Showing 1–7 of 7 results for author: Earls, C J

LLMs learn governing principles of dynamical systems, revealing an in-context neural scaling law

Authors: Toni J. B. Liu, Nicolas Boullé, Raphaël Sarfati, Christopher J. Earls

Abstract: Pretrained large language models (LLMs) are surprisingly effective at performing zero-shot tasks, including time-series forecasting. However, understanding the mechanisms behind such capabilities remains highly challenging due to the complexity of the models. We study LLMs' ability to extrapolate the behavior of dynamical systems whose evolution is governed by principles of physical interest. Our… ▽ More Pretrained large language models (LLMs) are surprisingly effective at performing zero-shot tasks, including time-series forecasting. However, understanding the mechanisms behind such capabilities remains highly challenging due to the complexity of the models. We study LLMs' ability to extrapolate the behavior of dynamical systems whose evolution is governed by principles of physical interest. Our results show that LLaMA 2, a language model trained primarily on texts, achieves accurate predictions of dynamical system time series without fine-tuning or prompt engineering. Moreover, the accuracy of the learned physical rules increases with the length of the input context window, revealing an in-context version of neural scaling law. Along the way, we present a flexible and efficient algorithm for extracting probability density functions of multi-digit numbers directly from LLMs. △ Less

Submitted 20 June, 2024; v1 submitted 1 February, 2024; originally announced February 2024.

Bayesian Deep Learning for Partial Differential Equation Parameter Discovery with Sparse and Noisy Data

Authors: Christophe Bonneville, Christopher J. Earls

Abstract: Scientific machine learning has been successfully applied to inverse problems and PDE discovery in computational physics. One caveat concerning current methods is the need for large amounts of ("clean") data, in order to characterize the full system response and discover underlying physical models. Bayesian methods may be particularly promising for overcoming these challenges, as they are naturall… ▽ More Scientific machine learning has been successfully applied to inverse problems and PDE discovery in computational physics. One caveat concerning current methods is the need for large amounts of ("clean") data, in order to characterize the full system response and discover underlying physical models. Bayesian methods may be particularly promising for overcoming these challenges, as they are naturally less sensitive to the negative effects of sparse and noisy data. In this paper, we propose to use Bayesian neural networks (BNN) in order to: 1) Recover the full system states from measurement data (e.g. temperature, velocity field, etc.). We use Hamiltonian Monte-Carlo to sample the posterior distribution of a deep and dense BNN, and show that it is possible to accurately capture physics of varying complexity, without overfitting. 2) Recover the parameters instantiating the underlying partial differential equation (PDE) governing the physical system. Using the trained BNN, as a surrogate of the system response, we generate datasets of derivatives that are potentially comprising the latent PDE governing the observed system and then perform a sequential threshold Bayesian linear regression (STBLR), between the successive derivatives in space and time, to recover the original PDE parameters. We take advantage of the confidence intervals within the BNN outputs, and introduce the spatial derivatives cumulative variance into the STBLR likelihood, to mitigate the influence of highly uncertain derivative data points; thus allowing for more accurate parameter discovery. We demonstrate our approach on a handful of example, in applied physics and non-linear dynamics. △ Less

Submitted 29 November, 2021; v1 submitted 5 August, 2021; originally announced August 2021.

Data-driven discovery of Green's functions with human-understandable deep learning

Authors: Nicolas Boullé, Christopher J. Earls, Alex Townsend

Abstract: There is an opportunity for deep learning to revolutionize science and technology by revealing its findings in a human interpretable manner. To do this, we develop a novel data-driven approach for creating a human-machine partnership to accelerate scientific discovery. By collecting physical system responses under excitations drawn from a Gaussian process, we train rational neural networks to lear… ▽ More There is an opportunity for deep learning to revolutionize science and technology by revealing its findings in a human interpretable manner. To do this, we develop a novel data-driven approach for creating a human-machine partnership to accelerate scientific discovery. By collecting physical system responses under excitations drawn from a Gaussian process, we train rational neural networks to learn Green's functions of hidden linear partial differential equations. These functions reveal human-understandable properties and features, such as linear conservation laws and symmetries, along with shock and singularity locations, boundary effects, and dominant modes. We illustrate the technique on several examples and capture a range of physics, including advection-diffusion, viscous shocks, and Stokes flow in a lid-driven cavity. △ Less

Submitted 11 March, 2022; v1 submitted 1 May, 2021; originally announced May 2021.

Comments: 54 pages, 23 figures

A Principled Approach to Design Using High Fidelity Fluid-Structure Interaction Simulations

Authors: Wensi Wu, Christophe Bonneville, Christopher J. Earls

Abstract: A high fidelity fluid-structure interaction simulation may require many days to run, on hundreds of cores. This poses a serious burden, both in terms of time and economic considerations, when repetitions of such simulations may be required (e.g. for the purpose of design optimization). In this paper we present strategies based on (constrained) Bayesian optimization (BO) to alleviate this burden. B… ▽ More A high fidelity fluid-structure interaction simulation may require many days to run, on hundreds of cores. This poses a serious burden, both in terms of time and economic considerations, when repetitions of such simulations may be required (e.g. for the purpose of design optimization). In this paper we present strategies based on (constrained) Bayesian optimization (BO) to alleviate this burden. BO is a numerical optimization technique based on Gaussian processes (GP) that is able to efficiently (with minimal calls to the expensive FSI models) converge towards some globally optimal design, as gauged using a black box objective function. In this study we present a principled design evolution that moves from FSI model verification, through a series of Bridge Simulations (bringing the verification case incrementally closer to the application), in order that we may identify material properties for an underwater, unmanned, autonomous vehicle (UUAV) sail plane. We are able to achieve fast convergence towards an optimal design, using a small number of FSI simulations (a dozen at most), even when selecting over several design parameters, and while respecting optimization constraints. △ Less

Submitted 21 August, 2020; originally announced August 2020.

Deep Learning for Classifying and Characterizing Atmospheric Ducting within the Maritime Setting

Authors: Hilarie Sit, Christopher J. Earls

Abstract: Real-time characterization of refractivity within the marine atmospheric boundary layer can provide valuable information that can potentially be used to mitigate the effects of atmospheric ducting on radar performance. Many duct characterization models are successful at predicting parameters from a specific refractivity profile associated with a given type of duct; however, the ability to classify… ▽ More Real-time characterization of refractivity within the marine atmospheric boundary layer can provide valuable information that can potentially be used to mitigate the effects of atmospheric ducting on radar performance. Many duct characterization models are successful at predicting parameters from a specific refractivity profile associated with a given type of duct; however, the ability to classify, and then subsequently characterize, various duct types is an important step towards a more comprehensive prediction model. We introduce a two-step approach using deep learning to differentiate sparsely sampled propagation factor measurements collected under evaporation ducting conditions with those collected under surface-based duct conditions in order to subsequently estimate the appropriate refractivity parameters based on that differentiation. We show that this approach is not only accurate, but also efficient; thus providing a suitable method for real-time applications. △ Less

Submitted 13 May, 2020; originally announced May 2020.

Comments: 13 pages, 3 figures

Gaussian Process Regression for Estimating EM Ducting Within the Marine Atmospheric Boundary Layer

Authors: Hilarie Sit, Christopher J. Earls

Abstract: We show that Gaussian process regression (GPR) can be used to infer the electromagnetic (EM) duct height within the marine atmospheric boundary layer (MABL) from sparsely sampled propagation factors within the context of bistatic radars. We use GPR to calculate the posterior predictive distribution on the labels (i.e. duct height) from both noise-free and noise-contaminated array of propagation fa… ▽ More We show that Gaussian process regression (GPR) can be used to infer the electromagnetic (EM) duct height within the marine atmospheric boundary layer (MABL) from sparsely sampled propagation factors within the context of bistatic radars. We use GPR to calculate the posterior predictive distribution on the labels (i.e. duct height) from both noise-free and noise-contaminated array of propagation factors. For duct height inference from noise-contaminated propagation factors, we compare a naive approach, utilizing one random sample from the input distribution (i.e. disregarding the input noise), with an inverse-variance weighted approach, utilizing a few random samples to estimate the true predictive distribution. The resulting posterior predictive distributions from these two approaches are compared to a "ground truth" distribution, which is approximated using a large number of Monte-Carlo samples. The ability of GPR to yield accurate and fast duct height predictions using a few training examples indicates the suitability of the proposed method for real-time applications. △ Less

Submitted 14 December, 2019; v1 submitted 25 May, 2019; originally announced May 2019.

Comments: 15 pages, 6 figures

Characterizing Evaporation Ducts Within the Marine Atmospheric Boundary Layer Using Artificial Neural Networks

Authors: Hilarie Sit, Christopher J. Earls

Abstract: We apply a multilayer perceptron machine learning (ML) regression approach to infer electromagnetic (EM) duct heights within the marine atmospheric boundary layer (MABL) using sparsely sampled EM propagation data obtained within a bistatic context. This paper explains the rationale behind the selection of the ML network architecture, along with other model hyperparameters, in an effort to demystif… ▽ More We apply a multilayer perceptron machine learning (ML) regression approach to infer electromagnetic (EM) duct heights within the marine atmospheric boundary layer (MABL) using sparsely sampled EM propagation data obtained within a bistatic context. This paper explains the rationale behind the selection of the ML network architecture, along with other model hyperparameters, in an effort to demystify the process of arriving at a useful ML model. The resulting speed of our ML predictions of EM duct heights, using sparse data measurements within MABL, indicates the suitability of the proposed method for real-time applications. △ Less

Submitted 26 August, 2019; v1 submitted 7 March, 2019; originally announced March 2019.

Comments: 13 pages, 7 figures