-
Uncertainty Quantification of Graph Convolution Neural Network Models of Evolving Processes
Authors:
Jeremiah Hauth,
Cosmin Safta,
Xun Huan,
Ravi G. Patel,
Reese E. Jones
Abstract:
The application of neural network models to scientific machine learning tasks has proliferated in recent years. In particular, neural network models have proved to be adept at modeling processes with spatial-temporal complexity. Nevertheless, these highly parameterized models have garnered skepticism in their ability to produce outputs with quantified error bounds over the regimes of interest. Hen…
▽ More
The application of neural network models to scientific machine learning tasks has proliferated in recent years. In particular, neural network models have proved to be adept at modeling processes with spatial-temporal complexity. Nevertheless, these highly parameterized models have garnered skepticism in their ability to produce outputs with quantified error bounds over the regimes of interest. Hence there is a need to find uncertainty quantification methods that are suitable for neural networks. In this work we present comparisons of the parametric uncertainty quantification of neural networks modeling complex spatial-temporal processes with Hamiltonian Monte Carlo and Stein variational gradient descent and its projected variant. Specifically we apply these methods to graph convolutional neural network models of evolving systems modeled with recurrent neural network and neural ordinary differential equations architectures. We show that Stein variational inference is a viable alternative to Monte Carlo methods with some clear advantages for complex neural network models. For our exemplars, Stein variational interference gave similar uncertainty profiles through time compared to Hamiltonian Monte Carlo, albeit with generally more generous variance.Projected Stein variational gradient descent also produced similar uncertainty profiles to the non-projected counterpart, but large reductions in the active weight space were confounded by the stability of the neural network predictions and the convoluted likelihood landscape.
△ Less
Submitted 16 February, 2024;
originally announced February 2024.
-
Enhancing Dynamical System Modeling through Interpretable Machine Learning Augmentations: A Case Study in Cathodic Electrophoretic Deposition
Authors:
Christian Jacobsen,
Jiayuan Dong,
Mehdi Khalloufi,
Xun Huan,
Karthik Duraisamy,
Maryam Akram,
Wanjiao Liu
Abstract:
We introduce a comprehensive data-driven framework aimed at enhancing the modeling of physical systems, employing inference techniques and machine learning enhancements. As a demonstrative application, we pursue the modeling of cathodic electrophoretic deposition (EPD), commonly known as e-coating. Our approach illustrates a systematic procedure for enhancing physical models by identifying their l…
▽ More
We introduce a comprehensive data-driven framework aimed at enhancing the modeling of physical systems, employing inference techniques and machine learning enhancements. As a demonstrative application, we pursue the modeling of cathodic electrophoretic deposition (EPD), commonly known as e-coating. Our approach illustrates a systematic procedure for enhancing physical models by identifying their limitations through inference on experimental data and introducing adaptable model enhancements to address these shortcomings. We begin by tackling the issue of model parameter identifiability, which reveals aspects of the model that require improvement. To address generalizability , we introduce modifications which also enhance identifiability. However, these modifications do not fully capture essential experimental behaviors. To overcome this limitation, we incorporate interpretable yet flexible augmentations into the baseline model. These augmentations are parameterized by simple fully-connected neural networks (FNNs), and we leverage machine learning tools, particularly Neural Ordinary Differential Equations (Neural ODEs), to learn these augmentations. Our simulations demonstrate that the machine learning-augmented model more accurately captures observed behaviors and improves predictive accuracy. Nevertheless, we contend that while the model updates offer superior performance and capture the relevant physics, we can reduce off-line computational costs by eliminating certain dynamics without compromising accuracy or interpretability in downstream predictions of quantities of interest, particularly film thickness predictions. The entire process outlined here provides a structured approach to leverage data-driven methods. Firstly, it helps us comprehend the root causes of model inaccuracies, and secondly, it offers a principled method for enhancing model performance.
△ Less
Submitted 16 January, 2024;
originally announced January 2024.
-
FP-IRL: Fokker-Planck-based Inverse Reinforcement Learning -- A Physics-Constrained Approach to Markov Decision Processes
Authors:
Chengyang Huang,
Siddhartha Srivastava,
Xun Huan,
Krishna Garikipati
Abstract:
Inverse Reinforcement Learning (IRL) is a compelling technique for revealing the rationale underlying the behavior of autonomous agents. IRL seeks to estimate the unknown reward function of a Markov decision process (MDP) from observed agent trajectories. However, IRL needs a transition function, and most algorithms assume it is known or can be estimated in advance from data. It therefore becomes…
▽ More
Inverse Reinforcement Learning (IRL) is a compelling technique for revealing the rationale underlying the behavior of autonomous agents. IRL seeks to estimate the unknown reward function of a Markov decision process (MDP) from observed agent trajectories. However, IRL needs a transition function, and most algorithms assume it is known or can be estimated in advance from data. It therefore becomes even more challenging when such transition dynamics is not known a-priori, since it enters the estimation of the policy in addition to determining the system's evolution. When the dynamics of these agents in the state-action space is described by stochastic differential equations (SDE) in It^{o} calculus, these transitions can be inferred from the mean-field theory described by the Fokker-Planck (FP) equation. We conjecture there exists an isomorphism between the time-discrete FP and MDP that extends beyond the minimization of free energy (in FP) and maximization of the reward (in MDP). We identify specific manifestations of this isomorphism and use them to create a novel physics-aware IRL algorithm, FP-IRL, which can simultaneously infer the transition and reward functions using only observed trajectories. We employ variational system identification to infer the potential function in FP, which consequently allows the evaluation of reward, transition, and policy by leveraging the conjecture. We demonstrate the effectiveness of FP-IRL by applying it to a synthetic benchmark and a biological problem of cancer cell dynamics, where the transition function is inaccessible.
△ Less
Submitted 17 June, 2023;
originally announced June 2023.
-
Multi-fidelity uncertainty quantification of particle deposition in turbulent pipe flow
Authors:
Yuan Yao,
Xun Huan,
Jesse Capecelatro
Abstract:
Particle deposition in fully-developed turbulent pipe flow is quantified taking into account uncertainty in electric charge, van der Waals strength, and temperature effects. A framework is presented for obtaining variance-based sensitivity in multiphase flow systems via a multi-fidelity Monte Carlo approach that optimally manages model evaluations for a given computational budget. The approach com…
▽ More
Particle deposition in fully-developed turbulent pipe flow is quantified taking into account uncertainty in electric charge, van der Waals strength, and temperature effects. A framework is presented for obtaining variance-based sensitivity in multiphase flow systems via a multi-fidelity Monte Carlo approach that optimally manages model evaluations for a given computational budget. The approach combines a high-fidelity model based on direct numerical simulation and a lower-order model based on a one-dimensional Eulerian description of the two-phase flow. Significant speedup is obtained compared to classical Monte Carlo estimation. Deposition is found to be most sensitive to electrostatic interactions and exhibits largest uncertainty for mid-sized (i.e., moderate Stokes number) particles.
△ Less
Submitted 10 May, 2022;
originally announced May 2022.
-
A Perspective on Regression and Bayesian Approaches for System Identification of Pattern Formation Dynamics
Authors:
Zhenlin Wang,
Bowei Wu,
Krishna Garikipati,
Xun Huan
Abstract:
We present two approaches to system identification, i.e. the identification of partial differential equations (PDEs) from measurement data. The first is a regression-based Variational System Identification procedure that is advantageous in not requiring repeated forward model solves and has good scalability to large number of differential operators. However it has strict data type requirements nee…
▽ More
We present two approaches to system identification, i.e. the identification of partial differential equations (PDEs) from measurement data. The first is a regression-based Variational System Identification procedure that is advantageous in not requiring repeated forward model solves and has good scalability to large number of differential operators. However it has strict data type requirements needing the ability to directly represent the operators through the available data. The second is a Bayesian inference framework highly valuable for providing uncertainty quantification, and flexible for accommodating sparse and noisy data that may also be indirect quantities of interest. However, it also requires repeated forward solutions of the PDE models which is expensive and hinders scalability. We provide illustrations of results on a model problem for pattern formation dynamics, and discuss merits of the presented methods.
△ Less
Submitted 7 March, 2020; v1 submitted 15 January, 2020;
originally announced January 2020.
-
Variational system identification of the partial differential equations governing microstructure evolution in materials: Inference over sparse and spatially unrelated data
Authors:
Z. Wang,
X. Huan,
K. Garikipati
Abstract:
Pattern formation is a widely observed phenomenon in diverse fields including materials physics, developmental biology and ecology, among many others. The physics underlying the patterns is specific to the mechanisms, and is encoded by partial differential equations (PDEs). With the aim of discovering hidden physics, we have previously presented a variational approach to identifying such systems o…
▽ More
Pattern formation is a widely observed phenomenon in diverse fields including materials physics, developmental biology and ecology, among many others. The physics underlying the patterns is specific to the mechanisms, and is encoded by partial differential equations (PDEs). With the aim of discovering hidden physics, we have previously presented a variational approach to identifying such systems of PDEs in the face of noisy data at varying fidelities (Computer Methods in Applied Mechanics and Engineering, 353:201-216, 2019). Here, we extend our variational system identification methods to address the challenges presented by image data on microstructures in materials physics. PDEs are formally posed as initial and boundary value problems over combinations of time intervals and spatial domains whose evolution is either fixed or can be tracked. However, the vast majority of microscopy techniques for evolving microstructure in a given material system deliver micrographs of pattern evolution over domains that bear no relation with each other at different time instants. The temporal resolution can rarely capture the fastest time scales that dominate the early dynamics, and noise abounds. Furthermore, data for evolution of the same phenomenon in a material system may well be obtained from different physical specimens. Against this backdrop of spatially unrelated, sparse and multi-source data, we exploit the variational framework to make judicious choices of weighting functions and identify PDE operators from the dynamics. A consistency condition arises for parsimonious inference of a minimal set of the spatial operators at steady state. It is complemented by a confirmation test that provides a sharp condition for acceptance of the inferred operators. The entire framework is demonstrated on synthetic data that reflect the characteristics of the experimental material microscopy images.
△ Less
Submitted 29 January, 2021; v1 submitted 11 January, 2020;
originally announced January 2020.
-
Variational system identification of the partial differential equations governing pattern-forming physics: Inference under varying fidelity and noise
Authors:
Zhenlin Wang,
Xun Huan,
Krishna Garikipati
Abstract:
We present a contribution to the field of system identification of partial differential equations (PDEs), with emphasis on discerning between competing mathematical models of pattern-forming physics. The motivation comes from developmental biology, where pattern formation is central to the development of any multicellular organism, and from materials physics, where phase transitions similarly lead…
▽ More
We present a contribution to the field of system identification of partial differential equations (PDEs), with emphasis on discerning between competing mathematical models of pattern-forming physics. The motivation comes from developmental biology, where pattern formation is central to the development of any multicellular organism, and from materials physics, where phase transitions similarly lead to microstructure. In both these fields there is a collection of nonlinear, parabolic PDEs that, over suitable parameter intervals and regimes of physics, can resolve the patterns or microstructures with comparable fidelity. This observation frames the question of which PDE best describes the data at hand. This question is particularly compelling because identification of the closest representation to the true PDE, while constrained by the functional spaces considered relative to the data at hand, immediately delivers insights to the physics underlying the systems. While building on recent work that uses stepwise regression, we present advances that leverage the variational framework and statistical tests. We also address the influences of variable fidelity and noise in the data.
△ Less
Submitted 18 June, 2019; v1 submitted 28 December, 2018;
originally announced December 2018.
-
Embedded Model Error Representation for Bayesian Model Calibration
Authors:
Khachik Sargsyan,
Xun Huan,
Habib N. Najm
Abstract:
Model error estimation remains one of the key challenges in uncertainty quantification and predictive science. For computational models of complex physical systems, model error, also known as structural error or model inadequacy, is often the largest contributor to the overall predictive uncertainty. This work builds on a recently developed framework of embedded, internal model correction, in orde…
▽ More
Model error estimation remains one of the key challenges in uncertainty quantification and predictive science. For computational models of complex physical systems, model error, also known as structural error or model inadequacy, is often the largest contributor to the overall predictive uncertainty. This work builds on a recently developed framework of embedded, internal model correction, in order to represent and quantify structural errors, together with model parameters, within a Bayesian inference context. We focus specifically on a Polynomial Chaos representation with additive modification of existing model parameters, enabling a non-intrusive procedure for efficient approximate likelihood construction, model error estimation, and disambiguation of model and data errors' contributions to predictive uncertainty. The framework is demonstrated on several synthetic examples, as well as on a chemical ignition problem.
△ Less
Submitted 12 February, 2019; v1 submitted 20 January, 2018;
originally announced January 2018.
-
Global Sensitivity Analysis and Estimation of Model Error, Toward Uncertainty Quantification in Scramjet Computations
Authors:
Xun Huan,
Cosmin Safta,
Khachik Sargsyan,
Gianluca Geraci,
Michael S. Eldred,
Zachary P. Vane,
Guilhem Lacaze,
Joseph C. Oefelein,
Habib N. Najm
Abstract:
The development of scramjet engines is an important research area for advancing hypersonic and orbital flights. Progress toward optimal engine designs requires accurate flow simulations together with uncertainty quantification. However, performing uncertainty quantification for scramjet simulations is challenging due to the large number of uncertain parameters involved and the high computational c…
▽ More
The development of scramjet engines is an important research area for advancing hypersonic and orbital flights. Progress toward optimal engine designs requires accurate flow simulations together with uncertainty quantification. However, performing uncertainty quantification for scramjet simulations is challenging due to the large number of uncertain parameters involved and the high computational cost of flow simulations. These difficulties are addressed in this paper by develo** practical uncertainty quantification algorithms and computational methods, and deploying them in the current study to large-eddy simulations of a jet in crossflow inside a simplified HIFiRE Direct Connect Rig scramjet combustor. First, global sensitivity analysis is conducted to identify influential uncertain input parameters, which can help reduce the systems stochastic dimension. Second, because models of different fidelity are used in the overall uncertainty quantification assessment, a framework for quantifying and propagating the uncertainty due to model error is presented. These methods are demonstrated on a nonreacting jet-in-crossflow test problem in a simplified scramjet geometry, with parameter space up to 24 dimensions, using static and dynamic treatments of the turbulence subgrid model, and with two-dimensional and three-dimensional geometries.
△ Less
Submitted 16 February, 2018; v1 submitted 29 July, 2017;
originally announced July 2017.
-
Compressive Sensing with Cross-Validation and Stop-Sampling for Sparse Polynomial Chaos Expansions
Authors:
Xun Huan,
Cosmin Safta,
Khachik Sargsyan,
Zachary P. Vane,
Guilhem Lacaze,
Joseph C. Oefelein,
Habib N. Najm
Abstract:
Compressive sensing is a powerful technique for recovering sparse solutions of underdetermined linear systems, which is often encountered in uncertainty quantification analysis of expensive and high-dimensional physical models. We perform numerical investigations employing several compressive sensing solvers that target the unconstrained LASSO formulation, with a focus on linear systems that arise…
▽ More
Compressive sensing is a powerful technique for recovering sparse solutions of underdetermined linear systems, which is often encountered in uncertainty quantification analysis of expensive and high-dimensional physical models. We perform numerical investigations employing several compressive sensing solvers that target the unconstrained LASSO formulation, with a focus on linear systems that arise in the construction of polynomial chaos expansions. With core solvers of l1_ls, SpaRSA, CGIST, FPC_AS, and ADMM, we develop techniques to mitigate overfitting through an automated selection of regularization constant based on cross-validation, and a heuristic strategy to guide the stop-sampling decision. Practical recommendations on parameter settings for these techniques are provided and discussed. The overall method is applied to a series of numerical examples of increasing complexity, including large eddy simulations of supersonic turbulent jet-in-crossflow involving a 24-dimensional input. Through empirical phase-transition diagrams and convergence plots, we illustrate sparse recovery performance under structures induced by polynomial chaos, accuracy and computational tradeoffs between polynomial bases of different degrees, and practicability of conducting compressive sensing for a realistic, high-dimensional physical application. Across test cases studied in this paper, we find ADMM to have demonstrated empirical advantages through consistent lower errors and faster computational times.
△ Less
Submitted 26 June, 2018; v1 submitted 28 July, 2017;
originally announced July 2017.
-
Design Analysis for Optimal Calibration of Diffusivity in Reactive Multilayers
Authors:
Manav Vohra,
Xun Huan,
Timothy P. Weihs,
Omar M. Knio
Abstract:
Calibration of the uncertain Arrhenius diffusion parameters for quantifying mixing rates in Zr-Al nanolaminate foils was performed in a Bayesian setting [Vohra et al., 2014]. The parameters were inferred in a low temperature regime characterized by homogeneous ignition and a high temperature regime characterized by self-propagating reactions in the multilayers. In this work, we extend the analysis…
▽ More
Calibration of the uncertain Arrhenius diffusion parameters for quantifying mixing rates in Zr-Al nanolaminate foils was performed in a Bayesian setting [Vohra et al., 2014]. The parameters were inferred in a low temperature regime characterized by homogeneous ignition and a high temperature regime characterized by self-propagating reactions in the multilayers. In this work, we extend the analysis to find optimal experimental designs that would provide the best data for inference. We employ a rigorous framework that quantifies the expected in- formation gain in an experiment, and find the optimal design conditions using numerical techniques of Monte Carlo, sparse quadrature, and polynomial chaos surrogates. For the low temperature regime, we find the optimal foil heating rate and pulse duration, and confirm through simulation that the optimal design indeed leads to sharper posterior distributions of the diffusion parameters. For the high temperature regime, we demonstrate potential for increase in the expected information gain of the posteriors by increasing sample size and reducing uncertainty in measurements. Moreover, posterior marginals are also produced to verify favorable experimental scenarios for this regime.
△ Less
Submitted 8 October, 2016;
originally announced October 2016.
