Showing 51–100 of 197 results for author: Karniadakis, G E

Dynamic spreading and infiltration of a molten sand droplet on a porous surface

Authors: Rahul Babu Koneru, Garrett Foresman, Alison Flatau, Zhen Li, Luis Bravo, Muthuvel Murugan, Anindya Ghoshal, George Em Karniadakis

Abstract: Compared to smooth surfaces, droplet spreading on porous surfaces is more complex and has relevance in many engineering applications. In this work, we investigate the infiltration dynamics of molten sand droplets on structured porous surfaces using the multiphase many-body dissipative particle dynamics (mDPD) method. We carry out three-dimensional simulations with different equilibrium contact ang… ▽ More Compared to smooth surfaces, droplet spreading on porous surfaces is more complex and has relevance in many engineering applications. In this work, we investigate the infiltration dynamics of molten sand droplets on structured porous surfaces using the multiphase many-body dissipative particle dynamics (mDPD) method. We carry out three-dimensional simulations with different equilibrium contact angles and surface porosities. The temporal evolution of the radius of the wetted area follows a power law, as in the case of a smooth surface. The infiltration rate on the other hand is dictated by the competition between spreading and capillary inhibition of the pores. Additionally, the temporal evolution of the droplet height and the contact angle on the porous surface is also presented. △ Less

Submitted 6 March, 2023; originally announced March 2023.

Neural Operator Learning for Long-Time Integration in Dynamical Systems with Recurrent Neural Networks

Authors: Katarzyna Michałowska, Somdatta Goswami, George Em Karniadakis, Signe Riemer-Sørensen

Abstract: Deep neural networks are an attractive alternative for simulating complex dynamical systems, as in comparison to traditional scientific computing methods, they offer reduced computational costs during inference and can be trained directly from observational data. Existing methods, however, cannot extrapolate accurately and are prone to error accumulation in long-time integration. Herein, we addres… ▽ More Deep neural networks are an attractive alternative for simulating complex dynamical systems, as in comparison to traditional scientific computing methods, they offer reduced computational costs during inference and can be trained directly from observational data. Existing methods, however, cannot extrapolate accurately and are prone to error accumulation in long-time integration. Herein, we address this issue by combining neural operators with recurrent neural networks, learning the operator map**, while offering a recurrent structure to capture temporal dependencies. The integrated framework is shown to stabilize the solution and reduce error accumulation for both interpolation and extrapolation of the Korteweg-de Vries equation. △ Less

Submitted 30 April, 2024; v1 submitted 3 March, 2023; originally announced March 2023.

Comments: 8 pages, 5 figures

A unified scalable framework for causal swee** strategies for Physics-Informed Neural Networks (PINNs) and their temporal decompositions

Authors: Michael Penwarden, Ameya D. Jagtap, Shandian Zhe, George Em Karniadakis, Robert M. Kirby

Abstract: Physics-informed neural networks (PINNs) as a means of solving partial differential equations (PDE) have garnered much attention in the Computational Science and Engineering (CS&E) world. However, a recent topic of interest is exploring various training (i.e., optimization) challenges - in particular, arriving at poor local minima in the optimization landscape results in a PINN approximation givin… ▽ More Physics-informed neural networks (PINNs) as a means of solving partial differential equations (PDE) have garnered much attention in the Computational Science and Engineering (CS&E) world. However, a recent topic of interest is exploring various training (i.e., optimization) challenges - in particular, arriving at poor local minima in the optimization landscape results in a PINN approximation giving an inferior, and sometimes trivial, solution when solving forward time-dependent PDEs with no data. This problem is also found in, and in some sense more difficult, with domain decomposition strategies such as temporal decomposition using XPINNs. We furnish examples and explanations for different training challenges, their cause, and how they relate to information propagation and temporal decomposition. We then propose a new stacked-decomposition method that bridges the gap between time-marching PINNs and XPINNs. We also introduce significant computational speed-ups by using transfer learning concepts to initialize subnetworks in the domain and loss tolerance-based propagation for the subdomains. Finally, we formulate a new time-swee** collocation point algorithm inspired by the previous PINNs causality literature, which our framework can still describe, and provides a significant computational speed-up via reduced-cost collocation point segmentation. The proposed methods form our unified framework, which overcomes training challenges in PINNs and XPINNs for time-dependent PDEs by respecting the causality in multiple forms and improving scalability by limiting the computation required per optimization iteration. Finally, we provide numerical results for these methods on baseline PDE problems for which unmodified PINNs and XPINNs struggle to train. △ Less

Submitted 18 September, 2023; v1 submitted 27 February, 2023; originally announced February 2023.

Journal ref: Journal of Computational Physics, 493, 2023, 112464

Learning stiff chemical kinetics using extended deep neural operators

Authors: Somdatta Goswami, Ameya D. Jagtap, Hessam Babaee, Bryan T. Susi, George Em Karniadakis

Abstract: We utilize neural operators to learn the solution propagator for the challenging chemical kinetics equation. Specifically, we apply the deep operator network (DeepONet) along with its extensions, such as the autoencoder-based DeepONet and the newly proposed Partition-of-Unity (PoU-) DeepONet to study a range of examples, including the ROBERS problem with three species, the POLLU problem with 25 sp… ▽ More We utilize neural operators to learn the solution propagator for the challenging chemical kinetics equation. Specifically, we apply the deep operator network (DeepONet) along with its extensions, such as the autoencoder-based DeepONet and the newly proposed Partition-of-Unity (PoU-) DeepONet to study a range of examples, including the ROBERS problem with three species, the POLLU problem with 25 species, pure kinetics of the syngas skeletal model for $CO/H_2$ burning, which contains 11 species and 21 reactions and finally, a temporally develo** planar $CO/H_2$ jet flame (turbulent flame) using the same syngas mechanism. We have demonstrated the advantages of the proposed approach through these numerical examples. Specifically, to train the DeepONet for the syngas model, we solve the skeletal kinetic model for different initial conditions. In the first case, we parametrize the initial conditions based on equivalence ratios and initial temperature values. In the second case, we perform a direct numerical simulation of a two-dimensional temporally develo** $CO/H_2$ jet flame. Then, we initialize the kinetic model by the thermochemical states visited by a subset of grid points at different time snapshots. Stiff problems are computationally expensive to solve with traditional stiff solvers. Thus, this work aims to develop a neural operator-based surrogate model to solve stiff chemical kinetics. The operator, once trained offline, can accurately integrate the thermochemical state for arbitrarily large time advancements, leading to significant computational gains compared to stiff integration schemes. △ Less

Submitted 23 February, 2023; originally announced February 2023.

Comments: 21 pages, 11 figures

Deep neural operators can predict the real-time response of floating offshore structures under irregular waves

Authors: Qianying Cao, Somdatta Goswami, Tapas Tripura, Souvik Chakraborty, George Em Karniadakis

Abstract: The use of neural operators in a digital twin model of an offshore floating structure can provide a paradigm shift in structural response prediction and health monitoring, providing valuable information for real-time control. In this work, the performance of three neural operators is evaluated, namely, deep operator network (DeepONet), Fourier neural operator (FNO), and Wavelet neural operator (WN… ▽ More The use of neural operators in a digital twin model of an offshore floating structure can provide a paradigm shift in structural response prediction and health monitoring, providing valuable information for real-time control. In this work, the performance of three neural operators is evaluated, namely, deep operator network (DeepONet), Fourier neural operator (FNO), and Wavelet neural operator (WNO). We investigate the effectiveness of the operators to accurately capture the responses of a floating structure under six different sea state codes $(3-8)$ based on the wave characteristics described by the World Meteorological Organization (WMO). The results demonstrate that these high-precision neural operators can deliver structural responses more efficiently, up to two orders of magnitude faster than a dynamic analysis using conventional numerical solvers. Additionally, compared to gated recurrent units (GRUs), a commonly used recurrent neural network for time-series estimation, neural operators are both more accurate and efficient, especially in situations with limited data availability. To further enhance the accuracy, novel extensions, such as wavelet-DeepONet and self-adaptive WNO, are proposed. Taken together, our study shows that FNO outperforms all other operators for approximating the map** of one input functional space to the output space as well as for responses that have small bandwidth of the frequency spectrum, whereas for learning the map** of multiple functions in the input space to the output space as well as for capturing responses within a large frequency spectrum, DeepONet with historical states provides the highest accuracy. △ Less

Submitted 30 November, 2023; v1 submitted 13 February, 2023; originally announced February 2023.

Comments: 40 pages, 16 figures, 13 tables

Learning bias corrections for climate models using deep neural operators

Authors: Aniruddha Bora, Khemraj Shukla, Shixuan Zhang, Bryce Harrop, Ruby Leung, George Em Karniadakis

Abstract: Numerical simulation for climate modeling resolving all important scales is a computationally taxing process. Therefore, to circumvent this issue a low resolution simulation is performed, which is subsequently corrected for bias using reanalyzed data (ERA5), known as nudging correction. The existing implementation for nudging correction uses a relaxation based method for the algebraic difference b… ▽ More Numerical simulation for climate modeling resolving all important scales is a computationally taxing process. Therefore, to circumvent this issue a low resolution simulation is performed, which is subsequently corrected for bias using reanalyzed data (ERA5), known as nudging correction. The existing implementation for nudging correction uses a relaxation based method for the algebraic difference between low resolution and ERA5 data. In this study, we replace the bias correction process with a surrogate model based on the Deep Operator Network (DeepONet). DeepONet (Deep Operator Neural Network) learns the map** from the state before nudging (a functional) to the nudging tendency (another functional). The nudging tendency is a very high dimensional data albeit having many low energy modes. Therefore, the DeepoNet is combined with a convolution based auto-encoder-decoder (AED) architecture in order to learn the nudging tendency in a lower dimensional latent space efficiently. The accuracy of the DeepONet model is tested against the nudging tendency obtained from the E3SMv2 (Energy Exascale Earth System Model) and shows good agreement. The overarching goal of this work is to deploy the DeepONet model in an online setting and replace the nudging module in the E3SM loop for better efficiency and accuracy. △ Less

Submitted 6 February, 2023; originally announced February 2023.

Deep neural operators can serve as accurate surrogates for shape optimization: A case study for airfoils

Authors: Khemraj Shukla, Vivek Oommen, Ahmad Peyvan, Michael Penwarden, Luis Bravo, Anindya Ghoshal, Robert M. Kirby, George Em Karniadakis

Abstract: Deep neural operators, such as DeepONets, have changed the paradigm in high-dimensional nonlinear regression from function regression to (differential) operator regression, paving the way for significant changes in computational engineering applications. Here, we investigate the use of DeepONets to infer flow fields around unseen airfoils with the aim of shape optimization, an important design pro… ▽ More Deep neural operators, such as DeepONets, have changed the paradigm in high-dimensional nonlinear regression from function regression to (differential) operator regression, paving the way for significant changes in computational engineering applications. Here, we investigate the use of DeepONets to infer flow fields around unseen airfoils with the aim of shape optimization, an important design problem in aerodynamics that typically taxes computational resources heavily. We present results which display little to no degradation in prediction accuracy, while reducing the online optimization cost by orders of magnitude. We consider NACA airfoils as a test case for our proposed approach, as their shape can be easily defined by the four-digit parametrization. We successfully optimize the constrained NACA four-digit problem with respect to maximizing the lift-to-drag ratio and validate all results by comparing them to a high-order CFD solver. We find that DeepONets have low generalization error, making them ideal for generating solutions of unseen shapes. Specifically, pressure, density, and velocity fields are accurately inferred at a fraction of a second, hence enabling the use of general objective functions beyond the maximization of the lift-to-drag ratio considered in the current work. △ Less

Submitted 1 February, 2023; originally announced February 2023.

Comments: 21 pages, 14 Figures

A Hybrid Deep Neural Operator/Finite Element Method for Ice-Sheet Modeling

Authors: QiZhi He, Mauro Perego, Amanda A. Howard, George Em Karniadakis, Panos Stinis

Abstract: One of the most challenging and consequential problems in climate modeling is to provide probabilistic projections of sea level rise. A large part of the uncertainty of sea level projections is due to uncertainty in ice sheet dynamics. At the moment, accurate quantification of the uncertainty is hindered by the cost of ice sheet computational models. In this work, we develop a hybrid approach to a… ▽ More One of the most challenging and consequential problems in climate modeling is to provide probabilistic projections of sea level rise. A large part of the uncertainty of sea level projections is due to uncertainty in ice sheet dynamics. At the moment, accurate quantification of the uncertainty is hindered by the cost of ice sheet computational models. In this work, we develop a hybrid approach to approximate existing ice sheet computational models at a fraction of their cost. Our approach consists of replacing the finite element model for the momentum equations for the ice velocity, the most expensive part of an ice sheet model, with a Deep Operator Network, while retaining a classic finite element discretization for the evolution of the ice thickness. We show that the resulting hybrid model is very accurate and it is an order of magnitude faster than the traditional finite element model. Further, a distinctive feature of the proposed model compared to other neural network approaches, is that it can handle high-dimensional parameter spaces (parameter fields) such as the basal friction at the bed of the glacier, and can therefore be used for generating samples for uncertainty quantification. We study the impact of hyper-parameters, number of unknowns and correlation length of the parameter distribution on the training and accuracy of the Deep Operator Network on a synthetic ice sheet model. We then target the evolution of the Humboldt glacier in Greenland and show that our hybrid model can provide accurate statistics of the glacier mass loss and can be effectively used to accelerate the quantification of uncertainty. △ Less

Submitted 26 January, 2023; originally announced January 2023.

L-HYDRA: Multi-Head Physics-Informed Neural Networks

Authors: Zongren Zou, George Em Karniadakis

Abstract: We introduce multi-head neural networks (MH-NNs) to physics-informed machine learning, which is a type of neural networks (NNs) with all nonlinear hidden layers as the body and multiple linear output layers as multi-head. Hence, we construct multi-head physics-informed neural networks (MH-PINNs) as a potent tool for multi-task learning (MTL), generative modeling, and few-shot learning for diverse… ▽ More We introduce multi-head neural networks (MH-NNs) to physics-informed machine learning, which is a type of neural networks (NNs) with all nonlinear hidden layers as the body and multiple linear output layers as multi-head. Hence, we construct multi-head physics-informed neural networks (MH-PINNs) as a potent tool for multi-task learning (MTL), generative modeling, and few-shot learning for diverse problems in scientific machine learning (SciML). MH-PINNs connect multiple functions/tasks via a shared body as the basis functions as well as a shared distribution for the head. The former is accomplished by solving multiple tasks with MH-PINNs with each head independently corresponding to each task, while the latter by employing normalizing flows (NFs) for density estimate and generative modeling. To this end, our method is a two-stage method, and both stages can be tackled with standard deep learning tools of NNs, enabling easy implementation in practice. MH-PINNs can be used for various purposes, such as approximating stochastic processes, solving multiple tasks synergistically, providing informative prior knowledge for downstream few-shot learning tasks such as meta-learning and transfer learning, learning representative basis functions, and uncertainty quantification. We demonstrate the effectiveness of MH-PINNs in five benchmarks, investigating also the possibility of synergistic learning in regression analysis. We name the open-source code "Lernaean Hydra" (L-HYDRA), since this mythical creature possessed many heads for performing important multiple tasks, as in the proposed method. △ Less

Submitted 5 January, 2023; originally announced January 2023.

MSC Class: 34F05; 62M45; 65L99; 65M99; 65N99

Reliable extrapolation of deep neural operators informed by physics or sparse observations

Authors: Min Zhu, Handi Zhang, Anran Jiao, George Em Karniadakis, Lu Lu

Abstract: Deep neural operators can learn nonlinear map**s between infinite-dimensional function spaces via deep neural networks. As promising surrogate solvers of partial differential equations (PDEs) for real-time prediction, deep neural operators such as deep operator networks (DeepONets) provide a new simulation paradigm in science and engineering. Pure data-driven neural operators and deep learning m… ▽ More Deep neural operators can learn nonlinear map**s between infinite-dimensional function spaces via deep neural networks. As promising surrogate solvers of partial differential equations (PDEs) for real-time prediction, deep neural operators such as deep operator networks (DeepONets) provide a new simulation paradigm in science and engineering. Pure data-driven neural operators and deep learning models, in general, are usually limited to interpolation scenarios, where new predictions utilize inputs within the support of the training set. However, in the inference stage of real-world applications, the input may lie outside the support, i.e., extrapolation is required, which may result to large errors and unavoidable failure of deep learning models. Here, we address this challenge of extrapolation for deep neural operators. First, we systematically investigate the extrapolation behavior of DeepONets by quantifying the extrapolation complexity via the 2-Wasserstein distance between two function spaces and propose a new behavior of bias-variance trade-off for extrapolation with respect to model capacity. Subsequently, we develop a complete workflow, including extrapolation determination, and we propose five reliable learning methods that guarantee a safe prediction under extrapolation by requiring additional information -- the governing PDEs of the system or sparse new observations. The proposed methods are based on either fine-tuning a pre-trained DeepONet or multifidelity learning. We demonstrate the effectiveness of the proposed framework for various types of parametric PDEs. Our systematic comparisons provide practical guidelines for selecting a proper extrapolation method depending on the available information, desired accuracy, and required inference speed. △ Less

Submitted 12 December, 2022; originally announced December 2022.

SMS: Spiking Marching Scheme for Efficient Long Time Integration of Differential Equations

Authors: Qian Zhang, Adar Kahana, George Em Karniadakis, Panos Stinis

Abstract: We propose a Spiking Neural Network (SNN)-based explicit numerical scheme for long time integration of time-dependent Ordinary and Partial Differential Equations (ODEs, PDEs). The core element of the method is a SNN, trained to use spike-encoded information about the solution at previous timesteps to predict spike-encoded information at the next timestep. After the network has been trained, it ope… ▽ More We propose a Spiking Neural Network (SNN)-based explicit numerical scheme for long time integration of time-dependent Ordinary and Partial Differential Equations (ODEs, PDEs). The core element of the method is a SNN, trained to use spike-encoded information about the solution at previous timesteps to predict spike-encoded information at the next timestep. After the network has been trained, it operates as an explicit numerical scheme that can be used to compute the solution at future timesteps, given a spike-encoded initial condition. A decoder is used to transform the evolved spiking-encoded solution back to function values. We present results from numerical experiments of using the proposed method for ODEs and PDEs of varying complexity. △ Less

Submitted 17 November, 2022; originally announced November 2022.

Comments: 14 pages, 7 figures

Report number: PNNL-SA-179601 MSC Class: 65M99

Augmented Physics-Informed Neural Networks (APINNs): A gating network-based soft domain decomposition methodology

Authors: Zheyuan Hu, Ameya D. Jagtap, George Em Karniadakis, Kenji Kawaguchi

Abstract: In this paper, we propose the augmented physics-informed neural network (APINN), which adopts soft and trainable domain decomposition and flexible parameter sharing to further improve the extended PINN (XPINN) as well as the vanilla PINN methods. In particular, a trainable gate network is employed to mimic the hard decomposition of XPINN, which can be flexibly fine-tuned for discovering a potentia… ▽ More In this paper, we propose the augmented physics-informed neural network (APINN), which adopts soft and trainable domain decomposition and flexible parameter sharing to further improve the extended PINN (XPINN) as well as the vanilla PINN methods. In particular, a trainable gate network is employed to mimic the hard decomposition of XPINN, which can be flexibly fine-tuned for discovering a potentially better partition. It weight-averages several sub-nets as the output of APINN. APINN does not require complex interface conditions, and its sub-nets can take advantage of all training samples rather than just part of the training data in their subdomains. Lastly, each sub-net shares part of the common parameters to capture the similar components in each decomposed function. Furthermore, following the PINN generalization theory in Hu et al. [2021], we show that APINN can improve generalization by proper gate network initialization and general domain & function decomposition. Extensive experiments on different types of PDEs demonstrate how APINN improves the PINN and XPINN methods. Specifically, we present examples where XPINN performs similarly to or worse than PINN, so that APINN can significantly improve both. We also show cases where XPINN is already better than PINN, so APINN can still slightly improve XPINN. Furthermore, we visualize the optimized gating networks and their optimization trajectories, and connect them with their performance, which helps discover the possibly optimal decomposition. Interestingly, if initialized by different decomposition, the performances of corresponding APINNs can differ drastically. This, in turn, shows the potential to design an optimal domain decomposition for the differential equation problem under consideration. △ Less

Submitted 29 September, 2023; v1 submitted 16 November, 2022; originally announced November 2022.

Comments: Accepted at Engineering Applications of Artificial Intelligence (EAAI)

Journal ref: Engineering Applications of Artificial Intelligence, Volume 126, Part B, November 2023, 107183

On the Geometry Transferability of the Hybrid Iterative Numerical Solver for Differential Equations

Authors: Adar Kahana, Enrui Zhang, Somdatta Goswami, George EM Karniadakis, Rishikesh Ranade, Jay Pathak

Abstract: The discovery of fast numerical solvers prompted a clear and rapid shift towards iterative techniques in many applications, especially in computational mechanics, due to the increased necessity for solving very large linear systems. Most numerical solvers are highly dependent on the problem geometry and discretization, facing issues when any of these properties change. The newly developed Hybrid I… ▽ More The discovery of fast numerical solvers prompted a clear and rapid shift towards iterative techniques in many applications, especially in computational mechanics, due to the increased necessity for solving very large linear systems. Most numerical solvers are highly dependent on the problem geometry and discretization, facing issues when any of these properties change. The newly developed Hybrid Iterative Numerical Transferable Solver (HINTS) combines a standard solver with a neural operator to achieve better performance, focusing on a single geometry at a time. In this work, we explore the "T" in HINTS, i.e., the geometry transferability properties of HINTS. We first propose to directly employ HINTS built for a specific geometry to a different but related geometry without any adjustments. In addition, we propose the integration of an operator level transfer learning with HINTS to even further improve the convergence of HINTS on new geometries and discretizations. We conduct numerical experiments for a Darcy flow problem and a plane-strain elasticity problem. The results show that both the direct application of HINTS and the transfer learning enhanced HINTS are able to accurately solve these problems on different geometries. In addition, using transfer learning, HINTS is able to converge to machine zero even faster than the direct application of HINTS. △ Less

Submitted 31 October, 2022; originally announced October 2022.

Comments: 12 pages, 8 figures, 1 table

How important are activation functions in regression and classification? A survey, performance comparison, and future directions

Authors: Ameya D. Jagtap, George Em Karniadakis

Abstract: Inspired by biological neurons, the activation functions play an essential part in the learning process of any artificial neural network commonly used in many real-world problems. Various activation functions have been proposed in the literature for classification as well as regression tasks. In this work, we survey the activation functions that have been employed in the past as well as the curren… ▽ More Inspired by biological neurons, the activation functions play an essential part in the learning process of any artificial neural network commonly used in many real-world problems. Various activation functions have been proposed in the literature for classification as well as regression tasks. In this work, we survey the activation functions that have been employed in the past as well as the current state-of-the-art. In particular, we present various developments in activation functions over the years and the advantages as well as disadvantages or limitations of these activation functions. We also discuss classical (fixed) activation functions, including rectifier units, and adaptive activation functions. In addition to discussing the taxonomy of activation functions based on characterization, a taxonomy of activation functions based on applications is presented. To this end, the systematic comparison of various fixed and adaptive activation functions is performed for classification data sets such as the MNIST, CIFAR-10, and CIFAR- 100. In recent years, a physics-informed machine learning framework has emerged for solving problems related to scientific computations. For this purpose, we also discuss various requirements for activation functions that have been used in the physics-informed machine learning framework. Furthermore, various comparisons are made among different fixed and adaptive activation functions using various machine learning libraries such as TensorFlow, Pytorch, and JAX. △ Less

Submitted 28 December, 2022; v1 submitted 6 September, 2022; originally announced September 2022.

Comments: 28 pages, 15 figures

A Hybrid Iterative Numerical Transferable Solver (HINTS) for PDEs Based on Deep Operator Network and Relaxation Methods

Authors: Enrui Zhang, Adar Kahana, Eli Turkel, Rishikesh Ranade, Jay Pathak, George Em Karniadakis

Abstract: Iterative solvers of linear systems are a key component for the numerical solutions of partial differential equations (PDEs). While there have been intensive studies through past decades on classical methods such as Jacobi, Gauss-Seidel, conjugate gradient, multigrid methods and their more advanced variants, there is still a pressing need to develop faster, more robust and reliable solvers. Based… ▽ More Iterative solvers of linear systems are a key component for the numerical solutions of partial differential equations (PDEs). While there have been intensive studies through past decades on classical methods such as Jacobi, Gauss-Seidel, conjugate gradient, multigrid methods and their more advanced variants, there is still a pressing need to develop faster, more robust and reliable solvers. Based on recent advances in scientific deep learning for operator regression, we propose HINTS, a hybrid, iterative, numerical, and transferable solver for differential equations. HINTS combines standard relaxation methods and the Deep Operator Network (DeepONet). Compared to standard numerical solvers, HINTS is capable of providing faster solutions for a wide class of differential equations, while preserving the accuracy close to machine zero. Through an eigenmode analysis, we find that the individual solvers in HINTS target distinct regions in the spectrum of eigenmodes, resulting in a uniform convergence rate and hence exceptional performance of the hybrid solver overall. Moreover, HINTS applies to equations in multidimensions, and is flexible with regards to computational domain and transferable to different discretizations. △ Less

Submitted 28 August, 2022; originally announced August 2022.

Comments: Main text: 28 pages, 6 figures. Supplementary Information: 18 pages, 11 figures, 4 tables, 4 algorithms

NeuralUQ: A comprehensive library for uncertainty quantification in neural differential equations and operators

Authors: Zongren Zou, Xuhui Meng, Apostolos F Psaros, George Em Karniadakis

Abstract: Uncertainty quantification (UQ) in machine learning is currently drawing increasing research interest, driven by the rapid deployment of deep neural networks across different fields, such as computer vision, natural language processing, and the need for reliable tools in risk-sensitive applications. Recently, various machine learning models have also been developed to tackle problems in the field… ▽ More Uncertainty quantification (UQ) in machine learning is currently drawing increasing research interest, driven by the rapid deployment of deep neural networks across different fields, such as computer vision, natural language processing, and the need for reliable tools in risk-sensitive applications. Recently, various machine learning models have also been developed to tackle problems in the field of scientific computing with applications to computational science and engineering (CSE). Physics-informed neural networks and deep operator networks are two such models for solving partial differential equations and learning operator map**s, respectively. In this regard, a comprehensive study of UQ methods tailored specifically for scientific machine learning (SciML) models has been provided in [45]. Nevertheless, and despite their theoretical merit, implementations of these methods are not straightforward, especially in large-scale CSE applications, hindering their broad adoption in both research and industry settings. In this paper, we present an open-source Python library (https://github.com/Crunch-UQ4MI), termed NeuralUQ and accompanied by an educational tutorial, for employing UQ methods for SciML in a convenient and structured manner. The library, designed for both educational and research purposes, supports multiple modern UQ methods and SciML models. It is based on a succinct workflow and facilitates flexible employment and easy extensions by the users. We first present a tutorial of NeuralUQ and subsequently demonstrate its applicability and efficiency in four diverse examples, involving dynamical systems and high-dimensional parametric and time-dependent PDEs. △ Less

Submitted 25 August, 2022; originally announced August 2022.

Comments: 27 pages, 12 figures

G2Φnet: Relating Genotype and Biomechanical Phenotype of Tissues with Deep Learning

Authors: Enrui Zhang, Bart Spronck, Jay D. Humphrey, George Em Karniadakis

Abstract: Many genetic mutations adversely affect the structure and function of load-bearing soft tissues, with clinical sequelae often responsible for disability or death. Parallel advances in genetics and histomechanical characterization provide significant insight into these conditions, but there remains a pressing need to integrate such information. We present a novel genotype-to-biomechanical-phenotype… ▽ More Many genetic mutations adversely affect the structure and function of load-bearing soft tissues, with clinical sequelae often responsible for disability or death. Parallel advances in genetics and histomechanical characterization provide significant insight into these conditions, but there remains a pressing need to integrate such information. We present a novel genotype-to-biomechanical-phenotype neural network (G2Φnet) for characterizing and classifying biomechanical properties of soft tissues, which serve as important functional readouts of tissue health or disease. We illustrate the utility of our approach by inferring the nonlinear, genotype-dependent constitutive behavior of the aorta for four mouse models involving defects or deficiencies in extracellular constituents. We show that G2Φnet can infer the biomechanical response while simultaneously ascribing the associated genotype correctly by utilizing limited, noisy, and unstructured experimental data. More broadly, G2Φnet provides a powerful method and a paradigm shift for correlating genotype and biomechanical phenotype quantitatively, promising a better understanding of their interplay in biological tissues. △ Less

Submitted 21 August, 2022; originally announced August 2022.

Comments: 41 pages, 9 figures

Physics-Informed Deep Neural Operator Networks

Authors: Somdatta Goswami, Aniruddha Bora, Yue Yu, George Em Karniadakis

Abstract: Standard neural networks can approximate general nonlinear operators, represented either explicitly by a combination of mathematical operators, e.g., in an advection-diffusion-reaction partial differential equation, or simply as a black box, e.g., a system-of-systems. The first neural operator was the Deep Operator Network (DeepONet), proposed in 2019 based on rigorous approximation theory. Since… ▽ More Standard neural networks can approximate general nonlinear operators, represented either explicitly by a combination of mathematical operators, e.g., in an advection-diffusion-reaction partial differential equation, or simply as a black box, e.g., a system-of-systems. The first neural operator was the Deep Operator Network (DeepONet), proposed in 2019 based on rigorous approximation theory. Since then, a few other less general operators have been published, e.g., based on graph neural networks or Fourier transforms. For black box systems, training of neural operators is data-driven only but if the governing equations are known they can be incorporated into the loss function during training to develop physics-informed neural operators. Neural operators can be used as surrogates in design problems, uncertainty quantification, autonomous systems, and almost in any application requiring real-time inference. Moreover, independently pre-trained DeepONets can be used as components of a complex multi-physics system by coupling them together with relatively light training. Here, we present a review of DeepONet, the Fourier neural operator, and the graph neural operator, as well as appropriate extensions with feature expansions, and highlight their usefulness in diverse applications in computational mechanics, including porous media, fluid mechanics, and solid mechanics. △ Less

Submitted 17 July, 2022; v1 submitted 8 July, 2022; originally announced July 2022.

Comments: 33 pages, 14 figures. arXiv admin note: text overlap with arXiv:2204.00997 by other authors

Fractional SEIR Model and Data-Driven Predictions of COVID-19 Dynamics of Omicron Variant

Authors: Min Cai, George Em Karniadakis, Changpin Li

Abstract: We study the dynamic evolution of COVID-19 cased by the Omicron variant via a fractional susceptible-exposedinfected-removed (SEIR) model. Preliminary data suggest that the symptoms of Omicron infection are not prominent and the transmission is therefore more concealed, which causes a relatively slow increase in the detected cases of the new infected at the beginning of the pandemic. To characteri… ▽ More We study the dynamic evolution of COVID-19 cased by the Omicron variant via a fractional susceptible-exposedinfected-removed (SEIR) model. Preliminary data suggest that the symptoms of Omicron infection are not prominent and the transmission is therefore more concealed, which causes a relatively slow increase in the detected cases of the new infected at the beginning of the pandemic. To characterize the specific dynamics, the Caputo-Hadamard fractional derivative is adopted to refined the classical SEIR model. Based on the reported data, we infer the fractional order, timedependent parameters, as well as unobserved dynamics of the fractional SEIR model via fractional physics-informed neural networks (fPINNs). Then, we make short-time predictions using the learned fractional SEIR model. △ Less

Submitted 23 May, 2022; originally announced May 2022.

Journal ref: Chaos 32, 071101 (2022)

Spiking Neural Operators for Scientific Machine Learning

Authors: Adar Kahana, Qian Zhang, Leonard Gleyzer, George Em Karniadakis

Abstract: The main computational task of Scientific Machine Learning (SciML) is function regression, required both for inputs as well as outputs of a simulation. Physics-Informed Neural Networks (PINNs) and neural operators (such as DeepONet) have been very effective in solving Partial Differential Equations (PDEs), but they tax computational resources heavily and cannot be readily adopted for edge computin… ▽ More The main computational task of Scientific Machine Learning (SciML) is function regression, required both for inputs as well as outputs of a simulation. Physics-Informed Neural Networks (PINNs) and neural operators (such as DeepONet) have been very effective in solving Partial Differential Equations (PDEs), but they tax computational resources heavily and cannot be readily adopted for edge computing. Here, we address this issue by considering Spiking Neural Networks (SNNs), which have shown promise in reducing energy consumption by two orders of magnitude or more. We present a SNN-based method to perform regression, which has been a challenge due to the inherent difficulty in representing a function's input domain and continuous output values as spikes. We first propose a new method for encoding continuous values into spikes based on a triangular matrix in space and time, and demonstrate its better performance compared to the existing methods. Next, we demonstrate that using a simple SNN architecture consisting of Leaky Integrate and Fire (LIF) activation and two dense layers, we can achieve relatively accurate function regression results. Moreover, we can replace the LIF with a trained Multi-Layer Perceptron (MLP) network and obtain comparable results but three times faster. Then, we introduce the DeepONet, consisting of a branch (typically a Fully-connected Neural Network, FNN) for inputs and a trunk (also a FNN) for outputs. We can build a spiking DeepONet by either replacing the branch or the trunk by a SNN. We demonstrate this new approach for classification using the SNN in the branch, achieving results comparable to the literature. Finally, we design a spiking DeepONet for regression by replacing its trunk with a SNN, and achieve good accuracy for approximating functions as well as inferring solutions of differential equations. △ Less

Submitted 12 October, 2022; v1 submitted 17 May, 2022; originally announced May 2022.

Comments: 16 pages, 6 figures and 4 tables

Scalable algorithms for physics-informed neural and graph networks

Authors: Khemraj Shukla, Mengjia Xu, Nathaniel Trask, George Em Karniadakis

Abstract: Physics-informed machine learning (PIML) has emerged as a promising new approach for simulating complex physical and biological systems that are governed by complex multiscale processes for which some data are also available. In some instances, the objective is to discover part of the hidden physics from the available data, and PIML has been shown to be particularly effective for such problems for… ▽ More Physics-informed machine learning (PIML) has emerged as a promising new approach for simulating complex physical and biological systems that are governed by complex multiscale processes for which some data are also available. In some instances, the objective is to discover part of the hidden physics from the available data, and PIML has been shown to be particularly effective for such problems for which conventional methods may fail. Unlike commercial machine learning where training of deep neural networks requires big data, in PIML big data are not available. Instead, we can train such networks from additional information obtained by employing the physical laws and evaluating them at random points in the space-time domain. Such physics-informed machine learning integrates multimodality and multifidelity data with mathematical models, and implements them using neural networks or graph networks. Here, we review some of the prevailing trends in embedding physics into machine learning, using physics-informed neural networks (PINNs) based primarily on feed-forward neural networks and automatic differentiation. For more complex systems or systems of systems and unstructured data, graph neural networks (GNNs) present some distinct advantages, and here we review how physics-informed learning can be accomplished with GNNs based on graph exterior calculus to construct differential operators; we refer to these architectures as physics-informed graph networks (PIGNs). We present representative examples for both forward and inverse problems and discuss what advances are needed to scale up PINNs, PIGNs and more broadly GNNs for large-scale engineering problems. △ Less

Submitted 16 May, 2022; originally announced May 2022.

Comments: 26 pages, 13 figures. arXiv admin note: text overlap with arXiv:2104.10013

Bayesian Physics-Informed Neural Networks for real-world nonlinear dynamical systems

Authors: Kevin Linka, Amelie Schafer, Xuhui Meng, Zongren Zou, George Em Karniadakis, Ellen Kuhl

Abstract: Understanding real-world dynamical phenomena remains a challenging task. Across various scientific disciplines, machine learning has advanced as the go-to technology to analyze nonlinear dynamical systems, identify patterns in big data, and make decision around them. Neural networks are now consistently used as universal function approximators for data with underlying mechanisms that are incomplet… ▽ More Understanding real-world dynamical phenomena remains a challenging task. Across various scientific disciplines, machine learning has advanced as the go-to technology to analyze nonlinear dynamical systems, identify patterns in big data, and make decision around them. Neural networks are now consistently used as universal function approximators for data with underlying mechanisms that are incompletely understood or exceedingly complex. However, neural networks alone ignore the fundamental laws of physics and often fail to make plausible predictions. Here we integrate data, physics, and uncertainties by combining neural networks, physics-informed modeling, and Bayesian inference to improve the predictive potential of traditional neural network models. We embed the physical model of a damped harmonic oscillator into a fully-connected feed-forward neural network to explore a simple and illustrative model system, the outbreak dynamics of COVID-19. Our Physics-Informed Neural Networks can seamlessly integrate data and physics, robustly solve forward and inverse problems, and perform well for both interpolation and extrapolation, even for a small amount of noisy and incomplete data. At only minor additional cost, they can self-adaptively learn the weighting between data and physics. Combined with Bayesian Neural Networks, they can serve as priors in a Bayesian Inference, and provide credible intervals for uncertainty quantification. Our study reveals the inherent advantages and disadvantages of Neural Networks, Bayesian Inference, and a combination of both and provides valuable guidelines for model selection. While we have only demonstrated these approaches for the simple model problem of a seasonal endemic infectious disease, we anticipate that the underlying concepts and trends generalize to more complex disease conditions and, more broadly, to a wide variety of nonlinear dynamical systems. △ Less

Submitted 24 May, 2022; v1 submitted 12 May, 2022; originally announced May 2022.

MSC Class: 62Mxx; 70Kxx ACM Class: G.3; J.3

Neural operator learning of heterogeneous mechanobiological insults contributing to aortic aneurysms

Authors: Somdatta Goswami, David S. Li, Bruno V. Rego, Marcos Latorre, Jay D. Humphrey, George Em Karniadakis

Abstract: Thoracic aortic aneurysm (TAA) is a localized dilatation of the aorta resulting from compromised wall composition, structure, and function, which can lead to life-threatening dissection or rupture. Several genetic mutations and predisposing factors that contribute to TAA have been studied in mouse models to characterize specific changes in aortic microstructure and material properties that result… ▽ More Thoracic aortic aneurysm (TAA) is a localized dilatation of the aorta resulting from compromised wall composition, structure, and function, which can lead to life-threatening dissection or rupture. Several genetic mutations and predisposing factors that contribute to TAA have been studied in mouse models to characterize specific changes in aortic microstructure and material properties that result from a wide range of mechanobiological insults. Assessments of TAA progression in vivo is largely limited to measurements of aneurysm size and growth rate. It has been shown that aortic geometry alone is not sufficient to predict the patient-specific progression of TAA but computational modeling of the evolving biomechanics of the aorta could predict future geometry and properties from initiating insults. In this work, we present an integrated framework to train a deep operator network (DeepONet)-based surrogate model to identify contributing factors for TAA by using FE-based datasets of aortic growth and remodeling resulting from prescribed insults. For training data, we investigate multiple types of TAA risk factors and spatial distributions within a constrained mixture model to generate axial--azimuthal maps of aortic dilatation and distensibility. The trained network is then capable of predicting the initial distribution and extent of the insult from a given set of dilatation and distensibility information. Two DeepONet frameworks are proposed, one trained on sparse information and one on full-field grayscale images, to gain insight into a preferred neural operator-based approach. Performance of the surrogate models is evaluated through multiple simulations carried out on insult distributions varying from fusiform to complex. We show that the proposed approach can predict patient-specific mechanobiological insult profile with a high accuracy, particularly when based on full-field images. △ Less

Submitted 8 May, 2022; originally announced May 2022.

Comments: 37 Pages, 16 Figures

Deep transfer operator learning for partial differential equations under conditional shift

Authors: Somdatta Goswami, Katiana Kontolati, Michael D. Shields, George Em Karniadakis

Abstract: Transfer learning (TL) enables the transfer of knowledge gained in learning to perform one task (source) to a related but different task (target), hence addressing the expense of data acquisition and labeling, potential computational power limitations, and dataset distribution mismatches. We propose a new TL framework for task-specific learning (functional regression in partial differential equati… ▽ More Transfer learning (TL) enables the transfer of knowledge gained in learning to perform one task (source) to a related but different task (target), hence addressing the expense of data acquisition and labeling, potential computational power limitations, and dataset distribution mismatches. We propose a new TL framework for task-specific learning (functional regression in partial differential equations (PDEs)) under conditional shift based on the deep operator network (DeepONet). Task-specific operator learning is accomplished by fine-tuning task-specific layers of the target DeepONet using a hybrid loss function that allows for the matching of individual target samples while also preserving the global properties of the conditional distribution of target data. Inspired by the conditional embedding operator theory, we minimize the statistical distance between labeled target data and the surrogate prediction on unlabeled target data by embedding conditional distributions onto a reproducing kernel Hilbert space. We demonstrate the advantages of our approach for various TL scenarios involving nonlinear PDEs under diverse conditions due to shift in the geometric domain and model dynamics. Our TL framework enables fast and efficient learning of heterogeneous tasks despite significant differences between the source and target domains. △ Less

Submitted 20 November, 2022; v1 submitted 20 April, 2022; originally announced April 2022.

Comments: 33 pages, 9 figures

Multifidelity Deep Operator Networks For Data-Driven and Physics-Informed Problems

Authors: Amanda A. Howard, Mauro Perego, George E. Karniadakis, Panos Stinis

Abstract: Operator learning for complex nonlinear systems is increasingly common in modeling multi-physics and multi-scale systems. However, training such high-dimensional operators requires a large amount of expensive, high-fidelity data, either from experiments or simulations. In this work, we present a composite Deep Operator Network (DeepONet) for learning using two datasets with different levels of fid… ▽ More Operator learning for complex nonlinear systems is increasingly common in modeling multi-physics and multi-scale systems. However, training such high-dimensional operators requires a large amount of expensive, high-fidelity data, either from experiments or simulations. In this work, we present a composite Deep Operator Network (DeepONet) for learning using two datasets with different levels of fidelity to accurately learn complex operators when sufficient high-fidelity data is not available. Additionally, we demonstrate that the presence of low-fidelity data can improve the predictions of physics-informed learning with DeepONets. We demonstrate the new multi-fidelity training in diverse examples, including modeling of the ice-sheet dynamics of the Humboldt glacier, Greenland, using two different fidelity models and also using the same physical model at two different resolutions. △ Less

Submitted 21 November, 2023; v1 submitted 19 April, 2022; originally announced April 2022.

Learning two-phase microstructure evolution using neural operators and autoencoder architectures

Authors: Vivek Oommen, Khemraj Shukla, Somdatta Goswami, Remi Dingreville, George Em Karniadakis

Abstract: Phase-field modeling is an effective but computationally expensive method for capturing the mesoscale morphological and microstructure evolution in materials. Hence, fast and generalizable surrogate models are needed to alleviate the cost of computationally taxing processes such as in optimization and design of materials. The intrinsic discontinuous nature of the physical phenomena incurred by the… ▽ More Phase-field modeling is an effective but computationally expensive method for capturing the mesoscale morphological and microstructure evolution in materials. Hence, fast and generalizable surrogate models are needed to alleviate the cost of computationally taxing processes such as in optimization and design of materials. The intrinsic discontinuous nature of the physical phenomena incurred by the presence of sharp phase boundaries makes the training of the surrogate model cumbersome. We develop a framework that integrates a convolutional autoencoder architecture with a deep neural operator (DeepONet) to learn the dynamic evolution of a two-phase mixture and accelerate time-to-solution in predicting the microstructure evolution. We utilize the convolutional autoencoder to provide a compact representation of the microstructure data in a low-dimensional latent space. DeepONet, which consists of two sub-networks, one for encoding the input function at a fixed number of sensors locations (branch net) and another for encoding the locations for the output functions (trunk net), learns the mesoscale dynamics of the microstructure evolution from the autoencoder latent space. The decoder part of the convolutional autoencoder then reconstructs the time-evolved microstructure from the DeepONet predictions. The trained DeepONet architecture can then be used to replace the high-fidelity phase-field numerical solver in interpolation tasks or to accelerate the numerical solver in extrapolation tasks. △ Less

Submitted 29 June, 2022; v1 submitted 11 April, 2022; originally announced April 2022.

Discovering and forecasting extreme events via active learning in neural operators

Authors: Ethan Pickering, Stephen Guth, George Em Karniadakis, Themistoklis P. Sapsis

Abstract: Extreme events in society and nature, such as pandemic spikes, rogue waves, or structural failures, can have catastrophic consequences. Characterizing extremes is difficult as they occur rarely, arise from seemingly benign conditions, and belong to complex and often unknown infinite-dimensional systems. Such challenges render attempts at characterizing them as moot. We address each of these diffic… ▽ More Extreme events in society and nature, such as pandemic spikes, rogue waves, or structural failures, can have catastrophic consequences. Characterizing extremes is difficult as they occur rarely, arise from seemingly benign conditions, and belong to complex and often unknown infinite-dimensional systems. Such challenges render attempts at characterizing them as moot. We address each of these difficulties by combining novel training schemes in Bayesian experimental design (BED) with an ensemble of deep neural operators (DNOs). This model-agnostic framework pairs a BED scheme that actively selects data for quantifying extreme events with an ensemble of DNOs that approximate infinite-dimensional nonlinear operators. We find that not only does this framework clearly beat Gaussian processes (GPs) but that 1) shallow ensembles of just two members perform best; 2) extremes are uncovered regardless of the state of initial data (i.e. with or without extremes); 3) our method eliminates "double-descent" phenomena; 4) the use of batches of suboptimal acquisition points compared to step-by-step global optima does not hinder BED performance; and 5) Monte Carlo acquisition outperforms standard optimizers in high-dimensions. Together these conclusions form the foundation of an AI-assisted experimental infrastructure that can efficiently infer and pinpoint critical situations across many domains, from physical to societal systems. △ Less

Submitted 20 September, 2022; v1 submitted 5 April, 2022; originally announced April 2022.

Comments: 25 pages, 8 figures, Submitted to Nature Computational Science

On the influence of over-parameterization in manifold based surrogates and deep neural operators

Authors: Katiana Kontolati, Somdatta Goswami, Michael D. Shields, George Em Karniadakis

Abstract: Constructing accurate and generalizable approximators for complex physico-chemical processes exhibiting highly non-smooth dynamics is challenging. In this work, we propose new developments and perform comparisons for two promising approaches: manifold-based polynomial chaos expansion (m-PCE) and the deep neural operator (DeepONet), and we examine the effect of over-parameterization on generalizati… ▽ More Constructing accurate and generalizable approximators for complex physico-chemical processes exhibiting highly non-smooth dynamics is challenging. In this work, we propose new developments and perform comparisons for two promising approaches: manifold-based polynomial chaos expansion (m-PCE) and the deep neural operator (DeepONet), and we examine the effect of over-parameterization on generalization. We demonstrate the performance of these methods in terms of generalization accuracy by solving the 2D time-dependent Brusselator reaction-diffusion system with uncertainty sources, modeling an autocatalytic chemical reaction between two species. We first propose an extension of the m-PCE by constructing a map** between latent spaces formed by two separate embeddings of input functions and output QoIs. To enhance the accuracy of the DeepONet, we introduce weight self-adaptivity in the loss function. We demonstrate that the performance of m-PCE and DeepONet is comparable for cases of relatively smooth input-output map**s. However, when highly non-smooth dynamics is considered, DeepONet shows higher accuracy. We also find that for m-PCE, modest over-parameterization leads to better generalization, both within and outside of distribution, whereas aggressive over-parameterization leads to over-fitting. In contrast, an even highly over-parameterized DeepONet leads to better generalization for both smooth and non-smooth dynamics. Furthermore, we compare the performance of the above models with another operator learning model, the Fourier Neural Operator, and show that its over-parameterization also leads to better generalization. Our studies show that m-PCE can provide very good accuracy at very low training cost, whereas a highly over-parameterized DeepONet can provide better accuracy and robustness to noise but at higher training cost. In both methods, the inference cost is negligible. △ Less

Submitted 9 March, 2022; originally announced March 2022.

Comments: 28 pages, 7 figures

Interfacing Finite Elements with Deep Neural Operators for Fast Multiscale Modeling of Mechanics Problems

Authors: Minglang Yin, Enrui Zhang, Yue Yu, George Em Karniadakis

Abstract: Multiscale modeling is an effective approach for investigating multiphysics systems with largely disparate size features, where models with different resolutions or heterogeneous descriptions are coupled together for predicting the system's response. The solver with lower fidelity (coarse) is responsible for simulating domains with homogeneous features, whereas the expensive high-fidelity (fine) m… ▽ More Multiscale modeling is an effective approach for investigating multiphysics systems with largely disparate size features, where models with different resolutions or heterogeneous descriptions are coupled together for predicting the system's response. The solver with lower fidelity (coarse) is responsible for simulating domains with homogeneous features, whereas the expensive high-fidelity (fine) model describes microscopic features with refined discretization, often making the overall cost prohibitively high, especially for time-dependent problems. In this work, we explore the idea of multiscale modeling with machine learning and employ DeepONet, a neural operator, as an efficient surrogate of the expensive solver. DeepONet is trained offline using data acquired from the fine solver for learning the underlying and possibly unknown fine-scale dynamics. It is then coupled with standard PDE solvers for predicting the multiscale systems with new boundary/initial conditions in the coupling stage. The proposed framework significantly reduces the computational cost of multiscale simulations since the DeepONet inference cost is negligible, facilitating readily the incorporation of a plurality of interface conditions and coupling schemes. We present various benchmarks to assess accuracy and speedup, and in particular we develop a coupling algorithm for a time-dependent problem, and we also demonstrate coupling of a continuum model (finite element methods, FEM) with a neural operator representation of a particle system (Smoothed Particle Hydrodynamics, SPH) for a uniaxial tension problem with hyperelastic material. What makes this approach unique is that a well-trained over-parametrized DeepONet can generalize well and make predictions at a negligible cost. △ Less

Submitted 25 February, 2022; originally announced March 2022.

Physics-informed neural networks for inverse problems in supersonic flows

Authors: Ameya D. Jagtap, Zhi** Mao, Nikolaus Adams, George Em Karniadakis

Abstract: Accurate solutions to inverse supersonic compressible flow problems are often required for designing specialized aerospace vehicles. In particular, we consider the problem where we have data available for density gradients from Schlieren photography as well as data at the inflow and part of wall boundaries. These inverse problems are notoriously difficult and traditional methods may not be adequat… ▽ More Accurate solutions to inverse supersonic compressible flow problems are often required for designing specialized aerospace vehicles. In particular, we consider the problem where we have data available for density gradients from Schlieren photography as well as data at the inflow and part of wall boundaries. These inverse problems are notoriously difficult and traditional methods may not be adequate to solve such ill-posed inverse problems. To this end, we employ the physics-informed neural networks (PINNs) and its extended version, extended PINNs (XPINNs), where domain decomposition allows deploying locally powerful neural networks in each subdomain, which can provide additional expressivity in subdomains, where a complex solution is expected. Apart from the governing compressible Euler equations, we also enforce the entropy conditions in order to obtain viscosity solutions. Moreover, we enforce positivity conditions on density and pressure. We consider inverse problems involving two-dimensional expansion waves, two-dimensional oblique and bow shock waves. We compare solutions obtained by PINNs and XPINNs and invoke some theoretical results that can be used to decide on the generalization errors of the two methods. △ Less

Submitted 23 February, 2022; originally announced February 2022.

Comments: 19 pages, 20 figures

Deep learning of inverse water waves problems using multi-fidelity data: Application to Serre-Green-Naghdi equations

Authors: Ameya D. Jagtap, Dimitrios Mitsotakis, George Em Karniadakis

Abstract: We consider strongly-nonlinear and weakly-dispersive surface water waves governed by equations of Boussinesq type, known as the Serre-Green-Naghdi system; it describes future states of the free water surface and depth averaged horizontal velocity, given their initial state. The lack of knowledge of the velocity field as well as the initial states provided by measurements lead to an ill-posed probl… ▽ More We consider strongly-nonlinear and weakly-dispersive surface water waves governed by equations of Boussinesq type, known as the Serre-Green-Naghdi system; it describes future states of the free water surface and depth averaged horizontal velocity, given their initial state. The lack of knowledge of the velocity field as well as the initial states provided by measurements lead to an ill-posed problem that cannot be solved by traditional techniques. To this end, we employ physics-informed neural networks (PINNs) to generate solutions to such ill-posed problems using only data of the free surface elevation and depth of the water. PINNs can readily incorporate the physical laws and the observational data, thereby enabling inference of the physical quantities of interest. In the present study, both experimental and synthetic (generated by numerical methods) training data are used to train PINNs. Furthermore, multi-fidelity data are used to solve the inverse water wave problem by leveraging both high- and low-fidelity data sets. The applicability of the PINN methodology for the estimation of the impact of water waves onto solid obstacles is demonstrated after deriving the corresponding equations. The present methodology can be employed to efficiently design offshore structures such as oil platforms, wind turbines, etc. by solving the corresponding ill-posed inverse water waves problem. △ Less

Submitted 6 February, 2022; originally announced February 2022.

Systems Biology: Identifiability analysis and parameter identification via systems-biology informed neural networks

Authors: Mitchell Daneker, Zhen Zhang, George Em Karniadakis, Lu Lu

Abstract: The dynamics of systems biological processes are usually modeled by a system of ordinary differential equations (ODEs) with many unknown parameters that need to be inferred from noisy and sparse measurements. Here, we introduce systems-biology informed neural networks for parameter estimation by incorporating the system of ODEs into the neural networks. To complete the workflow of system identific… ▽ More The dynamics of systems biological processes are usually modeled by a system of ordinary differential equations (ODEs) with many unknown parameters that need to be inferred from noisy and sparse measurements. Here, we introduce systems-biology informed neural networks for parameter estimation by incorporating the system of ODEs into the neural networks. To complete the workflow of system identification, we also describe structural and practical identifiability analysis to analyze the identifiability of parameters. We use the ultridian endocrine model for glucose-insulin interaction as the example to demonstrate all these methods and their implementation. △ Less

Submitted 3 February, 2022; originally announced February 2022.

Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons

Authors: Apostolos F Psaros, Xuhui Meng, Zongren Zou, Ling Guo, George Em Karniadakis

Abstract: Neural networks (NNs) are currently changing the computational paradigm on how to combine data with mathematical laws in physics and engineering in a profound way, tackling challenging inverse and ill-posed problems not solvable with traditional methods. However, quantifying errors and uncertainties in NN-based inference is more complicated than in traditional methods. This is because in addition… ▽ More Neural networks (NNs) are currently changing the computational paradigm on how to combine data with mathematical laws in physics and engineering in a profound way, tackling challenging inverse and ill-posed problems not solvable with traditional methods. However, quantifying errors and uncertainties in NN-based inference is more complicated than in traditional methods. This is because in addition to aleatoric uncertainty associated with noisy data, there is also uncertainty due to limited data, but also due to NN hyperparameters, overparametrization, optimization and sampling errors as well as model misspecification. Although there are some recent works on uncertainty quantification (UQ) in NNs, there is no systematic investigation of suitable methods towards quantifying the total uncertainty effectively and efficiently even for function approximation, and there is even less work on solving partial differential equations and learning operator map**s between infinite-dimensional function spaces using NNs. In this work, we present a comprehensive framework that includes uncertainty modeling, new and existing solution methods, as well as evaluation metrics and post-hoc improvement approaches. To demonstrate the applicability and reliability of our framework, we present an extensive comparative study in which various methods are tested on prototype problems, including problems with mixed input-output data, and stochastic problems in high dimensions. In the Appendix, we include a comprehensive description of all the UQ methods employed, which we will make available as open-source library of all codes included in this framework. △ Less

Submitted 19 January, 2022; originally announced January 2022.

SympOCnet: Solving optimal control problems with applications to high-dimensional multi-agent path planning problems

Authors: Tingwei Meng, Zhen Zhang, Jérôme Darbon, George Em Karniadakis

Abstract: Solving high-dimensional optimal control problems in real-time is an important but challenging problem, with applications to multi-agent path planning problems, which have drawn increased attention given the growing popularity of drones in recent years. In this paper, we propose a novel neural network method called SympOCnet that applies the Symplectic network to solve high-dimensional optimal con… ▽ More Solving high-dimensional optimal control problems in real-time is an important but challenging problem, with applications to multi-agent path planning problems, which have drawn increased attention given the growing popularity of drones in recent years. In this paper, we propose a novel neural network method called SympOCnet that applies the Symplectic network to solve high-dimensional optimal control problems with state constraints. We present several numerical results on path planning problems in two-dimensional and three-dimensional spaces. Specifically, we demonstrate that our SympOCnet can solve a problem with more than 500 dimensions in 1.5 hours on a single GPU, which shows the effectiveness and efficiency of SympOCnet. The proposed method is scalable and has the potential to solve truly high-dimensional path planning problems in real-time. △ Less

Submitted 14 January, 2022; originally announced January 2022.

A comprehensive and fair comparison of two neural operators (with practical extensions) based on FAIR data

Authors: Lu Lu, Xuhui Meng, Shengze Cai, Zhi** Mao, Somdatta Goswami, Zhongqiang Zhang, George Em Karniadakis

Abstract: Neural operators can learn nonlinear map**s between function spaces and offer a new simulation paradigm for real-time prediction of complex dynamics for realistic diverse applications as well as for system identification in science and engineering. Herein, we investigate the performance of two neural operators, and we develop new practical extensions that will make them more accurate and robust… ▽ More Neural operators can learn nonlinear map**s between function spaces and offer a new simulation paradigm for real-time prediction of complex dynamics for realistic diverse applications as well as for system identification in science and engineering. Herein, we investigate the performance of two neural operators, and we develop new practical extensions that will make them more accurate and robust and importantly more suitable for industrial-complexity applications. The first neural operator, DeepONet, was published in 2019, and the second one, named Fourier Neural Operator or FNO, was published in 2020. In order to compare FNO with DeepONet for realistic setups, we develop several extensions of FNO that can deal with complex geometric domains as well as map**s where the input and output function spaces are of different dimensions. We also endow DeepONet with special features that provide inductive bias and accelerate training, and we present a faster implementation of DeepONet with cost comparable to the computational cost of FNO. We consider 16 different benchmarks to demonstrate the relative performance of the two neural operators, including instability wave analysis in hypersonic boundary layers, prediction of the vorticity field of a flap** airfoil, porous media simulations in complex-geometry domains, etc. The performance of DeepONet and FNO is comparable for relatively simple settings, but for complex geometries and especially noisy data, the performance of FNO deteriorates greatly. For example, for the instability wave analysis with only 0.1% noise added to the input data, the error of FNO increases 10000 times making it inappropriate for such important applications, while there is hardly any effect of such noise on the DeepONet. We also compare theoretically the two neural operators and obtain similar error estimates for DeepONet and FNO under the same regularity assumptions. △ Less

Submitted 9 November, 2021; originally announced November 2021.

Gradient-enhanced physics-informed neural networks for forward and inverse PDE problems

Authors: Jeremy Yu, Lu Lu, Xuhui Meng, George Em Karniadakis

Abstract: Deep learning has been shown to be an effective tool in solving partial differential equations (PDEs) through physics-informed neural networks (PINNs). PINNs embed the PDE residual into the loss function of the neural network, and have been successfully employed to solve diverse forward and inverse PDE problems. However, one disadvantage of the first generation of PINNs is that they usually have l… ▽ More Deep learning has been shown to be an effective tool in solving partial differential equations (PDEs) through physics-informed neural networks (PINNs). PINNs embed the PDE residual into the loss function of the neural network, and have been successfully employed to solve diverse forward and inverse PDE problems. However, one disadvantage of the first generation of PINNs is that they usually have limited accuracy even with many training points. Here, we propose a new method, gradient-enhanced physics-informed neural networks (gPINNs), for improving the accuracy and training efficiency of PINNs. gPINNs leverage gradient information of the PDE residual and embed the gradient into the loss function. We tested gPINNs extensively and demonstrated the effectiveness of gPINNs in both forward and inverse PDE problems. Our numerical results show that gPINN performs better than PINN with fewer training points. Furthermore, we combined gPINN with the method of residual-based adaptive refinement (RAR), a method for improving the distribution of training points adaptively during training, to further improve the performance of gPINN, especially in PDEs with solutions that have steep gradients. △ Less

Submitted 1 November, 2021; originally announced November 2021.

DynG2G: An Efficient Stochastic Graph Embedding Method for Temporal Graphs

Authors: Mengjia Xu, Apoorva Vikram Singh, George Em Karniadakis

Abstract: Dynamic graph embedding has gained great attention recently due to its capability of learning low dimensional graph representations for complex temporal graphs with high accuracy. However, recent advances mostly focus on learning node embeddings as deterministic "vectors" for static graphs yet disregarding the key graph temporal dynamics and the evolving uncertainties associated with node embeddin… ▽ More Dynamic graph embedding has gained great attention recently due to its capability of learning low dimensional graph representations for complex temporal graphs with high accuracy. However, recent advances mostly focus on learning node embeddings as deterministic "vectors" for static graphs yet disregarding the key graph temporal dynamics and the evolving uncertainties associated with node embedding in the latent space. In this work, we propose an efficient stochastic dynamic graph embedding method (DynG2G) that applies an inductive feed-forward encoder trained with node triplet-based contrastive loss. Every node per timestamp is encoded as a time-dependent probabilistic multivariate Gaussian distribution in the latent space, hence we can quantify the node embedding uncertainty on-the-fly. We adopted eight different benchmarks that represent diversity in size (from 96 nodes to 87,626 and from 13,398 edges to 4,870,863) and diversity in dynamics. We demonstrate via extensive experiments on these eight dynamic graph benchmarks that DynG2G achieves new state-of-the-art performance in capturing the underlying temporal node embeddings. We also demonstrate that DynG2G can predict the evolving node embedding uncertainty, which plays a crucial role in quantifying the intrinsic dimensionality of the dynamical system over time. We obtain a universal relation of the optimal embedding dimension, $L_o$, versus the effective dimensionality of uncertainty, $D_u$, and we infer that $L_o=D_u$ for all cases. This implies that the uncertainty quantification approach we employ in the DynG2G correctly captures the intrinsic dimensionality of the dynamics of such evolving graphs despite the diverse nature and composition of the graphs at each timestamp. Moreover, this $L_0 - D_u$ correlation provides a clear path to select adaptively the optimum embedding size at each timestamp by setting $L \ge D_u$. △ Less

Submitted 27 April, 2022; v1 submitted 27 September, 2021; originally announced September 2021.

Comments: 13 pages, 10 figures

When Do Extended Physics-Informed Neural Networks (XPINNs) Improve Generalization?

Authors: Zheyuan Hu, Ameya D. Jagtap, George Em Karniadakis, Kenji Kawaguchi

Abstract: Physics-informed neural networks (PINNs) have become a popular choice for solving high-dimensional partial differential equations (PDEs) due to their excellent approximation power and generalization ability. Recently, Extended PINNs (XPINNs) based on domain decomposition methods have attracted considerable attention due to their effectiveness in modeling multiscale and multiphysics problems and th… ▽ More Physics-informed neural networks (PINNs) have become a popular choice for solving high-dimensional partial differential equations (PDEs) due to their excellent approximation power and generalization ability. Recently, Extended PINNs (XPINNs) based on domain decomposition methods have attracted considerable attention due to their effectiveness in modeling multiscale and multiphysics problems and their parallelization. However, theoretical understanding on their convergence and generalization properties remains unexplored. In this study, we take an initial step towards understanding how and when XPINNs outperform PINNs. Specifically, for general multi-layer PINNs and XPINNs, we first provide a prior generalization bound via the complexity of the target functions in the PDE problem, and a posterior generalization bound via the posterior matrix norms of the networks after optimization. Moreover, based on our bounds, we analyze the conditions under which XPINNs improve generalization. Concretely, our theory shows that the key building block of XPINN, namely the domain decomposition, introduces a tradeoff for generalization. On the one hand, XPINNs decompose the complex PDE solution into several simple parts, which decreases the complexity needed to learn each part and boosts generalization. On the other hand, decomposition leads to less training data being available in each subdomain, and hence such model is typically prone to overfitting and may become less generalizable. Empirically, we choose five PDEs to show when XPINNs perform better than, similar to, or worse than PINNs, hence demonstrating and justifying our new theory. △ Less

Submitted 18 October, 2022; v1 submitted 20 September, 2021; originally announced September 2021.

Comments: Published in SIAM Journal on Scientific Computing (SISC)

Journal ref: SIAM Journal on Scientific Computing Vol. 44, Iss. 5 (2022)

GFINNs: GENERIC Formalism Informed Neural Networks for Deterministic and Stochastic Dynamical Systems

Authors: Zhen Zhang, Yeonjong Shin, George Em Karniadakis

Abstract: We propose the GENERIC formalism informed neural networks (GFINNs) that obey the symmetric degeneracy conditions of the GENERIC formalism. GFINNs comprise two modules, each of which contains two components. We model each component using a neural network whose architecture is designed to satisfy the required conditions. The component-wise architecture design provides flexible ways of leveraging ava… ▽ More We propose the GENERIC formalism informed neural networks (GFINNs) that obey the symmetric degeneracy conditions of the GENERIC formalism. GFINNs comprise two modules, each of which contains two components. We model each component using a neural network whose architecture is designed to satisfy the required conditions. The component-wise architecture design provides flexible ways of leveraging available physics information into neural networks. We prove theoretically that GFINNs are sufficiently expressive to learn the underlying equations, hence establishing the universal approximation theorem. We demonstrate the performance of GFINNs in three simulation problems: gas containers exchanging heat and volume, thermoelastic double pendulum and the Langevin dynamics. In all the examples, GFINNs outperform existing methods, hence demonstrating good accuracy in predictions for both deterministic and stochastic systems. △ Less

Submitted 31 August, 2021; originally announced September 2021.

Physics-informed Neural Networks (PINNs) for Wave Propagation and Full Waveform Inversions

Authors: Majid Rasht-Behesht, Christian Huber, Khemraj Shukla, George Em Karniadakis

Abstract: We propose a new approach to the solution of the wave propagation and full waveform inversions (FWIs) based on a recent advance in deep learning called Physics-Informed Neural Networks (PINNs). In this study, we present an algorithm for PINNs applied to the 2D acoustic wave equation and test the model with both forward wave propagation and FWIs case studies. These synthetic case studies are design… ▽ More We propose a new approach to the solution of the wave propagation and full waveform inversions (FWIs) based on a recent advance in deep learning called Physics-Informed Neural Networks (PINNs). In this study, we present an algorithm for PINNs applied to the 2D acoustic wave equation and test the model with both forward wave propagation and FWIs case studies. These synthetic case studies are designed to explore the ability of PINNs to handle varying degrees of structural complexity using both teleseismic plane waves and seismic point sources. PINNs meshless formalism allows for a flexible implementation of the wave equation and different types of boundary conditions. For instance, our models demonstrate that PINN automatically satisfies absorbing boundary conditions, a serious computational challenge for common wave propagation solvers. Furthermore, a priori knowledge of the subsurface structure can be seamlessly encoded in PINNs formulation. We find that the current state-of-the-art PINNs provide good results for the forward model, even though spectral element or finite difference methods are more efficient and accurate. More importantly, our results demonstrate that PINNs yield excellent results for inversions on all cases considered and with limited computational complexity. Using PINNs as a geophysical inversion solver offers exciting perspectives, not only for the full waveform seismic inversions, but also when dealing with other geophysical datasets (e.g., magnetotellurics, gravity) as well as joint inversions because of its robust framework and simple implementation. △ Less

Submitted 26 August, 2021; originally announced August 2021.

Simulating progressive intramural damage leading to aortic dissection using an operator-regression neural network

Authors: Minglang Yin, Ehsan Ban, Bruno V. Rego, Enrui Zhang, Cristina Cavinato, Jay D. Humphrey, George Em Karniadakis

Abstract: Aortic dissection progresses via delamination of the medial layer of the wall. Notwithstanding the complexity of this process, insight has been gleaned by studying in vitro and in silico the progression of dissection driven by quasi-static pressurization of the intramural space by fluid injection, which demonstrates that the differential propensity of dissection can be affected by spatial distribu… ▽ More Aortic dissection progresses via delamination of the medial layer of the wall. Notwithstanding the complexity of this process, insight has been gleaned by studying in vitro and in silico the progression of dissection driven by quasi-static pressurization of the intramural space by fluid injection, which demonstrates that the differential propensity of dissection can be affected by spatial distributions of structurally significant interlamellar struts that connect adjacent elastic lamellae. In particular, diverse histological microstructures may lead to differential mechanical behavior during dissection, including the pressure--volume relationship of the injected fluid and the displacement field between adjacent lamellae. In this study, we develop a data-driven surrogate model for the delamination process for differential strut distributions using DeepONet, a new operator--regression neural network. The surrogate model is trained to predict the pressure--volume curve of the injected fluid and the damage progression field of the wall given a spatial distribution of struts, with in silico data generated with a phase-field finite element model. The results show that DeepONet can provide accurate predictions for diverse strut distributions, indicating that this composite branch-trunk neural network can effectively extract the underlying functional relationship between distinctive microstructures and their mechanical properties. More broadly, DeepONet can facilitate surrogate model-based analyses to quantify biological variability, improve inverse design, and predict mechanical properties based on multi-modality experimental data. △ Less

Submitted 24 August, 2021; originally announced August 2021.

Meta-learning PINN loss functions

Authors: Apostolos F Psaros, Kenji Kawaguchi, George Em Karniadakis

Abstract: We propose a meta-learning technique for offline discovery of physics-informed neural network (PINN) loss functions. We extend earlier works on meta-learning, and develop a gradient-based meta-learning algorithm for addressing diverse task distributions based on parametrized partial differential equations (PDEs) that are solved with PINNs. Furthermore, based on new theory we identify two desirable… ▽ More We propose a meta-learning technique for offline discovery of physics-informed neural network (PINN) loss functions. We extend earlier works on meta-learning, and develop a gradient-based meta-learning algorithm for addressing diverse task distributions based on parametrized partial differential equations (PDEs) that are solved with PINNs. Furthermore, based on new theory we identify two desirable properties of meta-learned losses in PINN problems, which we enforce by proposing a new regularization method or using a specific parametrization of the loss function. In the computational examples, the meta-learned losses are employed at test time for addressing regression and PDE task distributions. Our results indicate that significant performance improvement can be achieved by using a shared-among-tasks offline-learned loss function even for out-of-distribution meta-testing. In this case, we solve for test tasks that do not belong to the task distribution used in meta-training, and we also employ PINN architectures that are different from the PINN architecture used in meta-training. To better understand the capabilities and limitations of the proposed method, we consider various parametrizations of the loss function and describe different algorithm design options and how they may affect meta-learning performance. △ Less

Submitted 12 July, 2021; originally announced July 2021.

Learning Functional Priors and Posteriors from Data and Physics

Authors: Xuhui Meng, Liu Yang, Zhi** Mao, Jose del Aguila Ferrandis, George Em Karniadakis

Abstract: We develop a new Bayesian framework based on deep neural networks to be able to extrapolate in space-time using historical data and to quantify uncertainties arising from both noisy and gappy data in physical problems. Specifically, the proposed approach has two stages: (1) prior learning and (2) posterior estimation. At the first stage, we employ the physics-informed Generative Adversarial Networ… ▽ More We develop a new Bayesian framework based on deep neural networks to be able to extrapolate in space-time using historical data and to quantify uncertainties arising from both noisy and gappy data in physical problems. Specifically, the proposed approach has two stages: (1) prior learning and (2) posterior estimation. At the first stage, we employ the physics-informed Generative Adversarial Networks (PI-GAN) to learn a functional prior either from a prescribed function distribution, e.g., Gaussian process, or from historical data and physics. At the second stage, we employ the Hamiltonian Monte Carlo (HMC) method to estimate the posterior in the latent space of PI-GANs. In addition, we use two different approaches to encode the physics: (1) automatic differentiation, used in the physics-informed neural networks (PINNs) for scenarios with explicitly known partial differential equations (PDEs), and (2) operator regression using the deep operator network (DeepONet) for PDE-agnostic scenarios. We then test the proposed method for (1) meta-learning for one-dimensional regression, and forward/inverse PDE problems (combined with PINNs); (2) PDE-agnostic physical problems (combined with DeepONet), e.g., fractional diffusion as well as saturated stochastic (100-dimensional) flows in heterogeneous porous media; and (3) spatial-temporal regression problems, i.e., inference of a marine riser displacement field. The results demonstrate that the proposed approach can provide accurate predictions as well as uncertainty quantification given very limited scattered and noisy data, since historical data could be available to provide informative priors. In summary, the proposed method is capable of learning flexible functional priors, and can be extended to big data problems using stochastic HMC or normalizing flows since the latent space is generally characterized as low dimensional. △ Less

Submitted 7 June, 2021; originally announced June 2021.

AOSLO-net: A deep learning-based method for automatic segmentation of retinal microaneurysms from adaptive optics scanning laser ophthalmoscope images

Authors: Qian Zhang, Konstantina Sampani, Mengjia Xu, Shengze Cai, Yixiang Deng, He Li, Jennifer K. Sun, George Em Karniadakis

Abstract: Microaneurysms (MAs) are one of the earliest signs of diabetic retinopathy (DR), a frequent complication of diabetes that can lead to visual impairment and blindness. Adaptive optics scanning laser ophthalmoscopy (AOSLO) provides real-time retinal images with resolution down to 2 $μm$ and thus allows detection of the morphologies of individual MAs, a potential marker that might dictate MA patholog… ▽ More Microaneurysms (MAs) are one of the earliest signs of diabetic retinopathy (DR), a frequent complication of diabetes that can lead to visual impairment and blindness. Adaptive optics scanning laser ophthalmoscopy (AOSLO) provides real-time retinal images with resolution down to 2 $μm$ and thus allows detection of the morphologies of individual MAs, a potential marker that might dictate MA pathology and affect the progression of DR. In contrast to the numerous automatic models developed for assessing the number of MAs on fundus photographs, currently there is no high throughput image protocol available for automatic analysis of AOSLO photographs. To address this urgency, we introduce AOSLO-net, a deep neural network framework with customized training policies to automatically segment MAs from AOSLO images. We evaluate the performance of AOSLO-net using 87 DR AOSLO images and our results demonstrate that the proposed model outperforms the state-of-the-art segmentation model both in accuracy and cost and enables correct MA morphological classification. △ Less

Submitted 25 June, 2021; v1 submitted 5 June, 2021; originally announced June 2021.

Deep Kronecker neural networks: A general framework for neural networks with adaptive activation functions

Authors: Ameya D. Jagtap, Yeonjong Shin, Kenji Kawaguchi, George Em Karniadakis

Abstract: We propose a new type of neural networks, Kronecker neural networks (KNNs), that form a general framework for neural networks with adaptive activation functions. KNNs employ the Kronecker product, which provides an efficient way of constructing a very wide network while kee** the number of parameters low. Our theoretical analysis reveals that under suitable conditions, KNNs induce a faster decay… ▽ More We propose a new type of neural networks, Kronecker neural networks (KNNs), that form a general framework for neural networks with adaptive activation functions. KNNs employ the Kronecker product, which provides an efficient way of constructing a very wide network while kee** the number of parameters low. Our theoretical analysis reveals that under suitable conditions, KNNs induce a faster decay of the loss than that by the feed-forward networks. This is also empirically verified through a set of computational examples. Furthermore, under certain technical assumptions, we establish global convergence of gradient descent for KNNs. As a specific case, we propose the Rowdy activation function that is designed to get rid of any saturation region by injecting sinusoidal fluctuations, which include trainable parameters. The proposed Rowdy activation function can be employed in any neural network architecture like feed-forward neural networks, Recurrent neural networks, Convolutional neural networks etc. The effectiveness of KNNs with Rowdy activation is demonstrated through various computational experiments including function approximation using feed-forward neural networks, solution inference of partial differential equations using the physics-informed neural networks, and standard deep learning benchmark problems using convolutional and fully-connected neural networks. △ Less

Submitted 19 October, 2021; v1 submitted 20 May, 2021; originally announced May 2021.

Comments: 24 pages, 16 figures

Physics-informed neural networks (PINNs) for fluid mechanics: A review

Authors: Shengze Cai, Zhi** Mao, Zhicheng Wang, Minglang Yin, George Em Karniadakis

Abstract: Despite the significant progress over the last 50 years in simulating flow problems using numerical discretization of the Navier-Stokes equations (NSE), we still cannot incorporate seamlessly noisy data into existing algorithms, mesh-generation is complex, and we cannot tackle high-dimensional problems governed by parametrized NSE. Moreover, solving inverse flow problems is often prohibitively exp… ▽ More Despite the significant progress over the last 50 years in simulating flow problems using numerical discretization of the Navier-Stokes equations (NSE), we still cannot incorporate seamlessly noisy data into existing algorithms, mesh-generation is complex, and we cannot tackle high-dimensional problems governed by parametrized NSE. Moreover, solving inverse flow problems is often prohibitively expensive and requires complex and expensive formulations and new computer codes. Here, we review flow physics-informed learning, integrating seamlessly data and mathematical models, and implementing them using physics-informed neural networks (PINNs). We demonstrate the effectiveness of PINNs for inverse problems related to three-dimensional wake flows, supersonic flows, and biomedical flows. △ Less

Submitted 20 May, 2021; originally announced May 2021.

Parallel Physics-Informed Neural Networks via Domain Decomposition

Authors: Khemraj Shukla, Ameya D. Jagtap, George Em Karniadakis

Abstract: We develop a distributed framework for the physics-informed neural networks (PINNs) based on two recent extensions, namely conservative PINNs (cPINNs) and extended PINNs (XPINNs), which employ domain decomposition in space and in time-space, respectively. This domain decomposition endows cPINNs and XPINNs with several advantages over the vanilla PINNs, such as parallelization capacity, large repre… ▽ More We develop a distributed framework for the physics-informed neural networks (PINNs) based on two recent extensions, namely conservative PINNs (cPINNs) and extended PINNs (XPINNs), which employ domain decomposition in space and in time-space, respectively. This domain decomposition endows cPINNs and XPINNs with several advantages over the vanilla PINNs, such as parallelization capacity, large representation capacity, efficient hyperparameter tuning, and is particularly effective for multi-scale and multi-physics problems. Here, we present a parallel algorithm for cPINNs and XPINNs constructed with a hybrid programming model described by MPI $+$ X, where X $\in \{\text{CPUs},~\text{GPUs}\}$. The main advantage of cPINN and XPINN over the more classical data and model parallel approaches is the flexibility of optimizing all hyperparameters of each neural network separately in each subdomain. We compare the performance of distributed cPINNs and XPINNs for various forward problems, using both weak and strong scalings. Our results indicate that for space domain decomposition, cPINNs are more efficient in terms of communication cost but XPINNs provide greater flexibility as they can also handle time-domain decomposition for any differential equations, and can deal with any arbitrarily shaped complex subdomains. To this end, we also present an application of the parallel XPINN method for solving an inverse diffusion problem with variable conductivity on the United States map, using ten regions as subdomains. △ Less

Submitted 8 September, 2021; v1 submitted 20 April, 2021; originally announced April 2021.

Comments: 23 pages, 13 figures

A Caputo fractional derivative-based algorithm for optimization

Authors: Yeonjong Shin, Jérôme Darbon, George Em Karniadakis

Abstract: We propose a novel Caputo fractional derivative-based optimization algorithm. Upon defining the Caputo fractional gradient with respect to the Cartesian coordinate, we present a generic Caputo fractional gradient descent (CFGD) method. We prove that the CFGD yields the steepest descent direction of a locally smoothed objective function. The generic CFGD requires three parameters to be specified, a… ▽ More We propose a novel Caputo fractional derivative-based optimization algorithm. Upon defining the Caputo fractional gradient with respect to the Cartesian coordinate, we present a generic Caputo fractional gradient descent (CFGD) method. We prove that the CFGD yields the steepest descent direction of a locally smoothed objective function. The generic CFGD requires three parameters to be specified, and a choice of the parameters yields a version of CFGD. We propose three versions -- non-adaptive, adaptive terminal and adaptive order. By focusing on quadratic objective functions, we provide a convergence analysis. We prove that the non-adaptive CFGD converges to a Tikhonov regularized solution. For the two adaptive versions, we derive error bounds, which show convergence to integer-order stationary point under some conditions. We derive an explicit formula of CFGD for quadratic functions. We computationally found that the adaptive terminal (AT) CFGD mitigates the dependence on the condition number in the rate of convergence and results in significant acceleration over gradient descent (GD). For non-quadratic functions, we develop an efficient implementation of CFGD using the Gauss-Jacobi quadrature, whose computational cost is approximately proportional to the number of the quadrature points and the cost of GD. Our numerical examples show that AT-CFGD results in acceleration over GD, even when a small number of the Gauss-Jacobi quadrature points (including a single point) is used. △ Less

Submitted 5 April, 2021; originally announced April 2021.

MSC Class: 65K05; 65B99; 26A33

A physics-informed neural network for quantifying the microstructure properties of polycrystalline Nickel using ultrasound data

Authors: Khemraj Shukla, Ameya D. Jagtap, James L. Blackshire, Daniel Sparkman, George Em Karniadakis

Abstract: We employ physics-informed neural networks (PINNs) to quantify the microstructure of a polycrystalline Nickel by computing the spatial variation of compliance coefficients (compressibility, stiffness and rigidity) of the material. The PINN is supervised with realistic ultrasonic surface acoustic wavefield data acquired at an ultrasonic frequency of 5 MHz for the polycrystalline material. The ultra… ▽ More We employ physics-informed neural networks (PINNs) to quantify the microstructure of a polycrystalline Nickel by computing the spatial variation of compliance coefficients (compressibility, stiffness and rigidity) of the material. The PINN is supervised with realistic ultrasonic surface acoustic wavefield data acquired at an ultrasonic frequency of 5 MHz for the polycrystalline material. The ultrasonic wavefield data is represented as a deformation on the top surface of the material with the deformation measured using the method of laser vibrometry. The ultrasonic data is further complemented with wavefield data generated using a finite element based solver. The neural network is physically-informed by the in-plane and out-of-plane elastic wave equations and its convergence is accelerated using adaptive activation functions. The overarching goal of this work is to infer the spatial variation of compliance coefficients of materials using PINNs, which for ultrasound involves the spatially varying speed of the elastic waves. More broadly, the resulting PINN based surrogate model shows a promising approach for solving ill-posed inverse problems, often encountered in the non-destructive evaluation of materials. △ Less

Submitted 5 October, 2021; v1 submitted 25 March, 2021; originally announced March 2021.

Comments: 18 pages, 5 figures

Flow over an espresso cup: Inferring 3D velocity and pressure fields from tomographic background oriented schlieren videos via physics-informed neural networks

Authors: Shengze Cai, Zhicheng Wang, Frederik Fuest, Young-** Jeon, Callum Gray, George Em Karniadakis

Abstract: Tomographic background oriented schlieren (Tomo-BOS) imaging measures density or temperature fields in 3D using multiple camera BOS projections, and is particularly useful for instantaneous flow visualizations of complex fluid dynamics problems. We propose a new method based on physics-informed neural networks (PINNs) to infer the full continuous 3D velocity and pressure fields from snapshots of 3… ▽ More Tomographic background oriented schlieren (Tomo-BOS) imaging measures density or temperature fields in 3D using multiple camera BOS projections, and is particularly useful for instantaneous flow visualizations of complex fluid dynamics problems. We propose a new method based on physics-informed neural networks (PINNs) to infer the full continuous 3D velocity and pressure fields from snapshots of 3D temperature fields obtained by Tomo-BOS imaging. PINNs seamlessly integrate the underlying physics of the observed fluid flow and the visualization data, hence enabling the inference of latent quantities using limited experimental data. In this hidden fluid mechanics paradigm, we train the neural network by minimizing a loss function composed of a data mismatch term and residual terms associated with the coupled Navier-Stokes and heat transfer equations. We first quantify the accuracy of the proposed method based on a 2D synthetic data set for buoyancy-driven flow, and subsequently apply it to the Tomo-BOS data set, where we are able to infer the instantaneous velocity and pressure fields of the flow over an espresso cup based only on the temperature field provided by the Tomo-BOS imaging. Moreover, we conduct an independent PIV experiment to validate the PINN inference for the unsteady velocity field at a center plane. To explain the observed flow physics, we also perform systematic PINN simulations at different Reynolds and Richardson numbers and quantify the variations in velocity and pressure fields. The results in this paper indicate that the proposed deep learning technique can become a promising direction in experimental fluid mechanics. △ Less

Submitted 9 March, 2021; v1 submitted 3 March, 2021; originally announced March 2021.

Comments: 17 pages, supplementary materials attached