Showing 1–50 of 64 results for author: Stinis, P

Self-adaptive weights based on balanced residual decay rate for physics-informed neural networks and deep operator networks

Authors: Wenqian Chen, Amanda A. Howard, Panos Stinis

Abstract: Physics-informed deep learning has emerged as a promising alternative for solving partial differential equations. However, for complex problems, training these networks can still be challenging, often resulting in unsatisfactory accuracy and efficiency. In this work, we demonstrate that the failure of plain physics-informed neural networks arises from the significant discrepancy in the convergence… ▽ More Physics-informed deep learning has emerged as a promising alternative for solving partial differential equations. However, for complex problems, training these networks can still be challenging, often resulting in unsatisfactory accuracy and efficiency. In this work, we demonstrate that the failure of plain physics-informed neural networks arises from the significant discrepancy in the convergence speed of residuals at different training points, where the slowest convergence speed dominates the overall solution convergence. Based on these observations, we propose a point-wise adaptive weighting method that balances the residual decay rate across different training points. The performance of our proposed adaptive weighting method is compared with current state-of-the-art adaptive weighting methods on benchmark problems for both physics-informed neural networks and physics-informed deep operator networks. Through extensive numerical results we demonstrate that our proposed approach of balanced residual decay rates offers several advantages, including bounded weights, high prediction accuracy, fast convergence speed, low training uncertainty, low computational cost and ease of hyperparameter tuning. △ Less

Submitted 27 June, 2024; originally announced July 2024.

Comments: 13 figures, 4 tables

Report number: PNNL-SA-199965

Finite basis Kolmogorov-Arnold networks: domain decomposition for data-driven and physics-informed problems

Authors: Amanda A. Howard, Bruno Jacob, Sarah H. Murphy, Alexander Heinlein, Panos Stinis

Abstract: Kolmogorov-Arnold networks (KANs) have attracted attention recently as an alternative to multilayer perceptrons (MLPs) for scientific machine learning. However, KANs can be expensive to train, even for relatively small networks. Inspired by finite basis physics-informed neural networks (FBPINNs), in this work, we develop a domain decomposition method for KANs that allows for several small KANs to… ▽ More Kolmogorov-Arnold networks (KANs) have attracted attention recently as an alternative to multilayer perceptrons (MLPs) for scientific machine learning. However, KANs can be expensive to train, even for relatively small networks. Inspired by finite basis physics-informed neural networks (FBPINNs), in this work, we develop a domain decomposition method for KANs that allows for several small KANs to be trained in parallel to give accurate solutions for multiscale problems. We show that finite basis KANs (FBKANs) can provide accurate results with noisy data and for physics-informed training. △ Less

Submitted 28 June, 2024; originally announced June 2024.

Scientific machine learning for closure models in multiscale problems: a review

Authors: Benjamin Sanderse, Panos Stinis, Romit Maulik, Shady E. Ahmed

Abstract: Closure problems are omnipresent when simulating multiscale systems, where some quantities and processes cannot be fully prescribed despite their effects on the simulation's accuracy. Recently, scientific machine learning approaches have been proposed as a way to tackle the closure problem, combining traditional (physics-based) modeling with data-driven (machine-learned) techniques, typically thro… ▽ More Closure problems are omnipresent when simulating multiscale systems, where some quantities and processes cannot be fully prescribed despite their effects on the simulation's accuracy. Recently, scientific machine learning approaches have been proposed as a way to tackle the closure problem, combining traditional (physics-based) modeling with data-driven (machine-learned) techniques, typically through enriching differential equations with neural networks. This paper reviews the different reduced model forms, distinguished by the degree to which they include known physics, and the different objectives of a priori and a posteriori learning. The importance of adhering to physical laws (such as symmetries and conservation laws) in choosing the reduced model form and choosing the learning method is discussed. The effect of spatial and temporal discretization and recent trends toward discretization-invariant models are reviewed. In addition, we make the connections between closure problems and several other research disciplines: inverse problems, Mori-Zwanzig theory, and multi-fidelity methods. In conclusion, much progress has been made with scientific machine learning approaches for solving closure problems, but many challenges remain. In particular, the generalizability and interpretability of learned models is a major issue that needs to be addressed further. △ Less

Submitted 5 March, 2024; originally announced March 2024.

Report number: PNNL-SA-195444 MSC Class: 65MXX; 76MXX; 68T07

Multifidelity domain decomposition-based physics-informed neural networks and operators for time-dependent problems

Authors: Alexander Heinlein, Amanda A. Howard, Damien Beecroft, Panos Stinis

Abstract: Multiscale problems are challenging for neural network-based discretizations of differential equations, such as physics-informed neural networks (PINNs). This can be (partly) attributed to the so-called spectral bias of neural networks. To improve the performance of PINNs for time-dependent problems, a combination of multifidelity stacking PINNs and domain decomposition-based finite basis PINNs is… ▽ More Multiscale problems are challenging for neural network-based discretizations of differential equations, such as physics-informed neural networks (PINNs). This can be (partly) attributed to the so-called spectral bias of neural networks. To improve the performance of PINNs for time-dependent problems, a combination of multifidelity stacking PINNs and domain decomposition-based finite basis PINNs is employed. In particular, to learn the high-fidelity part of the multifidelity model, a domain decomposition in time is employed. The performance is investigated for a pendulum and a two-frequency problem as well as the Allen-Cahn equation. It can be observed that the domain decomposition approach clearly improves the PINN and stacking PINN approaches. Finally, it is demonstrated that the FBPINN approach can be extended to multifidelity physics-informed deep operator networks. △ Less

Submitted 6 June, 2024; v1 submitted 15 January, 2024; originally announced January 2024.

MSC Class: 65M22; 65M55; 68T07

Physics-Guided Continual Learning for Predicting Emerging Aqueous Organic Redox Flow Battery Material Performance

Authors: Yucheng Fu, Amanda Howard, Chao Zeng, Yunxiang Chen, Peiyuan Gao, Panos Stinis

Abstract: Aqueous organic redox flow batteries (AORFBs) have gained popularity in renewable energy storage due to their low cost, environmental friendliness and scalability. The rapid discovery of aqueous soluble organic (ASO) redox-active materials necessitates efficient machine learning surrogates for predicting battery performance. The physics-guided continual learning (PGCL) method proposed in this stud… ▽ More Aqueous organic redox flow batteries (AORFBs) have gained popularity in renewable energy storage due to their low cost, environmental friendliness and scalability. The rapid discovery of aqueous soluble organic (ASO) redox-active materials necessitates efficient machine learning surrogates for predicting battery performance. The physics-guided continual learning (PGCL) method proposed in this study can incrementally learn data from new ASO electrolytes while addressing catastrophic forgetting issues in conventional machine learning. Using a ASO anolyte database with a thousand potential materials generated by a 780 $\text{cm}^2$ interdigitated cell model, PGCL incorporates AORFB physics to optimize the continual learning task formation and training process. This achieves higher efficiency and robustness compared to the non-physics-guided continual learning while retaining previously learned battery material knowledge. The trained PGCL demonstrates its capability in assessing emerging ASO materials within the established parameter space when evaluated with the dihydroxyphenazine isomers. △ Less

Submitted 8 May, 2024; v1 submitted 13 December, 2023; originally announced December 2023.

Comments: 12 pages, 6 figures

Rethinking Skip Connections in Spiking Neural Networks with Time-To-First-Spike Coding

Authors: Youngeun Kim, Adar Kahana, Ruokai Yin, Yuhang Li, Panos Stinis, George Em Karniadakis, Priyadarshini Panda

Abstract: Time-To-First-Spike (TTFS) coding in Spiking Neural Networks (SNNs) offers significant advantages in terms of energy efficiency, closely mimicking the behavior of biological neurons. In this work, we delve into the role of skip connections, a widely used concept in Artificial Neural Networks (ANNs), within the domain of SNNs with TTFS coding. Our focus is on two distinct types of skip connection a… ▽ More Time-To-First-Spike (TTFS) coding in Spiking Neural Networks (SNNs) offers significant advantages in terms of energy efficiency, closely mimicking the behavior of biological neurons. In this work, we delve into the role of skip connections, a widely used concept in Artificial Neural Networks (ANNs), within the domain of SNNs with TTFS coding. Our focus is on two distinct types of skip connection architectures: (1) addition-based skip connections, and (2) concatenation-based skip connections. We find that addition-based skip connections introduce an additional delay in terms of spike timing. On the other hand, concatenation-based skip connections circumvent this delay but produce time gaps between after-convolution and skip connection paths, thereby restricting the effective mixing of information from these two paths. To mitigate these issues, we propose a novel approach involving a learnable delay for skip connections in the concatenation-based skip connection architecture. This approach successfully bridges the time gap between the convolutional and skip branches, facilitating improved information mixing. We conduct experiments on public datasets including MNIST and Fashion-MNIST, illustrating the advantage of the skip connection in TTFS coding architectures. Additionally, we demonstrate the applicability of TTFS coding on beyond image recognition tasks and extend it to scientific machine-learning tasks, broadening the potential uses of SNNs. △ Less

Submitted 1 December, 2023; originally announced December 2023.

Stacked networks improve physics-informed training: applications to neural networks and deep operator networks

Authors: Amanda A Howard, Sarah H Murphy, Shady E Ahmed, Panos Stinis

Abstract: Physics-informed neural networks and operator networks have shown promise for effectively solving equations modeling physical systems. However, these networks can be difficult or impossible to train accurately for some systems of equations. We present a novel multifidelity framework for stacking physics-informed neural networks and operator networks that facilitates training. We successively build… ▽ More Physics-informed neural networks and operator networks have shown promise for effectively solving equations modeling physical systems. However, these networks can be difficult or impossible to train accurately for some systems of equations. We present a novel multifidelity framework for stacking physics-informed neural networks and operator networks that facilitates training. We successively build a chain of networks, where the output at one step can act as a low-fidelity input for training the next step, gradually increasing the expressivity of the learned model. The equations imposed at each step of the iterative process can be the same or different (akin to simulated annealing). The iterative (stacking) nature of the proposed method allows us to progressively learn features of a solution that are hard to learn directly. Through benchmark problems including a nonlinear pendulum, the wave equation, and the viscous Burgers equation, we show how stacking can be used to improve the accuracy and reduce the required size of physics-informed neural networks and operator networks. △ Less

Submitted 20 November, 2023; v1 submitted 11 November, 2023; originally announced November 2023.

Efficient kernel surrogates for neural network-based regression

Authors: Saad Qadeer, Andrew Engel, Amanda Howard, Adam Tsou, Max Vargas, Panos Stinis, Tony Chiang

Abstract: Despite their immense promise in performing a variety of learning tasks, a theoretical understanding of the limitations of Deep Neural Networks (DNNs) has so far eluded practitioners. This is partly due to the inability to determine the closed forms of the learned functions, making it harder to study their generalization properties on unseen datasets. Recent work has shown that randomly initialize… ▽ More Despite their immense promise in performing a variety of learning tasks, a theoretical understanding of the limitations of Deep Neural Networks (DNNs) has so far eluded practitioners. This is partly due to the inability to determine the closed forms of the learned functions, making it harder to study their generalization properties on unseen datasets. Recent work has shown that randomly initialized DNNs in the infinite width limit converge to kernel machines relying on a Neural Tangent Kernel (NTK) with known closed form. These results suggest, and experimental evidence corroborates, that empirical kernel machines can also act as surrogates for finite width DNNs. The high computational cost of assembling the full NTK, however, makes this approach infeasible in practice, motivating the need for low-cost approximations. In the current work, we study the performance of the Conjugate Kernel (CK), an efficient approximation to the NTK that has been observed to yield fairly similar results. For the regression problem of smooth functions and logistic regression classification, we show that the CK performance is only marginally worse than that of the NTK and, in certain cases, is shown to be superior. In particular, we establish bounds for the relative test losses, verify them with numerical tests, and identify the regularity of the kernel as the key determinant of performance. In addition to providing a theoretical grounding for using CKs instead of NTKs, our framework suggests a recipe for improving DNN accuracy inexpensively. We present a demonstration of this on the foundation model GPT-2 by comparing its performance on a classification task using a conventional approach and our prescription. We also show how our approach can be used to improve physics-informed operator network training for regression tasks as well as convolutional neural network training for vision classification tasks. △ Less

Submitted 24 January, 2024; v1 submitted 28 October, 2023; originally announced October 2023.

Comments: 35 pages. software used to reach results available upon request, approved for release by Pacific Northwest National Laboratory

Report number: PNNL-SA-191858 MSC Class: 68T07; 65M99

Exploring Learned Representations of Neural Networks with Principal Component Analysis

Authors: Amit Harlev, Andrew Engel, Panos Stinis, Tony Chiang

Abstract: Understanding feature representation for deep neural networks (DNNs) remains an open question within the general field of explainable AI. We use principal component analysis (PCA) to study the performance of a k-nearest neighbors classifier (k-NN), nearest class-centers classifier (NCC), and support vector machines on the learned layer-wise representations of a ResNet-18 trained on CIFAR-10. We sh… ▽ More Understanding feature representation for deep neural networks (DNNs) remains an open question within the general field of explainable AI. We use principal component analysis (PCA) to study the performance of a k-nearest neighbors classifier (k-NN), nearest class-centers classifier (NCC), and support vector machines on the learned layer-wise representations of a ResNet-18 trained on CIFAR-10. We show that in certain layers, as little as 20% of the intermediate feature-space variance is necessary for high-accuracy classification and that across all layers, the first ~100 PCs completely determine the performance of the k-NN and NCC classifiers. We relate our findings to neural collapse and provide partial evidence for the related phenomenon of intermediate neural collapse. Our preliminary work provides three distinct yet interpretable surrogate models for feature representation with an affine linear model the best performing. We also show that leveraging several surrogate models affords us a clever method to estimate where neural collapse may initially occur within the DNN. △ Less

Submitted 26 September, 2023; originally announced September 2023.

Comments: 5 pages, 3 figures

Physics-informed machine learning of the correlation functions in bulk fluids

Authors: Wenqian Chen, Peiyuan Gao, Panos Stinis

Abstract: The Ornstein-Zernike (OZ) equation is the fundamental equation for pair correlation function computations in the modern integral equation theory for liquids. In this work, machine learning models, notably physics-informed neural networks and physics-informed neural operator networks, are explored to solve the OZ equation. The physics-informed machine learning models demonstrate great accuracy and… ▽ More The Ornstein-Zernike (OZ) equation is the fundamental equation for pair correlation function computations in the modern integral equation theory for liquids. In this work, machine learning models, notably physics-informed neural networks and physics-informed neural operator networks, are explored to solve the OZ equation. The physics-informed machine learning models demonstrate great accuracy and high efficiency in solving the forward and inverse OZ problems of various bulk fluids. The results highlight the significant potential of physics-informed machine learning for applications in thermodynamic state theory. △ Less

Submitted 1 September, 2023; originally announced September 2023.

Comments: 8 figures

Report number: PNNL-SA-189736

Physics-informed machine learning of redox flow battery based on a two-dimensional unit cell model

Authors: Wenqian Chen, Yucheng Fu, Panos Stinis

Abstract: In this paper, we present a physics-informed neural network (PINN) approach for predicting the performance of an all-vanadium redox flow battery, with its physics constraints enforced by a two-dimensional (2D) mathematical model. The 2D model, which includes 6 governing equations and 24 boundary conditions, provides a detailed representation of the electrochemical reactions, mass transport and hyd… ▽ More In this paper, we present a physics-informed neural network (PINN) approach for predicting the performance of an all-vanadium redox flow battery, with its physics constraints enforced by a two-dimensional (2D) mathematical model. The 2D model, which includes 6 governing equations and 24 boundary conditions, provides a detailed representation of the electrochemical reactions, mass transport and hydrodynamics occurring inside the redox flow battery. To solve the 2D model with the PINN approach, a composite neural network is employed to approximate species concentration and potentials; the input and output are normalized according to prior knowledge of the battery system; the governing equations and boundary conditions are first scaled to an order of magnitude around 1, and then further balanced with a self-weighting method. Our numerical results show that the PINN is able to predict cell voltage correctly, but the prediction of potentials shows a constant-like shift. To fix the shift, the PINN is enhanced by further constrains derived from the current collector boundary. Finally, we show that the enhanced PINN can be even further improved if a small number of labeled data is available. △ Less

Submitted 7 September, 2023; v1 submitted 31 May, 2023; originally announced June 2023.

Comments: 7 figures

Report number: PNNL-SA-185779

A multifidelity approach to continual learning for physical systems

Authors: Amanda Howard, Yucheng Fu, Panos Stinis

Abstract: We introduce a novel continual learning method based on multifidelity deep neural networks. This method learns the correlation between the output of previously trained models and the desired output of the model on the current training dataset, limiting catastrophic forgetting. On its own the multifidelity continual learning method shows robust results that limit forgetting across several datasets.… ▽ More We introduce a novel continual learning method based on multifidelity deep neural networks. This method learns the correlation between the output of previously trained models and the desired output of the model on the current training dataset, limiting catastrophic forgetting. On its own the multifidelity continual learning method shows robust results that limit forgetting across several datasets. Additionally, we show that the multifidelity method can be combined with existing continual learning methods, including replay and memory aware synapses, to further limit catastrophic forgetting. The proposed continual learning method is especially suited for physical problems where the data satisfy the same physical laws on each domain, or for physics-informed neural networks, because in these cases we expect there to be a strong correlation between the output of the previous model and the model on the current training domain. △ Less

Submitted 9 February, 2024; v1 submitted 7 April, 2023; originally announced April 2023.

Feature-adjacent multi-fidelity physics-informed machine learning for partial differential equations

Authors: Wenqian Chen, Panos Stinis

Abstract: Physics-informed neural networks have emerged as an alternative method for solving partial differential equations. However, for complex problems, the training of such networks can still require high-fidelity data which can be expensive to generate. To reduce or even eliminate the dependency on high-fidelity data, we propose a novel multi-fidelity architecture which is based on a feature space shar… ▽ More Physics-informed neural networks have emerged as an alternative method for solving partial differential equations. However, for complex problems, the training of such networks can still require high-fidelity data which can be expensive to generate. To reduce or even eliminate the dependency on high-fidelity data, we propose a novel multi-fidelity architecture which is based on a feature space shared by the low- and high-fidelity solutions. In the feature space, the projections of the low-fidelity and high-fidelity solutions are adjacent by constraining their relative distance. The feature space is represented with an encoder and its map** to the original solution space is effected through a decoder. The proposed multi-fidelity approach is validated on forward and inverse problems for steady and unsteady problems described by partial differential equations. △ Less

Submitted 27 March, 2023; v1 submitted 20 March, 2023; originally announced March 2023.

Comments: 12 figures

Report number: PNNL-SA-182880

A Multifidelity deep operator network approach to closure for multiscale systems

Authors: Shady E. Ahmed, Panos Stinis

Abstract: Projection-based reduced order models (PROMs) have shown promise in representing the behavior of multiscale systems using a small set of generalized (or latent) variables. Despite their success, PROMs can be susceptible to inaccuracies, even instabilities, due to the improper accounting of the interaction between the resolved and unresolved scales of the multiscale system (known as the closure pro… ▽ More Projection-based reduced order models (PROMs) have shown promise in representing the behavior of multiscale systems using a small set of generalized (or latent) variables. Despite their success, PROMs can be susceptible to inaccuracies, even instabilities, due to the improper accounting of the interaction between the resolved and unresolved scales of the multiscale system (known as the closure problem). In the current work, we interpret closure as a multifidelity problem and use a multifidelity deep operator network (DeepONet) framework to address it. In addition, to enhance the stability and accuracy of the multifidelity-based closure, we employ the recently developed "in-the-loop" training approach from the literature on coupling physics and machine learning models. The resulting approach is tested on shock advection for the one-dimensional viscous Burgers equation and vortex merging using the two-dimensional Navier-Stokes equations. The numerical experiments show significant improvement of the predictive ability of the closure-corrected PROM over the un-corrected one both in the interpolative and the extrapolative regimes. △ Less

Submitted 1 June, 2023; v1 submitted 15 March, 2023; originally announced March 2023.

Comments: 24 pages, 21 figures

Report number: PNNL-SA-182879

ViTO: Vision Transformer-Operator

Authors: Oded Ovadia, Adar Kahana, Panos Stinis, Eli Turkel, George Em Karniadakis

Abstract: We combine vision transformers with operator learning to solve diverse inverse problems described by partial differential equations (PDEs). Our approach, named ViTO, combines a U-Net based architecture with a vision transformer. We apply ViTO to solve inverse PDE problems of increasing complexity, namely for the wave equation, the Navier-Stokes equations and the Darcy equation. We focus on the mor… ▽ More We combine vision transformers with operator learning to solve diverse inverse problems described by partial differential equations (PDEs). Our approach, named ViTO, combines a U-Net based architecture with a vision transformer. We apply ViTO to solve inverse PDE problems of increasing complexity, namely for the wave equation, the Navier-Stokes equations and the Darcy equation. We focus on the more challenging case of super-resolution, where the input dataset for the inverse problem is at a significantly coarser resolution than the output. The results we obtain are comparable or exceed the leading operator network benchmarks in terms of accuracy. Furthermore, ViTO`s architecture has a small number of trainable parameters (less than 10% of the leading competitor), resulting in a performance speed-up of over 5x when averaged over the various test cases. △ Less

Submitted 15 March, 2023; originally announced March 2023.

Report number: PNNL-SA-182861

SDYN-GANs: Adversarial Learning Methods for Multistep Generative Models for General Order Stochastic Dynamics

Authors: Panos Stinis, Constantinos Daskalakis, Paul J. Atzberger

Abstract: We introduce adversarial learning methods for data-driven generative modeling of the dynamics of $n^{th}$-order stochastic systems. Our approach builds on Generative Adversarial Networks (GANs) with generative model classes based on stable $m$-step stochastic numerical integrators. We introduce different formulations and training methods for learning models of stochastic dynamics based on observat… ▽ More We introduce adversarial learning methods for data-driven generative modeling of the dynamics of $n^{th}$-order stochastic systems. Our approach builds on Generative Adversarial Networks (GANs) with generative model classes based on stable $m$-step stochastic numerical integrators. We introduce different formulations and training methods for learning models of stochastic dynamics based on observation of trajectory samples. We develop approaches using discriminators based on Maximum Mean Discrepancy (MMD), training protocols using conditional and marginal distributions, and methods for learning dynamic responses over different time-scales. We show how our approaches can be used for modeling physical systems to learn force-laws, dam** coefficients, and noise-related parameters. The adversarial learning approaches provide methods for obtaining stable generative models for dynamic tasks including long-time prediction and develo** simulations for stochastic systems. △ Less

Submitted 7 February, 2023; originally announced February 2023.

Comments: 7 figures

Report number: PNNL-SA-181736

A Hybrid Deep Neural Operator/Finite Element Method for Ice-Sheet Modeling

Authors: QiZhi He, Mauro Perego, Amanda A. Howard, George Em Karniadakis, Panos Stinis

Abstract: One of the most challenging and consequential problems in climate modeling is to provide probabilistic projections of sea level rise. A large part of the uncertainty of sea level projections is due to uncertainty in ice sheet dynamics. At the moment, accurate quantification of the uncertainty is hindered by the cost of ice sheet computational models. In this work, we develop a hybrid approach to a… ▽ More One of the most challenging and consequential problems in climate modeling is to provide probabilistic projections of sea level rise. A large part of the uncertainty of sea level projections is due to uncertainty in ice sheet dynamics. At the moment, accurate quantification of the uncertainty is hindered by the cost of ice sheet computational models. In this work, we develop a hybrid approach to approximate existing ice sheet computational models at a fraction of their cost. Our approach consists of replacing the finite element model for the momentum equations for the ice velocity, the most expensive part of an ice sheet model, with a Deep Operator Network, while retaining a classic finite element discretization for the evolution of the ice thickness. We show that the resulting hybrid model is very accurate and it is an order of magnitude faster than the traditional finite element model. Further, a distinctive feature of the proposed model compared to other neural network approaches, is that it can handle high-dimensional parameter spaces (parameter fields) such as the basal friction at the bed of the glacier, and can therefore be used for generating samples for uncertainty quantification. We study the impact of hyper-parameters, number of unknowns and correlation length of the parameter distribution on the training and accuracy of the Deep Operator Network on a synthetic ice sheet model. We then target the evolution of the Humboldt glacier in Greenland and show that our hybrid model can provide accurate statistics of the glacier mass loss and can be effectively used to accelerate the quantification of uncertainty. △ Less

Submitted 26 January, 2023; originally announced January 2023.

SMS: Spiking Marching Scheme for Efficient Long Time Integration of Differential Equations

Authors: Qian Zhang, Adar Kahana, George Em Karniadakis, Panos Stinis

Abstract: We propose a Spiking Neural Network (SNN)-based explicit numerical scheme for long time integration of time-dependent Ordinary and Partial Differential Equations (ODEs, PDEs). The core element of the method is a SNN, trained to use spike-encoded information about the solution at previous timesteps to predict spike-encoded information at the next timestep. After the network has been trained, it ope… ▽ More We propose a Spiking Neural Network (SNN)-based explicit numerical scheme for long time integration of time-dependent Ordinary and Partial Differential Equations (ODEs, PDEs). The core element of the method is a SNN, trained to use spike-encoded information about the solution at previous timesteps to predict spike-encoded information at the next timestep. After the network has been trained, it operates as an explicit numerical scheme that can be used to compute the solution at future timesteps, given a spike-encoded initial condition. A decoder is used to transform the evolved spiking-encoded solution back to function values. We present results from numerical experiments of using the proposed method for ODEs and PDEs of varying complexity. △ Less

Submitted 17 November, 2022; originally announced November 2022.

Comments: 14 pages, 7 figures

Report number: PNNL-SA-179601 MSC Class: 65M99

Vibrational Levels of a Generalized Morse Potential

Authors: Saad Qadeer, Garrett D. Santis, Panos Stinis, Sotiris S. Xantheas

Abstract: A Generalized Morse Potential (GMP) is an extension of the Morse Potential (MP) with an additional exponential term and an additional parameter that compensate for MP's erroneous behavior in the long range part of the interaction potential. Because of the additional term and parameter, the vibrational levels of the GMP cannot be solved analytically, unlike the case for the MP. We present several n… ▽ More A Generalized Morse Potential (GMP) is an extension of the Morse Potential (MP) with an additional exponential term and an additional parameter that compensate for MP's erroneous behavior in the long range part of the interaction potential. Because of the additional term and parameter, the vibrational levels of the GMP cannot be solved analytically, unlike the case for the MP. We present several numerical approaches for solving the vibrational problem of the GMP based on Galerkin methods, namely the Laguerre Polynomial Method (LPM), the Symmetrized Laguerre Polynomial Method (SLPM) and the Polynomial Expansion method (PEM) and apply them to the vibrational levels of the homonuclear diatomic molecules B$_2$, O$_2$ and F$_2$, for which high level theoretical Full CI potential energy surfaces and experimentally measured vibrational levels have been reported. Overall the LPM produces vibrational states for the GMP that are converged to within spectroscopic accuracy of 0.01 cm$^{-1}$ in between 1 and 2 orders of magnitude faster and with much fewer basis functions / grid points than the Colbert-Miller Discrete Variable Representation (CM-DVR) method for the three homonuclear diatomic molecules examined in this study. △ Less

Submitted 17 June, 2022; originally announced June 2022.

Comments: 18 pages, approved for release by Pacific Northwest National Laboratory (PNNL-SA-174299). A python library that fits and solves the GMP and similar potentials can be downloaded from https://gitlab.com/gds001uw/generalized-morse-solver

MSC Class: 65L15

Multifidelity Deep Operator Networks For Data-Driven and Physics-Informed Problems

Authors: Amanda A. Howard, Mauro Perego, George E. Karniadakis, Panos Stinis

Abstract: Operator learning for complex nonlinear systems is increasingly common in modeling multi-physics and multi-scale systems. However, training such high-dimensional operators requires a large amount of expensive, high-fidelity data, either from experiments or simulations. In this work, we present a composite Deep Operator Network (DeepONet) for learning using two datasets with different levels of fid… ▽ More Operator learning for complex nonlinear systems is increasingly common in modeling multi-physics and multi-scale systems. However, training such high-dimensional operators requires a large amount of expensive, high-fidelity data, either from experiments or simulations. In this work, we present a composite Deep Operator Network (DeepONet) for learning using two datasets with different levels of fidelity to accurately learn complex operators when sufficient high-fidelity data is not available. Additionally, we demonstrate that the presence of low-fidelity data can improve the predictions of physics-informed learning with DeepONets. We demonstrate the new multi-fidelity training in diverse examples, including modeling of the ice-sheet dynamics of the Humboldt glacier, Greenland, using two different fidelity models and also using the same physical model at two different resolutions. △ Less

Submitted 21 November, 2023; v1 submitted 19 April, 2022; originally announced April 2022.

Enhanced physics-constrained deep neural networks for modeling vanadium redox flow battery

Authors: QiZhi He, Yucheng Fu, Panos Stinis, Alexandre Tartakovsky

Abstract: Numerical modeling and simulation have become indispensable tools for advancing a comprehensive understanding of the underlying mechanisms and cost-effective process optimization and control of flow batteries. In this study, we propose an enhanced version of the physics-constrained deep neural network (PCDNN) approach [1] to provide high-accuracy voltage predictions in the vanadium redox flow batt… ▽ More Numerical modeling and simulation have become indispensable tools for advancing a comprehensive understanding of the underlying mechanisms and cost-effective process optimization and control of flow batteries. In this study, we propose an enhanced version of the physics-constrained deep neural network (PCDNN) approach [1] to provide high-accuracy voltage predictions in the vanadium redox flow batteries (VRFBs). The purpose of the PCDNN approach is to enforce the physics-based zero-dimensional (0D) VRFB model in a neural network to assure model generalization for various battery operation conditions. Limited by the simplifications of the 0D model, the PCDNN cannot capture sharp voltage changes in the extreme SOC regions. To improve the accuracy of voltage prediction at extreme ranges, we introduce a second (enhanced) DNN to mitigate the prediction errors carried from the 0D model itself and call the resulting approach enhanced PCDNN (ePCDNN). By comparing the model prediction with experimental data, we demonstrate that the ePCDNN approach can accurately capture the voltage response throughout the charge--discharge cycle, including the tail region of the voltage discharge curve. Compared to the standard PCDNN, the prediction accuracy of the ePCDNN is significantly improved. The loss function for training the ePCDNN is designed to be flexible by adjusting the weights of the physics-constrained DNN and the enhanced DNN. This allows the ePCDNN framework to be transferable to battery systems with variable physical model fidelity. △ Less

Submitted 3 March, 2022; originally announced March 2022.

Machine-learning custom-made basis functions for partial differential equations

Authors: Brek Meuris, Saad Qadeer, Panos Stinis

Abstract: Spectral methods are an important part of scientific computing's arsenal for solving partial differential equations (PDEs). However, their applicability and effectiveness depend crucially on the choice of basis functions used to expand the solution of a PDE. The last decade has seen the emergence of deep learning as a strong contender in providing efficient representations of complex functions. In… ▽ More Spectral methods are an important part of scientific computing's arsenal for solving partial differential equations (PDEs). However, their applicability and effectiveness depend crucially on the choice of basis functions used to expand the solution of a PDE. The last decade has seen the emergence of deep learning as a strong contender in providing efficient representations of complex functions. In the current work, we present an approach for combining deep neural networks with spectral methods to solve PDEs. In particular, we use a deep learning technique known as the Deep Operator Network (DeepONet), to identify candidate functions on which to expand the solution of PDEs. We have devised an approach which uses the candidate functions provided by the DeepONet as a starting point to construct a set of functions which have the following properties: i) they constitute a basis, 2) they are orthonormal, and 3) they are hierarchical i.e., akin to Fourier series or orthogonal polynomials. We have exploited the favorable properties of our custom-made basis functions to both study their approximation capability and use them to expand the solution of linear and nonlinear time-dependent PDEs. △ Less

Submitted 9 November, 2021; originally announced November 2021.

Comments: 35 pages, software used to reach results available upon request, approved for release by Pacific Northwest National Laboratory (PNNL-SA-168281)

Report number: PNNL-SA-168281 MSC Class: 65M70

Structure-preserving Sparse Identification of Nonlinear Dynamics for Data-driven Modeling

Authors: Kook** Lee, Nathaniel Trask, Panos Stinis

Abstract: Discovery of dynamical systems from data forms the foundation for data-driven modeling and recently, structure-preserving geometric perspectives have been shown to provide improved forecasting, stability, and physical realizability guarantees. We present here a unification of the Sparse Identification of Nonlinear Dynamics (SINDy) formalism with neural ordinary differential equations. The resultin… ▽ More Discovery of dynamical systems from data forms the foundation for data-driven modeling and recently, structure-preserving geometric perspectives have been shown to provide improved forecasting, stability, and physical realizability guarantees. We present here a unification of the Sparse Identification of Nonlinear Dynamics (SINDy) formalism with neural ordinary differential equations. The resulting framework allows learning of both "black-box" dynamics and learning of structure preserving bracket formalisms for both reversible and irreversible dynamics. We present a suite of benchmarks demonstrating effectiveness and structure preservation, including for chaotic systems. △ Less

Submitted 11 September, 2021; originally announced September 2021.

Machine learning structure preserving brackets for forecasting irreversible processes

Authors: Kook** Lee, Nathaniel A. Trask, Panos Stinis

Abstract: Forecasting of time-series data requires imposition of inductive biases to obtain predictive extrapolation, and recent works have imposed Hamiltonian/Lagrangian form to preserve structure for systems with reversible dynamics. In this work we present a novel parameterization of dissipative brackets from metriplectic dynamical systems appropriate for learning irreversible dynamics with unknown a pri… ▽ More Forecasting of time-series data requires imposition of inductive biases to obtain predictive extrapolation, and recent works have imposed Hamiltonian/Lagrangian form to preserve structure for systems with reversible dynamics. In this work we present a novel parameterization of dissipative brackets from metriplectic dynamical systems appropriate for learning irreversible dynamics with unknown a priori model form. The process learns generalized Casimirs for energy and entropy guaranteed to be conserved and nondecreasing, respectively. Furthermore, for the case of added thermal noise, we guarantee exact preservation of a fluctuation-dissipation theorem, ensuring thermodynamic consistency. We provide benchmarks for dissipative systems demonstrating learned dynamics are more robust and generalize better than either "black-box" or penalty-based approaches. △ Less

Submitted 23 June, 2021; originally announced June 2021.

Physics-constrained deep neural network method for estimating parameters in a redox flow battery

Authors: QiZhi He, Panos Stinis, Alexandre Tartakovsky

Abstract: In this paper, we present a physics-constrained deep neural network (PCDNN) method for parameter estimation in the zero-dimensional (0D) model of the vanadium redox flow battery (VRFB). In this approach, we use deep neural networks (DNNs) to approximate the model parameters as functions of the operating conditions. This method allows the integration of the VRFB computational models as the physical… ▽ More In this paper, we present a physics-constrained deep neural network (PCDNN) method for parameter estimation in the zero-dimensional (0D) model of the vanadium redox flow battery (VRFB). In this approach, we use deep neural networks (DNNs) to approximate the model parameters as functions of the operating conditions. This method allows the integration of the VRFB computational models as the physical constraints in the parameter learning process, leading to enhanced accuracy of parameter estimation and cell voltage prediction. Using an experimental dataset, we demonstrate that the PCDNN method can estimate model parameters for a range of operating conditions and improve the 0D model prediction of voltage compared to the 0D model prediction with constant operation-condition-independent parameters estimated with traditional inverse methods. We also demonstrate that the PCDNN approach has an improved generalization ability for estimating parameter values for operating conditions not used in the DNN training. △ Less

Submitted 4 March, 2022; v1 submitted 21 June, 2021; originally announced June 2021.

Time-dependent stochastic basis adaptation for uncertainty quantification

Authors: Ramakrishna Tipireddy, Panos Stinis, Alexandre M. Tartakovsky

Abstract: We extend stochastic basis adaptation and spatial domain decomposition methods to solve time varying stochastic partial differential equations (SPDEs) with a large number of input random parameters. Stochastic basis adaptation allows the determination of a low dimensional stochastic basis representation of a quantity of interest (QoI). Extending basis adaptation to time-dependent problems is chall… ▽ More We extend stochastic basis adaptation and spatial domain decomposition methods to solve time varying stochastic partial differential equations (SPDEs) with a large number of input random parameters. Stochastic basis adaptation allows the determination of a low dimensional stochastic basis representation of a quantity of interest (QoI). Extending basis adaptation to time-dependent problems is challenging because small errors introduced in the previous time steps of the low dimensional approximate solution accumulate over time and cause divergence from the true solution. To address this issue we have introduced an approach where the basis adaptation varies at every time step so that the low dimensional basis is adapted to the QoI at that time step. We have coupled the time-dependent basis adaptation with domain decomposition to further increase the accuracy in the representation of the QoI. To illustrate the construction, we present numerical results for one-dimensional time varying linear and nonlinear diffusion equations with random space-dependent diffusion coefficients. Stochastic dimension reduction techniques proposed in the literature have mainly focused on quantifying the uncertainty in time independent and scalar QoI. To the best of our knowledge, this is the first attempt to extend dimensional reduction techniques to time varying and spatially dependent quantities such as the solution of SPDEs. △ Less

Submitted 4 March, 2021; originally announced March 2021.

Optimal renormalization of multi-scale systems

Authors: Jacob Price, Brek Meuris, Madelyn Shapiro, Panos Stinis

Abstract: While model order reduction is a promising approach in dealing with multi-scale time-dependent systems that are too large or too expensive to simulate for long times, the resulting reduced order models can suffer from instabilities. We have recently developed a time-dependent renormalization approach to stabilize such reduced models. In the current work, we extend this framework by introducing a p… ▽ More While model order reduction is a promising approach in dealing with multi-scale time-dependent systems that are too large or too expensive to simulate for long times, the resulting reduced order models can suffer from instabilities. We have recently developed a time-dependent renormalization approach to stabilize such reduced models. In the current work, we extend this framework by introducing a parameter that controls the time-decay of the memory of such models and optimally selecting this parameter based on limited fully resolved simulations. First, we demonstrate our framework on the inviscid Burgers equation whose solution develops a finite-time singularity. Our renormalized reduced order models are stable and accurate for long times while using for their calibration only data from a full order simulation before the occurrence of the singularity. Furthermore, we apply this framework to the 3D Euler equations of incompressible fluid flow, where the problem of finite-time singularity formation is still open and where brute force simulation is only feasible for short times. Our approach allows us to obtain for the first time a perturbatively renormalizable model which is stable for long times and includes all the complex effects present in the 3D Euler dynamics. We find that, in each application, the renormalization coefficients display algebraic decay with increasing resolution, and that the parameter which controls the time-decay of the memory is problem-dependent. △ Less

Submitted 24 January, 2021; originally announced January 2021.

Comments: 28 pages, software used to reach results available upon request, approved for release by Pacific Northwest National Laboratory (PNNL-SA-159021). arXiv admin note: substantial text overlap with arXiv:1805.08766

Report number: PNNL-SA-159021 MSC Class: 65M99; 35D30; 35B44

Model reduction for a power grid model

Authors: **g Li, Panos Stinis

Abstract: We apply model reduction techniques to the DeMarco power grid model. The DeMarco model, when augmented by an appropriate line failure mechanism, can be used to study cascade failures. Here we examine the DeMarco model without the line failure mechanism and we investigate how to construct reduced order models for subsets of the state variables. We show that due to the oscillating nature of the solu… ▽ More We apply model reduction techniques to the DeMarco power grid model. The DeMarco model, when augmented by an appropriate line failure mechanism, can be used to study cascade failures. Here we examine the DeMarco model without the line failure mechanism and we investigate how to construct reduced order models for subsets of the state variables. We show that due to the oscillating nature of the solutions and the absence of timescale separation between resolved and unresolved variables, the construction of accurate reduced models becomes highly non-trivial since one has to account for long memory effects. In addition, we show that a reduced model which includes even a short memory is drastically better than a memoryless model. △ Less

Submitted 18 December, 2019; originally announced December 2019.

Comments: 27 pages

Report number: PNNL-SA-150116 MSC Class: 65C20; 65M99; 65L99; 65Z05

A Kinetic Monte Carlo Approach for Simulating Cascading Transmission Line Failure

Authors: Jacob Roth, David A. Barajas-Solano, Panos Stinis, Jonathan Weare, Mihai Anitescu

Abstract: In this work, cascading transmission line failures are studied through a dynamical model of the power system operating under fixed conditions. The power grid is modeled as a stochastic dynamical system where first-principles electromechanical dynamics are excited by small Gaussian disturbances in demand and generation around a specified operating point. In this context, a single line failure is in… ▽ More In this work, cascading transmission line failures are studied through a dynamical model of the power system operating under fixed conditions. The power grid is modeled as a stochastic dynamical system where first-principles electromechanical dynamics are excited by small Gaussian disturbances in demand and generation around a specified operating point. In this context, a single line failure is interpreted in a large deviation context as a first escape event across a surface in phase space defined by line security constraints. The resulting system of stochastic differential equations admits a transverse decomposition of the drift, which leads to considerable simplification in evaluating the quasipotential (rate function) and, consequently, computation of exit rates. Tractable expressions for the rate of transmission line failure in a restricted network are derived from large deviation theory arguments and validated against numerical simulations. Extensions to realistic settings are considered, and individual line failure models are aggregated into a Markov model of cascading failure inspired by chemical kinetics. Cascades are generated by traversing a graph composed of weighted edges representing transitions to degraded network topologies. Numerical results indicate that the Markov model can produce cascades with qualitative power-law properties similar to those observed in empirical cascades. △ Less

Submitted 15 December, 2019; originally announced December 2019.

MSC Class: 60H30; 68U20; 37H10

Enforcing constraints for time series prediction in supervised, unsupervised and reinforcement learning

Authors: Panos Stinis

Abstract: We assume that we are given a time series of data from a dynamical system and our task is to learn the flow map of the dynamical system. We present a collection of results on how to enforce constraints coming from the dynamical system in order to accelerate the training of deep neural networks to represent the flow map of the system as well as increase their predictive ability. In particular, we p… ▽ More We assume that we are given a time series of data from a dynamical system and our task is to learn the flow map of the dynamical system. We present a collection of results on how to enforce constraints coming from the dynamical system in order to accelerate the training of deep neural networks to represent the flow map of the system as well as increase their predictive ability. In particular, we provide ways to enforce constraints during training for all three major modes of learning, namely supervised, unsupervised and reinforcement learning. In general, the dynamic constraints need to include terms which are analogous to memory terms in model reduction formalisms. Such memory terms act as a restoring force which corrects the errors committed by the learned flow map during prediction. For supervised learning, the constraints are added to the objective function. For the case of unsupervised learning, in particular generative adversarial networks, the constraints are introduced by augmenting the input of the discriminator. Finally, for the case of reinforcement learning and in particular actor-critic methods, the constraints are added to the reward function. In addition, for the reinforcement learning case, we present a novel approach based on homotopy of the action-value function in order to stabilize and accelerate training. We use numerical results for the Lorenz system to illustrate the various constructions. △ Less

Submitted 17 May, 2019; originally announced May 2019.

Comments: 30 pages, 5 figures

Report number: PNNL-SA-143654 MSC Class: 37M05; 37M10; 62M45; 68Q32; 68T05

Improving solution accuracy and convergence for stochastic physics parameterizations with colored noise

Authors: Panos Stinis, Huan Lei, **g Li, Hui Wan

Abstract: Stochastic parameterizations are used in numerical weather prediction and climate modeling to help capture the uncertainty in the simulations and improve their statistical properties. Convergence issues can arise when time integration methods originally developed for deterministic differential equations are applied naively to stochastic problems. (Hodyss et al 2013, 2014) demonstrated that a corre… ▽ More Stochastic parameterizations are used in numerical weather prediction and climate modeling to help capture the uncertainty in the simulations and improve their statistical properties. Convergence issues can arise when time integration methods originally developed for deterministic differential equations are applied naively to stochastic problems. (Hodyss et al 2013, 2014) demonstrated that a correction term to various deterministic numerical schemes, known in stochastic analysis as the Itô correction, can help improve solution accuracy and ensure convergence to the physically relevant solution without substantial computational overhead. The usual formulation of the Itô correction is valid only when the stochasticity is represented by {\it white} noise. In this study, a generalized formulation of the Itô correction is derived for noises of any color. The formulation is applied to a test problem described by an advection-diffusion equation forced with a spectrum of fast processes. We present numerical results for cases with both constant and spatially varying advection velocities to show that, for the same time step sizes, the introduction of the generalized Itô correction helps to substantially reduce time integration error and significantly improve the convergence rate of the numerical solutions when the forcing term in the governing equation is rough (fast varying); alternatively, for the same target accuracy, the generalized Itô correction allows for the use of significantly longer time steps and hence helps to reduce the computational cost of the numerical simulation. △ Less

Submitted 17 October, 2019; v1 submitted 17 April, 2019; originally announced April 2019.

Comments: 18 pages, 2 figures; v2 includes section rearrangement and added details for the numerical implementation; v3 includes addition of sections, references and one figure

Report number: PNNL-SA-142475 MSC Class: 65C30; 86A10

A comparative study of physics-informed neural network models for learning unknown dynamics and constitutive relations

Authors: Ramakrishna Tipireddy, Paris Perdikaris, Panos Stinis, Alexandre Tartakovsky

Abstract: We investigate the use of discrete and continuous versions of physics-informed neural network methods for learning unknown dynamics or constitutive relations of a dynamical system. For the case of unknown dynamics, we represent all the dynamics with a deep neural network (DNN). When the dynamics of the system are known up to the specification of constitutive relations (that can depend on the state… ▽ More We investigate the use of discrete and continuous versions of physics-informed neural network methods for learning unknown dynamics or constitutive relations of a dynamical system. For the case of unknown dynamics, we represent all the dynamics with a deep neural network (DNN). When the dynamics of the system are known up to the specification of constitutive relations (that can depend on the state of the system), we represent these constitutive relations with a DNN. The discrete versions combine classical multistep discretization methods for dynamical systems with neural network based machine learning methods. On the other hand, the continuous versions utilize deep neural networks to minimize the residual function for the continuous governing equations. We use the case of a fedbatch bioreactor system to study the effectiveness of these approaches and discuss conditions for their applicability. Our results indicate that the accuracy of the trained neural network models is much higher for the cases where we only have to learn a constitutive relation instead of the whole dynamics. This finding corroborates the well-known fact from scientific computing that building as much structural information is available into an algorithm can enhance its efficiency and/or accuracy. △ Less

Submitted 2 April, 2019; originally announced April 2019.

Renormalization and blow-up for the 3D Euler equations

Authors: Jacob Price, Panos Stinis

Abstract: In recent work we have developed a renormalization framework for stabilizing reduced order models for time-dependent partial differential equations. We have applied this framework to the open problem of finite-time singularity formation (blow-up) for the 3D Euler equations of incompressible fluid flow. The renormalized coefficients in the reduced order models decay algebraically with time and reso… ▽ More In recent work we have developed a renormalization framework for stabilizing reduced order models for time-dependent partial differential equations. We have applied this framework to the open problem of finite-time singularity formation (blow-up) for the 3D Euler equations of incompressible fluid flow. The renormalized coefficients in the reduced order models decay algebraically with time and resolution. Our results for the behavior of the solutions are consistent with the formation of a finite-time singularity. △ Less

Submitted 27 July, 2018; v1 submitted 22 May, 2018; originally announced May 2018.

Report number: PNNL IR Number: PNNL-SA-134937 MSC Class: 65M99; 35D30; 35B44

Doing the impossible: Why neural networks can be trained at all

Authors: Nathan O. Hodas, Panos Stinis

Abstract: As deep neural networks grow in size, from thousands to millions to billions of weights, the performance of those networks becomes limited by our ability to accurately train them. A common naive question arises: if we have a system with billions of degrees of freedom, don't we also need billions of samples to train it? Of course, the success of deep learning indicates that reliable models can be l… ▽ More As deep neural networks grow in size, from thousands to millions to billions of weights, the performance of those networks becomes limited by our ability to accurately train them. A common naive question arises: if we have a system with billions of degrees of freedom, don't we also need billions of samples to train it? Of course, the success of deep learning indicates that reliable models can be learned with reasonable amounts of data. Similar questions arise in protein folding, spin glasses and biological neural networks. With effectively infinite potential folding/spin/wiring configurations, how does the system find the precise arrangement that leads to useful and robust results? Simple sampling of the possible configurations until an optimal one is reached is not a viable option even if one waited for the age of the universe. On the contrary, there appears to be a mechanism in the above phenomena that forces them to achieve configurations that live on a low-dimensional manifold, avoiding the curse of dimensionality. In the current work we use the concept of mutual information between successive layers of a deep neural network to elucidate this mechanism and suggest possible ways of exploiting it to accelerate training. We show that adding structure to the neural network that enforces higher mutual information between layers speeds training and leads to more accurate results. High mutual information between layers implies that the effective number of free parameters is exponentially smaller than the raw number of tunable weights. △ Less

Submitted 27 May, 2018; v1 submitted 13 May, 2018; originally announced May 2018.

Comments: The material is based on a poster from the 15th Neural Computation and Psychology Workshop "Contemporary Neural Network Models: Machine Learning, Artificial Intelligence, and Cognition" August 8-9, 2016, Drexel University, Philadelphia, PA, USA

Report number: PNNL-SA-127608

A data-driven framework for sparsity-enhanced surrogates with arbitrary mutually dependent randomness

Authors: Huan Lei, **g Li, Peiyuan Gao, Panos Stinis, Nathan Baker

Abstract: The challenge of quantifying uncertainty propagation in real-world systems is rooted in the high-dimensionality of the stochastic input and the frequent lack of explicit knowledge of its probability distribution. Traditional approaches show limitations for such problems. To address these difficulties, we have developed a general framework of constructing surrogate models on spaces of stochastic in… ▽ More The challenge of quantifying uncertainty propagation in real-world systems is rooted in the high-dimensionality of the stochastic input and the frequent lack of explicit knowledge of its probability distribution. Traditional approaches show limitations for such problems. To address these difficulties, we have developed a general framework of constructing surrogate models on spaces of stochastic input with arbitrary probability measure irrespective of the mutual dependencies between individual components and the analytical form. The present Data-driven Sparsity-enhancing Rotation for Arbitrary Randomness (DSRAR) framework includes a data-driven construction of multivariate polynomial basis for arbitrary mutually dependent probability measure and a sparsity enhancement rotation procedure. This sparsity-enhancing rotation method was initially proposed in our previous work [1] for Gaussian distributions, which may not be feasible for non-Gaussian distributions due to the loss of orthogonality after the rotation. To remedy such difficulties, we developed the new approach to construct orthonormal polynomials for arbitrary mutually dependent (amdP) randomness, ensuring the constructed basis maintains the orthogonality with respect to the density of the rotated random vector, where directly applying the regular polynomial chaos including arbitrary polynomial chaos (aPC) [2] shows limitations due to the assumption of the mutual independence between the components of the random inputs. The developed DSRAR framework leads to accurate recovery of a sparse representation of the target functions. The effectiveness of our method is demonstrated in challenging problems such as PDEs and realistic molecular systems where the underlying density is implicitly represented by a large collection of sample data, as well as systems with explicitly given non-Gaussian probabilistic measures. △ Less

Submitted 17 March, 2019; v1 submitted 20 April, 2018; originally announced April 2018.

Enforcing constraints for interpolation and extrapolation in Generative Adversarial Networks

Authors: Panos Stinis, Tobias Hagge, Alexandre M. Tartakovsky, Enoch Yeung

Abstract: We suggest ways to enforce given constraints in the output of a Generative Adversarial Network (GAN) generator both for interpolation and extrapolation (prediction). For the case of dynamical systems, given a time series, we wish to train GAN generators that can be used to predict trajectories starting from a given initial condition. In this setting, the constraints can be in algebraic and/or diff… ▽ More We suggest ways to enforce given constraints in the output of a Generative Adversarial Network (GAN) generator both for interpolation and extrapolation (prediction). For the case of dynamical systems, given a time series, we wish to train GAN generators that can be used to predict trajectories starting from a given initial condition. In this setting, the constraints can be in algebraic and/or differential form. Even though we are predominantly interested in the case of extrapolation, we will see that the tasks of interpolation and extrapolation are related. However, they need to be treated differently. For the case of interpolation, the incorporation of constraints is built into the training of the GAN. The incorporation of the constraints respects the primary game-theoretic setup of a GAN so it can be combined with existing algorithms. However, it can exacerbate the problem of instability during training that is well-known for GANs. We suggest adding small noise to the constraints as a simple remedy that has performed well in our numerical experiments. The case of extrapolation (prediction) is more involved. During training, the GAN generator learns to interpolate a noisy version of the data and we enforce the constraints. This approach has connections with model reduction that we can utilize to improve the efficiency and accuracy of the training. Depending on the form of the constraints, we may enforce them also during prediction through a projection step. We provide examples of linear and nonlinear systems of differential equations to illustrate the various constructions. △ Less

Submitted 19 June, 2019; v1 submitted 21 March, 2018; originally announced March 2018.

Comments: 29 pages; v2 has major text revision/restructuring, includes results for the Lorenz system and has several more references

Report number: PNNL-SA-133233 MSC Class: 68T05; 65L05; 37M10; 62M45; 68Q32

Mori-Zwanzig reduced models for uncertainty quantification

Authors: **g Li, Panos Stinis

Abstract: In many time-dependent problems of practical interest the parameters and/or initial conditions entering the equations describing the evolution of the various quantities exhibit uncertainty. One way to address the problem of how this uncertainty impacts the solution is to expand the solution using polynomial chaos expansions and obtain a system of differential equations for the evolution of the exp… ▽ More In many time-dependent problems of practical interest the parameters and/or initial conditions entering the equations describing the evolution of the various quantities exhibit uncertainty. One way to address the problem of how this uncertainty impacts the solution is to expand the solution using polynomial chaos expansions and obtain a system of differential equations for the evolution of the expansion coefficients. We present an application of the Mori-Zwanzig (MZ) formalism to the problem of constructing reduced models of such systems of differential equations. In particular, we construct reduced models for a subset of the polynomial chaos expansion coefficients that are needed for a full description of the uncertainty caused by uncertain parameters or initial conditions. Even though the MZ formalism is exact, its straightforward application to the problem of constructing reduced models for estimating uncertainty involves the computation of memory terms whose cost can become prohibitively expensive. For those cases, we present a Markovian reformulation of the MZ formalism which can lead to approximations that can alleviate some of the computational expense while retaining an accuracy advantage over reduced models that discard the memory altogether. Our results support the conclusion that successful reduced models need to include memory effects. △ Less

Submitted 6 March, 2018; originally announced March 2018.

Comments: 29 pages, 13 figures. arXiv admin note: substantial text overlap with arXiv:1212.6360, arXiv:1211.4285

Report number: PNNL-SA-132853 MSC Class: 65C20; 65M99; 41A10

Solving differential equations with unknown constitutive relations as recurrent neural networks

Authors: Tobias Hagge, Panos Stinis, Enoch Yeung, Alexandre M. Tartakovsky

Abstract: We solve a system of ordinary differential equations with an unknown functional form of a sink (reaction rate) term. We assume that the measurements (time series) of state variables are partially available, and we use recurrent neural network to "learn" the reaction rate from this data. This is achieved by including a discretized ordinary differential equations as part of a recurrent neural networ… ▽ More We solve a system of ordinary differential equations with an unknown functional form of a sink (reaction rate) term. We assume that the measurements (time series) of state variables are partially available, and we use recurrent neural network to "learn" the reaction rate from this data. This is achieved by including a discretized ordinary differential equations as part of a recurrent neural network training problem. We extend TensorFlow's recurrent neural network architecture to create a simple but scalable and effective solver for the unknown functions, and apply it to a fedbatch bioreactor simulation problem. Use of techniques from recent deep learning literature enables training of functions with behavior manifesting over thousands of time steps. Our networks are structurally similar to recurrent neural networks, but differences in design and function require modifications to the conventional wisdom about training such networks. △ Less

Submitted 5 October, 2017; originally announced October 2017.

Comments: 19 pages, 8 figures

Stochastic basis adaptation and spatial domain decomposition for PDEs with random coefficients

Authors: Ramakrishna Tipireddy, Panos Stinis, Alexandre Tartakovsky

Abstract: We present a novel uncertainty quantification approach for high-dimensional stochastic partial differential equations that reduces the computational cost of polynomial chaos methods by decomposing the computational domain into non-overlap** subdomains and adapting the stochastic basis in each subdomain so the local solution has a lower dimensional random space representation. The local solutions… ▽ More We present a novel uncertainty quantification approach for high-dimensional stochastic partial differential equations that reduces the computational cost of polynomial chaos methods by decomposing the computational domain into non-overlap** subdomains and adapting the stochastic basis in each subdomain so the local solution has a lower dimensional random space representation. The local solutions are coupled using the Neumann-Neumann algorithm, where we first estimate the interface solution then evaluate the interior solution in each subdomain using the interface solution as a boundary condition. The interior solutions in each subdomain are computed independently of each other, which reduces the operation count from $O(N^α)$ to $O(M^α),$ where $N$ is the total number of degrees of freedom, $M$ is the number of degrees of freedom in each subdomain, and the exponent $α>1$ depends on the uncertainty quantification method used. In addition, the localized nature of solutions makes the proposed approach highly parallelizable. We illustrate the accuracy and efficiency of the approach for linear and nonlinear differential equations with random coefficients. △ Less

Submitted 7 September, 2017; originally announced September 2017.

Renormalized Reduced Order Models with Memory for Long Time Prediction

Authors: Jacob Price, Panos Stinis

Abstract: We examine the challenging problem of constructing reduced models for the long time prediction of systems where there is no timescale separation between the resolved and unresolved variables. In previous work we focused on the case where there was only transfer of activity (e.g. energy, mass) from the resolved to the unresolved variables. Here we investigate the much more difficult case where ther… ▽ More We examine the challenging problem of constructing reduced models for the long time prediction of systems where there is no timescale separation between the resolved and unresolved variables. In previous work we focused on the case where there was only transfer of activity (e.g. energy, mass) from the resolved to the unresolved variables. Here we investigate the much more difficult case where there is two-way transfer of activity between the resolved and unresolved variables. Like in the case of activity drain out of the resolved variables, even if one starts with an exact formalism, like the Mori-Zwanzig (MZ) formalism, the constructed reduced models can become unstable. We show how to remedy this situation by using dynamic information from the full system to renormalize the MZ reduced models. In addition to being stabilized, the renormalized models can be accurate for very long times. We use the Korteweg-de Vries equation to illustrate the approach. The coefficients of the renormalized models exhibit rich structure, including algebraic time dependence and incomplete similarity. △ Less

Submitted 6 July, 2017; originally announced July 2017.

Comments: 19 pages plus appendices, four figures, software used to reach results available upon request, approved for release by PNNL (IR number PNNL-SA-127388)

Dynamic Loo** of a Free-Draining Polymer

Authors: Felix X. -F. Ye, Panos Stinis, Hong Qian

Abstract: We revisit the celebrated Wilemski-Fixman (WF) treatment for the loo** time of a free-draining polymer. The WF theory introduces a sink term into the Fokker-Planck equation for the $3(N+1)$-dimensional Ornstein-Uhlenbeck process of the polymer dynamics, which accounts for the appropriate boundary condition due to the formation of a loop. The assumption for WF theory is considerably relaxed. A pe… ▽ More We revisit the celebrated Wilemski-Fixman (WF) treatment for the loo** time of a free-draining polymer. The WF theory introduces a sink term into the Fokker-Planck equation for the $3(N+1)$-dimensional Ornstein-Uhlenbeck process of the polymer dynamics, which accounts for the appropriate boundary condition due to the formation of a loop. The assumption for WF theory is considerably relaxed. A perturbation method approach is developed that justifies and generalizes the previous results using either a Delta sink or a Heaviside sink. For both types of sinks, we show that under the condition of a small dimensionless $ε$, the ratio of capture radius to the Kuhn length, we are able to systematically produce all known analytical and asymptotic results obtained by other methods. This includes most notably the transition regime between the $N^2$ scaling of Doi, and $N\sqrt{N}/ε$ scaling of Szabo, Schulten, and Schulten. The mathematical issue at play is the non-uniform convergence of $ε\to 0$ and $N\to\infty$, the latter being an inherent part of the theory of a Gaussian polymer. Our analysis yields a novel term in the analytical expression for the loo** time with small $ε$, which is previously unknown. Monte Carlo numerical simulations corroborate the analytical findings. The systematic method developed here can be applied to other systems modeled by multi-dimensional Smoluchowski equations. △ Less

Submitted 26 October, 2017; v1 submitted 14 April, 2017; originally announced April 2017.

Comments: 20 pages, 4 figures

Basis adaptation and domain decomposition for steady partial differential equations with random coefficients

Authors: Ramkrishna Tipireddy, Panos Stinis, Alexandre Tartakovsky

Abstract: We present a novel approach for solving steady-state stochastic partial differential equations (PDEs) with high-dimensional random parameter space. The proposed approach combines spatial domain decomposition with basis adaptation for each subdomain. The basis adaptation is used to address the curse of dimensionality by constructing an accurate low-dimensional representation of the stochastic PDE s… ▽ More We present a novel approach for solving steady-state stochastic partial differential equations (PDEs) with high-dimensional random parameter space. The proposed approach combines spatial domain decomposition with basis adaptation for each subdomain. The basis adaptation is used to address the curse of dimensionality by constructing an accurate low-dimensional representation of the stochastic PDE solution (probability density function and/or its leading statistical moments) in each subdomain. Restricting the basis adaptation to a specific subdomain affords finding a locally accurate solution. Then, the solutions from all of the subdomains are stitched together to provide a global solution. We support our construction with numerical experiments for a steady-state diffusion equation with a random spatially dependent coefficient. Our results show that highly accurate global solutions can be obtained with significantly reduced computational costs. △ Less

Submitted 27 July, 2016; originally announced July 2016.

Comments: 26 pages, 13 figures

MSC Class: 65C20; 65N55; 41A10

Efficient failure probability calculation through mesh refinement

Authors: **g Li, Panos Stinis

Abstract: We present a novel way of accelerating hybrid surrogate methods for the calculation of failure probabilities. The main idea is to use mesh refinement in order to obtain improved local surrogates of low computation cost to simulate on. These improved surrogates can reduce significantly the required number of evaluations of the exact model (which is the usual bottleneck of failure probability calcul… ▽ More We present a novel way of accelerating hybrid surrogate methods for the calculation of failure probabilities. The main idea is to use mesh refinement in order to obtain improved local surrogates of low computation cost to simulate on. These improved surrogates can reduce significantly the required number of evaluations of the exact model (which is the usual bottleneck of failure probability calculations). Meanwhile the effort on evaluations of surrogates is dramatically reduced by utilizing low order local surrogates. Numerical results of the application of the proposed approach in several examples of increasing complexity show the robustness, versatility and gain in efficiency of the method. △ Less

Submitted 22 September, 2015; originally announced September 2015.

Comments: 22 pages

MSC Class: 65C20; 65M99; 41A10

A unified framework for mesh refinement in random and physical space

Authors: **g Li, Panos Stinis

Abstract: In recent work we have shown how an accurate reduced model can be utilized to perform mesh refinement in random space. That work relied on the explicit knowledge of an accurate reduced model which is used to monitor the transfer of activity from the large to the small scales of the solution. Since this is not always available, we present in the current work a framework which shares the merits and… ▽ More In recent work we have shown how an accurate reduced model can be utilized to perform mesh refinement in random space. That work relied on the explicit knowledge of an accurate reduced model which is used to monitor the transfer of activity from the large to the small scales of the solution. Since this is not always available, we present in the current work a framework which shares the merits and basic idea of the previous approach but does not require an explicit knowledge of a reduced model. Moreover, the current framework can be applied for refinement in both random and physical space. In this manuscript we focus on the application to random space mesh refinement. We study examples of increasing difficulty (from ordinary to partial differential equations) which demonstrate the efficiency and versatility of our approach. We also provide some results from the application of the new framework to physical space mesh refinement. △ Less

Submitted 22 September, 2015; originally announced September 2015.

Comments: 29 pages

MSC Class: 65C20; 65M99; 41A10

Model reduction and mesh refinement

Authors: Panagiotis Stinis

Abstract: In recent work we have presented a novel algorithm for mesh refinement which utilizes a reduced model. In particular, the reduced model is used to monitor the transfer of activity (e.g. mass, energy) from larger to smaller scales. When the activity transfer rate exceeds a prescribed tolerance, the algorithm refines the mesh accordingly. The algorithm was applied to the inviscid Burgers and focusin… ▽ More In recent work we have presented a novel algorithm for mesh refinement which utilizes a reduced model. In particular, the reduced model is used to monitor the transfer of activity (e.g. mass, energy) from larger to smaller scales. When the activity transfer rate exceeds a prescribed tolerance, the algorithm refines the mesh accordingly. The algorithm was applied to the inviscid Burgers and focusing Schrödinger equations to detect singularities. We offer here a simple proof of why the use of a reduced model for mesh refinement is a good idea. We do so by showing that by controlling the transfer of activity rate one controls the error caused inevitably by a finite resolution. △ Less

Submitted 25 February, 2014; originally announced February 2014.

Comments: 14 pages

MSC Class: 65M50; 35L67; 76B99; 65M70

Mesh refinement for uncertainty quantification through model reduction

Authors: **g Li, Panagiotis Stinis

Abstract: We present a novel way of deciding when and where to refine a mesh in probability space in order to facilitate the uncertainty quantification in the presence of discontinuities in random space. A discontinuity in random space makes the application of generalized polynomial chaos expansion techniques prohibitively expensive. The reason is that for discontinuous problems, the expansion converges ver… ▽ More We present a novel way of deciding when and where to refine a mesh in probability space in order to facilitate the uncertainty quantification in the presence of discontinuities in random space. A discontinuity in random space makes the application of generalized polynomial chaos expansion techniques prohibitively expensive. The reason is that for discontinuous problems, the expansion converges very slowly. An alternative to using higher terms in the expansion is to divide the random space in smaller elements where a lower degree polynomial is adequate to describe the randomness. In general, the partition of the random space is a dynamic process since some areas of the random space, particularly around the discontinuity, need more refinement than others as time evolves. In the current work we propose a way to decide when and where to refine the random space mesh based on the use of a reduced model. The idea is that a good reduced model can monitor accurately, within a random space element, the cascade of activity to higher degree terms in the chaos expansion. In terms, this facilitates the efficient allocation of computational sources to the areas of random space where they are more needed. For the Kraichnan-Orszag system, the prototypical system to study discontinuities in random space, we present theoretical results which show why the proposed method is sound and numerical results which corroborate the theory. △ Less

Submitted 19 February, 2014; originally announced February 2014.

Comments: 26 pages

MSC Class: 65C20; 65M99; 41A10

Mori-Zwanzig reduced models for uncertainty quantification II: Initial condition uncertainty

Authors: Panagiotis Stinis

Abstract: In a recent preprint (arXiv:1211.4285v1) we addressed the problem of constructing reduced models for time-dependent systems described by differential equations which involve uncertain parameters. In the current work, we focus on the construction of reduced models for systems of differential equations when the initial condition is uncertain. While for both cases the reduced models are constructed t… ▽ More In a recent preprint (arXiv:1211.4285v1) we addressed the problem of constructing reduced models for time-dependent systems described by differential equations which involve uncertain parameters. In the current work, we focus on the construction of reduced models for systems of differential equations when the initial condition is uncertain. While for both cases the reduced models are constructed through the Mori-Zwanzig formalism, the necessary estimation of the memory parameters is quite different. For the case of uncertain initial conditions we present an algorithm which allows to estimate on the fly the parameters appearing in the reduced model. The first part of the algorithm evolves the full system until the estimation of the parameters for the reduced model has converged. At the time instant that this happens, the algorithm switches to the evolution of only the reduced model with the estimated parameter values from the first part of the algorithm. The viscous Burgers equation with uncertain initial condition is used to illustrate the construction. △ Less

Submitted 27 December, 2012; originally announced December 2012.

Comments: 20 pages, 4 figures

MSC Class: 65C20; 65M99; 41A10

Mori-Zwanzig reduced models for uncertainty quantification I: Parametric uncertainty

Authors: Panagiotis Stinis

Abstract: In many time-dependent problems of practical interest the parameters entering the equations describing the evolution of the various quantities exhibit uncertainty. One way to address the problem of how this uncertainty impacts the solution is to expand the solution using polynomial chaos expansions and obtain a system of differential equations for the evolution of the expansion coefficients. We pr… ▽ More In many time-dependent problems of practical interest the parameters entering the equations describing the evolution of the various quantities exhibit uncertainty. One way to address the problem of how this uncertainty impacts the solution is to expand the solution using polynomial chaos expansions and obtain a system of differential equations for the evolution of the expansion coefficients. We present an application of the Mori-Zwanzig formalism to the problem of constructing reduced models of such systems of differential equations. In particular, we construct reduced models for a subset of the polynomial chaos expansion coefficients that are needed for a full description of the uncertainty caused by the uncertain parameters. We also present a Markovian reformulation of the Mori-Zwanzig reduced equation which replaces the solution of the orthogonal dynamics equation with an infinite hierarchy of ordinary differential equations. The viscous Burgers equation with uncertain viscosity parameter is used to illustrate the construction. For this example we provide a way to estimate the necessary parameters that appear in the reduced model without having to solve the full system. △ Less

Submitted 18 November, 2012; originally announced November 2012.

Comments: 21 pages, 2 figures

MSC Class: 65C20; 65M99; 41A10

Renormalized reduced models for singular PDEs

Authors: Panagiotis Stinis

Abstract: We present a novel way of constructing reduced models for systems of ordinary differential equations. The reduced models we construct depend on coefficients which measure the importance of the different terms appearing in the model and need to be estimated. The proposed approach allows the estimation of these coefficients on the fly by enforcing the equality of integral quantities of the solution… ▽ More We present a novel way of constructing reduced models for systems of ordinary differential equations. The reduced models we construct depend on coefficients which measure the importance of the different terms appearing in the model and need to be estimated. The proposed approach allows the estimation of these coefficients on the fly by enforcing the equality of integral quantities of the solution as computed from the original system and the reduced model. In particular, the approach combines the concepts of renormalization and effective field theory developed in the context of high energy physics and the Mori-Zwanzig formalism of irreversible statistical mechanics. It allows to construct stable reduced models of higher order than was previously possible. The method is applied to the problem of computing reduced models for ordinary differential equation systems resulting from Fourier expansions of singular (or near-singular) time-dependent partial differential equations. Results for the 1D Burgers and the 3D incompressible Euler equations are used to illustrate the construction. We also present, for the 1D Burgers and the 3D Euler equations, a simple and efficient recursive algorithm for calculating the higher order terms. △ Less

Submitted 17 November, 2011; v1 submitted 8 June, 2011; originally announced June 2011.

Comments: 30 pages, 5 color figures

MSC Class: 65M99; 35D30; 35B44

Journal ref: Commun. Appl. Math. Comput. Sci. 8 (2013) 39-66

Numerical computation of solutions of the critical nonlinear Schrodinger equation after the singularity

Authors: Panagiotis Stinis

Abstract: We present numerical results for the solution of the 1D critical nonlinear Schrodinger with periodic boundary conditions and initial data that give rise to a finite time singularity. We construct, through the Mori-Zwanzig formalism, a reduced model which allows us to follow the solution after the formation of the singularity. The computed post-singularity solution exhibits the same characteristics… ▽ More We present numerical results for the solution of the 1D critical nonlinear Schrodinger with periodic boundary conditions and initial data that give rise to a finite time singularity. We construct, through the Mori-Zwanzig formalism, a reduced model which allows us to follow the solution after the formation of the singularity. The computed post-singularity solution exhibits the same characteristics as the post-singularity solutions constructed recently by Terence Tao. △ Less

Submitted 11 October, 2010; originally announced October 2010.

Comments: 17 pages, 8 color figures

MSC Class: 65M99; 35D30; 35B44