Showing 1–50 of 70 results for author: Sandu, A

Preserving Nonlinear Constraints in Variational Flow Filtering Data Assimilation

Authors: Amit N. Subrahmanya, Andrey A. Popov, Reid J. Gomillion, Adrian Sandu

Abstract: Data assimilation aims to estimate the states of a dynamical system by optimally combining sparse and noisy observations of the physical system with uncertain forecasts produced by a computational model. The states of many dynamical systems of interest obey nonlinear physical constraints, and the corresponding dynamics is confined to a certain sub-manifold of the state space. Standard data assimil… ▽ More Data assimilation aims to estimate the states of a dynamical system by optimally combining sparse and noisy observations of the physical system with uncertain forecasts produced by a computational model. The states of many dynamical systems of interest obey nonlinear physical constraints, and the corresponding dynamics is confined to a certain sub-manifold of the state space. Standard data assimilation techniques applied to such systems yield posterior states lying outside the manifold, violating the physical constraints. This work focuses on particle flow filters which use stochastic differential equations to evolve state samples from a prior distribution to samples from an observation-informed posterior distribution. The variational Fokker-Planck (VFP) -- a generic particle flow filtering framework -- is extended to incorporate non-linear, equality state constraints in the analysis. To this end, two algorithmic approaches that modify the VFP stochastic differential equation are discussed: (i) VFPSTAB, to inexactly preserve constraints with the addition of a stabilizing drift term, and (ii) VFPDAE, to exactly preserve constraints by treating the VFP dynamics as a stochastic differential-algebraic equation (SDAE). Additionally, an implicit-explicit time integrator is developed to evolve the VFPDAE dynamics. The strength of the proposed approach for constraint preservation in data assimilation is demonstrated on three test problems: the double pendulum, Korteweg-de-Vries, and the incompressible Navier-Stokes equations. △ Less

Submitted 7 May, 2024; originally announced May 2024.

Report number: CSL-TR-24-1 MSC Class: 65C05; 62F15; 62F30; 35R30

Improving the Adaptive Moment Estimation (ADAM) stochastic optimizer through an Implicit-Explicit (IMEX) time-step** approach

Authors: Abhinab Bhattacharjee, Andrey A. Popov, Arash Sarshar, Adrian Sandu

Abstract: The Adam optimizer, often used in Machine Learning for neural network training, corresponds to an underlying ordinary differential equation (ODE) in the limit of very small learning rates. This work shows that the classical Adam algorithm is a first order implicit-explicit (IMEX) Euler discretization of the underlying ODE. Employing the time discretization point of view, we propose new extensions… ▽ More The Adam optimizer, often used in Machine Learning for neural network training, corresponds to an underlying ordinary differential equation (ODE) in the limit of very small learning rates. This work shows that the classical Adam algorithm is a first order implicit-explicit (IMEX) Euler discretization of the underlying ODE. Employing the time discretization point of view, we propose new extensions of the Adam scheme obtained by using higher order IMEX methods to solve the ODE. Based on this approach, we derive a new optimization algorithm for neural network training that performs better than classical Adam on several regression and classification problems. △ Less

Submitted 20 March, 2024; originally announced March 2024.

Report number: CSL-TR-2024-2

Simultaneous Optimal System and Controller Design for Multibody Systems with Joint Friction using Direct Sensitivities

Authors: Adwait Verulkar, Corina Sandu, Adrian Sandu, Daniel Dopico

Abstract: Real-world multibody systems are often subject to phenomena like friction, joint clearances, and external events. These phenomena can significantly impact the optimal design of the system and its controller. This work addresses the gradient-based optimization methodology for multibody dynamic systems with joint friction using a direct sensitivity approach for gradient computation. After a thorough… ▽ More Real-world multibody systems are often subject to phenomena like friction, joint clearances, and external events. These phenomena can significantly impact the optimal design of the system and its controller. This work addresses the gradient-based optimization methodology for multibody dynamic systems with joint friction using a direct sensitivity approach for gradient computation. After a thorough review of various friction models developed over the years, the Brown McPhee model has been found to be the most suitable for the study due to its accuracy for dynamic simulation and its compatibility with sensitivity analysis. The methodology supports co-design of the system and its controller, which is especially relevant for applications like robotics and servo-mechanical systems where the actuation and the design are highly dependent on each other. Numerical results are obtained using a new implementation of the MBSVT (Multi-Body Systems at Virginia Tech) software package; MBSVT 2.0 is reprogrammed in Julia for ease of implementation while maintaining high computational efficiency. Three case studies are provided to demonstrate the attractive properties of simultaneous optimal design and control approach for certain applications. △ Less

Submitted 25 December, 2023; originally announced December 2023.

Symplectic multirate generalized additive Runge-Kutta methods for Hamiltonian systems

Authors: Kevin Schäfers, Michael Günther, Adrian Sandu

Abstract: The generalized additive Runge-Kutta (GARK) framework provides a powerful approach for solving additively partitioned ordinary differential equations. This work combines the ideas of symplectic GARK schemes and multirate GARK schemes to efficiently solve additively partitioned Hamiltonian systems with multiple time scales. Order conditions, as well as conditions for symplecticity and time-reversib… ▽ More The generalized additive Runge-Kutta (GARK) framework provides a powerful approach for solving additively partitioned ordinary differential equations. This work combines the ideas of symplectic GARK schemes and multirate GARK schemes to efficiently solve additively partitioned Hamiltonian systems with multiple time scales. Order conditions, as well as conditions for symplecticity and time-reversibility, are derived in the general setting of non-separable Hamiltonian systems. Investigations of the special case of separable Hamiltonian systems are also carried out. We show that particular partitions may introduce stability issues, and discuss partitions that enable an implicit-explicit integration leading to improved stability properties. Higher-order symplectic multirate GARK schemes based on advanced composition techniques are discussed. The performance of the schemes is demonstrated by means of the Fermi-Pasta-Ulam problem. △ Less

Submitted 14 December, 2023; v1 submitted 7 June, 2023; originally announced June 2023.

Comments: 29 pages, 6 figures

MSC Class: 65L05; 65L06; 65L07; 65L20; 65P10 ACM Class: G.1.7

A Two-Level Galerkin Reduced Order Model for the Steady Navier-Stokes Equations

Authors: Dylan Park, Changhong Mou, Honghu Liu, Adrian Sandu, Traian Iliescu

Abstract: We propose, analyze, and investigate numerically a novel two-level Galerkin reduced order model (2L-ROM) for the efficient and accurate numerical simulation of the steady Navier-Stokes equations. In the first step of the 2L-ROM, a relatively low-dimensional nonlinear system is solved. In the second step, the Navier-Stokes equations are linearized around the solution found in the first step, and a… ▽ More We propose, analyze, and investigate numerically a novel two-level Galerkin reduced order model (2L-ROM) for the efficient and accurate numerical simulation of the steady Navier-Stokes equations. In the first step of the 2L-ROM, a relatively low-dimensional nonlinear system is solved. In the second step, the Navier-Stokes equations are linearized around the solution found in the first step, and a higher-dimensional system for the linearized problem is solved. We prove an error bound for the new 2L-ROM and compare it to the standard one level ROM (1L-ROM) in the numerical simulation of the steady Burgers equation. The 2L-ROM significantly decreases (by a factor of $2$ and even $3$) the 1L-ROM computational cost, without compromising its numerical accuracy. △ Less

Submitted 23 November, 2022; originally announced November 2022.

The Model Forest Ensemble Kalman Filter

Authors: Andrey A Popov, Adrian Sandu

Abstract: Traditional data assimilation uses information obtained from the propagation of one physics-driven model and combines it with information derived from real-world observations in order to obtain a better estimate of the truth of some natural process. However, in many situations multiple simulation models that describe the same physical phenomenon are available. Such models can have different source… ▽ More Traditional data assimilation uses information obtained from the propagation of one physics-driven model and combines it with information derived from real-world observations in order to obtain a better estimate of the truth of some natural process. However, in many situations multiple simulation models that describe the same physical phenomenon are available. Such models can have different sources. On one hand there are theory-guided models are constructed from first physical principles, while on the other there are data-driven models that are constructed from snapshots of high fidelity information. In this work we provide a possible way to make use of this collection of models in data assimilation by generalizing the idea of model hierarchies into model forests -- collections of high fidelity and low fidelity models organized in a gro** of model trees such as to capture various relationships between different models. We generalize the multifidelity ensemble Kalman filter that previously operated on model hierarchies into the model forest ensemble Kalman filter through a generalized theory of linear control variates. This new filter allows for much more freedom when treading the line between accuracy and speed. Numerical experiments with a high fidelity quasi-geostrophic model and two of its low fidelity reduced order models validate the accuracy of our approach. △ Less

Submitted 21 October, 2022; originally announced October 2022.

MSC Class: 62F15; 62M20; 65C05; 65M60; 76F70; 86A22; 93E11

Physics-informed neural networks for PDE-constrained optimization and control

Authors: Jostein Barry-Straume, Arash Sarshar, Andrey A. Popov, Adrian Sandu

Abstract: A fundamental problem in science and engineering is designing optimal control policies that steer a given system towards a desired outcome. This work proposes Control Physics-Informed Neural Networks (Control PINNs) that simultaneously solve for a given system state, and for the optimal control signal, in a one-stage framework that conforms to the underlying physical laws. Prior approaches use a t… ▽ More A fundamental problem in science and engineering is designing optimal control policies that steer a given system towards a desired outcome. This work proposes Control Physics-Informed Neural Networks (Control PINNs) that simultaneously solve for a given system state, and for the optimal control signal, in a one-stage framework that conforms to the underlying physical laws. Prior approaches use a two-stage framework that first models and then controls a system in sequential order. In contrast, a Control PINN incorporates the required optimality conditions in its architecture and in its loss function. The success of Control PINNs is demonstrated by solving the following open-loop optimal control problems: (i) an analytical problem, (ii) a one-dimensional heat equation, and (iii) a two-dimensional predator-prey problem. △ Less

Submitted 18 August, 2022; v1 submitted 6 May, 2022; originally announced May 2022.

Report number: CSL-TR-22-2 ACM Class: I.2.6; I.2.8; I.5.1; G.1.6; G.1.8

Eliminating Order Reduction on Linear, Time-Dependent ODEs with GARK Methods

Authors: Steven Roberts, Adrian Sandu

Abstract: When applied to stiff, linear differential equations with time-dependent forcing, Runge-Kutta methods can exhibit convergence rates lower than predicted by the classical order condition theory. Commonly, this order reduction phenomenon is addressed by using an expensive, fully implicit Runge-Kutta method with high stage order or a specialized scheme satisfying additional order conditions. This wor… ▽ More When applied to stiff, linear differential equations with time-dependent forcing, Runge-Kutta methods can exhibit convergence rates lower than predicted by the classical order condition theory. Commonly, this order reduction phenomenon is addressed by using an expensive, fully implicit Runge-Kutta method with high stage order or a specialized scheme satisfying additional order conditions. This work develops a flexible approach of augmenting an arbitrary Runge-Kutta method with a fully implicit method used to treat the forcing such as to maintain the classical order of the base scheme. Our methods and analyses are based on the general-structure additive Runge-Kutta framework. Numerical experiments using diagonally implicit, fully implicit, and even explicit Runge-Kutta methods confirm that the new approach eliminates order reduction for the class of problems under consideration, and the base methods achieve their theoretical orders of convergence. △ Less

Submitted 13 February, 2022; v1 submitted 19 January, 2022; originally announced January 2022.

MSC Class: 65L04; 65L20

Ensemble Variational Fokker-Planck Methods for Data Assimilation

Authors: Amit N Subrahmanya, Andrey A Popov, Adrian Sandu

Abstract: Particle flow filters solve Bayesian inference problems by smoothly transforming a set of particles into samples from the posterior distribution. Particles move in state space under the flow of an McKean-Vlasov-Ito process. This work introduces the Variational Fokker-Planck (VFP) framework for data assimilation, a general approach that includes previously known particle flow filters as special cas… ▽ More Particle flow filters solve Bayesian inference problems by smoothly transforming a set of particles into samples from the posterior distribution. Particles move in state space under the flow of an McKean-Vlasov-Ito process. This work introduces the Variational Fokker-Planck (VFP) framework for data assimilation, a general approach that includes previously known particle flow filters as special cases. The McKean-Vlasov-Ito process that transforms particles is defined via an optimal drift that depends on the selected diffusion term. It is established that the underlying probability density - sampled by the ensemble of particles - converges to the Bayesian posterior probability density. For a finite number of particles the optimal drift contains a regularization term that nudges particles toward becoming independent random variables. Based on this analysis, we derive computationally-feasible approximate regularization approaches that penalize the mutual information between pairs of particles, and avoid particle collapse. Moreover, the diffusion plays a role akin to a particle rejuvenation approach that aims to alleviate particle collapse. The VFP framework is very flexible. Different assumptions on prior and intermediate probability distributions can be used to implement the optimal drift, and localization and covariance shrinkage can be applied to alleviate the curse of dimensionality. A robust implicit-explicit method is discussed for the efficient integration of stiff McKean-Vlasov-Ito processes. The effectiveness of the VFP framework is demonstrated on three progressively more challenging test problems, namely the Lorenz '63, Lorenz '96 and the quasi-geostrophic equations. △ Less

Submitted 19 January, 2024; v1 submitted 27 November, 2021; originally announced November 2021.

Report number: CSL-TR-21-10 MSC Class: 65C05; 93E11; 62F15; 86A22

Adjoint-Matching Neural Network Surrogates for Fast 4D-Var Data Assimilation

Authors: Austin Chennault, Andrey A. Popov, Amit N. Subrahmanya, Rachel Cooper, Ali Haisam Muhammad Rafid, Anuj Karpatne, Adrian Sandu

Abstract: Data assimilation is the process of fusing information from imperfect computer simulations with noisy, sparse measurements of reality to obtain improved estimates of the state or parameters of a dynamical system of interest. The data assimilation procedures used in many geoscience applications, such as numerical weather forecasting, are variants of the our-dimensional variational (4D-Var) algorith… ▽ More Data assimilation is the process of fusing information from imperfect computer simulations with noisy, sparse measurements of reality to obtain improved estimates of the state or parameters of a dynamical system of interest. The data assimilation procedures used in many geoscience applications, such as numerical weather forecasting, are variants of the our-dimensional variational (4D-Var) algorithm. The cost of solving the underlying 4D-Var optimization problem is dominated by the cost of repeated forward and adjoint model runs. This motivates substituting the evaluations of the physical model and its adjoint by fast, approximate surrogate models. Neural networks offer a promising approach for the data-driven creation of surrogate models. The accuracy of the surrogate 4D-Var solution depends on the accuracy with each the surrogate captures both the forward and the adjoint model dynamics. We formulate and analyze several approaches to incorporate adjoint information into the construction of neural network surrogates. The resulting networks are tested on unseen data and in a sequential data assimilation problem using the Lorenz-63 system. Surrogates constructed using adjoint information demonstrate superior performance on the 4D-Var data assimilation problem compared to a standard neural network surrogate that uses only forward dynamics information. △ Less

Submitted 20 December, 2022; v1 submitted 16 November, 2021; originally announced November 2021.

Report number: CSL-TR-21-7 MSC Class: 34A55; 68T07; 90C30; 65L09

A Stochastic Covariance Shrinkage Approach to Particle Rejuvenation in the Ensemble Transform Particle Filter

Authors: Andrey A Popov, Amit N Subrahmanya, Adrian Sandu

Abstract: Rejuvenation in particle filters is necessary to prevent the collapse of the weights when the number of particles is insufficient to sample the high probability regions of the state space. Rejuvenation is often implemented in a heuristic manner by the addition of stochastic samples that widen the support of the ensemble. This work aims at improving canonical rejuvenation methodology by the introdu… ▽ More Rejuvenation in particle filters is necessary to prevent the collapse of the weights when the number of particles is insufficient to sample the high probability regions of the state space. Rejuvenation is often implemented in a heuristic manner by the addition of stochastic samples that widen the support of the ensemble. This work aims at improving canonical rejuvenation methodology by the introduction of additional prior information obtained from climatological samples; the dynamical particles used for importance sampling are augmented with samples obtained from stochastic covariance shrinkage. The ensemble transport particle filter, and its second order variant, are extended with the proposed rejuvenation approach. Numerical experiments show that modified filters significantly improve the analyses for low dynamical ensemble sizes. △ Less

Submitted 20 September, 2021; originally announced September 2021.

Report number: CSL-TR-20-4

Symplectic GARK methods for partitioned Hamiltonian systems

Authors: Michael Günther, Adrian Sandu, Kevin Schäfers, Antonella Zanna

Abstract: Generalized Additive Runge-Kutta schemes have shown to be a suitable tool for solving ordinary differential equations with additively partitioned right-hand sides. This work develops symplectic GARK schemes for additively partitioned Hamiltonian systems. In a general setting, we derive conditions for symplecticness, as well as symmetry and time-reversibility. We show how symplectic and symmetric s… ▽ More Generalized Additive Runge-Kutta schemes have shown to be a suitable tool for solving ordinary differential equations with additively partitioned right-hand sides. This work develops symplectic GARK schemes for additively partitioned Hamiltonian systems. In a general setting, we derive conditions for symplecticness, as well as symmetry and time-reversibility. We show how symplectic and symmetric schemes can be constructed based on schemes which are only symplectic, or only symmetric. Special attention is given to the special case of partitioned schemes for Hamiltonians split into multiple potential and kinetic energies. Finally we show how symplectic GARK schemes can leverage different time scales and evaluation costs for different potentials, and provide efficient numerical solutions by using different order for these parts. △ Less

Submitted 13 December, 2023; v1 submitted 6 March, 2021; originally announced March 2021.

MSC Class: 65L05; 65L06; 65L07; 65L020

A unified formulation of splitting-based implicit time integration schemes

Authors: Severiano González-Pinto, Domingo Hernández-Abreu, Maria S. Pérez-Rodríguez, Arash Sarshar, Steven Roberts, Adrian Sandu

Abstract: Splitting-based time integration approaches such as fractional steps, alternating direction implicit, operator splitting, and locally one-dimensional methods partition the system of interest into components and solve individual components implicitly in a cost-effective way. This work proposes a unified formulation of splitting time integration schemes in the framework of general-structure additive… ▽ More Splitting-based time integration approaches such as fractional steps, alternating direction implicit, operator splitting, and locally one-dimensional methods partition the system of interest into components and solve individual components implicitly in a cost-effective way. This work proposes a unified formulation of splitting time integration schemes in the framework of general-structure additive Runge-Kutta (GARK) methods. Specifically, we develop implicit-implicit (IMIM) GARK schemes, provide the order conditions and stability analysis for this class, and explain their application to partitioned systems of ordinary differential equations. We show that classical splitting methods belong to the IMIM GARK family, and therefore can be studied in this unified framework. New IMIM-GARK splitting methods are developed and tested using parabolic systems. △ Less

Submitted 20 December, 2021; v1 submitted 1 March, 2021; originally announced March 2021.

Report number: CSL-TR-2021-3 MSC Class: 65L05; 65L07

Multifidelity Ensemble Kalman Filtering Using Surrogate Models Defined by Physics-Informed Autoencoders

Authors: Andrey A Popov, Adrian Sandu

Abstract: Data assimilation is a Bayesian inference process that obtains an enhanced understanding of a physical system of interest by fusing information from an inexact physics-based model, and from noisy sparse observations of reality. The multifidelity ensemble Kalman filter (MFEnKF) recently developed by the authors combines a full-order physical model and a hierarchy of reduced order surrogate models i… ▽ More Data assimilation is a Bayesian inference process that obtains an enhanced understanding of a physical system of interest by fusing information from an inexact physics-based model, and from noisy sparse observations of reality. The multifidelity ensemble Kalman filter (MFEnKF) recently developed by the authors combines a full-order physical model and a hierarchy of reduced order surrogate models in order to increase the computational efficiency of data assimilation. The standard MFEnKF uses linear couplings between models, and is statistically optimal in case of Gaussian probability densities. This work extends MFEnKF to work with non-linear couplings between the models. Optimal nonlinear projection and interpolation operators are obtained by appropriately trained physics-informed autoencoders, and this approach allows to construct reduced order surrogate models with less error than conventional linear methods. Numerical experiments with the canonical Lorenz '96 model illustrate that nonlinear surrogates perform better than linear projection-based ones in the context of multifidelity filtering. △ Less

Submitted 10 March, 2021; v1 submitted 25 February, 2021; originally announced February 2021.

Report number: CSL-TR-21-1

Multirate Linearly-Implicit GARK Schemes

Authors: Michael Guenther, Adrian Sandu

Abstract: Many complex applications require the solution of initial-value problems where some components change fast, while others vary slowly. Multirate schemes apply different step sizes to resolve different components of the system, according to their dynamics, in order to achieve increased computational efficiency. The stiff components of the system, fast or slow, are best discretized with implicit base… ▽ More Many complex applications require the solution of initial-value problems where some components change fast, while others vary slowly. Multirate schemes apply different step sizes to resolve different components of the system, according to their dynamics, in order to achieve increased computational efficiency. The stiff components of the system, fast or slow, are best discretized with implicit base methods in order to ensure numerical stability. To this end, linearly implicit methods are particularly attractive as they solve only linear systems of equations at each step. This paper develops the Multirate GARK-ROS/ROW (MR-GARK-ROS/ROW) framework for linearly-implicit multirate time integration. The order conditions theory considers both exact and approximative Jacobians. The effectiveness of implicit multirate methods depends on the coupling between the slow and fast computations; an array of efficient coupling strategies and the resulting numerical schemes are analyzed. Multirate infinitesimal step linearly-implicit methods, that allow arbitrarily small micro-steps and offer extreme computational flexibility, are constructed. The new unifying framework includes existing multirate Rosenbrock(-W) methods as particular cases, and opens the possibility to develop new classes of highly effective linearly implicit multirate integrators. △ Less

Submitted 19 February, 2021; originally announced February 2021.

Report number: CSL-TR-21-02

Linearly Implicit Multistep Methods for Time Integration

Authors: Ross Glandon, Mahesh Narayanamurthi, Adrian Sandu

Abstract: Time integration methods for solving initial value problems are an important component of many scientific and engineering simulations. Implicit time integrators are desirable for their stability properties, significantly relaxing restrictions on timestep size. However, implicit methods require solutions to one or more systems of nonlinear equations at each timestep, which for large simulations can… ▽ More Time integration methods for solving initial value problems are an important component of many scientific and engineering simulations. Implicit time integrators are desirable for their stability properties, significantly relaxing restrictions on timestep size. However, implicit methods require solutions to one or more systems of nonlinear equations at each timestep, which for large simulations can be prohibitively expensive. This paper introduces a new family of linearly implicit multistep methods (LIMM), which only requires the solution of one linear system per timestep. Order conditions and stability theory for these methods are presented, as well as design and implementation considerations. Practical methods of order up to five are developed that have similar error coefficients, but improved stability regions, when compared to the widely used BDF methods. Numerical testing of a self-starting variable stepsize and variable order implementation of the new LIMM methods shows measurable performance improvement over a similar BDF implementation. △ Less

Submitted 20 November, 2020; originally announced November 2020.

Comments: 36 pages, 5 figures, submitted to SISC in May 2020

Report number: CSL-TR-19-13 MSC Class: 65L04; 65L05; 65L06

A fast time-step** strategy for dynamical systems equipped with a surrogate model

Authors: Steven Roberts, Andrey A Popov, Arash Sarshar, Adrian Sandu

Abstract: Simulation of complex dynamical systems arising in many applications is computationally challenging due to their size and complexity. Model order reduction, machine learning, and other types of surrogate modeling techniques offer cheaper and simpler ways to describe the dynamics of these systems but are inexact and introduce additional approximation errors. In order to overcome the computational d… ▽ More Simulation of complex dynamical systems arising in many applications is computationally challenging due to their size and complexity. Model order reduction, machine learning, and other types of surrogate modeling techniques offer cheaper and simpler ways to describe the dynamics of these systems but are inexact and introduce additional approximation errors. In order to overcome the computational difficulties of the full complex models, on one hand, and the limitations of surrogate models, on the other, this work proposes a new accelerated time-step** strategy that combines information from both. This approach is based on the multirate infinitesimal general-structure additive Runge-Kutta (MRI-GARK) framework. The inexpensive surrogate model is integrated with a small timestep to guide the solution trajectory, and the full model is treated with a large timestep to occasionally correct for the surrogate model error and ensure convergence. We provide a theoretical error analysis, and several numerical experiments, to show that this approach can be significantly more efficient than using only the full or only the surrogate model for the integration. △ Less

Submitted 14 February, 2022; v1 submitted 6 November, 2020; originally announced November 2020.

Report number: CSL-TR-20-6 MSC Class: 65L05; 65F99

Linearly implicit GARK schemes

Authors: Adrian Sandu, Michael Günther, Steven Roberts

Abstract: Systems driven by multiple physical processes are central to many areas of science and engineering. Time discretization of multiphysics systems is challenging, since different processes have different levels of stiffness and characteristic time scales. The multimethod approach discretizes each physical process with an appropriate numerical method; the methods are coupled appropriately such that th… ▽ More Systems driven by multiple physical processes are central to many areas of science and engineering. Time discretization of multiphysics systems is challenging, since different processes have different levels of stiffness and characteristic time scales. The multimethod approach discretizes each physical process with an appropriate numerical method; the methods are coupled appropriately such that the overall solution has the desired accuracy and stability properties. The authors developed the general-structure additive Runge-Kutta (GARK) framework, which constructs multimethods based on Runge-Kutta schemes. This paper constructs the new GARK-ROS/GARK-ROW families of multimethods based on linearly implicit Rosenbrock/Rosenbrock-W schemes. For ordinary differential equation models, we develop a general order condition theory for linearly implicit methods with any number of partitions, using exact or approximate Jacobians. We generalize the order condition theory to two-way partitioned index-1 differential-algebraic equations. Applications of the framework include decoupled linearly implicit, linearly implicit/explicit, and linearly implicit/implicit methods. Practical GARK-ROS and GARK-ROW schemes of order up to four are constructed. △ Less

Submitted 11 October, 2020; v1 submitted 4 August, 2020; originally announced August 2020.

Report number: CSL-TR-20-9 MSC Class: 65L05; 65L06; 65L07; 65L20

A Multifidelity Ensemble Kalman Filter with Reduced Order Control Variates

Authors: Andrey A Popov, Changhong Mou, Traian Iliescu, Adrian Sandu

Abstract: This work develops a new multifidelity ensemble Kalman filter (MFEnKF) algorithm based on linear control variate framework. The approach allows for rigorous multifidelity extensions of the EnKF, where the uncertainty in coarser fidelities in the hierarchy of models represent control variates for the uncertainty in finer fidelities. Small ensembles of high fidelity model runs are complemented by la… ▽ More This work develops a new multifidelity ensemble Kalman filter (MFEnKF) algorithm based on linear control variate framework. The approach allows for rigorous multifidelity extensions of the EnKF, where the uncertainty in coarser fidelities in the hierarchy of models represent control variates for the uncertainty in finer fidelities. Small ensembles of high fidelity model runs are complemented by larger ensembles of cheaper, lower fidelity runs, to obtain much improved analyses at only small additional computational costs. We investigate the use of reduced order models as coarse fidelity control variates in the MFEnKF, and provide analyses to quantify the improvements over the traditional ensemble Kalman filters. We apply these ideas to perform data assimilation with a quasi-geostrophic test problem, using direct numerical simulation and a corresponding POD-Galerkin reduced order model. Numerical results show that the two-fidelity MFEnKF provides better analyses than existing EnKF algorithms at comparable or reduced computational costs. △ Less

Submitted 1 July, 2020; originally announced July 2020.

Report number: CSL-TR-20-2 MSC Class: 62F15

Convergence Results for Implicit--Explicit General Linear Methods

Authors: Adrian Sandu

Abstract: This paper studies fixed-step convergence of implicit-explicit general linear methods. We focus on a subclass of schemes that is internally consistent, has high stage order, and favorable stability properties. Classical, index-1 differential algebraic equation, and singular perturbation convergence analyses results are given. For all these problems IMEX GLMs from the class of interest converge wit… ▽ More This paper studies fixed-step convergence of implicit-explicit general linear methods. We focus on a subclass of schemes that is internally consistent, has high stage order, and favorable stability properties. Classical, index-1 differential algebraic equation, and singular perturbation convergence analyses results are given. For all these problems IMEX GLMs from the class of interest converge with the full theoretical orders under general assumptions. The convergence results require the time steps to be sufficiently small, with upper bounds that are independent on the stiffness of the problem. △ Less

Submitted 8 April, 2020; originally announced April 2020.

Report number: CSL-TR-20-7

An Explicit Probabilistic Derivation of Inflation in a Scalar Ensemble Kalman Filter for Finite Step, Finite Ensemble Convergence

Authors: Andrey A Popov, Adrian Sandu

Abstract: This paper uses a probabilistic approach to analyze the converge of an ensemble Kalman filter solution to an exact Kalman filter solution in the simplest possible setting, the scalar case, as it allows us to build upon a rich literature of scalar probability distributions and non-elementary functions. To this end we introduce the bare-bones Scalar Pedagogical Ensemble Kalman Filter (SPEnKF). We sh… ▽ More This paper uses a probabilistic approach to analyze the converge of an ensemble Kalman filter solution to an exact Kalman filter solution in the simplest possible setting, the scalar case, as it allows us to build upon a rich literature of scalar probability distributions and non-elementary functions. To this end we introduce the bare-bones Scalar Pedagogical Ensemble Kalman Filter (SPEnKF). We show that in the asymptotic case of ensemble size, the expected value of both the analysis mean and variance estimate of the SPEnKF converges to that of the true Kalman filter, and that the variances of both tend towards zero, at each time moment. We also show that the ensemble converges in probability in the complementary case, when the ensemble is finite, and time is taken to infinity. Moreover, we show that in the finite-ensemble, finite-time case, variance inflation and mean correction can be leveraged to coerce the SPEnKF converge to its scalar Kalman filter counterpart. We then apply this framework to analyze perturbed observations and explain why perturbed observations ensemble Kalman filters underperform their deterministic counterparts. △ Less

Submitted 29 March, 2020; originally announced March 2020.

Report number: CSL-TR-20-1

A Stochastic Covariance Shrinkage Approach in Ensemble Transform Kalman Filtering

Authors: Andrey A Popov, Adrian Sandu, Elias D. Nino-Ruiz, Geir Evensen

Abstract: The Ensemble Kalman Filters (EnKF) employ a Monte-Carlo approach to represent covariance information, and are affected by sampling errors in operational settings where the number of model realizations is much smaller than the model state dimension. To alleviate the effects of these errors EnKF relies on model-specific heuristics such as covariance localization, which takes advantage of the spatial… ▽ More The Ensemble Kalman Filters (EnKF) employ a Monte-Carlo approach to represent covariance information, and are affected by sampling errors in operational settings where the number of model realizations is much smaller than the model state dimension. To alleviate the effects of these errors EnKF relies on model-specific heuristics such as covariance localization, which takes advantage of the spatial locality of correlations among the model variables. This work proposes an approach to alleviate sampling errors that utilizes a locally averaged-in-time dynamics of the model, described in terms of a climatological covariance of the dynamical system. We use this covariance as the target matrix in covariance shrinkage methods, and develop a stochastic covariance shrinkage approach where synthetic ensemble members are drawn to enrich both the ensemble subspace and the ensemble transformation. We additionally provide for a way in which this methodology can be localized similar to the state-of-the-art LETKF method, and that for a certain model setup, our methodology significantly outperforms it. △ Less

Submitted 3 June, 2022; v1 submitted 29 February, 2020; originally announced March 2020.

Report number: CSL-TR-20-3

Parallel implicit-explicit general linear methods

Authors: Steven Roberts, Arash Sarshar, Adrian Sandu

Abstract: High-order discretizations of partial differential equations (PDEs) necessitate high-order time integration schemes capable of handling both stiff and nonstiff operators in an efficient manner. Implicit-explicit (IMEX) integration based on general linear methods (GLMs) offers an attractive solution due to their high stage and method order, as well as excellent stability properties. The IMEX charac… ▽ More High-order discretizations of partial differential equations (PDEs) necessitate high-order time integration schemes capable of handling both stiff and nonstiff operators in an efficient manner. Implicit-explicit (IMEX) integration based on general linear methods (GLMs) offers an attractive solution due to their high stage and method order, as well as excellent stability properties. The IMEX characteristic allows stiff terms to be treated implicitly and nonstiff terms to be efficiently integrated explicitly. This work develops two systematic approaches for the development of IMEX GLMs of arbitrary order with stages that can be solved in parallel. The first approach is based on diagonally implicit multistage integration methods (DIMSIMs) of types 3 and 4. The second is a parallel generalization of IMEX Euler and has the interesting feature that the linear stability is independent of the order of accuracy. Numerical experiments confirm the theoretical rates of convergence and reveal that the new schemes are more efficient than serial IMEX GLMs and IMEX Runge-Kutta methods. △ Less

Submitted 21 April, 2020; v1 submitted 3 February, 2020; originally announced February 2020.

Report number: CSL-TR-19-12 MSC Class: 65L05; 65L20; 65L80

Goal-oriented a posteriori estimation of numerical errors in the solution of multiphysics systems

Authors: Mahesh Narayanamurthi, Ulrich Römer, Adrian Sandu

Abstract: This paper develops a general methodology for a posteriori error estimation in time-dependent multiphysics numerical simulations. The methodology builds upon the generalized-structure additive Runge--Kutta (GARK) approach to time integration. GARK provides a unified formulation of multimethods that simulate complex systems by applying different discretization formulas and/or different time steps t… ▽ More This paper develops a general methodology for a posteriori error estimation in time-dependent multiphysics numerical simulations. The methodology builds upon the generalized-structure additive Runge--Kutta (GARK) approach to time integration. GARK provides a unified formulation of multimethods that simulate complex systems by applying different discretization formulas and/or different time steps to individual components of the system. We derive discrete GARK adjoints and analyze their time accuracy. Based on the adjoint method, we establish computable a posteriori identities for the impacts of both temporal and spatial discretization errors on a given goal function. Numerical examples with reaction-diffusion systems illustrate the accuracy of the derived error measures. Local error decompositions are used to illustrate the power of this framework in adaptive refinements of both temporal and spatial meshes. △ Less

Submitted 23 January, 2020; originally announced January 2020.

Comments: 25 pages, 7 figures

Efficient implementation of partitioned stiff exponential Runge-Kutta methods

Authors: Mahesh Narayanamurthi, Adrian Sandu

Abstract: Multiphysics systems are driven by multiple processes acting simultaneously, and their simulation leads to partitioned systems of differential equations. This paper studies the solution of partitioned systems of differential equations using exponential Runge-Kutta methods. We propose specific multiphysics implementations of exponential Runge-Kutta methods satisfying stiff order conditions that wer… ▽ More Multiphysics systems are driven by multiple processes acting simultaneously, and their simulation leads to partitioned systems of differential equations. This paper studies the solution of partitioned systems of differential equations using exponential Runge-Kutta methods. We propose specific multiphysics implementations of exponential Runge-Kutta methods satisfying stiff order conditions that were developed in [Hochbruck et al., SISC, 1998] and [Luan and Osterman, JCAM, 2014]. We reformulate stiffly--accurate exponential Runge--Kutta methods in a way that naturally allows of the structure of multiphysics systems, and discuss their application to both component and additively partitioned systems. The resulting partitioned exponential methods only compute matrix functions of the Jacobians of individual components, rather than the Jacobian of the full, coupled system. We derive modified formulations of particular methods of order two, three and four, and apply them to solve a partitioned reaction-diffusion problem. The proposed methods retain full order for several partitionings of the discretized problem, including by components and by physical processes. △ Less

Submitted 2 December, 2019; originally announced December 2019.

Report number: CSL-TR-19-10 MSC Class: 65L05; 65L04; 65F60; 65M22

Implicit multirate GARK methods

Authors: Steven Roberts, John Loffeld, Arash Sarshar, Carol S. Woodward, Adrian Sandu

Abstract: This work considers multirate generalized-structure additively partitioned Runge-Kutta (MrGARK) methods for solving stiff systems of ordinary differential equations (ODEs) with multiple time scales. These methods treat different partitions of the system with different timesteps for a more targeted and efficient solution compared to monolithic single rate approaches. With implicit methods used acro… ▽ More This work considers multirate generalized-structure additively partitioned Runge-Kutta (MrGARK) methods for solving stiff systems of ordinary differential equations (ODEs) with multiple time scales. These methods treat different partitions of the system with different timesteps for a more targeted and efficient solution compared to monolithic single rate approaches. With implicit methods used across all partitions, methods must find a balance between stability and the cost of solving nonlinear equations for the stages. In order to characterize this important trade-off, we explore multirate coupling strategies, problems for assessing linear stability, and techniques to efficiently implement Newton iterations for stage equations. Unlike much of the existing multirate stability analysis which is limited in scope to particular methods, we present general statements on stability and describe fundamental limitations for certain types of multirate schemes. New implicit multirate methods up to fourth order are derived, and their accuracy and efficiency properties are verified with numerical tests. △ Less

Submitted 30 November, 2020; v1 submitted 30 October, 2019; originally announced October 2019.

Report number: CSL-TR-19-5 MSC Class: 65L06; 65L20

Adaptive Krylov-Type Time Integration Methods

Authors: Paul Tranquilli, Ross Glandon, Adrian Sandu

Abstract: The Rosenbrock-Krylov family of time integration schemes is an extension of Rosenbrock-W methods that employs a specific Krylov based approximation of the linear system solutions arising within each stage of the integrator. This work proposes an extension of Rosenbrock-Krylov methods to address stability questions which arise for methods making use of inexact linear system solution strategies. Two… ▽ More The Rosenbrock-Krylov family of time integration schemes is an extension of Rosenbrock-W methods that employs a specific Krylov based approximation of the linear system solutions arising within each stage of the integrator. This work proposes an extension of Rosenbrock-Krylov methods to address stability questions which arise for methods making use of inexact linear system solution strategies. Two approaches for improving the stability and efficiency of Rosenbrock-Krylov methods are proposed, one through direct control of linear system residuals and the second through a novel extension of the underlying Krylov space to include stage right hand side vectors. Rosenbrock-Krylov methods employing the new approaches show a substantial improvement in computational efficiency relative to prior implementations. △ Less

Submitted 6 October, 2019; originally announced October 2019.

MSC Class: 65L05

Biorthogonal Rosenbrock-Krylov time discretization methods

Authors: Ross Glandon, Paul Tranquilli, Adrian Sandu

Abstract: Many scientific applications require the solution of large initial-value problems, such as those produced by the method of lines after semi-discretization in space of partial differential equations. The computational cost of implicit time discretizations is dominated by the solution of nonlinear systems of equations at each time step. In order to decrease this cost, the recently developed Rosenbro… ▽ More Many scientific applications require the solution of large initial-value problems, such as those produced by the method of lines after semi-discretization in space of partial differential equations. The computational cost of implicit time discretizations is dominated by the solution of nonlinear systems of equations at each time step. In order to decrease this cost, the recently developed Rosenbrock-Krylov (ROK) time integration methods extend the classical linearly-implicit Rosenbrock(-W) methods, and make use of a Krylov subspace approximation to the Jacobian computed via an Arnoldi process. Since the ROK order conditions rely on the construction of a single Krylov space, no restarting of the Arnoldi process is allowed, and the iterations quickly become expensive with increasing subspace dimensions. This work extends the ROK framework to make use of the Lanczos biorthogonalization procedure for constructing Jacobian approximations. The resulting new family of methods is named biorthogonal ROK (BOROK). The Lanczos procedure's short two-term recurrence allows BOROK methods to utilize larger subspaces for the Jacobian approximation, resulting in increased numerical stability of the time integration at a reduced computational cost. Adaptive subspace size selection and basis extension procedures are also developed for the new schemes. Numerical experiments show that for stiff problems, where a large subspace used to approximate the Jacobian is required for stability, the BOROK methods outperform the original ROK methods. △ Less

Submitted 27 August, 2019; originally announced August 2019.

Report number: CSL-TR-19-8 MSC Class: 65L05

Journal ref: Applied Numerical Mathematics, 150 (2020) 233-251

Partitioned Exponential Methods for Coupled Multiphysics Systems

Authors: Mahesh Narayanamurthi, Adrian Sandu

Abstract: Multiphysics problems involving two or more coupled physical phenomena are ubiquitous in science and engineering. This work develops a new partitioned exponential approach for the time integration of multiphysics problems. After a possible semi-discretization in space, the class of problems under consideration is modeled by a system of ordinary differential equations where the right-hand side is a… ▽ More Multiphysics problems involving two or more coupled physical phenomena are ubiquitous in science and engineering. This work develops a new partitioned exponential approach for the time integration of multiphysics problems. After a possible semi-discretization in space, the class of problems under consideration is modeled by a system of ordinary differential equations where the right-hand side is a summation of two component functions, each corresponding to a given set of physical processes. The partitioned-exponential methods proposed herein evolve each component of the system via an exponential integrator, and information between partitions is exchanged via coupling terms. The traditional approach to constructing exponential methods, based on the variation-of-constants formula, is not directly applicable to partitioned systems. Rather, our approach to develo** new partitioned-exponential families is based on a general-structure additive formulation of the schemes. Two method formulations are considered, one based on a linear-nonlinear splitting of the right hand component functions, and another based on approximate Jacobians. The paper develops classical (non-stiff) order conditions theory for partitioned exponential schemes based on particular families of T-trees and B-series theory. Several practical methods of third order are constructed that extend the Rosenbrock-type and EPIRK families of exponential integrators. Several implementation optimizations specific to the application of these methods to reaction-diffusion systems are also discussed. Numerical experiments reveal that the new partitioned-exponential methods can perform better than traditional unpartitioned exponential methods on some problems. △ Less

Submitted 28 August, 2019; v1 submitted 25 August, 2019; originally announced August 2019.

Comments: Fixed a definition and other minor typos. Results remain unchanged

Report number: CSL-TR-19-6 MSC Class: 65L05; 65L04; 65F60; 65M22; 65Y05

Alternating Directions Implicit Integration in a General Linear Method Framework

Authors: Arash Sarshar, Steven Roberts, Adrian Sandu

Abstract: Alternating Directions Implicit (ADI) integration is an operator splitting approach to solve parabolic and elliptic partial differential equations in multiple dimensions based on solving sequentially a set of related one-dimensional equations. Classical ADI methods have order at most two, due to the splitting errors. Moreover, when the time discretization of stiff one-dimensional problems is based… ▽ More Alternating Directions Implicit (ADI) integration is an operator splitting approach to solve parabolic and elliptic partial differential equations in multiple dimensions based on solving sequentially a set of related one-dimensional equations. Classical ADI methods have order at most two, due to the splitting errors. Moreover, when the time discretization of stiff one-dimensional problems is based on Runge-Kutta schemes, additional order reduction may occur. This work proposes a new ADI approach based on the partitioned General Linear Methods framework. This approach allows the construction of high order ADI methods. Due to their high stage order, the proposed methods can alleviate the order reduction phenomenon seen with other schemes. Numerical experiments are shown to provide further insight into the accuracy, stability, and applicability of these new methods. △ Less

Submitted 3 December, 2019; v1 submitted 1 February, 2019; originally announced February 2019.

Report number: CSL-TR-2019-1 MSC Class: 65L05; 65L07

Journal ref: Journal of Computational and Applied Mathematics, 2019, 112619, ISSN 0377-0427

A Goal-Oriented Adaptive Discrete Empirical Interpolation Method

Authors: R. Stefanescu, A. Sandu

Abstract: In this study we propose a-posteriori error estimation results to approximate the precision loss in quantities of interests computed using reduced order models. To generate the surrogate models we employ Proper Orthogonal Decomposition and Discrete Empirical Interpolation Method. First order expansions of the components of the quantity of interest obtained as the product between the components gra… ▽ More In this study we propose a-posteriori error estimation results to approximate the precision loss in quantities of interests computed using reduced order models. To generate the surrogate models we employ Proper Orthogonal Decomposition and Discrete Empirical Interpolation Method. First order expansions of the components of the quantity of interest obtained as the product between the components gradient and model residuals are summed up to generate the error estimation result. Efficient versions are derived for explicit and implicit Euler schemes and require only one reduced forward and adjoint models and high-fidelity model residuals estimation. Then we derive an adaptive DEIM algorithm to enhance the accuracy of these quantities of interests. The adaptive DEIM algorithm uses dual weighted residuals singular vectors in combination with the non-linear term basis. Both the a-posteriori error estimation results and the adaptive DEIM algorithm were assessed using the 1D-Burgers and Shallow Water Equation models and the numerical experiments shows very good agreement with the theoretical results. △ Less

Submitted 10 January, 2019; originally announced January 2019.

Comments: 34 pages, 19 figures

ODE Test Problems: a MATLAB suite of initial value problems

Authors: Steven Roberts, Andrey A. Popov, Adrian Sandu

Abstract: ODE Test Problems (OTP) is an object-oriented MATLAB package offering a broad range of initial value problems which can be used to test numerical methods such as time integration methods and data assimilation (DA) methods. It includes problems that are linear and nonlinear, homogeneous and nonhomogeneous, autonomous and nonautonomous, scalar and high-dimensional, stiff and nonstiff, and chaotic an… ▽ More ODE Test Problems (OTP) is an object-oriented MATLAB package offering a broad range of initial value problems which can be used to test numerical methods such as time integration methods and data assimilation (DA) methods. It includes problems that are linear and nonlinear, homogeneous and nonhomogeneous, autonomous and nonautonomous, scalar and high-dimensional, stiff and nonstiff, and chaotic and nonchaotic. Many are real-world problems from fields such as chemistry, astrophysics, meteorology, and electrical engineering. OTP also supports partitioned ODEs for testing IMEX methods, multirate methods, and other multimethods. Functions for plotting solutions and creating movies are available for all problems, and exact solutions are provided when available. OTP is desgined for ease of use-meaning that working with and modifying problems is simple and intuitive. △ Less

Submitted 13 January, 2019; originally announced January 2019.

Report number: CSL-TR-19-1

Coupled Multirate Infinitesimal GARK Schemes for Stiff Systems with Multiple Time Scales

Authors: Steven Roberts, Arash Sarshar, Adrian Sandu

Abstract: Traditional time discretization methods use a single timestep for the entire system of interest and can perform poorly when the dynamics of the system exhibits a wide range of time scales. Multirate infinitesimal step (MIS) methods (Knoth and Wolke, 1998) offer an elegant and flexible approach to efficiently integrate such systems. The slow components are discretized by a Runge-Kutta method, and t… ▽ More Traditional time discretization methods use a single timestep for the entire system of interest and can perform poorly when the dynamics of the system exhibits a wide range of time scales. Multirate infinitesimal step (MIS) methods (Knoth and Wolke, 1998) offer an elegant and flexible approach to efficiently integrate such systems. The slow components are discretized by a Runge-Kutta method, and the fast components are resolved by solving modified fast differential equations. Sandu (2018) developed the Multirate Infinitesimal General-structure Additive Runge-Kutta (MRI-GARK) family of methods that includes traditional MIS schemes as a subset. The MRI-GARK framework allowed the construction of the first fourth order MIS schemes. This framework also enabled the introduction of implicit methods, which are decoupled in the sense that any implicitness lies entirely within the fast or slow integrations. It was shown by Sandu that the stability of decoupled implicit MRI-GARK methods has limitations when both the fast and slow components are stiff and interact strongly. This work extends the MRI-GARK framework by introducing coupled implicit methods to solve stiff multiscale systems. The coupled approach has the potential to considerably improve the overall stability of the scheme, at the price of requiring implicit stage calculations over the entire system. Two coupling strategies are considered. The first computes coupled Runge-Kutta stages before solving a single differential equation to refine the fast solution. The second alternates between computing coupled Runge-Kutta stages and solving fast differential equations. We derive order conditions and perform the stability analysis for both strategies. The new coupled methods offer improved stability compared to the decoupled MRI-GARK schemes. The theoretical properties of the new methods are validated with numerical experiments. △ Less

Submitted 2 March, 2020; v1 submitted 30 November, 2018; originally announced December 2018.

Comments: arXiv admin note: substantial text overlap with arXiv:1808.02759

Report number: CSL-TR-18-7 MSC Class: 65L05; 65L06

A Bayesian Approach to Multivariate Adaptive Localization in Ensemble-Based Data Assimilation with Time-Dependent Extensions

Authors: Andrey A Popov, Adrian Sandu

Abstract: Ever since its inception, the Ensemble Kalman Filter has elicited many heuristic methods that sought to correct it. One such method is localization---the thought that `nearby' variables should be highly correlated with `far away' variable not. Recognizing that correlation is a time-dependent property, adaptive localization is a natural extension to these heuristics. We propose a Bayesian approach… ▽ More Ever since its inception, the Ensemble Kalman Filter has elicited many heuristic methods that sought to correct it. One such method is localization---the thought that `nearby' variables should be highly correlated with `far away' variable not. Recognizing that correlation is a time-dependent property, adaptive localization is a natural extension to these heuristics. We propose a Bayesian approach to adaptive Schur-product localization for the DEnKF, and extend it to support multiple radii of influence. We test both the empirical validity of (multivariate) adaptive localization, and of our approach. We test a simple toy problem (Lorenz'96), extending it to a multivariate model, and a more realistic geophysical problem (1.5 Layer Quasi-Geostrophic). We show that the multivariate approach has great promise on the toy problem, and that the univariate approach leads to improved filter performance for the realistic geophysical problem. △ Less

Submitted 24 September, 2018; originally announced September 2018.

Report number: CSL-TR-18-6

A Class of Multirate Infinitesimal GARK Methods

Authors: Adrian Sandu

Abstract: Differential equations arising in many practical applications are characterized by multiple time scales. Multirate time integration seeks to solve them efficiently by discretizing each scale with a different, appropriate time step, while ensuring the overall accuracy and stability of the numerical solution. In a seminal paper Knoth and Wolke (APNUM, 1998) proposed a hybrid solution approach: discr… ▽ More Differential equations arising in many practical applications are characterized by multiple time scales. Multirate time integration seeks to solve them efficiently by discretizing each scale with a different, appropriate time step, while ensuring the overall accuracy and stability of the numerical solution. In a seminal paper Knoth and Wolke (APNUM, 1998) proposed a hybrid solution approach: discretize the slow component with an explicit Runge-Kutta method, and advance the fast component via a modified fast differential equation. The idea led to the development of multirate infinitesimal step (MIS) methods by Wensch et al. (BIT, 2009.)Günther and Sandu (BIT, 2016) explained MIS schemes as a particular case of multirate General-structure Additive Runge-Kutta (MR-GARK) methods. The hybrid approach offers extreme flexibility in the choice of the numerical solution process for the fast component. This work constructs a family of multirate infinitesimal GARK schemes (MRI-GARK) that extends the hybrid dynamics approachin multiple ways. Order conditions theory and stability analyses are developed, and practical explicit and implicit methods of up to order four are constructed. Numerical results confirm the theoretical findings. We expect the new MRI-GARK family to be most useful for systems of equations with widely disparate time scales, where the fast process is dispersive, and where the influence of the fast component on the slow dynamics is weak. △ Less

Submitted 1 February, 2022; v1 submitted 7 August, 2018; originally announced August 2018.

Report number: CSL-TR-2018-5 MSC Class: 65L05; 65L06

Design of High-Order Decoupled Multirate GARK Schemes

Authors: Arash Sarshar, Steven Roberts, Adrian Sandu

Abstract: Multirate time integration methods apply different step sizes to resolve different components of the system based on the local activity levels. This local selection of step sizes allows increased computational efficiency while achieving the desired solution accuracy. While the multirate idea is elegant and has been around for decades, multirate methods are not yet widely used in applications. This… ▽ More Multirate time integration methods apply different step sizes to resolve different components of the system based on the local activity levels. This local selection of step sizes allows increased computational efficiency while achieving the desired solution accuracy. While the multirate idea is elegant and has been around for decades, multirate methods are not yet widely used in applications. This is due, in part, to the difficulties raised by the construction of high order multirate schemes. Seeking to overcome these challenges, this work focuses on the design of practical high-order multirate methods using the theoretical framework of generalized additive Runge-Kutta (MrGARK) methods, which provides the generic order conditions and the linear and nonlinear stability analyses. A set of design criteria for practical multirate methods is defined herein: method coefficients should be generic in the step size ratio, but should not depend strongly on this ratio; unnecessary coupling between the fast and the slow components should be avoided; and the step size controllers should adjust both the micro- and the macro-steps. Using these criteria, we develop MrGARK schemes of up to order four that are explicit-explicit (both the fast and slow component are treated explicitly), implicit-explicit (implicit in the fast component and explicit in the slow one), and explicit-implicit (explicit in the fast component and implicit in the slow one). Numerical experiments illustrate the performance of these new schemes. △ Less

Submitted 20 December, 2021; v1 submitted 20 April, 2018; originally announced April 2018.

Report number: CSL-TR-18-4 MSC Class: 65L05; 65L06; 65L20

Journal ref: SIAM Journal on Scientific Computing, Vol. 41, No. 2, 2019, PP. A816-A847

A Learning Based Approach for Uncertainty Analysis in Numerical Weather Prediction Models

Authors: Azam Moosavi, Vishwas Rao, Adrian Sandu

Abstract: Complex numerical weather prediction models incorporate a variety of physical processes, each described by multiple alternative physical schemes with specific parameters. The selection of the physical schemes and the choice of the corresponding physical parameters during model configuration can significantly impact the accuracy of model forecasts. There is no combination of physical schemes that w… ▽ More Complex numerical weather prediction models incorporate a variety of physical processes, each described by multiple alternative physical schemes with specific parameters. The selection of the physical schemes and the choice of the corresponding physical parameters during model configuration can significantly impact the accuracy of model forecasts. There is no combination of physical schemes that works best for all times, at all locations, and under all conditions. It is therefore of considerable interest to understand the interplay between the choice of physics and the accuracy of the resulting forecasts under different conditions. This paper demonstrates the use of machine learning techniques to study the uncertainty in numerical weather prediction models due to the interaction of multiple physical processes. The first problem addressed herein is the estimation of systematic model errors in output quantities of interest at future times, and the use of this information to improve the model forecasts. The second problem considered is the identification of those specific physical processes that contribute most to the forecast uncertainty in the quantity of interest under specified meteorological conditions. The discrepancies between model results and observations at past times are used to learn the relationships between the choice of physical processes and the resulting forecast errors. Numerical experiments are carried out with the Weather Research and Forecasting (WRF) model. The output quantity of interest is the model precipitation, a variable that is both extremely important and very challenging to forecast. The physical processes under consideration include various micro-physics schemes, cumulus parameterizations, short wave, and long wave radiation schemes. The experiments demonstrate the strong potential of machine learning approaches to aid the study of model errors. △ Less

Submitted 20 February, 2018; originally announced February 2018.

Comments: 23 pages, 5 figures, 4 tables

Report number: CSL-2018-2

Adjoint Sensitivity Analysis of Hybrid Multibody Dynamical Systems

Authors: Sebastien Corner, Corina Sandu, Adrian Sandu

Abstract: Sensitivity analysis of multibody systems computes the derivatives of general cost functions that depend on the system solution with respect to parameters or initial conditions. This work develops adjoint sensitivity analysis for hybrid multibody dynamic systems. Hybrid systems are characterized by trajectories that are piecewise continuous in time, with finitely-many discontinuities being caused… ▽ More Sensitivity analysis of multibody systems computes the derivatives of general cost functions that depend on the system solution with respect to parameters or initial conditions. This work develops adjoint sensitivity analysis for hybrid multibody dynamic systems. Hybrid systems are characterized by trajectories that are piecewise continuous in time, with finitely-many discontinuities being caused by events such as elastic/inelastic impacts or sudden changes in constraints. The corresponding direct and adjoint sensitivity variables are also discontinuous at the time of events. The framework discussed herein uses a jump sensitivity matrix to relate the jump conditions for the direct and adjoint sensitivities before and after the time event, and provides analytical jump equations for the adjoint variables. The theoretical framework for sensitivities for hybrid systems is validated on a five-bar mechanism with non-smooth contacts. △ Less

Submitted 16 February, 2018; originally announced February 2018.

Comments: arXiv admin note: text overlap with arXiv:1710.04292

Modeling and sensitivity analysis methodology for hybrid dynamical systems

Authors: Sebastien Corner, Corina Sandu, Adrian Sandu

Abstract: This paper provides an analytical methodology to compute the sensitivities with respect to system parameters for any second order hybrid Ordinary Differential Equation (ODE) system. The hybrid ODE system is characterized by discontinuities in the velocity state variables due to an impulsive jump caused by an instantaneous impact in the motion of the system. The analytical methodology that solves t… ▽ More This paper provides an analytical methodology to compute the sensitivities with respect to system parameters for any second order hybrid Ordinary Differential Equation (ODE) system. The hybrid ODE system is characterized by discontinuities in the velocity state variables due to an impulsive jump caused by an instantaneous impact in the motion of the system. The analytical methodology that solves this problem is structured based on jum** conditions for both the state variables and the sensitivities matrix. The proposed analytical approach is of the benchmarked against a numerical method. △ Less

Submitted 11 October, 2017; originally announced October 2017.

Comments: Direct sensitivity analysis, Switching sensitivities, Constrained systems, Impulsive systems

Solving Parameter Estimation Problems with Discrete Adjoint Exponential Integrators

Authors: Ulrich Römer, Mahesh Narayanamurthi, Adrian Sandu

Abstract: The solution of inverse problems in a variational setting finds best estimates of the model parameters by minimizing a cost function that penalizes the mismatch between model outputs and observations. The gradients required by the numerical optimization process are computed using adjoint models. Exponential integrators are a promising family of time discretizations for evolutionary partial differe… ▽ More The solution of inverse problems in a variational setting finds best estimates of the model parameters by minimizing a cost function that penalizes the mismatch between model outputs and observations. The gradients required by the numerical optimization process are computed using adjoint models. Exponential integrators are a promising family of time discretizations for evolutionary partial differential equations. In order to allow the use of these discretizations in the context of inverse problems adjoints of exponential integrators are required. This work derives the discrete adjoint formulae for a W-type exponential propagation iterative methods of Runge-Kutta type (EPIRK-W). These methods allow arbitrary approximations of the Jacobian while maintaining the overall accuracy of the forward integration. The use of Jacobian approximation matrices that do not depend on the model state avoids the complex calculation of Hessians in the discrete adjoint formulae, and allows efficient adjoint code generation via algorithmic differentiation. We use the discrete EPIRK-W adjoints to solve inverse problems with the Lorenz-96 model and a computational magnetics benchmark test. Numerical results validate our theoretical derivations. △ Less

Submitted 8 April, 2017; originally announced April 2017.

Report number: CSL-TR-17-4 MSC Class: 34H05; 34K29; 34K35

Journal ref: Optimization Methods and Software, 33 (2018), 750-770

EPIRK-W and EPIRK-K time discretization methods

Authors: Mahesh Narayanamurthi, Paul Tranquilli, Adrian Sandu, Mayya Tokman

Abstract: Exponential integrators are special time discretization methods where the traditional linear system solves used by implicit schemes are replaced with computing the action of matrix exponential-like functions on a vector. A very general formulation of exponential integrators is offered by the Exponential Propagation Iterative methods of Runge-Kutta type (EPIRK) family of schemes. The use of Jacobia… ▽ More Exponential integrators are special time discretization methods where the traditional linear system solves used by implicit schemes are replaced with computing the action of matrix exponential-like functions on a vector. A very general formulation of exponential integrators is offered by the Exponential Propagation Iterative methods of Runge-Kutta type (EPIRK) family of schemes. The use of Jacobian approximations is an important strategy to drastically reduce the overall computational costs of implicit schemes while maintaining the quality of their solutions. This paper extends the EPIRK class to allow the use of inexact Jacobians as arguments of the matrix exponential-like functions. Specifically, we develop two new families of methods: EPIRK-W integrators that can accommodate any approximation of the Jacobian, and EPIRK-K integrators that rely on a specific Krylov-subspace projection of the exact Jacobian. Classical order conditions theories are constructed for these families. A practical EPIRK-W method of order three and an EPIRK-K method of order four are developed. Numerical experiments indicate that the methods proposed herein are computationally favorable when compared to existing exponential integrators. △ Less

Submitted 25 January, 2017; v1 submitted 23 January, 2017; originally announced January 2017.

Comments: Fixed spelling error, rewrote a sentence and moved a paragraph after rephrasing it. Fixed a small bug in the legend of figure 8b (results unchanged). Fixed a typo in figure caption. Fixed a typo in a sentence. Results unchanged

Report number: CSL-TR-17-2

Multivariate predictions of local reduced-order-model errors and dimensions

Authors: Azam Moosavi, Razvan Stefanescu, Adrian Sandu

Abstract: This paper introduces multivariate input-output models to predict the errors and bases dimensions of local parametric Proper Orthogonal Decomposition reduced-order models. We refer to these multivariate map**s as the MP-LROM models. We employ Gaussian Processes and Artificial Neural Networks to construct approximations of these multivariate map**s. Numerical results with a viscous Burgers mode… ▽ More This paper introduces multivariate input-output models to predict the errors and bases dimensions of local parametric Proper Orthogonal Decomposition reduced-order models. We refer to these multivariate map**s as the MP-LROM models. We employ Gaussian Processes and Artificial Neural Networks to construct approximations of these multivariate map**s. Numerical results with a viscous Burgers model illustrate the performance and potential of the machine learning based regression MP-LROM models to approximate the characteristics of parametric local reduced-order models. The predicted reduced-order models errors are compared against the multi-fidelity correction and reduced order model error surrogates methods predictions, whereas the predicted reduced-order dimensions are tested against the standard method based on the spectrum of snapshots matrix. Since the MP-LROM models incorporate more features and elements to construct the probabilistic map**s they achieve more accurate results. However, for high-dimensional parametric spaces, the MP-LROM models might suffer from the curse of dimensionality. Scalability challenges of MP-LROM models and the feasible ways of addressing them are also discussed in this study. △ Less

Submitted 13 January, 2017; originally announced January 2017.

Comments: 19 pages, 15 figures, 7 tables. arXiv admin note: substantial text overlap with arXiv:1511.02909

Report number: CSL-2017-1

LIRK-W: Linearly-implicit Runge-Kutta methods with approximate matrix factorization

Authors: Paul Tranquilli, Adrian Sandu, Hong Zhang

Abstract: This paper develops a new class of linearly implicit time integration schemes called Linearly-Implicit Runge-Kutta-W (LIRK-W) methods. These schemes are based on an implicit-explicit approach which does not require a splitting of the right hand side and allow for arbitrary, time dependent, and stage varying approximations of the linear systems appearing in the method. Several formulations of LIRK-… ▽ More This paper develops a new class of linearly implicit time integration schemes called Linearly-Implicit Runge-Kutta-W (LIRK-W) methods. These schemes are based on an implicit-explicit approach which does not require a splitting of the right hand side and allow for arbitrary, time dependent, and stage varying approximations of the linear systems appearing in the method. Several formulations of LIRK-W schemes, each designed for specific approximation types, and their associated order condition theories are presented. △ Less

Submitted 21 November, 2016; originally announced November 2016.

Report number: CSL-TR-16-7

A Numerical Investigation of Matrix-Free Implicit Time-Step** Methods for Large CFD Simulations

Authors: Arash Sarshar, Paul Tranquilli, Brent Pickering, Andrew McCall, Adrian Sandu, Christopher J. Roy

Abstract: This paper is concerned with the development and testing of advanced time-step** methods suited for the integration of time-accurate, real-world applications of computational fluid dynamics (CFD). The performance of several time discretization methods is studied numerically with regards to computational efficiency, order of accuracy, and stability, as well as the ability to treat effectively sti… ▽ More This paper is concerned with the development and testing of advanced time-step** methods suited for the integration of time-accurate, real-world applications of computational fluid dynamics (CFD). The performance of several time discretization methods is studied numerically with regards to computational efficiency, order of accuracy, and stability, as well as the ability to treat effectively stiff problems. We consider matrix-free implementations, a popular approach for time-step** methods applied to large CFD applications due to its adherence to scalable matrix-vector operations and a small memory footprint. We compare explicit methods with matrix-free implementations of implicit, linearly-implicit, as well as Rosenbrock-Krylov methods. We show that Rosenbrock-Krylov methods are competitive with existing techniques excelling for a number of problem types and settings. △ Less

Submitted 30 September, 2017; v1 submitted 22 July, 2016; originally announced July 2016.

Report number: Computational Science Lab CSL-TR-16-6 MSC Class: 65L05; 65L06; 65L20

Journal ref: Computers & Fluids, Volume 159, 15 Dec. 2017, PP. 53-63

Cluster Sampling Filters for Non-Gaussian Data Assimilation

Authors: Ahmed Attia, Azam Moosavi, Adrian Sandu

Abstract: This paper presents a fully non-Gaussian version of the Hamiltonian Monte Carlo (HMC) sampling filter. The Gaussian prior assumption in the original HMC filter is relaxed. Specifically, a clustering step is introduced after the forecast phase of the filter, and the prior density function is estimated by fitting a Gaussian Mixture Model (GMM) to the prior ensemble. Using the data likelihood functio… ▽ More This paper presents a fully non-Gaussian version of the Hamiltonian Monte Carlo (HMC) sampling filter. The Gaussian prior assumption in the original HMC filter is relaxed. Specifically, a clustering step is introduced after the forecast phase of the filter, and the prior density function is estimated by fitting a Gaussian Mixture Model (GMM) to the prior ensemble. Using the data likelihood function, the posterior density is then formulated as a mixture density, and is sampled using a HMC approach (or any other scheme capable of sampling multimodal densities in high-dimensional subspaces). The main filter developed herein is named "cluster HMC sampling filter" (ClHMC). A multi-chain version of the ClHMC filter, namely MC-ClHMC is also proposed to guarantee that samples are taken from the vicinities of all probability modes of the formulated posterior. The new methodologies are tested using a quasi-geostrophic (QG) model with double-gyre wind forcing and bi-harmonic friction. Numerical results demonstrate the usefulness of using GMMs to relax the Gaussian prior assumption in the HMC filtering paradigm. △ Less

Submitted 18 August, 2016; v1 submitted 13 July, 2016; originally announced July 2016.

A Parallel Implementation of the Ensemble Kalman Filter Based on Modified Cholesky Decomposition

Authors: Elias D. Nino, Adrian Sandu, Xinwei Deng

Abstract: This paper discusses an efficient parallel implementation of the ensemble Kalman filter based on the modified Cholesky decomposition. The proposed implementation starts with decomposing the domain into sub-domains. In each sub-domain a sparse estimation of the inverse background error covariance matrix is computed via a modified Cholesky decomposition; the estimates are computed concurrently on se… ▽ More This paper discusses an efficient parallel implementation of the ensemble Kalman filter based on the modified Cholesky decomposition. The proposed implementation starts with decomposing the domain into sub-domains. In each sub-domain a sparse estimation of the inverse background error covariance matrix is computed via a modified Cholesky decomposition; the estimates are computed concurrently on separate processors. The sparsity of this estimator is dictated by the conditional independence of model components for some radius of influence. Then, the assimilation step is carried out in parallel without the need of inter-processor communication. Once the local analysis states are computed, the analysis sub-domains are mapped back onto the global domain to obtain the analysis ensemble. Computational experiments are performed using the Atmospheric General Circulation Model (SPEEDY) with the T-63 resolution on the Blueridge cluster at Virginia Tech. The number of processors used in the experiments ranges from 96 to 2,048. The proposed implementation outperforms in terms of accuracy the well-known local ensemble transform Kalman filter (LETKF) for all the model variables. The computational time of the proposed implementation is similar to that of the parallel LETKF method (where no covariance estimation is performed). Finally, for the largest number of processors, the proposed parallel implementation is 400 times faster than the serial version of the proposed method. △ Less

Submitted 31 May, 2016; originally announced June 2016.

Comments: arXiv admin note: text overlap with arXiv:1605.08875

Report number: CSTR-3/2016 MSC Class: 62L20; 62M05; 62M20; 62P35

An Ensemble Kalman Filter Implementation Based on Modified Cholesky Decomposition for Inverse Covariance Matrix Estimation

Authors: Elias D. Nino, Adrian Sandu, Xinwei Deng

Abstract: This paper develops an efficient implementation of the ensemble Kalman filter based on a modified Cholesky decomposition for inverse covariance matrix estimation. This implementation is named EnKF-MC. Background errors corresponding to distant model components with respect to some radius of influence are assumed to be conditionally independent. This allows to obtain sparse estimators of the invers… ▽ More This paper develops an efficient implementation of the ensemble Kalman filter based on a modified Cholesky decomposition for inverse covariance matrix estimation. This implementation is named EnKF-MC. Background errors corresponding to distant model components with respect to some radius of influence are assumed to be conditionally independent. This allows to obtain sparse estimators of the inverse background error covariance matrix. The computational effort of the proposed method is discussed and different formulations based on various matrix identities are provided. Furthermore, an asymptotic proof of convergence with regard to the ensemble size is presented. In order to assess the performance and the accuracy of the proposed method, experiments are performed making use of the Atmospheric General Circulation Model SPEEDY. The results are compared against those obtained using the local ensemble transform Kalman filter (LETKF). Tests are performed for dense observations ($100\%$ and $50\%$ of the model components are observed) as well as for sparse observations (only $12\%$, $6\%$, and $4\%$ of model components are observed). The results reveal that the use of modified Cholesky for inverse covariance matrix estimation can reduce the impact of spurious correlations during the assimilation cycle, i.e., the results of the proposed method are of better quality than those obtained via the LETKF in terms of root mean square error. △ Less

Submitted 28 May, 2016; originally announced May 2016.

Report number: CSTR-2/2016

The Reduced-Order Hybrid Monte Carlo Sampling Smoother

Authors: Ahmed Attia, Razvan Stefanescu, Adrian Sandu

Abstract: Hybrid Monte-Carlo (HMC) sampling smoother is a fully non-Gaussian four-dimensional data assimilation algorithm that works by directly sampling the posterior distribution formulated in the Bayesian framework. The smoother in its original formulation is computationally expensive due to the intrinsic requirement of running the forward and adjoint models repeatedly. Here we present computationally ef… ▽ More Hybrid Monte-Carlo (HMC) sampling smoother is a fully non-Gaussian four-dimensional data assimilation algorithm that works by directly sampling the posterior distribution formulated in the Bayesian framework. The smoother in its original formulation is computationally expensive due to the intrinsic requirement of running the forward and adjoint models repeatedly. Here we present computationally efficient versions of the HMC sampling smoother based on reduced-order approximations of the underlying model dynamics. The schemes developed herein are tested numerically using the shallow-water equations model on Cartesian coordinates. The results reveal that the reduced-order versions of the smoother are capable of accurately capturing the posterior probability density, while being significantly faster than the original full order formulation. △ Less

Submitted 1 January, 2016; originally announced January 2016.

Comments: 32 pages, 2 figures

Report number: CSTR-1/2016

Robust data assimilation using $L_1$ and Huber norms

Authors: Vishwas Rao, Adrian Sandu, Michael Ng, Elias Nino-Ruiz

Abstract: Data assimilation is the process to fuse information from priors, observations of nature, and numerical models, in order to obtain best estimates of the parameters or state of a physical system of interest. Presence of large errors in some observational data, e.g., data collected from a faulty instrument, negatively affect the quality of the overall assimilation results. This work develops a sys… ▽ More Data assimilation is the process to fuse information from priors, observations of nature, and numerical models, in order to obtain best estimates of the parameters or state of a physical system of interest. Presence of large errors in some observational data, e.g., data collected from a faulty instrument, negatively affect the quality of the overall assimilation results. This work develops a systematic framework for robust data assimilation. The new algorithms continue to produce good analyses in the presence of observation outliers. The approach is based on replacing the traditional $Ł_2$ norm formulation of data assimilation problems with formulations based on $Ł_1$ and Huber norms. Numerical experiments using the Lorenz-96 and the shallow water on the sphere models illustrate how the new algorithms outperform traditional data assimilation approaches in the presence of data outliers. △ Less

Submitted 4 November, 2015; originally announced November 2015.

Comments: 25 pages, Submitted to SISC

Report number: CSL-TR-15-21

A Hybrid Monte-Carlo Sampling Smoother for Four Dimensional Data Assimilation

Authors: Ahmed Attia, Vishwas Rao, Adrian Sandu

Abstract: This paper constructs an ensemble-based sampling smoother for four-dimensional data assimilation using a Hybrid/Hamiltonian Monte-Carlo approach. The smoother samples efficiently from the posterior probability density of the solution at the initial time. Unlike the well-known ensemble Kalman smoother, which is optimal only in the linear Gaussian case, the proposed methodology naturally accommodate… ▽ More This paper constructs an ensemble-based sampling smoother for four-dimensional data assimilation using a Hybrid/Hamiltonian Monte-Carlo approach. The smoother samples efficiently from the posterior probability density of the solution at the initial time. Unlike the well-known ensemble Kalman smoother, which is optimal only in the linear Gaussian case, the proposed methodology naturally accommodates non-Gaussian errors and non-linear model dynamics and observation operators. Unlike the four-dimensional variational met\-hod, which only finds a mode of the posterior distribution, the smoother provides an estimate of the posterior uncertainty. One can use the ensemble mean as the minimum variance estimate of the state, or can use the ensemble in conjunction with the variational approach to estimate the background errors for subsequent assimilation windows. Numerical results demonstrate the advantages of the proposed method compared to the traditional variational and ensemble-based smoothing methods. △ Less

Submitted 18 May, 2015; originally announced May 2015.

Comments: 33 Pages

Report number: CSL-TR-19-2015