Showing 1–28 of 28 results for author: Sandu, A

Preserving Nonlinear Constraints in Variational Flow Filtering Data Assimilation

Authors: Amit N. Subrahmanya, Andrey A. Popov, Reid J. Gomillion, Adrian Sandu

Abstract: Data assimilation aims to estimate the states of a dynamical system by optimally combining sparse and noisy observations of the physical system with uncertain forecasts produced by a computational model. The states of many dynamical systems of interest obey nonlinear physical constraints, and the corresponding dynamics is confined to a certain sub-manifold of the state space. Standard data assimil… ▽ More Data assimilation aims to estimate the states of a dynamical system by optimally combining sparse and noisy observations of the physical system with uncertain forecasts produced by a computational model. The states of many dynamical systems of interest obey nonlinear physical constraints, and the corresponding dynamics is confined to a certain sub-manifold of the state space. Standard data assimilation techniques applied to such systems yield posterior states lying outside the manifold, violating the physical constraints. This work focuses on particle flow filters which use stochastic differential equations to evolve state samples from a prior distribution to samples from an observation-informed posterior distribution. The variational Fokker-Planck (VFP) -- a generic particle flow filtering framework -- is extended to incorporate non-linear, equality state constraints in the analysis. To this end, two algorithmic approaches that modify the VFP stochastic differential equation are discussed: (i) VFPSTAB, to inexactly preserve constraints with the addition of a stabilizing drift term, and (ii) VFPDAE, to exactly preserve constraints by treating the VFP dynamics as a stochastic differential-algebraic equation (SDAE). Additionally, an implicit-explicit time integrator is developed to evolve the VFPDAE dynamics. The strength of the proposed approach for constraint preservation in data assimilation is demonstrated on three test problems: the double pendulum, Korteweg-de-Vries, and the incompressible Navier-Stokes equations. △ Less

Submitted 7 May, 2024; originally announced May 2024.

Report number: CSL-TR-24-1 MSC Class: 65C05; 62F15; 62F30; 35R30

Improving the Adaptive Moment Estimation (ADAM) stochastic optimizer through an Implicit-Explicit (IMEX) time-step** approach

Authors: Abhinab Bhattacharjee, Andrey A. Popov, Arash Sarshar, Adrian Sandu

Abstract: The Adam optimizer, often used in Machine Learning for neural network training, corresponds to an underlying ordinary differential equation (ODE) in the limit of very small learning rates. This work shows that the classical Adam algorithm is a first order implicit-explicit (IMEX) Euler discretization of the underlying ODE. Employing the time discretization point of view, we propose new extensions… ▽ More The Adam optimizer, often used in Machine Learning for neural network training, corresponds to an underlying ordinary differential equation (ODE) in the limit of very small learning rates. This work shows that the classical Adam algorithm is a first order implicit-explicit (IMEX) Euler discretization of the underlying ODE. Employing the time discretization point of view, we propose new extensions of the Adam scheme obtained by using higher order IMEX methods to solve the ODE. Based on this approach, we derive a new optimization algorithm for neural network training that performs better than classical Adam on several regression and classification problems. △ Less

Submitted 20 March, 2024; originally announced March 2024.

Report number: CSL-TR-2024-2

Cheap Ways of Extracting Clinical Markers from Texts

Authors: Anastasia Sandu, Teodor Mihailescu, Sergiu Nisioi

Abstract: This paper describes the work of the UniBuc Archaeology team for CLPsych's 2024 Shared Task, which involved finding evidence within the text supporting the assigned suicide risk level. Two types of evidence were required: highlights (extracting relevant spans within the text) and summaries (aggregating evidence into a synthesis). Our work focuses on evaluating Large Language Models (LLM) as oppose… ▽ More This paper describes the work of the UniBuc Archaeology team for CLPsych's 2024 Shared Task, which involved finding evidence within the text supporting the assigned suicide risk level. Two types of evidence were required: highlights (extracting relevant spans within the text) and summaries (aggregating evidence into a synthesis). Our work focuses on evaluating Large Language Models (LLM) as opposed to an alternative method that is much more memory and resource efficient. The first approach employs a good old-fashioned machine learning (GOML) pipeline consisting of a tf-idf vectorizer with a logistic regression classifier, whose representative features are used to extract relevant highlights. The second, more resource intensive, uses an LLM for generating the summaries and is guided by chain-of-thought to provide sequences of text indicating clinical markers. △ Less

Submitted 17 March, 2024; originally announced March 2024.

Comments: https://github.com/nlp-unibuc/clpsych24-task

Simultaneous Optimal System and Controller Design for Multibody Systems with Joint Friction using Direct Sensitivities

Authors: Adwait Verulkar, Corina Sandu, Adrian Sandu, Daniel Dopico

Abstract: Real-world multibody systems are often subject to phenomena like friction, joint clearances, and external events. These phenomena can significantly impact the optimal design of the system and its controller. This work addresses the gradient-based optimization methodology for multibody dynamic systems with joint friction using a direct sensitivity approach for gradient computation. After a thorough… ▽ More Real-world multibody systems are often subject to phenomena like friction, joint clearances, and external events. These phenomena can significantly impact the optimal design of the system and its controller. This work addresses the gradient-based optimization methodology for multibody dynamic systems with joint friction using a direct sensitivity approach for gradient computation. After a thorough review of various friction models developed over the years, the Brown McPhee model has been found to be the most suitable for the study due to its accuracy for dynamic simulation and its compatibility with sensitivity analysis. The methodology supports co-design of the system and its controller, which is especially relevant for applications like robotics and servo-mechanical systems where the actuation and the design are highly dependent on each other. Numerical results are obtained using a new implementation of the MBSVT (Multi-Body Systems at Virginia Tech) software package; MBSVT 2.0 is reprogrammed in Julia for ease of implementation while maintaining high computational efficiency. Three case studies are provided to demonstrate the attractive properties of simultaneous optimal design and control approach for certain applications. △ Less

Submitted 25 December, 2023; originally announced December 2023.

Adversarial Training Using Feedback Loops

Authors: Ali Haisam Muhammad Rafid, Adrian Sandu

Abstract: Deep neural networks (DNN) have found wide applicability in numerous fields due to their ability to accurately learn very complex input-output relations. Despite their accuracy and extensive use, DNNs are highly susceptible to adversarial attacks due to limited generalizability. For future progress in the field, it is essential to build DNNs that are robust to any kind of perturbations to the data… ▽ More Deep neural networks (DNN) have found wide applicability in numerous fields due to their ability to accurately learn very complex input-output relations. Despite their accuracy and extensive use, DNNs are highly susceptible to adversarial attacks due to limited generalizability. For future progress in the field, it is essential to build DNNs that are robust to any kind of perturbations to the data points. In the past, many techniques have been proposed to robustify DNNs using first-order derivative information of the network. This paper proposes a new robustification approach based on control theory. A neural network architecture that incorporates feedback control, named Feedback Neural Networks, is proposed. The controller is itself a neural network, which is trained using regular and adversarial data such as to stabilize the system outputs. The novel adversarial training approach based on the feedback control architecture is called Feedback Looped Adversarial Training (FLAT). Numerical results on standard test problems empirically show that our FLAT method is more effective than the state-of-the-art to guard against adversarial attacks. △ Less

Submitted 23 August, 2023; v1 submitted 22 August, 2023; originally announced August 2023.

Neural Network Reduction with Guided Regularizers

Authors: Ali Haisam Muhammad Rafid, Adrian Sandu

Abstract: Regularization techniques such as $\mathcal{L}_1$ and $\mathcal{L}_2$ regularizers are effective in sparsifying neural networks (NNs). However, to remove a certain neuron or channel in NNs, all weight elements related to that neuron or channel need to be prunable, which is not guaranteed by traditional regularization. This paper proposes a simple new approach named "Guided Regularization" that pri… ▽ More Regularization techniques such as $\mathcal{L}_1$ and $\mathcal{L}_2$ regularizers are effective in sparsifying neural networks (NNs). However, to remove a certain neuron or channel in NNs, all weight elements related to that neuron or channel need to be prunable, which is not guaranteed by traditional regularization. This paper proposes a simple new approach named "Guided Regularization" that prioritizes the weights of certain NN units more than others during training, which renders some of the units less important and thus, prunable. This is different from the scattered sparsification of $\mathcal{L}_1$ and $\mathcal{L}_2$ regularizers where the the components of a weight matrix that are zeroed out can be located anywhere. The proposed approach offers a natural reduction of NN in the sense that a model is being trained while also neutralizing unnecessary units. We empirically demonstrate that our proposed method is effective in pruning NNs while maintaining performance. △ Less

Submitted 28 May, 2023; originally announced May 2023.

The Model Forest Ensemble Kalman Filter

Authors: Andrey A Popov, Adrian Sandu

Abstract: Traditional data assimilation uses information obtained from the propagation of one physics-driven model and combines it with information derived from real-world observations in order to obtain a better estimate of the truth of some natural process. However, in many situations multiple simulation models that describe the same physical phenomenon are available. Such models can have different source… ▽ More Traditional data assimilation uses information obtained from the propagation of one physics-driven model and combines it with information derived from real-world observations in order to obtain a better estimate of the truth of some natural process. However, in many situations multiple simulation models that describe the same physical phenomenon are available. Such models can have different sources. On one hand there are theory-guided models are constructed from first physical principles, while on the other there are data-driven models that are constructed from snapshots of high fidelity information. In this work we provide a possible way to make use of this collection of models in data assimilation by generalizing the idea of model hierarchies into model forests -- collections of high fidelity and low fidelity models organized in a gro** of model trees such as to capture various relationships between different models. We generalize the multifidelity ensemble Kalman filter that previously operated on model hierarchies into the model forest ensemble Kalman filter through a generalized theory of linear control variates. This new filter allows for much more freedom when treading the line between accuracy and speed. Numerical experiments with a high fidelity quasi-geostrophic model and two of its low fidelity reduced order models validate the accuracy of our approach. △ Less

Submitted 21 October, 2022; originally announced October 2022.

MSC Class: 62F15; 62M20; 65C05; 65M60; 76F70; 86A22; 93E11

A Meta-learning Formulation of the Autoencoder Problem for Non-linear Dimensionality Reduction

Authors: Andrey A. Popov, Arash Sarshar, Austin Chennault, Adrian Sandu

Abstract: A rapidly growing area of research is the use of machine learning approaches such as autoencoders for dimensionality reduction of data and models in scientific applications. We show that the canonical formulation of autoencoders suffers from several deficiencies that can hinder their performance. Using a meta-learning approach, we reformulate the autoencoder problem as a bi-level optimization proc… ▽ More A rapidly growing area of research is the use of machine learning approaches such as autoencoders for dimensionality reduction of data and models in scientific applications. We show that the canonical formulation of autoencoders suffers from several deficiencies that can hinder their performance. Using a meta-learning approach, we reformulate the autoencoder problem as a bi-level optimization procedure that explicitly solves the dimensionality reduction task. We prove that the new formulation corrects the identified deficiencies with canonical autoencoders, provide a practical way to solve it, and showcase the strength of this formulation with a simple numerical illustration. △ Less

Submitted 27 July, 2022; v1 submitted 14 July, 2022; originally announced July 2022.

Report number: CSL-21-9

Physics-informed neural networks for PDE-constrained optimization and control

Authors: Jostein Barry-Straume, Arash Sarshar, Andrey A. Popov, Adrian Sandu

Abstract: A fundamental problem in science and engineering is designing optimal control policies that steer a given system towards a desired outcome. This work proposes Control Physics-Informed Neural Networks (Control PINNs) that simultaneously solve for a given system state, and for the optimal control signal, in a one-stage framework that conforms to the underlying physical laws. Prior approaches use a t… ▽ More A fundamental problem in science and engineering is designing optimal control policies that steer a given system towards a desired outcome. This work proposes Control Physics-Informed Neural Networks (Control PINNs) that simultaneously solve for a given system state, and for the optimal control signal, in a one-stage framework that conforms to the underlying physical laws. Prior approaches use a two-stage framework that first models and then controls a system in sequential order. In contrast, a Control PINN incorporates the required optimality conditions in its architecture and in its loss function. The success of Control PINNs is demonstrated by solving the following open-loop optimal control problems: (i) an analytical problem, (ii) a one-dimensional heat equation, and (iii) a two-dimensional predator-prey problem. △ Less

Submitted 18 August, 2022; v1 submitted 6 May, 2022; originally announced May 2022.

Report number: CSL-TR-22-2 ACM Class: I.2.6; I.2.8; I.5.1; G.1.6; G.1.8

Ensemble Variational Fokker-Planck Methods for Data Assimilation

Authors: Amit N Subrahmanya, Andrey A Popov, Adrian Sandu

Abstract: Particle flow filters solve Bayesian inference problems by smoothly transforming a set of particles into samples from the posterior distribution. Particles move in state space under the flow of an McKean-Vlasov-Ito process. This work introduces the Variational Fokker-Planck (VFP) framework for data assimilation, a general approach that includes previously known particle flow filters as special cas… ▽ More Particle flow filters solve Bayesian inference problems by smoothly transforming a set of particles into samples from the posterior distribution. Particles move in state space under the flow of an McKean-Vlasov-Ito process. This work introduces the Variational Fokker-Planck (VFP) framework for data assimilation, a general approach that includes previously known particle flow filters as special cases. The McKean-Vlasov-Ito process that transforms particles is defined via an optimal drift that depends on the selected diffusion term. It is established that the underlying probability density - sampled by the ensemble of particles - converges to the Bayesian posterior probability density. For a finite number of particles the optimal drift contains a regularization term that nudges particles toward becoming independent random variables. Based on this analysis, we derive computationally-feasible approximate regularization approaches that penalize the mutual information between pairs of particles, and avoid particle collapse. Moreover, the diffusion plays a role akin to a particle rejuvenation approach that aims to alleviate particle collapse. The VFP framework is very flexible. Different assumptions on prior and intermediate probability distributions can be used to implement the optimal drift, and localization and covariance shrinkage can be applied to alleviate the curse of dimensionality. A robust implicit-explicit method is discussed for the efficient integration of stiff McKean-Vlasov-Ito processes. The effectiveness of the VFP framework is demonstrated on three progressively more challenging test problems, namely the Lorenz '63, Lorenz '96 and the quasi-geostrophic equations. △ Less

Submitted 19 January, 2024; v1 submitted 27 November, 2021; originally announced November 2021.

Report number: CSL-TR-21-10 MSC Class: 65C05; 93E11; 62F15; 86A22

Adjoint-Matching Neural Network Surrogates for Fast 4D-Var Data Assimilation

Authors: Austin Chennault, Andrey A. Popov, Amit N. Subrahmanya, Rachel Cooper, Ali Haisam Muhammad Rafid, Anuj Karpatne, Adrian Sandu

Abstract: Data assimilation is the process of fusing information from imperfect computer simulations with noisy, sparse measurements of reality to obtain improved estimates of the state or parameters of a dynamical system of interest. The data assimilation procedures used in many geoscience applications, such as numerical weather forecasting, are variants of the our-dimensional variational (4D-Var) algorith… ▽ More Data assimilation is the process of fusing information from imperfect computer simulations with noisy, sparse measurements of reality to obtain improved estimates of the state or parameters of a dynamical system of interest. The data assimilation procedures used in many geoscience applications, such as numerical weather forecasting, are variants of the our-dimensional variational (4D-Var) algorithm. The cost of solving the underlying 4D-Var optimization problem is dominated by the cost of repeated forward and adjoint model runs. This motivates substituting the evaluations of the physical model and its adjoint by fast, approximate surrogate models. Neural networks offer a promising approach for the data-driven creation of surrogate models. The accuracy of the surrogate 4D-Var solution depends on the accuracy with each the surrogate captures both the forward and the adjoint model dynamics. We formulate and analyze several approaches to incorporate adjoint information into the construction of neural network surrogates. The resulting networks are tested on unseen data and in a sequential data assimilation problem using the Lorenz-63 system. Surrogates constructed using adjoint information demonstrate superior performance on the 4D-Var data assimilation problem compared to a standard neural network surrogate that uses only forward dynamics information. △ Less

Submitted 20 December, 2022; v1 submitted 16 November, 2021; originally announced November 2021.

Report number: CSL-TR-21-7 MSC Class: 34A55; 68T07; 90C30; 65L09

Investigation of Nonlinear Model Order Reduction of the Quasigeostrophic Equations through a Physics-Informed Convolutional Autoencoder

Authors: Rachel Cooper, Andrey A. Popov, Adrian Sandu

Abstract: Reduced order modeling (ROM) is a field of techniques that approximates complex physics-based models of real-world processes by inexpensive surrogates that capture important dynamical characteristics with a smaller number of degrees of freedom. Traditional ROM techniques such as proper orthogonal decomposition (POD) focus on linear projections of the dynamics onto a set of spectral features. In th… ▽ More Reduced order modeling (ROM) is a field of techniques that approximates complex physics-based models of real-world processes by inexpensive surrogates that capture important dynamical characteristics with a smaller number of degrees of freedom. Traditional ROM techniques such as proper orthogonal decomposition (POD) focus on linear projections of the dynamics onto a set of spectral features. In this paper we explore the construction of ROM using autoencoders (AE) that perform nonlinear projections of the system dynamics onto a low dimensional manifold learned from data. The approach uses convolutional neural networks (CNN) to learn spatial features as opposed to spectral, and utilize a physics informed (PI) cost function in order to capture temporal features as well. Our investigation using the quasi-geostrophic equations reveals that while the PI cost function helps with spatial reconstruction, spatial features are less powerful than spectral features, and that construction of ROMs through machine learning-based methods requires significant investigation into novel non-standard methodologies. △ Less

Submitted 27 August, 2021; originally announced August 2021.

Report number: CSL-TR-21-5

Multifidelity Ensemble Kalman Filtering Using Surrogate Models Defined by Physics-Informed Autoencoders

Authors: Andrey A Popov, Adrian Sandu

Abstract: Data assimilation is a Bayesian inference process that obtains an enhanced understanding of a physical system of interest by fusing information from an inexact physics-based model, and from noisy sparse observations of reality. The multifidelity ensemble Kalman filter (MFEnKF) recently developed by the authors combines a full-order physical model and a hierarchy of reduced order surrogate models i… ▽ More Data assimilation is a Bayesian inference process that obtains an enhanced understanding of a physical system of interest by fusing information from an inexact physics-based model, and from noisy sparse observations of reality. The multifidelity ensemble Kalman filter (MFEnKF) recently developed by the authors combines a full-order physical model and a hierarchy of reduced order surrogate models in order to increase the computational efficiency of data assimilation. The standard MFEnKF uses linear couplings between models, and is statistically optimal in case of Gaussian probability densities. This work extends MFEnKF to work with non-linear couplings between the models. Optimal nonlinear projection and interpolation operators are obtained by appropriately trained physics-informed autoencoders, and this approach allows to construct reduced order surrogate models with less error than conventional linear methods. Numerical experiments with the canonical Lorenz '96 model illustrate that nonlinear surrogates perform better than linear projection-based ones in the context of multifidelity filtering. △ Less

Submitted 10 March, 2021; v1 submitted 25 February, 2021; originally announced February 2021.

Report number: CSL-TR-21-1

Restoring Uniqueness in MicroVM Snapshots

Authors: Marc Brooker, Adrian Costin Catangiu, Mike Danilov, Alexander Graf, Colm MacCarthaigh, Andrei Sandu

Abstract: Code initialization -- the step of loading code, executing static code, filling caches, and forming re-used connections -- tends to dominate cold-start time in serverless compute systems such as AWS Lambda. Post-initialization memory snapshots, cloned and restored on start, have emerged as a viable solution to this problem, with incremental snapshot and fast restore support in VMMs like Firecracke… ▽ More Code initialization -- the step of loading code, executing static code, filling caches, and forming re-used connections -- tends to dominate cold-start time in serverless compute systems such as AWS Lambda. Post-initialization memory snapshots, cloned and restored on start, have emerged as a viable solution to this problem, with incremental snapshot and fast restore support in VMMs like Firecracker. Saving memory introduces the challenge of managing high-value memory contents, such as cryptographic secrets. Cloning introduces the challenge of restoring the uniqueness of the VMs, to allow them to do unique things like generate UUIDs, secrets, and nonces. This paper examines solutions to these problems in the every microsecond counts context of serverless cold-start, and discusses the state-of-the-art of available solutions. We present two new interfaces aimed at solving this problem -- MADV\_WIPEONSUSPEND and SysGenId -- and compare them to alternative solutions. △ Less

Submitted 4 February, 2021; originally announced February 2021.

Partitioned Exponential Methods for Coupled Multiphysics Systems

Authors: Mahesh Narayanamurthi, Adrian Sandu

Abstract: Multiphysics problems involving two or more coupled physical phenomena are ubiquitous in science and engineering. This work develops a new partitioned exponential approach for the time integration of multiphysics problems. After a possible semi-discretization in space, the class of problems under consideration is modeled by a system of ordinary differential equations where the right-hand side is a… ▽ More Multiphysics problems involving two or more coupled physical phenomena are ubiquitous in science and engineering. This work develops a new partitioned exponential approach for the time integration of multiphysics problems. After a possible semi-discretization in space, the class of problems under consideration is modeled by a system of ordinary differential equations where the right-hand side is a summation of two component functions, each corresponding to a given set of physical processes. The partitioned-exponential methods proposed herein evolve each component of the system via an exponential integrator, and information between partitions is exchanged via coupling terms. The traditional approach to constructing exponential methods, based on the variation-of-constants formula, is not directly applicable to partitioned systems. Rather, our approach to develo** new partitioned-exponential families is based on a general-structure additive formulation of the schemes. Two method formulations are considered, one based on a linear-nonlinear splitting of the right hand component functions, and another based on approximate Jacobians. The paper develops classical (non-stiff) order conditions theory for partitioned exponential schemes based on particular families of T-trees and B-series theory. Several practical methods of third order are constructed that extend the Rosenbrock-type and EPIRK families of exponential integrators. Several implementation optimizations specific to the application of these methods to reaction-diffusion systems are also discussed. Numerical experiments reveal that the new partitioned-exponential methods can perform better than traditional unpartitioned exponential methods on some problems. △ Less

Submitted 28 August, 2019; v1 submitted 25 August, 2019; originally announced August 2019.

Comments: Fixed a definition and other minor typos. Results remain unchanged

Report number: CSL-TR-19-6 MSC Class: 65L05; 65L04; 65F60; 65M22; 65Y05

ODE Test Problems: a MATLAB suite of initial value problems

Authors: Steven Roberts, Andrey A. Popov, Adrian Sandu

Abstract: ODE Test Problems (OTP) is an object-oriented MATLAB package offering a broad range of initial value problems which can be used to test numerical methods such as time integration methods and data assimilation (DA) methods. It includes problems that are linear and nonlinear, homogeneous and nonhomogeneous, autonomous and nonautonomous, scalar and high-dimensional, stiff and nonstiff, and chaotic an… ▽ More ODE Test Problems (OTP) is an object-oriented MATLAB package offering a broad range of initial value problems which can be used to test numerical methods such as time integration methods and data assimilation (DA) methods. It includes problems that are linear and nonlinear, homogeneous and nonhomogeneous, autonomous and nonautonomous, scalar and high-dimensional, stiff and nonstiff, and chaotic and nonchaotic. Many are real-world problems from fields such as chemistry, astrophysics, meteorology, and electrical engineering. OTP also supports partitioned ODEs for testing IMEX methods, multirate methods, and other multimethods. Functions for plotting solutions and creating movies are available for all problems, and exact solutions are provided when available. OTP is desgined for ease of use-meaning that working with and modifying problems is simple and intuitive. △ Less

Submitted 13 January, 2019; originally announced January 2019.

Report number: CSL-TR-19-1

A Bayesian Approach to Multivariate Adaptive Localization in Ensemble-Based Data Assimilation with Time-Dependent Extensions

Authors: Andrey A Popov, Adrian Sandu

Abstract: Ever since its inception, the Ensemble Kalman Filter has elicited many heuristic methods that sought to correct it. One such method is localization---the thought that `nearby' variables should be highly correlated with `far away' variable not. Recognizing that correlation is a time-dependent property, adaptive localization is a natural extension to these heuristics. We propose a Bayesian approach… ▽ More Ever since its inception, the Ensemble Kalman Filter has elicited many heuristic methods that sought to correct it. One such method is localization---the thought that `nearby' variables should be highly correlated with `far away' variable not. Recognizing that correlation is a time-dependent property, adaptive localization is a natural extension to these heuristics. We propose a Bayesian approach to adaptive Schur-product localization for the DEnKF, and extend it to support multiple radii of influence. We test both the empirical validity of (multivariate) adaptive localization, and of our approach. We test a simple toy problem (Lorenz'96), extending it to a multivariate model, and a more realistic geophysical problem (1.5 Layer Quasi-Geostrophic). We show that the multivariate approach has great promise on the toy problem, and that the univariate approach leads to improved filter performance for the realistic geophysical problem. △ Less

Submitted 24 September, 2018; originally announced September 2018.

Report number: CSL-TR-18-6

A Machine Learning Approach to Adaptive Covariance Localization

Authors: Azam Moosavi, Ahmed Attia, Adrian Sandu

Abstract: Data assimilation plays a key role in large-scale atmospheric weather forecasting, where the state of the physical system is estimated from model outputs and observations, and is then used as initial condition to produce accurate future forecasts. The Ensemble Kalman Filter (EnKF) provides a practical implementation of the statistical solution of the data assimilation problem and has gained wide p… ▽ More Data assimilation plays a key role in large-scale atmospheric weather forecasting, where the state of the physical system is estimated from model outputs and observations, and is then used as initial condition to produce accurate future forecasts. The Ensemble Kalman Filter (EnKF) provides a practical implementation of the statistical solution of the data assimilation problem and has gained wide popularity as. This success can be attributed to its simple formulation and ease of implementation. EnKF is a Monte-Carlo algorithm that solves the data assimilation problem by sampling the probability distributions involved in Bayes theorem. Because of this, all flavors of EnKF are fundamentally prone to sampling errors when the ensemble size is small. In typical weather forecasting applications, the model state space has dimension $10^{9}-10^{12}$, while the ensemble size typically ranges between $30-100$ members. Sampling errors manifest themselves as long-range spurious correlations and have been shown to cause filter divergence. To alleviate this effect covariance localization dampens spurious correlations between state variables located at a large distance in the physical space, via an empirical distance-dependent function. The quality of the resulting analysis and forecast is greatly influenced by the choice of the localization function parameters, e.g., the radius of influence. The localization radius is generally tuned empirically to yield desirable results.This work, proposes two adaptive algorithms for covariance localization in the EnKF framework, both based on a machine learning approach. The first algorithm adapts the localization radius in time, while the second algorithm tunes the localization radius in both time and space. Numerical experiments carried out with the Lorenz-96 model, and a quasi-geostrophic model, reveal the potential of the proposed machine learning approaches. △ Less

Submitted 10 February, 2018; v1 submitted 1 January, 2018; originally announced January 2018.

Comments: 23 pages, 12 figures

Report number: CSTR-01

DATeS: A Highly-Extensible Data Assimilation Testing Suite v1.0

Authors: Ahmed Attia, Adrian Sandu

Abstract: A flexible and highly-extensible data assimilation testing suite, named DATeS, is described in this paper. DATeS aims to offer a unified testing environment that allows researchers to compare different data assimilation methodologies and understand their performance in various settings. The core of DATeS is implemented in Python and takes advantage of its object-oriented capabilities. The main com… ▽ More A flexible and highly-extensible data assimilation testing suite, named DATeS, is described in this paper. DATeS aims to offer a unified testing environment that allows researchers to compare different data assimilation methodologies and understand their performance in various settings. The core of DATeS is implemented in Python and takes advantage of its object-oriented capabilities. The main components of the package (the numerical models, the data assimilation algorithms, the linear algebra solvers, and the time discretization routines) are independent of each other, which offers great flexibility to configure data assimilation applications. DATeS can interface easily with large third-party numerical models written in Fortran or in C, and with a plethora of external solvers. △ Less

Submitted 1 July, 2018; v1 submitted 18 April, 2017; originally announced April 2017.

Report number: CSTR-5/2017

A Numerical Investigation of Matrix-Free Implicit Time-Step** Methods for Large CFD Simulations

Authors: Arash Sarshar, Paul Tranquilli, Brent Pickering, Andrew McCall, Adrian Sandu, Christopher J. Roy

Abstract: This paper is concerned with the development and testing of advanced time-step** methods suited for the integration of time-accurate, real-world applications of computational fluid dynamics (CFD). The performance of several time discretization methods is studied numerically with regards to computational efficiency, order of accuracy, and stability, as well as the ability to treat effectively sti… ▽ More This paper is concerned with the development and testing of advanced time-step** methods suited for the integration of time-accurate, real-world applications of computational fluid dynamics (CFD). The performance of several time discretization methods is studied numerically with regards to computational efficiency, order of accuracy, and stability, as well as the ability to treat effectively stiff problems. We consider matrix-free implementations, a popular approach for time-step** methods applied to large CFD applications due to its adherence to scalable matrix-vector operations and a small memory footprint. We compare explicit methods with matrix-free implementations of implicit, linearly-implicit, as well as Rosenbrock-Krylov methods. We show that Rosenbrock-Krylov methods are competitive with existing techniques excelling for a number of problem types and settings. △ Less

Submitted 30 September, 2017; v1 submitted 22 July, 2016; originally announced July 2016.

Report number: Computational Science Lab CSL-TR-16-6 MSC Class: 65L05; 65L06; 65L20

Journal ref: Computers & Fluids, Volume 159, 15 Dec. 2017, PP. 53-63

Efficient Construction of Local Parametric Reduced Order Models Using Machine Learning Techniques

Authors: Azam Moosavi, Razvan Stefanescu, Adrian Sandu

Abstract: Reduced order models are computationally inexpensive approximations that capture the important dynamical characteristics of large, high-fidelity computer models of physical systems. This paper applies machine learning techniques to improve the design of parametric reduced order models. Specifically, machine learning is used to develop feasible regions in the parameter space where the admissible ta… ▽ More Reduced order models are computationally inexpensive approximations that capture the important dynamical characteristics of large, high-fidelity computer models of physical systems. This paper applies machine learning techniques to improve the design of parametric reduced order models. Specifically, machine learning is used to develop feasible regions in the parameter space where the admissible target accuracy is achieved with a predefined reduced order basis, to construct parametric maps, to chose the best two already existing bases for a new parameter configuration from accuracy point of view and to pre-select the optimal dimension of the reduced basis such as to meet the desired accuracy. By combining available information using bases concatenation and interpolation as well as high-fidelity solutions interpolation we are able to build accurate reduced order models associated with new parameter settings. Promising numerical results with a viscous Burgers model illustrate the potential of machine learning approaches to help design better reduced order models. △ Less

Submitted 9 November, 2015; originally announced November 2015.

Comments: 28 pages, 15 figures, 6 tables

Application of approximate matrix factorization to high order linearly implicit Runge-Kutta methods

Authors: Hong Zhang, Adrian Sandu, Paul Tranquilli

Abstract: Linearly implicit Runge-Kutta methods with approximate matrix factorization can solve efficiently large systems of differential equations that have a stiff linear part, e.g. reaction-diffusion systems. However, the use of approximate factorization usually leads to loss of accuracy, which makes it attractive only for low order time integration schemes. This paper discusses the application of approx… ▽ More Linearly implicit Runge-Kutta methods with approximate matrix factorization can solve efficiently large systems of differential equations that have a stiff linear part, e.g. reaction-diffusion systems. However, the use of approximate factorization usually leads to loss of accuracy, which makes it attractive only for low order time integration schemes. This paper discusses the application of approximate matrix factorization with high order methods; an inexpensive correction procedure applied to each stage allows to retain the high order of the underlying linearly implicit Runge-Kutta scheme. The accuracy and stability of the methods are studied. Numerical experiments on reaction-diffusion type problems of different sizes and with different degrees of stiffness illustrate the efficiency of the proposed approach. △ Less

Submitted 17 August, 2014; v1 submitted 15 August, 2014; originally announced August 2014.

Optimization of Vehicle Dynamics based on Multibody Models using Adjoint Sensitivity Analysis

Authors: Yitao Zhu, Corina Sandu, Daniel Dopico, Adrian Sandu

Abstract: Multibody dynamics simulations have become widely used tools for vehicle systems analysis and design. As this approach evolves, it becomes able to provide additional information for various types of analyses. One very important direction is the optimization of multibody systems. Sensitivity analysis of multibody system dynamics is essential for design optimization. Dynamic sensitivities, when need… ▽ More Multibody dynamics simulations have become widely used tools for vehicle systems analysis and design. As this approach evolves, it becomes able to provide additional information for various types of analyses. One very important direction is the optimization of multibody systems. Sensitivity analysis of multibody system dynamics is essential for design optimization. Dynamic sensitivities, when needed, are often calculated by means of finite differences. However, depending of the number of parameters involved, this procedure can be computationally expensive. Moreover, in many cases the results suffer from low accuracy when real perturbations are used. This paper develops the adjoint sensitivity analysis of multibody systems in the context of penalty formulations. The resulting sensitivities are applied to perform dynamical optimization of a full vehicle system. △ Less

Submitted 3 November, 2014; v1 submitted 20 May, 2014; originally announced May 2014.

Comments: I tried to replace this paper with a new one which has corrected several errors in this paper. However, I didn't know how to replace it at that time, I submitted a new one "Dynamic Response Optimization of Complex Multibody Systems in a Penalty Formulation using Adjoint Sensitivity", the identifier is arXiv:1410.8422. Since I have already submitted that one, I want to withdraw this one

A Sampling Filter for Non-Gaussian Data Assimilation

Authors: Ahmed Attia, Adrian Sandu

Abstract: Data assimilation combines information from models, measurements, and priors to estimate the state of a dynamical system such as the atmosphere. The Ensemble Kalman filter (EnKF) is a family of ensemble-based data assimilation approaches that has gained wide popularity due its simple formulation, ease of implementation, and good practical results. Most EnKF algorithms assume that the underlying pr… ▽ More Data assimilation combines information from models, measurements, and priors to estimate the state of a dynamical system such as the atmosphere. The Ensemble Kalman filter (EnKF) is a family of ensemble-based data assimilation approaches that has gained wide popularity due its simple formulation, ease of implementation, and good practical results. Most EnKF algorithms assume that the underlying probability distributions are Gaussian. Although this assumption is well accepted, it is too restrictive when applied to large nonlinear models, nonlinear observation operators, and large levels of uncertainty. Several approaches have been proposed in order to avoid the Gaussianity assumption. One of the most successful strategies is the maximum likelihood ensemble filter (MLEF) which computes a maximum a posteriori estimate of the state assuming the posterior distribution is Gaussian. MLEF is designed to work with nonlinear and even non-differentiable observation operators, and shows good practical performance. However, there are limits to the degree of nonlinearity that MLEF can handle. This paper proposes a new ensemble-based data assimilation method, named the "sampling filter", which obtains the analysis by sampling directly from the posterior distribution. The sampling strategy is based on a Hybrid Monte Carlo (HMC) approach that can handle non-Gaussian probability distributions. Numerical experiments are carried out using the Lorenz-96 model and observation operators with different levels of non-linearity and differentiability. The proposed filter is also tested with shallow water model on a sphere with linear observation operator. The results show that the sampling filter can perform well even in highly nonlinear situations were EnKF and MLEF filters diverge. △ Less

Submitted 5 December, 2014; v1 submitted 27 March, 2014; originally announced March 2014.

Comments: 52 pages, 24 figures, 4 tables

Report number: CSTR-4/2014

An Optimization Framework to Improve 4D-Var Data Assimilation System Performance

Authors: Alexandru Cioaca, Adrian Sandu

Abstract: This paper develops a computational framework for optimizing the parameters of data assimilation systems in order to improve their performance. The approach formulates a continuous meta-optimization problem for parameters; the meta-optimization is constrained by the original data assimilation problem. The numerical solution process employs adjoint models and iterative solvers. The proposed framewo… ▽ More This paper develops a computational framework for optimizing the parameters of data assimilation systems in order to improve their performance. The approach formulates a continuous meta-optimization problem for parameters; the meta-optimization is constrained by the original data assimilation problem. The numerical solution process employs adjoint models and iterative solvers. The proposed framework is applied to optimize observation values, data weighting coefficients, and the location of sensors for a test problem. The ability to optimize a distributed measurement network is crucial for cutting down operating costs and detecting malfunctions. △ Less

Submitted 9 November, 2013; v1 submitted 8 August, 2013; originally announced August 2013.

Report number: CSL-TR-4-2013

Low-rank Approximations for Computing Observation Impact in 4D-Var Data Assimilation

Authors: Alexandru Cioaca, Adrian Sandu

Abstract: We present an efficient computational framework to quantify the impact of individual observations in four dimensional variational data assimilation. The proposed methodology uses first and second order adjoint sensitivity analysis, together with matrix-free algorithms to obtain low-rank approximations of ob- servation impact matrix. We illustrate the application of this methodology to important ap… ▽ More We present an efficient computational framework to quantify the impact of individual observations in four dimensional variational data assimilation. The proposed methodology uses first and second order adjoint sensitivity analysis, together with matrix-free algorithms to obtain low-rank approximations of ob- servation impact matrix. We illustrate the application of this methodology to important applications such as data pruning and the identification of faulty sensors for a two dimensional shallow water test system. △ Less

Submitted 18 July, 2013; originally announced July 2013.

Report number: CSL-TR-3-2013

Efficient methods for computing observation impact in 4D-Var data assimilation

Authors: Alexandru Cioaca, Adrian Sandu, Eric de Sturler

Abstract: This paper presents a practical computational approach to quantify the effect of individual observations in estimating the state of a system. Such an analysis can be used for pruning redundant measurements, and for designing future sensor networks. The mathematical approach is based on computing the sensitivity of the reanalysis (unconstrained optimization solution) with respect to the data. The c… ▽ More This paper presents a practical computational approach to quantify the effect of individual observations in estimating the state of a system. Such an analysis can be used for pruning redundant measurements, and for designing future sensor networks. The mathematical approach is based on computing the sensitivity of the reanalysis (unconstrained optimization solution) with respect to the data. The computational cost is dominated by the solution of a linear system, whose matrix is the Hessian of the cost function, and is only available in operator form. The right hand side is the gradient of a scalar cost function that quantifies the forecast error of the numerical model. The use of adjoint models to obtain the necessary first and second order derivatives is discussed. We study various strategies to accelerate the computation, including matrix-free iterative solvers, preconditioners, and an in-house multigrid solver. Experiments are conducted on both a small-size shallow-water equations model, and on a large-scale numerical weather prediction model, in order to illustrate the capabilities of the new methodology. △ Less

Submitted 18 July, 2013; v1 submitted 24 May, 2013; originally announced May 2013.

Report number: CSL-TR-2-2013

Implicit Simulation Methods for Stochastic Chemical Kinetics

Authors: Tae-Hyuk Ahn, Adrian Sandu, Xiaoying Han

Abstract: In biochemical systems some of the chemical species are present with only small numbers of molecules. In this situation discrete and stochastic simulation approaches are more relevant than continuous and deterministic ones. The fundamental Gillespie's stochastic simulation algorithm (SSA) accounts for every reaction event, which occurs with a probability determined by the configuration of the syst… ▽ More In biochemical systems some of the chemical species are present with only small numbers of molecules. In this situation discrete and stochastic simulation approaches are more relevant than continuous and deterministic ones. The fundamental Gillespie's stochastic simulation algorithm (SSA) accounts for every reaction event, which occurs with a probability determined by the configuration of the system. This approach requires a considerable computational effort for models with many reaction channels and chemical species. In order to improve efficiency, tau-lea** methods represent multiple firings of each reaction during a simulation step by Poisson random variables. For stiff systems the mean of this variable is treated implicitly in order to ensure numerical stability. This paper develops fully implicit tau-lea**-like algorithms that treat implicitly both the mean and the variance of the Poisson variables. The construction is based on adapting weakly convergent discretizations of stochastic differential equations to stochastic chemical kinetic systems. Theoretical analyses of accuracy and stability of the new methods are performed on a standard test problem. Numerical results demonstrate the performance of the proposed tau-lea** methods. △ Less

Submitted 14 March, 2013; originally announced March 2013.

Comments: Prepare submission