Showing 1–50 of 225 results for author: Kevrekidis, I

Active search for Bifurcations

Authors: Yorgos M. Psarellis, Themistoklis P. Sapsis, Ioannis G. Kevrekidis

Abstract: Bifurcations mark qualitative changes of long-term behavior in dynamical systems and can often signal sudden ("hard") transitions or catastrophic events (divergences). Accurately locating them is critical not just for deeper understanding of observed dynamic behavior, but also for designing efficient interventions. When the dynamical system at hand is complex, possibly noisy, and expensive to samp… ▽ More Bifurcations mark qualitative changes of long-term behavior in dynamical systems and can often signal sudden ("hard") transitions or catastrophic events (divergences). Accurately locating them is critical not just for deeper understanding of observed dynamic behavior, but also for designing efficient interventions. When the dynamical system at hand is complex, possibly noisy, and expensive to sample, standard (e.g. continuation based) numerical methods may become impractical. We propose an active learning framework, where Bayesian Optimization is leveraged to discover saddle-node or Hopf bifurcations, from a judiciously chosen small number of vector field observations. Such an approach becomes especially attractive in systems whose state x parameter space exploration is resource-limited. It also naturally provides a framework for uncertainty quantification (aleatoric and epistemic), useful in systems with inherent stochasticity. △ Less

Submitted 16 June, 2024; originally announced June 2024.

Comments: 27 pages, 7 figures

MSC Class: 37M20

On Learning what to Learn: heterogeneous observations of dynamics and establishing (possibly causal) relations among them

Authors: David W. Sroczynski, Felix Dietrich, Eleni D. Koronaki, Ronen Talmon, Ronald R. Coifman, Erik Bollt, Ioannis G. Kevrekidis

Abstract: Before we attempt to learn a function between two (sets of) observables of a physical process, we must first decide what the inputs and what the outputs of the desired function are going to be. Here we demonstrate two distinct, data-driven ways of initially deciding ``the right quantities'' to relate through such a function, and then proceed to learn it. This is accomplished by processing multiple… ▽ More Before we attempt to learn a function between two (sets of) observables of a physical process, we must first decide what the inputs and what the outputs of the desired function are going to be. Here we demonstrate two distinct, data-driven ways of initially deciding ``the right quantities'' to relate through such a function, and then proceed to learn it. This is accomplished by processing multiple simultaneous heterogeneous data streams (ensembles of time series) from observations of a physical system: multiple observation processes of the system. We thus determine (a) what subsets of observables are common between the observation processes (and therefore observable from each other, relatable through a function); and (b) what information is unrelated to these common observables, and therefore particular to each observation process, and not contributing to the desired function. Any data-driven function approximation technique can subsequently be used to learn the input-output relation, from k-nearest neighbors and Geometric Harmonics to Gaussian Processes and Neural Networks. Two particular ``twists'' of the approach are discussed. The first has to do with the identifiability of particular quantities of interest from the measurements. We now construct map**s from a single set of observations of one process to entire level sets of measurements of the process, consistent with this single set. The second attempts to relate our framework to a form of causality: if one of the observation processes measures ``now'', while the second observation process measures ``in the future'', the function to be learned among what is common across observation processes constitutes a dynamical model for the system evolution. △ Less

Submitted 10 June, 2024; originally announced June 2024.

RandONet: Shallow-Networks with Random Projections for learning linear and nonlinear operators

Authors: Gianluca Fabiani, Ioannis G. Kevrekidis, Constantinos Siettos, Athanasios N. Yannacopoulos

Abstract: Deep Operator Networks (DeepOnets) have revolutionized the domain of scientific machine learning for the solution of the inverse problem for dynamical systems. However, their implementation necessitates optimizing a high-dimensional space of parameters and hyperparameters. This fact, along with the requirement of substantial computational resources, poses a barrier to achieving high numerical accu… ▽ More Deep Operator Networks (DeepOnets) have revolutionized the domain of scientific machine learning for the solution of the inverse problem for dynamical systems. However, their implementation necessitates optimizing a high-dimensional space of parameters and hyperparameters. This fact, along with the requirement of substantial computational resources, poses a barrier to achieving high numerical accuracy. Here, inpsired by DeepONets and to address the above challenges, we present Random Projection-based Operator Networks (RandONets): shallow networks with random projections that learn linear and nonlinear operators. The implementation of RandONets involves: (a) incorporating random bases, thus enabling the use of shallow neural networks with a single hidden layer, where the only unknowns are the output weights of the network's weighted inner product; this reduces dramatically the dimensionality of the parameter space; and, based on this, (b) using established least-squares solvers (e.g., Tikhonov regularization and preconditioned QR decomposition) that offer superior numerical approximation properties compared to other optimization techniques used in deep-learning. In this work, we prove the universal approximation accuracy of RandONets for approximating nonlinear operators and demonstrate their efficiency in approximating linear nonlinear evolution operators (right-hand-sides (RHS)) with a focus on PDEs. We show, that for this particular task, RandONets outperform, both in terms of numerical approximation accuracy and computational cost, the ``vanilla" DeepOnets. △ Less

Submitted 8 June, 2024; originally announced June 2024.

MSC Class: 65M32; 65D12; 65J22; 41A35; 68T20; 65D15; 68T07; 68W20; 41A35

Integrating supervised and unsupervised learning approaches to unveil critical process inputs

Authors: Paris Papavasileiou, Dimitrios G. Giovanis, Gabriele Pozzetti, Martin Kathrein, Christoph Czettl, Ioannis G. Kevrekidis, Andreas G. Boudouvis, Stéphane P. A. Bordas, Eleni D. Koronaki

Abstract: This study introduces a machine learning framework tailored to large-scale industrial processes characterized by a plethora of numerical and categorical inputs. The framework aims to (i) discern critical parameters influencing the output and (ii) generate accurate out-of-sample qualitative and quantitative predictions of production outcomes. Specifically, we address the pivotal question of the sig… ▽ More This study introduces a machine learning framework tailored to large-scale industrial processes characterized by a plethora of numerical and categorical inputs. The framework aims to (i) discern critical parameters influencing the output and (ii) generate accurate out-of-sample qualitative and quantitative predictions of production outcomes. Specifically, we address the pivotal question of the significance of each input in sha** the process outcome, using an industrial Chemical Vapor Deposition (CVD) process as an example. The initial objective involves merging subject matter expertise and clustering techniques exclusively on the process output, here, coating thickness measurements at various positions in the reactor. This approach identifies groups of production runs that share similar qualitative characteristics, such as film mean thickness and standard deviation. In particular, the differences of the outcomes represented by the different clusters can be attributed to differences in specific inputs, indicating that these inputs are critical for the production outcome. Leveraging this insight, we subsequently implement supervised classification and regression methods using the identified critical process inputs. The proposed methodology proves to be valuable in scenarios with a multitude of inputs and insufficient data for the direct application of deep learning techniques, providing meaningful insights into the underlying processes. △ Less

Submitted 13 May, 2024; originally announced May 2024.

Nonlinear Manifold Learning Determines Microgel Size from Raman Spectroscopy

Authors: Eleni D. Koronaki, Luise F. Kaven, Johannes M. M. Faust, Ioannis G. Kevrekidis, Alexander Mitsos

Abstract: Polymer particle size constitutes a crucial characteristic of product quality in polymerization. Raman spectroscopy is an established and reliable process analytical technology for in-line concentration monitoring. Recent approaches and some theoretical considerations show a correlation between Raman signals and particle sizes but do not determine polymer size from Raman spectroscopic measurements… ▽ More Polymer particle size constitutes a crucial characteristic of product quality in polymerization. Raman spectroscopy is an established and reliable process analytical technology for in-line concentration monitoring. Recent approaches and some theoretical considerations show a correlation between Raman signals and particle sizes but do not determine polymer size from Raman spectroscopic measurements accurately and reliably. With this in mind, we propose three alternative machine learning workflows to perform this task, all involving diffusion maps, a nonlinear manifold learning technique for dimensionality reduction: (i) directly from diffusion maps, (ii) alternating diffusion maps, and (iii) conformal autoencoder neural networks. We apply the workflows to a data set of Raman spectra with associated size measured via dynamic light scattering of 47 microgel (cross-linked polymer) samples in a diameter range of 208nm to 483 nm. The conformal autoencoders substantially outperform state-of-the-art methods and results for the first time in a promising prediction of polymer size from Raman spectra. △ Less

Submitted 13 March, 2024; originally announced March 2024.

Comments: 51 pages, 12 figures, 4 tables

Intelligent Attractors for Singularly Perturbed Dynamical Systems

Authors: Daniel A. Serino, Allen Alvarez Loya, J. W. Burby, Ioannis G. Kevrekidis, Qi Tang

Abstract: Singularly perturbed dynamical systems, commonly known as fast-slow systems, play a crucial role in various applications such as plasma physics. They are closely related to reduced order modeling, closures, and structure-preserving numerical algorithms for multiscale modeling. A powerful and well-known tool to address these systems is the Fenichel normal form, which significantly simplifies fast d… ▽ More Singularly perturbed dynamical systems, commonly known as fast-slow systems, play a crucial role in various applications such as plasma physics. They are closely related to reduced order modeling, closures, and structure-preserving numerical algorithms for multiscale modeling. A powerful and well-known tool to address these systems is the Fenichel normal form, which significantly simplifies fast dynamics near slow manifolds through a transformation. However, the Fenichel normal form is difficult to realize in conventional numerical algorithms. In this work, we explore an alternative way of realizing it through structure-preserving machine learning. Specifically, a fast-slow neural network (FSNN) is proposed for learning data-driven models of singularly perturbed dynamical systems with dissipative fast timescale dynamics. Our method enforces the existence of a trainable, attracting invariant slow manifold as a hard constraint. Closed-form representation of the slow manifold enables efficient integration on the slow time scale and significantly improves prediction accuracy beyond the training data. We demonstrate the FSNN on several examples that exhibit multiple timescales, including the Grad moment system from hydrodynamics, two-scale Lorentz96 equations for modeling atmospheric dynamics, and Abraham-Lorentz dynamics modeling radiation reaction of electrons in a magnetic field. △ Less

Submitted 24 February, 2024; originally announced February 2024.

MSC Class: 37M21

Nonlinear Discrete-Time Observers with Physics-Informed Neural Networks

Authors: Hector Vargas Alvarez, Gianluca Fabiani, Ioannis G. Kevrekidis, Nikolaos Kazantzis, Constantinos Siettos

Abstract: We use Physics-Informed Neural Networks (PINNs) to solve the discrete-time nonlinear observer state estimation problem. Integrated within a single-step exact observer linearization framework, the proposed PINN approach aims at learning a nonlinear state transformation map by solving a system of inhomogeneous functional equations. The performance of the proposed PINN approach is assessed via two il… ▽ More We use Physics-Informed Neural Networks (PINNs) to solve the discrete-time nonlinear observer state estimation problem. Integrated within a single-step exact observer linearization framework, the proposed PINN approach aims at learning a nonlinear state transformation map by solving a system of inhomogeneous functional equations. The performance of the proposed PINN approach is assessed via two illustrative case studies for which the observer linearizing transformation map can be derived analytically. We also perform an uncertainty quantification analysis for the proposed PINN scheme and we compare it with conventional power-series numerical implementations, which rely on the computation of a power series solution. △ Less

Submitted 19 February, 2024; originally announced February 2024.

MSC Class: 37N30; 68T05; 93C55; 65D15

Polynomial Chaos Expansions on Principal Geodesic Grassmannian Submanifolds for Surrogate Modeling and Uncertainty Quantification

Authors: Dimitris G. Giovanis, Dimitrios Loukrezis, Ioannis G. Kevrekidis, Michael D. Shields

Abstract: In this work we introduce a manifold learning-based surrogate modeling framework for uncertainty quantification in high-dimensional stochastic systems. Our first goal is to perform data mining on the available simulation data to identify a set of low-dimensional (latent) descriptors that efficiently parameterize the response of the high-dimensional computational model. To this end, we employ Princ… ▽ More In this work we introduce a manifold learning-based surrogate modeling framework for uncertainty quantification in high-dimensional stochastic systems. Our first goal is to perform data mining on the available simulation data to identify a set of low-dimensional (latent) descriptors that efficiently parameterize the response of the high-dimensional computational model. To this end, we employ Principal Geodesic Analysis on the Grassmann manifold of the response to identify a set of disjoint principal geodesic submanifolds, of possibly different dimension, that captures the variation in the data. Since operations on the Grassmann require the data to be concentrated, we propose an adaptive algorithm based on Riemanniann K-means and the minimization of the sample Frechet variance on the Grassmann manifold to identify "local" principal geodesic submanifolds that represent different system behavior across the parameter space. Polynomial chaos expansion is then used to construct a map** between the random input parameters and the projection of the response on these local principal geodesic submanifolds. The method is demonstrated on four test cases, a toy-example that involves points on a hypersphere, a Lotka-Volterra dynamical system, a continuous-flow stirred-tank chemical reactor system, and a two-dimensional Rayleigh-Benard convection problem △ Less

Submitted 29 January, 2024; originally announced January 2024.

Comments: 50 pages, 17 figures

AI-Lorenz: A physics-data-driven framework for black-box and gray-box identification of chaotic systems with symbolic regression

Authors: Mario De Florio, Ioannis G. Kevrekidis, George Em Karniadakis

Abstract: Discovering mathematical models that characterize the observed behavior of dynamical systems remains a major challenge, especially for systems in a chaotic regime. The challenge is even greater when the physics underlying such systems is not yet understood, and scientific inquiry must solely rely on empirical data. Driven by the need to fill this gap, we develop a framework that learns mathematica… ▽ More Discovering mathematical models that characterize the observed behavior of dynamical systems remains a major challenge, especially for systems in a chaotic regime. The challenge is even greater when the physics underlying such systems is not yet understood, and scientific inquiry must solely rely on empirical data. Driven by the need to fill this gap, we develop a framework that learns mathematical expressions modeling complex dynamical behaviors by identifying differential equations from noisy and sparse observable data. We train a small neural network to learn the dynamics of a system, its rate of change in time, and missing model terms, which are used as input for a symbolic regression algorithm to autonomously distill the explicit mathematical terms. This, in turn, enables us to predict the future evolution of the dynamical behavior. The performance of this framework is validated by recovering the right-hand sides and unknown terms of certain complex, chaotic systems such as the well-known Lorenz system, a six-dimensional hyperchaotic system, and the non-autonomous Sprott chaotic system, and comparing them with their known analytical expressions. △ Less

Submitted 21 December, 2023; originally announced December 2023.

Comments: 28 pages, 15 figures, 9 tables

MSC Class: 34A34; 34A55; 70K55 ACM Class: J.2; G.1.7; I.2.0

Gappy local conformal auto-encoders for heterogeneous data fusion: in praise of rigidity

Authors: Erez Peterfreund, Iryna Burak, Ofir Lindenbaum, Jim Gimlett, Felix Dietrich, Ronald R. Coifman, Ioannis G. Kevrekidis

Abstract: Fusing measurements from multiple, heterogeneous, partial sources, observing a common object or process, poses challenges due to the increasing availability of numbers and types of sensors. In this work we propose, implement and validate an end-to-end computational pipeline in the form of a multiple-auto-encoder neural network architecture for this task. The inputs to the pipeline are several sets… ▽ More Fusing measurements from multiple, heterogeneous, partial sources, observing a common object or process, poses challenges due to the increasing availability of numbers and types of sensors. In this work we propose, implement and validate an end-to-end computational pipeline in the form of a multiple-auto-encoder neural network architecture for this task. The inputs to the pipeline are several sets of partial observations, and the result is a globally consistent latent space, harmonizing (rigidifying, fusing) all measurements. The key enabler is the availability of multiple slightly perturbed measurements of each instance:, local measurement, "bursts", that allows us to estimate the local distortion induced by each instrument. We demonstrate the approach in a sequence of examples, starting with simple two-dimensional data sets and proceeding to a Wi-Fi localization problem and to the solution of a "dynamical puzzle" arising in spatio-temporal observations of the solutions of Partial Differential Equations. △ Less

Submitted 20 December, 2023; originally announced December 2023.

Micro-Macro Consistency in Multiscale Modeling: Score-Based Model Assisted Sampling of Fast/Slow Dynamical Systems

Authors: Ellis R. Crabtree, Juan M. Bello-Rivas, Ioannis G. Kevrekidis

Abstract: A valuable step in the modeling of multiscale dynamical systems in fields such as computational chemistry, biology, materials science and more, is the representative sampling of the phase space over long timescales of interest; this task is not, however, without challenges. For example, the long term behavior of a system with many degrees of freedom often cannot be efficiently computationally expl… ▽ More A valuable step in the modeling of multiscale dynamical systems in fields such as computational chemistry, biology, materials science and more, is the representative sampling of the phase space over long timescales of interest; this task is not, however, without challenges. For example, the long term behavior of a system with many degrees of freedom often cannot be efficiently computationally explored by direct dynamical simulation; such systems can often become trapped in local free energy minima. In the study of physics-based multi-time-scale dynamical systems, techniques have been developed for enhancing sampling in order to accelerate exploration beyond free energy barriers. On the other hand, in the field of Machine Learning, a generic goal of generative models is to sample from a target density, after training on empirical samples from this density. Score based generative models (SGMs) have demonstrated state-of-the-art capabilities in generating plausible data from target training distributions. Conditional implementations of such generative models have been shown to exhibit significant parallels with long-established -- and physics based -- solutions to enhanced sampling. These physics-based methods can then be enhanced through coupling with the ML generative models, complementing the strengths and mitigating the weaknesses of each technique. In this work, we show that that SGMs can be used in such a coupling framework to improve sampling in multiscale dynamical systems. △ Less

Submitted 27 December, 2023; v1 submitted 9 December, 2023; originally announced December 2023.

Comments: 20 pages, 9 figures

MSC Class: 82C32 (Primary) 37M05; 60H10; 68T07; 82-08 (Secondary)

Tip** Points of Evolving Epidemiological Networks: Machine Learning-Assisted, Data-Driven Effective Modeling

Authors: Nikolaos Evangelou, Tianqi Cui, Juan M. Bello-Rivas, Alexei Makeev, Ioannis G. Kevrekidis

Abstract: We study the tip** point collective dynamics of an adaptive susceptible-infected-susceptible (SIS) epidemiological network in a data-driven, machine learning-assisted manner. We identify a parameter-dependent effective stochastic differential equation (eSDE) in terms of physically meaningful coarse mean-field variables through a deep-learning ResNet architecture inspired by numerical stochastic… ▽ More We study the tip** point collective dynamics of an adaptive susceptible-infected-susceptible (SIS) epidemiological network in a data-driven, machine learning-assisted manner. We identify a parameter-dependent effective stochastic differential equation (eSDE) in terms of physically meaningful coarse mean-field variables through a deep-learning ResNet architecture inspired by numerical stochastic integrators. We construct an approximate effective bifurcation diagram based on the identified drift term of the eSDE and contrast it with the mean-field SIS model bifurcation diagram. We observe a subcritical Hopf bifurcation in the evolving network's effective SIS dynamics, that causes the tip** point behavior; this takes the form of large amplitude collective oscillations that spontaneously -- yet rarely -- arise from the neighborhood of a (noisy) stationary state. We study the statistics of these rare events both through repeated brute force simulations and by using established mathematical/computational tools exploiting the right-hand-side of the identified SDE. We demonstrate that such a collective SDE can also be identified (and the rare events computations also performed) in terms of data-driven coarse observables, obtained here via manifold learning techniques, in particular Diffusion Maps. The workflow of our study is straightforwardly applicable to other complex dynamics problems exhibiting tip** point dynamics. △ Less

Submitted 10 November, 2023; v1 submitted 1 November, 2023; originally announced November 2023.

Comments: 22 pages, 12 figures

Machine Learning for the identification of phase-transitions in interacting agent-based systems

Authors: Nikolaos Evangelou, Dimitrios G. Giovanis, George A. Kevrekidis, Grigorios A. Pavliotis, Ioannis G. Kevrekidis

Abstract: Deriving closed-form, analytical expressions for reduced-order models, and judiciously choosing the closures leading to them, has long been the strategy of choice for studying phase- and noise-induced transitions for agent-based models (ABMs). In this paper, we propose a data-driven framework that pinpoints phase transitions for an ABM in its mean-field limit, using a smaller number of variables t… ▽ More Deriving closed-form, analytical expressions for reduced-order models, and judiciously choosing the closures leading to them, has long been the strategy of choice for studying phase- and noise-induced transitions for agent-based models (ABMs). In this paper, we propose a data-driven framework that pinpoints phase transitions for an ABM in its mean-field limit, using a smaller number of variables than traditional closed-form models. To this end, we use the manifold learning algorithm Diffusion Maps to identify a parsimonious set of data-driven latent variables, and show that they are in one-to-one correspondence with the expected theoretical order parameter of the ABM. We then utilize a deep learning framework to obtain a conformal reparametrization of the data-driven coordinates that facilitates, in our example, the identification of a single parameter-dependent ODE in these coordinates. We identify this ODE through a residual neural network inspired by a numerical integration scheme (forward Euler). We then use the identified ODE -- enabled through an odd symmetry transformation -- to construct the bifurcation diagram exhibiting the phase transition. △ Less

Submitted 29 October, 2023; originally announced October 2023.

Comments: 14 pages, 9 Figures

Nonlinear dimensionality reduction then and now: AIMs for dissipative PDEs in the ML era

Authors: Eleni D. Koronaki, Nikolaos Evangelou, Cristina P. Martin-Linares, Edriss S. Titi, Ioannis G. Kevrekidis

Abstract: This study presents a collection of purely data-driven workflows for constructing reduced-order models (ROMs) for distributed dynamical systems. The ROMs we focus on, are data-assisted models inspired by, and templated upon, the theory of Approximate Inertial Manifolds (AIMs); the particular motivation is the so-called post-processing Galerkin method of Garcia-Archilla, Novo and Titi. Its applicab… ▽ More This study presents a collection of purely data-driven workflows for constructing reduced-order models (ROMs) for distributed dynamical systems. The ROMs we focus on, are data-assisted models inspired by, and templated upon, the theory of Approximate Inertial Manifolds (AIMs); the particular motivation is the so-called post-processing Galerkin method of Garcia-Archilla, Novo and Titi. Its applicability can be extended: the need for accurate truncated Galerkin projections and for deriving closed-formed corrections can be circumvented using machine learning tools. When the right latent variables are not a priori known, we illustrate how autoencoders as well as Diffusion Maps (a manifold learning scheme) can be used to discover good sets of latent variables and test their explainability. The proposed methodology can express the ROMs in terms of (a) theoretical (Fourier coefficients), (b) linear data-driven (POD modes) and/or (c) nonlinear data-driven (Diffusion Maps) coordinates. Both Black-Box and (theoretically-informed and data-corrected) Gray-Box models are described; the necessity for the latter arises when truncated Galerkin projections are so inaccurate as to not be amenable to post-processing. We use the Chafee-Infante reaction-diffusion and the Kuramoto-Sivashinsky dissipative partial differential equations to illustrate and successfully test the overall framework. △ Less

Submitted 24 October, 2023; originally announced October 2023.

Comments: 27 pages, 22 figures

Self-similar blow-up solutions in the generalized Korteweg-de Vries equation: Spectral analysis, normal form and asymptotics

Authors: S. Jon Chapman, M. Kavousanakis, E. G. Charalampidis, I. G. Kevrekidis, P. G. Kevrekidis

Abstract: In the present work we revisit the problem of the generalized Korteweg-de Vries equation parametrically, as a function of the relevant nonlinearity exponent, to examine the emergence of blow-up solutions, as traveling waveforms lose their stability past a critical point of the relevant parameter $p$, here at $p=5$. We provide a {\it normal form} of the associated collapse dynamics and illustrate h… ▽ More In the present work we revisit the problem of the generalized Korteweg-de Vries equation parametrically, as a function of the relevant nonlinearity exponent, to examine the emergence of blow-up solutions, as traveling waveforms lose their stability past a critical point of the relevant parameter $p$, here at $p=5$. We provide a {\it normal form} of the associated collapse dynamics and illustrate how this captures the collapsing branch bifurcating from the unstable traveling branch. We also systematically characterize the linearization spectrum of not only the traveling states, but importantly of the emergent collapsing waveforms in the so-called co-exploding frame where these waveforms are identified as stationary states. This spectrum, in addition to two positive real eigenvalues which are shown to be associated with the symmetries of translation and scaling invariance of the original (non-exploding) frame features complex patterns of negative eigenvalues that we also fully characterize. We show that the phenomenology of the latter is significantly affected by the boundary conditions and is far more complicated than in the corresponding symmetric Laplacian case of the nonlinear Schr{ö}dinger problem that has recently been explored. In addition, we explore the dynamics of the unstable solitary waves for $p>5$ in the co-exploding frame. △ Less

Submitted 20 October, 2023; originally announced October 2023.

Comments: 33 pages, 16 figures

Learning Parametric Koopman Decompositions for Prediction and Control

Authors: Yue Guo, Milan Korda, Ioannis G. Kevrekidis, Qianxiao Li

Abstract: We present an approach to construct approximate Koopman-type decompositions for dynamical systems depending on static or time-varying parameters. Our method simultaneously constructs an invariant subspace and a parametric family of projected Koopman operators acting on this subspace. We parametrize both the projected Koopman operator family and the dictionary that spans the invariant subspace by n… ▽ More We present an approach to construct approximate Koopman-type decompositions for dynamical systems depending on static or time-varying parameters. Our method simultaneously constructs an invariant subspace and a parametric family of projected Koopman operators acting on this subspace. We parametrize both the projected Koopman operator family and the dictionary that spans the invariant subspace by neural networks and jointly train them with trajectory data. We show theoretically the validity of our approach, and demonstrate via numerical experiments that it exhibits significant improvements over existing methods in solving prediction problems, especially those with large state or parameter dimensions, and those possessing strongly non-linear dynamics. Moreover, our method enables data-driven solution of optimal control problems involving non-linear dynamics, with interesting implications on controllability. △ Less

Submitted 2 October, 2023; originally announced October 2023.

MSC Class: 47N70; 37N35; 49M99

Locating saddle points using gradient extremals on manifolds adaptively revealed as point clouds

Authors: A. Georgiou, H. Vandecasteele, J. M. Bello-Rivas, I. Kevrekidis

Abstract: Steady states are invaluable in the study of dynamical systems. High-dimensional dynamical systems, due to a separation of time-scales, often evolve towards a lower dimensional manifold $M$. We introduce an approach to locate saddle points (and other fixed points) that utilizes gradient extremals on such a priori unknown (Riemannian) manifolds, defined by adaptively sampled point clouds, with loca… ▽ More Steady states are invaluable in the study of dynamical systems. High-dimensional dynamical systems, due to a separation of time-scales, often evolve towards a lower dimensional manifold $M$. We introduce an approach to locate saddle points (and other fixed points) that utilizes gradient extremals on such a priori unknown (Riemannian) manifolds, defined by adaptively sampled point clouds, with local coordinates discovered on-the-fly through manifold learning. The technique, which efficiently biases the dynamical system along a curve (as opposed to exhaustively exploring the state space), requires knowledge of a single minimum and the ability to sample around an arbitrary point. We demonstrate the effectiveness of the technique on the Müller-Brown potential mapped onto an unknown surface (namely, a sphere). Previous work employed a similar algorithmic framework to find saddle points using Newton trajectories and gentlest ascent dynamics; we therefore also offer a brief comparison with these methods. △ Less

Submitted 28 September, 2023; originally announced September 2023.

Comments: 11 pages

Tasks Makyth Models: Machine Learning Assisted Surrogates for Tip** Points

Authors: Gianluca Fabiani, Nikolaos Evangelou, Tianqi Cui, Juan M. Bello-Rivas, Cristina P. Martin-Linares, Constantinos Siettos, Ioannis G. Kevrekidis

Abstract: We present a machine learning (ML)-assisted framework bridging manifold learning, neural networks, Gaussian processes, and Equation-Free multiscale modeling, for (a) detecting tip** points in the emergent behavior of complex systems, and (b) characterizing probabilities of rare events (here, catastrophic shifts) near them. Our illustrative example is an event-driven, stochastic agent-based model… ▽ More We present a machine learning (ML)-assisted framework bridging manifold learning, neural networks, Gaussian processes, and Equation-Free multiscale modeling, for (a) detecting tip** points in the emergent behavior of complex systems, and (b) characterizing probabilities of rare events (here, catastrophic shifts) near them. Our illustrative example is an event-driven, stochastic agent-based model (ABM) describing the mimetic behavior of traders in a simple financial market. Given high-dimensional spatiotemporal data -- generated by the stochastic ABM -- we construct reduced-order models for the emergent dynamics at different scales: (a) mesoscopic Integro-Partial Differential Equations (IPDEs); and (b) mean-field-type Stochastic Differential Equations (SDEs) embedded in a low-dimensional latent space, targeted to the neighborhood of the tip** point. We contrast the uses of the different models and the effort involved in learning them. △ Less

Submitted 25 September, 2023; originally announced September 2023.

Comments: 29 pages, 8 figures, 6 tables

On-Manifold Projected Gradient Descent

Authors: Aaron Mahler, Tyrus Berry, Tom Stephens, Harbir Antil, Michael Merritt, Jeanie Schreiber, Ioannis Kevrekidis

Abstract: This work provides a computable, direct, and mathematically rigorous approximation to the differential geometry of class manifolds for high-dimensional data, along with nonlinear projections from input space onto these class manifolds. The tools are applied to the setting of neural network image classifiers, where we generate novel, on-manifold data samples, and implement a projected gradient desc… ▽ More This work provides a computable, direct, and mathematically rigorous approximation to the differential geometry of class manifolds for high-dimensional data, along with nonlinear projections from input space onto these class manifolds. The tools are applied to the setting of neural network image classifiers, where we generate novel, on-manifold data samples, and implement a projected gradient descent algorithm for on-manifold adversarial training. The susceptibility of neural networks (NNs) to adversarial attack highlights the brittle nature of NN decision boundaries in input space. Introducing adversarial examples during training has been shown to reduce the susceptibility of NNs to adversarial attack; however, it has also been shown to reduce the accuracy of the classifier if the examples are not valid examples for that class. Realistic "on-manifold" examples have been previously generated from class manifolds in the latent of an autoencoder. Our work explores these phenomena in a geometric and computational setting that is much closer to the raw, high-dimensional input space than can be provided by VAE or other black box dimensionality reductions. We employ conformally invariant diffusion maps (CIDM) to approximate class manifolds in diffusion coordinates, and develop the Nyström projection to project novel points onto class manifolds in this setting. On top of the manifold approximation, we leverage the spectral exterior calculus (SEC) to determine geometric quantities such as tangent vectors of the manifold. We use these tools to obtain adversarial examples that reside on a class manifold, yet fool a classifier. These misclassifications then become explainable in terms of human-understandable manipulations within the data, by expressing the on-manifold adversary in the semantic basis on the manifold. △ Less

Submitted 23 August, 2023; originally announced August 2023.

Transporting Densities Across Dimensions

Authors: Michael Plainer, Felix Dietrich, Ioannis G. Kevrekidis

Abstract: Even the best scientific equipment can only partially observe reality. Recorded data is often lower-dimensional, e.g., two-dimensional pictures of the three-dimensional world. Combining data from multiple experiments then results in a marginal density. This work shows how to transport such lower-dimensional marginal densities into a more informative, higher-dimensional joint space by leveraging ti… ▽ More Even the best scientific equipment can only partially observe reality. Recorded data is often lower-dimensional, e.g., two-dimensional pictures of the three-dimensional world. Combining data from multiple experiments then results in a marginal density. This work shows how to transport such lower-dimensional marginal densities into a more informative, higher-dimensional joint space by leveraging time-delayed measurements from an observation process. This can augment the information from scientific equipment to construct a more coherent view. Classical transportation algorithms can be used when the source and target dimensions match. Our approach allows the transport of samples between spaces of different dimensions by exploiting information from the sample collection process. We reconstruct the surface of an implant from partial recordings of bacteria moving on it and construct a joint space for satellites orbiting the Earth by combining one-dimensional, time-delayed altitude measurements. △ Less

Submitted 25 May, 2023; originally announced May 2023.

MSC Class: 58K05; 60G30; 37C20; 62-07

Data-driven and Physics Informed Modelling of Chinese Hamster Ovary Cell Bioreactors

Authors: Tianqi Cui, Tom S. Bertalan, Nelson Ndahiro, Pratik Khare, Michael Betenbaugh, Costas Maranas, Ioannis G. Kevrekidis

Abstract: Fed-batch culture is an established operation mode for the production of biologics using mammalian cell cultures. Quantitative modeling integrates both kinetics for some key reaction steps and optimization-driven metabolic flux allocation, using flux balance analysis; this is known to lead to certain mathematical inconsistencies. Here, we propose a physically-informed data-driven hybrid model (a "… ▽ More Fed-batch culture is an established operation mode for the production of biologics using mammalian cell cultures. Quantitative modeling integrates both kinetics for some key reaction steps and optimization-driven metabolic flux allocation, using flux balance analysis; this is known to lead to certain mathematical inconsistencies. Here, we propose a physically-informed data-driven hybrid model (a "gray box") to learn models of the dynamical evolution of Chinese Hamster Ovary (CHO) cell bioreactors from process data. The approach incorporates physical laws (e.g. mass balances) as well as kinetic expressions for metabolic fluxes. Machine learning (ML) is then used to (a) directly learn evolution equations (black-box modelling); (b) recover unknown physical parameters ("white-box" parameter fitting) or -- importantly -- (c) learn partially unknown kinetic expressions (gray-box modelling). We encode the convex optimization step of the overdetermined metabolic biophysical system as a differentiable, feed-forward layer into our architectures, connecting partial physical knowledge with data-driven machine learning. △ Less

Submitted 4 May, 2023; originally announced May 2023.

Equation-Free Computations as DDDAS Protocols for Bifurcation Studies: A Granular Chain Example

Authors: M. O. Williams, Y. M. Psarellis, D. Pozharskiy, C. Chong, F. Li, J. Yang, P. G. Kevrekidis, I. G. Kevrekidis

Abstract: This chapter discusses the development and implementation of algorithms based on Equation-Free/Dynamic Data Driven Applications Systems (EF/DDDAS) protocols for the computer-assisted study of the bifurcation structure of complex dynamical systems, such as those that arise in biology (neuronal networks, cell populations), multiscale systems in physics, chemistry and engineering, and system modeling… ▽ More This chapter discusses the development and implementation of algorithms based on Equation-Free/Dynamic Data Driven Applications Systems (EF/DDDAS) protocols for the computer-assisted study of the bifurcation structure of complex dynamical systems, such as those that arise in biology (neuronal networks, cell populations), multiscale systems in physics, chemistry and engineering, and system modeling in the social sciences. An illustrative example demonstrates the experimental realization of a chain of granular particles (a so-called engineered granular chain). In particular, the focus is on the detection/stability analysis of time-periodic, spatially localized structures referred to as "dark breathers". Results in this chapter highlight, both experimentally and numerically, that the number of breathers can be controlled by varying the frequency as well as the amplitude of an "out of phase" actuation, and that a "snaking" structure in the bifurcation diagram (computed through standard, model-based numerical methods for dynamical systems) is also recovered through the EF/DDDAS methods operating on a black-box simulator. The EF/DDDAS protocols presented here are, therefore, a step towards general purpose protocols for performing detailed bifurcation analyses directly on laboratory experiments, not only on their mathematical models, but also on measured data. △ Less

Submitted 3 May, 2023; originally announced May 2023.

Comments: Accepted for publication as a chapter in the Handbook of Dynamic Data Driven Applications Systems

Some of the variables, some of the parameters, some of the times, with some physics known: Identification with partial information

Authors: Saurabh Malani, Tom S. Bertalan, Tianqi Cui, Jose L. Avalos, Michael Betenbaugh, Ioannis G. Kevrekidis

Abstract: Experimental data is often comprised of variables measured independently, at different sampling rates (non-uniform $Δ$t between successive measurements); and at a specific time point only a subset of all variables may be sampled. Approaches to identifying dynamical systems from such data typically use interpolation, imputation or subsampling to reorganize or modify the training data… ▽ More Experimental data is often comprised of variables measured independently, at different sampling rates (non-uniform $Δ$t between successive measurements); and at a specific time point only a subset of all variables may be sampled. Approaches to identifying dynamical systems from such data typically use interpolation, imputation or subsampling to reorganize or modify the training data $\textit{prior}$ to learning. Partial physical knowledge may also be available $\textit{a priori}$ (accurately or approximately), and data-driven techniques can complement this knowledge. Here we exploit neural network architectures based on numerical integration methods and $\textit{a priori}$ physical knowledge to identify the right-hand side of the underlying governing differential equations. Iterates of such neural-network models allow for learning from data sampled at arbitrary time points $\textit{without}$ data modification. Importantly, we integrate the network with available partial physical knowledge in "physics informed gray-boxes"; this enables learning unknown kinetic rates or microbial growth functions while simultaneously estimating experimental parameters. △ Less

Submitted 27 April, 2023; originally announced April 2023.

Comments: 25 pages, 15 figures

Implementation and (Inverse Modified) Error Analysis for implicitly-templated ODE-nets

Authors: Aiqing Zhu, Tom Bertalan, Beibei Zhu, Yifa Tang, Ioannis G. Kevrekidis

Abstract: We focus on learning unknown dynamics from data using ODE-nets templated on implicit numerical initial value problem solvers. First, we perform Inverse Modified error analysis of the ODE-nets using unrolled implicit schemes for ease of interpretation. It is shown that training an ODE-net using an unrolled implicit scheme returns a close approximation of an Inverse Modified Differential Equation (I… ▽ More We focus on learning unknown dynamics from data using ODE-nets templated on implicit numerical initial value problem solvers. First, we perform Inverse Modified error analysis of the ODE-nets using unrolled implicit schemes for ease of interpretation. It is shown that training an ODE-net using an unrolled implicit scheme returns a close approximation of an Inverse Modified Differential Equation (IMDE). In addition, we establish a theoretical basis for hyper-parameter selection when training such ODE-nets, whereas current strategies usually treat numerical integration of ODE-nets as a black box. We thus formulate an adaptive algorithm which monitors the level of error and adapts the number of (unrolled) implicit solution iterations during the training process, so that the error of the unrolled approximation is less than the current learning loss. This helps accelerate training, while maintaining accuracy. Several numerical experiments are performed to demonstrate the advantages of the proposed algorithm compared to nonadaptive unrollings, and validate the theoretical analysis. We also note that this approach naturally allows for incorporating partially known physical terms in the equations, giving rise to what is termed ``gray box" identification. △ Less

Submitted 9 April, 2023; v1 submitted 31 March, 2023; originally announced March 2023.

Discrete-Time Nonlinear Feedback Linearization via Physics-Informed Machine Learning

Authors: Hector Vargas Alvarez, Gianluca Fabiani, Nikolaos Kazantzis, Constantinos Siettos, Ioannis G. Kevrekidis

Abstract: We present a physics-informed machine learning (PIML) scheme for the feedback linearization of nonlinear discrete-time dynamical systems. The PIML finds the nonlinear transformation law, thus ensuring stability via pole placement, in one step. In order to facilitate convergence in the presence of steep gradients in the nonlinear transformation law, we address a greedy-wise training procedure. We a… ▽ More We present a physics-informed machine learning (PIML) scheme for the feedback linearization of nonlinear discrete-time dynamical systems. The PIML finds the nonlinear transformation law, thus ensuring stability via pole placement, in one step. In order to facilitate convergence in the presence of steep gradients in the nonlinear transformation law, we address a greedy-wise training procedure. We assess the performance of the proposed PIML approach via a benchmark nonlinear discrete map for which the feedback linearization transformation law can be derived analytically; the example is characterized by steep gradients, due to the presence of singularities, in the domain of interest. We show that the proposed PIML outperforms, in terms of numerical approximation accuracy, the traditional numerical implementation, which involves the construction--and the solution in terms of the coefficients of a power-series expansion--of a system of homological equations as well as the implementation of the PIML in the entire domain, thus highlighting the importance of continuation techniques in the training procedure of PIML. △ Less

Submitted 15 March, 2023; originally announced March 2023.

MSC Class: 65Pxx; 93C10; 68T07

Identifying Equivalent Training Dynamics

Authors: William T. Redman, Juan M. Bello-Rivas, Maria Fonoberova, Ryan Mohr, Ioannis G. Kevrekidis, Igor Mezić

Abstract: Study of the nonlinear evolution deep neural network (DNN) parameters undergo during training has uncovered regimes of distinct dynamical behavior. While a detailed understanding of these phenomena has the potential to advance improvements in training efficiency and robustness, the lack of methods for identifying when DNN models have equivalent dynamics limits the insight that can be gained from p… ▽ More Study of the nonlinear evolution deep neural network (DNN) parameters undergo during training has uncovered regimes of distinct dynamical behavior. While a detailed understanding of these phenomena has the potential to advance improvements in training efficiency and robustness, the lack of methods for identifying when DNN models have equivalent dynamics limits the insight that can be gained from prior work. Topological conjugacy, a notion from dynamical systems theory, provides a precise definition of dynamical equivalence, offering a possible route to address this need. However, topological conjugacies have historically been challenging to compute. By leveraging advances in Koopman operator theory, we develop a framework for identifying conjugate and non-conjugate training dynamics. To validate our approach, we demonstrate that it can correctly identify a known equivalence between online mirror descent and online gradient descent. We then utilize it to: identify non-conjugate training dynamics between shallow and wide fully connected neural networks; characterize the early phase of training dynamics in convolutional neural networks; uncover non-conjugate training dynamics in Transformers that do and do not undergo grokking. Our results, across a range of DNN architectures, illustrate the flexibility of our framework and highlight its potential for shedding new light on training dynamics. △ Less

Submitted 4 June, 2024; v1 submitted 17 February, 2023; originally announced February 2023.

Comments: 18 pages, 6 figures, 3 supplemental figures

Gentlest ascent dynamics on manifolds defined by adaptively sampled point-clouds

Authors: Juan M. Bello-Rivas, Anastasia Georgiou, Hannes Vandecasteele, Ioannis G. Kevrekidis

Abstract: Finding saddle points of dynamical systems is an important problem in practical applications such as the study of rare events of molecular systems. Gentlest ascent dynamics (GAD) is one of a number of algorithms in existence that attempt to find saddle points in dynamical systems. It works by deriving a new dynamical system in which saddle points of the original system become stable equilibria. GA… ▽ More Finding saddle points of dynamical systems is an important problem in practical applications such as the study of rare events of molecular systems. Gentlest ascent dynamics (GAD) is one of a number of algorithms in existence that attempt to find saddle points in dynamical systems. It works by deriving a new dynamical system in which saddle points of the original system become stable equilibria. GAD has been recently generalized to the study of dynamical systems on manifolds (differential algebraic equations) described by equality constraints and given in an extrinsic formulation. In this paper, we present an extension of GAD to manifolds defined by point-clouds, formulated using the intrinsic viewpoint. These point-clouds are adaptively sampled during an iterative process that drives the system from the initial conformation (typically in the neighborhood of a stable equilibrium) to a saddle point. Our method requires the reactant (initial conformation), does not require the explicit constraint equations to be specified, and is purely data-driven. △ Less

Submitted 23 April, 2023; v1 submitted 8 February, 2023; originally announced February 2023.

Physics-agnostic and Physics-infused machine learning for thin films flows: modeling, and predictions from small data

Authors: Cristina P. Martin-Linares, Yorgos M. Psarellis, Georgios Karapetsas, Eleni D. Koronaki, Ioannis G. Kevrekidis

Abstract: Numerical simulations of multiphase flows are crucial in numerous engineering applications, but are often limited by the computationally demanding solution of the Navier-Stokes (NS) equations. Here, we present a data-driven workflow where a handful of detailed NS simulation data are leveraged into a reduced-order model for a prototypical vertically falling liquid film. We develop a physics-agnosti… ▽ More Numerical simulations of multiphase flows are crucial in numerous engineering applications, but are often limited by the computationally demanding solution of the Navier-Stokes (NS) equations. Here, we present a data-driven workflow where a handful of detailed NS simulation data are leveraged into a reduced-order model for a prototypical vertically falling liquid film. We develop a physics-agnostic model for the film thickness, achieving a far better agreement with the NS solutions than the asymptotic Kuramoto-Sivashinsky (KS) equation. We also develop two variants of physics-infused models providing a form of calibration of a low-fidelity model (i.e. the KS) against a few high-fidelity NS data. Finally, predictive models for missing data are developed, for either the amplitude, or the full-field velocity and even the flow parameter from partial information. This is achieved with the so-called "Gappy Diffusion Maps", which we compare favorably to its linear counterpart, Gappy POD. △ Less

Submitted 29 January, 2023; originally announced January 2023.

Certified Invertibility in Neural Networks via Mixed-Integer Programming

Authors: Tianqi Cui, Thomas Bertalan, George J. Pappas, Manfred Morari, Ioannis G. Kevrekidis, Mahyar Fazlyab

Abstract: Neural networks are known to be vulnerable to adversarial attacks, which are small, imperceptible perturbations that can significantly alter the network's output. Conversely, there may exist large, meaningful perturbations that do not affect the network's decision (excessive invariance). In our research, we investigate this latter phenomenon in two contexts: (a) discrete-time dynamical system iden… ▽ More Neural networks are known to be vulnerable to adversarial attacks, which are small, imperceptible perturbations that can significantly alter the network's output. Conversely, there may exist large, meaningful perturbations that do not affect the network's decision (excessive invariance). In our research, we investigate this latter phenomenon in two contexts: (a) discrete-time dynamical system identification, and (b) the calibration of a neural network's output to that of another network. We examine noninvertibility through the lens of mathematical optimization, where the global solution measures the ``safety" of the network predictions by their distance from the non-invertibility boundary. We formulate mixed-integer programs (MIPs) for ReLU networks and $L_p$ norms ($p=1,2,\infty$) that apply to neural network approximators of dynamical systems. We also discuss how our findings can be useful for invertibility certification in transformations between neural networks, e.g. between different levels of network pruning. △ Less

Submitted 16 May, 2023; v1 submitted 27 January, 2023; originally announced January 2023.

Comments: 22 pages, 7 figures

From partial data to out-of-sample parameter and observation estimation with Diffusion Maps and Geometric Harmonics

Authors: Eleni D. Koronaki, Nikolaos Evangelou, Yorgos M. Psarellis, Andreas G. Boudouvis, Ioannis G. Kevrekidis

Abstract: A data-driven framework is presented, that enables the prediction of quantities, either observations or parameters, given sufficient partial data. The framework is illustrated via a computational model of the deposition of Cu in a Chemical Vapor Deposition (CVD) reactor, where the reactor pressure, the deposition temperature and feed mass flow rate are important process parameters that determine t… ▽ More A data-driven framework is presented, that enables the prediction of quantities, either observations or parameters, given sufficient partial data. The framework is illustrated via a computational model of the deposition of Cu in a Chemical Vapor Deposition (CVD) reactor, where the reactor pressure, the deposition temperature and feed mass flow rate are important process parameters that determine the outcome of the process. The sampled observations are high-dimensional vectors containing the outputs of a detailed CFD steady-state model of the process, i.e. the values of velocity, pressure, temperature, and species mass fractions at each point in the discretization. A machine learning workflow is presented, able to predict out-of-sample (a) observations (e.g. mass fraction in the reactor) given process parameters (e.g. inlet temperature); (b) process parameters given observation data; and (c) partial observations (e.g. temperature in the reactor) given other partial observations (e.g. mass fraction in the reactor). The proposed workflow relies on the manifold learning schemes Diffusion Maps and the associated Geometric Harmonics. Diffusion Maps is used for discovering a reduced representation of the available data, and Geometric Harmonics for extending functions defined on the manifold. In our work a special use case of Geometric Harmonics is formulated and implemented, which we call Double Diffusion Maps, to map from the reduced representation back to (partial) observations and process parameters. A comparison of our manifold learning scheme to the traditional Gappy-POD approach is provided: ours can be thought of as a "Gappy DMAP" approach. The presented methodology is easily transferable to application domains beyond reactor engineering. △ Less

Submitted 27 January, 2023; originally announced January 2023.

Journal ref: Computers & Chemical Engineering, Volume 178, 2023,

A Recursively Recurrent Neural Network (R2N2) Architecture for Learning Iterative Algorithms

Authors: Danimir T. Doncevic, Alexander Mitsos, Yue Guo, Qianxiao Li, Felix Dietrich, Manuel Dahmen, Ioannis G. Kevrekidis

Abstract: Meta-learning of numerical algorithms for a given task consists of the data-driven identification and adaptation of an algorithmic structure and the associated hyperparameters. To limit the complexity of the meta-learning problem, neural architectures with a certain inductive bias towards favorable algorithmic structures can, and should, be used. We generalize our previously introduced Runge-Kutta… ▽ More Meta-learning of numerical algorithms for a given task consists of the data-driven identification and adaptation of an algorithmic structure and the associated hyperparameters. To limit the complexity of the meta-learning problem, neural architectures with a certain inductive bias towards favorable algorithmic structures can, and should, be used. We generalize our previously introduced Runge-Kutta neural network to a recursively recurrent neural network (R2N2) superstructure for the design of customized iterative algorithms. In contrast to off-the-shelf deep learning approaches, it features a distinct division into modules for generation of information and for the subsequent assembly of this information towards a solution. Local information in the form of a subspace is generated by subordinate, inner, iterations of recurrent function evaluations starting at the current outer iterate. The update to the next outer iterate is computed as a linear combination of these evaluations, reducing the residual in this space, and constitutes the output of the network. We demonstrate that regular training of the weight parameters inside the proposed superstructure on input/output data of various computational problem classes yields iterations similar to Krylov solvers for linear equation systems, Newton-Krylov solvers for nonlinear equation systems, and Runge-Kutta integrators for ordinary differential equations. Due to its modularity, the superstructure can be readily extended with functionalities needed to represent more general classes of iterative algorithms traditionally based on Taylor series expansions. △ Less

Submitted 6 July, 2023; v1 submitted 22 November, 2022; originally announced November 2022.

Comments: manuscript (22 pages, 9 figures), supporting information (11 pages, 9 figures)

Quantifying the Structure of Disordered Materials

Authors: Thomas J. Hardin, Michael Chandross, Rahul Meena, Spencer Fajardo, Dimitris Giovanis, Ioannis G. Kevrekidis, Michael Falk, Michael Shields

Abstract: Durable interest in develo** a framework for the detailed structure of glassy materials has produced numerous structural descriptors that trade off between general applicability and interpretability. However, none approach the combination of simplicity and wide-ranging predictive power of the lattice-grain-defect framework for crystalline materials. Working from the hypothesis that the local ato… ▽ More Durable interest in develo** a framework for the detailed structure of glassy materials has produced numerous structural descriptors that trade off between general applicability and interpretability. However, none approach the combination of simplicity and wide-ranging predictive power of the lattice-grain-defect framework for crystalline materials. Working from the hypothesis that the local atomic environments of a glassy material are constrained by enthalpy minimization to a low-dimensional manifold in atomic coordinate space, we develop a novel generalized distance function, the Gaussian Integral Inner Product (GIIP) distance, in connection with agglomerative clustering and diffusion maps, to parameterize that manifold. Applying this approach to a two-dimensional model crystal and a three-dimensional binary model metallic glass results in parameters interpretable as coordination number, composition, volumetric strain, and local symmetry. In particular, we show that a more slowly quenched glass has a higher degree of local tetrahedral symmetry at the expense of cyclic symmetry. While these descriptors require post-hoc interpretation, they minimize bias rooted in crystalline materials science and illuminate a range of structural trends that might otherwise be missed. △ Less

Submitted 14 November, 2022; originally announced November 2022.

Comments: 18 pages, 4 figures

Report number: SAND2022-15908 O

Two novel families of multiscale staggered patch schemes efficiently simulate large-scale, weakly damped, linear waves

Authors: J. Divahar, A. J. Roberts, Trent W. Mattner, J. E. Bunder, Ioannis G. Kevrekidis

Abstract: Many multiscale wave systems exhibit macroscale emergent behaviour, for example, the fluid dynamics of floods and tsunamis. Resolving a large range of spatial scales typically requires a prohibitively high computational cost. The small dissipation in wave systems poses a significant challenge to further develo** multiscale modelling methods in multiple dimensions. This article develops and evalu… ▽ More Many multiscale wave systems exhibit macroscale emergent behaviour, for example, the fluid dynamics of floods and tsunamis. Resolving a large range of spatial scales typically requires a prohibitively high computational cost. The small dissipation in wave systems poses a significant challenge to further develo** multiscale modelling methods in multiple dimensions. This article develops and evaluates two families of equation-free multiscale methods on novel 2D staggered patch schemes, and demonstrates the power and utility of these multiscale schemes for weakly damped linear waves. A detailed study of sensitivity to numerical roundoff errors establishes the robustness of developed staggered patch schemes. Comprehensive eigenvalue analysis over a wide range of parameters establishes the stability, accuracy, and consistency of the multiscale schemes. Analysis of the computational complexity shows that the measured compute times of the multiscale schemes may be 10^5 times smaller than the compute time for the corresponding full-domain computation. This work provides the essential foundation for efficient large-scale simulation of challenging nonlinear multiscale waves. △ Less

Submitted 27 October, 2022; originally announced October 2022.

Comments: 35 pages, 12 figures, and 6 tables

A note on the control of processes exhibiting input multiplicity

Authors: Robert J. Lovelett, Yorgos M. Psarellis, Ioannis G. Kevrekidis, Manfred Morari

Abstract: Steady state multiplicity can occur in nonlinear systems, and this presents challenges to feedback control. Input multiplicity arises when the same steady state output values can be reached with system inputs at different values. Dynamic systems with input multiplicities equipped with controllers with integral action have multiple stationary points, which may be locally stable or not. This is unde… ▽ More Steady state multiplicity can occur in nonlinear systems, and this presents challenges to feedback control. Input multiplicity arises when the same steady state output values can be reached with system inputs at different values. Dynamic systems with input multiplicities equipped with controllers with integral action have multiple stationary points, which may be locally stable or not. This is undesirable for operation. For a 2x2 example system with three stationary points we demonstrate how to design a set of two single loop controllers such that only one of the stationary points is locally stable, thus effectively eliminating the "input multiplicity problem" for control. We also show that when MPC is used for the example system, all three closed-loop stationary points are stable. Depending on the initial value of the input variables, the closed loop system under MPC may converge to different steady state input instances (but the same output steady state). Therefore we computationally explore the basin boundaries of this closed loop system. It is not clear how MPC or other modern nonlinear controllers could be designed so that only specific equilibrium points are stable. △ Less

Submitted 4 October, 2022; originally announced October 2022.

Algorithmic (Semi-)Conjugacy via Koopman Operator Theory

Authors: William T. Redman, Maria Fonoberova, Ryan Mohr, Ioannis G. Kevrekidis, Igor Mezić

Abstract: Iterative algorithms are of utmost importance in decision and control. With an ever growing number of algorithms being developed, distributed, and proprietarized, there is a similarly growing need for methods that can provide classification and comparison. By viewing iterative algorithms as discrete-time dynamical systems, we leverage Koopman operator theory to identify (semi-)conjugacies between… ▽ More Iterative algorithms are of utmost importance in decision and control. With an ever growing number of algorithms being developed, distributed, and proprietarized, there is a similarly growing need for methods that can provide classification and comparison. By viewing iterative algorithms as discrete-time dynamical systems, we leverage Koopman operator theory to identify (semi-)conjugacies between algorithms using their spectral properties. This provides a general framework with which to classify and compare algorithms. △ Less

Submitted 13 September, 2022; originally announced September 2022.

Comments: 6 pages, 5 figures, accepted to IEEE CDC 2022

Data-driven Discovery of Chemotactic Migration of Bacteria via Machine Learning

Authors: Yorgos M. Psarellis, Seungjoon Lee, Tapomoy Bhattacharjee, Sujit S. Datta, Juan M. Bello-Rivas, Ioannis G. Kevrekidis

Abstract: E. coli chemotactic motion in the presence of a chemoattractant field has been extensively studied using wet laboratory experiments, stochastic computational models as well as partial differential equation-based models (PDEs). The most challenging step in bridging these approaches, is establishing a closed form of the so-called chemotactic term, which describes how bacteria bias their motion up ch… ▽ More E. coli chemotactic motion in the presence of a chemoattractant field has been extensively studied using wet laboratory experiments, stochastic computational models as well as partial differential equation-based models (PDEs). The most challenging step in bridging these approaches, is establishing a closed form of the so-called chemotactic term, which describes how bacteria bias their motion up chemonutrient concentration gradients, as a result of a cascade of biochemical processes. Data-driven models can be used to learn the entire evolution operator of the chemotactic PDEs (black box models), or, in a more targeted fashion, to learn just the chemotactic term (gray box models). In this work, data-driven Machine Learning approaches for learning the underlying model PDEs are (a) validated through the use of simulation data from established continuum models and (b) used to infer chemotactic PDEs from experimental data. Even when the data at hand are sparse (coarse in space and/or time), noisy (due to inherent stochasticity in measurements) or partial (e.g. lack of measurements of the associated chemoattractant field), we can attempt to learn the right-hand-side of a closed PDE for an evolving bacterial density. In fact we show that data-driven PDEs including a short history of the bacterial density field (e.g. in the form of higher-order in time PDEs in terms of the measurable bacterial density) can be successful in predicting further bacterial density evolution, and even possibly recovering estimates of the unmeasured chemonutrient field. The main tool in this effort is the effective low-dimensionality of the dynamics (in the spirit of the Whitney and Takens embedding theorems). The resulting data-driven PDE can then be simulated to reproduce/predict computational or experimental bacterial density profile data, and estimate the underlying (unmeasured) chemonutrient field evolution. △ Less

Submitted 24 August, 2022; originally announced August 2022.

Comments: 44 pages, 20 figures, 3 tables

Limits of Entrainment of Circadian Neuronal Networks

Authors: Yorgos M. Psarellis, Michail Kavousanakis, Michael A. Henson, Ioannis G. Kevrekidis

Abstract: Circadian rhythmicity lies at the center of various important physiological and behavioral processes in mammals, such as sleep, metabolism, homeostasis, mood changes and more. It has been shown that this rhythm arises from self-sustained biomolecular oscillations of a neuronal network located in the Suprachiasmatic Nucleus (SCN). Under normal circumstances, this network remains synchronized to the… ▽ More Circadian rhythmicity lies at the center of various important physiological and behavioral processes in mammals, such as sleep, metabolism, homeostasis, mood changes and more. It has been shown that this rhythm arises from self-sustained biomolecular oscillations of a neuronal network located in the Suprachiasmatic Nucleus (SCN). Under normal circumstances, this network remains synchronized to the day-night cycle due to signaling from the retina. Misalignment of these neuronal oscillations with the external light signal can disrupt numerous physiological functions and take a long-lasting toll on health and well-being. In this work, we study a modern computational neuroscience model to determine the limits of circadian synchronization to external light signals of different frequency and duty cycle. We employ a matrix-free approach to locate periodic steady states of the high-dimensional model for various driving conditions. Our algorithmic pipeline enables numerical continuation and construction of bifurcation diagrams w.r.t. forcing parameters. We computationally explore the effect of heterogeneity in the circadian neuronal network, as well as the effect of corrective therapeutic interventions, such as that of the drug molecule Longdaysin. Lastly, we employ unsupervised learning to construct a data-driven embedding space for representing neuronal heterogeneity. △ Less

Submitted 23 August, 2022; originally announced August 2022.

Comments: 14 pages, 15 figures

GANs and Closures: Micro-Macro Consistency in Multiscale Modeling

Authors: Ellis R. Crabtree, Juan M. Bello-Rivas, Andrew L. Ferguson, Ioannis G. Kevrekidis

Abstract: Sampling the phase space of molecular systems -- and, more generally, of complex systems effectively modeled by stochastic differential equations -- is a crucial modeling step in many fields, from protein folding to materials discovery. These problems are often multiscale in nature: they can be described in terms of low-dimensional effective free energy surfaces parametrized by a small number of "… ▽ More Sampling the phase space of molecular systems -- and, more generally, of complex systems effectively modeled by stochastic differential equations -- is a crucial modeling step in many fields, from protein folding to materials discovery. These problems are often multiscale in nature: they can be described in terms of low-dimensional effective free energy surfaces parametrized by a small number of "slow" reaction coordinates; the remaining "fast" degrees of freedom populate an equilibrium measure on the reaction coordinate values. Sampling procedures for such problems are used to estimate effective free energy differences as well as ensemble averages with respect to the conditional equilibrium distributions; these latter averages lead to closures for effective reduced dynamic models. Over the years, enhanced sampling techniques coupled with molecular simulation have been developed. An intriguing analogy arises with the field of Machine Learning (ML), where Generative Adversarial Networks can produce high dimensional samples from low dimensional probability distributions. This sample generation returns plausible high dimensional space realizations of a model state, from information about its low-dimensional representation. In this work, we present an approach that couples physics-based simulations and biasing methods for sampling conditional distributions with ML-based conditional generative adversarial networks for the same task. The "coarse descriptors" on which we condition the fine scale realizations can either be known a priori, or learned through nonlinear dimensionality reduction. We suggest that this may bring out the best features of both approaches: we demonstrate that a framework that couples cGANs with physics-based enhanced sampling techniques can improve multiscale SDE dynamical systems sampling, and even shows promise for systems of increasing complexity. △ Less

Submitted 9 December, 2023; v1 submitted 22 August, 2022; originally announced August 2022.

Comments: 26 pages, 14 figures, 3 tables

MSC Class: 82C32 (Primary) 37M05; 60H10; 68T07; 82-08 (Secondary)

Staggered grids for multidimensional multiscale modelling

Authors: J. Divahar, A. J. Roberts, Trent W. Mattner, J. E. Bunder, Ioannis G. Kevrekidis

Abstract: Numerical schemes for wave-like systems with small dissipation are often inaccurate and unstable due to truncation errors and numerical roundoff errors. Hence, numerical simulations of wave-like systems lacking proper handling of these numerical issues often fail to represent the physical characteristics of wave phenomena. This challenge gets even more intricate for multiscale modelling, especiall… ▽ More Numerical schemes for wave-like systems with small dissipation are often inaccurate and unstable due to truncation errors and numerical roundoff errors. Hence, numerical simulations of wave-like systems lacking proper handling of these numerical issues often fail to represent the physical characteristics of wave phenomena. This challenge gets even more intricate for multiscale modelling, especially in multiple dimensions. When using the usual collocated grid, about two-thirds of the resolved wave modes are incorrect with significant dispersion. But, numerical schemes on staggered grids (with alternating variable arrangement) are significantly less dispersive and preserve much of the wave characteristics. Also, the group velocity of the energy propagation in the numerical waves on a staggered grid is in the correct direction, in contrast to the collocated grid. For high accuracy and to preserve much of the wave characteristics, this article extends the concept of staggered grids in full-domain modelling to multidimensional multiscale modelling. Specifically, this article develops 120 multiscale staggered grids and demonstrates their stability, accuracy, and wave-preserving characteristic for equation-free multiscale modelling of weakly damped linear waves. But most characteristics of the developed multiscale staggered grids must also hold in general for multiscale modelling of many complex spatio-temporal physical phenomena such as the general computational fluid dynamics. △ Less

Submitted 25 July, 2022; originally announced July 2022.

Comments: 42 pages, 13 figures, and 1 table

Data-driven Control of Agent-based Models: an Equation/Variable-free Machine Learning Approach

Authors: Dimitrios G. Patsatzis, Lucia Russo, Ioannis G. Kevrekidis, Constantinos Siettos

Abstract: We present an Equation/Variable free machine learning (EVFML) framework for the control of the collective dynamics of complex/multiscale systems modelled via microscopic/agent-based simulators. The approach obviates the need for construction of surrogate, reduced-order models.~The proposed implementation consists of three steps: (A) from high-dimensional agent-based simulations, machine learning (… ▽ More We present an Equation/Variable free machine learning (EVFML) framework for the control of the collective dynamics of complex/multiscale systems modelled via microscopic/agent-based simulators. The approach obviates the need for construction of surrogate, reduced-order models.~The proposed implementation consists of three steps: (A) from high-dimensional agent-based simulations, machine learning (in particular, non-linear manifold learning (Diffusion Maps (DMs)) helps identify a set of coarse-grained variables that parametrize the low-dimensional manifold on which the emergent/collective dynamics evolve. The out-of-sample extension and pre-image problems, i.e. the construction of non-linear map**s from the high-dimensional input space to the low-dimensional manifold and back, are solved by coupling DMs with the Nystrom extension and Geometric Harmonics, respectively; (B) having identified the manifold and its coordinates, we exploit the Equation-free approach to perform numerical bifurcation analysis of the emergent dynamics; then (C) based on the previous steps, we design data-driven embedded wash-out controllers that drive the agent-based simulators to their intrinsic, imprecisely known, emergent open-loop unstable steady-states, thus demonstrating that the scheme is robust against numerical approximation errors and modelling uncertainty.~The efficiency of the framework is illustrated by controlling emergent unstable (i) traveling waves of a deterministic agent-based model of traffic dynamics, and (ii) equilibria of a stochastic financial market agent model with mimesis. △ Less

Submitted 5 August, 2022; v1 submitted 12 July, 2022; originally announced July 2022.

MSC Class: 37-xx; 65Pxx; 93-XX; 68Txx

Journal ref: Journal of Computational Physics, 478, 111953, 2023

Black and Gray Box Learning of Amplitude Equations: Application to Phase Field Systems

Authors: Felix P. Kemeth, Sergio Alonso, Blas Echebarria, Ted Moldenhawer, Carsten Beta, Ioannis G. Kevrekidis

Abstract: We present a data-driven approach to learning surrogate models for amplitude equations, and illustrate its application to interfacial dynamics of phase field systems. In particular, we demonstrate learning effective partial differential equations describing the evolution of phase field interfaces from full phase field data. We illustrate this on a model phase field system, where analytical approxi… ▽ More We present a data-driven approach to learning surrogate models for amplitude equations, and illustrate its application to interfacial dynamics of phase field systems. In particular, we demonstrate learning effective partial differential equations describing the evolution of phase field interfaces from full phase field data. We illustrate this on a model phase field system, where analytical approximate equations for the dynamics of the phase field interface (a higher order eikonal equation and its approximation, the Kardar-Parisi-Zhang (KPZ) equation) are known. For this system, we discuss data-driven approaches for the identification of equations that accurately describe the front interface dynamics. When the analytical approximate models mentioned above become inaccurate, as we move beyond the region of validity of the underlying assumptions, the data-driven equations outperform them. In these regimes, going beyond black-box identification, we explore different approaches to learn data-driven corrections to the analytically approximate models, leading to effective gray box partial differential equations. △ Less

Submitted 8 July, 2022; originally announced July 2022.

Learning black- and gray-box chemotactic PDEs/closures from agent based Monte Carlo simulation data

Authors: Seungjoon Lee, Yorgos M. Psarellis, Constantinos I. Siettos, Ioannis G. Kevrekidis

Abstract: We propose a machine learning framework for the data-driven discovery of macroscopic chemotactic Partial Differential Equations (PDEs) -- and the closures that lead to them -- from high-fidelity, individual-based stochastic simulations of E.coli bacterial motility. The fine scale, detailed, hybrid (continuum - Monte Carlo) simulation model embodies the underlying biophysics, and its parameters are… ▽ More We propose a machine learning framework for the data-driven discovery of macroscopic chemotactic Partial Differential Equations (PDEs) -- and the closures that lead to them -- from high-fidelity, individual-based stochastic simulations of E.coli bacterial motility. The fine scale, detailed, hybrid (continuum - Monte Carlo) simulation model embodies the underlying biophysics, and its parameters are informed from experimental observations of individual cells. We exploit Automatic Relevance Determination (ARD) within a Gaussian Process framework for the identification of a parsimonious set of collective observables that parametrize the law of the effective PDEs. Using these observables, in a second step we learn effective, coarse-grained "Keller-Segel class" chemotactic PDEs using machine learning regressors: (a) (shallow) feedforward neural networks and (b) Gaussian Processes. The learned laws can be black-box (when no prior knowledge about the PDE law structure is assumed) or gray-box when parts of the equation (e.g. the pure diffusion part) is known and "hardwired" in the regression process. We also discuss data-driven corrections (both additive and functional) of analytically known, approximate closures. △ Less

Submitted 25 May, 2022; originally announced May 2022.

Comments: 33 pages, 5 figures, 1 table

Learning Effective SDEs from Brownian Dynamics Simulations of Colloidal Particles

Authors: Nikolaos Evangelou, Felix Dietrich, Juan M. Bello-Rivas, Alex Yeh, Rachel Stein, Michael A. Bevan, Ioannis G. Kevrekidis

Abstract: We construct a reduced, data-driven, parameter dependent effective Stochastic Differential Equation (eSDE) for electric-field mediated colloidal crystallization using data obtained from Brownian Dynamics Simulations. We use Diffusion Maps (a manifold learning algorithm) to identify a set of useful latent observables. In this latent space we identify an eSDE using a deep learning architecture inspi… ▽ More We construct a reduced, data-driven, parameter dependent effective Stochastic Differential Equation (eSDE) for electric-field mediated colloidal crystallization using data obtained from Brownian Dynamics Simulations. We use Diffusion Maps (a manifold learning algorithm) to identify a set of useful latent observables. In this latent space we identify an eSDE using a deep learning architecture inspired by numerical stochastic integrators and compare it with the traditional Kramers-Moyal expansion estimation. We show that the obtained variables and the learned dynamics accurately encode the physics of the Brownian Dynamic Simulations. We further illustrate that our reduced model captures the dynamics of corresponding experimental data. Our dimension reduction/reduced model identification approach can be easily ported to a broad class of particle systems dynamics experiments/models. △ Less

Submitted 30 January, 2023; v1 submitted 30 April, 2022; originally announced May 2022.

Comments: 21 pages, 16 figures, 2 tables

Double Diffusion Maps and their Latent Harmonics for Scientific Computations in Latent Space

Authors: Nikolaos Evangelou, Felix Dietrich, Eliodoro Chiavazzo, Daniel Lehmberg, Marina Meila, Ioannis G. Kevrekidis

Abstract: We introduce a data-driven approach to building reduced dynamical models through manifold learning; the reduced latent space is discovered using Diffusion Maps (a manifold learning technique) on time series data. A second round of Diffusion Maps on those latent coordinates allows the approximation of the reduced dynamical models. This second round enables map** the latent space coordinates back… ▽ More We introduce a data-driven approach to building reduced dynamical models through manifold learning; the reduced latent space is discovered using Diffusion Maps (a manifold learning technique) on time series data. A second round of Diffusion Maps on those latent coordinates allows the approximation of the reduced dynamical models. This second round enables map** the latent space coordinates back to the full ambient space (what is called lifting); it also enables the approximation of full state functions of interest in terms of the reduced coordinates. In our work, we develop and test three different reduced numerical simulation methodologies, either through pre-tabulation in the latent space and integration on the fly or by going back and forth between the ambient space and the latent space. The data-driven latent space simulation results, based on the three different approaches, are validated through (a) the latent space observation of the full simulation through the Nyström Extension formula, or through (b) lifting the reduced trajectory back to the full ambient space, via Latent Harmonics. Latent space modeling often involves additional regularization to favor certain properties of the space over others, and the map** back to the ambient space is then constructed mostly independently from these properties; here, we use the same data-driven approach to construct the latent space and then map back to the ambient space. △ Less

Submitted 26 April, 2022; originally announced April 2022.

Comments: 25 pages,21 figures, 4 tables

Questionnaires to PDEs: From Disorganized Data to Emergent Generative Dynamic Models

Authors: David W. Sroczynski, Felix P. Kemeth, Ronald R. Coifman, Ioannis G. Kevrekidis

Abstract: Starting with sets of disorganized observations of spatially varying and temporally evolving systems, obtained at different (also disorganized) sets of parameters, we demonstrate the data-driven derivation of parameter dependent, evolutionary partial differential equation (PDE) models capable of generating the data. This tensor type of data is reminiscent of shuffled (multi-dimensional) puzzle til… ▽ More Starting with sets of disorganized observations of spatially varying and temporally evolving systems, obtained at different (also disorganized) sets of parameters, we demonstrate the data-driven derivation of parameter dependent, evolutionary partial differential equation (PDE) models capable of generating the data. This tensor type of data is reminiscent of shuffled (multi-dimensional) puzzle tiles. The independent variables for the evolution equations (their "space" and "time") as well as their effective parameters are all "emergent", i.e., determined in a data-driven way from our disorganized observations of behavior in them. We use a diffusion map based "questionnaire" approach to build a parametrization of our emergent space/time/parameter space for the data. This approach iteratively processes the data by successively observing them on the "space", the "time", and the "parameter" axes of a tensor. Once the data are organized, we use machine learning (here, neural networks) to approximate the operators governing the evolution equations in this emergent space. Our illustrative example is based on a previously developed vertex-plus-signaling model of Drosophila embryonic development. This allows us to discuss features of the process like symmetry breaking, translational invariance, and autonomousness of the emergent PDE model, as well as its interpretability. △ Less

Submitted 25 April, 2022; originally announced April 2022.

Comments: 9 main pages (8 figures) with 7 SI pages (5 figures) Submitted to PNAS Nexus

Staying the course: Locating equilibria of dynamical systems on Riemannian manifolds defined by point-clouds

Authors: Juan M. Bello-Rivas, Anastasia Georgiou, John Guckenheimer, Ioannis G. Kevrekidis

Abstract: We introduce a method to successively locate equilibria (steady states) of dynamical systems on Riemannian manifolds. The manifolds need not be characterized by an a priori known atlas or by the zeros of a smooth map. Instead, they can be defined by point-clouds and sampled as needed through an iterative process. If the manifold is an Euclidean space, our method follows isoclines, curves along whi… ▽ More We introduce a method to successively locate equilibria (steady states) of dynamical systems on Riemannian manifolds. The manifolds need not be characterized by an a priori known atlas or by the zeros of a smooth map. Instead, they can be defined by point-clouds and sampled as needed through an iterative process. If the manifold is an Euclidean space, our method follows isoclines, curves along which the direction of the vector field $X$ is constant. For a generic vector field $X$, isoclines are smooth curves and every equilibrium lies on isoclines. We generalize the definition of isoclines to Riemannian manifolds through the use of parallel transport: generalized isoclines are curves along which the directions of $X$ are parallel transports of each other. As in the Euclidean case, generalized isoclines of generic vector fields $X$ are smooth curves that connect equilibria of $X$. Our algorithm can be regarded as an extension of the method of Newton trajectories to the manifold setting when the manifold is unknown. This work is motivated by computational statistical mechanics, specifically high dimensional (stochastic) differential equations that model the dynamics of molecular systems. Often, these dynamics concentrate near low-dimensional manifolds and have transitions (saddle points with a single unstable direction) between metastable equilibria. We employ iteratively sampled data and isoclines to locate these saddle points. Coupling a black-box sampling scheme (e.g., Markov chain Monte Carlo) with manifold learning techniques (diffusion maps in the case presented here), we show that our method reliably locates equilibria of $X$. △ Less

Submitted 12 November, 2022; v1 submitted 21 April, 2022; originally announced April 2022.

MSC Class: 37M05; 53Z50; 65L99

Weakly Supervised Indoor Localization via Manifold Matching

Authors: Erez Peterfreund, Ioannis G. Kevrekidis, Ariel Jaffe

Abstract: Inferring the location of a mobile device in an indoor setting is an open problem of utmost significance. A leading approach that does not require the deployment of expensive infrastructure is fingerprinting, where a classifier is trained to predict the location of a device based on its captured signal. The main caveat of this approach is that acquiring a sufficiently large and accurate training s… ▽ More Inferring the location of a mobile device in an indoor setting is an open problem of utmost significance. A leading approach that does not require the deployment of expensive infrastructure is fingerprinting, where a classifier is trained to predict the location of a device based on its captured signal. The main caveat of this approach is that acquiring a sufficiently large and accurate training set may be prohibitively expensive. Here, we propose a weakly supervised method that only requires the location of a small number of devices. The localization is done by matching a low-dimensional spectral representation of the signals to a given sketch of the indoor environment. We test our approach on simulated and real data and show that it yields an accuracy of a few meters, which is on par with fully supervised approaches. The simplicity of our method and its accuracy with minimal supervision makes it ideal for implementation in indoor localization systems. △ Less

Submitted 7 February, 2022; originally announced February 2022.

Constructing coarse-scale bifurcation diagrams from spatio-temporal observations of microscopic simulations: A parsimonious machine learning approach

Authors: Evangelos Galaris, Gianluca Fabiani, Ioannis Gallos, Ioannis Kevrekidis, Constantinos Siettos

Abstract: We address a three-tier data-driven approach to solve the inverse problem in complex systems modelling from spatio-temporal data produced by microscopic simulators using machine learning. In the first step, we exploit manifold learning and in particular parsimonious Diffusion Maps using leave-one-out cross-validation (LOOCV) to both identify the intrinsic dimension of the manifold where the emerge… ▽ More We address a three-tier data-driven approach to solve the inverse problem in complex systems modelling from spatio-temporal data produced by microscopic simulators using machine learning. In the first step, we exploit manifold learning and in particular parsimonious Diffusion Maps using leave-one-out cross-validation (LOOCV) to both identify the intrinsic dimension of the manifold where the emergent dynamics evolve and for feature selection over the parametric space. In the second step, based on the selected features, we learn the right-hand-side of the effective partial differential equations (PDEs) using two machine learning schemes, namely shallow Feedforward Neural Networks (FNNs) with two hidden layers and single-layer Random Projection Networks(RPNNs) which basis functions are constructed using an appropriate random sampling approach. Finally, based on the learned black-box PDE model, we construct the corresponding bifurcation diagram, thus exploiting the numerical bifurcation analysis toolkit. For our illustrations, we implemented the proposed method to construct the one-parameter bifurcation diagram of the 1D FitzHugh-Nagumo PDEs from data generated by $D1Q3$ Lattice Boltzmann simulations. The proposed method was quite effective in terms of numerical accuracy regarding the construction of the coarse-scale bifurcation diagram. Furthermore, the proposed RPNN scheme was $\sim$ 20 to 30 times less costly regarding the training phase than the traditional shallow FNNs, thus arising as a promising alternative to deep learning for solving the inverse problem for high-dimensional PDEs. △ Less

Submitted 15 February, 2022; v1 submitted 31 January, 2022; originally announced January 2022.

Journal ref: Journal of Scientific Computing, 92, 34, 2022

A Spectral Analysis of the Nonlinear Schroedinger Equation in the Co-Exploding Frame

Authors: S. J. Chapman, M. E. Kavousanakis, E. G. Charalampidis, I. G. Kevrekidis, P. G. Kevrekidis

Abstract: The nonlinear Schroedinger model is a prototypical dispersive wave equation that features finite time blowup, either for supercritical exponents (for fixed dimension) or for supercritical dimensions (for fixed nonlinearity exponent). Upon identifying the self-similar solutions in the so-called "co-exploding frame", a dynamical systems analysis of their stability is natural, yet is complicated by t… ▽ More The nonlinear Schroedinger model is a prototypical dispersive wave equation that features finite time blowup, either for supercritical exponents (for fixed dimension) or for supercritical dimensions (for fixed nonlinearity exponent). Upon identifying the self-similar solutions in the so-called "co-exploding frame", a dynamical systems analysis of their stability is natural, yet is complicated by the mixed Hamiltonian-dissipative character of the relevant frame. In the present work, we study the spectral picture of the relevant linearized problem. We examine the point spectrum of 3 eigenvalue pairs associated with translation, $U(1)$ and conformal invariances, as well as the continuous spectrum. We find that two eigenvalues become positive, yet are attributed to symmetries and are thus not associated with instabilities. In addition to a vanishing eigenvalue, 3 more are found to be negative and real, while the continuous spectrum is nearly vertical and on the left-half (spectral) plane. Finally, the subtle effects of the boundaries are also assessed and their role in the observed weak eigenvalue oscillations is clarified. △ Less

Submitted 31 January, 2022; originally announced January 2022.

An Operator Theoretic View on Pruning Deep Neural Networks

Authors: William T. Redman, Maria Fonoberova, Ryan Mohr, Ioannis G. Kevrekidis, Igor Mezic

Abstract: The discovery of sparse subnetworks that are able to perform as well as full models has found broad applied and theoretical interest. While many pruning methods have been developed to this end, the naïve approach of removing parameters based on their magnitude has been found to be as robust as more complex, state-of-the-art algorithms. The lack of theory behind magnitude pruning's success, especia… ▽ More The discovery of sparse subnetworks that are able to perform as well as full models has found broad applied and theoretical interest. While many pruning methods have been developed to this end, the naïve approach of removing parameters based on their magnitude has been found to be as robust as more complex, state-of-the-art algorithms. The lack of theory behind magnitude pruning's success, especially pre-convergence, and its relation to other pruning methods, such as gradient based pruning, are outstanding open questions in the field that are in need of being addressed. We make use of recent advances in dynamical systems theory, namely Koopman operator theory, to define a new class of theoretically motivated pruning algorithms. We show that these algorithms can be equivalent to magnitude and gradient based pruning, unifying these seemingly disparate methods, and find that they can be used to shed light on magnitude pruning's performance during the early part of training. △ Less

Submitted 12 March, 2022; v1 submitted 27 October, 2021; originally announced October 2021.

Comments: 14 pages, 5 figures