Showing 1–50 of 51 results for author: Klus, S

Dynamical systems and complex networks: A Koopman operator perspective

Authors: Stefan Klus, Nataša Djurdjevac Conrad

Abstract: The Koopman operator has entered and transformed many research areas over the last years. Although the underlying concept$\unicode{x2013}$representing highly nonlinear dynamical systems by infinite-dimensional linear operators$\unicode{x2013}$has been known for a long time, the availability of large data sets and efficient machine learning algorithms for estimating the Koopman operator from data m… ▽ More The Koopman operator has entered and transformed many research areas over the last years. Although the underlying concept$\unicode{x2013}$representing highly nonlinear dynamical systems by infinite-dimensional linear operators$\unicode{x2013}$has been known for a long time, the availability of large data sets and efficient machine learning algorithms for estimating the Koopman operator from data make this framework extremely powerful and popular. Koopman operator theory allows us to gain insights into the characteristic global properties of a system without requiring detailed mathematical models. We will show how these methods can also be used to analyze complex networks and highlight relationships between Koopman operators and graph Laplacians. △ Less

Submitted 14 May, 2024; originally announced May 2024.

Data-driven approximation of Koopman operators and generators: Convergence rates and error bounds

Authors: Liam Llamazares-Elias, Samir Llamazares-Elias, Jonas Latz, Stefan Klus

Abstract: Global information about dynamical systems can be extracted by analysing associated infinite-dimensional transfer operators, such as Perron-Frobenius and Koopman operators as well as their infinitesimal generators. In practice, these operators typically need to be approximated from data. Popular approximation methods are extended dynamic mode decomposition (EDMD) and generator extended mode decomp… ▽ More Global information about dynamical systems can be extracted by analysing associated infinite-dimensional transfer operators, such as Perron-Frobenius and Koopman operators as well as their infinitesimal generators. In practice, these operators typically need to be approximated from data. Popular approximation methods are extended dynamic mode decomposition (EDMD) and generator extended mode decomposition (gEDMD). We propose a unified framework that leverages Monte Carlo sampling to approximate the operator of interest on a finite-dimensional space spanned by a set of basis functions. Our framework contains EDMD and gEDMD as special cases, but can also be used to approximate more general operators. Our key contributions are proofs of the convergence of the approximating operator and its spectrum under non-restrictive conditions. Moreover, we derive explicit convergence rates and account for the presence of noise in the observations. Whilst all these results are broadly applicable, they also refine previous analyses of EDMD and gEDMD. We verify the analytical results with the aid of several numerical experiments. △ Less

Submitted 19 June, 2024; v1 submitted 1 May, 2024; originally announced May 2024.

Data-driven network analysis using local delay embeddings

Authors: Stefan Klus, Hongyu Zhu

Abstract: Data-driven methods for the identification of the governing equations of dynamical systems or the computation of reduced surrogate models play an increasingly important role in many application areas such as physics, chemistry, biology, and engineering. Given only measurement or observation data, data-driven modeling techniques allow us to gain important insights into the characteristic properties… ▽ More Data-driven methods for the identification of the governing equations of dynamical systems or the computation of reduced surrogate models play an increasingly important role in many application areas such as physics, chemistry, biology, and engineering. Given only measurement or observation data, data-driven modeling techniques allow us to gain important insights into the characteristic properties of a system, without requiring detailed mechanistic models. However, most methods assume that we have access to the full state of the system, which might be too restrictive. We show that it is possible to learn certain global dynamical features from local observations using delay embedding techniques, provided that the system satisfies a localizability condition -- a property that is closely related to the observability and controllability of linear time-invariant systems. △ Less

Submitted 20 June, 2024; v1 submitted 30 January, 2024; originally announced January 2024.

Continuous optimization methods for the graph isomorphism problem

Authors: Stefan Klus, Patrick Gelß

Abstract: The graph isomorphism problem looks deceptively simple, but although polynomial-time algorithms exist for certain types of graphs such as planar graphs and graphs with bounded degree or eigenvalue multiplicity, its complexity class is still unknown. Information about potential isomorphisms between two graphs is contained in the eigenvalues and eigenvectors of their adjacency matrices. However, sym… ▽ More The graph isomorphism problem looks deceptively simple, but although polynomial-time algorithms exist for certain types of graphs such as planar graphs and graphs with bounded degree or eigenvalue multiplicity, its complexity class is still unknown. Information about potential isomorphisms between two graphs is contained in the eigenvalues and eigenvectors of their adjacency matrices. However, symmetries of graphs often lead to repeated eigenvalues so that associated eigenvectors are determined only up to basis rotations, which complicates graph isomorphism testing. We consider orthogonal and doubly stochastic relaxations of the graph isomorphism problem, analyze the geometric properties of the resulting solution spaces, and show that their complexity increases significantly if repeated eigenvalues exist. By restricting the search space to suitable subspaces, we derive an efficient Frank-Wolfe based continuous optimization approach for detecting isomorphisms. We illustrate the efficacy of the algorithm with the aid of various highly symmetric graphs. △ Less

Submitted 28 November, 2023; originally announced November 2023.

Koopman-Based Surrogate Models for Multi-Objective Optimization of Agent-Based Systems

Authors: Jan-Hendrik Niemann, Stefan Klus, Nataša Djurdjevac Conrad, Christof Schütte

Abstract: Agent-based models (ABMs) provide an intuitive and powerful framework for studying social dynamics by modeling the interactions of individuals from the perspective of each individual. In addition to simulating and forecasting the dynamics of ABMs, the demand to solve optimization problems to support, for example, decision-making processes naturally arises. Most ABMs, however, are non-deterministic… ▽ More Agent-based models (ABMs) provide an intuitive and powerful framework for studying social dynamics by modeling the interactions of individuals from the perspective of each individual. In addition to simulating and forecasting the dynamics of ABMs, the demand to solve optimization problems to support, for example, decision-making processes naturally arises. Most ABMs, however, are non-deterministic, high-dimensional dynamical systems, so objectives defined in terms of their behavior are computationally expensive. In particular, if the number of agents is large, evaluating the objective functions often becomes prohibitively time-consuming. We consider data-driven reduced models based on the Koopman generator to enable the efficient solution of multi-objective optimization problems involving ABMs. In a first step, we show how to obtain data-driven reduced models of non-deterministic dynamical systems (such as ABMs) that depend potentially nonlinearly on control inputs. We then use them in the second step as surrogate models to solve multi-objective optimal control problems. We first illustrate our approach using the example of a voter model, where we compute optimal controls to steer the agents to a predetermined majority, and then using the example of an epidemic ABM, where we compute optimal containment strategies in a prototypical situation. We demonstrate that the surrogate models effectively approximate the Pareto-optimal points of the ABM dynamics by comparing the surrogate-based results with test points, where the objectives are evaluated using the ABM. Our results show that when objectives are defined by the dynamic behavior of ABMs, data-driven surrogate models support or even enable the solution of multi-objective optimization problems. △ Less

Submitted 30 January, 2024; v1 submitted 30 June, 2023; originally announced June 2023.

Existence and Uniqueness of Solutions of the Koopman--von Neumann Equation on Bounded Domains

Authors: Marian Stengl, Patrick Gelß, Stefan Klus, Sebastian Pokutta

Abstract: The Koopman--von Neumann equation describes the evolution of a complex-valued wavefunction corresponding to the probability distribution given by an associated classical Liouville equation. Typically, it is defined on the whole Euclidean space. The investigation of bounded domains, particularly in practical scenarios involving quantum-based simulations of dynamical systems, has received little att… ▽ More The Koopman--von Neumann equation describes the evolution of a complex-valued wavefunction corresponding to the probability distribution given by an associated classical Liouville equation. Typically, it is defined on the whole Euclidean space. The investigation of bounded domains, particularly in practical scenarios involving quantum-based simulations of dynamical systems, has received little attention so far. We consider the Koopman--von Neumann equation associated with an ordinary differential equation on a bounded domain whose trajectories are contained in the set's closure. Our main results are the construction of a strongly continuous semigroup together with the existence and uniqueness of solutions of the associated initial value problem. To this end, a functional-analytic framework connected to Sobolev spaces is proposed and analyzed. Moreover, the connection of the Koopman--von Neumann framework to transport equations is highlighted. △ Less

Submitted 23 June, 2023; originally announced June 2023.

MSC Class: 35A05; 35F10; 37C30; 46E35; 47D06

Efficient Approximation of Molecular Kinetics using Random Fourier Features

Authors: Feliks Nüske, Stefan Klus

Abstract: Slow kinetic processes of molecular systems can be analyzed by computing dominant eigenpairs of the Koopman operator or its generator. In this context, the Variational Approach to Markov Processes (VAMP) provides a rigorous way of discerning the quality of different approximate models. Kernel methods have been shown to provide accurate and robust estimates for slow kinetic processes, but are sensi… ▽ More Slow kinetic processes of molecular systems can be analyzed by computing dominant eigenpairs of the Koopman operator or its generator. In this context, the Variational Approach to Markov Processes (VAMP) provides a rigorous way of discerning the quality of different approximate models. Kernel methods have been shown to provide accurate and robust estimates for slow kinetic processes, but are sensitive to hyper-parameter selection, and require the solution of large-scale generalized eigenvalue problems, which can easily become computationally demanding for large data sizes. In this contribution, we employ a stochastic approximation of the kernel based on random Fourier features (RFFs), to derive a small-scale dual eigenvalue problem which can easily be solved. We provide an interpretation of this procedure in terms of a finite randomly generated basis set. By combining the RFF approach and model selection by means of the VAMP score, we show that kernel parameters can be efficiently tuned, and accurate estimates of slow molecular kinetics can be obtained for several benchmarking systems, such as deca alanine and the NTL9 protein. △ Less

Submitted 15 June, 2023; v1 submitted 1 June, 2023; originally announced June 2023.

Transfer operators on graphs: Spectral clustering and beyond

Authors: Stefan Klus, Maia Trower

Abstract: Graphs and networks play an important role in modeling and analyzing complex interconnected systems such as transportation networks, integrated circuits, power grids, citation graphs, and biological and artificial neural networks. Graph clustering algorithms can be used to detect groups of strongly connected vertices and to derive coarse-grained models. We define transfer operators such as the Koo… ▽ More Graphs and networks play an important role in modeling and analyzing complex interconnected systems such as transportation networks, integrated circuits, power grids, citation graphs, and biological and artificial neural networks. Graph clustering algorithms can be used to detect groups of strongly connected vertices and to derive coarse-grained models. We define transfer operators such as the Koopman operator and the Perron-Frobenius operator on graphs, study their spectral properties, introduce Galerkin projections of these operators, and illustrate how reduced representations can be estimated from data. In particular, we show that spectral clustering of undirected graphs can be interpreted in terms of eigenfunctions of the Koopman operator and propose novel clustering algorithms for directed graphs based on generalized transfer operators. We demonstrate the efficacy of the resulting algorithms on several benchmark problems and provide different interpretations of clusters. △ Less

Submitted 14 February, 2024; v1 submitted 19 May, 2023; originally announced May 2023.

Low-rank tensor decompositions of quantum circuits

Authors: Patrick Gelß, Stefan Klus, Sebastian Knebel, Zarin Shakibaei, Sebastian Pokutta

Abstract: Quantum computing is arguably one of the most revolutionary and disruptive technologies of this century. Due to the ever-increasing number of potential applications as well as the continuing rise in complexity, the development, simulation, optimization, and physical realization of quantum circuits is of utmost importance for designing novel algorithms. We show how matrix product states (MPSs) and… ▽ More Quantum computing is arguably one of the most revolutionary and disruptive technologies of this century. Due to the ever-increasing number of potential applications as well as the continuing rise in complexity, the development, simulation, optimization, and physical realization of quantum circuits is of utmost importance for designing novel algorithms. We show how matrix product states (MPSs) and matrix product operators (MPOs) can be used to express certain quantum states, quantum gates, and entire quantum circuits as low-rank tensors. This enables the analysis and simulation of complex quantum circuits on classical computers and to gain insight into the underlying structure of the system. We present different examples to demonstrate the advantages of MPO formulations and show that they are more efficient than conventional techniques if the bond dimensions of the wave function representation can be kept small throughout the simulation. △ Less

Submitted 2 January, 2023; v1 submitted 19 May, 2022; originally announced May 2022.

MSC Class: 15A69; 47N50; 65C05; 68W40; 81P68

Koopman-based spectral clustering of directed and time-evolving graphs

Authors: Stefan Klus, Natasa Djurdjevac Conrad

Abstract: While spectral clustering algorithms for undirected graphs are well established and have been successfully applied to unsupervised machine learning problems ranging from image segmentation and genome sequencing to signal processing and social network analysis, clustering directed graphs remains notoriously difficult. Two of the main challenges are that the eigenvalues and eigenvectors of graph Lap… ▽ More While spectral clustering algorithms for undirected graphs are well established and have been successfully applied to unsupervised machine learning problems ranging from image segmentation and genome sequencing to signal processing and social network analysis, clustering directed graphs remains notoriously difficult. Two of the main challenges are that the eigenvalues and eigenvectors of graph Laplacians associated with directed graphs are in general complex-valued and that there is no universally accepted definition of clusters in directed graphs. We first exploit relationships between the graph Laplacian and transfer operators and in particular between clusters in undirected graphs and metastable sets in stochastic dynamical systems and then use a generalization of the notion of metastability to derive clustering algorithms for directed and time-evolving graphs. The resulting clusters can be interpreted as coherent sets, which play an important role in the analysis of transport and mixing processes in fluid flows. △ Less

Submitted 19 October, 2022; v1 submitted 6 April, 2022; originally announced April 2022.

A Dynamic Mode Decomposition Approach for Decentralized Spectral Clustering of Graphs

Authors: Hongyu Zhu, Stefan Klus, Tuhin Sahai

Abstract: We propose a novel robust decentralized graph clustering algorithm that is provably equivalent to the popular spectral clustering approach. Our proposed method uses the existing wave equation clustering algorithm that is based on propagating waves through the graph. However, instead of using a fast Fourier transform (FFT) computation at every node, our proposed approach exploits the Koopman operat… ▽ More We propose a novel robust decentralized graph clustering algorithm that is provably equivalent to the popular spectral clustering approach. Our proposed method uses the existing wave equation clustering algorithm that is based on propagating waves through the graph. However, instead of using a fast Fourier transform (FFT) computation at every node, our proposed approach exploits the Koopman operator framework. Specifically, we show that propagating waves in the graph followed by a local dynamic mode decomposition (DMD) computation at every node is capable of retrieving the eigenvalues and the local eigenvector components of the graph Laplacian, thereby providing local cluster assignments for all nodes. We demonstrate that the DMD computation is more robust than the existing FFT based approach and requires 20 times fewer steps of the wave equation to accurately recover the clustering information and reduces the relative error by orders of magnitude. We demonstrate the decentralized approach on a range of graph clustering problems. △ Less

Submitted 29 August, 2022; v1 submitted 25 February, 2022; originally announced March 2022.

Comments: Accepted by IEEE CCTA 2022

Journal ref: 2022 IEEE Conference on Control Technology and Applications (CCTA)

Koopman analysis of quantum systems

Authors: Stefan Klus, Feliks Nüske, Sebastian Peitz

Abstract: Koopman operator theory has been successfully applied to problems from various research areas such as fluid dynamics, molecular dynamics, climate science, engineering, and biology. Applications include detecting metastable or coherent sets, coarse-graining, system identification, and control. There is an intricate connection between dynamical systems driven by stochastic differential equations and… ▽ More Koopman operator theory has been successfully applied to problems from various research areas such as fluid dynamics, molecular dynamics, climate science, engineering, and biology. Applications include detecting metastable or coherent sets, coarse-graining, system identification, and control. There is an intricate connection between dynamical systems driven by stochastic differential equations and quantum mechanics. In this paper, we compare the ground-state transformation and Nelson's stochastic mechanics and demonstrate how data-driven methods developed for the approximation of the Koopman operator can be used to analyze quantum physics problems. Moreover, we exploit the relationship between Schrödinger operators and stochastic control problems to show that modern data-driven methods for stochastic control can be used to solve the stationary or imaginary-time Schrödinger equation. Our findings open up a new avenue towards solving Schrödinger's equation using recently developed tools from data science. △ Less

Submitted 28 June, 2022; v1 submitted 28 January, 2022; originally announced January 2022.

Deeptime: a Python library for machine learning dynamical models from time series data

Authors: Moritz Hoffmann, Martin Scherer, Tim Hempel, Andreas Mardt, Brian de Silva, Brooke E. Husic, Stefan Klus, Hao Wu, Nathan Kutz, Steven L. Brunton, Frank Noé

Abstract: Generation and analysis of time-series data is relevant to many quantitative fields ranging from economics to fluid mechanics. In the physical sciences, structures such as metastable and coherent sets, slow relaxation processes, collective variables dominant transition pathways or manifolds and channels of probability flow can be of great importance for understanding and characterizing the kinetic… ▽ More Generation and analysis of time-series data is relevant to many quantitative fields ranging from economics to fluid mechanics. In the physical sciences, structures such as metastable and coherent sets, slow relaxation processes, collective variables dominant transition pathways or manifolds and channels of probability flow can be of great importance for understanding and characterizing the kinetic, thermodynamic and mechanistic properties of the system. Deeptime is a general purpose Python library offering various tools to estimate dynamical models based on time-series data including conventional linear learning methods, such as Markov state models (MSMs), Hidden Markov Models and Koopman models, as well as kernel and deep learning approaches such as VAMPnets and deep MSMs. The library is largely compatible with scikit-learn, having a range of Estimator classes for these different models, but in contrast to scikit-learn also provides deep Model classes, e.g. in the case of an MSM, which provide a multitude of analysis methods to compute interesting thermodynamic, kinetic and dynamical quantities, such as free energies, relaxation times and transition paths. The library is designed for ease of use but also easily maintainable and extensible code. In this paper we introduce the main features and structure of the deeptime software. △ Less

Submitted 11 December, 2021; v1 submitted 28 October, 2021; originally announced October 2021.

Journal ref: Machine Learning: Science and Technology, Volume 3, Number 1, 2021

Symmetric and antisymmetric kernels for machine learning problems in quantum physics and chemistry

Authors: Stefan Klus, Patrick Gelß, Feliks Nüske, Frank Noé

Abstract: We derive symmetric and antisymmetric kernels by symmetrizing and antisymmetrizing conventional kernels and analyze their properties. In particular, we compute the feature space dimensions of the resulting polynomial kernels, prove that the reproducing kernel Hilbert spaces induced by symmetric and antisymmetric Gaussian kernels are dense in the space of symmetric and antisymmetric functions, and… ▽ More We derive symmetric and antisymmetric kernels by symmetrizing and antisymmetrizing conventional kernels and analyze their properties. In particular, we compute the feature space dimensions of the resulting polynomial kernels, prove that the reproducing kernel Hilbert spaces induced by symmetric and antisymmetric Gaussian kernels are dense in the space of symmetric and antisymmetric functions, and propose a Slater determinant representation of the antisymmetric Gaussian kernel, which allows for an efficient evaluation even if the state space is high-dimensional. Furthermore, we show that by exploiting symmetries or antisymmetries the size of the training data set can be significantly reduced. The results are illustrated with guiding examples and simple quantum physics and chemistry applications. △ Less

Submitted 26 June, 2021; v1 submitted 31 March, 2021; originally announced March 2021.

Journal ref: Machine Learning: Science and Technology, Volume 2, Number 4, 2021

Data-driven model reduction of agent-based systems using the Koopman generator

Authors: Jan-Hendrik Niemann, Stefan Klus, Christof Schütte

Abstract: The dynamical behavior of social systems can be described by agent-based models. Although single agents follow easily explainable rules, complex time-evolving patterns emerge due to their interaction. The simulation and analysis of such agent-based models, however, is often prohibitively time-consuming if the number of agents is large. In this paper, we show how Koopman operator theory can be used… ▽ More The dynamical behavior of social systems can be described by agent-based models. Although single agents follow easily explainable rules, complex time-evolving patterns emerge due to their interaction. The simulation and analysis of such agent-based models, however, is often prohibitively time-consuming if the number of agents is large. In this paper, we show how Koopman operator theory can be used to derive reduced models of agent-based systems using only simulation data. Our goal is to learn coarse-grained models and to represent the reduced dynamics by ordinary or stochastic differential equations. The new variables are, for instance, aggregated state variables of the agent-based model, modeling the collective behavior of larger groups or the entire population. Using benchmark problems with known coarse-grained models, we demonstrate that the obtained reduced systems are in good agreement with the analytical results, provided that the numbers of agents is sufficiently large. △ Less

Submitted 11 May, 2021; v1 submitted 14 December, 2020; originally announced December 2020.

Feature space approximation for kernel-based supervised learning

Authors: Patrick Gelß, Stefan Klus, Ingmar Schuster, Christof Schütte

Abstract: We propose a method for the approximation of high- or even infinite-dimensional feature vectors, which play an important role in supervised learning. The goal is to reduce the size of the training data, resulting in lower storage consumption and computational complexity. Furthermore, the method can be regarded as a regularization technique, which improves the generalizability of learned target fun… ▽ More We propose a method for the approximation of high- or even infinite-dimensional feature vectors, which play an important role in supervised learning. The goal is to reduce the size of the training data, resulting in lower storage consumption and computational complexity. Furthermore, the method can be regarded as a regularization technique, which improves the generalizability of learned target functions. We demonstrate significant improvements in comparison to the computation of data-driven predictions involving the full training data set. The method is applied to classification and regression problems from different application areas such as image recognition, system identification, and oceanographic time series analysis. △ Less

Submitted 15 March, 2021; v1 submitted 25 November, 2020; originally announced November 2020.

GraphKKE: Graph Kernel Koopman Embedding for Human Microbiome Analysis

Authors: Kateryna Melnyk, Stefan Klus, Grégoire Montavon, Tim Conrad

Abstract: More and more diseases have been found to be strongly correlated with disturbances in the microbiome constitution, e.g., obesity, diabetes, or some cancer types. Thanks to modern high-throughput omics technologies, it becomes possible to directly analyze human microbiome and its influence on the health status. Microbial communities are monitored over long periods of time and the associations betwe… ▽ More More and more diseases have been found to be strongly correlated with disturbances in the microbiome constitution, e.g., obesity, diabetes, or some cancer types. Thanks to modern high-throughput omics technologies, it becomes possible to directly analyze human microbiome and its influence on the health status. Microbial communities are monitored over long periods of time and the associations between their members are explored. These relationships can be described by a time-evolving graph. In order to understand responses of the microbial community members to a distinct range of perturbations such as antibiotics exposure or diseases and general dynamical properties, the time-evolving graph of the human microbial communities has to be analyzed. This becomes especially challenging due to dozens of complex interactions among microbes and metastable dynamics. The key to solving this problem is the representation of the time-evolving graphs as fixed-length feature vectors preserving the original dynamics. We propose a method for learning the embedding of the time-evolving graph that is based on the spectral analysis of transfer operators and graph kernels. We demonstrate that our method can capture temporary changes in the time-evolving graph on both created synthetic data and real-world data. Our experiments demonstrate the efficacy of the method. Furthermore, we show that our method can be applied to human microbiome data to study dynamic processes. △ Less

Submitted 19 November, 2020; v1 submitted 12 August, 2020; originally announced August 2020.

Kernel-based approximation of the Koopman generator and Schrödinger operator

Authors: Stefan Klus, Feliks Nüske, Boumediene Hamzi

Abstract: Many dimensionality and model reduction techniques rely on estimating dominant eigenfunctions of associated dynamical operators from data. Important examples include the Koopman operator and its generator, but also the Schrödinger operator. We propose a kernel-based method for the approximation of differential operators in reproducing kernel Hilbert spaces and show how eigenfunctions can be estima… ▽ More Many dimensionality and model reduction techniques rely on estimating dominant eigenfunctions of associated dynamical operators from data. Important examples include the Koopman operator and its generator, but also the Schrödinger operator. We propose a kernel-based method for the approximation of differential operators in reproducing kernel Hilbert spaces and show how eigenfunctions can be estimated by solving auxiliary matrix eigenvalue problems. The resulting algorithms are applied to molecular dynamics and quantum chemistry examples. Furthermore, we exploit that, under certain conditions, the Schrödinger operator can be transformed into a Kolmogorov backward operator corresponding to a drift-diffusion process and vice versa. This allows us to apply methods developed for the analysis of high-dimensional stochastic differential equations to quantum mechanical systems. △ Less

Submitted 25 December, 2020; v1 submitted 27 May, 2020; originally announced May 2020.

Kernel Autocovariance Operators of Stationary Processes: Estimation and Convergence

Authors: Mattes Mollenhauer, Stefan Klus, Christof Schütte, Péter Koltai

Abstract: We consider autocovariance operators of a stationary stochastic process on a Polish space that is embedded into a reproducing kernel Hilbert space. We investigate how empirical estimates of these operators converge along realizations of the process under various conditions. In particular, we examine ergodic and strongly mixing processes and obtain several asymptotic results as well as finite sampl… ▽ More We consider autocovariance operators of a stationary stochastic process on a Polish space that is embedded into a reproducing kernel Hilbert space. We investigate how empirical estimates of these operators converge along realizations of the process under various conditions. In particular, we examine ergodic and strongly mixing processes and obtain several asymptotic results as well as finite sample error bounds. We provide applications of our theory in terms of consistency results for kernel PCA with dependent data and the conditional mean embedding of transition probabilities. Finally, we use our approach to examine the nonparametric estimation of Markov transition operators and highlight how our theory can give a consistency analysis for a large family of spectral analysis methods including kernel-based dynamic mode decomposition. △ Less

Submitted 29 November, 2022; v1 submitted 2 April, 2020; originally announced April 2020.

Journal ref: Journal of Machine Learning Research 23(327) 1-34, 2022

Tensor-based algorithms for image classification

Authors: Stefan Klus, Patrick Gelß

Abstract: The interest in machine learning with tensor networks has been growing rapidly in recent years. We show that tensor-based methods developed for learning the governing equations of dynamical systems from data can, in the same way, be used for supervised learning problems and propose two novel approaches for image classification. One is a kernel-based reformulation of the previously introduced MANDy… ▽ More The interest in machine learning with tensor networks has been growing rapidly in recent years. We show that tensor-based methods developed for learning the governing equations of dynamical systems from data can, in the same way, be used for supervised learning problems and propose two novel approaches for image classification. One is a kernel-based reformulation of the previously introduced MANDy (multidimensional approximation of nonlinear dynamics), the other an alternating ridge regression in the tensor-train format. We apply both methods to the MNIST and fashion MNIST data set and show that the approaches are competitive with state-of-the-art neural network-based classifiers. △ Less

Submitted 28 November, 2019; v1 submitted 4 October, 2019; originally announced October 2019.

Journal ref: Algorithms, 12(11), 240, 2019

Data-driven approximation of the Koopman generator: Model reduction, system identification, and control

Authors: Stefan Klus, Feliks Nüske, Sebastian Peitz, Jan-Hendrik Niemann, Cecilia Clementi, Christof Schütte

Abstract: We derive a data-driven method for the approximation of the Koopman generator called gEDMD, which can be regarded as a straightforward extension of EDMD (extended dynamic mode decomposition). This approach is applicable to deterministic and stochastic dynamical systems. It can be used for computing eigenvalues, eigenfunctions, and modes of the generator and for system identification. In addition t… ▽ More We derive a data-driven method for the approximation of the Koopman generator called gEDMD, which can be regarded as a straightforward extension of EDMD (extended dynamic mode decomposition). This approach is applicable to deterministic and stochastic dynamical systems. It can be used for computing eigenvalues, eigenfunctions, and modes of the generator and for system identification. In addition to learning the governing equations of deterministic systems, which then reduces to SINDy (sparse identification of nonlinear dynamics), it is possible to identify the drift and diffusion terms of stochastic differential equations from data. Moreover, we apply gEDMD to derive coarse-grained models of high-dimensional systems, and also to determine efficient model predictive control strategies. We highlight relationships with other methods and demonstrate the efficacy of the proposed methods using several guiding examples and prototypical molecular dynamics problems. △ Less

Submitted 13 February, 2020; v1 submitted 23 September, 2019; originally announced September 2019.

Tensor-based computation of metastable and coherent sets

Authors: Feliks Nüske, Patrick Gelß, Stefan Klus, Cecilia Clementi

Abstract: Recent years have seen rapid advances in the data-driven analysis of dynamical systems based on Koopman operator theory and related approaches. On the other hand, low-rank tensor product approximations -- in particular the tensor train (TT) format -- have become a valuable tool for the solution of large-scale problems in a number of fields. In this work, we combine Koopman-based models and the TT… ▽ More Recent years have seen rapid advances in the data-driven analysis of dynamical systems based on Koopman operator theory and related approaches. On the other hand, low-rank tensor product approximations -- in particular the tensor train (TT) format -- have become a valuable tool for the solution of large-scale problems in a number of fields. In this work, we combine Koopman-based models and the TT format, enabling their application to high-dimensional problems in conjunction with a rich set of basis functions or features. We derive efficient algorithms to obtain a reduced matrix representation of the system's evolution operator starting from an appropriate low-rank representation of the data. These algorithms can be applied to both stationary and non-stationary systems. We establish the infinite-data limit of these matrix representations, and demonstrate our methods' capabilities using several benchmark data sets. △ Less

Submitted 10 August, 2021; v1 submitted 12 August, 2019; originally announced August 2019.

Kernel Conditional Density Operators

Authors: Ingmar Schuster, Mattes Mollenhauer, Stefan Klus, Krikamol Muandet

Abstract: We introduce a novel conditional density estimation model termed the conditional density operator (CDO). It naturally captures multivariate, multimodal output densities and shows performance that is competitive with recent neural conditional density models and Gaussian processes. The proposed model is based on a novel approach to the reconstruction of probability densities from their kernel mean e… ▽ More We introduce a novel conditional density estimation model termed the conditional density operator (CDO). It naturally captures multivariate, multimodal output densities and shows performance that is competitive with recent neural conditional density models and Gaussian processes. The proposed model is based on a novel approach to the reconstruction of probability densities from their kernel mean embeddings by drawing connections to estimation of Radon-Nikodym derivatives in the reproducing kernel Hilbert space (RKHS). We prove finite sample bounds for the estimation error in a standard density reconstruction scenario, independent of problem dimensionality. Interestingly, when a kernel is used that is also a probability density, the CDO allows us to both evaluate and sample the output density efficiently. We demonstrate the versatility and performance of the proposed model on both synthetic and real-world data. △ Less

Submitted 29 October, 2019; v1 submitted 27 May, 2019; originally announced May 2019.

Dimensionality Reduction of Complex Metastable Systems via Kernel Embeddings of Transition Manifolds

Authors: Andreas Bittracher, Stefan Klus, Boumediene Hamzi, Péter Koltai, Christof Schütte

Abstract: We present a novel kernel-based machine learning algorithm for identifying the low-dimensional geometry of the effective dynamics of high-dimensional multiscale stochastic systems. Recently, the authors developed a mathematical framework for the computation of optimal reaction coordinates of such systems that is based on learning a parametrization of a low-dimensional transition manifold in a cert… ▽ More We present a novel kernel-based machine learning algorithm for identifying the low-dimensional geometry of the effective dynamics of high-dimensional multiscale stochastic systems. Recently, the authors developed a mathematical framework for the computation of optimal reaction coordinates of such systems that is based on learning a parametrization of a low-dimensional transition manifold in a certain function space. In this article, we enhance this approach by embedding and learning this transition manifold in a reproducing kernel Hilbert space, exploiting the favorable properties of kernel embeddings. Under mild assumptions on the kernel, the manifold structure is shown to be preserved under the embedding, and distortion bounds can be derived. This leads to a more robust and more efficient algorithm compared to previous parametrization approaches. △ Less

Submitted 3 February, 2020; v1 submitted 18 April, 2019; originally announced April 2019.

Kernel methods for detecting coherent structures in dynamical data

Authors: Stefan Klus, Brooke E. Husic, Mattes Mollenhauer, Frank Noé

Abstract: We illustrate relationships between classical kernel-based dimensionality reduction techniques and eigendecompositions of empirical estimates of reproducing kernel Hilbert space (RKHS) operators associated with dynamical systems. In particular, we show that kernel canonical correlation analysis (CCA) can be interpreted in terms of kernel transfer operators and that it can be obtained by optimizing… ▽ More We illustrate relationships between classical kernel-based dimensionality reduction techniques and eigendecompositions of empirical estimates of reproducing kernel Hilbert space (RKHS) operators associated with dynamical systems. In particular, we show that kernel canonical correlation analysis (CCA) can be interpreted in terms of kernel transfer operators and that it can be obtained by optimizing the variational approach for Markov processes (VAMP) score. As a result, we show that coherent sets of particle trajectories can be computed by kernel CCA. We demonstrate the efficiency of this approach with several examples, namely the well-known Bickley jet, ocean drifter data, and a molecular dynamics problem with a time-dependent potential. Finally, we propose a straightforward generalization of dynamic mode decomposition (DMD) called coherent mode decomposition (CMD). Our results provide a generic machine learning approach to the computation of coherent sets with an objective score that can be used for cross-validation and the comparison of different methods. △ Less

Submitted 7 October, 2019; v1 submitted 16 April, 2019; originally announced April 2019.

On the Equivariance Properties of Self-adjoint Matrices

Authors: Michael Dellnitz, Bennet Gebken, Raphael Gerlach, Stefan Klus

Abstract: We investigate self-adjoint matrices $A\in\mathbb{R}^{n,n}$ with respect to their equivariance properties. We show in particular that a matrix is self-adjoint if and only if it is equivariant with respect to the action of a group $Γ_2(A)\subset \mathbf{O}(n)$ which is isomorphic to $\otimes_{k=1}^n\mathbf{Z}_2$. If the self-adjoint matrix possesses multiple eigenvalues -- this may, for instance, b… ▽ More We investigate self-adjoint matrices $A\in\mathbb{R}^{n,n}$ with respect to their equivariance properties. We show in particular that a matrix is self-adjoint if and only if it is equivariant with respect to the action of a group $Γ_2(A)\subset \mathbf{O}(n)$ which is isomorphic to $\otimes_{k=1}^n\mathbf{Z}_2$. If the self-adjoint matrix possesses multiple eigenvalues -- this may, for instance, be induced by symmetry properties of an underlying dynamical system -- then $A$ is even equivariant with respect to the action of a group $Γ(A) \simeq \prod_{i = 1}^k \mathbf{O}(m_i)$ where $m_1,\ldots,m_k$ are the multiplicities of the eigenvalues $λ_1,\ldots,λ_k$ of $A$. We discuss implications of this result for equivariant bifurcation problems, and we briefly address further applications for the Procrustes problem, graph symmetries and Taylor expansions. △ Less

Submitted 23 September, 2019; v1 submitted 22 February, 2019; originally announced February 2019.

MSC Class: 15B57; 15A24; 37G40; 41A58

Learning chemical reaction networks from trajectory data

Authors: Wei Zhang, Stefan Klus, Tim Conrad, Christof Schütte

Abstract: We develop a data-driven method to learn chemical reaction networks from trajectory data. Modeling the reaction system as a continuous-time Markov chain and assuming the system is fully observed, our method learns the propensity functions of the system with predetermined basis functions by maximizing the likelihood function of the trajectory data under $l^1$ sparse regularization. We demonstrate o… ▽ More We develop a data-driven method to learn chemical reaction networks from trajectory data. Modeling the reaction system as a continuous-time Markov chain and assuming the system is fully observed, our method learns the propensity functions of the system with predetermined basis functions by maximizing the likelihood function of the trajectory data under $l^1$ sparse regularization. We demonstrate our method with numerical examples using synthetic data and carry out an asymptotic analysis of the proposed learning procedure in the infinite-data limit. △ Less

Submitted 28 September, 2019; v1 submitted 13 February, 2019; originally announced February 2019.

MSC Class: 92C42; 62M86

Journal ref: Siam J. Appl. Dyn. Syst., 18(4), pp. 2000-2046, 2019

A kernel-based approach to molecular conformation analysis

Authors: Stefan Klus, Andreas Bittracher, Ingmar Schuster, Christof Schütte

Abstract: We present a novel machine learning approach to understanding conformation dynamics of biomolecules. The approach combines kernel-based techniques that are popular in the machine learning community with transfer operator theory for analyzing dynamical systems in order to identify conformation dynamics based on molecular dynamics simulation data. We show that many of the prominent methods like Mark… ▽ More We present a novel machine learning approach to understanding conformation dynamics of biomolecules. The approach combines kernel-based techniques that are popular in the machine learning community with transfer operator theory for analyzing dynamical systems in order to identify conformation dynamics based on molecular dynamics simulation data. We show that many of the prominent methods like Markov State Models, EDMD, and TICA can be regarded as special cases of this approach and that new efficient algorithms can be constructed based on this derivation. The results of these new powerful methods will be illustrated with several examples, in particular the alanine dipeptide and the protein NTL9. △ Less

Submitted 4 December, 2018; v1 submitted 28 September, 2018; originally announced September 2018.

Journal ref: Journal of Chemical Physics 149, 244109, 2018

Multidimensional approximation of nonlinear dynamical systems

Authors: Patrick Gelß, Stefan Klus, Jens Eisert, Christof Schütte

Abstract: A key task in the field of modeling and analyzing nonlinear dynamical systems is the recovery of unknown governing equations from measurement data only. There is a wide range of application areas for this important instance of system identification, ranging from industrial engineering and acoustic signal processing to stock market models. In order to find appropriate representations of underlying… ▽ More A key task in the field of modeling and analyzing nonlinear dynamical systems is the recovery of unknown governing equations from measurement data only. There is a wide range of application areas for this important instance of system identification, ranging from industrial engineering and acoustic signal processing to stock market models. In order to find appropriate representations of underlying dynamical systems, various data-driven methods have been proposed by different communities. However, if the given data sets are high-dimensional, then these methods typically suffer from the curse of dimensionality. To significantly reduce the computational costs and storage consumption, we propose the method MANDy which combines data-driven methods with tensor network decompositions. The efficiency of the introduced approach will be illustrated with the aid of several high-dimensional nonlinear dynamical systems. △ Less

Submitted 1 March, 2019; v1 submitted 7 September, 2018; originally announced September 2018.

MSC Class: 15A69; 34C15; 62J05; 65E05; 93B30

Journal ref: J. Comput. Nonlinear Dynam. 14, 061006 (2019)

Singular Value Decomposition of Operators on Reproducing Kernel Hilbert Spaces

Authors: Mattes Mollenhauer, Ingmar Schuster, Stefan Klus, Christof Schütte

Abstract: Reproducing kernel Hilbert spaces (RKHSs) play an important role in many statistics and machine learning applications ranging from support vector machines to Gaussian processes and kernel embeddings of distributions. Operators acting on such spaces are, for instance, required to embed conditional probability distributions in order to implement the kernel Bayes rule and build sequential data models… ▽ More Reproducing kernel Hilbert spaces (RKHSs) play an important role in many statistics and machine learning applications ranging from support vector machines to Gaussian processes and kernel embeddings of distributions. Operators acting on such spaces are, for instance, required to embed conditional probability distributions in order to implement the kernel Bayes rule and build sequential data models. It was recently shown that transfer operators such as the Perron-Frobenius or Koopman operator can also be approximated in a similar fashion using covariance and cross-covariance operators and that eigenfunctions of these operators can be obtained by solving associated matrix eigenvalue problems. The goal of this paper is to provide a solid functional analytic foundation for the eigenvalue decomposition of RKHS operators and to extend the approach to the singular value decomposition. The results are illustrated with simple guiding examples. △ Less

Submitted 16 March, 2020; v1 submitted 24 July, 2018; originally announced July 2018.

Feedback control of nonlinear PDEs using data-efficient reduced order models based on the Koopman operator

Authors: Sebastian Peitz, Stefan Klus

Abstract: In the development of model predictive controllers for PDE-constrained problems, the use of reduced order models is essential to enable real-time applicability. Besides local linearization approaches, Proper Orthogonal Decomposition (POD) has been most widely used in the past in order to derive such models. Due to the huge advances concerning both theory as well as the numerical approximation, a v… ▽ More In the development of model predictive controllers for PDE-constrained problems, the use of reduced order models is essential to enable real-time applicability. Besides local linearization approaches, Proper Orthogonal Decomposition (POD) has been most widely used in the past in order to derive such models. Due to the huge advances concerning both theory as well as the numerical approximation, a very promising alternative based on the Koopman operator has recently emerged. In this chapter, we present two control strategies for model predictive control of nonlinear PDEs using data-efficient approximations of the Koopman operator. In the first one, the dynamic control system is replaced by a small number of autonomous systems with different yet constant inputs. The control problem is consequently transformed into a switching problem. In the second approach, a bilinear surrogate model, is obtained via linear interpolation between two of these autonomous systems. Using a recent convergence result for Extended Dynamic Mode Decomposition (EDMD), convergence to the true optimum can be proved. We study the properties of these two strategies with respect to solution quality, data requirements, and complexity of the resulting optimization problem using the 1D Burgers Equation and the 2D Navier-Stokes Equations as examples. Finally, an extension for online adaptivity is presented. △ Less

Submitted 26 June, 2018; originally announced June 2018.

Analyzing high-dimensional time-series data using kernel transfer operator eigenfunctions

Authors: Stefan Klus, Sebastian Peitz, Ingmar Schuster

Abstract: Kernel transfer operators, which can be regarded as approximations of transfer operators such as the Perron-Frobenius or Koopman operator in reproducing kernel Hilbert spaces, are defined in terms of covariance and cross-covariance operators and have been shown to be closely related to the conditional mean embedding framework developed by the machine learning community. The goal of this paper is t… ▽ More Kernel transfer operators, which can be regarded as approximations of transfer operators such as the Perron-Frobenius or Koopman operator in reproducing kernel Hilbert spaces, are defined in terms of covariance and cross-covariance operators and have been shown to be closely related to the conditional mean embedding framework developed by the machine learning community. The goal of this paper is to show how the dominant eigenfunctions of these operators in combination with gradient-based optimization techniques can be used to detect long-lived coherent patterns in high-dimensional time-series data. The results will be illustrated using video data and a fluid flow example. △ Less

Submitted 16 May, 2018; originally announced May 2018.

Koopman Operator-Based Finite-Control-Set Model Predictive Control for Electrical Drives

Authors: Sören Hanke, Sebastian Peitz, Oliver Wallscheid, Stefan Klus, Joachim Böcker, Michael Dellnitz

Abstract: Predictive control of power electronic systems always requires a suitable model of the plant. Using typical physics-based white box models, a trade-off between model complexity (i.e. accuracy) and computational burden has to be made. This is a challenging task with a lot of constraints, since the model order is directly linked to the number of system states. Even though white-box models show suita… ▽ More Predictive control of power electronic systems always requires a suitable model of the plant. Using typical physics-based white box models, a trade-off between model complexity (i.e. accuracy) and computational burden has to be made. This is a challenging task with a lot of constraints, since the model order is directly linked to the number of system states. Even though white-box models show suitable performance in most cases, parasitic real-world effects often cannot be modeled satisfactorily with an expedient computational load. Hence, a Koopman operator-based model reduction technique is presented which directly links the control action to the system's outputs in a black-box fashion. The Koopman operator is a linear but infinite-dimensional operator describing the dynamics of observables of nonlinear autonomous dynamical systems which can be nicely applied to the switching principle of power electronic devices. Following this data-driven approach, the model order and the number of system states are decoupled which allows us to consider more complex systems. Extensive experimental tests with an automotive-type permanent magnet synchronous motor fed by an IGBT 2-level inverter prove the feasibility of the proposed modeling technique in a finite-set model predictive control application. △ Less

Submitted 27 September, 2019; v1 submitted 3 April, 2018; originally announced April 2018.

Eigendecompositions of Transfer Operators in Reproducing Kernel Hilbert Spaces

Authors: Stefan Klus, Ingmar Schuster, Krikamol Muandet

Abstract: Transfer operators such as the Perron--Frobenius or Koopman operator play an important role in the global analysis of complex dynamical systems. The eigenfunctions of these operators can be used to detect metastable sets, to project the dynamics onto the dominant slow processes, or to separate superimposed signals. We extend transfer operator theory to reproducing kernel Hilbert spaces and show th… ▽ More Transfer operators such as the Perron--Frobenius or Koopman operator play an important role in the global analysis of complex dynamical systems. The eigenfunctions of these operators can be used to detect metastable sets, to project the dynamics onto the dominant slow processes, or to separate superimposed signals. We extend transfer operator theory to reproducing kernel Hilbert spaces and show that these operators are related to Hilbert space representations of conditional distributions, known as conditional mean embeddings in the machine learning community. Moreover, numerical methods to compute empirical estimates of these embeddings are akin to data-driven methods for the approximation of transfer operators such as extended dynamic mode decomposition and its variants. One main benefit of the presented kernel-based approaches is that these methods can be applied to any domain where a similarity measure given by a kernel is available. We illustrate the results with the aid of guiding examples and highlight potential applications in molecular dynamics as well as video and text data analysis. △ Less

Submitted 13 August, 2019; v1 submitted 5 December, 2017; originally announced December 2017.

Journal ref: Journal of Nonlinear Science, 2019

Koopman operator-based model reduction for switched-system control of PDEs

Authors: Sebastian Peitz, Stefan Klus

Abstract: We present a new framework for optimal and feedback control of PDEs using Koopman operator-based reduced order models (K-ROMs). The Koopman operator is a linear but infinite-dimensional operator which describes the dynamics of observables. A numerical approximation of the Koopman operator therefore yields a linear system for the observation of an autonomous dynamical system. In our approach, by in… ▽ More We present a new framework for optimal and feedback control of PDEs using Koopman operator-based reduced order models (K-ROMs). The Koopman operator is a linear but infinite-dimensional operator which describes the dynamics of observables. A numerical approximation of the Koopman operator therefore yields a linear system for the observation of an autonomous dynamical system. In our approach, by introducing a finite number of constant controls, the dynamic control system is transformed into a set of autonomous systems and the corresponding optimal control problem into a switching time optimization problem. This allows us to replace each of these systems by a K-ROM which can be solved orders of magnitude faster. By this approach, a nonlinear infinite-dimensional control problem is transformed into a low-dimensional linear problem. In situations where the Koopman operator can be computed exactly using Extended Dynamic Mode Decomposition (EDMD), the proposed approach yields optimal control inputs. Furthermore, a recent convergence result for EDMD suggests that the approach can be applied to more complex dynamics as well. To illustrate the results, we consider the 1D Burgers equation and the 2D Navier--Stokes equations. The numerical experiments show remarkable performance concerning both solution times and accuracy. △ Less

Submitted 18 January, 2019; v1 submitted 18 October, 2017; originally announced October 2017.

Comments: arXiv admin note: text overlap with arXiv:1801.06419

Diffusion maps tailored to arbitrary non-degenerate Ito processes

Authors: Ralf Banisch, Zofia Trstanova, Andreas Bittracher, Stefan Klus, Peter Koltai

Abstract: We present two generalizations of the popular diffusion maps algorithm. The first generalization replaces the drift term in diffusion maps, which is the gradient of the sampling density, with the gradient of an arbitrary density of interest which is known up to a normalization constant. The second generalization allows for a diffusion map type approximation of the forward and backward generators o… ▽ More We present two generalizations of the popular diffusion maps algorithm. The first generalization replaces the drift term in diffusion maps, which is the gradient of the sampling density, with the gradient of an arbitrary density of interest which is known up to a normalization constant. The second generalization allows for a diffusion map type approximation of the forward and backward generators of general Ito diffusions with given drift and diffusion coefficients. We use the local kernels introduced by Berry and Sauer, but allow for arbitrary sampling densities. We provide numerical illustrations to demonstrate that this opens up many new applications for diffusion maps as a tool to organize point cloud data, including biased or corrupted samples, dimension reduction for dynamical systems, detection of almost invariant regions in flow fields, and importance sampling. △ Less

Submitted 10 October, 2017; originally announced October 2017.

Transition manifolds of complex metastable systems: Theory and data-driven computation of effective dynamics

Authors: Andreas Bittracher, Péter Koltai, Stefan Klus, Ralf Banisch, Michael Dellnitz, Christof Schütte

Abstract: We consider complex dynamical systems showing metastable behavior but no local separation of fast and slow time scales. The article raises the question of whether such systems exhibit a low-dimensional manifold supporting its effective dynamics. For answering this question, we aim at finding nonlinear coordinates, called reaction coordinates, such that the projection of the dynamics onto these coo… ▽ More We consider complex dynamical systems showing metastable behavior but no local separation of fast and slow time scales. The article raises the question of whether such systems exhibit a low-dimensional manifold supporting its effective dynamics. For answering this question, we aim at finding nonlinear coordinates, called reaction coordinates, such that the projection of the dynamics onto these coordinates preserves the dominant time scales of the dynamics. We show that, based on a specific reducibility property, the existence of good low-dimensional reaction coordinates preserving the dominant time scales is guaranteed. Based on this theoretical framework, we develop and test a novel numerical approach for computing good reaction coordinates. The proposed algorithmic approach is fully local and thus not prone to the curse of dimension with respect to the state space of the dynamics. Hence, it is a promising method for data-based model reduction of complex dynamical systems such as molecular dynamics. △ Less

Submitted 18 September, 2017; v1 submitted 28 April, 2017; originally announced April 2017.

Journal ref: Journal of Nonlinear Science, 28(2):471-512, 2018

Data-driven model reduction and transfer operator approximation

Authors: Stefan Klus, Feliks Nüske, Péter Koltai, Hao Wu, Ioannis Kevrekidis, Christof Schütte, Frank Noé

Abstract: In this review paper, we will present different data-driven dimension reduction techniques for dynamical systems that are based on transfer operator theory as well as methods to approximate transfer operators and their eigenvalues, eigenfunctions, and eigenmodes. The goal is to point out similarities and differences between methods developed independently by the dynamical systems, fluid dynamics,… ▽ More In this review paper, we will present different data-driven dimension reduction techniques for dynamical systems that are based on transfer operator theory as well as methods to approximate transfer operators and their eigenvalues, eigenfunctions, and eigenmodes. The goal is to point out similarities and differences between methods developed independently by the dynamical systems, fluid dynamics, and molecular dynamics communities such as time-lagged independent component analysis (TICA), dynamic mode decomposition (DMD), and their respective generalizations. As a result, extensions and best practices developed for one particular method can be carried over to other related methods. △ Less

Submitted 18 September, 2017; v1 submitted 29 March, 2017; originally announced March 2017.

Journal ref: Journal of Nonlinear Science, 28(3):985-1010, 2018

Continuous Relaxations for the Traveling Salesman Problem

Authors: Tuhin Sahai, Adrian Ziessler, Stefan Klus, Michael Dellnitz

Abstract: In this work, we aim to explore connections between dynamical systems techniques and combinatorial optimization problems. In particular, we construct heuristic approaches for the traveling salesman problem (TSP) based on embedding the relaxed discrete optimization problem into appropriate manifolds. We explore multiple embedding techniques -- namely, the construction of new dynamical systems on th… ▽ More In this work, we aim to explore connections between dynamical systems techniques and combinatorial optimization problems. In particular, we construct heuristic approaches for the traveling salesman problem (TSP) based on embedding the relaxed discrete optimization problem into appropriate manifolds. We explore multiple embedding techniques -- namely, the construction of new dynamical systems on the manifold of orthogonal matrices and associated Procrustes approximations of the TSP cost function. Using these dynamical systems, we analyze the local neighborhood around the optimal TSP solutions (which are equilibria) using computations to approximate the associated \emph{stable manifolds}. We find that these flows frequently converge to undesirable equilibria. However, the solutions of the dynamical systems and the associated Procrustes approximation provide an interesting biasing approach for the popular Lin--Kernighan heuristic which yields fast convergence. The Lin--Kernighan heuristic is typically based on the computation of edges that have a `high probability' of being in the shortest tour, thereby effectively pruning the search space. Our new approach, instead, relies on a natural relaxation of the combinatorial optimization problem to the manifold of orthogonal matrices and the subsequent use of this solution to bias the Lin--Kernighan heuristic. Although the initial cost of computing these edges using the Procrustes solution is higher than existing methods, we find that the Procrustes solution, when coupled with a homotopy computation, contains valuable information regarding the optimal edges. We explore the Procrustes based approach on several TSP instances and find that our approach often requires fewer $k$-opt moves than existing approaches. Broadly, we hope that this work initiates more work in the intersection of dynamical systems theory and combinatorial optimization. △ Less

Submitted 11 February, 2019; v1 submitted 17 February, 2017; originally announced February 2017.

Nearest-Neighbor Interaction Systems in the Tensor-Train Format

Authors: Patrick Gelß, Stefan Klus, Sebastian Matera, Christof Schütte

Abstract: Low-rank tensor approximation approaches have become an important tool in the scientific computing community. The aim is to enable the simulation and analysis of high-dimensional problems which cannot be solved using conventional methods anymore due to the so-called curse of dimensionality. This requires techniques to handle linear operators defined on extremely large state spaces and to solve the… ▽ More Low-rank tensor approximation approaches have become an important tool in the scientific computing community. The aim is to enable the simulation and analysis of high-dimensional problems which cannot be solved using conventional methods anymore due to the so-called curse of dimensionality. This requires techniques to handle linear operators defined on extremely large state spaces and to solve the resulting systems of linear equations or eigenvalue problems. In this paper, we present a systematic tensor-train decomposition for nearest-neighbor interaction systems which is applicable to a host of different problems. With the aid of this decomposition, it is possible to reduce the memory consumption as well as the computational costs significantly. Furthermore, it can be shown that in some cases the rank of the tensor decomposition does not depend on the network size. The format is thus feasible even for high-dimensional systems. We will illustrate the results with several guiding examples such as the Ising model, a system of coupled oscillators, and a CO oxidation model. △ Less

Submitted 4 April, 2017; v1 submitted 11 November, 2016; originally announced November 2016.

MSC Class: 15A69; 34C15; 60J27; 60K20; 80A30; 82C20

Journal ref: Journal of Computational Physics, Volume 341, Pages 140-162, 2017

Variational Koopman models: slow collective variables and molecular kinetics from short off-equilibrium simulations

Authors: Hao Wu, Feliks Nüske, Fabian Paul, Stefan Klus, Peter Koltai, Frank Noé

Abstract: Markov state models (MSMs) and Master equation models are popular approaches to approximate molecular kinetics, equilibria, metastable states, and reaction coordinates in terms of a state space discretization usually obtained by clustering. Recently, a powerful generalization of MSMs has been introduced, the variational approach (VA) of molecular kinetics and its special case the time-lagged indep… ▽ More Markov state models (MSMs) and Master equation models are popular approaches to approximate molecular kinetics, equilibria, metastable states, and reaction coordinates in terms of a state space discretization usually obtained by clustering. Recently, a powerful generalization of MSMs has been introduced, the variational approach (VA) of molecular kinetics and its special case the time-lagged independent component analysis (TICA), which allow us to approximate slow collective variables and molecular kinetics by linear combinations of smooth basis functions or order parameters. While it is known how to estimate MSMs from trajectories whose starting points are not sampled from an equilibrium ensemble, this has not yet been the case for TICA and the VA. Previous estimates from short trajectories, have been strongly biased and thus not variationally optimal. Here, we employ Koopman operator theory and ideas from dynamic mode decomposition (DMD) to extend the VA and TICA to non-equilibrium data. The main insight is that the VA and TICA provide a coefficient matrix that we call Koopman model, as it approximates the underlying dynamical (Koopman) operator in conjunction with the basis set used. This Koopman model can be used to compute a stationary vector to reweight the data to equilibrium. From such a Koopman-reweighted sample, equilibrium expectation values and variationally optimal reversible Koopman models can be constructed even with short simulations. The Koopman model can be used to propagate densities, and its eigenvalue decomposition provide estimates of relaxation timescales and slow collective variables for dimension reduction. Koopman models are generalizations of Markov state models, TICA and the linear VA and allow molecular kinetics to be described without a cluster discretization. △ Less

Submitted 22 January, 2017; v1 submitted 20 October, 2016; originally announced October 2016.

Tensor-based dynamic mode decomposition

Authors: Stefan Klus, Patrick Gelß, Sebastian Peitz, Christof Schütte

Abstract: Dynamic mode decomposition (DMD) is a recently developed tool for the analysis of the behavior of complex dynamical systems. In this paper, we will propose an extension of DMD that exploits low-rank tensor decompositions of potentially high-dimensional data sets to compute the corresponding DMD modes and eigenvalues. The goal is to reduce the computational complexity and also the amount of memory… ▽ More Dynamic mode decomposition (DMD) is a recently developed tool for the analysis of the behavior of complex dynamical systems. In this paper, we will propose an extension of DMD that exploits low-rank tensor decompositions of potentially high-dimensional data sets to compute the corresponding DMD modes and eigenvalues. The goal is to reduce the computational complexity and also the amount of memory required to store the data in order to mitigate the curse of dimensionality. The efficiency of these tensor-based methods will be illustrated with the aid of several different fluid dynamics problems such as the von Kármán vortex street and the simulation of two merging vortices. △ Less

Submitted 18 January, 2017; v1 submitted 21 June, 2016; originally announced June 2016.

MSC Class: 15A69; 37N10; 37L65

A Set-Oriented Numerical Approach for Dynamical Systems with Parameter Uncertainty

Authors: Michael Dellnitz, Stefan Klus, Adrian Ziessler

Abstract: In this article, we develop a set-oriented numerical methodology which allows to perform uncertainty quantification (UQ) for dynamical systems from a global point of view. That is, for systems with uncertain parameters we approximate the corresponding global attractors and invariant measures in the related stochastic setting. Our methods do not rely on generalized polynomial chaos techniques. Rath… ▽ More In this article, we develop a set-oriented numerical methodology which allows to perform uncertainty quantification (UQ) for dynamical systems from a global point of view. That is, for systems with uncertain parameters we approximate the corresponding global attractors and invariant measures in the related stochastic setting. Our methods do not rely on generalized polynomial chaos techniques. Rather, we extend classical set-oriented methods designed for deterministic dynamical systems to the UQ-context, and this allows us to analyze the long-term uncertainty propagation. The algorithms have been integrated into the software package GAIO, and we illustrate the use and efficiency of these techniques by a couple of numerical examples. △ Less

Submitted 25 April, 2016; originally announced April 2016.

Journal ref: SIAM Journal on Applied Dynamical Systems 16(1), 120-138, 2017

Towards tensor-based methods for the numerical approximation of the Perron-Frobenius and Koopman operator

Authors: Stefan Klus, Christof Schütte

Abstract: The global behavior of dynamical systems can be studied by analyzing the eigenvalues and corresponding eigenfunctions of linear operators associated with the system. Two important operators which are frequently used to gain insight into the system's behavior are the Perron-Frobenius operator and the Koopman operator. Due to the curse of dimensionality, computing the eigenfunctions of high-dimensio… ▽ More The global behavior of dynamical systems can be studied by analyzing the eigenvalues and corresponding eigenfunctions of linear operators associated with the system. Two important operators which are frequently used to gain insight into the system's behavior are the Perron-Frobenius operator and the Koopman operator. Due to the curse of dimensionality, computing the eigenfunctions of high-dimensional systems is in general infeasible. We will propose a tensor-based reformulation of two numerical methods for computing finite-dimensional approximations of the aforementioned infinite-dimensional operators, namely Ulam's method and Extended Dynamic Mode Decomposition (EDMD). The aim of the tensor formulation is to approximate the eigenfunctions by low-rank tensors, potentially resulting in a significant reduction of the time and memory required to solve the resulting eigenvalue problems, provided that such a low-rank tensor decomposition exists. Typically, not all variables of a high-dimensional dynamical system contribute equally to the system's behavior, often the dynamics can be decomposed into slow and fast processes, which is also reflected in the eigenfunctions. Thus, the weak coupling between different variables might be approximated by low-rank tensor cores. We will illustrate the efficiency of the tensor-based formulation of Ulam's method and EDMD using simple stochastic differential equations. △ Less

Submitted 10 November, 2016; v1 submitted 21 December, 2015; originally announced December 2015.

On the numerical approximation of the Perron-Frobenius and Koopman operator

Authors: Stefan Klus, Péter Koltai, Christof Schütte

Abstract: Information about the behavior of dynamical systems can often be obtained by analyzing the eigenvalues and corresponding eigenfunctions of linear operators associated with a dynamical system. Examples of such operators are the Perron-Frobenius and the Koopman operator. In this paper, we will review different methods that have been developed over the last decades to compute finite-dimensional appro… ▽ More Information about the behavior of dynamical systems can often be obtained by analyzing the eigenvalues and corresponding eigenfunctions of linear operators associated with a dynamical system. Examples of such operators are the Perron-Frobenius and the Koopman operator. In this paper, we will review different methods that have been developed over the last decades to compute finite-dimensional approximations of these infinite-dimensional operators - e.g. Ulam's method and Extended Dynamic Mode Decomposition (EDMD) - and highlight the similarities and differences between these approaches. The results will be illustrated using simple stochastic differential equations and molecular dynamics examples. △ Less

Submitted 20 October, 2016; v1 submitted 18 December, 2015; originally announced December 2015.

MSC Class: 34F05; 37M10; 37M25

Journal ref: Journal of Computational Dynamics, Volume 3, Issue 1, pp. 51 - 79, 2016

Sensing and Control in Symmetric Networks

Authors: Michael Dellnitz, Stefan Klus

Abstract: In engineering applications, one of the major challenges today is to develop reliable and robust control algorithms for complex networked systems. Controllability and observability of such systems play a crucial role in the design process. The underlying network structure may contain symmetries -- caused for example by the coupling of identical building blocks -- and these symmetries lead to repea… ▽ More In engineering applications, one of the major challenges today is to develop reliable and robust control algorithms for complex networked systems. Controllability and observability of such systems play a crucial role in the design process. The underlying network structure may contain symmetries -- caused for example by the coupling of identical building blocks -- and these symmetries lead to repeated eigenvalues in a generic way. This complicates the design of controllers since repeated eigenvalues might decrease the controllability of the system. In this paper, we will analyze the relationship between the controllability and observability of complex networked systems and graph symmetries using results from representation theory. Furthermore, we will propose an algorithm to compute sparse input and output matrices based on projections onto the isotypic components. We will illustrate our results with the aid of two guiding examples, a network with $ D_4 $ symmetry and the Petersen graph. △ Less

Submitted 29 July, 2015; originally announced July 2015.

Sparse Automatic Differentiation for Complex Networks of Differential-Algebraic Equations Using Abstract Elementary Algebra

Authors: Slaven Peles, Stefan Klus

Abstract: Most numerical solvers and libraries nowadays are implemented to use mathematical models created with language-specific built-in data types (e.g. real in Fortran or double in C) and their respective elementary algebra implementations. However, the built-in elementary algebra typically has limited functionality and often restricts the flexibility of mathematical models and the analysis types that c… ▽ More Most numerical solvers and libraries nowadays are implemented to use mathematical models created with language-specific built-in data types (e.g. real in Fortran or double in C) and their respective elementary algebra implementations. However, the built-in elementary algebra typically has limited functionality and often restricts the flexibility of mathematical models and the analysis types that can be applied to those models. To overcome this limitation, a number of domain-specific languages such as gPROMS or Modelica with more feature-rich built-in data types have been proposed. In this paper, we argue that if numerical libraries and solvers are designed to use abstract elementary algebra rather than the language-specific built-in algebra, modern mainstream languages can be as effective as any domain-specific language. We illustrate our ideas using the example of sparse Jacobian matrix computation. We implement an automatic differentiation method that takes advantage of sparse system structures and is straightforward to parallelize in a distributed memory setting. Furthermore, we show that the computational cost scales linearly with the size of the system. △ Less

Submitted 12 May, 2021; v1 submitted 4 May, 2015; originally announced May 2015.

Journal ref: International Journal of Numerical Analysis and Modeling. 14 (6). pp. 916-934 (2017)

Signal-Flow Based Runge-Kutta Methods for the Simulation of Complex Networks

Authors: Stefan Klus

Abstract: Complex dynamical networks appear in a wide range of physical, biological, and engineering systems. The coupling of subsystems with varying time scales often results in multirate behavior. During the simulation of highly integrated circuits, for example, only a few elements underlie changing signals whereas the major part -- usually up to 80 or even 90 per cent -- remains latent. Standard integrat… ▽ More Complex dynamical networks appear in a wide range of physical, biological, and engineering systems. The coupling of subsystems with varying time scales often results in multirate behavior. During the simulation of highly integrated circuits, for example, only a few elements underlie changing signals whereas the major part -- usually up to 80 or even 90 per cent -- remains latent. Standard integration schemes discretize the entire circuit with a single step size which is mainly limited by the accuracy requirements of the rapidly changing subcircuits. It is of a particular interest to speed up the simulation without a significant loss of accuracy. By exploiting the latency of the system, only a fraction of the equations has to be formulated and solved at a given time point. Günther and Rentrop suggest that multirate strategies must be based both on the numerical information of the integration scheme and on the topology of the circuit. In this paper, we will introduce a directed graph describing the interdependency of the underlying system and propose Runge--Kutta methods which utilize the signal flow of the system in order to identify and exploit inactive regions. Furthermore, we describe an extension of these methods to identify and exploit periodic subsystems. △ Less

Submitted 24 April, 2015; originally announced April 2015.

A Spectral Assignment Approach for the Graph Isomorphism Problem

Authors: Stefan Klus, Tuhin Sahai

Abstract: In this paper, we propose algorithms for the graph isomorphism (GI) problem that are based on the eigendecompositions of the adjacency matrices. The eigenvalues of isomorphic graphs are identical. However, two graphs $ G_A $ and $ G_B $ can be isospectral but non-isomorphic. We first construct a graph isomorphism testing algorithm for friendly graphs and then extend it to unambiguous graphs. We sh… ▽ More In this paper, we propose algorithms for the graph isomorphism (GI) problem that are based on the eigendecompositions of the adjacency matrices. The eigenvalues of isomorphic graphs are identical. However, two graphs $ G_A $ and $ G_B $ can be isospectral but non-isomorphic. We first construct a graph isomorphism testing algorithm for friendly graphs and then extend it to unambiguous graphs. We show that isomorphisms can be detected by solving a linear assignment problem. If the graphs possess repeated eigenvalues, which typically correspond to graph symmetries, finding isomorphisms is much harder. By repeatedly perturbing the adjacency matrices and by using properties of eigenpolytopes, it is possible to break symmetries of the graphs and iteratively assign vertices of $ G_A $ to vertices of $ G_B $, provided that an admissible assignment exists. This heuristic approach can be used to construct a permutation which transforms $ G_A $ into $ G_B $ if the graphs are isomorphic. The methods will be illustrated with several guiding examples. △ Less

Submitted 19 December, 2017; v1 submitted 4 November, 2014; originally announced November 2014.

MSC Class: 05C60; 05C50

A Traveling Salesman Learns Bayesian Networks

Authors: Tuhin Sahai, Stefan Klus, Michael Dellnitz

Abstract: Structure learning of Bayesian networks is an important problem that arises in numerous machine learning applications. In this work, we present a novel approach for learning the structure of Bayesian networks using the solution of an appropriately constructed traveling salesman problem. In our approach, one computes an optimal ordering (partially ordered set) of random variables using methods for… ▽ More Structure learning of Bayesian networks is an important problem that arises in numerous machine learning applications. In this work, we present a novel approach for learning the structure of Bayesian networks using the solution of an appropriately constructed traveling salesman problem. In our approach, one computes an optimal ordering (partially ordered set) of random variables using methods for the traveling salesman problem. This ordering significantly reduces the search space for the subsequent greedy optimization that computes the final structure of the Bayesian network. We demonstrate our approach of learning Bayesian networks on real world census and weather datasets. In both cases, we demonstrate that the approach very accurately captures dependencies between random variables. We check the accuracy of the predictions based on independent studies in both application domains. △ Less

Submitted 20 November, 2012; originally announced November 2012.