-
A Koopman Operator-Based Prediction Algorithm and its Application to COVID-19 Pandemic
Authors:
Igor Mezic,
Zlatko Drmac,
Nelida Crnjaric-Zic,
Senka Macesic,
Maria Fonoberova,
Ryan Mohr,
Allan Avila,
Iva Manojlovic,
Aleksandr Andrejcuk
Abstract:
The problem of prediction of behavior of dynamical systems has undergone a paradigm shift in the second half of the 20th century with the discovery of the possibility of chaotic dynamics in simple, physical, dynamical systems for which the laws of evolution do not change in time. The essence of the paradigm is the long term exponential divergence of trajectories. However, that paradigm does not ac…
▽ More
The problem of prediction of behavior of dynamical systems has undergone a paradigm shift in the second half of the 20th century with the discovery of the possibility of chaotic dynamics in simple, physical, dynamical systems for which the laws of evolution do not change in time. The essence of the paradigm is the long term exponential divergence of trajectories. However, that paradigm does not account for another type of unpredictability: the ``Black Swan" event. It also does not account for the fact that short-term prediction is often possible even in systems with exponential divergence. In our framework, the Black Swan type dynamics occurs when an underlying dynamical system suddenly shifts between dynamics of different types. A learning and prediction system should be capable of recognizing the shift in behavior, exemplified by ``confidence loss". In this paradigm, the predictive power is assessed dynamically and confidence level is used to switch between long term prediction and local-in-time prediction. Here we explore the problem of prediction in systems that exhibit such behavior. The mathematical underpinnings of our theory and algorithms are based on an operator-theoretic approach in which the dynamics of the system are embedded into an infinite-dimensional space. We apply the algorithm to a number of case studies including prediction of influenza cases and the COVID-19 pandemic. The results show that the predictive algorithm is robust to perturbations of the available data, induced for example by delays in reporting or sudden increase in cases due to increase in testing capability. This is achieved in an entirely data-driven fashion, with no underlying mathematical model of the disease.
△ Less
Submitted 26 April, 2023;
originally announced April 2023.
-
Predicting the Critical Number of Layers for Hierarchical Support Vector Regression
Authors:
Ryan Mohr,
Maria Fonoberova,
Zlatko Drmač,
Iva Manojlović,
Igor Mezić
Abstract:
Hierarchical support vector regression (HSVR) models a function from data as a linear combination of SVR models at a range of scales, starting at a coarse scale and moving to finer scales as the hierarchy continues. In the original formulation of HSVR, there were no rules for choosing the depth of the model. In this paper, we observe in a number of models a phase transition in the training error -…
▽ More
Hierarchical support vector regression (HSVR) models a function from data as a linear combination of SVR models at a range of scales, starting at a coarse scale and moving to finer scales as the hierarchy continues. In the original formulation of HSVR, there were no rules for choosing the depth of the model. In this paper, we observe in a number of models a phase transition in the training error -- the error remains relatively constant as layers are added, until a critical scale is passed, at which point the training error drops close to zero and remains nearly constant for added layers. We introduce a method to predict this critical scale a priori with the prediction based on the support of either a Fourier transform of the data or the Dynamic Mode Decomposition (DMD) spectrum. This allows us to determine the required number of layers prior to training any models.
△ Less
Submitted 21 December, 2020;
originally announced December 2020.
-
Applications of Koopman Mode Analysis to Neural Networks
Authors:
Iva Manojlović,
Maria Fonoberova,
Ryan Mohr,
Aleksandr Andrejčuk,
Zlatko Drmač,
Yannis Kevrekidis,
Igor Mezić
Abstract:
We consider the training process of a neural network as a dynamical system acting on the high-dimensional weight space. Each epoch is an application of the map induced by the optimization algorithm and the loss function. Using this induced map, we can apply observables on the weight space and measure their evolution. The evolution of the observables are given by the Koopman operator associated wit…
▽ More
We consider the training process of a neural network as a dynamical system acting on the high-dimensional weight space. Each epoch is an application of the map induced by the optimization algorithm and the loss function. Using this induced map, we can apply observables on the weight space and measure their evolution. The evolution of the observables are given by the Koopman operator associated with the induced dynamical system. We use the spectrum and modes of the Koopman operator to realize the above objectives. Our methods can help to, a priori, determine the network depth; determine if we have a bad initialization of the network weights, allowing a restart before training too long; speeding up the training time. Additionally, our methods help enable noise rejection and improve robustness. We show how the Koopman spectrum can be used to determine the number of layers required for the architecture. Additionally, we show how we can elucidate the convergence versus non-convergence of the training process by monitoring the spectrum, in particular, how the existence of eigenvalues clustering around 1 determines when to terminate the learning process. We also show how using Koopman modes we can selectively prune the network to speed up the training procedure. Finally, we show that incorporating loss functions based on negative Sobolev norms can allow for the reconstruction of a multi-scale signal polluted by very large amounts of noise.
△ Less
Submitted 21 June, 2020;
originally announced June 2020.
