-
Marginalization Consistent Mixture of Separable Flows for Probabilistic Irregular Time Series Forecasting
Authors:
Vijaya Krishna Yalavarthi,
Randolf Scholz,
Kiran Madhusudhanan,
Stefan Born,
Lars Schmidt-Thieme
Abstract:
Probabilistic forecasting models for joint distributions of targets in irregular time series are a heavily under-researched area in machine learning with, to the best of our knowledge, only three models researched so far: GPR, the Gaussian Process Regression model~\citep{Durichen2015.Multitask}, TACTiS, the Transformer-Attentional Copulas for Time Series~\cite{Drouin2022.Tactis, ashok2024tactis} a…
▽ More
Probabilistic forecasting models for joint distributions of targets in irregular time series are a heavily under-researched area in machine learning with, to the best of our knowledge, only three models researched so far: GPR, the Gaussian Process Regression model~\citep{Durichen2015.Multitask}, TACTiS, the Transformer-Attentional Copulas for Time Series~\cite{Drouin2022.Tactis, ashok2024tactis} and ProFITi \citep{Yalavarthi2024.Probabilistica}, a multivariate normalizing flow model based on invertible attention layers. While ProFITi, thanks to using multivariate normalizing flows, is the more expressive model with better predictive performance, we will show that it suffers from marginalization inconsistency: it does not guarantee that the marginal distributions of a subset of variables in its predictive distributions coincide with the directly predicted distributions of these variables. Also, TACTiS does not provide any guarantees for marginalization consistency. We develop a novel probabilistic irregular time series forecasting model, Marginalization Consistent Mixtures of Separable Flows (moses), that mixes several normalizing flows with (i) Gaussian Processes with full covariance matrix as source distributions and (ii) a separable invertible transformation, aiming to combine the expressivity of normalizing flows with the marginalization consistency of Gaussians. In experiments on four different datasets we show that moses outperforms other state-of-the-art marginalization consistent models, performs on par with ProFITi, but different from ProFITi, guarantee marginalization consistency.
△ Less
Submitted 11 June, 2024;
originally announced June 2024.
-
Probabilistic Forecasting of Irregular Time Series via Conditional Flows
Authors:
Vijaya Krishna Yalavarthi,
Randolf Scholz,
Stefan Born,
Lars Schmidt-Thieme
Abstract:
Probabilistic forecasting of irregularly sampled multivariate time series with missing values is an important problem in many fields, including health care, astronomy, and climate. State-of-the-art methods for the task estimate only marginal distributions of observations in single channels and at single timepoints, assuming a fixed-shape parametric distribution. In this work, we propose a novel mo…
▽ More
Probabilistic forecasting of irregularly sampled multivariate time series with missing values is an important problem in many fields, including health care, astronomy, and climate. State-of-the-art methods for the task estimate only marginal distributions of observations in single channels and at single timepoints, assuming a fixed-shape parametric distribution. In this work, we propose a novel model, ProFITi, for probabilistic forecasting of irregularly sampled time series with missing values using conditional normalizing flows. The model learns joint distributions over the future values of the time series conditioned on past observations and queried channels and times, without assuming any fixed shape of the underlying distribution. As model components, we introduce a novel invertible triangular attention layer and an invertible non-linear activation function on and onto the whole real line. We conduct extensive experiments on four datasets and demonstrate that the proposed model provides $4$ times higher likelihood over the previously best model.
△ Less
Submitted 21 May, 2024; v1 submitted 9 February, 2024;
originally announced February 2024.
-
Deep Learning for Fast Inference of Mechanistic Models' Parameters
Authors:
Maxim Borisyak,
Stefan Born,
Peter Neubauer,
Mariano Nicolas Cruz-Bournazou
Abstract:
Inferring parameters of macro-kinetic growth models, typically represented by Ordinary Differential Equations (ODE), from the experimental data is a crucial step in bioprocess engineering. Conventionally, estimates of the parameters are obtained by fitting the mechanistic model to observations. Fitting, however, requires a significant computational power. Specifically, during the development of ne…
▽ More
Inferring parameters of macro-kinetic growth models, typically represented by Ordinary Differential Equations (ODE), from the experimental data is a crucial step in bioprocess engineering. Conventionally, estimates of the parameters are obtained by fitting the mechanistic model to observations. Fitting, however, requires a significant computational power. Specifically, during the development of new bioprocesses that use previously unknown organisms or strains, efficient, robust, and computationally cheap methods for parameter estimation are of great value. In this work, we propose using Deep Neural Networks (NN) for directly predicting parameters of mechanistic models given observations. The approach requires spending computational resources for training a NN, nonetheless, once trained, such a network can provide parameter estimates orders of magnitude faster than conventional methods. We consider a training procedure that combines Neural Networks and mechanistic models. We demonstrate the performance of the proposed algorithms on data sampled from several mechanistic models used in bioengineering describing a typical industrial batch process and compare the proposed method, a typical gradient-based fitting procedure, and the combination of the two. We find that, while Neural Network estimates are slightly improved by further fitting, these estimates are measurably better than the fitting procedure alone.
△ Less
Submitted 5 December, 2023;
originally announced December 2023.
-
Deep Set Neural Networks for forecasting asynchronous bioprocess timeseries
Authors:
Maxim Borisyak,
Stefan Born,
Peter Neubauer,
Mariano Nicolas Cruz-Bournazou
Abstract:
Cultivation experiments often produce sparse and irregular time series. Classical approaches based on mechanistic models, like Maximum Likelihood fitting or Monte-Carlo Markov chain sampling, can easily account for sparsity and time-grid irregularities, but most statistical and Machine Learning tools are not designed for handling sparse data out-of-the-box. Among popular approaches there are vario…
▽ More
Cultivation experiments often produce sparse and irregular time series. Classical approaches based on mechanistic models, like Maximum Likelihood fitting or Monte-Carlo Markov chain sampling, can easily account for sparsity and time-grid irregularities, but most statistical and Machine Learning tools are not designed for handling sparse data out-of-the-box. Among popular approaches there are various schemes for filling missing values (imputation) and interpolation into a regular grid (alignment). However, such methods transfer the biases of the interpolation or imputation models to the target model. We show that Deep Set Neural Networks equipped with triplet encoding of the input data can successfully handle bio-process data without any need for imputation or alignment procedures. The method is agnostic to the particular nature of the time series and can be adapted for any task, for example, online monitoring, predictive control, design of experiments, etc. In this work, we focus on forecasting. We argue that such an approach is especially suitable for typical cultivation processes, demonstrate the performance of the method on several forecasting tasks using data generated from macrokinetic growth models under realistic conditions, and compare the method to a conventional fitting procedure and methods based on imputation and alignment.
△ Less
Submitted 5 December, 2023; v1 submitted 4 December, 2023;
originally announced December 2023.
-
Analytical solution for the long- and short-range every-pair-interactions model
Authors:
Fabiano L. Ribeiro,
Yunfei Li,
Stefan Born,
Diego Rybski
Abstract:
Many physical, biological, and social systems exhibit emergent properties that arise from the interactions between their components (cells). In this study, we systematically treat every-pair interactions (a) that exhibit power-law dependence on the Euclidean distance and (b) act in structures that can be characterized using fractal geometry. We analytically derive the mean interaction field of the…
▽ More
Many physical, biological, and social systems exhibit emergent properties that arise from the interactions between their components (cells). In this study, we systematically treat every-pair interactions (a) that exhibit power-law dependence on the Euclidean distance and (b) act in structures that can be characterized using fractal geometry. We analytically derive the mean interaction field of the cells and find that (i) in a long-range interaction regime, the mean interaction field increases following a power law with the size of the system, (ii) in a short-range interaction regime, the field saturates, and (iii) in the intermediate range
it follows a logarithmic behaviour. To validate our analytical solution, we perform numerical simulations. In the case of short-range interactions, we observe that discreteness significantly impacts the continuum approximation used in the derivation, leading to incorrect asymptotic behaviour in this regime. To address this issue, we propose an expansion that substantially improves the accuracy of the analytical expression. Furthermore, our results motivate us to explore a framework for estimating the fractal dimension of unknown structures. This approach offers an alternative to established methods such as box-counting or sandbox methods. Overall, we believe that our analytical work will have broad applicability in systems where every-pair interactions play a crucial role. The insights gained from this study can contribute to a better understanding of various complex systems and facilitate more accurate modelling and analysis in a wide range of disciplines.
△ Less
Submitted 15 July, 2023;
originally announced July 2023.
-
Forecasting Irregularly Sampled Time Series using Graphs
Authors:
Vijaya Krishna Yalavarthi,
Kiran Madhusudhanan,
Randolf Sholz,
Nourhan Ahmed,
Johannes Burchert,
Shayan Jawed,
Stefan Born,
Lars Schmidt-Thieme
Abstract:
Forecasting irregularly sampled time series with missing values is a crucial task for numerous real-world applications such as healthcare, astronomy, and climate sciences. State-of-the-art approaches to this problem rely on Ordinary Differential Equations (ODEs) which are known to be slow and often require additional features to handle missing values. To address this issue, we propose a novel mode…
▽ More
Forecasting irregularly sampled time series with missing values is a crucial task for numerous real-world applications such as healthcare, astronomy, and climate sciences. State-of-the-art approaches to this problem rely on Ordinary Differential Equations (ODEs) which are known to be slow and often require additional features to handle missing values. To address this issue, we propose a novel model using Graphs for Forecasting Irregularly Sampled Time Series with missing values which we call GraFITi. GraFITi first converts the time series to a Sparsity Structure Graph which is a sparse bipartite graph, and then reformulates the forecasting problem as the edge weight prediction task in the graph. It uses the power of Graph Neural Networks to learn the graph and predict the target edge weights. GraFITi has been tested on 3 real-world and 1 synthetic irregularly sampled time series dataset with missing values and compared with various state-of-the-art models. The experimental results demonstrate that GraFITi improves the forecasting accuracy by up to 17% and reduces the run time up to 5 times compared to the state-of-the-art forecasting models.
△ Less
Submitted 10 August, 2023; v1 submitted 22 May, 2023;
originally announced May 2023.
-
Scouting for dark showers at CMS and LHCb
Authors:
Susan Born,
Rohith Karur,
Simon Knapen,
Jessie Shelton
Abstract:
We assess the capabilities of the CMS and LHCb searches for low-$p_T$ displaced dimuon pairs to discover hidden valley models, using a newly-developed benchmark model that realizes a range of dimuon vertex topologies. We show that the data scouting techniques used in these searches provide unique sensitivity and we make some additional suggestions to further extend the scope of future experimental…
▽ More
We assess the capabilities of the CMS and LHCb searches for low-$p_T$ displaced dimuon pairs to discover hidden valley models, using a newly-developed benchmark model that realizes a range of dimuon vertex topologies. We show that the data scouting techniques used in these searches provide unique sensitivity and we make some additional suggestions to further extend the scope of future experimental searches.
△ Less
Submitted 26 September, 2023; v1 submitted 7 March, 2023;
originally announced March 2023.
-
When Bioprocess Engineering Meets Machine Learning: A Survey from the Perspective of Automated Bioprocess Development
Authors:
Nghia Duong-Trung,
Stefan Born,
Jong Woo Kim,
Marie-Therese Schermeyer,
Katharina Paulick,
Maxim Borisyak,
Mariano Nicolas Cruz-Bournazou,
Thorben Werner,
Randolf Scholz,
Lars Schmidt-Thieme,
Peter Neubauer,
Ernesto Martinez
Abstract:
Machine learning (ML) is becoming increasingly crucial in many fields of engineering but has not yet played out its full potential in bioprocess engineering. While experimentation has been accelerated by increasing levels of lab automation, experimental planning and data modeling are still largerly depend on human intervention. ML can be seen as a set of tools that contribute to the automation of…
▽ More
Machine learning (ML) is becoming increasingly crucial in many fields of engineering but has not yet played out its full potential in bioprocess engineering. While experimentation has been accelerated by increasing levels of lab automation, experimental planning and data modeling are still largerly depend on human intervention. ML can be seen as a set of tools that contribute to the automation of the whole experimental cycle, including model building and practical planning, thus allowing human experts to focus on the more demanding and overarching cognitive tasks. First, probabilistic programming is used for the autonomous building of predictive models. Second, machine learning automatically assesses alternative decisions by planning experiments to test hypotheses and conducting investigations to gather informative data that focus on model selection based on the uncertainty of model predictions. This review provides a comprehensive overview of ML-based automation in bioprocess development. On the one hand, the biotech and bioengineering community should be aware of the potential and, most importantly, the limitation of existing ML solutions for their application in biotechnology and biopharma. On the other hand, it is essential to identify the missing links to enable the easy implementation of ML and Artificial Intelligence (AI) tools in valuable solutions for the bio-community.
△ Less
Submitted 1 November, 2022; v1 submitted 2 September, 2022;
originally announced September 2022.
-
Fitting nonlinear models to continuous oxygen data with oscillatory signal variations via a loss based on DynamicTime War**
Authors:
Judit Aizpuru,
Annina Karolin Kemmer,
Jong Woo Kim,
Stefan Born,
Peter Neubauer,
Mariano N. Cruz Bournazou,
Tilman Barz
Abstract:
High throughput experimental systems play an important role in bioprocess development, as they provide an efficient way of analysing different experimental conditions and perform strain discrimination in previous phases to the industrial scale production. In the millilitre scale, these systems are combinations of parallel mini-bioreactors, liquid handling robots and automated workflows for data ha…
▽ More
High throughput experimental systems play an important role in bioprocess development, as they provide an efficient way of analysing different experimental conditions and perform strain discrimination in previous phases to the industrial scale production. In the millilitre scale, these systems are combinations of parallel mini-bioreactors, liquid handling robots and automated workflows for data handling and model based operation. For successfully monitoring cultivation conditions and improving the overall process quality by model-based approaches, a proper model identification is crucial. However, the quality and amount of measurements makes this task challenging considering the complexity of the bio-processes. TheDissolved Oxygen Tension is often the only measurement which is available online, and therefore, a good understanding of the errors in this signal is important for performing a robust estimation.Some of the expected errors will provoke uncertainties in the time-domain of the measurement, and in those cases, the common Weighted Least Squares estimation procedure can fail providing good results. Moreover, these errors will have even a larger effect in the fed-batch phase where bolus feeding is applied, as this generates fast dynamic responses in the signal. In the present work, an insilico study of the performance of Weighted Least Squares estimator is analysed when the expected time-uncertainties are present in the oxygen signal. As an alternative, a loss based on the Dynamic Time War** measure is proposed. The results show how this latter procedure outperforms the former reconstructing the oxygen signal, and in addition, returns less biased parameter estimates.
△ Less
Submitted 25 December, 2021;
originally announced December 2021.
-
Yformer: U-Net Inspired Transformer Architecture for Far Horizon Time Series Forecasting
Authors:
Kiran Madhusudhanan,
Johannes Burchert,
Nghia Duong-Trung,
Stefan Born,
Lars Schmidt-Thieme
Abstract:
Time series data is ubiquitous in research as well as in a wide variety of industrial applications. Effectively analyzing the available historical data and providing insights into the far future allows us to make effective decisions. Recent research has witnessed the superior performance of transformer-based architectures, especially in the regime of far horizon time series forecasting. However, t…
▽ More
Time series data is ubiquitous in research as well as in a wide variety of industrial applications. Effectively analyzing the available historical data and providing insights into the far future allows us to make effective decisions. Recent research has witnessed the superior performance of transformer-based architectures, especially in the regime of far horizon time series forecasting. However, the current state of the art sparse Transformer architectures fail to couple down- and upsampling procedures to produce outputs in a similar resolution as the input. We propose the Yformer model, based on a novel Y-shaped encoder-decoder architecture that (1) uses direct connection from the downscaled encoder layer to the corresponding upsampled decoder layer in a U-Net inspired architecture, (2) Combines the downscaling/upsampling with sparse attention to capture long-range effects, and (3) stabilizes the encoder-decoder stacks with the addition of an auxiliary reconstruction loss. Extensive experiments have been conducted with relevant baselines on four benchmark datasets, demonstrating an average improvement of 19.82, 18.41 percentage MSE and 13.62, 11.85 percentage MAE in comparison to the current state of the art for the univariate and the multivariate settings respectively.
△ Less
Submitted 25 August, 2022; v1 submitted 13 October, 2021;
originally announced October 2021.
-
Quasiconformal distortion of projective transformations and discrete conformal maps
Authors:
Stefan Born,
Ulrike Bücking,
Boris Springborn
Abstract:
We consider the quasiconformal dilatation of projective transformations of the real projective plane. For non-affine transformations, the contour lines of dilatation form a hyperbolic pencil of circles, and these are the only circles that are mapped to circles. We apply this result to analyze the dilatation of the circumcircle preserving piecewise projective interpolation between discretely confor…
▽ More
We consider the quasiconformal dilatation of projective transformations of the real projective plane. For non-affine transformations, the contour lines of dilatation form a hyperbolic pencil of circles, and these are the only circles that are mapped to circles. We apply this result to analyze the dilatation of the circumcircle preserving piecewise projective interpolation between discretely conformally equivalent triangulations. We show that another interpolation scheme, angle bisector preserving piecewise projective interpolation, is in a sense optimal with respect to dilatation. These two interpolation schemes belong to a one-parameter family.
△ Less
Submitted 31 March, 2017; v1 submitted 6 May, 2015;
originally announced May 2015.
-
Boundedness of Functions on Product Spaces by Sums of Functions on the Factors
Authors:
Stefan Born,
Alexander Dirmeier
Abstract:
We investigate sufficient conditions for real-valued functions on product spaces to be bounded from above by sums or products of functions which depend only on points in the respective factors.
We investigate sufficient conditions for real-valued functions on product spaces to be bounded from above by sums or products of functions which depend only on points in the respective factors.
△ Less
Submitted 31 December, 2013;
originally announced January 2014.
