-
Deterministic Uncertainty Propagation for Improved Model-Based Offline Reinforcement Learning
Authors:
Abdullah Akgül,
Manuel Haußmann,
Melih Kandemir
Abstract:
Current approaches to model-based offline Reinforcement Learning (RL) often incorporate uncertainty-based reward penalization to address the distributional shift problem. While these approaches have achieved some success, we argue that this penalization introduces excessive conservatism, potentially resulting in suboptimal policies through underestimation. We identify as an important cause of over…
▽ More
Current approaches to model-based offline Reinforcement Learning (RL) often incorporate uncertainty-based reward penalization to address the distributional shift problem. While these approaches have achieved some success, we argue that this penalization introduces excessive conservatism, potentially resulting in suboptimal policies through underestimation. We identify as an important cause of over-penalization the lack of a reliable uncertainty estimator capable of propagating uncertainties in the Bellman operator. The common approach to calculating the penalty term relies on sampling-based uncertainty estimation, resulting in high variance. To address this challenge, we propose a novel method termed Moment Matching Offline Model-Based Policy Optimization (MOMBO). MOMBO learns a Q-function using moment matching, which allows us to deterministically propagate uncertainties through the Q-function. We evaluate MOMBO's performance across various environments and demonstrate empirically that MOMBO is a more stable and sample-efficient approach.
△ Less
Submitted 6 June, 2024;
originally announced June 2024.
-
Latent variable model for high-dimensional point process with structured missingness
Authors:
Maksim Sinelnikov,
Manuel Haussmann,
Harri Lähdesmäki
Abstract:
Longitudinal data are important in numerous fields, such as healthcare, sociology and seismology, but real-world datasets present notable challenges for practitioners because they can be high-dimensional, contain structured missingness patterns, and measurement time points can be governed by an unknown stochastic process. While various solutions have been suggested, the majority of them have been…
▽ More
Longitudinal data are important in numerous fields, such as healthcare, sociology and seismology, but real-world datasets present notable challenges for practitioners because they can be high-dimensional, contain structured missingness patterns, and measurement time points can be governed by an unknown stochastic process. While various solutions have been suggested, the majority of them have been designed to account for only one of these challenges. In this work, we propose a flexible and efficient latent-variable model that is capable of addressing all these limitations. Our approach utilizes Gaussian processes to capture temporal correlations between samples and their associated missingness masks as well as to model the underlying point process. We construct our model as a variational autoencoder together with deep neural network parameterised encoder and decoder models, and develop a scalable amortised variational inference approach for efficient model training. We demonstrate competitive performance using both simulated and real datasets.
△ Less
Submitted 28 June, 2024; v1 submitted 8 February, 2024;
originally announced February 2024.
-
Estimating treatment effects from single-arm trials via latent-variable modeling
Authors:
Manuel Haussmann,
Tran Minh Son Le,
Viivi Halla-aho,
Samu Kurki,
Jussi V. Leinonen,
Miika Koskinen,
Samuel Kaski,
Harri Lähdesmäki
Abstract:
Randomized controlled trials (RCTs) are the accepted standard for treatment effect estimation but they can be infeasible due to ethical reasons and prohibitive costs. Single-arm trials, where all patients belong to the treatment group, can be a viable alternative but require access to an external control group. We propose an identifiable deep latent-variable model for this scenario that can also a…
▽ More
Randomized controlled trials (RCTs) are the accepted standard for treatment effect estimation but they can be infeasible due to ethical reasons and prohibitive costs. Single-arm trials, where all patients belong to the treatment group, can be a viable alternative but require access to an external control group. We propose an identifiable deep latent-variable model for this scenario that can also account for missing covariate observations by modeling their structured missingness patterns. Our method uses amortized variational inference to learn both group-specific and identifiable shared latent representations, which can subsequently be used for {\em (i)} patient matching if treatment outcomes are not available for the treatment group, or for {\em (ii)} direct treatment effect estimation assuming outcomes are available for both groups. We evaluate the model on a public benchmark as well as on a data set consisting of a published RCT study and real-world electronic health records. Compared to previous methods, our results show improved performance both for direct treatment effect estimation as well as for effect estimation via patient matching.
△ Less
Submitted 4 March, 2024; v1 submitted 6 November, 2023;
originally announced November 2023.
-
Practical Equivariances via Relational Conditional Neural Processes
Authors:
Daolang Huang,
Manuel Haussmann,
Ulpu Remes,
ST John,
Grégoire Clarté,
Kevin Sebastian Luck,
Samuel Kaski,
Luigi Acerbi
Abstract:
Conditional Neural Processes (CNPs) are a class of metalearning models popular for combining the runtime efficiency of amortized inference with reliable uncertainty quantification. Many relevant machine learning tasks, such as in spatio-temporal modeling, Bayesian Optimization and continuous control, inherently contain equivariances -- for example to translation -- which the model can exploit for…
▽ More
Conditional Neural Processes (CNPs) are a class of metalearning models popular for combining the runtime efficiency of amortized inference with reliable uncertainty quantification. Many relevant machine learning tasks, such as in spatio-temporal modeling, Bayesian Optimization and continuous control, inherently contain equivariances -- for example to translation -- which the model can exploit for maximal performance. However, prior attempts to include equivariances in CNPs do not scale effectively beyond two input dimensions. In this work, we propose Relational Conditional Neural Processes (RCNPs), an effective approach to incorporate equivariances into any neural process model. Our proposed method extends the applicability and impact of equivariant neural processes to higher dimensions. We empirically demonstrate the competitive performance of RCNPs on a large array of tasks naturally containing equivariances.
△ Less
Submitted 5 November, 2023; v1 submitted 19 June, 2023;
originally announced June 2023.
-
PAC-Bayesian Soft Actor-Critic Learning
Authors:
Bahareh Tasdighi,
Abdullah Akgül,
Manuel Haussmann,
Kenny Kazimirzak Brink,
Melih Kandemir
Abstract:
Actor-critic algorithms address the dual goals of reinforcement learning (RL), policy evaluation and improvement via two separate function approximators. The practicality of this approach comes at the expense of training instability, caused mainly by the destructive effect of the approximation errors of the critic on the actor. We tackle this bottleneck by employing an existing Probably Approximat…
▽ More
Actor-critic algorithms address the dual goals of reinforcement learning (RL), policy evaluation and improvement via two separate function approximators. The practicality of this approach comes at the expense of training instability, caused mainly by the destructive effect of the approximation errors of the critic on the actor. We tackle this bottleneck by employing an existing Probably Approximately Correct (PAC) Bayesian bound for the first time as the critic training objective of the Soft Actor-Critic (SAC) algorithm. We further demonstrate that online learning performance improves significantly when a stochastic actor explores multiple futures by critic-guided random search. We observe our resulting algorithm to compare favorably against the state-of-the-art SAC implementation on multiple classical control and locomotion tasks in terms of both sample efficiency and regret.
△ Less
Submitted 10 June, 2024; v1 submitted 30 January, 2023;
originally announced January 2023.
-
Fresnel reflection boundary for radiative transport lattice Boltzmann methods in highly scattering volume
Authors:
Albert Mink,
Kira Schediwy,
Marc Haussmann,
Clemens Posten,
Hermann Nirschl,
Mathias J. Krause
Abstract:
With its roots in kinetic theory, the lattice Boltzmann method (LBM) cannot only be used to solve complex fluid flows but also radiative transport in volume. The present work derives a novel Fresnel boundary scheme for radiative transport LBM, based on Fresnel's equation, which depicts the partly reflected radiation on surfaces. Driven from a boundary modeling and discussion on the microscopic lev…
▽ More
With its roots in kinetic theory, the lattice Boltzmann method (LBM) cannot only be used to solve complex fluid flows but also radiative transport in volume. The present work derives a novel Fresnel boundary scheme for radiative transport LBM, based on Fresnel's equation, which depicts the partly reflected radiation on surfaces. Driven from a boundary modeling and discussion on the microscopic level, incorporating Fresnel's equation, it is developed a boundary model for the mesoscopic radiative transport LBM. At an intermediate step, the Fresnel's equation is related to well known partial differential (Robin) equations, based on a bottom-up approach where the P1-Approximation is deployed. To connect the novel boundary scheme to the so derived target equation, a Chapman-Enskog expansion is examined in addition. Both techniques together, point out how to interpret microscopic modeling by the means of macroscopic expressions and as a consequence how, to chose simulation parameters according to the specific boundary. The numerical tests suggest that the proposed boundary is first order convergent. The paper closes with a showcase, where the novel boundary method for radiative transport LBM is applied to a setup with multiple LED spots.
△ Less
Submitted 20 July, 2021;
originally announced July 2021.
-
Evidential Turing Processes
Authors:
Melih Kandemir,
Abdullah Akgül,
Manuel Haussmann,
Gozde Unal
Abstract:
A probabilistic classifier with reliable predictive uncertainties i) fits successfully to the target domain data, ii) provides calibrated class probabilities in difficult regions of the target domain (e.g.\ class overlap), and iii) accurately identifies queries coming out of the target domain and rejects them. We introduce an original combination of Evidential Deep Learning, Neural Processes, and…
▽ More
A probabilistic classifier with reliable predictive uncertainties i) fits successfully to the target domain data, ii) provides calibrated class probabilities in difficult regions of the target domain (e.g.\ class overlap), and iii) accurately identifies queries coming out of the target domain and rejects them. We introduce an original combination of Evidential Deep Learning, Neural Processes, and Neural Turing Machines capable of providing all three essential properties mentioned above for total uncertainty quantification. We observe our method on five classification tasks to be the only one that can excel all three aspects of total calibration with a single standalone predictor. Our unified solution delivers an implementation-friendly and compute efficient recipe for safety clearance and provides intellectual economy to an investigation of algorithmic roots of epistemic awareness in deep neural nets.
△ Less
Submitted 8 March, 2022; v1 submitted 2 June, 2021;
originally announced June 2021.
-
Understanding Event-Generation Networks via Uncertainties
Authors:
Marco Bellagente,
Manuel Haußmann,
Michel Luchmann,
Tilman Plehn
Abstract:
Following the growing success of generative neural networks in LHC simulations, the crucial question is how to control the networks and assign uncertainties to their event output. We show how Bayesian normalizing flow or invertible networks capture uncertainties from the training and turn them into an uncertainty on the event weight. Fundamentally, the interplay between density and uncertainty est…
▽ More
Following the growing success of generative neural networks in LHC simulations, the crucial question is how to control the networks and assign uncertainties to their event output. We show how Bayesian normalizing flow or invertible networks capture uncertainties from the training and turn them into an uncertainty on the event weight. Fundamentally, the interplay between density and uncertainty estimates indicates that these networks learn functions in analogy to parameter fits rather than binned event counts.
△ Less
Submitted 1 October, 2021; v1 submitted 9 April, 2021;
originally announced April 2021.
-
Learning Partially Known Stochastic Dynamics with Empirical PAC Bayes
Authors:
Manuel Haussmann,
Sebastian Gerwinn,
Andreas Look,
Barbara Rakitsch,
Melih Kandemir
Abstract:
Neural Stochastic Differential Equations model a dynamical environment with neural nets assigned to their drift and diffusion terms. The high expressive power of their nonlinearity comes at the expense of instability in the identification of the large set of free parameters. This paper presents a recipe to improve the prediction accuracy of such models in three steps: i) accounting for epistemic u…
▽ More
Neural Stochastic Differential Equations model a dynamical environment with neural nets assigned to their drift and diffusion terms. The high expressive power of their nonlinearity comes at the expense of instability in the identification of the large set of free parameters. This paper presents a recipe to improve the prediction accuracy of such models in three steps: i) accounting for epistemic uncertainty by assuming probabilistic weights, ii) incorporation of partial knowledge on the state dynamics, and iii) training the resultant hybrid model by an objective derived from a PAC-Bayesian generalization bound. We observe in our experiments that this recipe effectively translates partial and noisy prior knowledge into an improved model fit.
△ Less
Submitted 26 February, 2021; v1 submitted 17 June, 2020;
originally announced June 2020.
-
Fluid-Structure Interaction Simulation of a Coriolis Mass Flowmeter using a Lattice Boltzmann Method
Authors:
Marc Haussmann,
Peter Reinshaus,
Stephan Simonis,
Hermann Nirschl,
Mathias J. Krause
Abstract:
In this paper we use a fluid-structure interaction (FSI) approach to simulate a Coriolis mass flowmeter (CMF). The fluid dynamics are calculated by the open source framework OpenLB, based on the lattice Boltzmann method (LBM). For the structural dynamics we employ the open source software Elmer, an implementation of the finite element method (FEM). A staggered coupling approach between the two sof…
▽ More
In this paper we use a fluid-structure interaction (FSI) approach to simulate a Coriolis mass flowmeter (CMF). The fluid dynamics are calculated by the open source framework OpenLB, based on the lattice Boltzmann method (LBM). For the structural dynamics we employ the open source software Elmer, an implementation of the finite element method (FEM). A staggered coupling approach between the two software packages is presented. The finite element mesh is created by the mesh generator Gmsh to ensure a complete open source workflow. The Eigenmodes of the CMF, which are calculated by modal analysis are compared with measurement data. Using the estimated excitation frequency, a fully coupled, partitioned, FSI simulation is applied to simulate the phase shift of the investigated CMF design. The calculated phaseshift values are in good agreement to the measurement data and verify the suitability of the model to numerically describe the working principle of a CMF.
△ Less
Submitted 8 May, 2020;
originally announced May 2020.
-
Deep Active Learning with Adaptive Acquisition
Authors:
Manuel Haussmann,
Fred A. Hamprecht,
Melih Kandemir
Abstract:
Model selection is treated as a standard performance boosting step in many machine learning applications. Once all other properties of a learning problem are fixed, the model is selected by grid search on a held-out validation set. This is strictly inapplicable to active learning. Within the standardized workflow, the acquisition function is chosen among available heuristics a priori, and its succ…
▽ More
Model selection is treated as a standard performance boosting step in many machine learning applications. Once all other properties of a learning problem are fixed, the model is selected by grid search on a held-out validation set. This is strictly inapplicable to active learning. Within the standardized workflow, the acquisition function is chosen among available heuristics a priori, and its success is observed only after the labeling budget is already exhausted. More importantly, none of the earlier studies report a unique consistently successful acquisition heuristic to the extent to stand out as the unique best choice. We present a method to break this vicious circle by defining the acquisition function as a learning predictor and training it by reinforcement feedback collected from each labeling round. As active learning is a scarce data regime, we bootstrap from a well-known heuristic that filters the bulk of data points on which all heuristics would agree, and learn a policy to warp the top portion of this ranking in the most beneficial way for the character of a specific data distribution. Our system consists of a Bayesian neural net, the predictor, a bootstrap acquisition function, a probabilistic state definition, and another Bayesian policy network that can effectively incorporate this input distribution. We observe on three benchmark data sets that our method always manages to either invent a new superior acquisition function or to adapt itself to the a priori unknown best performing heuristic for each specific data set.
△ Less
Submitted 27 June, 2019;
originally announced June 2019.
-
Bayesian Evidential Deep Learning with PAC Regularization
Authors:
Manuel Haussmann,
Sebastian Gerwinn,
Melih Kandemir
Abstract:
We propose a novel method for closed-form predictive distribution modeling with neural nets. In quantifying prediction uncertainty, we build on Evidential Deep Learning, which has been impactful as being both simple to implement and giving closed-form access to predictive uncertainty. We employ it to model aleatoric uncertainty and extend it to account also for epistemic uncertainty by converting…
▽ More
We propose a novel method for closed-form predictive distribution modeling with neural nets. In quantifying prediction uncertainty, we build on Evidential Deep Learning, which has been impactful as being both simple to implement and giving closed-form access to predictive uncertainty. We employ it to model aleatoric uncertainty and extend it to account also for epistemic uncertainty by converting it to a Bayesian Neural Net. While extending its uncertainty quantification capabilities, we maintain its analytically accessible predictive distribution model by performing progressive moment matching for the first time for approximate weight marginalization. The eventual model introduces a prohibitively large number of hyperparameters for stable training. We overcome this drawback by deriving a vacuous PAC bound that comprises the marginal likelihood of the predictor and a complexity penalty. We observe on regression, classification, and out-of-domain detection benchmarks that our method improves model fit and uncertainty quantification.
△ Less
Submitted 21 January, 2021; v1 submitted 3 June, 2019;
originally announced June 2019.
-
Deep-Learning Jets with Uncertainties and More
Authors:
Sven Bollweg,
Manuel Haussmann,
Gregor Kasieczka,
Michel Luchmann,
Tilman Plehn,
Jennifer Thompson
Abstract:
Bayesian neural networks allow us to keep track of uncertainties, for example in top tagging, by learning a tagger output together with an error band. We illustrate the main features of Bayesian versions of established deep-learning taggers. We show how they capture statistical uncertainties from finite training samples, systematics related to the jet energy scale, and stability issues through pil…
▽ More
Bayesian neural networks allow us to keep track of uncertainties, for example in top tagging, by learning a tagger output together with an error band. We illustrate the main features of Bayesian versions of established deep-learning taggers. We show how they capture statistical uncertainties from finite training samples, systematics related to the jet energy scale, and stability issues through pile-up. Altogether, Bayesian networks offer many new handles to understand and control deep learning at the LHC without introducing a visible prior effect and without compromising the network performance.
△ Less
Submitted 15 August, 2019; v1 submitted 22 April, 2019;
originally announced April 2019.
-
Brief increases in corticosterone affect morphology, stress responses, and telomere length, but not post-fledging movements, in a wild songbird
Authors:
Teresa M. Pegan,
David W. Winkler,
Mark F. Haussmann,
Maren N. Vitousek
Abstract:
Organisms are frequently exposed to challenges during development, such as poor weather and food shortage. Such challenges can initiate the hormonal stress response, which involves secretion of glucocorticoids. Although the hormonal stress response helps organisms deal with challenges, long-term exposure to high levels of glucocorticoids can have morphological, behavioral, and physiological conseq…
▽ More
Organisms are frequently exposed to challenges during development, such as poor weather and food shortage. Such challenges can initiate the hormonal stress response, which involves secretion of glucocorticoids. Although the hormonal stress response helps organisms deal with challenges, long-term exposure to high levels of glucocorticoids can have morphological, behavioral, and physiological consequences, especially during development. Glucocorticoids are also associated with reduced survival and telomere shortening. To investigate whether brief, acute exposures to glucocorticoids can also produce these phenotypic effects in free-living birds, we exposed wild tree swallow (Tachycineta bicolor) nestlings to a brief exogenous dose of cort once per day for five days and then measured their morphology, baseline and stress-induced corticosterone levels, and telomere length. We also deployed radio tags on a subset of nestlings, which allowed us to determine the age at which tagged nestlings left the nest (fledged) and their pattern of presence and absence at the natal site during the post-breeding period. Corticosterone-treated nestlings had lower mass, higher baseline and stress-induced corticosterone, and reduced telomeres; other metrics of morphology were affected weakly or not at all. Our treatment resulted in no significant effect on survival to fledging, fledge age, or age at first departure from the natal site, and we found no negative effect of corticosterone on inter-annual return rate. These results show that brief acute corticosterone exposure during development can have measurable effects on phenotype in free-living tree swallows. Corticosterone may therefore mediate correlations between rearing environment and phenotype in develo** organisms, even in the absence of prolonged stressors.
△ Less
Submitted 31 July, 2018;
originally announced August 2018.
-
LeMoNADe: Learned Motif and Neuronal Assembly Detection in calcium imaging videos
Authors:
Elke Kirschbaum,
Manuel Haußmann,
Steffen Wolf,
Hannah Sonntag,
Justus Schneider,
Shehabeldin Elzoheiry,
Oliver Kann,
Daniel Durstewitz,
Fred A. Hamprecht
Abstract:
Neuronal assemblies, loosely defined as subsets of neurons with reoccurring spatio-temporally coordinated activation patterns, or "motifs", are thought to be building blocks of neural representations and information processing. We here propose LeMoNADe, a new exploratory data analysis method that facilitates hunting for motifs in calcium imaging videos, the dominant microscopic functional imaging…
▽ More
Neuronal assemblies, loosely defined as subsets of neurons with reoccurring spatio-temporally coordinated activation patterns, or "motifs", are thought to be building blocks of neural representations and information processing. We here propose LeMoNADe, a new exploratory data analysis method that facilitates hunting for motifs in calcium imaging videos, the dominant microscopic functional imaging modality in neurophysiology. Our nonparametric method extracts motifs directly from videos, bypassing the difficult intermediate step of spike extraction. Our technique augments variational autoencoders with a discrete stochastic node, and we show in detail how a differentiable reparametrization and relaxation can be used. An evaluation on simulated data, with available ground truth, reveals excellent quantitative performance. In real video data acquired from brain slices, with no ground truth available, LeMoNADe uncovers nontrivial candidate motifs that can help generate hypotheses for more focused biological investigations.
△ Less
Submitted 22 February, 2019; v1 submitted 26 June, 2018;
originally announced June 2018.
-
Sampling-Free Variational Inference of Bayesian Neural Networks by Variance Backpropagation
Authors:
Manuel Haussmann,
Fred A. Hamprecht,
Melih Kandemir
Abstract:
We propose a new Bayesian Neural Net formulation that affords variational inference for which the evidence lower bound is analytically tractable subject to a tight approximation. We achieve this tractability by (i) decomposing ReLU nonlinearities into the product of an identity and a Heaviside step function, (ii) introducing a separate path that decomposes the neural net expectation from its varia…
▽ More
We propose a new Bayesian Neural Net formulation that affords variational inference for which the evidence lower bound is analytically tractable subject to a tight approximation. We achieve this tractability by (i) decomposing ReLU nonlinearities into the product of an identity and a Heaviside step function, (ii) introducing a separate path that decomposes the neural net expectation from its variance. We demonstrate formally that introducing separate latent binary variables to the activations allows representing the neural network likelihood as a chain of linear operations. Performing variational inference on this construction enables a sampling-free computation of the evidence lower bound which is a more effective approximation than the widely applied Monte Carlo sampling and CLT related techniques. We evaluate the model on a range of regression and classification tasks against BNN inference alternatives, showing competitive or improved performance over the current state-of-the-art.
△ Less
Submitted 12 June, 2019; v1 submitted 19 May, 2018;
originally announced May 2018.
-
Readout and control of a single nuclear spin with a meta-stable electron spin ancilla
Authors:
Sang-Yun Lee,
Matthias Widmann,
Torsten Rendler,
Marcus Doherty,
Thomas M. Babinec,
Sen Yang,
Moritz Eyer,
Petr Siyushev,
Birgit J. M. Haussmann,
Marko Loncar,
Zoltán Bodrog,
Adam Gali,
Neil Manson,
Helmut Fedder,
Jörg Wrachtrup
Abstract:
Electron and nuclear spins associated with point defects in insulators are promising systems for solid state quantum technology. While the electron spin usually is used for readout and addressing, nuclear spins are exquisite quantum bits and memory systems. With these systems single-shot readout of nearby nuclear spins as well as entanglement aided by the electron spin has been shown. While the el…
▽ More
Electron and nuclear spins associated with point defects in insulators are promising systems for solid state quantum technology. While the electron spin usually is used for readout and addressing, nuclear spins are exquisite quantum bits and memory systems. With these systems single-shot readout of nearby nuclear spins as well as entanglement aided by the electron spin has been shown. While the electron spin in this example is essential for readout it usually limits nuclear spin coherence. This has set of the quest for defects with spin-free ground states. Here, we isolate a hitherto unidentified defect in diamond and use it at room temperature to demonstrate optical spin polarization and readout with exceptionally high contrast (up to 45%), coherent manipulation of an individual excited triplet state spin, and coherent nuclear spin manipulation using the triplet electron spin as a meta-stable ancilla. By this we demonstrate nuclear magnetic resonance and Rabi oscillations of the uncoupled nuclear spin in the spin-free electronic ground state. Our study demonstrates that nuclei coupled to single metastable electron spins are useful quantum systems with long memory times despite electronic relaxation processes.
△ Less
Submitted 19 February, 2013;
originally announced February 2013.
