-
Quantifying intra-tumoral genetic heterogeneity of glioblastoma toward precision medicine using MRI and a data-inclusive machine learning algorithm
Authors:
Lujia Wang,
Hairong Wang,
Fulvio D'Angelo,
Lee Curtin,
Christopher P. Sereduk,
Gustavo De Leon,
Kyle W. Singleton,
Javier Urcuyo,
Andrea Hawkins-Daarud,
Pamela R. Jackson,
Chandan Krishna,
Richard S. Zimmerman,
Devi P. Patra,
Bernard R. Bendok,
Kris A. Smith,
Peter Nakaji,
Kliment Donev,
Leslie C. Baxter,
Maciej M. Mrugała,
Michele Ceccarelli,
Antonio Iavarone,
Kristin R. Swanson,
Nhan L. Tran,
Leland S. Hu,
**g Li
Abstract:
Glioblastoma (GBM) is one of the most aggressive and lethal human cancers. Intra-tumoral genetic heterogeneity poses a significant challenge for treatment. Biopsy is invasive, which motivates the development of non-invasive, MRI-based machine learning (ML) models to quantify intra-tumoral genetic heterogeneity for each patient. This capability holds great promise for enabling better therapeutic se…
▽ More
Glioblastoma (GBM) is one of the most aggressive and lethal human cancers. Intra-tumoral genetic heterogeneity poses a significant challenge for treatment. Biopsy is invasive, which motivates the development of non-invasive, MRI-based machine learning (ML) models to quantify intra-tumoral genetic heterogeneity for each patient. This capability holds great promise for enabling better therapeutic selection to improve patient outcomes. We proposed a novel Weakly Supervised Ordinal Support Vector Machine (WSO-SVM) to predict regional genetic alteration status within each GBM tumor using MRI. WSO-SVM was applied to a unique dataset of 318 image-localized biopsies with spatially matched multiparametric MRI from 74 GBM patients. The model was trained to predict the regional genetic alteration of three GBM driver genes (EGFR, PDGFRA, and PTEN) based on features extracted from the corresponding region of five MRI contrast images. For comparison, a variety of existing ML algorithms were also applied. The classification accuracy of each gene was compared between the different algorithms. The SHapley Additive exPlanations (SHAP) method was further applied to compute contribution scores of different contrast images. Finally, the trained WSO-SVM was used to generate prediction maps within the tumoral area of each patient to help visualize the intra-tumoral genetic heterogeneity. This study demonstrated the feasibility of using MRI and WSO-SVM to enable non-invasive prediction of intra-tumoral regional genetic alteration for each GBM patient, which can inform future adaptive therapies for individualized oncology.
△ Less
Submitted 29 December, 2023;
originally announced January 2024.
-
Why Do We Need Weight Decay in Modern Deep Learning?
Authors:
Maksym Andriushchenko,
Francesco D'Angelo,
Aditya Varre,
Nicolas Flammarion
Abstract:
Weight decay is a broadly used technique for training state-of-the-art deep networks, including large language models. Despite its widespread usage, its role remains poorly understood. In this work, we highlight that the role of weight decay in modern deep learning is different from its regularization effect studied in classical learning theory. For overparameterized deep networks, we show how wei…
▽ More
Weight decay is a broadly used technique for training state-of-the-art deep networks, including large language models. Despite its widespread usage, its role remains poorly understood. In this work, we highlight that the role of weight decay in modern deep learning is different from its regularization effect studied in classical learning theory. For overparameterized deep networks, we show how weight decay modifies the optimization dynamics enhancing the ever-present implicit regularization of SGD via the loss stabilization mechanism. In contrast, for underparameterized large language models trained with nearly online SGD, we describe how weight decay balances the bias-variance tradeoff in stochastic optimization leading to lower training loss. Moreover, we show that weight decay also prevents sudden loss divergences for bfloat16 mixed-precision training which is a crucial tool for LLM training. Overall, we present a unifying perspective from ResNets on vision tasks to LLMs: weight decay is never useful as an explicit regularizer but instead changes the training dynamics in a desirable way. Our code is available at https://github.com/tml-epfl/why-weight-decay.
△ Less
Submitted 6 October, 2023;
originally announced October 2023.
-
Uncertainty estimation under model misspecification in neural network regression
Authors:
Maria R. Cervera,
Rafael Dätwyler,
Francesco D'Angelo,
Hamza Keurti,
Benjamin F. Grewe,
Christian Henning
Abstract:
Although neural networks are powerful function approximators, the underlying modelling assumptions ultimately define the likelihood and thus the hypothesis class they are parameterizing. In classification, these assumptions are minimal as the commonly employed softmax is capable of representing any categorical distribution. In regression, however, restrictive assumptions on the type of continuous…
▽ More
Although neural networks are powerful function approximators, the underlying modelling assumptions ultimately define the likelihood and thus the hypothesis class they are parameterizing. In classification, these assumptions are minimal as the commonly employed softmax is capable of representing any categorical distribution. In regression, however, restrictive assumptions on the type of continuous distribution to be realized are typically placed, like the dominant choice of training via mean-squared error and its underlying Gaussianity assumption. Recently, modelling advances allow to be agnostic to the type of continuous distribution to be modelled, granting regression the flexibility of classification models. While past studies stress the benefit of such flexible regression models in terms of performance, here we study the effect of the model choice on uncertainty estimation. We highlight that under model misspecification, aleatoric uncertainty is not properly captured, and that a Bayesian treatment of a misspecified model leads to unreliable epistemic uncertainty estimates. Overall, our study provides an overview on how modelling choices in regression may influence uncertainty estimation and thus any downstream decision making process.
△ Less
Submitted 23 November, 2021;
originally announced November 2021.
-
On out-of-distribution detection with Bayesian neural networks
Authors:
Francesco D'Angelo,
Christian Henning
Abstract:
The question whether inputs are valid for the problem a neural network is trying to solve has sparked interest in out-of-distribution (OOD) detection. It is widely assumed that Bayesian neural networks (BNNs) are well suited for this task, as the endowed epistemic uncertainty should lead to disagreement in predictions on outliers. In this paper, we question this assumption and show that proper Bay…
▽ More
The question whether inputs are valid for the problem a neural network is trying to solve has sparked interest in out-of-distribution (OOD) detection. It is widely assumed that Bayesian neural networks (BNNs) are well suited for this task, as the endowed epistemic uncertainty should lead to disagreement in predictions on outliers. In this paper, we question this assumption and show that proper Bayesian inference with function space priors induced by neural networks does not necessarily lead to good OOD detection. To circumvent the use of approximate inference, we start by studying the infinite-width case, where Bayesian inference can be exact due to the correspondence with Gaussian processes. Strikingly, the kernels derived from common architectural choices lead to function space priors which induce predictive uncertainties that do not reflect the underlying input data distribution and are therefore unsuited for OOD detection. Importantly, we find the OOD behavior in this limiting case to be consistent with the corresponding finite-width case. To overcome this limitation, useful function space properties can also be encoded in the prior in weight space, however, this can currently only be applied to a specified subset of the domain and thus does not inherently extend to OOD data. Finally, we argue that a trade-off between generalization and OOD capabilities might render the application of BNNs for OOD detection undesirable in practice. Overall, our study discloses fundamental problems when naively using BNNs for OOD detection and opens interesting avenues for future research.
△ Less
Submitted 21 February, 2022; v1 submitted 12 October, 2021;
originally announced October 2021.
-
Are Bayesian neural networks intrinsically good at out-of-distribution detection?
Authors:
Christian Henning,
Francesco D'Angelo,
Benjamin F. Grewe
Abstract:
The need to avoid confident predictions on unfamiliar data has sparked interest in out-of-distribution (OOD) detection. It is widely assumed that Bayesian neural networks (BNN) are well suited for this task, as the endowed epistemic uncertainty should lead to disagreement in predictions on outliers. In this paper, we question this assumption and provide empirical evidence that proper Bayesian infe…
▽ More
The need to avoid confident predictions on unfamiliar data has sparked interest in out-of-distribution (OOD) detection. It is widely assumed that Bayesian neural networks (BNN) are well suited for this task, as the endowed epistemic uncertainty should lead to disagreement in predictions on outliers. In this paper, we question this assumption and provide empirical evidence that proper Bayesian inference with common neural network architectures does not necessarily lead to good OOD detection. To circumvent the use of approximate inference, we start by studying the infinite-width case, where Bayesian inference can be exact considering the corresponding Gaussian process. Strikingly, the kernels induced under common architectural choices lead to uncertainties that do not reflect the underlying data generating process and are therefore unsuited for OOD detection. Finally, we study finite-width networks using HMC, and observe OOD behavior that is consistent with the infinite-width case. Overall, our study discloses fundamental problems when naively using BNNs for OOD detection and opens interesting avenues for future research.
△ Less
Submitted 26 July, 2021;
originally announced July 2021.
-
Repulsive Deep Ensembles are Bayesian
Authors:
Francesco D'Angelo,
Vincent Fortuin
Abstract:
Deep ensembles have recently gained popularity in the deep learning community for their conceptual simplicity and efficiency. However, maintaining functional diversity between ensemble members that are independently trained with gradient descent is challenging. This can lead to pathologies when adding more ensemble members, such as a saturation of the ensemble performance, which converges to the p…
▽ More
Deep ensembles have recently gained popularity in the deep learning community for their conceptual simplicity and efficiency. However, maintaining functional diversity between ensemble members that are independently trained with gradient descent is challenging. This can lead to pathologies when adding more ensemble members, such as a saturation of the ensemble performance, which converges to the performance of a single model. Moreover, this does not only affect the quality of its predictions, but even more so the uncertainty estimates of the ensemble, and thus its performance on out-of-distribution data. We hypothesize that this limitation can be overcome by discouraging different ensemble members from collapsing to the same function. To this end, we introduce a kernelized repulsive term in the update rule of the deep ensembles. We show that this simple modification not only enforces and maintains diversity among the members but, even more importantly, transforms the maximum a posteriori inference into proper Bayesian inference. Namely, we show that the training dynamics of our proposed repulsive ensembles follow a Wasserstein gradient flow of the KL divergence with the true posterior. We study repulsive terms in weight and function space and empirically compare their performance to standard ensembles and Bayesian baselines on synthetic and real-world prediction tasks.
△ Less
Submitted 28 March, 2023; v1 submitted 22 June, 2021;
originally announced June 2021.
-
On Stein Variational Neural Network Ensembles
Authors:
Francesco D'Angelo,
Vincent Fortuin,
Florian Wenzel
Abstract:
Ensembles of deep neural networks have achieved great success recently, but they do not offer a proper Bayesian justification. Moreover, while they allow for averaging of predictions over several hypotheses, they do not provide any guarantees for their diversity, leading to redundant solutions in function space. In contrast, particle-based inference methods, such as Stein variational gradient desc…
▽ More
Ensembles of deep neural networks have achieved great success recently, but they do not offer a proper Bayesian justification. Moreover, while they allow for averaging of predictions over several hypotheses, they do not provide any guarantees for their diversity, leading to redundant solutions in function space. In contrast, particle-based inference methods, such as Stein variational gradient descent (SVGD), offer a Bayesian framework, but rely on the choice of a kernel to measure the similarity between ensemble members. In this work, we study different SVGD methods operating in the weight space, function space, and in a hybrid setting. We compare the SVGD approaches to other ensembling-based methods in terms of their theoretical properties and assess their empirical performance on synthetic and real-world tasks. We find that SVGD using functional and hybrid kernels can overcome the limitations of deep ensembles. It improves on functional diversity and uncertainty estimation and approaches the true Bayesian posterior more closely. Moreover, we show that using stochastic SVGD updates, as opposed to the standard deterministic ones, can further improve the performance.
△ Less
Submitted 22 June, 2021; v1 submitted 20 June, 2021;
originally announced June 2021.
-
Posterior Meta-Replay for Continual Learning
Authors:
Christian Henning,
Maria R. Cervera,
Francesco D'Angelo,
Johannes von Oswald,
Regina Traber,
Benjamin Ehret,
Sei** Kobayashi,
Benjamin F. Grewe,
João Sacramento
Abstract:
Learning a sequence of tasks without access to i.i.d. observations is a widely studied form of continual learning (CL) that remains challenging. In principle, Bayesian learning directly applies to this setting, since recursive and one-off Bayesian updates yield the same result. In practice, however, recursive updating often leads to poor trade-off solutions across tasks because approximate inferen…
▽ More
Learning a sequence of tasks without access to i.i.d. observations is a widely studied form of continual learning (CL) that remains challenging. In principle, Bayesian learning directly applies to this setting, since recursive and one-off Bayesian updates yield the same result. In practice, however, recursive updating often leads to poor trade-off solutions across tasks because approximate inference is necessary for most models of interest. Here, we describe an alternative Bayesian approach where task-conditioned parameter distributions are continually inferred from data. We offer a practical deep learning implementation of our framework based on probabilistic task-conditioned hypernetworks, an approach we term posterior meta-replay. Experiments on standard benchmarks show that our probabilistic hypernetworks compress sequences of posterior parameter distributions with virtually no forgetting. We obtain considerable performance gains compared to existing Bayesian CL methods, and identify task inference as our major limiting factor. This limitation has several causes that are independent of the considered sequential setting, opening up new avenues for progress in CL.
△ Less
Submitted 21 October, 2021; v1 submitted 1 March, 2021;
originally announced March 2021.
-
Annealed Stein Variational Gradient Descent
Authors:
Francesco D'Angelo,
Vincent Fortuin
Abstract:
Particle based optimization algorithms have recently been developed as sampling methods that iteratively update a set of particles to approximate a target distribution. In particular Stein variational gradient descent has gained attention in the approximate inference literature for its flexibility and accuracy. We empirically explore the ability of this method to sample from multi-modal distributi…
▽ More
Particle based optimization algorithms have recently been developed as sampling methods that iteratively update a set of particles to approximate a target distribution. In particular Stein variational gradient descent has gained attention in the approximate inference literature for its flexibility and accuracy. We empirically explore the ability of this method to sample from multi-modal distributions and focus on two important issues: (i) the inability of the particles to escape from local modes and (ii) the inefficacy in reproducing the density of the different regions. We propose an annealing schedule to solve these issues and show, through various experiments, how this simple solution leads to significant improvements in mode coverage, without invalidating any theoretical properties of the original algorithm.
△ Less
Submitted 18 March, 2021; v1 submitted 24 January, 2021;
originally announced January 2021.
-
Learning the Ising Model with Generative Neural Networks
Authors:
Francesco D'Angelo,
Lucas Böttcher
Abstract:
Recent advances in deep learning and neural networks have led to an increased interest in the application of generative models in statistical and condensed matter physics. In particular, restricted Boltzmann machines (RBMs) and variational autoencoders (VAEs) as specific classes of neural networks have been successfully applied in the context of physical feature extraction and representation learn…
▽ More
Recent advances in deep learning and neural networks have led to an increased interest in the application of generative models in statistical and condensed matter physics. In particular, restricted Boltzmann machines (RBMs) and variational autoencoders (VAEs) as specific classes of neural networks have been successfully applied in the context of physical feature extraction and representation learning. Despite these successes, however, there is only limited understanding of their representational properties and limitations. To better understand the representational characteristics of RBMs and VAEs, we study their ability to capture physical features of the Ising model at different temperatures. This approach allows us to quantitatively assess learned representations by comparing sample features with corresponding theoretical predictions. Our results suggest that the considered RBMs and convolutional VAEs are able to capture the temperature dependence of magnetization, energy, and spin-spin correlations. The samples generated by RBMs are more evenly distributed across temperature than those generated by VAEs. We also find that convolutional layers in VAEs are important to model spin correlations whereas RBMs achieve similar or even better performances without convolutional filters.
△ Less
Submitted 8 May, 2020; v1 submitted 15 January, 2020;
originally announced January 2020.
-
Chemical communication between synthetic and natural cells: a possible experimental design
Authors:
Giordano Rampioni,
Luisa Damiano,
Marco Messina,
Francesca D'Angelo,
Livia Leoni,
Pasquale Stano
Abstract:
The bottom-up construction of synthetic cells is one of the most intriguing and interesting research arenas in synthetic biology. Synthetic cells are built by encapsulating biomolecules inside lipid vesicles (liposomes), allowing the synthesis of one or more functional proteins. Thanks to the in situ synthesized proteins, synthetic cells become able to perform several biomolecular functions, which…
▽ More
The bottom-up construction of synthetic cells is one of the most intriguing and interesting research arenas in synthetic biology. Synthetic cells are built by encapsulating biomolecules inside lipid vesicles (liposomes), allowing the synthesis of one or more functional proteins. Thanks to the in situ synthesized proteins, synthetic cells become able to perform several biomolecular functions, which can be exploited for a large variety of applications. This paves the way to several advanced uses of synthetic cells in basic science and biotechnology, thanks to their versatility, modularity, biocompatibility, and programmability. In the previous WIVACE (2012) we presented the state-of-the-art of semi-synthetic minimal cell (SSMC) technology and introduced, for the first time, the idea of chemical communication between synthetic cells and natural cells. The development of a proper synthetic communication protocol should be seen as a tool for the nascent field of bio/chemical-based Information and Communication Technologies (bio-chem-ICTs) and ultimately aimed at building soft-wet-micro-robots. In this contribution (WIVACE, 2013) we present a blueprint for realizing this project, and show some preliminary experimental results. We firstly discuss how our research goal (based on the natural capabilities of biological systems to manipulate chemical signals) finds a proper place in the current scientific and technological contexts. Then, we shortly comment on the experimental approaches from the viewpoints of (i) synthetic cell construction, and (ii) bioengineering of microorganisms, providing up-to-date results from our laboratory. Finally, we shortly discuss how autopoiesis can be used as a theoretical framework for defining synthetic minimal life, minimal cognition, and as bridge between synthetic biology and artificial intelligence.
△ Less
Submitted 29 September, 2013;
originally announced September 2013.
