-
Conformal Prediction via Regression-as-Classification
Authors:
Etash Guha,
Shlok Natarajan,
Thomas Möllenhoff,
Mohammad Emtiyaz Khan,
Eugene Ndiaye
Abstract:
Conformal prediction (CP) for regression can be challenging, especially when the output distribution is heteroscedastic, multimodal, or skewed. Some of the issues can be addressed by estimating a distribution over the output, but in reality, such approaches can be sensitive to estimation error and yield unstable intervals.~Here, we circumvent the challenges by converting regression to a classifica…
▽ More
Conformal prediction (CP) for regression can be challenging, especially when the output distribution is heteroscedastic, multimodal, or skewed. Some of the issues can be addressed by estimating a distribution over the output, but in reality, such approaches can be sensitive to estimation error and yield unstable intervals.~Here, we circumvent the challenges by converting regression to a classification problem and then use CP for classification to obtain CP sets for regression.~To preserve the ordering of the continuous-output space, we design a new loss function and make necessary modifications to the CP classification techniques.~Empirical results on many benchmarks shows that this simple approach gives surprisingly good results on many practical problems.
△ Less
Submitted 11 April, 2024;
originally announced April 2024.
-
Space Physiology and Technology: Musculoskeletal Adaptations, Countermeasures, and the Opportunity for Wearable Robotics
Authors:
Shamas Ul Ebad Khan,
Re** John Varghese,
Panagiotis Kassanos,
Dario Farina,
Etienne Burdet
Abstract:
Space poses significant challenges for human physiology, leading to physiological adaptations in response to an environment vastly different from Earth. While these adaptations can be beneficial, they may not fully counteract the adverse impact of space-related stressors. A comprehensive understanding of these physiological adaptations is needed to devise effective countermeasures to support human…
▽ More
Space poses significant challenges for human physiology, leading to physiological adaptations in response to an environment vastly different from Earth. While these adaptations can be beneficial, they may not fully counteract the adverse impact of space-related stressors. A comprehensive understanding of these physiological adaptations is needed to devise effective countermeasures to support human life in space. This review focuses on the impact of the environment in space on the musculoskeletal system. It highlights the complex interplay between bone and muscle adaptation, the underlying physiological mechanisms, and their implications on astronaut health. Furthermore, the review delves into the deployed and current advances in countermeasures and proposes, as a perspective for future developments, wearable sensing and robotic technologies, such as exoskeletons, as a fitting alternative.
△ Less
Submitted 4 April, 2024;
originally announced April 2024.
-
SDXL Finetuned with LoRA for Coloring Therapy: Generating Graphic Templates Inspired by United Arab Emirates Culture
Authors:
Abdulla Alfalasi,
Esrat Khan,
Mohamed Alhashmi,
Raed Aldweik,
Davor Svetinovic
Abstract:
A transformative approach to mental health therapy lies at the crossroads of cultural heritage and advanced technology. This paper introduces an innovative method that fuses machine learning techniques with traditional Emirati motifs, focusing on the United Arab Emirates (UAE). We utilize the Stable Diffusion XL (SDXL) model, enhanced with Low-Rank Adaptation (LoRA), to create culturally significa…
▽ More
A transformative approach to mental health therapy lies at the crossroads of cultural heritage and advanced technology. This paper introduces an innovative method that fuses machine learning techniques with traditional Emirati motifs, focusing on the United Arab Emirates (UAE). We utilize the Stable Diffusion XL (SDXL) model, enhanced with Low-Rank Adaptation (LoRA), to create culturally significant coloring templates featuring Al-Sadu weaving patterns. This novel approach leverages coloring therapy for its recognized stress-relieving benefits and embeds deep cultural resonance, making it a potent tool for therapeutic intervention and cultural preservation. Specifically targeting Generalized Anxiety Disorder (GAD), our method demonstrates significant potential in reducing associated symptoms. Additionally, the paper delves into the broader implications of color and music therapy, emphasizing the importance of culturally tailored content. The technical aspects of the SDXL model and its LoRA fine-tuning showcase its capability to generate high-quality, culturally specific images. This research stands at the forefront of integrating mental wellness practices with cultural heritage, providing a groundbreaking perspective on the synergy between technology, culture, and healthcare. In future work, we aim to employ biosignals to assess the level of engagement and effectiveness of color therapy. A key focus will be to examine the impact of the Emirati heritage Al Sadu art on Emirati individuals and compare their responses with those of other nationalities. This will provide deeper insights into the cultural specificity of therapeutic interventions and further the understanding of the unique interplay between cultural identity and mental health therapy.
△ Less
Submitted 20 February, 2024;
originally announced March 2024.
-
Variational Learning is Effective for Large Deep Networks
Authors:
Yuesong Shen,
Nico Daheim,
Bai Cong,
Peter Nickl,
Gian Maria Marconi,
Clement Bazan,
Rio Yokota,
Iryna Gurevych,
Daniel Cremers,
Mohammad Emtiyaz Khan,
Thomas Möllenhoff
Abstract:
We give extensive empirical evidence against the common belief that variational learning is ineffective for large neural networks. We show that an optimizer called Improved Variational Online Newton (IVON) consistently matches or outperforms Adam for training large networks such as GPT-2 and ResNets from scratch. IVON's computational costs are nearly identical to Adam but its predictive uncertaint…
▽ More
We give extensive empirical evidence against the common belief that variational learning is ineffective for large neural networks. We show that an optimizer called Improved Variational Online Newton (IVON) consistently matches or outperforms Adam for training large networks such as GPT-2 and ResNets from scratch. IVON's computational costs are nearly identical to Adam but its predictive uncertainty is better. We show several new use cases of IVON where we improve finetuning and model merging in Large Language Models, accurately predict generalization error, and faithfully estimate sensitivity to data. We find overwhelming evidence that variational learning is effective.
△ Less
Submitted 6 June, 2024; v1 submitted 27 February, 2024;
originally announced February 2024.
-
Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AI
Authors:
Theodore Papamarkou,
Maria Skoularidou,
Konstantina Palla,
Laurence Aitchison,
Julyan Arbel,
David Dunson,
Maurizio Filippone,
Vincent Fortuin,
Philipp Hennig,
José Miguel Hernández-Lobato,
Aliaksandr Hubin,
Alexander Immer,
Theofanis Karaletsos,
Mohammad Emtiyaz Khan,
Agustinus Kristiadi,
Yingzhen Li,
Stephan Mandt,
Christopher Nemeth,
Michael A. Osborne,
Tim G. J. Rudner,
David Rügamer,
Yee Whye Teh,
Max Welling,
Andrew Gordon Wilson,
Ruqi Zhang
Abstract:
In the current landscape of deep learning research, there is a predominant emphasis on achieving high predictive accuracy in supervised tasks involving large image and language datasets. However, a broader perspective reveals a multitude of overlooked metrics, tasks, and data types, such as uncertainty, active and continual learning, and scientific data, that demand attention. Bayesian deep learni…
▽ More
In the current landscape of deep learning research, there is a predominant emphasis on achieving high predictive accuracy in supervised tasks involving large image and language datasets. However, a broader perspective reveals a multitude of overlooked metrics, tasks, and data types, such as uncertainty, active and continual learning, and scientific data, that demand attention. Bayesian deep learning (BDL) constitutes a promising avenue, offering advantages across these diverse settings. This paper posits that BDL can elevate the capabilities of deep learning. It revisits the strengths of BDL, acknowledges existing challenges, and highlights some exciting research avenues aimed at addressing these obstacles. Looking ahead, the discussion focuses on possible ways to combine large-scale foundation models with BDL to unlock their full potential.
△ Less
Submitted 2 June, 2024; v1 submitted 1 February, 2024;
originally announced February 2024.
-
Shadow: A Novel Loss Function for Efficient Training in Siamese Networks
Authors:
Alif Elham Khan,
Mohammad Junayed Hasan,
Humayra Anjum,
Nabeel Mohammed
Abstract:
Despite significant recent advances in similarity detection tasks, existing approaches pose substantial challenges under memory constraints. One of the primary reasons for this is the use of computationally expensive metric learning loss functions such as Triplet Loss in Siamese networks. In this paper, we present a novel loss function called Shadow Loss that compresses the dimensions of an embedd…
▽ More
Despite significant recent advances in similarity detection tasks, existing approaches pose substantial challenges under memory constraints. One of the primary reasons for this is the use of computationally expensive metric learning loss functions such as Triplet Loss in Siamese networks. In this paper, we present a novel loss function called Shadow Loss that compresses the dimensions of an embedding space during loss calculation without loss of performance. The distance between the projections of the embeddings is learned from inputs on a compact projection space where distances directly correspond to a measure of class similarity. Projecting on a lower-dimension projection space, our loss function converges faster, and the resulting classified image clusters have higher inter-class and smaller intra-class distances. Shadow Loss not only reduces embedding dimensions favoring memory constraint devices but also consistently performs better than the state-of-the-art Triplet Margin Loss by an accuracy of 5\%-10\% across diverse datasets. The proposed loss function is also model agnostic, upholding its performance across several tested models. Its effectiveness and robustness across balanced, imbalanced, medical, and non-medical image datasets suggests that it is not specific to a particular model or dataset but demonstrates superior performance consistently while using less memory and computation.
△ Less
Submitted 23 November, 2023;
originally announced November 2023.
-
The Memory Perturbation Equation: Understanding Model's Sensitivity to Data
Authors:
Peter Nickl,
Lu Xu,
Dharmesh Tailor,
Thomas Möllenhoff,
Mohammad Emtiyaz Khan
Abstract:
Understanding model's sensitivity to its training data is crucial but can also be challenging and costly, especially during training. To simplify such issues, we present the Memory-Perturbation Equation (MPE) which relates model's sensitivity to perturbation in its training data. Derived using Bayesian principles, the MPE unifies existing sensitivity measures, generalizes them to a wide-variety of…
▽ More
Understanding model's sensitivity to its training data is crucial but can also be challenging and costly, especially during training. To simplify such issues, we present the Memory-Perturbation Equation (MPE) which relates model's sensitivity to perturbation in its training data. Derived using Bayesian principles, the MPE unifies existing sensitivity measures, generalizes them to a wide-variety of models and algorithms, and unravels useful properties regarding sensitivities. Our empirical results show that sensitivity estimates obtained during training can be used to faithfully predict generalization on unseen test data. The proposed equation is expected to be useful for future research on robust and adaptive learning.
△ Less
Submitted 16 January, 2024; v1 submitted 30 October, 2023;
originally announced October 2023.
-
Model Merging by Uncertainty-Based Gradient Matching
Authors:
Nico Daheim,
Thomas Möllenhoff,
Edoardo Maria Ponti,
Iryna Gurevych,
Mohammad Emtiyaz Khan
Abstract:
Models trained on different datasets can be merged by a weighted-averaging of their parameters, but why does it work and when can it fail? Here, we connect the inaccuracy of weighted-averaging to mismatches in the gradients and propose a new uncertainty-based scheme to improve the performance by reducing the mismatch. The connection also reveals implicit assumptions in other schemes such as averag…
▽ More
Models trained on different datasets can be merged by a weighted-averaging of their parameters, but why does it work and when can it fail? Here, we connect the inaccuracy of weighted-averaging to mismatches in the gradients and propose a new uncertainty-based scheme to improve the performance by reducing the mismatch. The connection also reveals implicit assumptions in other schemes such as averaging, task arithmetic, and Fisher-weighted averaging. Our new method gives consistent improvements for large language models and vision transformers, both in terms of performance and robustness to hyperparameters.
△ Less
Submitted 19 October, 2023;
originally announced October 2023.
-
Exploiting Inferential Structure in Neural Processes
Authors:
Dharmesh Tailor,
Mohammad Emtiyaz Khan,
Eric Nalisnick
Abstract:
Neural Processes (NPs) are appealing due to their ability to perform fast adaptation based on a context set. This set is encoded by a latent variable, which is often assumed to follow a simple distribution. However, in real-word settings, the context set may be drawn from richer distributions having multiple modes, heavy tails, etc. In this work, we provide a framework that allows NPs' latent vari…
▽ More
Neural Processes (NPs) are appealing due to their ability to perform fast adaptation based on a context set. This set is encoded by a latent variable, which is often assumed to follow a simple distribution. However, in real-word settings, the context set may be drawn from richer distributions having multiple modes, heavy tails, etc. In this work, we provide a framework that allows NPs' latent variable to be given a rich prior defined by a graphical model. These distributional assumptions directly translate into an appropriate aggregation strategy for the context set. Moreover, we describe a message-passing procedure that still allows for end-to-end optimization with stochastic gradients. We demonstrate the generality of our framework by using mixture and Student-t assumptions that yield improvements in function modelling and test-time robustness.
△ Less
Submitted 26 June, 2023;
originally announced June 2023.
-
Memory-Based Dual Gaussian Processes for Sequential Learning
Authors:
Paul E. Chang,
Prakhar Verma,
S. T. John,
Arno Solin,
Mohammad Emtiyaz Khan
Abstract:
Sequential learning with Gaussian processes (GPs) is challenging when access to past data is limited, for example, in continual and active learning. In such cases, errors can accumulate over time due to inaccuracies in the posterior, hyperparameters, and inducing points, making accurate learning challenging. Here, we present a method to keep all such errors in check using the recently proposed dua…
▽ More
Sequential learning with Gaussian processes (GPs) is challenging when access to past data is limited, for example, in continual and active learning. In such cases, errors can accumulate over time due to inaccuracies in the posterior, hyperparameters, and inducing points, making accurate learning challenging. Here, we present a method to keep all such errors in check using the recently proposed dual sparse variational GP. Our method enables accurate inference for generic likelihoods and improves learning by actively building and updating a memory of past data. We demonstrate its effectiveness in several applications involving Bayesian optimization, active learning, and continual learning.
△ Less
Submitted 6 June, 2023;
originally announced June 2023.
-
Behavioral Forensics in Social Networks: Identifying Misinformation, Disinformation and Refutation Spreaders Using Machine Learning
Authors:
Euna Mehnaz Khan,
Ayush Ram,
Bhavtosh Rath,
Emily Vraga,
Jaideep Srivastava
Abstract:
With the ever-increasing spread of misinformation on online social networks, it has become very important to identify the spreaders of misinformation (unintentional), disinformation (intentional), and misinformation refutation. It can help in educating the first, stop** the second, and soliciting the help of the third category, respectively, in the overall effort to counter misinformation spread…
▽ More
With the ever-increasing spread of misinformation on online social networks, it has become very important to identify the spreaders of misinformation (unintentional), disinformation (intentional), and misinformation refutation. It can help in educating the first, stop** the second, and soliciting the help of the third category, respectively, in the overall effort to counter misinformation spread. Existing research to identify spreaders is limited to binary classification (true vs false information spreaders). However, people's intention (whether naive or malicious) behind sharing misinformation can only be understood after observing their behavior after exposure to both the misinformation and its refutation which the existing literature lacks to consider. In this paper, we propose a labeling mechanism to label people as one of the five defined categories based on the behavioral actions they exhibit when exposed to misinformation and its refutation. However, everyone does not show behavioral actions but is part of a network. Therefore, we use their network features, extracted through deep learning-based graph embedding models, to train a machine learning model for the prediction of the classes. We name our approach behavioral forensics since it is an evidence-based investigation of suspicious behavior which is spreading misinformation and disinformation in our case. After evaluating our proposed model on a real-world Twitter dataset, we achieved 77.45% precision and 75.80% recall in detecting the malicious actors, who shared the misinformation even after receiving its refutation. Such behavior shows intention, and hence these actors can rightfully be called agents of disinformation spread.
△ Less
Submitted 1 May, 2023;
originally announced May 2023.
-
Variational Bayes Made Easy
Authors:
Mohammad Emtiyaz Khan
Abstract:
Variational Bayes is a popular method for approximate inference but its derivation can be cumbersome. To simplify the process, we give a 3-step recipe to identify the posterior form by explicitly looking for linearity with respect to expectations of well-known distributions. We can then directly write the update by simply ``reading-off'' the terms in front of those expectations. The recipe makes t…
▽ More
Variational Bayes is a popular method for approximate inference but its derivation can be cumbersome. To simplify the process, we give a 3-step recipe to identify the posterior form by explicitly looking for linearity with respect to expectations of well-known distributions. We can then directly write the update by simply ``reading-off'' the terms in front of those expectations. The recipe makes the derivation easier, faster, shorter, and more general.
△ Less
Submitted 10 July, 2023; v1 submitted 27 April, 2023;
originally announced April 2023.
-
Interpretable histopathology-based prediction of disease relevant features in Inflammatory Bowel Disease biopsies using weakly-supervised deep learning
Authors:
Ricardo Mokhtari,
Azam Hamidinekoo,
Daniel Sutton,
Arthur Lewis,
Bastian Angermann,
Ulf Gehrmann,
Pal Lundin,
Hibret Adissu,
Junmei Cairns,
Jessica Neisen,
Emon Khan,
Daniel Marks,
Nia Khachapuridze,
Talha Qaiser,
Nikolay Burlutskiy
Abstract:
Crohn's Disease (CD) and Ulcerative Colitis (UC) are the two main Inflammatory Bowel Disease (IBD) types. We developed deep learning models to identify histological disease features for both CD and UC using only endoscopic labels. We explored fine-tuning and end-to-end training of two state-of-the-art self-supervised models for predicting three different endoscopic categories (i) CD vs UC (AUC=0.8…
▽ More
Crohn's Disease (CD) and Ulcerative Colitis (UC) are the two main Inflammatory Bowel Disease (IBD) types. We developed deep learning models to identify histological disease features for both CD and UC using only endoscopic labels. We explored fine-tuning and end-to-end training of two state-of-the-art self-supervised models for predicting three different endoscopic categories (i) CD vs UC (AUC=0.87), (ii) normal vs lesional (AUC=0.81), (iii) low vs high disease severity score (AUC=0.80). We produced visual attention maps to interpret what the models learned and validated them with the support of a pathologist, where we observed a strong association between the models' predictions and histopathological inflammatory features of the disease. Additionally, we identified several cases where the model incorrectly predicted normal samples as lesional but were correct on the microscopic level when reviewed by the pathologist. This tendency of histological presentation to be more severe than endoscopic presentation was previously published in the literature. In parallel, we utilised a model trained on the Colon Nuclei Identification and Counting (CoNIC) dataset to predict and explore 6 cell populations. We observed correlation between areas enriched with the predicted immune cells in biopsies and the pathologist's feedback on the attention maps. Finally, we identified several cell level features indicative of disease severity in CD and UC. These models can enhance our understanding about the pathology behind IBD and can shape our strategies for patient stratification in clinical trials.
△ Less
Submitted 16 May, 2023; v1 submitted 20 March, 2023;
originally announced March 2023.
-
The Lie-Group Bayesian Learning Rule
Authors:
Eren Mehmet Kıral,
Thomas Möllenhoff,
Mohammad Emtiyaz Khan
Abstract:
The Bayesian Learning Rule provides a framework for generic algorithm design but can be difficult to use for three reasons. First, it requires a specific parameterization of exponential family. Second, it uses gradients which can be difficult to compute. Third, its update may not always stay on the manifold. We address these difficulties by proposing an extension based on Lie-groups where posterio…
▽ More
The Bayesian Learning Rule provides a framework for generic algorithm design but can be difficult to use for three reasons. First, it requires a specific parameterization of exponential family. Second, it uses gradients which can be difficult to compute. Third, its update may not always stay on the manifold. We address these difficulties by proposing an extension based on Lie-groups where posteriors are parametrized through transformations of an arbitrary base distribution and updated via the group's exponential map. This simplifies all three difficulties for many cases, providing flexible parametrizations through group's action, simple gradient computation through reparameterization, and updates that always stay on the manifold. We use the new learning rule to derive a new algorithm for deep learning with desirable biologically-plausible attributes to learn sparse features. Our work opens a new frontier for the design of new algorithms by exploiting Lie-group structures.
△ Less
Submitted 8 March, 2023;
originally announced March 2023.
-
Frequency-domain Blind Quality Assessment of Blurred and Blocking-artefact Images using Gaussian Process Regression model
Authors:
Maryam Viqar,
Athar A. Moinuddin,
Ekram Khan,
M. Ghanbari
Abstract:
Most of the standard image and video codecs are block-based and depending upon the compression ratio the compressed images/videos suffer from different distortions. At low ratios, blurriness is observed and as compression increases blocking artifacts occur. Generally, in order to reduce blockiness, images are low-pass filtered which leads to more blurriness. Also, in bokeh mode images they are com…
▽ More
Most of the standard image and video codecs are block-based and depending upon the compression ratio the compressed images/videos suffer from different distortions. At low ratios, blurriness is observed and as compression increases blocking artifacts occur. Generally, in order to reduce blockiness, images are low-pass filtered which leads to more blurriness. Also, in bokeh mode images they are commonly seen: blurriness as a result of intentional blurred background while blocking artifact and global blurriness arising due to compression. Therefore, such visual media suffer from both blockiness and blurriness distortions. Along with this, noise is also commonly encountered distortion. Most of the existing works on quality assessment quantify these distortions individually. This paper proposes a methodology to blindly measure overall quality of an image suffering from these distortions, individually as well as jointly. This is achieved by considering the sum of absolute values of low and high-frequency Discrete Frequency Transform (DFT) coefficients defined as sum magnitudes. The number of blocks lying in specific ranges of sum magnitudes including zero-valued AC coefficients and mean of 100 maximum and 100 minimum values of these sum magnitudes are used as feature vectors. These features are then fed to the Machine Learning (ML) based Gaussian Process Regression (GPR) model, which quantifies the image quality. The simulation results show that the proposed method can estimate the quality of images distorted with the blockiness, blurriness, noise and their combinations. It is relatively fast compared to many state-of-art methods, and therefore is suitable for real-time quality monitoring applications.
△ Less
Submitted 5 March, 2023;
originally announced March 2023.
-
Simplifying Momentum-based Positive-definite Submanifold Optimization with Applications to Deep Learning
Authors:
Wu Lin,
Valentin Duruisseaux,
Melvin Leok,
Frank Nielsen,
Mohammad Emtiyaz Khan,
Mark Schmidt
Abstract:
Riemannian submanifold optimization with momentum is computationally challenging because, to ensure that the iterates remain on the submanifold, we often need to solve difficult differential equations. Here, we simplify such difficulties for a class of sparse or structured symmetric positive-definite matrices with the affine-invariant metric. We do so by proposing a generalized version of the Riem…
▽ More
Riemannian submanifold optimization with momentum is computationally challenging because, to ensure that the iterates remain on the submanifold, we often need to solve difficult differential equations. Here, we simplify such difficulties for a class of sparse or structured symmetric positive-definite matrices with the affine-invariant metric. We do so by proposing a generalized version of the Riemannian normal coordinates that dynamically orthonormalizes the metric and locally converts the problem into an unconstrained problem in the Euclidean space. We use our approach to simplify existing approaches for structured covariances and develop matrix-inverse-free $2^\text{nd}$-order optimizers for deep learning with low precision by using only matrix multiplications. Code: https://github.com/yorkerlin/StructuredNGD-DL
△ Less
Submitted 16 March, 2024; v1 submitted 19 February, 2023;
originally announced February 2023.
-
H-LPS: a hybrid approach for user's location privacy in location-based services
Authors:
Sonia Sabir,
Inayat Ali,
Eraj Khan
Abstract:
Applications providing location-based services (LBS) have gained much attention and importance with the notion of the internet of things (IoT). Users are utilizing LBS by providing their location information to third-party service providers. However, location data is very sensitive that can reveal user's private life to adversaries. The passive and pervasive data collection in IoT upsurges serious…
▽ More
Applications providing location-based services (LBS) have gained much attention and importance with the notion of the internet of things (IoT). Users are utilizing LBS by providing their location information to third-party service providers. However, location data is very sensitive that can reveal user's private life to adversaries. The passive and pervasive data collection in IoT upsurges serious issues of location privacy. Privacy-preserving location-based services are a hot research topic. Many anonymization and obfuscation techniques have been proposed to overcome location privacy issues. In this paper, we have proposed a hybrid location privacy scheme (H-LPS), a hybrid scheme mainly based on obfuscation and collaboration for protecting users' location privacy while using location-based services. Obfuscation naturally degrades the quality of service but provides more privacy as compared to anonymization. Our proposed scheme, H-LPS, provides a very high-level of privacy yet provides good accuracy for most of the users. The privacy level and service accuracy of H-LPS are compared with state-of-the-art location privacy schemes and it is shown that H-LPS could be a candidate solution for preserving user location privacy in location-based services.
△ Less
Submitted 15 December, 2022;
originally announced December 2022.
-
Bridging the Gap Between Target Networks and Functional Regularization
Authors:
Alexandre Piche,
Valentin Thomas,
Joseph Marino,
Rafael Pardinas,
Gian Maria Marconi,
Christopher Pal,
Mohammad Emtiyaz Khan
Abstract:
Bootstrap** is behind much of the successes of Deep Reinforcement Learning. However, learning the value function via bootstrap** often leads to unstable training due to fast-changing target values. Target Networks are employed to stabilize training by using an additional set of lagging parameters to estimate the target values. Despite the popularity of Target Networks, their effect on the opti…
▽ More
Bootstrap** is behind much of the successes of Deep Reinforcement Learning. However, learning the value function via bootstrap** often leads to unstable training due to fast-changing target values. Target Networks are employed to stabilize training by using an additional set of lagging parameters to estimate the target values. Despite the popularity of Target Networks, their effect on the optimization is still misunderstood. In this work, we show that they act as an implicit regularizer. This regularizer has disadvantages such as being inflexible and non convex. To overcome these issues, we propose an explicit Functional Regularization that is a convex regularizer in function space and can easily be tuned. We analyze the convergence of our method theoretically and empirically demonstrate that replacing Target Networks with the more theoretically grounded Functional Regularization approach leads to better sample efficiency and performance improvements.
△ Less
Submitted 3 January, 2024; v1 submitted 21 October, 2022;
originally announced October 2022.
-
Can Calibration Improve Sample Prioritization?
Authors:
Ganesh Tata,
Gautham Krishna Gudur,
Gopinath Chennupati,
Mohammad Emtiyaz Khan
Abstract:
Calibration can reduce overconfident predictions of deep neural networks, but can calibration also accelerate training? In this paper, we show that it can when used to prioritize some examples for performing subset selection. We study the effect of popular calibration techniques in selecting better subsets of samples during training (also called sample prioritization) and observe that calibration…
▽ More
Calibration can reduce overconfident predictions of deep neural networks, but can calibration also accelerate training? In this paper, we show that it can when used to prioritize some examples for performing subset selection. We study the effect of popular calibration techniques in selecting better subsets of samples during training (also called sample prioritization) and observe that calibration can improve the quality of subsets, reduce the number of examples per epoch (by at least 70%), and can thereby speed up the overall training process. We further study the effect of using calibrated pre-trained models coupled with calibration during training to guide sample prioritization, which again seems to improve the quality of samples selected.
△ Less
Submitted 15 November, 2022; v1 submitted 12 October, 2022;
originally announced October 2022.
-
SAM as an Optimal Relaxation of Bayes
Authors:
Thomas Möllenhoff,
Mohammad Emtiyaz Khan
Abstract:
Sharpness-aware minimization (SAM) and related adversarial deep-learning methods can drastically improve generalization, but their underlying mechanisms are not yet fully understood. Here, we establish SAM as a relaxation of the Bayes objective where the expected negative-loss is replaced by the optimal convex lower bound, obtained by using the so-called Fenchel biconjugate. The connection enables…
▽ More
Sharpness-aware minimization (SAM) and related adversarial deep-learning methods can drastically improve generalization, but their underlying mechanisms are not yet fully understood. Here, we establish SAM as a relaxation of the Bayes objective where the expected negative-loss is replaced by the optimal convex lower bound, obtained by using the so-called Fenchel biconjugate. The connection enables a new Adam-like extension of SAM to automatically obtain reasonable uncertainty estimates, while sometimes also improving its accuracy. By connecting adversarial and Bayesian methods, our work opens a new path to robustness.
△ Less
Submitted 10 December, 2023; v1 submitted 4 October, 2022;
originally announced October 2022.
-
Dual Parameterization of Sparse Variational Gaussian Processes
Authors:
Vincent Adam,
Paul E. Chang,
Mohammad Emtiyaz Khan,
Arno Solin
Abstract:
Sparse variational Gaussian process (SVGP) methods are a common choice for non-conjugate Gaussian process inference because of their computational benefits. In this paper, we improve their computational efficiency by using a dual parameterization where each data example is assigned dual parameters, similarly to site parameters used in expectation propagation. Our dual parameterization speeds-up in…
▽ More
Sparse variational Gaussian process (SVGP) methods are a common choice for non-conjugate Gaussian process inference because of their computational benefits. In this paper, we improve their computational efficiency by using a dual parameterization where each data example is assigned dual parameters, similarly to site parameters used in expectation propagation. Our dual parameterization speeds-up inference using natural gradient descent, and provides a tighter evidence lower bound for hyperparameter learning. The approach has the same memory cost as the current SVGP methods, but it is faster and more accurate.
△ Less
Submitted 19 January, 2022; v1 submitted 5 November, 2021;
originally announced November 2021.
-
Structured second-order methods via natural gradient descent
Authors:
Wu Lin,
Frank Nielsen,
Mohammad Emtiyaz Khan,
Mark Schmidt
Abstract:
In this paper, we propose new structured second-order methods and structured adaptive-gradient methods obtained by performing natural-gradient descent on structured parameter spaces. Natural-gradient descent is an attractive approach to design new algorithms in many settings such as gradient-free, adaptive-gradient, and second-order methods. Our structured methods not only enjoy a structural invar…
▽ More
In this paper, we propose new structured second-order methods and structured adaptive-gradient methods obtained by performing natural-gradient descent on structured parameter spaces. Natural-gradient descent is an attractive approach to design new algorithms in many settings such as gradient-free, adaptive-gradient, and second-order methods. Our structured methods not only enjoy a structural invariance but also admit a simple expression. Finally, we test the efficiency of our proposed methods on both deterministic non-convex problems and deep learning problems.
△ Less
Submitted 19 February, 2022; v1 submitted 22 July, 2021;
originally announced July 2021.
-
Subset-of-Data Variational Inference for Deep Gaussian-Processes Regression
Authors:
Ayush Jain,
P. K. Srijith,
Mohammad Emtiyaz Khan
Abstract:
Deep Gaussian Processes (DGPs) are multi-layer, flexible extensions of Gaussian processes but their training remains challenging. Sparse approximations simplify the training but often require optimization over a large number of inducing inputs and their locations across layers. In this paper, we simplify the training by setting the locations to a fixed subset of data and sampling the inducing inpu…
▽ More
Deep Gaussian Processes (DGPs) are multi-layer, flexible extensions of Gaussian processes but their training remains challenging. Sparse approximations simplify the training but often require optimization over a large number of inducing inputs and their locations across layers. In this paper, we simplify the training by setting the locations to a fixed subset of data and sampling the inducing inputs from a variational distribution. This reduces the trainable parameters and computation cost without significant performance degradations, as demonstrated by our empirical results on regression problems. Our modifications simplify and stabilize DGP training while making it amenable to sampling schemes for setting the inducing inputs.
△ Less
Submitted 17 July, 2021;
originally announced July 2021.
-
The Bayesian Learning Rule
Authors:
Mohammad Emtiyaz Khan,
Håvard Rue
Abstract:
We show that many machine-learning algorithms are specific instances of a single algorithm called the \emph{Bayesian learning rule}. The rule, derived from Bayesian principles, yields a wide-range of algorithms from fields such as optimization, deep learning, and graphical models. This includes classical algorithms such as ridge regression, Newton's method, and Kalman filter, as well as modern dee…
▽ More
We show that many machine-learning algorithms are specific instances of a single algorithm called the \emph{Bayesian learning rule}. The rule, derived from Bayesian principles, yields a wide-range of algorithms from fields such as optimization, deep learning, and graphical models. This includes classical algorithms such as ridge regression, Newton's method, and Kalman filter, as well as modern deep-learning algorithms such as stochastic-gradient descent, RMSprop, and Dropout. The key idea in deriving such algorithms is to approximate the posterior using candidate distributions estimated by using natural gradients. Different candidate distributions result in different algorithms and further approximations to natural gradients give rise to variants of those algorithms. Our work not only unifies, generalizes, and improves existing algorithms, but also helps us design new ones.
△ Less
Submitted 8 June, 2024; v1 submitted 9 July, 2021;
originally announced July 2021.
-
Knowledge-Adaptation Priors
Authors:
Mohammad Emtiyaz Khan,
Siddharth Swaroop
Abstract:
Humans and animals have a natural ability to quickly adapt to their surroundings, but machine-learning models, when subjected to changes, often require a complete retraining from scratch. We present Knowledge-adaptation priors (K-priors) to reduce the cost of retraining by enabling quick and accurate adaptation for a wide-variety of tasks and models. This is made possible by a combination of weigh…
▽ More
Humans and animals have a natural ability to quickly adapt to their surroundings, but machine-learning models, when subjected to changes, often require a complete retraining from scratch. We present Knowledge-adaptation priors (K-priors) to reduce the cost of retraining by enabling quick and accurate adaptation for a wide-variety of tasks and models. This is made possible by a combination of weight and function-space priors to reconstruct the gradients of the past, which recovers and generalizes many existing, but seemingly-unrelated, adaptation strategies. Training with simple first-order gradient methods can often recover the exact retrained model to an arbitrary accuracy by choosing a sufficiently large memory of the past data. Empirical results show that adaptation with K-priors achieves performance similar to full retraining, but only requires training on a handful of past examples.
△ Less
Submitted 27 October, 2021; v1 submitted 16 June, 2021;
originally announced June 2021.
-
Bridging the Gap Between Target Networks and Functional Regularization
Authors:
Alexandre Piché,
Valentin Thomas,
Rafael Pardinas,
Joseph Marino,
Gian Maria Marconi,
Christopher Pal,
Mohammad Emtiyaz Khan
Abstract:
Bootstrap** is behind much of the successes of deep Reinforcement Learning. However, learning the value function via bootstrap** often leads to unstable training due to fast-changing target values. Target Networks are employed to stabilize training by using an additional set of lagging parameters to estimate the target values. Despite the popularity of Target Networks, their effect on the opti…
▽ More
Bootstrap** is behind much of the successes of deep Reinforcement Learning. However, learning the value function via bootstrap** often leads to unstable training due to fast-changing target values. Target Networks are employed to stabilize training by using an additional set of lagging parameters to estimate the target values. Despite the popularity of Target Networks, their effect on the optimization is still misunderstood. In this work, we show that they act as an implicit regularizer which can be beneficial in some cases, but also have disadvantages such as being inflexible and can result in instabilities, even when vanilla TD(0) converges. To overcome these issues, we propose an explicit Functional Regularization alternative that is flexible and a convex regularizer in function space and we theoretically study its convergence. We conduct an experimental study across a range of environments, discount factors, and off-policiness data collections to investigate the effectiveness of the regularization induced by Target Networks and Functional Regularization in terms of performance, accuracy, and stability. Our findings emphasize that Functional Regularization can be used as a drop-in replacement for Target Networks and result in performance improvement. Furthermore, adjusting both the regularization weight and the network update period in Functional Regularization can result in further performance improvements compared to solely adjusting the network update period as typically done with Target Networks. Our approach also enhances the ability to networks to recover accurate $Q$-values.
△ Less
Submitted 7 September, 2023; v1 submitted 4 June, 2021;
originally announced June 2021.
-
A Novel Falling-Ball Algorithm for Image Segmentation
Authors:
Asra Aslam,
Ekram Khan,
Mohammad Samar Ansari,
M. M. Sufyan Beg
Abstract:
Image segmentation refers to the separation of objects from the background, and has been one of the most challenging aspects of digital image processing. Practically it is impossible to design a segmentation algorithm which has 100% accuracy, and therefore numerous segmentation techniques have been proposed in the literature, each with certain limitations. In this paper, a novel Falling-Ball algor…
▽ More
Image segmentation refers to the separation of objects from the background, and has been one of the most challenging aspects of digital image processing. Practically it is impossible to design a segmentation algorithm which has 100% accuracy, and therefore numerous segmentation techniques have been proposed in the literature, each with certain limitations. In this paper, a novel Falling-Ball algorithm is presented, which is a region-based segmentation algorithm, and an alternative to watershed transform (based on waterfall model). The proposed algorithm detects the catchment basins by assuming that a ball falling from hilly terrains will stop in a catchment basin. Once catchment basins are identified, the association of each pixel with one of the catchment basin is obtained using multi-criterion fuzzy logic. Edges are constructed by dividing image into different catchment basins with the help of a membership function. Finally closed contour algorithm is applied to find closed regions and objects within closed regions are segmented using intensity information. The performance of the proposed algorithm is evaluated both objectively as well as subjectively. Simulation results show that the proposed algorithms gives superior performance over conventional Sobel edge detection methods and the watershed segmentation algorithm. For comparative analysis, various comparison methods are used for demonstrating the superiority of proposed methods over existing segmentation methods.
△ Less
Submitted 12 May, 2021; v1 submitted 6 May, 2021;
originally announced May 2021.
-
Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning
Authors:
Alexander Immer,
Matthias Bauer,
Vincent Fortuin,
Gunnar Rätsch,
Mohammad Emtiyaz Khan
Abstract:
Marginal-likelihood based model-selection, even though promising, is rarely used in deep learning due to estimation difficulties. Instead, most approaches rely on validation data, which may not be readily available. In this work, we present a scalable marginal-likelihood estimation method to select both hyperparameters and network architectures, based on the training data alone. Some hyperparamete…
▽ More
Marginal-likelihood based model-selection, even though promising, is rarely used in deep learning due to estimation difficulties. Instead, most approaches rely on validation data, which may not be readily available. In this work, we present a scalable marginal-likelihood estimation method to select both hyperparameters and network architectures, based on the training data alone. Some hyperparameters can be estimated online during training, simplifying the procedure. Our marginal-likelihood estimate is based on Laplace's method and Gauss-Newton approximations to the Hessian, and it outperforms cross-validation and manual-tuning on standard regression and image classification datasets, especially in terms of calibration and out-of-distribution detection. Our work shows that marginal likelihoods can improve generalization and be useful when validation data is unavailable (e.g., in nonstationary settings).
△ Less
Submitted 15 June, 2021; v1 submitted 11 April, 2021;
originally announced April 2021.
-
Tractable structured natural gradient descent using local parameterizations
Authors:
Wu Lin,
Frank Nielsen,
Mohammad Emtiyaz Khan,
Mark Schmidt
Abstract:
Natural-gradient descent (NGD) on structured parameter spaces (e.g., low-rank covariances) is computationally challenging due to difficult Fisher-matrix computations. We address this issue by using \emph{local-parameter coordinates} to obtain a flexible and efficient NGD method that works well for a wide-variety of structured parameterizations. We show four applications where our method (1) genera…
▽ More
Natural-gradient descent (NGD) on structured parameter spaces (e.g., low-rank covariances) is computationally challenging due to difficult Fisher-matrix computations. We address this issue by using \emph{local-parameter coordinates} to obtain a flexible and efficient NGD method that works well for a wide-variety of structured parameterizations. We show four applications where our method (1) generalizes the exponential natural evolutionary strategy, (2) recovers existing Newton-like algorithms, (3) yields new structured second-order algorithms via matrix groups, and (4) gives new algorithms to learn covariances of Gaussian and Wishart-based distributions. We show results on a range of problems from deep learning, variational inference, and evolution strategies. Our work opens a new direction for scalable structured geometric methods.
△ Less
Submitted 17 January, 2022; v1 submitted 15 February, 2021;
originally announced February 2021.
-
Fast Variational Learning in State-Space Gaussian Process Models
Authors:
Paul E. Chang,
William J. Wilkinson,
Mohammad Emtiyaz Khan,
Arno Solin
Abstract:
Gaussian process (GP) regression with 1D inputs can often be performed in linear time via a stochastic differential equation formulation. However, for non-Gaussian likelihoods, this requires application of approximate inference methods which can make the implementation difficult, e.g., expectation propagation can be numerically unstable and variational inference can be computationally inefficient.…
▽ More
Gaussian process (GP) regression with 1D inputs can often be performed in linear time via a stochastic differential equation formulation. However, for non-Gaussian likelihoods, this requires application of approximate inference methods which can make the implementation difficult, e.g., expectation propagation can be numerically unstable and variational inference can be computationally inefficient. In this paper, we propose a new method that removes such difficulties. Building upon an existing method called conjugate-computation variational inference, our approach enables linear-time inference via Kalman recursions while avoiding numerical instabilities and convergence issues. We provide an efficient JAX implementation which exploits just-in-time compilation and allows for fast automatic differentiation through large for-loops. Overall, our approach leads to fast and stable variational inference in state-space GP models that can be scaled to time series with millions of data points.
△ Less
Submitted 17 July, 2020; v1 submitted 9 July, 2020;
originally announced July 2020.
-
Continual Deep Learning by Functional Regularisation of Memorable Past
Authors:
**bo Pan,
Siddharth Swaroop,
Alexander Immer,
Runa Eschenhagen,
Richard E. Turner,
Mohammad Emtiyaz Khan
Abstract:
Continually learning new skills is important for intelligent systems, yet standard deep learning methods suffer from catastrophic forgetting of the past. Recent works address this with weight regularisation. Functional regularisation, although computationally expensive, is expected to perform better, but rarely does so in practice. In this paper, we fix this issue by using a new functional-regular…
▽ More
Continually learning new skills is important for intelligent systems, yet standard deep learning methods suffer from catastrophic forgetting of the past. Recent works address this with weight regularisation. Functional regularisation, although computationally expensive, is expected to perform better, but rarely does so in practice. In this paper, we fix this issue by using a new functional-regularisation approach that utilises a few memorable past examples crucial to avoid forgetting. By using a Gaussian Process formulation of deep networks, our approach enables training in weight-space while identifying both the memorable past and a functional prior. Our method achieves state-of-the-art performance on standard benchmarks and opens a new direction for life-long learning where regularisation and memory-based methods are naturally combined.
△ Less
Submitted 8 January, 2021; v1 submitted 29 April, 2020;
originally announced April 2020.
-
Training Binary Neural Networks using the Bayesian Learning Rule
Authors:
Xiangming Meng,
Roman Bachmann,
Mohammad Emtiyaz Khan
Abstract:
Neural networks with binary weights are computation-efficient and hardware-friendly, but their training is challenging because it involves a discrete optimization problem. Surprisingly, ignoring the discrete nature of the problem and using gradient-based methods, such as the Straight-Through Estimator, still works well in practice. This raises the question: are there principled approaches which ju…
▽ More
Neural networks with binary weights are computation-efficient and hardware-friendly, but their training is challenging because it involves a discrete optimization problem. Surprisingly, ignoring the discrete nature of the problem and using gradient-based methods, such as the Straight-Through Estimator, still works well in practice. This raises the question: are there principled approaches which justify such methods? In this paper, we propose such an approach using the Bayesian learning rule. The rule, when applied to estimate a Bernoulli distribution over the binary weights, results in an algorithm which justifies some of the algorithmic choices made by the previous approaches. The algorithm not only obtains state-of-the-art performance, but also enables uncertainty estimation for continual learning to avoid catastrophic forgetting. Our work provides a principled approach for training binary neural networks which justifies and extends existing approaches.
△ Less
Submitted 17 August, 2020; v1 submitted 25 February, 2020;
originally announced February 2020.
-
Handling the Positive-Definite Constraint in the Bayesian Learning Rule
Authors:
Wu Lin,
Mark Schmidt,
Mohammad Emtiyaz Khan
Abstract:
The Bayesian learning rule is a natural-gradient variational inference method, which not only contains many existing learning algorithms as special cases but also enables the design of new algorithms. Unfortunately, when variational parameters lie in an open constraint set, the rule may not satisfy the constraint and requires line-searches which could slow down the algorithm. In this work, we addr…
▽ More
The Bayesian learning rule is a natural-gradient variational inference method, which not only contains many existing learning algorithms as special cases but also enables the design of new algorithms. Unfortunately, when variational parameters lie in an open constraint set, the rule may not satisfy the constraint and requires line-searches which could slow down the algorithm. In this work, we address this issue for positive-definite constraints by proposing an improved rule that naturally handles the constraints. Our modification is obtained by using Riemannian gradient methods, and is valid when the approximation attains a \emph{block-coordinate natural parameterization} (e.g., Gaussian distributions and their mixtures). We propose a principled way to derive Riemannian gradients and retractions from scratch. Our method outperforms existing methods without any significant increase in computation. Our work makes it easier to apply the rule in the presence of positive-definite constraints in parameter spaces.
△ Less
Submitted 25 October, 2020; v1 submitted 23 February, 2020;
originally announced February 2020.
-
Stein's Lemma for the Reparameterization Trick with Exponential Family Mixtures
Authors:
Wu Lin,
Mohammad Emtiyaz Khan,
Mark Schmidt
Abstract:
Stein's method (Stein, 1973; 1981) is a powerful tool for statistical applications, and has had a significant impact in machine learning. Stein's lemma plays an essential role in Stein's method. Previous applications of Stein's lemma either required strong technical assumptions or were limited to Gaussian distributions with restricted covariance structures. In this work, we extend Stein's lemma to…
▽ More
Stein's method (Stein, 1973; 1981) is a powerful tool for statistical applications, and has had a significant impact in machine learning. Stein's lemma plays an essential role in Stein's method. Previous applications of Stein's lemma either required strong technical assumptions or were limited to Gaussian distributions with restricted covariance structures. In this work, we extend Stein's lemma to exponential-family mixture distributions including Gaussian distributions with full covariance structures. Our generalization enables us to establish a connection between Stein's lemma and the reparamterization trick to derive gradients of expectations of a large class of functions under weak assumptions. Using this connection, we can derive many new reparameterizable gradient-identities that goes beyond the reach of existing works. For example, we give gradient identities when expectation is taken with respect to Student's t-distribution, skew Gaussian, exponentially modified Gaussian, and normal inverse Gaussian.
△ Less
Submitted 29 October, 2019;
originally announced October 2019.
-
VILD: Variational Imitation Learning with Diverse-quality Demonstrations
Authors:
Voot Tangkaratt,
Bo Han,
Mohammad Emtiyaz Khan,
Masashi Sugiyama
Abstract:
The goal of imitation learning (IL) is to learn a good policy from high-quality demonstrations. However, the quality of demonstrations in reality can be diverse, since it is easier and cheaper to collect demonstrations from a mix of experts and amateurs. IL in such situations can be challenging, especially when the level of demonstrators' expertise is unknown. We propose a new IL method called \un…
▽ More
The goal of imitation learning (IL) is to learn a good policy from high-quality demonstrations. However, the quality of demonstrations in reality can be diverse, since it is easier and cheaper to collect demonstrations from a mix of experts and amateurs. IL in such situations can be challenging, especially when the level of demonstrators' expertise is unknown. We propose a new IL method called \underline{v}ariational \underline{i}mitation \underline{l}earning with \underline{d}iverse-quality demonstrations (VILD), where we explicitly model the level of demonstrators' expertise with a probabilistic graphical model and estimate it along with a reward function. We show that a naive approach to estimation is not suitable to large state and action spaces, and fix its issues by using a variational approach which can be easily implemented using existing reinforcement learning methods. Experiments on continuous-control benchmarks demonstrate that VILD outperforms state-of-the-art methods. Our work enables scalable and data-efficient IL under more realistic settings than before.
△ Less
Submitted 15 September, 2019;
originally announced September 2019.
-
Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximations
Authors:
Wu Lin,
Mohammad Emtiyaz Khan,
Mark Schmidt
Abstract:
Natural-gradient methods enable fast and simple algorithms for variational inference, but due to computational difficulties, their use is mostly limited to \emph{minimal} exponential-family (EF) approximations. In this paper, we extend their application to estimate \emph{structured} approximations such as mixtures of EF distributions. Such approximations can fit complex, multimodal posterior distr…
▽ More
Natural-gradient methods enable fast and simple algorithms for variational inference, but due to computational difficulties, their use is mostly limited to \emph{minimal} exponential-family (EF) approximations. In this paper, we extend their application to estimate \emph{structured} approximations such as mixtures of EF distributions. Such approximations can fit complex, multimodal posterior distributions and are generally more accurate than unimodal EF approximations. By using a \emph{minimal conditional-EF} representation of such approximations, we derive simple natural-gradient updates. Our empirical results demonstrate a faster convergence of our natural-gradient method compared to black-box gradient-based methods with reparameterization gradients. Our work expands the scope of natural gradients for Bayesian inference and makes them more widely applicable than before.
△ Less
Submitted 6 November, 2020; v1 submitted 7 June, 2019;
originally announced June 2019.
-
Practical Deep Learning with Bayesian Principles
Authors:
Kazuki Osawa,
Siddharth Swaroop,
Anirudh Jain,
Runa Eschenhagen,
Richard E. Turner,
Rio Yokota,
Mohammad Emtiyaz Khan
Abstract:
Bayesian methods promise to fix many shortcomings of deep learning, but they are impractical and rarely match the performance of standard methods, let alone improve them. In this paper, we demonstrate practical training of deep networks with natural-gradient variational inference. By applying techniques such as batch normalisation, data augmentation, and distributed training, we achieve similar pe…
▽ More
Bayesian methods promise to fix many shortcomings of deep learning, but they are impractical and rarely match the performance of standard methods, let alone improve them. In this paper, we demonstrate practical training of deep networks with natural-gradient variational inference. By applying techniques such as batch normalisation, data augmentation, and distributed training, we achieve similar performance in about the same number of epochs as the Adam optimiser, even on large datasets such as ImageNet. Importantly, the benefits of Bayesian principles are preserved: predictive probabilities are well-calibrated, uncertainties on out-of-distribution data are improved, and continual-learning performance is boosted. This work enables practical deep learning while preserving benefits of Bayesian principles. A PyTorch implementation is available as a plug-and-play optimiser.
△ Less
Submitted 29 October, 2019; v1 submitted 6 June, 2019;
originally announced June 2019.
-
Approximate Inference Turns Deep Networks into Gaussian Processes
Authors:
Mohammad Emtiyaz Khan,
Alexander Immer,
Ehsan Abedi,
Maciej Korzepa
Abstract:
Deep neural networks (DNN) and Gaussian processes (GP) are two powerful models with several theoretical connections relating them, but the relationship between their training methods is not well understood. In this paper, we show that certain Gaussian posterior approximations for Bayesian DNNs are equivalent to GP posteriors. This enables us to relate solutions and iterations of a deep-learning al…
▽ More
Deep neural networks (DNN) and Gaussian processes (GP) are two powerful models with several theoretical connections relating them, but the relationship between their training methods is not well understood. In this paper, we show that certain Gaussian posterior approximations for Bayesian DNNs are equivalent to GP posteriors. This enables us to relate solutions and iterations of a deep-learning algorithm to GP inference. As a result, we can obtain a GP kernel and a nonlinear feature map while training a DNN. Surprisingly, the resulting kernel is the neural tangent kernel. We show kernels obtained on real datasets and demonstrate the use of the GP marginal likelihood to tune hyperparameters of DNNs. Our work aims to facilitate further research on combining DNNs and GPs in practical settings.
△ Less
Submitted 19 July, 2020; v1 submitted 5 June, 2019;
originally announced June 2019.
-
Scalable Training of Inference Networks for Gaussian-Process Models
Authors:
Jiaxin Shi,
Mohammad Emtiyaz Khan,
Jun Zhu
Abstract:
Inference in Gaussian process (GP) models is computationally challenging for large data, and often difficult to approximate with a small number of inducing points. We explore an alternative approximation that employs stochastic inference networks for a flexible inference. Unfortunately, for such networks, minibatch training is difficult to be able to learn meaningful correlations over function out…
▽ More
Inference in Gaussian process (GP) models is computationally challenging for large data, and often difficult to approximate with a small number of inducing points. We explore an alternative approximation that employs stochastic inference networks for a flexible inference. Unfortunately, for such networks, minibatch training is difficult to be able to learn meaningful correlations over function outputs for a large dataset. We propose an algorithm that enables such training by tracking a stochastic, functional mirror-descent algorithm. At each iteration, this only requires considering a finite number of input locations, resulting in a scalable and easy-to-implement algorithm. Empirical results show comparable and, sometimes, superior performance to existing sparse variational GP methods.
△ Less
Submitted 27 May, 2019;
originally announced May 2019.
-
A Generalization Bound for Online Variational Inference
Authors:
Badr-Eddine Chérief-Abdellatif,
Pierre Alquier,
Mohammad Emtiyaz Khan
Abstract:
Bayesian inference provides an attractive online-learning framework to analyze sequential data, and offers generalization guarantees which hold even with model mismatch and adversaries. Unfortunately, exact Bayesian inference is rarely feasible in practice and approximation methods are usually employed, but do such methods preserve the generalization properties of Bayesian inference ? In this pape…
▽ More
Bayesian inference provides an attractive online-learning framework to analyze sequential data, and offers generalization guarantees which hold even with model mismatch and adversaries. Unfortunately, exact Bayesian inference is rarely feasible in practice and approximation methods are usually employed, but do such methods preserve the generalization properties of Bayesian inference ? In this paper, we show that this is indeed the case for some variational inference (VI) algorithms. We consider a few existing online, tempered VI algorithms, as well as a new algorithm, and derive their generalization bounds. Our theoretical result relies on the convexity of the variational objective, but we argue that the result should hold more generally and present empirical evidence in support of this. Our work in this paper presents theoretical justifications in favor of online algorithms relying on approximate Bayesian methods.
△ Less
Submitted 10 December, 2019; v1 submitted 8 April, 2019;
originally announced April 2019.
-
Decision Support System for Renal Transplantation
Authors:
Ehsan Khan,
Avishek Choudhury,
Amy L Friedman,
Daehan Won
Abstract:
The burgeoning need for kidney transplantation mandates immediate attention. Mismatch of deceased donor-recipient kidney leads to post-transplant death. To ensure ideal kidney donor-recipient match and minimize post-transplant deaths, the paper develops a prediction model that identifies factors that determine the probability of success of renal transplantation, that is, if the kidney procured fro…
▽ More
The burgeoning need for kidney transplantation mandates immediate attention. Mismatch of deceased donor-recipient kidney leads to post-transplant death. To ensure ideal kidney donor-recipient match and minimize post-transplant deaths, the paper develops a prediction model that identifies factors that determine the probability of success of renal transplantation, that is, if the kidney procured from the deceased donor can be transplanted or discarded. The paper conducts a study envelo** data for 584 imported kidneys collected from 12 transplant centers associated with an organ procurement organization located in New York City, NY. The predicting model yielding best performance measures can be beneficial to the healthcare industry. Transplant centers and organ procurement organizations can take advantage of the prediction model to efficiently predict the outcome of kidney transplantation. Consequently, it will reduce the mortality rate caused by mismatching of donor-recipient kidney transplantation during the surgery. Keywords
△ Less
Submitted 11 December, 2018;
originally announced December 2018.
-
TD-Regularized Actor-Critic Methods
Authors:
Simone Parisi,
Voot Tangkaratt,
Jan Peters,
Mohammad Emtiyaz Khan
Abstract:
Actor-critic methods can achieve incredible performance on difficult reinforcement learning problems, but they are also prone to instability. This is partly due to the interaction between the actor and critic during learning, e.g., an inaccurate step taken by one of them might adversely affect the other and destabilize the learning. To avoid such issues, we propose to regularize the learning objec…
▽ More
Actor-critic methods can achieve incredible performance on difficult reinforcement learning problems, but they are also prone to instability. This is partly due to the interaction between the actor and critic during learning, e.g., an inaccurate step taken by one of them might adversely affect the other and destabilize the learning. To avoid such issues, we propose to regularize the learning objective of the actor by penalizing the temporal difference (TD) error of the critic. This improves stability by avoiding large steps in the actor update whenever the critic is highly inaccurate. The resulting method, which we call the TD-regularized actor-critic method, is a simple plug-and-play approach to improve stability and overall performance of the actor-critic methods. Evaluations on standard benchmarks confirm this.
△ Less
Submitted 25 February, 2019; v1 submitted 19 December, 2018;
originally announced December 2018.
-
Privacy-preserving data aggregation in resource-constrained sensor nodes in Internet of Things: A review
Authors:
Inayat Ali,
Sonia Sabir,
Eraj Khan
Abstract:
Privacy problems are lethal and getting more attention than any other issue with the notion of the Internet of Things (IoT). Since IoT has many application areas including smart home, smart grids, smart healthcare system, smart and intelligent transportation and many more. Most of these applications are fueled by the resource-constrained sensor network, such as Smart healthcare system is powered b…
▽ More
Privacy problems are lethal and getting more attention than any other issue with the notion of the Internet of Things (IoT). Since IoT has many application areas including smart home, smart grids, smart healthcare system, smart and intelligent transportation and many more. Most of these applications are fueled by the resource-constrained sensor network, such as Smart healthcare system is powered by Wireless Body Area Network (WBAN) and Smart home and weather monitoring systems are fueled by Wireless Sensor Networks (WSN). In the mentioned application areas sensor node life is a very important aspect of these technologies as it explicitly effects the network life and performance. Data aggregation techniques are used to increase sensor node life by decreasing communication overhead. However, when the data is aggregated at intermediate nodes to reduce communication overhead, data privacy problems becomes more vulnerable. Different Privacy-Preserving Data Aggregation (PPDA) techniques have been proposed to ensure data privacy during data aggregation in resource-constrained sensor nodes. We provide a review and comparative analysis of the state of the art PPDA techniques in this paper. The comparative analysis is based on Computation Cost, Communication overhead, Privacy Level, resistance against malicious aggregator, sensor node life and energy consumption by the sensor node. We have studied the most recent techniques and provide in-depth analysis of the minute steps involved in these techniques. To the best of our knowledge, this survey is the most recent and comprehensive study of PPDA techniques.
△ Less
Submitted 10 December, 2018;
originally announced December 2018.
-
SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradient
Authors:
Aaron Mishkin,
Frederik Kunstner,
Didrik Nielsen,
Mark Schmidt,
Mohammad Emtiyaz Khan
Abstract:
Uncertainty estimation in large deep-learning models is a computationally challenging task, where it is difficult to form even a Gaussian approximation to the posterior distribution. In such situations, existing methods usually resort to a diagonal approximation of the covariance matrix despite, the fact that these matrices are known to result in poor uncertainty estimates. To address this issue,…
▽ More
Uncertainty estimation in large deep-learning models is a computationally challenging task, where it is difficult to form even a Gaussian approximation to the posterior distribution. In such situations, existing methods usually resort to a diagonal approximation of the covariance matrix despite, the fact that these matrices are known to result in poor uncertainty estimates. To address this issue, we propose a new stochastic, low-rank, approximate natural-gradient (SLANG) method for variational inference in large, deep models. Our method estimates a "diagonal plus low-rank" structure based solely on back-propagated gradients of the network log-likelihood. This requires strictly less gradient computations than methods that compute the gradient of the whole variational objective. Empirical evaluations on standard benchmarks confirm that SLANG enables faster and more accurate estimation of uncertainty than mean-field methods, and performs comparably to state-of-the-art methods.
△ Less
Submitted 11 January, 2019; v1 submitted 11 November, 2018;
originally announced November 2018.
-
Fast yet Simple Natural-Gradient Descent for Variational Inference in Complex Models
Authors:
Mohammad Emtiyaz Khan,
Didrik Nielsen
Abstract:
Bayesian inference plays an important role in advancing machine learning, but faces computational challenges when applied to complex models such as deep neural networks. Variational inference circumvents these challenges by formulating Bayesian inference as an optimization problem and solving it using gradient-based optimization. In this paper, we argue in favor of natural-gradient approaches whic…
▽ More
Bayesian inference plays an important role in advancing machine learning, but faces computational challenges when applied to complex models such as deep neural networks. Variational inference circumvents these challenges by formulating Bayesian inference as an optimization problem and solving it using gradient-based optimization. In this paper, we argue in favor of natural-gradient approaches which, unlike their gradient-based counterparts, can improve convergence by exploiting the information geometry of the solutions. We show how to derive fast yet simple natural-gradient updates by using a duality associated with exponential-family distributions. An attractive feature of these methods is that, by using natural-gradients, they are able to extract accurate local approximations for individual model components. We summarize recent results for Bayesian deep learning showing the superiority of natural-gradient approaches over their gradient counterparts.
△ Less
Submitted 2 August, 2018; v1 submitted 12 July, 2018;
originally announced July 2018.
-
Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam
Authors:
Mohammad Emtiyaz Khan,
Didrik Nielsen,
Voot Tangkaratt,
Wu Lin,
Yarin Gal,
Akash Srivastava
Abstract:
Uncertainty computation in deep learning is essential to design robust and reliable systems. Variational inference (VI) is a promising approach for such computation, but requires more effort to implement and execute compared to maximum-likelihood methods. In this paper, we propose new natural-gradient algorithms to reduce such efforts for Gaussian mean-field VI. Our algorithms can be implemented w…
▽ More
Uncertainty computation in deep learning is essential to design robust and reliable systems. Variational inference (VI) is a promising approach for such computation, but requires more effort to implement and execute compared to maximum-likelihood methods. In this paper, we propose new natural-gradient algorithms to reduce such efforts for Gaussian mean-field VI. Our algorithms can be implemented within the Adam optimizer by perturbing the network weights during gradient evaluations, and uncertainty estimates can be cheaply obtained by using the vector that adapts the learning rate. This requires lower memory, computation, and implementation effort than existing VI methods, while obtaining uncertainty estimates of comparable quality. Our empirical results confirm this and further suggest that the weight-perturbation in our algorithm could be useful for exploration in reinforcement learning and stochastic optimization.
△ Less
Submitted 2 August, 2018; v1 submitted 13 June, 2018;
originally announced June 2018.
-
Beyond Unfolding: Exact Recovery of Latent Convex Tensor Decomposition under Reshuffling
Authors:
Chao Li,
Mohammad Emtiyaz Khan,
Zhun Sun,
Gang Niu,
Bo Han,
Shengli Xie,
Qibin Zhao
Abstract:
Exact recovery of tensor decomposition (TD) methods is a desirable property in both unsupervised learning and scientific data analysis. The numerical defects of TD methods, however, limit their practical applications on real-world data. As an alternative, convex tensor decomposition (CTD) was proposed to alleviate these problems, but its exact-recovery property is not properly addressed so far. To…
▽ More
Exact recovery of tensor decomposition (TD) methods is a desirable property in both unsupervised learning and scientific data analysis. The numerical defects of TD methods, however, limit their practical applications on real-world data. As an alternative, convex tensor decomposition (CTD) was proposed to alleviate these problems, but its exact-recovery property is not properly addressed so far. To this end, we focus on latent convex tensor decomposition (LCTD), a practically widely-used CTD model, and rigorously prove a sufficient condition for its exact-recovery property. Furthermore, we show that such property can be also achieved by a more general model than LCTD. In the new model, we generalize the classic tensor (un-)folding into reshuffling operation, a more flexible map** to relocate the entries of the matrix into a tensor. Armed with the reshuffling operations and exact-recovery property, we explore a totally novel application for (generalized) LCTD, i.e., image steganography. Experimental results on synthetic data validate our theory, and results on image steganography show that our method outperforms the state-of-the-art methods.
△ Less
Submitted 28 January, 2020; v1 submitted 22 May, 2018;
originally announced May 2018.
-
Vprop: Variational Inference using RMSprop
Authors:
Mohammad Emtiyaz Khan,
Zuozhu Liu,
Voot Tangkaratt,
Yarin Gal
Abstract:
Many computationally-efficient methods for Bayesian deep learning rely on continuous optimization algorithms, but the implementation of these methods requires significant changes to existing code-bases. In this paper, we propose Vprop, a method for Gaussian variational inference that can be implemented with two minor changes to the off-the-shelf RMSprop optimizer. Vprop also reduces the memory req…
▽ More
Many computationally-efficient methods for Bayesian deep learning rely on continuous optimization algorithms, but the implementation of these methods requires significant changes to existing code-bases. In this paper, we propose Vprop, a method for Gaussian variational inference that can be implemented with two minor changes to the off-the-shelf RMSprop optimizer. Vprop also reduces the memory requirements of Black-Box Variational Inference by half. We derive Vprop using the conjugate-computation variational inference method, and establish its connections to Newton's method, natural-gradient methods, and extended Kalman filters. Overall, this paper presents Vprop as a principled, computationally-efficient, and easy-to-implement method for Bayesian deep learning.
△ Less
Submitted 4 December, 2017;
originally announced December 2017.
-
Variational Adaptive-Newton Method for Explorative Learning
Authors:
Mohammad Emtiyaz Khan,
Wu Lin,
Voot Tangkaratt,
Zuozhu Liu,
Didrik Nielsen
Abstract:
We present the Variational Adaptive Newton (VAN) method which is a black-box optimization method especially suitable for explorative-learning tasks such as active learning and reinforcement learning. Similar to Bayesian methods, VAN estimates a distribution that can be used for exploration, but requires computations that are similar to continuous optimization methods. Our theoretical contribution…
▽ More
We present the Variational Adaptive Newton (VAN) method which is a black-box optimization method especially suitable for explorative-learning tasks such as active learning and reinforcement learning. Similar to Bayesian methods, VAN estimates a distribution that can be used for exploration, but requires computations that are similar to continuous optimization methods. Our theoretical contribution reveals that VAN is a second-order method that unifies existing methods in distinct fields of continuous optimization, variational inference, and evolution strategies. Our experimental results show that VAN performs well on a wide-variety of learning tasks. This work presents a general-purpose explorative-learning method that has the potential to improve learning in areas such as active learning and reinforcement learning.
△ Less
Submitted 15 November, 2017;
originally announced November 2017.
-
Conjugate-Computation Variational Inference : Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Models
Authors:
Mohammad Emtiyaz Khan,
Wu Lin
Abstract:
Variational inference is computationally challenging in models that contain both conjugate and non-conjugate terms. Methods specifically designed for conjugate models, even though computationally efficient, find it difficult to deal with non-conjugate terms. On the other hand, stochastic-gradient methods can handle the non-conjugate terms but they usually ignore the conjugate structure of the mode…
▽ More
Variational inference is computationally challenging in models that contain both conjugate and non-conjugate terms. Methods specifically designed for conjugate models, even though computationally efficient, find it difficult to deal with non-conjugate terms. On the other hand, stochastic-gradient methods can handle the non-conjugate terms but they usually ignore the conjugate structure of the model which might result in slow convergence. In this paper, we propose a new algorithm called Conjugate-computation Variational Inference (CVI) which brings the best of the two worlds together -- it uses conjugate computations for the conjugate terms and employs stochastic gradients for the rest. We derive this algorithm by using a stochastic mirror-descent method in the mean-parameter space, and then expressing each gradient step as a variational inference in a conjugate model. We demonstrate our algorithm's applicability to a large class of models and establish its convergence. Our experimental results show that our method converges much faster than the methods that ignore the conjugate structure of the model.
△ Less
Submitted 13 April, 2017; v1 submitted 13 March, 2017;
originally announced March 2017.