-
Differentiable Cost-Parameterized Monge Map Estimators
Authors:
Samuel Howard,
George Deligiannidis,
Patrick Rebeschini,
James Thornton
Abstract:
Within the field of optimal transport (OT), the choice of ground cost is crucial to ensuring that the optimality of a transport map corresponds to usefulness in real-world applications. It is therefore desirable to use known information to tailor cost functions and hence learn OT maps which are adapted to the problem at hand. By considering a class of neural ground costs whose Monge maps have a kn…
▽ More
Within the field of optimal transport (OT), the choice of ground cost is crucial to ensuring that the optimality of a transport map corresponds to usefulness in real-world applications. It is therefore desirable to use known information to tailor cost functions and hence learn OT maps which are adapted to the problem at hand. By considering a class of neural ground costs whose Monge maps have a known form, we construct a differentiable Monge map estimator which can be optimized to be consistent with known information about an OT map. In doing so, we simultaneously learn both an OT map estimator and a corresponding adapted cost function. Through suitable choices of loss function, our method provides a general approach for incorporating prior information about the Monge map itself when learning adapted OT maps and cost functions.
△ Less
Submitted 12 June, 2024;
originally announced June 2024.
-
Uniform Generalization Bounds on Data-Dependent Hypothesis Sets via PAC-Bayesian Theory on Random Sets
Authors:
Benjamin Dupuis,
Paul Viallard,
George Deligiannidis,
Umut Simsekli
Abstract:
We propose data-dependent uniform generalization bounds by approaching the problem from a PAC-Bayesian perspective. We first apply the PAC-Bayesian framework on `random sets' in a rigorous way, where the training algorithm is assumed to output a data-dependent hypothesis set after observing the training data. This approach allows us to prove data-dependent bounds, which can be applicable in numero…
▽ More
We propose data-dependent uniform generalization bounds by approaching the problem from a PAC-Bayesian perspective. We first apply the PAC-Bayesian framework on `random sets' in a rigorous way, where the training algorithm is assumed to output a data-dependent hypothesis set after observing the training data. This approach allows us to prove data-dependent bounds, which can be applicable in numerous contexts. To highlight the power of our approach, we consider two main applications. First, we propose a PAC-Bayesian formulation of the recently developed fractal-dimension-based generalization bounds. The derived results are shown to be tighter and they unify the existing results around one simple proof technique. Second, we prove uniform bounds over the trajectories of continuous Langevin dynamics and stochastic gradient Langevin dynamics. These results provide novel information about the generalization properties of noisy algorithms.
△ Less
Submitted 26 April, 2024;
originally announced April 2024.
-
Particle Denoising Diffusion Sampler
Authors:
Angus Phillips,
Hai-Dang Dau,
Michael John Hutchinson,
Valentin De Bortoli,
George Deligiannidis,
Arnaud Doucet
Abstract:
Denoising diffusion models have become ubiquitous for generative modeling. The core idea is to transport the data distribution to a Gaussian by using a diffusion. Approximate samples from the data distribution are then obtained by estimating the time-reversal of this diffusion using score matching ideas. We follow here a similar strategy to sample from unnormalized probability densities and comput…
▽ More
Denoising diffusion models have become ubiquitous for generative modeling. The core idea is to transport the data distribution to a Gaussian by using a diffusion. Approximate samples from the data distribution are then obtained by estimating the time-reversal of this diffusion using score matching ideas. We follow here a similar strategy to sample from unnormalized probability densities and compute their normalizing constants. However, the time-reversed diffusion is here simulated by using an original iterative particle scheme relying on a novel score matching loss. Contrary to standard denoising diffusion models, the resulting Particle Denoising Diffusion Sampler (PDDS) provides asymptotically consistent estimates under mild assumptions. We demonstrate PDDS on multimodal and high dimensional sampling tasks.
△ Less
Submitted 15 June, 2024; v1 submitted 9 February, 2024;
originally announced February 2024.
-
Nearly $d$-Linear Convergence Bounds for Diffusion Models via Stochastic Localization
Authors:
Joe Benton,
Valentin De Bortoli,
Arnaud Doucet,
George Deligiannidis
Abstract:
Denoising diffusions are a powerful method to generate approximate samples from high-dimensional data distributions. Recent results provide polynomial bounds on their convergence rate, assuming $L^2$-accurate scores. Until now, the tightest bounds were either superlinear in the data dimension or required strong smoothness assumptions. We provide the first convergence bounds which are linear in the…
▽ More
Denoising diffusions are a powerful method to generate approximate samples from high-dimensional data distributions. Recent results provide polynomial bounds on their convergence rate, assuming $L^2$-accurate scores. Until now, the tightest bounds were either superlinear in the data dimension or required strong smoothness assumptions. We provide the first convergence bounds which are linear in the data dimension (up to logarithmic factors) assuming only finite second moments of the data distribution. We show that diffusion models require at most $\tilde O(\frac{d \log^2(1/δ)}{\varepsilon^2})$ steps to approximate an arbitrary distribution on $\mathbb{R}^d$ corrupted with Gaussian noise of variance $δ$ to within $\varepsilon^2$ in KL divergence. Our proof extends the Girsanov-based methods of previous works. We introduce a refined treatment of the error from discretizing the reverse SDE inspired by stochastic localization.
△ Less
Submitted 5 March, 2024; v1 submitted 7 August, 2023;
originally announced August 2023.
-
On the Expected Size of Conformal Prediction Sets
Authors:
Guneet S. Dhillon,
George Deligiannidis,
Tom Rainforth
Abstract:
While conformal predictors reap the benefits of rigorous statistical guarantees on their error frequency, the size of their corresponding prediction sets is critical to their practical utility. Unfortunately, there is currently a lack of finite-sample analysis and guarantees for their prediction set sizes. To address this shortfall, we theoretically quantify the expected size of the prediction set…
▽ More
While conformal predictors reap the benefits of rigorous statistical guarantees on their error frequency, the size of their corresponding prediction sets is critical to their practical utility. Unfortunately, there is currently a lack of finite-sample analysis and guarantees for their prediction set sizes. To address this shortfall, we theoretically quantify the expected size of the prediction sets under the split conformal prediction framework. As this precise formulation cannot usually be calculated directly, we further derive point estimates and high-probability interval bounds that can be empirically computed, providing a practical method for characterizing the expected set size. We corroborate the efficacy of our results with experiments on real-world datasets for both regression and classification problems.
△ Less
Submitted 8 March, 2024; v1 submitted 12 June, 2023;
originally announced June 2023.
-
A Unified Framework for U-Net Design and Analysis
Authors:
Christopher Williams,
Fabian Falck,
George Deligiannidis,
Chris Holmes,
Arnaud Doucet,
Saifuddin Syed
Abstract:
U-Nets are a go-to, state-of-the-art neural architecture across numerous tasks for continuous signals on a square such as images and Partial Differential Equations (PDE), however their design and architecture is understudied. In this paper, we provide a framework for designing and analysing general U-Net architectures. We present theoretical results which characterise the role of the encoder and d…
▽ More
U-Nets are a go-to, state-of-the-art neural architecture across numerous tasks for continuous signals on a square such as images and Partial Differential Equations (PDE), however their design and architecture is understudied. In this paper, we provide a framework for designing and analysing general U-Net architectures. We present theoretical results which characterise the role of the encoder and decoder in a U-Net, their high-resolution scaling limits and their conjugacy to ResNets via preconditioning. We propose Multi-ResNets, U-Nets with a simplified, wavelet-based encoder without learnable parameters. Further, we show how to design novel U-Net architectures which encode function constraints, natural bases, or the geometry of the data. In diffusion models, our framework enables us to identify that high-frequency information is dominated by noise exponentially faster, and show how U-Nets with average pooling exploit this. In our experiments, we demonstrate how Multi-ResNets achieve competitive and often superior performance compared to classical U-Nets in image segmentation, PDE surrogate modelling, and generative modelling with diffusion models. Our U-Net framework paves the way to study the theoretical properties of U-Nets and design natural, scalable neural architectures for a multitude of problems beyond the square.
△ Less
Submitted 10 January, 2024; v1 submitted 31 May, 2023;
originally announced May 2023.
-
Error Bounds for Flow Matching Methods
Authors:
Joe Benton,
George Deligiannidis,
Arnaud Doucet
Abstract:
Score-based generative models are a popular class of generative modelling techniques relying on stochastic differential equations (SDE). From their inception, it was realized that it was also possible to perform generation using ordinary differential equations (ODE) rather than SDE. This led to the introduction of the probability flow ODE approach and denoising diffusion implicit models. Flow matc…
▽ More
Score-based generative models are a popular class of generative modelling techniques relying on stochastic differential equations (SDE). From their inception, it was realized that it was also possible to perform generation using ordinary differential equations (ODE) rather than SDE. This led to the introduction of the probability flow ODE approach and denoising diffusion implicit models. Flow matching methods have recently further extended these ODE-based approaches and approximate a flow between two arbitrary probability distributions. Previous work derived bounds on the approximation error of diffusion models under the stochastic sampling regime, given assumptions on the $L^2$ loss. We present error bounds for the flow matching procedure using fully deterministic sampling, assuming an $L^2$ bound on the approximation error and a certain regularity condition on the data distributions.
△ Less
Submitted 11 February, 2024; v1 submitted 26 May, 2023;
originally announced May 2023.
-
Generalization Bounds with Data-dependent Fractal Dimensions
Authors:
Benjamin Dupuis,
George Deligiannidis,
Umut Şimşekli
Abstract:
Providing generalization guarantees for modern neural networks has been a crucial task in statistical learning. Recently, several studies have attempted to analyze the generalization error in such settings by using tools from fractal geometry. While these works have successfully introduced new mathematical tools to apprehend generalization, they heavily rely on a Lipschitz continuity assumption, w…
▽ More
Providing generalization guarantees for modern neural networks has been a crucial task in statistical learning. Recently, several studies have attempted to analyze the generalization error in such settings by using tools from fractal geometry. While these works have successfully introduced new mathematical tools to apprehend generalization, they heavily rely on a Lipschitz continuity assumption, which in general does not hold for neural networks and might make the bounds vacuous. In this work, we address this issue and prove fractal geometry-based generalization bounds without requiring any Lipschitz assumption. To achieve this goal, we build up on a classical covering argument in learning theory and introduce a data-dependent fractal dimension. Despite introducing a significant amount of technical complications, this new notion lets us control the generalization error (over either fixed or random hypothesis spaces) along with certain mutual information (MI) terms. To provide a clearer interpretation to the newly introduced MI terms, as a next step, we introduce a notion of "geometric stability" and link our bounds to the prior art. Finally, we make a rigorous connection between the proposed data-dependent dimension and topological data analysis tools, which then enables us to compute the dimension in a numerically efficient way. We support our theory with experiments conducted on various settings.
△ Less
Submitted 10 July, 2023; v1 submitted 6 February, 2023;
originally announced February 2023.
-
A Multi-Resolution Framework for U-Nets with Applications to Hierarchical VAEs
Authors:
Fabian Falck,
Christopher Williams,
Dominic Danks,
George Deligiannidis,
Christopher Yau,
Chris Holmes,
Arnaud Doucet,
Matthew Willetts
Abstract:
U-Net architectures are ubiquitous in state-of-the-art deep learning, however their regularisation properties and relationship to wavelets are understudied. In this paper, we formulate a multi-resolution framework which identifies U-Nets as finite-dimensional truncations of models on an infinite-dimensional function space. We provide theoretical results which prove that average pooling corresponds…
▽ More
U-Net architectures are ubiquitous in state-of-the-art deep learning, however their regularisation properties and relationship to wavelets are understudied. In this paper, we formulate a multi-resolution framework which identifies U-Nets as finite-dimensional truncations of models on an infinite-dimensional function space. We provide theoretical results which prove that average pooling corresponds to projection within the space of square-integrable functions and show that U-Nets with average pooling implicitly learn a Haar wavelet basis representation of the data. We then leverage our framework to identify state-of-the-art hierarchical VAEs (HVAEs), which have a U-Net architecture, as a type of two-step forward Euler discretisation of multi-resolution diffusion processes which flow from a point mass, introducing sampling instabilities. We also demonstrate that HVAEs learn a representation of time which allows for improved parameter efficiency through weight-sharing. We use this observation to achieve state-of-the-art HVAE performance with half the number of parameters of existing models, exploiting the properties of our continuous-time formulation.
△ Less
Submitted 19 January, 2023;
originally announced January 2023.
-
From Denoising Diffusions to Denoising Markov Models
Authors:
Joe Benton,
Yuyang Shi,
Valentin De Bortoli,
George Deligiannidis,
Arnaud Doucet
Abstract:
Denoising diffusions are state-of-the-art generative models exhibiting remarkable empirical performance. They work by diffusing the data distribution into a Gaussian distribution and then learning to reverse this noising process to obtain synthetic datapoints. The denoising diffusion relies on approximations of the logarithmic derivatives of the noised data densities using score matching. Such mod…
▽ More
Denoising diffusions are state-of-the-art generative models exhibiting remarkable empirical performance. They work by diffusing the data distribution into a Gaussian distribution and then learning to reverse this noising process to obtain synthetic datapoints. The denoising diffusion relies on approximations of the logarithmic derivatives of the noised data densities using score matching. Such models can also be used to perform approximate posterior simulation when one can only sample from the prior and likelihood. We propose a unifying framework generalising this approach to a wide class of spaces and leading to an original extension of score matching. We illustrate the resulting models on various applications.
△ Less
Submitted 18 February, 2024; v1 submitted 7 November, 2022;
originally announced November 2022.
-
Generalisation under gradient descent via deterministic PAC-Bayes
Authors:
Eugenio Clerico,
Tyler Farghly,
George Deligiannidis,
Benjamin Guedj,
Arnaud Doucet
Abstract:
We establish disintegrated PAC-Bayesian generalisation bounds for models trained with gradient descent methods or continuous gradient flows. Contrary to standard practice in the PAC-Bayesian setting, our result applies to optimisation algorithms that are deterministic, without requiring any de-randomisation step. Our bounds are fully computable, depending on the density of the initial distribution…
▽ More
We establish disintegrated PAC-Bayesian generalisation bounds for models trained with gradient descent methods or continuous gradient flows. Contrary to standard practice in the PAC-Bayesian setting, our result applies to optimisation algorithms that are deterministic, without requiring any de-randomisation step. Our bounds are fully computable, depending on the density of the initial distribution and the Hessian of the training objective over the trajectory. We show that our framework can be applied to a variety of iterative optimisation algorithms, including stochastic gradient descent (SGD), momentum-based schemes, and damped Hamiltonian dynamics.
△ Less
Submitted 4 April, 2023; v1 submitted 6 September, 2022;
originally announced September 2022.
-
Ranking In Generalized Linear Bandits
Authors:
Amitis Shidani,
George Deligiannidis,
Arnaud Doucet
Abstract:
We study the ranking problem in generalized linear bandits. At each time, the learning agent selects an ordered list of items and observes stochastic outcomes. In recommendation systems, displaying an ordered list of the most attractive items is not always optimal as both position and item dependencies result in a complex reward function. A very naive example is the lack of diversity when all the…
▽ More
We study the ranking problem in generalized linear bandits. At each time, the learning agent selects an ordered list of items and observes stochastic outcomes. In recommendation systems, displaying an ordered list of the most attractive items is not always optimal as both position and item dependencies result in a complex reward function. A very naive example is the lack of diversity when all the most attractive items are from the same category. We model the position and item dependencies in the ordered list and design UCB and Thompson Sampling type algorithms for this problem. Our work generalizes existing studies in several directions, including position dependencies where position discount is a particular case, and connecting the ranking problem to graph theory.
△ Less
Submitted 1 January, 2024; v1 submitted 30 June, 2022;
originally announced July 2022.
-
A Continuous Time Framework for Discrete Denoising Models
Authors:
Andrew Campbell,
Joe Benton,
Valentin De Bortoli,
Tom Rainforth,
George Deligiannidis,
Arnaud Doucet
Abstract:
We provide the first complete continuous time framework for denoising diffusion models of discrete data. This is achieved by formulating the forward noising process and corresponding reverse time generative process as Continuous Time Markov Chains (CTMCs). The model can be efficiently trained using a continuous time version of the ELBO. We simulate the high dimensional CTMC using techniques develo…
▽ More
We provide the first complete continuous time framework for denoising diffusion models of discrete data. This is achieved by formulating the forward noising process and corresponding reverse time generative process as Continuous Time Markov Chains (CTMCs). The model can be efficiently trained using a continuous time version of the ELBO. We simulate the high dimensional CTMC using techniques developed in chemical physics and exploit our continuous time framework to derive high performance samplers that we show can outperform discrete time methods for discrete data. The continuous time treatment also enables us to derive a novel theoretical result bounding the error between the generated sample distribution and the true data distribution.
△ Less
Submitted 14 October, 2022; v1 submitted 30 May, 2022;
originally announced May 2022.
-
Chained Generalisation Bounds
Authors:
Eugenio Clerico,
Amitis Shidani,
George Deligiannidis,
Arnaud Doucet
Abstract:
This work discusses how to derive upper bounds for the expected generalisation error of supervised learning algorithms by means of the chaining technique. By develo** a general theoretical framework, we establish a duality between generalisation bounds based on the regularity of the loss function, and their chained counterparts, which can be obtained by lifting the regularity assumption from the…
▽ More
This work discusses how to derive upper bounds for the expected generalisation error of supervised learning algorithms by means of the chaining technique. By develo** a general theoretical framework, we establish a duality between generalisation bounds based on the regularity of the loss function, and their chained counterparts, which can be obtained by lifting the regularity assumption from the loss onto its gradient. This allows us to re-derive the chaining mutual information bound from the literature, and to obtain novel chained information-theoretic generalisation bounds, based on the Wasserstein distance and other probability metrics. We show on some toy examples that the chained generalisation bound can be significantly tighter than its standard counterpart, particularly when the distribution of the hypotheses selected by the algorithm is very concentrated.
Keywords: Generalisation bounds; Chaining; Information-theoretic bounds; Mutual information; Wasserstein distance; PAC-Bayes.
△ Less
Submitted 30 June, 2022; v1 submitted 2 March, 2022;
originally announced March 2022.
-
Neural Score Matching for High-Dimensional Causal Inference
Authors:
Oscar Clivio,
Fabian Falck,
Brieuc Lehmann,
George Deligiannidis,
Chris Holmes
Abstract:
Traditional methods for matching in causal inference are impractical for high-dimensional datasets. They suffer from the curse of dimensionality: exact matching and coarsened exact matching find exponentially fewer matches as the input dimension grows, and propensity score matching may match highly unrelated units together. To overcome this problem, we develop theoretical results which motivate th…
▽ More
Traditional methods for matching in causal inference are impractical for high-dimensional datasets. They suffer from the curse of dimensionality: exact matching and coarsened exact matching find exponentially fewer matches as the input dimension grows, and propensity score matching may match highly unrelated units together. To overcome this problem, we develop theoretical results which motivate the use of neural networks to obtain non-trivial, multivariate balancing scores of a chosen level of coarseness, in contrast to the classical, scalar propensity score. We leverage these balancing scores to perform matching for high-dimensional causal inference and call this procedure neural score matching. We show that our method is competitive against other matching approaches on semi-synthetic high-dimensional datasets, both in terms of treatment effect estimation and reducing imbalance.
△ Less
Submitted 1 March, 2022;
originally announced March 2022.
-
Conditional Simulation Using Diffusion Schrödinger Bridges
Authors:
Yuyang Shi,
Valentin De Bortoli,
George Deligiannidis,
Arnaud Doucet
Abstract:
Denoising diffusion models have recently emerged as a powerful class of generative models. They provide state-of-the-art results, not only for unconditional simulation, but also when used to solve conditional simulation problems arising in a wide range of inverse problems. A limitation of these models is that they are computationally intensive at generation time as they require simulating a diffus…
▽ More
Denoising diffusion models have recently emerged as a powerful class of generative models. They provide state-of-the-art results, not only for unconditional simulation, but also when used to solve conditional simulation problems arising in a wide range of inverse problems. A limitation of these models is that they are computationally intensive at generation time as they require simulating a diffusion process over a long time horizon. When performing unconditional simulation, a Schrödinger bridge formulation of generative modeling leads to a theoretically grounded algorithm shortening generation time which is complementary to other proposed acceleration techniques. We extend the Schrödinger bridge framework to conditional simulation. We demonstrate this novel methodology on various applications including image super-resolution, optimal filtering for state-space models and the refinement of pre-trained networks. Our code can be found at https://github.com/vdeborto/cdsb.
△ Less
Submitted 26 June, 2022; v1 submitted 27 February, 2022;
originally announced February 2022.
-
On Mixing Times of Metropolized Algorithm With Optimization Step (MAO) : A New Framework
Authors:
EL Mahdi Khribch,
George Deligiannidis,
Daniel Paulin
Abstract:
In this paper, we consider sampling from a class of distributions with thin tails supported on $\mathbb{R}^d$ and make two primary contributions. First, we propose a new Metropolized Algorithm With Optimization Step (MAO), which is well suited for such targets. Our algorithm is capable of sampling from distributions where the Metropolis-adjusted Langevin algorithm (MALA) is not converging or lacki…
▽ More
In this paper, we consider sampling from a class of distributions with thin tails supported on $\mathbb{R}^d$ and make two primary contributions. First, we propose a new Metropolized Algorithm With Optimization Step (MAO), which is well suited for such targets. Our algorithm is capable of sampling from distributions where the Metropolis-adjusted Langevin algorithm (MALA) is not converging or lacking in theoretical guarantees. Second, we derive upper bounds on the mixing time of MAO. Our results are supported by simulations on multiple target distributions.
△ Less
Submitted 1 December, 2021;
originally announced December 2021.
-
Conditionally Gaussian PAC-Bayes
Authors:
Eugenio Clerico,
George Deligiannidis,
Arnaud Doucet
Abstract:
Recent studies have empirically investigated different methods to train stochastic neural networks on a classification task by optimising a PAC-Bayesian bound via stochastic gradient descent. Most of these procedures need to replace the misclassification error with a surrogate loss, leading to a mismatch between the optimisation objective and the actual generalisation bound. The present paper prop…
▽ More
Recent studies have empirically investigated different methods to train stochastic neural networks on a classification task by optimising a PAC-Bayesian bound via stochastic gradient descent. Most of these procedures need to replace the misclassification error with a surrogate loss, leading to a mismatch between the optimisation objective and the actual generalisation bound. The present paper proposes a novel training algorithm that optimises the PAC-Bayesian bound, without relying on any surrogate loss. Empirical results show that this approach outperforms currently available PAC-Bayesian training methods.
△ Less
Submitted 24 February, 2022; v1 submitted 22 October, 2021;
originally announced October 2021.
-
Quantitative Uniform Stability of the Iterative Proportional Fitting Procedure
Authors:
George Deligiannidis,
Valentin De Bortoli,
Arnaud Doucet
Abstract:
We establish the uniform in time stability, w.r.t. the marginals, of the Iterative Proportional Fitting Procedure, also known as Sinkhorn algorithm, used to solve entropy-regularised Optimal Transport problems. Our result is quantitative and stated in terms of the 1-Wasserstein metric. As a corollary we establish a quantitative stability result for Schrödinger bridges.
We establish the uniform in time stability, w.r.t. the marginals, of the Iterative Proportional Fitting Procedure, also known as Sinkhorn algorithm, used to solve entropy-regularised Optimal Transport problems. Our result is quantitative and stated in terms of the 1-Wasserstein metric. As a corollary we establish a quantitative stability result for Schrödinger bridges.
△ Less
Submitted 22 October, 2021; v1 submitted 18 August, 2021;
originally announced August 2021.
-
Wide stochastic networks: Gaussian limit and PAC-Bayesian training
Authors:
Eugenio Clerico,
George Deligiannidis,
Arnaud Doucet
Abstract:
The limit of infinite width allows for substantial simplifications in the analytical study of over-parameterised neural networks. With a suitable random initialisation, an extremely large network exhibits an approximately Gaussian behaviour. In the present work, we establish a similar result for a simple stochastic architecture whose parameters are random variables, holding both before and during…
▽ More
The limit of infinite width allows for substantial simplifications in the analytical study of over-parameterised neural networks. With a suitable random initialisation, an extremely large network exhibits an approximately Gaussian behaviour. In the present work, we establish a similar result for a simple stochastic architecture whose parameters are random variables, holding both before and during training. The explicit evaluation of the output distribution allows for a PAC-Bayesian training procedure that directly optimises the generalisation bound. For a large but finite-width network, we show empirically on MNIST that this training approach can outperform standard PAC-Bayesian methods.
△ Less
Submitted 13 February, 2023; v1 submitted 17 June, 2021;
originally announced June 2021.
-
Fractal Structure and Generalization Properties of Stochastic Optimization Algorithms
Authors:
Alexander Camuto,
George Deligiannidis,
Murat A. Erdogdu,
Mert Gürbüzbalaban,
Umut Şimşekli,
Lingjiong Zhu
Abstract:
Understanding generalization in deep learning has been one of the major challenges in statistical learning theory over the last decade. While recent work has illustrated that the dataset and the training algorithm must be taken into account in order to obtain meaningful generalization bounds, it is still theoretically not clear which properties of the data and the algorithm determine the generaliz…
▽ More
Understanding generalization in deep learning has been one of the major challenges in statistical learning theory over the last decade. While recent work has illustrated that the dataset and the training algorithm must be taken into account in order to obtain meaningful generalization bounds, it is still theoretically not clear which properties of the data and the algorithm determine the generalization performance. In this study, we approach this problem from a dynamical systems theory perspective and represent stochastic optimization algorithms as random iterated function systems (IFS). Well studied in the dynamical systems literature, under mild assumptions, such IFSs can be shown to be ergodic with an invariant measure that is often supported on sets with a fractal structure. As our main contribution, we prove that the generalization error of a stochastic optimization algorithm can be bounded based on the `complexity' of the fractal structure that underlies its invariant measure. Leveraging results from dynamical systems theory, we show that the generalization error can be explicitly linked to the choice of the algorithm (e.g., stochastic gradient descent -- SGD), algorithm hyperparameters (e.g., step-size, batch-size), and the geometry of the problem (e.g., Hessian of the loss). We further specialize our results to specific problems (e.g., linear/logistic regression, one hidden-layered neural networks) and algorithms (e.g., SGD and preconditioned variants), and obtain analytical estimates for our bound.For modern neural networks, we develop an efficient algorithm to compute the developed bound and support our theory with various experiments on neural networks.
△ Less
Submitted 9 June, 2021;
originally announced June 2021.
-
Differentiable Particle Filtering via Entropy-Regularized Optimal Transport
Authors:
Adrien Corenflos,
James Thornton,
George Deligiannidis,
Arnaud Doucet
Abstract:
Particle Filtering (PF) methods are an established class of procedures for performing inference in non-linear state-space models. Resampling is a key ingredient of PF, necessary to obtain low variance likelihood and states estimates. However, traditional resampling methods result in PF-based loss functions being non-differentiable with respect to model and PF parameters. In a variational inference…
▽ More
Particle Filtering (PF) methods are an established class of procedures for performing inference in non-linear state-space models. Resampling is a key ingredient of PF, necessary to obtain low variance likelihood and states estimates. However, traditional resampling methods result in PF-based loss functions being non-differentiable with respect to model and PF parameters. In a variational inference context, resampling also yields high variance gradient estimates of the PF-based evidence lower bound. By leveraging optimal transport ideas, we introduce a principled differentiable particle filter and provide convergence results. We demonstrate this novel method on a variety of applications.
△ Less
Submitted 30 June, 2021; v1 submitted 15 February, 2021;
originally announced February 2021.
-
Stable ResNet
Authors:
Soufiane Hayou,
Eugenio Clerico,
Bobby He,
George Deligiannidis,
Arnaud Doucet,
Judith Rousseau
Abstract:
Deep ResNet architectures have achieved state of the art performance on many tasks. While they solve the problem of gradient vanishing, they might suffer from gradient exploding as the depth becomes large (Yang et al. 2017). Moreover, recent results have shown that ResNet might lose expressivity as the depth goes to infinity (Yang et al. 2017, Hayou et al. 2019). To resolve these issues, we introd…
▽ More
Deep ResNet architectures have achieved state of the art performance on many tasks. While they solve the problem of gradient vanishing, they might suffer from gradient exploding as the depth becomes large (Yang et al. 2017). Moreover, recent results have shown that ResNet might lose expressivity as the depth goes to infinity (Yang et al. 2017, Hayou et al. 2019). To resolve these issues, we introduce a new class of ResNet architectures, called Stable ResNet, that have the property of stabilizing the gradient while ensuring expressivity in the infinite depth limit.
△ Less
Submitted 18 March, 2021; v1 submitted 24 October, 2020;
originally announced October 2020.
-
Simulated tempering with irreversible Gibbs sampling techniques
Authors:
Fahim Faizi,
Pedro J. Buigues,
George Deligiannidis,
Edina Rosta
Abstract:
We present here two novel algorithms for simulated tempering simulations, which break detailed balance condition (DBC) but satisfy the skewed detailed balance to ensure invariance of the target distribution. The irreversible methods we present here are based on Gibbs sampling and concern breaking DBC at the update scheme of the temperature swaps. We utilise three systems as a test bed for our meth…
▽ More
We present here two novel algorithms for simulated tempering simulations, which break detailed balance condition (DBC) but satisfy the skewed detailed balance to ensure invariance of the target distribution. The irreversible methods we present here are based on Gibbs sampling and concern breaking DBC at the update scheme of the temperature swaps. We utilise three systems as a test bed for our methods: an MCMC simulation on a simple system described by a 1D double well potential, the Ising model and MD simulations on Alanine pentapeptide (ALA5). The relaxation times of inverse temperature, magnetic susceptibility and energy density for the Ising model indicate clear gains in sampling efficiency over conventional Gibbs sampling techniques with DBC and also over the conventionally used simulated tempering with Metropolis-Hastings (MH) scheme. Simulations on ALA5 with large number of temperatures indicate distinct gains in mixing times for inverse temperature and consequently the energy of the system compared to conventional MH. With no additional computational overhead, our methods were found to be more efficient alternatives to conventionally used simulated tempering methods with DBC. Our algorithms should be particularly advantageous in simulations of large systems with many temperature ladders, as our algorithms showed a more favorable constant scaling in Ising spin systems as compared with both reversible and irreversible MH algorithms. In future applications, our irreversible methods can also be easily tailored to utilize a given dynamical variable other than temperature to flatten rugged free energy landscapes.
△ Less
Submitted 13 August, 2020;
originally announced August 2020.
-
Hausdorff Dimension, Heavy Tails, and Generalization in Neural Networks
Authors:
Umut Şimşekli,
Ozan Sener,
George Deligiannidis,
Murat A. Erdogdu
Abstract:
Despite its success in a wide range of applications, characterizing the generalization properties of stochastic gradient descent (SGD) in non-convex deep learning problems is still an important challenge. While modeling the trajectories of SGD via stochastic differential equations (SDE) under heavy-tailed gradient noise has recently shed light over several peculiar characteristics of SGD, a rigoro…
▽ More
Despite its success in a wide range of applications, characterizing the generalization properties of stochastic gradient descent (SGD) in non-convex deep learning problems is still an important challenge. While modeling the trajectories of SGD via stochastic differential equations (SDE) under heavy-tailed gradient noise has recently shed light over several peculiar characteristics of SGD, a rigorous treatment of the generalization properties of such SDEs in a learning theoretical framework is still missing. Aiming to bridge this gap, in this paper, we prove generalization bounds for SGD under the assumption that its trajectories can be well-approximated by a \emph{Feller process}, which defines a rich class of Markov processes that include several recent SDE representations (both Brownian or heavy-tailed) as its special case. We show that the generalization error can be controlled by the \emph{Hausdorff dimension} of the trajectories, which is intimately linked to the tail behavior of the driving process. Our results imply that heavier-tailed processes should achieve better generalization; hence, the tail-index of the process can be used as a notion of "capacity metric". We support our theory with experiments on deep neural networks illustrating that the proposed capacity metric accurately estimates the generalization error, and it does not necessarily grow with the number of parameters unlike the existing capacity metrics in the literature.
△ Less
Submitted 22 May, 2021; v1 submitted 16 June, 2020;
originally announced June 2020.
-
Ensemble Rejection Sampling
Authors:
George Deligiannidis,
Arnaud Doucet,
Sylvain Rubenthaler
Abstract:
We introduce Ensemble Rejection Sampling, a scheme for exact simulation from the posterior distribution of the latent states of a class of non-linear non-Gaussian state-space models. Ensemble Rejection Sampling relies on a proposal for the high-dimensional state sequence built using ensembles of state samples. Although this algorithm can be interpreted as a rejection sampling scheme acting on an e…
▽ More
We introduce Ensemble Rejection Sampling, a scheme for exact simulation from the posterior distribution of the latent states of a class of non-linear non-Gaussian state-space models. Ensemble Rejection Sampling relies on a proposal for the high-dimensional state sequence built using ensembles of state samples. Although this algorithm can be interpreted as a rejection sampling scheme acting on an extended space, we show under regularity conditions that the expected computational cost to obtain an exact sample increases cubically with the length of the state sequence instead of exponentially for standard rejection sampling. We demonstrate this methodology by sampling exactly state sequences according to the posterior distribution of a stochastic volatility model and a non-linear autoregressive process. We also present an application to rare event simulation.
△ Less
Submitted 24 January, 2020;
originally announced January 2020.
-
Random walk algorithm for the Dirichlet problem for parabolic integro-differential equation
Authors:
G. Deligiannidis,
S. Maurer,
M. V. Tretyakov
Abstract:
We consider stochastic differential equations driven by a general Lévy processes (SDEs) with infinite activity and the related, via the Feynman-Kac formula, Dirichlet problem for parabolic integro-differential equation (PIDE). We approximate the solution of PIDE using a numerical method for the SDEs. The method is based on three ingredients: (i) we approximate small jumps by a diffusion; (ii) we u…
▽ More
We consider stochastic differential equations driven by a general Lévy processes (SDEs) with infinite activity and the related, via the Feynman-Kac formula, Dirichlet problem for parabolic integro-differential equation (PIDE). We approximate the solution of PIDE using a numerical method for the SDEs. The method is based on three ingredients: (i) we approximate small jumps by a diffusion; (ii) we use restricted jump-adaptive time-step**; and (iii) between the jumps we exploit a weak Euler approximation. We prove weak convergence of the considered algorithm and present an in-depth analysis of how its error and computational cost depend on the jump activity level. Results of some numerical experiments, including pricing of barrier basket currency options, are presented.
△ Less
Submitted 15 January, 2020;
originally announced January 2020.
-
Efficient Irreversible Monte Carlo samplers
Authors:
Fahim Faizi,
George Deligiannidis,
Edina Rosta
Abstract:
We present here two irreversible Markov chain Monte Carlo algorithms for general discrete state systems, one of the algorithms is based on the random-scan Gibbs sampler for discrete states and the other on its improved version, the Metropolized-Gibbs sampler. The algorithms we present incorporate the lifting framework with skewed detailed balance condition and construct irreversible Markov chains…
▽ More
We present here two irreversible Markov chain Monte Carlo algorithms for general discrete state systems, one of the algorithms is based on the random-scan Gibbs sampler for discrete states and the other on its improved version, the Metropolized-Gibbs sampler. The algorithms we present incorporate the lifting framework with skewed detailed balance condition and construct irreversible Markov chains that satisfy the balance condition. We have applied our algorithms to 1D 4-state Potts model. The integrated autocorrelation times for magnetisation and energy density indicate a reduction of the dynamical scaling exponent from $z \approx 1$ to $z \approx 1/2$. In addition, we have generalized an irreversible Metropolis-Hastings algorithm with skewed detailed balance, initially introduced by Turitsyn et al. (2011) for the mean field Ising model, to be now readily applicable to classical spin systems in general; application to 1D 4-state Potts model indicate a square root reduction of the mixing time at high temperatures.
△ Less
Submitted 6 May, 2020; v1 submitted 3 November, 2019;
originally announced November 2019.
-
Relaxing Bijectivity Constraints with Continuously Indexed Normalising Flows
Authors:
Rob Cornish,
Anthony L. Caterini,
George Deligiannidis,
Arnaud Doucet
Abstract:
We show that normalising flows become pathological when used to model targets whose supports have complicated topologies. In this scenario, we prove that a flow must become arbitrarily numerically noninvertible in order to approximate the target closely. This result has implications for all flow-based models, and especially Residual Flows (ResFlows), which explicitly control the Lipschitz constant…
▽ More
We show that normalising flows become pathological when used to model targets whose supports have complicated topologies. In this scenario, we prove that a flow must become arbitrarily numerically noninvertible in order to approximate the target closely. This result has implications for all flow-based models, and especially Residual Flows (ResFlows), which explicitly control the Lipschitz constant of the bijection used. To address this, we propose Continuously Indexed Flows (CIFs), which replace the single bijection used by normalising flows with a continuously indexed family of bijections, and which can intuitively "clean up" mass that would otherwise be misplaced by a single bijection. We show theoretically that CIFs are not subject to the same topological limitations as normalising flows, and obtain better empirical performance on a variety of models and benchmarks.
△ Less
Submitted 23 April, 2021; v1 submitted 30 September, 2019;
originally announced September 2019.
-
Non-Reversible Parallel Tempering: a Scalable Highly Parallel MCMC Scheme
Authors:
Saifuddin Syed,
Alexandre Bouchard-Côté,
George Deligiannidis,
Arnaud Doucet
Abstract:
Parallel tempering (PT) methods are a popular class of Markov chain Monte Carlo schemes used to sample complex high-dimensional probability distributions. They rely on a collection of $N$ interacting auxiliary chains targeting tempered versions of the target distribution to improve the exploration of the state-space. We provide here a new perspective on these highly parallel algorithms and their t…
▽ More
Parallel tempering (PT) methods are a popular class of Markov chain Monte Carlo schemes used to sample complex high-dimensional probability distributions. They rely on a collection of $N$ interacting auxiliary chains targeting tempered versions of the target distribution to improve the exploration of the state-space. We provide here a new perspective on these highly parallel algorithms and their tuning by identifying and formalizing a sharp divide in the behaviour and performance of reversible versus non-reversible PT schemes. We show theoretically and empirically that a class of non-reversible PT methods dominates its reversible counterparts and identify distinct scaling limits for the non-reversible and reversible schemes, the former being a piecewise-deterministic Markov process and the latter a diffusion. These results are exploited to identify the optimal annealing schedule for non-reversible PT and to develop an iterative scheme approximating this schedule. We provide a wide range of numerical examples supporting our theoretical and methodological contributions. The proposed methodology is applicable to sample from a distribution $π$ with a density $L$ with respect to a reference distribution $π_0$ and compute the normalizing constant. A typical use case is when $π_0$ is a prior distribution, $L$ a likelihood function and $π$ the corresponding posterior.
△ Less
Submitted 26 July, 2021; v1 submitted 8 May, 2019;
originally announced May 2019.
-
Bernoulli Race Particle Filters
Authors:
Sebastian M Schmon,
Arnaud Doucet,
George Deligiannidis
Abstract:
When the weights in a particle filter are not available analytically, standard resampling methods cannot be employed. To circumvent this problem state-of-the-art algorithms replace the true weights with non-negative unbiased estimates. This algorithm is still valid but at the cost of higher variance of the resulting filtering estimates in comparison to a particle filter using the true weights. We…
▽ More
When the weights in a particle filter are not available analytically, standard resampling methods cannot be employed. To circumvent this problem state-of-the-art algorithms replace the true weights with non-negative unbiased estimates. This algorithm is still valid but at the cost of higher variance of the resulting filtering estimates in comparison to a particle filter using the true weights. We propose here a novel algorithm that allows for resampling according to the true intractable weights when only an unbiased estimator of the weights is available. We demonstrate our algorithm on several examples.
△ Less
Submitted 3 March, 2019;
originally announced March 2019.
-
Unbiased Smoothing using Particle Independent Metropolis-Hastings
Authors:
Lawrence Middleton,
George Deligiannidis,
Arnaud Doucet,
Pierre E. Jacob
Abstract:
We consider the approximation of expectations with respect to the distribution of a latent Markov process given noisy measurements. This is known as the smoothing problem and is often approached with particle and Markov chain Monte Carlo (MCMC) methods. These methods provide consistent but biased estimators when run for a finite time. We propose a simple way of coupling two MCMC chains built using…
▽ More
We consider the approximation of expectations with respect to the distribution of a latent Markov process given noisy measurements. This is known as the smoothing problem and is often approached with particle and Markov chain Monte Carlo (MCMC) methods. These methods provide consistent but biased estimators when run for a finite time. We propose a simple way of coupling two MCMC chains built using Particle Independent Metropolis-Hastings (PIMH) to produce unbiased smoothing estimators. Unbiased estimators are appealing in the context of parallel computing, and facilitate the construction of confidence intervals. The proposed scheme only requires access to off-the-shelf Particle Filters (PF) and is thus easier to implement than recently proposed unbiased smoothers. The approach is demonstrated on a Lévy-driven stochastic volatility model and a stochastic kinetic model.
△ Less
Submitted 5 February, 2019;
originally announced February 2019.
-
Scalable Metropolis-Hastings for Exact Bayesian Inference with Large Datasets
Authors:
Robert Cornish,
Paul Vanetti,
Alexandre Bouchard-Côté,
George Deligiannidis,
Arnaud Doucet
Abstract:
Bayesian inference via standard Markov Chain Monte Carlo (MCMC) methods is too computationally intensive to handle large datasets, since the cost per step usually scales like $Θ(n)$ in the number of data points $n$. We propose the Scalable Metropolis-Hastings (SMH) kernel that exploits Gaussian concentration of the posterior to require processing on average only $O(1)$ or even $O(1/\sqrt{n})$ data…
▽ More
Bayesian inference via standard Markov Chain Monte Carlo (MCMC) methods is too computationally intensive to handle large datasets, since the cost per step usually scales like $Θ(n)$ in the number of data points $n$. We propose the Scalable Metropolis-Hastings (SMH) kernel that exploits Gaussian concentration of the posterior to require processing on average only $O(1)$ or even $O(1/\sqrt{n})$ data points per step. This scheme is based on a combination of factorized acceptance probabilities, procedures for fast simulation of Bernoulli processes, and control variate ideas. Contrary to many MCMC subsampling schemes such as fixed step-size Stochastic Gradient Langevin Dynamics, our approach is exact insofar as the invariant distribution is the true posterior and not an approximation to it. We characterise the performance of our algorithm theoretically, and give realistic and verifiable conditions under which it is geometrically ergodic. This theory is borne out by empirical results that demonstrate overall performance benefits over standard Metropolis-Hastings and various subsampling algorithms.
△ Less
Submitted 10 June, 2019; v1 submitted 28 January, 2019;
originally announced January 2019.
-
Boundary of the Range of a random walk and the Fölner property
Authors:
George Deligiannidis,
Sebastien Gouezel,
Zemer Kosloff
Abstract:
The range process $R_n$ of a random walk is the collection of sites visited by the random walk up to time $n$. In this work we deal with the question of whether the range process of a random walk or the range process of a cocycle over an ergodic transformation is almost surely a Fölner sequence and show the following results: %
(a) The size of the inner boundary $|\partial R_n|$ of the range of re…
▽ More
The range process $R_n$ of a random walk is the collection of sites visited by the random walk up to time $n$. In this work we deal with the question of whether the range process of a random walk or the range process of a cocycle over an ergodic transformation is almost surely a Fölner sequence and show the following results: %
(a) The size of the inner boundary $|\partial R_n|$ of the range of recurrent aperiodic random walks on $\mathbb{Z}^2$ with finite variance and aperiodic random walks in $\mathbb{Z}$ in the standard domain of attraction of the Cauchy distribution, divided by $\frac{n}{\log^2(n)}$, converges to a constant almost surely. %
(b) We establish a formula for the Fölner asymptotic of transient cocycles over an ergodic probability preserving transformation and use it to show that for transient random walk on groups which are not virtually cyclic, for almost every path, the range is not a Fölner sequence. %
(c) For aperiodic random walks in the domain of attraction of symmetric $α$- stable distributions with $1<α\leq 2$, we prove a sharp polynomial upper bound for the decay at infinity of $|\partial R_n|/|R_n|$. This last result shows that the range process of these random walks is almost surely a Fölner sequence.
△ Less
Submitted 18 December, 2018; v1 submitted 24 October, 2018;
originally announced October 2018.
-
Randomized Hamiltonian Monte Carlo as Scaling Limit of the Bouncy Particle Sampler and Dimension-Free Convergence Rates
Authors:
George Deligiannidis,
Daniel Paulin,
Alexandre Bouchard-Côté,
Arnaud Doucet
Abstract:
The Bouncy Particle Sampler is a Markov chain Monte Carlo method based on a nonreversible piecewise deterministic Markov process. In this scheme, a particle explores the state space of interest by evolving according to a linear dynamics which is altered by bouncing on the hyperplane tangent to the gradient of the negative log-target density at the arrival times of an inhomogeneous Poisson Process…
▽ More
The Bouncy Particle Sampler is a Markov chain Monte Carlo method based on a nonreversible piecewise deterministic Markov process. In this scheme, a particle explores the state space of interest by evolving according to a linear dynamics which is altered by bouncing on the hyperplane tangent to the gradient of the negative log-target density at the arrival times of an inhomogeneous Poisson Process (PP) and by randomly perturbing its velocity at the arrival times of an homogeneous PP. Under regularity conditions, we show here that the process corresponding to the first component of the particle and its corresponding velocity converges weakly towards a Randomized Hamiltonian Monte Carlo (RHMC) process as the dimension of the ambient space goes to infinity. RHMC is another piecewise deterministic non-reversible Markov process where a Hamiltonian dynamics is altered at the arrival times of a homogeneous PP by randomly perturbing the momentum component. We then establish dimension-free convergence rates for RHMC for strongly log-concave targets with bounded Hessians using coupling ideas and hypocoercivity techniques.
△ Less
Submitted 23 December, 2020; v1 submitted 13 August, 2018;
originally announced August 2018.
-
Unbiased Markov chain Monte Carlo for intractable target distributions
Authors:
Lawrence Middleton,
George Deligiannidis,
Arnaud Doucet,
Pierre E. Jacob
Abstract:
Performing numerical integration when the integrand itself cannot be evaluated point-wise is a challenging task that arises in statistical analysis, notably in Bayesian inference for models with intractable likelihood functions. Markov chain Monte Carlo (MCMC) algorithms have been proposed for this setting, such as the pseudo-marginal method for latent variable models and the exchange algorithm fo…
▽ More
Performing numerical integration when the integrand itself cannot be evaluated point-wise is a challenging task that arises in statistical analysis, notably in Bayesian inference for models with intractable likelihood functions. Markov chain Monte Carlo (MCMC) algorithms have been proposed for this setting, such as the pseudo-marginal method for latent variable models and the exchange algorithm for a class of undirected graphical models. As with any MCMC algorithm, the resulting estimators are justified asymptotically in the limit of the number of iterations, but exhibit a bias for any fixed number of iterations due to the Markov chains starting outside of stationarity. This "burn-in" bias is known to complicate the use of parallel processors for MCMC computations. We show how to use coupling techniques to generate unbiased estimators in finite time, building on recent advances for generic MCMC algorithms. We establish the theoretical validity of some of these procedures by extending existing results to cover the case of polynomially ergodic Markov chains. The efficiency of the proposed estimators is compared with that of standard MCMC estimators, with theoretical arguments and numerical experiments including state space models and Ising models.
△ Less
Submitted 15 June, 2020; v1 submitted 23 July, 2018;
originally announced July 2018.
-
Large Sample Asymptotics of the Pseudo-Marginal Method
Authors:
Sebastian M. Schmon,
George Deligiannidis,
Arnaud Doucet,
Michael K. Pitt
Abstract:
The pseudo-marginal algorithm is a variant of the Metropolis--Hastings algorithm which samples asymptotically from a probability distribution when it is only possible to estimate unbiasedly an unnormalized version of its density. Practically, one has to trade-off the computational resources used to obtain this estimator against the asymptotic variances of the ergodic averages obtained by the pseud…
▽ More
The pseudo-marginal algorithm is a variant of the Metropolis--Hastings algorithm which samples asymptotically from a probability distribution when it is only possible to estimate unbiasedly an unnormalized version of its density. Practically, one has to trade-off the computational resources used to obtain this estimator against the asymptotic variances of the ergodic averages obtained by the pseudo-marginal algorithm. Recent works optimizing this trade-off rely on some strong assumptions which can cast doubts over their practical relevance. In particular, they all assume that the distribution of the difference between the log-density and its estimate is independent of the parameter value at which it is evaluated. Under regularity conditions we show here that, as the number of data points tends to infinity, a space-rescaled version of the pseudo-marginal chain converges weakly towards another pseudo-marginal chain for which this assumption indeed holds. A study of this limiting chain allows us to provide parameter dimension-dependent guidelines on how to optimally scale a normal random walk proposal and the number of Monte Carlo samples for the pseudo-marginal method in the large-sample regime. This complements and validates currently available results.
△ Less
Submitted 2 December, 2019; v1 submitted 26 June, 2018;
originally announced June 2018.
-
Controlled Sequential Monte Carlo
Authors:
Jeremy Heng,
Adrian N. Bishop,
George Deligiannidis,
Arnaud Doucet
Abstract:
Sequential Monte Carlo methods, also known as particle methods, are a popular set of techniques for approximating high-dimensional probability distributions and their normalizing constants. These methods have found numerous applications in statistics and related fields; e.g. for inference in non-linear non-Gaussian state space models, and in complex static models. Like many Monte Carlo sampling sc…
▽ More
Sequential Monte Carlo methods, also known as particle methods, are a popular set of techniques for approximating high-dimensional probability distributions and their normalizing constants. These methods have found numerous applications in statistics and related fields; e.g. for inference in non-linear non-Gaussian state space models, and in complex static models. Like many Monte Carlo sampling schemes, they rely on proposal distributions which crucially impact their performance. We introduce here a class of controlled sequential Monte Carlo algorithms, where the proposal distributions are determined by approximating the solution to an associated optimal control problem using an iterative scheme. This method builds upon a number of existing algorithms in econometrics, physics, and statistics for inference in state space models, and generalizes these methods so as to accommodate complex static models. We provide a theoretical analysis concerning the fluctuation and stability of this methodology that also provides insight into the properties of related algorithms. We demonstrate significant gains over state-of-the-art methods at a fixed computational complexity on a variety of applications.
△ Less
Submitted 30 October, 2019; v1 submitted 28 August, 2017;
originally announced August 2017.
-
Piecewise-Deterministic Markov Chain Monte Carlo
Authors:
Paul Vanetti,
Alexandre Bouchard-Côté,
George Deligiannidis,
Arnaud Doucet
Abstract:
A novel class of non-reversible Markov chain Monte Carlo schemes relying on continuous-time piecewise-deterministic Markov Processes has recently emerged. In these algorithms, the state of the Markov process evolves according to a deterministic dynamics which is modified using a Markov transition kernel at random event times. These methods enjoy remarkable features including the ability to update…
▽ More
A novel class of non-reversible Markov chain Monte Carlo schemes relying on continuous-time piecewise-deterministic Markov Processes has recently emerged. In these algorithms, the state of the Markov process evolves according to a deterministic dynamics which is modified using a Markov transition kernel at random event times. These methods enjoy remarkable features including the ability to update only a subset of the state components while other components implicitly keep evolving and the ability to use an unbiased estimate of the gradient of the log-target while preserving the target as invariant distribution. However, they also suffer from important limitations. The deterministic dynamics used so far do not exploit the structure of the target. Moreover, exact simulation of the event times is feasible for an important yet restricted class of problems and, even when it is, it is application specific. This limits the applicability of these techniques and prevents the development of a generic software implementation of them. We introduce novel MCMC methods addressing these shortcomings. In particular, we introduce novel continuous-time algorithms relying on exact Hamiltonian flows and novel non-reversible discrete-time algorithms which can exploit complex dynamics such as approximate Hamiltonian dynamics arising from symplectic integrators while preserving the attractive features of continuous-time algorithms. We demonstrate the performance of these schemes on a variety of applications.
△ Less
Submitted 15 May, 2018; v1 submitted 17 July, 2017;
originally announced July 2017.
-
Exponential Ergodicity of the Bouncy Particle Sampler
Authors:
George Deligiannidis,
Alexandre Bouchard-Côté,
Arnaud Doucet
Abstract:
Non-reversible Markov chain Monte Carlo schemes based on piecewise deterministic Markov processes have been recently introduced in applied probability, automatic control, physics and statistics. Although these algorithms demonstrate experimentally good performance and are accordingly increasingly used in a wide range of applications, geometric ergodicity results for such schemes have only been est…
▽ More
Non-reversible Markov chain Monte Carlo schemes based on piecewise deterministic Markov processes have been recently introduced in applied probability, automatic control, physics and statistics. Although these algorithms demonstrate experimentally good performance and are accordingly increasingly used in a wide range of applications, geometric ergodicity results for such schemes have only been established so far under very restrictive assumptions. We give here verifiable conditions on the target distribution under which the Bouncy Particle Sampler algorithm introduced in \cite{P_dW_12} is geometrically ergodic. This holds whenever the target satisfies a curvature condition and has tails decaying at least as fast as an exponential and at most as fast as a Gaussian distribution. This allows us to provide a central limit theorem for the associated ergodic averages. When the target has tails thinner than a Gaussian distribution, we propose an original modification of this scheme that is geometrically ergodic. For thick-tailed target distributions, such as $t$-distributions, we extend the idea pioneered in \cite{J_G_12} in a random walk Metropolis context. We apply a change of variable to obtain a transformed target satisfying the tail conditions for geometric ergodicity. By sampling the transformed target using the Bouncy Particle Sampler and map** back the Markov process to the original parameterization, we obtain a geometrically ergodic algorithm.
△ Less
Submitted 27 August, 2017; v1 submitted 12 May, 2017;
originally announced May 2017.
-
Which ergodic averages have finite asymptotic variance?
Authors:
George Deligiannidis,
Anthony Lee
Abstract:
We show that the class of $L^2$ functions for which ergodic averages of a reversible Markov chain have finite asymptotic variance is determined by the class of $L^2$ functions for which ergodic averages of its associated jump chain have finite asymptotic variance. This allows us to characterize completely which ergodic averages have finite asymptotic variance when the Markov chain is an independen…
▽ More
We show that the class of $L^2$ functions for which ergodic averages of a reversible Markov chain have finite asymptotic variance is determined by the class of $L^2$ functions for which ergodic averages of its associated jump chain have finite asymptotic variance. This allows us to characterize completely which ergodic averages have finite asymptotic variance when the Markov chain is an independence sampler. In addition, we obtain a simple sufficient condition for all ergodic averages of $L^2$ functions of the primary variable in a pseudo-marginal Markov chain to have finite asymptotic variance.
△ Less
Submitted 23 July, 2016; v1 submitted 27 June, 2016;
originally announced June 2016.
-
The Correlated Pseudo-Marginal Method
Authors:
George Deligiannidis,
Arnaud Doucet,
Michael K. Pitt
Abstract:
The pseudo-marginal algorithm is a popular variant of the Metropolis--Hastings scheme which allows us to sample asymptotically from a target probability density $π$, when we are only able to estimate an unnormalized version of $π$ pointwise unbiasedly. It has found numerous applications in Bayesian statistics as there are many scenarios where the likelihood function is intractable but can be estim…
▽ More
The pseudo-marginal algorithm is a popular variant of the Metropolis--Hastings scheme which allows us to sample asymptotically from a target probability density $π$, when we are only able to estimate an unnormalized version of $π$ pointwise unbiasedly. It has found numerous applications in Bayesian statistics as there are many scenarios where the likelihood function is intractable but can be estimated unbiasedly using Monte Carlo samples. Using many samples will typically result in averages computed under this chain with lower asymptotic variances than the corresponding averages that use fewer samples. For a fixed computing time, it has been shown in several recent contributions that an efficient implementation of the pseudo-marginal method requires the variance of the log-likelihood ratio estimator appearing in the acceptance probability of the algorithm to be of order 1, which in turn usually requires scaling the number $N$ of Monte Carlo samples linearly with the number $T$ of data points. We propose a modification of the pseudo-marginal algorithm, termed the correlated pseudo-marginal algorithm, which is based on a novel log-likelihood ratio estimator computed using the difference of two positively correlated log-likelihood estimators. We show that the parameters of this scheme can be selected such that the variance of this estimator is order $1$ as $N,T\rightarrow\infty$ whenever $N/T\rightarrow 0$. By combining these results with the Bernstein-von Mises theorem, we provide an analysis of the performance of the correlated pseudo-marginal algorithm in the large $T$ regime. In our numerical examples, the efficiency of computations is increased relative to the standard pseudo-marginal algorithm by more than 20 fold for values of $T$ of a few hundreds to more than 100 fold for values of $T$ of around 10,000-20,000.
△ Less
Submitted 19 July, 2017; v1 submitted 16 November, 2015;
originally announced November 2015.
-
Relative Complexity of Random Walks in Random Scenery in the absence of a weak invariance principle for the local times
Authors:
George Deligiannidis,
Zemer Kosloff
Abstract:
We answer the question of Aaronson about the relative complexity of Random Walks in Random Sceneries driven by either aperiodic two dimensional random walks, two-dimensional Simple Random walk, or by aperiodic random walks in the domain of attraction of the Cauchy distribution. A key step is proving that the range of the random walk satisfies the Fölner property almost surely.
We answer the question of Aaronson about the relative complexity of Random Walks in Random Sceneries driven by either aperiodic two dimensional random walks, two-dimensional Simple Random walk, or by aperiodic random walks in the domain of attraction of the Cauchy distribution. A key step is proving that the range of the random walk satisfies the Fölner property almost surely.
△ Less
Submitted 1 June, 2015;
originally announced June 2015.
-
Optimal bounds for self-intersection local times
Authors:
George Deligiannidis,
Sergey Utev
Abstract:
For a random walk $S_n, n\geq 0$ in $\mathbb{Z}^d$, let $l(n,x)$ be its local time at the site $x\in \mathbb{Z}^d$. Define the $α$-fold self intersection local time $L_n(α) := \sum_{x} l(n,x)^α$, and let $L_n(α|ε, d)$ the corresponding quantity for $d$-dimensional simple random walk. Without imposing any moment conditions, we show that the variances of the local times $\mathop{var}(L_n(α))$ of any…
▽ More
For a random walk $S_n, n\geq 0$ in $\mathbb{Z}^d$, let $l(n,x)$ be its local time at the site $x\in \mathbb{Z}^d$. Define the $α$-fold self intersection local time $L_n(α) := \sum_{x} l(n,x)^α$, and let $L_n(α|ε, d)$ the corresponding quantity for $d$-dimensional simple random walk. Without imposing any moment conditions, we show that the variances of the local times $\mathop{var}(L_n(α))$ of any genuinely $d$-dimensional random walk are bounded above by the corresponding characteristics of the simple symmetric random walk in $\mathbb{Z}^d$, i.e. $\mathop{var}(L_n(α)) \leq C \mathop{var}[L_n(α|ε, d)]\sim K_{d,α}v_{d,α}(n)$. In particular, variances of local times of all genuinely $d$-dimensional random walks, $d\geq 4$, are similar to the $4$-dimensional symmetric case $\mathop{var}(L_n(α)) = O(n)$. On the other hand, in dimensions $d\leq 3$ the resemblance to the simple random walk $\liminf_{n\to \infty} \mathop{var}(L_n(α))/v_{d,α}(n)>0$ implies that the jumps must have zero mean and finite second moment.
△ Less
Submitted 3 June, 2015; v1 submitted 29 May, 2015;
originally announced May 2015.
-
Asymptotic variance of stationary reversible and normal Markov processes
Authors:
George Deligiannidis,
Magda Peligrad,
Sergey Utev
Abstract:
We obtain necessary and sufficient conditions for the regular variation of the variance of partial sums of functionals of discrete and continuous-time stationary Markov processes with normal transition operators. We also construct a class of Metropolis-Hastings algorithms which satisfy a central limit theorem and invariance principle when the variance is not linear in $n$.
We obtain necessary and sufficient conditions for the regular variation of the variance of partial sums of functionals of discrete and continuous-time stationary Markov processes with normal transition operators. We also construct a class of Metropolis-Hastings algorithms which satisfy a central limit theorem and invariance principle when the variance is not linear in $n$.
△ Less
Submitted 10 May, 2014;
originally announced May 2014.
-
Efficient implementation of Markov chain Monte Carlo when using an unbiased likelihood estimator
Authors:
Arnaud Doucet,
Michael Pitt,
George Deligiannidis,
Robert Kohn
Abstract:
When an unbiased estimator of the likelihood is used within a Metropolis--Hastings chain, it is necessary to trade off the number of Monte Carlo samples used to construct this estimator against the asymptotic variances of averages computed under this chain. Many Monte Carlo samples will typically result in Metropolis--Hastings averages with lower asymptotic variances than the corresponding Metropo…
▽ More
When an unbiased estimator of the likelihood is used within a Metropolis--Hastings chain, it is necessary to trade off the number of Monte Carlo samples used to construct this estimator against the asymptotic variances of averages computed under this chain. Many Monte Carlo samples will typically result in Metropolis--Hastings averages with lower asymptotic variances than the corresponding Metropolis--Hastings averages using fewer samples. However, the computing time required to construct the likelihood estimator increases with the number of Monte Carlo samples. Under the assumption that the distribution of the additive noise introduced by the log-likelihood estimator is Gaussian with variance inversely proportional to the number of Monte Carlo samples and independent of the parameter value at which it is evaluated, we provide guidelines on the number of samples to select. We demonstrate our results by considering a stochastic volatility model applied to stock index returns.
△ Less
Submitted 13 September, 2014; v1 submitted 5 October, 2012;
originally announced October 2012.
-
Variance of partial sums of stationary sequences
Authors:
George Deligiannidis,
Sergey Utev
Abstract:
Let $X_1,X_2,\ldots$ be a centred sequence of weakly stationary random variables with spectral measure $F$ and partial sums $S_n=X_1+\cdots+X_n$. We show that $\operatorname {var}(S_n)$ is regularly varying of index $γ$ at infinity, if and only if $G(x):=\int_{-x}^xF(\mathrm {d}x)$ is regularly varying of index $2-γ$ at the origin ($0<γ<2$).
Let $X_1,X_2,\ldots$ be a centred sequence of weakly stationary random variables with spectral measure $F$ and partial sums $S_n=X_1+\cdots+X_n$. We show that $\operatorname {var}(S_n)$ is regularly varying of index $γ$ at infinity, if and only if $G(x):=\int_{-x}^xF(\mathrm {d}x)$ is regularly varying of index $2-γ$ at the origin ($0<γ<2$).
△ Less
Submitted 21 October, 2013; v1 submitted 18 May, 2012;
originally announced May 2012.
-
An asymptotic variance of the self-intersections of random walks
Authors:
George Deligiannidis,
Sergey Utev
Abstract:
We present a Darboux-Wiener type lemma and apply it to obtain an exact asymptotic for the variance of the self-intersection of one and two-dimensional random walks. As a corollary, we obtain a central limit theorem for random walk in random scenery conjectured by Kesten and Spitzer in 1979.
We present a Darboux-Wiener type lemma and apply it to obtain an exact asymptotic for the variance of the self-intersection of one and two-dimensional random walks. As a corollary, we obtain a central limit theorem for random walk in random scenery conjectured by Kesten and Spitzer in 1979.
△ Less
Submitted 27 April, 2010;
originally announced April 2010.
