Can independent Metropolis beat crude Monte Carlo?
Abstract
Assume that we would like to estimate the expected value of a function with respect to a density . We prove that if is close enough under KL divergence to another density , an independent Metropolis sampler estimator that obtains samples from with proposal density , enriched with a variance reduction computational strategy based on control variates, achieves smaller asymptotic variance than that of the crude Monte Carlo estimator. The control variates construction requires no extra computational effort but assumes that the expected value of under is analytically available. We illustrate this result by calculating the marginal likelihood in a linear regression model with prior-likelihood conflict and a non-conjugate prior. Furthermore, we propose an adaptive independent Metropolis algorithm that adapts the proposal density such that its KL divergence with the target is being reduced. We demonstrate its applicability in a Bayesian logistic and Gaussian process regression problems and we rigorously justify our asymptotic arguments under easily verifiable and essentially minimal conditions.
keywords:
Markov chain Monte Carlo, independent Metropolis, variance reduction, control variate, Poisson equation.1 Introduction
1.1 The general problem
We address the following generic Monte Carlo estimation problem. We are given a probability density function on some state space which gives rise to a probability measure on . We want to estimate the expectation of a function with respect to given by
(1) |
and we assume that analytical integration of (1) is not feasible. The crude Monte Carlo estimation of this expectation simulates i.i.d. random variables and then produces the unbiased estimator
(2) |
with variance estimator of order . The true variance of is thus given by
(3) |
In some challenging problems often met in Bayesian inference, high energy physics and rare event simulation in finance and insurance, the variance is very large and variance reduction techniques are used to improve , see for example Owen (2013). One such method is importance sampling, in which we sample i.i.d. from some other probability density (importance function) on with respect to a measure and then use the (biased) estimator
(4) |
which can achieve a variance smaller than obtaining its smallest value when for all , see Kahn and Marshall (1953). Another one is the use of control variates: if there exist functions (control variates) such that are analytically available, then the estimator is unbiased and for appropriate values of the scalars has no larger variance than that of .
In many problems, a good choice of an importance function will not be available to estimate (1). This scenario is not rare when is high-dimensional, not only because it may be hard to both sample from and mimic the shape of , but also because the variance of (4) might not exist when has thinner tails than . The use of control variates might also be infeasible since when is a non-standard density, analytical expressions for are rarely available. Motivated by this, we aim to introduce alternative Markov chain Monte Carlo (MCMC) estimators of (1), based on the independent Metropolis algorithm, that can have smaller asymptotic variance than . At the same time these estimators should be easy to use without requiring to compute intractable expectations of control variates under the target.
An independent Metropolis estimator can be derived by constructing a Markov chain on which has as a stationary distribution by letting a density be the transition function, or proposal density, of the chain. The dynamics of the independent Metropolis algorithm require to move from any state by proposing a new location from and accept it with probability
(5) |
By collecting, at stationarity, dependent samples from the Markov chain, we construct the ergodic estimator
(6) |
which satisfies, under appropriate conditions (see section 2), a central limit theorem of the form
with the asymptotic variance given by
(7) |
1.2 Outline of the basic methodology
At first sight the estimator cannot compete with because it is based on dependent samples and therefore we expect that its variance should be larger. Indeed only when equals . Our key idea is that we can exploit the Markovian structure of the MCMC samples to build, with negligible extra computational cost, new estimators with variance smaller than . We derive these new estimators based on approximate solutions to the Poisson equation of the Markov chain that allow the construction of control variates for . There are two requirements for these estimators to achieve smaller variance than . The first is the ability to obtain analytically or for some function close to . The second is that the proposal density is close to . This is rigorously justified in our Theorem 2 which states that when is close enough to under KL-divergence, then the variance of the new estimator is smaller than . This gives a new perspective for Monte Carlo estimation of (1) using control variates that does not require the ability to obtain an importance density close to or to analytically evaluate for control variates . Unlike importance sampling where the optimal depends on in our estimator the optimal depends only on , and this implies that with no extra cost we can re-use the same MCMC samples to obtain variance reduction for many functions .
In the first part of our work, we derive the new ergodic estimator for independent Metropolis algorithms for any real valued functions together with a theoretical result that proves that such estimator is a good generic candidate for estimating expectations if the proposal density is close enough to the target density. To illustrate the potential of this new estimator, we visit a challenging and popular problem in Bayesian statistics that refers to calculation of marginal densities of the form where the likelihood and prior are non-conjugate and there is a likelihood-prior conflict. We show that our ergodic estimator can have significantly reduced variance compared to crude Monte Carlo. We further combine this estimator of the marginal likelihood with pseudo-marginal MCMC (Andrieu and Roberts, 2009) to sample over the model space without the need to obtain parameter estimates within each model.
In the second part, we build an adaptive independent Metropolis algorithm that updates every iterations the proposal density such that the KL-divergence is being reduced after every adaptation. This update is based on a recent strand of research that exploits stochastic gradient-based optimisation techniques for KL-divergence minimisation in variational inference (Ranganath et al., 2014) and adaptive MCMC (Gabrié et al., 2022). We provide rigorously justified asymptotic arguments for (i) ergodicity of the resulting Markov chain, (ii) convergence to distribution of the estimators, (iii) a weak law of large numbers when both and the MCMC sampling size go to infinity and (iv) convergence of the scalar estimators that are used to construct our control variates. These arguments show that our adaptive independent Metropolis algorithm, enriched with cost-free variance reduction ergodic estimator, can serve as a direct competitor to since it achieves smaller variance when approaches close enough, and can be a direct competitor to the importance sampling estimator when a good proposal density is not available.
To experimentally evaluate the estimator based on the adaptive independent Metropolis scheme, we considered Bayesian inference problems in logistic regression and Gaussian process regression. For several functions of interest , including non-linear functions of parameters such as the odds ratio, the variance reduction we achieved by adopting our estimator compared with the simple ergodic estimator of the adaptive independent Metropolis varied between and .
1.3 Related work
There is a vast literature of variance reduction in MCMC via control variates; for a long list of references and some critical discussion see Alexopoulos et al. (2023). A strand of research is based on Assaraf and Caffarel (1999) who noticed that a Hamiltonian operator together with a trial function are sufficient to construct an estimator with zero asymptotic variance. They considered a Hamiltonian operator of Schrödinger-type that led to a series of zero-variance estimators studied by Valle and Leisen (2010), Mira et al. (2013), Papamarkou et al. (2014), Belomestny et al. (2020), South et al. (2023), Oates et al. (2017), Barp et al. (2022), South et al. (2022), Oates et al. (2019). Another approach which is closely related to our proposed methodology attempts to minimise the asymptotic variance of the ergodic estimator of the Markov chain. One way to achieve this is based on the observation that the solution of the Poisson equation for (also called the fundamental equation) automatically leads to a zero-variance estimator for . Interestingly, a solution of this equation produces zero-variance estimators suggested by Assaraf and Caffarel (1999) for a specific choice of Hamiltonian operator. One of the rare examples that the Poisson equation has been solved exactly for discrete time Markov chains is the random scan Gibbs sampler where the target density is a multivariate Gaussian density and the goal is to estimate the mean of each components of the target density, see Dellaportas and Kontoyiannis (2012, 2009). Since direct solution of the Poisson equation is not available, approximate solutions have been suggested by Andradóttir et al. (1993), Atchadé and Perron (2005), Henderson (1997), Meyn (2008). Recently, Alexopoulos et al. (2023) provided a general framework that constructs estimators with reduced variance for random walk Metropolis and Metropolis-adjusted Langevin algorithms for means of each co-ordinate of a -dimensional target density.
Adaptive Markov chain Monte Carlo algorithms with certain ergodic properties were proposed by Haario et al. (2001) and Roberts and Rosenthal (2009). An adaptation scheme of independent Metropolis together with its convergence properties is provided by Holden et al. (2009). Andrieu and Moulines (2006) and Roberts and Rosenthal (2007) studied the ergodicity properties of adaptive MCMC algorithms and provided central limit theorems and weak laws of large numbers. Recently, Gabrié et al. (2022) and Brofos et al. (2022) suggest applying adaptive MCMC by adopting normalizing flows as independent Metropolis proposal densities, which are optimised by KL-divergence minimisation using the history of states. In our adaptive MCMC algorithm we also use KL-divergence minimisation to optimise the proposal density but unlike Gabrié et al. (2022); Brofos et al. (2022) we make use of all candidate states (rejected or accepted) to adapt . Specifically, we apply stochastic gradient methods based on the reprametrisation trick that are widely used for variational inference in machine learning (Titsias and Lázaro-Gredilla, 2014; Rezende et al., 2014; Kingma and Welling, 2013; Gianniotis et al., 2016; Kucukelbir et al., 2017; Salimans and Knowles, 2013; Tan and Nott, 2018).
Finally, a parallel path of research is based on adaptive importance sampling, see for example Richard and Zhang (2007); Cappé et al. (2008); Cornebise et al. (2008); Dellaportas and Tsionas (2019); Paananen et al. (2021). The goal is to target rather than and the adaptation process is somewhat different because it does not involve any Markov chain dynamics.
1.4 Outline of the paper
The rest of the paper is organised as follows. Section provides the methodological arguments and their rigorous justification for the construction of control variables for the ergodic estimator of the independent Metropolis algorithm together with illustrations with simulated and real data. Section deals with the construction of an adaptive independent Metropolis algorithm that adapts the proposal density such that its KL-divergence with the target is being reduced and illustrates its applicability with synthetic and real data. We conclude with a brief discussion in Section .
2 Control variates for Independent Metropolis algorithms
2.1 A new estimator based on control variates
We denote by the independent Metropolis algorithm defined on the state space with transition kernel defined via (5) and two probability measures and with corresponding target and proposal densities and . The transition kernel that generates the Markov chain based on is expressed by
where is Dirac’s measure centred at . Then, for any function we have
(8) |
Henderson (1997) observed that since the function has zero expectation with respect to , we can use it as control variate to reduce the variance of by constructing, using (8), the control variate based estimator which is based on samples and is given by
(9) |
for some scalar parameter . Since is reversible, the optimal that minimises the asymptotic variance in (7) is given by
(10) |
estimated by
(11) |
see Theorem 3 of Dellaportas and Kontoyiannis (2012). The optimum choice of is the solution of Poisson equation for :
(12) |
that achieves zero variance for the estimator for any value of and for , see Dellaportas and Kontoyiannis (2012). Clearly the solution of (12) is translation invariant. Therefore, in the following we will denote by the any member of the class of all translations of .
A research strategy that was used by Dellaportas and Kontoyiannis (2012) for random scan Gibbs samplers and by Alexopoulos et al. (2023) for random walk and Metropolis adjusted Langevin algorithms, is to find an estimator based on some approximation . We follow the same research avenue for by first proving the following Theorem:
Theorem 1
(Proof in the Appendix A.1) Assume that for a target density there exists a sequence of proposal distributions such that and for each proposal distribution the corresponding solution of the Poisson equation of for a function is . Then,
Theorem 1 guarantees that as the proposal density converges to the target density, the solution to the Poisson equation converges to the function . It is straightforward to observe that at the limit the acceptance ratio in (12) becomes one and the solution of the Poisson equation is just . Motivated by this limiting case, for any independent Metropolis sampler (where ) we propose to use the function in the general control variate based estimator in (9), which leads to the estimator
(13) |
Unlike the approach of Alexopoulos et al. (2023), the above estimator does not require any approximation of function because it simply assumes that is a good approximation to . However, (13) still requires the evaluation of the intractable integral for each sampled point . Since the transition kernel of the algorithm required one sample every time the chain was at state , a crude Monte Carlo estimator of this integral can be based on the (sample size one!) estimator , where all acceptance ratios are available at no additional cost. This suggests that (13) can be approximated by
(14) |
A final trick to achieve our final estimator is to use a further control variate to reduce the variance of the estimator (14), by reducing the variance of the sample size one estimate of the integral . If the expected value is analytically available then an immediate choice of control variate is the zero-mean random variable resulting to our proposed final estimator
(15) |
where is calculated from (11) with . Notice that depends on both the current value of the chain and the corresponding proposal value , so we will denote by and we obtain
The scalar is the least squares estimator that minimises the asymptotic variance of the estimator
so the optimal parameter is given by
(16) |
estimated by
(17) |
see, for example, Owen (2013). Notice that when , from (10) and (16) the regression coefficients are set to and the acceptance probability is , so we obtain the zero-variance estimator .
2.2 Connection to Rao–Blackwellisation by integrating out decision step
Another perspective of our proposed estimator is that it connects with and improves upon a basic Rao-Blackwellised estimator that integrates out the decision step at each Metropolis–Hastings iteration; see for instance Section 6.1 in Robert et al. (2018). Given that is the current state, is the proposal and is the uniform random variable to accept or reject , the IM estimator (or any other MCMC estimator) can be written as111Note that we apply some re-indexing to re-write as so that the counting of the samples starts at .
where we used that the next state is equal to either when the indicator function or otherwise. By taking the expectation under the uniform distribution to integrate out we obtain
which is the same with (14) for . Therefore, an interpretation of our final proposed estimator in (15) (when ) is that it adds a control variate to further reduce the variance associated with the proposed state in the above Rao–Blackwellised estimator.
2.3 Theoretical justifications
The asymptotic variance of is given by
(18) |
and the question that arises is when is smaller than . We define the space
where and are the corresponding densities of , and where . We state the following assumptions.
Assumption 1 (A1)
.
Assumption 2 (A2)
.
Assumption 3 (A3)
The Markov chain produced by the sampler is -irreducible and aperiodic with unique invariant measure , and there are functions , a small set , and a finite constant , such that the Lyapunov condition holds and .
Assumptions (A1), (A2) and (A3), guarantee standard theoretical MCMC results as shown in the following proposition.
Proposition 1
We now state our key result. By denoting the Kullback–Leibler(KL) divergence as
we provide a general result that connects in (18) with in (3) and in (7):
Theorem 2
Theorem 2 states that under fairly general assumptions of the functions for which we need to calculate , we can construct an sampler so that the ergodic estimator (15) achieves smaller variance of the corresponding crude Monte Carlo estimators (2) and the standard ergodic estimator of (6). This immediately suggests that a new methodological strategy to estimate with Monte Carlo is to use an sampler with a suitably chosen proposal density that has two properties: it approximates well and is analytically available. A plausible strategy is to approximate by mixture of normal or student-t densities which seems a much easier task than the standard importance sampling strategy which seeks to approximate in (4) with mixture of densities that approximate ; see, for example, Cappé et al. (2008). This strategy has another advantage when compared with importance sampling. When is a mixture of normal or student-t densities, expectations of is available for many functions . It is then straightforward to construct many estimators for many functions using the same sample from the chain. This is in contrast with the importance sampling methods that require a new importance function for every new function . An example of such flexibility is shown in the logistic regression examples of Section 3.3.2 where reduced variance estimators based on the same MCMC sample are derived for the first two moments of the parameters and the posterior odds.
2.4 Numerical illustrations
In subsection 2.4.1 we illustrate Theorem 2 in simple synthetic data examples while in subsection 2.4.2 we consider a more challenging Bayesian model selection example. We examine the relative performance of our estimator against the crude Monte Carlo estimator . Our measure of performance is based on the variance reduction factor which is defined as the ratio of . In our experiments we fix and produce independent repetitions and that are used to estimate a truncated version of the variance reduction factor by
(19) |
2.4.1 Synthetic data examples
We first construct a synthetic data example in which we assume that the target distribution is a standard normal distribution and the quantity of interest is the expected value of . We choose two different proposals , a normal distribution and a zero-mean student-t distribution with degrees of freedom . Independent Metropolis algorithms were initialised when a point drawn from the proposal distribution is accepted. independent repetitions of size samples were used to obtain the estimators and that were used to construct the box-plots of Figures 1 and 1. It is evident that as the proposal densities approach the target, the sample variance of the estimators decreases. The corresponding VRFs are depicted (shown in log space) in Figures 1 and 1. Notice that when the proposal density is far away from the target, has larger variance than that of . Furthermore, Figures 1 and 1 depict the theoretical lower bounds for the variance reduction factor (19) which are used in the proof of Theorem 2 in Appendix A.3. For these particular examples these bounds can be analytically derived, see (• ‣ B.1) and (• ‣ B.1) in Appendix B.1.
2.4.2 Bayesian model selection in non-conjugate linear regression
We consider a standard linear regression where is a column vector of observations, is an design matrix, is a column vector of regression coefficients and is an error vector distributed as . The Bayesian approach to model selection requires the calculation of posterior model probabilities for all models specified by all , the parameter vectors with elements the subsets of the set of elements of , and the corresponding design matrices . In turn, this requires a specification of prior model probabilities and the calculation of the marginal likelihood of model which is given by
(20) |
where and are the likelihood functions and parameter prior densities of model . It is well-known that the choice of the prior is of paramount importance in calculating posterior model probabilities and a careful specification is both necessary and an issue that has attracted a lot of interest in the literature; see, for example, Dellaportas et al. (2012). Perhaps the most popular and widely accepted default choice is the mixture of g-priors of Liang et al. (2008). An example of these priors that has been suggested by Liang et al. (2008) is given by
(21) |
This and all other choices of mixtures of -priors result in making integration of (20) intractable. This is a typical situation in Bayesian model choice literature where huge efforts have been made to propose ways to approximate such intractable marginal likelihoods.
It has often been noted (see for example Newton and Raftery (1994)) that an obvious solution would be to adopt a crude Monte Carlo estimator by sampling from the prior and estimating (20) as the expected likelihood function. However, since the posterior is often concentrated relative to the prior, Monte Carlo estimators will be very inefficient; see McCulloch and Rossi (1992) for a discussion. Although many strategies for increasing the efficiency of the crude Monte Carlo are available, it seems that there is nothing available to aid the estimation of the marginal likelihood of a linear regression model with the possibly most popular default prior density. Note that we cannot, at least readily, find functions of such that their analytical expectations with respect to the prior (21) are available, so is not easy to construct control variables to reduce the variance of the crude Monte Carlo estimator.
We test our estimator by estimating (20) in synthetic data where there is a clear prior-posterior conflict. We generate data points with predictors , and response . The algorithm with being the prior (21) requires a proposal density . Our estimator performs well when this proposal is both close to the target density (21) and the expectation is analytically available. We choose to use a proposal of the form
(22) |
For fixed and model the parameters can be estimated by an Expectation-Maximisation(EM) algorithm as follows. Let be an identity matrix with dimension equal to the dimension of . We fit a model based on 1000 data points generated from the generic prior
(23) |
and we obtain estimates and . Note that this algorithm is fitted only once and converges rapidly because all normal components have zero means. Then we sample from the density
(24) |
and we exploit the invariance of the KL divergence under linear transformation to obtain as samples generated from the density (22) that matches very well (21).
We chose by visual inspecting a Q-Q plot of the samples of (23) and (24). The algorithm ran for iterations and compared with the corresponding crude Monte Carlo through VRFs based on independent repetitions of the experiment. Results for all models are shown in Table 1 where it is evident that our estimator approximates well the marginal likelihood in all models. For comparative purposes we have included precise estimations of the marginal likelihoods obtained by first integrating and then performing numerical integration over . Notice that the difference of marginal likelihoods in -scale is used in Kass and Raftery (1995) as a way to interpret the weighted evidence against a null hypothesis with Bayes factors. It is also clear that as the parameter dimension increases, the variance of the crude Monte Carlo estimator and consequently the size of the variance reduction factor increase.
A realistic application of Bayesian model choice is when the size of models under comparison is large so estimation of all marginal likelihoods (20) is not feasible. In these cases model search MCMC algorithms that sample in the space of models provide a way to calculate posterior model probabilities only in models in which is not negligible; see, for example Liang et al. (2022). However, this typically requires the analytical availability of the marginal likelihood (20) for each model . Our unbiased independent Metropolis estimator of the marginal likelihood allows the bypass of this requirement by using a model search algorithm which is based on the pseudo-marginal MCMC proposed by Andrieu and Roberts (2009) and simply uses as target mass function our estimators of (20) for each model . For illustration, we performed a simple random walk MCMC algorithm on the discrete space of the models in our example and in Figure 2 it is shown that the pseudo-marginal MCMC based on our estimators performs very well and certainly much better than the corresponding crude Monte Carlo based algorithm. The precise estimates of Table 1 based on numerical integration are also shown.
Model | - | - | Numerical integration | VRF |
---|---|---|---|---|
93.19 | 93.19 | 93.19 | 4.29 | |
93.77 | 93.77 | 93.77 | 3.74 | |
77.23 | 77.08 | 77.16 | 8.71 | |
72.78 | 72.80 | 72.77 | 6.30 | |
93.35 | 93.34 | 93.35 | 5.01 | |
77.95 | 77.81 | 77.94 | 8.10 | |
73.52 | 73.66 | 73.51 | 4.02 | |
77.82 | 77.66 | 77.83 | 5.39 | |
73.57 | 73.36 | 73.55 | 16.74 | |
55.31 | 55.08 | 55.29 | 37.25 | |
78.53 | 78.30 | 78.53 | 30.34 | |
74.20 | 73.97 | 74.23 | 25.36 | |
55.39 | 55.49 | 55.34 | 49.12 | |
56.14 | 55.56 | 56.00 | 106.15 | |
55.91 | 57.71 | 55.93 | 4763.42 |
3 Adaptive independent Metropolis
In general, we consider the sequence of independent Metropolis adaptations identified by the proposal densities , indexed by corresponding parameter vectors . For each step , we have an update function and its stepsize . Then, the adaptive independent Metropolis can be implemented as shown in Algorithm 1.
Input: Parameterised proposal distribution , (intractable) target distribution , update function , stepsize , total number of adaptation iterations .
Initialisation Set and initialise {, , , } by {, , , }.
Return: Proposals and samples .
The ergodicity of the Markov chain is guaranteed by assumptions on the specific procedure of adaptation. Throughout this Section we will be assuming that all independent Metrpolis sampling chains have a state space and satisfy the following Assumption:
Assumption 4 (A4)
The parameters of the proposal density converge through the long run of Algorithm 1, i.e. , for some .
Moreover, in order to explore the properties of the adaptive MCMC, we need to rewrite (A2) and (A3) into their adaptation version:
Assumption 5 (A5)
for all with .
Assumption 6 (A6)
The Markov chain produced by the sampler is -irreducible and aperiodic with unique invariante measure for all , and there are functions , a small set , and a finite constant , such that the Lyapunov condition holds and .
3.1 Adaptation based on KL divergence
Theorem 2 and the numerical illustrations of the previous Section indicate that the key element of a good estimator of an sampler is the ability to obtain a good proposal density that is as close as possible, in terms of KL divergence, to the target . Therefore, in this Section we define the update function in Algorithm 1 as a gradient direction towards minimising the KL divergence . To further justify this choice, we present the following result from Theorem 2.
Corollary 1
Corollary 1 indicates that if we obtain a sequence of adaptations such that the proposal densities approach the target density as increases, we can achieve zero variance estimators. The requirement is the adaptation to satisfy
for a sequence of that converges to zero.
In practice, a way to obtain the sequence of the adapted proposal distributions is by applying gradient descent updates on the parameters in order to minimise the above KL divergence. The gradient descent update takes the form
(25) |
and is a step size parameter. This exact optimisation procedure is not feasible since the gradient is not available in closed-form, and instead we can apply stochastic optimisation (Robbins and Monro, 1951) where we use unbiased estimates of the gradients. More precisely, we can follow the recent literature in machine learning where such stochastic gradient optimisation methods are widely used for KL-based variational inference (Ranganath et al., 2014; Titsias and Lázaro-Gredilla, 2014; Kingma and Welling, 2013; Kucukelbir et al., 2017). We will assume that the independent Metropolis proposal is reparametrisable from a simpler distribution , which enables the use of efficient reparametrisation gradient methods (Titsias and Lázaro-Gredilla, 2014; Rezende et al., 2014; Kingma and Welling, 2013; Gianniotis et al., 2016; Salimans and Knowles, 2013; Tan and Nott, 2018). For instance, for the standard multivariate Gaussian proposal where and is the Cholesky factor of the covariance matrix, we can reparametrise as , and then consider as the update function in Algorithm 1 the unbiased gradient
(26) |
where is the negative entropy of the Gaussian distribution; see Appendix D. The simplest form of adaptation is to apply stochastic optimisation updates at each iteration of independent Metropolis, i.e. similarly to standard adaptive MCMC (Haario et al., 2001; Roberts and Rosenthal, 2009; Andrieu and Thoms, 2008). To perform the adaptation step at the -th sampling iteration we use the proposed state to evaluate the unbiased gradient in (26), and then perform a gradient step to obtain the new parameters by applying the update step in Algorithm 1 (line 3).
3.2 Theory and implementation of the adaptive independent Metropolis
3.2.1 Sample collection after adaptation
A trivial sub-optimal way to implement the general adaptation Algorithm 1 is to adapt until it reaches a final proposal density and ignore the samples generated by the adaptations . Then, the ergodic estimator (15) can be evaluated by using the samples from the non-adaptive algorithm and the Proposition 1 provides the necessary theoretical guarantees for such scheme.
3.2.2 Sample collection during adaptation
Clearly the previous sampling scheme of subsection 3.2.1 can be inefficient since it produces an estimator that ignores all samples from the adaptation phase. In this section, we wish to construct better estimators that make use of all samples, and to that end we will study the properties of the whole sequence of the independent Metropolis adaptations and the corresponding sequence of generated samples.
Laws of large numbers (LLN) and central limit theorems (CLT) have been proved for various cases of sequences, see Andrieu and Moulines (2006) and Roberts and Rosenthal (2007). However, some of the conditions required for these proofs are hard to verify in practice, see for example Andrieu and Moulines (2006). Roberts and Rosenthal (2007) prove a LLN that assumes that the function is bounded. We propose an adaptive algorithm in which adaptation takes place every iterations. This has two key advantanges. First, the update (25) is based on samples so the estimate (26) is based on samplers. Second, our estimator is now based on batches of size , , and this allows us to prove a LLN relaxing the assumption of boundness of .
Assume that we collect samples where denote the accepted samples and denote the corresponding proposal values from the proposal densities . Our estimator is
|
(27) |
with
(28) |
and
(29) |
Inputs: Parameterised proposal distribution , (intractable) target distribution , update function , stepsize , batch size , the number of batches , objective function .
Initialisation Set and initialise {, , , } by {, , , }.
Returns: Estimator in (27).
The resulting Algorithm 2 describes our adaptive independent Metropolis scheme and the following Theorem gives rigorous theoretical arguments.
3.3 Numerical illustrations
We evaluate the performance of the adaptive independent Metropolis algorithm with synthetic and real data applications. Our interest lies in comparing, with given number of iterations , the variance of the standard IM estimator in (6) and the variance of our IMCV estimator in (27) where . This comparison will be based on the ratio of the former to the latter variance estimated by the VRF in exactly the same way as in Section 2.4.
3.3.1 Adaptive IM with d-dimentional Gaussian target and Gaussian proposal
We consider a zero-mean and identity covariance -dimensional Gaussian target and an adapted Gaussian proposal initialised as where is a vector of ones and the matrix has elements if and if . Samples of size were drawn from adaptive independent Metropolis Algorithm 2 with adaptation scheme Algorithm 3 after burning-in adaptation MCMC steps. We set and . Ranges of variance reduction factors for the estimates of the expected values of each coordinate of the target are presented in Table 2.
Dimensions | ||||
---|---|---|---|---|
VRF | 41.2 - 87.5 | 20.3 - 41.6 | 6.2 - 12.2 | 2.9 - 7.6 |
In Appendix C we graphically present the results of Table 2 and we further investigate the VRF reduction with respect to the dimension . We prove that if the proposal after adaptation can be expressed as a normal density with some perturbations on both mean and covariance, then the VRF is bounded by a function which decreases at a rate of as increases.
3.3.2 Logistic regression
We consider binary classification based on binary labels and input data where are -dimensional input vectors. We assume a logistic regression log-likelihood where with being a -dimensional parameter vector. We place a -dimensional Gaussian prior on and we illustrate the performance of the adaptive independent Metropolis algorithm equipped with our control variate estimators in two datasets that have been commonly used in MCMC applications, see e.g. Girolami and Calderhead (2011) and Titsias and Dellaportas (2019). The Ripley dataset has sample size and number of covariates, whereas the Heart dataset has .
We estimate the posterior expected values of all parameters and of the odd ratio where is the average input vector. The proposal distribution is with and initially being a zero vector and the identity matrix respectively and being adapted with Algorithm 3 in Appendix D. Note that the required analytical expectations with respect to the proposal density are readily available, for example The variance reduction factors of our estimators against the standard IM estimator are shown in Table 3. The Ripley dataset achieves better VRFs than the Heart dataset because the lower dimension of the posterior implies a better approximation of the proposal to the target. Moreover, as the batch and sample sizes increase, we obtain higher variance reduction.
, | , | |||||
---|---|---|---|---|---|---|
Dataset | ||||||
Ripley | 42.4 - 50.5 | 40.7 - 50.1 | 43.6 | 41.4 - 57.2 | 42.6 - 54.5 | 52.5 |
Heart | 6.5 - 18.5 | 3.7 - 16.3 | 12.2 | 12.5 - 22.1 | 10.3 - 19.7 | 17.4 |
3.3.3 Gaussian process hyperparameters
In our final example we consider Bayesian estimation of kernel hyperparameters in Gaussian process (GP) models (Murray and Adams, 2010; Filippone and Girolami, 2014). We assume standard regression observed data , where each is an input vector and is the corresponding scalar response. We model each response using a Gaussian likelihood, i.e. and assign to the latent function a zero-mean GP distribution so that and is the kernel or covariance function. In our experiment, we use a GP having the squared-exponential kernel
so that overall the GP regression model depends on three hyperparameters , while the latent function variables can be analytically integrated out. To do Bayesian inference over the hyperparameters we place a Gaussian prior on log space, i.e. and then apply independent Metropolis sampling by using a multivariate normal density as the proposal distribution. This proposal was adapted to approximate the exact GP posterior . We applied the sampler and our estimator to two data sets: Boston housing (Harrison and Rubinfeld, 1978) and Pendulum (Lázaro-Gredilla et al., 2010). The Boston housing dataset has samples and number of covariates, whereas the Pendulum dataset has and . As the function we used both and where so that for the second case the expected value of gives an estimate of the posterior mean of each hyperparameter in the linear space. Note that the expectation of under the proposal density (needed for the evaluation of the control variate) is analytic by using the formula .
In this experiment, we set the number of batches to and the batch size to . The VRF scores are given in Table 4 which shows that the variance of is smaller than . Again, there is considerable variance reduction for all estimators in both log space and linear space.
Dataset | ||||||
---|---|---|---|---|---|---|
Boston | 4.6 | 2.7 | 11.6 | 8.1 | 4.5 | 7.5 |
Pendulum | 9.8 | 14.5 | 6.7 | 10.2 | 5.4 | 5.0 |
4 Discussion
Independent Metropolis algorithms have not attracted much interest in the MCMC community because their poor performance when compared with other Metropolis-based sampling strategies. We have attempted to revisit their applicability domain by pointing out that they can produce, without any extra computational effort, ergodic estimators of any function of interest equipped with control variates that have reduced asymptotic variance.
Our main result indicates that when the proposal density is close enough to the target density, these estimators can outperform the i.i.d. estimators based on crude Monte Carlo. Although this seems counter-intuitive since MCMC samples are not independent as those from crude Monte Carlo, the key idea is that the dependency of MCMC samples provides the ability to construct control variates through the theory of solutions to the corresponding Poisson equation. This opens a new methodological avenue to construct reduced variance estimators based on Monte Carlo that goes beyond the usual tools of importance sampling or standard control variates.
Incorporation of our key idea to an adaptive independent Metropolis algorithm based on reducing the KL-divergence between the target and the proposal density has produced a very efficient sampling strategy as shown in a series of synthetic and real data examples. To illustrate our methodology we revisited some common Bayesian statistics problems such as approximation of marginal likelihoods and sampling from Bayesian posterior distributions in logistic regression and Gaussian processes. We have shown that the applicability of independent Metropolis algorithms may be extended to produce solutions to many interesting popular statistical problems.
Further improvements of the adaptive independent Metropolis algorithm can be related to the stochastic gradient optimisation scheme that we currently use to fit the proposal distribution to the target distribution by minimising the KL-divergence. One direction is to reduce the variance of the stochastic gradients in the optimisation – a topic that has attracted a lot of attention by the machine learning community (Roeder et al., 2017; Miller et al., 2017; Geffner and Domke, 2020) and it deserves further investigation. A second direction is to exploit parallelisation in the computations of the independent Metropolis algorithm, which thanks to the independent form of the proposal distribution, and unlike other MCMC algorithms, can be largely parallelised (Jacob et al., 2011). The use of parallelisation is an interesting topic for future research since it could lead to faster adaptation and further improvements of the estimators.
Acknowledgements
We would like to thank Sam Livingstone, Max Hird and Arnaud Doucet for helpful comments.
Appendix A Appendix
A.1 Proof of Theorem 1
Proof A.4.
For each proposal density and solution of Poisson equation for a function , the Poisson equation is
Then, for every , since ,
Therefore, from the Poisson equation we obtain that which implies that .
A.2 Proof of Proposition 1
Proof A.5.
The convergence of transition kernel is proved by Brofos et al. (2022), Theorem 4.3 and proposition 4.4. Since the data collection takes place after adaptation the transition kernel of this part is invariant and the chain is reversible. Based on Theorem 2 of Dellaportas and Kontoyiannis (2012), the law of large numbers and the central limit theorem hold. Moreover, Theorem 3 of Dellaportas and Kontoyiannis (2012) provides the proof for the convergence of . The convergence of is immediate since it is a linear regression coefficient.
A.3 Proof of Theorem 2
The proof of the Theorem will be based on the following three Lemmas:
Lemma A.6.
Lemma A.7.
Proof A.8.
We first define the centralised version of as . The asymptotic variance (18) can be written as
(31) |
where is the -lag covariance of . This can be written as (Tan, 2006):
where and denotes the indicator function on set . Therefore, from (31) we obtain
(32) |
where . We will bound (A.8) by bounding both terms separately. For the first term, first observe that
(33) |
Then we have that
(since for the second term above, when , we have ) | ||||
(34) |
The second term of (A.8) is directly bounded by
(35) |
Combining (A.8) and (35) we obtain
To connect (30) with KL divergence, we need to bound the difference of variance under different measures via total variation distance and link to KL divergence by using Pinsker’s inequality. Thus, we need the following lemma.
Lemma A.9.
Proof A.10.
From (A1), there exists a such that
(36) |
By applying Cauchy-Schwarz inequality we obtain
where denotes the total variation distance which, by Pinsker’s inequality, is bounded in terms of the KL divergence as . Therefore,
Remark A.11.
Assume and define . By further use of the Cauchy-Schwarz inequality in the proof of Lemma A.9 we have
Also, the assumption can be restated as , so we have
(37) |
Proof A.12 (of Theorem 2).
(38) |
Since , we have . Therefore, from (A.12),
From (36) we have that and by setting in (A.11) in Remark A.11 we obtain
(39) |
Moreover, notice that is the centralised estimator and , so . Then,
(40) |
By combining Lemma A.7 and A.9 the bound (30) can be written as
(41) |
We now bound both terms of (A.12). The first is bounded by using (A.12):
For the second term of (A.12) we use (A.12) :
(42) | |||
(43) | |||
(44) |
The three terms I,II and III in equations (42), (43) and (44) respectively are bounded as follows: From (A.12), using ,
By Lemma A.9,
II | |||
Finally, by Pinsker’s inequality and the fact that we have
Thus,
(45) |
The true variance for crude Monte Carlo estimator is given by
and we need to prove the existence of of inequality:
(46) |
When , we define
and from (A.12), we know that on . Also, we have . Thus, the inequality (46) holds for all such that
(47) |
Thus, the that satisfy (47) belong to a subset of the ones that satisfy (46). Therefore, if the inequality (47) is satisfied then . Moreover, is immediate since the true variance of crude Monte Carlo is in (31), and the rest of the summation in (31) is no less than 0.
Remark A.13.
We can derive another upper bound for via spectral theory (Kipnis and Varadhan, 1986; Geyer, 1992; Rosenthal, 2003; Naesseth et al., 2020) as follows. For any square integrable function with respect to the stationary distribution ,
(48) |
where is the second largest eigenvalues of the Markov transition kernel. Wang (2022) also derives (48) and bounds the variance as
(49) |
Notice that this inequality uses the conjecture proposed by Liu (1996) that in a continuous space the spectral gap between the first and the second largest eigenvalues of the Markov transition kernel is . Wang (2022) also concludes this by incorrectly ignoring the non-negative first term of (A.8) from Tan (2006). We remark that the two different forms of bounds (30) and (49) do not affect the conclusion of Theorem 2. To see this, from (49) and the proof of Theorem 2, we have
By using the same proof, we have that
where . Also, the assumption should be strengthened to (A1).
A.4 Proof of Corollary 1
Proof A.14.
A.5 Proof of Theorem 3
Proof A.15.
For each adaptation step for , we define the within-batch estimator and without loss of generality we use the centralised version
(50) |
Note that . The estimator (27) is given by
We view the Markov chain as an extended B-variate chain on the state space with state , so the chain is . Denote by the i-th dimensional component of the state vector . Consider a sequence of adaptive proposal . Since is the transition kernel of , the transition kernel of new B-variate chain can be written as
-
1.
Under (A4), Theorem 4.3 of Brofos et al. (2022) guarantees that the sequence of independent Metropolis algorithms equipped with the adaptation of Algorithm 1 meets the ergodicity conditions stated by Theorem 1 of Roberts and Rosenthal (2007). Thus, the chain is ergodic. Moreover, the sequence of proposal densities converges to some in distribution and the transition kernels satisfy almost surely, where .
-
2.
We follow the proof of Theorem 5 of Roberts and Rosenthal (2007) who provided a weak LLN under the assumption that the function in (1) is strictly bounded. We can relax this condition by exploiting the fact that our function is given by a batch mean of many samples. Denote by the filtration generated by and a probability measure on the probability space . From (a), for any and any , we can choose such that , where is the usual total variation distance. Let where . Due to the diminishing condition, we can find a such that , when . Define the event to be all of the converged adaptive parameters.
By the coupling arguments provided by Roberts and Rosenthal (2007), for some we first construct an adaptive chain and its proposed samples with parameter sequence . Then, on event we construct a second chain and its proposed samples such that and are generated from . It is easy to show that for all and . Then, from the law of total expectation,
(51) By the law of large numbers applied to the within batch estimator in (50),
We can choose large enough, such that for all , . Thus, the first term of (2) can be bounded by . Then,
(52) The rest of the proof follows that of Roberts and Rosenthal (2007). We can choose a large such that . Then, the summation from to can be separated into three parts (denote [] the integer-part function): the head part from to , the tail part from to and the middle part contains intervals each of length :
Clearly that both of the head part and tail part can be bounded by , the middle part can be bounded by (52). Then, we have
By Markov’s inequality
we obtain
in probability as and .
- 3.
- 4.
Appendix B Analytics for examples
B.1 Analytical calculations for Section 2.4.1
We derive the analytical bounds for the toy examples in section 2.4.1.
-
•
The target density is and the proposal density is . Then,
Thus, because the exponential term is strictly decreasing. For (A.12), we have that
Since , and , we have that . Therefore,
Similarly, we have
Finally from (30), is bounded as follows:
(53) In section 2.4.1, we use numerical integration to estimate (• ‣ B.1) for a specific parameter .
-
•
The target density is and the proposal density is a student-t distribution with degrees of freedom with p.d.f given by
where denotes that gamma function. Then,
To find , we solve and then check , resulting to
Moreover,
Finally, we can estimate the bound (30) by Monte Carlo as follows.
(54) where and are samples from and are samples from .
B.2 Integrals for the marginal likelihood estimation of Section 2.4.2
From (21), the prior is
Assume is a -dimensional vector. The marginal likelihood (20) is
(55) |
where is a constant. Notice that (B.2) is a univariate integral so we use numerical integration to estimate it. Furthermore, in the static control variate term in (15), we need to compute the analytical expectation under the proposal density . In this example, the proposal is a discrete mixture of normals, see (22). Therefore,
(56) |
Moreover, to estimate the MCMC ratio, we need to calculate the pseudo-marginal prior by Monte Carlo with sample size (in our experiment set to 10) during each MCMC iteration,
Computational cost.
The computational cost for CMC and MCMC estimators is as follows. The matrix multiplication costs while the Cholesky decomposition of this matrix costs . Both these operations are common to both estimators and are needed to be preformed once at the beginning. Then given these two precomputations, sampling from the Gaussian costs while the likelihood evaluation costs (i.e. for computing the quadratic form ). This implies that the cost of the CMC estimator (given the precomputations of ) is where is the number of samples. This cost for large number of data is dominated by .
For the proposed MCMC estimator the evaluation of the expectation (56) in the static control variable has small cost where comes from computing efficiently the quadratic form using the precomputed Cholesky decomposition of . During MCMC, sampling from the -component mixture Gaussian proposal costs , while the evaluation of the M-H probability costs where is due to the evaluation of the pseudo-marginal prior and the mixture proposal. Given that we perform MCMC iterations to obtain the estimator the overall cost is dominated by which for large number of examples will be similar to CMC.
Appendix C Adaptive IM with different dimensional Gaussian target and Gaussian proposal
In Section 3.3.1, we illustrate the VRFs for the coordinate estimates of -dimensional Gaussian target and Gaussian proposal. Fig 3 provides a more intuitive display. It can be seen that, under the same optimisation algorithm and hyperparameters, as increases both the acceptance rate and the VRF decrease. We analyse here this relationship with Proposition C.16 under the following assumption:
Assumption 7 (A5)
For a Gaussian target , the proposal after adaptation can be expressed as where is a -dimensional vector with elements i.i.d., is diagonal with elements i.i.d. and is an orthogonal matrix.
Proposition C.16.
Under (A7), the expected KL divergence can be approximated by
Proof C.17.
Denote . Since the target is and the proposal is , by denoting with the trace of a matrix, we have that
Therefore,
In the Proof of Theorem 2 we have derived that the asymptotic variance can be bounded by . Thus, since ,
Therefore, with the increase of the dimensions, the decrease rate of the variance reduction factor is no faster than asymptotically. Moreover, increased precision in adaptation correlates with the amplification in the VRF.
Appendix D Adaptive IM algorithm with KL divergence instance
In all three experiments of Section 3.3 we use the adaptation algorithm combined with doubly stochastic variational inference (DSVI) (Titsias and Lázaro-Gredilla, 2014) and Adam optimisation (Kingma and Ba, 2014). The specific algorithm is described here in Algorithm 3.
Inputs: Gaussian proposal parameters , log-pdf of target distribution , stepsizes for two parameters (), batch size , the number of batches , objective function .
Initialisation Set and initialise {, , , } by {, , , }.
Returns: An estimate for (27).
References
- Alexopoulos et al. (2023) Alexopoulos, A., Dellaportas, P. and Titsias, M. K. (2023) Variance reduction for metropolis–hastings samplers. Statistics and Computing, 33(1), 6.
- Andradóttir et al. (1993) Andradóttir, S., Heyman, D. P. and Ott, T. J. (1993) Variance reduction through smoothing and control variates for markov chain simulations. ACM Transactions on Modeling and Computer Simulation (TOMACS), 3(3), 167–189.
- Andrieu and Moulines (2006) Andrieu, C. and Moulines, É. (2006) On the ergodicity properties of some adaptive mcmc algorithms. Annals of Applied Probability, 16(3), 1462–1505.
- Andrieu and Roberts (2009) Andrieu, C. and Roberts, G. O. (2009) The pseudo-marginal approach for efficient monte carlo computations. The Annals of Statistics, 37(2), 697–725.
- Andrieu and Thoms (2008) Andrieu, C. and Thoms, J. (2008) A tutorial on adaptive mcmc. Statistics and Computing, 18(4), 343–373.
- Assaraf and Caffarel (1999) Assaraf, R. and Caffarel, M. (1999) Zero-variance principle for monte carlo algorithms. Physical Review Letters, 83(23), 4682–4685.
- Atchadé and Perron (2005) Atchadé, Y. F. and Perron, F. (2005) Improving on the independent metropolis-hastings algorithm. Statistica Sinica, 15(1), 3–18.
- Barp et al. (2022) Barp, A., Oates, C. J., Porcu, E. and Girolami, M. (2022) A Riemann–Stein kernel method. Bernoulli, 28(4), 2181 – 2208.
- Belomestny et al. (2020) Belomestny, D., Iosipoi, L., Moulines, E., Naumov, A. and Samsonov, S. (2020) Variance reduction for markov chains with application to mcmc. Statistics and Computing, 30(4), 973–997.
- Brofos et al. (2022) Brofos, J., Gabrié, M., Brubaker, M. A. and Lederman, R. R. (2022) Adaptation of the independent metropolis-hastings sampler with normalizing flow proposals. In International Conference on Artificial Intelligence and Statistics, 5949–5986. PMLR.
- Cappé et al. (2008) Cappé, O., Douc, R., Guillin, A., Marin, J.-M. and Robert, C. P. (2008) Adaptive importance sampling in general mixture classes. Statistics and Computing, 18, 447–459.
- Cornebise et al. (2008) Cornebise, J., Moulines, É. and Olsson, J. (2008) Adaptive methods for sequential importance sampling with application to state space models. Statistics and Computing, 18(4), 461–480.
- Dellaportas et al. (2012) Dellaportas, P., Forster, J. J. and Ntzoufras, I. (2012) Joint specification of model space and parameter space prior distributions. Statistical Science, 232–246.
- Dellaportas and Kontoyiannis (2009) Dellaportas, P. and Kontoyiannis, I. (2009) Notes on using control variates for estimation with reversible mcmc samplers. arXiv preprint arXiv:0907.4160.
- Dellaportas and Kontoyiannis (2012) — (2012) Control variates for estimation based on reversible Markov chain Monte Carlo samplers. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 74(1), 133–161.
- Dellaportas and Tsionas (2019) Dellaportas, P. and Tsionas, M. G. (2019) Importance sampling from posterior distributions using copula-like approximations. Journal of Econometrics, 210(1), 45–57.
- Filippone and Girolami (2014) Filippone, M. and Girolami, M. A. (2014) Pseudo-marginal bayesian inference for gaussian processes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36, 2214–2226.
- Gabrié et al. (2022) Gabrié, M., Rotskoff, G. M. and Vanden-Eijnden, E. (2022) Adaptive Monte Carlo augmented with normalizing flows. Proceedings of the National Academy of Sciences of the United States of America, 119(10), e2109420119.
- Geffner and Domke (2020) Geffner, T. and Domke, J. (2020) Approximation based variance reduction for reparameterization gradients. In Advances in Neural Information Processing Systems (eds. H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan and H. Lin), vol. 33, 2397–2407. Curran Associates, Inc. URL: https://proceedings.neurips.cc/paper_files/paper/2020/file/193002e668758ea9762904da1a22337c-Paper.pdf.
- Geyer (1992) Geyer, C. J. (1992) Practical markov chain monte carlo. Statistical Science, 7(4), 473–483.
- Gianniotis et al. (2016) Gianniotis, N., Schnörr, C., Molkenthin, C. and Bora, S. (2016) Approximate variational inference based on a finite sample of gaussian latent variables. Pattern Analysis and Applications, 19, 475–485.
- Girolami and Calderhead (2011) Girolami, M. and Calderhead, B. (2011) Riemann manifold Langevin and Hamiltonian Monte Carlo methods. Journal of the Royal Statistical Society. Series B: Statistical Methodology, 73(2), 123–214.
- Haario et al. (2001) Haario, H., Saksman, E. and Tamminen, J. (2001) An adaptive Metropolis algorithm. Bernoulli, 7(2), 223–242.
- Harrison and Rubinfeld (1978) Harrison, D. and Rubinfeld, D. (1978) Hedonic Housing Prices and the Demand for Clean Air. Journal of Environmental Economics and Management, 5, 81–102.
- Henderson (1997) Henderson, S. G. (1997) Variance reduction via an approximating Markov process. Stanford University.
- Holden et al. (2009) Holden, L., Hauge, R. and Holden, M. (2009) Adaptive independent Metropolis-Hastings. Annals of Applied Probability, 19(1), 395–413.
- Jacob et al. (2011) Jacob, P., Robert, C. P. and Smith, M. H. (2011) Using parallel computation to improve independent metropolis–hastings based estimation. Journal of Computational and Graphical Statistics, 20(3), 616–635.
- Kahn and Marshall (1953) Kahn, H. and Marshall, A. W. (1953) Methods of reducing sample size in monte carlo computations. Journal of the Operations Research Society of America, 1(5), 263–278.
- Kass and Raftery (1995) Kass, R. E. and Raftery, A. E. (1995) Bayes factors. Journal of the American Statistical Association, 90(430), 773–795.
- Kingma and Ba (2014) Kingma, D. P. and Ba, J. (2014) Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
- Kingma and Welling (2013) Kingma, D. P. and Welling, M. (2013) Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114.
- Kipnis and Varadhan (1986) Kipnis, C. and Varadhan, S. R. (1986) Central limit theorem for additive functionals of reversible Markov processes and applications to simple exclusions. Communications in Mathematical Physics, 104(1), 1–19.
- Kucukelbir et al. (2017) Kucukelbir, A., Tran, D., Ranganath, R., Gelman, A. and Blei, D. M. (2017) Automatic differentiation variational inference. J. Mach. Learn. Res., 18(1), 430–474.
- Lázaro-Gredilla et al. (2010) Lázaro-Gredilla, M., Quiñnero-Candela, J., Rasmussen, C. E. and Figueiras-Vidal, A. R. (2010) Sparse spectrum gaussian process regression. Journal of Machine Learning Research, 11(63), 1865–1881.
- Liang et al. (2008) Liang, F., Paulo, R., Molina, G., Clyde, M. A. and Berger, J. O. (2008) Mixtures of g priors for bayesian variable selection. Journal of the American Statistical Association, 103(481), 410–423.
- Liang et al. (2022) Liang, X., Livingstone, S. and Griffin, J. (2022) Adaptive random neighbourhood informed markov chain monte carlo for high-dimensional bayesian variable selection. Statistics and Computing, 32(5), 84.
- Liu (1996) Liu, J. S. (1996) Metropolized independent sampling with comparisons to rejection sampling and importance sampling. Statistics and Computing, 6(2), 113–119.
- McCulloch and Rossi (1992) McCulloch, R. E. and Rossi, P. E. (1992) Bayes factors for nonlinear hypotheses and likelihood distributions. Biometrika, 79(4), 663–676.
- Meyn (2008) Meyn, S. (2008) Control techniques for complex networks. Cambridge University Press.
- Miller et al. (2017) Miller, A., Foti, N., D' Amour, A. and Adams, R. P. (2017) Reducing reparameterization gradient variance. In Advances in Neural Information Processing Systems (eds. I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan and R. Garnett), vol. 30. Curran Associates, Inc. URL: https://proceedings.neurips.cc/paper_files/paper/2017/file/325995af77a0e8b06d1204a171010b3a-Paper.pdf.
- Mira et al. (2013) Mira, A., Solgi, R. and Imparato, D. (2013) Zero variance Markov chain Monte carlo for Bayesian estimators. Statistics and Computing, 23(5), 653–662.
- Murray and Adams (2010) Murray, I. and Adams, R. P. (2010) Slice sampling covariance hyperparameters of latent gaussian models. Advances in neural information processing systems, 23.
- Naesseth et al. (2020) Naesseth, C., Lindsten, F. and Blei, D. (2020) Markovian score climbing: Variational inference with . Advances in Neural Information Processing Systems, 33, 15499–15510.
- Newton and Raftery (1994) Newton, M. A. and Raftery, A. E. (1994) Approximate bayesian inference with the weighted likelihood bootstrap. Journal of the Royal Statistical Society Series B: Statistical Methodology, 56(1), 3–26.
- Oates et al. (2019) Oates, C. J., Cockayne, J., Briol, F. X. and Girolami, M. (2019) Convergence rates for a class of estimators based on Stein’s method. Bernoulli, 25(2), 1141–1159.
- Oates et al. (2017) Oates, C. J., Girolami, M. and Chopin, N. (2017) Control functionals for Monte Carlo integration. Journal of the Royal Statistical Society. Series B: Statistical Methodology, 79(3), 695–718.
- Owen (2013) Owen, A. B. (2013) Monte Carlo theory, methods and examples. Stanford.
- Paananen et al. (2021) Paananen, T., Piironen, J., Bürkner, P. C. and Vehtari, A. (2021) Implicitly adaptive importance sampling. Statistics and Computing, 31(2), 16.
- Papamarkou et al. (2014) Papamarkou, T., Mira, A., Girolami, M. et al. (2014) Zero variance differential geometric Markov chain Monte Carlo algorithms. Bayesian Analysis, 9(1), 97–128.
- Ranganath et al. (2014) Ranganath, R., Gerrish, S. and Blei, D. (2014) Black box variational inference. In International Conference on Artificial Intelligence and Statistics, 814–822.
- Rezende et al. (2014) Rezende, D. J., Mohamed, S. and Wierstra, D. (2014) Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31st International Conference on Machine Learning. PMLR.
- Richard and Zhang (2007) Richard, J. F. and Zhang, W. (2007) Efficient high-dimensional importance sampling. Journal of Econometrics, 141(2), 1385–1411.
- Robbins and Monro (1951) Robbins, H. and Monro, S. (1951) A Stochastic Approximation Method. The Annals of Mathematical Statistics, 22(3), 400–407.
- Robert et al. (2018) Robert, C. P., Elvira, V., Tawn, N. and Wu, C. (2018) Accelerating mcmc algorithms. Wiley Interdisciplinary Reviews: Computational Statistics, 10(5), e1435.
- Roberts and Rosenthal (2007) Roberts, G. O. and Rosenthal, J. S. (2007) Coupling and ergodicity of adaptive Markov chain Monte Carlo algorithms. Journal of Applied Probability, 44(2), 458–475.
- Roberts and Rosenthal (2009) — (2009) Examples of adaptive MCMC. Journal of Computational and Graphical Statistics, 18(2), 349–367.
- Roeder et al. (2017) Roeder, G., Wu, Y. and Duvenaud, D. K. (2017) Sticking the landing: Simple, lower-variance gradient estimators for variational inference. In Advances in Neural Information Processing Systems (eds. I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan and R. Garnett), vol. 30. Curran Associates, Inc. URL: https://proceedings.neurips.cc/paper_files/paper/2017/file/e91068fff3d7fa1594dfdf3b4308433a-Paper.pdf.
- Rosenthal (2003) Rosenthal, J. S. (2003) Asymptotic variance and convergence rates of nearly-periodic markov chain monte carlo algorithms. Journal of the American Statistical Association, 98(461), 169–177.
- Salimans and Knowles (2013) Salimans, T. and Knowles, D. A. (2013) Fixed-form variational posterior approximation through stochastic linear regression. Bayesian Analysis, 8(4), 837–882.
- Smith and Tierney (1996) Smith, R. and Tierney, L. (1996) Exact transition probabilities for the independence Metropolis sampler. Preprint.
- South et al. (2022) South, L. F., Karvonen, T., Nemeth, C., Girolami, M. and Oates, C. J. (2022) Semi-exact control functionals from Sard’s method. Biometrika, 109(2), 351–367.
- South et al. (2023) South, L. F., Oates, C. J., Mira, A. and Drovandi, C. (2023) Regularized zero-variance control variates. Bayesian Analysis, 18(3), 865–888.
- Tan and Nott (2018) Tan, L. S. L. and Nott, D. J. (2018) Gaussian variational approximation with sparse precision matrices. Statistics and Computing, 28(2), 259–275.
- Tan (2006) Tan, Z. (2006) Monte carlo integration with acceptance-rejection. Journal of Computational and Graphical Statistics, 15(3), 735–752.
- Titsias and Dellaportas (2019) Titsias, M. K. and Dellaportas, P. (2019) Gradient-based adaptive markov chain monte carlo. In Advances in Neural Information Processing Systems, vol. 32.
- Titsias and Lázaro-Gredilla (2014) Titsias, M. K. and Lázaro-Gredilla, M. (2014) Doubly stochastic variational bayes for non-conjugate inference. In Proceedings of the 31st International Conference on Machine Learning. PMLR.
- Valle and Leisen (2010) Valle, L. D. and Leisen, F. (2010) A new multinomial model and a zero variance estimation. Communications in Statistics—Simulation and Computation®, 39(4), 846–859.
- Wang (2022) Wang, G. (2022) Exact convergence analysis of the independent Metropolis-Hastings algorithms. Bernoulli, 28(3), 2012–2033.