Moment Matching Denoising Gibbs Sampling

Abstract

Energy-Based Models (EBMs) offer a versatile framework for modeling complex data distributions. However, training and sampling from EBMs continue to pose significant challenges. The widely-used Denoising Score Matching (DSM) method Vincent (2011) for scalable EBM training suffers from inconsistency issues, causing the energy model to learn a ‘noisy’ data distribution. In this work, we propose an efficient sampling framework, (pseudo)-Gibbs sampling with moment matching, which enables effective sampling from the underlying clean model when given a ‘noisy’ model that has been well-trained via DSM. We explore the benefits of our approach compared to related methods and demonstrate how to scale the method to high-dimensional datasets.

1 Energy-Based Models

Energy-Based Models (EBMs) have attracted a lot of attention in the generative model literature Ngiam et al. (2011); Xie et al. (2016); Du and Mordatch (2019); Song and Ermon (2019). EBMs are a type of non-normalized probabilistic model that determines the probability density function without a known normalizing constant. For continuous data $x$, the density function of an EBM is specified as $q_{\theta}(x)=\exp(-f_{\theta}(x))/Z(\theta)$ where the $f_{\theta}(x)$ is a nonlinear function with parameter $\theta$ and $Z(\theta)=\int\exp(-f_{\theta}(x))\mathop{}\!\mathrm{d}{x}$ is the normalization constant that is independent of $x$. The energy parameterization allows for greater flexibility in model parameterization and the ability to model a wider range of probability distributions. However, the lack of a known normalizing constant makes training these models challenging. We start by giving a brief introduction of how to estimate $\theta$ in EBMs and refer the reader to Song and Kingma (2021) for a detailed overview of different training techniques for continuous EBMs.

Likelihood-based training: A classic method to learn $\theta$ is to minimize the KL divergence between the data distribution $p_{d}$ and the model density $q_{\theta}$, which is defined as

$${\mathrm{KL}}(p_{d}||q_{\theta}) \doteq-\int p_{d}(x)\log q_{\theta}(x)\mathop{}\!\mathrm{d}{x}\doteq-\int p_{d}(x)f_{\theta}(x)\mathop{}\!\mathrm{d}{x}-\log Z(\theta),$$ (1)

where we use $\doteq$ to denote the equivalence up to a constant that is independent of $\theta$. The integration of $p_{d}(x)$ can be approximated by Monte Carlo with the training dataset $\mathcal{X}_{train}=\{x_{1},\cdots,x_{N}\}\sim p_{d}(x)$; in this case, it is equivalent to the maximum likelihood estimate (MLE) Bishop and Nasrabadi (2006). However, for EBMs, minimizing the KL divergence requires the estimation of $Z(\theta)$, which is intractable for nonlinear $f_{\theta}(x)$ defined by a neural network. Various methods have been proposed to alleviate the intractability by introducing techniques like Markov chain Monte Carlo (MCMC) Hinton (2002); Nijkamp et al. (2019); Du et al. (2020); Gao et al. (2018) or adversarial training Kim and Bengio (2016); Zhai et al. (2016); Bose et al. (2018).

Score-based training: Alternatively, Hyvärinen (2005) proposes to minimize the Fisher divergence to learn $\theta$, which is defined as

$${\mathrm{FD}}(p_{d}||q_{\theta})=\frac{1}{2}\int p_{d}(x)||s_{p_{d}}(x)-s_{q_{\theta}}(x)||^{2}_{2}\mathop{}\!\mathrm{d}{x},$$ (2)

where we use $s_{p}(x)$ to denote the score function of distribution $p$: $s_{p}(x)\equiv\nabla_{x}\log p(x)$. Under certain regularity conditions, the Fisher divergence is equivalent to the score-matching (SM) objective Hyvärinen (2005),

$${\mathrm{FD}}(p_{d}||q_{\theta})\doteq\frac{1}{2}\int p_{d}(x)\left(||s_{q_{\theta}}(x)||_{2}^{2}+2\operatorname{Tr}(\nabla_{x}s_{q_{\theta}}(x))\right)\mathop{}\!\mathrm{d}{x},$$ (3)

which does not require estimation of the intractable $Z(\theta)$. However, this objective needs to calculate the Hessian trace $\nabla_{x}s_{q_{\theta}}(x)=-\nabla^{2}_{x}f_{\theta}(x)$ in every gradient step during training, which is computationally expensive and does not scale to high dimensional data or requires approximation Song et al. (2020). In this paper, we will focus on another training method, denoising score matching Vincent (2011), which overcomes the tractability and scalability issues mentioned above, and is introduced in the next section.

1.1 Denoising Score Matching

For the target data density $p_{d}(x)$, a noise distribution $p(\tilde{x}|x)=\operatorname{\mathcal{N}}(x,\sigma^{2}I)$ is introduced to construct a noised data distribution $\tilde{p}_{d}(\tilde{x})=\int p_{d}(x)p(\tilde{x}|x)\mathop{}\!\mathrm{d}{x}$. Denoising score matching (DSM) Vincent (2011) minimizes the Fisher divergence between the noised data distribution $\tilde{p}_{d}(\tilde{x})$ and an energy-based model $\tilde{q}_{\theta}(\tilde{x})=\exp(-f_{\theta}(\tilde{x}))/Z(\theta)$, with

$${\mathrm{FD}}(\tilde{p}_{d}||\tilde{q}_{\theta}) =\frac{1}{2}\int\tilde{p}(\tilde{x})||s_{\tilde{p}_{d}}(\tilde{x})-s_{\tilde{q}_{\theta}}(\tilde{x})||^{2}_{2}\mathop{}\!\mathrm{d}{\tilde{x}}$$

$$\doteq\frac{1}{2}\iint p(\tilde{x}|x)p_{d}(x)||\nabla_{\tilde{x}}\log p(\tilde{x}|x)-s_{\tilde{q}_{\theta}}(\tilde{x})||^{2}_{2}\mathop{}\!\mathrm{d}{\tilde{x}}\mathop{}\!\mathrm{d}{x}$$

$$\doteq\frac{1}{2}\iint p(\tilde{x}|x)p_{d}(x)\left\lVert\frac{\tilde{x}-x}{\sigma^{2}}+s_{\tilde{q}_{\theta}}(\tilde{x})\right\rVert_{2}^{2}\mathop{}\!\mathrm{d}{\tilde{x}}\mathop{}\!\mathrm{d}{x}\>,$$ (4)

where the last equation is due to $\nabla_{\tilde{x}}\log p(\tilde{x}|x)$ being tractable for the Gaussian distribution $p(\tilde{x}|x)$.

Compared to the KL or SM objectives, the DSM objective is scalable and well-defined when the data distribution is singular Zhang et al. (2020) and can alleviate the blindness problem of score matching Song and Ermon (2019); Wenliang and Kanagawa (2020); Zhang et al. (2022b). On the other hand, there is a notable disadvantage associated with the DSM objective: for a fixed $\sigma>0$, the DSM objective is not a consistent objective for learning the underlying data distribution $p_{d}$ since ${\mathrm{FD}}(\tilde{p}_{d}||\tilde{q}_{\theta})=0\implies\tilde{p}_{d}=\tilde{q}_{\theta}\neq p_{d}$. A common solution is to anneal $\sigma\rightarrow 0$ during training. However, Equation 4 is not defined when $\sigma=0$ since the division in Equation 4 will make $(\tilde{x}-x)/\sigma^{2}$ unbounded, which results in an inconsistent objective. Annealing $\sigma$ increases the variance of the training gradients Song and Kingma (2021); Wang et al. (2020), which makes the optimization challenging in practice.

To overcome the challenges, we propose an alternative data generation scheme: we use DSM with a fixed $\sigma>0$ to train a ‘noisy’ energy model and then construct a sampler which targets the underlying ‘clean’ model. Specifically, our contributions are summarized as follows:

• We demonstrate that for an EBM that learns a noisy data distribution, there exists a unique underlying clean model which recovers the true data distribution.

• We introduce a pseudo-Gibbs sampling scheme incorporating an analytical moment-matching approximation of the denoising distribution. This allows us to sample from the underlying clean model without requiring additional training.

• We illustrate how to scale our method for high-dimensional data and demonstrate the generation of high-quality images using only a single level of fixed noise. Furthermore, we showcase the application of our proposed method in multi-level noise scenarios, closely resembling a diffusion model.

2 Clean Model Identification

For a fixed $\sigma>0$, DSM can only learn a ‘noisy’ data distribution $\tilde{q}_{\theta}(\tilde{x})$ even in the ideal case where the Fisher divergence is exactly minimized, since ${\mathrm{FD}}(\tilde{p}_{d}||\tilde{q}_{\theta^{*}})=0\rightarrow\tilde{p}_{d}=\tilde{q}_{\theta}\neq p_{d}$. In this case, the following theorem shows that there exists a ‘clean’ model that is implicitly defined that learns the true data distribution.

Theorem 2.1 (Existence of the underlying clean model for optimal $\tilde{q}_{\theta}(\tilde{x})$).

When the Fisher divergence goes to 0, ${\mathrm{FD}}(\tilde{p}_{d}||\tilde{q}_{\theta})=0\rightarrow\tilde{p}_{d}=\tilde{q}_{\theta}$, there exists an unique underlying clean model $q(x)$ such that $\tilde{q}_{\theta}(\tilde{x})=\int q(x)p(\tilde{x}|x)\mathop{}\!\mathrm{d}{x}$ and $q(x)=p_{d}(x)$.

See Appendix A.1 for proof. This theorem shows that despite training an EBM on noisy data, there is an implicit model within it that can recover the true data distribution. Therefore, instead of annealing the noise $\sigma\rightarrow 0$ to recover the true data distribution, we will demonstrate how to directly sample from the implicitly-defined clean model given the noisy energy-based model in the next section.

We want to highlight that the ‘perfect fit’ assumption, i.e. achieving ${\mathrm{FD}}(\tilde{p}_{d}||\tilde{q}_{\theta})=0$, may not hold for a complex data distribution $p_{d}$ or underpowered EBMs. Therefore, we provide general sufficient conditions for the existence of the clean model for an imperfect EBM in Appendix A.2.

2.1 Gibbs Sampling with Gaussian Moment Matching

Given a well-trained noisy energy-based model $\tilde{q}_{\theta}(\tilde{x})=\tilde{p}_{d}(\tilde{x})$, the clean model has the form

$$q(x)=\int p(x|\tilde{x})\tilde{q}_{\theta}(\tilde{x})d\tilde{x},$$ (5)

where the denoising distribution can be written as $p(x|\tilde{x})\propto p(\tilde{x}|x)q(x)$. We notice that, since the noise distribution $p(\tilde{x}|x)=\operatorname{\mathcal{N}}(x,\sigma^{2}I)$ is known, a Gibbs sampling scheme can be constructed to sample from the underlying clean model if we know the denoising distribution $p(x|\tilde{x})$, with

$$\tilde{x}_{k-1}\sim p(\tilde{x}|x=x_{k-1}),\quad x_{k}\sim p(x|\tilde{x}=\tilde{x}_{k-1}),$$ (6)

where the initial sample $x_{0}\sim p_{0}(x)$ can be drawn from a standard Gaussian $p_{0}(x)=\mathcal{N}(0,I)$. However, as the denoising distribution $p(x|\tilde{x})$ is usually intractable for complex $p_{d}$, we propose an analytical Gaussian moment matching approximation of $p(x|\tilde{x})$.

Denote the mean and covariance of $p(x|\tilde{x})$ as

$$\mu(\tilde{x})=\langle x\rangle_{p(x|\tilde{x})},\quad\Sigma(\tilde{x})=\langle x^{2}\rangle_{p(x|\tilde{x})}-\langle x\rangle^{2}_{p(x|\tilde{x})}.$$ (7)

The classic Gaussian moment matching method Minka (2013) specifies a Gaussian approximation $p(x|\tilde{x})\approx\mathcal{N}(\mu(\tilde{x}),\Sigma(\tilde{x}))$, which matches the first and second moment of $p(x|\tilde{x})$. When $\tilde{q}_{\theta}=\tilde{p}_{d}$, the first mean of the denoised distribution has a well-known analytical form Song et al. (2021); Bao et al. (2022); Efron (2011); Robbins (1992)

$$\mu(\tilde{x})=\tilde{x}+\sigma^{2}s_{\tilde{q}_{\theta}}(\tilde{x});$$ (8)

we include the derivation in Appendix A.3. Using this identity, we can rewrite Equation 4 as

$$\mathrm{FD}(\tilde{p}_{d}||\tilde{q}_{\theta})\doteq\frac{1}{2\sigma^{4}}\iint p(\tilde{x}|x)p_{d}(x)\left\lVert x-\mu(\tilde{x})\right\rVert_{2}^{2}\mathop{}\!\mathrm{d}{\tilde{x}}\mathop{}\!\mathrm{d}{x},$$ (9)

where we can see that the Fisher divergence only depends on $\mu(\tilde{x})$. Since $\mathrm{FD}(\tilde{p}_{d}||\tilde{q}_{\theta})=0\Leftrightarrow q=p_{d}$ (Theorem 2.1), the function $\mu(\tilde{x})$ fully characterizes the distribution $q$. Therefore, $\mu(\tilde{x})$ and $p(\tilde{x}|x)=\operatorname{\mathcal{N}}(x,\sigma^{2}I)$ can provide sufficient information to determine $p(x|\tilde{x})\propto q(x)p(\tilde{x}|x)$. As a consequence, the following theorem shows that the covariance function can also be analytically derived.

Theorem 2.2 (Analytical Covariance Identity).

Given a clean model $q(x)$ such that $\int q(x)p(\tilde{x}|x)\mathop{}\!\mathrm{d}{x}=\tilde{q}_{\theta}(\tilde{x})=\tilde{p}_{d}(\tilde{x})$ with $p(\tilde{x}|x)=\operatorname{\mathcal{N}}(x,\sigma^{2}I)$, the $\mu(\tilde{x})$ and $\Sigma(\tilde{x})$ of the $p(x|\tilde{x})\propto q(x)p(\tilde{x}|x)$ has the following relations

$$\Sigma(\tilde{x})=\sigma^{2}\nabla_{\tilde{x}}\mu(\tilde{x})=\sigma^{4}\nabla_{\tilde{x}}^{2}\log\tilde{q}_{\theta}(\tilde{x})+\sigma^{2}I.$$ (10)

See Appendix A.3 for proof. This analytical covariance identity can be seen as a high-dimensional generalization of the 2nd-order Tweedie’s Formula Efron (2011); Robbins (1992). Therefore, the analytical full-covariance moment matching approximation can be written as

$$p(x|\tilde{x})\approx\operatorname{\mathcal{N}}(\tilde{x}+\sigma^{2}\nabla_{\tilde{x}}\log\tilde{q}_{\theta}(\tilde{x}),\sigma^{4}\nabla_{\tilde{x}}^{2}\log\tilde{q}_{\theta}(\tilde{x})+\sigma^{2}I).$$ (11)

We want to highlight that since the Gaussian moment matching is only an approximation of $p(x|\tilde{x})$, the sampling scheme in Equation 6 is a ‘pseudo’ Gibbs sampler unless the true $p(x|\tilde{x})$ is also a Gaussian distribution, which is not true for general non-Gaussian $p_{d}$. However, since $\mu(\tilde{x})$ and $p(\tilde{x}|x)=\operatorname{\mathcal{N}}(x,\sigma^{2}I)$ are already sufficient to specify $p(x|\tilde{x})$, it should be possible to derive expressions for higher-order moments which themselves involve only $\mu(\tilde{x})$ and $\sigma$; we leave this to future work. To our knowledge, the $\tilde{x}$-conditioned full covariance Gaussian moment matching approximation to $p(x|\tilde{x})$ has not been derived previously. In the next section, we briefly discuss the connections between our method and other related approaches.

2.2 Connection to Covariance Learning Approaches

Bengio et al. (2013) proposes to approximate the true posterior $p(x|\tilde{x})\propto p_{d}(x)p(\tilde{x}|x)$ with a variational distribution $q_{\theta}(x|\tilde{x})$. The parameter $\theta$ is then learned by minimizing the joint KL divergence

$${\mathrm{KL}}(p(\tilde{x}|x)p_{d}(x)\lVert q_{\theta}(x|\tilde{x}))\tilde{p}_{d}(\tilde{x}))\doteq-\iint p_{d}(x)p(\tilde{x}|x)\log q_{\theta}(x|\tilde{x})\mathop{}\!\mathrm{d}{\tilde{x}}\mathop{}\!\mathrm{d}{x},$$ (12)

where $\tilde{p}_{d}(\tilde{x})=\int p_{d}(x)p(\tilde{x}|x)\mathop{}\!\mathrm{d}{x}$. The joint KL divergence in Equation 12 encourages $q_{\theta}(x|\tilde{x})$ to match the moments of the true posterior $p(x|\tilde{x})$, and defines an upper bound of the marginal KL Zhang et al. (2019)

$${\mathrm{KL}}(p(\tilde{x}|x)p_{d}(x)\lVert q_{\theta}(x|\tilde{x}))\tilde{p}_{d}(\tilde{x}))\geq{\mathrm{KL}}(p_{d}(x)||q_{\theta}(x)),$$ (13)

where the model is implicitly defined as the marginal of the joint $q_{\theta}(x)=\int q_{\theta}(x|\tilde{x})\tilde{p}_{d}(\tilde{x})$. When $q_{\theta}(x|\tilde{x})$ is a consistent estimator of $p(x|\tilde{x})$, this asymptotic distribution of the Gibbs sampling will converge to the true data distribution $p_{d}(x)$ Bengio et al. (2013). For continuous data, the variational distribution is chosen as a Gaussian distribution $q_{\theta}(x|\tilde{x})=\operatorname{\mathcal{N}}(\mu_{\theta}(\tilde{x}),\Sigma_{\theta}(\tilde{x}))$, where the mean $\mu_{\theta}(\cdot)$ and the covariance $\Sigma_{\theta}(\cdot)$ are parameterized by neural networks. We note that the only difference between the KL and DSM objective (Equation 9) is that the KL objective additionally learns the covariance. We thus show that the optimal covariance under KL minimization is the proposed analytical covariance.

Theorem 2.3 (Optimal Gaussian Approximation).

Let $p(\tilde{x}|x)=\operatorname{\mathcal{N}}(0,\sigma^{2}I)$ and assume Gaussian distribution $q_{\theta}(x|\tilde{x})=\operatorname{\mathcal{N}}(\mu_{q}(\tilde{x}),\Sigma_{q}(\tilde{x}))$, then the optimal $q^{*}$ such that

$$q^{*}=\arg\min_{q}{\mathrm{KL}}(p(\tilde{x}|x)p_{d}(x)\lVert q(x|\tilde{x})\tilde{p}_{d}(\tilde{x}))$$ (14)

has the mean and covariance with the form

$$\mu_{q}^{*}(\tilde{x})=\langle x\rangle_{p(x|\tilde{x})},\quad\Sigma_{q}^{*}(\tilde{x})=\sigma^{2}\nabla_{\tilde{x}}\mu_{q}^{*}(\tilde{x}),$$ (15)

see Appendix A.4 for proof. Therefore, when the optimal mean function is learned $\mu_{\theta}(\tilde{x})=\mu_{q}^{*}(\tilde{x})$, the optimal $\Sigma(\tilde{x})$ can be analytically derived, making the learning of $\Sigma_{\theta}(\tilde{x})$ redundant. In addition to the training inefficiency caused by more parameters, the amortized covariance network may suffer from poor generalization Zhang et al. (2022a). Moreover, the KL objective is also not well-defined for learning data distributions which lie on a low-dimensional manifold, e.g. MNIST, see Section 4 for a detailed discussion. In this case, the learned $\Sigma_{\theta}(\tilde{x})$ may be a degenerate matrix, making the Gaussian density function $q(x|\tilde{x})$ ill-defined Zhang et al. (2020) which impedes the training, see Figure 5 for an example.

Paper Meng et al. (2021) proposes a higher-order score-matching loss to simultaneously learn both the first order score $\nabla_{\tilde{x}}\log\tilde{q}_{\theta}(\tilde{x})$ and the second order score $\nabla^{2}_{\tilde{x}}\log\tilde{q}_{\theta}(\tilde{x})$. However, our findings indicate that the mean function $\mu(\tilde{x})$ (or the first order score $\nabla_{\tilde{x}}\log p(\tilde{x})$) already contains all the moment information of the underlying true distribution $p_{d}$, and the optimal moment can be derived using the mean function. Therefore, learning the second-order score is redundant and may lead to sub-optimal inference.

2.3 Connection to Analytic DDPM

The recent paper Bao et al. (2022) considers a constrained variational family $q_{\theta}(x|\tilde{x})=\operatorname{\mathcal{N}}(\mu_{\theta}(\tilde{x}),\sigma_{q}^{2}I)$ in the context of diffusion model and derive the optimal $\sigma_{q}^{*}$ as

$$\sigma_{q}^{*2} =\arg\min_{\sigma_{q}}{\mathrm{KL}}(p(\tilde{x}|x)p_{d}(x)\lVert q_{\theta}(x|\tilde{x}))\tilde{p}_{d}(\tilde{x}))=\frac{1}{d}\left\langle\mathrm{Tr}\left(\mathrm{Cov}_{q(x|\tilde{x})}[x]\right)\right\rangle_{\tilde{p}_{d}(\tilde{x})},$$ (16)

which can also be rewritten using the score function

$$\sigma_{q}^{*2}=\sigma^{2}-\sigma^{4}/d\left\langle\left\lVert s_{q_{\theta}}(\tilde{x})\right\rVert_{2}^{2}\right\rangle_{\tilde{p}_{d}(\tilde{x})}.$$ (17)

In Appendix B, we provide a detailed derivation to show how this approximation can be linked to our method using the Fisher information identity Fisher (1925). This approximation has two potential limitations: first, compared to full covariance moment matching, the assumed isotropic covariance structure may be insufficiently flexible to capture the true posterior; second, the covariance is independent of $\tilde{x}$.

(a) Gaussian mixtures visualizations

(b) Density of $p(x|\tilde{x}^{\prime})$ (blue) and four $\tilde{x}^{\prime}$ points (green)

(c) Analytical full covariance moment matching

(d) Analytical isotropic covariance moment matching

(e) Learned diagonal covariance

Figure 1: Figure (a) shows the clean data distribution $p_{d}(x)$ and the corresponding noisy distribution $\tilde{p}_{d}(\tilde{x})$. Figure (b) shows 4 conditioned samples in the noisy space. Figures (c, d, e) visualize the true posterior $p(x|\tilde{x})$ (green) and three posterior approximations (orange). We find that only the proposed $\tilde{x}$-dependent analytical full-covariance moment matching can capture the variance of the true posterior, whereas the other two methods underestimate the variance.

The second assumption only holds when $\mu(\tilde{x})$ is a linear function of $\tilde{x}$ (e.g. when $p_{d}(x)$ is Gaussian) and does not hold for other non-Gaussian $p_{d}(x)$. Therefore, our $\tilde{x}$-dependent full-covariance approximation offers a more versatile approximation family, which ultimately results in a more precise estimation. However, in certain applications such as accelerating the sampling procedure of a diffusion model Bao et al. (2022), it is advantageous to use a $\tilde{x}$-independent isotropic covariance due to its inexpensive estimation. On the other hand, our $\tilde{x}$-dependent covariance necessitates the computation of the Hessian for each $\tilde{x}$, making it inefficient for high-dimensional data. In Section 3, we will explore approaches to mitigate this limitation.

2.4 Posterior Approximation Comparison

We now consider a toy example to compare the three denoising posterior approximations discussed above. Let $p_{d}(x)$ be a Mixture of Gaussians (MoG) $p_{d}(x)=\frac{1}{4}\sum_{k=1}^{k=4}g_{k}(x)$ whose components $g_{[1:4]}$ are 2D Gaussians with means $[-1,-1],[-1,1],[1,1],[1,-1]$ and isotropic covariance $\sigma_{g}^{2}I$ with $\sigma_{g}=0.2$. The noise distribution is $p(\tilde{x}|x)=\mathcal{N}(x,\sigma^{2}I)$ with $\sigma=0.2$, so $\tilde{p}(\tilde{x})=\int p(x)p(\tilde{x}|x)\mathop{}\!\mathrm{d}{x}$ is an MoG with the same component means and diagonal covariance $(\sigma_{g}^{2}+\sigma^{2})I$; see Figure 0(a) for a visualization. In this case the true posterior $p(x|\tilde{x})$ does not allow a tractable form. Fortunately, given a noisy sample $\tilde{x}^{\prime}$ and an evaluation point $x^{\prime}$, we can evaluate the true density $p(x|\tilde{x}^{\prime})$ using Bayes rule: $p(x^{\prime}|\tilde{x}=\tilde{x}^{\prime})=p(\tilde{x}^{\prime}|x=x^{\prime})p_{d}(x^{\prime})/\tilde{p}_{d}(\tilde{x}^{\prime})$. Figure 1 shows the true posteriors given four different $\tilde{x}^{\prime}$ where we use grid data in $x$-space to visualize the density.

To train the model, we sample 10,000 data points from $p_{d}$ as our training data. For the KL-trained Gibbs sampler described in Section 2.2, we use a network with 3 hidden layers with 400 hidden units, Swish activation Ramachandran et al. (2017) and output size 4 to generate both mean and log standard deviation of the Gaussian approximation. For the moment-matching Gibbs sampler (including both full and isotropic covariance), we use the same network architecture but with output size 1 to get the scalar energy and DSM as the training objective. Both networks are trained with batch size 100 and Adam Kingma and Ba (2014) optimizer with learning rate $1{\times}10^{-4}$ for 100 epochs. For the $\tilde{x}$-independent isotropic covariance, we use the Monte Carlo approximation to estimate the variance Bao et al. (2022) with 10000 samples from $\tilde{p}_{d}(\tilde{x})$.

Table 1: MMD evaluations of a single chain

Data	Learn diag.	Analytic iso.	Analytic full
MoGs	$0.929\pm 0.343$	$0.724\pm 0.361$	$\textbf{0.305}\pm 0.141$
Rings	$0.364\pm 0.044$	$0.006\pm 0.002$	$\textbf{0.005}\pm 0.001$
Roll	$0.053\pm 0.011$	$0.030\pm 0.001$	$\textbf{0.016}\pm 0.002$

Figure 1 visualizes the approximations to the denoising posterior $p(x|\tilde{x})$ estimated by each of the three methods described in the previous sections. We surprisingly find that although the KL objective in Equation 12 encourages $q_{\theta}(x|\tilde{x})$ to match the moments of $p(x|\tilde{x})$, the learned covariance in Figure 0(e) still underestimates the variance of the posterior. This shows the redundancy of covariance learning can degrade the variational approximation performance. Additionally, the $\tilde{x}$-independent covariance fails to account for the relative positions of $\tilde{x}^{\prime}$ and lacks the ability to predict the posterior’s elliptical shape due to its isotropic nature. In contrast, our $\tilde{x}$-dependent full covariance approximation overcomes these limitations, enabling more accurate predictions that capture the intricate geometry of the posterior distribution.

(a) True data

(b) Learn. diag.

(c) Analytic iso.

(d) Analytic full

(e) True data

(f) Learn. diag.

(g) Analytic iso.

(h) Analytic full

(i) True data

(j) Learn. diag.

(k) Analytic iso.

(l) Analytic full

Figure 2: Samples from a single chain Gibbs sampling

We then use the estimated posterior to conduct (pseudo) Gibbs sampling to generate samples. Specifically, we initialize the first sample $x_{0}\sim\operatorname{\mathcal{N}}(0,0.1)$ and run one Markov Chain with 10,000 time steps to generate 10,000 samples. In addition to the mixture of Gaussian datasets, we also train and generate samples from the 2D Swiss roll and two-ring datasets. For numerical evaluation, we calculate the Maximum Mean Discrepancy (MMD) Gretton et al. (2012) between 10k samples generated by a single-chain Gibbs sampler and 10k samples from the training dataset respectively. The kernel insides MMD is a sum over 5 Gaussian kernels with bandwidth ranging over $[2^{-2},2^{-1},2^{0},2^{1},2^{2}]$. The MMD results (including both mean and std) are calculated using 5 random seeds. We find that Gibbs sampling with the proposed analytical full covariance achieves the best results; numerical results are in Table 1, with a visual comparison in Figure 2.

3 Scalable Implementations for Image Data 

Scalable Diagonal Hessian Approximation As we discussed in Section 2.3, the proposed full covariance Gaussian approximation in Equation 10 requires calculating an $D{\times}D$ Hessian $\nabla^{2}_{\tilde{x}}\log\tilde{q}(\tilde{x})$ for each $\tilde{x}$ with size $D$, which brings both memory and computation difficulties for high-dimensional data. A naive diagonal Hessian method (only using the diagonal entries in the Hessian) will address the memory bottleneck but still needs $D$ times backward passes for the exact computation of the diagonal term Martens et al. (2012). In this paper, we use the following diagonal Hessian approximation Bekas et al. (2007),

$$\mathrm{Diag}(H)\approx 1/S\sum\nolimits_{s=1}^{S}v_{s}\odot Hv_{s},$$ (18)

where $v_{s}\sim p(v)$ is a Rademacher random variable with entries $\pm 1$ and $\odot$ denotes the element-wise product. This estimator will converge to the exact Hessian diagonals when $S\rightarrow\infty$ Bekas et al. (2007). The computation for each $v_{s}$ can be computed by two forward-backward passes. It is worth emphasizing that our $\tilde{x}$-dependent diagonal moment matching approach provides a comparable level of flexibility to the variational method proposed in Bengio et al. (2013) while eliminating the need for additional training of the diagonal covariance. Furthermore, our method remains more flexible than the isotropic $\tilde{x}$-independent moment matching method proposed by Bao et al. (2022).

Energy or Score Parameterization For the full-covariance moment matching in Equation 10, we require $\nabla^{2}_{\tilde{x}}\log\tilde{q}_{\theta}(\tilde{x})$ to be symmetric to obtain a valid Gaussian approximation. However, if we learn the score function $\nabla_{\tilde{x}}\log p(\tilde{x})=s_{\theta}(\tilde{x})$ using a network $s_{\theta}(\cdot):{\mathbb{R}}^{D}\rightarrow{\mathbb{R}}^{D}$, its Jacobian is not guaranteed to be symmetric. In this case, we follow Saremi et al. (2018) and directly parameterize the density function $\tilde{q}_{\theta}(\cdot)$ with a neural network $f_{\theta}(\cdot):{\mathbb{R}}^{D}\rightarrow{\mathbb{R}}$ and let the score function $\nabla_{\tilde{x}}\log\tilde{q}_{\theta}(\tilde{x})=-\nabla_{x}f_{\theta}(\tilde{x})$. This can be obtained by AutoDiff packages like PyTorch Paszke et al. (2017), and this parameterization guarantees $\nabla^{2}_{\tilde{x}}f_{\theta}(\tilde{x})$ to be symmetric. We also notice that when using the diagonal Hessian approximation (Equation 18), we only need entries in $\mathrm{Diag}(H)$ to be positive in order to obtain a valid Gaussian approximation. In this case, the score parameterization remains applicable and offers more efficient training compared to the energy parameterization. Therefore, the combination of full/diagonal covariance and energy/score parameterization provides a tradeoff between flexibility and inference speed, allowing for a flexible approach while maintaining computational efficiency during training.

4 Image Generation with a Single Noise Level

(a) Diagonal $\tilde{x}$-dependent covariance moment matching (Ours)

(b) Isotropic $\tilde{x}$-independent covariance moment matching Bao et al. (2022)

(c) Diagonal covariance learned by KL minimization Bengio et al. (2013)

Figure 3: Figures (a,b,c) show the MNIST experiment comparisons, where we compare samples generated by pseudo-Gibbs sampling with three different $q(x|\tilde{x})$. We plot samples from 25 independent Markov Chains with $t\in\{0,1,5,10,20\}$ time steps. We can find the samples generated by the proposed analytical covariance moment matching with diagonal approximation achieved the best sample quality.

We then apply the proposed method to model the grey-scale MNIST LeCun (1998) dataset. We use the standard U-Net architecture Song and Ermon (2019); Ronneberger et al. (2015) with a single fixed noise level $\sigma=0.5$; the effect of varying $\sigma$ is explored in Appendix C.1. For the KL training objective, the output channel size is 2 to generate both mean and log-std at the same time. For DSM training, we take the sum of the U-Net output to obtain the scalar energy evaluation which also relates to the product-of-experts model described in Saremi et al. (2018). We train both networks for 300 epochs with learning rate $1{\times}10^{-4}$ and batch-size 100.

(a) $\tilde{x}$-dep diag (Ours)

(b) $\tilde{x}$-dep diag. (KL)

(c) $\tilde{x}$-ind iso.

Figure 4: Figures (a,b,c) visualize the covariance approximations $q(x|\tilde{x}=x+\sigma\epsilon),\epsilon\sim\mathcal{N}(0,\sigma^{2}I)$ on 25 $\tilde{x}$ samples. We use a sigmoid function to map the real value noise into grayscale pixels for the visualization.

As discussed in Section 1.1, the KL divergence is not well-defined for manifold data distributions.

(a) KL loss

(b) DSM loss

Figure 5: Training loss comparison of two objectives. We plot the training loss every iteration during a total 300 epochs.

This limitation becomes evident when working with MNIST, where the presence of constant black pixels in the boundary areas leads to a rapid decrease towards 0 in the variance of $q_{\theta}(x|\tilde{x})$ during training. Consequently, the likelihood value $\log q_{\theta}(x|\tilde{x})$ tends to approach infinity, resulting in unstable training. In contrast, the DSM objective is well-defined for manifold data, providing a stable training process even in the presence of such boundary effects. Figure 5 provides a visual comparison of the two training procedures, demonstrating the improved stability and effectiveness of the DSM objective in handling manifold data distributions.

For the sample generation process, calculating the full-covariance Gaussian posterior becomes challenging. We therefore apply the scalable diagonal Hessian approximation described in Section 3 to approximate the diagonal Gaussian covariance of $p(x|\tilde{x})$. We find that the estimated diagonal Hessian occasionally contains small negative values due to approximation error; we, therefore, use the $\max(\cdot,{\epsilon})$ function with ${\epsilon}>0$ to ensure the positivity of the diagonal covariance. The $\tilde{x}$-independent isotropic covariance and the proposed $\tilde{x}$-dependent diagonal covariance share the same mean function.

(a) Data $x$

(b) $1$ sample

(c) $10$ samples

(d) $100$ samples

Figure 6: Visualizations of the diagonal covariance $q(x|\tilde{x}=x+\sigma\epsilon)$ with different number of Rademacher samples.

Figure 7: Samples from three Markov chains. We plot the samples every 10 Gibbs steps.

We first visualize the covariance estimated by three different methods in Figure 4. We use 100 Rademacher samples in estimating the diagonal Hessian (Equation 18) and 50,000 samples in estimating the isotropic variance (Equation 17). We find that both $\tilde{x}$-dependent diagonal covariance approximations can capture the posterior structure whereas the isotropic $\tilde{x}$-independent covariance is just Gaussian noise since the variance is shared between different digit and pixel locations. In Figure 3, we plot the sample comparison for three methods.

Since the isotropic covariance has the same variance in each dimension, the generated samples in Figure 2(b) contain white noise in the black background, whereas the proposed full-covariance sampler can generate a clean black background in Figure 2(a). On the other hand, the samples generated by the KL-trained Gibbs sampler (Figure 2(c)) have worse sample quality due to the unstable training.

We then apply the same method to model the more complicated CIFAR 10 Krizhevsky et al. (2009) dataset. We use the same U-Net structure as used in Song and Ermon (2020) and directly parameterize the score function rather than the energy function to speed up the training. The noise level is fixed at $0.3$. We train the model using Adam optimizer with learning rate $1{\times}10^{-4}$ and batch size 100 for 1000 epochs. We visualize the denoising posterior diagonal covariance in Figure 6 when using different numbers of Rademacher samples (Equation 18). We observe that better covariance estimation can be obtained by increasing the number of samples. To balance efficiency and accuracy, we use a sample number of 10 in the subsequent Gibbs sampling stage. Figure 7 shows three independent Markov chains with the samples plotted every 10 Gibbs steps, which demonstrates that sharp images can be generated with even one fixed level of noise.

Figure 8: FID evaluation with Increased Gibbs Steps. We can find the FID increases after 40 Gibbs steps.

Limitation: In the CIFAR experiment, we observe a mode collapse phenomenon when running multiple independent Markov chains for a longer time. This phenomenon is likely due to the small noise level $\sigma=0.3$, which prevents the sampler from exploring the full space, as commonly found with MCMC methods Robert et al. (1999). This effect is visually represented in Figure 8, where we assess the Fréchet Inception Distance (FID) values for 50,000 images sampled with varying numbers of Gibbs steps. Notably, the FID increases beyond 40 Gibbs steps, and visual evidence of mode collapse is observed (Figure 12). In the ensuing section, we will demonstrate the application of our method to settings with multiple noise levels, an approach that may help mitigate the issue of mode collapse.

5 Image Generation Using Multiple Noise Levels

Algorithm 1 Sampling with Langevin Dynamics

$\{\sigma_{t}\}_{t=0}^{T},\delta,K$

Initialize $x^{0}_{T}$

for $t\leftarrow T$ to $1$  do

     $\alpha_{t}\leftarrow\delta\sigma_{t}^{2}/\sigma_{0}^{2}$

     for $k\leftarrow 1$ to $K$ do

         Draw $z^{k}\sim\mathcal{N}(0,I)$

         $x^{k}_{t}{\leftarrow}x^{k-1}_{t}+\alpha_{t}s_{\theta}(x_{t}^{k-1},\sigma_{t})+\sqrt{2\alpha_{t}}z^{k}$

     end for

     $x_{t-1}^{0}\leftarrow x_{t}^{K}$

end for

return $x^{0}_{0}+\sigma^{2}_{0}s_{\theta}(x^{0}_{0},\sigma_{0})$

Algorithm 2 Sampling with the proposed pseudo Gibbs Sampling

$\{\sigma_{t}\}_{t=0}^{T},K$

Initialize $x^{0}_{T}$

for $t\leftarrow T$ to $1$  do

     for $k\leftarrow 1$ to $K$ do

         Draw $x^{k}_{t+1}\sim p(x_{t+1}|x_{t}=x^{k}_{t})$

         Draw $x^{k}_{t}\sim q_{\theta}(x_{t}|x_{t+1}=x^{k}_{t+1})$

     end for

     $x_{t-1}^{0}\leftarrow x_{t}^{K}$

end for

return $x^{0}_{0}+\sigma^{2}_{0}s_{\theta}(x^{0}_{0},\sigma_{0})$

The success of diffusion models and lessons from prior work on score-based generative models point to the importance of using multiple noise levels Ho et al. (2020); Song and Ermon (2019) when modelling data with complex multi-modal distributions. Intuitively, by learning to denoise data at a range of noise levels, a single network can learn both the fine and global structure of the distribution, which in turn allows for more effective sampling algorithms capable of efficiently exploring diverse modes (Song and Ermon, 2019). We therefore propose to adapt the denoising Gibbs sampling procedure to sample from distributions corrupted with multiple noise levels. For this purpose, we use a noise-conditioned score network trained by Song and Ermon (2020), who generated high-quality samples using a procedure inspired by annealed Langevin dynamics Welling and Teh (2011). This procedure involves generating samples from a sequence of distributions $p_{T}(x_{T}),...,p_{0}(x_{0})$, corrupted by progressively decreasing levels of Gaussian noise (parameterized via standard deviations $\sigma_{t}$, with $\sigma_{T}>,\cdots,>\sigma_{0}$). At a given step $t$ in the sequence, Langevin dynamics is used to sample from the corresponding noised distribution $p_{t}(x_{t})$, using the score network $s_{\theta}(x_{t},\sigma_{t})$ to approximate the gradient of the noised distribution. The outputs of this Langevin dynamics run are then used to initialize the same procedure at the next noise level, leading the sampling procedure to converge gradually towards the data distribution as the noise level tends to zero (i.e. $p_{0}(x_{0})\approx p_{d}(x),\sigma_{0}\approx 0$). The algorithm of the annealed Langevin dynamics with multi-level noise used in Song and Ermon (2019, 2020) is summarized in Algorithm 1.

We show that the proposed Gibbs sampling scheme can be directly applied to a pre-trained score-based generative model as a drop-in replacement for Langevin dynamics MCMC in the generation stage. At each noise level, we use samples $x_{t+1}$ from the previous noise level to initialise a Gibbs sampling chain targeting the marginal distribution at the current noise level $p_{t}(x_{t})$, which now plays the role of the ‘clean’ distribution in Equation 5. Therefore, the noisy distribution at time step $t$ is a Gaussian $p(x_{t+1}|x_{t})=\operatorname{\mathcal{N}}(x_{t},\sigma_{t+1}^{2}-\sigma_{t}^{2})$. The optimal denoising distribution $q(x_{t}|x_{t+1})$ is thus a function of the level of noise at step $t$ relative to the level of noise at the previous step $t+1$. For generation efficiency, we employ 3 Gibbs steps at each noise level, using 3 Rademacher samples to approximate the diagonal Hessian. The sampling procedure is summarized in Algorithm 2. In Figure 9 and 10, we visualize the samples from models that are trained on CIFAR10 and CelebA separately. Further experimental details can be found in Appendix C.2.

Table 2: CIFAR10 Inception and FID Scores

Model	Inception	FID
NCSNv2 (Langevin  (Song and Ermon, 2020))	$\mathbf{8.40\pm 0.07}$	$10.87$
NCSNv2 (Gibbs, Ours)	$8.28\pm 0.07$	$\mathbf{10.75}$
DDPM (Ho et al., 2020)	$9.46\pm 0.11$	3.17
NCSN++ (Song et al., 2021)	$9.89$	$2.20$

For direct comparison with the results of (Song and Ermon, 2020) on CIFAR10, we retain the same schedule of noise levels used to generate samples with Langevin dynamics. We generate 50000 samples using this approach and report FID and Inception scores in Table 2. Our multi-level Gibbs sampling scheme produces samples of equivalent quality to the multi-level Langevin dynamics of (Song and Ermon, 2020), confirming its applicability to complex natural image data. The FID is also notably superior to that of the single-noise level Gibbs sampling, and the samples exhibit significant visual diversity (Figure 9). This underlines the importance of employing multi-level noise in our approach. Recent advances in sampling strategies for score-based models leveraging the framework of stochastic differential equations (Song et al., 2021) have led to significant further improvements in generation quality as shown in Table 2; we leave the exploration of possible applications of our method to this framework to future work.

Figure 9: CIFAR 10 Samples

Figure 10: CelebA Samples

6 Conclusion

This paper focuses on addressing the inconsistency problem in training energy-based models (EBMs) using denoising score matching. Specifically, we identify the presence of an underlying clean model within a ‘noisy’ EBM and propose an efficient sampling scheme for the clean model. We demonstrate how this method can be effectively applied to high-dimensional data and showcase image generation results in both single and multi-level noise settings. More broadly, we hope our more accurate denoising posterior opens new avenues for future work on score-based methods in machine learning.

Appendix A Proof and Derivations

A.1 Proof of Theorem 2.1

The existence is straightforward, since ${\mathrm{FD}}(\tilde{p}_{d}||\tilde{q}_{\theta^{*}})=0\rightarrow\tilde{p}_{d}=\tilde{q}_{\theta^{*}}$, we can simply let $q(x)=p_{d}(x)$, which makes $\int q(x)p(\tilde{x}|x)\mathop{}\!\mathrm{d}{x}=\int p_{d}(x)p(\tilde{x}|x)\mathop{}\!\mathrm{d}{x}=\tilde{p}_{d}$. To show the uniqueness, we denote density $k({\epsilon})=\operatorname{\mathcal{N}}(0,\sigma^{2}I)$, so $\tilde{q}_{\theta}(\tilde{x})$ and $\tilde{p}_{d}(x)$ can be written as convolutions

$$\tilde{q}_{\theta}(\tilde{x})=q*k,\quad\tilde{p}_{d}(\tilde{x})=p_{d}*k,$$ (19)

we then have

$$\tilde{p}_{d}=\tilde{q}_{\theta} \Leftrightarrow q*k=p_{d}*k\Leftrightarrow\mathcal{F}(q)\mathcal{F}(k)=\mathcal{F}(p_{d})\mathcal{F}(k),$$ (20)

where $\mathcal{F}$ denotes the Fourier transform. Since the Fourier transform of a Gaussian is also a Gaussian, so $\mathcal{F}(k)>0$ everywhere, we have

$$\tilde{p}_{d}=\tilde{q}_{\theta^{*}} \Leftrightarrow\mathcal{F}(q)\cancel{\mathcal{F}(k)}=\mathcal{F}(p_{d})\cancel{\mathcal{F}(k)}\Leftrightarrow\mathcal{F}(q)=\mathcal{F}(p_{d})\Leftrightarrow q=p_{d}.$$ (21)

Therefore, $q=p_{d}$ is the unique distribution that makes $\tilde{p}_{d}=\tilde{q}_{\theta}$. This technique has also been used to construct spread KL divergence (we denote as $\widetilde{{\mathrm{KL}}}$)  Zhang et al. [2020], which is defined as $\widetilde{{\mathrm{KL}}}(p_{d}||q_{\theta})\equiv\mathrm{KL}(p_{d}*k||q_{\theta}*k)$ where $k(\epsilon)=N(0,\sigma^{2}I)$, to train implicit model $q_{\theta}$. Different from the DSM situation, when $\widetilde{{\mathrm{KL}}}(p_{d}||q_{\theta})=0$, the underlying model $q_{\theta}=p_{d}$ is directly available, whereas the EBM $\tilde{q}_{\theta}$ trained by DSM learns to be the noisy distribution $\tilde{q}_{\theta}=p_{d}*k$.

A.2 General Conditions Characterising the Existence of the Clean Model

In the previous section, we assume for a flexible neural network parameterized $f_{\theta}$, the energy-based model $\tilde{q}_{\theta}(\tilde{x})=\exp(-f(\tilde{x}))/Z(\theta)$ trained by Equation 4 can recover the target noisy data distribution $\tilde{q}_{\theta^{*}}=\tilde{p}_{d}$ so there exists an underlying model $q$ such that $\tilde{q}_{\theta^{*}}=q*k$ and $q=p_{d}$. This assumption is commonly used in the literature on score-based methods. For example, in the score-based diffusion models literature Song and Ermon [2019], Ho et al. [2020], Bao et al. [2022], for any data $x\in{\mathbb{R}}^{D}$, the score function $\nabla_{\tilde{x}}\log\tilde{q}_{\theta}(\tilde{x})$ is usually parameterized by a neural network $\mathrm{NN}_{\theta}(\cdot):{\mathbb{R}}^{D}\rightarrow{\mathbb{R}}^{D}$. However, this parameterization cannot guarantee $\mathrm{NN}_{\theta}(\tilde{x})$ is a conservative vector field, or in other words, there doesn’t exist a distribution $\tilde{q}_{\theta}(\tilde{x})$ such that $\nabla_{\tilde{x}}\tilde{q}_{\theta}(\tilde{x})=\nabla_{\tilde{x}}\log\tilde{q}_{\theta}(\tilde{x})$ and $\nabla^{2}_{\tilde{x}}\log\tilde{q}(\tilde{x})$ is symmetric Salimans and Ho [2021], Saremi [2019]. Therefore, perfect score estimation $\nabla_{\tilde{x}}\log\tilde{p}_{d}(\tilde{x})=\nabla_{\tilde{x}}\log\tilde{q}_{\theta}(\tilde{x})$ is implicitly assumed to allow an EBM interpretation.

However, the underlying clean model doesn’t always exist for imperfect model $\tilde{q}_{\theta}\neq\tilde{p}_{d}$. We here provide the sufficient and necessary conditions which guarantee the existence of the underlying clean model.

Theorem A.1 (Necessary and Sufficient conditions for the existence of the underlying clean model.).

For a model $\tilde{q}_{\theta}$ with the convolutional noise distribution $k(\epsilon)=\operatorname{\mathcal{N}}(0,\sigma^{2}I)$, there exists an underlying model $q$ such that $q*k=\tilde{q}$ if and only if $\mathcal{F}(\tilde{q}_{\theta})/\mathcal{F}(k)$ is positive semi-definite . Additionally, the underlying distribution $q$ can be written as

$$q=\mathcal{F}^{-1}(\mathcal{F}(\tilde{q}_{\theta})/\mathcal{F}(k)),$$ (22)

where $\mathcal{F}^{-1}$ is the inverse Fourier transform. This theorem is a straightforward corollary of Bochner’s Theorem . However, for the energy model $\tilde{q}_{\theta}(\tilde{x})\propto\exp{(-f_{\theta}(\tilde{x}))}$, it’s difficult to design a functioning family of $f$ that satisfies the positive semi-definite condition and have the tractable score function at the same time . We thus leave the design of better energy function parameterizations as a promising future direction.

A.3 Proof of Theorem 2.2

Derivation of the Mean Identity

We let $\tilde{q}_{\theta}(\tilde{x})=\int k(\tilde{x}|x)q_{\theta}(x)\mathop{}\!\mathrm{d}{\tilde{x}}$, where $k(\tilde{x}|x)=\mathcal{N}(x,\sigma^{2}I)$, we have

$$\nabla_{\tilde{x}}\log\tilde{q}_{\theta}(\tilde{x}) =\frac{\nabla_{\tilde{x}}\tilde{q}_{\theta}(\tilde{x})}{\tilde{q}_{\theta}(\tilde{x})}=\frac{\int\nabla_{\tilde{x}}k(\tilde{x}|x)q_{\theta}(x)\mathop{}\!\mathrm{d}{x}}{\tilde{q}_{\theta}(\tilde{x})}$$

$$=-\frac{1}{\sigma^{2}}\int\left((\tilde{x}-x)\frac{k(\tilde{x}|x)q_{\theta}(x)}{\tilde{q}_{\theta}(\tilde{x})}\right)\mathop{}\!\mathrm{d}{x}$$

$$\Longrightarrow\sigma^{2}\nabla_{\tilde{x}}\log\tilde{q}_{\theta}(\tilde{x})+\tilde{x} =\int x\frac{k(\tilde{x}|x)q_{\theta}(x)}{\tilde{q}_{\theta}(\tilde{x})}\mathop{}\!\mathrm{d}{x}=\langle x\rangle_{q_{\theta}(x|\tilde{x})}$$

where we define the model denoising posterior using Bayes rule $q_{\theta}(x|\tilde{x})\equiv k(\tilde{x}|x)q_{\theta}(x)/\tilde{q}_{\theta}(\tilde{x})$. The second equality is due to the following Gaussian distribution property

$$\nabla_{\tilde{x}}k(\tilde{x}|x)=\frac{1}{\sqrt{2\pi\sigma^{2}}}\nabla_{\tilde{x}}e^{\frac{-(\tilde{x}-x)^{2}}{2\sigma^{2}}}=-\frac{\tilde{x}-x}{\sigma^{2}}\frac{1}{\sqrt{2\pi\sigma^{2}}}e^{\frac{-(\tilde{x}-x)^{2}}{2\sigma^{2}}}=-\frac{\tilde{x}-x}{\sigma^{2}}k(\tilde{x}|x).$$ (23)

Derivations of the Analytical Full Covariance Identity

We have derived the mean identity

$$\mu_{q}(\tilde{x})\equiv\langle x\rangle_{q_{\theta}(x|\tilde{x})}=\sigma^{2}\nabla_{\tilde{x}}\log\tilde{q}_{\theta}(\tilde{x})+\tilde{x}.$$ (24)

Taking the gradient over $x$ in both side and scaling with $\sigma^{2}$, we have

$$\sigma^{2}\nabla_{\tilde{x}}\mu_{q}(\tilde{x})=\sigma^{4}\nabla_{\tilde{x}}^{2}\log\tilde{q}_{\theta}(\tilde{x})+\sigma^{2}I.$$ (25)

We can also expand the hessian of the $\log\tilde{q}_{\theta}(\tilde{x})$:

$$\nabla^{2}_{\tilde{x}}\log\tilde{q}_{\theta}(\tilde{x}) =-\frac{1}{\sigma^{2}}\int\nabla_{\tilde{x}}\left((\tilde{x}-x)\frac{k(\tilde{x}|x)q_{\theta}(x)}{\tilde{q}_{\theta}(\tilde{x})}\right)\mathop{}\!\mathrm{d}{x}$$

$$=-\frac{1}{\sigma^{2}}\int\frac{k(\tilde{x}|x)q_{\theta}(x)}{\tilde{q}_{\theta}(\tilde{x})}\mathop{}\!\mathrm{d}{x} +\frac{1}{\sigma^{2}}\int(\tilde{x}-x)\frac{\nabla_{\tilde{x}}k(\tilde{x}|x)\tilde{q}_{\theta}(\tilde{x})q_{\theta}(x)-\nabla_{\tilde{x}}\tilde{q}_{\theta}(\tilde{x})k(\tilde{x}|x)q_{\theta}(x)}{\tilde{q}^{2}_{\theta}(\tilde{x})}\mathop{}\!\mathrm{d}{x}$$

$$\Longrightarrow\sigma^{2}\nabla_{\tilde{x}}^{2}\log\tilde{q}_{\theta}(\tilde{x})+1 =\int(\tilde{x}-x)\frac{\nabla_{\tilde{x}}k(\tilde{x}|x)q_{\theta}(x)-\nabla_{\tilde{x}}\log\tilde{q}_{\theta}(\tilde{x})k(\tilde{x}|x)q_{\theta}(x)}{\tilde{q}_{\theta}(\tilde{x})}\mathop{}\!\mathrm{d}{x}$$

$$=\int(\tilde{x}-x) \frac{-\frac{1}{\sigma^{2}}(\tilde{x}-x)k(\tilde{x}|x)q_{\theta}(x)+\frac{1}{\sigma^{2}}(\tilde{x}-\langle x\rangle_{q_{\theta}(x|\tilde{x})})k(\tilde{x}|x)q_{\theta}(x)}{\tilde{q}_{\theta}(\tilde{x})}\mathop{}\!\mathrm{d}{x}$$

$$\Longrightarrow\sigma^{4}\nabla_{\tilde{x}}^{2}\log\tilde{q}_{\theta}(\tilde{x})+\sigma^{2}I =\int\left(-(\tilde{x}-x)^{2}+(\tilde{x}-x)(\tilde{x}-\langle x\rangle_{q_{\theta}(x|\tilde{x})})\right)q_{\theta}(x|\tilde{x})\mathop{}\!\mathrm{d}{x}$$

$$=\langle x^{2}\rangle_{q_{\theta}(x|\tilde{x})}-\langle x\rangle^{2}_{q_{\theta}(x|\tilde{x})}\equiv\Sigma_{q}(\tilde{x})$$

Therefore, we obtain the analytical full covariance identity.

$$\Sigma_{q}(\tilde{x})=\sigma^{2}\nabla_{\tilde{x}}\mu_{q}(\tilde{x}).$$ (26)

A.4 Proof of Theorem 2.3

Lemma A.2 (KL to Gaussian Bao et al. [2022]).

Let $p(x)$ be a distribution with mean $\mu_{p}$ and covariance $\Sigma_{p}$ and $q(x)=\operatorname{\mathcal{N}}(\mu_{q},\Sigma_{q})$, denote the differential entropy as $\mathrm{H}(p)\equiv-\int p(x)\log p(x)\mathop{}\!\mathrm{d}{x}$, we have

$${\mathrm{KL}}(p||q)={\mathrm{KL}}(\operatorname{\mathcal{N}}(\mu_{p},\Sigma_{p})||q)+\mathrm{H}(\operatorname{\mathcal{N}}(\mu_{p},\Sigma_{p}))-\mathrm{H}(p)$$ (27)

The proof can be found in Bao et al. [2022] Lemma 2.

We can then prove Theorem 2.3. Since $p(\tilde{x}|x)p_{d}(x)=p(x|\tilde{x})\tilde{p}_{d}(\tilde{x})$, where $\tilde{p}_{d}(\tilde{x})=\int p_{d}(x)p(\tilde{x}|x)\mathop{}\!\mathrm{d}{x}$, we have

$${\mathrm{KL}}(p(\tilde{x}|x)p_{d}(x)\lVert q(x|\tilde{x})\tilde{p}_{d}(\tilde{x})) =\left\langle{\mathrm{KL}}(p(x|\tilde{x})||q(x|\tilde{x}))\right\rangle_{\tilde{p}(\tilde{x})}$$ (28)

Assume Gaussian distribution $q(x|\tilde{x})=\operatorname{\mathcal{N}}(\mu_{q}(\tilde{x}),\Sigma_{q}(\tilde{x}))$and denote the mean and covariance of the true posterior are $\mu_{p}(\tilde{x})$ and $\Sigma_{p}(\tilde{x})$, then the optimal $q^{*}$ is

$$q^{*} =\arg\min_{q}{\mathrm{KL}}(p(\tilde{x}|x)p_{d}(x)\lVert q(x|\tilde{x})\tilde{p}_{d}(\tilde{x}))$$ (29)

$$=\arg\min_{q}\Big{\langle}{\mathrm{KL}}(p(x|\tilde{x})||q(x|\tilde{x}))\Big{\rangle}_{\tilde{p}(\tilde{x})}$$ (30)

$$=\arg\min_{q}\Big{\langle}{\mathrm{KL}}(\operatorname{\mathcal{N}}(\mu_{p},\Sigma_{p})||q(x|\tilde{x}))+\mathrm{H}(\operatorname{\mathcal{N}}(\mu_{p},\Sigma_{p}))-\mathrm{H}(p(x|\tilde{x}))\Big{\rangle}_{\tilde{p}(\tilde{x})}$$ (31)

$$=\arg\min_{q}\Big{\langle}{\mathrm{KL}}(\operatorname{\mathcal{N}}(\mu_{p},\Sigma_{p})||q(x|\tilde{x}))\Big{\rangle}_{\tilde{p}(\tilde{x})}+const..$$ (32)

Therefore, the optimal $q(x|\tilde{x})=\operatorname{\mathcal{N}}(\mu_{q}(\tilde{x}),\Sigma_{q}(\tilde{x}))$ under the joint KL has the mean and covariance $\mu^{*}_{q}(\tilde{x})=\mu_{p}(\tilde{x}),\Sigma^{*}_{q}(\tilde{x}))=\Sigma_{p}(\tilde{x})$.

Appendix B Connection to Analytical DDPM

Paper Bao et al. [2022] considers the constrained variational family $q_{\theta}(x|\tilde{x})=\operatorname{\mathcal{N}}(\mu_{\theta}(\tilde{x}),\sigma_{q}^{2}I)$ and derive the optimal $\sigma_{q}^{*}$ as

$$\sigma_{q}^{*2} =\arg\min_{\sigma_{q}}{\mathrm{KL}}(p(\tilde{x}|x)p_{d}(x)\lVert q_{\theta}(x|y))\tilde{p}_{d}(\tilde{x}))=\frac{1}{d}\left\langle\mathrm{Tr}\left(\mathrm{Cov}_{q(x|\tilde{x})}[x]\right)\right\rangle_{\tilde{p}_{d}(\tilde{x})},$$ (33)

which can also be rewritten using the score function

$$\sigma_{q}^{*2}=\sigma^{2}-\frac{\sigma^{4}}{d}\left\langle\left\lVert s_{q_{\theta}}(\tilde{x})\right\rVert_{2}^{2}\right\rangle_{\tilde{p}_{d}(\tilde{x})}.$$ (34)

To make a deep connection, we can also plug our analytical full covariance (Equation 10) into Equation 16

$$\sigma_{q}^{*2} =\sigma^{2}+\frac{\sigma^{4}}{d}\mathrm{Tr}\Big{\langle}\nabla_{x}^{2}\log q_{\theta}(\tilde{x})\Big{\rangle}_{\tilde{p}_{d}(\tilde{x})}$$

$$=\sigma^{2}-\frac{\sigma^{4}}{d}\mathrm{Tr}\left\langle s_{q_{\theta}}(\tilde{x})s_{q_{\theta}}(\tilde{x})^{T}\right\rangle_{\tilde{p}_{d}(\tilde{x})}=\sigma^{2}-\frac{\sigma^{4}}{d}\left\langle\left\lVert s_{q_{\theta}}(\tilde{x})\right\rVert_{2}^{2}\right\rangle_{\tilde{p}_{d}(\tilde{x})},$$ (35)

which recovers Equation 17, where the first equality is due to the well-known Fisher information identity Fisher [1925].

Appendix C Experiments

All the experiments conducted in this paper are run on one single NVDIA GTX 3090.

C.1 Effect of the Single Noise Choice on MNIST

Figure 11 shows the samples generated by our method with the EBM trained with difference $\sigma\in\{0.3,0.5,0.8\}$ in the noise distribution $p(\tilde{x}|x)$, we can find the image quality also heavily depends on the choice of the noise scale and $\sigma=0.5$ achieves the best visual quality, we then use this hyper-parameter in the subsequent comparisons.

(a) $\sigma=0.3$

(b) $\sigma=0.5$

(c) $\sigma=0.8$

Figure 11: Sample comparisons with different $\sigma$ value.

Figure 12: Mode Collapse visualization of 25 Markov chains, we plot the samples every 20 Gibbs steps, we can find less modes are covered if we run the Gibbs sampling for a longer time.

C.2 Multi-level Noise Details

For full details on the architecture and noise schedule used in the multi-level noise experiments in Section 5, we refer to Appendix B of [Song and Ermon, 2020]. For our multi-level Gibbs sampling procedure, we used 3 Gibbs steps at each noise level and 3 Rademacher samples for each diagonal Hessian computation. Following [Song and Ermon, 2020], we used a total of 232 noise levels, distributed according to their proposed geometric schedule, and applied a final denoising step in which the mean of the clean distribution conditioned on the final output of the sampling procedure is returned (the final output of the sampling procedure is a sample from the noised distribution from the noise distribution at the smallest noise level). This denoising step was previously found to improve FID scores [Jolicoeur-Martineau et al., 2021] significantly.