-
Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMC
Authors:
Yilun Du,
Conor Durkan,
Robin Strudel,
Joshua B. Tenenbaum,
Sander Dieleman,
Rob Fergus,
Jascha Sohl-Dickstein,
Arnaud Doucet,
Will Grathwohl
Abstract:
Since their introduction, diffusion models have quickly become the prevailing approach to generative modeling in many domains. They can be interpreted as learning the gradients of a time-varying sequence of log-probability density functions. This interpretation has motivated classifier-based and classifier-free guidance as methods for post-hoc control of diffusion models. In this work, we build up…
▽ More
Since their introduction, diffusion models have quickly become the prevailing approach to generative modeling in many domains. They can be interpreted as learning the gradients of a time-varying sequence of log-probability density functions. This interpretation has motivated classifier-based and classifier-free guidance as methods for post-hoc control of diffusion models. In this work, we build upon these ideas using the score-based interpretation of diffusion models, and explore alternative ways to condition, modify, and reuse diffusion models for tasks involving compositional generation and guidance. In particular, we investigate why certain types of composition fail using current techniques and present a number of solutions. We conclude that the sampler (not the model) is responsible for this failure and propose new samplers, inspired by MCMC, which enable successful compositional generation. Further, we propose an energy-based parameterization of diffusion models which enables the use of new compositional operators and more sophisticated, Metropolis-corrected samplers. Intriguingly we find these samplers lead to notable improvements in compositional generation across a wide set of problems such as classifier-guided ImageNet modeling and compositional text-to-image generation.
△ Less
Submitted 18 November, 2023; v1 submitted 22 February, 2023;
originally announced February 2023.
-
Continuous diffusion for categorical data
Authors:
Sander Dieleman,
Laurent Sartran,
Arman Roshannai,
Nikolay Savinov,
Yaroslav Ganin,
Pierre H. Richemond,
Arnaud Doucet,
Robin Strudel,
Chris Dyer,
Conor Durkan,
Curtis Hawthorne,
Rémi Leblond,
Will Grathwohl,
Jonas Adler
Abstract:
Diffusion models have quickly become the go-to paradigm for generative modelling of perceptual signals (such as images and sound) through iterative refinement. Their success hinges on the fact that the underlying physical phenomena are continuous. For inherently discrete and categorical data such as language, various diffusion-inspired alternatives have been proposed. However, the continuous natur…
▽ More
Diffusion models have quickly become the go-to paradigm for generative modelling of perceptual signals (such as images and sound) through iterative refinement. Their success hinges on the fact that the underlying physical phenomena are continuous. For inherently discrete and categorical data such as language, various diffusion-inspired alternatives have been proposed. However, the continuous nature of diffusion models conveys many benefits, and in this work we endeavour to preserve it. We propose CDCD, a framework for modelling categorical data with diffusion models that are continuous both in time and input space. We demonstrate its efficacy on several language modelling tasks.
△ Less
Submitted 15 December, 2022; v1 submitted 28 November, 2022;
originally announced November 2022.
-
Ferrotronics for the creation of band gaps in Graphene
Authors:
Qifang Wan,
Zhuocong Xiao,
Ahmed Kursumovic,
Judith. L. MacManus-Driscoll,
Colm Durkan
Abstract:
We experimentally demonstrate a simple graphene/ ferrolectric device, termed Ferrotronic (electronic effect from ferroelectric) device in which the band-structure of single-layer graphene is modified. The device architecture consists of graphene deposited on a ferroelectric substrate which encodes a periodic surface potential achieved through domain engineering. This structure takes advantage of t…
▽ More
We experimentally demonstrate a simple graphene/ ferrolectric device, termed Ferrotronic (electronic effect from ferroelectric) device in which the band-structure of single-layer graphene is modified. The device architecture consists of graphene deposited on a ferroelectric substrate which encodes a periodic surface potential achieved through domain engineering. This structure takes advantage of the nature of conduction through graphene to modulate the Fermi velocity of the charge carriers by the variations in surface potential, leading to the emergence of energy mini-bands and a band gap at the superlattice Brillouin zone boundary. Our work represents a simple route to building circuits whose functionality is controlled by the underlying substrate.
△ Less
Submitted 14 December, 2021;
originally announced December 2021.
-
Maximum Likelihood Training of Score-Based Diffusion Models
Authors:
Yang Song,
Conor Durkan,
Iain Murray,
Stefano Ermon
Abstract:
Score-based diffusion models synthesize samples by reversing a stochastic process that diffuses data to noise, and are trained by minimizing a weighted combination of score matching losses. The log-likelihood of score-based diffusion models can be tractably computed through a connection to continuous normalizing flows, but log-likelihood is not directly optimized by the weighted combination of sco…
▽ More
Score-based diffusion models synthesize samples by reversing a stochastic process that diffuses data to noise, and are trained by minimizing a weighted combination of score matching losses. The log-likelihood of score-based diffusion models can be tractably computed through a connection to continuous normalizing flows, but log-likelihood is not directly optimized by the weighted combination of score matching losses. We show that for a specific weighting scheme, the objective upper bounds the negative log-likelihood, thus enabling approximate maximum likelihood training of score-based diffusion models. We empirically observe that maximum likelihood training consistently improves the likelihood of score-based diffusion models across multiple datasets, stochastic processes, and model architectures. Our best models achieve negative log-likelihoods of 2.83 and 3.76 bits/dim on CIFAR-10 and ImageNet 32x32 without any data augmentation, on a par with state-of-the-art autoregressive models on these tasks.
△ Less
Submitted 20 October, 2021; v1 submitted 22 January, 2021;
originally announced January 2021.
-
SBI -- A toolkit for simulation-based inference
Authors:
Alvaro Tejero-Cantero,
Jan Boelts,
Michael Deistler,
Jan-Matthis Lueckmann,
Conor Durkan,
Pedro J. Gonçalves,
David S. Greenberg,
Jakob H. Macke
Abstract:
Scientists and engineers employ stochastic numerical simulators to model empirically observed phenomena. In contrast to purely statistical models, simulators express scientific principles that provide powerful inductive biases, improve generalization to new data or scenarios and allow for fewer, more interpretable and domain-relevant parameters. Despite these advantages, tuning a simulator's param…
▽ More
Scientists and engineers employ stochastic numerical simulators to model empirically observed phenomena. In contrast to purely statistical models, simulators express scientific principles that provide powerful inductive biases, improve generalization to new data or scenarios and allow for fewer, more interpretable and domain-relevant parameters. Despite these advantages, tuning a simulator's parameters so that its outputs match data is challenging. Simulation-based inference (SBI) seeks to identify parameter sets that a) are compatible with prior knowledge and b) match empirical observations. Importantly, SBI does not seek to recover a single 'best' data-compatible parameter set, but rather to identify all high probability regions of parameter space that explain observed data, and thereby to quantify parameter uncertainty. In Bayesian terminology, SBI aims to retrieve the posterior distribution over the parameters of interest. In contrast to conventional Bayesian inference, SBI is also applicable when one can run model simulations, but no formula or algorithm exists for evaluating the probability of data given parameters, i.e. the likelihood. We present $\texttt{sbi}$, a PyTorch-based package that implements SBI algorithms based on neural networks. $\texttt{sbi}$ facilitates inference on black-box simulators for practising scientists and engineers by providing a unified interface to state-of-the-art algorithms together with documentation and tutorials.
△ Less
Submitted 22 July, 2020; v1 submitted 17 July, 2020;
originally announced July 2020.
-
On Contrastive Learning for Likelihood-free Inference
Authors:
Conor Durkan,
Iain Murray,
George Papamakarios
Abstract:
Likelihood-free methods perform parameter inference in stochastic simulator models where evaluating the likelihood is intractable but sampling synthetic data is possible. One class of methods for this likelihood-free problem uses a classifier to distinguish between pairs of parameter-observation samples generated using the simulator and pairs sampled from some reference distribution, which implici…
▽ More
Likelihood-free methods perform parameter inference in stochastic simulator models where evaluating the likelihood is intractable but sampling synthetic data is possible. One class of methods for this likelihood-free problem uses a classifier to distinguish between pairs of parameter-observation samples generated using the simulator and pairs sampled from some reference distribution, which implicitly learns a density ratio proportional to the likelihood. Another popular class of methods fits a conditional distribution to the parameter posterior directly, and a particular recent variant allows for the use of flexible neural density estimators for this task. In this work, we show that both of these approaches can be unified under a general contrastive learning scheme, and clarify how they should be run and compared.
△ Less
Submitted 18 December, 2020; v1 submitted 10 February, 2020;
originally announced February 2020.
-
Neural Spline Flows
Authors:
Conor Durkan,
Artur Bekasov,
Iain Murray,
George Papamakarios
Abstract:
A normalizing flow models a complex probability density as an invertible transformation of a simple base density. Flows based on either coupling or autoregressive transforms both offer exact density evaluation and sampling, but rely on the parameterization of an easily invertible elementwise transformation, whose choice determines the flexibility of these models. Building upon recent work, we prop…
▽ More
A normalizing flow models a complex probability density as an invertible transformation of a simple base density. Flows based on either coupling or autoregressive transforms both offer exact density evaluation and sampling, but rely on the parameterization of an easily invertible elementwise transformation, whose choice determines the flexibility of these models. Building upon recent work, we propose a fully-differentiable module based on monotonic rational-quadratic splines, which enhances the flexibility of both coupling and autoregressive transforms while retaining analytic invertibility. We demonstrate that neural spline flows improve density estimation, variational inference, and generative modeling of images.
△ Less
Submitted 2 December, 2019; v1 submitted 10 June, 2019;
originally announced June 2019.
-
Cubic-Spline Flows
Authors:
Conor Durkan,
Artur Bekasov,
Iain Murray,
George Papamakarios
Abstract:
A normalizing flow models a complex probability density as an invertible transformation of a simple density. The invertibility means that we can evaluate densities and generate samples from a flow. In practice, autoregressive flow-based models are slow to invert, making either density estimation or sample generation slow. Flows based on coupling transforms are fast for both tasks, but have previou…
▽ More
A normalizing flow models a complex probability density as an invertible transformation of a simple density. The invertibility means that we can evaluate densities and generate samples from a flow. In practice, autoregressive flow-based models are slow to invert, making either density estimation or sample generation slow. Flows based on coupling transforms are fast for both tasks, but have previously performed less well at density estimation than autoregressive flows. We stack a new coupling transform, based on monotonic cubic splines, with LU-decomposed linear layers. The resulting cubic-spline flow retains an exact one-pass inverse, can be used to generate high-quality images, and closes the gap with autoregressive flows on a suite of density-estimation tasks.
△ Less
Submitted 5 June, 2019;
originally announced June 2019.
-
Autoregressive Energy Machines
Authors:
Charlie Nash,
Conor Durkan
Abstract:
Neural density estimators are flexible families of parametric models which have seen widespread use in unsupervised machine learning in recent years. Maximum-likelihood training typically dictates that these models be constrained to specify an explicit density. However, this limitation can be overcome by instead using a neural network to specify an energy function, or unnormalized density, which c…
▽ More
Neural density estimators are flexible families of parametric models which have seen widespread use in unsupervised machine learning in recent years. Maximum-likelihood training typically dictates that these models be constrained to specify an explicit density. However, this limitation can be overcome by instead using a neural network to specify an energy function, or unnormalized density, which can subsequently be normalized to obtain a valid distribution. The challenge with this approach lies in accurately estimating the normalizing constant of the high-dimensional energy function. We propose the Autoregressive Energy Machine, an energy-based model which simultaneously learns an unnormalized density and computes an importance-sampling estimate of the normalizing constant for each conditional in an autoregressive decomposition. The Autoregressive Energy Machine achieves state-of-the-art performance on a suite of density-estimation tasks.
△ Less
Submitted 11 April, 2019;
originally announced April 2019.
-
Unravelling the electrical properties of epitaxial Graphene nanoribbons
Authors:
Zhuocong Xiao,
Colm Durkan
Abstract:
The size-dependent electrical resistivity of single-layer graphene ribbons has been studied experimentally for ribbon widths from 16 nm to 320 nm. The experimental findings are that the resistivity follows a more dramatic trend than that seen for metallic nanowires of similar dimensions, due to a combination of surface scattering from the edges, band-gap related effects and shifts in the Fermi lev…
▽ More
The size-dependent electrical resistivity of single-layer graphene ribbons has been studied experimentally for ribbon widths from 16 nm to 320 nm. The experimental findings are that the resistivity follows a more dramatic trend than that seen for metallic nanowires of similar dimensions, due to a combination of surface scattering from the edges, band-gap related effects and shifts in the Fermi level due to edge effects. We show that the Charge Neutrality point switches polarity below a ribbon width of around 50 nm, and that at this point, the thermal coefficient of resistance is a maximum. The majority do** type therefore can be controlled by altering ribbon width. We also demonstrate that an alumina passivation layer has a significant effect on the mean free path of the charge carriers within the graphene, which can be probed directly via measurements of the width-dependent resistivity. We propose a model for conduction that takes edge and confinement effects into account.
△ Less
Submitted 23 May, 2019; v1 submitted 22 February, 2019;
originally announced February 2019.
-
Sequential Neural Methods for Likelihood-free Inference
Authors:
Conor Durkan,
George Papamakarios,
Iain Murray
Abstract:
Likelihood-free inference refers to inference when a likelihood function cannot be explicitly evaluated, which is often the case for models based on simulators. Most of the literature is based on sample-based `Approximate Bayesian Computation' methods, but recent work suggests that approaches based on deep neural conditional density estimators can obtain state-of-the-art results with fewer simulat…
▽ More
Likelihood-free inference refers to inference when a likelihood function cannot be explicitly evaluated, which is often the case for models based on simulators. Most of the literature is based on sample-based `Approximate Bayesian Computation' methods, but recent work suggests that approaches based on deep neural conditional density estimators can obtain state-of-the-art results with fewer simulations. The neural approaches vary in how they choose which simulations to run and what they learn: an approximate posterior or a surrogate likelihood. This work provides some direct controlled comparisons between these choices.
△ Less
Submitted 21 November, 2018;
originally announced November 2018.
-
Periodic ripples on thermally annealed Graphene on Cu (110) - Surface reconstruction or Moire pattern?
Authors:
Colm Durkan
Abstract:
We have used Ultrahigh Vacuum (UHV) Scanning tunneling microscopy (STM) to investigate the effect of thermal annealing of graphene grown by chemical vapour deposition (CVD) on a Cu(110) foil. We show that the annealing appears to induce a reconstruction of the Cu surface along the [210] direction, with a period of 1.43 nm. Such reconstructions have been ascribed to the tensile strain induced in th…
▽ More
We have used Ultrahigh Vacuum (UHV) Scanning tunneling microscopy (STM) to investigate the effect of thermal annealing of graphene grown by chemical vapour deposition (CVD) on a Cu(110) foil. We show that the annealing appears to induce a reconstruction of the Cu surface along the [210] direction, with a period of 1.43 nm. Such reconstructions have been ascribed to the tensile strain induced in the Cu surface by differential thermal expansion of it relative to the graphene overlayer, but we show that it is in fact a Moiré pattern due to interference between the graphene and the underlying atomic lattice as evidenced by the appearance of an odd-even transition only observed due to mis-orientation of the top layer of a crystal. This highlights that the analysis of STM measurements of graphene on metal surfaces should take such interference into account and that the graphene-Cu interface is more complex than previously thought.
△ Less
Submitted 9 October, 2017;
originally announced October 2017.
-
Towards Resolving Landauer's Paradox Through Direct Observation of Multiscale Ferroelastic-Ferroelectric Interplay
Authors:
Colm Durkan,
Asaf Hershkovitz,
Da** Chu,
James F. Scott,
Yachin Ivry
Abstract:
Electric-polarization reversibility in nano-ferroelectric structures renders them as a convenient platform for exploring phase transitions and develo** energy-efficient switching devices. However, the fundamental question of how ferroic domains switch, i.e. how the polarization changes from one state to another, is yet to be answered fully. There are contradicting models and a wide body of accum…
▽ More
Electric-polarization reversibility in nano-ferroelectric structures renders them as a convenient platform for exploring phase transitions and develo** energy-efficient switching devices. However, the fundamental question of how ferroic domains switch, i.e. how the polarization changes from one state to another, is yet to be answered fully. There are contradicting models and a wide body of accumulated data which disagree as to whether the switching requires domain nucleation. Moreover, ferroelectric domains switch under electric fields that are supposedly too weak to form nucleation sites, indicating that the level of disorder seen in real systems plays an important role. This longstanding so-called Landauer's paradox is the ferroelectric equivalent to the absence of raindrop formation in a dust-free vacuum, leading to supersaturated vapors that cannot exist otherwise, e.g. in spinodal decompositions or inhomogeneous nucleation environments. Here we show that polarization switching in ferroelectric-ferroelastic systems comprises domain types that differ by symmetry, lengthscale and switching energy. These domains switch simultaneously thanks to intermediate-range order of organized pinning sites, supporting the previously-unexplained coexistence of nucleation-and-growth and nucleation-frustrated mechanisms. Our treatment is applicable to other Kolmogorov-Avrami systems with multi-scale phase transitions. Finally, we demonstrate augmented electromechanical coupling based on the collective motion of pinning sites, which is promising for nano electro-mechanical and low-power switching devices.
△ Less
Submitted 12 August, 2016;
originally announced August 2016.
-
Temperature dependence of the ohmic conductivity and activation energy of Pb1+y(Zr0.3Ti0.7)O3 thin films
Authors:
D. P. Chu,
B. M. McGregor,
P. Migliorato,
C. Durkan,
M. E. Welland,
K. Hasegawa,
T. Shimoda
Abstract:
The ohmic conductivity of the sol-gel derived Pb1+y(Zr0.3Ti0.7)O3 thin films (with the excess lead y=0.0 to 0.4) are investigated using low frequency small signal alternate current (AC) and direct current (DC) methods. Its temperature dependence shows two activation energies of 0.26 and 0.12 eV depending on temperature range and excess Pb levels. The former is associated with Pb3+ acceptor cente…
▽ More
The ohmic conductivity of the sol-gel derived Pb1+y(Zr0.3Ti0.7)O3 thin films (with the excess lead y=0.0 to 0.4) are investigated using low frequency small signal alternate current (AC) and direct current (DC) methods. Its temperature dependence shows two activation energies of 0.26 and 0.12 eV depending on temperature range and excess Pb levels. The former is associated with Pb3+ acceptor centers, while the latter could be due to a different defect level yet to be identified.
△ Less
Submitted 5 June, 2001;
originally announced June 2001.
