-
SEEDS: Exponential SDE Solvers for Fast High-Quality Sampling from Diffusion Models
Authors:
Martin Gonzalez,
Nelson Fernandez,
Thuy Tran,
Elies Gherbi,
Hatem Hajri,
Nader Masmoudi
Abstract:
A potent class of generative models known as Diffusion Probabilistic Models (DPMs) has become prominent. A forward diffusion process adds gradually noise to data, while a model learns to gradually denoise. Sampling from pre-trained DPMs is obtained by solving differential equations (DE) defined by the learnt model, a process which has shown to be prohibitively slow. Numerous efforts on speeding-up…
▽ More
A potent class of generative models known as Diffusion Probabilistic Models (DPMs) has become prominent. A forward diffusion process adds gradually noise to data, while a model learns to gradually denoise. Sampling from pre-trained DPMs is obtained by solving differential equations (DE) defined by the learnt model, a process which has shown to be prohibitively slow. Numerous efforts on speeding-up this process have consisted on crafting powerful ODE solvers. Despite being quick, such solvers do not usually reach the optimal quality achieved by available slow SDE solvers. Our goal is to propose SDE solvers that reach optimal quality without requiring several hundreds or thousands of NFEs to achieve that goal. We propose Stochastic Explicit Exponential Derivative-free Solvers (SEEDS), improving and generalizing Exponential Integrator approaches to the stochastic case on several frameworks. After carefully analyzing the formulation of exact solutions of diffusion SDEs, we craft SEEDS to analytically compute the linear part of such solutions. Inspired by the Exponential Time-Differencing method, SEEDS use a novel treatment of the stochastic components of solutions, enabling the analytical computation of their variance, and contains high-order terms allowing to reach optimal quality sampling $\sim3$-$5\times$ faster than previous SDE methods. We validate our approach on several image generation benchmarks, showing that SEEDS outperform or are competitive with previous SDE solvers. Contrary to the latter, SEEDS are derivative and training free, and we fully prove strong convergence guarantees for them.
△ Less
Submitted 26 October, 2023; v1 submitted 23 May, 2023;
originally announced May 2023.
-
Realization Theory Of Recurrent Neural ODEs Using Polynomial System Embeddings
Authors:
Martin Gonzalez,
Thibault Defourneau,
Hatem Hajri,
Mihaly Petreczky
Abstract:
In this paper we show that neural ODE analogs of recurrent (ODE-RNN) and Long Short-Term Memory (ODE-LSTM) networks can be algorithmically embeddeded into the class of polynomial systems. This embedding preserves input-output behavior and can suitably be extended to other neural DE architectures. We then use realization theory of polynomial systems to provide necessary conditions for an input-outp…
▽ More
In this paper we show that neural ODE analogs of recurrent (ODE-RNN) and Long Short-Term Memory (ODE-LSTM) networks can be algorithmically embeddeded into the class of polynomial systems. This embedding preserves input-output behavior and can suitably be extended to other neural DE architectures. We then use realization theory of polynomial systems to provide necessary conditions for an input-output map to be realizable by an ODE-LSTM and sufficient conditions for minimality of such systems. These results represent the first steps towards realization theory of recurrent neural ODE architectures, which is is expected be useful for model reduction and learning algorithm analysis of recurrent neural ODEs.
△ Less
Submitted 1 August, 2022; v1 submitted 24 May, 2022;
originally announced May 2022.
-
Gaussian distributions on Riemannian symmetric spaces: statistical learning with structured covariance matrices
Authors:
Salem Said,
Hatem Hajri,
Lionel Bombrun,
Baba C. Vemuri
Abstract:
The Riemannian geometry of covariance matrices has been essential to several successful applications, in computer vision, biomedical signal and image processing, and radar data processing. For these applications, an important ongoing challenge is to develop Riemannian-geometric tools which are adapted to structured covariance matrices. The present paper proposes to meet this challenge by introduci…
▽ More
The Riemannian geometry of covariance matrices has been essential to several successful applications, in computer vision, biomedical signal and image processing, and radar data processing. For these applications, an important ongoing challenge is to develop Riemannian-geometric tools which are adapted to structured covariance matrices. The present paper proposes to meet this challenge by introducing a new class of probability distributions, Gaussian distributions of structured covariance matrices. These are Riemannian analogs of Gaussian distributions, which only sample from covariance matrices having a preassigned structure, such as complex, Toeplitz, or block-Toeplitz. The usefulness of these distributions stems from three features: (1) they are completely tractable, analytically or numerically, when dealing with large covariance matrices, (2) they provide a statistical foundation to the concept of structured Riemannian barycentre (i.e. Fréchet or geometric mean), (3) they lead to efficient statistical learning algorithms, which realise, among others, density estimation and classification of structured covariance matrices. The paper starts from the observation that several spaces of structured covariance matrices, considered from a geometric point of view, are Riemannian symmetric spaces. Accordingly, it develops an original theory of Gaussian distributions on Riemannian symmetric spaces, of their statistical inference, and of their relationship to the concept of Riemannian barycentre. Then, it uses this original theory to give a detailed description of Gaussian distributions of three kinds of structured covariance matrices, complex, Toeplitz, and block-Toeplitz. Finally, it describes algorithms for density estimation and classification of structured covariance matrices, based on Gaussian distribution mixture models.
△ Less
Submitted 12 May, 2017; v1 submitted 23 July, 2016;
originally announced July 2016.
-
Application of Stochastic Flows to the Sticky Brownian Motion Equation
Authors:
Hatem Hajri,
Caglar Mine,
Marc Arnaudon
Abstract:
We show how the theory of stochastic flows allows to recover in an elementary way a well known result of Warren on the sticky Brownian motion equation.
We show how the theory of stochastic flows allows to recover in an elementary way a well known result of Warren on the sticky Brownian motion equation.
△ Less
Submitted 27 December, 2016; v1 submitted 24 March, 2016;
originally announced March 2016.
-
On a coupling of solutions to the interface SDE on a star graph
Authors:
Hatem Hajri,
Marc Arnaudon
Abstract:
Inspired by Tsirelson proof of the non Brownian character of WalshBrownian motion ltration on three or more rays, we prove some results on aparticular coupling of solutions to the interface SDE on a star graph, recentlyintroduced. This coupling consists in two solutions which are independentgiven the driving Brownian motion. As a consequence, we deduce that if the stargraph contains 3 or more rays…
▽ More
Inspired by Tsirelson proof of the non Brownian character of WalshBrownian motion ltration on three or more rays, we prove some results on aparticular coupling of solutions to the interface SDE on a star graph, recentlyintroduced. This coupling consists in two solutions which are independentgiven the driving Brownian motion. As a consequence, we deduce that if the stargraph contains 3 or more rays, the argument of the solution at a xed time isindependent of the driving Brownian motion.
△ Less
Submitted 13 December, 2017; v1 submitted 27 January, 2016;
originally announced January 2016.
-
On flows associated to Tanaka's SDE and related works
Authors:
Hatem Hajri
Abstract:
We review the construction of flows associated to Tanaka's SDE from [9] and give an easy proof of the classification of these flows by means of probability measures on [0, 1]. Our arguments also simplify some proofs in the subsequent papers [2, 3, 7, 4].
We review the construction of flows associated to Tanaka's SDE from [9] and give an easy proof of the classification of these flows by means of probability measures on [0, 1]. Our arguments also simplify some proofs in the subsequent papers [2, 3, 7, 4].
△ Less
Submitted 13 January, 2015;
originally announced January 2015.
-
Stochastic flows and an interface SDE on metric graphs
Authors:
Hatem Hajri,
Olivier Raimond
Abstract:
This paper consists in the study of a stochastic differential equation on a metric graph, called an interface SDE $(\hbox{ISDE})$. To each edge of the graph is associated an independent white noise, which drives $(\hbox{ISDE})$ on this edge. This produces an interface at each vertex of the graph. We first do our study on star graphs with $N\ge 2$ rays. The case $N=2$ corresponds to the perturbed T…
▽ More
This paper consists in the study of a stochastic differential equation on a metric graph, called an interface SDE $(\hbox{ISDE})$. To each edge of the graph is associated an independent white noise, which drives $(\hbox{ISDE})$ on this edge. This produces an interface at each vertex of the graph. We first do our study on star graphs with $N\ge 2$ rays. The case $N=2$ corresponds to the perturbed Tanaka's equation recently studied by Prokaj \cite{MR18} and Le Jan-Raimond \cite{MR000} among others. It is proved that $(\hbox{ISDE})$ has a unique in law solution, which is a Walsh's Brownian motion. This solution is strong if and only if $N=2$.
Solution flows are also considered. There is a (unique in law) coalescing stochastic flow of
map**s $\p$ solving $(\hbox{ISDE})$. For $N=2$, it is the only solution flow. For $N\ge 3$, $\p$ is not a strong solution and by filtering $\p$ with respect to the
family of white noises, we obtain a (Wiener) stochastic flow of kernels solution of $(\hbox{ISDE})$.
There are no other Wiener solutions.
Our previous results \cite{MR501011} in hand, these results are extended to more general metric graphs.
The proofs involve the study of $(X,Y)$ a Brownian motion in a two dimensional quadrant obliquely reflected at the boundary, with time dependent
angle of reflection. We prove in particular that, when $(X\_0,Y\_0)=(1,0)$ and if $S$ is the first time $X$ hits $0$, then $Y\_S^2$ is a beta random variable of the second kind. We also calculate $\EE[L\_{σ\_0}]$, where $L$ is the local time accumulated at the boundary, and $σ\_0$ is the first time $(X,Y)$ hits $(0,0)$.
△ Less
Submitted 1 June, 2015; v1 submitted 14 October, 2013;
originally announced October 2013.
-
Stochastic flows on metric graphs
Authors:
Hatem Hajri,
Olivier Raimond
Abstract:
We study a simple stochastic differential equation driven by one Brownian motion on a general oriented metric graph whose solutions are stochastic flows of kernels. Under some condition, we describe the laws of all solutions. This work is a natural continuation of some previous papers by Hajri, Hajri-Raimond and Le Jan-Raimond where some particular graphs have been considered.
We study a simple stochastic differential equation driven by one Brownian motion on a general oriented metric graph whose solutions are stochastic flows of kernels. Under some condition, we describe the laws of all solutions. This work is a natural continuation of some previous papers by Hajri, Hajri-Raimond and Le Jan-Raimond where some particular graphs have been considered.
△ Less
Submitted 3 May, 2013;
originally announced May 2013.
-
On the Csáki-Vincze transformation
Authors:
Hatem Hajri
Abstract:
Cs aki and Vincze have de fined in 1961 a discrete transformation T which applies to simple random walks and is measure preserving. In this paper, we are interested in ergodic and assymptotic properties of T . We prove that T is exact : \cap_{k\geq 1} σ(T^k(S)) is trivial for each simple random walk S and give a precise description of the lost information at each step k. We then show that, in a su…
▽ More
Cs aki and Vincze have de fined in 1961 a discrete transformation T which applies to simple random walks and is measure preserving. In this paper, we are interested in ergodic and assymptotic properties of T . We prove that T is exact : \cap_{k\geq 1} σ(T^k(S)) is trivial for each simple random walk S and give a precise description of the lost information at each step k. We then show that, in a suitable scaling limit, all iterations of T "converge" to the corresponding iterations of the continous L evy transform of Brownian motion. Some consequences are also derived from these two results.
△ Less
Submitted 17 September, 2012; v1 submitted 2 September, 2012;
originally announced September 2012.
-
Tanaka's equation on the circle and stochastic flows
Authors:
Hatem Hajri,
Olivier Raimond
Abstract:
We define a Tanaka's equation on an oriented graph with two edges and two vertices. This graph will be embedded in the unit circle. Extending this equation to flows of kernels, we show that the laws of the flows of kernels $K$ solution of Tanaka's equation can be classified by pairs of probability measures $(m^+,m^-)$ on $[0,1]$, with mean 1/2. What happens at the first vertex is governed by…
▽ More
We define a Tanaka's equation on an oriented graph with two edges and two vertices. This graph will be embedded in the unit circle. Extending this equation to flows of kernels, we show that the laws of the flows of kernels $K$ solution of Tanaka's equation can be classified by pairs of probability measures $(m^+,m^-)$ on $[0,1]$, with mean 1/2. What happens at the first vertex is governed by $m^+$, and at the second by $m^-$. For each vertex $P$, we construct a sequence of stop** times along which the image of the whole circle by $K$ is reduced to $P$. We also prove that the supports of these flows contains a finite number of points, and that except for some particular cases this number of points can be arbitrarily large.
△ Less
Submitted 21 April, 2013; v1 submitted 19 March, 2012;
originally announced March 2012.
-
Discrete approximation to solution flows of Tanaka SDE related to Walsh Brownian motion
Authors:
Hatem Hajri
Abstract:
In a previous work, we have defined a Tanaka SDE related to Walsh Brownian motion which depends on kernels. It was shown that there are only one Wiener solution and only one flow of map**s solving this equation. In the terminology of Le Jan and Raimond, these are respectively the stronger and the weaker among all solutions. In this paper, we obtain these solutions as limits of discrete models.
In a previous work, we have defined a Tanaka SDE related to Walsh Brownian motion which depends on kernels. It was shown that there are only one Wiener solution and only one flow of map**s solving this equation. In the terminology of Le Jan and Raimond, these are respectively the stronger and the weaker among all solutions. In this paper, we obtain these solutions as limits of discrete models.
△ Less
Submitted 2 October, 2011; v1 submitted 10 January, 2011;
originally announced January 2011.
-
Stochastic flows related to Walsh Brownian motion
Authors:
Hatem Hajri
Abstract:
We define an equation on a simple graph which is an extension of Tanaka equation and the skew Brownian motion equation. We then apply the theory of transition kernels developped by Le Jan and Raimond and show that all the solutions can be classified by probability measures.
We define an equation on a simple graph which is an extension of Tanaka equation and the skew Brownian motion equation. We then apply the theory of transition kernels developped by Le Jan and Raimond and show that all the solutions can be classified by probability measures.
△ Less
Submitted 3 October, 2011; v1 submitted 8 January, 2011;
originally announced January 2011.
