Showing 1–7 of 7 results for author: Shaul, N

Bespoke Non-Stationary Solvers for Fast Sampling of Diffusion and Flow Models

Authors: Neta Shaul, Uriel Singer, Ricky T. Q. Chen, Matthew Le, Ali Thabet, Albert Pumarola, Yaron Lipman

Abstract: This paper introduces Bespoke Non-Stationary (BNS) Solvers, a solver distillation approach to improve sample efficiency of Diffusion and Flow models. BNS solvers are based on a family of non-stationary solvers that provably subsumes existing numerical ODE solvers and consequently demonstrate considerable improvement in sample approximation (PSNR) over these baselines. Compared to model distillatio… ▽ More This paper introduces Bespoke Non-Stationary (BNS) Solvers, a solver distillation approach to improve sample efficiency of Diffusion and Flow models. BNS solvers are based on a family of non-stationary solvers that provably subsumes existing numerical ODE solvers and consequently demonstrate considerable improvement in sample approximation (PSNR) over these baselines. Compared to model distillation, BNS solvers benefit from a tiny parameter space ($<$200 parameters), fast optimization (two orders of magnitude faster), maintain diversity of samples, and in contrast to previous solver distillation approaches nearly close the gap from standard distillation methods such as Progressive Distillation in the low-medium NFE regime. For example, BNS solver achieves 45 PSNR / 1.76 FID using 16 NFE in class-conditional ImageNet-64. We experimented with BNS solvers for conditional image generation, text-to-image generation, and text-2-audio generation showing significant improvement in sample approximation (PSNR) in all. △ Less

Submitted 2 March, 2024; originally announced March 2024.

Guided Flows for Generative Modeling and Decision Making

Authors: Qinqing Zheng, Matt Le, Neta Shaul, Yaron Lipman, Aditya Grover, Ricky T. Q. Chen

Abstract: Classifier-free guidance is a key component for enhancing the performance of conditional generative models across diverse tasks. While it has previously demonstrated remarkable improvements for the sample quality, it has only been exclusively employed for diffusion models. In this paper, we integrate classifier-free guidance into Flow Matching (FM) models, an alternative simulation-free approach t… ▽ More Classifier-free guidance is a key component for enhancing the performance of conditional generative models across diverse tasks. While it has previously demonstrated remarkable improvements for the sample quality, it has only been exclusively employed for diffusion models. In this paper, we integrate classifier-free guidance into Flow Matching (FM) models, an alternative simulation-free approach that trains Continuous Normalizing Flows (CNFs) based on regressing vector fields. We explore the usage of \emph{Guided Flows} for a variety of downstream applications. We show that Guided Flows significantly improves the sample quality in conditional image generation and zero-shot text-to-speech synthesis, boasting state-of-the-art performance. Notably, we are the first to apply flow models for plan generation in the offline reinforcement learning setting, showcasing a 10x speedup in computation compared to diffusion models while maintaining comparable performance. △ Less

Submitted 7 December, 2023; v1 submitted 22 November, 2023; originally announced November 2023.

Bespoke Solvers for Generative Flow Models

Authors: Neta Shaul, Juan Perez, Ricky T. Q. Chen, Ali Thabet, Albert Pumarola, Yaron Lipman

Abstract: Diffusion or flow-based models are powerful generative paradigms that are notoriously hard to sample as samples are defined as solutions to high-dimensional Ordinary or Stochastic Differential Equations (ODEs/SDEs) which require a large Number of Function Evaluations (NFE) to approximate well. Existing methods to alleviate the costly sampling process include model distillation and designing dedica… ▽ More Diffusion or flow-based models are powerful generative paradigms that are notoriously hard to sample as samples are defined as solutions to high-dimensional Ordinary or Stochastic Differential Equations (ODEs/SDEs) which require a large Number of Function Evaluations (NFE) to approximate well. Existing methods to alleviate the costly sampling process include model distillation and designing dedicated ODE solvers. However, distillation is costly to train and sometimes can deteriorate quality, while dedicated solvers still require relatively large NFE to produce high quality samples. In this paper we introduce "Bespoke solvers", a novel framework for constructing custom ODE solvers tailored to the ODE of a given pre-trained flow model. Our approach optimizes an order consistent and parameter-efficient solver (e.g., with 80 learnable parameters), is trained for roughly 1% of the GPU time required for training the pre-trained model, and significantly improves approximation and generation quality compared to dedicated solvers. For example, a Bespoke solver for a CIFAR10 model produces samples with Fréchet Inception Distance (FID) of 2.73 with 10 NFE, and gets to 1% of the Ground Truth (GT) FID (2.59) for this model with only 20 NFE. On the more challenging ImageNet-64$\times$64, Bespoke samples at 2.2 FID with 10 NFE, and gets within 2% of GT FID (1.71) with 20 NFE. △ Less

Submitted 29 October, 2023; originally announced October 2023.

On Kinetic Optimal Probability Paths for Generative Models

Authors: Neta Shaul, Ricky T. Q. Chen, Maximilian Nickel, Matt Le, Yaron Lipman

Abstract: Recent successful generative models are trained by fitting a neural network to an a-priori defined tractable probability density path taking noise to training examples. In this paper we investigate the space of Gaussian probability paths, which includes diffusion paths as an instance, and look for an optimal member in some useful sense. In particular, minimizing the Kinetic Energy (KE) of a path i… ▽ More Recent successful generative models are trained by fitting a neural network to an a-priori defined tractable probability density path taking noise to training examples. In this paper we investigate the space of Gaussian probability paths, which includes diffusion paths as an instance, and look for an optimal member in some useful sense. In particular, minimizing the Kinetic Energy (KE) of a path is known to make particles' trajectories simple, hence easier to sample, and empirically improve performance in terms of likelihood of unseen data and sample generation quality. We investigate Kinetic Optimal (KO) Gaussian paths and offer the following observations: (i) We show the KE takes a simplified form on the space of Gaussian paths, where the data is incorporated only through a single, one dimensional scalar function, called the \emph{data separation function}. (ii) We characterize the KO solutions with a one dimensional ODE. (iii) We approximate data-dependent KO paths by approximating the data separation function and minimizing the KE. (iv) We prove that the data separation function converges to $1$ in the general case of arbitrary normalized dataset consisting of $n$ samples in $d$ dimension as $n/\sqrt{d}\rightarrow 0$. A consequence of this result is that the Conditional Optimal Transport (Cond-OT) path becomes \emph{kinetic optimal} as $n/\sqrt{d}\rightarrow 0$. We further support this theory with empirical experiments on ImageNet. △ Less

Submitted 11 June, 2023; originally announced June 2023.

Improved Simulation of the Mass Charging for ASTROD I

Authors: Gang Bao, Wei-Tou Ni, D. N. A. Shaul, H. M. Araujo, Lei Liu, T. J. Sumner

Abstract: The electrostatic charging of the test mass in ASTROD I (Astrodynamical Space Test of Relativity using Optical Devices I) mission can affect the quality of the science data as a result of spurious Coulomb and Lorentz forces. To estimate the size of the resultant disturbances, credible predictions of charging rates and the charging noise are required. Using the GEANT4 software toolkit, we present… ▽ More The electrostatic charging of the test mass in ASTROD I (Astrodynamical Space Test of Relativity using Optical Devices I) mission can affect the quality of the science data as a result of spurious Coulomb and Lorentz forces. To estimate the size of the resultant disturbances, credible predictions of charging rates and the charging noise are required. Using the GEANT4 software toolkit, we present a detailed Monte Carlo simulation of the ASTROD I test mass charging due to exposure of the spacecraft to galactic cosmic-ray (GCR) protons and alpha particles (3He, 4He) in the space environment. A positive charging rate of 33.3 e+/s at solar minimum is obtained. This figure reduces by 50% at solar maximum. Based on this charging rate and factoring in the contribution of minor cosmic-ray components, we calculate the acceleration noise and stiffness associated with charging. We conclude that the acceleration noise arising from Coulomb and Lorentz effects are well below the ASTROD I acceleration noise limit at 0.1 mHz both at solar minimum and maximum. The coherent Fourier components due to charging are investigated, it needs to be studied carefully in order to ensure that these do not compromise the quality of science data in the ASTROD I mission. △ Less

Submitted 13 July, 2007; originally announced July 2007.

Comments: 20 pages, 14 figures, submitted to International Journal of Modern Physics D

Journal ref: INTERNATIONAL JOURNAL OF MODERN PHYSICS D, Vol.17, No.7, 965 (July 2008)

Simulation of ASTROD I test mass charging due to solar energetic particles

Authors: Lei Liu, Gang Bao, Wei-Tou Ni, D N A Shaul

Abstract: As ASTROD I travels through space, its test mass will accrue charge due to galactic cosmic-rays and solar energetic particles incident on the spacecraft. This test mass charge will result in Coulomb forces between the test mass and the surrounding electrodes. In earlier work using the GEANT4 toolkit, we predicted a net charging rate of nearly 9.0 +e/s from cosmic-ray protons between 0.1 and 1000… ▽ More As ASTROD I travels through space, its test mass will accrue charge due to galactic cosmic-rays and solar energetic particles incident on the spacecraft. This test mass charge will result in Coulomb forces between the test mass and the surrounding electrodes. In earlier work using the GEANT4 toolkit, we predicted a net charging rate of nearly 9.0 +e/s from cosmic-ray protons between 0.1 and 1000 GeV at solar maximum, and rising to 26.5 +e/s at solar minimum. Here we use GEANT4 to simulate the charging process due to solar energetic particle events and to estimate the magnitude of acceleration noise due to this charging. The predicted charging rates range from 2840 to 64300 +e/s, at peak intensity, for the 4 largest SEP events in September and October 1989. For the 2 larger events, the acceleration disturbances due to charging exceeds the ASTROD I acceleration noise budget requirement. Continuous discharge should be considered for suppressing this charging noise. The acceleration noise during the 2 small events is well below the design target, although during these events, the net charging rate will be dominated by these solar fluxes. △ Less

Submitted 26 April, 2007; originally announced April 2007.

Comments: 10 pages,8 figures, COSPAR2006 H0.1-1, submitted

ASTROD I Charging Simulation and Disturbances

Authors: Gang Bao, D N A Shaul, H M Araujo, Wei-Tou Ni, T J Sumner, Lei Liu

Abstract: ASTROD I is planned as a single spacecraft mission. It will use interferometric and pulse ranging techniques between the spacecraft and ground stations, to make high precision measurements of the parameters that describe the solar system, and to test relativistic gravity with improved accuracy. At the heart of the spacecraft is a test mass, which the spacecraft will follow using a drag-free cont… ▽ More ASTROD I is planned as a single spacecraft mission. It will use interferometric and pulse ranging techniques between the spacecraft and ground stations, to make high precision measurements of the parameters that describe the solar system, and to test relativistic gravity with improved accuracy. At the heart of the spacecraft is a test mass, which the spacecraft will follow using a drag-free control system. The mission critically depends on maintaining the geodesic motion of the test mass. Charging of the test mass due to cosmic rays and solar particles will disturb its geodesic motion. We have modelled the charging process using the GEANT4 toolkit and a simplified, geometrical model and estimate that the ASTROD I test mass will charge positively, at a rate of 24 +/-7 e+/s, due to cosmic ray protons and alpha particles (3He and 4He) at solar minimum. We have used the results of this simulation to estimate the magnitude of disturbances associated with test mass charging, for the worst-case scenario, taking into account uncertainties in the model and potential charging contributions from minor cosmic-ray components. △ Less

Submitted 24 April, 2007; originally announced April 2007.

Comments: 17 pages, 11 figures, submitted to General Relativity and Gravitation