-
Explicit Flow Matching: On The Theory of Flow Matching Algorithms with Applications
Authors:
Gleb Ryzhakov,
Svetlana Pavlova,
Egor Sevriugov,
Ivan Oseledets
Abstract:
This paper proposes a novel method, Explicit Flow Matching (ExFM), for training and analyzing flow-based generative models. ExFM leverages a theoretically grounded loss function, ExFM loss (a tractable form of Flow Matching (FM) loss), to demonstrably reduce variance during training, leading to faster convergence and more stable learning. Based on theoretical analysis of these formulas, we derived…
▽ More
This paper proposes a novel method, Explicit Flow Matching (ExFM), for training and analyzing flow-based generative models. ExFM leverages a theoretically grounded loss function, ExFM loss (a tractable form of Flow Matching (FM) loss), to demonstrably reduce variance during training, leading to faster convergence and more stable learning. Based on theoretical analysis of these formulas, we derived exact expressions for the vector field (and score in stochastic cases) for model examples (in particular, for separating multiple exponents), and in some simple cases, exact solutions for trajectories. In addition, we also investigated simple cases of diffusion generative models by adding a stochastic term and obtained an explicit form of the expression for score. While the paper emphasizes the theoretical underpinnings of ExFM, it also showcases its effectiveness through numerical experiments on various datasets, including high-dimensional ones. Compared to traditional FM methods, ExFM achieves superior performance in terms of both learning speed and final outcomes.
△ Less
Submitted 1 July, 2024; v1 submitted 5 February, 2024;
originally announced February 2024.
-
Black-Box Approximation and Optimization with Hierarchical Tucker Decomposition
Authors:
Gleb Ryzhakov,
Andrei Chertkov,
Artem Basharin,
Ivan Oseledets
Abstract:
We develop a new method HTBB for the multidimensional black-box approximation and gradient-free optimization, which is based on the low-rank hierarchical Tucker decomposition with the use of the MaxVol indices selection procedure. Numerical experiments for 14 complex model problems demonstrate the robustness of the proposed method for dimensions up to 1000, while it shows significantly more accura…
▽ More
We develop a new method HTBB for the multidimensional black-box approximation and gradient-free optimization, which is based on the low-rank hierarchical Tucker decomposition with the use of the MaxVol indices selection procedure. Numerical experiments for 14 complex model problems demonstrate the robustness of the proposed method for dimensions up to 1000, while it shows significantly more accurate results than classical gradient-free optimization methods, as well as approximation and optimization methods based on the popular tensor train decomposition, which represents a simpler case of a tensor network.
△ Less
Submitted 5 February, 2024;
originally announced February 2024.
-
PROTES: Probabilistic Optimization with Tensor Sampling
Authors:
Anastasia Batsheva,
Andrei Chertkov,
Gleb Ryzhakov,
Ivan Oseledets
Abstract:
We developed a new method PROTES for black-box optimization, which is based on the probabilistic sampling from a probability density function given in the low-parametric tensor train format. We tested it on complex multidimensional arrays and discretized multivariable functions taken, among others, from real-world applications, including unconstrained binary optimization and optimal control proble…
▽ More
We developed a new method PROTES for black-box optimization, which is based on the probabilistic sampling from a probability density function given in the low-parametric tensor train format. We tested it on complex multidimensional arrays and discretized multivariable functions taken, among others, from real-world applications, including unconstrained binary optimization and optimal control problems, for which the possible number of elements is up to $2^{100}$. In numerical experiments, both on analytic model functions and on complex problems, PROTES outperforms existing popular discrete optimization methods (Particle Swarm Optimization, Covariance Matrix Adaptation, Differential Evolution, and others).
△ Less
Submitted 22 May, 2023; v1 submitted 28 January, 2023;
originally announced January 2023.
-
Optimization of Functions Given in the Tensor Train Format
Authors:
Andrei Chertkov,
Gleb Ryzhakov,
Georgii Novikov,
Ivan Oseledets
Abstract:
Tensor train (TT) format is a common approach for computationally efficient work with multidimensional arrays, vectors, matrices, and discretized functions in a wide range of applications, including computational mathematics and machine learning. In this work, we propose a new algorithm for TT-tensor optimization, which leads to very accurate approximations for the minimum and maximum tensor eleme…
▽ More
Tensor train (TT) format is a common approach for computationally efficient work with multidimensional arrays, vectors, matrices, and discretized functions in a wide range of applications, including computational mathematics and machine learning. In this work, we propose a new algorithm for TT-tensor optimization, which leads to very accurate approximations for the minimum and maximum tensor element. The method consists in sequential tensor multiplications of the TT-cores with an intelligent selection of candidates for the optimum. We propose the probabilistic interpretation of the method, and make estimates on its complexity and convergence. We perform extensive numerical experiments with random tensors and various multivariable benchmark functions with the number of input dimensions up to $100$. Our approach generates a solution close to the exact optimum for all model problems, while the running time is no more than $50$ seconds on a regular laptop.
△ Less
Submitted 29 September, 2022;
originally announced September 2022.
-
Black box approximation in the tensor train format initialized by ANOVA decomposition
Authors:
Andrei Chertkov,
Gleb Ryzhakov,
Ivan Oseledets
Abstract:
Surrogate models can reduce computational costs for multivariable functions with an unknown internal structure (black boxes). In a discrete formulation, surrogate modeling is equivalent to restoring a multidimensional array (tensor) from a small part of its elements. The alternating least squares (ALS) algorithm in the tensor train (TT) format is a widely used approach to effectively solve this pr…
▽ More
Surrogate models can reduce computational costs for multivariable functions with an unknown internal structure (black boxes). In a discrete formulation, surrogate modeling is equivalent to restoring a multidimensional array (tensor) from a small part of its elements. The alternating least squares (ALS) algorithm in the tensor train (TT) format is a widely used approach to effectively solve this problem in the case of non-adaptive tensor recovery from a given training set (i.e., tensor completion problem). TT-ALS allows obtaining a low-parametric representation of the tensor, which is free from the curse of dimensionality and can be used for fast computation of the values at arbitrary tensor indices or efficient implementation of algebra operations with the black box (integration, etc.). However, to obtain high accuracy in the presence of restrictions on the size of the train data, a good choice of initial approximation is essential. In this work, we construct the ANOVA representation in the TT-format and use it as an initial approximation for the TT-ALS algorithm. The performed numerical computations for a number of multidimensional model problems, including the parametric partial differential equation, demonstrate a significant advantage of our approach for the commonly used random initial approximation. For all considered model problems we obtained an increase in accuracy by at least an order of magnitude with the same number of requests to the black box. The proposed approach is very general and can be applied in a wide class of real-world surrogate modeling and machine learning problems.
△ Less
Submitted 5 August, 2022;
originally announced August 2022.
-
Constructive TT-representation of the tensors given as index interaction functions with applications
Authors:
Gleb Ryzhakov,
Ivan Oseledets
Abstract:
This paper presents a method to build explicit tensor-train (TT) representations. We show that a wide class of tensors can be explicitly represented with sparse TT-cores, obtaining, in many cases, optimal TT-ranks. Numerical experiments show that our method outperforms the existing ones in several practical applications, including game theory problems. Theoretical estimations of the number of oper…
▽ More
This paper presents a method to build explicit tensor-train (TT) representations. We show that a wide class of tensors can be explicitly represented with sparse TT-cores, obtaining, in many cases, optimal TT-ranks. Numerical experiments show that our method outperforms the existing ones in several practical applications, including game theory problems. Theoretical estimations of the number of operations show that in some problems, such as permanent calculation, our methods are close to the known optimal asymptotics, which are obtained by a completely different type of methods.
△ Less
Submitted 15 June, 2022; v1 submitted 8 June, 2022;
originally announced June 2022.
-
Understanding DDPM Latent Codes Through Optimal Transport
Authors:
Valentin Khrulkov,
Gleb Ryzhakov,
Andrei Chertkov,
Ivan Oseledets
Abstract:
Diffusion models have recently outperformed alternative approaches to model the distribution of natural images, such as GANs. Such diffusion models allow for deterministic sampling via the probability flow ODE, giving rise to a latent space and an encoder map. While having important practical applications, such as estimation of the likelihood, the theoretical properties of this map are not yet ful…
▽ More
Diffusion models have recently outperformed alternative approaches to model the distribution of natural images, such as GANs. Such diffusion models allow for deterministic sampling via the probability flow ODE, giving rise to a latent space and an encoder map. While having important practical applications, such as estimation of the likelihood, the theoretical properties of this map are not yet fully understood. In the present work, we partially address this question for the popular case of the VP SDE (DDPM) approach. We show that, perhaps surprisingly, the DDPM encoder map coincides with the optimal transport map for common distributions; we support this claim theoretically and by extensive numerical experiments.
△ Less
Submitted 5 December, 2022; v1 submitted 14 February, 2022;
originally announced February 2022.
-
Optimal soil sampling design based on the maxvol algorithm
Authors:
Anna Petrovskaia,
Gleb Ryzhakov,
Ivan Oseledets
Abstract:
Spatial soil sampling is an integral part of a soil survey aimed at creating a soil map. We propose considering the soil sampling procedure as a task of optimal design. In practical terms, optimal experiments can reduce experimentation costs, as they allow the researcher to obtain one optimal set of points. We present a sampling design, based on the fundamental idea of selecting sample locations b…
▽ More
Spatial soil sampling is an integral part of a soil survey aimed at creating a soil map. We propose considering the soil sampling procedure as a task of optimal design. In practical terms, optimal experiments can reduce experimentation costs, as they allow the researcher to obtain one optimal set of points. We present a sampling design, based on the fundamental idea of selecting sample locations by performing an optimal design method called the maxvol algorithm. It is shown that the maxvol-base algorithm has a high potential for practical usage. Our method outperforms popular sampling methods in soil taxa prediction based on topographical features of the site and deals with massive agricultural datasets in a reasonable time.
△ Less
Submitted 18 March, 2021;
originally announced March 2021.
-
Two-phase approaches to optimal model-based design of experiments: how many experiments and which ones?
Authors:
Charlie Vanaret,
Philipp Seufert,
Jan Schwientek,
Gleb Karpov,
Gleb Ryzhakov,
Ivan Oseledets,
Norbert Asprion,
Michael Bortz
Abstract:
Model-based experimental design is attracting increasing attention in chemical process engineering. Typically, an iterative procedure is pursued: an approximate model is devised, prescribed experiments are then performed and the resulting data is exploited to refine the model. To help to reduce the cost of trial-and-error approaches, strategies for model-based design of experiments suggest experim…
▽ More
Model-based experimental design is attracting increasing attention in chemical process engineering. Typically, an iterative procedure is pursued: an approximate model is devised, prescribed experiments are then performed and the resulting data is exploited to refine the model. To help to reduce the cost of trial-and-error approaches, strategies for model-based design of experiments suggest experimental points where the expected gain in information for the model is the largest. It requires the resolution of a large nonlinear, generally nonconvex, optimization problem, whose solution may greatly depend on the starting point. We present two discretization strategies that can assist the experimenter in setting the number of relevant experiments and performing an optimal selection, and we compare them against two pattern-based strategies that are independent of the problem. The validity of the approaches is demonstrated on an academic example and two test problems from chemical engineering including a vapor liquid equilibrium and reaction kinetics.
△ Less
Submitted 22 January, 2021;
originally announced January 2021.
-
Gradient Descent-based D-optimal Design for the Least-Squares Polynomial Approximation
Authors:
V. P. Zankin,
G. V. Ryzhakov,
I. V. Oseledets
Abstract:
In this work, we propose a novel sampling method for Design of Experiments. This method allows to sample such input values of the parameters of a computational model for which the constructed surrogate model will have the least possible approximation error. High efficiency of the proposed method is demonstrated by its comparison with other sampling techniques (LHS, Sobol' sequence sampling, and Ma…
▽ More
In this work, we propose a novel sampling method for Design of Experiments. This method allows to sample such input values of the parameters of a computational model for which the constructed surrogate model will have the least possible approximation error. High efficiency of the proposed method is demonstrated by its comparison with other sampling techniques (LHS, Sobol' sequence sampling, and Maxvol sampling) on the problem of least-squares polynomial approximation. Also, numerical experiments for the Lebesgue constant growth for the points sampled by the proposed method are carried out.
△ Less
Submitted 2 October, 2018; v1 submitted 18 June, 2018;
originally announced June 2018.
-
Function approximation using gradient information with application to parametric and stochastic differential equations
Authors:
Gleb Ryzhakov,
Ivan Oseledets
Abstract:
In the paper we consider the problem of multivariate function approximation in polynomial basis. In order to solve this problem, we adjust the least squares method (LSM) by adding information about derivatives of the function. This modification allows reducing the number of evaluations of approximating function while kee** the accuracy at the appropriate level. We propose several techniques for…
▽ More
In the paper we consider the problem of multivariate function approximation in polynomial basis. In order to solve this problem, we adjust the least squares method (LSM) by adding information about derivatives of the function. This modification allows reducing the number of evaluations of approximating function while kee** the accuracy at the appropriate level. We propose several techniques for time-efficient calculation of derivatives in various applications. Numerical examples are given for comparison between the standard LSM and the proposed approach.
△ Less
Submitted 5 February, 2018;
originally announced February 2018.
