Showing 1–46 of 46 results for author: Dolgov, S

Tensor Quantum Programming

Authors: A. Termanova, Ar. Melnikov, E. Mamenchikov, N. Belokonev, S. Dolgov, A. Berezutskii, R. Ellerbrock, C. Mansell, M. Perelshtein

Abstract: Running quantum algorithms often involves implementing complex quantum circuits with such a large number of multi-qubit gates that the challenge of tackling practical applications appears daunting. To date, no experiments have successfully demonstrated a quantum advantage due to the ease with which the results can be adequately replicated on classical computers through the use of tensor network al… ▽ More Running quantum algorithms often involves implementing complex quantum circuits with such a large number of multi-qubit gates that the challenge of tackling practical applications appears daunting. To date, no experiments have successfully demonstrated a quantum advantage due to the ease with which the results can be adequately replicated on classical computers through the use of tensor network algorithms. Additionally, it remains unclear even in theory where exactly these advantages are rooted within quantum systems because the logarithmic complexity commonly associated with quantum algorithms is also present in algorithms based on tensor networks. In this article, we propose a novel approach called Tensor Quantum Programming, which leverages tensor networks for hybrid quantum computing. Our key insight is that the primary challenge of algorithms based on tensor networks lies in their high ranks (bond dimensions). Quantum computing offers a potential solution to this challenge, as an ideal quantum computer can represent tensors with arbitrarily high ranks in contrast to classical counterparts, which indicates the way towards quantum advantage. While tensor-based vector-encoding and state-readout are known procedures, the matrix-encoding required for performing matrix-vector multiplications directly on quantum devices remained unsolved. Here, we developed an algorithm that encodes Matrix Product Operators into quantum circuits with a depth that depends linearly on the number of qubits. It demonstrates effectiveness on up to 50 qubits for several matrices frequently encountered in differential equations, optimization problems, and quantum chemistry. We view this work as an initial stride towards the creation of genuinely practical quantum algorithms. △ Less

Submitted 20 March, 2024; originally announced March 2024.

Comments: 17 pages, 13 figures

TQCompressor: improving tensor decomposition methods in neural networks via permutations

Authors: V. Abronin, A. Naumov, D. Mazur, D. Bystrov, K. Tsarova, Ar. Melnikov, I. Oseledets, S. Dolgov, R. Brasher, M. Perelshtein

Abstract: We introduce TQCompressor, a novel method for neural network model compression with improved tensor decompositions. We explore the challenges posed by the computational and storage demands of pre-trained language models in NLP tasks and propose a permutation-based enhancement to Kronecker decomposition. This enhancement makes it possible to reduce loss in model expressivity which is usually associ… ▽ More We introduce TQCompressor, a novel method for neural network model compression with improved tensor decompositions. We explore the challenges posed by the computational and storage demands of pre-trained language models in NLP tasks and propose a permutation-based enhancement to Kronecker decomposition. This enhancement makes it possible to reduce loss in model expressivity which is usually associated with factorization. We demonstrate this method applied to the GPT-2$_{small}$. The result of the compression is TQCompressedGPT-2 model, featuring 81 mln. parameters compared to 124 mln. in the GPT-2$_{small}$. We make TQCompressedGPT-2 publicly available. We further enhance the performance of the TQCompressedGPT-2 through a training strategy involving multi-step knowledge distillation, using only a 3.1% of the OpenWebText. TQCompressedGPT-2 surpasses DistilGPT-2 and KnGPT-2 in comparative evaluations, marking an advancement in the efficient and effective deployment of models in resource-constrained environments. △ Less

Submitted 29 January, 2024; originally announced January 2024.

Tensor product algorithms for inference of contact network from epidemiological data

Authors: Sergey Dolgov, Dmitry Savostyanov

Abstract: We consider a problem of inferring contact network from nodal states observed during an epidemiological process. In a black-box Bayesian optimisation framework this problem reduces to a discrete likelihood optimisation over the set of possible networks. The high dimensionality of this set, which grows quadratically with the number of network nodes, makes this optimisation computationally challengi… ▽ More We consider a problem of inferring contact network from nodal states observed during an epidemiological process. In a black-box Bayesian optimisation framework this problem reduces to a discrete likelihood optimisation over the set of possible networks. The high dimensionality of this set, which grows quadratically with the number of network nodes, makes this optimisation computationally challenging. Moreover, the computation of the likelihood of a network requires estimating probabilities of the observed data to realise during the evolution of the epidemiological process on this network. A stochastic simulation algorithm struggles to estimate rare events of observing the data (corresponding to the ground truth network) during the evolution with a significantly different network, and hence prevents optimisation of the likelihood. We replace the stochastic simulation with solving the chemical master equation for the probabilities of all network states. This equation also suffers from the curse of dimensionality due to the exponentially large number of network states. We overcome this by approximating the probability of states in the tensor-train decomposition. This enables fast and accurate computation of small probabilities and likelihoods. Numerical simulations demonstrate efficient black-box Bayesian inference of the network. △ Less

Submitted 26 January, 2024; originally announced January 2024.

MSC Class: 15A69; 34A30; 37N25; 60J28; 62F15; 65F55; 90B15; 95C42

Statistical Proper Orthogonal Decomposition for model reduction in feedback control

Authors: Sergey Dolgov, Dante Kalise, Luca Saluzzi

Abstract: Feedback control synthesis for nonlinear, parameter-dependent fluid flow control problems is considered. The optimal feedback law requires the solution of the Hamilton-Jacobi-Bellman (HJB) PDE suffering the curse of dimensionality. This is mitigated by Model Order Reduction (MOR) techniques, where the system is projected onto a lower-dimensional subspace, over which the feedback synthesis becomes… ▽ More Feedback control synthesis for nonlinear, parameter-dependent fluid flow control problems is considered. The optimal feedback law requires the solution of the Hamilton-Jacobi-Bellman (HJB) PDE suffering the curse of dimensionality. This is mitigated by Model Order Reduction (MOR) techniques, where the system is projected onto a lower-dimensional subspace, over which the feedback synthesis becomes feasible. However, existing MOR methods assume at least one relaxation of generality, that is, the system should be linear, or stable, or deterministic. We propose a MOR method called Statistical POD (SPOD), which is inspired by the Proper Orthogonal Decomposition (POD), but extends to more general systems. Random samples of the original dynamical system are drawn, treating time and initial condition as random variables similarly to possible parameters in the model, and employing a stabilizing closed-loop control. The reduced subspace is chosen to minimize the empirical risk, which is shown to estimate the expected risk of the MOR solution with respect to the distribution of all possible outcomes of the controlled system. This reduced model is then used to compute a surrogate of the feedback control function in the Tensor Train (TT) format that is computationally fast to evaluate online. Using unstable Burgers' and Navier-Stokes equations, it is shown that the SPOD control is more accurate than Linear Quadratic Regulator or optimal control derived from a model reduced onto the standard POD basis, and faster than the direct optimal control of the original system. △ Less

Submitted 27 November, 2023; originally announced November 2023.

Numerical solution of the incompressible Navier-Stokes equations for chemical mixers via quantum-inspired Tensor Train Finite Element Method

Authors: Egor Kornev, Sergey Dolgov, Karan Pinto, Markus Pflitsch, Michael Perelshtein, Artem Melnikov

Abstract: The solution of computational fluid dynamics problems is one of the most computationally hard tasks, especially in the case of complex geometries and turbulent flow regimes. We propose to use Tensor Train (TT) methods, which possess logarithmic complexity in problem size and have great similarities with quantum algorithms in the structure of data representation. We develop the Tensor train Finite… ▽ More The solution of computational fluid dynamics problems is one of the most computationally hard tasks, especially in the case of complex geometries and turbulent flow regimes. We propose to use Tensor Train (TT) methods, which possess logarithmic complexity in problem size and have great similarities with quantum algorithms in the structure of data representation. We develop the Tensor train Finite Element Method -- TetraFEM -- and the explicit numerical scheme for the solution of the incompressible Navier-Stokes equation via Tensor Trains. We test this approach on the simulation of liquids mixing in a T-shape mixer, which, to our knowledge, was done for the first time using tensor methods in such non-trivial geometries. As expected, we achieve exponential compression in memory of all FEM matrices and demonstrate an exponential speed-up compared to the conventional FEM implementation on dense meshes. In addition, we discuss the possibility of extending this method to a quantum computer to solve more complex problems. This paper is based on work we conducted for Evonik Industries AG. △ Less

Submitted 23 May, 2023; v1 submitted 18 May, 2023; originally announced May 2023.

Self-reinforced polynomial approximation methods for concentrated probability densities

Authors: Tiangang Cui, Sergey Dolgov, Olivier Zahm

Abstract: Transport map methods offer a powerful statistical learning tool that can couple a target high-dimensional random variable with some reference random variable using invertible transformations. This paper presents new computational techniques for building the Knothe--Rosenblatt (KR) rearrangement based on general separable functions. We first introduce a new construction of the KR rearrangement --… ▽ More Transport map methods offer a powerful statistical learning tool that can couple a target high-dimensional random variable with some reference random variable using invertible transformations. This paper presents new computational techniques for building the Knothe--Rosenblatt (KR) rearrangement based on general separable functions. We first introduce a new construction of the KR rearrangement -- with guaranteed invertibility in its numerical implementation -- based on approximating the density of the target random variable using tensor-product spectral polynomials and downward closed sparse index sets. Compared to other constructions of KR arrangements based on either multi-linear approximations or nonlinear optimizations, our new construction only relies on a weighted least square approximation procedure. Then, inspired by the recently developed deep tensor trains (Cui and Dolgov, Found. Comput. Math. 22:1863--1922, 2022), we enhance the approximation power of sparse polynomials by preconditioning the density approximation problem using compositions of maps. This is particularly suitable for high-dimensional and concentrated probability densities commonly seen in many applications. We approximate the complicated target density by a composition of self-reinforced KR rearrangements, in which previously constructed KR rearrangements -- based on the same approximation ansatz -- are used to precondition the density approximation problem for building each new KR rearrangement. We demonstrate the efficiency of our proposed methods and the importance of using the composite map on several inverse problems governed by ordinary differential equations (ODEs) and partial differential equations (PDEs). △ Less

Submitted 4 March, 2023; originally announced March 2023.

A weighted subspace exponential kernel for support tensor machines

Authors: Kirandeep Kour, Sergey Dolgov, Peter Benner, Martin Stoll, Max Pfeffer

Abstract: High-dimensional data in the form of tensors are challenging for kernel classification methods. To both reduce the computational complexity and extract informative features, kernels based on low-rank tensor decompositions have been proposed. However, what decisive features of the tensors are exploited by these kernels is often unclear. In this paper we propose a novel kernel that is based on the T… ▽ More High-dimensional data in the form of tensors are challenging for kernel classification methods. To both reduce the computational complexity and extract informative features, kernels based on low-rank tensor decompositions have been proposed. However, what decisive features of the tensors are exploited by these kernels is often unclear. In this paper we propose a novel kernel that is based on the Tucker decomposition. For this kernel the Tucker factors are computed based on re-weighting of the Tucker matrices with tuneable powers of singular values from the HOSVD decomposition. This provides a mechanism to balance the contribution of the Tucker core and factors of the data. We benchmark support tensor machines with this new kernel on several datasets. First we generate synthetic data where two classes differ in either Tucker factors or core, and compare our novel and previously existing kernels. We show robustness of the new kernel with respect to both classification scenarios. We further test the new method on real-world datasets. The proposed kernel has demonstrated a higher test accuracy than the state-of-the-art tensor train multi-way multi-level kernel, and a significantly lower computational time. △ Less

Submitted 16 February, 2023; originally announced February 2023.

Smoothed Moreau-Yosida Tensor Train Approximation of State-constrained Optimization Problems under Uncertainty

Authors: Harbir Antil, Sergey Dolgov, Akwum Onwunta

Abstract: We propose an algorithm to solve optimization problems constrained by partial (ordinary) differential equations under uncertainty, with almost sure constraints on the state variable. To alleviate the computational burden of high-dimensional random variables, we approximate all random fields by the tensor-train decomposition. To enable efficient tensor-train approximation of the state constraints,… ▽ More We propose an algorithm to solve optimization problems constrained by partial (ordinary) differential equations under uncertainty, with almost sure constraints on the state variable. To alleviate the computational burden of high-dimensional random variables, we approximate all random fields by the tensor-train decomposition. To enable efficient tensor-train approximation of the state constraints, the latter are handled using the Moreau-Yosida penalty, with an additional smoothing of the positive part (plus/ReLU) function by a softplus function. In a special case of a quadratic cost minimization constrained by linear elliptic partial differential equations, and some additional constraint qualification, we prove strong convergence of the regularized solution to the optimal control. This result also proposes a practical recipe for selecting the smoothing parameter as a function of the penalty parameter. We develop a second order Newton type method with a fast matrix-free action of the approximate Hessian to solve the smoothed Moreau-Yosida problem. This algorithm is tested on benchmark elliptic problems with random coefficients, optimization problems constrained by random elliptic variational inequalities, and a real-world epidemiological model with 20 random variables. These examples demonstrate mild (at most polynomial) scaling with respect to the dimension and regularization parameters. △ Less

Submitted 4 July, 2024; v1 submitted 20 January, 2023; originally announced January 2023.

Comments: 28 pages

Tensor product approach to modelling epidemics on networks

Authors: Sergey V. Dolgov, Dmitry V. Savostyanov

Abstract: To improve mathematical models of epidemics it is essential to move beyond the traditional assumption of homogeneous well--mixed population and involve more precise information on the network of contacts and transport links by which a stochastic process of the epidemics spreads. In general, the number of states of the network grows exponentially with its size, and a master equation description suf… ▽ More To improve mathematical models of epidemics it is essential to move beyond the traditional assumption of homogeneous well--mixed population and involve more precise information on the network of contacts and transport links by which a stochastic process of the epidemics spreads. In general, the number of states of the network grows exponentially with its size, and a master equation description suffers from the curse of dimensionality. Almost all methods widely used in practice are versions of the stochastic simulation algorithm (SSA), which is notoriously known for its slow convergence. In this paper we numerically solve the chemical master equation for an SIR model on a general network using recently proposed tensor product algorithms. In numerical experiments we show that tensor product algorithms converge much faster than SSA and deliver more accurate results, which becomes particularly important for uncovering the probabilities of rare events, e.g. for number of infected people to exceed a (high) threshold. △ Less

Submitted 30 August, 2022; originally announced September 2022.

MSC Class: 15A69; 34A30; 37N25; 60J28; 65F55; 90B15; 95C42

Journal ref: Applied Mathematics and Computation 460:128290, 2024

Deep importance sampling using tensor trains with application to a priori and a posteriori rare event estimation

Authors: Tiangang Cui, Sergey Dolgov, Robert Scheichl

Abstract: We propose a deep importance sampling method that is suitable for estimating rare event probabilities in high-dimensional problems. We approximate the optimal importance distribution in a general importance sampling problem as the pushforward of a reference distribution under a composition of order-preserving transformations, in which each transformation is formed by a squared tensor-train decompo… ▽ More We propose a deep importance sampling method that is suitable for estimating rare event probabilities in high-dimensional problems. We approximate the optimal importance distribution in a general importance sampling problem as the pushforward of a reference distribution under a composition of order-preserving transformations, in which each transformation is formed by a squared tensor-train decomposition. The squared tensor-train decomposition provides a scalable ansatz for building order-preserving high-dimensional transformations via density approximations. The use of composition of maps moving along a sequence of bridging densities alleviates the difficulty of directly approximating concentrated density functions. To compute expectations over unnormalized probability distributions, we design a ratio estimator that estimates the normalizing constant using a separate importance distribution, again constructed via a composition of transformations in tensor-train format. This offers better theoretical variance reduction compared with self-normalized importance sampling, and thus opens the door to efficient computation of rare event probabilities in Bayesian inference problems. Numerical experiments on problems constrained by differential equations show little to no increase in the computational complexity with the event probability going to zero, and allow to compute hitherto unattainable estimates of rare event probabilities for complex, high-dimensional posterior densities. △ Less

Submitted 24 May, 2023; v1 submitted 5 September, 2022; originally announced September 2022.

Undersampling Raster Scans in Spectromicroscopy for reduced dose and faster measurements

Authors: Oliver Townsend, Silvia Gazzola, Sergey Dolgov, Paul Quinn

Abstract: Combinations of spectroscopic analysis and microscopic techniques are used across many disciplines of scientific research, including material science, chemistry and biology. X-ray spectromicroscopy, in particular, is a powerful tool used for studying chemical state distributions at the micro and nano scales. With the beam fixed, a specimen is typically rastered through the probe with continuous mo… ▽ More Combinations of spectroscopic analysis and microscopic techniques are used across many disciplines of scientific research, including material science, chemistry and biology. X-ray spectromicroscopy, in particular, is a powerful tool used for studying chemical state distributions at the micro and nano scales. With the beam fixed, a specimen is typically rastered through the probe with continuous motion and a range of multimodal data is collected at fixed time intervals. The application of this technique is limited in some areas due to: long scanning times to collect the data, either because of the area/volume under study or the compositional properties of the specimen; and material degradation due to the dose absorbed during the measurement. In this work, we propose a novel approach for reducing the dose and scanning times by undersampling the raster data. This is achieved by skip** rows within scans and reconstructing the x-ray spectromicroscopic measurements using low-rank matrix completion. The new method is robust and allows for x 5-6 reduction in sampling. Experimental results obtained on real data are illustrated. △ Less

Submitted 16 October, 2023; v1 submitted 15 August, 2022; originally announced August 2022.

Comments: 22 pages,11 figures

Data-driven Tensor Train Gradient Cross Approximation for Hamilton-Jacobi-Bellman Equations

Authors: Sergey Dolgov, Dante Kalise, Luca Saluzzi

Abstract: A gradient-enhanced functional tensor train cross approximation method for the resolution of the Hamilton-Jacobi-Bellman (HJB) equations associated to optimal feedback control of nonlinear dynamics is presented. The procedure uses samples of both the solution of the HJB equation and its gradient to obtain a tensor train approximation of the value function. The collection of the data for the algori… ▽ More A gradient-enhanced functional tensor train cross approximation method for the resolution of the Hamilton-Jacobi-Bellman (HJB) equations associated to optimal feedback control of nonlinear dynamics is presented. The procedure uses samples of both the solution of the HJB equation and its gradient to obtain a tensor train approximation of the value function. The collection of the data for the algorithm is based on two possible techniques: Pontryagin Maximum Principle and State Dependent Riccati Equations. Several numerical tests are presented in low and high dimension showing the effectiveness of the proposed method and its robustness with respect to inexact data evaluations, provided by the gradient information. The resulting tensor train approximation paves the way towards fast synthesis of the control signal in real-time applications. △ Less

Submitted 22 February, 2023; v1 submitted 10 May, 2022; originally announced May 2022.

MSC Class: 15A69; 15A23; 65F10; 65N22; 49J20; 49LXX; 49MXX

Optimizing semilinear representations for State-dependent Riccati Equation-based feedback control

Authors: Sergey Dolgov, Dante Kalise, Luca Saluzzi

Abstract: An optimized variant of the State Dependent Riccati Equations (SDREs) approach for nonlinear optimal feedback stabilization is presented. The proposed method is based on the construction of equivalent semilinear representations associated to the dynamics and their affine combination. The optimal combination is chosen to minimize the discrepancy between the SDRE control and the optimal feedback law… ▽ More An optimized variant of the State Dependent Riccati Equations (SDREs) approach for nonlinear optimal feedback stabilization is presented. The proposed method is based on the construction of equivalent semilinear representations associated to the dynamics and their affine combination. The optimal combination is chosen to minimize the discrepancy between the SDRE control and the optimal feedback law stemming from the solution of the corresponding Hamilton Jacobi Bellman (HJB) equation. Numerical experiments assess effectiveness of the method in terms of stability of the closed-loop with near-to-optimal performance. △ Less

Submitted 19 July, 2022; v1 submitted 23 February, 2022; originally announced February 2022.

TTRISK: Tensor Train Decomposition Algorithm for Risk Averse Optimization

Authors: Harbir Antil, Sergey Dolgov, Akwum Onwunta

Abstract: This article develops a new algorithm named TTRISK to solve high-dimensional risk-averse optimization problems governed by differential equations (ODEs and/or PDEs) under uncertainty. As an example, we focus on the so-called Conditional Value at Risk (CVaR), but the approach is equally applicable to other coherent risk measures. Both the full and reduced space formulations are considered. The algo… ▽ More This article develops a new algorithm named TTRISK to solve high-dimensional risk-averse optimization problems governed by differential equations (ODEs and/or PDEs) under uncertainty. As an example, we focus on the so-called Conditional Value at Risk (CVaR), but the approach is equally applicable to other coherent risk measures. Both the full and reduced space formulations are considered. The algorithm is based on low rank tensor approximations of random fields discretized using stochastic collocation. To avoid non-smoothness of the objective function underpinning the CVaR, we propose an adaptive strategy to select the width parameter of the smoothed CVaR to balance the smoothing and tensor approximation errors. Moreover, unbiased Monte Carlo CVaR estimate can be computed by using the smoothed CVaR as a control variate. To accelerate the computations, we introduce an efficient preconditioner for the KKT system in the full space formulation.The numerical experiments demonstrate that the proposed method enables accurate CVaR optimization constrained by large-scale discretized systems. In particular, the first example consists of an elliptic PDE with random coefficients as constraints. The second example is motivated by a realistic application to devise a lockdown plan for United Kingdom under COVID-19. The results indicate that the risk-averse framework is feasible with the tensor approximations under tens of random variables. △ Less

Submitted 1 December, 2022; v1 submitted 9 November, 2021; originally announced November 2021.

Comments: 28 pages

MSC Class: 93E20; 49J55; 49K20; 49K45; 90C15; 65D15; 15A69; 15A23

A Quantum Inspired Approach to Exploit Turbulence Structures

Authors: Nikita Gourianov, Michael Lubasch, Sergey Dolgov, Quincy Y. van den Berg, Hessam Babaee, Peyman Givi, Martin Kiffner, Dieter Jaksch

Abstract: Understanding turbulence is the key to our comprehension of many natural and technological flow processes. At the heart of this phenomenon lies its intricate multi-scale nature, describing the coupling between different-sized eddies in space and time. Here we introduce a new paradigm for analyzing the structure of turbulent flows by quantifying correlations between different length scales using me… ▽ More Understanding turbulence is the key to our comprehension of many natural and technological flow processes. At the heart of this phenomenon lies its intricate multi-scale nature, describing the coupling between different-sized eddies in space and time. Here we introduce a new paradigm for analyzing the structure of turbulent flows by quantifying correlations between different length scales using methods inspired from quantum many-body physics. We present results for interscale correlations of two paradigmatic flow examples, and use these insights along with tensor network theory to design a structure-resolving algorithm for simulating turbulent flows. With this algorithm, we find that the incompressible Navier-Stokes equations can be accurately solved within a computational space reduced by over an order of magnitude compared to direct numerical simulation. Our quantum-inspired approach provides a pathway towards conducting computational fluid dynamics on quantum computers. △ Less

Submitted 4 July, 2022; v1 submitted 10 June, 2021; originally announced June 2021.

Comments: Newest and final version of our article

Journal ref: Nature Computational Science (2022)

Conditional Deep Inverse Rosenblatt Transports

Authors: Tiangang Cui, Sergey Dolgov, Olivier Zahm

Abstract: We present a novel offline-online method to mitigate the computational burden of Bayesian inference, particularly in the regime where the posterior densities are computationally demanding to evaluate while real-time inference results are needed. In the offline phase, the proposed method learns the joint law of the parameter random variables and the observable random variables in the tensor-train (… ▽ More We present a novel offline-online method to mitigate the computational burden of Bayesian inference, particularly in the regime where the posterior densities are computationally demanding to evaluate while real-time inference results are needed. In the offline phase, the proposed method learns the joint law of the parameter random variables and the observable random variables in the tensor-train (TT) format. Then, in the online phase, the resulting order-preserving transport can be conditioned on newly observed data to characterize the posterior random variables in real-time. Compared with the state-of-the-art normalizing flows techniques, our proposed method relies on function approximation, for which we can provide a thorough performance analysis. The function approximation perspective allows us to significantly improve the capability of transport maps in challenging problems with high-dimensional observations and high-dimensional parameters. Capitalizing on this, we present novel heuristics to either reorder or reparametrize the variables to enhance the approximation power of TT. We then integrate the TT-based transport maps and the parameter reordering/reparametrization into a layered composite map to further improve the performance of the resulting inference. We demonstrate the efficiency of the proposed method on various statistical learning tasks involving ordinary differential equations (ODEs) and partial differential equations (PDEs). △ Less

Submitted 28 January, 2022; v1 submitted 8 June, 2021; originally announced June 2021.

Comments: 41 pages

Functional Tucker approximation using Chebyshev interpolation

Authors: Sergey Dolgov, Daniel Kressner, Christoph Strössner

Abstract: This work is concerned with approximating a trivariate function defined on a tensor-product domain via function evaluations. Combining tensorized Chebyshev interpolation with a Tucker decomposition of low multilinear rank yields function approximations that can be computed and stored very efficiently. The existing Chebfun3 algorithm [Hashemi and Trefethen, SIAM J. Sci. Comput., 39 (2017)]uses a si… ▽ More This work is concerned with approximating a trivariate function defined on a tensor-product domain via function evaluations. Combining tensorized Chebyshev interpolation with a Tucker decomposition of low multilinear rank yields function approximations that can be computed and stored very efficiently. The existing Chebfun3 algorithm [Hashemi and Trefethen, SIAM J. Sci. Comput., 39 (2017)]uses a similar format but the construction of the approximation proceeds indirectly, via a so called slice-Tucker decomposition. As a consequence, Chebfun3 sometimes uses unnecessarily many function evaluations and does not fully benefit from the potential of the Tucker decomposition to reduce, sometimes dramatically, the computational cost. We propose a novel algorithm Chebfun3F that utilizes univariate fibers instead of bivariate slices to construct the Tucker decomposition. Chebfun3F reduces the cost for the approximation in terms of the number of function evaluations for nearly all functions considered, typically by 75%, and sometimes by over 98%. △ Less

Submitted 23 January, 2023; v1 submitted 31 July, 2020; originally announced July 2020.

MSC Class: 15A69; 41A10; 41A63; 65D15

Deep composition of tensor-trains using squared inverse Rosenblatt transports

Authors: Tiangang Cui, Sergey Dolgov

Abstract: Characterising intractable high-dimensional random variables is one of the fundamental challenges in stochastic computation. The recent surge of transport maps offers a mathematical foundation and new insights for tackling this challenge by coupling intractable random variables with tractable reference random variables. This paper generalises the functional tensor-train approximation of the invers… ▽ More Characterising intractable high-dimensional random variables is one of the fundamental challenges in stochastic computation. The recent surge of transport maps offers a mathematical foundation and new insights for tackling this challenge by coupling intractable random variables with tractable reference random variables. This paper generalises the functional tensor-train approximation of the inverse Rosenblatt transport recently developed by Dolgov et al. (Stat Comput 30:603--625, 2020) to a wide class of high-dimensional non-negative functions, such as unnormalised probability density functions. First, we extend the inverse Rosenblatt transform to enable the transport to general reference measures other than the uniform measure. We develop an efficient procedure to compute this transport from a squared tensor-train decomposition which preserves the monotonicity. More crucially, we integrate the proposed order-preserving functional tensor-train transport into a nested variable transformation framework inspired by the layered structure of deep neural networks. The resulting deep inverse Rosenblatt transport significantly expands the capability of tensor approximations and transport maps to random variables with complicated nonlinear interactions and concentrated density functions. We demonstrate the efficiency of the proposed approach on a range of applications in statistical learning and uncertainty quantification, including parameter estimation for dynamical systems and inverse problems constrained by partial differential equations. △ Less

Submitted 18 October, 2021; v1 submitted 14 July, 2020; originally announced July 2020.

Comments: Found Comput Math (2021)

Efficient Structure-preserving Support Tensor Train Machine

Authors: Kirandeep Kour, Sergey Dolgov, Martin Stoll, Peter Benner

Abstract: An increasing amount of collected data are high-dimensional multi-way arrays (tensors), and it is crucial for efficient learning algorithms to exploit this tensorial structure as much as possible. The ever-present curse of dimensionality for high dimensional data and the loss of structure when vectorizing the data motivates the use of tailored low-rank tensor classification methods. In the presenc… ▽ More An increasing amount of collected data are high-dimensional multi-way arrays (tensors), and it is crucial for efficient learning algorithms to exploit this tensorial structure as much as possible. The ever-present curse of dimensionality for high dimensional data and the loss of structure when vectorizing the data motivates the use of tailored low-rank tensor classification methods. In the presence of small amounts of training data, kernel methods offer an attractive choice as they provide the possibility for a nonlinear decision boundary. We develop the Tensor Train Multi-way Multi-level Kernel (TT-MMK), which combines the simplicity of the Canonical Polyadic decomposition, the classification power of the Dual Structure-preserving Support Vector Machine, and the reliability of the Tensor Train (TT) approximation. We show by experiments that the TT-MMK method is usually more reliable computationally, less sensitive to tuning parameters, and gives higher prediction accuracy in the SVM classification when benchmarked against other state-of-the-art techniques. △ Less

Submitted 3 August, 2021; v1 submitted 12 February, 2020; originally announced February 2020.

Comments: 20 pages, 5 figures, 2 table, 2 Algorithm

Rank Bounds for Approximating Gaussian Densities in the Tensor-Train Format

Authors: Paul B. Rohrbach, Sergey Dolgov, Lars Grasedyck, Robert Scheichl

Abstract: Low-rank tensor approximations have shown great potential for uncertainty quantification in high dimensions, for example, to build surrogate models that can be used to speed up large-scale inference problems (Eigel et al., Inverse Problems 34, 2018; Dolgov et al., Statistics & Computing 30, 2020). The feasibility and efficiency of such approaches depends critically on the rank that is necessary to… ▽ More Low-rank tensor approximations have shown great potential for uncertainty quantification in high dimensions, for example, to build surrogate models that can be used to speed up large-scale inference problems (Eigel et al., Inverse Problems 34, 2018; Dolgov et al., Statistics & Computing 30, 2020). The feasibility and efficiency of such approaches depends critically on the rank that is necessary to represent or approximate the underlying distribution. In this paper, a-priori rank bounds for approximations in the functional tensor-train representation for the case of Gaussian models are developed. It is shown that under suitable conditions on the precision matrix, the Gaussian density can be approximated to high accuracy without suffering from an exponential growth of complexity as the dimension increases. These results provide a rigorous justification of the suitability and the limitations of low-rank tensor methods in a simple but important model case. Numerical experiments confirm that the rank bounds capture the qualitative behavior of the rank structure when varying the parameters of the precision matrix and the accuracy of the approximation. Finally, the practical relevance of the theoretical results is demonstrated in the context of a Bayesian filtering problem. △ Less

Submitted 27 November, 2020; v1 submitted 22 January, 2020; originally announced January 2020.

Comments: 25 pages, 6 figures

MSC Class: 15A23; 15A69; 65C60; 65D32; 65D15; 41A10

Solving differential Riccati equations: A nonlinear space-time method using tensor trains

Authors: Tobias Breiten, Sergey Dolgov, Martin Stoll

Abstract: Differential algebraic Riccati equations are at the heart of many applications in control theory. They are time-depent, matrix-valued, and in particular nonlinear equations that require special methods for their solution. Low-rank methods have been used heavily computing a low-rank solution at every step of a time-discretization. We propose the use of an all-at-once space-time solution leading to… ▽ More Differential algebraic Riccati equations are at the heart of many applications in control theory. They are time-depent, matrix-valued, and in particular nonlinear equations that require special methods for their solution. Low-rank methods have been used heavily computing a low-rank solution at every step of a time-discretization. We propose the use of an all-at-once space-time solution leading to a large nonlinear space-time problem for which we propose the use of a Newton-Kleinman iteration. Approximating the space-time problem in low-rank form requires fewer applications of the discretized differential operator and gives a low-rank approximation to the overall solution. △ Less

Submitted 14 December, 2019; originally announced December 2019.

Comments: 20 pages

Parallel time-dependent variational principle algorithm for matrix product states

Authors: Paul Secular, Nikita Gourianov, Michael Lubasch, Sergey Dolgov, Stephen R. Clark, Dieter Jaksch

Abstract: Combining the time-dependent variational principle (TDVP) algorithm with the parallelization scheme introduced by Stoudenmire and White for the density matrix renormalization group (DMRG), we present the first parallel matrix product state (MPS) algorithm capable of time evolving one-dimensional (1D) quantum lattice systems with long-range interactions. We benchmark the accuracy and performance of… ▽ More Combining the time-dependent variational principle (TDVP) algorithm with the parallelization scheme introduced by Stoudenmire and White for the density matrix renormalization group (DMRG), we present the first parallel matrix product state (MPS) algorithm capable of time evolving one-dimensional (1D) quantum lattice systems with long-range interactions. We benchmark the accuracy and performance of the algorithm by simulating quenches in the long-range Ising and XY models. We show that our code scales well up to 32 processes, with parallel efficiencies as high as 86%. Finally, we calculate the dynamical correlation function of a 201-site Heisenberg XXX spin chain with $1/r^2$ interactions, which is challenging to compute sequentially. These results pave the way for the application of tensor networks to increasingly complex many-body systems. △ Less

Submitted 12 June, 2020; v1 submitted 12 December, 2019; originally announced December 2019.

Comments: Version accepted for publication in Phys. Rev. B. Text clarified and references updated. Main text: 11 pages, 13 figures. Appendices: 3 pages, 3 figures. Supplemental material: 4 pages, 3 figures

Journal ref: Phys. Rev. B 101, 235123 (2020)

Tensor Decomposition Methods for High-dimensional Hamilton-Jacobi-Bellman Equations

Authors: Sergey Dolgov, Dante Kalise, Karl Kunisch

Abstract: A tensor decomposition approach for the solution of high-dimensional, fully nonlinear Hamilton-Jacobi-Bellman equations arising in optimal feedback control of nonlinear dynamics is presented. The method combines a tensor train approximation for the value function together with a Newton-like iterative method for the solution of the resulting nonlinear system. The tensor approximation leads to a pol… ▽ More A tensor decomposition approach for the solution of high-dimensional, fully nonlinear Hamilton-Jacobi-Bellman equations arising in optimal feedback control of nonlinear dynamics is presented. The method combines a tensor train approximation for the value function together with a Newton-like iterative method for the solution of the resulting nonlinear system. The tensor approximation leads to a polynomial scaling with respect to the dimension, partially circumventing the curse of dimensionality. A convergence analysis for the linear-quadratic case is presented. For nonlinear dynamics, the effectiveness of the high-dimensional control synthesis method is assessed in the optimal feedback stabilization of the Allen-Cahn and Fokker-Planck equations with a hundred of variables. △ Less

Submitted 15 March, 2021; v1 submitted 5 August, 2019; originally announced August 2019.

Comments: 26pp, to appear in SIAM J. Sci. Comput

Guaranteed a posteriori error bounds for low rank tensor approximate solutions

Authors: Sergey Dolgov, Tomáš Vejchodský

Abstract: We propose a guaranteed and fully computable upper bound on the energy norm of the error in low-rank Tensor Train (TT) approximate solutions of (possibly) high dimensional reaction-diffusion problems. The error bound is obtained from Euler-Lagrange equations for a complementary flux reconstruction problem, which are solved in the low-rank TT representation using the block Alternating Linear Scheme… ▽ More We propose a guaranteed and fully computable upper bound on the energy norm of the error in low-rank Tensor Train (TT) approximate solutions of (possibly) high dimensional reaction-diffusion problems. The error bound is obtained from Euler-Lagrange equations for a complementary flux reconstruction problem, which are solved in the low-rank TT representation using the block Alternating Linear Scheme. This bound is guaranteed to be above the energy norm of the total error, including the discretization error, the tensor approximation error, and the error in the solver of linear algebraic equations, although quadrature errors, in general, can pollute its evaluation. Numerical examples with the Poisson equation and the Schroedinger equation with the Henon-Heiles potential in up to 40 dimensions are presented to illustrate the efficiency of this approach. △ Less

Submitted 2 April, 2020; v1 submitted 21 May, 2019; originally announced May 2019.

MSC Class: 65N15; 65N30; 15A69; 15A23; 65F10; 65N22

Kriging in Tensor Train data format

Authors: Sergey Dolgov, Alexander Litvinenko, Dishi Liu

Abstract: Combination of low-tensor rank techniques and the Fast Fourier transform (FFT) based methods had turned out to be prominent in accelerating various statistical operations such as Kriging, computing conditional covariance, geostatistical optimal design, and others. However, the approximation of a full tensor by its low-rank format can be computationally formidable. In this work, we incorporate the… ▽ More Combination of low-tensor rank techniques and the Fast Fourier transform (FFT) based methods had turned out to be prominent in accelerating various statistical operations such as Kriging, computing conditional covariance, geostatistical optimal design, and others. However, the approximation of a full tensor by its low-rank format can be computationally formidable. In this work, we incorporate the robust Tensor Train (TT) approximation of covariance matrices and the efficient TT-Cross algorithm into the FFT-based Kriging. It is shown that here the computational complexity of Kriging is reduced to $\mathcal{O}(d r^3 n)$, where $n$ is the mode size of the estimation grid, $d$ is the number of variables (the dimension), and $r$ is the rank of the TT approximation of the covariance matrix. For many popular covariance functions the TT rank $r$ remains stable for increasing $n$ and $d$. The advantages of this approach against those using plain FFT are demonstrated in synthetic and real data examples. △ Less

Submitted 21 April, 2019; originally announced April 2019.

Comments: 19 pages,4 figures, 1 table, UNCECOMP 2019 3rd International Conference on Uncertainty Quantification in Computational Sciences and Engineering 24-26 June 2019, Crete, Greece https://2019.uncecomp.org/

Parallel cross interpolation for high-precision calculation of high-dimensional integrals

Authors: Sergey Dolgov, Dmitry Savostyanov

Abstract: We propose a parallel version of the cross interpolation algorithm and apply it to calculate high-dimensional integrals motivated by Ising model in quantum physics. In contrast to mainstream approaches, such as Monte Carlo and quasi Monte Carlo, the samples calculated by our algorithm are neither random nor form a regular lattice. Instead we calculate the given function along individual dimensions… ▽ More We propose a parallel version of the cross interpolation algorithm and apply it to calculate high-dimensional integrals motivated by Ising model in quantum physics. In contrast to mainstream approaches, such as Monte Carlo and quasi Monte Carlo, the samples calculated by our algorithm are neither random nor form a regular lattice. Instead we calculate the given function along individual dimensions (modes) and use this data to reconstruct its behaviour in the whole domain. The positions of the calculated univariate fibers are chosen adaptively for the given function. The required evaluations can be executed in parallel both along each mode (variable) and over all modes. To demonstrate the efficiency of the proposed method, we apply it to compute high-dimensional Ising susceptibility integrals, arising from asymptotic expansions for the spontaneous magnetisation in two-dimensional Ising model of ferromagnetism. We observe strong superlinear convergence of the proposed method, while the MC and qMC algorithms converge sublinearly. Using multiple precision arithmetic, we also observed exponential convergence of the proposed algorithm. Combining high-order convergence, almost perfect scalability up to hundreds of processes, and the same flexibility as MC and qMC, the proposed algorithm can be a new method of choice for problems involving high-dimensional integration, e.g. in statistics, probability, and quantum physics. △ Less

Submitted 27 March, 2019; originally announced March 2019.

MSC Class: 15A69; 15A23; 65D05; 65F99

Journal ref: Computer Physics Communications, 2019

Tensor product approach to quantum control

Authors: Diego Quiñones Valles, Sergey Dolgov, Dmitry Savostyanov

Abstract: In this proof-of-concept paper we show that tensor product approach is efficient for control of large quantum systems, such as Heisenberg spin wires, which are essential for emerging quantum computing technologies. We compute optimal control sequences using GRAPE method, applying the recently developed tAMEn algorithm to calculate evolution of quantum states represented in the tensor train format… ▽ More In this proof-of-concept paper we show that tensor product approach is efficient for control of large quantum systems, such as Heisenberg spin wires, which are essential for emerging quantum computing technologies. We compute optimal control sequences using GRAPE method, applying the recently developed tAMEn algorithm to calculate evolution of quantum states represented in the tensor train format to reduce storage. Using tensor product algorithms we can overcome the curse of dimensionality and compute the optimal control pulse for a 41 spin system on a single workstation with fully controlled accuracy and huge savings of computational time and memory. The use of tensor product algorithms opens new approaches for development of quantum computers with 50 to 100 qubits. △ Less

Submitted 4 March, 2019; v1 submitted 28 February, 2019; originally announced March 2019.

Comments: To appear in Proc. IMSE 2018

MSC Class: 65F30; 81Q93

Journal ref: in Integral Methods in Science and Engineering, eds: Constanda C., Harris P., 2019

A low-rank tensor method for PDE-constrained optimization with isogeometric analysis

Authors: Alexandra Bünger, Sergey Dolgov, Martin Stoll

Abstract: Isogeometric analysis (IGA) has become one of the most popular methods for the discretization of partial differential equations motivated by the use of NURBS for geometric representations in industry and science. A crucial challenge lies in the solution of the discretized equations, which we discuss in this talk with a particular focus on PDE-constrained optimization discretized using IGA. The dis… ▽ More Isogeometric analysis (IGA) has become one of the most popular methods for the discretization of partial differential equations motivated by the use of NURBS for geometric representations in industry and science. A crucial challenge lies in the solution of the discretized equations, which we discuss in this talk with a particular focus on PDE-constrained optimization discretized using IGA. The discretization results in a system of large mass and stiffness matrices, which are typically very costly to assemble. To reduce the computation time and storage requirements, low-rank tensor methods have become a promising tool. We present a framework for the assembly of these matrices in low-rank form as the sum of a small number of Kronecker products. For assembly of the smaller matrices only univariate integration is required. The resulting low rank Kronecker product structure of the mass and stiffness matrices can be used to solve a PDE-constrained optimization problem without assembling the actual system matrices. We present a framework which preserves and exploits the low-rank Kronecker product format for both the matrices and the solution. We use the block AMEn method to efficiently solve the corresponding KKT system of the optimization problem. We show several numerical experiments with 3D geometries to demonstrate that the low-rank assembly and solution drastically reduces the memory demands and computing times, depending on the approximation ranks of the domain. △ Less

Submitted 23 November, 2018; v1 submitted 16 November, 2018; originally announced November 2018.

Comments: 23 pages, 9 figures

Approximation and sampling of multivariate probability distributions in the tensor train decomposition

Authors: Sergey Dolgov, Karim Anaya-Izquierdo, Colin Fox, Robert Scheichl

Abstract: General multivariate distributions are notoriously expensive to sample from, particularly the high-dimensional posterior distributions in PDE-constrained inverse problems. This paper develops a sampler for arbitrary continuous multivariate distributions that is based on low-rank surrogates in the tensor-train format. We construct a tensor-train approximation to the target probability density funct… ▽ More General multivariate distributions are notoriously expensive to sample from, particularly the high-dimensional posterior distributions in PDE-constrained inverse problems. This paper develops a sampler for arbitrary continuous multivariate distributions that is based on low-rank surrogates in the tensor-train format. We construct a tensor-train approximation to the target probability density function using the cross interpolation, which requires a small number of function evaluations. For sufficiently smooth distributions the storage required for the TT approximation is moderate, scaling linearly with dimension. The structure of the tensor-train surrogate allows efficient sampling by the conditional distribution method. Unbiased estimates may be calculated by correcting the transformed random seeds using a Metropolis--Hastings accept/reject step. Moreover, one can use a more efficient quasi-Monte Carlo quadrature that may be corrected either by a control-variate strategy, or by importance weighting. We show that the error in the tensor-train approximation propagates linearly into the Metropolis--Hastings rejection rate and the integrated autocorrelation time of the resulting Markov chain. These methods are demonstrated in three computed examples: fitting failure time of shock absorbers; a PDE-constrained inverse diffusion problem; and sampling from the Rosenbrock distribution. The delayed rejection adaptive Metropolis (DRAM) algorithm is used as a benchmark. We find that the importance-weight corrected quasi-Monte Carlo quadrature performs best in all computed examples, and is orders-of-magnitude more efficient than DRAM across a wide range of approximation accuracies and sample sizes. Indeed, all the methods developed here significantly outperform DRAM in all computed examples. △ Less

Submitted 3 July, 2019; v1 submitted 2 October, 2018; originally announced October 2018.

Comments: 32 pages

MSC Class: 65D15; 65D32; 65C05; 65C40; 65C60; 62F15; 15A69; 15A23

Greedy low-rank algorithm for spatial connectome regression

Authors: Patrick Kürschner, Sergey Dolgov, Kameron Decker Harris, Peter Benner

Abstract: Recovering brain connectivity from tract tracing data is an important computational problem in the neurosciences. Mesoscopic connectome reconstruction was previously formulated as a structured matrix regression problem (Harris et al., 2016), but existing techniques do not scale to the whole-brain setting. The corresponding matrix equation is challenging to solve due to large scale, ill-conditionin… ▽ More Recovering brain connectivity from tract tracing data is an important computational problem in the neurosciences. Mesoscopic connectome reconstruction was previously formulated as a structured matrix regression problem (Harris et al., 2016), but existing techniques do not scale to the whole-brain setting. The corresponding matrix equation is challenging to solve due to large scale, ill-conditioning, and a general form that lacks a convergent splitting. We propose a greedy low-rank algorithm for connectome reconstruction problem in very high dimensions. The algorithm approximates the solution by a sequence of rank-one updates which exploit the sparse and positive definite problem structure. This algorithm was described previously (Kressner and Sirković, 2015) but never implemented for this connectome problem, leading to a number of challenges. We have had to design judicious stop** criteria and employ efficient solvers for the three main sub-problems of the algorithm, including an efficient GPU implementation that alleviates the main bottleneck for large datasets. The performance of the method is evaluated on three examples: an artificial "toy" dataset and two whole-cortex instances using data from the Allen Mouse Brain Connectivity Atlas. We find that the method is significantly faster than previous methods and that moderate ranks offer good approximation. This speedup allows for the estimation of increasingly large-scale connectomes across taxa as these data become available from tracing experiments. The data and code are available online. △ Less

Submitted 1 November, 2019; v1 submitted 16 August, 2018; originally announced August 2018.

MSC Class: 15A24; 15A83; 65F10 92C20; 94A08

Preconditioners and Tensor Product Solvers for Optimal Control Problems from Chemotaxis

Authors: Sergey Dolgov, John W. Pearson

Abstract: In this paper, we consider the fast numerical solution of an optimal control formulation of the Keller--Segel model for bacterial chemotaxis. Upon discretization, this problem requires the solution of huge-scale saddle point systems to guarantee accurate solutions. We consider the derivation of effective preconditioners for these matrix systems, which may be embedded within suitable iterative meth… ▽ More In this paper, we consider the fast numerical solution of an optimal control formulation of the Keller--Segel model for bacterial chemotaxis. Upon discretization, this problem requires the solution of huge-scale saddle point systems to guarantee accurate solutions. We consider the derivation of effective preconditioners for these matrix systems, which may be embedded within suitable iterative methods to accelerate their convergence. We also construct low-rank tensor-train techniques which enable us to present efficient and feasible algorithms for problems that are finely discretized in the space and time variables. Numerical results demonstrate that the number of preconditioned GMRES iterations depends mildly on the model parameters. Moreover, the low-rank solver makes the computing time and memory costs sublinear in the original problem size. △ Less

Submitted 22 June, 2018; originally announced June 2018.

Comments: 23 pages

MSC Class: 35Q93; 65F08; 65F10; 65N22; 92C17

A hybrid Alternating Least Squares -- TT Cross algorithm for parametric PDEs

Authors: Sergey Dolgov, Robert Scheichl

Abstract: We consider the approximate solution of parametric PDEs using the low-rank Tensor Train (TT) decomposition. Such parametric PDEs arise for example in uncertainty quantification problems in engineering applications. We propose an algorithm that is a hybrid of the alternating least squares and the TT cross methods. It computes a TT approximation of the whole solution, which is beneficial when multip… ▽ More We consider the approximate solution of parametric PDEs using the low-rank Tensor Train (TT) decomposition. Such parametric PDEs arise for example in uncertainty quantification problems in engineering applications. We propose an algorithm that is a hybrid of the alternating least squares and the TT cross methods. It computes a TT approximation of the whole solution, which is beneficial when multiple quantities of interest are sought. This might be needed, for example, for the computation of the probability density function (PDF) via the maximum entropy method [Kavehrad and Joseph, IEEE Trans. Comm., 1986]. The new algorithm exploits and preserves the block diagonal structure of the discretized operator in stochastic collocation schemes. This disentangles computations of the spatial and parametric degrees of freedom in the TT representation. In particular, it only requires solving independent PDEs at a few parameter values, thus allowing the use of existing high performance PDE solvers. In our numerical experiments, we apply the new algorithm to the stochastic diffusion equation and compare it with preconditioned steepest descent in the TT format, as well as with (multilevel) quasi-Monte Carlo and dimension-adaptive sparse grids methods. For sufficiently smooth random fields the new approach is orders of magnitude faster. △ Less

Submitted 5 July, 2018; v1 submitted 14 July, 2017; originally announced July 2017.

Comments: 29 pages

MSC Class: 65F10; 65F30; 65N22; 65N30; 65N35

Solving optimal control problems governed by random Navier-Stokes equations using low-rank methods

Authors: Peter Benner, Sergey Dolgov, Akwum Onwunta, Martin Stoll

Abstract: Many problems in computational science and engineering are simultaneously characterized by the following challenging issues: uncertainty, nonlinearity, nonstationarity and high dimensionality. Existing numerical techniques for such models would typically require considerable computational and storage resources. This is the case, for instance, for an optimization problem governed by time-dependent… ▽ More Many problems in computational science and engineering are simultaneously characterized by the following challenging issues: uncertainty, nonlinearity, nonstationarity and high dimensionality. Existing numerical techniques for such models would typically require considerable computational and storage resources. This is the case, for instance, for an optimization problem governed by time-dependent Navier-Stokes equations with uncertain inputs. In particular, the stochastic Galerkin finite element method often leads to a prohibitively high dimensional saddle-point system with tensor product structure. In this paper, we approximate the solution by the low-rank Tensor Train decomposition, and present a numerically efficient algorithm to solve the optimality equations directly in the low-rank representation. We show that the solution of the vorticity minimization problem with a distributed control admits a representation with ranks that depend modestly on model and discretization parameters even for high Reynolds numbers. For lower Reynolds numbers this is also the case for a boundary control. This opens the way for a reduced-order modeling of the stochastic optimal flow control with a moderate cost at all stages. △ Less

Submitted 17 March, 2017; originally announced March 2017.

Comments: 29 pages

Fast iterative solution of the Bethe-Salpeter eigenvalue problem using low-rank and QTT tensor approximation

Authors: Peter Benner, Sergey Dolgov, Venera Khoromskaia, Boris N. Khoromskij

Abstract: In this paper, we study and implement the structural iterative eigensolvers for the large-scale eigenvalue problem in the Bethe-Salpeter equation (BSE) based on the reduced basis approach via low-rank factorizations in generating matrices, introduced in the previous paper. The approach reduces numerical costs down to $\mathcal{O}(N_b^2)$ in the size of atomic orbitals basis set, $N_b$, instead of… ▽ More In this paper, we study and implement the structural iterative eigensolvers for the large-scale eigenvalue problem in the Bethe-Salpeter equation (BSE) based on the reduced basis approach via low-rank factorizations in generating matrices, introduced in the previous paper. The approach reduces numerical costs down to $\mathcal{O}(N_b^2)$ in the size of atomic orbitals basis set, $N_b$, instead of practically intractable $\mathcal{O}(N_b^6)$ complexity scaling for the direct diagonalization of the BSE matrix. As an alternative to rank approximation of the static screen interaction part of the BSE matrix, we propose to restrict it to a small active sub-block, with a size balancing the storage for rank-structured representations of other matrix blocks. We demonstrate that the enhanced reduced-block approximation exhibits higher precision within the controlled numerical cost, providing as well a distinct two-sided error estimate for the BSE eigenvalues. It is shown that further reduction of the asymptotic computational cost is possible due to ALS-type iteration in block tensor train (TT) format applied to the quantized-TT (QTT) tensor representation of both long eigenvectors and rank-structured matrix blocks. The QTT-rank of these entities possesses almost the same magnitude as the number of occupied orbitals in the molecular systems, $N_o$, hence the overall asymptotic complexity for solving the BSE problem can be estimated by $\mathcal{O}(\log(N_o) N_o^{2})$. We confirm numerically a considerable decrease in computational time for the presented iterative approach applied to various compact and chain-type molecules, while supporting sufficient accuracy. △ Less

Submitted 8 February, 2016; originally announced February 2016.

Comments: 23 pages, 11 figures

MSC Class: 65F30; 65F50; 65N35; 65F10

Polynomial Chaos Expansion of random coefficients and the solution of stochastic partial differential equations in the Tensor Train format

Authors: Sergey Dolgov, Boris N. Khoromskij, Alexander Litvinenko, Hermann G. Matthies

Abstract: We apply the Tensor Train (TT) decomposition to construct the tensor product Polynomial Chaos Expansion (PCE) of a random field, to solve the stochastic elliptic diffusion PDE with the stochastic Galerkin discretization, and to compute some quantities of interest (mean, variance, exceedance probabilities). We assume that the random diffusion coefficient is given as a smooth transformation of a Gau… ▽ More We apply the Tensor Train (TT) decomposition to construct the tensor product Polynomial Chaos Expansion (PCE) of a random field, to solve the stochastic elliptic diffusion PDE with the stochastic Galerkin discretization, and to compute some quantities of interest (mean, variance, exceedance probabilities). We assume that the random diffusion coefficient is given as a smooth transformation of a Gaussian random field. In this case, the PCE is delivered by a complicated formula, which lacks an analytic TT representation. To construct its TT approximation numerically, we develop the new block TT cross algorithm, a method that computes the whole TT decomposition from a few evaluations of the PCE formula. The new method is conceptually similar to the adaptive cross approximation in the TT format, but is more efficient when several tensors must be stored in the same TT representation, which is the case for the PCE. Besides, we demonstrate how to assemble the stochastic Galerkin matrix and to compute the solution of the elliptic equation and its post-processing, staying in the TT format. We compare our technique with the traditional sparse polynomial chaos and the Monte Carlo approaches. In the tensor product polynomial chaos, the polynomial degree is bounded for each random variable independently. This provides higher accuracy than the sparse polynomial set or the Monte Carlo method, but the cardinality of the tensor product set grows exponentially with the number of random variables. However, when the PCE coefficients are implicitly approximated in the TT format, the computations with the full tensor product polynomial set become possible. In the numerical experiments, we confirm that the new methodology is competitive in a wide range of parameters, especially where high accuracy and high polynomial degrees are required. △ Less

Submitted 11 March, 2015; originally announced March 2015.

Comments: This is a major revision of the manuscript arXiv:1406.2816 with significantly extended numerical experiments. Some unused material is removed

MSC Class: 15A69; 65F10; 60H15; 60H35; 65C30

Computation of the Response Surface in the Tensor Train data format

Authors: Sergey Dolgov, Boris N. Khoromskij, Alexander Litvinenko, Hermann G. Matthies

Abstract: We apply the Tensor Train (TT) approximation to construct the Polynomial Chaos Expansion (PCE) of a random field, and solve the stochastic elliptic diffusion PDE with the stochastic Galerkin discretization. We compare two strategies of the polynomial chaos expansion: sparse and full polynomial (multi-index) sets. In the full set, the polynomial orders are chosen independently in each variable, whi… ▽ More We apply the Tensor Train (TT) approximation to construct the Polynomial Chaos Expansion (PCE) of a random field, and solve the stochastic elliptic diffusion PDE with the stochastic Galerkin discretization. We compare two strategies of the polynomial chaos expansion: sparse and full polynomial (multi-index) sets. In the full set, the polynomial orders are chosen independently in each variable, which provides higher flexibility and accuracy. However, the total amount of degrees of freedom grows exponentially with the number of stochastic coordinates. To cope with this curse of dimensionality, the data is kept compressed in the TT decomposition, a recurrent low-rank factorization. PCE computations on sparse grids sets are extensively studied, but the TT representation for PCE is a novel approach that is investigated in this paper. We outline how to deduce the PCE from the covariance matrix, assemble the Galerkin operator, and evaluate some post-processing (mean, variance, Sobol indices), staying within the low-rank framework. The most demanding are two stages. First, we interpolate PCE coefficients in the TT format using a few number of samples, which is performed via the block cross approximation method. Second, we solve the discretized equation (large linear system) via the alternating minimal energy algorithm. In the numerical experiments we demonstrate that the full expansion set encapsulated in the TT format is indeed preferable in cases when high accuracy and high polynomial orders are required. △ Less

Submitted 11 June, 2014; originally announced June 2014.

Comments: 28 pages. Submitted to SIAM J. of Uncertainty Quantification

MSC Class: 15A69 65F10 60H15 60H35 65C30

A tensor decomposition algorithm for large ODEs with conservation laws

Authors: Sergey V. Dolgov

Abstract: We propose an algorithm for solution of high-dimensional evolutionary equations (ODEs and discretized time-dependent PDEs) in the Tensor Train (TT) decomposition, assuming that the solution and the right-hand side of the ODE admit such a decomposition with a low storage. A linear ODE, discretized via one-step or Chebyshev differentiation schemes, turns into a large linear system. The tensor decomp… ▽ More We propose an algorithm for solution of high-dimensional evolutionary equations (ODEs and discretized time-dependent PDEs) in the Tensor Train (TT) decomposition, assuming that the solution and the right-hand side of the ODE admit such a decomposition with a low storage. A linear ODE, discretized via one-step or Chebyshev differentiation schemes, turns into a large linear system. The tensor decomposition allows to solve this system for several time points simultaneously using an extension of the Alternating Least Squares algorithm. This method computes the TT approximation of the solution directly, without ever solving the original large problem, and encapsulates the Galerkin model reduction of the ODE. This allows an efficient estimation of the time discretization error, and hence provides a way to adapt the time steps. Besides, conservation laws can be preserved exactly in the reduced model by expanding the approximation subspace with the generating vectors of the linear invariants and correction of the euclidean norm. In numerical experiments with the transport and the chemical master equations, we demonstrate that the new method is faster than traditional time step** and stochastic simulation algorithms, whereas the invariants are preserved up to the machine precision irrespectively of the TT approximation accuracy. △ Less

Submitted 4 October, 2017; v1 submitted 31 March, 2014; originally announced March 2014.

MSC Class: 15A69; 33F05; 65F10; 65L05; 65M70; 34C14

Exact NMR simulation of protein-size spin systems using tensor train formalism

Authors: D. V. Savostyanov, S. V. Dolgov, J. M. Werner, Ilya Kuprov

Abstract: We introduce a new method, based on alternating optimization, for compact representation of spin Hamiltonians and solution of linear systems of algebraic equations in the tensor train format. We demonstrate the method's utility by simulating, without approximations, a 15N NMR spectrum of ubiquitin --- a protein containing several hundred interacting nuclear spins. Existing simulation algorithms fo… ▽ More We introduce a new method, based on alternating optimization, for compact representation of spin Hamiltonians and solution of linear systems of algebraic equations in the tensor train format. We demonstrate the method's utility by simulating, without approximations, a 15N NMR spectrum of ubiquitin --- a protein containing several hundred interacting nuclear spins. Existing simulation algorithms for the spin system and the NMR experiment in question either require significant approximations or scale exponentially with the spin system size. We compare the proposed method to the Spinach package that uses heuristic restricted state space techniques to achieve polynomial complexity scaling. When the spin system topology is close to a linear chain (e.g. for the backbone of a protein), the tensor train representation is more compact and can be computed faster than the sparse representation using restricted state spaces. △ Less

Submitted 18 June, 2014; v1 submitted 18 February, 2014; originally announced February 2014.

Comments: Accepted for publication in Phys Rev B

MSC Class: 15A69; 81T80; 82B28

Journal ref: Phys. Rev. B 90, 085139, 2014

One-site density matrix renormalization group and alternating minimum energy algorithm

Authors: Sergey V. Dolgov, Dmitry V. Savostyanov

Abstract: Given in the title are two algorithms to compute the extreme eigenstate of a high-dimensional Hermitian matrix using the tensor train (TT) / matrix product states (MPS) representation. Both methods empower the traditional alternating direction scheme with the auxiliary (e.g. gradient) information, which substantially improves the convergence in many difficult cases. Being conceptually close, these… ▽ More Given in the title are two algorithms to compute the extreme eigenstate of a high-dimensional Hermitian matrix using the tensor train (TT) / matrix product states (MPS) representation. Both methods empower the traditional alternating direction scheme with the auxiliary (e.g. gradient) information, which substantially improves the convergence in many difficult cases. Being conceptually close, these methods have different derivation, implementation, theoretical and practical properties. We emphasize the differences, and reproduce the numerical example to compare the performance of two algorithms. △ Less

Submitted 23 December, 2013; originally announced December 2013.

Comments: Submitted to the proceedings of ENUMATH 2013

MSC Class: 15A18; 15A69; 65F10; 65F15; 82B28; 82B20

Journal ref: Numerical Mathematics and Advanced Applications - ENUMATH 2013, in Lecture Notes in Computational Science and Engineering. 103:335-343, 2015

Simultaneous state-time approximation of the chemical master equation using tensor product formats

Authors: Sergey Dolgov, Boris Khoromskij

Abstract: We study the application of the novel tensor formats (TT, QTT, QTT-Tucker) to the solution of $d$-dimensional chemical master equations, applied mostly to gene regulating networks (signaling cascades, toggle switches, phage-$λ$). For some important cases, e.g. signaling cascade models, we prove good separability properties of the system operator. The Quantized tensor representations (QTT, QTT-Tuck… ▽ More We study the application of the novel tensor formats (TT, QTT, QTT-Tucker) to the solution of $d$-dimensional chemical master equations, applied mostly to gene regulating networks (signaling cascades, toggle switches, phage-$λ$). For some important cases, e.g. signaling cascade models, we prove good separability properties of the system operator. The Quantized tensor representations (QTT, QTT-Tucker) are employed in both state space and time, and the global state-time $(d+1)$-dimensional system is solved in the structured form by using the ALS-type iteration. This approach leads to the logarithmic dependence of the computational complexity on the system size. When possible, we compare our approach with the direct CME solution and some previously known approximate schemes, and observe a good potential of the newer tensor methods in simulation of relevant biological systems. △ Less

Submitted 13 November, 2013; originally announced November 2013.

Comments: This is an essentially improved version of the preprint [12]. This manuscript contains all the same numerical experiments, but some inaccuracies in the description of the modeling equations are corrected. Besides, more detailed introduction to the tensor methods is presented

MSC Class: 65F50; 15A69; 65F10; 82C31; 80A30; 34B08

Low-rank approximation in the numerical modeling of the Farley-Buneman instability in ionospheric plasma

Authors: S. V. Dolgov, A. P. Smirnov, E. E. Tyrtyshnikov

Abstract: We consider the numerical modeling of the Farley-Buneman instability development in the earth's ionosphere plasma. The ion behavior is governed by the kinetic Landau equation in the four-dimensional phase space, and since the finite difference discretization on a tensor product grid is used, this equation becomes the most computationally challenging part of the scheme. To relax the complexity and… ▽ More We consider the numerical modeling of the Farley-Buneman instability development in the earth's ionosphere plasma. The ion behavior is governed by the kinetic Landau equation in the four-dimensional phase space, and since the finite difference discretization on a tensor product grid is used, this equation becomes the most computationally challenging part of the scheme. To relax the complexity and memory consumption, an adaptive model reduction using the low-rank separation of variables, namely the Tensor Train format, is employed. The approach was verified via the prototype MATLAB implementation. Numerical experiments demonstrate the possibility of efficient separation of space and velocity variables, resulting in the solution storage reduction by a factor of order tens. △ Less

Submitted 27 August, 2013; originally announced August 2013.

MSC Class: 65K10; 82B28; 65M22; 65M06; 65K10

Computation of extreme eigenvalues in higher dimensions using block tensor train format

Authors: Sergey V. Dolgov, Boris N. Khoromskij, Ivan V. Oseledets, Dmitry V. Savostyanov

Abstract: We consider an approximate computation of several minimal eigenpairs of large Hermitian matrices which come from high--dimensional problems. We use the tensor train format (TT) for vectors and matrices to overcome the curse of dimensionality and make storage and computational cost feasible. Applying a block version of the TT format to several vectors simultaneously, we compute the low--lying eigen… ▽ More We consider an approximate computation of several minimal eigenpairs of large Hermitian matrices which come from high--dimensional problems. We use the tensor train format (TT) for vectors and matrices to overcome the curse of dimensionality and make storage and computational cost feasible. Applying a block version of the TT format to several vectors simultaneously, we compute the low--lying eigenstates of a system by minimization of a block Rayleigh quotient performed in an alternating fashion for all dimensions. For several numerical examples, we compare the proposed method with the deflation approach when the low--lying eigenstates are computed one-by-one, and also with the variational algorithms used in quantum physics. △ Less

Submitted 10 June, 2013; originally announced June 2013.

Comments: Submitted to Comput Phys Comm

MSC Class: 15A18; 15A69; 65F15; 81T17; 82B28; 82B20

Journal ref: Computer Physics Communications 185(4):1207-1216, 2014

Alternating minimal energy methods for linear systems in higher dimensions. Part II: Faster algorithm and application to nonsymmetric systems

Authors: Sergey V. Dolgov, Dmitry V. Savostyanov

Abstract: In this paper we accomplish the development of the fast rank-adaptive solver for tensor-structured symmetric positive definite linear systems in higher dimensions. In [arXiv:1301.6068] this problem is approached by alternating minimization of the energy function, which we combine with steps of the basis expansion in accordance with the steepest descent algorithm. In this paper we combine the same… ▽ More In this paper we accomplish the development of the fast rank-adaptive solver for tensor-structured symmetric positive definite linear systems in higher dimensions. In [arXiv:1301.6068] this problem is approached by alternating minimization of the energy function, which we combine with steps of the basis expansion in accordance with the steepest descent algorithm. In this paper we combine the same steps in such a way that the resulted algorithm works with one or two neighboring cores at a time. The recurrent interpretation of the algorithm allows to prove the global convergence and to estimate the convergence rate. We also propose several strategies, both rigorous and heuristic, to compute new subspaces for the basis enrichment in a more efficient way. We test the algorithm on a number of high-dimensional problems, including the non-symmetrical Fokker-Planck and chemical master equations, for which the efficiency of the method is not fully supported by the theory. In all examples we observe a convincing fast convergence and high efficiency of the proposed method. △ Less

Submitted 11 April, 2013; v1 submitted 3 April, 2013; originally announced April 2013.

Comments: Submitted to SIAM J Sci Comp

MSC Class: 15A69; 33F05; 65F10

Journal ref: SIAM J. Sci. Comput. 36(5): A2248-A2271, 2014

Alternating minimal energy methods for linear systems in higher dimensions. Part I: SPD systems

Authors: Sergey V. Dolgov, Dmitry V. Savostyanov

Abstract: We introduce a family of numerical algorithms for the solution of linear system in higher dimensions with the matrix and right hand side given and the solution sought in the tensor train format. The proposed methods are rank--adaptive and follow the alternating directions framework, but in contrast to ALS methods, in each iteration a tensor subspace is enlarged by a set of vectors chosen similarly… ▽ More We introduce a family of numerical algorithms for the solution of linear system in higher dimensions with the matrix and right hand side given and the solution sought in the tensor train format. The proposed methods are rank--adaptive and follow the alternating directions framework, but in contrast to ALS methods, in each iteration a tensor subspace is enlarged by a set of vectors chosen similarly to the steepest descent algorithm. The convergence is analyzed in the presence of approximation errors and the geometrical convergence rate is estimated and related to the one of the steepest descent. The complexity of the presented algorithms is linear in the mode size and dimension and the convergence demonstrated in the numerical experiments is comparable to the one of the DMRG--type algorithm. △ Less

Submitted 25 January, 2013; originally announced January 2013.

Comments: Submitted to SIAM J Num Anal

MSC Class: 15A69; 33F05; 65F10

Journal ref: SIAM J. Sci. Comput. 36(5): A2248-A2271, 2014

TT-GMRES: on solution to a linear system in the structured tensor format

Authors: Sergey V. Dolgov

Abstract: A adapted tensor-structured GMRES method for the TT format is proposed and investigated. The Tensor Train (TT) approximation is a robust approach to high-dimensional problems. One class of problems is solution of a linear system. In this work we study the convergence of the GMRES method in the presence of tensor approximations and provide relaxation techniques to improve its performance. Several n… ▽ More A adapted tensor-structured GMRES method for the TT format is proposed and investigated. The Tensor Train (TT) approximation is a robust approach to high-dimensional problems. One class of problems is solution of a linear system. In this work we study the convergence of the GMRES method in the presence of tensor approximations and provide relaxation techniques to improve its performance. Several numerical examples are presented. The method is also compared with a projection TT linear solver based on the ALS and DMRG methods. On a particular sPDE (high-dimensional parametric) problem, these methods manifest comparable performance, with a good preconditioner the TT-GMRES overcomes the ALS solver. △ Less

Submitted 24 June, 2012; originally announced June 2012.

Comments: To appear in: Russian Journal of Numerical Analysis and Mathematical Modelling

Superluminal Black Holes

Authors: D. S. Dolgov

Abstract: The new solution of the Einstein equations in empty space is presented. The solution is constructed using Schwarzschild solution but essentially differs from it. The basic properties of the solution are: the existence of a horizon which is a hyperboloid of one sheet moving along its axis with superluminal velocity, right signature of the metric outside the horizon and Minkovsky-flatness of it at… ▽ More The new solution of the Einstein equations in empty space is presented. The solution is constructed using Schwarzschild solution but essentially differs from it. The basic properties of the solution are: the existence of a horizon which is a hyperboloid of one sheet moving along its axis with superluminal velocity, right signature of the metric outside the horizon and Minkovsky-flatness of it at infinity outside the horizon. There is also a discussion in the last chapter, including comparing with recent astronomical observations. △ Less

Submitted 7 June, 1993; originally announced June 1993.

Comments: FIAN/TD/8-93 7 pages