Showing 1–45 of 45 results for author: Darve, E

Factor Fitting, Rank Allocation, and Partitioning in Multilevel Low Rank Matrices

Authors: Tetiana Parshakova, Trevor Hastie, Eric Darve, Stephen Boyd

Abstract: We consider multilevel low rank (MLR) matrices, defined as a row and column permutation of a sum of matrices, each one a block diagonal refinement of the previous one, with all blocks low rank given in factored form. MLR matrices extend low rank matrices but share many of their properties, such as the total storage required and complexity of matrix-vector multiplication. We address three problems… ▽ More We consider multilevel low rank (MLR) matrices, defined as a row and column permutation of a sum of matrices, each one a block diagonal refinement of the previous one, with all blocks low rank given in factored form. MLR matrices extend low rank matrices but share many of their properties, such as the total storage required and complexity of matrix-vector multiplication. We address three problems that arise in fitting a given matrix by an MLR matrix in the Frobenius norm. The first problem is factor fitting, where we adjust the factors of the MLR matrix. The second is rank allocation, where we choose the ranks of the blocks in each level, subject to the total rank having a given value, which preserves the total storage needed for the MLR matrix. The final problem is to choose the hierarchical partition of rows and columns, along with the ranks and factors. This paper is accompanied by an open source package that implements the proposed methods. △ Less

Submitted 29 October, 2023; originally announced October 2023.

A numerically stable communication-avoiding s-step GMRES algorithm

Authors: Zan Xu, Juan J. Alonso, Eric Darve

Abstract: Krylov subspace methods are extensively used in scientific computing to solve large-scale linear systems. However, the performance of these iterative Krylov solvers on modern supercomputers is limited by expensive communication costs. The $s$-step strategy generates a series of $s$ Krylov vectors at a time to avoid communication. Asymptotically, the $s$-step approach can reduce communication laten… ▽ More Krylov subspace methods are extensively used in scientific computing to solve large-scale linear systems. However, the performance of these iterative Krylov solvers on modern supercomputers is limited by expensive communication costs. The $s$-step strategy generates a series of $s$ Krylov vectors at a time to avoid communication. Asymptotically, the $s$-step approach can reduce communication latency by a factor of $s$. Unfortunately, due to finite-precision implementation, the step size has to be kept small for stability. In this work, we tackle the numerical instabilities encountered in the $s$-step GMRES algorithm. By choosing an appropriate polynomial basis and block orthogonalization schemes, we construct a communication avoiding $s$-step GMRES algorithm that automatically selects the optimal step size to ensure numerical stability. To further maximize communication savings, we introduce scaled Newton polynomials that can increase the step size $s$ to a few hundreds for many problems. An initial step size estimator is also developed to efficiently choose the optimal step size for stability. The guaranteed stability of the proposed algorithm is demonstrated using numerical experiments. In the process, we also evaluate how the choice of polynomial and preconditioning affects the stability limit of the algorithm. Finally, we show parallel scalability on more than 14,000 cores in a distributed-memory setting. Perfectly linear scaling has been observed in both strong and weak scaling studies with negligible communication costs. △ Less

Submitted 17 March, 2023; v1 submitted 15 March, 2023; originally announced March 2023.

Comments: 32 pages, 14 figures

A Hybrid Direct-Iterative Method for Solving KKT Linear Systems

Authors: Shaked Regev, Nai-Yuan Chiang, Eric Darve, Cosmin G. Petra, Michael A. Saunders, Kasia Świrydowicz, Slaven Peleš

Abstract: We propose a solution strategy for linear systems arising in interior method optimization, which is suitable for implementation on hardware accelerators such as graphical processing units (GPUs). The current gold standard for solving these systems is the LDL^T factorization. However, LDL^T requires pivoting during factorization, which substantially increases communication cost and degrades perform… ▽ More We propose a solution strategy for linear systems arising in interior method optimization, which is suitable for implementation on hardware accelerators such as graphical processing units (GPUs). The current gold standard for solving these systems is the LDL^T factorization. However, LDL^T requires pivoting during factorization, which substantially increases communication cost and degrades performance on GPUs. Our novel approach solves a large indefinite system by solving multiple smaller positive definite systems, using an iterative solve for the Schur complement and an inner direct solve (via Cholesky factorization) within each iteration. Cholesky is stable without pivoting, thereby reducing communication and allowing reuse of the symbolic factorization. We demonstrate the practicality of our approach and show that on large systems it can efficiently utilize GPUs and outperform LDL^T factorization of the full system. △ Less

Submitted 7 October, 2021; originally announced October 2021.

Comments: 22 pages, 9 figures, 7 tables

MSC Class: 15; 65; 68 ACM Class: G.1

On the fractional Laplacian of variable order

Authors: Eric Darve, Marta D'Elia, Roberto Garrappa, Andrea Giusti, Natalia L. Rubio

Abstract: We present a novel definition of variable-order fractional Laplacian on Rn based on a natural generalization of the standard Riesz potential. Our definition holds for values of the fractional parameter spanning the entire open set (0, n/2). We then discuss some properties of the fractional Poisson's equation involving this operator and we compute the corresponding Green function, for which we prov… ▽ More We present a novel definition of variable-order fractional Laplacian on Rn based on a natural generalization of the standard Riesz potential. Our definition holds for values of the fractional parameter spanning the entire open set (0, n/2). We then discuss some properties of the fractional Poisson's equation involving this operator and we compute the corresponding Green function, for which we provide some instructive examples for specific problems. △ Less

Submitted 2 September, 2021; originally announced September 2021.

Linear solvers for power grid optimization problems: a review of GPU-accelerated linear solvers

Authors: Kasia Swirydowicz, Eric Darve, Wesley Jones, Jonathan Maack, Shaked Regev, Michael A. Saunders, Stephen J. Thomas, Slaven Peles

Abstract: The linear equations that arise in interior methods for constrained optimization are sparse symmetric indefinite and become extremely ill-conditioned as the interior method converges. These linear systems present a challenge for existing solver frameworks based on sparse LU or LDL^T decompositions. We benchmark five well known direct linear solver packages using matrices extracted from power grid… ▽ More The linear equations that arise in interior methods for constrained optimization are sparse symmetric indefinite and become extremely ill-conditioned as the interior method converges. These linear systems present a challenge for existing solver frameworks based on sparse LU or LDL^T decompositions. We benchmark five well known direct linear solver packages using matrices extracted from power grid optimization problems. The achieved solution accuracy varies greatly among the packages. None of the tested packages delivers significant GPU acceleration for our test cases. △ Less

Submitted 13 August, 2021; v1 submitted 25 June, 2021; originally announced June 2021.

Trust Region Method for Coupled Systems of PDE Solvers and Deep Neural Networks

Authors: Kailai Xu, Eric Darve

Abstract: Physics-informed machine learning and inverse modeling require the solution of ill-conditioned non-convex optimization problems. First-order methods, such as SGD and ADAM, and quasi-Newton methods, such as BFGS and L-BFGS, have been applied with some success to optimization problems involving deep neural networks in computational engineering inverse problems. However, empirical evidence shows that… ▽ More Physics-informed machine learning and inverse modeling require the solution of ill-conditioned non-convex optimization problems. First-order methods, such as SGD and ADAM, and quasi-Newton methods, such as BFGS and L-BFGS, have been applied with some success to optimization problems involving deep neural networks in computational engineering inverse problems. However, empirical evidence shows that convergence and accuracy for these methods remain a challenge. Our study unveiled at least two intrinsic defects of these methods when applied to coupled systems of partial differential equations (PDEs) and deep neural networks (DNNs): (1) convergence is often slow with long plateaus that make it difficult to determine whether the method has converged or not; (2) quasi-Newton methods do not provide a sufficiently accurate approximation of the Hessian matrix; this typically leads to early termination (one of the stop** criteria of the optimizer is satisfied although the achieved error is far from minimal). Based on these observations, we propose to use trust region methods for optimizing coupled systems of PDEs and DNNs. Specifically, we developed an algorithm for second-order physics constrained learning, an efficient technique to calculate Hessian matrices based on computational graphs. We show that trust region methods overcome many of the defects and exhibit remarkable fast convergence and superior accuracy compared to ADAM, BFGS, and L-BFGS. △ Less

Submitted 16 May, 2021; originally announced May 2021.

Hierarchical Orthogonal Factorization: Sparse Least Squares Problems

Authors: Abeynaya Gnanasekaran, Eric Darve

Abstract: In this work, we develop a fast hierarchical solver for solving large, sparse least squares problems. We build upon the algorithm, spaQR (sparsified QR), that was developed by the authors to solve large sparse linear systems. Our algorithm is built on top of a Nested Dissection based multifrontal QR approach. We use low-rank approximations on the frontal matrices to sparsify the vertex separators… ▽ More In this work, we develop a fast hierarchical solver for solving large, sparse least squares problems. We build upon the algorithm, spaQR (sparsified QR), that was developed by the authors to solve large sparse linear systems. Our algorithm is built on top of a Nested Dissection based multifrontal QR approach. We use low-rank approximations on the frontal matrices to sparsify the vertex separators at every level in the elimination tree. Using a two-step sparsification scheme, we reduce the number of columns and maintain the ratio of rows to columns in each front without introducing any additional fill-in. With this improvised scheme, we show that the runtime of the algorithm scales as $\mathcal{O}(M \log N)$ and uses $\mathcal{O}(M)$ memory to store the factorization. This is achieved at the expense of a small and controllable approximation error. The end result is an approximate factorization of the matrix stored as a sequence of sparse orthogonal and upper-triangular factors and hence easy to apply/solve with a vector. Finally, we compare the performance of the spaQR algorithm in solving sparse least squares problems with direct multifrontal QR and CGLS iterative method with a standard diagonal preconditioner. △ Less

Submitted 4 March, 2021; v1 submitted 19 February, 2021; originally announced February 2021.

Comments: arXiv admin note: text overlap with arXiv:2010.06807

MSC Class: 65F08 (Primary); 65F25; 65F50; 65Y20; 65F05 (Secondary) ACM Class: G.1.3

Towards a Scalable Hierarchical High-order CFD Solver

Authors: Zan Xu, Léopold Cambier, Juan J. Alonso, Eric Darve

Abstract: Development of highly scalable and robust algorithms for large-scale CFD simulations has been identified as one of the key ingredients to achieve NASA's CFD Vision 2030 goals. In order to improve simulation capability and to effectively leverage new high-performance computing hardware, the most computationally intensive parts of CFD solution algorithms -- namely, linear solvers and preconditioners… ▽ More Development of highly scalable and robust algorithms for large-scale CFD simulations has been identified as one of the key ingredients to achieve NASA's CFD Vision 2030 goals. In order to improve simulation capability and to effectively leverage new high-performance computing hardware, the most computationally intensive parts of CFD solution algorithms -- namely, linear solvers and preconditioners -- need to achieve asymptotic behavior on massively parallel and heterogeneous architectures and preserve convergence rates as the meshes are refined further. In this work, we present a scalable high-order implicit Discontinuous Galerkin solver from the SU2 framework using a promising preconditioning technique based on algebraic sparsified nested dissection algorithm with low-rank approximations, and communication-avoiding Krylov subspace methods to enable scalability with very large processor counts. The overall approach is tested on a canonical 2D NACA0012 test case of increasing size to demonstrate its scalability on multiple processing cores. Both the preconditioner and the linear solver are shown to exhibit near-linear weak scaling up to 2,048 cores with no significant degradation of the convergence rate. △ Less

Submitted 5 January, 2021; originally announced January 2021.

Comments: 14 pages, 7 figures

Journal ref: AIAA Scitech 2021 Forum

ADCME: Learning Spatially-varying Physical Fields using Deep Neural Networks

Authors: Kailai Xu, Eric Darve

Abstract: ADCME is a novel computational framework to solve inverse problems involving physical simulations and deep neural networks (DNNs). This paper benchmarks its capability to learn spatially-varying physical fields using DNNs. We demonstrate that our approach has superior accuracy compared to the discretization approach on a variety of problems, linear or nonlinear, static or dynamic. Technically, we… ▽ More ADCME is a novel computational framework to solve inverse problems involving physical simulations and deep neural networks (DNNs). This paper benchmarks its capability to learn spatially-varying physical fields using DNNs. We demonstrate that our approach has superior accuracy compared to the discretization approach on a variety of problems, linear or nonlinear, static or dynamic. Technically, we formulate our inverse problem as a PDE-constrained optimization problem. We express both the numerical simulations and DNNs using computational graphs and therefore, we can calculate the gradients using reverse-mode automatic differentiation. We apply a physics constrained learning algorithm (PCL) to efficiently back-propagate gradients through iterative solvers for nonlinear equations. The open-source software which accompanies the present paper can be found at https://github.com/kailaix/ADCME.jl. △ Less

Submitted 24 November, 2020; originally announced November 2020.

Distributed Machine Learning for Computational Engineering using MPI

Authors: Kailai Xu, Weiqiang Zhu, Eric Darve

Abstract: We propose a framework for training neural networks that are coupled with partial differential equations (PDEs) in a parallel computing environment. Unlike most distributed computing frameworks for deep neural networks, our focus is to parallelize both numerical solvers and deep neural networks in forward and adjoint computations. Our parallel computing model views data communication as a node in… ▽ More We propose a framework for training neural networks that are coupled with partial differential equations (PDEs) in a parallel computing environment. Unlike most distributed computing frameworks for deep neural networks, our focus is to parallelize both numerical solvers and deep neural networks in forward and adjoint computations. Our parallel computing model views data communication as a node in the computational graph for numerical simulations. The advantage of our model is that data communication and computing are cleanly separated and thus provide better flexibility, modularity, and testability. We demonstrate using various large-scale problems that we can achieve substantial acceleration by using parallel solvers for PDEs in training deep neural networks that are coupled with PDEs. △ Less

Submitted 24 November, 2020; v1 submitted 2 November, 2020; originally announced November 2020.

Comments: 24 pages, 25 figures

Hierarchical Orthogonal Factorization: Sparse Square matrices

Authors: Abeynaya Gnanasekaran, Eric Darve

Abstract: In this work, we develop a new fast algorithm, spaQR -- sparsified QR, for solving large, sparse linear systems. The key to our approach is using low-rank approximations to sparsify the separators in a Nested Dissection based Householder QR factorization. First, a modified version of Nested Dissection is used to identify interiors/separators and reorder the matrix. Then, classical Householder QR i… ▽ More In this work, we develop a new fast algorithm, spaQR -- sparsified QR, for solving large, sparse linear systems. The key to our approach is using low-rank approximations to sparsify the separators in a Nested Dissection based Householder QR factorization. First, a modified version of Nested Dissection is used to identify interiors/separators and reorder the matrix. Then, classical Householder QR is used to factorize the interiors, going from the leaves to the root to the elimination tree. After every level of interior factorization, we sparsify the remaining separators by using low-rank approximations. This operation reduces the size of the separators without introducing any fill-in in the matrix. However, it introduces a small approximation error which can be controlled by the user. The resulting approximate factorization is stored as a sequence of sparse orthogonal and sparse upper-triangular factors. Hence, it can be applied efficiently to solve linear systems. Additionally, we further improve the algorithm by using a block diagonal scaling. Then, we show a systematic analysis of the approximation error and effectiveness of the algorithm in solving linear systems. Finally, we perform numerical tests on benchmark unsymmetric problems to evaluate the performance of the algorithm. The factorization time scales as $\mathcal{O}(N \log N)$ and the solve time scales as $\mathcal{O}(N)$. △ Less

Submitted 14 October, 2020; originally announced October 2020.

MSC Class: 65F08 (Primary); 65F25; 65F50; 65Y20; 65F05 (Secondary) ACM Class: G.1.3

Solving Inverse Problems in Steady-State Navier-Stokes Equations using Deep Neural Networks

Authors: Tiffany Fan, Kailai Xu, Jay Pathak, Eric Darve

Abstract: Inverse problems in fluid dynamics are ubiquitous in science and engineering, with applications ranging from electronic cooling system design to ocean modeling. We propose a general and robust approach for solving inverse problems in the steady-state Navier-Stokes equations by combining deep neural networks and numerical partial differential equation (PDE) schemes. Our approach expresses numerical… ▽ More Inverse problems in fluid dynamics are ubiquitous in science and engineering, with applications ranging from electronic cooling system design to ocean modeling. We propose a general and robust approach for solving inverse problems in the steady-state Navier-Stokes equations by combining deep neural networks and numerical partial differential equation (PDE) schemes. Our approach expresses numerical simulation as a computational graph with differentiable operators. We then solve inverse problems by constrained optimization, using gradients calculated from the computational graph with reverse-mode automatic differentiation. This technique enables us to model unknown physical properties using deep neural networks and embed them into the PDE model. We demonstrate the effectiveness of our method by computing spatially-varying viscosity and conductivity fields with deep neural networks (DNNs) and training the DNNs using partial observations of velocity fields. We show that the DNNs are capable of modeling complex spatially-varying physical fields with sparse and noisy data. Our implementation leverages the open access ADCME, a library for solving inverse modeling problems in scientific computing using automatic differentiation. △ Less

Submitted 18 November, 2020; v1 submitted 29 August, 2020; originally announced August 2020.

Second Order Accurate Hierarchical Approximate Factorization of Sparse SPD Matrices

Authors: Bazyli Klockiewicz, Léopold Cambier, Ryan Humble, Hamdi Tchelepi, Eric Darve

Abstract: We describe a second-order accurate approach to sparsifying the off-diagonal blocks in the hierarchical approximate factorizations of sparse symmetric positive definite matrices. The norm of the error made by the new approach depends quadratically, not linearly, on the error in the low-rank approximation of the given block. The analysis of the resulting two-level preconditioner shows that the prec… ▽ More We describe a second-order accurate approach to sparsifying the off-diagonal blocks in the hierarchical approximate factorizations of sparse symmetric positive definite matrices. The norm of the error made by the new approach depends quadratically, not linearly, on the error in the low-rank approximation of the given block. The analysis of the resulting two-level preconditioner shows that the preconditioner is second-order accurate as well. We incorporate the new approach into the recent Sparsified Nested Dissection algorithm [SIAM J. Matrix Anal. Appl., 41 (2020), pp. 715-746], and test it on a wide range of problems. The new approach halves the number of Conjugate Gradient iterations needed for convergence, with almost the same factorization complexity, improving the total runtimes of the algorithm. Our approach can be incorporated into other rank-structured methods for solving sparse linear systems. △ Less

Submitted 3 August, 2020; v1 submitted 1 July, 2020; originally announced July 2020.

Comments: 26 pages

Inverse Modeling of Viscoelasticity Materials using Physics Constrained Learning

Authors: Kailai Xu, Alexandre M. Tartakovsky, Jeff Burghardt, Eric Darve

Abstract: We propose a novel approach to model viscoelasticity materials using neural networks, which capture rate-dependent and nonlinear constitutive relations. However, inputs and outputs of the neural networks are not directly observable, and therefore common training techniques with input-output pairs for the neural networks are inapplicable. To that end, we develop a novel computational approach to bo… ▽ More We propose a novel approach to model viscoelasticity materials using neural networks, which capture rate-dependent and nonlinear constitutive relations. However, inputs and outputs of the neural networks are not directly observable, and therefore common training techniques with input-output pairs for the neural networks are inapplicable. To that end, we develop a novel computational approach to both calibrate parametric and learn neural-network-based constitutive relations of viscoelasticity materials from indirect displacement data in the context of multi-physics interactions. We show that limited displacement data hold sufficient information to quantify the viscoelasticity behavior. We formulate the inverse computation---modeling viscoelasticity properties from observed displacement data---as a PDE-constrained optimization problem and minimize the error functional using a gradient-based optimization method. The gradients are computed by a combination of automatic differentiation and physics constrained learning. The effectiveness of our method is demonstrated through numerous benchmark problems in geomechanics and porous media transport. △ Less

Submitted 9 May, 2020; originally announced May 2020.

Comments: 28 pages, 22 figures

Learning Constitutive Relations using Symmetric Positive Definite Neural Networks

Authors: Kailai Xu, Daniel Z. Huang, Eric Darve

Abstract: We present the Cholesky-factored symmetric positive definite neural network (SPD-NN) for modeling constitutive relations in dynamical equations. Instead of directly predicting the stress, the SPD-NN trains a neural network to predict the Cholesky factor of a tangent stiffness matrix, based on which the stress is calculated in the incremental form. As a result of the special structure, SPD-NN weakl… ▽ More We present the Cholesky-factored symmetric positive definite neural network (SPD-NN) for modeling constitutive relations in dynamical equations. Instead of directly predicting the stress, the SPD-NN trains a neural network to predict the Cholesky factor of a tangent stiffness matrix, based on which the stress is calculated in the incremental form. As a result of the special structure, SPD-NN weakly imposes convexity on the strain energy function, satisfies time consistency for path-dependent materials, and therefore improves numerical stability, especially when the SPD-NN is used in finite element simulations. Depending on the types of available data, we propose two training methods, namely direct training for strain and stress pairs and indirect training for loads and displacement pairs. We demonstrate the effectiveness of SPD-NN on hyperelastic, elasto-plastic, and multiscale fiber-reinforced plate problems from solid mechanics. The generality and robustness of the SPD-NN make it a promising tool for a wide range of constitutive modeling applications. △ Less

Submitted 1 April, 2020; originally announced April 2020.

Comments: 31 pages, 20 figures

Physics Constrained Learning for Data-driven Inverse Modeling from Sparse Observations

Authors: Kailai Xu, Eric Darve

Abstract: Deep neural networks (DNN) have been used to model nonlinear relations between physical quantities. Those DNNs are embedded in physical systems described by partial differential equations (PDE) and trained by minimizing a loss function that measures the discrepancy between predictions and observations in some chosen norm. This loss function often includes the PDE constraints as a penalty term when… ▽ More Deep neural networks (DNN) have been used to model nonlinear relations between physical quantities. Those DNNs are embedded in physical systems described by partial differential equations (PDE) and trained by minimizing a loss function that measures the discrepancy between predictions and observations in some chosen norm. This loss function often includes the PDE constraints as a penalty term when only sparse observations are available. As a result, the PDE is only satisfied approximately by the solution. However, the penalty term typically slows down the convergence of the optimizer for stiff problems. We present a new approach that trains the embedded DNNs while numerically satisfying the PDE constraints. We develop an algorithm that enables differentiating both explicit and implicit numerical solvers in reverse-mode automatic differentiation. This allows the gradients of the DNNs and the PDE solvers to be computed in a unified framework. We demonstrate that our approach enjoys faster convergence and better stability in relatively stiff problems compared to the penalty method. Our approach allows for the potential to solve and accelerate a wide range of data-driven inverse modeling, where the physical constraints are described by PDEs and need to be satisfied accurately. △ Less

Submitted 24 February, 2020; originally announced February 2020.

Comments: 44 pages, 13 figures

Learning Hidden Dynamics using Intelligent Automatic Differentiation

Authors: Kailai Xu, Dongzhuo Li, Eric Darve, Jerry M. Harris

Abstract: Many engineering problems involve learning hidden dynamics from indirect observations, where the physical processes are described by systems of partial differential equations (PDE). Gradient-based optimization methods are considered scalable and efficient to learn hidden dynamics. However, one of the most time-consuming and error-prone tasks is to derive and implement the gradients, especially in… ▽ More Many engineering problems involve learning hidden dynamics from indirect observations, where the physical processes are described by systems of partial differential equations (PDE). Gradient-based optimization methods are considered scalable and efficient to learn hidden dynamics. However, one of the most time-consuming and error-prone tasks is to derive and implement the gradients, especially in systems of PDEs where gradients from different systems must be correctly integrated together. To that purpose, we present a novel technique, called intelligent automatic differentiation (IAD), to leverage the modern machine learning tool $\texttt{TensorFlow}$ for computing gradients automatically and conducting optimization efficiently. Moreover, IAD allows us to integrate specially designed state adjoint method codes to achieve better performance. Numerical tests demonstrate the feasibility of IAD for learning hidden dynamics in complicated systems of PDEs; additionally, by incorporating custom built state adjoint method codes in IAD, we significantly accelerate the forward and inverse simulation. △ Less

Submitted 16 December, 2019; originally announced December 2019.

Comments: 25 pages, 10 figures

Adversarial Numerical Analysis for Inverse Problems

Authors: Kailai Xu, Eric Darve

Abstract: Many scientific and engineering applications are formulated as inverse problems associated with stochastic models. In such cases the unknown quantities are distributions. The applicability of traditional methods is limited because of their demanding assumptions or prohibitive computational consumptions; for example, maximum likelihood methods require closed-form density functions, and Markov Chain… ▽ More Many scientific and engineering applications are formulated as inverse problems associated with stochastic models. In such cases the unknown quantities are distributions. The applicability of traditional methods is limited because of their demanding assumptions or prohibitive computational consumptions; for example, maximum likelihood methods require closed-form density functions, and Markov Chain Monte Carlo needs a large number of simulations. We introduce adversarial numerical analysis, which estimates the unknown distributions by minimizing the discrepancy of statistical properties between observed random process and simulated random process. The discrepancy metric is computed with a discriminative neural network. We demonstrated numerically that the proposed methods can estimate the underlying parameters and learn complicated unknown distributions. △ Less

Submitted 15 October, 2019; originally announced October 2019.

Comments: 29 pages, 16 figures

Sparse Hierarchical Preconditioners Using Piecewise Smooth Approximations of Eigenvectors

Authors: Bazyli Klockiewicz, Eric Darve

Abstract: When solving linear systems arising from PDE discretizations, iterative methods (such as Conjugate Gradient, GMRES, or MINRES) are often the only practical choice. To converge in a small number of iterations, however, they have to be coupled with an efficient preconditioner. The efficiency of the preconditioner depends largely on its accuracy on the eigenvectors corresponding to small eigenvalues,… ▽ More When solving linear systems arising from PDE discretizations, iterative methods (such as Conjugate Gradient, GMRES, or MINRES) are often the only practical choice. To converge in a small number of iterations, however, they have to be coupled with an efficient preconditioner. The efficiency of the preconditioner depends largely on its accuracy on the eigenvectors corresponding to small eigenvalues, and unfortunately, black-box methods typically cannot guarantee sufficient accuracy on these eigenvectors. Thus, constructing the preconditioner becomes a problem-dependent task. However, for a large class of problems, including many elliptic equations, the eigenvectors corresponding to small eigenvalues are smooth functions of the PDE grid. In this paper, we describe a hierarchical approximate factorization approach which focuses on improving accuracy on the smooth eigenvectors. The improved accuracy is achieved by preserving the action of the factorized matrix on piecewise polynomial functions of the grid. Based on the factorization, we propose a family of sparse preconditioners with $O(n)$ or $O(n \log{n})$ construction complexities. Our methods exhibit the optimal $O(n)$ solution times in benchmarks run on large elliptic problems of different types, arising for example in flow or mechanical simulations. In the case of the linear elasticity equation the preconditioners are exact on the near-kernel rigid body modes. △ Less

Submitted 21 February, 2020; v1 submitted 8 July, 2019; originally announced July 2019.

Journal ref: SIAM Journal on Scientific Computing 42.6 (2020): A3907-A3931

Learning Constitutive Relations from Indirect Observations Using Deep Neural Networks

Authors: Daniel Z. Huang, Kailai Xu, Charbel Farhat, Eric Darve

Abstract: We present a new approach for predictive modeling and its uncertainty quantification for mechanical systems, where coarse-grained models such as constitutive relations are derived directly from observation data. We explore the use of a neural network to represent the unknown constitutive relations, compare the neural networks with piecewise linear functions, radial basis functions, and radial basi… ▽ More We present a new approach for predictive modeling and its uncertainty quantification for mechanical systems, where coarse-grained models such as constitutive relations are derived directly from observation data. We explore the use of a neural network to represent the unknown constitutive relations, compare the neural networks with piecewise linear functions, radial basis functions, and radial basis function networks, and show that the neural network outperforms the others in certain cases. We analyze the approximation error of the neural networks using a scaling argument. The training and predicting processes in our framework combine the finite element method, automatic differentiation, and neural networks (or other function approximators). Our framework also allows uncertainty quantification in the form of confidence intervals. Numerical examples on a multiscale fiber-reinforced plate problem and a nonlinear rubbery membrane problem from solid mechanics demonstrate the effectiveness of our framework. △ Less

Submitted 25 February, 2020; v1 submitted 29 May, 2019; originally announced May 2019.

Comments: 40 pages, 21 figures

The Neural Network Approach to Inverse Problems in Differential Equations

Authors: Kailai Xu, Eric Darve

Abstract: We proposed a framework for solving inverse problems in differential equations based on neural networks and automatic differentiation. Neural networks are used to approximate hidden fields. We analyze the source of errors in the framework and derive an error estimate for a model diffusion equation problem. Besides, we propose a way for sensitivity analysis, utilizing the automatic differentiation… ▽ More We proposed a framework for solving inverse problems in differential equations based on neural networks and automatic differentiation. Neural networks are used to approximate hidden fields. We analyze the source of errors in the framework and derive an error estimate for a model diffusion equation problem. Besides, we propose a way for sensitivity analysis, utilizing the automatic differentiation mechanism embedded in the framework. It frees people from the tedious and error-prone process of deriving the gradients. Numerical examples exhibit consistency with the convergence analysis and error saturation is noteworthily predicted. We also demonstrate the unique benefits neural networks offer at the same time: universal approximation ability, regularizing the solution, bypassing the curse of dimensionality and leveraging efficient computing frameworks. △ Less

Submitted 23 January, 2019; originally announced January 2019.

Comments: 32 pages, 9 figures

An Algebraic Sparsified Nested Dissection Algorithm Using Low-Rank Approximations

Authors: Léopold Cambier, Chao Chen, Erik G Boman, Sivasankaran Rajamanickam, Raymond S. Tuminaro, Eric Darve

Abstract: We propose a new algorithm for the fast solution of large, sparse, symmetric positive-definite linear systems, spaND -- sparsified Nested Dissection. It is based on nested dissection, sparsification and low-rank compression. After eliminating all interiors at a given level of the elimination tree, the algorithm sparsifies all separators corresponding to the interiors. This operation reduces the si… ▽ More We propose a new algorithm for the fast solution of large, sparse, symmetric positive-definite linear systems, spaND -- sparsified Nested Dissection. It is based on nested dissection, sparsification and low-rank compression. After eliminating all interiors at a given level of the elimination tree, the algorithm sparsifies all separators corresponding to the interiors. This operation reduces the size of the separators by eliminating some degrees of freedom but without introducing any fill-in. This is done at the expense of a small and controllable approximation error. The result is an approximate factorization that can be used as an efficient preconditioner. We then perform several numerical experiments to evaluate this algorithm. We demonstrate that a version using orthogonal factorization and block-diagonal scaling takes fewer CG iterations to converge than previous similar algorithms on various kinds of problems. Furthermore, this algorithm is provably guaranteed to never break down and the matrix stays symmetric positive-definite throughout the process. We evaluate the algorithm on some large problems and show it exhibits near-linear scaling. The factorization time is roughly O(N) and the number of iterations grows slowly with N. △ Less

Submitted 27 January, 2020; v1 submitted 9 January, 2019; originally announced January 2019.

Spectral Method for the Fractional Laplacian in 2D and 3D

Authors: Kailai Xu, Eric Darve

Abstract: A spectral method is considered for approximating the fractional Laplacian and solving the fractional Poisson problem in 2D and 3D unit balls. The method is based on the explicit formulation of the eigenfunctions and eigenvalues of the fractional Laplacian in the unit balls under the weighted $L^2$ space. The resulting method enjoys spectral accuracy for all fractional index $α\in (0,2)$ and is co… ▽ More A spectral method is considered for approximating the fractional Laplacian and solving the fractional Poisson problem in 2D and 3D unit balls. The method is based on the explicit formulation of the eigenfunctions and eigenvalues of the fractional Laplacian in the unit balls under the weighted $L^2$ space. The resulting method enjoys spectral accuracy for all fractional index $α\in (0,2)$ and is computationally efficient due to the orthogonality of the basis functions. We also proposed a numerical integration strategy for computing the coefficients. Numerical examples in 2D and 3D are shown to demonstrate the effectiveness of the proposed methods. △ Less

Submitted 17 December, 2018; originally announced December 2018.

Comments: 34 pages, 7 figures

Efficient Numerical Method for Models Driven by Lévy Process via Hierarchical Matrices

Authors: Kailai Xu, Eric Darve

Abstract: Modeling via fractional partial differential equations or a Lévy process has been an active area of research and has many applications. However, the lack of efficient numerical computation methods for general nonlocal operators impedes people from adopting such modeling tools. We proposed an efficient solver for the convection-diffusion equation whose operator is the infinitesimal generator of a L… ▽ More Modeling via fractional partial differential equations or a Lévy process has been an active area of research and has many applications. However, the lack of efficient numerical computation methods for general nonlocal operators impedes people from adopting such modeling tools. We proposed an efficient solver for the convection-diffusion equation whose operator is the infinitesimal generator of a Lévy process based on $\mathcal{H}$-matrix technique. The proposed Crank Nicolson scheme is unconditionally stable and has a theoretical $\mathcal{O}(h^2+Δt^2)$ convergence rate. The $\mathcal{H}$-matrix technique has theoretical $\mathcal{O}(N)$ space and computational complexity compared to $\mathcal{O}(N^2)$ and $\mathcal{O}(N^3)$ respectively for the direct method. Numerical experiments demonstrate the efficiency of the new algorithm. △ Less

Submitted 17 December, 2018; originally announced December 2018.

Comments: 40 pages, 16 figures

Isogeometric Collocation Method for the Fractional Laplacian in the 2D Bounded Domain

Authors: Kailai Xu, Eric Darve

Abstract: We consider the isogeometric analysis for fractional PDEs involving the fractional Laplacian in two dimensions. An isogeometric collocation method is developed to discretize the fractional Laplacian and applied to the fractional Poisson problem and the time-dependent fractional porous media equation. Numerical studies exhibit monotonous convergence with a rate of $\mathcal{O}(N^{-1})$, where $N$ i… ▽ More We consider the isogeometric analysis for fractional PDEs involving the fractional Laplacian in two dimensions. An isogeometric collocation method is developed to discretize the fractional Laplacian and applied to the fractional Poisson problem and the time-dependent fractional porous media equation. Numerical studies exhibit monotonous convergence with a rate of $\mathcal{O}(N^{-1})$, where $N$ is the degrees of freedom. A comparison with finite element analysis shows that the method enjoys higher accuracy per degree of freedom and has a better convergence rate. We demonstrate that isogeometric analysis offers a novel and promising computational tool for nonlocal problems. △ Less

Submitted 9 May, 2020; v1 submitted 18 December, 2018; originally announced December 2018.

Comments: 23 pages, 8 figures

A Robust Hierarchical Solver for Ill-conditioned Systems with Applications to Ice Sheet Modeling

Authors: Chao Chen, Leopold Cambier, Erik G. Boman, Sivasankaran Rajamanickam, Raymond S. Tuminaro, Eric Darve

Abstract: A hierarchical solver is proposed for solving sparse ill-conditioned linear systems in parallel. The solver is based on a modification of the LoRaSp method, but employs a deferred-compression technique, which provably reduces the approximation error and significantly improves efficiency. Moreover, the deferred-compression technique introduces minimal overhead and does not affect parallelism. As a… ▽ More A hierarchical solver is proposed for solving sparse ill-conditioned linear systems in parallel. The solver is based on a modification of the LoRaSp method, but employs a deferred-compression technique, which provably reduces the approximation error and significantly improves efficiency. Moreover, the deferred-compression technique introduces minimal overhead and does not affect parallelism. As a result, the new solver achieves linear computational complexity under mild assumptions and excellent parallel scalability. To demonstrate the performance of the new solver, we focus on applying it to solve sparse linear systems arising from ice sheet modeling. The strong anisotropic phenomena associated with the thin structure of ice sheets creates serious challenges for existing solvers. To address the anisotropy, we additionally developed a customized partitioning scheme for the solver, which captures the strong-coupling direction accurately. In general, the partitioning can be computed algebraically with existing software packages, and thus the new solver is generalizable for solving other sparse linear systems. Our results show that ice sheet problems of about 300 million degrees of freedom have been solved in just a few minutes using a thousand processors. △ Less

Submitted 29 November, 2018; v1 submitted 27 November, 2018; originally announced November 2018.

Comments: corrected misspelled author names

MSC Class: 65F99

Gaussian Quadrature Rule using ε-Quasiorthogonality

Authors: Pierre-David Létourneau, Eric Darve

Abstract: We introduce a new type of quadrature, known as approximate Gaussian quadrature (AGQ) rules using ε-quasiorthogonality, for the approximation of integrals of the form \int f(x)d α(x). The measure α(\cdot) can be arbitrary as long as it possesses finite moments μn for sufficiently large n. The weights and nodes associated with the quadrature can be computed in low complexity and their count is infe… ▽ More We introduce a new type of quadrature, known as approximate Gaussian quadrature (AGQ) rules using ε-quasiorthogonality, for the approximation of integrals of the form \int f(x)d α(x). The measure α(\cdot) can be arbitrary as long as it possesses finite moments μn for sufficiently large n. The weights and nodes associated with the quadrature can be computed in low complexity and their count is inferior to that required by classical quadratures at fixed accuracy on some families of integrands. Furthermore, we show how AGQ can be used to discretize the Fourier transform with few points in order to obtain short exponential representations of functions. △ Less

Submitted 12 November, 2018; originally announced November 2018.

Low-Rank Kernel Matrix Approximation Using Skeletonized Interpolation With Endo- or Exo-Vertices

Authors: Zixi Xu, Léopold Cambier, François-Henry Rouet, Pierre L'Eplatennier, Yun Huang, Cleve Ashcraft, Eric Darve

Abstract: The efficient compression of kernel matrices, for instance the off-diagonal blocks of discretized integral equations, is a crucial step in many algorithms. In this paper, we study the application of Skeletonized Interpolation to construct such factorizations. In particular, we study four different strategies for selecting the initial candidate pivots of the algorithm: Chebyshev grids, points on a… ▽ More The efficient compression of kernel matrices, for instance the off-diagonal blocks of discretized integral equations, is a crucial step in many algorithms. In this paper, we study the application of Skeletonized Interpolation to construct such factorizations. In particular, we study four different strategies for selecting the initial candidate pivots of the algorithm: Chebyshev grids, points on a sphere, maximally-dispersed and random vertices. Among them, the first two introduce new interpolation points (exo-vertices) while the last two are subsets of the given clusters (endo- vertices). We perform experiments using three real-world problems coming from the multiphysics code LS-DYNA. The pivot selection strategies are compared in term of quality (final rank) and efficiency (size of the initial grid). These benchmarks demonstrate that overall, maximally-dispersed vertices provide an accurate and efficient sets of pivots for most applications. It allows to reach near-optimal ranks while starting with relatively small sets of vertices, compared to other strategies. △ Less

Submitted 12 July, 2018; originally announced July 2018.

A distributed-memory hierarchical solver for general sparse linear systems

Authors: Chao Chen, Hadi Pouransari, Sivasankaran Rajamanickam, Erik G. Boman, Eric Darve

Abstract: We present a parallel hierarchical solver for general sparse linear systems on distributed-memory machines. For large-scale problems, this fully algebraic algorithm is faster and more memory-efficient than sparse direct solvers because it exploits the low-rank structure of fill-in blocks. Depending on the accuracy of low-rank approximations, the hierarchical solver can be used either as a direct s… ▽ More We present a parallel hierarchical solver for general sparse linear systems on distributed-memory machines. For large-scale problems, this fully algebraic algorithm is faster and more memory-efficient than sparse direct solvers because it exploits the low-rank structure of fill-in blocks. Depending on the accuracy of low-rank approximations, the hierarchical solver can be used either as a direct solver or as a preconditioner. The parallel algorithm is based on data decomposition and requires only local communication for updating boundary data on every processor. Moreover, the computation-to-communication ratio of the parallel algorithm is approximately the volume-to-surface-area ratio of the subdomain owned by every processor. We present various numerical results to demonstrate the versatility and scalability of the parallel algorithm. △ Less

Submitted 19 December, 2017; originally announced December 2017.

MSC Class: 65F50

On the numerical rank of radial basis function kernels in high dimension

Authors: Ruoxi Wang, Yingzhou Li, Eric Darve

Abstract: Low-rank approximations are popular methods to reduce the high computational cost of algorithms involving large-scale kernel matrices. The success of low-rank methods hinges on the matrix rank of the kernel matrix, and in practice, these methods are effective even for high-dimensional datasets. Their practical success motivates our analysis of the function rank, an upper bound of the matrix rank.… ▽ More Low-rank approximations are popular methods to reduce the high computational cost of algorithms involving large-scale kernel matrices. The success of low-rank methods hinges on the matrix rank of the kernel matrix, and in practice, these methods are effective even for high-dimensional datasets. Their practical success motivates our analysis of the function rank, an upper bound of the matrix rank. In this paper, we consider radial basis functions (RBF), approximate the RBF kernel with a low-rank representation that is a finite sum of separate products and provide explicit upper bounds on the function rank and the $L_\infty$ error for such approximations. Our three main results are as follows. First, for a fixed precision, the function rank of RBFs, in the worst case, grows polynomially with the data dimension. Second, precise error bounds for the low-rank approximations in the $L_\infty$ norm are derived in terms of the function smoothness and the domain diameters. Finally, a group pattern in the magnitude of singular values for RBF kernel matrices is observed and analyzed, and is explained by a grou** of the expansion terms in the kernel's low-rank representation. Empirical results verify the theoretical results. △ Less

Submitted 12 September, 2018; v1 submitted 23 June, 2017; originally announced June 2017.

Journal ref: SIAM Journal on Matrix Analysis and Applications, 2018, Vol. 39, No. 4 : pp. 1810-1835

Fast Low-Rank Kernel Matrix Factorization through Skeletonized Interpolation

Authors: Léopold Cambier, Eric Darve

Abstract: Integral equations are commonly encountered when solving complex physical problems. Their discretization leads to a dense kernel matrix that is block or hierarchically low-rank. This paper proposes a new way to build a low-rank factorization of those low-rank blocks at a nearly optimal cost of $\mathcal{O}(nr)$ for a $n \times n$ block submatrix of rank r. This is done by first sampling the kernel… ▽ More Integral equations are commonly encountered when solving complex physical problems. Their discretization leads to a dense kernel matrix that is block or hierarchically low-rank. This paper proposes a new way to build a low-rank factorization of those low-rank blocks at a nearly optimal cost of $\mathcal{O}(nr)$ for a $n \times n$ block submatrix of rank r. This is done by first sampling the kernel function at new interpolation points, then selecting a subset of those using a CUR decomposition and finally using this reduced set of points as pivots for a RRLU-type factorization. We also explain how this implicitly builds an optimal interpolation basis for the Kernel under consideration. We show the asymptotic convergence of the algorithm, explain his stability and demonstrate on numerical examples that it performs very well in practice, allowing to obtain rank nearly equal to the optimal rank at a fraction of the cost of the naive algorithm. △ Less

Submitted 6 May, 2019; v1 submitted 8 June, 2017; originally announced June 2017.

Sparse Hierarchical Solvers with Guaranteed Convergence

Authors: Kai Yang, Hadi Pouransari, Eric Darve

Abstract: Solving sparse linear systems from discretized PDEs is challenging. Direct solvers have in many cases quadratic complexity (depending on geometry), while iterative solvers require problem dependent preconditioners to be robust and efficient. Approximate factorization preconditioners, such as incomplete LU factorization, provide cheap approximations to the system matrix. However, even a highly accu… ▽ More Solving sparse linear systems from discretized PDEs is challenging. Direct solvers have in many cases quadratic complexity (depending on geometry), while iterative solvers require problem dependent preconditioners to be robust and efficient. Approximate factorization preconditioners, such as incomplete LU factorization, provide cheap approximations to the system matrix. However, even a highly accurate preconditioner may have deteriorating performance when the condition number of the system matrix increases. By increasing the accuracy on low-frequency errors, we propose a novel hierarchical solver with improved robustness with respect to the condition number of the linear system. This solver retains the linear computational cost and memory footprint of the original algorithm. △ Less

Submitted 12 March, 2017; v1 submitted 10 November, 2016; originally announced November 2016.

An efficient preconditioner for the fast simulation of a 2D Stokes flow in porous media

Authors: Pieter Coulier, Bryan Quaife, Eric Darve

Abstract: We consider an efficient preconditioner for boundary integral equation (BIE) formulations of the two-dimensional Stokes equations in porous media. While BIEs are well-suited for resolving the complex porous geometry, they lead to a dense linear system of equations that is computationally expensive to solve for large problems. This expense is further amplified when a significant number of iteration… ▽ More We consider an efficient preconditioner for boundary integral equation (BIE) formulations of the two-dimensional Stokes equations in porous media. While BIEs are well-suited for resolving the complex porous geometry, they lead to a dense linear system of equations that is computationally expensive to solve for large problems. This expense is further amplified when a significant number of iterations is required in an iterative Krylov solver such as GMRES. In this paper, we apply a fast inexact direct solver, the inverse fast multipole method (IFMM), as an efficient preconditioner for GMRES. This solver is based on the framework of $\mathcal{H}^{2}$-matrices and uses low-rank compressions to approximate certain matrix blocks. It has a tunable accuracy $\varepsilon$ and a computational cost that scales as $\mathcal{O} (N \log^2 1/\varepsilon)$. We discuss various numerical benchmarks that validate the accuracy and confirm the efficiency of the proposed method. We demonstrate with several types of boundary conditions that the preconditioner is capable of significantly accelerating the convergence of GMRES when compared to a simple block-diagonal preconditioner, especially for pipe flow problems involving many pores. △ Less

Submitted 14 September, 2016; originally announced September 2016.

Comments: 23 pages, 15 figures, 9 tables

Fast hierarchical solvers for sparse matrices using extended sparsification and low-rank approximation

Authors: Hadi Pouransari, Pieter Coulier, Eric Darve

Abstract: Inversion of sparse matrices with standard direct solve schemes is robust, but computationally expensive. Iterative solvers, on the other hand, demonstrate better scalability; but, need to be used with an appropriate preconditioner (e.g., ILU, AMG, Gauss-Seidel, etc.) for proper convergence. The choice of an effective preconditioner is highly problem dependent. We propose a novel fully algebraic s… ▽ More Inversion of sparse matrices with standard direct solve schemes is robust, but computationally expensive. Iterative solvers, on the other hand, demonstrate better scalability; but, need to be used with an appropriate preconditioner (e.g., ILU, AMG, Gauss-Seidel, etc.) for proper convergence. The choice of an effective preconditioner is highly problem dependent. We propose a novel fully algebraic sparse matrix solve algorithm, which has linear complexity with the problem size. Our scheme is based on the Gauss elimination. For a given matrix, we approximate the LU factorization with a tunable accuracy determined a priori. This method can be used as a stand-alone direct solver with linear complexity and tunable accuracy, or it can be used as a black-box preconditioner in conjunction with iterative methods such as GMRES. The proposed solver is based on the low-rank approximation of fill-ins generated during the elimination. Similar to H-matrices, fill-ins corresponding to blocks that are well-separated in the adjacency graph are represented via a hierarchical structure. The linear complexity of the algorithm is guaranteed if the blocks corresponding to well-separated clusters of variables are numerically low-rank. △ Less

Submitted 14 December, 2016; v1 submitted 26 October, 2015; originally announced October 2015.

Optimizing the adaptive fast multipole method for fractal sets

Authors: Hadi Pouransari, Eric Darve

Abstract: We have performed a detailed analysis of the fast multipole method (FMM) in the adaptive case, in which the depth of the FMM tree is non-uniform. Previous works in this area have focused mostly on special types of adaptive distributions, for example when points accumulate on a 2D manifold or accumulate around a few points in space. Instead, we considered a more general situation in which fractal s… ▽ More We have performed a detailed analysis of the fast multipole method (FMM) in the adaptive case, in which the depth of the FMM tree is non-uniform. Previous works in this area have focused mostly on special types of adaptive distributions, for example when points accumulate on a 2D manifold or accumulate around a few points in space. Instead, we considered a more general situation in which fractal sets, e.g., Cantor sets and generalizations, are used to create adaptive sets of points. Such sets are characterized by their dimension, a number between 0 and 3. We introduced a mathematical framework to define a converging sequence of octrees, and based on that, demonstrated how to increase $N \to \infty$. A new complexity analysis for the adaptive FMM is introduced. It is shown that the ${\cal{O}}(N)$ complexity is achievable for any distribution of particles, when a modified adaptive FMM is exploited. We analyzed how the FMM performs for fractal point distributions, and how optimal parameters can be picked, e.g., the criterion used to stop the subdivision of an FMM cell. A new subdividing double-threshold method is introduced, and better performance demonstrated. Parameters in the FMM are modeled as a function of particle distribution dimension, and the optimal values are obtained. A three dimensional kernel independent black box adaptive FMM is implemented and used for all calculations. △ Less

Submitted 11 August, 2015; originally announced August 2015.

MSC Class: 28A80; 65F99; 70F10

Journal ref: SIAM Journal on Scientific Computing 37.2 (2015): A1040-A1066

The inverse fast multipole method: using a fast approximate direct solver as a preconditioner for dense linear systems

Authors: Pieter Coulier, Hadi Pouransari, Eric Darve

Abstract: Although some preconditioners are available for solving dense linear systems, there are still many matrices for which preconditioners are lacking, in particular in cases where the size of the matrix $N$ becomes very large. There remains hence a great need to develop general purpose preconditioners whose cost scales well with the matrix size $N$. In this paper, we propose a preconditioner with broa… ▽ More Although some preconditioners are available for solving dense linear systems, there are still many matrices for which preconditioners are lacking, in particular in cases where the size of the matrix $N$ becomes very large. There remains hence a great need to develop general purpose preconditioners whose cost scales well with the matrix size $N$. In this paper, we propose a preconditioner with broad applicability and with cost $\mathcal{O}(N)$ for dense matrices, when the matrix is given by a smooth kernel. Extending the method using the same framework to general $\mathcal{H}^2$-matrices is relatively straightforward. These preconditioners have a controlled accuracy (machine accuracy can be achieved if needed) and scale linearly with $N$. They are based on an approximate direct solve of the system. The linear scaling of the algorithm is achieved by means of two key ideas. First, the $\mathcal{H}^2$-structure of the dense matrix is exploited to obtain an extended sparse system of equations. Second, fill-ins arising when performing the elimination are compressed as low-rank matrices if they correspond to well-separated interactions. This ensures that the sparsity pattern of the extended sparse matrix is preserved throughout the elimination, hence resulting in a very efficient algorithm with $\mathcal{O}(N \log(1/\varepsilon)^2 )$ computational cost and $\mathcal{O}(N \log 1/\varepsilon )$ memory requirement, for an error tolerance $0 < \varepsilon < 1$. The solver is inexact, although the error can be controlled and made as small as needed. These solvers are related to ILU in the sense that the fill-in is controlled. However, in ILU, most of the fill-in is simply discarded whereas here it is approximated using low-rank blocks, with a prescribed tolerance. Numerical examples are discussed to demonstrate the linear scaling of the method and to illustrate its effectiveness as a preconditioner. △ Less

Submitted 4 February, 2016; v1 submitted 7 August, 2015; originally announced August 2015.

Comments: Revised version Submitted to the SIAM Journal on Scientific Computing. 35 pages, 29 figures

Block Basis Factorization for Scalable Kernel Matrix Evaluation

Authors: Ruoxi Wang, Yingzhou Li, Michael W. Mahoney, Eric Darve

Abstract: Kernel methods are widespread in machine learning; however, they are limited by the quadratic complexity of the construction, application, and storage of kernel matrices. Low-rank matrix approximation algorithms are widely used to address this problem and reduce the arithmetic and storage cost. However, we observed that for some datasets with wide intra-class variability, the optimal kernel parame… ▽ More Kernel methods are widespread in machine learning; however, they are limited by the quadratic complexity of the construction, application, and storage of kernel matrices. Low-rank matrix approximation algorithms are widely used to address this problem and reduce the arithmetic and storage cost. However, we observed that for some datasets with wide intra-class variability, the optimal kernel parameter for smaller classes yields a matrix that is less well approximated by low-rank methods. In this paper, we propose an efficient structured low-rank approximation method -- the Block Basis Factorization (BBF) -- and its fast construction algorithm to approximate radial basis function (RBF) kernel matrices. Our approach has linear memory cost and floating-point operations for many machine learning kernels. BBF works for a wide range of kernel bandwidth parameters and extends the domain of applicability of low-rank approximation methods significantly. Our empirical results demonstrate the stability and superiority over the state-of-art kernel approximation algorithms. △ Less

Submitted 4 May, 2021; v1 submitted 3 May, 2015; originally announced May 2015.

Comments: 16 pages, 5 figures

Journal ref: SIAM Journal on Matrix Analysis and Applications, 2019, Vol. 40, No. 4 : pp. 1497-1526

A Fast and Memory Efficient Sparse Solver with Applications to Finite-Element Matrices

Authors: AmirHossein Aminfar, Eric Darve

Abstract: In this article, we introduce a fast and memory efficient solver for sparse matrices arising from the finite element discretization of elliptic partial differential equations (PDEs). We use a fast direct (but approximate) multifrontal solver as a preconditioner, and use an iterative solver to achieve a desired accuracy. This approach combines the advantages of direct and iterative schemes to arriv… ▽ More In this article, we introduce a fast and memory efficient solver for sparse matrices arising from the finite element discretization of elliptic partial differential equations (PDEs). We use a fast direct (but approximate) multifrontal solver as a preconditioner, and use an iterative solver to achieve a desired accuracy. This approach combines the advantages of direct and iterative schemes to arrive at a fast, robust and accurate solver. We will show that this solver is faster ($\sim$ 2x) and more memory efficient ($\sim$ 2--3x) than a conventional direct multifrontal solver. Furthermore, we will demonstrate that the solver is both a faster and more effective preconditioner than other preconditioners such as the incomplete LU preconditioner. Specific speed-ups depend on the matrix size and improve as the size of the matrix increases. The solver can be applied to both structured and unstructured meshes in a similar manner. We build on our previous work and utilize the fact that dense frontal and update matrices, in the multifrontal algorithm, can be represented as hierarchically off-diagonal low-rank (HODLR) matrices. Using this idea, we replace all large dense matrix operations in the multifrontal elimination process with $O(N)$ HODLR operations to arrive at a faster and more memory efficient solver. △ Less

Submitted 22 April, 2015; v1 submitted 10 October, 2014; originally announced October 2014.

Comments: 25 pages

The Inverse Fast Multipole Method

Authors: Sivaram Ambikasaran, Eric Darve

Abstract: This article introduces a new fast direct solver for linear systems arising out of wide range of applications, integral equations, multivariate statistics, radial basis interpolation, etc., to name a few. \emph{The highlight of this new fast direct solver is that the solver scales linearly in the number of unknowns in all dimensions.} The solver, termed as Inverse Fast Multipole Method (abbreviate… ▽ More This article introduces a new fast direct solver for linear systems arising out of wide range of applications, integral equations, multivariate statistics, radial basis interpolation, etc., to name a few. \emph{The highlight of this new fast direct solver is that the solver scales linearly in the number of unknowns in all dimensions.} The solver, termed as Inverse Fast Multipole Method (abbreviated as IFMM), works on the same data-structure as the Fast Multipole Method (abbreviated as FMM). More generally, the solver can be immediately extended to the class of hierarchical matrices, denoted as $\mathcal{H}^2$ matrices with strong admissibility criteria (weak low-rank structure), i.e., \emph{the interaction between neighboring cluster of particles is full-rank whereas the interaction between particles corresponding to well-separated clusters can be efficiently represented as a low-rank matrix}. The algorithm departs from existing approaches in the fact that throughout the algorithm the interaction corresponding to neighboring clusters are always treated as full-rank interactions. Our approach relies on two major ideas: (i) The $N \times N$ matrix arising out of FMM (from now on termed as FMM matrix) can be represented as an extended sparser matrix of size $M \times M$, where $M \approx 3N$. (ii) While solving the larger extended sparser matrix, \emph{the fill-in's that arise in the matrix blocks corresponding to well-separated clusters are hierarchically compressed}. The ordering of the equations and the unknowns in the extended sparser matrix is strongly related to the local and multipole coefficients in the FMM~\cite{greengard1987fast} and \emph{the order of elimination is different from the usual nested dissection approach}. Numerical benchmarks on $2$D manifold confirm the linear scaling of the algorithm. △ Less

Submitted 6 July, 2014; originally announced July 2014.

Comments: 25 pages, 28 figures

A Kalman filter powered by $\mathcal{H}^2$-matrices for quasi-continuous data assimilation problems

Authors: Judith Y. Li, Sivaram Ambikasaran, Eric F. Darve, Peter K. Kitanidis

Abstract: Continuously tracking the movement of a fluid or a plume in the subsurface is a challenge that is often encountered in applications, such as tracking a plume of injected CO$_2$ or of a hazardous substance. Advances in monitoring techniques have made it possible to collect measurements at a high frequency while the plume moves, which has the potential advantage of providing continuous high-resoluti… ▽ More Continuously tracking the movement of a fluid or a plume in the subsurface is a challenge that is often encountered in applications, such as tracking a plume of injected CO$_2$ or of a hazardous substance. Advances in monitoring techniques have made it possible to collect measurements at a high frequency while the plume moves, which has the potential advantage of providing continuous high-resolution images of fluid flow with the aid of data processing. However, the applicability of this approach is limited by the high computational cost associated with having to analyze large data sets within the time constraints imposed by real-time monitoring. Existing data assimilation methods have computational requirements that increase super-linearly with the size of the unknowns $m$. In this paper, we present the HiKF, a new Kalman filter (KF) variant powered by the hierarchical matrix approach that dramatically reduces the computational and storage cost of the standard KF from $\mathcal{O}(m^2)$ to $\mathcal{O}(m)$, while producing practically the same results. The version of HiKF that is presented here takes advantage of the so-called random walk dynamical model, which is tailored to a class of data assimilation problems in which measurements are collected quasi-continuously. The proposed method has been applied to a realistic CO$_2$ injection model and compared with the ensemble Kalman filter (EnKF). Numerical results show that HiKF can provide estimates that are more accurate than EnKF, and also demonstrate the usefulness of modeling the system dynamics as a random walk in this context. △ Less

Submitted 15 April, 2014; originally announced April 2014.

Comments: 18 pages, 7 figures. Water Resources Research, 2014

ACM Class: I.4.4; I.4.5; I.4.10; G.1.3; I.1.2

A Fast Block Low-Rank Dense Solver with Applications to Finite-Element Matrices

Authors: Amirhossein Aminfar, Sivaram Ambikasaran, Eric Darve

Abstract: This article presents a fast solver for the dense "frontal" matrices that arise from the multifrontal sparse elimination process of 3D elliptic PDEs. The solver relies on the fact that these matrices can be efficiently represented as a hierarchically off-diagonal low-rank (HODLR) matrix. To construct the low-rank approximation of the off-diagonal blocks, we propose a new pseudo-skeleton scheme, th… ▽ More This article presents a fast solver for the dense "frontal" matrices that arise from the multifrontal sparse elimination process of 3D elliptic PDEs. The solver relies on the fact that these matrices can be efficiently represented as a hierarchically off-diagonal low-rank (HODLR) matrix. To construct the low-rank approximation of the off-diagonal blocks, we propose a new pseudo-skeleton scheme, the boundary distance low-rank approximation, that picks rows and columns based on the location of their corresponding vertices in the sparse matrix graph. We compare this new low-rank approximation method to the adaptive cross approximation (ACA) algorithm and show that it achieves betters speedup specially for unstructured meshes. Using the HODLR direct solver as a preconditioner (with a low tolerance) to the GMRES iterative scheme, we can reach machine accuracy much faster than a conventional LU solver. Numerical benchmarks are provided for frontal matrices arising from 3D finite element problems corresponding to a wide range of applications. △ Less

Submitted 18 March, 2015; v1 submitted 20 March, 2014; originally announced March 2014.

Computing reaction rates in bio-molecular systems using discrete macro-states

Authors: Eric Darve, Ernest Ryu

Abstract: Computing reaction rates in biomolecular systems is a common goal of molecular dynamics simulations. The reactions considered often involve conformational changes in the molecule, either changes in the structure of a protein or the relative position of two molecules, for example when modeling the binding of a protein and ligand. Here we will consider the general problem of computing the rate of tr… ▽ More Computing reaction rates in biomolecular systems is a common goal of molecular dynamics simulations. The reactions considered often involve conformational changes in the molecule, either changes in the structure of a protein or the relative position of two molecules, for example when modeling the binding of a protein and ligand. Here we will consider the general problem of computing the rate of transfer from a subset A of the conformational space Omega to a subset B of Omega. It is assumed that A and B are associated with minimum energy basins and are long-lived states. Rates can be obtained using many different methods. We review some of the most popular approaches. We organize the different approaches roughly in chronological order and under four main categories: reactive flux, transition path sampling, conformation dynamics. The fourth class of methods, to which we do not give any specific name, in some sense attempts to combine features from transition path sampling and conformation dynamics. They include non-equilibrium umbrella sampling (Warmflash et al. [2007], Dickson et al. [2009b]), and weighted ensemble dynamics (Huber and Kim [1996]). △ Less

Submitted 2 July, 2013; originally announced July 2013.

Optimized M2L Kernels for the Chebyshev Interpolation based Fast Multipole Method

Authors: Matthias Messner, Bérenger Bramas, Olivier Coulaud, Eric Darve

Abstract: A fast multipole method (FMM) for asymptotically smooth kernel functions (1/r, 1/r^4, Gauss and Stokes kernels, radial basis functions, etc.) based on a Chebyshev interpolation scheme has been introduced in [Fong et al., 2009]. The method has been extended to oscillatory kernels (e.g., Helmholtz kernel) in [Messner et al., 2012]. Beside its generality this FMM turns out to be favorable due to its… ▽ More A fast multipole method (FMM) for asymptotically smooth kernel functions (1/r, 1/r^4, Gauss and Stokes kernels, radial basis functions, etc.) based on a Chebyshev interpolation scheme has been introduced in [Fong et al., 2009]. The method has been extended to oscillatory kernels (e.g., Helmholtz kernel) in [Messner et al., 2012]. Beside its generality this FMM turns out to be favorable due to its easy implementation and its high performance based on intensive use of highly optimized BLAS libraries. However, one of its bottlenecks is the precomputation of the multiple-to-local (M2L) operator, and its higher number of floating point operations (flops) compared to other FMM formulations. Here, we present several optimizations for that operator, which is known to be the costliest FMM operator. The most efficient ones do not only reduce the precomputation time by a factor up to 340 but they also speed up the matrix-vector product. We conclude with comparisons and numerical validations of all presented optimizations. △ Less

Submitted 20 November, 2012; v1 submitted 27 October, 2012; originally announced October 2012.

Extension and optimization of the FIND algorithm: computing Green's and less-than Green's functions (with technical appendix)

Authors: Song Li, Eric Darve

Abstract: The FIND algorithm is a fast algorithm designed to calculate certain entries of the inverse of a sparse matrix. Such calculation is critical in many applications, e.g., quantum transport in nano-devices. We extended the algorithm to other matrix inverse related calculations. Those are required for example to calculate the less-than Green's function and the current density through the device. For a… ▽ More The FIND algorithm is a fast algorithm designed to calculate certain entries of the inverse of a sparse matrix. Such calculation is critical in many applications, e.g., quantum transport in nano-devices. We extended the algorithm to other matrix inverse related calculations. Those are required for example to calculate the less-than Green's function and the current density through the device. For a 2D device discretized as an N_x x N_y mesh, the best known algorithms have a running time of O(N_x^3 N_y), whereas FIND only requires O(N_x^2 N_y). Even though this complexity has been reduced by an order of magnitude, the matrix inverse calculation is still the most time consuming part in the simulation of transport problems. We could not reduce the order of complexity, but we were able to significantly reduce the constant factor involved in the computation cost. By exploiting the sparsity and symmetry, the size of the problem beyond which FIND is faster than other methods typically decreases from a 130x130 2D mesh down to a 40x40 mesh. These improvements make the optimized FIND algorithm even more competitive for real-life applications. △ Less

Submitted 4 April, 2011; originally announced April 2011.

Fourier Based Fast Multipole Method for the Helmholtz Equation

Authors: Cris Cecka, Eric Darve

Abstract: The fast multipole method (FMM) has had great success in reducing the computational complexity of solving the boundary integral form of the Helmholtz equation. We present a formulation of the Helmholtz FMM that uses Fourier basis functions rather than spherical harmonics. By modifying the transfer function in the precomputation stage of the FMM, time-critical stages of the algorithm are accelerate… ▽ More The fast multipole method (FMM) has had great success in reducing the computational complexity of solving the boundary integral form of the Helmholtz equation. We present a formulation of the Helmholtz FMM that uses Fourier basis functions rather than spherical harmonics. By modifying the transfer function in the precomputation stage of the FMM, time-critical stages of the algorithm are accelerated by causing the interpolation operators to become straightforward applications of fast Fourier transforms, retaining the diagonality of the transfer function, and providing a simplified error analysis. Using Fourier analysis, constructive algorithms are derived to a priori determine an integration quadrature for a given error tolerance. Sharp error bounds are derived and verified numerically. Various optimizations are considered to reduce the number of quadrature points and reduce the cost of computing the transfer function. △ Less

Submitted 25 October, 2011; v1 submitted 20 November, 2009; originally announced November 2009.

Comments: 24 pages, 13 figures

MSC Class: 65T40