-
An Optimal Transport Approach for Computing Adversarial Training Lower Bounds in Multiclass Classification
Authors:
Nicolas Garcia Trillos,
Matt Jacobs,
Jakwang Kim,
Matthew Werenski
Abstract:
Despite the success of deep learning-based algorithms, it is widely known that neural networks may fail to be robust. A popular paradigm to enforce robustness is adversarial training (AT), however, this introduces many computational and theoretical difficulties. Recent works have developed a connection between AT in the multiclass classification setting and multimarginal optimal transport (MOT), u…
▽ More
Despite the success of deep learning-based algorithms, it is widely known that neural networks may fail to be robust. A popular paradigm to enforce robustness is adversarial training (AT), however, this introduces many computational and theoretical difficulties. Recent works have developed a connection between AT in the multiclass classification setting and multimarginal optimal transport (MOT), unlocking a new set of tools to study this problem. In this paper, we leverage the MOT connection to propose computationally tractable numerical algorithms for computing universal lower bounds on the optimal adversarial risk and identifying optimal classifiers. We propose two main algorithms based on linear programming (LP) and entropic regularization (Sinkhorn). Our key insight is that one can harmlessly truncate the higher order interactions between classes, preventing the combinatorial run times typically encountered in MOT problems. We validate these results with experiments on MNIST and CIFAR-$10$, which demonstrate the tractability of our approach.
△ Less
Submitted 17 January, 2024;
originally announced January 2024.
-
Nonlocal Approximation of Slow and Fast Diffusion
Authors:
Katy Craig,
Matt Jacobs,
Olga Turanova
Abstract:
Motivated by recent work on approximation of diffusion equations by deterministic interacting particle systems, we develop a nonlocal approximation for a range of linear and nonlinear diffusion equations and prove convergence of the method in the slow, linear, and fast diffusion regimes. A key ingredient of our approach is a novel technique for using the 2-Wasserstein and dual Sobolev gradient flo…
▽ More
Motivated by recent work on approximation of diffusion equations by deterministic interacting particle systems, we develop a nonlocal approximation for a range of linear and nonlinear diffusion equations and prove convergence of the method in the slow, linear, and fast diffusion regimes. A key ingredient of our approach is a novel technique for using the 2-Wasserstein and dual Sobolev gradient flow structures of the diffusion equations to recover the duality relation characterizing the pressure in the nonlocal-to-local limit. Due to the general class of internal energy densities that our method is able to handle, a byproduct of our result is a novel particle method for sampling a wide range of probability measures, which extends classical approaches based on the Fokker-Planck equation beyond the log-concave setting.
△ Less
Submitted 3 April, 2024; v1 submitted 18 December, 2023;
originally announced December 2023.
-
Optimal Transport of Linear Systems over Equilibrium Measures
Authors:
Karthik Elamvazhuthi,
Matt Jacobs
Abstract:
We consider the optimal transport problem over convex costs arising from optimal control of linear time-invariant(LTI) systems when the initial and target measures are assumed to be supported on the set of equilibrium points of the LTI system. In this case, the probability measures are singular with respect to the Lebesgue measure, thus not considered in previous results on optimal transport of li…
▽ More
We consider the optimal transport problem over convex costs arising from optimal control of linear time-invariant(LTI) systems when the initial and target measures are assumed to be supported on the set of equilibrium points of the LTI system. In this case, the probability measures are singular with respect to the Lebesgue measure, thus not considered in previous results on optimal transport of linear systems. This problem is motivated by applications, such as robotics, where the initial and target configurations of robots, represented by measures, are in equilibrium or stationary. Despite the singular nature of the measures, for many cases of practical interest, we show that the Monge problem has a solution by applying classical optimal transport results. Moreover, the problem is computationally tractable even if the state space of the LTI system is moderately high in dimension, provided the equilibrium set lives in a low dimensional space. In fact, for an important subclass of linear quadratic problems, such as control of the double integrator with linear quadratic cost, the optimal transport map happens to coincide with that of the Euclidean cost. We demonstrate our results by computing the optimal transport map for the minimum energy cost for a two dimensional double integrator, despite the fact that the state space is four dimensional due to position and velocity variables.
△ Less
Submitted 15 December, 2023;
originally announced December 2023.
-
Free boundary regularity for tumor growth with nutrients and diffusion
Authors:
Carson Collins,
Matt Jacobs,
Inwon Kim
Abstract:
In this paper, we study a tumor growth model where the growth is driven by nutrient availability and the tumor expands according to Darcy's law with a mechanical pressure resulting from the incompressibility of the cells. Our focus is on the free boundary regularity of the tumor patch that holds beyond topological changes. A crucial element in our analysis is establishing the regularity of the hit…
▽ More
In this paper, we study a tumor growth model where the growth is driven by nutrient availability and the tumor expands according to Darcy's law with a mechanical pressure resulting from the incompressibility of the cells. Our focus is on the free boundary regularity of the tumor patch that holds beyond topological changes. A crucial element in our analysis is establishing the regularity of the hitting time T, which records the first time the tumor patch reaches a given point. We achieve this by introducing a novel Hamilton-Jacobi-Bellman (HJB) interpretation of the pressure, which is of independent interest. The HJB structure is obtained by viewing the model as a limit of the Porous Media Equation (PME) and building upon a new variant of the AB estimate. Using the HJB structure, we establish a new Hopf-Lax type formula for the pressure variable. Combined with barrier arguments, the formula allows us to show that T is C^α, where αdepends only on the dimension, which translates into a mild nondegeneracy of the tumor patch evolution. Building on this and obstacle problem theory, we show that the tumor patch boundary is regular in spacetime except on a set of Hausdorff dimension at most $d-α$. On the set of regular points, we further show that the tumor patch is locally $C^{1,α}$ in space-time. This conclusively establishes that instabilities in the boundary evolution do not amplify arbitrarily high frequencies.
△ Less
Submitted 12 September, 2023;
originally announced September 2023.
-
Memory Efficient And Minimax Distribution Estimation Under Wasserstein Distance Using Bayesian Histograms
Authors:
Peter Matthew Jacobs,
Lekha Patel,
Anirban Bhattacharya,
Debdeep Pati
Abstract:
We study Bayesian histograms for distribution estimation on $[0,1]^d$ under the Wasserstein $W_v, 1 \leq v < \infty$ distance in the i.i.d sampling regime. We newly show that when $d < 2v$, histograms possess a special \textit{memory efficiency} property, whereby in reference to the sample size $n$, order $n^{d/2v}$ bins are needed to obtain minimax rate optimality. This result holds for the poste…
▽ More
We study Bayesian histograms for distribution estimation on $[0,1]^d$ under the Wasserstein $W_v, 1 \leq v < \infty$ distance in the i.i.d sampling regime. We newly show that when $d < 2v$, histograms possess a special \textit{memory efficiency} property, whereby in reference to the sample size $n$, order $n^{d/2v}$ bins are needed to obtain minimax rate optimality. This result holds for the posterior mean histogram and with respect to posterior contraction: under the class of Borel probability measures and some classes of smooth densities. The attained memory footprint overcomes existing minimax optimal procedures by a polynomial factor in $n$; for example an $n^{1 - d/2v}$ factor reduction in the footprint when compared to the empirical measure, a minimax estimator in the Borel probability measure class. Additionally constructing both the posterior mean histogram and the posterior itself can be done super--linearly in $n$. Due to the popularity of the $W_1,W_2$ metrics and the coverage provided by the $d < 2v$ case, our results are of most practical interest in the $(d=1,v =1,2), (d=2,v=2), (d=3,v=2)$ settings and we provide simulations demonstrating the theory in several of these instances.
△ Less
Submitted 19 July, 2023;
originally announced July 2023.
-
It begins with a boundary: A geometric view on probabilistically robust learning
Authors:
Leon Bungert,
Nicolás García Trillos,
Matt Jacobs,
Daniel McKenzie,
Đorđe Nikolić,
Qingsong Wang
Abstract:
Although deep neural networks have achieved super-human performance on many classification tasks, they often exhibit a worrying lack of robustness towards adversarially generated examples. Thus, considerable effort has been invested into reformulating Empirical Risk Minimization (ERM) into an adversarially robust framework. Recently, attention has shifted towards approaches which interpolate betwe…
▽ More
Although deep neural networks have achieved super-human performance on many classification tasks, they often exhibit a worrying lack of robustness towards adversarially generated examples. Thus, considerable effort has been invested into reformulating Empirical Risk Minimization (ERM) into an adversarially robust framework. Recently, attention has shifted towards approaches which interpolate between the robustness offered by adversarial training and the higher clean accuracy and faster training times of ERM. In this paper, we take a fresh and geometric view on one such method -- Probabilistically Robust Learning (PRL) (Robey et al., ICML, 2022). We propose a geometric framework for understanding PRL, which allows us to identify a subtle flaw in its original formulation and to introduce a family of probabilistic nonlocal perimeter functionals to address this. We prove existence of solutions using novel relaxation methods and study properties as well as local limits of the introduced perimeters.
△ Less
Submitted 30 May, 2023;
originally announced May 2023.
-
On the existence of solutions to adversarial training in multiclass classification
Authors:
Nicolas Garcia Trillos,
Matt Jacobs,
Jakwang Kim
Abstract:
We study three models of the problem of adversarial training in multiclass classification designed to construct robust classifiers against adversarial perturbations of data in the agnostic-classifier setting. We prove the existence of Borel measurable robust classifiers in each model and provide a unified perspective of the adversarial training problem, expanding the connections with optimal trans…
▽ More
We study three models of the problem of adversarial training in multiclass classification designed to construct robust classifiers against adversarial perturbations of data in the agnostic-classifier setting. We prove the existence of Borel measurable robust classifiers in each model and provide a unified perspective of the adversarial training problem, expanding the connections with optimal transport initiated by the authors in previous work and develo** new connections between adversarial training in the multiclass setting and total variation regularization. As a corollary of our results, we prove the existence of Borel measurable solutions to the agnostic adversarial training problem in the binary classification setting, a result that improves results in the literature of adversarial training, where robust classifiers were only known to exist within the enlarged universal $σ$-algebra of the feature space.
△ Less
Submitted 29 May, 2023; v1 submitted 28 April, 2023;
originally announced May 2023.
-
Lagrangian solutions to the Porous Media Equation and Reaction Diffusion Systems
Authors:
Matt Jacobs
Abstract:
In this paper, we construct global-in-time forward and backward Lagrangian flow maps along the pressure gradient generated by weak solutions of the Porous Media Equation. The main difficulty is that when the initial data has compact support, it is well-known that the pressure gradient is not a BV function. Thus, the theory of regular Lagrangian flows cannot be applied to construct the flow maps. T…
▽ More
In this paper, we construct global-in-time forward and backward Lagrangian flow maps along the pressure gradient generated by weak solutions of the Porous Media Equation. The main difficulty is that when the initial data has compact support, it is well-known that the pressure gradient is not a BV function. Thus, the theory of regular Lagrangian flows cannot be applied to construct the flow maps. To overcome this difficulty, we develop a new argument that combines Aronson-Bénilan type estimates with the quantitative Lagrangian flow theory of Crippa and De Lellis to show that certain doubly logarithmic quantities measuring the stability of flow maps do not blow up fast enough to prevent compactness. Our arguments are sufficiently flexible to handle the Hele-Shaw limit and a multispecies generalization of the Porous Media Equation where the equation is replaced by a coupled hyperbolic-parabolic system of reaction diffusion equations. As one application of our flow maps, we are able to construct solutions where different species cannot mix together if they were separated at initial time.
△ Less
Submitted 18 March, 2023; v1 submitted 2 August, 2022;
originally announced August 2022.
-
The Multimarginal Optimal Transport Formulation of Adversarial Multiclass Classification
Authors:
Nicolas Garcia Trillos,
Matt Jacobs,
Jakwang Kim
Abstract:
We study a family of adversarial multiclass classification problems and provide equivalent reformulations in terms of: 1) a family of generalized barycenter problems introduced in the paper and 2) a family of multimarginal optimal transport problems where the number of marginals is equal to the number of classes in the original classification problem. These new theoretical results reveal a rich ge…
▽ More
We study a family of adversarial multiclass classification problems and provide equivalent reformulations in terms of: 1) a family of generalized barycenter problems introduced in the paper and 2) a family of multimarginal optimal transport problems where the number of marginals is equal to the number of classes in the original classification problem. These new theoretical results reveal a rich geometric structure of adversarial learning problems in multiclass classification and extend recent results restricted to the binary classification setting. A direct computational implication of our results is that by solving either the barycenter problem and its dual, or the MOT problem and its dual, we can recover the optimal robust classification rule and the optimal adversarial strategy for the original adversarial problem. Examples with synthetic and real data illustrate our results.
△ Less
Submitted 26 May, 2023; v1 submitted 26 April, 2022;
originally announced April 2022.
-
Tumor Growth with Nutrients: Regularity and Stability
Authors:
Matt Jacobs,
Inwon Kim,
Jiajun Tong
Abstract:
In this paper we study a tumor growth model with nutrients. The model presents dynamic patch solutions due to the contact inhibition among the tumor cells. We show that when the nutrients do not diffuse and the cells do not die, the tumor density exhibits regularizing dynamics. In particular, we provide contraction estimates, exponential rate of asymptotic convergence, and boundary regularity of t…
▽ More
In this paper we study a tumor growth model with nutrients. The model presents dynamic patch solutions due to the contact inhibition among the tumor cells. We show that when the nutrients do not diffuse and the cells do not die, the tumor density exhibits regularizing dynamics. In particular, we provide contraction estimates, exponential rate of asymptotic convergence, and boundary regularity of the tumor patch. These results are in sharp contrast to the models either with nutrient diffusion or with death rate in tumor cells.
△ Less
Submitted 15 April, 2022;
originally announced April 2022.
-
Existence of solutions to reaction cross diffusion systems
Authors:
Matt Jacobs
Abstract:
Reaction cross diffusion systems are a two species generalization of the porous media equation. These systems play an important role in the mechanical modeling of living tissues and tumor growth. Due to their mixed parabolic-hyperbolic structure, even proving the existence of solutions to these equations is challenging. In this paper, we exploit the parabolic structure of the system to prove the s…
▽ More
Reaction cross diffusion systems are a two species generalization of the porous media equation. These systems play an important role in the mechanical modeling of living tissues and tumor growth. Due to their mixed parabolic-hyperbolic structure, even proving the existence of solutions to these equations is challenging. In this paper, we exploit the parabolic structure of the system to prove the strong compactness of the pressure gradient in L2. The key ingredient is the energy dissipation relation, which along with some compensated compactness arguments, allows us to upgrade weak convergence to strong convergence. As a consequence of the pressure compactness, we are able to prove the existence of solutions in a very general setting and pass to the Hele-Shaw/incompressible limit in any dimension.
△ Less
Submitted 26 July, 2021;
originally announced July 2021.
-
The back-and-forth method for Wasserstein gradient flows
Authors:
Matt Jacobs,
Wonjun Lee,
Flavien Léger
Abstract:
We present a method to efficiently compute Wasserstein gradient flows. Our approach is based on a generalization of the back-and-forth method (BFM) introduced by Jacobs and Léger to solve optimal transport problems. We evolve the gradient flow by solving the dual problem to the JKO scheme. In general, the dual problem is much better behaved than the primal problem. This allows us to efficiently ru…
▽ More
We present a method to efficiently compute Wasserstein gradient flows. Our approach is based on a generalization of the back-and-forth method (BFM) introduced by Jacobs and Léger to solve optimal transport problems. We evolve the gradient flow by solving the dual problem to the JKO scheme. In general, the dual problem is much better behaved than the primal problem. This allows us to efficiently run large-scale simulations for a large class of internal energies including singular and non-convex energies.
△ Less
Submitted 16 November, 2020;
originally announced November 2020.
-
Well-posedness and Regularity for a Polyconvex Energy
Authors:
Wilfrid Gangbo,
Matt Jacobs,
Inwon Kim
Abstract:
We prove the existence, uniqueness, and regularity of minimizers of a polyconvex functional in two and three dimensions, which corresponds to the $H^1$ projection of measure-preserving maps. Our result introduces a new criteria on the uniqueness of the minimizer, based on the smallness of the lagrange multiplier. No estimate on the second derivatives of the pressure is needed to get a unique globa…
▽ More
We prove the existence, uniqueness, and regularity of minimizers of a polyconvex functional in two and three dimensions, which corresponds to the $H^1$ projection of measure-preserving maps. Our result introduces a new criteria on the uniqueness of the minimizer, based on the smallness of the lagrange multiplier. No estimate on the second derivatives of the pressure is needed to get a unique global minimizer. As an application, we construct a minimizing movement scheme to construct $L^r$ solutions of the Navier-Stokes equation for a short time interval.
△ Less
Submitted 7 November, 2020;
originally announced November 2020.
-
Darcy's Law with a Source term
Authors:
Matt Jacobs,
Inwon Kim,
Jiajun Tong
Abstract:
We introduce a novel variant of the JKO scheme to approximate Darcy's law with a pressure dependent source term. By introducing a new variable that implicitly controls the source term, our scheme is still able to use the standard Wasserstein-2-metric even though the total mass changes over time. Leveraging the dual formulation of our scheme, we show that the discrete-in-time approximations satisfy…
▽ More
We introduce a novel variant of the JKO scheme to approximate Darcy's law with a pressure dependent source term. By introducing a new variable that implicitly controls the source term, our scheme is still able to use the standard Wasserstein-2-metric even though the total mass changes over time. Leveraging the dual formulation of our scheme, we show that the discrete-in-time approximations satisfy many useful properties expected for the continuum solutions, such as a comparison principle and uniform $L^1$-equicontinuity. Many of these properties are new even in the well-understood case where the growth term is absent. Finally, we show that our discrete approximations converge to a solution of the corresponding PDE system, including a tumor growth model with a general nonlinear source term.
△ Less
Submitted 16 June, 2020;
originally announced June 2020.
-
The $L^1$-contraction principle in optimal transport
Authors:
Matt Jacobs,
Inwon Kim,
Jiajun Tong
Abstract:
In this work, we use the JKO scheme to approximate a general class of diffusion problems generated by Darcy's law. Although the scheme is now classical, if the energy density is spatially inhomogeneous or irregular, many standard methods fail to apply to establish convergence in the continuum limit. To overcome these difficulties, we analyze the scheme through its dual problem and establish a nove…
▽ More
In this work, we use the JKO scheme to approximate a general class of diffusion problems generated by Darcy's law. Although the scheme is now classical, if the energy density is spatially inhomogeneous or irregular, many standard methods fail to apply to establish convergence in the continuum limit. To overcome these difficulties, we analyze the scheme through its dual problem and establish a novel $L^1$-contraction principle for the density variable. Notably, the contraction principle relies only on the existence of an optimal transport map and the convexity structure of the energy. As a result, the principle holds in a very general setting, and opens the door to using optimal-transport-based variational schemes to study a larger class of non-linear inhomogeneous parabolic equations.
△ Less
Submitted 16 June, 2020;
originally announced June 2020.
-
Computational methods for nonlocal mean field games with applications
Authors:
Siting Liu,
Matthew Jacobs,
Wuchen Li,
Levon Nurbekyan,
Stanley J. Osher
Abstract:
We introduce a novel framework to model and solve mean-field game systems with nonlocal interactions. Our approach relies on kernel-based representations of mean-field interactions and feature-space expansions in the spirit of kernel methods in machine learning. We demonstrate the flexibility of our approach by modeling various interaction scenarios between agents. Additionally, our method yields…
▽ More
We introduce a novel framework to model and solve mean-field game systems with nonlocal interactions. Our approach relies on kernel-based representations of mean-field interactions and feature-space expansions in the spirit of kernel methods in machine learning. We demonstrate the flexibility of our approach by modeling various interaction scenarios between agents. Additionally, our method yields a computationally efficient saddle-point reformulation of the original problem that is amenable to state-of-the-art convex optimization methods such as the primal-dual hybrid gradient method (PDHG). We also discuss potential applications of our methods to multi-agent trajectory planning problems.
△ Less
Submitted 28 April, 2020; v1 submitted 25 April, 2020;
originally announced April 2020.
-
A fast approach to optimal transport: The back-and-forth method
Authors:
Matt Jacobs,
Flavien Léger
Abstract:
We present an iterative method to efficiently solve the optimal transportation problem for a class of strictly convex costs which includes quadratic and p-power costs. Given two probability measures supported on a discrete grid with n points, we compute the optimal map using O(n) storage space and O(n log(n)) operations per iteration, with an approximately exponential convergence rate. Our approac…
▽ More
We present an iterative method to efficiently solve the optimal transportation problem for a class of strictly convex costs which includes quadratic and p-power costs. Given two probability measures supported on a discrete grid with n points, we compute the optimal map using O(n) storage space and O(n log(n)) operations per iteration, with an approximately exponential convergence rate. Our approach allows us to solve optimal transportation problems on spatial grids as large as 4096x4096 and 384x384x384 in a matter of minutes.
△ Less
Submitted 5 May, 2020; v1 submitted 28 May, 2019;
originally announced May 2019.
-
Weak solutions to the Muskat problem with surface tension via optimal transport
Authors:
Matt Jacobs,
Inwon Kim,
Alpár R. Mészáros
Abstract:
Inspired by recent works on the threshold dynamics scheme for multi-phase mean curvature flow (by Esedoglu-Otto and Laux-Otto), we introduce a novel framework to approximate solutions of the Muskat problem with surface tension. Our approach is based on interpreting the Muskat problem as a gradient flow in a product Wasserstein space. This perspective allows us to construct weak solutions via a min…
▽ More
Inspired by recent works on the threshold dynamics scheme for multi-phase mean curvature flow (by Esedoglu-Otto and Laux-Otto), we introduce a novel framework to approximate solutions of the Muskat problem with surface tension. Our approach is based on interpreting the Muskat problem as a gradient flow in a product Wasserstein space. This perspective allows us to construct weak solutions via a minimizing movements scheme. Rather than working directly with the singular surface tension force, we instead relax the perimeter functional with the heat content energy approximation of Esedoglu-Otto. The heat content energy allows us to show the convergence of the associated minimizing movement scheme in the Wasserstein space, and makes the scheme far more tractable for numerical simulations. Under a typical energy convergence assumption, we show that our scheme converges to weak solutions of the Muskat problem with surface tension. We then conclude the paper with a discussion on some numerical experiments and on equilibrium configurations.
△ Less
Submitted 15 September, 2020; v1 submitted 13 May, 2019;
originally announced May 2019.
-
Solving Large-Scale Optimization Problems with a Convergence Rate Independent of Grid Size
Authors:
Matt Jacobs,
Flavien Léger,
Wuchen Li,
Stanley Osher
Abstract:
We present a primal-dual method to solve L1-type non-smooth optimization problems independently of the grid size. We apply these results to two important problems : the Rudin-Osher-Fatemi image denoising model and the L1 earth mover's distance from optimal transport. Crucially, we provide analysis that determines the choice of optimal step sizes and we prove that our method converges independently…
▽ More
We present a primal-dual method to solve L1-type non-smooth optimization problems independently of the grid size. We apply these results to two important problems : the Rudin-Osher-Fatemi image denoising model and the L1 earth mover's distance from optimal transport. Crucially, we provide analysis that determines the choice of optimal step sizes and we prove that our method converges independently of the grid size. Our approach allows us to solve these problems on grids as large as 4096 by 4096 in a few minutes without parallelization.
△ Less
Submitted 23 May, 2018;
originally announced May 2018.
-
Connecting Global and Universal Rigidity
Authors:
Matthew Jacobs
Abstract:
A d-dimensional framework is an embedding of the vertices and edges of a graph in Euclidean space. A d-dimensional framework is globally rigid if every other d-dimensional framework with the same edge lengths has the same pairwise distances between the vertices. A graph is generically globally rigid in dimension d (d-GGR) if every generic framework is globally rigid. The d-dimensional framework of…
▽ More
A d-dimensional framework is an embedding of the vertices and edges of a graph in Euclidean space. A d-dimensional framework is globally rigid if every other d-dimensional framework with the same edge lengths has the same pairwise distances between the vertices. A graph is generically globally rigid in dimension d (d-GGR) if every generic framework is globally rigid. The d-dimensional framework of a d-GGR graph is universally rigid if for d' greater than or equal to d every d'-dimensional framework with the same edge lengths has the same pairwise distances between the vertices. We establish a strong connection between global and universal rigidity by showing that all 1 and 2-GGR graphs and an infinite number of higher dimensional d-GGR graphs have a generic universally rigid framework.
△ Less
Submitted 25 December, 2010; v1 submitted 17 November, 2010;
originally announced November 2010.
