-
Density Estimation via Measure Transport: Outlook for Applications in the Biological Sciences
Authors:
Vanessa Lopez-Marrero,
Patrick R. Johnstone,
Gilchan Park,
Xihaier Luo
Abstract:
One among several advantages of measure transport methods is that they allow for a unified framework for processing and analysis of data distributed according to a wide class of probability measures. Within this context, we present results from computational studies aimed at assessing the potential of measure transport techniques, specifically, the use of triangular transport maps, as part of a wo…
▽ More
One among several advantages of measure transport methods is that they allow for a unified framework for processing and analysis of data distributed according to a wide class of probability measures. Within this context, we present results from computational studies aimed at assessing the potential of measure transport techniques, specifically, the use of triangular transport maps, as part of a workflow intended to support research in the biological sciences. Scenarios characterized by the availability of limited amount of sample data, which are common in domains such as radiation biology, are of particular interest. We find that when estimating a distribution density function given limited amount of sample data, adaptive transport maps are advantageous. In particular, statistics gathered from computing series of adaptive transport maps, trained on a series of randomly chosen subsets of the set of available data samples, leads to uncovering information hidden in the data. As a result, in the radiation biology application considered here, this approach provides a tool for generating hypotheses about gene relationships and their dynamics under radiation exposure.
△ Less
Submitted 12 May, 2024; v1 submitted 26 September, 2023;
originally announced September 2023.
-
Stochastic Projective Splitting: Solving Saddle-Point Problems with Multiple Regularizers
Authors:
Patrick R. Johnstone,
Jonathan Eckstein,
Thomas Flynn,
Shinjae Yoo
Abstract:
We present a new, stochastic variant of the projective splitting (PS) family of algorithms for monotone inclusion problems. It can solve min-max and noncooperative game formulations arising in applications such as robust ML without the convergence issues associated with gradient descent-ascent, the current de facto standard approach in such situations. Our proposal is the first version of PS able…
▽ More
We present a new, stochastic variant of the projective splitting (PS) family of algorithms for monotone inclusion problems. It can solve min-max and noncooperative game formulations arising in applications such as robust ML without the convergence issues associated with gradient descent-ascent, the current de facto standard approach in such situations. Our proposal is the first version of PS able to use stochastic (as opposed to deterministic) gradient oracles. It is also the first stochastic method that can solve min-max games while easily handling multiple constraints and nonsmooth regularizers via projection and proximal operators. We close with numerical experiments on a distributionally robust sparse logistic regression problem.
△ Less
Submitted 24 June, 2021;
originally announced June 2021.
-
Single-Forward-Step Projective Splitting: Exploiting Cocoercivity
Authors:
Patrick R. Johnstone,
Jonathan Eckstein
Abstract:
This work describes a new variant of projective splitting for solving maximal monotone inclusions and complicated convex optimization problems. In the new version, cocoercive operators can be processed with a single forward step per iteration. In the convex optimization context, cocoercivity is equivalent to Lipschitz differentiability. Prior forward-step versions of projective splitting did not f…
▽ More
This work describes a new variant of projective splitting for solving maximal monotone inclusions and complicated convex optimization problems. In the new version, cocoercive operators can be processed with a single forward step per iteration. In the convex optimization context, cocoercivity is equivalent to Lipschitz differentiability. Prior forward-step versions of projective splitting did not fully exploit cocoercivity and required two forward steps per iteration for such operators. Our new single-forward-step method establishes a symmetry between projective splitting algorithms, the classical forward-backward splitting method (FB), and Tseng's forward-backward-forward method (FBF). The new procedure allows for larger stepsizes for cocoercive operators: the stepsize bound is $2β$ for a $β$-cocoercive operator, the same bound as has been established for FB. We show that FB corresponds to an unattainable boundary case of the parameters in the new procedure. Unlike FB, the new method allows for a backtracking procedure when the cocoercivity constant is unknown. Proving convergence of the algorithm requires some departures from the prior proof framework for projective splitting. We close with some computational tests establishing competitive performance for the method.
△ Less
Submitted 20 August, 2020; v1 submitted 24 February, 2019;
originally announced February 2019.
-
Projective Splitting with Forward Steps only Requires Continuity
Authors:
Patrick R. Johnstone,
Jonathan Eckstein
Abstract:
A recent innovation in projective splitting algorithms for monotone operator inclusions has been the development of a procedure using two forward steps instead of the customary proximal steps for operators that are Lipschitz continuous. This paper shows that the Lipschitz assumption is unnecessary when the forward steps are performed in finite-dimensional spaces: a backtracking linesearch yields a…
▽ More
A recent innovation in projective splitting algorithms for monotone operator inclusions has been the development of a procedure using two forward steps instead of the customary proximal steps for operators that are Lipschitz continuous. This paper shows that the Lipschitz assumption is unnecessary when the forward steps are performed in finite-dimensional spaces: a backtracking linesearch yields a convergent algorithm for operators that are merely continuous with full domain.
△ Less
Submitted 17 September, 2018;
originally announced September 2018.
-
Convergence Rates for Projective Splitting
Authors:
Patrick R. Johnstone,
Jonathan Eckstein
Abstract:
Projective splitting is a family of methods for solving inclusions involving sums of maximal monotone operators. First introduced by Eckstein and Svaiter in 2008, these methods have enjoyed significant innovation in recent years, becoming one of the most flexible operator splitting frameworks available. While weak convergence of the iterates to a solution has been established, there have been few…
▽ More
Projective splitting is a family of methods for solving inclusions involving sums of maximal monotone operators. First introduced by Eckstein and Svaiter in 2008, these methods have enjoyed significant innovation in recent years, becoming one of the most flexible operator splitting frameworks available. While weak convergence of the iterates to a solution has been established, there have been few attempts to study convergence rates of projective splitting. The purpose of this paper is to do so under various assumptions. To this end, there are three main contributions. First, in the context of convex optimization, we establish an $O(1/k)$ ergodic function convergence rate. Second, for strongly monotone inclusions, strong convergence is established as well as an ergodic $O(1/\sqrt{k})$ convergence rate for the distance of the iterates to the solution. Finally, for inclusions featuring strong monotonicity and cocoercivity, linear convergence is established.
△ Less
Submitted 8 August, 2018; v1 submitted 11 June, 2018;
originally announced June 2018.
-
Projective Splitting with Forward Steps: Asynchronous and Block-Iterative Operator Splitting
Authors:
Patrick R. Johnstone,
Jonathan Eckstein
Abstract:
This work is concerned with the classical problem of finding a zero of a sum of maximal monotone operators. For the projective splitting framework recently proposed by Combettes and Eckstein, we show how to replace the fundamental subproblem calculation using a backward step with one based on two forward steps. The resulting algorithms have the same kind of coordination procedure and can be implem…
▽ More
This work is concerned with the classical problem of finding a zero of a sum of maximal monotone operators. For the projective splitting framework recently proposed by Combettes and Eckstein, we show how to replace the fundamental subproblem calculation using a backward step with one based on two forward steps. The resulting algorithms have the same kind of coordination procedure and can be implemented in the same block-iterative and highly flexible manner, but may perform backward steps on some operators and forward steps on others. Prior algorithms in the projective splitting family have used only backward steps. Forward steps can be used for any Lipschitz-continuous operators provided the stepsize is bounded by the inverse of the Lipschitz constant. If the Lipschitz constant is unknown, a simple backtracking linesearch procedure may be used. For affine operators, the stepsize can be chosen adaptively without knowledge of the Lipschitz constant and without any additional forward steps. We close the paper by empirically studying the performance of several kinds of splitting algorithms on a large-scale rare feature selection problem.
△ Less
Submitted 20 August, 2020; v1 submitted 19 March, 2018;
originally announced March 2018.
-
Surface-sampled simulations of turbulent flow at high Reynolds number
Authors:
Neil D. Sandham,
Roderick Johnstone,
Christian T. Jacobs
Abstract:
A new approach to turbulence simulation, based on a combination of large-eddy simulation (LES) for the whole flow and an array of non-space-filling quasi-direct numerical simulations (QDNS), which sample the response of near-wall turbulence to large-scale forcing, is proposed and evaluated. The technique overcomes some of the cost limitations of turbulence simulation, since the main flow is treate…
▽ More
A new approach to turbulence simulation, based on a combination of large-eddy simulation (LES) for the whole flow and an array of non-space-filling quasi-direct numerical simulations (QDNS), which sample the response of near-wall turbulence to large-scale forcing, is proposed and evaluated. The technique overcomes some of the cost limitations of turbulence simulation, since the main flow is treated with a coarse-grid LES, with the equivalent of wall functions supplied by the near-wall sampled QDNS. Two cases are tested, at friction Reynolds number Re$_τ$=4200 and 20,000. The total grid node count for the first case is less than half a million and less than two million for the second case, with the calculations only requiring a desktop computer. A good agreement with published DNS is found at Re$_τ$=4200, both in terms of the mean velocity profile and the streamwise velocity fluctuation statistics, which correctly show a substantial increase in near-wall turbulence levels due to a modulation of near-wall streaks by large-scale structures. The trend continues at Re$_τ$=20,000, in agreement with experiment, which represents one of the major achievements of the new approach. A number of detailed aspects of the model, including numerical resolution, LES-QDNS coupling strategy and sub-grid model are explored. A low level of grid sensitivity is demonstrated for both the QDNS and LES aspects. Since the method does not assume a law of the wall, it can in principle be applied to flows that are out of equilibrium.
△ Less
Submitted 26 April, 2017;
originally announced April 2017.
-
Faster Subgradient Methods for Functions with Hölderian Growth
Authors:
Patrick R. Johnstone,
Pierre Moulin
Abstract:
The purpose of this manuscript is to derive new convergence results for several subgradient methods applied to minimizing nonsmooth convex functions with Hölderian growth. The growth condition is satisfied in many applications and includes functions with quadratic growth and weakly sharp minima as special cases. To this end there are three main contributions. First, for a constant and sufficiently…
▽ More
The purpose of this manuscript is to derive new convergence results for several subgradient methods applied to minimizing nonsmooth convex functions with Hölderian growth. The growth condition is satisfied in many applications and includes functions with quadratic growth and weakly sharp minima as special cases. To this end there are three main contributions. First, for a constant and sufficiently small stepsize, we show that the subgradient method achieves linear convergence up to a certain region including the optimal set, with error of the order of the stepsize. Second, if appropriate problem parameters are known, we derive a decaying stepsize which obtains a much faster convergence rate than is suggested by the classical $O(1/\sqrt{k})$ result for the subgradient method. Thirdly we develop a novel "descending stairs" stepsize which obtains this faster convergence rate and also obtains linear convergence for the special case of weakly sharp functions. We also develop an adaptive variant of the "descending stairs" stepsize which achieves the same convergence rate without requiring an error bound constant which is difficult to estimate in practice.
△ Less
Submitted 30 April, 2018; v1 submitted 1 April, 2017;
originally announced April 2017.
-
Local and Global Convergence of a General Inertial Proximal Splitting Scheme
Authors:
Patrick R. Johnstone,
Pierre Moulin
Abstract:
This paper is concerned with convex composite minimization problems in a Hilbert space. In these problems, the objective is the sum of two closed, proper, and convex functions where one is smooth and the other admits a computationally inexpensive proximal operator. We analyze a general family of inertial proximal splitting algorithms (GIPSA) for solving such problems. We establish finiteness of th…
▽ More
This paper is concerned with convex composite minimization problems in a Hilbert space. In these problems, the objective is the sum of two closed, proper, and convex functions where one is smooth and the other admits a computationally inexpensive proximal operator. We analyze a general family of inertial proximal splitting algorithms (GIPSA) for solving such problems. We establish finiteness of the sum of squared increments of the iterates and optimality of the accumulation points. Weak convergence of the entire sequence then follows if the minimum is attained. Our analysis unifies and extends several previous results.
We then focus on $\ell_1$-regularized optimization, which is the ubiquitous special case where the nonsmooth term is the $\ell_1$-norm. For certain parameter choices, GIPSA is amenable to a local analysis for this problem. For these choices we show that GIPSA achieves finite "active manifold identification", i.e. convergence in a finite number of iterations to the optimal support and sign, after which GIPSA reduces to minimizing a local smooth function. Local linear convergence then holds under certain conditions. We determine the rate in terms of the inertia, stepsize, and local curvature. Our local analysis is applicable to certain recent variants of the Fast Iterative Shrinkage-Thresholding Algorithm (FISTA), for which we establish active manifold identification and local linear convergence. Our analysis motivates the use of a momentum restart scheme in these FISTA variants to obtain the optimal local linear convergence rate.
△ Less
Submitted 8 February, 2016;
originally announced February 2016.
