-
Robust A-Optimal Experimental Design for Bayesian Inverse Problems
Authors:
Ahmed Attia,
Sven Leyffer,
Todd Munson
Abstract:
Optimal design of experiments for Bayesian inverse problems has recently gained wide popularity and attracted much attention, especially in the computational science and Bayesian inversion communities. An optimal design maximizes a predefined utility function that is formulated in terms of the elements of an inverse problem, an example being optimal sensor placement for parameter identification. T…
▽ More
Optimal design of experiments for Bayesian inverse problems has recently gained wide popularity and attracted much attention, especially in the computational science and Bayesian inversion communities. An optimal design maximizes a predefined utility function that is formulated in terms of the elements of an inverse problem, an example being optimal sensor placement for parameter identification. The state-of-the-art algorithmic approaches following this simple formulation generally overlook misspecification of the elements of the inverse problem, such as the prior or the measurement uncertainties. This work presents an efficient algorithmic approach for designing optimal experimental design schemes for Bayesian inverse problems such that the optimal design is robust to misspecification of elements of the inverse problem. Specifically, we consider a worst-case scenario approach for the uncertain or misspecified parameters, formulate robust objectives, and propose an algorithmic approach for optimizing such objectives. Both relaxation and stochastic solution approaches are discussed with detailed analysis and insight into the interpretation of the problem and the proposed algorithmic approach. Extensive numerical experiments to validate and analyze the proposed approach are carried out for sensor placement in a parameter identification problem.
△ Less
Submitted 5 May, 2023;
originally announced May 2023.
-
Compact representations of structured BFGS matrices
Authors:
Johannes J. Brust,
Zichao,
Di,
Sven Leyffer,
Cosmin G. Petra
Abstract:
For general large-scale optimization problems compact representations exist in which recursive quasi-Newton update formulas are represented as compact matrix factorizations. For problems in which the objective function contains additional structure, so-called structured quasi-Newton methods exploit available second-derivative information and approximate unavailable second derivatives. This article…
▽ More
For general large-scale optimization problems compact representations exist in which recursive quasi-Newton update formulas are represented as compact matrix factorizations. For problems in which the objective function contains additional structure, so-called structured quasi-Newton methods exploit available second-derivative information and approximate unavailable second derivatives. This article develops the compact representations of two structured Broyden-Fletcher-Goldfarb-Shanno update formulas. The compact representations enable efficient limited memory and initialization strategies. Two limited memory line search algorithms are described and tested on a collection of problems, including a real world large scale imaging application.
△ Less
Submitted 29 July, 2022;
originally announced August 2022.
-
Modeling Design and Control Problems Involving Neural Network Surrogates
Authors:
Dominic Yang,
Prasanna Balaprakash,
Sven Leyffer
Abstract:
We consider nonlinear optimization problems that involve surrogate models represented by neural networks. We demonstrate first how to directly embed neural network evaluation into optimization models, highlight a difficulty with this approach that can prevent convergence, and then characterize stationarity of such models. We then present two alternative formulations of these problems in the specif…
▽ More
We consider nonlinear optimization problems that involve surrogate models represented by neural networks. We demonstrate first how to directly embed neural network evaluation into optimization models, highlight a difficulty with this approach that can prevent convergence, and then characterize stationarity of such models. We then present two alternative formulations of these problems in the specific case of feedforward neural networks with ReLU activation: as a mixed-integer optimization problem and as a mathematical program with complementarity constraints. For the latter formulation we prove that stationarity at a point for this problem corresponds to stationarity of the embedded formulation. Each of these formulations may be solved with state-of-the-art optimization methods, and we show how to obtain good initial feasible solutions for these methods. We compare our formulations on three practical applications arising in the design and control of combustion engines, in the generation of adversarial attacks on classifier networks, and in the determination of optimal flows in an oil well network.
△ Less
Submitted 19 November, 2021;
originally announced November 2021.
-
Learning Symbolic Expressions: Mixed-Integer Formulations, Cuts, and Heuristics
Authors:
Jongeun Kim,
Sven Leyffer,
Prasanna Balaprakash
Abstract:
In this paper we consider the problem of learning a regression function without assuming its functional form. This problem is referred to as symbolic regression. An expression tree is typically used to represent a solution function, which is determined by assigning operators and operands to the nodes. The symbolic regression problem can be formulated as a nonconvex mixed-integer nonlinear program…
▽ More
In this paper we consider the problem of learning a regression function without assuming its functional form. This problem is referred to as symbolic regression. An expression tree is typically used to represent a solution function, which is determined by assigning operators and operands to the nodes. The symbolic regression problem can be formulated as a nonconvex mixed-integer nonlinear program (MINLP), where binary variables are used to assign operators and nonlinear expressions are used to propagate data values through nonlinear operators such as square, square root, and exponential. We extend this formulation by adding new cuts that improve the solution of this challenging MINLP. We also propose a heuristic that iteratively builds an expression tree by solving a restricted MINLP. We perform computational experiments and compare our approach with a mixed-integer program-based method and a neural-network-based method from the literature.
△ Less
Submitted 24 February, 2021; v1 submitted 16 February, 2021;
originally announced February 2021.
-
Stochastic Learning Approach to Binary Optimization for Optimal Design of Experiments
Authors:
Ahmed Attia,
Sven Leyffer,
Todd Munson
Abstract:
We present a novel stochastic approach to binary optimization for optimal experimental design (OED) for Bayesian inverse problems governed by mathematical models such as partial differential equations. The OED utility function, namely, the regularized optimality criterion, is cast into a stochastic objective function in the form of an expectation over a multivariate Bernoulli distribution. The pro…
▽ More
We present a novel stochastic approach to binary optimization for optimal experimental design (OED) for Bayesian inverse problems governed by mathematical models such as partial differential equations. The OED utility function, namely, the regularized optimality criterion, is cast into a stochastic objective function in the form of an expectation over a multivariate Bernoulli distribution. The probabilistic objective is then solved by using a stochastic optimization routine to find an optimal observational policy. The proposed approach is analyzed from an optimization perspective and also from a machine learning perspective with correspondence to policy gradient reinforcement learning. The approach is demonstrated numerically by using an idealized two-dimensional Bayesian linear inverse problem, and validated by extensive numerical experiments carried out for sensor placement in a parameter identification setup.
△ Less
Submitted 14 January, 2021;
originally announced January 2021.
-
Doing Moore with Less -- Leapfrogging Moore's Law with Inexactness for Supercomputing
Authors:
Sven Leyffer,
Stefan M. Wild,
Mike Fagan,
Marc Snir,
Krishna Palem,
Kazutomo Yoshii,
Hal Finkel
Abstract:
Energy and power consumption are major limitations to continued scaling of computing systems. Inexactness, where the quality of the solution can be traded for energy savings, has been proposed as an approach to overcoming those limitations. In the past, however, inexactness necessitated the need for highly customized or specialized hardware. The current evolution of commercial off-the-shelf(COTS)…
▽ More
Energy and power consumption are major limitations to continued scaling of computing systems. Inexactness, where the quality of the solution can be traded for energy savings, has been proposed as an approach to overcoming those limitations. In the past, however, inexactness necessitated the need for highly customized or specialized hardware. The current evolution of commercial off-the-shelf(COTS) processors facilitates the use of lower-precision arithmetic in ways that reduce energy consumption. We study these new opportunities in this paper, using the example of an inexact Newton algorithm for solving nonlinear equations. Moreover, we have begun develo** a set of techniques we call reinvestment that, paradoxically, use reduced precision to improve the quality of the computed result: They do so by reinvesting the energy saved by reduced precision.
△ Less
Submitted 12 October, 2016; v1 submitted 8 October, 2016;
originally announced October 2016.
-
Fast Response to Infection Spread and Cyber Attacks on Large-Scale Networks
Authors:
Sven Leyffer,
Ilya Safro
Abstract:
We present a strategy for designing fast methods of response to cyber attacks and infection spread on complex weighted networks. In these networks, nodes can be interpreted as primitive elements of the system, and weighted edges reflect the strength of interaction among these elements. The proposed strategy belongs to the family of multiscale methods whose goal is to approximate the system at mult…
▽ More
We present a strategy for designing fast methods of response to cyber attacks and infection spread on complex weighted networks. In these networks, nodes can be interpreted as primitive elements of the system, and weighted edges reflect the strength of interaction among these elements. The proposed strategy belongs to the family of multiscale methods whose goal is to approximate the system at multiple scales of coarseness and to obtain a solution of microscopic scale by combining the information from coarse scales. In recent years these methods have demonstrated their potential for solving optimization and analysis problems on large-scale networks. We consider an optimization problem that is based on the SIS epidemiological model. The objective is to detect the network nodes that have to be immunized in order to keep a low level of infection in the system.
△ Less
Submitted 30 June, 2012;
originally announced July 2012.
