-
BOIS: Bayesian Optimization of Interconnected Systems
Authors:
Leonardo D. González,
Victor M. Zavala
Abstract:
Bayesian optimization (BO) has proven to be an effective paradigm for the global optimization of expensive-to-sample systems. One of the main advantages of BO is its use of Gaussian processes (GPs) to characterize model uncertainty which can be leveraged to guide the learning and search process. However, BO typically treats systems as black-boxes and this limits the ability to exploit structural k…
▽ More
Bayesian optimization (BO) has proven to be an effective paradigm for the global optimization of expensive-to-sample systems. One of the main advantages of BO is its use of Gaussian processes (GPs) to characterize model uncertainty which can be leveraged to guide the learning and search process. However, BO typically treats systems as black-boxes and this limits the ability to exploit structural knowledge (e.g., physics and sparse interconnections). Composite functions of the form $f(x, y(x))$, wherein GP modeling is shifted from the performance function $f$ to an intermediate function $y$, offer an avenue for exploiting structural knowledge. However, the use of composite functions in a BO framework is complicated by the need to generate a probability density for $f$ from the Gaussian density of $y$ calculated by the GP (e.g., when $f$ is nonlinear it is not possible to obtain a closed-form expression). Previous work has handled this issue using sampling techniques; these are easy to implement and flexible but are computationally intensive. In this work, we introduce a new paradigm which allows for the efficient use of composite functions in BO; this uses adaptive linearizations of $f$ to obtain closed-form expressions for the statistical moments of the composite function. We show that this simple approach (which we call BOIS) enables the exploitation of structural knowledge, such as that arising in interconnected systems as well as systems that embed multiple GP models and combinations of physics and GP models. Using a chemical process optimization case study, we benchmark the effectiveness of BOIS against standard BO and sampling approaches. Our results indicate that BOIS achieves performance gains and accurately captures the statistics of composite functions.
△ Less
Submitted 28 November, 2023; v1 submitted 19 November, 2023;
originally announced November 2023.
-
New Paradigms for Exploiting Parallel Experiments in Bayesian Optimization
Authors:
Leonardo D. González,
Victor M. Zavala
Abstract:
Bayesian optimization (BO) is one of the most effective methods for closed-loop experimental design and black-box optimization. However, a key limitation of BO is that it is an inherently sequential algorithm (one experiment is proposed per round) and thus cannot directly exploit high-throughput (parallel) experiments. Diverse modifications to the BO framework have been proposed in the literature…
▽ More
Bayesian optimization (BO) is one of the most effective methods for closed-loop experimental design and black-box optimization. However, a key limitation of BO is that it is an inherently sequential algorithm (one experiment is proposed per round) and thus cannot directly exploit high-throughput (parallel) experiments. Diverse modifications to the BO framework have been proposed in the literature to enable exploitation of parallel experiments but such approaches are limited in the degree of parallelization that they can achieve and can lead to redundant experiments (thus wasting resources and potentially compromising performance). In this work, we present new parallel BO paradigms that exploit the structure of the system to partition the design space. Specifically, we propose an approach that partitions the design space by following the level sets of the performance function and an approach that exploits partially-separable structures of the performance function found. We conduct extensive numerical experiments using a reactor case study to benchmark the effectiveness of these approaches against a variety of state-of-the-art parallel algorithms reported in the literature. Our computational results show that our approaches significantly reduce the required search time and increase the probability of finding a global (rather than local) solution.
△ Less
Submitted 9 December, 2022; v1 submitted 3 October, 2022;
originally announced October 2022.
-
Data Analysis using Riemannian Geometry and Applications to Chemical Engineering
Authors:
Alexander Smith,
Benjamin Laubach,
Ivan Castillo,
Victor M. Zavala
Abstract:
We explore the use of tools from Riemannian geometry for the analysis of symmetric positive definite matrices (SPD). An SPD matrix is a versatile data representation that is commonly used in chemical engineering (e.g., covariance/correlation/Hessian matrices and images) and powerful techniques are available for its analysis (e.g., principal component analysis). A key observation that motivates thi…
▽ More
We explore the use of tools from Riemannian geometry for the analysis of symmetric positive definite matrices (SPD). An SPD matrix is a versatile data representation that is commonly used in chemical engineering (e.g., covariance/correlation/Hessian matrices and images) and powerful techniques are available for its analysis (e.g., principal component analysis). A key observation that motivates this work is that SPD matrices live on a Riemannian manifold and that implementing techniques that exploit this basic property can yield significant benefits in data-centric tasks such classification and dimensionality reduction. We demonstrate this via a couple of case studies that conduct anomaly detection in the context of process monitoring and image analysis.
△ Less
Submitted 23 March, 2022;
originally announced March 2022.
-
Unifying Theorems for Subspace Identification and Dynamic Mode Decomposition
Authors:
Sungho Shin,
Qiugang Lu,
Victor M. Zavala
Abstract:
This paper presents unifying results for subspace identification (SID) and dynamic mode decomposition (DMD) for autonomous dynamical systems. We observe that SID seeks to solve an optimization problem to estimate an extended observability matrix and a state sequence that minimizes the prediction error for the state-space model. Moreover, we observe that DMD seeks to solve a rank-constrained matrix…
▽ More
This paper presents unifying results for subspace identification (SID) and dynamic mode decomposition (DMD) for autonomous dynamical systems. We observe that SID seeks to solve an optimization problem to estimate an extended observability matrix and a state sequence that minimizes the prediction error for the state-space model. Moreover, we observe that DMD seeks to solve a rank-constrained matrix regression problem that minimizes the prediction error of an extended autoregressive model. We prove that existence conditions for perfect (error-free) state-space and low-rank extended autoregressive models are equivalent and that the SID and DMD optimization problems are equivalent. We exploit these results to propose a SID-DMD algorithm that delivers a provably optimal model and that is easy to implement. We demonstrate our developments using a case study that aims to build dynamical models directly from video data.
△ Less
Submitted 16 March, 2020;
originally announced March 2020.
-
On the Convergence of the Dynamic Inner PCA Algorithm
Authors:
Sungho Shin,
Alex D. Smith,
S. Joe Qin,
Victor M. Zavala
Abstract:
Dynamic inner principal component analysis (DiPCA) is a powerful method for the analysis of time-dependent multivariate data. DiPCA extracts dynamic latent variables that capture the most dominant temporal trends by solving a large-scale, dense, and nonconvex nonlinear program (NLP). A scalable decomposition algorithm has been recently proposed in the literature to solve these challenging NLPs. Th…
▽ More
Dynamic inner principal component analysis (DiPCA) is a powerful method for the analysis of time-dependent multivariate data. DiPCA extracts dynamic latent variables that capture the most dominant temporal trends by solving a large-scale, dense, and nonconvex nonlinear program (NLP). A scalable decomposition algorithm has been recently proposed in the literature to solve these challenging NLPs. The decomposition algorithm performs well in practice but its convergence properties are not well understood. In this work, we show that this algorithm is a specialized variant of a coordinate maximization algorithm. This observation allows us to explain why the decomposition algorithm might work (or not) in practice and can guide improvements. We compare the performance of the decomposition strategies with that of the off-the-shelf solver Ipopt. The results show that decomposition is more scalable and, surprisingly, delivers higher quality solutions.
△ Less
Submitted 12 March, 2020;
originally announced March 2020.
