-
Multi-objective hyperparameter optimization with performance uncertainty
Authors:
Alejandro Morales-Hernández,
Inneke Van Nieuwenhuyse,
Gonzalo Nápoles
Abstract:
The performance of any Machine Learning (ML) algorithm is impacted by the choice of its hyperparameters. As training and evaluating a ML algorithm is usually expensive, the hyperparameter optimization (HPO) method needs to be computationally efficient to be useful in practice. Most of the existing approaches on multi-objective HPO use evolutionary strategies and metamodel-based optimization. Howev…
▽ More
The performance of any Machine Learning (ML) algorithm is impacted by the choice of its hyperparameters. As training and evaluating a ML algorithm is usually expensive, the hyperparameter optimization (HPO) method needs to be computationally efficient to be useful in practice. Most of the existing approaches on multi-objective HPO use evolutionary strategies and metamodel-based optimization. However, few methods have been developed to account for uncertainty in the performance measurements. This paper presents results on multi-objective hyperparameter optimization with uncertainty on the evaluation of ML algorithms. We combine the sampling strategy of Tree-structured Parzen Estimators (TPE) with the metamodel obtained after training a Gaussian Process Regression (GPR) with heterogeneous noise. Experimental results on three analytical test functions and three ML problems show the improvement over multi-objective TPE and GPR, achieved with respect to the hypervolume indicator.
△ Less
Submitted 9 September, 2022;
originally announced September 2022.
-
Bi-objective Ranking and Selection Using Stochastic Kriging
Authors:
Sebastian Rojas Gonzalez,
Juergen Branke,
Inneke van Nieuwenhuyse
Abstract:
We consider bi-objective ranking and selection problems, where the goal is to correctly identify the Pareto optimal solutions among a finite set of candidates for which the two objective outcomes have been observed with uncertainty (e.g., after running a multiobjective stochastic simulation optimization procedure). When identifying these solutions, the noise perturbing the observed performance may…
▽ More
We consider bi-objective ranking and selection problems, where the goal is to correctly identify the Pareto optimal solutions among a finite set of candidates for which the two objective outcomes have been observed with uncertainty (e.g., after running a multiobjective stochastic simulation optimization procedure). When identifying these solutions, the noise perturbing the observed performance may lead to two types of errors: solutions that are truly Pareto-optimal can be wrongly considered dominated, and solutions that are truly dominated can be wrongly considered Pareto-optimal. We propose a novel Bayesian bi-objective ranking and selection method that sequentially allocates extra samples to competitive solutions, in view of reducing the misclassification errors when identifying the solutions with the best expected performance. The approach uses stochastic kriging to build reliable predictive distributions of the objective outcomes, and exploits this information to decide how to resample. Experimental results show that the proposed method outperforms the standard allocation method, as well as a well-known the state-of-the-art algorithm. Moreover, we show that the other competing algorithms also benefit from the use of stochastic kriging information; yet, the proposed method remains superior.
△ Less
Submitted 28 March, 2024; v1 submitted 5 September, 2022;
originally announced September 2022.
-
Constrained multi-objective optimization of process design parameters in settings with scarce data: an application to adhesive bonding
Authors:
Alejandro Morales-Hernández,
Sebastian Rojas Gonzalez,
Inneke Van Nieuwenhuyse,
Ivo Couckuyt,
Jeroen Jordens,
Maarten Witters,
Bart Van Doninck
Abstract:
Adhesive joints are increasingly used in industry for a wide variety of applications because of their favorable characteristics such as high strength-to-weight ratio, design flexibility, limited stress concentrations, planar force transfer, good damage tolerance, and fatigue resistance. Finding the optimal process parameters for an adhesive bonding process is challenging: the optimization is inher…
▽ More
Adhesive joints are increasingly used in industry for a wide variety of applications because of their favorable characteristics such as high strength-to-weight ratio, design flexibility, limited stress concentrations, planar force transfer, good damage tolerance, and fatigue resistance. Finding the optimal process parameters for an adhesive bonding process is challenging: the optimization is inherently multi-objective (aiming to maximize break strength while minimizing cost), constrained (the process should not result in any visual damage to the materials, and stress tests should not result in failures that are adhesion-related), and uncertain (testing the same process parameters several times may lead to different break strengths). Real-life physical experiments in the lab are expensive to perform. Traditional evolutionary approaches (such as genetic algorithms) are then ill-suited to solve the problem, due to the prohibitive amount of experiments required for evaluation. Although Bayesian optimization-based algorithms are preferred to solve such expensive problems, few methods consider the optimization of more than one (noisy) objective and several constraints at the same time. In this research, we successfully applied specific machine learning techniques (Gaussian Process Regression) to emulate the objective and constraint functions based on a limited amount of experimental data. The techniques are embedded in a Bayesian optimization algorithm, which succeeds in detecting Pareto-optimal process settings in a highly efficient way (i.e., requiring a limited number of physical experiments).
△ Less
Submitted 10 April, 2023; v1 submitted 16 December, 2021;
originally announced December 2021.
-
Multi-objective simulation optimization of the adhesive bonding process of materials
Authors:
Alejandro Morales-Hernández,
Inneke Van Nieuwenhuyse,
Sebastian Rojas Gonzalez,
Jeroen Jordens,
Maarten Witters,
Bart Van Doninck
Abstract:
Automotive companies are increasingly looking for ways to make their products lighter, using novel materials and novel bonding processes to join these materials together. Finding the optimal process parameters for such adhesive bonding process is challenging. In this research, we successfully applied Bayesian optimization using Gaussian Process Regression and Logistic Regression, to efficiently (i…
▽ More
Automotive companies are increasingly looking for ways to make their products lighter, using novel materials and novel bonding processes to join these materials together. Finding the optimal process parameters for such adhesive bonding process is challenging. In this research, we successfully applied Bayesian optimization using Gaussian Process Regression and Logistic Regression, to efficiently (i.e., requiring few experiments) guide the design of experiments to the Pareto-optimal process parameter settings.
△ Less
Submitted 9 December, 2021;
originally announced December 2021.
-
A survey on multi-objective hyperparameter optimization algorithms for Machine Learning
Authors:
Alejandro Morales-Hernández,
Inneke Van Nieuwenhuyse,
Sebastian Rojas Gonzalez
Abstract:
Hyperparameter optimization (HPO) is a necessary step to ensure the best possible performance of Machine Learning (ML) algorithms. Several methods have been developed to perform HPO; most of these are focused on optimizing one performance measure (usually an error-based measure), and the literature on such single-objective HPO problems is vast. Recently, though, algorithms have appeared that focus…
▽ More
Hyperparameter optimization (HPO) is a necessary step to ensure the best possible performance of Machine Learning (ML) algorithms. Several methods have been developed to perform HPO; most of these are focused on optimizing one performance measure (usually an error-based measure), and the literature on such single-objective HPO problems is vast. Recently, though, algorithms have appeared that focus on optimizing multiple conflicting objectives simultaneously. This article presents a systematic survey of the literature published between 2014 and 2020 on multi-objective HPO algorithms, distinguishing between metaheuristic-based algorithms, metamodel-based algorithms, and approaches using a mixture of both. We also discuss the quality metrics used to compare multi-objective HPO procedures and present future research directions.
△ Less
Submitted 15 November, 2022; v1 submitted 23 November, 2021;
originally announced November 2021.
