MALIBO: Meta-learning for Likelihood-free Bayesian Optimization

Various methods have been proposed to improve the data-efficiency of BO through meta-learning and have shown effectiveness in diverse applications (Andrychowicz et al., 2016).

One line of work focuses on the initialization of the optimization (initial design) by reducing the search space (Perrone et al., 2019; Li et al., 2022) or reusing promising configurations from similar tasks, where task similarity can be determined using hand-crafted meta-features (Feurer et al., 2014) or learned through neural networks (NNs) (Kim et al., 2017). One can also estimate the utility of a configuration using heuristics (Wistuba et al., 2015) or learning-based techniques (Volpp et al., 2020; Hsieh et al., 2021; Maraval et al., 2023). Transfer learning is also employed to modify the surrogate model using multi-task GPs (Swersky et al., 2013; Tighineanu et al., 2022, 2024), additive GP models (Golovin et al., 2017), weighted combinations of independent GPs (Wistuba et al., 2018; Feurer et al., 2022), shared feature representation learned across tasks (Perrone et al., 2018; Wistuba & Grabocka, 2021; Khazi et al., 2023) or pre-training surrogate models on large amount of diverse data (Chen et al., 2022; Müller et al., 2023).

Several methods simultaneously learn the initial design and modify the surrogate model. BOHAMIANN (Springenberg et al., 2016) adopts a Bayesian NN as the surrogate model, which is computationally expensive and hard to train. ABLR (Perrone et al., 2018) and BANNER (Berkenkamp et al., 2021) combine a NN to learn a shared feature representation across tasks and task-specific Bayesian linear regression (BLR) layers for scalable adaptation. While ABLR adapts to new tasks by fine-tuning the whole network, BANNER meta-learns a task-independent mean function and only fine-tunes the BLR layer during optimization. However, both methods are sensitive to changes in scale and noise across tasks. To address this, GC3P (Salinas et al., 2020) transforms the observed values via quantile transformation and fits a NN across all related tasks. Although GC3P warm-starts the optimization by using a NN to predict the mean for a GP on the target task, its scalability is limited by its GP surrogate.

Bayesian optimization does not require an explicit model of the likelihood of the observed values (Garnett, 2022) and can be done by directly approximating the acquisition function. The tree-structured Parzen estimator (TPE) (Bergstra et al., 2011) phrases BO as a density ratio estimation problem (Sugiyama et al., 2012) and uses the density ratio over ‘good’ and ‘bad’ configurations as an acquisition function. BORE (Tiao et al., 2021) estimates the density ratio through class probability estimation (Qin, 1998), which is equivalent to modeling the acquisition function with a binary classifier and can be parallellized (Oliveira et al., 2022). By transforming the acquisition function into a variational problem, likelihood-free Bayesian optimization (LFBO) (Song et al., 2022) uses the probabilistic predictions of a classifier to directly approximate the acquisition function. In this paper, we leverage the flexibility of likelihood-free acquisition functions and combine it with a meta-learning model to obtain a sample-efficient, scalable, and robust BO method.

BO aims to minimize a target black-box function $f:\mathcal{X}\rightarrow\mathbb{R}$ over $\mathbf{x}\in\mathcal{X}$. In the case of meta-learning, $T$ related black-box functions $\{f^{t}(\cdot)\}^{T}_{t=1}$ are given in advance, each with the same domain $\mathcal{X}$. The optimization is warm-started with previous evaluations on the related functions, $\mathcal{D}^{\text{meta}}=\{\mathcal{D}^{t}\}^{T}_{t=1}$ with $\mathcal{D}^{t}=\{(\mathbf{x}^{t}_{i},y^{t}_{i})\}^{N^{t}}_{i=1}$, where $y^{t}_{i}=f^{t}(\mathbf{x}^{t}_{i})+\epsilon^{t}$ are evaluations corrupted by noise $\epsilon^{t}$ and $N^{t}=|\mathcal{D}^{t}|$ is the number of observations collected from task $f^{t}$. Given a new task at step $N+1$ , BO proposes $\mathbf{x}_{N+1}$ and obtains a noisy observation from the target function $y_{N+1}=f(\mathbf{x}_{N+1})+\epsilon$, with $\epsilon$ drawn i.i.d. from some distribution $p_{\epsilon}$. To obtain the proposal $\mathbf{x}_{N+1}$, a probabilistic surrogate model is first fitted on $N$ previous observations on the target function $\mathcal{D}_{N}=\{(\mathbf{x}_{i},y_{i})\}^{N}_{i=1}$ and the related functions $\mathcal{D}^{\text{meta}}$. For simplicity, we denote $\mathcal{D}:=\mathcal{D}_{N}\cup\mathcal{D}^{\text{meta}}$. The resulting model is used to compute an acquisition function, such as, the expected utility of a given query $\mathbf{x}$,

where $U(y;\tau)$ is a chosen utility function with a threshold $\tau$ that decides the utility of observing $y$ at $\mathbf{x}$ and controls the exploration-exploitation trade-off. The predictive distribution $p(y\mid\mathbf{x},\mathcal{D})$ is given by the probabilistic surrogate model and the maximizer $\mathbf{x}_{N+1}=\operatorname*{arg\,max}_{\mathbf{x}\in\mathcal{X}}\alpha(\mathbf{x};\mathcal{D},\tau)$ is the proposed candidate. Acquisition functions that take the form of Equation 1 include Expected Improvement (EI) (Močkus, 1975) and Probability of Improvement (PI) (Kushner, 1964). Many others exist, such as UCB (Srinivas et al., 2010), Entropy Search (Hennig & Schuler, 2012; Hernández-Lobato et al., 2014; Wang & Jegelka, 2017) and Knowledge Gradient (Frazier et al., 2009).

Likelihood-free acquisition functions model the utility of a query without explicitly modeling the predictive distribution. For example, tree-structured Parzen estimators (TPE) (Bergstra et al., 2011) dismiss the surrogate for the outcomes and instead model two densities that split the observations w.r.t. a threshold $\tau$, namely $\ell(\mathbf{x})=p(\mathbf{x}\mid y\leq\tau,\mathcal{D}_{N})$ and $g(\mathbf{x})=p(\mathbf{x}\mid y>\tau,\mathcal{D}_{N})$ for the promising and non-promising data distributions, respectively. The threshold $\tau$ relates to the $\gamma$-th quantile of the observed outcomes via $\gamma=\Phi(\tau):=p(y\leq\tau)$. In fact, the resulting density ratio (DR) $\alpha^{\text{DR}}(\mathbf{x};\mathcal{D}_{N},\tau)=\ell(\mathbf{x})/g(\mathbf{x})$ is shown to have the same maximum as PI (Song et al., 2022; Garnett, 2022).

BORE (Tiao et al., 2021) improves several aspects of TPE by directly estimating the density ratio instead of solving the more challenging problem of modeling two independent densities as an intermediate step. It rephrases the density ratio estimation as a binary classification problem where all observations within the same class have the same importance. Specifically, they show $\alpha^{\text{DR}}(\mathbf{x};\mathcal{D}_{N},\tau)\propto C_{\bm{\theta}}(\mathbf{x})=p(k=1\mid\mathbf{x},D_{N},\tau)$, where $k=\mathbbm{1}(y\leq\tau)$ represents the binary class labels for classification and the classifier $C_{\bm{\theta}}$ has learnable parameters $\bm{\theta}$.

Likelihood-free Bayesian optimization (LFBO) (Song et al., 2022) directly learns an acquisition function in the form of Equation 1 through a classifier. By rephrasing the integral as a variational problem, LFBO involves solving a weighted classification problem with noisy labels for the class $k=1$, where the weights correspond to utilities. It is shown that the EI acquisition function, where $U(y;\tau):=\max(\tau-y,0)$, can be estimated by a classifier that optimizes the following objective:

The resulting classifier splits promising and non-promising configurations with probabilistic predictions that can be interpreted as the utility of queries, leading to scale-invariant models without noise assumptions and allowing the application of any classification methods (Song et al., 2022). Further details of the algorithms are provided in Appendix A.

MALIBO uses a structured neural network that combines a meta-learned, task-agnostic model with a task-specific layer. We show an overview in Figure 2 and provide details for the choices below. Following previous works (Perrone et al., 2018; Berkenkamp et al., 2021), our meta-learning model uses a deterministic, task-agnostic model to map the input into features $\bm{\Phi}=\phi(\mathbf{x})$, where $\phi:\mathcal{X}\rightarrow\mathbb{R}^{d}$ is a learnable feature map** shared across all tasks and $d$ is the predefined dimensionality of the feature space. We use a residual feedforward network (ResFFN) for learning $\phi$, which has been shown to be robust to network hyperparameters and generalizes well to different problems (Huang et al., 2020). To enable our model to provide good initial proposals, we introduce a task-agnostic mean prediction layer $m:\mathbb{R}^{d}\rightarrow\mathbb{R}$ that learns the promising areas from the related tasks. We refer to the combined task-agnostic components $m$ and $\phi$ as $g_{\bm{\omega}}$ (shown in blue), which is parameterized by $\bm{\omega}$. To allow adaptation on each task $t$, we use a task prediction layer $h_{\mathbf{z}_{t}}:\mathbb{R}^{d}\rightarrow\mathbb{R}$, which is parameterized by layer weights $\mathbf{z}_{t}\in\mathcal{Z}\subseteq\mathbb{R}^{d}$. Since each $\mathbf{z}_{t}$ embeds in a low dimensional latent space $\mathcal{Z}$ and is a unique vector for each task, we refer to $\mathbf{z}_{t}$ as the task embedding. We will train our model such that the $\{\mathbf{z}_{t}\}_{t=1}^{T}$ follow a known distribution $p(\mathcal{Z})$ and discuss below how to use this as a prior for target task adaptation. Lastly, in order to obtain classification outputs as in LFBO, we apply the sigmoid function to produce probabilistic class predictions $p(k=1\mid\mathbf{x})$. The prediction for an observation in task $t$ is then given by $C(\mathbf{x}_{t})=\sigma(m(\phi(\mathbf{x}))+h_{\mathbf{z}_{t}}(\phi(\mathbf{x})))=\sigma(m(\bm{\Phi})+\mathbf{z}_{t}^{\mathsf{T}}\bm{\Phi})$.

Directly optimizing $\mathcal{L}^{\text{{LFBO}}}$ to meta-learn our model would lead to task embeddings that do not conform to any particular prior task distribution $p(\mathcal{Z})$, and thus render task adaptation difficult and unreliable (Finn et al., 2018). Therefore, we regularize the task embeddings $\{\mathbf{z}_{t}\}_{t=1}^{T}$ during training to enable Bayesian inference. In addition, such regularization can also avoid overfitting in the task space $\mathcal{Z}$ and improves the generalization performance of our model. Specifically, we assume the prior of the task embeddings to be a multivariate normal (MVN), $p(\mathcal{Z})=\mathcal{N}(\mathbf{0},\mathbf{I})$ and apply a regularization term to bring the empirical distribution of the $\{\mathbf{z}_{t}\}_{t=1}^{T}$ close to the prior distribution. The loss used for training on the meta-data reads:

where the first term is the loss function from LFBO as in Equation 2, weighting the observations in the meta-data with improvements and the second term $\mathcal{R}$ is the regularization term weighted by $\lambda$. We regularize the empirical distribution of $\{\mathbf{z}_{t}\}_{t=1}^{T}$ to match the Gaussian prior in a tractable way (Saseendran et al., 2021):

where the first term matches the marginal cumulative distribution functions (CDFs) similar to a Kolmogrov-Smirnov (KS) test, and the second term matches the empirical covariance of the task embeddings to the covariance of the prior. The hyperparameters $\lambda_{\text{KS}}$ and $\lambda_{\text{Cov}}$ encode the trade-off between these two terms. We denote $F$ as the empirical CDF and Cov as the empirical covariance matrix. For more details we refer to Appendix C.

We only consider a uni-modal Gaussian prior in this work, as we will show it already demonstrates strong performance against other baselines. For more complex task distributions, one could extend it with multi-modal Gaussian prior (Saseendran et al., 2021).

After meta-training, the model can adapt to new tasks by estimating an embedding $\mathbf{z}$ based on the learned feature map** function $\phi$. In principle, one could use a maximum likelihood classifier obtained by directly optimizing Equation 2 w.r.t. $\mathbf{z}$. However, such a classifier does not consider the task uncertainty and would suffer from unreliable adaptation (Finn et al., 2018) and over-exploitation (Oliveira et al., 2022). Furthermore, when a potential disparity between the distribution of the meta-data and the non-i.i.d. data collected during optimization arises, a probabilistic model would be informed via uncertainty estimation and thereby can exploit the meta-learned knowledge less. Therefore, we propose to use a Bayesian approach for task adaptation, which makes our classifier uncertainty-aware and more exploratory.

Consider the task embedding $\mathbf{z}$ for the target task follows a distribution $p(\mathbf{z}\mid\mathcal{D}_{N})$ after $N$ observations, then the predictive distribution of our model can be written as

which accounts for the epistemic uncertainty in the task embedding. Since the parameters $\bm{\omega}$ of task-agnostic model $g_{\bm{\omega}}$ are fixed after meta-training, we denote our classifier as $C(\mathbf{x})$ for simplicity.

As there is no analytical way to evaluate the integration in Equation 5, we have to resort to approximation methods, such as Laplace approximation (Bishop & Nasrabadi, 2006), variational inference (Graves, 2011), and Markov chain Monte Carlo (Homan & Gelman, 2014). We consider Laplace approximation for $p(\mathbf{z}\mid\mathcal{D}_{N})$ as a fast and scalable method, and show its competitive performance against other, more expensive alternatives in Section D.4.

Laplace’s method fits a Gaussian distribution around the maximum-a-posteriori (MAP) estimate of the distribution and matches the second order derivative at the optimum. In the first step, we obtain the MAP estimate by maximizing the posterior of our classifier $C$ parameterized by $\mathbf{z}$. To be consistent with the regularization used during meta-training, we use a standard, isotropic Gaussian prior for the weights: $p(\mathbf{z})=\mathcal{N}(\mathbf{z}\mid\mathbf{0},\mathbf{I})$. Given observations $\mathcal{D}_{N}$, the negative log posterior $p(\mathbf{z}\mid\mathcal{D}_{N})$ is proportional to

where $\hat{k}=\sigma(m(\bm{\Phi})+\mathbf{z}^{\mathsf{T}}\bm{\Phi})$ is the class probability prediction and the MAP estimate of the weights given by $\mathbf{z}_{\text{MAP}}={\arg\min}_{\mathbf{z}\in\mathcal{Z}}\mathcal{L}^{\textsc{MALIBO}}$. As a second step, we compute the negative Hessian of the log posterior

which serves as the precision matrix for the approximated posterior $q(\mathbf{z})=\mathcal{N}(\mathbf{z}\mid\mathbf{z}_{\text{MAP}},\mathbf{\Sigma}_{N})$. Therefore Equation 5 can be approximated as

Having developed a meta-learning model, we now focus on how to utilize this model to encourage exploration and ensure reliable task adaptation.

In the early phase of optimization, every meta-learning model has to reason about the target task properties based only on the limited data available, which can lead to highly biased results and over-exploitation (Finn et al., 2018). Moreover, LFBO also suffers from similar issue even without meta-learning (Song et al., 2022). Therefore, we propose to use Thompson sampling based on task uncertainty for constructing a more exploratory acquisition function, and the resulting sampled predictions is generated by

Besides stronger exploration in the early phases of optimization, Thompson sampling also enables us to extend MALIBO to parallel BO by using multiple Thompson samples of the acquisition function in parallel. It is shown that this bypasses the sequential scheme of traditional BO, without introducing the common computational burden of more sophisticated methods (Kandasamy et al., 2018). We briefly explore this strategy in Appendix F.

Operating in a meta-learned feature space enables fast task adaptation for our Bayesian classifier. However, it relies on the assumption that the meta-data is sufficient and representative for the task distribution, which does not always hold in practice. Moreover, a distribution mismatch between observations $\mathcal{D}_{N}$ and meta-data $\mathcal{D}^{\text{meta}}$ can arise when $\mathcal{D}_{N}$ is generated by an optimization process while $\mathcal{D}^{\text{meta}}$ consists of, e.g., i.i.d. samples.

We employ a residual model independent of the meta-learning model, such that, even given non-informative features, our classifier is able to regress to an optimizer that operates in the input space $\mathcal{X}$. We propose to use gradient boosting (GB) (Friedman, 2001) as a residual prediction model for classification, which consists of an ensemble of weak learners that are sequentially trained to correct the errors from the previous ones. Specifically, we replace the first weak learner by a strong learner, i.e., our meta-learned classifier. With Thompson sampling, our classifier can be written as

where each $r_{i}$ represents the $i$-th trained base-learner for the error correction from gradient boosting. In addition to robust task adaptation, this approach offers two advantages: First, gradient boosting does not require an additional weighting scheme for combining different classifiers and automatically determines the weight of the meta-learned model; Second, gradient boosting demonstrates strong performance for LFBO on various benchmarks (Song et al., 2022), which makes our classifier achieve competitive performance even when meta-learning fails, as shown in Section D.3. The resulting residual model is trained solely on the target task data and thus might overfit in the early iterations with limited data. To avoid this, we apply gradient boosting only after a few iterations of Thompson sampling exploration and train it with early stop**. Note that this does not diminish the usefulness of the residual model, because our goal is to encourage exploration in early iterations as outlined in Section 4, and gradually rely more on the knowledge from the target task. We refer to Appendix G for more implementation details.

We compare our method against multiple baselines across all problems. As methods without meta-learning, we pick random search (Bergstra & Bengio, 2012), LFBO (Song et al., 2022) and Gaussian process (GP) (Snoek et al., 2012)) for our experiments. For meta-learning BO methods, we choose ABLR (Perrone et al., 2018), BaNNER (Berkenkamp et al., 2021), RGPE (Feurer et al., 2022), GC3P (Salinas et al., 2020), FSBO (Wistuba & Grabocka, 2021), MetaBO (Volpp et al., 2020), PFN (Müller et al., 2023) and DRE (Khazi et al., 2023) as representative algorithms. Additionally, we consider a simple baseline for extending LFBO with meta-learning, called LFBO+BB, which combines LFBO with bounding-box search space pruning (Perrone et al., 2019) as a meta-learning approach. For all LFBO-based methods, including MALIBO, we set the required threshold $\gamma=1/3$ following Song et al. (2022).

In order to aggregate performances across tasks, we use normalized regret as the quantitative performance measure for AutoML problems (Wistuba et al., 2018). This is defined as $\min_{\mathbf{x}\in\mathcal{X}_{N}}(f^{t}(\mathbf{x})-f^{t}_{\text{min}})/(f^{t}_{\text{max}}-f^{t}_{\text{min}})$, where $\mathcal{X}_{N}$ denotes the set of inputs that have been selected by an optimizer up to iteration $N$, $f^{t}_{\text{min}}$ and $f^{t}_{\text{max}}$ respectively represent the minimum and the maximum objective computed across all offline evaluations available for task $t$. We report the mean normalized regret across all tasks within a benchmark as the aggregated result. For all benchmarks, we report the results by mean and standard error across 100 random runs.

We first investigate the effect of Thompson sampling and the residual prediction model when optimizing a Forrester function (Sobester et al., 2008) as a toy example. By using the meta-learned model as shown in Figure 1, MALIBO performs task adaptation on a new Forrester function for $10$ iterations. We compare the results of MALIBO against a variant without the proposed Thompson sampling and gradient boosting, which only uses the approximated posterior predictive distribution in Equation 8 by probit approximation (Bishop & Nasrabadi, 2006) for the acquisition function. As shown in Figure 3, MALIBO without Thompson sampling fails to adapt the new task with little exploration and optimizes greedily around the local optimum. This greedy optimization occurs due to the strong dependence of LFBO on a good initialization to not over-exploit. In contrast, the proposed MALIBO allows the queries to cover both possible optima by encouraging explorations. In addition, gradient boosting performs the refinement beyond the smooth meta-learned acquisition function, which can be seen in the discontinuity in the predictions. By suppressing the predicted utility in the less promising areas, gradient boosting refines the acquisition function to focus on the lower value region. We provide an extensive ablation study on the effects of different components in MALIBO and refer to Appendix D for more details.

To confirm the scalability of MALIBO, we compare the runtime between methods, specifically the time required for the algorithm to propose a new candidate. As shown in Figure 4, the introduction of latent features and the Laplace approximation only adds negligible overhead compared to LFBO, while MALIBO’s runtime increases slowly with the number of observations. In contrast, all other meta-learning methods, except for LFBO+BB, are considerably slower than MALIBO. We include more detailed experimental results in Section E.2.

We empirically evaluate our method on various real-world optimization tasks, focusing on AutoML problems, including neural architecture search (NASBench201) (Dong & Yang, 2020), hyperparameter optimization for neural networks (HPOBench) (Klein & Hutter, 2019) and machine learning algorithms (HPO-B) (Pineda-Arango et al., 2021). In NASBench201, we consider designing a neural cell with $6$ discrete parameters, totaling $15,625$ unique architectures, evaluated on CIFAR-10, CIFAR-100 (Krizhevsky et al., 2009) and ImageNet-16 (Chrabaszcz et al., 2017). The goal is to find the optimal architecture for a neural network that yields the highest validation accuracy. For HPOBench, the aim is to find the optimal hyperparameters for a two-layer feed-forward regression network on four popular UCI datasets (Dua & Graff, 2017). The search space is $9$-dimensional and the optimization objective is the validation mean squared error after training with the corresponding network configuration. In HPO-B, the focus is on optimizing the hyperparameters for different machine learning models to maximize accuracy across various tasks. This benchmark comprises about $6$ million evaluations of hyperparameters, across $16$ search spaces that correspond to different machine learning models. Each search space varies in dimensionality ranging from $2$ to $18$ and includes several tasks, which are divided into training, validation and test tasks. Compared to the extensive evaluations in HPO-B, both HPOBench and NASBench201 have significantly fewer related tasks and serve as examples of performance with limited meta-data. We provide details for benchmarks in Appendix H.

To train and evaluate the meta-learning BO methods in HPOBench and NASBench201, we conduct our experiments in a leave-one-task-out way: all meta-learning methods use one task as the target task and all others as related tasks. In this way, every task in a benchmark is picked as the target task once. To construct the meta-datasets, we randomly select $512$ configuration-objective pairs from the related tasks, considering the limitations of RGPE in handling large meta-datasets. All meta-learning methods, except MetaBO, are trained from scratch for each independent run, to account for variations due to the randomly sampled meta-data. Because of its long training time, MetaBO is trained once for each target problem on more meta-data than other methods to avoid limiting its performance with a bad subsample and we show its results only for HPOBench and NASBench201. As for HPO-B, we utilize the provided meta-train and meta-validation dataset to train the meta-learning methods and evaluate all methods on the meta-test data. While all methods optimize the target tasks from scratch in the other two benchmarks, the first five initial observations in HPO-B is fixed as random seed and therefore we only show the performances starting after the initialization. We refer to Appendix G for more experimental details.

The aggregated results for all three benchmarks are summarized in Figure 5. It is evident that MALIBO consistently achieves strong anytime performance, surpassing other methods that either exhibit poor warm-starting or experience early saturation of performance. Notably, MALIBO outperforms other methods by a large margin in HPOBench, possibly because we focus on minimizing the validation error of a regression model in this benchmark. This task poses a significant challenge for GP-based regression models, as the observation values undergo abrupt changes and have varying scales across tasks, thereby violating the smoothness and noise assumptions inherent in these models. In most benchmarks, PFN and BaNNER exhibit comparable performance to GP, except in NASBench201. GC3P performs competitively only after the Copula process is fitted and LFBO matches its final performance. LFBO+BB exhibits similar performance as MALIBO in warm-starting and converges quickly, but the search space pruning technique forbids the method to explore regions beyond the promising areas in the meta-data, making its final performance even worse than its non-meta-learning counterpart. ABLR, RGPE and FSBO perform poorly on most of the benchmarks, except for HPO-B, because their meta-learning techniques require more meta-data for effective warm-starting, making them less data-efficient than MALIBO. MetaBO shows strong warm-starting performance in HPOBench while it fails in NASBench201. This is possibly due to the higher diversity in NASBench201 compared to HPOBench, and MetaBO fails to transfer knowledge from tasks that are significantly different from the target task. The poor task adaptation ability of MetaBO is also found by other studies (Wistuba & Grabocka, 2021; Wang et al., 2022). For more experimental results, we refer to Section E.3.

To understand the impact of each component within MALIBO, we conduct a quantitative ablation study and introduce the following variants:

• •

  MALIBO (Probit): Employs a probit approximation (detailed in Appendix B) for the marginalized form of the acquisition function. This variant does not use gradient boosting.

• •

  MALIBO (TS): Utilizes only Thompson sampling, omitting the gradient boosting component.

• •

  MALIBO (RES): Excludes the mean prediction layer $m(\cdot)$.

• •

  MALIBO (MEAN): Removes the task prediction layer $h_{\mathbf{z}_{t}}(\cdot)$ and utilizes only the task-agnostic component $g_{\bm{\omega}}$.

• •

  MALIBO (RF): Substitutes gradient boosting with a random forest (RF) classifier.

• •

  MALIBO (MLP): Replaces gradient boosting with a multi-layer perceptron (MLP) classifier.

The results illustrated in Figure 6 reveal several key insights. Due to the lack of a mean prediction layer, MALIBO (RES) exhibits the poorest warm-starting performance across all benchmarks, potentially leading to worse final performance. Although the mean prediction layer improves initial performance, relying solely on it may result in over-fitting to the meta-data due to insufficient exploration capabilities. As demonstrated by the performance of MALIBO (MEAN), while it achieves the best results among all variants in NASBench201, its performance on the other two benchmarks is inferior to the proposed method. In contrast, variants that include an uncertainty-aware task prediction layer, such as MALIBO (RES) and MALIBO, perform task adaptation more reliably. Although MALIBO (TS) also encourages exploration, the absence of a residual prediction model results in a significant performance decrease when the amount of meta-data is limited, as observed in HPOBench and NASBench201. The comparison of MALIBO (MLP) and MALIBO (RF) highlights the superiority of gradient boosting over other classifiers such as random forest and MLP for the residual prediction model. For more detailed experimental results, refer to Section D.6.

We use synthetic function ensembles (Berkenkamp et al., 2021) to test the robustness against heterogeneous noise in the data. We focus on the Hartmann3D function ensemble (Dixon, 1978), which is a three-dimensional problem with four local minima. Their locations and the global minimum vary across different functions. See Appendix H for more details.

To avoid biasing this experiment towards a single method, we use a heteroscedastic noise incompatible with any assumptions about the noise of any method. In particular, this violates the GP methods’ and ABLR’s assumption of homoscedastic, Gaussian noise. GC3P makes a similar assumption after the nonlinear transformation of the observation values, which does not translate to any well-known noise model. LFBO, LFBO+BB and MALIBO make no explicit noise assumptions, but optimize for the best mean. We choose a multiplicative noise, i.e., $y=f(\mathbf{x})\cdot(1+\epsilon\cdot n)$, where $n\sim\mathcal{N}(\mathbf{0},\mathbf{1})$. The noise corrupts observations with larger values more, while having a smaller effect on those with lower values. To see the robustness with different noise levels, we evaluate $\epsilon\in\{0,0.1,1.0\}$. For meta-training, we randomly sample $512$ noisy observations from $256$ functions in the ensemble. We show our results in Figure 7, where we can see across all noise levels, our method learns a meaningful prior for the optimization. The GP-based methods, despite their strong performance in the noise-free case, especially RGPE, degrade significantly with increasing noise levels.