Paper: Large Language Models to Enhance Bayesian Optimization (ICLR-2024)

Reviewer comment (Theory):
The authors used an LLM that models data as a sequence of tokens, but HPO data has no sequential structure. They attempt to address this by permutating the data to derive uncertainty, which the reviewer finds flawed. SMAC, which is based on a random forest, uses bootstrapping that is at least statistically motivated, and the reviewer missed a convincing argument for feeding non-sequential data into a sequential model and then applying a workaround to fix it.

Author response:
**Modelling non-sequential data.** We thank the reviewer for raising this point. Using an LLM for optimization of non-sequential data is central to our paper, and how to best model non-sequential data forms the basis of our investigations. LLMs, while being sequential models, possess extensive prior knowledge, exhibit strong few-shot learning capabilities, and flexibly incorporate contextual information. These are desirable properties for BO algorithms. Our approach effectively taps into these LLM capabilities to enhance BO, albeit necessitating data serialization.

We note that in other research domains, LLMs have been evaluated for processing non-sequential data. For example, [1-3], which demonstrated LLMs’ effectiveness on tabular regression, classification, and generation, respectively (where the data under consideration lacks sequential structure). More pertinent to our study, [4] employs a Transformer to sequentially process optimization results for HPO. Our empirical analysis further substantiates our method's effectiveness: despite data serialization, we can obtain a strong surrogate model (notably in few-shot scenarios, as shown in S4), along with an efficient candidate point sampler (S5) and an overall effective system (S6). 

Ultimately, the serialization of non-sequential data within our approach effectively elicits prior knowledge about problem space correlations, and capitalizes on the few-shot learning and contextual understanding capabilities of LLMs, resulting in an improved, sample-efficient search method. This advantage is vital in most BO applications, and we believe motivates our methodological choice. While performing BO with a sequential model is unconventional, we believe the strong performance and potential improvements of this novel approach is a valuable contribution and would inspire future research in this field.

**Empirical uncertainty.** Just to clarify, we permute the order of in-context examples, not the hyperparameters within each example. While MC sampling for uncertainty estimation may lack the principled grounding of GP/RF, it is straightforward to implement and widely accepted technique. For example, in deep ensemble literature [5], empirical variance is used for uncertainty estimation. Conceptually, if we consider the predictive distribution produced by an LLM as being induced by the weights of the LLM ($\theta_{LLM}$) and the few-shot prompt ($\theta_{prompt}$), i.e. $p(y| x; \theta_{LLM}, \theta_{prompt})$ (note that $(\theta_{prompt}, x)$ represents the complete prompt), our MC sampling is an approximation of the expectation $E_{\theta_{prompt}\sim p(\theta_{prompt})} [p(y|x; \theta_{LLM}, \theta_{prompt})]$, which marginalizes over the distribution of different prompts. Note, this is conceptually identical to the approach in deep ensembles, where $E_{w\sim p(w)}[p(y|x; w)]$. Intuitively, each differently prompted LLM can be considered an ensemble member, where we compute empirical variance from the ensemble. Indeed, we demonstrated that this approach of combining MC sampling with permutations of in-context examples improved calibration of uncertainty estimates, enhancing end-to-end performance.

[1] Dinh, T., Zeng, Y., Zhang, R., Lin, Z., Gira, M., Rajput, S., Sohn, J.Y., Papailiopoulos, D. and Lee, K., 2022. Lift: Language-interfaced fine-tuning for non-language machine learning tasks. Advances in Neural Information Processing Systems, 35, pp.11763-11784.

[2] Hegselmann, S., Buendia, A., Lang, H., Agrawal, M., Jiang, X. and Sontag, D., 2023, April. Tabllm: Few-shot classification of tabular data with large language models. In International Conference on Artificial Intelligence and Statistics (pp. 5549-5581). PMLR.

[3] Borisov, V., Sessler, K., Leemann, T., Pawelczyk, M. and Kasneci, G., 2022, September. Language Models are Realistic Tabular Data Generators. In The Eleventh International Conference on Learning Representations.

[4] Chen, Y., Song, X., Lee, C., Wang, Z., Zhang, R., Dohan, D., Kawakami, K., Kochanski, G., Doucet, A., Ranzato, M.A. and Perel, S., 2022. Towards learning universal hyperparameter optimizers with transformers. Advances in Neural Information Processing Systems, 35, pp.32053-32068.

[5] Lakshminarayanan, B., Pritzel, A. and Blundell, C., 2017. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30.

Reviewer comment (Experiments):
The strength of LLMs relies on good prompts, yet the paper lacks insights or ablation studies on how to design prompts for this problem. The reviewer questions whether the prompt templates shown in the paper were developed in the very first trial.

Author response:
Thank you for this suggestion. Our prompt design adheres to the framework outlined in S2.2, including three key components: (1) optimization problem description, (2) optimization history, and (3) explicit task instructions. We would like to re-emphasize that the primary objective of our research as to *investigate the feasibility* of enhancing BO with LLMs, and not designing highly robust or optimized prompts. Most of our efforts on prompt design were concentrated on (3), ensuring that LLAMBO behaves robustly from a system perspective, consistently generating predictions/samples in valid formats amenable for subsequent processing.

To further shed light on this process, we have further elaborated on the prompt design process in App C.2. This also includes a new ablation study wherein we methodically evaluated the impact of various components, specifically **(a)** omitting meta-data from context description (c.f. (1)) and **(b)** specific task instructions (c.f. (3)). Our ablation revealed that standard LLAMBO configuration outperformed other variants, underscoring the significance of each prompt component in enhancing overall performance. Our findings on **(a)** indicated that meta-data played a hand in improving search performance further, but excluding it did not significantly affect performance. On examining **(b)**, we also found that the task-specific instructions improved validity of proposed candidate points, leading to more efficient search. We elaborate on **(a)** further in **[P4]** below.

However, we view combining recent advances in automatic prompt tuning within our framework as a promising direction that could further improve the optimization performance of LLAMBO.

**Actions taken.** We have improved our manuscript with a (1) description of the prompt design process and (2) prompt ablation study in App C.

Reviewer comment (Presentation):
The paper is very dense and includes many results, but the main paper should be self-contained. The authors moved the discussion of related work to the appendix, which the reviewer opposes, arguing it is important to have related work in the main body so readers can situate the paper. The reviewer suggests moving the generative surrogate model description to the appendix instead, as it does not outperform the discriminative model.

Author response:
We appreciate this concrete recommendation, which we have followed by reorganizing the paper to include related works in the main text.

Reviewer comment (Experiments):
Papers using closed-sourced GPT models have the inherent flaw that we have no exact knowledge of how these models are trained, including training data. This limits scientific understanding of how to use them. Although the authors attempted to address this with private and artificial datasets, there is no clear explanation of how these relate to the GPT model's training distribution, making claims about meta-data leakage impossible.

Author response:
We acknowledge your concern regarding the potential for meta-data leakage, particularly the risks of LLM memorizing task-specific hyperparameters. To mitigate this, our evaluation included three private and three synthetic datasets. It is crucial to note that no *semantic* information that could be used to directly identify underlying datasets is provided (please see our prompts in App C). Our synthetic datasets, while generated from known DGPs, had controllable input dimensions and sampling distributions, set by us.

Regarding the meta-data in our prompts, we provided only high-level information, inclusively: [# of # of features (# categorical, # numerical), # of samples, and tasks described as either classification or regression]. This information is provided as HPT optimization should benefit from this information, for instance, to avoid overfitting on big-p-little-n tasks.

To concretely address your concerns, we point to three pieces of empirical evidence:
* In our warmstarting experiments (S3), we analyzed a no context zero-shot prompt, where no semantics or meta-data is provided (see App C), and observed the LLM was able to suggest configurations that exhibited stronger correlations (indicating dataset-agnostic knowledge of generalizable correlations in the optimization landscape) that improved optimization performance.
* In S4.2 and S5, we performed ablations that removed the aforementioned meta-data from the prompts **LLAMBO (UnInf)** and found that, while the performance degraded relative to the full prompt, the performance was consistently superior to considered baselines (see Figure 4, 6).
* **[New result]** To further address any concerns of meta-data leak, we perform an additional ablation study, where we removed any meta-data from the prompts and analyzed resulting end-to-end performance. This is included in App C.2. Expectedly, this ablation performs slightly worse, as meta-data can improve hyperparameter suggestion (e.g. specific hyperparameters better suited for large-p-small-n  to guard against overfitting). However, we note that the ablation was consistently superior against compared baselines, and achieved very similar optimization performance as the full setting. This highlights LLAMBO’s effectiveness despite the lack of metadata, indicating the performance goes beyond mere reliance on data memorization or leakage (as it operates without access to specific dataset information).

We hope this sufficiently addresses your concerns about meta-data leakage.

**Actions taken.** In response to your feedback, we have conducted an ablation study to assess the impact of including meta-data in prompts on end-to-end performance in App C.

Reviewer comment (Reproducibility):
GPT-3.5 is not publicly available and may be updated over time, which makes the reproducibility of the results questionable. The reviewer strongly recommends using publicly available LLMs such as Llama 2.

Author response:
The core message of this work is to showcase the potential of LLMs in BO, and to provide insights into their effective integration within the BO process. We wish to clarify that our claim is not that LLAMBO will work with *every* LLM, but rather that LLMs possess the potential for successful BO application.

In this context, we believe that demonstrating our approach’s effectiveness with *an* LLM (in this case, GPT-3.5), is sufficient. Our aim is not to compare various LLMs in BO, as we agree that performances will inevitably vary—some may perform worse, others better. This aspect, although outside our current scope, presents an exciting avenue for future research. Particularly, as LLMs gain traction in BO and other optimization tasks, understanding the strengths and weaknesses of different models becomes crucial. Identifying the most suitable LLM for specific scenarios, or even combining multiple LLMs to enhance optimization performance, are exciting possibilities. Future studies could benefit from benchmarks similar to those that have been proposed for [question answer (HellaSwag)](https://arxiv.org/abs/1905.07830) and [mathematical reasoning (GSM8K)](https://arxiv.org/abs/2110.14168).

Lastly, we appreciate your input on the new sentence and agree to exclude it from the revised manuscript as you recommended. We have incorporated the aforementioned discussion regarding the choice of LLMs into the Discussion section of our paper.

With thanks,

Authors of #8831

Reviewer comment (Experiments):
The Bayesmark benchmark library includes mostly simple HPO benchmarks with low-dimensional, continuous spaces and traditional ML models, which the reviewer considers outdated for state-of-the-art HPO. This choice may lead to results that conflict with established knowledge, and results might differ significantly if more challenging benchmarks like HPOBench were used.

Author response:
We appreciate your recommendation. Accordingly, we have expanded our analysis to include 24 additional tasks (8 datasets, 3 ML models) from HPOBench in Appendix E.4. In the interest of time, we note that we have only included results from one seeded run, but we plan to include results across 5 runs (which will be updated as they come in). Our expanded analysis reveals LLAMBO achieves strong performance across this extended set of results, further validating the efficacy and generalizability of our approach.

Reviewer comment (Evaluation):
All plots use iterations instead of wall-time on the x-axis, preventing conclusions about the overhead from querying the LLM, which might negate its benefits depending on the benchmark.

Author response:
**Clock time.** We have now included average clock time for LLAMBO and baselines in Appendix E.5.

Reviewer comment (Evaluation):
No uncertainty is shown in any plot, despite results being averaged over several runs/seeds.

Author response:
**Uncertainty.** We have updated all figures to include 95% CI.

Reviewer comment (Evaluation):
Some results appear contradictory to previously observed behaviors of approaches (see questions referenced by reviewer).

Author response:
7. When evaluated solely as a surrogate model, SMAC exhibits better predictive performance at lower values of $N$. A possible explanation is overfitting in GP with smaller $N$, whereas RF will benefit from the variance reduction effect from ensembling. At higher values of $N$, however, GP demonstrates enhanced uncertainty estimates and predictions, leading to better overall performance in end-to-end evaluations.

Reviewer comment (Writing):
In Appendix page 15: "2. Candidate Point Sampler H, is "]" intentional?"—possible typographical issue.

Author response:
**Typos.** These have been fixed.

Reviewer comment (Evaluation):
Standard deviation (or confidence interval) is not reported for every experiment. I think this is an important component for this kind of studies. If some results are statistically meaningless, the analyses are not much meaningful.

Author response:
Thank you for this suggestion. We originally omitted error bars to improve visual clarity in interest of limited space. We have since revised the manuscript to include 95% CI on all plots  (Fig 2 - Fig 8).

Reviewer comment (Presentation):
In prompt examples, do colors have some consistent indication? For example, texts in orange have some meaning? I think you can make consistency across examples and it can help understand the prompt examples.

Author response:
**Color legend.** Thank you for this suggestion, and have incorporated a color legend in the revised manuscript.

Reviewer comment (Evaluation):
Why is no context for warm-starting Bayesian optimization better than random, sobol, or hcube? The results of no context should be similar to the random initialization methods.

Author response:
**No context warmstarting.** No context warmstarting is expected to be better than random initialization. In fact, this was one of our initial sanity check to verify that LLMs contained useful prior knowledge about the optimization problem. This strategy relies on LLMs' intrinsic knowledge of hyperparameter correlations that are independent of specific problems. For instance, there are generalizable correlations, such as the interplay between tree depth and the number of estimators in Random Forests, which are crucial for managing the bias-variance tradeoff. Unlike random initialization methods, no context warmstarting exploits these inherent correlations identified by LLMs, thereby enhancing the optimization process.

Reviewer comment (Reproducibility):
In discriminative surrogate modeling, a method of LLAMBO utilizes both Monte Carlo sampling and shuffling of examples, or it only uses the shuffling?

Author response:
**Discriminative SM.** LLAMBO incorporates both MC sampling and permutation of in-context examples. This is in contrast to the **LLAMBO (MC)** variant, which exclusively relied on MC sampling. Our comparative analyses demonstrated that permutations improved calibration of uncertainty estimates obtained from MC sampling. We have updated the manuscript to be more clear about this distinction.

Reviewer comment (Reproducibility):
Could you elaborate how $\alpha$ is used in candidate point sampling?

Author response:
**$\alpha$.** The candidate sampler generates points, denoted as $h$, from the distribution $p(h|s’, \mathcal{D}\_n)$. This process is conditioned on a desired target value $s’$ and the existing observations $\mathcal{D}\_n$. The target value $s’$ is determined using the parameters $\alpha$, based on the formula $s’ = s\_{min} - \alpha \times (s\_{max}-s\_{min})$. Here, $s\_{max}$ (worst) and $s\_{min}$ (best) represent the extremities of objective values observed during the search, essentially defining the range of these values. 
Consequently, $s’$ is calculated by adjusting the best observed objective ($s_{min}$) by an amount proportional to the total observed range ($s\_{max} - s\_{min}$). To illustrate, an $\alpha$ value of $-0.1$ corresponds to targeting the 90th percentile of the observed values, while $\alpha = 0.1$ aims to improve upon the best value by the 10th percentile. Conversely, setting $\alpha$ to 0 targets the current best observed value. We set $\alpha=-0.1$ based on observations described in Figure 6 as a value that balanced diversity-performance. We hope this explanation clarifies the role of $\alpha$ in our sampling process.

Reviewer comment (Reproducibility):
Could you explain the details of end-to-end demonstration of LLAMBO? I think it is missing in the paper including the appendices.

Author response:
**End-to-end process.** We appreciate this suggestion. The end-to-end procedure iteratively performs three steps: **(1)** sample $M$ candidate points $\\{\tilde{h}\_m\\}\_{m=1}^M $. **(2)** evaluate $M$ points using the surrogate model, i.e. $p(s|\tilde{h}\_m)$ to obtain scores $\\{a(\tilde{h}\_m)\\}\_{m=1}^M$ according to an acquisition function. We use expected improvement (EI), $a(\tilde{h}\_m)= E[\max(p(s|\tilde{h}\_m) - f(h\_{best}), 0)]$. **(3)** select point with the highest score to evaluate next, $h = argmax\_{\tilde{h}\in\\{\tilde{h}\_m\\}\_{m=1}^M}a(\tilde{h})$. We have included these details in App C.

Reviewer comment (Reproducibility):
As I mentioned earlier, I would like to see the design process of prompts.

Author response:
Thank you for this suggestion. Our prompt design adheres to the framework outlined in S2.2, including three key components: (1) optimization problem description, (2) optimization history, and (3) explicit task instructions. Given that the primary objective of our research was to *investigate the feasibility* of enhancing BO with LLMs, we did not prioritize extensive refinement of the prompts design. Most of our efforts on prompt design were concentrated on (3), ensuring that LLAMBO behaves robustly from a system perspective, consistently generating predictions/samples in valid formats amenable for subsequent processing.

To further shed light on this process, we have further elaborated on the prompt design process in App C.2. This also includes a new ablation study wherein we methodically evaluated the impact of various components, specifically **(a)** omitting meta-data from context description (c.f. (1)) and **(b)** specific task instructions (c.f. (3)). Our ablation revealed that standard LLAMBO configuration outperformed other variants, underscoring the significance of each prompt component in enhancing overall performance. Our findings on **(a)** indicated that meta-data played a hand in improving search performance further, but excluding it did not significantly affect performance. On examining **(b)**, we also found that the task-specific instructions improved validity of proposed candidate points, leading to more efficient search.

**Actions taken.** We have improved our manuscript with a (1) description of the prompt design process and (2) prompt ablation study in App C.

Reviewer comment (Experiments):
This paper mainly compares with the LLAMBO with standard BO algorithms whereas many neural network-based BO algorithms have been developed recently to improve the modeling of standard BO algorithms, e.g., [R1]. This paper may also compare with it to further support the advantages of using LLM for BO.

Author response:
Thank you for pointing out the related work [R1]. This work utilized neural network-based surrogate models to enhance BO’s performance, particularly in high-dimensional problems. Additionally, the method improves computational efficiency by circumventing the need to invert large parameter matrices. In light of your suggestion, we have included a discussion of [R1] in the related works section of the revised manuscript. Furthermore, we empirically compared against it in App E.3. We observed that while STO is competitive against other baselines, especially during early stage of search, LLAMBO is consistently superior and found better solutions more efficiently.

Reviewer comment (Theory):
In page 8, could the author explain why a negative alpha will help improve the average regret? Intuitively, a negative alpha indicates that the sampling candidates usually perform no better than the best one when compared with the existing candidates, which therefore indicates that no improvement should be made through a negative alpha.

Author response:
* Thank you for this question. The use of a low negative $\alpha$ (e.g., $\alpha = -0.1$) aims to target the 'good' regions of the search space, specifically focusing on the top $10$% of observed points. This approach enables the LLM to exploit identified patterns in the existing observations and generate points in proximity to promising solutions. The underlying assumption here is that areas near good solutions are likely to yield similarly high-quality results. This exploitation of the search space could result in identifying candidates with lower regret. We note here that this approach is akin to the sampling strategy of TPE algorithms (which typically uses $\tau=0.25$, which corresponds to constructing density estimators with the top $25$% of observed points). Additionally, this 'good' region dynamically evolves as more points are sampled and evaluated. In our implementation, we set $\alpha=-0.1$ based on observations described in Figure 6 as a value that balanced diversity-performance.

Reviewer comment (Experiments):
What's the dimension of the empirical experiments that have been conducted in this paper? Any high-dimensional ones?

Author response:
* For details on the dimensionality of the BO tasks used in our experiments, please refer to App D.