Paper: High-dimensional Bayesian Optimization via Semi-supervised Learning with Optimized Unlabeled Data Sampling (ICLR-2024)

Reviewer comment (Theory):
The concept of adding synthetic data: In BO, the GP surrogate models the underlying objective in a principled manner which yields calibrated uncertainty estimates, assuming that it is properly trained. Since the GP in itself is principled, there is no clear justification that adding synthetic data is inherently beneficial. The proposed approach suggests that the GP is either un-calibrated or does not sufficiently extrapolate the observed data. There is no insight into which possible fault (of the GP) is being addressed, nor how the added synthetic data accomplishes it. Possible mechanisms could be: (1) The teacher MLP aggressively (and accurately) extrapolates in synthetic data generation; (2) The vanilla GP is simply under-confident, so adding synthetic data reduces uncertainty by adding more data; or (3) Some other reason or a combination of the two. There is a need for intuition as to how the modeling — not just the BO results — can improve by adding synthetic data.

Author response:
**The concept of adding synthetic data:**

We concur with the reviewer’s suggestion on viewing the proposed teacher-student model, which includes the student’s feedback to the teacher, as a mechanism for providing proper regularization to the teacher. This is a good way of looking at TS-BO. On the other hand, we want to emphasize that the regularization of the teacher is tailored specifically for the sequential optimization in BO. As such, we term it as “selective regularization” to distinguish it from other more commonly used regularizes.  
We elaborate on this “selective regularization” perspective further in the response to Point 4 below. 

In response to the current question, we believe that the synthetic data offers advantages in  both areas highlighted by the reviewer:

(a) The "selective regularization" improves the teacher's extrapolation power on synthetic data, more specifically, on the potentially high-value data points. This enhancement is a result of the combined efforts of the teacher, the unlabeled data sampler, and the student model. The sampler generates synthetic data which are potentially of high values. The student model, trained on the synthetic data, provides feedback to the teacher, effectively regularizing the teacher's predictions on these data points.

(b) Vanilla Gaussian Processes (GPs) often exhibit overly conservative behavior when dealing with scarce data, leading to poor extrapolation capabilities [1, 2]. In TSBO, the selective regularization mechanism empowers the teacher to produce more precise synthetic data in regions potentially with high target value. This process increases the accuracy of the predicted label of the synthetic data. As a result, leveraging such synthetic data as additional data to train the data query GP can help reduce its uncertainty in regions that attain a high target value, which well aligns with the goal of Bayesian optimization.

Reviewer comment (Theory):
The accuracy and role of the teacher: If the teacher is able to generate accurate data (synthetic or real), why not use it as a surrogate instead of the conventional GP? Statements such as _"Evaluating the performance of the student that is trained with random unlabeled data far away from the global optimum, may not provide relevant feedback for tuning the teacher toward finding the global optimum."_ suggest that it is the teacher's (and not the BO loop's) task to find the global optimum. If the teacher adds data that it believes is good to the BO surrogate, the teacher is indirectly conducting the optimization.

Author response:
**The accuracy and role of the teacher:**

This point raised by the reviewer brings up two important issues with respect to the role of teacher which we discussed as below:

(a) Direct utilization of the Teacher Model as the BO Surrogate: 

We've conducted an additional experiment to address whether the teacher model can be directly utilized as the surrogate in TSBO. We evaluate three different methods on Expression and Chemical Design. The first method is T-LBO, as the baseline. The second model is the proposed TSBO in the main paper and the third model is a variant of the proposed TSBO where the BO surrogate is the teacher model.

For all model, the hyper-parameter settings are the same as in the paper. The best evaluations of Expr and Penalized LogP after 250 queries among 5 seeds are reported in the table below.

We've also added this experiment with detailed setting in the Appendix in our revised version for further reference.

|Method| Sorrogate|  Expr ($\downarrow$ ) | Penalized LogP ($\uparrow$ ) |
| - | - | - | -|
|T-LBO (baseline)          | GP|  0.572±0.268 | 5.695±1.254 |
|TSBO  | GP| **0.240±0.168** | **25.02±4.794** |
|TSBO  | Teacher| 0.432±0.235 | 23.574±3.017 |

As shown in the table, even though the teacher are directly adopted as a data query model, TSBO still outperforms T-LBO. 

This suggests that the quality of the teacher is better than that of the GP surrogate in more standard BO methods that do not incorporate a teacher-student model.  We could use this teacher to replace the surrogate GP in these more standard BO methods to get better results. 

Meanwhile, most of existing BO methods prefer using GPs as the surrogate model due to their principled approach to calibrated uncertainty estimation. Though our teacher model could be more accurate than the standard GP surrogates, its uncertainty calibration is not as robust as that of GPs. Therefore, in our main paper, we proposed to supplement the teacher model with an additional data query GP. This GP incorporates both the synthetic data generated by the teacher and the labeled data. Empirically, this hybrid approach combines the best of the two worlds; it not only retains the improved accuracy of the teacher model but also enhances uncertainty estimation. The above experiment validates that this hybrid approach achieves the best final BO results.

(b) Intuition Behind the Teacher Model's Role: 

We concur with the reviewer’s suggestion on viewing the proposed teacher-student model, which includes the student’s feedback to the teacher, as a mechanism for providing proper regularization to the teacher. The teacher’s role can be understood in this "selective regularization" perspective, which will be elaborated in our response to the next question on “The role of the student”.

In brief, we posit that the superiority of the teacher model over traditional GPs as a BO surrogate stems from its enhanced ability to predict the labels of the data in regions of high target value. This is achieved by our BO-specific "selective regularization" of the teacher, implemented by combining the student model with an unlabeled data sampling strategy.

Reviewer comment (Novelty):
The role of the student: If the student is trained only on synthetic data, validated on a (promising) subset of the true data, and its loss is coupled with the teacher's, then the student's only role appears to be as a regularizer (i.e., to incorporate GP-like smoothness into the MLP) while producing approximately the same predictions on labeled data. Clarification is needed as to whether this is true, and if so, what difference the student offers compared to conventional regularization.

Author response:
**The role of the student:**

In agreement with the reviewer's perspective, the proposed teacher-student model, which includes the student’s feedback to the teacher, can be viewed as a mechanism for providing proper regularization to the teacher. On the other hand, we want to emphasize that the regularization of the teacher is tailored specifically for the sequential optimization in BO. As such, we term it as “selective regularization” to distinguish it from other more commonly used regularizes.

The BO context awareness in this “selective regularization” is implemented by incorporating the student’s feedback and an unlabeled data sampling procedure that is optimized for BO. 

To achieve the goal of BO, it is desirable to train the teacher model so that it can better predict data points that have high target value. We may achieve this objective by imposing regularization focusing on points of interest in the BO process, rather than indiscriminately across the entire optimization landscape. We achieve this through our unlabeled data sampling technique, which proposes potentially optimal points in the current optimization step. The student model, trained specifically on these points, provides selective feedback to the teacher, effectively fine-tuning it. This process can be thought of as a self-evaluation by the teacher-student model on potential high-value points before performing actual data querying.

Reviewer comment (Experiments):
Few tasks in results: Three benchmarks constitute a fairly sparse evaluation. It is unclear whether the proposed method makes sense on conventional test functions, and whether a couple of those could be added.

Author response:
**Few tasks in results:**

We evaluate TSBO on an additional challenging high-dimensional global optimization task: Topology Shape Fitting Task, first considered in T-LBO [3]. The goal is to find a pre-defined target binary topology image $\boldsymbol{\mathrm{x}}^* \in \{0, 1\}^{1\times 40\times40}$. The objective function is the cosine similarity between the target $\boldsymbol{\mathrm{x}}^*$ and an input image $\boldsymbol{\mathrm{x}}$: $f(\boldsymbol{\mathrm{x}})=\boldsymbol{\mathrm{x}}^T\boldsymbol{\mathrm{x}}^*/(\lVert \boldsymbol{\mathrm{x}}\rVert_2 \lVert \boldsymbol{\mathrm{x}}^*\rVert_2)$. The VAE used in this task is a convolutional neural network [3] with a latent space of dimension 20. It is pre-trained on 10,000 unlabeled topology pictures in the dataset from [4].


|Method | Topology ($\uparrow$ ) |
| -  | -|
|Sobol  |  0.781±0.007 |
|LS-BO  |  0.834±0.021 |
|W-LBO  |  0.842±0.012 |
|T-LBO  |  0.840±0.032 |
|TSBO | **0.862±0.022** |

We compare TSBO with all baselines considered in the manuscript. Each method starts with 100 labeled images, uniformly sampled from the dataset in [4]. We report the best score after 250 new queries, averaged across 3 random seeds. The hyper-parameter settings are the same as in the paper.

As shown in the table above, TSBO significantly outperforms baselines, demonstrating its effectiveness and versatility in global optimization tasks.

Reviewer comment (Experiments):
Question — Is the synthetically generated data added as with any other data point, or are they added with additional noise?

Author response:
**1.** The synthetic data are treated as other labeled data, i.e., no additional predicted noises involved in fitting the data query GP.

Reviewer comment (Experiments):
Question — Is the synthetically generated data ever replaced or substituted?

Author response:
**2.** Yes, the synthetic data are re-sampled at the beginning of each BO iteration. The computational overhead of sampling is relatively neglectable in light of costly efforts for data evaluations.

Reviewer comment (Evaluation):
Question — How (qualitatively) does the SGP differ from the BO GP in its prediction?

Author response:
**3.** The student GP (SGP) is fitted on synthetic data with pseudo labels. Its accuracy on labeled data tends to be worse than the vanilla BO GP. Thus, the student is only utilized to provide an effective regularization to the teacher via the feedback loss.

Reviewer comment (Evaluation):
Question — Does the parametrized sampling distribution discriminate regions that are believed to be good (like the GEV)?

Author response:
**4.** Yes, the data sampled by our parametrized sampling distribution tend to have higher evaluation values than the initial training data or data sampled randomly. For instance, in the final BO iteration for the molecule design task, the highest Penalized LogP value among the synthetic data generated by our sampler is larger than 20, whereas the maximal value of real molecules in the ZINC250 dataset is smaller than 4.

Reviewer comment (Novelty):
I have a broad top-view question. The premise of the paper is very reasonable in suggesting that the bottleneck for existing BO methods is the limited amount of labeled data available. However, labeled data is still required for training all the new components introduced in the paper (teacher model/parametrized sampling distribution). If semi-supervised learning is the key source for getting such improved results, why wouldn't any other semi-supervised learning approach (manifold/laplacian regularization etc) work equally well?

Author response:
Manifold Regularization and Laplacian Regularization are predicated on specific data assumptions. For instance, Laplacian Regularization penalizes the norm of the model's gradient when the density of unlabeled data is large. 

While these methods can potentially improve the modeling of the data geometry and hence improve the data query process when applied to Bayesian Optimization, the regularizations are applied across the entire data manifold and do not specifically target high-value regions, which are crucial in BO. In contrast, TSBO introduces a more nuanced, context-aware regularization (detailed in the next paragraph), particularly focused on areas with potentially high values. This targeted approach, as we hypothesize, is likely to be more effective within the scope of Bayesian Optimization.

The generation and utilization of unlabeled data within TSBO can be regarded as a form of BO specific "selective regularization" imposed on the teacher model. This enhancement is achieved by the combined efforts of the teacher, the unlabeled data sampler, and the student model: The sampler generates synthetic data which are potentially of high values. The student model, trained on the synthetic data, provides feedback to the teacher, effectively regularizing the teacher's predictions on these data points. This specialized form of regularization enables the teacher model to better extrapolate in regions of potential high values, providing valuable synthetic data that diminishes the uncertainty of the surrogate GP model. Consequently, this leads to improved sample efficiency, as demonstrated in our study.

Reviewer comment (Novelty):
There is a large body of work on both high-dimensional BO and latent space BO that is not discussed in the paper. Please see some references below. It is not necessary to compare with them experimentally but they deserve proper discussion in the paper as they are directly relevant to the studied problem. In fact, reference [9] below even considers including semi-supervised learning via an auxiliary network. Reference [6] is one of the key baselines because it re-trains the GP surrogate along with VAE parameters as we get more data.  

BO over high dimensional inputs

[1] Eriksson, D., Pearce, M., Gardner, J., Turner, R. D., & Poloczek, M. (2019). Scalable global optimization via local Bayesian optimization. Advances in neural information processing systems, 32.

[2] Eriksson, D., & Jankowiak, M. (2021, December). High-dimensional Bayesian optimization with sparse axis-aligned subspaces. In Uncertainty in Artificial Intelligence (pp. 493-503). PMLR.

[3] Papenmeier, L., Nardi, L., & Poloczek, M. (2022). Increasing the scope as you learn: Adaptive Bayesian optimization in nested subspaces. Advances in Neural Information Processing Systems, 35, 11586-11601.

[4] Letham, B., Calandra, R., Rai, A., & Bakshy, E. (2020). Re-examining linear embeddings for high-dimensional Bayesian optimization. Advances in neural information processing systems, 33, 1546-1558.

[5] Nayebi, A., Munteanu, A., & Poloczek, M. (2019, May). A framework for Bayesian optimization in embedded subspaces. In International Conference on Machine Learning (pp. 4752-4761). PMLR.

Latent space BO

[6] Maus, N., Jones, H., Moore, J., Kusner, M. J., Bradshaw, J., & Gardner, J. (2022). Local latent space bayesian optimization over structured inputs. Advances in Neural Information Processing Systems, 35, 34505-34518.

[7] Deshwal, A., & Doppa, J. (2021). Combining latent space and structured kernels for bayesian optimization over combinatorial spaces. Advances in Neural Information Processing Systems, 34, 8185-8200.

[8] Notin, P., Hernández-Lobato, J. M., & Gal, Y. (2021). Improving black-box optimization in VAE latent space using decoder uncertainty. Advances in Neural Information Processing Systems, 34, 802-814.

[9] Eissman, S., Levy, D., Shu, R., Bartzsch, S., & Ermon, S. (2018, January). Bayesian optimization and attribute adjustment. In Proc. 34th Conference on Uncertainty in Artificial Intelligence.

Author response:
We appreciate you for mentioning recent reference highly related to TSBO. All of them will be properly introduced in Section "Related Work" of an updated manuscript. Moreover, recent released Latent Space BO work will be discussed as well, such as [1, 2, 3]. 

Herein, we discuss the two VAE-based Latent Space BO approach that you specifically concerned: BO with attribute adjustment [4] ([9] in your reference order) and LOLBO [5] ([6] in your reference order). In convenience, we denote the first method as AABO. 

At a high-level, AABO and LOLBO introduce new loss functions to the VAE in the context of BO. AABO utilizes a target predictor in the latent space, and trains the VAE to simultaneously maximize the standard unlabeled ELBO objective and minimize the supervised MSE loss of the predictor. LOLBO, on the other hand, optimizes its VAE to maximize the standard ELBO and the supervised NMLL loss of the data query GP.

Both approaches aim to leverage semi-supervised learning to train a better VAE, and have achieved promising experimental results. However, TSBO focuses on systematically integrating unlabeled data with their predicted pseudo labels into the data query GP. We believe the proposed techniques in TSBO are complementary to the existing VAE-based Latent Space BO work. Furthermore, [6] and [9] can be integrated into the TSBO framework with ease.