Paper: Large-Batch, Iteration-Efficient Neural Bayesian Design Optimization (ICLR-2024)

Reviewer comment (Writing):
Upon first reading, I felt the title was somewhat cluttered.

Author response:
We are open to consider renaming the paper and we appreciate to know your feedback in this regards. We think we can drop ‘iteration-efficient’ and ‘design’ from the title.

Reviewer comment (Experiments):
Presumably for small use cases, optimising 2 simple objectives over 2 dimensions batchsize 2, i.e. the ideal use case for any GP-BO, the proposed method would suffer, is there a crossover from where more simple GP-BO methods fail and LBN-MOBO would be best?

Author response:
In our empirical studies on batch multi-objective Bayesian Optimization (BO), we've identified that a batch size of 100 serves as a threshold, beyond which traditional BO methods begin to encounter difficulties. Interestingly, there are some ingenious adaptations to BO that enable operation with batch sizes approaching 1000. However, for even larger batch sizes, the shift towards Bayesian Neural Networks (BNNs) in place of Gaussian Processes (GPs) and replacing the acquisition function with the 2MD function becomes essential. As we scale up to batch sizes of 10,000, only methods like Monte Carlo Dropout (MCDropout) and Deep Ensembles in combination with 2MD acquisition prove capable of effectively handling such extensive batch sizes.

Reviewer comment (Novelty):
Contribution is not enough. The innovation of the paper can be summarized into a new predictive model with a minor modification on the original Deep Ensembles model and a new multi-objective batch acquisition function. For the predictive model, the reason for modifying the uncertainty measurement part from aleatory noise to epistemic noise is unclear. Also, the benefit from this change is not verified in the paper. Moreover, the novelty of 2$\textit{M}$D acquisition compared to the other acquisition functions is not clear either, except for being more scalable.

Author response:
## The reason for modifying the uncertainty measurement part to include only epistemic uncertainty and its benefits. 
Uncertainty in BO primarily serves as a guide for effective exploration. Epistemic uncertainty represents a lack of knowledge about a phenomenon, which can be mitigated through additional observations. Conversely, aleatoric uncertainty relates to the inherent noise in the system and is not a function of data accumulation.

In our study, we worked with the underlying Native Forward Processes (NFPs) that are essentially noiseless. Consequently, addressing noisy NFPs is earmarked for future exploration (Limitation section). Given this noiseless scenario, aleatoric uncertainty is presumed negligible, leading us to focus solely on the epistemic uncertainty. This decision significantly simplified and stabilized our model's training phase. This means our training process closely resembles that of conventional neural network, avoiding potential complexities and instabilities in training Deep Ensembles [1][2]. This methodological choice not only enhances the robustness of LBN-MOBO but also increases its accessibility and practicality as any neural net architecture can be utilized.

## Usefulness of aleatoric uncertainty.
Prompted by the reviewer's insightful question, we explored a toy noisy NFP to see whether considering aleatoric uncertainty helps with the exploration. This example shows that in the presence of significant noise, using only the epistemic uncertainty could encourage sampling of noisy regions (likely due to Deep Ensembles mechanism), which will deteriorate the exploration. Very interestingly, including the aleatoric uncertainty and avoiding the regions with such uncertainty improves the exploration significantly. This has highly appealing implications for BO in the presence of noise and we plan to study it intensively in the future.  This is detailed in our supplementary material (Rebuttal PDF_LBN_MOBO_noisy.pdf). 

These findings highlight the potential challenges and benefits of aleatoric uncertainty in scenarios with inherent noise, paving the way for future research in this domain. 

## Comparing 2MD acquisition to the other acquisition functions, except for being more scalable.
The traditional approaches to acquisition, such as the Upper Confidence Bound (UCB), relies on some form of weighted sum of uncertainty and prediction. This weighting is crucial as it dictates the balance between exploration and exploitation within the optimization process. However, the scale differences between prediction and uncertainty necessitate careful tuning of this hyperparameter. Additionally, as the optimization progresses, the requirements for exploration or exploitation may change, potentially requiring ongoing adjustments to this weight. In the realm of multi-objective optimization, the complexity escalates further due to the need to fine-tune multiple parameters to achieve an optimal balance in exploring various objectives and their respective uncertainties.

In contrast, the 2MD acquisition function, as described in the paper (Section 4.2 and Equation 3), adopts a fundamentally different approach. Here, uncertainties and predictions are treated as separate optimization objectives, which are then jointly optimized to construct a Pareto front of potential candidates. This method does not need the summation of objectives, thereby remedies the issue of scale mismatch between objectives. Consequently, our approach is free from the constraints of hyperparameter tuning, simplifying the optimization process. It's important to note that, to our knowledge, there are no alternative methods capable of scaling to the large batch sizes required for addressing real-world problems of the magnitude we are tackling, positioning our algorithm as a unique solution in this domain.

Reviewer comment (Experiments):
More benchmark models should be considered in the two realistic tasks. In these two tasks, only two models are considered, i.e., modified deep ensemble + 2$\textit{M}$D and dropout + 2$\textit{M}$D. Therefore, whether the modified deep ensemble + 2MD indeed performs well enough is not clear. It would be great if the authors can also consider more models as benchmarks.

Author response:
## More benchmark algorithms other than BNNs on real problems.
In appendix section C5 you can find complementary experiments comparing LBN-MOBO using other algorithms that are partially or completely capable of handling such large batch sizes.

Reviewer comment (Presentation):
How is the “time” defined in Figures 1 and 3?

Author response:
## How is the “time” defined in Figures 1 and 3?
It is the summation of acquisition time plus training the surrogate (a BO cycle) for 10 iterations.

Reviewer comment (Evaluation):
How well does the modified deep ensemble quantify uncertainty, compared to the original deep ensemble?

Author response:
## How well does the modified deep ensembles quantify uncertainty, compared to the original deep ensemble?
There is a widely established way of implementing Deep Ensembles where the training of prediction networks is separated from the uncertainty estimation networks [2]. The proposed simplified implementation is identical to that approach when it comes to computing the epistemic uncertainty: we ensemble prediction networks to calculate the epistemic uncertainty.

Reviewer comment (Reproducibility):
How does 2MD work? It looks like a key ingredient of the new acquisition is the NSGA-II. However, this is not explained in the main body of the paper. Also, how to implement the acquisition function, one of the most important parts in this paper, is not clearly explained. The only relevant statement is the last three lines in page 6.

Author response:
## More details on the implementation of 2MD acquisition function and the role of NSGA-II.
2MD acquisition treats the surrogate model as a black box function and perform the acquisition by iteratively querying the surrogate while using the NSGA-II algorithm to guide the system toward the Pareto front of the the best solutions and highest epistemic uncertainties. These solutions are the candidates that we evaluate using the NFP to generate the next-generation dataset.

NSGA-II serves as a scalable multi-objective optimization inside the pipeline. If for a specific task one can come up with a more suitable optimizer, NSGA-II can easily be replaced. However since querying the Deep Ensembles surrogate is very inexpensive, NSGA-II proves to be a very effective and cost-efficient choice in practice.
To gain more information about its mechanism inside LBN-MOBO pipeline, please refer to our code: https://github.com/an-on-ym-ous/lbn_mobo

Reviewer comment (Theory):
In reality, the magnitude or the range of $F$ is usually unknown. How do the authors suggest to balance the tradeoff between the output of the predictive model and the uncertainty?

Author response:
## How to balance the trade-off between the output of the predictive model and the uncertainty?
In the 2MD acquisition function, uncertainties and predictions are handled as independent objectives. We optimize uncertainties and predictions as different objectives simultaneously to create a Pareto front of possible choices. Our method avoids combining the two, which means we don't have to worry about their different magnitudes. As a result, our method doesn't need complicated hyperparameter tuning, making the optimization process simpler.

Reviewer comment (Theory):
Although I like the simplicity of the approach, the reasoning behind choosing this instantiation of multiobjective optimization is not entirely clear. Please consider adding some more analysis about the principles behind the proposed acquisition function.

Author response:
## More analysis on 2MD acquisition function.
We have provided the following insights and analyses about our acquisition function: 
- In our paper, Section 5.4 is dedicated to analyzing the influence of uncertainty on the performance of the 2MD acquisition function. Through this analysis, we aim to emphasize the critical role that epistemic uncertainty plays in determining the overall quality of the LBN-MOBO approach.

- We have conducted a preliminary evaluation on the feasibility of applying the 2MD Pareto front approach to noisy Native Forward Processes (NFPs). This examination, detailed in a separate supplementary material (Rebuttal PDF-LBN_MOBO_noisy.pdf), explores the impact of incorporating aleatoric uncertainty into the acquisition function. Our findings reveal that relying solely on epistemic uncertainty in noisy scenarios can inadvertently lead the optimization process to select candidates from these noisy regions. In contrast, by taking into account aleatoric uncertainty and striving to minimize it during the optimization process, we can effectively avoid these noisy areas. This insight highlights the importance of considering both types of uncertainty in the optimization strategy when dealing with noisy NFPs.

- We have introduced a new subsection that delves into how the 2MD Pareto front can be paired with any arbitrary acquisition function. This aspect is crucial as it positions 2MD as a generic acquisition function adaptable to various contexts. In Section 5.1, we demonstrated this adaptability by coupling the 2MD approach with a range of Bayesian Neural Networks (BNNs). 

- In Section C3, we revisited the evaluations from Section 5.1, this time using a new problem, i.e., the DTLZ5 test suite. This problem has 20 input and 3 output dimensions. (When applying 2MD acquisition function, this translates to a 6D problem.) Although Monte Carlo Dropout (MCDropout) and Deep Ensembles showed the best runtime performance, the quality of the solutions across all surrogates was highly competitive. This finding suggests that the 2MD acquisition approach is capable of delivering strong performance with various surrogate models, indicating its versatility and effectiveness in handling complex multi-dimensional optimization tasks.

Reviewer comment (Theory):
The use of BNN as the surrogate model to enhance the performance is surely promising, however, BNN has many problems. For example, the tuning of its hyperparameters could be another optimization problem or the uncertainty provided by the BNNs could be inaccurate. However, these problems are not discussed in the paper. Furthermore, I don't understand why the proposed method only requires the epistemic uncertainty. There are no motivation, explanation, or insights about this choice and why it work.

Author response:
## Why does LBN-MOBO only needs epistemic uncertainty (and not aleatoric uncertainty)? 
Epistemic uncertainty represents the lack of knowledge about a phenomenon and can be reduced with more observation. The aleatoric uncertainty is a sign of inherent noise which cannot be reduced. In this paper we assumed that our underlying Native Forward Processes (NFPs) do not feature significant noise and left handling noisy NFPs for the future work (See Limitation section). In the absence of significant noise, the inclusion of aleatoric uncertainty does not affect the solutions. Thus, we decided to modify the Deep Ensembles to calculate only the epistemic uncertainty which is necessary for the exploration. This decision significantly simplifies and stabilizes our model's training [1] phase, as we don’t need to train variance networks typically needed for aleatoric uncertainty [2]. 

Prompted by the reviewer's insightful question, we explored a toy noisy NFP to see whether considering aleatoric uncertainty helps with the exploration. This example shows that in the presence of significant noise, using only the epistemic uncertainty could encourage sampling of noisy regions (likely due to Deep Ensembles mechanism), which will deteriorate the exploration. Very interestingly, including the aleatoric uncertainty and avoiding the regions with such uncertainty improves the exploration significantly. This has highly appealing implications for BO in the presence of noise and we plan to study it intensively in the future.  This is detailed in our supplementary material (Rebuttal PDF_LBN_MOBO_noisy.pdf). 

## Discussion of the problems inherent to BNNs (e.g., tuning and uncertainty quality).
The reviewer is right about these challenges of BNNs. While we don’t focus on improving on the BNNs side, our simplified approach in Sections 4.1 and 4.2 addresses both concerns to a great extent. Regarding the surrogate model, by excluding the aleatoric uncertainty we have avoided several complexities and instabilities during the training phase [1][2], making it completely similar to training a group of conventional neural networks. Moreover the way we calculate the epistemic uncertainty is identical to the original Deep Ensembles methods and it is among the best (quasi-)BNNs in terms of the trade off between the quality of uncertainty and scalability [3].

Reviewer comment (Experiments):
The experimental evaluation is very limited. It doesn't compare against other baselines in the main paper. In Section 5.1, it is not convincing to choose the surrogate model (inference method) by using only 1 synthetic experimental result.

Author response:
## More examples to show the advantage of our proposed surrogate over its possible alternatives.
We have included another example with larger input and output dimensions to demonstrate the robust performance and scalability of our pipeline. We used the DLTZ5 test, which has 20 input design parameters and 3 output dimensions, providing a rigorous challenge. The Section C3 in the revised manuscript shows that this problem behaves similarly to our previous tests. While the performance of all methods is comparable, dropout and Deep Ensembles stand out for their significantly higher scalability.

To align these experiments more closely with our real-world scenarios, we conducted additional tests using HMC and SGHMC surrogates (the two BNNs that could scale beyond 1000 sample batch sizes) with large batch sizes of 5000 and 10000 on the 30-dimensional ZDT3 problem (in a setup similar to Sections 5.1 and C1.4). The results, including runtime and objective values, were captured within a 24-hour GPU time frame.

| Method         | Batch Size | Objective | Runtime  |
|----------------|------------|-----------|----------|
| HMC            | 5000       | 0.2217    | 53433s   |
| HMC            | 10000      | Failed    | Failed   |
| SGHMC          | 5000       | 0.1921    | 43749s   |
| SGHMC          | 10000      | Failed    | Failed   |
| DeepEnsembles  | 5000       | **0.3001**    | **721s**     |
| DeepEnsembles  | 10000      | **0.3001**   | **1377s**    |

The data in our table illustrates that the Deep Ensemble not only achieves a better Pareto front but also does it so much faster compared to other BNNs. Furthermore, the table highlights the impracticality of using other BNNs in the context of our real-world problems, as we employed batch sizes exceeding 10,000 samples. This finding reinforces the effectiveness and efficiency of our chosen methods in handling large-scale, complex problems.

Reviewer comment (Reproducibility):
Section 4.1 only covers the modification for Deep Ensembles method. It is not clear how to apply the modification for other inference methods (SGHMC, HMC, DKL, IBNN), so as to compare in Figure 3.

Author response:
## Implementation details of using 2MD Pareto front with other BNNs 
In principle, one of the main advantages of 2MD acquisition function is that it only queries the surrogate models (without the need of gradients for example). 2MD performs by iteratively using these queried information and evolves the solution until a practically good convergence. We included a section and a conceptual figure in the appendix to clarify how it works (Section B2).

Reviewer comment (Novelty):
The concept of the 2MD acquisition function is quite similar to Upper Confidence Bound with a specific exploration factor. It seems that in UCB, both the prediction and the uncertainty are incorporated to compute the acquisition function, while 2MD use the two values as separate objective to optimize. Can the authors point out some differences between the UCB and 2MD?

Author response:
## Similarity and differences between UCB and 2MD Pareto front.
As reviewer correctly points out UCB and 2MD perform the optimization while exploiting the uncertainties in different ways.
UCB works based on a weighted sum of uncertainty and prediction. In UCB tuning is important as it regulates the trade-off between exploration and exploitation. Given the magnitude difference between prediction and uncertainty, one needs to tune this hyperparameter. Moreover, as we progress through the optimization the need for exploration or exploitation might change and as a result we might need to constantly adapt this hyperparameter. In the case of multi-objective optimization, the number of tuning parameters increases as we need to tune and find the right balance in exploring different objectives in the presence of their uncertainties. 

In the 2MD acquisition function, however, the uncertainties and predictions remain independent objectives and are jointly optimized to generate a Pareto front of viable candidates (Section 4.2 and Equation 3). Since each objective is optimized jointly with other objectives without any form of summation we do not need to worry about the scale difference and LBN-MOBO does not introduce extra hyperparameters. 

Unfortunately, we could not find any implementation of UCB extension capable of handling both multi-objective and batch optimization to compare it against our method. If pointed to such an implementation, we will be happy to compare it with our 2MD algorithm. To the best of our knowledge, qEHVI and qNEHVI are the state-of-the-art algorithms for multi-objective batch Bayesian optimization. We showed that these methods are not suitable for the large batch setting we are proposing (Section 3).

Reviewer comment (Experiments):
Is batch size b > 1000 a normal batch size in real-world problem? There seems to be no reference to any applications using such large batch.

Author response:
## Is batch size b > 1000 a normal batch size in real-world problem?
Absolutely! Specifically in the contex of manufacturing, fabrication and simulation (for which we have shown two example experiments in the paper). In many such cases we are capable of producing very large batches of data with almost the same cost of a single sample but iterative lab visits are costly. As a result, LBN-MOBO makes the overall process efficient by reducing the necessity of lab visits (second paragraph of the introduction).

Reviewer comment (Experiments):
The two real-world problems use b=15,000 for airfoil problem and b=20,000 for printer problem, on a total of 10 iterations. With such a large number of function evaluations (150,000 and 200,000), can LBN-MOBO outperform Evolutionary Computation methods, e.g., MOEA/D, NSGA-II? These two EC methods are quite powerful for solving multi-objective optimization problems.

Author response:
## Can LBN-MOBO outperform Evolutionary Computation methods?
Yes. In section C1.4 of appendix we have compared LBN-MOBO against NSGA-II and a few other advanced (not necessarily neural) BO methods on ZDT1, ZDT2, ZDT3, DLTZ1, and DLTZ4 test suits as well as on **both our real-world problems** (Sections C5.1 and C5.2). In all these experiments LBN-MOBO outperforms all other methods handily.