Paper:
Bayesian Optimisation is an important decision-making tool for high-stakes applications in drug discovery and materials design. An oft-overlooked modelling consideration however is the representation of input-dependent or heteroscedastic aleatoric uncertainty. The cost of misrepresenting this uncertainty as being homoscedastic could be high in drug discovery applications where neglecting heteroscedasticity in high throughput virtual screening could lead to a failed drug discovery program. In this paper, we propose a heteroscedastic Bayesian Optimisation scheme which both represents and optimises aleatoric noise in the suggestions. We consider cases such as drug discovery where we would like to minimise or be robust to aleatoric uncertainty but also applications such as materials discovery where it may be beneficial to maximise or be antifragile to aleatoric uncertainty. Our scheme features a heteroscedastic Gaussian Process (GP) as the surrogate model in conjunction with two acquisition heuristics. First, we extend the augmented expected improvement (AEI) heuristic to the heteroscedastic setting and second, we introduce a new acquisition function, aleatoric-penalised expected improvement (ANPEI) based on a simple scalarisation of the performance and noise objective. Both methods are capable of penalising or promoting aleatoric noise in the suggestions and yield improved performance relative to a naive implementation of homoscedastic Bayesian Optimisation on toy problems as well as a real-world optimisation problem.

Review:
SUMMARY OF REVIEW
This paper discusses an interesting problem of BO in the cases of robustness and antifragility to aleatoric noise/uncertainty. To tackle this problem, the authors replace the conventional homoscedastic GP model with a heteroscedastic GP model. In the case of robustness to aleatoric noise/uncertainty, the authors have modified EI by simply either scaling down [32] or subtracting from its value more when the aleatoric noise increases. In the case of antifragility to aleatoric noise/uncertainty, they do the opposite.
The modifications of EI to handle robustness and antifragility to aleatoric noise/uncertainty are simple and straightforward, one of which is similar to the augmented EI of [32].
There are some technical ambiguities, as detailed below. In particular, the choice of objective function (equation 9) for this problem needs to be justified and motivated by the practical applications described in Section 1.
Since no convergence guarantee is given, a more extensive empirical analysis with real datasets needs to be provided to better understand the performance and behavior of the proposed BO algorithms. In particular, though the authors have motivated their problem using the compelling applications of materials and drug discovery, no experimental result for these applications has been provided to support the motivation of this work.
DETAILED COMMENTS
The authors seem to motivate the significance of their problem of interest through the key applications of materials and drug discovery which I can appreciate. Unfortunately, experimental results in such applications were not available in this paper to "close the loop" in supporting the motivation of this work, begging the question whether their proposed BO algorithm indeed works for these key applications. For example, why is the FreeSolv hydration energy dataset not used for your experiments?
For Fig. 1, how exactly do you extract the error magnitudes from the FreeSolv hydration energy dataset? How do you exactly define the notion of calculated vs. experimental uncertainties? Fig. 1 shows that the noise peaks with a relatively high frequency at a single error magnitude value. Would the assumption of homoscedastic noise at this peaked value be detrimental to BO? A sensitivity analysis would be useful here.
For the soil phosphorus fraction dataset (Fig. 2), the skewed distribution of the measurements (few extremely large measurements and many small-valued measurements) may not be due to heteroscedastic aleatoric uncertainty. In fact, in the literature of earth/environmental science, such a dataset is often modeled using a log-Gaussian process (or log-normal kriging), that is, the log-measurements follow a GP:
Webster, R., and Oliver, M. 2007. Geostatistics for Environmental Scientists. John Wiley & Sons, 2nd edition.
Can the authors provide supporting evidence (in the form of references) that such a dataset is due to heteroscedastic aleatoric uncertainty?
On page 4, step 2 of the most likely heteroscedastic GP algorithm [28] cannot be understood: What is E[x]? Isn't G_1 a GP? Why is it able to accept x_i and D as inputs? How is z_i defined?
The authors say that "A note on the form of this variance estimator is give in Appendix B." There is no Appendix B.
Can the authors give a detailed discussion why is the expression of f(x) = g(x) + s(x) in equation 9 the right one to be minimized in practice (e.g., in the context of materials and drug discovery)? For example, do material scientists use such an objective function? Provide references. Furthermore, why is the same equation 9 being minimized for both the cases of robustness and antifragility to aleatoric uncertainty?
Caption of Fig. 3: I can't understand the sentence: "The combined objective, which when optimised maximises the sin wave subject to the minimisation of aleatoric noise". This was repeated in Section 5.5: "finds the first maximum as that which minimises aleatoric noise". Isn't the minimum of the aleatoric noise at the origin (Fig. 3b)?
Can the authors provide information on how much initial data was provided prior to running BO? How much data is used for learning the GP hyperparameters?
The performance advantage of heteroscedastic ANPEI over homoscedastic does not appear to be significant for Branin-Hoo function (Figs. 5b and 6b). Can the authors explain this?
Minor issues
There are two different font types of x in bold.
Page 5: fixed aleaotric
Which phrase is correct? "is obtained by subtracting the noise function from the 1D sinusoid" in the caption of Fig. 3 or "The objective function in all cases is the principal objective g(x) minus one standard deviation of the ground truth noise function s(x)"?
Fig. 5b: Shouldn't the vertical axis be labeled as ...+ Noise?
Fig. 6a: Shouldn't the vertical axis be labeled as ...- Noise?