Paper: Risk-Controlling Model Selection via Guided Bayesian Optimization (ICLR-2024)

Reviewer comment (Theory):
I find the definition of the region of interest to be under-motivated. It is not clear why the region of interest is defined in this way; it would be useful to further motivate the definition by some worked example or analysis, and the definition could be compared/contrasted with other possible ways for defining the region. It may be the case that the algorithm is primarily focused on the case where objectives are conflicting; if this is so, it should be stated more clearly, along with a description of how the algorithm will perform when this is not the case.

Author response:
> On the rational behind the definition of the ROI and expected limitations

We assume that there is a limited budget for searching the configuration space since each evaluation is very expensive (requires retraining of the model). We want to focus the optimization efforts on finding configurations that are: (i) likely to pass the test (the risk is not overly high), and are (ii) effective with respect to the free objective. The defined $\alpha^\textrm{max}$ specifies the limiting value that, if observed during testing, leads to rejecting the null hypothesis and declaring that the configuration is valid. Since the objectives are conflicting, as we get close to this limiting value, we improve in terms of the free objective. Ideally, we would like to find the configurations whose expected value is exactly $\alpha^\textrm{max}$. We include in the region of interest all the configurations that are likely to be associated with this ideal configuration. Specifically, we exclude configurations if the probability to correspond to an expected value $\alpha^\textrm{max}$ is lower than $\delta’$. These configurations are not relevant since they are likely to fail the test or they do not best minimize the free objective.  To improve clarity, we added clarifications around Eqs. (5), (6) and (7) and an illustration demonstrating these principles in Fig. E2.  

Regarding the effectiveness, it depends on the given budget and the shape of the frontier. In the limit of a large budget, sampling the entire front or only a portion of it, will be comparable. In addition, if the derivative of the free objective with respect to the constrained objective is small, the improvement in the free objective as we get closer to the limit is small as well. However, even in these cases the proposed method is not supposed to fail or perform worse than the other baselines. 

> On non-conflicting objectives

Indeed, if the objectives are not conflicting the defined region of interest may not be relevant. This case can be trivially solved as a constrained optimization problem, and the solution can be statistically tested and verified without any concern that it will not pass the test. Only when the objectives are conflicting, the minimizing solution satisfies the constraint tightly, and we need to find a set of configurations around this point in order to increase the chances that we would be able to find a valid configuration.

Reviewer comment (Experiments):
While the authors seem to be motivated by the problem of “expansive high-volume configuration spaces”, their experimental applications seem to have configuration spaces that are of a similar size as those in previous work. While performing better in these spaces is useful, it would also be interesting to see that this algorithm enables some new class of problem(s) to be addressed via MHT.

Author response:
> On “expansive high-volume configuration spaces” not demonstrated 

We corrected this to “expansive configuration spaces”.  We indeed present examples with configuration space similar in size to what was previously shown. The major difference is that we demonstrate expensive configuration spaces that require retraining of large models per each new configuration. In previous works, the configurations were set by hyperparameters that affect the way the model is used after the model is already trained, which is significantly less expensive, and can be performed with a relatively large budget.

Reviewer comment (Experiments):
The experiments section does not make a strong case for the current algorithm due to unexplained hyperparameter choices. Specifically, how is $\delta'$ chosen? This seems very important, and I see no mention of this in the experiments section.

Author response:
> On the choice of $\delta’$

We added explanations about the meaning and the choice of $\delta’$ in the revised manuscript. This hyperparameter controls the size of the region around $\alpha^\textrm{max}$.  We exclude configurations that correspond to $\alpha^\textrm{max}$ with probability that is less than $\delta’$. This is an empirical choice with a tradeoff between enlarging the region of interest to include other potential configurations and increasing the density of the configurations that are found close to $\alpha^\textrm{max}$. 

We use $\delta’=0.0001$.  Following your comment, we added an experiment to examine the influence of different $\delta’$ values (see Fig. C.5.). We observe that most of the time the exact choice of $\delta’$ does not lead to significant differences in the performance. Note that the width of the region is also influenced by the size of the validation data. When $k$ increases, the width decreases as we have more confidence in the observed empirical losses of being reflective of the actual expected loss.

Reviewer comment (Presentation):
The experiments section does not make a strong case for the current algorithm due to unclear choices for the limits of x-axes. For example, why should we only be concerned with $\alpha \in [0.165, 0.18]$ for Figure 3(a) or $\alpha \in [0.055, 0.07]$ in Figure 3(b)? These choices seem arbitrary, and make it hard to takeaway that the method is generally better than the simple baseline.

Author response:
> On the chosen ranges of $\alpha$ values

We added clarifications on the choice of these values in the revised manuscript. We define the range according to the values observed for the initial pool of configurations that is generated at the beginning of the BO procedure. The minimum and the maximum edge points obtained for each task appear in table B.1 and are also observed in the examples shown on Fig. C.4.  We choose values in between these extreme edge points, but not too close to either side, since too small values may not be statistically achievable and too large values are trivially satisfied (with tighter control not significantly improving the free objective). Note that in the chosen range even small differences in the limit values along the constrained axis, result in a significant difference for the free objective. As a general note, this choice is task and model specific, and for other tasks (for example pruning) the range is much wider.

Reviewer comment (Evaluation):
The experiments section does not make a strong case for the current algorithm due to insignificant results based on standard deviation. Specifically, Figure 3(a) and 3(b) do not show the proposed algorithm to be much better than a fairly naive baseline in what seems to be a highly-controlled setting.

Author response:
> On insignificant results for fairness and robustness 

We show in the experimental results that the proposed method is consistently better compared to all other baselines, almost everywhere across tasks and $\alpha$ values. It is true that for certain tasks the naïve baselines perform well but these baselines perform much worse in other settings. The naïve baselines spread the configurations uniformly or randomly in the configuration space. The multi-objective baselines spread them all over the Pareto front. It can happen that in some cases the resulting set contains configurations that reside in proximity to the testing limit, hence leading to efficient testing outcomes. However, more often it happens that the configurations are far from the limit, resulting in reduced efficiency. The proposed method always finds a dense set of configurations around the desired limit, thus is much more stable and efficient across different settings. We added an illustration in Fig. E2. to demonstrate this principle.

Reviewer comment (Reproducibility):
The experiments section would benefit from focusing on fewer results that are more clearly explained. Even examining the details in the appendix leaves many experimental choices a mystery.

Author response:
> On missing experimental details
 
Following your comment, we edited the experimental section and added further details and clarifications.

Reviewer comment (Novelty):
Since it is not clear why this definition of the region of interest is of particular significance (either via analysis or empirical results), I find the contribution to be incremental and short of the acceptance threshold, although I could be convinced otherwise by other reviews and/or author rebuttals.

Author response:
> On the rational behind the definition of the ROI and expected limitations

We assume that there is a limited budget for searching the configuration space since each evaluation is very expensive (requires retraining of the model). We want to focus the optimization efforts on finding configurations that are: (i) likely to pass the test (the risk is not overly high), and are (ii) effective with respect to the free objective. The defined $\alpha^\textrm{max}$ specifies the limiting value that, if observed during testing, leads to rejecting the null hypothesis and declaring that the configuration is valid. Since the objectives are conflicting, as we get close to this limiting value, we improve in terms of the free objective. Ideally, we would like to find the configurations whose expected value is exactly $\alpha^\textrm{max}$. We include in the region of interest all the configurations that are likely to be associated with this ideal configuration. Specifically, we exclude configurations if the probability to correspond to an expected value $\alpha^\textrm{max}$ is lower than $\delta’$. These configurations are not relevant since they are likely to fail the test or they do not best minimize the free objective.  To improve clarity, we added clarifications around Eqs. (5), (6) and (7) and an illustration demonstrating these principles in Fig. E2.  

Regarding the effectiveness, it depends on the given budget and the shape of the frontier. In the limit of a large budget, sampling the entire front or only a portion of it, will be comparable. In addition, if the derivative of the free objective with respect to the constrained objective is small, the improvement in the free objective as we get closer to the limit is small as well. However, even in these cases the proposed method is not supposed to fail or perform worse than the other baselines. 

> On non-conflicting objectives

Indeed, if the objectives are not conflicting the defined region of interest may not be relevant. This case can be trivially solved as a constrained optimization problem, and the solution can be statistically tested and verified without any concern that it will not pass the test. Only when the objectives are conflicting, the minimizing solution satisfies the constraint tightly, and we need to find a set of configurations around this point in order to increase the chances that we would be able to find a valid configuration.

Reviewer comment (Theory):
Can you give a more detailed explanation of the rationale behind the definition of the ROI? When is it expected to be most effective, and when may it fail?

Author response:
> On the rational behind the definition of the ROI and expected limitations

We assume that there is a limited budget for searching the configuration space since each evaluation is very expensive (requires retraining of the model). We want to focus the optimization efforts on finding configurations that are: (i) likely to pass the test (the risk is not overly high), and are (ii) effective with respect to the free objective. The defined $\alpha^\textrm{max}$ specifies the limiting value that, if observed during testing, leads to rejecting the null hypothesis and declaring that the configuration is valid. Since the objectives are conflicting, as we get close to this limiting value, we improve in terms of the free objective. Ideally, we would like to find the configurations whose expected value is exactly $\alpha^\textrm{max}$. We include in the region of interest all the configurations that are likely to be associated with this ideal configuration. Specifically, we exclude configurations if the probability to correspond to an expected value $\alpha^\textrm{max}$ is lower than $\delta’$. These configurations are not relevant since they are likely to fail the test or they do not best minimize the free objective.  To improve clarity, we added clarifications around Eqs. (5), (6) and (7) and an illustration demonstrating these principles in Fig. E2.  

Regarding the effectiveness, it depends on the given budget and the shape of the frontier. In the limit of a large budget, sampling the entire front or only a portion of it, will be comparable. In addition, if the derivative of the free objective with respect to the constrained objective is small, the improvement in the free objective as we get closer to the limit is small as well. However, even in these cases the proposed method is not supposed to fail or perform worse than the other baselines. 

> On non-conflicting objectives

Indeed, if the objectives are not conflicting the defined region of interest may not be relevant. This case can be trivially solved as a constrained optimization problem, and the solution can be statistically tested and verified without any concern that it will not pass the test. Only when the objectives are conflicting, the minimizing solution satisfies the constraint tightly, and we need to find a set of configurations around this point in order to increase the chances that we would be able to find a valid configuration.

Reviewer comment (Evaluation):
Can you more concretely characterize the efficiency of your method in relation to Pareto Testing? How much compute or time is actually being saved, and how much performance is lost compared to full Pareto Testing? This should be covered more thoroughly in the experiments section, as this seems to be the main comparison made prior to the experiments section.

Author response:
> On the efficiency of the proposed method with respect to Pareto Testing

We can compare the efficiency of the proposed method with respect to Pareto Testing from two viewpoints. The first viewpoint compares testing  efficiency (quantified via a  better minimization of the free objective) for a  given optimization budget. Pareto testing suggests first finding a sampled version of the Pareto front, and then orders and tests these configurations via sequential testing. We claim that most of the Pareto front is irrelevant, and we should focus only on the part of the front that is near the testing limit. For a given finite budget, our method yields a denser set of configurations near the limit, and therefore is expected to obtain tighter control. This is demonstrated in Fig. 3, 4, C1 and C2 and is illustrated in Fig. E.2. 
The second viewpoint is the budget required to match the ideal performance of Pareto Testing with a dense grid of configurations. As shown in Fig. C.3. with a budget of 50 configurations we can match the performance of Pareto testing with a dense grid of over 6K configurations.  From this viewpoint, our method has a computational advantage and can be applied to tuning hyperparameters that affect the training of large models, which was not exemplified before in Pareto Testing.

Reviewer comment (Experiments):
How is $\delta'$ chosen? This seems like an important parameter, but I cannot find a reference to its value in the experiments, nor how that value is chosen. It currently seems possible that equation (7) is not actually that important, but instead the ROI is just based on a heuristic choice of tolerance around $\alpha^{max}$. An ablation with respect to this parameter would also be helpful.

Author response:
> On the choice of $\delta’$

We added explanations about the meaning and the choice of $\delta’$ in the revised manuscript. This hyperparameter controls the size of the region around $\alpha^\textrm{max}$.  We exclude configurations that correspond to $\alpha^\textrm{max}$ with probability that is less than $\delta’$. This is an empirical choice with a tradeoff between enlarging the region of interest to include other potential configurations and increasing the density of the configurations that are found close to $\alpha^\textrm{max}$. 

We use $\delta’=0.0001$.  Following your comment, we added an experiment to examine the influence of different $\delta’$ values (see Fig. C.5.). We observe that most of the time the exact choice of $\delta’$ does not lead to significant differences in the performance. Note that the width of the region is also influenced by the size of the validation data. When $k$ increases, the width decreases as we have more confidence in the observed empirical losses of being reflective of the actual expected loss.

Reviewer comment (Experiments):
**Empirical investigation does not properly evaluate main contribution.** As far as I can tell, the main novel contribution is the method of defining the region of interest and the reference point, after which an existing MOBO algorithm is used, and an existing algorithm for testing the validity of the returned configuration is used. The method of defining the region of interest and the reference point is not theoretically backed, which means that its usefulness is supported purely by the empirical results. However, the empirical investigation compares the entire procedure to existing MOBO algorithms (under only the Pruning setting), and the existing MOBO algorithms do not undergo the testing procedure. It is unclear how much of the performance gain is due to the main contribution (region of interest and reference point) and how much is due to the testing. The paper should include ablation studies to empirically support the claim that the main contribution is useful. How does the proposed method perform without defining the region of interest? How about if it is only a lower bound instead of an interval? It may turn out that most of the performance gain is due to the testing, rendering the main contribution not useful.

Author response:
Thank you for pointing this out. We understand we did not explain clearly the exact baselines that were compared. All baselines consist of two stages: optimization followed by testing. We do not include baselines that do not contain testing, since they were shown in [1] to lack valid risk control (they do not satisfy Eq. (1)). It was also shown in [1] that Pareto Testing is preferable over other testing mechanisms. In this paper, we compare our method to baselines that are all equivalent to Pareto Testing, while utilizing different methods for optimization in the first stage (and then perform testing in the second stage): uniform, random and different multi-objective optimization algorithms. Therefore, the testing mechanism is the same for all presented methods, and the improvement achieved by our method is related to the different optimization mechanism that is designed for improving testing efficiency.

[1] Laufer-Goldshtein, Bracha, et al. "Efficiently Controlling Multiple Risks with Pareto Testing." The Eleventh International Conference on Learning Representations. 2022.‏

 > On the performance without defining the region of interest  

The baselines of the multi-objective optimizers correspond to the case that no region is defined, and the entire Pareto front is recovered. 

> On the performance with a single limit 

Following your comment, we added an ablation study (see Fig. C.7.) to examine the case of a single limit instead of the defined region. This  is shown to be inferior with respect to the full region in most cases, as expected.

Reviewer comment (Experiments):
Furthermore, it is not clear why no previous baselines are tested in the results for Figures 3 and 5. The empirical evaluation is the only support for the proposed method, and it needs to be more comprehensive.

Author response:
> On the comparison to MOBO baselines in Figs. 5 and 6. 

Following the reviewers’ comments, we added these comparisons. See extended results in Figs.  3, 4, C1 and C2.

Reviewer comment (Writing):
In the second paragraph of Sec. 3 Problem Formulation, should it be $L_i : \mathcal Y \times \mathcal Y \times \Lambda \rightarrow \mathbb R$ instead? In this case, $L_i$ in the preceding expectation should be $L_i(f_\lambda(X), Y, \lambda)$.

Author response:
> On notation in Section 3.

We changed the notation following your suggestion.

Reviewer comment (Theory):
In Equation (7), should the formula for $\ell_{\text{low}}$ and $\ell_{\text{high}}$ be swapped?

Author response:
> On swapped limits in Eq. (7)

We corrected the typo.

Reviewer comment (Reproducibility):
In Algorithm D.1 line 9, how exactly is $\mathbf \ell (\lambda_{n+1})$ evaluated? Using $\mathcal D_{\text{train}}$ or $\mathcal D_{\text{cal}}$ ? From the line "while running the BO procedure we only have access to finite-size validation data $|\mathcal D_{\text{val}}| = k$", it sounds like it should be $\mathcal D_{\text{cal}}$. But $\mathcal D_{\text{cal}}$ is not passed into Algorithm D.1, hence the confusion.

Author response:
> On the evaluation of the loss in Algorithm D.1. 

We use the validation data during BO. Following your comment, we added it as an input to the algorithm, and clarified its use for evaluating $\hat{\ell}$. The calibration data is used afterwards in the testing procedure.

Reviewer comment (Experiments):
Why aren't the previous MOBO algorithms tested in the results in Figures 3 and 5? What about the $n=1$ case prevents these baselines from being tested?

Author response:
> On the comparison to MOBO baselines in Figs. 5 and 6. 

Following the reviewers’ comments, we added these comparisons. See extended results in Figs.  3, 4, C1 and C2.

Reviewer comment (Theory):
The formulation of the region of interest seems unclear to me. I don’t understand how Eq. (5) – which is to compute the p-value - is derived. And is this inequality be able to apply for any loss function $l(\lambda)$? Are there any particular requirements regarding this loss function for Eq. (5) to hold? Eq. (6) is also suddenly introduced without clear explanation why $\alpha^\max$ is defined that way.

Author response:
Following your comment, we added clarifications around Eqs. (5), (6) and (7), and additional derivations and mathematical details in Appendix E. In short, for testing we receive an empirical loss evaluated over the calibration data, and we need to determine whether the configuration is valid or not.  For bounded losses, Hoeffding’s inequality quantifies the probability to obtain an empirical loss value $\hat{\ell}$  under a given expected loss $\ell$.  In appendix E, we show how to derive Eq. (5) from Hoeffding’s inequality. Eq. (6)  quantifies the value of $\hat{\ell}_1$ for which the p-value in (5) equals $\delta$. This is denoted as $\alpha^\textrm{max}$. In order to be valid the loss must be lower than $\alpha^\textrm{max}$. However, for effectively minimizing the free objective function, we should be  as close as possible to $\alpha^\textrm{max}$  (since increasing $\ell_1$ implies decreasing $\ell_2$). Therefore, we define in (7) the region of all configurations that are likely to be obtained under an expected value  $\alpha^\textrm{max}$. Specifically, we exclude configurations if their probability to be associated with $\alpha^\textrm{max}$ is lower than $\delta’$. The hyperparameter  $\delta’$ controls the region's width and is unrelated to the testing confidence level $\delta$. It is an empirical choice that trade-off between including more potential values, and obtaining a set of configurations that is dense around $\alpha^\textrm{max}$.  We added an experiment (see Fig. C.5.) showing that the method is generally  insensitive to the choice of $\delta’$.

Reviewer comment (Theory):
In Eq. (7), there is also no clear explanation on how the confidence interval is constructed that way. What is the role of $\delta’$ here? How can it be chosen in practice? What significant level is associated with this confidence interval?

Author response:
Specifically, we exclude configurations if their probability to be associated with $\alpha^\textrm{max}$ is lower than $\delta’$. The hyperparameter  $\delta’$ controls the region's width and is unrelated to the testing confidence level $\delta$. It is an empirical choice that trade-off between including more potential values, and obtaining a set of configurations that is dense around $\alpha^\textrm{max}$.  We added an experiment (see Fig. C.5.) showing that the method is generally  insensitive to the choice of $\delta’$.

Reviewer comment (Theory):
I’m not sure if I missed it but it seems like there is no proof of Theorem 5.1?

Author response:
We added the proof on Appendix E.

Reviewer comment (Experiments):
Among 4 test problems, three problems (classification fairness, classification robustness, VAE) actually only have a single-dimensional hyperparameter.

Author response:
Our experiments include two settings with a multi-dimensional hyperparameter. Specifically, we including the following five tasks:

Classification fairness – 1 dimension

Classification robustness – 1 dimension

Selective classification and robustness – 2 dimensions

VAE - 1 dimension

Transformer pruning - 3 dimensions

Note that compared to previous work (Pareto Testing),  we present four cases requiring model retraining for each new hyperparameter setting. This makes the optimization much more cost intensive and is enabled by our proposed approach, which focuses on a limited part of the Pareto front.

Reviewer comment (Experiments):
Finally, the range of the value $\alpha$ seems to be quite small to me, e.g., for the classification fairness problem, the range of $\alpha$ is only from 0.165 to 0.180 or for the robustness problem, the range of $\alpha$ is only from 0.055 to 0.070.

Author response:
We added clarifications on the choice of these values in the revised manuscript. We define the range according to the values observed for the initial pool of configurations that is generated at the beginning of the BO procedure. The minimum and the maximum edge points obtained for each task appear in table B.1 and are also observed in the examples shown on Fig. C.4.  We choose values in between these extreme edge points, but not too close to either side, since too small values may not be statistically achievable and too large values are trivially satisfied (with tighter control not significantly improving the free objective). Note that in the chosen range even small differences in the limit values along the constrained axis, result in a significant difference w.r.t the free objective. As a general note, this choice is data and model specific, and for other tasks (for example pruning) the range is much wider.

Reviewer comment (Writing):
This might be subjective, but many of the contents until page 5 are fairly standard, while later many details have to be put in the appendix. The narratives in section 6 and 7 are also not clear or confusing. The writing can be improved.

Author response:
We made major changes to the paper’s writing and clarified points that were unclear to the reviewers.

Reviewer comment (Theory):
The sign of (7) should be changed.

Author response:
We corrected the typo.

Reviewer comment (Evaluation):
The practical use of the proposed method. The significance level \sigma is usually not the crucial factor, but rather performance or business metrics related to the objective function.

Author response:
We agree that  performance or business metrics related to the objective function are crucial in many practical applications.  However, in many cases there are tradeoffs between these metrics (accuracy vs cost, fairness, robustness etc.), and it is important to constrain certain metrics while optimizing others. Our method presents a simple-yet-effective approach to solve this by a new guided BO method combined with statistical testing.  We guarantee $(\alpha,\delta)$-risk control, where $\alpha$ is the bound and $\delta$ is the significance level, reflecting the certainty level with respect to the randomness of the calibration data. If $\delta =0.1$, then we can guarantee that 90% of the time (over different calibration datasets) the risk does not exceed $\alpha$.

Reviewer comment (Experiments):
Though in the intro part the potentials of the proposed method are claimed to cover different areas, in the experiment section the test is only done on limited tasks.

Author response:
We demonstrate the performance in all the tasks mentioned in the introduction: algorithmic fairness, robustness to spurious correlations, generation capabilities in VAEs and cost reduction in Transformer pruning.

Reviewer comment (Experiments):
How does the proposed method perform in situations with extremely high-dimensional hyperparameter spaces?

Author response:
The proposed method consists of two stages: (i) Bayesian optimization, and  (ii) multiple hypothesis testing. The first stage is affected by the dimensionality of the hyperparameter space. We adopt Bayesian optimization techniques, which are effective in optimizing large hyperparameter spaces. For extremely high-dimensional hyperparameter spaces we can incorporate more advanced methods for effective hyperparameter optimization [1]. The second stage of statistical testing is performed over a proportion of the Pareto frontier in the objective space, and, therefore, is not affected by the hyperparameter dimension.   
 
[1] Daulton, Samuel, et al. "Multi-objective Bayesian optimization over high-dimensional search spaces." Uncertainty in Artificial Intelligence. PMLR, 2022.‏

Reviewer comment (Experiments):
How sensitive is the proposed method to the initial definition of the "region of interest"? Is there a risk of introducing bias based on this region?

Author response:
The region of interest is defined to include configurations that can be potentially verified by testing and are effective in minimizing the free objective. We demonstrate in the experimental results that introducing the “region of interest” is almost always preferable compared to the baseline methods that consider the full front. In addition, we added experiments showing that the specified region is preferable over a one-sided bound (Fig. C.7) and that the method is generally insensitive to the choice of $\delta’$ that controls the region’s width (Fig C.6).

Reviewer comment (Theory):
Given that Bayesian Optimization inherently deals with a trade-off between exploration and exploitation, how does the new approach ensure a balance, especially with the introduced "region of interest"?

Author response:
The overall approach does not rely on the optimization mechanism, as it suggests to perform optimization over a portion of the Pareto front in order to improve statistical efficiency while selecting a risk-controlling configuration.  
Here we implement this by defining a new reference point that encloses the region of interest. This can be coupled with several acquisition functions that are defined with respect to a reference point in the multi-objective case, such as hypervolume maximization, expected hypervolume maximization and probability of hypervolume improvement. These methods offer different exploration vs exploitation balancing possibilities. In our case, we found hypervolume maximization to be effective, where there is more emphasis on exploitation. It may be attributed to the fact that the region of interest narrows the search space focusing on a relatively small region.

Reviewer comment (Evaluation):
In practical terms, what would be the computational overhead of the proposed method compared to traditional BO or other hyper-parameter optimization techniques?

Author response:
The computational overhead of the proposed method compared to traditional hyperparameter techniques is in the testing stage. We need to evaluate the configurations in the chosen set, which requires a forward-pass of the model per tested setting. This is negligible with respect to the BO part that requires retraining the model N times.