Towards Explaining Hyperparameter Optimization via Partial Dependence Plots
Abstract
Automated hyperparameter optimization (HPO) approaches often do not provide valuable insights into the eﬀects of diﬀerent hyperparameters on the ﬁnal model performance.
This lack makes it diﬃcult to trust and understand the automated HPO process and its results.
We suggest using interpretable machine learning (IML) to gain insights from the experimental data obtained during HPO and discuss the popular case of Bayesian optimization (BO).
BO tends to focus on promising regions with potential high-performance conﬁgurations and thus induces a sampling bias.
Hence, many IML techniques, like Partial Dependence
Plots (PDP), carry the risk of generating biased interpretations.
By leveraging the posterior uncertainty of the BO surrogate model, we introduce a variant of the PDP with estimated conﬁdence bands.
In addition, we propose to partition the hyperparameter space to obtain more conﬁdent and reliable PDPs in relevant sub-regions.
In an experimental study, we provide quantitative evidence for the increased quality of the PDPs within sub-regions.
1. Introduction
AutoML systems mainly return well-performing conﬁgurations and leave users without insights into decisions of the optimization process.
Many data scientists do not trust the outcome of an AutoML system because of the lack of transparency (Drozdal et al., 2020).
Providing insights into the search process may help increase trust and facilitate interactive and exploratory processes: A data scientist could monitor the AutoML process and make changes to it (e.g., restricting or expanding the search space) already during optimization to anticipate unintended results.
Transparency, trust, and understanding of the inner workings of an AutoML system can be increased by interpreting the internal surrogate model of an AutoML approach.
For example, BO trains a surrogate model to approximate the relationship between hyperparameter conﬁgurations and model performance.
Hence, surrogate models implicitly contain information about the inﬂuence of hyperparameters.
If the interpretation of the surrogate matches with a data scientist’s expectation, conﬁdence in the correct functioning of the system may be increased.
If these do not match, it provides an opportunity to look either for bugs in the code or for new theoretical insights.
We propose to analyze surrogate models with methods from IML to provide insights about the results of HPO.
We focus on the PDP (Friedman, 2001) as it is a widely used method to visualize the average marginal eﬀect of single features on a black-box model’s prediction.
When applied to surrogate models, they provide information on how a speciﬁc hyperparameter inﬂuences the estimated model performance.
However, applying PDPs out of the box to surrogate models might lead to misleading conclusions.
Eﬃcient optimizers tend to focus on exploiting promising regions of the hyperparameter space while other regions are less explored, and therefore, a sampling bias in decision space is introduced.
Consequently, the surrogate model trained on such experimental data might be biased and thus poorly ﬁt underexplored regions.
It follows that PDPs for those surrogate models can be biased when calculated on the entire hyperparameter space.
Contributions: We study the problem of sampling bias in experimental data produced by AutoML systems and the resulting bias of the surrogate model and assess its implications on PDPs.
We then derive an uncertainty measure for PDPs of probabilistic surrogate models and visualize it as a conﬁdence band around the PDP mean estimate.
In addition, we propose a method that splits the hyperparameter space into interpretable sub-regions with varying uncertainties.
Thereby, we obtain sub-regions with more reliable and conﬁdent estimates for PDPs.
In the context of BO, we provide evidence for the usefulness of our proposed methods in an experimental study in which we optimize the architecture and hyperparameters of a deep neural network.
2. Background and Related Work
Recent research in AutoML started to question that their evaluation is often solely based on models’ predictive performance without considering interpretability (Hutter et al., 2014;
Pﬁsterer et al., 2019; Freitas, 2019; Xanthopoulos et al., 2020).
Interpreting AutoML systems can be categorized into interpreting the resulting ML model on the underlying dataset, and interpreting the HPO process itself.
In this paper, we focus on the latter aspect.
Let c : Λ → R be a black-box cost function that maps a hyperparameter conﬁguration
λ
=
( λ 1 , ..., λ d ) to the model error obtained by a learning algorithm with conﬁguration λ .
The goal of HPO is to ﬁnd λ ∗ ∈ arg min λ ∈ Λ c ( λ ).
Throughout the paper, we assume that a (probabilistic) surrogate model ˆ c : Λ → R is given as an approximation to c .
Partial Dependence for Hyperparameters.
Let S ⊂ { 1 , 2 , ..., d } denote an index set of features, and let C = { 1 , 2 , ..., d } \ S be its complement.
The PDP (Friedman, 2001) of ˆ c : Λ → R for a sample (cid:16) λ ( i ) C (cid:17) [Equation] ...,n ∼ P ( λ C ) and hyperparameter(s) S is deﬁned as
[Equation]
with ˆ m : Λ → R denoting the posterior mean.
For a ﬁxed i , ˆ m (cid:16) λ S , λ ( i ) C (cid:17) : Λ S → R is called the i -th individual conditional expectation (ICE) curve (Goldstein et al., 2015).
When analyzing the PDP of hyperparameters, we are usually interested in how their values λ S impact model performance uniformly across the hyperparameter space.
Therefore, we assume P to be the uniform distribution over Λ C .
Uncertainty Quantiﬁcation in PDPs.
Uncertainty of PDPs provides additional information about the reliability of the mean estimator.
Hutter et al.
(2014) quantify the model uncertainty speciﬁcally for random forests as surrogate models in BO by calculating the standard deviation of the marginal predictions of the individual trees.
Cafri and Bailey (2016) suggest a bootstrap approach for tree ensembles to quantify the uncertainties of eﬀects based on PDPs.
Greenwell (2017) implemented a method that marginalizes over the mean ± standard deviation of the ICE curves for each grid point.
A model-agnostic estimate based on uncertainty estimates for probabilistic models has not been proposed so far.
Subgroup PDPs.
Recently, some researchers started to focus on ﬁnding more reliable
PDP estimates within subgroups of observations.
Molnar et al.
(2020), for example, focus on problems in PDP estimation when features are correlated.
Gr¨omping (2020) looks at the same problem and also uses subgroup PDPs, where ICE curves are grouped regarding a correlated feature.
Britton (2019) applied a clustering approach to group ICE curves to ﬁnd interactions between features.
However, none of the approaches aim at ﬁnding subgroups where reliable PDP estimates have low uncertainty.
Also to the best of our knowledge, nothing similar exists for analyzing experimental data created by HPO algorithms.
3. Biased Sampling in HPO
PDPs for the marginal eﬀect of hyperparameters of surrogate models can be misleading.
Here, we show that this problem is due to the sampling and the resulting model bias.
Eﬃcient optimizers tend to exploit promising regions of the hyperparameter space while other regions are less explored.
Consequently, predictions of surrogate models are more accurate with less uncertainty in well-explored regions and less accurate with high uncertainty in under-explored regions.
This model bias aﬀects the PD estimate (see Figure 1).
ICE curves may be biased and less conﬁdent if they are computed in poorly learned regions.
Assuming uniformly distributed hyperparameters, poorly learned regions are incorporated in the PD estimate with the same weight as well-learned regions.
ICE curves belonging to regions with high uncertainty may obfuscate well-learned eﬀects of ICE curves belonging to other regions when they are aggregated to a PDP.
Hence, the model bias may also lead to a less reliable PD estimate.
PDPs visualizing only the mean estimator of Eq.
(1) do not provide insights into the reliability of the PD estimate and how it is aﬀected by the described model bias.
4. Quantifying Uncertainty in PDPs
Pointwise uncertainty estimates of a probabilistic model provide insights into the reliability of the prediction ˆ c ( λ ) for a speciﬁc conﬁguration λ .
This uncertainty directly correlates with how explored the region around λ is.
Hence, including the model’s uncertainty structure into the PD estimate enables a user to understand in which regions the PDP is more reliable and which parts of the PDP have to be interpreted with caution.
We now extend the PDP of Eq.
(1) to probabilistic surrogate models ˆ c (e.g., a GP).
Let λ S be a ﬁxed grid point and (cid:16)
λ ( i ) C (cid:17) i
=1 ,...,n ∼ P ( λ C ).
The vector of predicted performances at the grid point λ S is ˆ c ( λ S ) =
(cid:16) ˆ c (cid:16) λ S , λ
( i ) C (cid:17)(cid:17) i =1 ,...,n with (posterior) mean ˆ m ( λ S ) :
=
(cid:16) ˆ m (cid:16) λ S , λ ( i ) C (cid:17)(cid:17) i =1 ,...,n and a (posterior) covariance ˆ K ( λ S ) :
= (cid:16) ˆ k (cid:16)(cid:16) λ S , λ ( i ) C (cid:17) , (cid:16) λ S , λ ( j ) C (cid:17)(cid:17)(cid:17) i,j =1 ,...,n .
Thus, ˆ c S ( λ S ) = 1 n (cid:80) ni
=
ˆ c (cid:16) λ S , λ ( i ) C (cid:17) is a random variable itself.
The variance of ˆ c S ( λ S ) is ˆ
[Equation]
It is important to ensure that the kernel is correctly speciﬁed.
Eq.
(2) can be approximated empirically by treating the pairwise covariances as unknown, i.e.:
[Equation]
In Appendix A.2, we show that this is less sensitive to misspeciﬁcations in the kernel.
Note that the variance estimate and the mean estimate can also be applied to other probabilistic models, such as GAMLSS, transformation trees, or a random forest.
5. Partial Dependence Plots on Sub-regions
As discussed in Section 3, (eﬃcient) optimization may imply that the design is biased, which in turn can produce misleading analyses when IML methods are naively applied.
We now aim to identify sub-regions Λ (cid:48)
⊂ Λ of the hyperparameter space in which the PDP can be estimated with high conﬁdence, and separate those from sub-regions in which it cannot be estimated reliably.
We identify sub-regions in which poorly-learned eﬀects do not obfuscate the well-learned eﬀects along each grid point, allowing the user to draw conclusions more conﬁdently in these locations.
By splitting the entire hyperparameter space in disjoint and interpretable sub-regions which can be visualized and interpreted individually but also aggregated to a global view, the user gets a better understanding of how the optimazation and exploration of the sampling process inﬂuences hyperparameter eﬀects.
Thus, to combine these requirements, we introduce a simple, but interpretable tree-based partitioning procedure which splits the entire hyperparameter space in disjoint and interpretable sub-regions in such a way that we receive more conﬁdent PDP estimates for well-explored regions and less reliable estimates in under-explored regions.
This procedure can also be visualized via a tree structure as shown in Figure 2 for one partitioning step.
The PD estimate on the entire hyperparameter space Λ is computed by drawing the sample used for the Monte Carlo estimate ( λ ( i ) C ) i ∈N ∼ P ( λ C ), N := { 1 , 2 , ..., n } .
The PD estimate on a sub-region Λ (cid:48) ⊂ Λ will be approximated w.r.t.
( λ ( i ) C )
i ∈N (cid:48) with N (cid:48)
= { i ∈
N
} λ ( i ) ∈ Λ (cid:48) only.
Since we are interested in the marginal eﬀect of the hyperparameter(s) S at each λ S ∈ Λ S we will visualize the PD for the whole range Λ S .
Thus, we want to have sub-regions of the form Λ (cid:48)
=
Λ S × Λ (cid:48)
C with Λ (cid:48)
C ⊂ Λ C .
This corresponds to the average over the ICE curves i ∈ N (cid:48) .
The criterion to evaluate a speciﬁc partitioning is based on the idea of grouping ICE curves with similar uncertainty structure.
We evaluate the impurity of a PD estimate on a sub-region Λ (cid:48) by the associated set of observations N (cid:48)
= { i ∈ N } λ ( i ) C ∈ Λ (cid:48) C , also referred to as nodes, as follows:
For each grid point λ S , we use the L2 loss in L ( λ S , N (cid:48) ) to evaluate how uncertainty varies across all ICE estimates i ∈ N (cid:48) using ˆ s 2 S |N (cid:48) ( λ S ) :
= 1 |N (cid:48) | (cid:80) i ∈N (cid:48)
ˆ s 2 (cid:16) λ S , λ ( i ) C (cid:17) and aggregate over all grid points in R L 2 ( N (cid:48) ):
L (cid:0) λ S , N (cid:48) (cid:1) =
(cid:88) i ∈N (cid:16) ˆ s 2 (cid:16) λ S , λ
( i ) C (cid:17) − ˆ
s 2 S |N (cid:48) ( λ S ) (cid:17) 2 and R L 2 ( N (cid:48) )
= (cid:88)
G g =1 L ( λ ( g ) S , N (cid:48) ) .
The partitioning based on CART (Breiman et al., 1984) is described in Appendix B.
[Figure]
The chosen impurity measure of
Eq.
(4) measures the pointwise L 2distance between ICE curves of the variance function ˆ s 2 ( λ S , λ
( i ) C ) and its PD estimate ˆ s 2 S |N (cid:48)
( λ S ) within a sub-region
N (cid:48) .
It follows that the splits are based on ICE curve similarities regarding the uncertainty measure ˆ s 2 S .
This seems reasonable, as ICE curves in well-explored regions of the search space should, on average, have a lower uncertainty than the ones in less-explored regions.
In particular, we expect the PDP estimate in the sub-regions associated with low costs c (and thus high relevance for a user) to be more conﬁdent in well-explored regions of λ S .
This is illustrated in Figure 2 where the conﬁdence band in the left sub-region decreased compared to the conﬁdence band of the global PDP especially for grid points close to the optimal conﬁguration of λ S .
We argue that ICE curves of the variance function are on average lower and show a similar trend for well-explored regions of the Λ.
Thus, by splitting according to curve similarities, we receive at least one interpretable sub-region (containing the optimal conﬁguration) where the PDP can be interpreted more reliably.
6. Experimental Analysis
In this section, we show that the proposed tree-based partitioning procedure allows us to identify an interpretable sub-region that yields a more reliable and conﬁdent PDP estimate.
We investigate HPO in the context of a surrogate benchmark (Eggensperger et al., 2015) based on the LCBench data (Zimmer et al., 2021), incl.
2000 randomly sampled conﬁgurations of the small Auto-PyTorch space on 35 datasets.
For each dataset, we trained a random forest as empirical performance model which predicts the balanced validation error achieved by Auto-Pytorch Tabular for a given conﬁguration.
These surrogates serve as objective that is optimized via BO.
By design, the objective is cheaper to evaluate and allows to infer a ground-truth.
For each task, we ran BO to obtain the optimal hyperparameters and computed PDPs on the ﬁnal surrogate model (cf.
Appendix C.1 for more details).
We measure the reliability of a PDP, i.e., the degree to which a user can rely on the estimate of the PD estimate, by comparing it to the true PD c S ( λ S ) computed on the true objective function c .
More speciﬁcally, for every grid point λ ( g ) S , we compute the negative log-likelihood (NLL) of c S ( λ S ) under the distribution of ˆ c S ( λ S ) pointwise for every grid point λ ( g ) S .
We measure the conﬁdence by assessing ˆ s S ( λ S ) pointwise for every grid point.
We consider the mean conﬁdence (MC) across all grid points 1 G (cid:80) Gg =
1 ˆ s (cid:16) λ ( g ) S (cid:17) as well as the conﬁdence at the grid point closest to ˆ λ S (OC), with ˆ λ being the best conﬁguration evaluated by the optimizer.
When evaluating the performance of the tree-based partitioning, we report the above metrics on that sub-region which contains the best conﬁguration evaluated by the optimizer, assuming that this region is of particular interest for a user of HPO.
We compared the PDP in sub-regions after 6 splits with the global PDP by computing the relative improvement in conﬁdence (MC and OC) and NLL of the sub-regional PDP compared to those metrics for the global PDP.
The MC of the PDPs is on average reduced by 30% to 52%, depending on the hyperparameter (cf. Table 1).
At the optimal conﬁguration ˆ λ S the improvement even increases by 50% − 62%.
PDP estimates for all hyperparameters are on average clearly more conﬁdent in the relevant sub-regions when compared to the global ones, especially in the region around the optimal conﬁguration ˆ λ S .
Furthermore, NLL even improves while the MC decreases.
A more detailed overview on dataset level and a benchmark with other impurity measures can be found in Appendix C.2.
7. Conclusion
[Table] We showed that PD estimates for surrogate models ﬁtted on experimental data generated by HPO algorithms can be unreliable due to an underlying sampling bias.
We extended PDPs by an uncertainty estimate to provide the user with more information regarding the reliability of the mean estimator.
We introduced a tree-based partitioning approach for PDPs where we leverage the uncertainty estimator to decompose the hyperparameter space into interpretable, disjoint sub-regions.
Our experiments show that we generate, on average, more conﬁdent and reliable PD estimates in the sub-region containing the optimal conﬁguration compared.
While we mainly discussed GP surrogates on a numerical hyperparameter space, our methods are applicable to all kinds of distributional regression models and for mixed hyperparameter spaces.
In future, we will study our method on more complex, hierarchical conﬁguration spaces for neural architecture search. Acknowledgements
This work has been partially supported by the German Federal Ministry of Education and Research (BMBF) under Grant No. 01IS18036A.
The authors of this work take full responsibilities for its content.
Appendix A. Uncertainty Estimation A.1 Choice of Uncertainty Quantiﬁcation
Besides using the uncertainty estimate of the surrogate model to quantify the uncertainty for the PDP mean estimate (our method), it is also possible to estimate uncertainty w.r.t.
the variance over diﬀerent ICE curves (Greenwell, 2017).
However, if the uncertainty was estimated via computing the variance over ICE curves, we describe how the levels of the mean prediction vary along the λ S dimensions.
In contrast, we propose to capture model uncertainty along the λ S dimensions.
For example, consider a constant surrogate function ˆ c ( λ ) =
γ with high uncertainty estimation ˆ s 2 ( λ ) = 100.
Computing the variance over
ICE curves on this example will result in an uncertainty estimate of 0 (all ICE curves are identical).
Our method, however, would return a variance estimate of 100 and thus capture model uncertainty.
A.2 Covariance Estimates under Misspeciﬁcation of Kernels
In order to provide evidence for the claim that estimate (2) is more sensitive to misspeciﬁcations in the kernel (and thus in the covariance structure) than (3), we performed some prior experiments.
We assume that we are given an objective function that is generated by a Gaussian process (GP) with a Mat´ern-3/2 kernel.
In our experiments, that function was created by ﬁtting a GP on tuples (cid:0) λ
( i ) , y ( i ) (cid:1) i =1 ,..., 30 , with λ ( i ) ∼ Unif (cid:0) [ − 5 , 5] d (cid:1) and y ( i ) corresponding to the value of the d -dimensional Styblinski Tang function for λ ( i ) .
The posterior mean of this GP will further serve as our true objective c to pretend that we know the correct kernel speciﬁcation of the ground-truth.
Subsequently, we ﬁt both a GP surrogate model with correctly speciﬁed kernel (i.e., a
Mat´ern-3/2 kernel) and a surrogate model with a misspeciﬁed kernel (in our case, we chose a Gaussian kernel) to the data (cid:0)
λ ( i ) , c (cid:0) λ
( i ) (cid:1)(cid:1)
i =1 ,..., 30 .
In both cases, we compute the partial dependence plots (PDPs) for λ 1 with both variance estimates (2) and (3) and measure the negative log-likelihood of c S under the respective estimated PDP.
We performedrepetitions of the experiments for d ∈ { 3 , 5 , 8 } , respectively.
Figure 3 shows that the median of the NLL across all 50 replications is slightly lower for the covariance estimate (2).
However, the variance of the NLL is much higher for estimate (2) as compared to (3).
Table 2 conﬁrms that, when using variance estimate (2), the standard deviation of the NLL values is lower.
We conclude that the reliability of the estimate is particularly sensitive to a correct choice of the kernel function.
The NLL for the PDPs computed with variance estimate (3) is - independent of whether the kernel is correctly speciﬁed or not - less sensitive to misspeciﬁcations in the kernel.
Appendix B. Tree-based Partitioning Algorithm
The steps to partition the hyperparameter space Λ into two sub-regions based on an i.i.d.
uniformly distributed dataset ( λ ( i ) C ) i ∈N ∈ Λ C , N = { 1 , ..., n } , are shown in Algo.
The procedure is based on the CART splitting algorithm introduced by Breiman et al.
It is recursively repeated until a user-deﬁned stopping criterion is met (e.g., a maximum number of splits, a minimum size of a region, or a minimum improvement in each node).
For
[Figure]
[Table]
the leaf nodes, we calculate PDP and uncertainty estimate within the regarded sub-region by aggregating the respective ICE curves belonging to those nodes.
The identiﬁed sub-regions belonging to the leaf nodes can be deduced from split variables and split points in every step.
To receive a more conﬁdent PDP mean estimate, we would like to group those ICE curves that represent the same ranges of λ S well by exhibiting low and similar uncertainty in this range.
This is shown in Figure 4, where samples with high uncertainty over the entire range of λ S are grouped together (right sub-region).
Samples with low uncertainty close to the optimal conﬁguration of λ S and increasing uncertainties for less suitable conﬁgurations are grouped together by curve similarities in the left sub-region.
Number of Splits
Which PDPs are most interesting to look at depends on the question the user would like to answer.
If the main interest lies in understanding the optimization and exploring the sampling process, a user might want to keep the number of sub-regions relatively low by performing only a few partitioning steps.
Subsequently, one would investigate the overall structure of the sub-regions and the individual sub-regional PDPs.
If the user is more interested in interpreting hyperparameter eﬀects only in the most relevant sub-region(s) around the best conﬁgurations evaluated by the optimizer, a user potentially may want to split deeper and only look at sub-regions that are more conﬁdent than the global PDP.
input:
N output:
N t ∗ l and N t ∗ r for j ∈ C do for Every split t on hyperparameter λ
j do
N tl = { i ∈ N }
[Equation]
t
N tr = { i ∈ N } λ ( i
[Equation]
>t
I ( t ) =
R L 2 ( N tl )
+ R L 2 ( N tr ) end for
Choose t ∗ λ j ∈ arg
min t I ( t ) end for
Appendix C. Experimental Analysis C.1 Experimental Design
All experiments only require CPUs (and no GPUs) and were computed on a Linux cluster with 28-way Haswell-EP nodes, 1 core per node, and a memory limit of 2.2 GB per node.
The computational complexity of the PDP estimation with uncertainty is O ( G · n ) ·
O (ˆ c ), with O (ˆ c ) being the runtime complexity of single surrogate prediction, n denoting the size of the dataset to compute the Monte Carlo estimate and G being the number of grid points.
In the context of HPO, the general assumption is that evaluation time of ˆ c is negligibly low as compared to evaluation c .
So we argue that the runtime complexity of computing a PDP with uncertainty estimate can be neglected in this context.
When computing ICE curves and their variance estimates beforehand, the algorithmic complexity of Algorithmcorresponds to the algorithmic complexity of the tree splitting (Breiman et al., 1984).
In our experiments, the runtimes to compute the PDPs and perform the tree splitting lies within a few minutes.
We consider them to be negligible and will thus not report these.
All experimental data was downloaded from the LCBench project 2 .
As empirical performance model, we ﬁtted a random forest (ranger) to approximate the relationship between hyperparameters and balanced error rate (BER).
For every dataset, we performed a random search with 500 iterations and evaluation via 3-fold cross validation to choose reasonable for the hyperparameters represented in Table 3.
The empirical performance model acts as ground-truth in our experiments, and thus, we denote it by c .
This function was used to compute the true PDP c S .
We computed an initial random design of size 2 · d 3 .
We performed BO with a GP surrogate model with a Mat´ern-3/2 kernel and the LCB acquisition function a ( λ ) =
ˆ m ( λ ) + τ · ˆ s ( λ ) with diﬀerent values τ ∈ { 1 } .
A nugget eﬀect was modeled.
The maximum budget per BO run was set to 200 objective function evaluations.
We denote the best evaluated conﬁguration, measured by ˆ c , by ˆ λ .
[Table] [Table]
Based on the last surrogate model, we performed the partitioning in Algorithm 1 for a total number of 6 splits, with the diﬀerent splitting criteria (see Section C.2), with PDPs being computed with a G = 20 equidistant grid points, and n = 1000 samples for the Monte
Carlo approximation 4 .
All PDPs are computed with regards to a single feature, for a grid of G = 20 equidistant points, and the Monte Carlo estimate is computed with n = 1000 samples.
We illustrate the conﬁdence of a PDP by the width of its conﬁdence bands ˆ m S ( λ S ) ± q 1 − α/ 2 · ˆ s S ( λ S ), with q 1 − α/ 2 denoting the (1 − α/ 2)-quantile of a standard normal distribution.
C.2 Detailed Results
In Section 6 we evaluated the reliability of PDP estimation for the partitioning procedure proposed in Section 5.
The results presented in Section 5 are aggregated over a total number of 35 diﬀerent datasets.
In Tables 5 and 6 the relative improvement of the mean conﬁdence (MC) and negative loglikelihood (NLL) are presented on dataset level.
The mean and standard deviation are averaged over all hyperparameters.
Furthermore, the mean values of the features providing the highest and lowest relative improvement for each dataset are reported.
Following on that, Table 7 shows for each hyperparameter the the number of datasets for which the respective hyperparameter led to the highest (lowest) relative improvement for both evaluation metrics.
[Table]
To further study our suggested method, we now highlight a few individual experiments.
We chose one iteration of the shuttle dataset.
In the upper panel of Figure 5, we see that the true PDP estimate for max.
number of units is decreasing, while the globally estimated
PDP trend is increasing, and thus misleading.
Although the conﬁdence band already signals that the PDP cannot be reliably interpreted on the entire hyperparameter space, it remains challenging to draw any conclusions from it.
After performing 6 splits, we receive a conﬁdent and reliable PD estimate on an interpretable sub-region.
The same plots are depicted for the hyperparameter batch size in the lower
Dataset µ σ µ h µ l
[Table]
25 airlines   3 albert
[Table] [Table]
22 Australian
[Table] [Table]
13 car   32 christine
[Table]
7 connect
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
Dataset µ σ µ h µ l adult   8 airlines
[Table]
panel of Figure 5.
This example illustrates that the conﬁdence band might not always shrink uniformly over the entire range of λ S during the partitioning, but often particularly around the optimal conﬁguration ˆ λ S .
Split Criteria
In Section 5, we introduced Eq.
4 as split criteria within the tree-based partitioning of Algorithm 1.
This measure is based on splitting ICE curves based on curve similarities, which is especially suitable in the underlying context as explained in Section 5.
However, we also compared it to two other measures that are based on uncertainty estimates provided by the probabilistic surrogate model.
The ﬁrst one is also based on ICE curves of the variance function ˆ s 2 ( λ S , λ
( i ) C ) and its PD estimate ˆ s 2 S |N (cid:48)
( λ S ) within a sub-region
N (cid:48) .
However, instead of minimizing the distance between curves and group the associated
ICE curves regarding similar behavior, we can also minimize the area under ICE curves of the variance function.
The reasoning for this is as follows: If we aim for tight conﬁdence bands over the entire range of Λ S , we want the ICE curves of the variance function to be - on average - as low as possible.
This is equivalent to minimizing the average area under
ICE curves of the variance function.
Thus, the calculation of Eq.
4 changes such that we ﬁrst calculate the average area between each ICE curve of the uncertainty function and the respective sub-regional PDP
[Equation]
where ˆ s 2 S |N (cid:48)
[Equation]
λ
[Equation]
=
[Equation]
λ
[Equation]
λ
[Equation]
(cid:17) , and aggregate the quadratic value of it over all observations in the respective sub-region:
[Equation]
Second, we also used the uncertainty estimates of the probabilistic surrogate model for each observation of the testdata itself to deﬁne an impurity measure.
Therefore we calculated the squared deviation of each observation to the mean uncertainty within the respective node.
Hence, compared to the other two approaches, we do not group curves but the observations itself regarding their uncertainty.
We further refer to this approach as the variance (var) approach.
As a third measure that is not based on the uncertainty estimates
[Table]
we used the MSE of the posteriori mean estimate of the surrogate model as split criterion.
This is the most commonly used measure for regression trees and hence a solid baseline measure.
We compared the four impurity measures for the partitioning procedure over all datasets and hyperparameters.
We compare the results that we presented in Section 6 with the according results for the other three measures in Table 8.
The impurity measure based on curve similarities that we used for our analysis (L2) outperforms the other three measures on average for all hyperparameters regarding MC and especially regarding OC.
With regards to
NLL there is not one measure which outperforms all others, but rather all measures perform on average over all hyperparameters equally good
[Table]
Increased conﬁdence with more splits
Furthermore, it needs to be noted that by using our method the mean conﬁdence and NLL improve on average if we use six splits.
However, this does not mean that they improve by design when splitting into sub-regions.
As shown in Tables 5 and 6, improvements heavily depend on dataset and HP.
Diﬀerent factors inﬂuence the optimal number of splits, such as the sampling bias, size of the test-set and dimensionality of the HP space.
For some of our benchmarks, the best results are reached with fewer splits, as shown in Figure 6 where improvements in both metrics are made until
Split 2 and by splitting deeper, estimates get less accurate especially when sample sizes in sub-regions become very small.
Thus, the number of splits is a (useful and ﬂexible) control parameter in our method which can be determined within a human-in-the-loop approach (view plots after each split and stop when results are satisfying) or by deﬁning a quantitative measure (e.g. based on a threshold for conﬁdence improvement).
Appendix D. Code
All code related to this paper is made available via an anonymous repository 5 .
All methods are implemented within the folder R , and all code used to perform the experiments are provided in benchmarks .
All analyses shown in this paper in form of tables or ﬁgures can be reproduced via running the notebooks in analysis .
https://anonymous.4open.science/repository/a71006d4-e8df-475e-9848-03786f00bf99/README.
md