E XPLANATION U NCERTAINTY WITH D ECISION B OUNDARY A WARENESS
Anonymous authors
Paper under double-blind review
A BSTRACT
Post-hoc explanation methods have become increasingly depended upon for understanding black-box classiﬁers in high-stakes applications, precipitating a need for reliable explanations.
While numerous explanation methods have been proposed, recent works have shown that many existing methods can be inconsistent or unstable.
In addition, high-performing classiﬁers are often highly nonlinear and can exhibit complex behavior around the decision boundary, leading to brittle or misleading local explanations.
Therefore, there is an impending need to quantify the uncertainty of such explanation methods in order to understand when explanations are trustworthy.
We introduce a novel uncertainty quantiﬁcation method parameterized by a Gaussian Process model, which combines the uncertainty approximation of existing methods with a novel geodesic-based similarity which captures the complexity of the target black-box decision boundary.
The proposed framework is highly ﬂexible; it can be used with any black-box classiﬁer and feature attribution method to amortize uncertainty estimates for explanations.
We show theoretically that our proposed geodesic-based kernel similarity increases with the complexity of the decision boundary.
Empirical results on multiple tabular and image datasets show that our decision boundary-aware uncertainty estimate improves understanding of explanations as compared to existing methods.
I NTRODUCTION
Machine learning models are becoming increasingly prevalent in a wide variety of industries and applications.
In many such applications, the best performing model is opaque; post-hoc explainability methods are one of the crucial tools by which we understand and diagnose the model’s predictions.
Recently, many explainability methods, termed explainers , have been introduced in the category of local feature attribution methods.
That is, methods that return a real-valued score for each feature of a given data sample, representing the feature’s relative importance with respect to the sample prediction.
These explanations are local in that each data sample may have a different explanation.
Using local feature attribution methods therefore helps users better understand nonlinear and complex black-box models, since these models are not limited to using the same decision rules throughout the data distribution.
Recent works have shown that existing explainers can be inconsistent or unstable.
For example, given similar samples, explainers might provide different explanations (Alvarez-Melis & Jaakkola, 2018;
Slack et al., 2020).
When working in high-stakes applications, it is imperative to provide the user with an understanding of whether an explanation is reliable, potentially problematic, or even misleading.
A way to guide users regarding an explainer’s reliability is to provide corresponding uncertainty quantiﬁcation estimates.
One can consider explainers as function approximators; as such, standard techniques for quantifying the uncertainty of estimators can be utilized to quantify the uncertainty of explainers.
This is the strategy utilized by existing methods for producing uncertainty estimates of explainers (Slack et al., 2021; Schwab & Karlen, 2019).
However, we observe that for explainers, this is not sufﬁcient; because in addition to uncertainty due to the function approximation of explainers, explainers also have to deal with the uncertainty due to the complexity of the decision boundary (DB) of the blackbox model in the local region being explained.
[Figure]
Consider the following example: we are using a prediction model for a medical diagnosis using two features, level of physical activity and body mass index (BMI) (Fig. 1).
In order to understand the prediction and give actionable recommendations to the patient, we use a feature attribution method to evaluate the relative importance of each feature.
Because of the nonlinearity of the prediction model, patients A and B show very similar symptoms, but are given very different recommendations.
Note that while this issue is related to the notion of explainer uncertainty, measures of uncertainty that only consider the explainer would not capture this phenomenon.
This suggests that any notion of uncertainty is incomplete without capturing information related to the local behavior of the model.
Therefore, the ability to quantify uncertainty for DB-related explanation instability is desirable.
We approach this problem from the perspective of similarity: given two samples and their respective explanations, how closely related should the explanations be?
From the previous intuition, we deﬁne this similarity based on a geometric perspective of the DB complexity between these two points.
Speciﬁcally, we propose a novel geodesic-based kernel similarity metric, which we call the Weighted
Exponential Geodesic (WEG) kernel.
The WEG kernel encodes our expectation that two samples close in Euclidean space may not actually be similar if the DB within a local neighborhood of the samples is highly complex.
Using our similarity formulation, we propose the G aussian P rocess E xplanation Un C ertainty (GPEC) framework, which is an instance-wise, model-agnostic, and explainer-agnostic method to quantify the uncertainty of explanations.
The proposed notion of uncertainty is complementary to existing quantiﬁcation methods.
Existing methods primarily estimate the uncertainty related to the choice in model parameters and ﬁtting the explainer, which we call function approximation uncertainty , and does not capture uncertainty related to the DB.
GPEC can combine the DB-based uncertainty with function approximation uncertainty derived from any local feature attribution method.
In summary, we make the following contributions:
• We introduce a geometric perspective on capturing explanation uncertainty and deﬁne a novel geodesic-based similarity between explanations.
We prove theoretically that the proposed similarity captures the complexity of the decision boundary from a given black-box classiﬁer.
• We propose a novel Gaussian Process-based framework that combines A) uncertainty from decision boundary complexity and B) explainer-speciﬁc uncertainty to generate uncertainty estimates for any given feature attribution method and black box model.
• Empirical results show GPEC uncertainty improves understanding of feature attribution methods.
R ELATED W ORKS
Explanation Methods.
A wide variety of methods have been proposed for the purpose of improving transparency for pre-trained black-box prediction models (Guidotti et al., 2018; Barredo Arrieta et al., 2020).
Within this category of post-hoc methods, many methods focus on local explanations, that is, explaining individual predictions rather than the entire model.
Some of these methods generate explanations through local feature selection (Chen et al., 2018; Masoomi et al., 2020).
In this [Figure] work, we focus primarily on feature attribution methods, which return a real-valued score for each feature in the sample.
For example, LIME (Ribeiro et al., 2016) trains a local linear regression model to approximate the black-box model.
Lundberg & Lee (2017) generalizes LIME and ﬁve other feature attribution methods using the SHAP framework, which fulﬁll a number of desirable axioms.
While some methods such as LIME and its SHAP variant, KernelSHAP, are model-agnostic, others are designed for speciﬁc model architectures, such as neural networks (Bach et al., 2015; Shrikumar et al., 2017; Sundararajan et al., 2017; Erion et al., 2021), tree ensembles (Lundberg et al., 2020), or Bayesian neural networks (Bykov et al., 2020), taking advantage of those speciﬁc architectures.
Another class of method involves the training of separate surrogate models to explain the black-box model (Dabkowski & Gal, 2017; Chen et al., 2018; Schwab & Karlen, 2019;
Guo et al., 2018;
Jethani et al., 2022).
Once trained, surrogate-based methods are typically very fast; explanation generation generally only requires a single inference step from the surrogate model.
Explanation Uncertainty.
One option for improving the trustworthiness of explainers is to quantify the associated explanation uncertainty.
Bootstrap resampling techniques have been proposed as a way to estimate uncertainty from surrogate-based explainers (Schwab & Karlen, 2019; Schulz et al., 2022).
Guo et al.
(2018) also proposes a surrogate explainer parameterized with a Bayesian mixture model.
Alternatively, Bykov et al.
(2020) and Patro et al. (2019) introduce methods for explaining Bayesian neural networks, which can be transferred to non-Bayesian neural networks.
Covert & Lee (2021) derives an unbiased version of KernelSHAP and investigates an efﬁcient way of estimating its uncertainty.
Zhang et al. (2019) categorizes different sources of variance in LIME estimates. Several methods also investigate LIME and KernelSHAP in a Bayesian context; for example calculating a posterior over attributions (Slack et al., 2021), investigating the use of priors for explanations (Zhao et al., 2021), or using active learning to improve sampling (Saini & Prasad, 2022).
However, all existing methods for quantifying explanation uncertainty only consider the uncertainty of the explainer as a function approximator.
This work introduces an additional notion of uncertainty for explainers that takes into account the uncertainty of the explainer due to the DB.
U NCERTAINTY Q UANTIFICATION FOR B LACK -B OX E XPLAINERS
We now outline the GPEC framework, which we parameterize with a Gaussian Process (GP) regression model.
We deﬁne a vector-valued GP which is trained on data samples as input and explanations as labels.
More concretely, consider a data sample x 2 X ⇢ R d that we want to explain in the context of a black-box prediction model F : X !
[0 , 1] , in which the output corresponds to the probability for the positive class.
For simplicity, we consider the binary classiﬁer case; we extend to the multiclass case in Section C.3.
We apply any given feature attribution method E : X !
R s , where s is the dimension of the explanations e = E ( x ) .
Let us draw samples x 1 , . . .
, x M from the data distribution and generate their respective explanations e 1 , . . .
, e M .
We assume that each explanation e m is generated from an unobserved latent function E plus some independent Gaussian noise ⌘ m .
[Equation]
where k ( · , · ) is the speciﬁed kernel function and ⌧ m > 0 .
We decompose each explanation into two components, E ( x m ) and ⌘ m , which represent two separate sources of uncertainty: 1) a decision boundary-aware uncertainty which we capture using the kernel similarity, and 2) a function approximation uncertainty from the explainer.
After specifying E ( x m ) and ⌘ m , we can combine the two sources by calculating the posterior predictive distribution for the GP model; we take the 95% conﬁdence interval of this distribution to be the GPEC uncertainty estimate.
Due to space constraints, we provide an overview of GP regression in App.
B.2.
Function Approximation Uncertainty.
The ⌘ m component of Eq.
1 represents the uncertainty stemming from explainer speciﬁcation.
For example, this uncertainty can be captured from increased variance from undersampling (perturbation-based explainers) or from the increased bootstrap resampling variance from model misspeciﬁcation (surrogate-based explainers).
We can therefore deﬁne ⌧ based on the chosen explainer.
Explainers that include some estimate of uncertainty (e.g. BayesLIME, BayesSHAP, CXPlain) can be directly used to estimate ⌧ .
For example, CXPlain returns a distribution of feature importance values for a sample x m ; the variance of this distribution can be directly used as the variance of ⌘ m .
For other stochastic explanation methods that do not explicitly estimate uncertainty, we can estimate ⌧ empirically by resampling explanations for the same data sample:
[Equation]
where each E i ( x m ) is a sampled explanation for x m .
Alternatively, for deterministic explanation methods we can omit the noise and assume that the generated explanations are noiseless.
Decision Boundary-Aware Uncertainty.
In contrast, the E ( x m ) component of Eq.
1 draws possible functions from the GP prior that could have generated the observed explanations, with the function space deﬁned by the choice of kernel.
Intuitively, the kernel encodes our a priori deﬁnition of how similar two explanations should be given the similarity of the inputs.
In other words, how much information do we expect a given point x to provide for a nearby point x 0 with respect to their explanations?
We use this kernel speciﬁcation to deﬁne a source of uncertainty dependent on the behavior of nearby explanations.
In particular, we consider a novel kernel formulation that reﬂects the complexity of the DB in a local neighborhood of the samples; this is detailed in Section 4.
For any given kernel, we can interpret the distribution of possible functions E ( x m ) as an estimate of uncertainty based on the set of previously observed or sampled explanations.
W EIGHTED E XPONENTIAL G EODESIC K ERNEL
Intuitively, the GP kernel encodes the assumption that each individual explanation gives some information about the explanations around it; the choice of kernel deﬁnes the neighborhood and magnitude of this shared information.
In the GP framework introduced in Section 3, the kernels deﬁne the relationship or similarity between two explanations E ( x ) and E ( x 0 ) based soley on some function of their inputs x and x 0 .
For example, stationary kernels assign the same similarity for any two inputs regardless of where in the data manifold they are located.
Instead, we want to encode the assumption that when the DB is complex, knowing an explanation E ( x ) gives limited information about other nearby explanations (see Fig. 3).
4. G EOMETRY AND G EODESICS OF THE D ECISION B OUNDARY
We want to relate kernel similarity to the behavior of the black-box model; speciﬁcally, the complexity or smoothness of the DB.
Given any two points on the DB, the relative complexity of the boundary segment between them can be approximated by the segment length.
The simplest form that the DB can take is a linear boundary connecting the two points; this is exactly the minimum distance between the points.
As the complexity of the DB grows, there is a general corresponding increase in segment length.
X = (-3,3)
X = (-0.5,3)
X = (0.5,0)
X = (3,0)
R B F K e r n e l
W E G K e r n e l [Figure] Concretely, we deﬁne the DB as a ( d  1) -dimensional
Riemannian manifold embedded in R d .
Given the binary classiﬁer F , we deﬁne M = { x 2 R d : F ( x ) = 12 } representing the DB 1 of F .
Given this interpretation, we can deﬁne distances along the DB as geodesic distances in M .
[Equation]
The mapping  : [0 , 1] !
M , deﬁned such that  (0) = m
and  (1)
= m 0 , is a parametric representation of a 1-dimensional curve on M .
We can adapt geodesic distance in our kernel selection through the exponential geodesic (EG) kernel (Feragen et al., 2015), which is a generalization of the
Radial Basis Function (RBF) kernel substituting ` 2 distance with geodesic distance:
[Equation]
The EG kernel has been previously investigated in the context of Riemannian manifolds (Feragen et al., 2015; Feragen & Hauberg, 2016).
In particular, while prior work shows that the EG kernel fails to be positive deﬁnite for all values of  in non-Euclidean space, there exists large intervals of  > 0 for the EG kernel to be positive deﬁnite.
Appropriate values can be selected through grid search and cross validation; we assume that a valid value of  has been selected.
Note that the manifold we are interested in is the DB; applying the EG kernel on the data manifold (i.e. directly using k EG in the GPEC formulation) would not capture model complexity.
In addition, na¨ıvely using the EG kernel on the DB manifold would not capture the local complexity with respect to a given explanation; the similarity would be invariant to observed explanations.
We therefore need to relate the geodesic distances on M to samples in the data space X .
4. W EIGHTING THE D ECISION B OUNDARY
Consider a probability distribution p ( M ) with its support deﬁned over M .
We weight p ( M ) according to the ` 2 distance between M and a ﬁxed data sample x :
[Equation]
We evaluate the kernel function k ( x, x 0 ) by taking the expected value over the weighted distribution.
[Equation]
Our formulation is an example of a marginalized kernel (Tsuda et al., 2002): a kernel deﬁned on two observed samples x, x 0 and taking the expected value over some hidden variable.
Given that the underlying EG kernel is positive deﬁnite, it follows that the WEG kernel forms a valid kernel.
To evaluate the WEG kernel, we theoretically investigate two properties of the kernel.
Theoremshows that the WEG kernel is an extension of the EG kernel for data samples not directly on the DB.
Theorem 1.
Given two points x, x 0 2 M , then lim  !
1 k WEG ( x, x 0 ) =
k EG ( x, x 0 )
Proof details are in App.
C.1.
Intuitively, as we increase  , the manifold distribution closest to the points x, x 0 becomes weighted increasingly heavily.
At the limit, the weighting becomes concentrated entirely on the points x, x 0 themselves, which recovers the EG kernel.
Therefore we see that the WEG kernel is simply a weighting of the EG kernel, which is controlled by  .
Theorem 2 establishes the inverse relationship between DB complexity and WEG kernel similarity.
Given a black-box model with a piecewise linear DB, we show that this DB represents a local maximum with respect to WEG kernel similarity; i.e. as we perturb the DB to be nonlinear, kernel similarity decreases.
We ﬁrst deﬁne perturbations on DB.
Note that int ( S ) indicates the interior of a set S , f | S represents the function f restricted to S , and id indicates the identity mapping.
Deﬁnition 1.
Let { U ↵ } ↵ 2 I be charts of an atlas for a manifold P ⇢ R d , where I is a set of indices.
Let P and e
P be differentiable manifolds embedded in R d , where P is a Piecewise Linear (PL) manifold.
Let R : P !
e
P be a diffeomorphism.
We say e
P is a perturbation of P on the i th chart
if R satisﬁes the following two conditions.
There exists a compact subset K i [Equation]
R
[Equation] i ) =
id | P\ int ( K i ) and R | int[-( K i ) 
-]
( K i ) 6 = id | int ( K i ) .
There exists a linear homeomorphism between an open subset
[Equation]
Theorem 2.
Let P be a ( d – 1) -dimension PL manifold embedded in R d .
Let e
P be a perturbation of P and deﬁne ˜
k
( x, x 0 ) and k ( x, x 0 ) as the WEG kernel deﬁned on e
P and P respectively.
Then ˜ k
( x, x 0 )
<
k ( x, x 0 ) 8 x, x 0 2 R d .
Proof details are in App.
C.2.
Theorem 2 implies that, for any two ﬁxed points x, x 0 , their kernel similarity k WEG ( x, x 0 ) decreases as the black-box DB complexity increases.
Within GPEC, the explanations for x, x 0 become less informative for other nearby explanations and induce a higher explanation uncertainty estimate.
To improve the interpretation of the WEG kernel, we apply a normalization to scale the similarity values to be between [0 , 1] .
We construct the normalized kernel k ⇤ WEG as follows:
[Equation]
4. WEG K ERNEL A PPROXIMATION
In practice, the integral in Eq.
6 is intractable; we approximate the expected value using Monte
Carlo (MC) sampling with K samples:
[Equation]
Z m , Z m 0 are the normalization constants for q ( M | x, ⇢ ) and q ( M 0 | y, ⇢ ) , respectively.
We can similarly estimate these values using MC sampling:
[Equation]
GPEC Implementation Details.
Due to space constraints, the full GPEC training algorithm is detailed in Alg. 1 in App.
D. To use the GPEC framework, we ﬁrst train the GP regression model with a set of training samples.
Once the GP is trained, we can derive uncertainty estimates for test samples from the 95% conﬁdence interval of the posterior predictive distribution.
The WEG kernel requires additional setup time to use in the GP.
We ﬁrst calculate the EG kernel matrix by sampling the black-box DB, then weigh each element of the matrix per Eq. The estimation of [Figure] C
[Figure]
s G e r m a n C r e d i t [Figure] C e n s u s
[Figure]
a classiﬁer’s DB is an ongoing area of research; a number of methods have been proposed in the literature (e.g. Yan & Xu (2008); Karimi et al.
Once GPEC is trained, training cost is amortized during inference; estimating uncertainty for test samples using a GP generally has time complexity of O ( n 3 ) , which can be reduced to O ( n 2 ) using BBMM (Gardner et al., 2018), and further with variational methods (e.g., Hensman et al.
E XPERIMENTS
We evaluate how well GPEC captures 1) DB-aware uncertainty and 2) functional approximation uncertainty on a variety of datasets and classiﬁers.
In section 5.2 we compare the DB-aware uncertainty and approximation uncertainty components of GPEC.
Section 5.3 evaluates how GPEC captures DB complexity.
Section 5.4 evaluates how well GPEC combines sources of uncertainty.
Due to space constraints we have additional results in the appendix, including sensitivity analysis for GPEC parameters (F.5), execution time comparison F.1, and additional experiments on combining uncertainty (F.3).
All experiments were run on an internal cluster using AMD EPYC 7302 16-Core processors, and all source code will be made public.
5. E XPERIMENTAL S ETUP
Datasets and Models.
Experiments are performed on three tabular datasets (Census, Online Shoppers (Sakar et al., 2019), German Credit) from the UCI data repository (Dua & Graff, 2017) and two image datasets (MNIST (LeCun & Cortes, 2010) and fashion-MNIST (f-MNIST)
(Xiao et al., 2017)).
GPEC can be used with any black-box model; for our experiments we use XGBoost (Chen & Guestrin, 2016) with log-loss for the tabular datasets and a 4-layer Multi-Layered Perception (MLP) model for the image datasets.
Additional dataset details are outlined in App.
E.1.
Implementation Details.
For comparison purposes, we consider GPEC using two different kernels:
GPEC-WEG (WEG kernel) and GPEC-RBF (RBF Kernel).
Unless otherwise stated, we choose  = 1 . 0 and ⇢ = 0 .
1 (see App.
F.5 for experiments on parameter sensitivity).
For the Uncertainty
Visualization (Sec. 5.2) and Regularization Test (Sec. 5.3)
we use GPEC with the KernelSHAP explainer.
We train the GP parameterizing GPEC with BBMM (Gardner et al., 2018).
Samples from the DB are drawn using DBPS (Yan & Xu, 2008) for tabular datasets and DeepDIG (Karimi et al., 2019) for image datasets.
Geodesic distances are estimated using ISOMAP Tenenbaum et al.
Additional implementation details are available in App.
D.
[Figure]
Competing Methods.
We compare GPEC to three competing explanation uncertainty estimation methods.
BayesSHAP and BayesLIME (Slack et al., 2021) are extensions of KernelSHAP and
LIME, respectively, that ﬁt Bayesian linear regression models to perturbed data samples.
After ﬁtting, 95% credible intervals can be estimated by sampling the posterior distribution of the feature attributions.
CXPlain (Schwab & Karlen, 2019) trains a separate explanation model using a causalbased objective, and applies a bootstrap resampling approach to estimate explanation uncertainty.
We report the 95% conﬁdence interval from the set of bootstrapped explanations.
Unless otherwise stated, we use the default settings in the provided implementation for all three of these methods.
5. U NCERTAINTY V ISUALIZATION
In order to visualize explanation uncertainty, we train XGBoost binary classiﬁers using two selected features, which we take to be the black-box model.
Both GPEC and competing methods are used to quantify the uncertainty for the feature attributions of the test samples: a grid of 10,000 samples over the data domain.
Uncertainty estimates for the x-axis feature are plotted in Figure 4 (results for the y-axis variable are included in App.
F.4), where darker values in the heatmap indicate higher uncertainty.
GPEC training samples amortized uncertainty estimates distribution We ablate imation uncertainty component order to their respective
GPEC- shows that - nel attributes higher uncertainty to samples near nonlinearities DB.
In contrast, GPEC-RBF provides uncertainty estimates that relate only to the training samples; test sample uncertainty is proportional to distance from the training samples.
The competing methods BayesSHAP, BayesLIME, and CXPlain results in relatively uniform uncertainty estimates over the test samples.
CXPlain shows areas of higher uncertainty for Census, however the magnitude of these estimates are small.
The uncertainty estimates produced by these competing methods are unable to capture the properties of the black-box model.
5. R EGULARIZATION T EST
In this experiment we compare the average uncertainty of explanations as the model is increasingly regularized in order to assess the impact of restricting model complexity.
For XGBoost models, we vary the parameter  , which penalizes the number of leaves in the regression tree functions (Eq. 2 in
Chen & Guestrin (2016)).
For neural networks, we use two types of regularization.
First, we add an ` 2 penalty to the weights; this penalty increases as the parameter  increases.
Second, we change the
[Table]
ReLU activation functions to Softplus; a smooth approximation of ReLU with smoothness inversely proportional to a parameter  , which we vary (Dombrowski et al., 2019).
We observe in Table 1 that the average uncertainty estimate generated by GPEC-WEG decreases as regularization increases, suggesting that the uncertainty estimates reﬂect the overall complexity of the underlying blackbox model.
For the tabular datasets, the estimates for BayesSHAP, BayesLIME, and CXPlain stay relatively ﬂat.
Interestingly, the estimates from these methods decrease for the image datasets.
We hypothesize that the regularization for the neural network model also increases overall stability of the explanations.
GPEC can capture both the uncertainty from WEG kernel and also the estimated uncertainty from the function approximation for the explainer, which we demonstrate in Section 5.4.
5. C OMBINING A PPROXIMATION U NCERTAINTY
Using the WEG kernel with noisy explanation labels enables GPEC to incorporate uncertainty estimates from the explainer.
Explainers such as BayesSHAP provide an estimate of uncertainty for their explanations, which can be used as ﬁxed noise in the Gaussian likelihood for GPEC.
For explainers with no such estimate, we can empirically resample explanations for the same test sample and take the explanation variance as an estimate of uncertainty.
Here, we compare GPEC uncertainty results before and after applying explanation noise estimates.
We combine GPEC with two different explainers: BayesSHAP and Shapley Sampling Values (SSV) (Strumbelj & Kononenko, 2013).
The two selected explainers are different methods of approximating SHAP values, however the former has an inherent method for quantifying its uncertainty while the latter does not.
In Figure 5 row (A) we plot the heatmap for uncertainty estimates using GPEC using only the WEG Kernel and no added noise from the explainers.
In row (B) we add the additional uncertainty estimates from BayesSHAP and SSV.
The difference in quantiﬁed uncertainty is visualized in the row (C).
We observe that combining the uncertainty estimates from BayesSHAP and SSV in the GPEC Gaussian
Process formulation increases the overall uncertainty estimate.
L IMITATIONS AND C ONCLUSION
Generating uncertainty estimates for feature attribution explanations are essential for building reliable explanations.
We introduce a novel GP-based approach that can be used with any black-box classiﬁer and feature attribution method. GPEC generates uncertainty estimates for explanations that capture the complexity of the black-box model.
Experiments show that capturing this uncertainty improves understanding of the explanations and the black-box model itself.
Regarding limitations, GPEC relies on DB estimation methods which is an ongoing area of research.
Due to the time complexity of DB estimation, this can result in a tradeoff between computation time and approximation accuracy or sample bias.
However, the effects DB sampling time are minimized during inference as the DB only needs to be sampled during training.
Additionally, in its current implementation GPEC is limited to black-box classiﬁers; we leave the extension to regression as future work.