Active Surrogate Estimators: An Active Learning Approach to Label–Efficient Model Evaluation
Anonymous Author s)
AffiliationAddress
Abstract
We propose Active Surrogate Estimators (ASEs) , a new method for label-efficientmodel evaluation.
Evaluating model performance is a challenging and importantproblem when labels are expensive.
ASEs address this active testing problem usinga surrogate-based estimation approach that interpolates the errors of points withunknown labels, rather than forming a Monte Carlo estimator.
ASEs actively learnthe underlying surrogate, and we propose a novel acquisition strategy, XWED , thattailors this learning to the final estimation task.
We find that ASEs offer greaterlabel-efficiency than the current state-of-the-art when applied to challenging modelevaluation problems for deep neural networks.
Introduction
Machine learning research routinely assumes access to large, labeled datasets for supervised learning.
Practitioners often do not have that luxury.
Instead, no labeled data might be available initially,and label acquisition often involves significant cost: it might require medical experts to analyzex-ray images, physicists to perform experiments in particle accelerators, or chemists to synthesizemolecules.
In these situations, we should carefully acquire only those labels that are most informativefor the task at hand, maximizing performance while minimizing labeling cost [1, 62, 27, 43].
While a wide array of active learning methods [1, 62] have been developed to address the problemof training models when labels are expensive, much less attention has been given to the problem oflabel-efficient model evaluation .
Such model evaluation is important, e.g. for performance analysis,model selection, or hyperparameter optimization.
Indeed, Lowell et al.
[45] argue that a lack ofefficient approaches to model evaluation holds back adoption of active learning methods in practice.
Prior work in label-efficient model evaluation has focused on using importance sampling (IS) techniques to reduce the variance of Monte Carlo (MC) estimates [38, 59, 70].
In particular, recentapproaches [38, 18] have iteratively refined the proposal using information from previous evaluations,producing adaptive IS (AIS) estimators with state-of-the-art performance.
Though these methods provide substantial label-efficiency gains over naive sampling, they have someclear weaknesses that suggest improvements should be possible.
Firstly, the AIS weights assignedto samples depend on the acquisition proposal used at that iteration .
Thus, early samples are ofteninappropriately weighted, because we are yet to learn an effective proposal.
This can significantlyincrease the variance of the resulting estimator.
Secondly, MC estimates can be inefficient at dealingwith label noise: constraining the estimate to be a weighted sum of observed losses is limiting whenthe losses themselves are noisy.
Finally, IS methods can be restrictive on the acquisition strategiesused to choose points to label: acquisition must be stochastic with known probabilities.
To address these limitations, we introduce Active Surrogate Estimators (ASEs) , a new approach tolabel-efficient model evaluation that is based on interpolation rather than MC estimation.
Specifically,
ASEs actively learn a surrogate model to directly predict losses at all observed and unobserved points in the pool.
Here, the surrogate can take the form of any appropriate machine learning model,and we introduce a general framework for learning effective surrogates.
The interpolation-basednature of ASEs leads to a variety of advantages over MC estimates: ASEs better accommodatenoisy label observations, adapt quicker to new information, and provide complete flexibility in hownew datapoints are acquired, allowing for careful customizations that are not appropriate in the
AIS framework.
To exploit this, we propose a novel acquisition strategy, XWED , that tailors theacquisition towards points that will be most beneficial to the final estimation task.
To summarize, our main contributions are: (1) We introduce Active Surrogate Estimators; a newmethod for label-efficient model evaluation based on actively learning a surrogate model (§3).
We introduce XWED, a novel acquisition strategy that targets a weighted disagreement that ishighly relevant to our final surrogate-based prediction (§4).
(3) We provide theoretical insights into
ASEs’ errors (§5).
(4) We apply ASEs to the evaluation of deep neural networks trained for imageclassification, where we consistently outperform the current state-of-the-art (§7).
Label-Efficient Model Evaluation
We first describe the general setup of label-efficient model evaluation, also known as active testing[38].
We wish to evaluate the predictive model f : X → Y , mapping inputs x ∈ X to labels y ∈ Y .
Here, model evaluation refers to estimating the expected loss of the model predictions, that is, the risk
[Equation]
where we use capital letters for random variables, L is a loss function, and the expectation is over thetrue test distribution, which can be different to the distribution of the training data for f .
We makeno assumptions on f or the way in which it has been trained.
We only require that we can queryits predictions at arbitrary x .
Because we wish to estimate the risk of f , we keep f fixed and it doesnot change during evaluation.
Everything here applies to both classification and regression tasks.
Following existing approaches, we study active testing in the pool-based setting.
This is the typicalscenario experienced in practice [62]: samples x i can often be cheaply collected in the form ofunlabeled data, but the density for the data generating process itself is typically unknown.
Concretely,one assumes the existence of a pool of samples, with indices P = { 1 , . . .
, N } , drawn from the testdistribution, and for which we know input features x i but not labels y i , i ∈ P .
If we knew the labels at all test pool indices P , we could compute the empirical test risk ˆ r = 64 1 / N (cid:80) i ∈P
L ( f ( x i ) , y i ) .
When N is large, this provides an accurate estimate of r .
However, oneoften cannot afford to evaluate ˆ r : the cost associated with label acquisition is too high to acquirelabels for all points in P .
Instead, active testing methods must decide on a subset O ⊆ P of points to 67 label, where typically we have that M = |O| ≪ |P| = N .
The simplest approach would be to sample indices uniformly at random.
The resulting naive MCestimate, ˆ r iid = 1 / M (cid:80) i ∈O
L ( f ( x i ) , y i ) , represents the predominant practice of using randomlyconstructed test set instead of actively evaluating models.
However, the variance of ˆ r iid is oftenunacceptably high when M is small.
Methods for active testing seek to provide risk estimates withlower estimation error than ˆ r iid , without increasing the number of label acquisitions M .
Similar to active learning, it is advantageous in active testing to not select all points O upfront, butinstead to select them adaptively by using information from previous evaluations.
The mechanism bywhich points are chosen is known as an acquisition strategy .
As with active learning, it is typical toassume that the cost of labeling is dominant, and all other costs incurred during model evaluationare typically ignored.
Therefore, it can be worth (repeatedly) training surrogate models, despite theresulting expense, if this helps minimize the estimation error for a given label budget M .
2.1 Importance Sampling–Based Active Testing
Importance sampling (IS) [30, 51] is a classic strategy for forming MC estimates with lower variancethan naive MC.
IS samples points from a proposal X ∼ q instead of the original reference distribution,which we will denote p .
Adaptive importance sampling (AIS) [32, 48, 5] improves IS by refiningthe proposal
q using information from previous iterations.
However, traditional IS and AIS are onlyapplicable if we can evaluate the density of the reference distribution, at least up to normalization.
Thisis not the case in many relevant active testing scenarios, e.g. when x i are high-dimensional images.
IS cannot be applied directly in the pool-based setting because of the need to sample withoutreplacement.
However, in the context of a training objective for active learning, Farquhar et al.
[18] introduced LURE, which converts IS to this pool-based setting.
Specifically, they computeˆ r LURE =
[Equation]
M (cid:88)
[Equation]
where q ( i m ) is the probability of choosing to label sample i m at iteration m .
Kossen et al.
[38]showed that LURE can also be applied to active testing, for which the optimal proposal is
[Equation]
In practice, q ∗ is intractable as the distribution Y | X
= x
i m is unknown.
However, using ideasfrom AIS, one can adaptively improve q with information gathered from previous labels.
In particular, Kossen et al.
[38] introduce strategies to directly approximate (3) (see §6).
When q correctlyidentifies regions in the input where the loss is large, ˆ r LURE reduces variance compared to ˆ r iid .
Unlike standard AIS, adaptive LURE conveniently does not require knowledge of an explicit densityp ( x ) —a pool of samples suffices.
However, most techniques for reducing the variance of the weightsin the AIS literature (see e.g. Bugallo et al.
[5]) cannot currently be transferred over to adaptive
LURE.
In particular, we cannot employ methods like deterministic mixture AIS [50] to update historicweights using future proposal adaptations.
This means that poorly-calibrated early weights can persistand significantly increase the variance of the estimator.
Active Surrogate Estimators
We now introduce Active Surrogate Estimators (ASEs), our novel method for label-efficient modelevaluation.
Following the setup in §2, we assume a pool of unlabeled samples from which weiteratively choose points to label.
Unlike existing methods that rely on MC estimates, ASEs use aninterpolation-based approach to estimate model risk: a surrogate model directly predicts losses for allpoints in the test pool.
ASEs are further based around an adaptive and iterative procedure: we use anactive learning approach to carefully select the labels used to train our surrogate.
We first discuss theform that the ASE and surrogate take, and later cover our novel acquisition strategy in §4.
3.1 Deriving the ASE
The fundamental idea of ASEs is that of interpolation-based estimation: we wish to build a riskestimator of the form ˆ r ASE ( ϕ ) = 1 / N (cid:80) i ∈P g ( x i ; ϕ ) , where g ( x i ; ϕ ) is a flexible surrogate modelwith parameters ϕ
that approximates the conditional expected loss, E [ L ( f ( X ) , Y )
| X = x ] .
Though algorithmically quite different, the motivation behind this interpolation-based approach isquite similar to that of variational inference (VI) [74]: rather than relying on a sample-based approximation for ˆ r , we are converting our estimation into the optimization of a functional approximation.
Though, as with VI, this comes at the cost of weaker theoretical asymptotic guarantees compared with
MC estimation (see §5), it allows us to generalize more effectively when only a small number of labelsare available.
This trade-off is typically worthwhile in the small sample size regime of active testing.
To decide what form g should take, we first note that the theoretically optimal g —in terms ofmean squared error (MSE) for r —is simply the expected loss above, analogously to the optimalproposal for LURE.
While this is itself obviously intractable, we can learn an effective g by trying toapproximate this optimal g ∗ .
Though one could directly approximate this using a generic regressor,such as a neural network, this would disregard that we already know the loss function L , canevaluate f , and know how both are combined to give the optimal g ∗ .
In fact, the only unknown isthe conditional expectation Y | X = x for the test distribution.
We therefore introduce an auxiliary model π ( y | x ; ϕ ) of only this conditional.
Combining π , f , and
L , we construct g as g ( x ; ϕ ) =
E Y ∼ π ( ·| x ; ϕ )
[ L ( f ( x ) , Y )] , where the only learnable components of gare the parameters of π , ϕ .
With π introduced, we can finally write the ASE as
[Equation]
where we emphasize that the expectation over Y | X
= x
i is now with respect to π and not theunknown true label conditional.
Note that the fact that the ASE includes an explicit model ofthe label noise is a decisive advantage over prior work (see §8).
For classification, we can nowdirectly evaluate the expectation over the labels as a weighted sum over the predictive probabilities y π
( y | x i ; ϕ ) L ( f ( x i ) , y ) .
For regression, closed-form solutions of the expectation exist for somelosses, and we can always compute unbiased and accurate MC estimates using cheap samples from π .
Assembling g from L , f , and π will typically be advantageous to modeling g directly: a direct modelwould need to learn not only about predictions of f , but also how they relate to the test distribution.
In contrast, learning models π ( y | x ; ϕ ) of conditional outcomes is standard procedure.
This alsomakes it easy to include other prior information.
For example, initializing π on the training data cansometimes give the ASE a useful estimate of the risk before observing any test data at all.
Lastly, themodular construction of g allows us to quickly adapt the ASE to different models or loss functions.
Often, f , like π , will be a probabilistic model of outcomes.
However, it will generally be a highlyinappropriate choice for π , such that it is important to always learn a separate auxiliary model.
This is because we are essentially using π to predict the errors of f , and we should therefore aim todecorrelate the errors of π from those of f by training a separate π .
Further, we may want to chooseπ from a different model class than f , e.g. to allow for meaningful incorporation of uncertainties.
Lastly, while f is by construction fixed, we can adapt π during evaluation to reflect our updatedknowledge of the test label distribution, as we explain next.
3.2 Adaptive Refinement of ASEs
Inputs: Test pool { x i } i ∈P , acquisition strategy a , tar- get model f , initial auxiliary model π1: Initialize O = ∅ and U =
P 2: for m = 1 to M do 3:
Select i m = a ( π m − 1 , f, { x i } i ∈P , { y i } i ∈O ) 4:
Acquire label y
i m ∼ Y | x i m 5:
Update O ←
O
∪ i m
and U ← U \ i m 6:
Update auxiliary model π m (e.g. by retraining) 7: end for 8: Return ˆ r ASE as per (4) with π = π M
ASEs are adaptive: they iteratively improvetheir estimate by updating the surrogate asnew labels are acquired.
A crucial component to them doing this effectively is thatthey actively select the labels to acquire ateach iteration using an acquisition strategy.
As such, they can be thought of as performing estimation via active learning of theauxiliary model π , noting that the estimatethey produce is, in turn, fully defined bythis auxiliary model and the pool set.
To define this adaptive refinement process more formally, we denote with U and O the sets ofunobserved and observed points in the test pool.
These are initialized as U =
P
and
O = ∅ .
At each 162 subsequent iteration m of our adaptive refinement process, an acquisition strategy a selects an indexi m ∈ U of an unlabeled datapoint, for which we then acquire a label y
i m .
We remove i m from U , 164 and add it to
O .
Then, we obtain an improved ASE by retraining π , e.g. on the total observed testdata { ( x i , y i ) : i ∈ O} ; this retraining does not necessarily need to be performed at every iteration if 166 doing so would be computationally infeasible.
A summary of this approach is given in Algorithm 1.
Here, our acquisition strategy takes as input the current auxiliary model π m , the original model f , thepool of samples, and all previous evaluations, that is, we have i m = a ( π m − 1 , f, { x i } i ∈P , { y i } i ∈O ) .
A good acquisition strategy selects points such that the expected ASE error after M iterations isminimal [28].
While ASEs can be used with arbitrary acquisition strategies, we next introduce anovel acquisition strategy that we specifically design for use with ASEs.
The XWED Acquisition Function
It is important we tailor our acquisition strategy to the problem of active surrogate estimation, ratherthan apply more generic active learning approaches.
We need to acquire labels such that we producethe best ASE estimate given a limited labeling budget.
To do this, we propose a novel acquisitionfunction called the E x pected We ighted D isagreement, or XWED for short, that is designed specificallyto target improvements of the ASE estimate itself.
In line with acquisition strategies in active learning, we now require that π takes the form π ( y | x ; ϕ ) =
E Θ ∼ π ( · ; ϕ )
[ π ( y | x, Θ; ϕ )] , where Θ represents some model parameters we wish to learn about.
Ina BNN, Θ is the weights and biases of the network, while ϕ corresponds to the parameters of theposterior approximation given by previous observations (such that it contains all information fromprevious data).
To avoid clutter, we omit ϕ in the following presentation of acquisition functions.
For active learning, BALD [27] is an established acquisition strategies that acquires points with highepistemic uncertainty [13, 34].
For a predictive model π as above, BALD can be calculated as thedisagreement between the total predictive entropy and the expected entropies over posterior draws,
[Equation]
However, BALD, and other acquisition functions from active learning, are ill-suited to ASEs becausethey do not consider the effect the uncertainties in π have on the ASE estimate.
In particular, ASEsdepend on the loss L ( y, f ( x )) of the original model f at the pool samples, but neither the form of theloss function L nor the original model f are considered in BALD acquisition.
Acquisition for ASEs should instead target points that are both informative about Θ and which weexpect will contribute significantly to our final risk estimate.
The basis of XWED is therefore toweight the disagreement terms for particular inputs x using their corresponding loss L ( Y, f ( x )) :
[Equation]
With XWED, we acquire points with large epistemic uncertainty [13, 34] in the auxiliary π about theoutcomes only if they are relevant to the estimate ˆ r ASE .
As we show in §7, the XWED acquisitionfunction allows for significant improvements compared to BALD and other strategies.
XWED is compatible with a variety of model classes for π , such as Bayesian Neural Networks(BNNs)
[46, 21] and deep ensembles [41].
We emphasize that it is an acquisition strategy explicitlydesigned for active testing; it is not even defined in a more traditional active learning context.
Theoretical Analysis of the ASE Error
We next provide a theoretical analysis of ASEs to understand individual contributions to their error.
Specifically, we will break down the errors of ASEs into ( I ) error from using a finite evaluation pooland ( II ) error from imperfection of the surrogate, further breaking the latter down as ( IIA ) errorfrom any limited expressivity in our surrogate class and ( IIB ) error from imperfect training.
To start, let Φ be the final parameters obtained from the stochastic training procedure of the surrogateg , such that our learned surrogate is g ( · ; Φ) .
Here, Φ is a random variable with the stochasticityoriginating from (i) sampling the test pool itself, (ii) any stochasticity in the acquisition strategy, (iii)randomness in the acquired labels, and (iv) stochasticity in the optimization process of the surrogate.
Next, let X 0 ∼ p ( x ) be a hypothetical additional input sample that is independent from the testpool.
We can use this to define G ( ϕ ) =
E [ g ( X 0 ; ϕ )] as the true expectation with respect to the inputdistribution of a surrogate with parameters ϕ . In turn, we can define ϕ ∗ =
argmin ϕ | G ( ϕ ) − r | as theparameters which give the best approximation of the true expected loss E [ L ( f ( X ) , Y )
| X = x ] .
Notehere that ϕ ∗ is not random as it depends only on the fixed function class of g and the fixed true risk r .
Denoting our test pool as P = { X 1 , . .
X N } , where X i ∼ p ( x ) are independently sampled, we canwrite ˆ r ASE (Φ , P ) to make the dependency of the ASE estimator (4) on P and Φ explicit.
By using thetriangle inequality, we can then break down the error ∥
ˆ r ASE (Φ , P ) − r ∥ to the true model risk r as
[Equation]
where ∥ · ∥ is any valid norm, and we note that G ( ϕ ) is independent of X 0 and G ( ϕ ∗ ) is deterministic.
We now consider each of the terms in (7):( I ) =
∥
ˆ r ASE (Φ , P ) − G (Φ) ∥ is the error that originates from only evaluating the surrogate on a finiteset of pool of points.
As per standard results for MC estimation, this will generally decrease as
O ( 1 / √ N ) , such that it will be small for large test pools.
Note here that ˆ r ASE (Φ , P ) is typically notan unbiased estimator for G (Φ) as the set of input points available in the pool can influence thedistribution of Φ . However, this effect will typically be very small and the bias will diminish as
N increases. We can also, if desired, also eliminate it completely by evaluating g over a separatetest pool than that used during the acquisition of test labels and updating of g .
This variation on thestandard ASE estimator can also be useful when the pool is impractically large for performing labelacquisition, as we can evaluate the final learned g on a larger pool than used during its training.
( II ) =
∥ G (Φ) − r ∥ represents the error from the imperfection of the surrogate.
{+For the squared norm,it equals E Φ [( E X 
+}
[ g ( X 0 ; Φ)] − r ) 2 ] , such that it is the expected squared difference (over surrogateparameters) between the surrogate’s expectation over inputs and the true risk.
It can itself be furtherbroken down into terms ( IIA ) and ( IIB ) as described below.
Note that this term demonstrates thecritical dependency of ASE’s performance on the regressional performance of the surrogate itself.
( IIA ) = | G ( ϕ ∗ ) − r | gives the error from limits in the choice of the surrogate function class.
Itmeasures the difference between g ∗ and g ( · ; ϕ ∗ ) .
Whenever our surrogate function class contains thetrue expected loss, this term will be exactly zero.
Even when this is not the case, it should generallybe negligible for sensible choices of the surrogate function class. For example, if π is a deep neuralnetwork, the limiting factor will almost always be the training error from using finite data (i.e. term( IIB ) below), rather than the theoretical expressivity of the network.
( IIB ) =
∥ G (Φ) − G ( ϕ ∗ ) ∥ can be thought of as the error originating from the training of thesurrogate: it is the norm of the difference between the true expectation of the surrogate we havelearned and the true expectation of the best possible surrogate we could have learned.
5.1 Discussion and Empirical Analysis
In practice, ( IIB ) will typically dominate the ASE error.
If term ( I ) is significant, the pool size is alimiting factor, which can only be the case when our ASE estimation has been successful (assuming
M ≪ N ), as its implies we have a highly accurate estimate of the full empirical test risk ˆ r .
If term( IIA ) is significant, this means we are using an inappropriately weak model class for our surrogate.
To check these assertions hold in practice, we empirically estimate ( I ) and ( IIA ) for the experimentsin §7.1 at M = 1 : we find that they respectively make up only 0 .
00013% and 0 .
031% of the fullerror; thus ( IIB ) is completely dominant here.
Further characterizing the dominating term ( IIB ) in the fully general setting is unfortunately notfeasible: it equates to characterizing the error of a regression, which necessarily can only be done byconsidering a particular class of surrogate.
However, for a particular choice of surrogate, existingresults from learning theory could be applied to further categorize its convergence.
Here work on theconvergence of models in active learning [23, 24, 58, 7, 54] will be particularly appropriate, notingthat these all make specific assumptions about the model and acquisition strategy.
More specifically,for Gaussian process surrogates and acquisition strategies from Bayesian quadrature, the results of
Kanagawa & Hennig [31] could be applied.
On the other hand, for surrogates based on deep learningarchitectures, one could use results from the convergence of stochastic gradient descent methods [47],which typically give rates of O ( 1 / √ M ) under appropriate assumptions.
These will require unbiasedgradients in the optimization of the objective function, which we can achieve for ASEs with stochasticacquisition and the de-biasing scheme of Farquhar et al.
In summary, for large pool sizes and appropriate surrogate classes, ( IIB ) will be the dominantcontributor to the error, noting that the inequality in (7) becomes tight as one of the errors dominates.
As such, the convergence of ASEs essentially reduces to the convergence of the surrogate, and thus,by extension, the convergence of the auxiliary model.
Moreover, if we can ensure that this auxiliarymodel convergences to the true conditional output distribution as the number of label acquisitions Mincreases (this itself implies that our surrogate function class is sufficient powerful and so ( IIA ) = 0 ),we can thus ensure that the ASE estimator itself converges, noting that M → ∞ also predicates 268 N → ∞ (as N > M by construction), such that ( I ) → 0 .
Related Work
Active Learning.
Active learning (AL) [1, 62, 27, 21, 61] allows for the training of supervised modelsin a label-efficient manner, and it is a particular instance of optimal experimental design (OED)[44, 6, 60, 20].
OED formalizes the idea of targeted acquisition to maximize the information gainedwith respect to a target objective.
Relatedly, transductive AL [71, 68, 36] performs AL of modelpredictions with respect to a known set of points.
Filstroff et al.
[19] propose AL for decision making.
Active Testing.
Sawade et al.
[59] explore IS for active risk estimation.
They rely on vanilla ISwhich does not respect the pool-based setting [38].
Yilmaz et al.
[70] have recently proposed activetesting based on Poisson sampling.
Both [70] and [59] outperform simple baselines; however, theyuse non-adaptive acquisitions that cannot adjust to test data.
Of particular relevance is the current state-of-the-art approach by Kossen et al.
Building on the
LURE estimator, they propose an AIS-style estimate of the model test risk (§2.1).
To approximate theoptimal acquisition (3), they, like us, introduce an auxiliary model
π ( y | x ; ϕ ) that is (usually) adaptedas test labels are acquired.
Given restrictions of the LURE estimator, they always sample from
[Equation]
8) for label acquisition and do not actively improve the proposal, e.g. through active learning.
[2, 33, 40, 29] explore stratification for efficient evaluation of classifiers: a simple, non-adaptivecriterion is used to divide the test pool into strata, from which labels are selected uniformly at random.
These approaches can be improved by applying the method considered here within the strata.
Surrogates.
The use of surrogate models to replace expensive black-box functions is pervasivethroughout much of the sciences [53, 37, 10].
Bayesian optimisation (BO) [63, 4] and Bayesianquadrature (BQ) [49, 56, 3] are prominent examples of surrogate-based approaches that actively selectsamples.
BO iteratively finds the maximum of an unknown function, using a surrogate model and ideassimilar to active learning to select sample locations.
Unlike active learning, BO does not focus on aglobally accurate surrogate.
BQ obtains a Bayesian posterior for the integration of a Gaussian process(GP)
[55] surrogate against a known data density p ( x ) .
Despite biased estimates, BQ can drasticallyoutperform MC in low dimensions.
However, BQ does not scale well to high dimensions, where, often,
GP surrogates are not appropriate and explicit densities p ( x ) are unknown and hard to approximate.
Experimental Results
We next study the performance of ASEs for active testing in comparison to relevant baselines.
Concretely, we compare to naive MC and the current state-of-the-art LURE-based active testingapproach by Kossen et al.
As a general and common application, we consider the importantproblem of actively evaluating deep neural networks trained on high-dimensional image datasets.
Wegive full details on all experiments in the appendix.
7.1 Evaluation with Distribution Shift
[Figure]
at steps marked with grey lines.
We first explore model evaluation in a challenging distribution shift scenario.
Concretely, weare given a fixed model trained on 2000 digits of MNIST [42], where all sevens have beenremoved from the training set.
We then estimate the model risk on a test set with all classespresent in their normal proportions.
The testset has size N = 60000 but we can only affordto acquire labels for a small subset.
Such distribution shift settings are relevant in practice 313 and challenging for active testing: both ASEand LURE-based active testing needs to approximate the expected loss, and this is particularlyhard under distribution shift, where the initial πhas yet to learn about the test distribution.
Following [38], we use radial Bayesian Neural Networks [17] for f and π , with π initialized on thetraining set and regularly retrained during evaluation.
For LURE, we acquire labels by samplingfrom (8).
For the ASE, we use the XWED acquisition strategy (6).
Both ASE and LURE use thesame auxiliary model class π , but arrive at different network weights, ϕ ASE and ϕ LURE , because theirdistinct acquisition strategies lead to different test observations used for the re-training of π .
Main Result.
Figure 1 shows the MSE for ASE and baselines when estimating the unknown test poolrisk.
Note here that we are careful to use the best possible setup of the LURE baseline to construct achallenging benchmark, as shown in the next section.
We see that ASE improves on all baselineswhen used with the XWED acquisition strategy.
Further, LURE can struggle with variance, witherrors of individual runs occasionally spiking.
This occurs when there is a significant mismatchbetween observed loss and proposal, e.g. when an acquired point is ambiguous or mislabeled.
Thisdoes not happen for ASEs, as we do not weight true observations against predictions in the estimate.
7.2 Acquisition Strategies for ASE and LURE
We next perform an ablation to investigate different acquisition strategies besides XWED in thedistribution shift scenario: We select points with maximal BALD score, Eq.
Further, we acquirepoints by sampling from the expected loss; this is the exact same acquisition strategy used for
LURE-based active testing, cf.
We study additional choices for XWED acquisition in §B.
Figure 2 (a) shows that ASE is able to outperform naive MC for all acquisition strategies, but there arenoticeable variations in their performance.
The XWED acquisition, that we have proposed specifically
[Figure]
improvements much slower decrease early on, though does catch up with later. loss for sample the missing loss is high) almost exclusively, which initiallyworks well but later leads improve on in
In Figure 2 (b), we explore how LURE behaves under different acquisition strategies, such as samplingfrom XWED or BALD scores.
We find that LURE is noticeably more sensitive to the acquisitionstrategy than ASE: both XWED and BALD acquisition lead to high-variance estimates that no longeroutperform the naive MC baseline.
It seems one cannot expect acquisition strategies other thanexpected loss to perform well with LURE.
In contrast, ASEs can directly apply any acquisitionscheme without introducing such sensitivity because the acquisition does not affect which points areincluded in the estimate or how they are weighted
7.3 Evaluation of CNNs for Classification
Next, we study active testing of ResNets on more complex image classification datasets.
Concretely,we study the efficient evaluation of a Resnet-18 [25] on Fashion-MNIST [69] and CIFAR-10 [39],and of a WideResNet [73] on CIFAR-100.
In each case, we train the model to be evaluated, f , ona training set containing 40 000 points, and then use an evaluation pool of size N = 2000 .
As aground truth, we compare to the empirical risk on a held out set of 20 000 points.
For π , we use deepensembles [41] of the original ResNet models.
This experiment design is also used by [38].
In this experiment, there is no distribution shift between the training data for f and the data on whichwe evaluate it.
At the same time, the training set is large and, therefore, a lot of prior information isavailable to train the initial π .
Further, the number of test acquisitions is comparatively small ( M =50 ) and unlikely to affect π meaningfully, given the models have already seen 40 000 points fromthe training data.
Indeed, we find that updating π with this additional test data makes no noticeabledifference to performance.
To minimize computational costs, we therefore report results with π fixedthroughout this experiment, such that, here, π is equal for ASE and LURE.
A constant π leads to a constant LURE proposal (making the method more akin to IS than AIS) and constant prediction for ASE.
Figure 3 shows that ASEs perform effectively in this setting, significantly outperforming LUREand naive MC for all datasets.
We also display the error from the finite pool size, by comparingthe empirical risk of all N unobserved test points against our additional ground truth set and seethat ASE consistently performs similarly to this pool limit.
This suggests that the pool size N maybe the limit of the performance for ASEs here rather than the quality of the surrogate.
This experiment produces a rather unusual result: as π is fixed throughout, the ASE actually does notmake use of any information from the test labels!
Its accurate estimates of the test risk are entirelydue to information from the training data.
Note that LURE here has access to the same π as ASEin addition to up to 50 test labels.
Nevertheless, the surrogate prediction of the ASE at all poolpoints leads to much lower error than a LURE estimate.
This ‘zero-shot’ setting opens up interestingavenues for model evaluation when there are no test labels available at all.
However, in practice, one may not know if the test data follow the same distribution as the trainingdata.
Figure B.2 (a) shows that both LURE and ASE perform poorly when not retraining π in the
[Figure]
MNIST
[Figure]
distribution shift setting for any labeling budget M .
We therefore generally suggest regular adaptationof π for both ASE- and LURE-based active testing approaches.
And even without distribution shift,the information from test labels will always eventually become relevant as M increases.
Discussion
In the previous section, we have seen ASEs outperform MC-based alternatives in practice.
Here, wediscuss advantages of ASEs that are important to their practical success.
Actively Learning Estimators.
ASEs are flexible about which active learning strategy can be used.
In contrast, AIS-based strategies must balance competing goals: the acquisition strategy must bothfind informative data to improve the proposal in future iterations and provide importance-samplingweights that reduce the estimator variance.
For AIS, acquiring the most informative data with highprobability can make the estimator worse because the resulting acquisition probabilities are a badapproximation for the optimal q ∗ .
Prior work using LURE does not address this trade-off and alwaysacquires with the best approximation of q ∗ available.
ASE acquisitions can also be deterministic,while AIS requires full-support stochastic acquisition to avoid bias.
Surrogate Updates.
For AIS, improving the proposal distribution only improves future acquisitions,but does not retrospectively improve weights attached to earlier observations.
Bad initial acquisitionweights have a lasting influence on the variance of AIS.
In contrast, ASE makes better retrospectiveuse of information: the iterative refinement of g leads to updated predictions at all pool points.
Noisy Labels.
ASEs have a further advantage when label distributions are noisy , i.e. observations y arestochastic even for the true label conditional p ( y | x ) .
This is common in practice, because of ambiguous inputs near class-boundaries, mislabeled examples, or measurement error.
ASEs directly modelthe noisy outcome conditionals using π .
If π is accurate, the ASE error does not increase due to noisylabels, and the ASE can interpolate beyond the noise.
In contrast, AIS suffers badly: noisy observations form a direct part of the estimate, leading to increased variance; while the acquisition (8) modelsthe noise, this only affects weights, and noisy loss observations directly enter the estimate (2).
Chen& Choe [8] provide a detailed analysis of how stochastic observations affect IS estimators negatively.
Conclusions
We have introduced Active Surrogate Estimators (ASEs), a surrogate-based method for label-efficientmodel evaluation.
The surrogate is actively learned, and we introduced the XWED acquisitionfunction to directly account for the risk prediction task.
Our experiments show that ASEs and XWEDacquisition give state-of-the-art performance for efficient evaluation of deep image classifiers.
Checklist
For all authors...
(a) Do
the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes
(b) Did you describe the limitations of your work?
[Yes] See §5.
The ASE error is mainly
bounded by the error of the surrogate regressor.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] §G
discusses this in detail.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
w
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] Cf. §5.
(b) Did you include complete proofs of all theoretical results?
[Yes] Cf. §5.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] Full code for
reproducing the experiments is included in the supplement.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] We give all training details in §7 and §D.
(c) Did you report error bars (e.g., with respect to the random seed after running exper-
iments multiple times)?
[Yes] We repeat experiments and report error bars for all
experiments.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] We report this in §H.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Yes, we cite the
creators of the datasets and code libraries we use in §H.
(b) Did you mention the license of the assets?
[Yes] Yes, we mention licenses in §H.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] We release code for ASEs in the supplement.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A] Does not apply.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A] Does not apply.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A] Does not apply.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] Does not apply.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] Does not apply.