L EVERAGING U NLABELED D ATA TO P REDICT O UT - OF -D ISTRIBUTION P ERFORMANCE
Google Research, Blueshift team
A BSTRACT
Real-world machine learning deployments are characterized by mismatches between the source (training) and target (test) distributions that may cause performance drops.
In this work, we investigate methods for predicting the target domain accuracy using only labeled source data and unlabeled target data.
We propose Average Thresholded Conﬁdence (ATC), a practical method that learns a threshold on the model’s conﬁdence, predicting accuracy as the fraction of unlabeled examples for which model conﬁdence exceeds that threshold.
ATC outperforms previous methods across several model architectures, types of distribution shifts (e.g., due to synthetic corruptions, dataset reproduction, or novel subpopulations), and datasets (W ILDS , ImageNet, B REEDS , CIFAR, and MNIST).
In our experiments, ATC estimates target performance 2 – 4 ˆ more accurately than prior methods.
We also explore the theoretical foundations of the problem, proving that, in general, identifying the accuracy is just as hard as identifying the optimal predictor and thus, the efﬁcacy of any method rests upon (perhaps unstated) assumptions on the nature of the shift.
Finally, analyzing our method on some toy distributions, we provide insights concerning when it works.
I NTRODUCTION
Machine learning models deployed in the real world typically encounter examples from previously unseen distributions.
While the IID assumption enables us to evaluate models using held-out data from the source distribution (from which training data is sampled), this estimate is no longer valid in presence of a distribution shift.
Moreover, under such shifts, model accuracy tends to degrade (Szegedy et al., 2014; Recht et al., 2019; Koh et al., 2021).
Commonly, the only data available to the practitioner are a labeled training set (source) and unlabeled deployment-time data which makes the problem more difﬁcult.
In this setting, detecting shifts in the distribution of covariates is known to be possible (but difﬁcult) in theory (Ramdas et al., 2015), and in practice (Rabanser et al., 2018).
However, producing an optimal predictor using only labeled source and unlabeled target data is well-known to be impossible absent further assumptions (Ben-David et al., 2010; Lipton et al., 2018).
Two vital questions that remain are: (i) the precise conditions under which we can estimate a classiﬁer’s target-domain accuracy; and (ii) which methods are most practically useful.
To begin, the straightforward way to assess the performance of a model under distribution shift would be to collect labeled (target domain) examples and then to evaluate the model on that data.
However, collecting fresh labeled data from the target distribution is prohibitively expensive and time-consuming, especially if the target distribution is non-stationary.
Hence, instead of using labeled data, we aim to use unlabeled data from the target distribution, that is comparatively abundant, to predict model performance.
Note that in this work, our focus is not to improve performance on the target but, rather, to estimate the accuracy on the target for a given classiﬁer.
Recently, numerous methods have been proposed for this purpose (Deng & Zheng, 2021;
Chen et al., 2021; Jiang et al., 2021;
Deng et al., 2021; Guillory et al., 2021).
These methods either require calibration on the target domain to yield consistent estimates (Jiang et al., 2021; Guillory et al., 2021) or additional labeled data from several target domains to learn a linear regression function on a distributional distance that then predicts model performance (Deng et al., 2021; Deng & Zheng, 2021;
Guillory et al., 2021).
However, methods that require calibration on the target domain typically yield poor estimates since deep models trained and calibrated on source data are not, in general, calibrated on a (previously unseen) target domain (Ovadia et al., 2019).
Besides, methods that leverage labeled data from target domains rely on the fact that unseen target domains exhibit strong linear correlation with seen target domains on the underlying distance measure and, hence, can be rendered ineffective when such target domains with labeled data are unavailable (in Sec.
5.1 we demonstrate such a failure on a real-world distribution shift problem).
Therefore, throughout the paper, we assume access to labeled source data and only unlabeled data from target domain(s).
In this work, we ﬁrst show that absent assumptions on the source classiﬁer or the nature of the shift, no method of estimating accuracy will work generally (even in non-contrived settings).
To estimate accuracy on target domain perfectly , we highlight that even given perfect knowledge of the labeled source distribution (i.e., p s p x, y q ) and unlabeled target distribution (i.e., p t p x q ), we need restrictions on the nature of the shift such that we can uniquely identify the target conditional p t p y | x q .
Thus, in general, identifying the accuracy of the classiﬁer is as hard as identifying the optimal predictor.
Second, motivated by the superiority of methods that use maximum softmax probability (or logit) of a model for Out-Of-Distribution (OOD) detection (Hendrycks & Gimpel, 2016; Hendrycks et al., 2019), we propose a simple method that leverages softmax probability to predict model performance.
Our method, Average Thresholded Conﬁdence (ATC), learns a threshold on a score (e.g., maximum conﬁdence or negative entropy) of model conﬁdence on validation source data and predicts target domain accuracy as the fraction of unlabeled target points that receive a score above that threshold.
ATC selects a threshold on validation source data such that the fraction of source examples that receive the score above the threshold match the accuracy of those examples.
Our primary contribution in ATC is the proposal of obtaining the threshold and observing its efﬁcacy on (practical) accuracy estimation.
Importantly, our work takes a step forward in positively answering the question raised in
Deng & Zheng (2021); Deng et al.
(2021) about a practical strategy to select a threshold that enables accuracy prediction with thresholded model conﬁdence.
ATC is simple to implement with existing frameworks, compatible with arbitrary model classes, and dominates other contemporary methods.
Across several model architectures on a range of benchmark vision and language datasets, we verify that ATC outperforms prior methods by at least 2 – 4 ˆ in predicting target accuracy on a variety of distribution shifts.
In particular, we consider shifts due to common corruptions (e.g., ImageNet-C), natural distribution shifts due to dataset reproduction (e.g., ImageNet-v2, ImageNet-R), shifts due to novel subpopulations (e.g., B REEDS ), and distribution shifts faced in the wild (e.g., W ILDS ).
As a starting point for theory development, we investigate ATC on a simple toy model that models distribution shift with varying proportions of the population with spurious features, as in Nagarajan et al.
Finally, we note that although ATC achieves superior performance in our empirical evaluation, like all methods, it must fail (returns inconsistent estimates) on certain types of distribution shifts, per our impossibility result.
P RIOR W ORK
Out-of-distribution detection.
The main goal of OOD detection is to identify previously unseen examples, i.e., samples out of the support of training distribution.
To accomplish this, modern methods utilize conﬁdence or features learned by a deep network trained on some source data.
Hendrycks &
Gimpel (2016); Geifman & El-Yaniv (2017) used the conﬁdence score of an (already) trained deep model to identify OOD points.
Lakshminarayanan et al.
(2016) use entropy of an ensemble model to evaluate prediction uncertainty on OOD points.
To improve OOD detection with model conﬁdence,
Liang et al.
(2017) propose to use temperature scaling and input perturbations.
Jiang et al.
(2018) propose to use scores based on the relative distance of the predicted class to the second class.
Recently, residual ﬂow-based methods were used to obtain a density model for OOD detection (Zhang et al., 2020).
Ji et al.
(2021) proposed a method based on subfunction error bounds to compute unreliability per sample.
Refer to Ovadia et al.
; Ji et al.
(2021) for an overview and comparison of methods for prediction uncertainty on OOD data.
Predicting model generalization.
Understanding generalization capabilities of overparameterized models on in-distribution data using conventional machine learning tools has been a focus of a long line of work; representative research includes Neyshabur et al.
; Neyshabur (2017);
Neyshabur et al.
(2018); Dziugaite & Roy (2017)
; Bartlett et al.
; Zhou et al.
; Long & Sedghi (2019); Nagarajan & Kolter (2019a).
At a high level, this line of research bounds the generalization gap directly with complexity measures calculated on the trained model.
However, these bounds typically remain numerically loose relative to the true generalization error (Zhang et al., 2016;
Nagarajan & Kolter, 2019b).
On the other hand, another line of research departs from complexitybased approaches to use unseen unlabeled data to predict in-distribution generalization (Platanios et al., 2016; 2017; Garg et al., 2021; Jiang et al., 2021).
Relevant to our work are methods for predicting the error of a classiﬁer on OOD data based on unlabeled data from the target (OOD) domain.
These methods can be characterized into two broad categories: (i) Methods which explicitly predict correctness of the model on individual unlabeled points (Deng & Zheng, 2021; Jiang et al., 2021;
Deng et al., 2021); and (ii) Methods which directly obtain an estimate of error with unlabeled OOD data without making a point-wise prediction (Chen et al., 2021; Guillory et al., 2021; Chuang et al., 2020).
To achieve a consistent estimate of the target accuracy, Jiang et al.
; Guillory et al.
(2021) require calibration on target domain.
However, these methods typically yield poor estimates as deep models trained and calibrated on some source data are seldom calibrated on previously unseen domains (Ovadia et al., 2019).
Additionally, Deng & Zheng (2021); Guillory et al.
(2021) derive model-based distribution statistics on unlabeled target set that correlate with the target accuracy and propose to use a subset of labeled target domains to learn a (linear) regression function that predicts model performance.
However, there are two drawbacks with this approach: (i) the correlation of these distribution statistics can vary substantially as we consider different nature of shifts (refer to
Sec. 5.1, where we empirically demonstrate this failure); (ii) even if there exists a (hypothetical) statistic with strong correlations, obtaining labeled target domains (even simulated ones) with strong correlations would require signiﬁcant a priori knowledge about the nature of shift that, in general, might not be available before models are deployed in the wild.
Nonetheless, in our work, we only assume access to labeled data from the source domain presuming no access to labeled target domains or information about how to simulate them.
Moreover, unlike the parallel work of Deng et al. (2021), we do not focus on methods that alter the training on source data to aid accuracy prediction on the target data.
Chen et al.
(2021) propose an importance re-weighting based approach that leverages (additional) information about the axis along which distribution is shifting in form of “slicing functions”.
In our work, we make comparisons with importance re-weighting baseline from Chen et al.
(2021) as we do not have any additional information about the axis along which the distribution is shifting.
P ROBLEM S ETUP
Notation.
By ||¨|| , and x¨ , ¨y we denote the Euclidean norm and inner product, respectively.
For a vector v P R d , we use v j to denote its j th entry, and for an event E
we let I r E s denote the binary indicator of the event.
Suppose we have a multi-class classiﬁcation problem with the input domain X Ď R d and label space Y “ t 1 , 2 , . . .
, k u .
For binary classiﬁcation, we use Y “ t 0 , 1 u .
By D S and D T , we denote source and target distribution over X ˆ Y .
For distributions D S and D T , we deﬁne p S or p T as the corresponding probability density (or mass) functions.
A dataset S : “ tp x i , y i qu ni “ 1 „ p D S q n contains n points sampled i.i.d.
from D S .
Let F be a class of hypotheses mapping X to ∆ k ´ 1 where ∆ k ´ 1 is a simplex in k dimensions.
Given a classiﬁer f P F and datum p x, y q , we denote the 0error (i.e., classiﬁcation error) on that point by E p f p x q , y q : “ I “ y R arg max j P Y f
j p x q ‰ .
Given a model f P F , our goal in this work is to understand the performance of f on D T without access to labeled data from D T .
Note that our goal is not to adapt the model to the target data.
Concretely, we aim to predict accuracy of f on D T .
Throughout this paper, we assume we have access to the following: (i) model f ; (ii) previously-unseen (validation) data from D S ; and (iii) unlabeled data from target distribution D T .
3. A CCURACY E STIMATION : P OSSIBILITY AND I MPOSSIBILITY R ESULTS
First, we investigate the question of when it is possible to estimate the target accuracy of an arbitrary classiﬁer, even given knowledge of the full source distribution p s p x, y q and target marginal p t p x q .
Absent assumptions on the nature of shift, estimating target accuracy is impossible.
Even given access to p s p x, y q and p t p x q , the problem is fundamentally unidentiﬁable because p t p y | x q can shift arbitrarily.
In the following proposition, we show that absent assumptions on the classiﬁer f (i.e., when f can be any classiﬁer in the space of all classiﬁers on X ), we can estimate accuracy on the target data iff assumptions on the nature of the shift, together with p s p x, y q and p t p x q , uniquely identify the (unknown) target conditional p t p y | x q .
We relegate proofs from this section to App. A.
Proposition 1.
Absent further assumptions, accuracy on the target is identiﬁable iff p t p y | x q is uniquely identiﬁed given p s p x, y q and p t p x q .
Proposition 1 states that we need enough constraints on nature of shift such that p s p x, y q and p t p x q identiﬁes unique p t p y | x q .
It also states that under some assumptions on the nature of the shift, we can hope to estimate the model’s accuracy on target data.
We will illustrate this on two common assumptions made in domain adaptation literature: (i) covariate shift (Heckman, 1977; Shimodaira, 2000) and (ii) label shift (Saerens et al., 2002; Zhang et al., 2013; Lipton et al., 2018).
Under covariate shift assumption, that the target marginal support supp p p t p x qq is a subset of the source marginal support supp p p s p x qq and that the conditional distribution of labels given inputs does not change within support, i.e., p s p y | x q “ p t p y | x q , which, trivially, identiﬁes a unique target conditional p t p y | x q .
Under label shift, the reverse holds, i.e., the class-conditional distribution does not change ( p s p x | y q “ p t p x | y q ) and, again, information about p t p x
q uniquely determines the target conditional p t p y | x q (Lipton et al., 2018; Garg et al., 2020).
In these settings, one can estimate an arbitrary classiﬁer’s accuracy on the target domain either by using importance re-weighting with the ratio p t p x q{ p s p x q in case of covariate shift or by using importance re-weighting with the ratio p t p y q{ p s p y q in case of label shift.
While importance ratios in the former case can be obtained directly when p t p x q and p s p x q are known, the importance ratios in the latter case can be obtained by using techniques from Lipton et al.
; Azizzadenesheli et al.
; Alexandari et al.
As a corollary of Proposition 1, we now present a simple impossibility result, demonstrating that no single method can work for all families of distribution shift.
Corollary 1.
Absent assumptions on the classiﬁer f , no method of estimating accuracy will work in all scenarios, i.e., for different nature of distribution shifts.
Intuitively, this result states that every method of estimating accuracy on target data is tied up with some assumption on the nature of the shift and might not be useful for estimating accuracy under a different assumption on the nature of the shift.
For illustration, consider a setting where we have access to distribution p s p x, y q and p t p x q .
Additionally, assume that the distribution can shift only due to covariate shift or label shift without any knowledge about which one.
Then Corollary 1 says that it is impossible to have a single method that will simultaneously for both label shift and covariate shift as in the following example (we spell out the details in App.
A):
Example 1.
Assume binary classiﬁcation with p s p x q “ α ¨ φ p µ 1 q ` p 1 ´ α q ¨ φ p µ 2 q , p s p x | y “ 0 q “ φ p µ 1 q , p s p x | y “ 1 q “ φ p µ 2 q , and p t p x q “ β ¨ φ p µ 1 q ` p 1 ´ β q ¨ φ p µ 2 q where φ p µ q “ N p µ, 1 q , α, β P p 0 , 1 q , and α ‰ β .
Error of a classiﬁer f on target data is given by E 1 “ E p x,y q„ p s p x,y q ” [Equation] p [Equation]
q
[Equation]
ı under covariate shift and by E 2 “
E p x,y q„ p s p x,
[Equation]
q ”´
[Equation]
s
[Equation]
I
[Equation]
¯ I r f p x q ‰ y s ı under label shift.
In App.
A, we show that E 1 ‰ E 2 for all f .
Thus, given access to p s p x, y q , and p t p x q , any method that consistently estimates error of a classifer under covariate shift will give an incorrect estimate of error under label shift and vice-versa.
The reason is that the same p t p x q and p s p x, y q can correspond to error E 1 (under covariate shift) or error E 2 (under label shift) and determining which scenario one faces requires further assumptions on the nature of shift.
P REDICTING ACCURACY WITH A VERAGE T HRESHOLDED C ONFIDENCE
In this section, we present our method ATC that leverages a black box classiﬁer f and (labeled) validation source data to predict accuracy on target domain given access to unlabeled target data.
Throughout the discussion, we assume that the classiﬁer f is ﬁxed.
Before presenting our method, we introduce some terminology.
Deﬁne a score function s : ∆ k ´ 1 Ñ R that takes in the softmax prediction of the function f and outputs a scalar.
We want a score function such that if the score function takes a high value at a datum p x, y q then f is likely to be correct.
In this work, we explore two such score functions: (i) Maximum conﬁdence, i.e., s p
f p x qq “ max j P Y f
j p x q ; and (ii) Negative Entropy, i.e., s p
f p x qq “ ř j f j p x q log p f
j p x qq .
Our method identiﬁes a threshold t on source data D S such that the expected number of points that obtain a score less than t match the error of f on D S , i.e.,
[Equation]
and then our error estimate ATC D T p s q on the target domain D T is given by the expected number of target points that obtain a score less than t , i.e.,
[Equation]
In short, in (1), ATC selects a threshold on the score function such that the error in the source domain matches the expected number of points that receive a score below t and in (2), ATC predicts error on the target domain as the fraction of unlabeled points that obtain a score below that threshold t .
Note that, in principle, there exists a different threshold t 1 on the target distribution D T such that (1) is satisﬁed on D T .
However, in our experiments, the same threshold performs remarkably well.
The main empirical contribution of our work is to show that the threshold obtained with (1) might be used effectively in condunction with modern deep networks in a wide range of settings to estimate error on the target data.
In practice, to obtain the threshold with ATC, we minimize the difference between the expression on two sides of (1) using ﬁnite samples.
In the next section, we show that ATC precisely predicts accuracy on the OOD data on the desired line y “ x .
In App.
C, we discuss an alternate interpretation of the method and make connections with OOD detection methods.
E XPERIMENTS
We now empirical evaluate ATC and compare it with existing methods.
In each of our main experiment, keeping the underlying model ﬁxed, we vary target datasets and make a prediction of the target accuracy with various methods given access to only unlabeled data from the target.
Unless noted otherwise, all models are trained only on samples from the source distribution with the main exception of pre-training on a different distribution.
We use labeled examples from the target distribution to only obtain true error estimates.
Datasets.
First, we consider synthetic shifts induced due to different visual corruptions (e.g., shot noise, motion blur etc.)
under ImageNet-C (Hendrycks & Dietterich, 2019).
Next, we consider [Figure] natural shifts due to differences in the data collection process of ImageNet (Russakovsky et al., 2015), e.g, ImageNetv2 (Recht et al., 2019).
We also consider images with artistic renditions of object classes, i.e., ImageNet-R
(Hendrycks et al., 2021) and ImageNet-Sketch (Wang et al., 2019).
Note that renditions dataset only contains a subset 200 classes from ImageNet.
To include renditions dataset in our testbed, we include results on ImageNet restricted to these 200 classes (which we call
ImageNet-200) along with full ImageNet.
Second, we consider B REEDS (Santurkar et al., 2020) to assess robustness to subpopulation shifts, in particular, to understand how accuracy estimation methods behave when novel subpopulations not observed during training are introduced.
B REEDS leverages class hierarchy in ImageNet to createdatasets E NTITY -13, E NTITY -30, L IVING -17, N ON - LIVING -26.
We focus on natural and synthetic shifts as in ImageNet on same and different subpopulations in BREEDs.
Third, from W ILDS (Koh et al., 2021) benchmark, we consider FMoW-W ILDS (Christie et al., 2018), RxRx1-W ILDS (Taylor et al., 2019), Amazon-W ILDS
(Ni et al., 2019), CivilComments-W ILDS (Borkan et al., 2019) to consider distribution shifts faced in the wild.
Finally, similar to ImageNet, we consider (i) synthetic shifts (CIFAR-10-C) due to common corruptions; and (ii) natural shift (i.e., CIFARv2 (Recht et al., 2018)) on CIFAR-10 (Krizhevsky & Hinton, 2009).
On CIFAR-100, we just have synthetic shifts due to common corruptions.
For completeness, we also consider natural shifts on MNIST (LeCun et al., 1998) as in the prior work (Deng & Zheng, 2021).
We use three real shifted datasets, i.e., USPS (Hull, 1994), SVHN (Netzer et al., 2011) and
QMNIST (Yadav & Bottou, 2019).
We give a detailed overview of our setup in App.
F.
Architectures and Evaluation.
For ImageNet, B REEDS , CIFAR, FMoW-W ILDS , RxRx1-W ILDS datasets, we use DenseNet121 (Huang et al., 2017) and ResNet50 (He et al., 2016) architectures.
For
Amazon-W ILDS and CivilComments-W ILDS , we ﬁne-tune a DistilBERT-base-uncased (Sanh et al., 2019) model.
For MNIST, we train a fully connected multilayer perceptron.
We use standard training with benchmarked hyperparameters.
To compare methods, we report average absolute difference between the true accuracy on the target data and the estimated accuracy on the same unlabeled examples.
We refer to this metric as Mean Absolute estimation Error (MAE).
Along with MAE, we also show scatter plots to visualize performance at individual target sets.
Refer to App.
G for additional details on the setup.
Methods With ATC-NE, we denote ATC with negative entropy score function and with ATC-MC, we denote ATC with maximum conﬁdence score function.
For all methods, we implement post-hoc calibration on validation source data with Temperature Scaling (TS; Guo et al.
Below we brieﬂy discuss baselines methods compared in our work and relegate details to App.
E.
Average Conﬁdence (AC).
Error is estimated as the expected value of the maximum softmax conﬁdence on the target data, i.e, AC D T “ E [Equation] max [Equation]
p
[Equation]
Difference Of Conﬁdence (DOC).
We estimate error on target by subtracting difference of conﬁdences on source and target (as a surrogate to distributional distance Guillory et al.
from the error on source distribution, i.e, DOC D T [Equation] S
[Equation]
E [Equation] S r max
[Equation] P Y f
j p x qs .
This is referred to as DOC-Feat in (Guillory et al., 2021). [Table]
We include details about the sets considered in each shift
Table of calibration on source we observe that ATC achieves superior performance (lower MAE is better).
For language datasets, we use DistilBERT-base-uncased, for vision dataset we report results with DenseNet model with the exception of MNIST where we use FCN.
We include results on other architectures in App. H. For GDE post T and pre T estimates match since TS doesn’t alter the argmax prediction. Results reported by aggregating MAE numbers over 4 different seeds. We include results with standard deviation values in Table 3. Importance re-weighting (IM).
We estimate the error of the classiﬁer with importance re-weighting of 0-1 error in the pushforward space of the classiﬁer.
This corresponds to M ANDOLIN using one slice based on the underlying classiﬁer conﬁdence Chen et al.
Generalized Disagreement Equality (GDE).
Error is estimated as the expected disagreement of two models (trained on the same training set but with different randomization) on target data (Jiang et al., 2021), i.e., GDE D T “ E x „ D T r
I r f p x q ‰ f 1 p x qss where f and f 1 are the two models.
Note that GDE requires two models trained independently, doubling the computational overhead while training.
5. R ESULTS
In Table 1, we report MAE results aggregated by the nature of the shift in our testbed.
In Fig.and Fig.
1(right), we show scatter plots for predicted accuracy versus OOD accuracy on several datasets.
We include scatter plots for all datasets and parallel results with other architectures in
App. H.
In App.
H.1, we also perform ablations on CIFAR using a pre-trained model and observe that pre-training doesn’t change the efﬁcacy of ATC.
We predict accuracy on the target data before and after calibration with TS.
First, we observe that both ATC-NE and ATC-MC (even without TS) obtain signiﬁcantly lower MAE when compared with other methods (even with TS).
Note that with TS we observe substantial improvements in MAE for all methods.
Overall, ATC-NE (with TS) typically achieves the smallest MAE improving by more than 2 ˆ on CIFAR and by 3 – 4 ˆ on ImageNet over GDE (the next best alternative to ATC).
Alongside, we also observe that a linear ﬁt with robust regression (Siegel, 1982) on the scatter plot recovers a line close to x “ y for ATC-NE with TS while the line is far away from x “ y for other
[Figure]
novel populations,
ATC continues to dominate all other methods across all datasets in B REEDS .
Additionally, for different subpopulations in B REEDS setup, we observe a poor linear correlation of the estimated performance with the actual performance as shown in Fig.
3 (left)(we notice a similar gap in the linear ﬁt for all other methods).
Hence in such a setting, we would expect methods that ﬁne-tune a regression model on labeled target examples from shifts with one subpopulation will perform poorly on shifts with different subpopulations.
Corroborating this intuition, next, we show that even after ﬁtting a regression model for DOC on natural and synthetic shifts with source subpopulations, ATC without regression model continues to outperform DOC with regression model on shifts with novel subpopulation.
Fitting a regression model on B REEDS with DOC.
Using label target data from natural and synthetic shifts for the same subpopulation (same as source), we ﬁt a robust linear regression model (Siegel, 1982) to ﬁne-tune DOC as in Guillory et al.
We then evaluate the ﬁne-tuned
DOC (i.e., DOC with linear model) on natural and synthetic shifts from novel subpopulations on
B REEDS benchmark.
Although we observe signiﬁcant improvements in the performance of ﬁnetuned DOC when compared with DOC (without any ﬁne-tuning), ATC without any regression model continues to perform better (or similar) to that of ﬁne-tuned DOC on novel subpopulations (Fig.(middle)).
Refer to App.
H.2 for details and Table 5 for MAE on B REEDS with regression model.
I NVESTIGATING ATC ON T OY M ODEL
In this section, we propose and analyze a simple theoretical model that distills empirical phenomena from the previous section and highlights efﬁcacy of ATC.
Here, our aim is not to obtain a general model that captures complicated real distributions on high dimensional input space as the images in
ImageNet.
Instead to further our understanding, we focus on an easy-to-learn binary classiﬁcation task from Nagarajan et al.
(2020) with linear classiﬁers, that is rich enough to exhibit some of the same phenomena as with deep networks on real data distributions.
Consider a easy-to-learn binary classiﬁcation problem with two features x “ r x inv , x sp s P R 2 where x inv is fully predictive invariant feature with a margin γ ą 0 and x sp P t´ 1 , 1 u is a spurious feature (i.e., a feature that is correlated but not predictive of the true label).
Conditional on y , the distribution over x inv is given as follows: x inv |p
y “ 1 q „ U r γ, c s and x inv |p y “ 0 q „ U r´ c, ´ γ s , where c is a ﬁxed constant greater than γ .
For simplicity, we assume that label distribution on source is uniform on t´ 1 , 1 u .
x sp is distributed such that P s r x sp ¨ p 2 y ´ 1 q ą 0 s “ p sp , where p sp P p 0 .
q controls the degree of spurious correlation.
To model distribution shift, we simulate target data with different degree of spurious correlation, i.e., in target distribution P t r x sp ¨ p 2 y ´ 1 q ą 0 s “ p 1 sp P r 0 , 1 s .
Note that here we do not consider shifts in the label distribution but our result extends to arbitrary shifts in the label distribution as well.
In this setup, we examine linear sigmoid classiﬁers of the form f p x q “ ” 1 1 ` e wTx , e wTx 1 ` e wTx ı where w “ r w inv , w sp s P R 2 .
While there exists a linear classiﬁer with w “ r 1 , 0 s that correctly classiﬁes all the points with a margin γ , Nagarajan et al.
(2020) demonstrated that a linear classiﬁer will typically have a dependency on the spurious feature, i.e., w sp ‰ 0 .
They show that due to geometric skews, despite having positive dependencies on the invariant feature, a max-margin classiﬁer trained on ﬁnite samples relies on the spurious feature.
Refer to App.
D for more details on these skews.
In our work, we show that given a linear classiﬁer that relies on the spurious feature and achieves a non-trivial performance on the source (i.e., w inv ą 0 ), ATC with maximum conﬁdence score function consistently estimates the accuracy on the target distribution.
Theorem 1 (Informal) .
Given any classiﬁer with w inv ą 0 in the above setting, the threshold obtained in (1) together with ATC as in (2) with maximum conﬁdence score function obtains a consistent estimate of the target accuracy.
Consider a classiﬁer that depends positively on the spurious feature (i.e., w sp ą 0 ).
Then as the spurious correlation decreases in the target data, the classiﬁer accuracy on the target will drop and vice-versa if the spurious correlation increases on the target data.
Theorem 1 shows that the threshold identiﬁed with ATC as in (1) remains invariant as the distribution shifts and hence ATC as in (2) will correctly estimate the accuracy with shifting distributions.
Next, we illustrate Theorem 1 by simulating the setup empirically.
First we pick a arbitrary classiﬁer (which can also be obtained by training on source samples), tune the threshold on hold-out source examples and predict accuracy with different methods as we shift the distribution by varying the degree of spurious correlation.
Empirical validation and comparison with other methods.
Fig.
3(right) shows that as the degree of spurious correlation varies, our method accurately estimates the target performance where all other methods fail to accurately estimate the target performance.
Understandably, due to poor calibration of the sigmoid linear classiﬁer AC, DOC and GDE fail.
While in principle IM can perfectly estimate the accuracy on target in this case, we observe that it is highly sensitive to the number bins and choice of histogram binning (i.e., uniform mass or equal width binning).
We elaborate more on this in App.
D.
Biased estimation with ATC.
Now we discuss changes in the above setup where ATC yields inconsistent estimates.
We assumed that both in source and target x inv | y “ 1 is uniform between r γ, c s and x | y “ ´ 1 is uniform between r´ c, ´ γ s .
Shifting the support of target class conditional p t p x inv | y q may introduce a bias in ATC estimates, e.g., shrinking the support to c 1 ( ă c ) (while maintaining uniform distribution) in the target will lead to an over-estimation of the target performance with
ATC.
In App.
D.1, we elaborate on this failure and present a general (but less interpretable) classiﬁer dependent distribution shift condition where ATC is guaranteed to yield consistent estimates.
C ONCLUSION AND FUTURE WORK
In this work, we proposed ATC, a simple method for estimating target domain accuracy based on unlabeled target (and labeled source data).
ATC achieves remarkably low estimation error on several synthetic and natural shift benchmarks in our experiments.
Notably, our work draws inspiration from recent state-of-the-art methods that use softmax conﬁdences below a certain threshold for OOD detection (Hendrycks & Gimpel, 2016; Hendrycks et al., 2019) and takes a step forward in answering questions raised in Deng & Zheng (2021) about the practicality of threshold based methods.
Our distribution shift toy model justiﬁes ATC on an easy-to-learn binary classiﬁcation task.
In our experiments, we also observe that calibration signiﬁcantly improves estimation with ATC.
Since in binary classiﬁcation, post hoc calibration with TS does not change the effective threshold, in future work, we hope to extend our theoretical model to multi-class classiﬁcation to understand the efﬁcacy of calibration.
Our theory establishes that a classiﬁer’s accuracy is not, in general identiﬁed, from labeled source and unlabeled target data alone, absent considerable additional constraints on the target conditional p t p y | x q .
In light of this ﬁnding, we also hope to extend our understanding beyond the simple theoretical toy model to characterize broader sets of conditions under which ATC might be guaranteed to obtain consistent estimates.
Finally, we should note that while ATC outperforms previous approaches, it still suffers from large estimation error on datasets with novel populations, e.g., B REEDS .
We hope that our ﬁndings can lay the groundwork for future work for improving accuracy estimation on such datasets.
Reproducibility Statement We have been careful to ensure that our results are reproducible.
We have stored all models and logged all hyperparameters and seeds to facilitate reproducibility.
Note that throughout our work, we do not perform any hyperparameter tuning, instead, using benchmarked hyperparameters and training procedures to make our results easy to reproduce.
While, we have not released code yet, the appendix provides all the necessary details to replicate our experiments and results.
Moreover, we plan to release the code with a revised version of the manuscript
A CKNOWLEDGEMENT
Authors would like to thank Ariel Kleiner and Sammy Jerome as the problem formulation and motivation of this paper was highly inﬂuenced by initial discussions with them.
A PPENDIX A P ROOFS FROM S EC .
Before proving results from Sec. 3, we introduce some notations.
Deﬁne E p f p x q , y q : “ I “ y R arg max j P Y
f j p x q ‰ .
We express the population error on distribution D as E D p f q : “
E p x,y q„ D r E p
f p x q , y qs .
Proof of Proposition 1. Consider a binary classiﬁcation problem.
Assume P be the set of possible target conditional distribution of labels given p s p x, y q and p t p x q .
The forward direction is simple.
If P “ t p t p y | x qu is singleton given p s p x, y q and p t p x q , then the error of any classiﬁer f on the target domain is identiﬁed and is given by
[Equation]
For the reverse direction assume that given p t p x q and p s p x, y q , we have two possible distributions D T and D T 1 with p t p
y
| x q , p 1 t p y
| x q P P such that on some x with p t p x q ą 0 , we have p t p y
| x q ‰ p 1 t p y | x q .
Consider X M “ t x P X | p t p x q ą 0 and p t p y “ 1 | x q ‰ p 1 t p y “ 1 | x qu be the set of all input covariates where the two distributions differ.
We will now choose a classiﬁer f such that the error on the two distributions differ.
On a subset X 1 M “ t x P X | p t p x q ą 0 and p t p y “ 1 | x q ą p 1 t p y “ 1 | x qu , assume f p x q “ 0 and on a subset X 2 M “ t x P X | p t p x q ą 0 and p t p y “ 1 | x q ă p 1 t p y “ 1 | x qu , assume f p x q “ 1 .
We will show that the error of f on distribution with p t p y | x q is strictly greater than the error of f on distribution with p 1 t p y | x q .
Formally,
[Equation]
where the last step follows by construction of the set X 1 M and X 2 M .
Since E D T p
f
q ‰ E D T 1 p f q , given the information of p t p x q and p s p x, y q it is impossible to distinguish the two values of the error with classiﬁer f .
Thus, we obtain a contradiction on the assumption that p t p y | x q ‰ p 1 t p y | x q .
Hence, we must pose restrictions on the nature of shift such that P is singleton to to identify accuracy on the target.
Proof of Corollary 1.
The corollary follows directly from Proposition 1.
Since two different target conditional distribution can lead to different error estimates without assumptions on the classiﬁer, no method can estimate two different quantities from the same given information.
We illustrate this in
Example 1 next.
B E STIMATING ACCURACY IN COVARIATE SHIFT OR LABEL SHIFT
Accuracy estimation under covariate shift assumption Under the assumption that p t p y | x q “ p s p y | x q , accuracy on the target domain can be estimated as follows:
[Equation]
Accuracy estimation under label shift assumption Under the assumption that p t p x | y q “ p s p x | y q , accuracy on the target domain can be estimated as follows:
[Equation]
Estimating importance ratios p t p x q{ p s p x q is straightforward under covariate shift assumption when the distributions p t p x q and p s p x q are known.
For label shift, one can leverage moment matching approach called BBSE (Lipton et al., 2018) or likelihood minimization approach MLLS (Garg et al., 2020).
Below we discuss the objective of MLLS:
[Equation]
where W “ t w | @
y , w y ě
and ř ky “ 1 w y p s p y q “ 1 u .
MLLS objective is guaranteed to obtain consistent estimates for the importance ratios w ˚ p y q “ p t p y q{ p s p y q under the following condition.
Theorem 2 (Theorem 1 (Garg et al., 2020)) .
If the distributions t p p x q| y q : y “ 1 , . .
. , k u are strictly linearly independent, then w ˚ is the unique maximizer of the MLLS objective (9) .
We refer interested reader to Garg et al.
(2020) for details.
Above results of accuracy estimation under label shift and covariate shift can be extended to a generalized label shift and covariate shift settings.
Assume a function h : X Ñ Z such that y is independent of x given h p x q .
In other words h p x q contains all the information needed to predict label y .
With help of h , we can extend estimation to following settings: (i) Generalized covariate shift , i.e., p s p y | h p x qq “ p t p y | h p x qq and p s p
h p x qq ą 0 for all x P X t ; (ii) Generalized label shift , i.e., p s p
h p x q| y q “ p t p h p x q| y q and p s p y q ą 0 for all y P Y t .
By simply replacing x with h p x q in (6) and (9), we will obtain consistent error estimates under these generalized conditions.
Proof of Example 1.
Under covariate shift using (6), we get
[Equation]
Under label shift using (8), we get
[Equation]
Then E 1 ´ E 2 is given by
[Equation]
If α ą β , then E 1 ą E 2 and if α ă β , then E 1 ă E 2 .
Since E 1 ‰ E 2 for arbitrary f , given access to p s p x, y q , and p t p x q , any method that consistently estimates error under covariate shift will give an incorrect estimate under label shift and vice-versa.
The reason being that the same p t p x q and p s p x, y q can correspond to error E 1 (under covariate shift) or error E 2 (under label shift) either of which is not discernable absent further assumptions on the nature of shift.
C A LTERNATE INTERPRETATION OF ATC
Consider the following framework:
Given a datum p x, y q , deﬁne a binary classiﬁcation problem of whether the model prediction arg max f p x q was correct or incorrect.
In particular, if the model prediction matches the true label, then we assign a label 1 (positive) and conversely, if the model prediction doesn’t match the true label then we assign a label 0 (negative).
Our method can be interpreted as identifying examples for correct and incorrect prediction based on the value of the score function s p
f p x qq , i.e., if the score s p f p x qq is greater than or equal to the threshold t then our method predicts that the classiﬁer correctly predicted datum p x, y q and vice-versa if the score is less than t .
A method that can solve this task will perfectly estimate the target performance.
However, such an expectation is unrealistic.
Instead, ATC expects that most of the examples with score above threshold are correct and most of the examples below the threshold are incorrect.
More importantly, ATC selects a threshold such that the number of falsely identiﬁed correct predictions match falsely identiﬁed incorrect predictions on source distribution, thereby balancing incorrect predictions.
We expect useful estimates of accuracy with ATC if the threshold transfers to target, i.e. if the number of falsely identiﬁed correct predictions match falsely identiﬁed incorrect predictions on target.
This interpretation relates our method to the OOD detection literature where Hendrycks & Gimpel (2016);
Hendrycks et al.
(2019) highlight that classiﬁers tend to assign higher conﬁdence to in-distribution examples and leverage maximum softmax conﬁdence (or logit) to perform OOD detection.
D D ETAILS ON THE T OY M ODEL
Skews observed in this toy model In Fig.
4, we illustrate the toy model used in our empirical experiment.
In the same setup, we empirically observe that the margin on population with less density is large, i.e., margin is much greater than γ when the number of observed samples is small (in Fig.(d)).
Building on this observation, Nagarajan et al.
(2020) showed in cases when margin decreases with number of samples, a max margin classiﬁer trained on ﬁnite samples is bound to depend on the spurious features in such cases.
They referred to this skew as geometric skew .
Moreover, even when the number of samples are large so that we do not observe geometric skews,
Nagarajan et al.
(2020) showed that training for ﬁnite number of epochs, a linear classiﬁer will have a non zero dependency on the spurious feature.
They referred to this skew as statistical skew .
Due both of these skews, we observe that a linear classiﬁer obtained with training for ﬁnite steps on training data with ﬁnite samples, will have a non-zero dependency on the spurious feature.
We refer interested reader to Nagarajan et al.
(2020) for more details.
Proof of Theorem 1 Recall, we consider a easy-to-learn binary classiﬁcation problem with two features x “ r x inv , x sp s P R 2 where x inv is fully predictive invariant feature with a margin γ ą 0 and x sp P t´ 1 , 1 u is a spurious feature (i.e., a feature that is correlated but not predictive of the true label).
Conditional on y , the distribution over x inv is given as follows:
[Equation] [Equation]
where c is a ﬁxed constant greater than γ .
For simplicity, we assume that label distribution on source is uniform on t´ 1 , 1 u .
x sp is distributed such that P s r x sp ¨p 2 y ´ 1 q ą 0 s “ p sp , where p sp P p 0 .
q controls the degree of spurious correlation.
To model distribution shift, we simulate target data with different degree of spurious correlation, i.e., in target distribution P t r x sp ¨p 2 y ´ 1 q ą 0 s “ p 1 sp P r 0 , 1 s .
Note that here we do not consider shifts in the label distribution but our result extends to arbitrary shifts in the label distribution as well.
[Figure]
In this setup, we examine linear sigmoid classiﬁers of the form f p x q “
[Equation]
ı where w “ r w inv , w sp s P R 2 .
We show that given a linear classiﬁer that relies on the spurious feature and achieves a non-trivial performance on the source (i.e., w inv ą 0 ), ATC with maximum conﬁdence score function consistently estimates the accuracy on the target distribution.
Deﬁne X M “ t x | x sp ¨ p 2 y ´
q
ă
u
and X C “ t x | x sp ¨ p 2 y ´ 1 q ą 0 u .
Notice that in target distributions, we are changing the fraction of examples in X M and X C
but we are not changing the distribution of examples within individual set.
Theorem 3.
Given any classiﬁer f with w inv ą 0 in the above setting, assume that the threshold t is obtained with ﬁnite sample approximation of (1) , i.e., t is selected such that
[Equation]
where tp x i , y i qu ni “ 1 „ p D S q n are n samples from source distribution.
Fix a δ ą 0 .
Assuming n ě 2 log p 4 { δ q{p 1 ´ p sp q 2 , then the estimate of accuracy by ATC as in (2) satisﬁes the following with probability at least 1 ´ δ ,
[Equation]
where D T is any target distribution considered in our setting and c sp “ p 1 ´ p sp q if w sp ą
and c sp “ p sp otherwise.
Proof.
First we consider the case of w sp ą 0 .
The proof follows in two simple steps.
First we notice that the classiﬁer will make an error only on some points in X M and the threshold t will be selected such that the fraction of points in X M with maximum conﬁdence less than the threshold t will match the error of the classiﬁer on X M .
Classiﬁer with w sp ą 0 and w inv ą 0 will classify all the points in
X C correctly.
Second, since the distribution of points is not changing within X M and X C , the same threshold continues to work for arbitrary shift in the fraction of examples in X M , i.e., p 1 sp .
Note that when w sp ą 0 , the classiﬁer makes no error on points in X C and makes an error on a subset X err “ t x | x sp ¨ p 2 y ´
[Equation]
q ă
[Equation]
& p w inv x inv ` w sp x sp q
[Equation]
p
[Equation]
y ´ 1 q ď 0 u of X M , i.e., X err Ď X M .
Consider X thres “ t x | arg max
y P Y
f
y p x q ď t u as the set of points that obtain a score less than or equal to t .
Now we will show that ATC chooses a threshold t such that all points in X C gets a score above t , i.e., X thres Ď X M .
First note that the score of points close to the true separator in X C , i.e., at x 1 “ p γ, 1 q and x 2 “ p´ γ, ´ 1 q match.
In other words, score at x 1 matches with the score of x 2 by symmetricity, i.e.,
[Equation]
[Equation]
Hence, if t ě arg max y P Y f
y p x 1 q then we will have | X err | ă | X thres | which is contradiction violating deﬁnition of t as in (12).
Thus X thres Ď X M .
Now we will relate LHS and RHS of (12) with their expectations using Hoeffdings and DKW inequality to conclude (13).
Using Hoeffdings’ bound, we have with probability at least 1 ´ δ {
[Equation]
ÿ
[Equation]
With DKW inequality, we have with probability at least 1 ´ δ {
[Equation]
ÿ
[Equation]
for all t 1 ą 0 .
Combining (15) and (16) at t 1 “ t with deﬁnition (12), we have with probability at least 1 ´ δ {
[Equation]
Now for the case of w sp ă 0 , we can use the same arguments on X C .
That is, since now all the error will be on points in X C and classiﬁer will make no error X M , we can show that threshold t will be selected such that the fraction of points in X C with maximum conﬁdence less than the threshold t will match the error of the classiﬁer on X C .
Again, since the distribution of points is not changing within
X M and X C , the same threshold continues to work for arbitrary shift in the fraction of examples in
X M , i.e., p 1 sp .
Thus with similar arguments, we have
[Equation]
Using Hoeffdings’ bound, with probability at least 1 ´ δ { 2 , we have
[Equation]
With probability at least 1 ´ δ { 2 , we have
[Equation]
Combining (19) and (17), we get the desired result for w sp ą 0 .
For w sp ă 0 , we combine (20) and (18) to get the desired result.
[Figure]
Issues with IM in toy setting As described in App.
E, we observe that IM is sensitive to binning strategy.
In the main paper, we include IM result with uniform mass binning with 100 bins.
Empirically, we observe that we recover the true performance with IM if we use equal width binning with number of bins greater than 5.
Biased estimation with ATC in our toy model We assumed that both in source and target x inv | y “is uniform between r γ, c s and x | y “ ´ 1 is uniform between r´ c, ´ γ s .
Shifting the support of target class conditional p t p x inv | y q may introduce a bias in ATC estimates, e.g., shrinking the support to c 1 ( ă c ) (while maintaining uniform distribution) in the target will lead to an over-estimation of the target performance with ATC.
We show this failure in Fig.
The reason being that with the same threshold that we see more examples falsely identiﬁed as correct as compared to examples falsely identiﬁed as incorrect.
D. A M ORE G ENERAL R ESULT
Recall, for a given threshold t , we categorize an example p x, y q as a falsely identiﬁed correct prediction (ﬁcp) if the predicted label p y “ arg max f p x q is not the same as y but the predicted score
f p y p x q is greater than t .
Similarly, an example is falsely identiﬁed incorrect prediction (ﬁip) if the predicted label p y is the same as y but the predicted score
f p y p x q is less than t .
In general, we believe that our method will obtain consistent estimates in scenarios where the relative distribution of covariates doesn’t change among examples that are falsely identiﬁed as incorrect and examples that are falsely identiﬁed as correct.
In other words, ATC is expected to work if the distribution shift is such that falsely identiﬁed incorrect predictions match falsely identiﬁed correct prediction.
D. ATC PRODUCES CONSISTENT ESTIMATE ON SOURCE DISTRIBUTION
Proposition 2.
Given labeled validation data tp x i , y i qu ni “ 1 from a distribution D S and a model f , choose a threshold t as in (1) .
Then for δ ą 0 , with probability at least 1 ´ δ , we have
[Equation]
Proof.
The proof uses (i) Hoeffdings’ inequality to relate the accuracy with expected accuracy; and (ii) DKW inequality to show the concentration of the estimated accuracy with our proposed method.
Finally, we combine (i) and (ii) using the fact that at selected threshold t the number of false positives is equal to the number of false negatives.
Using Hoeffdings’ bound, we have with probability at least 1 ´ δ {
[Equation]
n ÿ
[Equation]
With DKW inequality, we have with probability at least 1 ´ δ {(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
ÿ
[Equation]

[Equation]
I „
[Equation]
 (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ď c log p 4 { δ q 2 n , (23) for all t 1 ą 0 .
Finally by deﬁnition, we have n ÿ
[Equation]
[Equation]
Combining (22), (23) at t 1 “ t , and (24), we have the desired result.
E B ASLINE M ETHODS
Importance-re-weighting (IM)
If we can estimate the importance-ratios p t p x q p s p x q with just the unlabeled data from the target and validation labeled data from source, then we can estimate the accuracy as on target as follows:
[Equation]
[Equation]
As previously discussed, this is particularly useful in the setting of covariate shift (within support) where importance ratios estimation has been explored in the literature in the past.
Mandolin (Chen et al., 2021) extends this approach.
They estimate importance-weights with use of extra supervision about the axis along which the distribution is shifting.
In our work, we experiment with uniform mass binning and equal width binning with the number of bins in r 5 , 10 , 50 s .
Overall, we observed that equal width binning works the best with 10 bins.
Hence throughout this paper we perform equal width binning with 10 bins to include results with IM.
Average Conﬁdence (AC)
If we expect the classiﬁer to be argmax calibrated on the target then average conﬁdence is equal to accuracy of the classiﬁer.
Formally, by deﬁnition of argmax calibration of f on any distribution D , we have
[Equation]
Difference Of Conﬁdence
We estimate the error on target by subtracting difference of conﬁdences on source and target (as a distributional distance (Guillory et al., 2021)) from expected error on source distribution, i.e, DOC
[Equation]
E
[Equation]
P
[Equation]
p x qs .
This is referred to as DOC-Feat in (Guillory et al., 2021).
Generalized Disagreement Equality (GDE) Jiang et al.
(2021) proposed average disagreement of two models (trained on the same training set but with different initialization and/or different data ordering) as a approximate measure of accuracy on the underlying data, i.e.,
[Equation]
They show that marginal calibration of the model is sufﬁcient to have expected test error equal to the expected of average disagreement of two models where the latter expectation is also taken over the models used to calculate disagreement.
F D ETAILS ON THE D ATASET S ETUP
In our empirical evaluation, we consider both natural and synthetic distribution shifts.
We consider shifts on ImageNet (Russakovsky et al., 2015), CIFAR Krizhevsky & Hinton (2009), FMoWW
[Table]
ImageNet setup.
First, we consider synthetic shifts induced to simulate 19 different visual corruptions (e.g., shot noise, motion blur, pixelation etc.)
each with 5 different intensities giving us a total ofdatasets under ImageNet-C (Hendrycks & Dietterich, 2019).
Next, we consider natural distribution shifts due to differences in the data collection process.
In particular, we consider 3 ImageNetv2 (Recht et al., 2019) datasets each using a different strategy to collect test sets.
We also evaluate performance on images with artistic renditions of object classes, i.e., ImageNet-R
(Hendrycks et al., 2021) and
ImageNet-Sketch (Wang et al., 2019) with hand drawn sketch images.
Note that renditions dataset only contains 200 classes from ImageNet.
Hence, in the main paper we include results on ImageNet restricted to these 200 classes, which we call as ImageNet-200, and relegate results on ImageNet with 1 k classes to appendix.
We also consider B REEDS benchmark (Santurkar et al., 2020) in our evaluation to assess robustness to subpopulation shifts, in particular, to understand how accuracy estimation methods behave when novel subpopulations not observed during training are introduced.
B REEDS leverages class hierarchy in ImageNet to repurpose original classes to be the subpopulations and deﬁnes a classiﬁcation task on superclasses.
Subpopulation shift is induced by directly making the subpopulations present in the training and test distributions disjoint.
Overall, B REEDS benchmark contains 4 datasets
E NTITY -13, E NTITY -30, L IVING -17, N ON - LIVING -26, each focusing on different subtrees in the hierarchy.
To generate B REEDS dataset on top of ImageNet, we use the open source library: https: //github.com/MadryLab/BREEDS-Benchmarks .
We focus on natural and synthetic shifts as in ImageNet on same and different subpopulations in BREEDs.
Thus for both the subpopulation (same or novel), we obtain a total of 99 target datasets.
CIFAR setup.
Similar to the ImageNet setup, we consider (i) synthetic shifts (CIFAR-10-C) due to common corruptions; and (ii) natural distribution shift (i.e., CIFARv2 (Recht et al., 2018;
Torralba et al., 2008))
due to differences in data collection strategy on on CIFAR-10 (Krizhevsky & Hinton, 2009).
On CIFAR-100, we just have synthetic shifts due to common corruptions.
FMoW- W ILDS setup.
In order to consider distribution shifts faced in the wild, we consider FMoWWILDS (Koh et al., 2021; Christie et al., 2018) from W ILDS benchmark, which contains satellite images taken in different geographical regions and at different times.
We obtain 12 different OOD target sets by considering images between years 2013 – 2016 and 2016 – 2018 and by considering ﬁve geographical regions as subpopulations (Africa, Americas, Oceania, Asia, and Europe) separately and together.
RxRx1– W ILDS setup.
Similar to FMoW, we consider RxRx1-W ILDS (Taylor et al., 2019) from
W ILDS benchmark, which contains image of cells obtained by ﬂuorescent microscopy and the task is to genetic treatments the cells received.
We obtain 3 target datasets with shift induced by batch effects which make it difﬁcult to draw conclusions from data across experimental batches.
Amazon- W ILDS setup.
For natural language task, we consider Amazon-W ILDS
(Ni et al., 2019) dataset from W ILDS benchmark, which contains review text and the task is get a corresponding star rating from 1 to 5 .
We obtain 2 target datasets by considered shifts induced due to different set of reviewers than the training set.
CivilComments- W ILDS setup.
We also consider CivilComments-W ILDS (Borkan et al., 2019) from
W ILDS benchmark, which contains text comments and the task is to classify them for toxicity.
We obtain 18 target datasets depending on whether a comment mentions each of the 8 demographic identities male, female, LGBTQ, Christian, Muslim, other religions, Black, and White.
MNIST setup.
For completeness, we also consider distribution shifts on MNIST (LeCun et al., 1998) digit classiﬁcation as in the prior work (Deng & Zheng, 2021).
We use three real shifted datasets, i.e.,
USPS (Hull, 1994), SVHN (Netzer et al., 2011) and QMNIST (Yadav & Bottou, 2019).
G D ETAILS ON THE E XPERIMENTAL S ETUP
All experiments were run on NVIDIA Tesla V100 GPUs.
We used PyTorch (Paszke et al., 2019) for experiments.
Deep nets We consider a 4-layered MLP.
The PyTorch code for 4-layer MLP is as follows:
nn.
Sequential(nn.
Flatten(), nn.
Linear(inputdim,5000, bias=
True)
, nn.ReLU(), nn.Linear(5000, 5000, bias=
True)
, nn.ReLU(), nn.Linear(5000, 50, bias=
True)
, nn.ReLU(), nn.
Linear(50, num label, bias=True) )
We mainly experiment convolutional nets.
In particular, we use ResNet18 (He et al., 2016), ResNet50, and DenseNet121 (Huang et al., 2017) architectures with their default implementation in PyTorch.
Whenever we initial our models with pre-trained models, we again use default models in PyTorch.
Hyperparameters and Training details As mentioned in the main text we do not alter the standard training procedures and hyperparameters for each task.
We present results at ﬁnal model, however, we observed that the same results extend to an early stopped model as well.
For completeness, we include these details below:
CIFAR10 and
CIFAR100 We train DenseNet121 and ResNet18 architectures from scratch.
We use
SGD training with momentum of 0 .
9 for 300 epochs.
We start with learning rate 0 . 1 and decay it by multiplying it with 0 .
1 every 100 epochs.
We use a weight decay of 5 ´ 4 .
We use batch size of 200 .
For CIFAR10, we also experiment with the same models pre-trained on ImageNet.
ImageNet For training, we use Adam with a batch size of 64 and learning rate 0 . 0001 .
Due to huge size of ImageNet, we could only train two models needed for GDE for 10 epochs.
Hence, for relatively small scale experiments, we also perform experiments on ImageNet subset with 200 classes, which we call as ImageNet-200 with the same training procedure.
These 200 classes are the same classes as in ImageNet-R dataset.
This not only allows us to train ImageNet for 50 epochs but also allows us to use ImageNet-R in our testbed.
On the both the datasets, we observe a similar superioriy with ATC.
Note that all the models trained here were initialized with a pre-trained ImageNet model with the last layer replaced with random weights.
FMoW
- WILDS
For all experiments, we follow Koh et al.
and use two architectures
DenseNet121 and ResNet50, both pre-trained on ImageNet.
We use the Adam optimizer (Kingma &
Ba, 2014) with an initial learning rate of 10 ´ 4 that decays by 0 .
96 per epoch, and train for 50 epochs and with a batch size of 64 .
RxRx1- WILDS
For all experiments, we follow Koh et al.
and use two architectures
DenseNet121 and ResNet50, both pre-trained on ImageNet.
We use Adam optimizer with a learning rate of 1 e ´ 4 and L2-regularization strength of 1 e ´ 5 with a batch size of 75 for 90 epochs.
We linearly increase the learning rate for 10 epochs, then decreasing it following a cosine learning rate schedule.
Finally, we pick the model that obtains highest in-distribution validation accuracy.
Amazon-
WILDS
For all experiments, we follow Koh et al.
and
ﬁnetuned DistilBERTbase-uncased models (Sanh et al., 2019), using the implementation from Wolf et al.
(2020), and with the following hyperparameter settings: batch size 8 ; learning rate 1 e ´ 5 with the AdamW optimizer (Loshchilov & Hutter, 2017); L2-regularization strength 0 .
01 ; 3 epochs with early stopping; and a maximum number of tokens of 512 .
CivilComments- WILDS For all experiments, we follow Koh et al.
and ﬁne-tuned DistilBERTbase-uncased models (Sanh et al., 2019), using the implementation from Wolf et al.
(2020) and with the following hyperparameter settings: batch size 16 ; learning rate 1 e ´ 5 with the AdamW optimizer (Loshchilov & Hutter, 2017) for 5 epochs; L2-regularization strength 0 .
01 ; and a maximum number of tokens of 300 .
Living17 and Nonliving26 from B REEDS For training, we use SGD with a batch size of 128 , weight decay of 10 ´ 4 , and learning rate 0 .
Models were trained until convergence.
Models were trained for a total of 450 epochs, with 10-fold learning rate drops every 150 epochs.
Note that since we want to evaluate models for novel subpopulations no pre-training was used.
We train two architectures
DenseNet121 and ResNet50.
Entity13 and Entity30 from B REEDS For training, we use SGD with a batch size of 128 , weight decay of 10 ´ 4 , and learning rate 0 .
Models were trained until convergence.
Models were trained for a total of 300 epochs, with 10-fold learning rate drops every 100 epochs.
Note that since we want to evaluate models for novel subpopulations no pre-training was used.
We train two architectures
DenseNet121 and ResNet50.
MNIST For MNIST, we train a MLP described above with SGD with momentum 0 .
9 and learning rate 0 .
01 for 50 epochs.
We use weight decay of 10 ´ 5 and batch size as 200 .
We have a single number for CivilComments because it is a binary classiﬁcation task.
For multiclass problems, ATC-NE and ATC-MC can lead to different ordering of examples when ranked with the corresponding scoring function.
Temperature scaling on top can further alter the ordering of examples.
The changed ordering of examples yields different thresholds and different accuracy estimates.
However for binary classiﬁcation, the two scoring functions are the same as entropy (i.e. p log p
p q ` p 1 ´ p q log p p q ) has a one-to-one mapping to the max conf for p P r 0 , 1 s .
Moreover, temperature scaling also doesn’t change the order of points for binary classiﬁcation problems.
Hence for the binary classiﬁcation problems, both the scoring functions with and without temperature scaling yield the same estimates.
We have made this clear in the updated draft.
Implementation for Temperature Scaling
We use temperature scaling implementation from https://github.com/kundajelab/abstention .
We use validation set (the same we use to obtain ATC threshold or DOC source error estimate) to tune a single temperature parameter.
For vision datasets, we train a DenseNet model with the exception of FCN model for MNIST dataset.
For language datasets, we ﬁne-tune a DistilBERT-base-uncased model.
For each of these models, we use the exact same setup as described Sec.
G. Importantly, to obtain errors on the same scale, we rescale all the errors by subtracting the error of Average Conﬁdence
H S UPPLEMENTARY R ESULTS H. CIFAR PRETRAINING A BLATION
[Figure] [Figure]
y
CIFAR10-Pretraining
[Figure]
H. B REEDS RESULTS WITH REGRESSION MODEL
[Figure]
Dataset
[Figure]
o ﬁt) DOC (w ﬁt) ATC-MC (Ours) (w/o ﬁt)
L IVING24 .
N ONLIVING29 .18 .19 .
E NTITY22 .8 .8 .
[Table]
[Figure]
i c t
[Figure]
d
A
[Figure]
u r
[Figure]
y
[Figure]
P r e d i c t e d
A c c u r a c y
[Figure]
P r e d i c t e d
A cc u r a c y
[Figure] [Figure]
i c t e d
A
[Figure]
u r a
[Figure]
y
[Figure]
P r e d i c t e d
A c c u r a c y
[Figure]
d i c t e d
A cc u r a c y
[Figure]
[Table] [Table]
Post T denotes use of TS calibration on source.
Across all observe that ATC achieves superior
(lower is better
TS doesn’t
[Table]