A SSESSING G ENERALIZATION VIA D ISAGREEMENT
Vaishnavh Nagarajan ∗† vaishnavh@
Christina Baek, J. Zico Kolter { kbaek,zkolter } @ .cmu.edu
A BSTRACT
We empirically show that the test error of deep networks can be estimated by training the same architecture on the same training set but with two different runs of Stochastic Gradient Descent (SGD), and then measuring the disagreement rate between the two networks on unlabeled test data .
This builds on — and is a stronger version of — the observation in Nakkiran & Bansal (2020), which requires the runs to be on separate training sets.
We further theoretically show that this peculiar phenomenon arises from the well-calibrated nature of ensembles of
SGD-trained models.
This ﬁnding not only provides a simple empirical measure to directly predict the test error using unlabeled test data but also establishes a new conceptual connection between generalization and calibration
I NTRODUCTION
Consider the following intriguing observation made in Nakkiran & Bansal (2020).
Train two networks of the same architecture to zero training error on two independently drawn datasets S 1 and
S 2 of the same size.
Both networks would achieve a test error (or equivalently, a generalization gap) of about the same value, denoted by (cid:15) .
Now, take a fresh unlabeled dataset U and measure the rate of disagreement of the predicted label between these two networks on U .
Based on the triangle inequality, one can quickly surmise that this disagreement rate could lie anywhere between 0 and 2 (cid:15) .
However, across various training set sizes and for various models like neural networks, kernel SVMs and decision trees, Nakkiran & Bansal (2020) (or N&B’20 in short) report that the disagreement rate not only linearly correlates with the test error (cid:15) , but nearly equals (cid:15) (see ﬁrst two plots in Fig 1).
What brings about this unusual equality?
Resolving this open question from N&B’20 could help us identify fundamental patterns in how neural networks make errors.
That might further shed insight into generalization and other poorly understood empirical phenomena in deep learning.
The ﬁrst two plots correspond to pairs of networks trained on independent datasets, and in the last two plots, on the same
The details are described in Sec 3.
In this work, we ﬁrst identify a stronger observation.
Consider two neural networks trained with the same hyperparameters and the same dataset , but with different random seeds (this could take the form e.g., of the data being presented in different random orders and/or by using a different random initialization of the network weights).
We would expect the disagreement rate in this setting to be much smaller than in N&B’20, since both models see the same data.
Yet, this is not the case: we observe on the SVHN (Netzer et al., 2011), CIFAR-10/100 (Krizhevsky et al., 2009) datasets, and for variants of Residual Networks (He et al., 2016) and Convolutional Networks (Lin et al., 2013), that the disagreement rate is still approximately equal to the test error (see last two plots in Fig 1), only slightly deviating from the behavior in N&B’20.
In fact, while N&B’20 show that the disagreement rate captures signiﬁcant changes in test error with varying training set sizes, we highlight a much stronger behavior: the disagreement rate is able to capture even minute variations in the test error under varying hyperparameters like width, depth and batch size .
Furthermore, we show that under certain training conditions, these properties even hold on many kinds of out-of-distribution data in the PACS dataset (Li et al., 2017), albeit not on all kinds.
The above observations not only raise deeper conceptual questions about the behavior of deep networks but also crucially yield a practical beneﬁt.
In particular, our disagreement rate does not require fresh labeled data (unlike the rate in N&B’20) and rather only requires fresh unlabeled data.
Hence, ours is a more meaningful and practical estimator of test accuracy (albeit, a slightly less accurate estimate at that).
Indeed, unsupervised accuracy estimation is valuable for real-time evaluation of models when test labels are costly or unavailable to due to privacy considerations (Donmez et al., 2010; Jaffe et al., 2015).
While there are also many other measures that correlate with generalization without access to even unlabeled test data (Jiang et al., 2018;
Yak et al., 2019; Jiang et al., 2020b;a;
Natekar & Sharma, 2020; Unterthiner et al., 2020), these require (a) computing intricate proportionality constants and (b) knowledge of the neural network weights/representations.
Disagreement, however, provides a direct estimate, and works even with a black-box model, which makes it practically viable when the inner details of the model are unavailable due to privacy concerns.
In the second part of our work, we theoretically investigate these observations.
Informally stated, we prove that if the ensemble learned from different stochastic runs of the training algorithm (e.g., across different random seeds) is well-calibrated (i.e., the predicted probabilities are neither overconﬁdent nor under-conﬁdent), then the disagreement rate equals the test error (in expectation over the training stochasticity).
Indeed, such kinds of SGD-trained deep network ensembles are known to be naturally calibrated in practice (Lakshminarayanan et al., 2017).
While we do not prove why calibration holds in practice, the fact the condition in our theorem is empirically satisﬁed implies that our theory offers a valuable insight into the practical generalization properties of deep networks.
Overall, our work establishes a new connection between generalization and calibration via the idea of disagreement.
This has both theoretical and practical implications in understanding generalization and the effect of stochasticity in SGD.
To summarize, our contributions are as follows:
We prove that for any stochastic learning algorithm, if the algorithm leads to a well-calibrated ensemble on a particular data distribution, then the ensemble satisﬁes the Generalization Disagreement Equality 1 (GDE) on that distribution, in expectation over the stochasticity of the algorithm .
Notably, our theory is general and makes no restrictions on the hypothesis class, the algorithm, the source of stochasticity, or the test distributions (which may be different from the training distribution).
We empirically show that for Residual Networks (He et al., 2016), convolutional neural networks (Lin et al., 2013) and fully connected networks, and on CIFAR-10/100 (Krizhevsky et al., 2009) and SVHN (Netzer et al., 2011), GDE is nearly satisﬁed, even on pairs of networks trained on the same data with different random seeds.
This yields a simple method that in practice accurately estimates the test error using unlabeled data in these settings.
We also empirically show that the corresponding ensembles are well-calibrated (according to our particular deﬁnition of calibration) in practice.
We do not, however, theoretically prove why this calibration holds.
We present preliminary observations showing that GDE is approximately satisﬁed even for certain distribution shifts within the PACS (Li et al., 2017) dataset.
This implies that the disagreement rate can be a promising estimator even for out-of-distribution accuracy.
We empirically ﬁnd that different sources of stochasticity in SGD are almost equally effective in terms of their effect on GDE and calibration of deep models trained with SGD.
We also explore the effect of pre-training on these phenomena.
R ELATED W ORKS
Understanding and predicting generalization.
Conventionally, generalization in deep learning has been studied through the lens of PAC-learning (Vapnik, 1971; Valiant, 1984).
Under this framework, generalization is roughly equivalent to bounding the size of the search space of a learning algorithm.
Representative works in this large area of research include Neyshabur et al.
(2014; 2017; 2018); Dziugaite & Roy (2017); Bartlett et al.
; Nagarajan & Kolter (2019b;c); Krishnan et al.
Several works have questioned whether these approaches are truly making progress toward understanding generalization in overparameterized settings (Belkin et al., 2018; Nagarajan & Kolter, 2019a; Jiang et al., 2020b; Dziugaite et al., 2020).
Subsequently, recent works have proposed unconventional ways to derive generalization bounds (Negrea et al., 2020; Zhou et al., 2020;
Garg et al., 2021).
Indeed, even our disagreement-based estimate marks a signiﬁcant departure from complexity-based approaches to generalization bounds.
Of particular relevance here is Garg et al.
(2021) who also leverage unlabeled data.
Their bound requires modifying the original training set and then performing a careful early stopping, and is thus inapplicable to (and becomes vacuous for) interpolating models.
While our estimate applies to the original training process, our guarantee applies only if we know a priori that the training procedure results in well-calibrated ensembles.
Finally, it is worth noting that much older work (Madani et al., 2004) has provided bounds on the test error as a function of (rather than based on a “direct estimate” of) disagreement.
However, these require the two runs to be on independent training sets.
While there has been research in unsupervised accuracy estimation (Donmez et al., 2010; Platanios et al., 2017; Jaffe et al., 2015; Steinhardt & Liang, 2016; ElSahar & Gall ´ e, 2019; Schelter et al., 2020; Chuang et al., 2020), the focus has been on out-of-distribution and/or specialized learning settings.
Hence, they require specialized training algorithms or extra information about the tasks.
Concurrent work Chen et al.
(2021) here has made similar discoveries regarding estimating accuracy via agreement, although their focus is more algorithmic than ours. We discuss this in Appendix A.
Reducing churn. A line of work has looked at reducing disagreement (termed there as “churn”) to make predictions more reproducible and easy-to-debug.
(Milani Fard et al., 2016; Jiang et al., 2021; Bhojanapalli et al., 2021). Bhojanapalli et al. (2021) further analyze how different sources of stochasticity can lead to non-trivial disagreement rates.
Calibration.
Calibration of a statistical model is the property that the probability obtained by the model reﬂects the true likelihood of the ground truth (Murphy & Epstein, 1967; Dawid, 1982).
A well-calibrated model provides an accurate conﬁdence on its prediction which is paramount for high-stake decision making and interpretability.
In the context of deep learning, several works (Guo et al., 2017; Lakshminarayanan et al., 2017; Fort et al., 2019; Wu & Gales, 2021; Bai et al., 2021;
Mukhoti et al., 2021) have found that while individual neural networks are usually over-conﬁdent about their predictions, ensembles of several independently and stochastically trained models tend to be naturally well-calibrated.
In particular, two types of ensembles have typically been studied, depending on whether the members are trained on independently sampled data also called bagging (Breiman, 1996) or on the same data but with different random seeds (e.g., different random initialization and data ordering) also called deep ensembles (Lakshminarayanan et al., 2017).
The latter typically achieves better accuracy and calibration (Nixon et al., 2020).
On the theoretical side, Allen-Zhu & Li (2020) have studied why deep ensembles outperform individual models in terms of accuracy.
Other works studied post-processing methods of calibration (Kumar et al., 2019), established relationships to conﬁdence intervals (Gupta et al., 2020), and derived upper bounds on calibration error either in terms of sample complexity or in terms of the accuracy (Bai et al., 2021;
Ji et al., 2021; Liu et al., 2019; Jung et al., 2020; Shabat et al., 2020).
The discussion in our paper complements the above works in multiple ways.
First, most works within the machine learning literature focus on top-class calibration , which is concerned only with the conﬁdence level of the top predicted class for each point.
The theory in our work, however, requires looking at the conﬁdence level of the model aggregated over all the classes.
We then empirically show that SGD ensembles are well-calibrated even in this class-aggregated sense.
Furthermore, we carefully investigate what sources of stochasticity result in well-calibrated ensembles.
Finally, we provide an exact formal relationship between generalization and calibration via the notion of disagreement, which is fundamentally different from existing theoretical calibration bounds.
Empirical phenomena in deep learning.
Broadly, our work falls in the area of research on identifying & understanding empirical phenomena in deep learning (Sedghi et al., 2019), especially in the context of overparameterized models that interpolate.
Some example phenomena include the generalization puzzle (Zhang et al., 2017; Neyshabur et al., 2014), double descent (Belkin et al., 2019; Nakkiran et al., 2020), and simplicity bias (Kalimeris et al., 2019; Arpit et al., 2017).
As stated earlier, we particularly build on N&B’20’s empirical observation of the Generalization Disagreement
Equality (GDE) in pairs of models trained on independently drawn datasets.
They provide a proof of GDE for 1-nearest-neigbhor classiﬁers under speciﬁc distributional assumptions, while our result is different, and much more generic.
Due to space constraints, we defer a detailed discussions of the relationship between our works in Appendix A.
D ISAGREEMENT T RACKS G ENERALIZATION E RROR
We demonstrate on various datasets and architectures that the test error can be estimated directly by training two runs of SGD and measuring their disagreement on an unlabeled dataset.
Importantly, we show that the disagreement rate can track even minute variations in the test error induced by varying hyperparameters.
Remarkably, this estimate does not require an independent labeled dataset.
Notations.
Let h : X → [ K ] denote a hypothesis from a hypothesis space H , where [ K ] denotes the set of K labels { 0 , 1 , . .
, K − 1 } .
Let D be a distribution over X × [ K ] .
We will use ( X, Y ) to denote the random variable with the distribution D , and ( x, y ) to denote speciﬁc values it can take.
Let A be a stochastic training algorithm that induces a distribution H A over hypotheses in H .
Let h, h (cid:48) ∼ H
A denote random hypotheses output by two independent runs of the training procedure.
We note that the stochasticity in A could arise from any arbitrary source.
This may arise from either the fact that each h is trained on a random dataset drawn from D or even a completely different distribution D (cid:48) .
The stochasticity could also arise from merely a different random initialization or data ordering.
Next, we denote the test error and disagreement rate for hypotheses h, h (cid:48) ∼ H
A by:
TestErr D ( h ) (cid:44) E D [
[Equation]
[ h ( X ) (cid:54) = Y ]] and Dis D ( h, h (cid:48) ) (cid:44) E D [ 1 [ h ( X
[Equation]
(cid:54) = h (cid:48)
( X )]] .
[Equation]
Let ˜ h denote the “ensemble” corresponding to h ∼ H A .
In particular, deﬁne
[Equation]
to be the probability value (between [0 , 1] ) given by the ensemble ˜ h for the k th class.
Note that the output of ˜ h is not a one-hot value based on plurality vote.
Main Experimental Setup.
We report our main observations on variants of Residual Networks, convolutional neural networks and fully connected networks trained with Momentum SGD on
CIFAR-10/100, and SVHN.
Each variation of the ResNet has a unique hyperparameter conﬁguration (See Appendix C.1 for details) and all models are (near) interpolating.
For each hyperparameter setting, we train two copies of models which experience two independent draws from one or more sources of stochasticity, namely 1.
random initialization (denoted by Init ) and/or 2.
ordering of a ﬁxed training dataset ( Order ) and/or 3.
different (disjoint) training data ( Data ).
We will use the term Diff to denote whether a source of stochasticity is “on”.
For example, DiffInit means that the two models have different initializations but see the same data in the same order.
In DiffOrder , models share the same initialization and see the same data, but in different orders.
In DiffData , the models share the initialization, but see different data.
In AllDiff , the two models differ in both data and in initialization (If the two models differ in data, the training data is split into two disjoint halves to ensure no overlap.).
The disagreement rate between a pair of models is computed as the proportion of the test data on which the (one-hot) predictions of the two models do not match.
Observations.
We provide scatter plots of test error of the ﬁrst run ( y ) vs disagreement error between the two runs ( x ) for CIFAR-10, SVHN and CIFAR-100 in Figures 1, 2 and 3 respectively (and for CNNs on CIFAR-10 in Fig 10).
Naively, we would expect these scatter plots to be arbitrarily distributed anywhere between y = 0 .
5 x (if the errors of the two models are disjoint) and x =
(if the errors are identical).
However, in all these scatter plots, we observe that test error and disagreement error lie very close to the diagonal line y
= x across different sources of stochasticity, while only slightly deviating in DiffInit/Order .
In particular, in AllDiff and DiffData , the points typically lie between y = x
and y = 0 .
9 x while in DiffInit and DiffOrder , the disagreement rate drops slightly (since the models are trained on the same data) and so the points typically lie between y = x and y = 1 .
3 x .
We quantify correlation via the R 2 coefﬁcient and Kendall’s
Ranking coefﬁcient ( tau ) reported on top of each scatter plot.
Indeed, we observe that these quantities are high in all the settings, generally above 0 . 85 .
If we focus only on the data-augmented or the non-data-augmented models, R 2 tends to range a bit lower, around 0 .
7 and τ around 0.6 (see Appendix D.8).
The positive observations about DiffInit and DiffOrder are surprising for two reasons.
First, when the second network is trained on the same dataset, we would expect its predictions to be largely aligned with the original network — naturally, the disagreement rate would be negligible, and the equality observed in N&B’20 would no longer hold.
Furthermore, since we calculate the disagreement rate without using a fresh labeled dataset, we would expect disagreement to be much less predictive of test error when compared to N&B’20.
Our observations defy both these expectations.
There are a few more noteworthy aspects.
In the low data regime where the test error is high, we would expect the models to be much less well-behaved.
However, consider the CIFAR-100 plots (Fig 3), and additionally, the plots in Fig 4 where we train on CIFAR-10 with just 2000 training points.
In both settings the network suffers an error as high as 0 .
5 to 0 .
Yet, we observe a behavior similar to the other settings (albeit with some deviations) — the scatter plot lies in y =
(1 ± 0 .
1) x (for AllDiff and DiffData ) and in y =
(1 ± 0 . 3) x (for DiffInit/Order ), and the correlation metrics are high.
Similar results were established in N&B’20 for AllDiff and DiffData .
Finally, it is important to highlight that each scatter plot here corresponds to varying certain hyperparameters that cause only mild variations in the test error.
Yet, the disagreement rate is able to capture those variations in the test error.
This is a stronger version of the ﬁnding in N&B’20 that disagreement captures larger variations under varying dataset size.
Effect of distribution shift and pre-training We study these observations in the context of the
PACS dataset, a popular domain generalization benchmark with four distributions, Photo ( P in short),
Art ( A ), Cartoon ( C ) and Sketch ( S ), all sharing the same 7 classes.
On any given domain, we train pairs of ResNet50 models.
Both models are either randomly initialized or ImageNet (Deng et al., 2009) pre-trained.
We then evaluate their test error and disagreement on all the domains.
As we see in Fig 5, the surprising phenomenon here is that there are many pairs of source-target domains where GDE is approximately satisﬁed despite the distribution shift.
Notably, for pre-trained models, with the exception of three pairs of source-target domains (namely, ( P , C ) , ( P , S ) , ( S , P ) ), GDE is satisﬁed approximately.
The other notable observation is that under distribution shift, pre-trained models can satisfy GDE, and often better than randomly initialized models.
This is counter-intuitive, since we would expect pre-trained models to be strongly predisposed towards speciﬁc kinds of features, resulting in models that disagree rarely.
See Appendix C.1 for hyperparameter details.
[Figure]
C ALIBRATION I MPLIES T HE GDE
We now formalize our main observation, like it was formalized in N&B’20 (although with minor differences to be more general).
In particular, we deﬁne “the Generalization Disagreement Equality” as the phenomenon that the test error equals the disagreement rate in expectation over h ∼ H A .
Deﬁnition 4.1.
The stochastic learning algorithm
A satisﬁes the Generalization Disagreement
Equality (GDE) on the distribution D if,
[Equation]
Note that the deﬁnition does not imply that the equality holds for each pair of h, h (cid:48) (which we observed empirically).
However, for simplicity, we will stick to the above “equality in expectation” as it captures the essence of the underlying phenomenon while being easier to analyze.
To motivate why proving this equality is non-trivial, let us look at the most natural hypothesis that N&B’ identify.
Imagine that all datapoints ( x, y ) are one of two types: (a) the datapoint is so “easy” that w.p. 1 over h ∼ H A , h ( x ) =
y (b) the datapoint is so “hard” that h ( x ) corresponds to picking a label uniformly at random.
In such a case, with a simple calculation, one can see that the above equality would hold not just in expectation over D , but even point-wise: for each x , the disagreement on x in expectation over H A would equal the error on x in expectation over H A (namely ( K − 1) /K
if x is hard, and 0 if easy).
Unfortunately, N&B’20 show that in practice, a signiﬁcant fraction of the points have disagreement larger than error and another fraction have error larger than disagreement (see Appendix D.4).
Surprisingly though, there is a delicate balance between these two types of points such that overall these disparities cancel each other out giving rise to the GDE.
What could create this delicate balance?
We identify that this can arise from the fact that the ensemble ˜ h is well-calibrated .
Informally, a well-calibrated model is one whose output probability for a particular class (i.e., the model’s “conﬁdence”) is indicative of the probability that the ground truth class is indeed that class (i.e., the model’s “accuracy”).
There are many ways in which calibration can be formalized.
Below, we provide a particular formalism called class-wise calibration.
Deﬁnition 4.2.
The ensemble model ˜
h satisﬁes class-wise calibration on D if for any conﬁdence value q ∈ [0 , 1] and for any class
k ∈
[ K ] ,
[Equation]
Next, we show that if the ensemble is class-wise calibrated on the distribution D , then GDE does hold on D .
Note however that shortly we show a more general result where even a weaker notion of calibration is sufﬁcient to prove GDE.
But since this stronger notion of calibration is easier to understand, and the proof sketch for this captures the key intuition of the general case, we will focus on this ﬁrst in detail.
It is worth emphasizing that besides requiring well-calibration on the (test) distribution, all our theoretical results are general.
We do not restrict the hypothesis class (it need not necessarily be neural networks), or the test/training distribution (they can be different, as long as calibration holds in the test distribution ), or where the stochasticity comes from (it need not necessarily come from the random seed or the data).
Theorem 4.1.
Given a stochastic learning algorithm A , if its corresponding ensemble ˜
h
satisﬁes class-wise calibration on D , then A satisﬁes the Generalization Disagreement Equality on D .
Proof.
(Sketch for binary classiﬁcation.
Details for full multi-class classiﬁcation are deferred to
App.
B.2.)
Let X q correspond to a “conﬁdence level set” in that X
q = { X ∈ X | ˜
h 0 ( X ) = q } .
Our key idea is to show that for a class-wise calibrated ensemble, GDE holds within each conﬁdence level set i.e., for each q ∈ [0 , 1] , the (expected) disagreement rate equals test error for the distribution D restricted to the support X q .
Since X is a combination of these level sets, it automatically follows that GDE holds over D .
It is worth contrasting this proof idea with the easy-hard explanation which requires showing that GDE holds point-wise, rather than conﬁdence-level-set-wise.
Now, let us calculate the disagreement on X q .
For any ﬁxed x in X q , the disagreement rate in expectation over h, h (cid:48) ∼ H
A corresponds to q (1 − q )
+(1 − q ) q = 2 q (1 − q ) .
This is simply the sum of the probability of the events that h predicts 0 and h (cid:48) predicts 1 , and vice versa.
Next, we calculate the test error on X q .
At any x , the expected error equals ˜ h 1 − y ( x ) .
From calibration, we have that exactly q fraction of X q has the true label 0 .
On these points, the error rate is ˜ h 1 ( x ) = 1 − q .
On the remaining 1 − q fraction, the true label is 1 , and hence the error rate on those is ˜ h 0 ( x ) = q .
The total error rate across both the class 0 and class 1 points is therefore q (1 − q )+(1 − q ) q = 2 q (1 − q ) .
Intuition.
Even though the proof is fairly simple, it may be worth demystifying it a bit further.
In short, a calibrated classiﬁer “knows” how much error it commits in different parts of the distribution i.e., over points where it has a conﬁdence of q , it commits an expected error of 1 − q .
With this in mind, it is easy to see why it is even possible to predict the test error without knowing the test labels: we can simply average the conﬁdence values of the ensemble to estimate test accuracy.
The expected disagreement error provides an alternative but less obvious route towards estimating test performance.
The intuition is that within any conﬁdence level set of a calibrated ensemble, the marginal distribution over the ground truth labels becomes identical to the marginal distribution over the one-hot predictions sampled from the ensemble.
Due to this, measuring the expected disagreement of the ensemble against itself becomes equivalent to measuring the expected disagreement of the ensemble against the ground truth .
The latter is nothing but the ensemble’s test error.
What is still surprising though is that in practice, we are able to get away with predicting test error by computing disagreement for a single pair of ensemble members — and this works even though an ensemble of two models is not well-calibrated, as we will see later in Table 1.
This suggests that the variance of the disagreement and test error (over the stochasticity of H A ) must be unusually small; indeed, we will empirically verify this in Table 1.
In Corollary B.1.1, we present some preliminary discussion on why the variance could be small, leaving further exploration for future work.
4. A MORE GENERAL RESULT : CLASS - WISE TO CLASS - AGGREGATED CALIBRATION
We will now show that GDE holds under a more relaxed notion of calibration, which holds “on average” over the classes rather than individually for each class.
Indeed, we demonstrate in a later section (see Appendix D.7) that this averaged notion of calibration holds more gracefully than class-wise calibration in practice.
Recall that in class-wise calibration we look at the conditional probability p
( Y = k | ˜
h k ( X ) = q ) for each k .
Here, we will take an average of these conditional probabilities by weighting the k th probability by p (˜ h k ( X ) = q ) .
The result is the following deﬁnition:
Deﬁnition 4.3.
The ensemble ˜ h
satisﬁes class-aggregated calibration on D if for each q ∈ [0 , 1] ,
[Equation]
Intuition.
The denominator here corresponds to the points where some class gets conﬁdence value q ; the numerator corresponds to the points where some class gets conﬁdence value q and that class also happens to be the ground truth.
Note however both the proportions involve counting a point x multiple times if ˜ h k ( x ) = q for multiple classes k .
In Appendix B.5, we discuss the relation between this new notion of calibration to existing deﬁnitions.
In Appendix B.1 Theorem B.1, we show that the above weaker notion of calibration is sufﬁcient to show GDE.
The proof of this theorem is a nontrivial generalization of the argument in the proof sketch of Theorem 4.1, and Theorem 4.1 follows as a straightforward corollary since class-wise calibration implies class-aggregated calibration.
Deviation from calibration.
For generality, we would like to consider ensembles that do not satisfy class-aggregated calibration precisely.
How much can a deviation from calibration hurt GDE?
To answer this question, we quantify calibration error as follows:
Deﬁnition 4.4.
The Class Aggregated Calibration Error (CACE) of an ensemble ˜ h on D is
[Equation]
In other words, for each conﬁdence value q , we look at the absolute difference between the left and right hand sides of Deﬁnition 4.3, and then weight the difference by the proportion of instances where a conﬁdence value of q is achieved.
It is worth keeping in mind that, while the absolute difference term lies in [0 , 1] , the weight terms alone would integrate to a value of K .
Therefore,
CACE D (˜ h ) can lie anywhere in the range [0 , K ] .
Note that CACE is different from the “expected calibration error (ECE)”
(Naeini et al., 2015; Guo et al., 2017) commonly used in the machine learning literature, which applies only to top-class calibration.
We show below that GDE holds approximately when the calibration error is low (and naturally, as a special case, holds perfectly when calibration error is zero).
The proof is deferred to Appendix B.3.
Theorem 4.2.
For any algorithm A , | E H A [ Dis D ( h, h (cid:48) )]
− E H A [ TestErr D ( h )] | ≤ CACE D
(˜ h ) .
Remark.
All our results hold more generally for any probabilistic classiﬁer ˜ h .
For example, if ˜ h was an individual calibrated neural network whose predictions are given by softmax probabilities, then
GDE holds for the neural network itself: the disagreement rate between two independently sampled one-hot predictions from that network would equal the test error of the softmax predictions.
E MPIRICAL A NALYSIS OF C LASS - AGGREGATED C ALIBRATION
Empirical evidence for theory.
As stated in the introduction, it is a well-established observation that ensembles of SGD trained models provide good conﬁdence estimates (Lakshminarayanan et al., 2017).
However, typically the output of these ensembles correspond to the average softmax probabilities of the individual models, rather than an average of the top-class predictions.
Our theory is however based upon the latter type of ensembles.
Furthermore, there exists many different evaluation metrics for calibration in literature, while we are particularly interested in the precise deﬁnition we have in Deﬁnition 4.3.
We report our observations keeping these considerations in mind.
In Figure 6, 7 and 8, we show that SGD ensembles do nearly satisfy class-aggregated calibration for all the sources of stochasticity we have considered .
In each plot, we report the conditional probability in the L.H.S of Deﬁnition 4.3 along the y axis and the conﬁdence value q along the x axis.
We observe that the plot closely follows the x = y line.
In fact, we observe that calibration holds across different sources of stochasticity.
We discuss this aspect in more detail in Appendix B.6.
[Table] For a more precise quantiﬁcation of how well calibration captures GDE, we also look at our notion of calibration error, namely CACE, which also acts as an upper bound on the difference between the test error and the disagreement rate.
We report CACE averaged over 100 models in Table(for CIFAR-10) and Table 3 (for CIFAR-100).
Most importantly, we observe that the CACE across different stochasticity settings correlates with the actual gap between the test error and the disagreement rate.
In particular, CACE for AllDiff/DiffData are about 2 to 3 times smaller than that for
DiffInit/Order , paralleling the behavior of | TestErr − Dis | in these settings.
Caveats.
While we believe our work provides a simple theoretical insight into how calibration leads to GDE, there are a few gaps that we do not address.
First, we do not provide a theoretical characterization of when we can expect good calibration (and hence, when we can expect GDE).
Therefore, if we do not know a priori that calibration holds in a particular setting, we would need labeled test data to verify if CACE is small.
This would defeat the purpose of using unlabeleddata-based disagreement to measure the test error.
(Thankfully, in in-distribution settings, it seems like we may be able to assume calibration for granted.)
Next, our theory sheds insight into why GDE holds in expectation over training stochasticity.
However, it is surprising that in practice the disagreement rate (and the test error) for a single pair of models lies close to this expectation.
This occurs even though two-model-ensembles are poorly calibrated (see Tables 1 and 3).
Finally, while CACE is an upper bound on the deviation from GDE, in practice CACE is only a loose bound, which could either indicate a mere lack of data/models or perhaps that our theory can be further reﬁned.
C ONCLUSION
Building on Nakkiran & Bansal (2020), we observe that remarkably, two networks trained on the same dataset, tend to disagree with each other on unlabeled data nearly as much as they disagree with the ground truth.
We’ve also theoretically shown that this property arises from the fact that
SGD ensembles are well-calibrated.
Broadly, these ﬁndings contribute to the larger pursuit of identifying and understanding empirical phenomena in deep learning.
Future work could shed light on why different sources of stochasticity surprisingly have a similar effect on calibration.
It is also important for future work in uncertainty estimation and calibration to develop a precise and exhaustive characterization of when calibration and GDE would hold.
On a different note, we hope our work inspires other novel ways to leverage unlabeled data to estimate generalization and also further cross-pollination of ideas between research in generalization and calibration.
A R ELATED WORK
Here we provide a more detailed discussion of some related work.
First we discuss how our results are distinct from and/or complement their other relevant ﬁndings.
First, N&B’20 provide a proof of
GDE speciﬁc to 1-Nearest Neighbor models trained on two independent datasets.
Our result does not restrict the hypothesis class, the algorithm or its stochasticity.
Second, N&B’20 identify a notion they term as “feature calibration” which can be thought of as a generalized version of calibration.
However, the instantiations of feature calibration that they empirically study are signiﬁcantly different from the standard notion we study.
Furthermore, they treat GDE and feature calibration as independent phenomena.
Conversely, we show that calibration in the standard sense implies GDE.
Finally, in their Appendix D.7.1, N&B’20 do report studies of ensembles where the members are trained on the same data.
But this is in an altogether independent context — the GDE-related experiments in N&B’20 are all reported only on ensembles of members trained on different data.
Hence, overall, their empirical results do not imply our GDE results.
In a concurrent work, Chen et al.
(2021) make similar discoveries regarding estimating accuracy via agreement by developing a sophisticated ensemble-learning algorithm and make connections to calibration.
Overall, their focus is more algorithmic, while our focus is primarily on understanding the nature of this phenomenon.
Furthermore, in comparison to all these works which focus on out-of-distribution settings, since we focus on the in-distribution setting, we are able to identify a much simpler approach that works with vanilla-trained blackbox models e.g., we examine the effects of different kinds of stochasticity, we introduce ensembles only as a vehicle to understand the phenomenon theoretically (rather than an algorithmic object), and we prove how GDE holds under certain novel notions of calibration weaker than the existing ones.
B A PPENDIX : A DDITIONAL THEORETICAL DISCUSSION B. P ROOF OF T HEOREM B.
We will now prove Theorem B.1 which states that if the ensemble ˜
h satisﬁes class-aggregated calibration, then the expected test error equals the expected disagreement rate.
The basic intuition behind why class-aggregated calibration implies GDE is similar to the argument for class-wise calibration — we can similarly argue that within a conﬁdence level set, the distribution over the predicted classes matches the distribution over the ground truth labels; therefore, measuring disagreement of the ensemble against the ensemble itself boils down to measuring disagreement against the ground truth.
However, the conﬁdence level sets here can involve counting the same data point multiple times, and this nuance needs to be handled carefully as can be seen from the proof in the appendix.
Theorem B.1.
Given a stochastic learning algorithm A , if its corresponding ensemble ˜ h satisﬁes class-aggregated calibration (Deﬁnition 4.3) on D , then A satisﬁes GDE on D (Deﬁnition 4.1).
Proof.
Before we delve into the details of the proof, we will outline the high-level proof idea which is similar to that proof sketch presented in Section 4.
First, we express the expected test error in terms of an integral over the conﬁdence values (Eq 23) and then plug in the deﬁnition for classaggregated calibration to get Eq 25.
For expected disagreement rate, we can analogously express it as an integral over the conﬁdence values, because the models are independent and the expectation naturally produces the conﬁdence values.
Note that the calibration assumption is not used for deriving the expected disagreement rate and the ﬁnal result (Eq 44) is equal to the expected test error (Eq 25), which completes our proof.
We’ll ﬁrst simplify the expected test error and then proceed to simplifying the expected disagreement rate to the same quantity.
Test Error Recall that the expected test error (which we will denote as ETE for short) corresponds to E
H
A [ p ( h ( X ) (cid:54) =
Y | h )] .
ETE (cid:44) E h ∼ H
A [ p ( h ( X ) (cid:54) =
Y | h )] (7)
[Equation]
exchanging expectations by Fubini’s theorem
[Equation]
For our further simpliﬁcations
[Equation]
we’ll explicitly deal with integrals rather than expectations
[Equation]
so we get,
[Equation]
We’ll also introduce ˜ h
( X ) as a r.v. as,
[Equation]
Over the next few steps
[Equation]
we’ll get rid of the integral over x .
First, splitting the joint distribution over the three r
[Equation]
v.s by conditioning on the latter two,
[Equation]
In the next few steps
[Equation]
we’ll simplify the integral over q by marginalizing over all but the k th dimension.
First
[Equation]
we rewrite the joint distribution of ˜ h
( X ) in terms of its K components.
For any k , let ˜
h
− k
( X ) and q − k denote the K − 1 dimensions of both vectors excluding their k th dimension.
Then,
ETE =
[Equation]
Rewriting q k as just q ,
[Equation] Finally
[Equation]
we have from the calibration in aggregate assumption that (cid:80) K − 1 k =0 p
( Y = k, ˜ h
k ( X ) = q ) = q
[Equation]
=
[Equation]
p (
[Equation]
h k ( X ) = q ) (Deﬁnition 4.3).
So, applying this, we get
[Equation]
Rearranging,
[Equation]
Disagreement Rate
The expected disagreement rate (denoted by EDR in short) is given by the probability that two i.i [Equation] d samples from ˜ h disagree with each other over draws of input from D , taken in expectation over draws from H A .
That is,
[Equation]
[Equation]
[Equation]
(exchanging expectations by Fubini’s Theorem)
[Equation]
Over the next few steps, we’ll write this in terms of ˜ h rather than h and h (cid:48) .
[Equation]
swapping the expectation and
[Equation]
summation)
[Equation]
(since h and h (cid:48) are i.i.d samples from H A )
[Equation] [Equation]
From here, we’ll deal with integrals instead of expectations.
[Equation]
Let us introduce the random variable ˜ h
( X ) as,
[Equation]
[Equation]
In the next few steps
[Equation]
we’ll get rid of the integral over x .
First [Equation] we split the joint distribution as,
[Equation]
bring to the front
[Equation]
Next
[Equation]
we’ll simplify the integral over q by marginalizing over all but the k th dimension
[Equation]
Rewriting q k as just q ,
[Equation]
This is indeed the same term as Eq 25, thus completing the proof.
B. P ROOF OF T HEOREM 4.1 AND V ARIANCE OF D ISAGREEMENT
Since Theorem 4.1 is a special case of Theorem B.1, we can easily prove the former:
Proof.
Observe that if ˜ h satisﬁes the class-wise calibration condition as in Deﬁnition 4.2, it must also satisfy class-aggregated calibration. [Equation] [Equation]
[Equation]
Then, we can invoke Theorem B.1 to claim that GDE holds.
The theorem above shows that in expectation, the disagreement of independent pairs of hypotheses is equal to the test error of a single hypothesis.
Now, we will drive an upper bound on the variance of the distribution.
This result corroborates the empirical observation where we can use as low as a single pair of independent models to accurate estimate the test error.
Corollary B.1.1.
If GDE holds and there exists κ ∈ (cid:2) 12 , 1 (cid:3) such that, for all h, h (cid:48) in the support of H
A ,
Dis
[Equation]
then
[Equation]
Proof.
First we write the expression for the exact variance of the disagreement:
[Equation]
By the assumption:
[Equation]
Since h and h (cid:48) are independent and identically distributed
[Equation]
the expectation of the cross term is equal to the product of each expectation and the second moments are equal:
[Equation]
Substituting the inequality back to (47):
[Equation]
By the GDE
[Equation]
we know that ETE = EDR :
[Equation]
Remark.
This corollary characterizes the worst-case behavior of the disagreement error’s variance and relates it to the expectation and variance of the test error distribution over H A .
Recent works (Neal et al., 2018; Nakkiran et al., 2020) have shown that the classical bias-variance trade-off exhibits unusual behaviors in the overparameterized regime where the model contains much more parameters than the number of data points.
In particular, Neal et al.
(2018) shows that the variance actually decreases as the number of parameters increases, which implies that the ﬁrst term of the
RHS in (55) is negligible.
Empirically, this is supported by the ﬁrst columns of Table 1 and Table 3, where we show the variance of the test error is small.
κ represents the amount of structure present in H A .
If κ = 1 , the assumption Dis ( h, h (cid:48) ) ≤
TestErr ( h ) + TestErr ( h (cid:48) ) is always true, since it follows directly from the triangle inequality; however, this would imply that there exists a pair of hypotheses in the support of H A that achieve the largest possible disagreement rate.
Empirical evidence (Nakkiran & Bansal, 2020) indicates that this may not the case.
Instead, deep models make mistakes in highly structured manner, suggesting κ is much smaller than 1 in practice.
On the other hand, if κ = 1 2 , the GDE holds pointwise for every hypotheses pair (up to the small stochasticity present in the distribution of TestErr ).
This is also unlikely since it would imply the disagreement rate variance is at most only a half of test error variance.
In practice, Table 1 and 3 show that the variance of disagreement is approximately equal to the variance of the test error, suggesting that κ falls somewhere between 12 and 1 .
This supports the hypothesis that the models make errors in a structured manner which gives rise to the observed phenomena.
B. D ISAGREEMENT P ROPERTY UNDER D EVIATION FROM C ALIBRATION
Recall from the main paper that we quantiﬁed deviation from class-aggregated calibration in terms of CACE.
Below, we provide the proof of Theorem 4.2, which shows that GDE holds approximately when CACE is low.
Proof.
Recall from the proof of Theorem B.1 that the expected test error ( ETE ) satisﬁes:
[Equation]
Recall that the second term on R.H.S is equal to the expected disagreement rate EDR .
Therefore,
[Equation]
Multiplying and dividing the inner term by (cid:80) K − 1 k =0 p (˜ h k ( X ) = q ) , |
[Table]
Note that it is possible to consider a more reﬁned deﬁnition of CACE that yields a tighter bound on the gap.
In particular, in the last series of equations, we can leave the 1 − q as it is, without upper bounding by 1 .
In practice, this tightens CACE by upto a value of 2 .
We however avoid considering the reﬁned deﬁnition as it is less intuitive as an error metric.
B. C ALIBRATION IS NOT A NECESSARY CONDITION FOR GDE
Theorem B.1 shows that calibration implies GDE.
Below, we show that the converse is not true.
That is, if the ensemble satisﬁes GDE, it is not necessarily the case that it satisﬁes class-aggregated calibration.
This means that calibration and GDE are not equivalent phenomena, but rather only that calibration may lead to the latter.
Proposition B.1.
For a stochastic algorithm A to satisfy GDE, it is not necessary that its corresponding ensemble ˜
h satisﬁes class-aggregated calibration.
Proof.
Consider an example where ˜ h assigns a probability of either 0 .
1 or 0 .
to class 0 .
In particular, assume that with 0 .
5 probability over the draws of ( x, y ) ∼ D , ˜
h 0 ( x ) = 0 .
1 and with
5 probability, ˜
h 0 ( x ) = 0 .
The expected disagreement rate ( EDR ) of this classiﬁer is given by
E [Equation] (cid:104)
[Equation]
(cid:1) =
[Equation]
Now, it can be veriﬁed that the binary classiﬁcation setting, class-aggregated and class-wise calibration are identical.
Therefore, letting p ( Y =
|
˜
h 0 ( X ) = 0 .
1) (cid:44) (cid:15) 1 and p ( Y =
| ˜
h 0 ( X ) = 0 .
2) (cid:44) (cid:15) 2 , our goal is to show that it is possible for (cid:15) 1 (cid:54)
= 0 .
1 or (cid:15) 2 (cid:54) = 0 .
2 and still have the expected test error ( ETE ) equal the EDR of 0 . 25 .
Now, the ETE on D conditioned on ˜ h 0 ( x ) = 0 .
1 is given by (0 .
1(1 − (cid:15) 1 ) + 0 .
9 (cid:15) 1 ) and on ˜
h 0 ( x ) = 0 .
2 is given by (0 .
2(1 − (cid:15) 2 ) + 0 .
8 (cid:15) 1 ) .
Thus, the ETE on D is given by 0 .
15 + 0 .
8 (cid:15) 1 + 0 .
6 (cid:15) 2 ) .
We want 0 .
15 + 0 .
8 (cid:15) 1 + 0 .
6 (cid:15) 2 ) = 0 .
25 or in other words, 0 . 8 (cid:15) 1 + 0 .
6 (cid:15) 2 = 0 .
Observe that while (cid:15) 1 = 0 . 1 and (cid:15) 2 = 0 .
2 is one possible solution where ˜ h would satisfy classwise calibration/class-aggregated calibration, there are also inﬁnitely many other solutions for this equality to hold (such as say (cid:15) 1 = 0 .
25 and (cid:15) 2 = 0 ) where calibration does not hold.
Thus, classaggregated/class-wise calibration is just one out of inﬁnitely many possible ways in which ˜ h could be conﬁgured to satisfy GDE.
B. C OMPARING CACE TO EXISTING NOTIONS OF CALIBRATION .
Calibration in machine learning literature (Guo et al., 2017; Nixon et al., 2019) is often concerned only with the conﬁdence level of the top predicted class for each point.
While top-class calibration is weaker than class-wise calibration, it is neither stronger nor weaker than class-aggregated calibration.
Class-wise calibration is a notion of calibration that has appeared originally under different names in Zadrozny & Elkan; Wu & Gales (2021).
On the other hand, the only closest existing notion to class-aggregated calibration seems to be that of static calibration in Nixon et al.
(2019), where it is only indirectly deﬁned.
Another existing notion of calibration for the multi-class setting is that of strong calibration (Vaicenavicius et al., 2019;
Widmann et al., 2019) which evaluates the accuracy of the model conditioned on ˜ h
( X ) taking a particular value in the K -simplex.
This is signiﬁcantly stronger than class-wise calibration since this would require about exp( K ) many equalities to hold rather than just the K equalities in Deﬁnition 4.2.
B. T HE EFFECT OF DIFFERENT STOCHASTICITY .
Compared to AllDiff/DiffData , DiffInit/Order is still well-calibrated, with only slight deviations.
Why is varying the training data almost as effective in calibration as varying the random seed?
One might propose the following natural hypothesis in the context of DiffOrder vs DiffData .
In the ﬁrst few steps of SGD, the data seen under two different reorderings are likely to not intersect at all, and hence the two trajectories would initially behave as though being trained on two independent datasets.
Further, if the ﬁrst few steps largely determine the kind of minimum that the training falls into, then it is reasonable to expect that the stochasticity in data and in ordering both have the same effect on calibration.
However, this hypothesis falls apart when we try to understand why two runs with the same ordering and different initialization ( DiffInit ) exhibits the same effect as DiffData .
Indeed, Fort et al.
(2019) have empirically shown that two such SGD runs explore diverse regions in the function space.
Hence, we believe that there is a more nuanced reason behind why different types of stochasticity have a similar effect on ensemble calibration.
One promising hypothesis for this could be the multi-view hypothesis from Allen-Zhu & Li (2020), which show that different random initializations could encourage the network to latch on to different predictive features (even when exposed to the same training set), and thus result in ensembles with better test accuracy.
Extending their study to understand similar effects on calibration would be a useful direction for future research.
C A PPENDIX : E XPERIMENTAL D ETAILS C. P AIRWISE D ISAGREEMENT
In this work, the main architectures we used are ResNet18 with the following hyperparameter conﬁgurations:
width multiplier: { 1 × , 2 ×} 2.
initial learning rate: { 0 . 1 , 0 . 05 } 3.
weight decay: { 0 . 0001 , 0 . 0 } 4.
minibatch size: { 200 , 100 } 5.
data augmentation: { No, Yes }
Width multiplier refers to how much wider the model is than the architecture presented in He et al.
(2016) (i.e. every ﬁlter width is multiplied by the width multiplier).
All models are trained with
SGD with momentum of 0 .
The learning rate decays 10 × every 50 epochs.
The training stops when the training accuracy reaches 100% .
For Convolutional Neural Network experiments, we use architectures similar to Network-inNetwork (Lin et al., 2013).
On a high level, the architecture contains blocks of 3 × 3 convolution followed by two 1 × 1 convolution (3 layers in total).
Each block has the same width and the ﬁnal layer is projected to output class number with another 1 × 1 convolution followed by a global average pooling layer to yield the ﬁnal logits.
Other differences from the original implementation are that we do not use dropout and add batch normalization layer is added after every layer.
The hyperparameters are:
1. depth: { 7, 10, 13 } 2.
width: { 128, 256, 384 } 3.
weight decay: { 0 . 001 , 0 . 0 } 4.
minibatch size: { 200, 100 , 300 }
All models are optimized with momentum of 0.9 and uses the same learning rate schedule as
ResNet18.
For Fully Connected Networks, we use:
1. depth: { 1,2,3,4 } 2.
width: { 128, 256, 384, 512 } 3.
weight decay: { 0 . 0 , 0 . 001 } 4.
minibatch size: { 100 , 200 , 300 }
All models are optimized with momentum of 0.9 and uses the same learning rate schedule as
ResNet18.
Distribution shift and pre-training experiments.
On all our experiments on the PACS dataset, we use ResNet50 (with Batch Normalization layers frozen and the ﬁnal fully-connected layer removed) as our featurizer and one linear layer as our classiﬁer.
All our models are trained until 3000 steps after reaching 0.995 training accuracy with the following hyperparameter conﬁgurations:
1. learning rate: 0 .2.
weight decay: 0. 3.
learning rate decay:
None 4.
minibatch size:5.
data augmentation:
Yes
On each of these domains, we train 5 pairs of ResNet50 with a linear layer on top, varying the random seeds (keeping hyperparameters constant).
Both models in a pair are trained on the same 80% of the data, and only differ in their initialization, data ordering and the augmentation on the data.
We then evaluate the test error and disagreement rate of all pairs on each of the four domains.
We consider both randomly initialized models and ImageNet pre-trained models (Deng et al., 2009).
For pre-trained models, only the linear layer is initialized differently between the two models in a pair.
C. E NSEMBLE
For ensembles, unless speciﬁed otherwise, we use the combination of the ﬁrst option for each hyperparameters in Section C.1.
C.2. F INITE -S AMPLE A PPROXIMATION OF CACE
For every ensemble experiment, we train a standard ResNet 18 model (width multiplier 1 × , initial learning rate 0 .
1 , weight decay 0 . 0001 , minibatch size 200 and no data augmentation).
To estimate the calibration, we use the testset D test .
We split [0 , 1] into 10 equally sized bins.
For a class k , we can group all
( x, y ) ∈ D test into different bins B ki according to ˜
h k ( x ) (all bins have boundaries that do not overlap with other bins).
In total, there are 10 × K bins.
[Equation]
Where upper and lower are the boundaries of the bin.
To mitigate the effect of insufﬁcient samples for some of the middling conﬁdence value in the middle (e.g. p = 0 . 5 ), we further aggregate the calibration accuracy over the classes into a single bin B i =
(cid:83) Kk =1 B ki in a weighted manner.
Concretely, for each bin, we sum over all the classes when computing the accuracy:
[Equation]
To quantify the how “far” the ensemble is from the ideal calibration level, we use the Class Aggregated Calibration Error (CACE) which is an average of how much each bin deviates from y =
x weighted by the number of samples in the bin:
[Equation]
where N B is number of bins (usually 10 in this paper unless speciﬁed otherwise), conf ( B i ) is the ideal conﬁdence level of the bin, which we set to the average conﬁdence of all data points in the bin.
This is the sample-based approximation of deﬁnition 4.4.
C.2. F INITE -S AMPLE A PPROXIMATION OF ECE
ECE is a widely used metric for measuing calibration.
For completeness, we will reproduce its approximation here.
Let ˆ Y be the class with highest probability under ˜
h
(we are omitting the dependency on X in the notation since it is clear):
[Equation]
We once again split [0 , 1] into 10 equally sized bins but do not divide further into K classes.
Each bin is constructed as:
[Equation]
upper
[Equation]
B i ) and ( x, y ) ∈ D test
[Equation]
With the same notation used for CACE, the accuracy is computed as:
[Equation]
Finally, the approximation of ECE is computed as the following:
[Equation]
[Figure]
[Figure]
[Figure]
D
[Figure]
C IFAR 100 C ALIBRATION T ABLE
Here we present the calibration error for ResNet18 trained on CIFAR100 with 10k training examples.
D. O THER DATASETS AND ARCHITECTURES
In Fig 10, we provide scatter plots for fully-connected networks (FCN) on MNIST, and convolutional networks (CNN) on CIFAR10.
We observe that when trained on the whole MNIST dataset, there is larger deviation from the x = y behavior (see left-most image).
But when we reduce the dataset size to 2000 , we recover the GDE observation on MNIST.
We observe that the CNN settings satisﬁes
GDE too.
[Table] [Table]
D. E RROR DISTRIBUTION OF E NSEMBLES
Here (Fig 11) we show the error distribution of the ensemble similar to N&B’20.
The x-axis of these plots represent 1 − ˜ h
y ( X ) in the context of our work.
As N&B’20 note, these plots are not bimodally distributed on zero error and random-classiﬁcation-level error (of K − 1 K where K is the number of classes).
This disproves the easy-hard hypothesis as discussed in the main paper.
As a side note, we observe that all these error distributions can be ﬁt well by a Beta distribution.
[Figure] [Figure]
[Figure]
D. C ALIBRATION C ONFIDENCE H ISTOGRAM
Here (Fig 12) we report the number of points that fall into each bin in calibration plots.
In other words, for each value of p , we report the number of times the ensemble ˜ h satisﬁes
˜ h k ( x ) ≈ p for some k and some x .
D. C OMBINING S TOCHASTICITY
In Fig 13, for the sake of completeness, we consider a setting where both the random initialization and the data ordering varies between two runs.
We call this setting the SameData setting.
We observe that this setting behaves similar to DiffData and DiffInit .
[Figure]
[Figure]
D. C LASS - WISE C ALIBRATION VS C LASS - AGGREGATED C ALIBRATION
In Fig 14, we report the calibration plots for a few random classes in the CIFAR10 and CIFAR setup and compare it with the class-aggregated calibration plots.
We observe that the class-wise plots have a lot more variance, indicating that calibration within each class may not always be perfect.
However, when aggregating across classes, calibration becomes much more well-behaved.
This suggests that the calibration is smoothed over all the classes.
It is worth noting that a similar effect also happens for ECE, although not reported here.
D. C ORRELATION VALUES IN F IG
In the main paper ﬁgures, we quantiﬁed correlated via R 2 and τ for scatter plots that include both data-augmented models and non-data-augmented models.
Here, we report these values speciﬁc to each group of models.
[Table] [Table]
D. M ORE THAN ONE PAIR OF MODELS
Here, we show the ResNet18 CIFAR-10 DiffInit experiments (no data augmentation) with only one pair of models v.s. the average of 4 pairs of models.
We see that while 4 pairs of models does improve the correlation, the improvement is only marginal.
This suggests that only using a single pair of models may be sufﬁcient for estimating the generalization error.
[Figure]
However, if we instead measure the distance from the y = x line.
Speciﬁcally, each pair’s deviation is measured as
[Equation]
The denominator normalizes the deviation so hyperparameters with different performance contribute equally.
Then, we observed that 1-pair achieves an average deviation of 0 .
112 while the 4-pair achieves 0 .
This shows that if one is interested in estimating the exact generalization error, using more pairs of models would help.