Original: Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shift
Revised: Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shift

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Recently, Miller et al.
Revised: Recently, Miller et al.

Original: [56] showed that a model’s in-distribution (ID) accuracy hasa strong linear correlation with its out-of-distribution (OOD) accuracy on several
Revised: [59] showed that a model’s in-distribution (ID) accuracy has a strong linear correlation with its out-of-distribution (OOD) accuracy on several

Original: OOD benchmarks — a phenomenon they dubbed “accuracy-on-the-line”.
Revised: OOD benchmarks — a phenomenon they dubbed “accuracy-on-the-line”.

Original: Whilea useful tool for model selection (i.e., the model most likely to perform the best OODis the one with highest ID accuracy), this fact does not help estimate theactual OOD performance of models without access to a labeled OOD validationset.
Revised: While a useful tool for model selection (i.e., the models with better ID accuracy are likely to have better OOD accuracy), this fact does not help estimate the actual OOD performance of models without access to a labeled OOD validation set.

Original: In this paper, we show a similar but surprising phenomenon also holds forthe agreement between pairs of neural network classifiers: whenever accuracyon-the-line holds, we observe that the OOD agreement between the predictionsof any two pairs of neural networks (with potentially different architectures) alsoobserves a strong linear correlation with their ID agreement.
Revised: In this paper, we show a similar but surprising phenomenon also holds for the agreement between pairs of neural network classifiers: whenever accuracy-onthe-line holds, we observe that the OOD agreement between the predictions of any two pairs of neural networks (with potentially different architectures) also observes a strong linear correlation with their ID agreement.

Original: Furthermore, weobserve that the slope and bias of OOD vs ID agreement closely matches that of
Revised: Furthermore, we observe that the slope and bias of OOD vs. ID agreement closely matches that of

Original: OOD vs ID accuracy.
Revised: OOD vs. ID accuracy.

Original: This phenomenon, which we call agreement-on-the-line ,has important practical applications: without any labeled data, we can predict the
Revised: This phenomenon, which we call agreement-on-the-line , has important practical applications: without any labeled data, we can predict the

Original: OOD accuracy of classifiers , since OOD agreement can be estimated with justunlabeled data.
Revised: OOD accuracy of classifiers , since OOD agreement can be estimated with just unlabeled data.

Original: Our prediction algorithm outperforms previous methods both inshifts where agreement-on-the-line holds and, surprisingly, when accuracy is noton the line.
Revised: Our prediction algorithm outperforms previous methods both in shifts where agreement-on-the-line holds and, surprisingly, when accuracy is not on the line.

Original: This phenomenon also provides new insights into deep neural networks:unlike accuracy-on-the-line, agreement-on-the-line appears to only hold for neuralnetwork classifiers.
Revised: This phenomenon also provides new insights into deep neural networks: unlike accuracy-on-the-line, agreement-on-the-line appears to only hold for neural network classifiers.

Original:  Introduction
Revised:  Introduction

Original: Machine learning operates well when models observe and make decisions on inputs coming fromthe same distribution as the training data.
Revised: Machine learning operates well when models observe and make decisions on data coming from the same distribution as the training data.

Original: Yet in the real world, this assumption rarely holds.
Revised: Yet in the real world, this assumption rarely holds.

Original: Environments are never fully controlled.
Revised: Environments are never fully controlled.

Original: Robots interact with their surroundings, effectively changingwhat they sees in the future.
Revised: Robots interact with their surroundings, effectively changing what they see in the future.

Original: Self-driving cars face constant distribution shift when driving tonew cities under changing weather conditions.
Revised: Self-driving cars face constant distribution shift when driving to new cities under changing weather conditions.

Original: Models trained on clinical data from one hospitalface challenges when deployed for a different hospital with different subpopulations.
Revised: Models trained on clinical data from one hospital face challenges when deployed for a different hospital with different subpopulations.

Original: Under these premises problem of estimating a  new  model  trained on ( in-distribution , or ID).
Revised: Under these premises, practitioners constantly face the problem of estimating a model’s performance on new data distributions ( out-of-distribution , or OOD) that are related to but different from the data distribution that the model was trained on ( in-distribution , or ID).

Original: Models may sometimes break catastrophically under new conditions, or may only suffer a small degredation.
Revised: Depending on the distribution shift, models may sometimes break catastrophically under new conditions, or may only suffer a small degradation in performance.

Original: Differentiating between such cases is crucial 
Revised: Differentiating between such cases is crucial in practice.

Original: Assessing OOD performance is difficult because in reality, labeled OOD data is often very costly toobtain.
Revised: Assessing OOD performance is difficult because in reality, labeled OOD data is often very costly to obtain.

Original: On the other hand, unlabeled OOD data is much easier to obtain.
Revised: On the other hand, unlabeled OOD data is much easier to obtain.

Original: A natural question is whetherwe can leverage unlabeled OOD data for estimating the OOD performance.
Revised: A natural question is whether we can leverage unlabeled OOD data for estimating the OOD performance.

Original: This paradigm of usingunlabeled data to predict the OOD generalization performance has received much attention recently[29, 9, 81, 22, 23, 10, 30].
Revised: This paradigm of using unlabeled data to predict the OOD generalization performance has received much attention recently [29, 9, 87, 22, 23, 10, 31].

Original: Though many different metrics have been proposed, their success varies 
Revised: Although many different metrics have been proposed, their success varies widely depending on the shift and the ID performance of the model. While it is

Original: [Figure]
Revised: [Figure]

Original: widely depending on the shift and the ID performance of the model.

Original: While it is in fact impossible fora method to always work with no assumptions [29], a major hurdle in using these methods is thatthere is currently no understanding of when they work or a recipe to detect when their predictionsmight be unreliable.
Revised: impossible for a method to always work with no assumptions [29], a major obstacle to using these methods is that there is currently no understanding of when they work or a recipe to detect when their predictions might be unreliable.

Original: In a separate but related line of work, Miller et al.
Revised: In a separate but related line of work, Miller et al.

Original: [56] demonstrated that in a wide variety ofcommon OOD prediction benchmarks such as CIFAR-10.1 [64], ImageNetV2 [64], CIFAR-10C[35], fMoW- WILDS [13], there exists an almost perfect positive linear correlation between the IDtest vs OOD accuracy of models.
Revised: [59] demonstrated that in a wide variety of common OOD prediction benchmarks such as CIFAR-10.1 [67], ImageNetV2 [67], CIFAR-10C [34], fMoW- WILDS [13], there exists an almost perfect positive linear correlation between the ID test vs. OOD accuracy of models.

Original: When this phenomenon, called accuracy-on-the-line , occurs,improving performance on the ID test data directly leads to improvements on OOD performance.
Revised: When this phenomenon, called accuracy-on-the-line , occurs, improving performance on the ID test data directly leads to improvements on OOD performance.

Original: Furthermore, if we have access to the slope and bias of this correlation, predicting OOD accuracybecomes straightforward.
Revised: Furthermore, if we have access to the slope and bias of this correlation, predicting OOD accuracy becomes straightforward.

Original: Unfortunately, accuracy-on-the-line is not universal.
Revised: Unfortunately, accuracy-on-the-line is not a universal phenomenon.

Original: In some datasets, suchas Camelyon17- WILDS [1], models with the same ID test performance had OOD performance thatvaried largely.
Revised: In some datasets, such as Camelyon17- WILDS [1], models with the same ID test performance have OOD performance that varied largely.

Original: Thus, while the accuracy-on-the-line phenomenon is interesting, its practical use issomewhat limited since determining whether accuracy-on-the-line holdsitself requires labeled OODdata.
Revised: Thus, while the accuracy-on-the-line phenomenon is interesting, its practical use is somewhat limited since determining whether accuracy-on-the-line holds requires labeled OOD data in the first place.

Original: In this work, we begin by observing an analogous phenomenon based upon agreement rather thanaccuracy.
Revised: In this work, we begin by observing an analogous phenomenon based upon agreement rather than accuracy.

Original: Specifically, if we consider pairs of neural network of classifiers, and look at the agreementof their predictions (the proportion of cases where they make the same prediction, which requiresno labeled data to compute), we find that there also often exists a strong linear correlation between
Revised: Specifically, if we consider pairs of neural network of classifiers, and look at the agreement of their predictions (the proportion of cases where they make the same prediction, which requires no labeled data to compute), we find that there also often exists a strong linear correlation between

Original: ID vs OOD agreement.
Revised: ID vs. OOD agreement.

Original: We call this phenomenon agreement-on-the-line .
Revised: We call this phenomenon agreement-on-the-line .

Original: Importantly, however, thisphenomenon appears to be tightly coupled with accuracy-on-the-line: when agreement-on-the-lineholds, accuracy-on-the-line also holds; and when agreement-on-the-line does not hold, neither doesaccuracy-on-the-line.
Revised: Importantly, however, this phenomenon appears to be tightly coupled with accuracy-on-the-line: when agreement-on-theline holds, accuracy-on-the-line also holds; and when agreement-on-the-line does not hold, neither does accuracy-on-the-line.

Original: Furthermore,in the case of neural network classifiers, when these propertieshold, the linear correlations of both accuracy-on-the-line and agreement-on-the-line appear to haveroughly the same slope and bias .
Revised: Furthermore, when these properties hold, the linear correlations of both accuracy-on-the-line and agreement-on-the-line appear to have roughly the same slope and bias .

Original: Interestingly, unlike accuracy-on-the-line, which appears to be ageneral phenomenon, agreement-on-the-line, especially the fact that the slope and bias of the linearcorrelation agree across accuracy and agreement, appears to occur only for neural networks.
Revised: Interestingly, unlike accuracy-on-the-line, which appears to be a general phenomenon, agreement-onthe-line, especially the fact that the slope and bias of the linear correlation agree across accuracy and agreement, appears to occur only for neural networks.

Original: Indeedthe phenomenon is quite unintuitive, given that there is no a-priori reason to believe that agreementand accuracy would be connected in such a manner; nonetheless, we find this phenomenon occursrepeatedly across multiple datasets and vastly different deep network architectures.
Revised: Indeed, the phenomenon is quite unintuitive, given that there is no a-priori reason to believe that agreement and accuracy would be connected in such a manner; nonetheless, we find this phenomenon occurs repeatedly across multiple datasets and vastly different neural network architectures.

Original: This phenomenon is of immediate practical interest.
Revised: This phenomenon is of immediate practical interest.

Original: Since agreement-on-the-line can be validatedwithout any labeled OOD data, we canimmediately use it as a proxy to assess whether accuracy-onthe-line holds, and thus whetherit is reasonable to use ID accuracy as model selection criteria forpicking a model on OOD data.
Revised: Since agreement-on-the-line can be validated without any labeled OOD data, we can use it as a proxy to assess whether accuracy-on-the-line holds, and thus whether ID accuracy is a reasonable OOD model selection criteria.

Original: Furthermore, since the slope and bias of the agreement-on-the-line fitcan also be estimated without labeled OOD data, (for the cases where agreement-on-the-line holds)we can use this approach to derive avery simple algorithm for estimating the OOD generalization of classifiers, without any access to labeled OOD data.
Revised: Furthermore, since the slope and bias of the agreement-on-the-line fit can also be estimated without labeled OOD data, (for the cases where agreement-on-the-line holds) we can use this approach to derive a simple algorithm for estimating the OOD generalization of classifiers, without any access to labeled OOD data.

Original: The approach outperforms competing methodsand predicts OOD test error with a mean absolute estimation error of ≤ 2% on datasets whereagreement-on-the-line holds.
Revised: The approach outperforms competing methods and predicts OOD test error with a mean absolute estimation error of ≤ 2% on datasets where agreement-on-the-line holds.

Original: On datasets where agreement-on-the-line does not hold, the method, asexpected does not perform as well, butsomewhat surprisingly still outperforms competing methodsin terms of predicting OOD performance.
Revised: On datasets where agreement-on-the-line does not hold, the method as expected does not perform as well, but surprisingly still outperforms competing methods in terms of predicting OOD performance.

Original: To summarize, our contributions are as follows:
Revised: To summarize, our contributions are as follows:

Original: We discover and empirically analyze the agreement-on-the-line phenomenon: that ID vsOOD agreement for pairs of  classifiers lies on a line precisely when the corresponding IDvs OOD accuracy also lies on a line.
Revised: We discover and empirically analyze the agreement-on-the-line phenomenon: that ID vs. OOD agreement for pairs of neural network classifiers lies on a line precisely when the corresponding ID vs. OOD accuracy also lies on a line.

Original: Furthermore,for thecase of neural network classifiers,the slope and bias of these two lines are approximately equal.2.
Revised: Furthermore, the slope and bias of these two lines are approximately equal.

Original: Exploiting this phenomenon, we develop a simple method for estimating the OOD performance of classifiers without any access to labeled OOD data (and by observing whetheragreement-on-the-line holds, the method also provides a “sanity check” that these estimatesare reasonable).
Revised: Leveraging this phenomenon, we develop a simple method for estimating the OOD performance of classifiers without any access to labeled OOD data (and by observing whether agreement-on-the-line holds, the method also provides a “sanity check” that these estimates are reasonable).

Original: The proposed method outperforms all competing baselines for this task.
Revised: The proposed method outperforms all competing baselines for this task.

Original:  Related Works
Revised:  Related Works

Revised: Accuracy-on-the-line Before Miller et al.

Revised: [59], several other works proposing new benchmarks for performance evaluation [58, 67, 69, 85] have also observed a strong linear correlation between

Original: Estimating ID generalization via agreement.
Revised: ID and OOD performance of models. Theoretical analysis of this phenomenon is limited. Miller et al. [59] constructs distribution shifts where accuracy-on-the-line can be observed assuming the data is Gaussian, however they do not incorporate any assumptions about the classifier.

Revised: Another work by Mania and Sra [55] analyzes the phenomenon under the assumption that given two models, it is unlikely that the lower accuracy model classifies a data point correctly while the higher accuracy model classifies it incorrectly.

Revised: Such model similarity has been observed by [56, 30].

Revised: Recent works have also observed some nonlinear correlations [47] and negative correlations in performance [79, 41].

Revised: In this work, we focus on the results from Miller et al.

Original:  Departing from  approaches based on uniformconvergence [60, 26, 3, 57], several recent works [41, 59, 83, 28] derive different approaches forestimating generalization error or comparing different models.
Revised: [59] and leave these nuances for future study. Estimating ID generalization via agreement. Departing from conventional approaches based on uniform convergence [63, 26, 3, 60], several recent works [62, 89, 28, 39] propose different approaches for estimating generalization error.

Original: In particular, this work is closelyrelated to Jiang et al.
Revised: In particular, this work is closely related to Jiang et al.

Original: [41], which shows that the disagreement between two models trained withdifferent random seeds closely tracks the ID generalization error of the models if the ensembles of themodels are well-calibrated.
Revised: [39], which shows that the disagreement between two models trained with different random seeds closely tracks the ID generalization error of the models, if the ensemble of the models are well-calibrated.

Original: Predicting ID generalization via disagreement has also previously beenproposed by Madani et al.
Revised: Predicting ID generalization via disagreement has also previously been proposed by Madani et al.

Original: [54] and Nakkiran and Bansal [58].
Revised: [54] and Nakkiran and Bansal [61].

Original: Our method also uses disagreementbut unlike these works, focuses on out-of-distribution generalization.
Revised: Our method also uses disagreement for estimating performance but, unlike these works, we focus on OOD generalization, and, more importantly, we do not require calibration or models with the same architecture.

Original: OOD generalization.
Revised: OOD generalization.

Original: The problem of characterizing generalization in  OOD setting is  more challenging than the ID setting and is relatively understudied.
Revised: The problem of characterizing generalization in the OOD setting is even more challenging than the ID setting.

Original: Ben-David et al.
Revised: Ben-David et al.

Original: [5] provides one of thefirst uniform-convergence-based bounds for domain adaptation .
Revised: [5] provides one of the first uniform-convergencebased bounds for domain adaptation , a related but harder framework of improving the OOD performance of models given unlabeled OOD data and labeled ID data.

Original: Several works [55, 18, 47] build onthis approach and extend it to other learning scenarios.
Revised: Several works [57, 18, 46] build on this approach and extend it to other learning scenarios.

Original: Most of these works attempt to bound thedifference between ID and OOD performance via a certain notion of closeness  (e.g., the total variationdistance and the H ∆ H divergence which is related to agreement) between the original distributionand shifted distribution and build on the uniform-convergence framework [65].
Revised: Most of these works attempt to bound the difference between ID and OOD performance via a certain notion of closeness between the original distribution and shifted distribution (e.g., the total variation distance and the H ∆ H divergence which is related to agreement), and build on the uniform-convergence framework [68].

Original: As pointed out by
Revised: As pointed out by

Original: Miller et al.
Revised: Miller et al.

Original: [56], these approaches providean upper bound on the OOD performance that growslooser as the distribution shift becomes larger, and these upper bounds do not capture the precisetrends observed in practice.
Revised: [59], these approaches provide upper bounds on the OOD performance that grows looser as the distribution shift becomes larger, and the bounds do not capture the precise trends observed in practice.

Original: Predicting the actual OOD performance using unlabeled data has gainedinterest in the past decade.
Revised: Predicting the actual OOD performance using unlabeled data has gained interest in the past decade.

Original: These methods can roughly be divided into three categories:1.
Revised: These methods can roughly be divided into three categories:

Original: Placing assumptions on the distribution shift.
Revised: Placing assumptions on the distribution shift.

Original: Donmez et al.
Revised: Donmez et al.

Original: [24] assume knowledge of themarginal of the shifted label distribution P ( y ) and show that OOD accuracy can be predicted if theshifted distribution satisfies several properties.
Revised: [24] assume knowledge of the marginal of the shifted label distribution P ( y ) and show that OOD accuracy can be predicted if the shifted distribution satisfies several properties.

Original: Steinhardt and Liang [70] work under the assumptionthat the data x can be separated into “views” that are conditionally independent given label y .
Revised: Steinhardt and Liang [74] work under the assumption that the data x can be separated into “views” that are conditionally independent given label y .

Original: Chenet al.
Revised: Chen et al.

Original: [10] assume prior knowledge about the shift and use an importance weighting procedure.2.
Revised: [10] assume prior knowledge about the shift and use an importance weighting procedure.

Original: Placing assumptions on the classifiers.
Revised: Placing assumptions on the classifiers.

Original: Given multiple classifiers of interest, Platanios et al.[61, 62] places assumptions on the classifiers and use logical constraints  to identify each one’s error.
Revised: Given multiple classifiers of interest, Platanios et al. [64, 65] form logical constraints based on assumptions about the hypothesis distribution to identify [Table]

Revised: individual classifier’s error.

Original: On the other hand, Jaffe et al.
Revised: On the other hand, Jaffe et al.

Original: [40] use a spectral-based approach under the assumption that classifiersmake independent errors.3.
Revised: [38] relates accuracy to the classifiers’ covariance matrix under the assumption that classifiers make independent errors and do better than random.

Original: Empirically measuring the distribution shift.
Revised: Empirically measuring the distribution shift.

Original: A group of works [27, 67, 22, 23] train aregression model over metrics that measure the severity of the distribution shift.
Revised: A group of works [27, 71, 22, 23] train a regression model over metrics that measure the severity of the distribution shift.

Original: Inspired by theobservation that the maximum softmax probability (or confidence) for OOD points is typicallylower [36, 34], Guillory et al.
Revised: Inspired by the observation that the maximum softmax probability (or confidence) for OOD points is typically lower [35, 34], Guillory et al.

Original: [30] and more recently Garg et al.
Revised: [31] and more recently Garg et al.

Original: [29] utilize model confidence topredict accuracy.
Revised: [29] utilize model confidence to predict accuracy.

Original: This method was extended upon by Chen et al.

Original: Chuang et al.
Revised: Chuang et al.

Original: [14] use a set ofdomain-invariant predictors as a proxy for the unknown, true target labels.
Revised: [14] uses agreement with a set of domain-invariant predictors as a proxy for the unknown, true target labels.

Revised: This method was extended upon by Chen et al.

Revised: [9] which improves the predictors by self-training.

Original: Yu et al.
Revised: Yu et al.

Original: [81] observedthat the distance between the model of interest f and a reference model trained on the pseudolabelsof f showed strong linear correlation with OOD accuracy.[Table]
Revised: [87] observed that the distance between the model of interest f and a reference model trained on the pseudolabels of f showed strong linear correlation with OOD accuracy.

Original: Though a large number of methods have been proposed, for the large majority, it is not well-understoodwhen they will work.
Revised: Though a large number of methods have been proposed, for the large majority, it is not wellunderstood when they will work.

Original: Intuitively, no method will work on all shifts without additional assumptions[29].
Revised: Intuitively, no method will work on all shifts without additional assumptions [29].

Original: But is there some simple general structure to shifts in the real world that allows us to reliablypredict OOD accuracy?
Revised: But is there some simple general structure to shifts in the real world that allows us to reliably predict OOD accuracy?

Original: Even if such a structure is not universal, can we easily inspect if this structureholds?
Revised: Even if such a structure is not universal, can we easily inspect if this structure holds?

Original: What is a plausible assumption we can make about the OOD behaviour of classifiers ?
Revised: What is a plausible assumption we can make about the OOD behaviour of classifiers ?

Original: Thenovelty and significance of our work comes from trying to better understand and address thesequestions, specifically for neural networks.
Revised: The novelty and significance of our work comes from trying to better understand and address these questions, specifically for neural networks.

Original: In this work, we observe a phenomenon related to, butstronger than accuracy-on-the-line that allows us to reliably predict the OOD accuracy of neuralnetworks.
Revised: In this work, we observe a phenomenon related to, but stronger than accuracy-on-the-line that allows us to reliably predict the OOD accuracy of neural networks.

Original:  The agreement-on-the-line phenomenon 
Revised:  The agreement-on-the-line phenomenon 3.1 Notation and setup

Original: 3.1 Notation and setup

Original: Let H denote a set of neural networks trained on ( X train , y train )
Revised: Let H denote a set of neural networks trained on ( X train , y train )

Original: = { ( x i , y i ) } m train i =1 sampled from D ID .
Revised: = { ( x i , y i ) } m train i =1 sampled from D ID .

Original: Given any pair of models h, h ′ ∈ H , for a distribution D , the expected accuracy and agreement aredefined as:
Revised: Given any pair of models h, h ′ ∈ H , for a distribution D , the expected accuracy and agreement are defined as:

Original: [Equation]
Revised: [Equation]

Original:  We assume access to  labeled validation set ( X val , y val ) =
Revised: Agr [Equation] [Equation] We assume access to a labeled validation set ( X val , y val ) =

Original: { ( x i , y i ) } m Val i =1 sampled from D ID thatallows us to estimate the ID accuracy (cid:100)
Revised: { ( x i , y i ) } m Val i =1 sampled from D ID that allows us to estimate the ID accuracy (cid:100)

Original: Acc ID ( h ) as the sample average of 1 { h ( x ) =
Revised: Acc ID ( h ) as the sample average of 1 { h ( x ) =

Original: y } over thevalidation set.
Revised: y } over the validation set.

Original: We do not assume access to a labeled OOD validation set, as this is often impracticalto obtain, and thereby cannot directly estimate (cid:100)
Revised: We do not assume access to a labeled OOD validation set, as this is often impractical to obtain, and thereby cannot directly estimate (cid:100)

Original: Acc OOD ( h ) in a similar manner.
Revised: Acc OOD ( h ) in a similar manner.

Original: Agreement, on the other hand, only requires access to unlabeled data.
Revised: Agreement, on the other hand, only requires access to unlabeled data.

Original: We assume access to unlabeled143 samples X
Revised: We assume access to unlabeled samples X

Original: [Equation] 1 from the shifted distribution of interestas D OOD .
Revised: OOD = { x i } m OOD i =1 from the shifted distribution of interest D OOD .

Original: Hence, we can estimate144 both the ID and OOD agreement as follows: 
Revised: Hence, we can estimate both the ID and OOD agreement as follows: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 3.2 Experimental setup
Revised: 3.2 Experimental setup

Original: We study the ID vs OOD accuracy and agreement between pairs of models across more than 20common OOD benchmarks and hundreds of independently trained neural networks.
Revised: We study the ID vs. OOD accuracy and agreement between pairs of models across more thancommon OOD benchmarks and hundreds of independently trained neural networks.

Original: Datasets.
Revised: Datasets.

Original: We present results on 8 dataset shifts in the main paper, and include results for otherdistribution shifts in the Appendix B. These 8 datasets span:
Revised: We present results on 8 dataset shifts in the main paper, and include results for other distribution shifts in the Appendix C. These 8 datasets span:

Original: Dataset reproductions: CIFAR-10.1 [64], CIFAR-10.2 [52] reproductions of CIFAR-10 [44] and ImageNetV2 [64] reproduction of ImageNet [21] 
Revised: Dataset reproductions: CIFAR-10.1 [67], CIFAR-10.2 [52] reproductions of CIFAR-10 [43] and ImageNetV2 [67] reproduction of ImageNet [21] 2.

Original: Synthetic corruptions: CIFAR-10C Fog and CIFAR-10C Snow [35]
Revised: Synthetic corruptions: CIFAR-10C Fog and CIFAR-10C Snow [34] 3.

Original: Real-world shifts from [43]: satellite images (fMoW- WILDS ), images from camera traps in the wildlife (iWildCam- WILDS [4]) and images of cancer tissue (Camelyon17- WILDS [1])
Revised: Real-world shifts from [42]: satellite images (fMoW- WILDS ), images from camera traps in the wildlife (iWildCam- WILDS [4]), and images of cancer tissue (Camelyon17- WILDS [1])

Original: [Figure]
Revised: Appendix C includes results on CINIC-10 [19], STL-10 [16], and other WILDS and CIFAR-10C benchmarks. We also investigate an analogous phenomenon for the F1-score used to assess the reading comprehension performance of language models on the Amazon-SQuAD benchmark [58].

Original: Models.
Revised: Models.

Original: For ImageNetV2, we evaluate 50 ImageNet pretrained models from the timm [77] package.
Revised: For ImageNetV2, we evaluate 50 ImageNet pretrained models from the timm [82] package.

Original: On all other shifts, we evaluate on all independently trained models in the testbed created andutilized by [56] consisting of ≥ 150 models for each shift.
Revised: On the remaining 7 shifts, we evaluate on all independently trained models in the testbed created and utilized by [59] consisting of ≥ 150 models for each shift.

Original: The evaluated models span a variety ofconvolutional neural networks (e.g. ResNet [31], DenseNet [38], EfficientNet [74], VGG [49]) aswell as various Vision Transformers [25].
Revised: The evaluated models span a variety of convolutional neural networks (e.g. ResNet [33], DenseNet [36], EfficientNet [78], VGG [49]) as well as various Vision Transformers [25].

Original: All architectures and models are listed in the Appendix C.
Revised: All architectures and models are listed in the Appendix D.

Original: Probit scaling.
Revised: Probit scaling.

Original: Miller et al.
Revised: Miller et al.

Original: [56] report their results after probit scaling Φ − 1 ( · ) 1 the ID vs OODaccuracies due to a better linear fit.
Revised: [59] report their results after probit scaling Φ − 1 ( · ) 1 the ID vs. OOD accuracies due to a better linear fit.

Original: We apply the same probit transform to both accuracy andagreement in our experiments.
Revised: We apply the same probit transform to both accuracy and agreement in our experiments.

Original: 3.3 Observations
Revised: 3.3 Observations

Original: We empirically observe a peculiar phenomenon in deep neural networks, which we call agreementon-the-line characterized by the following three properties:
Revised: We empirically observe a peculiar phenomenon in deep neural networks, which we call agreementon-the-line characterized by the following three properties:

Original: Prop( i)
Revised: Prop(i)

Original: When ID vs OOD accuracy observes a strong linear correlation ( ≥ 0 .
Revised: When ID vs. OOD accuracy observes a strong linear correlation ( ≥ 0 .

Original: 95 R 2 values), we seethat ID vs OOD agreement is also strongly linearly correlated .
Revised: 95 R 2 values), we see that ID vs. OOD agreement is also strongly linearly correlated .

Original: Prop(ii) When both accuracy and agreement observe strong linear correlation, we see that theselinear correlations have almost the same slope and bias .
Revised: Prop(ii) When both accuracy and agreement observe strong linear correlation, we see that these linear correlations have almost the same slope and bias .

Original: Prop(iii)
Revised: Prop(iii)

Original: When the linear correlation of ID vs OOD accuracy is weak ( ≤ 0 .
Revised: When the linear correlation of ID vs. OOD accuracy is weak ( ≤ 0 .

Original: 75 R 2 values), the linearcorrelation between ID and OOD agreement is similarly weak.
Revised: 75 R 2 values), the linear correlation between ID and OOD agreement is similarly weak.

Original: We show the agreement-on-the-line phenomenon on 8 datasets in Figure 1 and Table 1 (See Appendix B for other datasets).
Revised: We show the agreement-on-the-line phenomenon on 8 datasets in Figure 1 and Table 1.

Original: On CIFAR-10.1, CIFAR-10.2, ImageNetV2, CIFAR-10C Fog/Snow,and fMoW- WILDS , we find that both ID vs OOD accuracy and agreement observe strong linearcorrelations, and the linear fits have the same slope and bias (Prop(i), Prop(ii)).
Revised: On CIFAR10.1, CIFAR-10.2, ImageNetV2, CIFAR-10C Fog/Snow, and fMoW- WILDS , we find that both ID vs. OOD accuracy and agreement observe strong linear correlations, and the linear fits have the same slope and bias (Prop(i), Prop(ii)).

Original: On the other hand, onthe datasets Camelyon17- WILDS and iWildCam- WILDS where accuracy is not linearly correlated,agreement is also not linearly correlated (Prop(iii)).
Revised: On the other hand, on the datasets Camelyon17- WILDS and iWildCam- WILDS where accuracy is not linearly correlated, agreement is also not linearly correlated (Prop(iii)).

Revised: To ensure that the differences in the slopes is not statistically significant, we construct the following hypothesis test.

Revised: For each dataset, we randomly sample 1000 subsets of 10 models from the model

Revised: [Figure]

Original:  3.4 What makes agreement-on-the-line interesting?
Revised: testbed and compute the corresponding difference in the slope of the linear fits: Slope Acc − Slope Agr . In Table 1, we look at the 95 % confidence interval of the distribution of these slope differences for the 6 datasets where we observe a strong correlation. For all datasets with the exception of CIFAR10.2, the null hypothesis (i.e. difference in slope is 0) lies within the 95 % confidence interval thus cannot be rejected. 3.4 What makes agreement-on-the-line interesting?

Original: First, agreement can be estimated with just unlabeled data .
Revised: First, agreement can be estimated with just unlabeled data .

Original: Hence, the agreement-on-the-linephenomenon has important practical implications for both checking whether the distribution shiftobserves accuracy-on-the-line and predicting the actual value of OOD accuracy without any OODlabels.
Revised: Hence, the agreement-on-the-line phenomenon has important practical implications for both checking whether the distribution shift observes accuracy-on-the-line and predicting the actual value of OOD accuracy without any OOD labels.

Original: We present a method to estimate OOD error using this phenomenon in Section 4.
Revised: We present a method to estimate OOD error using this phenomenon in Section 4.

Original: Second, agreement-on-the-line does not directly follow from accuracy-on-the-line.
Revised: Second, agreement-on-the-line does not directly follow from accuracy-on-the-line.

Original: Prior work hasobserved that expected ID accuracy often equals ID agreement over pairs of models with the samearchitecture, trained on the same dataset but with different random seeds [41].
Revised: Prior work has observed that expected ID accuracy often equals ID agreement over pairs of models with the same architecture, trained on the same dataset but with different random seeds [39].

Original: Agreement-on-the-linegoes beyond these results in two ways: (i) agreement between models with different architectures(Fig. 2) and (ii) agreement between different checkpoints on the same training run (Fig. 4) is also on the ID vs OOD agreement line.
Revised: Agreement-on-the-line goes beyond these results in two ways: (i) agreement between models with different architectures (Fig. 2) and (ii) agreement between different checkpoints on the same training run (Fig. 4) is also on the ID vs. OOD agreement line.

Original: These ID/OOD agreements do not equal the expected
Revised: These ID/OOD agreements do not equal the expected

Original: ID/OOD accuracy.
Revised: ID/OOD accuracy.

Original: Indeed, understanding why agreement-on-the-line holds requires going beyondthe theoretical conditions presented in the prior work [41] which do not hold for this expanded set ofmodels.
Revised: Indeed, understanding why agreement-on-the-line holds requires going beyond the theoretical conditions presented in the prior work [39] which do not hold for this expanded set of models. Furthermore, the phenomenon of matching slopes is not due to the models performing well

Revised: ID and OOD (Accuracy can be thought of as the agreement with the perfect classifier).

Revised: Even pairs of bad models display this phenomenon.

Original: See Appendix E for further discussion.
Revised: See Appendix F for further discussion.

Original: Finally, we emphasize that there is something special about neural networks that makes the ID vs
Revised: Finally, we emphasize that there is something special about neural networks that makes the ID vs.

Original: OOD agreement trend identical to the ID vs OOD accuracy trend.
Revised: OOD agreement trend identical to the ID vs. OOD accuracy trend.

Original: This is unlike accuracy-on-theline that holds across a wide range of models including neural networks and classical approaches.
Revised: This is unlike accuracy-on-theline that holds across a wide range of models including neural networks and classical approaches.

Original: Figure 2 shows CIFAR-10 Test vs CIFAR-10C Fog accuracy and agreement of linear models (e.g.logistic and ridge regression) and various non-linear models (e.g. Kernel SVM [17], k-Nearest
Revised: Figure 2 shows CIFAR-10 Test vs. CIFAR-10C Fog accuracy and agreement of linear models (e.g. logistic and ridge regression) and various non-linear models (e.g. Kernel SVM [17], k-Nearest

Original: Neighbors, Random Forests [6], Random Features [15], AdaBoost [84]).
Revised: Neighbors, Random Forests [6], Random Features [15], AdaBoost [90]).

Original: See plots for other datasetsin Appendix B. We look at agreement between pairs of models from the same model family.
Revised: See plots for other datasets in Appendix C. We look at agreement between pairs of models from the same model family.

Original: While
Revised: While

Original: Prop(i) seems to hold for several other model families on several shifts, Prop(ii) only holds for neuralnetworks, i.e. the slope and bias of the agreement line do not match the slope and bias of the accuracyline for other model families.
Revised: Prop(i) seems to hold for several other model families on several shifts, Prop(ii) only holds for neural networks, i.e. the slope and bias of the agreement line do not match the slope and bias of the accuracy line for other model families.

Original:  A method for estimating OOD accuracy
Revised:  A method for estimating OOD accuracy

Original: In this section, we describe how the phenomenon of agreement-on-the-line (described in Section 3)offers a simple practical method to perform model selection and estimate accuracy under distributionshifts.
Revised: In this section, we describe how the phenomenon of agreement-on-the-line (described in Section 3) offers a simple practical method to perform model selection and estimate accuracy under distribution shifts.

Original: Recall from Section 3.1 that we have labeled ID validation data ( X val , y val ) and unlabeled
Revised: Recall from Section 3.1 that we have labeled ID validation data ( X val , y val ) and unlabeled

Original: OOD data X OOD .
Revised: OOD data X OOD .

Original: Model selection.
Revised: Model selection.

Original: Without OOD labeled data, can we determine which model is likely to achievethe best OOD performance?
Revised: Without OOD labeled data, can we determine which model is likely to achieve the best OOD performance?

Original: When accuracy-on-the-line holds and ID vs OOD accuracy is linearly correlated, we can simply pick the model with highest ID accuracy.
Revised: When accuracy-on-the-line holds and ID vs. OOD accuracy is linearly correlated, we can simply pick the model with highest ID accuracy.

Original: In practice, how doesone determine if accuracy-on-the-line holds without labeled OOD data?
Revised: In practice, how does one determine if accuracy-on-the-line holds without labeled OOD data?

Original: By Prop(i) and Prop(iii),agreement-on-the-line implies accuracy-on-the-line.
Revised: By Prop(i) and Prop(iii), agreement-on-the-line implies accuracy-on-the-line.

Original: Hence, we simply need to check if ID and OODagreement (which can be estimated as in (2)) are linearly correlated, in order to know if our modelselection criterion based on ID accuracy is valid.
Revised: Hence, we simply need to check if ID and OOD agreement (which can be estimated as in (2)) are linearly correlated, in order to know if our model selection criterion based on ID accuracy is valid.

Original: OOD error prediction.
Revised: OOD error prediction.

Original: Agreement-on-the-line allows us to go beyond model selection and actuallypredict OOD accuracy .
Revised: Agreement-on-the-line allows us to go beyond model selection and actually predict OOD accuracy .

Original: Intuitively, we can estimate the slope and bias of the agreement line with justunlabeled data.
Revised: Intuitively, we can estimate the slope and bias of the agreement line with just unlabeled data.

Original: By Prop(ii), they match the slope and bias of the accuracy line and hence, we canestimate the OOD accuracy by linearly transforming the ID accuracy (with the appropriate probitscaling).
Revised: By Prop(ii), they match the slope and bias of the accuracy line and hence, we can estimate the OOD accuracy by linearly transforming the ID accuracy (with the appropriate probit scaling).

Original: We formalize this intuition below and provide an algorithm for OOD accuracy estimation in Algorithm 1.
Revised: We formalize this intuition below and provide an algorithm for OOD accuracy estimation in Algorithm 1.

Revised: Implementation of our method is available at https://github.com/kebaek/ agreement-on-the-line .

Original: Recall (Section 3.1) that given n distinct models of interest H =
Revised: Recall (Section 3.1) that given n distinct models of interest H =

Original: { h i } ni =1 , we can estimate Acc ID ( h ) ,
Revised: { h i } ni =1 , we can estimate Acc ID ( h ) ,

Original: Agr ID ( h, h ′ ) and Agr OOD ( h, h ′ )
Revised: Agr ID ( h, h ′ ) and Agr OOD ( h, h ′ )

Original: as sample averages over ID labeled validation data and OOD unlabeled data for all h, h ′ ∈ H .
Revised: as sample averages over ID labeled validation data and OOD unlabeled data for all h, h ′ ∈ H .

Original: We now describe an estimator (cid:100)
Revised: We now describe an estimator (cid:100)

Original: Acc OOD ( h ) for the OOD accuracy of amodel h ∈ H .
Revised: Acc OOD ( h ) for the OOD accuracy of a model h ∈ H .

Original: From agreement-on-the-line, we know that when ID vs OOD agreement lies on a line for allh, h ′ ∈
Revised: From agreement-on-the-line, we know that when ID vs. OOD agreement lies on a line for all h, h ′ ∈

Original: H × H , ID vs OOD accuracy for all h ∈ H would approximately also lie on the same line:
Revised: H × H , ID vs. OOD accuracy for all h ∈ H would approximately also lie on the same line:

Original: [Equation]
Revised: [Equation]

Original: We estimate the slope and bias of the linear fit by performing linear regression after applying a probittransform on the agreements as follows.
Revised: We estimate the slope and bias of the linear fit by performing linear regression after applying a probit transform on the agreements as follows.

Original: [Equation]
Revised: [Equation]

Original: For each model h ∈ H , given its ID validation accuracy, one could simply plug the estimated slope ˆ aand bias ˆ b from (4), and (cid:100)
Revised: For each model h ∈ H , given its ID validation accuracy, one could simply plug the estimated slope ˆ a and bias ˆ b from (4), and (cid:100)

Original: Acc ID ( h ) (sample average over validation set) into (3) to get an estimate ofthe model’s
Revised: Acc ID ( h ) (sample average over validation set) into (3) to get an estimate of the model’s

Original: OOD accuracy.
Revised: OOD accuracy.

Original: We call this simple algorithm ALine-S.
Revised: We call this simple algorithm ALine-S.

Original: Notice that ALine-S does not directly use the OOD agreement estimates concerning the model ofinterest—we only use agreements indirectly via the estimates ˆ a and ˆ b .
Revised: Notice that ALine-S does not directly use the OOD agreement estimates concerning the model of interest—we only use agreements indirectly via the estimates ˆ a and ˆ b .

Original: We find that a better estimatorcan be obtained by directly using the model’s
Revised: We find that a better estimator can be obtained by directly using the model’s

Original: OOD agreement estimates via simple algebra as follows.
Revised: OOD agreement estimates via simple algebra as follows.

Original: First, note that for any pair of models h, h ′ ∈ H , it directly follows from (3)
Revised: First, note that for any pair of models h, h ′ ∈ H , it directly follows from (3)

Original: that
Revised: that

Original: [Equation]
Revised: [Equation]

Original: By substituting b = Φ − 1 ( Agr OOD ( h, h ′ )) − a · Φ − 1 ( Agr ID ( h, h ′ )) into (5), we can get that average
Revised: By substituting b = Φ − 1 ( Agr OOD ( h, h ′ )) − a · Φ − 1 ( Agr ID ( h, h ′ )) into (5), we can get that average

Original: OOD accuracy of any pair of models h, h ′ ∈ H is
Revised: OOD accuracy of any pair of models h, h ′ ∈ H is

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Original:  We can plug in estimates of the terms on the right hand side ( ˆ a from linear regression (4)) and therest from sample averages.
Revised: [Equation] [Table] [Equation] We can plug in estimates of the terms on the right hand side ( ˆ a from linear regression (4)) and the rest from sample averages.

Original: In this way, we can construct a system of linear equations of the form (6)involving “unknown” estimates of the  OOD accuracy of models and other “known” quantities whichwe solve  via linear regression to obtain the unknown estimates.
Revised: In this way, we can construct a system of linear equations of the form (6) involving “unknown” estimates of the probit transformed OOD accuracy of models and other “known” quantities. We solve the system via linear regression to obtain the unknown estimates.

Original: We call this procedure ALine-D, andit is described more explicitly in Algorithm 1.
Revised: We call this procedure ALine-D, and it is described more explicitly in Algorithm 1.

Original: Note that there must be at least 3 models in the set ofinterest H for the system of linear equations in (6) to have a unique solution.
Revised: Note that there must be at least 3 models in the set of interest H for the system of linear equations in (6) to have a unique solution.

Original:  Experiments
Revised:  Experiments

Original: Datasets and models.
Revised: Datasets and models.

Original: We evaluate our methods, the simple plug in of slope/bias estimate ALine-Sand the more involved ALine-D, on the same models and datasets from Section 3 and two additionaldatasets CIFAR10C-Saturate and RxRx1- WILDS (See AppendixB and C  for details on these datasets).
Revised: We evaluate our methods, the simple plug in of slope/bias estimate ALine-S and the more involved ALine-D, on the same models and datasets from Section 3 and two additional datasets CIFAR-10C-Saturate and RxRx1- WILDS (See Appendix C and D for details on these datasets).

Original: Specifically, we look at CIFAR-10.1, CIFAR-10.2, ImageNetV2, CIFAR-10C, fMoW- WILDS , and RxRx1- WILDS , where we observe a strong correlation.
Revised: Specifically, we look at CIFAR-10.1, CIFAR-10.2, ImageNetV2, CIFAR-10C, fMoWWILDS , and RxRx1- WILDS , where we observe a strong correlation.

Original: We also look at the performance of modelson datasets where we do not see a strong linear correlation, specifically Camelyon- WILDS andiWildCam- WILDS .
Revised: We also look at the performance of models on datasets where we do not see a strong linear correlation, specifically Camelyon- WILDS and iWildCam- WILDS .

Original: Baseline methods.
Revised: Baseline methods.

Original: We choose 4 existing unlabeled estimation methods for comparison: Average
Revised: We choose 4 existing unlabeled estimation methods for comparison: Average

Original: Threshold Confidence (ATC) by Garg et al.
Revised: Threshold Confidence (ATC) by Garg et al.

Original: [29], DOC-Feat in Guillory et al.
Revised: [29], DOC-Feat in Guillory et al.

Original: [30], Average Confidence(AC) in [36], and naive Agreement [54, 58, 41].
Revised: [31], Average Confidence (AC) in [35], and naive Agreement [54, 61, 39].

Revised: All of these methods, like ALine, are based on the softmax outputs of the model.

Revised: See Appendix A for more details about previous methods.

Original: We implement the version of ATC that performedbest in the paper, i.e. with negative entropy as the score function and temperature scaling to calibratethe models in-distribution.
Revised: We implement the version of ATC that performed best in the paper, i.e. with negative entropy as the score function and temperature scaling to calibrate the models in-distribution.

Original: Although DOC was deemed the best method in Guillory et al.
Revised: Although DOC was deemed the best method in Guillory et al.

Original: [30], weuse DOC-Feat since DOC requires information from multiple OOD datasets.
Revised: [31], we use DOC-Feat since DOC requires information from multiple OOD datasets.

Original: For ATC, DOC, and AC, consistent with the experimental design in Garg et al.
Revised: For ATC, DOC, and AC, consistent with the experimental design in Garg et al.

Original: [28], we report the best number achievedbetween before versus after temperature scaling.
Revised: [28], we report the best number achieved between before versus after temperature scaling.

Original: We also compare with the more recent, ProjNorm 
Revised: We also compare with the more recent, ProjNorm by Yu et al.

Revised: [87] which showed stronger linear correlation with OOD accuracy than Rotation [23] and ATC [29].

Revised: We compare with this method separately in Section 5.1, as they do not provide a way to directly estimate the OOD accuracy.

Revised: [Table]

Original: 1.0 Prediction 0.
Revised: 1.0 Prediction 0.

Original: OO D
Revised: OO D

Original: A cc u r a c y
Revised: A cc u r a c y

Original: CIFAR-10.
Revised: CIFAR-10.

Original: y =
Revised: y =

Original: x ATC
Revised: x ATC

Original: ALine-D ATC
Revised: ALine-D ATC

Original: ALine-D 0.
Revised: ALine-D 0.

Original: 1.0 Prediction 0.
Revised: 1.0 Prediction 0.

Original: OO D
Revised: OO D

Original: A cc u r a c y
Revised: A cc u r a c y

Original: CIFAR-10-C fog y =
Revised: CIFAR-10-C fog y =

Original: x ATC
Revised: x ATC

Original: ALine-D ATC
Revised: ALine-D ATC

Original: ALine-D 0.
Revised: ALine-D 0.

Original: 1.0 Prediction 0.
Revised: 1.0 Prediction 0.

Original: OO D
Revised: OO D

Original: A cc u r a c y
Revised: A cc u r a c y

Original: Camelyon17-WILDS y =
Revised: Camelyon17-WILDS y =

Original: x ATC
Revised: x ATC

Original: ALine-D ATC ALine-D by Yu et al. [81] which showed stronger linear correlation with OOD accuracy than Rotation [23] and
Revised: ALine-

Original:  ATC[29].
Revised: [Figure] ATC

Original: We compare with this method separately in Section 5.1, as they do not provide a way todirectly estimate the OOD accuracy.
Revised: ALine-D

Original: 5.1 Main results: comparison to other methods.
Revised: 5.1 Main results: comparison to other methods.

Original: In Table 2, we observe that ALine-D generally outperforms other methods on datasets whereagreement-on-the-line holds.
Revised: In Table 2, we observe that ALine-D generally outperforms other methods on datasets where agreement-on-the-line holds.

Original: On ImageNet to ImageNetV2 and CIFAR-10 to CIFAR-10C-Snow,
Revised: On ImageNet to ImageNetV2 and CIFAR-10 to CIFAR-10C-Snow,

Original: ATC performs marginally better.
Revised: ATC performs marginally better.

Original: As can be seen in Figure 3, ATC generally cannot accurately predictthe model’s
Revised: As can be seen in Figure 3, ATC generally cannot accurately predict the model’s

Original: OOD performance for models that do not perform very well.
Revised: OOD performance for models that do not perform very well.

Original: This is consistent withexperimental results in [29] and [81].
Revised: This is consistent with experimental results in [29] and [87].

Original: On the other hand, ALine performed equally well on “bad”models land “good” models.
Revised: On the other hand, ALine perform equally well on “bad” models land “good” models.

Original: In some sense, given a collection of models where we are interestedin the performance of each, ATC, AC, DOC-Feat, and Agreement only utilize information fromthe model of interest, whereas ALine utilizes the collective information from all models for eachindividual prediction.
Revised: In some sense, given a collection of models where we are interested in the performance of each, ATC, AC, DOC-Feat, and Agreement only utilize information from the model of interest, whereas ALine utilizes the collective information from all models for each individual prediction.

Original: As expected, on datasets where we did not observe a linear correlation between ID and OODagreement (and accuracy), ALine did not perform very well, with a mean absolute estimation errorof around 5% .
Revised: As expected, on datasets where we do not observe a linear correlation between ID and OOD agreement (and accuracy), ALine does not perform very well, with a mean absolute estimation error of around 5% .

Original: Interestingly, the other methods also did not perform very well on these datasets.
Revised: Interestingly, the other methods also do not perform very well on these datasets.

Original: No method successfully predicts the OOD accuracy for every distribution shift.
Revised: No method successfully predicts the OOD accuracy for every distribution shift.

Original: The advantage of ALine is that there is a concrete way to verify when the method will successfully predict the OODaccuracy (i.e. check whether agreement is on the line).
Revised: The advantage of ALine is that there is a concrete way to verify when the method will successfully predict the OOD accuracy (i.e. check whether agreement is on the line).

Original: Other prediction methods do not have anyway of characterizing when they will be successful.
Revised: Other prediction methods do not have any way of characterizing when they will be successful.

Original: Finally, we note that ALine-D actually surpassesprevious methods even when accuracy-on-the-line does not hold, suggesting that the algorithm hasadditional beneficial properties that require further study.
Revised: Finally, we note that ALine-D actually surpasses previous methods even when accuracy-on-the-line does not hold, suggesting that the algorithm has additional beneficial properties that require further study.

Original: 5.2 Correlation analysis
Revised: 5.2 Correlation analysis

Original: [Table]

Original: [Table]
Revised: [Table]

Original: Rather than predicting OOD accuracy, it couldbe useful to have a metric that just strongly correlates with the OOD accuracy, if the applicationsimply requires an understanding of relative performance such as model selection.
Revised: Rather than predicting OOD accuracy, it could be useful to have a metric that just strongly correlates with the OOD accuracy, if the application simply requires an understanding of relative performance such as model selection.

Original: Recently, Yuet al.
Revised: Recently, Yu et al.

Original: [81] proposed ProjNorm, a measurementthey show has a very strong linear correlationwith OOD accuracy, moreso than other recentmethods including Rotation [23] and ATC [29].
Revised: [87] proposed ProjNorm, a measurement they show has a very strong linear correlation with OOD accuracy, moreso than other recent methods including Rotation [23] and ATC [29].

Original: To compare Aline-D with Projnorm, we replicate the CIFAR10C study in Yu et al.
Revised: To compare Aline-D with ProjNorm, we replicate the CIFAR-10C study in Yu et al.

Original: [81], where theytrain a ResNet18 model and predict its performance across all corruptions and severity levels of CIFAR-10C (See their Table 1 in [81]).
Revised: [87], where they train a ResNet18 model and predict its performance across all corruptions and severity levels of CIFAR-10C (See their Table 1 in [87]).

Original: Since ALine-D is an algorithm that requires a set of modelsfor prediction, we use  29 pretrained models from the CIFAR-10 testbed of Miller et al.
Revised: Since ALine-D is an algorithm that requires a set of models for prediction, we use the 29 pretrained models from the CIFAR-10 testbed of Miller et al.

Original: [56], as theother models in the set.
Revised: [59], as the other models in the set.

Original: We look at the linear correlation of the estimates of OOD accuracy andthe true accuracy across all corruptions and find that ALine-D achieves stronger correlation than
Revised: We look at the linear correlation of the estimates of OOD accuracy and the true accuracy across all corruptions and find that ALine-D achieves stronger correlation than

Original: ProjNorm (Table 3).
Revised: ProjNorm (Table 3).

Original: See Appendix D for more experimental details. 
Revised: See Appendix E for more experimental details. [Figure]

Original: 5.3 Estimating performance along a training trajectory
Revised: 5.3 Estimating performance along a training trajectory

Original: [Figure] We assess whether ALine can be utilized even in situations where the practioner only cares aboutthe performance of a few models.
Revised: We assess whether ALine can be utilized even in situations where the practioner only cares about the performance of a few models.

Original: In such situations, one could efficiently gather many models bytraining a single model and saving checkpoints along the way.
Revised: In such situations, one could efficiently gather many models by training a single model and saving checkpoints along the way.

Original: We analyze whether our phenomenonis helpful for predicting such highly correlated hypotheses, instead of independently trained models.
Revised: We analyze whether our phenomenon is helpful for predicting such highly correlated hypotheses, instead of independently trained models.

Original: In Figure 4, we collect the ID and OOD test predictions every 5 epochs across CIFAR-10 training ofa single ResNet18 model and every epoch across ImageNet training of a ResNet50 model.
Revised: In Figure 4, we collect the ID and OOD test predictions every 5 epochs across CIFAR-10 training of a single ResNet18 model and every epoch across ImageNet training of a ResNet50 model.

Original: We seethat even the agreement between every pair of checkpoints of a model across training is enough toget a good linear fit that matches the slope and bias of ID vs OOD accuracy.
Revised: We see that even the agreement between every pair of checkpoints of a model across training is enough to get a good linear fit that matches the slope and bias of ID vs. OOD accuracy.

Original: By applying ALine-D tothese checkpoints, we get a very good estimate of the OOD performance of the model across trainingepochs though slightly worse than for a collection of independently trained models.
Revised: By applying ALine-D to these checkpoints, we get a very good estimate of the OOD performance of the model across training epochs though slightly worse than for a collection of independently trained models.

Original: This suggeststhat given a model of interest, ALine does not require practitioners to train a large number of models,but just train one and save its predictions across training iterations.
Revised: This suggests that given a model of interest, ALine does not require practitioners to train a large number of models, but just train one and save its predictions across training iterations.

Original: We do a more careful ablationstudy in Appendix F, looking at the number of models required for close estimates of accuracy.
Revised: We do a more careful ablation study in Appendix G, looking at the number of models required for close estimates of accuracy.

Original:  Conclusion
Revised:  Conclusion

Original: The contributions of this work are two-fold.
Revised: The contributions of this work are two-fold.

Original: First, we observe the agreement-on-the-line phenomenon,and show that it correlates strongly with accuracy-on-the-line over a range of datasets and models.
Revised: First, we observe the agreement-on-the-line phenomenon, and show that it correlates strongly with accuracy-on-the-line over a range of datasets and models.

Original: Wealso highlight that certain aspects of this phenomenon, namely the fact that the slope and bias of thelinear fit is largely the same across agreement and accuracy, are specific to neural networks, and thusfundamentally seem connected to these classes of models.
Revised: We also highlight that certain aspects of this phenomenon, namely the fact that the slope and bias of the linear fit is largely the same across agreement and accuracy, are specific to neural networks, and thus fundamentally seem connected to these classes of models.

Original: Second, using this empirical phenomenon,we propose a surprisingly simple but effective method for predicting OOD accuracy of classifiers,while only having access to unlabeled data from the new domain (and one that can be “sanity checked”via testing whether agreement-on-the line holds).
Revised: Second, using this empirical phenomenon, we propose a surprisingly simple but effective method for predicting OOD accuracy of classifiers, while only having access to unlabeled data from the new domain (and one that can be “sanity checked” via testing whether agreement-on-the line holds).

Original: Our method outperforms existing state-of-the-artapproaches to this problem.
Revised: Our method outperforms existing state-of-the-art approaches to this problem.

Original: Importantly, we do not claim that this phenomenon is universal, butwe found it to be true across an extensive range of neural networks and OOD benchmarks that weexperimented on.
Revised: Importantly, we do not claim that this phenomenon is universal, but we found it to be true across an extensive range of neural networks and OOD benchmarks that we experimented on.

Original: In addition to its practical relevance, this observation itself reveals something veryinteresting about the way neural networks learn, which we leave for future study.
Revised: In addition to its practical relevance, this observation itself reveals something very interesting about the way neural networks learn, which we leave for future study.

Original: A Experiments from Miller et al.
Revised: Acknowledgments and Disclosure of Funding

Revised: We thank Rohan Taori and Saurabh Garg for valuable discussions regarding the model testbeds and temperature scaling used in this work, respectively. Christina Baek was supported by a Presidential Fellowship sponsored by Carnegie Mellon University.

Revised: Yiding Jiang was supported by funding from the Bosch Center for Artificial Intelligence.

Revised: Aditi Raghunathan was supported by an Open

Revised: Philanthropy AI Fellowship.

Revised: A Baseline Methods

Revised: In Section 5, we compare ALine-S/D to ATC, DOC-Feat, AC, and Agreement.

Revised: We detail these methods below.

Revised: Given model h (the output after softmax) and unlabeled OOD data, the algorithms are

Revised: [Equation]

Revised: where s ( h ( x )) = (cid:80) j h j ( x ) log( h j ( x )) and t satisfies

Revised: [Equation]

Revised: Average Confidence (AC)

Revised: Given model h and unlabeled OOD data,

Revised: [Equation]

Revised: Agreement Given a pair of models h, h ′ , Agreement simply predicts their average OOD accuracy to be the agreement between the models (cid:100)

Revised: Agr OOD ( h, h ′ ) .

Original: To get a better understanding of the agreement-on-the-line phenomenon, we replicate a theoreticalexperiments in Miller et al.
Revised: To get a better understanding of the agreement-on-the-line phenomenon, we replicate a theoretical experiments in Miller et al.

Original: [56] using the same set of neural networks from their testbed.
Revised: [59] using the same set of neural networks from their testbed.

Original: Specifically,we look at CIFAR-10 with different added Gaussian noise (their Figure 4).
Revised: Specifically, we look at CIFAR-10 with different added Gaussian noise (their Figure 4).

Original: A.1 Matching gaussian noise
Revised: B.1 Matching gaussian noise

Original: Miller et al.
Revised: Miller et al.

Original: [56] conduct a theoretical analysis on a toy gaussian mixture model to better understandthe accuracy-on-the-line phenomena.
Revised: [59] conduct a theoretical analysis on a toy gaussian mixture model to better understand the accuracy-on-the-line phenomena.

Original: From their analysis, they predict that accuracy-on-the-lineoccurs if the covariances of the ID and OOD data are the same up to some constant scaling factor.
Revised: From their analysis, they predict that accuracy-on-the-line occurs if the covariances of the ID and OOD data are the same up to some constant scaling factor.

Original: Inspired by this, they show that accuracy-on-the-line holds stronger on CIFAR-10 data corrupted withgaussian noise that matches the covariance of CIFAR-10 test data versus isotropic gaussian noise.
Revised: Inspired by this, they show that accuracy-on-the-line holds stronger on CIFAR-10 data corrupted with gaussian noise that matches the covariance of CIFAR-10 test data versus isotropic gaussian noise.

Original: Interestingly, even for this simple setting, we similarly observe that the ID vs OOD agreement trendis stronger on covariance matched gaussian noise.
Revised: Interestingly, even for this simple setting, we similarly observe that the ID vs OOD agreement trend is stronger on covariance matched gaussian noise.

Original: B Correlation results on more datasets
Revised: C Correlation results on more datasets

Original: To verify the agreement-on-the-line phenomenon, we compare the linear correlation between ID vs
Revised: To verify the agreement-on-the-line phenomenon, we compare the linear correlation between ID vs

Original: OOD accuracy and agreement across a variety of distribution shifts.
Revised: OOD accuracy and agreement across a variety of distribution shifts.

Original: Given a set of n models, we plotthe ID vs OOD accuracy of each model in the set.
Revised: Given a set of n models, we plot the ID vs OOD accuracy of each model in the set.

Original: For agreement, we randomly pair each model withanother model in the set, and plot the ID vs OOD agreement of these n pairs.
Revised: For agreement, we randomly pair each model with another model in the set, and plot the ID vs OOD agreement of these n pairs.

Original: We provide R 2 valuesin the figure legends.
Revised: We provide R 2 values in the figure legends.

Original: Datasets In addition to the 8 datasets from the main body, we also observe the trend on other
Revised: Datasets In addition to the 8 datasets from the main body, we also observe the trend on other

Original: CIFAR10C corruptions [35].
Revised: CIFAR10C corruptions [34].

Original: For each corruption, we evaluate the models over data from all 5 severitylevels (both in Figure 1 of main body and appendix).
Revised: For each corruption, we evaluate the models over data from all 5 severity levels (both in Figure 1 of main body and appendix).

Original: We also look at shifting from CIFAR-10 to CINIC-10 [19] and CIFAR-10 to STL-10 [16] whichare shifts from changes in the image source.
Revised: We also look at shifting from CIFAR-10 to CINIC-10 [19] and CIFAR-10 to STL-10 [16] which are shifts from changes in the image source.

Original: Specifically, the CINIC-10 test dataset contains both
Revised: Specifically, the CINIC-10 test dataset contains both

Original: CIFAR-10 Test data and a selection of ImageNet images for CIFAR-10 class labels downsampled to32 × 32 .
Revised: CIFAR-10 Test data and a selection of ImageNet images for CIFAR-10 class labels downsampled to 32 × 32 .

Original: We only consider the downsampled ImageNet data as the OOD dataset.
Revised: We only consider the downsampled ImageNet data as the OOD dataset.

Original: Similarly, STL-10contains processed ImageNet images for CIFAR-10 class labels.
Revised: Similarly, STLcontains processed ImageNet images for CIFAR-10 class labels.

Original: Since STL-10 is an unsupervisedlearning dataset, we only utilize the labeled subset of STL-10 as the OOD dataset.
Revised: Since STL-10 is an unsupervised learning dataset, we only utilize the labeled subset of STL-10 as the OOD dataset.

Original: Additionally,
Revised: Additionally,

Original: STL-10 only contains 9 of the 10 CIFAR-10 classes, so we restrict the dataset to just those 9 classes.
Revised: STL-10 only contains 9 of the 10 CIFAR-10 classes, so we restrict the dataset to just those 9 classes.

Original: Finally, we add results for a real-world shift from batch effects in images of cells in RxRx1- WILDS .
Revised: Finally, we add results for a real-world shift from batch effects in images of cells in RxRx1- WILDS and a reading comprehension dataset Amazon-SQuAD [58] which looks at the reading comprehension performance of models trained on paragraphs derived from Wikipedia articles on Amazon product reviews.

Original: Models For ImageNetV2, we evaluate 49 ImageNet pretrained models from the timm [77] package.
Revised: Models For ImageNetV2, we evaluate 49 ImageNet pretrained models from the timm [82] package.

Original: See their repository for more details about the models.
Revised: See their repository for more details about the models.

Original: For RxRx1- WILDS we trained 36models of varying architecture and hyperparameters.
Revised: For RxRx1- WILDS we trainedmodels of varying architecture and hyperparameters.

Original: Specifically, we vary weight decay between[10 − 1 , 10 − 2 , 10 − 3 , 10 − 4 ,10 − 5 ] and optimizers between SGD, Adam, and AdamW. See C for architecture details.
Revised: Specifically, we vary weight decay between [10 − 1 , 10 − 2 , 10 − 3 , 10 − 4 ,

Original: Finally, we utilize all independently trained models from the CIFAR10, iWildCam- WILDS , fMoWWILDS , and Camelyon17- WILDS testbeds created and utilized by [56].
Revised: Finally, we utilize all independently trained models from the CIFAR10, iWildCam- WILDS , fMoWWILDS , and Camelyon17- WILDS testbeds created and utilized by [59].

Original: The hyperparameters used totrain these models are explained in great detail in Appendix B.2 of Miller et al.
Revised: The hyperparameters used to train these models are explained in great detail in Appendix B.2 of Miller et al.

Original: The architecturesof all models evaluated for the experiment are described in more detail in Appendix C.
Revised: The architectures of all models evaluated for the experiment are described in more detail in Appendix D.

Original: Pretrained vs Not Pretrained Miller et al.
Revised: Pretrained vs Not Pretrained Miller et al.

Original: [56] showed that for some shifts, the ID vs OODaccuracy of ImageNet pretrained models follow a different linear trend than models trained fromscratch in-distribution.
Revised: [59] showed that for some shifts, the ID vs OOD accuracy of ImageNet pretrained models follow a different linear trend than models trained from scratch in-distribution.

Original: In Figure 1 of the main body, we do not distinguish between pretrained andfrom scratch models as the trends for pretrained and from scratch models were the same for the 8datasets we chose.
Revised: In Figure 1 of the main body, we do not distinguish between pretrained and from scratch models as the trends for pretrained and from scratch models were the same for thedatasets we chose.

Original: Below, we divide results between pretrained and not pretrained models to be moreprecise.
Revised: Below, we divide results between pretrained and not pretrained models to be more precise.

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: (j) CIFAR10C Gaussian Noise
Revised: (j) CIFAR10C Gaussian Noise

Original: (m) CIFAR10C JPEG Compres- sion 
Revised: (m) CIFAR10C JPEG Compres- sion n

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: j) CIFAR10C Gaussian Noise m)
Revised: g Fog (j) CIFAR10C Gaussian Noise (k)

Original: CIFAR10C  JPEG Compres- sion 
Revised: CIFAR10C Glass Blur (m) CIFAR10C JPEG Compres- sion (n) CIFAR10C Motion Blur ( Pixelate

Original: [Figure]
Revised: [Figure]

Revised: C.3 NLP Datasets

Original: [Figure]
Revised: [Figure]

Original: Here, we plot the ID vs OOD F1 score(accuracy) of the models versus the ID (SQuAD) vs OOD(Amazon) F1-agreement, where F1-agreement is simply the average F1 score where one of themodels is treated as the  ground truth.
Revised: Here [Figure] we plot the ID vs OOD F1 score of the models versus the ID (SQuAD) vs OOD (Amazon) F1-agreement, where F1-agreement is simply the average F1 score where one of the models is treated as the labels or the ground truth.

Original: We look at 99 models from CodaLab.
Revised: Specifically, [Equation] [Equation]

Original:  We will include fullexperimental details in the next draft.
Revised: [Equation] This is analogous to how agreement relates to accuracy i.e. agreement can be thought of as accuracy where one of the models is treated as the ground truth. We look at 99 models from CodaLab. Note that the linear correlation of agreement is strong with R 2 is 0 .

Original: ALine-D achieves a mean estimation error of 0.028.
Revised: ALine-D achieves a mean estimation error of 0.028.

Original: B.4 Only Neural Networks
Revised: C.4 Only Neural Networks

Original: Here we observe the ID vs OOD accuracy and agreement trend of model families other than neuralnetworks.
Revised: Here we observe the ID vs OOD accuracy and agreement trend of model families other than neural networks.

Original: In the main body of the paper, we illustrated how the agreement-on-the-line phenomenonis specific to neural networks on CIFAR10-Fog, a synthetic shift.
Revised: In the main body of the paper, we illustrated how the agreement-on-the-line phenomenon is specific to neural networks on CIFAR10-Fog, a synthetic shift.

Original: Below, we illustrate this furtheron a data replication shift, CIFAR10.
Revised: Below, we illustrate this further on a data replication shift, CIFAR10.

Original: [Figure]
Revised: [Figure]

Original: and a real-world shift, fMoW- WILDS .
Revised: and a real-world shift, fMoW- WILDS .

Original: Note that the slope ofthe ID vs OOD agreement trend of neural network models is closest to the slope of the ID vs OODaccuracy trend.
Revised: Note that the slope of the ID vs OOD agreement trend of neural network models is closest to the slope of the ID vs OOD accuracy trend.

Original: Excluding neural networks [Figure] we observe that the agreement trend of random featuremodels [15] also has a similar slope to that of the accuracy trend for select shifts such as CIFAR-10.[Figure]
Revised: Excluding neural networks, we observe that the agreement trend of random feature models [15] also has a similar slope to that of the accuracy trend for select shifts such as CIFAR-10.

Original: [Figure]
Revised: [Figure]

Original: C Model Architectures
Revised: D Model Architectures

Original: We list the types of architectures and the corresponding number of models for each testbed.
Revised: We list the types of architectures and the corresponding number of models for each testbed.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ConViT [20]

Original: ConvNeXT [50]
Revised: ConvNeXT [50]

Original: CrossViT [8]
Revised: CrossViT [8]

Original: DenseNet [37]
Revised: DenseNet [36]

Original: DLA [80]
Revised: DLA [86]

Original: EfficientNet [73]
Revised: EfficientNet [77]

Original: HaloNet [76]
Revised: HaloNet [81]

Original: NFNet [7]
Revised: NFNet [7]

Original: ResNet [33]
Revised: ResNet [33]

Original: ResNeXT [78]
Revised: ResNeXT [83]

Original: Inception v3 [72]
Revised: Inception v3 [76]

Original: VGG [68]
Revised: VGG [72]

Original: [Table]
Revised: [Table]

Original: ResNet [33]
Revised: ResNet [33]

Original: [Table]

Original: ResNeXT [78]
Revised: ResNeXT [83]

Original: [Table]

Original: AlexNet [45]
Revised: AlexNet [44]

Original: [Table]

Original: DPN68 [11]
Revised: DPN68 [11]

Original: [Table]
Revised: [Table]

Original: DenseNet121 [37]
Revised: DenseNet121 [36]

Original: [Table]
Revised: [Table]

Original: GoogLeNet [71]
Revised: GoogLeNet [75]

Original: Xception [12]
Revised: Xception [12]

Original: [Table]

Original: ShuffleNet [82]
Revised: ShuffleNet [88]

Original: [Table]

Original: MobileNetV2 [66]
Revised: MobileNetV2 [70]

Original: VGG [68]
Revised: VGG [72]

Original: [Table]

Original: PNASNet [48]
Revised: PNASNet [48]

Original: Squeezenet [39]
Revised: Squeezenet [37]

Original: [Table]

Original: ViT [25]
Revised: ViT [25]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: D Section 5: Experimental DetailsD.1 Details for Experiment 5.2: Correlation analysis
Revised: E Section 5: Experimental Details E.1 Details for Experiment 5.2: Correlation analysis

Original: We replicate the correlation analysis experiment in Table 1 of Yu et al.
Revised: We replicate the correlation analysis experiment in Table 1 of Yu et al.

Original: [81] to compare the predictionperformance of ALine-D versus ProjNorm.
Revised: [87] to compare the prediction performance of ALine-D versus ProjNorm.

Original: Essentially, we want to see how strong the linearcorrelation is between the estimate of OOD accuracy versus the true OOD accuracy by looking at thecoefficients of determination R 2 and rank correlations ρ of the fit.
Revised: Essentially, we want to see how strong the linear correlation is between the estimate of OOD accuracy versus the true OOD accuracy by looking at the coefficients of determination R 2 and rank correlations ρ of the fit.

Original: We use the GitHub repository of
Revised: We use the GitHub repository of

Original: ProjNorm [81] found at https://github.com/yaodongyu/ProjNorm to replicate their correlationanalysis experiment found in their Table 1.
Revised: ProjNorm [87] found at https://github.com/yaodongyu/ProjNorm to replicate their correlation analysis experiment found in their Table 1.

Original: Using their repository, we train a base ResNet18 modelfor 20 epochs with their default hyperparameters
Revised: Using their repository, we train a base ResNet18 model for 20 epochs with their default hyperparameters

Original: • Batch Size: 128• Learning Rate: 0.001•
Revised: • Batch Size:• Learning Rate: 0.

Original:  Weight Decay: 0• Optimizer: SGD with Momentum 0.9• Pretrained:
Revised: • Weight Decay:• Optimizer: SGD with Momentum 0. • Pretrained:

Original: True
Revised: True

Original: using cosine learning rate decay [51].
Revised: using cosine learning rate decay [51].

Original: The repository uses the default implementation of ResNet18by torchvision.
Revised: The repository uses the default implementation of ResNet by torchvision.

Original: The CIFAR-10 images are resized to be 224 × 224 , then normalized.
Revised: The CIFAR-10 images are resized to be 224 × 224 , then normalized.

Original: To compute
Revised: To compute

Original: ProjNorm, we use the repository to train a reference ResNet18 model using the pseudolabels of thebase model for 500 iterations with the same hyperparameters as the base model.
Revised: ProjNorm, we use the repository to train a reference ResNet18 model using the pseudolabels of the base model for 500 iterations with the same hyperparameters as the base model.

Original: We use the ALine-Dalgorithm to predict the OOD accuracy of the base model.
Revised: We use the ALine-D algorithm to predict the OOD accuracy of the base model.

Original: ALine-D requires a model set H withat least 3 models so that the linear system of equations (Line 9 in Algorithm 1, Equation 6 in mainbody) has a unique solution.
Revised: ALine-D requires a model set H with at least 3 models so that the linear system of equations (Line 9 in Algorithm 1, Equation 6 in main body) has a unique solution.

Original: We use the 29 pretrained models from the CIFAR10 testbed as the othermodels in the model set.
Revised: We use the 29 pretrained models from the CIFAR10 testbed as the other models in the model set.

Original: D.2 Details for Experiment 5.3: Performance along a training trajectory
Revised: E.2 Details for Experiment 5.3: Performance along a training trajectory

Original: We train a ResNet18 model from scratch (no pretrained weights) on CIFAR10 for 150 epochs usingthe following hyperparameters:
Revised: We train a ResNet18 model from scratch (no pretrained weights) on CIFAR10 for 150 epochs using the following hyperparameters:

Original: • Batch Size: 100• Learning Rate: 0.1• Weight Decay:
Revised: • Batch Size:• Learning Rate: 0.

Original: 10 − [Table] •  Optimizer: SGD with Momentum 0.[Table]
Revised: • Weight Decay: 10 −• Optimizer: SGD with Momentum 0.

Original: • Pretrained:
Revised: • Pretrained:

Original: False
Revised: False

Original: We decay the learning rate by 0.1 at epoch 75 and epoch 113.
Revised: We decay the learning rate by 0.1 at epoch 75 and epoch 113.

Original: We use simple data augmentationconsisting of RandomCrop with padding = 4 , RandomHorizontalFlip, and Normalization.
Revised: We use simple data augmentation consisting of RandomCrop with padding = 4 , RandomHorizontalFlip, and Normalization.

Original: We modify
Revised: We modify

Original: ResNet18 to take in 32 × 32 input images.
Revised: ResNet18 to take in 32 × 32 input images.

Original: During training, we save the model weights every 5 training epochs.
Revised: During training, we save the model weights every 5 training epochs.

Original: Given this collection of models,we estimate the CIFAR-10.1 accuracy of each one of these models using the ALine-D procedure.
Revised: Given this collection of models, we estimate the CIFAR-10.1 accuracy of each one of these models using the ALine-D procedure.

Original: D.3 Hardware
Revised: E.3 Hardware

Original: All experiments were conducted using GeForce GTX 1080 and 2080 graphics cards.
Revised: All experiments were conducted using GeForce GTX 1080 and 2080 graphics cards.

Original: E The relationship between agreement and accuracy
Revised: F The relationship between agreement and accuracy

Original: This work is related to Jiang et al.
Revised: This work is related to Jiang et al.

Original: [41], which shows that the agreement between two models ofthe same architecture trained with different random seeds is approximately equal to the average IDtest accuracy of models if the ensemble consisting of the models is well-calibrated .
Revised: [39], which shows that the agreement between two models of the same architecture trained with different random seeds is approximately equal to the average ID test accuracy of models if the ensemble consisting of the models is well-calibrated .

Original: They call thisequality between accuracy and agreement Generalization Disagreement Equality (GDE).
Revised: They call this equality between accuracy and agreement Generalization Disagreement Equality (GDE).

Original: Let us ignore the probit transform for a moment and assume the linear correlation between ID vs
Revised: Let us ignore the probit transform for a moment and assume the linear correlation between ID vs

Original: OOD accuracy and agreement are strong without it.
Revised: OOD accuracy and agreement are strong without it.

Original: In this simplified scenario, agreement-on-the-lineimplies that, for datasets where both agreement and accuracy are strongly linearly correlated, if IDagreement of a pair of models is equal to their average ID accuracy, then their OOD agreementis equal to their OOD accuracy.
Revised: In this simplified scenario, agreement-on-the-line implies that, for datasets where both agreement and accuracy are strongly linearly correlated, if ID agreement of a pair of models is equal to their average ID accuracy, then their OOD agreement is equal to their OOD accuracy.

Original: Formally, when a shift satisfies accuracy-on-the-line, we knowby agreement-on-the-line that for any two models trained on ID samples h, h ′ ∈ H , the followingequations are approximately satisfied (ignoring probit transform)
Revised: Formally, when a shift satisfies accuracy-on-the-line, we know by agreement-on-the-line that for any two models trained on ID samples h, h ′ ∈ H , the following equations are approximately satisfied (ignoring probit transform)

Original: [Equation]
Revised: [Equation]

Original: ) and Agr OOD 
Revised: 12) and Agr OOD ( h, h ′ ) = a · Agr ID ( h, h ′ ) + b = ⇒

Original: [Equation]
Revised: [Equation]

Original: Acc
Revised: Acc

Original: [Equation]
Revised: [Equation]

Original:  for some slope a and intercept b .
Revised: (14) for some slope a and intercept b .

Original: Thus, if the “ID gap” between accuracy and agreement is 0, then the“OOD gap” is also 0.
Revised: Thus, if the “ID gap” between accuracy and agreement is 0, then the “OOD gap” is also 0.

Original: This may suggest something about calibration on shifts where accuracy-on-theline holds: if the ensemble of a pair of models is well calibrated ID, then by agreement-on-the-line
Revised: This may suggest something about calibration on shifts where accuracy-on-theline holds: if the ensemble of a pair of models is well calibrated ID, then by agreement-on-the-line

Original: GDE also holds OOD.
Revised: GDE also holds OOD.

Original: However, agreement-on-the-line goes beyond these results in two ways: (i) agreement betweenmodels with different architectures and (ii) agreement between different checkpoints on the sametraining run is also on the ID vs OOD agreement line.
Revised: However, agreement-on-the-line goes beyond these results in two ways: (i) agreement between models with different architectures and (ii) agreement between different checkpoints on the same training run is also on the ID vs OOD agreement line.

Original: Jiang et al.
Revised: Jiang et al.

Original: [41] does not guarantee GDEholds for these cases.
Revised: [39] does not guarantee GDE holds for these cases.

Original: As can be seen in Figure 10, for most pairs of models, the ID and OOD gapsbetween accuracy and agreement are not equal to 0 i.e. GDE does not occur ID or OOD.
Revised: As can be seen in Figure 10, for most pairs of models, the ID and OOD gaps between probit scaled accuracy and agreement are not equal to 0 i.e. GDE does not occur ID or OOD.

Original: Indeed,understanding why agreement-on-the-line holds requires going beyond the theoretical conditionspresented in the prior work [41] which do not hold for this expanded set of models.
Revised: Indeed, understanding why agreement-on-the-line holds requires going beyond the theoretical conditions presented in the prior work [39] which do not hold for this expanded set of models.

Original: [Figure]
Revised: [Figure]

Original: Is agreement-on-the-line a result of neural network performing well?
Revised: Is agreement-on-the-line a result of neural network performing well?

Original: In Figures 11 we look atthe trend between ID agreement vs accuracy to observe whether the accuracy of the models in thecollection plays a role in the success of ALine-D. Note that naturally, the gap between agreement andaverage accuracy is smaller for pairs of models that perform well.
Revised: In Figures 11 we look at the trend between ID agreement vs accuracy to observe whether the accuracy of the models in the collection plays a role in the success of ALine-D. Note that naturally, the gap between agreement and average accuracy is smaller for pairs of models that perform well.

Original: However, this does not necessarilycorrespond with better ALine-D performance on such models.
Revised: However, this does not necessarily correspond with better ALine-D performance on such models.

Original: On the right figure of 11, we  side -D’s Equation  side ), and observethat all pairs lie close to the diagonal, regardless of how large the  Acc OOD − Agr OOD is  the adjustment.
Revised: On the right figure of 11, we plot the right hand side of ALine-D’s Equation 6 versus the left hand side ("adjusted agreement"), and observe that all pairs lie close to the diagonal, regardless of how large the gap Acc OOD − Agr OOD is before the adjustment.

Original: This is to say that, all pairs of varying performance have strong linear correlationbetween ID vs OOD agreement, which we show in Figure12.
Revised: This is to say that, all pairs of varying performance have strong linear correlation between ID vs OOD agreement, which we show in Figure

Original: F Ablation 
Revised: G Ablation Study G.1 How

Original: Study F.1 How many models

Original: are necessary

Original: for

Original: ALine D to make accurate predictions
Revised: many models are necessary for ALine-D to make accurate predictions?

Original: [Figure]
Revised: [Figure]

Original: ALine-D is an algorithm that requires at least 3 models in the model set so that the system of linearequations (Line 9 in Algorithm 1) has a unique solution.
Revised: ALine-D is an algorithm that requires at least 3 models in the model set so that the system of linear equations (Line 9 in Algorithm 1) has a unique solution.

Original: Additionally, it may generally requiremore models for the slope and intercept of the agreement trend to match the slope and intercept ofthe accuracy trend.
Revised: Additionally, it may generally require more models for the slope and intercept of the agreement trend to match the slope and intercept of the accuracy trend.

Original: We observe the MAE (in % ) of the ALine-D estimates of the OOD accuracyfor model sets of varying sizes.
Revised: We observe the MAE (in % ) of the ALine-D estimates of the OOD accuracy for model sets of varying sizes.

Original: For each distribution shift, we randomly sample n models fromthe testbed to be our model set.
Revised: For each distribution shift, we randomly sample n models from the testbed to be our model set.

Original: n ranges from 3 to 463 in increments of 10 for CIFAR-10 relatedshifts, and 3 to 48 in increments of 5 for ImageNetV2.
Revised: n ranges from 3 to 463 in increments of 10 for CIFAR-10 related shifts [Figure] and 3 to 48 in increments of 5 for ImageNetV2.

Original: Our ablation study shows that the success of
Revised: Our ablation study shows that the success of

Original: ALine-D is not necessarily tied to the number of models.
Revised: ALine-D is not necessarily tied to the number of models.

Original: For CIFAR-10.1, we see a very quick dropin estimation error, and ALine-D performs well even for a small model set.
Revised: For CIFAR-10.1, we see a very quick drop in estimation error [Figure] and ALine-D performs well even for a small model set.

Original: Similarly, in ImageNetV2,we observe a decrease in estimation error as the number of models increases, however, the MAE isalready pretty low from the start ( 2 . 5% ).
Revised: Similarly, in ImageNetV2, we observe a decrease in estimation error as the number of models increases, however, the MAE is already pretty low from the start ( 2 . 5% ).

Original: On the other hand, in CIFAR-10C Fog, the estimation errordoes not decrease, but the error is quite low (below 1 . 8% ) from the start.
Revised: On the other hand, in CIFAR-10C Fog, the estimation error does not decrease, but the error is quite low (below 1 . 8% ) from the start.

Original: This short ablation studyseems to indicate that ALine-D performs pretty well when agreement-on-the-line holds even for asmall number of models ( < 15 models).
Revised: This short ablation study seems to indicate that ALine-D performs pretty well when agreement-on-the-line holds even for a small number of models ( < 15 models).

Original: Additionally, it is not always the case that more models willdecrease the estimation error further.
Revised: Additionally, it is not always the case that more models will decrease the estimation error further.

Original: F.2 Does a model set of varied architecture perform better than models of the samearchitecture?
Revised: G.2 Does a model set of varied architecture perform better than models of the same architecture?

Original: We study whether the diversity from varying the architecture of the models in the model set improvesthe performance of ALine-D. We look at the performance of ALine-D on CIFAR-10.1 and
Revised: We study whether the diversity from varying the architecture of the models in the model set improves the performance of ALine-D. We look at the performance of ALine-D on CIFAR-10.1 and

Original: CIFAR10C Fog over two model sets sampled from the CIFAR-10 testbed: (A) 20 PreActResNet18 [32] models, (B) 20 models of varying architecture.
Revised: CIFAR10C Fog over two model sets sampled from the CIFAR-10 testbed: (A) 20 PreActResNet18 [32] models, (B) 20 models of varying architecture.

Original: Similary for fMoW, we look at the performance over(A) 10 DenseNet121 [38] models , (B) 10 models of varying architecture.
Revised: Similary for fMoW, we look at the performance over (A) 10 DenseNet121 [36] models , (B) 10 models of varying architecture.

Original: We randomly samplethese models from the corresponding testbeds, and average our results over 10 seeds.
Revised: We randomly sample these models from the corresponding testbeds, and average our results over 10 seeds.

Original: Our results asshown below depends on the dataset.
Revised: Our results as shown below depends on the dataset.

Original: For CIFAR-10.1 and CIFAR-10C Fog, we see that ALine-Dperforms better on the diverse set consisting of many architecture types.
Revised: For CIFAR-10.1 and CIFAR-10C Fog, we see that ALine-D performs better on the diverse set consisting of many architecture types.

Original: On the other hand, for fMoW,
Revised: On the other hand, for fMoW,

Original: ALine-D performed better on the uniform set consisting of models of one architecture.
Revised: ALine-D performed better on the uniform set consisting of models of one architecture.

Original: [Figure]
Revised: [Figure]