Original: Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shift
Revised: Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shift

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Recently, Miller et al.
Revised: Recently, Miller et al.

Original: [56] showed that a model’s in-distribution (ID) accuracy hasa strong linear correlation with its out-of-distribution (OOD) accuracy on several
Revised: [56] showed that a model’s in-distribution (ID) accuracy hasa strong linear correlation with its out-of-distribution (OOD) accuracy on several

Original: OOD benchmarks — a phenomenon they dubbed “accuracy-on-the-line”.
Revised: OOD benchmarks — a phenomenon they dubbed “accuracy-on-the-line”.

Original: Whilea useful tool for model selection (i.e., the model most likely to perform the best
Revised: Whilea useful tool for model selection (i.e., the model most likely to perform the best

Original: OOD is the one with highest ID accuracy), this fact does not help estimate theactual OOD performance of models without access to a labeled OOD validationset.
Revised: OOD is the one with highest ID accuracy), this fact does not help estimate theactual OOD performance of models without access to a labeled OOD validationset.

Original: In this paper, we show a similar but surprising phenomenon also holds forthe agreement between pairs of neural network classifiers: whenever accuracyon-the-line holds, we observe that the OOD agreement between the predictionsof any two pairs of neural networks (with potentially different architectures) alsoobserves a strong linear correlation with their ID agreement.
Revised: In this paper, we show a similar but surprising phenomenon also holds forthe agreement between pairs of neural network classifiers: whenever accuracyon-the-line holds, we observe that the OOD agreement between the predictionsof any two pairs of neural networks (with potentially different architectures) alsoobserves a strong linear correlation with their ID agreement.

Original: Furthermore, weobserve that the slope and bias of OOD vs ID agreement closely matches that of
Revised: Furthermore, weobserve that the slope and bias of OOD vs ID agreement closely matches that of

Original: OOD vs ID accuracy.
Revised: OOD vs ID accuracy.

Original: This phenomenon, which we call agreement-on-the-line ,has important practical applications: without any labeled data, we can predict the
Revised: This phenomenon, which we call agreement-on-the-line ,has important practical applications: without any labeled data, we can predict the

Original: OOD accuracy of classifiers , since OOD agreement can be estimated with justunlabeled data.
Revised: OOD accuracy of classifiers , since OOD agreement can be estimated with justunlabeled data.

Original: Our prediction algorithm outperforms previous methods both inshifts where agreement-on-the-line holds and, surprisingly, when accuracy is noton the line.
Revised: Our prediction algorithm outperforms previous methods both inshifts where agreement-on-the-line holds and, surprisingly, when accuracy is noton the line.

Original: This phenomenon also provides new insights into deep neural networks:unlike accuracy-on-the-line, agreement-on-the-line appears to only hold for neuralnetwork classifiers.
Revised: This phenomenon also provides new insights into deep neural networks:unlike accuracy-on-the-line, agreement-on-the-line appears to only hold for neuralnetwork classifiers.

Original:  Introduction
Revised:  Introduction

Original: Machine learning operates well when models observe and make decisions on inputs coming fromthe same distribution as the training data.
Revised: Machine learning operates well when models observe and make decisions on inputs coming fromthe same distribution as the training data.

Original: Yet in the real world, this assumption rarely holds.
Revised: Yet in the real world, this assumption rarely holds.

Original: Environments are never fully controlled.
Revised: Environments are never fully controlled.

Original: Robots interact with their surroundings, effectively changingwhat they sees in the future.
Revised: Robots interact with their surroundings, effectively changingwhat they sees in the future.

Original: Self-driving cars face constant distribution shift when driving tonew cities under changing weather conditions.
Revised: Self-driving cars face constant distribution shift when driving tonew cities under changing weather conditions.

Original: Models trained on clinical data from one hospitalface challenges when deployed for a different hospital with different subpopulations.
Revised: Models trained on clinical data from one hospitalface challenges when deployed for a different hospital with different subpopulations.

Original: Under these premises problem of estimating a new model trained on ( in-distribution , or ID).
Revised: Under these premises problem of estimating a new model trained on ( in-distribution , or ID).

Original: Models may sometimes break catastrophically under new conditions, or may only suffer a small degredation.
Revised: Models may sometimes break catastrophically under new conditions, or may only suffer a small degredation.

Original: Differentiating between such cases is crucial
Revised: Differentiating between such cases is crucial

Original: Assessing OOD performance is difficult because in reality, labeled OOD data is often very costly toobtain.
Revised: Assessing OOD performance is difficult because in reality, labeled OOD data is often very costly toobtain.

Original: On the other hand, unlabeled OOD data is much easier to obtain.
Revised: On the other hand, unlabeled OOD data is much easier to obtain.

Original: A natural question is whetherwe can leverage unlabeled OOD data for estimating the OOD performance.
Revised: A natural question is whetherwe can leverage unlabeled OOD data for estimating the OOD performance.

Original: This paradigm of usingunlabeled data to predict the OOD generalization performance has received much attention recently[29, 9, 81, 22, 23, 10, 30].
Revised: This paradigm of usingunlabeled data to predict the OOD generalization performance has received much attention recently[29, 9, 81, 22, 23, 10, 30].

Original: Though many different metrics have been proposed, their success varies
Revised: Though many different metrics have been proposed, their success varies

Original: [Figure]
Revised: [Figure]

Original: widely depending on the shift and the ID performance of the model.
Revised: widely depending on the shift and the ID performance of the model.

Original: While it is in fact impossible fora method to always work with no assumptions [29], a major hurdle in using these methods is thatthere is currently no understanding of when they work or a recipe to detect when their predictionsmight be unreliable.
Revised: While it is in fact impossible fora method to always work with no assumptions [29], a major hurdle in using these methods is thatthere is currently no understanding of when they work or a recipe to detect when their predictionsmight be unreliable.

Original: In a separate but related line of work, Miller et al.
Revised: In a separate but related line of work, Miller et al.

Original: [56] demonstrated that in a wide variety ofcommon OOD prediction benchmarks such as CIFAR-10.1 [64], ImageNetV2 [64], CIFAR-10C[35], fMoW- WILDS [13], there exists an almost perfect positive linear correlation between the IDtest vs OOD accuracy of models.
Revised: [56] demonstrated that in a wide variety ofcommon OOD prediction benchmarks such as CIFAR-10.1 [64], ImageNetV2 [64], CIFAR-10C[35], fMoW- WILDS [13], there exists an almost perfect positive linear correlation between the IDtest vs OOD accuracy of models.

Original: When this phenomenon, called accuracy-on-the-line , occurs,improving performance on the ID test data directly leads to improvements on OOD performance.
Revised: When this phenomenon, called accuracy-on-the-line , occurs,improving performance on the ID test data directly leads to improvements on OOD performance.

Original: Furthermore, if we have access to the slope and bias of this correlation, predicting OOD accuracybecomes straightforward.
Revised: Furthermore, if we have access to the slope and bias of this correlation, predicting OOD accuracybecomes straightforward.

Original: Unfortunately, accuracy-on-the-line is not universal.
Revised: Unfortunately, accuracy-on-the-line is not universal.

Original: In some datasets, suchas Camelyon17- WILDS [1], models with the same ID test performance had OOD performance thatvaried largely.
Revised: In some datasets, suchas Camelyon17- WILDS [1], models with the same ID test performance had OOD performance thatvaried largely.

Original: Thus, while the accuracy-on-the-line phenomenon is interesting, its practical use issomewhat limited since determining whether accuracy-on-the-line holds itself requires labeled OODdata.
Revised: Thus, while the accuracy-on-the-line phenomenon is interesting, its practical use issomewhat limited since determining whether accuracy-on-the-line holds itself requires labeled OODdata.

Original: In this work, we begin by observing an analogous phenomenon based upon agreement rather thanaccuracy.
Revised: In this work, we begin by observing an analogous phenomenon based upon agreement rather thanaccuracy.

Original: Specifically, if we consider pairs of neural network of classifiers, and look at the agreementof their predictions (the proportion of cases where they make the same prediction, which requiresno labeled data to compute), we find that there also often exists a strong linear correlation between
Revised: Specifically, if we consider pairs of neural network of classifiers, and look at the agreementof their predictions (the proportion of cases where they make the same prediction, which requiresno labeled data to compute), we find that there also often exists a strong linear correlation between

Original: ID vs OOD agreement.
Revised: ID vs OOD agreement.

Original: We call this phenomenon agreement-on-the-line .
Revised: We call this phenomenon agreement-on-the-line .

Original: Importantly, however, thisphenomenon appears to be tightly coupled with accuracy-on-the-line: when agreement-on-the-lineholds, accuracy-on-the-line also holds; and when agreement-on-the-line does not hold, neither doesaccuracy-on-the-line.
Revised: Importantly, however, thisphenomenon appears to be tightly coupled with accuracy-on-the-line: when agreement-on-the-lineholds, accuracy-on-the-line also holds; and when agreement-on-the-line does not hold, neither doesaccuracy-on-the-line.

Original: Furthermore, in the case of neural network classifiers, when these propertieshold, the linear correlations of both accuracy-on-the-line and agreement-on-the-line appear to haveroughly the same slope and bias .
Revised: Furthermore, in the case of neural network classifiers, when these propertieshold, the linear correlations of both accuracy-on-the-line and agreement-on-the-line appear to haveroughly the same slope and bias .

Original: Interestingly, unlike accuracy-on-the-line, which appears to be ageneral phenomenon, agreement-on-the-line, especially the fact that the slope and bias of the linearcorrelation agree across accuracy and agreement, appears to occur only for neural networks.
Revised: Interestingly, unlike accuracy-on-the-line, which appears to be ageneral phenomenon, agreement-on-the-line, especially the fact that the slope and bias of the linearcorrelation agree across accuracy and agreement, appears to occur only for neural networks.

Original: Indeedthe phenomenon is quite unintuitive, given that there is no a-priori reason to believe that agreementand accuracy would be connected in such a manner; nonetheless, we find this phenomenon occursrepeatedly across multiple datasets and vastly different deep network architectures.
Revised: Indeedthe phenomenon is quite unintuitive, given that there is no a-priori reason to believe that agreementand accuracy would be connected in such a manner; nonetheless, we find this phenomenon occursrepeatedly across multiple datasets and vastly different deep network architectures.

Original: This phenomenon is of immediate practical interest.
Revised: This phenomenon is of immediate practical interest.

Original: Since agreement-on-the-line can be validatedwithout any labeled OOD data, we can immediately use it as a proxy to assess whether accuracy-onthe-line holds, and thus whether it is reasonable to use ID accuracy as model selection criteria forpicking a model on OOD data.
Revised: Since agreement-on-the-line can be validatedwithout any labeled OOD data, we can immediately use it as a proxy to assess whether accuracy-onthe-line holds, and thus whether it is reasonable to use ID accuracy as model selection criteria forpicking a model on OOD data.

Original: Furthermore, since the slope and bias of the agreement-on-the-line fitcan also be estimated without labeled OOD data, (for the cases where agreement-on-the-line holds)we can use this approach to derive a very simple algorithm for estimating the OOD generalization of classifiers, without any access to labeled OOD data.
Revised: Furthermore, since the slope and bias of the agreement-on-the-line fitcan also be estimated without labeled OOD data, (for the cases where agreement-on-the-line holds)we can use this approach to derive a very simple algorithm for estimating the OOD generalization of classifiers, without any access to labeled OOD data.

Original: The approach outperforms competing methodsand predicts OOD test error with a mean absolute estimation error of ≤ 2% on datasets whereagreement-on-the-line holds.
Revised: The approach outperforms competing methodsand predicts OOD test error with a mean absolute estimation error of ≤ 2% on datasets whereagreement-on-the-line holds.

Original: On datasets where agreement-on-the-line does not hold, the method, asexpected does not perform as well, but somewhat surprisingly still outperforms competing methodsin terms of predicting OOD performance.
Revised: On datasets where agreement-on-the-line does not hold, the method, asexpected does not perform as well, but somewhat surprisingly still outperforms competing methodsin terms of predicting OOD performance.

Original: To summarize, our contributions are as follows:
Revised: To summarize, our contributions are as follows:

Original: We discover and empirically analyze the agreement-on-the-line phenomenon: that ID vsOOD agreement for pairs of classifiers lies on a line precisely when the corresponding IDvs OOD accuracy also lies on a line.
Revised: We discover and empirically analyze the agreement-on-the-line phenomenon: that ID vsOOD agreement for pairs of classifiers lies on a line precisely when the corresponding IDvs OOD accuracy also lies on a line.

Original: Furthermore, for the case of neural network classifiers,the slope and bias of these two lines are approximately equal.2.
Revised: Furthermore, for the case of neural network classifiers,the slope and bias of these two lines are approximately equal.2.

Original: Exploiting this phenomenon, we develop a simple method for estimating the OOD performance of classifiers without any access to labeled OOD data (and by observing whetheragreement-on-the-line holds, the method also provides a “sanity check” that these estimatesare reasonable).
Revised: Exploiting this phenomenon, we develop a simple method for estimating the OOD performance of classifiers without any access to labeled OOD data (and by observing whetheragreement-on-the-line holds, the method also provides a “sanity check” that these estimatesare reasonable).

Original: The proposed method outperforms all competing baselines for this task.
Revised: The proposed method outperforms all competing baselines for this task.

Original:  Related Works
Revised:  Related Works

Original: Estimating ID generalization via agreement.
Revised: Estimating ID generalization via agreement.

Original: Departing from approaches based on uniformconvergence [60, 26, 3, 57], several recent works [41, 59, 83, 28] derive different approaches forestimating generalization error or comparing different models.
Revised: Departing from approaches based on uniformconvergence [60, 26, 3, 57], several recent works [41, 59, 83, 28] derive different approaches forestimating generalization error or comparing different models.

Original: In particular, this work is closelyrelated to Jiang et al.
Revised: In particular, this work is closelyrelated to Jiang et al.

Original: [41], which shows that the disagreement between two models trained withdifferent random seeds closely tracks the ID generalization error of the models if the ensembles of themodels are well-calibrated.
Revised: [41], which shows that the disagreement between two models trained withdifferent random seeds closely tracks the ID generalization error of the models if the ensembles of themodels are well-calibrated.

Original: Predicting ID generalization via disagreement has also previously beenproposed by Madani et al.
Revised: Predicting ID generalization via disagreement has also previously beenproposed by Madani et al.

Original: [54] and Nakkiran and Bansal [58].
Revised: [54] and Nakkiran and Bansal [58].

Original: Our method also uses disagreementbut unlike these works, focuses on out-of-distribution generalization.
Revised: Our method also uses disagreementbut unlike these works, focuses on out-of-distribution generalization.

Original: OOD generalization.
Revised: OOD generalization.

Original: The problem of characterizing generalization in OOD setting is more challenging than the ID setting and is relatively understudied.
Revised: The problem of characterizing generalization in OOD setting is more challenging than the ID setting and is relatively understudied.

Original: Ben-David et al.
Revised: Ben-David et al.

Original: [5] provides one of thefirst uniform-convergence-based bounds for domain adaptation .
Revised: [5] provides one of thefirst uniform-convergence-based bounds for domain adaptation .

Original: Several works [55, 18, 47] build onthis approach and extend it to other learning scenarios.
Revised: Several works [55, 18, 47] build onthis approach and extend it to other learning scenarios.

Original: Most of these works attempt to bound thedifference between ID and OOD performance via a certain notion of closeness (e.g., the total variationdistance and the H ∆ H divergence which is related to agreement) between the original distributionand shifted distribution and build on the uniform-convergence framework [65].
Revised: Most of these works attempt to bound thedifference between ID and OOD performance via a certain notion of closeness (e.g., the total variationdistance and the H ∆ H divergence which is related to agreement) between the original distributionand shifted distribution and build on the uniform-convergence framework [65].

Original: As pointed out by
Revised: As pointed out by

Original: Miller et al.
Revised: Miller et al.

Original: [56], these approaches provide an upper bound on the OOD performance that growslooser as the distribution shift becomes larger, and these upper bounds do not capture the precisetrends observed in practice.
Revised: [56], these approaches provide an upper bound on the OOD performance that growslooser as the distribution shift becomes larger, and these upper bounds do not capture the precisetrends observed in practice.

Original: Predicting the actual OOD performance using unlabeled data has gainedinterest in the past decade.
Revised: Predicting the actual OOD performance using unlabeled data has gainedinterest in the past decade.

Original: These methods can roughly be divided into three categories:1.
Revised: These methods can roughly be divided into three categories:1.

Original: Placing assumptions on the distribution shift.
Revised: Placing assumptions on the distribution shift.

Original: Donmez et al.
Revised: Donmez et al.

Original: [24] assume knowledge of themarginal of the shifted label distribution P ( y ) and show that OOD accuracy can be predicted if theshifted distribution satisfies several properties.
Revised: [24] assume knowledge of themarginal of the shifted label distribution P ( y ) and show that OOD accuracy can be predicted if theshifted distribution satisfies several properties.

Original: Steinhardt and Liang [70] work under the assumptionthat the data x can be separated into “views” that are conditionally independent given label y .
Revised: Steinhardt and Liang [70] work under the assumptionthat the data x can be separated into “views” that are conditionally independent given label y .

Original: Chenet al.
Revised: Chenet al.

Original: [10] assume prior knowledge about the shift and use an importance weighting procedure.2.
Revised: [10] assume prior knowledge about the shift and use an importance weighting procedure.2.

Original: Placing assumptions on the classifiers.
Revised: Placing assumptions on the classifiers.

Original: Given multiple classifiers of interest, Platanios et al.[61, 62] places assumptions on the classifiers and use logical constraints to identify each one’s error.
Revised: Given multiple classifiers of interest, Platanios et al.[61, 62] places assumptions on the classifiers and use logical constraints to identify each one’s error.

Original: On the other hand, Jaffe et al.
Revised: On the other hand, Jaffe et al.

Original: [40] use a spectral-based approach under the assumption that classifiersmake independent errors.3.
Revised: [40] use a spectral-based approach under the assumption that classifiersmake independent errors.3.

Original: Empirically measuring the distribution shift.
Revised: Empirically measuring the distribution shift.

Original: A group of works [27, 67, 22, 23] train aregression model over metrics that measure the severity of the distribution shift.
Revised: A group of works [27, 67, 22, 23] train aregression model over metrics that measure the severity of the distribution shift.

Original: Inspired by theobservation that the maximum softmax probability (or confidence) for OOD points is typicallylower [36, 34], Guillory et al.
Revised: Inspired by theobservation that the maximum softmax probability (or confidence) for OOD points is typicallylower [36, 34], Guillory et al.

Original: [30] and more recently Garg et al.
Revised: [30] and more recently Garg et al.

Original: [29] utilize model confidence topredict accuracy.
Revised: [29] utilize model confidence topredict accuracy.

Original: This method was extended upon by Chen et al.
Revised: This method was extended upon by Chen et al.

Original: Chuang et al.
Revised: Chuang et al.

Original: [14] use a set ofdomain-invariant predictors as a proxy for the unknown, true target labels.
Revised: [14] use a set ofdomain-invariant predictors as a proxy for the unknown, true target labels.

Original: Yu et al.
Revised: Yu et al.

Original: [81] observedthat the distance between the model of interest f and a reference model trained on the pseudolabelsof f showed strong linear correlation with OOD accuracy.
Revised: [81] observedthat the distance between the model of interest f and a reference model trained on the pseudolabelsof f showed strong linear correlation with OOD accuracy.

Original: [Table]
Revised: [Table]

Original: Though a large number of methods have been proposed, for the large majority, it is not well-understoodwhen they will work.
Revised: Though a large number of methods have been proposed, for the large majority, it is not well-understoodwhen they will work.

Original: Intuitively, no method will work on all shifts without additional assumptions[29].
Revised: Intuitively, no method will work on all shifts without additional assumptions[29].

Original: But is there some simple general structure to shifts in the real world that allows us to reliablypredict OOD accuracy?
Revised: But is there some simple general structure to shifts in the real world that allows us to reliablypredict OOD accuracy?

Original: Even if such a structure is not universal, can we easily inspect if this structureholds?
Revised: Even if such a structure is not universal, can we easily inspect if this structureholds?

Original: What is a plausible assumption we can make about the OOD behaviour of classifiers ?
Revised: What is a plausible assumption we can make about the OOD behaviour of classifiers ?

Original: Thenovelty and significance of our work comes from trying to better understand and address thesequestions, specifically for neural networks.
Revised: Thenovelty and significance of our work comes from trying to better understand and address thesequestions, specifically for neural networks.

Original: In this work, we observe a phenomenon related to, butstronger than accuracy-on-the-line that allows us to reliably predict the OOD accuracy of neuralnetworks.
Revised: In this work, we observe a phenomenon related to, butstronger than accuracy-on-the-line that allows us to reliably predict the OOD accuracy of neuralnetworks.

Original:  The agreement-on-the-line phenomenon
Revised:  The agreement-on-the-line phenomenon

Original: 3.1 Notation and setup
Revised: 3.1 Notation and setup

Original: Let H denote a set of neural networks trained on ( X train , y train )
Revised: Let H denote a set of neural networks trained on ( X train , y train )

Original: = { ( x i , y i ) } m train i =1 sampled from D ID .
Revised: = { ( x i , y i ) } m train i =1 sampled from D ID .

Original: Given any pair of models h, h ′ ∈ H , for a distribution D , the expected accuracy and agreement aredefined as:
Revised: Given any pair of models h, h ′ ∈ H , for a distribution D , the expected accuracy and agreement aredefined as:

Original: [Equation]
Revised: [Equation]

Original: We assume access to labeled validation set ( X val , y val ) =
Revised: We assume access to labeled validation set ( X val , y val ) =

Original: { ( x i , y i ) } m Val i =1 sampled from D ID thatallows us to estimate the ID accuracy (cid:100)
Revised: { ( x i , y i ) } m Val i =1 sampled from D ID thatallows us to estimate the ID accuracy (cid:100)

Original: Acc ID ( h ) as the sample average of 1 { h ( x ) =
Revised: Acc ID ( h ) as the sample average of 1 { h ( x ) =

Original: y } over thevalidation set.
Revised: y } over thevalidation set.

Original: We do not assume access to a labeled OOD validation set, as this is often impracticalto obtain, and thereby cannot directly estimate (cid:100)
Revised: We do not assume access to a labeled OOD validation set, as this is often impracticalto obtain, and thereby cannot directly estimate (cid:100)

Original: Acc OOD ( h ) in a similar manner.
Revised: Acc OOD ( h ) in a similar manner.

Original: Agreement, on the other hand, only requires access to unlabeled data.
Revised: Agreement, on the other hand, only requires access to unlabeled data.

Original: We assume access to unlabeled 143 samples X
Revised: We assume access to unlabeled 143 samples X

Original: [Equation]
Revised: [Equation]

Original: 1 from the shifted distribution of interest as D OOD .
Revised: 1 from the shifted distribution of interest as D OOD .

Original: Hence, we can estimate 144 both the ID and OOD agreement as follows:
Revised: Hence, we can estimate 144 both the ID and OOD agreement as follows:

Original: [Equation]
Revised: [Equation]

Original: 3.2 Experimental setup
Revised: 3.2 Experimental setup

Original: We study the ID vs OOD accuracy and agreement between pairs of models across more than 20common
Revised: We study the ID vs OOD accuracy and agreement between pairs of models across more than 20common

Original: OOD benchmarks and hundreds of independently trained neural networks.
Revised: OOD benchmarks and hundreds of independently trained neural networks.

Original: Datasets.
Revised: Datasets.

Original: We present results on 8 dataset shifts in the main paper, and include results for otherdistribution shifts in the Appendix B. These 8 datasets span:
Revised: We present results on 8 dataset shifts in the main paper, and include results for otherdistribution shifts in the Appendix B. These 8 datasets span:

Original: Dataset reproductions: CIFAR-10.1 [64], CIFAR-10.2 [52] reproductions of CIFAR-10 [44]
Revised: Dataset reproductions: CIFAR-10.1 [64], CIFAR-10.2 [52] reproductions of CIFAR-10 [44]

Original: and ImageNetV2 [64] reproduction of ImageNet [21]
Revised: and ImageNetV2 [64] reproduction of ImageNet [21]

Original: Synthetic corruptions: CIFAR-10C Fog and CIFAR-10C Snow [35]
Revised: Synthetic corruptions: CIFAR-10C Fog and CIFAR-10C Snow [35]

Original: Real-world shifts from [43]: satellite images (fMoW- WILDS ), images from camera traps in
Revised: Real-world shifts from [43]: satellite images (fMoW- WILDS ), images from camera traps in

Original: the wildlife (iWildCam- WILDS [4])
Revised: the wildlife (iWildCam- WILDS [4])

Original: and images of cancer tissue (Camelyon17- WILDS [1])
Revised: and images of cancer tissue (Camelyon17- WILDS [1])

Original: [Figure]
Revised: [Figure]

Original: Models.
Revised: Models.

Original: For ImageNetV2, we evaluate 50 ImageNet pretrained models from the timm [77] package.
Revised: For ImageNetV2, we evaluate 50 ImageNet pretrained models from the timm [77] package.

Original: On all other shifts, we evaluate on all independently trained models in the testbed created andutilized by [56] consisting of ≥ 150 models for each shift.
Revised: On all other shifts, we evaluate on all independently trained models in the testbed created andutilized by [56] consisting of ≥ 150 models for each shift.

Original: The evaluated models span a variety ofconvolutional neural networks (e.g. ResNet [31], DenseNet [38], EfficientNet [74], VGG [49]) aswell as various Vision Transformers [25].
Revised: The evaluated models span a variety ofconvolutional neural networks (e.g. ResNet [31], DenseNet [38], EfficientNet [74], VGG [49]) aswell as various Vision Transformers [25].

Original: All architectures and models are listed in the Appendix C.
Revised: All architectures and models are listed in the Appendix C.

Original: Probit scaling.
Revised: Probit scaling.

Original: Miller et al.
Revised: Miller et al.

Original: [56] report their results after probit scaling Φ − 1 ( · ) 1 the ID vs OODaccuracies due to a better linear fit.
Revised: [56] report their results after probit scaling Φ − 1 ( · ) 1 the ID vs OODaccuracies due to a better linear fit.

Original: We apply the same probit transform to both accuracy andagreement in our experiments.
Revised: We apply the same probit transform to both accuracy andagreement in our experiments.

Original: 3.3 Observations
Revised: 3.3 Observations

Original: We empirically observe a peculiar phenomenon in deep neural networks, which we call agreementon-the-line characterized by the following three properties:
Revised: We empirically observe a peculiar phenomenon in deep neural networks, which we call agreementon-the-line characterized by the following three properties:

Original: Prop(
Revised: Prop(

Original: i)
Revised: i)

Original: When ID vs OOD accuracy observes a strong linear correlation ( ≥ 0 .
Revised: When ID vs OOD accuracy observes a strong linear correlation ( ≥ 0 .

Original: 95 R 2 values), we seethat ID vs OOD agreement is also strongly linearly correlated .
Revised: 95 R 2 values), we seethat ID vs OOD agreement is also strongly linearly correlated .

Original: Prop(ii) When both accuracy and agreement observe strong linear correlation, we see that theselinear correlations have almost the same slope and bias .
Revised: Prop(ii) When both accuracy and agreement observe strong linear correlation, we see that theselinear correlations have almost the same slope and bias .

Original: Prop(iii)
Revised: Prop(iii)

Original: When the linear correlation of ID vs OOD accuracy is weak ( ≤ 0 .
Revised: When the linear correlation of ID vs OOD accuracy is weak ( ≤ 0 .

Original: 75 R 2 values), the linearcorrelation between ID and OOD agreement is similarly weak.
Revised: 75 R 2 values), the linearcorrelation between ID and OOD agreement is similarly weak.

Original: We show the agreement-on-the-line phenomenon on 8 datasets in Figure 1 and Table 1 (See Appendix B for other datasets).
Revised: We show the agreement-on-the-line phenomenon on 8 datasets in Figure 1 and Table 1 (See Appendix B for other datasets).

Original: On CIFAR-10.1, CIFAR-10.2, ImageNetV2, CIFAR-10C Fog/Snow,and fMoW- WILDS , we find that both ID vs OOD accuracy and agreement observe strong linearcorrelations, and the linear fits have the same slope and bias (Prop(i), Prop(ii)).
Revised: On CIFAR-10.1, CIFAR-10.2, ImageNetV2, CIFAR-10C Fog/Snow,and fMoW- WILDS , we find that both ID vs OOD accuracy and agreement observe strong linearcorrelations, and the linear fits have the same slope and bias (Prop(i), Prop(ii)).

Original: On the other hand, onthe datasets Camelyon17- WILDS and iWildCam- WILDS where accuracy is not linearly correlated,agreement is also not linearly correlated (Prop(iii)).
Revised: On the other hand, onthe datasets Camelyon17- WILDS and iWildCam- WILDS where accuracy is not linearly correlated,agreement is also not linearly correlated (Prop(iii)).

Original: 3.4 What makes agreement-on-the-line interesting?
Revised: 3.4 What makes agreement-on-the-line interesting?

Original: First, agreement can be estimated with just unlabeled data .
Revised: First, agreement can be estimated with just unlabeled data .

Original: Hence, the agreement-on-the-linephenomenon has important practical implications for both checking whether the distribution shiftobserves accuracy-on-the-line and predicting the actual value of OOD accuracy without any OODlabels.
Revised: Hence, the agreement-on-the-linephenomenon has important practical implications for both checking whether the distribution shiftobserves accuracy-on-the-line and predicting the actual value of OOD accuracy without any OODlabels.

Original: We present a method to estimate OOD error using this phenomenon in Section 4.
Revised: We present a method to estimate OOD error using this phenomenon in Section 4.

Original: Second, agreement-on-the-line does not directly follow from accuracy-on-the-line.
Revised: Second, agreement-on-the-line does not directly follow from accuracy-on-the-line.

Original: Prior work hasobserved that expected ID accuracy often equals ID agreement over pairs of models with the samearchitecture, trained on the same dataset but with different random seeds [41].
Revised: Prior work hasobserved that expected ID accuracy often equals ID agreement over pairs of models with the samearchitecture, trained on the same dataset but with different random seeds [41].

Original: Agreement-on-the-linegoes beyond these results in two ways: (i) agreement between models with different architectures(Fig. 2) and (ii) agreement between different checkpoints on the same training run (Fig. 4) is also on the ID vs OOD agreement line.
Revised: Agreement-on-the-linegoes beyond these results in two ways: (i) agreement between models with different architectures(Fig. 2) and (ii) agreement between different checkpoints on the same training run (Fig. 4) is also on the ID vs OOD agreement line.

Original: These ID/OOD agreements do not equal the expected
Revised: These ID/OOD agreements do not equal the expected

Original: ID/OOD accuracy.
Revised: ID/OOD accuracy.

Original: Indeed, understanding why agreement-on-the-line holds requires going beyondthe theoretical conditions presented in the prior work [41] which do not hold for this expanded set ofmodels.
Revised: Indeed, understanding why agreement-on-the-line holds requires going beyondthe theoretical conditions presented in the prior work [41] which do not hold for this expanded set ofmodels.

Original: See Appendix E for further discussion.
Revised: See Appendix E for further discussion.

Original: Finally, we emphasize that there is something special about neural networks that makes the ID vs
Revised: Finally, we emphasize that there is something special about neural networks that makes the ID vs

Original: OOD agreement trend identical to the ID vs OOD accuracy trend.
Revised: OOD agreement trend identical to the ID vs OOD accuracy trend.

Original: This is unlike accuracy-on-theline that holds across a wide range of models including neural networks and classical approaches.
Revised: This is unlike accuracy-on-theline that holds across a wide range of models including neural networks and classical approaches.

Original: Figure 2 shows CIFAR-10 Test vs CIFAR-10C Fog accuracy and agreement of linear models (e.g.logistic and ridge regression) and various non-linear models (e.g. Kernel SVM [17], k-Nearest
Revised: Figure 2 shows CIFAR-10 Test vs CIFAR-10C Fog accuracy and agreement of linear models (e.g.logistic and ridge regression) and various non-linear models (e.g. Kernel SVM [17], k-Nearest

Original: Neighbors, Random Forests [6], Random Features [15], AdaBoost [84]).
Revised: Neighbors, Random Forests [6], Random Features [15], AdaBoost [84]).

Original: See plots for other datasetsin
Revised: See plots for other datasetsin

Original: Appendix B. We look at agreement between pairs of models from the same model family.
Revised: Appendix B. We look at agreement between pairs of models from the same model family.

Original: While
Revised: While

Original: Prop(i) seems to hold for several other model families on several shifts, Prop(ii) only holds for neuralnetworks, i.e. the slope and bias of the agreement line do not match the slope and bias of the accuracyline for other model families.
Revised: Prop(i) seems to hold for several other model families on several shifts, Prop(ii) only holds for neuralnetworks, i.e. the slope and bias of the agreement line do not match the slope and bias of the accuracyline for other model families.

Original:  A method for estimating OOD accuracy
Revised:  A method for estimating OOD accuracy

Original: In this section, we describe how the phenomenon of agreement-on-the-line (described in Section 3)offers a simple practical method to perform model selection and estimate accuracy under distributionshifts.
Revised: In this section, we describe how the phenomenon of agreement-on-the-line (described in Section 3)offers a simple practical method to perform model selection and estimate accuracy under distributionshifts.

Original: Recall from Section 3.1 that we have labeled ID validation data ( X val , y val ) and unlabeled
Revised: Recall from Section 3.1 that we have labeled ID validation data ( X val , y val ) and unlabeled

Original: OOD data X OOD .
Revised: OOD data X OOD .

Original: Model selection.
Revised: Model selection.

Original: Without OOD labeled data, can we determine which model is likely to achievethe best OOD performance?
Revised: Without OOD labeled data, can we determine which model is likely to achievethe best OOD performance?

Original: When accuracy-on-the-line holds and ID vs OOD accuracy is linearly correlated, we can simply pick the model with highest ID accuracy.
Revised: When accuracy-on-the-line holds and ID vs OOD accuracy is linearly correlated, we can simply pick the model with highest ID accuracy.

Original: In practice, how doesone determine if accuracy-on-the-line holds without labeled OOD data?
Revised: In practice, how doesone determine if accuracy-on-the-line holds without labeled OOD data?

Original: By Prop(i) and Prop(iii),agreement-on-the-line implies accuracy-on-the-line.
Revised: By Prop(i) and Prop(iii),agreement-on-the-line implies accuracy-on-the-line.

Original: Hence, we simply need to check if ID and OODagreement (which can be estimated as in (2)) are linearly correlated, in order to know if our modelselection criterion based on ID accuracy is valid.
Revised: Hence, we simply need to check if ID and OODagreement (which can be estimated as in (2)) are linearly correlated, in order to know if our modelselection criterion based on ID accuracy is valid.

Original: OOD error prediction.
Revised: OOD error prediction.

Original: Agreement-on-the-line allows us to go beyond model selection and actuallypredict OOD accuracy .
Revised: Agreement-on-the-line allows us to go beyond model selection and actuallypredict OOD accuracy .

Original: Intuitively, we can estimate the slope and bias of the agreement line with justunlabeled data.
Revised: Intuitively, we can estimate the slope and bias of the agreement line with justunlabeled data.

Original: By Prop(ii), they match the slope and bias of the accuracy line and hence, we canestimate the OOD accuracy by linearly transforming the ID accuracy (with the appropriate probitscaling).
Revised: By Prop(ii), they match the slope and bias of the accuracy line and hence, we canestimate the OOD accuracy by linearly transforming the ID accuracy (with the appropriate probitscaling).

Original: We formalize this intuition below and provide an algorithm for OOD accuracy estimation in
Revised: We formalize this intuition below and provide an algorithm for OOD accuracy estimation in

Original: Algorithm 1.
Revised: Algorithm 1.

Original: Recall (Section 3.1) that given n distinct models of interest H =
Revised: Recall (Section 3.1) that given n distinct models of interest H =

Original: { h i } ni =1 , we can estimate Acc ID ( h ) ,
Revised: { h i } ni =1 , we can estimate Acc ID ( h ) ,

Original: Agr ID ( h, h ′ ) and Agr OOD ( h, h ′ )
Revised: Agr ID ( h, h ′ ) and Agr OOD ( h, h ′ )

Original: as sample averages over ID labeled validation data and OOD unlabeled data for all h, h ′ ∈ H .
Revised: as sample averages over ID labeled validation data and OOD unlabeled data for all h, h ′ ∈ H .

Original: We now describe an estimator (cid:100)
Revised: We now describe an estimator (cid:100)

Original: Acc OOD ( h ) for the OOD accuracy of amodel h ∈ H .
Revised: Acc OOD ( h ) for the OOD accuracy of amodel h ∈ H .

Original: From agreement-on-the-line, we know that when ID vs OOD agreement lies on a line for allh, h ′ ∈
Revised: From agreement-on-the-line, we know that when ID vs OOD agreement lies on a line for allh, h ′ ∈

Original: H
Revised: H

Original: × H , ID vs OOD accuracy for all h ∈ H would approximately also lie on the same line:
Revised: × H , ID vs OOD accuracy for all h ∈ H would approximately also lie on the same line:

Original: [Equation]
Revised: [Equation]

Original: We estimate the slope and bias of the linear fit by performing linear regression after applying a probittransform on the agreements as follows.
Revised: We estimate the slope and bias of the linear fit by performing linear regression after applying a probittransform on the agreements as follows.

Original: [Equation]
Revised: [Equation]

Original: For each model h ∈ H , given its ID validation accuracy, one could simply plug the estimated slope ˆ aand bias ˆ b from (4), and (cid:100)
Revised: For each model h ∈ H , given its ID validation accuracy, one could simply plug the estimated slope ˆ aand bias ˆ b from (4), and (cid:100)

Original: Acc ID ( h ) (sample average over validation set) into (3) to get an estimate ofthe model’s
Revised: Acc ID ( h ) (sample average over validation set) into (3) to get an estimate ofthe model’s

Original: OOD accuracy.
Revised: OOD accuracy.

Original: We call this simple algorithm ALine-S.
Revised: We call this simple algorithm ALine-S.

Original: Notice that ALine-S does not directly use the OOD agreement estimates concerning the model ofinterest—we only use agreements indirectly via the estimates ˆ a and ˆ b .
Revised: Notice that ALine-S does not directly use the OOD agreement estimates concerning the model ofinterest—we only use agreements indirectly via the estimates ˆ a and ˆ b .

Original: We find that a better estimatorcan be obtained by directly using the model’s
Revised: We find that a better estimatorcan be obtained by directly using the model’s

Original: OOD agreement estimates via simple algebra as follows.
Revised: OOD agreement estimates via simple algebra as follows.

Original: First, note that for any pair of models h, h ′ ∈ H , it directly follows from (3)
Revised: First, note that for any pair of models h, h ′ ∈ H , it directly follows from (3)

Original: that
Revised: that

Original: [Equation]
Revised: [Equation]

Original: By substituting b = Φ − 1 ( Agr OOD ( h, h ′ )) − a · Φ − 1 ( Agr ID ( h, h ′ )) into (5), we can get that average
Revised: By substituting b = Φ − 1 ( Agr OOD ( h, h ′ )) − a · Φ − 1 ( Agr ID ( h, h ′ )) into (5), we can get that average

Original: OOD accuracy of any pair of models h, h ′ ∈ H is
Revised: OOD accuracy of any pair of models h, h ′ ∈ H is

Original: [Equation]
Revised: [Equation]

Original: We can plug in estimates of the terms on the right hand side ( ˆ a from linear regression (4)) and therest from sample averages.
Revised: We can plug in estimates of the terms on the right hand side ( ˆ a from linear regression (4)) and therest from sample averages.

Original: In this way, we can construct a system of linear equations of the form (6)involving “unknown” estimates of the OOD accuracy of models and other “known” quantities whichwe solve via linear regression to obtain the unknown estimates.
Revised: In this way, we can construct a system of linear equations of the form (6)involving “unknown” estimates of the OOD accuracy of models and other “known” quantities whichwe solve via linear regression to obtain the unknown estimates.

Original: We call this procedure ALine-D, andit is described more explicitly in Algorithm 1.
Revised: We call this procedure ALine-D, andit is described more explicitly in Algorithm 1.

Original: Note that there must be at least 3 models in the set ofinterest H for the system of linear equations in (6) to have a unique solution.
Revised: Note that there must be at least 3 models in the set ofinterest H for the system of linear equations in (6) to have a unique solution.

Original:  Experiments
Revised:  Experiments

Original: Datasets and models.
Revised: Datasets and models.

Original: We evaluate our methods, the simple plug in of slope/bias estimate ALine-Sand the more involved ALine-D, on the same models and datasets from Section 3 and two additionaldatasets CIFAR10C-Saturate and RxRx1- WILDS (See Appendix B and C for details on these datasets).
Revised: We evaluate our methods, the simple plug in of slope/bias estimate ALine-Sand the more involved ALine-D, on the same models and datasets from Section 3 and two additionaldatasets CIFAR10C-Saturate and RxRx1- WILDS (See Appendix B and C for details on these datasets).

Original: Specifically, we look at CIFAR-10.1, CIFAR-10.2, ImageNetV2, CIFAR-10C, fMoW- WILDS , and
Revised: Specifically, we look at CIFAR-10.1, CIFAR-10.2, ImageNetV2, CIFAR-10C, fMoW- WILDS , and

Original: RxRx1- WILDS , where we observe a strong correlation.
Revised: RxRx1- WILDS , where we observe a strong correlation.

Original: We also look at the performance of modelson datasets where we do not see a strong linear correlation, specifically Camelyon- WILDS andiWildCam- WILDS .
Revised: We also look at the performance of modelson datasets where we do not see a strong linear correlation, specifically Camelyon- WILDS andiWildCam- WILDS .

Original: Baseline methods.
Revised: Baseline methods.

Original: We choose 4 existing unlabeled estimation methods for comparison: Average
Revised: We choose 4 existing unlabeled estimation methods for comparison: Average

Original: Threshold Confidence (ATC) by Garg et al.
Revised: Threshold Confidence (ATC) by Garg et al.

Original: [29], DOC-Feat in Guillory et al.
Revised: [29], DOC-Feat in Guillory et al.

Original: [30], Average Confidence(AC) in [36], and naive Agreement [54, 58, 41].
Revised: [30], Average Confidence(AC) in [36], and naive Agreement [54, 58, 41].

Original: We implement the version of ATC that performedbest in the paper, i.e. with negative entropy as the score function and temperature scaling to calibratethe models in-distribution.
Revised: We implement the version of ATC that performedbest in the paper, i.e. with negative entropy as the score function and temperature scaling to calibratethe models in-distribution.

Original: Although DOC was deemed the best method in Guillory et al.
Revised: Although DOC was deemed the best method in Guillory et al.

Original: [30], weuse DOC-Feat since DOC requires information from multiple OOD datasets.
Revised: [30], weuse DOC-Feat since DOC requires information from multiple OOD datasets.

Original: For ATC, DOC, and
Revised: For ATC, DOC, and

Original: AC, consistent with the experimental design in Garg et al.
Revised: AC, consistent with the experimental design in Garg et al.

Original: [28], we report the best number achievedbetween before versus after temperature scaling.
Revised: [28], we report the best number achievedbetween before versus after temperature scaling.

Original: We also compare with the more recent, ProjNorm
Revised: We also compare with the more recent, ProjNorm

Original: 1.0 Prediction 0.
Revised: 1.0 Prediction 0.

Original: OO D
Revised: OO D

Original: A cc u r a c y
Revised: A cc u r a c y

Original: CIFAR-10.
Revised: CIFAR-10.

Original: y =
Revised: y =

Original: x ATC
Revised: x ATC

Original: ALine-D ATC
Revised: ALine-D ATC

Original: ALine-D 0.
Revised: ALine-D 0.

Original: 1.0 Prediction 0.
Revised: 1.0 Prediction 0.

Original: OO D
Revised: OO D

Original: A cc u r a c y
Revised: A cc u r a c y

Original: CIFAR-10-C fog y =
Revised: CIFAR-10-C fog y =

Original: x ATC
Revised: x ATC

Original: ALine-D ATC
Revised: ALine-D ATC

Original: ALine-D 0.
Revised: ALine-D 0.

Original: 1.0 Prediction 0.
Revised: 1.0 Prediction 0.

Original: OO D
Revised: OO D

Original: A cc u r a c y
Revised: A cc u r a c y

Original: Camelyon17-WILDS y =
Revised: Camelyon17-WILDS y =

Original: x ATC
Revised: x ATC

Original: ALine-D ATC
Revised: ALine-D ATC

Original: ALine-D by Yu et al.
Revised: ALine-D by Yu et al.

Original: [81] which showed stronger linear correlation with OOD accuracy than Rotation [23] and
Revised: [81] which showed stronger linear correlation with OOD accuracy than Rotation [23] and

Original: ATC [29].
Revised: ATC [29].

Original: We compare with this method separately in Section 5.1, as they do not provide a way todirectly estimate the OOD accuracy.
Revised: We compare with this method separately in Section 5.1, as they do not provide a way todirectly estimate the OOD accuracy.

Original: 5.1 Main results: comparison to other methods.
Revised: 5.1 Main results: comparison to other methods.

Original: In Table 2, we observe that ALine-D generally outperforms other methods on datasets whereagreement-on-the-line holds.
Revised: In Table 2, we observe that ALine-D generally outperforms other methods on datasets whereagreement-on-the-line holds.

Original: On ImageNet to ImageNetV2 and CIFAR-10 to CIFAR-10C-Snow,
Revised: On ImageNet to ImageNetV2 and CIFAR-10 to CIFAR-10C-Snow,

Original: ATC performs marginally better.
Revised: ATC performs marginally better.

Original: As can be seen in Figure 3, ATC generally cannot accurately predictthe model’s
Revised: As can be seen in Figure 3, ATC generally cannot accurately predictthe model’s

Original: OOD performance for models that do not perform very well.
Revised: OOD performance for models that do not perform very well.

Original: This is consistent withexperimental results in [29] and [81].
Revised: This is consistent withexperimental results in [29] and [81].

Original: On the other hand, ALine performed equally well on “bad”models land “good” models.
Revised: On the other hand, ALine performed equally well on “bad”models land “good” models.

Original: In some sense, given a collection of models where we are interestedin the performance of each, ATC, AC, DOC-Feat, and Agreement only utilize information fromthe model of interest, whereas ALine utilizes the collective information from all models for eachindividual prediction.
Revised: In some sense, given a collection of models where we are interestedin the performance of each, ATC, AC, DOC-Feat, and Agreement only utilize information fromthe model of interest, whereas ALine utilizes the collective information from all models for eachindividual prediction.

Original: As expected, on datasets where we did not observe a linear correlation between ID and OODagreement (and accuracy), ALine did not perform very well, with a mean absolute estimation errorof around 5% .
Revised: As expected, on datasets where we did not observe a linear correlation between ID and OODagreement (and accuracy), ALine did not perform very well, with a mean absolute estimation errorof around 5% .

Original: Interestingly, the other methods also did not perform very well on these datasets.
Revised: Interestingly, the other methods also did not perform very well on these datasets.

Original: No method successfully predicts the OOD accuracy for every distribution shift.
Revised: No method successfully predicts the OOD accuracy for every distribution shift.

Original: The advantage of
Revised: The advantage of

Original: ALine is that there is a concrete way to verify when the method will successfully predict the OODaccuracy (i.e. check whether agreement is on the line).
Revised: ALine is that there is a concrete way to verify when the method will successfully predict the OODaccuracy (i.e. check whether agreement is on the line).

Original: Other prediction methods do not have anyway of characterizing when they will be successful.
Revised: Other prediction methods do not have anyway of characterizing when they will be successful.

Original: Finally, we note that ALine-D actually surpassesprevious methods even when accuracy-on-the-line does not hold, suggesting that the algorithm hasadditional beneficial properties that require further study.
Revised: Finally, we note that ALine-D actually surpassesprevious methods even when accuracy-on-the-line does not hold, suggesting that the algorithm hasadditional beneficial properties that require further study.

Original: 5.2 Correlation analysis
Revised: 5.2 Correlation analysis

Original: [Table]
Revised: [Table]

Original: Dataset

Original: [Table]
Revised: [Table]

Original: Rather than predicting OOD accuracy, it couldbe useful to have a metric that just strongly correlates with the OOD accuracy, if the applicationsimply requires an understanding of relative performance such as model selection.
Revised: Rather than predicting OOD accuracy, it couldbe useful to have a metric that just strongly correlates with the OOD accuracy, if the applicationsimply requires an understanding of relative performance such as model selection.

Original: Recently, Yuet al.
Revised: Recently, Yuet al.

Original: [81] proposed ProjNorm, a measurementthey show has a very strong linear correlationwith OOD accuracy, moreso than other recentmethods including Rotation [23] and ATC [29].
Revised: [81] proposed ProjNorm, a measurementthey show has a very strong linear correlationwith OOD accuracy, moreso than other recentmethods including Rotation [23] and ATC [29].

Original: To compare Aline-D with Projnorm, we replicate the CIFAR10C study in Yu et al.
Revised: To compare Aline-D with Projnorm, we replicate the CIFAR10C study in Yu et al.

Original: [81], where theytrain a ResNet18 model and predict its performance across all corruptions and severity levels of
Revised: [81], where theytrain a ResNet18 model and predict its performance across all corruptions and severity levels of

Original: CIFAR-10C (See their Table 1 in [81]).
Revised: CIFAR-10C (See their Table 1 in [81]).

Original: Since ALine-D is an algorithm that requires a set of modelsfor prediction, we use 29 pretrained models from the CIFAR-10 testbed of Miller et al.
Revised: Since ALine-D is an algorithm that requires a set of modelsfor prediction, we use 29 pretrained models from the CIFAR-10 testbed of Miller et al.

Original: [56], as theother models in the set.
Revised: [56], as theother models in the set.

Original: We look at the linear correlation of the estimates of OOD accuracy andthe true accuracy across all corruptions and find that ALine-D achieves stronger correlation than
Revised: We look at the linear correlation of the estimates of OOD accuracy andthe true accuracy across all corruptions and find that ALine-D achieves stronger correlation than

Original: ProjNorm (Table 3).
Revised: ProjNorm (Table 3).

Original: See Appendix D for more experimental details.
Revised: See Appendix D for more experimental details.

Original: 5.3 Estimating performance along a training trajectory
Revised: 5.3 Estimating performance along a training trajectory

Original: [Figure]
Revised: [Figure]

Original: So far in our experiments, ALine uses a large collection of models to get the linear fit and predict thetrue slope and bias of ID vs OOD accuracy.

Original: Weassess whether ALine can be utilized even in situations where the practioner only cares about the performance of a few models.
Revised: We assess whether ALine can be utilized even in situations where the practioner only cares about theperformance of a few models.

Original: In such situations, onecould efficiently gather many models by training asingle model and saving checkpoints along the way.
Revised: In such situations, one could efficiently gather many models by traininga single model and saving checkpoints along the way.

Original: We analyze whether our phenomenon is helpful forpredicting such highly correlated hypotheses, insteadof independently trained models.
Revised: We analyze whether our phenomenon is helpfulfor predicting such highly correlated hypotheses, instead of independently trained models.

Original: In Figure 4, wecollect the logits of the ID validation set and OODdataset every 5 epochs across the training of a single ResNet18 modeltrained on CIFAR-10 andcomputethe agreement between every pair.
Revised: In Figure4, we collect the logits of the ID validation set and OOD dataset every 5 epochs across CIFAR-10training of a single ResNet18 model and every epoch across ImageNEt training of a ResNet50 model.

Original: We see that eventhe agreement between the checkpoints of a model across training is enough to get a good linear fitthat matches the slope and bias of CIFAR-10 vs CIFAR-10.1 accuracy.
Revised: We see that even the agreement between every pair of checkpoints of a model across training isenough to get a good linear fit that matches the slope and bias of ID vs OOD accuracy.

Original: Thus, by applying ALine-Dto these checkpoints, we can get a very good estimate of the OOD performance of the model acrosstraining epochs.
Revised: Thus, byapplying ALine-D to these checkpoints, we can get a very good estimate of the OOD performanceof the model across training epochs.

Original: This suggests that given a model of interest, ALine does not require practitioners totrain a large number of models, but just train one and save its predictions across training iterations.
Revised: This suggests that given a model of interest, ALine does notrequire practitioners to train a large number of models, but just train one and save its predictionsacross training iterations.

Original: We do a more careful ablation study in Appendix F, looking at the number of models required forclose estimates of accuracy.
Revised: We do a more careful ablation study in Appendix F, looking at the numberof models required for close estimates of accuracy.

Original:  Conclusion
Revised:  Conclusion

Original: The contributions of this work are two-fold.
Revised: The contributions of this work are two-fold.

Original: First, we observe the agreement-on-the-line phenomenon,and show that it correlates strongly with accuracy-on-the-line over a range of datasets and models.
Revised: First, we observe the agreement-on-the-line phenomenon,and show that it correlates strongly with accuracy-on-the-line over a range of datasets and models.

Original: Wealso highlight that certain aspects of this phenomenon, namely the fact that the slope and bias of thelinear fit is largely the same across agreement and accuracy, are specific to neural networks, and thusfundamentally seem connected to these classes of models.
Revised: Wealso highlight that certain aspects of this phenomenon, namely the fact that the slope and bias of thelinear fit is largely the same across agreement and accuracy, are specific to neural networks, and thusfundamentally seem connected to these classes of models.

Original: Second, using this empirical phenomenon,we propose a surprisingly simple but effective method for predicting OOD accuracy of classifiers,while only having access to unlabeled data from the new domain (and one that can be “sanity checked”via testing whether agreement-on-the line holds).
Revised: Second, using this empirical phenomenon,we propose a surprisingly simple but effective method for predicting OOD accuracy of classifiers,while only having access to unlabeled data from the new domain (and one that can be “sanity checked”via testing whether agreement-on-the line holds).

Original: Our method outperforms existing state-of-the-artapproaches to this problem.
Revised: Our method outperforms existing state-of-the-artapproaches to this problem.

Original: Importantly, we do not claim that this phenomenon is universal, butwe found it to be true across an extensive range of neural networks and OOD benchmarks that weexperimented on.
Revised: Importantly, we do not claim that this phenomenon is universal, butwe found it to be true across an extensive range of neural networks and OOD benchmarks that weexperimented on.

Original: In addition to its practical relevance, this observation itself reveals something veryinteresting about the way neural networks learn, which we leave for future study.
Revised: In addition to its practical relevance, this observation itself reveals something veryinteresting about the way neural networks learn, which we leave for future study.

Original: A Experiments from Miller et al.
Revised: A Experiments from Miller et al.

Original: To get a better understanding of the agreement-on-the-line phenomenon, we replicate a theoreticalexperiments in Miller et al.
Revised: To get a better understanding of the agreement-on-the-line phenomenon, we replicate a theoreticalexperiments in Miller et al.

Original: [56] using the same set of neural networks from their testbed.
Revised: [56] using the same set of neural networks from their testbed.

Original: Specifically,we look at CIFAR-10 with different added Gaussian noise (their Figure 4).
Revised: Specifically,we look at CIFAR-10 with different added Gaussian noise (their Figure 4).

Original: A.1 Matching gaussian noise
Revised: A.1 Matching gaussian noise

Original: Miller et al.
Revised: Miller et al.

Original: [56] conduct a theoretical analysis on a toy gaussian mixture model to better understandthe accuracy-on-the-line phenomena.
Revised: [56] conduct a theoretical analysis on a toy gaussian mixture model to better understandthe accuracy-on-the-line phenomena.

Original: From their analysis, they predict that accuracy-on-the-lineoccurs if the covariances of the ID and OOD data are the same up to some constant scaling factor.
Revised: From their analysis, they predict that accuracy-on-the-lineoccurs if the covariances of the ID and OOD data are the same up to some constant scaling factor.

Original: Inspired by this, they show that accuracy-on-the-line holds stronger on CIFAR-10 data corrupted withgaussian noise that matches the covariance of CIFAR-10 test data versus isotropic gaussian noise.
Revised: Inspired by this, they show that accuracy-on-the-line holds stronger on CIFAR-10 data corrupted withgaussian noise that matches the covariance of CIFAR-10 test data versus isotropic gaussian noise.

Original: Interestingly, even for this simple setting, we similarly observe that the ID vs OOD agreement trendis stronger on covariance matched gaussian noise.
Revised: Interestingly, even for this simple setting, we similarly observe that the ID vs OOD agreement trendis stronger on covariance matched gaussian noise.

Original: B Correlation results on more datasets
Revised: B Correlation results on more datasets

Original: To verify the agreement-on-the-line phenomenon, we compare the linear correlation between ID vs
Revised: To verify the agreement-on-the-line phenomenon, we compare the linear correlation between ID vs

Original: OOD accuracy and agreement across a variety of distribution shifts.
Revised: OOD accuracy and agreement across a variety of distribution shifts.

Original: Given a set of n models, we plotthe ID vs OOD accuracy of each model in the set.
Revised: Given a set of n models, we plotthe ID vs OOD accuracy of each model in the set.

Original: For agreement, we randomly pair each model withanother model in the set, and plot the ID vs OOD agreement of these n pairs.
Revised: For agreement, we randomly pair each model withanother model in the set, and plot the ID vs OOD agreement of these n pairs.

Original: We provide R 2 valuesin the figure legends.
Revised: We provide R 2 valuesin the figure legends.

Original: Datasets In addition to the 8 datasets from the main body, we also observe the trend on other
Revised: Datasets In addition to the 8 datasets from the main body, we also observe the trend on other

Original: CIFAR10C corruptions [35].
Revised: CIFAR10C corruptions [35].

Original: For each corruption, we evaluate the models over data from all 5 severitylevels (both in Figure 1 of main body and appendix).
Revised: For each corruption, we evaluate the models over data from all 5 severitylevels (both in Figure 1 of main body and appendix).

Original: We also look at shifting from CIFAR-10 to CINIC-10 [19] and CIFAR-10 to STL-10 [16] whichare shifts from changes in the image source.
Revised: We also look at shifting from CIFAR-10 to CINIC-10 [19] and CIFAR-10 to STL-10 [16] whichare shifts from changes in the image source.

Original: Specifically, the CINIC-10 test dataset contains both
Revised: Specifically, the CINIC-10 test dataset contains both

Original: CIFAR-10 Test data and a selection of ImageNet images for CIFAR-10 class labels downsampled to32 × 32 .
Revised: CIFAR-10 Test data and a selection of ImageNet images for CIFAR-10 class labels downsampled to32 × 32 .

Original: We only consider the downsampled ImageNet data as the OOD dataset.
Revised: We only consider the downsampled ImageNet data as the OOD dataset.

Original: Similarly, STL-10contains processed ImageNet images for CIFAR-10 class labels.
Revised: Similarly, STL-10contains processed ImageNet images for CIFAR-10 class labels.

Original: Since STL-10 is an unsupervisedlearning dataset, we only utilize the labeled subset of STL-10 as the OOD dataset.
Revised: Since STL-10 is an unsupervisedlearning dataset, we only utilize the labeled subset of STL-10 as the OOD dataset.

Original: Additionally,
Revised: Additionally,

Original: STL-10 only contains 9 of the 10 CIFAR-10 classes, so we restrict the dataset to just those 9 classes.
Revised: STL-10 only contains 9 of the 10 CIFAR-10 classes, so we restrict the dataset to just those 9 classes.

Original: Finally, we add results for a real-world shift from batch effects in images of cells in RxRx1- WILDS .
Revised: Finally, we add results for a real-world shift from batch effects in images of cells in RxRx1- WILDS .

Original: Models For ImageNetV2, we evaluate 49 ImageNet pretrained models from the timm [77] package.
Revised: Models For ImageNetV2, we evaluate 49 ImageNet pretrained models from the timm [77] package.

Original: See their repository for more details about the models.
Revised: See their repository for more details about the models.

Original: For RxRx1- WILDS we trained 36models of varying architecture and hyperparameters.
Revised: For RxRx1- WILDS we trained 36models of varying architecture and hyperparameters.

Original: Specifically, we vary weight decay between[10 − 1 , 10 − 2 , 10 − 3 , 10 − 4 , 10 − 5 ] and optimizers between SGD, Adam, and AdamW. See C for architecture details.
Revised: Specifically, we vary weight decay between[10 − 1 , 10 − 2 , 10 − 3 , 10 − 4 , 10 − 5 ] and optimizers between SGD, Adam, and AdamW. See C for architecture details.

Original: Finally, we utilize all independently trained models from the CIFAR10, iWildCam- WILDS , fMoWWILDS , and Camelyon17- WILDS testbeds created and utilized by [56].
Revised: Finally, we utilize all independently trained models from the CIFAR10, iWildCam- WILDS , fMoWWILDS , and Camelyon17- WILDS testbeds created and utilized by [56].

Original: The hyperparameters used totrain these models are explained in great detail in Appendix B.2 of Miller et al.
Revised: The hyperparameters used totrain these models are explained in great detail in Appendix B.2 of Miller et al.

Original: The architecturesof all models evaluated for the experiment are described in more detail in Appendix C.
Revised: The architecturesof all models evaluated for the experiment are described in more detail in Appendix C.

Original: Pretrained vs Not Pretrained Miller et al.
Revised: Pretrained vs Not Pretrained Miller et al.

Original: [56] showed that for some shifts, the ID vs OODaccuracy of ImageNet pretrained models follow a different linear trend than models trained fromscratch in-distribution.
Revised: [56] showed that for some shifts, the ID vs OODaccuracy of ImageNet pretrained models follow a different linear trend than models trained fromscratch in-distribution.

Original: In Figure 1 of the main body, we do not distinguish between pretrained andfrom scratch models as the trends for pretrained and from scratch models were the same for the 8datasets we chose.
Revised: In Figure 1 of the main body, we do not distinguish between pretrained andfrom scratch models as the trends for pretrained and from scratch models were the same for the 8datasets we chose.

Original: Below, we divide results between pretrained and not pretrained models to be moreprecise.
Revised: Below, we divide results between pretrained and not pretrained models to be moreprecise.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: (j) CIFAR10C Gaussian Noise
Revised: (j) CIFAR10C Gaussian Noise

Original: (m) CIFAR10C JPEG Compres- sion
Revised: (m) CIFAR10C JPEG Compres- sion

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: (j) CIFAR10C Gaussian Noise
Revised: j) CIFAR10C Gaussian Noise

Original: m)
Revised: m)

Original: CIFAR10C JPEG Compres- sion
Revised: CIFAR10C JPEG Compres- sion

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: B.3 Only Neural Networks
Revised: B.3 Only Neural Networks

Revised: [Figure]

Original: Here we observe the ID vs OOD accuracy and agreement trend of model families other than neuralnetworks.
Revised: Here we observe the ID vs OOD accuracy and agreement trend of model families other than neuralnetworks.

Original: In the main body of the paper, we illustrated how the agreement-on-the-line phenomenonis specific to neural networks on CIFAR10-Fog, a synthetic shift.
Revised: In the main body of the paper, we illustrated how the agreement-on-the-line phenomenonis specific to neural networks on CIFAR10-Fog, a synthetic shift.

Original: Below, we illustrate this furtheron a data replication shift, CIFAR10.
Revised: Below, we illustrate this furtheron a data replication shift, CIFAR10.

Original: [Figure]
Revised: [Figure]

Original: and a real-world shift, fMoW- WILDS .
Revised: and a real-world shift, fMoW- WILDS .

Original: Note that the slope ofthe ID vs OOD agreement trend of neural network models is closest to the slope of the ID vs OODaccuracy trend.
Revised: Note that the slope ofthe ID vs OOD agreement trend of neural network models is closest to the slope of the ID vs OODaccuracy trend.

Original: Excluding neural networks, we observe that the agreement trend of random featuremodels [15] also has a similar slope to that of the accuracy trend for select shifts such as CIFAR-10.2.
Revised: Excluding neural networks, we observe that the agreement trend of random featuremodels [15] also has a similar slope to that of the accuracy trend for select shifts such as CIFAR-10.2.

Original: [Figure]
Revised: [Figure]

Original: C Model Architectures
Revised: C Model Architectures

Original: We list the types of architectures and the corresponding number of models for each testbed.
Revised: We list the types of architectures and the corresponding number of models for each testbed.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ConViT [20]
Revised: ConViT [20]

Original: ConvNeXT [50]
Revised: ConvNeXT [50]

Original: CrossViT [8]
Revised: CrossViT [8]

Original: DenseNet [37]
Revised: DenseNet [37]

Original: DLA [80]
Revised: DLA [80]

Original: EfficientNet [73]
Revised: EfficientNet [73]

Original: HaloNet [76]
Revised: HaloNet [76]

Original: NFNet [7]
Revised: NFNet [7]

Original: ResNet [33]
Revised: ResNet [33]

Original: ResNeXT [78]
Revised: ResNeXT [78]

Original: Inception v3 [72]
Revised: Inception v3 [72]

Original: VGG [68]
Revised: VGG [68]

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ResNeXT [78]
Revised: ResNeXT [78]

Original: [Table]
Revised: [Table]

Original: AlexNet [45]
Revised: AlexNet [45]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: DenseNet121 [37]
Revised: DenseNet121 [37]

Original: [Table]
Revised: [Table]

Original: GoogLeNet [71]
Revised: GoogLeNet [71]

Original: Xception [12]
Revised: Xception [12]

Original: [Table]
Revised: [Table]

Original: ShuffleNet [82]
Revised: ShuffleNet [82]

Original: [Table]
Revised: [Table]

Original: MobileNetV2 [66]
Revised: MobileNetV2 [66]

Original: VGG [68]
Revised: VGG [68]

Original: [Table]
Revised: [Table]

Original: PNASNet [48]
Revised: PNASNet [48]

Original: Squeezenet [39]
Revised: Squeezenet [39]

Original: [Table]
Revised: [Table]

Original: ViT [25]
Revised: ViT [25]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: D Section 5: Experimental DetailsD.1 Details for Experiment 5.2: Correlation analysis
Revised: D Section 5: Experimental DetailsD.1 Details for Experiment 5.2: Correlation analysis

Original: We replicate the correlation analysis experiment in Table 1 of Yu et al.
Revised: We replicate the correlation analysis experiment in Table 1 of Yu et al.

Original: [81] to compare the predictionperformance of ALine-D versus ProjNorm.
Revised: [81] to compare the predictionperformance of ALine-D versus ProjNorm.

Original: Essentially, we want to see how strong the linearcorrelation is between the estimate of OOD accuracy versus the true OOD accuracy by looking at thecoefficients of determination R 2 and rank correlations ρ of the fit.
Revised: Essentially, we want to see how strong the linearcorrelation is between the estimate of OOD accuracy versus the true OOD accuracy by looking at thecoefficients of determination R 2 and rank correlations ρ of the fit.

Original: We use the GitHub repository of
Revised: We use the GitHub repository of

Original: ProjNorm [81] found at https://github.com/yaodongyu/ProjNorm to replicate their correlationanalysis experiment found in their Table 1.
Revised: ProjNorm [81] found at https://github.com/yaodongyu/ProjNorm to replicate their correlationanalysis experiment found in their Table 1.

Original: Using their repository, we train a base ResNet18 modelfor 20 epochs with their default hyperparameters
Revised: Using their repository, we train a base ResNet18 modelfor 20 epochs with their default hyperparameters

Original: • Batch Size: • Learning Rate: 0.001• Weight Decay: 0• Optimizer: SGD with Momentum 0. • Pretrained:
Revised: • Batch Size: 128• Learning Rate: 0.001• Weight Decay: 0• Optimizer: SGD with Momentum 0.9• Pretrained:

Original: True
Revised: True

Original: using cosine learning rate decay [51].
Revised: using cosine learning rate decay [51].

Original: The repository uses the default implementation of ResNet18by torchvision.
Revised: The repository uses the default implementation of ResNet18by torchvision.

Original: The CIFAR-10 images are resized to be 224 × 224 , then normalized.
Revised: The CIFAR-10 images are resized to be 224 × 224 , then normalized.

Original: To compute
Revised: To compute

Original: ProjNorm, we use the repository to train a reference ResNet18 model using the pseudolabels of thebase model for 500 iterations with the same hyperparameters as the base model.
Revised: ProjNorm, we use the repository to train a reference ResNet18 model using the pseudolabels of thebase model for 500 iterations with the same hyperparameters as the base model.

Original: We use the ALine-Dalgorithm to predict the OOD accuracy of the base model.
Revised: We use the ALine-Dalgorithm to predict the OOD accuracy of the base model.

Original: ALine-D requires a model set H withat least 3 models so that the linear system of equations (Line 9 in Algorithm 1, Equation 6 in mainbody) has a unique solution.
Revised: ALine-D requires a model set H withat least 3 models so that the linear system of equations (Line 9 in Algorithm 1, Equation 6 in mainbody) has a unique solution.

Original: We use the 29 pretrained models from the CIFAR10 testbed as the othermodels in the model set.
Revised: We use the 29 pretrained models from the CIFAR10 testbed as the othermodels in the model set.

Original: D.2 Details for Experiment 5.3: Performance along a training trajectory
Revised: D.2 Details for Experiment 5.3: Performance along a training trajectory

Original: We train a ResNet18 model from scratch (no pretrained weights) on CIFAR10 for 150 epochs usingthe following hyperparameters:
Revised: We train a ResNet18 model from scratch (no pretrained weights) on CIFAR10 for 150 epochs usingthe following hyperparameters:

Original: • Batch Size:
Revised: • Batch Size:

Original: • Learning Rate: 0.1• Weight Decay:
Revised: • Learning Rate: 0.1• Weight Decay:

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: • Optimizer: SGD with Momentum 0.9• Pretrained:
Revised: • Optimizer: SGD with Momentum 0. [Table] • Pretrained:

Original: False
Revised: False

Original: [Table]
Revised: [Table]

Original: We decay the learning rate by 0.1 at epoch 75 and epoch 113.
Revised: We decay the learning rate by 0.1 at epoch 75 and epoch 113.

Original: We use simple data augmentationconsisting of RandomCrop with padding = 4 , RandomHorizontalFlip, and Normalization.
Revised: We use simple data augmentationconsisting of RandomCrop with padding = 4 , RandomHorizontalFlip, and Normalization.

Original: We modify
Revised: We modify

Original: ResNet18 to take in 32 × 32 input images.
Revised: ResNet18 to take in 32 × 32 input images.

Original: During training, we save the model weights every 5 training epochs.
Revised: During training, we save the model weights every 5 training epochs.

Original: Given this collection of models,we estimate the CIFAR-10.1 accuracy of each one of these models using the ALine-D procedure.
Revised: Given this collection of models,we estimate the CIFAR-10.1 accuracy of each one of these models using the ALine-D procedure.

Original: D.3 Hardware
Revised: D.3 Hardware

Original: All experiments were conducted using GeForce GTX 1080 and 2080 graphics cards.
Revised: All experiments were conducted using GeForce GTX 1080 and 2080 graphics cards.

Original: E The relationship between agreement and accuracy
Revised: E The relationship between agreement and accuracy

Original: This work is related to Jiang et al.
Revised: This work is related to Jiang et al.

Original: [41], which shows that the agreement between two models ofthe same architecture trained with different random seeds is approximately equal to the average IDtest accuracy of models if the ensemble consisting of the models is well-calibrated .
Revised: [41], which shows that the agreement between two models ofthe same architecture trained with different random seeds is approximately equal to the average IDtest accuracy of models if the ensemble consisting of the models is well-calibrated .

Original: They call thisequality between accuracy and agreement Generalization Disagreement Equality (GDE).
Revised: They call thisequality between accuracy and agreement Generalization Disagreement Equality (GDE).

Original: Let us ignore the probit transform for a moment and assume the linear correlation between ID vs
Revised: Let us ignore the probit transform for a moment and assume the linear correlation between ID vs

Original: OOD accuracy and agreement are strong without it.
Revised: OOD accuracy and agreement are strong without it.

Original: In this simplified scenario, agreement-on-the-lineimplies that, for datasets where both agreement and accuracy are strongly linearly correlated, if IDagreement of a pair of models is equal to their average ID accuracy, then their OOD agreementis equal to their OOD accuracy.
Revised: In this simplified scenario, agreement-on-the-lineimplies that, for datasets where both agreement and accuracy are strongly linearly correlated, if IDagreement of a pair of models is equal to their average ID accuracy, then their OOD agreementis equal to their OOD accuracy.

Original: Formally, when a shift satisfies accuracy-on-the-line, we knowby agreement-on-the-line that for any two models trained on ID samples h, h ′ ∈ H , the followingequations are approximately satisfied (ignoring probit transform)
Revised: Formally, when a shift satisfies accuracy-on-the-line, we knowby agreement-on-the-line that for any two models trained on ID samples h, h ′ ∈ H , the followingequations are approximately satisfied (ignoring probit transform)

Original: [Equation]
Revised: [Equation]

Original: ) and Agr OOD
Revised: 7) and Agr OOD

Original: [Equation]
Revised: [Equation]

Original: Acc
Revised: Acc

Original: [Equation]
Revised: [Equation]

Original: for some slope a and intercept b .
Revised: for some slope a and intercept b .

Original: Thus, if the “ID gap” between accuracy and agreement is 0, then the“OOD gap” is also 0.
Revised: Thus, if the “ID gap” between accuracy and agreement is 0, then the“OOD gap” is also 0.

Original: This may suggest something about calibration on shifts where accuracy-on-theline holds: if the ensemble of a pair of models is well calibrated ID, then by agreement-on-the-line
Revised: This may suggest something about calibration on shifts where accuracy-on-theline holds: if the ensemble of a pair of models is well calibrated ID, then by agreement-on-the-line

Original: GDE also holds OOD.
Revised: GDE also holds OOD.

Original: However, agreement-on-the-line goes beyond these results in two ways: (i) agreement betweenmodels with different architectures and (ii) agreement between different checkpoints on the sametraining run is also on the ID vs OOD agreement line.
Revised: However, agreement-on-the-line goes beyond these results in two ways: (i) agreement betweenmodels with different architectures and (ii) agreement between different checkpoints on the sametraining run is also on the ID vs OOD agreement line.

Original: Jiang et al.
Revised: Jiang et al.

Original: [41] does not guarantee GDEholds for these cases.
Revised: [41] does not guarantee GDEholds for these cases.

Original: As can be seen in Figure 9, for most pairs of models, the ID and OOD gapsbetween accuracy and agreement are not equal to 0 i.e. GDE does not occur ID or OOD.
Revised: As can be seen in Figure 9, for most pairs of models, the ID and OOD gapsbetween accuracy and agreement are not equal to 0 i.e. GDE does not occur ID or OOD.

Original: Indeed,understanding why agreement-on-the-line holds requires going beyond the theoretical conditionspresented in the prior work [41] which do not hold for this expanded set of models.
Revised: Indeed,understanding why agreement-on-the-line holds requires going beyond the theoretical conditionspresented in the prior work [41] which do not hold for this expanded set of models.

Original: [Figure]
Revised: [Figure]

Original: Is agreement-on-the-line a result of neural network performing well?
Revised: Is agreement-on-the-line a result of neural network performing well?

Original: In Figures 10 we look atthe trend between ID agreement vs accuracy to observe whether the accuracy of the models in thecollection plays a role in the success of ALine-D. Note that naturally, the gap between agreement andaverage accuracy is smaller for pairs of models that perform well.
Revised: In Figures 10 we look atthe trend between ID agreement vs accuracy to observe whether the accuracy of the models in thecollection plays a role in the success of ALine-D. Note that naturally, the gap between agreement andaverage accuracy is smaller for pairs of models that perform well.

Original: However, this does not necessarilycorrespond with better ALine-D performance on such models.
Revised: However, this does not necessarilycorrespond with better ALine-D performance on such models.

Original: On the right figure of 10, we plot theright hand side of ALine-D’s Equation 6 versus the left hand side ("adjusted agreement"), and observethat all pairs lie close to the diagonal, regardless of how large the gap Acc OOD − Agr OOD is before the adjustment.
Revised: On the right figure of 10, we plot theright hand side of ALine-D’s Equation 6 versus the left hand side ("adjusted agreement"), and observethat all pairs lie close to the diagonal, regardless of how large the gap Acc OOD − Agr OOD is before the adjustment.

Original: This is to say that, all pairs of varying performance have strong linear correlationbetween ID vs OOD agreement, which we show in Figure 11.
Revised: This is to say that, all pairs of varying performance have strong linear correlationbetween ID vs OOD agreement, which we show in Figure 11.

Original: F AblationStudy
Revised: F Ablation

Revised: Study

Original: F.1 How manymodels are
Revised: F.1 How many

Original: necessary

Original: ALine -D to make accurate predictions
Revised: models are necessary for ALine-D to make accurate predictions

Original: [Figure]
Revised: [Figure]

Original: ALine-D is an algorithm that requires at least 3 models in the model set so that the system of linearequations (Line 9 in Algorithm 1) has a unique solution.
Revised: ALine-D is an algorithm that requires at least 3 models in the model set so that the system of linearequations (Line 9 in Algorithm 1) has a unique solution.

Original: Additionally, it may generally requiremore models for the slope and intercept of the agreement trend to match the slope and intercept ofthe accuracy trend.
Revised: Additionally, it may generally requiremore models for the slope and intercept of the agreement trend to match the slope and intercept ofthe accuracy trend.

Original: We observe the MAE (in % ) of the ALine-D estimates of the OOD accuracyfor model sets of varying sizes.
Revised: We observe the MAE (in % ) of the ALine-D estimates of the OOD accuracyfor model sets of varying sizes.

Original: For each distribution shift, we randomly sample n models fromthe testbed to be our model set.
Revised: For each distribution shift, we randomly sample n models fromthe testbed to be our model set.

Original: n ranges from 3 to 463 in increments of 10 for CIFAR-10 relatedshifts, and 3 to 48 in increments of 5 for ImageNetV2.
Revised: n ranges from 3 to 463 in increments of 10 for CIFAR-10 relatedshifts, and 3 to 48 in increments of 5 for ImageNetV2.

Original: Our ablation study shows that the success of
Revised: Our ablation study shows that the success of

Original: ALine-D is not necessarily tied to the number of models.
Revised: ALine-D is not necessarily tied to the number of models.

Original: For CIFAR-10.1, we see a very quick dropin estimation error, and ALine-D performs well even for a small model set.
Revised: For CIFAR-10.1, we see a very quick dropin estimation error, and ALine-D performs well even for a small model set.

Original: Similarly, in ImageNetV2,we observe a decrease in estimation error as the number of models increases, however, the MAE isalready pretty low from the start ( 2 . 5% ).
Revised: Similarly, in ImageNetV2,we observe a decrease in estimation error as the number of models increases, however, the MAE isalready pretty low from the start ( 2 . 5% ).

Original: On the other hand, in CIFAR-10C Fog, the estimation errordoes not decrease, but the error is quite low (below 1 . 8% ) from the start.
Revised: On the other hand, in CIFAR-10C Fog, the estimation errordoes not decrease, but the error is quite low (below 1 . 8% ) from the start.

Original: This short ablation studyseems to indicate that ALine-D performs pretty well when agreement-on-the-line holds even for asmall number of models ( < 15 models).
Revised: This short ablation studyseems to indicate that ALine-D performs pretty well when agreement-on-the-line holds even for asmall number of models ( < 15 models).

Original: Additionally, it is not always the case that more models willdecrease the estimation error further.
Revised: Additionally, it is not always the case that more models willdecrease the estimation error further.

Original: F.2 Does a model set of varied architecture perform better than models of the samearchitecture?
Revised: F.2 Does a model set of varied architecture perform better than models of the samearchitecture?

Original: We study whether the diversity from varying the architecture of the models in the model set improvesthe performance of ALine-D. We look at the performance of ALine-D on CIFAR-10.1 and
Revised: We study whether the diversity from varying the architecture of the models in the model set improvesthe performance of ALine-D. We look at the performance of ALine-D on CIFAR-10.1 and

Original: CIFAR10C Fog over two model sets sampled from the CIFAR-10 testbed: (A) 20 PreActResNet18 [32] models, (B) 20 models of varying architecture.
Revised: CIFAR10C Fog over two model sets sampled from the CIFAR-10 testbed: (A) 20 PreActResNet18 [32] models, (B) 20 models of varying architecture.

Original: Similary for fMoW, we look at the performance over(A)
Revised: Similary for fMoW, we look at the performance over(A)

Original: 10 DenseNet121 [38] models , (B) 10 models of varying architecture.
Revised: 10 DenseNet121 [38] models , (B) 10 models of varying architecture.

Original: We randomly samplethese models from the corresponding testbeds, and average our results over 10 seeds.
Revised: We randomly samplethese models from the corresponding testbeds, and average our results over 10 seeds.

Original: Our results asshown below depends on the dataset.
Revised: Our results asshown below depends on the dataset.

Original: For CIFAR-10.1 and CIFAR-10C Fog, we see that ALine-Dperforms better on the diverse set consisting of many architecture types.
Revised: For CIFAR-10.1 and CIFAR-10C Fog, we see that ALine-Dperforms better on the diverse set consisting of many architecture types.

Original: On the other hand, for fMoW,
Revised: On the other hand, for fMoW,

Original: ALine-D performed better on the uniform set consisting of models of one architecture.
Revised: ALine-D performed better on the uniform set consisting of models of one architecture.

Original: [Figure]
Revised: [Figure]