Original: N O O NE R EPRESENTATION TO R ULE T HEM A LL : O VERLAPPING F EATURES OF T RAINING M ETHODS
Revised: N O O NE R EPRESENTATION TO R ULE T HEM A LL : O VERLAPPING F EATURES OF T RAINING M ETHODS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Despite being able to capture a range of features of the data, high accuracy models trained with supervision tend to make similar predictions.
Revised: Despite being able to capture a range of features of the data, high accuracy models trained with supervision tend to make similar predictions.

Original: This seemingly implies that high-performing models share similar biases regardless of training methodology, which would limit ensembling beneﬁts and render low-accuracy models as having little practical use.
Revised: This seemingly implies that high-performing models share similar biases regardless of training methodology, which would limit ensembling beneﬁts and render low-accuracy models as having little practical use.

Original: Against this backdrop, recent work has made very different training techniques, such as large-scale contrastive learning, yield competitively-high accuracy on generalization and robustness benchmarks.
Revised: Against this backdrop, recent work has developed quite different training techniques, such as large-scale contrastive learning, yielding competitively high accuracy on generalization and robustness benchmarks.

Original: This motivates us to revisit the assumption that models necessarily learn similar functions.
Revised: This motivates us to revisit the assumption that models necessarily learn similar functions.

Original: We conduct a large-scale empirical study of models across hyper-parameters, architectures, frameworks, and datasets.
Revised: We conduct a large-scale empirical study of models across hyper-parameters, architectures, frameworks, and datasets.

Original: We ﬁnd that model pairs that diverge more in training methodology display categorically different generalization behavior, producing increasingly uncorrelated errors.
Revised: We ﬁnd that model pairs that diverge more in training methodology display categorically different generalization behavior, producing increasingly uncorrelated errors.

Original: We show these models specialize in subdomains of the data, leading to higher ensemble performance: with just 2 models (each with ImageNet accuracy 76.5%), we can create ensembles with 83.4% (+7% boost).
Revised: We show these models specialize in subdomains of the data, leading to higher ensemble performance: with just 2 models (each with ImageNet accuracy ˜76.5%), we can create ensembles with 83.4% (+7% boost).

Original: Surprisingly, we ﬁnd that even signiﬁcantly low-accuracy models can be used to improve high-accuracy models.
Revised: Surprisingly, we ﬁnd that even signiﬁcantly low-accuracy models can be used to improve high-accuracy models.

Original: Finally, we show diverging training methodology yield representations that capture overlapping (but not supersetting) feature sets which, when combined, lead to increased downstream performance.
Revised: Finally, we show diverging training methodology yield representations that capture overlapping (but not supersetting) feature sets which, when combined, lead to increased downstream performance.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Over the years, the machine learning ﬁeld has developed myriad techniques for training neural networks.
Revised: Over the years, the machine learning ﬁeld has developed myriad techniques for training neural networks.

Original: In image classiﬁcation, these include data augmentation, regularization, architectures, losses, pre-training schemes, and more.
Revised: In image classiﬁcation, these include data augmentation, regularization, architectures, losses, pre-training schemes, and more.

Original: Such techniques have highlighted the ability of networks to capture diverse features of the data: textures/shapes (Geirhos et al., 2018), robust/non-robust features (Ilyas et al., 2019), and even features that ﬁt a random, pre-determined classiﬁer (Hoffer et al., 2018).
Revised: Such techniques have highlighted the ability of networks to capture diverse features of the data: textures/shapes (Geirhos et al., 2018), robust/non-robust features (Ilyas et al., 2019), and even features that ﬁt a random, pre-determined classiﬁer (Hoffer et al., 2018).

Original: Despite this representation-learning power, methods that yield high generalization performance seem to produce networks with little behavior diversity: models make similar predictions, with high-accuracy models rarely making mistakes that low-accuracy models predict correctly (Mania et al., 2019).
Revised: Despite this representation-learning power, methods that yield high generalization performance seem to produce networks with little behavior diversity: models make similar predictions, with high-accuracy models rarely making mistakes that low-accuracy models predict correctly (Mania et al., 2019).

Original: Additionally, the quality of features learned (e.g.: for downstream tasks) seems dictated by upstream performance (Kornblith et al., 2019).
Revised: Additionally, the quality of features learned (e.g.: for downstream tasks) seems dictated by upstream performance (Kornblith et al., 2019).

Original: Finally, training on subsets of the data yields low-accuracy models that don’t make performant ensembles (Nixon et al., 2020).
Revised: Finally, training on subsets of the data yields low-accuracy models that don’t make performant ensembles (Nixon et al., 2020).

Original: This seemingly suggests that high-performing models share similar biases, regardless of training methodology.
Revised: This seemingly suggests that high-performing models share similar biases, regardless of training methodology.

Original: Without behavior diversity, ensemble beneﬁts are limited to reducing noise, since models make correlated errors (Perrone & Cooper, 1992; Opitz & Maclin, 1999).
Revised: Without behavior diversity, ensemble beneﬁts are limited to reducing noise, since models make correlated errors (Perrone & Cooper, 1992; Opitz & Maclin, 1999).

Original: Without feature diversity, representations might not capture important features for downstream tasks, since feature reuse has been shown to be crucial for transfer learning (Neyshabur et al., 2020).
Revised: Without feature diversity, representations might not capture important features for downstream tasks, since feature reuse has been shown to be crucial for transfer learning (Neyshabur et al., 2020).

Original: Without knowing the effect of training methodology, one might conclude that low-accuracy models have no practical use, since their predictions would be dominated by high-accuracy ones.
Revised: Without knowing the effect of training methodology, one might conclude that low-accuracy models have no practical use, since their predictions would be dominated by high-accuracy ones.

Original: One open question is if these ﬁndings faced unavoidable selection bias, since the highest-performing models have historically been trained with similar supervised objectives on IID datasets.
Revised: One open question is if these ﬁndings faced unavoidable selection bias, since the highest-performing models have historically been trained with similar supervised objectives on IID datasets.

Original: Up until recently, this hypothesis was difﬁcult to test.
Revised: Up until recently, this hypothesis was difﬁcult to test.

Original: That changed with the recent success of large-scale contrastive learning, which produces competitively-high accuracy on standard generalization and robustness benchmarks (Radford et al., 2021;
Revised: That changed with the recent success of large-scale contrastive learning, which produces competitively-high accuracy on standard generalization and robustness benchmarks (Radford et al., 2021;

Original: Jia et al., 2021).
Revised: Jia et al., 2021).

Original: This motivates revisiting the question:
Revised: This motivates revisiting the question:

Original: How does training methodology affect learned representations and prediction behavior?
Revised: How does training methodology affect learned representations and prediction behavior?

Original: In this paper, we conduct a systematic empirical study of 82 models, which we train or collect, across hyper-parameters, architectures, objective functions, and datasets, including the latest high performing models CLIP, ALIGN, SimCLR, BiT, ViT-G/14, and MPL.
Revised: To settle these questions, we conduct a systematic empirical study of 82 models, which we train or collect, across hyper-parameters, architectures, objective functions, and datasets, including the latest high performing models CLIP, ALIGN, SimCLR, BiT, ViT-G/14, and MPL.

Original: In addition to using different techniques, these new models were trained on data collected very differently, allowing us to probe the effect of both training objective, as well as pre-training data.
Revised: In addition to using different techniques, these new models were trained on data collected very differently, allowing us to probe the effect of both training objective, as well as pre-training data.

Original: We categorize these models based on how their training methodologies diverge from a typical, base model and show:
Revised: We categorize these models based on how their training methodologies diverge from a typical, base model and show:

Original: Model pairs that diverge more in training methodology (reinitializations, hyper-parameters, architectures, frameworks, → dataset scales) produce increasingly uncorrelated errors.
Revised: Model pairs that diverge more in training methodology (in order: reinitializations (cid:41) hyperparameters (cid:41) architectures (cid:41) frameworks (cid:41) datasets) produce increasingly uncorrelated errors.

Original: Ensemble performance increases as error correlation decreases, due to higher ensemble efﬁciency.
Revised: Ensemble performance increases as error correlation decreases, due to higher ensemble efﬁciency.

Original: The most typical ImageNet model (ResNet-50, 76.5%), and its most different counterpart (ALIGN-ZS, 75.5%) yield 83.4% accuracy when ensembled, a +7% boost.
Revised: The most typical ImageNet model (ResNet-50, 76.5%), and its most different counterpart (ALIGN-ZS, 75.5%) yield 83.4% accuracy when ensembled, a +7% boost.

Original: Contrastively-learned models display categorically different generalization behavior, specializing in subdomains of the data, which explains the higher ensembling efﬁciency.
Revised: Contrastively-learned models display categorically different generalization behavior, specializing in subdomains of the data, which explains the higher ensembling efﬁciency.

Original: We show CLIP-S specializes in antropogenic images, whereas ResNet-50 excells in nature images.
Revised: We show CLIP-S specializes in antropogenic images, whereas ResNet-50 excels in nature images.

Original: Surprisingly, we ﬁnd that low-accuracy models can be useful if they are trained differently enough.
Revised: Surprisingly, we ﬁnd that low-accuracy models can be useful if they are trained differently enough.

Original: By combining a high-accuracy model (BiT-1k, 82.9%) with only low-accuracy models (max individual acc.
Revised: By combining a high-accuracy model (BiT-1k, 82.9%) with only low-accuracy models (max individual acc.

Original: 77.4%), we can create ensembles that yield as much as 86.7%.
Revised: 77.4%), we can create ensembles that yield as much as 86.7%.

Original: Diverging training methodology yield representations that capture overlapping (but not supersetting) feature sets which, when combined, lead to increased downstream performance (91.4% on Pascal VOC, using models with max individual accuracy 90.7%).
Revised: Diverging training methodology yield representations that capture overlapping (but not supersetting) feature sets which, when concatenated, lead to increased downstream performance (91.4% on Pascal VOC, using models with max individual accuracy 90.7%).

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Diversity in Ensembles .
Revised: Diversity in Ensembles .

Original: It is widely understood that good ensembles are made of models that are both accurate and make independent errors (Perrone & Cooper,1992; Opitz & Maclin, 1999; Wen et al., 2020).
Revised: It is widely understood that good ensembles are made of models that are both accurate and make independent errors (Perrone & Cooper, Opitz & Maclin , 1999; Wen et al., 2020).

Original: Beyond improving ensemble performance, ﬁnding diverse solutions that equally well explain the observations can help quantify model uncertainty (also known as epistemic uncertainty) – what the model does not know because training data was not appropriate (Kendall & Gal, 2017; 
Revised: Beyond improving ensemble performance, ﬁnding diverse solutions that equally well explain the observations can help quantify model uncertainty (also known as epistemic uncertainty) – what the model does not know because training data was not appropriate (Kendall & Gal, 2017; Fort et al., 2019).

Original: Fort et al., 2019).

Original: Many works have explored ways of ﬁnding such solutions (Izmailov et al., 2018).
Revised: Many works have explored ways of ﬁnding such solutions (Izmailov et al., 2018).

Original: Boostrapping (Freund et al., 1996) (ensembling models trained on subsets of the data) was found not to produce deep ensembles with higher accuracy than a single model trained on the entire dataset (Nixon et al., 2020).
Revised: Boostrapping (Freund et al., 1996) (ensembling models trained on subsets of the data) was found not to produce deep ensembles with higher accuracy than a single model trained on the entire dataset (Nixon et al., 2020).

Original: This emphasizes how much data deep neural networks need to achieve high performance.

Original: Another work has examined the effect of augmentation-induced prediction diversity on adversarial robustness (Liu et al., 2019).
Revised: Another work has examined the effect of augmentation-induced prediction diversity on adversarial robustness (Liu et al., 2019).

Original: More relevant to us, Wenzel et al.
Revised: More relevant to us, Wenzel et al.

Original: (2020) has explored the effect of random hyper-parameters, ﬁnding best ensembles when combining models that are both hyperparameter and weight-diverse, albeit still considering similar frameworksand architectures.
Revised: (2020) and Zaidi et al. (2021) have explored the effect of random hyper-parameters and architectures respectively ﬁnding best ensembles when combining diverse models, albeit still considering similar frameworks

Original: Model Behavior Similarity .
Revised: Model Behavior Similarity .

Original: These attempts were hindered as many high-performing techniques seem to produce similar prediction behavior.
Revised: These attempts were hindered as many high-performing techniques seem to produce similar prediction behavior.

Original: Mania et al.
Revised: Mania et al.

Original: (2019) demonstrates, via “dominance probabilities”, that high-accuracy models rarely make mistakes that low-accuracy models predict correctly.
Revised: (2019) demonstrates, via “dominance probabilities”, that high-accuracy models rarely make mistakes that low-accuracy models predict correctly.

Original: This indicates that, within the models studied, high-accuracy models “dominate” the predictions of low-accuracy ones.
Revised: This indicates that, within the models studied, high-accuracy models “dominate” the predictions of low-accuracy ones.

Original: Recht et al.
Revised: Recht et al.

Original: (2019) shows that out-of-distribution robustness seems correlated with in-distribution performance.
Revised: (2019) shows that out-of-distribution robustness seems correlated with in-distribution performance.

Original: Relatedly, Kornblith et al.
Revised: Relatedly, Kornblith et al.

Original: (2019) shows that upstream and downstream performance are very correlated.
Revised: (2019) shows that upstream and downstream performance are very correlated.

Original: These jointly indicate that high-accuracy models learn strictly better representations, diminishing the importance of low-accuracy solutions (even if they are diverse).
Revised: These jointly indicate that high-accuracy models learn strictly better representations, diminishing the importance of low-accuracy solutions (even if they are diverse).

Original: Finally, Fort et al.
Revised: Finally, Fort et al.

Original: (2019) shows that subspace-sampling methods for ensembling generate solutions that, while different in weight space, remain similar in function space, which gives rise to an insufﬁciently diverse set of predictions.
Revised: (2019) shows that subspace-sampling methods for ensembling generate solutions that, while different in weight space, remain similar in function space, which gives rise to an insufﬁciently diverse set of predictions.

Original: Contrastive-Learning Models; Different Large-Scale Datasets .
Revised: Contrastive-Learning Models; Different Large-Scale Datasets .

Original: This model behavior similarity might be explained by the fact that the training techniques that yield high performance on image classiﬁcation tasks have been relatively similar, mostly relying on supervised learning on ImageNet, optionally pre-training on a dataset with similar distribution.
Revised: This model behavior similarity might be explained by the fact that the training techniques that yield high performance on image classiﬁcation tasks have been relatively similar, mostly relying on supervised learning on ImageNet, optionally pre-training on a dataset with similar distribution.

Original: Recently, various works have demonstrated the effectiveness of learning from large-scale data using contrastive learning (Radford et al., 2021; Jia et al., 2021).
Revised: Recently, various works have demonstrated the effectiveness of learning from large-scale data using contrastive learning (Radford et al., 2021; Jia et al., 2021).

Original: They report impressive results on out-of-distribution benchmarks, and have been shown to have higher dominance probabilities (Andreassen et al., 2021).
Revised: They report impressive results on out-of-distribution benchmarks, and have been shown to have higher dominance probabilities (Andreassen et al., 2021).

Original: These represent some of the ﬁrst models to deviate from standard supervised training (or ﬁnetuning) on downstream data, while still yielding competitive accuracy.
Revised: These represent some of the ﬁrst models to deviate from standard supervised training (or ﬁnetuning) on downstream data, while still yielding competitive accuracy.

Original: They add to the set of high-performing training techniques (which include data augmentation (Cubuk et al., 2018; 2020), regularization (Srivastava et al., 2014;
Revised: They add to the set of high-performing training techniques, which include data augmentation (Cubuk et al., 2018; 2020), regularization (Srivastava et al., 2014;

Original: Szegedy et al., 2016; Ghiasi et al., 2018), architectures (Tan & Le, 2019; Dosovitskiy et al., 2020; Hu et al., 2018; Iandola et al., 2014;
Revised: Szegedy et al., 2016; Ghiasi et al., 2018), architectures (Tan & Le, 2019; Dosovitskiy et al., 2020; Hu et al., 2018; Iandola et al., 2014;

Original: Li et al., 2019; Szegedy et al., 2016; Simonyan & Zisserman, 2014;
Revised: Li et al., 2019; Szegedy et al., 2016; Simonyan & Zisserman, 2014;

Original: Sandler et al., 2018)
Revised: Sandler et al., 2018)

Original: , losses (Chen et al., 2020; Radford et al., 2021;
Revised: , losses (Chen et al., 2020; Radford et al., 2021;

Original: Jia et al., 2021), pre-training schemes (Kolesnikov et al., 2020; Pham et al., 2021),etc), and provide the motivation for revisiting the question of whether training methodology can yield different model behavior.
Revised: Jia et al., 2021), pre-training schemes (Kolesnikov et al., 2020; Pham et al., 2021), and provide the motivation for revisiting the question of whether training methodology can yield different model behavior.

Original:  M ETHOD 3. M ODEL C ATEGORIZATION
Revised:  M ETHOD 3. M ODEL C ATEGORIZATION

Original: In order to evaluate the performance of learned representations as a function of training methodology, we deﬁne the following categories, which classify model pairs based on their training differences:
Revised: In order to evaluate the performance of learned representations as a function of training methodology, we deﬁne the following categories, which classify model pairs based on their training differences:

Original: Reinits : identical models, just different in reinitialization.
Revised: Reinits : identical models, just different in reinitialization.

Original: Hyper-parameters : models of the same architecture trained with different hyper parameters (e
Revised: Hyper-parameters : models of the same architecture trained with different hyper parameters (e

Original: g.: weight decay, learning rate, initialization algorithm, etc).
Revised: g.: weight decay, learning rate, initialization scheme, etc).

Original: Architectures : models with different architectures, but still trained within the same framework and dataset (e.g.: ResNet and ViT, both with ImageNet supervision).
Revised: Architectures : models with different architectures, but still trained within the same framework and dataset (e.g.: ResNet and ViT, both with ImageNet supervision).

Original: Frameworks : models trained with different optimization objectives, but on same dataset (e.g.: ResNet and SimCLR, respectively supervised and contrastive learning on ImageNet).
Revised: Frameworks : models trained with different optimization objectives, but on same dataset (e g.: ResNet and SimCLR, respectively supervised and contrastive learning on ImageNet).

Original: Datasets : models trained on large-scale data (e.g.: CLIP or BiT – trained on WIT or JFT).
Revised: Datasets : models trained on large-scale data (e g.: CLIP or BiT – trained on WIT or JFT).

Original: In some sense, model categories can be supersets of one another: when we change a model architecture, we may also change the hyper-parameters used to train such architecture, to make sure that they are optimal for training this new setting.
Revised: In some sense, model categories can be supersets of one another: when we change a model architecture, we may also change the hyper-parameters used to train such architecture, to make sure that they are optimal for training this new setting.

Original: Unless stated otherwise, all ensembles are comprised of a ﬁxed base model, and another model belonging to one of the categories above.
Revised: Unless stated otherwise, all ensembles are comprised of a ﬁxed base model, and another model belonging to one of the categories above.

Original: This way, each category is deﬁned relative to the base model: model pairs in a given category will vary because
Revised: This way, each category is deﬁned relative to the base model: model pairs in a given category will vary because

Original: Model 2 is different than the base model along that axis.
Revised: Model 2 is different than the base model along that axis.

Original: The result is that as we navigate along model categories (Reinit → ... → Dataset), we will naturally be measuring the effect of increasingly dissimilar training methodology .
Revised: The result is that as we navigate along model categories (Reinit → ... → Dataset), we will naturally be measuring the effect of increasingly dissimilar training methodology .

Original: See Appendix Table 1 for details.
Revised: See Appendix Table 1 for details.

Original: 3. M ODEL S ELECTION
Revised: 3. M ODEL S ELECTION

Original: We collect representations and predictions for 82 models, across the many categories above.
Revised: We collect representations and predictions for 82 models, across the many categories above.

Original: We ﬁx ResNet-50, trained with RandAugment, as our base model.
Revised: We ﬁx ResNet-50, trained with RandAugment, as our base model.

Original: ResNet is a good candidate for a base model since it is one of the most typical ImageNet classiﬁcation models, and the de-facto standard baseline for this task.
Revised: ResNet is a good candidate for a base model since it is one of the most typical ImageNet classiﬁcation models, and the de-facto standard baseline for this task.

Original: In total, we train/collect models in the categories: 1) Reinit; 2) Hyperparameters (51): varying dropout, dropblock, learning rate, and weight decay, sometimes jointly; 3) Architectures (17): including EfﬁcientNet, ViT, DenseNet, VGG; 4) Framework (2): including
Revised: In total, we train/collect models in the categories: 1) Reinit; 2) Hyperparameters (51): varying dropout, dropblock, learning rate, and weight decay, sometimes jointly; 3) Architectures (17): including EfﬁcientNet, ViT, DenseNet, VGG; 4) Framework (2): including

Original: SimCLR, and models trained with distillation; and 5) Dataset (12): including CLIP, ALIGN, BiT, and more, trained on WIT (Radford et al., 2021), the ALIGN dataset, JFT (Sun et al., 2017), etc.
Revised: SimCLR, and models trained with distillation; and 5) Dataset (12): including CLIP, ALIGN, BiT, and more, trained on WIT (Radford et al., 2021), the ALIGN dataset, JFT (Sun et al., 2017), etc.

Original: We additionally collect high-performing models MPL, ALIGN (L-BFGS), ViT-G/14, BiT-1k, CLIP-L,
Revised: We additionally collect high-performing models MPL, ALIGN (L-BFGS), ViT-G/14, BiT-1k, CLIP-L,

Original: EfﬁcientNet-B3 for some of our analyses.
Revised: EfﬁcientNet-B3 for some of our analyses.

Original: These are some of the latest, highest-performing models for ImageNet classiﬁcation.
Revised: These are some of the latest, highest-performing models for ImageNet classiﬁcation.

Original: We found it necessary to calibrate all models using temperature scaling (Roelofs et al., 2020; Guo et al., 2017) to maximize ensemble performance.
Revised: We found it necessary to calibrate all models using temperature scaling (Roelofs et al., 2020; Guo et al., 2017) to maximize ensemble performance.

Original: Finally, unless stated otherwise, we only use models in the same narrow accuracy range (74-78% accuracy on ImageNet), which guarantees that the effects observed are indeed a function of diverging training methodology, and not of any given model being intrinsically more performant than another.
Revised: Finally, unless stated otherwise, we only use models in the same narrow accuracy range (74-78% accuracy on ImageNet), which guarantees that the effects observed are indeed a function of diverging training methodology, and not of any given model being intrinsically more performant than another.

Original: A complete list of models can be found in the Appendix.
Revised: A complete list of models can be found in the Appendix.

Original: 3. E NSEMBLING
Revised: 3. E NSEMBLING

Original: In our paper, we use ensembling as a tool for understanding: as such, our goal is not to ﬁnd methods to ensemble the highest accuracy models and reach state-of-the-art.
Revised: In our paper, we use ensembling as a tool for understanding: as such, our goal is not to ﬁnd methods to ensemble the highest accuracy models and reach state-of-the-art.

Original: Instead, we wish to use ensembling to probe whether/when training methodology yields uncorrelated (and therefore useful) predictions.
Revised: Instead, we wish to use ensembling to probe whether/when training methodology yields uncorrelated (and therefore useful) predictions.

Original: [Figure]
Revised: [Figure]

Revised: y

Original:  R ESULTS
Revised:  R ESULTS

Original: In order to understand whether model similarity reported in literature varies as a function of training methodology, we evaluate error correlation by measuring the number of test-set examples where one model predicts the correct class, and the other predicts an incorrect class.
Revised: In order to understand whether model similarity reported in literature varies as a function of training methodology, we evaluate error correlation by measuring the number of test-set examples where one model predicts the correct class, and the other predicts an incorrect class.

Original: We call this Error
Revised: We call this Error

Original: Inconsistency , as it complements the error consistency measure, deﬁned in Geirhos et al.
Revised: Inconsistency , as it complements the error consistency measure, deﬁned in Geirhos et al.

Original: We choose error inconsistency to measure error correlation because it allows us to connect the ensemble prediction diversity (which we are interested in quantifying) directly into performance.
Revised: We choose error inconsistency to measure error correlation because it allows us to connect the ensemble prediction diversity (which we are interested in quantifying) directly into performance.

Original: That is, when errors are consistent (i.e.: both models make a correct prediction, or both models make an incorrect prediction), this agreement directly translates into higher/lower accuracy after ensembling.
Revised: That is, when errors are consistent (i.e.: both models make a correct prediction, or both models make an incorrect prediction), this agreement directly translates into higher/lower accuracy after ensembling.

Original: When errors are inconsistent (i.e.: when one model is correct and another is incorrect), the ensemble prediction will be determined by the models’ conﬁdences.
Revised: When errors are inconsistent (i.e.: when one model is correct and another is incorrect), the ensemble prediction will be determined by the models’ conﬁdences.

Original: If a model is sufﬁciently conﬁdent, its prediction will “win” the ensembling procedure.
Revised: If a model is sufﬁciently conﬁdent, its prediction will “win” the ensembling procedure.

Original: If this model’s prediction was correct, we say that this prediction was “converted” into a correct ensemble prediction.
Revised: If this model’s prediction was correct, we say that this prediction was “converted” into a correct ensemble prediction.

Original: 4. A S TRAINING METHODOLOGY DIVERGES , ERRORS BECOME MORE UNCORRELATEDIn
Revised: 4. A S TRAINING METHODOLOGY DIVERGES , ERRORS BECOME MORE UNCORRELATED

Original:  Fig.
Revised: In Fig.

Original: 1 we see that, as we compare increasingly different training methodologies (“Reinit” → ... → “Dataset”), error inconsistency increases – the number of examples where only one model makes a correct prediction.
Revised: 1 we see that, as we compare increasingly different training methodologies (“Reinit” → ... → “Dataset”), error inconsistency increases – the number of examples where only one model makes a correct prediction.

Original: This indicates that as training methodology diverges, model predictions become dissimilar and, as a result, errors become uncorrelated.
Revised: This indicates that as training methodology diverges, model predictions become dissimilar and, as a result, errors become uncorrelated.

Revised: As the framework and dataset categories can be orthogonal, we note that models with the most error inconsistency tend to modify both.

Original: This also represents an opportunity for these uncorrelated mistakes to be converted into a correct prediction, when we ensemble the models.
Revised: This also represents an opportunity for these uncorrelated mistakes to be converted into a correct prediction, when we ensemble the models.

Original: Such an opportunity will be beneﬁcial if the conversion rate of these examples is higher than the decrease in number of examples where both models made correct predictions.
Revised: Such an opportunity will be beneﬁcial if the conversion rate of these examples is higher than the decrease in number of examples where both models made correct predictions.

Original: As we will see in the following section, this indeed happens.
Revised: As we will see in the following section, this indeed happens.

Original: 4. A S UNCORRELATED ERRORS INCREASE , SO DOES ENSEMBLE EFFICIENCY
Revised: 4. A S UNCORRELATED ERRORS INCREASE , SO DOES ENSEMBLE EFFICIENCY

Original: In order to assess whether these increased uncorrelated errors can be beneﬁcial, we measure how models in the various categories ensemble.
Revised: In order to assess whether these increased uncorrelated errors can be beneﬁcial, we measure how models in the various categories ensemble.

Original: That is, we ensemble our base model with models of the various categories, and measure the ensemble’s accuracy, relative to its member models.
Revised: That is, we ensemble our base model with models of the various categories, and measure the ensemble’s accuracy, relative to its member models.

Original: Per Fig. 2, as uncorrelated errors increase, so does ensemble performance (left).
Revised: Per Fig. 2, as uncorrelated errors increase, so does ensemble performance (left).

Original: Additionally, because we restrict our models to ones within the narrow accuracy range 74-78%, we guarantee that this relative improvement translates into absolute improvement (center), and is not due to any individual ensemble member being intrinsically better than another.
Revised: Additionally, because we restrict our models to ones within the narrow accuracy range 74-78%, we guarantee that this relative improvement translates into absolute improvement (center), and is not due to any individual ensemble member being intrinsically better than another.

Original: Relative to our base model ResNet- (76.5% on ImageNet), the most differently-trained model ALIGN-ZeroShot (75.5%) yields 83.4% top-1 accuracy when ensembled, a boost of nearly 7% accuracy.[Figure] Taken together
Revised: Relative to our base model ResNet- (76.5% on ImageNet), the most differently-trained model ALIGN-ZeroShot (75.5%) yields 83.4% top-1 accuracy when ensembled, a boost of nearly 7% accuracy.

Original: [Figure]
Revised: [Figure]

Original:  these results mean that combining models with diverse training methodologies can yield performance improvements that offset the decrease in examples where both models are correct.
Revised: Taken together, these results mean that combining models with diverse training methodologies can yield performance improvements that offset the decrease in examples where both models are correct.

Original: To explain this relative boost [Figure] we also measure the conversion rate of each ensemble – the rate at which inconsistent errors are converted into correct ensemble predictions.
Revised: To explain this relative boost, we also measure the conversion rate of each ensemble – the rate at which inconsistent errors are converted into correct ensemble predictions.

Original: Surprisingly, this rate also increases as training methods diverge (Fig. 2, right).
Revised: Surprisingly, this rate also increases as training methods diverge (Fig. 2, right).

Original: This means that different training methods not only yield more opportunities for ensemble beneﬁts, but also more efﬁcient ensembles.
Revised: This means that different training methods not only yield more opportunities for ensemble beneﬁts, but also more efﬁcient ensembles.

Original: 4. D ISSIMILAR TRAINING METHODOLOGIES CREATE SPECIALIZED MODELS
Revised: 4. D ISSIMILAR TRAINING METHODOLOGIES CREATE SPECIALIZED MODELS

Original: To understand how diverging training setups create efﬁcient ensembles, we measure the specialization of ensemble member models.
Revised: To understand how diverging training setups create efﬁcient ensembles, we measure the specialization of ensemble member models.

Original: We ﬁrst note that, for a model’s prediction to “win” the ensembling process, it only needs to be sufﬁciently more conﬁdent than the other model’s prediction.
Revised: We ﬁrst note that, for a model’s prediction to “win” the ensembling process, it only needs to be sufﬁciently more conﬁdent than the other model’s prediction.

Original: This means that we can measure specialization by the relative conﬁdence of two models.
Revised: This means that we can measure specialization by the relative conﬁdence of two models.

Original: We do this by measuring the angle distance θ in the conﬁdence-conﬁdence plot (see Fig.
Revised: We do this by measuring the angle distance θ in the conﬁdence-conﬁdence plot (see Fig.

Original: 3, top left).
Revised: 3, top left).

Original: When θ is high, Modelis more conﬁdent (relative to Model 2), and vice-versa.
Revised: When θ is high, Modelis more conﬁdent (relative to Model 2), and vice-versa.

Original: In Fig.
Revised: In Fig.

Original: 3, we use θ to understand the specialization of different models categories.
Revised: 3, we use θ to understand the specialization of different models categories.

Original: To simplify our analysis, we compare three different ensembles, from the categories Reinit (Model 2: ResNet-50),
Revised: To simplify our analysis, we compare three different ensembles, from the categories Reinit (Model 2: ResNet-50),

Original: Architecture (EfﬁcientNet-B3), and Dataset (CLIP-S), as they are representative of the spread of error correlation.
Revised: Architecture (EfﬁcientNet-B3), and Dataset (CLIP-S), as they are representative of the spread of error correlation.

Original: When we plot a histogram of examples as a function of θ , we ﬁnd that: 1) As observed in Fig.
Revised: When we plot a histogram of examples as a function of θ , we ﬁnd that: 1) As observed in Fig.

Original: 1, when training methods diverge, the models make mistakes in different sets of examples 2)
Revised: 1, when training methods diverge, the models make mistakes in different sets of examples 2)

Original: As training methods diverge, Model 1 tends to be more conﬁdent in examples where only Model 1 is correct, and vice-versa for Model 2.
Revised: As training methods diverge, Model 1 tends to be more conﬁdent in examples where only Model 1 is correct, and vice-versa for Model 2.

Original: This effect is most striking when we look at the
Revised: This effect is most striking when we look at the

Original: Dataset category, where Model 2 (CLIP-S) is signiﬁcantly more conﬁdent than Model 1 (ResNet-50) in examples where only CLIP-S predicts correctly, and vice-versa.
Revised: Dataset category, where Model 2 (CLIP-S) is signiﬁcantly more conﬁdent than Model 1 (ResNet-50) in examples where only CLIP-S predicts correctly, and vice-versa.

Original: This is in contrast with Reinit, where models don’t seem to be signiﬁcantly more conﬁdent than each other at all.
Revised: This is in contrast with Reinit, where models don’t seem to be signiﬁcantly more conﬁdent than each other at all.

Original: These results show: as training methodology diverges, model specialize to different data subdomains.
Revised: These results show: as training methodology diverges, model specialize to different data subdomains.

Original: 4. M ODEL SPECIALIZATION DEPENDS ON TRAINING METHODOLOGY
Revised: 4. M ODEL SPECIALIZATION DEPENDS ON TRAINING METHODOLOGY

Original: Next, we want to investigate what kind of data each model category specializes to.
Revised: Next, we want to investigate what kind of data each model category specializes to.

Original: In Fig.
Revised: In Fig.

Original: 4, we plot the same data as Fig. 3, but we divide the examples along their ImageNet class IDs.
Revised: 4, we plot the same data as Fig. 3, but we divide the examples along their ImageNet class IDs.

Original: This allows us to inspect whether a given model is more/less specialized in a given class.
Revised: This allows us to inspect whether a given model is more/less specialized in a given class.

Original: As before, we ﬁnd that, while models in the Architecture category demonstrate higher specialization (Model 1 tends to be more conﬁdent in examples where it is correct, and vice versa for Model 2),
Revised: As before, we ﬁnd that, while models in the Architecture category demonstrate higher specialization (Model 1 tends to be more conﬁdent in examples where it is correct, and vice versa for Model 2),

Original: [Figure]
Revised: [Figure]

Original: 1 prediction is likely to prevail at ensemble time
Revised: prediction is likely to prevail at ensemble time

Original: [Figure]
Revised: [Figure]

Original: this specialization does not seem to correlate with speciﬁc classes.
Revised: this specialization does not seem to correlate with speciﬁc classes.

Original: In contrast, not only do models of the Dataset category display more specialization, but this specialization seems to be correlated with groups of classes.
Revised: In contrast, not only do models of the Dataset category display more specialization, but this specialization seems to be correlated with groups of classes.

Original: In particular, CLIP-S seems to specialize to anthropogenic images, whereas
Revised: In particular, CLIP-S seems to specialize to anthropogenic images, whereas

Original: ResNet-50 seems to specialize to nature images.
Revised: ResNet-50 seems to specialize to nature images.

Original: We suspect this phenomena is a result of CLIP-S being trained on a dataset that was collected independently, and much differently than the one used to train our base 
Revised: We suspect this phenomena is a result of CLIP-S being trained on a dataset that was collected independently, and much differently than the one used to train our base model

Original: [Figure]
Revised: [Figure]

Original: 4. D IFFERENT ENOUGH LOWER - ACCURACY MODELS CAN IMPROVE ACCURACY
Revised: 4. D IFFERENT ENOUGH LOWER - ACCURACY MODELS CAN IMPROVE ACCURACY

Original: In classic machine learning, a common way to create high-performing ensembles is to create specialized models with bootstrapping (Freund et al., 1996), where models are trained on subsets of the data and subsequently ensembled.
Revised: In classic machine learning, a common way to create high-performing ensembles is to create specialized models with bootstrapping (Freund et al., 1996), where models are trained on subsets of the data and subsequently ensembled.

Original: In deep learning, bootstrapping has not been found to work well, since training on IID subsets of the data (with similar methodology) does not seem to yield ensembles that perform better than an individual model trained on the entire dataset (Nixon et al., 2020).
Revised: In deep learning, bootstrapping has not been found to work well, since training on IID subsets of the data (with similar methodology) does not seem to yield ensembles that perform better than an individual model trained on the entire dataset (Nixon et al., 2020).

Original: This seems to indicate that lower-accuracy models would have little practical beneﬁt for performance.
Revised: This seems to indicate that lower-accuracy models would have little practical beneﬁt for performance.

Original: In order to investigate this, and encouraged by the ﬁnding that differently-trained models can be experts in subdomains of the data, we ask whether lower-accuracy models can be beneﬁcial.
Revised: In order to investigate this, and encouraged by the ﬁnding that differently-trained models can be experts in subdomains of the data, we ask whether lower-accuracy models can be beneﬁcial.

Original: After all, some of the models in the Dataset category were trained on much larger-scale datasets (JFT, WIT), which are collected independently.
Revised: After all, some of the models in the Dataset category were trained on much larger-scale datasets (JFT, WIT), which are collected independently.

Original: To do this, we ﬁrst combine our base model, ResNet-50 (76.2%), with a lower-accuracy model trained very differently, CLIP-S-ZeroShot (63.3%, “Dataset” category), and observe a performance boost (77.7%).
Revised: To do this, we ﬁrst combine our base model, ResNet-50 (76.2%), with a lower-accuracy model trained very differently, CLIP-S-ZeroShot (63.3%, “Dataset” category), and observe a performance boost (77.7%).

Original: To push this idea further, we combine a high-accuracy model (BiT-1k, 82.85%) with only lower-accuracy models (max individual accuracy 77.44%), by greedily selecting ensemble members that maximize accuracy.
Revised: To push this idea further, we combine a high-accuracy model (BiT-1k, 82.85%) with only lower-accuracy models (max individual accuracy 77.44%), by greedily selecting ensemble members that maximize accuracy.

Original: With this procedure, we can create ensembles that yield as much as 86.66%.
Revised: With this procedure, we can create ensembles that yield as much as 86.66%.

Original: In Fig.
Revised: In Fig.

Original: 5, we repeat this procedure for 3 additional high-accuracy models and ﬁnd that lower-accuracy models can consistently improve the accuracy of high-accuracy models.
Revised: 5, we repeat this procedure for 3 additional high-accuracy models and ﬁnd that lower-accuracy models can consistently improve the accuracy of high-accuracy models.

Original: Importantly, the low-accuracy models need to be different enough in their training methodology (in terms of
Revised: Importantly, the low-accuracy models need to be different enough in their training methodology (in terms of

Original: Sec. 3.1 categories): for a given high-accuracy model, the most beneﬁcial lower-accuracy model to ensemble is one trained with a different loss and/or dataset, often both.
Revised: Sec. 3.1 categories): for a given high-accuracy model, the most beneﬁcial lower-accuracy model to ensemble is one trained with a different loss and/or dataset, often both.

Original: See table in Fig.
Revised: See table in Fig.

Original: 5 for details.
Revised: 5 for details.

Original: This result calls into question the idea that the information learned by high accuracy models is a strict superset of those learned by low accuracy models, which we explore further in sections below.
Revised: This result calls into question the idea that the information learned by high accuracy models is a strict superset of those learned by low accuracy models, which we explore further in sections below.

Original: 4. S PECIALIZATION COMES FROM OVERLAPPING ( BUT NOT SUPERSETTING ) REPRESENTATIONS
Revised: 4. S PECIALIZATION COMES FROM OVERLAPPING ( BUT NOT SUPERSETTING ) REPRESENTATIONS

Original: In order to explain why differently-trained models specialize, we posit that training methodology changes the features learned by individual models.
Revised: In order to explain why differently-trained models specialize, we posit that training methodology changes the features learned by individual models.

Original: In this view, models trained differently would have an overlapping (but not supersetting) set of features which, when combined, provide more information for classiﬁcation than any individual model’s representation could.
Revised: In this view, models trained differently would have an overlapping (but not supersetting) set of features which, when combined, provide more information for classiﬁcation than any individual model’s representation could.

Original: We test this directly by concatenating the features of our base model (ResNet-50) with those of the models above (ResNet-50, EfﬁcientNet-B0 and CLIP-S), and linearly evaluating these combined features on ImageNet classiﬁcation.
Revised: We test this directly by concatenating the features of our base model (ResNet-50) with those of the models above (ResNet-50, EfﬁcientNet-B0 and CLIP-S), and linearly evaluating these combined features on ImageNet classiﬁcation.

Original: More speciﬁcally, we ﬁrst randomly select features from the base model and each other model at inversely proportional rates.
Revised: More speciﬁcally, we ﬁrst randomly select features from the base model and each other model at inversely proportional rates.

Original: For example, we concatenate 25% of ResNet-50 features with 75% of CLIP-S features, yielding a ﬁnal embedding that is at most the same dimensionality of the ResNet-50 features.
Revised: For example, we concatenate 25% of ResNet-50 features with 75% of CLIP-S features, yielding a ﬁnal embedding that is at most the same dimensionality of the ResNet-50 features.

Original: This guarantees that any performance boost is not due to a higher number of dimensions being available, but by the quality of the features represented.
Revised: This guarantees that any performance boost is not due to a higher number of dimensions being available, but by the quality of the features represented.

Original: [Figure]
Revised: [Figure]

Original: In Fig.
Revised: In Fig.

Original: 6 (center)
Revised: 6 (center)

Original: we see that the best performance is obtained when features are combined.
Revised: we see that the best performance is obtained when features are combined.

Original: Additionally, we ﬁnd that combining features yields higher performance as training methodology diverges, with the best performing combination being ResNet-50 + CLIP-S. Concurrent work similarly ﬁnds that different training objectives yield different ﬁnal-layer feature embeddings (Grigg et al., 2021).
Revised: Additionally, we ﬁnd that combining features yields higher performance as training methodology diverges, with the best performing combination being ResNet-50 + CLIP-S. Concurrent work similarly ﬁnds that different training objectives yield different ﬁnal-layer feature embeddings (Grigg et al., 2021).

Original: This seems to conﬁrm the idea that the methods used to train networks can generate diverse features, which capture information that neither embedding alone has captured.
Revised: This seems to conﬁrm the idea that the methods used to train networks can generate diverse features, which capture information that neither embedding alone has captured.

Original: To push this further, we ask how efﬁciently these representations capture important features of the data.
Revised: To push this further, we ask how efﬁciently these representations capture important features of the data.

Original: To test this, we ﬁrst compute the covariance of each dimension of embeddings from both models (e.g.: ResNet-50 and CLIP-S).
Revised: To test this, we ﬁrst compute the covariance of each dimension of embeddings from both models (e.g.: ResNet-50 and CLIP-S).

Original: This gives us a ranking of highest to lowest covariance dimensions, which we use as a measure of diversity.
Revised: This gives us a ranking of highest to lowest covariance dimensions, which we use as a measure of diversity.

Original: We then select features in order of most diverse to least diverse, and linearly evaluate them on ImageNet classiﬁcation.
Revised: We then select features in order of most diverse to least diverse, and linearly evaluate them on ImageNet classiﬁcation.

Original: Fig.
Revised: Fig.

Original: 6 (right) shows how divergently-trained models yield more diverse (and therefore more compressible) features.
Revised: 6 (right) shows how divergently-trained models yield more diverse (and therefore more compressible) features.

Original: We tested multiples of 256 features, and found that Reinit models neededfeatures on average to achieve 76% accuracy.
Revised: We tested multiples of 256 features, and found that Reinit models neededfeatures on average to achieve 76% accuracy.

Original: In constrast, Hyperparameter models required 2057,
Revised: In constrast, Hyperparameter models required 2057,

Original: Architecture 1392, and Dataset only required 512 features.
Revised: Architecture 1392, and Dataset only required 512 features.

Original: 4. D OWNSTREAM TASK PERFORMANCE
Revised: 4. D OWNSTREAM TASK PERFORMANCE

Original: With the knowledge that diverse training leads to diverse feature embeddings, we ask whether these can transfer better to downstream tasks.
Revised: With the knowledge that diverse training leads to diverse feature embeddings, we ask whether these can transfer better to downstream tasks.

Original: In order to maximize performance, we pick three of the highest accuracy models on ImageNet, that are also representative of very diverse training setups.
Revised: In order to maximize performance, we pick three of the highest accuracy models on ImageNet, that are also representative of very diverse training setups.

Original: Meta-Pseudo Labels (MPL) was the previous SOTA on ImageNet, reporting 90.24% topaccuracy (Pham et al., 2021).
Revised: Meta-Pseudo Labels (MPL) was the previous SOTA on ImageNet, reporting 90.24% topaccuracy (Pham et al., 2021).

Original: It is trained with a teacher network that generate pseudo labels on unlabeled data to teach a student network, whose performance on the labeled set is then used to adapt the teacher.
Revised: It is trained with a teacher network that generate pseudo labels on unlabeled data to teach a student network, whose performance on the labeled set is then used to adapt the teacher.

Original: It is trained on the JFT-300M dataset.
Revised: It is trained on the JFT-300M dataset.

Original: ViT-G/14 is the current SOTA, with 90.45% topaccuracy (Zhai et al., 2021), when measured with EMA (Polyak & Juditsky, 1992).
Revised: ViT-G/14 is the current SOTA, with 90.45% topaccuracy (Zhai et al., 2021), when measured with EMA (Polyak & Juditsky, 1992).

Original: We obtained our image embeddings without the use of EMA, so we ﬁnd effective accuracy of 88.93% It is trained supervisedly, on JFT-3B, a larger version of the JFT-300M dataset.
Revised: We obtained our image embeddings without the use of EMA, so we ﬁnd effective accuracy of 88.93% It is trained supervisedly, on JFT-3B, a larger version of the JFT-300M dataset.

Original: Finally, CLIP-S is a constrastive learning model, yielding 75.7% top-1 accuracy (with L-BFGS linear evaluation of its features on ImageNet)
Revised: Finally, CLIP-S is a constrastive learning model trained on the WIT dataset, yielding 75.7% top-1 accuracy (with L-BFGS linear evaluation of its features on ImageNet)

Original: (Radford et al., 2021).
Revised: (Radford et al., 2021).

Original: It is trained with constrastive learning, on the WIT dataset.

Original: Despite being a lower accuracy model, as we will see, it is a useful model.
Revised: Despite being a lower accuracy model, as we will see, it is a useful model.

Original: In order to test the downstream generalization ability of these models’ learned representations, we linearly evaluate them on Pascal VOC (Everingham et al., 2010) using the same evaluation method as
Revised: In order to test the downstream generalization ability of these models’ learned representations, we linearly evaluate them on Pascal VOC (Everingham et al., 2010) using the same evaluation method as

Original: Kornblith et al.
Revised: Kornblith et al.

Original: Pascal VOC is a multi-label image classiﬁcation benchmark, with 20 diverse classes ranging from vehicles, animals, household objects and people.
Revised: Pascal VOC is a multi-label image classiﬁcation benchmark, with 20 diverse classes ranging from vehicles, animals, household objects and people.

Original: This diversity of scenes make it an interesting downstream task to study.
Revised: This diversity of scenes make it an interesting downstream task to study.

Original: Performance on this benchmark is measured by 11-point mAP. We measure it according to the deﬁnition in Everingham et al.
Revised: Performance on this benchmark is measured by 11-point mAP Everingham et al.

Original: 90 ImageNet Accuracy
Revised: 90 ImageNet Accuracy

Original: P a s c a l
Revised: P a s c a l

Original: V O C 11 - p
Revised: V O C 11 - p

Original: o
Revised: o

Original: i n t m A P ( L - B F
Revised: i n t m A P ( L - B F

Original: G S )
Revised: G S )

Original: MPL
Revised: MPL

Original: ViT-G/
Revised: ViT-G/

Original: CLIP-S
Revised: CLIP-S

Original: ViT-G/14MPL
Revised: ViT-G/14MPL

Original: MPLCLIP-S
Revised: MPLCLIP-S

Original: ViT-G/14CLIP-S 0.
Revised: ViT-G/14CLIP-S 0.

Original: E rr
Revised: E rr

Original: o r I n c o n s i s t e n c y
Revised: o r I n c o n s i s t e n c y

Original: ( I m a g e N e t ) 0.
Revised: ( I m a g e N e t ) 0.

Original: ConcatenatedRepresentations 1.
Revised: ConcatenatedRepresentations 1.

Original: P a s c a l
Revised: P a s c a l

Original: V O C 11 - p
Revised: V O C 11 - p

Original: o
Revised: o

Original: i n t m A P ( S t a c k e d G e n e r a li z a t i o n )
Revised: i n t m A P ( S t a c k e d G e n e r a li z a t i o n )

Original: MPL + ViT-G/ 89.
Revised: MPL + ViT-G/ 89.

Original: MPL + CLIP-S 88.
Revised: MPL + CLIP-S 88.

Original: ViT-G/14+ CLIP-S 91.
Revised: ViT-G/14+ CLIP-S 91.

Original: In Fig.7 we ﬁnd that, surprisingly, the highest performing model on ImageNet (MPL) is not the best performing model on VOC.
Revised: In Fig.7 we ﬁnd that, surprisingly, the highest performing model on ImageNet (MPL) is not the best performing model on VOC.

Original: We also see that the contrastive model CLIP-S deviates from the linear trend described in Kornblith et al.
Revised: We also see that the contrastive model CLIP-S deviates from the linear trend described in Kornblith et al.

Revised: This is surprising since this linear trend was previously reported with really high correlation.

Original: Indeed, when compared with the other models, CLIP-S yields the most prediction diversity on ImageNet.
Revised: Indeed, when compared with the other models, CLIP-S yields the most prediction diversity on ImageNet.

Original: We additionally test how the combined features perform by concatenating them pairwise and performing linear evaluation on top of these feature combinations – known as Stacked Generalization (Wolpert, 1992).
Revised: We additionally test how the combined features perform by concatenating them pairwise and performing linear evaluation on top of these feature combinations – known as Stacked Generalization (Wolpert, 1992).

Original: The highest performing model combinations (on VOC) are ones which combine differently trained models.
Revised: The highest performing model combinations (on VOC) are ones which combine differently trained models.

Original: Further, the best combinations do not even include MPL, which indicates that diversity can be better indicator of downstream performance than any single model’s accuracy.
Revised: Further, the best combinations do not even include MPL, which indicates that diversity can be better indicator of downstream performance than any single model’s accuracy.

Original: We posit that the reason diversely-trained model combinations yield highest performance on this downstream task is due to the diversity of features learned, which provide higher coverage of features captured that explain the data.
Revised: We posit that the reason diversely-trained model combinations yield highest performance on this downstream task is due to the diversity of features learned, which provide higher coverage of features captured that explain the data.

Original: This allows for better feature reuse, which is crucial for transfer learning (Neyshabur et al., 2020).
Revised: This allows for better feature reuse, which is crucial for transfer learning (Neyshabur et al., 2020).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have shown that diverse training methodologies, in particular, training with diverse optimization objectives on different large-scale datasets, can produce models that generate uncorrelated errors.
Revised: We have shown that diverse training methodologies, in particular, training with diverse optimization objectives on different large-scale datasets, can produce models that generate uncorrelated errors.

Original: These models ensemble more efﬁciently, attaining higher ensembling accuracy, since their different training setups allows them to specialize to different subdomains of the data.
Revised: These models ensemble more efﬁciently, attaining higher ensembling accuracy, since their different training setups allows them to specialize to different subdomains of the data.

Original: Due to this specialization, different-enough models can be useful for achieving high accuracies, even if they display low accuracies individually.
Revised: Due to this specialization, different-enough models can be useful for achieving high accuracies, even if they display low accuracies individually.

Original: Finally, we have also shown that they learn overlapping (but not supersetting) features, and that combining their embeddings can boost downstream performance.
Revised: Finally, we have also shown that they learn overlapping (but not supersetting) features, and that combining their embeddings can boost downstream performance.

Original: The importance of behavior diversity has been highlighted in different ﬁelds.
Revised: The importance of behavior diversity has been highlighted in different ﬁelds.

Original: In signal processing, sensor fusion – the combination of signals from multiple sensors into a more accurate estimation – relies on these sensors making independent errors.
Revised: In signal processing, sensor fusion – the combination of signals from multiple sensors into a more accurate estimation – relies on these sensors making independent errors.

Original: In deep reinforcement learning, it has been hypothesized that maximizing the coverage of possible behaviors of an agent may help it acquire the skills that are useful (Eysenbach et al., 2018).
Revised: In deep reinforcement learning, it has been hypothesized that maximizing the coverage of possible behaviors of an agent may help it acquire the skills that are useful (Eysenbach et al., 2018).

Original: In image classiﬁcation, the model is our agent, its predictions the behavior, and ensembling our sensor fusion.
Revised: In image classiﬁcation, the model is our agent, its predictions the behavior, and ensembling our sensor fusion.

Original: Our work demonstrates that this diversity of features is possible, and highlights a key question: why didn’t SGD learn it?
Revised: Our work demonstrates that this diversity of features is possible, and highlights a key question: why didn’t SGD learn it?

Original: Perhaps there exists an objective that can produce a single best/supersetting representation but, as we’ve shown, none of our existing training methodologies have found it.
Revised: Perhaps there exists an objective that can produce a single best/supersetting representation but, as we’ve shown, none of our existing training methodologies have found it.

Original: If we look closely, our results may also provide clues for why this is the case.
Revised: If we look closely, our results may also provide clues for why this is the case.

Original: To ﬁnd models with categorically different generalization behavior, we had to train them with objectives that do not directly maximize the classiﬁcation accuracy we are ultimately interested in.
Revised: To ﬁnd models with categorically different generalization behavior, we had to train them with objectives that do not directly maximize the classiﬁcation accuracy we are ultimately interested in.

Original: In our collective search for novelty, we have stumbled upon the decade-old ﬁnding that some problems are best solved by methods that ignore the objective, since “the objective function does not necessarily reward the stepping stones in the search space that ultimately lead to the objective” (Lehman and Stanley, 2008).
Revised: In our collective search for novelty, we have stumbled upon the decade-old ﬁnding that some problems are best solved by methods that ignore the objective, since “the objective function does not necessarily reward the stepping stones in the search space that ultimately lead to the objective” (Lehman and Stanley, 2008).

Original: Together, our results provide encouragement for machine learning practitioners to continue creating methodologies to train high-performing models, including new frameworks & datasets, in order to expand the types of features our models learn, behaviors they exhibit, and novel ways to train them. 
Revised: Together, our results provide encouragement for machine learning practitioners to continue creating methodologies to train high-performing models, including new frameworks & datasets, in order to expand the types of features our models learn, behaviors they exhibit, and novel ways to train them. A CKNOWLEDGMENTS

Revised: We would like to thank the following people for their early feedback, thoughtful discussions and help over the course of this project: Becca Roelofs, Ben Poole, Simon Kornblith, Niki Parmar, Ben

Revised: Caine, Keren Gu, Ethan Dyer, Anders Andreassen, Katherine Heller, Chiyuan Zhang, Jon Shlens,

Revised: Tyler Scott, Gamaleldin Elsayed, Rosanne Liu, Dan Hendrycks, Samy Bengio, Hieu Pham, Thomas

Revised: Unterthiner, Chao Jia, Mostafa Dehghani, Neil Houlsby, Lucas Beyer, Josip Djolonga, Rodolphe

Original:  A A PPENDIX A.M ODEL C ATEGORIZATION
Revised: Jenatton, Benham Neyshabur, Alec Radford and Sam Altman. A A PPENDIX A. C ONCURRENT WORK Many concurrent and recent papers have found related conclusions to ours. Here, we highlight a few of them.

Revised: Abnar et al.

Revised: (2021) found that when training on the same objective, as we increase the upstream accuracy, the performance of downstream tasks saturates.

Revised: This is in line with our results in Sec. 4.7, that optimizing for a single objective might not capture diverse enough features for downstream performance.

Revised: D’Amour et al.

Revised: (2020) show that, as a result of underspeciﬁcation in training, models are treated as equivalent based on their training domain performance, but can behave very differently in deployment domains.

Revised: Additionally, other recent work also ﬁnds that optimizing for diversity is beneﬁcial:

Revised: Roy et al.

Revised: (2022) found that using a diverse ensemble (mixing losses and datasets) improves OOD performance in dermatology tasks.

Revised: Sinha et al.

Revised: (2020) explicitly optimizes for diversity of predictions using an adversarial loss, leading to improved OOD performance.

Revised: A. M ODEL C ATEGORIZATION

Original: In some sense, model categories are supersets of one another: when we change a model architecture, we also change the hyper-parameters used to train such architecture, to make sure that they are optimal for training this new setting.
Revised: In some sense, model categories are supersets of one another: when we change a model architecture, we also change the hyper-parameters used to train such architecture, to make sure that they are optimal for training this new setting.

Original: In a similar fashion, when we change the training framework (supervised to contrastive), not only do we change hyper parameters, but the architecture also changes to best suit the new learning framework.
Revised: In a similar fashion, when we change the training framework (supervised to contrastive), not only do we change hyper parameters, but the architecture also changes to best suit the new learning framework.

Original: Finally, when we change dataset scales (e.g.: ImageNet →
Revised: Finally, when we change dataset scales (e.g.: ImageNet →

Original: WIT), we use the framework, architecture, and hyper parameters that allow the best performance possible on the new dataset.
Revised: WIT), we use the framework, architecture, and hyper parameters that allow the best performance possible on the new dataset.

Original: A. M ODEL L IST P ER C ATEGORY
Revised: A. M ODEL L IST P ER C ATEGORY

Original: In Table 1, we list all models we used in our main analysis, Figs. 1, 2, 3, 4, 6, along with their training methodologies, calibration temperatures, individual ImageNet accuracy, error inconsistency (relative to our base model ResNet-50), and ensemble accuracy (with our base model ResNet-50).
Revised: In Table 1, we list all models we used in our main analysis, Figs. 1, 2, 3, 4, 6, along with their training methodologies, calibration temperatures, individual ImageNet accuracy, error inconsistency (relative to our base model ResNet-50), and ensemble accuracy (with our base model ResNet-50).

Original: We selected these models by controlling for their individual accuracy, which helps guarantee our analysis concerns training methodology, not any individual model being inherently better.
Revised: We selected these models by controlling for their individual accuracy, which helps guarantee our analysis concerns training methodology, not any individual model being inherently better.

Original: In table 2, we list the high-accuracy models used in Figs.
Revised: In table 2, we list the high-accuracy models used in Figs.

Original: 5 and 7.
Revised: 5 and 7.

Original: These models are higher accuracy, and provide a stronger base for ensembling.
Revised: These models are higher accuracy, and provide a stronger base for ensembling.

Original: Table 3 lists low accuracy models, but which are trained with very different methodologies (relative to the typical model).
Revised: Table 3 lists low accuracy models, but which are trained with very different methodologies (relative to the typical model).

Original: They are therefore still useful (as we show), and were used in Sec.
Revised: They are therefore still useful (as we show), and were used in Sec.

Original: 4.5 and Fig.
Revised: 4.5 and Fig.

Original: Finally, Table 4 lists other models that we trained and analyzed, but which did not reach our target accuracy range 74-78%, so were not included in the main paper analysis.
Revised: Finally, Table 4 lists other models that we trained and analyzed, but which did not reach our target accuracy range 74-78%, so were not included in the main paper analysis.

Original: A. A DDITIONAL E RROR A NALYSIS
Revised: A. A DDITIONAL E RROR A NALYSIS

Original: In Fig.
Revised: In Fig.

Original: 8, we reproduce Fig.
Revised: 8, we reproduce Fig.

Original: 1 (left), and additionally show how the number of examples where both models predict correct (center), as well as examples where both models predict incorrectly (right), both decrease.
Revised: 1 (left), and additionally show how the number of examples where both models predict correct (center), as well as examples where both models predict incorrectly (right), both decrease.

Original: The added prediction diversity that comes with ensembling models trained differently will only be beneﬁcial if the ensemble can efﬁciently [Figure] [Table]
Revised: The added prediction diversity that comes with ensembling models trained differently will only be beneﬁcial if the ensemble can efﬁciently convert the examples on the left, to compensate for the decreased number of examples in the center.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: A. L-BFGS DETAILS
Revised: A. L-BFGS DETAILS

Original: In order to train the linear classiﬁers, we use L-BFGS in the same setup/implementation described in Kornblith et al.
Revised: In order to train the linear classiﬁers, we use L-BFGS in the same setup/implementation described in Kornblith et al.

Original: We train the linear heads on top of the pre-logit representations without augmentation.
Revised: We train the linear heads on top of the pre-logit representations without augmentation.

Original: We ﬁnd it useful to normalize each representation before any operations (concatenation, subsampling, etc).
Revised: We ﬁnd it useful to normalize each representation before any operations (concatenation, subsampling, etc).

Original: In the ﬁrst part of Section 4.6, the portions of each representation are picked randomly (e.g.: random 25% dimensions of ResNet, and random 75% dimensions of CLIP), and then concatenated.
Revised: In the ﬁrst part of Section 4.6, the portions of each representation are picked randomly (e.g.: random 25% dimensions of ResNet, and random 75% dimensions of CLIP), and then [Table] [Table] [Table]

Original:  In the second part, the two representations are ﬁrst concatenated then dimensions are selected based on the diversity criteria.
Revised: [Table] [Table] concatenated. In the second part, the two representations are ﬁrst concatenated then dimensions are selected based on the diversity criteria

Original:  A. E XTRA FIGURES FOR REVIEWERS
Revised: . A. E XTRA FIGURES FOR REVIEWERS

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]

Original: suspect 
Revised: suspect this is because

Original: [Figure]
Revised: [Figure]