Original: T HE B OOTSTRAP F RAMEWORK : G ENERALIZATION T HROUGH THE L ENS OF O NLINE O PTIMIZATION
Revised: T HE B OOTSTRAP F RAMEWORK : G ENERALIZATION T HROUGH THE L ENS OF O NLINE O PTIMIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose a new framework for reasoning about generalization in deep learning.
Revised: We propose a new framework for reasoning about generalization in deep learning.

Original: The core idea is to couple the Real World, where optimizers take stochastic gradient steps on the empirical loss, to an Ideal World, where optimizers take steps on the population loss.
Revised: The core idea is to couple the Real World, where optimizers take stochastic gradient steps on the empirical loss, to an Ideal World, where optimizers take steps on the population loss.

Original: This leads to an alternate decomposition of test error into: (1) the Ideal World test error plus (2) the gap between the two worlds.
Revised: This leads to an alternate decomposition of test error into: (1) the Ideal World test error plus (2) the gap between the two worlds.

Original: If the gap (2) is universally small, this reduces the problem of generalization in ofﬂine learning to the problem of optimization in online learning.
Revised: If the gap (2) is universally small, this reduces the problem of generalization in ofﬂine learning to the problem of optimization in online learning.

Original: We then give empirical evidence that this gap between worlds can be small in realistic deep learning settings, in particular supervised image classiﬁcation.
Revised: We then give empirical evidence that this gap between worlds can be small in realistic deep learning settings, in particular supervised image classiﬁcation.

Original: For example, CNNs generalize better than MLPs on image distributions in the Real World, but this is “because” they optimize faster on the population loss in the Ideal World.
Revised: For example, CNNs generalize better than MLPs on image distributions in the Real World, but this is “because” they optimize faster on the population loss in the Ideal World.

Original: This suggests our framework is a useful tool for understanding generalization in deep learning, and lays a foundation for future research in the area.
Revised: This suggests our framework is a useful tool for understanding generalization in deep learning, and lays a foundation for future research in the area.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The goal of a generalization theory in supervised learning is to understand when and why trained models have small test error.
Revised: The goal of a generalization theory in supervised learning is to understand when and why trained models have small test error.

Original: The classical framework of generalization decomposes the test error of a model f t as:
Revised: The classical framework of generalization decomposes the test error of a model f t as:

Original: [Equation]
Revised: [Equation]

Original: and studies each part separately (e.g. Vapnik & Chervonenkis (1971);
Revised: and studies each part separately (e.g. Vapnik & Chervonenkis (1971);

Original: Blumer et al.
Revised: Blumer et al.

Original: ; ShalevShwartz & Ben-David (2014)).
Revised: ; ShalevShwartz & Ben-David (2014)).

Original: Many works have applied this framework to study generalization of deep networks (e.g. Bartlett (1997); Bartlett et al.
Revised: Many works have applied this framework to study generalization of deep networks (e.g. Bartlett (1997); Bartlett et al.

Original: ; Bartlett & Mendelson (2002); Anthony &
Revised: ; Bartlett & Mendelson (2002); Anthony &

Original: Bartlett (2009); Neyshabur et al. (2015b); Dziugaite & Roy (2017); Bartlett et al.
Revised: Bartlett (2009); Neyshabur et al. (2015b); Dziugaite & Roy (2017); Bartlett et al.

Original: ; Neyshabur et al.
Revised: ; Neyshabur et al.

Original: (2017); Harvey et al.
Revised: (2017); Harvey et al.

Original: (2017); Golowich et al.
Revised: (2017); Golowich et al.

Original: Arora et al.
Revised: Arora et al.

Original: (2018; 2019); Allen-Zhu et al.
Revised: (2018; 2019); Allen-Zhu et al.

Original: ; Long & Sedghi (2019); Wei & Ma (2019)).
Revised: ; Long & Sedghi (2019); Wei & Ma (2019)).

Original: However, there are at least two obstacles to understanding generalization of modern neural networks via the classical approach.
Revised: However, there are at least two obstacles to understanding generalization of modern neural networks via the classical approach.

Original: Modern methods can interpolate , reaching TrainError ≈ 0 , while still performing well.
Revised: Modern methods can interpolate , reaching TrainError ≈ 0 , while still performing well.

Original: In these settings, the decomposition of Equation (1) does not actually reduce test error into two easier subproblems.{+different subproblems: it amounts to writing TestError = 
 + TestError.+}
Revised: In these settings, the decomposition of Equation (1) does not actually reduce test error into two {+different subproblems: it amounts to writing TestError = 
 + TestError.+}

Original: That is understanding the generalization gap here is exactly as hard as understanding the test error itself.
Revised: That is, understanding the generalization gap here is exactly equivalent to understanding the test error itself.

Original: Most if not all techniques for understanding the generalization gap (e.g. uniform convergence, VC-dimension, regularization, stability, margins) remain vacuous (Zhang et al., 2017; Belkin et al., 2018a;b; Nagarajan & Kolter, 2019) and not predictive (Nagarajan & Kolter, 2019; Jiang et al., 2019) for overparameterized deep networks.
Revised: Most if not all techniques for understanding the generalization gap (e.g. uniform convergence, VC-dimension, regularization, stability, margins) remain vacuous (Zhang et al., 2017; Belkin et al., 2018a;b; Nagarajan & Kolter, 2019) and not predictive (Nagarajan & Kolter, 2019; Jiang et al., 2019) for overparameterized deep networks.

Original: In this work, we propose an alternate approach to understanding generalization to help overcome these obstacles.
Revised: In this work, we propose an alternate approach to understanding generalization to help overcome these obstacles.

Original: The key idea is to consider an alternate decomposition:
Revised: The key idea is to consider an alternate decomposition:

Original: [Equation]
Revised: [Equation]

Original: where f t is the neural-network after t optimization steps (the “Real World”), and f iid t is a network trained identically to f t , but using fresh samples from the distribution in each mini-batch step (the
Revised: where f t is the neural-network after t optimization steps (the “Real World”), and f iid t is a network trained identically to f t , but using fresh samples from the distribution in each mini-batch step (the

Original: “Ideal World”).
Revised: “Ideal World”).

Original: That is, f iid t is the result of optimizing on the population loss for t steps, while f t is the result of optimizing on the empirical loss as usual (we deﬁne this more formally later).
Revised: That is, f iid t is the result of optimizing on the population loss for t steps, while f t is the result of optimizing on the empirical loss as usual (we deﬁne this more formally later).

Original: This leads to a different decoupling of concerns, and proposes an alternate research agenda to understand generalization.
Revised: This leads to a different decoupling of concerns, and proposes an alternate research agenda to understand generalization.

Original: To understand generalization in the bootstrap framework, it is sufﬁcient to understand:
Revised: To understand generalization in the bootstrap framework, it is sufﬁcient to understand:

Original: (A) Online Learning: How quickly models optimize on the population loss, in the inﬁnite-data regime (the Ideal World).
Revised: (A) Online Learning: How quickly models optimize on the population loss, in the inﬁnite-data regime (the Ideal World).

Original: (B) Finite-Sample Deviations: How closely models behave in the ﬁnite-data vs. inﬁnite-data regime (the bootstrap error).
Revised: (B) Finite-Sample Deviations: How closely models behave in the ﬁnite-data vs. inﬁnite-data regime (the bootstrap error).

Original: Although neither of these points are theoretically understood for deep networks, they are closely related to rich areas in optimization and statistics, whose tools have not been brought fully to bear on the problem of generalization.
Revised: Although neither of these points are theoretically understood for deep networks, they are closely related to rich areas in optimization and statistics, whose tools have not been brought fully to bear on the problem of generalization.

Original: The ﬁrst part (A) is purely a question in online stochastic optimization: We have access to a stochastic gradient oracle for a population loss function, and we are interested in how quickly an online optimization
Revised: The ﬁrst part (A) is purely a question in online stochastic optimization: We have access to a stochastic gradient oracle for a population loss function, and we are interested in how quickly an online optimization

Original: algorithm
Revised: algorithm

Original: (e.g. SGD, Adam) reaches small population loss.
Revised: (e.g. SGD, Adam) reaches small population loss.

Original: This problem is well-studied in the online learning literature for convex functions (Bubeck, 2011; Hazan, 2019; Shalev-Shwartz et al., 2011), and is an active area of research in nonconvex settings (Jin et al., 2017; Lee et al., 2016; Jain & Kar, 2017; Gao et al., 2018; Yang et al., 2018; Maillard & Munos, 2010).
Revised: This problem is well-studied in the online learning literature for convex functions (Bubeck, 2011; Hazan, 2019; Shalev-Shwartz et al., 2011), and is an active area of research in nonconvex settings (Jin et al., 2017; Lee et al., 2016; Jain & Kar, 2017; Gao et al., 2018; Yang et al., 2018; Maillard & Munos, 2010).

Original: In the context of neural networks, optimization is usually studied on the empirical loss landscape (Arora et al., 2019; Allen-Zhu et al., 2019), but we propose studying optimization on the population loss landscape directly.
Revised: In the context of neural networks, optimization is usually studied on the empirical loss landscape (Arora et al., 2019; Allen-Zhu et al., 2019), but we propose studying optimization on the population loss landscape directly.

Original: This highlights a key difference in our approach: we never compare test and train quantities— we only consider test quantities.
Revised: This highlights a key difference in our approach: we never compare test and train quantities— we only consider test quantities.

Original: The second part (B) involves approximating fresh samples with “reused” samples, and reasoning about behavior of certain functions under this approximation.
Revised: The second part (B) involves approximating fresh samples with “reused” samples, and reasoning about behavior of certain functions under this approximation.

Original: This is closely related to the nonparametric bootstrap in statistics (Efron, 1979; Efron & Tibshirani, 1986), where sampling from the population distribution is approximated by sampling with replacement from an empirical distribution.
Revised: This is closely related to the nonparametric bootstrap in statistics (Efron, 1979; Efron & Tibshirani, 1986), where sampling from the population distribution is approximated by sampling with replacement from an empirical distribution.

Original: Bootstrapped estimators are widely used in applied statistics, and their theoretical properties are known in certain cases (e.g. Hastie et al.
Revised: Bootstrapped estimators are widely used in applied statistics, and their theoretical properties are known in certain cases (e.g. Hastie et al.

Original: ; James et al.
Revised: ; James et al.

Original: ; Efron & Hastie (2016);
Revised: ; Efron & Hastie (2016);

Original: Van der Vaart (2000)).
Revised: Van der Vaart (2000)).

Original: Although current bootstrap theory does not apply to neural networks, it is conceivable that these tools could eventually be extended to our setting.
Revised: Although current bootstrap theory does not apply to neural networks, it is conceivable that these tools could eventually be extended to our setting.

Original: Experimental Validation.
Revised: Experimental Validation.

Original: Beyond the theoretical motivation, our main experimental claim is that the bootstrap decomposition is actually useful: in realistic settings, the bootstrap error is often small, and the performance of real classiﬁers is largely captured by their performance in the Ideal World.
Revised: Beyond the theoretical motivation, our main experimental claim is that the bootstrap decomposition is actually useful: in realistic settings, the bootstrap error is often small, and the performance of real classiﬁers is largely captured by their performance in the Ideal World.

Original: Figure 1 shows one example of this, as a preview of our more extensive experiments in Section 4.
Revised: Figure 1 shows one example of this, as a preview of our more extensive experiments in Section 4.

Original: We plot the test error of a ResNet and an MLP on a CIFAR-10-like task, over increasing minibatch
Revised: We plot the test error of a ResNet and an MLP on a CIFAR-10-like task, over increasing minibatch

Original: SGD iterations.
Revised: SGD iterations.

Original: The Real World is trained on 50K samples for 100 epochs.
Revised: The Real World is trained on 50K samples for 100 epochs.

Original: The Ideal World is trained on 5 million samples with a single pass.
Revised: The Ideal World is trained on 5 million samples with a single pass.

Original: Notice that the bootstrap error is small for both architectures, although the generalization gap can be large.
Revised: Notice that the bootstrap error is small for both architectures, although the generalization gap can be large.

Original: In particular, the convnet generalizes better than the MLP on ﬁnite data, but this is “because” it optimizes faster on the population loss with inﬁnite data.
Revised: In particular, the convnet generalizes better than the MLP on ﬁnite data, but this is “because” it optimizes faster on the population loss with inﬁnite data.

Original: Our Contributions.
Revised: Our Contributions.

Original: • Framework: We propose the Bootstrap Framework for understanding generalization in deep learning, which connects ofﬂine generalization to online optimization.
Revised: • Framework: We propose the Bootstrap Framework for understanding generalization in deep learning, which connects ofﬂine generalization to online optimization.

Original: (Section 2).
Revised: (Section 2).

Original: • Validation: We give evidence that the bootstrap error is small in realistic settings for supervised image classiﬁcation, by conducting extensive experiments on large-scale tasks (including variants of CIFAR-10 and ImageNet) for many architectures (Section 4).
Revised: • Validation: We give evidence that the bootstrap error is small in realistic settings for supervised image classiﬁcation, by conducting extensive experiments on large-scale tasks (including variants of CIFAR-10 and ImageNet) for many architectures (Section 4).

Original: Thus, The generalization of models in ofﬂine learning is largely determined by their optimization speed in online learning.
Revised: Thus, The generalization of models in ofﬂine learning is largely determined by their optimization speed in online learning.

Original: • Implications: We highlight how our framework can unify and yield insight into important phenomena in deep learning, including implicit bias, model selection, data-augmentation and pretraining (Section 5).
Revised: • Implications: We highlight how our framework can unify and yield insight into important phenomena in deep learning, including implicit bias, model selection, data-augmentation and pretraining (Section 5).

Original: For example, we observe a surprising phenomena in practice, which is captured by our framework: The same techniques (architectures and training methods) are used in practice in both over- and under-parameterized regimes.
Revised: For example, we observe a surprising phenomena in practice, which is captured by our framework: The same techniques (architectures and training methods) are used in practice in both over- and under-parameterized regimes.

Original: Additional Related Work.
Revised: Additional Related Work.

Original: The bootstrap error is also related to algorithmic stability (e.g. Bousquet & Elisseeff (2001);
Revised: The bootstrap error is also related to algorithmic stability (e.g. Bousquet & Elisseeff (2001);

Original: Hardt et al.
Revised: Hardt et al.

Original: (2016)), since both quantities involve replacing samples with fresh samples.
Revised: (2016)), since both quantities involve replacing samples with fresh samples.

Original: However, stability-based generalization bounds cannot tightly bound the bootstrap error, since there are many settings where the generalization gap is high, but bootstrap error is low.
Revised: However, stability-based generalization bounds cannot tightly bound the bootstrap error, since there are many settings where the generalization gap is high, but bootstrap error is low.

Original:  T HE B OOSTRAP F RAMEWORK
Revised:  T HE B OOSTRAP F RAMEWORK

Original: Here we more formally describe the bootstrap framework and our main claims.
Revised: Here we more formally describe the bootstrap framework and our main claims.

Original: Let F denote a learning algorithm, including architecture and optimizer.
Revised: Let F denote a learning algorithm, including architecture and optimizer.

Original: We consider optimizers which can be used in online learning, such as stochastic gradient descent and variants.
Revised: We consider optimizers which can be used in online learning, such as stochastic gradient descent and variants.

Original: Let Train F ( D , n, t ) denote training in the “Real World”: using the architecture and optimizer speciﬁed by F , on a train set of n samples from distribution D , for t optimizer steps.
Revised: Let Train F ( D , n, t ) denote training in the “Real World”: using the architecture and optimizer speciﬁed by F , on a train set of n samples from distribution D , for t optimizer steps.

Original: Let Train F
Revised: Let Train F

Original: ( D , ∞ , t ) denote this same optimizer operating on the population loss (the “Ideal World”).
Revised: ( D , ∞ , t ) denote this same optimizer operating on the population loss (the “Ideal World”).

Original: Note that these two procedures use identical architectures, learning-rate schedules, mini-batch size, etc – the only difference is, the Ideal World optimizer sees a fresh minibatch of samples in each optimization step, while the Real World reuses samples in minibatches.
Revised: Note that these two procedures use identical architectures, learning-rate schedules, mini-batch size, etc – the only difference is, the Ideal World optimizer sees a fresh minibatch of samples in each optimization step, while the Real World reuses samples in minibatches.

Original: Let the Real and Ideal World trained models be:
Revised: Let the Real and Ideal World trained models be:

Original: [Equation]
Revised: [Equation]

Original: We now claim that for all t until the Real World converges, these two models f t , f iid t have similar test performance.
Revised: We now claim that for all t until the Real World converges, these two models f t , f iid t have similar test performance.

Original: In our main claims, we differ slightly from the presentation in the Introduction in that we consider the “soft-error” of classiﬁers instead of their hard-errors.
Revised: In our main claims, we differ slightly from the presentation in the Introduction in that we consider the “soft-error” of classiﬁers instead of their hard-errors.

Original: The soft-accuracy of classiﬁers is deﬁned as the softmax probability on the correct label, and ( soft-error ) :
Revised: The soft-accuracy of classiﬁers is deﬁned as the softmax probability on the correct label, and ( soft-error ) :

Original: = 1 − ( soft-accuracy ) .
Revised: = 1 − ( soft-accuracy ) .

Original: Equivalently, this is the expected error oftemperature-1 samples from the softmax distribution.
Revised: Formally, let p ( x ) be the softmax distribution of the classiﬁer on input x , as a distribution over the label space Y . Then the soft-error is E x,y ∼D [1 − p ( x ) y ] .

Original: Formally, deﬁne ε as the bootstrap error – the gap in soft-error between Real and Ideal worlds at time t :
Revised: Now deﬁne ε as the bootstrap error – the gap in soft-error between Real and Ideal worlds at time t :

Original: [Equation]
Revised: [Equation]

Original: Our main experimental claim is that the bootstrap error ε is uniformly small in realistic settings.
Revised: Our main experimental claim is that the bootstrap error ε is uniformly small in realistic settings.

Original: Claim 1 (Bootstrap Approximation, informal) For choices of ( n, D , F ) corresponding to realistic settings in deep learning for supervised image classiﬁcation, the bootstrap error ε ( n, D , F , t ) is small for all t ≤ T 0 .
Revised: Claim 1 (Bootstrap Approximation, informal) For choices of ( n, D , F ) corresponding to realistic settings in deep learning for supervised image classiﬁcation, the bootstrap error ε ( n, D , F , t ) is small for all t ≤ T 0 .

Original: The “stopping time” T 0 is deﬁned as the time when the Real World reaches small training error (we use 1%) – that is, when Real World training has essentially converged.
Revised: The “stopping time” T 0 is deﬁned as the time when the Real World reaches small training error (we use 1%) – that is, when Real World training has essentially converged.

Original: The restriction on t ≤ T 0 is necessary, since as t → ∞ the Ideal World will continue to improve, but the Real World will at some point essentially stop changing (when train error ≈ 0 ).
Revised: The restriction on t ≤ T 0 is necessary, since as t → ∞ the Ideal World will continue to improve, but the Real World will at some point essentially stop changing (when train error ≈ 0 ).

Original: However, we claim that these worlds are close for “as long as we can hope”— as long as the Real World optimizer is still moving.
Revised: However, we claim that these worlds are close for “as long as we can hope”— as long as the Real World optimizer is still moving.

Original: Error vs. Soft-Error.
Revised: Error vs. Soft-Error.

Original: We chose to measure soft-error instead of hard-error in our framework for both empirical and theoretically-motivated reasons.
Revised: We chose to measure soft-error instead of hard-error in our framework for both empirical and theoretically-motivated reasons.

Original: Empirically, we found that the bootstrap gap is often smaller with respect to soft-errors.
Revised: Empirically, we found that the bootstrap gap is often smaller with respect to soft-errors.

Original: Theoretically, we want to deﬁne the bootstrap gap such that it converges to 0 as data and model size are scaled to inﬁnity.
Revised: Theoretically, we want to deﬁne the bootstrap gap such that it converges to 0 as data and model size are scaled to inﬁnity.

Original: Speciﬁcally, if we consider an overparameterized scaling limit where the Real World models always interpolate the train data, then
Revised: Speciﬁcally, if we consider an overparameterized scaling limit where the Real World models always interpolate the train data, then

Original: Distributional Generalization (Nakkiran & Bansal, 2020) implies that the bootstrap gap for test error will not converge to 0 on distributions with non-zero Bayes risk.
Revised: Distributional Generalization (Nakkiran & Bansal, 2020) implies that the bootstrap gap for test error will not converge to 0 on distributions with non-zero Bayes risk.

Original: Roughly, this is because the Ideal
Revised: Roughly, this is because the Ideal

Original: World classiﬁer will converge to the Bayes optimal one ( argmax y p ( y | x ) ), while the Real World interpolating classiﬁer will converge to a sampler from p ( y | x ) .
Revised: World classiﬁer will converge to the Bayes optimal one ( argmax y p ( y | x ) ), while the Real World interpolating classiﬁer will converge to a sampler from p ( y | x ) .

Original: Considering soft-errors instead of errors nulliﬁes this issue.
Revised: Considering soft-errors instead of errors nulliﬁes this issue.

Original: We elaborate further on the differences between the worlds in Section 6.
Revised: We elaborate further on the differences between the worlds in Section 6.

Original:  E XPERIMENTAL S ETUP
Revised:  E XPERIMENTAL S ETUP

Original: Our bootstrap framework could apply to any setting where an iterative optimizer for online learning is applied in an ofﬂine setting.
Revised: Our bootstrap framework could apply to any setting where an iterative optimizer for online learning is applied in an ofﬂine setting.

Original: In this work we primarily consider stochastic gradient descent (SGD) applied to deep neural networks for image classiﬁcation.
Revised: In this work we primarily consider stochastic gradient descent (SGD) applied to deep neural networks for image classiﬁcation.

Original: This setting is well-developed both in practice and in theory, and thus serves as an appropriate ﬁrst step to vet theories of generalization, as done in many recent works (e.g. Jiang et al.
Revised: This setting is well-developed both in practice and in theory, and thus serves as an appropriate ﬁrst step to vet theories of generalization, as done in many recent works (e.g. Jiang et al.

Original: ; Neyshabur et al.
Revised: ; Neyshabur et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: Arora et al.
Revised: Arora et al.

Original: Our work does not depend on overparameterization— it holds for both under and over parameterized networks, though it is perhaps most interesting in the overparameterized setting.
Revised: Our work does not depend on overparameterization— it holds for both under and over parameterized networks, though it is perhaps most interesting in the overparameterized setting.

Original: We now describe our datasets and experimental methodology.
Revised: We now describe our datasets and experimental methodology.

Original: 3. D ATASETS
Revised: 3. D ATASETS

Original: Measuring the bootstrap error in realistic settings presents some challenges, since we do not have enough samples to instantiate the Ideal World.
Revised: Measuring the bootstrap error in realistic settings presents some challenges, since we do not have enough samples to instantiate the Ideal World.

Original: For example, for a Real World CIFAR-10 network trained on 50K samples for 100 epochs, the corresponding Ideal World training would requiremillion samples (fresh samples in each epoch).
Revised: For example, for a Real World CIFAR-10 network trained on 50K samples for 100 epochs, the corresponding Ideal World training would requiremillion samples (fresh samples in each epoch).

Original: Since we do not have 5 million samples for CIFAR10, we use the following datasets as proxies.
Revised: Since we do not have 5 million samples for CIFAR10, we use the following datasets as proxies.

Original: More details, including sample images, in Appendix D.
Revised: More details, including sample images, in Appendix D.

Original: CIFAR-5m. We construct a dataset of 6 million synthetic CIFAR-10-like images, by sampling from the CIFAR-10 Denoising Diffusion generative model of Ho et al.
Revised: CIFAR-5m. We construct a dataset of 6 million synthetic CIFAR-10-like images, by sampling from the CIFAR-10 Denoising Diffusion generative model of Ho et al.

Original: (2020), and labeling the unconditional samples by a 98.5% accurate Big-Transfer model (Kolesnikov et al., 2019).
Revised: (2020), and labeling the unconditional samples by a 98.5% accurate Big-Transfer model (Kolesnikov et al., 2019).

Original: These are synthetic images, but close to CIFAR-10 for research purposes.
Revised: These are synthetic images, but close to CIFAR-10 for research purposes.

Original: For example, a ResNet-18 trained on 50K samples from CIFAR-5m reaches 92% test accuracy on CIFAR-10 test set.
Revised: For example, a ResNet-18 trained on 50K samples from CIFAR-5m reaches 92% test accuracy on CIFAR-10 test set.

Original: We use 5 million images for training, and reserve the rest for the test set.
Revised: We use 5 million images for training, and reserve the rest for the test set.

Original: We release this dataset for public use.
Revised: We release this dataset for public use.

Original: ImageNet-DogBird.
Revised: ImageNet-DogBird.

Original: To test our framework in more complex settings, with real images, we construct a distribution based on ImageNet ILSVRC-2012 (Russakovsky et al., 2015).
Revised: To test our framework in more complex settings, with real images, we construct a distribution based on ImageNet ILSVRC-2012 (Russakovsky et al., 2015).

Original: Recall that we need a setting with a large number of samples relative to the difﬁculty of the task: if the Real World performs well with few samples and few epochs, then we can simulate it in the Ideal World.
Revised: Recall that we need a setting with a large number of samples relative to the difﬁculty of the task: if the Real World performs well with few samples and few epochs, then we can simulate it in the Ideal World.

Original: Thus, we construct a simpler binary classiﬁcation task out of ImageNet by collapsing classes into the superclasses “hunting dog” and “bird.”
Revised: Thus, we construct a simpler binary classiﬁcation task out of ImageNet by collapsing classes into the superclasses “hunting dog” and “bird.”

Original: This is a roughly balanced task with 155K total images.
Revised: This is a roughly balanced task with 155K total images.

Original: 3. M ETHODOLOGY
Revised: 3. M ETHODOLOGY

Original: For experiments on CIFAR-5m, we exactly simulate the Real and Ideal worlds as described in Section 2.
Revised: For experiments on CIFAR-5m, we exactly simulate the Real and Ideal worlds as described in Section 2.

Original: That is, for every Real World architecture and optimizer we consider, we construct the corresponding Ideal World by executing the exact same training code, but using fresh samples in each epoch.
Revised: That is, for every Real World architecture and optimizer we consider, we construct the corresponding Ideal World by executing the exact same training code, but using fresh samples in each epoch.

Original: The rest of the training procedure remains identical, including data-augmentation and learning-rate schedule.
Revised: The rest of the training procedure remains identical, including data-augmentation and learning-rate schedule.

Original: For experiments on ImageNet-DogBird, we do not have enough samples to exactly simulate the Ideal World.
Revised: For experiments on ImageNet-DogBird, we do not have enough samples to exactly simulate the Ideal World.

Original: Instead, we approximate the Ideal World by using the full training set ( N = 155 K ) and data-augmentation.
Revised: Instead, we approximate the Ideal World by using the full training set ( N = 155 K ) and data-augmentation.

Original: Formally, this corresponds to approximating
Revised: Formally, this corresponds to approximating

Original: Train F ( D , ∞ , t ) by Train F ( D , 155 K, t ) .
Revised: Train F ( D , ∞ , t ) by Train F ( D , 155 K, t ) .

Original: In practice, we train the Real World on n = 10 K samples for 120 epochs, so we can approximate this with less than 8 epochs on the full 155 K train set.
Revised: In practice, we train the Real World on n = 10 K samples for 120 epochs, so we can approximate this with less than 8 epochs on the full 155 K train set.

Original: Since we train with data augmentation (crop+resize+ﬂip), each of the 8 repetitions of each sample will undergo different random augmentations, and thus this plausibly approximates fresh samples.
Revised: Since we train with data augmentation (crop+resize+ﬂip), each of the 8 repetitions of each sample will undergo different random augmentations, and thus this plausibly approximates fresh samples.

Original: [Figure]
Revised: [Figure]

Original: Stopping time.
Revised: Stopping time.

Original: We stop both Real and Ideal World training when the Real World reaches a small value of train error (which we set as 1% in all experiments).
Revised: We stop both Real and Ideal World training when the Real World reaches a small value of train error (which we set as 1% in all experiments).

Original: This stopping condition is necessary, as described in Section 2.
Revised: This stopping condition is necessary, as described in Section 2.

Original: Thus, for experiments which report test performance “at the end of training”, this refers to either when the target number of epochs is reached, or when Real World training has converged ( < 1% train error).
Revised: Thus, for experiments which report test performance “at the end of training”, this refers to either when the target number of epochs is reached, or when Real World training has converged ( < 1% train error).

Original: We always compare Real and Ideal Worlds after the exact same number of train iterations.
Revised: We always compare Real and Ideal Worlds after the exact same number of train iterations.

Original:  M AIN E XPERIMENTS
Revised:  M AIN E XPERIMENTS

Original: We now give evidence to support our main experimental claim, that the bootstrap error ε is often small for realistic settings in deep learning for image classiﬁcation.
Revised: We now give evidence to support our main experimental claim, that the bootstrap error ε is often small for realistic settings in deep learning for image classiﬁcation.

Original: In all experiments in this section, we instantiate the same model and training procedure in the Real and Ideal Worlds, and observe that the test soft-error is close at the end of training.
Revised: In all experiments in this section, we instantiate the same model and training procedure in the Real and Ideal Worlds, and observe that the test soft-error is close at the end of training.

Original: Full experimental details are in Appendix C.2.
Revised: Full experimental details are in Appendix C.2.

Original: CIFAR-5m. In Figure 2a we consider a variety of standard architectures on CIFAR-5m, from fullyconnected nets to modern convnets.
Revised: CIFAR-5m. In Figure 2a we consider a variety of standard architectures on CIFAR-5m, from fullyconnected nets to modern convnets.

Original: In the Real World, we train these architectures with SGD on n = 50 K samples from CIFAR-5m, for 100 total epochs.
Revised: In the Real World, we train these architectures with SGD on n = 50 K samples from CIFAR-5m, for 100 total epochs.

Original: We then construct the corresponding Ideal
Revised: We then construct the corresponding Ideal

Original: Worlds for each architecture, trained in the same way with fresh samples each epoch.
Revised: Worlds for each architecture, trained in the same way with fresh samples each epoch.

Original: Figure 2a shows the test soft-error of the trained classiﬁers in the Real and Ideal Worlds at the end of training.
Revised: Figure 2a shows the test soft-error of the trained classiﬁers in the Real and Ideal Worlds at the end of training.

Original: Observe that test performance is very close in Real and Ideal worlds, although the Ideal World sees 100 × unique samples during training.
Revised: Observe that test performance is very close in Real and Ideal worlds, although the Ideal World sees 100 × unique samples during training.

Original: To test our framework for more diverse architectures, we also sample 500 random architectures from the DARTS search space (Liu et al., 2019).
Revised: To test our framework for more diverse architectures, we also sample 500 random architectures from the DARTS search space (Liu et al., 2019).

Original: These are deep convnets of varying width and depth, which range in size from 70k to 5.5 million parameters.
Revised: These are deep convnets of varying width and depth, which range in size from 70k to 5.5 million parameters.

Original: Figure 2b shows the Real and Ideal World test performance at the end of training— these are often within 3% .
Revised: Figure 2b shows the Real and Ideal World test performance at the end of training— these are often within 3% .

Original: ImageNet: DogBird.
Revised: ImageNet: DogBird.

Original: We now test various ImageNet architectures on ImageNet-DogBird.
Revised: We now test various ImageNet architectures on ImageNet-DogBird.

Original: The Real
Revised: The Real

Original: World models are trained with SGD on n = 10 K samples with standard ImageNet data augmentation.
Revised: World models are trained with SGD on n = 10 K samples with standard ImageNet data augmentation.

Original: We approximate the Ideal World by training on 155 K samples as described above.
Revised: We approximate the Ideal World by training on 155 K samples as described above.

Original: Figure 3a plots the Real vs. Ideal World test error at the end of training, for various architectures.
Revised: Figure 3a plots the Real vs. Ideal World test error at the end of training, for various architectures.

Original: Figure 3b shows this for ResNet-18s of varying widths.
Revised: Figure 3b shows this for ResNet-18s of varying widths.

Original:  D EEP P HENOMENA THROUGH THE B OOTSTRAP L ENS
Revised:  D EEP P HENOMENA THROUGH THE B OOTSTRAP L ENS

Original: Here we show that our bootstrap framework can be insightful to study phenomena and design choices in deep learning.
Revised: Here we show that our bootstrap framework can be insightful to study phenomena and design choices in deep learning.

Original: For example, many effects in the Real World can be seen through their corresponding effects in the Ideal World.
Revised: For example, many effects in the Real World can be seen through their corresponding effects in the Ideal World.

Original: Full details for experiments provided in Appendix C.
Revised: Full details for experiments provided in Appendix C.

Original: Model Selection in the Over- and Under-parameterized Regimes.
Revised: Model Selection in the Over- and Under-parameterized Regimes.

Original: Much of theoretical work in deep learning focuses on overparameterized networks, which are large enough to ﬁt their train sets.
Revised: Much of theoretical work in deep learning focuses on overparameterized networks, which are large enough to ﬁt their train sets.

Original: [Figure]
Revised: [Figure]

Original: However, in modern practice, state-of-the-art networks can be either over or under -parameterized, depending on the scale of data.
Revised: However, in modern practice, state-of-the-art networks can be either over or under -parameterized, depending on the scale of data.

Original: For example, SOTA models on 300 million JFT images or 1 billion
Revised: For example, SOTA models on 300 million JFT images or 1 billion

Original: Instagram images are underﬁtting, due to the massive size of the train set (Sun et al., 2017; Mahajan et al., 2018).
Revised: Instagram images are underﬁtting, due to the massive size of the train set (Sun et al., 2017; Mahajan et al., 2018).

Original: In NLP, modern models such as GPT-3 and T5 are trained on massive internet-text datasets, and so are solidly in the underparameterized regime (Kaplan et al., 2020;
Revised: In NLP, modern models such as GPT-3 and T5 are trained on massive internet-text datasets, and so are solidly in the underparameterized regime (Kaplan et al., 2020;

Original: Brown et al., 2020; Raffel et al., 2019).
Revised: Brown et al., 2020; Raffel et al., 2019).

Original: We highlight one surprising aspect of this situation:
Revised: We highlight one surprising aspect of this situation:

Original: The same techniques (architectures and training methods) are used in practice in both over
Revised: The same techniques (architectures and training methods) are used in practice in both over

Original: - and
Revised: - and

Original: under
Revised: under

Original: -parameterized
Revised: -parameterized

Original: regimes.
Revised: regimes.

Original: For example, ResNet-101 is competitive both on 1 billion images of Instagram (when it is underparameterized) and on 50k images of CIFAR-10 (when it is overparameterized).
Revised: For example, ResNet-101 is competitive both on 1 billion images of Instagram (when it is underparameterized) and on 50k images of CIFAR-10 (when it is overparameterized).

Original: This observation was made recently in Bornschein et al.
Revised: This observation was made recently in Bornschein et al.

Original: (2020) for overparameterized architectures,
Revised: (2020) for overparameterized architectures,

Original: and we observe it extends to the underparameterized regime.
Revised: and we observe it extends to the underparameterized regime.

Original: It is apriori surprising that the same architectures do well in both over and underparameterized regimes, since there are very different considerations in each regime.
Revised: It is apriori surprising that the same architectures do well in both over and underparameterized regimes, since there are very different considerations in each regime.

Original: In the overparameterized regime, architecture matters for generalization reasons: there are many ways to ﬁt the train set, and some architectures lead SGD to minima that generalizes better.
Revised: In the overparameterized regime, architecture matters for generalization reasons: there are many ways to ﬁt the train set, and some architectures lead SGD to minima that generalizes better.

Original: In the underparameterized regime, architecture matters for purely optimization reasons: all models will have small generalization gap with 1 billion+ samples, but we seek models which are capable of reaching low values of test loss, and which do so quickly (with few optimization steps).
Revised: In the underparameterized regime, architecture matters for purely optimization reasons: all models will have small generalization gap with 1 billion+ samples, but we seek models which are capable of reaching low values of test loss, and which do so quickly (with few optimization steps).

Original: Thus, it should be surprising that in practice, we use essentially the same architectures in both regimes.
Revised: Thus, it should be surprising that in practice, we use essentially the same architectures in both regimes.

Original: Our work suggests that these phenomena are closely related: If the boostrap error is small, then we should expect that architectures which optimize well in the inﬁnite-data (underparameterized) regime also generalize well in the ﬁnite-data (overparameterized) regime.
Revised: Our work suggests that these phenomena are closely related: If the boostrap error is small, then we should expect that architectures which optimize well in the inﬁnite-data (underparameterized) regime also generalize well in the ﬁnite-data (overparameterized) regime.

Original: This uniﬁes the two apriori different principles guiding model-selection in over and under-parameterized regimes, and helps understand why the same architectures are used in both regimes.
Revised: This uniﬁes the two apriori different principles guiding model-selection in over and under-parameterized regimes, and helps understand why the same architectures are used in both regimes.

Original: Implicit Bias via Explicit Optimization.
Revised: Implicit Bias via Explicit Optimization.

Original: Much recent theoretical work has focused on the implicit bias of gradient descent (e.g. Neyshabur et al.
Revised: Much recent theoretical work has focused on the implicit bias of gradient descent (e.g. Neyshabur et al.

Original: (2015a);
Revised: (2015a);

Original: Soudry et al.
Revised: Soudry et al.

Original: ; Gunasekar et al.
Revised: ; Gunasekar et al.

Original: (2018b;a);
Revised: (2018b;a);

Original: Ji & Telgarsky (2019); Chizat & Bach (2020)).
Revised: Ji & Telgarsky (2019); Chizat & Bach (2020)).

Original: For overparameterized networks, there are many minima of the empirical loss, some which have low test error and others which have high test error.
Revised: For overparameterized networks, there are many minima of the empirical loss, some which have low test error and others which have high test error.

Original: Thus studying why interpolating networks generalize amounts to studying why SGD is “biased” towards ﬁnding empirical minima with low population loss.
Revised: Thus studying why interpolating networks generalize amounts to studying why SGD is “biased” towards ﬁnding empirical minima with low population loss.

Original: Our framework suggests an alternate perspective: instead of directly trying to characterize which empirical minima SGD reaches, it may be sufﬁcient to study why SGD optimizes quickly on the population loss.
Revised: Our framework suggests an alternate perspective: instead of directly trying to characterize which empirical minima SGD reaches, it may be sufﬁcient to study why SGD optimizes quickly on the population loss.

Original: That is, instead of studying implicit bias of optimization on the empirical loss, we could study explicit properties of optimization on the population loss.
Revised: That is, instead of studying implicit bias of optimization on the empirical loss, we could study explicit properties of optimization on the population loss.

Original: The following experiment highlights this approach.
Revised: The following experiment highlights this approach.

Original: Consider the D-CONV and D-FC architectures introduced recently by Neyshabur (2020).
Revised: Consider the D-CONV and D-FC architectures introduced recently by Neyshabur (2020).

Original: D-CONV is a deep convolutional network and D-FC is its fully-connected counterpart: an MLP which subsumes the convnet in expressive capacity.
Revised: D-CONV is a deep convolutional network and D-FC is its fully-connected counterpart: an MLP which subsumes the convnet in expressive capacity.

Original: That is,
Revised: That is,

Original: D-FC is capable of representing all functions that D-CONV can represent, since it replaces all conv layers with fully-connected layers and unties all the weights.
Revised: D-FC is capable of representing all functions that D-CONV can represent, since it replaces all conv layers with fully-connected layers and unties all the weights.

Original: Both networks reach close to 0 train
Revised: Both networks reach close to 0 train

Original: [Figure]
Revised: [Figure]

Original: error on 50K samples from CIFAR-5m, but the convnet generalizes much better.
Revised: error on 50K samples from CIFAR-5m, but the convnet generalizes much better.

Original: The traditional explanation for this is that the “implicit bias” of SGD biases the convnet to a better-generalizing minima than the MLP.
Revised: The traditional explanation for this is that the “implicit bias” of SGD biases the convnet to a better-generalizing minima than the MLP.

Original: We show that, in fact, this generalization is captured by the fact that DCONV optimizes much faster on the population loss than D-FC.
Revised: We show that, in fact, this generalization is captured by the fact that DCONV optimizes much faster on the population loss than D-FC.

Original: Figure 5c shows the test and train errors of both networks when trained on 50K samples from CIFAR-5m, in the Real and Ideal Worlds.
Revised: Figure 5c shows the test and train errors of both networks when trained on 50K samples from CIFAR-5m, in the Real and Ideal Worlds.

Original: Observe that the Real and Ideal world test performances are nearly identical.
Revised: Observe that the Real and Ideal world test performances are nearly identical.

Original: Sample Size.
Revised: Sample Size.

Original: In Figure 4, we consider the effect of varying the train set size in the Real World.
Revised: In Figure 4, we consider the effect of varying the train set size in the Real World.

Original: Note that in this case, the Ideal World does not change.
Revised: Note that in this case, the Ideal World does not change.

Original: There are two effects of increasing n : (1) The stopping time extends— Real World training continues for longer before converging.
Revised: There are two effects of increasing n : (1) The stopping time extends— Real World training continues for longer before converging.

Original: And (2) the bootstrap error decreases.
Revised: And (2) the bootstrap error decreases.

Original: Of these, (1) is the dominant effect.
Revised: Of these, (1) is the dominant effect.

Original: Figure 4a illustrates this behavior in detail by considering a single model: ResNet-18 on CIFAR-5m. We plot the Ideal
Revised: Figure 4a illustrates this behavior in detail by considering a single model: ResNet-18 on CIFAR-5m. We plot the Ideal

Original: World behavior of ResNet-18, as well as different Real Worlds for varying n .
Revised: World behavior of ResNet-18, as well as different Real Worlds for varying n .

Original: All Real Worlds are stopped when they reach < 1% train error, as we do throughout this work.
Revised: All Real Worlds are stopped when they reach < 1% train error, as we do throughout this work.

Original: After this point their test performance is essentially ﬂat (shown as faded lines).
Revised: After this point their test performance is essentially ﬂat (shown as faded lines).

Original: However, until this stopping point, all Real
Revised: However, until this stopping point, all Real

Original: Worlds are roughly close to the Ideal World, becoming closer with larger n .
Revised: Worlds are roughly close to the Ideal World, becoming closer with larger n .

Original: These learning curves are representative of most architectures in our experiments.
Revised: These learning curves are representative of most architectures in our experiments.

Original: Figure 4b shows the same architectures of Figure 2a, trained on various sizes of train sets from CIFAR-5m. The Real and Ideal worlds may deviate from each other at small n , but become close for realistically large values of n .
Revised: Figure 4b shows the same architectures of Figure 2a, trained on various sizes of train sets from CIFAR-5m. The Real and Ideal worlds may deviate from each other at small n , but become close for realistically large values of n .

Original: Data Augmentation.
Revised: Data Augmentation.

Original: Data augmentation in the Ideal World corresponds to randomly augmenting each fresh sample before training on it (as opposed to re-using the same sample for multiple augmentations).
Revised: Data augmentation in the Ideal World corresponds to randomly augmenting each fresh sample before training on it (as opposed to re-using the same sample for multiple augmentations).

Original: There are 3 potential effects of data augmentation in our framework: (1) it can affect the Ideal World optimization, (2) it can affect the bootstrap gap, and (3) it can affect the Real World stopping time (time until training converges).
Revised: There are 3 potential effects of data augmentation in our framework: (1) it can affect the Ideal World optimization, (2) it can affect the bootstrap gap, and (3) it can affect the Real World stopping time (time until training converges).

Original: We ﬁnd that the dominant factors are (1) and (3), though data augmentation does typically reduce the bootstrap gap as well.
Revised: We ﬁnd that the dominant factors are (1) and (3), though data augmentation does typically reduce the bootstrap gap as well.

Original: Figure 5a shows the effect of data augmentation on ResNet-18 for CIFAR-5m. In this case, data augmentation does not change the Ideal World much, but it extends the time until the Real World training converges.
Revised: Figure 5a shows the effect of data augmentation on ResNet-18 for CIFAR-5m. In this case, data augmentation does not change the Ideal World much, but it extends the time until the Real World training converges.

Original: This view suggests that good data augmentations should (1) not hurt optimization in the Ideal World (i.e., not destroy true samples much), and (2) obstruct optimization in the Real World (so the Real World can improve for longer before converging).
Revised: This view suggests that good data augmentations should (1) not hurt optimization in the Ideal World (i.e., not destroy true samples much), and (2) obstruct optimization in the Real World (so the Real World can improve for longer before converging).

Original: This is aligned with the “afﬁnity” and “diversity” view of data augmentations in Gontijo-Lopes et al.
Revised: This is aligned with the “afﬁnity” and “diversity” view of data augmentations in Gontijo-Lopes et al.

Original: See Appendix B.2 for more ﬁgures, including examples where data augmentation hurts the Ideal World.
Revised: See Appendix B.2 for more ﬁgures, including examples where data augmentation hurts the Ideal World.

Original: Pretraining.
Revised: Pretraining.

Original: Figure 5b shows the effect of pretraining for Image-GPT (Chen et al., 2020), a transformer pretrained for generative modeling on ImageNet.
Revised: Figure 5b shows the effect of pretraining for Image-GPT (Chen et al., 2020), a transformer pretrained for generative modeling on ImageNet.

Original: We ﬁne-tune iGPT-S on 2K samples of
Revised: We ﬁne-tune iGPT-S on 2K samples of

Original: CIFAR-10 (not CIFAR-5m, since we have enough samples in this case) and compare initializing from an early checkpoint vs. the ﬁnal pretrained model.
Revised: CIFAR-10 (not CIFAR-5m, since we have enough samples in this case) and compare initializing from an early checkpoint vs. the ﬁnal pretrained model.

Original: The fully-pretrained model generalizes better in the Real World, and also optimizes faster in the Ideal World.
Revised: The fully-pretrained model generalizes better in the Real World, and also optimizes faster in the Ideal World.

Original: An additional experiment of
Revised: An additional experiment of

Original: ImageNet-pretraining for MLPs is in Appendix B.4.
Revised: ImageNet-pretraining for MLPs is in Appendix B.4.

Original: Random Labels.
Revised: Random Labels.

Original: Our approach of comparing Real and Ideal worlds also captures generalization in the random-label experiment of Zhang et al.
Revised: Our approach of comparing Real and Ideal worlds also captures generalization in the random-label experiment of Zhang et al.

Original: Speciﬁcally, if we train on a distribution with purely random labels, both Real and Ideal world models will have trivial test error.
Revised: Speciﬁcally, if we train on a distribution with purely random labels, both Real and Ideal world models will have trivial test error.

Original: [Figure]
Revised: [Figure]

Original:  D IFFERENCES BETWEEN THE W ORLDS
Revised:  D IFFERENCES BETWEEN THE W ORLDS

Original: In our framework, we only compare the test soft-error of models in the Real and Ideal worlds.
Revised: In our framework, we only compare the test soft-error of models in the Real and Ideal worlds.

Original: We do not claim these models are close in all respects— in fact, this is not true.
Revised: We do not claim these models are close in all respects— in fact, this is not true.

Original: For example,
Revised: For example,

Original: Figure 6 shows the same ResNet-18s trained in the Introduction (Figure 1), measuring three different metrics in both worlds.
Revised: Figure 6 shows the same ResNet-18s trained in the Introduction (Figure 1), measuring three different metrics in both worlds.

Original: Notably, the test loss diverges drastically between the Real and Ideal worlds, although the test soft-error (and to a lesser extent, test error) remains close.
Revised: Notably, the test loss diverges drastically between the Real and Ideal worlds, although the test soft-error (and to a lesser extent, test error) remains close.

Original: This is because training to convergence in the Real World will cause the network weights to grow unboundedly, and the softmax distribution to concentrate (on both train and test).
Revised: This is because training to convergence in the Real World will cause the network weights to grow unboundedly, and the softmax distribution to concentrate (on both train and test).

Original: In contrast, training in the Ideal World will generally not cause weights to diverge, and the softmax will remain diffuse.
Revised: In contrast, training in the Ideal World will generally not cause weights to diverge, and the softmax will remain diffuse.

Original: This phenomenon also means that the Error and Soft-Error are close in the Real World, but can be slightly different in the Ideal World, which is consistent with our experiments.
Revised: This phenomenon also means that the Error and Soft-Error are close in the Real World, but can be slightly different in the Ideal World, which is consistent with our experiments.

Original:  C ONCLUSION AND D ISCUSSION
Revised:  C ONCLUSION AND D ISCUSSION

Original: We propose the Bootstrap Framework for understanding generalization in deep learning.
Revised: We propose the Bootstrap Framework for understanding generalization in deep learning.

Original: Our approach is to compare the Real World, where optimizers take steps on the empirical loss, to an Ideal
Revised: Our approach is to compare the Real World, where optimizers take steps on the empirical loss, to an Ideal

Original: World, where optimizers have inﬁnite data and take steps on the population loss.
Revised: World, where optimizers have inﬁnite data and take steps on the population loss.

Original: We ﬁnd that in modern settings, the test performance of models is close between these worlds.
Revised: We ﬁnd that in modern settings, the test performance of models is close between these worlds.

Original: This establishes a new connection between the ﬁelds of generalization and online learning: models which learn quickly (online) also generalize well (ofﬂine).
Revised: This establishes a new connection between the ﬁelds of generalization and online learning: models which learn quickly (online) also generalize well (ofﬂine).

Original: Our framework thus provides a new lens on deep phenomena, and lays a promising route towards theoretically understanding generalization in deep learning.
Revised: Our framework thus provides a new lens on deep phenomena, and lays a promising route towards theoretically understanding generalization in deep learning.

Original: Our work takes ﬁrst steps towards characterizing the bootstrap error, but fully understanding this is an important area for future study.
Revised: Our work takes ﬁrst steps towards characterizing the bootstrap error, but fully understanding this is an important area for future study.

Original: The bootstrap error is not universally small for all models and learning tasks: for example, we found the gap was larger at limited sample sizes and without data augmentation.
Revised: The bootstrap error is not universally small for all models and learning tasks: for example, we found the gap was larger at limited sample sizes and without data augmentation.

Original: Moreover, it can be large in simple settings like linear regression (Appendix A), or settings when the Real World test error is non-monotonic (e.g. due to epoch double-decent (Nakkiran et al., 2020)).
Revised: Moreover, it can be large in simple settings like linear regression (Appendix A), or settings when the Real World test error is non-monotonic (e.g. due to epoch double-decent (Nakkiran et al., 2020)).

Original: Nevertheless, the gap appears to be small in realistic deep learning settings, and we hope that future work can help understand why.
Revised: Nevertheless, the gap appears to be small in realistic deep learning settings, and we hope that future work can help understand why.

Original: A T OY E XAMPLE
Revised: A T OY E XAMPLE

Original: Here we present a theoretically-inspired toy example, giving a simple setting where the bootstrap gap is small, but the generalization gap is large.
Revised: Here we present a theoretically-inspired toy example, giving a simple setting where the bootstrap gap is small, but the generalization gap is large.

Original: We also give an analogous example where the bootstrap error is large.
Revised: We also give an analogous example where the bootstrap error is large.

Original: The purpose of these examples is (1) to present a simple setting where the bootstrap framework can be more useful than studying the generalization gap.
Revised: The purpose of these examples is (1) to present a simple setting where the bootstrap framework can be more useful than studying the generalization gap.

Original: And (2) to illustrate that the bootstrap gap is not always small, and can be large in certain standard settings.
Revised: And (2) to illustrate that the bootstrap gap is not always small, and can be large in certain standard settings.

Original: We consider the following setup.
Revised: We consider the following setup.

Original: Let us pass to a regression setting, where we have a distribution over ( x, y ) ∈
Revised: Let us pass to a regression setting, where we have a distribution over ( x, y ) ∈

Original: R d × R , and we care about mean-square-error instead of classiﬁcation error.
Revised: R d × R , and we care about mean-square-error instead of classiﬁcation error.

Original: That is, for a model f , we have TestMSE ( f ) :
Revised: That is, for a model f , we have TestMSE ( f ) :

Original: = E x,y [( f ( x ) − y ) 2 ] .
Revised: = E x,y [( f ( x ) − y ) 2 ] .

Original: Both our examples are from the following class of distributions in dimension d = 1000 .
Revised: Both our examples are from the following class of distributions in dimension d = 1000 .

Original: [Equation]
Revised: [Equation]

Original: where β ∗ ∈ R d is the ground-truth, and σ is a pointwise activation function.
Revised: where β ∗ ∈ R d is the ground-truth, and σ is a pointwise activation function.

Original: The model family is linear,
Revised: The model family is linear,

Original: [Equation]
Revised: [Equation]

Original: We draw n samples from the distribution, and train the model f β using full-batch gradient descent on the empirical loss:
Revised: We draw n samples from the distribution, and train the model f β using full-batch gradient descent on the empirical loss:

Original: [Equation]
Revised: [Equation]

Original: We chose β ∗ =
Revised: We chose β ∗ =

Original: e 1 , and covariance V to be diagonal with 10 eigenvalues of 1 and the remaining eigenvalues of 0 .
Revised: e 1 , and covariance V to be diagonal with 10 eigenvalues of 1 and the remaining eigenvalues of 0 .

Original: That is, x is essentially 10-dimensional, with the remaining coordinates “noise.”
Revised: That is, x is essentially 10-dimensional, with the remaining coordinates “noise.”

Original: The two distributions are instances of the above setting for different choices of parameters.
Revised: The two distributions are instances of the above setting for different choices of parameters.

Original: • Setting A. Linear activation σ ( x ) = x .
Revised: • Setting A. Linear activation σ ( x ) = x .

Original: With n = 20 train samples.
Revised: With n = 20 train samples.

Original: • Setting B. Sign activation σ ( x ) = sgn
Revised: • Setting B. Sign activation σ ( x ) = sgn

Original: ( x ) .
Revised: ( x ) .

Original: With n = 100 train samples.
Revised: With n = 100 train samples.

Original: Setting A is a standard well-speciﬁed
Revised: Setting A is a standard well-speciﬁed

Original: linear regression setting.
Revised: linear regression setting.

Original: Setting B is a misspeciﬁed regression setting.
Revised: Setting B is a misspeciﬁed regression setting.

Original: Figure 7 shows the Real and Ideal worlds in these settings, for gradient-descent on the empirical loss (with step-size η = 0 . 1 ).
Revised: Figure 7 shows the Real and Ideal worlds in these settings, for gradient-descent on the empirical loss (with step-size η = 0 . 1 ).

Original: Observe that in the well-speciﬁed Setting A, the Ideal
Revised: Observe that in the well-speciﬁed Setting A, the Ideal

Original: World performs much better than the Real World, and the bootstrap framework is not as useful.
Revised: World performs much better than the Real World, and the bootstrap framework is not as useful.

Original: However, in the misspeciﬁed Setting B, the bootstrap gap remains small even as the generalization gap grows.
Revised: However, in the misspeciﬁed Setting B, the bootstrap gap remains small even as the generalization gap grows.

Original: This toy example is contrived to help isolate factors important in more realistic settings.
Revised: This toy example is contrived to help isolate factors important in more realistic settings.

Original: We have observed behavior similar to Setting B in other simple settings with real data, such as regression on
Revised: We have observed behavior similar to Setting B in other simple settings with real data, such as regression on

Original: MNIST/Fashion-MNIST, as well as in the more complex settings in the body of this paper.
Revised: MNIST/Fashion-MNIST, as well as in the more complex settings in the body of this paper.

Original: B A DDITIONAL F IGURES B. DARTS A RCHITECTURES
Revised: B A DDITIONAL F IGURES B. DARTS A RCHITECTURES

Original: Figure 8 shows the Real vs Ideal world for trained random DARTS architectures.
Revised: Figure 8 shows the Real vs Ideal world for trained random DARTS architectures.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Figure 9 shows the effect of data-augmentation in the Ideal World, for several selected architectures on CIFAR-5m. Recall that data augmentation in the Ideal World corresponds to randomly augmenting each fresh sample once, as opposed to augmenting the same sample multiple times.
Revised: Figure 9 shows the effect of data-augmentation in the Ideal World, for several selected architectures on CIFAR-5m. Recall that data augmentation in the Ideal World corresponds to randomly augmenting each fresh sample once, as opposed to augmenting the same sample multiple times.

Original: We train with SGD using the same hyperparameters as the main experiments (described in Appendix C.2).
Revised: We train with SGD using the same hyperparameters as the main experiments (described in Appendix C.2).

Original: We use standard CIFAR-10 data augmentation: random crop and random horizontal ﬂip.
Revised: We use standard CIFAR-10 data augmentation: random crop and random horizontal ﬂip.

Original: The test performance without augmentation is shown as solid lines, and with augmentation as dashed lines.
Revised: The test performance without augmentation is shown as solid lines, and with augmentation as dashed lines.

Original: Note that VGG and ResNet do not behave differently with augmentation, but augmentation signiﬁcantly hurts AlexNet and the MLP.
Revised: Note that VGG and ResNet do not behave differently with augmentation, but augmentation signiﬁcantly hurts AlexNet and the MLP.

Original: This may be because VGG and ResNet have global spatial pooling, which makes them (partially) shift-invariant, and thus more amenable to the random cropping.
Revised: This may be because VGG and ResNet have global spatial pooling, which makes them (partially) shift-invariant, and thus more amenable to the random cropping.

Original: In contrast, augmentation hurts the architectures without global pooling, perhaps because for these architectures, augmented samples appear more out-of-distribution.
Revised: In contrast, augmentation hurts the architectures without global pooling, perhaps because for these architectures, augmented samples appear more out-of-distribution.

Original: Figure 10a shows the same architectures and setting as Figure 2 but trained without data augmentation.
Revised: Figure 10a shows the same architectures and setting as Figure 2 but trained without data augmentation.

Original: That is, we train on 50K samples from CIFAR-5m, using SGD with cosine decay and initial learning rate { 0 . 1 , 0 . 01 , 0 . 001 } .
Revised: That is, we train on 50K samples from CIFAR-5m, using SGD with cosine decay and initial learning rate { 0 . 1 , 0 . 01 , 0 . 001 } .

Original: Figure 10b shows learning curves with and without data augmentation of a ResNet-18 on n = 10 k samples.
Revised: Figure 10b shows learning curves with and without data augmentation of a ResNet-18 on n = 10 k samples.

Original: This is the analogous setting of Figure 5a in the body, which is for n = 50 k samples.
Revised: This is the analogous setting of Figure 5a in the body, which is for n = 50 k samples.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Figure 11 shows several experiments with the Adam optimizer (Kingma & Ba, 2014) in place of
Revised: Figure 11 shows several experiments with the Adam optimizer (Kingma & Ba, 2014) in place of

Original: SGD.
Revised: SGD.

Original: We train all architectures on 50K samples from CIFAR-5m, with data-augmentation, batchsize 128, using Adam with default parameters (lr= 0 .
Revised: We train all architectures on 50K samples from CIFAR-5m, with data-augmentation, batchsize 128, using Adam with default parameters (lr= 0 .

Original: 001 , β 1 = 0 .
Revised: 001 , β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: P RETRAINING
Revised: P RETRAINING

Original: Figure 12 shows the effect of pretraining for an MLP (3x2048) on CIFAR-5m, by comparing training from scratch (random initialization) to training from an ImageNet-pretrained initialization.
Revised: Figure 12 shows the effect of pretraining for an MLP (3x2048) on CIFAR-5m, by comparing training from scratch (random initialization) to training from an ImageNet-pretrained initialization.

Original: The pretrained MLP generalizes better in the Real World, and also optimizes faster in the Ideal World.
Revised: The pretrained MLP generalizes better in the Real World, and also optimizes faster in the Ideal World.

Original: We ﬁne tune on 50K samples from CIFAR-5m, with no data-augmentation.
Revised: We ﬁne tune on 50K samples from CIFAR-5m, with no data-augmentation.

Original: For ImageNet-pretraining, we train the MLP[3x2048] on full ImageNet (224px, 1000 classes), using
Revised: For ImageNet-pretraining, we train the MLP[3x2048] on full ImageNet (224px, 1000 classes), using

Original: Adam with default settings, and batchsize 1024.
Revised: Adam with default settings, and batchsize 1024.

Original: We use standard ImageNet data augmentation (random resized crop + horizontal ﬂip) and train for 500 epochs.
Revised: We use standard ImageNet data augmentation (random resized crop + horizontal ﬂip) and train for 500 epochs.

Original: This MLP achieves test accuracy 21% and train accuracy 30% on ImageNet.
Revised: This MLP achieves test accuracy 21% and train accuracy 30% on ImageNet.

Original: For ﬁne-tuning, we adapt the network to 32px input size by resizing the ﬁrst layer ﬁlters from 224x224 to 32x32 via bilinear interpolation.
Revised: For ﬁne-tuning, we adapt the network to 32px input size by resizing the ﬁrst layer ﬁlters from 224x224 to 32x32 via bilinear interpolation.

Original: We then replace the classiﬁcation layer, and ﬁne-tune the entire network on CIFAR-5m.
Revised: We then replace the classiﬁcation layer, and ﬁne-tune the entire network on CIFAR-5m.

Original: [Figure]
Revised: [Figure]

Original: Figure 13 shows test soft-error, test error
Revised: Figure 13 shows test soft-error, test error, and test loss for VGG-11 from Figure 1.

Original: [Figure]
Revised: [Figure]

Revised: Here we show the results of several of our experiments if we measure the bootstrap gap with respect to Test Error instead of SoftError.

Revised: The bootstrap gap is often reasonably small even with respect to

Revised: Error, though it is not as well behaved as SoftError.

Revised: Figure 14 shows the same setting as Figure 2a in the body, but measuring Error instaed of SoftError.

Revised: B.6. T RAINING WITH MSE

Revised: We can measure Test Error even for networks which do not naturally output a probability distribution.

Revised: Here, we train various architectures on CIFAR-5m using the squared-loss (MSE) directly on logits, with no softmax layer.

Revised: This follows the methodology in Hui & Belkin (2020).

Original: and test loss for VGG-11 from Figure 1.
Revised: We train all Real-World models using SGD, batchsize 128, momentum 0 . 9 , initial learning rate 0 . 002 with cosine decay for 100 epochs. Figure 15 shows the Test Error and Test Loss in the Real and Ideal Worlds.

Revised: The bootstrap gap, with respect to test error, for MSE-trained networks is reasonably small – though there are deviations in the low error regime.

Revised: Compare this to Figure 2a, which measures the SoftError for networks trained with cross-entropy.

Revised: [Figure]

Original: C A PPENDIX : E XPERIMENTAL D ETAILS C. I NTRODUCTION E XPERIMENT
Revised: C A PPENDIX : E XPERIMENTAL D ETAILS C. I NTRODUCTION E XPERIMENT

Original: All architectures trained in the Real World with n = 50 K samples from CIFAR-5m, using SGD with batch size 128 , cross-entropy loss, initial learning rate 0 .
Revised: All architectures trained in the Real World with n = 50 K samples from CIFAR-5m, using SGD with batch size 128 , cross-entropy loss, initial learning rate 0 .

Original: 1 , cosine learning rate decay, for 100 epochs, with random crop+horizontal ﬂip data-augmentation (i.e. standard CIFAR-10 data augmentation).
Revised: 1 , cosine learning rate decay, for 100 epochs, with random crop+horizontal ﬂip data-augmentation (i.e. standard CIFAR-10 data augmentation).

Original: The MLP was additional trained with momentum 0 .
Revised: The MLP was additional trained with momentum 0 .

Original: 9 to improve convergence.
Revised: 9 to improve convergence.

Original: We plot median over 10 trials (with random sampling of the train set, random initialization, and random
Revised: We plot median over 10 trials (with random sampling of the train set, random initialization, and random

Original: SGD order in each trial).
Revised: SGD order in each trial).

Original: We use an MLP with 5 hidden layers of width 2048, with batch norm.
Revised: We use an MLP with 5 hidden layers of width 2048, with batch norm.

Original: Figure 1 plots the Test
Revised: Figure 1 plots the Test

Original: Soft-Error and Train Soft-Errors over the course of training.
Revised: Soft-Error and Train Soft-Errors over the course of training.

Original: C. M AIN E XPERIMENTS
Revised: C. M AIN E XPERIMENTS

Original: For CIFAR-5m we use the following architectures: AlexNet (Krizhevsky et al., 2012), VGG (Simonyan & Zisserman, 2015), ResNet (He et al., 2016), DenseNet (Huang et al., 2017).
Revised: For CIFAR-5m we use the following architectures: AlexNet (Krizhevsky et al., 2012), VGG (Simonyan & Zisserman, 2015), ResNet (He et al., 2016), DenseNet (Huang et al., 2017).

Original: The Myrtle architecture is a 5-layer CNN introduced by (Page, 2018).
Revised: The Myrtle architecture is a 5-layer CNN introduced by (Page, 2018).

Original: All our experiments are implemented in
Revised: All our experiments are implemented in

Original: PyTorch (Paszke et al., 2019).
Revised: PyTorch (Paszke et al., 2019).

Original: In the Real World, we train these architectures on n = 50 K samples from CIFAR-5m using cross-entropy loss.
Revised: In the Real World, we train these architectures on n = 50 K samples from CIFAR-5m using cross-entropy loss.

Original: All models are trained with SGD with batchsize 128, initial learning rate { 0 . 1 , 0 . 01 , 0 . 001 } , cosine learning rate decay, for 100 total epochs, with data augmentation: random horizontal ﬂip and RandomCrop(32, padding=4) .
Revised: All models are trained with SGD with batchsize 128, initial learning rate { 0 . 1 , 0 . 01 , 0 . 001 } , cosine learning rate decay, for 100 total epochs, with data augmentation: random horizontal ﬂip and RandomCrop(32, padding=4) .

Original: We plot median over 10 trials (with random sampling of the train set, random initialization, and random SGD order in each trial).
Revised: We plot median over 10 trials (with random sampling of the train set, random initialization, and random SGD order in each trial).

Original: DARTS Architectures.
Revised: DARTS Architectures.

Original: We sample architectures from the DARTS search space (Liu et al., 2019), as implemented in the codebase of Dong & Yang (2020).
Revised: We sample architectures from the DARTS search space (Liu et al., 2019), as implemented in the codebase of Dong & Yang (2020).

Original: We follow the parameters used for CIFAR10 in Dong & Yang (2020), while also varying width and depth for added diversity.
Revised: We follow the parameters used for CIFAR10 in Dong & Yang (2020), while also varying width and depth for added diversity.

Original: Speciﬁcally, we use 4 nodes, number of cells ∈ { 1 , 5 } , and width ∈ { 16 , 64 } .
Revised: Speciﬁcally, we use 4 nodes, number of cells ∈ { 1 , 5 } , and width ∈ { 16 , 64 } .

Original: We train all DARTS architectures with
Revised: We train all DARTS architectures with

Original: SGD, batchsize 128, initial learning rate 0 .
Revised: SGD, batchsize 128, initial learning rate 0 .

Original: 1 , cosine learning rate decay, for 100 total epochs, with random crop+ﬂip data-augmentation.
Revised: 1 , cosine learning rate decay, for 100 total epochs, with random crop+ﬂip data-augmentation.

Original: ImageNet: DogBird All architectures for ImageNet-DogBird are trained with SGD, batchsize 128, learning rate 0 .
Revised: ImageNet: DogBird All architectures for ImageNet-DogBird are trained with SGD, batchsize 128, learning rate 0 .

Original: 01 , momentum 0 .
Revised: 01 , momentum 0 .

Original: 9 , for 120 epochs, with standard ImageNet data augmentation (random resized crop to 224px, horizontal ﬂip).
Revised: 9 , for 120 epochs, with standard ImageNet data augmentation (random resized crop to 224px, horizontal ﬂip).

Original: We report medians over 10 trials for each architecture.
Revised: We report medians over 10 trials for each architecture.

Original: We additional include the ImageNet architectures: BagNet (Brendel & Bethge, 2019), MobileNet (Sandler et al., 2018), and ResNeXt (Xie et al., 2017).
Revised: We additional include the ImageNet architectures: BagNet (Brendel & Bethge, 2019), MobileNet (Sandler et al., 2018), and ResNeXt (Xie et al., 2017).

Original: The architectures SCONV9 and SCONV refer to the S-CONV architectures deﬁned by Neyshabur (2020), instantiated for ImageNet with base-width 48, image size 224, and kernel size { 9 , 33 } respectively.
Revised: The architectures SCONV9 and SCONV refer to the S-CONV architectures deﬁned by Neyshabur (2020), instantiated for ImageNet with base-width 48, image size 224, and kernel size { 9 , 33 } respectively.

Original: C. I MPLICIT B IAS
Revised: C. I MPLICIT B IAS

Original: We use the D-CONV architecture from (Neyshabur, 2020), with base width 32 , and the corresponding D-FC architecture.
Revised: We use the D-CONV architecture from (Neyshabur, 2020), with base width 32 , and the corresponding D-FC architecture.

Original: PyTorch speciﬁcation of these architectures are provided in Appendix E for convenience.
Revised: PyTorch speciﬁcation of these architectures are provided in Appendix E for convenience.

Original: We train both architectures with SGD, batchsize 128, initial learning rate 0 .
Revised: We train both architectures with SGD, batchsize 128, initial learning rate 0 .

Original: 1 , cosine learning rate decay, for 100 total epochs, with random crop + horizontal ﬂip data-augmentation.
Revised: 1 , cosine learning rate decay, for 100 total epochs, with random crop + horizontal ﬂip data-augmentation.

Original: We plot median errors over 10 trials.
Revised: We plot median errors over 10 trials.

Original: C. I MAGE -GPT F INETUNING
Revised: C. I MAGE -GPT F INETUNING

Original: We ﬁne-tune iGPT-S, using the publicly available pretrained model checkpoints from Chen et al.
Revised: We ﬁne-tune iGPT-S, using the publicly available pretrained model checkpoints from Chen et al.

Original: The “Early” checkpoint in Figure 5b refers to checkpoint 131000, and the “Final” checkpoint is 1000000.
Revised: The “Early” checkpoint in Figure 5b refers to checkpoint 131000, and the “Final” checkpoint is 1000000.

Original: Following Chen et al.
Revised: Following Chen et al.

Original: (2020), we use Adam with ( lr = 0 .
Revised: (2020), we use Adam with ( lr = 0 .

Original: 003 , β 1 = 0 .
Revised: 003 , β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: 95) , and batchsize 128.
Revised: 95) , and batchsize 128.

Original: We do not use data augmentation.
Revised: We do not use data augmentation.

Original: For simplicity, we differ slightly from Chen et al.
Revised: For simplicity, we differ slightly from Chen et al.

Original: (2020) in that we simply attach the classiﬁcation head to the [average-pooled] last transformer layer, and we ﬁne-tune using only classiﬁcation loss and not the joint generative+classiﬁcation loss used in Chen et al.
Revised: (2020) in that we simply attach the classiﬁcation head to the [average-pooled] last transformer layer, and we ﬁne-tune using only classiﬁcation loss and not the joint generative+classiﬁcation loss used in Chen et al.

Original: Note that we ﬁne-tune the entire model, not just the classiﬁcation head.
Revised: Note that we ﬁne-tune the entire model, not just the classiﬁcation head.

Original: D A PPENDIX : D ATASETS D. CIFAR-5 M
Revised: D A PPENDIX : D ATASETS D. CIFAR-5 M

Original: CIFAR-5m is a dataset of 6 million synthetic CIFAR-10-like images.
Revised: CIFAR-5m is a dataset of 6 million synthetic CIFAR-10-like images.

Original: The images are RGB 32 × 32 px.
Revised: The images are RGB 32 × 32 px.

Original: We generate samples from the Denoising Diffusion generative model of Ho et al.
Revised: We generate samples from the Denoising Diffusion generative model of Ho et al.

Original: (2020) trained on the CIFAR-10 train set (Krizhevsky, 2009).
Revised: (2020) trained on the CIFAR-10 train set (Krizhevsky, 2009).

Original: We use the publicly available trained model and sampling code provided by the authors at https://github.com/hojonathanho/ diffusion .
Revised: We use the publicly available trained model and sampling code provided by the authors at https://github.com/hojonathanho/ diffusion .

Original: We then label these unconditional samples by a 98.5% accurate Big-Transfer model (Kolesnikov et al., 2019).
Revised: We then label these unconditional samples by a 98.5% accurate Big-Transfer model (Kolesnikov et al., 2019).

Original: Speciﬁcally, we use the pretrained BiT-M-R152x2 model, ﬁne-tuned on
Revised: Speciﬁcally, we use the pretrained BiT-M-R152x2 model, ﬁne-tuned on

Original: CIFAR-10 using the author-provided code at https://github.com/google-research/ big_transfer .
Revised: CIFAR-10 using the author-provided code at https://github.com/google-research/ big_transfer .

Original: We use 5 million images for training, and reserve the remaining images for the test set.
Revised: We use 5 million images for training, and reserve the remaining images for the test set.

Original: We release this dataset publicly (URL omitted for anonymity during review.)
Revised: We release this dataset publicly (URL omitted for anonymity during review.)

Original: Random samples from CIFAR-5m are shown in Figure 16.
Revised: Random samples from CIFAR-5m are shown in Figure 18.

Original: For comparison, we show random samples from CIFAR-10 in Figure 15.
Revised: For comparison, we show random samples from CIFAR-10 in Figure 17.

Original: D. I MAGENET : D OG B IRD
Revised: D. I MAGENET : D OG B IRD

Original: The ImageNet-DogBird task is constructed by collapsing classes from ImageNet.
Revised: The ImageNet-DogBird task is constructed by collapsing classes from ImageNet.

Original: The task is to distinguish dogs from birds.
Revised: The task is to distinguish dogs from birds.

Original: The dogs are all ImageNet classes under the WordNet synset “hunting dog” (including 63 ImageNet classes) and birds are all classes under synset “bird” (including
Revised: The dogs are all ImageNet classes under the WordNet synset “hunting dog” (including 63 ImageNet classes) and birds are all classes under synset “bird” (including

Original: ImageNet classes).
Revised: ImageNet classes).

Original: The listing of the ImageNet wnids included in each class is provided below.
Revised: The listing of the ImageNet wnids included in each class is provided below.

Original: This is a relatively easy task compared to full ImageNet: A ResNet-18 trained on 10K samples from
Revised: This is a relatively easy task compared to full ImageNet: A ResNet-18 trained on 10K samples from

Original: ImageNet-DogBird, with standard ImageNet data augmentation, can achieve test accuracy 95%.
Revised: ImageNet-DogBird, with standard ImageNet data augmentation, can achieve test accuracy 95%.

Original: Hunting Dogs (n2087122): n02091831, n02097047, n02088364, n02094433, n02097658, n02089078, n02090622, n02095314, n02102040, n02097130, n02096051, n02098105, n02095889, n02100236, n02099267, n02102318, n02097474, n02090721, n02102973, n02095570, n02091635, n02099429, n02090379, n02094258, n02100583, n02092002, n02093428, n02098413, n02097298, n02093754, n02096177, n02091032, n02096437, n02087394, n02092339, n02099712, n02088632, n02093647, n02098286, n02096585, n02093991, n02100877, n02094114, n02101388, n02089973, n02088094, n02088466, n02093859, n02088238, n02102480, n02101556, n02089867, n02099601, n02102177, n02101006, n02091134, n02100735, n02099849, n02093256, n02097209, n02091467, n02091244, n
Revised: Hunting Dogs (n2087122): n02091831, n02097047, n02088364, n02094433, n02097658, n02089078, n02090622, n02095314, n02102040, n02097130, n02096051, n02098105, n02095889, n02100236, n02099267, n02102318, n02097474, n02090721, n02102973, n02095570, n02091635, n02099429, n02090379, n02094258, n02100583, n02092002, n02093428, n02098413, n02097298, n02093754, n02096177, n02091032, n02096437, n02087394, n02092339, n02099712, n02088632, n02093647, n02098286, n02096585, n02093991, n02100877, n02094114, n02101388, n02089973, n02088094, n02088466, n02093859, n02088238, n02102480, n02101556, n02089867, n02099601, n02102177, n02101006, n02091134, n02100735, n02099849, n02093256, n02097209, n02091467, n02091244, n

Original: Birds (n1503061): n01855672, n01560419, n02009229, n01614925, n01530575, n01798484, n02007558, n01860187, n01820546, n01817953, n01833805, n02058221, n01806567, n01558993, n02056570, n01797886, n02018207, n01828970, n02017213, n02006656, n01608432, n01818515, n02018795, n01622779, n01582220, n02013706, n01534433, n02027492, n02012849, n02051845, n01824575, n01616318, n02002556, n01819313, n01806143, n02033041, n01601694, n01843383, n02025239, n02002724, n01843065, n01514859, n01796340, n01855032, n01580077, n01807496, n01847000, n01532829, n01537544, n01531178, n02037110, n01514668, n02028035, n01795545, n01592084, n01518878, n01829413, n02009912, n
Revised: Birds (n1503061): n01855672, n01560419, n02009229, n01614925, n01530575, n01798484, n02007558, n01860187, n01820546, n01817953, n01833805, n02058221, n01806567, n01558993, n02056570, n01797886, n02018207, n01828970, n02017213, n02006656, n01608432, n01818515, n02018795, n01622779, n01582220, n02013706, n01534433, n02027492, n02012849, n02051845, n01824575, n01616318, n02002556, n01819313, n01806143, n02033041, n01601694, n01843383, n02025239, n02002724, n01843065, n01514859, n01796340, n01855032, n01580077, n01807496, n01847000, n01532829, n01537544, n01531178, n02037110, n01514668, n02028035, n01795545, n01592084, n01518878, n01829413, n02009912, n

Original: E D-CONV, D-FC A RCHITECTURE D ETAILS
Revised: E D-CONV, D-FC A RCHITECTURE D ETAILS

Original: For convenience, we provide PyTorch speciﬁcations for the D-CONV and D-FC architectures from
Revised: For convenience, we provide PyTorch speciﬁcations for the D-CONV and D-FC architectures from

Original: Neyshabur (2020) which we use in this work.
Revised: Neyshabur (2020) which we use in this work.

Original: D-CONV.
Revised: D-CONV.

Original: This model has 6563498 parameters.
Revised: This model has 6563498 parameters.

Original: Network( (features): Sequential( (0): Conv2d(3, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1) , bias=False) (1):BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=
Revised: Network( (features): Sequential( (0): Conv2d(3, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1) , bias=False) (1):BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=

Original: True) (2): ReLU(inplace=
Revised: True) (2): ReLU(inplace=

Original: True) (3): Conv2d(32, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1) , bias=False) (4):BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=
Revised: True) (3): Conv2d(32, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1) , bias=False) (4):BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=

Original: True) (5): ReLU(inplace=
Revised: True) (5): ReLU(inplace=

Original: True) (6): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1) , bias=False) (7): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=
Revised: True) (6): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1) , bias=False) (7): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=

Original: True) (8): ReLU(inplace=
Revised: True) (8): ReLU(inplace=

Original: True) (9): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1) , bias=False) (10):BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=
Revised: True) (9): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1) , bias=False) (10):BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=

Original: True) (11): ReLU(inplace=
Revised: True) (11): ReLU(inplace=

Original: True) (12): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1) , bias=False) (
Revised: True) (12): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1) , bias=False) (

Original: ):BatchNorm2d
Revised: ):BatchNorm2d

Original: , eps=1e-05, momentum
Revised: , eps=1e-05, momentum

Original: 0.1, affine=
Revised: 0.1, affine=

Original: True) (14): ReLU(inplace=
Revised: True) (14): ReLU(inplace=

Original: ): Conv2d
Revised: ): Conv2d

Original: , kernel
Revised: , kernel

Original: size
Revised: size

Original: , stride
Revised: , stride

Original: padding
Revised: padding

Original: , bias=False)
Revised: , bias=False)

Original: :BatchNorm2d
Revised: :BatchNorm2d

Original: eps
Revised: eps

Original: 1e
Revised: 1e

Original: , affine
Revised: , affine

Original: ): ReLU
Revised: ): ReLU

Original: inplace
Revised: inplace

Original: ): Conv2d
Revised: ): Conv2d

Original: kernel
Revised: kernel

Original: = =
Revised: = =

Original: , bias=False)
Revised: , bias=False)

Original: BatchNorm2d
Revised: BatchNorm2d

Original: eps
Revised: eps

Original: 1e
Revised: 1e

Original: affine
Revised: affine

Original: ReLU
Revised: ReLU

Original: Conv2d
Revised: Conv2d

Original: kernel
Revised: kernel

Original: , bias=False)
Revised: , bias=False)

Original: BatchNorm2d
Revised: BatchNorm2d

Original: eps
Revised: eps

Original: 1e
Revised: 1e

Original: affine
Revised: affine

Original: ReLU
Revised: ReLU

Original: Sequential
Revised: Sequential

Original: Linear
Revised: Linear

Original: features=
Revised: features=

Original: features
Revised: features

Original: , bias=False)
Revised: , bias=False)

Original: BatchNorm1d
Revised: BatchNorm1d

Original: 1e
Revised: 1e

Original: momentum
Revised: momentum

Original: affine
Revised: affine

Original: ReLU
Revised: ReLU

Original: inplace
Revised: inplace

Original: Dropout
Revised: Dropout

Original: inplace=
Revised: inplace=

Original: Linear
Revised: Linear

Original: _features=
Revised: _features=

Original: out_features=10, bias=
Revised: out_features=10, bias=

Original: [Table]
Revised: [Table]

Original: Network( features
Revised: Network( features

Original: Sequential( (0): Linear(in_features=3072, out_features=32768, bias=False) (1): BatchNorm1d(
Revised: Sequential( (0): Linear(in_features=3072, out_features=32768, bias=False) (1): BatchNorm1d(

Original: (4): BatchNorm1d(16384, eps=1e-05, momentum=0.1, affine=
Revised: (4): BatchNorm1d(16384, eps=1e-05, momentum=0.1, affine=

Original: True) (5): ReLU(inplace=
Revised: True) (5): ReLU(inplace=

Original: True)
Revised: True)

Original: (6): Linear(
Revised: (6): Linear(

Original: _features=16384, out_features=16384, bias=False) (7): BatchNorm1d(16384, eps=1e-05, momentum=0.1, affine=
Revised: _features=16384, out_features=16384, bias=False) (7): BatchNorm1d(16384, eps=1e-05, momentum=0.1, affine=

Original: True) (8): ReLU(inplace=
Revised: True) (8): ReLU(inplace=

Original: True) (9): Linear(in_features=16384, out_features=8192, bias=False) (10): BatchNorm1d(8192, eps=1e-05, momentum=0.1, affine=
Revised: True) (9): Linear(in_features=16384, out_features=8192, bias=False) (10): BatchNorm1d(8192, eps=1e-05, momentum=0.1, affine=

Original: True) (11): ReLU(inplace=
Revised: True) (11): ReLU(inplace=

Original: True)
Revised: True)

Original: (12): Linear(in_features=8192, out_features=8192, bias=False) (13): BatchNorm1d(8192, eps=1e-05, momentum=0.1, affine=
Revised: (12): Linear(in_features=8192, out_features=8192, bias=False) (13): BatchNorm1d(8192, eps=1e-05, momentum=0.1, affine=

Original: True) (14): ReLU(inplace=
Revised: True) (14): ReLU(inplace=

Original: True)
Revised: True)

Original: (15): Linear(in_features=8192, out_features=4096, bias=False) (16): BatchNorm1d(4096, eps=1e-05, momentum=0.1, affine=
Revised: (15): Linear(in_features=8192, out_features=4096, bias=False) (16): BatchNorm1d(4096, eps=1e-05, momentum=0.1, affine=

Original: True) (17): ReLU(inplace=
Revised: True) (17): ReLU(inplace=

Original: True)
Revised: True)

Original: (18): Linear(in_features=4096, out_features=4096, bias=False) (19): BatchNorm1d(4096, eps=1e-05, momentum=0.1, affine=
Revised: (18): Linear(in_features=4096, out_features=4096, bias=False) (19): BatchNorm1d(4096, eps=1e-05, momentum=0.1, affine=

Original: True) (20): ReLU(inplace=
Revised: True) (20): ReLU(inplace=

Original: True) (21): Linear(in_features=4096, out_features=2048, bias=False) (22): BatchNorm1d(2048, eps=1e-05, momentum=0.1, affine=True) (23): ReLU(inplace=True) )
Revised: True) (21): Linear(in_features=4096, out_features=2048, bias=False) (22): BatchNorm1d(2048, eps=1e-05, momentum=0.1, affine=True) (23): ReLU(inplace=True) )

Original: (classifier): Sequential
Revised: (classifier): Sequential

Original: (0): Linear
Revised: (0): Linear

Original: _features=2048, out_features=2048, bias=False) (1) BatchNorm1d 2048, eps=1e
Revised: (in_features=2048, out_features=2048, bias=False) (1): BatchNorm1d(2048, eps=1e-

Original: =0. , affine
Revised: , momentum=0.1, affine=

Original:  ReLU(inplace=
Revised: True) (2): ReLU(inplace=

Original: True) (3) Dropout(p=0.5, inplace=
Revised: True) (3): Dropout(p=0.5, inplace=

Original: False) (4) Linear(in_features=2048, out_features
Revised: False) (4): Linear(in_features=2048, out_features

Revised: , bias