Original: O N THE E DGE OF B ENIGN O VERFITTING : L ABEL N OISE AND O VERPARAMETERIZATION L EVEL
Revised: O N THE E DGE OF B ENIGN O VERFITTING : L ABEL N OISE AND O VERPARAMETERIZATION L EVEL

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Studies on benign overfitting provide insights for the success of overparameterized deep learning models.
Revised: Studies on benign overfitting provide insights for the success of overparameterized deep learning models.

Original: In this work, we examine whether overfitting is truly benign in real-world classification tasks.
Revised: In this work, we examine whether overfitting is truly benign in real-world classification tasks.

Original: We start with the observation that a ResNet model overfits benignly on Cifar10 but not benignly on ImageNet.
Revised: We start with the observation that a ResNet model overfits benignly on Cifar10 but not benignly on ImageNet.

Original: To understand why benign overfitting fails in the ImageNet experiment, we theoretically analyze benign overfitting under a more restrictive setup where the number of parameters is not significantly larger than the number of data points.
Revised: To understand why benign overfitting fails in the ImageNet experiment, we theoretically analyze benign overfitting under a more restrictive setup where the number of parameters is not significantly larger than the number of data points.

Original: Under this mild overparameterization setup, our analysis identifies a phase change: unlike in the previous heavy overparameterization settings, benign overfitting can now fail in the presence of label noise.
Revised: Under this mild overparameterization setup, our analysis identifies a phase change: unlike in the previous heavy overparameterization settings, benign overfitting can now fail in the presence of label noise.

Original: Our analysis explains our empirical observations, and is validated by a set of control experiments with ResNets.
Revised: Our analysis explains our empirical observations, and is validated by a set of control experiments with ResNets.

Original: Our work highlights the importance of understanding implicit bias in underfitting regimes as a future direction.
Revised: Our work highlights the importance of understanding implicit bias in underfitting regimes as a future direction.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Modern deep learning models achieve good generalization performances even with more parameters than data points.
Revised: Modern deep learning models achieve good generalization performances even with more parameters than data points.

Original: This surprising phenomenon is referred to as benign overfitting and differs from the canonical learning regime where good generalization requires limiting the model complexity (Mohri et al., 2018).
Revised: This surprising phenomenon is referred to as benign overfitting and differs from the canonical learning regime where good generalization requires limiting the model complexity (Mohri et al., 2018).

Original: One widely accepted explanation for benign overfitting is that optimization algorithms benefit from implicit bias and find good solutions among the interpolating ones under the overparametrized settings.
Revised: One widely accepted explanation for benign overfitting is that optimization algorithms benefit from implicit bias and find good solutions among the interpolating ones under the overparametrized settings.

Original: The implicit bias can vary from problem to problem.
Revised: The implicit bias can vary from problem to problem.

Original: Examples include the min-norm solution in regression settings or the max-margin solution in classification settings (Gunasekar et al., 2018a; Soudry et al., 2018; Gunasekar et al., 2018b).
Revised: Examples include the min-norm solution in regression settings or the max-margin solution in classification settings (Gunasekar et al., 2018a; Soudry et al., 2018; Gunasekar et al., 2018b).

Original: These types of bias in optimization can further result in good generalization performances (Bartlett et al., 2020;
Revised: These types of bias in optimization can further result in good generalization performances (Bartlett et al., 2020;

Original: Zou et al., 2021;
Revised: Zou et al., 2021;

Original: Frei et al., 2022).
Revised: Frei et al., 2022).

Original: These studies provide novel insights, yet they sometimes differ from the deep learning practice: state-of-the-art models, despite being overparameterized, often do not interpolate the data points ( e.g. , He et al.
Revised: These studies provide novel insights, yet they sometimes differ from the deep learning practice: state-of-the-art models, despite being overparameterized, often do not interpolate the data points ( e.g. , He et al.

Original: ; Devlin et al.
Revised: ; Devlin et al.

Original: We first examine the existence of benign overfitting in realistic setups.
Revised: We first examine the existence of benign overfitting in realistic setups.

Original: In the rest of this work, we term benign overfitting as the observation that validation performance does not drop while the model fits more training data points  .
Revised: In the rest of this work, we term benign overfitting as the observation that validation performance does not drop while the model fits more training data points 1 .

Original: We test whether ResNet (He et al., 2016) models overfit data benignly for image classification on CIFAR10 and ImageNet.
Revised: We test whether ResNet (He et al., 2016) models overfit data benignly for image classification on CIFAR10 and ImageNet.

Original: Our results are shown in Figure 1 below.
Revised: Our results are shown in Figure 1 below.

Revised: In particular, we first trained ResNet18 on CIFAR10 for 200 epochs and the model interpolates the training data.

Revised: In addition, we also trained ResNet50 on ImageNet for 500 epochs, as opposed to the common schedule that stops at 90 epochs.

Original: [Figure]
Revised: Surprisingly, we found that although benign overfitting happens on the CIFAR10 dataset, overfitting is not benign on the ImageNet dataset—the test loss increased as the model further fits the train set. More precisely, the ImageNet experiment does not overfit benignly since the best model is achieved in the middle of the training. The different overfitting behaviors cannot be explained by known analysis for classification tasks, as no negative results have been studied yet. Motivated by the above observation, our work aims to understand the cause for the two different overfitting behaviors in ImageNet and CIFAR10, and to reconcile the empirical phenomenon with previous analysis on benign overfitting.

Revised: Our first hint comes from the level of overparameterization.

Revised: Previous results on benign overfitting in the classification setting usually requires that p = ω ( n ) ,

Original: [Table]
Revised: [Figure]

Original: ClassificationNoisy
Revised: ClassificationNoiseless

Revised: ClassificationNoisy

Original: RegressionNoisy
Revised: RegressionNoisy

Original: [Table] p = Θ( n )
Revised: Mild overparameterization p = Θ( n )

Original: Benign (Ours)
Revised: Benign (Ours)

Original: Not Benign (Ours)
Revised: Not Benign (Ours)

Original: Not Benign (Bartlett et al., 2020)
Revised: Not Benign (Bartlett et al., 2020)

Original: [Table] p = ω ( n )
Revised: Heavy overparameterization p = ω ( n )

Original: Benign (Cao et al., 2021)
Revised: Benign (Cao et al., 2021)

Original: (Wang et al., 2021a)
Revised: (Wang et al., 2021a)

Original: Benign (Chatterji et al., 2021) (Frei et al., 2022)
Revised: Benign (Chatterji et al., 2021) (Frei et al., 2022)

Original: (Wang et al., 2021a)
Revised: (Wang et al., 2021a)

Original: Benign (Bartlett et al., 2020)
Revised: Benign (Bartlett et al., 2020)

Original: (Zou et al., 2021) In particular, we first trained ResNet18 on CIFAR10 for 200 epochs and the model interpolates the training data. In addition, we also trained ResNet50 on ImageNet for 500 epochs, as opposed to the common schedule that stops at 90 epochs. Surprisingly, we found that although benign overfitting happens on the CIFAR10 dataset, overfitting is not benign on the ImageNet dataset—the test loss increased as the model further fits the train set.
Revised: (Zou et al., 2021) where p denotes the number of parameters and n denotes the training sample size (Wang et al., 2021a;

Original: More precisely, the ImageNet experiment does not overfit benignly since the best model is achieved in the middle of the training.

Original: The different overfitting behaviors cannot be explained by known analysis for classification tasks, as no negative results have been studied yet.

Original: Motivated by the above observation, our work aims to understand the cause for the two different overfitting behaviors in ImageNet and CIFAR10, and to reconcile the empirical phenomenon with previous analysis on benign overfitting.

Original: Our first hint comes from the level of overparameterization.

Original: Previous results on benign overfitting in the classification setting usually requires that p = ω ( n ) , where p denotes the number of parameters and n denotes the training sample size (Wang et al., 2021a;

Original: Cao et al., 2021; Chatterji et al., 2021;
Revised: Cao et al., 2021; Chatterji et al., 2021;

Original: Frei et al., 2022).
Revised: Frei et al., 2022).

Original: However, in practice many deep learning models fall in the mild overparameterization regime, where the number of parameters is only slightly larger than the number of samples despite overparameterization.
Revised: However, in practice many deep learning models fall in the mild overparameterization regime, where the number of parameters is only slightly larger than the number of samples despite overparameterization.

Original: In our case, the sample size n = 10in ImageNet, whereas the parameter size p ≈ 10 7 in ResNets.
Revised: In our case, the sample size n = 10in ImageNet, whereas the parameter size p ≈ 10 7 in ResNets.

Original: To close the gap, we study the overfitting behavior of classification models under the mild overparameterization setup where p =
Revised: To close the gap, we study the overfitting behavior of classification models under the mild overparameterization setup where p =

Original: Θ( n ) (this is sometimes referred to as the asymptotic regime).
Revised: Θ( n ) (this is sometimes referred to as the asymptotic regime).

Original: In particular, following Wang et al.
Revised: In particular, following Wang et al.

Original: (2021a)
Revised: (2021a)

Original: ; Cao et al.
Revised: ; Cao et al.

Original: ; Chatterji et al.
Revised: ; Chatterji et al.

Original: Frei et al.
Revised: Frei et al.

Original: (2022), we analyze the solution of stochastic gradient descent for the Gaussian mixture models.
Revised: (2022), we analyze the solution of stochastic gradient descent for the Gaussian mixture models.

Original: We found that a phase change happens when we move from p =
Revised: We found that a phase change happens when we move from p =

Original: Ω( n log n ) (studied in Wang et al.
Revised: Ω( n log n ) (studied in Wang et al.

Original: (2021a))
Revised: (2021a))

Original: to p =
Revised: to p =

Original: Θ( n ) .
Revised: Θ( n ) .

Original: Unlike previous analysis, we show that benign overfitting now provably fails in the presence of label noise (see Table 1 and Figure 2).
Revised: Unlike previous analysis, we show that benign overfitting now provably fails in the presence of label noise (see Table 1 and Figure 2).

Original: This aligns with our empirical findings as
Revised: This aligns with our empirical findings as

Original: ImageNet is known to suffer from mislabelling and multi-labels (Yun et al., 2021; Shankar et al., 2020).
Revised: ImageNet is known to suffer from mislabelling and multi-labels (Yun et al., 2021; Shankar et al., 2020).

Original: More specifically, our analysis (see Theorem 3.1 for details) under the mild overparameterization ( p = Θ( n ) ) setup supports the following statements that align with our empirical observations in
Revised: More specifically, our analysis (see Theorem 3.1 for details) under the mild overparameterization ( p = Θ( n ) ) setup supports the following statements that align with our empirical observations in

Original: Figure 1 and 2:
Revised: Figure 1 and 2:

Original: • When the labels are noiseless, benign overfitting holds under similar conditions as in previous analyses.
Revised: • When the labels are noiseless, benign overfitting holds under similar conditions as in previous analyses.

Original: • When the labels are noisy, the interpolating solution can provably lead to a positive excess risk that does not diminish with the sample size.
Revised: • When the labels are noisy, the interpolating solution can provably lead to a positive excess risk that does not diminish with the sample size.

Original: • When the labels are noisy, early stopping can provably lead to better generalization performances compared to interpolating models.
Revised: • When the labels are noisy, early stopping can provably lead to better generalization performances compared to interpolating models.

Original: 4 5 6 7 8 9 10 11 12 13log p 456789 l o g n 0.
Revised: 4 5 6 7 8 9 10 11 12 13log p 456789 l o g n 0.

Original: (a) Noiseless 4 5 6 7 8 9 10 11 12 13log p 456789 l o g n
Revised: (a) Noiseless 4 5 6 7 8 9 10 11 12 13log p 456789 l o g n

Original: Mild
Revised: Mild

Original: Heavy 0.
Revised: Heavy 0.

Original: (b) Noisy 4 5 6 7 8 9 10 11 12 13log p 456789 l o g n
Revised: (b) Noisy 4 5 6 7 8 9 10 11 12 13log p 456789 l o g n

Original: Mild
Revised: Mild

Original: Heavy 0.
Revised: Heavy 0.

Original: [Figure]
Revised: [Figure]

Original: Our analysis is supported by both synthetic and deep learning experiments.
Revised: Our analysis is supported by both synthetic and deep learning experiments.

Original: First, we train a linear classifier in Figure 2.
Revised: First, we train a linear classifier in Figure 2.

Original: Figure (a) shows that training on noiseless data does not cause overfitting,
Revised: Figure (a) shows that training on noiseless data does not cause overfitting,

Original: Figure (b) shows that training on noisy data with a label noise ratio 0.4 causes overfitting with mild overparameterization, and Figure (c) shows that applying early-stopping on the same noisy data can effectively avoid overfitting.
Revised: Figure (b) shows that training on noisy data with a label noise ratio 0.4 causes overfitting with mild overparameterization, and Figure (c) shows that applying early-stopping on the same noisy data can effectively avoid overfitting.

Original: Besides, we find that overfitting consistently happens under noisy regimes, despite the theoretical guarantee that the model overfits benignly when p = ω ( n ) .
Revised: Besides, we find that overfitting consistently happens under noisy regimes, despite the theoretical guarantee that the model overfits benignly when p = ω ( n ) .

Original: Furthermore, our analysis is validated by control experiments in Section 4 with ResNets.
Revised: Furthermore, our analysis is validated by control experiments in Section 4 with ResNets.

Original: We first show that injecting random labels into the CIFAR10 dataset indeed leads to the failure of benign overfitting.
Revised: We first show that injecting random labels into the CIFAR10 dataset indeed leads to the failure of benign overfitting.

Original: We further show that, surprisingly, increasing the width of ResNet alone can alleviate overfitting incorrect labels, as our theory predicted.
Revised: We further show that, surprisingly, increasing the width of ResNet alone can alleviate overfitting incorrect labels, as our theory predicted.

Original: More broadly, the empirical and theoretical results in our work point out that modern models may not operate under the benign interpolating regime.
Revised: More broadly, the empirical and theoretical results in our work point out that modern models may not operate under the benign interpolating regime.

Original: Hence, it highlights the importance of characterizing implicit bias when the model, though mildly overparameterized, does not interpolate the training data.
Revised: Hence, it highlights the importance of characterizing implicit bias when the model, though mildly overparameterized, does not interpolate the training data.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Although the benign overfitting phenomenon has been systematically studied both empirically (Belkin et al., 2018b; 2019; Nakkiran et al., 2021b) and theoretically (see later), our work differs in that we aim to understand why benign overfitting fails in the classification setup.
Revised: Although the benign overfitting phenomenon has been systematically studied both empirically (Belkin et al., 2018b; 2019; Nakkiran et al., 2021b) and theoretically (see later), our work differs in that we aim to understand why benign overfitting fails in the classification setup.

Original: In particular, among all theoretical studies, the closest ones to us are Chatterji et al.
Revised: In particular, among all theoretical studies, the closest ones to us are Chatterji et al.

Original: (2021), Wang et al.
Revised: (2021), Wang et al.

Original: (2021b)
Revised: (2021b)

Original: and
Revised: and

Original: Cao et al.
Revised: Cao et al.

Original: (2021), as the analyses are done under the classification setup with Gaussian mixture models.
Revised: (2021), as the analyses are done under the classification setup with Gaussian mixture models.

Original: Our work provides new results by moving from heavy overparameterization p = ω ( n ) to mild overparameterization p =
Revised: Our work provides new results by moving from heavy overparameterization p = ω ( n ) to mild overparameterization p =

Original: Θ( n ) .
Revised: Θ( n ) .

Original: We found that this new condition leads to an analysis that is consistent with our empirical observations, and can explain why benign overfitting fails.
Revised: We found that this new condition leads to an analysis that is consistent with our empirical observations, and can explain why benign overfitting fails.

Original: In short, compared to all previous studies on benign overfitting in classification tasks, we focus on identifying the condition that breaks benign overfitting in our ImageNet experiments.
Revised: In short, compared to all previous studies on benign overfitting in classification tasks, we focus on identifying the condition that breaks benign overfitting in our ImageNet experiments.

Original: The comparison is summarized in Table 1.
Revised: The comparison is summarized in Table 1.

Original: More related works on benign overfitting: Researchers have made a lot of efforts to generalize the notation of benign overfitting beyond the novel work on linear regression (Bartlett et al., 2020), e.g. , variants of linear regression (Tsigler et al., 2020; Muthukumar et al., 2020;
Revised: More related works on benign overfitting: Researchers have made a lot of efforts to generalize the notation of benign overfitting beyond the novel work on linear regression (Bartlett et al., 2020), e.g. , variants of linear regression (Tsigler et al., 2020; Muthukumar et al., 2020;

Original: Zou et al., 2021), linear classification (Liang et al., 2020a; Belkin et al., 2018a) with different distribution assumptions(instead of Gaussian mixture model), kernel-based estimators (Liang et al., 2018; 2020b; Mei et al., 2022), neural networks (Frei et al., 2022).
Revised: Zou et al., 2021), linear classification (Liang et al., 2020a; Belkin et al., 2018a) with different distribution assumptions (instead of Gaussian mixture model), kernel-based estimators (Liang et al., 2018; 2020b; Mei et al., 2022), neural networks (Frei et al., 2022).

Original: Among them, Cao et al.
Revised: Among them, Cao et al.

Original: (2022) also identifies a phase change in benign overfitting when the overfitting can be provably harmful with a different data distribution under noiseless regimes.
Revised: (2022) also identifies a phase change in benign overfitting when the overfitting can be provably harmful with a different data distribution under noiseless regimes.

Original: Gaussian Mixture Model (GMM) represents the data distribution where the input is drawn from a
Revised: Gaussian Mixture Model (GMM) represents the data distribution where the input is drawn from a

Original: Gaussian distribution with different centers for each class.
Revised: Gaussian distribution with different centers for each class.

Original: The model was widely studied in hidden
Revised: The model was widely studied in hidden

Original: Markov models, anomaly detection, and many other fields (Hastie et al., 2001;
Revised: Markov models, anomaly detection, and many other fields (Hastie et al., 2001;

Original: Xuan et al., 2001;
Revised: Xuan et al., 2001;

Original: Reynolds, 2009; Zong et al., 2018).
Revised: Reynolds, 2009; Zong et al., 2018).

Original: A closely related work is Jin (2009) which analyzes the lower bound for excess risk of the Gaussian Mixture Model under noiseless regimes.
Revised: A closely related work is Jin (2009) which analyzes the lower bound for excess risk of the Gaussian Mixture Model under noiseless regimes.

Original: However, their analysis cannot be directly extended to either the overparameterization or noisy label regimes.
Revised: However, their analysis cannot be directly extended to either the overparameterization or noisy label regimes.

Original: Another closely related work (Mai et al., 2019) focuses on the GMM setting with mild overparameterization, but they require a noiseless label regime and rely on a small signal-to-noise ratio on the input, and thus cannot be generalized to our theoretical results.
Revised: Another closely related work (Mai et al., 2019) focuses on the GMM setting with mild overparameterization, but they require a noiseless label regime and rely on a small signal-to-noise ratio on the input, and thus cannot be generalized to our theoretical results.

Original: Asymptotic (Mildly Overparameterized) Regimes.
Revised: Asymptotic (Mildly Overparameterized) Regimes.

Original: This paper considers asymptotic regimes where the ratio of parameter dimension and the sample size is upper and lower bounded by absolute constants.
Revised: This paper considers asymptotic regimes where the ratio of parameter dimension and the sample size is upper and lower bounded by absolute constants.

Original: Previous work studied this setup with different focus than benign overfitting ( e.g. , double descent), both in regression perspective (Hastie et al., 2019) and classification perspective (Sur et al., 2019; Mai et al., 2019;
Revised: Previous work studied this setup with different focus than benign overfitting ( e.g. , double descent), both in regression perspective (Hastie et al., 2019) and classification perspective (Sur et al., 2019; Mai et al., 2019;

Original: Deng et al., 2019).
Revised: Deng et al., 2019).

Original: We study the mild overparameterization case because it generally happens in realistic machine learning tasks, where the number of parameters exceeds the number of training samples but not extremely.
Revised: We study the mild overparameterization case because it generally happens in realistic machine learning tasks, where the number of parameters exceeds the number of training samples but not extremely.

Original:  O VERFITTING UNDER M ILD O VERPARAMETERIZATION
Revised:  O VERFITTING UNDER M ILD O VERPARAMETERIZATION

Original: We observed in Figure 1 that training ResNets on CIFAR10 and ImageNet can result in different overfitting behaviors.
Revised: We observed in Figure 1 that training ResNets on CIFAR10 and ImageNet can result in different overfitting behaviors.

Original: This discrepancy was not reflected in the previous analysis of benign overfitting in classification tasks.
Revised: This discrepancy was not reflected in the previous analysis of benign overfitting in classification tasks.

Original: In this section, we provide a theoretical analysis by studying the Gaussian mixture model under the mild overparameterization setup.
Revised: In this section, we provide a theoretical analysis by studying the Gaussian mixture model under the mild overparameterization setup.

Original: Our analysis shows that mild overparameterization along with label noise can break benign overfitting.
Revised: Our analysis shows that mild overparameterization along with label noise can break benign overfitting.

Original: 3. O VERPARAMETERIZED G AUSSIAN M IXTURE M ODEL
Revised: 3. O VERPARAMETERIZED G AUSSIAN M IXTURE M ODEL

Original: In this subsection, we study the generalization performance of linear models on the Gaussian Mixture
Revised: In this subsection, we study the generalization performance of linear models on the Gaussian Mixture

Original: Model (GMM).
Revised: Model (GMM).

Original: We assume that linear models are obtained by solving logistic regression with stochastic gradient descent (SGD).
Revised: We assume that linear models are obtained by solving logistic regression with stochastic gradient descent (SGD).

Original: This simplified model may help explain phenomenons in neural networks, since previous works show that neural networks converge to linear models as the width goes to infinity (Arora et al., 2019; Allen-Zhu et al., 2019).
Revised: This simplified model may help explain phenomenons in neural networks, since previous works show that neural networks converge to linear models as the width goes to infinity (Arora et al., 2019; Allen-Zhu et al., 2019).

Original: We next introduce GMM under two setups of overparameterized linear classification: the noiseless regime and the noisy regime.
Revised: We next introduce GMM under two setups of overparameterized linear classification: the noiseless regime and the noisy regime.

Original: Noiseless Regime.
Revised: Noiseless Regime.

Original: Let y ∼ Unif {− 1 , 1 } ∈ R denote the ground truth label.
Revised: Let y ∼ Unif {− 1 , 1 } ∈ R denote the ground truth label.

Original: The corresponding feature is generated by
Revised: The corresponding feature is generated by

Original: [Equation]
Revised: [Equation]

Original: where ϵ denotes noise drawn from a subGaussian distribution.
Revised: where ϵ denotes noise drawn from a subGaussian distribution, and µ ∈ R p denotes the signal vector.

Original: We denote the dataset D
Revised: We denote the dataset D

Original: = { ( x i , y i ) } i ∈ [ n ] where ( x i , y i ) are generated by the above mechanism.
Revised: = { ( x i , y i ) } i ∈ [ n ] where ( x i , y i ) are generated by the above mechanism.

Original: Noisy Regime with contamination rate ρ .
Revised: Noisy Regime with contamination rate ρ .

Original: For the noisy regime, we first generate noiseless data ( x , y ) using the noiseless regime, and then consider the data point ( x , ˜ y ) where ˜ y is the contaminated version of y .
Revised: For the noisy regime, we first generate noiseless data ( x , y ) using the noiseless regime, and then consider the data point ( x , ˜ y ) where ˜ y is the contaminated version of y .

Original: Formally, given the contamination rate ρ , the contaminated label ˜ y
Revised: Formally, given the contamination rate ρ , the contaminated label ˜ y

Original: =
Revised: =

Original: − y with probability ρ and ˜ y
Revised: − y with probability ρ and ˜ y

Original: = y with probability 1 − ρ .
Revised: = y with probability 1 − ρ .

Original: And the returned dataset is ˜ D
Revised: And the returned dataset is ˜ D

Original: = { ( x i , ˜ y i ) } .
Revised: = { ( x i , ˜ y i ) } .

Original: For simplicity, we assume that the data points in the train set are linearly separable in both noiseless and noisy regimes.
Revised: For simplicity, we assume that the data points in the train set are linearly separable in both noiseless and noisy regimes.

Original: This assumption holds almost surely under mild overparameterization.
Revised: This assumption holds almost surely under mild overparameterization.

Original: Besides, we make the following assumptions about the data distribution:
Revised: Besides, we make the following assumptions about the data distribution:

Original: Assumption 1 (Assumptions on the data distribution) 
Revised: Assumption 1 (Assumptions on the data distribution) .

Original: A

Revised: A

Original: The feature noise ϵ is drawn from the Gaussian distribution, i e., ϵ ∼ N (0 , σ 2 I ) .
Revised: The feature noise ϵ is drawn from the Gaussian distribution, i.e., ϵ ∼ N (0 , σ 2 I ) .

Original: A2 The signal-to-noise ratio satisfies ∥ µ ∥ σ ≥ c √ log n for a given constant c .
Revised: A2 The signal-to-noise ratio satisfies ∥ µ ∥ σ ≥ c √ log n for a given constant c .

Original: A3 The ratio p/n = r > 1 is a fixed constant.
Revised: A3 The ratio p/n = r > 1 is a fixed constant.

Original: The three assumptions are all crucial but can be made slightly more general (see Section 5).
Revised: The three assumptions are all crucial but can be made slightly more general (see Section 5).

Original: The first
Revised: The first

Original: Assumption [A1] stems from the requirement to derive a lower bound for excess risk under a noisy regime.
Revised: Assumption [A1] stems from the requirement to derive a lower bound for excess risk under a noisy regime.

Original: The second Assumption [A2] is widely used in the analysis (Chatterji et al., 2021;
Revised: The second Assumption [A2] is widely used in the analysis (Chatterji et al., 2021;

Original: Frei et al.,
Revised: Frei et al.,

Original: For a smaller ratio, the model may be unable to learn even under the noiseless regime and return vacuous bounds.
Revised: For a smaller ratio, the model may be unable to learn even under the noiseless regime and return vacuous bounds.

Original: The third Assumption [A3] differs from the previous analysis, where we consider a mild overparameterization instead of heavy overparameterization (i.e., p = ω ( n ) ).
Revised: The third Assumption [A3] differs from the previous analysis, where we consider a mild overparameterization instead of heavy overparameterization (i.e., p = ω ( n ) ).

Original: 3. T RAINING P ROCEDURE
Revised: 3. T RAINING P ROCEDURE

Original: We consider the multi-pass SGD training with logistic loss ℓ ( w ; x , y )
Revised: We consider the multi-pass SGD training with logistic loss ℓ ( w ; x , y )

Original: =
Revised: =

Original: log(1 + exp( − y x ⊤ w )) .
Revised: log(1 + exp( − y x ⊤ w )) .

Original: During each epoch, each data is visited exactly once randomly without replacement.
Revised: During each epoch, each data is visited exactly once randomly without replacement.

Original: Formally, at the beginning of each epoch E , we uniformly random sample a permutation P E : { 1 , ..., n } → { 1 , ..., n } , then at iteration t , given the learning rate η we have
Revised: Formally, at the beginning of each epoch E , we uniformly random sample a permutation P E : { 1 , ..., n } → { 1 , ..., n } , then at iteration t , given the learning rate η we have

Original: [Equation]
Revised: [Equation]

Original: where x ( t ) =
Revised: where x ( t ) =

Original: x P E ( i ) , y ( t ) =
Revised: x P E ( i ) , y ( t ) =

Original: y P E ( i ) , given t = nE
Revised: y P E ( i ) , given t = nE

Original: + i, 1 ≤ i ≤ n .
Revised: + i, 1 ≤ i ≤ n .

Original: Under the above procedure, Proposition 3.1 shows that the classifier under the GMM regime with multi-pass SGD training will converge in the direction of the max-margin interpolating classifier.
Revised: Under the above procedure, Proposition 3.1 shows that the classifier under the GMM regime with multi-pass SGD training will converge in the direction of the max-margin interpolating classifier.

Original: Proposition 3.1 (Interpolator of multi-pass SGD under GMM regime, from Nacson et al.
Revised: Proposition 3.1 (Interpolator of multi-pass SGD under GMM regime, from Nacson et al.

Original: Under the regime of GMM with logistic loss, denote the iterates in multi-pass SGD by w ( t ) .
Revised: Under the regime of GMM with logistic loss, denote the iterates in multi-pass SGD by w ( t ) .

Original: Then for any initialization w (0) , the iterates w ( t ) converges to the max-margin solution almost surely, namely,
Revised: Then for any initialization w (0) , the iterates w ( t ) converges to the max-margin solution almost surely, namely,

Original: [Equation]
Revised: [Equation]

Original: where ˜ w = argmin w ∈ R d ∥ w ∥ 2 s.t.
Revised: where ˜ w = argmin w ∈ R d ∥ w ∥ 2 s.t.

Original: w ⊤ x i ≥ 1 , ∀ i ∈ [ n ] denotes the max-margin solution.
Revised: w ⊤ x i ≥ 1 , ∀ i ∈ [ n ] denotes the max-margin solution.

Original: For simplicity, we denote w + ( t ) as the parameter at iteration t in the noiseless setting, and w − ( t ) as the parameter in the noisy setting.
Revised: For simplicity, we denote w + ( t ) as the parameter at iteration t in the noiseless setting, and w − ( t ) as the parameter in the noisy setting.

Original: By the proposition above, we know that both w + ( ∞ ) and w − ( ∞ ) are max-margin classifiers on the training data points.
Revised: By the proposition above, we know that both w + ( ∞ ) and w − ( ∞ ) are max-margin classifiers on the training data points.

Original: During the evaluation process, we also focus on the 0-1 loss, where the population 0-1 loss is L 01 ( w ) =
Revised: During the evaluation process, we also focus on the 0-1 loss, where the population 0-1 loss is L 01 ( w ) =

Original: P ( y x ⊤ w < 0) .
Revised: P ( y x ⊤ w < 0) .

Original: 3. M AIN T HEOREM
Revised: 3. M AIN T HEOREM

Original: Based on the above assumptions and discussions, we state the following Theorem 3.1, indicating the different performances between noiseless and noisy settings.
Revised: Based on the above assumptions and discussions, we state the following Theorem 3.1, indicating the different performances between noiseless and noisy settings.

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: We consider the above GMM regime with Assumption [A1-A3].
Revised: We consider the above GMM regime with Assumption [A1-A3].

Original: Specifically, denote the noise level by ρ and the mild overparameterization ratio by r = p/n .
Revised: Specifically, denote the noise level by ρ and the mild overparameterization ratio by r = p/n .

Original: Then there exists absolute constant c 1 , c 2 , c 3 , c 4 , c 5 > 0 such that the following statements hold with probability 1 at least 1 − c 1 /n :
Revised: Then there exists absolute constant c 1 , c 2 , c 3 , c 4 , c 5 > 0 such that the following statements hold with probability 2 at least 1 − c 1 /n :

Original: Under the noiseless setting, the max-margin classifier w + ( ∞ ) obtained from SGD has a non-vacuous 0-1 loss, namely,
Revised: Under the noiseless setting, the max-margin classifier w + ( ∞ ) obtained from SGD has a non-vacuous 0-1 loss, namely,

Original: [Equation]
Revised: [Equation]

Original: Under the noisy setting, the max-margin classifier w − ( ∞ ) has vacuous 0-1 loss with a constant lower bound, namely, the following inequality holds for any training sample size n ,
Revised: Under the noisy setting, the max-margin classifier w − ( ∞ ) has vacuous 0-1 loss with a constant lower bound, namely, the following inequality holds for any training sample size n ,

Original: [Equation]
Revised: [Equation]

Original: Under the noisy setting, if the learning rate satisfies η <
Revised: Under the noisy setting, if the learning rate satisfies η <

Original: [Equation]
Revised: [Equation]

Original: , there exists a time t such that the trained early-stopping classifier w − ( t ) has non-vacuous 0-1 loss, namely,
Revised: , there exists a time t such that the trained early-stopping classifier w − ( t ) has non-vacuous 0-1 loss, namely,

Original: [Equation]
Revised: [Equation]

Original: Intuitively, Theorem 3.1 illustrates that although SGD leads to benign overfitting under noiseless regimes (statement 1), it provably overfits when the labels are noisy (statement 2).
Revised: Intuitively, Theorem 3.1 illustrates that although SGD leads to benign overfitting under noiseless regimes (statement 1), it provably overfits when the labels are noisy (statement 2).

Original: In particular, it
Revised: In particular, it

Original: [Figure]
Revised: [Figure]

Original: incurs Ω(1) error on noiseless data and hence would incur Ω(1) + ρ error on noisy labeled data, since label noise in the test set is independent of the algorithm.
Revised: incurs Ω(1) error on noiseless data and hence would incur Ω(1) + ρ error on noisy labeled data, since label noise in the test set is independent of the algorithm.

Original: Furthermore, statement 3 shows that overfitting is avoidable through early stopping.
Revised: Furthermore, statement 3 shows that overfitting is avoidable through early stopping.

Original: One may doubt the fast convergence rate  in Statement One and Statement Three, which seems too good to be true.
Revised: One may doubt the fast convergence rate 3 in Statement One and Statement Three, which seems too good to be true.

Original: The strange phenomenon happens because we distinguish  the randomness in the training set from the randomness in the test set.
Revised: The strange phenomenon happens because we distinguish between the randomness in the training set and the randomness in the test set.

Original: Note that the high probability guarantee is in order O (1 /n ) [Figure] and therefore, the total 0-1 loss is approximate O (1 /n ) after union bound.
Revised: Note that the high probability guarantee is in order O (1 /n ) , and therefore, the total 0-1 loss is approximate O (1 /n ) after union bound.

Original: We refer to Cao et al.
Revised: We refer to Cao et al.

Original: ; Wang et al.
Revised: ; Wang et al.

Original: (2021a)
Revised: (2021a)

Original: for similar types of bounds.
Revised: for similar types of bounds.

Original: Remark 1 (Comparison against heavy overparameterization) .
Revised: Remark 1 (Comparison against heavy overparameterization) .

Original: Previous work usually analyze the
Revised: Previous work usually analyze the

Original: GMM model under the heavy overparameterization regime, e.g. , p = Ω( n 2 )
Revised: GMM model under the heavy overparameterization regime, e.g. , p = Ω( n 2 )

Original: (Cao et al., 2021;
Revised: (Cao et al., 2021;

Original: Chatterji et al., 2021) or
Revised: Chatterji et al., 2021) or

Original: p = Ω( n log( n ))
Revised: p = Ω( n log( n ))

Original: (Wang et al., 2021a).
Revised: (Wang et al., 2021a).

Original: In comparison, our paper focuses on the mild overparameterization regime, where p =
Revised: In comparison, our paper focuses on the mild overparameterization regime, where p =

Original: Θ( n ) .
Revised: Θ( n ) .

Original: We note that this leads to a phase change that the overfitting model under noisy settings now provably overfits.
Revised: We note that this leads to a phase change that the overfitting model under noisy settings now provably overfits.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Theorem 3.1 claims the crucial role of label noise and overparameterization level in benign overfitting.
Revised: Theorem 3.1 claims the crucial role of label noise and overparameterization level in benign overfitting.

Original: This section aims at providing experimental evidence to validate the statements.
Revised: This section aims at providing experimental evidence to validate the statements.

Revised: However, as the dataset size is upper bounded, the criterion generalization error converges to zero is hard to verify in experiments.

Revised: Instead, we assume that (1) the gap of verification accuracy between the best-epoch model and the last-epoch model can work as an efficient proxy for the gap between the Bayesian optimal accuracy and the validation accuracy of the last-epoch model, and (2) training the model for long-enough epochs can work as an honest proxy for training for infinite epochs.

Revised: Therefore, as stated in the introduction part, we adopt a different notion of benign overfitting, namely, validation performance does not drop while the model fits more training data points.

Original: Specifically, we focus on the noisy CIFAR10 dataset, where we randomly flip a portion of the labels.
Revised: Specifically, we focus on the noisy CIFAR10 dataset, where we randomly flip a portion of the labels.

Original: From the label noise perspective, Section 4.1 demonstrates that ResNets on noisy CIFAR10 would dramatically overfit non-benignly ( c.f. , ResNets on clean CIFAR10 overfit benignly).
Revised: From the label noise perspective, Section 4.1 demonstrates that ResNets on noisy CIFAR10 would dramatically overfit non-benignly ( c.f. , ResNets on clean CIFAR10 overfit benignly).

Original: From the overparameterization level perspective, Section 4.2 demonstrates that a heavier overparameterization level helps increase the last-iterate performance and hence leads to benign overfitting.
Revised: From the overparameterization level perspective, Section 4.2 demonstrates that a heavier overparameterization level helps increase the last-iterate performance and hence leads to benign overfitting.

Original: 4. L ABEL N OISE M ATTERS : O VERFITTING IN N OISY CIFAR

Original: In Section 3.1, we prove that under noisy label regimes with mild overparameterization, early-stopping classifiers and interpolators perform differently.

Original: This section aims to verify if the phenomenon empirically happens in the real-world dataset.

Original: Specifically, we generate a noisy CIFAR10 dataset where each label is randomly flipped with probability ρ , We train the ResNets again and the results show that the test error first increases and then dramatically decreases, demonstrating that the interpolator performs worse than the models in the middle of the training.

Original: Width Factor
Revised: Width Factor

Original: V a li d a t i o n
Revised: V a li d a t i o n

Original: A cc ( % )
Revised: A cc ( % )

Original: Last Epoch
Revised: Last Epoch

Original: Best Epoch
Revised: Best Epoch

Original: [Figure]
Revised: [Figure]

Revised: 4. L ABEL N OISE M ATTERS : O VERFITTING IN N OISY CIFAR

Revised: In Section 3.1, we prove that under noisy label regimes with mild overparameterization, early-stopping classifiers and interpolators perform differently.

Revised: This section aims to verify if the phenomenon empirically happens in the real-world dataset.

Revised: Specifically, we generate a noisy CIFAR10 dataset where each label is randomly flipped with probability ρ , We train the ResNets again and the results show that the test error first increases and then dramatically decreases, demonstrating that the interpolator performs worse than the models in the middle of the training.

Original: Setup.
Revised: Setup.

Original: The base dataset is CIFAR10, where each sample is randomly flipped with probability ρ ∈ { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 } .
Revised: The base dataset is CIFAR10, where each sample is randomly flipped with probability ρ ∈ { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 } .

Original: We use ResNet18 and SGD to train the model with cosine learning rate decay.
Revised: We use ResNet18 and SGD to train the model with cosine learning rate decay.

Original: For each model, we train for 200 epochs, test the validation accuracy and plot the training accuracy and validation accuracy in Figure 3.
Revised: For each model, we train for 200 epochs, test the validation accuracy and plot the training accuracy and validation accuracy in Figure 3.

Original: More details can be found in the code.
Revised: More details can be found in the code.

Original: Figure 3 illustrates a similar phenomenon to the results in linear models under mild overparameterization analysis (see Section 3.1).
Revised: Figure 3 illustrates a similar phenomenon to the results in linear models under mild overparameterization analysis (see Section 3.1).

Original: Precisely, the interpolator achieves suboptimal accuracy under the mild overparameterization regimes, but we can still find a better classifier through early stopping.
Revised: Precisely, the interpolator achieves suboptimal accuracy under the mild overparameterization regimes, but we can still find a better classifier through early stopping.

Original: In
Revised: In

Original: Figure 3, such a phenomenon is manifested as the validation accuracy curve increases and decreases.
Revised: Figure 3, such a phenomenon is manifested as the validation accuracy curve increases and decreases.

Original: We also notice that the degree becomes sharper as the noise level increases.
Revised: We also notice that the degree becomes sharper as the noise level increases.

Original: 4. A CHIEVE B ENIGN O VERFITTING BY I NCREASING O VERPARAMETERIZATION L EVEL
Revised: 4. A CHIEVE B ENIGN O VERFITTING BY I NCREASING O VERPARAMETERIZATION L EVEL

Original: Theorem 3.1 implies that although noisy label leads to bad overfitting, increasing overparameterization can turn overfitting benign.
Revised: Theorem 3.1 implies that although noisy label leads to bad overfitting, increasing overparameterization can turn overfitting benign.

Original: Specifically, when the overparameterization ratio r = p/n increases,
Revised: Specifically, when the overparameterization ratio r = p/n increases,

Original: (1) The last-iterate classifier (max-margin classifier) may generalize better (see Argument 2 in Theorem 3.1), since the lower bound ρc 3 r exp( − c 3 rρ ) decreases with r .
Revised: (1) The last-iterate classifier (max-margin classifier) may generalize better (see Argument 2 in Theorem 3.1), since the lower bound ρc 3 r exp( − c 3 rρ ) decreases with r .

Original: The early-stopping classifiers have similar generalization performances (see Argument 3 in Theorem 3.1), since the upper bound is in order n − c 4 which converges to zero as the sample size goes to infinity.
Revised: The early-stopping classifiers have similar generalization performances (see Argument 3 in Theorem 3.1), since the upper bound is in order n − c 4 which converges to zero as the sample size goes to infinity.

Original: Figure 2 demonstrates the two observations on the synthetic dataset (linear models).
Revised: Figure 2 demonstrates the two observations on the synthetic dataset (linear models).

Original: We next show similar observations on noisy CIFAR-10.
Revised: We next show similar observations on noisy CIFAR-10.

Original: Specifically, We conduct an ablation study on CIFAR dataset with a fixed label noise ratio ρ = 0 .
Revised: Specifically, We conduct an ablation study on CIFAR dataset with a fixed label noise ratio ρ = 0 .

Original: We increase the width of ResNets and effectively turn
Revised: We increase the width of ResNets and effectively turn

Original: ResNet into Wide-ResNets.
Revised: ResNet into Wide-ResNets.

Original: By varying the width factor W of WideResNet16- W , we show that these observations remain in the deep learning regime.
Revised: By varying the width factor W of WideResNet16- W , we show that these observations remain in the deep learning regime.

Original: Setup.
Revised: Setup.

Original: The base dataset is CIFAR10, where each sample is randomly flipped with probability ρ = 0 .
Revised: The base dataset is CIFAR10, where each sample is randomly flipped with probability ρ = 0 .

Original: We use the WideResNet16- W for W ∈ { 1 , 1 . 25 , 1 . 5 , 1 . 75 , 2 , 3 , 4 } and use SGD to train the model with cosine learning rate decay.
Revised: We use the WideResNet16- W for W ∈ { 1 , 1 . 25 , 1 . 5 , 1 . 75 , 2 , 3 , 4 } and use SGD to train the model with cosine learning rate decay.

Original: For each model, we train for 200 epochs, test the validation accuracy and compare the best validation performance and the validation performance upon convergence in Figure 4.
Revised: For each model, we train for 200 epochs, test the validation accuracy and compare the best validation performance and the validation performance upon convergence in Figure 4.

Original: Figure 4 verifies our theoretical observation where the early stopped classifier enjoys generalization performance nearly independent of overparameterization level, again stressing the necessity to study the bias of neural networks beyond the interpolation regime.
Revised: Figure 4 verifies our theoretical observation where the early stopped classifier enjoys generalization performance nearly independent of overparameterization level, again stressing the necessity to study the bias of neural networks beyond the interpolation regime.

Original: Besides, we observe that the validation performance of the converged classifier increases with respect to the overparameterization level.
Revised: Besides, we observe that the validation performance of the converged classifier increases with respect to the overparameterization level.

Original: Moreover, we notice that the increment is more significant for smaller overparameterization level, which is qualitatively similar to ρc 3 r exp( − c 3 rρ ) .
Revised: Moreover, we notice that the increment is more significant for smaller overparameterization level, which is qualitatively similar to ρc 3 r exp( − c 3 rρ ) .

Original: We further observe the epoch-wise double descent phenomenonfrom the curve in Figure 4, where the accuracy first decreases and then increases while the model reaches interpolation (Nakkiran et al., 2020; Heckel et al., 2021).
Revised: We further observe the epoch-wise double descent phenomenon in Figure 4, where the accuracy first decreases and then increases while the model reaches interpolation (Nakkiran et al., 2020; Heckel et al., 2021).

Original: Stephenson et al.
Revised: Stephenson et al.

Original: (2021) pointed out that the phenomenon is also closely related to label noise and overparameterization level.
Revised: (2021) argues that the phenomenon is also closely related to label noise and overparameterization level.

Original: Their work differs from ours in that they focus on studying the training dynamics and removing the double-descent phenomenon.
Revised: Their work differs from ours in that they focus on studying the training dynamics and removing the double-descent phenomenon.

Original: We do not observe a similar phenomenon in the ImageNet experiment with mild overparameterization in 500 epochs.
Revised: We do not observe a similar phenomenon in the ImageNet experiment with mild overparameterization in 500 epochs.

Original: We leave the discussion of epoch-wise double descent under mild overparameterization as future work.
Revised: We leave the discussion of epoch-wise double descent under mild overparameterization as future work.

Original:  C HALLENGES IN P ROVING T HEOREM 3. 
Revised:  C HALLENGES IN P ROVING T HEOREM 3.

Original: This section provides more details about the three statements in Theorem 3.1 with milder assumptions than those in Assumption 1.
Revised: This section provides more details about the three statements in Theorem 3.1 with milder assumptions than those in Assumption 1.

Original: We also explain why existing analysis cannot be readily applied.
Revised: We also explain why existing analysis cannot be readily applied.

Original: Assumption 2.
Revised: Assumption 2.

Original: The following assumptions are more general,
Revised: The following assumptions are more general,

Original: A4 The noise ϵ in x is generated from a σ -subGaussian distribution.
Revised: A4 The noise ϵ in x is generated from a σ -subGaussian distribution.

Original: A

Revised: A

Original: The signal-to-noise ratio satisfies ∥ µ ∥ σ ≥ c 6 (cid:0) pn (cid:1) 12 .
Revised: The signal-to-noise ratio satisfies ∥ µ ∥ σ ≥ c 6 (cid:0) pn (cid:1) 12 .

Original: [-A
-]
Revised: [-A
-]A6 The signal

Original: The signal-to-noise ratio satisfies ∥ µ ∥ σ = ω (cid:16)(cid:0) pn (cid:1) 14 (cid:17) .
Revised: -to-noise ratio satisfies [Equation]

Original: We compare the assumptions in Assumption 1 and Assumption 2.
Revised: We compare the assumptions in Assumption 1 and Assumption 2.

Original: Assumption [A4] is a relaxation of
Revised: Assumption [A4] is a relaxation of

Original: Assumption [A1], and Assumption [A5, A6] can be obtained by Assumption [A2, A3].
Revised: Assumption [A1], and Assumption [A5, A6] can be obtained by Assumption [A2, A3].

Original: Therefore, we conclude that Assumption 2 is weaker than Assumption 1.
Revised: Therefore, we conclude that Assumption 2 is weaker than Assumption 1.

Original: We next introduce the generalized version of the three arguments in Theorem 3.1, including Theorem 5.1, Theorem 5.2 and Theorem 5.3.
Revised: We next introduce the generalized version of the three arguments in Theorem 3.1, including Theorem 5.1, Theorem 5.2 and Theorem 5.3.

Original: Theorem 5.1 (Statement One) .
Revised: Theorem 5.1 (Statement One) .

Original: Under the noiseless setting, for a fixed δ 1 ≥ max { c 7 n , exp( − c 8 p ) } , under Assumption [A2, A4, A5], there exists constant c 2 > 0 such that the following statement holds with probability at least 1 − δ 1 ,
Revised: Under the noiseless setting, for a fixed δ 1 ≥ max { c 7 n , exp( − c 8 p ) } , under Assumption [A2, A4, A5], there exists constant c 2 > 0 such that the following statement holds with probability at least 1 − δ 1 ,

Original: [Equation]
Revised: [Equation]

Original: Theorem 5.1 implies that the interpolator under noiseless regimes converges to zero as the sample size n goes to infinity.
Revised: Theorem 5.1 implies that the interpolator under noiseless regimes converges to zero as the sample size n goes to infinity.

Original: The proof of Theorem 5.1 depends on bounding the projection of the classifier w on µ -direction, which relies on a sketch of the classification margin.
Revised: The proof of Theorem 5.1 depends on bounding the projection of the classifier w on µ -direction, which relies on a sketch of the classification margin.

Original: We defer the whole proof to the Appendix due to space limitations.
Revised: We defer the whole proof to the Appendix due to space limitations.

Original: Previous results on noiseless GMM ( e.g. , Cao et al.
Revised: Previous results on noiseless GMM ( e.g. , Cao et al.

Original: ; Wang et al.
Revised: ; Wang et al.

Original: (2021a)) rely on the heavy overparameterization p = ω ( n ) and assumption (cid:0) pn (cid:1) 14 ≤ ∥ µ ∥ σ ≤ pn .
Revised: (2021a)) rely on the heavy overparameterization p = ω ( n ) and assumption (cid:0) pn (cid:1) 14 ≤ ∥ µ ∥ σ ≤ pn .

Original: In contrast, our results only require ∥ µ ∥ σ ≥ (cid:0) pn (cid:1) 12 as well as ∥ µ ∥ σ ≥ c √ log n , and therefore, can be deployed in the mild overparameterization regimes.
Revised: In contrast, our results only require ∥ µ ∥ σ ≥ (cid:0) pn (cid:1) 12 as well as ∥ µ ∥ σ ≥ c √ log n , and therefore, can be deployed in the mild overparameterization regimes.

Original: Therefore, the existing results cannot directly imply Theorem 5.1.
Revised: Therefore, the existing results cannot directly imply Theorem 5.1.

Original: Theorem 5.2 (Statement Two) .
Revised: Theorem 5.2 (Statement Two) .

Original: Under the noisy regime with noisy level ρ and mild overparameterization ratio r = p/n , for a fixed δ 2 ≥ max { exp( − c 8 p ) , c 9 exp( − c 10 η 2 n ) } , under Assumption [A1,
Revised: Under the noisy regime with noisy level ρ and mild overparameterization ratio r = p/n , for a fixed δ 2 ≥ max { exp( − c 8 p ) , c 9 exp( − c 10 η 2 n ) } , under Assumption [A1,

Original: A3], there exists constant c 3 such that the following statement holds with probability at least 1 − δ 2 ,
Revised: A3], there exists constant c 3 such that the following statement holds with probability at least 1 − δ 2 ,

Original: [Equation]
Revised: [Equation]

Original: Therefore, w − ( ∞ ) has constant excess risk, given that r and ρ are both constant.
Revised: Therefore, w − ( ∞ ) has constant excess risk, given that r and ρ are both constant.

Original: Theorem 5.2 proves a constant lower bound for interpolators in noisy settings.
Revised: Theorem 5.2 proves a constant lower bound for interpolators in noisy settings.

Original: Compared to Theorem 5.1, Theorem 5.2 show that noisy and noiseless regimes perform differently under mild overparameterization.
Revised: Compared to Theorem 5.1, Theorem 5.2 show that noisy and noiseless regimes perform differently under mild overparameterization.

Original: The core of the proof lies in controlling the distance between the center of wrong labeled samples and the point µ , which further leads to an upper bound of | µ ⊤ w − ( ∞ ) | .
Revised: The core of the proof lies in controlling the distance between the center of wrong labeled samples and the point µ , which further leads to an upper bound of | µ ⊤ w − ( ∞ ) | .

Original: One can then derive the corresponding 0-1 loss for classifier w − ( ∞ ) .
Revised: One can then derive the corresponding 0-1 loss for classifier w − ( ∞ ) .

Original: We defer the whole proof to the
Revised: We defer the whole proof to the

Original: Appendix due to space limitations.
Revised: Appendix due to space limitations.

Original: Previous results ( e.g. , Chatterji et al.
Revised: Previous results ( e.g. , Chatterji et al.

Original: ; Wang et al.
Revised: ; Wang et al.

Original: (2021a))
Revised: (2021a))

Original: mainly focus on deriving the non-vacuous bound for noisy GMM, which also relies on the heavy overparameterization assumption p = ω ( n ) .
Revised: mainly focus on deriving the non-vacuous bound for noisy GMM, which also relies on the heavy overparameterization assumption p = ω ( n ) .

Original: Instead, our results show that the interpolator dramatically fails and suffers from a constant lower bound under mild overparameterization regimes.
Revised: Instead, our results show that the interpolator dramatically fails and suffers from a constant lower bound under mild overparameterization regimes.

Original: Therefore, heavy overparameterization performs differently from mild overparameterization cases.
Revised: Therefore, heavy overparameterization performs differently from mild overparameterization cases.

Original: We finally remark that although it is still an open problem when the phase change happens, we conjecture that realistic training procedures are more close to the mild overparameterization regime according to the experiment results.
Revised: We finally remark that although it is still an open problem when the phase change happens, we conjecture that realistic training procedures are more close to the mild overparameterization regime according to the experiment results.

Original: Theorem 5.3 (Statement Three) .
Revised: Theorem 5.3 (Statement Three) .

Original: Under the noisy regime, consider the learning rate η < 1 c 5 n max i ∥ x i ∥ 2 where ( x i , ˜ y i )
Revised: Under the noisy regime, consider the learning rate η < 1 c 5 n max i ∥ x i ∥ 2 where ( x i , ˜ y i )

Original: ∈ D denotes the data point.
Revised: ∈ D denotes the data point.

Original: For a fixed δ≥ max { c 11 n , c 12 exp( − c 13 η 2 n ) } , under Assumption [A2, A4, A6], the following statement holds with probability at least 1 − δ 3 ,
Revised: For a fixed δ≥ max { c 11 n , c 12 exp( − c 13 η 2 n ) } , under Assumption [A2, A4, A6], the following statement holds with probability at least 1 − δ 3 ,

Original: [Equation]
Revised: [Equation]

Original: Therefore, theabove bound converges to zero as sample size n goes to infinity underthe Assumption [A6].
Revised: Therefore, the bound converges to zero as sample size n goes to infinity under Assumption [A6].

Original: Theorem 5.3 derives that the bound again converges to zero by considering early-stopping.
Revised: Theorem 5.3 derives that the bound again converges to zero by considering early-stopping.

Original: Studying early-stopping classifiers is meaningful since people usually cannot ideally obtain the interpolation during training in practice.
Revised: Studying early-stopping classifiers is meaningful since people usually cannot ideally obtain the interpolation during training in practice.

Original: The derivation of Theorem 5.3 relies on the analysis of one-pass SGD.
Revised: The derivation of Theorem 5.3 relies on the analysis of one-pass SGD.

Original: Different from the previous approaches where we can directly assume ∥ w + ( ∞ ) ∥
Revised: Different from the previous approaches where we can directly assume ∥ w + ( ∞ ) ∥

Original: =
Revised: =

Original: ∥ w − ( ∞ ) ∥
Revised: ∥ w − ( ∞ ) ∥

Original: =without loss of generality, the classifier w − ( t ) is trained in this case and we need to first bound it.
Revised: =without loss of generality, the classifier w − ( t ) is trained in this case and we need to first bound it.

Original: We then define a surrogate classifier and show that (a) the surrogate classifier is close to the trained classifier for a sufficiently small learning rate, and (b) the surrogate classifier can return a satisfying projection on the direction µ .
Revised: We then define a surrogate classifier and show that (a) the surrogate classifier is close to the trained classifier for a sufficiently small learning rate, and (b) the surrogate classifier can return a satisfying projection on the direction µ .

Original: Therefore, we bound the term µ ⊤ w − ( t ) /
Revised: Therefore, we bound the term µ ⊤ w − ( t ) /

Original: ∥ w − ( t ) ∥ which leads to the results.
Revised: ∥ w − ( t ) ∥ which leads to the results.

Original: We defer the whole proof to the Appendix due to space limitations.
Revised: We defer the whole proof to the Appendix due to space limitations.

Original: One may wonder whether we can apply the results of stability-based bound (Bousquet et al., 2002;
Revised: One may wonder whether we can apply the results of stability-based bound (Bousquet et al., 2002;

Original: Hardt et al., 2016) into the analysis since the training process is convex.
Revised: Hardt et al., 2016) into the analysis since the training process is convex.

Original: However, the analysis might not be proper due to a bad Lipschitz constant during the training process.
Revised: However, the analysis might not be proper due to a bad Lipschitz constant during the training process.

Original: Therefore, the stability-based analysis may only return vacuous bound under such regimes.
Revised: Therefore, the stability-based analysis may only return vacuous bound under such regimes.

Original: Besides, the previous results on convex optimization with one-pass SGD ( e.g. , Sekhari et al.
Revised: Besides, the previous results on convex optimization with one-pass SGD ( e.g. , Sekhari et al.

Original: (2021)) cannot be directly applied to the analysis since most results on one-pass SGD are expectation bounds, while we provide a high probability bound in Theorem 5.3.
Revised: (2021)) cannot be directly applied to the analysis since most results on one-pass SGD are expectation bounds, while we provide a high probability bound in Theorem 5.3.

Original:  C ONCLUSIONS AND D ISCUSSIONS
Revised:  C ONCLUSIONS AND D ISCUSSIONS

Original: In this work, we aim to understand why benign overfitting happens in training ResNet on CIFAR10, but fails on ImageNet.
Revised: In this work, we aim to understand why benign overfitting happens in training ResNet on CIFAR10, but fails on ImageNet.

Original: We start by identifying a phase change in the theoretical analysis of benign overfitting.
Revised: We start by identifying a phase change in the theoretical analysis of benign overfitting.

Original: We found that when the model parameter is in the same order as the number of data points, benign overfitting would fail due to label noise.
Revised: We found that when the model parameter is in the same order as the number of data points, benign overfitting would fail due to label noise.

Original: We conjecture that the noise in labels leads to the different behaviors in CIFAR10 and ImageNet.
Revised: We conjecture that the noise in labels leads to the different behaviors in CIFAR10 and ImageNet.

Original: We verify the conjecture by injecting label noise into CIFAR10 and adopting self-training in ImageNet.
Revised: We verify the conjecture by injecting label noise into CIFAR10 and adopting self-training in ImageNet.

Original: The results support our hypothesis.
Revised: The results support our hypothesis.

Original: Our work also left many questions unanswered.
Revised: Our work also left many questions unanswered.

Original: First, our theoretical and empirical evidence shows that realistic deep learning models may not work in the interpolating scheme.
Revised: First, our theoretical and empirical evidence shows that realistic deep learning models may not work in the interpolating scheme.

Original: Still, although there is a larger number of parameters than data points, the model generalizes well.
Revised: Still, although there is a larger number of parameters than data points, the model generalizes well.

Original: Understanding the implicit bias in deep learning when the model underfits is still open.
Revised: Understanding the implicit bias in deep learning when the model underfits is still open.

Original: A closely related topic would be algorithmic stability (Bousquet et al., 2002; Hardt et al., 2016), however, the benefit of overparameterization within the stability framework still requires future studies.
Revised: A closely related topic would be algorithmic stability (Bousquet et al., 2002; Hardt et al., 2016), however, the benefit of overparameterization within the stability framework still requires future studies.

Original: Second, the GMM model provides a convenient way for analysis, but how the number of parameters in the linear setup relates to that in the neural network remains unclear.
Revised: Second, the GMM model provides a convenient way for analysis, but how the number of parameters in the linear setup relates to that in the neural network remains unclear.

Revised: Third, although the conclusions in this paper can be generalized to other regimes, e.g., GMM with Gradient Descent, the possible extension to neural networks as in Cao et al.

Revised: (2022) is still unclear and we will consider that in future works.

Original: Appendix A A DDITIONAL RELATED WORKS ON LABEL NOISE
Revised: Appendix A A DDITIONAL RELATED WORKS ON LABEL NOISE

Original: Label noise often exists in real world datasets, e.g. , in ImageNet (Yun et al., 2021; Shankar et al., 2020).
Revised: Label noise often exists in real world datasets, e.g. , in ImageNet (Yun et al., 2021; Shankar et al., 2020).

Original: However, its effect on generalization remains debatable.
Revised: However, its effect on generalization remains debatable.

Original: Recently, Damian et al.
Revised: Recently, Damian et al.

Original: (2021) claim that in regression settings, label noise may prefer flat global minimizers and thus helps generalization.
Revised: (2021) claim that in regression settings, label noise may prefer flat global minimizers and thus helps generalization.

Original: Another line of work claims that label noise hurts the effectiveness of empirical risk minimization in classification regimes, and proposes to improve the model performance by explicit regularization (Mignacco et al., 2020) or robust training (Brodley et al., 1996;
Revised: Another line of work claims that label noise hurts the effectiveness of empirical risk minimization in classification regimes, and proposes to improve the model performance by explicit regularization (Mignacco et al., 2020) or robust training (Brodley et al., 1996;

Original: Guan et al., 2011;
Revised: Guan et al., 2011;

Original: Huang et al., 2020).
Revised: Huang et al., 2020).

Original: Among them, Bagherinezhad et al.
Revised: Among them, Bagherinezhad et al.

Original: (2018) studies how to refine the labels in
Revised: (2018) studies how to refine the labels in

Original: ImageNet using label propagation to improve the model performance, demonstrating the importance of the label in ImageNet.
Revised: ImageNet using label propagation to improve the model performance, demonstrating the importance of the label in ImageNet.

Original: Besides, Nakkiran et al.
Revised: Besides, Nakkiran et al.

Original: (2021a) find that interpolation does harm to generalization in the presence of label noise.
Revised: (2021a) find that interpolation does harm to generalization in the presence of label noise.

Original: However, they do not relate their findings with the scale of overparameterization.
Revised: However, they do not relate their findings with the scale of overparameterization.

Original: This paper mainly falls in the latter branch, which theoretically analyzes how label noise acts in the mild overparameterization regimes.
Revised: This paper mainly falls in the latter branch, which theoretically analyzes how label noise acts in the mild overparameterization regimes.

Original: B D ETAILED P ROOFS B. P ROOF OF T HEOREM 5. 
Revised: B D ETAILED P ROOFS B. P ROOF OF T HEOREM 5.

Original: The first statement in Theorem 3.1 states that the interpolator has a non-vacuous bound in a noiseless setting, which is a direct corollary of the following Theorem 5.1.
Revised: The first statement in Theorem 3.1 states that the interpolator has a non-vacuous bound in a noiseless setting, which is a direct corollary of the following Theorem 5.1.

Original: The proof of Theorem 5.1 mainly depends on bounding the projection of the classifier w on µ , which relies on a sketch of the classification margin.
Revised: The proof of Theorem 5.1 mainly depends on bounding the projection of the classifier w on µ , which relies on a sketch of the classification margin.

Original: Theorem 5.1 (Statement One) .
Revised: Theorem 5.1 (Statement One) .

Original: Under the noiseless setting, for a fixed δ 1 ≥ max { c 7 n , exp( − c 8 p ) } , under Assumption [A2, A4, A5], there exists constant c 2 > 0 such that the following statement holds with probability at least 1 − δ 1 ,
Revised: Under the noiseless setting, for a fixed δ 1 ≥ max { c 7 n , exp( − c 8 p ) } , under Assumption [A2, A4, A5], there exists constant c 2 > 0 such that the following statement holds with probability at least 1 − δ 1 ,

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem 5.1.
Revised: Proof of Theorem 5.1.

Original: We denote the classifier w + ( ∞ ) by w during the proof for simplicity.
Revised: We denote the classifier w + ( ∞ ) by w during the proof for simplicity.

Original: Due to
Revised: Due to

Original: Proposition 3.1, the final classifier converges to its max-margin solution.
Revised: Proposition 3.1, the final classifier converges to its max-margin solution.

Original: Without loss of generality, let the final classifier w satisfy ∥ w ∥ = 1 .
Revised: Without loss of generality, let the final classifier w satisfy ∥ w ∥ = 1 .

Original: Therefore, the following equation for margin γ ( · ) holds since w is max-margin solution:
Revised: Therefore, the following equation for margin γ ( · ) holds since w is max-margin solution:

Original: [Equation]
Revised: [Equation]

Original: where γ ( w ) = min i
Revised: where γ ( w ) = min i

Original: y i x
Revised: y i x

Original: ⊤ i
Revised: ⊤ i

Original: w denotes the margin of classifier w for the dataset.
Revised: w denotes the margin of classifier w for the dataset.

Original: We next consider the margin for the classifier µ / ∥ µ ∥ .
Revised: We next consider the margin for the classifier µ / ∥ µ ∥ .

Original: Note that the margin can be rewritten as
Revised: Note that the margin can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: We note that
Revised: We note that

Original: y i
Revised: y i

Original: ϵ ⊤ i µ / ∥ µ ∥ is σ -subGaussian due to the definition of subGaussian random vector.
Revised: ϵ ⊤ i µ / ∥ µ ∥ is σ -subGaussian due to the definition of subGaussian random vector.

Original: Therefore, due to Claim C.1, we have
Revised: Therefore, due to Claim C.1, we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is due to Assumption [A2].
Revised: where the last inequality is due to Assumption [A2].

Original: We next bound the term µ ⊤ w via the above margin.
Revised: We next bound the term µ ⊤ w via the above margin.

Original: From one hand, we notice that by the definition of the margin function,
Revised: From one hand, we notice that by the definition of the margin function,

Original: [Equation]
Revised: [Equation]

Original: From the other hand, we rewrite the margin as
Revised: From the other hand, we rewrite the margin as

Original: [Equation]
Revised: [Equation]

Original: The right hand side can be bounded as
Revised: The right hand side can be bounded as

Original: [Equation]
Revised: [Equation]

Original: where the final equation is due to Claim C.2.
Revised: where the final equation is due to Claim C.2.

Original: Therefore, combining the above Equation 1, Equationand Equation 3, we bound the projection of w + on µ as:
Revised: Therefore, combining the above Equation 1, Equationand Equation 3, we bound the projection of w + on µ as:

Original: [Equation]
Revised: [Equation]

Original: where the last equation is due to Assumption [A5].
Revised: where the last equation is due to Assumption [A5].

Original: We rewrite Equation (4) as µ ⊤ w ≥ c 5 ∥ µ ∥ , then we can bound the 0-1 loss as follows for a given constant c 6 :
Revised: We rewrite Equation (4) as µ ⊤ w ≥ c 5 ∥ µ ∥ , then we can bound the 0-1 loss as follows for a given constant c 6 :

Original: [Equation]
Revised: [Equation]

Original: Due to Assumption [A2], ∥ µ ∥ /σ > c √  n , and therefore, by setting c 2 = c 6 c 2 , we have
Revised: Due to Assumption [A2], ∥ µ ∥ /σ > c √ log n , and therefore, by setting c 2 = c 6 c 2 , we have

Original: [Equation]
Revised: [Equation]

Original: The proof is done.
Revised: The proof is done.

Original: B. P ROOF OF T HEOREM 5. 
Revised: B. P ROOF OF T HEOREM 5.

Original: Theorem 5.1 shows that the interpolator can be non-vacuous under noiseless regimes with mild overparameterization.
Revised: Theorem 5.1 shows that the interpolator can be non-vacuous under noiseless regimes with mild overparameterization.

Original: However, things can be much different in noisy regimes.
Revised: However, things can be much different in noisy regimes.

Original: Statement Two proves a vacuous lower bound for interpolators in noisy settings, which can be derived by the following
Revised: Statement Two proves a vacuous lower bound for interpolators in noisy settings, which can be derived by the following

Original: Theorem 5.2.
Revised: Theorem 5.2.

Original: The core of the proof lies in controlling the distance between the center of wrong labeled samples and the point µ , which further leads to an upper bound of | µ ⊤ w − ( ∞ ) | .
Revised: The core of the proof lies in controlling the distance between the center of wrong labeled samples and the point µ , which further leads to an upper bound of | µ ⊤ w − ( ∞ ) | .

Original: One can then derive the corresponding 0-1 loss for classifier w − ( ∞ ) .
Revised: One can then derive the corresponding 0-1 loss for classifier w − ( ∞ ) .

Original: Theorem 5.2 (Statement Two) .
Revised: Theorem 5.2 (Statement Two) .

Original: Under the noisy regime with noisy level ρ and mild overparameterization ratio r = p/n , for a fixed δ 2 ≥ max { exp( − c 8 p ) , c 9 exp( − c 10 η 2 n ) } , under Assumption [A1,
Revised: Under the noisy regime with noisy level ρ and mild overparameterization ratio r = p/n , for a fixed δ 2 ≥ max { exp( − c 8 p ) , c 9 exp( − c 10 η 2 n ) } , under Assumption [A1,

Original: A3], there exists constant c 3 such that the following statement holds with probability at least 1 − δ 2 ,
Revised: A3], there exists constant c 3 such that the following statement holds with probability at least 1 − δ 2 ,

Original: [Equation]
Revised: [Equation]

Original: Therefore, w − ( ∞ ) has constant excess risk, given that r and ρ are both constant.
Revised: Therefore, w − ( ∞ ) has constant excess risk, given that r and ρ are both constant.

Original: Proof.
Revised: Proof.

Original: We denote w − ( ∞ ) as w for simplicity, and assume that ∥ w ∥ = 1 without loss of generality.
Revised: We denote w − ( ∞ ) as w for simplicity, and assume that ∥ w ∥ = 1 without loss of generality.

Original: Let y denote the original label and ˜ y denote its corrupted label.
Revised: Let y denote the original label and ˜ y denote its corrupted label.

Original: Without loss of generality, we consider those samples with y i = 

Revised: Without loss of generality, we consider those samples with y i =

Original: while ˜ y
Revised: while ˜ y

Original: i = − 1 , which are indexed by K
Revised: i = − 1 , which are indexed by K

Original: =
Revised: =

Original: { i : y i = 1 , ˜ y
Revised: { i : y i = 1 , ˜ y

Original: i = − 1 } . Consider the center point of K , which is
Revised: i = − 1 } . Consider the center point of K , which is

Original: [Equation]
Revised: [Equation]

Original: Due to the interpolation in Proposition 3.1 and ˜ y
Revised: Due to the interpolation in Proposition 3.1 and ˜ y

Original: = − 1 , we derive that
Revised: = − 1 , we derive that

Original: [Equation]
Revised: [Equation]

Original: Case 1: µ ⊤ w < 0 .
Revised: Case 1: µ ⊤ w < 0 .

Original: In this case, L 01 ( w ) naturally has a lower bound of 1 / 2 since it even fails in the center point µ .
Revised: In this case, L 01 ( w ) naturally has a lower bound of 1 / 2 since it even fails in the center point µ .

Original: Case 2: µ ⊤ w > 0 .
Revised: Case 2: µ ⊤ w > 0 .

Original: In this case, the classifier w satisfies µ ⊤ w > 0 and ¯ x ⊤K w < 0 .
Revised: In this case, the classifier w satisfies µ ⊤ w > 0 and ¯ x ⊤K w < 0 .

Original: Therefore, the distance between µ and ¯ x ⊤K must be less than the distance from µ to its projection on the separating hyperplane that perpendicular to w through the origin.
Revised: Therefore, the distance between µ and ¯ x ⊤K must be less than the distance from µ to its projection on the separating hyperplane that perpendicular to w through the origin.

Original: Formally,
Revised: Formally,

Original: [Equation]
Revised: [Equation]

Original: Note that ϵ i is independent and σ -subGaussian, and therefore applying Claim C.2, we have that
Revised: Note that ϵ i is independent and σ -subGaussian, and therefore applying Claim C.2, we have that

Original: [Equation]
Revised: [Equation]

Original: i ∈K ϵ
Revised: i ∈K ϵ

Original: i (cid:13)(cid:13) ≲ σ (cid:113) p |K| .
Revised: i (cid:13)(cid:13) ≲ σ (cid:113) p |K| .

Original: Besides, we derive by Claim C.3 that |K| ≳
Revised: Besides, we derive by Claim C.3 that |K| ≳

Original: ρn .
Revised: ρn .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: We next consider the corresponding test error of w , where we consider the test error on noiseless regime instead of noisy regime.
Revised: We next consider the corresponding test error of w , where we consider the test error on noiseless regime instead of noisy regime.

Original: Note that the two arguments are equivalent, we refer to Claim C.4 for more details.
Revised: Note that the two arguments are equivalent, we refer to Claim C.4 for more details.

Original: We rewrite Equation 6 as | µ ⊤ w | ≤ cσ (cid:113) d |K| ≲ σ (cid:113) pρn , where we abuse the notation c > 0 as a fixed constant.
Revised: We rewrite Equation 6 as | µ ⊤ w | ≤ cσ (cid:113) d |K| ≲ σ (cid:113) pρn , where we abuse the notation c > 0 as a fixed constant.

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: where ϵ is sampled from Gaussian distribution, and Φ denotes the CDF of standard Gaussian Random
Revised: where ϵ is sampled from Gaussian distribution, and Φ denotes the CDF of standard Gaussian Random

Original: Variable.
Revised: Variable.

Original: Case 1.
Revised: Case 1.

Original: If c (cid:113) pρn ≤ 2 , then Φ( − c (cid:113) pρn ) ≥ Φ( − 2) .
Revised: If c (cid:113) pρn ≤ 2 , then Φ( − c (cid:113) pρn ) ≥ Φ( − 2) .

Original: Case 2.
Revised: Case 2.

Original: If c (cid:113) pρn > 2 , note that Φ( − t ) ≥ ( 1 t − 1 t 3 ) exp( − t 2 / 2) ≥ 1 /t 2 exp( − t 2 / 2) if t > 2 .
Revised: If c (cid:113) pρn > 2 , note that Φ( − t ) ≥ ( 1 t − 1 t 3 ) exp( − t 2 / 2) ≥ 1 /t 2 exp( − t 2 / 2) if t > 2 .

Original: Therefore, Φ( − c (cid:113) pρn ) ≥
Revised: Therefore, Φ( − c (cid:113) pρn ) ≥

Original: [Equation]
Revised: [Equation]

Original: ≥
Revised: ≥

Original: [Equation]
Revised: [Equation]

Original: − c 2 pρn ) .
Revised: − c 2 pρn ) .

Original: Taking the above two cases together and denoting r = p/n , we have
Revised: Taking the above two cases together and denoting r = p/n , we have

Original: [Equation]
Revised: [Equation]

Original: which is a constant lower bound under Assumption [A3].
Revised: which is a constant lower bound under Assumption [A3].

Original: The proof is done.
Revised: The proof is done.

Original: B. P ROOF OF T HEOREM 5. 
Revised: B. P ROOF OF T HEOREM 5.

Original: We apologize for the following typos made in Theorem 5.3 in the main text.
Revised: We apologize for the following typos made in Theorem 5.3 in the main text.

Original: For consistency, the dimension d should be written as p and the learning rate λ should be written as η .
Revised: For consistency, the dimension d should be written as p and the learning rate λ should be written as η .

Original: Furthermore,
Revised: Furthermore,

Original: Theorem 5.3 needs an initialization from zero during the training, which is widely used in linear models (Bartlett et al., 2020).
Revised: Theorem 5.3 needs an initialization from zero during the training, which is widely used in linear models (Bartlett et al., 2020).

Original: Statement Two shows that the interpolator fails in the noisy regime with mild overparameterization.
Revised: Statement Two shows that the interpolator fails in the noisy regime with mild overparameterization.

Original: How can we derive a non-vacuous bound under such regimes?
Revised: How can we derive a non-vacuous bound under such regimes?

Original: The key is early-stopping.
Revised: The key is early-stopping.

Original: To show that, Statement Three provides a non-vacuous bound for early-stopping classifiers in noisy regimes, which is induced by the following Theorem 5.3.
Revised: To show that, Statement Three provides a non-vacuous bound for early-stopping classifiers in noisy regimes, which is induced by the following Theorem 5.3.

Original: Theorem 5.3 (Statement Three) .
Revised: Theorem 5.3 (Statement Three) .

Original: Under the noisy regime, consider the learning rate η <
Revised: Under the noisy regime, consider the learning rate η <

Original: [Equation]
Revised: [Equation]

Revised: max

Revised: [Equation]

Original: i ∥ 2 where ( x i , ˜ y i )
Revised: i ∥ 2 where ( x i , ˜ y i )

Original: ∈ D denotes the data point.
Revised: ∈ D denotes the data point.

Original: For a fixed δ≥ max { c 11 n , c 12 exp( − c 13 η 2 n ) } , under Assumption [A2, A4, A6], the following statement holds with probability at least 1 − δ 3 ,
Revised: For a fixed δ≥ max { c 11 n , c 12 exp( − c 13 η 2 n ) } , under Assumption [A2, A4, A6], the following statement holds with probability at least 1 − δ 3 ,

Original: [Equation]
Revised: [Equation]

Original: Therefore, theabove bound converges to zero as sample size n goes to infinity underthe Assumption [A6].
Revised: Therefore, the bound converges to zero as sample size n goes to infinity under Assumption [A6].

Original: To show the relationship between Theorem 5.3 and Theorem 3.1, one can directly use Assumption [A2,
Revised: To show the relationship between Theorem 5.3 and Theorem 3.1, one can directly use Assumption [A2,

Original: A3] in Theorem 5.3 to reach generalization bound in Theorem 3.1 (Statement Three).
Revised: A3] in Theorem 5.3 to reach generalization bound in Theorem 3.1 (Statement Three).

Original: The derivation of Theorem 5.3 relies on the analysis on one-pass SGD, where we show that one-pass SGD is sufficient to reach non-vacuous bound.
Revised: The derivation of Theorem 5.3 relies on the analysis on one-pass SGD, where we show that one-pass SGD is sufficient to reach non-vacuous bound.

Original: The proof of Theorem 5.3 again, relies on bounding the term c but in a different way.
Revised: The proof of Theorem 5.3 again, relies on bounding the term c but in a different way.

Original: Different from the previous approaches where we can directly assume ∥w
Revised: Different from the previous approaches where we can directly assume ∥

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: ∥ w
Revised: 1 , the classifier w

Original: [Equation]
Revised: [Equation]

Original: (∞ ) ∥ = 1 , the classifier w − ( t ) is trained in this case and we need to first bound it.
Revised: ( t ) is trained in this case and we need to first bound it.

Original: We then define a surrogate classifier and show that (a) the surrogate classifier is close to the trained classifier for a sufficiently small learning rate, and (b) the surrogate classifier can return satisfying projection on the direction µ .
Revised: We then define a surrogate classifier and show that (a) the surrogate classifier is close to the trained classifier for a sufficiently small learning rate, and (b) the surrogate classifier can return satisfying projection on the direction µ .

Original: Therefore, we bound the term µ ⊤ w − ( t ) /
Revised: Therefore, we bound the term µ ⊤ w − ( t ) /

Original: ∥ w − ( t ) ∥ which leads to the results.
Revised: ∥ w − ( t ) ∥ which leads to the results.

Original: Proof.
Revised: Proof.

Original: We abuse the notation w n to represent w − ( t ) which is returned by one-pass SGD.
Revised: We abuse the notation w n to represent w − ( t ) which is returned by one-pass SGD.

Original: We first lower bound the term µ ⊤ w n ∥ w n ∥ , where µ is the optimal classification direction.
Revised: We first lower bound the term µ ⊤ w n ∥ w n ∥ , where µ is the optimal classification direction.

Original: To achieve the goal, we bound the term µ ⊤ w n and ∥ w n ∥ individually.
Revised: To achieve the goal, we bound the term µ ⊤ w n and ∥ w n ∥ individually.

Original: Before diving into the proof, we first introduce a surrogate classifier ˜ w n = 12 η (cid:80) t x t ˜ y t .
Revised: Before diving into the proof, we first introduce a surrogate classifier ˜ w n = 12 η (cid:80) t x t ˜ y t .

Original: From the definition, we have that for update step size η ,
Revised: From the definition, we have that for update step size η ,

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have that
Revised: Therefore, we have that

Original: [Equation]
Revised: [Equation]

Original: Bounding the term µ ⊤ w .
Revised: Bounding the term µ ⊤ w .

Original: We fist bound the different between µ ⊤ w t and µ ⊤ ˜ w t .
Revised: We fist bound the different between µ ⊤ w t and µ ⊤ ˜ w t .

Original: We note that
Revised: We note that

Original: To bound the above different, the fist step is to bound the term max t ∈ [ n ] µ ⊤ x t ˜ y t .
Revised: To bound the above different, the fist step is to bound the term max t ∈ [ n ] µ ⊤ x t ˜ y t .

Original: Since µ ⊤ ϵ t / ∥ µ ∥ is σ -subGuassian, we have that with probability 1 − δ 1 with δ 
Revised: Since µ ⊤ ϵ t / ∥ µ ∥ is σ -subGuassian, we have that with probability 1 − δ 1 with δ 1 ≲ 1 /

Original: [Equation]
Revised: [Equation]

Original: We then bound the term exp( − ˜ y
Revised: We then bound the term exp( − ˜ y

Original: t x ⊤ t w t ) − 1 2(1+exp( − ˜ y
Revised: t x ⊤ t w t ) − 1 2(1+exp( − ˜ y

Original: t x ⊤ t w t )) .
Revised: t x ⊤ t w t )) .

Original: Note that | exp( u ) − 1 | ≤ 2 | u | when | u | < 12 .
Revised: Note that | exp( u ) − 1 | ≤ 2 | u | when | u | < 12 .

Original: By the iteration in Equation 8, we have that
Revised: By the iteration in Equation 8, we have that

Original: [Equation]
Revised: [Equation]

Original: where the first equation is due to the iteration, and the last equation is due to η ≤ 1 2 n max i ∥
Revised: where the first equation is due to the iteration, and the last equation is due to η ≤ 1 2 n max i ∥

Original: x i ∥ 2 (by setting c 5 > 2 ).
Revised: x i ∥ 2 (by setting c 5 > 2 ).

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Combining Equation 10 and Equation 11, we have that with probability at least 1 − δ 1 ,
Revised: Combining Equation 10 and Equation 11, we have that with probability at least 1 − δ 1 ,

Original: [Equation]
Revised: [Equation]

Original: On the other hand, we show the bound for µ ⊤ ˜ w n .
Revised: On the other hand, we show the bound for µ ⊤ ˜ w n .

Original: Note that ˜ w t
Revised: Note that ˜ w t

Original: +

Revised: +

Original: =
Revised: =

Original: η/ 2 (cid:80) t x t ˜ y t , therefore,
Revised: η/ 2 (cid:80) t x t ˜ y t , therefore,

Original: [Equation]
Revised: [Equation]

Original: where we denote I i ( ρ ) ∈ {− 1 , 1 } as a random variable which takes value − 1 with probability ρ and takes value +1 with probability 1 − ρ .
Revised: where we denote I i ( ρ ) ∈ {− 1 , 1 } as a random variable which takes value − 1 with probability ρ and takes value +1 with probability 1 − ρ .

Original: Due to Claim C.3, we have 1 n (cid:80) t I i ( ρ )
Revised: Due to Claim C.3, we have 1 n (cid:80) t I i ( ρ )

Original: ≳
Revised: ≳

Original: ρ , where we note that I ( ρ ) ∈ {− 1 , 1 } .
Revised: ρ , where we note that I ( ρ ) ∈ {− 1 , 1 } .

Original: Besides, since µ ⊤ ϵ t ˜ y t is ∥ µ ∥ σ -subGaussian, we have that with probability 1 − 1 n ,
Revised: Besides, since µ ⊤ ϵ t ˜ y t is ∥ µ ∥ σ -subGaussian, we have that with probability 1 − 1 n ,

Original: [Equation]
Revised: [Equation]

Original: where the term log( n ) comes from the probability 1 /n .
Revised: where the term log( n ) comes from the probability 1 /n .

Original: In summary, we have that
Revised: In summary, we have that

Original: [Equation]
Revised: [Equation]

Original: where the last equation follows Assumption [A2].
Revised: where the last equation follows Assumption [A2].

Original: Combining Equation (12) and Equation (13), we have
Revised: Combining Equation (12) and Equation (13), we have

Original: [Equation]
Revised: [Equation]

Original: We additionally note that Equation 14 holds by choosing proper constant in Equation (11) (and in the choice of η ).
Revised: We additionally note that Equation 14 holds by choosing proper constant in Equation (11) (and in the choice of η ).

Original: Bounding the norm ∥ w n ∥ .
Revised: Bounding the norm ∥ w n ∥ .

Original: We next bound the norm ∥ w n ∥ .
Revised: We next bound the norm ∥ w n ∥ .

Original: Before that, we first bound the norm ∥
Revised: Before that, we first bound the norm ∥

Original: ˜ w t ∥ .
Revised: ˜ w t ∥ .

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: where the last equation is due to Claim C.2 by choosing probability δ ≳ exp( − p )
Revised: where the last equation is due to Claim C.2 by choosing probability δ ≳ exp( − p )

Original: /n .
Revised: /n .

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: Note that according to the iteration in Equation 8, we have that
Revised: Note that according to the iteration in Equation 8, we have that

Original: [Equation]
Revised: [Equation]

Original: where the last equation is due to η ≤
Revised: where the last equation is due to η ≤

Original: [Equation]
Revised: [Equation]

Original: Therefore, we bound the norm w t as
Revised: Therefore, we bound the norm w t as

Original: [Equation]
Revised: [Equation]

Original: Combining Equation 14 and Equation
Revised: Combining Equation 14 and Equation

Original: [Equation]
Revised: [Equation]

Original: we derive that with probability at least 1 − c/n ,
Revised: we derive that with probability at least 1 − c/n ,

Original: [Equation]
Revised: [Equation]

Original: We next consider the probability on the test point, note that given the dataset ( x i , y i ) and taking probability on the testing point ( x , y ) , we have
Revised: We next consider the probability on the test point, note that given the dataset ( x i , y i ) and taking probability on the testing point ( x , y ) , we have

Original: [Equation]
Revised: [Equation]

Original: where y x ⊤ w = µ ⊤ w
Revised: where y x ⊤ w = µ ⊤ w

Original: + y ϵ ⊤ w and y ϵ ⊤ w / ∥ w ∥ is σ -subGaussian.
Revised: + y ϵ ⊤ w and y ϵ ⊤ w / ∥ w ∥ is σ -subGaussian.

Original: Plugging Equation 17 into the above equation, we have that with high probability,
Revised: Plugging Equation 17 into the above equation, we have that with high probability,

Original: [Equation]
Revised: [Equation]

Original: If ∥ µ ∥ /σ > (cid:112) p/n ,
Revised: If ∥ µ ∥ /σ > (cid:112) p/n ,

Original: [Equation]
Revised: [Equation]

Original: µ ∥ /σ .
Revised: µ ∥ /σ .

Original: If ∥ µ ∥ /σ < (cid:112) p/n ,
Revised: If ∥ µ ∥ /σ < (cid:112) p/n ,

Original: [Equation]
Revised: [Equation]

Original: pn ≳
Revised: pn ≳

Original: [Equation]
Revised: [Equation]

Original: ∥ 2 σ 2 (cid:112) n/p , which is large when ∥ µ ∥ /σ = ω
Revised: ∥ 2 σ 2 (cid:112) n/p , which is large when ∥ µ ∥ /σ = ω

Original: (( p/n ) 1 / 4 ) .
Revised: (( p/n ) 1 / 4 ) .

Original: Therefore, the above bound is non-vacuous if ∥ µ ∥ /σ = ω
Revised: Therefore, the above bound is non-vacuous if ∥ µ ∥ /σ = ω

Original: (( p/n ) 1 / 4 ) .
Revised: (( p/n ) 1 / 4 ) .

Original: Note that for given a constant r = p/n , under Assumption [A2], we have
Revised: Note that for given a constant r = p/n , under Assumption [A2], we have

Original: [Equation]
Revised: [Equation]

Original: The proof is done.
Revised: The proof is done.

Original: C T ECHNICAL L EMMAS
Revised: C T ECHNICAL L EMMAS

Original: We next provide some technical claims.
Revised: We next provide some technical claims.

Original: The first claims bound the maximum of a sequence of subGuassian random variables:
Revised: The first claims bound the maximum of a sequence of subGuassian random variables:

Original: Claim C.1 (Maximum of a sequence of subGuassian random variables.) .
Revised: Claim C.1 (Maximum of a sequence of subGuassian random variables.) .

Original: For a sequence of σsubGaussian random variables X 1 , . . .
Revised: For a sequence of σsubGaussian random variables X 1 , . . .

Original: X n ∈ R .
Revised: X n ∈ R .

Original: We have that with probability at least 1 − δ ,
Revised: We have that with probability at least 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: Specifically, under the condition that δ ≳ 1 /n , we have
Revised: Specifically, under the condition that δ ≳ 1 /n , we have

Original: [Equation]
Revised: [Equation]

Original: Proof of Claim C.1.
Revised: Proof of Claim C.1.

Original: By taking union bound, we have
Revised: By taking union bound, we have

Original: [Equation]
Revised: [Equation]

Original: By setting u = √ 2 σ ( √ log n + (cid:112) log(1 /δ )) , we have
Revised: By setting u = √ 2 σ ( √ log n + (cid:112) log(1 /δ )) , we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, with probability at least 1 − δ , we have
Revised: Therefore, with probability at least 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Original: The next claim bounds the norm of sum of epsilon.
Revised: The next claim bounds the norm of sum of epsilon.

Original: Claim C.2.
Revised: Claim C.2.

Original: Under Assumption [A4] which assumes that ϵ i are subGaussian, we have that with probability at least 1 − δ , (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)n (cid:88) i ∈ [ n ] ϵ i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≤ 4 σ
Revised: Under Assumption [A4] which assumes that ϵ i are subGaussian, we have that with probability at least 1 − δ , (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)n (cid:88) i ∈ [ n ] ϵ i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≤ 4 σ

Original: [Equation]
Revised: [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: + 2 σ
Revised: + 2 σ

Original: [Equation]
Revised: [Equation]

Original: log(1 /δ )
Revised: log(1 /δ )

Original: n .
Revised: n .

Original: Specifically, under the condition that δ ≳ exp( − 4 p ) , it holds that (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)n (cid:88) i ∈ [ n ] ϵ i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
Revised: Specifically, under the condition that δ ≳ exp( − 4 p ) , it holds that (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)n (cid:88) i ∈ [ n ] ϵ i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Note that since ϵ i is σ 2 -subGaussian, its summation (cid:80) i ∈ [ n ] ϵ i is nσ 2 -subGaussian.
Revised: Note that since ϵ i is σ 2 -subGaussian, its summation (cid:80) i ∈ [ n ] ϵ i is nσ 2 -subGaussian.

Original: Therefore, we have that
Revised: Therefore, we have that

Original: [Equation]
Revised: [Equation]

Original: The next claim shows that the number of noisy data is approximately in the same order with ρn .
Revised: The next claim shows that the number of noisy data is approximately in the same order with ρn .

Original: Claim C.3 (Number of noisy data) .
Revised: Claim C.3 (Number of noisy data) .

Original: Denote |K| as the number of samples with flipped labels.
Revised: Denote |K| as the number of samples with flipped labels.

Original: With probability at least 1 − δ , there exists a constant c such that
Revised: With probability at least 1 − δ , there exists a constant c such that

Original: [Equation]
Revised: [Equation]

Original: Furthermore, when δ ≳ 2 exp( − 18 ρ 2 n ) , we have that with probability at least 1 − δ , |K| ≳ 1 2 ρn and |K| ≲ 3 2 ρn.
Revised: Furthermore, when δ ≳ 2 exp( − 18 ρ 2 n ) , we have that with probability at least 1 − δ , |K| ≳ 1 2 ρn and |K| ≲ 3 2 ρn.

Original: Proof.
Revised: Proof.

Original: Note that K = (cid:80) i u i , where u i are drawn from independent Bernoulli distribution with parameter ρ .
Revised: Note that K = (cid:80) i u i , where u i are drawn from independent Bernoulli distribution with parameter ρ .

Original: Therefore, by Hoeffding’s inequality, there exists constant c such that with probability 1 − 2 exp (cid:16) − ct 2 n (cid:17) ,
Revised: Therefore, by Hoeffding’s inequality, there exists constant c such that with probability 1 − 2 exp (cid:16) − ct 2 n (cid:17) ,

Original: [Equation]
Revised: [Equation]

Original: By setting t = (cid:113) nc log( 2 δ ) , we have that with probability at least 1 − δ ,
Revised: By setting t = (cid:113) nc log( 2 δ ) , we have that with probability at least 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: We then rewrite the constant c to reach the above conclusion.
Revised: We then rewrite the constant c to reach the above conclusion.

Original: The next Claim C.4 shows that analyzing L 01 loss on noiseless data is sufficient under noisy regimes.
Revised: The next Claim C.4 shows that analyzing L 01 loss on noiseless data is sufficient under noisy regimes.

Original: Claim C.4.
Revised: Claim C.4.

Original: Under the noisy regime, the 0-1 population loss on noiseless data L 01 ( w ) =
Revised: Under the noisy regime, the 0-1 population loss on noiseless data L 01 ( w ) =

Original: P ( y x ⊤ w < 0) has the same order with the 0-1 population loss (excess risk) on noisy data ˜
Revised: P ( y x ⊤ w < 0) has the same order with the 0-1 population loss (excess risk) on noisy data ˜

Original: L 01 ( w ) − η =
Revised: L 01 ( w ) − η =

Original: P (˜ y x ⊤ w < 0) − η .
Revised: P (˜ y x ⊤ w < 0) − η .

Original: Therefore, we analyze L 01 ( w ) in this paper as a surrogate of ˜ L 01 ( w ) .
Revised: Therefore, we analyze L 01 ( w ) in this paper as a surrogate of ˜ L 01 ( w ) .

Original: Proof.
Revised: Proof.

Original: Note that we here consider the L 01 on the noiseless data, and one can get ˜ L 01 on the noisy data by just adding ρ to L 01 .
Revised: Note that we here consider the L 01 on the noiseless data, and one can get ˜ L 01 on the noisy data by just adding ρ to L 01 .

Original: That is to say,
Revised: That is to say,

Original: [Equation]
Revised: [Equation]

Original: And therefore ˜ L 01 − η has the same order with L 01 .
Revised: And therefore ˜ L 01 − η has the same order with L 01 .

Original: D E XPERIMENTS D. I MPLEMENTED T ASKS
Revised: D E XPERIMENTS D. I MPLEMENTED T ASKS

Original: We conduct experiments on three public available datasets along with synthetic GMM data.
Revised: We conduct experiments on three public available datasets along with synthetic GMM data.

Original: First, we briefly introduce each dataset.
Revised: First, we briefly introduce each dataset.

Original: ImageNet is an image classification dataset containing 1.2 million pictures belonging todifferent classes.
Revised: ImageNet is an image classification dataset containing 1.2 million pictures belonging todifferent classes.

Original: CIFAR10 is an image classification dataset containing 60000 pictures belonging to 10 different classes.
Revised: CIFAR10 is an image classification dataset containing 60000 pictures belonging to 10 different classes.

Original: Penn Tree Bank(PTB) is a dataset containing natural sentences withdifferent tokens.
Revised: Penn Tree Bank(PTB) is a dataset containing natural sentences withdifferent tokens.

Original: We use this dataset for language modelling task.
Revised: We use this dataset for language modelling task.

Original: Synthetic GMM data is generated following the setup in Section 3.1.
Revised: Synthetic GMM data is generated following the setup in Section 3.1.

Original: We generate data for n ∈ { 2 4 , ..., 2 9 } , p ∈ { 2 4 , ..., 2 9 } , ρ ∈ { 0 , 0 . 4 } , | µ | = 40 , σ = 1 , n ≤ p .
Revised: We generate data for n ∈ { 2 4 , ..., 2 9 } , p ∈ { 2 4 , ..., 2 9 } , ρ ∈ { 0 , 0 . 4 } , | µ | = 40 , σ = 1 , n ≤ p .

Original: We then describe experiment details for each dataset.
Revised: We then describe experiment details for each dataset.

Original: For ImageNet , we train a ResNet50 from scratch.
Revised: For ImageNet , we train a ResNet50 from scratch.

Original: We use standard cross entropy loss.
Revised: We use standard cross entropy loss.

Original: We first train the model for 90 epochs using SGD as optimizer, with initial learning rate 1 e − 1 , momentum 0 .and
Revised: We first train the model for 90 epochs using SGD as optimizer, with initial learning rate 1 e − 1 , momentum 0 .and

Original: weight decay 1 e − 4 .
Revised: weight decay 1 e − 4 .

Original: The learning rate will decay by a factor of 10 for every 30 iterations.
Revised: The learning rate will decay by a factor of 10 for every 30 iterations.

Original: Then we train the model for another 410 epochs, with initial learning rate 1 e − 3 ,momentum 0 .
Revised: Then we train the model for another 410 epochs, with initial learning rate 1 e − 3 ,momentum 0 .

Original: 9 and weight decay 1 e − 4 and learning rate will decay by a factor of 1 . 25 for every 50 iterations.
Revised: 9 and weight decay 1 e − 4 and learning rate will decay by a factor of 1 . 25 for every 50 iterations.

Original: For CIFAR10 , we randomly flip the label with probability { 0 , 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 } and for each train a ResNet18 from scratch.
Revised: For CIFAR10 , we randomly flip the label with probability { 0 , 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 } and for each train a ResNet18 from scratch.

Original: We use standard cross entropy loss.
Revised: We use standard cross entropy loss.

Original: We train each model forepochs using SGD as optimizer, with learning rate 1 e − 1 , momentum 0 .
Revised: We train each model forepochs using SGD as optimizer, with learning rate 1 e − 1 , momentum 0 .

Original: 9 and weight decay 5 e − 4 .
Revised: 9 and weight decay 5 e − 4 .

Original: We use a cosine learning rate decay scheduler.
Revised: We use a cosine learning rate decay scheduler.

Original: For ablation study, we fix the flip probability to be 0 . 

Revised: For ablation study, we fix the flip probability to be 0 .

Original: and train WideResNet16- W on the noisy dataset with W ∈ 1 , 1 . 25 , 1 . 5 , 1 . 75 , 2 , 3 , 4 .
Revised: and train WideResNet16- W on the noisy dataset with W ∈ 1 , 1 . 25 , 1 . 5 , 1 . 75 , 2 , 3 , 4 .

Original: We repeat each experiments for four times.
Revised: We repeat each experiments for four times.

Original: We train each model for 200 epochs using SGD as optimizer, with learning rate 1 e − 1 , momentum 0 .
Revised: We train each model for 200 epochs using SGD as optimizer, with learning rate 1 e − 1 , momentum 0 .

Original: 9 and weight decay 5 e − 4 .
Revised: 9 and weight decay 5 e − 4 .

Original: We use a cosine learning rate decay scheduler
Revised: We use a cosine learning rate decay scheduler

Original: [Figure]
Revised: [Figure]

Original: For Penn Tree Bank , we train a standard transformer from scratch.
Revised: For Penn Tree Bank , we train a standard transformer from scratch.

Original: We train one model forepochs using ADAM as optimizer, with learning rate 5 e − 4 , beta (0 . 9 , 0 . 98) and weight decay 1 e − 2 .
Revised: We train one model forepochs using ADAM as optimizer, with learning rate 5 e − 4 , beta (0 . 9 , 0 . 98) and weight decay 1 e − 2 .

Original: We use an inverse square learning rate scheduler.
Revised: We use an inverse square learning rate scheduler.

Original: We train another model for 140 epochs using SGD as optimizer with learning rate 5.0.
Revised: We train another model for 140 epochs using SGD as optimizer with learning rate 5.0.

Original: We use a step learning rate schedule with step size 1 and gamma 0 .
Revised: We use a step learning rate schedule with step size 1 and gamma 0 .

Original: For Synthetic GMM data , we train a linear classifier for each dataset, initialized from 0 .
Revised: For Synthetic GMM data , we train a linear classifier for each dataset, initialized from 0 .

Original: We use logistic loss.
Revised: We use logistic loss.

Original: We train each model using SGD as optimizer with learning rate 1 e − 5 until training loss decrease below 0 .
Revised: We train each model using SGD as optimizer with learning rate 1 e − 5 until training loss decrease below 0 .

Original: D. O VERFITTING ON PTB WITH LANGUAGE MODELLING
Revised: D. O VERFITTING ON PTB WITH LANGUAGE MODELLING

Original: In this subsection, we show that the training a small Transformer on PTB for language modelling also leads to overfitting, as shown in Figure 5.
Revised: In this subsection, we show that the training a small Transformer on PTB for language modelling also leads to overfitting, as shown in Figure 5.

Original: This is expected based on our analysis.
Revised: This is expected based on our analysis.

Original: If we view language modelling as a classification task: predicting the next word based on a prefixed sequence, then the ground truth label will naturally be noisy.
Revised: If we view language modelling as a classification task: predicting the next word based on a prefixed sequence, then the ground truth label will naturally be noisy.

Original: D. M ORE GMM E XPERIMENTS
Revised: D. M ORE GMM E XPERIMENTS

Original: We hereby present more GMM experiments results.
Revised: We hereby present more GMM experiments results.

Original: We consider signal-to-noise ratio between { 10 , 20 , 40 , 80 , 160 } and the result is shown in figure 6.
Revised: We consider signal-to-noise ratio between { 10 , 20 , 40 , 80 , 160 } and the result is shown in figure 6.

Original: Notice that according to our theory, we can roughly separate the GMM models into three cases ignoring some technical conditions.
Revised: Notice that according to our theory, we can roughly separate the GMM models into three cases ignoring some technical conditions.

Original: [Equation]
Revised: [Equation]

Original: ( p/n ) 1 / 4 ) which corresponds to the case where [A6] doesn’t hold.
Revised: ( p/n ) 1 / 4 ) which corresponds to the case where [A6] doesn’t hold.

Original: In this case, as the signal-to-noise ratio is too small, even on noiseless data, the model will suffer a constant excess risk.
Revised: In this case, as the signal-to-noise ratio is too small, even on noiseless data, the model will suffer a constant excess risk.

Original: Case 2 ∥ µ ∥ /σ
Revised: Case 2 ∥ µ ∥ /σ

Original: = O
Revised: = O

Original: (( p/n ) 1 / 4 + √ log n ) while p = Θ( n ) .
Revised: (( p/n ) 1 / 4 + √ log n ) while p = Θ( n ) .

Original: In this case the signal-to-noise ratio is great enough, however the overparameterization is mild, hence although the model can benign overfit on noiseless data, it will fail to benigh overfit noisy data.
Revised: In this case the signal-to-noise ratio is great enough, however the overparameterization is mild, hence although the model can benign overfit on noiseless data, it will fail to benigh overfit noisy data.

Original: However early stopping on the noisy data will still gives a classifier that have nonvacuous generalization guarantee.
Revised: However early stopping on the noisy data will still gives a classifier that have nonvacuous generalization guarantee.

Original: Case 3 ∥ µ ∥ /σ
Revised: Case 3 ∥ µ ∥ /σ

Original: = O
Revised: = O

Original: (( p/n ) 1 / 4 + √ log n ) while p = ω ( n ) .
Revised: (( p/n ) 1 / 4 + √ log n ) while p = ω ( n ) .

Original: In this heavy overparameteized regime, the model will benignly overfit both the noisy and clean data.
Revised: In this heavy overparameteized regime, the model will benignly overfit both the noisy and clean data.

Original: Our simulation aligns with the theoretical prediction as (
Revised: Our simulation aligns with the theoretical prediction as (

Original: ) For
Revised: ) For

Original: small signal-to-noise ratio as 5 , we observe that when p/n grows to 2 7 , the linear model starts to suffer an constant excess risk even for the noiseless data.
Revised: small signal-to-noise ratio as 5 , we observe that when p/n grows to 2 7 , the linear model starts to suffer an constant excess risk even for the noiseless data.

Original: For the noisy data, we need a larger signal-to-noise ratio compared with the overparameterization.
Revised: For the noisy data, we need a larger signal-to-noise ratio compared with the overparameterization.

Original: This corresponds to Case 1 in our prediction and explain the constant excess risk in Figure 6(a) and the
Revised: This corresponds to Case 1 in our prediction and explain the constant excess risk in Figure 6(a) and the

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: overfitting in the heavy overparameterized regime in Figure 6(b), Figure 6(c), Figure 6(e) and
Revised: overfitting in the heavy overparameterized regime in Figure 6(b), Figure 6(c), Figure 6(e) and

Original: (2) We observe similar effect as in Figure 2 for ∥ µ ∥ σ spanning across { 10 , 20 , 40 , 80 , 160 } that in the mild overparameterized regime, non-benign overfitting happens consistently, while for the heavy overparameterized regime, benign overfitting takes place.
Revised: (2) We observe similar effect as in Figure 2 for ∥ µ ∥ σ spanning across { 10 , 20 , 40 , 80 , 160 } that in the mild overparameterized regime, non-benign overfitting happens consistently, while for the heavy overparameterized regime, benign overfitting takes place.

Original: (3) We also notes that using technique as early stopping prevents overfitting and gives a classifier with generalization guarantee for ∥ µ ∥ σ spanning across { 10 , 20 , 40 , 80 , 160 } as predicted.
Revised: (3) We also notes that using technique as early stopping prevents overfitting and gives a classifier with generalization guarantee for ∥ µ ∥ σ spanning across { 10 , 20 , 40 , 80 , 160 } as predicted.

Original: E D ISCUSSION E. C OMPARISON BETWEEN REGRESSION AND CLASSIFICATION
Revised: E D ISCUSSION E. C OMPARISON BETWEEN REGRESSION AND CLASSIFICATION

Original: Although Bartlett et al.
Revised: Although Bartlett et al.

Original: (2020) shows similar findings in regression regimes, we emphisize that the techniques can be pretty different.
Revised: (2020) shows similar findings in regression regimes, we emphisize that the techniques can be pretty different.

Original: The difference between regression and classification problems can be split into two folds:
Revised: The difference between regression and classification problems can be split into two folds:

Original: (a) technical difference.
Revised: (a) technical difference.

Original: Regression problem usually permits closed form solution through matrix pseudo-inverse while classification problem does not.
Revised: Regression problem usually permits closed form solution through matrix pseudo inverse while classification problem does not.

Original: Therefore, one needs to use other terms, e g. , margin, to study the generalization bound.
Revised: Therefore, one needs to use other terms, e.g. , margin, to study the generalization bound.

Original: Besides, the commonly-used operations in regression settings, e.g. , variance-bias decomposition does not apply in classification settings.
Revised: Besides, the commonly-used operations in regression settings, e.g. , variance-bias decomposition does not apply in classification settings.

Original: Therefore, we need some different techniques during the proof.
Revised: Therefore, we need some different techniques during the proof.

Original: For example, when proving Theorem3.1.2 we need to choose those samples with wrong labels, which is impossible in regression regimes.
Revised: For example, when proving Theorem3.1.2 we need to choose those samples with wrong labels, which is impossible in regression regimes.

Original: (b) different label noise type.
Revised: (b) different label noise type.

Original: In regression problems, the label noise is usually subGaussian type.
Revised: In regression problems, the label noise is usually subGaussian type.

Original: However, the label noise is totally different in classification problems and therefore their results cannot be extended to the classification settings.
Revised: However, the label noise is totally different in classification problems and therefore their results cannot be extended to the classification settings.

Original: However, as the most commonly-used learning task, the analysis on classification is important.
Revised: However, as the most commonly-used learning task, the analysis on classification is important.

Revised: E. D ETECTING B ENIGN O VERFITTING IN P RACTICE

Revised: Benign overfitting refers to that large models can fit data well without giving poor generalization performance (Bartlett et al., 2021), indicating the models operate outside the realm of uniform convergence.

Revised: There could be multiple mathematical instantiations of the statement.

Revised: For example,

Revised: Bartlett et al.

Revised: (2020) bound the generalization error of the last-epoch model with sample size and model size to characterize benign overfitting, and prove that the validation accuracy approaches

Revised: Bayesian optimal as sample size increases.

Revised: Our theoretical claims (Statements 1 and 3 of Theorem 3.1) follow the same criterion.

Revised: We show that the validation error of a fully trained linear model approaches the label error, and hence is Bayesian optimal.

Revised: However, the criterion generalization error converges to zero is hard to verify in experiments, as it is hard to determine when the model size and sample complexity are large enough for the generalization bound to be sufficiently small, given that the data set size is upper bounded.

Revised: Hence we adopt two mild assumptions to test whether benign overfitting happens empirically

Revised: Training the model for long-enough epochs (at least three times that of the standard practice) can work as an honest proxy for training for infinite epochs.

Revised: The gap of verification accuracy between the best-epoch model and the last-epoch model can work as an efficient proxy for the gap between the Bayesian optimal accuracy and the validation accuracy of the last-epoch model.

Revised: This assumption is supported by our theoretical predictions (Statements 2 and 3 of Theorem 3.1).

Revised: Hence we adopt the different notion of benign overfitting, as the reviewer mentioned, ’validation performance does not drop while the model fits more training data points’.

Revised: This notion is

Revised: 1. coherent to the intuitive definition embed all of the noise of the labels into the parameter estimate —without harming prediction accuracy ; 2.

Revised: easily experimentally verifiable; 3. closely related to theoretical predictions in our work and previous works, in the sense that based on theoretical prediction, in the mild overparameterization regime (our work), the drop will be significant regardless of model size, and in the heavy overparameterization regime (previous works), the drop will diminish when the model size and sample complexity increases.

Revised: Finally, we would like to stress that the similarity between the overparameterization through model size and training a model longer is not required for our analysis.