Original: Benign, Tempered, or Catastrophic: A Taxonomy of Overﬁtting
Revised: Benign, Tempered, or Catastrophic: A Taxonomy of Overﬁtting

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: The practical success of overparameterized neural networks has motivated therecent scientiﬁc study of interpolating methods – learning methods which are ableﬁt training data perfectly.
Revised: The practical success of overparameterized neural networks has motivated therecent scientiﬁc study of interpolating methods – learning methods which are ableto ﬁt training data perfectly.

Original: Empirically, certain interpolating methods can ﬁt noisytraining data without catastrophically bad test performance, which deﬁes standardintuitions from statistical learning theory.
Revised: Empirically, certain interpolating methods can ﬁt noisytraining data without catastrophically bad test performance, which deﬁes standardintuitions from statistical learning theory.

Original: Aiming to explain this, a large body ofrecent work has studied benign overﬁtting , a behavior seen in certain asymptoticsettings under which interpolating methods approach Bayes-optimality, even in thepresence of noise.
Revised: Aiming to explain this, a large body ofrecent work has studied benign overﬁtting , a behavior seen in certain asymptoticsettings under which interpolating methods approach Bayes-optimality, even in thepresence of noise.

Original: In this work, we argue that, while benign overﬁtting has beeninstructive to study, real interpolating methods like deep networks do not ﬁt benignly.
Revised: In this work, we argue that, while benign overﬁtting has beeninstructive to study, real interpolating methods like deep networks do not ﬁt benignly.

Original: That is, noise in the train set leads to suboptimal generalization, suggestingthat these methods fall in an intermediate regime between benign and catastrophicoverﬁtting, where asymptotic risk is neither unbounded nor Bayes-optimal.
Revised: That is, noise in the train set leads to suboptimal generalization, suggestingthat these methods fall in an intermediate regime between benign and catastrophicoverﬁtting, where asymptotic risk is neither unbounded nor Bayes-optimal.

Original: Wecall this behavior tempered overﬁtting .
Revised: Wecall this behavior tempered overﬁtting .

Original: We ﬁrst  kernel (ridge) regression, obtaining conditions on the ridge parameter and kernel eigenspectrum under which KRexhibits each of the three behaviors, thus obtaining a relatively complete pictureof overﬁtting for kernel regression.
Revised: We ﬁrst study kernel (ridge) regression,obtaining conditions on the ridge parameter and kernel eigenspectrum under which KR exhibits each of the three behaviors, thus obtaining a relatively complete pictureof overﬁtting for kernel regression.

Original: We ﬁnd that kernels with powerlaw spectra,which are ubiquitous in practice, display tempered overﬁtting.
Revised: We ﬁnd that kernels with powerlaw spectra,which are ubiquitous in practice, display tempered overﬁtting.

Original: We then provideempirical evidence that deep neural networks trained to interpolation are tempered,while those stopped early are benign.
Revised: We then provideempirical evidence that deep neural networks trained to interpolation are tempered,while those stopped early are benign.

Original:  Introduction
Revised:  Introduction

Original: In the last decade, the dramatic success of overparameterized deep neural networks (DNNs), indeﬁance of classical intuitions from statistical learning theory, has inspired the ﬁeld to reexaminethe theoretical foundations of generalization.
Revised: In the last decade, the dramatic success of overparameterized deep neural networks (DNNs), indeﬁance of classical intuitions from statistical learning theory, has inspired the ﬁeld to reexaminethe theoretical foundations of generalization.

Original: Classical statistical learning theory suggests that analgorithm which interpolates (i e. perfectly ﬁts) its training data will typically catastrophically overﬁtat test time, generalizing no better than a random function 1 .
Revised: Classical statistical learning theory suggests that analgorithm which interpolates (i e. perfectly ﬁts) its training data will typically catastrophically overﬁtat test time, generalizing no better than a random function 1 .

Original: Figure 1c illustrates the catastrophicoverﬁtting classically expected of an interpolating method.
Revised: Figure 1c illustrates the catastrophicoverﬁtting classically expected of an interpolating method.

Original: Yet DNNs can interpolate their trainingdata (even in the presence of noise) and generalize well nonetheless [Neyshabur et al., 2015, Zhanget al., 2017]— which motivated the need for a new theoretical framework, replacing classical statistical learning theory, within which to understand this phenomenon. 
Revised: Yet DNNs can interpolate their trainingdata (even in the presence of noise) and generalize well nonetheless [Neyshabur et al., 2015, Zhanget al., 2017]. This suggests the need for a new theoretical framework, replacing classical statistical learning theory, within which to understand this phenomenon. [Figure]

Revised: Catastrophic

Original: This need eventually led to the identiﬁcation and study of benign overﬁtting [Bartlett et al.
Revised: This need eventually led to the identiﬁcation and study of benign overﬁtting [Bartlett et al.

Original: , 2020](also called “harmless interpolation” Muthukumar et al.
Revised: , 2020](also called “harmless interpolation” Muthukumar et al.

Original: [2020]), a phenomenon in which certainmethods that perfectly ﬁt the training data still approach Bayes-optimal generalization in the limit oflarge trainset size.
Revised: [2020]), a phenomenon in which certainmethods that perfectly ﬁt the training data still approach Bayes-optimal generalization in the limit oflarge trainset size.

Original: Intuitively speaking, benignly-overﬁtting methods ﬁt the target function globally,yet ﬁt the noise only locally, and the addition of more label noise does not asymptotically degradegeneralization.
Revised: Intuitively speaking, benignly-overﬁtting methods ﬁt the target function globally,yet ﬁt the noise only locally, and the addition of more label noise does not asymptotically degradegeneralization.

Original: Figure 1a illustrates a simple method that is asymptotically benign.
Revised: Figure 1a illustrates a simple method that is asymptotically benign.

Original: The study ofbenign overﬁtting has proven fruitful, leading to rich mathematical insights into high-dimentionallearning 2 .
Revised: The study ofbenign overﬁtting has proven fruitful, leading to rich mathematical insights into high-dimensionallearning 2 .

Original: And benign overﬁtting is certainly closer to the real behavior of DNNs compared to thecatastrophic overﬁtting expected from classical intuition.
Revised: And benign overﬁtting is certainly closer to the real behavior of DNNs compared to thecatastrophic overﬁtting expected from classical intuition.

Original: Curves show test classiﬁcation error vs. training label ﬂip probability for a Wide ResNet 28 ⇥trained to interpolation on binary CIFAR-10 (animals vs. vehicles) for different training sizes n .

Original: This is a noise proﬁle , as discussed in Section 2.

Original: That said, simple experiments reveal that manystandard DNNs do not overﬁt benignly : whentraining on noisy data, DNNs do not diverge catastrophically, but neither do they approach Bayesoptimal risk.
Revised: That said, simple experiments reveal that many standard DNNs do not overﬁt benignly : when trainingon noisy data, DNNs do not diverge catastrophically, but neither do they approach Bayes-optimal risk.

Original: Instead, they converge to a predictorthat is neither catastrophic nor optimal, but rathersomewhere in between— with error that increasesas the noise in the data increases.
Revised: Instead, they converge to a predictor that is neither catastrophic nor optimal, but rather somewherein between— with error that increases as the noise in the data increases.

Original: Figure 2 depictssuch an experiment: a ResNet is trained on a binary variant of CIFAR-10 with varying amountsof training label noise and increasing n , revealing that more train noise indeed causes more testerror, even for large n .
Revised: Figure 2 depicts such anexperiment: a ResNet is trained on a binary variant of CIFAR-10 with varying amounts of traininglabel noise and increasing n , revealing that more train noise indeed causes more test error, evenfor large n .

Original: The observation that interpolating DNNs are harmed by label noise iswell-known (e.g. Zhang et al.
Revised: The observation that interpolating DNNs are harmed by label noise is well-known (e.g. Zhang et al.

Original: [2017]), and thelinear shape of noise proﬁle in Figure 2 is consistent with the prediction of the Feature Calibration Conjecture of Nakkiran and Bansal [2020].
Revised: [2017]), and the linear shape of noise proﬁle in Figure 2 is consistent with the predictionof the Feature Calibration Conjecture of Nakkiran and Bansal [2020].

Original: These observations suggest that, in the search fora paradigm to understand modern interpolatingmethods, we should identify and study a regimeintermediate between benign and catastrophic.
Revised: These observations suggestthat, in the search for a paradigm to understand modern interpolating methods, we should identifyand study a regime intermediate between benign and catastrophic.

Original: 1.1 Summary of Contributions
Revised: 1.1 Summary of Contributions

Original: In this work we formally identify an intermediate regime between benign and catastrophic overﬁtting,which we call tempered overﬁtting , because the noise’s harmful effect is tempered but still nonzero.
Revised: In this work we formally identify an intermediate regime between benign and catastrophic overﬁtting,which we call tempered overﬁtting , because the noise’s harmful effect is tempered but still nonzero.

Original: We ﬁnd that, even as the number of training examples n approaches inﬁnity , both DNNs trainedto interpolation and (ridgeless) kernel regression (KR) using certain common kernels fall intothis intermediate regime, as do common methods like 1-nearest-neighbors and piecewise linearinterpolation (as in Figure 1b).
Revised: We ﬁnd that, even as the number of training examples n approaches inﬁnity , both DNNs trainedto interpolation and (ridgeless) kernel regression (KR) using certain common kernels fall intothis intermediate regime, as do common methods like 1-nearest-neighbors and piecewise linearinterpolation (as in Figure 1b).

Original: Our tempered regime completes the taxonomy of overﬁtting: anylearning procedure is either benign, tempered, or catastrophic in the asymptotic limit.
Revised: Our tempered regime completes the taxonomy of overﬁtting: anylearning procedure is either benign, tempered, or catastrophic in the asymptotic limit.

Original: We begin in Section 2 with preliminaries, formal deﬁnitions of the three regimes of overﬁtting, anda taxonomy of some common ML methods according to these regimes.
Revised: We begin in Section 2 with preliminaries, formal deﬁnitions of the three regimes of overﬁtting, anda taxonomy of some common ML methods according to these regimes.

Original: We generally consider analgorithm’s limiting behavior as n ! 1 , as in the study of statistical consistency 3 .
Revised: We generally consider analgorithm’s limiting behavior as n ! 1 , as in the study of statistical consistency 3 .

Original: We are generallymore interested in algorithms’ asymptotic test risk than in the fact that they interpolate, and weconsider non-interpolating methods as well as interpolating methods.
Revised: We are generally We that consistency is benign overﬁtting more interested in algorithms’ asymptotic test risk than in the fact that they interpolate, and weconsider non-interpolating methods as well as interpolating methods.

Original: In Section 3, we study these three regimes for kernel regression (KR).
Revised: In Section 3, we study these three regimes for kernel regression (KR).

Original: Using recent spectral theoriesof the generalization of KR, we obtain conditions on the ridge parameter and kernel eigenspectrumunder which KR falls into each of the three regimes.
Revised: Using recent spectral theoriesof the generalization of KR, we obtain conditions on the ridge parameter and kernel eigenspectrumunder which KR falls into each of the three regimes.

Original: Importantly, we ﬁnd that ridgeless kernels withpowerlaw spectra, including the Laplace kernel and ReLU fully-connected NTKs, are asymptoticallytempered , not benign.
Revised: Importantly, we ﬁnd that ridgeless kernels withpowerlaw spectra, including the Laplace kernel and ReLU fully-connected NTKs, are asymptoticallytempered , not benign.

Original: We conﬁrm our theory with experiments on real and synthetic data.
Revised: We conﬁrm our theory with experiments on real and synthetic data.

Original: In Section 4, we empirically study overﬁtting for DNNs.
Revised: In Section 4, we empirically study overﬁtting for DNNs.

Original: We give evidence that standard interpolating
Revised: We give evidence that standard interpolating

Original: DNNs exhibit tempered overﬁtting, and not benign overﬁtting, which highlights the relevance of ourframework for understanding modern methods.
Revised: DNNs exhibit tempered overﬁtting, and not benign overﬁtting, which highlights the relevance of ourframework for understanding modern methods.

Original: We conclude with a discussion in Section 5.
Revised: We conclude with a discussion in Section 5.

Original: 1.2 Relation to Prior Work
Revised: 1.2 Relation to Prior Work

Original:  Our work is inspired by recent developments in the theory and empirics of interpolating methods[Bartlett et al., 2020, Belkin et al., 2019c,d, Ji et al., 2021, Koehler et al., 2021, Liang and Rakhlin,2018, 2020, Muthukumar et al., 2020, Rakhlin and Zhai, 2019, Tsigler and Bartlett, 2020].
Revised: [Figure] Our work is inspired by recent developments inthe theory and empirics of interpolating methods[Bartlett et al., 2020, Belkin et al., 2019c,d, Ji et al. ,2021, Koehler et al., 2021, Liang and Rakhlin,2018, 2020, Muthukumar et al., 2020, Rakhlin and Zhai, 2019, Tsigler and Bartlett, 2020].

Original: Many ofthese works prove that certain interpolating estimators are statistically consistent in certain settings—demonstrating benign overﬁtting .
Revised: Many ofthese works prove that certain interpolating estimators are statistically consistent in certain settings—demonstrating benign overﬁtting .

Original: In contrast, we argue that a wide range of interpolators, including
Revised: In contrast, weargue that a wide range of interpolators, including

Original: DNNs used in practice, are not benign .
Revised: DNNs used in practice, are not benign .

Original: Our work is compatible with prior work because our formalismdiffers in several crucial ways, which leads us to different conclusions.
Revised: Our workis compatible with prior work because our formalism differs in several crucial ways, which leads usto different conclusions.

Original: Brieﬂy, many prior works assume that the target task always lies in “high enough dimension” relativeto the number of samples n , while our work considers the limit n ! 1 for tasks of ﬁxed dimension 91 d — as is the realistic asymptotic in practice.
Revised: Brieﬂy, many prior works assume that the targettask always lies in “high enough dimension” relative to the number of samples n , while our workconsiders the limit n ! 1 for tasks of ﬁxed di- 89 mension d — as is the realistic asymptotic in practice.

Original: For example, in the linear regression setting, Bartlettet al.
Revised: For example, in the linear regression setting, Bartlett et al.

Original: [2020] highlight that benign overﬁtting occurs most robustly when input dimension grows fasterthan the sample size .
Revised: [2020] highlight that benign overﬁtting occurs most robustly when input dimensiongrows faster than the sample size .

Original: Other works explicitly scale the ambient problem dimension and sample sizeto inﬁnity together, at a proportional rate [Hastie et al., 2019, Liang and Rakhlin, 2018,Mei and
Revised: Other works explicitly scale the ambient problem dimension andsample size to inﬁnity together, at a proportionalrate [Hastie et al., 2019, Liang and Rakhlin, 2018,

Original:  Montanari, 2019].
Revised: Mei and Montanari, 2019].

Original: Taking a different approach, Frei et al.
Revised: Taking a different approach, Frei et al.

Original: [2022] prove that interpolating two-layernetworks can achieve close to optimal test error on certain distributions— but require an assumptionthat n  ⌦ ( d ) .
Revised: [2022] prove that interpolatingtwo-layer networks can achieve close to optimaltest error on certain distributions— but require an assumption that n  ⌦ ( d ) .

Original: And Koehler et al. [2021] state a generalization bound for interpolators that decays to0 with n , but also require n  d .
Revised: And Koehler et al.[2021] state a generalization bound for interpolators that decays to 0 with n , but also require n  d .

Original: In summary, a bulk of prior benign overﬁtting results apply in theregime where n is large, but still restricted to be smaller than the dimension of the problem, whereaswe do not consider such restrictions.
Revised: In summary, a bulk of prior benign overﬁtting results apply in the regime where n is large, butstill restricted to be smaller than the dimension of the problem, whereas we do not consider suchrestrictions.

Original: Our work is compatible with prior works which observe that excess risk of certain interpolatingmethods decays with ambient dimension— thus high-dimensional problems enjoy a “blessing ofdimensionality.”
Revised: Our work is compatible with prior works which observe that excess risk of certain interpolatingmethods decays with ambient dimension— thus high-dimensional problems enjoy a “blessing ofdimensionality.”

Original: For example, the simplicial interpolation scheme of Belkin et al.
Revised: For example, the simplicial interpolation scheme of Belkin et al.

Original: [2018a] has excessrisk that decays as O (2  d ) for ambient dimension d .
Revised: [2018a] has excessrisk that decays as O (2  d ) for ambient dimension d .

Original: And Rakhlin and Zhai [2019] show that kernelridgeless regression with the Laplace kernel is inconsistent in any ﬁxed dimension — but with alower-bound that decays with dimension.
Revised: And Rakhlin and Zhai [2019] show that kernelridgeless regression with the Laplace kernel is inconsistent in any ﬁxed dimension — but with alower-bound that decays with dimension.

Original: This agrees with our conclusions, that many interpolatingmethods are tempered on real distributions, which have ﬁxed dimension.
Revised: This agrees with our conclusions, that many interpolatingmethods are tempered on real distributions, which have ﬁxed dimension.

Original:  The Three Types of Overﬁtting
Revised:  The Three Types of Overﬁtting

Original: Here we formally present our taxonomy of overﬁtting, delineating the three types of asymptoticbehaviors which learning procedures can exhibit.
Revised: Here we formally present our taxonomy of overﬁtting, delineating the three types of asymptoticbehaviors which learning procedures can exhibit.

Original: 2.1 Deﬁnitions
Revised: 2.1 Deﬁnitions

Original: We consider the standard setting of supervised learning.
Revised: We consider the standard setting of supervised learning.

Original: For simplicity, we present deﬁnitions forregression, but these are readily extended to classiﬁcation.
Revised: For simplicity, we present deﬁnitions forregression, but these are readily extended to classiﬁcation.

Original: We wish to learn a function b f : X !
Revised: We wish to learn a function b f : X !

Original: R 114 from a size- n dataset
Revised: R 118 from a size- n dataset

Original: D n ⌘ { ( x i , y i ) } ni =1 ⇠ D , where D is a joint distribution over X ⇥ R , and
Revised: D n ⌘ { ( x i , y i ) } ni =1 ⇠ D , where D is a joint distribution over X ⇥ R , and

Original: X is the input domain.
Revised: X is the input domain.

Original: We evaluate the generalization performance of b f by the mean squaredrisk : R ( b f ) :=
Revised: We evaluate the generalization performance of b f by the mean squaredrisk : R ( b f ) :=

Original: E x,y ⇠ D h ( b f ( x )  y )i .
Revised: E x,y ⇠ D h ( b f ( x )  y )i .

Original: The Bayes-optimal regression function is given by f ⇤ :=
Revised: The Bayes-optimal regression function is given by f ⇤ :=

Original: argmin
Revised: argmin

Original: f R ( f ) , where the minimization is over all measurable functions, and has risk R ⇤ which iscalled the irreducible risk .
Revised: f R ( f ) , where the minimization is over all measurable functions, and has risk R ⇤ which iscalled the irreducible risk .

Original: The excess risk of any function b f is given by R ( b f ) :
Revised: The excess risk of any function b f is given by R ( b f ) :

Original: = R ( b f )  R ⇤ .
Revised: = R ( b f )  R ⇤ .

Original: Wesay an estimator achieves interpolation if b f n ( x i ) =
Revised: Wesay an estimator achieves interpolation if b f n ( x i ) =

Original: y i for all ( x i , y i )
Revised: y i for all ( x i , y i )

Original: 2 D n .
Revised: 2 D n .

Original: These deﬁnitions arereadily generalized to classiﬁcation, using classiﬁcation error in place of MSE.
Revised: These deﬁnitions arereadily generalized to classiﬁcation, using classiﬁcation error in place of MSE.

Original: Learning Procedure.
Revised: Learning Procedure.

Original: The central objects in our taxonomy are learning procedures .
Revised: The central objects in our taxonomy are learning procedures .

Original: Our deﬁnitionof learning procedure is fully general, which allows us to discuss anything from neural networksto 1-nearest-neighbors.
Revised: Our deﬁnitionof learning procedure is fully general, which allows us to discuss anything from neural networksto 1-nearest-neighbors.

Original: Informally, a learning procedure is a fully speciﬁcation of which model tooutput on a given train set, speciﬁed for all sizes of train sets.
Revised: Informally, a learning procedure is a fully speciﬁcation of which model tooutput on a given train set, speciﬁed for all sizes of train sets.

Original: Formally, a learning procedure A :=
Revised: Formally, a learning procedure A :=

Original: { A n } n is a sequence of (potentially stochastic) functions,indexed by sample size n 2 N .
Revised: { A n } n is a sequence of (potentially stochastic) functions,indexed by sample size n 2 N .

Original: At each n , the function A n : D n 7!
Revised: At each n , the function A n : D n 7!

Original: b f n inputs a train set D nand outputs a “model” b f n : X !
Revised: b f n inputs a train set D nand outputs a “model” b f n : X !

Original: R .
Revised: R .

Original: Note that this n -dependence allows learning proceduresto be non-uniform , and use different algorithms for each sample size n .
Revised: Note that this n -dependence allows learning proceduresto be non-uniform , and use different algorithms for each sample size n .

Original: For example, it allowsprocedures which scale-up the DNN model size along with the sample size n , or which vary thekernel bandwidth with n .
Revised: For example, it allowsprocedures which scale-up the DNN model size along with the sample size n , or which vary thekernel bandwidth with n .

Original: The expected risk of a learning procedure on n examples from distribution
Revised: The expected risk of a learning procedure on n examples from distribution

Original: D is R n := E A n , D n [ R ( A n ( D n ))] .
Revised: D is R n := E A n , D n [ R ( A n ( D n ))] .

Original: 2.2 The Taxonomy
Revised: 2.2 The Taxonomy

Original: We shall categorize learning procedures in terms of their asymptotic expected risk .
Revised: We shall categorize learning procedures in terms of their asymptotic expected risk .

Original: We handleregression and K -class classiﬁcation settings separately, due to their different loss scalings.
Revised: We handleregression and K -class classiﬁcation settings separately, due to their different loss scalings.

Original: As thenumber of samples n ! 1 , the sequence of expected risks {R n } can behave in one of three ways:
Revised: As thenumber of samples n ! 1 , the sequence of expected risks {R n } can behave in one of three ways:

Original: Regression
Revised: Regression

Original: Classiﬁcation
Revised: Classiﬁcation

Original: Benign
Revised: Benign

Original: [Equation]
Revised: [Equation]

Original: Tempered
Revised: Tempered

Original: [Equation]
Revised: [Equation]

Original: Catastrophic
Revised: Catastrophic

Original: [Equation]
Revised: [Equation]

Original: This deﬁnes our taxonomy.
Revised: This deﬁnes our taxonomy.

Original: There is technically a fourth option – that the limit does not exist – but,to our knowledge, this does not describe any non-pathological algorithms.
Revised: There is technically a fourth option – that the limit does not exist – but,to our knowledge, this does not describe any non-pathological algorithms.

Original: All together, this set ofbehaviors is exhaustive, describing every possible learning procedure.
Revised: All together, this set ofbehaviors is exhaustive, describing every possible learning procedure.

Original: Note that the deﬁnitions forclassiﬁcation and regression are identical, except for the bounds at 1 replaced by the error of thepredictor choosing a uniformly random label ( 1  1 K ).
Revised: Note that the deﬁnitions forclassiﬁcation and regression are identical, except for the bounds at 1 replaced by the error of thepredictor choosing a uniformly random label ( 1  1 K ).

Original: 2.3 Applying Our Taxonomy
Revised: 2.3 Applying Our Taxonomy

Original: Our taxonomy handles general learning procedures, even those which do not interpolate their trainsets 4 .
Revised: Our taxonomy handles general learning procedures, even those which do not interpolate their trainsets 4 .

Original: Thus, we can apply it to describe many existing methods in machine learning.
Revised: Thus, we can apply it to describe many existing methods in machine learning.

Original: In doing so,we reﬁne the language around statistical consistency: Many methods were known to be statisticallyinconsistent (i.e. not benign), but we highlight that there are two ways to be inconsistent: temperedor catastrophic.
Revised: In doing so,we reﬁne the language around statistical consistency: Many methods were known to be statisticallyinconsistent (i.e. not benign), but we highlight that there are two ways to be inconsistent: temperedor catastrophic.

Original: Benign
Revised: Benign

Original: Tempered
Revised: Tempered

Original: Catastrophic
Revised: Catastrophic

Original: • Early-stopped DNNs • KR with ridge • k -NN ( k ⇠ log n ) • Nadaraya-Watson kernel smoothing w/Hilbert kernel •
Revised: • Early-stopped DNNs • KR with ridge • k -NN ( k ⇠ log n ) • Nadaraya-Watson kernel smoothing w/Hilbert kernel •

Original: Interpolating DNNs • Laplace KR • ReLU NTKs • k -NN (constant k ) • Simplicial interpolation • Gaussian KR • Critically-parameterized regression
Revised: Interpolating DNNs • Laplace KR • ReLU NTKs • k -NN (constant k ) • Simplicial interpolation • Gaussian KR • Critically-parameterized regression

Original: To illustrate our taxonomy, we give several examples of known results in Table 1.
Revised: To illustrate our taxonomy, we give several examples of known results in Table 1.

Original: Any statisticallyconsistent method is by deﬁnition benign : this includes non-interpolating methods such as earlystopped DNNs [Ji et al., 2021] and k -nearest-neighbors with k ⇠ log n [Chaudhuri and Dasgupta,2014, Cover and Hart, 1967], as well as interpolating methods such as Nadarya-Watson kernelsmoothing [Devroye et al., 1998] and k-NN schemes
Revised: Any statisticallyconsistent method is by deﬁnition benign : this includes non-interpolating methods such as earlystopped DNNs [Ji et al., 2021] and k -nearest-neighbors with k ⇠ log n [Chaudhuri and Dasgupta,2014, Cover and Hart, 1967], as well as interpolating methods such as Nadarya-Watson kernelsmoothing [Devroye et al., 1998] and k-NN schemes

Original: [Belkin et al., 2018a, 2019c].
Revised: [Belkin et al., 2018a, 2019c].

Original: Catastrophic learning procedures are also abundant.
Revised: Catastrophic learning procedures are also abundant.

Original: Parametric models which are “criticallyparameterized” (i.e. at the double descent peak)
Revised: Parametric models which are “criticallyparameterized” (i.e. at the double descent peak)

Original: overﬁt catastrophically: this includes Random
Revised: overﬁt catastrophically: this includes Random

Original: Feature regression with number of features p = n [Mei and Montanari, 2019], and, more generally,for a broad class of linear and random feature methods [Holzmüller, 2021].
Revised: Feature regression with number of features p = n [Mei and Montanari, 2019], and, more generally,for a broad class of linear and random feature methods [Holzmüller, 2021].

Original: Generic empirical-riskminimization for a hypothesis class with VC-dimension > n will also overﬁt catastrophically in theworst case.
Revised: Generic empirical-riskminimization for a hypothesis class with VC-dimension > n will also overﬁt catastrophically in theworst case.

Original: It turns out that RBF kernel regression under natural assumptions also catastrophicallyoverﬁts, as we show in Section 3.
Revised: It turns out that RBF kernel regression under natural assumptions also catastrophicallyoverﬁts, as we show in Section 3.

Original: Finally, many methods, from classical to modern, exhibit tempered overﬁtting.
Revised: Finally, many methods, from classical to modern, exhibit tempered overﬁtting.

Original: First, it is wellknown that 1-nearest-neighbor converges to an asymptotic risk that is ﬁnite, but bounded away from
Revised: First, it is wellknown that 1-nearest-neighbor converges to an asymptotic risk that is ﬁnite, but bounded away from

Original: Bayes optimal 5 — thus, it is tempered.
Revised: Bayes optimal 5 — thus, it is tempered.

Original: Further, any “underﬁtting” method, such as Empirical Risk
Revised: Further, any “underﬁtting” method, such as Empirical Risk

Original: Minimization over a hypothesis class with VC-dimension ⌧ n will be tempered, as long as thishypothesis class does not contain the ground-truth function.
Revised: Minimization over a hypothesis class with VC-dimension ⌧ n will be tempered, as long as thishypothesis class does not contain the ground-truth function.

Original: In Section 4, we empirically demonstratethat many standard DNNs exhibit tempered overﬁtting, when trained to interpolation.
Revised: In Section 4, we empirically demonstratethat many standard DNNs exhibit tempered overﬁtting, when trained to interpolation.

Original: We complementthis with theoretical results in Section 3, showing that kernel regression with the Laplace kernel, aswell as with the ReLU
Revised: We complementthis with theoretical results in Section 3, showing that kernel regression with the Laplace kernel, aswell as with the ReLU

Original: NTK [Bietti and Bach, 2020, Li et al., 2020, Nguyen et al., 2021] exhibitstempered overﬁtting.
Revised: NTK [Bietti and Bach, 2020, Li et al., 2020, Nguyen et al., 2021] exhibitstempered overﬁtting.

Original: Thus, our new category of tempered overﬁtting is not merely a theoreticalpossibility, but rather captures many natural and practically-used learning methods.
Revised: Thus, our new category of tempered overﬁtting is not merely a theoreticalpossibility, but rather captures many natural and practically-used learning methods.

Original: 2.4 Noise Proﬁles
Revised: 2.4 Noise Proﬁles

Original: To study asymptotic risk, we use a metric we call a noise proﬁle .
Revised: To study asymptotic risk, we use a metric we call a noise proﬁle .

Original: A noise proﬁle characterizes thesensitivity of a learning procedure to noise in the training set.
Revised: A noise proﬁle characterizes thesensitivity of a learning procedure to noise in the training set.

Original: For a given learning procedure A anddata distribution D , a noise proﬁle P n describes how the asymptotic risk varies with respect to  , thelevel of artiﬁcial noise added to training targets.
Revised: For a given learning procedure A anddata distribution D , a noise proﬁle P n describes how the asymptotic risk varies with respect to  , thelevel of artiﬁcial noise added to training targets.

Original: Formally, the noise proﬁle P A is
Revised: Formally, the noise proﬁle P A is

Original: [Equation]
Revised: [Equation]

Original: where D (  ) denotes n i.i.d.
Revised: where D (  ) denotes n i.i.d.

Original: samples from the distribution D , with  -level of label noise added.
Revised: samples from the distribution D , with  -level of label noise added.

Original: Thelabel noise  2 R denotes a kind of noise that depends on the problem setting:  is the variance ofadditive Gaussian noise for regression settings, or the label-ﬂip probability for classiﬁcation settings.
Revised: Thelabel noise  2 R denotes a kind of noise that depends on the problem setting:  is the variance ofadditive Gaussian noise for regression settings, or the label-ﬂip probability for classiﬁcation settings.

Original: We cannot empirically evaluate the n !
Revised: We cannot empirically evaluate the n !

Original: 1 limit exactly, so we estimate it using asymptotics fromﬁnite but large sample sizes.
Revised: 1 limit exactly, so we estimate it using asymptotics fromﬁnite but large sample sizes.

Original: As shown in Figure 2, noise proﬁles are easily plotted and reveal at aglance whether an learning procedure is benign, tempered, or catastrophic.
Revised: As shown in Figure 2, noise proﬁles are easily plotted and reveal at aglance whether an learning procedure is benign, tempered, or catastrophic.

Original:  Overﬁtting in Kernel Regression
Revised:  Overﬁtting in Kernel Regression

Original: Before characterizing the asymptotic behavior for DNNs via experiment, we ﬁrst turn to a simplerlearning method for which these three regimes can be theoretically understood.
Revised: Before characterizing the asymptotic behavior for DNNs via experiment, we ﬁrst turn to a simplerlearning method for which these three regimes can be theoretically understood.

Original: We study kernel regression (KR), a widely-used nonparameteric learning algorithm.
Revised: We study kernel regression (KR), a widely-used nonparameteric learning algorithm.

Original: Theoretical interest in thisalgorithm has increased signiﬁcantly in recent years due to the discovery that trained neural networksconverge to KR in the inﬁnite-width and inﬁnite-time limit Jacot et al.
Revised: Theoretical interest in thisalgorithm has increased signiﬁcantly in recent years due to the discovery that trained neural networksconverge to KR in the inﬁnite-width and inﬁnite-time limit Jacot et al.

Original: [2018], suggesting that insightsinto KR simultaneously shed light on overparameterized neural networks.
Revised: [2018], suggesting that insightsinto KR simultaneously shed light on overparameterized neural networks.

Original: KR is fully speciﬁed by a positive-semideﬁnite kernel function K : R d ⇥ R d !
Revised: KR is fully speciﬁed by a positive-semideﬁnite kernel function K : R d ⇥ R d !

Original: R and a ridgeparameter   0 .
Revised: R and a ridgeparameter   0 .

Original: As described above, we allow the training set D n to contain n samples ( x i , y i )
Revised: As described above, we allow the training set D n to contain n samples ( x i , y i )

Original: ⇠ D , 189 where here we assume that y i =
Revised: ⇠ D , 193 where here we assume that y i =

Original: f ⇤ ( x i )+ ⌘ i , where ⌘ i ⇠ N (0 ,  2 ) .
Revised: f ⇤ ( x i )+ ⌘ i , where ⌘ i ⇠ N (0 ,  2 ) .

Original: KR returns the predicted functionb f given by
Revised: KR returns the predicted functionb f given by

Original: [Equation]
Revised: [Equation]

Original: where K ( D n , D n ) is the “data-data kernel matrix" with components K ( D n , D n ) ij =
Revised: where K ( D n , D n ) is the “data-data kernel matrix" with components K ( D n , D n ) ij =

Original: K ( x i , x j ) , and 192 K ( x,
Revised: K ( x i , x j ) , and 196 K ( x,

Original: D ) is a row vector with components K ( x, D ) i
Revised: D ) is a row vector with components K ( x, D ) i

Original: = K ( x, x i ) , and Y is a column vector of targetlabels 6 .
Revised: = K ( x, x i ) , and Y is a column vector of targetlabels 6 .

Original: Existing literature already provides examples of KR exhibiting all three asymptotic behaviors.
Revised: Existing literature already provides examples of KR exhibiting all three asymptotic behaviors.

Original: Benignoverﬁtting has been analyzed for highly-overparameterized linear regression [Bartlett et al., 2020,
Revised: Benignoverﬁtting has been analyzed for highly-overparameterized linear regression [Bartlett et al., 2020,

Original: Belkin et al., 2019b, Hastie et al., 2019, Muthukumar et al., 2020], a special case of KR.
Revised: Belkin et al., 2019b, Hastie et al., 2019, Muthukumar et al., 2020], a special case of KR.

Original: It iswell-known that KR with a positive ridge value (with appropriate scaling conventions) is consistent[Christmann and Steinwart, 2007].
Revised: It iswell-known that KR with a positive ridge value (with appropriate scaling conventions) is consistent[Christmann and Steinwart, 2007].

Original: Furthermore, in 1D, a Laplace kernel approaches piecewise linearinterpolation as n ! 

Revised: Furthermore, in 1D, a Laplace kernel approaches piecewise linearinterpolation as n !

Original: [Belkin et al., 2018a], which exhibits tempered overﬁtting, and Rakhlin and
Revised: [Belkin et al., 2018a], which exhibits tempered overﬁtting, and Rakhlin and

Original: Zhai [2019] proved that the Laplace kernel does not overﬁt benignly in (ﬁxed) dimension greaterthan one.
Revised: Zhai [2019] proved that the Laplace kernel does not overﬁt benignly in (ﬁxed) dimension greaterthan one.

Original: Finally, it is known in experimental folklore (though not theoretically, to our knowledge)that KR with a Gaussian kernel and zero ridge tends to yield poorly-conditioned kernel matrices andcatastrophic behavior.
Revised: Finally, it is known in experimental folklore (though not theoretically, to our knowledge)that KR with a Gaussian kernel and zero ridge tends to yield poorly-conditioned kernel matrices andcatastrophic behavior.

Original: Here we derive general conditions under which KR falls into each regime,solidifying these various observations into a uniﬁed picture.
Revised: Here we derive general conditions under which KR falls into each regime,solidifying these various observations into a uniﬁed picture.

Original: As our chief tool for obtaining these conditions, we use recently-derived closed-form expressionsfor the generalization MSE of KR in terms of the eigenspectrum of the kernel (with respect to thedata measure) and the eigendecomposition of the target function.
Revised: As our chief tool for obtaining these conditions, we use recently-derived closed-form expressionsfor the generalization MSE of KR in terms of the eigenspectrum of the kernel (with respect to thedata measure) and the eigendecomposition of the target function.

Original: These eigenvalues {  i } 1 i =1 and 208 orthonormal eigenfunctions {  i } 1 i =1 are given by
Revised: These eigenvalues {  i } 1 i =1 and 212 orthonormal eigenfunctions {  i } 1 i =1 are given by

Original: [Equation]
Revised: [Equation]

Original: We note that  i  0 because a kernel must be positive-semideﬁnite and that
Revised: We note that  i  0 because a kernel must be positive-semideﬁnite and that

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: i =
Revised: i =

Original: Tr [ K ] ⌘
Revised: Tr [ K ] ⌘

Original: E x ⇠ p [ K ( x, x )] , which we assume is ﬁnite.
Revised: E x ⇠ p [ K ( x, x )] , which we assume is ﬁnite.

Original: We shall assume that the eigenvalues are indexed indecreasing order, with  i   j if i < j .
Revised: We shall assume that the eigenvalues are indexed indecreasing order, with  i   j if i < j .

Original: Because the eigenfunctions form a complete basis, we canalso decompose the target function as f ⇤ ( x ) =
Revised: Because the eigenfunctions form a complete basis, we canalso decompose the target function as f ⇤ ( x ) =

Original: P i v i
Revised: P i v i

Original:  i ( x ) , where { v i } 1 i =1 are the eigencoefﬁcients.
Revised:  i ( x ) , where { v i } 1 i =1 are the eigencoefﬁcients.

Original: Recent results have shown that, remarkably, knowledge of the eigenvalues and eigencoefﬁcients (butnot the eigenfunctions) is typically sufﬁcient to approximate test MSE.
Revised: Recent results have shown that, remarkably, knowledge of the eigenvalues and eigencoefﬁcients (butnot the eigenfunctions) is typically sufﬁcient to approximate test MSE.

Original: Bordelon et al.
Revised: Bordelon et al.

Original: [2020] and
Revised: [2020] and

Original: Jacot et al.
Revised: Jacot et al.

Original: [2020] derived equivalent expressions for test MSE using the replica method and randommatrix theory respectively, and Simon et al.
Revised: [2020] derived equivalent expressions for test MSE using the replica method and randommatrix theory respectively, and Simon et al.

Original: [2021] then found that this result can be derived andexpressed more simply in terms of modewise learnabilities {L i } 1 i
Revised: [2021] then found that this result can be derived andexpressed more simply in terms of modewise learnabilities {L i } 1 i

Original: =1 , a set of scores in [0 , 1] which 218 indicate how well each eigenmode is learned at a given n .
Revised: =1 , a set of scores in [0 , 1] which 222 indicate how well each eigenmode is learned at a given n .

Original: These approximations are nonrigorous(relying on informal but well-accepted methods from statistical physics), but comparisons with true
Revised: These approximations are nonrigorous(relying on informal but well-accepted methods from statistical physics), but comparisons with true

Original: MSE using real and synthetic datasets generally reveal a close match, particularly at large n .
Revised: MSE using real and synthetic datasets generally reveal a close match, particularly at large n .

Original: Usingthe formulation of Simon et al.
Revised: Usingthe formulation of Simon et al.

Original: [2021], test MSE R n is approximated by
Revised: [2021], test MSE R n is approximated by

Original: [Equation]
Revised: [Equation]

Original: This MSE includes noise on test labels; to compute the excess risk, one would simply subtract  2 .
Revised: This MSE includes noise on test labels; to compute the excess risk, one would simply subtract  2 .

Original: Here we study the asymptotic behavior of E n as n ! 1 for varying eigenspectra and ridge values.
Revised: Here we study the asymptotic behavior of E n as n ! 1 for varying eigenspectra and ridge values.

Original: The ﬁtting regime of the kernel is then given by
Revised: The ﬁtting regime of the kernel is then given by

Original: E : (a) if lim n !
Revised: E : (a) if lim n !

Original: 1 E =  2 (the Bayes-optimal MSE), 225 then ﬁtting is benign , (b) if lim n !
Revised: 1 E =  2 (the Bayes-optimal MSE), 229 then ﬁtting is benign , (b) if lim n !

Original: 1 E 2 (  2 , 1 ) , then ﬁtting is tempered , and (c) if lim n !
Revised: 1 E 2 (  2 , 1 ) , then ﬁtting is tempered , and (c) if lim n !

Original: 1 E = 1 , then ﬁtting is catastrophic .
Revised: 1 E = 1 , then ﬁtting is catastrophic .

Original: We obtain conditions on the kernel eigenspectrum under which the kernelﬁts according to each of these three regimes.
Revised: We obtain conditions on the kernel eigenspectrum under which the kernelﬁts according to each of these three regimes.

Original: Our proofs rely on several (quite weak) technical assumptions on {  i } i and { v i } i , the most important 229 of which is that the target function does not place weight in zero-eigenvalue modes (i.e. outside thekernel’s RKHS).
Revised: Our proofs rely on several (quite weak) technical assumptions on {  i } i and { v i } i , the most important 233 of which is that the target function does not place weight in zero-eigenvalue modes (i.e. outside thekernel’s RKHS).

Original: We defer enumeration and discussion of these conditions to Appendix A, wherethey are captured by Assumption A.1.
Revised: We defer enumeration and discussion of these conditions to Appendix A, wherethey are captured by Assumption A.1.

Original: Our result is the following:
Revised: Our result is the following:

Original: Theorem 3.1 (KRR trichotomy) .
Revised: Theorem 3.1 (KRR trichotomy) .

Original: For {  i } 1 i =1 and { v } 1 i =1 satisfying Assumption A.1,  2 > 0 , and
Revised: For {  i } 1 i =1 and { v } 1 i =1 satisfying Assumption A.1,  2 > 0 , and

Original: E n given by Equation 4,
Revised: E n given by Equation 4,

Original: (a) If  > 0 , then lim n !
Revised: (a) If  > 0 or  i = i

Revised:  1 log  ↵ i for some ↵ > 1 , then lim n !

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original:  2 .(b)
Revised: 2 .(b)

Original: If  = 0 and  i = i
Revised: If  = 0 and  i = i

Original:  ↵ for some ↵ > 1 , then lim n 
Revised:  ↵ for some ↵ > 1 , then lim n !

Original: [Equation] .(c)
Revised: 1 E n = ↵ 2 .(c)

Original: If  = 0 and  i  i [-+
-] 
Revised: If  = 0 and  i  i [-+
-]+1

Original: i [Equation] i ( i +1)
Revised: i  log i ( i +1)

Original:  log( i +1) for all i , then lim n !
Revised:  log( i +1) for all i , then lim n !

Original: 1 E n = 1 .
Revised: 1 E n = 1 .

Original: We defer the proof to Appendix A.
Revised: We defer the proof to Appendix A.

Original: The proof proceeds by ﬁrst showing that asymptotic MSE isdominated by the noise, and then ﬁnding lim n !
Revised: The proof proceeds by ﬁrst showing that asymptotic MSE isdominated by the noise, and then ﬁnding lim n !

Original: 1 E 0 in each of the three cases.
Revised: 1 E 0 in each of the three cases.

Original: Theorem 3.1 can besummarized as follows: a ridge parameter leads to benign ﬁtting, powerlaw decay of eigenvaluesleads to tempered overﬁtting, and eigenvalue decay at least as fast as i  log i leads to catastrophicoverﬁtting 7 .
Revised: Theorem 3.1 can besummarized as follows: a ridge parameter leads to benign ﬁtting, powerlaw decay of eigenvaluesleads to tempered overﬁtting, and eigenvalue decay at least as fast as i  log i leads to catastrophicoverﬁtting 7 .

Original: We note that the three conditions of Theorem 3.1 are not exhaustive; in particular, spectracan asymptotically decay as powerlaws without matching the exact form in (b), or be O
Revised: We note that the three conditions of Theorem 3.1 are not exhaustive; in particular, spectracan asymptotically decay as powerlaws without matching the exact form in (b), or be O

Original: ( i  log i )without obeying the condition for (c), and there of course exist decay rates intermediate betweenpowerlaws and
Revised: ( i  log i )without obeying the condition for (c), and there of course exist decay rates intermediate betweenpowerlaws and

Original: i  log i .
Revised: i  log i .

Original: Nonetheless, we suggest these three conditions heuristically capture thespectral decay of virtually all kernels used in practice and conjecture that a similar result withexhaustive conditions could be proved.
Revised: Nonetheless, we suggest these three conditions heuristically capture thespectral decay of virtually all kernels used in practice and conjecture that a similar result withexhaustive conditions could be proved.

Original: Theorem 3.1 has several consequences for KR with familiar kernels and the training of inﬁnitewidth networks.
Revised: Theorem 3.1 has several consequences for KR with familiar kernels and the training of inﬁnitewidth networks.

Original: For illustrative purposes, we contrast the Gaussian (RBF) kernel K  ( x 1 , x 2 ) =
Revised: For illustrative purposes, we contrast the Gaussian (RBF) kernel K G ( x 1 , x 2 ) =

Original: e  w  2 | x 1  x 2 | 22 with the Laplace kernel K  ( x 1 , x 2 ) =
Revised: e  w  2 || x 1  x 2 || 22 with the Laplace kernel K L ( x 1 , x 2 ) =

Original: e  w  1 | x 1  x 2 | 1 , where w is a width parameter.
Revised: e  w  1 || x 1  x 2 || 2 , where w is a width parameter.

Original: With data drawn from a d -dimensional manifold, the Gaussian and Laplace kernels have eigenspectrathat decay like  i ⇠
Revised: With data drawn from a d -dimensional manifold, the Gaussian and Laplace kernels have eigenspectrathat decay like  i ⇠

Original: e  i 2 /d (as can be seen by taking a Fourier transform of K  ) and  i ⇠ i  ( d +1)
Revised: e  i 2 /d (as can be seen by taking a Fourier transform of K G ) and  i ⇠ i  ( d +1)

Original: /d ,respectively [Bietti and Bach, 2020].
Revised: /d ,respectively [Bietti and Bach, 2020].

Original: We note that ReLU NTKs, restricted to the hypersphere, have thesame eigendecay as the Laplace kernel [Chen and Xu, 2020, Geifman et al., 2020].
Revised: We note that ReLU NTKs, restricted to the hypersphere, have thesame eigendecay as the Laplace kernel [Chen and Xu, 2020, Geifman et al., 2020].

Original: The consequencesof Theorem 3.1 include the following:
Revised: The consequencesof Theorem 3.1 include the following:

Original: • KR with a ridge parameter and any sufﬁciently-expressive kernel will ﬁt benignly as n ! 1 .•
Revised: • KR with a ridge parameter and any sufﬁciently-expressive kernel will ﬁt benignly as n ! 1 .•

Original: Ridgeless KR with Laplace kernels or ReLU NTKs will exhibit tempered overﬁtting , approaching benignness as dimension grows 8 .
Revised: Ridgeless KR with Laplace kernels or ReLU NTKs will exhibit tempered overﬁtting , approaching benignness as dimension grows 8 .•

Original: • Ridgeless KR with Gaussian kernels will exhibit catastrophic overﬁtting .
Revised: Ridgeless KR with Gaussian kernels will exhibit catastrophic overﬁtting .•

Original: • The asymptotic behavior of KR with non-ReLU NTKs depends on the activation function .Virtually
Revised: The asymptotic behavior of KR with non-ReLU NTKs depends on the activation function .Virtually

Original: any kernel on the d -sphere (including the Gaussian kernel) can be realized as the NTK of a wide network with a specially-designed activation function [Simon et al., 2022],and thus there exist choices of activation function that yield catastrophic overﬁtting.
Revised: any kernel on the d -sphere (including the Gaussian kernel) can be realized as theNTK of a wide network with a proper choice of activation function [Simon et al., 2022]

Original: • The asymptotic behavior of KR with non-ReLU NTKs depends on the activation function.

Original: However, virtually any kernel on the d -sphere (including the Gaussian kernel) can be realizedas the NTK of a wide network with a specially-designed activation function [Simon et al.

Original: ,2022], and thus there exist choices of activation function that yield catastrophic overﬁtting.
Revised: andthus there exist choices of activation function that yield catastrophic overﬁtting.

Original: • Early stopping is known to act as an effective ridge parameter for wide networks [Ali et al.
Revised: • Early stopping is known to act as an effective ridge parameter for wide networks [Ali et al.

Original: ,2019], and thus we should expect that early-stopped wide networks will ﬁt benignly.[Figure]
Revised: ,2019], and thus we should expect that early-stopped wide networks will ﬁt benignly.

Original:  Experiments
Revised:  Experiments

Original: Here we report experimental results conﬁrming our theoretical conclusions for kernel regression, andshowing that DNNs can be either benign or tempered, depending on whether they are stopped early.
Revised: Here we report experimental results conﬁrming our theoretical conclusions for kernel regression, andshowing that DNNs can be either benign or tempered, depending on whether they are stopped early.

Original: We provide complete experimental details in Appendix B. 
Revised: We provide complete experimental details in Appendix B. [Figure] [Figure] [Figure]

Original: 4.1 Experiments on Kernel Regression
Revised: 4.1 Experiments on Kernel Regression

Original: We begin by running KR with (a) Gaussian kernel + ridge parmeter, (b) Laplace kernel + no ridgeparameter, and (c) Gaussian kernel + no ridge parameter, which should exhibit benign, tempered,and catastrophic behavior, respectively, according to Theorem 3.1.
Revised: We begin by running KR with (a) Gaussian kernel + ridge parmeter, (b) Laplace kernel + no ridgeparameter, and (c) Gaussian kernel + no ridge parameter, which should exhibit benign, tempered,and catastrophic behavior, respectively, according to Theorem 3.1.

Original: The results, shown in Figure 3,conﬁrm our theoretical predictions.
Revised: The results, shown in Figure 3,conﬁrm our theoretical predictions.

Original: Figure 3c also shows that increasing input dimension tends toboth improve MSE at ﬁxed n and increase the critical n at which MSE begins to diverge, suggestingthat high dimensionality (which is known to encourage tempered methods to be benign) also acts toprevent catastrophic methods from diverging.
Revised: Figure 3c also shows that increasing input dimension tends toboth improve MSE at ﬁxed n and increase the critical n at which MSE begins to diverge, suggestingthat high dimensionality (which is known to encourage tempered methods to be benign) also acts toprevent catastrophic methods from diverging.

Original: 4.2 Experiments on Deep Neural Networks
Revised: 4.2 Experiments on Deep Neural Networks

Original: We ﬁrst present experimental results for Wide ResNets ( 28 ⇥ 10 ) trained on standard image datasets.
Revised: We ﬁrst present experimental results for Wide ResNets ( 28 ⇥ 10 ) trained on standard image datasets.

Original: We train on MSE loss and measure test classiﬁcation error, and consider interpolation to mean atraining accuracy of 100%.
Revised: We train on MSE loss and measure test classiﬁcation error, and consider interpolation to mean atraining accuracy of 100%.

Original: For each dataset, we train at varying n and with varying label noise ,quantiﬁed by the probability with which each training label is uniformly resampled.
Revised: For each dataset, we train at varying n and with varying label noise ,quantiﬁed by the probability with which each training label is uniformly resampled.

Original: We are primarilyinterested in asymptotic noise-ﬁtting behavior at large n .
Revised: We are primarilyinterested in asymptotic noise-ﬁtting behavior at large n .

Revised: 0.4 Train Label Noise 0 .0 .0 .0 .0 .0 .

Revised: C l e a n

Revised: T e s t

Original: Figure 2 and Figure 4a show noise proﬁles for ResNets trained to interpolation on two-class and10-class CIFAR-10, respectively.
Revised: E rr o r Benign Catastrophic 0/1 SVHN, early stopped 0. 0.4 Train Label Noise 0 .0 .0 .0 .0 .0 .

Revised: C l e a n

Revised: T e s t

Revised: E rr o r

Revised: Benign

Revised: Catastrophic 0/1 SVHN, overﬁt n

Revised: [Figure]

Revised: 9k

Revised: [Figure]

Revised: 18k n

Revised: [Figure]

Revised: 36k

Revised: [Figure]

Revised: Figure 2 and Figure 4a show noise proﬁles for ResNets trained to interpolation on two-class and10-class CIFAR-10, respectively.

Original: In both cases, ﬁtting is tempered, and the noise proﬁle rapidlyapproaches a limiting, fairly linear curve as n ! 1 .
Revised: In both cases, ﬁtting is tempered, and the noise proﬁle rapidlyapproaches a limiting, fairly linear curve as n ! 1 .

Original: Figure 5 shows noise proﬁles for ResNets trained on SVHN, both stopped early (Figure 5a) andtrained to interpolation (Figure 5b).
Revised: Figure 5 shows noise proﬁles for ResNets trained on SVHN, both stopped early (Figure 5a) andtrained to interpolation (Figure 5b).

Original: The early-stopped ResNets approaches benign ﬁtting as ngrows, with low error even at sizable noise levels, while the ResNets trained to interpolation quicklyconverge to a tempered noise proﬁle.
Revised: The early-stopped ResNets approaches benign ﬁtting as ngrows, with low error even at sizable noise levels, while the ResNets trained to interpolation quicklyconverge to a tempered noise proﬁle.

Original: In Figure Ba, we plot MSE vs. training time for an MLP trainedon synthetic noisy data sampled uniformly on the hypersphere, showing this transition betweenregimes directly.
Revised: In Figure 7, we plot MSE vs. training time for an MLP trainedon synthetic noisy data sampled uniformly on the hypersphere, showing this transition betweenregimes directly.

Original: Finally, Figure 4b shows the noise proﬁle for an MLP trained to interpolation on anextremely simple synthetic dataset, where the ground-truth function is constant .
Revised: Finally, Figure 4b shows the noise proﬁle for an MLP trained to interpolation on anextremely simple synthetic dataset, where the ground-truth function is constant .

Original: Even in this simplesetting, interpolating DNNs are not benign— they do not successfully learn the constant function(experimental details in Appendix B.3).
Revised: Even in this simplesetting, interpolating DNNs are not benign— they do not successfully learn the constant function(experimental details in Appendix B.3).

Original:  Conclusion
Revised:  Conclusion

Original: In this paper we study the nature of overﬁtting in learning methods which interpolate their training data.
Revised: In this paper we study the nature of overﬁtting in learning methods which interpolate their training data.

Original: Much of contemporary theory and experimental work categorizes overﬁtting as either catastrophicor benign.
Revised: Much of contemporary theory and experimental work categorizes overﬁtting as either catastrophicor benign.

Original: In contrast, we observe that many natural learning procedures, including DNNs used inpractice, overﬁt in a manner that is neither benign nor catastrophic— but rather in an intermediateregime.
Revised: In contrast, we observe that many natural learning procedures, including DNNs used inpractice, overﬁt in a manner that is neither benign nor catastrophic— but rather in an intermediateregime.

Original: We identify and formally deﬁne this regime, which we call tempered overﬁtting .
Revised: We identify and formally deﬁne this regime, which we call tempered overﬁtting .

Original: We present empirical evidence of learning procedures that exhibit tempered overﬁtting, on bothsynthetic and natural data, using kernel machines and deep neural networks.
Revised: We present empirical evidence of learning procedures that exhibit tempered overﬁtting, on bothsynthetic and natural data, using kernel machines and deep neural networks.

Original: We show temperedover-ﬁtting can be quantiﬁed in terms of noise proﬁles, which measure how asymptotic performancedepends on noise in the train distribution.
Revised: We show temperedover-ﬁtting can be quantiﬁed in terms of noise proﬁles, which measure how asymptotic performancedepends on noise in the train distribution.

Original: For kernel regression, we provide a theoretical result in theform of a trichotomy: conditions on problem parameters which yield each of the three regimes ofoverﬁtting.
Revised: For kernel regression, we provide a theoretical result in theform of a trichotomy: conditions on problem parameters which yield each of the three regimes ofoverﬁtting.

Original: Our work presents an initial study of tempered overﬁtting, and lays the framework for future studyof what we believe is a rich and relevant regime for modern learning.
Revised: Our work presents an initial study of tempered overﬁtting, and lays the framework for future studyof what we believe is a rich and relevant regime for modern learning.

Original: We hope this frameworkinspires further investigation into tempered overﬁtting for more complex models, both theoreticallyand experimentally.
Revised: We hope this frameworkinspires further investigation into tempered overﬁtting for more complex models, both theoreticallyand experimentally.

Revised: For example, it is open to understand which conditions on neural networkarchitecture and data distribution lead to benign, tempered, or catastrophic overﬁtting— an analogueof our “kernel trichotomy” for deep networks.