Original: K NOWLEDGE D ISTILLATION AS S EMIPARAMETRIC I NFERENCE
Revised: K NOWLEDGE D ISTILLATION AS S EMIPARAMETRIC I NFERENCE

Original: A BSTRACT
Revised: A BSTRACT

Original: A popular approach to model compression is to train an inexpensive student model to mimic the class probabilities of a highly accurate but cumbersome teacher model.
Revised: A popular approach to model compression is to train an inexpensive student model to mimic the class probabilities of a highly accurate but cumbersome teacher model.

Original: Surprisingly, this two-step knowledge distillation process often leads to higher accuracy than training the student directly on labeled data.
Revised: Surprisingly, this two-step knowledge distillation process often leads to higher accuracy than training the student directly on labeled data.

Original: To explain and enhance this phenomenon, we cast knowledge distillation as a semiparametric inference problem with the optimal student model as the target, the unknown Bayes class probabilities as nuisance, and the teacher probabilities as a plug-in nuisance estimate.
Revised: To explain and enhance this phenomenon, we cast knowledge distillation as a semiparametric inference problem with the optimal student model as the target, the unknown Bayes class probabilities as nuisance, and the teacher probabilities as a plug-in nuisance estimate.

Original: By adapting modern semiparametric tools, we derive new guarantees for the prediction error of standard distillation and develop two enhancements—cross-ﬁtting and loss correction—to mitigate the impact of teacher overﬁtting and underﬁtting on student performance.
Revised: By adapting modern semiparametric tools, we derive new guarantees for the prediction error of standard distillation and develop two enhancements—cross-ﬁtting and loss correction—to mitigate the impact of teacher overﬁtting and underﬁtting on student performance.

Original: We validate our ﬁndings empirically on both tabular and image data and observe consistent improvements from our knowledge distillation enhancements.
Revised: We validate our ﬁndings empirically on both tabular and image data and observe consistent improvements from our knowledge distillation enhancements.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Knowledge distillation (KD) (Craven & Shavlik, 1996; Breiman & Shang, 1996; Bucila et al., 2006;
Revised: Knowledge distillation (KD) (Craven & Shavlik, 1996; Breiman & Shang, 1996; Bucila et al., 2006;

Original: Li et al., 2014; Ba & Caruana, 2014; Hinton et al., 2015) is a widely used model compression technique that enables the deployment of highly accurate predictive models on devices such as phones, watches, and virtual assistants (Stock et al., 2020).
Revised: Li et al., 2014; Ba & Caruana, 2014; Hinton et al., 2015) is a widely used model compression technique that enables the deployment of highly accurate predictive models on devices such as phones, watches, and virtual assistants (Stock et al., 2020).

Original: KD operates by training a compressed student model to mimic the predicted class probabilities of an expensive, high-quality teacher model.
Revised: KD operates by training a compressed student model to mimic the predicted class probabilities of an expensive, high-quality teacher model.

Original: Remarkably and across a wide variety of domains (Hinton et al., 2015; Sanh et al., 2019; Jiao et al., 2019; Liu et al., 2018; Tan et al., 2018; Fakoor et al., 2020), this two-step process often leads to higher accuracy than training the student directly on the raw labeled dataset.
Revised: Remarkably and across a wide variety of domains (Hinton et al., 2015; Sanh et al., 2019; Jiao et al., 2019; Liu et al., 2018; Tan et al., 2018; Fakoor et al., 2020), this two-step process often leads to higher accuracy than training the student directly on the raw labeled dataset.

Original: While the practice of KD is now well developed, a general theoretical understanding of its successes and failures is still lacking.
Revised: While the practice of KD is now well developed, a general theoretical understanding of its successes and failures is still lacking.

Original: As we detail below, a number of authors have argued that the success of KD lies in the more precise “soft labels” provided by the teacher’s predicted class probabilities.
Revised: As we detail below, a number of authors have argued that the success of KD lies in the more precise “soft labels” provided by the teacher’s predicted class probabilities.

Original: Recently,
Revised: Recently,

Original: Menon et al.
Revised: Menon et al.

Original: (2020) observed that these teacher probabilities can serve as a proxy for the Bayes probabilities (i.e., the true class probabilities) and that the closer the teacher and Bayes probabilities, the better the student’s performance should be.
Revised: (2020) observed that these teacher probabilities can serve as a proxy for the Bayes probabilities (i.e., the true class probabilities) and that the closer the teacher and Bayes probabilities, the better the student’s performance should be.

Original: Building on this observation, we cast KD as a plug-in approach to semiparametric inference (Kosorok, 2007): that is, we view KD as ﬁtting a student model ˆ f in the presence of nuisance (the Bayes probabilities p 0 ) with the teacher’s probabilities ˆ p as a plug-in estimate of p 0 .
Revised: Building on this observation, we cast KD as a plug-in approach to semiparametric inference (Kosorok, 2007): that is, we view KD as ﬁtting a student model ˆ f in the presence of nuisance (the Bayes probabilities p 0 ) with the teacher’s probabilities ˆ p as a plug-in estimate of p 0 .

Original: This insight allows us to adapt modern tools from semiparametric inference to analyze the error of a distilled student in Sec. 3.
Revised: This insight allows us to adapt modern tools from semiparametric inference to analyze the error of a distilled student in Sec. 3.

Original: Our analysis also reveals two distinct failure modes of KD: one due to teacher overﬁtting and data reuse and the other due to teacher underﬁtting from model misspeciﬁcation or insufﬁcient training.
Revised: Our analysis also reveals two distinct failure modes of KD: one due to teacher overﬁtting and data reuse and the other due to teacher underﬁtting from model misspeciﬁcation or insufﬁcient training.

Original: In Sec. 4, we introduce and analyze two complementary KD enhancements that correct for these failures: cross-ﬁtting —a popular technique from semiparametric inference (see, e.g., Chernozhukov et al., 2018)—mitigates teacher overﬁtting through data partitioning while loss correction mitigates teacher underﬁtting by reducing the bias of the plug-in estimate ˆ p .
Revised: In Sec. 4, we introduce and analyze two complementary KD enhancements that correct for these failures: cross-ﬁtting —a popular technique from semiparametric inference (see, e.g., Chernozhukov et al., 2018)—mitigates teacher overﬁtting through data partitioning while loss correction mitigates teacher underﬁtting by reducing the bias of the plug-in estimate ˆ p .

Original: The latter enhancement was inspired by the orthogonal machine learning (Chernozhukov et al., 2018; Foster & Syrgkanis, 2019) approach to semiparametric inference which suggests a particular adjustment for the teacher’s log probabilities.
Revised: The latter enhancement was inspired by the orthogonal machine learning (Chernozhukov et al., 2018; Foster & Syrgkanis, 2019) approach to semiparametric inference which suggests a particular adjustment for the teacher’s log probabilities.

Original: We argue in Sec. 4 that this orthogonal correction minimizes the teacher bias but often at the cost of unacceptably large variance.
Revised: We argue in Sec. 4 that this orthogonal correction minimizes the teacher bias but often at the cost of unacceptably large variance.

Original: Our proposed correction avoids this variance explosion by balancing the bias and variance terms in our generalization bounds.
Revised: Our proposed correction avoids this variance explosion by balancing the bias and variance terms in our generalization bounds.

Original: In Sec. 5, we complement our theoretical analysis with a pair of experiments demonstrating the value of our enhancements on six real classiﬁcation problems.
Revised: In Sec. 5, we complement our theoretical analysis with a pair of experiments demonstrating the value of our enhancements on six real classiﬁcation problems.

Original: On ﬁve real tabular datasets, cross-ﬁtting and loss correction improve student performance by up to 4% AUC over vanilla KD.
Revised: On ﬁve real tabular datasets, cross-ﬁtting and loss correction improve student performance by up to 4% AUC over vanilla KD.

Original: Furthermore, on
Revised: Furthermore, on

Original: CIFAR-10 (Krizhevsky & Hinton, 2009), a benchmark image classiﬁcation dataset, our enhancements improve vanilla KD accuracy by up to 1.5% when the teacher model overﬁts.
Revised: CIFAR-10 (Krizhevsky & Hinton, 2009), a benchmark image classiﬁcation dataset, our enhancements improve vanilla KD accuracy by up to 1.5% when the teacher model overﬁts.

Original: Related work.
Revised: Related work.

Original: Since we cannot review the vast literature on KD in its entirety, we point the interested reader to Gou et al.
Revised: Since we cannot review the vast literature on KD in its entirety, we point the interested reader to Gou et al.

Original: for a recent overview of the ﬁeld.
Revised: for a recent overview of the ﬁeld.

Original: We devote this section to reviewing theoretical advances in the understanding of KD and summarize complementary empirical studies and applications of in the extended literature review in App. A.
Revised: We devote this section to reviewing theoretical advances in the understanding of KD and summarize complementary empirical studies and applications of in the extended literature review in App. A.

Original: A number of papers have argued that the availability of soft class probabilities from the teacher rather than hard labels enables us to improve training of the student model.
Revised: A number of papers have argued that the availability of soft class probabilities from the teacher rather than hard labels enables us to improve training of the student model.

Original: This was hypothesized in Hinton et al.
Revised: This was hypothesized in Hinton et al.

Original: (2015) with empirical justiﬁcation.
Revised: (2015) with empirical justiﬁcation.

Original: Phuong & Lampert (2019) consider the case in which the teacher is a ﬁxed linear classiﬁer and the student is either a linear model or a deep linear network.
Revised: Phuong & Lampert (2019) consider the case in which the teacher is a ﬁxed linear classiﬁer and the student is either a linear model or a deep linear network.

Original: They show that the student can learn the teacher perfectly if the number of training examples exceeds the ambient dimension.
Revised: They show that the student can learn the teacher perfectly if the number of training examples exceeds the ambient dimension.

Original: Vapnik & Izmailov (2015) discuss the setting of learning with privileged information where one has additional information at training time which is not available at test time.
Revised: Vapnik & Izmailov (2015) discuss the setting of learning with privileged information where one has additional information at training time which is not available at test time.

Original: Lopez-Paz et al. (2015) draw a connection between this and KD, arguing that KD is effective because the teacher learns a better representation allowing the student to learn at a faster rate.
Revised: Lopez-Paz et al. (2015) draw a connection between this and KD, arguing that KD is effective because the teacher learns a better representation allowing the student to learn at a faster rate.

Original: They hypothesize that a teacher’s class probabilities enable student improvement by indicating how difﬁcult each point is to classify.
Revised: They hypothesize that a teacher’s class probabilities enable student improvement by indicating how difﬁcult each point is to classify.

Original: Tang et al.
Revised: Tang et al.

Original: argue using empirical evidence that label smoothing and reweighting of training examples using the teacher’s predictions are key to the success of KD.
Revised: argue using empirical evidence that label smoothing and reweighting of training examples using the teacher’s predictions are key to the success of KD.

Original: Mobahi et al.
Revised: Mobahi et al.

Original: (2020) analyzed the case of self-distillation in which the student and teacher function classes are identical.
Revised: (2020) analyzed the case of self-distillation in which the student and teacher function classes are identical.

Original: Focusing on kernel ridge regression models, they proved that self-distillation can act as increased regularization strength.
Revised: Focusing on kernel ridge regression models, they proved that self-distillation can act as increased regularization strength.

Original: Bu et al.
Revised: Bu et al.

Original: (2020) considers more generic model compression in a rate-distortion framework, where the rate is the size of the student model and distortion is the difference in excess risk between the teacher and the student.
Revised: (2020) considers more generic model compression in a rate-distortion framework, where the rate is the size of the student model and distortion is the difference in excess risk between the teacher and the student.

Original: Menon et al.
Revised: Menon et al.

Original: (2020) consider the case of losses such that the population risk is linear in the Bayes class probabilities.
Revised: (2020) consider the case of losses such that the population risk is linear in the Bayes class probabilities.

Original: They consider distilled empirical risk and Bayes distilled empirical risk which are the risk computed using the teacher class probabilities and Bayes class probabilities respectively rather than the observed label.
Revised: They consider distilled empirical risk and Bayes distilled empirical risk which are the risk computed using the teacher class probabilities and Bayes class probabilities respectively rather than the observed label.

Original: They show that the variance of the Bayes distilled empirical risk is lower than the empirical risk.
Revised: They show that the variance of the Bayes distilled empirical risk is lower than the empirical risk.

Original: Then using analysis from Maurer & Pontil (2009); Bennett (1962), they derive the excess risk of the distilled empirical risk as a function of the (cid:96) 2 distance between the teacher’s class probabilities and the Bayes class probabilities.
Revised: Then using analysis from Maurer & Pontil (2009); Bennett (1962), they derive the excess risk of the distilled empirical risk as a function of the (cid:96) 2 distance between the teacher’s class probabilities and the Bayes class probabilities.

Original: We signiﬁcantly depart from Menon et al.
Revised: We signiﬁcantly depart from Menon et al.

Original: (2020) in multiple ways: i) our Thm. 1 allows for the common practice of data re-use, ii) our results cover the standard KD losses SEL and ACE which are non-linear in p 0 , iii) we use localized Rademacher analysis to achieve tight fast rates for standard KD losses, and iv) we use techniques from semiparametric inference to improve upon vanilla KD.
Revised: (2020) in multiple ways: i) our Thm. 1 allows for the common practice of data re-use, ii) our results cover the standard KD losses SEL and ACE which are non-linear in p 0 , iii) we use localized Rademacher analysis to achieve tight fast rates for standard KD losses, and iv) we use techniques from semiparametric inference to improve upon vanilla KD.

Original:  K NOWLEDGE D ISTILLATION B ACKGROUND
Revised:  K NOWLEDGE D ISTILLATION B ACKGROUND

Original: We consider a multiclass classiﬁcation problem with k classes and n training datapoints z i =( x
Revised: We consider a multiclass classiﬁcation problem with k classes and n training datapoints z i =( x

Original: i ,y i ) sampled independently from some distribution P .
Revised: i ,y i ) sampled independently from some distribution P .

Original: Each feature vector x belongs to a set X , each label vector y ∈{ e 1 ,...,e j }⊂{ 0 , 1 } k is a one-hot encoding of the class label, and the conditional probability of observing each label is the Bayes class probability function p 0 ( x )
Revised: Each feature vector x belongs to a set X , each label vector y ∈{ e 1 ,...,e k }⊂{ 0 , 1 } k is a one-hot encoding of the class label, and the conditional probability of observing each label is the Bayes class probability function p 0 ( x )

Original: =
Revised: =

Original: E [ Y | X = x ] .
Revised: E [ Y | X = x ] .

Original: Our aim is to identify a scoring rule f : X → R k that minimizes a prediction loss on average under the distribution P .
Revised: Our aim is to identify a scoring rule f : X → R k that minimizes a prediction loss on average under the distribution P .

Original: Knowledgedistillation.
Revised: Knowledgedistillation.

Original: Knowledgedistillation(KD)isatwo-steptrainingprocesswhereoneﬁrstuses alabeleddatasettotrainateachermodelandthentrainsastudentmodeltopredicttheteacher’spredicted class probabilities.
Revised: Knowledgedistillation(KD)isatwo-steptrainingprocesswhereoneﬁrstuses alabeleddatasettotrainateachermodelandthentrainsastudentmodeltopredicttheteacher’spredicted class probabilities.

Original: Typically the teacher model is larger and more cumbersome, while the student is smaller and more efﬁcient.
Revised: Typically the teacher model is larger and more cumbersome, while the student is smaller and more efﬁcient.

Original: Knowledge distillation was ﬁrst motivated by model compression (Bucila et al., 2006), to ﬁnd compact yet high-performing models to be deployed (such as on mobile devices).
Revised: Knowledge distillation was ﬁrst motivated by model compression (Bucila et al., 2006), to ﬁnd compact yet high-performing models to be deployed (such as on mobile devices).

Original: In training the student to match the teacher’s prediction probability, there are several types of loss functions that are commonly used.
Revised: In training the student to match the teacher’s prediction probability, there are several types of loss functions that are commonly used.

Original: Let ˆ p ( x ) ∈
Revised: Let ˆ p ( x ) ∈

Original: R k be the teacher’s vector of predicted class probabilities, f ( x ) ∈
Revised: R k be the teacher’s vector of predicted class probabilities, f ( x ) ∈

Original: R k be the student model’s output, and [ k ] (cid:44) { 1 , 2 ,...,k } .
Revised: R k be the student model’s output, and [ k ] (cid:44) { 1 , 2 ,...,k } .

Original: The most popular distillation loss functions 1 (cid:96) ( z ; f ( x ) , ˆ p ( x )) include the squared error logit (SEL) loss (Ba & Caruana, 2014) (cid:96)
Revised: The most popular distillation loss functions 1 (cid:96) ( z ; f ( x ) , ˆ p ( x )) include the squared error logit (SEL) loss (Ba & Caruana, 2014) (cid:96)

Original: [Equation]
Revised: [Equation]

Original: ( x
Revised: ( x

Original: [Equation]
Revised: [Equation]

Original: (SEL) and the annealed cross-entropy (ACE) loss (Hinton et al., 2015)
Revised: (SEL) and the annealed cross-entropy (ACE) loss (Hinton et al., 2015)

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: l
Revised: l

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: (cid:17) (ACE) for an inverse temperature β > 0 .
Revised: (cid:17) (ACE) for an inverse temperature β > 0 .

Original: These loss functions measure the divergence between the probabilities predicted by the teacher and the student.
Revised: These loss functions measure the divergence between the probabilities predicted by the teacher and the student.

Original: A student model trained with knowledge distillation often performs better than the same model trained from scratch (Bucila et al., 2006; Hinton et al., 2015).
Revised: A student model trained with knowledge distillation often performs better than the same model trained from scratch (Bucila et al., 2006; Hinton et al., 2015).

Original: In Secs.
Revised: In Secs.

Original: 3 and 4, we will adapt modern tools from semiparametric inference to understand and enhance this phenomenon.
Revised: 3 and 4, we will adapt modern tools from semiparametric inference to understand and enhance this phenomenon.

Original:  D ISTILLATION AS S EMIPARAMETRIC I NFERENCE
Revised:  D ISTILLATION AS S EMIPARAMETRIC I NFERENCE

Original: In semiparametric inference (Kosorok, 2007), one aims to estimate a target parameter or function f 0 , but that estimation depends on an auxiliary nuisance function p 0 that is unknown and not of primary interest.
Revised: In semiparametric inference (Kosorok, 2007), one aims to estimate a target parameter or function f 0 , but that estimation depends on an auxiliary nuisance function p 0 that is unknown and not of primary interest.

Original: We cast the knowledge distillation process as a semiparametric inference problem, by treating the unknown Bayes class probabilities p 0 as nuisance and the teacher’s predicted probabilities as a plug-in estimate of that nuisance.
Revised: We cast the knowledge distillation process as a semiparametric inference problem, by treating the unknown Bayes class probabilities p 0 as nuisance and the teacher’s predicted probabilities as a plug-in estimate of that nuisance.

Original: This perspective allows us bound the generalization of the student in terms of the mean squared error (MSE) between the teacher and the Bayes probabilities.
Revised: This perspective allows us bound the generalization of the student in terms of the mean squared error (MSE) between the teacher and the Bayes probabilities.

Original: In the next section (Sec. 4) we use techniques from semiparametric inference to enhance the performance of the student.
Revised: In the next section (Sec. 4) we use techniques from semiparametric inference to enhance the performance of the student.

Original: The interested reader could consult Tsiatis (2007) for more details on semiparametric inference.
Revised: The interested reader could consult Tsiatis (2007) for more details on semiparametric inference.

Original: Our analysis starts from taking the following perspective on distillation.
Revised: Our analysis starts from taking the following perspective on distillation.

Original: For a given pointwise loss function (cid:96) ( z ; f ( x ) ,p 0 ( x )) , we view the goal of the student as minimizing an oracle population loss over a function class F ,
Revised: For a given pointwise loss function (cid:96) ( z ; f ( x ) ,p 0 ( x )) , we view the goal of the student as minimizing an oracle population loss over a function class F ,

Original: [Equation]
Revised: [Equation]

Original: The main hurdle is that this is objective depends on the unknown Bayes probabilities p 0 .
Revised: The main hurdle is that this is objective depends on the unknown Bayes probabilities p 0 .

Original: We view the teacher’s model ˆ p as an approximate version of p 0 and bound the distillation error of the student as a function of the teacher’s estimation error.
Revised: We view the teacher’s model ˆ p as an approximate version of p 0 and bound the distillation error of the student as a function of the teacher’s estimation error.

Original: Typical semiparametric inference considers cases where f 0 is a ﬁnite dimensional parameter; however recent work of Foster & Syrgkanis (2019) extends this framework to inﬁnite dimensional models f 0 and to develop statistical learning theory with a nuisance component framework.
Revised: Typical semiparametric inference considers cases where f 0 is a ﬁnite dimensional parameter; however recent work of Foster & Syrgkanis (2019) extends this framework to inﬁnite dimensional models f 0 and to develop statistical learning theory with a nuisance component framework.

Original: The distillation problem ﬁts exactly into this setup.
Revised: The distillation problem ﬁts exactly into this setup.

Original: Bounds on vanilla KD
Revised: Bounds on vanilla KD

Original: As a ﬁrst step we derive a vanilla bound on the error of the distilled student model without any further modiﬁcations of the distillation process, i.e., we assume that the student is trained on the same data as the teacher and is trained by running empirical risk minimization (ERM) on the plug-in loss, plugging in the teacher’s model instead of p 0 , i.e., ˆ f =argmin f ∈F L n ( f, ˆ p ) for L n ( f, ˆ p ) (cid:44) E n [ (cid:96) ( Z
Revised: As a ﬁrst step we derive a vanilla bound on the error of the distilled student model without any further modiﬁcations of the distillation process, i.e., we assume that the student is trained on the same data as the teacher and is trained by running empirical risk minimization (ERM) on the plug-in loss, plugging in the teacher’s model instead of p 0 , i.e., ˆ f =argmin f ∈F L n ( f, ˆ p ) for L n ( f, ˆ p ) (cid:44) E n [ (cid:96) ( Z

Original: [Equation]
Revised: [Equation]

Original: f ( X ) , ˆ p ( X ))]
Revised: f ( X ) , ˆ p ( X ))]

Original: (Vanilla KD) where E n [ X ]= 1 n (cid:80) n i =1 X
Revised: (Vanilla KD) where E n [ X ]= 1 n (cid:80) n i =1 X

Original: i denotes the empirical expectation of a random variable.
Revised: i denotes the empirical expectation of a random variable.

Original: Technical deﬁnitions Before presenting our main theorem we introduce some technical notation.
Revised: Technical deﬁnitions Before presenting our main theorem we introduce some technical notation.

Original: For a vector valued function f that takes as input a random variable X , we use the shorthand notation (cid:107) f (cid:107) p,q (cid:44) (cid:107)(cid:107) f
Revised: For a vector valued function f that takes as input a random variable X , we use the shorthand notation (cid:107) f (cid:107) p,q (cid:44) (cid:107)(cid:107) f

Original: ( X ) (cid:107) p (cid:107) L q = E (cid:2) (cid:107) f
Revised: ( X ) (cid:107) p (cid:107) L q = E (cid:2) (cid:107) f

Original: ( X ) (cid:107) qp (cid:3) 1 /q .
Revised: ( X ) (cid:107) qp (cid:3) 1 /q .

Original: Let ∇ φ and ∇ π denote the partial derivatives of (cid:96) ( z ; φ,π ) , with respect to its second and third input correspondingly and ∇ φπ the Jacobian of cross partial derivatives,
Revised: Let ∇ φ and ∇ π denote the partial derivatives of (cid:96) ( z ; φ,π ) , with respect to its second and third input correspondingly and ∇ φπ the Jacobian of cross partial derivatives,

Original: i.e., [ ∇ φπ (cid:96) ( z ; φ,π )] i,j = ∂ 

Revised: i.e., [ ∇ φπ (cid:96) ( z ; φ,π )] i,j = ∂

Original: ∂φ j ∂π i (cid:96) ( z ; φ,π ) .
Revised: ∂φ j ∂π i (cid:96) ( z ; φ,π ) .

Original: Finally, let q f,p ( x )=
Revised: Finally, let q f,p ( x )=

Original: E [ ∇ φπ (cid:96) ( Z ; f ( X ) ,p ( X )) | X = x ] and γ f,p ( x )=
Revised: E [ ∇ φπ (cid:96) ( Z ; f ( X ) ,p ( X )) | X = x ] and γ f,p ( x )=

Original: E U ∼ Unif ([0 , 1])
Revised: E U ∼ Unif ([0 , 1])

Original: [ q f,Up +(1 − U ) p 0 ( x )] .
Revised: [ q f,Up +(1 − U ) p 0 ( x )] .

Original: Critical radius Finally, we need to deﬁne the notion of the critical radius (see, e.g., Wainwright (2019, 14.1.1)) of a function class, which typically provides tight learning rates for statistical learning theory tasks.
Revised: Critical radius Finally, we need to deﬁne the notion of the critical radius (see, e.g., Wainwright (2019, 14.1.1)) of a function class, which typically provides tight learning rates for statistical learning theory tasks.

Original: For any function class F we deﬁne the localized Rademacher complexity as:
Revised: For any function class F we deﬁne the localized Rademacher complexity as:

Original: [Equation]
Revised: [Equation]

Original: where (cid:15) i are i.i.d.
Revised: where (cid:15) i are i.i.d.

Original: random variables taking values equiprobably in {− 1 , 1 } .
Revised: random variables taking values equiprobably in {− 1 , 1 } .

Original: The critical radius of a class F , taking values in [ − H,H ] , is the smallest  solution δ n to the inequality R ( δ ; F ) ≤ δ 2 H .
Revised: The critical radius of a class F , taking values in [ − H,H ] , is the smallest positive solution δ n to the inequality R ( δ ; F ) ≤ δ 2 H .

Original: Theorem 1 (Vanilla KD analysis) .
Revised: Theorem 1 (Vanilla KD analysis) .

Original: Suppose f 0 belongs to a convex set F satisfying the (cid:96) 2 /(cid:96) 4 ratio condition sup f ∈F (cid:107) f − f 0 (cid:107) 2 , 4 / (cid:107)
Revised: Suppose f 0 belongs to a convex set F satisfying the (cid:96) 2 /(cid:96) 4 ratio condition sup f ∈F (cid:107) f − f 0 (cid:107) 2 , 4 / (cid:107)

Original: f − f 0 (cid:107) 2 , 2 ≤ C and that the teacher estimates ˆ p ∈ P from the same dataset used to train the student.
Revised: f − f 0 (cid:107) 2 , 2 ≤ C and that the teacher estimates ˆ p ∈ P from the same dataset used to train the student.

Original: Let δ n,ζ = δ n + c 0 (cid:113) log( c 1 /ζ )
Revised: Let δ n,ζ = δ n + c 0 (cid:113) log( c 1 /ζ )

Original: n for universal constants c 0 ,c 1 and δ n an upper bound on the critical radius of the function class
Revised: n for universal constants c 0 ,c 1 and δ n an upper bound on the critical radius of the function class

Original: [Equation]
Revised: [Equation]

Original: Let µ ( z ) =
Revised: Let µ ( z ) =

Original: sup φ (cid:107)∇ φ (cid:96) ( z ; φ, ˆ p ( x )) (cid:107) 2 , and assume that, for all z ∈ Z and π , the loss (cid:96) ( z ; φ,π ) is σ -strongly convex in φ  and  uniformly bounded in [ − H,H ]for all z,φ,π .
Revised: sup φ (cid:107)∇ φ (cid:96) ( z ; φ, ˆ p ( x )) (cid:107) 2 , and assume that the loss (cid:96) ( z ; φ,π ) is σ -strongly convex in φ for each z and that each g ∈G is uniformly bounded in [ − H,H ] .

Original: Then the Vanilla KD ˆ f satisﬁes
Revised: Then the Vanilla KD ˆ f satisﬁes

Original: [Equation]
Revised: [Equation]

Revised: ˆ

Original: [Equation]
Revised: [Equation]

Revised: =

Revised: [Equation]

Revised: [Equation]

Revised: +

Original:  2 ) with probability at least 1 − ζ.
Revised: [Equation] p [Equation] 22 , 2 ) with probability at least 1 − ζ.

Original: Thm. 1, proved in App.
Revised: Thm. 1, proved in App.

Original: C, shows that vanilla distillation yields an accurate student whenever the teacher generalizes well (i.e., (cid:107) ˆ p − p 0 (cid:107) 2 , 2 is small) and the student and teacher model classes F and P are not too complex.
Revised: C, shows that vanilla distillation yields an accurate student whenever the teacher generalizes well (i.e., (cid:107) ˆ p − p 0 (cid:107) 2 , 2 is small) and the student and teacher model classes F and P are not too complex.

Original: The (cid:96) 2 /(cid:96) 4 ratio requirement can be removed at the expense of replacing (cid:107) µ (cid:107) 4 by (cid:107) µ (cid:107) ∞
Revised: The (cid:96) 2 /(cid:96) 4 ratio requirement can be removed at the expense of replacing (cid:107) µ (cid:107) 4 by (cid:107) µ (cid:107) ∞

Original: =
Revised: =

Original: sup z | µ ( z ) | in the ﬁnal bound.
Revised: sup z | µ ( z ) | in the ﬁnal bound.

Original: Moreover, we highlight that the strong convexity requirement for (cid:96) is satisﬁed by all standard distillation objectives including SEL and ACE, as it is strong convexity with respect to the output of f and not the parameters of f .
Revised: Moreover, we highlight that the strong convexity requirement for (cid:96) is satisﬁed by all standard distillation objectives including SEL and ACE, as it is strong convexity with respect to the output of f and not the parameters of f .

Original: Even this requirement could be removed, but this would yield slow rate bounds of the form: (cid:107) ˆ f − f 0 (cid:107) 22 , 

Revised: Even this requirement could be removed, but this would yield slow rate bounds of the form: (cid:107) ˆ f − f 0 (cid:107) 22 ,

Original: =
Revised: =

Original: O ( δ n,ζ + (cid:107) γ (cid:62) f 0 , ˆ p (ˆ p − p 0 )
Revised: O ( δ n,ζ + (cid:107) γ (cid:62) f 0 , ˆ p (ˆ p − p 0 )

Original: (cid:107) 22 , 2 ) .
Revised: (cid:107) 22 , 2 ) .

Original: Failure modes of vanilla KD Thm.
Revised: Failure modes of vanilla KD Thm.

Original: 1 also hints at two distinct ways in which vanilla distillation could fail.
Revised: 1 also hints at two distinct ways in which vanilla distillation could fail.

Original: First, since the student only learns from the teacher and does not have access to the original labels, we would expect the student to be erroneous when the teacher probabilities are inaccurate due to model misspeciﬁcation, an overly restrictive teacher function class, or insufﬁcient training.
Revised: First, since the student only learns from the teacher and does not have access to the original labels, we would expect the student to be erroneous when the teacher probabilities are inaccurate due to model misspeciﬁcation, an overly restrictive teacher function class, or insufﬁcient training.

Original: Prop. 2, proved in App.
Revised: Prop. 2, proved in App.

Original: D, conﬁrms that, in the worst case, student error suffers from inaccuracy due to this teacher underﬁtting even when both the student and teacher belong to low complexity model classes.
Revised: D, conﬁrms that, in the worst case, student error suffers from inaccuracy due to this teacher underﬁtting even when both the student and teacher belong to low complexity model classes.

Original: Proposition 2 (Impact of teacher underﬁtting on vanilla KD) .
Revised: Proposition 2 (Impact of teacher underﬁtting on vanilla KD) .

Original: There exists a classiﬁcation problem in which the following properties all hold simultaneously with high probability for f 0 =log( p 0 ) :
Revised: There exists a classiﬁcation problem in which the following properties all hold simultaneously with high probability for f 0 =log( p 0 ) :

Original: • The teacher learns ˆ p ( x )= 1 n (1+ λ ) (cid:80) ni
Revised: • The teacher learns ˆ p ( x )= 1 n (1+ λ ) (cid:80) ni

Original: =

Revised: =

Original: y i for all x ∈X via ridge regression with λ =
Revised: y i for all x ∈X via ridge regression with λ =

Original: Θ(1 /n 1 / 4 ) .
Revised: Θ(1 /n 1 / 4 ) .

Original: • Vanilla KD with SEL loss and constant ˆ f satisﬁes (cid:107)
Revised: • Vanilla KD with SEL loss and constant ˆ f satisﬁes (cid:107)

Original: ˆ f − f 0 (cid:107) 22 , 2 ≥ (cid:107) γ (cid:62) f 0 ,p 0 (ˆ p − p 0 ) (cid:107) 22 , 2 = Ω( 1 √ n ) , matching the dependence of the Thm.
Revised: ˆ f − f 0 (cid:107) 22 , 2 ≥ (cid:107) γ (cid:62) f 0 ,p 0 (ˆ p − p 0 ) (cid:107) 22 , 2 = Ω( 1 √ n ) , matching the dependence of the Thm.

Original: 1 upper bound up to a constant factor.
Revised: 1 upper bound up to a constant factor.

Original: • Enhanced KD with SEL loss, ˆ γ ( t ) =
Revised: • Enhanced KD with SEL loss, ˆ γ ( t ) =

Original: diag( 1ˆ p ( t ) ) , and constant ˆ f satisﬁes (cid:107) ˆ f − f 0 (cid:107) 22 , 2 = O ( 1 n ) .
Revised: diag( 1ˆ p ( t ) ) , and constant ˆ f satisﬁes (cid:107) ˆ f − f 0 (cid:107) 22 , 2 = O ( 1 n ) .

Original: Second, the critical radius in Thm. 1 depends on the complexity of the teacher model class P .
Revised: Second, the critical radius in Thm. 1 depends on the complexity of the teacher model class P .

Original: If P has a large critical radius, then the student error bound suffers due to potential teacher overﬁtting even if the teacher generalizes well .
Revised: If P has a large critical radius, then the student error bound suffers due to potential teacher overﬁtting even if the teacher generalizes well .

Original: Prop. 3, proved in App.
Revised: Prop. 3, proved in App.

Original: E, shows that, in the worst case, this teacher overﬁtting penalty is unavoidable and does in fact lead to increased student error.
Revised: E, shows that, in the worst case, this teacher overﬁtting penalty is unavoidable and does in fact lead to increased student error.

Original: This occurs as the student only has access to the teacher’s training set probabilities which, due to overﬁtting, need not reﬂect its test set probabilities.
Revised: This occurs as the student only has access to the teacher’s training set probabilities which, due to overﬁtting, need not reﬂect its test set probabilities.

Original: Proposition 3 (Impact of teacher overﬁtting on vanilla KD) .
Revised: Proposition 3 (Impact of teacher overﬁtting on vanilla KD) .

Original: There exists a classiﬁcation problem in which the following properties all hold simultaneously with high probability for f 0 =log( p 0 ) :
Revised: There exists a classiﬁcation problem in which the following properties all hold simultaneously with high probability for f 0 = E [log( p 0 ( X ))] :

Original: • The critical radius δ n of the teacher-student function class G in Thm.
Revised: • The critical radius δ n of the teacher-student function class G in Thm.

Original: 1 is a non-vanishing constant, due to the complexity of the teacher’s function class.
Revised: 1 is a non-vanishing constant, due to the complexity of the teacher’s function class.

Original: • The Vanilla KD error (cid:107) ˆ f − f 0 (cid:107) 22 , 2 for constant ˆ f with SEL loss is lower bounded by a non-vanishing constant, matching the δ n dependence of the Thm.
Revised: • The Vanilla KD error (cid:107) ˆ f − f 0 (cid:107) 22 , 2 for constant ˆ f with SEL loss is lower bounded by a non-vanishing constant, matching the δ n dependence of the Thm.

Original: 1 upper bound up to a constant factor.
Revised: 1 upper bound up to a constant factor.

Original: • Enhanced KD with SEL loss, ˆ γ ( t ) =0 , and constant ˆ f satisﬁes (cid:107) ˆ fCF − f 0 (cid:107) 22 , 2 =
Revised: • Enhanced KD with SEL loss, ˆ γ ( t ) =0 , and constant ˆ f satisﬁes (cid:107) ˆ f − f 0 (cid:107) 22 , 2 =

Original: O ( n − 4 / (4+ d ) ) .
Revised: O ( n − 4 / (4+ d ) ) .

Original: These examples serve to lower bound student performance in the worst case by the teacher’s critical radius and class probability MSE, matching the upper bounds given in Thm. 1.
Revised: These examples serve to lower bound student performance in the worst case by the teacher’s critical radius and class probability MSE, matching the upper bounds given in Thm. 1.

Original: However, we note that in other better-case scenarios vanilla distillation can perform better than the upper-bounding Thm.would imply.
Revised: However, we note that in other better-case scenarios vanilla distillation can perform better than the upper-bounding Thm.would imply.

Original: In the next section, we adapt and generalize techniques from semiparametric inference to mitigate the effects of teacher overﬁtting and underﬁtting in all cases.
Revised: In the next section, we adapt and generalize techniques from semiparametric inference to mitigate the effects of teacher overﬁtting and underﬁtting in all cases.

Original:  E NHANCING K NOWLEDGE D ISTILLATION
Revised:  E NHANCING K NOWLEDGE D ISTILLATION

Original: To address the two distinct inefﬁciencies of vanilla distillation revealed in Sec. 3, we will adapt and generalize two distinct techniques from semiparametric inference: orthogonal correction and cross-ﬁtting.
Revised: To address the two distinct inefﬁciencies of vanilla distillation revealed in Sec. 3, we will adapt and generalize two distinct techniques from semiparametric inference: orthogonal correction and cross-ﬁtting.

Original: 4. C OMBATING TEACHER UNDERFITTING WITH LOSS CORRECTION
Revised: 4. C OMBATING TEACHER UNDERFITTING WITH LOSS CORRECTION

Original: We can view the plug-in distillation loss (cid:96) ( z ; f ( x ) , ˆ p ( x )) as a zeroth order Taylor approximation to the ideal loss (cid:96) ( z ; f ( x ) ,p 0 ( x )) around ˆ p .
Revised: We can view the plug-in distillation loss (cid:96) ( z ; f ( x ) , ˆ p ( x )) as a zeroth order Taylor approximation to the ideal loss (cid:96) ( z ; f ( x ) ,p 0 ( x )) around ˆ p .

Original: An ideal ﬁrst-order approximation would take the form
Revised: An ideal ﬁrst-order approximation would take the form

Original: [Equation]
Revised: [Equation]

Original: z
Revised: z

Original: [Equation]
Revised: [Equation]

Original: f
Revised: f

Original: [Equation]
Revised: [Equation]

Original: However, its computation also requires knowledge of p 0 .
Revised: However, its computation also requires knowledge of p 0 .

Original: Nevertheless, since p 0 ( x )=
Revised: Nevertheless, since p 0 ( x )=

Original: E [ Y | X = x ] , we can always construct an unbiased estimate of the ideal ﬁrst order term by replacing p 0 ( x ) with y :
Revised: E [ Y | X = x ] , we can always construct an unbiased estimate of the ideal ﬁrst order term by replacing p 0 ( x ) with y :

Original: [Equation]
Revised: [Equation]

Original: ortho (
Revised: ortho (

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: , ˆ p
Revised: , ˆ p

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: For standard distillation base losses like SEL and ACE, the orthogonal loss (1) has an especially simple form, as ∇ π (cid:96) ( z ; f ( x ) , ˆ p ( x )) is linear in f .
Revised: For standard distillation base losses like SEL and ACE, the orthogonal loss (1) has an especially simple form, as ∇ π (cid:96) ( z ; f ( x ) , ˆ p ( x )) is linear in f .

Original: Indeed, this is true more generally for the following class of Bregman divergence losses.
Revised: Indeed, this is true more generally for the following class of Bregman divergence losses.

Original: Deﬁnition 1 (Bregman divergence losses) .
Revised: Deﬁnition 1 (Bregman divergence losses) .

Original: Any Bregman divergence loss function of the form (cid:96) ( z ; f ( x ) ,p ( x ))
Revised: Any Bregman divergence loss function of the form (cid:96) ( z ; f ( x ) ,p ( x ))

Original: (cid:44) Ψ( f ( x ))
Revised: (cid:44) Ψ( f ( x ))

Original: − Ψ( g ( p ( x )))
Revised: − Ψ( g ( p ( x )))

Original: −(cid:104)∇ g Ψ( g ( p ( x ))) ,f ( x ) − g ( p ( x ))
Revised: −(cid:104)∇ g Ψ( g ( p ( x ))) ,f ( x ) − g ( p ( x ))

Original: (cid:105) has (cid:96) ortho ( z ; f ( x ) ,p
Revised: (cid:105) has (cid:96) ortho ( z ; f ( x ) ,p

Original: [Equation]
Revised: [Equation]

Original: +const
Revised: +const

Original: [Equation]
Revised: [Equation]

Original: ) with the second term bilinear in f ( x ) and y − p ( x ) .
Revised: ) with the second term bilinear in f ( x ) and y − p ( x ) .

Original: For the SEL loss, Ψ( s ) =
Revised: For the SEL loss, Ψ( s ) =

Original: 12 (cid:107) s (cid:107) 22 , g ( p ) =
Revised: 12 (cid:107) s (cid:107) 22 , g ( p ) =

Original: log( p ) , and the correction matrix ∇ p g ( p ( x )) (cid:62) ∇ 2 gg Ψ( g ( p ( x )))
Revised: log( p ) , and the correction matrix ∇ p g ( p ( x )) (cid:62) ∇ 2 gg Ψ( g ( p ( x )))

Original: =
Revised: =

Original: diag( 1 p ( x ) ) .
Revised: diag( 1 p ( x ) ) .

Original: Similarly, the ACE loss falls into the class of Bregman divergence losses.
Revised: Similarly, the ACE loss falls into the class of Bregman divergence losses.

Original: We will show that orthogonal correction (1) can signiﬁcantly improve student bias due to teacher underﬁtting; however, for our standard distillation losses (SEL and ACE), the same orthogonal correction term often introduces unreasonably large variance due to division by small probabilities appearing in the correction matrix (see Deﬁnition 1).
Revised: We will show that orthogonal correction (1) can signiﬁcantly improve student bias due to teacher underﬁtting; however, for our standard distillation losses (SEL and ACE), the same orthogonal correction term often introduces unreasonably large variance due to division by small probabilities appearing in the correction matrix (see Deﬁnition 1).

Original: To grant ourselves more ﬂexibility in balancing bias and variance, we propose and analyze a family of γ -corrected losses , parameterized by a matrix valued function
Revised: To grant ourselves more ﬂexibility in balancing bias and variance, we propose and analyze a family of γ -corrected losses , parameterized by a matrix valued function

Original: [Equation]
Revised: [Equation]

Original: →
Revised: →

Original: [Equation]
Revised: [Equation]

Original: ×
Revised: ×

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: to mimic the bilinear structure of Bregman orthogonal losses (2).
Revised: to mimic the bilinear structure of Bregman orthogonal losses (2).

Original: Note that we can always recover the vanilla distillation loss by taking γ ≡ 0 .
Revised: Note that we can always recover the vanilla distillation loss by taking γ ≡ 0 .

Original: We denote the associated population and empirical risks by
Revised: We denote the associated population and empirical risks by

Original: [Equation]
Revised: [Equation]

Original: Observe that at p 0 the correction term is mean-zero and hence L D ( f,p 0 ,γ ) is independent of γ
Revised: Observe that at p 0 the correction term is mean-zero and hence L D ( f,p 0 ,γ ) is independent of γ

Original: [Equation]
Revised: [Equation]

Original: The γ -corrected loss has strong connections to the literature on Neyman orthogonality (Chernozhukov et al., 2018;
Revised: The γ -corrected loss has strong connections to the literature on Neyman orthogonality (Chernozhukov et al., 2018;

Original: Chernozhukov et al., 2016; Nekipelov et al., 2018; Chernozhukov et al., 2018; Foster & Syrgkanis, 2019).
Revised: Chernozhukov et al., 2016; Nekipelov et al., 2018; Chernozhukov et al., 2018; Foster & Syrgkanis, 2019).

Original: In particular, if the function γ is set appropriately, then one can show that the γ -corrected loss function satisﬁes the condition of a Neyman orthogonal loss deﬁned by Foster &
Revised: In particular, if the function γ is set appropriately, then one can show that the γ -corrected loss function satisﬁes the condition of a Neyman orthogonal loss deﬁned by Foster &

Original: Syrgkanis (2019).
Revised: Syrgkanis (2019).

Original: We begin our analysis by showing a general lemma for any estimator ˆ f , which adapts the main theorem of Foster & Syrgkanis (2019) to account for approximate orthogonality; the proof can be found in App.
Revised: We begin our analysis by showing a general lemma for any estimator ˆ f , which adapts the main theorem of Foster & Syrgkanis (2019) to account for approximate orthogonality; the proof can be found in App.

Original: F.
Revised: F.

Original: Lemma 4 (Algorithm-agnostic analysis) .
Revised: Lemma 4 (Algorithm-agnostic analysis) .

Original: Consider any estimation algorithm that produces an estimate ˆ f with small plug-in excess risk, i.e.,
Revised: Consider any estimation algorithm that produces an estimate ˆ f with small plug-in excess risk, i.e.,

Original: [Equation]
Revised: [Equation]

Original: If the loss L D is σ -strongly convex with respect to f and F is a convex set, then
Revised: If the loss L D is σ -strongly convex with respect to f and F is a convex set, then

Original: [Equation]
Revised: [Equation]

Original: If, in addition, sup z,φ,π,i ∈
Revised: If, in addition, sup z,φ,π,i ∈

Original: [ d ]
Revised: [ d ]

Original: [Equation]
Revised: [Equation]

Original: φ i ππ (cid:96) ( z ; φ,π ) (cid:107) op ≤ M , then
Revised: φ i ππ (cid:96) ( z ; φ,π ) (cid:107) op ≤ M , then

Original: [Equation]
Revised: [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: Connection to Neyman orthogonality Remarkably, if we set γ = q f 0 , ˆ p , then the γ -corrected loss is Neyman orthogonal (Foster & Syrgkanis, 2019), and the student MSE bound depends only on the squared MSE of the teacher.
Revised: Connection to Neyman orthogonality Remarkably, if we set γ = q f 0 , ˆ p , then the γ -corrected loss is Neyman orthogonal (Foster & Syrgkanis, 2019), and the student MSE bound depends only on the squared MSE of the teacher.

Original: Moreover, q f 0 , ˆ p is an observable quantity for any Bregman divergence loss (Deﬁnition 1) as q f 0 , ˆ p is independent of f 0 .
Revised: Moreover, q f 0 , ˆ p is an observable quantity for any Bregman divergence loss (Deﬁnition 1) as q f 0 , ˆ p is independent of f 0 .

Original: However, we note that this setting of the γ can lead to larger variance, i.e., the achievable excess risk can be much larger than the excess risk without the correction.
Revised: However, we note that this setting of the γ can lead to larger variance, i.e., the achievable excess risk can be much larger than the excess risk without the correction.

Original: For instance, in the case of the SEL loss q f 0 , ˆ p ( x ) = 1ˆ p ( x ) , which can be excessively large when ˆ p is close to 0 , leading to a large increase in the variance of our loss.
Revised: For instance, in the case of the SEL loss q f 0 , ˆ p ( x ) = 1ˆ p ( x ) , which can be excessively large when ˆ p is close to 0 , leading to a large increase in the variance of our loss.

Original: Thus, in a departure from the standard approach in semiparametric inference, we will be choosing γ in practice to balance bias and variance.
Revised: Thus, in a departure from the standard approach in semiparametric inference, we will be choosing γ in practice to balance bias and variance.

Original: Example instantiation of student’s estimation algorithm
Revised: Example instantiation of student’s estimation algorithm

Original: If we use plug-in empirical risk minimization , i.e., ˆ f = argmin f ∈F L n ( f, ˆ p,γ ) , to estimate f 0 with ˆ p estimated on an independent sample, then the results of Maurer & Pontil (2009) directly imply that as long as the loss function (cid:96) ( z ; φ,π ) is uniformly bounded in [ − H,H ] , then, with probability at least 1 − δ , (cid:15) 
Revised: If we use plug-in empirical risk minimization , i.e., ˆ f = argmin f ∈F L n ( f, ˆ p,γ ) , to estimate f 0 with ˆ p estimated on an independent sample, then the results of Maurer & Pontil (2009) directly imply that as long as the loss function (cid:96) ( z ; φ,π ) is uniformly bounded in [ − H,H ] , then, with probability at least 1 − δ , (cid:15) ( ˆ f, [Equation] p

Original: [Equation]
Revised: [Equation]

Revised: =

Original: O
Revised: O

Original: [Equation]
Revised: [Equation]

Original: where τ ( n )= N ∞ (1 /n, F , 2 n ) and N ∞ ( (cid:15), F ,m ) is the (cid:96) ∞ empirical covering number of function class
Revised: where τ ( n )= N ∞ (1 /n, F , 2 n ) and N ∞ ( (cid:15), F ,m ) is the (cid:96) ∞ empirical covering number of function class

Original: F intheworst-caseoverallrealizationsof m datapointsandatapproximationlevel (cid:15) .
Revised: F intheworst-caseoverallrealizationsof m datapointsandatapproximationlevel (cid:15) .

Original: Thisresulthastwo drawbacks: it is a slow rate result that scales as 1 / √ n for parametric or bounded Vapnik–Chervonenkis (VC)-dimension classes, and it requires the student to be ﬁt on a completely separate dataset from the teacher’s.
Revised: Thisresulthastwo drawbacks: it is a slow rate result that scales as 1 / √ n for parametric or bounded Vapnik–Chervonenkis (VC)-dimension classes, and it requires the student to be ﬁt on a completely separate dataset from the teacher’s.

Original: In the next theorem, we address both of these drawbacks: i) we invoke localized Rademacher complexity analysis to provide a fast rate result which would be of the order of 1 /n for VC or parametric function classes, and ii) we use a more sophisticated data-partitioning technique called cross-ﬁtting, which allows the student to be trained using all of the available teacher data.
Revised: In the next theorem, we address both of these drawbacks: i) we invoke localized Rademacher complexity analysis to provide a fast rate result which would be of the order of 1 /n for VC or parametric function classes, and ii) we use a more sophisticated data-partitioning technique called cross-ﬁtting, which allows the student to be trained using all of the available teacher data.

Original: 4. C OMBATING TEACHER OVERFITTING WITH CROSS - FITTING
Revised: 4. C OMBATING TEACHER OVERFITTING WITH CROSS - FITTING

Original: We now describe a more sophisticated version of data partitioning to make use of all data points in our student estimation, while at the same time not suffering from the sample complexity of the teacher’s function space.
Revised: We now describe a more sophisticated version of data partitioning to make use of all data points in our student estimation, while at the same time not suffering from the sample complexity of the teacher’s function space.

Original: This approach is referred to as cross-ﬁtting (CF) in the semiparametric inference literature (see, e.g., Chernozhukov et al.
Revised: This approach is referred to as cross-ﬁtting (CF) in the semiparametric inference literature (see, e.g., Chernozhukov et al.

Original: Partition the dataset into B equally sized folds P 1 ,...,P B . 2.
Revised: Partition the dataset into B equally sized folds P 1 ,...,P B . 2.

Original: For each fold t ∈ [ B ] estimate ˆ p ( t ) and ˆ γ ( t ) using all the out-of-fold data points.
Revised: For each fold t ∈ [ B ] estimate ˆ p ( t ) and ˆ γ ( t ) using all the out-of-fold data points.

Original: Estimate ˆ f by minimizing the empirical loss:
Revised: Estimate ˆ f by minimizing the empirical loss:

Original: [Equation]
Revised: [Equation]

Original: In other words, the nuisance estimates (ˆ γ ( t ) , ˆ p ( t ) ) that are evaluated on the data points in fold t when ﬁtting the student in step 3, are estimated only using data points outside of P t .
Revised: In other words, the nuisance estimates (ˆ γ ( t ) , ˆ p ( t ) ) that are evaluated on the data points in fold t when ﬁtting the student in step 3, are estimated only using data points outside of P t .

Original: Theorem 5 (Enhanced KD analysis) .
Revised: Theorem 5 (Enhanced KD analysis) .

Original: Suppose f 0 belongs to a convex set F .
Revised: Suppose f 0 belongs to a convex set F .

Original: Let δ n,ζ = δ n + c 0 (cid:113) log( c 1 /ζ )
Revised: Let [Equation]

Original: n for universal constants c 0 ,c 1 and δ n an upper bound on the critical radius of the class
Revised: for universal constants c 0 , c 1 and δ n/B an upper bound on the critical radius of the class

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: 1] } for each t ∈ [ B ] .
Revised: ] } for each t ∈ [ B ] .

Original: Assume that, for all z ∈Z , φ , and π , the loss (cid:96) ( z ; φ,π ) is uniformly bounded in [ − H,H ] and σ -strongly convex with respect to φ , and let µ ( z ) =
Revised: Let µ ( z ) =

Original: sup f ∈F ,t ∈
Revised: sup f ∈F ,t ∈

Original: [ B ] (cid:13)(cid:13) ∇ φ (cid:96)  ( z ; f ( X ) , ˆ p ( t ) ( x ))
Revised: [ B ] (cid:13)(cid:13) ∇ φ (cid:96) ˆ γ ( t ) ( z ; f ( X ) , ˆ p ( t ) ( x ))

Original: (cid:13)(cid:13) 2  .
Revised: (cid:13)(cid:13) 2 , and assume that, with probability 1 for each t ∈ [ B ] , the loss (cid:96) ˆ γ ( t ) ( z ; φ, ˆ p ( t ) ( x )) is σ -strongly convex in φ for each z and each g ∈ G (ˆ p ( t ) , ˆ γ ( t ) ) is uniformly bounded in [ − H,H ] .

Original: Moreover, suppose that the function class F satisﬁes the (cid:96) 2 /(cid:96) 4 ratio condition: sup f ∈F 
Revised: Moreover, suppose that the function class F satisﬁes the (cid:96) 2 /(cid:96) 4 ratio condition: sup f ∈F (cid:107) f − f 0 (cid:107) 2 , 4 (cid:107) f − f 0 (cid:107) 2 , 2 ≤ C. If ˆ f is the output of Enhanced KD, then, with probability at least 1 − ζ ,

Original: [Equation]
Revised: [Equation]

Original: If ˆ f is the output of Enhanced KD, then, with probability at least 1 − ζ ,

Original: [Equation]
Revised: [Equation]

Original: The proof is found in App.
Revised: The proof is found in App.

Original: G. Observe that, unlike Thm. 1, the function classes G (ˆ p ( t ) , ˆ γ ( t ) ) in the
Revised: G. Observe that, unlike Thm. 1, the function classes G (ˆ p ( t ) , ˆ γ ( t ) ) in the

Original: Thm. 5 do not vary the teacher’s model over P but rather evaluate p at the speciﬁc out-of-fold estimates ˆ p ( t ) and only vary f ∈F .
Revised: Thm. 5 do not vary the teacher’s model over P but rather evaluate p at the speciﬁc out-of-fold estimates ˆ p ( t ) and only vary f ∈F .

Original: Since in practice the teacher’s model can be quite complex, removing this dependence on the sample complexity of the teacher’s function space can bring immense improvement with the critical radius of G (ˆ p ( t ) , ˆ γ ( t ) ) signiﬁcantly smaller than  G .
Revised: Since in practice the teacher’s model can be quite complex, removing this dependence on the sample complexity of the teacher’s function space can bring immense improvement with the critical radius of G (ˆ p ( t ) , ˆ γ ( t ) ) signiﬁcantly smaller than that of G from Thm. 1.

Original: For instance, suppose that the loss function (cid:96) ˆ γ ( t ) ( z ; f, ˆ p ( t ) ) is L -Lipschitz with respect to f and that F is a VC-subgraph class with VC dimension d F .
Revised: For instance, suppose that the loss function (cid:96) ˆ γ ( t ) ( z ; f, ˆ p ( t ) ) is L -Lipschitz with respect to f and that F is a VC-subgraph class with VC dimension d F .

Original: Then the critical radius of the function class G (ˆ p ( t ) , ˆ γ ( t ) ) is of order (cid:112)
Revised: Then the critical radius of the function class G (ˆ p ( t ) , ˆ γ ( t ) ) is of order (cid:112)

Original: d F log( n ) /n for any choice of (ˆ p ( t ) , ˆ γ ( t ) )
Revised: d F log( n ) /n for any choice of (ˆ p ( t ) , ˆ γ ( t ) )

Original: (see, e.g., Foster & Syrgkanis, 2019, Sec. 4.2).
Revised: (see, e.g., Foster & Syrgkanis, 2019, Sec. 4.2).

Original: However, under the same conditions, the critical radius of the teacher-student function class G in
Revised: However, under the same conditions, the critical radius of the teacher-student function class G in

Original: Thm. 1 will still depend on the teacher’s function space.
Revised: Thm. 1 will still depend on the teacher’s function space.

Original: If P is also a VC-subgraph class with VC dimension d P (cid:29) d F , then the critical radius of G will be of the much larger order (cid:112) d P log( n ) /n .
Revised: If P is also a VC-subgraph class with VC dimension d P (cid:29) d F , then the critical radius of G will be of the much larger order (cid:112) d P log( n ) /n .

Original: We can also see in the bound of Thm.
Revised: We can also see in the bound of Thm.

Original: 5 the interplay between bias and variance introduced by γ .
Revised: 5 the interplay between bias and variance introduced by γ .

Original: In particular, the part of the bound that depends on ˆ γ ( t ) can be further simpliﬁed as(cid:113)
Revised: In particular, the part of the bound that depends on ˆ γ ( t ) can be further simpliﬁed as

Original: [Equation]
Revised: [Equation]

Original: where the terms respectively encode the increase in variance and decrease in bias from employing loss correction.
Revised: where the terms respectively encode the increase in variance and decrease in bias from employing loss correction.

Original: Notably, Thm. 5 implies that CF without γ -correction (i.e., ˆ γ ( t ) ( x )=0 ) is sufﬁcient to reduce student error due to teacher overﬁtting but may still be susceptible to excessive student error due to teacher underﬁtting.
Revised: Notably, Thm. 5 implies that CF without γ -correction (i.e., ˆ γ ( t ) ( x )=0 ) is sufﬁcient to reduce student error due to teacher overﬁtting but may still be susceptible to excessive student error due to teacher underﬁtting.

Original: These qualitative predictions accord with our experimental observations in Sec.
Revised: These qualitative predictions accord with our experimental observations in Sec.

Original: 5 and Fig.
Revised: 5 and Fig.

Original: 4. B IASED STOCHASTIC GRADIENT DESCENT ANALYSIS
Revised: 4. B IASED STOCHASTIC GRADIENT DESCENT ANALYSIS

Original: When the set of candidate prediction rules f θ is parameterized by a vector θ ∈ R d , we may alternatively ﬁt θ via stochastic gradient descent (SGD) (Robbins & Monro, 1951; Bottou & Bousquet, 2008) on the γ -corrected objective L D ( f θ , ˆ p, ˆ γ ) .
Revised: When the set of candidate prediction rules f θ is parameterized by a vector θ ∈ R d , we may alternatively ﬁt θ via stochastic gradient descent (SGD) (Robbins & Monro, 1951; Bottou & Bousquet, 2008) on the γ -corrected objective L D ( f θ , ˆ p, ˆ γ ) .

Original: With a minibatch size of 1 and a starting point θ 0 , the parameter updates take the form
Revised: With a minibatch size of 1 and a starting point θ 0 , the parameter updates take the form

Original: [Equation]
Revised: [Equation]

Revised: [ n ] .

Original: Ideally, these updates would converge to a minimizer of the ideal risk L ( θ ; p 0 ) =
Revised: Ideally, these updates would converge to a minimizer of the ideal risk L ( θ ; p 0 ) =

Original: L D ( f θ ,p 0 ) .
Revised: L D ( f θ ,p 0 ) .

Original: Our next result shows that, if the teacher ˆ p is independent of ( W t ) t ∈ [ n ] , then the SGD updates (4) have excess ideal risk governed by a bias term ζ (ˆ γ ) and a variance term σ (ˆ γ ) 2 /n .
Revised: Our next result shows that, if the teacher ˆ p is independent of ( W t ) t ∈ [ n ] , then the SGD updates (4) have excess ideal risk governed by a bias term ζ (ˆ γ ) and a variance term σ (ˆ γ ) 2 /n .

Original: Here, σ 20 ( θ ) represents the baseline stochastic gradient variance that would be incurred if SGD were run directly on the ideal risk L ( θ ; p 0 ) rather than our surrogate risk.
Revised: Here, σ 20 ( θ ) represents the baseline stochastic gradient variance that would be incurred if SGD were run directly on the ideal risk L ( θ ; p 0 ) rather than our surrogate risk.

Original: Our proof in App.
Revised: Our proof in App.

Original: H builds upon the biased SGD bounds of Ajalloeian & Stich (2020).
Revised: H builds upon the biased SGD bounds of Ajalloeian & Stich (2020).

Original: Theorem 6 (Biased SGD analysis) .
Revised: Theorem 6 (Biased SGD analysis) .

Original: Suppose that the loss L ( θ ; p 0 ) is λ -strongly smooth in θ .
Revised: Suppose that the loss L ( θ ; p 0 ) is λ -strongly smooth in θ .

Original: Deﬁne the bias and root-variance parameters
Revised: Deﬁne the bias and root-variance parameters

Original: [Equation]
Revised: [Equation]

Original: for σ 20 ( θ ) [Equation]
Revised: for σ 20 ( θ ) (cid:44) (cid:80)

Original: W
Revised: i

Original: [Equation]
Revised: [Equation]

Original: X

Original: [Equation]
Revised: [Equation]

Original: p
Revised: [ ∇

Original: [Equation]
Revised: [Equation]

Original:  X ))] the unbiased SGD variance.
Revised: ( W ; f θ ( X ) , p 0 ( X ))] the unbiased SGD variance.

Original: If
Revised: If

Original: F[-
-]
Revised: F[-
-]

Revised: [Equation]

Original: =
Revised: =

Original: L ( θ 0 ; p 0 ) −min θ
Revised: L ( θ 0 ; p 0 ) −

Original: [Equation]
Revised: [Equation]

Original: R d L ( θ ; p 0 ) , thentheiterates { θ t } nt=
Revised: R d L ( θ ; p 0 ) , thentheiterates { θ t } nt

Original: [Equation]

Original:  ofthelosscorrectedSGDalgorithmsatisfymin
Revised: =1 ofthelosscorrectedSGDalgorithmsatisfy

Original: [Equation]
Revised: [Equation]

Original: If, in addition, L ( θ ; p 0 ) is µ -strongly convex in θ , then the iterates satisfy
Revised: If, in addition, L ( θ ; p 0 ) is µ -strongly convex in θ , then the iterates satisfy

Original: [Equation]
Revised: [Equation]

Original: Similar to Thm.
Revised: Similar to Thm.

Original: 5, the bound in Thm. 6, portrays the interplay of bias and variance as ˆ γ ranges from 0 to q f θ , ˆ p (recall that q f θ , ˆ p is independent of f θ for any Bregman loss).
Revised: 5, the bound in Thm. 6, portrays the interplay of bias and variance as ˆ γ ranges from 0 to q f θ , ˆ p (recall that q f θ , ˆ p is independent of f θ for any Bregman loss).

Original: In particular, the part of the bound for strongly convex losses that depends on ˆ γ can be further simpliﬁed to:
Revised: In particular, the part of the bound for strongly convex losses that depends on ˆ γ can be further simpliﬁed to:

Original: [Equation]
Revised: [Equation]

Original: This has a very intuitive form: the ﬁrst term is the impact of ˆ γ ( X ) on the variance, which is also related to the square of the noise of y , divided by the standard error scaling.
Revised: This has a very intuitive form: the ﬁrst term is the impact of ˆ γ ( X ) on the variance, which is also related to the square of the noise of y , divided by the standard error scaling.

Original: The second controls how ˆ γ improves the bias introduced by the error in the teacher’s ˆ p .
Revised: The second controls how ˆ γ improves the bias introduced by the error in the teacher’s ˆ p .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We complement our theoretical analysis with a pair of experiments demonstrating the practical beneﬁts of cross-ﬁtting and loss correction on six real-world classiﬁcation tasks.
Revised: We complement our theoretical analysis with a pair of experiments demonstrating the practical beneﬁts of cross-ﬁtting and loss correction on six real-world classiﬁcation tasks.

Original: Throughout, we use the SEL loss.
Revised: Throughout, we use the SEL loss and report mean performance ± 1 standard error across 5 independent runs.

Original: Code to replicate all experiments can be found at https://github.com/microsoft/semiparametric-distillation , and supplementary experimental details and results can be found in App.
Revised: Code to replicate all experiments can be found at https://github.com/microsoft/semiparametric-distillation , and supplementary experimental details and results can be found in App.

Original: I.
Revised: I.

Original: Selecting the loss correction matrix ˆ γ Motivated by the analyses in Sec. 4, for each training point ( x,y ) , we will select our correction matrix ˆ γ ( x ) to balance bias and variance by minimizing a pointwise upper bound on the loss correction error (5) (ideally with a closed-form solution to avoid excessive computational overhead).
Revised: Selecting the loss correction matrix ˆ γ Motivated by the analyses in Sec. 4, for each training point ( x,y ) , we will select our correction matrix ˆ γ ( x ) to balance bias and variance by minimizing a pointwise upper bound on the loss correction error (5) (ideally with a closed-form solution to avoid excessive computational overhead).

Original: 3 To eliminate dependence on the unobserved p 0 , we observe that the bias term (cid:107) ( γ f θ , ˆ p ( x ) − [Equation] γ [Equation]
Revised: 3 To eliminate dependence on the unobserved p 0 , we observe that the bias term (cid:107) ( γ f θ , ˆ p ( x ) − ˆ γ ( x )) (cid:62) (ˆ p ( x ) − p 0 ( x ))

Original: x ))

Original: [Equation]

Original: p

Original: [Equation] [Equation] [Equation] 22 ={+(cid:107) 
+}
Revised: {+(cid:107) 
+}

Original: O (

Original: [Equation]

Original: q

Original: [Equation] θ , [Equation] p ( x ) − ˆ γ ( x ) (cid:107) 2op ) up to additive terms independent of ˆ γ .
Revised: = O ( (cid:107) q f θ , ˆ p ( x ) − ˆ γ ( x ) (cid:107) 2op ) up to additive terms independent of ˆ γ .

Original: We introduce a tunable hyperparameter α> 0 to trade off between this bias bound and the variance term in (5) and select ˆ γ ( x )=
Revised: We introduce a tunable hyperparameter α> 0 to trade off between this bias bound and the variance term in (5) and select ˆ γ ( x )=

Original: diag( v ( x )) to minimize:
Revised: diag( v ( x )) to minimize:

Original: [Equation]
Revised: [Equation]

Original: Since the conditional expectation involves the unknown quantity p 0 , we approximate E [ (cid:107)  y − ˆ p ( x ) (cid:107) 22 | x ] by its sample (cid:107)  y − ˆ p ( x ) (cid:107) 22 . 

Revised: Since the conditional expectation involves the unknown quantity p 0 , we estimate E [ (cid:107) v ( x )( y − ˆ p ( x )) (cid:107) 22 | x ] with its sample (cid:107) v ( x )( y − ˆ p ( x )) (cid:107) 22 .

Original: This objective is quadratic in v ( x ) and thus has a closed-form solution.
Revised: This objective is quadratic in v ( x ) and thus has a closed-form solution.

Original: Given ˆ γ ( x ) , the student’s loss-corrected objective is equivalent to a square loss with labels log( p ( x ))
Revised: Given ˆ γ ( x ) , the student’s loss-corrected objective is equivalent to a square loss with labels log( p ( x ))

Original: +ˆ γ ( x ) (cid:62) ( y − p ( x )) .
Revised: +ˆ γ ( x ) (cid:62) ( y − p ( x )) .

Original: Tabular data.
Revised: Tabular data.

Original: We ﬁrst validate our KD enhancements on ﬁve real-world tabular datasets—FICO (FIC),
Revised: We ﬁrst validate our KD enhancements on ﬁve real-world tabular datasets—FICO (FIC),

Original: StumbleUpon (Eve; Liu et al., 2017), and Adult, Higgs, and MAGIC from Dheeru & Karra Taniskidou (2017)—with random forest (Breiman, 2001) students and teachers.
Revised: StumbleUpon (Eve; Liu et al., 2017), and Adult, Higgs, and MAGIC from Dheeru & Karra Taniskidou (2017)—with random forest (Breiman, 2001) students and teachers.

Original: In Fig.
Revised: In Fig.

Original: 1a, we examine the impact of varying student model capacity for a ﬁxed high-capacity teacher with 500 trees on FICO.
Revised: 1a, we examine the impact of varying student model capacity for a ﬁxed high-capacity teacher with 500 trees on FICO.

Original: This setting lends itself to teacher overﬁtting, and we ﬁnd that cross-ﬁtting consistently improves upon vanilla KD by up to 4 AUC percentage points.
Revised: This setting lends itself to teacher overﬁtting, and we ﬁnd that cross-ﬁtting consistently improves upon vanilla KD by up to 4 AUC percentage points.

Original: In Fig.
Revised: In Fig.

Original: 1b we explore the impact of teacher underﬁtting by limiting the teacher’s maximum tree depth on Adult.
Revised: 1b we explore the impact of teacher underﬁtting by limiting the teacher’s maximum tree depth on Adult.

Original: Here we observe consistent gains from loss correction with student performance exceeding even that of the teacher for smaller maximum tree depths.
Revised: Here we observe consistent gains from loss correction with student performance exceeding even that of the teacher for smaller maximum tree depths.

Original: Analogous results for the remaining datasets can be found in App.
Revised: Analogous results for the remaining datasets can be found in App.

Original: I.1.
Revised: I.1.

Original: [Figure]
Revised: [Figure]

Original: ) FICO dataset, when teacher overﬁts
Revised: dataset

Original: [Figure]
Revised: [Figure]

Original: Image data.
Revised: Image data.

Original: We next validate our KD enhancements on the image classiﬁcation dataset CIFAR-
Revised: We next validate our KD enhancements on the image classiﬁcation dataset CIFAR-

Original: (Krizhevsky & Hinton, 2009).
Revised: (Krizhevsky & Hinton, 2009).

Original: We pair a residual network (ResNet-8) student with teacher networks of varying depths (ResNet-14/20/32/44/56)
Revised: We pair a residual network (ResNet-8) student with teacher networks of varying depths (ResNet-14/20/32/44/56)

Original: (He et al., 2016).
Revised: (He et al., 2016).

Original: It has been observed that larger and deeper teachers need not yield better students, as the teacher might overﬁt to the training set (Cho & Hariharan, 2019; M ¨ uller et al., 2019).
Revised: It has been observed that larger and deeper teachers need not yield better students, as the teacher might overﬁt to the training set (Cho & Hariharan, 2019; M¨uller et al., 2019).

Original: To induce this overﬁtting, we turn off data augmentation (random horizontal ﬂipping and cropping).
Revised: To induce this overﬁtting, we turn off data augmentation (random horizontal ﬂipping and cropping).

Original: We compare students trained with vanilla KD and enhanced KD with and without loss correction in Fig.
Revised: We compare students trained with Vanilla KD and Enhanced KD with and without loss correction in Fig.

Original: We ﬁnd that cross-ﬁtting consistently reduces the effect of teacher overﬁtting with largest impact realized for the deepest models.
Revised: We ﬁnd that cross-ﬁtting consistently reduces the effect of teacher overﬁtting with largest impact realized for the deepest models.

Original: This effect is most evident in the  test loss, where the vanilla KD student incurs signiﬁcantly larger loss than the cross-ﬁtted student.
Revised: This effect is most evident in the cross-entropy test loss, where the Vanilla KD student incurs signiﬁcantly larger loss than the cross-ﬁtted student.

Original: For both accuracy and test loss, employing loss correction on top of cross-ﬁtting provides an additional small performance boost.
Revised: For both accuracy and test loss, employing loss correction on top of cross-ﬁtting provides an additional small performance boost.

Original: Ablation on the effect of the loss correction hyperparameter α .
Revised: Effect of the loss correction hyperparameter α .

Original: Our hyperparameter α controls the tradeoff between bias and variance in loss correction.
Revised: Our hyperparameter α controls the tradeoff between bias and variance in loss correction.

Original: When α is very small, the objective is close to the vanilla KD objective.
Revised: When α is very small, the objective is close to the vanilla KD objective.

Original: When α is large, the objective is closer tothe Neyman-orthogonal loss.
Revised: When α is large, the objective is closer to the Neyman-orthogonal loss.

Original: In Figure 3, we show the effect of varying α , with ResNet-8 as the student and ResNet-20 as the teacher, on the CIFAR-10 dataset.
Revised: In Figure 3, we show the effect of varying α , with ResNet-8 as the student and ResNet-20 as the teacher, on the CIFARdataset.

Original: Large values of α lead to high variance and thus lower test accuracy.
Revised: Large values of α lead to high variance and thus lower test accuracy.

Original: Intermediate values of α improves onboth thevanillaKD objective, which correspondsto α =0 andon theorthogonal objective  ( α = ∞ ).
Revised: Intermediate values of α improves on both the Vanilla KD objective, which corresponds to α =0 and on the orthogonal objective An alternative estimate that performs slightly worse is (cid:107) v ( x )ˆ p ( x )(1 − ˆ p ( x )) (cid:107) 22 .

Original:  The test accuracy drops sharply beyond some threshold of α as the variance becomes due to the terms 
Revised: [Figure] ( α = ∞ ). The test accuracy drops sharply beyond some threshold of α as the variance becomes too high (due to the terms q ˆ p ( x )=diag (cid:16)

Original: [Figure]
Revised: [Figure]

Revised: x

Original: [Figure] 
Revised: [Figure] [Figure] x [Figure]

Revised: ), causing training to become unstable.

Revised: [Figure]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We developed a new analysis of knowledge distillation under the lens of semiparametric inference.
Revised: We developed a new analysis of knowledge distillation under the lens of semiparametric inference.

Original: By framing the KD process as learning with plug-in estimation in the presence of nuisance, we obtained new generalization bounds for distillation and new lower bounds highlighting the susceptibility of KD to teacher overﬁtting and underﬁtting.
Revised: By framing the KD process as learning with plug-in estimation in the presence of nuisance, we obtained new generalization bounds for distillation and new lower bounds highlighting the susceptibility of KD to teacher overﬁtting and underﬁtting.

Original: To address these failure modes, we introduced two complementary KD enhancements—cross-ﬁtting and loss correction—which improve student performance both in theory and in practice.
Revised: To address these failure modes, we introduced two complementary KD enhancements—cross-ﬁtting and loss correction—which improve student performance both in theory and in practice.

Original: Past work has shown that augmenting the student training set with synthetic data from a generative model (e [Figure] g., a generative adversarial network (Liu et al., 2018) or MUNGE (Bucila et al., 2006)) often leads to improved student performance.
Revised: Past work has shown that augmenting the student training set with synthetic data from a generative model (e.g., a generative adversarial network (Liu et al., 2018) or MUNGE (Bucila et al., 2006)) often leads to improved student performance.

Original: A natural next step is to prove an analogue of Thm. 5 for synthetic augmentation to understand when this strategy successfully mitigates the impact of teacher overﬁtting.
Revised: A natural next step is to prove an analogue of Thm. 5 for synthetic augmentation to understand when this strategy successfully mitigates the impact of teacher overﬁtting.

Original: In addition, two tantalizing open questions are, ﬁrst, whether other techniques from semiparametric inference, such as targeted maximum likelihood (Van Der Laan & Rubin, 2006), can be used to improve KD performance and, second, whether a semiparametric perspective can explain the surprising success of self-distillation (Furlanello et al., 2018) and
Revised: In addition, two tantalizing open questions are, ﬁrst, whether other techniques from semiparametric inference, such as targeted maximum likelihood (Van Der Laan & Rubin, 2006), can be used to improve KD performance and, second, whether a semiparametric perspective can explain the surprising success of self-distillation (Furlanello et al., 2018) and

Original: noisy student training (Xie et al., 2020) through which students routinely outperform their teachers.
Revised: noisy student training (Xie et al., 2020) through which students routinely outperform their teachers.

Original: A E XTENDED LITERATURE REVIEW
Revised: A E XTENDED LITERATURE REVIEW

Original: We point the interested reader to Gou et al.
Revised: We point the interested reader to Gou et al.

Original: (2020) for a sweeping survey of the many developments in knowledge distillation over the past half decade.
Revised: (2020) for a sweeping survey of the many developments in knowledge distillation over the past half decade.

Original: In addition to the references discussing theoretical aspects of knowledge distillation provided in Sec. 1, we highlight here a number of empirical investigations of why distillation works.
Revised: In addition to the references discussing theoretical aspects of knowledge distillation provided in Sec. 1, we highlight here a number of empirical investigations of why distillation works.

Original: Cho & Hariharan (2019) show that larger teacher models do not necessarily improve the performance of student models as parsimonious student models are not able to mimic the teacher model.
Revised: Cho & Hariharan (2019) show that larger teacher models do not necessarily improve the performance of student models as parsimonious student models are not able to mimic the teacher model.

Original: They suggest early stopping in training large teacher neural networks as means of regularizing.
Revised: They suggest early stopping in training large teacher neural networks as means of regularizing.

Original: Cheng et al.
Revised: Cheng et al.

Original: (2020) show that when applied to image data, distillation allows the student neural net to learn multiple visual concepts simultaneously, while in knowledge distillation neural networks learn concepts sequentially.
Revised: (2020) demonstrate that when applied to image data, distillation allows the student neural net to learn multiple visual concepts simultaneously, while, when learning from raw data, neural networks learn concepts sequentially.

Original: Knowledge distillation has also been used for adversarial attacks (Papernot et al., 2016b; Ross & Doshi-Velez, 2017; Gil et al., 2019; Goldblum et al., 2020), data security (Papernot et al., 2016a;
Revised: Knowledge distillation has also been used for adversarial attacks (Papernot et al., 2016b; Ross & Doshi-Velez, 2017; Gil et al., 2019; Goldblum et al., 2020), data security (Papernot et al., 2016a;

Original: Lopes et al., 2017; Wang et al., 2019), image processing (Li & Hoiem, 2017; Wang et al., 2017; Chen et al., 2018;
Revised: Lopes et al., 2017; Wang et al., 2019), image processing (Li & Hoiem, 2017; Wang et al., 2017; Chen et al., 2018;

Original: Li et al., 2017), natural language processing (Nakashole & Flauger, 2017;
Revised: Li et al., 2017), natural language processing (Nakashole & Flauger, 2017;

Original: Mou et al., 2016; Hu et al., 2018;
Revised: Mou et al., 2016; Hu et al., 2018;

Original: Freitag et al., 2017), and speech processing (Chebotar & Waters, 2016;
Revised: Freitag et al., 2017), and speech processing (Chebotar & Waters, 2016;

Original: Lu et al., 2017; Watanabe et al., 2017; Oord et al., 2018;
Revised: Lu et al., 2017; Watanabe et al., 2017; Oord et al., 2018;

Original: Shen et al., 2018).
Revised: Shen et al., 2018).

Original: B G LOSSARY
Revised: B G LOSSARY

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original: [Table]
Revised: [Table]

Revised: f

Original: [Equation]
Revised: [Equation]

Revised: argmin

Original: Partial derivative of (cid:96) ( z,φ,π ) with respect to the second input
Revised: [Equation] [Table] [Equation] ∇

Original: [Table]
Revised: [Table]

Revised: Partial derivative of (cid:96) ( z ; φ,π ) with respect to the second input ∇ π

Original: Partial derivative of (cid:96) ( z,φ,π ) with respect to the third input 
Revised: Partial derivative of (cid:96) ( z ; φ,π ) with respect to the third input ∇ φπ

Original: [Table]
Revised: [Equation]

Revised: p

Original: [Equation]
Revised: [Equation]

Original: Localized Rademacher complexity of function class F
Revised: Localized Rademacher complexity of function class F

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: n

Original: [Table]
Revised: [Table]

Revised: γ -corrected loss (cid:96) γ ( z

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table]
Revised: f ( x )

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table]
Revised: p ( x ))

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: Population γ -risk L D ( f,p,γ )

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table]
Revised: X

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: X

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: Empirical γ -risk L n ( f,p,γ )

Original: [Equation]
Revised: [Equation]

Original: C P ROOF OF T HM . 1: V ANILLA DISTILLATION ANALYSIS
Revised: C P ROOF OF T HM . 1: V ANILLA DISTILLATION ANALYSIS

Original: First by localized Rademacher analysis (see Lemma 11 of Foster & Syrgkanis (2019)) if δ n upper bounds the critical radius of the function class G and if we denote with (cid:96)  ˆ p,f ( w )= (cid:96) ( w ; f ( x ) , ˆ p ( x ))  , then w.p. 1 − ζ :
Revised: Introduce the shorthand (cid:96) f, ˆ p ( z )= (cid:96) ( z ; f ( x ) , ˆ p ( x )) . Since δ n upper bounds the critical radius of the function class G , the localized Rademacher analysis of Foster & Syrgkanis (2019, Lem. 11) implies

Original: [Equation]
Revised: [Equation]

Original:  Moreover,if we let µ ( w )= sup φ (cid:107)∇ φ (cid:96) ( w ; φ, ˆ p ( x )) (cid:107) 2 , then we have by Cauchy-Scwarz inequality:
Revised: with probability at least 1 − ζ . Moreover, by Cauchy-Scwharz,

Original: [Equation]
Revised: [Equation]

Original: If we further assume that the function class F satisﬁes an (cid:96) 2 /(cid:96) 4  condition that:
Revised: By the assumed (cid:96) 2 /(cid:96) 4 ratio condition we therefore have

Original: [Equation]
Revised: [Equation]

Original: then:
Revised: Plugging this bound into Lemma 4 (which holds irrespective of whether data re-use, sample splitting

Original: [Equation]
Revised: [Equation]

Original: Plugging in the bound above to Lemma 4 (which holds irrespective of whether data re-use, sample splitting, or cross-ﬁtting is employed) and applyingagain the arithmetic-geometric mean inequality yields σ
Revised: or cross-ﬁtting is employed) and applying the arithmetic-geometric mean inequality yields σ

Original: [Equation]
Revised: [Equation]

Revised: instantiate

Original: D P ROOF OF P ROP . 2: I MPACT OF TEACHER UNDERFITTING ON VANILLA DISTILLATION
Revised: D P ROOF OF P ROP . 2: I MPACT OF TEACHER UNDERFITTING ON VANILLA DISTILLATION

Original: Suppose that, for known (cid:15)> 0 ,p 0 belongs to the set
Revised: Suppose that p 0 does not vary with x and, for known (cid:15)> 0 , belongs to the set

Original: [Equation] } and does not vary with x  .
Revised: P = { p : p j ( x ) ∈ [ (cid:15), 1] , ∀ x ∈X , j ∈ [ k ] } .

Original: As all quantities in this proof are independent of x , we will omit the dependence on x whenever appropriate.
Revised: As all quantities in this proof are independent of x , we will omit the dependence on x whenever convenient.

Original: Consider the constant teacher estimate ˆ p = ¯ y 1+{+
 ++} λ  obtained via ridge regression with regularization strength λ  and ¯ y (cid:44) 1 n (cid:80) ni
Revised: Consider the constant teacher estimate ˆ p = ¯ y {+
 ++} λ ∨ (cid:15) obtained via ridge regression with regularization strength λ ≤ 1 and ¯ y (cid:44) 1 n (cid:80) ni

Original: =

Revised: =

Original: y i .
Revised: y i .

Original: A constant student prediction rule in
Revised: A constant student prediction rule in

Original: [Equation]
Revised: [Equation]

Original:  trained via Vanilla KD with SEL loss yields ˆ f ( x)=log( ¯ y 1+ λ ) and satisﬁes
Revised: [Equation] ∈ [ k ] } trained via Vanilla KD with SEL loss yields ˆ f ( x ) =log(ˆ p ) .

Revised: Suppose that, unbeknownst to the teacher and student, the true p 0 satisﬁes the more stringent condition p 0 ,j ≥ 2 (cid:15) for all j ∈ [ k ] .

Revised: Then the student satisﬁes

Original: [Equation]
Revised: [Equation]

Original: by the concavity of the logarithm.
Revised: by the concavity of the logarithm and the choice λ ≤ 1 .

Original: Since
Revised: Since

Original: [Equation]
Revised: [Equation]

Original: by Bernstein’s inequality (Bernstein, 1946), we have
Revised: by Bernstein’s inequality (Bernstein, 1946), we have

Original: P

Original: [Equation]

Original: (cid:107)

Original: [Equation]

Original: [Equation]

Original: ˆ

Original: [Equation]
Revised: [Equation]

Original: 0 , ∀ [Equation] [ k ]) [Equation]
Revised: whenever

Original: λp

Original: [Equation]
Revised: [Equation]

Original: ≥ ¯ y j − p 0 ,j , ∀ j ∈ [ k ]) ≥ 1 − ζ whenever λ ≥ (cid:113) 2 n(cid:15) log( kζ )+ 43 1 n(cid:15) log( kζ ) .

Original: Moreover, since limsup n →∞
Revised: Moreover, since limsup n →∞

Original: [Equation]
Revised: [Equation]

Original: √ [Equation] ) =1 with probability 1 by the law of the iterated logarithm,(cid:107) γ (cid:62)
Revised: = 1 with probability 1 by the law of the iterated logarithm,

Original: [Equation]
Revised: [Equation]

Revised: Ω(min

Original: [Equation]
Revised: [Equation]

Original: p

Original: [Equation]
Revised: [Equation]

Original: ( p 0 − ˆ p ) (cid:107) 22 , 2 =Ω(min(1 ,λ 2 )) with probability 1 whenever  λ ≥ (cid:113)2loglog( n ) n(cid:15) . The choice λ
Revised: λ 2 )) with probability 1 whenever 1 ≥ λ ≥ (cid:113)

Original: [Equation]
Revised: [Equation]

Original: max
Revised: . The choice

Original: [Equation]
Revised: [Equation]

Original: now yields the ﬁrst two advertised claims.
Revised: now yields the ﬁrst two advertised claims.

Original: The ﬁnal claim follows directly from Thm.
Revised: The ﬁnal claim follows directly from Thm.

Original: 5 with B = O (1) as ˆ γ ( t ) =
Revised: 5 with B = O (1) as ˆ γ ( t ) =

Original: γ f 0 , ˆ p and the critical radius of G (ˆ p ( t ) , ˆ γ ( t ) ) satisﬁes δ n/B
Revised: γ f 0 , ˆ p and the critical radius of G (ˆ p ( t ) , ˆ γ ( t ) ) satisﬁes δ n/B

Original: = O ( (cid:112)
Revised: = O ( (cid:112)

Original: Bk/n ) by Wainwright (2019, Ex. 13.8).
Revised: Bk/n ) by Wainwright (2019, Ex. 13.8).

Original: E P ROOF OF P ROP . 3: I MPACT OF TEACHER OVERFITTING ON VANILLA DISTILLATION
Revised: E P ROOF OF P ROP . 3: I MPACT OF TEACHER OVERFITTING ON VANILLA DISTILLATION

Original: Suppose thatthe Bayes probability function p 0 has Lipschitz gradient and, for known (cid:15)> 0 , belongs to the set
Revised: Suppose that p 0 has Lipschitz gradient and, for known (cid:15)> 0 , belongs to the set

Original: P = { p : p j ( x ) ∈ [ (cid:15), 1] , ∀ x [Equation] j ∈ [ k ] } .
Revised: P = { p : p j ( x ) ∈ [ (cid:15), 1] , ∀ x ∈X , j ∈ [ k ] } .

Original: Suppose moreover that X ∈ R d has Lebesgue density bounded away from 0 and ∞ and that (cid:15)< 14 E [ p 0 ,j ( X )(1 − p 0 ,j ( X )) 2 ] E [(1 − p 0 ,j ( X )) /p 0 ,j ( X )] for each j .
Revised: Suppose moreover that X ∈ R d has Lebesgue density bounded away from 0 and ∞ and that (cid:15)< 1 4 E [ p 0 ,j ( X )(1 − p 0 ,j ( X )) 2 ] E [(1 − p 0 ,j ( X )) /p 0 ,j ( X )] for each j .

Original: Consider the teacher estimates ˆ p j ( x )=max( (cid:15), ˜ p
Revised: Consider the teacher estimates ˆ p j ( x )=max( (cid:15), ˜ p

Original: j ( x )) for ˜ p the Nadaraya-Watson kernel smoothing estimator (Nadaraya, 1964;
Revised: j ( x )) for ˜ p the Nadaraya-Watson kernel smoothing estimator (Nadaraya, 1964;

Original: Watson, 1964)
Revised: Watson, 1964)

Original: [Equation]
Revised: ˜ p ( x ) (cid:44) (cid:26)

Original: y i if x = x i 
Revised: y i if x = x i (cid:80) ni

Original: [Equation] ni = [Equation]{+=
 y i K (( x − x i ) /h ) /+}
Revised: {+=
 y i K (( x − x i ) /h ) /+}

Original: ) / (cid:80) ni =1 K (( x − x i ) /h )
Revised: (cid:80) ni =1 K (( x − x i ) /h )

Original: otherwise with kernel K ( x ) = (cid:107) x (cid:107) − a 

Revised: otherwise with kernel K ( x ) = (cid:107) x (cid:107) − a

Original: I [ (cid:107) x
Revised: I [ (cid:107) x

Original: (cid:107) 2 ≤ 1] , a ∈ (0 ,d/ 2) , and h = n − 1 / (4+ d ) .
Revised: (cid:107) 2 ≤ 1] , a ∈ (0 ,d/ 2) , and h = n − 1 / (4+ d ) .

Original: By Belkin et al.
Revised: By Belkin et al.

Original: (2019, Thm. 1), the teacher satisﬁes
Revised: (2019, Thm. 1), the teacher satisﬁes

Original: E [ (cid:107) p 0 − ˆ p (cid:107) 22 , 2 ]= O ( n − 4 / (4+ d ) ) .
Revised: E [ (cid:107) p 0 − ˆ p (cid:107) 22 , 2 ]= O ( n − 4 / (4+ d ) ) .

Original: Now instantiate the notation of Thm. 1, and consider a student prediction rule trained to learn a constant prediction rule via Vanilla
Revised: Now instantiate the notation of Thm. 1, and consider a student prediction rule trained to learn a constant prediction rule via Vanilla

Original: KD with the SEL loss and
Revised: KD with the SEL loss and

Original: [Equation] all x 
Revised: F = { f : f ( x )= f

Revised: ( x (cid:48) ) ∈ [log( (cid:15) ) , 0] k for all x,x (cid:48) ∈X} .

Original: [Equation]
Revised: [Equation]

Original: Since ˜ p exactly interpolates the observed labels (i.e., ˜ p ( x i ) =
Revised: Since ˜ p exactly interpolates the observed labels (i.e., ˜ p ( x i ) =

Original: y i ), the critical radius of the teacher-student function class
Revised: y i ), the critical radius of the teacher-student function class

Original: G satisﬁes δ n = Ω(1) .
Revised: G satisﬁes δ n = Ω(1) .

Original: Moreover, since the student only has access to the teacher’s training set probabilities, its estimate
Revised: Moreover, since the student only has access to the teacher’s training set probabilities, its estimate

Original: ˆ f ( x )= 1 n (cid:80) ni
Revised: ˆ f ( x )= 1 n (cid:80) ni

Original: =1 log(max( y i ,(cid:15) )) is inconsistent for the optimal constant rule f 0 ( x )=E [log( p 0 ( X ))] as f 0 , [Equation] ( x )
Revised: =1 log(max( y i ,(cid:15) )) is inconsistent for the optimal constant rule f 0 ( x )=

Original: [Equation]

Original: x )

Original: [Equation]

Original: [log [Equation] p 0 ,
Revised: E [log( p 0 ( X ))] as

Original: [Equation]
Revised: [Equation]

Original: X )

Original: [Equation]

Original: [Equation]

Original: ] by Taylor’s theorem with Lagrange remainder.
Revised: by Taylor’s theorem with Lagrange remainder.

Original: This non-vanishing student error reﬂects the non-vanishing critical radius δ n of the composite student-teacher function class G deﬁned in Thm. 1; since the student function class F has low complexity, the complexity of G is driven by the highly ﬂexible interpolating teacher.
Revised: This non-vanishing student error reﬂects the non-vanishing critical radius δ n of the composite student-teacher function class G deﬁned in Thm. 1; since the student function class F has low complexity, the complexity of G is driven by the highly ﬂexible interpolating teacher.

Original: Next, instantiate the notation of Thm. 5, and consider a student prediction rule ˆ f trained via Enhanced KD with SEL loss, ˆ γ ( t ) =0 ,
Revised: Next, instantiate the notation of Thm. 5, and consider a student prediction rule ˆ f trained via Enhanced KD with SEL loss, ˆ γ ( t ) =0 ,

Original: B = O (1) , and F (6).
Revised: B = O (1) , and F (6).

Original: The critical radius of G (ˆ p ( t ) , ˆ γ ( t ) ) satisﬁes δ n/B
Revised: The critical radius of G (ˆ p ( t ) , ˆ γ ( t ) ) satisﬁes δ n/B

Original: = O ( (cid:112)
Revised: = O ( (cid:112)

Original: Bk/n ) by Wainwright (2019, Ex. 13.8).
Revised: Bk/n ) by Wainwright (2019, Ex. 13.8).

Original: Moreover, each cross-ﬁtted teacher satisﬁes E [ (cid:107) p 0 − ˆ p ( t ) (cid:107) 22 , 2 ]=
Revised: Moreover, each cross-ﬁtted teacher satisﬁes E [ (cid:107) p 0 − ˆ p ( t ) (cid:107) 22 , 2 ]=

Original: O
Revised: O

Original: n − 4 / (4+ d ) ) by Belkin et al.
Revised: n − 4 / (4+ d ) ) by Belkin et al.

Original: (2019, Thm. 1), so, by Chebyshev’s and Jensen’s inequalities, with probability at least 1 − ζ/ 2 , 
Revised: (2019, Thm. 1), so, by Chebyshev’s and Jensen’s inequalities, with probability at least 1 − ζ/ 2 , [Equation] +

Original: [Equation]
Revised: [Equation]

Original: for all t.
Revised: for all t.

Original: Therefore, Thm. 5 implies that
Revised: Therefore, Thm. 5 implies that

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − ζ .
Revised: with probability at least 1 − ζ .

Original: F P ROOF OF L EMMA 4: A LGORITHM - AGNOSTIC ANALYSIS
Revised: F P ROOF OF L EMMA 4: A LGORITHM - AGNOSTIC ANALYSIS

Original: First we deﬁne for any functional L ( f ) the Frechet derivative as:
Revised: First we deﬁne for any functional L ( f ) the Frechet derivative as:

Original: [Equation]
Revised: [Equation]

Original: When L is an operator of the form: E [ g ( f ( X ))] 
Revised: When L is an operator of the form: E [ g ( f ( X ))] , then: D f L ( f )

Original: [Equation] then: D f L ( f )[ ν ]= E [ ∇ g ( f ( X )) (cid:62) ν ( X )] .
Revised: [ ν ]= E [ ∇ g ( f ( X )) (cid:62) ν ( X )] .

Original: By  σ -strong convexity of L D
Revised: By the σ -strong convexity of L D

Original: [Equation]
Revised: [Equation]

Original: 6 we have that
Revised: 6 we have that

Original: [Equation]
Revised: [Equation]

Original: Furthermore, our excess risk assumption and the optimality of f 0 give us σ
Revised: Furthermore, our excess risk assumption and the optimality of f 0 give us σ

Original: [Equation]
Revised: [Equation]

Original: By Taylor’s theorem with integral remainder,
Revised: By Taylor’s theorem with integral remainder,

Original: [Equation]
Revised: [Equation]

Original: whenever ∇ φπ (cid:96) is well-deﬁned.
Revised: whenever ∇ φπ (cid:96) is well-deﬁned.

Original: We can now invoke the expansion (7) and Cauchy-Schwarz to obtain the bound
Revised: We can now invoke the expansion (7) and Cauchy-Schwarz to obtain the bound

Original: [Equation]
Revised: [Equation]

Original: Thus combining all the above inequalities: σ
Revised: Thus combining all the above inequalities: σ

Original: [Equation]
Revised: [Equation]

Original: By an AM-GM inequality, for all a,b ≥ 0 : a · b ≤ 1 2 ( 2 σ a 2 + σ 2 b 2 ) .
Revised: By an AM-GM inequality, for all a,b ≥ 0 : a · b ≤ 12 ( [Equation] a [Equation]

Revised: σ 2 b 2 ) .

Original: Applying this to the product of norms on the RHS and re-arranging yields
Revised: Applying this to the product of norms on the RHS and re-arranging yields

Original: [Equation]
Revised: [Equation]

Original: To get the ﬁnal ineguality, observe that:
Revised: To get the ﬁnal inequality, observe that:

Original: [Equation]
Revised: [Equation]

Original: Moreover, by the boundedness of the third derivative, we have:
Revised: Moreover, by the boundedness of the third derivative, we have:

Original: [Equation]
Revised: [Equation]

Original: Combining all the above yields the ﬁnal bound.
Revised: Combining all the above yields the ﬁnal bound.

Original: G P ROOF OF T HM . 5: C ROSS - FITTED ERM ANALYSIS
Revised: G P ROOF OF T HM . 5: C ROSS - FITTED ERM ANALYSIS

Original: Let L n,t denote the empirical loss over the samples in the t -th fold and ˆ p ( t ) , ˆ γ ( t ) the nuisance functions used on the samples in the k -th fold.
Revised: Let L n,t denote the empirical loss over the samples in the t -th fold and ˆ p ( t ) , ˆ γ ( t ) the nuisance functions used on the samples in the k -th fold.

Original: For any t ∈ [ K ] and conditional on ˆ p ( t ) , ˆ γ ( t ) , suppose that δ n upper bounds the critical radius of the function class
Revised: For any t ∈ [ K ] and conditional on ˆ p ( t ) , ˆ γ ( t ) , suppose that δ n upper bounds the critical radius of the function class

Original: G (ˆ p ( t ) , ˆ γ ( t ) ) , then by Lemma 11 of Foster & Syrgkanis (2019), 7 if we denote with (cid:96) t,f ( w )= (cid:96) ˆ γ ( t ) ( w ; f ( x ) , ˆ p ( t ) ( x 
Revised: G (ˆ p ( t ) , ˆ γ ( t ) ) , then by Lemma 11 of Foster & Syrgkanis (2019), 7 if we denote with (cid:96) t,f ( z )= (cid:96) ˆ γ ( t ) ( z ; f ( x ) , ˆ p ( t ) ( x )) , w.p. 1 − ζ :

Original: [Equation]
Revised: [Equation]

Original: Moreover, we have that by the deﬁnition of cross-ﬁtted ERM:
Revised: Moreover, we have that by the deﬁnition of cross-ﬁtted ERM:

Original: [Equation]
Revised: [Equation]

Original: Thus we have that w.p. 1 − ζB :
Revised: Thus we have that w.p. 1 − ζB :

Original: [Equation]
Revised: [Equation]

Original:  Moreover, if we let µ ( w )=sup φ,t (cid:13)(cid:13) ∇ φ (cid:96) ( w ; φ, ˆ p ( t ) ( x ))
Revised: + [Equation] ζ (cid:33) Moreover, if we let µ ( z )=sup φ,t (cid:13)(cid:13) ∇ φ (cid:96) ( z ; φ, ˆ p ( t ) ( x ))

Original: (cid:13)(cid:13) 2 , then we have by Cauchy-Schwarz inequality:
Revised: (cid:13)(cid:13) 2 , then we have by Cauchy-Schwarz inequality:

Original: [Equation]
Revised: [Equation]

Original: −
Revised: t [Equation] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: µ (cid:107)
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original: If we further assume that the function class F satisﬁes an (cid:96) 2 /(cid:96) 4 condition that:
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: then w.p
Revised: [Table] [Equation] [Table] If we further assume that the function class F satisﬁes an (cid:96) 2 /(cid:96) 4 condition that:

Original: [Equation]
Revised: [Equation]

Original:  1 − ζ :
Revised: then w.p. 1 − ζ :

Original: [Equation]
Revised: [Equation]

Original: Applying Lemma 4 for any ˆ p ( t ) , ˆ γ ( t ) and averaging the ﬁnal inequality we get:
Revised: Applying Lemma 4 for any ˆ p ( t ) , ˆ γ ( t ) and averaging the ﬁnal inequality we get:

Original: [Equation]
Revised: [Equation]

Original:  Plugging in the bound above to Lemma 4 and applying the AM-GM inequality and Jensen’s inequality [Equation] yields:
Revised: [Equation] Plugging in the bound above to Lemma 4 and applying the AM-GM inequality and Jensen’s inequality, yields:

Original: [Equation]
Revised: [Equation]

Original: H P ROOF OF T HM . 6: B IASED SGD ANALYSIS
Revised: H P ROOF OF T HM . 6: B IASED SGD ANALYSIS

Revised: Below

Original: Below, for any integer s , we deﬁne the operator norm of any vector v ∈ R s and any tensor T operating on R s as(cid:107) v (cid:107) op (cid:44) (cid:107) v (cid:107)and (cid:107) T (cid:107) op
Revised: [Equation] for any integer s [Equation] we deﬁne the operator norm of any vector v ∈ R s and any tensor T operating on R s as

Original: (cid:44) sup v : (cid:107) v (cid:107) 2 =1 (cid:107) T 
Revised: [Equation] and (cid:107) T (cid:107) [Equation] (cid:44)

Original: [ v ] (cid:107) op .
Revised: [Equation]

Original: Recall the deﬁnition
Revised: Recall the deﬁnition

Original: [Equation]
Revised: [Equation]

Original: Observe that since E [ Y | X = x ]= p 0 ( x ) , we can write for any γ :
Revised: Observe that since E [ Y | X = x ]= p 0 ( x ) , we can write for any γ :

Original: [Equation]
Revised: [Equation]

Original: Thus we also have that:
Revised: Thus we also have that:

Original: [Equation]
Revised: [Equation]

Original: Given this observation, we can decompose the gradient that is used in our SGD algorithm into a bias and variance component, when viewed from the perspective of a biased SGD algorithm for the population oracle loss:
Revised: Given this observation, we can decompose the gradient that is used in our SGD algorithm into a bias and variance component, when viewed from the perspective of a biased SGD algorithm for the population oracle loss:

Original: [Equation]
Revised: [Equation]

Original: The following two lemmas bound the gradient bias and noise terms.
Revised: The following two lemmas bound the gradient bias and noise terms.

Original: Lemma 7 (Gradient bias) .
Revised: Lemma 7 (Gradient bias) .

Original: If sup x,φ,π (cid:107) E [ ∇ ππφ (cid:96) ( W ; φ,π ) | X
Revised: If sup x,φ,π (cid:107) E [ ∇ ππφ (cid:96) ( W ; φ,π ) | X

Original: = x ] (cid:107) op ≤ M , then for any parameter vector θ and functions p and γ , we have: 
Revised: = x ] (cid:107) op ≤ M , then for any parameter vector θ and functions p and γ , we have: [Equation] γ f [Equation]

Revised: ,p

Original: [Equation]
Revised: [Equation]

Revised: γ )

Revised: [Equation]

Revised: p − p 0 )

Revised: [Equation]

Revised: [Equation]

Revised: and

Revised: [Equation]

Original: Proof
Revised: Proof

Original: By Taylor’s theorem with integral remainder and Lagrange remainder respectively the SGD bias for each parameter i takes the form
Revised: By Taylor’s theorem with integral remainder and Lagrange remainder respectively the SGD bias for each parameter i takes the form

Original: [Equation]
Revised: [Equation]

Original: Furthermore, our operator norm assumption and Cauchy-Schwarz imply
Revised: Furthermore, our operator norm assumption and Cauchy-Schwarz imply

Original: [Equation]
Revised: [Equation]

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: by the triangle inequality and Jensen’s inequality we ﬁnd that
Revised: by the triangle inequality and Jensen’s inequality we ﬁnd that

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: ∇ θ f (cid:62) θ ( q f θ ,p − γ ) (cid:62) ( p − p 0 ) (cid:13)(cid:13) 2 , 2 + M 2 (cid:107)∇ θ f θ (cid:107) F, 2 (cid:107) p − p 0 (cid:107) 22 , 4 .

Original: Lemma 8 (Gradient Variance) .
Revised: Lemma 8 (Gradient Variance) .

Original: Deﬁne For any parameter θ and functions p and γ , (cid:113)E
Revised: Deﬁne For any parameter θ and functions p and γ , (cid:113)

Original: [-[ (cid:107) n ( W ; θ,p,γ ) (cid:107) 22 ] ≤ σ 0 ( θ )+ (cid:113) E [ (cid:107)∇ θ f θ ( X ) (cid:62) γ ( X ) (cid:62) ( Y − p 
 ( X )) (cid:107) 22 ]+ (cid:13)(cid:13) ∇ θ f (cid:62)-]
Revised: [-[ (cid:107) n ( W ; θ,p,γ ) (cid:107) 22 ] ≤ σ 0 ( θ )+ (cid:113) E [ (cid:107)∇ θ f θ ( X ) (cid:62) γ ( X ) (cid:62) ( Y − p 
 ( X )) (cid:107) 22 ]+ (cid:13)(cid:13) ∇ θ f (cid:62)-][Equation]

Original: θ ( γ f θ ,p − γ ) (cid:62) ( p − p 0 )

Original: (cid:13)(cid:13) 2 2 , 2 .

Original: Proof
Revised: Proof

Original: For each i ∈ [ d ] , deﬁne the shorthand
Revised: For each i ∈ [ d ] , deﬁne the shorthand

Original: [Equation]
Revised: [Equation]

Original: for some convex combination ¯ p ( X ) of p ( X ) and p 0 ( X ) .
Revised: for some convex combination ¯ p ( X ) of p ( X ) and p 0 ( X ) .

Original: We begin by bounding the target expectation using Cauchy-Schwarz
Revised: We begin by bounding the target expectation using Cauchy-Schwarz

Original: [Equation]
Revised: [Equation]

Original: We next employ the law of total variance to rewrite the variance terms:
Revised: We next employ the law of total variance to rewrite the variance terms:

Original: [Equation]
Revised: [Equation]

Original: Finally, we control Var[ Z i ] using Cauchy-Schwarz (cid:115)(cid:88)
Revised: Finally, we control Var[ Z i ] using Cauchy-Schwarz (cid:115)(cid:88)

Original: [Equation]
Revised: [Equation]

Original: The two claims of Thm. 6 now follow from Theorems 2 and 3 of Ajalloeian & Stich (2020) respectively, with the parameters σand ζ instantiated with quantities σ 2 ( γ ) and ζ ( γ ) of Lemmas 7 and
Revised: The two claims of Thm. 6 now follow from Theorems 2 and 3 of Ajalloeian & Stich (2020) respectively, with the parameters σand ζ instantiated with quantities σ 2 ( γ ) and ζ ( γ ) of Lemmas 7 and

Original: . I E XPERIMENT D ETAILS AND A DDITIONAL R ESULTS I. T ABULAR DATA
Revised: . I E XPERIMENT D ETAILS AND A DDITIONAL R ESULTS I. T ABULAR DATA

Original: We use cross-ﬁtting with 10 folds.
Revised: We use cross-ﬁtting with 10 folds.

Original: The student is trained using the SEL loss with clipped teacher class probabilities max(ˆ p ( x ) ,(cid:15) ) for (cid:15) =10 − 3 .
Revised: The student is trained using the SEL loss with clipped teacher class probabilities max(ˆ p ( x ) ,(cid:15) ) for (cid:15) =10 − 3 .

Original: The λ hyperparameter of the loss correction was chosen by cross-validation with 5 folds.
Revised: The α hyperparameter of the loss correction was chosen by cross-validation with 5 folds.

Original: We repeat the experiments 5 times to measure the mean and standard deviation.
Revised: We repeat the experiments 5 times to measure the mean and standard deviation.

Original: For the overﬁtting experiment, we use a random forest with 500 trees as the teacher and a random forest with 1-40 trees as the student.
Revised: For the overﬁtting experiment, we use a random forest with 500 trees as the teacher and a random forest with 1-40 trees as the student.

Original: We also evaluate the impact of teacher underﬁtting by limiting the teacher’s maximum tree depth (from 1 to 20).
Revised: We also evaluate the impact of teacher underﬁtting by limiting the teacher’s maximum tree depth (from 1 to 20).

Original: Lower depth corresponds to greater underﬁtting.
Revised: Lower depth corresponds to greater underﬁtting.

Original: The teacher has 100 trees, and the student has 10 trees.
Revised: The teacher has 100 trees, and the student has 10 trees.

Original: For all of the datasets, loss correction successfully mitigates the teacher’s underﬁtting and thus improves the student’s performance.
Revised: For all of the datasets, loss correction successfully mitigates the teacher’s underﬁtting and thus improves the student’s performance.

Original: The effect is most pronounced when the teacher underﬁts more heavily (has lower tree depth).
Revised: The effect is most pronounced when the teacher underﬁts more heavily (has lower tree depth).

Original: We show the full results for all 5 of the datasets in Figs.
Revised: We show the full results for all 5 of the datasets in Figs.

Original: 4 and
Revised: 4 and

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: I. I MAGE DATA (CIFAR-10)
Revised: I. I MAGE DATA (CIFAR-10)

Original: We use SGD with initial learning rate 0.1, momentum 0.9, and batch size 128 to train for 200 epochs.
Revised: We use SGD with initial learning rate 0.1, momentum 0. [Figure] and batch size 128 to train for 200 epochs.

Original: We use the standard learning rate decay schedule, where the learning rate is divided by 5 at epoch 60, 120, and 160.
Revised: We use the standard learning rate decay schedule, where the learning rate is divided by 5 at epoch 60, 120, and 160.

Revised: For loss correction, we select the value of the hyperparameter α that yields the highest accuracy on a held-out validation set.

Revised: For cross-ﬁtting, we use 10 folds.