Original: S UPERVISION C OMPLEXITY AND ITS R OLE IN K NOWLEDGE D ISTILLATION
Revised: S UPERVISION C OMPLEXITY AND ITS R OLE IN K NOWLEDGE D ISTILLATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Despite the popularity and efficacy of knowledge distillation, there is limited understanding of why it helps.
Revised: Despite the popularity and efficacy of knowledge distillation, there is limited understanding of why it helps.

Original: In order to study the generalization behavior of a distilled student, we propose a new theoretical framework that leverages supervision complexity : a measure of alignment between teacher-provided supervision and the student’s neural tangent kernel .
Revised: In order to study the generalization behavior of a distilled student, we propose a new theoretical framework that leverages supervision complexity : a measure of alignment between teacher-provided supervision and the student’s neural tangent kernel .

Original: The framework highlights a delicate interplay among the teacher’s accuracy, the student’s margin with respect to the teacher predictions, and the complexity of the teacher predictions.
Revised: The framework highlights a delicate interplay among the teacher’s accuracy, the student’s margin with respect to the teacher predictions, and the complexity of the teacher predictions.

Original: Specifically, it provides a rigorous justification for the utility of various techniques that are prevalent in the context of distillation, such as early stopping and temperature scaling.
Revised: Specifically, it provides a rigorous justification for the utility of various techniques that are prevalent in the context of distillation, such as early stopping and temperature scaling.

Original: Our analysis further suggests the use of online distillation , where a student receives increasingly more complex supervision from teachers in different stages of their training.
Revised: Our analysis further suggests the use of online distillation , where a student receives increasingly more complex supervision from teachers in different stages of their training.

Original: We demonstrate efficacy of online distillation and validate the theoretical findings on a range of image classification benchmarks and model architectures.
Revised: We demonstrate efficacy of online distillation and validate the theoretical findings on a range of image classification benchmarks and model architectures.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Knowledge distillation (KD)
Revised: Knowledge distillation (KD)

Original: (Bucilu ˇ a et al., 2006; Hinton et al., 2015) is a popular method of compressing a large “teacher” model into a more compact “student” model.
Revised: (Bucilu ˇ a et al., 2006; Hinton et al., 2015) is a popular method of compressing a large “teacher” model into a more compact “student” model.

Original: In its most basic form, this involves training the student to fit the teacher’s predicted label distribution or soft labels for each sample.
Revised: In its most basic form, this involves training the student to fit the teacher’s predicted label distribution or soft labels for each sample.

Original: There is strong empirical evidence that distilled students usually perform better than students trained on raw dataset labels (Hinton et al., 2015;
Revised: There is strong empirical evidence that distilled students usually perform better than students trained on raw dataset labels (Hinton et al., 2015;

Original: Furlanello et al., 2018; Stanton et al., 2021;
Revised: Furlanello et al., 2018; Stanton et al., 2021;

Original: Gou et al., 2021).
Revised: Gou et al., 2021).

Original: Multiple works have devised novel KD procedures that further improve the student model performance (see Gou et al.
Revised: Multiple works have devised novel KD procedures that further improve the student model performance (see Gou et al.

Original: and references therein).
Revised: and references therein).

Original: Simultaneously, several works have aimed to rigorously formalize why KD can improve the student model performance.
Revised: Simultaneously, several works have aimed to rigorously formalize why KD can improve the student model performance.

Original: Some prominent observations from this line of work are that (self-)distillation induces certain favorable optimization biases in the training objective (Phuong & Lampert, 2019; Ji & Zhu, 2020), lowers
Revised: Some prominent observations from this line of work are that (self-)distillation induces certain favorable optimization biases in the training objective (Phuong & Lampert, 2019; Ji & Zhu, 2020), lowers

Original: variance of the objective (Menon et al., 2021; Dao et al., 2021; Ren et al., 2022), increases regularization towards learning “simpler” functions (Mobahi et al., 2020), transfers information from different data views (Allen-Zhu & Li, 2020), and scales per-example gradients based on the teacher’s confidence (Furlanello et al., 2018; Tang et
Revised: variance of the objective (Menon et al., 2021; Dao et al., 2021; Ren et al., 2022), increases regularization towards learning “simpler” functions (Mobahi et al., 2020), transfers information from different data views (Allen-Zhu & Li, 2020), and scales per-example gradients based on the teacher’s confidence (Furlanello et al., 2018; Tang et

Original: al
Revised: al

Original: Despite this remarkable progress there are still many open problems and unexplained phenomena around knowledge distillation; to name a few:
Revised: Despite this remarkable progress there are still many open problems and unexplained phenomena around knowledge distillation; to name a few:

Original: — Why do soft labels (sometimes) help?
Revised: — Why do soft labels (sometimes) help?

Original: It is agreed that teacher’s soft predictions carry information about class similarities (Hinton et al., 2015; Furlanello et al., 2018), and that this softness of predictions has a regularization effect similar to label smoothing (Yuan et al., 2020).
Revised: It is agreed that teacher’s soft predictions carry information about class similarities (Hinton et al., 2015; Furlanello et al., 2018), and that this softness of predictions has a regularization effect similar to label smoothing (Yuan et al., 2020).

Original: Nevertheless, KD also works in binary classification settings with limited class similarity information (M¨uller et al., 2020).
Revised: Nevertheless, KD also works in binary classification settings with limited class similarity information (M¨uller et al., 2020).

Original: How exactly the softness of teacher predictions (controlled by a temperature parameter) affects the student learning remains far from well understood.
Revised: How exactly the softness of teacher predictions (controlled by a temperature parameter) affects the student learning remains far from well understood.

Original: — The role of capacity gap .
Revised: — The role of capacity gap .

Original: There is evidence that when there is a significant capacity gap between the teacher and the student, the distilled model usually falls behind its teacher (Mirzadeh et al., 2020; Cho & Hariharan, 2019; Stanton et al., 2021).
Revised: There is evidence that when there is a significant capacity gap between the teacher and the student, the distilled model usually falls behind its teacher (Mirzadeh et al., 2020; Cho & Hariharan, 2019; Stanton et al., 2021).

Original: It is unclear whether this is due to difficulties in optimization, or due to insufficient student capacity.
Revised: It is unclear whether this is due to difficulties in optimization, or due to insufficient student capacity.

Original: — What makes a good teacher?
Revised: — What makes a good teacher?

Original: Sometimes less accurate models are better teachers (Cho & Hariharan, 2019; Mirzadeh et al., 2020).
Revised: Sometimes less accurate models are better teachers (Cho & Hariharan, 2019; Mirzadeh et al., 2020).

Original: Moreover, early stopped or exponentially averaged models are often better teachers (Ren et al., 2022).
Revised: Moreover, early stopped or exponentially averaged models are often better teachers (Ren et al., 2022).

Original: A comprehensive explanation of this remains elusive.
Revised: A comprehensive explanation of this remains elusive.

Original:  The aforementioned wide range of phenomena suggest that there is a complex interplay between teacher accuracy, softness of teacher-provided targets, and complexity of the distillation objective.
Revised: [Figure] The aforementioned wide range of phenomena suggest that there is a complex interplay between teacher accuracy, softness of teacher-provided targets, and complexity of the distillation objective.

Original: This paper provides a new theoretically grounded perspective on KD through the lens of supervision complexity .
Revised: This paper provides a new theoretically grounded perspective on KD through the lens of supervision complexity .

Original: In a nutshell, this quantifies why certain targets (e.g., temperature-scaled teacher probabilities) may be “easier” for a student model to learn compared to others (e.g., raw one-hot labels), owing to better alignment with the student’s neural tangent kernel ( NTK ) (Jacot et al., 2018; Lee et al., 2019).
Revised: In a nutshell, this quantifies why certain targets (e.g., temperature-scaled teacher probabilities) may be “easier” for a student model to learn compared to others (e.g., raw one-hot labels), owing to better alignment with the student’s neural tangent kernel ( NTK ) (Jacot et al., 2018; Lee et al., 2019).

Original: In particular, we provide a novel theoretical analysis (§2, Thm. 3 and 4) of the role of supervision complexity on kernel classifier generalization, and use this to derive a new generalization bound for distillation (Prop. 5).
Revised: In particular, we provide a novel theoretical analysis (§2, Thm. 3 and 4) of the role of supervision complexity on kernel classifier generalization, and use this to derive a new generalization bound for distillation (Prop. 5).

Original: The latter highlights how student generalization is controlled by a balance of the teacher generalization , the student’s margin with respect to the teacher predictions, and the complexity of the teacher’s predictions.
Revised: The latter highlights how student generalization is controlled by a balance of the teacher generalization , the student’s margin with respect to the teacher predictions, and the complexity of the teacher’s predictions.

Original: Based on the preceding analysis, we establish the conceptual and practical efficacy of a simple online distillation approach (§4), wherein the student is fit to progressively more complex targets, in the form of teacher predictions at various checkpoints during its training.
Revised: Based on the preceding analysis, we establish the conceptual and practical efficacy of a simple online distillation approach (§4), wherein the student is fit to progressively more complex targets, in the form of teacher predictions at various checkpoints during its training.

Original: This method can be seen as guiding the student in the function space (see Fig. 1), and leads to better generalization compared to offline distillation.
Revised: This method can be seen as guiding the student in the function space (see Fig. 1), and leads to better generalization compared to offline distillation

Original: We provide empirical results on a range of image classification benchmarks confirming the value of online distillation, particularly for students with weak inductive biases.
Revised: We provide empirical results on a range of image classification benchmarks confirming the value of online distillation, particularly for students with weak inductive biases.

Original: Beyond practical benefits, the supervision complexity view yields new insights into distillation:
Revised: Beyond practical benefits, the supervision complexity view yields new insights into distillation:

Original: — The role of temperature scaling and early-stopping .
Revised: — The role of temperature scaling and early-stopping .

Original: Temperature scaling and early-stopping of the teacher have proven effective for KD.
Revised: Temperature scaling and early-stopping of the teacher have proven effective for KD.

Original: We show that both of these techniques reduce the supervision complexity, at the expense of also lowering the classification margin.
Revised: We show that both of these techniques reduce the supervision complexity, at the expense of also lowering the classification margin.

Original: Online distillation manages to smoothly increase teacher complexity, without degrading the margin.
Revised: Online distillation manages to smoothly increase teacher complexity, without degrading the margin.

Original: — Teaching a weak student .
Revised: — Teaching a weak student .

Original: We show that for students with weak inductive biases, and/or with much less capacity than the teacher, the final teacher predictions are often as complex as dataset labels , particularly during the early stages of training.
Revised: We show that for students with weak inductive biases, and/or with much less capacity than the teacher, the final teacher predictions are often as complex as dataset labels , particularly during the early stages of training.

Original: In contrast, online distillation allows the supervision complexity to progressively increase, thus allowing even a weak student to learn.
Revised: In contrast, online distillation allows the supervision complexity to progressively increase, thus allowing even a weak student to learn.

Original: — NTK and relational transfer .
Revised: — NTK and relational transfer .

Original: We show that online distillation is highly effective at matching the teacher and student NTK matrices.
Revised: We show that online distillation is highly effective at matching the teacher and student NTK matrices.

Original: This transfers relational knowledge in the form of examplepair similarity, as opposed to standard distillation which only transfers per-example knowledge .
Revised: This transfers relational knowledge in the form of examplepair similarity, as opposed to standard distillation which only transfers per-example knowledge .

Original: Problem setting.
Revised: Problem setting.

Original: We focus on classification problems from input domain X to d classes.
Revised: We focus on classification problems from input domain X to d classes.

Original: We are given a training set of n labeled examples { ( x 1 , y 1 ) , . . .
Revised: We are given a training set of n labeled examples { ( x 1 , y 1 ) , . . .

Original: , ( x n , y n ) } , with one-hot encoded labels y i ∈ { 0 , 1 } d .
Revised: , ( x n , y n ) } , with one-hot encoded labels y i ∈ { 0 , 1 } d .

Original: Typically, a model f θ : X → R d is trained with the softmax cross-entropy loss:
Revised: Typically, a model f θ : X → R d is trained with the softmax cross-entropy loss:

Original: [Equation]
Revised: [Equation]

Original:  where σ ( · ) is the softmax function.
Revised: (1) where σ ( · ) is the softmax function.

Original: In standard KD, given a trained teacher model g : X → R d that outputs logits, one trains a student model f θ : X → R d to fit the teacher predictions.
Revised: In standard KD, given a trained teacher model g : X → R d that outputs logits, one trains a student model f θ : X → R d to fit the teacher predictions.

Original: Hinton et al.
Revised: Hinton et al.

Original: (2015) propose the following KD loss:
Revised: (2015) propose the following KD loss:

Original: [Equation]
Revised: [Equation]

Original: where temperature τ > 0 controls the softness of teacher predictions.
Revised: where temperature τ > 0 controls the softness of teacher predictions.

Original: To highlight the effect of KD and simplify exposition, we assume that the student is not trained with the dataset labels.
Revised: To highlight the effect of KD and simplify exposition, we assume that the student is not trained with the dataset labels.

Original:  S UPERVISION COMPLEXITY AND GENERALIZATION
Revised:  S UPERVISION COMPLEXITY AND GENERALIZATION

Original: One apparent difference between standard training and KD (Eq. 1 and 2) is that the latter modifies the targets that the student attempts to fit.
Revised: One apparent difference between standard training and KD (Eq. 1 and 2) is that the latter modifies the targets that the student attempts to fit.

Original: The targets used during distillation ensure a better generalization for the student; what is the reason for this?
Revised: The targets used during distillation ensure a better generalization for the student; what is the reason for this?

Original: Towards answering this question, we present a new perspective on KD in terms of supervision complexity .
Revised: Towards answering this question, we present a new perspective on KD in terms of supervision complexity .

Original: To begin, we show how the generalization of a kernel-based classifier is controlled by a measure of alignment between the target labels and the kernel matrix.
Revised: To begin, we show how the generalization of a kernel-based classifier is controlled by a measure of alignment between the target labels and the kernel matrix.

Original: We first treat binary kernel-based classifiers (Thm. 3), and later extend our analysis to multiclass kernel-based classifiers (Thm. 4).
Revised: We first treat binary kernel-based classifiers (Thm. 3), and later extend our analysis to multiclass kernel-based classifiers (Thm. 4).

Original: Finally, by leveraging the neural tangent kernel machinery, we discuss the implications of our analysis for neural classifiers in §2.2.[Figure]
Revised: Finally, by leveraging the neural tangent kernel machinery, we discuss the implications of our analysis for neural classifiers in §2.2.

Original: 2. S UPERVISION COMPLEXITY CONTROLS KERNEL MACHINE GENERALIZATION
Revised: 2. S UPERVISION COMPLEXITY CONTROLS KERNEL MACHINE GENERALIZATION

Revised: The notion of supervision complexity is easiest to introduce and study for kernel-based classifiers.

Original: We briefly review some necessary background (Scholkopf & Smola, 2001).
Revised: We briefly review some necessary background (Scholkopf & Smola, 2001).

Original: Let k : X
Revised: Let k : X

Original: ×
Revised: ×

Original: X
Revised: X

Original: → R be a positive semidefinite kernel defined over an input space X .
Revised: → R be a positive semidefinite kernel defined over an input space X .

Original: Any such kernel uniquely determines a reproducing kernel Hilbert space (RKHS) H of functions from X to R .
Revised: Any such kernel uniquely determines a reproducing kernel Hilbert space (RKHS) H of functions from X to R .

Original: This RKHS is the completion of the set of functions of form f ( x ) = (cid:80) mi
Revised: This RKHS is the completion of the set of functions of form f ( x ) = (cid:80) m i

Original: =

Revised: =

Original: α i k
Revised: α i k

Original: ( x i , x ) , with x i ∈ X , α i ∈ R .
Revised: ( x i , x ) , with x i ∈ X , α i ∈ R .

Original: Any f ( x ) = (cid:80) mi =

Revised: Any f ( x ) = (cid:80) mi =

Original: α i k
Revised: α i k

Original: ( x i , x ) ∈ H has (RKHS) norm ∥
Revised: ( x i , x ) ∈ H has (RKHS) norm ∥

Original: f ∥ 2 H =
Revised: f ∥ 2 H =

Original: (cid:80) mi =1 (cid:80) mj
Revised: (cid:80) mi =1 (cid:80) mj

Original: =

Revised: =

Original: α i α j k
Revised: α i α j k

Original: ( x i , x j ) =
Revised: ( x i , x j ) =

Original: α ⊤ K α , where α = ( α 1 , . . . , α n ) ⊤ and K i,j = k
Revised: α ⊤ K α , where α = ( α 1 , . . . , α n ) ⊤ and K i,j = k

Original: ( x i , x j ) .
Revised: ( x i , x j ) .

Original: Intuitively, ∥ f ∥ 2 H measures the smoothness of f , e.g., for a Gaussian kernel it measures the Fourier spectrum decay of f (Scholkopf & Smola, 2001).
Revised: Intuitively, ∥ f ∥ 2 H measures the smoothness of f , e.g., for a Gaussian kernel it measures the Fourier spectrum decay of f (Scholkopf & Smola, 2001).

Original: For simplicity, we start with the case of binary classification.
Revised: For simplicity, we start with the case of binary classification.

Original: Suppose { ( X i , Y i ) } i ∈ [ n ] are n i.i.d.
Revised: Suppose { ( X i , Y i ) } i ∈ [ n ] are n i.i.d.

Original: examples sampled from some probability distribution on X × Y , with Y ⊂ R , where positive and negative labels correspond to distinct classes.
Revised: examples sampled from some probability distribution on X × Y , with Y ⊂ R , where positive and negative labels correspond to distinct classes.

Original: Let K i,j = k
Revised: Let K i,j = k

Original: ( X i , X j ) denote the kernel matrix, and
Revised: ( X i , X j ) denote the kernel matrix, and

Original: Y =
Revised: Y =

Original: ( Y 1 , . . .
Revised: ( Y 1 , . . .

Original: , Y n ) ⊤ be the concatenation of all training labels.
Revised: , Y n ) ⊤ be the concatenation of all training labels.

Original: Definition 1 (Supervision complexity) .
Revised: Definition 1 (Supervision complexity) .

Original: The supervision complexity of targets Y 1 , . . .
Revised: The supervision complexity of targets Y 1 , . . .

Original: , Y n with respect to a kernel k is defined to be Y ⊤ K − 1 Y in cases when K is invertible, and + ∞ otherwise.
Revised: , Y n with respect to a kernel k is defined to be Y ⊤ K − 1 Y in cases when K is invertible, and + ∞ otherwise.

Original: We now establish how supervision complexity controls the smoothness of the optimal kernel classifier.
Revised: We now establish how supervision complexity controls the smoothness of the optimal kernel classifier.

Original: Consider a classifier obtained by solving a regularized kernel classification problem:
Revised: Consider a classifier obtained by solving a regularized kernel classification problem:

Original: [Equation]
Revised: [Equation]

Original: where ℓ is a loss function and λ > 0 .
Revised: where ℓ is a loss function and λ > 0 .

Original: The following proposition shows whenever the supervision complexity is small, the RKHS norm of any optimal solution f ∗ will also be small (see Appendix B for a proof).
Revised: The following proposition shows whenever the supervision complexity is small, the RKHS norm of any optimal solution f ∗ will also be small (see Appendix B for a proof).

Original: This is an important learning bias that shall help us explain certain aspects of KD.
Revised: This is an important learning bias that shall help us explain certain aspects of KD.

Original: Proposition 2.
Revised: Proposition 2.

Original: Assume that K is full rank almost surely; ℓ ( y, y ′ ) ≥ 0 , ∀ y, y ′ ∈ Y ; and ℓ ( y, y )
Revised: Assume that K is full rank almost surely; ℓ ( y, y ′ ) ≥ 0 , ∀ y, y ′ ∈ Y ; and ℓ ( y, y )

Original: = 0 , ∀ y ∈ Y .
Revised: = 0 , ∀ y ∈ Y .

Original: Then, with probability 1, for any solution f ∗ of (3) , we have ∥ f ∗ ∥
Revised: Then, with probability 1, for any solution f ∗ of (3) , we have ∥ f ∗ ∥

Original: 2 H ≤ Y ⊤ K − 1 Y .
Revised: 2 H ≤ Y ⊤ K − 1 Y .

Original: Equipped with the above result, we now show how supervision complexity controls generalization.
Revised: Equipped with the above result, we now show how supervision complexity controls generalization.

Original: In the following, let ϕ γ : R → [0 , 1] be the margin loss (Mohri et al., 2018) with scale γ > 0 :
Revised: In the following, let ϕ γ : R → [0 , 1] be the margin loss (Mohri et al., 2018) with scale γ > 0 :

Original: [Equation]
Revised: [Equation]

Original: Theorem 3.
Revised: Theorem 3.

Original: Assume that κ = sup x ∈X k ( x, x ) < ∞ and K is full rank almost surely.
Revised: Assume that κ = sup x ∈X k ( x, x ) < ∞ and K is full rank almost surely.

Original: Further, assume that ℓ ( y, y ′ ) ≥ 0 , ∀ y, y ′ ∈ Y and ℓ ( y, y ) = 0 , ∀ y ∈ Y .
Revised: Further, assume that ℓ ( y, y ′ ) ≥ 0 , ∀ y, y ′ ∈ Y and ℓ ( y, y ) = 0 , ∀ y ∈ Y .

Original: Let M 

Revised: Let M

Original: =
Revised: =

Original: γ √ n/ (2 √ κ ) ⌉ .
Revised: γ √ n/ (2 √ κ ) ⌉ .

Original: Then, with probability at least 1 − δ , for any solution f ∗ of problem in Eq.
Revised: Then, with probability at least 1 − δ , for any solution f ∗ of problem in Eq.

Original: (3), we have
Revised: (3), we have

Revised: P

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The proof is available in Appendix B. One can compare Thm.
Revised: The proof is available in Appendix B. One can compare Thm.

Original: 3 with the standard Rademacher bound for kernel classifiers (Bartlett & Mendelson, 2002).
Revised: 3 with the standard Rademacher bound for kernel classifiers (Bartlett & Mendelson, 2002).

Original: The latter typically consider learning over functions with RKHS norm bounded by a constant M > 0 .
Revised: The latter typically consider learning over functions with RKHS norm bounded by a constant M > 0 .

Original: The corresponding complexity term then decays as O ( (cid:112) M
Revised: The corresponding complexity term then decays as O ( (cid:112) M

Original: · Tr ( K ) /n ) , which is data-independent .
Revised: · Tr ( K ) /n ) , which is data-independent .

Original: Consequently, such a bound cannot adapt to the intrinsic “difficulty” of the targets Y .
Revised: Consequently, such a bound cannot adapt to the intrinsic “difficulty” of the targets Y .

Original: In contrast, Thm. 3 considers functions with RKHS norm bounded by the data-dependent supervision complexity term.
Revised: In contrast, Thm. 3 considers functions with RKHS norm bounded by the data-dependent supervision complexity term.

Original: This results in a more informative generalization bound, which captures the “difficulty” of the targets.
Revised: This results in a more informative generalization bound, which captures the “difficulty” of the targets.

Original: Here, we note that Arora et al.
Revised: Here, we note that Arora et al.

Original: characterized the generalization of an overparameterized two-layer neural network via a term closely related to the supervision complexity (see §5 for additional discussion).
Revised: characterized the generalization of an overparameterized two-layer neural network via a term closely related to the supervision complexity (see §5 for additional discussion).

Original: The supervision complexity Y ⊤ K − 1 Y is small whenever Y is aligned with top eigenvectors of K and/or Y has small scale.
Revised: The supervision complexity Y ⊤ K − 1 Y is small whenever Y is aligned with top eigenvectors of K and/or Y has small scale.

Original: Furthermore, one cannot make the bound close to zero by just reducing the scale of targets, as one would need a small γ to control the margin loss that would otherwise increase due to student predictions getting closer to zero (as the student aims to match Y i ).
Revised: Furthermore, one cannot make the bound close to zero by just reducing the scale of targets, as one would need a small γ to control the margin loss that would otherwise increase due to student predictions getting closer to zero (as the student aims to match Y i ).

Original: To better understand the role of supervision complexity, it is instructive to consider two special cases that lead to a poor generalization bound: (1) uninformative features , and (2) uninformative labels .
Revised: To better understand the role of supervision complexity, it is instructive to consider two special cases that lead to a poor generalization bound: (1) uninformative features , and (2) uninformative labels .

Original: Complexity under uninformative features.
Revised: Complexity under uninformative features.

Original: Suppose the kernel matrix K is diagonal, so that the kernel provides no information on example-pair similarity; i.e., the kernel is “uninformative”.
Revised: Suppose the kernel matrix K is diagonal, so that the kernel provides no information on example-pair similarity; i.e., the kernel is “uninformative”.

Original: An application of Cauchy-Schwarz reveals the key expression in the second term in (5) satisfies:n
Revised: An application of Cauchy-Schwarz reveals the key expression in the second term in (5) satisfies:

Original: [Equation]
Revised: [Equation]

Original: Consequently, this term is least constant in order, and does not vanish as n → ∞ .
Revised: Consequently, this term is least constant in order, and does not vanish as n → ∞ .

Original: Complexity under uninformative labels.
Revised: Complexity under uninformative labels.

Original: Suppose the labels Y i are purely random, and independent from inputs X i .
Revised: Suppose the labels Y i are purely random, and independent from inputs X i .

Original: Conditioned on { X i } , Y ⊤ K − 1 Y concentrates around its mean by the
Revised: Conditioned on { X i } , Y ⊤ K − 1 Y concentrates around its mean by the

Original: Hanson-Wright inequality (Vershynin, 2018).
Revised: Hanson-Wright inequality (Vershynin, 2018).

Original: Hence, ∃ ϵ ( K, δ, n ) such that with probability ≥ 1 − δ ,Y ⊤ K [Equation]
Revised: Hence, ∃ ϵ ( K, δ, n ) such that with probability ≥ 1 − δ ,

Original: Y  ≥ E [Equation] Y i } (cid:2)Y [Equation]
Revised: Y ⊤ K − 1 Y ≥ E { Y i } (cid:2)

Original:  K [Equation] Y (cid:3)
Revised: Y ⊤ K − 1 Y (cid:3)

Original: [Equation] Y
Revised: − ϵ

Original: [Equation]
Revised: = E (cid:2)

Original: Tr [Equation]
Revised: Y 21 (cid:3)

Original:  K [Equation] 1 ) − ϵ .
Revised: Tr( K − 1 ) − ϵ .

Original: Thus, with the same probability,
Revised: Thus, with the same probability,

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is by Cauchy-Schwarz.
Revised: where the last inequality is by Cauchy-Schwarz.

Original: For sufficiently large n , the quantity E (cid:2) Y 21 (cid:3) ndominates ϵ Tr ( K ) , rendering the bound of Thm.
Revised: For sufficiently large n , the quantity E (cid:2) Y 21 (cid:3) ndominates ϵ Tr ( K ) , rendering the bound of Thm.

Original: 3 close to a constant.
Revised: 3 close to a constant.

Original: 2. E XTENSIONS : MULTICLASS CLASSIFICATION AND NEURAL NETWORKS
Revised: 2. E XTENSIONS : MULTICLASS CLASSIFICATION AND NEURAL NETWORKS

Original: We now show that a result similar to Thm.
Revised: We now show that a result similar to Thm.

Original: 3 holds for multiclass classification as well.
Revised: 3 holds for multiclass classification as well.

Original: In addition, we also discuss how our results are instructive about the behavior of neural networks.
Revised: In addition, we also discuss how our results are instructive about the behavior of neural networks.

Original: Extension to multiclass classification.
Revised: Extension to multiclass classification.

Original: Let { ( X i , Y i ) } i ∈
Revised: Let { ( X i , Y i ) } i ∈

Original: [ n ] be drawn i.i.d.
Revised: [ n ] be drawn i.i.d.

Original: from a distribution over
Revised: from a distribution over

Original: X × Y , where Y ⊂ R d .
Revised: X × Y , where Y ⊂ R d .

Original: Let k : X × X
Revised: Let k : X × X

Original: → R d × d be a matrix-valued positive definite kernel and H be the corresponding vector-valued RKHS.
Revised: → R d × d be a matrix-valued positive definite kernel and H be the corresponding vector-valued RKHS.

Original: As in the binary classification case, we consider a kernel problem in Eq.
Revised: As in the binary classification case, we consider a kernel problem in Eq.

Original: Let Y ⊤ =
Revised: Let Y ⊤ =

Original: ( Y ⊤ 1 , . . .
Revised: ( Y ⊤ 1 , . . .

Original: , Y ⊤ n ) and K be the kernel matrix of training examples:
Revised: , Y ⊤ n ) and K be the kernel matrix of training examples:

Original: [Equation]
Revised: [Equation]

Original:  For f : X → R d and a labeled example ( x, y ) , let ρ f ( x, y )
Revised: [Equation] For f : X → R d and a labeled example ( x, y ) , let ρ f ( x, y )

Original: =
Revised: =

Original: f ( x ) y − max y ′ ̸
Revised: f ( x ) y − max y ′ ̸

Original: = y f ( x ) y ′ be the prediction margin .
Revised: = y f ( x ) y ′ be the prediction margin .

Original: Then, the following analogue of Thm. 3 holds (see Appendix C for the proof).
Revised: Then, the following analogue of Thm. 3 holds (see Appendix C for the proof).

Original: Theorem 4.
Revised: Theorem 4.

Original: Assume that κ = sup x ∈X ,y ∈
Revised: Assume that κ = sup x ∈X ,y ∈

Original: [ d ] k ( x, x ) y,y < ∞ , and K is full rank almost surely.
Revised: [ d ] k ( x, x ) y,y < ∞ , and K is full rank almost surely.

Original: Further, assume that ℓ ( y, y ′ ) ≥ 0 , ∀ y, y ′ ∈ Y and ℓ ( y, y ) =
Revised: Further, assume that ℓ ( y, y ′ ) ≥ 0 , ∀ y, y ′ ∈ Y and ℓ ( y, y ) =

Original: 0 , ∀ y ∈ Y .
Revised: 0 , ∀ y ∈ Y .

Original: Let M 

Revised: Let M

Original: =
Revised: =

Original: ⌈ γ √ n/ (4 d √ κ ) ⌉ .
Revised: ⌈ γ √ n/ (4 d √ κ ) ⌉ .

Original: Then, with probability at least 1 − δ , for any solution f ∗ of problem in Eq.
Revised: Then, with probability at least 1 − δ , for any solution f ∗ of problem in Eq.

Original: [Equation]
Revised: [Equation]

Original: Implications for neural classifiers .
Revised: Implications for neural classifiers .

Original: Our analysis has so far focused on kernel-based classifiers.
Revised: Our analysis has so far focused on kernel-based classifiers.

Original: While neural networks are not exactly kernel methods, many aspects of their performance can be understood via a corresponding linearized neural network (see Ortiz-Jim´enez et al.
Revised: While neural networks are not exactly kernel methods, many aspects of their performance can be understood via a corresponding linearized neural network (see Ortiz-Jim´enez et al.

Original: and references therein).
Revised: and references therein).

Original: We follow this approach, and given a neural network f θ with current weights θ 0 , we consider the corresponding linearized neural network comprising the linear terms of the Taylor expansion of f θ ( x ) around θ 

Revised: We follow this approach, and given a neural network f θ with current weights θ 0 , we consider the corresponding linearized neural network comprising the linear terms of the Taylor expansion of f θ ( x ) around θ

Original: (Jacot et al., 2018; Lee et al., 2019):
Revised: (Jacot et al., 2018; Lee et al., 2019):

Original: f lin θ ( x ) ≜ f θ 0 ( x )+ ∇ θ f θ 

Revised: f lin θ ( x ) ≜ f θ 0 ( x )+ ∇ θ f θ

Original: ( x ) ⊤ ( θ − θ 0 ) .
Revised: ( x ) ⊤ ( θ − θ 0 ) .

Original: Let ω ≜ θ − θ 0 .
Revised: Let ω ≜ θ − θ 0 .

Original: This network f lin ω ( x ) is a linear function with respect to the parameters ω , but is generally non-linear with respect to the input x .
Revised: This network f lin ω ( x ) is a linear function with respect to the parameters ω , but is generally non-linear with respect to the input x .

Original: Note that ∇ θ f θ 

Revised: Note that ∇ θ f θ

Original: ( x ) acts as a feature representation, and induces the neural tangent kernel ( NTK) k 0 ( x, x ′ )
Revised: ( x ) acts as a feature representation, and induces the neural tangent kernel ( NTK

Original: [-= ∇ θ f θ 0 ( x ) ⊤ ∇ θ f θ 
-]
Revised: [-= ∇ θ f θ 0 ( x ) ⊤ ∇ θ f θ 
-][Equation]

Original: ( x ′ ) ∈ R d × d .
Revised: x [Equation]

Original: Given a labeled dataset S = { ( x i , y i ) } i ∈ [ n ] and a loss function L ( f ; S ) , the dynamics of gradient flow with learning rate η > 0 for f lin ω can be fully characterized in the function space, and depends only on the predictions at θ 0 and the NTK k 0 :
Revised: Given a labeled dataset S = { ( x i , y i ) } i ∈ [ n ] and a loss function L ( f ; S ) , the dynamics of gradient flow with learning rate η > 0 for f lin ω can be fully characterized in the function space, and depends only on the predictions at θ 0 and the NTK k 0 :

Original: [Equation]
Revised: [Equation]

Original: where f ( x 1: n ) ∈ R nd denotes the concatenation of predictions on training examples and
Revised: where f ( x 1: n ) ∈ R nd denotes the concatenation of predictions on training examples and

Original: K 0 ( x ′ , x 1: n ) = ∇ θ f θ 0 ( x ′ )
Revised: K 0 ( x ′ , x 1: n ) = ∇ θ f θ 0 ( x ′ )

Original: ⊤ ∇ θ f θ 

Revised: ⊤ ∇ θ f θ

Original: ( x 1: n ) .
Revised: ( x 1: n ) .

Original: Lee et al.
Revised: Lee et al.

Original: (2019) show that as one increases the width of the network or when ( θ − θ 0 ) does not change much during training, the dynamics of the linearized and original neural network become close.
Revised: (2019) show that as one increases the width of the network or when ( θ − θ 0 ) does not change much during training, the dynamics of the linearized and original neural network become close.

Original: When f θ is sufficiently overparameterized and L is convex with respect to ω , then ω t converges to an interpolating solution.
Revised: When f θ is sufficiently overparameterized and L is convex with respect to ω , then ω t converges to an interpolating solution.

Original: Furthermore, for the mean squared error objective, the solution has the minimum norm.
Revised: Furthermore, for the mean squared error objective, the solution has the minimum Euclidean norm (Gunasekar et al., 2017).

Original: As the Euclidean norm of ω corresponds to norm of f lin ω ( x ) − f θ 0 ( x ) in the vectorvalued RKHS H corresponding to k 0 , training a linearized network to interpolation is equivalent to solving the following with a small λ > 0 :
Revised: As the Euclidean norm of ω corresponds to norm of f lin ω ( x ) − f θ 0 ( x ) in the vector-valued RKHS H corresponding to k 0 , training a linearized network to interpolation is equivalent to solving the following with a small λ > 0 :

Original: [Equation]
Revised: [Equation]

Original: Our generalization bounds (Thm. 3 and 4) apply tof θ 0 ( x ) + h ∗ {+with supervision complexity of residual targets y i − f θ 
+} ( x  ) , as shifting the hypothesis class { h ∈ H :
Revised: Therefore, the generalization bounds of Thm. 3 and 4 apply to h ∗ {+with supervision complexity of residual targets y i − f θ 
+} ( x i ) .

Original: {+However, we are interested in the performance of f θ 
 + h ∗ . As the proofs of these results rely on bounding the Rademacher complexity of hypothesis sets of form { h ∈ H :+} ∥ h ∥ ≤ M }  by a constant function does not change its empirical Rademacher complexity.
Revised: {+However, we are interested in the performance of f θ 
 + h ∗ . As the proofs of these results rely on bounding the Rademacher complexity of hypothesis sets of form { h ∈ H :+} ∥ h ∥ ≤ M } , and shifting a hypothesis set by a constant function does not change the Rademacher complexity (see Remark 7), these proofs can be easily modified to handle hypotheses shifted by the constant function f θ 0 .

Original:  K NOWLEDGE DISTILLATION : A SUPERVISION COMPLEXITY LENS
Revised:  K NOWLEDGE DISTILLATION : A SUPERVISION COMPLEXITY LENS

Original: We now turn to KD, and explore how supervision complexity affects student’s generalization.
Revised: We now turn to KD, and explore how supervision complexity affects student’s generalization.

Original: We show that student’s generalization depends on three terms: the teacher generalization , the student’s margin with respect to the teacher predictions, and the complexity of the teacher’s predictions.
Revised: We show that student’s generalization depends on three terms: the teacher generalization , the student’s margin with respect to the teacher predictions, and the complexity of the teacher’s predictions.

Original: 3. T RADE - OFF BETWEEN TEACHER ACCURACY , MARGIN , AND COMPLEXITY
Revised: 3. T RADE - OFF BETWEEN TEACHER ACCURACY , MARGIN , AND COMPLEXITY

Original: Consider the binary classification setting of §2, and a fixed teacher g : X → R that outputs a logit.
Revised: Consider the binary classification setting of §2, and a fixed teacher g : X → R that outputs a logit.

Original: Let { ( X i , Y ∗ i ) } i ∈
Revised: Let { ( X i , Y ∗ i ) } i ∈

Original: [ n ] be n i.i.d.
Revised: [ n ] be n i.i.d.

Original: labeled examples, where Y ∗ i ∈ {− 1 , 1 } denotes the ground truth labels.
Revised: labeled examples, where Y ∗ i ∈ {− 1 , 1 } denotes the ground truth labels.

Original: For temperature τ > 0 , let Y i ≜ 2 σ ( g ( X i ) /τ )
Revised: For temperature τ > 0 , let Y i ≜ 2 σ ( g ( X i ) /τ )

Original: − 1 ∈ [ − 1 , +1] denote the teacher’s soft predictions , for sigmoid function σ : z (cid:55)→ (1 + exp( − z ))
Revised: − 1 ∈ [ − 1 , +1] denote the teacher’s soft predictions , for sigmoid function σ : z (cid:55)→ (1 + exp( − z ))

Original: − 1 .
Revised: − 1 .

Original: Our key observation is: if the teacher predictions Y i are accurate enough and have significantly lower complexity compared to ground truth labels Y ∗ i , then a student kernel method (cf.
Revised: Our key observation is: if the teacher predictions Y i are accurate enough and have significantly lower complexity compared to ground truth labels Y ∗ i , then a student kernel method (cf.

Original: Eq.
Revised: Eq.

Original: (3)) trained with Y i can generalize better than the one trained with Y ∗ i .
Revised: (3)) trained with Y i can generalize better than the one trained with Y ∗ i .

Original: The following result quantifies the trade-off between teacher accuracy, student prediction margin, and teacher prediction complexity (see Appendix B.3 for a proof).
Revised: The following result quantifies the trade-off between teacher accuracy, student prediction margin, and teacher prediction complexity (see Appendix B.3 for a proof).

Original: Proposition 5.
Revised: Proposition 5.

Original: Assume that κ = sup x ∈X k ( x, x ) < ∞ and K is full rank almost surely, ℓ ( y, y ′ ) ≥ 0 , ∀ y, y ′ ∈ Y , and ℓ ( y, y )
Revised: Assume that κ = sup x ∈X k ( x, x ) < ∞ and K is full rank almost surely, ℓ ( y, y ′ ) ≥ 0 , ∀ y, y ′ ∈ Y , and ℓ ( y, y )

Original: = 0 , ∀ y ∈ Y .
Revised: = 0 , ∀ y ∈ Y .

Original: Let Y i
Revised: Let Y i

Original: and
Revised: and

Original: Y ∗ i be defined as above.
Revised: Y ∗ i be defined as above.

Original: Let
Revised: Let

Original: M 0 = ⌈ γ √ n/ (2 √ κ ) ⌉ .
Revised: M 0 = ⌈ γ √ n/ (2 √ κ ) ⌉ .

Original: Then, with probability at least 1 − δ , any solution f ∗ of problem (3) satisfies 
Revised: Then, with probability at least 1 − δ , any solution f ∗ of problem (3) satisfies [Equation]

Original: P X,Y ∗ ( Y ∗ f ∗ ( X ) ≤ 0)
Revised: [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: Note that a similar result is easy to establish for multiclass classification using Thm. 4.
Revised: Note that a similar result is easy to establish for multiclass classification using Thm. 4.

Original: The first term in the above accounts for the misclassification rate of the teacher.
Revised: The first term in the above accounts for the misclassification rate of the teacher.

Original: While this term is not irreducible (it is possible for a student to perform better than its teacher), generally a student performs worse that its teacher, especially when there is a significant teacher-student capacity gap.
Revised: While this term is not irreducible (it is possible for a student to perform better than its teacher), generally a student performs worse that its teacher, especially when there is a significant teacher-student capacity gap.

Original: The second term is student’s empirical margin loss w.r.t.
Revised: The second term is student’s empirical margin loss w.r.t.

Original: teacher predictions.
Revised: teacher predictions.

Original: This captures the price of making teacher predictions too soft.
Revised: This captures the price of making teacher predictions too soft.

Original: Intuitively, the softer (i.e., closer to zero) teacher predictions are, the harder it is for the student to learn the classification rule.
Revised: Intuitively, the softer (i.e., closer to zero) teacher predictions are, the harder it is for the student to learn the classification rule.

Original: The third term accounts for the supervision complexity and the margin parameter γ .
Revised: The third term accounts for the supervision complexity and the margin parameter γ .

Original: Thus, one has to choose γ carefully to achieve a good balance between empirical margin loss and margin-normalized supervision complexity.
Revised: Thus, one has to choose γ carefully to achieve a good balance between empirical margin loss and margin-normalized supervision complexity.

Original: The effect of temperature.
Revised: The effect of temperature.

Original: For a fixed margin parameter γ > 0 , increasing the temperature τ makes teacher’s predictions Y i softer.
Revised: For a fixed margin parameter γ > 0 , increasing the temperature τ makes teacher’s predictions Y i softer.

Original: On the one hand, the reduced scale decreases the supervision complexity Y ⊤ K − 1 Y .
Revised: On the one hand, the reduced scale decreases the supervision complexity Y ⊤ K − 1 Y .

Original: Moreover, we shall see that in the case of neural networks the complexity decreases even further due to Y becoming more aligned with top eigenvectors of K .
Revised: Moreover, we shall see that in the case of neural networks the complexity decreases even further due to Y becoming more aligned with top eigenvectors of K .

Original: On the other hand, the scale of predictions of the (possibly interpolating) student f ∗ will decrease too, increasing the empirical margin loss.
Revised: On the other hand, the scale of predictions of the (possibly interpolating) student f ∗ will decrease too, increasing the empirical margin loss.

Original: This suggests that setting the value of τ is not trivial: the optimal value can be different based on the kernel k and teacher logits g
Revised: This suggests that setting the value of τ is not trivial: the optimal value can be different based on the kernel k and teacher logits g

Original: ( X i ) .
Revised: ( X i ) .

Original: 3. F ROM O FFLINE TO O NLINE KNOWLEDGE DISTILLATION
Revised: 3. F ROM O FFLINE TO O NLINE KNOWLEDGE DISTILLATION

Original: We identified that supervision complexity plays a key role in determining the efficacy of a distillation procedure.
Revised: We identified that supervision complexity plays a key role in determining the efficacy of a distillation procedure.

Original: The supervision from a fully trained teacher model can prove to be very complex for a student model in an early stage of its training (Fig. 1c).
Revised: The supervision from a fully trained teacher model can prove to be very complex for a student model in an early stage of its training (Fig. 1c).

Original: This raises the question: is there value in providing progressively difficult supervision to the student ?
Revised: This raises the question: is there value in providing progressively difficult supervision to the student ?

Original: In this section, we describe a simple online distillation method, where the the teacher is updated during the student training.
Revised: In this section, we describe a simple online distillation method, where the the teacher is updated during the student training.

Original: Over the course of their training, neural models learn functions of increasing complexity (Kalimeris et al., 2019).
Revised: Over the course of their training, neural models learn functions of increasing complexity (Kalimeris et al., 2019).

Original: This provides a natural way to construct a set of teachers with varying prediction complexities.
Revised: This provides a natural way to construct a set of teachers with varying prediction complexities.

Original: Similar to Jin et al.
Revised: Similar to Jin et al.

Original: (2019), for practical considerations of not training the teacher and the student simultaneously, we assume the availability of teacher checkpoints over the course of its training.
Revised: (2019), for practical considerations of not training the teacher and the student simultaneously, we assume the availability of teacher checkpoints over the course of its training.

Original: Given m teacher checkpoints at times T = { t i } i ∈ [ m ] , during the t -th step of distillation, the student receives supervision from the teacher checkpoint at time min { t ′ ∈ T : t ′ > t } .
Revised: Given m teacher checkpoints at times T = { t i } i ∈ [ m ] , during the t -th step of distillation, the student receives supervision from the teacher checkpoint at time min { t ′ ∈ T : t ′ > t } .

Original: Note that the student is trained for the same number of epochs in total as in offline distillation.
Revised: Note that the student is trained for the same number of epochs in total as in offline distillation.

Original: Throughout the paper we use the term “online distillation” for this approach (cf.
Revised: Throughout the paper we use the term “online distillation” for this approach (cf.

Original: Algorithm 1 of Appendix A).
Revised: Algorithm 1 of Appendix A).

Original: Online distillation can be seen as guiding the student network to follow the teacher’s trajectory in function space (see Fig. 1).
Revised: Online distillation can be seen as guiding the student network to follow the teacher’s trajectory in function space (see Fig. 1).

Original: Given that NTK can be interpreted as a principled notion of example similarity and controls which examples affect each other during training (Charpiat et al., 2019), it is desirable for the student to have an NTK similar to that of its teacher at each time step.
Revised: Given that NTK can be interpreted as a principled notion of example similarity and controls which examples affect each other during training (Charpiat et al., 2019), it is desirable for the student to have an NTK similar to that of its teacher at each time step.

Original: To test whether online distillation also transfers NTK, we propose to measure similarity between the final student and final teacher NTKs.
Revised: To test whether online distillation also transfers NTK, we propose to measure similarity between the final student and final teacher NTKs.

Original: For computational efficiency we work with
Revised: For computational efficiency we work with

Original: NTK matrices corresponding to a batch of b examples ( bd × bd matrices).
Revised: NTK matrices corresponding to a batch of b examples ( bd × bd matrices).

Original: Explicit computation of even batch NTK matrices can be costly, especially when the number of classes d is large.
Revised: Explicit computation of even batch NTK matrices can be costly, especially when the number of classes d is large.

Original: We propose to view student and teacher batch NTK matrices (denoted by K f and K g respectively) as operators and measure their similarity by comparing their behavior on random vectors: sim( K f , K g ) =
Revised: We propose to view student and teacher batch NTK matrices (denoted by K f and K g respectively) as operators and measure their similarity by comparing their behavior on random vectors: sim( K f , K g ) =

Original: E v ∼N (0 ,I ) [ ⟨ K f v, K g v ⟩ / ( ∥ K f v ∥
Revised: E v ∼N (0 ,I ) [ ⟨ K f v, K g v ⟩ / ( ∥ K f v ∥

Original: ∥
Revised: ∥

Original: K g v ∥ )] .
Revised: K g v ∥ )] .

Original: Note that the cosine distance is used to account for scale differences of K g and K f .
Revised: Note that the cosine distance is used to account for scale differences of K g and K f .

Original: The kernel-vector products appearing in this similarity measure above can be computed efficiently without explicitly constructing the kernel matrices.
Revised: The kernel-vector products appearing in this similarity measure above can be computed efficiently without explicitly constructing the kernel matrices.

Original: For example, K f v = ∇ θ f θ ( x 1: b ) ⊤ ( ∇ θ f θ ( x 1: b ) v ) can be computed with one vector-Jacobian product followed by a Jacobian-vector product.
Revised: For example, K f v = ∇ θ f θ ( x 1: b ) ⊤ ( ∇ θ f θ ( x 1: b ) v ) can be computed with one vector-Jacobian product followed by a Jacobian-vector product.

Original: The former can be computed efficiently using backpropagation, while the latter can be computed efficiently using forward-mode differentiation.
Revised: The former can be computed efficiently using backpropagation, while the latter can be computed efficiently using forward-mode differentiation.

Original:  E XPERIMENTAL RESULTS
Revised:  E XPERIMENTAL RESULTS

Original: We now present experimental results to showcase the importance of supervision complexity in distillation, and to establish efficacy of online distillation. 
Revised: We now present experimental results to showcase the importance of supervision complexity in distillation, and to establish efficacy of online distillation. [Table] [Table]

Original: 4. E XPERIMENTAL SETUP
Revised: 4. E XPERIMENTAL SETUP

Original: We consider standard image classification benchmarks: CIFAR-10, CIFAR-100, and TinyImageNet.
Revised: We consider standard image classification benchmarks: CIFAR-10, CIFAR-100, and TinyImageNet.

Original: Additionally, we derive a binary classification task from CIFAR-100 by grouping the[Table] [Table] first and last 50 classes into two meta-classes.
Revised: Additionally, we derive a binary classification task from CIFAR-100 by grouping the first and last 50 classes into two meta-classes.

Original: We consider teacher and student architectures that are ResNets
Revised: We consider teacher and student architectures that are ResNets

Original: (He et al., 2016), VGGs (Simonyan & Zisserman, 2015), and MobileNets (Howard et al., 2019) of various depths.
Revised: (He et al., 2016), VGGs (Simonyan & Zisserman, 2015), and MobileNets (Howard et al., 2019) of various depths.

Original: As a student architecture with relatively weaker inductive biases, we also consider the LeNet-5 (LeCun et al., 1998) with 8 times wider hidden layers.
Revised: As a student architecture with relatively weaker inductive biases, we also consider the LeNet-5 (LeCun et al., 1998) with 8 times wider hidden layers.

Original: We use standard hyperparameters (Appendix A) to train these models.
Revised: We use standard hyperparameters (Appendix A) to train these models.

Original: We compare (1) regular one-hot training (without any distillation), (2) regular offline distillation using the temperature-scaled softmax crossentropy, and (3) online distillation using the same loss.
Revised: We compare (1) regular one-hot training (without any distillation), (2) regular offline distillation using the temperature-scaled softmax crossentropy, and (3) online distillation using the same loss.

Original: For CIFAR-10 and binary CIFAR-100, we also consider training with mean-squared error (MSE) loss and its corresponding KD loss: 
Revised: For CIFAR-10 and binary CIFAR-100, we also consider training with mean-squared error (MSE) loss and its corresponding KD loss: L

Original: [Equation]
Revised: [Equation]

Original: The MSE loss allows for interpolation in case of one-hot labels y i , making it amenable to the analysis in §2 and §3.
Revised: The MSE loss allows for interpolation in case of one-hot labels y i , making it amenable to the analysis in §2 and §3.

Original: Moreover, Hui & Belkin (2021) show that under standard training, the CE and MSE losses perform similarly; as we shall see, the same is true for distillation as well.
Revised: Moreover, Hui & Belkin (2021) show that under standard training, the CE and MSE losses perform similarly; as we shall see, the same is true for distillation as well.

Revised: As mentioned in Sec. 1, in all KD experiments student networks receive supervision only through a knowledge distillation loss (i.e., dataset labels are not used).

Revised: This choice help us decrease differences between the theory and experiments.

Revised: Furthermore, in our preliminary experiments we observed that this choice does not result in student performance degradation (see Appendix D).

Original: 4. R ESULTS AND DISCUSSION
Revised: 4. R ESULTS AND DISCUSSION

Original: Tables 1, 2, 3 and Table 5 (Appendix D) present the results (mean and standard deviation of test accuracy over 3 random trials).
Revised: Tables 1, 2, 3 and Table 5 (Appendix D) present the results (mean and standard deviation of test accuracy over 3 random trials).

Original: First, we see that online distillation with proper temperature scaling typically yields the most accurate student.
Revised: First, we see that online distillation with proper temperature scaling typically yields the most accurate student.

Original: The gains over regular distillation are particularly pronounced when there is a large teacher-student gap.
Revised: The gains over regular distillation are particularly pronounced when there is a large teacher-student gap.

Original: For example, on CIFAR-100, ResNet to LeNet distillation with temperature scaling appears to hit a limit of ∼ 60% accuracy.
Revised: For example, on CIFAR-100, ResNet to LeNet distillation with temperature scaling appears to hit a limit of ∼ 60% accuracy.

Original: Online distillation however manages to further increase accuracy by +6% , which is a ∼ 20% increase compared to standard training.
Revised: Online distillation however manages to further increase accuracy by +6% , which is a ∼ 20% increase compared to standard training.

Original: Second, the similar results on binary CIFAR-100 shows that “dark knowledge” in the form of membership information in multiple classes is not necessary for distillation to succeed.
Revised: Second, the similar results on binary CIFAR-100 shows that “dark knowledge” in the form of membership information in multiple classes is not necessary for distillation to succeed.

Original: The results also demonstrate that knowledge distillation with the MSE loss of (10) has a qualitatively similar behavior to KD with CE objective.
Revised: The results also demonstrate that knowledge distillation with the MSE loss of (10) has a qualitatively similar behavior to KD with CE objective.

Original: We use these MSE models to highlight the role of supervision complexity.
Revised: We use these MSE models to highlight the role of supervision complexity.

Original: As an instructive case, we consider a LeNet-5x8 network trained on binary
Revised: As an instructive case, we consider a LeNet-5x8 network trained on binary

Original: CIFAR-100 with the standard MSE loss function.
Revised: CIFAR-100 with the standard MSE loss function.

Original: For a given checkpoint of this network and a given set of m labeled ( test ) examples { ( X i , Y i ) } i ∈ [ m ] , we compute the adjusted supervision complexity 1 /n (cid:112)
Revised: For a given checkpoint of this network and a given set of m labeled ( test ) examples { ( X i , Y i ) } i ∈ [ m ] , we compute the adjusted supervision complexity 1 /n (cid:112)

Original: ( Y − f
Revised: ( Y − f

Original: ( X 1: m ))
Revised: ( X 1: m ))

Original: ⊤ K − 1 ( Y − f
Revised: ⊤ K − 1 ( Y − f

Original: ( X 1: m )) ·
Revised: ( X 1: m )) ·

Original: Tr ( K ) , where f denotes the current predictionfunction, and K is derived from the current NTK.
Revised: Tr ( K ) , where f denotes the current prediction

Original: Note that the subtraction of initial predictions is the appropriate way to measure complexity given the form of the optimization problem (9). As the training NTK matrix becomes aligned with dataset labels during training (see Baratin et al. (2021) and Fig. 7 of Appendix D), we pick { X i } i ∈ m to be a set of 2 12 test examples.
Revised: [Table]

Original: Comparison of supervision complexities. We compare the adjusted supervision complexities of random labels, dataset labels, and predictions of an offline and online ResNet-56 teacher predictions with respect to various checkpoints of the LeNet-5x8 network. The results presented in Fig. 1c indicate that the dataset labels and offline teacher predictions are as complex as random labels in
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: A d j u s t e d s u p e
Revised: A d j u s t e d s u p e

Original: r
Revised: r

Original: v
Revised: v

Original: i s i o n
Revised: i s i o n

Original: c o m p l e x i t y o v e r t a r g e t no r m
Revised: c o m p l e x i t y o v e r t a r g e t n

Original: [Figure]
Revised: [Figure]

Original: 20 Temperature 0.
Revised: 20 Temperature 0.

Original: A d j u s t e d s u p e
Revised: A d j u s t e d s u p e

Original: r
Revised: r

Original: v
Revised: v

Original: i s i o n
Revised: i s i o n

Original: c o m p l e x i t y 0.
Revised: c o m p l e x i t y 0.

Original: A d j u s t e d s u p e
Revised: A d j u s t e d s u p e

Original: r
Revised: r

Original: v
Revised: v

Original: i s i o n
Revised: i s i o n

Original: c o m p l e x i t y o v e r t a r g e tn
Revised: c o m p l e x i t y o v e r t a r g e t

Original: [Figure]
Revised: [Figure]

Original:  r m
Revised: o r m

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: A d j u s t e d s u p e
Revised: A d j u s t e d s u p e

Original: r
Revised: r

Original: v
Revised: v

Original: i s i o n
Revised: i s i o n

Original: [Figure]

Original:  o m p l e x i t y
Revised: c o m p l e x i t y

Original: [Figure]
Revised: [Figure]

Original:  the beginning.
Revised: Comparison of supervision complexities. We compare the adjusted supervision complexities of random labels, dataset labels, and predictions of an offline and online ResNet-56 teacher predictions with respect to various checkpoints of the LeNet-5x8 network. The results presented in Fig. 1c indicate that the dataset labels and offline teacher predictions are as complex as random labels in the beginning.

Original: After some initial decline, the complexity of these targets increases as the network starts to overfit.
Revised: After some initial decline, the complexity of these targets increases as the network starts to overfit.

Original: Given the lower bound on the supervision complexity of random labels (see §2), this increase means that the NTK spectrum becomes less uniform (see Fig.
Revised: Given the lower bound on the supervision complexity of random labels (see §2), this increase means that the NTK spectrum becomes less uniform (see Fig.

Original: 6 of Appendix D).
Revised: 6 of Appendix D).

Original: In contrast to these static targets, the complexity of the online teacher predictions smoothly increases, and is significantly smaller for most of the epochs.
Revised: In contrast to these static targets, the complexity of the online teacher predictions smoothly increases, and is significantly smaller for most of the epochs.

Original: To account for softness differences of the various targets, we consider plotting the adjusted supervision complexity normalized by the target norm √ m ∥
Revised: To account for softness differences of the various targets, we consider plotting the adjusted supervision complexity normalized by the target norm √ m ∥

Original: Y ∥ 2 .
Revised: Y ∥ 2 .

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2a, the normalized complexity of offline and online teacher predictions is smaller compared to the dataset labels, indicating a better alignment with top eigenvectors of the LeNet NTK.
Revised: 2a, the normalized complexity of offline and online teacher predictions is smaller compared to the dataset labels, indicating a better alignment with top eigenvectors of the LeNet NTK.

Original: Importantly, we see that the predictions of an online teacher have significantly lower normalized complexity in the critical early stages of training.
Revised: Importantly, we see that the predictions of an online teacher have significantly lower normalized complexity in the critical early stages of training.

Original: Similar observations hold when complexities are measured with respect to a ResNet-20 network (see Appendix D).
Revised: Similar observations hold when complexities are measured with respect to a ResNet-20 network (see Appendix D).

Original: Average teacher complexity.
Revised: Average teacher complexity.

Original: Ren et al.
Revised: Ren et al.

Original: (2022) observed that teacher predictions fluctuate over time, and showed that using exponentially averaged teachers improves knowledge distillation.
Revised: (2022) observed that teacher predictions fluctuate over time, and showed that using exponentially averaged teachers improves knowledge distillation.

Original: Fig.
Revised: Fig.

Original: 2c demonstrates that the supervision complexity of an online teacher predictions is always slightly larger than that of the average of predictions of teachers of the last 10 preceding epochs.
Revised: 2c demonstrates that the supervision complexity of an online teacher predictions is always slightly larger than that of the average of predictions of teachers of the last 10 preceding epochs.

Original: Effect of temperature scaling.
Revised: Effect of temperature scaling.

Original: As discussed earlier, higher temperature makes the teacher predictions softer, decreasing their norm.
Revised: As discussed earlier, higher temperature makes the teacher predictions softer, decreasing their norm.

Original: This has a large effect on supervision complexity (Fig. 2b).
Revised: This has a large effect on supervision complexity (Fig. 2b).

Original: Even when one controls for the norm of the predictions, the complexity still decreases (Fig. 2b).
Revised: Even when one controls for the norm of the predictions, the complexity still decreases (Fig. 2b).

Original: NTK similarity.
Revised: NTK similarity.

Original: Remarkably, we observe that across all of our experiments, the final test accuracy of the student is strongly correlated with the similarity of final teacher and student NTKs (see Fig.and Fig.
Revised: Remarkably, we observe that across all of our experiments, the final test accuracy of the student is strongly correlated with the similarity of final teacher and student NTKs (see Fig.and Fig.

Original: 8 of Appendix D).
Revised: 9 of Appendix D).

Original: This cannot be explained by better matching the teacher predictions.
Revised: This cannot be explained by better matching the teacher predictions.

Original:  In fact, we see that the final fidelity (the rate of classification agreement of a teacher-student pair) measured on training set has no clear relationship with test accuracy.
Revised: [Figure] In fact, we see that the final fidelity (the rate of classification agreement of a teacher-student pair) measured on training set has no clear relationship with test accuracy.

Original: Furthermore, we see that online
Revised: Furthermore, we see that online

Original: KD results in better NTK transfer without an explicit regularization loss enforcing suchtransfer.
Revised: KD results in better NTK transfer without an explicit regularization loss enforcing such

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Due to space constraints, we briefly discuss the existing works that are the most closely related to our exploration in this paper (see Appendix E for a more comprehensive account of related work.)
Revised: Due to space constraints

Original: [Figure]
Revised: [Figure]

Revised: we briefly discuss the existing works that are the most closely related to our exploration in this paper (see Appendix E for a more comprehensive account of related work.)

Original: Supervision complexity.
Revised: Supervision complexity.

Original: The key quantity in our work is supervision complexity Y ⊤ K − 1 Y .
Revised: The key quantity in our work is supervision complexity Y ⊤ K − 1 Y .

Original: Cristianini et al.
Revised: Cristianini et al.

Original: (2001) introduced a related quantity Y ⊤ K Y called kernel-target alignment (KTA) and derived a generalization bound with it for expected Parzen window classifiers.
Revised: (2001) introduced a related quantity Y ⊤ K Y called kernel-target alignment (KTA) and derived a generalization bound with it for expected Parzen window classifiers.

Original: Deshpande et al.
Revised: Deshpande et al.

Original: (2021) use KTA for model selection in transfer learning.
Revised: (2021) use KTA for model selection in transfer learning.

Original: Ortiz-Jim ´ enez et al.
Revised: Ortiz-Jim ´ enez et al.

Original: (2021) demonstrate that when NTK-target alignment is high, learning is faster and generalizes better.
Revised: (2021) demonstrate that when NTK-target alignment is high, learning is faster and generalizes better.

Original: Arora et al.
Revised: Arora et al.

Original: (2019) prove a generalization bound for overparameterized two-layer neural networks with NTK parameterization, trained with gradient flow.
Revised: (2019) prove a generalization bound for overparameterized two-layer neural networks with NTK parameterization, trained with gradient flow.

Original: Their bound is roughly (cid:0) Y ⊤ ( K ∞ )
Revised: Their bound is roughly (cid:0) Y ⊤ ( K ∞ )

Original: − 1 Y (cid:1) 1 / 2 / √ n , where K ∞ is the expected NTK matrix at a random initialization.
Revised: − 1 Y (cid:1) 1 / 2 / √ n , where K ∞ is the expected NTK matrix at a random initialization.

Original: Our bound of Thm.
Revised: Our bound of Thm.

Original: 3 can be seen as a generalization of this result for all kernel methods, including linearized neural networks of any depth and sufficient width, with the only difference of using the empirical NTK matrix.
Revised: 3 can be seen as a generalization of this result for all kernel methods, including linearized neural networks of any depth and sufficient width, with the only difference of using the empirical NTK matrix.

Original: Belkin et al.
Revised: Belkin et al.

Original: (2018) warns that bounds based on RKHS complexity of the learned function can fail to explain the good generalization of kernel methods under label noise.
Revised: (2018) warns that bounds based on RKHS complexity of the learned function can fail to explain the good generalization of kernel methods under label noise.

Original: Mobahi et al.
Revised: Mobahi et al.

Original: (2020) prove that for kernel methods with RKHS norm regularization, self-distillation increases regularization strength.
Revised: (2020) prove that for kernel methods with RKHS norm regularization, self-distillation increases regularization strength.

Original: Phuong & Lampert (2019), while studying self-distillation of deep linear networks, derive a bound of the transfer risk that depends on the distribution of the acute angle between teacher parameters and data points.
Revised: Phuong & Lampert (2019), while studying self-distillation of deep linear networks, derive a bound of the transfer risk that depends on the distribution of the acute angle between teacher parameters and data points.

Original: This is in spirit related to supervision complexity as it measures an “alignment ” between the distillation objective and data.
Revised: This is in spirit related to supervision complexity as it measures an “alignment ” between the distillation objective and data.

Original: Ji & Zhu (2020) extend this results to linearized neural networks, showing that ∆ ⊤ z K − 1 ∆ z , where ∆ z is the logit change during training, plays a key role in estimating the bound.
Revised: Ji & Zhu (2020) extend this results to linearized neural networks, showing that ∆ ⊤ z K − 1 ∆ z , where ∆ z is the logit change during training, plays a key role in estimating the bound.

Original: Their bound is qualitatively different than ours, and ∆ ⊤ z K − 1 ∆ z becomes ill-defined for hard labels.
Revised: Their bound is qualitatively different than ours, and ∆ ⊤ z K − 1 ∆ z becomes ill-defined for hard labels.

Original: Non-static teachers.
Revised: Non-static teachers.

Original: Multiple works consider various approaches that train multiple students simultaneously to distill either from each other or from an ensemble (see, e.g., Zhang et al., 2018;
Revised: Multiple works consider various approaches that train multiple students simultaneously to distill either from each other or from an ensemble (see, e.g., Zhang et al., 2018;

Original: Anil et al., 2018; Guo et al., 2020).
Revised: Anil et al., 2018; Guo et al., 2020).

Original: Zhou et al.
Revised: Zhou et al.

Original: (2018) and Shi et al.
Revised: (2018) and Shi et al.

Original: (2021) train teacher and student together while having a common architecture trunk and regularizing teacher predictions to close that of students, respectively.
Revised: (2021) train teacher and student together while having a common architecture trunk and regularizing teacher predictions to close that of students, respectively.

Original: Jin et al.
Revised: Jin et al.

Original: (2019) study route constrained optimization which is closest to the online distillation in §3.2.
Revised: (2019) study route constrained optimization which is closest to the online distillation in §3.2.

Original: They employ a few teacher checkpoints to perform a multi-round
Revised: They employ a few teacher checkpoints to perform a multi-round

Original: KD.
Revised: KD.

Original: We complement this line of work by highlighting the role of supervision complexity and by demonstrating that online distillation can be very powerful for students with weak inductive biases.F UTURE WORK
Revised: We complement this line of work by highlighting the role of supervision complexity and by demonstrating that online distillation can be very powerful for students with weak inductive biases.

Original: There are several potential directions for future work.

Original: Adaptive temperature scaling for online distillation, where the teacher predictions are smoothened so as to ensure low target complexity, is one such direction.

Original: Another avenue is to explore alternative ways to smoothen teacher prediction besides temperature scaling; e.g., can one perform sample-dependent scaling?

Original: There is large potential for improving online KD by making more informed choices for the frequency and positions of teacher checkpoints, and controlling how much the student is trained in between teacher updates.

Original: Finally, while we demonstrated that online distillation results in a better alignment with the teacher’s NTK matrix, understanding why this happens is an open and interesting problem.

Original: A H YPERPARAMETERS AND IMPLEMENTATION DETAILS
Revised: A H YPERPARAMETERS AND IMPLEMENTATION DETAILS

Original: [Table]
Revised: [Table]

Original: In all experiments we use stochastic gradient descent optimizer with 128 batch size and 0.9 Nesterov momentum.
Revised: In all experiments we use stochastic gradient descent optimizer with 128 batch size and 0.9 Nesterov momentum.

Original: The starting learning rates are presented in Table 4.
Revised: The starting learning rates are presented in Table 4.

Original: All models for CIFAR datasets are trained for 256 epochs, with a learning schedule that divides the learning rate by 10 at epochs 96, 192, and 224.
Revised: All models for CIFAR datasets are trained for 256 epochs, with a learning schedule that divides the learning rate by 10 at epochs 96, 192, and 224.

Original: All models for Tiny ImageNet are trained for 200 epochs, with a learning rate schedule that divides the learning rate by 10 at epochs 75 and 135.
Revised: All models for Tiny ImageNet are trained for 200 epochs, with a learning rate schedule that divides the learning rate by 10 at epochs 75 and 135.

Original: The learning rate is warmed-up linearly to its initial value in the first 10 and 5 epochs for CIFAR and Tiny ImageNet models respectively.
Revised: The learning rate is warmed-up linearly to its initial value in the first 10 and 5 epochs for CIFAR and Tiny ImageNet models respectively.

Original: All
Revised: All

Original: VGG and ResNet models use 2e-4 weight decay, while MobileNet models use 1e-5 weight decay.
Revised: VGG and ResNet models use 2e-4 weight decay, while MobileNet models use 1e-5 weight decay.

Original: The LeNet-5 uses ReLU activations.
Revised: The LeNet-5 uses ReLU activations.

Original: We use the CIFAR variants of ResNets in experiments with
Revised: We use the CIFAR variants of ResNets in experiments with

Original: CIFAR-10 or (binary) CIFAR-100 datasets.
Revised: CIFAR-10 or (binary) CIFAR-100 datasets.

Original: Tiny ImageNet examples are resized to 224x224 resolution to suit the original ResNet, VGG and MobileNet architectures.
Revised: Tiny ImageNet examples are resized to 224x224 resolution to suit the original ResNet, VGG and MobileNet architectures.

Original: In all experiments we use standard data augmentation – random cropping and random horizontal flip.
Revised: In all experiments we use standard data augmentation – random cropping and random horizontal flip.

Original: In all online learning methods we consider one teacher checkpoint per epoch.
Revised: In all online learning methods we consider one teacher checkpoint per epoch.

Original: B P ROOFS
Revised: B P ROOFS

Original: In this appendix we present the deferred proofs.
Revised: In this appendix we present the deferred proofs.

Original: We use the following definition of Rademacher complexity (Mohri et al., 2018).
Revised: We use the following definition of Rademacher complexity (Mohri et al., 2018).

Original: Definition 6 (Rademacher complexity) .
Revised: Definition 6 (Rademacher complexity) .

Original: Let G be a family of functions from Z to R , and Z 1 , . . .
Revised: Let G be a family of functions from Z to R , and Z 1 , . . .

Original: , Z n be n i.i.d.
Revised: , Z n be n i.i.d.

Original: examples from a distribution P on Z .
Revised: examples from a distribution P on Z .

Original: Then, the empirical Rademacher complexity of G with respect to ( Z 1 , . . .
Revised: Then, the empirical Rademacher complexity of G with respect to ( Z 1 , . . .

Original: , Z n ) is defined as
Revised: , Z n ) is defined as

Original: [Equation]
Revised: [Equation]

Original: where σ i are independent Rademacher random variables (i.e., uniform random variables taking values in {− 1 , 1 } ).
Revised: where σ i are independent Rademacher random variables (i.e., uniform random variables taking values in {− 1 , 1 } ).

Original: The Rademacher complexity of G is then defined as
Revised: The Rademacher complexity of G is then defined as

Original: [Equation]
Revised: [Equation]

Revised: Remark 7.

Revised: Shifting the hypothesis class G by a constant function does not change the empirical

Revised: Rademacher complexity:

Original: [Equation]
Revised: [Equation]

Original: Given the kernel classification setting described in Sec. 2.1, we first prove a slightly more general variant of a classical generalization gap bound in Bartlett & Mendelson (2002, Theorem 21).
Revised: Given the kernel classification setting described in Sec. 2. [Equation] we first prove a slightly more general variant of a classical generalization gap bound in Bartlett & Mendelson (2002, Theorem 21) [Equation]

Original: Theorem 7.
Revised: Theorem 8.

Original: Assume sup x ∈X k
Revised: Assume sup x ∈X k

Original: ( x, x ) < ∞ .
Revised: ( x, x ) < ∞ .

Original: Fix any constant M > 0 .
Revised: Fix any constant M > 0 .

Original: Then with probability at least 1 − δ , every function f ∈ H with ∥ f
Revised: Then with probability at least 1 − δ , every function f ∈ H with ∥ f

Original: ∥
Revised: ∥

Original: H ≤ M satisfies
Revised: H ≤ M satisfies

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let F = { f ∈ H : ∥ f ∥ ≤ M } and consider the following class of functions:
Revised: Let F = { f ∈ H : ∥ f ∥ ≤ M } and consider the following class of functions:

Original: [Equation]
Revised: [Equation]

Original: By the standard Rademacher complexity classification generalization bound (Mohri et al., 2018,
Revised: By the standard Rademacher complexity classification generalization bound (Mohri et al., 2018,

Original: Theorem 3.3), for any δ > 0 , with probability at least 1 − δ , the following holds for all f ∈ F :
Revised: Theorem 3.3), for any δ > 0 , with probability at least 1 − δ , the following holds for all f ∈ F :

Original: [Equation]
Revised: [Equation]

Original: Therefore, with probability at least 1 − δ , for all f ∈ F
Revised: Therefore, with probability at least 1 − δ , for all f ∈ F

Original: [Equation]
Revised: [Equation]

Original: To finish the proof, we upper bound (cid:98) R n ( G ) :
Revised: To finish the proof, we upper bound (cid:98) R n ( G ) :

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: where the third line is due to Ledoux & Talagrand (1991).
Revised: where the third line is due to Ledoux & Talagrand (1991).

Original: By Lemma 22 of Bartlett & Mendelson (2002), we thus conclude that
Revised: By Lemma 22 of Bartlett & Mendelson (2002), we thus conclude that

Original: [Equation]
Revised: [Equation]

Original: B. P ROOF OF P ROPOSITION
Revised: B. P ROOF OF P ROPOSITION

Original: Proof of Proposition 2.
Revised: Proof of Proposition 2.

Original: As K is a full rank matrix almost surely, then with probability 1 there exists a vector α ∈ R n , such that K α = Y .
Revised: As K is a full rank matrix almost surely, then with probability 1 there exists a vector α ∈ R n , such that K α = Y .

Original: Consider the function f ( x ) = (cid:80) n i ={+ni =
+}
Revised: Consider the function f ( x ) = (cid:80) {+ni =
+}

Original: α i k
Revised: α i k

Original: ( X i , x ) ∈ H .
Revised: ( X i , x ) ∈ H .

Original: Clearly, f ( X i ) =
Revised: Clearly, f ( X i ) =

Original: Y i , ∀ i ∈ [ n ] .
Revised: Y i , ∀ i ∈ [ n ] .

Original: Furthermore, ∥ f ∥ 2 H =
Revised: Furthermore, ∥ f ∥ 2 H =

Original: α ⊤ K α =
Revised: α ⊤ K α =

Original: Y ⊤ K − 1 Y .
Revised: Y ⊤ K − 1 Y .

Original: The existence of such f ∈ H with zero empirical loss and the assumptions on the loss function imply that any optimal solution of problem (3) has a norm at most Y ⊤ K − 1 Y .
Revised: The existence of such f ∈ H with zero empirical loss and the assumptions on the loss function imply that any optimal solution of problem (3) has a norm at most Y ⊤ K − 1 Y .

Original: B. P ROOF OF T HM .
Revised: B. P ROOF OF T HM .

Original: Proof.
Revised: Proof.

Original: To get a generalization bound for f ∗ it is tempting to use Thm. 7 with M = ∥
Revised: To get a generalization bound for f ∗ it is tempting to use Thm. 8 with M = ∥

Original: f ∗ ∥ .
Revised: f ∗ ∥ .

Original: However, ∥ f ∗ ∥ is a random variable depending on the training data and is an invalid choice for the constant
Revised: However, ∥ f ∗ ∥ is a random variable depending on the training data and is an invalid choice for the constant

Original: M .
Revised: M .

Original: This issue can be resolved by paying a small logarithmic penalty.
Revised: This issue can be resolved by paying a small logarithmic penalty.

Original: For any M ≥ M 0 = (cid:108) γ √ n 2 √ κ (cid:109) the bound of Thm. 7 is vacuous.
Revised: For any M ≥ M 0 = (cid:108) γ √ n 2 √ κ (cid:109) the bound of Thm. 8 is vacuous.

Original: Let us consider the set of integers
Revised: Let us consider the set of integers

Original: M = { 1 , 2 , . . .
Revised: M = { 1 , 2 , . . .

Original: , M 0 } and write Thm. 7 for each element of M with δ/M 0 failure probability.
Revised: , M 0 } and write Thm. 8 for each element of M with δ/M 0 failure probability.

Original: By union bound, we have that with probability at least 1 − δ , all instances of Thm. 7 with M chosen from M hold simultaneously.
Revised: By union bound, we have that with probability at least 1 − δ , all instances of Thm. 8 with M chosen from M hold simultaneously.

Original: If Y ⊤ K − 1 Y ≥ M 0 , then the desired bound holds trivially, as the right-hand side becomes at least 1.
Revised: If Y ⊤ K − 1 Y ≥ M 0 , then the desired bound holds trivially, as the right-hand side becomes at least 1.

Original: Otherwise, we set M = (cid:108)(cid:112)
Revised: Otherwise, we set M = (cid:108)(cid:112)

Original: Y ⊤ K − 1 Y (cid:109) ∈ M and consider the corresponding part of the union bound.
Revised: Y ⊤ K − 1 Y (cid:109) ∈ M and consider the corresponding part of the union bound.

Original: We thus have that with at least 1 − δ probability, every function f ∈ F with ∥ f ∥ ≤ M satisfies 
Revised: We thus have that with at least 1 − δ probability, every function f ∈ F with ∥ f ∥ ≤ M satisfies [Equation] (cid:114)

Original: [Equation]
Revised: [Equation]

Original: As by Prop. 2 any optimal solution f ∗ has norm at most (cid:112)
Revised: As by Prop. 2 any optimal solution f ∗ has norm at most (cid:112)

Original: Y ⊤ K − 1 Y and M ≤ (cid:112)
Revised: Y ⊤ K − 1 Y and M ≤ (cid:112)

Original: Y ⊤ K − 1 Y + 1 , we have with probability at least 1 − δ , 
Revised: Y ⊤ K − 1 Y + 1 , we have with probability at least 1 − δ , P [Equation]

Original: [Equation]
Revised: [Equation]

Original: B. P ROOF OF P ROP .
Revised: B. P ROOF OF P ROP .

Original: Prop. 5 is a simple corollary of Thm. 3.
Revised: Prop. 5 is a simple corollary of Thm. 3.

Original: Proof.
Revised: Proof.

Original: We have that
Revised: We have that

Original: [Equation]
Revised: [Equation]

Original: The rest follows from bounding P X ( g ( X ) f ∗ ( X ) ≤ 0) using Thm. 3.
Revised: The rest follows from bounding P X ( g ( X ) f ∗ ( X ) ≤ 0) using Thm. 3.

Original: C E XTENSION TO MULTICLASS CLASSIFICATION
Revised: C E XTENSION TO MULTICLASS CLASSIFICATION

Original: Let us now consider the case of multiclass classification with d classes.
Revised: Let us now consider the case of multiclass classification with d classes.

Original: Let k : X
Revised: Let k : X

Original: ×
Revised: ×

Original: X
Revised: X

Original: → R d × d be a matrix-valued positive definite kernel.
Revised: → R d × d be a matrix-valued positive definite kernel.

Original: For every x ∈ X and a ∈ R d , let k x a = k ( · , x ) a be the function from X to R d defined the following way:
Revised: For every x ∈ X and a ∈ R d , let k x a = k ( · , x ) a be the function from X to R d defined the following way:

Original: [Equation]
Revised: [Equation]

Original: With any such kernel k there is a unique vector-valued RKHS H of functions from X to R d .
Revised: With any such kernel k there is a unique vector-valued RKHS H of functions from X to R d .

Original: This
Revised: This

Original: RKHS is the completion of span (cid:8) k x a : x ∈ X , a ∈ R d (cid:9) , with the following inner product:
Revised: RKHS is the completion of span (cid:8) k x a : x ∈ X , a ∈ R d (cid:9) , with the following inner product:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: For any f ∈ H , the norm ∥ f ∥ H is defined as (cid:112) ⟨ f, f ⟩ H .
Revised: For any f ∈ H , the norm ∥ f ∥ H is defined as (cid:112) ⟨ f, f ⟩ H .

Original: Therefore, if f ( x ) = (cid:80) ni
Revised: Therefore, if f ( x ) = (cid:80) ni

Original: =

Revised: =

Original: k x i a i then
Revised: k x i a i then

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: where a ⊤ = ( a ⊤ 1 , . .
Revised: where a ⊤ = ( a ⊤ 1 , . .

Original: . , a ⊤ n ) ∈ R nd and
Revised: . , a ⊤ n ) ∈ R nd and

Original: [Equation]
Revised: [Equation]

Original:  Suppose { ( X i , Y i ) } i ∈ [ n ] are n i.i.d.
Revised: = [Equation] Suppose { ( X i , Y i ) } i ∈ [ n ] are n i.i.d.

Original: examples sampled from some probability distribution on X ×Y , where Y ⊂ R d .
Revised: examples sampled from some probability distribution on X ×Y , where Y ⊂ R d .

Original: As in the binary classification case, we consider the regularized kernel problem (3).
Revised: As in the binary classification case, we consider the regularized kernel problem (3).

Original: Let Y ⊤ =
Revised: Let Y ⊤ =

Original: ( Y ⊤ 1 , . . .
Revised: ( Y ⊤ 1 , . . .

Original: , Y ⊤ n ) be the concatenation of targets.
Revised: , Y ⊤ n ) be the concatenation of targets.

Original: The following proposition is the analog of Prop. 2 in this vector-valued setting.
Revised: The following proposition is the analog of Prop. 2 in this vector-valued setting.

Original: Proposition 8.
Revised: Proposition 9.

Original: Assume K is full rank almost surely.
Revised: Assume K is full rank almost surely.

Original: Assume also ℓ ( y, y ′ ) ≥ 0 , ∀ y, y ′ ∈ Y , and ℓ ( y, y )
Revised: Assume also ℓ ( y, y ′ ) ≥ 0 , ∀ y, y ′ ∈ Y , and ℓ ( y, y )

Original: = 0 , ∀ y ∈ Y .
Revised: = 0 , ∀ y ∈ Y .

Original: Then, with probability 1, for any solution f ∗ of (3) , we have that
Revised: Then, with probability 1, for any solution f ∗ of (3) , we have that

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Proof.
Revised: Proof.

Original: With probability 1, the kernel matrix K is full rank.
Revised: With probability 1, the kernel matrix K is full rank.

Original: Therefore, there exists a vector a ⊤ =
Revised: Therefore, there exists a vector a ⊤ =

Original: ( a ⊤ 1 , . .
Revised: ( a ⊤ 1 , . .

Original: . , a ⊤ n ) ∈ R nd , with a i ∈ R d , such that K a = Y .
Revised: . , a ⊤ n ) ∈ R nd , with a i ∈ R d , such that K a = Y .

Original: Consider the function f ( x ) = (cid:80)
Revised: Consider the function f ( x ) = (cid:80)

Original: ni
Revised: ni

Original: =1 k X i a i ∈ H .
Revised: =1 k X i a i ∈ H .

Original: Clearly, f ( X i ) =
Revised: Clearly, f ( X i ) =

Original: Y i , ∀ i ∈ [ n ] .
Revised: Y i , ∀ i ∈ [ n ] .

Original: Furthermore,
Revised: Furthermore,

Original: [Equation]
Revised: [Equation]

Original: [Equation] The existence of such f ( x ) ∈ H with zero empirical loss and assumptions on the loss function imply that any optimal solution of problem (3) has a norm at most Y ⊤ K − 1 Y .
Revised: The existence of such f ( x ) ∈ H with zero empirical loss and assumptions on the loss function imply that any optimal solution of problem (3) has a norm at most Y ⊤ K − 1 Y .

Original: As in the main text, for a hypothesis f : X → R d and a labeled example ( x, y ) , let ρ f ( x, y )
Revised: As in the main text, for a hypothesis f : X → R d and a labeled example ( x, y ) , let ρ f ( x, y )

Original: =
Revised: =

Original: f ( x ) y − max y ′ ̸
Revised: f ( x ) y − max y ′ ̸

Original: = y f ( x ) y ′ be the prediction margin .
Revised: = y f ( x ) y ′ be the prediction margin .

Original: We now restate Thm. 4 and present a proof.
Revised: We now restate Thm. 4 and present a proof.

Original: Theorem 9 (Thm. 4 restated) .
Revised: Theorem 10 (Thm. 4 restated) .

Original: Assume that κ = sup x ∈X ,y ∈
Revised: Assume that κ = sup x ∈X ,y ∈

Original: [ d ] k ( x, x ) y,y < ∞ , and K is full rank almost surely.
Revised: [ d ] k ( x, x ) y,y < ∞ , and K is full rank almost surely.

Original: Further, assume that ℓ ( y, y ′ ) ≥ 0 , ∀ y, y ′ ∈ Y and ℓ ( y, y ) = 0 , ∀ y ∈ Y .
Revised: Further, assume that ℓ ( y, y ′ ) ≥ 0 , ∀ y, y ′ ∈ Y and ℓ ( y, y ) = 0 , ∀ y ∈ Y .

Original: Let
Revised: Let

Original: M 0 = ⌈ γ √ n/ (4 d √ κ ) ⌉ .
Revised: M 0 = ⌈ γ √ n/ (4 d √ κ ) ⌉ .

Original: Then, with probability at least 1 − δ , for any solution f ∗ of problem in
Revised: Then, with probability at least 1 − δ , for any solution f ∗ of problem in

Original: Eq.
Revised: Eq.

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Consider the class of functions F = { f ∈ H :
Revised: Consider the class of functions F = { f ∈ H :

Original: ∥ f ∥ ≤ M } for some M > 0 .
Revised: ∥ f ∥ ≤ M } for some M > 0 .

Original: By Theoremof Kuznetsov et al.
Revised: By Theoremof Kuznetsov et al.

Original: , for any γ > 0 and δ > 0 , with probability at least 1 − δ , the following bound holds for all f ∈ F :
Revised: , for any γ > 0 and δ > 0 , with probability at least 1 − δ , the following bound holds for all f ∈ F :

Original: [Equation]
Revised: [Equation]

Original: where ˜ F = { ( x, y ) (cid:55)→ f ( x ) y : f ∈ F , y ∈
Revised: where ˜ F = { ( x, y ) (cid:55)→ f ( x ) y : f ∈ F , y ∈

Original: [ d ] } .
Revised: [ d ] } .

Original: Next we upper bound the empirical Rademacher complexity of ˜ F :
Revised: Next we upper bound the empirical Rademacher complexity of ˜ F :

Original: [Equation]
Revised: [Equation]

Original: Jensen’s inequality)
Revised: (cid:88) [Equation] (cid:13)(cid:13) (cid:13)(cid:13)(cid:13) [Equation]

Revised: H

Original: [Equation]
Revised: [Equation]

Revised:  (Jensen’s inequality) (40)

Original: [Equation]
Revised: [Equation]

Revised: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation] 
Revised: [Equation] [Equation] [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: The proof is concluded with the same reasoning of the proof of Thm. 
Revised: The proof is concluded with the same reasoning of the proof of Thm. 3.

Original: [Table] D A DDITIONAL RESULTS 
Revised: D A DDITIONAL RESULTS AND DISCUSSION

Original: In this appendix we present additional results  to support the main findings of this work.
Revised: In this appendix we present additional results and discussion to support the main findings of this work.

Original: 5 presents the comparison of standard training, offline distillation, and online distillation on
Revised: On early stopped teachers. Cho & Hariharan (2019) observe that sometimes offline KD works better with early stopped teachers. Such teachers have worse accuracy and perhaps results in a smaller student margin, but they also have a significantly smaller supervision complexity (see Fig. 2a), which provides a possible explanation for this phenomenon. Teaching students with weak inductive biases.

Revised: As we saw earlier, a fully trained teacher can have predictions as complex as random labels for a weak student at initialization.

Original: CIFAR-10.
Revised: This low alignment of student NTK and teacher predictions can result in memorization. In contrast, an early stopped teacher captures simple patterns and has a better alignment with the student NTK, allowing the student to learn these patterns in a generalizable fashion. This feature learning improves the student NTK and allows learning more complex patterns in future iterations.

Revised: We hypothesize that this is the mechanism that allows online distillation to outperform offline distillation in some cases.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: A d j u s t e d s u p e

Original: r v is i o n c o m p l e x i t y *
Revised: i

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure] A d j u s t e d s u p e r
Revised: CIFAR-10 results.

Original: v
Revised: Table 5 presents the comparison of standard training, offline distillation, and online distillation on CIFAR-10.

Original: i s i o n c o m p l e x i t y * o v e r t a r g e t n o r m [Figure]
Revised: We see that the results are qualitatively similar to CIFARresults.

Original: Comparison of supervision complexities.
Revised: Comparison of supervision complexities.

Original: In the main text, we have introduced the notion of adjusted supervision complexity , which for a given neural network f ( x ) with NTK k and a set of labeled examples { ( X i , Y i ) } i ∈ [ m ] is defined as:
Revised: In the main text, we have introduced the notion of adjusted supervision complexity , which for a given neural network f ( x ) with NTK k and a set of labeled examples { ( X i , Y i ) } i ∈ [ m ] is defined as:

Original: [Equation]
Revised: [Equation]

Original: As discussed earlier, the subtraction of initial predictions is the appropriate way to measure complexity given the form of the optimization problem (9).
Revised: As discussed earlier, the subtraction of initial predictions is the appropriate way to measure complexity given the form of the optimization problem (9).

Original: Nevertheless, it is meaningful to consider the following quantity as well:
Revised: Nevertheless, it is meaningful to consider the following quantity as well:

Original: [Equation]
Revised: [Equation]

Original: (44) in order to measure “alignment” of targets Y with the NTK k .
Revised: in order to measure “alignment” of targets Y with the NTK k .

Original: We call this quantity adjusted supervision complexity* .
Revised: We call this quantity adjusted supervision complexity* .

Original: We compare the adjusted supervision complexities* of random labels, dataset labels, and predictions of an offline and online ResNet-56 teacher predictions with respect to various checkpoints of the LeNet-5x8 network.
Revised: We compare the adjusted supervision complexities* of random labels, dataset labels, and predictions of an offline and online ResNet-56 teacher predictions with respect to various checkpoints of the LeNet-5x8 network.

Original: The results presented in Fig.
Revised: The results presented in Fig.

Original: 4 are remarkably similar to the results with adjusted supervision complexity (Fig. 1c and Fig. 2a).
Revised: 4 are remarkably similar to the results with adjusted supervision complexity (Fig. 1c and Fig. 2a).

Original: We therefore, focus only on adjusted supervision complexity of (43) when comparing various targets.
Revised: We therefore, focus only on adjusted supervision complexity of (46) when comparing various targets.

Original: The only other experiment where we compute adjusted supervision complexities* (i.e., without subtracting the current predictions from labels) is presented in Fig.
Revised: The only other experiment where we compute adjusted supervision complexities* (i.e., without subtracting the current predictions from labels) is presented in Fig.

Original: 7, where the goal is to demonstrate that training labels become aligned with the training NTK matrix over the course of training.
Revised: 7, where the goal is to demonstrate that training labels become aligned with the training NTK matrix over the course of training.

Original: Fig.
Revised: Fig.

Original: 5 presents the comparison of adjusted supervision complexities, but with respect to a ResNetnetwork, instead of a LeNet-5x8 network.
Revised: 5 presents the comparison of adjusted supervision complexities, but with respect to a ResNetnetwork, instead of a LeNet-5x8 network.

Original: Again, we see that in the early epochs dataset labels and offline teacher predictions are almost as complex as random labels.
Revised: Again, we see that in the early epochs dataset labels and offline teacher predictions are almost as complex as random labels.

Original: Unlike the case of the LeNet5x8 network, random labels, dataset labels, and offline teacher predictions do not exhibit a U-shaped behavior.
Revised: Unlike the case of the LeNet5x8 network, random labels, dataset labels, and offline teacher predictions do not exhibit a U-shaped behavior.

Original: As for the LeNet-5x8 network, the shape of these curves is in agreement with the behavior of the condition number of the NTK (Fig. 6).
Revised: As for the LeNet-5x8 network, the shape of these curves is in agreement with the behavior of the condition number of the NTK (Fig. 6).

Original: Importantly, we still observe that the complexity of  the online teacher predictions is significantly smaller compared to the other targets, even when we account for the norm of predictions.
Revised: Importantly, we still observe that the complexity of [Figure] [Figure] the online teacher predictions is significantly smaller compared to the other targets, even when we account for the norm of predictions.

Revised: The effect of frequency of teacher checkpoints.

Revised: As mentioned earlier, throughout this paper we used one teacher checkpoint per epoch.

Revised: While this served our goal of establishing efficacy of online distillation, this choice is prohibitive for large teacher networks.

Revised: To understand the effect of the frequency of teacher checkpoints, we conduct an experiment on CIFAR-100 with ResNetand LeNet-5x8 student with varying frequency of teacher checkpoints.

Revised: In particular, we consider checkpointing the teacher once in every { 1 , 2 , 4 , 8 , 16 , 32 , 64 , 128 } epochs.

Original: NTK similarity.
Revised: The results presented in Fig. 8 show that reducing the teacher checkpointing frequency to once in 16 epochs results in only a minor performance drop for online distillation with τ = 4 . On label supervision in KD.

Revised: So far in all distillation methods dataset labels were not used as an additional source of supervision for students.

Revised: However, in practice it is common to train a student with a convex combination of knowledge distillation and standard losses: (1 − α )

Revised: L ce + α L kd-ce .

Revised: To verify that the choice of α = 1 does not produce unique conclusions regarding efficacy of online distillation, we do experiments on CIFAR-100 with varying values of α .

Revised: The results presented in

Revised: Table 6 confirm our main conclusions on online distillation.

Revised: Furthermore, we observe that picking α = 1 does not result in significant degradation of student performance.

Original: Finally, Fig.
Revised: The effect of τ . To confirm that our main conclusions regarding online distillation do not depend on the temperature value, we present additional experiments on CIFAR-100 with τ = 2 in Table 7. NTK similarity. Fig.

Original: 8 presents additional evidence that (a) training NTK similarity of the final student and the teacher is correlated with the final student test accuracy; and (b) that online distillation manages to transfer teacher NTK better. 
Revised: 9 presents additional evidence that (a) training NTK similarity of the final student and the teacher is correlated with the final student test accuracy; and (b) that online distillation manages to transfer teacher NTK better. [Figure] p [Figure]

Revised: r v i s i

Revised: [Figure]

Original: E E XPANDED DISCUSSION ON RELATED WORK
Revised: c [Figure] p l [Figure]

Revised: i

Revised: [Figure]

Revised: d j u s t e d s u p e

Revised: r

Revised: v

Revised: i s i o n

Revised: c o m p l

Revised: [Figure]

Revised: i

Revised: [Figure]

Revised: [Figure]

Original:  The key contributions of this work are the demonstration of the role of supervision complexity in student generalization, and the establishment of online knowledge distillation as a theoretically grounded and effective method.
Revised: [Table] [Table] E E XPANDED DISCUSSION ON RELATED WORK The key contributions of this work are the demonstration of the role of supervision complexity in student generalization, and the establishment of online knowledge distillation as a theoretically grounded and effective method.

Original: Both supervision complexity and online distillation have a number of relevant precedents in the literature that are worth comment.
Revised: Both supervision complexity and online distillation have a number of relevant precedents in the literature that are worth comment.

Original: Transferring knowledge beyond logits .
Revised: Transferring knowledge beyond logits .

Original: In the seminal works of Buciluˇa et al.
Revised: In the seminal works of Buciluˇa et al.

Original: ; Hinton et al.
Revised: ; Hinton et al.

Original: (2015) transferred “knowledge” is in the form of output probabilities.
Revised: (2015) transferred “knowledge” is in the form of output probabilities.

Original: Later works suggest other notions of “knowledge“ and other ways of transferring knowledge (Gou et al., 2021).These include [Figure] [Figure]
Revised: Later works suggest other notions of “knowledge“ and other ways of transferring knowledge (Gou et al., 2021).

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure] [Figure] activations of intermediate layers (Romero et al., 2015), attention maps (Zagoruyko & Komodakis, 2017), classifier head parameters (Chen et al., 2022), and various notions of example similarity (Passalis & Tefas, 2018; Park et al., 2019; Tung & Mori, 2019; Tian et al., 2020; He & Ozay, 2021).
Revised: These include activations of intermediate layers (Romero et al., 2015), attention maps (Zagoruyko & Komodakis, 2017), classifier head parameters (Chen et al., 2022), and various notions of example similarity (Passalis & Tefas, 2018; Park et al., 2019; Tung & Mori, 2019; Tian et al., 2020; He & Ozay, 2021).

Original: Transferring teacher NTK matrix belongs to this latter category of methods.
Revised: Transferring teacher NTK matrix belongs to this latter category of methods.

Original: Zhang et al.
Revised: Zhang et al.

Original: (2022) propose to transfer a low-rank approximation of a feature map corresponding the teacher NTK
Revised: (2022) propose to transfer a low-rank approximation of a feature map corresponding the teacher NTK.

Original: [Figure]

Original: Non-static teachers .
Revised: Non-static teachers .

Original: Some works on KD consider non-static teachers.
Revised: Some works on KD consider non-static teachers.

Original: In order to bridge teacherstudent capacity gap, Mirzadeh et al.
Revised: In order to bridge teacherstudent capacity gap, Mirzadeh et al.

Original: (2020) propose to perform a few rounds of distillation with teachers of increasing capacity.
Revised: (2020) propose to perform a few rounds of distillation with teachers of increasing capacity.

Original: In deep mutual learning (Zhang et al., 2018; Chen et al., 2020), codistillation (Anil et al., 2018), and collaborative learning (Guo et al., 2020), multiple students are trained simultaneously, distilling from each other or from an ensemble.
Revised: In deep mutual learning (Zhang et al., 2018; Chen et al., 2020), codistillation (Anil et al., 2018), and collaborative learning (Guo et al., 2020), multiple students are trained simultaneously, distilling from each other or from an ensemble.

Original: In Zhou et al.
Revised: In Zhou et al.

Original: (2018) and Shi et al.
Revised: (2018) and Shi et al.

Original: (2021), the teacher and the student are trained together.
Revised: (2021), the teacher and the student are trained together.

Original: In the former they have a common architecture trunk, while in the latter the teacher is penalized to keep its predictions close to the student’s predictions.
Revised: In the former they have a common architecture trunk, while in the latter the teacher is penalized to keep its predictions close to the student’s predictions.

Original: The closest method to the online distillation method of this work is route constrained optimization (Jin et al., 2019), where a few teacher checkpoints are selected for a multi-round distillation. 
Revised: The closest method to the online distillation method of this work is route constrained optimization (Jin et al., 2019), where a few teacher checkpoints are selected for a multi-round distillation. Rezagholizadeh et al.

Revised: (2022) employ a similar procedure but with an annealed temperature that decreases linearly with training time, followed by a phase of training with dataset labels only.

Original: The idea of distilling from checkpoints also appears in Yang et al.
Revised: The idea of distilling from checkpoints also appears in Yang et al.

Original: (2019), where a network is trained with a cosine learning rate schedule, simultaneously distilling from the checkpoint of the previous learning rate cycle.
Revised: (2019), where a network is trained with a cosine learning rate schedule, simultaneously distilling from the checkpoint of the previous learning rate cycle.

Original: Fundamental understanding of distillation .
Revised: Fundamental understanding of distillation .

Original: The effects of temperature, teacher-student capacity gap, optimization time, data augmentations, and other training details is non-trivial (Cho & Hariharan, 2019; Beyer et al., 2022; Stanton et al., 2021).
Revised: The effects of temperature, teacher-student capacity gap, optimization time, data augmentations, and other training details is non-trivial (Cho & Hariharan, 2019; Beyer et al., 2022; Stanton et al., 2021).

Original: It has been hypothesized and shown to some extent that teacher soft predictions capture class similarities, which is beneficial for the student (Hinton et al., 2015;
Revised: It has been hypothesized and shown to some extent that teacher soft predictions capture class similarities, which is beneficial for the student (Hinton et al., 2015;

Original: Furlanello et al., 2018; Tang et al., 2020).
Revised: Furlanello et al., 2018; Tang et al., 2020).

Original: Yuan et al.
Revised: Yuan et al.

Original: (2020) demonstrate that this softness of teacher predictions also has a regularization effect, similar to label smoothing.
Revised: (2020) demonstrate that this softness of teacher predictions also has a regularization effect, similar to label smoothing.

Original: Menon et al.
Revised: Menon et al.

Original: (2021) argue that teacher predictions are sometimes closer to the Bayes classifier than the hard labels of the dataset, reducing the variance of the training objective.
Revised: (2021) argue that teacher predictions are sometimes closer to the Bayes classifier than the hard labels of the dataset, reducing the variance of the training objective.

Original: The vanilla knowledge distillation loss also introduces some optimization biases.
Revised: The vanilla knowledge distillation loss also introduces some optimization biases.

Original: Mobahi et al.
Revised: Mobahi et al.

Original: (2020) prove that for kernel methods with RKHS norm regularization, self-distillation increases regularization strength, resulting in smaller norm RKHS norm solutions.
Revised: (2020) prove that for kernel methods with RKHS norm regularization, self-distillation increases regularization strength, resulting in smaller norm RKHS norm solutions.

Original: Phuong & Lampert (2019) prove that in a self-distillation setting, deep linear networks trained with gradient flow converge to the projection of teacher parameters into the data span, effectively recovering teacher parameters when the number of training points is large than the number of parameters.
Revised: Phuong & Lampert (2019) prove that in a self-distillation setting, deep linear networks trained with gradient flow converge to the projection of teacher parameters into the data span, effectively recovering teacher parameters when the number of training points is large than the number of parameters.

Original: They derive a bound of the transfer risk that depends on the distribution of the acute angle between teacher parameters and data points.
Revised: They derive a bound of the transfer risk that depends on the distribution of the acute angle between teacher parameters and data points.

Original: This is in spirit related to supervision complexity as it measures an “alignment ” between the distillation objective and data Ji & Zhu (2020) extend this results to linearized neural networks, showing that the quantity ∆ ⊤ z K − 1 ∆ z , where ∆ z is the logit change during training, plays a key role in estimating the bound.
Revised: This is in spirit related to supervision complexity as it measures an “alignment ” between the distillation objective and data Ji & Zhu (2020) extend this results to linearized neural networks, showing that the quantity ∆ ⊤ z K − 1 ∆ z , where ∆ z is the logit change during training, plays a key role in estimating the bound.

Original: The resulting bound is qualitatively different compared to ours, and the ∆ ⊤ z K − 1 ∆ z becomes ill-defined for hard labels.
Revised: The resulting bound is qualitatively different compared to ours, and the ∆ ⊤ z K − 1 ∆ z becomes ill-defined for hard labels.

Original: Supervision complexity .
Revised: Supervision complexity .

Original: The key quantity in our work is Y ⊤ K − 1 Y .
Revised: The key quantity in our work is Y ⊤ K − 1 Y .

Original: Cristianini et al.
Revised: Cristianini et al.

Original: (2001) introduced a related quantity Y ⊤ K Y called kernel-target alignment and derived a generalization bound with it for expected Parzen window classifiers.
Revised: (2001) introduced a related quantity Y ⊤ K Y called kernel-target alignment and derived a generalization bound with it for expected Parzen window classifiers.

Original: As an easy-to-compute proxy to supervision complexity, Deshpande et al.
Revised: As an easy-to-compute proxy to supervision complexity, Deshpande et al.

Original: (2021) use kernel-target alignment for model selection in transfer learning.
Revised: (2021) use kernel-target alignment for model selection in transfer learning.

Original: Ortiz-Jim ´ enez et al.
Revised: Ortiz-Jim´enez et al.

Original: (2021) demonstrate that when NTK-target alignment is high, learning is faster and generalizes better.
Revised: (2021) demonstrate that when NTK-target alignment is high, learning is faster and generalizes better.

Original: Arora et al.
Revised: Arora et al.

Original: (2019) prove a generalization bound for overparameterized two-layer neural networks with NTK parameterization trained with gradient flow.
Revised: (2019) prove a generalization bound for overparameterized two-layer neural networks with NTK parameterization trained with gradient flow.

Original: Their bound is approximately (cid:113) Y ⊤ ( K ∞ )
Revised: Their bound is approximately (cid:113) Y ⊤ ( K ∞ )

Original: − 1 Y / √ n , where K ∞ is the expected NTK matrix at a random initialization.
Revised: − 1 Y / √ n , where K ∞ is the expected NTK matrix at a random initialization.

Original: Our bound of Thm.
Revised: Our bound of Thm.

Original: 3 can be seen as a generalization of this result for all kernel methods, including linearized neural networks of any depth and sufficient width, with the only difference of using the empirical NTK matrix.
Revised: 3 can be seen as a generalization of this result for all kernel methods, including linearized neural networks of any depth and sufficient width, with the only difference of using the empirical NTK matrix.

Original: Belkin et al.
Revised: Belkin et al.

Original: (2018) warns that bounds based on RKHS complexity of the learned function can fail to explain the good generalization capabilities of kernel methods in presence of label noise. 
Revised: (2018) warns that bounds based on RKHS complexity of the learned function can fail to explain the good generalization capabilities of kernel methods in presence of label noise. Future work.

Revised: There are several potential directions for future work.

Revised: Adaptive temperature scaling for online distillation, where the teacher predictions are smoothened so as to ensure low target complexity, is one such direction.

Revised: Another avenue is to explore alternative ways to smoothen teacher prediction besides temperature scaling; e.g., can one perform sample-dependent scaling?

Revised: There is large potential for improving online KD by making more informed choices for the frequency and positions of teacher checkpoints, and controlling how much the student is trained in between teacher updates.

Revised: Finally, while we demonstrated that online distillation results in a better alignment with the teacher’s NTK matrix, understanding why this happens is an open and interesting problem.

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: Binary CIFAR-

Revised: [Figure]

Revised: LeNet

Revised: [Figure]

Revised: 5x8 student

Revised: [Figure]

Revised: [Figure]

Revised: (g) Tiny ImageNet

Revised: [Figure]

Revised: MobileNet- V3-35 student

Revised: [Figure]

Revised: MobileNet-V3-

Revised: [Figure]

Revised: ImageNet

Revised: [Figure]

Revised: MobileNet

Revised: [Figure]

Revised: 35 student

Revised: [Figure]

Revised: MobileNet

Revised: [Figure]

Revised: [Figure]

Revised: (i) Tiny ImageNet, VGG-16 stu- dent, ResNet-101 teacher

Revised: [Figure]