Original: F EATURE K ERNEL D ISTILLATION
Revised: F EATURE K ERNEL D ISTILLATION

Original: A BSTRACT
Revised: A BSTRACT

Original: Trained Neural Networks (NNs) can be viewed as data-dependent kernel machines, with predictions determined by the inner product of last-layer representations across inputs, referred to as the feature kernel .
Revised: Trained Neural Networks (NNs) can be viewed as data-dependent kernel machines, with predictions determined by the inner product of last-layer representations across inputs, referred to as the feature kernel .

Original: We explore the relevance of the feature kernel for Knowledge Distillation (KD), using a mechanistic understanding of an NN’s optimisation process.
Revised: We explore the relevance of the feature kernel for Knowledge Distillation (KD), using a mechanistic understanding of an NN’s optimisation process.

Original: We extend the theoretical analysis of Allen-Zhu & Li (2020) to show that a trained NN’s feature kernel is highly dependent on its parameter initialisation, which biases different initialisations of the same architecture to learn different data attributes in a multi-view data setting.
Revised: We extend the theoretical analysis of Allen-Zhu & Li (2020) to show that a trained NN’s feature kernel is highly dependent on its parameter initialisation, which biases different initialisations of the same architecture to learn different data attributes in a multi-view data setting.

Original: This enables us to prove that KD using only pairwise feature kernel comparisons can improve NN test accuracy in such settings, with both single & ensemble teacher models, whereas standard training without KD fails to generalise.
Revised: This enables us to prove that KD using only pairwise feature kernel comparisons can improve NN test accuracy in such settings, with both single & ensemble teacher models, whereas standard training without KD fails to generalise.

Original: We further use our theory to motivate practical considerations for improving student generalisation when using distillation with feature kernels, which allows us to propose a novel approach: Feature Kernel Distillation (FKD).
Revised: We further use our theory to motivate practical considerations for improving student generalisation when using distillation with feature kernels, which allows us to propose a novel approach: Feature Kernel Distillation (FKD).

Original: Finally, we experimentally corroborate our theory in the image classiﬁcation setting, showing that FKD is amenable to ensemble distillation, can transfer knowledge across datasets, and outperforms both vanilla KD & other feature kernel based KD baselines across a range of standard architectures & datasets.
Revised: Finally, we experimentally corroborate our theory in the image classiﬁcation setting, showing that FKD is amenable to ensemble distillation, can transfer knowledge across datasets, and outperforms both vanilla KD & other feature kernel based KD baselines across a range of standard architectures & datasets.

Original:  I NTRODUCTION & B ACKGROUND
Revised:  I NTRODUCTION & B ACKGROUND

Original: A prevailing belief in the Deep Learning community is that feature learning, where data-dependent features are acquired during training, is crucial to explaining the empirical success of Neural Networks (NNs)
Revised: A prevailing belief in the Deep Learning community is that feature learning, where data-dependent features are acquired during training, is crucial to explaining the empirical success of Neural Networks (NNs)

Original: (Fort et al., 2020; Baratin et al., 2021).
Revised: (Fort et al., 2020; Baratin et al., 2021).

Original: A comparison in this regard is often made to kernel methods (Jacot et al., 2018), which can be thought of as feature selection methods from a ﬁxed data-independent set of features.
Revised: A comparison in this regard is often made to kernel methods (Jacot et al., 2018), which can be thought of as feature selection methods from a ﬁxed data-independent set of features.

Original: This separation has been caricaturised as a distinction between feature learning and kernel learning regimes (Chizat et al., 2019; Yang & Hu, 2020; Woodworth et al., 2020) of NN training.
Revised: This separation has been caricaturised as a distinction between feature learning and kernel learning regimes (Chizat et al., 2019; Yang & Hu, 2020; Woodworth et al., 2020) of NN training.

Original: Though less amenable to theoretical analysis compared to kernel regimes, feature learning regimes have the capability to capture more of the complex empirical phenomenon that one can observe in NNs due to parameter-space non-convexity, such as: i) how ensembling trained NNs differing solely in their independent parameter initialisations can lead to improvements in predictive accuracy & uncertainty (Lakshminarayanan et al., 2017; Allen-Zhu & Li, 2020), or ii) the effectiveness of knowledge distillation with both single & ensemble teacher models (Buciluˇa et al., 2006; Hinton et al., 2015).
Revised: Though less amenable to theoretical analysis compared to kernel regimes, feature learning regimes have the capability to capture more of the complex empirical phenomenon that one can observe in NNs due to parameter-space non-convexity, such as: i) how ensembling trained NNs differing solely in their independent parameter initialisations can lead to improvements in predictive accuracy & uncertainty (Lakshminarayanan et al., 2017; Allen-Zhu & Li, 2020), or ii) the effectiveness of knowledge distillation with both single & ensemble teacher models (Buciluˇa et al., 2006; Hinton et al., 2015).

Original: This implies that in order to understand ensembling & knowledge distillation (KD) in NNs, we need to understand the mechanisms of NN feature learning regimes.
Revised: This implies that in order to understand ensembling & knowledge distillation (KD) in NNs, we need to understand the mechanisms of NN feature learning regimes.

Original: Ensembling can be loosely summarised as aggregating predictions from multiple models, & is used widely across machine learning (ML) to improve performance (Dietterich, 2000; Breiman, 2001).
Revised: Ensembling can be loosely summarised as aggregating predictions from multiple models, & is used widely across machine learning (ML) to improve performance (Dietterich, 2000; Breiman, 2001).

Original: Conversely, knowledge distillation (KD), the idea of transferring knowledge from a teacher model to a student model, has garnered the most attention with NNs.
Revised: Conversely, knowledge distillation (KD), the idea of transferring knowledge from a teacher model to a student model, has garnered the most attention with NNs.

Original: Remarkably, via KD it is possible to signiﬁcantly improve a single student’s generalisation with knowledge from a more powerful teacher, or an ensemble of teachers.
Revised: Remarkably, via KD it is possible to signiﬁcantly improve a single student’s generalisation with knowledge from a teacher model, or an ensemble of teachers.

Original: This means that the single student’s model has enough ﬂexibility to generalise well (relative to the teacher), thus one must factor in the optimisation process (such as the parameter initialisation) in order to explain the mechanisms of ensembling and KD in NNs.
Revised: This means that the single student’s model has enough ﬂexibility to generalise well (relative to the teacher), thus one must factor in the optimisation process (such as the parameter initialisation) in order to explain the mechanisms of ensembling and KD in NNs.

Original: To describe KD, suppose we have N input-target ( x , y ) ∈ R d × R C S data pairs ˆ D
Revised: To describe KD, suppose we have N input-target ( x , y ) ∈ R d × R C S data pairs ˆ D

Original: = { ( x i , y i ) } Ni = sampled i.i.d.
Revised: = { ( x i , y i ) } Ni = sampled i.i.d.

Original: (independent & identically distributed) from some distribution D , and a student NN architecture f S ( x ) = W S · h S ( x , θ S ) .
Revised: (independent & identically distributed) from some distribution D , and a student NN architecture f S ( x ) = W S · h S ( x , θ S ) .

Original: Here h S ( x , θ S ) ∈
Revised: Here h S ( x , θ S ) ∈

Original: R m × 1 is a student-speciﬁc feature extractor model (e.g. MLP, CNN, ResNet, or Transformer) with parameters θ S , and
Revised: R m × 1 is a student-speciﬁc feature extractor model (e.g. MLP, CNN, ResNet, or Transformer) with parameters θ S , and

Original: W S ∈ R C S × m is a parameter matrix for the last layer.
Revised: W S ∈ R C S × m is a parameter matrix for the last layer.

Original: Assume also that we have loss:
Revised: Assume also that we have loss:

Original: L ( θ S , W S )
Revised: L ( θ S , W S )

Original: = 1 N (cid:80) Ni
Revised: = 1 N (cid:80) Ni

Original: =
Revised: =

Original: L ( f S ( x i ) , y i ) which we seek to minimise over θ S , W S in the hope that f S can generalise to unseen ( x , y ) pairs.
Revised: L ( f S ( x i ) , y i ) which we seek to minimise over θ S , W S in the hope that f S can generalise to unseen ( x , y ) pairs.

Original: L is typically cross-entropy in the classiﬁcation setting.
Revised: L is typically cross-entropy in the classiﬁcation setting.

Original: Vanilla KD (Hinton et al., 2015) distils knowledge from a trained teacher network f T ( x )
Revised: Vanilla KD (Hinton et al., 2015) distils knowledge from a trained teacher network f T ( x )

Original: = W T · h T ( x , θ T ) ∈ R C T to a student by regularising student f S towards the teacher f T :
Revised: = W T · h T ( x , θ T ) ∈ R C T to a student by regularising student f S towards the teacher f T :

Original: [Equation]
Revised: [Equation]

Original: for temperature τ> 0 & regularisation λ KD > 0 hyperparameters.
Revised: for temperature τ> 0 & regularisation λ KD > 0 hyperparameters.

Original: Note, this is only valid if C T = C S .
Revised: Note, this is only valid if C T = C S .

Original: Following Hinton et al.
Revised: Following Hinton et al.

Original: , many methods have been proposed using different quirks of NNs to distil knowledge from teacher to student.
Revised: , many methods have been proposed using different quirks of NNs to distil knowledge from teacher to student.

Original: A relevant line of work involves encouraging the student to match how similar/related the teacher views two inputs x , x (cid:48) to be (Passalis & Tefas, 2018;
Revised: A relevant line of work involves encouraging the student to match how similar/related the teacher views two inputs x , x (cid:48) to be (Passalis & Tefas, 2018;

Original: Tung &
Revised: Tung &

Original: Mori, 2019; Park et al., 2019).
Revised: Mori, 2019; Park et al., 2019).

Original: These approaches have the beneﬁt of being agnostic to teacher/student architectures & prediction spaces C T & C S , but as of yet remain heuristically motivated.
Revised: These approaches have the beneﬁt of being agnostic to teacher/student architectures & prediction spaces C T & C S , but as of yet remain heuristically motivated.

Original: In this work, we explore such approaches under the more general framework of NN feature kernel (the kernel induced by the inner product of last-layer features h ) learning, allowing us to provide the missing theoretical justiﬁcation.
Revised: In this work, we explore such approaches under the more general framework of NN feature kernel (the kernel induced by the inner product of last-layer features h ) learning, allowing us to provide the missing theoretical justiﬁcation.

Original: Moreover, we use our theoretical insights to introduce practical improvements for FKD in Section 4, which we show outperform these previous works in Section 5.
Revised: Moreover, we use our theoretical insights to introduce practical improvements for FKD in Section 4, which we show outperform these previous works in Section 5.

Original: Allen-Zhu & Li (2020) provide the ﬁrst theoretical exposition of the mechanisms by which vanilla
Revised: Allen-Zhu & Li (2020) provide the ﬁrst theoretical exposition of the mechanisms by which vanilla

Original: KD and ensembling improve generalisation in NNs.
Revised: KD and ensembling improve generalisation in NNs.

Original: To this end, the authors introduce the notion of multi-view data, which is when a class in a multi-class classiﬁcation problem has multiple identifying features/attributes.
Revised: To this end, the authors introduce the notion of multi-view data, which is when a class in a multi-class classiﬁcation problem has multiple identifying features/attributes.

Original: For example, an image of a car can be discerned by i) wheels, ii) windows, or iii) headlights.
Revised: For example, an image of a car can be discerned by i) wheels, ii) windows, or iii) headlights.

Original: The key idea is that the NN parameter initialisation, and its random correlations with certain attributes, will bias the NN to learn only a subset of the entire set of attributes pertaining to a given class.
Revised: The key idea is that the NN parameter initialisation, and its random correlations with certain attributes, will bias the NN to learn only a subset of the entire set of attributes pertaining to a given class.

Original: When presented with single-view data lacking the class-identifying attribute that the
Revised: When presented with single-view data lacking the class-identifying attribute that the

Original: NN has learnt, the NN will not generalise.
Revised: NN has learnt, the NN will not generalise.

Original: For example, an NN that has learnt to classify cars based on if they have headlights will not generalise to a side-on image of a car that occludes headlights.
Revised: For example, an NN that has learnt to classify cars based on if they have headlights will not generalise to a side-on image of a car that occludes headlights.

Original: The implication then is that ensembling NNs works in part because independent parameter initialisations learn independent sets of attributes, so more data features will be learnt across the ensemble model.
Revised: The implication then is that ensembling NNs works in part because independent parameter initialisations learn independent sets of attributes, so more data features will be learnt across the ensemble model.

Original: Moreover, Allen-Zhu & Li (2020) argue that vanilla KD in NNs works because the features learnt by the teacher model (or models) are imparted to the student via soft teacher labels that capture ambiguity in a given data input (such as an image of a car whose headlights look like the eyes of a cat).
Revised: Moreover, it is argued that vanilla KD in NNs works because the features learnt by the teacher model (or models) are imparted to the student via soft teacher labels that capture ambiguity in a given data input (such as an image of a car whose headlights look like the eyes of a cat).

Original: This is fundamentally different to ensembling in strongly convex feature selection problems, such as linear or random features (Rahimi & Recht, 2007) models with (cid:96) 2 regularisation.
Revised: This is fundamentally different to ensembling in strongly convex feature selection problems, such as linear or random features (Rahimi & Recht, 2007) models with (cid:96) 2 regularisation.

Original: In such cases, different initialisations reach the same unique optimum, and additional noise must be added to ensure predictive diversity in the ensemble (Matthews et al., 2017).
Revised: In such cases, different initialisations reach the same unique optimum, and additional noise must be added to ensure predictive diversity in the ensemble (Matthews et al., 2017).

Original: These analyses suggest that it is not possible to fully explain KD or ensembling in NNs without feature learning, thus motivating our study of Feature Kernel Distillation , where one performs KD on NN features directly.
Revised: These analyses suggest that it is not possible to fully explain KD or ensembling in NNs without feature learning, thus motivating our study of Feature Kernel Distillation , where one performs KD on NN features directly.

Original: Our contributions Feature learning can be thought of as when the feature kernel , induced by the inner product of last-layer representations in a NN, changes during training (Yang & Hu, 2020), and kernel learning in NNs can be thought of as when this kernel is constant.
Revised: Our contributions Feature learning can be thought of as when the feature kernel , induced by the inner product of last-layer representations in a NN, changes during training (Yang & Hu, 2020), and kernel learning in NNs can be thought of as when this kernel is constant.

Original: In this work, we take a feature learning perspective of knowledge distillation (KD).
Revised: In this work, we take a feature learning perspective of knowledge distillation (KD).

Original: We ﬁrst highlight the importance of the feature kernel by viewing trained NNs as data-dependent kernel machines, & use this to motivate
Revised: We ﬁrst highlight the importance of the feature kernel by viewing trained NNs as data-dependent kernel machines, & use this to motivate

Original: Feature Kernel Distillation (FKD).
Revised: Feature Kernel Distillation (FKD).

Original: In FKD, we aim to ensure that the student’s feature kernel is well suited for improved generalisation, using both the teacher’s data-dependent feature kernel as well as an understanding of the student NN’s optimisation process.
Revised: In FKD, we aim to ensure that the student’s feature kernel is well suited for improved generalisation, using both the teacher’s data-dependent feature kernel as well as an understanding of the student NN’s optimisation process.

Original: In Section 3, we adapt the framework of Allen-Zhu & Li (2020) to show that FKD offers the same generalisation beneﬁts as found in vanilla KD in a multi-view data setting, and is further amenable to ensemble distillation.
Revised: In Section 3, we adapt the framework of Allen-Zhu & Li (2020) to show that FKD offers the same generalisation beneﬁts as found in vanilla KD in a multi-view data setting, and is further amenable to ensemble distillation.

Original: We then derive practical considerations from our insights in Section 4, to improve FKD through an understanding of the NN’s feature learning optimisation process, compared to previous methods which implicitly used the feature kernel for KD.
Revised: We then derive practical considerations from our insights in Section 4, to improve FKD through an understanding of the NN’s feature learning optimisation process, compared to previous methods which implicitly used the feature kernel for KD.

Original: Finally, in Section 5, we provide experimental support that our theoretical claims extend to standard image classiﬁcation settings, by: verifying that FKD is amenable to ensemble distillation; can transfer knowledge across datasets with different prediction spaces (unlike vanilla KD); and outperforms vanilla KD & previous feature kernel based distillation methods over a range of architectures on CIFAR-100 and ImageNet-1K.
Revised: Finally, in Section 5, we provide experimental support that our theoretical claims extend to standard image classiﬁcation settings, by: verifying that FKD is amenable to ensemble distillation; can transfer knowledge across datasets with different prediction spaces (unlike vanilla KD); and outperforms vanilla KD & previous feature kernel based distillation methods over a range of architectures on CIFAR-100 and ImageNet-1K.

Original:  M OTIVATION FOR FEATURE KERNEL DISTILLATION
Revised:  M OTIVATION FOR FEATURE KERNEL DISTILLATION

Original: [Figure]
Revised: [Figure]

Original: One obvious limitation of vanilla KD is that student f S and teacher f T need to share prediction spaces, i.e.
Revised: One obvious limitation of vanilla KD is that student f S and teacher f T need to share prediction spaces, i.e.

Original: C S = C T .
Revised: C S = C T .

Original: In many situations, we may have a teacher network trained on a dataset with a different number of classes than the student’s dataset, and it is not clear how one could apply vanilla knowledge distillation.
Revised: In many situations, we may have a teacher network trained on a dataset with a different number of classes than the student’s dataset, and it is not clear how one could apply vanilla knowledge distillation.

Original: One possibility could be to regularise directly in feature space by comparing h S and h T element-wise, but again this either requires same teacher-student last-layer sizes or additional projection layers.
Revised: One possibility could be to regularise directly in feature space by comparing h S and h T element-wise, but again this either requires same teacher-student last-layer sizes or additional projection layers.

Original: To eschew such unnecessary complications, we take the perspective of NNs as data-dependent kernel machines.
Revised: To eschew such unnecessary complications, we take the perspective of NNs as data-dependent kernel machines.

Original: Deﬁne an NN’s feature kernel to be:
Revised: Deﬁne an NN’s feature kernel to be:

Original: Deﬁnition 1 (Feature Kernel) .
Revised: Deﬁnition 1 (Feature Kernel) .

Original: Suppose we have parameters θ and last-layer NN feature extractor h ( · , θ ): R d (cid:55)→ R m .
Revised: Suppose we have parameters θ and last-layer NN feature extractor h ( · , θ ): R d (cid:55)→ R m .

Original: For two inputs x i , x j ∈ R m , the feature kernel k is the kernel induced by the inner product of h ( x i , θ ) and h ( x j , θ ) , that is: k
Revised: For two inputs x i , x j ∈ R m , the feature kernel k is the kernel induced by the inner product of h ( x i , θ ) and h ( x j , θ ) , that is: k

Original: ( x i , x j ) def = (cid:104) h
Revised: ( x i , x j ) def = (cid:104) h

Original: ( x i , θ ) , h ( x j , θ ) (cid:105) .
Revised: ( x i , θ ) , h ( x j , θ ) (cid:105) .

Original: At initialisation, it is well known that in the inﬁnite NN-width limit, with appropriate scaling, the feature kernel k converges almost surely to a deterministic kernel known as the Neural Network
Revised: At initialisation, it is well known that in the inﬁnite NN-width limit, with appropriate scaling, the feature kernel k converges almost surely to a deterministic kernel known as the Neural Network

Original: Gaussian Process (NNGP) kernel (Neal, 2012; Lee et al., 2018; Matthews et al., 2018; Yang, 2019).
Revised: Gaussian Process (NNGP) kernel (Neal, 2012; Lee et al., 2018; Matthews et al., 2018; Yang, 2019).

Original: Yang & Hu (2020) show that there is a parameterisation-dependent dichotomy between kernel & feature learning regimes for inﬁnite-width NNs, where the feature kernel k is constant or changes during training, respectively.
Revised: Yang & Hu (2020) show that there is a parameterisation-dependent dichotomy between kernel & feature learning regimes for inﬁnite-width NNs, where the feature kernel k is constant or changes during training, respectively.

Original: It has been widely demonstrated that a crucial component of the success of ﬁnite-width NNs is their ability to ﬂexibly learn features, and indeed the feature kernel, from data during training (Fort et al., 2020; Aitchison, 2020;
Revised: It has been widely demonstrated that a crucial component of the success of ﬁnite-width NNs is their ability to ﬂexibly learn features, and indeed the feature kernel, from data during training (Fort et al., 2020; Aitchison, 2020;

Original: Chen et al., 2020b; Maddox et al., 2021).
Revised: Chen et al., 2020b; Maddox et al., 2021).

Original: To see the importance of the feature kernel, note that for a ﬁxed θ with many common loss functions
Revised: To see the importance of the feature kernel, note that for a ﬁxed θ with many common loss functions

Original: L , and some mild assumptions on strong convexity (which could be enforced e.g. with standard (cid:96) 2 regularisation), the optimal W is uniquely determined and k determines the entire predictive function f ∗ ( · ) .
Revised: L , and some mild assumptions on strong convexity (which could be enforced e.g. with standard (cid:96) 2 regularisation), the optimal W is uniquely determined and k determines the entire predictive function f ∗ ( · ) .

Original: For example, with squared error, L ( f ( x ) , y )
Revised: For example, with squared error, L ( f ( x ) , y )

Original: = (cid:107) f
Revised: = (cid:107) f

Original: ( x ) − y (cid:107) 22 , and (cid:96) 2 regularisation strength λ > 0 , a trained NN is precisely kernel ridge regression with the data-dependent feature kernel k , whose job is to measure how similar different inputs are.
Revised: ( x ) − y (cid:107) 22 , and (cid:96) 2 regularisation strength λ > 0 , a trained NN is precisely kernel ridge regression with the data-dependent feature kernel k , whose job is to measure how similar different inputs are.

Original: Thus, all teacher knowledge is contained in its feature kernel, k T , so the feature kernel can act as our primary distillation target, as depicted in Fig.
Revised: Thus, all teacher knowledge is contained in its feature kernel, k T , so the feature kernel can act as our primary distillation target, as depicted in Fig.

Original: We show a corresponding result for cross-entropy loss in App. A.
Revised: We show a corresponding result for cross-entropy loss in App. A.

Original: Fig.
Revised: Fig.

Original: 2 corroborates our claims.
Revised: 2 corroborates our claims.

Original: For a ResNet20v1 (He et al., 2016) ‘reference’ model trained on CIFAR10 with cross entropy, we plot test class prediction confusion matrices between said model and: a i r p l a n e a u t o m o b il e s h
Revised: For a ResNet20v1 (He et al., 2016) ‘reference’ model trained on CIFAR10 with cross entropy, we plot test class prediction confusion matrices between said model and: a i r p l a n e a u t o m o b il e s h

Original: i p t
Revised: i p t

Original: r u c k c a t b i r d h
Revised: r u c k c a t b i r d h

Original: o r s e d ee r d o g f
Revised: o r s e d ee r d o g f

Original: r
Revised: r

Original: o g
Revised: o g

Original: Reference model prediction a i r p l a n e a u t o m o b il e s h
Revised: Reference model prediction a i r p l a n e a u t o m o b il e s h

Original: i p t
Revised: i p t

Original: r u c k c a t b i r d h
Revised: r u c k c a t b i r d h

Original: o r s e d ee r d o g f
Revised: o r s e d ee r d o g f

Original: r
Revised: r

Original: o g
Revised: o g

Original: A l t e r n a t e m o d e l p r e d i c t i o n vs Retrained last layer a i r p l a n e a u t o m o b il e s h
Revised: A l t e r n a t e m o d e l p r e d i c t i o n vs Retrained last layer a i r p l a n e a u t o m o b il e s h

Original: i p t
Revised: i p t

Original: r u c k c a t b i r d h
Revised: r u c k c a t b i r d h

Original: o r s e d ee r d o g f
Revised: o r s e d ee r d o g f

Original: r
Revised: r

Original: o g
Revised: o g

Original: Reference model prediction vs Independent init 

Revised: Reference model prediction vs Independent init

Original: N o r m a li s e d f r e q u e n c y i) a retrained version where all but the last-layer parameters are ﬁxed (hence fully determined by the reference model’s feature kernel as per App.
Revised: N o r m a li s e d f r e q u e n c y i) a retrained version where all but the last-layer parameters are ﬁxed (hence fully determined by the reference model’s feature kernel as per App.

Original: A), & ii) an independent model trained from a different initialisation.
Revised: A), & ii) an independent model trained from a different initialisation.

Original: As expected, there is signiﬁcantly more disagreement across test predictions for models with different initialisations than those which share feature kernels.
Revised: As expected, there is signiﬁcantly more disagreement across test predictions for models with different initialisations than those which share feature kernels.

Original: This suggests: a) different initialisations bias the same architecture to learn different features, and b) the feature kernel (largely) determines a model’s test predictions.
Revised: This suggests: a) different initialisations bias the same architecture to learn different features, and b) the feature kernel (largely) determines a model’s test predictions.

Original: Experimental details and a breakdown of the predictive disagreements can be found in App.
Revised: Experimental details and a breakdown of the predictive disagreements can be found in App.

Original: F.
Revised: F.

Original: Having described the feature kernel as a central object in any NN, we use this to motivate our proposed FKD, where we treat the teacher’s feature kernel, k T , as a key distillation target for the student’s feature kernel, k S .
Revised: Having described the feature kernel as a central object in any NN, we use this to motivate our proposed FKD, where we treat the teacher’s feature kernel, k T , as a key distillation target for the student’s feature kernel, k S .

Original: Encouraging similarity across feature kernels shares useful features that the teacher has learnt with the student, which we theoretically show in Section 3.
Revised: Encouraging similarity across feature kernels shares useful features that the teacher has learnt with the student, which we theoretically show in Section 3.

Original: We deﬁne the FKD student loss function via an additive regularisation term between feature kernels:
Revised: We deﬁne the FKD student loss function via an additive regularisation term between feature kernels:

Original: [Equation]
Revised: [Equation]

Original: where λ KD > 0 is the regularisation strength, D is some (pseudo-)distance over kernels, and the student feature kernel k S =
Revised: where λ KD > 0 is the regularisation strength, D is some (pseudo-)distance over kernels, and the student feature kernel k S =

Original: k θ is written to make explicit the dependence on student feature extractor parameters θ .
Revised: k θ is written to make explicit the dependence on student feature extractor parameters θ .

Original: We stress that Eq.
Revised: We stress that Eq.

Original: (2) does not require matching prediction (nor feature) spaces between teacher and student, allowing us to apply FKD across tasks, architectures, and datasets.
Revised: (2) does not require matching prediction (nor feature) spaces between teacher and student, allowing us to apply FKD across tasks, architectures, and datasets.

Original: We train θ using a stochastic optimisation algorithm such as stochastic gradient descent.
Revised: We consider Eq.

Original: For that reason, we require D to be amenable to stochastic mini-batching, therefore, in this work we choose
Revised: (2) with D set to:

Original: [Equation]
Revised: [Equation]

Original: where we can approximate the expectation with  an average of pairwise differences in a minibatch.
Revised: with expectation approximated by an average over a minibatch.

Revised: In this work we choose p = 2 , so that √ D gives the Frobenius norm of the difference in feature kernel gram matrices over a batch.

Original:  T HEORETICAL ANALYSIS FOR FKD
Revised:  T HEORETICAL ANALYSIS FOR FKD

Original: We now adapt the theoretical framework of Allen-Zhu & Li (2020), which is restricted to vanilla
Revised: We now adapt the theoretical framework of Allen-Zhu & Li (2020), which is restricted to vanilla

Original: KD, to demonstrate the generalisation beneﬁts of FKD over standard training.
Revised: KD, to demonstrate the generalisation beneﬁts of FKD over standard training.

Original: Note that FKD distils knowledge by comparing different data points, whereas vanilla KD compares a single data point across classes: this core difference is reﬂected throughout our analysis relative to Allen-Zhu &
Revised: Note that FKD distils knowledge by comparing different data points, whereas vanilla KD compares a single data point across classes: this core difference is reﬂected throughout our analysis relative to Allen-Zhu &

Original: Li (2020).
Revised: Li (2020).

Original: We ﬁrst describe the mutli-view data setting & CNN architecture we consider, before recalling that standard training without KD fails to generalise well.
Revised: We ﬁrst describe the multi-view data setting & CNN architecture we consider, before recalling that standard training without KD fails to generalise well.

Original: We then provide our main theoretical result, Theorem 2, which shows that FKD improves student test performance.
Revised: We then provide our main theoretical result, Theorem 2, which shows that FKD improves student test performance.

Original: Though our theoretical results are limited to a speciﬁc scenario, inspired by real-world data (Allen-Zhu & Li, 2020), & NN architecture, we believe the setup we consider is apt: it is simple enough to be tractable, yet rich enough to display the merits of FKD.
Revised: Though our theoretical results are limited to a speciﬁc scenario, inspired by real-world data (Allen-Zhu & Li, 2020), & NN architecture, we believe the setup we consider is apt: it is simple enough to be tractable, yet rich enough to display the merits of FKD.

Original: Moreover, we ﬁnd in Section 5 that our conclusions generalise to standard architectures & image datasets.
Revised: Moreover, we ﬁnd in Section 5 that our conclusions generalise to standard architectures & image datasets.

Original: In the interest of space & readability, we focus on providing intuition in this section, and ﬁll in remaining details/proofs in the appendix.
Revised: In the interest of space & readability, we focus on providing intuition in this section, and ﬁll in remaining details/proofs in the appendix.

Original: Multi-view data.
Revised: Multi-view data.

Original: We consider the data classiﬁcation problem introduced by Allen-Zhu & Li (2020), with C classes and inputs x with P patches each of dimension d ,
Revised: We consider the data classiﬁcation problem introduced by Allen-Zhu & Li (2020), with C classes and inputs x with P patches each of dimension d ,

Original: meaning x ∈
Revised: meaning x ∈

Original: ( R d ) P .
Revised: ( R d ) P .

Original: For each class c , we suppose that there exist two attributes v c, 1 , v c, 2 ∈ R d .
Revised: For each class c , we suppose that there exist two attributes v c, 1 , v c, 2 ∈ R d .

Original: For x belonging to class c , the attributes found in patches of x will include v c, 1 and v c, 2 , as well as a random selection of out-of-class attributes { v c (cid:48) ,l } c (cid:48) (cid:54)
Revised: For x belonging to class c , the attributes found in patches of x will include v c, 1 and v c, 2 , as well as a random selection of out-of-class attributes { v c (cid:48) ,l } c (cid:48) (cid:54)

Original: =
Revised: =

Original: c,l ∈ [2] .
Revised: c,l ∈ [2] .

Original: This denotes the multi-view nature of the data distribution.
Revised: This denotes the multi-view nature of the data distribution.

Original: In the true data-generating distribution D , we suppose that a proportion µ of the data ( x , y ) is singleview, which means that only one of v c, 1 or v c, 2 is present in x when ( x , y ) is from class c .
Revised: In the true data-generating distribution D , we suppose that a proportion µ of the data ( x , y ) is singleview, which means that only one of v c, 1 or v c, 2 is present in x when ( x , y ) is from class c .

Original: These will be the data for which standard training fails to generalise.
Revised: These will be the data for which standard training fails to generalise.

Original: A precise deﬁnition of multi-view data is presented in App. B.1.1.
Revised: A precise deﬁnition of multi-view data is presented in App. B.1.1.

Original: Allen-Zhu & Li (2020) argue that this multi-view setting provides a compelling proxy for standard image datasets such as CIFAR-10/100. Intuition: FKD on multi-view
Revised: Allen-Zhu & Li (2020) argue that this multi-view setting provides a compelling proxy for standard image datasets such as CIFAR-10/100 Krizhevsky (2009).

Original: data
Revised: [Equation]

Original: Suppose we have an image classiﬁcation task, with cat & car just two out of many classes.
Revised: Suppose we have an image classiﬁcation task, with cat & car just two out of many classes.

Original: For the car class, v c, 1 could correspond to headlights, whilst v c, 2 could correspond to wheels.
Revised: For the car class, v c, 1 could correspond to headlights, whilst v c, 2 could correspond to wheels.

Original: We would then expect v c, 1 to also appear in patches of an input image, x cat , corresponding to a cat with headlight-like eyes.
Revised: We would then expect v c, 1 to also appear in patches of an input image, x cat , corresponding to a cat with headlight-like eyes.

Original: Allen-Zhu & Li (2020) show that a single trained model is biased to learn exactly one of v c, 1 or v c, 2 , depending on its parameter initialisation.
Revised: Allen-Zhu & Li (2020) show that a single trained model is biased to learn exactly one of v c, 1 or v c, 2 , depending on its parameter initialisation.

Original: W.L.O.G, suppose that the student is biased to learn v c, 2 & not v c, 1 .
Revised: W.L.O.G, suppose that the student is biased to learn v c, 2 & not v c, 1 .

Original: If the teacher model has learnt v c, 1 , this means that the teacher model knows there is a similarity between x cat & any car image, x car , that displays headlights.
Revised: If the teacher model has learnt v c, 1 , this means that the teacher model knows there is a similarity between x cat & any car image, x car , that displays headlights.

Original: Mathematically, we show that this corresponds to a large value for k T ( x cat , x car ) .
Revised: Mathematically, we show that this corresponds to a large value for k T ( x cat , x car ) .

Original: Our FKD regularisation forces the student to also have a large value for k S ( x cat , x car ) , ensuring that attribute v c, 1 is also learnt by the student network.
Revised: Our FKD regularisation forces the student to also have a large value for k S ( x cat , x car ) , ensuring that attribute v c, 1 is also learnt by the student network.

Original: Without distillation, a student NN which has learnt v c, 2 & not v c, 1 will not generalise to front-on images of cars that hide wheels.
Revised: Without distillation, a student NN which has learnt v c, 2 & not v c, 1 will not generalise to front-on images of cars that hide wheels.

Original: Convolutional NN & corresponding feature kernel.
Revised: Convolutional NN & corresponding feature kernel.

Original: Like Allen-Zhu & Li (2020), for our theoretical analyis we consider a single hidden-layer convolutional NN (CNN) with sum-pooling.
Revised: Like Allen-Zhu & Li (2020), for our theoretical analyis we consider a single hidden-layer convolutional NN (CNN) with sum-pooling.

Original: For each class c ∈ [ C ] , we suppose that the CNN has m channels, giving Cm channels in total.
Revised: For each class c ∈ [ C ] , we suppose that the CNN has m channels, giving Cm channels in total.

Original: For channel r and class c , we suppose that we have weights θ c,r ∈ R d .
Revised: For channel r and class c , we suppose that we have weights θ c,r ∈ R d .

Original: This gives output for class c by
Revised: This gives output for class c by

Original: [Equation]
Revised: [Equation]

Original: where for ease of analysis (cid:94) ReLU is ReLU-like but with continuous gradient, see App.
Revised: where for ease of analysis (cid:94) ReLU is ReLU-like but with continuous gradient, see App.

Original: B.2.
Revised: B.2.

Original: To deﬁne the feature kernel for this CNN f , we recast f ( x )=
Revised: Before we consider FKD, we must ﬁrst deﬁne the feature kernel for this CNN f . To do so, we recast f ( x )=

Original: W · h ( x , θ ) , where h ( x , θ ) ∈ R Cm and W ∈ R C × Cm satisfying, for r ∈ [ m ] , c ∈
Revised: W · h ( x , θ ) , where h ( x , θ ) ∈ R Cm and W ∈ R C × Cm satisfying, for r ∈ [ m ] , c ∈

Original: [ C ] , c (cid:48) ∈
Revised: [ C ] , c (cid:48) ∈

Original: [ C ] : 
Revised: [ C ] : h ( x , θ ) r +( c − 1) m def = P (cid:88) p = (cid:94) ReLU ( (cid:104) θ c,r , x p (cid:105) ) , and W c (cid:48) ,r +( c − 1) m def = 1 { c = c (cid:48) } . Given that the feature kernel is k ( x , x (cid:48) )

Revised: def = (cid:104) h

Revised: ( x , θ ) , h

Original: [Equation]
Revised: [Equation]

Original: and W
Revised: x

Original: [Equation]
Revised: [Equation]

Original: Given that the feature kernel is k ( x , x (cid:48) )

Original: def = (cid:104) h ( x , θ ) , h ( x (cid:48) , θ ) (cid:105) , we now have that:
Revised: , θ ) (cid:105) , we now have that:

Original: [Equation]
Revised: [Equation]

Original: We ﬁrst recall that standard training of the model f with gradient descent and cross entropy loss fails to generalise on half the µ proportion of data that is single-view..com/
Revised: We ﬁrst recall that standard training of the model f with gradient descent and cross entropy loss fails to generalise on half the µ proportion of data that is single-view.

Original: en-us/research/blog/three-mysteries-in-deep-

Original: learning-ensemble-knowledge-distillation-and-self-distillation/

Original: Theorem 1 (Standard training fails, Theorem 1 of Allen-Zhu & Li (2020)) .
Revised: Theorem 1 (Standard training fails, Theorem 1 of Allen-Zhu & Li (2020)) .

Original: For sufﬁciently many classes C and channels m ∈ [ polylog ( C ) , C ] , with learning rate η ≤
Revised: For sufﬁciently many classes C and channels m ∈ [ polylog ( C ) , C ] , with learning rate η ≤

Original: [Equation]
Revised: [Equation]

Original: , training time
Revised: , training time

Original: T ∗ = poly ( C ) η , and multi-view data distribution (App. B.1.1), the trained model f
Revised: T ∗ = poly ( C ) η , and multi-view data distribution (App. B.1.1), the trained model f

Original: ( T ∗ ) satisﬁes with probability at least 1 − e − Ω( log 2 ( C )) :
Revised: ( T ∗ ) satisﬁes with probability at least 1 − e − Ω( log 2 ( C )) :

Original: • Training accuracy is perfect:
Revised: • Training accuracy is perfect:

Original: For all ( x , y ) ∈ ˆ D , y = argmax c
Revised: For all ( x , y ) ∈ ˆ D , y = argmax c

Original: f
Revised: f

Original: ( T ∗ ) c
Revised: ( T ∗ ) c

Original: [Equation]
Revised: [Equation]

Original: • Test accuracy is bad but consistent: P ( x , y ) ∼D (cid:2)
Revised: • Test accuracy is bad but consistent: P ( x , y ) ∼D (cid:2)

Original: y (cid:54) =
Revised: y (cid:54) =

Original: argmax c
Revised: argmax c

Original: f
Revised: f

Original: ( T ∗ ) c
Revised: ( T ∗ ) c

Original: [Equation]
Revised: [Equation]

Original: ∈ [0 . 49 µ, 0 .
Revised: ∈ [0 . 49 µ, 0 .

Original: 51 µ ] .
Revised: 51 µ ] .

Original: Now we are ready to show that regularising the student model towards the teacher model’s feature kernel, as in FKD, improves test accuracy.
Revised: Now we are ready to show that regularising the student model towards the teacher model’s feature kernel, as in FKD, improves test accuracy.

Original: We suppose our teacher model is an ensemble of E ≥models { f e } Ee =1 , each with corresponding feature kernel k e , trained as standard on the same data with independent initialisations θ e 0 .
Revised: We suppose our teacher model is an ensemble of E ≥models { f e } Ee =1 , each with corresponding feature kernel k e , trained as standard on the same data with independent initialisations θ e 0 .

Original: We average k e over e ∈
Revised: We average k e over e ∈

Original: [ E ] to obtain our teacher feature kernel:
Revised: [ E ] to obtain our teacher feature kernel:

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: x (cid:48)
Revised: x (cid:48)

Original: [Equation]
Revised: [Equation]

Original: This is akin to concatenating all features in { h e } Ee =1 into a ECm -dimensional feature vector, albeit without the additional computational baggage.
Revised: This is akin to concatenating all features in { h e } Ee =1 into a ECm -dimensional feature vector, albeit without the additional computational baggage.

Original: We then have our main theoretical result:
Revised: We then have our main theoretical result:

Original: Theorem 2 (FKD improves student generalisation and is better with larger ensemble) .
Revised: Theorem 2 (FKD improves student generalisation and is better with larger ensemble) .

Original: Given an arbitrary (cid:15) > 0 .
Revised: Given an arbitrary (cid:15) > 0 .

Original: For any ensemble size E of teacher NNs trained as in Theorem 1 and sufﬁciently many classes C , for m = polylog ( C ) , with learning rate η ≤ 1 poly ( C ) , and training time T ∗ = poly ( C ) η , the ensemble teacher knowledge can be distilled into a single student model f
Revised: For any ensemble size E of teacher NNs trained as in Theorem 1 and sufﬁciently many classes C , for m = polylog ( C ) , with learning rate η ≤ 1 poly ( C ) , and training time T ∗ = poly ( C ) η , the ensemble teacher knowledge can be distilled into a single student model f

Original: ( T ∗ ) using only teacher feature kernel k T , Eq.
Revised: ( T ∗ ) using only teacher feature kernel k T , Eq.

Original: (7) , such that with probability at least 1 − e − Ω( log 2 ( C
Revised: (7) , such that with probability at least 1 − e − Ω( log 2 ( C

Original: • Training accuracy is perfect:
Revised: • Training accuracy is perfect:

Original: For all ( x , y ) ∈ ˆ D , y = argmax c
Revised: For all ( x , y ) ∈ ˆ D , y = argmax c

Original: f
Revised: f

Original: ( T ∗ ) c ( x ) .
Revised: ( T ∗ ) c ( x ) .

Original: • Test accuracy is good: P ( x , y ) ∼D (cid:2) y (cid:54) =
Revised: • Test accuracy is good: P ( x , y ) ∼D (cid:2) y (cid:54) =

Original: argmax c
Revised: argmax c

Original: f
Revised: f

Original: ( T ∗ ) c
Revised: ( T ∗ ) c

Original: ( x ) (cid:3) ≤ ( 1 2 E +1 + (cid:15) ) µ .
Revised: ( x ) (cid:3) ≤ ( 1 2 E +1 + (cid:15) ) µ .

Original: To outline the proof of Theorem 2, we ﬁrst show, in Lemma 1, that a single trained NN’s feature kernel (which we deﬁned in Eq.
Revised: Proof outline for Theorem 2 , we ﬁrst show, in Lemma 1, that a single trained NN’s feature kernel (which we deﬁned in Eq.

Original: (6)) can detect if two inputs share a data attribute that the NN has learnt due to its weight initialisation.
Revised: (6)) can detect if two inputs share a data attribute that the NN has learnt due to its weight initialisation.

Original: We extend this result to an ensemble teacher in App.
Revised: We extend this result to an ensemble teacher in App.

Original: C.2, showing that the ensemble teacher feature kernel detects the union of all data attributes learnt by individual { k e } Ee =1 .
Revised: C.2, showing that the ensemble teacher feature kernel detects the union of all data attributes learnt by individual { k e } Ee =1 .

Original: This simpliﬁes our calculations when showing that our distillation regulariser, Eq.
Revised: This simpliﬁes our calculations when showing that our distillation regulariser, Eq.

Original: (3), is effective for improved student generalisation.
Revised: (3), is effective for improved student generalisation.

Original: The full proof can be found in App.
Revised: The full proof can be found in App.

Original: C.
Revised: C.

Original: Intuition: FKD with ensemble of teachers
Revised: Intuition: FKD with ensemble of teachers

Original: To parse Theorem 2, suppose we have a single teacher i.e. E =1 .
Revised: To parse Theorem 2, suppose we have a single teacher i.e. E =1 .

Original: Then, the test error is essentially 0 .
Revised: Then, the test error is essentially 0 .

Original: 25 µ in Theorem 2.
Revised: 25 µ in Theorem 2.

Original: The explanation is that both the student & teacher networks independently learn one of { v c,l } 2 l =1 for each class c .
Revised: The explanation is that both the student & teacher networks independently learn one of { v c,l } 2 l =1 for each class c .

Original: Either vanilla KD (Theorem 4 of Allen-Zhu & Li (2020)) or our feature kernel approach allow the student to learn the union of the independent attributes learnt by the student and the teacher, so for only a quarter of the single-view test data x s will the student not have learnt the useful class attribute present in x s .
Revised: Either vanilla KD (Theorem 4 of Allen-Zhu & Li (2020)) or our feature kernel approach allow the student to learn the union of the independent attributes learnt by the student and the teacher, so for only a quarter of the single-view test data x s will the student not have learnt the useful class attribute present in x s .

Original: For general ensemble size
Revised: For general ensemble size

Original: E , the story is the same: the student & E teachers each independently learn one of the two useful attributes { v c,l } l ∈ 2 for all c ∈ [ C ] .
Revised: E , the story is the same: the student & E teachers each independently learn one of the two useful attributes { v c,l } l ∈ 2 for all c ∈ [ C ] .

Original: Distilling allows the student to learn the union of these attributes, which means that the student will fail on only 1 2 E +1 of the single-view data.
Revised: Distilling allows the student to learn the union of these attributes, which means that the student will fail on only 1 2 E +1 of the single-view data.

Original:  FKD IN PRACTICE
Revised:  FKD IN PRACTICE

Original: Next, we highlight practical considerations for implementing FKD derived from our theory.
Revised: Next, we highlight practical considerations for implementing FKD derived from our theory.

Original: Pseudocode and PyTorch-style code for our FKD implementation are given in Algs.
Revised: Pseudocode and PyTorch-style code for our FKD implementation are given in Algs.

Original: 1 and 2 respectively.
Revised: 1 and 2 respectively.

Original: Correlation kernel.
Revised: Correlation kernel.

Original: We propose D ( ρ θ , ρ T ) as a regulariser in FKD instead of D ( k θ , k T ) , where:
Revised: We propose D ( ρ θ , ρ T ) as a regulariser in FKD instead of D ( k θ , k T ) , where:

Original: [Equation]
Revised: [Equation]

Original: deﬁnes feature correlation kernel ρ z , corresponding to feature kernel k z , for z ∈ [ E ] ∪{ θ } .
Revised: deﬁnes feature correlation kernel ρ z , corresponding to feature kernel k z , for z ∈ [ E ] ∪{ θ } .

Original: The reason we use correlation kernels is that they normalise data, so that ρ ( x , x )=1 ∀ x , which zeros diagonal differences in Eq.
Revised: The reason we use correlation kernels is that they normalise data, so that ρ ( x , x )=1 ∀ x , which zeros diagonal differences in Eq.

Original: (3): we hope FKD allows the student to learn from the teacher features shared between different inputs.
Revised: (3): we hope FKD allows the student to learn from the teacher features shared between different inputs.

Original: Non-zero diagonal differences, like in Similarity-Preserving (SP) KD (Tung & Mori, 2019), encourage the student to learn noise as we show in App.
Revised: Non-zero diagonal differences, like in Similarity-Preserving (SP) KD (Tung & Mori, 2019), encourage the student to learn noise as we show in App.

Original: D.1, and we hypothesise that this contributes to the improved performance we observe of FKD over SP in Section 5.
Revised: D.1, and we hypothesise that this contributes to the improved performance we observe of FKD over SP in Section 5.

Original: Moreover, this normalisation helps balance individual teacher’s inﬂuence in an ensemble teacher, and ensures that FKD does not need a temperature hyperparameter τ , which produces soft labels in vanilla KD.
Revised: Moreover, this normalisation helps balance individual teacher’s inﬂuence in an ensemble teacher, and ensures that FKD does not need a temperature hyperparameter τ , which produces soft labels in vanilla KD.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Feature regularisation.
Revised: Feature regularisation.

Original: One downside to using the correlation kernel is that our FKD regularisation, D ( ρ θ , ρ T ) , becomes invariant to the scale of k θ .
Revised: One downside to using the correlation kernel is that our FKD regularisation, D ( ρ θ , ρ T ) , becomes invariant to the scale of k θ .

Original: For example, replacing k θ ( x , x (cid:48) ) with (cid:112) M ( x ) M ( x (cid:48) )
Revised: For example, replacing k θ ( x , x (cid:48) ) with (cid:112) M ( x ) M ( x (cid:48) )

Original: k θ ( x , x (cid:48) ) , for any
Revised: k θ ( x , x (cid:48) ) , for any

Original: M ( x ): R d (cid:55)→ R + , leaves the student correlation kernel unchanged.
Revised: M ( x ): R d (cid:55)→ R + , leaves the student correlation kernel unchanged.

Original: This may lead to degeneracies when training θ , & large variations in k ( x , x ) over x may harm generalisation (as evidenced by the fact that input normalisation is common across ML, from linear models to NNs).
Revised: This may lead to degeneracies when training θ , & large variations in k ( x , x ) over x may harm generalisation (as evidenced by the fact that input normalisation is common across ML, from linear models to NNs).

Original: Moreover, our proof of Theorem 2 is not quantitative, in that we only show k θ ( x , x (cid:48) )
Revised: Moreover, our proof of Theorem 2 is not quantitative, in that we only show k θ ( x , x (cid:48) )

Original: =
Revised: =

Original: ˜Θ(1) in the number of classes C up to polylogarithmic factor in C ,
Revised: ˜Θ(1) in the number of classes C up to polylogarithmic factor in C ,

Original: when x (cid:54) = x (cid:48) share a data attribute v c,l that has been learned by parameters θ .
Revised: when x (cid:54) = x (cid:48) share a data attribute v c,l that has been learned by parameters θ .

Original: These insights motivate us to regularise the student feature h during FKD training to control the norm of k ( x , x ) across inputs x .
Revised: These insights motivate us to regularise the student feature h during FKD training to control the norm of k ( x , x ) across inputs x .

Original: We use an additive (cid:96) 2 regularisation 1 B (cid:80)
Revised: We use an additive (cid:96) 2 regularisation 1 B (cid:80)

Original: Bb =1 (cid:107) h
Revised: Bb =1 (cid:107) h

Original: ( x b , θ ) (cid:107) 22 = 1 B (cid:80)
Revised: ( x b , θ ) (cid:107) 22 = 1 B (cid:80)

Original: Bb =1 k ( x b , x b ) with regularisation strength λ FR > 0 , for each minibatch { ( x b , y b ) }
Revised: Bb =1 k ( x b , x b ) with regularisation strength λ FR > 0 , for each minibatch { ( x b , y b ) }

Original: Bb =1 .
Revised: Bb =1 .

Original: Fig.
Revised: Fig.

Original: 3 shows that using Feature Regularisation (FR) encourages a more even spread of k ( x , x ) across inputs for a student VGG8 network trained with a VGG13 teacher model on
Revised: 3 shows that using Feature Regularisation (FR) encourages a more even spread of k ( x , x ) across inputs for a student VGG8 network trained with a VGG13 teacher model on

Original: CIFAR-100.
Revised: CIFAR-100.

Original: Corresponding plots for other architectures can be found in Fig.
Revised: Corresponding plots for other architectures can be found in Fig.

Original: Similar to Dauphin & Cubuk (2021), we ﬁnd that FR improves generalisation & provide ablations in Section 5.
Revised: Similar to Dauphin & Cubuk (2021), we ﬁnd that FR improves generalisation & provide ablations in Section 5.

Original: Algorithm
Revised: Algorithm

Original: Require: Maximum number of iterations T ∗ , batch size B , learning rate η , teacher correlation kernel ρ T ,
Revised: Require: Maximum number of iterations T ∗ , batch size B , learning rate η , teacher correlation kernel ρ T ,

Original: FKD regularisation λ KD > 0 , Feature regularisation λ FR > 0 .
Revised: FKD regularisation λ KD > 0 , Feature regularisation λ FR > 0 .

Original: Initialise student parameters θ 0 , W 0 .
Revised: Initialise student parameters θ 0 , W 0 .

Original: for iteration t = 0 , . . .
Revised: for iteration t = 0 , . . .

Original: , T ∗ do
Revised: , T ∗ do

Original: Sample minibatch ( x Bi , y Bi )
Revised: Sample minibatch ( x Bi , y Bi )

Original: Bi =1 i.i.d.
Revised: Bi =1 i.i.d.

Original: ∼ ˆ D .
Revised: ∼ ˆ D .

Original: Compute loss L = 1 B (cid:80) Bi
Revised: Compute loss L = 1 B (cid:80) Bi

Original: =1 L ( f ( x Bi ) , y Bi )
Revised: =1 L ( f ( x Bi ) , y Bi )

Original: + 2 λ KD B
Revised: + 2 λ KD B

Original: ( B − 1) (cid:80) Bi (cid:54) =
Revised: ( B − 1) (cid:80) Bi (cid:54) =

Original: j (cid:0) ρ θ t ( x Bi , x Bj ) − ρ T ( x Bi , x Bj ) (cid:1) 2 .
Revised: j (cid:0) ρ θ t ( x Bi , x Bj ) − ρ T ( x Bi , x Bj ) (cid:1) 2 .

Original: Add feature regularisation L =
Revised: Add feature regularisation L =

Original: L + λ FR B (cid:80) Bi
Revised: L + λ FR B (cid:80) Bi

Original: =1 (cid:13)(cid:13)
Revised: =1 (cid:13)(cid:13)

Original: h ( x Bi , θ t ) (cid:13)(cid:13) 2 2  (optional).
Revised: h ( x Bi , θ t ) (cid:13)(cid:13) 2 2 , (optional).

Original: Update parameters θ t +1 ← θ t − η ∇ θ L , W t +1 ← W t − η ∇ W L . end for return { θ T ∗ , W T ∗ }
Revised: Update parameters θ t +1 ← θ t − η ∇ θ L , W t +1 ← W t − η ∇ W L . end for return { θ T ∗ , W T ∗ }

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Due to space concerns, App. F contains further experiments & any missing experimental details.
Revised: Due to space concerns, App. F contains further experiments & any missing experimental details.

Original: Ensemble distillation.
Revised: Ensemble distillation.

Original: We ﬁrst verify that larger ensemble teacher size, E , further improves FKD student performance as suggested by Theorem 2.
Revised: We ﬁrst verify that larger ensemble teacher size, E , further improves FKD student performance as suggested by Theorem 2.

Original: This is conﬁrmed in
Revised: This is conﬁrmed in

Original: Fig. 4, using VGG8 for all student & teacher networks on the CIFAR-100 dataset.
Revised: Fig. 4, using VGG8 for all student & teacher networks on the CIFAR-100 dataset.

Original: [Figure]
Revised: [Figure]

Original: We also plot the test accuracy of the ensemble teacher of size E , whose predictive probabilities are averaged over individual teachers, as well as the test accuracy of an undistilled student model.
Revised: We also plot the test accuracy of the ensemble teacher across sizes E , whose predictive probabilities are averaged over individual teachers, as well as the test accuracy of an undistilled student model.

Original: We see that FKD consistently outperforms vanilla
Revised: We see that FKD consistently outperforms vanilla

Original: KD, and both distillation methods outperform the teacher in the ‘self-distillation’ setting of E =

Revised: KD, and both distillation methods outperform the teacher in the ‘self-distillation’ setting of E =

Original: (Furlanello et al., 2018; Zhang et al., 2019).
Revised: (Furlanello et al., 2018; Zhang et al., 2019).

Original: Moreover, FKD allows a single student to match teacher performance when E =2 , before positive but diminishing returns with larger E relative to the teacher ensemble.
Revised: Moreover, FKD allows a single student to match teacher performance when E =2 , before positive but diminishing returns with larger E relative to the teacher ensemble.

Original: [Table]
Revised: [Table]

Original: Dataset Transfer.
Revised: Dataset Transfer.

Original: We next show FKD can transfer knowledge across similar datasets.
Revised: We next show FKD can transfer knowledge across similar datasets.

Original: From a ﬁxed VGG13 teacher network trained on CIFAR100, we distil to student
Revised: From a ﬁxed VGG13 teacher network trained on CIFAR100, we distil to student

Original: VGG8 NNs on CIFAR-10,
Revised: VGG8 NNs on CIFAR-10,

Original: STL-10 & Tiny-ImageNet.
Revised: STL-10 & Tiny-ImageNet.

Original: As no student dataset has 100 classes, unlike CIFAR-100, it is not clear how one can use vanilla KD (Hinton et al., 2015) in this case.
Revised: As no student dataset has 100 classes, unlike CIFAR-100, it is not clear how one can use vanilla KD (Hinton et al., 2015) in this case.

Original: We thus compare FKD to other feature kernel based KD methods: Relational KD (RKD) (Park et al., 2019) & Similarity-Preserving (SP) KD (Tung & Mori, 2019).
Revised: We thus compare FKD to other feature kernel based KD methods: Relational KD (RKD) (Park et al., 2019) & Similarity-Preserving (SP) KD (Tung & Mori, 2019).

Original: In Table
Revised: In Table

Original: [Table]
Revised: [Table]

Original: we see that FKD without feature regularisation outperforms both baselines across all datasets, and that feature regularisation (FR) further improves FKD performance, highlighting the beneﬁt of our practical considerations in Section 4.
Revised: we see that FKD without feature regularisation outperforms both baselines across all datasets, and that feature regularisation (FR) further improves FKD performance, highlighting the beneﬁt of our practical considerations in Section 4.

Original: The improved performance is particularly stark on STL-10 (which we downsize to 32x32 resolution), where FKD improves student performance by 2.75%.
Revised: The improved performance is particularly stark on STL-10 (which we downsize to 32x32 resolution), where FKD improves student performance by 2.75%.

Original: STL-10 is well suited for FKD as it has only 5K labeled inputs but 100K unlabeled datapoints, which can be used in our feature kernel regulariser, D ( ρ θ , ρ T )
Revised: STL-10 is well suited for FKD as it has only 5K labeled inputs but 100K unlabeled datapoints, which can be used in our feature kernel regulariser, D ( ρ θ , ρ T )

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Comparison on CIFAR-100 and ImageNet.
Revised: Comparison on CIFAR-100 and ImageNet.

Original: We next compare FKD to various knowledge distillation baselines on CIFAR-100 and ImageNet, across a selection of teacher/student architectures.
Revised: Finally, we compare FKD to various knowledge distillation baselines on CIFAR-100 and ImageNet, across a selection of teacher/student architectures.

Original: We see in Table 2 that FKD consistently outperforms: vanilla KD (Hinton et al., 2015), RKD (Park et al., 2019), and SP (Tung & Mori, 2019).
Revised: We see in Table 2 that FKD consistently outperforms: vanilla KD (Hinton et al., 2015), RKD (Park et al., 2019), and SP (Tung & Mori, 2019).

Original: Moreover, FKD either matches or outperforms the highperforming Contrastive Representational Distillation (Tian et al., 2020).
Revised: Moreover, FKD either matches or outperforms the high-performing Contrastive Representational Distillation (Tian et al., 2020).

Original: We use the exact same teacher checkpoints provided by Tian et al.
Revised: We use the exact same teacher checkpoints used by Tian et al.

Original: and
Revised: and

Original: Chen et al.
Revised: Chen et al.

Original: (2021) for CIFAR-100 and ImageNet respectively to ensure fair comparison.
Revised: (2021) for CIFAR-100 and ImageNet respectively to ensure fair comparison.

Original: We ﬁnd, like in Table 1, that feature regularisation consistently improves FKD performance and that even without feature regularisation, FKD outperforms all feature kernel based KD methods.
Revised: We ﬁnd, like in Table 1, that feature regularisation consistently improves FKD performance and that even without feature regularisation, FKD outperforms all feature kernel based KD methods.

Original: This implies that using the correlation kernel to zero out diagonal differences, as described in Section 4, indeed helps improve student performance.
Revised: This implies that using the correlation kernel to zero out diagonal differences, as described in Section 4, indeed helps improve student performance.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: NN Knowledge Distillation.
Revised: NN Knowledge Distillation.

Original: Following Hinton et al.
Revised: Following Hinton et al.

Original: , there has been much interest in expanding KD in NNs (Romero et al., 2014; Zagoruyko & Komodakis, 2016; Passalis & Tefas, 2018;
Revised: , there has been much interest in expanding KD in NNs (Romero et al., 2014; Zagoruyko & Komodakis, 2016; Passalis & Tefas, 2018;

Original: Zhang et al., 2018; Yu et al., 2019; Chen et al., 2020a; Tian et al., 2020).
Revised: Zhang et al., 2018; Yu et al., 2019; Chen et al., 2020a; Tian et al., 2020).

Original: Most similar to FKD are
Revised: Most similar to FKD are

Original: Park et al.
Revised: Park et al.

Original: Tung & Mori (2019) who also use relations between inputs to distil knowledge (albeit not from the feature kernel learning perspective and without our theoretical justiﬁcation), as well as Qian et al.
Revised: Tung & Mori (2019) who also use relations between inputs to distil knowledge (albeit not from the feature kernel learning perspective and without our theoretical justiﬁcation), as well as Qian et al.

Original: (2020) who focus on reducing computational costs of expensive kernel matrix operations, whereas FKD uses only pairwise interactions between inputs.
Revised: (2020) who focus on reducing computational costs of full-batch kernel matrix operations.

Original: App.
Revised: App.

Original: D highlights in more detail the differences of FKD compared to previous pairwise feature kernel based KD methods.
Revised: D highlights in more detail the differences of FKD compared to previous pairwise feature kernel based KD methods.

Original: Allen-Zhu & Li (2020) made the ﬁrst theoretical connection using the mechanisms of ensembling in NNs to explain the success of vanilla KD in NNs, which we extend for feature kernel based KD.
Revised: Allen-Zhu & Li (2020) made the ﬁrst theoretical connection using the mechanisms of ensembling in NNs to explain the success of vanilla KD in NNs, which we extend for feature kernel based KD.

Original: Ensembling NNs.
Revised: Ensembling NNs.

Original: Ensembling NNs has long been studied for improving predictive accuracy (Hansen & Salamon, 1990; Krogh et al., 1995) with particular recent focus towards uncertainty quantiﬁcation & Bayesian inference (Lakshminarayanan et al., 2017; Ovadia et al., 2019; Zaidi et al., 2020; Pearce et al., 2020;
Revised: Ensembling NNs has long been studied for improving predictive accuracy (Hansen & Salamon, 1990; Krogh et al., 1995) with particular recent focus towards uncertainty quantiﬁcation & Bayesian inference (Lakshminarayanan et al., 2017; Ovadia et al., 2019; Zaidi et al., 2020; Pearce et al., 2020;

Original: He et al., 2020; Wilson & Izmailov, 2020; Wenzel et al., 2020;
Revised: He et al., 2020; Wilson & Izmailov, 2020; Wenzel et al., 2020;

Original: D’Angelo & Fortuin, 2021; Schut et al., 2021) and predictive diversity (Fort et al., 2019; D’Amour et al., 2020).
Revised: D’Angelo & Fortuin, 2021; Schut et al., 2021) and predictive diversity (Fort et al., 2019; D’Amour et al., 2020).

Original: On the topic of Bayesian inference, the feature kernel has also been studied under the name of Neural Linear Model (Riquelme et al., 2018; Ober & Rasmussen, 2019), and extensions treating features h as inputs to standard Gaussian Process kernels are known by the name of Deep
Revised: On the topic of Bayesian inference, the feature kernel has also been studied under the name of Neural Linear Model (Riquelme et al., 2018; Ober & Rasmussen, 2019), and extensions treating features h as inputs to standard Gaussian Process kernels are known by the name of Deep

Original: Kernel Learning (Wilson et al., 2016; Ober et al., 2021; van Amersfoort et al., 2021).
Revised: Kernel Learning (Wilson et al., 2016; Ober et al., 2021; van Amersfoort et al., 2021).

Original: NN Feature learning.
Revised: NN Feature learning.

Original: A recent ﬂurry of work has focussed on characterising & understanding the importance of feature learning in NNs (Chizat et al., 2019;
Revised: A recent ﬂurry of work has focussed on characterising & understanding the importance of feature learning in NNs (Chizat et al., 2019;

Original: Fort et al., 2020; Baratin et al., 2021;
Revised: Fort et al., 2020; Baratin et al., 2021;

Original: Lee et al., 2020; Aitchison, 2020; Ghorbani et al., 2020), fuelled in part by the development that wide NNs become (Neural Tangent) Kernel machines in certain regimes (Jacot et al., 2018; Lee et al., 2019; Yang & Littwin, 2021), thus forgoing feature learning.
Revised: Lee et al., 2020; Aitchison, 2020; Ghorbani et al., 2020), fuelled in part by the development that wide NNs become (Neural Tangent) Kernel machines in certain regimes (Jacot et al., 2018; Lee et al., 2019; Yang & Littwin, 2021), thus forgoing feature learning.

Original: The consensus in these works is that there are gaps between NTK theory & practical NNs that cannot be explained without feature learning.
Revised: The consensus in these works is that there are gaps between NTK theory & practical NNs that cannot be explained without feature learning.

Original: However, Yang & Hu (2020) proved that feature-learning is still possible with inﬁnitewidth NNs, and also that feature learning is equivalent to feature kernel learning in inﬁnite-width
Revised: However, Yang & Hu (2020) proved that feature-learning is still possible with inﬁnitewidth NNs, and also that feature learning is equivalent to feature kernel learning in inﬁnite-width

Original: NNs.
Revised: NNs.

Original: This motivates our study of the feature kernel as a key object for distillation.
Revised: This motivates our study of the feature kernel as a key object for distillation.

Original: Regularising the feature kernel to the true target covariance kernel was suggested by Yoo et al.
Revised: Regularising the feature kernel to the true target covariance kernel was suggested by Yoo et al.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have theoretically shown that the feature kernel is a valid object for Knowledge Distillation (KD) in Neural Networks (NNs), by extending the analysis of Allen-Zhu & Li (2020) in a multiview data setting.
Revised: We have theoretically shown that the feature kernel is a valid object for Knowledge Distillation (KD) in Neural Networks (NNs) by extending the analysis of Allen-Zhu & Li (2020), which focused on vanilla KD (Hinton et al., 2015).

Original: Further, we used an understanding of a student NN’s optimisation process to motivate practical considerations when using feature kernels for distillation. These include using the feature correlation kernel & using feature regularisation, and we term our approach Feature
Revised: Further, we used our theoretical insights to motivate practical considerations when using feature kernels for distillation, such as using the feature correlation kernel & using feature regularisation, to improve on previous feature based KD methods.

Original:  Kernel Distillation (FKD).
Revised: We term our approach Feature Kernel Distillation (FKD), and note that FKD is more widely applicable than vanilla KD, as it beneﬁts from being agnostic to teacher and student prediction spaces.

Original: Experimentally, we have demonstrated that FKD is amenable to ensemble distillation as suggested by our theory, is able to transfer knowledge across similar datasets and that FKD outperforms vanilla KD & other feature kernel based KD methods across a variety of architectures on CIFAR-100, and ImageNet-1K.
Revised: Experimentally, we have demonstrated that FKD is amenable to ensemble distillation as suggested by our theory, is able to transfer knowledge across similar datasets and that FKD outperforms vanilla KD & previous feature kernel based KD methods across a variety of architectures on CIFAR-100, and ImageNet-1K.

Original: Limitations & future work.
Revised: Limitations & future work.

Original: Though feature learning is central to our results, we stress that there are still gaps between our theory and practice to understanding NN ensembling & KD,as demonstrated by the divergence between ensemble teacher & FKD in Fig.
Revised: Though feature learning is central to our results, we stress that there are still gaps between our theory & practice to understanding NN ensembling & KD, demonstrated by the divergence between ensemble teacher & FKD in Fig.

Original: 4 for larger ensemble size.
Revised: 4 for larger ensemble size.

Original: This could be due to a number of reasons, such as the multi-view data setting not being able to capture the full complexity of real-world data, the role of hierarchical feature learning between layers in a deep NN, or the importance of minibatching in stochastic gradient descent.
Revised: This could be due to: the multi-view data setting not being able to capture the full complexity of real-world data, the role of hierarchical feature learning between layers in a deep NN, or the importance of minibatching in stochastic gradient descent.

Original: Other future work could involve using the multi-view data setting to analyse  the beneﬁt of ensembling NNs for uncertainty quantiﬁcation, or improving FKD further to compete with state-of-the-art KD methods.
Revised: Other future work could apply the multi-view data setting to analyse uncertainty quantiﬁcation in NN ensembles, assess the impact of different FKD regularisation metrics in Eq. (3), or improve FKD further to compete with state-of-the-art KD methods.

Revised: Acknowledgements We thank Emilien Dupont, Yee Whye Teh and Sheheryar Zaidi for helpful feedback on this work.

Original: APPENDIX: FEATURE KERNEL DISTILLATION A F EATURE KERNEL DEPENDENCE WITH CROSS - ENTROPY LOSS
Revised: APPENDIX: FEATURE KERNEL DISTILLATION A F EATURE KERNEL DEPENDENCE WITH CROSS - ENTROPY LOSS

Original: Suppose, we have n data points with ﬁxed feature extractor h = h ( X ) ∈ R n × p , trainable last layer weights W ∈ R p × C and targets Y ∈ R n × C .
Revised: Suppose, we have n data points with ﬁxed feature extractor h = h ( X ) ∈ R n × p , trainable last layer weights W ∈ R p × C and targets Y ∈ R n × C .

Original: In Section 2, we described how with Mean Squared Error loss the trained NN is precisely kernel ridge regression using the feature kernel, when there is (cid:96) 2 regularisation on the last layer weights.
Revised: In Section 2, we described how with Mean Squared Error loss the trained NN is precisely kernel ridge regression using the feature kernel, when there is (cid:96) 2 regularisation on the last layer weights.

Original: Hence, the feature kernel exactly determines the predictions.
Revised: Hence, the feature kernel exactly determines the predictions.

Original: We now prove the same for cross entropy loss.
Revised: We now prove the same for cross entropy loss.

Original: That is to say, we wish to show that given the feature kernel k , the predictive probabilities/logits at the optimal W ∗ are independent of both W ∗ and h .
Revised: That is to say, we wish to show that given the feature kernel k , the predictive probabilities/logits at the optimal W ∗ are independent of both W ∗ and h .

Original: We deﬁne the loss, for some regularisation λ > 0 (purely to enforce strong convexity) by
Revised: We deﬁne the loss, for some regularisation λ > 0 (purely to enforce strong convexity) by

Original: [Equation]
Revised: [Equation]

Original: where h i ∈ R 1 × p is the i th row of h and
Revised: where h i ∈ R 1 × p is the i th row of h and

Original: [Equation]
Revised: [Equation]

Original: is cross entropy loss.
Revised: is cross entropy loss.

Original: Proposition 1.
Revised: Proposition 1.

Original: Let feature extractor h ( · ) , training inputs X , training targets Y , (cid:96) 2 regularisation λ > 0 all be ﬁxed.
Revised: Let feature extractor h ( · ) , training inputs X , training targets Y , (cid:96) 2 regularisation λ > 0 all be ﬁxed.

Original: Suppose we are given a test point x (cid:48) with features h (cid:48) , then the test prediction logits h (cid:48)
Revised: Suppose we are given a test point x (cid:48) with features h (cid:48) , then the test prediction logits h (cid:48)

Original: W ∗ at optimal W ∗ can be expressed solely in terms of feature kernel evaluations k ( · , · )=
Revised: W ∗ at optimal W ∗ can be expressed solely in terms of feature kernel evaluations k ( · , · )=

Original: (cid:104) h ( · ) , h ( · ) (cid:105) .
Revised: (cid:104) h ( · ) , h ( · ) (cid:105) .

Original: Proof.
Revised: Proof.

Original: If we differentiate L ( W ) and set to zero we get:
Revised: If we differentiate L ( W ) and set to zero we get:

Original: [Equation]
Revised: [Equation]

Original: where p ∈ R n × C is the result of applying softmax to each row of the logits h (cid:48)
Revised: where p ∈ R n × C is the result of applying softmax to each row of the logits h (cid:48)

Original: W ∗ .
Revised: W ∗ .

Original: Let h i denote the extracted features for training input x i .
Revised: Let h i denote the extracted features for training input x i .

Original: Now recall that (cid:104) h
Revised: Now recall that (cid:104) h

Original: i , h j (cid:105) =
Revised: i , h j (cid:105) =

Original: k ( x i , x j ) is the feature kernel evaluated at x i , x j .
Revised: k ( x i , x j ) is the feature kernel evaluated at x i , x j .

Original: We multiply Eq.
Revised: We multiply Eq.

Original: (9) by the test-data feature vector h (cid:48) to give:
Revised: (9) by the test-data feature vector h (cid:48) to give:

Original: [Equation]
Revised: [Equation]

Original: but h (cid:48)
Revised: but h (cid:48)

Original: W ∗ are precisely the logits for test point x ∗ , and likewise p i,l is the vector of probability predictions at training point i .
Revised: W ∗ are precisely the logits for test point x ∗ , and likewise p i,l is the vector of probability predictions at training point i .

Original: Hence, we could solve Eq.
Revised: Hence, we could solve Eq.

Original: (10) numerically for logits/predictive probabilities given only the feature kernel (without h ( · ) or W ∗ ) , and we see that the feature kernel once again determines logit/prediction probabilities at the optimal last layer parameters like for squared error loss, albeit this time implicitly for cross entropy loss.
Revised: (10) numerically for logits/predictive probabilities given only the feature kernel (without h ( · ) or W ∗ ) , and we see that the feature kernel once again determines logit/prediction probabilities at the optimal last layer parameters like for squared error loss, albeit this time implicitly for cross entropy loss.

Original: B S ETUP FOR TRAINING ON M ULTI - VIEW DATA
Revised: B S ETUP FOR TRAINING ON M ULTI - VIEW DATA

Original: Before we prove Theorem 2,  we need to recall the data and training setup of Allen-Zhu & Li (2020) for completeness and self-containedness.
Revised: Before we prove Theorem 2, which demonstrates the generalisation beneﬁts of FKD theoretically, we need to recall the data and training setup of Allen-Zhu & Li (2020) for completeness and selfcontainedness.

Original: Unless otherwise stated, everything in this section (App. B) is a simpliﬁed version of the setup in Allen-Zhu & Li (2020).
Revised: Unless otherwise stated, everything in this section (App. B) is a simpliﬁed version of the setup in Allen-Zhu & Li (2020).

Original: Our resultsshould hold in the more general version too, but we present the setup in a simpliﬁed setting here for readability and convenience, without sacriﬁcing the key messages and intuitions.
Revised: Our results hold in the more general version too, but we present the setup in a simpliﬁed setting here for readability and convenience, without sacriﬁcing the key messages and intuitions: our focus is for our theoretical and practical contributions to guide each other and align as much as possible, as opposed to e.g. maximising the generality of our theory.

Original: B. M ULTI - VIEW DATA DISTRIBUTION
Revised: B. M ULTI - VIEW DATA DISTRIBUTION

Original: Recall that we consider a C -class classiﬁcation problem over P -patch inputs, where each patch has dimension d , so our inputs are described by x = ( x 1 , . . .
Revised: Recall that we consider a C -class classiﬁcation problem over P -patch inputs, where each patch has dimension d , so our inputs are described by x = ( x 1 , . . .

Original: , x P ) ∈ ( R d ) P .
Revised: , x P ) ∈ ( R d ) P .

Original: For simplicity, we take
Revised: For simplicity, we take

Original: P = C 2 and d = poly ( C ) for a large polynomial.
Revised: P = C 2 and d = poly ( C ) for a large polynomial.

Original: Like Allen-Zhu & Li (2020), we use ˜ O, ˜Θ , ˜Ω to hide polylogarithmic factors in the number of classes, C , which we take to be sufﬁciently large.
Revised: Like Allen-Zhu & Li (2020), we use ˜ O, ˜ Θ , ˜ Ω to hide polylogarithmic factors in the number of classes, C , which we take to be sufﬁciently large.

Original: We assume that each class c ∈ [ C ] has exactly two attributes v c, 1 , v c, 2 ∈ R d , which are orthonormal for simplicity, 7 such that:
Revised: We assume that each class c ∈ [ C ] has exactly two attributes v c, 1 , v c, 2 ∈ R d , which are orthonormal for simplicity, 7 such that:

Original: [Equation]
Revised: [Equation]

Original: ] is the set of all attributes.
Revised: is the set of all attributes.

Original: B.1. D ATA GENERATING MECHANISM
Revised: B.1. D ATA GENERATING MECHANISM

Original: Let our data distribution for a data pair ( x , y ) ∼ D be deﬁned as D = µ D s + (1 − µ ) D m , for multiview & single-view distributions D m & D s respectively.
Revised: Let our data distribution for a data pair ( x , y ) ∼ D be deﬁned as D = µ D s + (1 − µ ) D m , for multiview & single-view distributions D m & D s respectively.

Original: ( x , y ) ∼ D are generated as follows:
Revised: ( x , y ) ∼ D are generated as follows:

Original: Sample y ∈
Revised: Sample y ∈

Original: [ C ] uniformly at random 2.
Revised: [ C ] uniformly at random.

Original: Sample a set V (cid:48)
Revised: Sample a set V (cid:48)

Original: ( x ) of attributes uniformly at random from { v c (cid:48) , 1 , v c (cid:48) , 2 } c (cid:48) (cid:54) =
Revised: ( x ) of attributes uniformly at random from { v c (cid:48) , 1 , v c (cid:48) , 2 } c (cid:48) (cid:54) =

Original: y each with probability sC and denote V ( X ) =
Revised: y each with probability sC and denote V ( X ) =

Original: V (cid:48)(
Revised: V (cid:48)

Original:  x ) ∪ { v y, 1 , v y, 2 } as the set of attribute vectors used in data x .
Revised: ( x ) ∪ { v y, 1 , v y, 2 } as the set of attribute vectors used in data x .

Original: We take s = C 0 .
Revised: We take s = C 0 .

Original: Intuition:
Revised: Intuition:

Original: V (cid:48)
Revised: V (cid:48)

Original: ( x ) correspond to the ambiguous attributes present in x , such as the cat whose eyes look like car headlights.
Revised: ( x ) correspond to the ambiguous attributes present in x , such as the cat whose eyes look like car headlights.

Original: These V (cid:48) are crucial in our proofs as the FKD regularisation between ambiguous images ensures that the student learns attributes it would have otherwise missed.
Revised: These V (cid:48) are crucial in our proofs as the FKD regularisation between ambiguous images ensures that the student learns attributes it would have otherwise missed.

Original: For each v ∈ V ( x ) , pick C p (where C p is a global constant) many disjoint patches in [ P ] uniformly at random and denote this set as P v ( x ) .
Revised: For each v ∈ V ( x ) , pick C p (where C p is a global constant) many disjoint patches in [ P ] uniformly at random and denote this set as P v ( x ) .

Original: Denote P ( x ) = ∪ v ∈V ( x ) P v ( x ) : all other patches p (cid:54)∈ P ( x ) will contain noise only.If x is single view, pick a value
Revised: Denote P ( x ) = ∪ v ∈V ( x ) P v ( x ) : all other patches p (cid:54)∈ P ( x ) will contain noise only.

Original:  ˆ l = ˆ l ( x ) ∈{ 1 , 2 } uniformly at random.
Revised: If x is single view, pick a value ˆ l = ˆ l ( x ) ∈{ 1 , 2 } uniformly at random.

Original: ˆ l corresponds to the attribute v y, ˆ l that is present in x , with v y, 3 − ˆ l missing.
Revised: ˆ l corresponds to the attribute v y, ˆ l that is present in x , with v y, 3 − ˆ l missing

Original: For each p ∈ P v ( x ) for some v ∈ V ( x ) : 
Revised: For each p ∈ P v ( x ) for some v ∈ V ( x ) : [Equation] p

Original: [Equation]
Revised: [Equation]

Original:  where  ξ p ∼ N (0 , σ 2 p I d ) is independent random noise, with σ p = 1 √ d polylog ( C ) .
Revised: z p [Equation] where α p,v (cid:48) ∈ [0 , 1 C 1 . 5 ] represents feature noise and ξ p ∼ N (0 , σ 2 p I d ) is independent random noise, with σ p = 1 √ d polylog ( C ) .

Original: The coefﬁcients z p ≥ 0 satisfy:
Revised: The coefﬁcients z p ≥ 0 satisfy:

Original: (a) If x is multi view, • When v ∈ { v y, 1 , v y, [Equation]
Revised: (a) If x is multi view, • When v ∈ { v y, 1 , v y, 2 } ,

Original: [Equation]
Revised: [Equation]

Original: Intuition:
Revised: Intuition

Original: These conditions ensure that both attributes for class y are equally likely to be learnt, averaged over different random initialisations of parameters.
Revised: These conditions ensure that both attributes for class y are equally likely to be learnt, averaged over different random initialisations of parameters.

Original: • When v ∈ V (cid:48) ( x ) ,
Revised: • When v ∈ V (cid:48) ( x [Equation]

Original: [Equation]
Revised: [Equation]

Original: (b) If x is single view, • When v = v y, ˆ l , (cid:80) p ∈P v ( x ) z p 
Revised: (b) If x is single view, • When v = v y, ˆ l , (cid:80) p ∈P v ( x ) z p =•

Original: [Equation]

Original: • When v = v y, 3 − ˆ l , (cid:80) p ∈P v ( x ) z p =
Revised: When v = v y, 3 − ˆ l , (cid:80) p ∈P v ( x ) z p =

Original: C − 0 .•
Revised: C − 0 .•

Original: When v ∈ V (cid:48)
Revised: When v ∈ V (cid:48)

Original: ( x ) , (cid:80) p ∈P v ( x ) z p = Γ where Γ = 1 polylog ( C ) Intuition: This is where the single view name comes from, as C − 0 .
Revised: ( x ) , (cid:80) p ∈P v ( x ) z p = Γ where Γ = 1 polylog [Equation] C ) Intuition: This is where the single view name comes from, as C − 0 .

Original: 2 (cid:28) 

Revised: 2 (cid:28)

Original: we see that v y, 3 − ˆ l is barely present in x . 6.
Revised: we see that v y, 3 − ˆ l is barely present in x . 6.

Original: For each p ∈ [ P ] \P v ( x ) :
Revised: For each p ∈ [ P ] \P v ( x ) :

Original: [Equation]
Revised: [Equation]

Original: where ξ p ∼ N (0 , 1 Cd I d ) is independent random noise. 
Revised: for feature noise α p,v (cid:48) ∈ [0 , 1 C 1 . 5 ] and ξ p ∼ N (0 , 1 Cd I d ) is independent random noise. Intuition: One can think of the zero feature noise setting α p,v (cid:48) = 0 ∀ p, v (cid:48) for simplicity.

Revised: But the general formulation above renders the problem unlearnable by linear classiﬁer, as the maximum permissible feature noise across patches dominates the minimum possible signal:

Revised: [Equation]

Original: Remark It is possible to allow more relaxed assumptions, e.g. on z p , as in Allen-Zhu & Li (2020).
Revised: Remark It is possible to allow more relaxed assumptions, e.g. on z p , as in Allen-Zhu & Li (2020).

Original: Training data Recall we have D = µ D s
Revised: Training data Recall we have D = µ D s

Original: + (1 − µ ) D m , so that a proportion (1 − µ ) of the data is multi-view.
Revised: + (1 − µ ) D m , so that a proportion (1 − µ ) of the data is multi-view.

Original: Our training data, ˆ D , is N independent samples from D .
Revised: Our training data, ˆ D , is N independent samples from D .

Original: Letting ˆ D
Revised: Letting ˆ D

Original: =
Revised: =

Original: ˆ D m ∪ ˆ D s denote the split into multi and single view training data.
Revised: ˆ D m ∪ ˆ D s denote the split into multi and single view training data.

Original: We let µ = 1 poly ( C ) and we suppose | N | =
Revised: We let µ = 1 poly ( C ) and we suppose | N | =

Original: C 1 .
Revised: C 1 .

Original: 2 µ so that each label c appears at least Ω(1) in ˆ D s . 
Revised: 2 µ so that each label c appears at least Ω(1) in ˆ D s . [Figure]

Original: B. S MOOTHED R E LU CNN
Revised: B. S MOOTHED R E LU CNN

Original: Our theoretical analysis considers a single hidden layer CNN with sum-pooling, f , such that for class c :
Revised: Our theoretical analysis considers a single hidden layer CNN with sum-pooling, f , such that for class c :

Original: [Equation]
Revised: [Equation]

Original: For a threshold (cid:37) = 1 polylog ( C ) , we deﬁne the smoothed ReLU function as:
Revised: For a threshold (cid:37) = 1 polylog ( C ) , we deﬁne the smoothed ReLU function as:

Original: [Equation]
Revised: [Equation]

Original: which has continuous monotonic gradient denoted as (cid:94) ReLU (cid:48) .0.4 x 0. y
Revised: which has continuous monotonic gradient denoted as (cid:94) ReLU (cid:48) .

Original: B. H YPERPARAMETER VALUES IN SETUP
Revised: B. H YPERPARAMETER VALUES IN SETUP

Original: Hyperparameter values used in the theoretical setup are given in Table 3.[Table] [Table]
Revised: Hyperparameter values used in the theoretical setup are given in Table 3.

Original: B. S TANDARD TRAINING
Revised: B. S TANDARD TRAINING

Original: We deﬁne our empirical loss by
Revised: We deﬁne our empirical loss by:

Original: L = 1 N (cid:88) i ∈

Original: [ N ]
Revised: [Equation]

Original: L ( f ( x i ) , y i ) where L is the cross entropy loss deﬁned in Eq.
Revised: where L is the cross entropy loss deﬁned in Eq.

Original: We randomly initialise parameters θ 0 c,r i.i.d.
Revised: We randomly initialise parameters θ 0 c,r i.i.d.

Original: ∼ N (0 , σ 20 ) for σ 20 = 1 C .
Revised: ∼ N (0 , σ 20 ) for σ 20 = 1 C .

Original: Standard training in the theoretical analysis comprises of full-batch gradient descent on the empirical loss L with learning rate η ≤ [Equation] and for T = [Equation]
Revised: Standard training in the theoretical analysis comprises of full-batch gradient descent on the empirical loss L with learning rate η ≤ 1 poly ( C ) and for T = poly ( C ) η iterations.

Original: C ) η iterations.

Original: C P ROOF OF T HEOREM
Revised: C P ROOF OF T HEOREM

Original: We restate Theorem 2, where recall E denotes the teacher ensemble size and C is the number of classes: 
Revised: We restate Theorem 2, where recall E denotes the teacher ensemble size and C is the number of classes: [Table] C

Original:  Theorem 2 (FKD improves student generalisation and is better with larger ensemble) .
Revised: [Table] Theorem 2 (FKD improves student generalisation and is better with larger ensemble) .

Original: Given an arbitrary (cid:15) > 0 .
Revised: Given an arbitrary (cid:15) > 0 .

Original: For any ensemble size E of teacher NNs trained as in Theorem 1 and sufﬁciently many classes C , for m = polylog ( C ) , with learning rate η ≤ 1 poly ( C ) , and training time T ∗ = poly ( C ) η , the ensemble teacher knowledge can be distilled into a single student model f
Revised: For any ensemble size E of teacher NNs trained as in Theorem 1 and sufﬁciently many classes C , for m = polylog ( C ) [Table] with learning rate η ≤ 1 poly ( C ) , and training time T ∗ = poly ( C ) η [Table]

Revised: the ensemble teacher knowledge can be distilled into a single student model f

Original: ( T ∗ ) using only teacher feature kernel k T , Eq.
Revised: ( T ∗ ) using only teacher feature kernel k T , Eq.

Original: (7) , such that with probability at least 1 − e − Ω( log 2 ( C )) :
Revised: (7) , such that with probability at least 1 − e − Ω( log 2 ( C )) :

Original: • Training accuracy is perfect:
Revised: • Training accuracy is perfect:

Original: For all ( x , y ) ∈ ˆ D , y = argmax c
Revised: For all ( x , y ) ∈ ˆ D , y = argmax c

Original: f
Revised: f

Original: ( T ∗ ) c ( x ) .
Revised: ( T ∗ ) c ( x ) .

Original: • Test accuracy is good: P (x , y ) ∼D (cid:2) y (cid:54) = argmax c f ( T ∗ ) c
Revised: • Test accuracy is good: P (

Original: [Equation]
Revised: [Equation]

Original: Outline of proof of Theorem
Revised: Outline of proof of Theorem

Original: We ﬁrst analyse the feature kernel k for a single trained model, in App.
Revised: We ﬁrst analyse the feature kernel k for a single trained model, in App.

Original: C.1.
Revised: C.1.

Original: We then extend our results to an ensembled teacher model feature kernel in App.
Revised: We then extend our results to an ensembled teacher model feature kernel in App.

Original: C.2.
Revised: C.2.

Original: We next outline our kernel distillation training scheme and provide a key result concerning how the student’s parameters become increasingly correlated with the teacher’s learnt views in Apps.
Revised: We next outline our kernel distillation training scheme and provide a key result concerning how the student’s parameters become increasingly correlated with the teacher’s learnt views in Apps.

Original: C.3 and C. 4.
Revised: C.3 and C. 4.

Original: We combine all threads to prove the ﬁnal result in App.
Revised: We combine all threads to prove the ﬁnal result in App.

Original: C.5.
Revised: C.5.

Original: C. F EATURE KERNEL FOR A SINGLE TRAINED MODEL
Revised: C

Revised: F EATURE KERNEL FOR A SINGLE TRAINED MODEL

Original: To prove Theorem 2, we ﬁrst calculate the feature kernel k ∗ for a single trained model with trained parameters θ ∗ from initialisation θ 0 .
Revised: To prove Theorem 2, we ﬁrst calculate the feature kernel k ∗ for a single trained model with trained parameters θ ∗ from initialisation θ 0 .

Original: The point of this exercise is to show that the feature kernel k ∗ is able to detect whether two inputs x , x (cid:48) share a common attribute which is in the subset of attributes that was learnt by the trained parameters θ ∗ , due to random correlation with initialised parameters θ 0 .
Revised: The point of this exercise is to show that the feature kernel k ∗ is able to detect whether two inputs x , x (cid:48) share a common attribute which is in the subset of attributes that was learnt by the trained parameters θ ∗ , due to random correlation with initialised parameters θ 0 .

Original: This is formally shown in Lemma 1.
Revised: This is formally shown in Lemma 1.

Original: Recall the deﬁnition of the feature kernel k for our CNN architecture:
Revised: Recall the deﬁnition of the feature kernel k for our CNN architecture:

Original: [Equation]
Revised: [Equation]

Original: Let us ﬁrst make a few more useful deﬁnitions.
Revised: Let us ﬁrst make a few more useful deﬁnitions.

Original: For l = 1 , 2 , we have:
Revised: For l = 1 , 2 , we have:

Original: Deﬁnition 2.
Revised: Deﬁnition 2.

Original: [Equation]
Revised: [Equation]

Original: Intuition:
Revised: Intuition:

Original: Z c,l ( x ) is a scalar constant describing the strength, in x , of the presence of attribute v c,l (where c may either correspond to true class y or to a different class; the latter has the effect of producing ‘soft labels’ when using vanilla knowledge distillation).
Revised: Z c,l ( x ) is a scalar constant describing the strength, in x , of the presence of attribute v c,l (where c may either correspond to true class y or to a different class; the latter has the effect of producing ‘soft labels’ when using vanilla knowledge distillation).

Original: Deﬁnition 3.
Revised: Deﬁnition 3.

Original: [Equation]
Revised: [Equation]

Original: Deﬁnition 4.
Revised: Deﬁnition 4.

Original: [Equation]
Revised: [Equation]

Original: : Υ ∗ c,l , Φ ∗ c,l are both parameter-dependent, data-independent scalars that describe the amount that attribute v c,l has been ‘learnt’ by the network parameters.
Revised: Intuition: Υ ∗ c,l , Φ ∗ c,l are both parameter-dependent, data-independent scalars that describe the amount that attribute v c,l has been ‘learnt’ by the network parameters.

Original: Υ ∗ c,l appears in the feature kernel whereas Φ ∗ c,l appears in the function predictions f
Revised: Υ ∗ c,l appears in the feature kernel whereas Φ ∗ c,l appears in the function predictions f

Original: c directly, for our particular architecture in
Revised: c directly, for our particular architecture in

Original: Eq.
Revised: Eq.

Original: (4) (Allen-Zhu & Li, 2020).
Revised: (4) (Allen-Zhu & Li, 2020).

Original: Lemma 1.
Revised: Lemma 1.

Original: We have, for x , x (cid:48) ∼ ˆ D m , 
Revised: We have, for x , x (cid:48) ∼ ˆ D m , k ∗

Original: [Equation]
Revised: [Equation]

Original: where s M ( x , x(cid:48) )
Revised: where s M [Equation] x , x

Original:  = 
Revised: [Equation] ) = [Equation] [Equation]

Original: |{ ( c, l ) : v c,l ∈ V ( x ) ∩ V ( x (cid:48) ) and ( c, 3 − l ) / ∈ M}| is the number of shared attributes of x and x (cid:48) , that are also members of the set of attributes learnt by the single trained model with parameters θ ∗ , and M is deﬁned in Fact A.e .
Revised: c, l ) : [Equation] c,l ∈ V ( x ) ∩ V ( x (cid:48) ) and ( c, 3 − l ) / ∈ M}| is the number of shared attributes of x and x (cid:48) , that are also members of the set of attributes learnt by the single trained model with parameters θ ∗ , and M is deﬁned in Fact A.e below.

Original: Proof.
Revised: Proof.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: First, deﬁne the hidden-layer activations for class c and channel r to be:
Revised: First, deﬁne the hidden-layer activations for class c and channel r to be:

Original: Deﬁnition 5.
Revised: Deﬁnition 5.

Original: [Equation]
Revised: [Equation]

Original: We know from Theorem C.2.
Revised: We know from Theorem C.2.

Original: of Allen-Zhu & Li (2020)
Revised: of Allen-Zhu & Li (2020)

Original: that, for multi-view x ∈ ˆ D m and some c ∈ [ C ] :
Revised: that, for multi-view x ∈ ˆ D m and some c ∈ [ C ] :

Original: Fact A.a For every p ∈ P v c,l ( x ) and for l = 1 , 2 , we have: (cid:104) θ ∗ [Equation]
Revised: Fact A.a For every p ∈ P v c,l ( x ) and for l = 1 , 2 , we have: (cid:104) θ ∗ c,r , x p (cid:105) = (cid:104) θ ∗ c,r , v c,l (cid:105)

Original:  ˜ o
Revised: z p ± ˜ o

Original: ( σ 0 )
Revised: ( σ 0 )

Original: Fact A.b
Revised: Fact A.b

Original: For every p ∈ P ( x ) \ ( P v c, 1 ( x ) ∪ P v c, 2 ( x )) , we have |(cid:104) θ ∗ c,r , x p (cid:105)| ≤ ˜ O
Revised: For every p ∈ P ( x ) \ ( P v c, 1 ( x ) ∪ P v c, 2 ( x )) , we have |(cid:104) θ ∗ c,r , x p (cid:105)| ≤ ˜ O

Original: ( σ 0 )
Revised: ( σ 0 )

Original: Fact A.c For every p ∈
Revised: Fact A.c For every p ∈

Original: [ P ] \P ( x ) , we have |(cid:104) θ ∗ c,r , x p (cid:105)| ≤ ˜ O
Revised: [ P ] \P ( x ) , we have |(cid:104) θ ∗ c,r , x p (cid:105)| ≤ ˜ O

Original: ( σ 0 √ C )
Revised: ( σ 0 √ C )

Original: Fact A.d
Revised: Fact A.d

Original: For every r ∈ [ m ] \M 0 c , every l ∈ [2] , it holds that (cid:104) θ ∗ c,r , v c,l (cid:105) ≤ ˜ O ( σ 0 ) , where:
Revised: For every r ∈ [ m ] \M 0 c , every l ∈ [2] , it holds that (cid:104) θ ∗ c,r , v c,l (cid:105) ≤ ˜ O ( σ 0 ) , where:

Original: [Equation]
Revised: [Equation]

Revised: +

Original: (cid:27) .
Revised: [Equation]

Original: Note from Proposition B.1 of Allen-Zhu & Li (2020), that m 0 def = |M 0 c | =
Revised: Note from Proposition B.1 of Allen-Zhu & Li (2020), that m 0 def = |M 0 c | =

Original: O ( log 5 ( C )) with probability at least 1 − e − Ω( log 5 ( C )) .
Revised: O ( log 5 ( C )) with probability at least 1 − e − Ω( log 5 ( C )) .

Original: Intuition:
Revised: Intuition:

Original: M 0 c denotes the key channels in [ m ] which have ‘won the lottery’ and are relevant for class c , in that in the C → ∞ limit the predictions for f c are the same as if one forgets the other channels, as shown in Allen-Zhu & Li (2020).
Revised: M 0 c denotes the key channels in [ m ] which have ‘won the lottery’ and are relevant for class c , in that in the C → ∞ limit the predictions for f c are the same as if one forgets the other channels, as shown in Allen-Zhu & Li (2020).

Original: Fact A.e
Revised: Fact A.e

Original: For every p ∈ P v c,l ( x ) and r ∈ [ m ] , if ( c, 3 − l ) ∈ M , then we have: |(cid:104) θ ∗ c,r , x p (cid:105)| ≤ ˜ O
Revised: For every p ∈ P v c,l ( x ) and r ∈ [ m ] , if ( c, 3 − l ) ∈ M , then we have: |(cid:104) θ ∗ c,r , x p (cid:105)| ≤ ˜ O

Original: ( σ 0 ) , where:
Revised: ( σ 0 ) , where:

Original: [Equation]
Revised: [Equation]

Original: Intuition:
Revised: Intuition:

Original: M denotes the  attributes v c,l which are more likely to be learnt by the NN parameters (compared to their fellow class attributes v c, 3 − l ) because of correlations of the initial parameters with such attributes.
Revised: M denotes the data attributes v c,l which are more likely to be learnt by the NN parameters (compared to their fellow class attributes v c, 3 − l ) because of correlations of the initial parameters with such attributes.

Original: So, Fact A.e is saying that if ( c, 3 − l ) ∈ M , then the attribute ( c, l ) is not learnt at all during standard single model training.
Revised: So, Fact A.e is saying that if ( c, 3 − l ) ∈ M , then the attribute ( c, l ) is not learnt at all during standard single model training.

Original: Thus, 
Revised: Thus, Ψ c,r ( x ) = 2 (cid:88) l = [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: First, note that |P v c (cid:48) ,l (cid:48)
Revised: First, note that |P v c (cid:48) ,l (cid:48)

Original: ( x ) | =
Revised: ( x ) | =

Original: C p is constant ∀ c (cid:48) , l (cid:48) .
Revised: C p is constant ∀ c (cid:48) , l (cid:48) .

Original: From Fact A.b , Eq.
Revised: From Fact A.b , Eq.

Original: (14) can be easily seen to be ˜ O
Revised: (14) can be easily seen to be ˜ O

Original: σ 40 s ) =
Revised: σ 40 s ) =

Original: ˜ O ( C − 1 . 8 ) as σ 20 = 1 C and s = C 0 . 2 , and likewise Eq.
Revised: ˜ O ( C − 1 . 8 ) as σ 20 = 1 C and s = C 0 . 2 , and likewise Eq.

Original: (15) can be seen to be ˜ O
Revised: (15) can be seen to be ˜ O

Original: (( σ 0 √ C ) 4 P ) =
Revised: (( σ 0 √ C ) 4 P ) =

Original: ˜ O ( C − 2 ) as P = C 2 , by Fact A.c .
Revised: ˜ O ( C − 2 ) as P = C 2 , by Fact A.c .

Original: Summing these equations over m and C will be bounded above by O ( 1 polylog ( C ) ) .
Revised: We note here that summing these equations over m and C will be bounded above by O ( 1 polylog ( C ) ) .

Original: Now, let’s consider Eq.
Revised: Now, let’s consider Eq.

Original: Let notation v c, 1 , v c, 2 ∈ S denote either v c, 1 or v c, 2 ∈ S for some set
Revised: Let notation v c, 1 , v c, 2 ∈ S denote either v c, 1 or v c, 2 ∈ S for some set

Original: S , and v c, 1 , v c, 2 / ∈ S denote neither v c, 1 nor v c, 2 ∈ S .
Revised: S , and v c, 1 , v c, 2 / ∈ S denote neither v c, 1 nor v c, 2 ∈ S .

Original: There are three cases to consider:
Revised: There are three cases to consider:

Original: If v c, 1 , v c, 2 / ∈ V ( x ) , then Eq.
Revised: If v c, 1 , v c, 2 / ∈ V ( x ) , then Eq.

Original: (13) is zero.
Revised: (13) is zero.

Original: Else if ∃ l ∈ [2] such that v c,l ∈ V ( x ) and ( c, 3 − l ) ∈ M , then by Fact A
Revised: Else if ∃ l ∈ [2] such that v c,l ∈ V ( x ) and ( c, 3 − l ) ∈ M , then by Fact A

Original: e , we have Eq.
Revised: e , we have Eq.

Original: (13) is ˜ O ( σ 40 )=
Revised: (13) is ˜ O ( σ 40 )=

Original: ˜ O ( C − 2 ) .
Revised: ˜ O ( C − 2 ) .

Original: Intuition:
Revised: Intuition:

Original: This setting is when only attribute ( c, l ) appears in x , but attribute ( c, 3 − l ) was dominant at initialisation so ( c, l ) has not been learnt by θ ∗ .
Revised: This setting is when only attribute ( c, l ) appears in x , but attribute ( c, 3 − l ) was dominant at initialisation so ( c, l ) has not been learnt by θ ∗ .

Original: Else we have that
Revised: Else, we have that:

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: Putting this all together with Fact A.a , we see that 
Revised: Putting this all together with Fact A.a , we see that Ψ

Original: [Equation]
Revised: [Equation]

Original: Now if r / ∈ M 0 c  and ( c, 1) , ( c, 2) ∈ s M ( x , x ) ,  by Fact A.d , we have that Ψ c,r ( x )=
Revised: Now, if r / ∈ M 0 c (i.e. neuron r is not dominant at initialisation), and ( c, 1) , ( c, 2) ∈ s M ( x , x ) , the by Fact A.d , we have that Ψ c,r ( x )=

Original: ˜ O ( C − 1 . 8 ) .
Revised: ˜ O ( C − 1 . 8 ) .

Original: On the other hand, recall (cid:37) = 1 polylog ( C ) and m 

Revised: On the other hand, recall (cid:37) = 1 polylog ( C ) and m

Original: =
Revised: =

Original: O ( log 5 ( C )) .
Revised: O ( log 5 ( C )) .

Original: And also, recall that if z > (cid:37) , then (cid:94) ReLU ( z )
Revised: And also, recall that if z > (cid:37) , then (cid:94) ReLU ( z )

Original: =
Revised: =

Original: z + O ( (cid:37) ) .
Revised: z + O ( (cid:37) ) .

Original: Moreover, by Claim C.11 of Allen-Zhu & Li (2020) we know that ∃ r (cid:48) , l (cid:48) such that (cid:104) θ ∗ c,r (cid:48) , v c,l (cid:48) (cid:105) =
Revised: Moreover, by Claim C.11 of Allen-Zhu & Li (2020) we know that ∃ r (cid:48) , l (cid:48) such that (cid:104) θ ∗ c,r (cid:48) , v c,l (cid:48) (cid:105) =

Original: Ω( 1 m 0 ) .
Revised: Ω( 1 m 0 ) .

Original: Hence, for any r ∈ M 0 c , we know that either:
Revised: Hence, for any r ∈ M 0 c , we know that either:

Original: [Equation]
Revised: [Equation]

Revised: Ψ c,r ( x ) = 1 { ( c, 1) , ( c, 2)

Original: or
Revised: [Equation] s M ( x , x [Equation] O

Revised: [Equation]

Revised: O

Original: [Equation]
Revised: [Equation]

Original: Moreover, Ψ c,r ( x ) =
Revised: Moreover, Ψ c,r ( x ) =

Original: ˜ O (1) ∀ r , by e.g. Lemma C.21 of Allen-Zhu & Li (2020).
Revised: ˜ O (1) ∀ r , by e.g. Lemma C.21 of Allen-Zhu & Li (2020).

Original: And so it’s easy to see, for (cid:37) small enough we have:
Revised: And so it can be seen, for (cid:37) small enough we have:

Original: [Equation]
Revised: [Equation]

Original: where we also use Fact A.e above, such that it is not possible for both (cid:104) θ ∗ c,r , v c,l (cid:105) + and (cid:104) θ ∗ c,r , v c, 3 − l (cid:105) + to be large.
Revised: where we also use Fact A.e above, such that it is not possible for both (cid:104) θ ∗ c,r , v c,l (cid:105) + and (cid:104) θ ∗ c,r , v c, 3 − l (cid:105) + to be large.

Original: 9 Note also from e.g. the proof of Allen-Zhu & Li (2020) Theorem 1, that if ( c, 3 − l ) / ∈ M , then max r (cid:104) θ ∗ c,r , v c,l (cid:105) = ˜Θ(1) .
Revised: 8 Note also from e.g. the proof of Allen-Zhu & Li (2020) Theorem 1, that if ( c, 3 − l ) / ∈ M , then max r (cid:104) θ ∗ c,r , v c,l (cid:105) = ˜Θ(1) .

Original: Thus, we see that the contribution to k from the m class- c channels is:[Equation] [Equation] ˜ O
Revised: Thus, we see that the contribution to k from the m class- c channels/neurons is:

Original: ( 1 C 1 .

Original: [Equation]
Revised: [Equation]

Original: else (17)

Original: Summing k c over [ C ] completes the proof of the lemma.
Revised: Summing k c over [ C ] completes the proof of the lemma.

Original: We now use Lemma 1 to analyse k ∗ and ρ ∗ ( x , x (cid:48) )
Revised: We now use Lemma 1 to analyse k ∗ and ρ ∗ ( x , x (cid:48) )

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: First we look at Υ ∗ c,l .
Revised: First we look at Υ ∗ c,l .

Original: Recall that (cid:104) θ ∗ c,r , v c,l (cid:105) =
Revised: Recall that (cid:104) θ ∗ c,r , v c,l (cid:105) =

Original: ˜ O (1) for all c ∈ [ C ] , l ∈ [2] , r ∈ [ m ] , and likewise so is M 0 c .
Revised: ˜ O (1) for all c ∈ [ C ] , l ∈ [2] , r ∈ [ m ] , and likewise so is M 0 c .

Original: More speciﬁcally,
Revised: More speciﬁcally,

Original: • If ( c, l ) ∈ M , then we have from the proof of Theorem 1 in Allen-Zhu & Li (2020) that (cid:80) r (cid:104) θ ∗ c,r , v c,l (cid:105) + ≥ Ω
Revised: • If ( c, l ) ∈ M , then we have from the proof of Theorem 1 in Allen-Zhu & Li (2020) that (cid:80) r (cid:104) θ ∗ c,r , v c,l (cid:105) + ≥ Ω

Original: ( log ( C )) , and so max r (cid:104) θ ∗ c,r , v c,l (cid:105)+ = ˜Θ(1) .
Revised: ( log ( C )) , and so max r (cid:104) θ ∗ c,r , v c,l (cid:105)

Revised: [Equation]

Original: • If neither ( c, 1) nor ( c, 2) are in M , then Claim C.10 of Allen-Zhu & Li (2020) shows us that both max r (cid:104) θ ∗ c,r , v c, 1 (cid:105) + , max r (cid:104) θ ∗ c,r , v c, 2 (cid:105) + are ˜Θ(1) .
Revised: • If neither ( c, 1) nor ( c, 2) are in M , then Claim C.10 of Allen-Zhu & Li (2020) shows us that both max r (cid:104) θ ∗ c,r , v c, 1 (cid:105) + , max r (cid:104) θ ∗ c,r , v c, 2 (cid:105) + are ˜Θ(1) .

Original: Combining these facts, we have that:
Revised: Combining these facts [Equation] we have that:

Original: [Equation]
Revised: [Equation]

Original: and so from Lemma 1, we have:
Revised: and so from Lemma [Equation] we have:

Revised: [Equation]

Revised: [Equation]

Revised: Finally

Original: [Equation]
Revised: [Equation]

Original: Finally, we arrive at an expression for the correlation kernel ρ ∗ of a trained single model by
Revised: we arrive at an expression for the correlation kernel ρ ∗ of a trained single model by

Original: [Equation]
Revised: [Equation]

Original:  where we deﬁne that s M ( x )
Revised: (21) where we deﬁne that s M ( x )

Original: =
Revised: =

Original: s M ( x , x ) is the number of attributes in x that have been learnt by the trained network θ ∗ , and is s (1 ± o (1)) with high probability.
Revised: s M ( x , x ) is the number of attributes in x that have been learnt by the trained network θ ∗ , and is s (1 ± o (1)) with high probability.

Original: Intuition:
Revised: Intuition:

Original: Compare ρ ∗ in Eq.
Revised: Compare ρ ∗ in Eq.

Original: (21) to the ‘soft’ probability labels p τ ∈ R C of Allen-Zhu & Li (2020) (Claim
Revised: (21) to the ‘soft’ probability labels p τ ∈ R C of Allen-Zhu & Li (2020) (Claim

Original: F.4) with temperature τ =1 log 2 ( C [Equation] : p
Revised: F.4) with temperature τ =

Original: [Equation]
Revised: [Equation]

Original: if

Original: [Equation]
Revised: [Equation]

Original:  or v [Equation] is in V ( x ) else where s ( x ) is the number of indices c
Revised: if v c, 1 or v c, 2 is in V ( x ) else where s ( x ) is the number of indices c

Original: ∈
Revised: ∈

Original: [ C ] such that v c, 1 or v c, 2 is in V ( x ) .
Revised: [ C ] such that v c, 1 or v c, 2 is in V ( x ) .

Original: Note, the setting of Allen-Zhu & Li (2020) is with a large ˜Ω(1) ensemble, so every attribute is learnt (akin to M being empty for us).
Revised: Note, the setting of Allen-Zhu & Li (2020) is with a large ˜Ω(1) ensemble, so every attribute is learnt (akin to M being empty for us).

Original: In the case that M = {} being empty, if x (cid:54) =
Revised: In the case that M = {} being empty, if x (cid:54) =

Original: x (cid:48) and they share at least one feature, then from App.
Revised: x (cid:48) and they share at least one feature, then from App.

Original: B.1.1 with high probability they will share exactly one feature, so that s M ( x , x (cid:48) )
Revised: B.1.1 with high probability they will share exactly one feature, so that s M ( x , x (cid:48) )

Original: = 1 .
Revised: = 1 .

Original: Moreover, s ( x ) =
Revised: Moreover, s ( x ) =

Original: s M ( x ) ≈ s M ( x (cid:48) ) , hence we see that Eq.
Revised: s M ( x ) ≈ s M ( x (cid:48) ) , hence we see that Eq.

Original: (21) matches roughly with p τc ( x ) , but without the need for a temperature hyperparameter.
Revised: (21) matches roughly with p τc ( x ) , but without the need for a temperature hyperparameter.

Original: We see that vanilla KD learns new attributes by comparing a single data point x between classes, and giving larger target labels to the classes where ambiguous attributes learnt by the teacher are present in x .
Revised: We see that vanilla KD learns new attributes by comparing a single data point x between classes, and giving larger target labels to the classes where ambiguous attributes learnt by the teacher are present in x .

Original: On the other hand, ρ ∗ gives higher values to data pairs x , x (cid:48) that share attributes that have been learnt by the trained model, and as we will see later this is how FKD learns new attributes in the student. 
Revised: On the other hand, ρ ∗ gives higher values to data pairs x , x (cid:48) that share attributes that have been learnt by the trained model, and as we will see later this is how FKD learns new attributes in the student. [Equation]

Original: C. E NSEMBLED TEACHER
Revised: E NSEMBLED TEACHER

Original: To summarise what we have done so far in this section, we have seen in Eqs.
Revised: To summarise what we have done so far in App. C.1, we have seen in Eqs.

Original: (20) and (21) that it is possible, for a single trained model θ ∗ , to simplify both the feature kernel k ∗ ( x , x (cid:48) ) and correlation kernel ρ ∗ ( x , x (cid:48) ) in terms of the number of shared attributes between x , x (cid:48) which are also learnt by the trained model.
Revised: (20) and (21) that it is possible, for a single trained model θ ∗ , to simplify both the feature kernel k ∗ ( x , x (cid:48) ) and correlation kernel ρ ∗ ( x , x (cid:48) ) in terms of the number of shared attributes between x , x (cid:48) which are also learnt by the trained model.

Original: The set of attributes learnt by the  trained model is captured by the set
Revised: The set of attributes learnt by the single trained model is captured by the set

Original: M =

Original: [Equation]
Revised: [Equation]

Original: where θ 0 was the random parameter initialisation for θ ∗ .
Revised: where θ 0 was the random parameter initialisation for θ ∗ .

Original: From Fact A.e , we know that if ( c, 3 − l ) ∈M , then the attribute v c,l has not been learnt by the network.
Revised: From Fact A.e , we know that if ( c, 3 − l ) ∈M , then the attribute v c,l has not been learnt by the network.

Original: Consider now an ensemble of E = Θ(1) independently trained networks, { θ ∗ e } Ee =1 , with an averaged feature kernel:
Revised: Consider now an ensemble of E = Θ(1) independently trained networks, { θ ∗ e } Ee =1 , with an averaged feature kernel:

Original: [Equation]
Revised: [Equation]

Original: Suppose { θ e, 0 } Ee =1 denotes the corresponding independent parameter initialisations.
Revised: Suppose { θ e, 0 } E e =1 denotes the corresponding independent parameter initialisations.

Original: Then, for each e ∈ [ E ] , let us deﬁne:
Revised: Then, for each e ∈ [ E ] [Equation] let us deﬁne:

Original: [Equation]
Revised: [Equation]

Original: Note that these M e are completely independent sets due to the independent initialisations, and also by Proposition B.2 of Allen-Zhu & Li (2020), we know that
Revised: Note that these M e are completely independent sets due to the independent initialisations, and also by Proposition B.2 of Allen-Zhu & Li (2020), we know that

Original: [Equation]
Revised: [Equation]

Revised: c, 2) ∈ M e (cid:3) ≥ 1 − o (1) ∀ c

Revised: ∈ [ C ] , e ∈

Revised: [ E ] .

Original: Therefore,
Revised: Therefore,

Revised: C ≥ |M e | ≥ C (1 − o p (1))

Original: [Equation]
Revised: ∀ e ∈ [ E ] .

Original: Moreover [Equation] Eq.
Revised: Moreover, Eq.

Original: (11) & Proposition B.2 of Allen-Zhu & Li (2020) tell us that each of the two attributes v c, 1 , v c, 2 are equally likely to be in M e (and so learnt in the multi-view setup), This means that:
Revised: (11) & Proposition B.2 of Allen-Zhu & Li (2020) tell us that each of the two attributes v c, 1 , v c, 2 are equally likely to be in M e (and so learnt in the multi-view setup), This means that:

Original: [Equation]
Revised: [Equation]

Original: Deﬁne M T = (cid:84)
Revised: Deﬁne M T = (cid:84)

Original: Ee =1 M e .
Revised: Ee =1 M e .

Original: From Eq.
Revised: From Eq.

Original: (19) and the deﬁnition of k T , we see that:
Revised: (19) and the deﬁnition of k T , we see that:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where for the reader’s convenience, we redeﬁne:
Revised: where for the reader’s convenience, we redeﬁne:

Original: [Equation]
Revised: [Equation]

Original: Intuition:
Revised: Intuition:

Original: We see that only for those attributes ( c, l ) such that ( c, 3 − l )
Revised: We see that only for those attributes ( c, l ) such that ( c, 3 − l )

Original: ∈
Revised: ∈

Original: M T does the ensembled teacher k T miss the fact that we should have a strong ˜Θ(1) kernel value between x , x (cid:48) .
Revised: M T does the ensembled teacher k T miss the fact that we should have a strong ˜Θ(1) kernel value between x , x (cid:48) .

Original: This is when |V ( x ) ∩ V ( x (cid:48) )
Revised: This is when |V ( x ) ∩ V ( x (cid:48) )

Original: | = { v c,l } is non-empty (or in other words, when s M T ( x , x (cid:48) )
Revised: | = { v c,l } is non-empty (or in other words, when s M T ( x , x (cid:48) )

Original: (cid:54) =
Revised: (cid:54) =

Original: |V ( x ) ∩V
Revised: |V ( x ) ∩V

Original: ( x (cid:48) ) | ), and hence there should be a large kernel value k T ( x , x (cid:48) ) .
Revised: ( x (cid:48) ) | ), and hence there should be a large kernel value k T ( x , x (cid:48) ) .

Original: So we see that only |M T | of the attributes are not learnt by the teacher, which is a fraction |M
Revised: So we see that only |M T | of the attributes are not learnt by the teacher, which is a fraction |M

Original: T | 2 C = 12 E (1 − o (1)) of all the attributes.
Revised: T | 2 C = 12 E (1 − o (1)) of all the attributes.

Original: These missed attributes are where the[-
 [Equation]-] 1  test error in Theorem 2 comes from (teacher ensemble of size E, and plus 1 for the attributes learnt from the student’s initialisation too).
Revised: These missed attributes are where the[-
 [Equation]-] 1 2 E +1 test error in Theorem 2 comes from (teacher ensemble of size E, and plus 1 for the attributes learnt from the student’s initialisation too).

Original: What’s more, we can decompose the teacher’s feature kernel k T = (cid:80)
Revised: What’s more, we can decompose the teacher’s feature kernel k T = (cid:80)

Original: c
Revised: c

Original: k T c into contributions k T c from each class c , like in Eq.
Revised: k T c into contributions k T c from each class c , like in Eq.

Original: From Eq.
Revised: From Eq.

Original: (17), we see that the contribution to the teacher’s feature kernel from class , for x (cid:54) =
Revised: (17), we see that the contribution to the teacher’s feature kernel from class , for x (cid:54) =

Original: x (cid:48) ;
Revised: x (cid:48) ;

Original: [Equation]
Revised: [Equation]

Original: else, (24) is able to decipher between whether or not x , x (cid:48) share an attribute from class c for all attributes apart from those ( c, l ) such that ( c, 3 − l ) ∈ M T .
Revised: is able to decipher between whether or not x , x (cid:48) share an attribute from class c for all attributes apart from those ( c, l ) such that ( c, 3 − l ) ∈ M T .

Original: C. T RAINING SCHEME FOR FKD
Revised: C. T RAINING SCHEME FOR FKD

Original: We note at this point that we are morally done in terms of proving Theorem 2, with Eqs.
Revised: We note at this point that we are morally done in terms of proving Theorem 2, with Eqs.

Original: (23) and (24), our key results telling us that the (ensemble) teacher kernel k T can identify when two inputs share common attributes that have been learnt by the (ensemble) teacher, and more speciﬁcally that k T c can do so when said common attribute is from class c .
Revised: (23) and (24), our key results telling us that the (ensemble) teacher kernel k T can identify when two inputs share common attributes that have been learnt by the (ensemble) teacher, and more speciﬁcally that k T c can do so when said common attribute is from class c .

Original: What remains is a repackaging of the proof techniques of Allen-Zhu & Li (2020) (particularly for their Theorem 4 regarding self-distillation), that knowledge distillation (this time only using feature kernels instead of temperature-scaled logits, and with explicit dependence on teacher ensemble size) can improve generalisation performance of a student.
Revised: What remains is a repackaging of the proof techniques of Allen-Zhu & Li (2020) (particularly for their Theorem 4 regarding self-distillation), that knowledge distillation (this time only using feature kernels instead of temperature-scaled logits, and with explicit dependence on teacher ensemble size) can improve generalisation performance of a student.

Original: For convenience, the theoretical analysis of Allen-Zhu & Li (2020) introduces some slight discrepancies between the actual practical weight updates of vanilla KD Hinton et al.
Revised: For convenience, the theoretical analysis of Allen-Zhu & Li (2020) introduces some slight discrepancies between the actual practical weight updates of vanilla KD Hinton et al.

Original: (2015), i.e. the gradients of:
Revised: (2015), i.e. the gradients of:

Original: [Equation]
Revised: [Equation]

Original: and the weight updates in their theoretical exposition.
Revised: and the weight updates in their theoretical exposition.

Original: Namely,1.
Revised: Namely,

Original: The authors assume that a temperature-dependent threshold caps the logits to give soft labels:
Revised: The authors assume that a temperature-dependent threshold caps the logits to give soft labels:

Original: [Equation]
Revised: [Equation]

Original: The authors truncate the negative part of the gradient of the KD regularisation to only encourage logits to increase not decrease, with weight updates for θ c,r on input x :
Revised: The authors truncate the negative part of the gradient of the KD regularisation to only encourage logits to increase not decrease, with weight updates for θ c,r on input x :

Original: [Equation]
Revised: [Equation]

Original: where p τ, T are the temperature-scaled teacher labels.
Revised: where p τ, T are the temperature-scaled teacher labels.

Original: The authors scale the output of both student and teacher models by a (polylogarithmic) factor, in order to ensure that both reach the threshold to give soft labels in Item 1.
Revised: The authors scale the output of both student and teacher models by a (polylogarithmic) factor, in order to ensure that both reach the threshold to give soft labels in Item 1.

Original: above.
Revised: above.

Original: Self-distillation (Furlanello et al., 2018; Zhang et al., 2019) distils a single teacher and a single student of same architecture into the student, like a size 2 ensemble.
Revised: Self-distillation (Furlanello et al., 2018; Zhang et al., 2019) distils a single teacher and a single student of same architecture into the student, like an ensemble of size 2 (student+teacher).

Original: Allen-Zhu & Li (2020) modify the training scheme for their theoretical analysis of self-distillation so that the student is ﬁrst trained on its own in order
Revised: Allen-Zhu & Li (2020) modify the training scheme for their theoretical analysis of self-distillation so that the student is ﬁrst trained on its own in order

Original: separate learning its own attributes/features from those of the teacher.
Revised: separate learning its own attributes/features from those of the teacher.

Original: Our analysis covers a similar scheme.
Revised: Our analysis covers a similar scheme.

Original: These modiﬁcations are justiﬁed in that they make the theoretical analysis more convenient, whilst illustrating the main mechanisms by which KD works, which is to share ‘dark knowledge’ that is held in the teacher (in the form of the multi-view attributes that have been acquired by the teacher due to its parameter initialisation), with the student.
Revised: These modiﬁcations are justiﬁed in that they make the theoretical analysis more convenient, whilst illustrating the main mechanisms by which KD works, which is to share ‘dark knowledge’ that is held in the teacher (in the form of the multi-view attributes that have been acquired by the teacher due to its parameter initialisation), with the student.

Original: In the same vein, we now introduce some modiﬁcations to the practical implementation of FKD we propose in Alg.
Revised: In the same vein, we now introduce some modiﬁcations to the practical implementation of FKD we propose in Alg.

Original: 1 to aid our theoretical analysis, and describe the main mechanisms by which FKD works, corroborating our initial analyses in Section 2 and App.
Revised: 1 to aid our theoretical analysis, and describe the main mechanisms by which FKD works, corroborating our initial analyses in Section 2 and App.

Original: A about how the feature kernel is a crucial object in any NN and captures all the ‘dark knowledge’ that a teacher network could possess in the multi-view data setting.
Revised: A about how the feature kernel is a crucial object in any NN and captures all the ‘dark knowledge’ that a teacher network could possess in the multi-view data setting.

Original: It is likely possible to extend our proof of Theorem 2 with different modiﬁcations/training schemes, but given that the focus of this work is to introduce FKD as a principled alternative to vanilla KD with certain advantages such as prediction-space independence, and that the multi-view setting we consider is a plausible simpliﬁcation of real world data (as demonstrated in Allen-Zhu & Li (2020)), we leave this to future work.
Revised: It is likely possible to extend our proof of Theorem 2 with different modiﬁcations/training schemes, but given that the focus of this work is to introduce FKD as a principled alternative to vanilla KD with certain advantages such as prediction-space independence, and that the multi-view setting we consider is a plausible simpliﬁcation of real world data (as demonstrated in Allen-Zhu & Li (2020)), we leave this to future work.

Original: We stress that any simpliﬁcations to the update rule in Alg. 1 for this section can be efﬁciently computed, only requiring access to pairwise evaluations of the student and teacher feature kernels, if need be.
Revised: We stress that any simpliﬁcations to the update rule in Alg. 1 for this section can be efﬁciently computed, only requiring access to pairwise evaluations of the student and teacher feature kernels, if need be.

Original: Modiﬁed training regime for FKD
Revised: Modiﬁed training regime for FKD

Original: We ﬁrst suppose that the student is trained as standard (as in App.
Revised: We ﬁrst suppose that the student is trained as standard (as in App.

Original: B.4) for T 1 = poly ( C ) η steps, and learns its own subset of attributes M S , dependent on its initialisation θ 0 s , before being trained with the FKD objective:
Revised: B.4) for T 1 = poly ( C ) η steps, and learns its own subset of attributes M S , dependent on its initialisation θ 0 s , before being trained with the FKD objective:

Original: Intuition:
Revised: Intuition:

Original: This mirrors the self-distillation setup of Allen-Zhu & Li  Theorem 4.
Revised: This mirrors the self-distillation setup of Allen-Zhu & Li (2020) Theorem 4.

Original: The idea being that the student ﬁrst learns M S before picking up the other attributes that the teacher has access to.
Revised: The idea being that the student ﬁrst learns M S before picking up the other attributes that the teacher has access to.

Original: For a given feature kernel k , we threshold the feature kernel k
Revised: For a given feature kernel k , we threshold the feature kernel k

Original: based on value, to deﬁne a modiﬁcation ˜
Revised: based on value, to deﬁne a modiﬁcation ˜

Original: k
Revised: k

Original: such that
Revised: such that

Original: [Equation]
Revised: [Equation]

Original: else Intuition This condition delineates between the setting where x , x (cid:48) share common attributes learnt by student parameters θ T 1 in the initial phase of training (i.e.  whether s M s ( x , x (cid:48) ) nonempty or empty).
Revised: This condition delineates between the setting where x , x (cid:48) share common attributes learnt by student parameters θ T 1 in the initial phase of training (i.e. delineates between whether s M s ( x , x (cid:48) ) nonempty or empty).

Original: To see this: note that
Revised: To see this: note that

Original: if v c,l ∈ V ( x ) ∩ V ( x (cid:48) ) and ( c, 3 − l ) / ∈ M S then we know from
Revised: if v c,l ∈ V ( x ) ∩ V ( x (cid:48) ) and ( c, 3 − l ) / ∈ M S then we know from

Original: Allen-Zhu & Li (2020) that Φ T 1 c,l ≥ Ω( log ( C )) .
Revised: Allen-Zhu & Li (2020) that Φ T 1 c,l ≥ Ω( log ( C )) .

Original: Hence max r (cid:104) θ ∗ c,r , v c,l (cid:105) ≥ Ω( log − 4 ( C )) as the number of active neurons m 0 =
Revised: Hence max r (cid:104) θ ∗ c,r , v c,l (cid:105) ≥ Ω( log − 4 ( C )) as the number of active neurons m 0 =

Original: |M 0 c | =
Revised: |M 0 c | =

Original: O ( log 5 C ) , and so it’s easy to see that for large enough m we have k ( x , x (cid:48) )
Revised: O ( log 5 C ) , and so it’s easy to see that for large enough m we have k ( x , x (cid:48) )

Original: ≥ 1 m 2 via
Revised: ≥ 1 m 2 via

Original: Lemma 1.
Revised: Lemma 1.

Original: On the other hand, if { v c,l } = V ( x ) ∩ V ( x (cid:48) ) and ( c, 3 − l ) ∈M S then from Lemmawe know that k ( x , x (cid:48) )
Revised: On the other hand, if { v c,l } = V ( x ) ∩ V ( x (cid:48) ) and ( c, 3 − l ) ∈M S then from Lemmawe know that k ( x , x (cid:48) )

Original: =
Revised: =

Original: ˜ O ( C − 0 . 8 ) (cid:28) 1 m 2 .
Revised: ˜ O ( C − 0 . 8 ) (cid:28) 1 m 2 .

Original: Similar to Allen-Zhu & Li (2020), we also truncate our FKD regularisation to only encourage kernel values to increase, and not decrease.
Revised: Similar to Allen-Zhu & Li (2020), we also truncate our FKD regularisation to only encourage kernel values to increase, and not decrease.

Original: For any input pair x 1 , x 2 , we have parameter update:
Revised: For any input pair x 1 , x 2 , we have parameter update:

Original: [Equation]
Revised: [Equation]

Revised: where recall Ψ c,r ( x ) def =

Revised: P

Original: [Equation]
Revised: [Equation]

Original: where recall
Revised: p =

Original: [Equation]
Revised: [Equation]

Original:  , x p (cid:105) ) so that ∇ θ c,r Ψ c,r ( x) =
Revised: ReLU ( (cid:104) θ c,r , x p (cid:105) ) so that ∇ θ c,r Ψ c,r ( x

Original: P (cid:88) p =(cid:94) ReLU (cid:48) ( (cid:104) θ c,r , x p (cid:105) ) x p and k c ( x , x (cid:48) ) def = m (cid:88) r
Revised: [Equation] =

Original: = Ψ c,r ( x )Ψ c,r ( x (cid:48) ) were deﬁned in Deﬁnition 5 and Eq.
Revised: [Equation] x p and [Equation] were deﬁned in Deﬁnition 5 and Eq.

Original: Intuition:
Revised: Intuition:

Original: If the loss was(
Revised: If the loss was

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: )then the gradient with respect to θ c,r would be:
Revised: then the gradient with respect to θ c,r would be:

Original: [Equation]
Revised: [Equation]

Original: so the only differences with Eq.
Revised: so the only differences with Eq.

Original: (25) are truncating (cid:0) ˜
Revised: (25) are truncating (cid:0) ˜

Original: k c ( x 1 , x 2 ) − ˜ k T c ( x 1 , x 2 ) (cid:1) − and also the thresholding to obtain ˜ k .
Revised: k c ( x 1 , x 2 ) − ˜ k T c ( x 1 , x 2 ) (cid:1) − and also the thresholding to obtain ˜ k .

Original: To summarise, after training the student on its own for T 1 steps (such that we are in the setting of
Revised: To summarise, after training the student on its own for T 1 steps (such that we are in the setting of

Original: Theorem 1) to reach parameters θ T 1 s , we update for T 2 = poly ( C ) η steps as (hiding S subscript):
Revised: Theorem 1) to reach parameters θ T 1 s , we update for T 2 = poly ( C ) η steps as (hiding S subscript):

Original: [Equation]
Revised: [Equation]

Original: C. F EATURE CORRELATION GROWTHS
Revised: C. F EATURE CORRELATION GROWTHS

Original: We now seek to analyse to what extent the attributes { v c,l } c,l are learnt during our T 2 FKD training steps.
Revised: We now seek to analyse to what extent the attributes { v c,l } c,l are learnt during our T 2 FKD training steps.

Original: The central objects describing how much v c,l has been learnt by parameters θ are:
Revised: The central objects describing how much v c,l has been learnt by parameters θ are:

Original: [Equation]
Revised: [Equation]

Original: and 
Revised: and Φ tc

Original: [Equation]
Revised: [Equation]

Original:  Intuition: Φ c,l is a data-independent quantity that reﬂects the strength of correlation with feature v c,l by parameters θ .
Revised: = [Equation] as well as Ψ c,r ( x ) as deﬁned above. Intuition: Φ c,l is a data-independent quantity that reﬂects the strength of correlation with feature v c,l by parameters θ .

Original: On the other hand, Ψ c,r ( x ) is a data-dependent quantity that reﬂects the activation of channel r for class c with input x .
Revised: On the other hand, Ψ c,r ( x ) is a data-dependent quantity that reﬂects the activation of channel r for class c with input x .

Original: Also recall that
Revised: Also recall that

Original: [Equation]
Revised: [Equation]

Original: and deﬁne V c,r,l ( x ) (which is convenient for calculating the size of gradient updates for θ c,r ):
Revised: and deﬁne V c,r,l ( x ) (which is convenient for calculating the size of gradient updates for θ c,r ):

Original: Deﬁnition 6.
Revised: Deﬁnition 6.

Original: [Equation]
Revised: [Equation]

Original: Like how Lemma 1 simpliﬁed the feature kernel in terms of data-dependent Z c,l ( x ) and dataindependent Υ c,l , we have a result from Allen-Zhu & Li (2020) to simplify function predictions f c in terms of Z c,l ( x ) and Φ c,l :
Revised: Like how Lemma 1 simpliﬁed the feature kernel in terms of data-dependent Z c,l ( x ) and dataindependent Υ c,l , we have a result from Allen-Zhu & Li (2020) to simplify function predictions f c in terms of Z c,l ( x ) and Φ c,l :

Original: Claim 1 (Claim F.7 from Allen-Zhu & Li (2020)) .
Revised: Claim 1 (Claim F.7 from Allen-Zhu & Li (2020)) .

Original: For every t ≤ T 1 + T 2 , every c ∈ [ C ] , every (
Revised: For every t ≤ T 1 + T 2 , every c ∈ [ C ] , every (

Original: x , y ) ∈
Revised: x , y ) ∈

Original: ˆ D (or every test sample ( x , y ) ∼ D with probability 1 − e  log 2 ( C ) ):f
Revised: ˆ D (or every test sample ( x , y ) ∼ D with probability 1 − e − Ω( log 2 ( C )) ):

Original: [Equation]
Revised: [Equation]

Original: We also have the following facts from Allen-Zhu & Li (2020) regarding the correlation of gradient ∇ θ c,r Ψ tc,r ( x ) with v c,l for ( x , y ) ∈
Revised: We also have the following facts from Allen-Zhu & Li (2020) regarding the correlation of gradient ∇ θ c,r Ψ tc,r ( x ) with v c,l for ( x , y ) ∈

Original: ˆ D , l ∈ [2] and r ∈ [ m ] :
Revised: ˆ D , l ∈ [2] and r ∈ [ m ] :

Original: Claim 2 (c.f. Claim F.6 of Allen-Zhu & Li (2020)) .
Revised: Claim 2 (c.f. Claim F.6 of Allen-Zhu & Li (2020)) .

Original: For every t ≤ T 1 + T 2 , for every ( x , y ) ∈ ˆ D , every c ∈ [ C ] , r ∈ [ m ] and l ∈ [2] :
Revised: For every t ≤ T 1 + T 2 , for every ( x , y ) ∈ ˆ D , every c ∈ [ C ] , r ∈ [ m ] and l ∈ [2] :

Original: • If v c, 1 , v c, 2 ∈ V ( x ) , then (cid:104)∇ θ c,r Ψ tc,r ( x ) , v c,l (cid:105) ≥ (cid:0) V c,r,l ( x ) − ˜ O
Revised: • If v c, 1 , v c, 2 ∈ V ( x ) , then (cid:104)∇ θ c,r Ψ tc,r ( x ) , v c,l (cid:105) ≥ (cid:0) V c,r,l ( x ) − ˜ O

Original: σ p P ) (cid:1) • (cid:104)∇ θ c,r Ψ tc,r , v c,l (cid:105) ≤ (cid:0) 1 { ( v c,l ∈ V ( x ) } V c,r,l ( x ) + ˜ O
Revised: σ p P ) (cid:1) • (cid:104)∇ θ c,r Ψ t c,r , v c,l (cid:105) ≤ (cid:0) 1 { ( v c,l ∈ V ( x ) } V c,r,l ( x ) + ˜ O

Original: ( C − 2 ) (cid:1) •
Revised: ( C − 2 ) (cid:1) •

Original: For every i (cid:54)=
Revised: For every i (cid:54)

Original: c , |(cid:104)−∇ θ c,r Ψ tc,r ( x ) , v i,l (cid:105)| ≤ ˜

Original: O ( C − 1 . 5 )
Revised: [Equation]

Original: Disclaimer Technically, Allen-Zhu & Li (2020) only show Claims 1 and 2 for t ≤ T 1 and one would need to use similar proof techniques (such as their inductive hypothesis F.1) to show the case for T 1 ≤ t ≤ T 2 , which we skip for conciseness.
Revised: Disclaimer Technically, Allen-Zhu & Li (2020) only show Claims 1 and 2 for t ≤ T 1 and one would need to use similar proof techniques (such as their inductive hypothesis F.1) to show the case for T 1 ≤ t ≤ T 2 , which we skip for conciseness.

Original: We now study the growth of the student’s Φ c,l , for those ( c, l ) which have been learnt by the teacher but not the student:
Revised: We now study the growth of the student’s Φ c,l , for those ( c, l ) which have been learnt by the teacher but not the student:

Original: Lemma 2 (Correlation Growth for attributes learnt by teacher) .
Revised: Lemma 2 (Correlation Growth for attributes learnt by teacher) .

Original: For every c ∈ [ C ] , l ∈ [2] , T 2 ≥ t ≥ T 1 , such that ( c, 3 − l ) / ∈ M T , suppose Φ tc,l ≤ 12 m , then we have:
Revised: For every c ∈ [ C ] , l ∈ [2] , T 2 ≥ t ≥ T 1 , such that ( c, 3 − l ) / ∈ M T , suppose Φ tc,l ≤ 12 m , then we have:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For any c ∈ [ C ] , r ∈ [ m ] , l ∈ [2] , we have 
Revised: For any c ∈ [ C ] , r ∈ [ m ] , l ∈ [2] , we have from Claim 2‘

Original: [Equation]
Revised: [Equation]

Original: Note that as µ ≤ 1 poly ( C ) , we can suppose that both x 1 , x 2 are multi-view data.
Revised: Note that as µ ≤ 1 poly ( C ) , we can suppose that both x 1 , x 2 are multi-view data.

Original: Using Claim 2, we have that
Revised: Using Claim 2, we have that

Original: [Equation]
Revised: [Equation]

Original: Let r = argmax r (cid:48) ∈ [ m ] {(cid:104) θ tc,r (cid:48) , v c,l (cid:105)} , such that deﬁnitely [Equation] θ tc,r , v c,l [Equation]
Revised: Let r = argmax r (cid:48) ∈ [ m ] {(cid:104) θ tc,r (cid:48) , v c,l (cid:105)} , such that deﬁnitely (cid:104) θ tc,r , v c,l (cid:105)≥ ˜Ω(Φ tc,l ) because m = polylog ( C ) .

Original: ˜Ω(Φ tc,l ) because m = polylog ( C ) .

Original: But if x is multi-view and v c,l ∈ V ( x ) such that (cid:80) p ∈P vc,l z 4 p = Θ(1) , and also by Fact A.a we have that:
Revised: But if x is multi-view and v c,l ∈ V ( x ) such that (cid:80) p ∈P vc,l z 4 p = Θ(1) , and also by Fact A.a we have that:

Original: [Equation]
Revised: [Equation]

Original: Now, we have assumed that ( c, 3 − l ) / ∈ M T , such that the teacher model has learnt attribute ( c, l ) and satisﬁes ˜ k T c ( x , x (cid:48) )
Revised: Now, we have assumed that ( c, 3 − l ) / ∈ M T , such that the teacher model has learnt attribute ( c, l ) and satisﬁes ˜ k T c ( x , x (cid:48) )

Original: = 1 when v c,l ∈ V ( x ) ∩ V ( x (cid:48) )
Revised: = 1 when v c,l ∈ V ( x ) ∩ V ( x (cid:48) )

Original: (for large enough polylogarithmic m ).
Revised: (for large enough polylogarithmic m ).

Original: Also, it is simple to see that when v c,l ∈ V ( x ) ∩ V ( x (cid:48) ) & v c, 3 − l / ∈ V ( x )
Revised: Also, it is simple to see that when v c,l ∈ V ( x ) ∩ V ( x (cid:48) ) & v c, 3 − l / ∈ V ( x )

Original: ∩ V ( x (cid:48) ) , for large enough m , that Φ tc,l ≤ 12 m implies that k c ( x , x (cid:48) ) ≤ 1 m 2 by Lemma 1, and so ˜ k c ( x , x (cid:48) )
Revised: ∩ V ( x (cid:48) ) , for large enough m , that Φ tc,l ≤ 12 m implies that k c ( x , x (cid:48) ) ≤ 1 m 2 by Lemma 1, and so ˜ k c ( x , x (cid:48) )

Original: = 0 , i.e. the student has not (yet) learnt v c,l .
Revised: = 0 , i.e. the student has not (yet) learnt v c,l .

Original: So we see there are two more conditions that must be satisﬁed in order for (cid:0) 
Revised: So we see there are two more conditions that must be satisﬁed in order for (cid:0) ˜

Original: ˜ k T [Equation] + =1 > 0 :
Revised: k T c ( x 1 , x 2 ) − ˜ k c ( x 1 , x 2 ) (cid:1) + =1 > 0 :

Original: v c,l ∈ V ( x 1 )
Revised: v c,l ∈ V ( x 1 )

Original: ∩ V ( x 2 )
Revised: ∩ V ( x 2 )

Original: so that ˜ k T c ( x 1 , x 2 ) =2.
Revised: so that ˜ k T c ( x 1 , x 2 ) =2.

Original: v c, 3 − l / ∈ V ( x 1 )
Revised: v c, 3 − l / ∈ V ( x 1 )

Original: ∩ V ( x 2 )
Revised: ∩ V ( x 2 )

Original: so that ˜ k c ( x 1 , x 2 ) = 0 .
Revised: so that ˜ k c ( x 1 , x 2 ) = 0 .

Original: Going back tothe App.
Revised: Going back to App.

Original: B.1.1, we know that these conditions occur with probability s 2 C 2 (1 − o (1)) for independently sampled x 1 , x 2 ∼ ˆ D .
Revised: B.1.1, we know that these conditions occur with probability s 2 C 2 (1 − o (1)) for independently sampled x 1 , x 2 ∼ ˆ D .

Original: Finally, putting everything together we have that:
Revised: Finally, putting everything together we have that:

Original: [Equation]
Revised: [Equation]

Original: summing over r (cid:48) ∈ [ m ] and noting (cid:104) ∆ θ tc,r (cid:48) , v c,l (cid:105) ≥ [-
-] up to small error (as σ p P = 1 poly [Equation] C ) for a large polynomial) gives us our result.
Revised: summing over r (cid:48) ∈ [ m ] and noting (cid:104) ∆ θ tc,r (cid:48) , v c,l (cid:105) ≥ [-
-]0 ∀ r (cid:48) , up to small error (as σ p P = 1 poly ( C ) for a large polynomial), gives us our result.

Original: Lemma 2 immediately gives us the following corollaries, because Φ T 1 i,l ≥ ˜Ω( σ 0 )  and (cid:94) ReLU (cid:48) is increasing.
Revised: Lemma 2 immediately gives us the following corollaries, because Φ T 1 c,l ≥ ˜Ω( σ 0 ) from Allen-Zhu & Li (2020) Induction Hypothesis F.1.g and (cid:94) ReLU (cid:48) is increasing.

Original: Corollary 1.
Revised: Corollary 1.

Original: Deﬁne iteration threshold T 

Revised: Deﬁne iteration threshold T

Original: = ˜Θ( C 2 ηs 2 σ 70 ) = ˜Θ( C 5 . 1 η ) , then for every ( c, l ) such that ( c, 3 − l ) / ∈ M T we have:Φ T 1 + T 2 c,l
Revised: = ˜Θ( C 2 ηs 2 σ 70 ) = ˜Θ( C 5 . 1 η ) , then for every ( c, l ) such that ( c, 3 − l ) / ∈ M T we have:

Original: [Equation]
Revised: [Equation]

Original: But likewise, we can also bound the growth of Φ c,l
Revised: But likewise, we can also bound the growth of Φ c,l

Original: Lemma 3.
Revised: Lemma 3.

Original: If ( c, 3 − l ) ∈ M S \M T , once Φ tc,l ≥ (cid:113) loglog ( C ) m , it no longer gets updated (for large
Revised: If ( c, 3 − l ) ∈ M S \M T , once Φ tc,l ≥ (cid:113) loglog ( C ) m , it no longer gets updated (for large

Original: C ):
Revised: C ):

Original: Proof.
Revised: Proof.

Original: Recall Deﬁnitions 3 and 4 that Φ ∗ c,l = (cid:80)
Revised: Recall Deﬁnitions 3 and 4 that Φ ∗ c,l = (cid:80)

Original: r ∈
Revised: r ∈

Original: [ m ] [ (cid:104) θ tc,r , v c,l (cid:105) ]
Revised: [ m ] [ (cid:104) θ tc,r , v c,l (cid:105) ]

Original: + and Υ tc,l = (cid:80) r ∈ [ m ] [ (cid:104) θ
Revised: + and Υ tc,l = (cid:80) r ∈ [ m ] [ (cid:104) θ

Original: tc,r , v c,l (cid:105) ]
Revised: tc,r , v c,l (cid:105) ]

Original: +
Revised: +

Original: Hence by Cauchy-Schwarz we have:
Revised: Hence by Cauchy-Schwarz we have:

Original: [Equation]
Revised: [Equation]

Original: for large enough C .
Revised: for large enough C .

Original: Thus, if x 1 , x 2 are both multi-view and v c,l ∈ V ( x [Equation] ∩ [Equation]
Revised: Thus, if x 1 , x 2 are both multi-view and v c,l ∈ V ( x 1 ) ∩ V ( x 2 ) , by Lemma 1 we must have

Original: ( x 2 ) , by Lemma 1 we must have

Original: [Equation]
Revised: [Equation]

Original: It’s also not difﬁcult to check that any other possible setting for x 1 , x 2 , and V ( x 1 ) ∩ V ( x 2 ) will lead to (cid:0)
Revised: It’s also not difﬁcult to check that any other possible setting for x 1 , x 2 , and V ( x 1 ) ∩ V ( x 2 ) will lead to (cid:0)

Original: ˜ k T c ( x 1 , x 2 ) − ˜ k
Revised: ˜ k T c ( x 1 , x 2 ) − ˜ k

Original: c ( x 1 , x 2 ) (cid:1) + = 0 , and hence (cid:0) ˜ k T c ( x 1 , x 2 ) − ˜ k c ( x 1 , x 2 ) (cid:1) + =∀ x 1 , x
Revised: c ( x 1 , x 2 ) (cid:1) + = 0 , and hence [Equation]

Original: C. W RAPPING UP PROOF OF T HEOREM
Revised: C. W RAPPING UP PROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: We are now ready to wrap up our proof.
Revised: We are now ready to wrap up our proof.

Original: Recall from the proof of Theorem 1 in Allen-Zhu &
Revised: Recall from the proof of Theorem 1 in Allen-Zhu &

Original: Li (2020) [Equation] that after the initial phase of T 1 steps of student training on its own:
Revised: Li (2020), that after the initial phase of T 1 steps of student training on its own:

Original: [Equation]
Revised: [Equation]

Original: and more speciﬁcally:
Revised: and more speciﬁcally:

Original: • If ( c, 3 − l ) / ∈ M S , then 
Revised: • If ( c, 3 − l ) / ∈ M S , then Φ T 1 c,

Original: [Equation]
Revised: [Equation]

Revised: ≥ Ω( log ( C ))

Original: This gives us perfect test accuracy on the multi-view data, and 50% accuracy on the singleview data, so 0 .
Revised: This gives us perfect test accuracy on the multi-view data, and 50% accuracy on the singleview data, so 0 .

Original: 5 µ test accuracy overall without distillation, as per Theorem1.
Revised: 5 µ test accuracy overall without distillation, as per Theorem

Original: • Moreover, we have that if ( c, 3 − l ) / ∈ M T and ( c, 3 − l ) ∈ M S , then by Corollary 1 and Lemma 3:
Revised: • Moreover, we have that if ( c, 3 − l ) / ∈ M T and ( c, 3 − l ) ∈ M S , then by Corollary 1 and Lemma 3:

Original: [Equation]
Revised: [Equation]

Original: We see that this change in Φ T 1 + T 2 c,l after FKD training is much smaller than Ω( log ( C )) and so the student after FKD training still has perfect multi-view accuracy, as well as correct predictions on any single-view data that possess the attributes learnt in the initial phase of training.
Revised: We see that this change in Φ T 1 + T 2 c,l after FKD training is much smaller than Ω( log ( C )) and so the student after FKD training still has perfect multi-view accuracy, as well as correct predictions on any single-view data that possess the attributes learnt in the initial phase of training.

Original: On the other hand, for single-view data, we know that if we have data point x , y and attribute v c,l ∈ V ( x ) , such that c (cid:54) =
Revised: On the other hand, for single-view data, we know that if we have data point x , y and attribute v c,l ∈ V ( x ) , such that c (cid:54) =

Original: y , then (cid:80) p ∈P vc,l ( x ) z p =
Revised: y , then (cid:80) p ∈P vc,l ( x ) z p =

Original: Γ =
Revised: Γ =

Original: O ( 1 polylog ( C ) ) , as deﬁned in App. B.1.1.
Revised: O ( [Equation] polylog ( C ) ) , as deﬁned in App. B.1.1.

Original: Hence for small enough Γ ( (cid:28) 1 m ) we have that if ( c, 3 − l) / ∈ M T and thesingle view data x is of class c with ˆ l ( x ) =
Revised: Hence for small enough Γ ( (cid:28) 1 m ) we have that if ( c 3 − l / ∈ M T and the

Revised: single view

Original:  lthen we have correct prediction, as per Claim 1.
Revised: data x is of class c with ˆ l

Revised: x

Revised: = l then we have correct prediction, as

Revised: per

Original:  Combining these means that we have correct prediction for any single-view data x of class c , and ˆ l ( x ) =
Revised: Claim Combining these means that we have correct prediction for any single-view data x of class c , and ˆ l ( x ) =

Original: l such that ( c, 3 − l ) / ∈ M T ∩ M s .
Revised: l such that ( c, 3 − l ) / ∈ M T ∩ M s .

Original: By the independence of these sets we have that |M T ∩ M s | =
Revised: By the independence of these sets we have that |M T ∩ M s | =

Original: (2 − E ) k
Revised: (2 − E ) k

Original: (1 − o (1))
Revised: (1 − o (1))

Original: this means we have test error less than (2 − E − 1 + (cid:15) ) µ for any (cid:15) > 0 , for large enough C as required.
Revised: this means we have test error less than (2 − E − 1 + (cid:15) ) µ for any (cid:15) > 0 , for large enough C as required.

Original: D D IFFERENCES BETWEEN FKD & OTHER FEATURE KERNEL BASED KD METHODS
Revised: D D IFFERENCES BETWEEN FKD & OTHER FEATURE KERNEL BASED KD METHODS

Original: In this section, we highlight how our FKD approach overcomes some of the shortcomings of previous feature kernel based KD methods which only use pairwise evaluations of the feature kernel: SP (Tung & Mori, 2019) and RKD (Park et al., 2019).
Revised: In this section, we highlight how our FKD approach overcomes some of the shortcomings of previous feature kernel based KD methods which only use pairwise evaluations of the feature kernel: SP (Tung & Mori, 2019) and RKD (Park et al., 2019).

Original: One advantage of FKD relative to these previous works is that we have shown FKD is amenable to ensemble distillation.
Revised: One advantage of FKD relative to these previous works is that we have shown FKD is amenable to ensemble distillation.

Original: Moreover, it goes without saying that Feature Regularisation, which arises naturally thanks to our feature kernel learning perspective in Section 4, is already a signiﬁcant departure that improves FKD relative to SP & RKD.
Revised: Moreover, it goes without saying that Feature Regularisation, which arises naturally thanks to our feature kernel learning perspective in Section 4, is already a signiﬁcant departure that improves FKD relative to SP & RKD.

Original: However, even without FR we observe in Section 5 that FKD outperforms both RKD & SP across different datasets and architectures, which warrants explanation.
Revised: However, even without FR we observe in Section 5 that FKD outperforms both RKD & SP across different datasets and architectures, which warrants explanation.

Original: D. I MPORTANCE OF ZERO DIAGONAL DIFFERENCES : SP (T UNG & M ORI , 2019)
Revised: D. I MPORTANCE OF ZERO DIAGONAL DIFFERENCES : SP (T UNG & M ORI , 2019)

Original: First, we consider diagonal kernel differences, k  ( x , x ) − k T ( x , x ) for ﬁxed x , and motivate using zero diagonal differences, which is not present in SP (Tung & Mori, 2019) but is in FKD thanks to our use of the correlation kernel.
Revised: First, we consider diagonal kernel differences, k S ( x , x ) − k T ( x , x ) for ﬁxed x , and motivate using zero diagonal differences, which is not present in SP (Tung & Mori, 2019) but is in FKD thanks to our use of the correlation kernel.

Original: Fig.
Revised: Fig.

Original: 6 displays this comparison between FKD & SP graphically.
Revised: 6 displays this comparison between FKD & SP graphically.

Original: Intuition: Downside of non-zero diagonal differences
Revised: Intuition: Downside of non-zero diagonal differences

Original: The key intuition, which we detail below using our theoretical setup, is that non-zero diagonal differences k
Revised: The key intuition, which we detail below using our theoretical setup, is that non-zero diagonal differences k

Original: ( x , x ) − k T ( x , x ) (cid:54)
Revised: ( x , x ) − k T ( x , x ) (cid:54)

Original: = 

Revised: =

Original: encourages the student to learn noise in input x , compared to when we have zero diagonal differences k
Revised: encourage the student to learn noise in input x , compared to when we have zero diagonal differences k

Original: ( x , x ) − k T ( x , x ) = 0 .
Revised: ( x , x ) − k T ( x , x ) = 0 .

Original: In the latter case, we only have non-zero differences for k ( x , x (cid:48) )
Revised: In the latter case, we only have non-zero differences for k ( x , x (cid:48) )

Original: − k T ( x , x (cid:48) )
Revised: − k T ( x , x (cid:48) )

Original: where x (cid:54) = x (cid:48) .
Revised: where x (cid:54) = x (cid:48) .

Revised: x x( k x , x 0 k x , x 0 ) 2 for SP 0.

Revised: x x( k x , x 0 k x , x 0 ) 2 for FKD 0.

Original: Diagonal updates Consider our parameter update Eq.
Revised: Diagonal updates Consider our parameter update Eq.

Original: (25) when x 1 = x 2 = x :
Revised: (25) when x 1 = x 2 = x :

Original: If we didn’t have zero diagonal differences and instead ˜ k c ( x , x ) − ˜ k T c
Revised: If we didn’t have zero diagonal differences and instead ˜ k c ( x , x ) − ˜ k T c

Original: ( x , x ) = − 1 , then:
Revised: ( x , x ) = − 1 , then:

Original: [Equation]
Revised: [Equation]

Original: x x( k x , x 0 k x , x 0 ) 2 for SP 0.

Original: x x( k x , x 0 k x , x 0 ) 2 for FKD 0.

Original: Now suppose v = v c, 1 ∈ V ( x ) .
Revised: Now suppose v = v c, 1 ∈ V ( x ) .

Original: For each p ∈ P v ( x ) , recall (from App. B.1.1) that:
Revised: For each p ∈ P v ( x ) , recall (from App. B.1.1) that:

Original: [Equation]
Revised: [Equation]

Revised: where we assume zero feature noise for simplicity.

Original: We then see that (c.f. Claim C.13 of Allen-Zhu & Li (2020)):
Revised: We then see that (c.f. Claim C.13 of Allen-Zhu & Li (2020)):

Original: [Equation]
Revised: [Equation]

Original: ) (cid:94)
Revised: + O

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  as (cid:104) v, ξ p (cid:105) =
Revised: d ) , as (cid:104) v, ξ p (cid:105) =

Original: O ( 1 √ d ) 
Revised: O ( 1 √ d ) with high probability.

Original: But at the same time (by e.g. Claim F.6 of Allen-Zhu & Li (2020)):[Equation] [Equation] =
Revised: But at the same time (by e.g. Claim F.6 of Allen-Zhu & Li (2020)):

Original: [Equation]
Revised: [Equation]

Original: V

Original: [Equation] [Equation] [Equation] ) and note that that V c,r, 1 ( x ) ≤ Θ(1) by deﬁnition.
Revised: and note that that V c,r, 1 ( x ) ≤ Θ(1) by deﬁnition.

Original: Moreover, in order for the student network to learn attribute v c, 1 , then eventually it must satisfy max r (cid:48) (cid:104) θ c,r (cid:48) , v (cid:105) =
Revised: Moreover, in order for the student network to learn attribute v c, 1 , then eventually it must satisfy max r (cid:48) (cid:104) θ c,r (cid:48) , v (cid:105) =

Original: ˜ O
Revised: ˜ O

Original: (1) from Allen-Zhu & Li (2020).
Revised: (1) from Allen-Zhu & Li (2020).

Original: Thus, for (cid:37) small enough, if r = argmax r (cid:48) (cid:104) θ c,r (cid:48) , v (cid:105) , we have (cid:94) ReLU (cid:48)
Revised: Thus, for (cid:37) small enough, if r = argmax r (cid:48) (cid:104) θ c,r (cid:48) , v (cid:105) , we have (cid:94) ReLU (cid:48)

Original: ( (cid:104) θ c,r , x p (cid:105) )
Revised: ( (cid:104) θ c,r , x p (cid:105) )

Original: = 1 .
Revised: = 1 .

Original: Non-diagonal updates On the other hand, if x 1 (cid:54) =
Revised: Non-diagonal updates On the other hand, if x 1 (cid:54) =

Original: x 2 and we have v ∈ V ( x 1 ) ∩ V ( x 2 ) :
Revised: x 2 and we have v ∈ V ( x 1 ) ∩ V ( x 2 ) :

Original: ∆ θ c,r ( x 1 , x 2 ) = η (cid:88) j =1 ,Ψ c,r ( x j ) P (cid:88) p = (cid:94) ReLU (cid:48) ( (cid:104) θ c,r , x 3 − j,p (cid:105) )
Revised: [Equation]

Original: x3 − j,p , and so if x 1 = x then (cid:104) ∆
Revised: x

Original: [Equation]
Revised: [Equation]

Revised: ReLU

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Revised: , and so

Original: , x 2 ) , ξ [Equation] (cid:105) = ˜Θ( η
Revised: if x 1 = x and ξ p denotes the random noise in x 1 ,p , then:

Original: [Equation]
Revised: [Equation]

Original: Ψ c,r ( x 2 ) (cid:94) ReLU (cid:48) ( (cid:104) θ c,r , x 1 ,p (cid:105) )

Original: + O ( 1 √ d ) , as (cid:104) x 2 ,p , ξ p (cid:105) =
Revised: ( 1 √ d ) , as (cid:104) x 2 ,p , ξ p (cid:105) =

Original: O ( 1 √ d ) .
Revised: O ( 1 √ d ) .

Original: But this time we have (cid:104) ∆ θ c,r ( x 1 , x 2 ) , v (cid:105) =
Revised: But this time we have [Equation]

Original: η (cid:0) Ψ c,r ( x 1 ) V c,r, 1 ( x2 ) + Ψ c,r ( x 2 ) V c,r, 1 ( x1 )(cid:1) (1 ± o (1)) , We see that whereas in diagonal updates ∆ θ c,r ( x , x ) the increments for (cid:104) ∆ θ c,
Revised: We see that whereas in diagonal updates ∆ θ c,r ( x , x ) the increments for (cid:104) ∆ θ c,r ( x , x ) , ξ p (cid:105) and (cid:104) ∆ θ c,r ( x , x ) , v (cid:105) are 1:1, for non-diagonal updates ∆ θ c,r ( x 1 , x 2 ) they are 1:2 respectively.

Original: Thus, the parameter updates for θ c,r with non-zero diagonal differences in feature kernels are more likely to learn noise, ξ p , compared to our zero diagonal updates which rely only on non-diagonal ∆ θ c,r ( x 1 , x 2 ) for x 1 (cid:54) = x 2 .
Revised: Thus, the parameter updates for θ c,r with non-zero diagonal differences in feature kernels are more likely to learn noise, ξ p , compared to our zero diagonal updates which rely only on non-diagonal ∆ θ c,r ( x 1 , x 2 ) for x 1 (cid:54) = x 2 .

Original: This is why we zero out diagonal differences for FKD, using the feature correlation matrix in practice.
Revised: This is why we zero out diagonal differences for FKD, using the feature correlation matrix in practice.

Original: D. P ROBLEM OF HOMOGENEOUS NN S IN RKD (P ARK ET AL ., 2019)
Revised: D. P ROBLEM OF HOMOGENEOUS NN S IN RKD (P ARK ET AL ., 2019)

Original: The distance-wise version of RKD (Park et al., 2019) is as follows: for x , x (cid:48) , we calculate ψ T ( x , x (cid:48) )
Revised: The distance-wise version of RKD (Park et al., 2019) is as follows: for x , x (cid:48) , we calculate ψ T ( x , x (cid:48) )

Original: = (cid:107) h T ( x , θ T )
Revised: = (cid:107) h T ( x , θ T )

Original: − h T ( x (cid:48) , θ T ) (cid:107) 2 , where recall h T is the last-layer teacher feature extractor.
Revised: − h T ( x (cid:48) , θ T ) (cid:107) 2 , where recall h T is the last-layer teacher feature extractor.

Original: Likewise, we also calculate ψ S ( x , x (cid:48) )
Revised: Likewise, we also calculate ψ S ( x , x (cid:48) )

Original: = (cid:107) h S
Revised: = (cid:107) h S

Original: ( x , θ S )
Revised: ( x , θ S )

Original: − h S ( x (cid:48) , θ S ) (cid:107) 2 .
Revised: − h S ( x (cid:48) , θ S ) (cid:107) 2 .

Original: The RKD loss adds λ KD E x , x (cid:48)
Revised: The RKD loss adds λ KD E x , x (cid:48)

Original: [( ψ T ( x , x (cid:48) ) − ψ S ( x , x (cid:48) )) 2 ] to the student’s training loss.
Revised: [( ψ T ( x , x (cid:48) ) − ψ S ( x , x (cid:48) )) 2 ] to the student’s training loss.

Original: 10 While RKD (Park et al., 2019) does ensure zero diagonal differences,
Revised: 9 While RKD (Park et al., 2019) does ensure zero diagonal differences,

Original: i.e. that ψ T ( x , x ) − ψ S ( x , x ) = 0 , it suffers from a related issue, due to the homogeneity of NNs that use ReLU nonlinearity, which is ubiquitous in image classiﬁcation tasks.
Revised: i.e. that ψ T ( x , x ) − ψ S ( x , x ) = 0 , it suffers from a related issue, due to the homogeneity of NNs that use ReLU nonlinearity, which is ubiquitous in image classiﬁcation tasks.

Original: Suppose we take x and deﬁne x (cid:48)
Revised: Suppose we take x and deﬁne x (cid:48)

Original: =
Revised: =

Original: M x for some M > 0 .
Revised: M x for some M > 0 .

Original: For example, think of taking a cat image and multiplying all the pixel values by M .
Revised: For example, think of taking a cat image and multiplying all the pixel values by M .

Original: For ReLU (C)NNs without bias parameters, we have that h ( x , θ ) is 1-homogeneous: h
Revised: For ReLU (C)NNs without bias parameters, we have that h ( x , θ ) is 1-homogeneous: h

Original: ( x (cid:48) , θ )
Revised: ( x (cid:48) , θ )

Original: =
Revised: =

Original: Mh ( x , θ ) .
Revised: Mh ( x , θ ) .

Original: This means that it is likely  we will have ψ T ( x , M x ) − ψ S ( x , M x ) (cid:54) = 0 .
Revised: This means that it is likely (depending on the norms of the features h S and h T ) that we will have ψ T ( x , M x ) − ψ S ( x , M x ) (cid:54) = 0 .

Original: But a cat image multiplied by some scalar M is still a cat image, hence RKD runs into the same problems as in App.
Revised: But a cat image multiplied by some scalar M is still a cat image, hence RKD runs into the same problems as in App.

Original: D.1 of learning noise in x .
Revised: D.1 of learning noise in x .

Original: On the other hand for FKD: correlation kernel ρ S ( x , M x )
Revised: On the other hand for FKD: correlation kernel ρ S ( x , M x )

Original: =
Revised: =

Original: ρ T ( x , M x )=1 , hence ρ S ( x , M x ) − ρ T ( x , M x )=

Revised: ρ T ( x , M x )=1 , hence ρ S ( x , M x ) − ρ T ( x , M x )=

Original: ∀ x ∈ R d , M > 0 .
Revised: ∀ x ∈ R d , M > 0 .

Original: E P Y T ORCH - STYLE PSEUDOCODE FOR FKD
Revised: E P Y T ORCH - STYLE PSEUDOCODE FOR FKD

Original: In Alg. 2, we provide PyTorch-style Paszke et al.
Revised: In Alg. 2, we provide PyTorch-style Paszke et al.

Original: (2019b) pseudocode for the distillation and feature regularisation losses in FKD.
Revised: (2019b) pseudocode for the distillation and feature regularisation losses in FKD.

Original: We note that FKD only requires pairwise computations of feature (correlations) kernels.
Revised: We note that FKD only requires pairwise computations of feature (correlations) kernels.

Original: This alleviates the need for matrix multiplication/inversion operations with batch-by-batch size matrices, which is beneﬁcial for scalability.
Revised: This alleviates the need for matrix multiplication/inversion operations with batch-by-batch size matrices, which is beneﬁcial for scalability.

Original: F E XPERIMENTAL DETAILS AND FURTHER RESULTS F. F IG . 2: PREDICTIVE DISAGREEMENT ACROSS INDEPENDENT INITIALISATIONS VS RETRAINED LAST LAYER
Revised: F E XPERIMENTAL DETAILS AND FURTHER RESULTS F. F IG . 2: PREDICTIVE DISAGREEMENT ACROSS INDEPENDENT INITIALISATIONS VS RETRAINED LAST LAYER

Original: All models are ResNet20v1 trained with standard hyperparameters:
Revised: All models are ResNet20v1 trained with standard hyperparameters:

Original: • 160 epochs training time with batch size 128 and learning rate 0.1 which is decayed by a factor of 10 after epochs 80 and 120.
Revised: • 160 epochs training time with batch size 128 and learning rate 0.1 which is decayed by a factor of 10 after epochs 80 and 120.

Original: • SGD optimiser with momentum 0.9 and weight decay of 0.0001.
Revised: • SGD optimiser with momentum 0.9 and weight decay of 0.0001.

Revised: [Table]

Original: • CIFAR10 data is normalised in each channel such that the training data is zero mean and unit standard deviation.
Revised: • CIFAR10 data is normalised in each channel such that the training data is zero mean and unit standard deviation.

Original: Random crops and horizontal ﬂips used as data augmentation.
Revised: Random crops and horizontal ﬂips used as data augmentation.

Original: • All models are initialised with Kaiming initialisation
Revised: • All models are initialised with Kaiming initialisation

Original: He et al.
Revised: He et al.

Original: Out of 10000 test points, the predictive disagreements between a ‘reference’ model and either: independent initialisations (top row) or retrained last layers (bottom row) are depicted in Table 4.
Revised: Out of 10000 test points, the predictive disagreements between a ‘reference’ model and either: independent initialisations (top row) or retrained last layers (bottom row) are depicted in Table 4.

Original: We see two clear trends.
Revised: We see two clear trends.

Original: First, the retrained last layer has much fewer disagreements with the reference model than an independent initialisation model, highlighting the importance of the feature[Table] kernel.
Revised: First, the retrained last layer has much fewer disagreements with the reference model than an independent initialisation model, highlighting the importance of the feature kernel.

Original: Secondly, the vast majority of disagreements between independent initialisations are where one of the models is correct.
Revised: Secondly, the vast majority of disagreements between independent initialisations are where one of the models is correct.

Original: This reinforces our intuition/theoretical analysis that ensembling NN works because different initialisations bias the models to capture different useful features, and hence ensemble distillation (via feature kernels) can improve student performance.
Revised: This reinforces our intuition/theoretical analysis that ensembling NN works because different initialisations bias the models to capture different useful features, and hence ensemble distillation (via feature kernels) can improve student performance.

Original: [Table]
Revised: [Table]

Original: F. F IG . 3: A DDITIONAL FEATURE KERNEL HISTOGRAMS
Revised: F. F IG . 3: A DDITIONAL FEATURE KERNEL HISTOGRAMS

Original: In Fig.
Revised: In Fig.

Original: 7, we provide additional plots to Fig.
Revised: 7, we provide additional plots to Fig.

Original: 3 that depict the difference in distribution (over x ) of feature kernel values k ( x , x ) , for FKD with and without Feature Regularisation (FR). 
Revised: 3 that depict the difference in distribution (over x ) of feature kernel values k ( x , x ) , for FKD with and without Feature Regularisation (FR). [Figure] i s t o g r a m

Revised: D e n s i t y

Revised: [Figure]

Original: Negative hypothesis.
Revised: Negative hypothesis.

Original: We originally hypothesised that FR could beneﬁt FKD, in addition to balancing the distribution of k ( x , x ) , as it could reduce the sparsity in the NN last-layer representation activations, which is consistent with the proofs of Theorems 1 and 2.
Revised: We originally hypothesised that FR could beneﬁt FKD, in addition to balancing the distribution of k ( x , x ) , as it could reduce the sparsity in the NN last-layer representation activations, which is consistent with the proofs of Theorems 1 and 2.

Original: Indeed, the NN predictions are dominated by a select few neurons who ‘have won the lottery’ (Allen-Zhu & Li, 2020) on account of being most correlated with one of the attributes { v c ; l } l ∈ [2] ,c ∈ [ C ] at random initialisation.
Revised: Indeed, the NN predictions are dominated by a select few neurons who ‘have won the lottery’ (Allen-Zhu & Li, 2020) on account of being most correlated with one of the attributes { v c ; l } l ∈ [2] ,c ∈ [ C ] at random initialisation.

Original: In our analysis, as few as O ( log 5 ( m ) out of m neurons could be inactive.
Revised: In our analysis, as few as O ( log 5 ( m ) out of m neurons could be inactive.

Original: From our feature kernel learning perspective, this seems like a highly undesirable phenomenon, because if k( x , x (cid:48) ) = (cid:104) h ( x ; θ ) , h ( x (cid:48) ; θ ) (cid:105) = (cid:80)
Revised: From our feature kernel learning perspective, this seems like a highly undesirable phenomenon, because if k

Original: [Equation]

Original: x , θ )

Original: [Equation]
Revised: [Equation]

Original: , θ ) only has useful contributions from a [Figure] [Figure] [Figure]{+= (cid:80) Cmr =
+}
Revised: {+= (cid:80) Cmr =
+}

Original: [Figure] H i s t o g ra m D e n s i t y
Revised: h r ( x , θ ) h r

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure] [Figure] [Figure] dominant minority of r ∈ [ Cm ] , then we are not utilising the full capacity of the model.
Revised: ( x (cid:48) , θ ) only has useful contributions from a dominant minority of r ∈ [ Cm ] , then we are not utilising the full capacity of the model.

Original: FR seemed appropriate to reduce this sparsity as (cid:96) 2 regularisation is known to promote non-sparse solutions (Van Den Doel et al., 2013).
Revised: FR seemed appropriate to reduce this sparsity as (cid:96) 2 regularisation is known to promote non-sparse solutions (Van Den Doel et al., 2013).

Original: However, we experimentally found the opposite to our hypothesis: that
Revised: However, we experimentally found the opposite to our hypothesis: that

Original: FR trained FKD student had more inactive neurons relative to FKD students trained without FR.
Revised: FR trained FKD student had more inactive neurons relative to FKD students trained without FR.

Original: This again highlights that, while (we believe) our results in this work provide compelling evidence to highlight the validity of feature kernel based distillation, there are still gaps between our theory and practice, and further questions to be answered in future work.
Revised: This again highlights that, while (we believe) our results in this work provide compelling evidence to highlight the validity of feature kernel based distillation, there are still gaps between our theory and practice, and further questions to be answered in future work.

Original: F. F IG . 4: E NSEMBLE DISTILLATION
Revised: F. F IG . 4: E NSEMBLE DISTILLATION

Original: All individual VGG8 networks that made up the teacher ensemble were trained using the default training regime from Tian et al.
Revised: All individual VGG8 networks that made up the teacher ensemble were trained using the default training regime from Tian et al.

Original: (2020), with independent parameter initialisations.
Revised: (2020), with independent parameter initialisations.

Original: Indeed, all of our experiments in Section 5 used Tian et al.
Revised: Indeed, all of our experiments in Section 5 used Tian et al.

Original: (2020)’s excellent open-source PyTorch codebase (Paszke et al., 2019a).
Revised: (2020)’s excellent open-source PyTorch codebase (Paszke et al., 2019a).

Original: For student networks, we used the training regime for vanilla KD from Tian et al.
Revised: For student networks, we used the training regime for vanilla KD from Tian et al.

Original: (2020) for all ensemble sizes.
Revised: (2020) for all ensemble sizes.

Original: For FKD, we used the hyperparameters from our ResNet50 → VGG8 experiment in
Revised: For FKD, we used the hyperparameters from our ResNet50 → VGG8 experiment in

Original: Table 2 for all ensemble sizes.
Revised: Table 2 for all ensemble sizes.

Original: F. T ABLE 1: D ATASET T RANSFER
Revised: F. T ABLE 1: D ATASET T RANSFER

Original: The VGG13 teacher checkpoint is provided by Tian et al.
Revised: The VGG13 teacher checkpoint is provided by Tian et al.

Original: For both CIFAR-10 and STL10, all student networks are trained for 160 epochs with batch size 64 using SGD with momentum, with learning rate decays at epochs 80, 120, 150.
Revised: For both CIFAR-10 and STL10, all student networks are trained for 160 epochs with batch size 64 using SGD with momentum, with learning rate decays at epochs 80, 120, 150.

Original: The student trained without KD used default hyperparameters from Tian et al.
Revised: The student trained without KD used default hyperparameters from Tian et al.

Original: (2020), which are indeed strong hyperparameters for standard training.
Revised: (2020), which are indeed strong hyperparameters for standard training.

Original: For FKD, RKD (Park et al., 2019) and SP (Tung & Mori, 2019), we tuned the learning rate, learning rate decay, and KD regularisation strength λ KD on a labeled validation set of size 5000 for CIFAR-10 and 1000 for STL-10, before retraining using best hyperparameters on the full training(+unlabeled) dataset.
Revised: For FKD, RKD (Park et al., 2019) and SP (Tung & Mori, 2019), we tuned the learning rate, learning rate decay, and KD regularisation strength λ KD on a labeled validation set of size 5000 for CIFAR-10 and 1000 for STL-10, before retraining using best hyperparameters on the full training(+unlabeled) dataset.

Original: We also tuned the FR regularisation strength, λ FR for FKD when FR was used.
Revised: We also tuned the FR regularisation strength, λ FR for FKD when FR was used.

Original: All RKD and SP hyperparameters were tuned in a large window around their default values from Tian et al.
Revised: All RKD and SP hyperparameters were tuned in a large window around their default values from Tian et al.

Original: (2020), which were all author recommended.
Revised: (2020), which were all author recommended.

Original: For FKD, we allowed λ KD to range in [1,1000], and λ FR to range in [0,20].
Revised: For FKD, we allowed λ KD to range in [1,1000], and λ FR to range in [0,20].

Original: All hyperparameters sweeps were conducted using
Revised: All hyperparameters sweeps were conducted using

Original: Bayes search.
Revised: Bayes search.

Original: For STL-10, we used a batch size of 512 for all KD methods’ regularisation terms, compared to 64 for the standard cross-entropy loss.
Revised: For STL-10, we used a batch size of 512 for all KD methods’ regularisation terms, compared to 64 for the standard cross-entropy loss.

Original: This was due to the fact that STL-10 has only 5K labeled datapoints, and we wanted to ensure that the student used as much of the unlabeled data as possible for each feature-kernel based KD method’s additional regularisation term during 160 epochs of training.
Revised: This was due to the fact that STL-10 has only 5K labeled datapoints, and we wanted to ensure that the student used as much of the unlabeled data as possible for each feature-kernel based KD method’s additional regularisation term during 160 epochs of training.

Original: 512 batch size was the maximum power of 2 before we ran into memory issues on a 11GB
Revised: 512 batch size was the maximum power of 2 before we ran into memory issues on a 11GB

Original: VRAM GPU, which occured for the RKD method.
Revised: VRAM GPU, which occured for the RKD method.

Original: Both CIFAR-10 and STL-10 data are normalised in each channel such that the training data is zero mean and unit standard deviation.
Revised: Both CIFAR-10 and STL-10 data are normalised in each channel such that the training data is zero mean and unit standard deviation.

Original: Random crops and horizontal ﬂips used as data augmentation.
Revised: Random crops and horizontal ﬂips used as data augmentation.

Original: STL-10 images are downsized from 96x96 to 32x32 resolution.
Revised: STL-10 images are downsized from 96x96 to 32x32 resolution.

Original: F. T ABLE 2: CIFAR-100 AND I MAGE N ET COMPARISON
Revised: F. T ABLE 2: CIFAR-100 AND I MAGE N ET COMPARISON

Original: CIFAR-100.
Revised: CIFAR-100.

Original: All networks were trained for 240 epochs with batch size 64, with learning rate decay at epochs 150, 180, 210 using SGD with momentum.
Revised: All networks were trained for 240 epochs with batch size 64, with learning rate decay at epochs 150, 180, 210 using SGD with momentum.

Original: All teacher networks use the exact same checkpoints as provided by Tian et al.
Revised: All teacher networks use the exact same checkpoints as provided by Tian et al.

Original: Learning rate, learning rate decay, λ KD , and λ FR (when used) were tuned as in App.
Revised: Learning rate, learning rate decay, λ KD , and λ FR (when used) were tuned as in App.

Original: F.4 on a validation of size 5000 .
Revised: F.4 on a validation of size 5000 .

Original: All other hyperparameters were set to the default values used by Tian et al.
Revised: All other hyperparameters were set to the default values used by Tian et al.

Original: The CIFAR-100 data is normalised in each channel such that the training data is zero mean and unit standard deviation, with random crops and horizontal ﬂips used for data augmentation.
Revised: The CIFAR-100 data is normalised in each channel such that the training data is zero mean and unit standard deviation, with random crops and horizontal ﬂips used for data augmentation.

Original: All results provided denote the test set accuracy at the end of the 240 epochs of training.
Revised: All results provided denote the test set accuracy at the end of the 240 epochs of training.

Original: ImageNet.
Revised: ImageNet.

Original: The ImageNet dataset (ILSVRC-2012) consists of about 1.3 million training images and 50,000 validation images from 1,000 classes.
Revised: The ImageNet dataset (ILSVRC-2012) consists of about 1.3 million training images and 50,000 validation images from 1,000 classes.

Original: Each training image is extracted as a randomly sampled 224x224 crop or its horizontal ﬁlp without any padding operation.
Revised: Each training image is extracted as a randomly sampled 224x224 crop or its horizontal ﬂip without any padding operation.

Original: All teacher networks use the exact same checkpoints as provided by Chen et al.
Revised: All teacher networks use the exact same checkpoints as provided by Chen et al.

Original: The initial learning rate is 0.1 and divided by 10 at 30 and 60 of the total 90 training epochs.
Revised: The initial learning rate is 0.1 and divided by 10 at 30 and 60 of the total 90 training epochs.

Original: We set the mini-batch size to 256 and the weight decay to 10 − 4 .
Revised: We set the mini-batch size to 256 and the weight decay to 10 − 4 .

Original: λ KD , and λ FR (when used) were tuned as in App.
Revised: λ KD , and λ FR (when used) were tuned as in App.

Original: F.4 on a validation of size 5000 except using Bayes search.
Revised: F.4 on a validation of size 5000 except using Bayes search.

Original: All results are reported in a single trial.
Revised: All results are reported in a single trial.

Original: All other hyperparameters were set to the default values used by Chen et al.
Revised: All other hyperparameters were set to the default values used by Chen et al.

Original: All results provided denote the Top-1 test accuracy (%).
Revised: All results provided denote the Top-1 test accuracy (%).

Original: Accuracy of baselines were reported in Tian et al.
Revised: Accuracy of baselines were reported in Tian et al.

Original: F. S ENSITIVITY TO λ KD
Revised: F. S ENSITIVITY TO λ KD

Original: In Fig.
Revised: In Fig.

Original: 8, we plot the sensitivity of FKD to the strength of the distillation regularisation λ KD in Eq.
Revised: 8, we plot the sensitivity of FKD to the strength of the distillation regularisation λ KD in Eq.

Original: (2) for the VGG13 → VGG8 experiment on CIFAR-100.
Revised: (2) for the VGG13 → VGG8 experiment on CIFAR-100.

Original: We see that a well tuned λ KD ( ≈ 300 here) is important for best student generalisation.
Revised: We see that a well tuned λ KD ( ≈ 300 here) is important for best student generalisation.

Original: Feature regularisation λ FR = 20 in Fig.
Revised: Feature regularisation λ FR = 20 in Fig.

Original: F. T ABLE 5: A NALYSES IN N EURAL M ACHINE T RANSLATION
Revised: F. T ABLE 5: A NALYSES IN N EURAL M ACHINE T RANSLATION

Original: In this section, We performed analyses for a neural machine translation (NMT) task proposed by
Revised: In this section, We performed analyses for a neural machine translation (NMT) task proposed by

Original: Tan et al.
Revised: Tan et al.

Original: In the analyses, we could only obtain data for En-De (from English to German) translation since links to the datasets for other languages are broken.
Revised: In the analyses, we could only obtain data for En-De (from English to German) translation since links to the datasets for other languages are broken.

Original: Therefore, we employed a self-distillation method on a pre-trained English model for En-De translation as follows: 
Revised: Therefore, we employed a self-distillation method on a pre-trained English model for En-De translation as follows: [Figure]

Original:  • We train a single teacher transformer model on the IWSLT dataset for English (Tan et al., 2019).
Revised: T e s t a cc u r a c y ( % ) VGG VGG8 on CIFAR- • We train a single teacher transformer model on the IWSLT dataset for English (Tan et al., 2019).

Original: • We perform self-distillation on the teacher model for En-De translation (Tan et al., 2019).
Revised: • We perform self-distillation on the teacher model for En-De translation (Tan et al., 2019).

Original: • We did not search for optimal hyperparameters, and used default parameters of the code provided by the authors of Tan et al.
Revised: • We did not search for optimal hyperparameters, and used default parameters of the code provided by the authors of Tan et al.

Original: The results are given in Table 
Revised: The results are given in Table 5.

Original: We ﬁrst note that, we adapted vanilla KD and our FKD for sequential data in the KD loss (2) of (Tan et al., 2019) in this task.

Original: More precisely, we ﬁrst computed vanilla KD and FKD on token

Original: [Figure]
Revised: [Table]

Original: [Figure]

Original: cc u r [Figure] c y [Figure]
Revised: We ﬁrst note that, we adapted vanilla KD and our FKD for sequential data in the KD loss (2) of (Tan et al., 2019) in this task.

Original: VGG VGG8 on CIFAR- [Table] probabilities, and added these loss functions to the KD loss (eq 2 of (Tan et al., 2019)) in KD and
Revised: More precisely, we ﬁrst computed vanilla KD and FKD on token probabilities, and added these loss functions to the KD loss (eq 2 of (Tan et al., 2019)) in KD and

Original: FKD.
Revised: FKD.

Original: In the results, aggregating vanilla KD with the KD loss (eq 2 of (Tan et al., 2019)) improved accuracy from 27.49 to 27.51.
Revised: In the results, aggregating vanilla KD with the KD loss (eq 2 of (Tan et al., 2019)) improved accuracy from 27.49 to 27.51.

Original: However, FKD further boosted BLEU to 27.64.
Revised: However, FKD further boosted BLEU to 27.64.

Original: We then replaced KD loss (eq 2 of (Tan et al., 2019)) with FKD for training.
Revised: We then replaced KD loss (Eq. 2 of (Tan et al., 2019)) with FKD for training.

Original: Remarkably, FKD further boosted the BLEU to 27.79.
Revised: Remarkably, FKD further boosted the BLEU to 27.79.

Original: These results suggest that the proposed FKD can be applied in NMT tasks, successfully.
Revised: These results suggest that the proposed FKD can be applied in NMT tasks, successfully.

Original: We hope that these results will motivate researchers to employ FKD in various different NLP tasks including but not limited to multilingual NMT, named entity recognition and question answering.
Revised: We hope that these results will motivate researchers to employ FKD in various different NLP tasks including but not limited to multilingual NMT, named entity recognition and question answering.

Original: F. T ABLE 6: A NALYSES IN A UTOMATIC S PEECH R ECOGNITION
Revised: F. T ABLE 6: A NALYSES IN A UTOMATIC S PEECH R ECOGNITION

Original: In this section, we used a CRDNN model (VGG + LSTM,GRU,LiGRU+ DNN) on the TIMIT dataset.
Revised: In this section, we used a CRDNN model (VGG + LSTM,GRU,LiGRU+ DNN) on the TIMIT dataset.

Original: In this experiment, we used a distillation approach proposed by Gao et al.
Revised: In this experiment, we used a distillation approach proposed by Gao et al.

Original: (2020) for ASR tasks as follows:
Revised: (2020) for ASR tasks as follows:

Original: • We train a single teacher model on the TIMIT dataset Ravanelli et al.
Revised: • We train a single teacher model on the TIMIT dataset Ravanelli et al.

Original: • We perform self-distillation on the teacher Gao et al.
Revised: • We perform self-distillation on the teacher Gao et al.

Original: • We did not search for optimal hyperparameters, and used default parameters of the SpeechBrain Library.
Revised: • We did not search for optimal hyperparameters, and used default parameters of the SpeechBrain Library.

Original: • In this task, replacing CTC/NLL distillation losses with KD (Hinton et al., 2015) did not converge.
Revised: • In this task, replacing CTC/NLL distillation losses with KD (Hinton et al., 2015) did not converge.

Original: Additional investigation with hyperparameter search is needed.
Revised: Additional investigation with hyperparameter search is needed.

Original: We used phoneme error rate (PER) to measure accuracy of models.
Revised: We used phoneme error rate (PER) to measure accuracy of models.

Original: [Table]
Revised: [Table]

Original: The results are given in Table 6.
Revised: The results are given in Table 6.

Original: Similar to the NMT task, we adapted vanilla KD and our FKD for sequential data as follows:
Revised: Similar to the NMT task, we adapted vanilla KD and our FKD for sequential data as follows:

Original: We ﬁrst computed vanilla KD and FKD loss functions on token probabilities, and then added to the total loss (eq 7 of Gao et al.
Revised: We ﬁrst computed vanilla KD and FKD loss functions on token probabilities, and then added to the total loss (eq 7 of Gao et al.

Original: In the analyses, Vanilla KD (Hinton et al., 2015) increased the PER from 12.80 to 12.86.
Revised: In the analyses, Vanilla KD (Hinton et al., 2015) increased the PER from 12.80 to 12.86.

Original: However, FKD further improved the
Revised: However, FKD further improved the

Original: PER from 12.80 to 12.59.
Revised: PER from 12.80 to 12.59.

Original: In this task, training models by replacing CTC/NLL distillation losses (eq 4 or 5 of Gao et al.
Revised: In this task, training models by replacing CTC/NLL distillation losses (eq 4 or 5 of Gao et al.

Original: with KD (Hinton et al., 2015)
Revised: with KD (Hinton et al., 2015)

Original: and FKD did not converge.
Revised: and FKD did not converge.

Original: In conclusion, these results propound that FKD can be applied for different tasks, i.e., image classiﬁcation, NMT and ASR, boosting accuracy of baseline distillation methods.
Revised: In conclusion, these results propound that FKD can be applied for different tasks, i.e., image classiﬁcation, NMT and ASR, boosting accuracy of baseline distillation methods.

Original: We hope that these initial results will motivate researchers in different communities (computer vision, NLP, and ASR) to further expound and apply FKD in additional sub-tasks.
Revised: We hope that these initial results will motivate researchers in different communities (computer vision, NLP, and ASR) to further expound and apply FKD in additional sub-tasks.