Original: K NOWLEDGE DISTILLATION VIA SOFTMAX REGRESSION REPRESENTATION LEARNING
Revised: K NOWLEDGE DISTILLATION VIA SOFTMAX REGRESSION REPRESENTATION LEARNING

Original: This paper addresses the problem of model compression via knowledge distillation.
Revised: This paper addresses the problem of model compression via knowledge distillation.

Original: We advocate for a method that optimizes the output feature of the penultimate layer of the student network and hence is directly related to representation learning.
Revised: We advocate for a method that optimizes the output feature of the penultimate layer of the student network and hence is directly related to representation learning.

Original: To this end, we ﬁrstly propose a direct feature matching approach which focuses on optimizing the student’s penultimate layer only.
Revised: To this end, we ﬁrstly propose a direct feature matching approach which focuses on optimizing the student’s penultimate layer only.

Original: Secondly and more importantly , because feature matching does not take into account the classiﬁcation problem at hand, we propose a second approach that decouples representation learning and classiﬁcation and utilizes the teacher’s pre-trained classiﬁer to train the student’s penultimate layer feature.
Revised: Secondly and more importantly , because feature matching does not take into account the classiﬁcation problem at hand, we propose a second approach that decouples representation learning and classiﬁcation and utilizes the teacher’s pre-trained classiﬁer to train the student’s penultimate layer feature.

Original: In particular, for the same input image, we wish the teacher’s and student’s feature to produce the same output when passed through the teacher’s classiﬁer, which is achieved with a simple Lloss.
Revised: In particular, for the same input image, we wish the teacher’s and student’s feature to produce the same output when passed through the teacher’s classiﬁer, which is achieved with a simple Lloss.

Original: Our method is extremely simple to implement and straightforward to train and is shown to consistently outperform previous state-of-the-art methods over a large set of experimental settings including different (a) network architectures, (b) teacher-student capacities, (c) datasets, and (d) domains.
Revised: Our method is extremely simple to implement and straightforward to train and is shown to consistently outperform previous state-of-the-art methods over a large set of experimental settings including different (a) network architectures, (b) teacher-student capacities, (c) datasets, and (d) domains.

Original: The code will be available at https://github.com/jingyang2017/KD_SRRL .
Revised: The code is available at https://github.com/jingyang2017/KD_SRRL .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recently, there has been a great amount of research effort to make Convolutional Neural Networks (CNNs) lightweight so that they can be deployed in devices with limited resources.
Revised: Recently, there has been a great amount of research effort to make Convolutional Neural Networks (CNNs) lightweight so that they can be deployed in devices with limited resources.

Original: To this end, several approaches for model compression have been proposed, including network pruning (Han et al., 2016; Lebedev & Lempitsky, 2016), network quantization (Rastegari et al., 2016; Wu et al., 2016), knowledge transfer/distillation (Hinton et al., 2015; Zagoruyko & Komodakis, 2017), and neural architecture search (Zoph & Le, 2017; Liu et al., 2018).
Revised: To this end, several approaches for model compression have been proposed, including network pruning (Han et al., 2016; Lebedev & Lempitsky, 2016), network quantization (Rastegari et al., 2016; Wu et al., 2016), knowledge transfer/distillation (Hinton et al., 2015; Zagoruyko & Komodakis, 2017), and neural architecture search (Zoph & Le, 2017; Liu et al., 2018).

Original: Knowledge distillation (Buciluˇa et al., 2006; Hinton et al., 2015) aims to transfer knowledge from one network (the so-called “teacher”) to another (the so-called “student”).
Revised: Knowledge distillation (Buciluˇa et al., 2006; Hinton et al., 2015) aims to transfer knowledge from one network (the so-called “teacher”) to another (the so-called “student”).

Original: Typically, the teacher is a high-capacity model capable of achieving high accuracy, while the student is a compact model with much fewer parameters, thus also requiring much less computation.
Revised: Typically, the teacher is a high-capacity model capable of achieving high accuracy, while the student is a compact model with much fewer parameters, thus also requiring much less computation.

Original: The goal of knowledge distillation is to use the teacher to improve the training of the student and push its accuracy closer to that of the teacher.
Revised: The goal of knowledge distillation is to use the teacher to improve the training of the student and push its accuracy closer to that of the teacher.

Original: The rationale behind knowledge distillation can be explained from an optimization perspective: there is evidence that high capacity models (i.e. the teacher) can ﬁnd good local minima due to over-parameterization (Du & Lee, 2018; Soltanolkotabi et al., 2018).
Revised: The rationale behind knowledge distillation can be explained from an optimization perspective: there is evidence that high capacity models (i.e. the teacher) can ﬁnd good local minima due to over-parameterization (Du & Lee, 2018; Soltanolkotabi et al., 2018).

Original: In knowledge distillation, such models are used to facilitate the optimization of lower capacity models (i.e. the student) during training.
Revised: In knowledge distillation, such models are used to facilitate the optimization of lower capacity models (i.e. the student) during training.

Original: For example, in the seminal work of (Hinton et al., 2015), the softmax outputs of the teacher provide extra supervisory signals of inter-class similarities which facilitate the training of the student.
Revised: For example, in the seminal work of (Hinton et al., 2015), the softmax outputs of the teacher provide extra supervisory signals of inter-class similarities which facilitate the training of the student.

Original: In other inﬂuential works, intermediate representations extracted from the teacher such as feature tensors (Romero et al., 2015) or attention maps (Zagoruyko & Komodakis, 2017) have been used to deﬁne auxiliary loss functions used in the optimization of the student.
Revised: In other inﬂuential works, intermediate representations extracted from the teacher such as feature tensors (Romero et al., 2015) or attention maps (Zagoruyko & Komodakis, 2017) have been used to deﬁne auxiliary loss functions used in the optimization of the student.

Original: Training a network whose output feature representation is rich and powerful has been shown crucial for achieving high accuracy for the subsequent classiﬁcation task in recent works in both unsupervised and supervised learning, see for example (Chen et al., 2020;
Revised: Training a network whose output feature representation is rich and powerful has been shown crucial for achieving high accuracy for the subsequent classiﬁcation task in recent works in both unsupervised and supervised learning, see for example (Chen et al., 2020;

Original: He et al., 2020) and (Kang et al., 2020).
Revised: He et al., 2020) and (Kang et al., 2020).

Original: Hence, in this paper, we are advocating for representation learning-based knowledge distillation by optimizing the student’s penultimate layer output feature.
Revised: Hence, in this paper, we are advocating for representation learning-based knowledge distillation by optimizing the student’s penultimate layer output feature.

Original: If we are able to do this effectively, we expect (and show experimentally) to end up with a student network which can generalize better than one trained with logit matching as in the KD paper of (Hinton et al., 2015).
Revised: If we are able to do this effectively, we expect (and show experimentally) to end up with a student network which can generalize better than one trained with logit matching as in the KD paper of (Hinton et al., 2015).

Original: Main contributions: To accomplish the aforementioned goal we propose two loss functions: The ﬁrst loss function , akin to (Romero et al., 2015; Zagoruyko & Komodakis, 2017), is based on direct feature matching but focuses on optimizing the student’s penultimate layer feature, only.
Revised: Main contributions: To accomplish the aforementioned goal we propose two loss functions: The ﬁrst loss function , akin to (Romero et al., 2015; Zagoruyko & Komodakis, 2017), is based on direct feature matching but focuses on optimizing the student’s penultimate layer feature only.

Original: Because direct feature matching might be difﬁcult due to the lower representation capacity of the student and, more importantly, is detached from the classiﬁcation task at hand, we also propose a second loss function : we propose to decouple representation learning and classiﬁcation and utilize the teacher’s pre-trained classiﬁer to train the student’s penultimate layer feature.
Revised: Because direct feature matching might be difﬁcult due to the lower representation capacity of the student and, more importantly, is detached from the classiﬁcation task at hand, we also propose a second loss function : we propose to decouple representation learning and classiﬁcation and utilize the teacher’s pre-trained classiﬁer to train the student’s penultimate layer feature.

Original: In particular, for the same input image, we wish the teacher’s and student’s feature to produce the same output when passed through the teacher’s classiﬁer, which is achieved with a simple L 2 loss. See Fig. 
Revised: In particular, for the same input image, we wish the teacher’s and student’s feature to produce the same output when passed through the teacher’s classiﬁer, which is achieved with a simple L 2 loss (see Fig. 1).

Original: This softmax regression projection is used to retain from the student’s feature the information that is relevant to classiﬁcation, but since the projection matrix is pre-trained (learned during the teacher’s training phase) this does not compromise the representational power of the student’s feature.
Revised: This softmax regression projection is used to retain from the student’s feature the information that is relevant to classiﬁcation, but since the projection matrix is pre-trained (learned during the teacher’s training phase) this does not compromise the representational power of the student’s feature.

Original: Main results: Our method has two advantages: (1) It is simple and straightforward to implement.
Revised: Main results: Our method has two advantages: (1) It is simple and straightforward to implement.

Original: (2) It consistently outperforms state-of-the-art methods over a large set of experimental settings including different (a) network architectures (WideResNets, ResNets, MobileNets), (b) teacherstudent capacities, (c) datasets (CIFAR-10/100, ImageNet), and (d) domains (real-to-binary).
Revised: (2) It consistently outperforms state-of-the-art methods over a large set of experimental settings including different (a) network architectures (WideResNets, ResNets, MobileNets), (b) teacherstudent capacities, (c) datasets (CIFAR-10/100, ImageNet), and (d) domains (real-to-binary).

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Knowledge transfer: In theseminal work of (Hinton et al., 2015), knowledge is deﬁned as the teacher’s outputs after the ﬁnal softmax layer.
Revised: Knowledge transfer: In the work of (Hinton et al., 2015), knowledge is deﬁned as the teacher’s outputs after the ﬁnal softmax layer.

Original: The softmax outputs carry richer information than one-hot labels because they provide extra supervision signals in terms of the inter-class similarities learned by the teacher.
Revised: The softmax outputs carry richer information than one-hot labels because they provide extra supervision signals in terms of the inter-class similarities learned by the teacher.

Original: In a similar fashion to (Hinton et al., 2015), intermediate representations ex- tracted from the teacher such as feature tensors (Romero et al., 2015) or attention maps (Zagoruyko & Komodakis, 2017) have been used to deﬁne loss functions used to facilitate the optimization of the student.
Revised: In a similar fashion to (Hinton et al., 2015), intermediate representations extracted from the teacher such as feature tensors (Romero et al., 2015) or attention maps (Zagoruyko & Komodakis, 2017) have been used to deﬁne loss functions used to facilitate the optimization of the student.

Original: Trying to match the whole feature tensor, as in FitNets (Romero et al., 2015), is hard and, in certain circumstances, such an approach may adversely affect the performance and convergence of the student.
Revised: Trying to match the whole feature tensor, as in FitNets (Romero et al., 2015), is hard and, in certain circumstances, such an approach may adversely affect the performance and convergence of the student.

Original: To relax the assumption of FitNet, Attention Transfer (AT) was proposed in (Zagoruyko & Komodakis, 2017) where knowledge takes the form of attention maps which are summaries of the energies of the feature tensors over the channel dimension.
Revised: To relax the assumption of FitNet, Attention Transfer (AT) was proposed in (Zagoruyko & Komodakis, 2017) where knowledge takes the form of attention maps which are summaries of the energies of the feature tensors over the channel dimension.

Original: An extension of (Zagoruyko & Komodakis, 2017) using Maximum Mean Discrepancy of the network activations as a loss term for distillation was proposed in (Huang & Wang, 2017).
Revised: An extension of (Zagoruyko & Komodakis, 2017) using Maximum Mean Discrepancy of the network activations as a loss term for distillation was proposed in (Huang & Wang, 2017).

Original: Cho & Hariharan (2019) showed that very accurate networks are “too good” to be good teachers and proposed to mitigate this with early stopping of the teacher’s training.
Revised: Cho & Hariharan (2019) showed that very accurate networks are “too good” to be good teachers and proposed to mitigate this with early stopping of the teacher’s training.

Original: Recently, the work of (Heo et al., 2019a) studied the location within the network at which feature distillation should be applied and proposed margin ReLU and a speciﬁcally designed distance function that transfers only the useful (positive) information from the teacher to the student.
Revised: Recently, the work of (Heo et al., 2019a) studied the location within the network at which feature distillation should be applied and proposed margin ReLU and a speciﬁcally designed distance function that transfers only the useful (positive) information from the teacher to the student.

Original: More recently, Li et al.
Revised: More recently, Li et al.

Original: (Li et al., 2020a) proposed to supervise the block-wise architecture search by the architecture knowledge distilled from a teacher model.
Revised: (Li et al., 2020a) proposed to supervise the blockwise architecture search by the architecture knowledge distilled from a teacher model.

Original: Another NAS based method was proposed in (Guan et al., 2020), in which a student-to-teacher loss is used to ﬁnd the aggregation weights that match the learning ability of the student.
Revised: Another NAS based method was proposed in (Guan et al., 2020), in which a student-to-teacher loss is used to ﬁnd the aggregation weights that match the learning ability of the student.

Original: Passalis et al.
Revised: Passalis et al.

Original: (2020) claimed that traditional KD ignores information plasticity during the training process, and proposed to model the information ﬂow through the various layers of the teacher.
Revised: (2020) claimed that traditional KD ignores information plasticity during the training process, and proposed to model the information ﬂow through the various layers of the teacher.

Original: Feature relationship transfer: Another line of knowledge distillation methods focus on exploring transferring the relationship between features, rather than the actual features themselves.
Revised: Feature relationship transfer: Another line of knowledge distillation methods focus on exploring transferring the relationship between features, rather than the actual features themselves.

Original: In (Yim et al., 2017), feature correlations are captured by computing the Gram matrix of features across layers for both teacher and student and then applying an L 2 loss on pairs of teacher-student Gram matrices.
Revised: In (Yim et al., 2017), feature correlations are captured by computing the Gram matrix of features across layers for both teacher and student and then applying an L 2 loss on pairs of teacher-student Gram matrices.

Original: The limitation of this work is the high computational cost, which is addressed to some extent in (Lee et al., 2018) by compressing the feature maps by singular value decomposition.
Revised: The limitation of this work is the high computational cost, which is addressed to some extent in (Lee et al., 2018) by compressing the feature maps by singular value decomposition.

Original: Park et al.
Revised: Park et al.

Original: (2019) proposed a relational knowledge distillation method which computes distance-wise and angle-wise relations of each embedded feature vector.
Revised: (2019) proposed a relational knowledge distillation method which computes distance-wise and angle-wise relations of each embedded feature vector.

Original: This idea is further explored in (Peng et al., 2019) and (Liu et al., 2019).
Revised: This idea is further explored in (Peng et al., 2019) and (Liu et al., 2019).

Original: In (Peng et al., 2019), Taylor series expansion is proposed to better capture the correlation between multiple instances.
Revised: In (Peng et al., 2019), Taylor series expansion is proposed to better capture the correlation between multiple instances.

Original: In (Liu et al., 2019), the instance feature and relationships are considered as vertexes and edges respectively in a graph and instance relationship graph is proposed to model the feature space transformation across layers.
Revised: In (Liu et al., 2019), the instance feature and relationships are considered as vertexes and edges respectively in a graph and instance relationship graph is proposed to model the feature space transformation across layers.

Original: Inspired by the observation that semantically similar inputs should have similar activation patterns, (Tung & Mori, 2019) proposed a similarity-preserving knowledge distillation method which guides the student to mimic the teacher with respect to generating similar or dissimilar activations.
Revised: Inspired by the observation that semantically similar inputs should have similar activation patterns, (Tung & Mori, 2019) proposed a similarity-preserving knowledge distillation method which guides the student to mimic the teacher with respect to generating similar or dissimilar activations.

Original: More recently, (Jain et al., 2020) proposed to matching the student output with the teacher’s by distilling the knowledge through a quantized visual words space.
Revised: More recently, (Jain et al., 2020) proposed to matching the student output with the teacher’s by distilling the knowledge through a quantized visual words space.

Original: Li et al.
Revised: Li et al.

Original: (2020b) proposed the local correlation exploration framework to represent the relationships of local regions in the feature space which contains more details and discriminative patterns.
Revised: (2020b) proposed the local correlation exploration framework to represent the relationships of local regions in the feature space which contains more details and discriminative patterns.

Original: Finally, a similar connection between distillation and representation learning was very recently made in (Tian et al., 2020) which uses contrastive learning for knowledge distillation.
Revised: Finally, a similar connection between distillation and representation learning was very recently made in (Tian et al., 2020) which uses contrastive learning for knowledge distillation.

Original: We note that our loss is not related to the one used in (Tian et al., 2020), is simpler, and as shown in Section 5, outperforms it for all of our experiments, often by a signiﬁcant margin.
Revised: We note that our loss is not related to the one used in (Tian et al., 2020), is simpler, and as shown in Section 5, outperforms it for all of our experiments, often by a signiﬁcant margin.

Original:  M ETHOD
Revised:  M ETHOD

Original: We denote by T and S the teacher and student networks, respectively.
Revised: We denote by T and S the teacher and student networks respectively.

Original: We split these networks into two parts: (i)
Revised: We split these networks into two parts: (i)

Original: A convolutional feature extractor f Net , Net =
Revised: A convolutional feature extractor f Net , Net =

Original: { T, S }  the output of which at the i -th layer is a feature tensor F Net i ∈
Revised: { T, S } , the output of which at the i -th layer is a feature tensor F Net i ∈

Original: [Equation]
Revised: [Equation]

Original: i , where C Net i is the output feature dimensionality, and H i , W i the output spatial dimensions.
Revised: , where C [Equation] i is the output feature dimensionality, and H i , W i the output spatial dimensions.

Original: We also denote by h Net = (cid:80) H L h =1 (cid:80) W L w =1 F NetL ∈ (cid:60) C NetL the last layer feature representation learned by f Net .
Revised: We also denote by h Net = (cid:80) H L h =1 (cid:80) W L w =1 F NetL ∈ R C NetL the last layer feature representation learned by f Net .

Original: (ii)
Revised: (ii)

Original: A projection matrix W Net ∈ (cid:60) C NetL × K which projects the feature representation h Net into K class logits z Neti , i = 1 , . .
Revised: A projection matrix W Net ∈ R C NetL × K which projects the feature representation h Net into K class logits z Neti , i = 1 , . .

Original: , K  followed by the softmax function s ( z i ) =
Revised: , K , followed by the softmax function s ( z i ) =

Original: exp( z i /τ ) (cid:80) j exp( z j /τ ) with temperature τ ( τ = 1 for Cross Entropy loss) which put together form a softmax regression classiﬁer into K classes.
Revised: exp( z i /τ ) (cid:80) j exp( z j /τ ) with temperature τ ( τ = 1 for Cross Entropy loss) which put together form a softmax regression classiﬁer into K classes.

Original: Knowledge Distillation (KD) (Hinton et al., 2015) trains the student with the following loss:
Revised: Knowledge Distillation (KD) (Hinton et al., 2015) trains the student with the following loss:

Original: [Equation]
Revised: [Equation]

Original: so that the discrepancy between the teacher’s and student’s classiﬁers is directly minimized.
Revised: so that the discrepancy between the teacher’s and student’s classiﬁers is directly minimized.

Original: FitNets (Romero et al., 2015) match intermediate feature representations.
Revised: FitNets (Romero et al., 2015) match intermediate feature representations.

Original: For the i -th layer, the following loss is deﬁned:
Revised: For the i -th layer, the following loss is deﬁned:

Original: [Equation]
Revised: [Equation]

Original: where r ( . ) is a function for matching the feature tensor dimensions.
Revised: where r ( . ) is a function for matching the feature tensor dimensions.

Original: In our work, we propose to minimize the discrepancy between the representations h T and h S .
Revised: In our work, we propose to minimize the discrepancy between the representations h T and h S .

Original: To accomplish this goal, we propose to use two losses.
Revised: To accomplish this goal, we propose to use two losses.

Original: The ﬁrst one is an L 2 feature matching loss:
Revised: The ﬁrst one is an L 2 feature matching loss:

Original: [Equation]
Revised: [Equation]

Original: where for notational simplicity we dropped the dependency on r ( . ) .
Revised: where for notational simplicity we dropped the dependency on r ( . ) .

Original: Hence, L FM loss is a simpliﬁed
Revised: Hence, L FM loss is a simpliﬁed

Original: FitNet loss which focuses only on the ﬁnal representation learned.
Revised: FitNet loss which focuses only on the ﬁnal representation learned.

Original: The intuition for this is that this feature is directly connected to the classiﬁer and hence imposing the student’s feature to be similar to that of the teacher could have more impact on classiﬁcation accuracy.
Revised: The intuition for this is that this feature is directly connected to the classiﬁer and hence imposing the student’s feature to be similar to that of the teacher could have more impact on classiﬁcation accuracy.

Original: Moreover, it might be questionable why one should optimize for other intermediate representations as in (Romero et al., 2015) especially when the student is a network of lower representational capacity.
Revised: Moreover, it might be questionable why one should optimize for other intermediate representations as in (Romero et al., 2015) especially when the student is a network of lower representational capacity.

Original: In Section 4:
Revised: In Section 4:

Original: Where should the losses be applied?
Revised: Where should the losses be applied?

Original: , we conﬁrm that L FM alone has a positive impact but feature matching in other layers is not helpful.
Revised: , we conﬁrm that L FM alone has a positive impact but feature matching in other layers is not helpful.

Original: We found L FM to be effective but only to limited extent.
Revised: We found L FM to be effective but only to limited extent.

Original: One disadvantage of L FM and, in general, of all feature matching losses e.g. (Romero et al., 2015; Zagoruyko & Komodakis, 2017), is that it treats each channel dimension in the feature space independently, and ignores the inter-channel dependencies of the feature representations h S and h T for the ﬁnal classiﬁcation.
Revised: One disadvantage of L FM and, in general, of all feature matching losses e.g. (Romero et al., 2015; Zagoruyko & Komodakis, 2017), is that it treats each channel dimension in the feature space independently, and ignores the inter-channel dependencies of the feature representations h S and h T for the ﬁnal classiﬁcation.

Original: This is in contrast to the original logit matching loss proposed by Hinton et al.
Revised: This is in contrast to the original logit matching loss proposed by Hinton et al.

Original: in (Hinton et al., 2015) which directly targets classiﬁcation accuracy.
Revised: in (Hinton et al., 2015) which directly targets classiﬁcation accuracy.

Original: To alleviate the aforementioned problem, in this work, we propose a second loss for optimizing h S which is directly linked with classiﬁcation accuracy.
Revised: To alleviate the aforementioned problem, in this work, we propose a second loss for optimizing h S which is directly linked with classiﬁcation accuracy.

Original: To this end, we will use the teacher’s pre-trained Softmax Regression (SR) classiﬁer.
Revised: To this end, we will use the teacher’s pre-trained Softmax Regression (SR) classiﬁer.

Original: Let us denote by p the output of the teacher network when fed with some input image x .
Revised: Let us denote by p the output of the teacher network when fed with some input image x .

Original: Let us also feed the same image through the student network to obtain feature h S ( x ) .
Revised: Let us also feed the same image through the student network to obtain feature h S ( x ) .

Original: Finally let us pass h S ( x ) through the teacher’s SR classiﬁer to obtain output q .
Revised: Finally let us pass h S ( x ) through the teacher’s SR classiﬁer to obtain output q .

Original: See also Fig.
Revised: See also Fig.

Original: Our loss is deﬁned as:
Revised: Our loss is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: At this point, we make the following two observations: (1) If p = q (and since the teacher’s classiﬁer is frozen), then this implies that h S ( x ) = h T ( x ) which shows that indeed Eq.
Revised: At this point, we make the following two observations: (1) If p = q (and since the teacher’s classiﬁer is frozen), then this implies that h S ( x ) = h T ( x ) which shows that indeed Eq.

Original: (4) optimizes the student’s feature representation h S ( h T is also frozen).
Revised: (4) optimizes the student’s feature representation h S ( h T is also frozen).

Original: The loss of Eq.(4) can be written as:
Revised: The loss of Eq.(4) can be written as:

Original: [Equation]
Revised: [Equation]

Original: Now let us now write KD loss in a similar way:
Revised: Now let us now write KD loss in a similar way:

Original: [Equation]
Revised: [Equation]

Original: By comparing Eq.
Revised: By comparing Eq.

Original: (5) with Eq.
Revised: (5) with Eq.

Original: (6), we see that the only difference in our method is that the frozen, pre-trained teacher’s classiﬁer is used for both teacher and the student.
Revised: (6), we see that the only difference in our method is that the frozen, pre-trained teacher’s classiﬁer is used for both teacher and the student.

Original: On the contrary, in KD, W S is also optimized.
Revised: On the contrary, in KD, W S is also optimized.

Original: This gives more degrees of freedom to the optimization algorithm, in particular, to adjust the weights of both the student’s feature extractor f S and the student’s classiﬁer W S in order to minimize the loss.
Revised: This gives more degrees of freedom to the optimization algorithm, in particular, to adjust the weights of both the student’s feature extractor f S and the student’s classiﬁer W S in order to minimize the loss.

Original: This has an impact on the learning of the student’s feature representation h S which, in turn, hinders the generalization capability of the student on the test set.
Revised: This has an impact on the learning of the student’s feature representation h S which, in turn, hinders the generalization capability of the student on the test set.

Original: We conﬁrm this hypothesis with the experiment of Section 4: Transferability of representations .
Revised: We conﬁrm this hypothesis with the experiment of Section 4: Transferability of representations .

Original: Finally, we note that we found that, in practice, an L 2 loss between the logits:
Revised: Finally, we note that we found that, in practice, an L 2 loss between the logits:

Original: [Equation]
Revised: [Equation]

Original: (7) works slightly better than the cross-entropy loss.
Revised: (7) works slightly better than the cross-entropy loss.

Original: The comparison between different types of losses for L SR is given in the supplementary material.
Revised: The comparison between different types of losses for L SR is given in the appendix.

Original: [Table]
Revised: [Table]

Original: Overall, in our method, we train the student network using three losses:
Revised: Overall, in our method, we train the student network using three losses:

Original: [Equation]
Revised: [Equation]

Original: where α and β are the weights used to scale the losses.
Revised: where α and β are the weights used to scale the losses.

Original: The teacher network is pretrained and ﬁxed during training the student.
Revised: The teacher network is pretrained and ﬁxed during training the student.

Original: L CE is the standard loss based on ground truth labels for the task in hand (e.g. cross-entropy loss for image classiﬁcation).
Revised: L CE is the standard loss based on ground truth labels for the task in hand (e.g. cross-entropy loss for image classiﬁcation).

Original: Note that this results in a very simple algorithm for training the student summarized in Algorithm 1.
Revised: Note that this results in a very simple algorithm for training the student, summarized in Algorithm 1.

Original:  A BLATION S TUDIES
Revised:  A BLATION S TUDIES

Original: We conducted a set of ablation studies on CIFAR-100 (see Section 5.1) using a Wide ResNet (WRN) for both teacher (WRN-40-4) and student (WRN-16-4) (for network deﬁnitions, see Section 5).
Revised: We conducted a set of ablation studies on CIFAR-100 (see Section 5.1) using a Wide ResNet (WRN) for both teacher (WRN-40-4) and student (WRN-16-4) (for network deﬁnitions, see Section 5).

Original: Are both L FM and L SR useful?
Revised: Are both L FM and L SR useful?

Original: To answer this question, we ran 3 experiments: using L FM alone, L SR alone, and combining them together L FM + L SR .
Revised: To answer this question, we ran 3 experiments: using L FM alone, L SR alone, and combining them together L FM + L SR .

Original: The results of Table 1 (ﬁrst 3 rows) clearly show that all proposed variants offer signiﬁcant gains: when using L FM and L SR alone, ∼ 1% and ∼ 2% improvements in Top-1 accuracy were obtained.
Revised: The results of Table 1 (ﬁrst 3 rows) clearly show that all proposed variants offer signiﬁcant gains: when using L FM and L SR alone, ∼ 1% and ∼ 2% improvements in Top-1 accuracy were obtained.

Original: Moreover, when combining them together, an additional ∼ 0 .
Revised: Moreover, when combining them together, an additional ∼ 0 .

Original: 4% improvement was gained.
Revised: 4% improvement was gained.

Original: Importantly, the results show that L SR is signiﬁcantly more effective than L FM .
Revised: Importantly, the results show that L SR is signiﬁcantly more effective than L FM .

Original: We further note at this point that we found that L FM offers diminishing gains on ImageNet experiments.
Revised: We further note at this point that we found that L FM offers diminishing gains on ImageNet experiments.

Original: Where should the losses be applied?
Revised: Where should the losses be applied?

Original: The proposed losses can be applied at other layers of the networks too.
Revised: The proposed losses can be applied at other layers of the networks too.

Original: This is straightforward for L FM .
Revised: This is straightforward for L FM .

Original: We can also extend L SR to more layers, by transferring the mean feature of the student at each layer to the corresponding layer of the teacher using an
Revised: We can also extend L SR to more layers, by transferring the mean feature of the student at each layer to the corresponding layer of the teacher using an

Original: AdaIN layer (Huang & Belongie, 2017).
Revised: AdaIN layer (Huang & Belongie, 2017).

Original: On one hand, applying the losses early in the network could ensure that the subsequent layers receive “better” features.
Revised: On one hand, applying the losses early in the network could ensure that the subsequent layers receive “better” features.

Original: On the other hand, features produced by early layers are not specialised to a particular class.
Revised: On the other hand, features produced by early layers are not specialised to a particular class.

Original: Thus, applying the distillation losses towards the end of the network, where the activations encode discriminative, task-related features should lead to potentially stronger models.
Revised: Thus, applying the distillation losses towards the end of the network, where the activations encode discriminative, task-related features should lead to potentially stronger models.

Original: The results from Table 1 (last 3 rows) conﬁrm our hypothesis:
Revised: The results from Table 1 (last 3 rows) conﬁrm our hypothesis:

Original: By applying the loss at multiple points in the network actually rather hurts accuracy.
Revised: Applying the loss at multiple points in the network actually rather hurts accuracy.

Original: Teacher-student similarity: The overall aim of knowledge distillation is to make the student mimic the teacher’s output, so that the student is able to obtain similar performance to that of the teacher.
Revised: Teacher-student similarity: The overall aim of knowledge distillation is to make the student mimic the teacher’s output, so that the student is able to obtain similar performance to that of the teacher.

Original: Therefore, to see how well the student mimics the teacher, we measured the similarity between the teacher’s and student’s outputs using (a) the KL divergence between the teacher’s and student’s outputs, and (b) the cross-entropy loss between the student’s predictions and the ground truth labels.
Revised: Therefore, to see how well the student mimics the teacher, we measured the similarity between the teacher’s and student’s outputs using (a) the KL divergence between the teacher’s and student’s outputs, and (b) the cross-entropy loss between the student’s predictions and the ground truth labels.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: From Table 2, it can be observed that KD (Hinton et al., 2015) reduces the KL divergence with the teacher’s output offering ∼ 1 .
Revised: From Table 2, it can be observed that KD (Hinton et al., 2015) reduces the KL divergence with the teacher’s output offering ∼ 1 .

Original: 5% accuracy gain.
Revised: 5% accuracy gain.

Original: AT (Zagoruyko & Komodakis, 2017) also decreases the KL divergence with the teacher’s output offering a smaller accuracy gain of ∼ 1 . 0% .
Revised: AT (Zagoruyko & Komodakis, 2017) also decreases the KL divergence with the teacher’s output offering a smaller accuracy gain of ∼ 1 . 0% .

Original: Moreover, both proposed losses L FM and L SR and their combination L FM + L SR show considerably high similarity compared to the KD and AT.
Revised: Moreover, both proposed losses L FM and L SR and their combination L FM + L SR show considerably high similarity compared to the KD and AT.

Original: This similarity is one of the main reasons for the improved student’s accuracy offered by our method.
Revised: This similarity is one of the main reasons for the improved student’s accuracy offered by our method.

Original: [Figure]
Revised: [Figure]

Original: Representations distance: Table 3 shows thecalculated L 2 distance between the teacher and student representations h T and h S .
Revised: Representations distance: Table 3 shows the L 2 distance between the teacher and student representations h T and h S .

Original: The results, presented in Table 3, clearly show that both L FM and L SR narrow the distance, with their combination being the closest to the teacher.
Revised: The results, presented in Table 3, clearly show that both L FM and L SR narrow the distance, with their combination being the closest to the teacher.

Original: Normalized Mutual Information (NMI): Moreover, we calculated the NMI (Manning et al., 2008) which is a balanced metric that can be used to determine the quality of feature clustering.
Revised: Normalized Mutual Information (NMI): Moreover, we calculated the NMI (Manning et al., 2008) which is a balanced metric that can be used to determine the quality of feature clustering.

Original: The results, presented in Table 3, show that L FM + L SR has the highest NMI score meaning that the features are better clustered.
Revised: The results, presented in Table 3, show that L FM + L SR has the highest NMI score, meaning that the features are better clustered.

Original: Qualitative results are shown in Figure 2 which visualizes the features h S and h T .
Revised: Qualitative results are shown in Figure 2, which visualizes the features h S and h T .

Original: It can be observed that L FM + L SR is able to learn more discriminative features, which also correlates with quantitative accuracy gains.
Revised: It can be observed that L FM + L SR is able to learn more discriminative features, which also correlates with quantitative accuracy gains.

Original: Transferability of representations: Following (Tian et al., 2020), this section aims to compare the representational power of the learned student’s representation h s .
Revised: Transferability of representations: Following (Tian et al., 2020), this section aims to compare the representational power of the learned student’s representation h s .

Original: To this end, we trained the student on CIFAR100, and then used it as a frozen feature extractor on top of which we train a linear classiﬁer for 2 datasets: STL10 Coates et al.
Revised: To this end, we trained the student on CIFAR100, and then used it as a frozen feature extractor on top of which we train a linear classiﬁer for 2 datasets: STL10 Coates et al.

Original: (2011) and CIFAR100.
Revised: (2011) and CIFAR100.

Original: We compare the transfer ability of KD, CRD, L FM , L SR , and L FM + L SR .
Revised: We compare the transfer ability of KD, CRD, L FM , L SR , and L FM + L SR .

Original: The superiority of the proposed losses over KD on
Revised: The superiority of the proposed losses over KD on

Original: STL is evident.
Revised: STL is evident.

Original: Importantly, L SR largely outperforms KD which conﬁrms our analysis of Eqs.
Revised: Importantly, L SR largely outperforms KD which conﬁrms our analysis of Eqs.

Original: (5) and (6).
Revised: (5) and (6).

Original: The best results on STL are obtained by CRD.
Revised: The best results on STL are obtained by CRD.

Original: However, on CIFAR100, which is the target distillation dataset our method outperforms CRD.
Revised: However, on CIFAR100, which is the target distillation dataset our method outperforms CRD.

Original:  C OMPARISON WITH STATE - OF - THE - ART
Revised:  C OMPARISON WITH STATE - OF - THE - ART

Original: We thoroughly evaluated the effectiveness of our method across multiple (a) network architectures (ResNet (He et al., 2016), Wide ResNet (Zagoruyko & Komodakis, 2016), MobileNetV2 (Sandler et al., 2018), MobileNet (Howard et al., 2017)) with different teacher-student capacities, (b) datasets
Revised: We thoroughly evaluated the effectiveness of our method across multiple (a) network architectures (ResNet (He et al., 2016), Wide ResNet (Zagoruyko & Komodakis, 2016), MobileNetV2 (Sandler et al., 2018), MobileNet (Howard et al., 2017)) with different teacher-student capacities, (b) datasets

Original: [Table] [Table]
Revised: [Table] [Table]

Original: (CIFAR10/100, ImageNet), and (c) domains (real-valued and binary networks).
Revised: (CIFAR10/100, ImageNet), and (c) domains (real-valued and binary networks).

Original: The training details for all experiments are provided in the supplementary material.
Revised: The training details for all experiments are provided in the appendix.

Original: We denote with ResNet-N a Residual Network with N convolutional layers (He et al., 2016).
Revised: We denote with ResNet-N a Residual Network with N convolutional layers (He et al., 2016).

Original: We denote with WRN- D - k a WRN architecture with D layers and an expansion rate of k (Zagoruyko & Komodakis, 2017).
Revised: We denote with WRN- D - k a WRN architecture with D layers and an expansion rate of k (Zagoruyko & Komodakis, 2017).

Original: For the above mentioned settings, we compare our method with KD (Hinton et al., 2015) and
Revised: For the above mentioned settings, we compare our method with KD (Hinton et al., 2015) and

Original: AT (Zagoruyko & Komodakis, 2017), and the more recent methods of OFD (Heo et al., 2019a),
Revised: AT (Zagoruyko & Komodakis, 2017), and the more recent methods of OFD (Heo et al., 2019a),

Original: RKD (Park et al., 2019), CRD (Tian et al., 2020).
Revised: RKD (Park et al., 2019), CRD (Tian et al., 2020).

Original: Overview of results: From our experiments, we conclude that our approach offers consistent gains across all of the above scenarios, outperforming all methods considered for all settings.
Revised: Overview of results: From our experiments, we conclude that our approach offers consistent gains across all of the above scenarios, outperforming all methods considered for all settings.

Original: Notably, our method is particularly effective for the most difﬁcult datasets (i.e. CIFAR-100 and ImageNet).
Revised: Notably, our method is particularly effective for the most difﬁcult datasets (i.e. CIFAR-100 and ImageNet).

Original: 5. CIFAR-10/ 
Revised: 5. CIFAR-10/

Original: For CIFAR-10, Top-1 performance of our method is shown in Table 5.
Revised: For CIFAR-10, Top-1 performance of our method is shown in Table 5.

Original: We tested three cases representing different network architectures for student and teacher networks: the ﬁrst two experiments are with WRNs.
Revised: We tested three cases representing different network architectures for student and teacher networks: the ﬁrst two experiments are with WRNs.

Original: The following three experiments are with ResNets.
Revised: The following three experiments are with ResNets.

Original: In the last experiment, teacher and student have different network architectures.
Revised: In the last experiment, teacher and student have different network architectures.

Original: Overall, our method achieves the best results for all cases, with KD (Hinton et al., 2015) closely following.
Revised: Overall, our method achieves the best results for all cases, with KD (Hinton et al., 2015) closely following.

Original: For CIFAR-100 (Krizhevsky & Hinton, 2009), we experimented with several student-teacher network pairs using different structures.
Revised: For CIFAR-100 (Krizhevsky & Hinton, 2009), we experimented with several student-teacher network pairs using different structures.

Original: Experiments are grouped in three sets.
Revised: Experiments are grouped in three sets.

Original: The ﬁrst shows performance for different teacher and student capacities using WRNs: poor student - good teacher (WRN-16-2; WRN-40-4), descent student - good teacher (WRN-10-10; WRN-16-10); good student - good teacher (WRN-16-4; WRN-40-4).
Revised: The ﬁrst shows performance for different teacher and student capacities using WRNs: poor student - good teacher (WRN-16-2; WRN-40-4), descent student - good teacher (WRN-10-10; WRN-16-10); good student - good teacher (WRN-16-4; WRN-40-4).

Original: In the second set, we show that these results hold when using a different architecture, ResNet in this case.
Revised: In the second set, we show that these results hold when using a different architecture, ResNet in this case.

Original: The ﬁnal set is designed to show performance when teacher and student have different architectures (MobileNetV2, ResNet and WRN).
Revised: The ﬁnal set is designed to show performance when teacher and student have different architectures (MobileNetV2, ResNet and WRN).

Original: Top-1 performance of our method is shown in Table 10.
Revised: Top-1 performance of our method is shown in Table 11.

Original: We observe that for almost all conﬁgurations, our method achieves consistent and signiﬁcant accuracy gains over prior work.
Revised: We observe that for almost all conﬁgurations, our method achieves consistent and signiﬁcant accuracy gains over prior work.

Original: Furthermore, it is hard to tell which is the second best method as the remaining methods have their own advantages for different conﬁgurations.
Revised: Furthermore, it is hard to tell which is the second best method as the remaining methods have their own advantages for different conﬁgurations.

Original: For WRN experiments, OFD ranks second.
Revised: For WRN experiments, OFD ranks second.

Original: For ResNet and mixed structure experiments, CRD ranks second.
Revised: For ResNet and mixed structure experiments, CRD ranks second.

Original: More comparisons with other methods and results obtained by combining our method with KD and AT are provided in the supplementary material.
Revised: More comparisons with other methods and results obtained by combining our method with KD and AT are provided in the supplementary material.

Original: Further improvements could be obtained by combining our method with others but this requires a comprehensive investigation which goes beyond the scope of this paper.
Revised: Further improvements could be obtained by combining our method with others but this requires a comprehensive investigation which goes beyond the scope of this paper.

Original: [Table]
Revised: [Table]

Original: -the-art on ImageNet.
Revised: -the-art on ImageNet.

Original: [Table]
Revised: [Table]

Original: 5. I MAGE N ET -1K
Revised: 5. I MAGE N ET -1K

Original: Our experiments include two pairs of networks which are popular settings for ImageNet (Russakovsky et al., 2015).
Revised: Our experiments include two pairs of networks which are popular settings for ImageNet (Russakovsky et al., 2015).

Original: The ﬁrst is distillation from ResNet-34 to ResNet-18 and the second one is distillation from ResNet-50 to MobileNet (Howard et al., 2017).
Revised: The ﬁrst is distillation from ResNet-34 to ResNet-18 and the second one is distillation from ResNet-50 to MobileNet (Howard et al., 2017).

Original: Note that, following (Tian et al., 2020) on ImageNet, for KD, we set the weight for  KL loss to 0.9, the weight for cross-entropy loss to 0.5 which helps to obtain better accuracy.
Revised: Note that, following (Tian et al., 2020) on ImageNet, for KD, we set the weight for the KL loss to 0.9, the weight for cross-entropy loss to 0.5 which helps to obtain better accuracy.

Original: Our results are presented in Table 7.
Revised: Our results are presented in Table 7.

Original: Again, we observe that our method achieves signiﬁcant improvements over all  methods.
Revised: Again, we observe that our method achieves signiﬁcant improvements over all competing methods.

Original: Moreover, there is no method which is consistently second: for ResNet-34 to ResNet-18 experiment, RKD is the second best while for ResNet-50 to MobileNet,
Revised: Moreover, there is no method which is consistently second: for ResNet-34 to ResNet-18 experiment, RKD is the second best while for ResNet-50 to MobileNet,

Original: CRD is the second best.
Revised: CRD is the second best.

Original: Notably, for the latter experiment, CRD reduces the gap between the teacher and the student by 1 . 27% , while our method narrows it by 2 .
Revised: Notably, for the latter experiment, CRD reduces the gap between the teacher and the student by 1 . 27% , while our method narrows it by 2 .

Original: Overall, our results on ImageNet validate the scalability of our method, and show that, when applied to a large-scale dataset, we achieve an even more favourable performance compared against competing methods.
Revised: Overall, our results on ImageNet validate the scalability of our method [Table] and show that, when applied to a large-scale dataset, we achieve an even more favourable performance compared against competing methods.

Original:  used to distill a binary student.
Revised: 34 is used to distill a binary student.

Original: Real-to-binary distillation results on ImageNet-1K : a real-valued ResNetis used to distill a binary student.
Revised: Real-to-binary distillation results on ImageNet-1K: a real-valued ResNetis used to distill a binary student.

Original: OFD result might be suboptimal.
Revised: OFD result might be suboptimal.

Original: [Table]
Revised: [Table]

Original: 5. B INARY N ETWORKS DISTILLATION
Revised: 5. B INARY N ETWORKS DISTILLATION

Original: Training highly accurate binary neural networks (i.e. the most extreme case of quantization) is a very challenging task (Rastegari et al., 2016;
Revised: Training highly accurate binary neural networks (i.e. the most extreme case of quantization) is a very challenging task (Rastegari et al., 2016;

Original: Bulat & Tzimiropoulos, 2019), and to this end, knowledge distillation appears to be a promising direction.
Revised: Bulat & Tzimiropoulos, 2019), and to this end, knowledge distillation appears to be a promising direction.

Original: In this section, we present results by applying distillation for the task of training binary student networks guided by real-valued teacher networks.
Revised: In this section, we present results by applying distillation for the task of training binary student networks guided by real-valued teacher networks.

Original: The network architecture is kept the same for both the student and the teacher in this case: speciﬁcally we used a ResNet using the modiﬁcations described in (Bulat & Tzimiropoulos, 2019).
Revised: The network architecture is kept the same for both the student and the teacher in this case: speciﬁcally we used a ResNet using the modiﬁcations described in (Bulat & Tzimiropoulos, 2019).

Original: Table 8 presents our results.
Revised: Table 8 presents our results.

Original: Again, we observe that our method outperforms all methods considerably, showing that it can effectively transfer knowledge between different domains.
Revised: Again, we observe that our method outperforms all methods considerably, showing that it can effectively transfer knowledge between different domains.

Original: Note that, it was not clear to us where to place the distillation position for OFD, so although we included our result for this method, we emphasize that this result might be suboptimal. 
Revised: Note that it was not clear to us where to place the distillation position for OFD, so although we included our result for this method, we emphasize that this result might be suboptimal. 5. F ACIAL LANDMARK DETECTION

Revised: Given a face image, the task is to localise a set of facial landmarks in terms of their (x,y) coordinates.

Original: C ONCLUSION
Revised: This is often solved by using a CNN to directly regress the (x,y) coordinates of the facial landmarks. In order to show the suitability of our method for this problem, we use the WFLW Wu et al. (2018) dataset, which is one of the hardest benchmarks for this task. Performance is measured in terms of Normalised Mean Error (lower is better), which is the standard metric for the problem.

Revised: In our experiment, we use a ResNet50 as the teacher and a ResNet8 as the student.

Revised: The results, shown in

Revised: Table 9, conﬁrm the superior performance of our method when compared to other state-of-the-art methods.

Revised: [Table]

Revised:  C ONCLUSION

Original: We presented a method for knowledge distillation that optimizes the output feature of the penultimate layer of the student network and hence is directly related to representation learning.
Revised: We presented a method for knowledge distillation that optimizes the output feature of the penultimate layer of the student network and hence is directly related to representation learning.

Original: A key to our method is the newly proposed Softmax Regression Loss which was found necessary for effective representation learning.
Revised: A key to our method is the newly proposed Softmax Regression Loss which was found necessary for effective representation learning.

Original: We showed that our method consistently outperforms other stateof-the-art distillation methods for a wide range of experimental settings including multiple network architectures (ResNet, Wide ResNet, MobileNet) with different teacher-student capacities, datasets (CIFAR10/100, ImageNet) and domains (real-valued and binary networks).
Revised: We showed that our method consistently outperforms other stateof-the-art distillation methods for a wide range of experimental settings including multiple network architectures (ResNet, Wide ResNet, MobileNet) with different teacher-student capacities, datasets (CIFAR10/100, ImageNet), and domains (real-valued and binary networks).

Original: A A PPENDIX A. S TUDY ON THE HYPER - PARAMETER α AND β
Revised: A A PPENDIX A. S TUDY OF THE HYPER - PARAMETERS α AND β

Original: Wedid only  a  basic search to ﬁnd the best hyper-parameters.
Revised: We only performed a very basic search to ﬁnd the best hyper-parameters.

Original: First, we ﬁx α and search for the best β .
Revised: First, we ﬁx α and search for the best β .

Original: Then, we used the best β  and search for the best α .
Revised: Then, we used the best β , and search for the best α .

Original: This is suboptimal compared to a full grid search over α and β .
Revised: This is sub-optimal compared to a full grid search over α and β .

Original: Furthermore,  we searched only for 2 values: 1 and 5.
Revised: Furthermore, after some preliminary experimentation, we considered only for 2 values: 1 and 5.

Original: Notably, we found that for all teacher-student pairs but (T:WRN40 4,S:MV2)
Revised: Notably, we found that for all teacher-student pairs but (T:WRN40 4,

Original:  alpha=1 is the optimal value.
Revised: S:MV2) alpha=1 is the optimal value.

Original: Furthermore, on ImageNet, the optimal values were α = 1 , β = 1 for all teacher-student pairs considered.
Revised: Furthermore, on ImageNet, the optimal values were α = 1 , β = 1 for all teacher-student pairs considered.

Original: A. D ATASETS AND TRAINING DETAILS
Revised: A. D ATASETS AND TRAINING DETAILS

Original: CIFAR-10 CIFAR-10 is a popular image classiﬁcation dataset consisting of 50,000 training and 10,000 testing images equally distributed across 10 classes.
Revised: CIFAR-10 CIFAR-10 is a popular image classiﬁcation dataset consisting of 50,000 training and 10,000 testing images equally distributed across 10 classes.

Original: All images are of resolution 32 × 32 px.
Revised: All images are of resolution 32 × 32 px.

Original: Following (Zagoruyko & Komodakis, 2017), during training, we randomly cropped and horizontally ﬂipped the images.
Revised: Following (Zagoruyko & Komodakis, 2017), during training, we randomly cropped and horizontally ﬂipped the images.

Original: The ResNet models were trained for 350 epochs using SGD.
Revised: The ResNet models were trained for 350 epochs using SGD.

Original: The initial learning rate was set to 0.1, and then it was reduced by a factor of 10 at epochs 150, 250 and 320.
Revised: The initial learning rate was set to 0.1, and then it was reduced by a factor of 10 at epochs 150, 250 and 320.

Original: Similarly, the WRN models were trained for 200 epochs with a learning rate of 0.1 that was subsequently 
Revised: Similarly, the WRN models were trained for 200 epochs with a learning rate of 0.1 that was subsequently reduced by 5 at epochs 60, 120 and 160. In all experiments, we set the dropout rate to 0.

Original: For traditional KD (Hinton et al., 2015), we set α = 0 .
Revised: For traditional KD (Hinton et al., 2015), we set α = 0 .

Original: 9 and T = 4 .
Revised: 9 and T = 4 .

Original: For AT (Zagoruyko &
Revised: For AT (Zagoruyko &

Original: Komodakis, 2017), as in (Zagoruyko & Komodakis, 2017; Tung & Mori, 2019), we set the weight of distillation loss to 1000.
Revised: Komodakis, 2017), as in (Zagoruyko & Komodakis, 2017; Tung & Mori, 2019), we set the weight of distillation loss to 1000.

Original: We note, that for our experiments, the AT loss is added after each layer group for WRN and the last two groups for ResNet as in (Zagoruyko & Komodakis, 2017).
Revised: We note that, in our experiments, the AT loss is added after each layer group for WRN and the last two groups for ResNet as in (Zagoruyko & Komodakis, 2017).

Original: Following OFD (Heo et al., 2019a), we set the weight of distillation loss to 10 − 3 .
Revised: Following OFD (Heo et al., 2019a), we set the weight of distillation loss to 10 − 3 .

Original: For RKD (Park et al., 2019), we set β 1 = 25 for distance, and β 2 = 50 for angle, as described in (Park et al., 2019; Tian et al., 2020).
Revised: For RKD (Park et al., 2019), we set β 1 = 25 for distance, and β 2 = 50 for angle, as described in (Park et al., 2019; Tian et al., 2020).

Original: We did not compare with CRD (Tian et al., 2020) on CIFAR-10 because, in our experiments, we found that their parameter setting (used in their paper for CIFAR-100 and ImageNet-1K) does not obtain good performance on CIFAR-10.
Revised: We did not compare with CRD (Tian et al., 2020) on CIFAR-10 because, in our experiments, we found that their parameter setting (used in their paper for CIFAR-100 and ImageNet-1K) does not obtain good performance on CIFAR-10.

Original: CIFAR-100 For CIFAR-100 (Krizhevsky & Hinton, 2009), we used a standard data augmentation scheme (Zagoruyko & Komodakis, 2017) including padding 4 pixels prior to random cropping and horizontal ﬂipping.
Revised: CIFAR-100 For CIFAR-100 (Krizhevsky & Hinton, 2009), we used a standard data augmentation scheme (Zagoruyko & Komodakis, 2017) including padding 4 pixels prior to random cropping and horizontal ﬂipping.

Original: We used SGD with weight decay 5e-4 and momentum 0.9.
Revised: We used SGD with weight decay 5e-4 and momentum 0.9.

Original: Batch size was set to 128.
Revised: Batch size was set to 128.

Original: Learning rate was set to 0.1; then decayed by 0.1 at epochs 100, 150, until training reached 200 epochs (Heo et al., 2019a).
Revised: Learning rate was set to 0.1; then decayed by 0.1 at epochs 100, 150, until training reached 200 epochs (Heo et al., 2019a).

Original: ImageNet-1K Images are cropped to 224 × 224 pixels for both training and evaluation.
Revised: ImageNet-1K Images are cropped to 224 × 224 pixels for both training and evaluation.

Original: We used
Revised: We used

Original: SGD with Nesterov momentum 0.9, weight decay 1 e − 4 , initial learning rate 0.2 which was then dropped by a factor of 10 every 30 epochs, training in total for 100 epochs (for CRD we trained with 10 more epochs as suggested by the authors).
Revised: SGD with Nesterov momentum 0.9, weight decay 1 e − 4 , initial learning rate 0.2 which was then dropped by a factor of 10 every 30 epochs, training in total for 100 epochs (for CRD we trained with 10 more epochs as suggested by the authors).

Original: Batch size was set to 512.
Revised: Batch size was set to 512.

Original: For simplicity and to enable a fair comparison, we used pretrained PyTorch models Paszke et al.
Revised: For simplicity and to enable a fair comparison, we used pretrained PyTorch models Paszke et al.

Original: (2017) as teacher networks Heo et al. (2019a)
Revised: (2017) as teacher networks Heo et al. (2019a)

Original: ; Tian et al.
Revised: ; Tian et al.

Original: For binary experiments, we used Adam as the optimizer with initial learning 0.002 which was then reduced by a factor of 10 every 30 epochs, training in total forepochs.
Revised: For binary experiments, we used Adam as the optimizer with initial learning 0.002 which was then reduced by a factor of 10 every 30 epochs, training in total forepochs.

Original: A. A DDITIONAL ABLATION STUDIES
Revised: A. A DDITIONAL ABLATION STUDIES

Original: Different losses for L SR : This part expands Section 4 of our paper by evaluating different losses for L SR .
Revised: Different losses for L SR : This part expands Section 4 of our paper by evaluating different losses for L SR .

Original: The following loss functions are compared:
Revised: The following loss functions are compared:

Original: L2 loss: L SR − L 2 ( p, q ) =
Revised: L2 loss: L SR − L 2 ( p, q ) =

Original: (cid:107) p − q (cid:107) 2 .
Revised: (cid:107) p − q (cid:107) 2 .

Original: This is the loss used in Section 4 of our paper 2.
Revised: This is the loss used in Section 4 of our paper 2.

Original: Cross Entropy loss (CE) with label y :
Revised: Cross Entropy loss (CE) with label y :

Original: L SR − CE ( q, y )
Revised: L SR − CE ( q, y )

Original: =
Revised: =

Original: H ( q, y ) .
Revised: H ( q, y ) .

Original: KL loss with temperature τ Hinton et al.
Revised: KL loss with temperature τ Hinton et al.

Original: (2015): L SR − KL ( p, q ) =
Revised: (2015): L SR − KL ( p, q ) =

Original: KL ( q/τ, p/τ ) .
Revised: KL ( q/τ, p/τ ) .

Original: The results, presented in Table 9, show that all loss functions offer signiﬁcant improvement gains while L FM + L SR − L 2 achieves the best accuracy.
Revised: The results, presented in Table 10, show that all loss functions offer signiﬁcant improvement gains while L FM + L SR − L 2 achieves the best accuracy.

Original: Therefore, in our paper, L SR − L 2 is used in all cases.
Revised: Therefore, in our paper, L SR − L 2 is used in all cases.

Original: [Table]
Revised: [Table]

Original: Combining our method with KD and AT:
Revised: Combining our method with KD and AT:

Original: Table 10 shows additional comparisons on CIFAR by combining our method with AT Zagoruyko & Komodakis (2017) and KD Hinton et al.
Revised: Table 11 shows additional comparisons on CIFAR by combining our method with AT Zagoruyko & Komodakis (2017) and KD Hinton et al.

Original: (2015), respectively.
Revised: (2015), respectively.

Original: The results show that a straightforward combination did not provide satisfactory results, however it could be possible that a more comprehensive investigation might prove to be beneﬁcial.
Revised: The results show that a straightforward combination did not provide satisfactory results, however it could be possible that a more comprehensive investigation might prove to be beneﬁcial

Original:  A. A DDITIONAL COMPARISONS
Revised: . A. A DDITIONAL COMPARISONS

Original: This section provides additional comparisons using the evaluation framework of CRD Tian et al.
Revised: This section provides additional comparisons using the evaluation framework of CRD Tian et al.

Original: Comparisons include distillation between models with the same architecture (e.g. ResNet to ResNet20) and between different architectures (e.g. ResNet50 to MobileNetV2). 
Revised: Comparisons include distillation between models with the same architecture (e.g. ResNet to ResNet20) and between different architectures (e.g. ResNet50 to MobileNetV2). In order to

Original: In order to maximize the fairness of the comparison, we followed their experimental setting.
Revised: [Table] maximize the fairness of the comparison [Table] we followed their experimental setting.

Original: Thus, we did not choose the training parameters, teacher-student architecture pairs or methods to compare against
Revised: Thus, we did not choose the training parameters, teacher-student architecture pairs or methods to compare against

Original: [Table]
Revised: [Table]

Original: The competing methods included are:
Revised: The competing methods included are:

Original: • Classic: Knowledge Distillation (KD) Hinton et al.
Revised: • Classic: Knowledge Distillation (KD) Hinton et al.

Original: FitNet Romero et al.
Revised: FitNet Romero et al.

Original: , Attention Transfer (AT) Zagoruyko & Komodakis (2017).
Revised: , Attention Transfer (AT) Zagoruyko & Komodakis (2017).

Original: • Most recent:
Revised: • Most recent:

Original: Similarity-Preserving KD (SP) (Tung & Mori, 2019), Correlation Congruence (CC) (Peng et al., 2019), Variational Information Distillation (VID) (
Revised: Similarity-Preserving KD (SP) (Tung & Mori, 2019), Correlation Congruence (CC) (Peng et al., 2019), Variational Information Distillation (VID) (

Original: Ahn et al., 2019), Relational Knowledge Distillation (RKD) (Park et al., 2019), Distillation of Activation Boundaries (AB) (Heo et al., 2019b), Factor Transfer (FT) (Kim et al., 2018), Flow of Solution (FSP)
Revised: Ahn et al., 2019), Relational Knowledge Distillation (RKD) (Park et al., 2019), Distillation of Activation Boundaries (AB) (Heo et al., 2019b), Factor Transfer (FT) (Kim et al., 2018), Flow of Solution (FSP)

Original: (Yim et al., 2017) and Contrastive Representation Distillation (CRD) (Tian et al., 2020).
Revised: (Yim et al., 2017) and Contrastive Representation Distillation (CRD) (Tian et al., 2020).

Original: [Table]
Revised: [Table]

Original: [Table] [Table] [Table] [Table]
Revised: SP 73.

Original: [Table] [Table] [Table] [Table]
Revised: CC 73.

Original: [Table] [Table] [Table]
Revised: VID 74.

Original: [Table] [Table] [Table] [Table]
Revised: RKD 73.

Original: [Table] [Table] [Table] [Table]
Revised: PKT 74.

Original: [Table]
Revised: AB 72.

Original: [Table] [Table] [Table] [Table]
Revised: FT 73.

Original: [Table] [Table] [Table] [Table]
Revised: FSP 72.

Original: [Table] [Table] [Table] [Table]
Revised: NST 73.

Original: [Table] [Table] [Table] [Table]
Revised: CRD 75.

Original: [Table] [Table] [Table] [Table]
Revised: Ours 75.

Original: [Table]AB 66. FT 61. NST 58.
Revised: [Table]

Original: CRD 69.

Original: Ours 69.
Revised: Ours 69.