Original: C ONTRASTIVE R EPRESENTATION D ISTILLATION
Revised: C ONTRASTIVE R EPRESENTATION D ISTILLATION

Original: Anonymous authors
Revised: Tian yonglong@ edu Krishnan

Original: Paper under double-blind review
Revised: Phillip Isola MIT CSAIL phillipi@mit.edu

Original: A BSTRACT
Revised: A BSTRACT

Original: Often we wish to transfer representational knowledge from one neural network to another.
Revised: Often we wish to transfer representational knowledge from one neural network to another.

Original: Examples include distilling a large network into a smaller one, transferring knowledge from one sensory modality to a second, or ensembling a collection of models into a single estimator.
Revised: Examples include distilling a large network into a smaller one, transferring knowledge from one sensory modality to a second, or ensembling a collection of models into a single estimator.

Original: Knowledge distillation, the standard approach to these problems, minimizes the KL divergence between the probabilistic outputs of a teacher and student network.
Revised: Knowledge distillation, the standard approach to these problems, minimizes the KL divergence between the probabilistic outputs of a teacher and student network.

Original: We demonstrate that this objective ignores important structural knowledge of the teacher network.
Revised: We demonstrate that this objective ignores important structural knowledge of the teacher network.

Original: This motivates an alternative objective by which we train a student to capture signiﬁcantly more information in the teacher’s representation of the data.
Revised: This motivates an alternative objective by which we train a student to capture signiﬁcantly more information in the teacher’s representation of the data.

Original: We formulate this objective as contrastive learning.
Revised: We formulate this objective as contrastive learning.

Original: Experiments demonstrate that our resulting new objective outperforms knowledge distillation and other cutting-edge distillers on a variety of knowledge transfer tasks, including single model compression, ensemble distillation, and cross-modal transfer.
Revised: Experiments demonstrate that our resulting new objective outperforms knowledge distillation and other cutting-edge distillers on a variety of knowledge transfer tasks, including single model compression, ensemble distillation, and cross-modal transfer.

Original: Our method sets a new state-of-the-art in many transfer tasks, and sometimes even outperforms the teacher network when combined with knowledge distillation.
Revised: Our method sets a new state-of-the-art in many transfer tasks, and sometimes even outperforms the teacher network when combined with knowledge distillation.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Knowledge distillation (KD) transfers knowledge from one deep learning model (the teacher) to another (the student).
Revised: Knowledge distillation (KD) transfers knowledge from one deep learning model (the teacher) to another (the student).

Original: The objective originally proposed by Hinton et al.
Revised: The objective originally proposed by Hinton et al.

Original: (2015) minimizes the KL divergence between the teacher and student outputs.
Revised: (2015) minimizes the KL divergence between the teacher and student outputs.

Original: This formulation makes intuitive sense when the output is a distribution, e.g., a probability mass function over classes.
Revised: This formulation makes intuitive sense when the output is a distribution, e.g., a probability mass function over classes.

Original: However, often we instead wish to transfer knowledge about a representation .
Revised: However, often we instead wish to transfer knowledge about a representation .

Original: For example, in the problem of “cross-modal distillation", we may wish to transfer the representation of an image processing network to a sound (Aytar et al., 2016) or to depth (Gupta et al., 2016) processing network, such that deep features for an image and the associated sound or depth features are highly correlated.
Revised: For example, in the problem of “cross-modal distillation", we may wish to transfer the representation of an image processing network to a sound (Aytar et al., 2016) or to depth (Gupta et al., 2016) processing network, such that deep features for an image and the associated sound or depth features are highly correlated.

Original: In such cases, the KL divergence is undeﬁned.
Revised: In such cases, the KL divergence is undeﬁned.

Original: Representational knowledge is structured – the dimensions exhibit complex interdependencies.
Revised: Representational knowledge is structured – the dimensions exhibit complex interdependencies.

Original: The original KD objective introduced in Hinton et al. (2015) treats all dimensions as independent, conditioned on the input.
Revised: The original KD objective introduced in (Hinton et al., 2015) treats all dimensions as independent, conditioned on the input.

Original: Let y T be the output of the teacher and y S be the output of the student.
Revised: Let y T be the output of the teacher and y S be the output of the student.

Original: Then the original KD objective function, ψ , has the fully factored form: ψ ( y S , y T ) =
Revised: Then the original KD objective function, ψ , has the fully factored form: ψ ( y S , y T ) =

Original: (cid:80) i φ i ( y Si , y Ti ) ∗ .
Revised: (cid:80) i φ i ( y Si , y Ti ) ∗ .

Original: Such a factored objective is insufﬁcient for transferring structural knowledge, i.e. dependencies between output dimensions i and j .
Revised: Such a factored objective is insufﬁcient for transferring structural knowledge, i.e. dependencies between output dimensions i and j .

Original: This is similar to the situation in image generation where an Lobjective produces blurry results, due to independence assumptions between output dimensions.
Revised: This is similar to the situation in image generation where an Lobjective produces blurry results, due to independence assumptions between output dimensions.

Original: To overcome this problem, we would like an objective that captures correlations and higher order output dependencies.
Revised: To overcome this problem, we would like an objective that captures correlations and higher order output dependencies.

Original: To achieve this, in this paper we leverage the family of contrastive objectives (Gutmann & Hyvärinen, 2010; Oord et al., 2018; Arora et al., 2019; Hjelm et al., 2018).
Revised: To achieve this, in this paper we leverage the family of contrastive objectives (Gutmann & Hyvärinen, 2010; Oord et al., 2018; Arora et al., 2019; Hjelm et al., 2018).

Original: These objective functions have been used successfully in recent years for density estimation and representation learning, especially in self-supervised settings.
Revised: These objective functions have been used successfully in recent years for density estimation and representation learning, especially in self-supervised settings.

Original: Here we adapt them to the task of knowledge distillation from one deep network to another.
Revised: Here we adapt them to the task of knowledge distillation from one deep network to another.

Original: We show that it is important to work in representation space, similar to recent works such as Zagoruyko & Komodakis (2016a); Romero et al.
Revised: We show that it is important to work in representation space, similar to recent works such as Zagoruyko & Komodakis (2016a); Romero et al.

Original: However, note that the loss functions used in those works do not explicitly try to capture correlations or higher-order dependencies in representational space.
Revised: However, note that the loss functions used in those works do not explicitly try to capture correlations or higher-order dependencies in representational space.

Original: Our objective maximizes a lower-bound to the mutual information between the teacher and student representations.

Original: We ﬁnd that this results in better performance on several knowledge transfer tasks.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: /QVePCji1Z/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4dua3H1FpHst7M0nQj+hQ8pAzaqzUeOqXK27VnYOsEi8nFchR75e/eoOYpRFKwwTVuuu5ifEzqgxnAqelXqoxoWxMh9i1VNIItZ/ND52SM6sMSBgrW9KQufp7IqOR1pMosJ0RNSO97M3E/7xuasJrP+MySQ1KtlgUpoKYmMy+JgOukBkxsYQyxe2thI2ooszYbEo2BG/55VXSuqh6btVrXFZqN3kcRTiBUzgHD66gBndQhyYwQHiGV3hzHpwX5935WLQWnHzmGP7A+fwB5jmM/A==</latexit> 
Revised: /QVePCji1Z/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4dua3H1FpHst7M0nQj+hQ8pAzaqzUeOqXK27VnYOsEi8nFchR75e/eoOYpRFKwwTVuuu5ifEzqgxnAqelXqoxoWxMh9i1VNIItZ/ND52SM6sMSBgrW9KQufp7IqOR1pMosJ0RNSO97M3E/7xuasJrP+MySQ1KtlgUpoKYmMy+JgOukBkxsYQyxe2thI2ooszYbEo2BG/55VXSuqh6btVrXFZqN3kcRTiBUzgHD66gBndQhyYwQHiGV3hzHpwX5935WLQWnHzmGP7A+fwB5jmM/A==</latexit> Our objective maximizes a lower-bound to the mutual information between the teacher and student representations. We ﬁnd that this results in better performance on several knowledge transfer tasks.

Original: We conjecture that this is because the contrastive objective better transfers all the information in the teacher’s representation, rather than only transferring knowledge about conditionally independent output class probabilities.
Revised: We conjecture that this is because the contrastive objective better transfers all the information in the teacher’s representation, rather than only transferring knowledge about conditionally independent output class probabilities.

Original: Somewhat surprisingly, the contrastive objective even improves results on the originally proposed task of distilling knowledge about class probabilities, for example, compressing a large CIFAR100 network into a smaller one that performs almost as well.
Revised: Somewhat surprisingly, the contrastive objective even improves results on the originally proposed task of distilling knowledge about class probabilities, for example, compressing a large CIFAR100 network into a smaller one that performs almost as well.

Original: We believe this is because the correlations between different class probabilities contains useful information that regularizes the learning problem.
Revised: We believe this is because the correlations between different class probabilities contains useful information that regularizes the learning problem.

Original: Our paper forges a connection between two literatures that have evolved mostly independently: knowledge distillation and representation learning .
Revised: Our paper forges a connection between two literatures that have evolved mostly independently: knowledge distillation and representation learning .

Original: This connection allows us to leverage strong methods from representation learning to signiﬁcantly improve the SOTA on knowledge distillation.
Revised: This connection allows us to leverage strong methods from representation learning to signiﬁcantly improve the SOTA on knowledge distillation.

Original: Our contributions are:
Revised: Our contributions are:

Original: A contrastive-based objective for transferring knowledge between deep networks.
Revised: A contrastive-based objective for transferring knowledge between deep networks.

Original: Applications to model compression, cross-modal transfer, and ensemble distillation.
Revised: Applications to model compression, cross-modal transfer, and ensemble distillation.

Original: Benchmarking 12 recent distillation methods; CRD outperforms all other methods, e
Revised: Benchmarking 12 recent distillation methods; CRD outperforms all other methods, e

Original: g., 57% average relative improvement over the original KD (Hinton et al., 2015) † , which, surprisingly, performs the second best.
Revised: g., 57% average relative improvement over the original KD (Hinton et al., 2015) † , which, surprisingly, performs the second best.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: The seminal work of Bucilu ˇ a et al.
Revised: The seminal work of Buciluˇa et al.

Original: (2006) and Hinton et al.
Revised: (2006) and Hinton et al.

Original: (2015) introduced the idea of knowledge distillation between large, cumbersome models into smaller, faster models without losing too much generalization power.
Revised: (2015) introduced the idea of knowledge distillation between large, cumbersome models into smaller, faster models without losing too much generalization power.

Original: The general motivation was that at training time, the availability of computation allows “slop" in model size, and potentially faster learning.
Revised: The general motivation was that at training time, the availability of computation allows “slop" in model size, and potentially faster learning.

Original: But computation and memory constraints at inference time necessitate the use of smaller models.
Revised: But computation and memory constraints at inference time necessitate the use of smaller models.

Original: Buciluˇa et al.
Revised: Buciluˇa et al.

Original: (2006) achieve this by matching output logits; Hinton et al.
Revised: (2006) achieve this by matching output logits; Hinton et al.

Original: (2015) introduced the idea of temperature in the softmax outputs to better represent smaller probabilities in the output of a single sample.
Revised: (2015) introduced the idea of temperature in the softmax outputs to better represent smaller probabilities in the output of a single sample.

Original: These smaller probabilities provide useful information about the learned representation of the teacher model; some tradeoff between large temperatures (which increase entropy) or small temperatures tend to provide the highest transfer of knowledge between student and teacher.
Revised: These smaller probabilities provide useful information about the learned representation of the teacher model; some tradeoff between large temperatures (which increase entropy) or small temperatures tend to provide the highest transfer of knowledge between student and teacher.

Original: The method in Li et al. (2014) was also closely related to Hinton et al.
Revised: The method in (Li et al., 2014) was also closely related to (Hinton et al., 2015).

Original: Attention transfer (Zagoruyko & Komodakis, 2016a) focuses on the features maps of the network as opposed to the output logits.
Revised: Attention transfer (Zagoruyko & Komodakis, 2016a) focuses on the features maps of the network as opposed to the output logits.

Original: Here the idea is to elicit similar response patterns in the teacher and student feature maps (called “attention").
Revised: Here the idea is to elicit similar response patterns in the teacher and student feature maps (called “attention").

Original: However, only feature maps with the same spatial  resolution can be combined in this approach, which is a signiﬁcant limitation since it requires student and teacher networks with very similar architectures.
Revised: However, only feature maps with the same spatial [Table] resolution can be combined in this approach, which is a signiﬁcant limitation since it requires student and teacher networks with very similar architectures.

Original: This technique achieves state of the art results[Table] for distillation (as measured by the generalization of the student network).
Revised: This technique achieves state of the art results for distillation (as measured by the generalization of the student network).

Original: FitNets (Romero et al., 2014) also deal with intermediate representations by using regressions to guide the feature activations of the student network.
Revised: FitNets (Romero et al., 2014) also deal with intermediate representations by using regressions to guide the feature activations of the student network.

Original: Since Zagoruyko & Komodakis (2016a) do a weighted form of this regression, they tend to perform better.
Revised: Since Zagoruyko & Komodakis (2016a) do a weighted form of this regression, they tend to perform better.

Original: Other papers (Yim et al., 2017; Huang & Wang, 2017; Kim et al., 2018;
Revised: Other papers (Yim et al., 2017; Huang & Wang, 2017; Kim et al., 2018;

Original: Yim et al., 2017; Huang & Wang, 2017; Ahn et al.,  2019) have enforced various criteria based on representations.
Revised: Yim et al., 2017; Huang & Wang, 2017; Ahn et al., 2019; Koratana et al., 2019) have enforced various criteria based on representations.

Original: The contrastive objective we use in this paper is the same as that used in Tian et al.
Revised: The contrastive objective we use in this paper is the same as that used in CMC (Tian et al., 2019).

Original: But we derive from a different perspective and give a rigorous proof that our objective is a lower bound on mutual information.
Revised: But we derive from a different perspective and give a rigorous proof that our objective is a lower bound on mutual information.

Original: Our objective is also related to the InfoNCE and NCE objectives introduced in (Oord et al., 2018; Gutmann & Hyvärinen, 2010).
Revised: Our objective is also related to the InfoNCE and NCE objectives introduced in (Oord et al., 2018; Gutmann & Hyvärinen, 2010).

Original: (Oord et al., 2018) use contrastive learning in the context of self-supervised representations learning.
Revised: Oord et al. (2018) use contrastive learning in the context of self-supervised representations learning.

Original: They show that their objective maximizes a lower bound on mutual information.
Revised: They show that their objective maximizes a lower bound on mutual information.

Original: A very related approach is used in Hjelm et al.
Revised: A very related approach is used in (Hjelm et al., 2018).

Original: InfoNCE and NCE are closely related but distinct from adversarial learning (Goodfellow et al., 2014).
Revised: InfoNCE and NCE are closely related but distinct from adversarial learning (Goodfellow et al., 2014).

Original: In Goodfellow (2014), it is shown that the NCE objective of Gutmann & Hyvärinen (2010) can lead to maximum likelihood learning, but not the adversarial objective.
Revised: In (Goodfellow, 2014), it is shown that the NCE objective of Gutmann & Hyvärinen (2010) can lead to maximum likelihood learning, but not the adversarial objective.

Original:  M ETHOD
Revised:  M ETHOD

Original: The key idea of contrastive learning is very general: learn a representation that is close in some metric space for “positive” pairs and push apart the representation between “negative” pairs.
Revised: The key idea of contrastive learning is very general: learn a representation that is close in some metric space for “positive” pairs and push apart the representation between “negative” pairs.

Original: Fig.
Revised: Fig.

Original: 1 gives a visual explanation for how we structure contrastive learning for the three tasks we consider: model compression, cross-modal transfer and ensemble distillation
Revised: 1 gives a visual explanation for how we structure contrastive learning for the three tasks we consider: model compression, cross-modal transfer and ensemble distillation

Original: . 3. C ONTRASTIVE L OSS
Revised: . 3. C ONTRASTIVE L OSS

Original: Given two deep neural networks, a teacher f T and a student f S .
Revised: Given two deep neural networks, a teacher f T and a student f S .

Original: Let x be the network input; we denote representations at the penultimate layer (before logits) as f T ( x ) and f S ( x ) respectively.
Revised: Let x be the network input; we denote representations at the penultimate layer (before logits) as f T ( x ) and f S ( x ) respectively.

Original: Let x i represent a training sample, and x j another randomly chosen sample.
Revised: Let x i represent a training sample, and x j another randomly chosen sample.

Original: We would like to push closer the representations f S ( x i ) and f T ( x i ) while pushing apart f S ( x i ) and f T ( x j ) .
Revised: We would like to push closer the representations f S ( x i ) and f T ( x i ) while pushing apart f S ( x i ) and f T ( x j ) .

Original: For ease of notation, we deﬁne random variables S and T for the student and teacher’s representations of the data respectively:
Revised: For ease of notation, we deﬁne random variables S and T for the student and teacher’s representations of the data respectively:

Original: [Equation]
Revised: [Equation]

Original: Intuitively speaking, we will consider the joint distribution p ( S, T ) and the product of marginal distributions p ( S ) p ( T ) , so that, by maximizing KL divergence between these distributions, we can maximize the mutual information between student and teacher representations.
Revised: Intuitively speaking, we will consider the joint distribution p ( S, T ) and the product of marginal distributions p ( S ) p ( T ) , so that, by maximizing KL divergence between these distributions, we can maximize the mutual information between student and teacher representations.

Original: To setup an appropriate loss that can achieve this aim, let us deﬁne a distribution q with latent variable C which decides whether a tuple ( f T ( x i ) , f S ( x j )) was drawn from the joint ( C = 1 ) or product of marginals ( [Equation] = 0 ):
Revised: To setup an appropriate loss that can achieve this aim, let us deﬁne a distribution q with latent variable C which decides whether a tuple ( f T ( x i ) , f S ( x j )) was drawn from the joint ( C = 1 ) or product of marginals ( C = 0 ):

Original: [Equation]
Revised: [Equation]

Original: Now, suppose in our data, we are given 1 congruent pair (drawn from the joint distribution, i.e. the same input provided to T and S ) for every N incongruent pairs (drawn from the product of marginals; independent randomly drawn inputs provided to T and S ).
Revised: Now, suppose in our data, we are given 1 congruent pair (drawn from the joint distribution, i.e. the same input provided to T and S ) for every N incongruent pairs (drawn from the product of marginals; independent randomly drawn inputs provided to T and S ).

Original: Then the priors on the latent C are:
Revised: Then the priors on the latent C are:

Original: [Equation]
Revised: [Equation]

Original: By simple manipulation and Bayes’ rule, the posterior for class C = 1 is given by:[Equation] Next
Revised: By simple manipulation and Bayes’ rule, the posterior for class C = 1 is given by:

Original: [Equation]
Revised: [Equation]

Original:  we observe a connection to mutual information as follows:
Revised: Next, we observe a connection to mutual information as follows:

Original: [Equation]
Revised: [Equation]

Original: Then taking expectation on both sides w.r.t.
Revised: Then taking expectation on both sides w.r.t.

Original: p ( T, S ) (equivalently w.r.t.
Revised: p ( T, S ) (equivalently w.r.t.

Original: q ( T, S | C = 1) ) and rearranging, gives us:
Revised: q ( T, S | C = 1) ) and rearranging, gives us:

Original: [Equation]
Revised: [Equation]

Original: where I ( T ; S ) is the mutual information between the distributions of the teacher and student embeddings.
Revised: where I ( T ; S ) is the mutual information between the distributions of the teacher and student embeddings.

Original: Thus maximizing E q ( T,S | C =1) log q ( C = 1 | T, S ) w.r.t.
Revised: Thus maximizing E q ( T,S | C =1) log q ( C = 1 | T, S ) w.r.t.

Original: the parameters of the student network
Revised: the parameters of the student network

Original: S increases a lower bound on mutual information.
Revised: S increases a lower bound on mutual information.

Original: However, we do not know the true distribution q ( C = 1 | T, S ) ; instead we estimate it by ﬁtting a model h : {T , S} → [0 , 1] to samples from the data distribution q ( C =| T, S ) , where T and S represent the domains of the embeddings.
Revised: However, we do not know the true distribution q ( C = 1 | T, S ) ; instead we estimate it by ﬁtting a model h : {T , S} → [0 , 1] to samples from the data distribution q ( C =| T, S ) , where T and S represent the domains of the embeddings.

Original: We maximize the log likelihood of the data under this model (a binary classiﬁcation problem):
Revised: We maximize the log likelihood of the data under this model (a binary classiﬁcation problem):

Original: [Equation]
Revised: [Equation]

Original: We term h the critic since we will be learning representations that optimize the critic’s score.
Revised: We term h the critic since we will be learning representations that optimize the critic’s score.

Original: Assuming sufﬁciently expressive h , h ∗ ( T, S ) =
Revised: Assuming sufﬁciently expressive h , h [Equation]

Original: q ( C = 1 | T, S ) (via Gibbs’ inequality; see Sec. 6.2.
Revised: Gibbs’ inequality; see Sec. 6.2.

Original: for proof), so we can rewrite Eq.
Revised: for proof), so we can rewrite Eq.

Original: 9 in terms of h ∗ :
Revised: 9 in terms of h ∗ :

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Therefore, we see that the optimal critic is an estimator whose expectation lower-bounds mutual information.
Revised: Therefore, we see that the optimal critic is an estimator whose expectation lower-bounds mutual information.

Original: We wish to learn a student that maximizes the mutual information between its representation and the teacher’s, suggesting the following optimization problem:
Revised: We wish to learn a student that maximizes the mutual information between its representation and the teacher’s, suggesting the following optimization problem:

Original: [Equation]
Revised: [Equation]

Original: An apparent difﬁculty here is that the optimal critic h ∗ depends on the current student.
Revised: An apparent difﬁculty here is that the optimal critic h ∗ depends on the current student.

Original: We can circumvent this difﬁculty by weakening the bound in (12) to: 
Revised: We can circumvent this difﬁculty by weakening the bound in (12) to: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst line comes about by simply adding N E q ( T,S | C =0)
Revised: The ﬁrst line comes about by simply adding N E q ( T,S | C =0)

Original: [log(1 − h ∗ ( T, S ))] to the bound in (12).
Revised: [log(1 − h ∗ ( T, S ))] to the bound in (12).

Original: This term is strictly negative, so the inequality holds.
Revised: This term is strictly negative, so the inequality holds.

Original: The last line follows from the fact that L critic ( h ∗ ) upper-bounds L critic ( h ) .
Revised: The last line follows from the fact that L critic ( h ∗ ) upper-bounds L critic ( h ) .

Original: Optimizing (15) w.r.t.
Revised: Optimizing (15) w.r.t.

Original: the student we have:
Revised: the student we have:

Original: [Equation]
Revised: [Equation]

Original: our ﬁnal learning problem
Revised: our ﬁnal learning problem

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: which demonstrates that we may jointly optimize f S at the same time as we learn h .
Revised: which demonstrates that we may jointly optimize f S at the same time as we learn h .

Original: We note that due to (16), f S ∗ = arg max f S L critic ( h ) , for any h , also is a representation that optimizes a lower-bound (a weaker one) on mutual information, so our formulation does not rely on h being optimized perfectly.
Revised: We note that due to (16), f S ∗ = arg max f S L critic ( h ) , for any h , also is a representation that optimizes a lower-bound (a weaker one) on mutual information, so our formulation does not rely on h being optimized perfectly.

Original: We may choose to represent h with any family of functions that satisfy h : {T , S} → [0 , 1] .
Revised: We may choose to represent h with any family of functions that satisfy h : {T , S} → [0 , 1] .

Original: In practice, we use the following:
Revised: In practice, we use the following:

Original: [Equation]
Revised: [Equation]

Original: where M is the cardinality of the dataset and τ is a temperature that adjusts the concentration level.
Revised: where M is the cardinality of the dataset and τ is a temperature that adjusts the concentration level.

Original: In practice, since the dimensionality of S and T may be different, g S and g T linearly transform them into the same dimension and further normalize them by L -2 norm before the inner product.
Revised: In practice, since the dimensionality of S and T may be different, g S and g T linearly transform them into the same dimension and further normalize them by L -2 norm before the inner product.

Original: The form of Eq.
Revised: The form of Eq.

Original: (18) is inspired by NCE (Gutmann & Hyvärinen, 2010).
Revised: (18) is inspired by NCE (Gutmann & Hyvärinen, 2010; Wu et al., 2018).

Original: Our formulation is similar to the InfoNCE loss (Oord et al., 2018) in that we maximize a lower bound on the mutual information.
Revised: Our formulation is similar to the InfoNCE loss (Oord et al., 2018) in that we maximize a lower bound on the mutual information.

Original: However we use a different objective and bound, which in our experiments we found to be more effective than InfoNCE.
Revised: However we use a different objective and bound, which in our experiments we found to be more effective than InfoNCE.

Original: Implementation.
Revised: Implementation.

Original: Theoretically, larger N in Eq.
Revised: Theoretically, larger N in Eq.

Original: 16 leads to tighter lower bound on MI.
Revised: 16 leads to tighter lower bound on MI.

Original: In practice, to avoid using very large batch size, we  implement a memory buffer that stores latent features of each data sample computed from previous batches.
Revised: In practice, to avoid using very large batch size, we follow Wu et al. and implement a memory buffer that stores latent features of each data sample computed from previous batches.

Original: Therefore, during training we can efﬁciently retrieve a large number of negative samples from the memory buffer without recomputing their features.
Revised: Therefore, during training we can efﬁciently retrieve a large number of negative samples from the memory buffer.

Original: 3. K NOWLEDGE D ISTILLATION O BJECTIVE
Revised: 3. K NOWLEDGE D ISTILLATION O BJECTIVE

Original: The knowledge distillation loss was proposed in Hinton et al.
Revised: The knowledge distillation loss was proposed in Hinton et al.

Original: In addition to the regular cross-entropy loss between the student output y S and one-hot label y , it asks the student network output to be as similar as possible to the teacher output by minimizing the cross-entropy between their output probabilities.
Revised: In addition to the regular cross-entropy loss between the student output y S and one-hot label y , it asks the student network output to be as similar as possible to the teacher output by minimizing the cross-entropy between their output probabilities.

Original: The complete objective is:
Revised: The complete objective is:

Original: [Equation]
Revised: [Equation]

Original:  where ρ is the temperature, α is a balancing weight, and σ is softmax function.
Revised: (20) where ρ is the temperature, α is a balancing weight, and σ is softmax function.

Original: H ( σ ( z T /ρ ) , σ ( z S /ρ )) is further decomposed into KL ( σ ( z T /ρ ) | σ ( z S /ρ )) and a constant entropy H ( σ ( z T /ρ )) .
Revised: H ( σ ( z T /ρ ) , σ ( z S /ρ )) is further decomposed into KL ( σ ( z T /ρ ) | σ ( z S /ρ )) and a constant entropy H ( σ ( z T /ρ )) .

Original: 3. C ROSS -M ODAL T RANSFER L OSS
Revised: 3. C ROSS -M ODAL T RANSFER L OSS

Original: In the cross-modal transfer task shown in Fig. 1(b), a teacher network is trained on a source modality
Revised: In the cross-modal transfer task shown in Fig. 1(b), a teacher network is trained on a source modality

Original: X with large-scale labeled dataset.
Revised: X with large-scale labeled dataset.

Original: We then wish to transfer the knowledge to a student network, but adapt it to another dataset or modality Y .
Revised: We then wish to transfer the knowledge to a student network, but adapt it to another dataset or modality Y .

Original: But the features of the teacher network are still valuable to help with learning of the student on another domain.
Revised: But the features of the teacher network are still valuable to help with learning of the student on another domain.

Original: In this transfer task, we use the contrastive loss Eq.
Revised: In this transfer task, we use the contrastive loss Eq.

Original: 10 to match the features of the student and teacher.
Revised: 10 to match the features of the student and teacher.

Original: Additionally, we also consider other distillation objectives, such as KD discussed in previous section, Attention Transfer Zagoruyko &
Revised: Additionally, we also consider other distillation objectives, such as KD discussed in previous section, Attention Transfer Zagoruyko &

Original: Komodakis (2016a) and FitNet Romero et al.
Revised: Komodakis (2016a) and FitNet Romero et al.

Original: Such transfer is conducted on a paired but unlabeled dataset D = { ( x i , y i ) | i = 1 , ...,
Revised: Such transfer is conducted on a paired but unlabeled dataset D = { ( x i , y i ) | i = 1 , ...,

Original: L, x i ∈ X , y i ∈ Y} .
Revised: L, x i ∈ X , y i ∈ Y} .

Original: In this scenario, there is no true label y of such data for the original training task on the source modality, and therefore we ignore the
Revised: In this scenario, there is no true label y of such data for the original training task on the source modality, and therefore we ignore the

Original: H ( y, y S ) term in all objectives that we test.
Revised: H ( y, y S ) term in all objectives that we test.

Original: Prior cross-modal work Aytar et al.
Revised: Prior cross-modal work Aytar et al.

Original: ; Hoffman et al.
Revised: ; Hoffman et al.

Original: (2016b;a) uses either L 2 regression or KL-divergence.
Revised: (2016b;a) uses either L 2 regression or KL-divergence.

Original: 3. E NSEMBLE D ISTILLATION L OSS
Revised: 3. E NSEMBLE D ISTILLATION L OSS

Original: In the case of ensemble distillation shown in 1(c), we have M > 1 teacher networks, f T i and one student network f S .
Revised: In the case of ensemble distillation shown in 1(c), we have M > 1 teacher networks, f T i and one student network f S .

Original: We adopt the contrastive framework by deﬁning multiple pair-wise contrastive losses between features of each teacher network f T i and the student network f S .
Revised: We adopt the contrastive framework by deﬁning multiple pair-wise contrastive losses between features of each teacher network f T i and the student network f S .

Original: These losses are summed together to give the ﬁnal loss (to be minimized):
Revised: These losses are summed together to give the ﬁnal loss (to be minimized):

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate our contrastive representation distillation (CRD) framework in three knowledge distillation tasks: (a) model compression of a large network to a smaller one; (b) cross-modal knowledge transfer; (c) ensemble distillation from a group of teachers to a single student network.
Revised: We evaluate our contrastive representation distillation (CRD) framework in three knowledge distillation tasks: (a) model compression of a large network to a smaller one; (b) cross-modal knowledge transfer; (c) ensemble distillation from a group of teachers to a single student network.

Original: Datasets (1) CIFAR-100 (Krizhevsky & Hinton, 2009) contains 50K training images with 0.5K images per class and 10K test images.
Revised: Datasets (1) CIFAR-100 (Krizhevsky & Hinton, 2009) contains 50K training images with 0.5K images per class and 10K test images.

Original: (2) ImageNet (Deng et al., 2009) provides 1.2 million images from 1K classes for training and 50K for validation.
Revised: (2) ImageNet (Deng et al., 2009) provides 1.2 million images from 1K classes for training and 50K for validation.

Original: (3) STL-10 (Coates et al., 2011) consists of a training set of 5K labeled images from 10 classes and 100K unlabeled images, and a test set of 8K images.
Revised: (3) STL-10 (Coates et al., 2011) consists of a training set of 5K labeled images from 10 classes and 100K unlabeled images, and a test set of 8K images.

Original: (4) TinyImageNet
Revised: (4) TinyImageNet

Original: (Deng et al., 2009) has 200 classes, each with 500 training images andvalidaton images.
Revised: (Deng et al., 2009) has 200 classes, each with 500 training images andvalidaton images.

Original: (5) NYU-Depth V2 (Silberman et al., 2012) consists of 1449 indoor images, each labeled with dense depth image and semantic map. 
Revised: (5) NYU-Depth V2 (Silberman et al., 2012) consists of 1449 indoor images, each labeled with dense depth image and semantic map. [Table] [Table]

Original: 4. M ODEL C OMPRESSION
Revised: 4. M ODEL C OMPRESSION

Original: Setup We experiment on CIFAR-100 and ImageNet with student-teacher combinations of various capacity, such as ResNet (He et al., 2016) or Wide ResNet (WRN) (Zagoruyko & Komodakis, 2016b).
Revised: Setup We experiment on CIFAR-100 and ImageNet with student-teacher combinations of various capacity, such as ResNet (He et al., 2016) or Wide ResNet (WRN) (Zagoruyko & Komodakis, 2016b).

Original: Results on CIFAR100 Table 1 and Table 2 compare top-1 accuracies of different distillation objectives (for details, see Section 6.1).
Revised: Results on CIFAR100 Table 1 and Table 2 compare top-1 accuracies of different distillation objectives (for details, see Section 6.1).

Original: Table 1 investigates students and teachers of the same architectural style, while Table 2 focuses on students and teachers from different architectures.
Revised: Table 1 investigates students and teachers of the same architectural style, while Table 2 focuses on students and teachers from different architectures.

Original: We observe that our  loss, which we call CRD (Contrastive Representation Distillation), consistently outperforms all other distillation objectives, including the original KD (an average relative improvement of 57%).
Revised: We observe that our [Figure] [Table] loss, which we call CRD (Contrastive Representation Distillation), consistently outperforms all other distillation objectives, including the original KD (an average relative improvement of 57%).

Original: Surprisingly, we found that KD works pretty well and none of the other methods consistently outperforms
Revised: Surprisingly, we found that KD works pretty well and none of the other methods consistently outperforms

Original: KD on their own.
Revised: KD on their own.

Original: Another observation is that, while switching the teacher student combinations from same to different architectural styles, methods that distill intermediate representations tend[Table] [Table] to perform worse than methods that distill from the last several layers.
Revised: Another observation is that, while switching the teacher student combinations from same to different architectural styles, methods that distill intermediate representations tend to perform worse than methods that distill from the last several layers.

Original: For example, the Attention
Revised: For example, the Attention

Original: Transfer (AT) and FitNet methods even underperform the vanilla student.
Revised: Transfer (AT) and FitNet methods even underperform the vanilla student.

Original: In contrast, PKT, SP and
Revised: In contrast, PKT, SP and

Original: CRD that operate on last several layers performs well.
Revised: CRD that operate on last several layers performs well.

Original: This might because, architectures of different styles have their own solution paths mapping from the input to the output [Table] and enforcing the mimic of intermediate representations thus might conﬂict with such inductive bias.[Figure]
Revised: This might because, architectures of different styles have their own solution paths mapping from the input to the output, and enforcing the mimic of intermediate representations thus might conﬂict with such inductive bias.

Original: [Table]

Original: Capturing inter-class correlations.
Revised: Capturing inter-class correlations.

Original: In Fig.
Revised: In Fig.

Original: 2, we compute the difference between the correlation matrices of the teacher’s and student’s logits; for three different students: vanilla student without distillation, trained by AT, KD or CRD (our method).
Revised: 2, we compute the difference between the correlation matrices of the teacher’s and student’s logits; for three different students: vanilla student without distillation, trained by AT, KD or CRD (our method).

Original: It is clear that the CRD objective captures the most correlation structure in the logit as shown by the smaller differences between teacher and student.
Revised: It is clear that the CRD objective captures the most correlation structure in the logit as shown by the smaller differences between teacher and student.

Original: This is reﬂected in reduced error rates.
Revised: This is reﬂected in reduced error rates.

Original: Results on ImageNet For a fair comparison with Zagoruyko & Komodakis (2016a) and Lan et al.
Revised: Results on ImageNet For a fair comparison with Zagoruyko & Komodakis (2016a) and Lan et al.

Original: (2018), we adopt the models from these papers, ResNet-34 as the teacher and ResNet-18 as the student.
Revised: (2018), we adopt the models from these papers, ResNet-34 as the teacher and ResNet-18 as the student.

Original: As shown in Table 3, the gap of top-1 accuracy between the teacher and student is 3 .
Revised: As shown in Table 3, the gap of top-1 accuracy between the teacher and student is 3 .

Original: The AT method reduces this gap by 0 .
Revised: The AT method reduces this gap by 0 .

Original: 95% , while ours narrow it by 1 .
Revised: 95% , while ours narrow it by 1 .

Original: 42% , a 50% relative improvement.
Revised: 42% , a 50% relative improvement.

Original: Results on ImageNet validates the scalability of our CRD.
Revised: Results on ImageNet validates the scalability of our CRD.

Original: Transferability of representations We are interested in representations , and a primary goal of representation learning is to acquire general knowledge, that is, knowledge that transfers to tasks or datasets that were unseen during training.
Revised: Transferability of representations We are interested in representations , and a primary goal of representation learning is to acquire general knowledge, that is, knowledge that transfers to tasks or datasets that were unseen during training.

Original: Therefore, we test if the representations we distill transfer well.
Revised: Therefore, we test if the representations we distill transfer well.

Original: A WRN-16-2 student either distills from a WRN-40-2 teacher, or is trained from scratch on
Revised: A WRN-16-2 student either distills from a WRN-40-2 teacher, or is trained from scratch on

Original: CIFAR100.
Revised: CIFAR100.

Original: The student serves as a frozen representation extractor (the layer prior to the logit) for images from STL-10 or TinyImageNet (all images downsampled to 32x32).
Revised: The student serves as a frozen representation extractor (the layer prior to the logit) for images from STL-10 or TinyImageNet (all images downsampled to 32x32).

Original: We then train a linear classiﬁer to perform 10-way (for STL-10) or 200-way (for TinyImageNet) classiﬁcation to quantify the transferability of the representations.
Revised: We then train a linear classiﬁer to perform 10-way (for STL-10) or 200-way (for TinyImageNet) classiﬁcation to quantify the transferability of the representations.

Original: We compare CRD with multiple baselines such as KD and
Revised: We compare CRD with multiple baselines such as KD and

Revised: [Table]

Revised: [Figure]

Original: AT in Table 4.
Revised: AT in Table 4.

Original: In this setting, all distillation methods except FitNet improve transferability of the learned representations on both STL-10 and TinyImageNet.
Revised: In this setting, all distillation methods except FitNet improve transferability of the learned representations on both STL-10 and TinyImageNet.

Original: While the teacher performs the best on the original CIFAR100 dataset, its representations transfer the worst to the other two datasets.
Revised: While the teacher performs the best on the original CIFAR100 dataset, its representations transfer the worst to the other two datasets.

Original: This is perhaps the teacher’s representations are biased towards the original task.
Revised: This is perhaps the teacher’s representations are biased towards the original task.

Original: Surprisingly, the student with CRD+KD distillation not only matches its teacher on CIFAR100 (see Table 4), but also transfers much better than the teacher, e.g., 3.6 % improvement (STL-10) and 4.1 % on TinyImageNet.
Revised: Surprisingly, the student with CRD+KD distillation not only matches its teacher on CIFAR100 (see Table 4), but also transfers much better than the teacher, e.g., 3.6 % improvement (STL-10) and 4.1 % on TinyImageNet.

Original: 4. C ROSS - MODAL TRANSFER
Revised: 4. C ROSS - MODAL TRANSFER

Original: We consider a practical setting where modality X has large amount of labeled data while modality Y does not.
Revised: We consider a practical setting where modality X has large amount of labeled data while modality Y does not.

Original: Transfering knowledge from X to Y is a common challenge.
Revised: Transfering knowledge from X to Y is a common challenge.

Original: For example, while large-scale
Revised: For example, while large-scale

Original: [Table] [Figure] RGB datasets are easily accessible, other modalities such as depth images are much harder to label at scale but have wide applications.
Revised: RGB datasets are easily accessible, other modalities such as depth images are much harder to label at scale but have wide applications.

Original: We demonstrate the potential of CRD for cross-modal transfer in two scenarios: (a) transfer from luminance to chrominance; (b) transfer from RGB to depth images.
Revised: We demonstrate the potential of CRD for cross-modal transfer in two scenarios: (a) transfer from luminance to chrominance; (b) transfer from RGB to depth images.

Original: Transferring from Luminance to Chrominace.
Revised: Transferring from Luminance to Chrominace.

Original: We work on Lab color space, where L represents
Revised: We work on Lab color space, where L represents

Original: Luminance and ab Chrominance.
Revised: Luminance and ab Chrominance.

Original: We ﬁrst train an L network on TinyImageNet with supervision.
Revised: We ﬁrst train an L network on TinyImageNet with supervision.

Original: Then we transfer knowledge from this L network to ab network on the unlabeled set of STL-10 with different objectives, including FitNet, KD, KD+AT and CRD.
Revised: Then we transfer knowledge from this L network to ab network on the unlabeled set of STL-10 with different objectives, including FitNet, KD, KD+AT and CRD.

Original: For convenience, we use the same architecture for student and teacher (they can also be different).
Revised: For convenience, we use the same architecture for student and teacher (they can also be different).

Original: Finally, we evaluate the knowlege of ab network by two means: (1) linear probing : we freeze the ab network and train a linear classiﬁer on top of features from different layers to perform 10-way classiﬁcation on STL-10 ab images.
Revised: Finally, we evaluate the knowlege of ab network by two means: (1) linear probing : we freeze the ab network and train a linear classiﬁer on top of features from different layers to perform 10-way classiﬁcation on STL-10 ab images.

Original: This is a common practice (Alain & Bengio, 2016; Zhang et al., 2017) to evaluate the quality of network representations; (b) fully ﬁnetuning : we fully ﬁnetune the ab network to obtain the best accuracy.
Revised: This is a common practice (Alain & Bengio, 2016; Zhang et al., 2017) to evaluate the quality of network representations; (b) fully ﬁnetuning : we fully ﬁnetune the ab network to obtain the best accuracy.

Original: We also use as a baseline the ab network that is randomly initialized rather than distilled.
Revised: We also use as a baseline the ab network that is randomly initialized rather than distilled.

Original: Architectures investigated include AlexNet, ResNet-18 and ResNet-50.
Revised: Architectures investigated include AlexNet, ResNet-18 and ResNet-50.

Original: The results shown in Figure 3 show that
Revised: The results shown in Figure 3 show that

Original: CRD is more efﬁcient for transferring inter-modal knowledge than other methods.
Revised: CRD is more efﬁcient for transferring inter-modal knowledge than other methods.

Original: Besides, we also note KD+AT does not improve upon KD, possibly because attention of luminance and chrominance are different and harder to transfer.
Revised: Besides, we also note KD+AT does not improve upon KD, possibly because attention of luminance and chrominance are different and harder to transfer.

Original: Transferring from RGB to Depth.
Revised: Transferring from RGB to Depth.

Original: We transfer the knowledge of a ResNet-18 teacher pretrained on ImageNet to a 5-layer student CNN operating on depth images.
Revised: We transfer the knowledge of a ResNet-18 teacher pretrained on ImageNet to a 5-layer student CNN operating on depth images.

Original: We follow a similar transferring procedure on NYU-Depth training set, except that we use a trick of contrasting between local and global features, proposed by Hjelm et al.
Revised: We follow a similar transferring procedure on NYU-Depth training set, except that we use a trick of contrasting between local and global features, proposed by Hjelm et al.

Original: (2018), to overcome the problem of insufﬁcient data samples in the depth domain.
Revised: (2018), to overcome the problem of insufﬁcient data samples in the depth domain.

Original: Then the student network is further trained to predict semantic segmentation  maps from depth images.
Revised: Then the student network is further trained to predict semantic segmentation [Table] [Figure] maps from depth images.

Original: We note that both knowledge transfer and downstream training are conducted on the same set of images, i.e., the training set.
Revised: We note that both knowledge transfer and downstream training are conducted on the same set of images, i.e., the training set.

Original: Table 5 reports the average pixel prediction accuracy and mean Intersection-over-Union across all classes.
Revised: Table 5 reports the average pixel prediction accuracy and mean Intersection-over-Union across all classes.

Original: All distillation methods can transfer knowledge from the RGB ResNet to the depth CNN.
Revised: All distillation methods can transfer knowledge from the RGB ResNet to the depth CNN.

Original: FitNet surpasses KD and KD+AT.
Revised: FitNet surpasses KD and KD+AT.

Original: CRD signiﬁcantly outperforms all other methods.
Revised: CRD signiﬁcantly outperforms all other methods.

Original: 4. D ISTILLATION FROM AN ENSEMBLE
Revised: 4. D ISTILLATION FROM AN ENSEMBLE

Original: Better classiﬁcation performance is often achieved by ensembles of deep networks, but these are usually too expensive for inference time, and distillation into a single network is a desirable task.
Revised: Better classiﬁcation performance is often achieved by ensembles of deep networks, but these are usually too expensive for inference time, and distillation into a single network is a desirable task.

Original: We investigate the KL-divergence based KD and our CRD Metric (%) Random Init. KD
Revised: We investigate the KL-divergence based KD and our CRD for this task, using the loss of Sec. 21.

Original: KD+AT

Original: FitNet

Original: CRD

Original: Pix.

Original: Acc.

Original: [Table]

Original: [Table] mIoU 35. [Figure] The network structures of each teacher and student are identical here, but an ensemble of multiple teachers can still provide rich knowledge for the student.
Revised: The network structures of each teacher and student are identical here, but an ensemble of multiple teachers can still provide rich knowledge for the student.

Original: To compare between KD and CRD on
Revised: To compare between KD and CRD on

Original: CIFAR100 dataset, we use WRN-16-2 and ResNet-20, whose single model error rates are 26 .
Revised: CIFAR100 dataset, we use WRN-16-2 and ResNet-20, whose single model error rates are 26 .

Original: 7% and 30 .
Revised: 7% and 30 .

Original: 9% respectively.
Revised: 9% respectively.

Original: The results of distillation are presented in Figure 4, where we vary the number of ensembled teachers.
Revised: The results of distillation are presented in Figure 4, where we vary the number of ensembled teachers.

Original: CRD with 8 teachers decreases the error rate of WRN-16-2 to 23 .
Revised: CRD with 8 teachers decreases the error rate of WRN-16-2 to 23 .

Original: and
Revised: and

Original: ResNet20 to 28 .
Revised: ResNet20 to 28 .

Original: In addition, CRD works consistently better than KD in all settings we test.
Revised: In addition, CRD works consistently better than KD in all settings we test.

Original: These observations suggest that CRD is capable of distilling an ensemble of models into a single one which performs signiﬁcantly better than a model of the same size that is trained from scratch. 
Revised: These observations suggest that CRD is capable of distilling an ensemble of models into a single one which performs signiﬁcantly better than a model of the same size that is trained from scratch. 4. A BLATIVE S TUDY

Revised: InfoNCE v.s. Ours.

Original: 4. H YPER - PARAMETERS AND COMPUTATION OVERHEAD
Revised: InfoNCE (Oord et al., 2018) is an alternative contrastive objective which select a single positive out from a set of distractors via a softmax function. We compare InfoNCE with our contrastive objective Eq 18, when using the same number of negatives. The last two rows in Tableshow that our objective outperforms InfoNCE in 4 out of 5 teacher-student combinations. Negative Sampling.

Revised: In this paper, we consider two negative sampling plocies when giving an anchor x i : (1) x jj (cid:54)

Revised: =

Revised: i for the unsupervised case when we have no labels, or (2) x jy j (cid:54) =

Revised: y i for supervised case, where y i represents the label associated with sample x i .

Revised: One might imagine that the ﬁrst sampling strategy will increase the intra-class variance as we may push apart positives and negatives from the same underlying class, while the second would not.

Revised: We quantitatively measure and report the gap between these two strategies in Table 6, which shows that the classiﬁcation accuracy by the second sampling strategy is 0 .

Revised: 81% higher for our objective and 0 .

Revised: 62% higher for InfoNCE than the ﬁrst one.

Original:  There are two main hyper-parameters in our contrastive objectives: (1) the number of negative samples N in Eq.
Revised: [Table] [Figure] H YPER - PARAMETERS AND COMPUTATION OVERHEAD There are two main hyper-parameters in our contrastive objectives: (1) the number of negative samples N in Eq.

Original: 18, and (2) temperature τ which modulates the softmax probability.
Revised: 18, and (2) temperature τ which modulates the softmax probability.

Original: We adopt
Revised: We adopt

Original: WRN-40-2 as teacher and WRN-16-2 as student for parameter analysis.
Revised: WRN-40-2 as teacher and WRN-16-2 as student for parameter analysis.

Original: Experiments are conducted on CIFAR100 and the results are shown in Figure 5.
Revised: Experiments are conducted on CIFAR100 and the results are shown in Figure 5.

Original: Number of negatives N We have validated different N : 16 , 64 , 256 , 1024 , 4096 , 16384 .
Revised: Number of negatives N We have validated different N : 16 , 64 , 256 , 1024 , 4096 , 16384 .

Original: As shown in Figure 5(a), increasing N leads to improved performance.
Revised: As shown in Figure 5(a), increasing N leads to improved performance.

Original: However, the difference of error rate between N = 4096 and N = 16384 is less than 0 .
Revised: However, the difference of error rate between N = 4096 and N = 16384 is less than 0 .

Original: Therefore, we use N = 16384 for reporting the accuracy while in practice N = 4096 should sufﬁce.
Revised: Therefore, we use N = 16384 for reporting the accuracy while in practice N = 4096 should sufﬁce.

Original: Temperature τ We varied τ between 0 .
Revised: Temperature τ We varied τ between 0 .

Original: 02 and 0 .
Revised: 02 and 0 .

Original: As Figure 5(b) illustrates, both extremely high or low temperature lead to a sub-optimal solution.
Revised: As Figure 5(b) illustrates, both extremely high or low temperature lead to a sub-optimal solution.

Original: In general, temperatures between 0 . 05 and 0 .
Revised: In general, temperatures between 0 . 05 and 0 .

Original: 2 work well on CIFAR100.
Revised: 2 work well on CIFAR100.

Original: All experiments but those on ImageNet use a temperature of 0 . 1 .
Revised: All experiments but those on ImageNet use a temperature of 0 . 1 .

Original: For ImageNet, we use τ = 0 .
Revised: For ImageNet, we use τ = 0 .

Original: The optimal temperature may vary across different datasets and require further tuning.
Revised: The optimal temperature may vary across different datasets and require further tuning.

Original: Computational Cost We use ResNet-18 on ImageNet for illustration.
Revised: Computational Cost We use ResNet-18 on ImageNet for illustration.

Original: CRD uses extra 260 MFLOPs, which is about 12% of the original 2 GFLOPs.
Revised: CRD uses extra 260 MFLOPs, which is about 12% of the original 2 GFLOPs.

Original: In practice, we did not notice signiﬁcant difference of training time on ImageNet (e.g., 1.75 epochs/hour v.s. 1.67 epochs/hour on two Titan-V GPUs).
Revised: In practice, we did not notice signiﬁcant difference of training time on ImageNet (e.g., 1.75 epochs/hour v.s. 1.67 epochs/hour on two Titan-V GPUs).

Original: The memory bank for storing all 128 -d features of ImageNet only costs around 600 MB memory, and therefore we store it on GPU memory.
Revised: The memory bank for storing all 128 -d features of ImageNet only costs around 600 MB memory, and therefore we store it on GPU memory.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have developed a novel technique for neural network distillation, using the concept of contrastive objectives, which are usually used for representation learning.
Revised: We have developed a novel technique for neural network distillation, using the concept of contrastive objectives, which are usually used for representation learning.

Original: We experimented with our objective on[Figure] a number of applications such as model compression, cross-modal transfer and ensemble distillation, outperforming other distillation objectives by signiﬁcant margins in all these tasks.
Revised: We experimented with our objective on a number of applications such as model compression, cross-modal transfer and ensemble distillation, outperforming other distillation objectives by signiﬁcant margins in all these tasks.

Original: Our contrastive objective is the only distillation objective that consistently outperforms knowledge distillation across a wide variety of knowledge transfer tasks.
Revised: Our contrastive objective is the only distillation objective that consistently outperforms knowledge distillation across a wide variety of knowledge transfer tasks.

Original: Prior objectives only surpass KD when combined with
Revised: Prior objectives only surpass KD when combined with

Original: KD.
Revised: KD.

Original: Contrastive learning is a simple and effective objective with practical beneﬁts
Revised: Contrastive learning is a simple and effective objective with practical beneﬁts.

Revised: Acknowledgments.

Revised: We thank Baoyun Peng for providing the code of CC (Peng et al., 2019) and

Revised: Frederick Tung for verifying our reimplementation of SP (Tung & Mori, 2019).

Revised: This research was supported in part by Google Cloud and iFlytek.

Original:  A PPENDIX 6. O THER M ETHODS
Revised:  A PPENDIX 6. O THER M ETHODS

Original: We compare to the following other state-of-the-art methods from the literature:
Revised: We compare to the following other state-of-the-art methods from the literature:

Original: Knowledge Distillation (KD) (Hinton et al., 2015) 2.
Revised: Knowledge Distillation (KD) (Hinton et al., 2015) 2.

Original: Fitnets:
Revised: Fitnets:

Original: Hints for thin deep nets (Romero et al., 2014) 3.
Revised: Hints for thin deep nets (Romero et al., 2014) 3.

Original: Attention Transfer (AT) (Zagoruyko & Komodakis, 2016a) 4.
Revised: Attention Transfer (AT) (Zagoruyko & Komodakis, 2016a) 4.

Original: Similarity-Preserving Knowledge Distillation (SP) (Tung & Mori, 2019); 5.
Revised: Similarity-Preserving Knowledge Distillation (SP) (Tung & Mori, 2019); 5.

Original: Correlation Congruence (CC) (Peng et al., 2019) 6.
Revised: Correlation Congruence (CC) (Peng et al., 2019) 6.

Original: Variational information distillation for knowledge transfer (VID) (Ahn et al., 2019) 7.
Revised: Variational information distillation for knowledge transfer (VID) (Ahn et al., 2019) 7.

Original: Relational Knowledge Distillation (RKD) (Park et al., 2019) 8.
Revised: Relational Knowledge Distillation (RKD) (Park et al., 2019) 8.

Original: Learning deep representations with probabilistic knowledge transfer (PKT) (Passalis & Tefas, 2018) 9.
Revised: Learning deep representations with probabilistic knowledge transfer (PKT) (Passalis & Tefas, 2018) 9.

Original: Knowledge transfer via distillation of activation boundaries formed by hidden neurons (AB) (Heo et al., 2019) 
Revised: Knowledge transfer via distillation of activation boundaries formed by hidden neurons (AB) (Heo et al., 2019) 10.

Original: Paraphrasing complex network: Network compression via factor  (FT) (Kim et al., 2018) 11.
Revised: Paraphrasing complex network: Network compression via factor transfer (FT) (Kim et al., 2018) 11.

Original: A gift from knowledge distillation: Fast optimization, network minimization and transfer learning (FSP) (Yim et al., 2017) 12.
Revised: A gift from knowledge distillation: Fast optimization, network minimization and transfer learning (FSP) (Yim et al., 2017) 12.

Original: Like what you like: Knowledge distill via neuron selectivity transfer (NST) (Huang & Wang, 2017)
Revised: Like what you like: Knowledge distill via neuron selectivity transfer (NST) (Huang & Wang, 2017)

Original: 6. C ONTRASTIVE LOSS – DETAILS 6.2. 
Revised: 6. C ONTRASTIVE LOSS – DETAILS 6.2. P ROOF THAT h ∗ ( T, S ) = q ( C = 1 | T, S )

Original: [Equation]

Original: We wish to model some true distribution q ( C | T
Revised: We wish to model some true distribution q ( C | T

Original: = t, S = s ) .
Revised: = t, S = s ) .

Original: C is a binary variable, so we can model q ( C | T
Revised: C is a binary variable, so we can model q ( C | T

Original: = t, S = s ) as a Bernoulli distribution with a single parameter h
Revised: = t, S = s ) as a Bernoulli distribution with a single parameter h

Original: ( S = s, T = t ) ∈
Revised: ( S = s, T = t ) ∈

Original: [0 , 1] , deﬁning for convenience h (cid:48)
Revised: [0 , 1] , deﬁning for convenience h (cid:48)

Original: ( C = 1 , S = s, T = t )
Revised: ( C = 1 , S = s, T = t )

Original: =
Revised: =

Original: h ( S = s, T = t ) and h (cid:48)
Revised: h ( S = s, T = t ) and h (cid:48)

Original: ( C = 0 , S = s, T = t )
Revised: ( C = 0 , S = s, T = t )

Original: = 1 − h ( S = s, T = t ) .
Revised: = 1 − h ( S = s, T = t ) .

Original: The log likelihood function is:
Revised: The log likelihood function is:

Original: [Equation]
Revised: [Equation]

Original: By Gibbs’ inequality, the max likelihood ﬁt is 
Revised: By Gibbs’ inequality, the max likelihood ﬁt is h

Original: [Equation]
Revised: [Equation]

Original: q
Revised: C = c, S = s,

Original: [Equation]
Revised: [Equation]

Original:  C = c | S= s
Revised: = q ( C = c | S

Original: [Equation]

Original:  T = t ) , which also implies that h ( S = s, T = t ) =
Revised: = s, T = t ) , which also implies that h ( S = s, T = t ) =

Original: q ( C = 1 | S = s, T = t ) .
Revised: q ( C = 1 | S = s, T = t ) .

Original: We now demonstrate that our objective in Eq.
Revised: We now demonstrate that our objective in Eq.

Original: (10) is proportional to a summation over terms Eq.
Revised: (10) is proportional to a summation over terms Eq.

Original: (22) for all s ∈ S and t ∈ T .
Revised: (22) for all s ∈ S and t ∈ T .

Original: [Equation]
Revised: [Equation]

Original: Notice that (26) is proportional to Eq.
Revised: Notice that (26) is proportional to Eq.

Original: (10) from the main paper.
Revised: (10) from the main paper.

Original: For sufﬁciently expressive h , then, each term inside the expectation in Eq.
Revised: For sufﬁciently expressive h , then, each term inside the expectation in Eq.

Original: (23) can be maximized, resulting in h ∗ ( T = t, S = s )
Revised: (23) can be maximized, resulting in h ∗ ( T = t, S = s )

Original: =
Revised: =

Original: q ( C = 1 | T = t, S = t ) for all s and t .
Revised: q ( C = 1 | T = t, S = t ) for all s and t .

Original: 6. N ETWORK A RCHITECTURES
Revised: 6. N ETWORK A RCHITECTURES

Original: Wide Residual Network (WRN) (Zagoruyko & Komodakis, 2016b).
Revised: Wide Residual Network (WRN) (Zagoruyko & Komodakis, 2016b).

Original: WRN-d-w represnets wide resnet with depth d and width factor w . resnet (He et al., 2016).
Revised: WRN-d-w represnets wide resnet with depth d and width factor w . resnet (He et al., 2016).

Original: We use resnet-d to represent cifar -style resnet with 3 groups of basic blocks, each with 16 , 32 , and 64 channels respectively.
Revised: We use resnet-d to represent cifar -style resnet with 3 groups of basic blocks, each with 16 , 32 , and 64 channels respectively.

Original: In our experiments, resnet8 x4 and resnet32 x indicate a 4 times wider network (namely, with 64 , 128 , and 256 channels for each of the block)
Revised: In our experiments, resnet8 x4 and resnet32 x indicate a 4 times wider network (namely, with 64 , 128 , and 256 channels for each of the block)

Original: ResNet (He et al., 2016).
Revised: ResNet (He et al., 2016).

Original: ResNet-d represents ImageNet -style ResNet with Bottleneck blocks and more channels.
Revised: ResNet-d represents ImageNet -style ResNet with Bottleneck blocks and more channels.

Original: MobileNetV2 Sandler et al.
Revised: MobileNetV2 Sandler et al.

Original: In our experiments, we use a width multiplier of 0 .
Revised: In our experiments, we use a width multiplier of 0 .

Original: vgg (Simonyan & Zisserman, 2014).
Revised: vgg (Simonyan & Zisserman, 2014).

Original: the vgg net used in our experiments are adapted from its original
Revised: the vgg net used in our experiments are adapted from its original

Original: ImageNet counterpart.
Revised: ImageNet counterpart.

Original: ShufﬂeNetV1 (Zhang et al., 2018a), ShufﬂeNetV2 (Tan et al., 2019).
Revised: ShufﬂeNetV1 (Zhang et al., 2018a), ShufﬂeNetV2 (Tan et al., 2019).

Original: ShufﬂeNets are proposed for efﬁcient training and we adapt them to input of size 32x32.
Revised: ShufﬂeNets are proposed for efﬁcient training and we adapt them to input of size 32x32.

Original: 6. I MPLEMENTATION D ETAILS
Revised: 6. I MPLEMENTATION D ETAILS

Original: All methods evaluated in our experiments use SGD.
Revised: All methods evaluated in our experiments use SGD.

Original: For CIFAR-100, we initialize the learning rate as 0.05, and decay it by 0.1 every 30 epochs after the ﬁrst 150 epochs until the last 240 epoch.
Revised: For CIFAR-100, we initialize the learning rate as 0.05, and decay it by 0.1 every 30 epochs after the ﬁrst 150 epochs until the last 240 epoch.

Original: For MobileNetV2, ShufﬂeNetV1 and ShufﬂeNetV2, we use a learning rate of 0.01 as this learning rate is optimal for these models in a grid search, while 0.05 is optimal for other models.
Revised: For MobileNetV2, ShufﬂeNetV1 and ShufﬂeNetV2, we use a learning rate of 0.01 as this learning rate is optimal for these models in a grid search, while 0.05 is optimal for other models.

Original: For ImageNet, we follow the standard PyTorch practice but train for 10 more epochs.
Revised: For ImageNet, we follow the standard PyTorch practice but train for 10 more epochs.

Original: Batch size is 64 for CIFAR-100 or 256 for ImageNet.
Revised: Batch size is 64 for CIFAR-100 or 256 for ImageNet.

Original: The student is trained by a combination of cross-entropy classiﬁcation objective and a knowledge distillation objective, shown as follows:
Revised: The student is trained by a combination of cross-entropy classiﬁcation objective and a knowledge distillation objective, shown as follows:

Original: [Equation]
Revised: [Equation]

Original:  For the weight balance factor β , we directly use the optimal value from the original paper if it is speciﬁed, or do a grid search with teacher WRN-40-2 and student WRN-16-2.
Revised: [Equation] For the weight balance factor β , we directly use the optimal value from the original paper if it is speciﬁed, or do a grid search with teacher WRN-40-2 and student WRN-16-2.

Original: This results in the following list of β used for different objectives:
Revised: This results in the following list of β used for different objectives:

Original: Fitnets (Romero et al., 2014): β =2.
Revised: Fitnets (Romero et al., 2014): β =2.

Original: AT (Zagoruyko & Komodakis, 2016a): β =3.
Revised: AT (Zagoruyko & Komodakis, 2016a): β =3.

Original: SP (Tung & Mori, 2019): β =4.
Revised: SP (Tung & Mori, 2019): β =4.

Original: CC (Peng et al., 2019): β = 0 .5.
Revised: CC (Peng et al., 2019): β = 0 .5.

Original: VID (Ahn et al., 2019): β =6.
Revised: VID (Ahn et al., 2019): β =6.

Original: RKD (Park et al., 2019): β 1 = 25 for distance and β 2 = 50 for angle.
Revised: RKD (Park et al., 2019): β 1 = 25 for distance and β 2 = 50 for angle.

Original: For this loss, we combine both term following the original paper.
Revised: For this loss, we combine both term following the original paper.

Original: PKT (Passalis & Tefas, 2018): β =8.
Revised: PKT (Passalis & Tefas, 2018): β =8.

Original: AB (Heo et al., 2019): β = 0 , distillation happens in a separate pre-training stage where only distillation objective applies.
Revised: AB (Heo et al., 2019): β = 0 , distillation happens in a separate pre-training stage where only distillation objective applies.

Original: FT (Kim et al., 2018): β =10.
Revised: FT (Kim et al., 2018): β =10.

Original: FSP (Yim et al., 2017): β = 0 , distillation happens in a separate pre-training stage where only distillation objective applies.
Revised: FSP (Yim et al., 2017): β = 0 , distillation happens in a separate pre-training stage where only distillation objective applies.

Original: NST (Huang & Wang, 2017): β =12.
Revised: NST (Huang & Wang, 2017): β =12.

Original: CRD: β = 0 .
Revised: CRD: β = 0 .

Original: 8 , in general β ∈ [0 . 5 , 1 . 5] works reasonably well.
Revised: 8 , in general β ∈ [0 . 5 , 1 . 5] works reasonably well.

Original: For KD(Hinton et al., 2015), we follow Eq.
Revised: For KD(Hinton et al., 2015) we follow Eq.

Original: and set α = 0 .
Revised: and set α = 0 .

Original: 9 and T = 4 .
Revised: 9 and T = 4 .

Original: 6. V ISUALIZATION OF THE CORRELATION DISCREPANCY
Revised: 6. V ISUALIZATION OF THE CORRELATION DISCREPANCY

Original: We visualize the correlation discrepancy for different distillation objectives across various combinations of student and teacher networks.
Revised: We visualize the correlation discrepancy for different distillation objectives across various combinations of student and teacher networks.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 6, our contrastive distillation objective siginiﬁcantly outperforms other objectives, in terms of minimizing the correlation discrepancy between student and teacher networks.
Revised: 6, our contrastive distillation objective siginiﬁcantly outperforms other objectives, in terms of minimizing the correlation discrepancy between student and teacher networks.

Original: The normalized correlation coefﬁcients are computed at the logit layer.
Revised: The normalized correlation coefﬁcients are computed at the logit layer.

Original: 6. C OMBINING DIFFERENT DISTILLATION OBJECTIVES
Revised: 6. C OMBINING DIFFERENT DISTILLATION OBJECTIVES

Original: Table 6 shows the effect of combining various objectives with KD.
Revised: Table 7 shows the effect of combining various objectives with KD.

Original: Most methods is able to slightly outperform KD after this combination.
Revised: Most methods is able to slightly outperform KD after this combination.

Original: We also the check the compatibility of our distillation objective with KD and PKT.
Revised: We also the check the compatibility of our distillation objective with KD and PKT.

Original: The combination of CRD and KD/PKT further improves over single
Revised: The combination of CRD and KD/PKT further improves over single

Original: CRD objective.
Revised: CRD objective.

Original: 6. A DDITIONAL RESULTS ON T RANSFERABILITY OF REPRESENTATIONS
Revised: 6. A DDITIONAL RESULTS ON T RANSFERABILITY OF REPRESENTATIONS

Original: In addtion to Table 4 shown in the main text, we also evaluate the transferabilit of the representations of various other distillation methods and summarize the results in Table 7.
Revised: In addtion to Table 4 shown in the main text, we also evaluate the transferabilit of the representations of various other distillation methods and summarize the results in Table 8.

Original: In general, CRD achieves the best transferring accuracy in 7 out of 10 settings.
Revised: In general, CRD achieves the best transferring accuracy in 7 out of 10 settings.

Original: [Figure]

Original: 6. D EEP M UTUAL L EARNING S ETTING
Revised: 6. D EEP M UTUAL L EARNING S ETTING

Original: In Zhang et al. (2018b) a Deep Mutual Learning setting was proposed where the teacher and student networks are trained simultaneously rather than sequentially.
Revised: In (Zhang et al., 2018b), a Deep Mutual Learning setting was proposed where the teacher and student networks are trained simultaneously rather than sequentially.

Original: The beneﬁt is that not only the student networks but also the teacher models will be improved.
Revised: The beneﬁt is that not only the student networks but also the teacher models will be improved.

Original: Here we investigate the possibility of incorporating different distillation objectives into this training framework.
Revised: Here we investigate the possibility of incorporating different distillation objectives into this training framework.

Original: The results are summarizied in
Revised: The results are summarizied in

Original: Table 8.
Revised: Table 9.

Original: We observe that in general logit based distillation methods is better than non-logit based distillation methods in this setting, i e., KD performs the best.
Revised: We observe that in general logit based distillation methods is better than non-logit based distillation methods in this setting, i.e., KD performs the best.

Original: We conjecture that during the mutual training phase, KD performs like an advanced label smoothing regularization and does not require the logit to be very accurate.
Revised: We conjecture that during the mutual training phase, KD performs like an advanced label smoothing regularization and does not require the logit to be very accurate.

Original: But the feature-based distillation methods are hard to learn knowledge from the teacher model when it is handicapped.
Revised: But the feature-based distillation methods are hard to learn knowledge from the teacher model when it is handicapped.

Original: On the other hand, we notice that the combination of
Revised: On the other hand, we notice that the combination of

Original: KD and CRD leads to better performance on the student side, as shown in the last row of Table 8.
Revised: KD and CRD leads to better performance on the student side, as shown in the last row of Table 9. [Figure]

Original: 6. S TANDARD D EVIATION FOR R ESULTS ON CIFAR-100 B ENCHMARK
Revised: 6. S TANDARD D EVIATION FOR R ESULTS ON CIFAR-100 B ENCHMARK

Original: The standard deviation over multiple runs on CIFAR-100 benchmark is provided in Table 9 for student and teacher models of the same architectural type, and in Table 10 for student and teacher models of different architectural types.
Revised: The standard deviation over multiple runs on CIFAR-100 benchmark is provided in Table 10 for student and teacher models of the same architectural type, and in Table 11 for student and teacher models of different architectural types.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: KD 70.
Revised: KD 70.

Original: FitNet 70.
Revised: FitNet 70.

Original: AT 70.
Revised: AT 70.

Original: SP 71.
Revised: SP 71.

Original: CC 70.
Revised: CC 70.

Original: VID 71.
Revised: VID 71.

Original: RKD 71.
Revised: RKD 71.

Original: PKT 71.
Revised: PKT 71.

Original: AB 70.
Revised: AB 70.

Original: FT 71.
Revised: FT 71.

Original: FSP 69.
Revised: FSP 69.

Original: n/a n/a n/a n/a
Revised: n/a n/a n/a n/a

Original: NST 70.
Revised: NST 70.

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: Vanilla
Revised: [Table] [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: T
Revised: Vanilla

Original: KD

Original: [Table]
Revised: [Table]

Original: [Table][Table] [Table] AT
Revised: [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: SP

Original: [Table] [Table]
Revised: [Table] [Table]

Original: CC
Revised: S

Original: [Table]
Revised: [Table]

Original: CRD

Original: [Table]
Revised: [Table]

Original: CRD

Original: [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] Vanilla 73. S
Revised: [Table]

Original: KD
Revised: [Table] [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: AT 73.
Revised: [Table] [Table] [Table] [Table]

Original: SP 72.
Revised: [Table] [Table] [Table] [Table]

Original: CC 73.
Revised: [Table] [Table] [Table] [Table]

Original: CRD 75.
Revised: [Table] [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Standard deviation is provided.

Original: [Table]
Revised: [Table]