Original: Respecting Transfer Gap in Knowledge Distillation
Revised: Respecting Transfer Gap in Knowledge Distillation

Original: Abstract
Revised: Abstract

Original: Knowledge distillation (KD) is essentially a process of transferring a teacher model’s behavior, e.g. , network response, to a student model.
Revised: Knowledge distillation (KD) is essentially a process of transferring a teacher model’s behavior, e.g. , network response, to a student model.

Original: The network response serves as additional supervision to formulate the machine domain ( machine for short), which uses the data collected from the human domain ( human for short) as a transfer set.
Revised: The network response serves as additional supervision to formulate the machine domain ( machine for short), which uses the data collected from the human domain ( human for short) as a transfer set.

Original: Traditional KD methods hold an underlying assumption that the data collected in both human domain and machine domain are both independent and identically distributed (IID).
Revised: Traditional KD methods hold an underlying assumption that the data collected in both human domain and machine domain are both independent and identically distributed (IID).

Original: We point out that this naïve assumption is unrealistic and there is indeed a transfer gap between the two domains.
Revised: We point out that this naïve assumption is unrealistic and there is indeed a transfer gap between the two domains.

Original: Although the gap offers the student model external knowledge from the machine domain, the imbalanced teacher knowledge would make us incorrectly estimate how much to transfer from teacher to student per sample on the non-IID transfer set.
Revised: Although the gap offers the student model external knowledge from the machine domain, the imbalanced teacher knowledge would make us incorrectly estimate how much to transfer from teacher to student per sample on the non-IID transfer set.

Original: To tackle this challenge, we propose Inverse Probability Weighting Distillation (IPWD) that estimates the propensity score of a training sample belonging to the machine domain, and assigns its inverse amount to compensate for under-represented samples.
Revised: To tackle this challenge, we propose Inverse Probability Weighting Distillation (IPWD) that estimates the propensity score of a training sample belonging to the machine domain, and assigns its inverse amount to compensate for under-represented samples.

Original: Experiments on CIFAR-100 and ImageNet demonstrate the effectiveness of IPWD for both two-stage distillation and one-stage self-distillation.
Revised: Experiments on CIFAR-100 and ImageNet demonstrate the effectiveness of IPWD for both two-stage distillation and one-stage self-distillation.

Original:  Introduction
Revised:  Introduction

Original: Knowledge distillation (KD) [21] transfers knowledge from a teacher model, e g. , a big, cumbersome, and energy-inefficient network, to a student model, e.g. , a small, light and energy-efficient network, to improve the performance of the student model.
Revised: Knowledge distillation (KD) [20] transfers knowledge from a teacher model, e g. , a big, cumbersome, and energy-inefficient network, to a student model, e.g. , a small, light, and energy-efficient network, to improve the performance of the student model.

Original: A common intuition is that a teacher with better performance will teach a stronger student.
Revised: A common intuition is that a teacher with better performance will teach a stronger student.

Original: However, recent studies find that the teacher’s accuracy is not a good indicator of the resultant student performance [8].
Revised: However, recent studies find that the teacher’s accuracy is not a good indicator of the resultant student performance [8].

Original: For example, a poorly-trained teacher with early stopping can still teach a better student [8, 11, 77]; or, a teacher with a smaller model size than the student is also a good teacher [77]; or, a teacher with the same architecture as the student helps to improve the student—self-distillation [13, 82, 81, 27].
Revised: For example, a poorly-trained teacher with early stopping can still teach a better student [8, 11, 67]; or, a teacher with a smaller model size than the student is also a good teacher [67]; or, a teacher with the same architecture as the student helps to improve the student—self-distillation [13, 71, 70, 25].

Original: Should we view KD in a perspective of domain transfer [12, 63], we would better understand the above counter-intuitive findings.
Revised: Should we view KD in the perspective of domain transfer [12, 55], we would better understand the above counter-intuitive findings.

Original: From Figure 1, we can see that teacher predictions and ground-truth labels indeed behave differently.
Revised: From Figure 1, we can see that teacher predictions and ground-truth labels indeed behave differently.

Original: Although the teacher is trained on the balanced dataset, its predicted probability distribution over the dataset is imbalanced.
Revised: Although the teacher is trained on the balanced dataset, its predicted probability distribution over the dataset is imbalanced.

Original: Even on the same training set with the same model parameter, teachers with different temperature τ yield different “soft label” distributions from the ground-truth ones.
Revised: Even on the same training set with the same model parameter, teachers with different temperature τ yield different “soft label” distributions from the ground-truth ones.

Original: This implies that human and teacher knowledge are from different domains, and there is a transfer gap that drives the “dark knowledge” [21] transferring from teacher to student—regardless of “strong” or “weak” teachers, it is a valid transfer as long as there is a gap.
Revised: This implies that human and teacher knowledge is from different domains, and there is a transfer gap that drives the “dark knowledge” [20] transferring from teacher to student— regardless of “strong” or “weak” teachers, it is a valid transfer as long as there is a gap.

Original: However, the transfer gap affects the distillation performance on the under-represented classes, i.e. , classes on the tail of teacher predictions, which is overlooked in recent studies.
Revised: However, the transfer gap affects the distillation performance of the under-represented classes, i.e. , classes on the tail of teacher predictions, which is overlooked in recent studies.

Original: Take CIFAR-100 as an example.
Revised: Take CIFAR-100 as an example.

Original: We rank and divide the 100 classes into 4 groups according to the ranks of predicted
Revised: We rank and divide the 100 classes into four groups according to the ranks of predicted

Original: [Figure]
Revised: [Figure]

Original: probability.
Revised: probability.

Original: As shown in Table 1, compared to vanilla training, KD achieves better performance in all the subgroups.
Revised: As shown in Table 1, compared to vanilla training, KD achieves better performance in all the subgroups.

Original: However, the increase in the top 25 classes is much higher than that in the lastclasses, i [Table] e. , averagely 5.14% vs. 0.85%.
Revised: However, the increase in the top 25 classes is much higher than that in the last 25 classes, i.e. , averagely 5.14% vs. 0.85%.

Original: We ask: what causes the gap from the first place; or more specifically, why does the teacher’s non-uniform distributed predictions implies the gap?
Revised: We ask: what causes the gap in the first place; or more specifically, why does the teacher’s non-uniform distributed predictions implies the gap?

Original: We answer in an invariance vs. equivariance learning point of view [4, 69]:
Revised: We answer from an invariance vs. equivariance learning point of view [4, 60]:

Original: [Table]
Revised: [Table]

Original: Human domain: context invariance .
Revised: Human domain: context invariance .

Original: The discriminative generalization is the ability to learn both context-invariant and classequivariant information from the diverse training samples per class.
Revised: The discriminative generalization is the ability to learn both context-invariant and classequivariant information from the diverse training samples per class.

Original: The human domain only provides context-invariant class-specific information, i.e. , hard targets.
Revised: The human domain only provides context-invariant class-specific information, i.e. , hard targets.

Original: We normally collect a balanced dataset to formulate human domain.
Revised: We normally collect a balanced dataset to formulate human domain.

Original: Machine domain: context equivariance .
Revised: Machine domain: context equivariance .

Original: Teacher models often use a temperature variable to preserve the context.
Revised: Teacher models often use a temperature variable to preserve the context.

Original: The temperature allows the teacher to represent a sample not only by its contextinvariant class-specific information, but also its context-equivariant information.
Revised: The temperature allows the teacher to represent a sample not only by its contextinvariant class-specific information, but also its context-equivariant information.

Original: For example, a dog image with  soft label 0.8 · dog + 0.2 · wolf may imply that the dog has wolf-like contextual attributes such as “fluffy coat” and “upright ears”.
Revised: For example, a dog image with a soft label 0.8 · dog + 0.2 · wolf may imply that the dog has wolf-like contextual attributes such as “fluffy coat” and “upright ears”.

Original: Although the context-invariance ( i.e. , class) is balanced in the training data, the context-equivariance ( i.e. , context) is imbalanced because the context balance is not considered in class-specific data collection [67].
Revised: Although the context-invariance ( i.e. , class) is balanced in the training data, the context-equivariance ( i.e. , context) is imbalanced because the context balance is not considered in class-specific data collection [58].

Original: To construct the transfer set for the machine domain, the teacher model annotates each sample after seeing others, i.e. , being pre-trained on the whole set.
Revised: To construct the transfer set for the machine domain, the teacher model annotates each sample after seeing others, i.e. , being pre-trained on the whole set.

Original: Interestingly, the diverse context results in a long-tailed imbalanced distribution, which is exactly reflected in Figure 1.
Revised: Interestingly, the diverse context results in a long-tailed imbalanced distribution, which is exactly reflected in Figure 1.

Original: In other words, the teacher’s knowledge is imbalanced even though the teacher is trained on a class-balanced dataset.
Revised: In other words, the teacher’s knowledge is imbalanced even though the teacher is trained on a class-balanced dataset.

Original: Now we are ready to point out how the transfer gap is not properly addressed in conventional KD methods.
Revised: Now we are ready to point out how the transfer gap is not properly addressed in conventional KD methods.

Original: Conventional KD calculates the Cross-Entropy (CE) loss between the ground-truth label and  student’s prediction, and the Kullback–Leibler (KL) divergence [33] loss between the teacher’s and student’s predictions, where a constant weight is assigned for the two losses.
Revised: Conventional KD calculates the Cross-Entropy (CE) loss between the ground-truth label and the student’s prediction, and the Kullback–Leibler (KL) divergence [30] loss between the teacher’s and student’s predictions, where a constant weight is assigned for the two losses.

Original: This is essentially based on the underlying assumption that the data in both the human and machine domains are IID.
Revised: This is essentially based on the underlying assumption that the data in both the human and machine domains are IID.

Original: Based on the analysis of context equivariance, we argue that the assumption is unrealistic, i.e. , the teacher’s knowledge is imbalanced .
Revised: Based on the analysis of context equivariance, we argue that the assumption is unrealistic, i.e. , the teacher’s knowledge is imbalanced .

Original: Therefore, a constant sample weight for the KL loss would be a bottleneck.
Revised: Therefore, a constant sample weight for the KL loss would be a bottleneck.

Original: In this paper, we propose a simple yet effective method, Inverse ProbabilityWeighting
Revised: In this paper, we propose a simple yet effective method, Inverse Probability

Original:  Distillation (IPWD), which compensates for the training samples that are under-weighted in the machine domain.
Revised: Weighting Distillation (IPWD), which compensates for the training samples that are under-weighted in the machine domain.

Original: For each training sample x , we first estimate its machine-domain propensity score P ( x | machine ) by comparing class-aware and context-aware predictions.
Revised: For each training sample x , we first estimate its machine-domain propensity score P ( x | machine ) by comparing class-aware and context-aware predictions.

Original: A sample with a low propensity score would have a high confidence from class-aware predictions and a low confidence from context-aware predictions.
Revised: A sample with a low propensity score would have a high confidence from class-aware predictions and a low confidence from context-aware predictions.

Original: Then, IPWD assigns the inverse probability 1 /P ( x | machine ) as the sample weight for the KL loss to highlight the under-represented samples.
Revised: Then, IPWD assigns the inverse probability 1 /P ( x | machine ) as the sample weight for the KL loss to highlight the under-represented samples.

Original: In this way, IPWD generates a pseudo-population [37, 26] to deal withthe imbalanced knowledge.
Revised: In this way, IPWD generates a pseudo-population [34, 24] to deal with imbalanced knowledge.

Original: We evaluate our proposed IPWD on two typical knowledge distillation settings: two-stage teacherstudent distillation and one-stage self-distillation.
Revised: We evaluate our proposed IPWD on two typical knowledge distillation settings: two-stage teacherstudent distillation and one-stage self-distillation.

Original: Experiments conducted on CIFAR-100 [32] and
Revised: Experiments conducted on CIFAR-100 [29] and

Original: ImageNet [10] demonstrate the effectiveness and generality of our IPWD.
Revised: ImageNet [10] demonstrate the effectiveness and generality of our IPWD.

Original: Our contributions are three-fold:
Revised: Our contributions are three-fold:

Original: • We formulate KD as a domain transfer problem and argue that the naïve IID assumption on machine domain neglects the imbalanced knowledge due to  transfer gap.
Revised: • We formulate KD as a domain transfer problem and argue that the naïve IID assumption on machine domain neglects the imbalanced knowledge due to the transfer gap.

Original: • We propose Inverse Probability Weighting Distillation (IPWD) which compensate for the samples that are overlooked in the machine domain to tackle the imbalanced knowledge in transfer gap.
Revised: • We propose Inverse Probability Weighting Distillation (IPWD) which compensate for the samples that are overlooked in the machine domain to tackle the imbalanced knowledge in transfer gap.

Original: • Experiments on CIFAR-100 and ImageNet for both two-stage distillation and one-stage selfdistillation show that the proper handling of the transfer gap is a promising direction in KD.
Revised: • Experiments on CIFAR-100 and ImageNet for both two-stage distillation and one-stage selfdistillation show that the proper handling of the transfer gap is a promising direction in KD.

Original:  Related Work
Revised:  Related Work

Original: Knowledge distillation (KD) was first introduced to transferthe knowledge from an effective but cumbersome model to a smaller and  efficient model [21].
Revised: Knowledge distillation (KD) was first introduced to transfer knowledge from an effective but cumbersome model to a smaller and more efficient model [20].

Original: The knowledge can be formulated in either output space [21, 28, 35, 78, 77, 43, 61, 85, 31] or representation space [54, 25, 79, 30, 50, 19, 66, 7, 27].
Revised: The knowledge can be formulated in either output space [20, 26, 32, 68, 67, 38, 53, 73, 28] or representation space [48, 23, 69, 27, 44, 18, 57, 7, 25].

Original: KD has attracted a wide interest in theory, methodology, and applications [15].
Revised: KD has attracted a wide interest in theory, methodology, and applications [14].

Original: For applications, KD has shown its great potential in various areas, including but not limited to classification [36, 53, 39, 23], detection [34, 59, 70], segmentation [18, 44, 38] for visual recognition tasks, and visual question answering [46, 1, 48], video captioning [49, 84], and text-to-image synthesis [64] for vision-language tasks.
Revised: For applications, KD has shown its great potential in various areas, including but not limited to classification [33, 47, 36, 21], detection [31, 52, 61], segmentation [17, 39, 35] for visual recognition tasks, and visual question answering [41, 1, 42], video captioning [43, 72], and text-to-image synthesis [56] for vision-language tasks.

Original: Recent studies further discussed how and why KD works.
Revised: Recent studies further discussed how and why KD works.

Original: Specifically, Müller et al.
Revised: Specifically, Müller et al.

Original: [45] and Shen et al.
Revised: [40] and Shen et al.

Original: [58] empirically analyzed the effect of label smoothing on KD.
Revised: [51] empirically analyzed the effect of label smoothing on KD.

Original: Cho et al.
Revised: Cho et al.

Original: [8], Dong et al.
Revised: [8], Dong et al.

Original: [11], and Yuan et al.
Revised: [11], and Yuan et al.

Original: [77] pointed out that early stopping is a good regularization for a better teacher.
Revised: [67] pointed out that early stopping is a good regularization for a better teacher.

Original: Yuan et al.
Revised: Yuan et al.

Original: [77] further found that a poorly trained teacher, even a model smaller than the student, can improve the performance of the student.
Revised: [67] further found that a poorly trained teacher, even a model smaller than the student, can improve the performance of the student.

Original: Besides, Memon et al.
Revised: Besides, Memon et al.

Original: [41] and Zhou et al.
Revised: [37] and Zhou et al.

Original: [85] proposed a bias-variance trade-off perspective for KD.
Revised: [73] proposed a bias-variance trade-off perspective for KD.

Original: In this paper, we point out that existing KD methods hold an underlying assumption that the IID training samples are also IID in the machine domain, which overlooks the transfer gap.
Revised: In this paper, we point out that existing KD methods hold an underlying assumption that the IID training samples are also IID in the machine domain, which overlooks the transfer gap.

Original: Self-distillation is a special case of KD, which uses the student network itself as the teacher instead of the cumbersome model, i.e. , the teacher and student models have the same architecture [13, 82, 81, 27].
Revised: Self-distillation is a special case of KD, which uses the student network itself as the teacher instead of the cumbersome model, i.e. , the teacher and student models have the same architecture [13, 71, 70, 25].

Original: This process can be executed in iterations and produce a stronger ensemble model [13].
Revised: This process can be executed in iterations and produce a stronger ensemble model [13].

Original: Similar to
Revised: Similar to

Original: KD, traditional self-distillation follows a two-stage process: first pre-training a student model as the teacher, and then distilling the knowledge from the pre-trained model to a new student model.
Revised: KD, traditional self-distillation follows a two-stage process: first pre-training a student model as the teacher, and then distilling the knowledge from the pre-trained model to a new student model.

Original: In order to perform the teacher-student optimization in one generation, recent studies [75, 31] proposed one-stage self-distillation that adopts student models at earlier epochs as teacher models.
Revised: In order to perform the teacher-student optimization in one generation, recent studies [65, 28] proposed one-stage self-distillation that adopts student models at earlier epochs as teacher models.

Original: These one-stage self-distillation methods outperform vanilla students by large margins.
Revised: These one-stage self-distillation methods outperform vanilla students by large margins.

Original: In this paper, we also evaluate the effectiveness of our IPWD as a plug-in in one-stage self-distillation.
Revised: In this paper, we also evaluate the effectiveness of our IPWD as a plug-in in one-stage self-distillation.

Original: Inverse Probability Weighting (IPW)
Revised: Inverse Probability Weighting (IPW)

Original: [55, 37, 26, 5], also known as inverse probability of treatment weighting or inverse propensity weighting, was proposed to correct the selection bias when the observations are non-IID.
Revised: [49, 34, 24, 5], also known as inverse probability of treatment weighting or inverse propensity weighting, was proposed to correct the selection bias when the observations are non-IID.

Original: IPW uses the inverse of the probability ( i.e. , propensity score) that the individual would be assigned to the treatment group to reweight the samples.
Revised: IPW uses the inverse of the probability ( i.e. , propensity score) that the individual would be assigned to the treatment group to reweight the samples.

Original: Propensity-weighting techniques have been widely applied and studied in many areas [57], such as causal inference [26], complete-case analysis [37], machine learning [9, 6, 62], and recommendation systems [57, 72, 3].
Revised: Propensity-weighting techniques have been widely applied and studied in many areas [50], such as causal inference [24], complete-case analysis [34], machine learning [9, 6, 54], and recommendation systems [50, 62, 3].

Original: In this paper, we view the distillation process as a domain transfer problem and adopt IPW to dynamically assign the weight to each training sample for the distillation loss.
Revised: In this paper, we view the distillation process as a domain transfer problem and adopt IPW to dynamically assign the weight to each training sample for the distillation loss.

Original:  Analysis 3.1 Knowledge Distillation (KD)
Revised:  Analysis 3.1 Knowledge Distillation (KD)

Original: We view knowledge distillation from a perspective of domain transfer, and take the image classification task as the case study.
Revised: We view knowledge distillation from a perspective of domain transfer, and take the image classification task as the case study.

Original: Suppose that the training data D = {X , Y} = { ( x, y ) } contains x as the input ( e.g. , image) and y ∈ R C as its ground-truth annotation ( e.g. , one-hot label), where C denotes the number of classes.
Revised: Suppose that the training data D = {X , Y} = { ( x, y ) } contains x as the input ( e.g. , image) and y ∈ R C as its ground-truth annotation ( e.g. , one-hot label), where C denotes the number of classes.

Original: A standard solution to train the classifier θ uses the cross-entropy loss as the objective:
Revised: A standard solution to train the classifier θ uses the cross-entropy loss as the objective:

Original: [Equation]
Revised: [Equation]

Original: where ℓ cls ( x, y ) =
Revised: where ℓ cls ( x, y ) =

Original: H ( y s , y ) is the classification loss for sample x , H ( p, q ) =
Revised: H ( y s , y ) is the classification loss for sample x , H ( p, q ) =

Original: [Equation]
Revised: [Equation]

Original: Ci =1 − q i log p
Revised: Ci =1 − q i log p

Original: i denotes the cross entropy between p and q , y s = f
Revised: i denotes the cross entropy between p and q , y s = f

Original: ( x ; θ ) denotes the model’s output probability given x , i.e. , y sk =
Revised: ( x ; θ ) denotes the model’s output probability given x , i.e. , y sk =

Original: [Equation]
Revised: [Equation]

Original: z
Revised: z

Original: [Equation]
Revised: [Equation]

Original: Ci =
Revised: Ci =

Original: [Equation]
Revised: [Equation]

Original: exp ( z si ) , where z s is the output logits of the model.
Revised: exp ( z si ) , where z s is the output logits of the model.

Original: The hard targets provide context-invariant class-specific information from the human domain.
Revised: The hard targets provide context-invariant class-specific information from the human domain.

Original: An assumption held behind
Revised: An assumption held behind

Original: Eq.
Revised: Eq.

Original: (1) is that the samples are independent and identically distributed (IID) in the training and test set.
Revised: (1) is that the samples are independent and identically distributed (IID) in the training and test set.

Original: KD adopts a teacher model θ t to generate soft targets as extra supervisions, i.e. , context-equivariant information.
Revised: KD adopts a teacher model θ t to generate soft targets as extra supervisions, i.e. , context-equivariant information.

Original: To formulate the machine domain, traditional KD methods commonly use the training set
Revised: To formulate the machine domain, traditional KD methods commonly use the training set

Original: D to construct the transfer set D t using the same copy of X , i.e. , D t = { ( x, y t ) } where y t = f
Revised: D to construct the transfer set D t using the same copy of X , i.e. , D t = { ( x, y t ) } where y t = f

Original: ( x ; θ t ) and x ∈ X .
Revised: ( x ; θ t ) and x ∈ X .

Original: Traditional KD approaches use the KL divergence [33] loss for knowledge transfer:
Revised: Traditional KD approaches use the KL divergence [30] loss for knowledge transfer:

Original: [Equation]
Revised: [Equation]

Original: where ℓ dist ( x, y t ; θ ) = τ 2 · [ H ( y sτ , y tτ ) − H
Revised: where ℓ dist ( x, y t ; θ ) = τ 2 · [ H ( y sτ , y tτ ) − H

Original: ( y tτ , y tτ )] denotes the distillation loss for sample x .
Revised: ( y tτ , y tτ )] denotes the distillation loss for sample x .

Original: Normally, the outputs of the student and teacher are softened using a temperature τ , i.e. , y sτ,k = exp( z sk /τ )
Revised: Normally, the outputs of the student and teacher are softened using a temperature τ , i.e. , y sτ,k = exp( z sk /τ )

Original: (cid:80) Ci =1 exp ( z si /τ ) and y tτ,k = exp
Revised: (cid:80) Ci =1 exp ( z si /τ ) and y tτ,k = exp

Original: ( z tk /τ ) (cid:80)
Revised: ( z tk /τ ) (cid:80)

Original: Ci
Revised: Ci

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ti /τ ) .
Revised: ti /τ ) .

Original: The overall objective combines L cls and L dist as:
Revised: The overall objective combines L cls and L dist as:

Original: [Equation]
Revised: [Equation]

Original: where α and β are the hyper-parameters.
Revised: where α and β are the hyper-parameters.

Original: The underlying assumption of traditional KD behind Eq.
Revised: The underlying assumption of traditional KD behind Eq.

Original: (2) is that the transfer set D t is an unbiased approximation of the machine domain.
Revised: (2) is that the transfer set D t is an unbiased approximation of the machine domain.

Original: However, the observed long-tailed and temperature-sensitive distributions of teacher’s predictions in Figure 1 rationally challenge this assumption.
Revised: However, the observed long-tailed and temperature-sensitive distributions of teacher’s predictions in Figure 1 rationally challenge this assumption.

Original: As a result, samples with lower P ( x | machine ) are under-represented during the distillation process, which affects the unbiasedness of knowledge transfer.
Revised: As a result, samples with lower P ( x | machine ) are under-represented during the distillation process, which affects the unbiasedness of knowledge transfer.

Original: This analysis indicates that Eq.
Revised: This analysis indicates that Eq.

Original: (2) is not optimal to utilize the teacher’s imbalanced knowledge.
Revised: (2) is not optimal to utilize the teacher’s imbalanced knowledge.

Original: 3.2 Transfer Gap in KD
Revised: 3.2 Transfer Gap in KD

Original: [Equation]
Revised: [Equation]

Original: We interpret the transfer gap and its confounding effect from the perspective of causal inference.
Revised: We interpret the transfer gap and its confounding effect from the perspective of causal inference.

Original: Figure 2 illustrates the causal relations between the image X , training data D
Revised: Figure 2 illustrates the causal relations between the image X , training data D

Original: = { ( x, y ) } , teacher’s parameters θ t and teacher’s output Y t in KD.
Revised: = { ( x, y ) } , teacher’s parameters θ t and teacher’s output Y t in KD.

Original: Overall,
Revised: Overall,

Original: D and θ t jointly act as the confounder of X and Y t in the transfer set.
Revised: D and θ t jointly act as the confounder of X and Y t in the transfer set.

Original: First, the training set D and transfer set of teacher model D t = { ( x, y t ) } share the same image set, and X = x is sampled from the image set of D , i.e., D serves the cause of X .
Revised: First, the training set D and transfer set of teacher model D t = { ( x, y t ) } share the same image set, and X = x is sampled from the image set of D , i.e., D serves the cause of X .

Original: Second, the teacher θ t is trained on D , and y t is calculated based on θ t and x , i.e., y t
Revised: Second, the teacher θ t is trained on D , and y t is calculated based on θ t and x , i.e., y t

Original: = f
Revised: = f

Original: ( x ; θ t ) .
Revised: ( x ; θ t ) .

Original: Therefore, X and θ t are the cause of Y t .
Revised: Therefore, X and θ t are the cause of Y t .

Original: Note that the transfer set is constructed based on the images on D and teacher model θ t .
Revised: Note that the transfer set is constructed based on the images on D and teacher model θ t .

Original: Therefore, we regard the transfer set D t , the joint of D and θ t , as the confounder of X and Y t .
Revised: Therefore, we regard the transfer set D t , the joint of D and θ t , as the confounder of X and Y t .

Original: Although D is balanced when considering the context-invariant class-specific information, the context information (e.g., attributes) is overlooked, which makes the D imbalanced on context.
Revised: Although D is balanced when considering the context-invariant class-specific information, the context information (e.g., attributes) is overlooked, which makes the D imbalanced on context.

Original: As shown in Figure 1, such  imbalanced context leads to an imbalanced transfer set D t and further affects the distillation performance of teacher’s knowledge.
Revised: As shown in Figure 1, such an imbalanced context leads to an imbalanced transfer set D t and further affects the distillation performance of teacher’s knowledge.

Original: To overcome such confounding effect, a commonly used technique is intervention via P ( y t | do ( x )) instead of P ( y t | x ) , which is formulated as P ( y t | do ( x ))
Revised: To overcome such confounding effect, a commonly used technique is intervention via P ( y t | do ( x )) instead of P ( y t | x ) , which is formulated as P ( y t | do ( x ))

Original: = (cid:80) D t P ( y | x, D t ) P ( D t )
Revised: = (cid:80) D t P ( y | x, D t ) P ( D t )

Original: = (cid:80) D t P ( x,y t , D t ) P ( x |D t ) .
Revised: = (cid:80) D t P ( x,y t , D t ) P ( x |D t ) .

Original: This transformation suggests that we can use the inverse of propensity score, 1 /P ( x |D t ) , as sample weight to implement the intervention and overcome the confounding effect.
Revised: This transformation suggests that we can use the inverse of propensity score, 1 /P ( x |D t ) , as sample weight to implement the intervention and overcome the confounding effect.

Original: Thanks to the causalitybased theory [55, 5], we can use the Inverse Probability Weighting (IPW) technique to overcome the confounding effect brought by the transfer gap.
Revised: Thanks to the causalitybased theory [49, 5], we can use the Inverse Probability Weighting (IPW) technique to overcome the confounding effect brought by the transfer gap.

Original: i npu t s t u d e n t ba c k b on e f e a t u r e
Revised: I npu t s t u d e n t ba c k b on e f e a t u r e

Original: L k dK D  h e ad y k d
Revised: L k d D i s t ill h e ad y s

Original: i npu t s t u d e n t ba c k b on e f e a t u r e
Revised: I npu t s t u d e n t ba c k b on e f e a t u r e

Original: C L S h e ad
Revised: C L S h e ad

Original: K D  h e ad y c l s y k d
Revised: D i s t ill h e ad y c l s y s

Original: L c l s
Revised: L c l s

Original: L i p w d w
Revised: L i p w d w

Original: Eq.
Revised: Eq.

Original: Eq. Eq.
Revised: Eq. Eq.

Original: [Figure]
Revised: [Figure]

Original:  Method
Revised:  Method

Original: We propose a simple yet effective method, Inverse Probability Weighting Distillation (IPWD), to respect the transfer gap and imbalance knowledge in KD.
Revised: We propose a simple yet effective method, Inverse Probability Weighting Distillation (IPWD), to respect the transfer gap and imbalance knowledge in KD.

Original: In this section, we first introduce the overall framework of IPWD, then present the implementation details.
Revised: In this section, we first introduce the overall framework of IPWD, then present the implementation details.

Original: 4.1 Inverse Probability Weighting for KD
Revised: 4.1 Inverse Probability Weighting for KD

Original: As analyzed in Section 3, the IID training samples in the human domain are no longer IID in the machine domain.
Revised: As analyzed in Section 3, the IID training samples in the human domain are no longer IID in the machine domain.

Original: Simply assuming the training set as the perfect transfer set may lead to the selection bias: samples that match “head” knowledge are over-represented and easy to be observed, while samples that match “tail” knowledge are under-represented and hard to be observed.
Revised: Simply assuming the training set as the perfect transfer set may lead to the selection bias: samples that match “head” knowledge are over-represented and easy to be observed, while samples that match “tail” knowledge are under-represented and hard to be observed.

Original: This would suppress the transfer of “tail” knowledge.
Revised: This would suppress the transfer of “tail” knowledge.

Original: The analysis from the perspective of causal inference in
Revised: The analysis from the perspective of causal inference in

Original: Section 3.2 suggests that we can use Inverse Probability Weighting (IPW) for debiased distillation.
Revised: Section 3.2 suggests that we can use Inverse Probability Weighting (IPW) for debiased distillation.

Original: In short, IPW generates a pseudo-population where under-represented samples are assigned with large weights and over-represented samples are assigned with small weights.
Revised: In short, IPW generates a pseudo-population where under-represented samples are assigned with large weights and over-represented samples are assigned with small weights.

Original: The weight for sample x is determined as the inverse of its probability, also known as propensity score, to the domain d ∈ { human , machine } , i.e. , w x | d = 1 /p ( x | d ) .
Revised: The weight for sample x is determined as the inverse of its probability, also known as propensity score, to the domain d ∈ { human , machine } , i.e. , w x | d = 1 /p ( x | d ) .

Original: We adopt IPW to KD and obtain the following objective for sample x :
Revised: We adopt IPW to KD and obtain the following objective for sample x :

Revised: ℓ ( x ; θ ) = (cid:88)

Original: [Equation]
Revised: [Equation]

Original: 4.2 Implementation
Revised: 4.2 Implementation

Original: Since the training and test data are normally IID in the human domain, we safely and rationally use the empirical risk.
Revised: Since the training and test data are normally IID in the human domain, we safely and rationally use the empirical risk.

Original: Therefore, we assign a constant weight to each sample when calculating the classification loss.
Revised: Therefore, we assign a constant weight to each sample when calculating the classification loss.

Original: For sure, the training and test samples can be both non-IID, e.g. , long-tailed recognition tasks, which is out of the scope of this paper.
Revised: For sure, the training and test samples can be both non-IID, e.g. , long-tailed recognition tasks, which is out of the scope of this paper.

Original: As analyzed in Section 1, the assumption held by traditional KD, i.e. , both P ( X | human ) and
Revised: As analyzed in Section 1, the assumption held by traditional KD, i.e. , both P ( X | human ) and

Original: P ( X | machine ) are IID, is unrealistic in practice.
Revised: P ( X | machine ) are IID, is unrealistic in practice.

Original: Therefore, we should consider the propensity score P ( x | machine ) as a sample-specific value for the distillation loss to improve the generalization.
Revised: Therefore, we should consider the propensity score P ( x | machine ) as a sample-specific value for the distillation loss to improve the generalization.

Original: Traditional IPW estimates the propensity score using logistic regression, i.e. , ˆ P ( x | machine )
Revised: Traditional IPW estimates the propensity score using logistic regression, i.e. , ˆ P ( x | machine )

Original: = 1 / (1 + exp( − z x )) , where z x is the logit for x .
Revised: = 1 / (1 + exp( − z x )) , where z x is the logit for x .

Original: Since the ground-truth annotation of P ( x | machine ) is not available, it is not practicable to directly train the regression model in a fully-supervised manner.
Revised: Since the ground-truth annotation of P ( x | machine ) is not available, it is not practicable to directly train the regression model in a fully-supervised manner.

Original: Therefore, we estimate the propensity score in an unsupervised way.
Revised: Therefore, we estimate the propensity score in an unsupervised way.

Original: Recall that the samples with high propensity are over-represented in the transfer set.
Revised: Recall that the samples with high propensity are over-represented in the transfer set.

Original: As a result, the student model would learn less from the under-represented group via distillation.
Revised: As a result, the student model would learn less from the under-represented group via distillation.

Original: Therefore, we use a classification-trained (CLS-trained) classifier for the human domain as reference, and assume that
Revised: Therefore, we use a classification-trained (CLS-trained) classifier for the human domain as reference, and assume that

Original: a
Revised: a

Original: KD-trained classifier for the machine domain is more confident for the over-represented group than the CLS-trained classifier.
Revised: KD-trained classifier for the machine domain is more confident for the over-represented group than the CLS-trained classifier.

Original: We compare the outputs of two classifiers to identify whether a sample is under-represented in the machine domain.
Revised: We compare the outputs of two classifiers to identify whether a sample is under-represented in the machine domain.

Original: Suppose that the KD-trained output is y kd and the
Revised: Suppose that the KD-trained output is y kd and the

Original: CLS-trained output is y cls .
Revised: CLS-trained output is y cls .

Original: The assumption implies that the logit z x is negatively correlated with
Revised: The assumption implies that the logit z x is negatively correlated with

Original: [Figure]
Revised: [Figure]

Original: H ( y kd , y ) and positively correlated with H ( y cls , y ) , where H ( · , · ) is the cross-entropy defined in
Revised: H ( y kd , y ) and positively correlated with H ( y cls , y ) , where H ( · , · ) is the cross-entropy defined in

Original: Section 3.
Revised: Section 3.

Original: Considering the range of logit, we estimate z x as z x =
Revised: Considering the range of logit, we estimate z x as z x =

Original: log
Revised: log

Original: H
Revised: H

Original: ( y cls ,y ) H ( y kd ,y ) .
Revised: ( y cls ,y ) H ( y kd ,y ) .

Original: Figure 3 illustrates the comparison between traditional KD and our IPWD.
Revised: Figure 3 illustrates the comparison between traditional KD and our IPWD.

Original: We take the KD-trained student’s output y s as y kd , and train an extra classifier head ( i.e. , “CLS head” in Figure 3(b)) to calculate y cls , which is optimized using the cross-entropy loss L cls with the ground-truth labels.
Revised: We take the KD-trained student’s output y s as y kd , and train an extra classifier head ( i.e. , “CLS head” in Figure 3(b)) to calculate y cls , which is optimized using the cross-entropy loss L cls with the ground-truth labels.

Original: As shown in Table 5, we empirically found that directly using y s and y cls would lead to a high variance.
Revised: As shown in Table 5, we empirically found that directly using y s and y cls would lead to a high variance.

Original: For example, a wrongly classified sample may have a extremely large loss and heavily suppress the distillation of other samples.
Revised: For example, a wrongly classified sample may have an extremely large loss and heavily suppress the distillation of other samples.

Original: Therefore, we normalize the logits by dividing them by the standard deviation σ s and σ cls , i.e. , (cid:101)
Revised: Therefore, we normalize the logits by dividing them by the standard deviation σ s and σ cls , i.e. , (cid:101)

Original: y s =
Revised: y s =

Original: exp( z sk /σ s ) (cid:80) Ci =1 exp ( z si /σ s ) and (cid:101) y cls = exp ( z clsk /σ cls ) (cid:80)
Revised: exp( z sk /σ s ) (cid:80) Ci =1 exp ( z si /σ s ) and (cid:101) y cls = exp ( z clsk /σ cls ) (cid:80)

Original: Ci =1 exp ( z clsi /σ cls ) .
Revised: Ci =1 exp ( z clsi /σ cls ) .

Original: In this way, the outputs are at the same scale with a standard deviation equal to 1, which helps to reduce the variance.
Revised: In this way, the outputs are at the same scale with a standard deviation equal to 1, which helps to reduce the variance.

Original: We finally take (cid:101)
Revised: We finally take (cid:101)

Original: y s as y kd and (cid:101)
Revised: y s as y kd and (cid:101)

Original: y cls as y cls .
Revised: y cls as y cls .

Original: Combining z x into the propensity score, we have:
Revised: Combining z x into the propensity score, we have:

Original: [Equation]
Revised: [Equation]

Original: and the estimated weight ˆ w x for sample x is ˆ
Revised: and the estimated weight ˆ w x for sample x is ˆ

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: Figure 4 illustrates some examples of training samples and their assigned weights.
Revised: Figure 4 illustrates some examples of training samples and their assigned weights.

Original: Under-represented samples, for which the KD-trained classifier is less confident than the CLS-trained classifier, are assigned with a large weight (Figure 4(a)). Over-represented samples, for which the KD-trained classifier is more confident than the CLS-trained classifier, are assigned with a small weight (Figure 4(c)).
Revised: Under-represented samples, for which the KD-trained classifier is less confident than the CLS-trained classifier, are assigned with a large weight (Figure 4(a)). Over-represented samples, for which the KD-trained classifier is more confident than the CLS-trained classifier, are assigned with a small weight (Figure 4(c)).

Original: Samples for which the two classifiers behave similarly are assigned with a balanced weight (Figure 4(b)).
Revised: Samples for which the two classifiers behave similarly are assigned with a balanced weight (Figure 4(b)).

Original: The weighted distillation loss is formulated as:
Revised: The weighted distillation loss is formulated as:

Original: [Equation]
Revised: [Equation]

Original: Our final IPWD objective is formulated as:
Revised: Our final IPWD objective is formulated as:

Original: [Equation]
Revised: [Equation]

Original: where α is a trade-off hyper-parameter between classification and distillation.
Revised: where α is a trade-off hyper-parameter between classification and distillation.

Original: Limitations and negative societal impacts. .
Revised: Limitations and negative societal impacts. .

Original: As introduced in Section 4.2, we estimate the propensity score by comparing the heads of the student model.
Revised: As introduced in Section 4.2, we estimate the propensity score by comparing the heads of the student model.

Original: Therefore, the estimation relies on the quality of the student model.
Revised: Therefore, the estimation relies on the quality of the student model.

Original: A poor student may not correctly estimate the propensity, which may further suppress the effectiveness of IPWD.
Revised: A poor student may not correctly estimate the propensity, which may further suppress the effectiveness of IPWD.

Original: Also, we hold an assumption that the training and test samples are IID in the human domain, which may not be valid for long-tailed tasks.
Revised: Also, we assume that the training and test samples are IID in the human domain, which may not be valid for long-tailed tasks.

Original: To the best of our knowledge, as our work is purely an algorithm for knowledge distillation, we haven’t found any negative societal impact.
Revised: To the best of our knowledge, as our work is purely an algorithm for knowledge distillation, we haven’t found any negative societal impact.

Original: [Table]
Revised: [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: NST [25]
Revised: NST [23]

Original: [Table]
Revised: [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: KD [21] 73.
Revised: KD [20] 73.

Original: [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: CRD [66] 74.
Revised: CRD [57] 74.

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table]

Original: [Table] [Table] [Table] WSLD ∗ [85] 73.
Revised: WSLD ∗ [73] 73.

Original: [Table]

Original: [Table] [Table] [Table] IPWD 
Revised: IPWD 74.

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original:  Experiments
Revised:  Experiments

Original: We take the image classification task as a case study to evaluate the effectiveness and generalizability of our IPWD.
Revised: We take the image classification task as a case study to evaluate the effectiveness and generalizability of our IPWD.

Original: Following previous works [66, 85, 31], we conduct experiments with two settings, two-stage distillation and one-stage self-distillation.
Revised: Following previous works [57, 73, 28], we conduct experiments with two settings, two-stage distillation and one-stage self-distillation.

Original: 5.1 Datasets and Settings
Revised: 5.1 Datasets and Settings

Original: Datasets .
Revised: Datasets .

Original: We conducted experiments on CIFAR-100 [32] and ImageNet [10].
Revised: We conducted experiments on CIFAR-100 [29] and ImageNet [10].

Original: CIFAR-100 contains 50K images in the training set and 10K images in the test set from 100 classes.
Revised: CIFAR-100 contains 50K images in the training set and 10K images in the test set from 100 classes.

Original: ImageNet provides 1.2M images in the training set and 50K images in the validation set from 1K classes.
Revised: ImageNet provides 1.2M images in the training set and 50K images in the validation set from 1K classes.

Original: Settings .
Revised: Settings .

Original: Two-stage distillation is the conventional setting that pre-trains a teacher model at the first stage and transfers the knowledge to a student model at the second stage.
Revised: Two-stage distillation is the conventional setting that pre-trains a teacher model at the first stage and transfers the knowledge to a student model at the second stage.

Original: Commonly, the teacher is a larger model, and the student is a smaller model.
Revised: Commonly, the teacher is a larger model, and the student is a smaller model.

Original: For self-distillation, the teacher and student have the same architecture.
Revised: For self-distillation, the teacher and student have the same architecture.

Original: One-stage self-distillation aims to complete the teacher-student optimization simultaneously [75, 31], i.e. , the pre-training and transfer processes are reduced to one.
Revised: One-stage self-distillation aims to complete the teacher-student optimization simultaneously [65, 28], i.e. , the pre-training and transfer processes are reduced to one.

Original: 5.2 Two-stage Distillation
Revised: 5.2 Two-stage Distillation

Original: Baseline methods .
Revised: Baseline methods .

Original: For two-stage distillation, following Tian et al.
Revised: For two-stage distillation, following Tian et al.

Original: [66] and Zhou et al.
Revised: [57] and Zhou et al.

Original: [85], we considered the following methods as baselines: KD [21], FitNet [54], AT [79], SP [68], CC [52],
Revised: [73], we considered the following methods as baselines: KD [20], FitNet [48], AT [69], SP [59], CC [46],

Original: VID [2], RKD [50], PKT [51], FSP [76], AB [20], FT [30], NST [25], CRD [66], SSKD [74], and
Revised: VID [2], RKD [44], PKT [45], FSP [66], AB [19], FT [27], NST [23], CRD [57], SSKD [64], and

Original: WSLD [85].
Revised: WSLD [73].

Original: In particular, WSLD [85] is the most related work to us, which proposed a bias-variance trade-off perspective for KD and also assigns different weights to each training sample.
Revised: In particular, WSLD [73] is the most related work to us, which proposed a bias-variance trade-off perspective for KD and also assigns different weights to each training sample.

Original: Similarly, the weight is positive related to the cross-entropy loss of student’s output.
Revised: Similarly, the weight is positive related to the cross-entropy loss of student’s output.

Original: The main differences between our IPWD and WSLD are as follows.
Revised: The main differences between our IPWD and WSLD are as follows.

Original: First, our formulation of samples weights is theoretically guaranteed by the causal theory behind Inverse Probability Weighting (IPW) [55, 37, 26, 5].
Revised: First, our formulation of samples weights is theoretically guaranteed by the causal theory behind Inverse Probability Weighting (IPW) [49, 34, 24, 5].

Original: Second,
Revised: Second,

Original: WSLD estimates the sample weight using both the student model and the teacher model.
Revised: WSLD estimates the sample weight using both the student model and the teacher model.

Original: As a comparison, we use the student model with two different classifier heads to guarantee that the capacities of the compared models are close.
Revised: As a comparison, we use the student model with two different classifier heads to guarantee that the capacities of the compared models are close.

Original: Implementation .
Revised: Implementation .

Original: For experiments on CIFAR-100, we followed CRD [66] based on the open-sourced code.
Revised: For experiments on CIFAR-100, we followed CRD [57] based on the open-sourced code.

Original: We set the trade-off hyper-parameter α = 5 in Eq.
Revised: We set the trade-off hyper-parameter α = 5 in Eq.

Original: (7) and the temperature τ = 10 .
Revised: (7) and the temperature τ = 10 .

Original: Other training details were the same as CRD [66] and provided in the appendix.
Revised: Other training details were the same as CRD [57] and provided in the appendix.

Original: For ImageNet, we followed
Revised: For ImageNet, we followed

Original: [Table]
Revised: [Table]

Original: Zhou et al.
Revised: Zhou et al.

Original: [85] to conduct experiments based on their open-sourced code.
Revised: [73] to conduct experiments based on their open-sourced code.

Original: We used the same hyper parameters as WSLD [85], i.e. , α as 2.5 and τ as 2.
Revised: We used the same hyper parameters as WSLD [73], i.e. , α as 2.5 and τ as 2.

Original: Comparison with baseline methods .
Revised: Comparison with baseline methods .

Original: Table 2 shows the results of student models on CIFAR-100 with different teacher-student architectures, which can be grouped into same architecture style and different architecture style.
Revised: Table 2 shows the results of student models on CIFAR-100 with different teacher-student architectures, which can be grouped into same architecture style and different architecture style.

Original: Note that the results of WSLD reported in [85] used a different pre-trained teacher model.
Revised: Note that the results of WSLD reported in [73] used a different pre-trained teacher model.

Original: Since some training techniques like early-stopping [8, 11, 77] may improve the distillation performance, we reimplemented WSLD using the same teacher model for a fair comparison.
Revised: Since some training techniques like early-stopping [8, 11, 67] may improve the distillation performance, we reimplemented WSLD using the same teacher model for a fair comparison.

Original: Overall, our IPWD outperforms KD by large margins and outperforms other baseline methods on most of the architectures, which demonstrates the effectiveness of our IPWD.
Revised: Overall, our IPWD outperforms KD by large margins and outperforms other baseline methods on most of the architectures, which demonstrates the effectiveness of our IPWD.

Original: In particular, the improvement with the same architecture style is smaller than the different style.
Revised: In particular, the improvement with the same architecture style is smaller than the different style.

Original: The reason is that the different architecture style reflects the bigger gap between the human domain and machine domain.
Revised: The reason is that the different architecture style reflects the bigger gap between the human domain and machine domain.

Original: Since our
Revised: Since our

Original: IPWD weights the training samples to address the non-IID problem, IPWD successfully outperforms
Revised: IPWD weights the training samples to address the non-IID problem, IPWD successfully outperforms

Original: KD and other state-of-the-art methods by large margins when the transfer gap is significant.
Revised: KD and other state-of-the-art methods by large margins when the transfer gap is significant.

Original: [Table]
Revised: [Table]

Original: Note that SSKD [74] achieves higher performance because of (1) a better teacher model, and (2) data augmentation for structured knowledge distillation.
Revised: Note that SSKD [64] achieves higher performance because of (1) a better teacher model, and (2) data augmentation for structured knowledge distillation.

Original: We further apply IPWD to SSKD as a plugin by weighting the logit distillation objective and keeping the structured knowledge distillation terms unchanged.
Revised: We further apply IPWD to SSKD as a plugin by weighting the logit distillation objective and keeping the structured knowledge distillation terms unchanged.

Original: Table 3 shows that our IPWD can consistently improve SSKD by 0.5~1.0% for different architectures.
Revised: Table 3 shows that our IPWD can consistently improve SSKD by 0.5~1.0% for different architectures.

Original: These results indicate that our IPWD is a good complementary to distillation methods.
Revised: These results indicate that our IPWD is a good complementary to distillation methods.

Original: Table 4 further shows the comparison on ImageNet.
Revised: Table 4 further shows the comparison on ImageNet.

Original: Following CRD [66] and WSLD [85], we used two teacherstudent architectures as the representatives.
Revised: Following CRD [57] and WSLD [73], we used two teacherstudent architectures as the representatives.

Original: For the same architecture style, our IPWD improves KD by 1.21%, and achieves competitive performance compared to WSLD.
Revised: For the same architecture style, our IPWD improves KD by 1.21%, and achieves competitive performance compared to WSLD.

Original: For the different architecture style, the improvement of WSLD over KD drops from 1.37% to 1.03%.
Revised: For the different architecture style, the improvement of WSLD over KD drops from 1.37% to 1.03%.

Original: As a comparison, our IPWD improves KD by 2.16%, and outperforms WSLD by 1.13%.
Revised: As a comparison, our IPWD improves KD by 2.16%, and outperforms WSLD by 1.13%.

Original: This improvement on the large-scale dataset further demonstrates the effectiveness of our IPWD to bridge the transfer gap when the student and teacher model have different architecture styles, which is more practical in real-world applications.
Revised: This improvement on the large-scale dataset further demonstrates the effectiveness of our IPWD in bridging the transfer gap when the student and teacher model have different architecture styles, which is more practical in real-world applications.

Original: Ablation study: technical designs .
Revised: Ablation study: technical designs .

Original: As introduced in Section 4.2, we used an extra classifier head to produce CLS-trained output, and normalized the logits to reduce the variance for propensity estimation.
Revised: As introduced in Section 4.2, we used an extra classifier head to produce CLS-trained output, and normalized the logits to reduce the variance for propensity estimation.

Original: Note that WSLD [85] uses the teacher model to estimate the sample weight, and the teacher model is also trained with the cross-entropy loss.
Revised: Note that WSLD [73] uses the teacher model to estimate the sample weight, and the teacher model is also trained with the cross-entropy loss.

Original: Therefore, we considered an alternative 
Revised: Therefore, we considered an alternative which replaces the classification head with the teacher model to produce the classification-aware

Original: [Table]
Revised: [Table]

Original: that which replaces the classification head with the teacher model to produce the classification-aware output.
Revised: output.

Original: To evaluate the contribution of logits normalization
Revised: To evaluate the contribution of logits normalization

Original: [Table]
Revised: [Table]

Original: we considered an alternative that the logits are not normalized by the standard deviation.
Revised: we considered an alternative that the logits are not normalized by the standard deviation.

Original: Results in Table 5 verify the contribution of each design.
Revised: Results in Table 5 verify the contribution of each design.

Original: Without the classification head and logit normalization, the training is hard to converge or the performance is much worse.
Revised: Without the classification head and logit normalization, the training is hard to converge or the performance is much worse.

Original: As a comparison, either the classification head or logit normalization helps with stable training.
Revised: As a comparison, either the classification head or logit normalization helps with stable training.

Original: Besides, a combination of both further improves the performance and achieves the best results.
Revised: Besides, a combination of both further improves the performance and achieves the best results.

Original: The crash of training is due to the high variance of sample weights.
Revised: The crash of training is due to the high variance of sample weights.

Original: Since the teacher model is well pre-trained and has more parameters, it has a larger capacity than the student model.
Revised: Since the teacher model is well pre-trained and has more parameters, it has a larger capacity than the student model.

Original: Differently, an extra head with a shared backbone guarantees a similar capacity.
Revised: Differently, an extra head with a shared backbone guarantees a similar capacity.

Original: Also, the normalization will avoid an extremely large or small CE loss, which further reduces the variance.
Revised: Also, the normalization will avoid an extremely large or small CE loss, which further reduces the variance.

Original: Teacher trained with label smoothing.
Revised: Teacher trained with label smoothing.

Original: Recent works [45, 58] observed that KD performs poorly with label smoothing.
Revised: Recent works [40, 51] observed that KD performs poorly with label smoothing.

Original: Similar to KD, the performance of IPWD drops when the teacher model is trained with label smoothing, but still outperforms KD.
Revised: Similar to KD, the performance of IPWD drops when the teacher model is trained with label smoothing, but still outperforms KD.

Original: However, we found that the improvement of
Revised: However, we found that the improvement of

Original: IPWD compared to KD also decreases with label smoothing.
Revised: IPWD compared to KD also decreases with label smoothing.

Original: For example, on CIFAR-100, given
Revised: For example, on CIFAR-100, given

Original: ResNet50 as teacher and MobileNetV2 as student, IPWD outperforms KD by 1.12% (69.67% vs.
Revised: ResNet50 as teacher and MobileNetV2 as student, IPWD outperforms KD by 1.12% (69.67% vs.

Original: 68.55%) without label smoothing, but the improvements drops to 0.56% (66.79% vs. 66.23%) with label smoothing.
Revised: 68.55%) without label smoothing, but the improvement drops to 0.56% (66.79% vs. 66.23%) with label smoothing.

Original: Given resnet32x4 as teacher and ShuffleNetV1 as student, IPWD outperforms
Revised: Given resnet32x4 as teacher and ShuffleNetV1 as student, IPWD outperforms

Original: KD by 1.52% (75.79% vs. 74.27%) without label smoothing, but the improvement drops to 0.53% (73.27% vs. 72.74%) with label smoothing.
Revised: KD by 1.52% (75.79% vs. 74.27%) without label smoothing, but the improvement drops to 0.53% (73.27% vs. 72.74%) with label smoothing.

Original: We observed that teacher trained with label smoothing produces more balanced predictions compared to teacher trained without label smoothing.
Revised: We observed that teacher trained with label smoothing produces more balanced predictions compared to teacher trained without label smoothing.

Original: Therefore, the results are consistent with our hypothesis that IPWD helps to bridge the transfer gap especially when the context information of teacher is imbalanced.
Revised: Therefore, the results are consistent with our hypothesis that IPWD helps to bridge the transfer gap especially when the context information of teacher is imbalanced.

Original: 5.3 One-stage Self-Distillation
Revised: 5.3 One-stage Self-Distillation

Original: Baseline methods and metrics .
Revised: Baseline methods and metrics .

Original: For one-stage self-distillation, we apply our method to the stateof-the-art PS-KD [31] method as a plug-in, and consider label smoothing (LS) method and two self-distillation methods, CS-KD [78] and TF-KD [77], as baselines.
Revised: For one-stage self-distillation, we apply our method to the stateof-the-art PS-KD [28] method as a plug-in, and consider label smoothing (LS) method and two self-distillation methods, CS-KD [68] and TF-KD [67], as baselines.

Original: PS-KD proposed a one-stage framework that progressively distills the knowledge of a model itself to soften the one-hot supervisions as regularization.
Revised: PS-KD proposed a one-stage framework that progressively distills the knowledge of a model itself to soften the one-hot supervisions as regularization.

Original: The knowledge is transferred using a conventional distillation loss.
Revised: The knowledge is transferred using a conventional distillation loss.

Original: As for metrics, besides top-1 and top-5 accuracy, we follow Kim et al.
Revised: As for metrics, besides top-1 and top-5 accuracy, we follow Kim et al.

Original: [31] to report expected calibration error (ECE, %) and the area under the risk-coverage curve (AURC, × 10 3 ).
Revised: [28] to report expected calibration error (ECE, %) and the area under the risk-coverage curve (AURC, × 10 3 ).

Original: A low ECE indicates well-calibrated predictions, and a low AURC represents the well-separation of correct and incorrect predictions.
Revised: A low ECE indicates well-calibrated predictions, and a low AURC represents the well-separation of correct and incorrect predictions.

Original: Implementation .
Revised: Implementation .

Original: We follow all the training details of PS-KD for a fair comparison.
Revised: We follow all the training details of PS-KD for a fair comparison.

Original: Specifically, the architectures we considered are ResNet-18[17], ResNet-101 [16],  ResNeXt-29 [73] (cardinality=8, width=64), and DenseNet-121 [24] (growth rate=32).
Revised: Specifically, the architectures we considered are ResNet-18 [16], ResNet-101 [15], ResNeXt-29 [63] (cardinality=8, width=64), and DenseNet-121 [22] (growth rate=32).

Original: During training, PS-KD gradually determine how much the student learns from the teacher’s knowledge.
Revised: During training, PS-KD gradually determines how much the student learns from the teacher’s knowledge.

Original: The formulation is:
Revised: The formulation is:

Original: [Equation]
Revised: [Equation]

Original: where the trade-off parameter α t = α T × t/T , T is the number of total epochs ( e.g. , 300), t is the current epoch, and α T is a hyperparameter.
Revised: where the trade-off parameter α t = α T × t/T , T is the number of total epochs ( e.g. , 300), t is the current epoch, and α T is a hyperparameter.

Original: Compared to Eq.
Revised: Compared to Eq.

Original: (8), our IPWD applied on PS-KD is formulated as
Revised: (8), our IPWD applied on PS-KD is formulated as

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: Since both the student and teacher models are poor at early epochs, the weight estimation is not accurate at early epochs, which may lead to a worse self-teacher.
Revised: Since both the student and teacher models are poor at early epochs, the weight estimation is not accurate at early epochs, which may lead to a worse self-teacher.

Original: Therefore, we apply IPWD at the last 75 epochs over the total 300 epochs.
Revised: Therefore, we apply IPWD at the last 75 epochs over the total 300 epochs.

Original: Comparison with baseline methods .
Revised: Comparison with baseline methods .

Original: Table 6 shows the results of one-stage self-distillation methods over four architectures.
Revised: Table 6 shows the results of one-stage self-distillation methods over four architectures.

Original: Our IPWD can effectively and constantly improve the top-1 accuracy of
Revised: Our IPWD can effectively and constantly improve the top-1 accuracy of

Original: PS-KD by 0.33% ∼ 0.82% with different architectures.
Revised: PS-KD by 0.33% ∼ 0.82% with different architectures.

Original: Besides, our IPWD siginifantly lowers the
Revised: Besides, our IPWD significantly lowers the

Original: ECE and AURC of PS-KD.
Revised: ECE and AURC of PS-KD.

Original: These results demonstrate the effectiveness of our IPWD.
Revised: These results demonstrate the effectiveness of our IPWD.

Original: [Table]
Revised: [Table]

Original: Ablation study: IPWD stage .
Revised: Ablation study: IPWD stage .

Original: We conduct an ablation study to analyze whether IPWD should be started from an early stage ( e.g. , the beginning of training) or a late stage ( e.g. , last 1/4 of the epochs).
Revised: We conduct an ablation study to analyze whether IPWD should be started from an early stage ( e.g. , the beginning of training) or a late stage ( e.g. , last 1/4 of the epochs).

Original: We take ResNeXt-29 as an example.
Revised: We take ResNeXt-29 as an example.

Original: As shown in Table 7, applying IPWD from the beginning slightly outperforms PS-KD and under-performs the student modal that applies IPWD only at the late stage by large margins.
Revised: As shown in Table 7, applying IPWD from the beginning slightly outperforms PS-KD and under-performs the student modal that applies IPWD only at the late stage by large margins.

Original: As the student model is poorly trained at the early stage, the weight estimation is inaccurate and hurts the performance of self-teacher.
Revised: As the student model is poorly trained at the early stage, the weight estimation is inaccurate and hurts the performance of self-teacher.

Original: These results indicate that the quality of estimated weight and distillation performance relies on the student model and self-teacher.
Revised: These results indicate that the quality of estimated weight and distillation performance relies on the student model and self-teacher.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we point out that conventional KD methods hold an invalid IID assumption and do not properly address the transfer gap between the context-invariant human domain and the contextequivariant machine domain, especially the imbalance knowledge of the teacher model on the transfer set.
Revised: In this paper, we point out that conventional KD methods hold an invalid IID assumption and does not properly address the transfer gap between the context-invariant human domain and the context-equivariant machine domain, especially the imbalanced knowledge of the teacher model on the transfer set.

Original: We further proposed a simple yet effective method, Inverse Probability Weighting
Revised: We further proposed a simple yet effective method, Inverse Probability Weighting

Original: Distillation (IPWD), to deal with the imbalanced knowledge caused by transfer gap.
Revised: Distillation (IPWD), to deal with the imbalanced knowledge caused by transfer gap.

Original: In the future, we will extend our IPWD to (1) tasks beyond classification, like detection and segmentation, and (2) long-tailed tasks where the training samples in the human domain are also non-IID.
Revised: In the future, we will extend our IPWD to (1) tasks beyond classification, like detection and segmentation, and (2) long-tailed tasks where the training samples in the human domain are also non-IID.

Original: Acknowledgement
Revised: Acknowledgment

Original: We thank anonymous ACs and reviewers for their valuable discussion and insightful suggestions.
Revised: We thank anonymous ACs and reviewers for their valuable discussion and insightful suggestions.

Original: This research is supported by the National Research Foundation, Singapore under its AI Singapore
Revised: This research is supported by the National Research Foundation, Singapore under its AI Singapore

Original: Programme (AISG Award
Revised: Programme (AISG Award

Original: No: AISG2-RP-2021-022) and Alibaba-NTU Singapore Joint Research
Revised: No: AISG2-RP-2021-022) and Alibaba-NTU Singapore Joint Research

Original: Institute (JRI).
Revised: Institute (JRI).

Original: Appendix A Experimental Settings A.1 Two-stage Distillation
Revised: Checklist

Original: Implementation . We follow the training details of Tian et al.

Original: [66] for CIFAR-100 and Zhou et al. [85] for ImageNet. Specifically, for CIFAR-100, we set the mini-batch size as 64 and an initial learning rate as 0.05.
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: We train the model for 240 epochs. The learning rate is decayed by 10 every 30 epochs after 150 epochs. We initialize the learning rate as 0.01 for MobileNetV2, ShuffleNetV and ShuffleNetV2, and as 0.05 for other models.
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: The experiments are conducted using one NVIDIA TITAN RTX GPU. For ImageNet, we train the model for 100 epochs. We set the mini-batch size as 256, an initial learning rate as 0.1, and decay it by 10 every 30 epochs.
Revised: [Yes] See Section 4.

Original: The experiments are conducted using four Tesla V100 GPUs. Architectures . We follow Tian et al. [66] for the choice of network architectures.
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: Specifically, “WRN-d-w” denotes Wide Residual Network (WRN) [80] with depth d and width factor w . “resnet-d” represents cifar-style ResNet [16] with 3 groups of basic blocks, each with 16, 32, and 64 channels respectively. For example, resnet8x4 and resnet32x4 represent a 4 times wider network, i.e. , with 64, 128, and 256 channels respectively. “ResNet-d” represents the ImageNet-style ResNet with
Revised: [Yes] To the best of our knowledge, as our work is purely an algorithm for knowledge distillation, we haven’t found any negative societal impact.

Original: Bottleneck blocks and more channels. MobileNetV2 [56] has a width multiplier of 0.5. “vgg” denotes VGGNet [60] that is adapted from its original ImageNet counterpart. “ShuffleNetV1” [83], “ShuffleNetV2” [40] are adapted with input size as 32x32.
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: A.2 One-
Revised: [Yes] 2.

Original: stage Self-Distillation

Original: Implementation . We follow all the training details of PS-KD [31]. The standard data argumentation schemes are 32x32 random crop after padding with 4 pixels and random horizontal flip. The networks are trained for 300 epochs using SGD with a momentum of 0.9.
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: The learning rate is decayed by 10 at 150 and 225 epochs. We set the mini-batch size as 128, an initial learning rate as 0.1, and a weight decay as 0.0005. The experiments are conducted using four Tesla V100 GPUs. Metrics .
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: Besides top-1 and top-5 accuracies, we follow Kim et al.

Original: [31] and report Expected calibration error (ECE) and Area under risk-coverage curve (AURC) for evaluation. Expected calibration error (ECE) [47] is used to evaluate the confidence calibration performance of models, i.e. , the expected gap between accuracy and confidence. Specifically, the samples are partitioned by confidence into M bins B 1 , · · · , B M .
Revised: [N/A] 3.

Original: The m -th bin B m contains samples with confidence within [ m − 1 M , mM ] . For N samples, ECE is formulated as: [Equation] where Acc ( B m ) denotes the accuracy of samples in B m , and Conf ( B m ) denotes the average confidence of samples in B m .
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: A lower BCE indicates a well-calibrated model. Area under risk-coverage curve (AURC) [14] measures how well predictions are ordered by confidence values. Specifically, we can determine a threshold for classification, where only samples with confidence higher than the threshold are accepted.
Revised: [Yes] Please refer to the supplemental material.

Original: After that, we can obtain the proportion of covered samples to the whole dataset, i.e. , coverage, and define the risk as the error rate computed by using the covered samples. AURC is defined as the area under the risk-coverage curve. A lower AURC indicates that the correct and incorrect predictions are well-separable by confidence values. A.3 License of Assets
Revised: (b) Did you specify all the training details (e

Original: We reimplemented WSLD 2 and SSKD 3 based on their open-resourced codes. Both WSLD and SSKD did not mention the license in their open-resourced codes. B Additional Results B.1 Architecture styles Due to page limitation, we did not include all the results of different architecture styles on CIFARin the main paper.
Revised: g., data splits, hyperparameters, how they were chosen)?

Original: We provide additional results in Table 8.
Revised: [Yes] The details are in Section 5.

Original: Results show that our IPWD achieves competitive performances and outperforms KD by large margins. B.2 Feature-based Methods
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: Note that our proposed IPWD is a logit-based distillation method. An interesting question is whether the reweighting strategy can work with feature-based distillation methods. We select ReviewKD as an example, which is a recent representative feature-based distillation method.
Revised: [Yes] We run the experiments with five different random seeds. (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: As shown in Figure 9, the gap between ReviewKD+IPWD and ReviewKD is very marginal, which indicates that IPWD cannot promote feature-based distillation. The possible reasons are two-fold. First, the logit knowledge of label y is long-tailed but the representation knowledge of sample x may be relatively balanced. Second, as pointed out by Kang et al.
Revised: [Yes] See Appendix. 4.

Original: [29], “data imbalance might not be an issue in learning highquality representations” for long-tailed classification, which implies that the reweighting strategy is not compatible at feature level. B.3 Long-tailed Methods for KD Note that Figure 1 shows that there seems long-tailed property of teacher predictions.
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: An interesting question is whether it could be fixed by techniques for long-tailed classification. We take LA [42] [Table]
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [Table] ReviewKD + IPWD 76.
Revised: [Yes] See Appendix.

Original: as the recent representative technique for long-tailed classification. LA proposed a logit adjusted softmax cross-entropy loss by applying a class prior to each logit. LA does not require extra modules (compared to TDE [65]), post-hoc logit adjustment (compared to LADE [22]), or ensemble of multiple models (compared to RIDE [71]).
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: Following LA, we applied the class prior to the student output when calculating the KL divergence distillation loss. We found that KD+LA underperforms KD by averagely 0.5% on CIFAR-100. The possible reason is that the introduced prior indirectly breaks the teacher’s knowledge for each training sample, which hurts the effectiveness of distillation. These results indicate that logit-adjust-based long-tailed techniques are not applicable to the issue of KD.
Revised: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: We have conducted ablation studies on the components of our proposed IPWD. In the appendix, we further provide ablation studies on the technical designs. We take two-stage distillation on CIFAR-100 as an example.
Revised: [Yes] All the used existing assets are open-sourced for research.

Original: Recall that the sample weights are estimated based on the two types of student’s outputs, KD-trained output y kd from the student’s original head and CLS-trained output y cls from the student’s extra head. We further conduct ablations on the selection of two outputs. For CE-aware output y cls , an straightforward alternative is using an extra model that has the same architecture as student with totally different parameters. This extra model is trained using the crossentropy classification loss.
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: In other words, the difference is whether the visual backbone is shared for the two outputs. We denote this alternative as IPWD ∗ . Table 10 shows the comparison. Overall,
Revised: [Yes] The benchmark dataset does not contain personally identifiable information or offensive content.

Original: IPWD ∗ achieves competitive results compared to IPWD. However, training an extra model leads to more memory and time cost. Therefore, our design that takes an extra head is both effective and efficient. [Table]
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [Table] For KD-trained output y kd [Table] an straightforward alternative is using an extra distillation head like the classification head.
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: Different from the original head of the student [Table] the distillation head is trained only using the distillation loss, which simply mimics the teacher’s output without ground-truth annotations. We denote this alternative as IPWD † .
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: Table 11 shows the comparison. Overall, IPWD † slightly underperforms IPWD with different four architectures. The reason is that the extra distillation head only learns from the teacher model, which is sensitive to the teacher’s performance and weights hard samples more. These two ablation studies further verify the effectiveness of our techinical designs.
Revised: [N/A]