Original: Respecting Transfer Gap in Knowledge Distillation
Revised: Respecting Transfer Gap in Knowledge Distillation

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Knowledge distillation (KD) is essentially a process of transferring a teachermodel’s behavior, e.g., network response, to a student model.
Revised: Knowledge distillation (KD) is essentially a process of transferring a teacher model’s behavior, e.g. , network response, to a student model.

Original: The network responseserves as additional supervision to formulate the machine domain ( machine forshort), which uses the data collected from the human domain ( human for short) asa transfer set.
Revised: The network response serves as additional supervision to formulate the machine domain ( machine for short), which uses the data collected from the human domain ( human for short) as a transfer set.

Original: Traditional KD methods hold an underlying assumption that the datacollected in both human domain and machine domain are both independent andidentically distributed (IID).
Revised: Traditional KD methods hold an underlying assumption that the data collected in both human domain and machine domain are both independent and identically distributed (IID).

Original: We point out that this naïve assumption is unrealisticand there is indeed a transfer gap between the two domains.
Revised: We point out that this naïve assumption is unrealistic and there is indeed a transfer gap between the two domains.

Original: Although the gap offersthe student model external knowledge from the machine domain, the imbalancedteacher knowledge would make us incorrectly estimate how much to transfer fromteacher to student per sample on the non-IID transfer set.
Revised: Although the gap offers the student model external knowledge from the machine domain, the imbalanced teacher knowledge would make us incorrectly estimate how much to transfer from teacher to student per sample on the non-IID transfer set.

Original: To tackle this challenge,we propose Inverse Probability Weighting Distillation (IPWD) that estimates thepropensity of a training sample belonging to the machine domain, and assignsits inverse amount to compensate for under-represented samples.
Revised: To tackle this challenge, we propose Inverse Probability Weighting Distillation (IPWD) that estimates the propensity score of a training sample belonging to the machine domain, and assigns its inverse amount to compensate for under-represented samples.

Original: Experimentson CIFAR-100 and ImageNet demonstrate the effectiveness of IPWD for bothtwo-stage distillation and one-stage self-distillation.
Revised: Experiments on CIFAR-100 and ImageNet demonstrate the effectiveness of IPWD for both two-stage distillation and one-stage self-distillation.

Original:  Introduction
Revised:  Introduction

Original: Knowledge distillation (KD)
Revised: Knowledge distillation (KD)

Original: [18] transfers knowledge from a teacher model, e.g. , a big, cumbersome,and energy-inefficient network, to a student model, e.g. , a small, light, and energy-efficient network,to improve the performance of the student model.
Revised: [21] transfers knowledge from a teacher model, e g. , a big, cumbersome, and energy-inefficient network, to a student model, e.g. , a small, light and energy-efficient network, to improve the performance of the student model.

Original: A common intuition is that a teacher with betterperformance will teach a stronger student.
Revised: A common intuition is that a teacher with better performance will teach a stronger student.

Original: However, recent studies find that the teacher’s accuracy isnot a good indicator of the resultant student performance [8].
Revised: However, recent studies find that the teacher’s accuracy is not a good indicator of the resultant student performance [8].

Original: For example, a poorly-trained teacherwith early stopping can still teach a better student [8, 11, 63]; or, a teacher with a smaller model sizethan the student is also a good teacher [63]; or, a teacher with the same architecture as the studenthelps to improve the student—self-distillation [13, 67, 66, 22].
Revised: For example, a poorly-trained teacher with early stopping can still teach a better student [8, 11, 77]; or, a teacher with a smaller model size than the student is also a good teacher [77]; or, a teacher with the same architecture as the student helps to improve the student—self-distillation [13, 82, 81, 27].

Original: Should we view KD in a perspective of domain transfer [12, 52], we would better understand theabove counter-intuitive findings.
Revised: Should we view KD in a perspective of domain transfer [12, 63], we would better understand the above counter-intuitive findings.

Original: From Figure 1, we can see that teacher predictions and ground-truthlabels indeed behave differently.
Revised: From Figure 1, we can see that teacher predictions and ground-truth labels indeed behave differently.

Original: Although the teacher is trained on the balanced dataset, its predictedprobability distribution over the dataset is imbalanced.
Revised: Although the teacher is trained on the balanced dataset, its predicted probability distribution over the dataset is imbalanced.

Original: Even on the same training set with the samemodel parameter, teachers with different temperature τ yield different “soft label” distributionsfrom the ground-truth ones.
Revised: Even on the same training set with the same model parameter, teachers with different temperature τ yield different “soft label” distributions from the ground-truth ones.

Original: This implies that human and teacher knowledge are from differentdomains, and there is a transfer gap that drives the “dark knowledge” [18] transferring from teacherto student—regardless of “strong” or “weak” teachers, it is a valid transfer as long as there is a gap.
Revised: This implies that human and teacher knowledge are from different domains, and there is a transfer gap that drives the “dark knowledge” [21] transferring from teacher to student—regardless of “strong” or “weak” teachers, it is a valid transfer as long as there is a gap.

Original: However, the transfer gap affects the distillation performance on the under-represented classes, i.e. ,classes on the tail of teacher predictions, which is overlooked in recent studies.
Revised: However, the transfer gap affects the distillation performance on the under-represented classes, i.e. , classes on the tail of teacher predictions, which is overlooked in recent studies.

Original: Taking CIFAR-100 asan example, we rank and divide the 100 classes into 4 groups according to the ranks of predictedprobability.
Revised: Take CIFAR-100 as an example. We rank and divide the 100 classes into 4 groups according to the ranks of predicted [Figure] probability.

Original: Compared to vanilla training, KD achieves better performance in all the subgroups.
Revised: As shown in Table 1, compared to vanilla training, KD achieves better performance in all the subgroups.

Original: However,going deeper into each subgroup, we found that the increase in the top 25 classes is much 
Revised: However, the increase in the top 25 classes is much higher than that in the lastclasses, i

Original: [Figure]
Revised: [Table]

Original: higher than that in the last 25 classes, i.e., averagely 5.14% vs. 0.85%.
Revised: e. , averagely 5.14% vs. 0.85%.

Original: We ask: what causes the gapfrom the first place; or more specifically, why does the teacher’s non-uniform distributed predictionsimplies the gap?
Revised: We ask: what causes the gap from the first place; or more specifically, why does the teacher’s non-uniform distributed predictions implies the gap?

Original: We answer in an invariance vs. equivariance learning point of view [4, 57]: [Table] [Figure]
Revised: We answer in an invariance vs. equivariance learning point of view [4, 69]:

Original: [Table]
Revised: [Table]

Original: Human domain: context invariance .
Revised: Human domain: context invariance .

Original: The discriminative generalization is theability to learn both context-invariant andclass-equivariant information from thediverse training samples per class.
Revised: The discriminative generalization is the ability to learn both context-invariant and classequivariant information from the diverse training samples per class.

Original: Thehuman domain only provides contextinvariant class-specific information, i.e. ,hard targets.
Revised: The human domain only provides context-invariant class-specific information, i.e. , hard targets.

Original: We normally collect a balancedtraining dataset to formulatethe human domain.
Revised: We normally collect a balanced dataset to formulate human domain.

Original: Machine domain: context equivariance .
Revised: Machine domain: context equivariance .

Original: Teacher models often use a temperature variable topreserve the contexts.
Revised: Teacher models often use a temperature variable to preserve the context.

Original: The temperature allows the teacher to represent a sample not only by itscontext-invariant class-specific information, but also its context-equivariant information.
Revised: The temperature allows the teacher to represent a sample not only by its contextinvariant class-specific information, but also its context-equivariant information.

Original: For example,a dog image with soft label 0.8 · dog + 0.2 · wolf may imply that the dog has wolf-like contextualattributes such as “fluffy coat” and “upright ears”.
Revised: For example, a dog image with soft label 0.8 · dog + 0.2 · wolf may imply that the dog has wolf-like contextual attributes such as “fluffy coat” and “upright ears”.

Original: Although the context-invariance ( i.e. , class) isbalanced in the training data, the context-equivariance ( i.e. , context) is imbalanced because the contextbalance is not considered in class-specific data collection [55].
Revised: Although the context-invariance ( i.e. , class) is balanced in the training data, the context-equivariance ( i.e. , context) is imbalanced because the context balance is not considered in class-specific data collection [67].

Original: To construct the transfer set for themachine domain, the teacher model annotates each sample after seeing others, i.e. , being pre-trainedon the whole set.
Revised: To construct the transfer set for the machine domain, the teacher model annotates each sample after seeing others, i.e. , being pre-trained on the whole set.

Original: Interestingly, the diverse context results in a long-tailed imbalanced distribution,which is exactly reflected in Figure 1.
Revised: Interestingly, the diverse context results in a long-tailed imbalanced distribution, which is exactly reflected in Figure 1.

Original: In other words, the teacher’s knowledge is imbalanced eventhough the teacher is trained on a class-balanced dataset.
Revised: In other words, the teacher’s knowledge is imbalanced even though the teacher is trained on a class-balanced dataset.

Original: Now we are ready to point out how the transfer gap is not properly addressed in conventional KDmethods.
Revised: Now we are ready to point out how the transfer gap is not properly addressed in conventional KD methods.

Original: Conventional KD calculates the Cross-Entropy (CE) loss between the ground-truth labeland student’s prediction, and the Kullback–Leibler (KL) divergence [27] loss between the teacher’sand student’s predictions, where a constant weight is assigned for both CE loss and KL loss.
Revised: Conventional KD calculates the Cross-Entropy (CE) loss between the ground-truth label and student’s prediction, and the Kullback–Leibler (KL) divergence [33] loss between the teacher’s and student’s predictions, where a constant weight is assigned for the two losses.

Original: This isessentially based on the underlying assumption that the data in both the human and machine domainsare IID.
Revised: This is essentially based on the underlying assumption that the data in both the human and machine domains are IID.

Original: Based on the analysis of context equivariance, we argue that the assumption is unrealistic,i.e. , the teacher’s knowledge is imbalanced.
Revised: Based on the analysis of context equivariance, we argue that the assumption is unrealistic, i.e. , the teacher’s knowledge is imbalanced .

Original: Therefore, a constant sample weight for the KL losswould be a bottleneck.
Revised: Therefore, a constant sample weight for the KL loss would be a bottleneck.

Original: In this paper, we propose a simple yet effective method, Inverse Probability 
Revised: In this paper, we propose a simple yet effective method, Inverse Probability Weighting

Original: Weighting Distillation (IPWD), which compensates for the training samples that are under-weightedin the machine domain.
Revised: Distillation (IPWD), which compensates for the training samples that are under-weighted in the machine domain.

Original: For each training sample x , we first estimate its machine-domain propensityscore P ( x | machine ) by comparing class-aware and context-aware predictions.
Revised: For each training sample x , we first estimate its machine-domain propensity score P ( x | machine ) by comparing class-aware and context-aware predictions.

Original: A sample with a lowpropensity score would have a high confidence from class-aware predictions and a low confidencefrom context-aware predictions.
Revised: A sample with a low propensity score would have a high confidence from class-aware predictions and a low confidence from context-aware predictions.

Original: Then, IPWD assigns the inverse probability 1 /P ( x | machine ) asthe sample weight for the KL loss to highlight the under-represented samples.
Revised: Then, IPWD assigns the inverse probability 1 /P ( x | machine ) as the sample weight for the KL loss to highlight the under-represented samples.

Original: In this way, IPWDgenerates a pseudo-population [31, 21] to deal with the imbalanced knowledge.
Revised: In this way, IPWD generates a pseudo-population [37, 26] to deal with the imbalanced knowledge.

Original: We evaluate our proposed IPWD on two typical knowledge distillation settings: two-stage teacherstudent distillation and one-stage self-distillation.
Revised: We evaluate our proposed IPWD on two typical knowledge distillation settings: two-stage teacherstudent distillation and one-stage self-distillation.

Original: Experiments conducted on CIFAR-100 [26] and
Revised: Experiments conducted on CIFAR-100 [32] and

Original: ImageNet [10] demonstrate the effectiveness and generality of our IPWD.
Revised: ImageNet [10] demonstrate the effectiveness and generality of our IPWD.

Original: Our contributions are three-fold:
Revised: Our contributions are three-fold:

Original: • We formulate KD as a domain transfer problem and argue that the naïve IID assumption on machinedomain neglects the imbalanced knowledge due to transfer gap.
Revised: • We formulate KD as a domain transfer problem and argue that the naïve IID assumption on machine domain neglects the imbalanced knowledge due to transfer gap.

Original: • We propose Inverse Probability Weighting Distillation (IPWD) which compensate for the samplesthat are overlooked in the machine domain to tackle the imbalanced knowledge in transfer 
Revised: • We propose Inverse Probability Weighting Distillation (IPWD) which compensate for the samples that are overlooked in the machine domain to tackle the imbalanced knowledge in transfer gap.

Original: gap.• Experiments on CIFAR-100 and ImageNet for both two-stage distillation and one-stage selfdistillation show that the proper handling of the transfer gap is a promising direction in KD.
Revised: • Experiments on CIFAR-100 and ImageNet for both two-stage distillation and one-stage selfdistillation show that the proper handling of the transfer gap is a promising direction in KD.

Original:  Related Work
Revised:  Related Work

Original: Knowledge distillation (KD) was first introduced to transfer the knowledge from an effective butcumbersome model to a smaller and efficient model [18].
Revised: Knowledge distillation (KD) was first introduced to transfer the knowledge from an effective but cumbersome model to a smaller and efficient model [21].

Original: The knowledge can be formulated ineither output space [18, 23, 29, 64, 63, 35, 50, 69, 25] or representation space [45, 20, 65, 24, 41,16, 54, 7, 22].
Revised: The knowledge can be formulated in either output space [21, 28, 35, 78, 77, 43, 61, 85, 31] or representation space [54, 25, 79, 30, 50, 19, 66, 7, 27].

Original: KD has attracted a wide interest in theory, methodology, and applications [14].
Revised: KD has attracted a wide interest in theory, methodology, and applications [15].

Original: For applications, KD has shown its great potential in various areas, including but not limited toclassification [30, 44, 33, 19], detection [28, 49, 58], segmentation [15, 36, 32] for visual recognitiontasks, and visual question answering [38, 1, 39], video captioning [40, 68], and text-to-imagesynthesis [53] for vision-language tasks.
Revised: For applications, KD has shown its great potential in various areas, including but not limited to classification [36, 53, 39, 23], detection [34, 59, 70], segmentation [18, 44, 38] for visual recognition tasks, and visual question answering [46, 1, 48], video captioning [49, 84], and text-to-image synthesis [64] for vision-language tasks.

Original: Recent studies further discussed how and why KD works.
Revised: Recent studies further discussed how and why KD works.

Original: Specifically, Müller et al.
Revised: Specifically, Müller et al.

Original: [37] and Shen et al.
Revised: [45] and Shen et al.

Original: [48] empirically analyzed the effect of label smoothingon KD.
Revised: [58] empirically analyzed the effect of label smoothing on KD.

Original: Cho et al.
Revised: Cho et al.

Original: [8], Dong et al.
Revised: [8], Dong et al.

Original: [11], and Yuan et al.
Revised: [11], and Yuan et al.

Original: [63] pointed out that early stopping is a goodregularization for a better teacher.
Revised: [77] pointed out that early stopping is a good regularization for a better teacher.

Original: Yuan et al.
Revised: Yuan et al.

Original: [63] further found that a poorly trained teacher, evena model smaller than the student, can improve the performance of the student.
Revised: [77] further found that a poorly trained teacher, even a model smaller than the student, can improve the performance of the student.

Original: Besides, Memon etal.
Revised: Besides, Memon et al.

Original: [34] and Zhou et al.
Revised: [41] and Zhou et al.

Original: [69] proposed a bias-variance trade-off perspective for KD.
Revised: [85] proposed a bias-variance trade-off perspective for KD.

Original: In this paper, wepoint out that existing KD methods hold an underlying assumption that the IID training samples arealso IID in the machine domain, which overlooks the transfer gap.
Revised: In this paper, we point out that existing KD methods hold an underlying assumption that the IID training samples are also IID in the machine domain, which overlooks the transfer gap.

Original: Self-distillation is a special case of KD, which uses the student network itself as the teacher instead ofthe cumbersome model, i.e. , the teacher and student models have the same architecture [13, 67, 66, 22].
Revised: Self-distillation is a special case of KD, which uses the student network itself as the teacher instead of the cumbersome model, i.e. , the teacher and student models have the same architecture [13, 82, 81, 27].

Original: This process can be executed in iterations and produce a stronger ensemble model [13].
Revised: This process can be executed in iterations and produce a stronger ensemble model [13].

Original: Similar to
Revised: Similar to

Original: KD, traditional self-distillation follows a two-stage process: first pre-training a student model asthe teacher, and then distill the knowledge from the pre-trained model to a new student model.
Revised: KD, traditional self-distillation follows a two-stage process: first pre-training a student model as the teacher, and then distilling the knowledge from the pre-trained model to a new student model.

Original: Inorder to perform the teacher-student optimization in one generation, recent studies [61, 25] proposedone-stage self-distillation that adopts student models at earlier epochs as teacher models.
Revised: In order to perform the teacher-student optimization in one generation, recent studies [75, 31] proposed one-stage self-distillation that adopts student models at earlier epochs as teacher models.

Original: Theseone-stage self-distillation methods outperform vanilla students by large margins.
Revised: These one-stage self-distillation methods outperform vanilla students by large margins.

Original: In this paper, wealso evaluate the effectiveness of our IPWD as a plug-in in one-stage self-distillation.
Revised: In this paper, we also evaluate the effectiveness of our IPWD as a plug-in in one-stage self-distillation.

Original: Inverse Probability Weighting (IPW)
Revised: Inverse Probability Weighting (IPW)

Original: [46, 31, 21, 5], also known as inverse probability of treatmentweighting or inverse propensity weighting, was proposed to correct the selection bias when theobservations are non-IID.
Revised: [55, 37, 26, 5], also known as inverse probability of treatment weighting or inverse propensity weighting, was proposed to correct the selection bias when the observations are non-IID.

Original: IPW uses the inverse of the probability ( i.e. , propensity score) that theindividual would be assigned to the treatment group to reweight the samples.
Revised: IPW uses the inverse of the probability ( i.e. , propensity score) that the individual would be assigned to the treatment group to reweight the samples.

Original: Propensity-weightingtechniques have been widely applied and studied in many areas [47] such as causal inference [21],complete-case analysis [31], machine learning [9, 6, 51], and recommendation systems [47, 59, 3].
Revised: Propensity-weighting techniques have been widely applied and studied in many areas [57], such as causal inference [26], complete-case analysis [37], machine learning [9, 6, 62], and recommendation systems [57, 72, 3].

Original: In this paper, we view the distillation process as a domain transfer problem and adopt IPW todynamically assign the weight to each training sample for the distillation loss.
Revised: In this paper, we view the distillation process as a domain transfer problem and adopt IPW to dynamically assign the weight to each training sample for the distillation loss.

Original:  Background
Revised:  Analysis 3.1 Knowledge Distillation (KD)

Original: In this paper, we view knowledge distillation(KD) from a perspective of domain transfer, and takethe image classification task as the case study.
Revised: We view knowledge distillation from a perspective of domain transfer, and take the image classification task as the case study.

Original: Suppose that the training data D = {X , Y} = { ( x, y ) }contains x as the input ( e.g. , image) and y ∈ R C as its ground-truth annotation ( e.g. , one-hotlabel), where C denotes the number of classes.
Revised: Suppose that the training data D = {X , Y} = { ( x, y ) } contains x as the input ( e.g. , image) and y ∈ R C as its ground-truth annotation ( e.g. , one-hot label), where C denotes the number of classes.

Original: A standard solution to train the classifier θ uses thecross-entropy loss as the objective:
Revised: A standard solution to train the classifier θ uses the cross-entropy loss as the objective:

Original: [Equation]
Revised: [Equation]

Original: where ℓ cls ( x, y ) =
Revised: where ℓ cls ( x, y ) =

Original: H ( y s , y ) is the classification loss for sample x , H ( p, q ) 
Revised: H ( y s , y ) is the classification loss for sample x , H ( p, q ) =

Original: =
Revised: [Equation]

Original: (cid:80) Ci =1 − q i log p idenotes the cross entropy between p and q , y s = f
Revised: Ci =1 − q i log p i denotes the cross entropy between p and q , y s = f

Original: ( x ; θ ) denotes the model’s output probability given x , i.e. , y sk = 
Revised: ( x ; θ ) denotes the model’s output probability given x , i.e. , y sk = [Equation] z

Original: [Equation]
Revised: [Equation]

Original: =1 exp ( z si ) , where z s is the output logits of the model.
Revised: Ci = [Equation] exp ( z si ) , where z s is the output logits of the model.

Original: The hard targets providecontext-invariant class-specific information from the human domain.
Revised: The hard targets provide context-invariant class-specific information from the human domain.

Original: An assumption held behind
Revised: An assumption held behind

Original: Eq.
Revised: Eq.

Original: (1) is that the samples are independent and identically distributed (IID) in the training and test set.
Revised: (1) is that the samples are independent and identically distributed (IID) in the training and test set.

Original: KD adopts a teacher model θ t to generate soft targets as extra supervisions, i.e. , context-equivariantinformation.
Revised: KD adopts a teacher model θ t to generate soft targets as extra supervisions, i.e. , context-equivariant information.

Original: To formulate the machine domain, traditional KD methods commonly use the training set
Revised: To formulate the machine domain, traditional KD methods commonly use the training set

Original: D to construct the transfer set D t using the same copy of X , i.e. , D t = { ( x, y t ) } where y t = f
Revised: D to construct the transfer set D t using the same copy of X , i.e. , D t = { ( x, y t ) } where y t = f

Original: ( x ; θ t )and x ∈ X .
Revised: ( x ; θ t ) and x ∈ X .

Original: Traditional KD approaches use the KL divergence [27] loss for knowledge transfer:
Revised: Traditional KD approaches use the KL divergence [33] loss for knowledge transfer:

Original: [Equation]
Revised: [Equation]

Original: where ℓ dist ( x, y t ; θ ) = τ 2 · [ H ( y sτ , y tτ ) − H
Revised: where ℓ dist ( x, y t ; θ ) = τ 2 · [ H ( y sτ , y tτ ) − H

Original: ( y tτ , y tτ )] denotes the distillation loss for sample x .
Revised: ( y tτ , y tτ )] denotes the distillation loss for sample x .

Original: Normally, the outputs of the student and teacher are softened using a temperature τ , i.e. , y sτ,k =exp( z sk /τ )
Revised: Normally, the outputs of the student and teacher are softened using a temperature τ , i.e. , y sτ,k = exp( z sk /τ )

Original: (cid:80) Ci =1 exp ( z [Equation] ) and y tτ,k = 
Revised: (cid:80) Ci =1 exp ( z si /τ ) and y tτ,k = exp ( z tk /τ ) (cid:80)

Revised: Ci

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ti [Equation]
Revised: ti /τ ) .

Original: The overall objective combines L cls and L dist as:
Revised: The overall objective combines L cls and L dist as:

Original: [Equation]
Revised: [Equation]

Original: where α and β are the hyper-parameters.
Revised: where α and β are the hyper-parameters.

Original: The underlying assumption of traditional KD behind Eq.
Revised: The underlying assumption of traditional KD behind Eq.

Original: (2)is that the transfer set D t is an unbiased approximation of the machine domain.
Revised: (2) is that the transfer set D t is an unbiased approximation of the machine domain.

Original: However, the observedlong-tailed and temperature-sensitive distributions of teacher’s predictions in Figure 1 rationallychallenge this assumption.
Revised: However, the observed long-tailed and temperature-sensitive distributions of teacher’s predictions in Figure 1 rationally challenge this assumption.

Original: As a result, samples with lower P ( x | machine ) are under-representedduring the distillation process, which affects the unbiasedness of knowledge transfer.
Revised: As a result, samples with lower P ( x | machine ) are under-represented during the distillation process, which affects the unbiasedness of knowledge transfer.

Original: This analysisindicates that Eq.
Revised: This analysis indicates that Eq.

Original: (2) is not optimal to utilize the teacher’s imbalanced knowledge.
Revised: (2) is not optimal to utilize the teacher’s imbalanced knowledge.

Original: Method

Original: We propose a simple yet effective method, Inverse Probability Weighting Distillation (IPWD), torespect the transfer gap and imbalance knowledge in KD.

Original: In this section, we first introduce the overallframework of IPWD, then present the implementation details.

Original: 4.1 Transfer Gap in KD
Revised: 3.2 Transfer Gap in KD

Original: 𝑿

Original: [Equation]
Revised: [Equation]

Original: Wefirst interpret the transfer gap and its confounding effectfrom the perspective of causal inference.
Revised: We interpret the transfer gap and its confounding effect from the perspective of causal inference.

Original: Figure 2 illustratesthe causal relations between the image X , training data D ={ ( x, y ) } , teacher’s parameters θ t and teacher’s output Y t in KD.
Revised: Figure 2 illustrates the causal relations between the image X , training data D = { ( x, y ) } , teacher’s parameters θ t and teacher’s output Y t in KD.

Original: Overall, D and θ t jointly act as the confounder of X and Y t in the transfer set.
Revised: Overall, D and θ t jointly act as the confounder of X and Y t in the transfer set.

Original: First, the training set D and transfer setof teacher model D t = { ( x, y t ) } share the same image set, and X = x is sampled from the imageset of D , i.e., D serves the cause of X .
Revised: First, the training set D and transfer set of teacher model D t = { ( x, y t ) } share the same image set, and X = x is sampled from the image set of D , i.e., D serves the cause of X .

Original: Second, the teacher θ t is trained on D , and y t is calculatedbased on θ t and x , i.e., y t
Revised: Second, the teacher θ t is trained on D , and y t is calculated based on θ t and x , i.e., y t

Original: = f
Revised: = f

Original: ( x ; θ t ) .
Revised: ( x ; θ t ) .

Original: Therefore, X and θ t are the cause of Y t .
Revised: Therefore, X and θ t are the cause of Y t .

Original: Note that the transferset is constructed based on the images on D and teacher model θ t .
Revised: Note that the transfer set is constructed based on the images on D and teacher model θ t .

Original: Therefore, we regard the transferset D t , the joint of D and θ t , as the confounder of x and y t .
Revised: Therefore, we regard the transfer set D t , the joint of D and θ t , as the confounder of X and Y t .

Original: Although D is balanced when considering the context-invariant class-specific information, the contextinformation (e.g., attributes) is overlooked, which makes the D imbalanced on context.
Revised: Although D is balanced when considering the context-invariant class-specific information, the context information (e.g., attributes) is overlooked, which makes the D imbalanced on context.

Original: As shownin Figure 1, such imbalanced context leads to an imbalanced transfer set D t and further affects thedistillation performance of teacher’s knowledge.
Revised: As shown in Figure 1, such imbalanced context leads to an imbalanced transfer set D t and further affects the distillation performance of teacher’s knowledge.

Original: To overcome such confounding effect, a commonly used technique is intervention via P ( y t | do ( x )) instead of P ( y t | x ) , which is formulated as P ( y t | do ( x ))
Revised: To overcome such confounding effect, a commonly used technique is intervention via P ( y t | do ( x )) instead of P ( y t | x ) , which is formulated as P ( y t | do ( x ))

Original: = (cid:80) D t P ( y | x, D t ) P ( D t )
Revised: = (cid:80) D t P ( y | x, D t ) P ( D t )

Original: = (cid:80) D t P ( x,y t , D t ) P ( x |D t ) .
Revised: = (cid:80) D t P ( x,y t , D t ) P ( x |D t ) .

Original: This transformation suggests that we can use the inverse of propensity score P ( x |D t )  as sampleweight to implement the intervention and overcome the confounding effect.
Revised: This transformation suggests that we can use the inverse of propensity score, 1 /P ( x |D t ) , as sample weight to implement the intervention and overcome the confounding effect.

Original: Thanks to the causalitybased theory [46, 5], we can use the Inverse Probability Weighting (IPW) technique to overcome theconfounding effect brought by the transfer gap.
Revised: Thanks to the causalitybased theory [55, 5], we can use the Inverse Probability Weighting (IPW) technique to overcome the confounding effect brought by the transfer gap.

Original: I npu
Revised: I npu

Original: t s t u d e n t ba c k b on e f e a t u r e
Revised: t s t u d e n t ba c k b on e f e a t u r e

Original: L k d
Revised: L k d

Original: D i s t ill h e ad y s
Revised: D i s t ill h e ad y s

Original: I npu
Revised: I npu

Original: t s t u d e n t ba c k b on e f e a t u r e
Revised: t s t u d e n t ba c k b on e f e a t u r e

Original: C L S h e ad
Revised: C L S h e ad

Original: D i s t ill h e ad y c l s y s
Revised: D i s t ill h e ad y c l s y s

Original: L c l s
Revised: L c l s

Original: L i p w d w
Revised: L i p w d w

Original: Eq.
Revised: Eq.

Original: [Figure] [Figure]
Revised: Eq.

Original: Eq.
Revised: Eq.

Original: [Figure]
Revised: [Figure]

Revised:  Method

Revised: We propose a simple yet effective method, Inverse Probability Weighting Distillation (IPWD), to respect the transfer gap and imbalance knowledge in KD.

Revised: In this section, we first introduce the overall framework of IPWD, then present the implementation details.

Original: 4.2 Inverse Probability Weighting for KD
Revised: 4.1 Inverse Probability Weighting for KD

Original: As analyzed in Section 3, the IID training samples in the human domain are no longer IID in themachine domain.
Revised: As analyzed in Section 3, the IID training samples in the human domain are no longer IID in the machine domain.

Original: Simply assuming the training set as the perfect transfer set may lead to the selectionbias: samples that match “head” knowledge are over-represented and easy to be observed, whilesamples that match “tail” knowledge are under-represented and hard to be observed.
Revised: Simply assuming the training set as the perfect transfer set may lead to the selection bias: samples that match “head” knowledge are over-represented and easy to be observed, while samples that match “tail” knowledge are under-represented and hard to be observed.

Original: This wouldsuppress the transfer of “tail” knowledge.
Revised: This would suppress the transfer of “tail” knowledge.

Original: The analysis from the perspective of causal inference in
Revised: The analysis from the perspective of causal inference in

Original: Section 4.1 suggests that we can use Inverse Probability Weighting (IPW) for debiased distillation.
Revised: Section 3.2 suggests that we can use Inverse Probability Weighting (IPW) for debiased distillation.

Original: In short, IPW generates a pseudo-population where under-represented samples are assigned withlarge weights and over-represented samples are assigned with small weights.
Revised: In short, IPW generates a pseudo-population where under-represented samples are assigned with large weights and over-represented samples are assigned with small weights.

Original: The weight for samplex is determined as the inverse of its probability, also known as propensity score, to the domaind ∈ { human , machine } , i.e. , w x | d = 1 /p ( x | d ) .
Revised: The weight for sample x is determined as the inverse of its probability, also known as propensity score, to the domain d ∈ { human , machine } , i.e. , w x | d = 1 /p ( x | d ) .

Original: We adopt IPW to KD and obtain the followingobjective for sample x :
Revised: We adopt IPW to KD and obtain the following objective for sample x :

Original: [Equation]
Revised: ℓ ( x ; θ ) = (cid:88)

Original: [Equation]
Revised: [Equation]

Original: 4.3 Implementation
Revised: 4.2 Implementation

Original: Since the training and test data are normally IID in the human domain, we safely and rationallyuse the empirical risk.
Revised: Since the training and test data are normally IID in the human domain, we safely and rationally use the empirical risk.

Original: Therefore, we assign a constant weight to each sample when calculating theclassification loss.
Revised: Therefore, we assign a constant weight to each sample when calculating the classification loss.

Original: For sure, the training and test samples can be  non-IID, e.g. , long-tailed recognitiontasks, which is out of the scope of this paper.
Revised: For sure, the training and test samples can be both non-IID, e.g. , long-tailed recognition tasks, which is out of the scope of this paper.

Original: As analyzed in Section 1, the assumption held by traditional KD, i.e. , both P ( X | human ) and
Revised: As analyzed in Section 1, the assumption held by traditional KD, i.e. , both P ( X | human ) and

Original: P ( X | machine ) are IID, is unrealistic in practice.
Revised: P ( X | machine ) are IID, is unrealistic in practice.

Original: Therefore, we should consider the propensity  P ( x | machine ) as a sample-specific value for the distillation loss to improve the generalization.
Revised: Therefore, we should consider the propensity score P ( x | machine ) as a sample-specific value for the distillation loss to improve the generalization.

Original: Traditional IPW estimates the propensity  using logistic regression, i.e. , ˆ P ( x | machine )
Revised: Traditional IPW estimates the propensity score using logistic regression, i.e. , ˆ P ( x | machine )

Original: = 1 / (1 +exp( − z x ))  where z x is the logit for x .
Revised: = 1 / (1 + exp( − z x )) , where z x is the logit for x .

Original: Since the ground-truth annotation of P ( x | machine ) is notavailable, it is not practicable to directly train the regression model in a fully-supervised manner.
Revised: Since the ground-truth annotation of P ( x | machine ) is not available, it is not practicable to directly train the regression model in a fully-supervised manner.

Original: Therefore, we estimate the propensity  in an unsupervised way.
Revised: Therefore, we estimate the propensity score in an unsupervised way.

Original: Recall that the samples with high propensity are over-represented in the transfer set.
Revised: Recall that the samples with high propensity are over-represented in the transfer set.

Original: As a result, thestudent model would learn less from the under-represented group via distillation.
Revised: As a result, the student model would learn less from the under-represented group via distillation.

Original: Therefore, we use aclassification-trained (CLS-trained) classifier for the human domain as reference, and assume that
Revised: Therefore, we use a classification-trained (CLS-trained) classifier for the human domain as reference, and assume that

Original: a
Revised: a

Original: KD-trained classifier for the machine domain is more confident for the over-represented group thanthe CLS-trained classifier.
Revised: KD-trained classifier for the machine domain is more confident for the over-represented group than the CLS-trained classifier.

Original: We compare the outputs of two classifiers to identify whether a sampleis under-represented in the machine domain.
Revised: We compare the outputs of two classifiers to identify whether a sample is under-represented in the machine domain.

Original: Suppose that the KD-trained output is y kd and the
Revised: Suppose that the KD-trained output is y kd and the

Original: CLS-trained output is y cls .
Revised: CLS-trained output is y cls .

Original: The assumption implies that the logit z  is negatively correlated with 
Revised: The assumption implies that the logit z x is negatively correlated with [Figure]

Original: H ( y kd , y ) and positively correlated with H ( y cls , y ) , where H ( · , · ) is the cross-entropy defined in
Revised: H ( y kd , y ) and positively correlated with H ( y cls , y ) , where H ( · , · ) is the cross-entropy defined in

Original: Section 3.
Revised: Section 3.

Original: Considering the range of logit, we estimate z x as z x 
Revised: Considering the range of logit, we estimate z x as z x = log

Original: [Equation]
Revised: H ( y cls ,y ) H ( y kd ,y ) .

Original: Figure 3 illustrates the comparison between traditional KD and our IPWD.
Revised: Figure 3 illustrates the comparison between traditional KD and our IPWD.

Original: We take the KD-trainedstudent’s output y s as y kd , and train an extra classifier head ( i.e. , “CLS head” in Figure 3(b)) tocalculate y cls , which is optimized using the cross-entropy loss L cls with the ground-truth labels.
Revised: We take the KD-trained student’s output y s as y kd , and train an extra classifier head ( i.e. , “CLS head” in Figure 3(b)) to calculate y cls , which is optimized using the cross-entropy loss L cls with the ground-truth labels.

Original: Asshown in Table 5, we empirically find that directly using y s and y cls would lead to a high variance.[Figure]
Revised: As shown in Table 5, we empirically found that directly using y s and y cls would lead to a high variance.

Original: ) honeycomb apiary bee hen cock blackgrouse canoe paddle snorkel

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: # vulture

Original: [Figure]

Original: Africanwilddog hyena dhole racecar sportscar convertible ant groundbeetle tigerbeetle

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure] [Figure] [Figure] For example, a wrongly classified sample may have a extremely large loss and heavily suppress thedistillation of other samples.
Revised: For example, a wrongly classified sample may have a extremely large loss and heavily suppress the distillation of other samples.

Original: Therefore, we normalize the logits by dividing them by the standarddeviation σ s and σ cls , i.e. , (cid:101)
Revised: Therefore, we normalize the logits by dividing them by the standard deviation σ s and σ cls , i.e. , (cid:101)

Original: y s =exp
Revised: y s =

Original: [Equation]

Original:  (cid:80) Ci =1 exp ( z si /σ s ) and (cid:101) y cls = exp [Equation]
Revised: exp( z sk /σ s ) (cid:80) Ci =1 exp ( z si /σ s ) and (cid:101) y cls = exp ( z clsk /σ cls ) (cid:80)

Original: clsk

Original: [Equation]

Original: cls ) (cid:80)

Original: Ci =1 exp ( z clsi /σ cls ) .
Revised: Ci =1 exp ( z clsi /σ cls ) .

Original: In this way, theoutputs are at the same scale with a standard deviation equal to 1, which helps to reduce the variance.
Revised: In this way, the outputs are at the same scale with a standard deviation equal to 1, which helps to reduce the variance.

Original: We finally take (cid:101)
Revised: We finally take (cid:101)

Original: y s as y kd and (cid:101)
Revised: y s as y kd and (cid:101)

Original: y cls as y cls .
Revised: y cls as y cls .

Original: Combining z x into propensity, we have:
Revised: Combining z x into the propensity score, we have:

Original: [Equation]
Revised: [Equation]

Original: 5) and the estimated weight ˆ w x for sample x is ˆ w x=
Revised: and the estimated weight ˆ w x for sample x is ˆ [Equation] x

Original: [Equation]
Revised: [Equation]

Original: Figure 4 illustratessome examples of training samples and their assigned weights.
Revised: Figure 4 illustrates some examples of training samples and their assigned weights.

Original: Under-represented samples, forwhich the KD-trained classifier is less confident than the CLS-trained classifier, are assigned with alarge weight (Figure 4(a)).
Revised: Under-represented samples, for which the KD-trained classifier is less confident than the CLS-trained classifier, are assigned with a large weight (Figure 4(a)).

Original: Over-represented samples, for which the KD-trained classifier is moreconfident than the CLS-trained classifier, are assigned with a small weight (Figure 4(c)).
Revised: Over-represented samples, for which the KD-trained classifier is more confident than the CLS-trained classifier, are assigned with a small weight (Figure 4(c)).

Original: Samplesfor which the two classifiers behave similarly are assigned with a balanced weight (Figure 4(b)).
Revised: Samples for which the two classifiers behave similarly are assigned with a balanced weight (Figure 4(b)).

Original: Theweighted distillation loss is formulated as:
Revised: The weighted distillation loss is formulated as:

Original: [Equation]
Revised: [Equation]

Original: Our final IPWD objective is formulated as:
Revised: Our final IPWD objective is formulated as:

Original: [Equation]
Revised: [Equation]

Original: where α is a trade-off hyper-parameter between classification and distillation.
Revised: where α is a trade-off hyper-parameter between classification and distillation.

Original: Limitations and negative societal impacts. .
Revised: Limitations and negative societal impacts. .

Original: Aswe introduced in Section 4.3, we estimate thepropensity by comparing the heads of the student model.
Revised: As introduced in Section 4.2, we estimate the propensity score by comparing the heads of the student model.

Original: Therefore, theaccuracy of estimation relieson the quality of the student model.
Revised: Therefore, the estimation relies on the quality of the student model.

Original: A pooror small student may notbe able to correctly estimate thepropensity, which may further suppress the effectiveness of IPWD.
Revised: A poor student may not correctly estimate the propensity, which may further suppress the effectiveness of IPWD.

Original: Also, we hold an assumption thatthe training and test samples are IID in the human domain, which may not be valid for long-tailedtasks.
Revised: Also, we hold an assumption that the training and test samples are IID in the human domain, which may not be valid for long-tailed tasks.

Original: Besides, we found that our IPWD does not work with feature-based distillation methods (see

Original: Appendix for more results).

Original: To the best of our knowledge, as our work is purely an algorithm forknowledge distillation, we haven’t found any negative societal impact. 
Revised: To the best of our knowledge, as our work is purely an algorithm for knowledge distillation, we haven’t found any negative societal impact. [Table] [Table] [Table]

Revised: [Table]

Original: Experiments
Revised: [Table] [Table] [Table] [Table]

Revised: NST [25]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: KD [21] 73.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: CRD [66] 74.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: WSLD ∗ [85] 73.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: IPWD

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original:  We take the image classification task as a case study to evaluate the effectiveness and generalizabilityof our IPWD.
Revised: [Table] [Table]  Experiments We take the image classification task as a case study to evaluate the effectiveness and generalizability of our IPWD.

Original: Following previous works [54, 69, 25], we conduct experiments with two settings,two-stage distillation and one-stage self-distillation.[Table]
Revised: Following previous works [66, 85, 31], we conduct experiments with two settings, two-stage distillation and one-stage self-distillation.

Original: 5.1 Datasets and Settings
Revised: 5.1 Datasets and Settings

Original: Datasets .
Revised: Datasets .

Original: We conducted experiments on CIFAR-100 [26] and ImageNet [10].
Revised: We conducted experiments on CIFAR-100 [32] and ImageNet [10].

Original: CIFAR-100 contains50K images in the training set and 10K images in the test set from 100 classes.
Revised: CIFAR-100 contains 50K images in the training set and 10K images in the test set from 100 classes.

Original: ImageNet provides1.2M images in the training set and 50K images in the validation set from 1K classes.
Revised: ImageNet provides 1.2M images in the training set and 50K images in the validation set from 1K classes.

Original: Settings .
Revised: Settings .

Original: We considered two settings for knowledge distillation, two-stage distillation and one-stageself-distillation.

Original: Two-stage distillation is the conventional setting that pre-trains a teacher model atthe first stage and transfers the knowledge to a student model at the second stage.
Revised: Two-stage distillation is the conventional setting that pre-trains a teacher model at the first stage and transfers the knowledge to a student model at the second stage.

Original: Commonly, theteacher is a larger model, and the student is a smaller model.
Revised: Commonly, the teacher is a larger model, and the student is a smaller model.

Original: For self-distillation, the teacher andstudent have the same architecture.
Revised: For self-distillation, the teacher and student have the same architecture.

Original: One-stage self-distillation aims to complete the teacher-studentoptimization simultaneously [61, 25], i.e. , the pre-training and transfer processes are reduced to one.
Revised: One-stage self-distillation aims to complete the teacher-student optimization simultaneously [75, 31], i.e. , the pre-training and transfer processes are reduced to one.

Original: 5.2 Two-stage Distillation
Revised: 5.2 Two-stage Distillation

Original: Baseline methods .
Revised: Baseline methods .

Original: For two-stage distillation, following Tian et al.
Revised: For two-stage distillation, following Tian et al.

Original: [54] and Zhou et al.
Revised: [66] and Zhou et al.

Original: [69], weconsidered the following methods as baselines: KD [18], FitNet [45], AT [65], SP [56], CC [43],
Revised: [85], we considered the following methods as baselines: KD [21], FitNet [54], AT [79], SP [68], CC [52],

Original: VID [2], RKD [41], PKT [42], FSP [62], AB [17], FT [24], NST [20], CRD [54], SSKD [60], and
Revised: VID [2], RKD [50], PKT [51], FSP [76], AB [20], FT [30], NST [25], CRD [66], SSKD [74], and

Original: WSLD [69].
Revised: WSLD [85].

Original: In particular, WSLD [69] is the most related work to us, which proposed a bias-variancetrade-off perspective for KD and also assigns different weights to each training sample.
Revised: In particular, WSLD [85] is the most related work to us, which proposed a bias-variance trade-off perspective for KD and also assigns different weights to each training sample.

Original: Similarly, theweight is positive related to the cross-entropy loss of student’s output.
Revised: Similarly, the weight is positive related to the cross-entropy loss of student’s output.

Original: The main differences betweenour IPWD and WSLD are as follows.
Revised: The main differences between our IPWD and WSLD are as follows.

Original: First, our formulation of samples weights is theoreticallyguaranteed by the causal theory behind Inverse Probability Weighting (IPW) [46, 31, 21, 5].
Revised: First, our formulation of samples weights is theoretically guaranteed by the causal theory behind Inverse Probability Weighting (IPW) [55, 37, 26, 5].

Original: Second,
Revised: Second,

Original: WSLD estimates the sample weight using both the student model and the teacher model.
Revised: WSLD estimates the sample weight using both the student model and the teacher model.

Original: As acomparison, we use the student model with two different classifier heads to guarantee that thecapacities of the compared models are close.
Revised: As a comparison, we use the student model with two different classifier heads to guarantee that the capacities of the compared models are close.

Original: Implementation .
Revised: Implementation .

Original: For experiments on CIFAR-100, we followed CRD [54] based on the open-sourcedcode.
Revised: For experiments on CIFAR-100, we followed CRD [66] based on the open-sourced code.

Original: We set the trade-off hyper-parameter α = 5 in Eq.
Revised: We set the trade-off hyper-parameter α = 5 in Eq.

Original: (7) and the temperature τ = 10 .
Revised: (7) and the temperature τ = 10 .

Original: Othertraining details were the same as CRD [54] and provided in the appendix.
Revised: Other training details were the same as CRD [66] and provided in the appendix.

Original: For ImageNet, we followed 
Revised: For ImageNet, we followed [Table]

Original: Zhou et al.
Revised: Zhou et al.

Original: [69] to conduct experiments based on their open-sourced code.
Revised: [85] to conduct experiments based on their open-sourced code.

Original: We used the same hyperparameters as WSLD [69], i.e. , α as 2.5 and τ as 2.
Revised: We used the same hyper parameters as WSLD [85], i.e. , α as 2.5 and τ as 2.

Original: Comparison with baseline methods .
Revised: Comparison with baseline methods .

Original: Table 2 shows the results of student models on CIFAR-100with twelve teacher-student architectures, which can be grouped into same architecture style anddifferent architecture style.
Revised: Table 2 shows the results of student models on CIFAR-100 with different teacher-student architectures, which can be grouped into same architecture style and different architecture style.

Original: Note that the results of WSLD reported in [69] used a different pre-trainedteacher model.
Revised: Note that the results of WSLD reported in [85] used a different pre-trained teacher model.

Original: Since some training techniques like label smoothing [48] and early-stoping [8, 11, 63]may improve the distillation performance, we reimplemented WSLD using the same teacher model[Table] for a fair comparison.
Revised: Since some training techniques like early-stopping [8, 11, 77] may improve the distillation performance, we reimplemented WSLD using the same teacher model for a fair comparison.

Original: Overall, our IPWD outperforms KD by large margins and outperforms otherbaseline methods on most of the architectures, which demonstrates the effectiveness of our IPWD.
Revised: Overall, our IPWD outperforms KD by large margins and outperforms other baseline methods on most of the architectures, which demonstrates the effectiveness of our IPWD.

Original: Inparticular, the improvement with the same architecture style is smaller than the different architecturestyle.
Revised: In particular, the improvement with the same architecture style is smaller than the different style.

Original: The reason is that the different architecture style reflects the huge gap between the humandomain and machine domain.
Revised: The reason is that the different architecture style reflects the bigger gap between the human domain and machine domain.

Original: Since our IPWD weights the training samples to address the non-IIDproblem and respect the transfer gap, IPWD successfully outperforms KD and other state-of-the-artmethods by large margins when the  gap is significant.
Revised: Since our IPWD weights the training samples to address the non-IID problem, IPWD successfully outperforms KD and other state-of-the-art methods by large margins when the transfer gap is significant.

Original: [Table]
Revised: [Table]

Original: Note that SSKD [60] achieves higher performance because of (1) a better teachermodel, and (2) data augmentation forstructured knowledge distillation.
Revised: Note that SSKD [74] achieves higher performance because of (1) a better teacher model, and (2) data augmentation for structured knowledge distillation.

Original: Wefurther apply IPWD to SSKD as a plugin by weighting the logit distillation objective and keeping the structured knowledge distillation terms unchanged.
Revised: We further apply IPWD to SSKD as a plugin by weighting the logit distillation objective and keeping the structured knowledge distillation terms unchanged.

Original: Table 3 shows that our IPWD can furtherconsistently improve SSKD by 0.5~1.0%for different architectures.
Revised: Table 3 shows that our IPWD can consistently improve SSKD by 0.5~1.0% for different architectures.

Original: These resultsindicate that our IPWD is a good complementary to distillation methods thatare not limited to logit knowledge.
Revised: These results indicate that our IPWD is a good complementary to distillation methods.

Original: Table 4 further shows the comparisonon ImageNet.
Revised: Table 4 further shows the comparison on ImageNet.

Original: Following CRD [54]and WSLD [69], we used two teacherstudent architectures as the representatives.
Revised: Following CRD [66] and WSLD [85], we used two teacherstudent architectures as the representatives.

Original: For the same architecture style,our IPWD improves KD by 1.21%, andachieves competitive performance compared to WSLD.
Revised: For the same architecture style, our IPWD improves KD by 1.21%, and achieves competitive performance compared to WSLD.

Original: For the different architecture style, the improvement of WSLDover KD drops from 1.37% to 1.03%.
Revised: For the different architecture style, the improvement of WSLD over KD drops from 1.37% to 1.03%.

Original: As a comparison, our IPWD improves KD by 2.16%, andoutperforms WSLD by 1.13%.
Revised: As a comparison, our IPWD improves KD by 2.16%, and outperforms WSLD by 1.13%.

Original: This improvement on the large-scale dataset further demonstratesthe effectiveness of our IPWD to bridge the transfer gap when the student and teacher model havedifferent architecture styles, which is more practical in real-world applications.
Revised: This improvement on the large-scale dataset further demonstrates the effectiveness of our IPWD to bridge the transfer gap when the student and teacher model have different architecture styles, which is more practical in real-world applications.

Original: Ablation study: technical designs .
Revised: Ablation study: technical designs .

Original: As introduced in Section 4.3, we used an extra classifier head toproduce CE-aware output, and normalized the logits to reduce the variance for propensity estimation.
Revised: As introduced in Section 4.2, we used an extra classifier head to produce CLS-trained output, and normalized the logits to reduce the variance for propensity estimation.

Original: Note that WSLD [69] uses the teacher model to estimate the sample weight, and the teacher modelis also trained with the cross-entropy(CE) loss.
Revised: Note that WSLD [85] uses the teacher model to estimate the sample weight, and the teacher model is also trained with the cross-entropy loss.

Original: Therefore, we considered an alternative that whichreplaces the classification head with the teacher model to produce the classification-aware  output.
Revised: Therefore, we considered an alternative that which replaces the classification head with the teacher model to produce the classification-aware [Table] [Table] output.

Original: To evaluate the contribution of logits normalization, we considered an alternative that the logitsare not normalized by the standard deviation.
Revised: To evaluate the contribution of logits normalization [Table] we considered an alternative that the logits are not normalized by the standard deviation.

Original: Results in Table 5 verify the contribution of eachdesign.
Revised: Results in Table 5 verify the contribution of each design.

Original: Without the classification head and logit normalization, the training is hard to converge or theperformance is much worse.
Revised: Without the classification head and logit normalization, the training is hard to converge or the performance is much worse.

Original: As a comparison, either the classification head or logit normalizationhelps with stable training.
Revised: As a comparison, either the classification head or logit normalization helps with stable training.

Original: Besides, a combination of both further improves the performance andachieves the best results.
Revised: Besides, a combination of both further improves the performance and achieves the best results.

Original: The crash of training is due to the high variance of sample weights.
Revised: The crash of training is due to the high variance of sample weights.

Original: Since[Table] [Table] the teacher model is well pre-trained and has more parameters, it has a larger capacity than the studentmodel.
Revised: Since the teacher model is well pre-trained and has more parameters, it has a larger capacity than the student model.

Original: Differently, an extra head with a shared backbone guarantees a similar capacity.
Revised: Differently, an extra head with a shared backbone guarantees a similar capacity.

Original: Also, thenormalization will avoid an extremely large or small CE loss, which further reduces the variance. 
Revised: Also, the normalization will avoid an extremely large or small CE loss, which further reduces the variance. Teacher trained with label smoothing.

Revised: Recent works [45, 58] observed that KD performs poorly with label smoothing.

Revised: Similar to KD, the performance of IPWD drops when the teacher model is trained with label smoothing, but still outperforms KD.

Revised: However, we found that the improvement of

Original: 5.3 One-stage Self-Distillation
Revised: IPWD compared to KD also decreases with label smoothing. For example, on CIFAR-100, given ResNet50 as teacher and MobileNetV2 as student, IPWD outperforms KD by 1.12% (69.67% vs. 68.55%) without label smoothing, but the improvements drops to 0.56% (66.79% vs. 66.23%) with label smoothing.

Revised: Given resnet32x4 as teacher and ShuffleNetV1 as student, IPWD outperforms

Revised: KD by 1.52% (75.79% vs. 74.27%) without label smoothing, but the improvement drops to 0.53% (73.27% vs. 72.74%) with label smoothing.

Revised: We observed that teacher trained with label smoothing produces more balanced predictions compared to teacher trained without label smoothing.

Revised: Therefore, the results are consistent with our hypothesis that IPWD helps to bridge the transfer gap especially when the context information of teacher is imbalanced.

Revised: 5.3 One-stage Self-Distillation

Original: Baseline methods and metrics .
Revised: Baseline methods and metrics .

Original: For one-stage self-distillation, we apply our method to the stateof-the-art PS-KD [25] method as a plug-in, and consider label smoothing (LS) method and twoself-distillation methods, CS-KD [64] and TF-KD [63], as baselines.
Revised: For one-stage self-distillation, we apply our method to the stateof-the-art PS-KD [31] method as a plug-in, and consider label smoothing (LS) method and two self-distillation methods, CS-KD [78] and TF-KD [77], as baselines.

Original: PS-KD proposed a one-stageframework that progressively distills the knowledge of a model itself to soften the one-hot supervisionsas regularization.
Revised: PS-KD proposed a one-stage framework that progressively distills the knowledge of a model itself to soften the one-hot supervisions as regularization.

Original: The knowledge is transferred using a conventional distillation loss.
Revised: The knowledge is transferred using a conventional distillation loss.

Original: As for metrics,besides top-1 and top-5 accuracy, we follow Kim et al.
Revised: As for metrics, besides top-1 and top-5 accuracy, we follow Kim et al.

Original: [25] to report expected calibration error (ECE,%) and the area under the risk-coverage curve (AURC, × 10 3 ).
Revised: [31] to report expected calibration error (ECE, %) and the area under the risk-coverage curve (AURC, × 10 3 ).

Original: A low ECE indicates well-calibratedpredictions, and a low AURC represents the well-separation of correct and incorrect predictions. 
Revised: A low ECE indicates well-calibrated predictions, and a low AURC represents the well-separation of correct and incorrect predictions. Implementation .

Revised: We follow all the training details of PS-KD for a fair comparison.

Revised: Specifically, the architectures we considered are ResNet-18 [17], ResNet-101 [16], ResNeXt-29 [73] (cardinality=8, width=64), and DenseNet-121 [24] (growth rate=32).

Revised: During training, PS-KD gradually determine how much the student learns from the teacher’s knowledge.

Original: Comparison with baseline methods .
Revised: The formulation is: [Equation] where the trade-off parameter α t = α T × t/T , T is the number of total epochs ( e.g. , 300), t is the current epoch, and α T is a hyperparameter. Compared to Eq.

Revised: (8), our IPWD applied on PS-KD is formulated as

Revised: [Equation]

Revised: Since both the student and teacher models are poor at early epochs, the weight estimation is not accurate at early epochs, which may lead to a worse self-teacher.

Revised: Therefore, we apply IPWD at the last 75 epochs over the total 300 epochs.

Original:  Table 6 shows the results of one-stage self-distillation methodsover four architectures.
Revised: Comparison with baseline methods . Table 6 shows the results of one-stage self-distillation methods over four architectures.

Original: Our IPWD can effectively and constantly improve the top-1 accuracy of
Revised: Our IPWD can effectively and constantly improve the top-1 accuracy of

Original: PS-KD by 0.33% ∼ 0.82% with different architectures.
Revised: PS-KD by 0.33% ∼ 0.82% with different architectures.

Original: Besides, our IPWD siginifantly lowers the
Revised: Besides, our IPWD siginifantly lowers the

Original: ECE and AURC of PS-KD.
Revised: ECE and AURC of PS-KD.

Original: These results demonstrate the effectiveness of our IPWD. 
Revised: These results demonstrate the effectiveness of our IPWD. [Table]

Original: Conclusion
Revised: Ablation study: IPWD stage . We conduct an ablation study to analyze whether IPWD should be started from an early stage ( e.g. , the beginning of training) or a late stage ( e.g. , last 1/4 of the epochs). We take ResNeXt-29 as an example. As shown in Table 7, applying IPWD from the beginning slightly outperforms PS-KD and under-performs the student modal that applies IPWD only at the late stage by large margins.

Revised: As the student model is poorly trained at the early stage, the weight estimation is inaccurate and hurts the performance of self-teacher.

Revised: These results indicate that the quality of estimated weight and distillation performance relies on the student model and self-teacher.

Revised:  Conclusion

Original: In this paper, we we point out that conventional KD methods hold an invalid IID assumption anddoes not properly address the transfer gap between the context-invariant human domain and thecontext-equivariant machine domain, especially the imbalance knowledge of the teacher model onthe transfer set.
Revised: In this paper, we we point out that conventional KD methods hold an invalid IID assumption and does not properly address the transfer gap between the context-invariant human domain and the context-equivariant machine domain, especially the imbalance knowledge of the teacher model on the transfer set.

Original: We further proposed a simple yet effective method, Inverse Probability Weighting
Revised: We further proposed a simple yet effective method, Inverse Probability Weighting

Original: Distillation (IPWD), to deal with the imbalanced knowledge caused by transfer gap.
Revised: Distillation (IPWD), to deal with the imbalanced knowledge caused by transfer gap.

Original: In the future,we will extend our IPWD to (1) tasks beyond classification, like detection and segmentation, and (2)long-tailed tasks where the training samples in the human domain are also non-IID.
Revised: In the future, we will extend our IPWD to (1) tasks beyond classification, like detection and segmentation, and (2) long-tailed tasks where the training samples in the human domain are also non-IID.

Original: Checklist
Revised: Acknowledgement

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: We thank anonymous ACs and reviewers for their valuable discussion and insightful suggestions. This research is supported by the National Research Foundation, Singapore under its AI Singapore

Original: contributions and scope?
Revised: Programme (AISG Award No: AISG2-RP-2021-022) and Alibaba-NTU Singapore Joint Research

Original: [Yes]
Revised: Institute (JRI).

Original: (b) Did you describe the limitations of your work?
Revised: Appendix A Experimental Settings A.1 Two-stage Distillation

Original: [Yes] See Section 4.
Revised: Implementation .

Original: (c) Did you discuss any potential negative societal impacts of your work? [Yes] SeeAppendix.(d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: We follow the training details of Tian et al. [66] for CIFAR-100 and Zhou et al.

Original: them? [Yes]
Revised: [85] for ImageNet.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: Specifically, for CIFAR-100, we set the mini-batch size as 64 and an initial learning rate as 0.05. We train the model for 240 epochs.

Original: [N/A]
Revised: The learning rate is decayed by 10 every 30 epochs after 150 epochs. We initialize the learning rate as 0.01 for MobileNetV2, ShuffleNetV and ShuffleNetV2, and as 0.05 for other models.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: The experiments are conducted using one NVIDIA

Original: [N/A]
Revised: TITAN RTX GPU.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: For ImageNet, we train the model for 100 epochs. We set the mini-batch size as 256, an initial learning rate as 0.1, and decay it by 10 every 30 epochs.

Original: mental results (either in the supplemental material or as a URL)?
Revised: The experiments are conducted using four Tesla V100 GPUs. Architectures .

Original: [Yes] Please refer to
Revised: We follow Tian et al.

Original:  the supplemental material.
Revised: [66] for the choice of network architectures. Specifically, “WRN-d-w” denotes Wide Residual Network (WRN) [80] with depth d and width factor w . “resnet-d” represents cifar-style ResNet [16] with 3 groups of basic blocks, each with 16, 32, and 64 channels respectively. For example, resnet8x4 and resnet32x4 represent a 4 times wider network, i.e. , with 64, 128, and 256 channels respectively.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: “ResNet-d” represents the ImageNet-style ResNet with Bottleneck blocks and more channels. MobileNetV2 [56] has a width multiplier of 0.5.

Original: how they
Revised: “vgg” denotes VGGNet [60] that is adapted from its original ImageNet counterpart. “ShuffleNetV1” [83], “ShuffleNetV2” [40] are adapted with input size as 32x32. A.2 One- stage Self-Distillation

Original: were chosen)? [Yes] The details are in Section 5.
Revised: Implementation . We follow all the training details of PS-KD [31].

Original: (c) Did you report error bars (e.g., with respect to the random seed after running exper-
Revised: The standard data argumentation schemes are 32x32 random crop after padding with 4 pixels and random horizontal flip. The networks are trained for 300 epochs using SGD with a momentum of 0.9.

Original: iments multiple times)?
Revised: The learning rate is decayed by 10 at 150 and 225 epochs. We set the mini-batch size as 128, an initial learning rate as 0.1, and a weight decay as 0.0005.

Original: [Yes] We run the experiments with five different random
Revised: The experiments are conducted using four Tesla V100 GPUs.

Original: seeds.
Revised: Metrics .

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: Besides top-1 and top-5 accuracies, we follow Kim et al. [31] and report Expected calibration error (ECE) and Area under risk-coverage curve (AURC) for evaluation.

Original: [Yes] See Appendix.
Revised: Expected calibration error (ECE) [47] is used to evaluate the confidence calibration performance of models, i.e. , the expected gap between accuracy and confidence. Specifically, the samples are partitioned by confidence into M bins B 1 , · · · , B M . The m -th bin B m contains samples with confidence within [ m − 1 M , mM ] .

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: For N samples, ECE is formulated as: [Equation] where Acc ( B m ) denotes the accuracy of samples in B m , and Conf ( B m ) denotes the average confidence of samples in B m . A lower BCE indicates a well-calibrated model.

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: Area under risk-coverage curve (AURC) [14] measures how well predictions are ordered by confidence values. Specifically, we can determine a threshold for classification, where only samples with confidence higher than the threshold are accepted. After that, we can obtain the proportion of covered samples to the whole dataset, i.e. , coverage, and define the risk as the error rate computed by using the covered samples.

Original: [Yes]
Revised: AURC is defined as the area under the risk-coverage curve. A lower AURC indicates that the correct and incorrect predictions are well-separable by confidence values. A.3 License of Assets We reimplemented WSLD 2 and SSKD 3 based on their open-resourced codes.

Original: (b) Did you mention the license  of  the assets?
Revised: Both WSLD and SSKD did not mention the license in their open-resourced codes. B Additional Results B.1 Architecture styles Due to page limitation, we did not include all the results of different architecture styles on CIFARin the main paper. We provide additional results in Table 8.

Original: [Yes] See Appendix.
Revised: Results show that our IPWD achieves competitive performances and outperforms KD by large margins. B.2 Feature-based Methods Note that our proposed IPWD is a logit-based distillation method. An interesting question is whether the reweighting strategy can work with feature-based distillation methods.

Revised: We select ReviewKD as an example, which is a recent representative feature-based distillation method.

Revised: As shown in Figure 9, the gap between ReviewKD+IPWD and ReviewKD is very marginal, which indicates that IPWD cannot promote feature-based distillation.

Revised: The possible reasons are two-fold.

Revised: First, the logit knowledge of label y is long-tailed but the representation knowledge of sample x may be relatively balanced.

Original: (c) Did you include any new assets either in  the supplemental material or as a URL?
Revised: Second, as pointed out by Kang et al. [29], “data imbalance might not be an issue in learning highquality representations” for long-tailed classification, which implies that the reweighting strategy is not compatible at feature level. B.3 Long-tailed Methods for KD Note that Figure 1 shows that there seems long-tailed property of teacher predictions.

Revised: An interesting question is whether it could be fixed by techniques for long-tailed classification.

Revised: We take LA [42]

Revised: [Table]

Original: [Yes]
Revised: [Table]

Revised: ReviewKD + IPWD 76.

Revised: as the recent representative technique for long-tailed classification.

Original: The code is attached.
Revised: LA proposed a logit adjusted softmax cross-entropy loss by applying a class prior to each logit. LA does not require extra modules (compared to TDE [65]), post-hoc logit adjustment (compared to LADE [22]), or ensemble of multiple models (compared to RIDE [71]). Following LA, we applied the class prior to the student output when calculating the KL divergence distillation loss. We found that KD+LA underperforms KD by averagely 0.5% on CIFAR-100.

Revised: The possible reason is that the introduced prior indirectly breaks the teacher’s knowledge for each training sample, which hurts the effectiveness of distillation.

Revised: These results indicate that logit-adjust-based long-tailed techniques are not applicable to the issue of KD.

Revised: B.4 Ablation Studies

Revised: We have conducted ablation studies on the components of our proposed IPWD.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: In the appendix, we further provide ablation studies on the technical designs. We take two-stage distillation on CIFAR-100 as an example. Recall that the sample weights are estimated based on the two types of student’s outputs, KD-trained output y kd from the student’s original head and CLS-trained output y cls from the student’s extra head. We further conduct ablations on the selection of two outputs.

Original: [Table]
Revised: For CE-aware output y cls , an straightforward alternative is using an extra model that has the same architecture as student with totally different parameters. This extra model is trained using the crossentropy classification loss.

Original: using/curating? [Yes] All the used existing assets are open-sourced for research.
Revised: In other words, the difference is whether the visual backbone is shared for the two outputs. We denote this alternative as IPWD ∗ .

Original: [Table]
Revised: Table 10 shows the comparison. Overall,

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: IPWD ∗ achieves competitive results compared to IPWD. However, training an extra model leads to more memory and time cost. Therefore, our design that takes an extra head is both effective and efficient.

Original: [Table]
Revised: [Table]

Original: information or offensive content?

Original: [Yes] The benchmark dataset does not contain

Original: [Table]
Revised: [Table]

Original: personally identifiable information or offensive content.
Revised: For KD-trained output y kd

Original: [Table]
Revised: [Table]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: an straightforward alternative is using an extra distillation head like the classification head. Different from the original head of the student

Original: [Table]
Revised: [Table]

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: the distillation head is trained only using the distillation loss, which simply mimics the teacher’s output without ground-truth annotations.

Original: [Table] applicable? [N/A]
Revised: We denote this alternative as IPWD † .

Original: [Table]
Revised: Table 11 shows the comparison.

Original: (b) Did you describe any potential participant risks, with links to Institutional Review [Table] Board (IRB) approvals, if applicable? [N/A]
Revised: Overall, IPWD † slightly underperforms IPWD with different four architectures.

Original: [Table] (c) Did you include the estimated hourly wage paid toparticipants and the total amount [Table] spent on participant compensation?
Revised: The reason is that the extra distillation head only learns from the teacher model, which is sensitive to the teacher’s performance and weights hard samples more.

Original: [N/A] [Table]
Revised: These two ablation studies further verify the effectiveness of our techinical designs.