Original: Respecting Transfer Gap in Knowledge Distillation
Revised: Respecting Transfer Gap in Knowledge Distillation

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Knowledge distillation (KD) is essentially a process of transferring a teachermodel’s behavior, e.g., network response, to a student model.
Revised: Knowledge distillation (KD) is essentially a process of transferring a teachermodel’s behavior, e.g., network response, to a student model.

Original: The network responseserves as additional supervision to formulate the machine domain ( machine forshort), which uses the data collected from the human domain ( human for short) asa transfer set.
Revised: The network responseserves as additional supervision to formulate the machine domain ( machine forshort), which uses the data collected from the human domain ( human for short) asa transfer set.

Original: Traditional KD methods hold an underlying assumption that the datacollected in both human domain and machine domain are both independent andidentically distributed (IID).
Revised: Traditional KD methods hold an underlying assumption that the datacollected in both human domain and machine domain are both independent andidentically distributed (IID).

Original: We point out that this naïve assumption is unrealisticand there is indeed a transfer gap between the two domains.
Revised: We point out that this naïve assumption is unrealisticand there is indeed a transfer gap between the two domains.

Original: Although the gap offersthe student model external knowledge from the machine domain, the imbalancedteacher knowledge would make us incorrectly estimate how much to transfer fromteacher to student per sample on the non-IID transfer set.
Revised: Although the gap offersthe student model external knowledge from the machine domain, the imbalancedteacher knowledge would make us incorrectly estimate how much to transfer fromteacher to student per sample on the non-IID transfer set.

Original: To tackle this challenge,we propose Inverse Probability Weighting Distillation (IPWD) that estimates thepropensity of a training sample belonging to the machine domain, and assignsits inverse amount to compensate for under-represented samples.
Revised: To tackle this challenge,we propose Inverse Probability Weighting Distillation (IPWD) that estimates thepropensity of a training sample belonging to the machine domain, and assignsits inverse amount to compensate for under-represented samples.

Original: Experimentson CIFAR-100 and ImageNet demonstrate the effectiveness of IPWD for bothtwo-stage distillation and one-stage self-distillation.
Revised: Experimentson CIFAR-100 and ImageNet demonstrate the effectiveness of IPWD for bothtwo-stage distillation and one-stage self-distillation.

Original:  Introduction
Revised:  Introduction

Original: Knowledge distillation (KD)
Revised: Knowledge distillation (KD)

Original: [18] transfers knowledge from a teacher model, e.g. , a big, cumbersome,and energy-inefficient network, to a student model, e g. , a small, light, and energy-efficient network,to improve the performance of the student model.
Revised: [18] transfers knowledge from a teacher model, e.g. , a big, cumbersome,and energy-inefficient network, to a student model, e.g. , a small, light, and energy-efficient network,to improve the performance of the student model.

Original: A common intuition is that a teacher with betterperformance will teach a stronger student.
Revised: A common intuition is that a teacher with betterperformance will teach a stronger student.

Original: However, recent studies find that the teacher’s accuracy isnot a good indicator of the resultant student performance [8].
Revised: However, recent studies find that the teacher’s accuracy isnot a good indicator of the resultant student performance [8].

Original: For example, a poorly-trained teacherwith early stopping can still teach a better student [8, 11, 63]; or, a teacher with a smaller model sizethan the student is also a good teacher [63]; or, a teacher with the same architecture as the studenthelps to improve the student—self-distillation [13, 67, 66, 22].
Revised: For example, a poorly-trained teacherwith early stopping can still teach a better student [8, 11, 63]; or, a teacher with a smaller model sizethan the student is also a good teacher [63]; or, a teacher with the same architecture as the studenthelps to improve the student—self-distillation [13, 67, 66, 22].

Original: Should we view KD in a perspective of domain transfer [12, 52], we would better understand theabove counter-intuitive findings.
Revised: Should we view KD in a perspective of domain transfer [12, 52], we would better understand theabove counter-intuitive findings.

Original: From Figure 1, we can see that teacher predictions and ground-truthlabels indeed behave differently.
Revised: From Figure 1, we can see that teacher predictions and ground-truthlabels indeed behave differently.

Original: Although the teacher is trained on the balanced dataset, its predictedprobability distribution over the dataset is imbalanced.
Revised: Although the teacher is trained on the balanced dataset, its predictedprobability distribution over the dataset is imbalanced.

Original: Even on the same training set with the samemodel parameter, teachers with different temperature τ yield different “soft label” distributionsfrom the ground-truth ones.
Revised: Even on the same training set with the samemodel parameter, teachers with different temperature τ yield different “soft label” distributionsfrom the ground-truth ones.

Original: This implies that human and teacher knowledge are from differentdomains, and there is a transfer gap that drives the “dark knowledge” [18] transferring from teacherto student—regardless of “strong” or “weak” teachers, it is a valid transfer as long as there is agap.
Revised: This implies that human and teacher knowledge are from differentdomains, and there is a transfer gap that drives the “dark knowledge” [18] transferring from teacherto student—regardless of “strong” or “weak” teachers, it is a valid transfer as long as there is a gap.

Revised: However, the transfer gap affects the distillation performance on the under-represented classes, i.e. ,classes on the tail of teacher predictions, which is overlooked in recent studies.

Revised: Taking CIFAR-100 asan example, we rank and divide the 100 classes into 4 groups according to the ranks of predictedprobability.

Revised: Compared to vanilla training, KD achieves better performance in all the subgroups.

Original:  We ask: what causes the gap from the first place; or more specifically, why does the teacher’s non-uniform distributed predictions implies the gap in Figure 1?
Revised: However, going deeper into each subgroup, we found that the increase in the top 25 classes is much [Figure] higher than that in the last 25 classes, i.e., averagely 5.14% vs. 0.85%. We ask: what causes the gapfrom the first place; or more specifically, why does the teacher’s non-uniform distributed predictionsimplies the gap?

Original: We answer in an invariance vs.equivariance learning point of view [4, 57]: 
Revised: We answer in an invariance vs. equivariance learning point of view [4, 57]: [Table] [Figure] [Table]

Original: Human domain: context invariance .
Revised: Human domain: context invariance .

Original: The discriminative generalization is the ability to learn both context-invariant and class-equivariant information from the diverse training samples per class.The
Revised: The discriminative generalization is theability to learn both context-invariant andclass-equivariant information from thediverse training samples per class.

Original: [Figure] human domain only provides context-invariant class-specific information, i.e. , hard targets.
Revised: Thehuman domain only provides contextinvariant class-specific information, i.e. ,hard targets.

Original: Wenormally collect a balanced training dataset to  the  domain.
Revised: We normally collect a balanced training dataset to formulate the human domain.

Original: Machine domain: context equivariance .
Revised: Machine domain: context equivariance .

Original: Teacher models often use a temperature variable topreserve the contexts.
Revised: Teacher models often use a temperature variable topreserve the contexts.

Original: The temperature allows the teacher to represent a sample not only by itscontext-invariant class-specific information, but also its context-equivariant information.
Revised: The temperature allows the teacher to represent a sample not only by itscontext-invariant class-specific information, but also its context-equivariant information.

Original: For example,a dog image with soft label 0.8 · dog + 0.2 · wolf may imply that the dog has wolf-like contextualattributes such as “fluffy coat” and “upright ears”.
Revised: For example,a dog image with soft label 0.8 · dog + 0.2 · wolf may imply that the dog has wolf-like contextualattributes such as “fluffy coat” and “upright ears”.

Original: Although the context-invariance ( i.e. , class) isbalanced in the training data, the context-equivariance ( i.e. , context) is imbalanced because the contextbalance is not considered in class-specific data collection [55].
Revised: Although the context-invariance ( i.e. , class) isbalanced in the training data, the context-equivariance ( i.e. , context) is imbalanced because the contextbalance is not considered in class-specific data collection [55].

Original: To construct the transfer set for themachine domain, the teacher model annotates each sample after seeing others, i.e. , being pre-trainedon the whole set.
Revised: To construct the transfer set for themachine domain, the teacher model annotates each sample after seeing others, i.e. , being pre-trainedon the whole set.

Original: Interestingly, the diverse context results in a long-tailed imbalanced distribution,which is exactly reflected in Figure 1.
Revised: Interestingly, the diverse context results in a long-tailed imbalanced distribution,which is exactly reflected in Figure 1.

Original: In other words, the teacher’s knowledge is imbalanced eventhough the teacher is trained on a class-balanced dataset.
Revised: In other words, the teacher’s knowledge is imbalanced eventhough the teacher is trained on a class-balanced dataset.

Original: Now we are ready to point out how the transfer gap is not properly addressed in conventional KDmethods.
Revised: Now we are ready to point out how the transfer gap is not properly addressed in conventional KDmethods.

Original: Conventional KD calculates the Cross-Entropy (CE) loss between the ground-truth labeland student’s prediction, and the Kullback–Leibler (KL) divergence [27] loss between the teacher’sand student’s predictions, where a constant weight is assigned for both CE loss and KL loss.
Revised: Conventional KD calculates the Cross-Entropy (CE) loss between the ground-truth labeland student’s prediction, and the Kullback–Leibler (KL) divergence [27] loss between the teacher’sand student’s predictions, where a constant weight is assigned for both CE loss and KL loss.

Original: This isessentially based on the underlying assumption that the data in both the human and machine domainsare IID.
Revised: This isessentially based on the underlying assumption that the data in both the human and machine domainsare IID.

Original: Based on the analysis of context equivariance, we argue that the assumption is unrealistic,i.e. , the teacher’s knowledge is imbalanced.
Revised: Based on the analysis of context equivariance, we argue that the assumption is unrealistic,i.e. , the teacher’s knowledge is imbalanced.

Original: Therefore, a constant sample weight for the KL losswould be a bottleneck.
Revised: Therefore, a constant sample weight for the KL losswould be a bottleneck.

Original: In this paper, we propose a simple yet effective method, Inverse Probability
Revised: In this paper, we propose a simple yet effective method, Inverse Probability

Original: Weighting Distillation (IPWD), which compensates for the training samples that are under-weightedin the machine domain.
Revised: Weighting Distillation (IPWD), which compensates for the training samples that are under-weightedin the machine domain.

Original: For each training sample x , we first estimate its machine-domain propensityscore P ( x | machine ) by comparing class-aware and context-aware predictions.
Revised: For each training sample x , we first estimate its machine-domain propensityscore P ( x | machine ) by comparing class-aware and context-aware predictions.

Original: A sample with a lowpropensity score would have a high confidence from class-aware predictions and a low confidencefrom context-aware predictions.
Revised: A sample with a lowpropensity score would have a high confidence from class-aware predictions and a low confidencefrom context-aware predictions.

Original: Then, IPWD assigns the inverse probability 1 /P ( x | machine ) asthe sample weight for the KL loss to highlight the under-represented samples.
Revised: Then, IPWD assigns the inverse probability 1 /P ( x | machine ) asthe sample weight for the KL loss to highlight the under-represented samples.

Original: In this way, IPWDgenerates a pseudo-population [31, 21] to deal with the imbalanced knowledge.
Revised: In this way, IPWDgenerates a pseudo-population [31, 21] to deal with the imbalanced knowledge.

Original: We evaluate our proposed IPWD on two typical knowledge distillation settings: two-stage teacherstudent distillation and one-stage self-distillation.
Revised: We evaluate our proposed IPWD on two typical knowledge distillation settings: two-stage teacherstudent distillation and one-stage self-distillation.

Original: Experiments conducted on CIFAR-100 [26] and
Revised: Experiments conducted on CIFAR-100 [26] and

Original: ImageNet [10] demonstrate the effectiveness and generality of our IPWD.
Revised: ImageNet [10] demonstrate the effectiveness and generality of our IPWD.

Original: Our contributions are three-fold:
Revised: Our contributions are three-fold:

Original: • We formulate KD as a domain transfer problem and argue that the naïve IID assumption on machinedomain neglects the imbalanced knowledge due to transfer gap.
Revised: • We formulate KD as a domain transfer problem and argue that the naïve IID assumption on machinedomain neglects the imbalanced knowledge due to transfer gap.

Original: • We propose Inverse Probability Weighting Distillation (IPWD) which compensate for the samplesthat are overlooked in the machine domain to tackle the imbalanced knowledge in transfer
Revised: • We propose Inverse Probability Weighting Distillation (IPWD) which compensate for the samplesthat are overlooked in the machine domain to tackle the imbalanced knowledge in transfer

Original: gap.• Experiments on CIFAR-100 and ImageNet for both two- stage distillation and one-stage selfdistillation show that the proper handling of the transfer gap is a promising direction in KD.
Revised: gap.• Experiments on CIFAR-100 and ImageNet for both two-stage distillation and one-stage selfdistillation show that the proper handling of the transfer gap is a promising direction in KD.

Original:  Related Work
Revised:  Related Work

Original: Knowledge distillation (KD) was first introduced to transfer the knowledge from an effective butcumbersome model to a smaller and efficient model [18].
Revised: Knowledge distillation (KD) was first introduced to transfer the knowledge from an effective butcumbersome model to a smaller and efficient model [18].

Original: The knowledge can be formulated ineither output space [18, 23, 29, 64, 63, 35, 50, 69, 25] or representation space [45, 20, 65, 24, 41,16, 54, 7, 22].
Revised: The knowledge can be formulated ineither output space [18, 23, 29, 64, 63, 35, 50, 69, 25] or representation space [45, 20, 65, 24, 41,16, 54, 7, 22].

Original: KD has attracted a wide interest in theory, methodology, and applications [14].
Revised: KD has attracted a wide interest in theory, methodology, and applications [14].

Original: For applications, KD has shown its great potential in various areas, including but not limited toclassification [30, 44, 33, 19], detection [28, 49, 58], segmentation [15, 36, 32] for visual recognitiontasks, and visual question answering [38, 1, 39], video captioning [40, 68], and text-to-imagesynthesis [53] for vision-language tasks.
Revised: For applications, KD has shown its great potential in various areas, including but not limited toclassification [30, 44, 33, 19], detection [28, 49, 58], segmentation [15, 36, 32] for visual recognitiontasks, and visual question answering [38, 1, 39], video captioning [40, 68], and text-to-imagesynthesis [53] for vision-language tasks.

Original: Recent studies further discussed how and why KD works.
Revised: Recent studies further discussed how and why KD works.

Original: Specifically, Müller et al.
Revised: Specifically, Müller et al.

Original: [37] and Shen et al.
Revised: [37] and Shen et al.

Original: [48] empirically analyzed the effect of label smoothingon KD.
Revised: [48] empirically analyzed the effect of label smoothingon KD.

Original: Cho et al.
Revised: Cho et al.

Original: [8], Dong et al.
Revised: [8], Dong et al.

Original: [11], and Yuan et al.
Revised: [11], and Yuan et al.

Original: [63] pointed out that early stopping is a goodregularization for a better teacher.
Revised: [63] pointed out that early stopping is a goodregularization for a better teacher.

Original: Yuan et al.
Revised: Yuan et al.

Original: [63] further found that a poorly trained teacher, evena model smaller than the student, can improve the performance of the student.
Revised: [63] further found that a poorly trained teacher, evena model smaller than the student, can improve the performance of the student.

Original: Besides, Memon etal.
Revised: Besides, Memon etal.

Original: [34] and Zhou et al.
Revised: [34] and Zhou et al.

Original: [69] proposed a bias-variance trade-off perspective for KD.
Revised: [69] proposed a bias-variance trade-off perspective for KD.

Original: In this paper, wepoint out that existing KD methods hold an underlying assumption that the IID training samples arealso IID in the machine domain, which overlooks the transfer gap.
Revised: In this paper, wepoint out that existing KD methods hold an underlying assumption that the IID training samples arealso IID in the machine domain, which overlooks the transfer gap.

Original: Self-distillation is a special case of KD, which uses the student network itself as the teacher instead ofthe cumbersome model, i.e. , the teacher and student models have the same architecture [13, 67, 66, 22].
Revised: Self-distillation is a special case of KD, which uses the student network itself as the teacher instead ofthe cumbersome model, i.e. , the teacher and student models have the same architecture [13, 67, 66, 22].

Original: This process can be executed in iterations and produce a stronger ensemble model [13].
Revised: This process can be executed in iterations and produce a stronger ensemble model [13].

Original: Similar to
Revised: Similar to

Original: KD, traditional self-distillation follows a two-stage process: first pre-training a student model asthe teacher, and then distill the knowledge from the pre-trained model to a new student model.
Revised: KD, traditional self-distillation follows a two-stage process: first pre-training a student model asthe teacher, and then distill the knowledge from the pre-trained model to a new student model.

Original: Inorder to perform the teacher-student optimization in one generation, recent studies [61, 25] proposedone-stage self-distillation that adopts student models at earlier epochs as teacher models.
Revised: Inorder to perform the teacher-student optimization in one generation, recent studies [61, 25] proposedone-stage self-distillation that adopts student models at earlier epochs as teacher models.

Original: Theseone-stage self-distillation methods outperform vanilla students by large margins.
Revised: Theseone-stage self-distillation methods outperform vanilla students by large margins.

Original: In this paper, wealso evaluate the effectiveness of our IPWD as a plug-in in one-stage self-distillation.
Revised: In this paper, wealso evaluate the effectiveness of our IPWD as a plug-in in one-stage self-distillation.

Original: Inverse Probability Weighting (IPW)
Revised: Inverse Probability Weighting (IPW)

Original: [46, 31, 21, 5], also known as inverse probability of treatmentweighting or inverse propensity weighting, was proposed to correct the selection bias when theobservations are non-IID.
Revised: [46, 31, 21, 5], also known as inverse probability of treatmentweighting or inverse propensity weighting, was proposed to correct the selection bias when theobservations are non-IID.

Original: IPW uses the inverse of the probability ( i.e. , propensity score) that theindividual would be assigned to the treatment group to reweight the samples.
Revised: IPW uses the inverse of the probability ( i.e. , propensity score) that theindividual would be assigned to the treatment group to reweight the samples.

Original: Propensity-weightingtechniques have been widely applied and studied in many areas [47] such as causal inference [21],complete-case analysis [31], machine learning [9, 6, 51], and recommendation systems [47, 59, 3].
Revised: Propensity-weightingtechniques have been widely applied and studied in many areas [47] such as causal inference [21],complete-case analysis [31], machine learning [9, 6, 51], and recommendation systems [47, 59, 3].

Original: Inthis paper, we view the distillation process as a domain transfer problem and adopt IPW to dynamicallyassign the weight to each training sample for the distillation loss.
Revised: In this paper, we view the distillation process as a domain transfer problem and adopt IPW todynamically assign the weight to each training sample for the distillation loss.

Original: Recently, Zhou et al.

Original: [69] proposeda reweighting method to balance the bias-variance trade-off for KD, which compares the outputs ofthe student model and teacher model to calculate the sample weight.

Original: Differently, our formulation ofsamples weights is theoretically guaranteed by the causal theory of IPW [46, 31, 21, 5].

Original:  Background
Revised:  Background

Original: In this paper, we view knowledge distillation (KD) from a perspective of domain transfer, and takethe image classification task as the case study.
Revised: In this paper, we view knowledge distillation (KD) from a perspective of domain transfer, and takethe image classification task as the case study.

Original: Suppose that the training data D = {X , Y} = { ( x, y ) }contains x as the input ( e.g. , image) and y ∈ R C as its ground-truth annotation ( e.g. , one-hotlabel), where C denotes the number of classes.
Revised: Suppose that the training data D = {X , Y} = { ( x, y ) }contains x as the input ( e.g. , image) and y ∈ R C as its ground-truth annotation ( e.g. , one-hotlabel), where C denotes the number of classes.

Original: A standard solution to train the classifier θ uses thecross-entropy loss as the objective:
Revised: A standard solution to train the classifier θ uses thecross-entropy loss as the objective:

Original: [Equation]
Revised: [Equation]

Original: where ℓ cls ( x, y ) =
Revised: where ℓ cls ( x, y ) =

Original: H ( y s , y ) is the classification loss for sample x , H ( p, q )
Revised: H ( y s , y ) is the classification loss for sample x , H ( p, q )

Original: =
Revised: =

Original: (cid:80) Ci =1 − q i log p idenotes the cross entropy between p and q , y s = f
Revised: (cid:80) Ci =1 − q i log p idenotes the cross entropy between p and q , y s = f

Original: ( x ; θ ) denotes the model’s output probabilitygiven x , i.e. , y s k = [Equation] =
Revised: ( x ; θ ) denotes the model’s output probability given x , i.e. , y sk =

Original: [Equation]
Revised: [Equation]

Original:  exp ( z si ) , where z s is the output logits of the model.
Revised: =1 exp ( z si ) , where z s is the output logits of the model.

Original: The hard targets providecontext-invariant class-specific information from the human domain.
Revised: The hard targets providecontext-invariant class-specific information from the human domain.

Original: An assumption held behind
Revised: An assumption held behind

Original: Eq.
Revised: Eq.

Original: (1) is that the samples are independent and identically distributed (IID) in the training and test set.
Revised: (1) is that the samples are independent and identically distributed (IID) in the training and test set.

Original: KD adopts a teacher model θ t to generate soft targets as extra supervisions, i.e. , context-equivariantinformation.
Revised: KD adopts a teacher model θ t to generate soft targets as extra supervisions, i.e. , context-equivariantinformation.

Original: To formulate the machine domain, traditional KD methods commonly use the training setI npu t s t u d e n t ba c k b on e f e a t u r e L k d
Revised: To formulate the machine domain, traditional KD methods commonly use the training set

Original: D i s t ill h e ad y s

Original: I npu

Original: t s t u d e n t ba c k b on e f e a t u r e

Original: C L S h e ad

Original: D i s t ill h e ad y c l s y s

Original: L c l s

Original: L i p w d w

Original: Eq.

Original: Eq. Eq. [Figure] D to construct the transfer set D t using the same copy of X , i.e. , D t = { ( x, y t ) } where y t = f
Revised: D to construct the transfer set D t using the same copy of X , i.e. , D t = { ( x, y t ) } where y t = f

Original: ( x ; θ t )and x ∈ X .
Revised: ( x ; θ t )and x ∈ X .

Original: Traditional KD approaches use the KL divergence [27] loss for knowledge transfer:
Revised: Traditional KD approaches use the KL divergence [27] loss for knowledge transfer:

Original: [Equation]
Revised: [Equation]

Original: ) where ℓ dist ( x, y t ; θ ) = τ 2 · [ H ( y sτ , y tτ ) 
Revised: where ℓ dist ( x, y t ; θ ) = τ 2 · [ H ( y sτ , y tτ ) − H

Original: − H ( y tτ , y tτ )] denotes the distillation loss for sample x .
Revised: ( y tτ , y tτ )] denotes the distillation loss for sample x .

Original: Normally, the outputs of the student and teacher are softened using a temperature τ , i.e. , y sτ,k =
Revised: Normally, the outputs of the student and teacher are softened using a temperature τ , i.e. , y sτ,k =exp( z sk /τ )

Revised: (cid:80) Ci =1 exp ( z

Original: [Equation]
Revised: [Equation]

Original: Ci =
Revised: ) and y tτ,k =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) and y tτ,k = exp ( z tk /τ ) (cid:80)
Revised: ti

Original: Ci =1 exp ( z ti /τ ) .
Revised: [Equation]

Original: The overall objective combines L cls and L dist as:
Revised: The overall objective combines L cls and L dist as:

Original: [Equation]
Revised: [Equation]

Original: ) where α and β are the hyper-parameters.
Revised: where α and β are the hyper-parameters.

Original: The underlying assumption of traditional KD behind Eq.
Revised: The underlying assumption of traditional KD behind Eq.

Original: (2)is that the transfer set D t is an unbiased approximation of the machine domain.
Revised: (2)is that the transfer set D t is an unbiased approximation of the machine domain.

Original: However, the observedlong-tailed and temperature-sensitive distributions of teacher’s predictions in Figure 1 rationallychallenge this assumption.
Revised: However, the observedlong-tailed and temperature-sensitive distributions of teacher’s predictions in Figure 1 rationallychallenge this assumption.

Original: As a result, samples with lower P ( x | machine ) are under-representedduring the distillation process, which affects the unbiasedness of knowledge transfer.
Revised: As a result, samples with lower P ( x | machine ) are under-representedduring the distillation process, which affects the unbiasedness of knowledge transfer.

Original: This analysisindicates that Eq.
Revised: This analysisindicates that Eq.

Original: (2) is not optimal to utilize the teacher’s imbalanced knowledge.
Revised: (2) is not optimal to utilize the teacher’s imbalanced knowledge.

Original:  Method
Revised:  Method

Original: We propose a simple yet effective method, Inverse Probability Weighting Distillation (IPWD), torespect the transfer gap and imbalance knowledge in KD.
Revised: We propose a simple yet effective method, Inverse Probability Weighting Distillation (IPWD), torespect the transfer gap and imbalance knowledge in KD.

Original: In this section, we first introduce the overallframework of IPWD, then present the implementation details.
Revised: In this section, we first introduce the overallframework of IPWD, then present the implementation details.

Original: 4.1 Inverse Probability Weighting for KD
Revised: 4.1 Transfer Gap in KD

Revised: 𝑿 [Equation]

Revised: We first interpret the transfer gap and its confounding effectfrom the perspective of causal inference.

Revised: Figure 2 illustratesthe causal relations between the image X , training data D ={ ( x, y ) } , teacher’s parameters θ t and teacher’s output Y t in

Revised: KD.

Revised: Overall, D and θ t jointly act as the confounder of X and

Revised: Y t in the transfer set.

Revised: First, the training set D and transfer setof teacher model D t = { ( x, y t ) } share the same image set, and X = x is sampled from the imageset of D , i.e., D serves the cause of X .

Revised: Second, the teacher θ t is trained on D , and y t is calculatedbased on θ t and x , i.e., y t

Revised: = f

Revised: ( x ; θ t ) .

Revised: Therefore, X and θ t are the cause of Y t .

Revised: Note that the transferset is constructed based on the images on D and teacher model θ t .

Revised: Therefore, we regard the transferset D t , the joint of D and θ t , as the confounder of x and y t .

Revised: Although D is balanced when considering the context-invariant class-specific information, the contextinformation (e.g., attributes) is overlooked, which makes the D imbalanced on context.

Revised: As shownin Figure 1, such imbalanced context leads to an imbalanced transfer set D t and further affects thedistillation performance of teacher’s knowledge.

Revised: To overcome such confounding effect, a commonly used technique is intervention via P ( y t | do ( x )) instead of P ( y t | x ) , which is formulated as P ( y t | do ( x ))

Revised: = (cid:80) D t P ( y | x, D t ) P ( D t )

Revised: = (cid:80) D t P ( x,y t , D t ) P ( x |D t ) .

Revised: This transformation suggests that we can use the inverse of propensity score P ( x |D t ) as sampleweight to implement the intervention and overcome the confounding effect.

Revised: Thanks to the causalitybased theory [46, 5], we can use the Inverse Probability Weighting (IPW) technique to overcome theconfounding effect brought by the transfer gap.

Revised: I npu

Revised: t s t u d e n t ba c k b on e f e a t u r e

Revised: L k d

Revised: D i s t ill h e ad y s

Revised: I npu

Revised: t s t u d e n t ba c k b on e f e a t u r e

Revised: C L S h e ad

Revised: D i s t ill h e ad y c l s y s

Revised: L c l s

Revised: L i p w d w

Revised: Eq.

Revised: [Figure]

Revised: [Figure]

Revised: Eq.

Revised: [Figure]

Revised: 4.2 Inverse Probability Weighting for KD

Original: As analyzed in Section 3, the IID training samples in the human domain are no longer IID in themachine domain.
Revised: As analyzed in Section 3, the IID training samples in the human domain are no longer IID in themachine domain.

Original: Simply assuming the training set as the perfect transfer set may lead to the selectionbias: samples that match “head” knowledge are over-represented and easy to be observed, whilesamples that match “tail” knowledge are under-represented and hard to be observed.
Revised: Simply assuming the training set as the perfect transfer set may lead to the selectionbias: samples that match “head” knowledge are over-represented and easy to be observed, whilesamples that match “tail” knowledge are under-represented and hard to be observed.

Original: This wouldsuppress the transfer of “tail” knowledge.
Revised: This wouldsuppress the transfer of “tail” knowledge.

Original: Thanks to the causality-based theory [46, 5], we can usethe Inverse Probability Weighting (IPW) technique to overcome the confounding effect brought bythe transfer gap.
Revised: The analysis from the perspective of causal inference in Section 4.1 suggests that we can use Inverse Probability Weighting (IPW) for debiased distillation.

Original: In short, IPW generates a pseudo-population where under-represented samples areassigned with large weights and over-represented samples are assigned with small weights.
Revised: In short, IPW generates a pseudo-population where under-represented samples are assigned withlarge weights and over-represented samples are assigned with small weights.

Original: Theweight for sample x is determined as the inverse of its probability, also known as propensity score,to the domain d ∈ { human , machine } , i.e. , w x | d = 1 /p ( x | d ) .
Revised: The weight for samplex is determined as the inverse of its probability, also known as propensity score, to the domaind ∈ { human , machine } , i.e. , w x | d = 1 /p ( x | d ) .

Original: We adopt IPW to KD and obtain thefollowing objective for sample x :
Revised: We adopt IPW to KD and obtain the followingobjective for sample x :

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: 4.2 Implementation
Revised: 4.3 Implementation

Original: Since the training and test data are normally IID in the human domain, we safely and rationallyuse the empirical risk.
Revised: Since the training and test data are normally IID in the human domain, we safely and rationallyuse the empirical risk.

Original: Therefore, we assign a constant weight to each sample when calculating theclassification loss.
Revised: Therefore, we assign a constant weight to each sample when calculating theclassification loss.

Original: For sure, the training and test samples can be non-IID, e.g. , long-tailed tasks,which is out of the scope of this paper.
Revised: For sure, the training and test samples can be non-IID, e.g. , long-tailed recognitiontasks, which is out of the scope of this paper.

Original: As analyzed in Section 1, the assumption held by traditional KD, i.e. , both P ( X | human ) and
Revised: As analyzed in Section 1, the assumption held by traditional KD, i.e. , both P ( X | human ) and

Original: P ( X | machine ) are IID, is unrealistic in practice.
Revised: P ( X | machine ) are IID, is unrealistic in practice.

Original: Therefore, we should consider the propensity
Revised: Therefore, we should consider the propensity

Original: P ( x | machine ) as a sample-specific value for the distillation loss to improve the generalization.
Revised: P ( x | machine ) as a sample-specific value for the distillation loss to improve the generalization.

Original: Traditional IPW estimates the propensity using logistic regression, i.e. , ˆ P ( x | machine )
Revised: Traditional IPW estimates the propensity using logistic regression, i.e. , ˆ P ( x | machine )

Original: = 1 / (1 +exp( − z x )) where z x is the logit for x .
Revised: = 1 / (1 +exp( − z x )) where z x is the logit for x .

Original: Since the ground-truth annotation of P ( x | machine ) is notavailable, it is not practicable to directly train the regression model in a fully-supervised manner.
Revised: Since the ground-truth annotation of P ( x | machine ) is notavailable, it is not practicable to directly train the regression model in a fully-supervised manner.

Original: Therefore, we estimate the propensity in an unsupervised way.
Revised: Therefore, we estimate the propensity in an unsupervised way.

Original: Recall that the samples with high propensity are over-represented in the transfer set.
Revised: Recall that the samples with high propensity are over-represented in the transfer set.

Original: As a result,the student model would learn more from the over-represented groupthan the under-representedgroup via distillation.
Revised: As a result, thestudent model would learn less from the under-represented group via distillation.

Original: Therefore, we use a classification-trained (CLS-trained) classifier for thehuman domain as reference, and assume that a KD-trained classifier for the machine domain is moreconfident for the over-represented group than the CLS-trained classifier.
Revised: Therefore, we use aclassification-trained (CLS-trained) classifier for the human domain as reference, and assume that a KD-trained classifier for the machine domain is more confident for the over-represented group thanthe CLS-trained classifier.

Original: We compare the outputs oftwo classifiers to identify whether a sample is under-represented in the machine domain.
Revised: We compare the outputs of two classifiers to identify whether a sampleis under-represented in the machine domain.

Original: Supposethat the KD-trained output is y kd and the CLS-trained output is y cls .
Revised: Suppose that the KD-trained output is y kd and the CLS-trained output is y cls .

Original: The assumption implies thatthe logit z is negatively correlated with H ( y kd , y ) and positively correlated with H ( y cls , y ) , where H ( · , · ) is the cross-entropy defined in Section 3.
Revised: The assumption implies that the logit z is negatively correlated with H ( y kd , y ) and positively correlated with H ( y cls , y ) , where H ( · , · ) is the cross-entropy defined in Section 3.

Original: Considering the range of logit, we estimate z x as:
Revised: Considering the range of logit, we estimate z x as z x

Original: [Equation]
Revised: [Equation]

Original: Figure 2 illustrates the comparison between traditional KD and our IPWD.
Revised: Figure 3 illustrates the comparison between traditional KD and our IPWD.

Original: We take the KD-trainedstudent’s output y s as y kd , and train an extra classifier head ( i.e. , “CLS head” in Figure 2(b)) tocalculate y cls , which is optimized using the cross-entropy loss L cls with the ground-truth labels.
Revised: We take the KD-trainedstudent’s output y s as y kd , and train an extra classifier head ( i.e. , “CLS head” in Figure 3(b)) tocalculate y cls , which is optimized using the cross-entropy loss L cls with the ground-truth labels.

Original: Asshown in Table 4, we empirically find that directly using y s and y cls would lead to a high variance. 
Revised: Asshown in Table 5, we empirically find that directly using y s and y cls would lead to a high variance. [Figure]

Revised: ) honeycomb apiary bee hen cock blackgrouse canoe paddle snorkel

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Original: For example, a wrongly classified sample may have a extremely large loss and heavily suppress thedistillation of other samples.
Revised: # vulture [Figure] Africanwilddog hyena dhole racecar sportscar convertible ant groundbeetle tigerbeetle [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: For example, a wrongly classified sample may have a extremely large loss and heavily suppress thedistillation of other samples.

Original: Therefore, we normalize the logits by dividing them by the standarddeviation σ s and σ cls , i.e. , (cid:101) y s = 
Revised: Therefore, we normalize the logits by dividing them by the standarddeviation σ s and σ cls , i.e. , (cid:101) y s = exp

Original: [Equation]
Revised: [Equation]

Original:  and (cid:101) y cls 
Revised: (cid:80) Ci =1 exp ( z si /σ s ) and (cid:101) y cls = exp

Original: [Equation]
Revised: [Equation]

Original: z
Revised: clsk

Original: [Equation]
Revised: [Equation]

Revised: cls ) (cid:80)

Revised: Ci =1 exp ( z clsi /σ cls ) .

Original: In this way, theoutputs are at the same scale with a standard deviation equal to 1, which helps to reduce the variance.
Revised: In this way, theoutputs are at the same scale with a standard deviation equal to 1, which helps to reduce the variance.

Original: We finally take (cid:101)
Revised: We finally take (cid:101)

Original: y s as y kd and (cid:101)
Revised: y s as y kd and (cid:101)

Original: y cls as y cls .
Revised: y cls as y cls .

Original: Combining Eq. (5) into propensity, we have:
Revised: Combining z x into propensity, we have:

Original: [Equation]
Revised: [Equation]

Original:  and the estimated weight ˆ w x for sample x is:
Revised: 5) and the estimated weight ˆ w x for sample x is ˆ w x =

Original: [Equation]
Revised: [Equation]

Original: Figure 3 illustrates some examples of training samples and their assigned weights.
Revised: Figure 4 illustratessome examples of training samples and their assigned weights.

Original: Under-representedsamples, for which the KD-trained classifier is less confident than the C: A classifier, are assignedwith a large weight (Figure 3(a)).
Revised: Under-represented samples, forwhich the KD-trained classifier is less confident than the CLS-trained classifier, are assigned with alarge weight (Figure 4(a)).

Original: Over-represented samples, for which the KD-trained classifieris more confident than the CLS-trained classifier, are assigned with a small weight (Figure 3(c)).
Revised: Over-represented samples, for which the KD-trained classifier is moreconfident than the CLS-trained classifier, are assigned with a small weight (Figure 4(c)).

Original: Samples for which the two classifiers behave similarly are assigned with a balanced weight (Figure3(b)).
Revised: Samplesfor which the two classifiers behave similarly are assigned with a balanced weight (Figure 4(b)).

Original: The weighted distillation loss is formulated as:
Revised: Theweighted distillation loss is formulated as:

Original: [Equation]
Revised: [Equation]

Original: Our final IPWD objective is formulated as:
Revised: Our final IPWD objective is formulated as:

Original: [Equation]
Revised: [Equation]

Original: where α is a trade-off hyper-parameter between classification and distillation.
Revised: where α is a trade-off hyper-parameter between classification and distillation.

Original: [Figure]

Original: ) honeycomb apiary bee hen cock blackgrouse canoe paddle snorkel

Original: [Figure]

Original: [Figure]

Original: 𝑦 !"

Original: # 𝑦 !"

Original: [Figure]

Original: 𝑦 # 𝑦 # vulture

Original: [Figure]

Original: [Figure]

Original: ) (bottom

Original: [Figure]

Original: Africanwilddog hyena dhole racecar sportscar convertible ant groundbeetle tigerbeetle

Original: [Figure]

Original: [Figure]

Original: # 𝑦 !"

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: seashore goose duck

Original: [Figure]

Original: violin acousticguitar

Original: [Figure]

Original: Limitations and negative societal impacts. .
Revised: Limitations and negative societal impacts. .

Original: As we introduced in Section 4.2, we estimate thepropensity by comparing the heads of the student model.
Revised: As we introduced in Section 4.3, we estimate thepropensity by comparing the heads of the student model.

Original: Therefore, the accuracy of estimation relieson the quality of the student model.
Revised: Therefore, the accuracy of estimation relieson the quality of the student model.

Original: A poor or small student may not be able to correctly estimate thepropensity, which may further suppress the effectiveness of IPWD.
Revised: A poor or small student may not be able to correctly estimate thepropensity, which may further suppress the effectiveness of IPWD.

Original: Results in Table 6 also confirmthis point. Also, we hold an assumption that the training and test samples are IID in the humandomain, which may not be valid for long-tailed tasks.
Revised: Also, we hold an assumption thatthe training and test samples are IID in the human domain, which may not be valid for long-tailedtasks. Besides, we found that our IPWD does not work with feature-based distillation methods (see Appendix for more results).

Original: To the best of our knowledge, as our work ispurely an algorithm for knowledge distillation, we haven’t found any negative societal impact.
Revised: To the best of our knowledge, as our work is purely an algorithm forknowledge distillation, we haven’t found any negative societal impact.

Original:  Experiments
Revised:  Experiments

Original: We take the image classification task as a case study to evaluate the effectiveness and generalizabilityof our IPWD.
Revised: We take the image classification task as a case study to evaluate the effectiveness and generalizabilityof our IPWD.

Original: Following previous works [54, 69, 25], we conduct experiments with two settings,two-stage distillation and one-stage self-distillation. 
Revised: Following previous works [54, 69, 25], we conduct experiments with two settings,two-stage distillation and one-stage self-distillation. [Table]

Original: 5.1 Datasets and Settings
Revised: 5.1 Datasets and Settings

Original: Datasets .
Revised: Datasets .

Original: We conducted experiments on CIFAR-100 [26] and ImageNet [10].
Revised: We conducted experiments on CIFAR-100 [26] and ImageNet [10].

Original: CIFAR-100 contains50K images in the training set and 10K images in the test set from 100 classes.
Revised: CIFAR-100 contains50K images in the training set and 10K images in the test set from 100 classes.

Original: ImageNet provides1.2M images in the training set and 50K images in the validation set from 1K classes.
Revised: ImageNet provides1.2M images in the training set and 50K images in the validation set from 1K classes.

Original: Settings .
Revised: Settings .

Original: We considered two settings for knowledge distillation, two-stage distillation and one-stageself-distillation.
Revised: We considered two settings for knowledge distillation, two-stage distillation and one-stageself-distillation.

Original: Two-stage distillation is the conventional setting that pre-trains a teacher model atthe first stage and transfers the knowledgefrom the teacher to a student model at the second stage.
Revised: Two-stage distillation is the conventional setting that pre-trains a teacher model atthe first stage and transfers the knowledge to a student model at the second stage.

Original: Commonly, the teacher is a larger model, and the student is a smaller model.
Revised: Commonly, theteacher is a larger model, and the student is a smaller model.

Original: For self-distillation, theteacher and student have the same architecture.
Revised: For self-distillation, the teacher andstudent have the same architecture.

Original: As a comparison, one-stage self-distillation aims tocomplete the teacher-student optimization simultaneously [61, 25], i.e. , the pre-training and transferprocesses are reduced to one.
Revised: One-stage self-distillation aims to complete the teacher-studentoptimization simultaneously [61, 25], i.e. , the pre-training and transfer processes are reduced to one.

Original: 5.2 Two-stage Distillation
Revised: 5.2 Two-stage Distillation

Original: Baseline methods .
Revised: Baseline methods .

Original: For two-stage distillation, following Tian et al.
Revised: For two-stage distillation, following Tian et al.

Original: [54] and Zhou et al.
Revised: [54] and Zhou et al.

Original: [69], weconsidered the following methods as baselines: KD [18], FitNet [45], AT [65], SP [56], CC [43],
Revised: [69], weconsidered the following methods as baselines: KD [18], FitNet [45], AT [65], SP [56], CC [43],

Original: VID [2], RKD [41], PKT [42], FSP [62], AB [17], FT [24], NST [20], CRD [54], SSKD [60], and
Revised: VID [2], RKD [41], PKT [42], FSP [62], AB [17], FT [24], NST [20], CRD [54], SSKD [60], and

Original: WSLD [69].
Revised: WSLD [69].

Original: In particular, WSLD [69] is the most related work to us, which proposed a bias-variancetrade-off perspective for KD and also assigns different weights to each training sample.
Revised: In particular, WSLD [69] is the most related work to us, which proposed a bias-variancetrade-off perspective for KD and also assigns different weights to each training sample.

Original: Similarly, theweight is positive related to the cross-entropy loss of student’s output.
Revised: Similarly, theweight is positive related to the cross-entropy loss of student’s output.

Original: The main differences betweenour IPWD and WSLD are as follows.
Revised: The main differences betweenour IPWD and WSLD are as follows.

Original: First, our formulation of samples weights is theoreticallyguaranteed by the causal theory behind Inverse Probability Weighting (IPW) [46, 31, 21, 5].
Revised: First, our formulation of samples weights is theoreticallyguaranteed by the causal theory behind Inverse Probability Weighting (IPW) [46, 31, 21, 5].

Original: Second,
Revised: Second,

Original: [Table] [Table] estimates the sample weight using both the student model and the teacher model.
Revised: WSLD estimates the sample weight using both the student model and the teacher model.

Original: As acomparison, we use the student model with two different classifier heads to guarantee that thecapacities of the compared models are close.
Revised: As acomparison, we use the student model with two different classifier heads to guarantee that thecapacities of the compared models are close.

Original: Implementation .
Revised: Implementation .

Original: For experiments on CIFAR-100, we followed CRD [54] based on the open-sourcedcode.
Revised: For experiments on CIFAR-100, we followed CRD [54] based on the open-sourcedcode.

Original: We set the trade-off hyper-parameter α = 5 in Eq.
Revised: We set the trade-off hyper-parameter α = 5 in Eq.

Original: (9) and the temperature τ = 10 .
Revised: (7) and the temperature τ = 10 .

Original: Othertraining details were the same as CRD [54] and provided in the appendix.
Revised: Othertraining details were the same as CRD [54] and provided in the appendix.

Original: For ImageNet, we followed
Revised: For ImageNet, we followed

Original: Zhou et al.
Revised: Zhou et al.

Original: [69] to conduct experiments based on their open-sourced code.
Revised: [69] to conduct experiments based on their open-sourced code.

Original: We used the same hyperparameters as WSLD [69], i.e. , α as 2.5 and τ as 2.
Revised: We used the same hyperparameters as WSLD [69], i.e. , α as 2.5 and τ as 2.

Original: Comparison with baseline methods .
Revised: Comparison with baseline methods .

Original: Table 1 shows the results of student models on CIFAR-100with twelve teacher-student architectures, which can be grouped into same architecture style anddifferent architecture style.
Revised: Table 2 shows the results of student models on CIFAR-100with twelve teacher-student architectures, which can be grouped into same architecture style anddifferent architecture style.

Original: Note that the results of WSLD reported in [69] used a different pre-trainedteacher model.
Revised: Note that the results of WSLD reported in [69] used a different pre-trainedteacher model.

Original: Since some training techniques like label smoothing [48] and early-stoping [8, 11, 63]may improve the distillation performance, we reimplemented WSLD using the same teacher modelfor a fair comparison.
Revised: Since some training techniques like label smoothing [48] and early-stoping [8, 11, 63]may improve the distillation performance, we reimplemented WSLD using the same teacher model [Table] for a fair comparison.

Original: Overall, our IPWD outperforms KD by large margins and outperforms otherbaseline methods on most of the architectures, which demonstrates the effectiveness of our IPWD.
Revised: Overall, our IPWD outperforms KD by large margins and outperforms otherbaseline methods on most of the architectures, which demonstrates the effectiveness of our IPWD.

Original: Inparticular, the improvement with the same architecture style is smaller than the different architecturestyle.
Revised: Inparticular, the improvement with the same architecture style is smaller than the different architecturestyle.

Original: The reason is that the different architecture style reflects the huge gap between the humandomain and machine domain.
Revised: The reason is that the different architecture style reflects the huge gap between the humandomain and machine domain.

Original: Since our IPWD weights the training samples to address the non-IIDproblem and respect the transfer gap, IPWD successfully outperforms KD and other state-of-the-artmethods by large margins when the gap is significant.denotes the usage of an extra classification head. “logits
Revised: Since our IPWD weights the training samples to address the non-IIDproblem and respect the transfer gap, IPWD successfully outperforms KD and other state-of-the-artmethods by large margins when the gap is significant.

Original: [Table]
Revised: [Table]

Original: Note that SSKD [60] achieves higher performance because of (1) a better teacher model [Table] and (2) dataaugmentation for structured knowledge distillation.
Revised: Note that SSKD [60] achieves higher performance because of (1) a better teachermodel, and (2) data augmentation forstructured knowledge distillation.

Original: We further apply IPWD to SSKD as a plug-inby weighting the logit distillation objective and keeping the structured knowledge distillation termsunchanged.
Revised: Wefurther apply IPWD to SSKD as a plugin by weighting the logit distillation objective and keeping the structured knowledge distillation terms unchanged.

Original: Table 2 shows that our IPWD can further consistently improve SSKD by 0.5~1.0% fordifferent architectures.
Revised: Table 3 shows that our IPWD can furtherconsistently improve SSKD by 0.5~1.0%for different architectures.

Original: These results indicate that our IPWD is a good complementary to distillationmethods that are not limited to logit knowledge.[Table] NST [20] 70. FT [24] — — 69.
Revised: These resultsindicate that our IPWD is a good complementary to distillation methods thatare not limited to logit knowledge.

Original: FSP [62] 70.

Original: AB [17] — — 68.

Original: RKD [41] 70.

Original: KD [18] 70.

Original: Overhaul [16] 71.

Original: CRD [54] 71.

Original: SSKD [60] 71.

Original: DGKD [50] 71.

Original: WSLD [69] 72.

Original: IPWD 71.

Original: Table 3 further shows the comparisonon ImageNet.
Revised: Table 4 further shows the comparisonon ImageNet.

Original: Following CRD [54]and WSLD [69], we used two teacherstudent architectures as the representatives.
Revised: Following CRD [54]and WSLD [69], we used two teacherstudent architectures as the representatives.

Original: For the same architecture style,our IPWD improves KD by 1.21%, andachieves competitive performance compared to WSLD.
Revised: For the same architecture style,our IPWD improves KD by 1.21%, andachieves competitive performance compared to WSLD.

Original: For the different architecture style, the improvement of WSLDover KD drops from 1.37% to 1.03%.
Revised: For the different architecture style, the improvement of WSLDover KD drops from 1.37% to 1.03%.

Original: Asa comparison, our IPWD improves KDby 2.16%, and outperforms WSLD by1.13%.
Revised: As a comparison, our IPWD improves KD by 2.16%, andoutperforms WSLD by 1.13%.

Original: This improvement on the largescale dataset further demonstrates the effectiveness of our IPWD to bridge thetransfer gap when the student and teachermodel have different architecture styles,which is more practical in real-world applications.
Revised: This improvement on the large-scale dataset further demonstratesthe effectiveness of our IPWD to bridge the transfer gap when the student and teacher model havedifferent architecture styles, which is more practical in real-world applications.

Original: Ablation study: technical designs .
Revised: Ablation study: technical designs .

Original: Asintroduced in Section 4.2, we used an extra classifier head to produce CE-awareoutput, and normalized the logits to reduce the variance for propensity estimation.
Revised: As introduced in Section 4.3, we used an extra classifier head toproduce CE-aware output, and normalized the logits to reduce the variance for propensity estimation.

Original: Note that WSLD [69] uses theteacher model to estimate the sample weight, and the teacher model is also trained with the crossentropy (CE) loss.
Revised: Note that WSLD [69] uses the teacher model to estimate the sample weight, and the teacher modelis also trained with the cross-entropy (CE) loss.

Original: Therefore, we considered an alternative that which replaces the classification headwith the teacher model to produce the classification-aware output.
Revised: Therefore, we considered an alternative that whichreplaces the classification head with the teacher model to produce the classification-aware output.

Original: To evaluate the contribution oflogits normalization, we considered an alternative that the logits are not normalized by the standarddeviation.
Revised: To evaluate the contribution of logits normalization, we considered an alternative that the logitsare not normalized by the standard deviation.

Original: Results in Table 4 verify the contribution of each design.
Revised: Results in Table 5 verify the contribution of eachdesign.

Original: Without the classification headand logit normalization, the training is hard to converge or the performance is much worse.
Revised: Without the classification head and logit normalization, the training is hard to converge or theperformance is much worse.

Original: As acomparison, either the classification head or logit normalization helps with stable training.
Revised: As a comparison, either the classification head or logit normalizationhelps with stable training.

Original: Besides, acombination of both further improves the performance and achieves the best results.
Revised: Besides, a combination of both further improves the performance andachieves the best results.

Original: The crash oftraining is due to the high variance of sample weights.
Revised: The crash of training is due to the high variance of sample weights.

Original: Since  the teacher model is well pre-trained andhas more parameters, it has a larger capacity than the student model.
Revised: Since [Table] [Table] the teacher model is well pre-trained and has more parameters, it has a larger capacity than the studentmodel.

Original: Differently, an extra head with ashared backbone guarantees a similar capacity.
Revised: Differently, an extra head with a shared backbone guarantees a similar capacity.

Original: Also, the normalization will avoid an extremely largeor small CE loss, which further reduces the variance.[Table]
Revised: Also, thenormalization will avoid an extremely large or small CE loss, which further reduces the variance.

Original: 5.3 One-stage Self-Distillation
Revised: 5.3 One-stage Self-Distillation

Original: Baseline methods and metrics .
Revised: Baseline methods and metrics .

Original: For one-stage self-distillation, we apply our method to the stateof-the-art PS-KD [25] method as a plug-in, and consider label smoothing (LS) method and twoself-distillation methods, CS-KD [64] and TF-KD [63], as baselines.
Revised: For one-stage self-distillation, we apply our method to the stateof-the-art PS-KD [25] method as a plug-in, and consider label smoothing (LS) method and twoself-distillation methods, CS-KD [64] and TF-KD [63], as baselines.

Original: PS-KD proposed a one-stageframework that progressively distills the knowledge of a model itself to soften the one-hot supervisionsas regularization.
Revised: PS-KD proposed a one-stageframework that progressively distills the knowledge of a model itself to soften the one-hot supervisionsas regularization.

Original: The knowledge is transferred using a conventional distillation loss.
Revised: The knowledge is transferred using a conventional distillation loss.

Original: As for metrics,besides top-1 and top-5 accuracy, we follow Kim et al.
Revised: As for metrics,besides top-1 and top-5 accuracy, we follow Kim et al.

Original: [25] to report expected calibration error (ECE,%) and the area under the risk-coverage curve (AURC, × 10 3 ).
Revised: [25] to report expected calibration error (ECE,%) and the area under the risk-coverage curve (AURC, × 10 3 ).

Original: A low ECE indicates well-calibratedpredictions, and a low AURC represents the well-separation of correct and incorrect predictions.[Table]
Revised: A low ECE indicates well-calibratedpredictions, and a low AURC represents the well-separation of correct and incorrect predictions.

Original: Implementation .

Original: We follow all thetraining details of PS-KD for a fair comparison.

Original: Since both the student andteacher models are poor at early epochs,the weight estimation is not accurateat early epochs, which may lead to aworse self-teacher.

Original: Therefore, we apply

Original: IPWD at the last 75 epochs over thetotal 300 epochs.

Original: Comparison with baseline methods .
Revised: Comparison with baseline methods .

Original: Table 5 shows the results of one-stage self-distillation methodsover four architectures.
Revised: Table 6 shows the results of one-stage self-distillation methodsover four architectures.

Original: Our IPWD can effectively and constantly improve the top-1 accuracy of
Revised: Our IPWD can effectively and constantly improve the top-1 accuracy of

Original: PS-KD by 0.33% ∼ 0.82% with different architectures.
Revised: PS-KD by 0.33% ∼ 0.82% with different architectures.

Original: Besides, our IPWD siginifantly lowers the
Revised: Besides, our IPWD siginifantly lowers the

Original: ECE and AURC of PS-KD.
Revised: ECE and AURC of PS-KD.

Original: These results demonstrate the effectiveness of our IPWD.
Revised: These results demonstrate the effectiveness of our IPWD.

Original: Ablation study: IPWD stage .

Original: We conduct an ablation study to analyze whether IPWD should bestarted from an early stage ( e.g. , the beginning of training) or a late stage ( e.g. , last 1/4 of the epochs).

Original: We take ResNeXt-29 as an example.

Original: As shown in Table 6, applying IPWD from the beginningslightly outperforms PS-KD and under-performs the student modal that applies IPWD only at thelate stage by large margins.

Original: As the student model is poorly trained at the early stage, the weightestimation is inaccurate and hurts the performance of self-teacher.

Original: These results indicate that thequality of estimated weight and distillation performance relies on the student model and self-teacher.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we we point out that conventional KD methods hold an invalid IID assumption anddoes not properly address the transfer gap between the context-invariant human domain and thecontext-equivariant machine domain, especially the imbalance knowledge of the teacher model onthe transfer set.
Revised: In this paper, we we point out that conventional KD methods hold an invalid IID assumption anddoes not properly address the transfer gap between the context-invariant human domain and thecontext-equivariant machine domain, especially the imbalance knowledge of the teacher model onthe transfer set.

Original: We further proposed a simple yet effective method, Inverse Probability Weighting
Revised: We further proposed a simple yet effective method, Inverse Probability Weighting

Original: Distillation (IPWD), to deal with the imbalanced knowledge caused by transfer gap.
Revised: Distillation (IPWD), to deal with the imbalanced knowledge caused by transfer gap.

Original: In the future,we will extend our IPWD to (1) tasks beyond classification, like detection and segmentation, and (2)long-tailed tasks where the training samples in the human domain are also non-IID.
Revised: In the future,we will extend our IPWD to (1) tasks beyond classification, like detection and segmentation, and (2)long-tailed tasks where the training samples in the human domain are also non-IID.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See Section 4.
Revised: [Yes] See Section 4.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] SeeAppendix.(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: [Yes] SeeAppendix.(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]3.
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes] Please refer to
Revised: [Yes] Please refer to

Original: the supplemental material.
Revised: the supplemental material.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes] The details are in Section 5.
Revised: [Yes] The details are in Section 5.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running exper-
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running exper-

Original: iments multiple times)?
Revised: iments multiple times)?

Original: [Yes] We run the experiments with five different randomseeds.
Revised: [Yes] We run the experiments with five different random seeds.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Appendix.
Revised: [Yes] See Appendix.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] See Appendix.
Revised: [Yes] See Appendix.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Original: The code is attached.
Revised: The code is attached.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: [Yes
Revised: [Table]

Original:  All the used existing assets are open-sourced for research.e) Did you discuss
Revised: using/curating? [Yes] All the used existing assets are open-sourced for research.

Original: whether
Revised: [Table]

Original:  the data you are using curating contains personally 
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: identifiable
Revised: [Table]

Original: information or offensive content?
Revised: information or offensive content?

Original: [Yes] The benchmark dataset does not contain 
Revised: [Yes] The benchmark dataset does not contain [Table]

Original: personally identifiable information or offensive content.
Revised: personally identifiable information or offensive content.

Revised: [Table]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original:  (a) Did you include the full text of instructions given to participants and screenshots, if 
Revised: [Table] (a) Did you include the full text of instructions given to participants and screenshots, if [Table]

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original:  (b) Did you describe any potential participant risks, with links to Institutional Review 
Revised: [Table] (b) Did you describe any potential participant risks, with links to Institutional Review [Table]

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original:  (c) Did you include the estimated hourly wage paid to participants and the total amount 
Revised: [Table] (c) Did you include the estimated hourly wage paid to participants and the total amount [Table]

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Revised: [Table]