Original: Teach Less, Learn More: On the Undistillable Classes in Knowledge Distillation
Revised: Teach Less, Learn More: On the Undistillable Classes in Knowledge Distillation

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Knowledge distillation (KD) can effectively compress neural networks by traininga smaller network (student) to simulate the behavior of a larger one (teacher).
Revised: Knowledge distillation (KD) can effectively compress neural networks by traininga smaller network (student) to simulate the behavior of a larger one (teacher).

Original: A counter-intuitive observation is that a more expansive teacher does not makea better student, but the reasons for this phenomenon remain unclear.
Revised: A counter-intuitive observation is that a more expansive teacher does not makea better student, but the reasons for this phenomenon remain unclear.

Original: In thispaper, we demonstrate that this is directly attributed to the presence of undistillableclasses : when trained with distillation, the teacher’s knowledge of some classesis incomprehensible to the student model.
Revised: In thispaper, we demonstrate that this is directly attributed to the presence of undistillableclasses : when trained with distillation, the teacher’s knowledge of some classesis incomprehensible to the student model.

Original: We observe that while KD improvesthe overall accuracy, it is at the cost of the model becoming inaccurate in theseundistillable classes.
Revised: We observe that while KD improvesthe overall accuracy, it is at the cost of the model becoming inaccurate in theseundistillable classes.

Original: After establishing their widespread existence in state-ofthe-art distillation methods, we illustrate their correlation with the capacity gap between teacher and student models.
Revised: After establishing their widespread existence in state-ofthe-art distillation methods, we illustrate their correlation with the capacity gap between teacher and student models.

Original: Finally, we present a simple “Teach Less
Revised: Finally, we present a simple “Teach Less

Original: Learn More” (TLLM) framework to identify and discard the undistillable classesduring training.
Revised: Learn More” (TLLM) framework to identify and discard the undistillable classesduring training.

Original: We validate the effectiveness of our approach on multiple datasetswith varying network architectures.
Revised: We validate the effectiveness of our approach on multiple datasetswith varying network architectures.

Original: In all settings, our proposed method is able toexceed the performance of competitive state-of-the-art techniques.
Revised: In all settings, our proposed method is able toexceed the performance of competitive state-of-the-art techniques.

Original:  Introduction
Revised:  Introduction

Original: More accurate machine learning models often demand more computation and memory at test time,making them difficult to deploy on computational-constraint devices.
Revised: More accurate machine learning models often demand more computation and memory at test time,making them difficult to deploy on computational-constraint devices.

Original: Knowledge distillation (KD)alleviates this burden by transferring the knowledge from an expensive teacher model to a cheapstudent model to improve the performance of this more efficient network.
Revised: Knowledge distillation (KD)alleviates this burden by transferring the knowledge from an expensive teacher model to a cheapstudent model to improve the performance of this more efficient network.

Original: Over the past severalyears, we have witnessed a huge success in knowledge distillation and a surge of related literature ondesigning better distillation techniques.
Revised: Over the past severalyears, we have witnessed a huge success in knowledge distillation and a surge of related literature ondesigning better distillation techniques.

Original: An intuitive thought of knowledge distillation is that better teachers are supposed to make betterstudents since a larger model can learn more visually meaningful and task-related information and astudent model can presumably benefit from learning with more informative knowledge.
Revised: An intuitive thought of knowledge distillation is that better teachers are supposed to make betterstudents since a larger model can learn more visually meaningful and task-related information and astudent model can presumably benefit from learning with more informative knowledge.

Original: However,recent works show that such intuition is incorrect.
Revised: However,recent works show that such intuition is incorrect.

Original: In fact, prior works found that students distilledfrom a bigger teacher, one with more parameters and higher accuracy, can perform worse than is getting more severe a large researches [52, 32,50,
Revised: In fact, prior works found that students distilledfrom a bigger teacher, one with more parameters and higher accuracy, can perform worse than the same students distilled from a smaller teacher [19, 7, 32,

Revised: The situation is getting more severe when training on a large-scale, challenging dataset such as ImageNet [3].

Revised: More surprisingly, the model with the same architecture [43] or even a model with lower performance [15, can be used as a teacher network to perform knowledge distillation which plays a role of regularization [28, 20,

Original: 31] devoted to solving this problem but few minimal  been put into understanding this phenomenon
Revised: Indeed a large body of researches [49, 31, 47, 14, 30] devoted to solving this problem but few minimal efforts have been put into understanding this phenomenon

Original: In this paper, we propose a new data-centric perspective on the phenomenon of “larger teacher, worsestudent”.
Revised: In this paper, we propose a new data-centric perspective on the phenomenon of “larger teacher, worsestudent”.

Original: In contrast to the previous works, we cast the inefficacy of large teachers as a fundamentalconsequence of the class-dependent bias in KD.
Revised: In contrast to the previous works, we cast the inefficacy of large teachers as a fundamentalconsequence of the class-dependent bias in KD.

Original: Specifically, we claim that:
Revised: Specifically, we claim that:

Original: The undistillable classes are a direct result of the inefficacy of large teachers in distillation.
Revised: The undistillable classes are a direct result of the inefficacy of large teachers in distillation.

Original: Recall that prior works attribute the failure of large teachers in KD to the capacity mismatch betweenthe teacher and student model.
Revised: Recall that prior works attribute the failure of large teachers in KD to the capacity mismatch betweenthe teacher and student model.

Original: In other words, the student model cannot learn the teacher’s knowledgedue to their insufficient expressive power.
Revised: In other words, the student model cannot learn the teacher’s knowledgedue to their insufficient expressive power.

Original: Indeed, we find that their claim is true for some classesthat posses bad distillability .
Revised: Indeed, we find that their claim is true for some classesthat posses bad distillability .

Original: We posit that these classes cause seemingly mismatched capacity overall samples in the dataset.
Revised: We posit that these classes cause seemingly mismatched capacity overall samples in the dataset.

Original: The empirical study supports our hypothesis.
Revised: The empirical study supports our hypothesis.

Original: For samples from the undistillable classes, the featurerepresentation similarity between the teacher and student model is extremely low compared tothe regular classes.
Revised: For samples from the undistillable classes, the featurerepresentation similarity between the teacher and student model is extremely low compared tothe regular classes.

Original: Furthermore, perhaps less surprisingly, the distilled model could obtain loweraccuracy on these undistillable classes than their vanilla-trained counterpart.
Revised: Furthermore, perhaps less surprisingly, the distilled model could obtain loweraccuracy on these undistillable classes than their vanilla-trained counterpart.

Original: To further corroborateour theory, we show that the number of undistillable classes increases when the capacity gap betweenthe two models increases.
Revised: To further corroborateour theory, we show that the number of undistillable classes increases when the capacity gap betweenthe two models increases.

Original: After verifying our hypothesis, a natural question would be raised: to what extent do these undistillableclasses exist?
Revised: After verifying our hypothesis, a natural question would be raised: to what extent do these undistillableclasses exist?

Original: When discussing the property of knowledge distillation, previous literature typicallyevaluates vanilla knowledge distillation [13].
Revised: When discussing the property of knowledge distillation, previous literature typicallyevaluates vanilla knowledge distillation [12].

Original: However, considering the evolution of distillationtechniques over the past several years, we believe it is necessary to expand our study.
Revised: However, considering the evolution of distillationtechniques over the past several years, we believe it is necessary to expand our study.

Original: To this end, weinvestigate the existence of the undistillable classes over three standard datasets (CIFAR100 [19],
Revised: To this end, weinvestigate the existence of the undistillable classes over three standard datasets (CIFAR100 [18],

Original: ImageNet1K [5], CUB-200 [39]) with more than 20 modern distillation techniques.
Revised: ImageNet1K [4], CUB-200 [38]) with more than 20 modern distillation techniques.

Original: We also includethe evaluation of advanced architectures, i.e., vision transformer [36].
Revised: We also includethe evaluation of advanced architectures, i.e., vision transformer [35].

Original: Our study shows that theundistillable classes generally exist, usually taking over from 10% to 50% of the total classes,depending on the distillation methods, datasets, and teacher-student pairs.
Revised: Our study shows that theundistillable classes generally exist, usually taking over from 10% to 50% of the total classes,depending on the distillation methods, datasets, and teacher-student pairs.

Original: To resolve this issue, we present a novel distillation framework called “Teach Less, Learn More”distillation (TLLM).
Revised: To resolve this issue, we present a novel distillation framework called “Teach Less, Learn More”distillation (TLLM).

Original: It is motivated by the successful teaching strategy in pedagogy[34], whereteachers are encouraged to do less rote learning and provide more spaces for students to explore anddiscover their talents.
Revised: It is motivated by the successful teaching strategy in pedagogy[33], whereteachers are encouraged to do less rote learning and provide more spaces for students to explore anddiscover their talents.

Original: Our methodology is conceptually similar: we aim to identify the undistillableclasses and remove them entirely during distillation, allowing the student model to explore therepresentations of these classes with their own capacity.
Revised: Our methodology is conceptually similar: we aim to identify the undistillableclasses and remove them entirely during distillation, allowing the student model to explore therepresentations of these classes with their own capacity.

Original: Specifically, we record the per-class teachingcurve (distillation losses) and calculate its change over a moving window of K epochs.
Revised: Specifically, we record the per-class teachingcurve (distillation losses) and calculate its change over a moving window of K epochs.

Original: If the teachingcurve for some particular classes is stagnated or upward, we consider the knowledge transfer processfor these classes is no longer necessary.
Revised: If the teachingcurve for some particular classes is stagnated or upward, we consider the knowledge transfer processfor these classes is no longer necessary.

Original: Then, we remove these classes in distillation (if it is an outputbased KD, we replace them with the label smoothing technique to compensate for the regularizationeffect).
Revised: Then, we remove these classes in distillation (if it is an outputbased KD, we replace them with the label smoothing technique to compensate for the regularizationeffect).

Original: Our approach, despite its simplicity, is proven to be effective in preventing the adverse effectbrought by the undistillable classes and improving the overall accuracy.
Revised: Our approach, despite its simplicity, is proven to be effective in preventing the adverse effectbrought by the undistillable classes and improving the overall accuracy.

Original: Overall, we emphasize our contributions are the following: 1) we analytically show that the undistillable classes are the cause of inefficacy of the large teacher model; 2) we demonstrate that the accuracyof these undistillable classes drops after distillation, and our extensive experiments showing this is ageneral phenomenon, irrespective of distillation methods, datasets, or teacher-student pairs; 3) wepropose a simple yet effective framework to identify and discard undistillable classes and illustratethe effectiveness our approach with extensive experiments.
Revised: Overall, we emphasize our contributions are the following: 1) we analytically show that the undistillable classes are the cause of inefficacy of the large teacher model; 2) we demonstrate that the accuracyof these undistillable classes drops after distillation, and our extensive experiments showing this is ageneral phenomenon, irrespective of distillation methods, datasets, or teacher-student pairs; 3) wepropose a simple yet effective framework to identify and discard undistillable classes and illustratethe effectiveness our approach with extensive experiments.

Original:  On the Undistillable Classes in Knowledge Distillation2.1 Formal Definition
Revised:  On the Undistillable Classes in Knowledge Distillation2.1 Formal Definition

Original: In this section, we formally define the undistillable classes.
Revised: In this section, we formally define the undistillable classes.

Original: Considering an multi-class predictor
Revised: Considering an multi-class predictor

Original: F : R d → R k as student and another predictor G : R d → R k as teacher.
Revised: F : R d → R k as student and another predictor G : R d → R k as teacher.

Original: Giving a traing dataset
Revised: Giving a training sample

Original: S = { ( x i , y i ) ni
Revised: S = { ( x i , y i ) ni

Original: =1 } ∼ P n for distribution P over a set of instance X and label Y =
Revised: =1 } ∼ P n for distribution P over a set of instance X and label Y =

Original: [ L ] =
Revised: [ L ] =

Original: , L ] .
Revised: , L ] .

Original: In the standard training, the goal is to approximate the risk R ( f ; S ) of functional class f ∈ F via the
Revised: In the standard training, the goal is to approximate the risk R ( f ; S ) of functional class f ∈ F via the

Original: For student network, the samples from undistillable classes clearly fail to match the representation from teacher model empirical risk ˆ R ( f ; S )
Revised: For student network, the samples from undistillable classes clearly fail to match the representation from teacher model. empirical risk ˆ R ( f ; S )

Original: = 1 N (cid:80) n l ( y , f ( x )) , for any bounded loss.
Revised: = 1 N (cid:80) n l ( y , f ( x )) , for any bounded loss.

Original: For knowledge distillation, we canrewrite the distilled risk as ˜ R ( f ; S )
Revised: For knowledge distillation, we canrewrite the distilled risk as ˜ R ( f ; S )

Original: = 1 N (cid:80) n l ( g ( x ) , f ( x )) , where the functional class g ∈ G .
Revised: = 1 N (cid:80) n l ( g ( x ) , f ( x )) , where the functional class g ∈ G .

Original: The conventional setting consider all classes equally.
Revised: The conventional setting consider all classes equally.

Original: Without loss of generality, we will consider the
Revised: Without loss of generality, we will consider the

Original: S =
Revised: S =

Original: S r (cid:83) S u ∼ P n , where we split the data over distribution P into two categories, each categorycontains non-overlapping instances S u
Revised: S r (cid:83) S u ∼ P n , where we split the data over distribution P into two categories, each categorycontains non-overlapping instances S u

Original: = { ( x i , y ui ) ni =1 } , Y u =
Revised: = { ( x i , y ui ) ni =1 } , Y u =

Original: [ L ] =
Revised: [ L ] =

Original: { L n +1 , L n +2 , · · · , L } , S r =
Revised: { L n +1 , L n +2 , · · · , L } , S r =

Original: S \ S u and | ·
Revised: S \ S u and | ·

Original: | c be a appropriate distillability measurement.
Revised: | c be a appropriate distillability measurement.

Original: In our context, we define the distillabilityfunction | · | c as the difference in performance between the distilled student model with the trainfrom-scratch counterpart under certain evaluation metrics.

Original: Let E ( · ) be an evaluation metric thatmeasures the performance of a particular multi-class predictor on S .

Original: Then, formally, we define| · | c = E ( F kd ) − E ( F vanilla ) .

Original: Finally, we define that S u are a set of samples from the undistillableclasses in distillation iff | f ; S u | c ≪ | f ; S r | c .
Revised: We define that S u are a set of samplesfrom the undistillable classes in distillation iff | f ; S u | c ≪ | f ; S r | c .

Original: 2.2 Mismatched Representation for Undistillable Classes
Revised: 2.2 Mismatched Representation for Undistillable Classes

Original: The previous section formally defines the undistillable classes as classes with bad distillability.
Revised: The previous section formally defines the undistillable classes as classes with bad distillability.

Original: Inthis section, we develop a criterion to measure the distillability of a class quantitatively.
Revised: Inthis section, we develop a criterion to measure the distillability of a class quantitatively.

Original: Recallthat the student is typically encouraged to mimic the behavior of the teacher model.
Revised: Recallthat the student is typically encouraged to mimic the behavior of the teacher model.

Original: Therefore, aconverged student should impeccably match the teacher’s behavior in an ideal setting.
Revised: Therefore, aconverged student should impeccably match the teacher’s behavior in an ideal setting.

Original: It impliesthat their feature representations should be the same as the teachers’, given the same input on bothmodels.
Revised: It impliesthat their feature representations should be the same as the teachers’, given the same input on bothmodels.

Original: As a result, we harness the similarity of feature representations between teacher and studentto reflect the level of distillability of the student model.
Revised: As a result, we harness the similarity of feature representations between teacher and studentto reflect the level of distillability of the student model.

Original: Inspired by Zhu et al.
Revised: Inspired by Zhu et al.

Original: [52], we use the
Revised: [49], we use the

Original: Center Kernel Alignment (CKA) [18] to calculate the representation similarity between two neuralnetworks.
Revised: Center Kernel Alignment (CKA) [17] to calculate the representation similarity between two neuralnetworks.

Original: To create a typical case of capacity mismatch, we follow the previous works [4] to usea ResNet56 as our pre-trained teacher model and a modified ResNet24 as the student model (see
Revised: To create a typical case of capacity mismatch, we follow the previous works [3] to usea ResNet56 as our pre-trained teacher model and a modified ResNet24 as the student model (see

Original: Appendix for details).
Revised: Appendix for details).

Original: We perform standard feature knowledge distillation [28] on stage four, whichcontains nine consecutive convolutional blocks.
Revised: We perform standard feature knowledge distillation [27] on stage four, whichcontains nine consecutive convolutional blocks.

Original: We choose to compare the network representation atconvergence.
Revised: We choose to compare the network representation atconvergence.

Original: For the CKA score figure, we primarily focus the score on the diagonal, which indicatesthe representation similarity of the convolutional layer at the same position in the network.
Revised: For the CKA score figure, we primarily focus the score on the diagonal, which indicatesthe representation similarity of the convolutional layer at the same position in the network.

Original: In orderto compare the representation similarity of samples from different types of classes, we deliberatelychoose 12 test samples from distinct classes.
Revised: In orderto compare the representation similarity of samples from different types of classes, we deliberatelychoose 12 test samples from distinct classes.

Original: Note that we use samples from the test set to ensure thatthe observed behavior is not due to ad-hoc memorization.
Revised: Note that we use samples from the test set to ensure thatthe observed behavior is not due to ad-hoc memorization.

Original: In the middle of Figure 1, the feature similarity between teacher and student on each correspondinglayer is presented.
Revised: In the middle of Figure 1, the feature similarity between teacher and student on each correspondinglayer is presented.

Original: We can observe that the CKA scores for most layers are less than 0.6, and forsome layers are even less than 0.5.
Revised: We can observe that the CKA scores for most layers are less than 0.6, and forsome layers are even less than 0.5.

Original: These results indicate that despite the student indeed learningto match the teacher’s representations, their features are far from perfect alignment.
Revised: These results indicate that despite the student indeed learningto match the teacher’s representations, their features are far from perfect alignment.

Original: Notably, therepresentation similarity score is calculated based on the average score over a batch.
Revised: Notably, therepresentation similarity score is calculated based on the average score over a batch.

Original: Therefore, wedecouple the matrices and check the CKA score for each sample individually.
Revised: Therefore, wedecouple the matrices and check the CKA score for each sample individually.

Original: On the left side andright side of Figure 1, we give the CKA matrices at the last three layers for 12 samples.
Revised: On the left side andright side of Figure 1, we give the CKA matrices at the last three layers for 12 samples.

Original: Each  side 
Revised: Each sidecontains six samples. We notice a clear trend: the CKA scores for samples from the left side areobviously much lower than those on the right side, indicating that the mismatched representationsare not uniformly distributed over all samples. Therefore, we name the classes on the left side asundistillable classes . Our observation contradicts the conclusion from the prior work, which attributes

Original: - - -     -15 - -    15 ∆ A cc ( % ) ∆PA
Revised: - - -     -15 - -    15 ∆ A cc ( % ) ∆PA

Original: T:ResNet32x4 S: ResNet8x4 Pearson‘s R = 0.
Revised: T:ResNet32x4 S: ResNet8x4 Pearson‘s R = 0.

Original: - - -     -15 - -    15 ∆ A cc ( % ) ∆PA
Revised: - - -     -15 - -    15 ∆ A cc ( % ) ∆PA

Original: T:ResNet50 S: MobileNetV2 Pearson‘s R = 0.
Revised: T:ResNet50 S: MobileNetV2 Pearson‘s R = 0.

Original: - - -     -15 - -    15 ∆ A cc ( % ) ∆PA
Revised: - - -     -15 - -    15 ∆ A cc ( % ) ∆PA

Original: T:ResNet32x4 S: ShuffleNetV1 Pearson‘s R = 0.
Revised: T:ResNet32x4 S: ShuffleNetV1 Pearson‘s R = 0.

Original: contains six samples. We notice a clear trend: the CKA scores for samples from the left side areobviously much lower than those on the right side, indicating that the mismatchedrepresentationsare not uniformly distributed over all samples. Therefore, we name the classes on the left side asundistillable classes . Our observation contradicts the conclusion from the prior work, which attributesthe mismatched capacity as a general issue for all data [52, 4, 20].
Revised: the mismatched capacity as a general issue for all data [49, 3, 19].

Original: We later show that at the left topof Figure 4, the accuracy for the undistillable classes drops after training with KD compared to theirtrain-from-scratch counterpart.
Revised: We later show that at the left topof Figure 4, the accuracy for the undistillable classes drops after training with KD compared to theirtrain-from-scratch counterpart.

Original: 2.3 Bad Distillability hurts Accuracy
Revised: 2.3 Bad Distillability hurts Accuracy

Original: [Figure]

Original: As aforementioned, a model would contain some classes with bad distillability.
Revised: As aforementioned, a model would contain some classes with bad distillability.

Original: But what is theconsequence of distilling knowledge to these classes?
Revised: But what is theconsequence of distilling knowledge to these classes?

Original: In this section, we show that the quality of KD(i.e., accuracy) is highly correlated to their distillability.
Revised: In this section, we show that the quality of KD(i.e., accuracy) is highly correlated to their distillability.

Original: Here we record the difference in accuracybetween the vanilla trained model (training without distillation) and the model trained with knowledgedistillation.
Revised: Here we record the difference in accuracybetween the vanilla trained model (training without distillation) and the model trained with knowledgedistillation.

Original: Particularly, we adopt per-class accuracy:
Revised: Particularly, we adopt per-class accuracy:

Original: [Equation]
Revised: [Equation]

Original:  where m ∈ Y .
Revised: (1) where j ∈ Y .

Original: The motivation behind this metric is straightforward: if the ∆ACC m > 0 , then form th class, the effect of KD is positive; otherwise, the KD hurts the student’s performance on m thclass.
Revised: The motivation behind this metric is straightforward: if the ∆ACC m > 0 , then form th class, the effect of KD is positive; otherwise, the KD hurts the student’s performance on m thclass.

Original: To demonstrate the correlation between per-class accuracy and per-class distillability, a naiveapproach is to compare the ∆ ACC with the CKA scores, as shown in the previous section.
Revised: To demonstrate the correlation between per-class accuracy and per-class distillability, a naiveapproach is to compare the ∆ ACC with the CKA scores, as shown in the previous section.

Original: However,
Revised: However,

Original: CKA measures per-layer feature representation similarity; simply averaging over the total number oflayers may be inaccurate, not to mention that the teacher model usually contains more layers thanstudents.
Revised: CKA measures per-layer feature representation similarity; simply averaging over the total number oflayers may be inaccurate, not to mention that the teacher model usually contains more layers thanstudents.

Original: Therefore, we present a new metric based on average prediction agreement:
Revised: Therefore, we present a new metric based on average prediction agreement:

Original: [Equation]
Revised: [Equation]

Original: where σ ( z t ) =
Revised: where σ ( z t ) =

Original: exp( z i ) (cid:80) j exp( z i ) and σ ( z s ) =
Revised: exp( z i ) (cid:80) j exp( z i ) and σ ( z s ) =

Original: exp( z s ) (cid:80) j exp( z s ) is the output of softmax function for teacher modeland student model, respectively.
Revised: exp( z s ) (cid:80) j exp( z s ) is the output of softmax function for teacher modeland student model, respectively.

Original: Equation 2 is inspired by the fact that if the feature representationsbetween two models are exactly the same, these two models ought to produce the same top-1 label.
Revised: Equation 2 is inspired by the fact that if the feature representationsbetween two models are exactly the same, these two models ought to produce the same top-1 label.

Original: Similar to ∆ Acc, here we focus on the difference in prediction agreement ∆ PA between two models;one is trained from scratch, and another is trained with distillation:
Revised: Similar to ∆ Acc, here we focus on the difference in prediction agreement ∆ PA between two models;one is trained from scratch, and another is trained with distillation:

Original: [Equation]
Revised: [Equation]

Original: Specifically, we put these two metrics in a scatter plot chart for three varying teacher-student pairson CIFAR100.
Revised: Specifically, we put these two metrics in a scatter plot chart for three varying teacher-student pairson CIFAR100.

Original: Figure 2 reveals that for most of the classes with ∆ACC ≤ 0 , its corresponding∆PA ≤ 0 as well, indicating that if the student become disagrees with the teacher’s prediction,the teacher’s supervision can hurt student’s performance on this particular class.
Revised: Figure 2 reveals that for most of the classes with ∆ACC ≤ 0 , its corresponding∆PA ≤ 0 as well, indicating that if the student become disagrees with the teacher’s prediction,the teacher’s supervision can hurt student’s performance on this particular class.

Original: In addition, wecan observe that the level of disagreement is linearly correlated to the level of performance change: when measured with the Pearson’s r, it obtained over 0.65 with a p-value less than 0.0005 on allthree teacher-student combination, showing a statistically significant linear correlation between thesetwo metrics.
Revised: In addition, wecan observe that the level of disagreement is linearly correlated to the level of performance change:when measured with the Pearson’s r, it obtained over 0.65 with a p-value less than 0.0005 on allthree teacher-student combination, showing a statistically significant linear correlation between thesetwo metrics.

Original: In conclusion, for any class in the student model, if it is undistillable, it certainly has∆ACC m < 0 .
Revised: In conclusion, for any class in the student model, if it is undistillable, it certainly has∆ACC m < 0 .

Original: 2.4 The Larger the Capacity Gap, the More the Undistillable Classes
Revised: The Larger the Capacity Gap, the More the Undistillable Classes

Original: Now, we redirect our attention to the correlation between the level of capacity gap and the presenceof the undistillable classes.
Revised: Now, we redirect our attention to the correlation between the level of capacity gap and the presenceof the undistillable classes.

Original: We propose to leverage multiple teacher networks with varying capacityto distilled a single, low-capacity student.
Revised: We propose to leverage multiple teacher networks with varying capacityto distilled a single, low-capacity student.

Original: Specifically, we select the ResNet20 as our student model,and pick ResNet44, ResNet56, and ResNet110 as three individual teacher model.
Revised: Specifically, we select the ResNet20 as our student model,and pick ResNet44, ResNet56, and ResNet110 as three individual teacher model.

Original: For each set ofexperiments, we adopt one teacher model from the above to train a student model via vanilla KD [13].
Revised: For each set ofexperiments, we adopt one teacher model from the above to train a student model via vanilla KD [12].

Original: This procedure mirrors that of mismatched capacity experiments in Cho et al.
Revised: This procedure mirrors that of mismatched capacity experiments in Cho et al.

Original: [4], except that our goalis to investigate the relationship of per-class accuracy with the capacity gap.
Revised: [3], except that our goalis to investigate the relationship of per-class accuracy with the capacity gap.

Original: As shown in Figure 3,we observe a clear relation between increase of the number undistillable classes, as increase in thecapacity gap between models.
Revised: As shown in Figure 3,we observe a clear relation between increase of the number undistillable classes, as increase in thecapacity gap between models.

Original: Specifically, the number of total undistillable classes for the smalleststudent model (ResNet44) is almost tripled compare to the largest largest model (ResNet101).
Revised: Specifically, the number of total undistillable classes for the smalleststudent model (ResNet44) is almost tripled compare to the largest largest model (ResNet101).

Original: 2.5 On the Existence of the Undistillable Classes
Revised: 2.5 On the Existence of the Undistillable Classes

Original: The knowledge distillation is well-known for its “free” gain over compact models on average onan i.i.d test set.
Revised: The knowledge distillation is well-known for its “free” gain over compact models on average onan i.i.d test set.

Original: Modern distillation techniques only focus on improving the overall accuracy of thestudent model, with an intuition that the distillation is beneficial for all classes.
Revised: Modern distillation techniques only focus on improving the overall accuracy of thestudent model, with an intuition that the distillation is beneficial for all classes.

Original: Indeed, the overallaccuracy is paramount, but the per-class accuracy is also important in many cases.
Revised: Indeed, the overallaccuracy is paramount, but the per-class accuracy is also important in many cases.

Revised: For instance, theengineers may want to avoid their face recognition system mistakenly recognizing a black womanas a man with brown skin.

Original: Nevertheless, thedistillation method inevitably brings class-dependent bias into the student model.
Revised: Nevertheless, the distillation method inevitably brings class-dependentbias into the student model.

Original: As briefly shownin Figure 4, the distilled student model shows obvious class-dependent bias on all datasets.
Revised: As briefly shown in Figure 4, the distilled student model shows crystalclass-dependent bias on all datasets.

Original: It isalso worth noting that the accuracy drops dramatically for some undistillable classes.
Revised: It is also worth noting that the accuracy drops dramatically forsome undistillable classes.

Original: For instance,on ImageNet1K, there are 127 classes that lose over 4% accuracy compared to their vanilla trainedcounterpart, and more than nine classes lose over 10% on the test set.
Revised: For instance, on ImageNet1K, there are 127 classes that lose over 4%accuracy compared to their vanilla trained counterpart, and more than nine classes lose over 10% onthe test set.

Original: In recent years, the techniques of distillation have evolved dramatically, i.e., feature-based distillationand self-supervision-based distillation [35].
Revised: In recent years, the techniques of distillation have evolved dramatically, i.e., feature-based distillationand self-supervision-based distillation [34].

Original: Therefore, to ensure the undistillable classes are nota unique byproduct of the vanilla KD, we extend our experiments to include 26 state-of-the-artdistillation frameworks, each of them evaluated by five different teacher-student pairs on CIFAR100.
Revised: Therefore, to ensure the undistillable classes are nota unique byproduct of the vanilla KD, we extend our experiments to include 26 state-of-the-artdistillation frameworks, each of them evaluated by five different teacher-student pairs on CIFAR100.

Original: We categorize them into two groups (output-based KD and feature-based KD).
Revised: We categorize them into two groups (output-based KD and feature-based KD).

Original: Table 1 give a list of all [Table] methods that we used for evaluation.
Revised: Table 1 give a list of allmethods that we used for evaluation.

Original: We only present the results for 13 of them due to limited space;the complete analysis and more evaluation of ImageNet-1K can be found in Appendix.
Revised: We only present the results for 13 of them due to limited space;the complete analysis and more evaluation of ImageNet-1K can be found in Appendix.

Original: Figure 5shows the number of undistillable classes.
Revised: Figure 5shows the number of undistillable classes.

Original: It is surprising to show that all of them contain undistillableclasses.
Revised: It is surprising to show that all of them contain undistillable

Original:  For instance, the Resnet32x4:Resnet8x4 with vanilla KD has 52 undistillable classes, morethe half of the total classes.
Revised: [Table] classes. For instance, the Resnet32x4:Resnet8x4 with vanilla KD has 52 undistillable classes, morethe half of the total classes.

Original: Despite so many undistillable classes, all distillation methods improvethe vanilla student model, some by a large margin.
Revised: Despite so many undistillable classes, all distillation methods improvethe vanilla student model, some by a large margin.

Original: Such a phenomenon indicates that the effect ofknowledge distillation is non-uniform at the class level.
Revised: Such a phenomenon indicates that the effect ofknowledge distillation is non-uniform at the class level.

Original: Moreover, the proportion of the undistillableclasses is taking over from 10% to 50% over the total number of classes.
Revised: Moreover, the proportion of the undistillableclasses is taking over from 10% to 50% over the total number of classes.

Original: Therefore, it is critical andnecessary to control and reduce the number of undistillable classes in the student network.
Revised: Therefore, it is critical andnecessary to control and reduce the number of undistillable classes in the student network.

Original:  Teach Less, Learn More (TLLM): Identifying and Discarding theUndistillable Classes
Revised:  Teach Less, Learn More (TLLM): Identifying and Discarding theUndistillable Classes

Original: We have stressed the adverse effect and characterized the properties of the undistillable classes fromthe previous section.
Revised: We have stressed the adverse effect and characterized the properties of the undistillable classes fromthe previous section.

Original: To this end, we give a simple framework, named “Teach Less, Learn More”
Revised: To this end, we give a simple framework, named “Teach Less, Learn More”distillation (TLLM), which analogy to a successful teaching principal in the pedagogy, to build better KD algorithms. Our idea is conceptually similar to the TLLM in the pedagogy, where the studentis encouraged to have more autonomy and devise their own solutions to problems. In practice, thestudent model is supervised by both teacher’s knowledge and the ground truth labels.

Revised: Training with

Original: Resnet20x4:
Revised: Resnet20x4:

Original: Resnet8x
Revised: Resnet8x

Original: WRN-40-2: WRN-16Resnet50: MobileNetV
Revised: WRN-40-2: WRN-16Resnet50: MobileNetV

Original: Resnet32x4: ShuffleNetV
Revised: Resnet32x4: ShuffleNetV

Original: VGG13: VGG
Revised: VGG13: VGG

Original: K D
Revised: K D

Original: F i t N e t    40 >
Revised: F i t N e t    40 >

Original: A T
Revised: A T

Original: S P
Revised: S P

Original: F S P V I D
Revised: F S P V I D

Original: R K D
Revised: R K D

Original: P K T
Revised: P K T

Original: A B
Revised: A B

Original: F T C C
Revised: F T C C

Original: C R D N S T
Revised: C R D N S T

Original: Number of Undisillable Classes data
Revised: Number of Undisillable Classes data

Original: Hintlayers Outputlayer Inputlayers
Revised: Hintlayers Outputlayer Inputlayers

Original: Distill
Revised: Distill

Original: Teacher
Revised: Teacher

Original: Student
Revised: Student

Original: Inference Stage
Revised: Inference Stage

Original: Student a cc beforekd afterkd [Figure] cc beforekd afterkd [Figure]
Revised: Student a cc beforekd afterkd a cc beforekd afterkd class

Original: [Figure] ~ ~ ~ ~ ~Knowledge distillation class # distillation (TLLM), which analogy to a successful teaching principal in the pedagogy, to build better KD algorithms.
Revised: # “ Undistillable Class ” Regular Class ~ ~ ~ ~ ~

Original: Our idea is conceptually similar to the TLLM in the pedagogy, where the studentis encouraged to have more autonomy and devise their own solutions to problems.

Original: In practice, thestudent model is supervised by both teacher’s knowledge and the ground truth labels.

Original: Training withthe model ’s task-oriented objectives is critical to the eventual performance [13], and provably tocorrect the wrong gradient directions that are brought by the teacher’s mistaken knowledge [16].
Revised: Knowledge distillation class # the model’s task-oriented objectives is critical to the eventual performance [12], and provably tocorrect the wrong gradient directions that are brought by the teacher’s mistaken knowledge [15].

Original: Therefore, what we do is simple: identify the undistillable classes and then remove them from thedistillation process.
Revised: Therefore, what we do is simple: identify the undistillable classes and then remove them fromthe distillation process.

Original: As a result, the student can learn these classes directly from the ground truthlabels without interference.
Revised: As a result, the student can learn these classes directly from the groundtruth labels without interference.

Original: The critical challenge of TLLM is how to findthe undistillable classes during training.
Revised: The critical challenge of TLLM is how to find the undistillableclasses during training.

Original: A naiveyet computational expansive approach is firstto train the model to convergence with distillation and then retrain it by manually eliminatingthe undistillable classes according to the performance on the test set.
Revised: A naive yet computational expansive approach is first to train the model toconvergence with distillation and then retrain it by manually eliminating the undistillable classesaccording to the performance on the test set.

Original: Instead, we propose tomonitor the teaching curves (distillation losses)and set up a criterion to find the undistillableclasses during training that are more computationally efficient.
Revised: Instead, we propose to monitor the teaching curves(distillation losses) and set up a criterion to find the undistillable classes during training that are morecomputationally efficient.

Original: Specifically, we pay attention to the per-classteaching curve, i.e., teaching curve l ( y m , f ( x ))on m th class, instead of the averaged distillationloss over the entire datasets.
Revised: Specifically, we pay attention to the per-class [Figure] teaching curve, i.e., teaching curve l ( y m , f ( x ))on m th class, instead of the averaged distillation loss over the entire datasets.

Original: The teaching curve is calculated on the validation set.
Revised: Theteaching curve is calculated on the validationset.

Original: Then, we set a moving window over K epochs, which is started at S th epoch and end at ( S + K ) th epoch.
Revised: Then, we set a moving window over Kepochs, which is started at S th epoch and endat ( S + K ) th epoch.

Original: During the distillation process, we record and update the per-class teaching curve [ l S ( y m , f ( x )) , l S + K ( y m , f ( x ))] .
Revised: During the distillation process, we record and update the per-class teaching curve [ l S ( y m , f ( x )) , l S + K ( y m , f ( x ))] .

Original: [Table] If l S ( y m [Table] f ( x ))− l S + K ( y m
Revised: Ifl S ( y m , f ( x ))

Original: [Table]

Original:  f ( x )) ≤ η , 0 ≤ η [Table] meaning that the loss is moving upward in the
Revised: − l S + K ( y m , f ( x )) ≤ η , 0 ≤ η ,meaning that the loss is moving upward in the

Original: K epochs moving window.
Revised: K epochs moving window.

Original: Figure 3 gives an example of the teaching curve for an undistillableclass.
Revised: Figure 3 gives an example of the teaching curve for an undistillableclass.

Original: After training with approximately 50 epochs, the loss for this particular class suddenly jumps.
Revised: After training with approximately 50 epochs, the loss for this particular class suddenly jumps.

Original: Therefore, our method can capture such behavior, then identify and remove these undistillable classesduring training.
Revised: Therefore, our method can capture such behavior, then identify and remove these undistillable classesduring training.

Original: It is worth noting that these classes remain in the task-oriented objective for thestudent, making sure that the student can still learn from these data.
Revised: It is worth noting that these classes remain in the task-oriented objective for the [Table] student, making sure that the student can still learn from these data.

Original: Our approach is very efficientsince the extra computational cost is linearly correlated with the number of classes, usually not a bignumber.
Revised: Our approach is very efficientsince the extra computational cost is linearly correlated with the number of classes, usually not a bignumber [Table]

Original:  Experiment
Revised:  Experiment

Original: 4.1 Main Results
Revised: 4.1 Main Results

Original: In this section, we evaluate our proposed method on CIFAR100 [19] and ImageNet [5] under variousdistillation settings.
Revised: In this section, we evaluate our proposed method on CIFAR100 [18] and ImageNet [4] under variousdistillation settings.

Original: We refer reader to the Appendix for the implementation details.
Revised: We refer reader to the Appendix for the implementation details.

Original: CIFAR100.
Revised: CIFAR100.

Original: As aforementioned, our approach is orthogonal to other distillation techniques.
Revised: As aforementioned, our approach is orthogonal to other distillation techniques.

Original: Therefore, we investigate the performance of TLLM when combined with other methods.
Revised: Therefore, we investigate the performance of TLLM when combined with other methods.

Original: We reportthe experimental results on CIFAR100, which average over five trials.
Revised: We reportthe experimental results on CIFAR100, which average over five trials.

Original: Specifically, we compare sixdistillation methods, and for each of them, we add our method to theirs.
Revised: Specifically, we compare sixdistillation methods, and for each of them, we add our method to theirs.

Original: As shown in Table 2, ourmethod consistently improve the overall accuracy for baseline.
Revised: As shown in Table 2, ourmethod consistently improve the overall accuracy for baseline.

Original: For instance, we improve the vanilla
Revised: For instance, we improve the vanilla

Original: KD [13] by 2 . 54% .
Revised: KD [12] by 2 . 54% .

Original: The performance gain mostly comes from the increased accuracy on undistillableclasses.
Revised: The performance gain mostly comes from the increased accuracy on undistillableclasses.

Original: Later, we will show that our approach also reduces the number of undistillable classes.
Revised: Later, we will show that our approach also reduces the number of undistillable classes.

Original: Itis also surprising to find that some methods that were once inferior to other approaches, i.e., theaccuracy of OH [11] is 0 .
Revised: Itis also surprising to find that some methods that were once inferior to other approaches, i.e., theaccuracy of OH [10] is 0 .

Original: 65% lower than KR [2] on ResNet32x4/ResNet8x4, obtain higher accuracyafter applying TLLM, showing that many “outdated” algorithms have the potential to be qualitymethods.
Revised: 65% lower than KR [2] on ResNet32x4/ResNet8x4, obtain higher accuracyafter applying TLLM, showing that many “outdated” algorithms have the potential to be qualitymethods.

Original: Moreover, our approach is practical for both output-based (i.e., KD [13])and feature-baseddistillation [28, 2, 44] algorithms.
Revised: Moreover, our approach is practical for both output-based (i.e., KD [12])and feature-baseddistillation [27, 2, 42] algorithms.

Original: ImageNet.
Revised: ImageNet.

Original: ImageNet is a large-scale classification dataset that contains 1.28 million images fortraining and 50k images for validation.
Revised: ImageNet is a large-scale classification dataset that contains 1.28 million images fortraining and 50k images for validation.

Original: Unlike conventional papers that only report the performanceof CNN models, we also consider vision transformer (ViT), an advanced network architecture thathas drawn increasing attention lately.
Revised: Unlike conventional papers that only report the performanceof CNN models, we also consider vision transformer (ViT), an advanced network architecture thathas drawn increasing attention lately.

Original: For CNN-based architectures, we evaluate two teacher-student pairs: 1) ResNet32 as a pretrainedteacher and ResNet18 as a student model, a widely adopted setting follows CRD [35], 2) ResNet50as a pretrained teacher, and MobileNetV1 as a student model, a less conventional setting that somepapers have used.
Revised: For CNN-based architectures, we evaluate two teacher-student pairs: 1) ResNet32 as a pretrainedteacher and ResNet18 as a student model, a widely adopted setting follows CRD [34], 2) ResNet50as a pretrained teacher, and MobileNetV2 as a student model, a less conventional setting that somepapers have used.

Original: The experimental results are reported in Table 7.
Revised: The experimental results are reported in Table 3.

Original: Our TLLM, which is built basedon KR [9], achieves a considerable improvement over its baseline.
Revised: Our TLLM, which is built basedon KR [8], achieves a considerable improvement over its baseline.

Original: It is also worth mentioning thatthe performance of TLLM outperforms other state-of-the-art distillation methods.[Table] [Table]
Revised: It is also worth mentioning thatthe performance of TLLM outperforms other state-of-the-art distillation methods.

Original: For ViT architectures [Table] we evaluate two types of teacher-student pairs: 1) the RegNetY [27] and
Revised: For ViT architectures, we evaluate two types of teacher-student pairs: 1) the RegNetY [26] and

Original: ViT-Tiny/ViT-Small [7] pairs are the popular setting in DeiT [36] which apply a CNN model as ateacher network to distill a vision transformer, 2) the CaiT-S23 [37] and ViT-Tiny pairs adopt anadvanced ViT model as a teacher, whose performance is slightly inferior to RegNetY. 
Revised: ViT-Tiny/ViT-Small [6] pairs are the popular setting in DeiT [35] which apply a CNN model as ateacher network to distill a vision transformer, 2) the CaiT-S23 [36] and ViT-Tiny pairs adopt anadvanced ViT model as a teacher, whose performance is slightly inferior to RegNetY. We compare [Table]

Original: We comparewith DeiT [36], a benchmark that uses teachers’ hard labels as dark knowledge to supervise the studentnetwork.
Revised: [Table] with DeiT [35] [Table] a benchmark that uses teachers’ hard labels as dark knowledge to supervise the studentnetwork.

Original: We notice that in Table 4, we achieve higher accuracy than DeiT on all teacher-student pairs.
Revised: We notice that in Table [Table] we achieve higher accuracy than DeiT on all teacher-student pairs.

Original: Particularly, when the capacity gap between teacher and student is large, i.e., RegNetY/ViT-Tiny and
Revised: Particularly, when the capacity gap between teacher and student is large, i [Table] e., RegNetY/ViT-Tiny and

Original: CaiT-S24/ViT-Small, the improvement in the validation set is more notable.
Revised: CaiT-S24/ViT-Small, the improvement in the validation set is more notable.

Original: 4.2 Ablation Study
Revised: 4.2 Ablation Study

Original: Number of the Undistillable classes.
Revised: Number of the Undistillable classes.

Original: Our method specifically identified the undistillable classes andthen discarded them during distillation.
Revised: Our method specifically identified the undistillable classes andthen discarded them during distillation.

Original: Therefore, we conduct experiments to check whether ourapproach reduces the number of undistillable classes.
Revised: Therefore, we conduct experiments to check whether ourapproach reduces the number of undistillable classes.

Original: In the Appendix, we show that our proposed
Revised: In the Appendix, we show that our proposed

Original: TLLM significantly reduces the number of undistillable classes in the student models, allowingbalanced performance improvement.
Revised: TLLM significantly reduces the number of undistillable classes in the student models, allowingbalanced performance improvement.

Original: Does TLLM allow larger teacher to make better student?
Revised: Does TLLM allow larger teacher to make better student?

Original: Our approach is motivated toresolve the “larger teacher, worse student problem”; thus, it is natural to verify whether our approachcan solve this issue.
Revised: Our approach is motivated toresolve the “larger teacher, worse student problem”; thus, it is natural to verify whether our approachcan solve this issue.

Original: In the Appendix, we show that our approach allows the larger teacher to distillstudents with higher accuracy.
Revised: In the Appendix, we show that our approach allows the larger teacher to distillstudents with higher accuracy.

Revised: Are undistillable classes consistent across multiple experiments?

Revised: Yes and no.

Revised: Yes, since we findsome classes consistently hurt by knowledge distillation over several repeated experiments.

Revised: No,because some classes are undistillable at one experiment and then become distillable in the secondround.

Revised: We provide more evidences in the Appendix.

Original:  Conclusion
Revised:  Conclusion

Original: This paper studies the “larger teacher, worse student” phenomena in knowledge distillation.
Revised: This paper studies the “larger teacher, worse student” phenomena in knowledge distillation.

Original: Unlikeprior works, which attribute the inefficacy of large teachers as a capacity mismatch issue on samples,we consider the presence of undistillable classes as the direct cause of mismatched capacity.
Revised: Unlikeprior works, which attribute the inefficacy of large teachers as a capacity mismatch issue on samples,we consider the presence of undistillable classes as the direct cause of mismatched capacity.

Original: Ourpaper verified that these classes lower the ability of the student model to match the teacher’s featurerepresentation and predictive distribution, making the accuracy drops considerably in some arbitraryclasses, therefore, hurting the overall performance.
Revised: Ourpaper verified that these classes lower the ability of the student model to match the teacher’s featurerepresentation and predictive distribution, making the accuracy drops considerably in some arbitraryclasses, therefore, hurting the overall performance.

Original: We further demonstrate that the existence ofundistillable classes is universal, irrespective of distillation methods, datasets, and student-teacherconfigurations.
Revised: We further demonstrate that the existence ofundistillable classes is universal, irrespective of distillation methods, datasets, and student-teacherconfigurations.

Original: Finally, we propose a “teacher less, learn more” (TLLM) method to alleviate thisissue by targeting and removing samples from the undistillable classes.
Revised: Finally, we propose a “teacher less, learn more” (TLLM) method to alleviate thisissue by targeting and removing samples from the undistillable classes.

Original: Our experiments have shownthat our TLLM can greatly eliminate the adverse effect of undistillable classes to improve the overallperformance significantly.
Revised: Our experiments have shownthat our TLLM can greatly eliminate the adverse effect of undistillable classes to improve the overallperformance significantly.

Original: Overall, we believe that our analysis and proposed method provides aninteresting and practical new perspective on designing distillation algorithms.
Revised: Overall, we believe that our analysis and proposed method provides aninteresting and practical new perspective on designing distillation algorithms.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes](b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes](c)
Revised: [Yes]

Original:  Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes](d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]2.
Revised: [Yes]2.

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A](b)
Revised: [N/A]

Original:  Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?

Original: [No] The code isproprietary.
Revised: [No] The code is proprietary.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: (b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?

Original: (c) Did you report error bars (e
Revised: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experi- ments multiple times)?
Revised: g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] error bars are not reported for some experiments because
Revised: [Yes] error bars are not reported for some experiments because

Original: it would be too computationally expensive
Revised: it would be too computationally expensive

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type
Revised: (d) Did you include the total amount of compute and the type of resources used (e

Original: of GPUs, internal cluster, or cloud provider)?
Revised: g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]
Revised: [No]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [Yes]
Revised: [Yes]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [No] We use standard benchmark in all of our
Revised: [No] We use standard benchmark in all of our

Original: experiments.
Revised: experiments.

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: [Table]
Revised: Board (IRB) approvals, if applicable?

Original: [Table]
Revised: [N/A]

Original: [Table] [Table] [Table]
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [Table]
Revised: [N/A]