T EACHER ’ S PET : UNDERSTANDING AND MITIGATING BIASES IN DISTILLATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Knowledge distillation is widely used as a means of improving the performance of a relatively simple “student” model using the predictions from a complex “teacher” model.
Several works have shown that distillation signiﬁcantly boosts the student’s overall performance; however, are these gains uniform across all data subgroups?
In this paper, we show that distillation can harm performance on certain subgroups, e.g., classes with few associated samples, compared to the vanilla student trained using the one-hot labels.
We trace this behaviour to errors made by the teacher distribution being transferred to and ampliﬁed by the student model.
To mitigate this problem, we present techniques which soften the teacher inﬂuence for subgroups where it is less reliable.
Experiments on several image classiﬁcation benchmarks show that these modiﬁcations of distillation maintain boost in overall accuracy, while additionally ensuring improvement in subgroup performance.
I NTRODUCTION
Knowledge distillation is a technique for improving the performance of a “student” model using the predictions from a “teacher” model.
At its core, distillation involves replacing the one-hot training labels with the teacher’s predicted label distribution.
Empirically, distillation has proven successful for model compression (Bucil ˇ a et al., 2006; Hinton et al., 2015), improving the performance of a ﬁxed model architecture (Anil et al., 2018; Furlanello et al., 2018), and semi-supervised learning (Radosavovic et al., 2018).
Theoretically, several works (Lopez-Paz et al., 2016; Mobahi et al., 2020;
Tang et al., 2020; Menon et al., 2020; Zhang & Sabuncu, 2020; Ji & Zhu, 2020; Allen-Zhu & Li, 2020;
Zhou et al., 2021; Dao et al., 2021) have studied how distillation affects learning.
Put together, both strands of work further the understanding of when and why distillation helps.
In this paper, we are similarly motivated to better understand the mechanics of distillation, but pose a slightly different question: does distillation help all data subgroups uniformly?
Or, do its overall gains come at the expense of degradation of performance on certain subgroups?
To our knowledge, there has been no systematic study (empirical or otherwise) of this question.
This consideration is topical given the study of fairness of machine learning algorithms on under-represented subgroups (Hardt et al., 2016;
Buolamwini & Gebru, 2018;
Chzhen et al., 2019; Sagawa et al., 2020a).
Our ﬁrst ﬁnding is that even in standard settings — e.g., on image classiﬁcation benchmarks such as CIFAR — distillation can disproportionately harm performance on subgroups deﬁned by the individual classes (see Figure 1).
To discern the source of this behaviour, we ablate the teacher and student architectures (§3.2), dataset complexity (§3.3), label frequencies (§3.4).
These point to the potential harms of distillation when the teacher makes conﬁdent mispredictions on a subgroup.
We emphasise that this phenomenon is not simply the teacher’s worst class accuracy translating to student; instead, distillation ampliﬁes worst class imbalance much more than the teacher.
Having identiﬁed a potential limitation of distillation, we present two simple techniques to remedy it.
These apply per-subgroup mixing weights between the teacher and one-hot labels, and per-subgroup margins respectively (§4).
Intuitively, these limit the inﬂuence of teacher predictions on subgroups it models poorly.
Experiments on image classiﬁcation benchmarks show that these methods typically maintain a boost in overall accuracy, while ensuring a more equitable improvement across subgroups.
In sum, this work provides novel insights into distillation performance, with three main contributions:
[Figure]
A cc u r a c y i m p
r
o v e
[Figure]
(i) we identify a hitherto unexplored issue with distillation, namely, that its improvements in overall accuracy may come at the expense of harming accuracy on certain subgroups (§3.1).
Such a ﬁnding is topical given the widespread practical use of generic learning paradigms such as distillation, and the increasing societal applications of learning systems more broadly.
(ii) we ablate potential sources for the above phenomenon (§3.2, §3.3, §3.4)
and in the process identify certain characteristics of data (e.g., skewed label distributions) where it can manifest.
This systematic empirical analysis aims at understanding a non-trivial phenomenon, rather than empirically comparing different learning methods.
(iii) we propose two simple modiﬁcations of distillation that mitigate the above problem, based on applying per-subgroup mixing weights and margins (§4); these perform well empirically (§5).
While the paper’s analysis is empirical rather than theoretical, we note that our focus is on systematic analysis aiming at understanding a non-trivial phenomenon, rather than merely empirical comparison :
Section §3 is devoted to carefully understanding the extent of, and causes for, the non-uniform gains of distillation.
This is in line with works which empirically analyse neural network phenomena, e.g.
Zhang et al.
; Müller et al. ; Nakkiran et al. ; Neyshabur et al.  B ACKGROUND AND RELATED WORK
Knowledge distillation .
Consider a multi-class classiﬁcation problem over instances X and labels Y =
[ L ] .
= { 1 , . . .
, L } .
Given a training set S = { ( x n , y n ) } Nn =1 drawn from some distribution P , we seek a classiﬁer h : X → Y that minimises the misclassiﬁcation error E avg ( h ) .
= P ( h ( x ) (cid:54) = y ) .
In practice, one may learn logits f : X → R L to minimise
ˆ R ( f ) = 1 N (cid:80) Nn =1 (cid:96) ( y n , f ( x n )) , where (cid:96) is a loss function such as the softmax cross-entropy, which for softmax probabilities
p y
( x ) ∝ exp( f y ( x )) is (cid:96) ( y, f ( x )) .
= − log p y ( x ) .
One may then classify the sample via h ( x ) =
arg max y ∈
[ L ] f y ( x ) .
Knowledge distillation (Bucilˇa et al., 2006; Hinton et al., 2015) employs the logits f t : X → R L of a “teacher” model to train a “student” model.
The latter learns logits f s : X → R L to minimise:
[Equation]
[Equation]
where α ∈ [0 , 1] .
Here, one converts the teacher logits to probabilities p t : X → ∆ L for simplex ∆ , e.g. via a softmax transformation p t y (cid:48)
( x ) ∝ exp( f t y (cid:48) ( x )) .
The second term smooths the student labels based on the teacher’s conﬁdence that they explain the sample.
The ﬁrst term includes the original label, so as to prevent incorrect teacher predictions from overwhelming the student.
One further important trick is temperature scaling of the teacher logits, so that p t y (cid:48)
( x ) ∝ exp( T − 1 · f t y (cid:48) ( x )) .
Setting T (cid:29) 0 makes p t more uniform, thus preventing overconﬁdent predictions (Guo et al., 2017).
Average versus subgroup performance .
The above exposition treats the misclassiﬁcation error
E avg ( h ) as the fundamental performance measure of interest.
However, suppose the data contains subgroups G = { 1 , . . .
, G } .
Deﬁning the per-subgroup errors err g ( h ) .
= P ( h ( x ) (cid:54) =
y | g ) , we have
E avg ( h ) =
(cid:80) g ∈ G P ( g ) · err g ( h ) , which may mask errors on samples with P ( g ) ∼ 0 (Sagawa et al., 2020a;b; Sohoni et al., 2020).
To this end, one may instead measure the balanced error (Menon et al., 2013)
E bal ( h ) .
= (cid:80) g ∈ G 1 | G | · err g ( h ) which treats the subgroup distribution as uniform, or the worst-subgroup error (Sagawa et al., 2020a;b; Sohoni et al., 2020)
E max ( h ) .
= max g ∈ G err g ( h ) , which focusses on the worst-performing subgroup.
An intermediary is the average of the k worstperforming subgroups (Williamson & Menon, 2019): for i th largest per-subgroup error err [ i ] ( h ) ,
E top − k ( h ) .
= 1 k (cid:80) ki =1 err [ i ] ( h ) .
The deﬁnition of subgroups is a domain-speciﬁc consideration.
One special case is where each label deﬁnes a subgroup (i.e., G = Y ), and P ( y ) is skewed.
In such long-tail settings (Buda et al., 2017), classiﬁers with good average performance can perform poorly on “tail” labels where P ( y ) ∼ 0 .
Related work .
There is limited prior study that dissects distillation’s overall gains per subgroup.
Zhao et al.
(2020) showed that in incremental learning settings, distillation can be biased towards recently observed classes.
We show that even in ofﬂine settings, distillation can harm certain classes.
Recently, Zhou et al.
(2021) studied the standard aggregate performance E avg of distillation, which was tied to a certain subset of “regularisation samples”.
By contrast, our primary concern is to understand the subgroup performance of distillation.
Study of the fairness of machine learning algorithms on under-represented data subgroups has received recent attention (Dwork et al., 2012;
Hardt et al., 2016; Buolamwini & Gebru, 2018;
Chzhen et al., 2019).
This has prompted dissection of the performance of techniquess such as dimensionality reduction (Samadi et al., 2018), increasing model capacity (Sagawa et al., 2020a), and selective classiﬁcation (Jones et al., 2021).
We follow the general spirit of such works, studying a more delicate setting involving two separate models (the student and teacher), each with their own inductive biases.
We present more discussion of related directions in §6.
In a related effort, Hooker et al.
(2019) explore how model compression may harm subgroup performance compared to the original model.
A RE DISTILLATION ’ S GAINS UNIFORM ?
We demonstrate that the gains of distillation are not uniform across subgroups : speciﬁcally, considering subgroups deﬁned by classes, distillation can harm the student’s performance on the “hardest” few classes (§3.1).
To understand the genesis of this problem, we perform ablations (cf.
Table 1) that establish its existence in settings where there are insufﬁcient samples to model certain classes, either due to the number of classes being large (§3.3), or the class distribution being skewed (§3.4).
We then identify that the student may amplify the teacher’s errors (§3.6).
Finally, we corroborate these results for a more general notion of subgroup in a fairness dataset (§3.5).
3. D ISTILLATION HURTS SUBGROUP PERFORMANCE
To begin, we consider the effect of distillation on a standard image classiﬁcation benchmark, namely,
ImageNet.
We employ a self-distillation (Furlanello et al., 2018) setup, with ResNet-34 teacher and student models, trained with standard hyperparameter choices (see Appendix B).
Following Cho &
Hariharan (2019), we use early stopping on the teacher model.
We now ask: what is the impact of distillation on the student’s overall and per-class performance?
The ﬁrst question has an expected answer: distillation improves the student’s average accuracy by +0 .
4% (see the Baseline setting in
Table 1).
Judged by this conventional metric, distillation is thus a success.
A more nuanced picture emerges when we break down the source of the above improvement.
We compute the per-class accuracies for the one-hot and distillation models, to understand how the overall gains of distillation are distributed.
Figure 2 shows that these gains are non-uniform: distillation [Figure] [Figure] [Figure]
[Table]
in fact hurts the worst- k class performance for k ≤ 40 .
(See Appendix C for a detailed per-class breakdown.)
Thus, distillation may harm the student on classes that it already ﬁnds difﬁcult.
At the same time we note that distillation does improve many classes, such as the classes with relatively high accuracies from the one-hot student.
Given that average accuracy improves, it is worth asking whether the above is a cause for concern: does it matter that performance on subgroups corresponding to the “hardest” classes suffers?
While ultimately a domain-speciﬁc consideration, in general exacerbating subgroup errors may lead to issues from the fairness perspective.
Indeed, we shall see that distillation can also harm in settings where the subgroups correspond to sensitive variables; we discuss this further in §3.5.
At this stage, it is apposite to ask whether the above is an isolated ﬁnding, or indicative of a deeper issue.
We thus study each of the following in turn: (i) does the ﬁnding hold in settings beyond self-distillation?
(ii) does the ﬁnding hold for other datasets, or is it simply due to the idiosyncrasies of ImageNet?
(iii) what are some general characteristics of settings where the problem is manifest?
3. I S DISTILLATION BIASED BY THE MODEL SIZE OR ARCHITECTURE ?
Having begun with a self-distillation setup, we now demonstrate that similar ﬁndings hold when the student and teacher architectures differ.
Continuing with the ImageNet dataset, in the second row in Table 1 we report statistics for the overall average accuracy and average accuracy over the worst 10 classes when distilled ResNet-50 student from a stronger teacher: Efﬁcient-Net
(Tan & Le, 2021) teacher achieving 85.7% accuracy.
Again, we see improved average accuracy and harmed
Worst subgroup accuracy, composed of the 10 classes with the lowest accuracy according to teacher’s performance.
In Table 2, we report statistics when varying teacher and student architectures.
The detrimental effect of distillation on hard class performance holds across all scenarios: thus, our earlier results were not speciﬁc to self-distillation.
For self-distillation settings, smaller models appear to incur greater losses on the worst-class error.
When distilling between different architectures (e.g., from ResNet-50 to ResNet-18), we observe that even average accuracy may not improve, as noted in Cho & Hariharan (2019).
There is however no clear trend between the difference in architectures and drop in worst class performance.
Finally, we note that there is little change in the teacher’s and student’s worst-
k classe; see Figure 6 (Appendix).
3. I S DISTILLATION BIASED BY A LARGE NUMBER OF CLASSES ?
Having seen that ImageNet consistently demonstrates a performance degradation on certain classes, we now repeat the same analysis on a smaller image classiﬁcation benchmark.
We return to the selfdistillation setup, using ResNet-56 models on CIFAR-100.
On this dataset, Table 1 shows a (perhaps more expected) result: distillation boosts both the average and worst- 1 class performance.
This indicates that, at a minimum, the behaviour of distillation’s performance gains are problem-speciﬁc; on CIFAR, distillation appears a complete win for both the average and subgroup accuracy.
One plausible hypothesis is that the tension between average and subgroup performance only manifests on problems with many labels, which might informally be considered “harder”.
To conﬁrm this further while ﬁxing the dataset, we randomly select 10% of classes from the ImageNet dataset, and only keep examples corresponding to those classes across the train and validation sets.
Consistent with the results for CIFAR-100, we again ﬁnd that the worst-10 class accuracy is not harmed under distillation (see the right side of Table 1 where we report results on ResNet-34 self-distillation).
The above indicates that for problems with a few, balanced labels, there may not be a tension between average and worst-subgroup performance under distillation.
However, we now show that even for problems with relatively few labels, one may harm subgroup performance if there is label imbalance .
3. I S DISTILLATION BIASED BY CLASS IMBALANCE ?
We now consider a long-tail setting, where the training label distribution P ( y ) is highly non-uniform.
Following the long-tail learning literature (Cui et al., 2019; Cao et al., 2019; Kang et al., 2020), we construct “long-tailed” ( LT ) versions of the above datasets, wherein the training set is down-sampled so as to achieve a particular label skew.
For ImageNet, we use the long-tailed version from Liu et al.
For other datasets, we down-sample labels to follow P ( y = i )
∝ 1 µ i for constant µ and i ∈
[ L ] (Cui et al., 2019).
The ratio of the most to least frequent class is set to 100 .
From the Long tail setting in Table 1, we note that on both CIFAR-100-LT and ImageNet-LT, accuracy over the hardest classes drops (we report results from self-distillation using ResNet-56 for CIFARand ResNet-50 ImageNet).
The former is particularly interesting, given that the standard CIFARshows gains amongst the hardest classes.
This provides evidence that for some “harder” problems — e.g., where there are insufﬁciently many samples from which to model a particular class — there may be a tension between the average and subgroup performance.
3. B EYOND CLASSES : OTHER CHOICES OF SUBGROUPS
Our analysis thus far has focused on subgroups deﬁned by classes.
This choice is natural for longtailed problems, where it is important to ensure good model performance on rare classes (Kang et al., 2020).
In other problems, different choices of subgroups may be appropriate.
For example, in problems arising in fairness, one may deﬁne subgroups based on certain sensitive attributes (e.g., sex, race).
In such settings, does one similarly see varying gains from distillation across subgroups?
We conﬁrm this can indeed hold on the UCI Adult dataset using random forest models (details in
Appendix C.3).
This data involves the task of predicting if an individual’s income is ≥ 50 K or not, and possesses subgroups deﬁned by the individual’s race and sex.
Akin to the preceding results, we ﬁnd that distillation can signiﬁcantly improve overall accuracy, at the expense of degrading accuracy on certain rare subgroups, e.g., Black women; see Table 1, and Table 8 (Appendix).
This further corroborates our basic observation on the non-uniform distribution of distillation’s gains.
A distinct notion of subgroup was recently considered in Zhou et al.
(2021), who identiﬁed the impact of certain “regularisation samples” on distillation.
These are a subset of training samples which were seen to degrade the overall performance of distillation.
It is of interest whether such a subgroup relates to our previously studied subgroups of “hard” classes; e.g., is there an abundance of regularisation samples in such subgroups, which might explain the poor performance of distillation?
In
Appendix C.4, we study the relationship between regularisation samples, and the per-label subgroups [Figure] from our analysis; we ﬁnd that, in general, these may be complementary notions.
We further analyze the effect of the technique proposed in Zhou et al.
(2021) on average and subgroup accuracies in §
3. W HY DOES DISTILLATION HURT CERTAIN SUBGROUPS ?
The above has established that in a range of scenarios, distillation can hurt performance on subgroups deﬁned by individual classes.
However, a ﬁrm understanding of why this happens remains elusive.
To study this, we consider ResNet-56 self-distillation on CIFAR-100-LT — which showed a stark gap between the average and subgroup (i.e., worst- 1 class) performance — and dissect the logits of the teacher and distilled student.
(See the Appendix for plots where the teacher and student architectures differ.)
Across classes, we seek to understand : (i) how aligned are the student and teacher accuracies
(ii) how reliable are the models’ probability estimates ?
(iii) how do the models’ conﬁdences behave
For a test 1 example ( x, y ) and predicted label distribution p ( x ) ∈
∆ L , we thus compute each models’ accuracy, log-loss (cid:96) log ( y, p ( x ))
= − log p y ( x ) , and margin (Koltchinskii & Panchenko, 2002)
(cid:96) marg ( y, p ( x )) =
p y ( x ) − max y (cid:48)
(cid:54) =
y p (cid:48)
y ( x ) .
Note that the latter may be negative if the model predicts the incorrect label for the example.
Figure 3 shows these metrics on 10 class buckets: these are created by sorting the 100 classes according to the teacher accuracy, and then creating 10 buckets of classes.
Within each bucket, we compute the average of the metric speciﬁed above.
Remarkably, for 5 out of 10 class buckets, average margins are negative , suggesting that the teacher is often wrong yet conﬁdent in predicting these classes.
On these buckets, the student accuracy generally worsens compared to the teacher.
Further, log-loss increases across all buckets (including those where accuracy improves), indicating reduced conﬁdence in the true class of the distilled student.
This points at a potential source of the poorer performance on the worst- 1 accuracy.
Recall that the distilled student’s aim is to mimic the teacher’s logits on the training samples.
This is a proxy to the student’s true goal, which is mimicking these logits on test samples, so as to attain similar generalisation performance as the teacher.
When such generalisation happens, the student can thus be expected to roughly inherit the teacher’s per-class performance; in settings like the above, this unfortunately implies it will perform poorly on those classes with negative teacher margin.
I MPROVING SUBGROUP PERFORMANCE UNDER DISTILLATION
We now study simple means of correcting distillation to prevent the degradation of subgroup performance identiﬁed above.
These leverage the insight that the behaviour is potentially a result of of of use leverage the the the teacher conﬁdently mispredicting on some subgroups.
In the following, for concreteness and simplicity, we focus on subgroups that are given by the individual classes.
4. D ISTILLATION WITH ADAPTIVE MIXING WEIGHTS
In §3.6, we saw that distillation can hurt on classes where the teacher is inherently inaccurate.
Such inaccuracy may in fact be ampliﬁed by the student, which is hardly desirable.
An intuitive ﬁx is to simply rely less on the teacher for classes where it performs poorly, or is otherwise not conﬁdent; instead, the student can simply fall back onto the one-hot training labels themselves.
Formally, for per-class mixing weights ( α 1 , . . .
, α L ) ∈ [0 , 1] L , the student can minimise
[Equation]
This objective introduces a mixing weight α y per-class, which allows us to weigh between teacher predictions and one-hot labels for each class independently.
By contrast, in the standard distillation setup equation 1 we only have a single weight α that is common for all classes.
How do we choose the weights α y ?
In the standard distillation objective equation 1, one only needs to tune a single scalar α , which is amenable to, e.g., cross-validation.
By contrast, equation 2 involves a single scalar for each label, which makes any attempt at grid search infeasible.
Following the observations in §3.6, we propose the following intuitive setting of α y given teacher predictions p t :
[Equation]
In words, Equation 3 places greater faith in the teacher model for those classes which it predicts correctly with conﬁdence , i.e., with large average margin γ avg .
When this margin is negative — so that the teacher is incorrect on average, which can occur on classes that are rare in the training set — we set α y = 0 , and completely ignore the teacher predictions.
The above requires estimating the expectation E x | y [ · ] , which requires access to a labelled sample.
This may be done using a holdout set; we shall follow this in our subsequent experiments.
4. D ISTILLATION WITH PER - CLASS MARGINS
Our second approach for improving distillation on harder classes is to leverage recent developments in long-tail learning , where the goal is to improve performance on rare classes.
Speciﬁcally, Khan et al.
; Cao et al.
; Tang et al.
Ren et al.
; Menon et al.
Wang et al.
(2021) proposed a variant of the softmax cross-entropy with margins
ρ yy (cid:48) between label pairs:
[Equation]
Intuitively, this penalises predicting label y (cid:48) instead of y when ρ yy (cid:48) is large.
For training label distribution π , Cao et al.
(2019) proposed to set ρ yy (cid:48)
∝ exp( π − 1 / 4 y ) , so that rare labels receive a higher weight when misclassiﬁed.
Khan et al.
; Ren et al.
; Menon et al.
; Wang et al.
(2021) showed gains with ρ yy (cid:48)
∝ π y (cid:48)
π y , so that rare labels are not confused with common ones.
We adapt such techniques to our setting, with the intuition that we ought to increase the student penalty for misclassifying those “hard” classes that the teacher has difﬁculty modeling.
We thus choose ρ yy (cid:48)
=
α y (cid:48)
α y , where α y is the adaptive per-class mixing weight from the previous section.
This discourages the model from confusing “hard” labels y with “easy” labels y (cid:48) , when α y (cid:48) > α y .
To avoid a division by 0 issue, we add a small offset to α y when it becomes 0.
We may understand the effect of Equation 4 by studying how it impacts the Bayes-optimal student model predictions, i.e., the optimal predictions in the inﬁnite sample limit, and without a model capacity restriction.
We have the following.
Lemma 1.
Let (cid:96) be the loss of Equation 4, with ρ yy (cid:48) .
= α y (cid:48)
α y .
Let f ∗ be the Bayes-optimal student scores that minimise R dist ( f ) .
=
E x ( p t ( x )) (cid:62) (cid:96) ( f ( x )) .
Then, ∀ x ∈ X , y ∈
[ L ] , f ∗ y ( x ) = log p
t
y
( x ) α y .
[Table]
Lemma 1 illustrates that using per-class margins encourages the student to mimic the teacher predictions p t ( x ) , but with an important modiﬁcation: we up-weight the probabilities for classes that the teacher does poorly on ( α y ∼ 0 ).
Intuitively, this makes it easier for the student to improve performance on classes with small teacher margin.
Relation to existing work .
Previous works varied distillation supervision across examples towards improving average accuracy.
Proposals included weighting samples based on the ratio (Tang et al., 2019;
Zhou et al., 2021), and difference (Zhang et al., 2020) between student and teacher score.
Similarly, Zhou et al.
(2020) proposed to only apply distillation on samples the teacher gets correct.
R ESULTS FOR ADAPTIVE DISTILLATION METHODS
We now present results that further corroborate the potential non-uniform gains of distillation, and the ability to mitigate this with the techniques of the previous section.
We emphasise here that our goal is expressly not to improve over the state-of-the-art in distillation techniques; rather, we wish to verify the key principles identiﬁed in the preceding study, which considers distillation from a novel angle (i.e., in terms of subgroup rather than average performance).
Setup .
We report results on the datasets used in §3: CIFAR-100, ImageNet; and long-tailed ( LT ) versions of the same.
For brevity, we report results under self-distillation.
(For results with varying architectures, see the Appendix.)
Thus, for each dataset, we train a one-hot teacher ResNet model, which is distilled to a student ResNet of the same depth.
We use ResNet-56 models for CIFAR, and
ResNet-50 models for all other datasets.
We employ the same hyper-parameters as used in §3, except we use non-early stopped teachers for consistency across datasets; see the Appendix for details.
We compare: (i) one-hot training of the student (ii) standard distillation, i.e., minimising Equation(iii)
AdaAlpha , our proposed distillation objective with adaptive mixing between one-hot and teacher labels Equation 2, and α as per Equation 3 (iv) AdaMargin , our proposed distillation objective with adaptive margins (Equation 4), and ρ yy (cid:48) =
α y (cid:48)
α y .
We summarise each method by reporting: (i) the standard mean accuracy over all classes; (ii) the accuracy over the worst- 1 class; and (iii) the mean accuracy over worst- 10 (and worst- 100 for the LT datasets) classes.
For the Ada-* methods, per §4, creating the label-dependent α y requires estimating the generalisation performance of the teacher.
To do this, we create a random holdout split of the training set.
For non-LT datasets, we randomly split into 80% (new train) – 20% (dev).
For LT datasets, for each class we hold out k examples into the dev set ( k = 50 for Imagenet-LT, k = 20 for CIFAR-100-LT), or half of examples for a class if the total number of per class examples is at most 2 k .
We train an initial teacher on the new train slice of data, and estimate its per-class performance on the holdout dev slice.
These are used to estimate α y as per, e.g., Equation 3.
Table 3 summarises the results for all methods.
We make the following observations.
[Table]
AdaAlpha improves mean accuracy over vanilla distillation .
The proposed AdaAlpha method consistently and signiﬁcantly improves standard mean accuracy over vanilla distillation.
Thus,
AdaAlpha does not sacriﬁce the gains offered by distillation on average class performance, which is desirable.
Other techniques sometimes perform slightly worse than standard distillation on this metric; however, as we now see, this is compensated by gains on other important dimensions.
AdaAlpha improves worst-accuracy over distillation .
The proposed method consistently improves the worst-class accuracy compared to standard distillation:
Thus, the technique largely fulﬁl their design goal of improving performance on “hard” classes, while not overly sacriﬁcing average-case performance.
In most cases, these improve both the average and worst-class accuracy, indicating that softening the teacher inﬂuence can be broadly beneﬁcial.
Comparison of AdaAlpha and AdaMargin .
In the Appendix, we report per class statistics for
CIFAR-100 LT.
AdaMargin ﬂattens both the margin and log-loss distributions, reducing conﬁdence on the poorly classiﬁed, tail classes.
AdaAlpha consistently increases log-loss across classes, and improves margins on few buckets, leading to a positive margin on one bucket where all other methods give negative margins.
Intuitively, AdaMargin tries to more aggressively control the worst-class accuracy; when this succeeds, there is a large payoff, but there is also greater risk of overﬁtting.
Additional ablations .
We conﬁrm that the success of AdaAlpha is not immediately replicated by simpler baselines: (i) remove hardest 10% , which removes the distillation loss component on bottom 10% labels according to the per class margins found using Equation 3.
It helps analyze whether there is any additional gain beyond simply removing teacher’s supervision where it is arguably wrong.
(ii) shufﬂe temperatures , which randomly shufﬂes the per-class α y values used in AdaAlpha.
This determines whether the precise choice of which labels to up- or down-weight is important; (iii) the adaptive distillation scheme of Zhou et al.
(2021), where distillation is weighted differently across examples depending on the teacher and student scores.
In Table 4, we ﬁnd that the ﬁrst two methods work worse than the proposed AdaAlpha method, indicating that the precise choice of which labels to up- or down-weight is important, and that it does not sufﬁce to merely ignore the teacher on entire subgroups.
The adaptive distillation scheme Zhou et al.
(2021) is also not as effective as AdaAlpha; see Appendix for more such results.
D ISCUSSION AND OTHER APPROACHES
Our goal of ensuring equitable performance across classes can be seen as encouraging fairness across subgroups deﬁned by the classes.
This is subtly different to the classical fairness literature (Calders & Verwer, 2010; Dwork et al., 2012; Hardt et al., 2016), wherein the subgroups are deﬁned by certain sensitive attributes .
Broadly, fairness techniques attempt to learn models that predict the target label accurately , but the subgroup label poorly ; these are inadmissible for our setting, wherein the two labels exactly coincide.
Ensuring fairness across subgroups deﬁned by the classes has been studied in Mohri et al.
; Williamson & Menon (2019); Sagawa et al.
(2020a), who proposed algorithms to explicitly minimise the worst-class (as opposed to the average) loss.
Adapting such algorithms to the distillation setting is of interest for future work.
More broadly, the intent of the analysis in this paper is to better understand settings where distillation can implicitly hurt certain under-represented subgroups.
E THICS STATEMENT
This work studies settings wherein distillation can harm performance on subgroups.
The aim of such a study is to help identify and mitigate potential adverse effects from distillation in practical applications where fairness considerations are important.
Further, our proposed techniques were shown to improve the subgroup performance of distillation, and thus could be useful for preventing such adverse behaviour.
As a limitation, we considered natural deﬁnitions of such subgroups, but an important practical consideration is settings where subgroup membership is unknown.
R EPRODUCIBILITY STATEMENT
For all experiments we used the default hyperparameters from previous works (see Appendix B for details).
A P ROOFS
Proof of Lemma 1.
We may write
[Equation]
y α y (cid:48)
α
[Equation]
(cid:48)
[Equation]
y ( x ) 

[Equation]
log 
[Equation]
[Equation]
(cid:88) y (cid:48) (cid:54) =
y e ln α y (cid:48) − ln α y · e f y (cid:48)
( x ) − f y ( x ) 
 = log  
[Equation]
(cid:88) y (cid:48) (cid:54) =
y e ¯ f y (cid:48)
( x ) − ¯ f y ( x ) 

[Equation]
e ¯ f
[Equation]
( x )
[Equation]
(cid:48) (
[Equation]
where ¯ f y ( x ) .
=
f y ( x ) + ln α y .
The population loss is
[Equation]
where ¯ p s y ( x ) ∝ exp( ¯ f y ( x )) .
Thus, at optimality we must have ¯ p s ( x ) =
p t ( x ) , or ¯ f y ( x ) = log p
t
y ( x ) .
By deﬁnition of ¯ f , we thus see that f y ( x ) = log p t y ( x ) − log
α y = log p t y ( x ) α y .
B D ETAILS OF EXPERIMENTS B. A RCHITECTURE
We use ResNet with batch norm (He et al., 2016) for all our experiments with the following conﬁgurations.
For CIFAR, we experiment with ResNet-56 and ResNet-32.
For ImageNet, we use ResNet-50.
We list the architecture conﬁgurations in terms of (n layer , n ﬁlter , stride) corresponding to each ResNet block in Table 5.
[Table]
Conﬁguration: [(n layer , n ﬁlter , stride)]
CIFAR ResNet- CIFAR ResNet-
[Table] [Table]
[Table]
B. T RAINING SET
For all datasets, we train using SGD and weight decay 10 − 4 for CIFAR, and 0 .
5 × 10 − 4 for Imagenet datasets.
We have the following dataset speciﬁc settings.
CIFAR-100 .
We train for 450 epochs with an initial learning rate of 1.0, with a linear warmup in the ﬁrst 15 epochs, and an annealed learning rate schedule.
We drop the learning rate by a factor of 10 at epochs number: 200, 300 and 400.
We use a mini-batch size of 1024.
We use SGD with Nesterov momentum of 0.9.
[Figure]
We zoom in on the “tail” bucket (comprising the 100 “hardest” classes), and further split them into 10 “tail sub
As in Figure 3, the teacher is seen to conﬁdently mispredict most samples on the last few misplaced conﬁdence being transferred to the student.
For our distillation experiments we train only with the cross-entropy objective against the teacher’s logits.
For each method we ﬁnd the best temperature from the list of values: { 1 , 2 , 3 , 4 , 5 } .
ImageNet .
We train for 90 epochs with an initial learning rate of 0.8, with a linear warmup in the ﬁrst 5 epochs, and an annealed learning rate schedule.
We drop the learning rate by a factor of 10 at epochs number: 30, 60 and 80.
We use a mini-batch size of 1024.
For our distillation experiments we train with the distillation objective as deﬁned in Equation 1 setting α = 0 .
For each method we ﬁx the temperature to 0 .
Long-tail (LT) datasets .
We follow setup as in the non-long tail version, except for the learning rate schedule, which we change to follow the cosine schedule (Loshchilov & Hutter, 2017).
C A DDITIONAL E XPERIMENTS
We present additional experiments to those in the body.
C. F URTHER VARYING DATASETS AND MODEL ARCHITECTURES
On Imagenet, we summarise statistics for three models: the early-stopped teacher, distilled student, and the one-hot (non-distilled) student.
As with CIFAR-100-LT, we sort classes by teacher accuracy, and bucket them into 10 groups.
Owing to the larger number of labels, we further zoom into the “tail” bucket (comprising the 100 “hardest” classes), and split them into 10 sub-buckets.
From Figure 4, the distilled student performs worse than its one-hot counterpart on the last bucket; this is in keeping with our results in Table 1.
Figure 5 shows logit statistics for additional settings to considered in the body.
On ImageNet-LT, e.g., we see again that the margin of the teacher model systematically worsens and becomes negative on the hardest classes.
In Table 6 we report results from the inherently long-tailed iNaturalist 2018 dataset (Van Horn &
Perona, 2017).
Our observations made for other considered datasets hold: adaptive margin method improves over both one hot and plain distillation in terms of the worst class accuracy.
We also observe, how the average accuracy improves.
C. L OGIT PLOTS UNDER A DA -* METHODS
Figure 7 shows the logit statistics under the proposed AdaMargin and AdaAlpha methods on CIFAR100 LT.
We see that AdaMargin can generally improve the student margin and accuracy on the hardest classes, while also reducing the log-loss.
This conﬁrms that the gains of the method come from improving behaviour of the scores on these hard classes.
[Figure] [Figure]
C. R ESULTS ON A DULT DATASET
We report the results of an experiment on the UCI Adult dataset.
This data comprises ∼ 48 K examples, with the target being a binary label denoting whether or not an individual has income ≥
K. The data is mildly imbalanced, with 24% of samples being positive.
Inspired by Dao et al.
(2021), we consider a random forest based distillation setup: we use a teacher model that is a random forest classiﬁer comprising 500 trees with a maximum depth of 20 , and a student model that is a random forest regressor comprising 1 tree with a maximum depth of 20 .
The teacher model achieves a test (balanced) accuracy of 81 . 8% .
[Table]
[Figure]
few buckets, including ﬂipping the bucket 5 to have positive margin.
[Table]
We perform distillation by feeding the student model the teacher’s prediction scores, mixed in with the binary training labels with a weight α = 0 .
Distillation improves the student’s overall (balanced) accuracy signiﬁcantly, from 76 . 2% to 79 .
However, this gain is not distributed uniformly: using per-label subgroups, we ﬁnd that distillation helps the positive class by +7 .
4% , but hurts the negative class by − 1 . 2% .
While by itself suggestive of asymmetry in distillation performance, the data admits an arguably more natural subgroup creation, based on available sex and sex features.
For example, we ﬁnd that amongst low-income males, distillation hurts by − 2 .
2% ; further restricting to those who are Asian Paciﬁc-Islander, the degradation is − 5 .
This conﬁrms that in scenarios where fairness may be a consideration, a naïve application of distillation may be inadmissible.
C. A NALYSIS OF REGULARISATION SAMPLES
Recently, Zhou et al.
(2021) proposed the notion of regularisation samples to understand how distillation’s performance can be improved.
In brief, such samples correspond to cases where the teacher’s prediction on the training label is less than the distilled student’s prediction on this label; these may be shown to correspond to cases where a certain notion of “variance reduction” dominates
[Table]
a notion of “bias reduction”.
Given our analysis above of the asymmetric effects of distillation on certain subgroups, it is natural to consider whether or not these relate to the presence of regularisation samples in these groups.
Figure 8(a) visualises the distribution of regularisation samples inside subgroups deﬁned by 10 label buckets.
where the labels are sorted in descending order of label frequency.
Here, we compare the predicted probabilities of the teacher and ﬁnal distilled student models on all training samples (as was done in the analysis of Zhou et al.
Interestingly, we see that the tail buckets tend to have very few regularisation samples; i.e., for rare labels, the teacher prediction on the training label is generally higher than that of the distilled student model.
We conﬁrm this in Figure 8(b).
While the analysis of Zhou et al. (2021) was primarily for training samples — since the aim in identifying regularisation samples was to mitigate their inﬂuence during training — we may also identify the breakdown of such samples on test data.
Figure 9(a) shows that, compared to the training set, there are in absolute terms more such samples across nearly every label bucket; however, there is again no clear correlation between the label bucket and the fraction of such samples.
In particular, the tail bucket is again the one with the fewest regularisation samples.
This is corroborated by the probability scores of the teacher and student in Figure 9(b).
Overall, this results suggest that the existing notion of regularisation samples may not, by themselves, be sufﬁcient to predict the poor performance of distillation on certain subgroups deﬁned by labels.
C. I MPACT OF REPEATED DISTILLATION
In the body, we showed that performing distillation once can harm worst-class accuracy.
However, what is the effect of repeating this process, and distilling using the resulting student as a new teacher?
Does the worst-class accuracy get further harmed?
Table 9 shows that on CIFAR-100 LT, repeating distillation can indeed harm worst-class performance, even though average performance remains roughly similar.
This further highlights the potential tension between average and worst-case performance under distillation.
[Figure]
[Figure] [Figure]
∆
[Figure]
One-hot 44 .3 .
Distillation 1 × 45 .
Distillation 2 × 45 .
Distillation 3 × 44 .0 .87 .44 .