What Makes a “Good” Data Augmentation in Knowledge Distillation – A Statistical Perspective Project: http://huanwang.tech/Good-DA-in-KD [Figure] 𝑥 !
[Figure] t [Figure]
Howtodefine“stronger”DA [Figure] ) and T. stddev, showing that T. stddev well captures the “goodness” of different DA schemes in KD. The most striking fact from this plot may be: T. stddev is purely calculated with the teacher (no any student used) while it can “predict” the relative order of the student ’s performance, implying the “goodness” of DA in KD probably is student-invariant.
Abstract
Knowledge distillation (KD) is a general neural network training approach that uses a teacher model to guide the student model.
Existing works mainly study KD from the network output side ( e.g. , trying to design a better KD loss function), while few have attempted to understand it from the input side.
Especially, its interplay with data augmentation (DA) has not been well understood.
In this paper, we ask: Why do some DA schemes ( e.g. , CutMix) inherently perform much better than others in KD?
What makes a “good” DA in KD?
Our investigation from a statistical perspective suggests that a good DA scheme should reduce the covariance of the teacher-student cross-entropy .
A practical metric, the stddev of teacher’s mean probability (T. stddev), is further presented and well justified empirically.
Besides the theoretical understanding, we also introduce a new entropy-based data-mixing DA scheme, CutMixPick , to further enhance CutMix.
Extensive empirical studies support our claims and demonstrate how we can harvest considerable performance gains simply by using a better DA scheme in knowledge distillation.
Introduction
Deep neural networks (DNNs) are the de facto methodology in many artificial intelligence areas nowadays [25, 37].
How to effectively train a deep network has been a central topic for decades.
In the past several years, efforts have mainly focused on better architecture design ( e.g. , batch normalization [20], residual blocks [14], dense connections [19]) and better loss functions ( e.g. , label smoothing [43, 30], contrastive loss [18], large-margin softmax [26]) than the standard cross-entropy (CE) loss.
Knowledge distillation (KD)
[17] is a training method that falls into the second group.
In
KD, a stronger network – called teacher – is introduced to guide the learning of the original network – called student – by minimizing the discrepancy between the representations of the two networks,
[Equation]
where D KL represents KL divergence [24]; α ∈ (0 , 1) is a factor to balance the two loss terms;
L CE denotes the cross-entropy loss; y is the one-hot label and p ( t ) , p ( s ) stand for the teacher’s and student’s output probabilities over the classes; τ is a temperature constant [17] to smooth predicted probabilities.
KD allows us to train smaller, more efficient neural networks without compromising on accuracy, which facilitates deploying deep learning in resource constrained environments ( e.g. , on mobile devices).
KD has found plenty of applications in many tasks [6, 49, 13, 21, 50].
Most existing KD methods have attempted to improve it by proposing better KD loss functions applied at the network outputs [34, 31, 45].
Few works have considered KD from the input side .
Especially, the interplay between KD and data augmentation (DA) [40] has not been well understood so far (note, by DA here, we mean the conventional DA concept: generating a new input by transforming one or multiple inputs .
A broader scope of DA may involve the neural network , e.g. , dropout can be seen as a kind of DA [3].
We do not consider this type of DA in this paper due to the limited length). In this work, we ask: What makes a “good” data augmentation in knowledge distillation ?
A clear answer to this question has many benefits.
First, theoretically, it can help us towards a better understanding about how data augmentation plays a role in KD.
Second, practically, it can bring us considerable performance gain – in Fig.
2, we show test error rates using the standard CE loss (no teacher) vs. using KD loss (with a teacher). As seen, a stronger DA can lower the test error rate and admit more training iterations without overfitting in KD.
It is pretty obvious to see that “Flip+Crop” is stronger than “Flip” alone in Fig.
However, for other DA schemes, such as Mixup [54] vs. AutoAugment [8], which one is stronger?
Not very clear. We thus desire a principled way ( e.g. , a concrete metric) to make the vague concept “stronger” exact.
Presenting such a theoretically sound metric and empirically validating its effectiveness is the goal of this paper.
[Figure]
Intuitively, a good DA should enrich the input data and expose more knowledge of the teacher so that the student can generalize better.
We formalize this idea from a statistical learning perspective.
Specifically, we will show a good DA scheme is defined by a lower variance (or covariance) of the teacher
’s mean output probability over different input samples, which ultimately leads to a lower generalization gap for the student.
The proposed theory is well justified by our extensive empirical studies on CIFAR and Tiny ImageNet datasets with various pairs.
The proposed theory well explains why CutMix is better than other alternatives (such as
Mixup [54], AutoAugment [8]) in KD.
In addition to the new theoretical results, we also propose an entropy-based data picking scheme to select more informative samples for KD, which can deliver even lower variance of the teacher’s mean probability as well as lower generalization error of the student.
We make the following contributions in this paper:
• We present a proven proposition (Proposition 3.1) that precisely answers what defines a better DA in KD:
Given a fixed teacher, a better DA is the one that gives a lower variance of the teacher’s mean probability.
• The proposition is well justified empirically on standard image classification datasets with many teacher-student pairs.
• An entropy-based data picking scheme is introduced to further reduce the variance of the teacher ’s mean probability, which can further advance CutMix, the prior state-of-the-art DA approach among those evaluated in this paper.
• Empirically, we show how the presented theory can benefit in practice – we can simply enhance the existing KD methods by using a stronger DA and prolonged training iterations.
Related Work
Knowledge Distillation (KD) .
The general idea of knowledge distillation is to guide the training of a student model through a (stronger) teacher model (or an ensemble of models).
It was pioneered by Buciluˇa et al.
[4] and later refined by Hinton et al.
[17], who coined the term.
Since its debut, KD has seen extensive application in vision and language tasks [6, 15, 49, 21, 50].
Many variants have been proposed regarding the central question in KD, that is, how to define the knowledge transferred from the teacher to the student.
Examples of such knowledge definitions include feature distance [35], feature map attention [53], feature distribution [32], activation boundary [16], inter-sample distance relationship [31, 34, 27, 46], and mutual information [45]. Several works [30, 39, 5] investigate the connection between label smoothing and knowledge distillation. Another line of works ( e.g. , [44]) attempts to understand KD more theoretically.
Over the past several years, the progress has been made primarily for intermediate feature maps and network outputs ( i.e. , through a better loss function).
In contrast, our goal is to improve the KD performance at the input end with the help of data augmentation.
We will show this path is as effective and also has much potential for future research.
Data Augmentation (DA) .
Deep neural networks are prone to overfitting, i.e. , building input-target mappings using undesirable or irrelevant features (like noise) in the data.
Data augmentation is a prevailing technique to curb overfitting [40].
In classification tasks, data augmentation aims to explicitly provide data with label-invariant transformations (such as random crop, horizontal flip, color jittering, Cutout [12]) during training so that the model can learn representations robust to those nuisance factors.
Recently, more advanced data augmentation methods were proposed, which not only transform the input, but also transform the target.
For example, Mixup [54] linearly mixes two images with the labels mixed by the same linear interpolation.
Manifold Mixup [48] is similar to
Mixup but conducts the mix operation in the feature level instead of pixel level; CutMix [51] pastes a patch cut from an image onto another image with the label decided by the area ratio of the two parts.
AutoAugment [8] is a strong DA method that finds an optimal augmentation policy from a large search space via reinforcement learning.
When both the input and target are transformed simultaneously, the key is to maintain a semantic correspondence between the new input and new target.
Unlike these methods, which focus on general classification using the cross-entropy loss, our work investigates the interplay between data augmentation and knowledge distillation loss and proposes new data augmentation specifically for knowledge distillation. Some recent KD works also involve the utilization of DA in KD, such as [2, 9]. Especially, [2] also employs Mixup and prolonged training to enhance the student performance in KD, akin to ours.
Yet it is worthwhile to note that our paper is substantially different from theirs, in that we are seeking the theoretical reason explaining how to define a better DA in KD to deliver better performance; these works mainly investigate in an empirical fashion, with no theoretical results presented. Meanwhile, our work is not limited to one specific DA (see Sec. 5). We target a general theoretical understanding which can apply to a broad scope of DA schemes (fortunately, as our experiments show, the proposed theory and the derived DA “goodness” measure indeed capture it).
The fact that [2] utilizes Mixup [54] and prolonged training to deliver 82.8% top-1 accuracy with resnet50 [14] on ImageNet [11] can be a direct proof that the principle proposed in this work can bring us very promising practical benefits. One recent work [10] also conducts empirical studies of the impact of DA on KD.
They first apply DA ( e.g. , Mixup/CutMix) to the teacher training then conduct the KD step as usual (no extra DA in this step). Our investigation is the exact opposite to their setup: We train the teacher as usual (no Mixup/CutMix), then in the KD step we employ a more advanced DA ( e.g. , Mixup/CutMix).
Interestingly, they conclude that the teacher trained with Mixup/CutMix hurts the student’s generalization ability, while we consistently see student performance boost via a stronger DA. Another recent work [42] utilizes KD to re-label mixed samples in Mixup to fix the inaccurate labelling problem of Mixup.
Their work shows that KD can be used to make Mixup more generally useful, which is orthogonal to the topic of this work. The work of [29] presents a statistical perspective to understand how the softened probabilities in KD are better than the one-hot hard labels.
Our work is inspired by their Bayes teacher notion.
This said, our work is different from theirs in that we focus on explaining how data augmentation plays a role in KD and answer what characterizes a good DA, while they attempt to answer why the knowledge distillation loss is better than the standard cross-entropy loss.
Theoretical Investigation 3.1 Prerequisites: Multi-Class Classification with KD
Given a training set S = { ( x n , y n ) } Nn =1 ∼ D N , where D is the joint distribution for input-output random variable pair ( x, y ) , the goal in multi-class classification is to pin down a predictor f : X → R C from a hypothesis class H , where X is the input space and C refers to the number of classes.
The predictor f is supposed to minimize the true risk
[Equation]
where L stands for the loss objective function ( e.g. , cross-entropy); the subscript D of R D is to emphasize that the true risk is defined on the data distribution.
The true risk is approximated in practice on a separate test set.
For training, the predictor aims to minimize the empirical risk defined on the training sequence S :
[Equation]
where e y ∈ { 0 , 1 } C is a one-hot vector indicating the label y ∈
[ C ] = { 1 , 2 , · · · , C } .
The subscript
S of R S is to emphasize the empirical risk is defined on the finite sampled data points.
In the context of KD, the one-hot hard target vector
e y is replaced with a probability vector p ( t ) ( x ) ∈
R C + , giving us the empirical distilled risk of f :
[Equation]
where the super-script t indicates the fixed teacher model.
The theory concerning why Eq.
(4) is better than Eq.
(3) has been established in [29].
Interested readers may refer to their paper for more details.
Next, we look into how the data augmentation plays a role in KD.
3.2 What Makes a “Good” DA in KD?
Intuitively, a better DA should provide more information, i.e. , expose more knowledge of the teacher so that the student can absorb more and thus generalize better.
We make this idea rigorous as follows.
Proposition 3.1.
Given a bounded loss function and a fixed teacher model with the empirical distilled risk defined in Eq.
(4), for any predictor f , consider two sampled sequences S 1 ∈ D N and S 2 ∈ D N , they are made up of N elements sampled from the same distribution D , while not i.i.d.
(especially when data augmentation is employed).
If the elements in S 1 present a larger correlation than those in S 2 , then the student’s generalization gap trained on S 1 will be greater than that trained on S 2 :
[Equation] [Equation]
Proof.
Let ∆ =
ˆ R S ( f ) − R D ( f ) .
By the definition of variance, E S [∆ 2 ] =
Var S [∆] + (E S [∆]) 2 .
To make the notation clearer, we define q ( x i )
= − p ( t ) ( x i ) ⊤ log( f ( x i )) .
(1) Since R D ( f ) is a constant (albeit unknown),
[Equation]
where the second last equation is because each element x i in S is drawn from the same distribution D .
From the RHS of the last equation, we can clearly see that for S 1 , S 2 , E S [∆] is the same.
(2) Then we consider Var S [∆] .
Again, since R D ( f ) is a constant, we only need to consider
[Equation]
where Cov[ · , · ] stands for covariance.
From the last item in Eq.
(7), we can see that the covariance part is different for different sampled S ’s. If the samples in S 1 present a larger correlation than those in S 2 , we will have Var S 1 [∆] >
Var S 2 [∆] , which further leads to increased generalization gap for the student, i.e. , E S
[( R S 1 ( f ) − R D ( f )) 2 ] > E S 2 [( R S 2 ( f ) − R D ( f )) 2 ] .
The proof is finished.
Practical Use: Stddev of Teacher’s Mean Probability (T. stddev) .
Note in Proposition 3.1, the predictor f ( i.e. , the student model) can be any one (not necessarily a converged model).
Each student would have an order of different DA schemes regarding which is better “in its opinion”. Presumably, different students will lead to different such orders.
For practical use, we must pick a certain student as oracle to conduct the evaluation. Here, we can play a trick that can make it rather simple to use our theory – assume there is a student that performs exactly the same as the teacher (this assumption is not unpractical, since one straightforward example is to use the teacher as student). Then we have
[Equation]
In this case, we only need the covariance of the teacher’s probability to measure the “goodness” of a certain DA technique, no need for the student .
Despite not using any information of the student, the proposed metric turns out to correlate surprisingly well with the student’s performance (see Fig. 4).
Based on Eq.
(8), when using S 1 vs. S 2 as training sequence, the fundamental factor answering for the variance gap of Var S [∆] boils down to the covariance in { p ( t ) ( x i ) } Ni =1 .
That is, a larger covariance in { p ( t ) ( x i ) } Ni =1 leads to a larger covariance in { p ( t ) ( x i )
⊤ log( p ( t ) ( x i )) }
Ni =1 in Eq.
(8), which ultimately leads to a higher Var S [∆] in Eq.
The next step is to find a feasible way to estimate the covariance in { p ( t ) ( x i ) } Ni =1 .
Consider the average variable (denoted as u here) of several random variables { p ( t ) ( x
[Equation] [Equation]
Its variance (or equivalently, stddev) inherently takes into account the covariance among its addends.
Therefore, we can use the variance of u as a proxy for the covariance in { p ( t ) ( x k ) } Kk =1 :
[Equation] [Figure]
where K is a pre-defined number of samples of set S ∗ to realize the averaging effect ( e.g. , K = 640 in our experiments for CIFAR100 and Tiny ImageNet – see Appendix Sec. A.3 for a concrete calculation example).
Note, u can be interpreted as the teacher
’s mean probability over K input samples.
If there is a large covariance among { p ( t ) ( x k ) } Kk =1 , m will be large (in an element-wise sense).
Then the average of m over classes, i.e. , the ¯ m (we do this averaging simply because we desire a scalar metric), is a good indicator to capture such covariance.
We thus formally introduce ¯ m , the (averaged)
T. stddev, as the proposed metric to measure the quality of a data augmentation scheme .
A lower ¯ m implies a better DA by our definition.
In the experiments, we will show this metric defined purely using the teacher can accurately characterize the generalization error of the students after distillation.
Evaluated Algorithms 4.1 Extend Existing DA Approaches for KD
Given an existing DA approach, this section explains how we adapt it properly to the case of KD.
Specifically, let x 0 denote the raw data, x denote the transformed data by the standard augmentation (random crop and flip).
Illustrated in Fig. 3(b), we will add the DA following x to obtain x ′ .
Unlike the common data augmentation where only the transformed input is fed into the network, we keep both the input x and x ′ for the training (as such, the number of input examples is doubled).
The consideration of keeping both inputs is to maintain the information path for the original input x so that we can easily see how the added information path of x ′ leads to a difference.
For x , its loss is still the original KD loss, consisting of the cross-entropy loss and the KL divergence (Eq.
Of special note is that, for x ′ , its loss is only the KL divergence, i.e. , we do not use the labels assigned by the DA algorithm (for example, in the original Mixup and CutMix, they assign a linearly interpolated label to the augmented sample) because these labels can actually be wrong (see Appendix Sec. A.4 for concrete examples on ImageNet). In fact, not using the hard label has another bonus. A dataset augmentation scheme which employs CE loss has to provide corresponding labels as supervisory information.
In order to maintain the semantic correspondence, it cannot admit very extreme transformations for data augmentation.
In contrast, in the Mixup/CutMix+KD setting described above, the data augmentation scheme need not worry about the labels as they are assigned by the teacher – the recent work [2] refers this kind of utilization of DA as function matching of the teacher. As a result, it can admit a broader set of transformations to expose the teacher’s knowledge more completely. This reflects that data augmentation in KD has more freedom than in CE. Among all the data augmentation techniques evaluated in this paper, we will show CutMix works best .
The fundamental reason for its success, as we will show, is that it achieves a much lower variance of the teacher ’s mean probability, implying it produces more diverse data than its counterparts.
4.2 CutMixPick: Enhancing CutMix with Entropy-Based Data Picking
In this section, we propose a data picking scheme to further reduce the variance of the teacher’s mean probability.
The idea is partly inspired by active learning [38].
In active learning, the learner enjoys the freedom to query the data instances to be labeled for training by an oracle ( i.e. , the teacher in our case) [38].
Since the augmented data can vary in their quality, we can introduce a certain criterion to pick the more valuable data for the student.
Intuitively, we regard a sample with more information is of higher quality.
Therefore, we take
Shannon’s entropy of the teacher’s output probability as a natural measure to select samples,
[Equation]
Note this formula is in exactly the same form of Eq.
Empirically, we will also show the data selected by this formula indeed results in lower ¯ m and better test loss for the student.
Concretely, given a batch of data, we first apply CutMix to obtain a bunch of augmented samples.
Then sort all the augmented samples by Eq.
(11) in ascending order and keep the top r (a pre-defined percentage constant, r = 0 .
5 in our experiments) samples.
This simple technique can be rather effective according to our empirical study.
Another seemingly potential alternative is to use the student’s entropy as the picking metric.
The intuition behind this is that a high-entropy sample in the view of the student can be regarded as a hard example, too.
Learning with these hard examples may expand the student’s knowledge by squeezing its blind spots.
Despite this intuitively plausible explanation, in practice, we will show this scheme actually under-performs the former student-agnostic scheme in Eq.
(11), which is a bit surprising.
Experimental Results
Datasets and Networks .
We evaluate our method primarily on the CIFAR100 [23] and Tiny
ImageNet * datasets. CIFAR100 has 100 object classes (32 × 32 RGB images).
Each class hasimages for training and 100 images for testing.
Tiny ImageNet is a small version of ImageNet [11] with 200 classes (64 × 64 RGB images).
Each class has 500 images for training, 50 for validation and 50 for testing.
To thoroughly evaluate our methods, we benchmark them on various standard network architectures: vgg [41], resnet [14], wrn [52], MobileNetV2 [36], ShuffleV2 [28]. We will also include results on ImageNet100 (a randomly drawn 100-class subset of ImageNet) and ImageNet.
Benchmark Methods .
In addition to the standard cross-entropy training and the original KD method [17], we also compare with the state-of-the-art distillation approach, Contrastive Representation Distillation (CRD) [45].
It is important to note that our method focuses on improving KD by using better inputs , while CRD improves KD at the output end ( i.e. , a better loss function).
Therefore, they are orthogonal and we will show they can be combined together to deliver even better results.
Hyper-Parameter Settings .
The temperature τ of knowledge distillation is set to 4 following CRD [45].
Loss weight α = 0 .
9 (Eq.
For CIFAR100 and Tiny ImageNet, training batch size is 64; the original number of total training epochs is 240, with learning rate (LR) decayed at epoch 150, 180, and 210 by multiplier 0.1.
The initial LR is 0.05.
All these settings are the same as CRD [45] for fair comparison.
Note, in our experiments we will present the results of more training iterations.
If the number of total epochs is scaled by a factor k , the epochs after which learning rate is decayed is also be scaled by k .
For example, if we train a network for CIFAR100 for 480 epochs ( k = 2 ) in total, the learning rate will be decayed at epoch 300, 360, and 420.
We use PyTorch [33] to conduct all our experiments.
For CIFAR100, we adopt the pretrained teacher models from CRD † for fair comparison.
For Tiny ImageNet and ImageNet100, we train our own teacher models.
For ImageNet, we use torchvision models following CRD [45].
Data Augmentation Schemes .
We investigate the following popular DA schemes:
• Identity : This augmentation scheme simply makes a copy of each batch data during training, which should be the lower bound of all DA schemes discussed here since it adds no new information.
• Flip : Random horizontal flip. • Flip+Crop : Random horizontal flip and random crop.
This is the standard DA extensively used in 2D image recognition task (such as on CIFAR and ImageNet datasets). • Cutout [12]: Cutout occludes a small random patch of an image.
• AutoAugment [8]: AutoAugment is an ensemble of a collected DA schemes.
The DA policy is automatically selected by reinforcement learning instead of manually.
• Mixup [54]: Mixup applies linear interpolation between two inputs and applies the same linear interpolation to their labels to make the new label for the augmented sample.
• CutMix [51]: CutMix cuts a small patch from a source image and pastes it to another source image. The resulted image is considered as a new input.
The target for the new input is a linear interpolation from the two source labels.
5.1 Empirical Verification of Our Proposition
Before presenting results, one point worth mentioning is that, in this section we use test loss instead of accuracy as the measure of student’s performance in KD, because (1) all the formulas in Sec.
3 are derived using numerical loss instead of accuracy; (2) more importantly, it is observed that accuracy can mismatch with loss – a model may achieve a better accuracy, meanwhile higher loss too [29], which we also observe several times in our experiments. Using accuracy as measure would prevent us from seeing the correlation between T. stddev and student’s performance (see Appendix Fig. 6 for an example). Potential extension of our theory from loss to accuracy is left for future work.
In Fig. 4, we plot the scatters of S. test loss and T. stddev. The x/y-axis value of each data point is averaged by at least three random runs (see Tabs 3/4 and Tabs 5/6 for detailed numbers).
(1) In terms of T. stddev, there is a rough trend ( e.g. , on the vgg13/vgg8 pair on CIFAR100): Identity < Flip < Flip+Crop < Cutout <
AutoAugment < Mixup < CutMix.
These inequalities are wellaligned with our intuition.
E.g. , AutoAugment [8] includes Cutout [12] in its transformation pool, thus should be stronger than Cutout. This is faithfully reflected by the T. stddev on many pairs. (2) Obviously, S. test loss poses a clear positive correlation with T. stddev. Per our theory, lower
T. stddev should lead to better generalization risk for the student.
This is generally well verified in these plots.
We do see some minor counterexamples.
Possible reasons are: 1)
The test loss is obtained on the test set with finite samples, which is only an approximation of the true risk defined on distribution; 2)
The teacher’s mean probability is also evaluated on finite data.
Despite them, the general picture from Fig. 4 still confirms the positive correlation between S. test loss and T. stddev.
The correlation is actually very significant as the p-values indicate.
(3) Importantly, note we only need the teacher to define the quality of a certain DA in KD, no need for the student .
This is more clear if we examine the results in Tabs.
3/4 and Tabs.
Taking vgg13/vgg8 and vgg13/MobileNetV2 as an example, the students are starkly different but both the student’s performance correlates well with T. stddev.
This observation implies that the “goodness” of
DA in KD is probably student-invariant .
This, notably, is a great advantage in practice since we can decide which DA should be used for the best performance simply using a formula (Eq.
with a few network forwards, without having to train the students physically.
5.2 Boosting KD with Stronger DA
In this section, we present more results to show how the proposed theory can benefit in practice – we can harvest considerable performance gain simply by using a stronger DA in KD.
Prolonged Training .
Notably, a stronger DA produces more diverse data, implying more information.
Intuitively, it should take a student model more training iterations (if the batch size does not change) to fully absorb the excessive information.
That is, a stronger DA conceivably takes more training iterations to fully exhibit its potential.
This intuition is confirmed in Fig.
2 (and also two more pairs wrn_40_2/wrn_16_2 and vgg13/vgg8 in the Appendix).
Thus, in our experiments, we will also report results with prolonged training iterations for maximized performance.
Results on CIFAR100 .
The results on CIFAR100 dataset are shown in Tab. 1.
(1) Comparing the row “KD+CutMix” to “KD", we see CutMix improves the student accuracies on all the pairs.
On the pair resnet32x4/ShuffleV2, the improvement is very significant (more thanpercentage point).
(2) Comparing the row “KD+CutMixPick” to “KD+CutMix”, we see 6/7 pairs [Figure] [Figure] CutoutAutoAugmentMixupCutMixCutmixPick
[Figure] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
ImageNet
[Figure]
, Tiny ImageNet
[Figure]
CutoutAutoAugmentMixupCutMixCutmixPick
[Figure]
The detailed numbers are deferred to Appendix (Tab. 3, Tab
[Figure]
are improved further, showing the proposed data picking scheme works in most cases. (3) Finally, “KD+CutMixPick” scheme can be combined with more training iterations (960 epochs), which delivers even higher accuracies. If comparing our best results (KD+CutMixPick 960 ) with those of CRD (though this is not an apples-to-apples comparison since the two methods focus on different aspects to improve KD), we can see our approach outperforms CRD on 6/7 pairs.
In the last two rows of Tab. 1, when CRD [45] is armed with our proposed “CutMixPick” and more training iterations, its results can be further advanced consistently .
This demonstrates that our method is general and can readily work with those methods focusing on better KD loss functions.
Results on Tiny ImageNet .
We also evaluate CutMix and CutMixPick on a more challenging dataset,
Tiny ImageNet.
Similar to the case on CIFAR100, we have results on different teacher-student pairs, shown in Tab. 2.
For prolonged training, we train for 480 epochs instead of 960 to save time.
Most claims on the CIFAR100 dataset are also validated here: (1) “KD+CutMix” is better than KD, which is verified on all pairs.
(2) “KD+CutMixPick” is better than “KD+CutMix”, verified on 6/7 pairs.
The exception pair is resnet56/resnet20, where adding data picking decreases the accuracy slightly by 0.11%.
(3) When “KD+CutMixPick” is trained for prolonged iterations, the students perform best.
[Table] [Table] We further evaluate our DA methods equipped with CRD [45], shown in the last two rows of Tab. 2.
Our “CutMixPick” method further advances the prior SOTA on 5 pairs.
When CRD+CutMixPick is trained for 480 epochs (instead of 240), further improvement can be observed on 6 of 7 pairs. Apply DA to More KD Methods .
Notably, we achieve the above performance boosting simply using the original KD loss [17] [Table] with no bells and whistles .
This justifies one of our motivations in this paper, i [Table] e. , existing KD methods [34, 31, 45] mainly improve KD at the network output side via better loss functions, while we propose to improve KD at the input side and show this path is just as promising.
Actually, this performance boosting effect is generic – we also applied CutMix to another 5 top-performing KD methods on CIFAR100: AT [53], CC [34], SP [46], PKT [32], and VID [1].
All the pairs see accuracy gains; half of them are even improved by more than 1% point. Results on ImageNet100 and ImageNet . These results are deferred to Appendix A.1.
In general, we observe that the correlation between T. stddev and S. test loss become weaker on ImageNet and ImageNet.
This is because these two datasets are inherently harder than CIFAR100 and Tiny ImageNet. Nevertheless, the p-value of the correlation is still below 5% on ImageNet100, i.e. , still statistically significant, suggesting our theory can generalize to large-resolution ( 224 × 224 ) datasets.  Conclusion
In this paper, we attempt to precisely answer what makes a good data augmentation in knowledge distillation.
By analyzing the generalization gap of the student under different sampling schemes, we reach the conclusion that a good data augmentation scheme should reduce the variance of the cross-entropy ( i.e. , the distilled risk) between the teacher and student.
Based on this, we propose a new metric, the stddev of the teacher
’s mean probability (T. stddev), as a feasible measure of the quality of data augmentation techniques.
Empirical studies with various teacher-student pairs confirm the efficacy of the proposed metric for data augmentation quality in KD.
Besides the theoretical understanding, we also develop an entropy-based data picking scheme to further enhance the prior best augmentation scheme (CutMix) in KD.
Finally, we show how we can obtain considerable KD performance gains simply by using a stronger DA guided by the proposed theory.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See our Appendix.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See our Appendix.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Sec. 3.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Sec. 3. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See our Appendix.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See our Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] All of our CIFAR100/Tiny ImageNet/ImageNet results are averaged by at least three random runs, mean and stddev reported.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See our Appendix. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] See our Appendix.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] We include the code link.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] The data we use are all publicly available.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes] The data we use contains no personally identifiable information or offensive content.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[No] No crowdsourcing in this work.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[No] No crowdsourcing in this work.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [No] No crowdsourcing in this work.
A Appendix A.1 More Results
Detailed numerical results of S. test loss and T. stddev on CIFAR100 and Tiny ImageNet . See Tab. 3, Tab. 4, Tab. 5, Tab. 6. These tables are the numerical results that we use to plot Fig. 4.
[Table]
[Table]
ImageNet100 and ImageNet Results .
See Fig.
5, Tab. 7 and Tab. 8 for the results.
In general, we find it is harder to verify the proposed proposition on these two more challenging datasets – As seen, the coefficients turn smaller while p-values larger than the cases on CIFAR100 and Tiny ImageNet.
On ImageNet100, the p-values are below (or very close to) 5%, suggesting the correlation is still strong.
On ImageNet, however, the results show a weak correlation between T. stddev and S. test loss, against our theory.
Currently, we are not very sure why this happens sorely on ImageNet.
After all,
ImageNet100 is a subset of ImageNet and the proposed theory works fairly well on ImageNet100.
One possible reason may be – full ImageNet historically was shown especially hard to make KD work, e.g. , in [7], they mentioned “ (Page 4) ...
it is still a mystery why no teacher improves accuracy on ImageNet.
Despite multiple recent papers in knowledge distillation, experiments on ImageNet are rarely reported.
The few that do report find that standard setting of knowledge distillation fails on ImageNet [26] or perform an experiment with a small portion of ImageNet [21] ”.
Besides, we may notice the authors of CRD [45] mentioned in their GitHub issue ‡ : “ I have been struggling a bit to get KD work as well on ImageNet with ResNet-18 as the student network ”.
We conceive that the weak correlation on ImageNet may be related to this KD underperformance on ImageNet and may be related to the number of classes of the dataset.
We shall continue to investigate this in our future work.
[Table]
[Table]
Discussion: This paper discovers that “more correlation in the data, worse generalization ability”.
Is this an already-known fact in statistical learning [22, 47]?
Our work is starkly different from the established theory in [22, 47] in that they study the dependency in the input data , while an important point in our theory is that we consider the teacher’s output of the input, not the input per se.
Namely, the proposed theory must be discussed in the context of KD .
Existing works [22, 47] clearly are not in this scope.
Use test accuracy vs. test loss as the measure of student’s performance .
In the paper, we mentioned we use test loss instead of test accuracy as the measure of student’s performance.
The primary consideration, as discussed in the main paper, is that accuracy may be misaligned with loss sometimes ( i.e. , ideally we expect higher accuracy coincides with lower loss ; while in practice, we observe several counterexamples).
Here we show an example of calculating the correlation between
T. stddev and S. test accuracy in Fig.
As seen, if accuracy used, we observe no positive correlation between T. stddev and S. performance (which is the S. test accuracy here), while the correlation is strong if we use test loss.
This shows the importance of using the correct measure for student’s performance in a theoretical investigation of KD problems, as also noted by [29].
KD vs. CE loss on wrn_40_2/wrn_16_2 and vgg13/vgg8 .
See Fig.
These plots show more examples that we can harvest considerable performance gain simply by using a strong DA with prolonged training.
A.2 More Explanations from Eq. (7) to Eq. (8)
In Proposition 3.1, the main idea is to have lower covariance among samples (in terms of the teacher’s outputs), as suggested by Eq.
Then, it seems that the most straightforward idea is to use the covariance (or possibly the normalized covariance, i.e. , correlation) of the teacher’s output as metric to capture the dependency among samples.
Why does the paper not use this idea?
We first show this idea does not work as well as our proposed metric.
Then we explain why.
Given a batch of input images, the teacher’s output probability is P ∈ R N × C , where N is the batch size, C is the number of classes.
Then we consider the covariance and correlation of P and try
to
[Figure]
[Figure]
[Table]
construct a new metric from them:
[Equation]
We collect this (mean) covariance v (a scalar) and correlation r (a scalar) over many batches and then take the average of all the collected v and r ,
[Equation]
where K is the total number of batches (10 epochs in our experiments, K = 7 , 818 ).
Then, we replace the proposed “T. stddev” with ¯ v and ¯ r in Tab. 3, resulting in two new tables: Tab.and Tab. 10.
As seen, the foremost impression of Tab. 9 and Tab. 10 is that they have many red uparrows, which implies the metric goes against the supposed order, undermining its efficacy.
E.g. , AutoAugment is stronger than Cutout (in that Autoaugment includes Cutout as a part; also AutoAugment performs better than Cutout as the test losses indicate).
However, by ¯ v (see Tab. 9), AutoAugment is ranked “ weaker ” than Cutout with teacher resnet56 and vgg13.
As for Tab. 10, the correlation metric ¯ r is even worse than ¯ v in the sense that it incurs even more red uparrows than ¯ v .
Especially, CutMixPick (T. ent.) is the best DA (delivering the lowest test loss) while by ¯ r it “ underperforms ” all the other DA schemes except Identity and Autoaugment, with the teacher resnet56.
Why does the covariance metric ¯ v not work as well as our proposed metric?
The covariance metric arises directly from our proposition, if the proposition is correct, why does it not work well?
Fundamentally, the reason is that the covariance among samples is hard to estimate accurately.
When we calculate the covariance matrix Cov( P, P ) , P is of shape N × C .
Note, each row of P
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
correlation analysis in our paper. is an attribute (or random variable, RV) and each column of P is an observation § .
The number of observations of P is the number of classes, which is a constant for a given dataset (in the current case, it is 100 for the CIFAR100 dataset).
Intuitively, given two random variables, in order to estimate the covariance between them, merely 100 observations is not very abundant.
Namely, the limited observations render the direct estimation of the covariance among samples inaccurate .
This can explain the counterexamples between ¯ v and test loss in Tab. 9.
(If the covariance metric is accurately estimated, it would not be surprising that the correlation in
Tab. 10 is inaccurate, either.
Especially, the normalization itself may not be desired.
We put the results of using correlation as metric simply for a reference.)
Then, how does the proposed metric resolve this “limited observation” problem?
The “smart thing” of the proposed metric is that it considers the sum of RVs instead of the RV itself.
In statistics, it is well-known that, when multiple RVs are added together, the variance (or equivalently, stddev) of the sum RV will take into account the covariance among its members, as shown below:
[Equation]
where X i (with a little abuse of notation here) is the teacher’s output probability of the i -th example.
Essentially, we want to estimate the covariance term in RHS ( i.e. , (cid:80) 1 ≤ i<j ≤ N Cov
( X i , X j ) ).
Now, what we do in the paper is to use the LHS ( i.e. , Var( (cid:80) Ni =1 X i ) ) as a proxy of the covariance term.
Clearly, there is a hidden assumption here to allow us to use such a proxy: the first term of RHS ( i.e. , (cid:80)
Ni [Equation] Var( X [Equation]
) ) stays (nearly) the same for different samples ( i.e. , different batches) of { X i } .
In our work, there are two conditions to make this hidden assumption hold in practice.
(1) The augmented images are based on the original images.
We can still consider them in the same domain (in our
[Figure]
higher (or lower) than the one right above it .
If “S. test loss” change is at a different direction from the “ ¯ v ” change, we highlight the arrow in red.
[Table]
Proposition 3.1, X i is actually assumed to be drawn from the same distribution),
so Var( X i ) is close for different X i .
(2) We have abundant samples ( N = 640 in our experiments for CIFAR100 and
Tiny ImageNet).
The sum effect of so many samples makes (cid:80)
{+Ni =
+}
Var( X i ) stabilize to a constant.
We can actually come up with counterexamples that intentionally break the hidden assumption.
E.g. , consider a “bad” DA which turns all input images to a constant image.
Then the input batch will become a repetition of just one image.
When we input such a batch to the teacher, the teacher’s output probability will be the same.
Then the Var( (cid:80) Ni =1 X i ) term will become zero.
It cannot be a legitimate proxy for the covariance term anymore as the first term of RHS now is zero.
We are aware of the existence of such counterexamples.
Notably, they do not affect the validity of the proposed metric.
Here are the reasons – For one thing, such bad DA rarely appears in practice.
The strong correlation per se already demonstrates the efficacy of the proposed metric in practical cases.
For another thing, there is an important design in our training setup to avoid such counterexamples – Note we add the augmented images to the original images instead of replacing them (see Fig. 3).
Therefore, in a batch, it is unlikely that all the images degenerate to one image.
Namely, the counterexample we just mentioned never appear in practice.
Then how to understand such counterexample?
Why it appears to go against our theory?
Fundamentally, the counterexample we just mentioned breaks one of the key assumptions of our proposition in the first place – each individual sample (including the original image and the augmented image ) obeys the same true (unknown) data distribution D .
The bad DA transforms all input images to a constant image, which already means the individual sample is not drawn from the true data distribution since randomly drawing from the true data distribution will not give us the same image.
To further confirm this, we implement the “bad” DA and re-check the validity of the proposed metric ¯ m with wrn_40_2/wrn_16_2 on CIFAR100.
When using the “bad” DA, we have the metric value
[Table]
¯ m = 0 . 022439 .
Compare this to Tab. 3, we will notice it is ranked the highest ( i.e. , the worst ).
This agrees with our expectation.
In short, the counterexamples above do not affect the efficacy of our proposed metric in practice.
A.3 Example of Calculating T. Stddev
Take CIFAR100 for an example.
It has 50,000 training samples.
With batch size 64, one epoch amounts to 782 batches.
During training, the K in Eq.
(9) is set to 10.
During the iteration of training data loader, we collect the teacher’s outputs.
Every K/ 2 training batches (the division of 2 is because we append the augmented images to the original images), we obtain a matrix of shape [ K × batch _ size , num _ classes] .
Then we average this matrix along the 1st axis to give us a vector, i.e. , the u in Eq.
In brief, every K/ 2 batches, we have one sample of u .
In total, we run the data loader for 10 epochs (10 is empirically set; more epochs should give us more accurate estimation but we just find 10 is good enough), which gives us 7820/5=1654 u ’s, i.e. , a matrix of shape [1654 , num _ classes] .
Then we calculate the variance along the 1st axis, giving us a vector of shape [num _ classes] , i.e. , the m in Eq.
Finally, we average the stddev elements in m to obtain the proposed scalar metric.
For more details, please check our code at: http://github.com/MingSun-Tse/Good-DA-in-KD.
A.4 CutMix Sample Analysis on ImageNet
CutMix sample analysis and why KD is naturally suited to exploit CutMix .
During the KD training of resnet34/resnet18 on ImageNet, we recorded the CutMix samples on which the teacher disagrees with the CutMix scheme on the label.
We call this label disagreement issue .
As show in Fig. 8.
there exist cases where the image cut from one image covers the salient object in the other.
For example, the cab in (a) completely covers the ground beetle.
In this case, using the label by CutMix does not make sense anymore.
A similar problem appears on (b).
Note that these misleading labels by CutMix are rectified when the teacher is employed to guide the student.
The teacher assigns the correct label “cab” to (a) and “Yorkshire terrier” to (b) (which is still not the true label “Tibetan terrier” but it is clearly more relevant and “Tibetan terrier” is also in the toppredictions).
For (c) and (d), they pose a problem more than occlusion: the foreground cut in (c) is labeled as “acoustic guitar”, however, the cut is too small for us to make it out without knowing the label.
Meanwhile, the background object “Arabian camel” is occluded.
Then the grids in the picture turn out to be the most salient part.
If we look at the predictions of the teacher, “shopping cart” and “shopping basket” clearly make more sense than either of the original two labels.
A similar issue happens on (d), where the “Indian elephant” is largely occluded.
The foreground cut is labeled “quill” but the bottle in the middle is more salient.
Thus the teacher predicted it as “coffeepot”, “milk can”, etc .
In order to see how severe the label disagreement issue is, we counted the number of these synthetic samples and found that on more than half of the samples (52.1%) produced by CutMix, the teacher model and CutMix hold a different view regarding the label.
Many of these suffer from the problem shown in Fig.
The KD loss can rectify these label mistakes.
This further shows the interplay between KD and DA: KD thrives on DA and in turn, some DA schemes are more reasonable for KD (than CE) where a teacher can supply more relevant labels.
[Figure]
A.5 Dataset License and Hardware Condition
The four datasets used in this paper (CIFAR100, Tiny ImageNet, ImageNet100, ImageNet) are all publicly available.
One KD experiment with one NIVIDA 2080Ti GPU on CIFAR100 takes aroundto 6 hrs; while an experiment on Tiny ImageNet, ImageNet100, or ImageNet may take up to 24 hrs on the 2080Ti GPU.
In general, the experimenting is not compute intensive .
For details, please refer to our GitHub code: http://github.com/MingSun-Tse/Good-DA-in-KD.
A.6 Limitations and Potential Negative Societal Impacts
Limitations .
The proposed metric is calculated with finite training samples.
The precision of the metric values will necessarily depend on the number of training samples used.
In this regard, we have tried our best to avoid the influence of statistical variations (using abundant samples).
The presented results in the paper are shown stable.
Especially, the correlation in Tabs.
1 and 2 is strong.
This fact itself is a proof that the limitation of our paper in this regard is small.
Potential Negative Societal Impacts .
Simply put, this work proposes theories and algorithms that can boost the performance of a network with the aid of a larger network ( i.e. , the teacher-student distillation), that is, make it smaller, faster, and possibly consume less energy in practical applications.
We focus on the classification task, which is generally the foundation of the many up-stream computer vision tasks like detection and segmentation.
Therefore, this work potentially has a broad application especially in the computer vision areas.
The algorithm itself has few negative societal issues, but when it makes many AI-driven technologies applicable in practice, the impact really depends on how humans use them.
This actually falls into the general ethical discussion on whether AI is good or not.
Beyond this scope, this work does not have specific negative societal impacts brought by its potential application, to our best knowledge.