What Makes a “Good” Data Augmentation in Knowledge Distillation – A Statistical Perspective
Project: http://huanwang.tech/Good-DA-in-KD
[Figure]
𝑥 !
[Figure]
t
[Figure]
Howtodefine“stronger”DA
[Figure]
) and T. stddev, showing that T. stddev well captures the “goodness” of different DA schemes in KD.
The most striking fact from this plot may be: T. stddev is purely calculated with the teacher (no any student used) while it can “predict” the relative order of the student ’s performance, implying the “goodness” of DA in KD probably is student-invariant.
Abstract
Knowledge distillation (KD) is a general neural network training approach that uses a teacher model to guide the student model.
Existing works mainly study KD from the network output side ( e.g. , trying to design a better KD loss function), while few have attempted to understand it from the input side.
Especially, its interplay with data augmentation (DA) has not been well understood.
In this paper, we ask:
Why do some DA schemes ( e.g. , CutMix) inherently perform much better than others in KD?
What makes a “good” DA in KD?
Our investigation from a statistical perspective suggests that a good DA scheme should reduce the covariance of the teacher-student cross-entropy .
A practical metric, the stddev of teacher’s mean probability (T. stddev), is further presented and well justified empirically.
Besides the theoretical understanding, we also introduce a new entropy-based data-mixing
DA scheme, CutMixPick , to further enhance CutMix.
Extensive empirical studies support our claims and demonstrate how we can harvest considerable performance gains simply by using a better DA scheme in knowledge distillation.
Introduction
Deep neural networks (DNNs) are the de facto methodology in many artificial intelligence areas nowadays [23, 35].
How to effectively train a deep network has been a central topic for decades.
In the past several years, efforts have mainly focused on better architecture design ( e.g. , batch normalization [19], residual blocks [13], dense connections [18]) and better loss functions ( e.g. , label smoothing [41, 28], contrastive loss [17], large-margin softmax [24]) than the standard cross-entropy (CE) loss.
Knowledge distillation (KD)
[16] is a training method that falls into the second group.
In
KD, a stronger network – called teacher – is introduced to guide the learning of the original network – called student – by minimizing the discrepancy between the representations of the two networks,
[Equation]
where D KL represents KL divergence [22]; α ∈ (0 , 1) is a factor to balance the two loss terms;
L CE denotes the cross-entropy loss; y is the one-hot label and p ( t ) , p ( s ) stand for the teacher’s and student’s output probabilities over the classes; τ is a temperature constant [16] to smooth predicted probabilities.
KD allows us to train smaller, more efficient neural networks without compromising on accuracy, which facilitates deploying deep learning in resource constrained environments ( e.g. , on mobile devices).
KD has found plenty of applications in many tasks [6, 46, 12, 20, 47].
Most existing KD methods have attempted to improve it by proposing better KD loss functions applied at the network outputs [32, 29, 43].
Few works have considered KD from the input side .
Especially, the interplay between KD and data augmentation (DA) [38] has not been well understood so far (note, by DA here, we mean the conventional DA concept: generating a new input by transforming one or multiple inputs .
A broader scope of DA may involve the neural network , e.g. , dropout can be seen as a kind of DA [3].
We do not consider this type of DA in this paper due to the limited length).
In this work, we ask: What makes a “good” data augmentation in knowledge distillation ?
A clear answer to this question has many benefits.
First, theoretically, it can help us towards a better understanding about how data augmentation plays a role in KD.
Second, practically, it can bring us considerable performance gain – in Fig.
2, we show test error rates using the standard CE loss (no teacher) vs. using KD loss (with a teacher).
As seen, a stronger DA can lower the test error rate and admit more training iterations without overfitting in KD.
It is pretty obvious to see that “Flip+Crop” is stronger than “Flip” alone in Fig.
However, for other DA schemes, such as Mixup [51] vs. AutoAugment [7], which one is stronger?
Not very clear.
We thus desire a principled way ( e.g. , a concrete metric) to make the vague concept “stronger” exact.
Presenting such a theoretically sound metric and empirically validating its effectiveness is the goal of this paper.
[Figure]
Intuitively, a good DA should enrich the input data and expose more knowledge of the teacher so that the student can generalize better.
We formalize this idea from a statistical learning perspective.
Specifically, we will show a good DA scheme is defined by a lower variance (or covariance) of the teacher
’s mean output probability over different input samples, which ultimately leads to a lower generalization gap for the student.
The proposed theory is well justified by our extensive empirical studies on CIFAR and Tiny ImageNet datasets with various pairs.
The proposed theory well explains why CutMix is better than other alternatives (such as
Mixup [51], AutoAugment [7]) in KD.
In addition to the new theoretical results, we also propose an entropy-based data picking scheme to select more informative samples for KD, which can deliver even lower variance of the teacher’s mean probability as well as lower generalization error of the student.
We make the following contributions in this paper:
• We present a proven proposition (Proposition 3.1) that precisely answers what defines a better DA in KD:
Given a fixed teacher, a better DA is the one that gives a lower variance of the teacher’s mean probability.
• The proposition is well justified empirically on standard image classification datasets with many teacher-student pairs.
• An entropy-based data picking scheme is introduced to further reduce the variance of the teacher
’s mean probability, which can further advance CutMix, the prior state-of-the-art DA approach among those evaluated in this paper.
• Empirically, we show how the presented theory can benefit in practice – we can simply enhance the existing KD methods by using a stronger DA and prolonged training iterations.
Related Work
Knowledge Distillation (KD) .
The general idea of knowledge distillation is to guide the training of a student model through a (stronger) teacher model (or an ensemble of models).
It was pioneered by
Buciluˇa et al.
[4] and later refined by Hinton et al.
[16], who coined the term.
Since its debut, KD has seen extensive application in vision and language tasks [6, 14, 46, 20, 47].
Many variants have been proposed regarding the central question in KD, that is, how to define the knowledge transferred from the teacher to the student.
Examples of such knowledge definitions include feature distance [33], feature map attention [50], feature distribution [30], activation boundary [15], inter-sample distance relationship [29, 32, 25, 44], and mutual information [43].
Several works [28, 37, 5] investigate the connection between label smoothing and knowledge distillation.
Another line of works ( e.g. , [42]) attempts to understand KD more theoretically.
Over the past several years, the progress has been made primarily for intermediate feature maps and network outputs ( i.e. , through a better loss function).
In contrast, our goal is to improve the KD performance at the input end with the help of data augmentation.
We will show this path is as effective and also has much potential for future research.
Data Augmentation (DA) .
Deep neural networks are prone to overfitting, i.e. , building input-target mappings using undesirable or irrelevant features (like noise) in the data.
Data augmentation is a prevailing technique to curb overfitting [38].
In classification tasks, data augmentation aims to explicitly provide data with label-invariant transformations (such as random crop, horizontal flip, color jittering, Cutout [11]) during training so that the model can learn representations robust to those nuisance factors.
Recently, more advanced data augmentation methods were proposed, which not only transform the input, but also transform the target.
For example, Mixup [51] linearly mixes two images with the labels mixed by the same linear interpolation.
Manifold Mixup [45] is similar to
Mixup but conducts the mix operation in the feature level instead of pixel level; CutMix [48] pastes a patch cut from an image onto another image with the label decided by the area ratio of the two parts.
AutoAugment [7] is a strong DA method that finds an optimal augmentation policy from a large search space via reinforcement learning.
When both the input and target are transformed simultaneously, the key is to maintain a semantic correspondence between the new input and new target.
Unlike these methods, which focus on general classification using the cross-entropy loss, our work investigates the interplay between data augmentation and knowledge distillation loss and proposes new data augmentation specifically for knowledge distillation.
Some recent KD works also involve the utilization of DA in KD, such as [2, 8].
Especially, [2] also employs Mixup and prolonged training to enhance the student performance in KD, akin to ours.
Yet it is worthwhile to note that our paper is substantially different from theirs, in that we are seeking the theoretical reason explaining how to define a better DA in KD to deliver better performance; these works mainly investigate in an empirical fashion, with no theoretical results presented.
Meanwhile, our work is not limited to one specific DA (see Sec. 5).
We target a general theoretical understanding which can apply to a broad scope of DA schemes (fortunately, as our experiments show, the proposed theory and the derived DA “goodness” measure indeed capture it).
The fact that [2] utilizes Mixup [51] and prolonged training to deliver 82.8% top-1 accuracy with resnet50 [13] on ImageNet [10] can be a direct proof that the principle proposed in this work can bring us very promising practical benefits.
One recent work [9] also conducts empirical studies of the impact of DA on KD.
They first apply
DA ( e.g. , Mixup/CutMix) to the teacher training then conduct the KD step as usual (no extra DA in this step).
Our investigation is the exact opposite to their setup: We train the teacher as usual (no Mixup/CutMix), then in the KD step we employ a more advanced DA ( e.g. , Mixup/CutMix).
Interestingly, they conclude that the teacher trained with Mixup/CutMix hurts the student’s generalization ability, while we consistently see student performance boost via a stronger DA.
Another recent work [40] utilizes KD to re-label mixed samples in Mixup to fix the inaccurate labelling problem of Mixup.
Their work shows that KD can be used to make Mixup more generally useful, which is orthogonal to the topic of this work.
The work of [27] presents a statistical perspective to understand how the softened probabilities in KD are better than the one-hot hard labels.
Our work is inspired by their Bayes teacher notion.
This said, our work is different from theirs in that we focus on explaining how data augmentation plays a role in KD and answer what characterizes a good DA, while they attempt to answer why the knowledge distillation loss is better than the standard cross-entropy loss.
Theoretical Investigation 3.1 Prerequisites: Multi-Class Classification with KD
Given a training set S = { ( x n , y n ) } Nn =1 ∼ D N , where D is the joint distribution for input-output random variable pair ( x, y ) , the goal in multi-class classification is to pin down a predictor f : X → R C from a hypothesis class H , where X is the input space and C refers to the number of classes.
The predictor f is supposed to minimize the true risk
[Equation]
where L stands for the loss objective function ( e.g. , cross-entropy); the subscript D of R D is to emphasize that the true risk is defined on the data distribution.
The true risk is approximated in practice on a separate test set.
For training, the predictor aims to minimize the empirical risk defined on the training sequence S :
[Equation]
where e y ∈ { 0 , 1 } C is a one-hot vector indicating the label y ∈
[ C ] = { 1 , 2 , · · · , C } .
The subscript
S of R S is to emphasize the empirical risk is defined on the finite sampled data points.
In the context of KD, the one-hot hard target vector
e y is replaced with a probability vector p ( t ) ( x ) ∈
R C + , giving us the empirical distilled risk of f :
[Equation]
where the super-script t indicates the fixed teacher model.
The theory concerning why Eq.
(4) is better than Eq.
(3) has been established in [27].
Interested readers may refer to their paper for more details.
Next, we look into how the data augmentation plays a role in KD.
3.2 What Makes a “Good” DA in KD?
Intuitively, a better DA should provide more information, i.e. , expose more knowledge of the teacher so that the student can absorb more and thus generalize better.
We make this idea rigorous as follows.
Proposition 3.1.
Given a bounded loss function and a fixed teacher model with the empirical distilled risk defined in Eq.
(4), for any predictor f , consider two sampled sequences S 1 ∈ D N and S 2 ∈ D N , they are made up of N elements sampled from the same distribution D , while not i.i.d.
(especially when data augmentation is employed).
If the elements in S 1 present a larger correlation than those in S 2 , then the student’s generalization gap trained on S 1 will be greater than that trained on S 2 :
[Equation] [Equation]
Proof.
Let ∆ =
ˆ R S ( f ) − R D ( f ) .
By the definition of variance, E S [∆ 2 ] =
Var S [∆] + (E S [∆]) 2 .
To make the notation clearer, we define q ( x i )
= − p
( t ) ( x i )
⊤ log( f ( x i )) .
(1) Since R D ( f ) is a constant (albeit unknown),
[Equation]
where the second last equation is because each element x i in S is drawn from the same distribution
D .
From the RHS of the last equation, we can clearly see that for S 1 , S 2 , E S [∆] is the same.
(2) Then we consider Var S [∆] .
Again, since R D ( f ) is a constant, we only need to consider
[Equation]
where Cov[ · , · ] stands for covariance.
From the last item in Eq.
(7), we can see that the covariance part is different for different sampled S ’s. If the samples in S 1 present a larger correlation than those in S 2 , we will have Var S 1 [∆] >
Var S 2 [∆] , which further leads to increased generalization gap for the student, i.e. , E S
[( R S 1 ( f ) − R D ( f )) 2 ] > E S 2 [( R S 2 ( f ) − R D ( f )) 2 ] .
The proof is finished.
Practical Use: Stddev of Teacher’s Mean Probability (T. stddev) .
Note in Proposition 3.1, the predictor f ( i.e. , the student model) can be any one (not necessarily a converged model).
Each student would have an order of different DA schemes regarding which is better “in its opinion”.
Presumably, different students will lead to different such orders.
For practical use, we must pick a certain student as oracle to conduct the evaluation.
Here, we can play a trick that can make it rather simple to use our theory – assume there is a student that performs exactly the same as the teacher (this assumption is not unpractical, since one straightforward example is to use the teacher as student).
Then we have
[Equation]
In this case, we only need the covariance of the teacher’s probability to measure the “goodness” of a certain DA technique, no need for the student .
Despite not using any information of the student, the proposed metric turns out to correlate surprisingly well with the student’s performance (see Fig. 4).
Based on Eq.
(8), when using S 1 vs. S 2 as training sequence, the fundamental factor answering for the variance gap of Var S [∆] boils down to the covariance in { p ( t ) ( x i ) } Ni =1 .
That is, a larger covariance in { p ( t ) ( x i ) }
Ni =1 leads to a larger covariance in { p ( t ) ( x i )
⊤ log( p ( t ) ( x i )) }
Ni =1 in Eq.
(8), which ultimately leads to a higher Var S [∆] in Eq.
The next step is to find a feasible way to estimate the covariance in { p ( t ) ( x i ) } Ni =1 .
Consider the average variable (denoted as u here) of several random variables { p ( t ) ( x
[Equation]
[Equation]
Its variance (or equivalently, stddev) inherently takes into account the covariance among its addends.
Therefore, we can use the variance of u as a proxy for the covariance in { p ( t ) ( x k ) }
Kk =1 :
[Equation]
[Figure]
where K is a pre-defined number of samples of set S ∗ to realize the averaging effect ( e.g. , K = 640 in our experiments for CIFAR100 and Tiny ImageNet – see Appendix Sec. ??
for a concrete calculation example).
Note, u can be interpreted as the teacher
’s mean probability over K input samples.
If there is a large covariance among { p ( t ) ( x k ) } Kk =1 , m will be large (in an element-wise sense).
Then the average of m over classes, i.e. , the ¯ m (we do this averaging simply because we desire a scalar metric), is a good indicator to capture such covariance.
We thus formally introduce ¯ m , the (averaged)
T. stddev, as the proposed metric to measure the quality of a data augmentation scheme .
A lower ¯ m implies a better DA by our definition.
In the experiments, we will show this metric defined purely using the teacher can accurately characterize the generalization error of the students after distillation.
Evaluated Algorithms 4.1 Extend Existing DA Approaches for KD
Given an existing DA approach, this section explains how we adapt it properly to the case of KD.
Specifically, let x 0 denote the raw data, x denote the transformed data by the standard augmentation (random crop and flip).
Illustrated in Fig. 3(b), we will add the DA following x to obtain x ′ .
Unlike the common data augmentation where only the transformed input is fed into the network, we keep both the input x and x ′ for the training (as such, the number of input examples is doubled).
The consideration of keeping both inputs is to maintain the information path for the original input x so that we can easily see how the added information path of x ′ leads to a difference.
For x , its loss is still the original KD loss, consisting of the cross-entropy loss and the KL divergence (Eq.
Of special note is that, for x ′ , its loss is only the KL divergence, i.e. , we do not use the labels assigned by the DA algorithm (for example, in the original Mixup and CutMix, they assign a linearly interpolated label to the augmented sample) because these labels can actually be wrong (see Appendix Sec. ??
for concrete examples on ImageNet).
In fact, not using the hard label has another bonus.
A dataset augmentation scheme which employs CE loss has to provide corresponding labels as supervisory information.
In order to maintain the semantic correspondence, it cannot admit very extreme transformations for data augmentation.
In contrast, in the Mixup/CutMix+KD setting described above, the data augmentation scheme need not worry about the labels as they are assigned by the teacher – the recent work [2] refers this kind of utilization of DA as function matching of the teacher.
As a result, it can admit a broader set of transformations to expose the teacher’s knowledge more completely.
This reflects that data augmentation in KD has more freedom than in CE.
Among all the data augmentation techniques evaluated in this paper, we will show CutMix works best .
The fundamental reason for its success, as we will show, is that it achieves a much lower variance of the teacher
’s mean probability, implying it produces more diverse data than its counterparts.
4.2 CutMixPick: Enhancing CutMix with Entropy-Based Data Picking
In this section, we propose a data picking scheme to further reduce the variance of the teacher’s mean probability.
The idea is partly inspired by active learning [36].
In active learning, the learner enjoys the freedom to query the data instances to be labeled for training by an oracle ( i.e. , the teacher in our case) [36].
Since the augmented data can vary in their quality, we can introduce a certain criterion to pick the more valuable data for the student.
Intuitively, we regard a sample with more information is of higher quality.
Therefore, we take
Shannon’s entropy of the teacher’s output probability as a natural measure to select samples,
[Equation]
Note this formula is in exactly the same form of Eq.
Empirically, we will also show the data selected by this formula indeed results in lower ¯ m and better test loss for the student.
Concretely, given a batch of data, we first apply CutMix to obtain a bunch of augmented samples.
Then sort all the augmented samples by Eq.
(11) in ascending order and keep the top r (a pre-defined percentage constant, r = 0 .
5 in our experiments) samples.
This simple technique can be rather effective according to our empirical study.
Another seemingly potential alternative is to use the student’s entropy as the picking metric.
The intuition behind this is that a high-entropy sample in the view of the student can be regarded as a hard example, too.
Learning with these hard examples may expand the student’s knowledge by squeezing its blind spots.
Despite this intuitively plausible explanation, in practice, we will show this scheme actually under-performs the former student-agnostic scheme in Eq.
(11), which is a bit surprising.
Experimental Results
Datasets and Networks .
We evaluate our method primarily on the CIFAR100 [21] and Tiny
ImageNet * datasets.
CIFAR100 has 100 object classes (32 × 32 RGB images).
Each class hasimages for training and 100 images for testing.
Tiny ImageNet is a small version of ImageNet [10] with 200 classes (64 × 64 RGB images).
Each class has 500 images for training, 50 for validation and 50 for testing.
To thoroughly evaluate our methods, we benchmark them on various standard network architectures: vgg [39], resnet [13], wrn [49], MobileNetV2 [34], ShuffleV2 [26].
We will also include results on ImageNet100 (a randomly drawn 100-class subset of ImageNet) and ImageNet.
Benchmark Methods .
In addition to the standard cross-entropy training and the original KD method [16], we also compare with the state-of-the-art distillation approach, Contrastive Representation Distillation (CRD) [43].
It is important to note that our method focuses on improving KD by using better inputs , while CRD improves KD at the output end ( i.e. , a better loss function).
Therefore, they are orthogonal and we will show they can be combined together to deliver even better results.
Hyper-Parameter Settings .
The temperature τ of knowledge distillation is set to 4 following
CRD [43].
Loss weight α = 0 .
9 (Eq.
For CIFAR100 and Tiny ImageNet, training batch size is 64; the original number of total training epochs is 240, with learning rate (LR) decayed at epoch 150, 180, and 210 by multiplier 0.1.
The initial LR is 0.05.
All these settings are the same as CRD [43] for fair comparison.
Note, in our experiments we will present the results of more training iterations.
If the number of total epochs is scaled by a factor k , the epochs after which learning rate is decayed is also be scaled by k .
For example, if we train a network for CIFAR100 for 480 epochs ( k = 2 ) in total, the learning rate will be decayed at epoch 300, 360, and 420.
We use PyTorch [31] to conduct all our experiments.
For CIFAR100, we adopt the pretrained teacher models from CRD † for fair comparison.
For Tiny ImageNet and ImageNet100, we train our own teacher models.
For ImageNet, we use torchvision models following CRD [43].
Data Augmentation Schemes .
We investigate the following popular DA schemes:
• Identity : This augmentation scheme simply makes a copy of each batch data during training, which should be the lower bound of all DA schemes discussed here since it adds no new information.
• Flip : Random horizontal flip.
• Flip+Crop : Random horizontal flip and random crop.
This is the standard DA extensively used in 2D image recognition task (such as on CIFAR and ImageNet datasets).
• Cutout [11]: Cutout occludes a small random patch of an image.
• AutoAugment [7]: AutoAugment is an ensemble of a collected DA schemes.
The DA policy is automatically selected by reinforcement learning instead of manually.
• Mixup [51]: Mixup applies linear interpolation between two inputs and applies the same linear interpolation to their labels to make the new label for the augmented sample.
• CutMix [48]: CutMix cuts a small patch from a source image and pastes it to another source image.
The resulted image is considered as a new input.
The target for the new input is a linear interpolation from the two source labels.
5.1 Empirical Verification of Our Proposition
Before presenting results, one point worth mentioning is that, in this section we use test loss instead of accuracy as the measure of student’s performance in KD, because (1) all the formulas in Sec.
3 are derived using numerical loss instead of accuracy; (2) more importantly, it is observed that accuracy can mismatch with loss – a model may achieve a better accuracy, meanwhile higher loss too [27], which we also observe several times in our experiments.
Using accuracy as measure would prevent us from seeing the correlation between T. stddev and student’s performance (see Appendix Fig. ??
for an example).
Potential extension of our theory from loss to accuracy is left for future work.
In Fig.
4, we plot the scatters of S. test loss and T. stddev.
The x/y-axis value of each data point is averaged by at least three random runs (see Tabs ?? / ?? and Tabs ?? for detailed numbers).
(1) In terms of T. stddev, there is a rough trend ( e.g. , on the vgg13/vgg8 pair on CIFAR100): Identity <
Flip < Flip+Crop < Cutout <
AutoAugment < Mixup < CutMix.
These inequalities are wellaligned with our intuition.
E.g. , AutoAugment [7] includes Cutout [11] in its transformation pool, thus should be stronger than Cutout.
This is faithfully reflected by the T. stddev on many pairs.
(2) Obviously, S. test loss poses a clear positive correlation with T. stddev.
Per our theory, lower
T. stddev should lead to better generalization risk for the student.
This is generally well verified in these plots.
We do see some minor counterexamples.
Possible reasons are: 1)
The test loss is obtained on the test set with finite samples, which is only an approximation of the true risk defined on distribution; 2)
The teacher’s mean probability is also evaluated on finite data.
Despite them, the general picture from Fig.
4 still confirms the positive correlation between S. test loss and T. stddev.
The correlation is actually very significant as the p-values indicate.
(3) Importantly, note we only need the teacher to define the quality of a certain DA in KD, no need for the student .
This is more clear if we examine the results in Tabs. ?? / ??
and Tabs. ??
Taking vgg13/vgg8 and vgg13/MobileNetV2 as an example, the students are starkly different but both the student’s performance correlates well with T. stddev.
This observation implies that the “goodness” of
DA in KD is probably student-invariant .
This, notably, is a great advantage in practice since we can decide which DA should be used for the best performance simply using a formula (Eq.
with a few network forwards, without having to train the students physically.
5.2 Boosting KD with Stronger DA
In this section, we present more results to show how the proposed theory can benefit in practice – we can harvest considerable performance gain simply by using a stronger DA in KD.
Prolonged Training .
Notably, a stronger DA produces more diverse data, implying more information.
Intuitively, it should take a student model more training iterations (if the batch size does not change) to fully absorb the excessive information.
That is, a stronger DA conceivably takes more training iterations to fully exhibit its potential.
This intuition is confirmed in Fig.
2 (and also two more pairs wrn_40_2/wrn_16_2 and vgg13/vgg8 in the Appendix).
Thus, in our experiments, we will also report results with prolonged training iterations for maximized performance.
Results on CIFAR100 .
The results on CIFAR100 dataset are shown in Tab. 1.
(1) Comparing the row “KD+CutMix” to “KD", we see CutMix improves the student accuracies on all the pairs.
On the pair resnet32x4/ShuffleV2, the improvement is very significant (more thanpercentage point).
(2) Comparing the row “KD+CutMixPick” to “KD+CutMix”, we see 6/7 pairs
[Figure]
[Figure]
CutoutAutoAugmentMixupCutMixCutmixPick
[Figure]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
ImageNet
[Figure]
, Tiny ImageNet
[Figure]
CutoutAutoAugmentMixupCutMixCutmixPick
[Figure]
The detailed numbers are deferred to Appendix (Tab. ??
, Tab. ?? , Tab. ?? , Tab. ?? ).
are improved further, showing the proposed data picking scheme works in most cases.
(3) Finally, “KD+CutMixPick” scheme can be combined with more training iterations (960 epochs), which delivers even higher accuracies.
If comparing our best results (KD+CutMixPick 960 ) with those of
CRD (though this is not an apples-to-apples comparison since the two methods focus on different aspects to improve KD), we can see our approach outperforms CRD on 6/7 pairs.
In the last two rows of Tab. 1, when CRD [43] is armed with our proposed “CutMixPick” and more training iterations, its results can be further advanced consistently .
This demonstrates that our method is general and can readily work with those methods focusing on better KD loss functions.
Results on Tiny ImageNet .
We also evaluate CutMix and CutMixPick on a more challenging dataset,
Tiny ImageNet.
Similar to the case on CIFAR100, we have results on different teacher-student pairs, shown in Tab. 2.
For prolonged training, we train for 480 epochs instead of 960 to save time.
Most claims on the CIFAR100 dataset are also validated here: (1) “KD+CutMix” is better than KD, which is verified on all pairs.
(2) “KD+CutMixPick” is better than “KD+CutMix”, verified on 6/7 pairs.
The exception pair is resnet56/resnet20, where adding data picking decreases the accuracy slightly by 0.11%.
(3) When “KD+CutMixPick” is trained for prolonged iterations, the students perform best.
[Table] [Table]
We further evaluate our DA methods equipped with CRD [43], shown in the last two rows of Tab. 2.
Our “CutMixPick” method further advances the prior SOTA on 5 pairs.
When CRD+CutMixPick is trained for 480 epochs (instead of 240), further improvement can be observed on 6 of 7 pairs.
Apply DA to More KD Methods .
Notably, we achieve the above performance boosting simply using the original KD loss [16]
[Table]
with no bells and whistles .
This justifies one of our motivations in this paper,
i
[Table]
e. , existing KD methods [32, 29, 43] mainly improve KD at the network output side via better loss functions, while we propose to improve KD at the input side and show this path is just as promising.
Actually, this performance boosting effect is generic – we also applied CutMix to another 5 top-performing KD methods on CIFAR100: AT [50], CC [32], SP [44], PKT [30], and VID [1].
All the pairs see accuracy gains; half of them are even improved by more than 1% point.
Results on ImageNet100 and ImageNet .
These results are deferred to Appendix ?? .
In general, we observe that the correlation between T. stddev and S. test loss become weaker on ImageNet and ImageNet.
This is because these two datasets are inherently harder than CIFAR100 and Tiny
ImageNet.
Nevertheless, the p-value of the correlation is still below 5% on ImageNet100, i.e. , still statistically significant, suggesting our theory can generalize to large-resolution ( 224 × 224 ) datasets.
Conclusion
In this paper, we attempt to precisely answer what makes a good data augmentation in knowledge distillation.
By analyzing the generalization gap of the student under different sampling schemes, we reach the conclusion that a good data augmentation scheme should reduce the variance of the cross-entropy ( i.e. , the distilled risk) between the teacher and student.
Based on this, we propose a new metric, the stddev of the teacher
’s mean probability (T. stddev), as a feasible measure of the quality of data augmentation techniques.
Empirical studies with various teacher-student pairs confirm the efficacy of the proposed metric for data augmentation quality in KD.
Besides the theoretical understanding, we also develop an entropy-based data picking scheme to further enhance the prior best augmentation scheme (CutMix) in KD.
Finally, we show how we can obtain considerable KD performance gains simply by using a stronger DA guided by the proposed theory. Disclosure of Funding
This work originates from Huan’s internship at MERL, and is finished eventually after he returns to Northeastern University as a research assistant.
This work is thus fully supported by MERL and Northeastern University.
There is no third-party funding or support in any form.
There are no competing interests to disclose.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See our Appendix.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See our Appendix.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Sec. 3.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Sec. 3. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See our Appendix.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See our Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] All of our CIFAR100/Tiny ImageNet/ImageNet results are averaged by at least three random runs, mean and stddev reported.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See our Appendix. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] See our Appendix.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] We include the code link.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] The data we use are all publicly available.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes] The data we use contains no personally identifiable information or offensive content.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[No] No crowdsourcing in this work.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[No] No crowdsourcing in this work.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[No] No crowdsourcing in this work.
[-Ni =
-]