What Makes a “Good” Data Augmentation in Knowledge Distillation – A Statistical Perspective
Anonymous Author(s)
AffiliationAddress email
Abstract
Knowledge distillation (KD) is a general neural network training approach thatuses a teacher to guide a student.
Existing works mainly study KD from thenetwork output side ( e.g. , how to design a better KD loss function), while fewhave attempted to understand it from the input side.
Especially, its interplaywith data augmentation (DA) has not been well understood.
In this paper, weask: Why do some DA schemes ( e.g. , CutMix) inherently perform much betterthan others in KD?
What characterizes a “good” DA in KD?
Our investigationfrom a statistical perspective suggests that a good DA scheme should reduce thevariance of the teacher
’s mean probability, which will eventually lead to a lowergeneralization error gap for the student.
Besides the theoretical understanding, wealso propose a new entropy-based data-mixing DA scheme to enhance CutMix.
Extensive empirical studies support our claims and demonstrate how we can harvest considerable performance gains simply by using a better DA scheme in distillation.
Introduction
Deep neural networks (DNNs) are the top performing method in many artificial intelligence areas [15,27].
How to effectively train a deep network for classification has been a central topic for decades.
In the past several years, efforts have mainly focused on better architecture design ( e g. , batchnormalization [11], residual blocks [6], dense connections [10]) and better loss functions ( e g. , labelsmoothing [31, 20], contrastive loss [9], large-margin softmax [16]) than the standard cross-entropy(CE) loss.
Knowledge distillation (KD)
[8] is a training method that falls in the second group.
In
KD, a stronger network – called teacher – is introduced to guide the learning of the original network –called student – by minimizing the discrepancy between the representations of the two networks,
[Equation]
where D KL represents KL divergence [14]; α ∈ (0 , 1) is a factor to balance the two loss terms;
L CE denotes the cross-entropy loss; y is the one-hot label and p ( t ) , p ( s ) stand for the teacher’s andstudent’s output probabilities over the classes; τ is a temperature constant [8] to smooth predictedprobabilities.
KD allows us to train smaller, more efficient neural networks without compromising onaccuracy, which facilitates deploying deep learning in resource constrained environments ( e.g. , onmobile devices).
KD has found plenty of applications in many tasks [2, 35, 12, 36].
Most existing KD methods have attempted to improve it by proposing better KD loss functionsapplied at the network outputs [24, 21, 32].
Few works have considered KD from the input side.
Especially, the interplay between KD and data augmentation (DA) [29] has not been well understoodso far.
In this work, we ask: What defines a “better” data augmentation in knowledge distillation ?
Aclear answer to this question has many benefits.
First, theoretically, it can help us towards a betterunderstanding about how data augmentation plays a role in KD.
Second, practically, it can bring us considerable performance gain – in Fig.
1, we empirically show that a stronger DA can lowerthe test error rate and admit more training iterations without overfitting.
The figure shows test errorrates using standard cross entropy loss (no teacher) as well as using knowledge distillation loss(with a teacher).
It is pretty obvious to see that “Flip+Crop” is stronger than “Flip” alone in Fig.
However, for other DA schemes, such as Mixup [40] vs. AutoAugment [3], which one is stronger?
We desire a principled way (for example, a concrete metric) to make the vague concept “stronger”exact.
Presenting such a theoretically sound metric and empirically validating its effectiveness is thegoal of this paper.
[Figure]
Intuitively, a better DA should enrich the inputdata and expose more knowledge of the teacherso that the student can generalize better.
Weformalize this idea from a statistical learningperspective.
Specifically, we will show a good
DA scheme is defined by a lower variance (orcovariance) of the teacher
’s mean output probability over different input samples, which ultimately leads to a lower generalization gap forthe student.
The proposed theory is well justifiedby our extensive empirical studies on CIFAR100 and Tiny ImageNet datasets with variouspairs.
The proposed theory well explains why
CutMix is better than other alternatives (such as
Mixup [40], AutoAugment [3]) in KD.
Besides the new theoretical results, we also propose an entropy-based data picking scheme toselect more informative samples for KD, whichcan deliver even lower variance of the teacher’smean probability as well as lower generalizationerror of the student.
We make the followingcontributions in this paper:
• We present a proven proposition that precisely answers what defines a better DA in KD:
Given a fixed teacher, a better DA is the one that gives a lower variance of the teacher’smean output probability.•
The proposition is well justified empirically on standard image classification datasets with
many teacher-student pairs.• An entropy-based data picking scheme is introduced to further reduce the variance of theteacher’s mean probability, which can further advance CutMix, the prior state-of-the-art DAapproach among those evaluated in this paper.• Empirically, we show how the presented theory can benefit in practice –
we can simply
enhance the existing KD methods by using a stronger DA and prolonged training iterations.
Related Work
Knowledge Distillation .
The general idea of knowledge distillation is to guide the training of a studentmodel through a stronger pretrained teacher model (or an ensemble of models).
It was pioneeredby Buciluˇa et al.
[1] and later refined by Hinton et al.
[8], who coined the term.
Since its debut,knowledge distillation has seen extensive application in vision and language tasks [2, 35, 12, 36].
Many variants have been proposed regarding the central question in knowledge distillation, thatis, how to define the knowledge that is supposed to be transferred from the teacher to the student.
Examples of such knowledge definitions include feature distance [25], feature map attention [39],feature distribution [22], activation boundary [7], inter-sample distance relationship [21, 24, 17, 33],and mutual information [32].
Over the past several years, the progress has been made primarilyfor intermediate feature maps and network outputs ( i.e. , through a better loss function).
In contrast,our goal in this paper is to improve the KD performance at the input end with the help of dataaugmentation.
We will show this path is as effective and also has much potential for future research.
[Figure]
Data Augmentation .
Deep neural networks are prone to overfitting, i.e. , building input-targetmappings using undesirable or irrelevant features (like noise) in the data.
Data augmentation isa prevailing technique to curb overfitting [29].
In classification tasks, data augmentation aims toexplicitly provide data with label-invariant transformations (such as random crop, horizontal flip,color jittering, Cutout [5]) during training so that the model can learn representations robust to thosenuisance factors.
Recently, more advanced data augmentation methods were proposed, which notonly transform the input, but also transform the target.
For example, Mixup [40] linearly mixes twoimages with the labels mixed by the same linear interpolation.
Manifold Mixup [34] is similar to
Mixup but conducts the mix operation in the feature level instead of pixel level; CutMix [37] pastesa patch cut from an image onto another image with the label decided by the area ratio of the twoparts.
AutoAugment [3] is a strong DA method that finds an optimal augmentation policy froma large search space that minimizes a task loss.
When both the input and target are transformedsimultaneously, the key is to maintain a semantic correspondence between the new input and newtarget.
Although these methods have been proven effective, one lingering concern is the reasoningbehind them.
Specifically, it is easy to come up with examples where the semantic correspondence ispoorly kept (see our supplementary material for examples of CutMix on ImageNet).
Unlike thesemethods, which focus on general classification using the cross-entropy loss, our work investigatesthe interplay between data augmentation and knowledge distillation loss and the proposed new dataaugmentation is specifically for knowledge distillation.
One recent work [4] also conducts empirical study about the impact of data augmentation onknowledge distillation.
They first apply data augmentation ( e.g. , mixup/CutMix) to the teacher thenconduct the distillation step as usual (no extra data augmentation in this step); our investigation is theexact opposite to their setup: we train the teacher as usual (not applying mixup/CutMix), then in thedistillation step we employ a more advanced data augmentation ( e.g. , mixup/CutMix).
Interestingly,they conclude that the teacher trained with mixup/
CutMix hurts the student’s generalization ability,while we consistently see a student performance boost via a stronger DA in the KD step.
The work of [19] presents a statistical perspective to understand how the softened probabilities in KDis better than the one-hot hard labels.
Our work is inspired by their Bayes teacher notion.
This said,our work is different from theirs in that we focus on explaining how data augmentation plays a rolein KD and answer what characterizes a good DA, while they attempt to answer why the KD loss isbetter than the standard cross-entropy loss.
Theoretical Investigation
Prerequisites: Multi-Class Classification with KD
Given a training set S = { ( x n , y n ) } Nn =1 ∼ D N , where D is the joint distribution for input-outputrandom variable pair ( x, y ) , the goal in multi-class classification is to pin down a predictor f : X →
R C from hypothesis class H ( C : the number of classes), which is supposed to minimize the true risk
[Equation]
where L stands for the loss objective function ( e.g. , cross-entropy); the subscript D of R D is toemphasize the true risk is defined on the data distribution.
The true risk is approximated in practiceon a separate test set.
For training, the predictor aims to minimize the empirical risk defined on the training sequence S ,
[Equation]
where e y ∈ { 0 , 1 } C is a one-hot vector indicating the label y ∈
[ C ] = { 1 , 2 , · · · , C } .
The subscript
S of R S is to emphasize the empirical risk is defined on the finite sampled data points.
In the context of KD, the one-hot hard target vector
e y is replaced with a probability vector p ( t ) ( x ) ∈
R C + , giving us the empirical distilled risk of f
[Equation]
where the super-script t indicates the fixed teacher model.
The theory concerning why Eq.
(4) isbetter than Eq.
(3) has been established in [19].
Interested readers may refer to their paper for moredetails.
Next, we look into how the data augmentation plays a role in KD.
3.2 What Makes a “Good” DA in Distillation?
Intuitively, a better DA should provide more information, i.e. , expose more knowledge of the teacherso that the student can absorb more and thus generalizes better.
We make this idea rigorous as follows.
Proposition 1 .
Given a bounded loss function and a fixed teacher model with the empirical distilledrisk defined in Eq.
(4), for any predictor f , consider two sampled sequences S 1 ∈ D N and S 2 ∈ D N ,they are made up of elements sampled from the same distribution D , while not i.i.d.
(especially whenwe use data augmentation).
If the elements in S 1 present a larger correlation than those in S 2 , thenthe student’s generalization gap trained on S 1 will be greater than that trained on S 2 :
[Equation]
Proof .
Let ∆ =
ˆ R S ( f ) − R D ( f ) .
By the definition of variance, E S [∆ 2 ] =
Var S [∆] + (E S [∆]) 2 .
Tomake the notation clearer, we define q ( x i ) =
p ( t ) ( x i ) ⊤ log( f ( x i )) .(1)
Since R D ( f ) is a constant (albeit unknown),
[Equation]
where the second last equation above is due to the fact that each element x i in S is drawn the sameunderlying distribution D .
From the above derivation, we can clearly see that for S 1 , S 2 , E S [∆] isthe same.
(2) Then we consider Var S [∆] .
Again, since R D ( f ) is a constant,
[Equation]
where Cov[ · , · ] stands for covariance.
From the last item in Eq.
(7), we can see that the covariancepart is different for different sampled S .
If the samples in S 1 present a larger correlation than those in
S 2 , we will have Var S 1 [∆] >
Var S 2 [∆] , which further leads to increased generalization gap for thestudent, E S
[( R S 1 ( f ) − R D ( f )) 2 ] > E S 2 [( R S 2 ( f ) − R D ( f )) 2 ] .
□
Practical use: Std of the teacher’s mean probability .
Note in Proposition 1, the predictor f
( i.e. , thestudent model) can be any one (not necessarily a converged model).
Clearly, we want to characterizethe student model after KD training .
Then, for a well-trained student f with enough capacity (whichis the case in practice), f ( x ) (after softmax to make a valid probability distribution) is actually prettyclose to p ( t ) ( x ) .
Thus, we can further reduce the covariance term in Eq.
[Equation]
This formula implies an important fact that we only need the covariance of the teacher’s probabilityto measure the “goodness” of a certain DA technique, no need for the student.
This fact will beempirically justified in our experiments with different networks as the student (see Tabs.
1 and 2).
Based on Eq.
(8), when using S 1 vs. S 2 as training sequence, the fundamental factor answeringfor the variance gap of Var S [∆] boils down to the covariance in { p ( t ) ( x i ) } Ni =1 .
That is, a largercovariance in { p ( t ) ( x i ) } N i =1 leads to a larger covariance in { p ( t ) ( x i )
⊤ log( p ( t ) ( x i ))
} N i
=1 in Eq.
(8)(for a converged student in consideration), which ultimately leads to a higher Var S [∆] in Eq.
The next step is to find a feasible way to estimate the covariance in { p ( t ) ( x i ) } Ni =1 .
Consider theaverage variable (denoted as u here) of several random variables { p ( t ) ( x k ) }
Kk =1 ,
[Equation]
Its variance inherently takes into account the covariance among its addends.
Therefore, we can usethe variance (or equivalently, its std) of u as a proxy for the covariance in { p ( t ) ( x k ) } Kk =1 :
[Equation]
where K is a pre-defined number of samples of set S ∗ to realize the averaging effect ( K = 640 in ourexperiments 1 ).
Note, u can be interpreted as the teacher
’s mean probability over K input samples.
If there is a large covariance among { p ( t ) ( x k ) } Kk =1 , m will be large (in an element-wise sense).
Thenthe average of m over classes (we do this averaging simply because we desire a scalar metric) is agood indicator to capture such covariance.
We thus formally introduce ¯ m , the (averaged) std of theteacher
’s
mean probability, as the proposed metric to measure the quality of a data augmentationscheme .
A lower ¯ m implies a better DA by our definition.
In the experiments, we will show thismetric defined merely using the teacher can accurately characterize the generalization error of thestudents after distillation.
Evaluated Algorithms
4.1 Extend Existing DA Approaches for KD
Given an existing DA approach, this section explains how we adapt it properly to the case of KD.
Specifically, let x 0 denote the raw data, x denote the transformed data by the standard augmentation(random crop and flip).
Illustrated in Fig.
2(b), we will add the DA following x to obtain x ′ .
Unlikethe common data augmentation where only the transformed input is fed into the network, we keepboth the input x and x ′ for the training (as such, the number of input examples is increased, typicallydoubled).
The consideration of keeping both inputs is to maintain the information path for the originalinput x so that we can easily see how the added information path of x ′ leads to a difference.
For x , its loss is still the original KD loss, consisting of the cross-entropy loss and the KL divergence(Eq.
Of special note is that, for x ′ , its loss is only the KL divergence, i.e. , we do not use thelabels assigned by the DA algorithm (for example, in the original Mixup and CutMix, they assign alinearly interpolated label to the augmented sample) because these labels can actually be wrong (seeour supplementary material for concrete examples on ImageNet).
Among all the data augmentation techniques evaluated in this paper, we will show CutMix works best .
The fundamental reason for its success, as we will show, is that it achieves a much lower variance ofthe teacher’s mean probability, implying it produces more diverse data than its counterparts.
4.2 CutMixPick: Enhance CutMix with Entropy-based Data Picking
In this section, we propose a data picking scheme to further reduce the variance of the teacher’s meanprobability.
The idea is partly inspired by active learning [28].
In active learning, the learner enjoysthe freedom to query the data instances to be labeled for training by an oracle ( i.e. , the teacher in ourcase) [28].
Since the augmented data can vary in their quality, we can introduce certain criterion topick the more valuable data for the student.
Intuitively, we regard a sample with more information is of higher quality.
Therefore, we take
Shannon’s entropy of the teacher’s output probability as a natural measure to select samples,
[Equation]
Note this formula is exactly in the same form of Eq.
(8) where a converged student is considered.
Empirically, we will also show the data selected by this formula indeed results in lower ¯ m and bettertest error of the student.
Concretely, given a batch of data, we first apply CutMix to obtain a bunch of augmented samples.
Then sort all the augmented samples by Eq.
(11) in ascending order and keep the top r (a pre-definedpercentage constant, r = 0 .
5 in our experiments) samples.
This simple technique can be rather effective according to our empirical study (see Tabs.
1 and 2).
Another seemingly potential alternative is to use the student’s entropy as the picking metric.
Theintuition behind this is that a high-entropy sample in the view of the student can be regarded as a hardexample.
Learning on these hard examples may expand the student’s knowledge by reducing its blindspots.
Despite this intuitively plausible explanation, in practice, we will show this scheme actuallyunder-performs the former student-agnostic one in Eq.
[Table]
Experimental Results
Datasets and Networks .
We evaluate our method on the CIFAR-100 [13] and Tiny ImageNet 2datasets.
CIFAR-100 has 100 object classes (32 × 32 RGB images).
Each class has 500 images fortraining and 100 images for testing.
Tiny ImageNet is a small version of ImageNet with 200 classes(64 × 64 RGB images).
Each class has 500 images for training, 50 for validation and 50 for testing.
To thoroughly evaluate our methods, we benchmark them on various standard network architectures:
VGG [30], ResNet [6], WRN (Wide-ResNet) [38], MobileNetV2 [26], ShuffleNetV2 [18].
Benchmark Methods .
In addition to the standard cross-entropy training and the original KDmethod [8], we also compare with the state-of-the-art distillation approach Contrastive Representation
Distillation (CRD) [32].
It is important to note that our method focuses on improving KD by usingbetter inputs , while CRD improves KD at the output end ( i.e. , a better loss function).
Therefore, theyare orthogonal and we will show they can be combined together to deliver even better results.
Hyper-parameter Settings .
The temperature τ of knowledge distillation is set to 4.
Loss weightα =
9 (Eq.
For CIFAR-100 and Tiny ImageNet, training batch size is 64; the original numberof total training epochs is 240, with learning rate (LR) decayed at epoch 150, 180, and 210 bymultiplier 0.1.
The initial LR is 0.05.
All these settings are the same as CRD [32] for fair comparison.
Note, in our experiments we will show the results of more training iterations.
If the number of totalepochs is scaled by a factor k , the epochs after which learning rate is decayed is also be scaled by k .
For example, if we train a network for CIFAR-100 for 480 epochs ( k = 2 ) in total, the epochs afterwhich the learning rate is decayed will be 300, 360, and 420.
We use PyTorch [23] to conduct all our experiments.
For CIFAR-100, we adopt the pretrained teachermodels from CRD 3 for fair comparison with it.
For Tiny ImageNet, we train our own teacher models.
Our code and trained models will be made publicly available .
Data Augmentation Schemes .
We investigate the following popular DA schemes:
• Identity : This augmentation scheme simply makes a copy of each batch data during training, whichshould be the lower bound of all DA schemes discussed
here
since it adds no new information.
• Flip : Horizontal flip.• Flip+Crop : Horizontal flip and random crop.
This is the standard DA extensively used in 2Dimage recognition task (such as on CIFAR and ImageNet datasets).• Cutout [5]: Cutout occludes a small random patch
of an image
• AutoAugment [3]: AutoAugment is an ensemble of a collected DA schemes.
The DA policy isautomatically selected by reinforcement learning instead of manually.
• Mixup [40]: Mixup applies linear interpolation between two inputs and applies the same linear
interpolation to their labels to make the new label for the augmented sample.
• CutMix [37]: CutMix cuts a small patch from one source image and pastes it to another sourceimage, the resulted image considered as a new input.
The target for the new input is a linearinterpolation from the two source labels.
5.1 Empirical Verification of Our Theory
Before presenting results, one thing worth mention is that, in this section we do not use accuracyas the measure of student’s performance in KD, because (1) all the formulas in Sec.
3 are derivedusing numerical error instead of accuracy; (2) more importantly, it is well-known that accuracy canmismatch with error – a model may achieve a better accuracy, meanwhile higher error too [19], whichwe also observed many times in our experiments.
In Tab. 1 (CIFAR-100) and Tab. 2 (Tiny ImageNet), we present the std of the teacher’s meanprobability (Eq.
and the student’s test error after distillation.
We have the following observations.(1)
In Tab. 1, foremost, we see in terms of the std of the teacher’s mean probability, Identity < Flip< Flip+Crop < Cutout <
AutoAugment < Mixup < CutMix.
These inequalities are well-alignedwith our intuition.
For example, AutoAugment [3] includes Cutout [5] in its transformation pool,thus should be stronger than Cutout, which is consistently reflected by the std of the teacher’s meanprobability on different pairs.(2) Per our theory, lower std of the teacher
’s mean probability should lead to better generalizationrisk for the student.
This is also generally well verified in Tab.
We see only six counterexamples(highlighted by red arrow in Tab. 1).
Possible reasons are: 1)
The test error is obtained on the testset with finite samples, which is only an approximation of the true error defined on distribution; 2)
The teacher’s mean probability is also evaluated on finite data.
Despite them, the general picturefrom Tab. 1 still confirms the positive correlation between the std of teacher
’s mean probabilityand student’s test error.
The correlation is actually very significant as the p-values indicate (see oursupplementary material for detailed p-values).
All the results are averaged by 3 random runs .
The up (or down) arrow indicates the result is higher (or lower) than the one right above it .
If “Test err.”
change is at a different direction from the “Mean prob.
std” change, we highlight the arrow in red
[Table]
The coherence between std of the teacher’smean probability and student’s test error is robust.
We also see it on the more challenging
Tiny ImageNet, as shown in Tab. 2.
Of particular note in Tab. 2 is that, Mixup is shown weakerthan AutoAugment here (in the sense of a higher“Std of mean prob.”), contrary to the CIFAR-100case (see Tab. 1, VGG13 columns) where Mixupis stronger.
Interestingly, the student’s test errorsalso well resonate with this change: The test error of VGG8 and MobileNetV2 using Mixupis higher than that using AutoAugment.
Thisdemonstrates the efficacy of the proposed metricin Eq.
(10) underpinned by Proposition 1.(4)
Importantly, note we only need the teacherto define the quality of a certain DA in KD, noneed for the student .
This is verified in Tabs.
1and 2, as when different students are used, thecoherence between std of the teacher’s meanprobability and student’s test error is steadilymaintained.
Notably, this is a great advantagein practice since we can decide which DA should be used for the best performance simply usingformulas (Eq.
with a few network forwards, without having to fully train the students.
5.2 Boosting KD with Stronger DA
In this section, we present more results to show how the proposed theory can benefit in practice – wecan harvest considerable performance gain simply by using a stronger DA in KD.
Prolonged training .
Notably, a stronger DA produces more diverse data, implying more information.
Intuitively, it should take a student model more training iterations (if the batch size does not change)
[Table] [Table]
[Table]
to fully absorb the excessive information.
That is, a stronger DA may take more training iterations to
[Table]
fully exhibit its potential.
This intuition is confirmed in Fig.
1 (and also two more pairs WRN-402/WRN-16-2 and VGG13/VGG8 in the supplementary material).
Thus, in our experiments, we willalso report results with prolonged training iterations for maximized performance.
Results on CIFAR-100 .
The results on CIFAR-100 dataset are shown in Tab.
Comparing the row “KD+CutMix” to “KD", we see the proposed CutMix scheme improves theaccuracies of 6/7 teacher-student pairs.
On the pair ResNet32x4/ShuffleNetV2, the improvementis very significant (more than 1 percentage point).
(2) Comparing the row “KD+CutMixPick” to“KD+CutMix”, we see 6/7 pairs are improved further, showing the proposed data picking schemeworks in most cases.
(3) Finally, “KD+CutMixPick” scheme can be combined with more training iterations, which delivers even higher accuracies.
If comparing our best results (KD+CutMixPick 960 )with those of CRD (though this is not an apples-to-apples comparison since the two methods focuson different aspects to improve KD), we can see our approach outperforms CRD on 6/7 pairs.
In the last two rows of Tab. 3, when CRD [32], one of the SOTA KD algorithms, is armed with ourproposed “CutMixPick” and more training iterations, its results can be further advanced consistently .
This demonstrates that the proposed schemes are general and can readily work with those methodsfocusing on better KD loss functions.
Results on Tiny ImageNet .
We also evaluate CutMix and CutMixPick on a more challenging dataset,
Tiny ImageNet.
Similar to the case on CIFAR-100, we have results on different teacher-student pairs,shown in Tab. 4.
For prolonged training, we train for 480 epochs instead of 960 to save time.
Mostclaims on the CIFAR-100 dataset are also validated here: (1) “KD+CutMix” is better than KD, whichis verified on all pairs.
(2) “KD+CutMixPick” is better than “KD+CutMix”, verified on 6 pairs.
Theexception pair is ResNet56/ResNet20, where adding data picking decreases the accuracy slightly by0.11%.
(3) When “KD+CutMixPick” is trained for prolonged iterations, the students achieve the bestperformances.
We further evaluate our DA methods equipped with CRD [32], shown in the last two rows of Tab. 4.
Our “CutMixPick” method further advances the prior SOTA on 5 pairs.
When CRD+CutMixPick istrained for 480 epochs (instead of 240), further improvement can be observed on 6 of 7 pairs.
The performance boosting is simple and generic .
Notably, we achieve the above performanceboosting simply using the original KD loss [8], with no bells and whistles .
This justifies one of ourmotivations in this paper, i.e. , existing KD methods [24, 21, 32] mainly improve KD at the networkoutput side via better loss functions, while we propose to improve KD at the input end and show thispath is just as promising .
Actually, this performance boosting effect is generic , not limited to CRD.
In the supplementary material, we present more results of applying CutMix to another 5 KD methods.
All of the evaluated pairs see accuracy gains; half of them are even improved by more than 1% point.
Conclusion
In this paper, we attempt to precisely answer what makes a good data augmentation in knowledgedistillation.
By analyzing the generalization gap of the student under different sampling schemes,we reach the conclusion that a good data augmentation scheme should reduce the variance of theempirical distilled risk between the teacher and student.
Based on this, we propose a new metric,std of the teacher
’s mean probability, as a feasible measure of the quality of data augmentationtechniques.
Empirical studies with various teacher-student pairs confirm the efficacy of the proposedmetric for data augmentation quality in KD.
Besides the theoretical understanding, we also develop anentropy-based data picking scheme to further enhance the prior best augmentation scheme (CutMix)in KD.
Finally, we show how we can obtain considerable KD performance gains simply by using astronger DA guided by the proposed theory.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See supplementary material.(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] Seesupplementary material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Sec. 3.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Sec. 3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See supplemen-
tary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See supplementary material.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See Tab. 3, Tab. 4, and Fig.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See supplementary material.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] See supplementary material.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] The data we use are all publicly available.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes] The data we use contains no personally
identifiable information or offensive content.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[No] No crowdsourcing in this work.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[No] No crowdsourcing in this work.
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[No] No crowdsourcing in this work.