T RADEOFFS IN D ATA A UGMENTATION : A N E MPIRICAL S TUDY
Anonymous authors
Paper under double-blind review
A BSTRACT
Though data augmentation has become a standard component of deep neural network training, the underlying mechanism behind the effectiveness of these techniques remains poorly understood.
In practice, augmentation policies are often chosen using heuristics of distribution shift or augmentation diversity.
Inspired by these, we conduct an empirical study to quantify how data augmentation improves model generalization.
We introduce two interpretable and easy-to-compute measures: Afﬁnity and Diversity.
We ﬁnd that augmentation performance is predicted not by either of these alone but by jointly optimizing the two.
[Figure]
[Table]
[Figure]
I NTRODUCTION
Models that achieve state-of-the-art in image classiﬁcation often use heavy data augmentation strategies.
The best techniques use various transforms applied sequentially and stochastically.
Though the effectiveness of this is well-established, the mechanism through which these transformations work is not well-understood.
Since early uses of data augmentation, it has been assumed that augmentation works because it simulates realistic samples from the true data distribution: “[augmentation strategies are] reasonable since the transformed reference data is now extremely close to the original data.
In this way, the amount of training data is effectively increased" (Bellegarda et al., 1992).
Because of this, augmentations have often been designed with the heuristic of incurring minimal distribution shift from the training data.
This rationale does not explain why unrealistic distortions such as cutout (DeVries & Taylor, 2017),
SpecAugment (Park et al., 2019), and mixup (Zhang et al., 2017) signiﬁcantly improve generalization performance.
Furthermore, methods do not always transfer across datasets— Cutout , for example, is useful on CIFAR-10 and not on ImageNet (Lopes et al., 2019).
Additionally, many augmentation policies heavily modify images by stochastically applying multiple transforms to a single image.
Based on this observation, some have proposed that augmentation strategies are effective because they increase the diversity of images seen by the model.
In this complex landscape, claims about diversity and distributional similarity remain unveriﬁed heuristics.
Without more precise data augmentation science, ﬁnding state-of-the-art strategies requires brute force that can cost thousands of GPU hours (Cubuk et al., 2018; Zhang et al., 2019).
This highlights a need to specify and measure the relationship between the original training data and the augmented dataset, as relevant to a given model’s performance.
In this paper, we quantify these heuristics.
Seeking to understand the mechanisms of augmentation, we focus on single transforms as a foundation.
We present an empirical study of 204 different augmentations on CIFAR-10 and 225 on ImageNet, varying both broad transform families and ﬁner transform parameters.
To better understand current state of the art augmentation policies, we additionally measure 58 composite augmentations on ImageNet and three state of the art augmentations on CIFAR-10.
Our contributions are:
We introduce Afﬁnity and Diversity: interpretable, easy-to-compute metrics for parametrizing augmentation performance.
Afﬁnity quantiﬁes how much an augmentation shifts the training data distribution.
Diversity quantiﬁes the complexity of the augmented data with respect to the model and learning procedure.
We ﬁnd that performance is dependent on both metrics.
In the Afﬁnity-Diversity plane, the best augmentation strategies jointly optimize the two (see Fig 1).
We connect augmentation to other familiar forms of regularization, such as ` 2 and learning rate scheduling, observing common features of the dynamics: performance can be improved and training accelerated by turning off regularization at an appropriate time.
We ﬁnd that performance is only improved when a transform increases the total number of unique training examples.
The utility of these new training examples is informed by the augmentation’s Afﬁnity and Diversity.
R ELATED W ORK
Since early uses of data augmentation in training neural networks, there has been an assumption that effective transforms for data augmentation are those that produce images from an “overlapping but different" distribution (Bengio et al., 2011; Bellegarda et al., 1992).
Indeed, elastic distortions as well as distortions in the scale, position, and orientation of training images have been used on
MNIST (Ciregan et al., 2012; Sato et al., 2015; Simard et al., 2003; Wan et al., 2013), while horizontal ﬂips, random crops, and random distortions to color channels have been used on CIFAR-10 and
ImageNet (Krizhevsky et al., 2012; Zagoruyko & Komodakis, 2016; Zoph et al., 2017).
For object detection and image segmentation, one can also use object-centric cropping (Liu et al., 2016) or cut-and-paste new objects (Dwibedi et al., 2017;
Fang et al., 2019; Ngiam et al., 2019).
In contrast, researchers have also successfully used less domain-speciﬁc transformations, such as
Gaussian noise (Ford et al., 2019; Lopes et al., 2019), input dropout (Srivastava et al., 2014), erasing random patches of the training samples (DeVries & Taylor, 2017; Park et al., 2019; Zhong et al., 2017), and adversarial noise (Szegedy et al., 2013).
Mixup (Zhang et al., 2017) and Sample Pairing (Inoue, 2018) are two augmentation methods that use convex combinations of training samples.
It is also possible to improve generalization by combining individual transformations.
For example, reinforcement learning has been used to choose more optimal combinations of data augmentation transformations (Ratner et al., 2017; Cubuk et al., 2018).
Follow-up research has lowered the computation cost of such optimization, by using population based training (Ho et al., 2019), density matching (Lim et al., 2019), adversarial policy-design that evolves throughout training (Zhang et al., 2019), or a reduced search space (Cubuk et al., 2019).
Despite producing unrealistic outputs, such combinations of augmentations can be highly effective in different tasks (Berthelot et al., 2019; Tan & Le, 2019; Tan et al., 2019; Xie et al., 2019a;b).
Across these different examples, the role of distribution shift in training remains unclear.
Lim et al.
; Hataya et al.
(2020) have found augmentation policies by minimizing the distance between the distributions of augmented data and clean data.
Recent work found that after training with augmented data, ﬁne-tuning on clean training data can be beneﬁcial (He et al., 2019), while Touvron et al.
(2019) found it beneﬁcial to ﬁne-tune with a test-set resolution that aligns with the training-set resolution.
The true input-space distribution from which a training dataset is drawn remains elusive.
To better understand the effect of distribution shift on performance, many works attempt to estimate it.
Often these techniques require training secondary models, such as those based on variational methods (Goodfellow et al., 2014; Kingma & Welling, 2014;
Nowozin et al., 2016; Blei et al., 2017).
Others have augmented the training set by modelling the data distribution directly (Tran et al., 2017).
Recent work has suggested that even unrealistic distribution modelling can be beneﬁcial (Dai et al., 2017).
These methods try to specify the distribution separately from the model they are trying to optimize.
As a result, they are insensitive to any interaction between the model and data distribution.
Instead, we are interested in a measure of how much the data shifts along directions that are most relevant to the model’s performance.
M ETHODS
We performed extensive experiments with various augmentations on CIFAR-10 and ImageNet.
Experiments on CIFAR-10 used the WRN-28-2 model (Zagoruyko & Komodakis, 2016), trained for 78k steps with cosine learning rate decay.
Results are the mean over 10 initializations and reported errors (often too small to show on ﬁgures) are the standard error on the mean.
Details on the error analysis are in Sec.
C.
Experiments on ImageNet used the ResNet-50 model (He et al., 2016), trained for 112.6k steps with a weight decay rate of 1e-4, and a learning rate of 0.2, which is decayed by 10 at epochs 30, 60, and 80.
Images were pre-processed by dividing each pixel value by 255 and normalizing by the dataset statistics.
Random crop was also applied on all ImageNet models.
These pre-processed data without further augmentation are “clean data” and a model trained on it is the “clean baseline”.
We followed the same implementation details as Cubuk et al.
(2018) 1 , including for most augmentation operations.
Further implementation details are in Sec.
A.
Unless speciﬁed otherwise, data augmentation was applied following standard practice: each time an image is drawn, the given augmentation is applied with a given probability.
We call this mode dynamic augmentation.
Due to whatever stochasticity is in the transform itself (such as randomly selecting the location for a crop) or in the policy (such as applying a ﬂip only with 50% probability), the augmented image could be different each time.
Thus, most of the tested augmentations increase the number of possible distinct images that can be shown during training.
We also performed select experiments by training with static augmentation.
In static augmentation, the augmentation policy (one or more transforms) is applied once to the entire clean training set.
Static augmentation does not change the number of unique images in the dataset.
3. A FFINITY : A SIMPLE METRIC FOR DISTRIBUTION SHIFT
Thus far, heuristics of distribution shift have motivated design of augmentation policies.
Inspired by this focus, we introduce a simple metric to quantify how augmentation shifts data with respect to the decision boundary of the clean baseline model
We start by noting that a trained model is often sensitive to the distribution of the training data.
That is, model performance varies greatly between new samples from the true data distribution and samples from a shifted distribution.
Importantly, the model’s sensitivity to distribution shift is not purely a function of the input data distribution, since training dynamics and the model’s implicit biases affect performance.
Because the goal of augmentation is improving model performance, measuring shifts with respect to the distribution captured by the model is more meaningful than measuring shifts in the distribution of the input data alone.
We thus deﬁne Afﬁnity to be the difference between the validation accuracy of a model trained on clean data and tested on clean data, and the accuracy of the same model tested on an augmented validation set.
Here, the augmentation is applied to the validation dataset in one pass, as a static augmentation.
More formally we deﬁne:
Deﬁnition 1.
Let D train and D val be training and validation datasets drawn IID from the same clean data distribution, and let D 0 val be derived from D val by applying a stochastic augmentation strategy, a , once to each image in D val , D 0 val =
{ ( a ( x ) , y ) : 8 ( x, y ) 2 D val } .
Further let m be a model trained on D train and A ( m, D ) denote the model’s accuracy when evaluated on dataset D .
The Afﬁnity,
T [ a ; m ; D val ] , is given by
[Equation]
With this deﬁnition, Afﬁnity of zero represents no shift and a negative number suggests that the augmented data is out-of-distribution for the model.
In Fig.
2 we illustrate Afﬁnity with a two-class classiﬁcation task on a mixture of two Gaussians.
Augmentation in this example comprises shift of the means of the Gaussians of the validation data compared to those used for training.
Under this shift, we calculate both Afﬁnity and KL divergence of the shifted data with respect to the original data.
Afﬁnity changes only when the shift in the data is with respect to the model’s decision boundary, whereas the KL divergence changes even when data is shifted in the direction that is irrelevant to the classiﬁcation task.
In this way, Afﬁnity captures what is relevant to a model: shifts that impact predictions.
This same metric has been used as a measure of a model’s robustness to image corruptions that do not change images’ semantic content (Azulay & Weiss, 2018; Dodge & Karam, 2017; Ford et al., 2019; Hendrycks & Dietterich, 2019; Rosenfeld et al., 2018; Yin et al., 2019).
Here we, turn this around and use it to quantify the shift of augmented data compared to clean data.
Afﬁnity has the following advantages as a metric:
It is easy to measure.
It requires only clean training of the model in question.
It is independent of any confounding interaction between the data augmentation and the training process, since augmentation is only used on the validation set and applied statically.
It is a measure of distance sensitive to properties of both the data distribution and the model.
We gain conﬁdence in this metric by comparing it to other potential model-dependent measures of distribution shift.
We consider the mean log likelihood of augmented test images(Grathwohl et al., 2019), and the Watanabe–Akaike information criterion (WAIC) (Watanabe, 2010).
These other metrics have high correlation with Afﬁnity.
Details can be found in Sec.
F.
3. D IVERSITY : A MEASURE OF AUGMENTATION COMPLEXITY
Inspired by the observation that multi-factor augmentation policies such as FlipLR + Crop + Cutout and RandAugment (Cubuk et al., 2019) greatly improve performance, we propose another axis on which to view augmentation policies, which we dub Diversity .
This measure is intended to quantify the intuition that augmentations prevent models from over-ﬁtting by increasing the number of samples in the training set; the importance of this is shown in Sec. 4.3.
Based on the intuition that more diverse data should be more difﬁcult for a model to ﬁt, we propose a model-based measure.
The Diversity metric in this paper is the ﬁnal training loss of a model trained with a given augmentation:
Deﬁnition 2.
Let a be an augmentation and D 0 train be the augmented training data resulting from applying the augmentation, a , stochastically.
Further, let L train be the training loss for a model, m , trained on D 0 train .
We deﬁne the Diversity, D [ a ; m ; D train ] , as
[Equation]
As with Afﬁnity, this deﬁnition of Diversity has the advantage that it can capture model-dependent elements, i.e. it is informed by the class of priors implicit in choosing a model and optimization scheme as well as by the stopping criterion used in training. One shortcoming of the above measure is the computational cost of evaluation. For each augmentation strategy of interest one must train an independent model and computing the training loss is as computationally costly as determining ﬁnal test accuracy.
For some purposes this cost may not be a problem.
For instance, many successful augmentation strategies are usable across a wide range of models.
This re-usability helps differ the initial search cost. Here we focus on this metric as a tool for understanding. Nonetheless, with computational cost in mind, in Sec. E we consider two alternative measures of diversity that are substantially less costly to evaluate.
The ﬁrst of these is again using the training loss but at an earlier point during training.
We ﬁnd that even after ten epochs of training the early time loss correlates well with its value at the end of training.
Another potential diversity measure is the entropy of the transformed data, D Ent .
This is inspired by the intuition that augmentations with more degrees of freedom perform better.
For discrete transformations, we consider the conditional entropy of the augmented data.
[Equation]
Here x 2 X is a clean training image and x 0 2 X 0 is an augmented image.
This measure has the property that it can be evaluated without any training or reference to model architecture.
However, the appropriate entropy for continuously-varying transforms is less straightforward.
Lastly, in Sec.
E we consider a fourth proxy for Diversity, the training time needed for a model to reach a given training accuracy threshold.
In Sec.
E, we show that these metrics correlate well with each other.
In the remaining sections we describe how the complementary metrics of Diversity and Afﬁnity can be used to characterize and understand augmentation performance.
R ESULTS 4. A UGMENTATION PERFORMANCE IS DETERMINED BY BOTH A FFINITY AND D IVERSITY
Despite the original inspiration to mimic realistic transformations and minimize distribution shift, many state-of-the-art augmentations yield unrealistic images.
This suggests that distribution shift alone does not fully describe or predict augmentation performance.
Figure 3(a) (left) measures Afﬁnity across 204 different augmentations for CIFAR-10 andfor ImageNet respectively.
We ﬁnd that for the most important augmentations—those that help performance—Afﬁnity is a poor predictor of accuracy.
Furthermore, we ﬁnd many successful augmentations with low Afﬁnity.
For example, Rotate(fixed, 45deg, 50%) , Cutout(16) , and combinations of FlipLR , Crop(32) , and Cutout(16) all have Afﬁnity <  15% and test accuracy > 2% above clean baseline on CIFAR-10.
Augmentation details are in Sec.
B.
As Afﬁnity does not fully characterize the performance of an augmentation, we seek another metric.
To assess the importance of an augmentation’s complexity, we measure Diversity across the same set of augmentations.
We ﬁnd that Diversity is complementary in explaining how augmentations can increase test performance.
As shown in Fig. 3(b) and (c), Afﬁnity and Diversity together provide a much clearer parameterization of an augmentation policy’s beneﬁt to performance.
For a ﬁxed level of Diversity, augmentations with higher Afﬁnity are consistently better.
Similarly, for a ﬁxed Afﬁnity, it is generally better to have higher Diversity.
To make this more quantitative we consider pairs of augmentation strategies ( a, a 0 ) and measure the fraction of pairs satisfying { ( a, a 0 ) :
A ( a ) >
A ( a 0 ) and T ( a ) > T ( a 0 ) or D ( a ) > D ( a 0 ) } (3)
[Figure]
A simple case study is presented in Fig.
The probability of the transform Rotate(fixed, 60deg) is varied.
The accuracy and Afﬁnity are not monotonically related, with the peak accuracy falling at an intermediate value of Afﬁnity.
Similarly, accuracy is correlated with Diversity for low probability transformations but does not track for higher probabilities.
The optimal probability for
Rotate(fixed, 60deg) lies at an intermediate value of Afﬁnity and Diversity.
To situate the tested augmentations—mostly single transforms—within the context of the state-of-theart, we tested three high-performance augmentations from literature: mixup (Zhang et al., 2017),
AutoAugment (Cubuk et al., 2018), and RandAugment (Cubuk et al., 2019).
These are highlighted with a star marker in Fig. 3(b).
More than either of the metrics alone, Afﬁnity and Diversity together provide a useful parameterization of an augmentation’s performance.
We now turn to investigating the utility of this tool for explaining other observed phenomena of data augmentations.
4. T URNING AUGMENTATIONS OFF MAY ADJUST A FFINITY , D IVERSITY , AND PERFORMANCE
The term “regularizer” is ill-deﬁned in the literature, often referring to any technique used to reduce generalization error without necessarily reducing training error (Goodfellow et al., 2016).
With this deﬁnition, it is widely acknowledged that commonly-used augmentations act as regularizers (Hernández-García & König, 2018; Zhang et al., 2016; Dao et al., 2019).
Though this is a broad deﬁnition, we notice another commonality across seemingly different kinds of regularizers: various regularization techniques yield boosts in performance (or at least no degradation) if the regularization is turned off at the right time during training.
For instance:
Decaying a large learning rate on an appropriate schedule can be better than maintaining a large learning rate throughout training (Zagoruyko & Komodakis, 2016).
Turning off ` 2 at the right time does not hurt performance (Golatkar et al., 2019).
Relaxing model constraints mid-training can boost performance (d’Ascoli et al., 2019).
Fine-tuning on clean data can improve ﬁnal accuracy (He et al., 2019).
To further study augmentation as a regularizer, we compare the constant augmentation case (with the same augmentation throughout) to the case where the augmentation is turned off partway through training and training is completed with clean data.
For each transform, we test over a range of switchoff points and select the one that yields the best ﬁnal validation or test accuracy on CIFAR-10 and
ImageNet respectively.
The Switch-off Lift is the resulting increase in ﬁnal test accuracy, compared to training with augmented data the entire time.
[Figure]
For some poor-performing augmentations, this gain can actually bring the test accuracy above the baseline, as shown in Fig. 5(b).
We additionally observe (Fig. 5(a)) that this test accuracy improvement can happen quite rapidly for both augmentations and for the other regularizers tested.
This suggests an opportunity to accelerate training without hurting performance by appropriately switching off regularization.
We call this a slingshot effect.
Interestingly, we ﬁnd the best time for turning off an augmentation is not always close to the end of training, contrary to what is shown in He et al.
For example, without switching,
FlipUD(100%) decreases test accuracy by almost 50% compared to clean baseline.
When the augmentation is used for only the ﬁrst third of training, ﬁnal test accuracy is above the baseline.
He et al.
(2019) hypothesized that the gain from turning augmentation off is due to recovery from a distribution shift.
Indeed, for many detrimental transformations, the test accuracy gained by turning off the augmentation merely recovers the clean baseline performance.
A few of the tested augmentations, such as FlipLR(100%) , are fully deterministic.
Thus, each time an image is drawn in training, it is augmented the same way.
When such an augmentation is turned off partway through training, the model then sees images—the clean ones—that are now new.
Indeed, when FlipLR(100%) is switched off at the right time, its ﬁnal test accuracy exceeds that of FlipLR(50%) without switching.
In this way, switching augmentation off may adjust for not only low Afﬁnity but also low Diversity.
4. I NCREASED EFFECTIVE TRAINING SET SIZE IS CRUCIAL FOR DATA AUGMENTATION
Most augmentations we tested and those used in practice have inherent stochasticity and thus may alter a given training image differently each time the image is drawn.
In the typical dynamic training mode, these augmentations increase the number of unique inputs seen across training epochs.
To further study how augmentations act as regularizers, we seek to discriminate this increase in effective dataset size from other effects.
We train models with static augmentation, as described in Sec. 3.
This altered training set is used without further modiﬁcation during training so that the number of unique training inputs is the same between the augmented and the clean training settings.
For almost all tested augmentations, using static augmentation yields lower test accuracy than the clean baseline.
Where static augmentation shows a gain (versions of crop ), the difference is less than the standard error on the mean.
As in the dynamic case, poorer performance in the static case is for transforms that have lower Afﬁnity and lower Diversity.
Static augmentations also always perform worse than their non-deterministic, dynamic counterparts, as shown in Fig.
This may be because the Diversity of a static augmentation is always less than the dynamic case (see also Sec. E).
The decrease in Diversity in the static case suggests a connection between Diversity and the number of training examples.
Together, these results point to the following conclusion: Increased effective training set size is crucial to the performance beneﬁt of data augmentation.
An augmentation’s Afﬁnity and Diversity inform how useful the additional training examples are.
D ISCUSSION
In this work, we conducted an empirical study in an attempt to characterize the essential factors leading to successful augmentation policies.
This builds a foundation for using these metrics to quantify and design more complex and powerful combinations of augmentations.
We further hope that Afﬁnity and Diversity may serve as useful metrics of distribution shift more generally.
Though earlier work has often explicitly focused on just one of these metrics, chosen priors have implicitly ensured reasonable values for both.
One way to achieve Diversity is to use combinations of many single augmentations, as in AutoAugment (Cubuk et al., 2018).
Because transforms and hyperparameters in Cubuk et al.
(2018) were chosen by optimizing performance on proxy tasks, the optimal policies include high and low Afﬁnity transforms.
Fast AutoAugment (Lim et al., 2019),
CTAugment (Berthelot et al., 2019; Sohn et al., 2020), and differentiable RandAugment (Cubuk et al., 2019) all aim to increase Afﬁnity by what Lim et al.
(2019) called “density-matching”.
However these methods use the search space of AutoAugment and thus inherit its Diversity.
On the other hand, Adversarial AutoAugment (Zhang et al., 2019) focused on increasing Diversity by optimizing policies to increase the training loss.
While this method did not explicitly aim to increase
Afﬁnity, it also used transforms and hyperparameters from the AutoAugment search space which implicitly have high Afﬁnity.
Without such a prior, the goal of maximizing training loss with no other constraints would lead to data augmentation policies that erase all the information from the images.
C ONCLUSION
We attempted to quantify common intuition that more in-distribution and more diverse augmentation policies perform well.
To this end, we introduced two easy-to-compute metrics, Afﬁnity and Diversity, intended to measure to what extent a given augmentation is in-distribution and how complex the augmentation is to learn.
Because they are model-dependent, these metrics capture the data shifts that affect model performance.
We hope our ﬁndings provide a foundation for continued scientiﬁc study of data augmentation and more general distribution shifts.