How much Data is Augmentation Worth?
Abstract
Despite the clear performance benefits of data augmentations, little is known about why they are so effective.
In this paper, we disentangle several key mechanisms through which data augmentations operate.
Establishing an exchange rate between augmented and additional real data, we find that augmentations can provide nearly the same performance gains as additional data samples for in-domain generalization and even greater performance gains for out-of-distribution test sets.
We also find that neural networks with hard-coded invariances underperform those with invariances learned via data augmentations.
Our experiments suggest that these benefits to generalization arise from the additional stochasticity conferred by randomized augmentations, leading to flatter minima.
1. Introduction
Even with the proliferation of large-scale image datasets, deep neural networks for computer vision parameterize highly flexible model families and often contain orders of magnitude more parameters than the size of their training sets.
As a result, large models trained on limited datasets still have the capacity for improvement (LeCun et al., 1998a).
The importance of data augmentation for boosting performance leads us to wonder whether it benefits training through more complex mechanisms than simply adding more data.
In addition to adding extra samples, augmentation may regularize models by promoting invariance.
Just as classifiers learn to make the same prediction across samples
Published at the ICML
Workshop on Spurious Correlations, Invariance and Stability .
Baltimore, Maryland, USA.
Copyright 2022 by the author(s).
in each class, data augmentations (DA) encourage models to make consistent predictions across augmented views of each sample.
Data augmentations thus promote invariance by imposing implicit model constraints that can assist in the underdetermined model regime.
We can use our apriori beliefs about which invariances are present in the data to design augmentations.
The need to incorporate invariances in neural networks has motivated the development of architectures that are instead explicitly constrained to be equivariant to transformations (Weiler and Cesa, 2019;
Finzi et al., 2020).
If the downstream effects of data augmentations were attributable solely to invariance, then we could replace DA with explicit model constraints.
However, data augmentations may affect training dynamics beyond imposing constraints.
In addition to promoting invariance, augmentation serves as an extra source of stochasticity.
Under DA, randomization during training comes not only from randomly selecting samples from the dataset to form batches but also from sampling transformations with which to augment data (Fort et al., 2022).
Stochastic optimization is associated with benefits in non-convex problems wherein the optimizer can bias parameters towards flatter minima (Jastrz˛ebski et al., 2017;
Geiping et al., 2021; Liu et al., 2021).
In this paper, we revisit the role of data augmentation, as it may be more multi-faceted than the classical view of extra data:
• We quantify the relationship between augmented views of training samples and extra data.
We find that augmentations can confer comparable benefits to independently drawn samples on in-domain test sets and even stronger benefits on slightly out-of-distribution testing.
• We observe that models which learn invariances via data augmentation consistently outperform architectures that are instead constrained with equivariance to the same transformations, suggesting that DA regularizes models beyond invariance.
Moreover, the standard model of data augmentation dictates that one should choose transforms under which the distribution is invariant, and yet we show that invariances which are uncharacteristic of the data distribution still benefit performance.
[Figure]
[Figure] [Figure]
• We consider a sampling strategy for augmentation that reduces randomness during training by averaging each batch over all transformed views of an image at once and find that DA exhibits flatness-seeking behavior and greater stochasticity during late stages of training.
2. Augmentations as Additional Data
A central role of data augmentation is to serve as extra data and expand limited datasets used for training large models.
In this section, we quantify this property, conducting a series of experiments to quantify the exchange rates , which indicate how much data an augmentation is worth – a number of additional samples which yields the same performance boost as the augmentation policy.
We conduct these experiments on the CINIC-10 dataset (Darlow et al., 2018), a drop-in replacement for CIFAR10 (Krizhevsky, 2009) which contains numerous additional samples.
This allows us to train models with augmented data on dataset sizes similar to CIFAR-10, but compare to reference models trained without augmentations on larger datasets.
We further use this replacement to illustrate a behavior of augmentations that is often underappreciated:
We evaluate the accuracy of the same models trained on
CINIC-10 not only on a CINIC-10 validation set, but also the CIFAR-10 validation set.
Both datasets are nearly indistinguishable using simple summary statistics (Darlow et al., 2018), yet there is a minor distribution shift caused by different image processing protocols during dataset curation.
We argue that this is a reasonable test case for practical scenarios in which there could be even a minor uncertainty about the distribution of test data.
How much extra data is gained through data augmentations and how does this number change as the number of base samples increase, or the data distribution shifts?
Figureshows that the validation accuracy for this model is well modeled by power laws of the form f ( x ) =
ax − c + b for both validation sets.
We then evaluate the relative difference between these power laws in terms of absolute number of extra samples gained through augmentation and extrapolate to larger sample sizes.
Under this view, we see that data augmentations are transient.
Investigating Figure 1a, we find that for some amount of base samples from the original dataset, the effectiveness of each augmentation peaks and the augmentation effectively generates the most extra data; then with more base samples, the benefits of all augmentations diminish up to a point where we extrapolate that they would not be beneficial any more and may in fact hurt performance.
Here, we also see a reversal of the trends from our previous study on diversity.
The diverse augmentations such as TrivialAug peak early.
Yet, the consistent, and least diverse, augmentations of horizontal flips falls off slowest, showing that with enough real samples, diverse, but inconsistent policies make poorer augmentations.
Yet, this behavior is highly contingent on the evaluation on in-domain data.
Even, the shift to evaluation on CIFAR10 in Figure 1b reveals that while the transient behavior described in the previous paragraph is still present, the actual extra data gained is much more significant.
3. Augmentations are Worth More Than Invariance
A different angle on data augmentations is that they encourage invariance to image transformations by enforcing the assignment of identical labels across transformations of each training sample.
If the success of data augmentations can be attributed solely to invariance, then we can build exactly invariant models which achieve comparable accuracy when trained without data augmentation.
Several works propose mechanisms for constraining neural network layers to be invariant, and we will leverage these in our study.
In this section, we find that the benefits of data augmentation can
[Figure]
T e s t
A cc u r a c y
[Figure]
[Figure]
[Figure]
only partially be recovered by building invariant models.
Moreover, we discover that augmentations which reflect invariances not present in the data distribution can still offer performance boosts, indicating that the benefits of data augmentation are not limited to invariances which exist in the data distribution.
One might think that the ability to remain only approximately invariant to transformations is a valuable feature of data augmentations, since a particular invariance might be inconsistent with the data distribution (Finzi et al., 2021).
However, a notable property of training on augmented data is that the loss function does not distinguish between raw and augmented images, and so the model is not able to overcome invariances which are inconsistent with the training labels.
Therefore, when models are trained with data augmentations, instead of determining whether invariances are consistent with the distribution, they learn only approximate invariances since they may not perfectly fit the augmented training data, and their invariance might not generalize to test data.
This fact makes it all the more mysterious that exactly invariant networks underperform those trained with augmentation.
3.1. Invariant Neural Networks Without Augmentation
In order to probe the benefits of invariance without augmentations, we adopt the following three methods for constructing invariant networks.
Prediction averaging: Insert augmented views of a sample into the network and average the corresponding predictions (Shanmugam et al., 2021; Gandikota et al., 2021).
We use this procedure during both training and inference.
Note that this method still involves passing augmented data into the model.
E2CNN:
General E(2)-Equivariant Steerable CNN (E2CNN) (Weiler and Cesa, 2019) constrains convolutional kernels to reflect a group equivariance.
LieConv: LieConv (Finzi et al., 2020) is a convolutional layer which is equivariant to a user-specified Lie group, yielding entire models which possess this equivariance property.
We use LieConv layers in a
ResNet-18 model, which we will refer to as LieConvNet.
We employ the above methods to train ResNet-18 models on CIFAR-10 which are exactly invariant to horizontal flips.
Figure 2a compares their performance.
E2CNN, LieConvNet, and prediction averaging models all improve performance over non-invariant models trained on non-augmented data, there remains a considerable accuracy gap between invariant neural networks and the same architecture trained instead with horizontal flip data augmentation.
3.2. Out-Of-Distribution Augmentations Still Improve Performance
Previously, we observed the performance benefits of data augmentations which promote invariances consistent with the data distribution, or approximately so, even over existent invariant neural networks.
But can it still be useful to augment our data with a transformation that generates samples completely outside the support of the data distribution and which are inconsistent with any label?
To answer this question, we construct a synthetic dataset in which the exact invariances are known.
We begin by randomly sampling a single base image from each CIFAR-10 class.
We then construct 10 classes by rotating each of the base images, so that all samples in a class correspond to rotations of a single image.
Thus, the classification task at hand is to determine which base image was rotated to form the test sample.
We randomly sample rotations from each of these classes to serve as training data and another disjoint set of rotations to serve as test data.
We then use horizontal flip and random crop data augmentations to generate out-of-distribution samples, since horizontally flipped or cropped image views cannot be formed merely via rotation.
Note that this experiment is distinct from typical covariate shift setups where the
[Table]
distribution of data domains differs, but the support is far from disjoint and may even be identical.
In Figure
[Table]
we see that these out-of-distribution augmentations are beneficial nonetheless.
Notably, random crops, which can generate significantly more unique views than horizontal flips, yield massive performance boosts for identifying rotated images, even though we know that the cropped samples are out-of-distribution.
We also see in this figure that random crops are especially useful if we instead use as our test set rotations of other samples from
CIFAR-10 than those used from training.
Specifically, we assign a base test image and its rotations the same label as the base image from the training set with the same
CIFAR-10 label.
This experiment supports the observations from Section 2 that augmentations can be particularly beneficial for OOD generalization.
4. Data Augmentation As a Source of Stochasticity During Training
Typical neural network loss functions contain individual terms for each training sample.
During optimization, randomly sampling batches of data points (i.e. terms in the loss) and computing gradients on batches rather than the full loss gives rise to stochasticity.
Augmentations increase the number of terms in the loss function, often to such an extent that we never sample the same term twice during training.
Since data augmentations expand and diversify the body of samples, and equivalently terms of the loss function, available for sampling, they may serve as additional sources of stochasticity during optimization.
As neural networks are typically trained with first-order optimizers, this boost in stochasticity can be examined by measuring the variety of gradients across batches during training.
If data augmentations do in fact increase the variety of gradients, then they could as a result cause us to find qualitatively different minima.
Stochastic optimization is thought to be associated with flat minima of the loss landscape which are in turn associated with superior generalization (Jastrz˛ebski et al., 2017; Huang et al., 2020; Liu et al., 2021).
Moreover, this flatness-seeking behavior may be the effect not only of the augmented loss function but also how we sample it.
In this section, we put this hypothesis to the test.
We measure the standard deviation of gradients during optimization for models trained with and without data augmentations, and we quantify the flatness of the minima these variants find.
We describe how we measure both stochasticity and flatness in the appendix.
We construct experiments that disentangle the augmented loss function from the additional stochasticity produced by randomly sampling augmented views.
We consider a “same batch" strategy in which gradient updates are averaged over multiple views of a single image, resulting in lower stochasticity.
We also consider “fixed views" experiments in which we pre-compute a frozen set of augmented views per element of the training set, which can then be sampled during training.
In Table 1, we see that for each augmentation policy, applying augmentations randomly results in the most stochasticity at the end of training, while including multiple random views in the same batch (Hoffer et al., 2020) results in less.
Sampling augmentations from a fixed set of four views per sample (denoted “fixed views”) results in even less stochasticity, and including each of the four views in every batch results in the least stochasticity (denoted “fixed vews”, “same batch”).
This ordering, which holds across all data augmentations we try, is consistent with the intuition that more randomness in augmentation leads to more stochasticity in training.
Including each of the four views for each member of the batch entirely removes the randomization of data augmentations but still preserves the same exact possible combinations of base images available for sampling under all other training setups we consider, allowing us to train on the same augmented loss function but without the extra randomization from sampling augmented views.
In this case, we have 4 views per base image andbase images, making a total of 4 × 128 = 512 samples in the batch including augmentations.
We further observe that flatness correlates strongly with late-training stochasticity.
Models trained without augmentation or with non-random augmentation, where all views are seen in each batch, exhibit less stochasticity at the end of training and find sharper minima.
While previous works have associated SGD with flatness-seeking behavior (Jastrz˛ebski et al., 2017; Geiping et al., 2021), the findings here indicate that data augmentations can also contribute to this phenomenon.
Simply put, training with randomized data augmentations finds flatter minima, and models trained with strong data augmentations such as RandAugment and
TrivialAugment lie at especially flat minima.
5. Discussion
This work promotes an all-encompassing understanding of neural network training which incorporates the underappreciated factor that is data augmentation, covering the effective extra data , invariance and stochasticity provided by augmentations.
Data augmentation has had a profound impact on the performance of neural networks, but their precise role has not been well understood; for example, if augmentations are simply a heuristic for learning certain symmetries, should we not prefer to directly encode these symmetries through advances in group equivariant networks?
We first establish a new lens through which to view the efficacy of specific augmentations, when we discuss exchange rates and their power laws, and we then probe the underlying phenomena that contribute to their success.
We uncover that the effective extra data gained from augmentations is not simply a matter of learned invariance, as out-of-domain transformations still improve performance, and we measure the close correlation between the stochasticity gained from augmentations and the flatness of landscapes.
This work promotes an all-encompassing understanding of neural network training which incorporates the often ignored factor that is data augmentation.
A. Background and Related Work
Data Augmentations in Computer Vision.
Data augmentations have been a staple of deep learning, used to deform handwritten digits as early as Yaeger et al.
; LeCun et al.
(1998a)
or to improve oversampling on class-imbalanced datasets (Chawla et al., 2002).
These early works hypothesize that data augmentations are necessary to prevent overfitting when training neural networks since they typically contain many more parameters than training data points (LeCun et al., 1998a).
We restrict our study to augmentations which act on a single sample and do not modify labels.
Namely, we study augmentations which can be written as ( T ( x ) , y ) , where ( x , y ) denotes an input-label pair, and T ∼ T is a random transformation sampled from a distribution of such transformations.
For a broad and thorough discussion on image augmentations, their categorization, and applications to computer vision, see (Shorten and Khoshgoftaar, 2019; Xu et al., 2022).
We consider basic geometric (random crops, flips, perspective) and photometric (jitter, blur, contrast) transformations, and common augmentation policies, such as AutoAug (Cubuk et al., 2019a), RandAug (Cubuk et al., 2019b), AugMix (Hendrycks et al., 2020) and TrivialAug (Müller and Hutter, 2021) which combine basic augmentations.
Understanding the Role of Augmentation and Invariance.
A number of works, such as Hernández-García and
König (2018), propose that data augmentations (DA) induce implicit regularization.
Empirical evaluations describe useful augmentations as “label preserving”, namely they do not significantly change the conditional probability over labels (Taylor and Nitschke, 2018).
Gontijo-Lopes et al.
(2020b;a) investigate empirical notions of consistency and diversity.
They measure consistency (referred to as affinity) evaluating models trained without augmentation on augmented validation accuracy yielded.
They also measure diversity as the ratio of training loss of a model trained with augmentations and a model trained without them and conclude that strong data augmentations should be both consistent and diverse.
A trade-off between diversity and consistency is also seen in NLP applications (Kim et al., 2021).
In contrast to Gontijo-Lopes et al. (2020b),
Marcu and Prügel-Bennett (2021) find that the value of data augmentations cannot be measured by how much they deform the data distribution.
Other work proposes to learn invariances parameterized as augmentations from the data (Benton et al., 2020), investigates the number of samples required to learn an invariance (Balestriero et al., 2022b), uncovers the tendency of augmentations to sacrifice performance on some classes in exchange for gains on others (Balestriero et al., 2022a), or argues that data augmentations cause models to misrepresent uncertainty (Kapoor et al., 2022).
Theoretical investigations in Chen et al.
formalize data augmentations as label-preserving group actions and discuss an inherent invariance-variance trade-off.
Variance regularization also arises when modeling augmentations for kernel classifiers (Dao et al., 2019).
For a binary classifier with finite VC dimension, the bound on expected risk can be reduced through additional data generated via augmentations until inconsistency between augmented and real data distributions overwhelms would-be gains (He et al., 2019b).
The regularizing effect of data augmentations is investigated in LeJeune et al.
(2019) who propose a model under which continuous augmentations increase the smoothness of neural network decision boundaries.
Rajput et al.
(2019) similarly find that linear classifiers trained with sufficient augmentations can approximate the maximum margin solution.
Hanin and Sun (2021) relate data augmentations to stochastic optimization.
A different angle towards understanding invariances through data augmentations is presented in Zhu et al.
(2021), where the effect of
DA in increasing the theoretical sample cover of the distribution is investigated, and augmentations can reduce the amount of data required, if they “cover” the real distribution.
In language applications, the efficacy of augmentations have been probed with counterfactual examples Kaushik et al.
Zmigrod et al.
; Hall Maudslay et al.
Human-generated counterfactual augmentations of a given example towards a target label have been shown to be effective augmentations.
These examples interestingly flip the previously discussed roles in the invariance-variance trade-off, generating examples that only differ in causal features, but are otherwise near-invariant in other features.
Stochastic Optimization and Neural Network Training.
The implicit regularization of SGD is regarded as an essential component for neural network generalization (An, 1996; Neyshabur et al., 2017).
Stochastic training which randomizes gradients can drive parameters into flatter minima, associated with superior generalization (Jastrz˛ebski et al., 2017; Huang et al., 2020; Liu et al., 2021).
In fact, Geiping et al.
(2021) find that neural networks trained with non-stochastic full batch gradient descent require explicit flatness-seeking regularizers in order to achieve comparable test accuracy.
Data augmentations provide an additional source of stochasticity during training on top of batch sampling, which we will investigate in this work.
A window into the effect of data augmentations on stochasticity is “batch augmentation” (Hoffer et al., 2020; Fong and
[Table]
(cid:76) (cid:72) (cid:81) (cid:87) (cid:49)(cid:82)(cid:3)(cid:36)(cid:88)(cid:74)(cid:17) (cid:43)(cid:82)(cid:85)(cid:76)(cid:93)(cid:17)(cid:3)(cid:41)(cid:79)(cid:76)(cid:83)(cid:3)(cid:9)(cid:3)(cid:53)(cid:68)(cid:81)(cid:71)(cid:17)(cid:3)(cid:38)(cid:85)(cid:82)(cid:83) (cid:55)(cid:85)(cid:76)(cid:89)(cid:17)(cid:3)(cid:36)(cid:88)(cid:74)(cid:17)(cid:3)(cid:9)(cid:3)(cid:43)(cid:82)(cid:85)(cid:76)(cid:93)(cid:17)(cid:3)(cid:41)(cid:79)(cid:76)(cid:83)(cid:3)(cid:9)(cid:3)(cid:53)(cid:68)(cid:81)(cid:71)(cid:17)(cid:3)(cid:38)(cid:85)(cid:82)(cid:83) (cid:53)(cid:68)(cid:81)(cid:71)(cid:17)(cid:3)(cid:36)(cid:88)(cid:74)(cid:17)(cid:3)(cid:9)(cid:3)(cid:43)(cid:82)(cid:85)(cid:76)(cid:93)(cid:17)(cid:3)(cid:41)(cid:79)(cid:76)(cid:83)(cid:3)(cid:9)(cid:3)(cid:53)(cid:68)(cid:81)(cid:71)(cid:17)(cid:3)(cid:38)(cid:85)(cid:82)(cid:83)
[Table]
(cid:49) (cid:82) (cid:85) (cid:80) (cid:3) (cid:82) (cid:73) (cid:3) (cid:54) (cid:87) (cid:71) (cid:3) (cid:82) (cid:73) (cid:3) (cid:42) (cid:85) (cid:68) (cid:71) (cid:76) (cid:72) (cid:81) (cid:87) (cid:49)(cid:82)(cid:3)(cid:36)(cid:88)(cid:74)(cid:17) (cid:43)(cid:82)(cid:85)(cid:76)(cid:93)(cid:17)(cid:3)(cid:41)(cid:79)(cid:76)(cid:83)(cid:3)(cid:9)(cid:3)(cid:53)(cid:68)(cid:81)(cid:71)(cid:17)(cid:3)(cid:38)(cid:85)(cid:82)(cid:83) (cid:55)(cid:85)(cid:76)(cid:89)(cid:17)(cid:3)(cid:36)(cid:88)(cid:74)(cid:17)(cid:3)(cid:9)(cid:3)(cid:43)(cid:82)(cid:85)(cid:76)(cid:93)(cid:17)(cid:3)(cid:41)(cid:79)(cid:76)(cid:83)(cid:3)(cid:9)(cid:3)(cid:53)(cid:68)(cid:81)(cid:71)(cid:17)(cid:3)(cid:38)(cid:85)(cid:82)(cid:83) (cid:53)(cid:68)(cid:81)(cid:71)(cid:17)(cid:3)(cid:36)(cid:88)(cid:74)(cid:17)(cid:3)(cid:9)(cid:3)(cid:43)(cid:82)(cid:85)(cid:76)(cid:93)(cid:17)(cid:3)(cid:41)(cid:79)(cid:76)(cid:83)(cid:3)(cid:9)(cid:3)(cid:53)(cid:68)(cid:81)(cid:71)(cid:17)(cid:3)(cid:38)(cid:85)(cid:82)(cid:83)
Vedaldi, 2019), also termed “repeated augmentations” (Berman et al., 2019; Touvron et al., 2021; Wightman et al., 2021).
Batch augmentation strategies average over multiple augmentations of each sample in the batch, resulting in gradients with less randomness that if a single augmentation is chosen per sample.
Although this strategy is not well understood, it is employed in several modern training recipes.
Fort et al.
(2022) find that batch augmentation via decreasing the number of independent samples in a batch and including multiple augmentations of each, which decreases the stochasticity from augmentation while increasing the stochasticity from batch sampling, can boost accuracy.
They conclude that the additional stochasticity from data augmentation is harmful rather than helpful, although they do not compare to cases where training data is augmented but the additional stochasticity is eliminated entirely.
We ablate away the stochasticity introduced by
DA entirely, and we instead find that this stochasticity has the positive benefit of discovering flatter minima.
We fuse together the above three topics and explore the role of data augmentations play in learning invariance and in increasing stochasticity during late stages of training.
In doing so, we fill in several gaps in the literature discussed in this section.
Unlike other works which measure the effectiveness of data augmentations in terms of accuracy boosts, we compare the benefits of augmentations to those achieved by instead collecting more data.
While other works have studied the role of data augmentations in learning invariance, we find that even invariances which have no relationship to invariances in the training data distribution are still effective.
Finally, we develop an understanding of batch augmentation by showing that stochastically applied augmentations increase gradient noise during training, leading to qualitatively distinct minima.
B. Experimental Setup
For all sections if not otherwise mentioned, we run the following protocol.
We train the model (in the main body a
ResNet-18), with stochastic gradient descent for 60000 steps with a batch size of 128.
This corresponds to 160 epochs for a dataset of size 48000 .
For experiments with 8 × or larger enlarged datasets, we set a minimum number of 40 epochs, e.g. corresponding to 120000 steps for the 8 × experiments, but otherwise do not modify the number of gradient steps when increasing or decreasing the dataset size.
We linearly warm up the learning rate for the first 2000 steps (about 5 epochs) up to peak rate of 0 .
and then decay to zero by a half-cycle of cosine annealing.
For all experiments, we include a standard weight decay of 5 e − 4 and train with Nesterov momentum of 0 .
The data is shuffled randomly after every epoch and we record validation accuracy every 10000 steps.
All training runs are non-deterministic based on stochasticity due to random shuffling and cudnn non-determinism.
We run at least five trials for each experiment in the main body and three trials for each in the supplementary material.
In each plot, the standard deviation is shaded.
For five trials this corresponds close to a 97 .
5% confidence interval.
We use CIFAR-10 in its default configuration.
For CINIC-10, we clean and resample the train and test sets.
We first remove all CIFAR-10 train and test images from the dataset, we then further remove all exactly duplicated images and missing images, merge all remaining images and sample a new validation set of 10000 images.
We provide code to replicate the creation of this cleaned dataset with the supplementary material.
Overall we recover a new training set of size 193523.
For
CIFAR-10-C experiments in the supp.
material, we report average accuracy over all transformations in CIFAR-10-C with a severity of 3.
For all experiments where we consider only a subset of the existing data (e.g. each experiment with less than 193523 samples for CINIC-10), we sample a new subset of the training set for each experiment separately, to rule out confounding effects of good or bad splits of the training data, especially for smaller subset sizes.
For experiments in the main body where data augmentations are randomly sampled a finite number of times, we store all augmentations in a database ( lmdb ) that is recreated in each run.
As a result, each experiment contains a fixed set of finite views of each original datapoint, but these views are randomized across experiments.
Due to random shuffling, samples from this enlarged dataset are drawn randomly and multiple views of the image are only guaranteed to occur in the same batch in the batch augmentation experiments in Section 5.
To create the table of exchange rates, we first compute the mean validation accuracy
CINICfor each experiment.
We then train the reference models for
CINICsubset sizes of 1000 , 2000 , 3000 , 6000 , 12000 , 24000 , 48000 , 96000 , 128000 , 144000 , 168000 , 180000 , 192000 .
To cross-reference the average validation accuracy of these reference models with our data augmentation experiments, we assert that validation accuracies are monotonically increasing as subset sizes increase and fit a linear spline f ref for interpolation.
We then compute the exchange ratios of Table 1 via f − 1 ref ( x ) /b , for the base dataset size b which is 48000 in Table 1 and input mean validation accuracy x for each experiment.
For values outside the interval spanned by the minimal and maximal validation accuracies of the reference data, we reuse the power laws of the form f p ( x ) =
ax − c + b described in Sec.
3.3 and again compute f − 1 p ( x ) /b .
We mark these extrapolated values by a ∗ in the table.
To estimate parameters a, b, c for f p ( x ) = ax − c + b in the exchange rate table and Sec.
3.3 we use a non-linear leastsquares algorithm, initialized from starting parameters that describe the curve for no augmentations.
For this we use the
Levenberg-Marquardt implementation of MINPACK , as wrapped in scipy .
Measuring stochasticity.
We measure stochasticity during training as follows:
We train a model on a given training set and augmentation strategy, and we freeze the model every 10 epochs to estimate the standard deviation (formally the norm of parameter-wise standard deviations) of its gradients over randomly sampled batches comprising 128 base images, the same batch size used during training.
That is, we measure the square root of the average squared distance between a randomly sampled batch gradient and the mean gradient.
We adopt a filter-normalized distance function (Li et al., 2018; Huang et al., 2020) to account for invariances in neural networks whereby shrinking the parameters in convolutional filters may not effect the network’s output but may make the model more sensitive to parameter perturbations of a fixed size.
Measuring flatness.
We adopt the flatness measurements from Huang et al.
(2020) as these measurements are non-local, do not require Hessian computations which are dubious for non-smooth ReLU networks, and they are consistent with our filter-normalized gradient standard deviation measurements.
Specifically, we measure the average filter-normalized distance in random directions from the trained model parameters before we reach a loss function value of 1 . 0 , where loss is evaluated on the non-augmented dataset.
Under this metric, larger values correspond to flatter minima where parameters can be perturbed further without greatly increasing loss.
We use the same ResNet-18 models trained in the stochasticity experiments above with the same exact augmentation setups.
The ResNet-18 model employed in the model is a modern variant (He et al., 2015; 2019a) and contains the usual CIFARstem consisting of a single 3 × 3 convolutional layer without pooling, instead of the ImageNet stem (of two convolutional layers with stride and max-pooling).
For experiments in the supplementary material, we further consider a ResNet- (i.e. three stages and a single block per stage)
(He et al., 2015), a VGG-11 (Simonyan and Zisserman, 2014) with batch normalization, and a ConvMixer architecture (Trockman and Kolter, 2022) of depth 8 with hidden dimension 128 and spatial kernel size of 7.
We implement and run all experiments in PyTorch and make use of torchvision implementations for a range of data augmentations investigated in this work.
We provide code to replicate all experiments with the supplementary material.
B.1. Hyperparameters for Augmentations
For each augmentation we broadly follow established defaults.
For completion we record these, and additional details here.
Horiz.
Flips:
A data point is flipped horizontally with probability 0.5.
Det.
Horiz.
Flips:
Deterministic horizontal flips.
For 1 × , this corresponds to flipping every data point.
2 × corresponds to both flips being contained in the dataset.
Vert.
Flips:
A data point is flipped vertically with probability 0.5.
Det.
Vert.
Flips:
Deterministic vertical flips.
For 1 × , this corresponds to flipping every data point.
2 × corresponds to both flips being contained in the dataset.
Random Crops: The image is padded by with zero-padding by 4 pixels in each direction and then a image of size 32 ×is cropped (This is classical random cropping for CIFAR-10).
Flips&Crops: Both random crops and horizontal flips are employed, as described above.
Perspectives: Performs a random perspective transform with probability 0.5 with bilinear resampling.
Jitter: Color jitter, randomly transforming contrast, hue and brightness of the image.
For each distortion, sample a new scale uniformly from [0 . 5 , 1 . 5] .
Blur: Blurs the image with a Gaussian blur with σ = 3 .
AutoAug: Employ the augmentation policy of Cubuk et al. (2019a), with the CIFAR-10 policy.
AugMix: The augmentation policy of Hendrycks et al.
RandAug:
The augmentation policy of Cubuk et al.
(2019b), again with the CIFAR-10 policy.
TrivialAug: The augmentation policy of Müller and Hutter (2021) in its “wide" configuration.
AutoAug&Flips&Crops: The AutoAug policy followed by random horizontal flips and random crops as described above.
AugMix&Flips&Crops: The Augmix policy followed by random horizontal flips and random crops as described above.
RandAug&Flips&Crops:
The RandAug policy followed by random horizontal flips and random crops as described above.
TrivialAug&Flips&Crops: The TrivialAug policy followed by random horizontal flips and random crops as described above.
B.2. Data Licensing
We investigate MNIST (LeCun et al., 1998b), CIFAR-10 and CIFAR-100 (Krizhevsky, 2009), EMNIST (Cohen et al., 2017), CIFAR10-C (Hendrycks and Dietterich, 2018) and CINIC-10 (Darlow et al., 2018) and refer to these publications for additional details.
We remove duplicates and missing data from CINIC-10 as described above.
B.3. Computational Setup and Costs
We use an academic cluster with NVIDIA RTXA4000 cards and NVIDIA GTX2080ti cards.
Each job is scheduled on a single GPU and the default setting of 60000 gradient steps takes roughly an hour to train and evaluate.
Including all preliminary experiments we estimate a total usage of about 400 GPU days for this project.
To replicate all experiments in the main body without repeated trials, we estimate a requirement of about 15 GPU days.
C. Broader Impact
We foresee no direct negative societal consequences from this work.
We do think that data augmentations are a beneficial tool, especially in applications with only limited data, or where data curation is expensive.
We argue that knowing how to exchange a smaller (but verified and curated) dataset for a larger dataset that is not augmented, but also due to its size less curated, is helpful to the community.
D. Additional Results
Under our experimental setup, we can contrast the accuracy of augmented datasets in Figure 4 with reference models trained on larger unaugmented datasets from the true data distribution (black horizontal bars).
We see that, for example,
TrivialAug&Flips&Crops can generate enlarged datasets that match the performance of reference models trained on the entire, unaugmented 192000 sample dataset.
We formalize this notion in Table 2.
This table contains scaling factors that quantify how much the dataset size effectively increases when training with augmentations.
We compute this quantity by matching each augmented model with the reference model that achieves the same val.
accuracy and computing the ratio of reference dataset size to base size.
We include additional information about augmentations and hyperparameters for each
[Figure]
5 6 7 8 9 100k 3 4 5 6 7 8 9 1M 3 4 5 6 7 8 9 10M 0.
[Figure] [Figure]
&Crops
Horiz
[Figure]
Perspective
TrivialAug
Size of Dataset Generated from 48k Base Samples
C
I N I C 10 V a l .
A cc u r ac y 192k
[Figure]
samples 48k
[Figure]
4 5 6 7 8 9 100k 3 4 5 6 7 8 9 1M 3 4 5 6 7 8 9 10M 0.
[Figure]
of Dataset Generated from 48k Base Samples
[Figure]
experiment in the appendix.
We show these exchange rates again for both CINIC-10 and CIFAR-10 val.
data.
At a glance, this table reveals consistency (as measured e.g. in Gontijo-Lopes et al.
(2020b))
in the 1 × row, then shows diversity of each augmentation in the trend in each row, and examines robustness by cross-reference to the CIFAR-10 evaluations.
This reference to out-of-domain data is especially relevant to practical use, and we find, for example for AugMix(Hendrycks et al., 2020) that behavior on in-domain data is not indicative of robust performance.
AugMix, which was designed for robustness, does not improve in performance on in-domain data as more views are sampled.
However, for out-of-domain data AugMix views do improve performance, and combined with flips and crops actually generate the largest amount of effective data.
We also include “negative” augmentations, such as vertical flips and blur that are especially inconsistent.
In the case of vertical flips, an augmentation strategy which is neither diverse nor consistent, applying the augmentation enough still yields performance benefits and tangible extra data, indicating that diversity and inconsistency alone paint an incomplete picture of the benefits of augmentation.
We further probe the idea that ”out-of-domain" augmentations can improve performance later on.
We include additional material for section 3 in a series of figures and tables.
Table 3 is an extended version of table 1 in the main body, including repetitions up to 32 × and ablating the number of steps.
Behavior is consistent over additional repetitions, so we chose not to include these additional rows in the main body.
Table 4 and Table 5 are then variants of this table where validation accuracy is evaluated on CIFAR-10 and CIFAR-10-C, respectively.
CIFAR-10-C is a significant distribution shift that cannot be mitigated by additional CINIC-10 data, only training on, e.g. blurred samples, provides robustness to this distortion.
We further find that training with horizontal flips in our experimental setup is quickly disadvantageous.
E. Additional Datasets and Models
We further verify that the findings discussed in the main body are not limited to the choice of dataset and model therein.
We repeat Fig.1 and Table 1 for a range of models.
We include a tiny ResNet-8 in Figure 7 and Table 6, a VGG-11 in Figureand Table 7 and a ConvMixer (as representative of modern ConvNet/Transformer variations) in Figure 9 and Table 8.
We then further repeat these experiments with models trained on the MNIST training set in Figure 10 and Table 9, CIFARin Figure 11 and Table 10, as well as EMNIST in Figure 12 and Table 11.
We further include repeated experiments for Sec. 5 on CIFAR-100 in Figure 13 and Table 12.
[Table] [Table]
[Table] [Table]
[Table] [Table]
Crops
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
03 Perspectives
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
07 Blur
[Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
&Crops 0.
[Table] [Table]
[Table] [Table]
[Table]
&Crops 0.
[Table] [Table]
[Table] [Table]
&Crops 0.
[Table] [Table]
[Table] [Table]
0.88 Blur 0.
AutoAug 0.
1.77 AugMix 0.
1.13 RandAug 0.
- TrivialAug 0.
AutoAug&Flips&Crops 0.
4.00* 4.00* 4.00* 4.08* Augmix&Flips&Crops 0.
3.74 RandAug&Flips&Crops 0.
4.00* 4.00* 4.00* - TrivialAug&Flips&Crops 0.
[Table] [Table]
[Table] [Table]
Perspectives
[Table]
16* ✓ ✓ ✓ ✓ ✓ ✓
[Table] [Table]
* ✓
[Table] [Table]
[Table] [Table]
* Blur ✓ ✓ ✓ ✓ ✓ ✓ ✓
[Table]
✓ ✓ ✓ ✓ ✓ ✓ ✓
[Table]
✓ ✓ ✓ ✓ ✓ ✓ ✓
RandAug ✓ ✓ ✓ ✓ ✓ ✓ ✓
TrivialAug ✓ ✓ ✓ ✓ ✓ ✓ ✓
[Table]
✓ ✓ ✓ ✓ ✓ ✓ ✓
AugMix
[Table]
✓ ✓ ✓ ✓ ✓ ✓ ✓
[Table]
[Table]
64* ✓ ✓ ✓ ✓ ✓ ✓
[Table]
Crops ✓ ✓ ✓ ✓ ✓ ✓ ✓
[Figure]
[Figure]
[Table]
1.80 AugMix 1.
1.67 RandAug 0.
2.68 TrivialAug 0.
AutoAug&Flips&Crops 0.
3.00 AugMix&Flips&Crops 0.
[Table]
3.22 RandAug&Flips&Crops 0.
[Table] [Table]
[Table] [Table]
[Table] [Table]
92 TrivialAug&Flips&Crops
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Figure]
[Table]
1.51 AugMix 0.
1.00 RandAug 0.
2.19 TrivialAug 0.
AutoAug&Flips&Crops 0.
4.00* 4.00* AugMix&Flips&Crops 0.
3.12 RandAug&Flips&Crops 0.
3.92 TrivialAug&Flips&Crops 0.
[Figure]
[Table]
2.07 AugMix 0.
1.70 RandAug 0.
2.28 TrivialAug 0.
AutoAug&Flips&Crops 0.
2.91 AugMix&Flips&Crops 0.
2.20 RandAug&Flips&Crops 0.
2.89 TrivialAug&Flips&Crops 0.
[Figure]
[Table]
AutoAug 0.
4.38* 7.84* 7.84* ✓
AugMix 1.
1.15 RandAug 1.
4.38* ✓ ✓ ✓ 1.
✓
TrivialAug 0.
4.38* ✓ 1.
✓
AutoAug&Flips&Crops 0.
0.26 AugMix&Flips&Crops 0.
0.24 RandAug&Flips&Crops 0.
0.23 TrivialAug&Flips&Crops 0.
[Figure]
Size of Dataset Generated from
[Figure]
Base Samples
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
AutoAug 1.
13.79* 2.03 AugMix 1.
1.99 RandAug 1.
✓ ✓ ✓ ✓
TrivialAug 0.
✓ ✓
AutoAug&Flips&Crops 0.
0.25 AugMix&Flips&Crops 0.
0.16 RandAug&Flips&Crops 0.
0.21 TrivialAug&Flips&Crops 0.
[Figure]
[Figure] [Figure]
[Table]
[Figure]
i e n t S t a nd a r d
D e v .