H OW M UCH D ATA A RE A UGMENTATIONS W ORTH ? A N I NVESTIGATION INTO S CALING L AWS , I NVARIANCE , AND I MPLICIT R EGULARIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Despite the clear performance benefits of data augmentations, little is known about why they are so effective.
In this paper, we disentangle several key mechanisms through which data augmentations operate.
Establishing an exchange rate between augmented and additional real data, we find that in out-of-distribution testing scenarios, augmentations which yield samples that are diverse, but inconsistent with the data distribution can be even more valuable than additional training data.
Moreover, we find that data augmentations which encourage invariances can be more valuable than invariance alone, especially on small and medium sized training sets.
Following this observation, we show that augmentations induce additional stochasticity during training, effectively flattening the loss landscape.
I NTRODUCTION
Even with the proliferation of large-scale image datasets, deep neural networks for computer vision represent highly flexible model families and often contain orders of magnitude more parameters than the size of their training sets.
As a result, large models trained on limited datasets still have the capacity for improvement.
To make up for this data shortage, standard operating procedure involves diversifying training data by augmenting samples with randomly applied transformations that preserve semantic content.
These augmented samples expand the volume of data available for training, resulting in downstream performance benefits that one might expect from a larger dataset.
However, the now profound significance of data augmentation (DA) for boosting performance suggests that its benefits may be more nuanced than previously believed.
In addition to adding extra samples, augmentation promotes invariance by encouraging models to make consistent predictions across augmented views of each sample.
The need to incorporate invariances in neural networks has motivated the development of architectures that are explicitly constrained to be equivariant to transformations (Weiler & Cesa, 2019; Finzi et al., 2020).
If the downstream effects of data augmentations were attributable solely to invariance, then we could replace
DA with explicit model constraints.
However, if explicit constraints cannot replicate the benefits of augmentation, then augmentations may affect training dynamics beyond imposing constraints.
Finally, augmentation may improve training by serving as an extra source of stochasticity.
Under DA, randomization during training comes not only from randomly selecting samples from the dataset to form batches but also from sampling transformations with which to augment data (Fort et al., 2022).
Stochastic optimization is associated with benefits in non-convex problems wherein the optimizer can bias parameters towards flatter minima (Jastrz˛ebski et al., 2018; Geiping et al., 2021; Liu et al., 2021a).
In this paper, we re-examine the role of data augmentation.
In particular, we quantify the effects of data augmentation in expanding available training data, promoting invariance, and acting as a source of stochasticity during training.
In summary:
• We quantify the relationship between augmented views of training samples and extra data, evaluating the benefits of augmentations as the number of samples rises.
We find that augmentations can confer comparable benefits to independently drawn samples on in-domain test sets and even stronger benefits on out-of-distribution testing.
• We observe that models that learn invariances via data augmentation provide additional regularization compared to invariant architectures and we show that invariances that are uncharacteristic of the data distribution still benefit performance.
• We then clarify the regularization benefits gained from augmentations through measurements of flatness and gradient noise showing how DA exhibits flatness-seeking behavior.
R ELATED W ORK
Data Augmentations in Computer Vision.
Data augmentations have been a staple of deep learning, used to deform handwritten digits as early as Yaeger et al.
(1996) and LeCun et al.
(1998), or to improve oversampling on class-imbalanced datasets (Chawla et al., 2002).
These early works hypothesize that data augmentations are necessary to prevent overfitting when training neural networks since they typically contain many more parameters than training data points (LeCun et al., 1998).
We restrict our study to augmentations which act on a single sample and do not modify labels.
Namely, we study augmentations which can be written as ( T ( x ) , y ) , where ( x , y ) denotes an input-label pair, and T ∼ T is a random transformation sampled from a distribution of such transformations.
For a broad and thorough discussion on image augmentations, their categorization, and applications to computer vision, see Shorten & Khoshgoftaar (2019) and Xu et al.
We consider basic geometric (random crops, flips, perspective) and photometric (jitter, blur, contrast) transformations, and common augmentation policies, such as AutoAug (Cubuk et al., 2019), RandAug (Cubuk et al., 2020), AugMix (Hendrycks et al., 2020) and TrivialAug (Müller & Hutter, 2021) which combine basic augmentations.
Understanding the Role of Augmentation and Invariance.
Works such as Hernández-García & König (2018) propose that data augmentations (DA) induce implicit regularization.
Empirical evaluations describe useful augmentations as “label preserving”, namely they do not significantly change the conditional probability over labels (Taylor & Nitschke, 2018).
Gontijo-Lopes et al.
(2020b;a) investigate empirical notions of consistency and diversity, and variations in dataset scales (Steiner et al., 2022).
They measure consistency (referred to as affinity) as the performance of models trained without augmentation on augmented validation sets.
They also measure diversity as the ratio of training loss of a model trained with and without augmentations and conclude that strong data augmentations should be both consistent and diverse, an effect also seen in Kim et al.
In contrast to Gontijo-Lopes et al. (2020b), Marcu & Prugel-Bennett (2022) find that the value of data augmentations cannot be measured by how much they deform the data distribution.
Other work proposes to learn invariances parameterized as augmentations from the data (Benton et al., 2020), investigates the number of samples required to learn an invariance (Balestriero et al., 2022b), uncovers the tendency of augmentations to sacrifice performance on some classes in exchange for gains on others (Balestriero et al., 2022a), or argues that data augmentations cause models to misrepresent uncertainty (Kapoor et al., 2022).
Theoretical investigations in Chen et al.
(2020a)
formalize data augmentations as label-preserving group actions and discuss an inherent invariance-variance trade-off.
Variance regularization also arises when modeling augmentations for kernel classifiers (Dao et al., 2019).
For a binary classifier with finite VC dimension, the bound on expected risk can be reduced through additional data generated via augmentations until inconsistency between augmented and real data distributions overwhelms would-be gains (He et al., 2019b).
The regularizing effect of data augmentations is investigated in
LeJeune et al.
(2019) who propose a model under which continuous augmentations increase the smoothness of neural network decision boundaries.
Rajput et al.
(2019) similarly find that linear classifiers trained with sufficient augmentations can approximate the maximum margin solution.
Hanin & Sun (2021) relate data augmentations to stochastic optimization.
A different angle towards understanding invariances through data augmentations is presented in Zhu et al.
(2021), where the effect of DA in increasing the theoretical sample cover of the distribution is investigated, and augmentations can reduce the amount of data required, if they “cover” the real distribution.
Stochastic Optimization and Neural Network Training.
The implicit regularization of SGD is regarded as an essential component for neural network generalization (An, 1996; Neyshabur et al., 2017).
Stochastic training which randomizes gradients can drive parameters into flatter minima, associated with superior generalization (Jastrz˛ebski et al., 2018; Huang et al., 2020; Liu et al., 2021a).
In fact, Geiping et al.
(2021) find that neural networks trained with non-stochastic full batch gradient descent require explicit flatness-seeking regularizers in order to achieve comparable test accuracy.
[Figure]
Data augmentations provide an additional source of stochasticity during training on top of batch sampling, which we will investigate in this work.
We fuse together the above three topics and explore the role of data augmentations play in learning invariance and in increasing stochasticity during late stages of training.
In doing so, we fill in several gaps in the literature discussed in this section.
Unlike other works which measure the effectiveness of data augmentations in terms of accuracy boosts for fixed sample sizes, we compare the benefits of augmentations to those achieved by instead collecting more data.
While other works have studied the role of data augmentations in learning invariance, we find that even invariances which have no relationship to invariances in the training data distribution are still effective.
Finally, we develop an understanding of batch augmentation by showing that stochastically applied augmentations increase gradient noise during training, leading to qualitatively distinct minima.
A UGMENTATIONS AS A DDITIONAL D ATA
A central role of data augmentation is to serve as extra data and expand limited datasets used for training large models.
In this section, we quantify this property, conducting a series of experiments culminating in measurements exchange rates , which indicate exactly how much data an augmentation is worth – the number of additional data samples which would yield the same performance gain as the augmentation.
Such exchange rates constitute a novel angle for quantifying the practical benefits of data augmentations and conceptualize qualitative notions of consistency, diversity, and robustness to distribution shifts, and allow us to observe the properties of data augmentations over a wider range of dataset sizes.
We conduct these experiments on subsets of the CINIC-10 dataset (Darlow et al., 2018), a drop-in replacement for CIFAR-10 (Krizhevsky, 2009), which contains ∼ 200 000 samples.
This setup allows us to train models with augmented data on dataset subsets similar to CIFAR-10, but compare to reference models trained without augmentations on larger datasets.
We start with
ResNet-18 architectures, evaluating their exchange rate behavior, but consider other architectures in
Section 3.1.2.
Our dataset setup is quite specific to CIFAR-10/CINIC-10, revisiting this experiment e.g. for ImageNet would require running a pairing such as ImageNet/JFT-300 (Sun et al., 2017), which would be prohibitive for the number of experiments in this work.
Nevertheless we include several validations on CIFAR-100, EMNIST, MNIST in Appendix B to verify that the behaviors observed are not specific to CIFAR-10.
3. E XCHANGE R ATES : H OW M ANY S AMPLES ARE A UGMENTATIONS W ORTH ?
We visualize the validation accuracies for a range of models trained with select augmentations in
Figure 1 (left).
The validation behavior of these models can be well described by power laws of the form f ( x ) =
ax − c + b , describing the relationship between number of samples and validation accuracy.
From these power laws, we derive the exchange rates of various augmentations compared to the reference curve of un-augmented models f ref = a ′ x − c ′ + b ′ .
For an augmentation policy described by
f aug = ax − c + b , we define its exchange rate via
[Equation]
for a base dataset size x .
We visualize this quantity in Figure 1 (right).
This metric measures exactly the gain in accuracy between the un-augmented and augmented power laws shown on the left-hand side and converts this quantity into additional samples by evaluating on how much more data the reference models would need to realize the same accuracy as the augmented model.
A core property of augmentations that becomes evident in this analysis is the relationship between consistency of augmented samples with the underlying data distribution (Taylor & Nitschke, 2018;
Gontijo-Lopes et al., 2020b;
He et al., 2019b) and diversity of extra data (Gontijo-Lopes et al., 2020b).
In Figure 1, we find that an augmentation strategy, such as TrivialAug (Müller & Hutter, 2021) (a policy consisting of a random draw from a table of 14 common photo- and geometric transforms, applied in tandem with horizontal flips and random crops), shown in orange, is highly diverse, generating a large amount of effective extra samples when the number of base samples is small.
However, this policy also falls off quickest as more base samples are added: The policy is ultimately inconsistent with the underlying data distributions and when enough real samples are present, gains through augmentation deteriorate.
On the flip side, augmenting with only horizontal flips, shown in green, is less diverse, and hence yields limited impact for smaller dataset sizes.
However, the augmentation is much more consistent with the data distribution, and as such horizontal flips are beneficial even for large dataset sizes.
Takeaway :
The impact of augmentations is linked to dataset size; diverse but inconsistent augmentations provide large gains for smaller sizes but are hindrances at scale.
Estimation of the value of future data collection should take this effect of augmentations on scaling into account.
3.1. M EASURING D IVERSITY AND C ONSISTENCY DIRECTLY
We can disambiguate the effects of consistency and diversity further by analyzing these augmentations for a moment not as randomly applied augmentations, but as fixed enlargements of the dataset.
In
Figure 2 (right), we see that a replacement of each sample with an augmented sample generated by TrivialAug is detrimental to performance (see single repetition plotted in red).
As such, TrivialAug is inconsistent with the data distribution.
On the other hand, evaluating the gains realized by replicating each sample in the dataset a fixed number of times via the augmentation policy, we also see that TrivialAug leads to significantly more diversity in this controlled experiment, compared to random flips (Figure 2, left).
Another way to look at Figure 2 is again as a measurement of extra data.
Simply replacing each base sample by a single augmented sample is not a benefit (see single repetition plotted in red), yet we quickly find that large gains can be attributed simply to the duplication (green) and quadruplication (purple) of existing data.
[Figure]
[Figure]
E ff ec ti v e E x t r a
S a m p l e s fr o m
A ug .
[Figure] [Figure]
E ff ec ti v e E x t r a
S a m p l e s fr o m
A ug .
[Figure]
3.1. E XCHANGE R ATES FOR M ODEL S CALING AND M ODEL V ARIATIONS
Figure 3 evaluates the effective samples gained from augmentation as model width increases (left) and model architecture changes (right).
For both plots, the references f ref are based on the unaugmented models with that same configuration of width or architecture.
We find that model width of the evaluated ResNet-18 reliably correlates with the gains from augmentations to larger dataset sizes.
Evaluating different architectures, we find that while global behavior is similar for all models, the exact exchange rates are similar for the convolutional architectures of ResNet, PyramidNet (Han et al., 2017) and VGG (Simonyan & Zisserman, 2014), mirroring their closely related inductive biases.
On the other hand, from the two transformer architectures, we find that a notably limited benefit from augmentations versus real data for the Swin-Transformer (Liu et al., 2021b; 2022), but a large benefit for the ConvMixer (Trockman & Kolter, 2022).
In direct comparison of both architectures, the SwinTransformer contains a large number of features designed specifically for vision tasks, whereas ConvMixer is much more general, so that their differing inductive biases are again reflected in Figure 3.
Takeaway
Relative gains through augmentations as sample sizes scale are broadly consistent across model widths and architectures, and absolute gains increase with model capacity
. 3. E XCHANGE R ATES IN O UT - OF -D ISTRIBUTION S ETTINGS
A benefit of extra data generated from augmentations that is often underappreciated is illustrated in
Figure 4, showing exchange rates of the same models trained on CINIC-10 as analyzed in Figure 1, but evaluated on the CIFAR-10 validation set (left).
Comparing CINIC-10 and CIFAR-10, both datasets are nearly indistinguishable using simple summary statistics (Darlow et al., 2018), yet there is a minor distribution shift caused by different image processing protocols during dataset curation.
In this setting, diverse augmentations are now quickly on-par with models trained on many more samples.
Figure
50k 100k 150k 200k 250k 300k 50k [Figure] 150k 200k [Figure]
Random
[Figure]
E ff ec ti v e E x t r a
S a m p l e s fr o m
A ug .
2 3 4 5 6 7 89 10k 2 3 4 5 6 7 89 100k 2 3 −400k −300k −200k −100k 100k 200k 300k 400k Repetitions
Random
[Figure]
E ff ec ti v e E x t r a
S a m p l e s fr o m
A ug . shows that this effect is apparent, even if base samples are replaced by a fixed number of augmented samples.
Four augmented samples are quickly worth more than additional real samples, and with enough base samples, even replacing each sample by a fixed augmented version is beneficial.
These effects can be exaggerated by evaluating on the CIFAR-10-C dataset of common corruptions applied to CIFAR-10 (Hendrycks & Dietterich, 2018).
There, we quickly find that evaluating exchange rates for this large distribution shift, that the stronger augmentations evaluated, quickly produces benefits beyond what a collection containing even substantial amounts of in-domain data would achieve.
The increased diversity in these augmentations broadens the support of the data distribution, and the support of the CIFAR-10 dataset appears to be well contained within the transformed data.
In practical applications in domain adaptation (Zhang et al., 2019; Ahuja et al., 2021) and unsupervised learning (Chen et al., 2020b), actively broadening the support of the data distribution in the face of uncertainty is advantageous as suddenly each augmentation is effectively worth much more data.
Takeaway : Data augmentations broaden the support of training data, which significantly extends their usefulness even on larger dataset scales in OOD testing scenarios.
A UGMENTATIONS AND I NVARIANCE
[Figure]
The success of augmentations is often attributed to the invariances encoded into the model by enforcing the assignment of identical labels across transformations of each training sample.
If the success of data augmentations can be attributed solely to invariance, then we can build exactly invariant models that achieve comparable accuracy when trained without data augmentation.
Several works propose such mechanisms for constraining neural network layers to be invariant, and we will leverage these in our study.
4. I NVARIANT N EURAL N ETWORKS W ITHOUT A UGMENTATION
In order to probe the benefits of invariance without augmentations, we evaluate the following three methods for constructing invariant networks, for the case of invariance to horizontal flips:
Prediction averaging: Insert augmented views of a sample into the model and average the corresponding predictions (Simonyan & Zisserman, 2014).
We use this procedure during both training and inference , and refer to the approach applied to a ResNet as flip-invariant ResNet-18.
Orbit Selection: An invariance can also be enforced via orbit selection (Gandikota et al., 2022), which corresponds to a normalization of the invariance.
Here, an orbit mapping uniquely selects a
[Figure]
single element from the group of transformation before the data is passed into the model.
E2CNN: General E(2)-Equivariant Steerable CNN (E2CNN) (Weiler & Cesa, 2019) constrains convolutional kernels to reflect a group equivariance.
We instantiate an E2CNN with the same architecture as the ResNet-18 studied so far.
In Section 3, we saw that horizontal flips are consistent with the CIFAR-10 distribution, so it may not be surprising that horizontal flip invariant networks perform better than those trained with neither augmentations nor invariance constraints.
Moreover, networks trained with data augmentations outperform invariant models for small and medium sample sizes.
However, we observe in Figurethat all investigated invariant architectures (red, green, blue) catch up to models trained with random augmentations (in orange) as we increase the number of base samples.
We will see in Sectionthat data augmentations serve as flatness-seeking regularizers, and the benefits of this additional regularization wane as the number of samples increases.
4. O UT -O F -D ISTRIBUTION A UGMENTATIONS S TILL I MPROVE P ERFORMANCE
Previously, we observed the performance benefits of data augmentations which promote invariances consistent with the data distribution, or approximately so.
But can it still be useful to augment our data with a transformation that generates samples completely outside the support of the data distribution and which are inconsistent with any label?
To answer this question, we construct a synthetic dataset in which the exact invariances are known.
We begin by randomly sampling a single base image from each CIFAR-10 class.
We then construct 10 classes by continuously rotating each of the base images, so that all samples in a class correspond to rotations of a single image.
Thus, the classification task at hand is to determine which base image was rotated to form the test sample.
We randomly sample rotations from each of these classes to serve as training data and another disjoint set of rotations to serve as test data.
We then use horizontal flip and random crop data augmentations to generate out-of-distribution samples, since horizontally flipped or cropped image views cannot be formed merely via rotation.
Note that this experiment is distinct from typical co-variate shift setups where the distribution of data domains differs, but the support is far from disjoint and may even be identical.
In Figure 7, we see that these out-of-distribution augmentations are beneficial nonetheless.
Notably, random crops, which can generate significantly more unique views than horizontal flips, yield massive performance boosts for identifying rotated images, even though we know that the cropped samples are out-of-distribution.
We also see in this figure that random crops are especially useful if we instead use as our test set rotations of samples from CIFAR-10 that were not used for training.
Specifically, we assign a base test image and its rotations the same label as the base image from the training set with the same CIFAR-10 label.
This experiment supports the observations from Section 3 that augmentations can be particularly beneficial for OOD generalization.
Takeaway : Comparing invariant architectures to augmentations, we find that augmentations dominate on smaller scales, but invariant architectures catch up in the large-sample regime.
Augmentations can provide benefits even on apparently unrelated invariances, which is particularly helpful for OOD generalization.
[Table]
(cid:87)
[Table]
A UGMENTATIONS A S A S OURCE OF S TOCHASTICITY D URING T RAINING
Typical loss functions are summed over training samples.
During optimization, gradients are computed using small mini-batches of random samples, resulting in stochasticity.
Augmentations increase the number of available data, often so much that we never sample the same data twice.
Since data augmentations expand and diversify the training set, they may serve as additional sources of stochasticity during optimization.
If data augmentations do increase the variety of gradients, they could as a result cause us to find qualitatively different minima.
Stochastic optimization is thought to be associated with flat minima of the loss landscape which are in turn associated with superior generalization (Jastrz˛ebski et al., 2018; Huang et al., 2020; Liu et al., 2021a).
This flatness-seeking behavior may be the effect of both the augmented loss function and also how we sample it.
To put this hypothesis to the test, we measure the standard deviation of gradients during optimization for models trained with and without data augmentations, and we quantify the flatness of the corresponding minima.
We construct experiments that disentangle the augmented loss function from the additional stochasticity produced by sampling augmentations.
We consider a “same batch" strategy in which gradient updates are averaged over multiple views of a single image, resulting in lower stochasticity.
We also consider “fixed views" where we repeat a frozen set of augmented views per element of the training set, as in Figure 2.
5. M EASURING STOCHASTICITY
To measure stochasticity, we train a model on a given training set and augmentation strategy, and we freeze the model every 10 epochs to estimate the standard deviation (formally the norm of parameterwise standard deviations) of its gradients over randomly sampled batches comprising 128 base images, the same batch size used during training.
That is, we measure the square root of the average squared distance between a randomly sampled batch gradient and the mean gradient.
We adopt a filter-normalized distance function (Li et al., 2018; Huang et al., 2020) to account for invariances in neural networks whereby shrinking the parameters in convolutional filters may not effect the network’s output but may make the model more sensitive to parameter perturbations of a fixed size.
In Figure 8, we see that non-augmented datasets actually yield noisier gradients early in training, but this noise vanishes rapidly as over-fitting occurs.
In contrast, randomly applied data augmentations result in flatter curves, indicating that the added diversity of views available for sampling preserves stochasticity later in training.
We also see that for each augmentation policy, applying augmentations randomly results in the most stochasticity late in training, while including multiple random views in the same batch (Hoffer et al., 2020; Fort et al., 2022) results in less.
Sampling augmentations from a fixed set of four views per sample (denoted “fixed views”) results in even less stochasticity, and including each of the four views in every batch results in the least stochasticity (denoted “fixed vews”, “same batch”).
This ordering, which holds across all data augmentations we try, is consistent with the intuition that more randomness in augmentation leads to more stochasticity in training, notably only manifesting during later epochs.
We will now see that the late-training stochasticity we measure correlates strongly with the flatness of the minima these optimization procedures find.
5. M EASURING F LATNESS
We adopt the flatness measurements from Huang et al.
(2020) as these measurements are non-local, do not require Hessian computations which are dubious for non-smooth ReLU networks, and they are consistent with our filter-normalized gradient standard deviation measurements.
Specifically, we measure the average filter-normalized distance in random directions from the trained model parameters before we reach a loss function value of 1 . 0 , where loss is evaluated on the non-augmented dataset.
Under this metric, larger values correspond to flatter minima as parameters can be perturbed further without increasing loss.
We use the same ResNet-18 models trained in the stochasticity experiments above with the same exact augmentation setups.
Investigating Figure 8 and Figure 9, (see also Table 4), we observe that flatness correlates strongly with late-training stochasticity.
Models trained without augmentation or with non-random augmentation, where all views are seen in each batch, are less stochastic at the end of training and find sharper minima.
While previous works have associated SGD with flatness-seeking behavior (Jastrz˛ebski et al., 2018; Geiping et al., 2021), data augmentations appear to contribute to this phenomenon.
Simply put, training with randomized data augmentations finds flatter minima, and models trained with strong data augmentations lie at especially flat minima.
Takeaway : Randomly applied augmentations provide benefits beyond invariance by flattening the loss landscape, which is reflected in both measurements of flatness after training and measurements of gradient noise late in the training.
5. D ATASET SCALING AND F LATNESS
[Figure]
Figure 9 directly measures flatness for several dataset scales.
We first notice that base models become flatter (with respect to their base samples) as the number of samples increases.
Surprisingly, stronger augmentations can produce this effect quicker and raise flatness values even for lower sample sizes.
As a notable example, TrivialAug produces models that remain relatively flat for all sample sizes considered.
Furthermore, all augmentations converge to similar flatness in the sample size limit, as regularization becomes less relevant in the large data regime.
Over all plots we can even correlate the number of samples gained through augmentations and flatness and find that for all weaker augmentations, flatness of the solution is strongly correlated with the number of extra samples that are gained from the augmentation.
Takeaway :
Strong data augmentations flatten the loss landscape to levels otherwise only reached with significantly larger datasets.
C ONCLUSION
Data augmentations have a profound impact on the performance of neural networks, but their precise role has not been well understood; for example, if augmentations are simply a heuristic for learning certain symmetries, should we not prefer to directly encode these symmetries through advances in group equivariant networks?
Through the lens of exchange rates and power laws, we observe the gains through augmentations as datasets scale and domains change.
We find that augmentations dominating invariant architectures on smaller scales, but, scale in opposite ways.
Augmentations are further distinguished from invariances in the way they can improve performance even out-of-distribution.
Ultimately we find that we can connect these findings to the regularization effect induced by data augmentations, which we also measure, showing how augmentations flatten the loss landscape.
This work promotes an all-encompassing understanding of neural network training, shedding light on the nuanced but significant role of data augmentation in the success of deep learning.
E THICS S TATEMENT
We foresee no direct negative societal consequences from this work.
We do think that data augmentations are beneficial, especially in applications with only limited data, or where data curation is expensive.
We argue that knowing how to exchange a smaller (but verified and curated) dataset for a larger dataset that is not augmented, but also due to its size less curated is hopefully helpful to the community.
R EPRODUCIBILITY S TATEMENT
We use an academic cluster with NVIDIA RTXA4000 cards and also NVIDIA GTX2080ti cards.
Each job is scheduled on a single GPU and the default setting of 60000 gradient steps takes roughly an hour to train and evaluate.
Including all preliminary experiments we estimate a total usage of about 400 GPU days for this project.
To replicate all experiments in the main body without repeated trials, we estimate a requirement of about 10 GPU days.
We provide code with the supplementary material to do so.
A E XPERIMENTAL S ETUP
For all sections if not otherwise mentioned, we run the following protocol.
We train the model (in the main body a ResNet-18), with stochastic gradient descent for 60000 steps with a batch size of 128.
This corresponds to 160 epochs for a dataset of size 48000 .
We keep this number of gradient steps fixed when increasing or decreasing the dataset size.
We linearly warm up the learning rate for the first 2000 steps (about 5 epochs) up to peak rate of 0 .
and then decay to zero by a half-cycle of cosine annealing.
For all experiments, we include a standard weight decay of 5 e − 4 and train with Nesterov momentum of 0 .
The data is shuffled randomly after every epoch and we record validation accuracy every 10000 steps.
All training runs are non-deterministic based on stochasticity due to random shuffling and cudnn non-determinism.
We run at least five trials for each experiment in the main body and three trials for each in the supplementary material.
In each plot, the standard deviation is shaded.
For five trials this corresponds close to a 97 .
5% confidence interval. We use CIFAR-10 in its default configuration.
For CINIC-10, we clean and resample the train and test sets.
We first remove all CIFAR-10 train and test images from the dataset, we then further remove all exactly duplicated images and missing images, merge all remaining images and sample a new validation set of 10000 images.
We provide code to replicate the creation of this cleaned dataset with the supplementary material.
Overall we recover a new training set of size 193523.
For
CIFAR-10-C experiments in the supp.
material, we report average accuracy over all transformations in CIFAR-10-C with a severity of 3.
For all experiments where we consider only a subset of the existing data (e.g. each experiment with less than 193523 samples for CINIC-10), we sample a new subset of the training set for each experiment separately, to rule out confounding effects of good or bad splits of the training data, especially for smaller subset sizes.
This new test set for CINIC-10 turn out to be harder than the CIFAR-10 test set, but we verify that the hyperparameters discussed above would result in more than 95% accuracy when training with CIFAR training data.
For experiments in the main body where data augmentations are randomly sampled a finite number of times, we store all augmentations in a database ( lmdb ) that is recreated in each run.
As a result, each experiment contains a fixed set of finite views of each original datapoint, but these views are randomized across experiments.
Due to random shuffling, samples from this enlarged dataset are drawn randomly and multiple views of the image are only guaranteed to occur in the same batch in the batch augmentation experiments in Section 5.
To compute measurements of exchange rates, we first compute the mean validation accuracy
CINIC-10 for each experiment.
We then train the reference models for CINIC-10 subset sizes of 1000 , 2000 , 3000 , 6000 , 12000 , 24000 , 48000 , 96000 , 128000 , 144000 , 168000 , 180000 , 192000 .
To estimate parameters a, b, c for f p ( x ) = ax − c + b in the exchange rate plots in all sections we use a non-linear least-squares algorithm, initialized from starting parameters that describe the curve for no augmentations.
For this we use the Levenberg-Marquardt implementation of MINPACK , as wrapped in scipy .
To cross-reference the average validation accuracy of these reference models with our data augmentation experiments in Table 1, we assert that validation accuracies are monotonically increasing as subset sizes increase and fit a linear spline f ref for interpolation.
We then compute the exchange ratios of Table 1 via f − 1 ref ( x ) /b , for the base dataset size b which is 48000 in Table 1 and input mean validation accuracy x for each experiment.
For values outside the interval spanned by the minimal and maximal validation accuracies of the reference data, we reuse the power laws of the form f p ( x ) =
ax − c + b described in Sec.
3.3 and again compute f − 1 p ( x ) /b .
We mark these extrapolated values by a ∗ in the table.
The ResNet-18 model employed in the model is a modern variant (He et al., 2016; 2019a) and contains the usual CIFAR-10 stem consisting of a single 3 × 3 convolutional layer without pooling, instead of the ImageNet stem (of two convolutional layers with stride and max-pooling).
For experiments in the supplementary material, we further consider a ResNet-8 (i.e. three stages and a single block per stage)
(He et al., 2016), a VGG-11 (Simonyan & Zisserman, 2014) with batch normalization, and a ConvMixer architecture (Trockman & Kolter, 2022) of depth 8 with hidden dimension 128 and spatial kernel size of 7.
We implement and run all experiments in PyTorch and make use of torchvision implementations for a range of data augmentations investigated in this work.
We provide code to replicate all experiments with the supplementary material.
A. H YPERPARAMETERS FOR A UGMENTATIONS
For each augmentation we broadly follow established defaults.
For completion we record these, and additional details here.
Horiz.
Flips:
A data point is flipped horizontally with probability 0.5.
Det.
Horiz.
Flips:
Deterministic horizontal flips.
For 1 × , this corresponds to flipping every data point.
2 × corresponds to both flips being contained in the dataset.
Vert.
Flips:
A data point is flipped vertically with probability 0.5.
Det.
Vert.
Flips:
Deterministic vertical flips.
For 1 × , this corresponds to flipping every data point.
2 × corresponds to both flips being contained in the dataset.
Random Crops: The image is padded by with zero-padding by 4 pixels in each direction and then a image of size 32 × 32 is cropped (This is classical random cropping for CIFAR-10).
Flips&Crops: Both random crops and horizontal flips are employed, as described above.
Perspectives: Performs a random perspective transform with probability 0.5 with bilinear resampling.
Jitter: Color jitter, randomly transforming contrast, hue and brightness of the image.
For each distortion, sample a new scale uniformly from [0 . 5 , 1 . 5] .
Blur: Blurs the image with a Gaussian blur with σ = 3 .
AutoAug: Employ the augmentation policy of Cubuk et al. (2019), with the CIFAR-10 policy.
AugMix: The augmentation policy of Hendrycks et al.
RandAug:
The augmentation policy of Cubuk et al.
(2020), again with the CIFAR-10 policy.
TrivialAug: The augmentation policy of Müller & Hutter (2021) in its “wide" configuration.
AutoAug&Flips&Crops: The AutoAug policy followed by random horizontal flips and random crops as described above.
AugMix&Flips&Crops: The Augmix policy followed by random horizontal flips and random crops as described above.
RandAug&Flips&Crops: The RandAug policy followed by random horizontal flips and random crops as described above.
TrivialAug&Flips&Crops: The TrivialAug policy followed by random horizontal flips and random crops as described above.
A. D ATA L ICENSING
We investigate, MNIST (LeCun et al., 1998), CIFAR-10 and CIFAR-100 (Krizhevsky, 2009), EMNIST (Cohen et al., 2017), CIFAR10-C (Hendrycks & Dietterich, 2018) and CINIC-10 (Darlow et al., 2018) and refer to these publications for additional details.
We remove duplicates and missing data from CINIC-10 as described in the experimental setup.
[Figure]
B A DDITIONAL E XPERIMENTAL R ESULTS
We include additional material for the experimental section in a series of figures and tables.
Table 1 is an extended table of the exchange rates for CINIC-10 for a sample size of 48000, including repetitions up to 32 × and ablating the number of steps (This is a slice of Figure 1).
Behavior is consistent over additional repetitions, so we chose not to include these additional rows in the main body.
This table further includes additional data augmentations not featured in the main body, over which behavior is [Figure] [Figure] consistent.
Table 2 and Table 3 are then variants of this table where validation accuracy is evaluated on CIFAR-10 and CIFAR-10-C, respectively.
CIFAR-10-C is a significant distribution shift that cannot be mitigated by additional CINIC-10 data, only training on, e.g. blurred samples, provides robustness to this distortion.
We further find that training with horizontal flips in our experimental setup is quickly disadvantageous.
To provide additional clarification for Figure 2, we also provide slices through this plot at sample sizes of 24000, 48000, 96000 and 192000 in Figure 10 for CINIC validation data, Figure 11 for
CIFAR-10 and Figure 12 for CIFAR-10-C data.
Further, Figure 13 shows the data points underlying
Figure 3 for additional clarification.
An additional table comparing the end-of-training stochasticity in Figure 8 and flatness measurements in Figure 9 for ease of references is Table 4.
C A DDITIONAL D ATASETS AND M ODELS
We further verify that the findings discussed in the main body are not limited to the choice of dataset and model therein and provide additional reproduction aside from the investigation into model architectures in Figure 3.
We repeat the tabular representation of exchange rates and a simplfied form of Figure 11 for a tiny ResNet-8 in Figure 14 and Table 5, a VGG-11 in Figure 15 and Table 6 and a
ConvMixer (as representative of modern ConvNet/Transformer variations) in Figure 16 and Table 7.
We then further repeat these experiments with models trained on the MNIST training set in Figureand Table 8, CIFAR-100 in Figure 18 and Table 9, as well as EMNIST in Figure 19 and Table 10.
We further include repeated experiments for Sec. 5 on CIFAR-100 in Figure 20 and Table 11.
[Figure]
[Table]
1.00 Horiz.
Flips 0.
- Det.
Horiz.
Flips 0.
- - - - - - Vert.
Flips 0.
1.20 Det.
Vert.
Flips 0.
- - - - - - Random Crops 0.
2.74 Flips&Crops 0.
- Perspectives 0.
Jitter 0.
0.88 Blur 0.
AutoAug 0.
1.77 AugMix 0.
1.13 RandAug 0.
- TrivialAug 0.
AutoAug&Flips&Crops 0.
4.00* 4.00* 4.00* 4.08* Augmix&Flips&Crops 0.
[Table]
[Table] [Table]
[Table] [Table]
[Table]
0.11 Det
[Table]
[Table]
- - - - - Random Crops 0.
[Table] [Table]
[Table]
[Table] [Table]
67 Flips&Crops 0.
[Table]
[Table]
[Table]
0.25 Perspectives 34.16* ✓ ✓ ✓ ✓ ✓ ✓
Jitter
[Table]
81* 190.76* ✓ 16.99* 7.26* 3.
163.08* Blur ✓ ✓ ✓ ✓ ✓ ✓ ✓
AutoAug ✓ ✓ ✓ ✓ ✓ ✓ ✓
AugMix ✓ ✓ ✓ ✓ ✓ ✓ ✓
RandAug ✓ ✓ ✓ ✓ ✓ ✓ ✓
TrivialAug ✓ ✓ ✓ ✓ ✓ ✓ ✓
AutoAug&Flips&Crops ✓ ✓ ✓ ✓ ✓ ✓ ✓
AugMix&Flips&Crops ✓ ✓ ✓ ✓ ✓ ✓ ✓
RandAug&Flips&Crops 91.64* ✓ ✓ ✓ ✓ ✓ ✓
TrivialAug&Flips&Crops ✓ ✓ ✓ ✓ ✓ ✓ ✓
[Table]
[Table]
- - - - - - Vert.
Flips 0.
1.09 Det.
Vert.
Flips 0.
- - - - - - Random Crops 0.
- 2.62 Flips&Crops 0.
3.83 Perspectives 0.
Jitter 1.
1.21 Blur 0.
AutoAug 0.
1.80 AugMix 1.
1.67 RandAug 0.
2.68 TrivialAug 0.
AutoAug&Flips&Crops 0.
3.00 AugMix&Flips&Crops 0.
3.22 RandAug&Flips&Crops 0.
[Table]
TrivialAug
[Table]
Crops
[Table] [Figure] [Table] [Table]
1.00 Horiz.
Flips 1.
1.86 Det.
Horiz.
Flips 0.
- - - - - - Vert.
Flips 0.
0.97 Det.
Vert.
Flips 0.
- - - - - - Random Crops 0.
- 2.24 Flips&Crops 0.
3.15 Perspectives 0.
Jitter 0.
0.82 Blur 0.
AutoAug 0.
1.51 AugMix 0.
1.00 RandAug 0.
2.19 TrivialAug 0.
AutoAug&Flips&Crops 0.
4.00* 4.00* AugMix&Flips&Crops 0.
[Table]
[Figure]
[Table]
1.66 Det.
Horiz.
Flips 1.
- - - - - - Vert.
Flips 0.
0.49 Det.
Vert.
Flips 0.
- - - - - - Random Crops 0.
1.26 Flips&Crops 0.
1.72 Perspectives 0.
Jitter 0.
1.26 Blur 0.
AutoAug 0.
2.07 AugMix 0.
1.70 RandAug 0.
2.28 TrivialAug 0.
AutoAug&Flips&Crops 0.
2.91 AugMix&Flips&Crops 0.
2.20 RandAug& [Table] &Crops [Table]
TrivialAug
[Table]
Crops
[Table]
[Figure]
I
[Figure]
I C
[Figure]
V a l .
A cc
[Figure]
r ac y
[Figure]
[Table]
Augmentation
[Table]
8x 16x 32x rand (160) rand (640) - 1.
1.00 Horiz.
Flips 0.
0.25 Det.
Horiz.
Flips 0.02* - - - - - - - Vert.
Flips 0.
0.22 Det.
Vert.
Flips 0.02* - - - - - - - Random Crops 1.
✓ 1.
✓ ✓ - 37.32* Flips&Crops 0.
0.23 Perspectives 1.
✓ 1.
Jitter 1.
1.10 Blur 1.
AutoAug 0.
4.38* 7.84* 7.84* ✓
AugMix 1.
1.15 RandAug 1.
4.38* ✓ ✓ ✓ 1.
✓
TrivialAug 0. 4.38* ✓ 1. ✓ AutoAug&Flips&Crops 0.
0.26 AugMix&Flips&Crops 0.
0.24 RandAug&Flips&Crops 0.
0.23 TrivialAug&Flips&Crops 0.
[Figure]
[Table] [Figure] &Flips&Crops [Figure]
[Figure]
[Figure] [Figure] [Figure]
[Table]
AutoAug 1. 13.79* 2.03 AugMix 1. 1.99 RandAug 1. ✓ ✓ ✓ ✓
TrivialAug 0. ✓ ✓ AutoAug&Flips&Crops 0. 0.25 AugMix&Flips&Crops 0.
0.16 RandAug&Flips&Crops 0. 0.21 TrivialAug&Flips&Crops 0.
[Figure]
Augmentation [Table]
No Augmentation - - 46.
[Table]
Horiz.
Flip & Rand.
Crop
No
No 46.
Yes
No 42.
[Figure]
t S t a nd a r d D e v .
D A LTERNATIVE P OWER L AWS
In the main body, we propose to investigate power laws of the form f ( x ) =
ax − c + b and fit these to measured data to estimate scaling rates for accuracy with respect to number of samples.
We believe this choice to be sound in principle, as over all variants, e.g. in Figure 1 (left), these curves fit experimental results well.
In this section, though, we want to validate this choice and discuss possible alternatives.
D. E QUATIONS SUGGESTED VIA S YMBOLIC R EGRESSION
To potentially discover alternative functional forms we turn to symbolic regression using machine learning scoring rules (Cranmer et al., 2020) using the implementation of Cranmer (2022).
We search for symbolic expressions containing elementary operations and exponential functions with a symbolic complexity less or equal than 10.
For exponentiation, we limit complexity in the exponent to 1.
We search for 24000 iterations and then extract the functional form of the equation f , discarding the constants obtained during discovery, and use standard nonlinear regression, as described previously to fit constants for each curve f and f aug .
We symbolically invert the found equation f and compute f − 1 ( f aug )( x ) − x as described in Section 3.
To run a representative example with higher complexity, we re-evaluate Figure 1 and discover a new functional form as described above using data from baseline experiments without augmentation.
{+We first note that we refind a result close to our original hypothesis with f sym ( x ) = x 0 . 
+}
− 1 .
2693417 , at complexity 5.
Yet, we find the following function with complexity 10 using symbolic regression:
[Equation]
i
e. the functional form
[Equation]
We plot results for a re-analysis of Figure 1 with this form in Figure 21.
Unsurprisingly, this is a better fit for the baseline without augmentations, data on which this function was found.
Interestingly though, this does not lead to significant qualitative changes.
The perspective transform is valued differently based on this new fit of the baseline, but trends are broadly consistent.
If we take this to the extreme, and search for a functional form specifically without atoms used previously (containing now only additions, subtractions, divisions and multiplications), we find
[Equation]
from which we fit
[Equation]
[Figure]
[Figure]
which we also use for an exemplary reanalysis and show in Figure 22.
This agains fits the baseline very well, but for example fits the result with flips and crops less well than Figure 1.
We could ultimately also search for functions with higher complexity.
For example, this is a symbolic regression result with larger complexity:
[Equation]
from which we extract
[Equation]
and which we visualize in Figure 23.
Here, we note the close relationship to the previously found result at complexity 10, and similar conclusions hold.
D. A LTERNATIVE S UGGESTIONS
Alternatively
[Equation]
we know that the functional form f ( x ) =
ax − c + b can only locally describe the relationship between accuracy and data samples, as the resulting function is potentially unbounded above for some choices of ( a, b, c ) , yet accuracy is strictly bounded by 1 . 0 .
As, such, we might wonder about fitting a globally accurate form, such as f ( x ) = a tanh ( x − c ) + b , which is bounded.
We include a re-interpretation with this form in Figure 24, but again find no broad qualitative changes.
[Figure]
[Figure]
[Figure]
E E XPERIMENTAL A BLATIONS
To validate the experimental setup executed in the main body of this work, we run a series of comparative studies.
First, we construct a CINIC-10 setup in this work with a novel, sanitized test set that makes it hard to compare validation accuracies to baseline CIFAR-10 and CINIC-10.
As such, we verify our implementations of all architectures and chosen budget and training setup in
Table 12, where we find these implementations to perform as expected, and reach high performance on CIFAR-10.
For reference, we also record the number of parameters for each investigated model here.
Secondly, we vary the budget used in our study.
In all other experiments in this work we consider a budget of 60000 steps (which is 160 epochs at batch size 128 and a dataset size of 48000 ).
In
Figure 25, we check, for the case of TrivialAug, how the results of Figure 1 vary, as the budget is varied.
There, we find that qualitative behavior is stable across all considered budget variations, but quantitative results depend on the chosen budget.
Third, we switch from validation accuracy after training to peak validation accuracy observed during training in Figure 26.
Peak validation accuracy during training as an oracle would return earlystopping results, if the investigated models would overfit to their training data.
However, we find in Figure 26 that behavior and analysis are almost indistinguishable under this metric, validating the choice of final validation accuracy without early stopping and the choice of fixed budgets
[Table]
[Figure]
We further include additional model architectures in Figure 27, validating that model family is a much stronger predictor of qualitative behavior than model size, by including additional results for the larger ResNet-110, ResNet-152, and the smaller ResNet-8 and ResNet-20.
[Figure]
P ea k C I N I C10 V a l . A cc u r ac y
[Figure]
[Figure]
[Figure]
I m a g e N e t V a l .
A cc u r ac y
[Figure]
F I MAGE N ET R ESULTS
We also investigate how well these power laws would fit ImageNet training runs (Russakovsky et al., 2015).
ImageNet is not an ideal setting for our analysis, given that the dataset is relatively small compared to its complexity.
Given unlimited compute one would rather train reference models on larger splits, e
[Figure]
g. ImageNet-21k or JFT-300m (Sun et al., 2017) and compare the validation accuracy of those reference models to ImageNet models trained with augmentations, as discussed in Section
[Figure]
To train these models we follow the recently proposed training regime of Wightman et al.
We train for 65536 steps (about 100 epochs) using the LAMB optimizer with a peak learning rate of 8e − 3 and cosine decay after a warmup of 3125 steps.
We additionally apply label smoothing with 0 .
We train a ResNet-50 model as described in He et al. (2019a).
As unaugmented baseline, we consider a pre-processing of centered crops of size 224 .
For the augmented variant, we augment with random resized crops (with ratios between 3 / 4 and 4 / 3 as usual) of size 224 and random horizontal flips.
For TrivialAug, we use TrivialAug as described on top of these random resized crops and flips.
In all cases, we validate by resizing to 256 × 256 and center cropping to 224 × 224 .