Data-Ef ﬁ cient Augmentation for Training Neural Networks
Anonymous Author(s)
Af ﬁ liation Address email
Abstract
Data augmentation is essential to achieve state-of-the-art performance in manydeep learning applications.
However, the most effective augmentation techniquesbecome computationally prohibitive for even medium-sized datasets.
To addressthis, we propose a rigorous technique to select subsets of data points that whenaugmented, closely capture the training dynamics of full data augmentation.
Weﬁ rst show that data augmentation, modeled as additive perturbations, improveslearning and generalization by relatively enlarging and perturbing the smallersingular values of the network Jacobian, while preserving its prominent directions.
Then, we propose a framework to iteratively extract small subsets of trainingdata that when augmented, closely capture the alignment of the fully augmented
Jacobian with labels/residuals.
We prove that stochastic gradient descent applied toaugmented subsets found by our approach have similar training dynamics to that offully augmented data.
Our experiments demonstrate that our method outperformsstate-of-the-art by 7 . 7% on CIFAR10 with 6.3x speedup and 4 .
7% on SVHNwith 2.2x speedup, using 10% and 30% augmented subsets respectively.
Besides,augmenting 10% and 30% subsets from our method beats random baselines by7 .
9% and 5 .
3% on TinyImageNet, and by 7 . 6% and 2 .
3% on ImageNet.
Introduction
Standard (weak) data augmentation transforms the training examples with e g. rotations or crops forimages and trains on the transformed examples in place of the original training data.
While weakaugmentation is effective and computationally inexpensive, strong data augmentation (in additionto weak augmentation) is a key component in achieving nearly all state-of-the-art results in deeplearning applications [34].
However, the most effective strong data augmentation techniques increasethe training time by orders of magnitude.
First, they often have a very expensive pipeline togenerate transformations that best improves generalization [5, 16, 23, 39].
Second, appendingtransformed examples to the training data is often much more effective than training only on the(strongly or weakly) transformed examples in-place of the original data.
Importantly, appendingone transformed example is often much more effective than training on two transformed examplesin place of every original training data, which has the same computational cost as that of training onoriginal images appended with only one transformation per image ( c.f. Appendix D.5).
Hence, toobtain the state-of-the-art performance, multiple augmented examples are added for every singledata point and to each training iteration [15, 39].
In this case, even if producing transformations arecheap, such methods increases the size of the training data by orders of magnitude.
As a result, stateof-the-art data augmentation techniques become computationally prohibitive for even medium-sizedreal-world problems.
For example, the state-of-the-art augmentation of [39] increases training timeof ResNet20 on CIFAR10 by 13x on an Nvidia A40 GPU.
To make state-of-the-art data augmentation more ef ﬁ cient and scalable, an effective approach is tocarefully select a small subset of the training data such that augmenting only the subset providessimilar training dynamics to that of full data augmentation.
If such a subset can be quickly found,it would directly lead to a signi ﬁ cant reduction in storage and training costs.
First, while standardin-place augmentation can be applied to the entire data, the strong and expensive transformationscan be only produced for the examples in the subset.
Besides, only the transformed elements of thesubset can be appended to the training data.
In addition, when the data is larger than the trainingbudget, one can train on random subsets (with standard in-place augmentation) and augment coresets(by strong augmentation and/or appending transformations) to achieve a superior performance.
Despite the ef ﬁ ciency and scalability that it can provide, this direction has remained largely unexplored.
Existing studies are limited to fully training a network and subsampling data points basedon its loss or in ﬂ uence for augmentation in subsequent training runs [21].
However, this method isprohibitive for large datasets, provides a marginal improvement over augmenting random subsets,and does not provide any theoretical guarantee for the performance of the network trained on theaugmented subsets.
Besides, when the data contains mislabeled examples, augmentation methodsthat append transformations to the data (e.g. by max loss) degrade the performance by selecting andappending several noisy labels.
A major challenge in ﬁ nding the most effective data points for augmentation is to theoretically understand how data augmentation affects the optimization and generalization of neural networks.
Existingtheoretical results are mainly limited to simple linear classi ﬁ ers and analyze data augmentation asenlarging the span of the training data [39], providing a regularization effect [4, 10, 36, 39], enlargingthe margin of a linear classi ﬁ er [32], or having a variance reduction effect [7].
However, such toolsdo not provide insights on the effect of data augmentation on training deep neural networks.
Here, we study the effect of label invariant data augmentation on training dynamics of overparameterized neural networks.
Theoretically, we model data augmentation by bounded additive perturbations[32], and analyze its effect on neural network Jacobian matrix containing all its ﬁ rst-order partialderivatives [1].
We show that label invariant additive data augmentation proportionally enlarges butmore importantly perturbs the singular values of the Jacobian, particularly smaller ones, while maintaining its prominent directions.
In doing so, data augmentation regularizes training with bounded butvarying perturbations to the gradients and prevents over ﬁ tting.
Empirically, we show that our analysisholds for various strong augmentations, e.g., AutoAugment [8], CutOut [11], and AugMix [15].
Next, we develop a rigorous method to iteratively ﬁ nd small weighted subsets (coresets) that whenaugmented, closely capture the alignment between the Jacobian of the full augmented data with thelabel/residual vector.
We show that the most effective subsets for data augmentation are the set ofexamples with the most centrally located gradients, and can be obtained by maximizing a submodularfunction.
Such subsets can be ef ﬁ ciently extracted using a fast greedy algorithm which operates onsmall dimensional gradient proxies, with only a small additional cost.
We prove that augmenting thecoresets guarantees similar training dynamics to that of full data augmentation.
We also show thataugmenting our coresets achieve a superior accuracy in presence of noisy labeled examples.
We demonstrate the effectiveness of our approach applied to CIFAR10 (ResNet20,
WideResNet-28-10), CIFAR10-IB (ResNet32), SVHN (ResNet32), noisy-CIFAR10 (ResNet20),
Caltech256 (ResNet18,ResNet50), TinyImageNet (ResNet50), and ImageNet (ResNet50) comparedto random and max-loss baselines [21].
We show the effectiveness of our approach (in presence ofstandard augmentation) in the following cases:
• When producing augmentations is expensive and/or they are appended to the training data:we show that for the augmentation method of [39] applied to CIFAR10/ResNet20 it is 3.43x fasterto train on the whole dataset and augment our coresets of size 30% than to train and augment thewhole dataset.
At the same time we achieve 75% of the accuracy improvement of training on andaugmenting the full data with [39].• When data is larger than the training budget: we show that we can achieve 71.99% testaccuracy on ResNet50/ImageNet when training on and augmenting only 30% subsets for 90epochs.
Compared to AutoAugment [8] despite using only 30% subsets, we achieve 86% ofthe original reported accuracy while boasting 5x speed-up in training time.
Similarly, onCaltech256/ResNet18, training on and augmenting 10% coresets with [8] yield 65.4% accuracy,improving over random 10% subsets by 5.8% and over weak augmentation only by 17.4%.
• When data contains mislabeled examples: We show that training on and strongly augmenting50% subsets using our method on CIFAR-10 with 50% noisy labels achieves 76.20% test accuracy.
This actually improves performance compared to training on and strongly augmenting the full data.
Additional Related Work
Strong data augmentation methods achieve state-of-the-art performance by ﬁ nding the set of transformations for every example that best improves the performance.
Methods like AutoAugment [8],
RandAugment [9], and Faster RandAugment [9] search over a (possibly large) space of transformations to ﬁ nd sequences of transformations that best improves generalization [8, 9, 25, 39].
Othertechniques involve a very expensive pipeline for generating the transformations.
For example, someuse Generative Adversarial Networks to directly learn new transformations [2, 25, 28, 33].
Strongaugmentations like Smart Augmentation [23], Neural Style Transfer-based [16], and GAN-based augmentations [5] require an expensive forward pass through a deep network for input transformations.
For example, [16] increases training time by 2.8x for training ResNet18 on Caltech256.
Strong data augmentation methods either replace the original example by its transformed version,or add the generated transformations to the training data.
Crucially, augmenting the training datawith transformations is much more effective in improving the generalization performance.
Hence,the most effective data augmentation methods such as that of [39] and AugMix [15] append thetransformed examples to the training data.
In Appendix D.5, we show that even for cheaper strongaugmentation methods such as AutoAugment [8], while replacing the original training examples withtransformations may decrease the performance, appending the augmentations signi ﬁ cantly improvesthe performance.
Appending the training data with augmentations, however, increase the trainingtime by orders of magnitude.
For example, AugMix [15] that outperforms AutoAugment increasesthe training time by at least 3x by appending extra augmented examples, and [39] increases trainingtime by 13x due to appending and forwarding additional augmented examples through the model.
Problem Formulation
We begin by formally describing the problem of learning from augmented data.
Consider a dataset
D train =
( X train , y train ) , where X train = ( x 1 , · · · , x n
) ∈ R d × n is the set of n normalized datapoints x i ∈ [0 , 1] d , from the index set V , and y train =
( y 1 , · · · , y n ) ∈ { y ∈ { ν 1 , ν 2 , · · · , ν C }} with{ ν j } Cj =1 ∈ [0 , 1] .
Following [32] we model data augmentation as an arbitrary bounded additive perturbation � , with � � �≤ � 0 .
For a given � 0 and the set of all possible transformations A , we study the transformations selected from S ⊆
A satisfying S =
{ T i ∈
A | � T i ( x ) − x [-(cid:107)≤ (cid:15) 
-]�≤ � 0 ∀ x ∈ X train } .
While 123 the additive perturbation model cannot represent all augmentations, most real-world augmentationsare bounded to preserve the regularities of natural images.
Thus, we see the effects of additiveaugmentation on the singular spectrum holds even under real-world augmentation settings (Fig. 3).
This model is indeed limited when applied to augmentations that cannot be reduced to perturbations,such as horizontal/vertical ﬂ ips and large translations.
We extend our theoretical analysis toaugmentations modeled as arbitrary linear transforms (e.g. as mentioned, horizontal ﬂ ips) in B.5.
The set of augmentations at iteration t generating r augmented examples per data point can bespeci ﬁ ed, with abuse of notation, as D taug = { � ri [-=
-]=1 ( T ti ( X train ) , y train ) } , where |D taug | = rn and
T ti ( X train ) transforms all the training data points with the set of transformations T ti ⊂ S at iteration 132 t .
We denote X taug = { � ri =1 T ti ( X train ) } and y taug = { � ri =1 y train } .
Let f ( W , x ) be an arbitrary neural network with m vectorized (trainable) parameters W ∈ R m .
Weassume that the network is trained using (stochastic) gradient descent with learning rate η to minimizethe squared loss L over the original and augmented training examples D t =
{D train ∪ D taug }with associated index set V t , at every iteration t .
I.e., L ( W t , X ) :
=� i ∈ V t L i ( W t , x i ) :
= � ( x i ,y i )
∈ D t � f
( W t , x i )
− y i � 22 .
The gradient update at iteration t is given by
[Equation]
where X t = { X train ∪ X taug } and y t = { y train ∪ y taug } are the set of original and augmentedexamples and their labels, J ( W , X ) ∈ R n × m is the Jacobian matrix associated with f , andr t = f ( W t , X ) − y is the residual.
We further assume that J is smooth with Lipschitz constant
L .
I.e., � J ( W , x i ) − J ( W , x j ) �≤ L � x
i − x j � ∀ x i , x j ∈ X .
Thus, for any transformation
T j ∈ S , we have � J ( W , x i ) − J ( W , T j ( x i ))
�≤ L � 0 .
Denoting J
= J ( W , X train ) and ˜ J
=
J ( W ,T j ( X train )) ,we get ˜ J
= J + E , where E is the perturbation matrix with � E � 2 ≤� E � F ≤√ nL � 0 .
Data Augmentation Improves Learning
In this section, we analyze the effect of data augmentation on training dynamics of neural networks,and show that data augmentation can provably prevent over ﬁ tting.
To do so, we leverage the recentresults that characterize the training dynamics based on properties of neural network Jacobian and thecorresponding Neural Tangent Kernel (NTK) [17] de ﬁ ned as Θ =
J ( W , X ) J ( W , X ) T .
Formally:
[Equation]
where Θ = U Λ U T = � i =
λ i u
i
u
Ti is the eigendecomposition of the NTK [1].
Although theconstant NTK assumption holds only in the in ﬁ nite width limit, [22] found close empirical agreementbetween the NTK dynamics and the true dynamics for wide but practical networks, such as wide
ResNet architectures [40].
Eq.
(2) shows that training dynamics depend on the alignment of the
NTK with the residual vector at every iteration t .
Next, we prove that for small perturbations � 0 ,data augmentation prevents over ﬁ tting and improves generalization by proportionally enlarging andperturbing smaller eigenvalues of the NTK relatively more, while preserving its prominent directions.
4.1 Effect of Augmentation on Eigenvalues of the NTK
We ﬁ rst investigate the effect of data augmentation on the singular values of the Jacobian, and usethis result to bound the change in the eigenvalues of the NTK.
To characterize the effect of dataaugmentation on singular values of the perturbed Jacobian ˜ J , we rely on Weyl’s theorem [38]stating that under bounded perturbations E , no singular value can move more than the norm of theperturbations.
Formally, | ˜ σ i − σ i | ≤ � E � 2 , where ˜ σ i and σ i are the singular values of the perturbed 163 and original Jacobian respectively.
Crucially, data augmentation affects larger and smaller singularvalues differently.
Let P be orthogonal projection onto the column space of J T , and
P ⊥ =
I − P bethe projection onto its orthogonal complement subspace.
Then, the singular values of the perturbed
Jacobian ˜ J T are ˜ σ 2 i =
( σ i + µ i ) 2 + ζ 2 i , where | µ i | ≤ � P E � 2 , and σ min ( P ⊥ E ) ≤ ζ i ≤ � P ⊥ E � 2 ,σ min the smallest singular value of J T [35].
Since the eigenvalues of the projection matrix P areeither 0 or 1, as the number of dimensions m grows, for bounded perturbations we get that on averageµ 2 i
= O (1) and ζ 2 i
= O ( m ) .
Thus, the second term dominates and increase of small singular valuesunder perturbation is proportional to √ m .
However, for larger singular values, ﬁ rst term dominatesand hence ˜
σ i − σ i ∼ = µ i .
Thus in general, small singular values can become proportionally larger,while larger singular values remain relatively unchanged.
The following Lemma characterizes theexpected change to the eignvalues of the NTK.
Lemma 4.1.
Data augmentation as additive perturbations bounded by small � 0 results in thefollowing expected change to the eigenvalues of the NTK:
[Equation]
where p i := P ( ˜ σ i − σ i < 0) is the probability that σ i decreases as a result of data augmentation,and is smaller for smaller singular values.
All the proofs can be found in the Appendix.
Next, we discuss the effect of data augmentation onsingular vectors of the Jacobian and show that it mainly affects the non-prominent directions of the
Jacobian spectrum, but to a smaller extent compared to the singular values.
4.2 Effect of Augmentation on Eigenvectors of the NTK
Here, we focus on characterizing the effect of data augmentation on the eigenspace of the NTK.
Let the singular subspace decomposition of the Jacobian be J = U Σ V T .
Then for the NTK, wehave Θ
=
J J T =
U Σ V T V Σ U T =
U Σ 2 U T (since V T V = I ).
Hence, the perturbation of theeigenspace of the NTK is the same as perturbation of the left singular subspace of the Jacobian J .
[Figure]
Suppose σ i are singular values of the Jacobian.
Let the perturbed Jacobian be ˜
J = J + E , anddenote the eigengap γ
= min { σ i − σ i
+1 :
i = 1 , · · · , r } where σ r +1 := 0 .
Assuming γ 0 ≥ 2 � E � 2 , 188 a combination of Wedin’s theorem [37] and Mirsky’s inequality [27] implies
[Equation]
This result provides an upper-bound on the change of every left singular vectors of the Jacobian.
However as we discuss below, data augmentation affects larger and smaller singular directionsdifferently.
To see the effect of data augmentation on every singular vectors of the Jacobian, letthe subspace decomposition of Jacobian be J = U Σ V T
=
U s Σ s V
Ts + U n Σ n V Tn , where U sassociated with nonzero singular values, spans the column space of J , which is also called thesignal subspace, and U n , associated with zero singular values ( Σ n = 0 ), spans the orthogonal spaceof U s , which is also called the noise subspace.
Similarly, let the subspace decomposition of theperturbed Jacobian be ˜ J =
˜ U ˜ Σ
˜ V T = ˜
U s ˜ Σ s ˜ V Ts + ˜ U n ˜ Σ n ˜ V Tn , and ˜ U s = U s +
Δ U s , where Δ U sis the perturbation of the singular vectors that span the signal subspace.
Then the following generalﬁ rst-order expression for the perturbation of the orthogonal subspace due to perturbations of the
Jacobian characterize the change of the singular directions: Δ U s =
U n U Tn EV s Σ − 1 s [24].
Wesee that singular vectors associated to larger singular values are more robust to data augmentation,compared to others.
Note that in general singular vectors are more robust than singular values.
Fig.
1 shows the effect of perturbations with [-(cid:15) 
-]� 0 = 8 , 16 on singular values and singular vectors of the
Jacobian matrix for a 1 hidden layer MLP trained on MNIST, and ResNet20 trained on CIFAR10.
Ascalculating the entire Jacobian spectrum is computationally prohibitive, data is subsampled from 3classes.
We report the effect of other real-world augmentation techniques, such as random crops, ﬂ ips,rotations and Autoaugment [8] - which includes translations, contrast, and brightness transforms - in
Appendix C. We observe that data augmentation increases smaller singular values relatively more.
On the other hand, it affects prominent singular vectors of the Jacobian to a smaller extent.
4.3 Augmentation Improves Training & Generalization
By making relatively larger changes to the smaller singular values of the Jacobian with a highprobability, and making relatively larger perturbations to non-prominent singular vectors, dataaugmentation results in bounded but varying perturbations to particularly the nuisance space of the
Jacobian during the training.
Under label-preserving transformations, data augmentation preventsthe network parameters from over ﬁ tting over the nuisance space.
This results in a potentially largertraining loss, but better generalization performance, as shown in Appendix D. Theorem B.1 in the
Appendix characterizes the expected training dynamics resulted by data augmentation.
At the same time, the relative growth of the smaller singular values resulted by data augmentationimproves the generalization performance of the network.
In general, learning along small eigenvectorsof the Jacobian is slower [6].
Data augmentation speeds up training along these dimensions byincreasing the eigenvalues of the smaller eigendirections of the NTK, while preserving eigenvectors,and hence enhance learning along these (harder to learn) components.
The following Lemma capturesthe improvement in the generalization performance, as a result of data augmentation.
Lemma 4.2.
Assume gradient descent with learning rate η is applied to train a neural networkwith constant NTK and Lipschitz constant L , on data points augmented with additive perturbations bounded by � 0 as de ﬁ ned in Sec. 3.
Let σ min be the minimum singular value of Jacobian J associatedwith training data X train .
With probability 1 − δ , generalization error of the network trained withgradient descent on augmented data X aug enjoys the following bound:
[Equation]
Effective Subsets for Data Augmentation
Here, we focus on identifying subsets of data that when augmented similarly improve generalizationand prevent over ﬁ tting.
To do so, our key idea is to ﬁ nd subsets of data points that when augmented,closely capture the alignment of the NTK (or equivalently the Jacobian) corresponding to thefull augmented data with the residual vector, J ( W t , X taug ) T r taug .
If such subsets can be found,augmenting only the subsets will change the NTK and its alignment with the residual in a similar wayas that of full data augmentation, and will result in similar improved training dynamics.
However,generating the full set of transformations X taug is often very expensive, particularly for strongaugmentations and large datasets.
Hence, generating the transformations, and then extracting thesubsets may not provide a considerable overall speedup.
In the following, we show that weighted subsets (coresets) S that closely estimate the alignment of the
Jacobian associated to the original data with the residual vector J T ( W t , X train ) r train can closelyestimate the alignment of the Jacobian of the full augmented data and
the corresponding residual
J T ( W t , X taug ) r taug .
Thus, the most effective subsets for augmentation can be directly found fromthe training data.
Formally, subsets S t ∗ weighted by γ tS that capture the alignment of the full Jacobianwith residual by an error of at most ξ can be found by solving the following optimization problem:
[Equation]
Solving the above optimization problem is NP-hard.
However, as we discuss in the Appendix A, anear optimal subset can be found by minimizing the Frobenius norm of a matrix G S , in which the i throw contains the euclidean distance between data point i and its closest element in the subset S , inthe gradient space.
Formally, [ G S ] i. =
min j � ∈ S � J T ( W t , x i ) r i − J T ( W t , x j � ) r j � � .
Intuitively, 248 such subsets contain the set of medoids of the dataset in the gradient space.
Medoids of a datasetare de ﬁ ned as the most centrally located elements in the dataset [19].
The weight of every elementj ∈ S is the number of data points closest to it in the gradient space, i.e., γ j = � i ∈
V I [ j =arg min j � ∈ S � J T ( W t , x i ) r
i − J T ( W t , x j � ) r j � � ] .
The set of medoids can be found by solving thefollowing submodular 1 cover problem:
[Equation]
where C ≥ � G S � F is a constant.
The classical greedy algorithm provides a logarithmic approxima- 254 tion for the above submodular maximization problem, i.e., | S | ≤ (1 + ln ( n )) .
It starts with the empty 255 set S [-
-]0 = ∅ , and at each iteration τ , it selects the training example e ∈ V that maximizes the marginalgain F ( e | S τ ) =
F ( S τ ∪ { e } ) − F ( S τ ) .
Formally, S τ = S τ − 1 ∪ { arg max e ∈ V F ( e | S τ − 1 ) } .
The
O ( nk ) computational complexity of the greedy algorithm can be reduced to O ( n ) using randomizedmethods [29] and further improved using lazy evaluation [26] and distributed implementations [31].
The rows of the matrix G can be ef ﬁ ciently upper-bounded using the gradient of the loss w.r.t.
theinput to the last layer of the network, which has been shown to capture the variation of the gradientnorms closely [18].
The above upper-bound is only marginally more expensive than calculating thevalue of the loss.
Hence the subset can be found ef ﬁ ciently.
Better approximations can be obtainedby considering earlier layers in addition to the last two, at the expense of greater computational cost.
At every iteration t during training, we select a coreset from every class c ∈
[ C ] separately, and applythe set of transformations { T ti } ri =1 only to the elements of the coresets, i.e., X taug = { ∪ ri =1 T ti ( X S t ) } .
266 We divide the weight of every element j in the coreset equally among its transformations, i.e. theﬁ nal weight ρ tj = γ tj /r if j ∈ S t .
We apply the gradient descent updates in Eq.
(1) to the weighted
Jacobian matrix of X t =
X t aug or X t =
{ X train ∪ X t aug } (viewing ρ t as ρ t ∈ R n ) as follows:
[Equation]
[Table]
The pseudocode is in Alg.1, Appendix.
The following Lemma upper bounds the difference betweenthe alignment of the Jacobian and residual for augmented coreset vs. full augmented data.
Lemma 5.1.
Let S be a coreset that captures the alignment of the full data NTK with residualwith an error of at most ξ as in Eq.
Augmenting the coreset with perturbations bounded by� 0 ≤ n 32 √ L captures the alignment of the fully augmented data with the residual by an error of at most
[Equation]
5.1 Coreset vs. Max-loss Data Augmentation
In the initial phase of training the NTK goes through rapid changes.
This determines the ﬁ nal basinof convergence and network’s ﬁ nal performance [12].
Regularizing deep networks by weight decayor data augmentation mainly affects this initial phase and matters little afterwards [13].
Crucially,augmenting coresets that closely capture the alignment of the NTK with the residual during thisinitial phase results in less over ﬁ tting and improved generalization performance.
On the other hand,augmenting points with maximum loss early in training decreases the alignment between the NTKand the label vector and impedes learning and convergence.
After this initial phase when the networkhas good prediction performance, the gradients for majority of data points become small.
Here, thealignment is mainly captured by the elements with the maximum loss.
Thus, as training proceeds, theintersection between the elements of the coresets and examples with maximum loss increases.
Wevisualize this pattern in Appendix D.
The following Theorem characterizes the training dynamics oftraining on the full data and the augmented coresets, using our additive perturbation model.
Theorem 5.2.
Let L i be β -smooth, L be λ -smooth and satisfy the α -PL condition, that is forα > 0 , �∇ L ( W ) � 2 ≥ α L ( W ) for all weights W .
Let f be Lipschitz in X with constant L � , and¯ L = max { L, L � } .
Let G 0 be the gradient at initializaion, σ max the maximum singular value ofthe coreset Jacobian at initialization.
Choosing � 0 ≤ 1 σ max √ ¯ Ln and running SGD on full data withaugmented coreset using constant step size η = αλβ , result in: the following bound:
[Equation]
293 Theorem 5.2 shows that training on full data and augmented coresets converges with the same rateas training on the fully augmented data, to a close neighborhood of the optimal solution.
The sizeof the neighborhood depends on the error of the coreset ξ in Eq.
(5), and the error in capturing thealignment of the full augmented data with the residual derived in Lemma 5.1.
The ﬁ rst term decreaseas the size of the coreset grows, and the second term depends on the network structure.
We alsoanalyze convergence of training only on the augmented coresets, and augmentations modelled asarbitrary linear transformations using a linear model [39] in Appendix B.
Experiments
Setup and baselines.
We extensively evaluate the performance of our approach in three differentsettings.
Firstly, to evaluate the quality of our coresets for augmentation we consider training only oncoresets and their augmentations.
Secondly, we investigate the effect of adding augmented coresetsto the full training data.
Finally, we consider adding augmented coresets to random subsets.
Wecompare our coresets with max-loss loss and/or random subsets as baselines. For all methods, we [Table] [Table]
[Table]
select a new augmentation subset every R epochs.
We note that while the original maxloss method[21] selects points from a fully trained model hence limits to only one subset throughout training, tomaximize fairness, our max-loss baseline selects a new subset at every subset selection step.
For allexperiments, standard weak augmentations (random crop and horizontal ﬂ ips) are always performedon both the original and strongly augmented data.
6.1 Training on Coresets and their Augmentations
First, we evaluate the effectiveness of our approach for training on the coresets and their augmentations.
Our main goal here is to compare the performance of training on and augmenting coresets vs.random and max-loss subsets.
Tab. 1 shows the test accuracy for training ResNet20 and Wide-ResNeton CIFAR10 when we only train on small augmented coresets of size 0 .
1% to 1% selected at everyepoch ( R = 1 ), and ResNet18 on Caltech256 using coresets of size 5% to 50% with R = 5 .
We seethat the augmented coresets outperform augmented random subsets by a large margin.
This clearlyshows the effectiveness of augmenting the coresets, and the importance of capturing the alignment ofthe NTK with the residual for data augmentation.
Note that for CIFAR10 experiments, training onthe augmented max-loss points did not even converge in absence of full data.
Generalization across augmentation techniques Our coresets are not dependent on the type ofdata augmentation.
To con ﬁ rm this, we show the superior generalization performance of our methodin Tab. 2 on ResNet18/R=5 on coresets vs random subsets of Caltech256, augmented with CutOut[11], AugMix [15], and noise perturbations (color jitter, gaussian blur).
For example, On 30%subsets, we obtain 28.2%, 29.3%, 20.2% improvement over random when using CutOut, AugMix,and perturbation augmentations respectively.
6.2 Training on Full Data and Augmented Coresets
Next, we study the effectiveness of our method for training on full data and augmented coresets.
Tab. 4 demonstrates the accuracy improvement resulted by augmenting subsets of size 5%, 10%,and 30% selected by our method vs. max-loss and random over full data augmentation.
We observethat augmenting coresets effectively improves generalization, and outperforms augmenting randomand max-loss subsets across different models and datasets.
In Appendix D, we also show that ourapproach is effective when trained with full data even for small augmentation subset sizes of 0 .
2%and 0 .
5% with R = 1 .
We also report results on TinyImageNet/ResNet50 ( R =15 ) in Tab. 3, wherewe show that augmenting coresets outperforms baseline max-loss and random subsets - For 30%subsets, we improve over max-loss and random on TinyImageNet by 3.7% and 4.4% respectively.
Training speedup In Fig.
2, we measure the improvement in training time in the case of trainingon full data and augmenting subsets of various sizes.
While our method yields similar or slightlylower speed-up to the max-loss policy and random approach respectively, our resulting accuracy [Table] outperforms these two approaches on average.
For example, for SVHN/Resnet32 using ,we sacri ﬁ ce 10% of the speed-up to obtain an additional 24 .
8% of the gain in accuracy from full augmentation when compared to a random subset of the same size.
We also provide -clock times for in
[Figure]
Augmenting noisy labeled data.
We also experimentally con ﬁ rm the robustness of our coresets tolabel noise.
Tab. 5 shows the result of augmenting coresets vs. max-loss and random subsets
ofdifferent sizes selected from CIFAR10 with 50% label noise for training ResNet20.
Augmentingcoresets selected by our method not only outperforms max-loss and random, but provides a superiorperformance over full data augmentation.
This con ﬁ rms the effectiveness of the coresets in capturingthe alignment of the NTK with the residual of clean data.
6.3 Training on Random Data and Augmented Coresets
Finally, we evaluate the performance of our method for training on random subsets and augmentingcoresets.
We report results on ImageNet and TinyImageNet on ResNet50 (90 epochs, R =15 ).
In
Tab. 6, we train on random subsets and augment random subsets and coresets of the same size.
Weshow that our results hold for large-scale datasets, where we improve by 7 . 6% , 2 . 3% , and 1 . 3% overrandom with 10% , 30% , and 50% subsets respectively on ImageNet.
[Table]
Conclusion
We showed that data augmentation improves training and generalization by relatively enlargingand perturbing the smaller singular values of the neural network Jacobian while preserving itsprominent directions.
Then, we proposed a framework to iteratively extract small coresets oftraining data that when augmented, closely capture the alignment of the fully augmented Jacobianwith the label/residual vector.
We showed the effectiveness of augmenting coresets in providing asuperior generalization performance when added to the full data, in presence of noisy labels, or as astandalone subset.
While under local smoothness of images, our additive perturbation can be appliedto model many bounded transformations such as small rotations, crops, shearing, and pixel-wisetransformations like sharpening, blurring, color distortions, structured adversarial perturbation [25],the additive perturbation model is indeed limited when applied to augmentations that cannot bereduced to perturbations, such as horizontal/vertical ﬂ ips and large translations.
Further theoreticalanalysis of complex data augmentations is indeed an interesting direction for future work.[-A Proof of Main Results A.1 Proof for Lemma 4.
 Proof.-]