Data-Efﬁcient Augmentation for Training Neural Networks
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Data augmentation is essential to achieve state-of-the-art performance in manydeep learning applications.
However, the most effective augmentation techniquesbecome computationally prohibitive for even medium-sized datasets.
To addressthis, we propose a rigorous technique to select subsets of data points that whenaugmented, closely capture the training dynamics of full data augmentation.
Weﬁrst show that data augmentation, modeled as additive perturbations, improveslearning and generalization by relatively enlarging and perturbing the smallersingular values of the network Jacobian, while preserving its prominent directions.
Then, we propose a framework to iteratively extract small subsets of trainingdata that when augmented, closely capture the alignment of the fully augmented
Jacobian with labels/residuals.
We prove that stochastic gradient descent applied toaugmented subsets found by our approach have similar training dynamics to that offully augmented data.
Our experiments demonstrate that our method outperformsstate-of-the-art by 7 . 7% on CIFAR10 with 6.3x speedup and 4 .
7% on SVHNwith 2.2x speedup, using 10% and 30% augmented subsets respectively.
Besides,augmenting 10% and 30% subsets from our method beats random baselines by7 .
9% and 5 .
3% on TinyImageNet, and by 7 . 6% and 2 .
3% on ImageNet.
Introduction
Standard data augmentation expands the training data by applying transformations, such as rotationsor crops for images, to the original training examples.
Due to its effectiveness, data augmentation isa key component in achieving nearly all state-of-the-art results in deep learning applications [32].
However, the most effective (strong) data augmentation techniques increase the training time by ordersof magnitude.
First, they often have a very expensive pipeline to generate transformations that bestimproves generalization [4, 15, 22].
Importantly, to obtain the state-of-the-art performance, multiplestrongly augmented examples are added for every single data point and to each training iteration [14,37].
As a result, state-of-the-art data augmentation techniques become computationally prohibitivefor even medium-sized real-world problems.
For example, the state-of-the-art augmentation of [37]increases training time of ResNet20 on CIFAR10 by 13x on an Nvidia A40 GPU.
To make data augmentation more efﬁcient and scalable, an effective approach is to carefully selecta small subset of the training data such that augmenting only the subset provides similar trainingdynamics to that of full data augmentation.
If such a subset can be quickly found, it would directlylead to a signiﬁcant reduction in storage and training costs.
Despite the efﬁciency and scalabilitythat it can provide, this direction has remained largely unexplored.
Existing studies are limited tofully training a network and subsampling data points based on its loss or inﬂuence for augmentationin subsequent training runs [20].
However, this method is prohibitive for large datasets, providesa marginal improvement over augmenting random subsets, and does not provide any theoreticalguarantee for the performance of the network trained on the augmented subsets.
A major challenge in ﬁnding the most effective data points for augmentation is to theoretically understand how data augmentation affects the optimization and generalization of neural networks.
Existingtheoretical results are mainly limited to simple linear classiﬁers and analyze data augmentation asenlarging the span of the training data [37], providing a regularization effect [3, 9, 34, 37], enlargingthe margin of a linear classiﬁer [30], or having a variance reduction effect [6].
However, such toolsdo not provide insights on the effect of data augmentation on training deep neural networks.
Here, we study the effect of label invariant data augmentation on training dynamics of overparameterized neural networks.
Theoretically, we model data augmentation by bounded additive perturbations[30], and analyze its effect on neural network Jacobian matrix containing all its ﬁrst-order partialderivatives [1].
We show that label invariant data augmentation proportionally enlarges but moreimportantly perturbs the singular values of the Jacobian, particularly smaller ones, while maintainingits prominent directions.
In doing so, data augmentation regularizes training with bounded but varyingperturbations to the gradients and prevents overﬁtting.
Empirically, we show that our analysis holdsfor various strong augmentations, e.g., AutoAugment [7], CutOut [10], and AugMix [14].
Next, we develop a rigorous method to iteratively ﬁnd small weighted subsets (coresets) that whenaugmented, closely capture the alignment between the Jacobian of the full augmented data with thelabel/residual vector.
We show that the most effective subsets for data augmentation are the set ofexamples with the most centrally located gradients, and can be obtained by maximizing a submodularfunction.
Such subsets can be efﬁciently extracted using a fast greedy algorithm which operates onsmall dimensional gradient proxies, with only a small additional cost.
We prove that augmenting thecoresets guarantees similar training dynamics to that of full data augmentation.
We demonstrate the effectiveness of our approach applied to CIFAR10 (ResNet20,WideResNet28-10), CIFAR10-IB (ResNet32), SVHN (ResNet32), noisy-CIFAR10 (ResNet20), Caltech256(ResNet18,ResNet50), TinyImageNet (ResNet50), and ImageNet (ResNet50) compared to randomand max-loss baselines [20].
We show that augmenting coresets found by our approach outperformsthe state-of-the-art even in absence of the full data.
For instance, when small augmented subsetsof size 30% found by our approach are appended to CIFAR10, we attain 75% of the improvementin test accuracy compared to augmenting the full dataset while enjoying a 3.4x speedup in trainingtime.
In the absence of full data, training and augmenting tiny coresets of size 1% can achieve 74 .
7%accuracy on CIFAR10/ResNet20 while providing a 39x speedup compared to using the full dataset.
On CIFAR10 with 50% noisy labels, augmenting 50% of the training data outperforms full dataaugmentation.
For larger dataset experiments, training ResNet18 on subsets of size 10% and 30%respectively yields 5.8% and 0.8% gain in test accuracy on Caltech256, while training ResNet50 on
ImageNet yields 7.6% and 2.3% gain.
Additional Related Work
Strong data augmentation methods achieve state-of-the-art performance by ﬁnding the set of transformations for every example that best improves the performance.
Methods like AutoAugment [7],
RandAugment [8], and Faster RandAugment [8] search over a (possibly large) space of transformations to ﬁnd sequences of transformations that best improves generalization [7, 8, 24, 37].
Othertechniques involve a very expensive pipeline for generating the transformations.
For example, someuse Generative Adversarial Networks to directly learn new transformations [2, 24, 27, 31].
Strongaugmentations like Smart Augmentation [22], Neural Style Transfer-based [15], and GAN-based augmentations [4] require an expensive forward pass through a deep network for input transformations.
For example, [15] increases training time by 2.8x for training ResNet18 on Caltech256.
Strong data augmentation methods either replace the original example by its transformed version,or add the generated transformations to the training data.
Crucially, augmenting the training datawith transformations is much more effective in improving the generalization performance.
Hence,the most effective data augmentation methods such as that of [37] and AugMix [14] append thetransformed examples to the training data.
In Appendix D, we show that even for cheaper strongaugmentation methods such as AutoAugment [7], while replacing the original training examples withtransformations may decrease the performance, appending the augmentations signiﬁcantly improvesthe performance.
Appending the training data with augmentations, however, increase the trainingtime by orders of magnitude.
For example, AugMix [14] that outperforms AutoAugment increasesthe training time by at least 3x by appending extra augmented examples, and [37] increases trainingtime by 13x due to appending and forwarding additional augmented examples through the model.
Problem Formulation
We begin by formally describing the problem of learning from augmented data.
Consider a dataset
D train =
( X train , y train ) , where X train = ( x 1 , · · · , x n
) ∈ R d × n is the set of n normalized datapoints x i ∈ [0 , 1] d , from the index set V , and y train =
( y 1 , · · · , y n ) ∈ { y ∈ { ν 1 , ν 2 , · · · , ν C }} with{ ν j } Cj =1 ∈ [0 , 1] .
Following [30] we model data augmentation as an arbitrary bounded additiveperturbation (cid:15) , with (cid:107) (cid:15) (cid:107)≤ (cid:15) 0 .
For a given (cid:15) 0 and the set of all possible transformations A , we studythe transformations selected from S ⊆
A satisfying S =
{ T i ∈
A | (cid:107) T i ( x ) − x (cid:107)≤ (cid:15)
∀ x ∈ X train } .
While this model cannot represent all augmentations, we empirically show in Appendix C and in ourexperiments that our theory holds for various weak and strong augmentations.
We separately analyzethe effect of linear transformations on training dynamics of a linear classiﬁer in Appendix B
The set of augmentations at iteration t generating r augmented examples per data point can bespeciﬁed, with abuse of notation, as D taug = { (cid:83) ri
=
( T ti ( X train ) , y train ) } , where |D taug | = rn and
T ti ( X train ) transforms all the training data points with the set of transformations T ti ⊂ S at iterationt .
We denote X taug = { (cid:83) ri =1 T ti ( X train ) } and y taug = { (cid:83) ri =1 y train } .
Let f ( W , x ) be an arbitrary neural network with m vectorized (trainable) parameters W ∈ R m .
Weassume that the network is trained using (stochastic) gradient descent with learning rate η to minimizethe squared loss L over the original and augmented training examples D t =
{D train ∪ D taug }with associated index set V t , at every iteration t .
I.e., L ( W t , X ) :
= 12 (cid:80) i ∈ V t L
i ( W t , x i ) :
=12 (cid:80) ( x i ,y i ) ∈D t (cid:107) f
( W t , x i ) − y
i
(cid:107) 22 .
The gradient update at iteration t is given by
W t +1 =
W t − η ∇L ( W t , X ) , s.t
[Equation]
) where X t = { X train ∪ X taug } and y t = { y train ∪ y taug } are the set of original and augmentedexamples and their labels, J ( W , X ) ∈ R n × m is the Jacobian matrix associated with f , andr t = f
( W t , X ) − y is the residual.
We further assume that J is smooth with Lipschitz constant
L .
I.e., (cid:107)J ( W , x i ) −J ( W , x j ) (cid:107)≤ L (cid:107) x
i − x j (cid:107) ∀ x i , x j ∈ X .
Thus, for any transformation
T j ∈ S , we have (cid:107)J ( W , x i ) −J ( W , T j ( x i ))
(cid:107)≤ L(cid:15) 0 .
Denoting J
= J ( W , X train ) and ˜ J
=
J ( W ,T j ( X train )) ,we get ˜ J
= J + E , where E is the perturbation matrix with (cid:107) E (cid:107) 2 ≤(cid:107)
E (cid:107) F ≤√ nL(cid:15) 0 .
Data Augmentation Improves Learning
In this section, we analyze the effect of data augmentation on training dynamics of neural networks,and show that data augmentation can provably prevent overﬁtting.
To do so, we leverage the recentresults that characterize the training dynamics based on properties of neural network Jacobian and thecorresponding Neural Tangent Kernel (NTK) [16] deﬁned as Θ = J ( W , X ) J
( W , X ) T .
Formally:
[Equation]
where Θ = U Λ U T
= (cid:80) i =
λ i u
i
u
Ti is the eigendecomposition of the NTK [1].
Although theconstant NTK assumption holds only in the inﬁnite width limit, [21] found close empirical agreementbetween the NTK dynamics and the true dynamics for wide but practical networks, such as wide
ResNet architectures [38].
Eq.
(2) shows that training dynamics depend on the alignment of the
NTK with the residual vector at every iteration t .
Next, we prove that for small perturbations (cid:15) 0 ,data augmentation prevents overﬁtting and improves generalization by proportionally enlarging andperturbing smaller eigenvalues of the NTK relatively more, while preserving its prominent directions.
4.1 Effect of Augmentation on Eigenvalues of the NTK
We ﬁrst investigate the effect of data augmentation on the singular values of the Jacobian, and usethis result to bound the change in the eigenvalues of the NTK.
To characterize the effect of dataaugmentation on singular values of the perturbed Jacobian ˜ J , we rely on Weyl’s theorem [36]stating that under bounded perturbations E , no singular value can move more than the norm of theperturbations.
Formally, | ˜ σ i − σ i |≤ (cid:107) E (cid:107) 2 , where ˜ σ i and σ i are the singular values of the perturbedand original Jacobian respectively.
Crucially, data augmentation affects larger and smaller singular
[Figure]
values differently.
Let P be orthogonal projection onto the column space of J T , and
P ⊥ =
I − P bethe projection onto its orthogonal complement subspace.
Then, the singular values of the perturbed
Jacobian ˜ J T are ˜ σ 2 i =
( σ i + µ i ) 2 + ζ 2 i , where | µ i |≤ (cid:107) P E (cid:107) 2 , and σ min ( P ⊥ E ) ≤ ζ i ≤ (cid:107) P ⊥ E (cid:107) 2 ,σ min the smallest singular value of J T [33].
Since the eigenvalues of the projection matrix P areeither 0 or 1, as the number of dimensions m grows, for bounded perturbations we get that on averageµ 2 i
= O (1) and ζ 2 i
= O ( m ) .
Thus, the second term dominates and increase of small singular valuesunder perturbation is proportional to √ m .
However, for larger singular values, ﬁrst term dominatesand hence ˜
σ i − σ i ∼ = µ i .
Thus in general, small singular values can become proportionally larger,while larger singular values remain relatively unchanged.
The following Lemma characterizes theexpected change to the eignvalues of the NTK.
Lemma 4.1.
Data augmentation as additive perturbations bounded by small (cid:15) 0 results in thefollowing expected change to the eigenvalues of the NTK:
[Equation]
where p i := P ( ˜ σ i − σ i < 0) is the probability that σ i decreases as a result of data augmentation,and is smaller for smaller singular values.
All the proofs can be found in the Appendix.
Next, we discuss the effect of data augmentation onsingular vectors of the Jacobian and show that it mainly affects the non-prominent directions of the
Jacobian spectrum, but to a smaller extent compared to the singular values.
4.2 Effect of Augmentation on Eigenvectors of the NTK
Here, we focus on characterizing the effect of data augmentation on the eigenspace of the NTK.
Let the singular subspace decomposition of the Jacobian be J = U Σ V T .
Then for the NTK, wehave Θ
=
J J T =
U Σ V T V Σ U T =
U Σ 2 U T (since V T V = I ).
Hence, the perturbation of theeigenspace of the NTK is the same as perturbation of the left singular subspace of the Jacobian J .
Suppose σ i are singular values of the Jacobian.
Let the perturbed Jacobian be ˜
J = J + E , anddenote the eigengap γ
= min { σ i − σ i
+1 :
i = 1 , · · · , r } where σ r +1 := 0 .
Assuming γ 0 ≥ 2 (cid:107) E (cid:107) 2 ,a combination of Wedin’s theorem [35] and Mirsky’s inequality [26] implies
[Equation]
This result provides an upper-bound on the change of every left singular vectors of the Jacobian.
However as we discuss below, data augmentation affects larger and smaller singular directionsdifferently.
To see the effect of data augmentation on every singular vectors of the Jacobian, letthe subspace decomposition of Jacobian be J = U Σ V T
=
U s Σ s V
Ts + U n Σ n V Tn , where U sassociated with nonzero singular values, spans the column space of J , which is also called thesignal subspace, and U n , associated with zero singular values ( Σ n = 0 ), spans the orthogonal spaceof U s , which is also called the noise subspace.
Similarly, let the subspace decomposition of theperturbed Jacobian be ˜ J =
˜ U ˜ Σ
˜ V T = ˜
U s ˜ Σ s ˜ V Ts + ˜ U n ˜ Σ n ˜ V Tn , and ˜ U s =
U s + ∆ U s , where ∆ U sis the perturbation of the singular vectors that span the signal subspace.
Then the following generalﬁrst-order expression for the perturbation of the orthogonal subspace due to perturbations of the
Jacobian characterize the change of the singular directions:
∆ U s =
U n U Tn EV s Σ − 1 s [23].
We
Require: The dataset D = { ( x i , y i ) } ni =1 , number of iterations T .
Ensure: Output model parameters W T .
1: for t = 1 , · · · , T do 2:
S t = ∅ , X taug = ∅ .
3: for c ∈ { 1 , · · · , C } do 4:
S tc = greedy ( V c ) (cid:46) Extract a coreset from class c by solving Eq.
(7) 5: γ j = (cid:80) i ∈
V c I [ j =arg min j (cid:48) ∈ S (cid:107)J T ( W t , x i ) r
i −J T ( W t , x j (cid:48) )
r j (cid:48) (cid:107) ]
(cid:46)
Coreset weights 6:
X taug = { X aug ∪ {∪ ri =1 T ti ( X S tc ) }} (cid:46) Augment the coreset 7: ρ tj = γ tj /r 8: end for 9:
Update the parameters W t using weighted gradient descent on X taug or { X train ∪ X taug } .
10: end for see that singular vectors associated to larger singular values are more robust to data augmentation,compared to others.
Note that in general singular vectors are more robust than singular values.
Fig.
1 shows the effect of perturbations with (cid:15)
= 8 , 16 on singular values and singular vectors of the
Jacobian matrix for a 1 hidden layer MLP trained on MNIST, and ResNet20 trained on CIFAR10.
Ascalculating the entire Jacobian spectrum is computationally prohibitive, data is subsampled from 3classes.
We report the effect of other real-world augmentation techniques, such as random crops, ﬂips,rotations and Autoaugment [7] - which includes translations, contrast, and brightness transforms - in
Appendix C. We observe that data augmentation increases smaller singular values relatively more.
On the other hand, it affects prominent singular vectors of the Jacobian to a smaller extent.
4.3 Augmentation
Improves Training & Generalization
By making relatively larger changes to the smaller singular values of the Jacobian with a highprobability, and making relatively larger perturbations to non-prominent singular vectors, dataaugmentation results in bounded but varying perturbations to particularly the nuisance space of the
Jacobian during the training.
Under label-preserving transformations, data augmentation preventsthe network parameters from overﬁtting over the nuisance space.
This results in a potentially largertraining loss, but better generalization performance, as shown in Appendix D. Theorem B.1 in the
Appendix characterizes the expected training dynamics resulted by data augmentation.
At the same time, the relative growth of the smaller singular values resulted by data augmentationimproves the generalization performance of the network.
In general, learning along small eigenvectorsof the Jacobian is slower [5].
Data augmentation speeds up training along these dimensions byincreasing the eigenvalues of the smaller eigendirections of the NTK, while preserving eigenvectors,and hence enhance learning along these (harder to learn) components.
The following Lemma capturesthe improvement in the generalization performance, as a result of data augmentation.
Lemma 4.2.
Assume gradient descent with learning rate η is applied to train a neural networkwith constant NTK and Lipschitz constant L , on data points augmented with additive perturbationsbounded by (cid:15) 0 as deﬁned in Sec.
Let σ min be the minimum singular value of Jacobian J associatedwith training data X train .
With probability 1 − δ , generalization error of the network trained withgradient descent on augmented data X aug enjoys the following bound:
[Equation]
Effective Subsets for Data Augmentation
Here, we focus on identifying subsets of data that when augmented similarly improve generalizationand prevent overﬁtting.
To do so, our key idea is to ﬁnd subsets of data points that when augmented,closely capture the alignment of the NTK (or equivalently the Jacobian) corresponding to thefull augmented data with the residual vector, J ( W t , X taug ) T r taug .
If such subsets can be found,augmenting only the subsets will change the NTK and its alignment with the residual in a similar wayas that of full data augmentation, and will result in similar improved training dynamics.
However, generating the full set of transformations X taug is often very expensive, particularly for strongaugmentations and large datasets.
Hence, generating the transformations, and then extracting thesubsets may not provide a considerable overall speedup.
In the following, we show that weighted subsets (coresets) S that closely estimate the alignment of the
Jacobian associated to the original data with the residual vector J T ( W t , X train ) r train can closelyestimate the alignment of the Jacobian of the full augmented data and
the corresponding residual
J T ( W t , X taug ) r taug .
Thus, the most effective subsets for augmentation can be directly found fromthe training data.
Formally, subsets S t ∗ weighted by γ tS that capture the alignment of the full Jacobianwith residual by an error of at most ξ can be found by solving the following optimization problem:
[Equation]
Solving the above optimization problem is NP-hard.
However, as we discuss in the Appendix A, anear optimal subset can be found by minimizing the Frobenius norm of a matrix G S , in which the i throw contains the euclidean distance between data point i and its closest element in the subset S , inthe gradient space.
Formally, [ G S ] i. =
min j (cid:48) ∈ S (cid:107)J T ( W t , x i ) r i
[Equation]
J T ( W t , x j (cid:48) )
r j (cid:48) (cid:107) .
Intuitively,such subsets contain the set of medoids of the dataset in the gradient space.
Medoids of a datasetare deﬁned as the most centrally located elements in the dataset [18].
The weight of every elementj ∈ S is the number of data points closest to it in the gradient space, i.e., γ j = (cid:80) i ∈
V I [ j =
[Equation]
S
[Equation]
T ( W t , x i ) r
i − J T ( W t , x j (cid:48) )
r j (cid:48) (cid:107) ] .
The set of medoids can be found by solving thefollowing submodular 1 cover problem:
[Equation]
where C ≥ (cid:107) G S (cid:107) F is a constant.
The classical greedy algorithm provides a logarithmic approximation for the above submodular maximization problem, i.e., | S |≤ (1 + ln ( n )) .
It starts with the emptyset S
= ∅ , and at each iteration τ , it selects the training example e ∈ V that maximizes the marginalgain F ( e | S τ ) =
F ( S τ ∪ { e } ) − F ( S τ ) .
Formally, S τ = S τ − 1 ∪ { arg max e ∈ V F ( e | S τ − 1 ) } .
The
O ( nk ) computational complexity of the greedy algorithm can be reduced to O ( n ) using randomizedmethods [28] and further improved using lazy evaluation [25] and distributed implementations [29].
The rows of the matrix G can be efﬁciently upper-bounded using the gradient of the loss w.r.t.
theinput to the last layer of the network, which has been shown to capture the variation of the gradientnorms closely [17].
The above upper-bound is only marginally more expensive than calculating thevalue of the loss.
Hence the subset can be found efﬁciently.
Better approximations can be obtainedby considering earlier layers in addition to the last two, at the expense of greater computational cost.
At every iteration t during training, we select a coreset from every class c ∈
[ C ] separately, and applythe set of transformations { T ti } ri =1 only to the elements of the coresets, i.e., X taug = {∪ ri =1 T ti ( X S t ) } .
We divide the weight of every element j in the coreset equally among its transformations, i.e. theﬁnal weight ρ tj = γ tj /r if j ∈ S t .
We apply the gradient descent updates in Eq.
(1) to the weighted
Jacobian matrix of X t = X taug or X t
=
{ X train ∪ X taug } (viewing ρ t as ρ t ∈ R n ) as follows:
[Equation]
The pseudocode is in Alg.
The following Lemma upper bounds the difference between thealignment of the Jacobian and residual for augmented coreset vs. full augmented data.
Lemma 5.1.
Let S be a coreset that captures the alignment of the full data NTK with residualwith an error of at most ξ as in Eq.
Augmenting the coreset with perturbations bounded by(cid:15) 0 ≤ n 32 √ L captures the alignment of the fully augmented data with the residual by an error of at most
[Equation]
Coreset vs. Max-loss
Data Augmentation
In the initial phase of training the NTK goes through rapid changes.
This determines the ﬁnal basinof convergence and network’s ﬁnal performance [11].
Regularizing deep networks by weight decay
[Table]
or data augmentation mainly affects this initial phase and matters little afterwards [12].
Crucially,augmenting coresets that closely capture the alignment of the NTK with the residual during thisinitial phase results in less overﬁtting and improved generalization performance.
On the other hand,augmenting points with maximum loss early in training decreases the alignment between the NTKand the label vector and impedes learning and convergence.
After this initial phase when the networkhas good prediction performance, the gradients for majority of data points become small.
Here, thealignment is mainly captured by the elements with the maximum loss.
Thus, as training proceeds, theintersection between the elements of the coresets and examples with maximum loss increases.
Wevisualize this pattern in Appendix D.
The following Theorem characterizes the training dynamics oftraining on the full data and the augmented coresets, using our additive perturbation model.
Theorem 5.2.
Let L i be β -smooth, L be λ -smooth and satisfy the α -PL condition, that is forα > 0 , (cid:107)∇L ( W ) (cid:107) 2 ≥ α L ( W ) for all weights W .
Let f be Lipschitz in X with constant L (cid:48) , and¯ L = max { L, L (cid:48) } .
Let G 0 be the gradient at initializaion, σ max the maximum singular value ofthe coreset Jacobian at initialization.
Choosing (cid:15) 0 ≤
[Table]
[Equation]
and running SGD on full data withaugmented coreset using constant step size η = αλβ , result in: the following bound:
[Equation]
266 Theorem 5.2 shows that training on full data and augmented coresets converges with the same rateas training on the fully augmented data, to a close neighborhood of the optimal solution.
The sizeof the neighborhood depends on the error of the coreset ξ in Eq.
(5), and the error in capturing thealignment of the full augmented data with the residual derived in Lemma 5.1.
The ﬁrst term decreaseas the size of the coreset grows, and the second term depends on the network structure.
We alsoanalyze convergence of training only on the augmented coresets, and augmentations modelled asarbitrary linear transformations using a linear model [37] in Appendix B.
Experiments
Setup and baselines.
We extensively evaluate the performance of our approach in three differentsettings.
Firstly, to evaluate the quality of our coresets for augmentation we consider training only oncoresets and their augmentations.
Secondly, we investigate the effect of adding augmented coresetsto the full training data.
Finally, we consider adding augmented coresets to random subsets.
Wecompare our coresets with max-loss loss and/or random subsets as baselines.
For all methods, weselect a new augmentation subset every R epochs.
We note that while the original maxloss method[20] selects points from a fully trained model hence limits to only one subset throughout training, tomaximize fairness, our max-loss baseline selects a new subset at every subset selection step.
[Table]
[Table] [Table]
Data and augmentation.
We apply our method to training ResNet20 and Wide-ResNet-28-10 on
CIFAR10, and ResNet32 on CIFAR10-IMB (Long-Tailed CIFAR10 with Imbalance factor of 100following [19]) and SVHN datasets.
We train Caltech256 [13] on ImageNet-pretrained ResNet18,and include experiments with random initialization in Appendix D. TinyImageNet and ImageNet aretrained on with ResNet50.
For all experiments, a new set of default augmentations (random crop andhorizontal ﬂip) are applied every R epochs to the original data.
We use [37] for CIFAR10/SVHN, and
AutoAugment [7] for Caltech256, TinyImageNet, and ImageNet as the strong augmentation method.
Note that we append strong augmentations rather than apply them in-replace, which we show to bemore effective in Appendix D. All results are averaged over 5 runs using an Nvidia A40 GPU.
6.1 Training on Coresets and their Augmentations
First, we evaluate the effectiveness of our approach for training on the coresets and their augmentations.
Our main goal here is to show the accuracy improvement resulted from augmenting the coresets.
In particular instead of focusing on the quality of the coresets, we aim at showing the effectiveness ofthe coresets for augmentation.
Tab. 1 shows the test accuracy for training ResNet20 and Wide-ResNeton CIFAR
when we only train on small augmented coresets of size 0 .
1% to 1% selected at everyepoch ( R = 1 ), and ResNet18 on Caltech256 using coresets of size 5% to 50% with R = 5 .
We seethat the augmented coresets outperform augmented random subsets by a large margin.
This clearlyshows the effectiveness of augmenting the coresets, and the importance of capturing the alignment ofthe NTK with the residual for data augmentation.
Note that for CIFAR10 experiments, training onthe augmented max-loss points did not even converge in absence of full data.
Generalization across augmentation techniques Our coresets are not dependent on the type ofdata augmentation.
To conﬁrm this, we show the superior generalization performance of our methodin Tab. 2 on ResNet18/R=5 on coresets vs random subsets of Caltech256, augmented with CutOut[10], AugMix [14], and noise perturbations (color jitter, gaussian blur).
For example, On 30%subsets, we obtain 28.2%, 29.3%, 20.2% improvement over random when using CutOut, AugMix,and perturbation augmentations respectively.
6.2 Training on Full Data and Augmented Coresets
Next, we study the effectiveness of our method for training on full data and augmented coresets.
Tab. 4 demonstrates the accuracy improvement resulted by augmenting subsets of size 5%, 10%,and 30% selected by our method vs. max-loss and random over full data augmentation.
We observethat augmenting coresets effectively improves generalization, and outperforms augmenting randomand max-loss subsets across different models and datasets.
In Appendix D, we also show that ourapproach is effective when trained with full data even for small augmentation subset sizes of 0 .
2%and 0 .
5% with R = 1 .
We also report results on TinyImageNet/ResNet50 ( R =15 ) in Tab. 3, wherewe show that augmenting coresets outperforms baseline max-loss and random subsets - For 30%subsets, we improve over max-loss and random on TinyImageNet by 3.7% and 4.4% respectively.
[Table]
Training speedup In Fig.
2, we measure the improvement in training time in the case of trainingon full data and augmenting subsets of various sizes.
While our method yields similar or slightlylower speed-up to the max-loss policy and random approach respectively, our resulting accuracyoutperforms these two approaches on average.
For example, for SVHN/Resnet32 using 30% coresets,we sacriﬁce 10% of the speed-up to obtain an additional 24 .
8% of the gain in accuracy from full dataaugmentation when compared to a random subset of the same size.
We also provide wall-clock timesfor ﬁnding coresets from Caltech256 and TinyImageNet in Appendix D.
[Figure]
Augmenting noisy labeled data.
We also experimentally conﬁrm the robustness of our coresets tolabel noise.
Tab. 5 shows the result of augmenting coresets vs. max-loss and random subsets
ofdifferent sizes selected from CIFAR10 with 50% label noise for training ResNet20.
Augmentingcoresets selected by our method not only outperforms max-loss and random, but provides a superiorperformance over full data augmentation.
This conﬁrms the effectiveness of the coresets in capturingthe alignment of the NTK with the residual of clean data.
6.3 Training on Random Data and Augmented Coresets
Finally, we evaluate the performance of our method for training on random subsets and augmentingcoresets.
We report results on ImageNet and TinyImageNet on ResNet50 (90 epochs, R =15 ).
In
Tab. 6, we train on random subsets and augment random subsets and coresets of the same size.
Weshow that our results hold for large-scale datasets, where we improve by 7 . 6% , 2 . 3% , and 1 . 3% overrandom with 10% , 30% , and 50% subsets respectively on ImageNet.
[Table]
Conclusion
We showed that data augmentation improves training and generalization by relatively enlargingand perturbing the smaller singular values of the neural network Jacobian while preserving itsprominent directions.
Then, we proposed a framework to iteratively extract small coresets of trainingdata that when augmented, closely capture the alignment of the fully augmented Jacobian with thelabel/residual vector.
We showed the effectiveness of augmenting coresets in providing a superiorgeneralization performance when added to the full data, in presence of noisy labels, or as a standalonesubset.