D ATA - EFFICIENT A UGMENTATION FOR T RAINING N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Data augmentation is essential to achieve state-of-the-art performance in many deep learning applications.
However, modern data augmentation techniques become computationally prohibitive for large datasets.
To address this, we propose a rigorous technique to select subsets of data points that when augmented, closely capture the training dynamics of full data augmentation.
We ﬁrst show that data augmentation, modeled as additive perturbations, speeds up learning by enlarging the smaller singular values of the network Jacobian.
Then, we propose a framework to iteratively extract small subsets of training data that when augmented, closely capture the alignment of the fully augmented Jacobian with label/residual vector.
We prove that stochastic gradient descent applied to augmented subsets found by our approach have similar training dynamics to that of fully augmented data.
Our experiments demonstrate that our method outperforms state-of-the-art max-loss strategy by 7 . 7% on CIFAR10 while achieving 6.3x speedup, and by 4 . 7% on
SVHN while achieving 2.2x speedup, using 10% and 30% subsets, respectively.
I NTRODUCTION
Data augmentation expands the training data by applying transformations, such as rotations or crops for images, to the original training examples.
Due to its effectiveness, data augmentation is a key component in achieving nearly all state-of-the-art results in deep learning applications (Shorten &
Khoshgoftaar, 2019).
The most effective data augmentation techniques often search over a (possibly large) space of transformations to ﬁnd sequences of transformations that speeds up training the most (Cubuk et al., 2019; 2020; Luo et al., 2020; Wu et al., 2020).
In addition, multiple augmented examples are usually generated for a single data point to obtain better results, increasing the size of the training data by orders of magnitude.
As a result, state-of-the-art data augmentation techniques become computationally prohibitive for large real-world problems ( c.f. Fig. 1).
To make data augmentation more efﬁcient and scalable, an effective approach is to carefully select a small subset of the training data such that augmenting only the subset have similar training dynamics to that of full data augmentation.
If such a subset can be quickly found, it would directly lead to a signiﬁcant reduction in storage and training costs, and lower costs incurred from selecting and tuning the optimal set of transformations to apply.
Despite the efﬁciency and scalability that it can provide, this direction has remained largely unexplored.
Existing studies are limited to fully training a network and subsampling data points based on its loss or inﬂuence for augmentation in subsequent training runs (Kuchnik & Smith, 2018).
However, this method is prohibitive for large datasets, provides
[Figure]
a marginal improvement over augmenting random subsets, and does not provide any theoretical guarantee for the performance of the network trained on the augmented subsets.
A major challenge in ﬁnding the most effective data points for augmentation is to theoretically understand how data augmentation affects the optimization and generalization of neural networks.
Existing theoretical results are mainly limited to simple linear classiﬁers and analyze data augmentation as enlarging the span of the training data (Wu et al., 2020), providing a regularization effect (Bishop, 1995; Dao et al., 2019; Wager et al., 2013; Wu et al., 2020), enlarging the margin of a linear classiﬁer (Rajput et al., 2019), or having a variance reduction effect (Chen et al., 2019).
However, such tools do not provide insights on the effect of data augmentation on training deep neural networks.
Here, we study the effect of label invariant data augmentation modeled by small additive perturbations (Rajput et al., 2019) on training dynamics of overparameterized neural networks.
In particular, we rely on recent results that characterize training dynamics of neural networks based on the alignment of the labels/residuals with the singular subspace of the network’s Jacobian matrix containing all its ﬁrst-order partial derivatives (Arora et al., 2019).
We show that label invariant data augmentation enlarges smaller singular values of the information space, and prove that this will speed up training.
Next, we develop a rigorous method to iteratively ﬁnd small weighted subsets (coresets) that when augmented, closely capture the alignment of the full augmented data with the label/residual, at every point during the training.
Augmenting the coresets guarantees similar training dynamics to that of full data augmentation.
Our key observation is that early in training, this alignment is best captured by data points that are highly representative of their classes.
However towards the end of training when the network converges, data points with maximum loss best capture this alignment.
Data augmentation has been empirically shown to mainly affect the initial phase of training (Golatkar et al., 2019), which crucially determines the ﬁnal basin of convergence (Fort et al., 2020).
Better selection of points by our method during this initial phase explains the superior accuracy improvement resulted by augmenting our coresets vs. the max-loss and the better ﬁnal generalization performance.
Importantly, we show that our coresets can be provably trained on even in absence of full data, and even when a high fraction of labels are noisy.
We demonstrate the effectiveness of our approach applied to training ResNet20, ResNet32, WideResNet on CIFAR10, CIFAR10-IB and SVHN, compared to random and max-loss baselines (Kuchnik & Smith, 2018).
We show that augmenting coresets found by our approach outperforms the stateof-the-art even in absence of the full data.
For instance, when small augmented subsets of size 30% found by our approach are appended to CIFAR10, we attain 75% of the improvement in test accuracy compared to augmenting the full dataset while enjoying a 3.4x speedup in training time.
Even in the absence of full data, training and augmenting tiny coresets of size 1% can achieve 74 .
7% accuracy on
CIFAR10/ResNet20 while providing a 39x speedup compared to using the full dataset.
On CIFAR with 50% noisy labels, augmenting 50% of the training data outperforms full data augmentation.
P ROBLEM F ORMULATION
We begin by formally describing the problem of learning from augmented data.
Consider a dataset D train =
( X train , y train ) , where X train = ( x 1 , · · · , x n
) ∈ R d × n is the set of n normalized data points x i ∈ [0 , 1] d , from the index set V , i.e., i ∈ V
= { 1 , · · · , n } , and y train =
( y 1 , · · · , y n ) ∈ { y ∈ { ν 1 , ν 2 , · · · , ν C }} with { ν j } Cj =1 ∈ [0 , 1] .
Assume at every step t during training, we have a set of augmented examples D t aug generated by a set of label-invariant transformations.
In particular, following (Rajput et al., 2019) we model data augmentation as an arbitrary bounded additive perturbation (cid:15) , with (cid:107) (cid:15) (cid:107)≤ (cid:15) 0 .
Formally, for a given upper bound (cid:15) 0 and the set of all possible transformations A , we study the transformations selected from S ⊆
A satisfying
[Equation]
Under the smoothness constraint of images, where adjacent pixels have close intensities, such as small translations, crops, rotations, and for other pixel-wise augmentation methods such as sharpening, blurring, and color distortions (Cubuk et al., 2020), (cid:15) 0 is small.
This model is also especially suitable for modelling state-of-the-art augmentation techniques such as structured adversarial perturbation (Luo et al., 2020), in which pixel intensities are changed minimally.
While in our analysis we focus on small perturbations ε 0 , our experiments use variety of strong and weak augmentations.
Although other transformations such as data synthesis (Baluja & Fischer, 2017; Mirza & Osindero, 2014), semantic augmentation (Wang et al., 2019), large translations, crops, rotations, and ﬂips are still valid under the additive perturbation model, they can be more effectively modelled using matrices of linear transforms (Wu et al., 2020), as we analyze in Appendix B.3.
In practice, multiple augmentations are generated for each example x i , and each augmented data point can be a combination of multiple transformations, e.g. random cropping and rotating followed by horizontal ﬂipping.
The set of augmentations at iteration t generating r augmented examples per data point can be speciﬁed, with abuse of notation, as D taug = { (cid:83) ri =
( T ti ( X train ) , y train ) } , where |D taug | = rn and T ti ( X train ) transforms all the training data points with the set of transformations
T ti ⊂ S at iteration t .
We denote X taug = { (cid:83) ri =1 T ti ( X train ) } and y taug = { (cid:83) ri =1 y train } .
Let f ( W , x ) be an arbitrary neural network with m vectorized (trainable) parameters W ∈ R m .
We assume that the network is trained using (stochastic) gradient descent with learning rate η to minimize the squared loss L over the original and augmented training examples D t = {D train ∪ D taug } with associated index set V t , at every iteration t :
[Equation]
The gradient update at iteration t is given by
[Equation]
where X t = { X train ∪ X taug } and y t = { y train ∪ y taug } are the set of original and augmented examples and their labels, J ( W , X ) ∈ R n × m is the Jacobian matrix associated with f , and r t = f ( W t , X ) − y is the residual.
We further assume that J is smooth with Lipschitz constant L :
[Equation]
This trivially holds for linear models, and when W is bounded, it holds for deep ReLU, and generally for networks with any activation φ with bounded derivatives φ (cid:48) and φ (cid:48)(cid:48)
(Jordan & Dimakis, 2020).
Under this assumption, augmentation as deﬁned in Eq.
(1) results in bounded perturbations to the
Jacobian matrix.
I.e., for any transformation T j ∈ S , we have (cid:107)J ( W , x i ) − J ( W , T j ( x i )) (cid:107)≤ L(cid:15) 0 .
Using the shorthand notations J = J ( W , X train ) and
˜ J
= J ( W , T j ( X train )) , we obtain ˜ J
= J + E , where E is the perturbation matrix with (cid:107) E (cid:107) 2 ≤ (cid:107) E (cid:107) F ≤ √ nL(cid:15) 0 .
D ATA A UGMENTATION S PEEDS U P L EARNING
In this section, we analyze the effect of data augmentation on training dynamics of neural networks, and show that data augmentation can provably speed up learning.
To do so, we leverage the recent results that characterize the training dynamics based on properties of neural network Jacobian matrix and the corresponding Neural Tangent Kernel (NTK)
(Jacot et al., 2018) deﬁned as Θ =
J ( W , X ) J ( W , X ) T .
Formally (Arora et al., 2019):
[Equation]
where Θ = U Λ U T
= (cid:80) i =
λ i u
i
u
Ti is the eigendecomposition of the NTK.
Although the constant
NTK assumption holds only in the inﬁnite width limit, Lee et al.
(2019) found close empirical agreement between the NTK dynamics and the true dynamics for wide but practical networks, such as wide ResNet architectures (Zagoruyko & Komodakis, 2016).
Eq.
(5) shows that training dynamics depend on the alignment of the NTK with the residual vector at every iteration t .
In particular, shrinkage of residuals along the directions associated with larger eigenvalues of the NTK is fast and happens early during the training, while learning along the space associated with the small eigenvalues is slow and happens later.
In the following, we prove that for small perturbations (cid:15) 0 , data augmentation speeds up training by enlarging smaller eigenvalues of the NTK, while decreasing larger eigenvalues with a high probability.
Intuitively, this can be characterized as decreasing the learning rate for dimensions with larger gradient and slightly increasing the learning in dimensions with smaller gradients, and having a regularization effect by slightly perturbing the eigenvectors.
[Figure]
3. E FFECT OF D ATA A UGMENTATION ON THE E IGENVALUES OF THE NTK
We ﬁrst investigate the effect of data augmentation on the singular values of the Jacobian, and use this result to bound the change in the eigenvalues of the NTK.
To characterise the effect of data augmentation on singular values of the perturbed Jacobian ˜ J , we rely on Weyl’s theorem (Weyl, 1912) stating that under bounded perturbations E , no singular value can move more than the norm of the perturbations.
Formally, | ˜ σ i − σ i |≤ (cid:107) E (cid:107) 2 , where ˜ σ i and σ i are the singular values of the perturbed and original Jacobian respectively.
Crucially, data augmentation affects larger and smaller singular values differently.
Let P be orthogonal projection onto the column space of J T , and
P ⊥ =
I − P be the projection onto its orthogonal complement subspace.
Then, the singular values of the perturbed
Jacobian ˜ J T are ˜ σ 2 i =
( σ i + µ i ) 2 + ζ 2 i , where | µ i |≤ (cid:107) P E (cid:107) 2 , and σ min ( P ⊥ E ) ≤ ζ i ≤ (cid:107) P ⊥ E (cid:107) 2 , σ min the smallest singular value of J T (Stewart, 1979).
Since the eigenvalues of the projection matrix P are either 0 or 1, as the number of dimensions m grows, for bounded perturbations we get that on average µ 2 i
= O (1) and ζ 2 i
= O ( m ) .
Thus, the second term dominates and increase of small singular values under perturbation is proportional to √ m .
However, for larger singular values, ﬁrst term dominates and hence ˜
σ i − σ i ∼ = µ i .
Thus in general, small singular values can become signiﬁcantly and proportionally larger, while larger singular values remain relatively unchanged.
Empirically, we observed that largest singular values often decrease by data augmentation.
Fig.
2 shows the effect of data augmentation on singular values of the Jacobian matrix for a 1 hidden layer MLP trained on MNIST, ResNet32 trained on SVHN, and ResNet20 trained on CIFAR10.
As calculating the entire Jacobian spectrum is computationally prohibitive, data is subsampled fromclasses.
For all datasets, we used the data augmentation techniques described in the Experiments section, such as rotation, translation, contrast, brightness.
It can be observed that for less diverse datasets such as MNIST that have smaller singular values in general, data augmentation can easily enlarge the singular values.
On the other hand, for more diverse datasets such as CIFAR10 that generally have larger singular values, augmentation cannot change the singular values considerably.
In both cases largest singular values became smaller.
The following Lemma characterizes the expected change to the eignvalues of the NTK.
Lemma 3.1 Data augmentation as additive perturbations bounded by small (cid:15) 0 results in the following expected change to the eigenvalues of the NTK:
[Equation]
(6) where p i := P ( ˜ σ i − σ i < 0) is the probability that σ i decreases as a result of data augmentation.
All the proofs can be found in the Appendix.
From Lemma 3.1 and Eq.
(5) we see that augmentation can speed up learning at every epoch by σ i (1 − 2 p i ) (cid:107) E (cid:107) + (cid:107) E (cid:107) 2 / 3 along dimensions with smaller singular values for which p i ≤ 0 . 5 , and by (cid:107) E (cid:107) 2 / 3 along dimensions with for which singular values do not change ( p i ≈ 0 . 5 ).
It can also speed up learning along dimensions for which singular values decrease ( p i > 0 . 5 ) as long as (cid:107) E (cid:107)≥ 3 σ i (2 p i − 1) .
This is mainly relevant for smaller singular values under strong augmentation.
3. E FFECT OF D ATA A UGMENTATION ON THE E IGENVECTORS OF THE NTK
Next, we focus on characterizing the effect of data augmentation on the eigenspace of the NTK.
Let the singular subspace decomposition of the Jacobian be J = U Σ V T .
Then for the NTK, we have Θ =
J J T =
U Σ V T V Σ U T =
U Σ 2 U T (since V T V = I ).
Hence, the perturbation of the eigenspace of the NTK is the same as perturbation of the left singular subspace of the Jacobian J .
Suppose σ i are singular values of the Jacobian.
Let the perturbed Jacobian be ˜
J = J + E , and denote the eigengap γ
= min { σ i − σ i
+1 :
i = 1 , · · · , r } where σ r +1 := 0 .
Assuming γ 0 ≥ 2 (cid:107) E (cid:107) 2 , a combination of Wedin’s theorem (Wedin, 1972) and Mirsky’s inequality (Mirsky, 1960) (the counterpart of Weyl’s inequality (Weyl, 1912) for singular values) implies that
[Equation]
This result provides an upper-bound on the change of every left singular vectors of the Jacobian.
However as we discuss below, data augmentation affects larger and smaller singular directions differently.
To see the effect of data augmentation on every singular vectors of the Jacobian, let the subspace decomposition of Jacobian be J = U Σ V T
=
U s Σ s V
Ts + U n Σ n V Tn , where U s associated with nonzero singular values, spans the column space of J , which is also called the signal subspace, and U n , associated with zero singular values ( Σ n = 0 ), spans the orthogonal space of U s , which is also called the noise subspace.
Similarly, let the subspace decomposition of the perturbed Jacobian be ˜ J
=
˜
U ˜ Σ
˜ V T = ˜
U s ˜ Σ s ˜ V Ts + ˜ U n ˜ Σ n ˜ V Tn , and ˜ U s =
U s + ∆ U s , where ∆ U s is the perturbation of the singular vectors that span the signal subspace.
Then the following general ﬁrst-order expression for the perturbation of the orthogonal subspace due to perturbations of the Jacobian characterize the change of the singular directions:
∆ U s =
U n U Tn EV s Σ − 1 s (Li et al., 1993).
We see that while singular vectors associated to larger singular values are more robust to data augmentation, singular vectors associated with small singular values are more affected.
We also note that singular vectors are more robust to perturbations than singular values.
3. A UGMENTATION I MPROVES T RAINING AND G ENERALIZATION
As discussed, data augmentation increases the smaller eigenvalues of the NTK with a high probability, and is likely to slightly decrease the largest eigenvalues.
Besides, eigenvectors (in particular those associated with larger eigenvalues) are generally more robust than eigenvalues.
The following Theorem characterizes the expected improvement in the training dynamics resulted by data augmentation.
Theorem 3.2 Gradient descent with learning rate η applied to a neural network with constant NTK and Lipschitz constant L , and data points D aug augmented with r additive perturbations bounded by (cid:15) 0 as deﬁned in Eq.
(1) results in the following training dynamics:
[Equation]
(8) where E with (cid:107) E (cid:107)≤ √ nL(cid:15) 0 is the perturbation to the Jacobian, and p i :=
P ( ˜ σ i − σ i < 0) is the probability that σ i decreases as a result of data augmentation.
Corollary 3.3 Under the same assumptions as in Theorem 3.2 let σ min be the minimum singular value of Jacobian J associated with training data X train , then probability 1 − δ generalization error of the network trained with gradient descent on augmented data X aug enjoys the following bound:
[Equation]
M OST E FFECTIVE S UBSETS FOR D ATA A UGMENTATION
In Sec.
3 we discussed the effect of data augmentation on changing the alignment of the NTK with the residual, and how it improves training.
Here, we focus on identifying the most effective subsets for data augmentation.
Our key idea is to ﬁnd subsets of data points that when augmented, closely capture the alignment of the NTK (or equivalently the Jacobian) corresponding to the full augmented data with the residual vector, J ( W t , X taug ) T r taug .
If such subsets can be found, augmenting only the subsets will change the NTK and its alignment with the residual in a similar way as that of full data augmentation, and will result in similar improved training dynamics.
Effectively, such augmented subsets can be trained on along with the full non-augmented data, or without it to further speed up training.
However, generating the full set of transformations X taug is often very expensive.
This is exacerbated in the case of strong augmentations.
Hence, generating the transformations, and then extracting the subsets may not provide a considerable overall speedup.
In the following, we show that weighted subsets (coresets) S that closely estimate the alignment of the
Jacobian associated to the original data with the residual vector J T ( W t , X train ) r train can closely estimate the alignment of the Jacobian of the full augmented data and
the corresponding residual
J T ( W t , X taug ) r taug .
Thus, the most effective subsets for augmentation can be directly found from the training data.
Formally, subsets S t ∗ weighted by γ tS that capture the alignment of the full Jacobian with residual by and error of at most ξ can be found by solving the following optimization problem:
[Equation]
Solving the above optimization problem is NP-hard.
However, as we discuss in the Appendix A.6, a near optimal subset can be found by minimizing the Frobenius norm of a matrix G S , in which the i th row contains the euclidean distance between data point i and its closest element in the subset S , in the gradient space.
Formally, [ G S ] i. =
min j (cid:48) ∈ S (cid:107)J T ( W t , x i ) r i − J T ( W t , x j
[Equation]
) r j
[Equation]
(cid:107) .
Intuitively, such subsets contains the set of medoids of the dataset in the gradient space, where the medoids of a dataset are deﬁned as the most centrally located elements in the dataset (Kaufman et al., 1987).
The weight of every element j ∈ S is the number of data points closest to it in the gradient space, i.e., γ j = (cid:80) i ∈
[Equation] [Equation]
The set of medoids can be found by solving the following submodular 1 cover problem:
[Equation]
where C ≥ (cid:107) G S (cid:107) F is a constant.
The classical greedy algorithm provides a logarithmic approximation for the above submodular maximization problem, i.e., | S |≤ (1+ Ln ( n )) .
It starts with the empty set S 0 = ∅ , and at each iteration τ , it selects the training example e ∈ V that maximizes the marginal gain F ( e | S τ )
=
F ( S τ ∪ { e } ) − F ( S τ ) .
Formally, S τ = S τ − 1 ∪ { arg max e ∈ V F ( e | S τ − 1 ) } .
The
O ( nk ) computational complexity of the greedy algorithm can be reduced to O ( n ) using randomized methods (Mirzasoleiman et al., 2015), and further improved using lazy evaluation (Minoux, 1978) and distributed implementations (Mirzasoleiman et al., 2013).
The rows of the matrix G can be efﬁciently upper-bounded using the gradient of the loss w.r.t.
the input to the last layer of the network, which has been shown to capture the variation of the gradient norms closely (Katharopoulos & Fleuret, 2018).
The above upper-bound is only marginally more expensive than calculating the value of the loss, and hence the subset can be found efﬁciently.
Better approximations can also be obtained by considering earlier layers in addition to the last two, at the expense of greater computational cost.
At every iteration t during training, we select a coreset from every class c ∈
[ C ] separately, and apply the set of transformations { T ti } ri =1 only to the elements of the coresets, i.e., X taug = {∪ ri =1 T ti ( X S t ) } .
We divide the weight of every element j in the coreset equally among its transformations, i.e. the ﬁnal weight ρ tj = γ tj /r if j ∈ S t .
We apply the gradient descent updates in Eq.
(3) to the weighted
Jacobian matrix of X t = X taug or X t
=
{ X train ∪ X taug } (viewing ρ t as ρ t ∈ R n ) as follows:
[Equation]
[Figure]
The pseudo code is given in Alg.
The following Lemma upper-bounds the difference between the alignment of the Jacobian and residual for augmented coreset vs. full augmented data.
Lemma 4.1 Let S be a coreset that captures the alignment of the full data NTK with residual with an error of at most ξ as in Eq.
Augmenting the coreset with perturbations bounded by (cid:15) 0 ≤ n 32 √ L captures the alignment of the fully augmented data with the residual by an error of at most
[Equation]
Importantly, since the augmented coresets capture the alignment of the Jacobian of a fully augmented data with residual, they can be trained on in absence of the full data to gain more speedup.
This is not the case for existing methods such as max-loss.
Besides, the gradient of noisy labeled data points do not cluster in the gradient space (Mirzasoleiman et al., 2020b).
Thus, only clean data points are selected for augmentation.
This results in an improved performance, as we will show in Sec. 5.1.
4. C ORESET VS . M AX - LOSS S TRATEGY FOR D ATA A UGMENTATION
In the initial phase of training the NTK goes through rapid changes, which determines the ﬁnal basin of convergence and network’s ﬁnal performance (Fort et al., 2020).
Regularizing deep networks by weight decay or data augmentation mainly affects this initial phase and matters little afterwards (Golatkar et al., 2019).
Crucially, augmenting coresets that closely capture the alignment of the NTK with the residual during this initial phase results in a considerable increase in the speed of convergence and generalization performance.
On the other hand, augmenting points with maximum loss early in training decreases the alignment between the NTK and the label vector and slows down learning and convergence.
Fig.
3b illustrates the superior performance of coreset vs max-loss augmentation.
After this initial phase (highlighted in green in Fig. 3a) when the network have a good prediction performance, the gradients for the majority of data point becomes very small.
Here, the alignment is mainly captured by the elements with the maximum loss.
Hence, as training proceeds, the intersection between the elements of the coresets and examples with maximum loss increases ( c.f. Fig. 3a).
The following Theorem characterizes the training dynamics of training on the full data and the augmented coresets, using the additive perturbation model in Eq.
Theorem 4.2 Let L i be β -smooth, L be λ -smooth and satisfy the α -PL condition, that is for α > 0 , (cid:107)∇L ( W ) (cid:107) 2 ≥ α L ( W ) for all weights W .
Let f be Lipschitz in X with constant L (cid:48) , and ¯ L = max { L, L (cid:48) } .
Let G 0 be the gradient at initializaion, σ max the maximum singular value of the coreset
Jacobian at initialization.
Choosing (cid:15) 0 ≤ 1 σ max √ ¯ Ln and running SGD on full data with augmented coreset using constant step size η = αλβ , we get the following convergence bound:
[Equation]
[Table]
The above Theorem shows that training on full data and augmented coresets converges with the same rate as of training on the fully augmented data, to a close neighborhood of the optimal solution.
The size of the neighborhood depends on (1) the error of the coreset ξ in Eq.
(4), and (2) the error in capturing the alignment of the full augmented data with the residual derived in Lemma 4.1.
The ﬁrst term decrease as the size of the coreset grows, and the second term depends on the network structure.
We also analyze convergence of training only on the augmented coresets, and augmentations modelled as arbitrary linear transformations using a linear model (Wu et al., 2020) in Appendix B.1, B.3.
E XPERIMENTS
Setup and baselines.
We evaluate the performance of our approach in two different settings.
First, we consider the case where full data is small enough to be trained on.
Here, we study the effect of augmenting coresets on the generalization performance of clean and noisy labeled data.
Next, we focus on the case where the full training data is too large to be trained on.
In this setting, we train only on the augmented coresets.
In both cases we use subsets with maximum loss, and random subsets as baselines.
For all methods, we select a new augmentation subset every R epochs.
We note that while the original method of (Kuchnik & Smith, 2018) selects points based on loss of a fully trained model, to maximize fairness, our max-loss baseline selects a new subset at every subset selection step.
Data and augmentation.
We apply our method to training ResNet20 and Wide-Resnet-28-10 on
CIFAR10, and ResNet32 on CIFAR10-IMB (Long-Tailed CIFAR10 with Imbalance factor offollowing (Kim & Kim, 2020)) and SVHN datasets.
All models are trained for 200 epochs using
SGD with 0.9 momentum and learning rate warm-up.
We use the strong augmentation proposed by (Wu et al., 2020) to generate 4 distinct augmented examples by randomly sampling 2 augmentations from the same set used by (Cubuk et al., 2019; 2020) to apply to each example.
A new set of default augmentations (random crop and horizontal ﬂip) are also applied every R epochs to the original data.
We individually measure average time taken for subset selection, gradient descent, and subset augmentation per epoch to compute total training time and speedups.
All results are averaged overruns using an Nvidia A40 GPU, and full results are reported in the Appendix.
5. T RAINING ON F ULL D ATA AND A UGMENTED S UBSETS
Table 1 demonstrates the accuracy improvement resulted by augmenting subsets of size 5%, 10% and 30% selected by our method vs. max-loss and random over full data augmentation.
We observe that augmenting coresets effectively improves generalization, and outperforms augmenting random (a) CIFAR
[Figure]
[Table] [Table]
[Table]
and max-loss subsets across different models and datasets.
Fig.
5 shows the speedup vs. test accuracy of augmenting subsets of different size selected from CIFAR10 and SVHN by our method vs baselines.
We see that our method outperforms the baseline while providing a similar speedup as that of max-loss.
On SVHN, we achieve 80 .
% of the augmentation accuracy gain by augmenting only 30% size subsets while providing 2.2x speedup, while max-loss only achieves 76 .
4% with the same speedup.
In the Appendix, we also show that our approach is effective when trained with full data even for small augmentation subset sizes of 0 .
2% and 0 .
5% selected every epoch.
Augmenting noisy labeled data.
Table 2 shows the result of augmenting coresets vs. max-loss and random subsets of different sizes selected from CIFAR10 with 50% label noise for training ResNet20.
Note that augmenting coresets selected by our method not only outperforms max-loss and random, but provides a superior performance over full data augmentation.
This clearly shows the effectiveness of the coresets in capturing the alignment of the NTK with the residual of clean data points.
5. T RAINING ONLY ON AUGMENTED SUBSETS IN ABSENCE OF FULL DATA
We now evaluate the effectiveness of our approach for training only on the augmented coresets.
Our main goal here is to show the accuracy improvement resulted from augmenting the coresets.
In particular instead of focusing on the quality of the coresets, we aim at showing the effectiveness of the coresets for augmentation.
Table 3 shows the test accuracy for training ResNet20 and Wide-ResNet on CIFAR10 when we only train on small augmented coresets of size 0 .
1% to 1% selected at every epoch ( R = 1 ).
We see that while the non-augmented coreset performs worse than random in most cases, the augmented coresets outperform augmented random subsets by a large margin.
That is, the improvement from augmenting the coresets is signiﬁcantly larger than random data points.
This clearly shows the effectiveness of augmenting the coresets, and the importance of capturing the alignment of the NTK with the residual for data augmentation.
Note that training on the augmented max-loss points did not even converge in absence of full data.
Finally, Fig.
3c shows the fraction of data selected for augmentation during training of ResNet20 on CIFAR10.
We see that the coresets achieve a superior performance by carefully selecting certain data points more often than others.
C ONCLUSION
We showed that data augmentation improves training and generalization by enlarging the smaller singular values of the neural network Jacobian.
Then, we proposed a framework to iteratively extract small subsets of training data that when augmented, closely capture the alignment of the fully augmented Jacobian with the label/residual vector.
We showed the effectiveness of augmenting the coresets selected by our method to provide a superior generalization performance when added to the full data, in presence of noisy labels, or as a standalone subset.
R EPRODUCIBILITY S TATEMENT
Our experimental setup along with training details are introduced in Section 5.
Full results containing standard deviations can be found in Appendix C. All code use for our empirical results are also available in the supplementary material, with along with instructions on how to run them.
All proofs for non-trivial theoretical results in the main paper can be found in Appendix A, while additional theoretical results and their respective proofs can be found in Appendix B
S UPPLEMENTARY M ATERIAL : D ATA -E FFICIENT A UGMENTATION FOR T RAINING N EURAL N ETWORKS A P ROOF OF M AIN R ESULTS A. P ROOF FOR L EMMA 3.
Proof Let δ i :=
˜ σ i − σ i , where P ( δ i < 0)
=
p i .
Assuming uniform probability between −(cid:107) E (cid:107) to 0 , and between 0 to (cid:107) E (cid:107) , we have pdf ρ i ( x ) for δ i :
ρ i ( x ) =
  p i (cid:107) E (cid:107) , if − (cid:107) E (
cid:107)≤ x <1 − p
i (cid:107) E (cid:107) , 0 ≤ x ≤ (cid:107) E (cid:107) 0 , otherwise (15)
Taking expectation,
E (˜ σ i − σ i ) =
E ( δ i ) =
∞ (cid:90)
−∞ xρ i ( x ) dx (16)
=
(cid:90) −(cid:107)
E (cid:107) x
p i (cid:107) E (cid:107) dx + (cid:90) (cid:107) E (cid:107) x
1 − p
i (cid:107) E (cid:107) dx (17) = −(cid:107) E (cid:107) p
i + (1 − p i ) (cid:107) E (cid:107) (18) =
(1 − 2 p i ) (cid:107) E (cid:107) (19)
We also have
E ( δ 2 i ) = ∞ (cid:90)
−∞ x 2 ρ i ( x ) dx (20) = 0 (cid:90) −(cid:107) E (cid:107) x 2 p i (cid:107) E (cid:107) dx + (cid:90) (cid:107)
[Equation]
Thus
[Equation]
we have
[Equation]
A. P ROOF OF T HEOREM 3.
Using Jensen’s inequality
[Equation]
we have
[Equation]
A. P ROOF OF C OROLLARY 3.
Under the assumptions of Theorem 5.1 of Arora et al.
(2019), i.e. where the minimum eigenvalue of the NTK is λ min ( J J T ) ≥ λ 0 for a constant λ 0 > 0 , and training data X of size n sampled i.i.d.
from distribution D and 1-Lipschitz loss L , we have that with probability δ/ 3 , training the over-parameterized neural network with gradient descent for t ≥ Ω (cid:16) 1 nλ 0 log nδ (cid:17) iterations results in the following population loss L D (generalization error)
[Equation]
with high probability of at least 1 − δ over random initialization and training samples.
Hence, using λ min , σ min to denote minimum eigen and singular value respectively of the NTK corresponding to full data, we get
[Equation]
For augmented dataset X aug , we have ˜ σ i ≤ σ i + √ nL(cid:15) 0 , hence the improvement in the generalization error is at most
[Equation]
Combining these two results, we obtain Corollary 3.3.
A. P ROOF OF L EMMA 4. Proof
[Equation]
Applying deﬁnition of coresets
[Equation]
we obtain
[Equation]
A. P ROOF OF T HEOREM 4.
Proof In this proof, as shorthand notation
[Equation]
we use X f and X train interchangeably.
We further use
X c to represent the coreset selected from the full data, and X c aug to represent the augmented coreset.
By Theorem 1 of Bassilyet al.
(2018) , under the α -PL assumption for L and interpolation assumption (i
[Equation]
e. for every sequence W 1 , W 2 , . . .
such that lim t →∞ L ( W t , X ) = 0 , we have that the loss for each data point lim t →∞ L ( W t , x i ) = 0 ), the convergence of SGD with constant step size is given by
[Equation]
Using Jensen’s inequality, we have
[Equation]
A. F INDING S UBSETS
Let S be a subset of training data points.
Furthermore, assume that there is a mapping π w,S : V → S that for every W assigns every data point i ∈ V to its closest element j ∈ S ,
i
[Equation]
e. j = π w,S ( i ) =
[Equation]
max j (cid:48) ∈ S s ij (cid:48) ( W ) , where s ij ( W ) =
C −
[Equation]
W
[Equation]
, x
[Equation]
W
[Equation]
x
[Equation]
is the similarity between gradients of i and j , and C ≥ max ij s ij ( W ) is a constant.
Consider a matrix G π w,S ∈
R n × m , in which every row i contains gradient of π w ( i ) , i.e., [ G π w,S ] i. =
J T ( W t , x π w,S ( i ) )
r π w,S ( i ) .
The Frobenius norm of the matrix G π w provides an upper-bound on the error of the weighted subset S in capturing the alignment of the residuals of the full training data with the Jacobian matrix.
Formally,
[Equation]
where the weight vector γ S t ∈ R | S | contains the number of elements that are mapped to every element j ∈ S by mapping π w,S , i.e. γ j = (cid:80) i ∈ V 1 [ π w,S ( i )
=
j ] .
Hence, the set of training points that closely estimate the projection of the residuals of the full training data on the Jacobian spectrum can be obtained by ﬁnding a subset S that minimizes the Frobenius norm of matrix G π w,S .
B A DDITIONAL T HEORETICAL R ESULTS B. C ONVERGENCE ANALYSIS FOR TRAINING ON THE CORESET AND ITS AUGMENTATION
Theorem B.1 Let L i be β -smooth, L be λ -smooth and satisfy the α -PL condition, that is for α > 0 , (cid:107)∇L ( W , X ) (cid:107) 2 ≥ α L ( W , X ) for all weights W .
Let ξ upper-bound the normed difference in gradients between the weighted coreset and full dataset.
Assume that the network f ( W , X ) is
Lipschitz in W , X with Lipschitz constant L and L’ respectively, and ¯ L = max { L, L (cid:48) } .
Let Gthe gradient over the full dataset at initialization, σ max the maximum Jacobian singular value at initialization.
Choosing perturbation bound (cid:15) 0 ≤
[Equation]
Ln where σ max is the maximum singular value of the coreset Jacobian and n is the size of the original dataset, running SGD on the coreset and its augmentation using constant step size η = αλβ , we get the following convergence bound:
[Equation]
where X c + c aug represents the dataset containing the (weighted) coreset and its augmentation.
Proof As in the proof for Theorem 4.2, we begin with the following inequality
[Equation]
Thus
[Equation]
we can write
[Equation] [Equation]
The rest of the proof is similar to that of Theorem 4 . 2 .
B. L EMMA FOR EIGENVALUES OF CORESET
The following Lemma characterizes the sum of eigenvalues of the NTK associated with the coreset.
Lemma B.2 Let ξ be an upper bound of the normed difference in gradient of the weighted coreset and the original dataset,
i
[Equation]
e. for full data X and its corresponding coreset X S with weights γ S , and respective residuals r , r S , we have the bound (cid:107)J T ( W t , X ) r t − γ S J T ( W t , X S ) r tS (cid:107)≤ ξ .
Let { λ i } ki =1 be the eigenvalues of the NTK associated with the coreset.
Then we have that
[Equation]
Proof Let singular values of coreset Jacobian be σ i .
Let J T ( W t , X ) r t = γ S J T ( W t , X S ) r tS + ξ S where (cid:107) ξ S (cid:107)≤ ξ .
[Equation]
We can make the following observations: For overparameterized networks, with bounded activation functions and labels, e.g. softmax and one-hot encoding, the norm of the residual vector is bounded, and the gradient norm is likely to be much larger than residual, especially when dimension of gradient is large.
In this case, the Jacobian matrix associated with small weighted coresets found by solving
Eq.
(11), have large singular values.
B. A UGMENTATION AS L INEAR T RANSFORMATION : L INEAR M ODEL A NALYSIS
We introduce a simpliﬁed linear model to extend our theoretical analysis to augmentations modelled as linear transformation matrices F applied to the original training data.
These augmentations are also originally studied by Wu et al.
In this section, we speciﬁcally study the effect of these augmentations using a linear model when applied to coresets.
B. L INEAR CASE (1 INPUT LAYER , 1 OUTPUT LAYER )
Lemma B.3 (Augmented coreset gradient bounds: Linear) Let f be a simple linear model with weights W ∈ R d × C where f ( W , x i )
=
W T x i , trained on mean squared loss function L .
Let
F ∈ R d × d be a common linear augmentation matrix with norm (cid:107) F (cid:107) with augmentation x
aug i given by F x i .
Let coreset be of size k and full dataset be of size n .
Further assume that the predicted label of x i and its augmentation F x i are sufﬁciently close, i.e. there exists ω such that W T ( F x i ) =
[Equation]
i
[Equation]
z
[Equation] [Equation]
z i (cid:107)≤ ω ∀ i .
Let ξ upper-bound the normed difference in gradients between the weighted coreset and full dataset.
Then, the normed difference between the gradient of the augmented full data and augmented coreset is given by
[Equation]
for some (small) constant ξ .
Proof By our assumption, we can begin with,
[Equation]
Furthermore, by Mirzasoleiman et al. (2020a), we know that sum of the coreset weights γ s j is given by (cid:80) k
=1 j =1 γ s j ≤ n .
Hence,
[Equation]
i ∈
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
i
[Equation] [Equation]
Corollary B
[Equation]
In the simpliﬁed linear case above
[Equation]
the difference in gradients of the full training data with its augmentations ( ∇L ( W , X f + aug ) ) and gradients of the coreset with its augmentations ( ∇L ( W , X c + c aug ) ) can be bounded by
[Equation]
Proof Applying Eq.
(79) and Lemma B
[Equation] [Equation]
we obtain
[Equation]
Theorem B.5 (Convergence of linear model) Let f be a linear model with weights W and augmentation be represented by the common linear transformation F .
Let L i be β -smooth, L be λ -smooth and satisfy the α -PL condition, that is for α > 0 , (cid:107)∇L ( W , X ) (cid:107) 2 ≥ α L ( W , X ) for all weights W .
Let ξ upper-bound the normed difference in gradients between the weighted coreset and full dataset and ω bound W T ( F x i )
=
W T x
i
+ z
i , (cid:107) z i (cid:107)≤ ω ∀ i .
Let G (cid:48) 0 be the gradient over the full dataset and its augmentations at initialization.
Then, running SGD on the size k coreset with its augmentation using constant step size η = αλβ , we get the following convergence bound:
[Equation]
Proof From Bassilyet al.
(2018) , we have
[Equation]
Using Jensen’s inequality, we have
[Equation]
where the last inequality follows from applying Corollary B.4.
C A PPENDIX C: E XTRA EXPERIMENTS C. F ULL R ESULTS FOR T ABLE
This section contains full experiment results including standard deviations and the full augmentation benchmark for Table 1 .
Augmenting coresets of size 10% achieves 51% of the improvement obtained from augmentation of the full data and further enjoys a 6x speedup in training time on CIFAR10.
This speedup becomes more signiﬁcant when using strong augmentation techniques which are mostly computationally demanding, especially when applied to the entire dataset.
[Table]
C. S PEED - UP MEASUREMENTS
We measure the improvement in training time in the case of training on full data and augmenting subsets of various sizes.
While our method yields similar or slightly lower speed-up to the max-loss policy and random approach respectively, our resulting accuracy outperforms these two approaches.
For example, for SVHN/Resnet32 using 30% coresets, we sacriﬁce 10% of the speed-up to obtain an additional 24 .
8% of the gain in accuracy from full data augmentation when compared to a random subset of the same size.
[Table]
T RAINING ON FULL DATA AND AUGMENTING SMALL SUBSETS RE - SELECTED EVERY EPOCH
We apply our proposed method to select a new subset for augmentation every epoch
(i
[Table]
e. using
R = 1 ) and compare our results with other approaches using accuracy and percentage of data not selected (NS).
We see that while the max-loss policy selects a small fraction of data points over and over and random uniformly selects all the data points, our approach successfully ﬁnds the smallest subset of data points that are the most crucial for data augmentation.
Hence, it can achieve a superior accuracy than max-loss policy, while augmenting only slightly more examples.
This conﬁrms the data-efﬁciency of our approach.
This is especially evident when using coresets of size 0 .
Furthermore, despite the random baseline using a signiﬁcantly larger percentage of data, it is outperformed by our approach in both data-efﬁciency and accuracy.
We emphasize that results in this table is different from that of Table
[Table]
as default augmentations on the full training data are performed once every R = 1 epochs instead of every R = 20 epochs.
Since selecting subsets at every epoch can be computationally expensive, we only perform these experiments on small coresets and hence still enjoy good speedups compared to full data augmentation.
This shows that our approach is still effective at very small subset sizes, hence can be computationally efﬁcient even when subsets are re-selected every epoch.
[Table]
C. A DDITIONAL VISUALIZATIONS FOR TRAINING ON CORESETS AND ITS AUGMENTATIONSM EASURING TRAINING DYNAMICS OVER TIME
We include additional visualizations in Figure 5 for training on coresets and its augmentations as supplementary plots to Figure 3c and Table 3.
We plot metrics obtained during each point (epoch) of the training process based on percentage of data selected/used and test accuracy achieved.
All metrics are averaged over 5 runs and obtained using R = 1 .
These plots demonstrate that coreset augmentation approaches outperform random augmentation baselines throughout the training process.
Furthermore, they show that augmentation of coresets result in a larger increase in test accuracy compared to augmentation of randomly selected training examples, especially for small subset sizes.
[Figure]
C. I NTERSECTION OF MAX - LOSS POLICY AND CORESETS
Figure 3 a depicts the increase in intersection between max-loss subsets and coresets over time.
In addition, we also aggregate 30% subsets selected every R = 20 epochs using both approaches over the entire training process to compute intersection between the top N selected data points.
Our plots in Figure 6 suggest that a similar pattern holds in this setting.
We also qualitatively visualize this in
Figure 7.