A DA A UG : L EARNING C LASSAND I NSTANCEADAPTIVE D ATA A UGMENTATION P OLICIES
A BSTRACT
Data augmentation is an effective way to improve the generalization capability of modern deep learning models.
However, the underlying augmentation methods mostly rely on handcrafted operations.
Moreover, an augmentation policy useful to one dataset may not transfer well to other datasets.
Therefore, Automated
Data Augmentation (AutoDA) methods, like AutoAugment and Population-based
Augmentation , have been proposed recently to automate the process of searching for optimal augmentation policies.
However, the augmentation policies found are not adaptive to the dataset used, hindering the effectiveness of these AutoDA methods.
In this paper, we propose a novel AutoDA method called AdaAug to efﬁciently learn adaptive augmentation policies in a class-dependent and potentially instance-dependent manner.
Our experiments show that the adaptive augmentation policies learned by our method transfer well to unseen datasets such as the Oxford Flowers, Oxford-IIT Pets, FGVC Aircraft, and Stanford Cars datasets when compared with other AutoDA baselines.
In addition, our method also achieves a state-of-the-art performance on the CIFAR-10, CIFAR-100, and SVHN datasets.
I NTRODUCTION
Data augmentation is a common way to enhance the robustness of deep learning models by augmenting the datasets used for model training.
Applying popular data augmentation operations such as randomized cropping, horizontal ﬂipping, and color shifting to image data has become a standard procedure in modern image recognition models (Krizhevsky et al., 2012;
Shorten & Khoshgoftaar, 2019).
Over the years, various augmentation methods using advanced operations have been proposed.
Examples include occlusion-based operations like Cutout (Devries & Taylor, 2017) that randomly masks part of an image to avoid overﬁtting, label-mixing operations like CutMix (Yun et al., 2019) that replaces the occluded part in Cutout with a different image patch, and Mixup (Zhang et al., 2018) that interpolates two images with their corresponding one-hot encoded labels.
While these hand-crafted data augmentation methods can improve model generalization, choosing the operations and their corresponding parameters is often decided manually to make the augmentation scheme effective for the task at hand.
Despite the manual efforts involved, an augmentation policy that is useful for a particular dataset does not often generalize well to other datasets (Cubuk et al., 2019).
To tackle this problem, a series of recent studies has been conducted to automate the process of ﬁnding an effective data augmentation policy for a target dataset.
These automated data augmentation (AutoDA) methodologies show impressive results on several benchmark image datasets (Cubuk et al., 2019;
Ho et al., 2019; Lim et al., 2019; Cubuk et al., 2020; Hataya et al., 2020; Hendrycks et al., 2020;
Li et al., 2020; Cheung & Yeung, 2021).
The manual tuning of augmentation parameters can also be addressed by using generative approaches, such as training a generative adversarial network (GAN) to create new artiﬁcial images directly (Antoniou et al., 2017; Tran et al., 2017; Jha & Cecotti, 2020;
Yorioka et al., 2020; Zhao et al., 2020).
However, these generative models are often hard to implement and computationally expensive to train.
In supervised learning, data augmentation is considered to be a naive way to inject inductive biases, such as translational invariance, to a classiﬁer.
With the recent advances in representation learning, data augmentation has become a major approach to the learning of good representations.
For example, there is an increasing trend to replace convolutional neural networks (CNNs) with transformers in computer vision (Carion et al., 2020; Dosovitskiy et al., 2021).
Being a more generic architecture, transformers do not come with additional inductive bias such as translational invariance in CNNs, thereby requiring more data to learn the effective invariance properties.
Data augmentation serves as an effective way to achieve this goal (Touvron et al., 2020).
In addition, recent self-supervised models rely heavily on data augmentation to create different views of the same data and learn the robust representations through contrastive learning (Chen et al., 2020;
Grill et al., 2020;
He et al., 2020).
However, an improper choice of augmentation operations, especially with excessive strength, may impose a wrong inductive bias to the models and lead to performance degradation (Chen et al., 2020; Xiao et al., 2021).
Consequently, there is a need to enrich the current data augmentation methods, especially with a better choice of the data augmentation policy, to take advantage of the recent advances in representation learning.
Previous AutoDA methods attempt to ﬁnd an optimal augmentation policy to augment a given dataset.
However, most of the discovered policies are not adaptive to variations of the dataset.
Even in a dataset in which an augmentation policy is found to be effective, the same augmentation scheme is applied equally to all classes.
This can limit the potential of data diversity brought about by data augmentation.
For example, in digit classiﬁcation, ﬂip-invariance is useful for the digits “0”, “1” and “8” but not for the other digits; in shape classiﬁcation, shearing-invariance is useful for “triangles” but not for “rectangles”.
None of the previous AutoDA methods can learn an adaptive class-dependent augmentation policy.
To address this limitation, we propose AdaAug , as an AutoDA method, to learn a class-dependent and potentially instance-dependent augmentation policy efﬁciently.
Despite the attractive potential of such an adaptive scheme if it can indeed be realized, actually learning an adaptive augmentation policy poses at least two major technical challenges.
First, the search space for the per-class, per-instance augmentation policies is very large, rendering it intractable when trying to maintain an individual policy for each class or even each realization of the input data.
Second, the gradient information of the augmentation parameters is hard to obtain as the operation selection process and the transformations are non-differentiable.
Optimizing the augmentation policy efﬁciently is a challenging problem to address.
In this work, AdaAug employs a recognition model to learn the underlying augmentation policy for each data instance and takes an alternating exploit-and-explore procedure to update the augmentation policy using a differentiable workﬂow.
In the exploitation pass, AdaAug trains a classiﬁer for a number of steps, followed by the exploration pass which validates the classiﬁer and updates the policy to minimize the validation loss.
The intuition behind our design of AdaAug is that such an alternating procedure can learn augmentations that help generalize the trained model to unseen validation data.
For example, rotational invariance would be found to be a desirable inductive bias that the model should learn if the validation set contains some similar but rotated versions of the training images.
Our goal is to capture such information and assign a higher probability to use rotation for augmentation in this case.
An application scenario would be a computer vision task for drones where the unseen data may contain different kinds of rotated images.
To summarize, our contributions are listed as follows:
• We introduce a novel AutoDA method to learn a class-dependent and potentially instancedependent augmentation policy for each data instance.
• We propose a differentiable workﬂow to search for the augmentation policy efﬁciently.
• We demonstrate that the policies learned by our method transfer better to unseen datasets, such as Oxford Flowers, Oxford-IIT Pets, FGVC Aircraft, and Stanford Cars, when compared to other AutoDA baselines.
• We demonstrate a state-of-the-art performance on the CIFAR-10, CIFAR-100, and SVHN datasets.
R ELATED W ORK
Automated Data Augmentation.
Several AutoDA methods have been proposed to compose the augmentation operations automatically.
AutoAugment (Cubuk et al., 2019) learns to generate the probability and magnitude when applying different augmentation operations as a policy using reinforcement learning (RL).
It alternately generates an augmentation policy to train a child model and updates the policy generator using the validation performance as a reward.
Since it is computationally expensive to train the child models repeatedly, several techniques have been proposed subsequently in an attempt to reduce the search effort.
Fast AutoAugment (Lim et al., 2019) uses Bayesian optimization (BO) to tune the augmentation parameters.
Population-based Augmentation (PBA) (Ho et al., 2019) exploits population-based training (PBT) to search for an optimal augmentation policy schedule by training multiple parallel child models using an evolutionary approach.
RandAugment (Cubuk et al., 2020) applies the augmentation operations uniformly and reduces the search space signiﬁcantly by covering only the number of operators and the global augmentation magnitude.
Instead of using the validation performance to evaluate the augmentation quality, Adversarial AutoAugment (Zhang et al., 2020) uses an adversarial objective to learn the augmentation policy.
MODALS (Cheung & Yeung, 2021) utilizes PBA to search for an optimal latent space augmentation policy and augment data from any modality not limited to image data.
Differentiable Data Augmentation.
In addition to using RL, BO and PBT to optimize the augmentation parameters, there exist related methods that modify the otherwise discrete search procedures to make them end-to-end differentiable.
This results in a more efﬁcient optimization procedure and a more precise policy than RL and PBT as the search space is continuous.
In AdaAug , part of our contribution is to design a differentiable workﬂow to learn the augmentation policy.
For previous differentiable augmentation approaches, Faster AutoAugment (Hataya et al., 2020) proposes a differentiable Relaxed Bernoulli distribution to sample the candidate augmentation functions and estimate the gradients of the non-differentiable augmentation magnitude using the Stop Gradient estimator.
Speciﬁcally, it optimizes a density matching loss between the training and validation data.
DADA (Li et al., 2020) differentiates through the discrete policy sampling process using GumbelSoftmax trick. While AutoDA and differentiable data augmentation have been shown to be successful in improving the generalization performance of deep learning models, the learned augmentation policy is often applied uniformly to the whole dataset, meaning that all classes and instances share the same augmentation policy.
On the contrary, each class and even each data instance receives an adaptive policy to augment in our proposed method.
Adaptive Data Augmentation.
Attempts have been made to apply adaptive data augmentation at a class or subgroup level.
Hauberg et al.
(2016) model the transformations within each class and use statistical models to augment the dataset.
The approach shows improvement in the small MNIST dataset and its variants.
However, the augmentation operations are limited to spatial transformations.
In addition, observations of the data must be locatable and alignable, making it difﬁcult to extend to most other computer vision tasks.
Recently, CAMEL (Goel et al., 2021) adopts a ﬁner data-generation method by ﬁxing the classiﬁers that fail on a subgroup of a class.
It uses CycleGAN to learn different variations of the same training data within a subgroup.
However, CAMEL requires the specifying of the subgroup information manually and assumes that the subgroups only exist within the same class.
MetaAugment (Zhou et al., 2021) learns a sample-wise weighting scheme and a global probability parameter to control the sampling of augmentation transformations.
In AdaAug , the learned policy can capture class-dependent transformation automatically and also instance-dependent information, such as the light intensity of an image, across different classes.
A DA A UG 3. S EARCH S PACE
Let T be a set of augmentation operations where τ j denotes the j -th operation (e.g., “rotation”) in the set.
We formulate an augmentation policy as the probability p and magnitude λ in applying the augmentation operations.
Here, p is a probability vector with each entry p
j ∈ [0 , 1]; (cid:80)
| T | j =1 p j =and λ is a vector with each entry λ j ∈ [0 , 1] , where p j and λ j are the probability and magnitude, respectively, of applying the operation τ j .
Mathematically, τ j : X → X is a mapping from the input space X to itself.
For an image x ∈ X , τ j transforms it with the magnitude parameter λ j that 𝑓 ! 𝑓 !
𝑓 !
Training data x 𝑓 ! 𝑔 " 𝜎 𝑠𝑚 𝜏 # (𝑥,𝜆 # ) 𝜏 $ (𝑥,𝜆 $ ) 𝜏 % (𝑥,𝜆 % ) k times …
Augmented training data x’ 𝐿 !"
#$% (𝛼,𝛽) 𝑓 !
ℎ & 𝑝 𝜆 𝜏 # (𝑥,𝜆 # ) 𝜏 # (𝑥,𝜆 # ) 𝜏 # (𝑥,𝜆 # ) 𝜏 # (𝑥,𝜆 # ) 𝜏 # (𝑥,𝜆 # ) … … 𝑓 ! 𝑓 !
+ 𝜆 𝑝 𝑔 " 𝐿 &#’$( (𝛼,𝛽,𝛾)
Validation data x
Augmented validation data 𝑥. (( ! ) … speciﬁes the strength of the transformation (e.g., degree of rotation), i.e., x (cid:55)→ τ j ( x ; λ j ) . Note that some operations like ﬂipping do not depend on the magnitude parameter. In a training pass, given an input data x , an augmentation policy ( p, λ ) , and the number of operators k , we sample k operations according to p and apply them with their corresponding magnitudes speciﬁed by λ : [Equation]
Here, T ( t ) , 1 ≤ t ≤ k, denotes applying the t -th operation.
Our goal is to learn an augmentation policy function π θ : x (cid:55)→ ( p, λ ) to generate an adaptive augmentation policy that optimizes the generalization performance and is dependent on the input data (see Figure 1 and Algorithm 1).
3. S EARCH A LGORITHM
Exploitation.
AdaAug uses a feature extraction network f α : X → Z to map an input space to a latent space, a dense layer g β : Z → Y to map a latent space to a label space, and a projection function h γ : Z → P × Λ to map a latent space (representation) to a probability and magnitude space, where p =
[0 , 1] | T | and (cid:107) p (cid:107) 1 = 1 , ∀ p ∈ P and λ = [0 , 1] | T | , ∀ λ ∈ Λ .
In our case, the functions f, g, h (with the subscripts dropped for notational simplicity) are implemented as neural networks with network weights α, β, γ , respectively.
The softmax operation is applied to the ﬁrst half of the output from h to get the probabilities and the sigmoid function is applied to the other half to get the magnitudes.
The policy network π θ ( x ) can then connect the class and image information to the augmentation policy via h ◦ f ( x ) with parameters θ
= ( γ, α ) .
In an exploitation pass, given the training data x , the policy function generates the data-dependent augmentation policy ( p, λ )
=
π θ ( x ) and augments x to give ˆ x using Equation (1).
Here, ˆ x is treated as a new unseen training example and is used to train the classiﬁcation model g ◦ f
(ˆ x ) by minimizing the crossentropy loss: min α,β L train ( α, β ) .
Note that there is a discrete sampling procedure in Equation (1).
During the update, the gradient of α does not involve the computations in the policy network.
Exploration.
In the exploration pass, AdaAug generates the augmentation policy ( p, λ ) =
π θ ( x ) given the validation data x .
The | T | operations are applied to x separately with the corresponding magnitudes in λ .
The augmented validation data are passed to the feature extraction network f individually to get the latent representations, which are then summed based on their weights in the probability vector p .
The mixed representation is passed to g for computing the predicted labels:
[Equation]
AdaAug updates γ to minimize the validation loss: min γ L valid ( α, β, γ ) .
As we make no assumption that the augmentation operations are differentiable, we follow prior approaches (Bengio et al., 2013; Li et al., 2020) using a straight-through gradient estimator to optimize the augmentation magnitudes.
Speciﬁcally, the gradient of the magnitudes is estimated with respect to each input pixel value x h,w of the augmented data, i.e., ∂ ˆ x h,w ∂λ j = 1 .
The gradient can then be calculated by:
[Equation]
Algorithm
1: procedure S EARCH ( D train , D valid , T , k, m, r ) 2:
Initialize α, β, γ 3: for r steps do # Exploitation 4:
Sample a mini-batch d train ∈ D train 5: for ( x, y ) ∈
d train do 6: ( p, λ ) =
π θ ( x ) 7: ˆ x = augment
( x ) (cid:46) augment computes Eq.8: ˆ y =
g ◦ f
(ˆ x ) 9:
L train ( α, β ) =
CrossEntropyLoss (ˆ y, y ) 10: ( α, β ) ← arg min α,β L train ( α, β ) 11: if r is divisible by m then # Exploration 12:
Sample a mini-batch d valid ∈ D valid 13: for ( x, y ) ∈
d valid do 14: ( p, λ ) = π
[Equation] ( x ) 15: ˆ y = explore ( x ) (cid:46) explore computes Eq.16:
L valid ( α, β, γ ) =
CrossEntropyLoss (ˆ y, y ) 17: γ ← arg min γ L valid ( α, β, γ ) return α, γ
Relation to Neural Architecture Search (NAS).
Tuning of the augmentation policy parameters bears some similarity to the optimization of the network architecture weights in DARTS from the
NAS literature (Liu et al., 2019).
DARTS prescribes different computation paths to different operation cells and relaxes the computation to a mixture of the operations weighted by learnable weights for each path.
With w as the model parameters and α as the weights for the computation paths, the search algorithm solves a bi-level optimization problem:
[Equation]
While DARTS optimizes the architecture weights, AdaAug optimizes the projection parameter γ that decides the augmentation weights and magnitudes in the exploration.
DARTS solves the optimization by a ﬁrst-order and ﬁnite-difference approximation of the architecture gradient.
AdaAug samples the augmentation operations and treats the augmented data as missing training data ˆ X train .
By absorbing the augmentation parameter in the training dataset, we avoid the complex bi-level optimization and simplify the exploitation procedure into training a standard classiﬁer:
[Equation]
Relation to Density Matching.
Data augmentation can be regarded as a density matching problem between the training and validation data (Ratner et al., 2017; Tran et al., 2017; Hataya et al., 2020;
Lim et al., 2019).
From this perspective, AdaAug improves model generalization by matching the density of D train with the density of the augmented D valid .
In the outer optimization objective in
Equation (5), AdaAug minimizes the classiﬁcation loss with the augmentation parameters γ over the same optimal model parameters α ∗ , β ∗ learned from ˆ D train .
In so doing, it approximately reduces the distance between the densities of the augmented D valid and D train .
3. I NFERENCE
Like most automated augmentation pipelines (Cubuk et al., 2019; Ho et al., 2019; Lim et al., 2019;
Hataya et al., 2020; Li et al., 2020), AdaAug searches for the augmentation policy on a small dataset using a small model and then applies the learned policy network π θ to train with a larger dataset or model.
We call the process of applying a searched policy to augment a new dataset as inference time.
Regarding this workﬂow, RandAugment argues that the different search spaces in search time and inference time makes the augmentation policy unable to adjust the regularization strength to different target datasets and models (Cubuk et al., 2020).
Therefore, it proposes to search for the global magnitude and number of operators for each case.
To address this concern, AdaAug utilizes three diversity parameters to ﬁne-tune the regularization strength of the found policy for large datasets and models.
First, the number of operators k is set to 1 at search time, but a larger value of k can be used at inference time.
To control the selection of more diverse operations, a temperature parameter T is introduced in the softmax function: p i =
[Equation] [Equation] [Equation]
Setting a larger T value allows the operations with lower probability being sampled more often in the same order.
Last, AdaAug perturbs the magnitude value with the parameter δ so the perturbed magnitude ˆ λ is given as ˆ λ ∼ Uniform ( λ − δ, λ + δ ) .
This allows the augmented data to show slight variations even with the same operation on the same input data.
In practice, we can perform grid search on these three diversity parameters with the other hyperparameters using a holdout validation set.
E XPERIMENTS AND R ESULTS
In this section, we explain our experimental design followed by presenting the results.
We evaluate the empirical performance of AdaAug in two experiments: AdaAug-transfer and AdaAug-direct.
We select the comparison baselines that use the validation performance as the evaluation method to learn the augmentation policy.
For all the experiments, we report the average test-set error rate as the performance metric.
Each model is evaluated three times with different random initializations.
Augmentation operations.
We match the operations adopted by AutoAugment.
In addition to the 16 operations proposed previously (ShearX, ShearY, TranslateX, TranslateY, Rotate, AutoContrast,
Invert, Equalize, Solarize, Posterize, Contrast, Color, Brightness, Sharpness, Cutout, and Sample
Pairing), we add the Identity operation for not applying data augmentation.
For the simple baseline, we apply random horizontal ﬂip, color jittering, color normalization, and Cutout with a 16 × 16 patch size.
Our method and other baselines apply the found policy on top of these standard augmentations.
Policy search.
We follow the setup adopted by AutoAugment (Cubuk et al., 2019) to use 4, training images for CIFAR-10 and CIFAR-100, and 1,000 training images for SVHN.
The remaining images are used as the validation set.
We use Wide-ResNet-40-2 (Zagoruyko & Komodakis, 2016) as the feature extraction network for all searches.
We implement h as a linear layer and update the policy parameter γ after every 10 training steps using the Adam optimizer with a learning rate of 0.001 and a batch size of 128.
AdaAug-transfer.
In the ﬁrst experiment, we investigate how well the learned augmentation policy can transfer to unseen datasets.
We search for the optimal augmentation policy on the CIFARdataset and use the learned policy to train with four ﬁne-grained classiﬁcation datasets: Oxford
Flowers (Nilsback & Zisserman, 2008), Oxford-IIIT Pets (Em et al., 2017), FGVC Aircraft (Maji et al., 2013), and Stanford Cars (Krause et al., 2013).
We compare the test error rate with AutoAugment, Fast AutoAugment, DADA, and RandAugment using their published policies for CIFAR-100.
For all the tested datasets, we compare the transfer results when training the ResNet-50 model (He et al., 2016) for 180 epochs from scratch and ﬁne-tuning the ResNet-50 model pretrained on ImageNet for 100 epochs.
We use the cosine learning rate decay with one annealing cycle (Loshchilov & Hutter, 2017), initial learning rate of 0.1, weight decay 1e-4 and gradient clipping parameter 5.
AdaAug-direct .
In the second experiment, we search for the optimal augmentation policy on a small subset of the target dataset and use the learned policy to train the full dataset.
The purpose of the experiment is to demonstrate that while the AdaAug policy can adapt to other unseen datasets, it can also achieve competitive performances on the seen datasets with more training data.
We compare
AdaAug-direct with state-of-the-art AutoDA methods using the same evaluation datasets: CIFAR10, CIFAR-100 (Krizhevsky & Hinton, 2009), and SVHN (Netzer et al., 2011).
We test our method [Table] [Table] using Wide-ResNet-40-2 and Wide-ResNet-28-10.
At inference time [Table] we set the temperature T =and magnitude perturbation δ = 0 .
3 and search for the number of operators k ∈ { 1 , 2 , 3 , 4 } using a holdout validation set [Table] like RandAugment (Cubuk et al., 2020).
For the hyperparameters
[Table]
we follow AutoAugment, PBA
[Table]
and Fast AutoAugment if possible.
4. A DA A UG - TRANSFER
Table 1 and Table 2 show that the AdaAug policy outperforms the other baselines when training and ﬁne-tuning the ResNet-50 model on the Flowers, Pets, Aircraft, and Cars datasets, respectively.
These baselines apply the same augmentation policy to all datasets.
Such a policy may not be optimal to the target domain.
In contrast, AdaAug adapts the augmentation policy to individual image classes and instances automatically.
The AdaAug policy network applies different augmentation policies to unseen images according to their similarity to the classes that AdaAug has seen in the search.
To further justify our claim, we show the distribution of the augmentation parameters for different tasks in Appendix A.5.4.
The four datasets have many classes but few training examples per class.
The negative effect of using a non-adaptive data augmentation policy is likely to be more imminent than a situation where its dataset has fewer classes but more examples per class.
4. A DA A UG - DIRECT
CIFAR-10 and CIFAR-100 .
The policies learned by AdaAug mostly achieve either comparable or better performances than the baselines for both WRN-40-2 and WRN-28-10 models on the CIFAR10 and CIFAR-100 datasets (see Table 3).
We visualize the augmentation policy of CIFAR-10 by applying the policy to the validation data and averaging the predicted augmentation probability of each image for each class (see Figure 2).
The policy contains all types of augmentations at some degree.
Among the operations, Flip dominates the policy.
It is in line with the augmentation selected manually as people ﬁnd applying horizontal ﬂipping to CIFAR-10 can improve the prediction accuracy.
The policy learned by AdaAug puts higher emphasis on Invert and Equalize, which are also reported by PBA.
The focus on Brightness is also aligned with the policy in AutoAugment.
Although there are minor variations in the importance of some operations between the policies learned by AdaAug , AutoAugment, and PBA, their empirical performance is similar.
SVHN .
AdaAug performs comparably to the baselines on the core set of SVHN.
We visualize the augmentation policy for SVHN in Figure 2.
We found that AutoContrast, Invert, and Solarize receive much attention in SVHN.
This makes sense because the speciﬁc color of the number and background is irrelevant to the prediction.
This is consistent with the ﬁndings by AutoAugment and
PBA.
The Flip operation receives a signiﬁcantly higher probability in digits “0”, “1”, and “8”.
It is likely because these three digits appear similar after ﬂipping.
This shows that AdaAug captures not only a dataset-speciﬁc augmentation policy, but also class-dependent augmentation, which cannot be achieved by other baselines.
In addition to the augmentation probability, we visualize the learned augmentation magnitude in Appendix A.5.2. [Table]
ImageNet .
We also validate our method on large-scale ImageNet dataset.
AdaAug improves the top-1 accuracy 1% over the ResNet-50 baseline (see Appendix A.1.1).
Although some baselines like AutoAugment produce similar performance or slightly outperform our method in the AdaAugdirect experiment, AdaAug uses less computational effort in searching for the policy.
Speciﬁcally,
AutoAugment takes 5,000 GPU hours to search for the CIFAR-10 policy, while AdaAug takes only 3.3 GPU hours on an old GeForce GTX 1080 GPU card (see Appendix A.4).
D ISCUSSION
Instance-dependent augmentation.
The AdaAug-direct results show that the learned AdaAug policy can capture dataset- and class-dependent augmentations on CIFAR-10, CIFAR-100, and SVHN.
We further investigate whether AdaAug can learn instance-dependent information.
According to the architecture, AdaAug takes the output from the last layer of a CNN network as the image representation and uses it to predict the augmentation parameters.
The image representation contains the class information and potentially some image features.
Empirically, we ﬁrst examine if AdaAug generates different augmentation policies for different instances even within the same class.
In Appendix A.5.3, we plot the standard deviations of the predicted augmentation probabilities of the image instances for each class.
It is observed that even within the same class, the predicted augmentation policy is slightly different across instances.
It is a clue that the AdaAug policy captures some instance-level augmentation information.
Qualitatively, we show the augmented examples of a ﬂower image using the AdaAug policy in Figure 3.
The input image is relatively darker than the other images.
Apparently, AdaAug is aware of this property and applies more brightness-related augmentation to lighten up the image.
We observe similar behaviour of AdaAug in other classes, but there is inadequate empirical support to conclude a general rule on how the network decides the instance-aware augmentation as policies are learned in a data-driven way.
k = k =
k = k
=m =m = 5 m = 10 m = 15 m =
k
=k
=k =k =𝛿 = 0.1 𝛿 = 0.3 𝛿 = 0.5 𝛿 = 0.7 𝛿 = 0.
Quality of augmented data. We compare the augmented images from AdaAug with RandAugment under different augmentation strengths in Figure 3.
In terms of augmentation diversity, RandAugment produces more variations of the input image.
However, not all augmented images produced are plausible.
As RandAugment applies the augmentations uniformly, some augmented ﬂower images show a strange color difference to the original image.
With an increasing number of operators and magnitude, the ﬂower object is sometimes translated out of the frame and results in a black image.
This may affect the learning performance as color can be an important feature in classifying different species of ﬂowers.
For AdaAug , the augmented images are more visually appealing.
Ablation study.
In this section, we study the effects of AdaAug-transfer on Oxford 102 Flowers and AdaAug-direct on CIFAR-10 using three alternative search conﬁgurations.
First, does the use of a nonlinear projection deliver a better performance?
We replace the linear layer in h with a 2-layer MLP using a 128 hidden layer size with ReLU activation.
Second, we study whether the class-dependent and instance-dependent conﬁguration improves model accuracy.
We remove the class and instance information by replacing the policy network with a ﬁxed vector, which decides the augmentation probabilities and magnitudes for the entire dataset.
In the third setting, we mix the augmented images in the input space instead of the latent space and observe the effects.
Our results show that the use of class- and instance-adaptive augmentation policies contribute larger improvements in AdaAug-direct.
In AdaAug-transfer, using a nonlinear projection harms the prediction performance.
A possible reason is that the nonlinear projection is more likely to overﬁt the search dataset and fail to generalize to unseen datasets.
Moreover, combining the augmentation path in the latent space learns a better policy (see Table 4).
[Table]
C ONCLUSION
In this work, we propose a novel AutoDA approach, AdaAug , to learn class- and instance-adaptive augmentation policies efﬁciently.
We demonstrate that the found policy transfers well to unseen datasets while achieving state-of-the-art results on the seen datasets.
We provide evidence to show that the learned adaptive policy captures class- and instance-level information.
We think that
AdaAug can show further gains over existing baselines when applied to datasets with more dissimilar underlying augmentation rules among the data classes, and with fewer training examples for each class.
It is also promising to investigate in the future if the proposed adaptive augmentation method can improve the performance of other computer vision and representation learning tasks.
A CKNOWLEDGMENTS
This research has been made possible by the Hong Kong PhD Fellowship provided to the ﬁrst author and research grants (General Research Fund project 16204720 and Collaborative Research Fund project C6030-18GF from the Research Grants Council of Hong Kong; Amazon Web Services Machine Learning Research Award 2020) provided to the second author.
A A PPENDIX A. A DDITIONAL E XPERIMENTS A.1. L ARGE - SCALE D ATASET
For large-scale dataset AdaAug improves the top-1 accuracy 1% over the ImageNet ResNetbaseline (see Table 5).
The performance gain is similar to previous AutoDA methods and validates the positive effect of AdaAug on complex dataset
. A.1. A RCHITECTURE T RANSFER
We also provide the experimental results when using the learned Augmentation policy to train the
Shake-Shake (26 2x96d) model on Reduced CIFAR-10 and Reduced SVHN in Table [Table]
[Table]
A. A DDITIONAL A BLATION S TUDY
To clarify the improvements of AdaAug
[Table]
we compare the performance of AdaAug under different settings in Table 7: Simple : standard data augmentation is applied; Random :
AdaAug is applied with randomly initialized h γ while keeping the diversity parameters the same; AdaAug (w/o diversity) :
AdaAug is applied without the diversity parameters; AdaAug : AdaAug is applied with learned h γ and the diversity parameters.
[Table]
A. F INE - TUNING OF THE DIVERSITY PARAMETERS
The following experiments study the sensitivity of the temperature T and magnitude perturbation δ on the Flower dataset by changing the values of T and δ around the default values used in our original experiments.
The ﬁne-tuning of the diversity parameters reduces the test-set error rate of
AdaAug-transfer on the Flower dataset from 3.63 to 3.49.
A. E FFICIENCY OF P OLICY S EARCH
We compare the GPU hours needed to search the augmentation policy between different Automated
Data Augmentation methods in Table 9.
Among the baselines, AdaAug is more efﬁcient than
AutoAugment, PBA and Fast AutoAugment.
A. M ORE ANALYSIS ON A DA A UG POLICY LEARNING A.5. C ONVERGENCE OF POLICY LEARNING
Here, we provide some insights and empirical evidences for the convergence in policy training.
In particular, Figure 4 shows the training and validation losses when learning the CIFAR-10 augmentation policy.
The training and validation losses converge to some ﬁxed values towards the end of training.
In addition, we also visualize the change of the augmentation parameters ( p and µ ) in Figure 5.
The magnitude parameters start at a smaller value and converge towards the end of training.
For the augmentation probability, most of the candidates are stabilized after certain epochs while
[Table]
[Table] some others are updated more frequently.
These are the evidences that show the convergence of our proposed policy training method.
For a more thorough analysis for the policy convergence, we would like to leave it as the future work.
A.5. A NALYSIS OF LEARNED AUGMENTATION MAGNITUDE
Complement with the augmentation probability in Figure 2, Figure 6 shows the augmentation magnitude for CIFAR-10 and SVHN.
We observe that the policy magnitude λ also shows slight variations among different classes and instances.
Although the observation is less prominent and is harder to interpret when compared to the augmentation probability p , the learned augmentation magnitude adapts to different data samples
. A.5. V ARIANCE OF LEARNED AUGMENTATION POLICY GROUPED BY CLASS
In Figure [Table] we plot the standard deviations of the predicted augmentation probabilities of the image instances grouped by its class label.
It is observed that even within the same class, the predicted augmentation policy is different across instances.
It is a clue that the AdaAug policy captures some instance-level augmentation information.
A.5. D ISTRIBUTIONS OF THE AUGMENTATION PARAMETERS IN A DA A UG - TRANSFER
In Figure 8, we show the distributions of the augmentation parameters when transferring the learned policy to the Flower, Pet, Car and Aircraft datasets.
The distributions of the augmentation parameters show slight differences between different tasks.
Apparently, we observe that the colour transformations, for example Colour, Invert and Solarize are less preferred in the Flower and Pet datasets, while shearing operations are relatively more favourable in the Car and Aircraft dataset.
The observation of using less color transformation for the Flower dataset aligns with the ﬁndings from Lee et al.
Although some differences in the augmentation distributions may be less prominent and harder to interpret when compared to the AdaAug-direct cases, the policy shows its adaptations to different tasks.
A. L IMITATIONS
Although AdaAug is differentiable and efﬁcient in search, it requires forming different augmentation paths and passing the images through each path.
Compared to standard training, where we forward b images in one mini-batch, AdaAug processes b · | T | images in the exploration pass.
At inference time, AdaAug keeps a pre-trained policy network ( π θ ) to augment the new dataset.
This adds extra computational effort when compared to other AutoDA policies, which may be a concern if the image resolution and model size are large but the computational resources are limited.