PAGANDA: An Adaptive Task-Independent Automatic Data Augmentation
Abstract
Data augmentation is a useful technique to enlarge the size of the training set and prevent overﬁtting for different machine learning tasks when training data is scarce.
However, current data augmentation techniques rely heavily on human design and domain knowledge, and existing automated approaches are yet to fully exploit the latent features in the training dataset.
In this paper we propose Parallel Adaptive GAN Data Augmentation (PAGANDA), where the training set adaptively enriches itself with sample images automatically constructed from
Generative Adversarial Networks (GANs) trained in parallel.
We demonstrate by experiments that our data augmentation strategy, with little modelspeciﬁc considerations, can be easily adapted to cross-domain deep learning/machine learning tasks such as image classiﬁcation and image inpainting, while signiﬁcantly improving model performance in both tasks.
Our source code and experimental details are available at https: //github.com/
miaojiang1987/ k-folder-data-augmentation-gan/ .
1. Introduction
Deep learning and machine learning models produce highly successful results when given sufﬁcient training data.
However, when training data is scarce, overﬁtting will occur and the resulting model will generalize poorly.
Data augmentation(DA) ameliorates such issues by enlarging the original data set and making more effective use of the information in existing data.
Much prior work has centered on data augmentation strategies based on human design, including heuristic data augmentation strategies such as crop, mirror, rotation and distortion (Krizhevsky et al., 2012; Simard et al., 2003), interpolating through labeled data points in feature spaces (DeVries & Taylor, 2017), and adversarial data augmentation strategies based on (Teo et al., 2008; Fawzi et al., 2016).
These methods have greatly aided many deep learning tasks across several domains such as classiﬁcation (Krizhevsky et al., 2012),
image segmentation (Yang et al., 2017) and image reconstruction/inpainting (Alvarez-Gila et al., 2017).
Despite their success, these DA methods generally require domain-speciﬁc expert knowledge, manual operations and extensive amount of tuning depending on actual contexts (Ciresan et al.
; Dosovitskiy et al., 2016).
In particular, the need to directly operate on existing data with domain knowledge prevents many previous data augmentation strategies from being applicable to more general settings.
To circumvent the need for speciﬁc domain knowledge in data augmentation, more recent work (Antoniou et al., 2017) utilizes generative adversarial networks(GANs)
(Goodfellow et al., 2014) to produce images that better encode features in the latent space of training data.
By alternatively optimizing the generator G and the discriminator D in the GAN, the GAN is able to produce images similar to the original data and effectively complement the training set.
It has been shown in experiments (Antoniou et al., 2017) that GAN-based methods have indeed signiﬁcantly boosted the performance of classiﬁers under limited data through automatic augmentation, but applications into other tasks are yet to be explored.
Furthermore, given the computational complexity of GANs, a natural way to reduce runtime is to consider parallelism (Intrator et al., 2018; Durugkar et al., 2016).
In view of these considerations, we propose in this paper
Parallel Adaptive Generative Adversarial Network Data
Augmentation (PAGANDA), where the training set adaptively enriches itself with sample images automatically constructed from Generative Adversarial Networks (GANs) trained in parallel.
Our contributions can be summarized as follows
: • We propose a general adaptive black-box data augmentation strategy to diversify enhance training data, with no task-speciﬁc requirements.
• We also include in our model a novel K -fold parallel framework, which helps make the most use of the existing data.
• Experiments over various datasets and tasks demonstrate the effectiveness of our method in different context.
2. Related Work
Data Augmentation(DA)
Previous work on data augmentation can be classiﬁed into several groups.
Traditional
Heuristic DA strategies such as crop, mirror, rotation and distortion (Krizhevsky et al., 2012;
Simard et al., 2003) have found their way in many deep classiﬁcation tasks, but these method generally require domain-speciﬁc expert knowledge, manual operations and extensive amount of tuning depending on actual contexts (Ciresan et al.
; Dosovitskiy et al., 2016).
Other DA methods used interpolation through labeled data points in feature spaces (DeVries & Taylor, 2017), but their dependence on class labels makes them inapplicable for tasks with weak or no supervision.
Adversarial Data Augmentation strategies (Teo et al., 2008;
Fawzi et al., 2016) choose from a select number of transformation operations to maximize the loss function of the end classiﬁcation model involved in the task.
While good motivations for our methods, these methods make strong assumptions over the types of augmentation and are difﬁcult to generalize.
(Ratner et al., 2017; Cubuk et al., 2018) transform the problem of choosing data augmentation strategies into a reinforcement learning policy search problems, but the choice of augmentation methods are still limited and the reinforcement learning algorithms have non-trivial computation overhead in addition to the main task.
ML problems with limited data For classﬁcation with limited samples, (Salamon & Bello, 2017) proposed a convolutional neural network(CNN) to classify environmental sounds with limited samples.
Other algorithms have been proposed in (Frid-Adar et al., 2018; Zhu et al., 2017), yet many of them have assumptions/constraints that hurts their capacity for generalization.
For unsupervised learning models, recent research on sample complexity reduction in GAN training seeks to reparametrize the input noise using variational inference (Gurumurthy et al., 2017; Nowozin et al., 2016), but this method has severe mathematical limitation that prevents further generalization.
(Wang et al., 2018) adopts transfer learning techniques to train a new GAN for limited data from a pre-trained GAN network.
While effective, this approach requires a pre-trained network in the ﬁrst place and doesn’t apply to the cases when data is scarce.
Parallel/Distributed GANs (Intrator et al., 2018; Durugkar et al., 2016) proposed the ﬁrst distributed multidiscriminator generative adversarial models, yet these models require large datasets to train and have great computational complexity.
Moreover, these models are trained on ﬁxed given datasets, meaning they are still susceptible to the inherent biases of the training data.
3. Proposed method
In this section we describe the details of Parallel Adaptive Generative Adversarial Network Data Augmentation (PAGANDA).
Our method consists of three interrelated components: generative data augmentation, parallel image generation with fold division, and adaptive weight adjustment.
3.1. Generative Data Augmentation
To ensure that make full use of the information contained in the existing images, the ﬁrst part of our method involves generative data augmentation , which constructs varied images given the training set by repeatedly generating samples from and adding samples to the training set using a generative adversarial net.
We start off with a limited training set, and consecutively run the generative adversarial net using the set.
After running a ﬁxed number t of regular training epochs, we proceed to the augmentation epoch where the augmentation is conducted.
During the augmentation epoch, we extract a number of sample images from the generator G using standard procedures of sample image generation as described in (Radford et al., 2015; Gulrajani et al., 2017).
For this batch of samples, we calculate the Inception Score(IS) as deﬁned by (Salimans et al., 2016) to measure the authenticity of the images generated, which we denote as w .
Here the Inception Score provides a metric of the power of generator to produce realistic images: the higher the value of w , the more power the corresponding generator G .
This batch of images are then added back into the original training set for subsequent augmentation epochs.
We alternate running t regular training epochs and the augmentation epoch for a ﬁxed number of times or until convergence.
Figure 1 is a ﬂow-chart of our procedure.
Notice that our procedure is agnostic to the speciﬁc architecture of generative adversarial net used to augment the training data.
Since GANs capture the information in the latent feature space of the images and translate such information into generated images, our method has the capacity to reveal the potential features that are possibly not visually evident in the original training images.
Moreover, compared with many other data augmentation strategies which require one to pre-deﬁne the operations to be carried on the images, our method automatically enriches the training set and does not require human intervention.
3.2. Parallel Image Generation with Fold Division
The second part of our method consists of a parallel data generation strategy, inspired by K -fold cross validation in machine learning (Bishop, 2006).
Dividing the training data into K folds at the beginning, we run in parallel K independent generators { G i } Ki =1 .
Each generator G i is trained on one of data groups, and each data group i consists of
K − 1 folds of the training set, except for the i -th fold.
After images are generated in each generator G i in the training epochs, the sample images produced by each generator during the augmentation epoch are fed back into the respective training data groups.
To allow for maximal usage of each generated image, we insert the images in a way such that the images generated by one generator G i are sent to the training data groups corresponding to all other K − 1 generators except for that corresponding to G i .
This is to insure that the different generators in parallel have access to as many varied data pieces as possible in subsequent steps of training, so as to prevent overﬁtting and bolster the robustness of our strategy.
Figure 2 demonstrates our algorithm.
3.3. Adaptive Generator Weighting
Furthermore, to determine which generators are the most effective in generating authentic images, we introduce adaptive generator weighting at each augmentation epoch.
At the initial stage, all the generators are treated equally.
Before the batch of sample images generated by one generator G i are sent to the data group corresponding to other K −generators, we collect the inception scores { w i } Ki =1 computed in section 3.1.
Since higher inception scores imply better performance of the generator, we deﬁne the generator weight p
i of a generator G i as p
[Equation]
and use this weight to determine how many images should be sampled from generator G i to be sent to other data groups for subsequent training in the very next augmentation epoch.
When the total number of samples to be collected from generators are ﬁxed, this method enables generators with better realistic image generation power to contribute more to the future training data groups.
More realistic training sets thus augmented, in turn, exert more positive inﬂuence on the images to be generated.
Note that all three strategies introduced go hand in hand, with no need for model speciﬁc considerations.
As demonstrated by our experiments Section 4, training different
GANs in parallel from different folds of data substantially boosts the quality of the training set and that of the generated images.
4. Experiments 4.1. Multitask Experimental Settings
To illustrate the effectiveness of PAGANDA for multiple machine learning tasks, we have applied our data augmentation method to two tasks: image classiﬁcation and image inpainting.
For image classiﬁcation we constructed our dataset from Imagenet and Cifar-10 by randomly drawing 5000 images from each dataset respectively and applied PAGANDA on these reduced datasets.
The augmented datasets are then used to train an AlexNet CNN classiﬁer, and the classiﬁcation results are compared with the results obtained from an
AlexNet trained on the corresponding original unaugmented datasets.
For image inpainting, we constructed our datasets from
Places dataset.
We chose images from the Ocean subset from Places to obtain the reduced Places Dataset. [Figure]
To ensure the parallelism of the experiments, we trained our model in a multi-threaded environment to make simultaneously training.
Under such a setting, all the data groups are trained at the same time, and each GAN model corresponding to each data group is trained in a separate thread.
All of our experiments are conducted on a server with Tesla-V
GPU (32GB RAM, 7.8 TeraFLOPS) and Intel Xeon Processor E5 (2.00 GHz).
4.2. Evaluation on Classiﬁcation
For our experiments on classiﬁcation, we ﬁrst augment the reduced Cifar-10 and reduced Imagenet datasets, and then train the CNN classiﬁer with the augmented dataset.
The classiﬁer accuracies with and without augmentation are listed in Table 1 below.
[Table]
4.3. Evaluation on Inpainting
For the task of inpainting, we augment the reduced dataset constructed in the experiment.
Without loss of generality, we train a WGAN-GP model for inpainting from the augmented dataset.
We then select testing images that are not selected in the training set, and add to them gray masks covering the center part of these images.
We then applied our trained WGAN-GP to generate patches that cover the masked portion of the inpainting image.
Figure 4 lists a couple of generated images with and without augmentation.
Visual comparisons demonstrate the effectiveness of our method.
5. Conclusion and Future Work
In sum, our paper shows that PAGANDA effectively improves the performances for different machine learning tasks with little task-speciﬁc considerations.
Our strategy is not only simple to implement, but also demonstrates capability to generate onto different settings since it does not require speciﬁc information about the task being analyzed.
As a further step, we are investigating the relationship between our proposed approach and other established methods.
We hope to apply our idea to other generative models such as VAE (Kingma & Welling, 2014) and further optimize our strategy using recent theoretical advances, and wish to investigate the scenarios where the tasks involved are interrelated.
Application wise, we are aiming to apply our parallel GAN model to multi-modal image synthesis/generation where training data is limited. Acknowledgements We thank the Digital Science Center of Indiana University for supporting our research with GPUs.