A UTOREGRESSIVE G ENERATIVE A DVERSARIAL N ETWORKS / C ONFERENCE S UBMISSIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Generative Adversarial Networks (GANs) learn a generative model by playing an adversarial game between a generator and an auxiliary discriminator, which classiﬁes data samples vs. generated ones.
However, it does not explicitly model feature co-occurrences in samples.
In this paper, we propose a novel Autoregressive Generative Adversarial Network (ARGAN), that models the latent distribution of data using an autoregressive model, rather than relying on binary classiﬁcation of samples into data/generated categories.
In this way, feature co-occurrences in samples can be more efﬁciently captured.
Our model was evaluated on two widely used datasets: CIFAR-10 and STL-10.
Its performance is competitive with respect to other GAN models both quantitatively and qualitatively.
I NTRODUCTION
Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) are a new type of generative model with wide-spread success.
For example, they can be used to generate photo-realistic face and bedroom images (Radford et al., 2015).
Natural images contain a rich amount of features with speciﬁc co-occurrence conﬁgurations.
A generative model should be able to learn these conﬁgurations in order to perform well.
In a GAN setting, a discriminator is used to learn features in data and helps the generator to imitate them.
The generator should be able to produce samples with certain feature conﬁgurations – which are seen as likely by the discriminator – to satisfy the discriminator’s criteria.
However, GAN’s performance does not scale well to large datasets with high sample variation such as ImageNet (Russakovsky et al., 2015).
It can generate realistic textures, but the global consistency is still lacking (Salimans et al., 2016).
Recently several variations of GAN have been proposed (Arjovsky et al., 2017; Mao et al., 2016; Zhao et al., 2016) to improve image generation.
However the issue of global inconsistency still persists.
We postulate that the issue of global inconsistency has to do with the discriminator of a GAN and especially its classiﬁcation nature to model detailed global structure.
The discriminator architecture is similar to ones used in classiﬁcation or recognition tasks, whose aim is modeling p ( y | x ) but not p ( x ) , where y , x , p ( x ) represent class probabilities, data sample and data distribution, respectively.
However the overall aim of generative modeling is learning p ( x ) .
A good “teacher” a.k.a.
discriminator, should be able to learn p ( x ) in order to teach it to its “student” a.k.a.
generator, but the discriminator only learns p ( x ) implicitly by modeling p ( y | x ) in an adversarial way.
However, direct modeling of p ( x ) is time consuming and poses difﬁculties for capturing global structure in high resolution images (van den Oord et al., 2016).
In order to alleviate this problem, we propose a latent space modeling where an adversarial encoder learns the high dimensional feature vectors of real samples that are distinguishable from the fake ones.
The distinguishability criteria is the modeling error of an autoregressive model that tries to model latent space distribution of real images p ( f ) where f represents high-dimensional features.
A good adversarial encoder should come up with features that produce low error if they belong to real images and high error if they belong to fake ones.
Different from the basic GAN, p ( y | x ) , discriminator, is not modeled but rather p ( f | x ) , the encoder, and p ( f ) , the autoregressive model.
Our proposed Autoregressive GAN (ARGAN) learns features in the real data distribution without classiﬁcation but instead by modeling the latent space of the encoder with an autoregressive model.
Since an autoregressive model factorizes latent space and learns features in a conditional way, it can model feature co-occurrences more effectively than a classiﬁer as used in conventional GAN.
Furthermore, we propose a combination of ARGAN and Patch-GAN (Isola et al., 2016).
In this setting, Patch-GAN ensures realistic local features while ARGAN deploys its capacity at the global level.
We show that PARGAN improves both qualitative and quantitative results of the generated images.
We will release the source code and model data upon paper acceptance.
R ELATED W ORK
In various GAN models, the discriminator can be partitioned into two parts, a feature learner (encoder) and an assigner, which assigns values to distinguish two distributions either data or model.
A vanilla GAN learns features by using an adversarial encoder (E) and a linear classiﬁer (c) as assigner (Fig. 1).
The Wasserstein GAN (WGAN) (Arjovsky et al., 2017) follows a similar idea and uses an adversarial encoder with a linear separator (s) as assigner, which separates real features from generated ones on a single dimensional space.
Energy-based GAN (EBGAN) (Zhao et al., 2016)
slightly departs from this trend and uses an encoder to learn features and a decoder (D) as assigner, which reconstructs input of encoder.
In EBGAN, different from the previous methods, energy value is not the output of the assigner but the reconstruction.
In the proposed ARGAN model, features are learned with an encoder, and an autoregressive model (R) is used as assigner.
Similar to EBGAN, energy value is not the assigner’s output but rather its difference from the target, which corresponds to the modeling error of the assigner.
For the generator all models are the same, a noise sample (z) is fed into a generator (G) that maps it into the data space.
Another type of generative model is proposed by van den Oord et al.
(2016) and Theis & Bethge (2015), who model the joint distribution of pixels, p ( x ) , with an RNN in an autoregressive way.
During test time samples are generated by sampling one pixel at a time from the model.
In our work, autoregressive modeling is used in the latent space, and the features incoming to autoregressor are learned in an adversarial manner.
Hence instead of modeling data in the data space, they are modeled in the feature space, which is learned by an adversarial encoder concurrently.
Modeling distribution in the latent space can focus on semantics of data rather than low level features like correlation between adjacent pixels.
Furthermore, our model runs as fast as vanilla GAN during test time since recurrent modeling is only used during training as an auxiliary model.
The closest work to our method is Denoising Feature Matching Warde-Farley & Bengio (2017).
Even though Denoising Feature Matching uses density estimation in the latent space, there are major differences which makes learning dynamic of our model totally different then theirs. (i) Their method is complementary to GAN objective while our method can be learned standalone. (ii) More importantly their discriminator (encoder + classiﬁer) are trained as in original GAN objective which means that features learned from the data distribution are based on classiﬁer’s feedback not on density model’s.
This crucial difference make both works different than one another.
(iii)
In our model feature co-occurrences is modeled explicitly.
(iv) Motivation for both works are totally different.
M ETHOD 3. P RELIMINARY
GAN is a two player min-max game between a generator G and a discriminator D , as formulated by
Eq. 1.
G takes a noise sample z and maps it into data space
G
( z ; θ g ) , while D tries to discriminate between samples coming from G and real samples x .
Overal, D ’s aim is to learn features that separate samples coming from p g , generator’s distribution, and samples coming from p data , real data distribution.
G ’s aim is to produce samples that are indistinguishable from the real ones.
In practice, the objective for G is changed with Eq.
3 in order to improve gradient ﬂow, especially during the initial part of the training.
At each iteration, D maximizes Eq.
2 while G maximizes
Eq.
This procedure continues until it reaches the equilibrium p g ≈ p data , which is a saddle point of Eq.
[Equation]
3. A UTOREGRESSIVE GAN (ARGAN)
The discriminator of GAN can be separated into two parts: D =
d ◦ E where E is an adversarial encoder, and d is a linear binary classiﬁer.
The objective of E is to produce linearly separable distributions of features f real ∼ E ( x ) and f fake ∼ E ( G ( z )) .
Instead of using a classiﬁer d to separate distributions, we propose to use an autoregressive model, R , to model the latent distribution of real samples p ( f real ) .
This model should produce a small error for samples coming from p ( f real ) and a large error for samples coming from p ( f fake ) so that it can differentiate one distribution from the other.
An overview of the proposed ARGAN architecture is depicted in Fig.
When compared to vanilla
GAN (Fig. 1), this model replaces the classiﬁer layer with a factorization of features and autoregressive modeling of the factorized features.
The autoregressive model can be represented with a multi-layer LSTM or other types of RNN models.
Instead of using class discrimination as the learning signal, the autoregressive model’s prediction loss is used.
The intuition is that an autoregressive model would model the feature distribution better than fully connected layers by factorizing f and modeling conditional distributions:
[Equation]
The proposed Autoregressive GAN (ARGAN) is formulated as in Eq.
5, where G is the generator function
, E is the encoder, L is the loss function as deﬁned in Eq.
6, and θ G , θ E , θ R are the parameters for generator, encoder, and autoregressive model, respectively.
[Equation]
[Equation]
In practice, we found that Eq.
5 is unstable due to θ R ’s maximization of the second term being unbounded.
Instead θ R is optimized only over the ﬁrst term of Eq.
In this case, the objective for
G and E stays the same (Eq. 7), while θ R is optimized only on real sample features that are coming from E ( x ) but not on E ( G ( z ))
(Eq.
[Equation]
When θ E maximizes Eq.
7, it tries to learns features that yield low error for real samples and high error for generated ones.
In this way, E exploits structures that are common on data samples but not on generated ones.
When θ R maximizes Eq.
8, it ﬁts to features coming from real examples only.
We have not used any distribution function at the output layer of R since it is not meant to be sampled but rather used during training as an auxiliary model.
Using the model only during training removes test time sampling error (Husz ´ ar, 2015), which happens when conditional samples come from model distribution rather than data distribution.
We do make the independence assumption when factorizing features in order to reduce computation time during training.
Instead of conditioning single features on another, feature groups are conditioned on other feature groups by making the assumption that features in the same groups are independent from each other.
Fig.
3 shows two types of factorization we use to model Spatial-wise
ARGAN (S-ARGAN) and Channel-wise ARGAN (C-ARGAN).
Even though the independence assumption does not strictly hold, in practice both of them work equally well.
3. P ATCH -GAN + S-ARGAN (PARGAN)
Unlike vanilla GAN which produces single value for each sample, Patch-GAN (Isola et al., 2016), (Li & Wand, 2016) takes multiple patches from a sample and produces a value for each patch.
Hence for a single sample, multiple scores from different regions are evaluated.
Even though Patch-GAN can help the generator to produce locally realistic samples, it does not take the global arrangement into account.
Isola et al.
(2016) used Patch-GAN combined with L1 loss for image-to-image translation.
While Patch-GAN ensures realism of local content, a pixel-wise loss with ground truth is used to enforce global arrangement.
For image generation as in our work, there is no ground truth to relate with G ( z ) , hence it is not clear how to enforce global arrangement.
Therefore, in this paper, we propose S-ARGAN to ensure
global
1: θ G , θ E , θ R ← initialize network parameters 2: repeat 3: x (1) , ..., x ( M ) ∼ p data ( x ) (cid:46) Sample from data distribution 4: z (1) , ..., z ( M ) ∼ p noise ( z )
(cid:46) Sample from noise distribution 5: f ( j ) r ← E ( x ( j ) ) , for j = 1,
,M (cid:46) Compute features for training examples 6: f ( j )
f ← E ( G ( z ( j ) )) , for j
= 1,
,M (cid:46) Compute features for generated samples 7: p ( j ) r ← R ( f ( j ) r ) , for j = 1,
,M (cid:46) Compute predictions for training examples 8: p ( j ) f ← R ( f ( j ) f ) , for j
= 1,...
,M (cid:46) Compute predictions for training examples 9:
L θ G ←
[Table]
M (cid:80) Mj
=
[Table]
(cid:80) N f i =
[Table]
| p
[Table]
j ) f i − f
[Table]
j ) f
i +1 | (cid:46)
Compute loss for generator 10:
L θ E ←
[Table]
M (cid:80) Mj
=1 (cid:80) N f i =
| p ( j ) r i − f ( j ) r i +1 | −
[Table]
M (cid:80) Mj
=1 (cid:80) N f i =
| p ( j ) f i − f ( j ) f
i +1 | 11:
L θ R ← 1 M (cid:80) Mj
=1 (cid:80) N f i =
| p ( j ) r i − f ( j ) r i +1 | (cid:46)
Compute loss for autoregressive model 12: θ G ← θ G − ∇ θ G L θ G (cid:46) Update parameters of generator 13: θ E ← θ E − ∇ θ E L θ E (cid:46) Update parameters of encoder 14:
[Equation]
(cid:46) Update parameters of autoregressive model 15: until convergence consistency.
In this case, Patch-GAN decides whether individual patches are realistic or not, while
S-ARGAN examines whether their arrangement is spatially consistent.
Beware that S-ARGAN by itself can give feedback into individual channels, however using it with Patch-GAN makes training easier and produces better results.
Fig.
4 shows how the latent space of the encoder is factorized and feeds into S-ARGAN and Patch-GAN.
The objective of the Patch-GAN for discriminator and generator is given in Eq.
9 and Eq.
10 where
D =
d ⊕ E , ⊕ is convolution operation, E ( . )
∈
R h × w × c is the same encoder used in previous section and d ∈ R c × 1 × 1 × 1 is the convolutional kernel.
D ( . ) produces an activation map with size R h × w .
Each activation is matched to its corresponding target by using least square loss.
We use the least square version of GAN objective (Mao et al., 2016) rather than vanilla GAN objective (Eq. 2, Eq. 3) since vanilla GAN produces high error for generator later in the training.
PARGAN objective is a simple summation of ARGAN objective (Eq. 7), Eq.
8) and Patch-GAN objective (Eq. 9), Eq. 10) without any hyperparameter.
[Equation]
E XPERIMENTS
We have conducted experiments on CIFAR-10 (Krizhevsky et al.)
, STL-10 (Coates et al., 2011) and CelebA (Liu et al., 2015) databases.
CIFAR-10 has 60,000 images of 32 × 32 pixels withclasses: airplane, automobile, bird, cat, dear, dog, frog, horse, ship, and truck.
We have used 50, training set images to train our model.
STL-10 is a subset of ImageNet and more complex than
CIFAR-10.
There are 13,000 labeled and 100,000 unlabeled images with 96 × 96 pixels resolution.
CelebA contains 200K aligned face images.
The images are cropped to leave face are roughly.
All of our experiments have been conducted without any labels from datasets.
There is no agreed-upon evaluation metric for generative models (Theis et al., 2015).
One commonly used metric is the Inception score (Salimans et al., 2016) which is well correlated with human judgment.
Inception score is exp ( E x [ KL ( p ( y | x )
|| p ( y ))]) , where p ( y | x ) is the conditional class probability which should ideally have low entropy, and p ( y ) is the marginal class probability which should ideally have high entropy.
Higher values of inception score are better.
For evaluation, 50, images are generated and fed into the ofﬁcial Inception score code 1 with 10 splits.
The network architecture for each dataset is similar to Radford et al.
; details are provided in the Appendix.
We have used transposed convolution with strides 2 in generator until it matches the size of the data.
In the discriminator, convolutions with stride 2 are used until it reaches a pre-deﬁned feature map size.
BatchNorm (Ioffe & Szegedy, 2015) is used in both generator and discriminator, except the last layer of generator and ﬁrst layer of discriminator.
When C-ARGAN is used, the depth of the last layer in E restricted to 256 in order to reduce computation time during training.
In the case of S-ARGAN and PARGAN, the depth of the last layer is 512 .
We have used single layer
LSTM for S-ARGAN, and multi-layer LSTM for C-ARGAN since single level produces blurry results.
ADAM is used as optimizer with learning rate 0 . 0001 and β 1 = 0 .
Interestingly, the L2 loss could not produce results as good as L1 loss in R ’s objective (Eq. 6).
Hence, we have conducted all our experiments using L1 loss in the autoregressive model.
In the case of C-ARGAN and S-ARGAN, the generator is updated t times to catch encoder.
We found t to be an important hyper-parameter to tune.
By using grid search, we choose t = 3 in all our experiments.
With this setting, the error of the generator closely follows the error of the encoder.
4. Q UALITATIVE R ESULTS
Generated images for the proposed S-ARGAN, C-ARGAN and PARGAN for CIFAR-10 are shown in Fig.
Note that none of the images in this paper are cherry picked.
Overall images look sharp even though not all of them feature recognizable objects.
For CIFAR-10, only 32x32 resolution is explored.
Fig.
6 shows 48x48 pixel image generation from STL-10 dataset for S-ARGAN, CARGAN and PARGAN.
We have also used PARGAN on the 96x96 STL-10 to show it can scale to higher resolutions (Fig. 7).
Certain objects like car, ship, horse, and some other animals are recognizable.
For CelebA generation we have used SW-ARGAN with 64x64 resolution (Fig. 8).
The face images looks realistic in general with small artifacts in certain samples.
4. Q UANTITATIVE R ESULTS
Inception scores for the proposed S-ARGAN, C-ARGAN and PARGAN on CIFAR-10 and STL10 are given in Table 1.
We compare our results with other baseline GAN models 3 like DCGAN (Radford et al., 2015), EBGAN (Dai et al., 2017), WGAN (Gulrajani et al., 2017) rather than extensions such as D2GAN (Dinh Nguyen et al., 2017), MGGAN (Hoang et al., 2017), DFM (Warde-Farley & Bengio, 2017).
These extensions can be combined with our model in the same way as they are combined with vanilla GAN.
C-ARGAN and S-ARGAN show competitive results with vanilla GAN, while PARGAN outperforms them.
PARGAN is competitive with WGAN-GP on the same architecture (DCGAN) even though WGAN-GP performs better with ResNet (Gulrajani et al., 2017).
[Table]
C ONCLUSIONS AND F UTURE W ORK
We have proposed a novel GAN model and successfully trained it on CIFAR-10 and STLdatasets.
It models the latent distribution of real samples, which is learned via adversarial training.
Besides, we have combined the proposed model with Patch-GAN to further improve the stability of training and the quality of generated images.
In this work we have used a simple autoregressive model.
Our model can be further enhanced by more advanced latent space modeling, such as bidirectional modules (Schuster & Paliwal, 1997), 2D LSTM (Theis & Bethge, 2015), hierarchical modules (Mehri et al., 2017) etc.
As future work we would like to extend this method to ImageNet which has even larger sample variation.
Combining our models with MGGAN (Hoang et al., 2017) is another potential research direction.
A N ETWORK A RCHITECTURES
Network architecture for CIFAR-10:
• Generator: FC(100,512*4*4)-BN-ReLU-DC(512,256;4c2s)-BN-ReLUDC(256,128;4c2s)-BN-ReLU-DC(128,3;4c2s)-Tanh • Feature Learner: CV(3,64;4c2s)-BN-LRec-CV(64,128;4c2s)-BN-LRecCV(128,256;4c2s) -BN-LRec-CV(256,512;4c2s)-BN-LRec • Autoregressive model for C-ARGAN: 2 layer LSTM, hidden size: • Autoregressive model for S-ARGAN: 1 layer LSTM, hidden size: • Autoregressive model for PARGAN: 1 layer LSTM, hidden size: • Patch-wise discriminator for PARGAN: CV(512,1;1c1s)
Network architecture for STL-10:
• Generator: FC(100,512*3*3)-BN-ReLU-DC(512,256;4c2s)-BN-ReLUDC(256,256;4c2s)-BN-ReLU-DC(256,128;4c2s)-BN-ReLU-DC(128,3;4c2s)-Tanh • Feature Learner: CV(3,64;4c2s)-BN-LRec-CV(64,128;4c2s)-BN-LRecCV(128,256;4c2s) -BN-LRec-CV(256,512;4c2s)-BN-LRec • Autoregressive model for C-ARGAN: 2 layer LSTM, hidden size: • Autoregressive model for S-ARGAN: 1 layer LSTM, hidden size: • Autoregressive model for PARGAN:
• Patch-wise discriminator for PARGAN: CV(512,1;1c1s) FC(100,512) denotes fully connected layer from 100 to 512 dimensions, BN is batch-normalization, DC(512,256;4c2s) is transposed convolution with input channels 512 and output channels 256, while 4c2s denotes ﬁlter size 4 and stride 2 (fractional stride in this case).
CV is convolution layer and LRec is leaky ReLU.