S EMANTICALLY D ECOMPOSING THE L ATENT S PACES OF G ENERATIVE A DVERSARIAL N ETWORKS
zlipton
A BSTRACT
We propose a new algorithm for training generative adversarial networks that jointly learns latent codes for both identities (e g. individual humans) and observations (e.g. speciﬁc photographs).
By ﬁxing the identity portion of the latent codes, we can generate diverse images of the same subject and by ﬁxing the observation portion, we can traverse the manifold of subjects while maintaining contingent aspects such as lighting and pose.
Our algorithm features a pairwise training scheme in which each sample from the generator consists of two images with a common identity code.
Corresponding samples from the real dataset consist of two distinct photographs of the same subject.
In order to fool the discriminator, the generator must produce pairs that are photorealistic, distinct and appear to depict the same individual.
We augment both the DCGAN and BEGAN approaches with
Siamese discriminators to facilitate pairwise training.
Experiments with human judges and an off-the-shelf face veriﬁcation system demonstrate our algorithm’s ability to generate convincing, identity-matched photographs
I NTRODUCTION
In many domains, a suitable generative process might consist of several stages.
To generate a photograph of a product, we might wish to ﬁrst sample from the space of products, and then from the space of photographs of that product .
Given such disentangled representations in a multistage generative process, an online retailer might diversify its catalog, depicting products in a wider variety of settings.
A retailer could also ﬂip the process, imagining new products in a ﬁxed setting.
Datasets for such domains often contain many labeled identities with fewer observations of each (e.g. a collection of face portraits with thousands of people and ten photos of each).
While we may know the identity of the subject in each photograph, we may not know the contingent aspects of the observation (such as lighting, pose and background).
This kind of data is ubiquitous; given a set of commonalities, we might want to incorporate this structure into our latent representations.
Generative adversarial networks (GANs) learn mappings from latent codes z in some low-dimensional space Z to points in the space of natural data X (Goodfellow et al., 2014).
They achieve this power through an adversarial training scheme pitting a generative model G : Z (cid:55)→ X against a discriminative model D : X (cid:55)→ [0 , 1] in a minimax game.
While GANs are popular, owing to their ability to generate high-ﬁdelity images, they do not, in their original form, explicitly disentangle the latent factors according to known commonalities.
In this paper , we propose Semantically Decomposed GANs (SD-GANs), which encourage a speciﬁed portion of the latent space to correspond to a known source of variation.
The technique decomposes the latent code Z into one portion Z I corresponding to identity, and the remaining portion Z O corresponding to the other contingent aspects of observations.
SD-GANs learn through a pairwise training scheme in which each sample from the real dataset consists of two distinct images with a common identity.
Each sample from the generator consists of a pair of images with common z I ∈
Z I but differing z O ∈ Z O .
In order to fool the discriminator, the generator must not only produce diverse and photorealistic images, but also images that depict the same identity when z I is ﬁxed.
For SD-GANs, we modify the discriminator so that it can determine whether a pair of samples constitutes a match.
As a case study, we experiment with a dataset of face photographs, demonstrating that SD-GANs can generate contrasting images of the same subject (Figure 1; interactive web demo in footnote on previous page).
The generator learns that certain properties are free to vary across observations but not identity.
For example, SD-GANs learn that pose, facial expression, hirsuteness, grayscale vs. color, and lighting can all vary across different photographs of the same individual.
On the other hand, the aspects that are more salient for facial veriﬁcation remain consistent as we vary the observation code
z O .
We also train SD-GANs on a dataset of product images, containing multiple photographs of each product from various perspectives (Figure 4).
We demonstrate that SD-GANs trained on faces generate stylistically-contrasting, identity-matched image pairs that human annotators and a state-of-the-art face veriﬁcation algorithm recognize as depicting the same subject.
On measures of identity coherence and image diversity, SD-GANs perform comparably to a recent conditional GAN method (Odena et al., 2017); SD-GANs can also imagine new identities, while conditional GANs are limited to generating existing identities from the training data.
S EMANTICALLY D ECOMPOSED G ENERATIVE A DVERSARIAL N ETWORKS
Before introducing our algorithm, we brieﬂy review the prerequisite concepts.
2. GAN PRELIMINARIES
GANs leverage the discriminative power of neural networks to learn generative models.
The generative model G ingests latent codes z , sampled from some known prior P Z , and produces G ( z ) , a sample of an implicit distribution P G .
The learning process consists of a minimax game between G , parameterized by θ G , and a discriminative model D , parameterized by θ D .
In the original formulation, the discriminative model tries to maximize log likelihood, yielding
[Equation]
Training proceeds as follows:
For k iterations, sample one minibatch from the real distribution P R and one from the distribution of generated images P G , updating discriminator weights θ D to increase
V ( G, D ) by stochastic gradient ascent.
Then sample a minibatch from P Z , updating θ G to decrease
V ( G, D ) by stochastic gradient descent.
Zhao et al.
(2017b)
propose energy-based GANs (EBGANs), in which the discriminator can be viewed as an energy function.
Speciﬁcally, they devise a discriminator consisting of an autoencoder:
D ( x ) =
D d ( D e ( x )) .
In the minimax game, the discriminator’s weights are updated to minimize the reconstruction error L ( x ) =
|| x − D ( x ) || for real data, while maximizing the error L ( G ( z )) for the generator.
More recently, Berthelot et al.
(2017) extend this work, introducing Boundary
Equilibrium GANs (BEGANs), which optimize the Wasserstein distance (reminiscent of Wasserstein
GANs (Arjovsky et al., 2017)) between autoencoder loss distributions, yielding the formulation:
[Equation]
Additionally, they introduce a method for stabilizing training.
Positing that training becomes unstable when the discriminator cannot distinguish between real and generated images, they introduce a new hyperparameter γ , updating the value function on each iteration to maintain a desired ratio between the two reconstruction errors: E [ L ( G ( z ))]
= γ E [ L ( x )] .
The BEGAN model produces what appear to us, subjectively, to be the sharpest images of faces yet generated by a GAN.
In this work, we adapt both the DCGAN (Radford et al., 2016) and BEGAN algorithms to the SD-GAN training scheme.
2. SD-GAN FORMULATION
Consider the data’s identity as a random variable I in a discrete index set I .
We seek to learn a latent representation that conveniently decomposes the variation in the real data into two parts: 1) due to I , and 2) due to the other factors of variation in the data, packaged as a random variable O .
Ideally, the decomposition of the variation in the data into I and O should correspond exactly to a decomposition of the latent space Z
=
Z I × Z O .
This would permit convenient interpolation and other operations on the inferred subspaces Z I and Z O .
A conventional GAN samples I, O from their joint distribution.
Such a GAN’s generative model samples directly from an unstructured prior over the latent space.
It does not disentangle the variation in O and I , for instance by modeling conditional distributions P G ( O | I = i ) , but only models their average with respect to the prior on I .
Our SD-GAN method learns such a latent space decomposition, partitioning the coordinates of Z into two parts representing the subspaces, so that any z ∈ Z can be written as the concatenation [ z I ;
z O ] of its identity representation z I ∈ R
d I =
Z I and its contingent aspect representation z O ∈ R d O =
Z O .
SD-GANs achieve this through a pairwise training scheme in which each sample from the real data consists of x 1 , x 2 ∼ P R ( x | I = i ) , a pair of images with a common identity i ∈ I .
Each sample from the generator consists of G ( z 1 ) , G ( z 2 ) ∼ P G ( z | Z I =
z I ) , a pair of images generated from a common identity vector z
I ∈
Z I but i.i.d.
observation vectors z 1 O , z
2 O ∈ Z O .
We assign identity-matched pairs from P R the label 1 and z I -matched pairs from P G the label 0 .
The discriminator can thus learn to reject pairs for either of two primary reasons: 1) not photorealistic or 2) not plausibly depicting the same subject.
See Algorithm 1 for SD-GAN training pseudocode.
[Figure]
2. SD-GAN DISCRIMINATOR ARCHITECTURE
With SD-GANs, there is no need to alter the architecture of the generator.
However, the discriminator must now act upon two images, producing a single output.
Moreover, the effects of the two input images x 1 , x 2 on the output score are not independent.
Two images might be otherwise photorealistic but deserve rejection because they clearly depict different identities.
To this end, we devise two novel discriminator architectures to adapt DCGAN and BEGAN respectively.
In both cases, we ﬁrst separately encode each image using the same convolutional neural network D e (Figure 2).
We choose this Siamese setup (Bromley, 1994; Chopra et al., 2005) as our problem is symmetrical in the images, and thus it’s sensible to share weights between the encoders.
To adapt DCGAN, we stack the feature maps D e ( x 1 ) and D e ( x 2 ) along the channel axis, applying one additional strided convolution.
This allows the network to further aggregate information from the two images before ﬂattening and fully connecting to a sigmoid output.
For BEGAN, because the discriminator is an autoencoder, our architecture is more complicated.
After encoding each image, we concatenate the representations [ D e ( x 1 ); D e ( x 2 )]
∈ R 2( d I + d O ) and apply one fully connected bottleneck layer R 2( d I + d O )
⇒ R d I +
d O with linear activation.
In alignment with BEGAN, the
SD-BEGAN bottleneck has the same dimensionality as the tuple of latent codes ( z I , z 1 O , z 2 O ) that generated the pair of images.
Following the bottleneck, we apply a second FC layer R d I +
d O ⇒
R 2( d I + d O ) , taking the ﬁrst d I + d O components of its output to be the input to the ﬁrst decoder and the second d I + d O components to be the input to the second decoder.
The shared intermediate layer gives SD-BEGAN a mechanism to push apart matched and unmatched pairs.
We specify our exact architectures in full detail in Appendix E.
E XPERIMENTS
We experimentally validate SD-GANs using two datasets: 1) the MS-Celeb-1M dataset of celebrity face images (Guo et al., 2016) and 2) a dataset of shoe images collected from Amazon (McAuley et al., 2015).
Both datasets contain a large number of identities (people and shoes, respectively) with multiple observations of each.
The “in-the-wild” nature of the celebrity face images offers a richer test bed for our method as both identities and contingent factors are signiﬁcant sources of variation.
In contrast, Amazon’s shoe images tend to vary only with camera perspective for a given product, making this data useful for sanity-checking our approach.
Faces From the aligned face images in the MS-Celeb-1M dataset, we select 12,500 celebrities at random and 8 associated images of each, resizing them to 64 x 64 pixels.
We split the celebrities into subsets of 10,000 (training), 1,250 (validation) and 1,250 (test).
The dataset has a small number of duplicate images and some label noise (images matched to the wrong celebrity).
We detect and remove duplicates by hashing the images, but we do not rid the data of label noise.
We scale the pixel values to [ − 1 , 1] , performing no additional preprocessing or data augmentation.
Shoes Synthesizing novel product images is another promising domain for our method.
In our shoes dataset, product photographs are captured against white backgrounds and primarily differ in orientation and distance.
Accordingly, we expect that SD-GAN training will allocate the observation latent space to capture these aspects.
We choose to study shoes as a prototypical example of a category of product images.
The Amazon dataset contains around 3,000 unique products with the category “Shoe” and multiple product images.
We use the same 80% , 10% , 10% split and again hash the images to ensure that the splits are disjoint.
There are 6 . 2 photos of each product on average.
3. T RAINING DETAILS
We train SD-DCGANs on both of our datasets for 500,000 iterations using batches of 16 identitymatched pairs.
To optimize SD-DCGAN, we use the Adam optimizer (Kingma & Ba, 2015) with hyperparameters α = 2e − 4 , β 1 = 0 .
5 , β 2 = 0 .
999 as recommended by Radford et al.
We also consider a non-Siamese discriminator that simply stacks the channels of the pair of real or fake images before encoding (SD-DCGAN-SC).
As in (Radford et al., 2016), we sample latent vectors z ∼ Uniform ([ − 1 , 1] 100 ) .
For SD-GANs, we partition the latent codes according to z
I ∈ R d I , z O ∈ R 100 − d
I using values of d I =
Our algorithm can be trivially applied with k -wise training (vs. pairwise).
To explore the effects of using k > 2 , we also experiment with an SD-DCGAN where we sample k = 4 instances each from
P G ( z | Z I =
z I ) for some z
I ∈
Z I and from P R ( x | I = i ) for some i ∈
I .
For all experiments, unless otherwise stated, we use d I = 50 and k = 2 .
We also train an SD-BEGAN on both of our datasets.
The increased complexity of the SD-BEGAN model signiﬁcantly increases training time, limiting our ability to perform more-exhaustive hyperparameter validation (as we do for SD-DCGAN).
We use the Adam optimizer with the default hyperparameters from (Kingma & Ba, 2015) for our SD-BEGAN experiments.
While results from our SD-DCGAN k = 4 model are compelling, an experiment with a k = 4 variant of SD-BEGAN resulted in early mode collapse (Appendix F); hence, we excluded SD-BEGAN k = 4 from our evaluation.
We also compare to a DCGAN architecture trained using the auxiliary classiﬁer GAN (AC-GAN) method (Odena et al., 2017).
AC-GAN differs from SD-GAN in two key ways: 1) random identity codes
z I are replaced by a one-hot embedding over all the identities in the training set (matrix of size 10000 x 50 ); 2) the AC-GAN method encourages that generated photos depict the proper identity by tasking its discriminator with predicting the identity of the generated or real image.
Unlike SD-GANs, the AC-DCGAN model cannot imagine new identities; when generating from AC-DCGAN (for our quantitative comparisons to SD-GANs), we must sample a random identity from those existing in the training data. [Table]
3. E VALUATION
The evaluation of generative models is a fraught topic.
Quantitative measures of sample quality can be poorly correlated with each other (Theis et al., 2016).
Accordingly, we design an evaluation to match conceivable uses of our algorithm.
Because we hope to produce diverse samples that humans deem to depict the same person, we evaluate the identity coherence of SD-GANs and baselines using both a pretrained face veriﬁcation model and crowd-sourced human judgments obtained through
Amazon’s Mechanical Turk platform.
3.2. Q UANTITATIVE
Recent advancements in face veriﬁcation using deep convolutional neural networks (Schroff et al., 2015; Parkhi et al., 2015; Wen et al., 2016) have yielded accuracy rivaling humans.
For our evaluation, we procure FaceNet , a publicly-available face veriﬁer based on the Inception-ResNet architecture (Szegedy et al., 2017).
The FaceNet model was pretrained on the CASIA-WebFace dataset (Yi et al., 2014) and achieves 98 .
6% accuracy on the LFW benchmark (Huang et al., 2012).
FaceNet ingests normalized, 160 x 160 color images and produces an embedding f ( x ) ∈ R 128 .
The training objective for FaceNet is to learn embeddings that minimize the L 2 distance between matched pairs of faces and maximize the distance for mismatched pairs.
Accordingly, the embedding space yields a function for measuring the similarity between two faces x 1 and x 2 : D ( x 1 , x 2 ) =
|| f
( x 1 ) − f ( x 2 ) || 22 .
Given two images, x 1 and x 2 , we label them as a match if D ( x 1 , x 2 ) ≤ τ v where τ v is the accuracy-maximizing threshold on a class-balanced set of pairs from MS-Celeb-1M validation data.
We use the same threshold for evaluating both real and synthetic data with FaceNet.
We compare the performance of FaceNet on pairs of images from the MS-Celeb-1M test set against generated samples from our trained SD-GAN models and AC-DCGAN baseline.
To match FaceNet’s training data, we preprocess all images by resizing from 64 x 64 to 160 x 160 , normalizing each image individually.
We prepare 10,000 pairs from MS-Celeb-1M, half identity-matched and half unmatched.
From each generative model, we generate 5,000 pairs each with z 1 I =
z 2 I and 5,000 pairs with z 1 I (cid:54) =
z 2 I .
For each sample, we draw observation vectors z
O randomly.
We also want to ensure that identity-matched images produced by the generative models are diverse.
To this end, we propose an intra-identity sample diversity (ID-Div) metric.
The multi-scale structural similarity (MS-SSIM)
(Wang et al., 2004) metric reports the similarity of two images on a scale from 0 (no resemblance) to 1 (identical images).
We report 1 minus the mean MS-SSIM for all pairs of identity-matched images as ID-Div.
To measure the overall sample diversity (All-Div), we also compute 1 minus the mean similarity of 10 k pairs with random identities.
In Table 1, we report the area under the receiver operating characteristic curve (AUC), accuracy, and false accept rate (FAR) of FaceNet (at threshold τ v ) on the real and generated data.
We also report our proposed diversity statistics.
FaceNet veriﬁes pairs from the real data with 87% accuracy compared to 86% on pairs from our SD-BEGAN model.
Though this is comparable to the accuracy achieved on pairs from the AC-DCGAN baseline, our model produces samples that are more diverse in pixel space (as measured by ID-Div and All-Div).
FaceNet has a higher but comparable FAR for pairs from SD-GANs than those from AC-DCGAN; this indicates that SD-GANs may produce images that are less semantically diverse on average than AC-DCGAN.
We also report the combined memory footprint of G and D for all methods in Table 1.
For conditional
GAN approaches, the number of parameters grows linearly with the number of identities in the training data.
Especially in the case of the AC-GAN, where the discriminator computes a softmax over all identities, linear scaling may be prohibitive.
While our 10 k-identity subset of MS-Celeb-1M requires a 131 MB AC-DCGAN model, an AC-DCGAN for all 1 M identities would be over 8 GB, with more than 97 % of the parameters devoted to the weights in the discriminator’s softmax layer.
In contrast, the complexity of SD-GAN is constant in the number of identities.
3.2. Q UALITATIVE
In addition to validating that identity-matched SD-GAN samples are veriﬁed by FaceNet, we also demonstrate that humans are similarly convinced through experiments using Mechanical Turk.
For these experiments, we use balanced subsets of 1,000 pairs from MS-Celeb-1M and the most promising generative methods from our FaceNet evaluation.
We ask human annotators to determine if each pair depicts the “same person” or “different people”.
Annotators are presented with batches of ten pairs at a time.
Each pair is presented to three distinct annotators and predictions are determined by majority vote.
Additionally, to provide a benchmark for assessing the quality of the Mechanical Turk ensembles, we (the authors) manually judged 200 pairs from MS-Celeb-1M. Results are in Table 1.
For all datasets, human annotators on Mechanical Turk answered “same person” less frequently than
FaceNet when the latter uses the accuracy-maximizing threshold τ v .
Even on real data, balanced so that 50% of pairs are identity-matched, annotators report “same person” only 28% of the time (compared to the 41% of FaceNet).
While annotators achieve higher accuracy on pairs from ACDCGAN than pairs from SD-BEGAN, they also answer “same person” 16% more often for ACDCGAN pairs than real data.
In contrast, annotators answer “same person” at the same rate for
SD-BEGAN pairs as real data.
This may be attributable to the lower sample diversity produced by
AC-DCGAN.
Samples from SD-DCGAN and SD-BEGAN are shown in Figures 3 and 1 respectively.
R ELATED WORK
Style transfer and novel view synthesis are active research areas.
Early attempts to disentangle style and content manifolds used factored tensor representations (Tenenbaum & Freeman, 1997; Vasilescu & Terzopoulos, 2002; Elgammal & Lee, 2004; Tang et al., 2013), applying their results to face image synthesis.
More recent work focuses on learning hierarchical feature representations using deep convolutional neural networks to separate identity and pose manifolds for faces (Zhu et al., 2013;
Reed et al., 2014; Zhu et al., 2014;
Yang et al., 2015; Kulkarni et al., 2015; Oord et al., 2016; Yan et al., 2016) and products (Dosovitskiy et al., 2015).
Gatys et al.
(2016) use features of a convolutional network, pretrained for image recognition, as a means for discovering content and style vectors.
Since their introduction (Goodfellow et al., 2014), GANs have been used to generate increasingly highquality images (Radford et al., 2016; Zhao et al., 2017b; Berthelot et al., 2017).
Conditional GANs (cGANs), introduced by Mirza & Osindero (2014), extend GANs to generate class-conditional data.
Odena et al.
(2017) propose auxiliary classiﬁer GANs, combining cGANs with a semi-supervised discriminator (Springenberg, 2015).
Recently, cGANs have been used to ingest text (Reed et al., 2016) and full-resolution images (Isola et al., 2017; Liu et al., 2017; Zhu et al., 2017) as conditioning information, addressing a variety of image-to-image translation and style transfer tasks.
Chen et al.
(2016) devise an information-theoretic extension to GANs in which they maximize the mutual information between a subset of latent variables and the generated data.
Their unsupervised method appears to disentangle some intuitive factors of variation, but these factors may not correspond to those explicitly disentangled by SD-GANs.
Several related papers use GANs for novel view synthesis of faces.
Tran et al.
; Huang et al.
Yin et al.
(2017a;b);
Zhao et al.
(2017a)
all address synthesis of different body/facial poses conditioned on an input image (representing identity) and a ﬁxed number of pose labels.
Antipov et al.
(2017) propose conditional GANs for synthesizing artiﬁcially-aged faces conditioned on both a face image and an age vector.
These approaches all require explicit conditioning on the relevant factor (such as rotation, lighting and age) in addition to an identity image.
In contrast, SD-GANs can model these contingent factors implicitly (without supervision).
Mathieu et al.
(2016) combine GANs with a traditional reconstruction loss to disentangle identity.
While their approach trains with an encoder-decoder generator, they enforce a variational bound on the encoder embedding, enabling them to sample from the decoder without an input image.
Experiments with their method only address small ( 28 x 28 ) grayscale face images, and their training procedure is complex to reproduce.
In contrast, our work offers a simpler approach and can synthesize higher-resolution, color photographs.
One might think of our work as offering the generative view of the Siamese networks often favored for learning similarity metrics (Bromley, 1994; Chopra et al., 2005).
Such approaches are used for discriminative tasks like face or signature veriﬁcation that share the many classes with few examples structure that we study here.
In our work, we adopt a Siamese architecture in order to enable the discriminator to differentiate between matched and unmatched pairs.
Recent work by Liu & Tuzel (2016) propose a GAN architecture with weight sharing across multiple generators and discriminators, but with a different problem formulation and objective from ours.
D ISCUSSION
Our evaluation demonstrates that SD-GANs can disentangle those factors of variation corresponding to identity from the rest.
Moreover, with SD-GANs we can sample never-before-seen identities, a beneﬁt not shared by conditional GANs.
In Figure 3, we demonstrate that by varying the observation vector
z O , SD-GANs can change the color of clothing, add or remove sunnies, or change facial pose.
They can also perturb the lighting, color saturation, and contrast of an image, all while keeping the apparent identity ﬁxed.
We note, subjectively, that samples from SD-DCGAN tend to appear less photorealistic than those from SD-BEGAN.
Given a generator trained with SD-GAN, we can independently interpolate along the identity and observation manifolds (Figure 5).
On the shoe dataset, we ﬁnd that the SD-DCGAN model produces convincing results.
As desired, manipulating z I while keeping z O ﬁxed yields distinct shoes in consistent poses (Figure 4).
The identity code z I appears to capture the broad categories of shoes (sneakers, ﬂip-ﬂops, boots, etc.).
Surprisingly, neither original BEGAN nor SD-BEGAN can produce diverse shoe images (Appendix G).
In this paper, we presented SD-GANs, a new algorithm capable of disentangling factors of variation according to known commonalities.
We see several promising directions for future work.
One logical extension is to disentangle latent factors corresponding to more than one known commonality.
We also plan to apply our approach in other domains such as identity-conditioned speech synthesis.
A CKNOWLEDGEMENTS
The authors would like to thank Anima Anandkumar, John Berkowitz and Miller Puckette for their helpful feedback on this work.
This work used the Extreme Science and Engineering Discovery
Environment (XSEDE), which is supported by National Science Foundation grant number ACI1053575 (Towns et al., 2014).
GPUs used in this research were donated by the NVIDIA Corporation.
A E STIMATING LATENT CODES
We estimate latent vectors for unseen images and demonstrate that the disentangled representations of SD-GANs can be used to depict the estimated identity with different contingent factors.
In order to ﬁnd a latent vector ˆ z such that G (ˆ z ) (pretrained G ) is similar to an unseen image x , we can minimize the distance between x and G (ˆ z ) :
min ˆ z || G (ˆ z ) − x || 22 (Lipton & Tripathi, 2017).
In Figure 6, we depict estimation and linear interpolation across both subspaces for two pairs of images using SD-BEGAN.
We also display the corresponding source images being estimated.
For both pairs, ˆ z I (identity) is consistent in each row and ˆ z O (observation) is consistent in each column.
B P AIRWISE DISCRIMINATION OF EMBEDDINGS AND ENCODINGS
In Section 3.1, we describe an AC-GAN (Odena et al., 2017) baseline which uses an embedding matrix over real identities as latent identity codes ( G : i, z O
(cid:55)→ ˆx ).
In place of random identity vectors, we tried combining this identity representation with pairwise discrimination (in the style of
SD-GAN).
In this experiment, the discriminator receives either either two real images with the same identity ( x 1 i , x 2 i ) , or a real image with label i and synthetic image with label i ( x 1 i , G ( i, z O )) .
All other hyperparameters are the same as in our SD-DCGAN experiment (Section 3.1).
We show results in Figure 7.
In Appendix C, we detail a modiﬁcation of the DR-GAN (Tran et al., 2017) method which uses an encoding network G e to transform images to identity representations ( G d : G e ( x ) , z
O (cid:55)→ ˆx ).
We also tried combining this encoder-decoder approach with pairwise discrimination.
The discriminator receives either two real images with the same identity ( x 1 i , x 2 i ) , or ( x 1 i , G d ( G e ( x 1 i ) , z O ) .
We show results in Figure 8.
While these experiments are exploratory and not part of our principle investigation, we ﬁnd the results to be qualitatively promising.
We are not the ﬁrst to propose pairwise discrimination with pairs of (real, real) or (real, fake) images in GANs (Pathak et al., 2016; Isola et al., 2017).
C E XPLORATORY EXPERIMENT WITH DR-GAN S
Tran et al.
(2017) propose Disentangled Representation learning-GAN (DR-GAN), an approach to face frontalization with similar setup to our SD-GAN algorithm.
The (single-image) DR-GAN generator G (composition of G e and G d ) accepts an input image x , a pose code c , and a noise vector z .
The DR-GAN discriminator receives either x or ˆx =
G d ( G e ( x ) , c , z ) .
In the style of (Springenberg, 2015), the discriminator is tasked with determining not only if the image is real or fake, but also classifying the pose c , suggesting a disentangled representation to the generator.
Through their experiments, they demonstrate that DR-GAN can explicitly disentangle pose and illumination ( c ) from the rest of the latent space ( G e ( x ); z ).
In addition to our AC-DCGAN baseline (Odena et al., 2017), we tried modifying DR-GAN to only disentangle identity (rather than both identity and pose in the original paper).
We used the
DCGAN (Radford et al., 2016) discriminator architecture (Table 4) as G e , linearly projecting the ﬁnal convolutional layer to G e ( x ) ∈ R 50 (in alignment with our SD-GAN experiments).
We altered the discriminator to predict the identity of x or ˆx , rather than pose information (which is unknown in our experimental setup).
With these modiﬁcations, G e ( x ) is analogous to z I in the SD-GAN generator, and z is analogous to z O .
Furthermore, this setup is identical to the AC-DCGAN baseline except that the embedding matrix is replaced by an encoding network G e .
Unfortunately, we found that the generator quickly learned to produce a single output image ˆx for each input x
regardless of observation code z (Figure 9).
Accordingly, we excluded this experiment from our evaluation (Section 3.2).
D I MAGINING IDENTITIES WITH AC-GAN
As stated in Section 3.1, AC-GANs Odena et al.
(2017) provide no obvious way to imagine new identities.
For our evaluation (Section 3.2), the AC-GAN generator receives identity input z I ∈ [0 , 1] 10000 : a one-hot over all identities.
One possible approach to imagining new identities would be to query a trained AC-GAN generator with
a random vector z
I such that (cid:80)
i
=
z I
[ i ] = 1 .
We found that this strategy produced little identity variety (Figure 10) compared to the normal one-hot strategy (Figure 11) and excluded it from our evaluation.
E A RCHITECTURE D ESCRIPTIONS
We list here the full architectural details for our SD-DCGAN and SD-BEGAN models.
In these descriptions, k is the number of images that the generator produces and discriminator observes per identity (usually 2 for pairwise training), and d I is the number of dimensions in the latent space
Z I (identity).
In our experiments, dimensionality of Z O is always 100 − d I .
As a concrete example, the bottleneck layer of the SD-BEGAN discriminator autoencoder (“fc2” in Table 6) with k = 2 , d I =has output dimensionality 150 .
We emphasize that generators are parameterized by k in the tables only for clarity and symmetry with the discriminators.
Implementations need not modify the generator; instead, k can be collapsed into the batch size.
For the stacked-channels versions of these discriminators, we simply change the number of input image channels from 3 to 3 k and set k = 1 wherever k appears in the table.
[Table]
[Table] [Table]
[Table] [Table]
F F ACE S AMPLES
We present samples from each model reported in Table 1 for qualitative comparison.
In each matrix, z I is the same across all images in a row and
z O is the same across all images in a column.
We draw identity and observation vectors randomly for these samples.
G S HOE S AMPLES
We present samples from an SD-DCGAN and SD-BEGAN trained on our shoes dataset