M EMORIZATION P RECEDES G ENERATION : L EARNING U NSUPERVISED GAN S WITH M EMORY N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose an approach to address two undesired properties of unsupervised
GANs.
First, since GANs use only a continuous latent distribution to embed multiple classes or clusters of a dataset, GANs often do not correctly handle the structural discontinuity between disparate classes in a latent space.
Second, discriminators of GANs easily forget about past generated samples by generators, incurring instability during adversarial training.
We argue that these two infamous problems of unsupervised GANs can be largely alleviated by a memory structure to which both generators and discriminators can access.
Generators can effectively store a large amount of training samples that are needed to understand the underlying cluster distribution, which eases the structure discontinuity problem.
At the same time, discriminators can memorize previously generated samples, which mitigate the forgetting problem.
We propose a novel end-to-end GAN model named memoryGAN , that involves a memory network that can be trained in an unsupervised manner, and integrable to many existing models of GANs.
With evaluations on multiple datasets including Fashion-MNIST, CelebA, CIFAR10, and Chairs, we show that our model is probabilistically interpretable, and generates image samples of high visual ﬁdelity.
We also show that our memoryGAN also achieves the state-of-the-art inception scores among unsupervised GAN models on the CIFAR10 dataset, without additional tricks or weaker divergences.
I NTRODUCTION
Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) are one of emerging branches of unsupervised models for deep neural networks.
It consists of two neural networks named generator and discriminator that compete each other in a zero-sum game framework.
GANS have been successfully applied to multiple generation tasks, including image syntheses ( e.g .
(Reed et al., 2016b;
Radford et al., 2016; Zhang et al., 2016a)), image super-resolution ( e.g .
(Ledig et al., 2017; Snderby et al., 2017)), image colorization ( e.g .
(Zhang et al., 2016b)), to name a few.
Despite such remarkable progress, GANs are notoriously difﬁcult to train.
Currently, such training instability problems have mostly tackled by ﬁnding better distance measures ( e.g .
(Li et al., 2015;
Nowozin et al., 2016;
Arjovsky & Bottou, 2017; Arjovsky et al., 2017; Gulrajani et al., 2017; Warde-Farley & Bengio, 2017; Mroueh et al., 2017; Mroueh & Sercu, 2017)) or regularizers ( e.g . (Salimans et al., 2016;
Metz et al., 2017; Che et al., 2017; Berthelot et al., 2017)).
In this paper, we address two undesired properties of unsupervised GANs that cause instability during training.
The ﬁrst one is that GANs use a unimodal continuous latent space ( e.g . Gaussian distribution), and thus fail to handle structural discontinuity between different classes or clusters.
It partly attributes to the infamous mode collapsing problem.
For example, GANs embed both building and cats into a common continuous latent distribution, even though there is no intermediate structure between them.
Hence, even a perfect generator would produce unrealistic images for some latent codes that reside in intermediate regions of two disparate classes.
Fig.1 (a,c) visualize this problem with examples of afﬁne-transformed MNIST and Fashion-MNIST datasets.
There always exist latent space regions that cause unrealistic samples (red boxes) between different classes (blue boxes at the corners).
Another problem is the forgetting behavior of the discriminator about past synthesized samples by the generator, during adversarial training of GANs.
The catastrophic forgetting has explored in deep
[Figure]
neural network research, such as (Kirkpatrick et al., 2016; Kemker et al., 2017).
In the context of
GANs, Shrivastava et al.
(2017) address the problem that the discriminator often focuses only on the latest input images.
Since the loss functions of two networks depend on each other’s performance, such forgetting behavior results in serious instability, including causing the divergence of adversarial training, or making the generator re-introduce artifacts that the discriminator has forgotten.
In this work, we claim that a simple memorization module can effectively mitigate both infamous instability issues of unsupervised GANs.
First, to ease the structure discontinuity problem, the memory can store a large amount of training samples that help the generator better understand the underlying class or cluster distribution of a dataset.
Fig.1 (b,d) illustrate some examples that key values of our memory network successfully learn implicit image clusters.
Thus, we can separate cluster sampling with latent variable sampling in the generative process, which helps reduce the structural discontinuity of a latent space.
Second, to mitigate the forgetting problem, the memory can memorize previously generated samples by the generator, including even rare ones.
It makes the discriminator train robust to temporal proximity of speciﬁc batch inputs.
Based on these intuitions, we propose a novel end-to-end GAN model named memoryGAN , that involves a life-long memory network to which both generator and discriminator can access.
Our model can learn a highly multi-modal latent space of data in an unsupervised way, without additional tricks or weaker divergences.
Moreover, the proposed memory structure is orthogonal to generator and discriminator design, and thus is integrable to many existing variants of GANs.
In our experiments, we show that our model can generate image samples of competitive visual ﬁdelity on several benchmark datasets, including Fashion-MNIST (Xiao et al., 2017), CelebA (Liu et al., 2015),
CIFAR10 (Krizhevsky, 2009), and Chairs (Aubry et al., 2014).
Our memoryGAN also achieves the state-of-the-art inception scores among unsupervised GAN models on the CIFAR10 dataset.
We summarize the contributions of this paper as follows
• We propose MemoryGAN as a novel unsupervised framework to resolve the two key instability issues of existing GAN training, the structural discontinuity in a latent space and the forgetting problem of GANs.
To the best of our knowledge, our model is a ﬁrst attempt to incorporate a memory network module with an unsupervised GAN model.
• In our experiments, we show that our model is probabilistically interpretable by visualizing data likelihoods, learned categorical priors, and posterior distributions of memory slots.
We qualitatively show that our model generates realistic image samples of high visual ﬁdelity on CIFAR10, CelebA, and Chairs dataset.
We also outperform existing unsupervised GAN models in terms of inception scores on the CIFAR10 dataset.
We plan to make public source codes and pretrained model parameters.
R ELATED W ORK
Memory networks .
Augmenting neural networks with memory has been heavily studied recently (Bahdanau et al., 2014; Graves et al., 2014; Sukhbaatar et al., 2015).
In these early memory models, computational requirement necessitates the memory size to be small.
Some networks such as (Weston et al., 2014; Xu et al., 2016) use large-scale memory but its size is ﬁxed prior to training.
Recently, Kaiser et al.
(2017) extend (Santoro et al., 2016;
Rae et al., 2016) and propose a large-scale life-long memory network, which does not need to be reset during training.
It exploits nearest-neighbor algorithms for efﬁcient memory lookup, and thus scales to a large memory size.
Unlike previous approaches, our memory network in the discriminator is designed based on the Gaussian mixture model and incremental Expectation-Maximization (EM) algorithm.
We will further specify the uniqueness of read, write and sampling mechanism of our memory network in section 3.1.
There have been a few models that strengthen the memorization capability of generative models.
Li et al.
(2016) use additional trainable parameter matrices as a form of memory for deep generative models, but they do not consider the GAN framework.
Arici & Celikyilmaz (2016) use RBMs as an associative memory, to transfer some knowledge of the discriminator’s feature distribution to the generator.
However, they do not address the two problems of our interest, the structural discontinuity and the forgetting problem; as a result, their memory structure is completely different with ours.
Structural discontinuity in a latent space .
Previous works have addressed this problem by splitting the latent vector into multiple subsets, and allocating a separate distribution for each data cluster.
That is, many conditional GANs concatenate random noises with vectorized external information like class labels or text embedding, which serve as cluster identiﬁers, (Mirza & Osindero, 2014;
Gauthier, 2015; Reed et al., 2016b; Zhang et al., 2016a;
Reed et al., 2016a; Odena et al., 2017;
Dash et al., 2017).
In an unsupervised setting, however, such networks are not applicable because they require supervision of conditional information.
Some image editing GANs and cross-domain transfer GANs extract the content information from an input data using an auxiliary encoder network (Yan et al., 2016;
Perarnau et al., 2016; Antipov et al., 2017; Denton et al., 2016;
Lu et al., 2017;
Zhang et al., 2017; Isola et al., 2017; Taigman et al., 2017;
Kim et al., 2017; Zhu et al., 2017).
However, they require additional auxiliary encoder network, and transform only a given image rather than generating from the entire sample space.
On the other hand, MemoryGAN learns clusters of data without supervision, additional encoder network, or a source image to edit from.
Fogetting problem .
It is a well-known problem that the discriminator of GANs is prone to forgetting past samples that the generator synthetizes.
Multiple studies such as (Salimans et al., 2016;
Shrivastava et al., 2017) address the forgetting problem in GANs and make a consensus on the need for memorization mechanism.
In order for GANs to be less sensitive to temporal proximity of speciﬁc batch inputs, Salimans et al.
(2016) add a regularization term of the (cid:96) 2 -distance between current network parameters and the running average of previous parameters, to prevent the network from revisiting previous parameters during training.
Shrivastava et al.
(2017) modify the adversarial training algorithm to involve a buffer of synthetic images generated by the previous generator network.
Instead of adding a regularization term or modifying GAN algorithm, we explicitly increase the model’s memorization capacity by introducing a life-long memory into the discriminator.
T HE M EMORY GAN
Fig.2 shows the proposed MemoryGAN architecture, which includes a novel discriminator network named Discriminative Memory Network (DMN) and a generator network as Memory Conditional
Generative Network (MCGN).
We describe DMN and MCGN in section 3.1–3.3, and then discuss how our MemoryGAN can resolve the two instability issues in section 3.4.
3. T HE D ISCRIMINATIVE M EMORY N ETWORK
The DMN consists of an inference network and a memory network.
The inference network µ is a convolutional neural network (CNN), whose input is a datapoint x ∈ R D and output is a normalized query vector q = µ ( x ) ∈ R M with (cid:107) q (cid:107) = 1 .
Then the memory module takes the query vector as input and decides whether x is real or fake: y ∈ { 0 , 1 } .
[Figure]
Keys
P(!)
P(%|+ = 1)
Q R
P(U = 1|)) " {Q , ,Q . ,…,Q X }
P(%|))
Slot Histogram {ℎ , ,ℎ . ,…,ℎ X }
Values {+ , ,+ . ,…,+ X } {+ _ ; ,…,+ _ = }
[Figure]
Our memory network partly borrows some mechanisms of the life-long memory network (Kaiser et al., 2017), that is free to increase a memory size, and has no need to be reset during training.
Our memory also uses the k-nearest neighbor indexing for efﬁcient memory lookup and adopts the Least
Recently Used (LRU) scheme for memory update.
Nonetheless, there are several novel features of our memory structure as follows.
First, our method is probabilistically interpretable; we can easily compute the data likelihood, categorical prior and posterior distribution of memory indices.
Second, our memory learns the approximate distribution of a query by maximizing likelihood using an incremental EM algorithm.
Third, our memory is optimized from the GAN loss, instead of the memory loss proposed in (Kaiser et al., 2017).
Fifth, our method tracks the slot histogram to determine the degree of contributions of each sample to the slot and sampling purpose.
Below we ﬁrst discuss how to compute the discriminative probability p ( y | x ) for a given real or fake sample x at inference, using the memory and the learned inference network (section 3.1.1).
We then explain how to update the memory during training (section 3.1.2).
3.1. T HE D ISCRIMINATIVE O UTPUT
For a given sample x , we ﬁrst ﬁnd out which memory slots should be referred for computing the discriminative probability.
We use c ∈ { 1 , 2 , . . .
, N } to denote a memory slot index.
We represent the posterior distribution over memory indices using a Gaussian mixture model as
[Equation]
where q = µ ( x ) and σ 2 is a ﬁxed variance.
The categorical prior of the memory indices, p ( c ) , is directly obtained by nomalizing the slot histogram: p ( c = i )
=
h i
[Equation]
β
[Equation]
Nj =
[Equation]
h j + β ) , where β (= 10 − 8 ) is a small smoothing constant for numerical stability.
We further simplify Eq.(1) using (cid:107) µ ( x ) (cid:107)
=
(cid:107) K
i (cid:107) = 1 and eliminating the common denominators:
[Equation]
By using p ( y = 1 | c
= i, x ) = v i and Eq.(2), we can estimate the discriminative probability p
( y = 1 | x ) by marginalizing the joint probability p ( y = 1 , c | x ) with respect to c :
[Equation]
However, in practice, it is not scalable to exhaustively sum over the whole memory of size N for each x ; we approximate the predictive probability by using a subset of keys.
We choose indices of k slots S
= { s 1 , ..., s k } that have k -largest posterior probability ( e.g .
we use k = 128 ∼ 256 ):
[Equation]
For efﬁciency, we use approximated discriminative output p ( y | x ) ≈ (cid:80)
i ∈
S
v i p ( c = i | x ) .
We clip p ( y | x ) into [ (cid:15), 1 − (cid:15) ] with a small constant (cid:15) = 0 .
001 for numerical stability.
3.1. M EMORY U PDATE
The memory keys and values are updated during the training.
We adopt both a conventional memory update mechanism and an incremental EM algorithm.
We denote a training sample x (cid:48) and its label y (cid:48) that is 1 for real and 0 for fake.
For each x (cid:48) ,
we ﬁrst ﬁnd k -nearest slots S y as done in Eq.(4), except that we here use the conditional posterior p ( c | x, v c = y (cid:48) ) instead of p ( c | x ) , in order to perform the
EM algorithm using only the slots that belong to the same class with y (cid:48) .
Then we update the memory in two different ways, according to whether v ( S y ) contains the correct label y (cid:48) or not.
If there is no correct label in the slots of S y , we ﬁnd the oldest memory slot by n a = argmax a ( S y ) , and set to
K n a ← q (cid:48)
=
µ ( x (cid:48) ) , v n a ← y (cid:48) , a n a ← 0 , and h n a ← 1 N (cid:80) Ni =
1 h i .
If S y contains a correct value, the memory keys are updated using the following modiﬁed incremental EM algorithm for T iterations.
In the expectation step, we compute posterior γ ti = p ( c i | x ) for i ∈ S y , by applying previous keys ˆ K t − 1 i and ˆ h t − 1 i to Eq.(2).
In the maximization step, we update the required sufﬁcient statistics.
[Equation]
where t ∈ 1 , ..., T , γ 0 = 0 ,
ˆ K 1 i =
K i , ˆ h 1 i
=
αh i and α = 0 .
After T iterations, we update the slots of S y by K
i ←
ˆ K ti and h
i ← ˆ h ti .
The decay rate α controls how much it exponentially reduces the contribution of old queries to the slot position of the means of mixture components.
The α value is important for the performance, give than old queries that were used to update a key previously are unlikely to be ﬁt to the current mixture distribution, since the inference network µ itself updates during the training.
Finally, it is worth noting that this memory updating mechanism is orthogonal to any adversarial training algorithm, because it is performed separately while the discriminator is updated.
Moreover, adding our memory module does not affects the running speed of the model at test time, since the memory is updated only at training time.
3. T HE M EMORY -C ONDITIONAL G ENERATIVE N ETWORK
Our generative network MCGN is largely based on the conditional GANs of InfoGAN (Chen et al., 2016), However, one key difference is that the MCGN synthesizes a sample not only conditioned on a random noise vector z ∈ R D z , but also on conditional memory information.
That is, in addition to sample z from a Gaussian, the MCGN samples a memory index i from P ( c = i | v c = 1)
=
h i v i (cid:80)
Nj h j , which reﬂects the exact appearance frequency of the memory cell i within real data.
Finally, the generator synthesizes a fake sample from the concatenated representation [ K i , z ] , where K i is a key value vector of the memory index i .
Unlike other conditional GANs, MCGN requires neither additional annotation nor any external encoder network.
Instead, MCGN makes use of conditional memory information that DMN learns in an unsupervised way.
Recall that DMN learns the Gaussian mixture memory with only the query representation q (cid:48)
=
µ ( x (cid:48) ) of each training sample x (cid:48) and its indicator y (cid:48) .
Finally, we summarize the training algorithm of MemoryGAN in Algorithm 1.
3. T HE O BJECTIVE F UNCTION
Our objective is based on that of InfoGAN (Chen et al., 2016), which maximizes the mutual information between a subset of latent variables and observations.
We add another mutual information loss term between K i and G ( z, K i ) , to ensure that the structural information is consistant between do 2:
Sample Sample
Update by the φ ←
+ η ∇
L 5:
Find S data the minibatch 6:
Initialize γ 0 s ← 0 , ˆ h 0 s ← αh s and ˆ K 0 s ← K s for s ∈ S y 7: for number
EM iterations do 8:
Estimate posterior γ ts for s ∈ S y 9: ˆ h ts ← ˆ h t − 1 s + γ ts − γ t − 1 s for s ∈ S y 10: ˆ
K ts ← ˆ K t − 1 s + γ ts − γ t − 1 s ˆ h
ts ( q i − ˆ K ts ) for s ∈ S y 11:
Update GMM h s y ← ˆ h
Ts y , K s y ←
ˆ K Ts y for s y ∈ S
y 12:
Sample a minibatch of noises z ∼ p z ) and memory indices c ∼ p ( c | v = 1) .
Update by gradient descent θ ← θ − η ∇ θ L a sampled memory information
K
i and a generated sample G ( z, K i ) from it:
[Equation]
(6) where ˆ I is the mean-squared error
ˆ I = (cid:107)
q − K i (cid:107) 2 .
We defer the derivation of Eq.(6) to Appendix A.1.
Finally, the modiﬁed GAN objective can be written with the lower bound of mutual information, with a hyperparameter λ ( e.g .
we use λ = 2 ·
[Equation]
3. H OW DOES memoryGAN M ITIGATE THE T WO I NSTABILITY I SSUES ?
Our memoryGAN implicitly learns the joint distribution p ( x, z, c )
=
p ( x | z, c ) p ( z ) p ( c ) , for which we assume the continuous latent variable z and the discrete memory latent variable c are independent.
This assumption reﬂects our intuition that when we synthesize a new sample ( e.g . an image), we separate the modeling of its class or cluster ( e.g . an object category) from the representation of other image properties, attributes, or styles ( e.g . the rotation and translation of the object).
Such separation of modeling duties can largely alleviate the structural discontinuity problem in our model.
For data synthesis, we sample K i and z as an input to the generator.
K i represents one of underlying clusters of training data that the DMN learns in a form of key values.
Thus, z does not need to care about the class discontinuity but focus on the attributes or styles of synthesized samples.
Our model suffers less from the forgetting problem, intuitively thanks to the explicit memory.
The memory network memorizes high-level representation of clusters of real and fake samples in a form of key value vectors.
Moreover, the DMN allocates memory slots even for infrequent real samples and maintains the learned slot histogram h nonzero for them ( i.e . p ( c | v c = 1) (cid:54) = 0 ).
It opens a chance of sampling from rare ones for a synthesized sample, although their chance could be low.
E XPERIMENTS
We present examples of probabilistic interpretation of MemoryGAN to understand how it works in section 4.1.
We show both qualitative and quantitative results of image generation in section 4.2 on
Fashion-MNIST (Xiao et al., 2017), CelebA (Liu et al., 2015), CIFAR10 (Krizhevsky, 2009), and
Chairs (Aubry et al., 2014).
We perform ablation experiments to demonstrate the usefulness of key components of our model in section 4.3.
We present more experimental results in Appendix B.
We use DCGAN (Radford et al., 2016) for our inference network and generator, except the experiments on CIFAR10 and CelebA, for which we use the one similar to WGAN-GP ResNet (Gulrajani et al., 2017).
We use minibatches of size 64 , a learning rate of 2 · 10 − 4 , and Adam (Kingma & Ba, 2014) optimizer for all experiments.
[Figure]
og - li k e li hood 1e+ 1e3.
Iterations
S o r t e d s l o t i nd i ce s
Memory Slots −1.
−1. −1.
−1. −1.
[Figure]
1e-
P
[Figure]
s t
[Figure]
r i
[Figure]
r
[Figure] [Figure]
[Table]
DFM (Warde-Farley & Bengio, 2017)
[Table] [Table]
P ROBABILISTIC I NTERPRETATION
Fig.3 presents some probabilistic interpretation of MemoryGAN to gain intuitions of how it works.
We train MemoryGAN using Fashion-MNIST (Xiao et al., 2017), with a hyperparameter setting of z ∈ R 2 , the memory size N = 4 , 096 , the key dimension M = 256 , the number of nearest neighbors k = 128 , and λ = 0 .
Fig.3(a) shows the log-likelihood p ( X | y
= 1) of 10 , 000 real test images ( i.e . unseen for training), while learning the model up to reaching its highest inception score.
Although the inference network keeps updated during adversarial training, the GMM-based memory successfully tracks its approximate distribution; as a result, the likelihood keeps increasing.
Fig.3(b) show posteriors p ( c | x ) of nine randomly selected memory slots for a a random input image (shown at the left-most column in the red box).
We observe that the memory slots are properly activated, in that the memory slots with highest posteriors indeed include the information of the same class and similar styles like( e.g . shapes or colors) to the input image.
The slot images below the bars are drawn by the generator using its slot key value
V i and a random noise z .
Finally, Fig.3(c) shows categorical prior distribution p ( c ) of memory slots.
At the initial state, the inference network is not well trained, so it uses only a small number of slots even for different examples.
As training proceeds, the prior spreads along N memory slots, which means the GMM-based memory fully utilizes the memory slots to distribute training samples according to the clusters of the data distribution.
4. I MAGE G ENERATION P ERFORMANCE
We perform both quantitative and qualitative evaluations on our model’s ability to generate realistic images.
Table 1 shows that our model outperforms state-of-the-art unsupervised GAN models in terms of inception scores with no additional divergence measure or auxiliary network on the CIFAR10 dataset.
Interestingly, we use the exact same implementation of DCGAN only except our memory network, and the improvement is signiﬁcant from 5.35 to 8.04.
We carry out no other hyperparameter tuning for our method, except for using Layer Normalization(Ba et al., 2016) instead of Batch Normalization and using ELU activation function instead of ReLU and Leaky ReLU.
Un-
[Figure]
[Table]
fortunately, except CIFAR10, there are no reported inception scores for other datasets, on which we do not compare quantitatively.
Fig.4 shows generated samples on CIFAR10, CelebA, and Chairs dataset, on which our model achieves competitive visual ﬁdelity.
We observe that more regular the shapes of classes are, the more realistic generated images are.
For example, car or chair images are more realistic, while the faces of dogs and cats , hairs , and sofas are less.
We set N = 16 , 384 , M = 512 , k =
[Table]
and λ = 1 e − 6 for all datasets.
We use z ∈ R 128 for DCGAN and z ∈ R 16 for MemoryGAN.
4. A BLATION S TUDY
We performe a series of ablation experiments to demonstrate that key components of MemoryGAN indeed improve the inception scores.
The variants are as follows.
(i) (– EM) is our MemoryGAN but adopts the memory updating rule of Kaiser et al.
(2017) ( i.e . K t ← K t − 1 + q (cid:107) K t − 1 + q (cid:107) ).
(ii) (– MCGN) removes the slot-sampling process from the generative network, which equals to the DCGAN that uses
DMN as discriminator.
(iii) (– Memory) is equivalent to the original DCGAN.
Results in 2 show that each of proposed components of our MemoryGAN makes signiﬁcant contribution to its outstanding image generation performance.
As expected, without the memory network, the performance is the worst over all datasets.
C ONCLUSION
We proposed a novel end-to-end unsupervised GAN model named memoryGAN , that effectively learns a highly multi-modal latent space and does not suffer from forgetting.
Empirically, we showed that our model achieved the state-of-the-art inception scores among unsupervised GAN models on the CIFAR10 dataset.
We also demonstrated that our model generates realistic image samples of high visual ﬁdelity on Fashion-MNIST, CIFAR10, CelebA, and Chairs datasets.
As an interesting future work, we can extend our model for few-shot generation that synthesizes rare samples.
A A PPENDIX A: M ORE T ECHNICAL D ETAILS A. D ERIVATION OF T HE O BJECTIVE F UNCTION
Our objective is based on that of InfoGAN (Chen et al., 2016), which maximizes the mutual information between a subset of latent variables and observations.
We add another mutual information loss term between K i and G ( z, K i ) , to ensure that the structural information is consistant between a sampled memory information K i and a generated sample G ( z, K i ) from it:
[Equation]
(8) where Q is a Gaussian distribution N
( K i ; q, I ) .
Since maximizing the lower bound is equivalent to minimizing the mean-squared error ˆ I = (cid:107)
q − K i (cid:107) 2 , the modiﬁed GAN objective can be written with the lower bound of mutual information, with a hyperparameter λ
(we use λ = 2 · 10 − 6 ).
[Equation]
B A PPENDIX B: M ORE E XPERIMENTAL R ESULTS B. E XPERIMENTAL S ETTINGS
We summarize the experimental setting used for each dataset in Table 3.
The test sample size indicates the number of samples that we use to evaluate inception scores.
For Chair and CelebA dataaset, we do not compute the inception scores.
The test classiﬁer indicates which image classiﬁer is used for the inception scores.
For afﬁne-MNIST and Fashion-MNIST, we use a simpliﬁed AlexNet and ResNet, respectively.
The noise dimension is the dimension of z , and the image size indicates the height and width of training images.
For CIFAR10, Chair, and CelebA, we linearly decay the learning rate: α t
= α 0 · (1 − γ − 1 · t/ 1 , 000) , where t is the iteration, and γ is the number of minibatches per epoch.
B. M ORE INTERPOLATION E XAMPLES FOR V ISUALIZING S TRUCTURE IN A L ATENT S PACE
We present more interpolation examples similar to Fig.1.
We use Fashion-MNIST (Xiao et al., 2017) and afﬁne transformed MNIST, in which we randomly rotate MNIST images by 20 ◦ and randomly place it in a square image of 40 pixels.
First, we randomly ﬁx a memory slot c , and randomly choose four noise vectors { z 1 , . . .
, z 4 } .
Then, we interpolate the four noise vectors to extract intermediate noise vectors ˆ z = (1 − e )
z i + ez j , where i, j ∈ { 1 , 2 , 3 , 4 } and e ∈ (0 , 1) , and visualize the generated samples
G (ˆ z, c ) .
Fig.5 shows the results.
We set memory size N = 4 , 096 , the key dimension M = 256 , the number of nearest neighbors k = 128 , and λ = 0 .
01 for both datasets.
We use z ∈ R 32 for DCGAN and z ∈ R 2 for MemoryGAN .
In both datasets, the DMN successfully learns to separate a latent space into multiple modes, using Gaussian mixture memory module. [Figure]
FIg.6 presents results of the same experiments on the Chair dataset.
We here show an example that includes failure cases in (d), where interestingly some unrealistic chair images are observed as the back of chair rotates from left to right.
Fig.7 shows the enlarged versions of Fig.4 for better visibility.
[Figure] [Figure]