G ENERATIVE
M ODELS
OF
V ISUALLY
G ROUNDED
I MAGINATION
Anonymous authors
Paper under double-blind review
A BSTRACT
It is easy for people to imagine what a man with pink hair looks like, even if they have never seen such a person before.
We call the ability to create images of novel semantic concepts visually grounded imagination .
In this paper, we show how we can modify variational auto-encoders to perform this task.
Our method uses a novel training objective, and a novel product-of-experts inference network, which can handle partially speciﬁed (abstract) concepts in a principled and efﬁcient way.
We also propose a set of easy-to-compute evaluation metrics that capture our intuitive notions of what it means to have good visual imagination, namely correctness, coverage, and compositionality (the 3 C’s ).
Finally, we perform a detailed comparison of our method with two existing joint image-attribute VAE methods (the JMVAE method of (Suzuki et al., 2017) and the BiVCCA method of (Wang et al., 2016b)) by applying them to two datasets: the MNIST-with-attributes dataset (which we introduce here), and the CelebA dataset (Liu et al., 2015).
I NTRODUCTION
Consider the following two-party communication game: a speaker thinks of a visual concept C , such as “men with black hair”, and then generates a description y of this concept, which she sends to a listener; the listener interprets the description y , by creating an internal representation z , which captures its “meaning”.
We can think of z as representing a set of “mental images” which depict the concept C .
To test whether the listener has correctly “understood” the concept, we ask him to draw a set of real images S
= { x s : s = 1 : S } , which depict the concept C .
He then sends these back to the speaker, who checks to see if the images correctly match the concept C .
We call this process visually grounded imagination .
In this paper, we represent concept descriptions in terms of a ﬁxed length vector of discrete attributes
A .
This allows us to specify an exponentially large set of concepts using a compact, combinatorial representation.
In particular, by specifying different subsets of attributes, we can generate concepts at different levels of granularity or abstraction.
We can arrange these concepts into a compositional abstraction hierarchy , as shown in Figure 1.
This is a DAG in which nodes represent concepts, and an edge from a node to its parent is added whenever we drop one of the attributes from the child’s concept deﬁnition.
We can describe a concept by creating the attribute vector y O , in which we only specify the value of the attributes in the subset O ⊆ A ; the remaining attributes are unspeciﬁed, and are assumed to take all possible legal values.
For example, consider the following concepts, in order of increasing abstraction:
C msb =
(male , smiling , blackhair) , C ∗ sb =
( ∗ , smiling , blackhair) , and C ∗∗ b = ( ∗ , ∗ , blackhair) , where the attributes are gender, smiling or not, and hair color, and ∗ represents “don’t care”.
A good model should be able to generate images from different levels of the abstraction hierarchy, as shown in Figure 1.
(This is in contrast to most prior work on conditional generative models of images, which assume that all attributes are fully speciﬁed, which corresponds to sampling only from leaf nodes in the hierarchy.)
In Section 2, we show how we can extend the variational autoencoder (VAE) framework of (Kingma & Welling, 2014) to create models which can perform this task.
The ﬁrst extension is to modify the model to the “multi-modal” setting where we have both an image, x , and an attribute vector, y .
More precisely, we assume a joint generative model of the form p ( x , y , z )
=
p ( z ) p ( x | z ) p ( y | z ) , where p ( z ) is the prior over the latent variable z , p ( x | z ) is our image decoder, and p ( y | z ) is our description
[Figure]
decoder.
We additionally assume that the description decoder factorizes over the speciﬁed attributes in the description, so p ( y O | z ) =
(cid:81)
k ∈O p
( y k | z ) .
We further extend the VAE by devising a novel objective function, which we call the TELBO , for training the model from paired data, D = { ( x n , y n ) } .
However, at test time, we will allow unpaired data (either just a description or just an image).
Hence we ﬁt three inference networks: q ( z | x , y ) , q ( z | x ) and q ( z | y ) .
This way we can embed an image or a description into the same shared latent space (using q ( z | x ) and q ( z | y ) , respectively); this lets us “translate” images into descriptions or vice versa, by computing p ( y | x ) = (cid:82)
d z p ( y | z ) q ( z | x ) and p ( x | y ) =
(cid:82) d z p ( x | z ) q ( z | y ) .
To handle abstract concepts (i.e., partially observed attribute vectors), we use a method based on the product of experts (POE) (Hinton, 2002).
In particular, our inference network for attributes has the form q ( z | y O )
∝ p ( z ) (cid:81) k ∈O q
( z | y k ) .
If no attributes are speciﬁed, the posterior is equal to the prior.
As we condition on more attributes, the posterior becomes narrower, which corresponds to specifying a more precise concept.
This enables us to generate a more diverse set of images to represent abstract concepts, and a less diverse set of images to represent concrete concepts, as we show below.
Section 3 discusses how to evaluate the performance of our method in an objective way.
Speciﬁcally, we ﬁrst “ground” the description by generating a set of images, S ( y O ) =
{ x s ∼ p ( x | y O ) : s = 1 :
S } .
We then check that all the sampled images in S ( y O ) are consistent with the speciﬁed attributes y O
(we call this correctness ).
We also check that the set of images “spans” the extension of the concept, by exhibiting suitable diversity (c.f.
(Young et al., 2014)).
Concretely, we check that the attributes that were not speciﬁed (e.g., gender in C ∗ sb above) vary across the different images; we call this coverage .
Finally, we want the set of images to have high correctness and coverage even if the concept y O has a combination of attribute values that have not been seen in training.
For example, if we train on
C msb =
(male , smiling , blackhair) , and C fnb =
(female , notsmiling , blackhair) , we should be able to test on C mnb
= (male , notsmiling , blackhair) , and C fsb =
(female , smiling , blackhair) .
We will call this property compositionality .
Being able to generate plausible images in response to truly compositionally novel queries is the essence of imagination.
Together, we call these criteria the 3 C’s of visual imagination .
Section 5 reports experimental results on two different datasets.
The ﬁrst dataset is a modiﬁed version of MNIST, which we call MNIST-with-attributes (or MNIST-A), in which we “render” modiﬁed versions of a single MNIST digit on a 64x64 canvas, varying its location, orientation and size.
The second dataset is CelebA (Liu et al., 2015), which consists of over 200k face images, annotated with 40 binary attributes.
We show that our method outperforms previous methods on these datasets.
The contributions of this paper are threefold.
First, we present a novel extension to VAEs in the multimodal setting, introducing a principled new training objective (the TELBO), and deriving an interpretation of a previously proposed objective (JMVAE) (Wang et al., 2016a) as a valid alternative in Appendix A.1.
Second, we present a novel way to handle missing data in inference networks based on a product of experts.
Third, we present novel criteria (the 3 C’s) for evaluating conditional generative models of images, that extends prior work by considering the notion of visual abstraction and imagination.
M ETHODS
We start by describing standard VAEs, to introduce notation.
We then discuss our extensions to handle the multimodal and the missing input settings.
Standard VAEs.
A variational autoencoder (Kingma & Welling, 2014) is a latent variable model of the form p θ ( x , z )
= p θ ( z ) p θ ( x | z ) , where p θ ( z ) is the prior (we assume it is Gaussian, p θ ( z ) =
N ( z | 0 , I ) , although this assumption can be relaxed), and p θ ( x | z ) is the likelihood (sometimes called the decoder), usually represented by a neural network.
To perform approximate posterior inference, we ﬁt an inference network (sometimes called the encoder) of the form q φ ( z | x ) , so as to maximize
L ( θ , φ ) =
E ˆ p ( x ) [elbo( x , θ , φ )] , where ˆ p ( x ) = 1 N (cid:80)
Nn =1 δ x n ( x ) is the empirical distribution, and
ELBO is the evidence lower bound:
[Equation]
Here KL( p, q ) is the Kullback Leibler divergence between distributions p and q .
Due to the nonnegativity of the KL divergence, elbo( x , θ , φ ) ≤ log p θ ( x ) , so this is a form of approximate maximum likelihood training.
Joint VAEs and the TELBO.
We extend the VAE to model images and attributes by deﬁning the joint distribution p θ ( x , y , z ) =
p θ ( z ) p θ ( x | z ) p θ ( y | z ) , where p θ ( x | z ) is the image decoder (we use the DCGAN architecture from (Radford et al., 2015)), and p θ ( y | z ) is an MLP for the attribute vector.
The corresponding training objective which we want to maximize becomes L ( θ , φ ) =
E ˆ p ( x , y )
[elbo( x , y , θ , φ )] , where ˆ p ( x , y )
= 1 N (cid:80) Nn
=1 δ x n ( x ) δ y n ( y n ) is the empirical distribution derived from paired data, and the joint ELBO is given by
[Equation]
Here λ y ≥ 1 is a weighting term to prevent the image likelihood p θ ( x | z ) from dominating the lower dimensional attribute likelihood p θ ( y | z ) .
We call this the JVAE (joint VAE) model.
In addition to ﬁtting the joint model, we need to train unpaired inference networks q φ x ( z | x ) and q φ y ( z | y ) , so we can embed images and attributes into the same shared latent space.
A natural objective to ﬁt, say, q φ x ( z | x ) is to maximize the following:
[Equation]
where the last term is constant wrt φ x , and hence can be dropped.
We can use a similar method to ﬁt q φ y ( z | y ) .
Combining these gives the following triple ELBO ( TELBO ) objective:
[Equation]
elbo( x , y , θ , φ ) + elbo( x , θ , φ x ) + elbo( y , θ , φ y ) (cid:3)
In practice, we also use a scaling factor β y for the likelihood term in elbo( y ) to balance the relative contribution of likelihoods across the unimodal ELBO terms.
Since we are training the generative model only on aligned data, and simply retroﬁtting inference networks, we freeze the p θ ( x | z ) and p θ ( y | z ) terms when training the last two ELBO terms above.
In Section 4, we compare this to other methods for training joint VAEs that have been proposed in the literature.
Handling missing attributes.
In order to handle missing attributes at test time, we use a product of experts model, where each attribute instantiates an expert.
Concretely, our q ( z | y ) inference network has the following form: q ( z | y O )
∝ p ( z ) (cid:81) k ∈O q
( z | y k ) , where q ( z | y k ) =
N ( z | µ k ( y k ) , C k ( y k )) is the k th Gaussian “expert”, and p ( z ) =
N ( z | µ 0 = 0 , C 0 = I ) is the prior.
Unlike the product of
[Figure]
experts model in (Hinton, 2002), our model multiplies Gaussians, not Bernoullis, so the product has a closed form solution namely q ( z | y O ) =
N ( z | µ , C ) , where C − 1 = (cid:80) k C − 1 k and µ =
C ( (cid:80) k
C − 1 k µ k ) , and the sum is over all the observed attributes.
This equation has a simple intuitive interpretation: If we do not observe any attributes, the posterior reduces to the prior.
As we observe more attributes, the posterior becomes narrower, since the (positive deﬁnite) precision matrices add up, reﬂecting the increased speciﬁcity of the concept being speciﬁed, as illustrated in Figure(middle).
We always include the prior term, p ( z ) , in the product, since without it, the posterior q φ y ( z | y O ) may not be well-conditioned when we are missing attributes, as illustrated in Figure(right).
For more implementation-level details on the model architectures, see Appendix A.4.
E VALUATION METRICS : T HE 3C’ S OF V ISUAL I MAGINATION
To evaluate the quality of a set of generated images, S ( y O )
= { x s ∼ p ( x | y O ) : s = 1 : S } , we apply a multi-label classiﬁer to each image, to convert it to a predicted attribute vector, ˆ y ( x ) .
This attribute classiﬁer is trained on a large dataset of images and attributes, and is held constant across all methods that are being evaluated.
It plays the role of a human observer.
This is similar in spirit to generative adversarial networks (Goodfellow et al., 2014), that declare a generated image to be good enough if a binary classiﬁer cannot distinguish it from a real image.
(Both approaches avoid the problems mentioned in (Theis et al., 2016) related to evaluating generative image models in terms of their likelihood.)
However, the attribute classiﬁer checks not only that the images look realistic, but also that they have the desired attributes.
To quantify this, we deﬁne the correctness as the fraction of attributes for each generated image that match those speciﬁed in the concept’s description: correctness( S , y O )
=|S| (cid:80) x ∈S 1 |O| (cid:80) k
∈O I (ˆ y ( x ) k
= y k ) .
However, we also want to measure the diversity of values for the unspeciﬁed or missing attributes, M =
A \ O .
We do this by comparing q k , the empirical distribution over values for attribute k induced by the generated set S , to p k , the true distribution for this attribute induced by the training set.
We measure the difference between these distributions using the Jensen-Shannon divergence, since it is symmetric and satisﬁes 0 ≤ JS( p, q ) ≤ 1 .
We then deﬁne the coverage as follows: coverage( S, y O )
= 1 |M| (cid:80) k ∈M (1 − JS( p k , q k )) .
If desired, we can combine correctness and coverage into a single number, by computing the JS divergence between p k and q k for all attributes, where, for observed attributes, p k is a delta function and q k is the empirical distribution.
However, we ﬁnd it helpful to report correctness and coverage separately.
Note that our metric is different from the inception score proposed in (Salimans et al., 2016).
That is deﬁned as follows:
inception =
exp (cid:0) E ˆ p ( x ) [KL( p ( y | x ) , p ( y ))] (cid:1) , where y is a class label.
Expanding the term inside the exponential, we get
[Equation]
A high inception score means that the distribution p ( y | x ) has low entropy, so the generated images match some class, but that the marginal p ( y ) has high entropy, so the images are diverse.
However, the inception score was created to evaluate unconditional generative models of images, so it does not check if the generated images are consistent with the concept y O , and the degree of diversity does not vary in response to the level of abstraction of the concept.
Finally, we can assess how well the model understands compositionality , by checking correctness of its generated images in response to test concepts y O that differ in at least one attribute from the training concepts.
We call this a compositional split of the data.
This is much harder than a standard iid split, since we are asking the model to predict the effects of novel combinations of attributes, which it has not seen before (and which might actually be impossible).
Note that abstraction is different from compositionality – in abstraction we are asking the model to predict the effects of dropping certain attributes instead of predicting novel combinations of attributes.
R ELATED W ORK
In this section, we brieﬂy mention some of the most closely related prior work.
Conditional models.
Many conditional generative image models of the form p ( y | x ) have been proposed recently, where y can be a class label (e.g., (Radford et al., 2015)), a vector of attributes (e.g., (Yan et al., 2016)), a sentence (e.g., (Reed et al., 2016)), another image (e.g., (Isola et al., 2017)), etc.
Such models are usually based on VAEs or GANs.
However, we are more interested in learning a shared latent space from either descriptions y or images x , which means we need to use a joint, symmetric, model.
Joint models.
Several papers use the same joint VAE model as us, but they differ in how it is trained.
In particular, the BiVCCA objective of (Wang et al., 2016b) has the form L ( θ , φ ) =
E ˆ p ( x , y )
[ J ( x , y , θ , φ )] , where
[Equation]
This method results in the model generating the mean image corresponding to each concept, due to the
E q φ y ( z | y ) log p θ ( x , y | z ) term, which requires that z ’s sampled from q φ y ( z | y n ) be good at generating all the the different x n ’s which co-occur with y n .
We show this empirically in Section 5.
This problem can be partially compensated for by increasing µ , but that reduces the KL( q φ ( z | y ) , p θ ( z )) penalty, which is required to ensure q φ y ( z | y ) is a broad distribution with good coverage of the concept.
The JMVAE objective of (Suzuki et al., 2017) has the form
[Equation]
At ﬁrst glance, this objective looks odd, since forcing q φ ( z | y ) to be close to q φ ( z | x , y ) seems undesirable, since the latter will typically be close to a delta function, since there is little posterior uncertainty in z once we see the image x .
However, in Appendix A.1, we use results from (Hoffman & Johnson, 2016) to show that E ˆ p ( x , y ) (cid:104) KL( q φ ( z | x , y ) , q φ y ( z | y ))
(cid:105) can be written in terms of
KL( q avg φ ( z | y ) , q φ y ( z | y )) , where q avg φ ( z | y )
=
E ˆ p ( x | y )
[ q φ ( z | x , y )] is the aggregated posterior over z induced by all images x which are associated with description y .
This ensures that q φ y ( z | y ) will cover the embeddings of all the images associated with concept y , resulting in sharp image samples.
However, since there is no KL( q φ y ( z | y ) , p θ ( z )) term, the diversity of the samples is slightly reduced compared to TELBO, as we show empirically in Section 5.
The SCAN method of (Higgins et al., 2017c)
ﬁrst ﬁts a standard β -VAE model (Higgins et al., 2017b) on unlabeled images, then they freeze p ( x | z ) and q ( z | x ) and ﬁt an encoder and decoder network for attribute vectors, using as a regularizer KL( q ( z | x ) , q ( z | y )) ; this encourages the posterior given labels to cover the posterior for corresponding images.
Although these authors consider compositionality, they do not consider abstraction.
In Appendix A.3, we show that, by construction, β -VAEs can only organize the latent space around visual concepts, so matching the latent distributions of abstract concepts (such as parity in MNIST) will likely result in a poor ﬁt.
Our approach always takes the concepts into consideration, permitting well-organized latent spaces even in the presence of non-visual concepts.
Handling missing inputs.
Conditional generative models of images, of the form p ( x | y ) , have problems with missing input attributes, as do inference networks q ( z | y ) for VAEs.
(Hoffman, 2017) uses MCMC to ﬁt a latent Gaussian model, which can in principle handle missing data; however, he initializes the Markov chain with the posterior mode computed by an inference network, which cannot easily handle missing inputs.
One approach we can use, if we have a joint model, is to estimate or impute the missing values, as follows: ˆ y = arg max y M p ( y M | y O ) , where p ( y M , y O ) models dependencies between attributes.
We can then sample images using p ( x | ˆ y ) .
This approach was used in (Yan et al., 2016) to handle the case where some of the pixels being passed into an inference network were not observed.
However, conditioning on an imputed value will give different results from not conditioning on the missing inputs; only the latter will increase the posterior uncertainty in order to correctly represent less precise concepts with broader support.
Gaussian embeddings.
There are many papers that embed images and text into points in a vector space.
However, we want to represent concepts of different levels of abstraction, and therefore want to map images and text to regions of latent space.
There are some prior works that use Gaussian embeddings for words (Vilnis & McCallum, 2015; Athiwaratkun & Wilson, 2017), sometimes in conjunction with images (Mukherjee & Hospedales, 2016;
Ren et al., 2016).
Our method differs from these approaches in several ways.
First, we maximize the likelihood of ( x , y ) pairs, whereas the above methods learn a Gaussian embedding using a contrastive loss.
Second, our PoE formulation ensures that the covariance of the posterior q ( z | y O ) is adaptive to the data that we condition on.
In particular, it becomes narrower as we observe more attributes (because the precision matrices sum up), which is a property not shared by other embedding methods.
Abstraction and compositionality.
Young et al.
(2014) represent the extension of a concept (described by a noun phrase) in terms of a set of images whose captions match the phrase.
By contrast, we use a parametric probability distribution in a latent space that can generate new images.
Vendrov et al.
(2016) use order embeddings, where they explicitly learn subsumption-like relationships by learning a space that respects a partial order.
In contrast, we reason about generality of concepts via the uncertainty induced by their latent representation.
There has been some work on compositionality in the language/vision literature (see e.g., Atzmon et al.
; Johnson et al.
; Agrawal et al.
(2017)), but
none of these papers use generative models, which is arguably a much more stringent test of whether a model has truly “understood” the meaning of the components which are being composed.
E XPERIMENTAL RESULTS
In this section, we ﬁt the JVAE model to two different datasets (MNIST-A and CelebA), using the TELBO objective, as well as BiVCCA and JMVAE.
We measure the quality of the resulting model using the 3 C’s, and show that our method of handling missing data behaves in a qualitatively reasonable way.
5. MNIST-A
Dataset.
In this section, we report results on the MNIST-A dataset.
This is created by modifying the original MNIST dataset as follows.
We ﬁrst create a compositional concept hierarchy usingdiscrete attributes, corresponding to class label (10 values), location (4 values), orientation (3 values), and size (2 values).
Thus there are 10x2x3x4=240 unique concepts in total.
We then sample ∼example images of each concept, and create both an iid and compositional split of the data.
See
Appendix A.2 for details.
Models and algorithms.
We train the JVAE model on this dataset using TELBO, BiVCCA and
JMVAE objectives.
We use Adam (Kingma & Ba, 2015) for optimization, with a learning rate of 0.0001, and a minibatch size of 64.
We train all models for 500,000 steps (we generally found that the models do not tend to overﬁt in our experiments).
For the image models, p ( x | z ) and q ( z | x ) , we use the DCGAN architecture from (Radford et al., 2015).
For the attribute models, p ( y k | z ) and q ( z | y k ) , we use MLPs.
For the joint inference network, q ( z | x , y ) , we use a CNN combined with an MLP.
We use d = 10 latent dimensions for all models.
We choose the hyperparameters for each method so [Figure] as to maximize correctness on a validation set.
See Appendix A.4 for further details on the model architectures.
Evaluation.
To measure correctness and coverage, we ﬁrst train the observation classiﬁer on the full iid dataset, where it gets to an accuracy of 91.18% for class label, 90.56% for scale, 92.23% for orientation, and 100% for location.
Consequently, it is a reliable way to assess the quality of samples from various generative models (see Appendix A.5 for details).
We then compute correctness and coverage on the iid dataset, and coverage on the comp dataset.
Familiar concrete concepts.
We start by assessing the quality of the models in the simplest setting, which is where the test concepts are fully speciﬁed (i.e., all attributes are known), and the concepts have been seen before in the training set (i.e., we are using the iid split).
Figure 4a shows the correctness scores for the three methods.
(Since the test concepts are fully grounded, coverage is not well deﬁned, since there are no missing attributes.)
We see that TELBO has a correctness of 82.08%, which is close to that of JMVAE (85.15%); both methods signiﬁcantly outperform BiVCCA (67.38%).
To gain more insight, Figure 3 shows some samples from each of these methods for a leaf concept chosen at random.
We see that the images generated by BiVCCA are very blurry, for reasons we discussed in Section 4.
Note that these blurry images are correctly detected by the attribute classiﬁer.
We also see that the JMVAE samples all look good (in this example).
Most of the samples from
TELBO are also good, although there is one error (correctly detected by the attribute classiﬁer).
[Figure]
[Figure]
[Figure]
We report standard deviation across 5 splits of the test set.
[Figure]
[Figure]
Novel abstract concepts.
Next we assess the quality of the models when the test concepts are abstract, i.e., one or more attributes are not speciﬁed.
(Note that the model was never trained on such abstract concepts.)
Figure 4a shows that the correctness scores for JMVAE seems to drop somewhat (from about 85% to about 81.5%), although it remains steady for TELBO and BiVCCA.
We also see that the coverage of TELBO is higher than the other methods, due to the use of the
KL( q φ y ( z | y ) , p θ ( z )) regularizer, as we discussed in Section 4.
Figure 4b illustrates how the methods respond to concepts of different levels of abstraction.
The samples from the TELBO seem to be more diverse, which is consistent with the numbers in Figure 4a.
Compositionally novel concrete concepts.
Finally we assess the quality of the models when the test concepts are fully speciﬁed, but have not been seen before (i.e., we are using the comp split).
Figure 4a shows some quantitative results.
We see that the correctness for TELBO and JMVAE has dropped from about 82% to about 75%, since this task is much harder, and requires “strong generalization”.
However, as before, we see that both TELBO and JMVAE outperform BiVCCA, which has a correctness of about 69%.
See Appendix A.6 qualitative results and more details.
5. C ELEB A
In this section, we report results on the CelebA dataset (Liu et al., 2015).
In particular, we use the version that was used in (Perarnau et al., 2016), which selects 18 visually distinctive attributes; see
Appendix A.7 for details.
Figure 5 shows some sample qualitative results.
On the top left, we show some images which were generated by the three methods given the concept shown in the left column.
TELBO and JMVAE generate realistic and diverse images; BiVCCA just generates the mean image.
On the bottom left, we show what happens when we drop some attributes, thus specifying more abstract concepts.
We see a greater diversity in the samples, but also a slight drop in image quality.
On the top right, we show some examples of visual imagination, where we ask the models to generate images from the concept “bald female”, which does not occur in the training set.
4 (We omit the results from BiVCCA, which are uniformly poor.)
We see that both TELBO and JMVAE can sometimes do a fairly reasonable job (although these are admittedly cherry picked results).
Finally, the bottom right illustrates an interesting bias in the dataset: if we ask the model to generate images where we do not specify the value of the eyeglasses attribute, nearly all of the samples fail to included glasses, since the prior probability of this attribute is rare (about 6%).
[Figure]
C ONCLUSIONS AND FUTURE WORK
We have shown how to create generative models which can “imagine” compositionally novel concrete and abstract visual concepts.
In the future we would like to explore richer forms of description, beyond attribute vectors, such as natural language text, as well as compositional descriptions of scenes, which will require dealing with a variable number of objects.
A A PPENDIX A. A NALYSIS OF JMVAE OBJECTIVE
The JMVAE objective of (Suzuki et al., 2017) has the form
[Equation]
Let us focus on the KL( q φ ( z | x , y ) | q φ y ( z | y )) term.
Let Y be the set of unique labels (attribute vectors) in the training set, X i be the indices of the images associated with label
y
i
[Equation]
and let N i
= |X
i | be the size of that set.
Then we can write
[Equation]
As explained in (Hoffman & Johnson, 2016)
[Equation]
we can rewrite this by treating the index n ∈ { 1 , · · · , N i } as a random variable
[Equation]
with prior q ( n | y i )
= 1 /N
i .
Also
[Equation]
let us deﬁne the likelihood q ( z | n, y i )
=
q φ ( z | x n , y i ) .
Using this notation
[Equation]
we can show that the above average KL becomes
[Equation]
where
[Equation]
is the average of the posteriors for that concept, and q ( n | z , y i ) is the posterior over the indices for all the possible examples from the set X i , given that the latent code is z and the description is y i .
The KL( q avg φ ( z | y i ) | q φ y ( z | y i )) term in Equation (3) tells us that JMVAE encourages the inference network for descriptions, q φ y ( z | y i ) , to be close to the average of the posteriors induced by each of the images x n associated with y i .
Since each q φ ( z | x n , y i ) is close to a delta function (since there is little posterior uncertainty when conditioning on an image), we are essentially requiring that q φ ( z | y i ) cover the embeddings of each of these images.
A. D ETAILS ON THE MNIST-A DATASET
We created the MNIST-A dataset as follows.
Given an image in the original MNIST dataset, we ﬁrst sample a discrete scale label (big or small), an orientation label (clockwise, upright, and anticlockwise), and a location label (top-left, top-right, bottom-left, bottom-right).
Next, we converted this vector of discrete attributes into a vector of continuous transformation parameters, using the procedure described below:
• Scale: For big, we sample scale values from a Gaussian centered at 0.9 with a standard deviation of 0.1, while for small we sample from a Gaussian centered at 0.6 with a standard deviation of 0.1.
In all cases, we reject and draw a sample again if we get values outside the range [0 . 4 , 1 . 0] , to avoid artifacts from upsampling or problems with illegible (small) digits.
• Orientation: For the clockwise label, we sample the amount of rotation to apply for a digit from a Gaussian centered at +45 degrees, with a standard deviation of 10 degrees.
For anti-clockwise, we use a Gaussian at -45 degrees, with a standard deviation of 10 degrees.
For upright, we set the rotation to be 0 degrees always.
• Location: For location, we place Gaussians at the centers of the four quadrants in the image, and then apply an offset of image_size/16 to shift the centers a bit towards the corresponding corners.
We then use a standard deviation of image_size/16 and sample locations for centers of the digits.
We reject and draw the sample again if we ﬁnd that the location for the center would place the extremities of the digit outside of the canvas.
Finally, we generate the image as follows.
We ﬁrst take an empty black canvas of size 64x64 , rotate the original 28x28 MNIST image, and then scale and translate the image and paste it on the canvas.
(We use bicubic interpolation for scaling and resizing the images.)
Finally, we use the method of (Salakhutdinov & Murray, 2008) to binarize the images.
See Figure 6 for example images generated in this way.
We repeat the above process of sampling labels, and applying corresponding transformations, to generate images 10 times for each image in the original MNIST dataset.
Each trial samples labels from a uniform categorical distribution over the sample space for the corresponding attribute.
Thus, we get a new MNIST-A dataset with 700,000 images from the original MNIST dataset of 70, images.
We split the images into a train, val and test set of 85%, 5%, and 10% of the data respectively to create the IID split.
To create the compositional split, we split the 10x2x3x4=240 possible label combinations by the sample train/val/test split, giving us splits of the dataset with non-overlapping label combinations. [Figure]
A. β -VAE vs. LIKELIHOOD SCALING IN JOINT MODELS
[Figure]
β -VAE Higgins et al.
(2017a) is an approach that aims to learn disentangled latent spaces by tweaking the evidence lower bound to prioritize matching the prior (which is assumed to be disentangled).
Our hypothesis/observation is that one needs labels in order to truly disentangle and organize information in latent spaces for high level concepts.
To study this systematically, we set up an experiment where we learn a 2d latent space for MNIST digits with two attributes.
First is parity: odd vs. even, and second is magnitude: value < 5 or > = 5 .
We call this dataset MNIST-2bit.
In Figure 7(a), we show the results of ﬁtting a 2d β -VAE model (Higgins et al., 2017a) to the images in MNIST-2bit, ignoring the attributes.
We perform a hyperparameter sweep over β , and pick the one that gives the best looking latent space (this corresponds to a value of β = 10) .
At each point z in the latent 2d space, we show a single image sampled from p ( x | z ) .
To derive the colors for each point in latent space, we proceed as follows: we embed each training image
x
(with label y
( x ) ) into latent space, by computing ˆ z ( x ) =
E q ( z | x ) [ z ] .
We then associate label y ( x ) with this point in space.
To derive the label for an arbitrary point z , we lookup the closest embedded training image (using (cid:96)distance in z space), and use its corresponding label.
We see that the latent space is useful for autoencoding (since the generated images look good), but it does not capture the relevant semantic properties of parity and magnitude.
In fact, we argue that there is no way of forcing the model to learn a latent space that captures such high level conceptual properties from images alone.
In Figure 7(b), we show the results of ﬁtting a joint VAE model to MNIST-2bit, by optimizing elbo( x, y ) on images and attributes ( i.e. , we do not include the uni-modality elbo( x ) and elbo( y ) terms in this experiment.)
Now the color codes are derived from p ( y | z ) rather than using nearest neighbor retrieval.
We see that the latent space autoencodes well, and also captures the 4 relevant types of concepts.
In particular, the regions are all convex and linearly seperable, which facilitates the learning of a good imagination function q ( z | y ) , interpolation, retrieval, and other latent-space tasks.
A. D ETAILS OF THE NEURAL NETWORK ARCHITECTURES
As explained in the main paper, we ﬁt the joint graphical model p ( x, y, z )
= p ( z ) p ( x | z ) p ( y | z ) with inference networks q ( z | x, y ) , q ( z | x ) , and q ( z | y ) .
Thus, our overall model is made up of three encoders (denoted with q ) and two decoders (denoted with p ).
Across all models we use the exponential linear unit (ELU) which is a leaky non-linearity often used to train VAEs.
We explain the architectures in more detail below.
• Image decoder, p ( x | z ) : Our architecture for the image decoder exactly follows the standard DCGAN architecture from (Radford et al., 2015), where the input to the model is the latent state of the VAE.
• Label decoder, p ( y | z ) : Our label decoder assumes a factorized output space p ( y | z ) = (cid:81)
k ∈A p
( y k | z ) , where y k is each individual attribute.
We parameterize each p ( y k | z ) with a two-layer MLP with 128 hidden units each.
We apply a small amount of (cid:96) 2 regularization to the weight matrices.
• Image and Label encoder, q ( z | x, y ) :
Our architecture (Figure 8) for the image-label encoder ﬁrst separately processes the images and the labels, and then concatenates them downstream in the network and then passes the concatenated features through a multi-layered perceptron.
More speciﬁcally, we have convolutional layers which process image into 32 , 64 , 128 ,feature maps with strides 1 , 2 , 2 , 2 in the corresponding layers.
We use batch normalization in the convolutional layers before applying the ELU non-linearity.
On the label encoder side, we ﬁrst encode the each attribute label into a 32d continuous vector and then pass each individual attribute vector through a 2-layered MLP with 512 hidden dimensions each.
For example, for MNIST-A we have 4 attributes, which gives us 4 vectors of 512d.
We then concatenate these vectors and pass it through a two layer MLP.
Finally we concatenate this label feature with the image feature after the convolutional layers (after ﬂattening the conv-features) and then pass the result through a 2 layer MLP to predict the mean ( µ ) and standard deviation ( σ ) for the latent space gaussian.
Following standard practice, we predict log σ for the standard deviation in order to get values which are positive.
• Image encoder, q ( z | x ) : The image encoder (Figure 9a) uses the same architecture to process the image as the image feature extractor in q ( z | x, y ) network described above.
After the conv-features, we pass the result through a 3-layer MLP to get the latent state mean and standard deviation vectors following the procedure described above.
• Label encoder, q ( z | y ) :
The label encoder (Figure 9b) part of the architecture uses the same design choices to process the labels as the label encoder part in the q ( z | x, y ) network.
After obtaining the concatenated label feature vectors, we pass the result through a 4-layered MLP
Class [10]
Scale [2]
Orientation [3] Location [4] concat (2048)
64x64x
Image [64x64x1] flatten (1024) concat (1536)
µ            32x32x 16x16x
[Figure]
64x64x flatten (1024) 32x32x 16x16x 8x8x Image [64x64x1] µ  [Figure]
for the
[Figure] z | x ) network [Figure]
concat (2048)           µ [Figure]
with 512 hidden dimensions each and then ﬁnally obtain the mean ( µ ) and log σ values for each dimension in the latent state of the VAE.
CelebA model architecture Our design choices for CelebA closely mirror the models we built for MNIST-A. One primary difference is that we use a latent dimensionality of 18 in our CelebA experiments which matches the number of attributes we model.
Meanwhile, the architectures of the image encoder, image decoder ( i.e. DCGAN), are exactly identical to what is described above for
MNIST-A execept that encoders take as input a 3-channel RGB image, while decoders produce a 3-channel output.
We replace the Bernoulli likelihood with Quantized Normal likelihood (which is basically gaussian likelihood with uniform noise).
In terms of the label encoder q ( z | y ) , we follow Figure 9b quite closely, except that we get as inputcategorical (embedded) class labels as input, and we process the labels through a single hidden layer before concatenation and two hidden layers post concatenation (as opposed to two and four used in
Figure 9b).
Finally, the joint encoder q ( z | x, y ) , is again based heavily on Figure 8 where we feed as inputlabels as opposed to 4, process them through a single layer mlp of 512d, concatenate them, and then pass the result through a two hidden layer mlp of 512 d.
At this point we concatenate the result with the image feature through the image feature head in Figure 8.
Finally, we process the feature through another 512d single hidden layer mlp to produce the µ, σ values.
A. O UTPUTS OF OBSERVATION CLASSIFIER ON GENERATED IMAGES
Figure 10 shows some images sampled from our TELBO model trained on
MNIST-A.
It also shows the attributes that are predicted by the attribute classiﬁer.
We see that the classiﬁer often produces reasonable results that we as humans would also agree with.
Thus, it acts as a reasonable proxy for humans classifying the labels for the generated images
. A. C OMPOSITIONAL GENRALIZATION ON MNIST-A: Q UALITATIVE R ESULTS AND D ETAILS
We next show some examples of compositional generalization on MNIST-A on a validation set of queries.
For the compositinal experiments we reused the parameters of the best models on the iid splits for all the models, and trained the models for ∼ 160 K iterations.
All other design choices were the same.
Figure 11 shows some qualitative results.
A. D ETAILS ON C ELEB A
CelebA consists of 202,599 face colored images and 40 attribute binary vectors.
We use the version of this dataset that was used in (Perarnau et al., 2016); this uses a subset of 18 visually distinctive attributes, and preprocesses each image so they are aligned, cropped, and scaled down to 64 x 64.
We use the ofﬁcial train and test partitions, 182K for training and 20K for testing.
Note that this is an iid split, so the attribute vectors in the test set all occur in the training set, even though the images and people are unique.
In total, the original dataset with 40 attributes speciﬁed a set of 96486 unique visual concepts, while our dataset of 18 attributes spans 3690 different visual concepts.
In Section 5.2, we claim that our generations of “Bald” and “Female” images are from a compositionally novel concept.
Our claim comes with a minor caveat/clariﬁcation: the concept bald = 1 and male = 0 does occur in 9 training examples, but they are all incorrect labelings, as shown in Figure 12!
Further, we see that the images generated from our model (shown in Figure 5) are qualitatively very different from any of the images here, showing that the model has not memorized these examples.
[Figure] [Figure]