G ENERATIVE M ODELS OF V ISUALLY G ROUNDED I MAGINATION
Jonathan Huang
A BSTRACT
It is easy for people to imagine what a man with pink hair looks like, even if they have never seen such a person before.
We call the ability to create images of novel semantic concepts visually grounded imagination .
In this paper, we show how we can modify variational auto-encoders to perform this task.
Our method uses a novel training objective, and a novel product-of-experts inference network, which can handle partially speciﬁed (abstract) concepts in a principled and efﬁcient way.
We also propose a set of easy-to-compute evaluation metrics that capture our intuitive notions of what it means to have good visual imagination, namely correctness, coverage, and compositionality (the 3 C’s ).
Finally, we perform a detailed comparison of our method with two existing joint image-attribute VAE methods (the JMVAE method of (Suzuki et al., 2017) and the BiVCCA method of (Wang et al., 2016b)) by applying them to two datasets: the MNIST-with-attributes dataset (which we introduce here), and the CelebA dataset (Liu et al., 2015).
I NTRODUCTION
Consider the following two-party communication game: a speaker thinks of a visual concept C , such as “men with black hair”, and then generates a description y of this concept, which she sends to a listener; the listener interprets the description y , by creating an internal representation z , which captures its “meaning”.
We can think of z as representing a set of “mental images” which depict the concept C .
To test whether the listener has correctly “understood” the concept, we ask him to draw a set of real images S
= { x s : s = 1 : S } , which depict the concept C .
He then sends these back to the speaker, who checks to see if the images correctly match the concept C .
We call this process visually grounded imagination .
In this paper, we represent concept descriptions in terms of a ﬁxed length vector of discrete attributes
A .
This allows us to specify an exponentially large set of concepts using a compact, combinatorial representation.
In particular, by specifying different subsets of attributes, we can generate concepts at different levels of granularity or abstraction.
We can arrange these concepts into a compositional abstraction hierarchy , as shown in Figure 1.
This is a directed acyclic graph (DAG) in which nodes represent concepts, and an edge from a node to its parent is added whenever we drop one of the attributes from the child’s concept deﬁnition.
Note that we dont make any assumptions about the order in which the attributes are dropped (that is, dropping the attribute “smiling” is just as valid as dropping “female” in Figure 1).
Thus, the tree shown in the ﬁgure is just a subset extracted from the full DAG of concepts, shown for illustration purposes.
We can describe a concept by creating the attribute vector y O , in which we only specify the value of the attributes in the subset O ⊆ A ; the remaining attributes are unspeciﬁed, and are assumed to take all possible legal values.
For example, consider the following concepts, in order of increasing abstraction:
C msb =
(male , smiling , blackhair) , C ∗ sb =
( ∗ , smiling , blackhair) , and C ∗∗ b = ( ∗ , ∗ , blackhair) , where the attributes are gender, smiling or not, and hair color, and ∗ represents “don’t care”.
A good model should be able to generate images from different levels of the abstraction hierarchy, as shown in Figure 1.
(This is in contrast to most prior work on conditional generative models of images, which assume that all attributes are fully speciﬁed, which corresponds to sampling only from leaf nodes in the hierarchy.) [Figure]
In Section 2, we show how we can extend the variational autoencoder (VAE) framework of (Kingma & Welling, 2014) to create models which can perform this task.
The ﬁrst extension is to modify the model to the “multi-modal” setting where we have both an image, x , and an attribute vector, y .
More precisely, we assume a joint generative model of the form p ( x , y , z )
=
p ( z ) p ( x | z ) p ( y | z ) , where p ( z ) is the prior over the latent variable z , p ( x | z ) is our image decoder, and p ( y | z ) is our description decoder.
We additionally assume that the description decoder factorizes over the speciﬁed attributes in the description, so p ( y O | z ) =
(cid:81)
k ∈O p
( y k | z ) .
We further extend the VAE by devising a novel objective function, which we call the TELBO , for training the model from paired data, D = { ( x n , y n ) } .
However, at test time, we will allow unpaired data (either just a description or just an image).
Hence we ﬁt three inference networks: q ( z | x , y ) , q ( z | x ) and q ( z | y ) .
This way we can embed an image or a description into the same shared latent space (using q ( z | x ) and q ( z | y ) , respectively); this lets us “translate” images into descriptions or vice versa, by computing p ( y | x ) = (cid:82)
d z p ( y | z ) q ( z | x ) and p ( x | y ) =
(cid:82) d z p ( x | z ) q ( z | y ) .
To handle abstract concepts (i.e., partially observed attribute vectors), we use a method based on the product of experts (POE) (Hinton, 2002).
In particular, our inference network for attributes has the form q ( z | y O )
∝ p ( z ) (cid:81) k ∈O q
( z | y k ) .
If no attributes are speciﬁed, the posterior is equal to the prior.
As we condition on more attributes, the posterior becomes narrower, which corresponds to specifying a more precise concept.
This enables us to generate a more diverse set of images to represent abstract concepts, and a less diverse set of images to represent concrete concepts, as we show below.
Section 3 discusses how to evaluate the performance of our method in an objective way.
Speciﬁcally, we ﬁrst “ground” the description by generating a set of images, S ( y O ) =
{ x s ∼ p ( x | y O ) : s = 1 :
S } .
We then check that all the sampled images in S ( y O ) are consistent with the speciﬁed attributes y O
(we call this correctness ).
We also check that the set of images “spans” the extension of the concept, by exhibiting suitable diversity (c.f.
(Young et al., 2014)).
Concretely, we check that the attributes that were not speciﬁed (e.g., gender in C ∗ sb above) vary across the different images; we call this coverage .
Finally, we want the set of images to have high correctness and coverage even if the concept y O has a combination of attribute values that have not been seen in training.
For example, if we train on
C msb =
(male , smiling , blackhair) , and C fnb =
(female , notsmiling , blackhair) , we should be able to test on C mnb
= (male , notsmiling , blackhair) , and C fsb =
(female , smiling , blackhair) .
We will call this property compositionality .
Being able to generate plausible images in response to truly compositionally novel queries is the essence of imagination.
Together, we call these criteria the 3 C’s of visual imagination .
Section 5 reports experimental results on two different datasets.
The ﬁrst dataset is a modiﬁed version of MNIST, which we call MNIST-with-attributes (or MNIST-A), in which we “render” modiﬁed versions of a single MNIST digit on a 64x64 canvas, varying its location, orientation and size.
The second dataset is CelebA (Liu et al., 2015), which consists of over 200k face images, annotated with 40 binary attributes.
We show that our method outperforms previous methods on these datasets.
The contributions of this paper are threefold.
First, we present a novel extension to VAEs in the multimodal setting, introducing a principled new training objective (the TELBO), and deriving an interpretation of a previously proposed objective (JMVAE) (Wang et al., 2016a) as a valid alternative in Appendix A.1.
Second, we present a novel way to handle missing data in inference networks based on a product of experts.
Third, we present novel criteria (the 3 C’s) for evaluating conditional generative models of images, that extends prior work by considering the notion of visual abstraction and imagination.
M ETHODS
We start by describing standard VAEs, to introduce notation.
We then discuss our extensions to handle the multimodal and the missing input settings.
Standard VAEs.
A variational autoencoder (Kingma & Welling, 2014) is a latent variable model of the form p θ ( x , z )
= p θ ( z ) p θ ( x | z ) , where p θ ( z ) is the prior (we assume it is Gaussian, p θ ( z ) =
N ( z | 0 , I ) , although this assumption can be relaxed), and p θ ( x | z ) is the likelihood (sometimes called the decoder), usually represented by a neural network.
To perform approximate posterior inference, we ﬁt an inference network (sometimes called the encoder) of the form q φ ( z | x ) , so as to maximize
L ( θ , φ ) =
E ˆ p ( x ) [elbo( x , θ , φ )] , where ˆ p ( x ) = 1 N (cid:80)
Nn =1 δ x n ( x ) is the empirical distribution, and
ELBO is the evidence lower bound:
[Equation]
Here KL( p, q ) is the Kullback Leibler divergence between distributions p and q .
By default, β = λ = 1 , in which case we will just write elbo( x , θ , φ ) . However, by using β > 1 we can encourage the posterior to be closer to the factorial prior p ( z ) = N ( z | 0 , I ) , which encouarges the latent factors to be “disentangled”, as proved in (Achille & Soatto, 2017); this is known as the β -VAE trick (Higgins et al., 2017b). And allowing λ > 1 will be useful later, when we have multiple modalities.
Joint VAEs and the TELBO.
We extend the VAE to model images and attributes by deﬁning the joint distribution p θ ( x , y , z ) =
p θ ( z ) p θ ( x | z ) p θ ( y | z ) , where p θ ( x | z ) is the image decoder (we use the DCGAN architecture from (Radford et al., 2015)), and p θ ( y | z ) is an MLP for the attribute vector.
The corresponding training objective which we want to maximize becomes L ( θ , φ ) =
E ˆ p ( x , y )
[elbo( x , y , θ , φ )] , where ˆ p ( x , y )
= 1 N (cid:80) Nn
=1 δ x n ( x ) δ y n ( y n ) is the empirical distribution derived from paired data, and the joint ELBO is given by
[Equation]
We call this the JVAE (joint VAE) model. We usually set β = 1 , but set λ y /λ x > 1 to to scale up the likelihood from the low dimensional attribute vector, p θ ( y | z ) , to match the likelihood from the high dimensional image, p θ ( x | z ) .
In addition to ﬁtting the joint model, we need to train unpaired inference networks q φ x ( z | x ) and q φ y ( z | y ) , so we can embed images and attributes into the same shared latent space.
A natural objective to ﬁt, say, q φ x ( z | x ) is to maximize the following:
[Equation]
where the last term is constant wrt φ x , and hence can be dropped.
We can use a similar method to ﬁt q φ y ( z | y ) .
Combining these gives the following triple ELBO ( TELBO ) objective:
[Equation]
where λ and γ scale the log likelihood terms log p ( y | z ) ; we set these parameters using a validation set. Since we are training the generative model only on aligned data, and simply retroﬁtting inference [Figure]
networks, we freeze the p θ x ( x | z ) and p θ y ( y | z ) terms when training the last two ELBO terms above, and just optimize q φ x ( z | x ) and q φ y ( z | y ) terms. Alternatively, we can ﬁrst ﬁt the joint model, and then ﬁt the unimodal inference networks.
In Section 4, we compare this to other methods for training joint VAEs that have been proposed in the literature.
Handling missing attributes.
In order to handle missing attributes at test time, we use a product of experts model, where each attribute instantiates an expert.
Concretely, our q ( z | y ) inference network has the following form: q ( z | y O )
∝ p ( z ) (cid:81) k ∈O q
( z | y k ) , where q ( z | y k ) =
N ( z | µ k ( y k ) , C k ( y k )) is the k th Gaussian “expert”, and p ( z ) =
N ( z | µ 0 = 0 , C 0 = I ) is the prior.
Unlike the product of experts model in (Hinton, 2002), our model multiplies Gaussians, not Bernoullis, so the product has a closed form solution namely q ( z | y O ) =
N ( z | µ , C ) , where C − 1 = (cid:80) k C − 1 k and µ =
C ( (cid:80) k
C − 1 k µ k ) , and the sum is over all the observed attributes.
This equation has a simple intuitive interpretation: If we do not observe any attributes, the posterior reduces to the prior.
As we observe more attributes, the posterior becomes narrower, since the (positive deﬁnite) precision matrices add up, reﬂecting the increased speciﬁcity of the concept being speciﬁed, as illustrated in Figure(middle).
We always include the prior term, p ( z ) , in the product, since without it, the posterior q φ y ( z | y O ) may not be well-conditioned when we are missing attributes, as illustrated in Figure(right).
For more implementation-level details on the model architectures, see Appendix A.4.
E VALUATION METRICS : T HE 3C’ S OF V ISUAL I MAGINATION
To evaluate the quality of a set of generated images, S ( y O )
= { x s ∼ p ( x | y O ) : s = 1 : S } , we apply a multi-label classiﬁer to each image, to convert it to a predicted attribute vector, ˆ y ( x ) .
This attribute classiﬁer is trained on a large dataset of images and attributes, and is held constant across all methods that are being evaluated.
It plays the role of a human observer.
This is similar in spirit to generative adversarial networks (Goodfellow et al., 2014), that declare a generated image to be good enough if a binary classiﬁer cannot distinguish it from a real image.
(Both approaches avoid the problems mentioned in (Theis et al., 2016) related to evaluating generative image models in terms of their likelihood.)
However, the attribute classiﬁer checks not only that the images look realistic, but also that they have the desired attributes.
To quantify this, we deﬁne the correctness as the fraction of attributes for each generated image that match those speciﬁed in the concept’s description: correctness( S , y O )
=|S| (cid:80) x ∈S 1 |O| (cid:80) k
∈O I (ˆ y ( x ) k
= y k ) .
However, we also want to measure the diversity of values for the unspeciﬁed or missing attributes, M =
A \ O .
We do this by comparing q k , the empirical distribution over values for attribute k induced by the generated set S , to p k , the true distribution for this attribute induced by the training set.
We measure the difference between these distributions using the Jensen-Shannon divergence, since it is symmetric and satisﬁes 0 ≤ JS( p, q ) ≤ 1 .
We then deﬁne the coverage as follows: coverage( S, y O )
= 1 |M| (cid:80) k ∈M (1 − JS( p k , q k )) .
If desired, we can combine correctness and coverage into a single number, by computing the JS divergence between p k and q k for all attributes, where, for observed attributes, p k is a delta function and q k is the empirical distribution (we call this JS-overall ). This gives us a convenient way to pick hyperparameters.
However, for analysis, we ﬁnd it helpful to report correctness and coverage separately.
Note that our metric is different from the inception score proposed in (Salimans et al., 2016).
That is deﬁned as follows:
inception =
exp (cid:0) E ˆ p ( x ) [KL( p ( y | x ) , p ( y ))] (cid:1) , where y is a class label.
Expanding the term inside the exponential, we get
[Equation]
A high inception score means that the distribution p ( y | x ) has low entropy, so the generated images match some class, but that the marginal p ( y ) has high entropy, so the images are diverse.
However, the inception score was created to evaluate unconditional generative models of images, so it does not check if the generated images are consistent with the concept y O , and the degree of diversity does not vary in response to the level of abstraction of the concept.
Finally, we can assess how well the model understands compositionality , by checking correctness of its generated images in response to test concepts y O that differ in at least one attribute from the training concepts.
We call this a compositional split of the data.
This is much harder than a standard iid split, since we are asking the model to predict the effects of novel combinations of attributes, which it has not seen before (and which might actually be impossible).
Note that abstraction is different from compositionality – in abstraction we are asking the model to predict the effects of dropping certain attributes instead of predicting novel combinations of attributes.
R ELATED W ORK
In this section, we brieﬂy mention some of the most closely related prior work.
Conditional models.
Many conditional generative image models of the form p ( y | x ) have been proposed recently, where y can be a class label (e.g., (Radford et al., 2015)), a vector of attributes (e.g., (Yan et al., 2016)), a sentence (e.g., (Reed et al., 2016)), another image (e.g., (Isola et al., 2017)), etc.
Such models are usually based on VAEs or GANs.
However, we are more interested in learning a shared latent space from either descriptions y or images x , which means we need to use a joint, symmetric, model.
Joint models.
Several papers use the same joint VAE model as us, but they differ in how it is trained.
In particular, the BiVCCA objective of (Wang et al., 2016b) has the form L ( θ , φ ) =
E ˆ p ( x , y )
[ J ( x , y , θ , φ )] , where
[Equation]
This method results in the model generating the mean image corresponding to each concept, due to the
E q φ y ( z | y ) log p θ ( x , y | z ) term, which requires that z ’s sampled from q φ y ( z | y n ) be good at generating all the different x n ’s which co-occur with y n .
We show this empirically in Section 5.
This problem can be partially compensated for by increasing µ , but that reduces the KL( q φ ( z | y ) , p θ ( z )) penalty, which is required to ensure q φ y ( z | y ) is a broad distribution with good coverage of the concept.
The JMVAE objective of (Suzuki et al., 2017) has the form L ( θ , φ ) =
E ˆ p ( x , y ) [ J ( x , y , θ , φ )] , where J ( x , y , θ , φ ) = elbo 1 ,λ, 1 ( x , y , θ , φ ) − α (cid:104) KL( q φ ( z | x , y ) , q φ y ( z | y ))
+ KL( q φ ( z | x , y ) , q φ x ( z | x )) (cid:105) At ﬁrst glance, forcing q φ ( z | y ) to be close to q φ ( z | x , y ) seems undesirable, since the latter will typically be close to a delta function, since there is little posterior uncertainty in z once we see the image x .
However, in Appendix A.1, we use results from (Hoffman & Johnson, 2016) to show that E ˆ p ( x , y ) (cid:104) KL( q φ ( z | x , y ) , q φ y ( z | y ))
(cid:105) can be written in terms of KL( q avg φ ( z | y ) , q φ y ( z | y )) , where q avg φ ( z | y ) =
E ˆ p ( x | y )
[ q φ ( z | x , y )] is the aggregated posterior over z induced by all images x which are associated with description y .
This ensures that q φ y ( z | y ) will cover the embeddings of all the images associated with concept y .
However, since there is no KL( q φ y ( z | y ) , p θ ( z )) term, the diversity of the samples is slightly reduced for novel concepts compared to TELBO, as we show empirically in Section 5.
On the ﬂip side, the beneﬁt of using the aggregated posterior to ﬁt the q ( z | y ) inference network is that one can expect sharper images, as this ensures we will sample z ∼ q ( z | y ) which have been seen by the image decoder p θ ( x | z ) during joint training.
If the aggregated posterior does not exactly match the prior (which is known to happen in VAE-type models, see Hoffman &
Johnson (2016)) then regularizing with respect to the prior (as TELBO does) can generate samples in parts of space not seen by the image decoder, which can potentially lead to less “correct” samples.
Again, our empirical ﬁndings in Section 5 conﬁrm this tradeoff between correctness and coverage implicit in choices of TELBO vs. JMVAE.
The SCAN method of (Higgins et al., 2017c)
ﬁrst ﬁts a standard β -VAE model (Higgins et al., 2017b) on unlabeled images (or rather, features derived from images using a pre-trained denoising autoencoder) by maximizing L ( θ x , φ x ) = E ˆ p ( x ) [elbo 1 ,β x ( x , θ x , φ x )] . They then ﬁt a second VAE by maximizing L ( θ y , φ y ) = E ˆ p ( x , y ) (cid:2) J ( x , y , θ y , φ y , φ x ) (cid:3) , where
J ( x , y , θ y , φ y , φ x ) = elbo 1 ,β y ( y , θ y , [Equation] This is very similar to JMVAE, since q φ x ( z | x ) ≈ q φ ( z | x , y ) , when ( x , y ) is a matching pair of images and labels. An important difference, however, is that SCAN treats the attribute vectors y as atomic symbols; this has the advantage that there is no need to handle missing inputs, but the disadvantage that they cannot infer the meaning of unseen attribute combinations at test time, unless they are “taught” them by having them paired with images.
Also, they rely on β x > 1 as a way to get compositionality, assuming that a disentangled latent space will sufﬁce. However, in Appendix A.3, we show that unsupervised learning of the latent space given images alone can result in poor results when some of the attributes in the compositional concept hierarchy are non-visual, such as parity of an MNIST digit.
Our approach always takes the labels into consideration when learning the latent space, permitting well-organized latent spaces even in the presence of non-visual concepts (c.f.
the difference between PCA and LDA).
Handling missing inputs.
Conditional generative models of images, of the form p ( x | y ) , have problems with missing input attributes, as do inference networks q ( z | y ) for VAEs.
(Hoffman, 2017) uses MCMC to ﬁt a latent Gaussian model, which can in principle handle missing data; however, he initializes the Markov chain with the posterior mode computed by an inference network, which cannot easily handle missing inputs.
One approach we can use, if we have a joint model, is to estimate or impute the missing values, as follows: ˆ y = arg max y M p ( y M | y O ) , where p ( y M , y O ) models dependencies between attributes.
We can then sample images using p ( x | ˆ y ) .
This approach was used in (Yan et al., 2016) to handle the case where some of the pixels being passed into an inference network were not observed.
However, conditioning on an imputed value will give different results from not conditioning on the missing inputs; only the latter will increase the posterior uncertainty in order to correctly represent less precise concepts with broader support.
Gaussian embeddings.
There are many papers that embed images and text into points in a vector space.
However, we want to represent concepts of different levels of abstraction, and therefore want to map images and text to regions of latent space.
There are some prior works that use Gaussian embeddings for words (Vilnis & McCallum, 2015; Athiwaratkun & Wilson, 2017), sometimes in conjunction with images (Mukherjee & Hospedales, 2016;
Ren et al., 2016).
Our method differs from these approaches in several ways.
First, we maximize the likelihood of ( x , y ) pairs, whereas the above methods learn a Gaussian embedding using a contrastive loss.
Second, our PoE formulation ensures that the covariance of the posterior q ( z | y O ) is adaptive to the data that we condition on.
In particular, it becomes narrower as we observe more attributes (because the precision matrices sum up), which is a property not shared by other embedding methods.
Abstraction and compositionality.
Young et al.
(2014) represent the extension of a concept (described by a noun phrase) in terms of a set of images whose captions match the phrase.
By contrast, we use a parametric probability distribution in a latent space that can generate new images.
Vendrov et al.
(2016) use order embeddings, where they explicitly learn subsumption-like relationships by learning a space that respects a partial order.
In contrast, we reason about generality of concepts via the uncertainty induced by their latent representation.
There has been some work on compositionality in the language/vision literature (see e.g., Atzmon et al.
; Johnson et al.
; Agrawal et al.
(2017)), but
none of these papers use generative models, which is arguably a much more stringent test of whether a model has truly “understood” the meaning of the components which are being composed.
E XPERIMENTAL RESULTS
In this section, we ﬁt the JVAE model to two different datasets (MNIST-A and CelebA), using the TELBO objective, as well as BiVCCA and JMVAE.
We measure the quality of the resulting model using the 3 C’s, and show that our method of handling missing data behaves in a qualitatively reasonable way.
5. MNIST-A
Dataset.
In this section, we report results on the MNIST-A dataset.
This is created by modifying the original MNIST dataset as follows.
We ﬁrst create a compositional concept hierarchy usingdiscrete attributes, corresponding to class label (10 values), location (4 values), orientation (3 values), and size (2 values).
Thus there are 10x2x3x4=240 unique concepts in total.
We then sample ∼example images of each concept, and create both an iid and compositional split of the data.
See
Appendix A.2 for details.
Models and algorithms.
We train the JVAE model on this dataset using TELBO, BiVCCA and
JMVAE objectives.
We use Adam (Kingma & Ba, 2015) for optimization, with a learning rate of 0.0001, and a minibatch size of 64.
We train all models for 250,000 steps (we generally found that the models do not tend to overﬁt in our experiments). Our models typically take around a day to train on
NVIDIA Titan X GPUs. For the image models, p ( x | z ) and q ( z | x ) , we use the DCGAN architecture from (Radford et al., 2015).
Our generated images are of size 64 × 64, as in Radford et al.
For the attribute models, p ( y k | z ) and q ( z | y k ) , we use MLPs.
For the joint inference network, q ( z | x , y ) , we use a CNN combined with an MLP.
We use d = 10 latent dimensions for all models.
We choose the hyperparameters for each method so as to maximize JS-overall, which is an overall measure of correctness and coverage (see Section 3) on a validation set of attribute queries.
See Appendix A. for further details on the model architectures.
Evaluation.
To measure correctness and coverage, we ﬁrst train the observation classiﬁer on the full iid dataset, where it gets to an accuracy of 91.18% for class label, 90.56% for scale, 92.23% for orientation, and 100% for location.
Consequently, it is a reliable way to assess the quality of samples from various generative models (see Appendix A.5 for details).
We then compute correctness and coverage on the iid dataset, and coverage on the comp dataset.
Familiar concrete concepts.
We start by assessing the quality of the models in the simplest setting, which is where the test concepts are fully speciﬁed (i.e., all attributes are known), and the concepts have been seen before in the training set (i.e., we are using the iid split).
Figure 4a shows the correctness scores for the three methods.
(Since the test concepts are fully grounded, coverage is not well deﬁned, since there are no missing attributes.)
We see that TELBO has a correctness of 82.08%, which is close to that of JMVAE (85.15%); both methods signiﬁcantly outperform BiVCCA (67.38%).
To gain more insight, Figure 3 shows some samples from each of these methods for a leaf concept chosen at random.
We see that the images generated by BiVCCA are very blurry, for reasons we discussed in Section 4.
Note that these blurry images are correctly detected by the attribute classiﬁer.
We also see that the JMVAE samples all look good (in this example).
Most of the samples from
TELBO are also good, although there is one error (correctly detected by the attribute classiﬁer).
Novel abstract concepts.
Next we assess the quality of the models when the test concepts are abstract, i.e., one or more attributes are not speciﬁed.
(Note that the model was never trained on [Figure] [Figure] such abstract concepts.)
Figure 4a shows that the correctness scores for JMVAE seems to drop somewhat (from about 85% to about 81.5%), although it remains steady for TELBO and BiVCCA.
We also see that the coverage of TELBO is higher than the other methods, due to the use of the
KL( q φ y ( z | y ) , p θ ( z )) regularizer, as we discussed in Section 4.
Figure 4b illustrates how the methods respond to concepts of different levels of abstraction.
The samples from the TELBO seem to be more diverse, which is consistent with the numbers in Figure 4a.
Compositionally novel concrete concepts.
Finally we assess the quality of the models when the test concepts are fully speciﬁed, but have not been seen before (i.e., we are using the comp split).
Figure 4a shows some quantitative results.
We see that the correctness for TELBO and JMVAE has dropped from about 82% to about 75%, since this task is much harder, and requires “strong generalization”.
However, as before, we see that both TELBO and JMVAE outperform BiVCCA, which has a correctness of about 69%.
See Appendix A.7 qualitative results and more details.
5. C ELEB A
In this section, we report results on the CelebA dataset (Liu et al., 2015).
In particular, we use the version that was used in (Perarnau et al., 2016), which selects 18 visually distinctive attributes, and generate images of size 64 × 64; see Appendix A.8 for more details on the CelebA dataset and Appendix A.4 for details of the model architectures.
Figure 5 shows some sample qualitative results.
On the top left, we show some images which were generated by the three methods given the concept shown in the left column.
TELBO and JMVAE generate realistic and diverse images. That is, the generated images are generally of males, with mouth slightly open and smiling attributes present in the images. On the other hand, BiVCCA just generates the mean image.
On the bottom left, we show what happens when we drop some attributes, thus specifying more abstract concepts.
We see that when we drop the gender, we get a mixture of both male and female images for both TELBO and JMVAE. Going further, when we drop the “smiling” attribute, we see that the samples now comprise of people who are smiling as well as not smiling, and we see a mixture of genders in the samples. Further, while we see a greater diversity in the samples, we also notice a slight drop in image quality (presumably because none of the approaches has seen supervision with just ‘abstract’ concepts).
See Appendix A.9 for more qualitative examples on CelebA. On the top right, we show some examples of visual imagination, where we ask the models to generate images from the concept “bald female”, which does not occur in the training set.
4 (We omit the results from BiVCCA, which are uniformly poor.)
We see that both TELBO and JMVAE can sometimes do a fairly reasonable job (although these are admittedly cherry picked results).
Finally, the bottom right illustrates an interesting bias in the dataset: if we ask the model to generate images where we do not specify the value of the eyeglasses attribute, nearly all of the samples fail to included glasses, since the prior probability of this attribute is rare (about 6%).
[Figure]
Abstract generation: gender=*
Abstract generation: gender=*,smiling=*
TELBO
JMVAE
BiVCCA
TELBO
JMVAE
Method
[Figure]
C ONCLUSIONS AND FUTURE WORK
We have shown how to create generative models which can “imagine” compositionally novel concrete and abstract visual concepts.
In the future we would like to explore richer forms of description, beyond attribute vectors, such as natural language text, as well as compositional descriptions of scenes, which will require dealing with a variable number of objects. A CKNOWLEDGMENTS
We would like to thank Hernan Moraldo for his help in writing the JVAE library, and Alex Alemi for valuable insights on TELBO and JMVAE.
Finally we would like to thank Devi Parikh for advice on the CelebA experiments, and Stefan Lee and Yash Goyal for feedback on an initial version of this draft.
A A PPENDIX A. A NALYSIS OF JMVAE OBJECTIVE
The JMVAE objective of (Suzuki et al., 2017) has the form
[Equation]
Let us focus on the KL( q φ ( z | x , y ) | q φ y ( z | y )) term.
Let Y be the set of unique labels (attribute vectors) in the training set, X i be the indices of the images associated with label
y
i
[Equation]
and let N i
= |X
i | be the size of that set.
Then we can write
[Equation]
As explained in (Hoffman & Johnson, 2016)
[Equation]
we can rewrite this by treating the index n ∈ { 1 , · · · , N i } as a random variable
[Equation]
with prior q ( n | y i )
= 1 /N
i .
Also
[Equation]
let us deﬁne the likelihood q ( z | n, y i )
=
q φ ( z | x n , y i ) .
Using this notation
[Equation]
we can show that the above average KL becomes
[Equation]
where
[Equation]
is the average of the posteriors for that concept, and q ( n | z , y i ) is the posterior over the indices for all the possible examples from the set X i , given that the latent code is z and the description is y i .
The KL( q avg φ ( z | y i ) | q φ y ( z | y i )) term in Equation (3) tells us that JMVAE encourages the inference network for descriptions, q φ y ( z | y i ) , to be close to the average of the posteriors induced by each of the images x n associated with y i .
Since each q φ ( z | x n , y i ) is close to a delta function (since there is little posterior uncertainty when conditioning on an image), we are essentially requiring that q φ ( z | y i ) cover the embeddings of each of these images.
A. D ETAILS ON THE MNIST-A DATASET
We created the MNIST-A dataset as follows.
Given an image in the original MNIST dataset, we ﬁrst sample a discrete scale label (big or small), an orientation label (clockwise, upright, and anticlockwise), and a location label (top-left, top-right, bottom-left, bottom-right).
Next, we converted this vector of discrete attributes into a vector of continuous transformation parameters, using the procedure described below:
• Scale: For big, we sample scale values from a Gaussian centered at 0.9 with a standard deviation of 0.1, while for small we sample from a Gaussian centered at 0.6 with a standard deviation of 0.1.
In all cases, we reject and draw a sample again if we get values outside the range [0 . 4 , 1 . 0] , to avoid artifacts from upsampling or problems with illegible (small) digits.
• Orientation: For the clockwise label, we sample the amount of rotation to apply for a digit from a Gaussian centered at +45 degrees, with a standard deviation of 10 degrees.
For anti-clockwise, we use a Gaussian at -45 degrees, with a standard deviation of 10 degrees.
For upright, we set the rotation to be 0 degrees always.
• Location: For location, we place Gaussians at the centers of the four quadrants in the image, and then apply an offset of image_size/16 to shift the centers a bit towards the corresponding corners.
We then use a standard deviation of image_size/16 and sample locations for centers of the digits.
We reject and draw the sample again if we ﬁnd that the location for the center would place the extremities of the digit outside of the canvas.
Finally, we generate the image as follows.
We ﬁrst take an empty black canvas of size 64x64 , rotate the original 28x28 MNIST image, and then scale and translate the image and paste it on the canvas.
(We use bicubic interpolation for scaling and resizing the images.)
Finally, we use the method of (Salakhutdinov & Murray, 2008) to binarize the images.
See Figure 6 for example images generated in this way.
We repeat the above process of sampling labels, and applying corresponding transformations, to generate images 10 times for each image in the original MNIST dataset.
Each trial samples labels from a uniform categorical distribution over the sample space for the corresponding attribute.
Thus, we get a new MNIST-A dataset with 700,000 images from the original MNIST dataset of 70, images.
We split the images into a train, val and test set of 85%, 5%, and 10% of the data respectively to create the IID split.
To create the compositional split, we split the 10x2x3x4=240 possible label combinations by the sample train/val/test split, giving us splits of the dataset with non-overlapping label combinations.
[Figure]
[Figure]
[Table]
β -VAE Higgins et al.
(2017a) is an approach that aims to learn disentangled latent spaces. It does this by modifying the ELBO objective, so that it scales the KL( q ( z | x ) , p ( z )) term by a factor β > 1 . This gives rise to disentangled spaces since the prior p ( z ) =
N ( z | 0 , I ) is factorized (see (Achille & Soatto, 2017) for details). However, to learn latent spaces that correspond to high level concepts, this is not sufﬁcient: we need to use labeled data as well.
To illustrate this, we set up an experiment where we learn a 2d latent space for standard MNIST digit images, but where we replace the label with two binary attributes: parity (odd vs. even) and magnitude (value < 5 or > = 5 ).
We call this dataset MNIST-2bit.
In Figure 7(a), we show the results of ﬁtting a 2d β -VAE model (Higgins et al., 2017b) to the images in MNIST-2bit, ignoring the attributes .
We perform a hyperparameter sweep over β , and pick the one that gives the best looking latent space (this corresponds to a value of β = 10) .
At each point z in the latent 2d space, we show a single image sampled from p ( x | z ) .
To derive the colors for each point in latent space, we proceed as follows: we embed each training image
x
(with label y
( x ) ) into latent space, by computing ˆ z ( x ) =
E q ( z | x ) [ z ] .
We then associate label y ( x ) with this point in space.
To derive the label for an arbitrary point z , we lookup the closest embedded training image (using (cid:96) 2 distance in z space), and use its corresponding label.
We see that the latent space is useful for autoencoding (since the generated images look good), but it does not capture the relevant semantic properties of parity and magnitude.
In fact, we argue that there is no way of forcing the model to learn a latent space that captures such high level conceptual properties from images alone.
In Figure 7(b), we show the results of ﬁtting a joint VAE model to MNIST-2bit, by optimizing elbo( x, y ) on images and attributes ( i.e. , we do not include the uni-modality elbo( x ) and elbo( y ) terms in this experiment.)
Now the color codes are derived from p ( y | z ) rather than using nearest neighbor retrieval.
We see that the latent space autoencodes well, and also captures the 4 relevant types of concepts.
In particular, the regions are all convex and linearly seperable, which facilitates the learning of a good imagination function q ( z | y ) , interpolation, retrieval, and other latent-space tasks.
A skeptic might complain that we have created an arbitrary partitioning of the data, that is unrelated to the appearance of the objects, and that learning such concepts is therefore “unnatural”.
But consider an agent interacting with an environment by touching digits on a screen.
Suppose the amount of reward they get depends on whether the digit that they touch is small or big, or odd or even.
In such an environment, it would be very useful for the agent to structure its internal representation to capture the concepts of magnitude and parity, rather than in terms of low level visual similarity. (In fact, (Scarf et al., 2011) showed that pigeons can learn simple numerical concepts, such as magnitude, by rewarding them for doing exactly this!) Language can be considered as the realization of such concepts, which enables agents to share useful information about their common environments more easily. A. D ETAILS OF THE NEURAL NETWORK ARCHITECTURES
As explained in the main paper, we ﬁt the joint graphical model p ( x, y, z )
= p ( z ) p ( x | z ) p ( y | z ) with inference networks q ( z | x, y ) , q ( z | x ) , and q ( z | y ) .
Thus, our overall model is made up of three encoders (denoted with q ) and two decoders (denoted with p ).
Across all models we use the exponential linear unit (ELU) which is a leaky non-linearity often used to train VAEs.
We explain the architectures in more detail below.
MNIST-A model architecture
• Image decoder, p ( x | z ) : Our architecture for the image decoder exactly follows the standard DCGAN architecture from (Radford et al., 2015), where the input to the model is the latent state of the VAE.
• Label decoder, p ( y | z ) : Our label decoder assumes a factorized output space p ( y | z ) = (cid:81)
k ∈A p
( y k | z ) , where y k is each individual attribute.
We parameterize each p ( y k | z ) with a two-layer MLP with 128 hidden units each.
We apply a small amount of (cid:96) 2 regularization to the weight matrices.
• Image and Label encoder, q ( z | x, y ) :
Our architecture (Figure 8) for the image-label encoder ﬁrst separately processes the images and the labels, and then concatenates them downstream in the network and then passes the concatenated features through a multi-layered perceptron.
More speciﬁcally, we have convolutional layers which process image into 32 , 64 , 128 ,feature maps with strides 1 , 2 , 2 , 2 in the corresponding layers.
We use batch normalization in the convolutional layers before applying the ELU non-linearity.
On the label encoder side, we ﬁrst encode the each attribute label into a 32d continuous vector and then pass each individual attribute vector through a 2-layered MLP with 512 hidden dimensions each.
For example, for MNIST-A we have 4 attributes, which gives us 4 vectors of 512d.
We then concatenate these vectors and pass it through a two layer MLP.
Finally we concatenate this label feature with the image feature after the convolutional layers (after ﬂattening the conv-features) and then pass the result through a 2 layer MLP to predict the mean ( µ ) and standard deviation ( σ ) for the latent space gaussian.
Following standard practice, we predict log σ for the standard deviation in order to get values which are positive.
Class [10]
Scale [2]
Orientation [3] Location [4] concat (2048)
64x64x
Image [64x64x1] flatten (1024) concat (1536)
µ
[Figure]
• Image encoder, q ( z | x ) : The image encoder (Figure 9a) uses the same architecture to process the image as the image feature extractor in q ( z | x, y ) network described above.
After the conv-features, we pass the result through a 3-layer MLP to get the latent state mean and standard deviation vectors following the procedure described above.
• Label encoder, q ( z | y ) :
The label encoder (Figure 9b) part of the architecture uses the same design choices to process the labels as the label encoder part in the q ( z | x, y ) network.
After obtaining the concatenated label feature vectors, we pass the result through a 4-layered MLP with 512 hidden dimensions each and then ﬁnally obtain the mean ( µ ) and log σ values for each dimension in the latent state of the VAE.
MNIST-A Observation Classiﬁer Model We next describe the architecuture of the observation classiﬁer we use for evaluating the 3C’s on the MNIST-A dataset. The observation classiﬁer is a convolutional neural network, with the ﬁrst convolutional layer with ﬁlters of size 5 × 5, andchannels, followed by a 2 × 2 pooling layer applied with a stride of 2. This is followed by another convolutional layer with 5 × 5 ﬁlter size and 64 output channels. This is followed by another 2 ×pooling layer of stride 2.
After this, the network has four heads (corresponding to each attribute), each of which is an MLP with a single hidden layer (of size 1024), with dropout applied to the activations.
The ﬁnal layer of the MLP outputs the logits for classifying each attribute into the corresponding categorical labels associated with it. We train this model from scratch on the MNIST-A dataset using stochastic gradient descent, batch size of 64 and a learning rate of 10 − 4 . CelebA model architecture Our design choices for CelebA closely mirror the models we built for MNIST-A. One primary difference is that we use a latent dimensionality of 18 in our CelebA experiments which matches the number of attributes we model.
Meanwhile, the architectures of the image encoder, image decoder ( i.e. DCGAN), are exactly identical to what is described above for
MNIST-A execept that encoders take as input a 3-channel RGB image, while decoders produce a 64x64x flatten (1024) [Figure] Image [64x64x1] µ
[Figure]
concat (2048)           µ  [Figure] 3-channel output. We replace the Bernoulli likelihood with Quantized Normal likelihood (which is basically gaussian likelihood with uniform noise).
In terms of the label encoder q ( z | y ) , we follow Figure 9b quite closely, except that we get as inputcategorical (embedded) class labels as input, and we process the labels through a single hidden layer before concatenation and two hidden layers post concatenation (as opposed to two and four used in
Figure 9b).
Finally, the joint encoder q ( z | x, y ) , is again based heavily on Figure 8 where we feed as inputlabels as opposed to 4, process them through a single layer mlp of 512d, concatenate them, and then pass the result through a two hidden layer mlp of 512 d.
At this point we concatenate the result with the image feature through the image feature head in Figure 8.
Finally, we process the feature through another 512d single hidden layer mlp to produce the µ, σ values.
A. O UTPUTS OF OBSERVATION CLASSIFIER ON GENERATED IMAGES
Figure 10 shows some images sampled from our TELBO model trained on
MNIST-A.
It also shows the attributes that are predicted by the attribute classiﬁer.
We see that the classiﬁer often produces reasonable results that we as humans would also agree with.
Thus, it acts as a reasonable proxy for humans classifying the labels for the generated images.
A. H YPERPARAMTER C HOICES FOR TELBO, JMVAE, B I VCCA ON MNIST-A
We discuss more hyperparameter choices for the different objectives and how they impact performance on the MNIST-A dataset. Across all the objectives we set λ x =1, and vary λ y . In addition, we also discuss how the private hyperparamter choices for each loss, γ for TELBO, α for JMVAE, as in Wang et al.
(2016a)) and µ for BiVCCA affect performance.
We use the JS-overall metric for picking hyperparameters, as explained in the main paper.
[Figure]
[Figure]
Effect of λ y :
We search for λ y values in the set { 1 , 50 , 100 } for all objectives.
In general, we ﬁnd the setting of λ y in the elbo terms to be critical for good performance (especially on correctness).
For example, at λ y =1, we ﬁnd that correctness numbers for the best performing
TELBO model drop to 60.47 ( ± 0.34) (from 82.08 ( ± 0.56) at λ y =50) on the validation set for iid queries.
Similar trends can be observed for the JMVAE and BiVCCA objectives as well (with λ y =
10 being the best setting for BiVCCA, λ y =50 for JMVAE).
We have seen qualitative evidence which shows that the likelihood scaling for λ y affects how disentangled the latent space is along the speciﬁed attributes.
When the latent space is not grouped or organized as per high-level attributes (see Figure 7 for example), the posterior distribution for a given concept is multimodal, which is hard for a gaussian inference network q ( z | y ) to capture.
This leads to poor correctness values.
Effect of γ :
In addition to the λ y scaling term which is common across all objectives,
TELBO has a γ scaling factor which controls how we scale the log p ( y | z ) term in the elbo γ, 1 ( y , θ y , φ y ) term.
We sweep values of { 1 , 50 , 100 } for this parameter.
In general, we ﬁnd that the effect of this term is smaller on the performance
than the λ y term.
Based on the setting of this parameter, we ﬁnd that, for example, the correctness values for fully speciﬁed queries change from 82.08 ( ± 0.56) at γ =50 to 80.27 ( ± 0.38) at γ =1 on validation set for iid queries.
Effect of α : We generally ﬁnd that α =1.0 works best for JMVAE across the different choices explored in Wang et al.
(2016a), namely, { 0 . 01 , 0 . 1 , 1 . 0 } .
For example, decreasing the value of α to 0.1 or 0.01 reduces correctness for fully sepciﬁed queries from 85.63 ( ± 0.29) to 77.58 ( ± 0.23) at 0.1 and 74.57 ( ± 0.44) at 0.01 respectively on the validation set for iid queries.
Effect of µ : For BiVCCA, we ran a search for µ over { 0 . 3 , 0 . 5 , 0 . 7 } , running each training experiment four times, and picked the best hyperparameter choice across the runs.
We found that µ =0.7 was the best value, however the performance difference across different choices was not very large.
Intuitively, higher values of µ should lead to improved performance compared to lower values of µ .
This is because lower values of µ mean that we put more weight on the elbo term with a q ( z
| x ) inference network than the one with a q ( z | y ) inference network, which results in sharper samples.
A. C OMPOSITIONAL GENRALIZATION ON MNIST-A: Q UALITATIVE R ESULTS AND D ETAILS
We next show some examples of compositional generalization on MNIST-A on a validation set of queries.
For the compositinal experiments we reused the parameters of the best models on the iid splits for all the models, and trained the models for ∼ 160 K iterations.
All other design choices were the same.
Figure 11 shows some qualitative results. [Figure]
A. D ETAILS ON C ELEB A
CelebA consists of 202,599 face colored images and 40 attribute binary vectors.
We use the version of this dataset that was used in (Perarnau et al., 2016); this uses a subset of 18 visually distinctive attributes, and preprocesses each image so they are aligned, cropped, and scaled down to 64 x 64.
We use the ofﬁcial train and test partitions, 182K for training and 20K for testing.
Note that this is an iid split, so the attribute vectors in the test set all occur in the training set, even though the images and people are unique.
In total, the original dataset with 40 attributes speciﬁed a set of 96486 unique visual concepts, while our dataset of 18 attributes spans 3690 different visual concepts.
In Section 5.2, we claim that our generations of “Bald” and “Female” images are from a compositionally novel concept.
Our claim comes with a minor caveat/clariﬁcation: the concept bald = 1 and male = 0 does occur in 9 training examples, but they are all incorrect labelings, as shown in Figure 12!
Further, we see that the images generated from our model (shown in Figure 5) are qualitatively very different from any of the images here, showing that the model has not memorized these examples. A. M ORE RESULTS ON C ELEB A
Finally, we show further qualitative examples of performance on the CelebA dataset.
We focus on the TELBO and JMVAE objectives here, since BiVCCA generally produces poor samples (see Figure 5). Figure 13 (middle) shows some example generations for the concept speciﬁed by the attributes (top). We see that both TELBO and JMVAE produce correct images when provided the full attribute queries (ﬁrst two rows). However, when we stop specifying attribute “male” or “not male” (female), we see that TELBO provides more diverse samples, spanning both male and female (compared to JMVAE).
This ties into the explanation in Appendix A.1, where we show how one can interpret JMVAE as optimizing for the KL( q avg φ ( z | y i )
| q φ y ( z | y i )) to ﬁt the unimodal inference network q φ y ( z | y i ) .
Since JMVAE only reasons about the “aggregate” posterior as opposed to the prior (which TELBO reasons about), it has the tendency to generate less diverse samples when shown unseen concepts.
[Figure]