L ATENT N ORMALIZING F LOWS FOR M ANY - TO -M ANY C ROSS -D OMAIN M APPINGS
A BSTRACT
Learned joint representations of images and text form the backbone of several important cross-domain tasks such as image captioning.
Prior work mostly maps both domains into a common latent representation in a purely supervised fashion.
This is rather restrictive, however, as the two domains follow distinct generative processes.
Therefore, we propose a novel semi-supervised framework, which models shared information between domains and domain-speciﬁc information separately.
The information shared between the domains is aligned with an invertible neural network.
Our model integrates normalizing ﬂow-based priors for the domain-speciﬁc information, which allows us to learn diverse many-to-many mappings between the two domains.
We demonstrate the effectiveness of our model on diverse tasks, including image captioning and text-to-image synthesis.
I NTRODUCTION
Joint image-text representations ﬁnd application in cross-domain tasks such as imageconditioned text generation (captioning; Mao et al., 2015; Karpathy & Fei-Fei, 2017;
Xu et al., 2018) and text-conditioned image synthesis (Reed et al., 2016).
Yet, image and text distributions follow distinct generative processes, making joint generative modeling of the two distributions challenging.
Current state-of-the-art models for learning joint image-text distributions encode the two domains in a common shared latent space in a fully supervised setup (Gu et al., 2018; Wang et al., 2019).
While such approaches can model supervised information in the shared latent space, they do not preserve domain-speciﬁc information.
However, as the domains under consideration, e . g . images and texts, follow distinct generative processes, many-to-many mappings naturally emerge – there are many likely captions for a given image and vice versa.
Therefore, it is crucial to also encode domain-speciﬁc variations in the latent space to enable many-to-many mappings.
State-of-the-art models for cross-domain synthesis leverage conditional variational autoencoders (VAEs, cVAEs; Kingma & Welling, 2014) or generative adversarial networks (GANs; Goodfellow et al., 2014) for learning conditional distributions.
However, such generative models ( e . g ., Wang et al., 2017; Aneja et al., 2019) enforce a Gaussian prior in the latent space.
Gaussian priors can result in strong regularization or posterior collapse as they impose stringent constraints while modeling complex distributions in the latent space (Tomczak & Welling, 2018).
This severely limits the accuracy and diversity of the cross-domain generative model.
Recent work (Ziegler & Rush, 2019; Bhattacharyya et al., 2019) has found normalizing ﬂows (Dinh et al., 2015) advantageous for modeling complex distributions in the latent space.
Normalizing ﬂows can capture a high degree of multimodality in the latent space through a series of transformations from a simple distribution to a complex data-dependent prior.
Ziegler & Rush (2019) apply normalizing ﬂow-based priors in the latent space of unconditional variational autoencoders for discrete distributions and character-level modeling.
We propose to leverage normalizing ﬂows to overcome the limitations of existing cross-domain generative models in capturing heterogeneous distributions and introduce a novel semi-supervised
Latent Normalizing Flows for Many-to-Many Mappings (LNFMM) framework.
We exploit normalizing ﬂows (Dinh et al., 2015) to capture complex joint distributions in the latent space of our model (Fig. 1).
Moreover, since the domains under consideration, e . g . images and texts, have different generative processes, the latent representation for each distribution is modeled such that it contains both shared cross-domain information as well as domain-speciﬁc information.
The latent dimensions constrained by supervised information from paired data model the common (semantic) information across images and texts.
The diversity within the image and text distributions, e .
g .
different visual or textual styles, are encoded in the residual latent dimensions, thus preserving domain-speciﬁc variation.
We can hence synthesize diverse samples from a distribution given a reference point in the other domain in a many-to-many setup.
We show the beneﬁts of our learned many-to-many latent spaces for real-world image captioning and text-to-image synthesis tasks on the COCO dataset (Lin et al., 2014).
Our model outperforms the current state of the art for image captioning w . r . t .
the Bleu and CIDEr metrics for accuracy as well as on various diversity metrics.
Additionally, we also show improvements in diversity metrics over the state of the art in text-to-image generation.
R ELATED W ORK
Diverse image captioning.
Recent work on image captioning introduces stochastic behavior in captioning and thus encourages diversity by mapping an image to many captions.
Vijayakumar et al.
(2018) sample captions from a very high-dimensional space based on word-to-word Hamming distance and parts-of-speech information, respectively.
To overcome the limitation of sampling from a high-dimensional space, Shetty et al.
; Dai et al.
; Li et al.
(2018) build on Generative Adversarial Networks (GANs) and modify the training objective of the generator, matching generated captions to human captions.
While GAN-based models can generate diverse captions by sampling from a noise distribution, they suffer on accuracy due to the inability of the model to capture the true underlying distribution.
Wang et al.
Aneja et al.
(2019), therefore, leverage conditional Variational Autoencoders (cVAEs) to learn latent representations conditioned on images based on supervised information and sequential latent spaces, respectively, to improve accuracy and diversity.
Without supervision, cVAEs with conditional Gaussian priors suffer from posterior collapse.
This results in a strong trade-off between accuracy and diversity; e .
g .
Aneja et al.
(2019) learn sequential latent spaces with a Gaussian prior to improve diversity, but suffer on perceptual metrics.
Moreover, sampling captions based only on supervised information limits the diversity in the captions.
In this work we show that by learning complex multimodal priors, we can model text distributions efﬁciently in the latent space without speciﬁc supervised clustering information and generate captions that are more diverse and accurate.
Diverse text-to-image synthesis.
State-of-the-art methods for text-to-image synthesis are based on conditional GANs (Reed et al., 2016).
Much of the research for text-conditioned image generation has focused on generating high-resolution images similar to the ground truth.
Zhang et al.
(2017; 2019b) introduce a series of generators in different stages for high-resolution images.
AttnGAN (Xu et al., 2018) and MirrorGAN (Qiao et al., 2019) aim at synthesizing ﬁne-grained image features by attending to different words in the text description.
Dash et al.
(2017) condition image generation on class information in addition to texts.
Yin et al.
(2019) use a Siamese architecture to generate images with similar high-level semantics but different low-level semantics.
In this work, we instead focus on generating diverse images for a given text with powerful latent semantic spaces, unlike GANs with
Gaussian priors, which fail to capture the true underlying distributions and result in mode collapse.
Normalizing ﬂows & Variational Autoencoders.
Normalizing ﬂows (NF) are a class of density estimation methods that allow exact inference by transforming a complex distribution to a simple distribution using the change-of-variables rule.
Dinh et al.
(2015) develop ﬂow-based generative models with afﬁne transformations to make the computation of the Jacobian efﬁcient.
Recent works (Dinh et al., 2017; Kingma & Dhariwal, 2018; Ardizzone et al., 2019; Behrmann et al., 2019) extend ﬂow-based generative models to multi-scale architectures to model complex dependencies across dimensions.
Vanilla Variational Autoencoders (VAEs; Kingma & Welling, 2014) consider simple
Gaussian priors in the latent space.
Simple priors can provide very strong constraints, resulting in poor latent representations (Hoffman & Johnson, 2016).
Recent work has, therefore, considered modeling complex priors in VAEs.
Particularly, Wang et al.
; Tomczak & Welling (2018) propose mixtures of Gaussians with predeﬁned clusters, Chen et al.
(2017) use neural autoregressive model priors, and van den Oord et al.
(2017) use discrete models in the latent space, which improves results for image synthesis.
Ziegler & Rush (2019) learn a prior based on normalizing ﬂows to model multimodal discrete distributions of character-level texts in the latent spaces with nonlinear ﬂow layers.
However, this invertible layer is difﬁcult to be optimized in both directions.
Bhattacharyya et al.
(2019) learn conditional priors based on normalizing ﬂows to model conditional distributions in the latent space of cVAEs.
In this work, we learn a conditional prior using normalizing ﬂows in the latent space of our variational inference model, modeling joint complex distributions in the latent space, particularly of images and texts for diverse cross-domain many-to-many mappings.
M ETHOD
To learn joint distributions p µ ( x t , x v ) of texts and images that follow distinct generative processes with ground-truth distributions p t ( x t ) and p v ( x v ) , respectively, in a semi-supervised setting, we formulate a novel joint generative model based on variational inference: Latent Normalizing Flows for Many-to-Many Mappings (LNFMM).
Our model deﬁnes a joint probability distribution over the data { x t , x v } and latent variables z with a distribution p µ ( x t , x v , z ) =
p µ ( x t , x v | z ) p µ ( z ) , parameterized by µ .
We maximize the likelihood of p µ ( x t , x v ) using a variational posterior q θ ( z | x t , x v ) , parameterized by variables θ .
As we are interested in jointly modeling distributions with distinct generative processes, e .
g . images and texts, the choice of the latent distribution is crucial.
Mapping to a shared latent distribution can be very restrictive (Xu et al., 2018).
We begin with a discussion of our variational posterior q θ ( z | x t , x v ) and its the factorization in our LNFMM model, followed by our normalizing ﬂow-based priors, which enable q θ ( z | x t , x v ) to be complex and multimodal, allowing for diverse many-to-many mappings.
Factorizing the latent posterior.
We choose a novel factorized posterior distribution with both shared and domain-speciﬁc components.
The shared component z s is learned with supervision from paired image-text data and encodes information common to both domains.
The domain-speciﬁc components encode information that is unique to each domain, thus preserving the heterogeneous structure of the data in the latent space.
Speciﬁcally, consider z t and z v as the latent variables to model text and image distributions.
Recall from above that z s denotes the shared latent variable for supervised learning, which encodes information shared between the data points x t and x v .
Given this supervised information, the residual information speciﬁc to each domain is encoded in z (cid:48) t and z (cid:48) v .
This leads to the factorization of the variational posterior of our LNFMM model with z t =
[ z s z (cid:48) t ] and z v =
[ z s z (cid:48) v ] ,
[Equation]
Next, we derive our LNFMM model in detail.
Since directly maximizing the log-likelihood of p µ ( x t , x v ) with the variational posterior is intractable, we derive the log-evidence lower bound for learning the posterior distributions of the latent variables z
= { z s , z (cid:48) t , z (cid:48) v } .
3. D ERIVING THE L OG -E VIDENCE L OWER B OUND
Maximizing the marginal likelihood p µ ( x t , x v ) given a set of observation points { x t , x v } is generally intractable.
Therefore, we develop a variational inference framework that maximizes a variational lower bound on the data log-likelihood – the log-evidence lower bound (ELBO) with the proposed factorization in Eq.
[Equation]
where z = { z s , z (cid:48) t , z (cid:48) v } are the latent variables.
The ﬁrst expectation term is the reconstruction error.
The second expectation term minimizes the KL-divergence between the variational posterior q θ ( z | x t , x v ) and a prior p φ ( z ) .
Taking into account the factorization in Eq.
[Equation] we now derive the
ELBO for our LNFMM model.
We ﬁrst rewrite the reconstruction term as
[Equation]
which assumes conditional independence given the domain-speciﬁc latent dimensions z (cid:48) t , z (cid:48) v and the shared latent dimensions z s .
Thus, the reconstruction term can be further simpliﬁed as
[Equation]
[Equation]
Next, we simplify the KL-divergence term on the right of Eq.
We use the chain rule along with
Eq.
(1) to obtain
[Equation]
This assumes a factorized prior of the form [Equation] t ( z [Equation]
|
[Equation]
s )
[Equation] v ( z [Equation] v | z s ) , consistent with our conditional independence assumptions, given that information speciﬁc to each distribution is encoded in { z (cid:48) t , z (cid:48) v } .
The ﬁnal ELBO can then be expressed as
[Equation]
, x
[Equation]
In the standard VAE formulation (Kingma & Welling, 2014), the priors corresponding to p φ t ( z (cid:48) t | z s ) and p φ v ( z (cid:48) v | z s ) are modeled as standard normal distributions.
However, Gaussian priors limit the expressiveness of the model in the latent space since they result in strong constraints on the posterior (Tomczak & Welling, 2018; Razavi et al., 2019; Ziegler & Rush, 2019).
Speciﬁcally, optimizing with a Gaussian prior pushes the posterior distribution towards the mean, limiting diversity and hence generative power (Tomczak & Welling, 2018).
This is especially true for complex multimodal image and text distributions.
Furthermore, alternatives like Gaussian mixture model-based priors (Wang et al., 2017) also suffer from similar drawbacks and additionally depend on predeﬁned heuristics like the number of components in the mixture model.
Analogously, the VampPrior (Tomczak &
Welling, 2018) depends on a predeﬁned number of pseudo-inputs to learn the prior in the latent space.
Similar to Ziegler & Rush (2019)
; Bhattacharyya et al.
(2019), which learn priors based on exact inference models, we propose to learn the conditional priors p φ t ( z (cid:48)
t | z s ) and p φ v ( z (cid:48) v | z s ) jointly with the variational posterior in Eq.
(1) using normalizing ﬂows.
3. V ARIATIONAL I NFERENCE WITH N ORMALIZING F LOW - BASED P RIORS
Normalizing ﬂows are exact inference models, which can map simple distributions to complex densities through a series of K t invertible mappings,
[Equation]
This allows us to transform a simple base density (cid:15) ∼ p ( (cid:15) ) to a complex multimodal conditional prior p φ t ( z (cid:48) t | z s ) (and correspondingly to p φ v ( z (cid:48) v | z s ) ).
The likelihood of the latent variables under the base density can be obtained using the change-of-variables formula.
A composition of invertible mappings f iφ t , parameterized by parameters φ t , is learned such that (cid:15) =
f − 1 φ t ( z (cid:48) t | z s ) .
The loglikelihood with Jacobian J iφ t =
∂f iφt / [Equation] 1 φt , assuming f 0 φ t
= I is the identity, can be expressed as
[Equation]
Using data-dependent and non-volume preserving transformations
[Equation]
multimodal priors can be jointly learned in the latent space [Equation] allowing for more complex posteriors and better solutions of the evidence lower bound.
Using Eq.
[Equation] the ELBO with normalizing ﬂow-based priors can be expressed by rewriting the KL-divergence terms in Eq.
(6) as (analogously for the other term)
[Equation]
Next, we describe our complete model for learning joint distributions with latent normalizing ﬂows using Eqs.
(6) and (8), which enables many-to-many mappings between domains.
3. L ATENT N ORMALIZING F LOW M ODEL FOR M ANY - TO -M ANY M APPINGS
f ϕ s K ﬂow layers
Image Encoder
Image Decoder p ( ϵ ) f
ϕ t
K t ﬂow layers p ( ϵ ) f
ϕ v K v ﬂow layers −log p ( x t ) −log p ( x v ) x v x t p ϕ t ( z ′  t | z s ) p ϕ v ( z ′  v | z s )
z s z s z ′  v z ′
t
We illustrate our complete model in Fig.
It consists of two domain-speciﬁc encoders to learn the domainspeciﬁc latent posterior distributions q θ 2 ( z (cid:48) t | x t , z s ) and q θ 3 ( z (cid:48) v | x t , z s ) .
As the shared latent variable
z s encodes information common to both domains, it holds that q θ 1 ( z s | x t , x v ) =
q θ 1 ( z s | x t )
=
q θ 1 ( z s | x v ) for a matching pair of data points ( x t , x v ) .
Therefore, each encoder must be able to model the common supervised information independently for every matching pair ( x t , x v ) .
We enforce this by splitting the output dimensions of each encoder into z v =
[ z s z (cid:48) v ] and z t =
[ z s z (cid:48) t ] , respectively (cf. Eq. 1), and constraining the supervised latent dimensions to encode the same information.
We assume that the shared latent code z s has d (cid:48) dimensions.
We propose to learn the posterior distribution q θ 1 ( z s | x t , x v ) as the shared latent space between two domain-speciﬁc autoencoders.
One simple method to induce sharing is by minimizing the mean-squared error between the encodings.
However, this is not ideal given that x t and x v follow different highly multimodal generative processes.
We, therefore, learn an invertible mapping f φ s : R d (cid:48)
→ R d (cid:48) with invertible neural networks such that the d (cid:48) -dimensional latent code z s can be transformed between the domains v and t .
Let z v with d ≥ d (cid:48) dimensions be the encoded latent variable for distribution p v ( x v ) .
A bijective mapping f φ s : ( z v ) d (cid:48)
(cid:55)→ ( z t ) d (cid:48) is learned with an invertible mapping analogous to Eq.
(7) as
[Equation]
Here, ( · ) d (cid:48) denotes restricting the latent code to the d (cid:48) -dimensional shared part.
f φ s is an invertible neural network with afﬁne coupling layers (Dinh et al., 2015), i .
e .
f φ s = f
Kφ s ◦
f K − 1 φ s ◦ · · · ◦ f 1 φ s .
This makes it easy to compute Jacobians J iφ s =
∂f iφs / ∂f i − 1 φs formulated as triangular matrices for the layers of the invertible neural network.
Again, we assume that f 0 φ s =
I is the identity.
The number of dimensions for domain-speciﬁc information, i .
e .
the dimensionality of z (cid:48) t and z (cid:48) v , can be different for the two domains, depending on the complexity of each distribution.
Note that by conditioning on the dimensions with a supervision signal, we minimize the redundancy in the dimensions for unsupervised information without disentangling the dimensions.
The multimodal conditional prior in q θ 2 ( z (cid:48) t | z s , x t ) and q θ 3 ( z (cid:48) v | z s , x v ) is modeled with non-volume-preserving normalizing ﬂow models with Eq.
(7) (Dinh et al., 2017), parameterized by φ t and φ v , respectively.
The data log-likelihood terms in Eqs.
(4) and (6) are deﬁned as the (negative) reconstruction errors.
Let g θ v : x v (cid:55)→ z v with θ v = { θ 1 , θ 3 } and g θ t : x t (cid:55)→ z t with θ t = { θ 1 , θ 2 } denote the image and text encoders, respectively.
Further, h ω v : z v (cid:55)→ x v and h ω t : z t (cid:55)→ x t are the decoders with parameters ω v and ω t corresponding to images and texts, respectively, with ˜ x v = h ω v ( g θ v ( x v )) and ˜ x t = h ω t ( g θ t ( x t )) denoting the decoded image and text samples.
For texts, we consider the output probability of the j th word ( x t ) j of the ground-truth sentence x t given the previous reconstructed words (˜ x t ) 0: j − 1 from the text decoder, and deﬁne the reconstruction error as L rec t ( x t , ˜ x t ) = − (cid:80) j log p µ (cid:0) ( x t ) j | (˜ x t ) 0: j − 1 (cid:1) .
For images, the reconstruction loss between the input image x v and reconstructed image ˜ x v from the image decoder is taken as
L recv ( x v , ˜ x v ) = (cid:107) x v − ˜ x v (cid:107) , where (cid:107) · (cid:107) is either the (cid:96) 1 or (cid:96) 2 norm.
Furthermore, in Eq.
(9) we deﬁne log q θ 1 ( f φ s (( z v ) d (cid:48) )
| x t ) as the cost of mapping ( z v ) d (cid:48) to the latent space of texts under the transformation f φ s ; we use the mean-squared error between the encoded text representations ( z t ) d (cid:48) and the transformed image representation f φ s (( z v ) d (cid:48) ) of the paired data ( x t , x v ) (see Ardizzone et al., 2019). [Table] [Table]
Starting from the ELBO on the right-hand side of Eq.
(6), the learned latent priors in Eq.
(8), the invertible mapping from Eq.
(9), and plugging in the reconstruction terms as just deﬁned, the overall objective of our semi-supervised generative model framework to be minimized is given by
[Equation]
Here, µ = { θ, φ s , φ v , φ t , ω v , ω t } are all parameters to be learned; λ i , i = { 1 , . .
. , 5 } are regularization parameters.
Recall that ˜ x t and ˜ x v are decoded text and image samples, respectively. We assume the prior p φ s ( z s ) on the shared latent space to be uniform.
The strength of this uniform prior on the shared dimensions can be controlled with the regularization parameter λ 1 .
Our model allows for bi-directional many-to-many mappings.
In detail, given a data point x v from the image domain with latent encoding z v , we ﬁrst map it to the text domain through the invertible transformation z s =
f φ s (( z v ) d (cid:48) ) .
We can now generate diverse texts by sampling from the learned latent
prior p φ t ( z (cid:48) t | z s ) .
A similar procedure is followed for sampling images given text through the learned prior p φ v ( z (cid:48) v | z s ) .
For conditional generation tasks, as we do not have to sample from the supervised latent space, we ﬁnd a “uniform” prior p φ s ( z s ) to be advantageous in practice as it loosens the constraints on the decoders.
Alternatively, a more complex ﬂow-based prior could also be used here to enable sampling of the shared semantic space.
We show the effectiveness of our joint latent normalizing ﬂow-based priors on real-world tasks, i .
e .
diverse image captioning and text-to-image synthesis.
E XPERIMENTS
To validate our method for learning many-to-many mappings to provide latent joint distributions, one of the important real-world tasks is that of image-to-text or text-to-image synthesis.
To that end, we perform experiments on the COCO dataset (Lin et al., 2014).
It contains 82,783 training and 40,504 validation images, each with ﬁve captions.
Following Wang et al.
; Mao et al. (2015) for image captioning, we use 118,287 data points for training and evaluate on 1,000 test images.
For text-to-image synthesis, the training set contains 82,783 images and 40,504 validation data points at test time (Reed et al., 2016; Huang et al., 2017).
Architecture details can be found in the Appendix.
4. I MAGE C APTIONING
We evaluate our approach against methods that generate diverse captions for a given image.
We compare against methods based on (conditional) variational autoencoders, AG-CVAE (Wang et al.
, ∗ Our implementation
[Table]
[Table] [Table]
in a
[Table]
water.
[Table]
Two
[Table]
while standing in the water. • Two
[Table]
water [Table] [Table] group
[Table]
around
[Table] [Table] room with furniture on the [Table]
2016), which uses additional supervision based on information about objects in the images for a Gaussian mixture model in the latent space
[Table]
and Seq-CVAE (Aneja et al., 2019), which models sequential latent spaces with Gaussian priors.
We also include Div-BS (Vijayakumar et al., 2018) based on beam search and POS (Aneja et al., 2018), which uses additional supervision from images. Additionally, we include different ablations to show the effectiveness of various components of our approach. LNFMM-MSE does not contain the invertible neural network f φ s , i . e .
we directly minimize the mean squared error between ( z t ) d (cid:48) and ( z v ) d (cid:48) in Eq.
We ﬁx the image encodings of a VGG-16 encoder in LNFMM (pre-trained) for comparison to image captioning methods with pre-trained image features.
Furthermore, LNFMM-TXT contains the latent ﬂow for unsupervised information f φ t only for the text distribution.
In this setting, we have z v =
z s and thus z v is transformed through the invertible neural network as f φ s ( z v ) .
To show that our framework can be applied in the semi-supervised setting, we also consider limited labeled data with only 30% of the paired training data for supervision.
The remaining training data is included as unpaired data for modeling domain-speciﬁc information.
Evaluation .
We evaluate the accuracy with Bleu (B) 1-4 (Papineni et al., 2002), CIDEr (C; Vedantam et al., 2015), ROUGE (R; Lin, 2004), METEOR (M; Denkowski & Lavie, 2014), and SPICE (S; Anderson et al., 2016).
For evaluating diversity, we consider the metrics of Wang et al.
Aneja et al.
Uniqueness is the percentage of unique captions generated on the test set.
Novel sentences are the captions that were never observed in the training data.
m-Bleu-4 computes Bleufor each diverse caption with respect to remaining diverse captions per image.
The Bleu-4 obtained is averaged across all images.
Div-n is the ratio of distinct n -grams to the total number of words generated per set of diverse captions.
Results .
In Table 1 we show the caption evaluation metrics in the oracle setting, i .
e .
taking the maximum score for each accuracy metric over all the candidate captions.
We consider 100 samples z , consistent with previous methods.
The cVAE baseline with an image-conditioned Gaussian prior does not perform well on all metrics, showing the inability of the Gaussian prior to model meaningful latent spaces representative of the multimodal nature the underlying data distribution.
The overall trend across metrics is that our LNFMM model improves the upper bound on Bleu and CIDEr while being comparable on the Rouge and Spice metrics.
Comparing the accuracy of the baseline LNFMM-MSE with LNFMM, we can conclude that learning the shared posterior distribution of z s with our invertible mapping is better than directly minimizing the mean squared error in the latent space due to differences in the complexity of the distributions.
Also note that LNFMM (pre-trained) with ﬁxed image-encoded representations has better performance compared to AG-CVAE and Seq-CVAE, in particular. This highlights that the LNFMM [Figure]
Sample [Figure] [Figure] learns representations in the latent space that are representative of the underlying data distribution even for pre-trained features.
Table 2 considers a more realistic setting (as ground-truth captions are not always available) where, instead of comparing against the reference captions of the test set, reference captions for images from the training set most similar to the test image are retrieved.
The generated captions are then ranked with the CIDEr score (Mao et al., 2015).
While Div-BS has very good accuracy across metrics due to the wide search space, our LNFMM model gives state-of-the-art accuracy on various Bleu metrics and especially the CIDEr score, which is known to correlate well with human evaluations.
More interestingly, compared to AG-CVAE with conditional Gaussian mixture priors based on object (class) information, our LNFMM model, which does not encode any additional supervised information in the latent space, outperforms the former on all accuracy metrics by a large margin.
Moreover, the recent GXN (Gu et al., 2018) and M 3 DGAN (Ma et al., 2019) also study bi-directional synthesis with joint models in Gaussian latent spaces.
Ma et al.
(2019) additionally model attention in the latent space.
From Table 4, we see that our method considerably outperforms the competing methods, validating the importance of complex priors in the latent space for image-text distributions.
This again highlights that the complex joint distribution of images and texts captured by our LNFMM model is more representative of the ground-truth data distribution.
We additionally experiment with limited labelled training data (30% labeled) , which shows that our approach copes well with limited paired data.
We ﬁnally compare LMFMM against LNFMM-TXT to show the importance of joint learning of image and text generative models.
With a generative model only for texts, the joint distribution cannot be captured effectively in the latent space.
With diversity being an important goal here, we show in Table 3 that our LNFMM method improves diversity across all metrics, with a 6.5% improvement in unique captions generated in the test set and 4741/5000 captions not previously seen in the training set.
Our generated captions for a given image also show more diversity with low mutual overlap (mBLEU) compared to the state of the art.
We, moreover, observe high n -gram diversity for the generated captions of each image.
Div-BS with high accuracy has limited diversity as it can repeat the n -grams in different captions.
POS and AG-CVAE, due to guided supervision in the latent space, offer diversity but model only syntactic or semantic diversity, respectively (Wang & Chan, 2019).
Our proposed LNFMM model in Table 5 shows a range of diverse captions with different semantics and syntactic structure.
Therefore, we conclude that the proposed LNFMM can effectively learn semantically meaningful joint latent representations without any additional object or text-guided supervision.
The data-dependent learnt priors are thus promising for synthesizing captions with high human-correlated accuracy as well as diversity.
4. T EXT - TO -I MAGE S YNTHESIS
Given a text description, we are now interested in generating diverse images representative of the domain-speciﬁc structure of images.
To that end, we include a discriminator to improve the image quality of our image decoder.
Note that this does not affect the joint latent space of the LNFMM model.
We evaluate our method against state-of-the-art approaches such as AttnGAN (Xu et al., 2018), HD-GAN (Zhang et al., 2018b), StackGAN (Zhang et al., 2017), and GAN-INT-CLS (Reed et al., 2016).
While our main goal is to encourage text-conditioned diversity in the generated samples, the current state-of-the-art for text-to-image generation aims at improving the realism of the generated images.
Note that various GANs can be integrated with the image decoder of our framework as desired.
Evaluation.
As we are interested in modeling diversity, we study the diversity in generated images using the Inference via Optimization (IvOM; Srivastava et al., 2017) and LPIPS (Zhang et al., 2018a) metrics against the state-of-the-art AttnGAN.
Given the text, for each matching image, IvOM ﬁnds the closest image the model is capable of generating.
Thus, it shows whether the model can match the diversity of the ground-truth distribution.
LPIPS (Zhang et al., 2018a) evaluates diversity by computing pairwise perceptual similarities using a deep neural network.
Additionally, we also report the Inception score (Salimans et al., 2016). [Table]
Results.
In Table 6, our method improves over AttnGAN for both IvOM and LPIPS scores, showing that our method can effectively model the image semantics conditioned on the texts in the latent space, as well as generate diverse images for a given caption.
Note that AttnGAN uses extra supervision to improve the inception score.
However, it is unclear if this improves the visual quality of the generated images as pointed out by Zhang et al.
(2018b).
We improve the IS over HD-GAN, which does not use additional supervision.
Qualitative examples in Fig.
3 show that our LNFMM model generates diverse images, e .
g ., closeup images of food items as well as different orientations of the baseball player in the ﬁeld.
In Fig.we additionally see that given a caption, images generated by our LNFMM model capture detailed semantics of the test images compared to that of AttnGAN, showing the representative power of our latent space.
C ONCLUSION
We present a novel and effective semi-supervised LNFMM framework for diverse bi-directional many-to-many mappings with learnt priors in the latent space, which enables modeling joint imagetext distributions.
Particularly, we model domain-speciﬁc information conditioned on the shared information between the two domains with normalizing ﬂows, thus preserving the heterogeneous structure of the data in the latent space.
Our extensive experiments with bi-directional synthesis show that our latent space can effectively model data-dependent priors, which enable highly accurate and diverse generated samples of images or texts. Acknowledgement.
This work has been supported by the German Research Foundation as part of the Research Training Group Adaptive Preparation of Information from Heterogeneous Sources (AIPHES) under grant
No. GRK 1994/1.
A A PPENDIX A. N ETWORK A RCHITECTURE
We provide the details of the network architecture of Fig. 2.
Image pipeline.
The network consists of an image encoder built upon VGG-16.
4096-dimensional activations of input images are extracted from the fully-connected layer of VGG-16.
This is followed by a 2048-dimensional fully-connected layer with ReLU activations.
We then project it to the latent space with a 1056-dimensional fully-connected layer.
The image pipeline has d (cid:48)
=
dim z s =and dim z (cid:48) v = 64 dimensions.
In the image decoder, we leverage the architecture of Zhang et al. (2019a) to synthesize images of 64 × 64 or 256 × 256 dimensions.
The input to the decoder has dimensionality 1056.
For the image generation experiments, we additionally apply the discriminator of Zhang et al.
(2019b) to the output of the image decoder.
Text pipeline.
We use a bidirectional GRU with two layers and a hidden size of 1024 as text encoder.
This outputs 1024-dimensional latent representations for sentences.
For text, dim z (cid:48) t = 32 .
The text decoder is a LSTM with one layer and hidden size of 512.
Flow modules.
Our network consists of two ﬂow modules for conditional priors on image and text domains and an invertible neural network to exchange supervised information.
Invertible neural network for supervision ( f φ s ) : This invertible neural network consists of 12 ﬂow layers and input dimensionality of 992.
Each ﬂow consists of conditional afﬁne coupling layers followed by a switch layer (Dinh et al., 2015).
Latent ﬂow for conditional prior on images ( f φ v ) : We map the 64 dimensions of the image encodings from the image encoder to a Gaussian with normalizing ﬂows with 16 layers of ﬂow and 512 hidden channels.
Each ﬂow consists of conditional afﬁne coupling layers followed by a switch layer (Dinh et al., 2017).
Latent ﬂow for conditional prior on texts ( f φ t ) : We map the 32 dimensions of the text encodings from the text encoder to a Gaussian with normalizing ﬂows with 16 layers of ﬂow and 1024 hidden channels.
Each ﬂow consists of a conditional activation normalization layer followed by conditional afﬁne coupling layers.
Invertible 1 × 1 convolutions are applied to the output of the afﬁne coupling layers, which is followed by a switch layer (Kingma & Dhariwal, 2018).
A. D IVERSITY IN I MAGE C APTIONING
We show more qualitative examples of the captions generated by our LNFMM model in Table 7.
The example captions show syntactic as well as semantic diversity.
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Figure]
We additionally show diverse images generated by our LNFMM model in Fig.
Our generated images can successfully capture the text semantics and also exhibit image speciﬁc diversity, e .
g .
in style and orientation of objects.
Furthermore, to show that the latent space captures the joint distribution, we show the images generated by our model with IvOM by ﬁnding a z v conditioned on the input text that is most likely to have generated the test image.
We show the images generated for the z v most likely to have generated the image.
Our generated samples in Fig.
7 capture the details in the images, showing that our LNFMM model learns powerful latent representations.
A. V ISUALIZATION OF LATENT SPACES
In Table 8 we show the t-SNE (van der Maaten & Hinton, 2008) visualization of the latent space of texts conditioned on an image.
In the examples shown, we observe global patterns like ‘table with foods’ across all captions of an image and local patterns in clusters like ‘table with several’ or ‘a dinner table with’ showing image-conditioned representation as well as domain-speciﬁc variation.
[Table]
Similarly, in Table 9 we can observe clusters in image space conditioned on texts.
Here again, we can observe global patterns like ‘close-up of food’ and local A. V ALIDATION WITH H UMAN E VALUATION
In addition to validating the performance of our LNFMM method on image captioning for accuracy in Table 1 and for diversity Table 3 with various measures
[Table]
we conduct a human evaluation of the quality of captions generated by our LNFMM approach against Div-BS (Vijayakumar et al., 2018) for accuracy and diversity.
We presented ﬁve captions each for a set of images and for each method to four human annotators.
The human annotators assessed the captions for accuracy and diversity on a scale from 1 to 10.
Here, accuracy is deﬁned as how well the captions describe the details in a given image and diversity can be syntactic diversity or semantic diversity.
In Fig. 8 ( left ), the score for each image is the average score given by the four annotators and Fig.
( right ) shows the scores for the captions of each image from each annotator.
Here, our ﬁndings are similar to those in Table 1 and Table 3.
The captions generated by Div-BS were assessed to have diversity scores in the range of [5 , 8] , while the diversity scores fell into the range [7 , 10] for our LNFMM method.
Furthermore, the accuracy of the captions of each image generated with our
LNFMM was found to be better compared to Div-BS, thereby showing that LNFMM can generate captions that are both diverse and accurate.
[Figure]
[Table]