O N THE L IMITATIONS OF M ULTIMODAL VAE S
A BSTRACT
Multimodal variational autoencoders (VAEs) have shown promise as efﬁcient generative models for weakly-supervised data.
Yet, despite their advantage of weak supervision, they exhibit a gap in generative quality compared to unimodal VAEs, which are completely unsupervised.
In an attempt to explain this gap, we uncover a fundamental limitation that applies to a large family of mixture-based multimodal VAEs.
We prove that the sub-sampling of modalities enforces an undesirable upper bound on the multimodal ELBO and thereby limits the generative quality of the respective models.
Empirically, we showcase the generative quality gap on both synthetic and real data and present the tradeoffs between different variants of multimodal VAEs.
We ﬁnd that none of the existing approaches fulﬁlls all desired criteria of an effective multimodal generative model when applied on more complex datasets than those used in previous benchmarks.
In summary, we identify, formalize, and validate fundamental limitations of VAE-based approaches for modeling weakly-supervised data and discuss implications for real-world applications.
I NTRODUCTION
In recent years, multimodal VAEs have shown great potential as efﬁcient generative models for weakly-supervised data, such as pairs of images or paired images and captions.
Previous works (Wu and Goodman, 2018; Shi et al., 2019; Sutter et al., 2020) demonstrate that multimodal VAEs leverage weak supervision to learn generalizable representations, useful for downstream tasks (Dorent et al., 2019;
Minoura et al., 2021) and for the conditional generation of missing modalities (Lee and van der
Schaar, 2021).
However, despite the advantage of weak supervision, state-of-the-art multimodal
VAEs consistently underperform when compared to simple unimodal VAEs in terms of generative quality.
This paradox serves as a starting point for our work, which aims to explain the observed lack of generative quality in terms of a fundamental limitation that underlies existing multimodal VAEs.
What is limiting the generative quality of multimodal VAEs?
We ﬁnd that the sub-sampling of modalities during training leads to a problem that affects all mixture-based multimodal VAEs—a family of models that subsumes the MMVAE (Shi et al., 2019), MoPoE-VAE (Sutter et al., 2021), and a special case of the MVAE (Wu and Goodman, 2018).
We prove that modality sub-sampling enforces an undesirable upper bound on the multimodal ELBO and thus prevents a tight approximation of the joint distribution when there is modality-speciﬁc variation in the data.
Our experiments demonstrate that modality sub-sampling can explain the gap in generative quality compared to unimodal VAEs and that the gap typically increases with each additional modality.
Through extensive ablations on three different datasets, we validate the generative quality gap between unimodal and multimodal
VAEs and present the tradeoffs between different approaches.
Our results raise serious concerns about the utility of multimodal VAEs for real-world applications.
We show that none of the existing approaches fulﬁlls all desired criteria (Shi et al., 2019; Sutter et al., 2020) of an effective multimodal generative model when applied to slightly more complex datasets than used in previous benchmarks.
In particular, we demonstrate that generative coherence (Shi et al., 2019) cannot be guaranteed for any of the existing approaches, if the information shared between modalities cannot be predicted in expectation across modalities.
Our ﬁndings are particularly relevant for applications on datasets with a relatively high degree of modality-speciﬁc variation, which is a typical characteristic of many real-world datasets (Baltrušaitis et al., 2019).
R ELATED WORK
First, to put multimodal VAEs into context, let us point out that there is a long line of research focused on learning multimodal generative models based on a wide variety of methods.
There are several notable generative models with applications on pairs of modalities (e.g., Ngiam et al., 2011;
Srivastava and Salakhutdinov, 2014; Wu and Goodman, 2019; Lin et al., 2021; Ramesh et al., 2021), as well as for the specialized task of image-to-image translation (e.g., Huang et al., 2018; Choi et al., 2018; Liu et al., 2019).
Moreover, generative models can use labels as side information (Ilse et al., 2019; Tsai et al., 2019; Wieser et al., 2020); for example, to guide the disentanglement of shared and modality-speciﬁc information (Tsai et al., 2019).
In contrast, multimodal VAEs do not require strong supervision and can handle a large and variable number of modalities efﬁciently.
They learn a joint distribution over multiple modalities, but also enable the inference of latent representations, as well as the conditional generation of missing modalities, given any subset of modalities (Wu and
Goodman, 2018; Shi et al., 2019; Sutter et al., 2021).
Multimodal VAEs are an extension of VAEs (Kingma and Welling, 2014) and they belong to the class of multimodal generative models with encoder-decoder architectures (Baltrušaitis et al., 2019).
The ﬁrst multimodal extensions of VAEs (Suzuki et al., 2016; Hsu and Glass, 2018; Vedantam et al., 2018) use separate inference networks for every subset of modalities, which quickly becomes intractable as the number of inference networks required grows exponentially with the number of modalities.
Starting with the seminal work of Wu and Goodman (2018), multimodal VAEs were developed as an efﬁcient method for multimodal learning.
In particular, multimodal VAEs enable the inference of latent representations, as well as the conditional generation of missing modalities, given any subset of input modalities.
Different types of multimodal VAEs were devised by decomposing the joint encoder as a product (Wu and Goodman, 2018), mixture (Shi et al., 2019), or mixture of products (Sutter et al., 2021) of unimodal encoders respectively.
A commonality between these approaches is the sub-sampling of modalities during training—a property we will use to deﬁne the family of mixture-based multimodal VAEs.
For the MMVAE and MoPoE-VAE, the sub-sampling is a direct consequence of deﬁning the joint encoder as a mixture distribution over different subsets of modalities.
Further, our analysis includes a special case of the MVAE without ELBO sub-sampling, which can be seen as another member of the family of mixture-based multimodal VAEs (Sutter et al., 2021).
The MVAE was originally proposed with “ELBO sub-sampling”, an additional training paradigm that was later found to result in an incorrect bound on the joint distribution (Wu and Goodman, 2019).
While this training paradigm is also based on the sub-sampling of modalities, the objective differs from mixture-based multimodal VAEs in that the MVAE does not reconstruct the missing modalities from the set of sub-sampled modalities.
Table 1 provides an overview of the different variants of mixture-based multimodal VAEs and the properties that one can infer from empirical results in previous works (Shi et al., 2019; 2021;
Sutter et al., 2021).
Most importantly, there appears to be a tradeoff between generative quality and generative coherence (i.e., the ability to generate semantically related samples across modalities).
Our work explains why the generative quality is worse for models that sub-sample modalities (Section 4) and shows that a tighter approximation of the joint distribution can be achieved without sub-sampling (Section 4.3).
Through systematic ablations, we validate the proposed theoretical limitations and showcase the tradeoff between generative quality and generative coherence (Section 5.1).
Our experiments also reveal that generative coherence cannot be guaranteed for more complex datasets than those used in previous benchmarks (Section 5.2).
M ULTIMODAL VAE S , IN DIFFERENT FLAVORS
Let X :=
{ X 1 , . . .
, X M } be a set of random vectors describing M modalities and let x := { x 1 , . . .
, x M } be a sample from the joint distribution p ( x 1 , . . .
, x M ) .
For conciseness, denote subsets of modalities by subscripts; for example, X { 1 , 3 } or x { 1 , 3 } respectively for modalities 1 and 3.
Throughout this work, we assume that all modalities are described by discrete random vectors (e.g., pixel values), so that we can assume non-negative entropy and conditional entropy terms.
Deﬁnitions for all required information-theoretic quantities are provided in Appendix A.
[Table]
3. T HE MULTIMODAL ELBO
Deﬁnition 1.
Let p θ ( z | x ) be a stochastic encoder [Table] parameterized by θ [Table]
that takes multiple modalities as input. Let q φ ( x | z ) be a variational decoder (for all modalities)
[Table]
parameterized by φ
[Table]
and let q ( z ) be a prior. The multimodal evidence lower bound (ELBO) on E p ( x ) [log p ( x )] is deﬁned as
[Equation] [Equation]
The multimodal ELBO (Deﬁnition 1), ﬁrst introduced by Wu and Goodman (2018), is the objective maximized by all multimodal VAEs and it forms a variational lower bound on the expected logevidence.
The ﬁrst term denotes the estimated log-likelihood of all modalities and the second term is the KL-divergence between the stochastic encoder and the prior.
We take an information-theoretic perspective using the variational information bottleneck (VIB) from Alemi et al.
and employ the standard notation used in multiple previous works (Alemi et al., 2017; Poole et al., 2019).
Similar to the latent variable model approach, the VIB derives the ELBO as a variational lower bound on the expected log-evidence, but, in addition, the VIB is a more general framework for optimization that allows us to reason about the underlying information-theoretic quantities of interest (for details on the VIB and its notation, please see Appendix B.1).
Note that the above deﬁnition of the multimodal ELBO requires that the complete set of modalities is available.
To overcome this limitation and to learn the inference networks for different subsets of modalities, existing models use different decompositions of the joint encoder, as summarized in
Table 1.
Recent work shows that existing models can be generalized by formulating the joint encoder as a mixture of products of experts (Sutter et al., 2021).
Analogously, in the following, we generalize existing models to deﬁne the family of mixture-based multimodal VAEs.
3. T HE FAMILY OF MIXTURE - BASED MULTIMODAL VAE S
Now we introduce the family of mixture-based multimodal VAEs, which subsumes the MMVAE,
MoPoE-VAE, and a special case of the MVAE without ELBO sub-sampling.
We ﬁrst deﬁne an encoder that generalizes the decompositions used by existing models:
Deﬁnition 2.
Let S =
{ ( A, ω A ) | A ⊆ { 1 , . . .
, M } , A (cid:54) = ∅ , ω
A ∈ [0 , 1] } be an arbitrary set of nonempty subsets A of modalities and corresponding mixture coefﬁcients ω A , such that (cid:80)
A ∈S ω A = 1 .
Deﬁne the stochastic encoder to be a mixture distribution: p S θ (
[Equation]
In the above deﬁnition and throughout this work, we write A ∈ S to abbreviate ( A, ω A ) ∈ S .
To deﬁne the family of mixture-based multimodal VAEs, we restrict the family of models optimizing the multimodal ELBO to the subfamily of models that use a mixture-based stochastic encoder.
Deﬁnition 3.
The family of mixture-based multimodal VAEs is comprised of all models that maximize the multimodal ELBO using a stochastic encoder p S
θ ( z | x ) that is consistent with Deﬁnition 2.
In particular, we deﬁne the family in terms of all models that maximize the following objective:
[Equation]
In Appendix B.2, we show that the objective L S ( x ; θ, φ ) is a lower bound on L ( x ; θ, φ ) (which makes it an ELBO) and explain how, for different choices of the set of subsets S , the objective L S ( x ; θ, φ ) relates to the objectives of the MMVAE, MoPoE-VAE, and MVAE without ELBO sub-sampling.
From a computational perspective, a characteristic of mixture-based multimodal VAEs is the subsampling of modalities during training, which is a direct consequence of deﬁning the encoder as a mixture distribution over subsets of modalities.
The sub-sampling of modalities can be viewed as the extraction of a subset x A ∈ x , where A indexes one subset of modalities that is drawn from the model-speciﬁc set of subsets S .
The only member of the family of mixture-based multimodal
VAEs that forgoes sub-sampling, deﬁnes a trivial mixture over a single subset, the complete set of modalities (Sutter et al., 2021).
M ODALITY SUB - SAMPLING LIMITS THE MULTIMODAL ELBO 4. A N INTUITION ABOUT THE PROBLEM
Before we delve into the details, let us illustrate how modality sub-sampling affects the likelihood estimation, and hence the multimodal ELBO.
Consider the likelihood estimation using the objective L S : (cid:88)
[Equation] [Equation]
) where A denotes a subset of modalities and
ω
A the respective mixture weight.
Crucially, the stochastic encoder p θ ( z | x A ) encodes a subset of modalities.
What seems to be a minute detail, can have a profound impact on the likelihood estimation, because the precise estimation of all modalities depends on information from all modalities.
In trying to reconstruct all modalities from incomplete information, the model can learn an inexact, average prediction; however, it cannot reliably predict modality-speciﬁc information, such as the background details in an image given a concise verbal description of its content.
In the following, we formalize the above intuition by showing that, in the presence of modalityspeciﬁc variation, modality sub-sampling enforces an undesirable upper bound on the multimodal
ELBO and therefore prevents a tight approximation of the joint distribution.
4. A FORMALIZATION OF THE PROBLEM
Theorem 1 states our main theoretical result, which describes a non-trivial limitation of mixture-based multimodal VAEs.
Our result shows that the sub-sampling of modalities enforces an undesirable upper bound on the approximation of the joint distribution when there is modality-speciﬁc variation in the data.
This limitation conﬂicts with the goal of modeling real-world multimodal data, which typically exhibits a considerable degree of modality-speciﬁc variation.
Theorem 1.
Each mixture-based multimodal VAE (Deﬁnition 3) approximates the expected logevidence up to an irreducible discrepancy ∆( X , S ) that depends on the model-speciﬁc mixture distribution S as well as on the amount of modality-speciﬁc information in X .
For the maximization of L S ( x ; θ, φ ) and every value of θ and φ , the following inequality holds:
[Equation]
where
[Equation]
In particular, the generative discrepancy is always greater than or equal to zero and it is independent of θ and φ and thus remains constant during the optimization.
A proof is provided in Appendix B.5 and it is based on Lemmas 1 and 2.
Theorem 1 formalizes the rationale that, in the general case, cross-modal prediction cannot recover information that is speciﬁc to the target modalities that are unobserved due to modality sub-sampling.
In general, the conditional entropy H ( X { 1 ,...,M }\ A | X A ) measures the amount of information in one subset of random vectors X { 1 ,...,M }\ A that is not shared with another subset X A .
In our context, the sub-sampling of modalities yields a discrepancy ∆( X , S ) that is a weighted average of conditional entropies H ( X { 1 ,...,M }\ A | X A ) of the modalities X { 1 ,...,M }\ A unobserved by the encoder given an observed subset X A .
Hence, ∆( X , S ) describes the modality-speciﬁc information that cannot be recovered by cross-modal prediction, averaged over all subsets of modalities.
Theorem 1 applies to the MMVAE, MoPoE-VAE, and a special case of the MVAE without ELBO subsampling, since all of these models belong to the class of mixture-based multimodal VAEs.
However, ∆( X , S ) can vary signiﬁcantly between different models, depending on the mixture distribution deﬁned by the respective model and on the amount of modality-speciﬁc variation in the data.
In the following, we show that without modality sub-sampling ∆( X , S ) vanishes, whereas for the MMVAE and MoPoE-VAE, ∆( X , S ) typically increases with each additional modality.
In Section 5, we provide empirical support for each of these theoretical statements.
4. I MPLICATIONS OF T HEOREM
First, we consider the case of no modality sub-sampling, for which it is easy to show that the generative discrepancy vanishes.
Corollary 1.
Without modality sub-sampling, ∆( X , S ) = 0 .
A proof is provided in Appendix B.6.
The result from Corollary 1 applies to the MVAE without
ELBO sub-sampling and suggests that this model should yield a tighter approximation of the joint distribution and hence a better generative quality compared to mixture-based multimodal VAEs that sub-sample modalities.
Note that this does not imply that a model without modality sub-sampling is superior to one that uses sub-sampling and that there can be an inductive bias that favors sub-sampling despite the approximation error it incurs.
Especially, Corollary 1 does not imply that the variational approximation is tight for the MVAE; for instance, the model can be underparameterized or simply misspeciﬁed due to simplifying assumptions, such as the PoE-factorization (Kurle et al., 2019).
Second, we consider how additional modalities might affect the generative discrepancy.
Corollarypredicts an increased generative discrepancy (and hence, a decline of generative quality) when we increase the number of modalities for the MMVAE and MoPoE-VAE.
Corollary 2 (informal) .
For the MMVAE and MoPoE-VAE, the generative discrepancy increases with each additional modality, if the new modality is sufﬁciently diverse.
A proof is provided in Appendix B.7.
The notion of diversity requires a more formal treatment of the underlying information-theoretic quantities, which we defer to Appendix B.7.
Intuitively, a new modality is sufﬁciently diverse, if it does not add too much redundant information with respect to the existing modalities.
In special cases when there is a lot of redundant information, ∆( X , S ) can decrease given an additional modality, but it does not vanish in any one of these cases.
Only if there is very little modality-speciﬁc information in all modalities, we have ∆( X , S ) → 0 for the MMVAE and MoPoE-VAE.
This condition requires modalities to be extremely similar, which does not apply to most multimodal datasets, where ∆( X , S ) typically represents a large part of the total variation.
In summary, Theorem 1 formalizes how the family of mixture-based multimodal VAEs is fundamentally limited for the task of approximating the joint distribution, and Corollaries 1 and 2 connect this result to existing models—the MMVAE, MoPoE-VAE, and MVAE without ELBO sub-sampling.
We now turn to the experiments, where we present empirical support for the limitations described by
Theorem 1 and its Corollaries.
E XPERIMENTS
Figure 1 presents the three considered datasets.
PolyMNIST (Sutter et al., 2021) is a simple, synthetic dataset with ﬁve image modalities that allows us to conduct systematic ablations.
Translated-PolyMNIST is a new dataset that adds a small tweak—the downscaling and random translation of digits—to demonstrate the limitations of existing methods when shared information cannot be predicted in expectation across modalities.
Finally, Caltech Birds (CUB; Wah et al., 2011;
Shi et al., 2019) is used to validate the limitations on a more realistic dataset with two modalities, images and captions.
Please note that we use CUB with real images and not the simpliﬁed version based on precomputed ResNet-features that was used in Shi et al.
and Shi et al.
For a more detailed description of the three considered datasets, please see Appendix C.1.
[Figure]
In total, more than 400 models were trained, requiring approximately 1.5 GPU years of compute on a single NVIDIA GeForce RTX 2080 Ti GPU.
For the experiments in the main text, we use the publicly available code from Sutter et al.
(2021) and
in Appendix C.3 we also include ablations using the publicly available code from Shi et al.
(2019), which implements importance sampling and alternative
ELBO objectives.
To provide a fair comparison across methods, we use the same architectures and similar capacities for all models.
For each unimodal VAE, we make sure to decrease the capacity by reducing the latent dimensionality proportionally with respect to the number of modalities.
Additional information on architectures, hyperparameters, and evaluation metrics is provided in Appendix C.
5. T HE GENERATIVE QUALITY GAP
We assume that an increase in the generative discrepancy ∆( X , S ) is associated with a drop of generative quality.
However, we want to point out that there can also be an inductive bias that favors modality sub-sampling despite the approximation error that it incurs.
In fact, our experiments reveal a fundamental tradeoff between generative quality and generative coherence when shared information can be predicted in expectation across modalities.
We measure generative quality in terms of Fréchet inception distance (FID; Heusel et al., 2017), a standard metric for evaluating the quality of generated images.
Lower FID represents better generative quality and the values typically correlate well with human perception (Borji, 2019).
In addition, in Appendix C we provide log-likelihood values, as well as qualitative results for all modalities including captions, for which FID cannot be computed.
Figure 2 presents the generative quality across a range of β values.
To relate different methods, we compare models with the best FID respectively, because different methods can reach their optima at different β values.
As described by Theorem 1, mixture-based multimodal VAEs that sub-sample modalities (MMVAE and MoPoE-VAE) exhibit a pronounced generative quality gap compared to unimodal VAEs.
When we compare the best models, we observe a gap of more than 60 points on both
PolyMNIST and Translated-PolyMNIST, and about 30 points on CUB images.
Qualitative results (Figure 9 in Appendix C.3) conﬁrm that this gap is clearly visible in the generated samples and that it applies not only to image modalities, but also to captions.
In contrast, the MVAE (without ELBO sub-sampling) reaches the generative quality of unimodal VAEs, which is in line with our theoretical result from Corollary 1.
For completeness, in Appendix C.3, we also report joint log-likelihoods, latent classiﬁcation performance, as well as additional FIDs for all modalities.
Figure 3 examines how the generative quality is affected when we vary the number of modalities.
Notably, for the MMVAE and MoPoE-VAE, the generative quality deteriorates almost continuously with the number of modalities, which is in line with our theoretical result from Corollary 2.
Interestingly, for the MVAE, the generative quality on Translated-PolyMNIST also decreases slightly, but the change is comparatively small.
Figure 11 in Appendix C.3, shows a similar trend even when we control for modality-speciﬁc differences by generating PolyMNIST using the same background image for all modalities.
[Figure]
[Figure]
In summary, the results from Figure 2 and Figure 3 provide empirical support for the existence of a generative quality gap between unimodal and mixture-based multimodal VAEs that sub-sample modalities.
The results verify that the approximation of the joint distribution improves for models without sub-sampling, which manifests in better generative quality.
In contrast, the gap increases disproportionally with each additional modality for both the MMVAE and MoPoE-VAE.
Hence, the presented results support all of the theoretical statements from Sections 4.2 and 4.3.
5. L ACK OF GENERATIVE COHERENCE ON MORE COMPLEX DATA
Apart from generative quality, another desired criterion (Shi et al., 2019; Sutter et al., 2020) for an effective multimodal generative model is generative coherence , which measures a model’s ability to generate semantically related samples across modalities.
To be consistent with Sutter et al.
(2021), we compute the leave-one-out coherence (see Appendix C.2), which means that the input to each model consists of all modalities except the one that is being conditionally generated.
On CUB, we resort to a qualitative evaluation of coherence, because there is no ground truth annotation of shared factors and the proxies used in Shi et al.
and Shi et al.
(2021) do not yield meaningful estimates when applied to the conditionally generated images from models that were trained on real images.
In terms of generative coherence, Figure 4 reveals that the positive results from previous work do not translate to more complex datasets.
As a baseline, for PolyMNIST (Figure 4a) we replicate the coherence results from Sutter et al.
(2021) for a range of β values.
Consistent with previous work (Shi et al., 2019; 2021; Sutter et al., 2020; 2021), we ﬁnd that the MMVAE and MoPoE-VAE exhibit
[Figure]
superior coherence compared to the MVAE.
Though, it was not apparent from previous work that
MVAE’s coherence can improve signiﬁcantly with increasing β values, which can be of independent interest for future work.
On Translated-PolyMNIST (Figure 4b), the stark decline of all models makes it evident that coherence cannot be guaranteed when shared information cannot be predicted in expectation across modalities.
Our qualitative results (Figure 10 in Appendix C.3) conﬁrm that not a single multimodal VAE is able to conditionally generate coherent examples and, for the most part, not any digits at all.
To verify that the lack of coherence is not an artifact of our implementation, we have checked that the encoders and decoders have sufﬁcient capacity such that digits show up in most self-reconstructions.
On CUB (Figure 4c), for which coherence cannot be computed, the qualitative results for conditional generation verify that none of the existing approaches generates images that are both of sufﬁciently high quality and coherent with respect to the given caption.
Overall, the negative results on Translated-PolyMNIST and CUB showcase the limitations of existing approaches when applied to more complex datasets than those used in previous benchmarks.
D ISCUSSION
Implications and scope Our experiments lend empirical support to the proposed theoretical limitations of mixture-based multimodal VAEs.
On both synthetic and real data, our results showcase the generative limitations of multimodal VAEs that sub-sample modalities.
However, our results also reveal that none of the existing approaches (including those without sub-sampling) fulﬁll all desired criteria (Shi et al., 2019; Sutter et al., 2020) of an effective multimodal generative model.
More broadly, our results showcase the limitations of existing VAE-based approaches for modeling weakly-supervised data in the presence of modality-speciﬁc information, and in particular when shared information cannot be predicted in expectation across modalities.
The Translated-PolyMNIST dataset demonstrates this problem in a simple setting, while the results on CUB conﬁrm that similar issues can be expected on more realistic datasets.
For future work, it would be interesting to generate simulated data where the discrepancy ∆( X , S ) can be measured exactly and where it is gradually increased by an adaptation of the dataset in a way that increases only the modality-speciﬁc variation.
Furthermore, it is worth noting that Theorem 1 applies to all multimodal VAEs that optimize Equa- tion (2), which is a lower bound on the multimodal ELBO for models that sub-sample modalities.
Our theory predicts the same discrepancy for models that optimize a tighter bound (e.g., via Equation (28)), because the discrepancy ∆( X , S ) derives from the likelihood term, which is equal for Equations (2) and (28).
In Appendix C.3 we verify that the discrepancy can also be observed for the MMVAE with the original implementation from Shi et al.
(2019) that uses a tighter bound.
Further analysis of the different bounds can be an interesting direction for future work.
Model selection and generalization
Our results raise fundamental questions regarding model selection and generalization, as generative quality and generative coherence do not necessarily go hand in hand.
In particular, our experiments demonstrate that FIDs and log-likelihoods do not reﬂect the problem of lacking coherence and without access to ground truth labels (on what is shared between modalities) coherence metrics cannot be computed.
As a consequence, it can be difﬁcult to perform model selection on more realistic multimodal datasets, especially for less interpretable types of modalities, such as DNA sequences.
Hence, for future work it would be interesting to design alternative metrics for generative coherence that can be applied when shared information is not annotated.
For the related topic of generalization, it can be illuminating to consider what would happen, if one could arbitrarily “scale things up”.
In the limit of inﬁnite i.i.d.
data, perfect generative coherence could be achieved by a model that memorizes the pairwise relations between training examples from different modalities.
However, would this yield a model that generalizes out of distribution (e.g., under distribution shift)?
We believe that for future work it would be worthwhile to consider out-of-distribution generalization performance (e.g., Montero et al., 2021)
in addition to generative quality and coherence.
Limitations In general, the limitations and tradeoffs presented in this work apply to a large family of multimodal VAEs, but not necessarily to other types of generative models, such as generative adversarial networks (Goodfellow et al., 2014).
Where current VAEs are limited by the reconstruction of modality-speciﬁc information, other types of generative models might offer less restrictive objectives.
Similar to previous work, we have only considered models with simple priors, such as
Gauss and Laplace distributions with independent dimensions.
Further, we have not considered models with modality-speciﬁc latent spaces, which seem to yield better empirical results (Hsu and
Glass, 2018; Sutter et al., 2020; Daunhawer et al., 2020), but currently lack theoretical grounding.
Modality-speciﬁc latent spaces offer a potential solution to the problem of cross-modal prediction by providing modality-speciﬁc context from the target modalities to each decoder.
However, more work is required to establish guarantees for the identiﬁability and disentanglement of shared and modality-speciﬁc factors, which might only be possible for VAEs under relatively strong assumptions (Locatello et al., 2019; 2020; Gresele et al., 2019; von Kügelgen et al., 2021).
C ONCLUSION
In this work, we have identiﬁed, formalized, and demonstrated several limitations of multimodal
VAEs.
Across different datasets, this work revealed a signiﬁcant gap in generative quality between unimodal and mixture-based multimodal VAEs.
We showed that this apparent paradox can be explained by the sub-sampling of modalities, which enforces an undesirable upper bound on the multimodal ELBO and therefore limits the generative quality of the respective models.
While the sub-sampling of modalities allows these models to learn the inference networks for different subsets of modalities efﬁciently, there is a notable tradeoff in terms of generative quality.
Finally, we studied two failure cases—Translated-PolyMNIST and CUB—that demonstrate the limitations of multimodal
VAEs when applied to more complex datasets than those used in previous benchmarks.
For future work, we believe that it is crucial to be aware of the limitations of existing methods as a ﬁrst step towards developing new methods that achieve more than incremental improvements for multimodal learning.
We conjecture that there are at least two potential strategies to circumvent the theoretical limitations of multimodal VAEs.
First, the sub-sampling of modalities can be combined with modality-speciﬁc context from the target modalities.
Second, cross-modal reconstruction terms can be replaced with less restrictive objectives that do not require an exact prediction of modalityspeciﬁc information.
Finally, we urge future research to design more challenging benchmarks and to compare multimodal generative models in terms of both generative quality and coherence across a range of hyperparameter values, to present the tradeoff between these metrics more transparently. A CKNOWLEDGEMENTS
ID and KC were supported by the SNSF grant #200021_188466 .
Special thanks to Alexander
Marx, Nicolò Ruggeri, Maxim Samarin, Yuge Shi and Mario Wieser for helpful discussions and/or feedback on the manuscript.
R EPRODUCIBILITY S TATEMENT
For all theoretical statements, we provide detailed derivations and state the necessary assumptions.
For our main theoretical results, we present empirical support on both synthetic and real data.
To ensure empirical reproducibility, the results of each experiment and every ablation were averaged over multiple seeds and are reported with standard deviations.
All of the used datasets are either public or can be generated from publicly available resources using the code that we provide in the supplementary material.
Information about implementation details, hyperparameter settings, and evaluation metrics are included in Appendix C
A D EFINITIONS
Let X , Y , and Z denote the support sets of three discrete random vectors X , Y , and Z respectively.
Let p X ( x ) , p Y ( y ) , and p Z ( z ) denote the respective marginal distributions and note that we will leave out the subscripts (e.g., p ( x ) instead of p X ( x ) ) when it is clear from context which distribution we are referring to.
Analogously, we write shorthand p ( y | x ) for the conditional distribution of Y given X and p ( x , y ) for the joint distribution of X and Y .
The entropy of X is deﬁned as
[Equation]
The conditional entropy of X given Y is deﬁned as
[Equation]
The joint entropy of X and Y is deﬁned as
[Equation]
The Kullback-Leibler divergence of the discrete probability distribution P from the discrete probability distribution Q is deﬁned as
[Equation]
assuming that P and Q are deﬁned on the same support set X .
The cross-entropy of the discrete probability distribution Q from the discrete probability distribution
P is deﬁned as
[Equation]
assuming that P and Q are deﬁned on the same support set X .
The mutual information of X and Y is deﬁned as
[Equation]
The conditional mutual information of X and Y given Z is deﬁned as
[Equation]
Recall that we assume discrete random vectors (e.g., pixel values) and therefore can assume nonnegative entropy, conditional entropy and conditional mutual information terms (Cover and Thomas, 2012).
For continuous random variables, all of the above sums can be replaced with integrals.
The only information-theoretic quantities for which in this work we use continuous random vectors are the KL-divergence and mutual information, both of which are always non-negative.
B P ROOFS B. I NFORMATION - THEORETIC DERIVATION OF THE MULTIMODAL ELBO
Proposition 1 relates the multimodal ELBO (Deﬁnition 1) to the expected log-evidence, the quantity that is being approximated by all likelihood-based generative models including VAEs.
The derivation is based on a straightforward extension of the variational information bottleneck (VIB; Alemi et al., 2017).
We include the result mainly for the purpose of illustration—to clarify the notation, as well as the relation between the multimodal ELBO and the underlying information-theoretic quantities of interest: the entropy, conditional entropy, and mutual information.
Notation Readers who are familiar with latent variable models, but may be less familiar with the information-theoretic perspective on VAEs, please keep in mind the following notational differences.
In contrast to the latent variable model perspective, which deﬁnes a variational posterior (typically denoted by the letter q ) and a stochastic decoder (typically denoted by the letter p ), the VIB deﬁnes a stochastic encoder p θ ( z | x ) and variational decoder q φ ( x | z ) .
Moreover, the VIB makes no assumptions about the true posterior.
Also note that latent variable models tend to write the ELBO with respect to the log-evidence log p ( x ) , but information-theoretic approaches write the ELBO with respect to the expected log-evidence E p ( x ) [log p ( x )] ; though, it is still assumed that the estimation of the ELBO is based on a ﬁnite sample from p ( x ) .
Proposition 1.
The multimodal ELBO forms a variational lower bound on the expected log-evidence:
[Equation]
Proof.
First, notice that the expected log-evidence is equal to the negative entropy − H ( X ) =
E p ( x ) [log p ( x )] .
Given any random variable Z , the entropy can be decomposed into conditional entropy and mutual information terms: H ( X ) =
H ( X | Z )
+ I ( X ; Z ) .
The expected log-evidence relates to the multimodal ELBO as follows:
[Equation]
where the inequality follows from the variational approximations of the respective terms.
As in Alemi et al.
[Equation]
we can use the following variational bounds.
For the conditional entropy
[Equation]
we have
[Equation]
where q φ ( x | z ) is a variational decoder that is parameterized by φ .
For the mutual information [Equation] we have
[Equation]
where q ( z ) is a prior.
Hence, the multimodal ELBO forms a variational lower bound on the expected log-evidence:
[Equation]
where
[Equation]
denotes the (non-negative) variational approximation gap.
B. R ELATION BETWEEN THE DIFFERENT OBJECTIVES
Proposition 2 relates the multimodal ELBO L from Deﬁnition 1 to the objective L S , which is a general formulation of the objective maximized by all mixture-based multimodal VAEs.
Compared to previous mixture-based formulations (Shi et al., 2019; Sutter et al., 2020), our formulation is more general in that it allows for arbitrary subsets with non-uniform mixture coefﬁcients.
Further, the derivation quantiﬁes the approximation gap between L and L S , where the latter corresponds to the objectives that are actually being optimized in the implementations of the MMVAE, MoPoE-VAE, and MVAE without sub-sampling.
Proposition 2.
For every stochastic encoder p S θ ( z | x ) that is consistent with Deﬁnition 2, the following inequality holds:
[Equation]
Proof.
Recall the multimodal ELBO from Deﬁnition 1:
[Equation]
For the encoder p θ ( z | x )
[Equation]
plug in the mixture-based encoder p S θ ( z | x
[Equation]
= (cid:80)
[Equation]
from Deﬁnition 2 and re-write as follows:
[Equation]
[Equation]
In Equation (31)
[Equation]
CE ( p [Equation] q ) denotes the cross-entropy between distributions p and q .
For Equation (32)
[Equation]
decompose both cross-entropy terms using CE ( p, q )
=
H ( p ) + D KL ( p || q ) and notice that the respective entropy terms cancel out.
The inequality (Equation (33)) follows from the non-negativity of the KL-divergence.
This concludes the proof that L S ( x ; θ, φ ) forms a lower bound on L ( x ; θ, φ ) .
Objectives of individual models Sutter et al.
(2021) already showed that Equation (28) subsumes the objectives of the MMVAE, MoPoE-VAE, and MVAE without ELBO sub-sampling.
However, in their actual implementation, all of these methods take the sum out of the KL-divergence term (e.g., see Shi et al., 2019, Equation 3), which corresponds to the objective L S .
To see how L S recovers the objectives of the individual models, simply plug in the model-speciﬁc deﬁnition of S into Equation (33) and use uniform mixture coefﬁcients
ω A = 1 / |S| for all subsets.
For the MVAE without ELBO sub-sampling, S is comprised of only one subset, the complete set of modalities { x 1 , . . .
, x M } .
For the MMVAE, S is comprised of the set of unimodal subsets {{ x 1 } , . .
, { x M }} .
For the MoPoE-VAE, S is comprised of the powerset P ( M ) \ {∅} .
Further implementation details, such as importance sampling and ELBO sub-sampling, are discussed in Appendix C.3.
B. O BJECTIVE L S IS A SPECIAL CASE OF THE VIB
Lemma 1.
L S ( x ; θ, φ ) is a special case of the variational information bottleneck (VIB) objective
[Equation]
where the encoding Z A = f ψ ( X A ) is a function of a subset X A , the terms H ψ ( X | Z A ) and
I ψ ( X A ; Z A ) denote variational upper bounds of H ( X | Z A ) and I ( X A ; Z A ) respectively, and ψ summarizes the parameters of these variational estimators.
Proof.
We start from L S , the objective optimized by all mixture-based multimodal VAEs.
Recall from Deﬁnition 3:
[Equation]
Each term within the sum is comprised of two terms: ( i ) the log-likelihood estimation based on a variational decoder q φ ( x | z ) ; ( ii ) the regularization of the stochastic encoder p θ ( z | x A ) with respect to a variational prior q ( z ) .
The sampled encoding z ∼ p θ ( z | x A ) can be viewed as the output of a function Z A = f θ ( X A ) of a subset of modalities.
To see the relation to the underlying information terms H ( X | Z A ) and I ( X A ; Z A ) , we undo the variational approximation for ( i ) and ( ii ) by re-introducing the unobserved ground truth decoder p ( x | z ) and the ground truth prior p ( z ) .
For ( i ) , we have
[Equation]
For ( ii ) , we have
[Equation]
Since L S ( x ; θ, φ ) is being maximized, ( i ) is being maximized, while ( ii ) is being minimized.
The maximization of ( i ) is equal to the minimization of a variational upper bound on H ( X | Z A ) .
Similarly, the minimization of ( ii ) is equal to the minimization of a variational upper bound on
I ( X A ; Z A ) .
Hence, we have established that L S ( x ; θ, φ ) is a special case of the more general VIB objective (Equation (35)) where the information terms are estimated with a mixture-based multimodal
VAE that is parameterized by ψ = { θ, φ } .
B. D ECOMPOSITION OF THE CONDITIONAL ENTROPY FOR SUBSETS OF MODALITIES
Lemma 2.
Let X A ⊆
X be some subset of modalites.
If Z A = f ( X A ) , where f is some function of the subset X A , then the following equality holds:
[Equation] [Equation]
Proof.
When Z A is a function of a subset X A ⊆
X , we have the Markov chain
Z A ← X A −− X { 1 ,...,M }\ A , since Z A is a function of the (observed) subset of modalities and depends on the remaining (unobserved) modalities only through X A .
We can re-write H ( X | Z A ) as follows:
[Equation]
Equation (44) applies the deﬁnition of the conditional mutual information.
Equation (45) is based on the conditional independence X | =
Z A | X A implied by the Markov chain.
Equation (46) removes the “known” information that we condition on.
Finally, Equation (47) follows from X A ⊆ X , which implies that I ( X ; X A )
=
H ( X A ) and I ( X ; X A | Z A ) =
H ( X A | Z A ) .
B. P ROOF OF T HEOREM
Theorem 1.
Each mixture-based multimodal VAE (Deﬁnition 3) approximates the expected logevidence up to an irreducible discrepancy ∆( X , S ) that depends on the model-speciﬁc mixture distribution S as well as on the amount of modality-speciﬁc information in X .
For the maximization of L S ( x ; θ, φ ) and every value of θ and φ , the following inequality holds:
[Equation]
where
[Equation]
In particular, the generative discrepancy is always greater than or equal to zero and it is independent of θ and φ and thus remains constant during the optimization.
Proof.
Lemma 1 shows that all mixture-based multimodal VAEs approximate the expected logevidence via the more general VIB objective
[Equation]
where the encoding Z A = f ψ ( X A ) is a function of a subset X A ⊆ X .
The fact that Z A is a function of a subset , permits the following decomposition of the conditional entropy (see Lemma 2):
[Equation]
In particular, Equation (49) holds for every Z A
= f ψ ( X A ) and thus for every value ψ .
Further, notice that H ( X { 1 ,...,M }\ A | X A ) is independent of the learned encoding Z A and thus remains constant during the optimization with respect to ψ .
Hence, for every value ψ , the following inequality holds:
[Equation]
which means that the minimization of H ψ ( X | Z A ) is lower-bound by H ( X { 1 ,...,M }\ A | X A ) , even if H ψ ( X | Z A ) is a tight estimator of H ( X | Z A ) .
Analogously, for the optimization of the VIB objective (Equation (48)), for every value ψ , the following inequality holds:
[Equation]
where ∆( X , S ) is independent of ψ and thus remains constant during the optimization.
Consequently, ∆( X , S ) represents an irreducible error for the optimization of the VIB objective.
For mixture-based multimodal VAEs, Lemma 1 shows that L S ( x ; θ, φ ) is a special case of the VIB objective with ψ
= ( θ, φ ) .
Hence, for every value of θ and φ , the following inequality holds:
[Equation]
The exact value of ∆( X , S ) depends on the deﬁnition of the mixture distribution S , as well as on the amount of modality-speciﬁc variation in the data.
In particular, ∆( X , S ) > 0 , if there is any subset
A ∈ S with ω A > 0 for which H
( X { 1 ,...,M }\ A | X A ) > 0 .
B. P ROOF OF C OROLLARY
Corollary 1.
Without modality sub-sampling, ∆( X , S ) = 0 .
Proof.
Without modality sub-sampling, S is comprised of only one subset, the complete set of modalities { 1 , . . .
, M } , and therefore X
A = X and X { 1 ,...,M }\ A = ∅ .
It follows that ∆( X , S ) =
H ( X { 1 ,...,M }\ A | X A ) =
H ( ∅ | X ) = 0 , since the conditional entropy of the empty set is zero.
B. P ROOF OF C OROLLARY
Corollary 2.
For the MMVAE and MoPoE-VAE, the generative discrepancy increases given an additional modality X M +1 , if the new modality is sufﬁciently diverse in the following sense:
[Equation]
where S denotes the model-speciﬁc mixture distribution over the set of subsets of modalities given modalities X 1 , .
, X M and S + is the respective mixture distribution over the extended set of subsets of modalities given X 1 , . . .
, X M +1 .
Proof.
Let X M +1 be the new modality, let X
+
:= { X 1 , . . .
, X M +1 } denote the extended set of modalities, and let S + denote the new mixture distribution over subsets given X + .
Note that all subsets from S are still contained in S + , but that S + contains new subsets in addition to those in S .
Further, due to the re-weighting of mixture coefﬁcients, S + can have different mixture coefﬁcients for the subsets it shares with S .
We denote by S − :=
{ ( A, ω + A ) ∈ S
+ : A (cid:54)∈ S} the set of new subsets and let ω + A denote the new mixture coefﬁcients, where typically ω A (cid:54) =
ω
+ A due to the re-weighting.
We are interested in the change of the generative discrepancy, when we add modality X M +1 : [Equation] [Equation]
[Equation]
Re-write the right hand side in terms of subsets that are contained in both S and S + and subsets that are only contained in S + .
For this, we decompose the ﬁrst term as follows
[Equation]
where the last equation follows from
[Equation]
We can use the decomposition from Equation (63) to re-write the right hand side of Equation (59) by collecting the corresponding terms for H ( X { 1 ,...,M }\ A | X A ) :
[Equation]
Notice that in Equation (66) only the ﬁrst term can be negative, due to the re-weighting of mixture coefﬁcients for terms that do not contain X M +1 .
Hence, in the general case, the generative discrepancy can only decrease, if the mixture coefﬁcients change in such a way that the ﬁrst term in Equation (66) dominates the other two terms.
For the relevant special case of uniform mixture weights, which applies to both the MMVAE and
MoPoE-VAE, we can further decompose Equation (66) into ( i ) information shared between X and
X M +1 , and ( ii ) information that is speciﬁc to X or X M +1 .
Using uniform mixture coefﬁcients
ω A = 1 |S| and ω + A = 1 |S + | for all subsets, we can factor out the coefﬁcients and re-write Equation (66) as follows:
[Equation]
where the second term already denotes information that is speciﬁc to X M +1 .
Hence, we decompose the ﬁrst and last terms corresponding to ( i ) and ( ii ) .
For the ﬁrst term from Equation (67), we have
[Equation] [Equation]
For the last term from Equation (67), we have
[Equation]
where we can further decompose
[Equation]
Collecting all corresponding terms from Equations (69), (71) and (74)
[Equation]
we can re-write Equation (67) as follows:
[Equation]
For both the MMVAE and MoPoE, the ﬁrst and last terms cancel out, which can see by plugging in the respective deﬁnitions of S into the above equation.
Recall that for the MMVAE, S is comprised of the set of unimodal subsets {{ x 1 } , . .
. , { x M }} and thus S + is comprised of {{ x 1 } , . .
, { x M +1 }} .
For the MoPoE-VAE, S is comprised of the powerset P ( M ) \ {∅} and thus S + is comprised of the powerset P ( M + 1) \ {∅} .
Hence, for the MMVAE and MoPoE-VAE, we have shown that ∆( X + , S + ) − ∆( X , S ) is equal to the following expression:
[Equation]
where the information is decomposed into:
( i ) information shared between X and X M +1 (term (80)), and ( ii ) information that is speciﬁc to X or
X M +1 (the ﬁrst and
second
terms in (
) respectively),
and where only ( i ) can be negative since |S + | > |S| .
This concludes the proof of Corollary 2, showing that ∆( X + , S + ) − ∆( X , S ) > 0 , if X M +1 is sufﬁciently diverse in the sense that ( ii ) >
( i ) .
C E XPERIMENTS C. D ESCRIPTION OF THE DATASETS PolyMNIST
The PolyMNIST dataset, introduced in Sutter et al.
(2021), combines the MNIST dataset (LeCun et al., 1998) with crops from ﬁve different background images to create ﬁve synthetic image modalities.
Each sample from the data is a set of ﬁve MNIST images (with digits of the same class) overlayed on 28 × 28 crops from ﬁve different background images.
Figure 1a showssamples from the PolyMNIST dataset; each column represents one sample and each row represents one modality.
The dataset provides a convenient testbed for the evaluation of generative coherence, because by design only the digit information is shared between modalities.
Translated-PolyMNIST
This new dataset is conceptually similar to PolyMNIST in that a digit label is shared between ﬁve synthetic image modalities.
The difference is that in the creation of the dataset, we change the size and position of the digit, as shown in Figure 1b.
Technically, instead of overlaying a full-sized 28 × 28 MNIST digit on a patch from the respective background image, we downsample the MNIST digit by a factor of two and place it at a random ( x, y )
-coordinate
within the 28 × 28 background patch.
Conceptually, these transformations leave the shared information between modalities (i.e., the digit label) unaffected and only serve to make it more difﬁcult to predict the shared information across modalities on expectation.
Caltech Birds (CUB)
The extended CUB dataset from Shi et al.
(2019) is comprised of two modalities, images and captions.
Each image from Caltech-Birds (CUB-200-2011 Wah et al., 2011) is coupled with 10 crowdsourced descriptions of the respective bird.
Figure 1c shows ﬁve samples from the dataset.
It is important to note that we use the CUB dataset with real images , instead of the simpliﬁed version based on precomputed ResNet-features that was used in Shi et al.
C. I MPLEMENTATION DETAILS
Our experiments are based on the publicly available code from Sutter et al.
(2021), which already provides an implementation of PolyMNIST.
A notable difference in our implementation is that we employ ResNet architectures, because we found that the previously used convolutional neural networks did not have sufﬁcient capacity for the more complex datasets we use.
For internal consistency, we use ResNets for PolyMNIST as well.
We have veriﬁed that there is no signiﬁcant difference compared to the results from Sutter et al.
(2021) when we change to ResNets.
Hyperparameters All models were trained using the Adam optimizer (Kingma and Ba, 2015) with learning rate 5e-4 and a batch size of 256.
For image modalities we estimate likelihoods using Laplace distributions and for captions we employ one-hot categorical distributions.
Models were trained for 500, 1000, and 150 epochs on PolyMNIST, Translated-PolyMNIST, and CUB respectively.
Similar to previous work, we use Gaussian priors and a latent space with 512 dimensions for PolyMNIST and 64 dimensions for CUB.
For a fair comparison, we reduce the latent dimensionality of unimodal VAEs proportionally (wrt.
the number of modalities) to control for capacity.
For the β -ablations, we use β ∈ { 3e-4, 3e-3, 3e-1, 1, 3, 9 } and, in addition, 32 for CUB.
Evaluation metrics For the evaluation of generative quality , we use the Fréchet inception distance (FID; Heusel et al., 2017), a standard metric for evaluating the quality of generated images.
In
Appendix C.3, we also provide log-likelihoods and qualitative results for both images and captions.
To compute generative coherence , we adopt the deﬁnitions from previous works (Shi et al., 2019;
Sutter et al., 2021).
Generative coherence requires annotation on what is shared between modalities; for example, in both PolyMNIST and Translated-PolyMNIST the digit label is shared by design.
For a single generated example ˆ x m ∼ q φ ( x m | z ) from modality m , the generative coherence is computed as the following indicator:
[Equation]
where y is a ground-truth class label and g m is a pretrained classiﬁer (learned on the training data from modality m ) that outputs a predicted class label.
To compute the conditional coherence accuracy , we average the coherence values over a set of N conditionally generated examples, where N is typically the size of the test set.
In particular, when ˆ x m ∼ q φ ( x m | z ) is conditionally generated from z ∼ p θ ( z | x A )
such that A
= { 1 , . . .
, M } \ m , the metric is speciﬁed as the leave-one-out conditional coherence accuracy , because the input consists of all modalities except the one that is being generated.
When it is clear from context which metric is used, we refer to the (leave-oneout) conditional coherence accuracy simply as generative coherence.
For PolyMNIST, we use the pretrained digit classiﬁers that are provided in the publicly available code from Sutter et al.
(2021) and
for Translated-PolyMNIST we train the classiﬁers from scratch with the same architectures that are used for the VAE encoders.
Notably, the new pretrained digit classiﬁers have a classiﬁcation accuracy between 93.5–96.9% on the test set of the respective modality, which means that it is possible to predict the digits fairly well with the given architectures.
C. A DDITIONAL EXPERIMENTAL RESULTS
Linear classiﬁcation Shi et al.
(2019) propose linear classiﬁcation as a measure of latent factorization, to judge the quality of learned representations and to assess how well the information decomposes into shared and modality-speciﬁc features.
Figure 6 shows the linear classiﬁcation accuracy on the learned representations.
The results suggest that not only does the generative coherence decline when we switch from PolyMNIST to Translated-PolyMNIST, but also the quality of the learned representations.
While a low classiﬁcation accuracy does not imply that there is no digit information encoded in the latent representation (after all, digits show up in most self-reconstructions), the result demonstrates that a linear classiﬁer cannot extract the digit information.
Log-likelihoods and qualitative results Figure 7 shows the generative quality in terms of joint log-likelihoods.
We observe a similar ranking of models as with FID, but we notice that the gap between MVAE and MoPoE-VAE appears less pronounced.
The reason for this discrepancy is that, to be consistent with Sutter et al.
(2021), we estimate joint log-likelihoods given all modalities—a procedure that resembles reconstruction more than it does unconditional generation.
It can be of independent interest that log-likelihoods might overestimate the generative quality for unconditional generation for certain types of models.
Qualitative results for unconditional generation (Figure 9) support the hypothesis that the presented log-likelihoods do not reﬂect the visible lack of generative quality for the MoPoE-VAE.
Further, qualitative results for conditional generation (Figure 10) indicate a lack of diversity for both the MMVAE and MoPoE-VAE: even though we draw different samples from the posterior, the respective conditionally generated samples (i.e., the ten samples along each column) show little diversity in terms of backgrounds or writing styles.
Repeated modalities To check if the generative quality gap is also present when modalities have similar modality-speciﬁc variation, we use PolyMNIST with “repeated” modalities generated from the same background image (illustrated in Figure 5).
We vary the number of modalities from 2 to 5, but in contrast to the results from Figure 3, we now use repeated modalities.
Figure 11 conﬁrms that the generative quality of both the MVAE and MoPoE-VAE deteriorates with each additional modality, even in this simpliﬁed setting with repeated modalities.
In comparison, the generative quality of the MVAE is much closer to the unimodal VAE for any number of modalities.
These results lend further support to the theoretical statements from Corollaries 1 and 2.
MMVAE with the ofﬁcial implementation The empirical results of the MMVAE in Section 5 are based on a simpliﬁed version of the model that was proposed by Shi et al.
In particular, we use the re-implementation from Sutter et al.
(2021), which optimizes the standard ELBO and not the doubly reparameterized ELBO gradient estimator (DReG, Tucker et al., 2019) with importance sampling that is used in the ofﬁcial implementation from Shi et al.
Further, the re-implementation does not parameterize the prior but uses a ﬁxed, standard normal prior instead.
To verify that these implementation differences do not affect the core results—the generative quality gap and the lack of coherence—we conducted experiments using the MMVAE with the ofﬁcial implementation from Shi et al.
Figure 12 shows the β -ablation for PolyMNIST and it conﬁrms that there is still a clear gap in generative quality between the unimodal VAE and the
MMVAE when we use the ofﬁcial implementation.
For Translated-PolyMNIST (not shown) the results are similar; in particular, we have veriﬁed that generative coherence for cross generation is random, even if we limit the dataset to two modalities.
MVAE with ELBO sub-sampling
For the MVAE, Wu and Goodman (2018) introduce ELBO sub-sampling as an additional training strategy to learn the inference networks for different subsets of modalities.
In our notation, ELBO sub-sampling can be described by the following objective:
[Equation]
where S denotes some set of subsets of modalities.
Wu and Goodman (2018) experiment with different choices for S , but throughout all of their experiments they use at least the set of unimodal subsets {{ x 1 } , . . .
, { x M }} , which yields the following objective:
[Equation]
It is important to note that the above objective differs from the objective optimized by all mixturebased multimodal VAEs (Deﬁnition 3) in that there are no cross-modal reconstructions in Equation (84).
As a consequence, ELBO sub-sampling puts more weight on the approximation of the marginal distributions compared to the conditionals and therefore does not optimize a proper bound on the joint distribution (Wu and Goodman, 2019).
Figure 13 shows the PolyMNIST β -ablation comparing MVAE with and without ELBO sub-sampling.
MVAE + denotes the model with ELBO sub-sampling using objective (84).
Notably, MVAE + achieves signiﬁcantly better generative coherence, while both models perform similarly in terms of generative quality (both in terms of FID and joint log-likelihood).
Hence, even though the MVAE + optimizes an incorrect bound on the joint distribution (Wu and Goodman, 2019), our results suggest that the learned models behave quite similar in practice, which can be of independent interest for future work.
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]