S CALABLE M ULTIMODAL V ARIATIONAL A UTOENCODERS WITH SURROGATE JOINT POSTERIOR
Anonymous authors
Paper under double-blind review
A BSTRACT
To obtain a joint representation from multimodal data in variational autoencoders (VAEs), it is important to infer the representation from arbitrary subsets of modalities after learning.
A scalable way to achieve this is to aggregate the inferences of each modality as experts.
A state-of-the-art approach to learning this aggregation of experts is to encourage all modalities to be reconstructed and crossgenerated from arbitrary subsets.
However, this learning may be insufﬁcient if cross-generation is difﬁcult.
Furthermore, to evaluate its objective function, exponential generation paths concerning the number of modalities are required.
To alleviate these problems, we propose to explicitly minimize the divergence between inferences from arbitrary subsets and the surrogate joint posterior that approximates the true joint posterior.
We also proposed using a gradient origin network, a deep generative model that learns inferences without using an inference network, thereby reducing the need for additional parameters by introducing the surrogate posterior.
We demonstrate that our method performs better than existing scalable multimodal VAEs in inference and generation.
I NTRODUCTION
Our world is characterized by many kinds of information.
This multimodal information can be used to form common concepts.
For instance, by recognizing information, such as the landscape of the beach and the salty scent of the ocean, we can form a concept about the sea.
In multimodal learning (Baltruˇsaitis et al., 2018), it has been studied as the learning of joint representations from multimodal data, especially using variational autoencoders (VAEs) (Kingma & Welling, 2013;
Rezende et al., 2014).
The goal of multimodal representation learning with VAEs (Suzuki et al., 2016; Vedantam et al., 2018; Wu & Goodman, 2018; Shi et al., 2019) is to learn all given multimodal data and to infer a joint representation from arbitrary subsets of modalities.
This representation is useful as an input to predict the corresponding label information or as an intermediate representation to generate other modalities, i.e., cross-generations.
However, this is difﬁcult because a joint representation must be inferred even when some of the modalities are missing at test time.
The most naive way to deal with this difﬁculty is to have inference models for all possible combinations of modalities, which is exponentially necessary given the number of modalities.
A scalable way to address this is to consider the inferences for each modality as experts and combine them to approximate the inferences from an arbitrary subset of modalities.
There are two types of methods for aggregating inferences: products of experts (PoE) (Hinton, 2002) and mixtures of experts (MoE); VAEs that use these as approximate posteriors are proposed as
MVAE (Wu & Goodman, 2018) and MMVAE (Shi et al., 2019), respectively.
Each method has different advantages that exist in a trade-off relationship, i.e., PoE increases the amount of mutual information with joint representation by increasing the number of modalities, while MoE has the effect of encouraging cross-generation between modalities.
Sutter et al.
(2021) deﬁned the objective of multimodal VAEs to be that the inference of all possible subsets approximate the true joint posterior, and derived a generalized evidence lower bound (ELBO) as a lower bound of that objective.
This ELBO can be viewed as using a mixture of products of experts (MoPoE) as an aggregation method for inference, which is to generate all modalities from PoE-aggregated inference of arbitrary subsets.
In other words, this model, called MoPoE-VAE, tries to achieve the above objective of multimodal VAEs by minimizing all reconstruction and crossgeneration errors in all modality combinations.
They showed that this is a generalization that has the advantages of both PoE and MoE. However, these properties of MoPoE-VAE imply that inference may not be sufﬁciently learned when cross-generation is difﬁcult.
Furthermore, the number of paths for this reconstruction and cross-generation in MoPoE-VAE is required to be exponential with respect to the number of modalities.
Therefore, we pursue a scalable method that realizes the above objective of multimodal VAEs.
In this paper, we propose a different approach to achieve this objective.
We ﬁrst prepare a surrogate joint posterior, which approximates the true joint posterior given all modalities, and then minimize the divergence between the PoE-aggregated inferences and this surrogate posterior.
The approximation of surrogate posterior is perform by the usual VAE optimization, and we simultaneously perform this optimization and the divergence minimization above.
The advantages of this new approach are twofold.
First, if the surrogate model is sufﬁciently close to the true posterior, then optimizing this objective function explicitly achieves the objective of multimodal VAEs deﬁned in MoPoE, i.e, approximating inferences from arbitrary subsets to the true posterior.
Second, unlike MoPoE, this method is not exponentially more expensive.
This is because this method is based on divergence minimization of all inference models, rather than reconstruction or cross-generation of all modalities in combination.
Although this method requires an additional surrogate model, the size is not exponential as in previous similar methods (Suzuki et al., 2016;
Vedantam et al., 2018), but only doubles the number of parameters in encoders.
Therefore, we call this method a scalable multimodal VAE (SMVAE) .
Furthermore, to reduce the number of parameters in the surrogate posterior, we propose to use variational gradient origin networks (VGONs) (Bond-Taylor & Willcocks, 2020) for training the surrogate posterior.
VGONs are deep generative models that can rapidly acquire amortized inference to representations without a trainable network.
We call our method that uses VGONs for surrogate posterior SMVGON.
Experiments show that the proposed method performs better than conventional methods in inferring joint representations and generating modalities on commonly used multimodal benchmarks.
In particular, we show that the proposed method improves the performance in situations where the conventional methods have difﬁculty in cross-generation.
We also conﬁrm that the performance of
SMVGONs with VGONs is comparable to that of SMVAEs when the input modalities are relatively simple, without the need for additional inference networks for surrogate posteriors.
P RELIMINARY 2. M ULTIMODAL VAE S
Suppose that we are given an i.i.d.
dataset { X ( i ) } N i =1 , where each example is a set of M modalities
X ( i ) =
{ x ( i ) m } M m =1 and where each modality x ( i ) m =
[ x ( i ) 1 m , ..., x ( i ) D m m ] has its own dimension D m and domain.
We assume that these examples are derived from data distribution p data ( X ) and that each example X ( i ) has a corresponding common latent concept z ( i ) =
[ z ( i ) 1 , ..., z ( i ) j , ..., z ( i ) J ] ∈ R J , i.e., a joint representation.
Given a training set { X ( i ) } Ni =1 , our objective is to infer a joint representation from the subset of modalities 1 X s ⊆ X and to generate another subset X ′ s ⊆ X via that representation.
In other words, we aim to obtain an inference p ( z | X ) and a joint distribution p ( X, z ) =
Q x m ∈ X p ( x m | z ) p ( z ) using a training set.
In this study, we designate the generation when X s =
X ′ s as reconstruction and when
X s ∩ X ′ s = ∅ as cross-generation .
To achieve this, we use VAEs (Kingma & Welling, 2013; Rezende et al., 2014), which are deep generative models that can learn inference in addition to generative models.
Therefore, we can acquire representations of data by inference.
The generative model p Θ ( X | z ) =
Q x m ∈
X p θ m ( x m | z ) , where Θ = { θ m } Mm =1 is a set of parameters, is parameterized by deep neural networks, and the prior of the latent variable is set as a standard Gaussian p ( z ) =
N ( 0 , I ) .
The objective function of VAEs is to maximize the marginal log-likelihood of the multimodal dataset: P Ni =1 log p Θ ( X i ) .
However, since it is tractable to optimize this likelihood directly, we introduce an inference model q Φ ( z | X ) to approximate the posterior p Θ ( z | X ) , and instead optimize the following lower bound of the marginal log-likelihood given X , which is called the ELBO:
[Equation]
Here, the optimization of ELBO with respect to the parameters of the inference model corresponds to minimizing the following Kullback-Leiber (KL) divergence:
[Equation]
In other words, maximizing the ELBO encourages bringing the true posterior closer to the inference model.
After optimizing Equation 1, we aim to infer a joint representation z from an arbitrary subset of modalities X s .
However, this is challenging because Equation 1 only models joint inferences from all modalities.
Therefore, for inferences from a subset of modalities X s , other modality inputs are missing, which results in the inability to infer the representation properly (Suzuki et al., 2016).
A scalable way to alleviate this difﬁculty is to consider the inference model for each modality as an expert and approximate the inference model for arbitrary modalities by combining them.
There are two ways to combine experts: product and mixture.
2.1. A GGREGATED INFERENCE WITH P O E AND M O E
A joint inference given all modalities X is expressed as the following using PoE (Hinton, 2002):
[Equation]
In general, PoE cannot be calculated as closed-form, but if we restrict each expert to Gaussian q φ m ( z | x m ) =
N (cid:0) µ m , diag( σ 2 m )) (cid:1) , then Equation 3 is calculable as q PoE ( z | X ) =
N ( µ , diag( σ 2 )) , where µ j = P m µ jm σ − 2 jm P m σ − 2 jm and σ 2 j = 1 P m σ − 2 jm .
Therefore, the Gaussian PoE enables us to infer the representation from subset X s .
However, if we optimize this inference model with Equation 1, then the inference of arbitrary modalities is not optimized explicitly.
Wu & Goodman (2018) proposed to train ELBOs with PoEaggregated inferences of all modality combinations P X s ∈X L (Θ s , Φ s ; X s ) ; however, since the number of ELBOs increases exponentially, they used the subsampled version.
The model that learns
PoE inference by this objective function is called MVAE (Wu & Goodman, 2018).
Another method to combine experts is to use MoE:
[Equation]
where w m is a mixing coefﬁcient of each expert and where
P x m ∈
X w m = 1 .
Shi et al.
(2019) proposed a VAE with MoE-aggregated inference models called MMVAE.
There is a trade-off between the approximate posterior with PoE and MoE. PoE can perform inference given any subset and MVAE can learn inference explicitly during training.
PoE also has the property that increasing the size of the subset, i.e., the number of input modalities, leads to an increase in the information content of the inferred latent variables.
However, the objective function of
MVAE does not guarantee that the inferences from any subset are the same, since it only learns reconstruction and not cross-generation.
Sutter et al.
(2021) also pointed out that its objective function by subsampling is not a valid lower bound.
MMVAE based on MoE, on the other hand, has a valid ELBO and learns to explicitly encourage cross-generation, which ensures that the inferences from each modality remain consistent.
However, this method only learns inferences from a single modality, and increasing the number of modalities does not necessarily increase the information content of the representation.
[Equation]
Sutter et al.
(2021) considered optimization on all possible subsets, i.e., the powerset P ( X ) , and reestablished the minimization of the following KL divergence as the objective of multimodal VAEs:
[Equation]
They then proposed a generalized ELBO that is a valid lower bound on the log-likelihood p Θ ( X ) and whose maximization corresponds to the minimization of Equation 5:
[Equation]
where the form of the posterior of q MoPoE ( z | X ) is called MoPoE and is deﬁned as follows:
[Equation]
A feature of the model that maximizes Equation 6, called MoPoE-VAE, is that it learns to generate all modalities from all possible subsets, including reconstruction and cross-generation, which improves the performance of inference and generation from all subsets.
In addition, they showed that this
MoPoE-VAE is a generalization of MVAE and MMVAE, speciﬁcally MVAE when the summation in Equation 7 is taken only for a set of all modalities X and MMVAE when it is taken for the single modalities { x m } M m =1 .
Therefore, MoPoE-VAE can be regarded as having the advantages of both
MVAE and MMVAE.
2. V ARIATIONAL GRADIENT ORIGIN NETWORKS
VGONs (Bond-Taylor & Willcocks, 2020) are deep generative models that can perform amortized inference equivalent to VAEs without an inference network.
Given an observed variable x and a latent variable z , let the generative model be p θ ( x | z ) , and the inference model be q φ, z 0 ( z ) =
N ( µ φ ( z 0 ) , σ 2 φ ( z 0 ) I ) with the mean and variance mapped by trainable linear transformations ( µ φ and σ 2 φ ) from a parameter z 0 .
This objective function, ELBO, given x is deﬁned as follows:
[Equation]
In VGONs, the estimator for z 0 is calculated by a single update based on the gradient of Equation 8:
[Equation]
9) where z 0 = 0 , i.e, the initial value of z 0 is set to the origin.
Using this estimator ˆ z 0 , we can obtain inference as z ∼ q φ, ˆ z
( z ) and can evaluate the objective function as L V GON ( θ, φ, ˆ z 0 ; x ) .
In learning, we also backpropagate the gradient of the objective function through Equation 9, which allowed us to learn both inference and generation simultaneously as in VAEs.
M ETHODS 3. I SSUES OF M O P O E-VAE
MoPoE-VAE aims at the objective of Equation 5 by maximizing the ELBO of Equation 6.
The two are equivalent in variational inference, but have different implications for learning including generative models, i.e., whether the Equation 5 is achieved depends on the performance of all modalities of reconstruction and cross-generation.
Therefore, in situations where cross-generation is difﬁcult, this objective might not be fully achieved, speciﬁcally, the inference and generation of some modalities might be signiﬁcantly worse than others.
Another issue is that the computational cost of evaluating Equation 6 increases exponentially with the number of modalities.
Speciﬁcally, we consider a computational path that generates modalities from inferred representations.
In this case, since there are 2 M representations to be inferred by
MoPoE, the generation path required to evaluate Equation 6 is M × 2 M ; therefore, as the number of modalities increases, the evaluation of the ELBO becomes more intractable and might also lead to insufﬁcient learning of cross-generation.
3. A SURROGATE JOINT POSTERIOR AND THE REFORMULATION OF THE OBJECTIVE
To address the above issues, we aim to explicitly realize Equation 5.
However, since a true posterior cannot be obtained, we prepare a surrogate joint posterior q Λ ( z | X ) , which is a Gaussian distribution parameterized by a neural network and should approximate the true posterior p Θ ( z | X ) .
We then reformulate our objective as minimizing the KL divergence between subset inferences and this surrogate posterior:
[Equation]
Here, we consider each inference from a subset q φ c ( z | X c ) to be expressed in PoE form.
Then, since the surrogate posterior is deﬁned as Gaussian, this KL divergence can be calculated analytically.
If D KL ( q Λ ( z | X ) || p Θ ( z | X ))
≈ 0 is satisﬁed, then this optimization problem is the same as Equation 5.
As explained in Section 2.1, minimizing the KL divergence between the approximate posterior (in this case, the surrogate posterior) and the true posterior is equivalent to maximizing the ELBO. Therefore, we can ﬁrst maximize the ELBO L (Θ , Λ; X ) sufﬁciently, and then ﬁx the surrogate posterior and perform the optimization of Equation 10.
However, this requires a two-step learning process, which can be time consuming. Therefore, we propose to optimize these two steps simultaneously, referring to existing multimodal VAEs with additional inference models such as JMVAE (Suzuki et al., 2016) (see Section 4), i.e., maximize the following objective function:
[Equation]
where β is a hyperparameter that adjusts for the effect of KL divergence in the objective function.
We train this objective function in an end-to-end fashion. Empirically, we have found that this end-toend optimization results in higher performance than the two-stage optimization (see Appendix D). At test time, we infer representations from subsets using a PoE-aggregated inference model q φ c ( z | X c ) .
The advantages of this objective function are as follows.
First, inference models from arbitrary subsets are learned not by cross-generation, but by explicitly minimizing the divergence between them and the surrogate posterior.
Moreover, the surrogate posterior is learned based on the maximization of ELBO L (Θ , Λ; X ) , i.e., based on the reconstruction in all modalities, rather than crossgeneration.
Therefore, this objective can mitigate the impact of the difﬁculty of cross-generation on the performance of inference and generation.
Secondly, the computational cost can be signiﬁcantly reduced compared to MoPoE-VAE.
There is only one pass for generation in evaluating this objective function for each modality in L (Θ , Λ; X ) .
In addition, the second term of Equation 11 computes the inference for all possible subsets, which is obtained by aggregating the inferences for each modality; therefore, the inference pass is also only once for each modality.
We call the proposed model a scalable multimodal VAE (SMVAE) because it is scalable to the number of modalities while explicitly optimizing toward the objective of multimodal VAEs.
3. R EDUCING THE PARAMETERS IN SURROGATE INFERENCE MODELS
Although the proposed method alleviate the issues of previous multimodal VAEs, it also has the disadvantage of requiring more parameters for the inference model.
The added surrogate posterior requires as many parameters as all of the unimodal inference models, i.e., the parameter size of the inference model is twice as large as that of previous models such as MoPoE-VAE.
Note that in existing studies of multimodal VAEs with additional inference models, such as JMVAE, the parameter size increases exponentially because inference models are required for each subset.
In comparison, SMVAE keeps the number of required parameters linear with respect to the number of modalities.
To reduce the number of required parameters, we propose to use VGONs to train the surrogate posterior and generative models for all modalities.
In the VGON framework, the surrogate posterior given multimodal inputs can be deﬁned as q λ, z 0 ( z ) =
N ( µ λ ( z 0 ) , σ 2 λ ( z 0 ) I ) , where the estimator of z 0 is calculated as ˆ z 0 = ∇ z
L V GON (Θ , λ, z 0 ; X ) .
Therefore, the objective function of the proposed method using VGONs is as follows:
[Equation]
We refer to this model as SMVGON to distinguish it from SMVAE.
Note that the surrogate posterior is used to approximate the aggregated inference model during training and is not used during testing; therefore, it is possible to use other iterative inference algorithms.
In this study, however, we used VGONs, which can perform inference in a single iteration, to avoid slow training.
R ELATED WORKS
For learning unsupervised joint representations from multimodal data, autoencoder-based methods have been used (Ngiam et al., 2011; Silberer & Lapata, 2014); however, they have difﬁculties in the challenge of missing modalities.
Joint representation learning using deep belief networks (Hinton, 2009) or deep Boltzmann machines (Salakhutdinov & Hinton, 2009) can adequately handle missing modalities (Srivastava & Salakhutdinov, 2012; Srivastava et al., 2012; Sohn et al., 2014), but has difﬁculty scaling to large dataset.
Early multimodal learning using VAEs dealt with cross-generation of two modalities, such as conditional VAEs (Sohn et al., 2015; Kingma et al., 2014) and conditional multi-modal autoencoders (Pandey & Dukkipati, 2017).
However, because these methods directly learn conditional generative models, they cannot perform bidirectional generation between modalities, nor can they obtain joint latent representations of different modalities.
Some studies of multimodal VAEs require additional inference models (Suzuki et al., 2016;
Vedantam et al., 2018; Korthals et al., 2019; Wu & Goodman, 2019).
Suzuki et al.
(2016) pointed out that VAEs trained on all modalities have the problem of missing modalities in inference, and propose to approximate the unimodal inference models to explicitly approach the joint inference model of VAEs. The objective function of JMVAE is the ELBO of VAE given multimodal inputs minus the KL divergence corresponding to the above approximation.
3 Vedantam et al.
(2018) introduced TELBO, where the objective function is composed of the sum of ELBOs that take each set of modalities as input, using the same models as JMVAE. Korthals et al. (2019) introduced M 2 VAE that can be regarded as a combination of JMVAE and TELBO, and Wu & Goodman (2019) proposed VAEVAE that excludes a KL divergence term between joint inference and prior from it. They are learned end-to-end with a single objective function, except for TELBO.
The challenge with these models is that they require inference models for all modality combinations, i.e., the number of inference models grows exponentially with the number of modalities.
Our proposed method is similar to these methods; however, unlike these methods, required memory cost is linear in the number of modalities. Kutuzova et al. (2021) proposed to introduce additional unimodal inference models instead of a joint inference model, and to reduce the number of required inference models to the square of the number of modalities.
However, this also does not scale with the number of modalities compared to our linear cost model.
MMJSD proposed by Sutter et al. (2020) is closer to our work, i.e., it encourages unimodal inference to be closer to joint inference from all modalities.
The differences between our work and
MMJSD are twofold. First, while MMJSD only considers unimodal inference, our work approximates joint inference with any combination of unimodals, which allows to learn inferences from arbitrary modalities. The second is the design of a joint inference. MMJSD considers a dynamic prior that combines all unimodal inferences and considers it a joint inference.
However, there is no guarantee that this dynamic prior approximates the true posterior as in the surrogate posterior of SMVAE. In addition, since MMJSD uses the same networks for a joint inference and unimodal inferences, it might be difﬁcult for this joint inference to achieve all of the approximations to the true posterior and to inferences from arbitrary modalities.
The other topic in multimodal VAEs is the acquisition of modality-speciﬁc latent variables, apart from the global latent variables that embed all modalities (Tsai et al., 2018; Hsu & Glass, 2018;
Sutter et al., 2020).
Our study is in a different direction from these, since we are addressing the possibility of inferring a single joint representation from arbitrary subset of modalities.
In the experiments, we conﬁrm that SMVAE achieves high performance with a single latent variable in a task where the conventional method achieves by adding modality-speciﬁc latent variables.
Recently, contrastive-based multimodal (multi-view) learning (Tian et al., 2019; Alayrac et al., 2020; Tsai et al., 2021) has been gaining attention instead of autoencoder-based, but these aim to acquire coordinated representations (Baltruˇsaitis et al., 2018), i.e., they do not assume inference from a subset of arbitrary modalities.
E XPERIMENTS
We use MNIST-SVHN-Text (Sutter et al., 2020), Bimodal CelebA (Sutter et al., 2020), and PolyMNIST (Sutter et al., 2021) datasets for evaluation.
MNIST-SVHN-Text is the MNISTSVHN (Shi et al., 2019) with the addition of the text modality, which represents the English name of each number as text and randomly changes the starting index to give a diversity.
Bimodal CelebA is a dataset of face images from the CelebA (Liu et al., 2015) with additional text describing the corresponding attribute labels.
The PolyMNIST dataset consists of a set of ﬁve different MNIST images with different backgrounds and handwriting styles, each of which is considered a different modality.
Different modalities of the same example have the same digits.
We conﬁrm that the proposed model realizes the objective of Equation 5, i.e., it can properly infer representations and generate modalities from arbitrary subsets.
To evaluate this quantitatively, we estimate the performance of the representation by training a linear classiﬁer under inference from all modalities and predicting by it from the representation inferred from subsets.
If this evaluation results in high performance on different subsets, it means that the inference from the subsets is successful.
We also perform cross-generation from arbitrary modalities and evaluate the performance of the generated modalities by pre-trained classiﬁers from Sutter et al.
The interpretation of this evaluation is the same as in the case of representation.
For the network architectures, we follow Sutter et al.
(2020), with 32-dimensional latent variables for MNIST-SVHN-Text and Bimodal CelebA and 256 for PolyMNIST.
The distributions for images are set to Laplace, and Text is set to categorical.
To align the scaling of the likelihoods for each modality, we divided each likelihood by the dimension of each modality and multiplied the largest dimension value by all likelihoods so that the coefﬁcient of the likelihood term with the largest dimension was 1.
We use Adam (Kingma & Ba, 2014) for optimization, with a learning rate of 0.001, and a mini-batch size of 256.
We set β = 1 , and train 50 epochs for MNIST-SVHN-Text and
Bimodal CelebA and 300 epochs for PolyMNIST.
All experimental results are the average of three runs.
We implement them using PyTorch (Paszke et al., 2019) and Pixyz (Suzuki et al., 2021).
5. MNIST-SVHN-T EXT
Table 1 shows the results of the performance of representations in MNIST-SVHN-Text.
MVAE has better performance for joint representations, but poorer performance for representations inferred from single modalities.
MMVAE has better performance for inference from single modalities, but worse performance for joint inference.
The performance of MoPoE compensates for the shortcomings of both methods.
In contrast, the proposed method outperforms MoPoE in inference from
[Table]
[Table]
most of the subsets, though it is lower than MMVAE for single modalities.
This indicates that approximations that minimize the divergence from the surrogate posterior are more effective than computationally expensive approximations based on cross-generation.
In addition, the performance of SMVGON is almost the same as that of SMVAE, which means that the surrogate posterior using VGON is sufﬁcient as a joint inference model even without inference networks.
See Appendix for the results of inference and generation when training with VGON alone.
This result conﬁrms that the performance of the surrogate posterior is sufﬁcient for this dataset.
The results for generation in Table 2 are similar to those of the inference of the representation.
It can be seen that the performance of the previous work on generating SVHN from subsets is greatly degraded.
This is thought to be due to the difﬁculty in cross-generation to SVHN, which resulted in poor learning.
On the other hand, the proposed method is able to generate from subsets with the same performance as from all modalities.
This shows that the proposed method achieves our objective, i.e., to make inference from any modality as good as from all modalities.
Figure 1 shows the results of SVHN generation given different modalities for MoPoE-VAE, SMVAE, and SMVGON.
We can see that MoPoE-VAE fails to generate SVHN from both MNIST and
Text modalities.
On the other hand, SMVAE can generate corresponding and diverse SVHN images from each modality.
Sutter et al.
(2020) attempted this generation by introducing modality-speciﬁc latent variables, but these results suggest that our proposed SMVAE can perform cross-generation of SVHN with only a single latent variable.
In addition, it can be conﬁrmed that SMVGON can generate images with similar performance, indicating that our objective can be achieved without inference networks.
5. B IMODAL C ELEB A
Table 3 shows the results of representation and generation for Bimodal CelebA. On this dataset,
SMVAE performs better than the other methods.
We believe this is due to the challenging nature of this dataset.
Although this dataset is bimodal, it has a larger diversity of images and text compared to
MNIST-SVHN-Text.
Therefore, it becomes difﬁcult to learn reconstructions and cross-generations from a single modality, which makes previous methods that rely on this to learn inference insufﬁ-
[Figure]
[Table]
cient.
Our proposed method, on the other hand, learns the surrogate joint posterior by reconstructing all modalities, and approximates the inferences from arbitrary subsets to be close to it.
Therefore, our method is less sensitive to the difﬁculty of cross-generation.
On the other hand, SMVGON has not reached the same level of performance as SMVAE.
This may be due to the difﬁculty of the multimodal dataset, which makes VGON’s inference insufﬁcient.
For the training results of VGON alone on this dataset, please refer to Appendix C. For qualitative generation results, see Appendix E.
5. P OLY MNIST
Figure 2 shows the results of PolyMNIST.
First, we can see that SMVAE outperforms the existing methods in both representation and generation.
Also, the generation results show that SMVAE is able to generate corresponding numbers more clearly than MoPoE-VAE.
This result conﬁrms that
SMVAE is a scalable model not only in terms of computational cost but also in terms of performance when the number of modalities increases.
On the other hand, the performance of SMVGON is lower than that of other models in both generation and representation.
This might be because, as in the case of bimodal CelebA, inference by VGON becomes difﬁcult as the number of modalities increases.
In this study, we proposed SMVAE, which approximates inference from subsets by explicitly minimizing the divergence from the surrogate posterior distribution.
The model is scalable with respect to the number of modalities and shows high performance in inference and generation, especially in the case where cross-generation is difﬁcult.
Furthermore, to reduce the parameters of the inference model, we introduced SMVGON using VGON, which can learn inference without inference networks.
The remaining task of this study is to ﬁnd a way to introduce a surrogate posterior distribution that will result in good inference with low memory and computational costs.
VGON performed well on MNIST-SVHN-Text, but did not perform well on challenging problem settings such as Bimodal CelebA and PolyMNIST.
In the future, we would like to address this issue and pursue effective methods for a wide range of multimodal datasets.
ETHICS STATEMENT
This work is a state-of-the-art method for multimodal VAEs, which will have an impact on the machine learning domain due to its scalability.
Our method can cross-generate from one modality to another corresponding modality; therefore, we have to be aware that it can be used to generate non-existent facial images, text, spoken voices etc.
In addition, since the attributes in the CelebA dataset used for training in this study have biases (Prabhu et al., 2019) it is necessary to remove these biases when training the proposed method in practice.
R EPRODUCIBILITY S TATEMENT
All codes and datasets necessary to reproduce the experiments in this paper will be available after the paper review is concluded.
The hyperparameters and other details in the training were written at the beginning of Section and the architecture of the networks is the same as in Sutter et al.
A C OMPARISON BETWEEN SMVAE AND JMVAE WHEN THE NUMBER OF MODALITIES IS
When the number of modalities is two, the objective function of JMVAE (Suzuki et al., 2016) is as follows: [Equation]
[Equation]
Compared to Equation 11, the difference is that the direction of KL divergence is reversed and aggregation by PoE is not used.
Table 4 shows the results of comparing SMVAE and JMVAE on the Bimodal CelebA dataset.
The performance of the representations is almost the same, but the generation results are better for SMVAE.
This may indicate that the KL divergence direction of
SMVAE is better for cross-generation.
Table 5 shows the inference and generation performance of VAE and VGON for multimodal inputs.
It can be seen that VGON can generally learn inference models equivalent to VAE without inference networks, although its performance is slightly lower than that of VAE.
C C OMPARISON OF VAE AND VGON IN B IMODAL C ELEB A
Table 6 shows the comparison between VAE and VGON in Bimodal CelebA.
The results conﬁrm that the performance of VGON is lower than that of VAE in all tasks.
D E FFECTIVENESS OF END - TO - END LEARNING IN SMVAE
Figure 3 compares the results of end-to-end learning in SMVAE, i.e., optimizing Equation 11, and the results of two steps, i.e., ﬁrst optimizing surrogate inference and then minimizing KL divergence with inference from arbitrary modalities.
The dataset is PolyMNIST; in two stages, 200 epochs were trained in the ﬁrst stage and 100 epochs in the second stage.
The results show that the end-to-end method gives better results.
This may be due to the fact that the two distributions are more easily approximated by training them simultaneously.
E C ONDITIONAL GENERATION ON B IMODAL C ELEB A
Figure 4 shows the results of conditional generation of Bimodal CelebA given a text modality.
We used SMVAE as the model.
[Table]
[Table]
[Table]
F R ANDOM IMAGE GENERATION
Figure 5 shows the results of generating all modalities from random noise in the latent space for each dataset.
We used SMVAE as the model.
Each sample was generally generated correspondingly across modalities but the text for Bimodal CelebA was not generated well.
[Figure]