L EARNING M ULTIMODAL VAE S THROUGH M UTUAL S UPERVISION
A BSTRACT
Multimodal variational autoencoders (VAEs) seek to model the joint distribution over heterogeneous data (e.g. vision, language), whilst also capturing a shared representation across such modalities.
Prior work has typically combined information from the modalities by reconciling idiosyncratic representations directly in the recognition model through explicit products, mixtures, or other such factorisations.
Here we introduce a novel alternative, the M utually sup E rvised M ultimodal
VA E (MEME), that avoids such explicit combinations by repurposing semisupervised VAEs to combine information between modalities implicitly through mutual supervision.
This formulation naturally allows learning from partiallyobserved data where some modalities can be entirely missing—something that most existing approaches either cannot handle, or do so to a limited extent.
We demonstrate that MEME outperforms baselines on standard metrics across both partial and complete observation schemes on the MNIST-SVHN (image–image) and CUB (image–text) datasets 1 .
We also contrast the quality of the representations learnt by mutual supervision against standard approaches and observe interesting trends in its ability to capture relatedness between data.
I NTRODUCTION
Modelling the generative process underlying heterogenous data, particularly data spanning multiple perceptual modalities such as vision or language, can be enormously challenging.
Consider for example, the case where data spans across photographs and sketches of objects.
Here, a data point, comprising of an instance from each modality, is constrained by the fact that the instances are related and must depict the same underlying abstract concept.
An effective model not only needs to faithfully generate data in each of the different modalities, it also needs to do so in a manner that preserves the underlying relation between modalities.
Learning a model over multimodal data thus relies on the ability to bring together information from idiosyncratic sources in such a way as to overlap on aspects they relate on, while remaining disjoint otherwise.
Variational autoencoders (VAEs) (Kingma & Welling, 2014) are a class of deep generative models that are particularly well-suited for multimodal data as they employ the use of encoders — learnable mappings from high-dimensional data to lower-dimensional representations—that provide the means to combine information across modalities.
They can also be adapted to work in situations where instances are missing for some modalities; a common problem where there are inherent difﬁculties in obtaining and curating heterogenous data.
Much of the work in multimodal VAEs involves exploring different ways to model and formalise the combination of information with a view to improving the quality of the learnt models (see § 2).
Prior approaches typically combine information through explicit speciﬁcation as products (Wu &
Goodman, 2018), mixtures (Shi et al., 2019), combinations of such (Sutter et al., 2021), or through additional regularisers on the representations (Suzuki et al., 2016; Sutter et al., 2020).
Here, we explore an alternative approach that leverages advances in semi-supervised VAEs (Siddharth et al.,
Encoding
Prior
Reg.
Data (a) VAE
Encoding Encoding
Combine
Prior
Modality Modality
Reg.
(b) Typical Multimodal VAE
Encoding
Encoding (prior)
Encoding Encoding (prior)
Modality
Modality Modality
Modality+ Reg.
Reg.
(c) MEME (ours)
; Joy et al., 2021) to repurpose existing regularisation in the VAE framework as an implicit means by which information is combined across modalities (see Figure 1).
We develop a novel formulation for multimodal VAEs that views the combination of information through a semi-supervised lens, as mutual supervision between modalities.
We term this approach
M utually sup E rvised M ultimodal VA E (MEME).
Our approach not only avoids the need for additional explicit combinations, but it also naturally extends to learning in the partially-observed setting—something that most prior approaches cannot handle.
We evaluate MEME on standard metrics for multimodal VAEs across both partial and complete data settings, on the typical multimodal data domains, MNIST-SVHN (image-image) and the less common but notably more complex
CUB (image-text), and show that it outperforms prior work on both.
We additionally investigate the capability of MEMEs ability to capture the ‘relatedness’, a notion of semantic similarity, between modalities in the latent representation; in this setting we also ﬁnd that MEME outperforms prior work considerably.
R ELATED WORK
Prior approaches to multimodal VAEs can be broadly categorised in terms of the explicit combination of representations (distributions), namely concatenation and factorization.
Concatenation : Models in this category learn joint representation by either concatenating the inputs themselves or their modality-speciﬁc representations.
Examples for the former includes early work in multimodal VAEs such as the JMVAE (Suzuki et al., 2016), triple ELBO (Vedantam et al., 2018) and MFM (Tsai et al., 2019), which deﬁne a joint encoder over concatenated multimodal data.
Such approaches usually require the training of auxiliary modality-speciﬁc components to handle the partially-observed setting, with missing modalities, at test time.
They also cannot learn from partially-observed data.
In very recent work, Gong et al.
(2021) propose VSAE where the latent representation is constructed as the concatenation of modality-speciﬁc encoders.
Inspired by VAEs that deal with imputing pixels in images such as VAEAC (Ivanov et al., 2019), Partial VAE (Ma et al., 2018), MIWAE (Mattei & Frellsen, 2019), HI-VAE (Naz ´ abal et al., 2020) and pattern-set mixture model
(Ghalebikesabi et al., 2021), VSAE can learn in the partially-observed setting by incorporating a modality mask.
This, however, introduces additional components such as a collective proposal network and a mask generative network, while ignoring the need for the joint distribution over data to capture some notion of the relatedness between modalities.
Factorization :
In order to handle missing data at test time without auxiliary components, recent work propose to factorize the posterior over all modalities as the product (Wu & Goodman, 2018) or mixture (Shi et al., 2019) of modality-speciﬁc posteriors (experts).
Following this, Sutter et al.
(2021) proposes to combine the two approaches (MoPoE-VAE) to improve learning in settings where the number of modalities exceeds two.
In contrast to these methods, mmJSD (Sutter et al., 2020) combines information not in the posterior, but in a “dynamic prior”, deﬁned as a function (either mixture or product) over the modality-speciﬁc posteriors as well as pre-deﬁned prior.
Table 1 provides a high-level summary of prior work.
Note that all the prior approaches have some explicit form of joint representation or distribution, where some of them induces the need for auxiliary components to deal with missing data at test time, while others are established without signiﬁcant theoretical beneﬁts.
By building upon a semi-supervised framework, our method MEME circumvents this issue to learn representations through mutual supervision between modalities, and is able to deal with missing data at train or test time naturally without additional components.
[Table]
M ETHOD
Consider a scenario where we are given data spanning two modalities, s and t , curated as pairs ( s , t ) .
For example this could be an “image” and associated “caption” of an observed scene.
We will further assume that some proportion of observations have one of the modalities missing, leaving us with partially-observed data.
Using D s , t to denote the proportion containing fully observed pairs from both modalities, and D s , D t for the proportion containing observations only from modality s and t respectively, we can decompose the data as D = D s ∪ D t ∪ D s , t .
In aid of clarity, we will introduce our method by conﬁning attention to this bi-modal case, providing a discussion on generalising beyond two modalities later.
Following established notation in the literature on VAEs, we will denote the generative model using p , latent variable using z , and the encoder, or recognition model, using q .
Subscripts for the generative and recognition models, where indicated, denote the parameters of deep neural networks associated with that model.
3. S EMI -S UPERVISED VAEs
To develop our approach we draw inspiration from semi-supervised VAEs which use additional information, typically data labels, to extend the generative model.
This facilitates learning tasks such as disentangling latent representations and performing intervention through conditional generation.
In particular, we will build upon the work of Joy et al.
(2021), who suggests to supervise latent representations in VAEs with partial label information by forcing the encoder, or recognition model, to channel the ﬂow of information as s → z → t .
They demonstrate that the model learns latent representations, z , of data, s , that can be faithfully identiﬁed with label information t .
[Equation]
Figure 2 shows a modiﬁed version of the graphical model from Joy et al.
(2021), extracting just the salient components, and avoiding additional constraints therein.
The label, here t , is denoted as partially observed as not all observations s have associated labels.
Note that, following the information ﬂow argument, the generative model factorises as p θ,ψ ( s , z , t ) =
p θ
[Equation]
s | z )
p ψ
[Equation]
z | t ) p
[Equation]
t ) (solid arrows) whereas the recognition model factorises as q φ,ϕ ( t , z | s ) =
q ϕ ( t | z ) q φ ( z | s ) (dashed arrows).
This autoregressive formulation of both the generative and recognition models is what enables the “supervision” of the latent representation of s by the label, t , via the conditional prior p ψ ( z | t ) as well as the classiﬁer q ϕ ( t | z ) .
The corresponding objective for supervised data, derived as the (negative) variational free energy or evidence lower bound (ELBO) of the model is
[Equation]
with the generative and recognition model parameterised by Θ = { θ, ϑ } and Φ = { φ, ϕ } respectively.
A derivation of this objective can be found in Appendix A.
3. M UTUAL S UPERVISION
Procedurally, a semi-supervised VAE is already multimodal.
Beyond viewing labels as a separate data modality, for more typical multimodal data (vision, language), one would just need to replace labels with data from the appropriate modality, and adjust the corresponding encoder and decoder to handle such data.
Conceptually however, this simple replacement can be problematic.
Supervised learning encapsulates a very speciﬁc imbalance in information between observed data and the labels—that labels do not encode information beyond what is available in the observation itself.
This is a consequence of the fact that labels are typically characterised as projections of the data into some lower-dimensional conceptual subspace such as the set of object classes one may encounter in images, for example.
Such projections cannot introduce additional information into the system, implying that the information in the data subsumes the information in the labels, i.e. that the conditional entropy of label t given data s is zero: H ( t | s )
= 0 .
Supervision-based models typically incorporate this information imbalance as a feature, as observed in the speciﬁc correspondences and structuring enforced between their label y and latent z in Joy et al.
Multimodal data of the kind considered here, on the other hand, does not exhibit this feature.
Rather than being characterised as a projection from one modality to another, they are better understood as idiosyncratic projections of an abstract concept into distinct modalities—for example, as an image of a bird or a textual description of it.
In this setting, no one modality has all the information, as each modality can encode unique perspectives opaque to the other.
More formally, this implies that both the conditional entropies H ( t | s ) and H
( s | t ) are ﬁnite.
Based on this insight we symmetrise the semi-supervised VAE formulation by additionally constructing a mirrored version, where we swap s and t along with their corresponding parameters, i.e. the generative model now uses the parameters Φ and the recognition model now uses the parameters Θ .
This has the effect of also incorporating the information ﬂow in the opposite direction to the standard case as t → z → s , ensuring that the modalities are now mutually supervised .
This approach forces each encoder to act as an encoding distribution when information ﬂows one way, but also act as a prior distribution when the information ﬂows the other way.
Extending the semi-supervised
VAE objective (6), we construct a bi-directional objective for MEME
[Equation]
where both information ﬂows are weighted equally.
On a practical note, we ﬁnd that it is important to ensure that parameters are shared appropriately when mirroring the terms, and that the variance in the gradient estimator is controlled effectively.
Please see Appendices B to D for further details.
3. L EARNING FROM P ARTIAL O BSERVATIONS
In practice, prohibitive costs on multimodal data collection and curation imply that observations can frequently be partial, i.e., have missing modalities.
One of the main beneﬁts of the method introduced here is its natural extension to the case of partial observations on account of its semisupervised underpinnings.
Consider, without loss of generality, the case where we observe modality s , but not its pair t .
Recalling the autoregressive generative model p ( s , z , t )
= p ( s | z ) p ( z | t ) p ( t ) we can derive a lower bound on the log-evidence
[Equation]
Estimating the integral p ( z ) = (cid:82) p ( z | t ) p ( t ) d t highlights another conceptual difference between a (semi-)supervised setting and a multimodal one.
When t is seen as a label, this typically implies that one could possibly compute the integral exactly by explicit marginalisation over its support, or at the very least, construct a reasonable estimate through simple Monte-Carlo integration.
In Joy et al.
(2021), the authors extend the latter approach through importance sampling with the “inner” encoder q ( t | z ) , to construct a looser lower bound to (3).
In the multimodal setting however, this poses serious difﬁculties as the domain of the variable t is not simple categorical labels, but rather complex continuous-valued data.
This rules out exact marginalisation, and renders further importance-sampling practically infeasible on account of the quality of samples one can expect from the encoder q ( t | z ) which itself is being learnt from data.
To overcome this issue and to ensure a ﬂexible alternative, we adopt an approach inspired by the VampPrior (Tomczak & Welling, 2018).
Noting that our formulation includes a conditional prior p ψ ( z | t ) , we introduce learnable pseudo-samples λ t = { u t i } Ni =1 to estimate the prior as p λ t ( z )
= 1 N (cid:80) Ni =
1 p ψ ( z | u t i ) .
Our objective for when t is unobserved is thus
[Equation]
where the equivalent objective for when s is missing can be derived in a similar way.
For a dataset D containing partial observations the overall objective (to maximise) becomes
[Equation]
This treatment of unobserved data distinguishes our approach from alternatives such as that of Shi et al.
(2019), where model updates for missing modalities are infeasible.
Whilst there is the possibility to perform multimodal learning in the weakly supervised case as introduced by Wu & Goodman (2018), their approach directly affects the posterior distribution, whereas ours only affects the regularization of the embedding during training.
At test time, Wu & Goodman (2018) will produce different embeddings depending on whether all modalities are present, which is typically at odds with the concept of placing the embeddings of related modalities in the same region of the latent space.
Our approach does not suffer from this issue as the posterior remains unchanged regardless of whether the other modality is present or not.
Learning with MEME
Given the overall objective in (5), we train MEME through maximumlikelihood estimation of the objective over a dataset D .
Each observation from the dataset is optimised using the relevant term in the right-hand side of (5), through the use of standard stocastic gradient descent methods.
Note that training the objective involves learning all the (neural network) parameters ( θ, ψ, φ, ϕ ) in the fully-observed, bi-directional case.
When training with a partial observation, say just s , all parameters except the relevant likelihood parameter ϕ (for q ϕ ( t | z ) ) are learnt.
Note that the encoding for data in the domain of t is still computed through the learnable pseudo-samples λ t .
This is reversed when training on an observation with just t .
Generalisation beyond two modalities We conﬁne our attention here to the bi-modal case for two important reasons.
Firstly, the number of modalities one typically encounters in the multimodal setting is fairly small to begin with.
This is often a consequence of its motivation from embodied perception, where one is restricted by the relatively small number of senses available (e.g. sight, sound, proprioception).
Furthermore, the vast majority of prior work on multimodal VAEs only really consider the bimodal setting (cf. § 2).
Secondly, it is quite straightforward to extend MEME to settings beyond the bimodal case, by simply incorporating existing explicit combinations (e.g. mixtures or products) on top of the implicit combination discussed here, we provide further explanation in Appendix E.
Our focus in this work lies in exploring and analysing the utility of implicit combination in the multimodal setting, and our formulation and experiments reﬂect this focus.
E XPERIMENTS 4. L EARNING FROM P ARTIALLY O BSERVED D ATA
In this section, we evaluate the performance of MEME following standard multimodal VAE metrics as proposed in Shi et al.
Since our model beneﬁts from its implicit latent regularisation and is able to learn from partially-observed data, here we evaluate MEME’s performance when different proportions of data are missing in either or both modalities during training.
The two metrics used are cross coherence to evaluate the semantic consistency in the reconstructions, as well as latent accuracy in a classiﬁcation task to quantitatively evaluate the representation learnt in the latent space.
We demonstrate our results on two datasets, namely an image ↔ image dataset MNISTSVHN (LeCun et al., 2010; Netzer et al., 2011), which is commonly used to evaluate multimodal
VAEs (Shi et al., 2019; Shi et al., 2021; Sutter et al., 2020; 2021); as well as the more challenging, but less common, image ↔ caption dataset CUB (Welinder et al., 2010).
[Figure] [Figure]
Following standard approaches, we represented image likelihoods using Laplace distributions, and a categorical distribution for caption data.
The latent variables are parameterised by Gaussian distributions.
In line with previous research (Shi et al., 2019; Massiceti et al., 2018), simple convolutional architectures were used for both MNIST-SVHN and for CUB images and captions.
For details on training and exact architectures see Appendix K; we also provide tabularised results in Appendix H.
Cross Coherence Here, we focus mainly on the model’s ability to reconstruct one modality, say, t , given another modality, s , as input, while preserving the conceptual commonality between the two.
In keeping with Shi et al.
(2019), we report the cross coherence score on MNIST-SVHN as the percentage of matching digit predictions of the input and output modality obtained from a pre-trained classiﬁer.
On CUB we perform canonical correlation analysis (CCA) on input-output pairs of cross generation to measure the correlation between these samples.
For more details on the computation of CCA values we refer to Appendix G.
In Figure 5 we plot cross coherence for MNIST-SVHN and display correlation results for CUB in
Figure 6, across different partial-observation schemes.
The x -axis represents the proportion of data that is paired, while the subscript to the method (see legends) indicates the modality that is presented.
For instance, MEME MNIST with f = 0 .
25 indicates that only 25% of samples are paired, and the other 75% only contain MNIST digits, and MEME SPLIT with f = 0 .
25 indicates that the 75% contains a mix of MNIST and SVHN samples that are unpaired and never observed together, i.e we alternate depending on the iteration, the remaining 25% contain paired samples.
We provide qualitative results in Figure 3 and Figure 4.
We can see that our model is able to obtain higher coherence scores than the baselines including
MVAE (Wu & Goodman, 2018) and MMVAE (Shi et al., 2019) in the fully observed case, f = 1 . 0 , as well as in the case of partial observations, f < 1 . 0 .
This holds true for both MNIST-SVHN and
CUB 2 .
It is worth pointing out that the coherence between SVHN and MNIST is similar for both partially observing MNIST or SVHN, i.e. generating MNIST digits from SVHN is more robust to which modalities are observed during training (Figure 5 Right).
However, when generating SVHN from MNIST, this is not the case, as when partially observing MNIST during training the model struggles to generate appropriate SVHN digits.
This behaviour is somewhat expected since the information needed to generate an MNIST digit is typically subsumed within an SVHN digit (e.g. there is little style information associated with MNIST), making generation from SVHN to MNIST easier, and from MNIST to SVHN more difﬁcult.
Moreover, we also hypothesise that observing
MNIST during training provides greater clustering in the latent space, which seems to aid cross generating SVHN digits.
We provide additional t-SNE plots in Appendix H.3 to justify this claim.
For CUB we can see in Figure 6 that MEME consistently obtains higher correlations than MVAE across all supervision rates, and higher than MMVAE in the fully supervised case.
Generally, crossgenerating images yields higher correlation values, possibly due to the difﬁculty in generating semantically meaningful text with relatively simplistic convolutional architectures.
We would like to highlight that partially observing captions typically leads to poorer performance when cross- information.
generating captions.
We hypothesise that is due to the difﬁculty in generating the captions and the fact there is a limited amount of captions data in this setting.
Latent Accuracy To gauge the quality of the learnt representations we follow previous work (Higgins et al., 2017; Kim & Mnih, 2018;
Shi et al., 2019; Sutter et al., 2021) and ﬁt a linear classiﬁer that predicts the input digit from the latent samples.
The accuracy of predicting the input digit using this classiﬁer indicates how well the latents can be separated in a linear manner.
In Figure 7, we plot the latent accuracy on MNIST and SVHN against the fraction of observation.
We can see that MEME outperforms MVAE on both MNIST and SVHN under the fully-observed scheme (i.e. when observation fractions is 1 . 0 ).
We can also notice that the latent accuracy of
MVAE is rather lopsided, with the performance on MNIST to be as high as 0 .
88 when only 1/16 of the data is observed, while SVHN predictions remain almost random even when all data are used; this indicates that MVAE relies heavily on MNIST to extract digit information.
On the other hand,
MEME’s latent accuracy observes a steady increase as observation fractions grow in both modalities.
It is worth noting that both models performs better on MNIST than SVHN in general—this is unsurprising as it is easier to disentangle digit information from MNIST, however our experiments here show that MEME does not completely disregard the digits in SVHN like MVAE does, resulting in more balanced learned representations.
It is also interesting to see that MVAE obtains a higher latent accuracy than MEME for low supervision rates.
This is due to MVAE learning to construct representations for each modality in a completely separate sub-space in the latent space, we provide a t-SNE plot to demonstrate this in Appendix H.1.
Ablation Studies To study the effect of modelling and data choices on performance, we perform two ablation studies: one varying the number of pseudo-samples for the prior, and the other evaluating how well the model leverages partially observed data over fully observed data.
We ﬁnd that performance degrades, as expected, with fewer pseudo-samples, and that the model trained with additional partially observed data does indeed improve.
See Appendix J for details.
4. E VALUATING R ELATEDNESS
Now that we have established that the representation learned by MEME contains rich class information from the inputs, we also wish to analyse the relationship between the encodings of different modalities by studying their “relatedness”, i.e. semantic similarity.
The probabilistic nature of the learned representations suggests the use of probability distance functions as a measure of relatedness, where a low distance implies closely related representations and vice versa.
In the following experiments we use the 2-Wasserstein distance, W 2 , a probability metric with a closed-form expression for Gaussian distributions (see Appendix F for more details).
Speciﬁcally, we compute d ij = W 2 ( q ( z | s i ) (cid:107) q ( z | t j ) ) , where q ( z | s i ) and q ( z | t j ) are the individual encoders, for all combination of pairs { s i , t j } in the mini-batch, i.e { s i , t j } , for i, j ∈ { 1 . . .
, M } where M is the number of elements in the mini-batch.
General Relatedness In this experiment we wish to highlight the disparity in measured relatedness between paired vs. unpaired multimodal data.
To do so, we plot d ij on a histogram and color-code the histogram by whether the corresponding data pair { s i , t j } shows the same concept, e.g. same digit for MNIST-SVHN and same image-caption pair for CUB.
Ideally, we should observe smaller distances between encoding distributions for data pairs that are related, and larger for ones that are not.
To investigate this, we plot d ij on a histogram for every mini-batch; ideally we should see higher densities at closer distances for points that are paired, and higher densities at further distances for unpaired points.
In Figure 8, we see that MEME (left) does in fact yields higher mass at lower distance values for paired multimodal samples (orange) than it does for unpaired ones (blue).
This effect is not so pronounced in MMVAE and not present at all in MVAE.
This demonstrates MEME’s capability of capturing relatedness between multimodal samples in its latent space, and the quality of its representation.
Class-contextual Relatedness To offer more insights on the relatedness of representations within classes, we construct a distance matrix K ∈ R 10 × 10 for the MNIST-SVHN dataset, where each element K i,j corresponds to the average W 2 distance between encoding distributions of class i of
MNIST and j of SVHN.
A perfect distance matrix will consist of a diagonal of all zeros and positive values in the off-diagonal.
See the class distance matrix in Figure 9 (top row), generated with models trained on fully observed multimodal data.
It is clear that our model (left) produces much lower distances on the diagonal, i.e. when input classes for the two modalities are the same, and higher distances off diagonal where input classes are different.
A clear, lower-valued diagonal can also be observed for MMVAE (middle), however it is less distinct compared to MEME, since some of the mismatched pairs also obtains smaller values.
The distance matrix for MVAE (right), on the other hand, does not display a diagonal at all, reﬂecting poor ability to identify relatedness or extract class information through the latent.
To closely examine which digits are considered similar by the model, we construct dendrograms to visualise the hierarchical clustering of digits by relatedness, as seen in Figure 9 (bottom row).
We see that our model (left) is able to obtain a clustering of conceptually similar digits.
In particular, digits with smoother writing proﬁle such as 3 , 5 , 8 , along with 6 and 9 are clustered together (right hand side of dendrogram), and the digits with sharp angles, such as 4 and 7 are clustered together.
The same trend is not observed for MMVAE nor MVAE.
It is also important to note the height of each bin, where higher values indicate greater distance between clusters.
Generally the clusters obtained in MEME are further separated for MMVAE, demonstrating more distinct clustering across classes.
D ISCUSSION
Here we have presented a method which faithfully deals with partially observed modalities in
VAEs.
Through leveraging recent advances in semi-supervised VAEs, we construct a model which is amenable to multi-modal learning when modalities are partially observed.
Speciﬁcally, our method employs mutual supervision by treating the uni-modal encoders individually and minimizing a KL between them to ensure embeddings for are pertinent to one another.
This approach enables us to successfully learn a model when either of the modalites are partially observed.
Furthermore, our model is able to naturally extract an indication of relatedness between modalities.
We demonstrate our approach on the MNIST-SVHN and CUB datasets, where training is performed on a variety of different observations rates.
Ethics Statement We believe there are no inherent ethical concerns within this work, as all datasets and motivations do not include or concern humans.
As with every technological advancement there is always the potential for miss-use, for this work though, we can not see a situation where this method may act adversarial to society.
In fact, we believe that multi-modal representation learning in general holds many beneﬁts, for instance in language translation which removes the need to translate to a base language (normally English) ﬁrst.
A D ERIVATION OF THE O BJECTIVE
The variational lower bound for the case when s and t are both observed, following the notation in
Figure 2, derives as:
[Equation]
Following Joy et al.
(2021), assuming s | = t | z and applying Bayes rule we have
[Equation]
where q φ,ϕ ( t | s
[Equation]
z which can be substituted into the lower bound to obtain
[Equation]
B E FFICIENT G RADIENT E STIMATION
Given the objective in (6), note that the ﬁrst term is quite complex, and requires estimating a weight ratio that involves an additional integral for q ϕ,φ ( t | s ) .
This has a signiﬁcant effect, as the naive
Monte-Carlo estimator of ∇ φ,ϕ E q φ ( z | s ) (cid:20)
[Equation]
can be very noisy, and prohibit learning effectively.
To mitigate this, we note that the ﬁrst term in (7) computes gradients for the encoder parameters ( φ, ϕ ) through a ratio of probabilities, whereas the second term does so through log probabilities.
Numerically, the latter is a lot more stable to learn from than the former, and so we simply drop the ﬁrst term in (7) by employing a stop gradient on the ratio q ϕ ( t | z ) q ϕ,φ ( t | s ) .
We further support this change with empirical results (cf.
Figure 10) that show how badly the signal-to-noise ratio (SNR) is affected for the gradients with respect to the encoder parameters.
We further note that Joy et al.
(2021) perform a similar modiﬁcation also motivated by an empirical study, but where they detach the sampled z —we ﬁnd that our simpliﬁcation that detaches the weight itself works more stably and effectively.
C W EIGHT S HARING
Another critical issue with na¨ıvely training using (6), is that in certain situations q ϕ ( t | z ) struggles to learn features (typically style) for t , consequently making it difﬁcult to generate realistic samples.
This is due to the information entering the latent space only coming from s , which contains all of the information needed to reconstruct s , but does not necessarily contain the information needed to reconstruct a corresponding t .
Consequently, the term p θ ( s | z ) will learn appropriate features (like a standard VAE decoder), but the term q ϕ ( t | z ) will fail to do so.
In situations like this, where the information in t is not subsumed by the information in s , there is no way for the model to know how to reconstruct a t .
Introducing weight sharing into the bidirectional objective (2) removes this issue, as there is equal opportunity for information from both modalities to enter the latent space, consequently enabling appropriate features to be learned in the decoders p
θ ( s | z ) and p ϕ ( t | z ) , which subsequently allow cross generations to be performed.
Furthermore, we also observe that when training with (2) we are able to obtain much more balanced likelihoods Table 2.
In this setting we train two models separately using (6) with s =
MNIST and SVHN and then with t = SVHN and s = MNIST respectively.
At test time, we then ‘ﬂip’ the modalities and the corresponding networks, allowing us to obtain marginal likelihoods in each direction.
Clearly we see that we only obtain reasonable marginal likelihoods in the direction for which we train.
Training with the bidirectional objective completely removes this deﬁciency, as we now introduce a balance between the modalities.
[Table]
D R EUSING A PPROXIMATE P OSTERIOR MC S AMPLE
When approximating q ϕ,φ ( t | s ) through MC sampling, we ﬁnd that it is essential for numerical stability to include the sample from the approximate posterior.
Before considering why, we must ﬁrst outline the numerical implementation of q ϕ,φ ( t | s ) , which for K samples z 1: K ∼ q φ ( z | s ) is computed using the LogSumExp trick as:
[Equation]
where the ratio q ϕ ( t | z ) q ϕ
[Equation]
φ ( t | s ) is computed as exp { log q ϕ ( t | z ) − log q ϕ,φ ( t | s ) } .
Given that the
LogSumExp trick is deﬁned as:
[Equation]
where x ∗ = max { x 1 , . . .
, x N } .
The ratio will be computed as q ϕ ( t | z ) q
[Equation]
where z ∗ = arg max z 1: K log q ϕ ( t | z k ) .
For numerical stability, we require that log q ϕ ( t | z ) (cid:54)(cid:29) log q ϕ ( t | z ∗ ) , otherwise the computation may blow up when taking the exponent.
To enforce this, we need to include the sample z into the LogSumExp function, doing so will cause the ﬁrst two terms to either cancel if z = z ∗ or yield a negative value, consequently leading to stable computation when taking the exponent.
E E XTENSION BEYOND THE B I -M ODAL CASE
Here we offer further detail on how MEME can be extended beyond the bi-modal case, i.e. when the number of modalities M > 2 .
Note that the central thesis in MEME is that the evidence lower bound (ELBO) offers an implicit way to regularise different representations if viewed from the posteriorprior perspective, which can be used to build effective multimodal DGMs that are additionally applicable to partially-observed data.
In MEME, we explore the utility of this implicit regularisation in the simplest possible manner to show that a direct application of this to the multi-modal setting would involve the case where M = 2 .
The way to extend, say for M = 3 , involves additionally employing an explicit combination for two modalities in the prior (instead of just 1).
This additional combination could be something like a mixture or product, following from previous approaches.
More formally, if we were to denote the implicit regularisation between posterior and prior as R i ( ., . ) , and an explicit regularisation function
R e ( ., . ) , and the three modalities as m 1 , m 2 , and m 3 , this would mean we would compute
[Equation]
assuming that R e was commutative, as is the case for products and mixtures.
There are indeed more terms to compute now compared to M = 2 , which only needs R i ( m 1 , m 2 ) , but note that R i is still crucial—it does not diminish because we are additionally employing R e .
As stated in prior work(Suzuki et al., 2016; Wu & Goodman, 2018; Shi et al., 2019), we follow the reasoning that the actual number of modalities, at least when considering embodied perception, is not likely to get much larger, so the increase in number of terms, while requiring more computation, is unlikely to become intractable.
Note that prior work on multimodal VAEs also suffer when extending the number of modalities in terms of the number of paths information ﬂows through.
We do not explore this setting empirically as our priary goal is to highlight the utility of this implicit regularisation for multi-modal DGMs, and its effectiveness at handling partially-observed data.
F C LOSED F ORM EXPRESSION FOR W ASSERTEIN D ISTANCE BETWEEN TWO G AUSSIANS
The Wassertein-2 distance between two probability measures µ and ν on R n is deﬁned as
[Equation]
with X ∼ µ and Y ∼ ν .
Given µ = N ( m 1 , Σ 1 ) and ν = N ( m 2 , Σ 2 ) , the 2-Wassertein is then given as
[Equation]
For a detailed proof please see (Givens & Shortt, 2002).
G C ANONICAL C ORRELATION A NALYSIS
Following Shi et al.
; Massiceti et al.
(2018), we report cross-coherence scores for CUB using Canonical Correlation Analysis (CCA).
Given paired observations x 1 ∈ R n 1 and x 2 ∈ R n 2 ,
CCA learns projection weights W T 1 ∈ R n 1 × k and W T 2 ∈ R n 2 × k which minimise the correlation between the projections W T 1 x 1 and W T 2 x 2 .
The correlations between a data pair { ˜ x 1 , ˜ x 2 } can thus be calculated as
[Equation]
(12) where φ ( x n ) = W Tn ˜ x n − avg ( W Tn ˜ x n ) .
Following Shi et al.
(2019), we use feature extractors to pre-process the data.
Speciﬁcally, features for image data are generated from an off-the-shelf ResNet-101 network.
For text data, we ﬁrst ﬁt a FastText model on all sentences, resulting in a 300- d projection for each word Bojanowski et al.
[Table]
H A DDITIONAL R ESULTS H. MVAE L ATENT A CCURACIES
The superior accuracy in latent accuracy when classifying MNIST from MVAE is due to a complete failure to construct a joint representation, which is evidenced in its failure to perform crossgeneration.
Failure to construct joint representations aids latent classiﬁcation, as the encoders just learn to construct representations for single modalities, this then provides more ﬂexibility and hence better classiﬁcation.
In Figure
[Table]
we further provide a t-SNE plot to demonstrate that MVAE places representations for MNIST modality in completely different parts of the latent space to SVHN.
Here
[Table]
bill
[Table]
[Figure]
this is white a all is with flat beak and black a , ’s a curved for light has a body is.
[Figure]
we can see that representations for each modality are completely separated, meaning that there is no shared representation.
Furthermore, MNIST is well clustered, unlike SVHN.
Consequently it is far easier for the classiﬁer to predict the MNIST digit as the representations do not contain any information associated with SVHN.
[Table]
[Table]
002 ± 0.001 -0.000 ± 0.004 0.000 ± 0.
0.001 ± 0.
MEME SPLIT 0.106 ± 0.000 0.064 ± 0.
0.046 ± 0.
0.031 ± 0.
0.027 ± 0.
MVAE SPLIT 0.106 ± 0.000 -0.002 ± 0.001 0.000 ± 0.
0.000 ± 0.005 -0.001 ± 0.
Caption → Image
Model
GT f
= 1 .f
= 0 .f = 0 .f
= 0 .
MEME Image 0.106 ± 0.000 0.074 ± 0.
0.058 ± 0.
0.051 ± 0.
0.046 ± 0.
MMVAE Image 0.106 ± 0.000 0.058 ± 0.
- - - MVAE Image 0.106 ± 0.000 -0.002 ± 0.001 -0.002 ± 0.000 -0.002 ± 0.001 -0.001 ± 0.
Ours Captions 0.106 ± 0.000 0.074 ± 0.
0 .059 ± 0.
0.050 ± 0.
0.053 ± 0.
MMVAE Captions 0.106 ± 0.000 0.058 ± 0.
- - - MVAE Captions 0.106 ± 0.000 0.002 ± 0.001 -0.001 ± 0.002 -0.003 ± 0.002 -0.002 ± 0.
MEME SPLIT 0.106 ± 0.000 0.074 ± 0.
0.061 ± 0.
0.047 ± 0.
0.049 ± 0.
MVAE SPLIT 0.106 ± 0.000 -0.002 ± 0.001 -0.002 ± 0.002 -0.002 ± 0.001 -0.002 ± 0.
H.
G ENERATIVE C APABILITY
We report the mutual information between the parameters ω of a pre-trained classiﬁer and the labels y for a corresponding reconstruction x .
The mutual information gives us an indication of the amount of information we would gain about ω for a label y given x , this provides an indicator to how outof-distribution x is.
If x is a realistic reconstruction, then there will be a low MI, conversely, an un-realistic x will manifest as a high MI as there is a large amount of information to be gained about ω .
The MI for this setting is given as
[Equation]
Rather than using dropout Gal (2016); Smith & Gal (2018) which requires an ensemble of multiple classiﬁers
[Table]
we instead replace the last layer with a sparse variational GP.
This allows us to estimate p ( y | x, D ) =
(cid:82) p ( y | x, ω ) p ( ω | D )
d ω using Monte Carlo samples and similarly estimate E p ( ω |D )
[ H [ p ( y | x , ω )]] .
We display the MI scores in Table 6, where we see that our model is able to obtain superior results
[Table] [Table]
SNE
[Table]
LOTS W HEN
[Table]
BOTH
[Table]
In Figure 20 we can see that partially observing MNIST leads to less structure in the latent space.
[Table]
I MMVAE BASELINE WITH L APLACE P OSTERIOR AND P RIOR
The difference in results between our implementation of MVAE and the ones in the paper (Shi et al., 2019), is becuase we restrict MEME to use Gaussian distributions for the posterior and prior, and therefore we adopt Gaussian posteriors and priors for all three models to ensure like-for-like comparison.
Better results for MMVAE can be obtained by using Laplace posteriors and priors, and In Table 7 we display coherence scores using our implementation of MMVAE using a Laplace posterior and prior.
Our implementation is inline with the results reported in Shi et al.
, indicating that the baseline for MMVAE is accurate.
J A BLATION S TUDIES
Here we carry out two ablation studies to test the hypotheses: 1) How sensitive is the model to the number of pseudo samples in λ and 2)
What is the effect of training the model using only paired data for a given fraction of the dataset.
J. S ENSITIVITY TO NUMBER OF PSEUDO - SAMPLES
In Figure 21 we plot results where the number of pseudo samples is varied for different observation rates.
Ideally we expect to see the results decrease in their performance as the number of pseudo-samples is minimised.
This is due to the number of components being present in the mixture
p λ t ( z ) =
1 N (cid:80) Ni =
1 p ψ ( z | u t i ) , also being decreased, thus reducing the its ability to approximate the true prior p ( z ) =
(cid:82) t p ψ ( z | t ) p ( t ) dt .
As expected lower observation rates are more sensitive, due to a higher dependence on the prior approximation, and a higher number of pseudo samples typically leads to better results.
J. T RAINING USING ONLY PAIRED DATA
Here we test the models ability to leverage partially observed data to improve the results.
If the model is successfully able to leverage the partially observed samples, then we should see a decrease in the efﬁcacy if we train the model using only paired samples, i.e. a model trained with 25% paired and 75% partially observed should perform improve the results over a model trained with only the 25% paired data.
In other words we omit, the ﬁrst two partially observed terms in (5), discarding D s and D t .
In Figure 22 we can see that the model is able to use the partially observed modalities to improve its results.
K T RAINING D ETAILS
The architechtures are very simple and cean easily be implemented in popular deep learning frameworks such as Pytorch and Tensorﬂow.
However, we do provide a release of the codebase at the following location: https://github.com/thwjoy/meme.
MNIST-SVHN We provide the architectures used in Table 8b and Table 8a.
We used the Adam optimizer with a learning rate of 0 . 0005 and beta values of (0 . 9 , 0 . 999) for 100 epochs, training consumed around 2Gb of memory.
CUB We provide the architectures used in Table 8c and Table 8d.
We used the Adam optimizer with a learning rate of 0 . 0001 and beta values of (0 . 9 , 0 . 999) for 300 epochs, training consumed around 3Gb of memory. [Figure] [Table] [Figure]
[Table]