Paper:
Multimodal Generative Models for Scalable Weakly-Supervised Learning
Multiple modalities often co-occur when describing natural phenomena. Learning a joint representation of these modalities should yield deeper and more useful representations. Previous generative approaches to multi-modal input either do not learn a joint distribution or require additional computation to handle missing data. Here, we introduce a multimodal variational autoencoder (MVAE) that uses a product-of-experts inference network and a sub-sampled training paradigm to solve the multi-modal inference problem. Notably, our model shares parameters to efficiently learn under any combination of missing modalities. We apply the MVAE on four datasets and match state-of-the-art performance using many fewer parameters. In addition, we show that the MVAE is directly applicable to weaklysupervised learning, and is robust to incomplete supervision. We then consider two case studies, one of learning image transformations-edge detection, colorization, segmentation-as a set of modalities, followed by one of machine translation between two languages. We find appealing results across this range of tasks.

Review:
This paper presents a generative approach to multimodal deep learning based on a product-of-experts (PoE) inference network. The main idea is to assume the joint distribution over all modalities factorises into a product of single-modality data-generating distributions when conditioned on the latent space, and use this to derive the structure and factorisation of the variational posterior. The proposed model shares parameters to efficiently handle any combination of missing modalities, and experiments indicate the model’s efficacy on various benchmark datasets.  The idea is intuitive, the exposition is well-written and easy to follow, and the results are thorough and compelling. I have a few questions / comments, mainly about the relationship of this work with respect to previous approaches ([15] and [21] in the text).  Firstly, the following line in the introduction is incorrect: “While fully-supervised deep learning approaches [15, 21] can learn to bridge modalities, generative approaches promise to capture the joint distribution across modalities and flexibly support missing data”. Both [15] and [21] are unsupervised generative approaches used to model the joint distribution, and have the machinery to deal with missing modalities. Furthermore, [21] employs a restricted Boltzmann machine, which is itself a product-of-experts model, so I would like to see a better justification for how the PoE is incorporated in the proposed approach, and how this PoE differs from [21].  Secondly, the sub-sampled training paradigm in section 2.2 is also quite similar to the denoising training approaches used in [15] and [21] (which drop out modalities randomly), so I think this could be acknowledged and the proposed approach contrasted. Lines 106-107 are also a bit unclear: is the procedure to compute the ELBO using either (1) all modalities, (2) a single modality, or (3) k randomly selected modalities; each with equal probability? I would rewrite that sentence for clarity.  Lastly, two related points: (1) while the approach is benchmarked against BiVCCA, JMVAE, and CVAE, there are no comparisons some of the less recent approaches to multimodal deep learning ([15] and [21]); (2) while the experiments are comprehensive on MNIST and CelebA, these benchmarks are a little contrived in the multimodal learning case, and I believe the paper would be strengthened with a result on a more real-world multimodal problem. An additional experiment on the MIR Flickr dataset would address both of these points.  Minor nitpicks: - Figure 1 caption talks about PoE, but the diagram refers to PoG. - Line 126, I would change “we’ve argued...” to a more formal tone. Ie. “we have previously argued in Section 2...”  [15] Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. Multimodal deep learning. In Proceedings of the 28th international conference on machine learning (ICML-11), pages 689–696, 2011. [21] Nitish Srivastava and Ruslan R Salakhutdinov. Multimodal learning with deep boltzmann machines. In Advances in neural information processing systems, pages 2222–2230, 2012.   --AFTER REBUTTAL-- I have read the authors' response and other reviews and my overall assessment has not changed, but there is one critical aspect that should be addressed.  After discussions with the AC and reviewers, we believe the claim in line 73 is not correct - the correct posterior p being factorised doesn't necessarily mean the optimal approximate posterior q when using a given family of distributions for q will also be factorised. I the argument should be changed to indicate the factorisation of p was inspiration for the form of q rather than claiming the "correct" q must be factorised, and making the argument specific to the multi-modality application as well.  I have a number of other comments that I feel should be addressed before the final version: 1) The authors recognise that comparison to an RBM is interesting (given it is also a PoE) and promise a new RBM baseline in the final draft. To clarify, I think this should be the same deep architecture, trained with a shared RBM representation, rather than just a single layer RBM, to make it a fair comparison (something like [21]. Obviously the proposed approach is end-to-end and does not require layer-wise training like a Deep Boltzmann machine, an important distinction that should be emphasised.  2) The response document states that "the shared representation in [15] is not a probabilistic measure", but this does not prevent it from being a generative model as the modality-specific latent representations are stochastic and it is still modelling the data-generating distribution. As such, I think it would be incorrect to state (as in the original draft), that [15] and [21] are not generative models. I believe this should be rectified for both, not just for [21]. 3) The additional text translation task is compelling - but I assume that the baseline performances will be reported on this task as well? 4) My comment on the sub-sampling training of the model was, I believe, only partially addressed. In particular, I still think the section detailing this process could be written more clearly, in terms of what data is used and how this relates to previous "denoising training" procedures (eg. [15] and [21]), which I believe are effectively sub-sampling in any case.  All in all, I think this would be a good addition to the conference.