Paper: Score-Based Multimodal Autoencoders (ICLR-2024)

Reviewer comment (Novelty):
Missing connection and comparison of the methods in the related work section to the proposed method. In what way are SBM-* improving over these methods? I also find that such discussion is missing in the experimental results.

Author response:
*“Missing connection and comparison of the methods in the related work section to the proposed method. In what way are SBM improving over these methods?...“*

Multimodal VAEs (MoPoE, MVAE, MMVAE, MMVAE+, MVTCAE) learn a joint representation (newer approaches like MMVAE+ added modality-specific representation as well) and forcing the decoder to use this joint representation reduces the generation quality of each modality while helping to improve the coherence (we have seen a similar effect with fine-tuning our decoder to increase coherence see Section A.4.2). Our work, on the other hand, does not learn a joint representation but learns a joint distribution over individual representation, this way the decoder and individual learned representation remains intact and decoder uses only modality-specific representation. In addition, our method exhibits superior coherence in unconditional generation. When no modality has been observed, previous work fails to recover the joint representation. In contrast, our approach samples from a valid joint distribution, resulting in enhanced coherence.

Reviewer comment (Experiments):
No ablation studies on the components of the proposed method. This would also help gaining a more intuitive understanding of the proposed method, which is lacking in the current version.

Author response:
*“No ablation studies on the components of the proposed method.”*

The main components of our approach are the score-based model and coherence-guidance EBM. We already have included the results with and without EBM and we have studied the scalability of the score-based model in the presence of a different number of modalities (when trained with 2, 3, all the way up to 10 modalities shown in Figure 8 - (a,b,c)). 
In the revised version, we also added Table 4 to the paper to explain the effects of beta on the score-based model.
Additionally, we added another section to explore the effect of decoder fine-tuning to improve the coherence. 
If you can think of other ablation studies that are missing from our paper, we are willing to include them.

Reviewer comment (Experiments):
Limited experimental evaluation using only image data. Both PolyMNIST and CelebAMask are image datasets. It would be beneficial to add experiments on text audio modalities as well.

Author response:
*“Limited experimental evaluation using only image data. Both PolyMNIST and CelebAMask are image datasets. It would be beneficial to add experiments on text audio modalities as well”*

We chose PolyMnist for its simplicity in performing experiments on multiple modalities. Studying the behavior of multimodal VAEs in the presence of different numbers of missing modalities requires a tractable dataset and problem setting. 
We also added CelebA-MASK-HQ with the attributes as an additional non-image modality. In addition to that, we have added audio-image modality from the MHD dataset [1] (please see the revised version).

Reviewer comment (Novelty):
I find the novelty of the method a bit limiting. Given that the experimental results do not report any standard deviations makes it hard to judge the weight of the contribution.

Author response:
*“I find the novelty of the method a bit limiting. Given that the experimental results do not report any standard deviations makes it hard to judge the weight of the contribution.”*

In our graphs, we plot the standard deviation as a shade in the figures (more visible when zoomed). We report the mean over three-experiment runs on our figures and tables. We have edited the table to include the standard deviation.

Reviewer comment (Evaluation):
It has been shown by several works that FID does not provide an adequate evaluation of a generative model (see for example work by Sajjadi et al Assessing Generative Models via Precision and Recall). Are you sure that none of the considered models experience a mode collapse? It would be better to present these results with newer metrics like Improved Precision and Recall [1] or Delaunay Component Analysis [2] below). With the latter, you could also gain more insights into each generated modality by analysing the geometry of their representations.  

[1] Kynkäänniemi et al, Improved Precision and Recall Metric for Assessing Generative Models, NeurIPS 2019  

[2] Poklukar et al, Delaunay Component Analysis for Evaluation of Data Representations, ICLR 2022

Author response:
*“It has been shown by several works that FID does not provide an adequate evaluation of a generative model…”*

Despite the extensive research for introducing alternative metrics for evaluating generative quality for images, FID has been extensively used in different generative model evaluations [2,3,4] (to name a few), including in all of the baselines in Multimodal VAEs, mostly to achieve comparison consistency.  

*“It would be better to present these results with newer metrics like Improved Precision and Recall [1] or Delaunay Component Analysis [2] below)...”*

Thank you for your suggestions, however, it is worth noting that our method is not learning a joint representation for data modalities and we rely on the learned representation from individual VAEs for each modality. We are claiming that learning the correlation among latent representations of individual modalities is enough to achieve coherence.  

*“ Are you sure that none of the considered models experience a mode collapse? “*

We have added section A.5 to investigate the possibility of mode collapse. We unconditionally generate all the modalities 10000 times and count the present labels using a pretrained classifier. All methods cover all modes (different digits) for each modality, however, MVTCAE has less balanced coverage compared to our approach and MMVAE+ .

Reviewer comment (Evaluation):
When calculating conditional accuracy, do you randomise the observed modalities? For example, in Fig 5, how many times did you repeat this experiment? I believe that without std it is hard to draw conclusions.

Author response:
*“When calculating conditional accuracy, do you randomize the observed modalities? For example, in Fig 5, how many times did you repeat this experiment? I believe that without std it is hard to draw conclusions.”*

We have reported the standard deviation as shades under the main graph (we ran each  3 times). We use the whole test dataset to calculate the results shown, so yes, multiple data points are given to generate the results. We also revised the tables to include standard deviation.

Reviewer comment (Experiments):
I do not see the benefits of keeping both SBM-VAE and SBM-RAE in the experiments. First, it is not clear what version is better and why. The authors do not discuss this at all. Second, it hinders the readability of the results.

Author response:
*“I do not see the benefits of keeping both SBM-VAE and SBM-RAE in the experiments.”*

Actually, there is no dominant version across all experiments. Our method is independent of how the individual latent representations are being learned, so we show this flexibility by combining it with both variational autoencoders and regularized deterministic autoencoders.

Reviewer comment (Evaluation):
I do not fully understand the unconditional coherence evaluation. Do you sample z_1:M, then generate all x_1:M modalities and evaluate their agreement? In Fig 6 what are ”similar modalities”? Please add some more explanation.

Author response:
*“I do not fully understand the unconditional coherence evaluation”*

Yes, your interpretation is correct. They are first initialized from Gaussian noise. We then sample from the score-based models by starting from the noise and iteratively updating the samples. Final samples are fed to each decoder to generate images. The similar modalities are the number of modalities that agree (same digit) as classified by a pre-trained classifier.

Reviewer comment (Theory):
To me it is unclear why the score-matching model would lead to a coherent shared latent space, and I would appreciate if authors would clarify that. While in the first step of training we have a prior \( p(z_{1:M})=\prod \mathcal{N}(0, \sigma I) \), what are the modelling assumptions on the parametric prior \( p_{\theta}(z_{1:M}) \) in the second step?

Author response:
*“To me it is unclear why the score-matching model would lead to a coherent shared latent space, and I would appreciate if authors would clarify that…”*

Our approach leverages the fact that SBM implicitly learns the underlying data distribution by capturing the correlation among their input variables, which are here the latent representation learned by uni-variate VAEs. SBMs offer tractable joint or marginal sampling, which has been used to sample the latent representation of a missing modality. Therefore, our approach does not require learning an explicit joint representation which is common in the prior multi-modal VAEs.

 *“what are the modelling assumptions on the parametric prior in the second step?”*

In the second step we learn the parametric prior which has been parameterized by a UNET.
We don’t enforce any specific assumption for training the SBM.

Reviewer comment (Evaluation):
The comparison with MVTCAE [2] from e.g. Figures 4 and 5 is quite important and should be further commented, with maybe more insights (e.g., different \( \beta \) values, average performance across modalities for conditional generation, see below) for at least two reasons. First, as it does not sub-sample modalities during training, MVTCAE is not subject to the limitations for generative quality uncovered in [1]. Second, MVTCAE and the proposed model are really similar in performance, and from the results it is unclear which one performs better overall (at least in the first experimental setting).

Author response:
*“dfferent $\beta$ values, average performance across modalities for conditional generation...”*

We have added Table 3 in the paper listing the average performance (average of each modality given the rest) of each model as well as the performance for individual modalities in Figure 13 and Figure 14.  MVTCAE results in better accuracy while having a higher FID score due to existing joint representation. When none of the modalities has been observed (unconditional generation), our approach is superior to all of the prior multimodal VAEs including MVTCAE. This is a very important property in scenarios like generating a piece of coherent music consisting of multiple instruments (with each instrument as a modality).

Reviewer comment (Evaluation):
Why reporting only the results on the last modality in e.g. Figure 4? (Even though also results on the first modality are available, in the Appendix). This is only a partial insight on model performance, and results should be averaged across modalities for conditional generation (not modalities used for inference, modalities used for generation) to show performance is consistently good. To back up this point, results on the third modality (Figure 3) indicate that MVTCAE has poor generative quality. However, FIDs in Figure 4 for the last modality indicate otherwise. Hence, it seems that in evaluating model performance, one might want to control for the effect of choice of modality.

Author response:
*“Why reporting only the results on the last modality in e.g. Figure 4?” *

We have added Table 3, Figure 13, and Figure 14 in the revised version.

Reviewer comment (Experiments):
Did the authors do an ablation for different values of \( \beta \) for the compared models in e.g. Extended PolyMNIST? In the Appendix it is stated that the \( \beta \) was chosen using the validation set. How exactly was it chosen? Should it be by looking at the ELBO value on the validation set, one should be careful as likelihood values prove often not to be representative when it comes to performance of multimodal VAEs. For instance, with models that subsample modalities, high likelihoods for conditional reconstruction across modalities can be obtained by producing average-looking samples (since the modality-specific information about the sample to be reconstructed cannot be inferred). I think it would be important to report results for the compared models across \( \beta \) values instead of reporting results only for a single value. To back up this point, the conditional generation results from Figure 3 for e.g. MVTCAE seem rather different from what reported in the original work [2] on PolyMNIST, in which the authors use a much higher \( \beta \) value (\( \beta=2.5 \) I think).

Author response:
*“Did the authors do an ablation for different values of beta for the compared models in e.g. Extended PolyMNIST”*

Please see the revised section of A.4.2

*“In the Appendix it is stated that the model was chosen using the validation set. How exactly was it chosen”*

Models were selected using a combination of the task metrics (fid and coherence in both conditional and unconditional settings) to address the existing tradeoff. Note that MVTCAE has a tradeoff between conditional and unconditional performance with respect to different beta. For example, MVTCAE performs better conditionally with lower beta but worse unconditionally. The experiments can be replicated using the zip file we have attached.

Reviewer comment (Writing):
I would change "alternative" to "novel" in the Abstract. Not really clear what the approach is "alternative" to otherwise.

Author response:
*“I would change "alternative" to "novel" in the Abstract. Not really clear what the approach is "alternative" to otherwise.”*

We employed the term 'alternative' to emphasize the distinctiveness of our method compared to conventional Multimodal VAE techniques, which typically rely on joint representations.

Reviewer comment (Writing):
In the Introduction, I can suggest to make the difference between "Scalability" and "Conditional modality gain" clearer. As I understand it, scalability refers to the fact that a multimodal VAE trained on a given number of modalities should perform at least as well as the same model trained on fewer modalities. On the other hand, conditional modality gain refers to test-time conditional generation performance improving when more modalities are given for inference. I would suggest to make a clearer distinction in the text between the two concepts.

Author response:
*"Scalability" and "Conditional modality gain":*

Thank you for pointing that out. We refer to Scalability as computational efficiency when adding modalities while performing comparably to a model trained with a lower number of modalities. We have edited that line to make it more clear.