Paper: Multi-modal Latent Diffusion (ICLR-2024)

Reviewer comment (Theory):
The paper employs diffusion on latent variables concatenated from different modalities. One concern is whether the data distributions corresponding to vectors from these various modalities significantly diverge. If so, does utilizing the same noise schedule result in a lack of synchronization between the denoising and noise-adding processes across the different modalities?

Author response:
We can analyze the problem separating two different time regimes: large diffusion times, and small ones. 

Large diffusion times **cannot** be the source of synchronization problems. Indeed it is widely known that under loose assumptions [Collet2008], the discrepancy (in KL sense) between time varying probability densities and their final (Gaussian in this case) steady state distributions decreases exponentially fast. In the language of our work: $KL(q(r^{A_1},t) | \rho(r^{A_1})\leq C\exp(-\lambda t)$, where the parameters $C,\lambda>0$ are independent of time. This result holds for any starting distribution $q(r^{A_1},0)$ [Collet2008]. 

For small diffusion times, the question of whether different modalities respond differently to noise injection is important. In Appendix A.2, “Latent space robustness against diffusion perturbation” of the submitted paper, we discuss in length about this matter (see also Figure 5). 

It is important to notice that it is this very question, together with the information theoretic analysis presented in Section 4.2, that motivates the multi-time diffusion process as an alternative to the inpainting approach (which is based on a single diffusion time).

* [Collet2008] J.F. Collet and F. Malrieu. Logarithmic sobolev inequalities for inhomogeneous markov semigroups. ESAIM: Probability and Statistics, 12:492–504, 2008

Reviewer comment (Experiments):
The optimization speeds of different modalities may inherently vary, leading to discrepancies in performance. The proposed method might achieve satisfactory results with simplistic datasets, but training becomes substantially more challenging when scaled to extensive, real-world data scenarios.

Author response:
We respectfully disagree, this concern is speculative. In our experiments, we have not observed any issues related to the optimization speed raised by different modalities. The performance of our method has remained robust and consistent across various datasets.

As explained in the submitted version of the paper, we “ avoid any form of interference in the back-propagated gradients corresponding to the uni-modal reconstruction losses. Consequently, gradient conflict issues [Javaloy et al., 2022], where stronger modalities pollute weaker ones, are avoided.” (see Section 3, below Equation 2 in our paper).

The only impact of scaling to larger, higher-resolution datasets is on the computational cost of the (single, amortized) score network, which scales with the dimensions of the concatenated latent sizes. However, this does not inherently make training more challenging from a methodological standpoint.

Reviewer comment (Experiments):
Within the model's framework, a discrepancy arises between the training and inference stages in terms of the number of modalities. For instance, during training, the model might handle three modalities: A, B, and C. However, in a scenario where inference is desired based solely on modality A to predict B, would a masked C still be necessitated? This raises questions about the model's flexibility and adaptability to accommodate various generative scenarios with different modalities, and its capacity to dynamically adjust without compromising the integrity of the generative process.

Author response:
In the scenario you describe, where the model is trained on three modalities (A, B, and C) but inference is desired based on modality A to “predict” B, it is not necessary to involve modality C. If the goal is to ensure that modality C does not influence the generation from A to B, our method simply considers C as a missing modality during the inference stage. This flexibility is an integral part of our model's design, ensuring that it can dynamically adjust to different conditions without compromising the integrity of the generative process.

Reviewer comment (Evaluation):
The quantitative comparisons on the CUB dataset should be integrated into the main text. The coherence metric for image->caption has not improved, necessitating a comprehensive comparative analysis and case demonstrations of caption generation. Additionally, the visual representations in this section of the paper are unclear, making them difficult to interpret.

Author response:
Due to space limitations, we cannot move the CUB experiments to the main. The reviewer indicates that results in table 16, Appendix E.4 show that in one particular case, image->caption, the coherence of MLD is not the best. A careful look at the table shows that the performance of MLD in this task is on par with alternatives, and better for all other tasks (caption-> image, joint quality, caption->image quality). 

Image to caption is a very difficult task, for which even specialized models [Li2023] relying on LLMs require metrics that only loosely measure coherency, and prompt hints that instill information for text generation. We believe this is an exciting area of research, but it is outside the scope of our objectives in this paper.

The visual representations we display in the main and appendix for CUB images are in-line with the results obtained by all the works we compare to ( see fig. 7 of [Palumbo2023], see fig 4 of [Daunhawer2022]). Simply put, image generation quality suffers heavily in most of the competitors, and when it does not, coherency is very bad. This is exactly the quality/coherence tradeoff that has been pointed out in the literature and that we study in our work.

* [Li2023] J. Li, et al. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. https://arxiv.org/abs/2301.12597
* [Palumbo2023] E. Palumbo, et al. MMVAE+: Enhancing the Generative Quality of Multimodal VAEs Without Compromises, ICLR 2023, https://openreview.net/pdf?id=BYHy9WwxFU
* [Daunhawer2022] Imant Daunhawer, et al. On the Limitations of Multimodal VAEs, ICLR 2022, https://arxiv.org/pdf/2110.04121.pdf

Reviewer comment (Experiments):
Considering that the differences in certain metrics on the CUB dataset are not particularly pronounced, it is recommended to augment the study with a comparative analysis on the Bimodal CelebA dataset.

Author response:
Thank you for the suggestion! We performed additional experiments (see Appendix E.5) with the CelebAMask-HQ dataset [Lee2020] as done in [Wesego2023], which consists of face images, each having a segmentation mask, and attributes, so 3 modalities. Our results are clear: the method we propose outperforms competitors from the state of the art, even with this new dataset.

* [Lee2020] Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image manipulation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
* [Wesego2023] Daniel Wesego and Amirmohammad Rooshenas, Score-Based Multimodal Autoencoders. https://arxiv.org/abs/2305.15708

Reviewer comment (Reproducibility):
The authors have not made their code available, which makes it hard to reproduce the experimental results.

Author response:
We have now made code available by uploading it as an anonymous zip supplement.

Reviewer comment (Writing):
Clarity in Presentation: While the paper is comprehensive, certain sections, especially those with plenty of mathematical formulations, might benefit from further simplification or more intuitive explanations for a broader audience.

Author response:
We do agree that the formulation might hinder accessibility of our work to a broader audience. However, we think that the mathematical rigor we adopt is a requirement for a venue like ICLR. To align with the reviewer’s request, we can commit to writing a new section in the Appendix called “Intuitive summary” which we hope will help the readers grasp at an intuitive level the key ideas of our method, without using the language of mathematics. Please refer to Appendix A.5 in the new version of the paper for a preliminary draft of such a discussion.

Reviewer comment (Experiments):
Dataset Diversity in Experiments: The current experiments focus primarily on simple or low-resolution datasets. Would the MLD approach's efficacy be consistent when tested on more popular, high-resolution datasets? Expanding the experimental evaluation to include such datasets might enhance the paper's applicability and appeal to the broader research community.

Author response:
In our research, we aim to demonstrate that the widely recognized concept of "latent fusion" through the product of experts, mixture of experts or combination thereof from the literature is not sufficient, particularly in terms of balancing generative quality and coherence. Our approach, which uses joint latent diffusion processes, effectively enables a new form of "latent fusion", whereby latent modalities jointly coevolve in the reverse, generative process. The datasets we selected for this study are not just adequate but crucial for drawing a meaningful comparison with the current state-of-the-art methods in "latent fusion". 

In light of a similar suggestion from Reviewer SqcE, we performed additional experiments (**see Appendix E.5**) with the **CelebAMask-HQ dataset** [Lee2020] as done in [Wesego2023], which consists of face images, each having a segmentation mask, and attributes, so 3 modalities. Our results are clear: the method we propose outperforms competitors from the state of the art, even with this new dataset.
Other kinds of datasets, for example those including videos, are not appropriate for our goal because computing the coherence metric is not currently possible.

* [Lee2020] Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image manipulation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
* [Wesego2023] Daniel Wesego and Amirmohammad Rooshenas, Score-Based Multimodal Autoencoders. https://arxiv.org/abs/2305.15708

Reviewer comment (Experiments):
Real-world Applications: The paper could be enriched by providing more real-world applications or use-cases to showcase the practical significance of MLD.

Author response:
In our exploration of the industrial applications of MLD, we have directed our focus toward the automotive industry, recognizing the potential impact of our research in this field. Specifically, we have conducted extensive experiments of MLD on automotive-related datasets. However, due to the private nature of these datasets, we are unable to share them publicly and have consequently omitted these specific tests from our paper.

Despite this limitation, we can share insights into the practical use-cases we investigated. One key application of MLD in automotive technology is in mitigating the issues caused by faulty car sensors. In these applications, the text modality is absent, which hinders the use of many available pre-trained generative models (see also a comment by Reviewer 865d). Our model has shown promising results for in-car applications such as parking assistance, especially in situations where sensors are either faulty or their vision is occluded. These tests showcase the robustness and adaptability of MLD in handling real-world challenges in the automotive sector.

Reviewer comment (Reproducibility):
How is the architecture of the autoencoders, such as the image AE and text AE, designed? In the context of stable diffusion, pure convolutional networks are utilized for both encoding and decoding. Does the MLD approach adopt a similar design for images?

Author response:
In our MLD approach, we have emphasized the importance of tailoring each autoencoder to its specific input modality to achieve best performance. This principle is reflected in our design choices for image, audio, and text autoencoders. We apologize if references to supplementary material were not clear, but all details concerning the architectures are already reported in Appendix D. For the case of images, we consider the same convolutional architectures of the competitors, which combine convolutional and fully connected blocks. Additional details concerning other modalities are available in Appendix D.

Note that our method is conceptually different from the Stable Diffusion (SD) approach, where pure convolutional networks are adopted. In that work, a “latent space” is only motivated by the need for reduced computational complexity of the diffusion model. In SD, the “latent” input to the diffusion model is a downsized image with 4 channels (details may vary depending on the SD version). In the VAE literature we are considering, latent space has a different meaning, and usually corresponds to a flattened vector which belongs to a real space of a given dimension. This difference has important practical implications: with our approach, it is much simpler to **mix** latent spaces of different modalities, whereas with SD such mixing would require careful engineering effort. Finally, recall that SD is unidirectional: text-to-image, but not the other way around.

Reviewer comment (Experiments):
How have you determined the latent dimensions for each modality, and what criteria influenced your decision on their dimensionality?

Author response:
This is a good question which is a research line per se! The manifold hypothesis [Gorban2018] tells us that natural data resides on a lower-dimensional space, and we tend to use small latent sizes, in accordance with what has been done in the literature. We have experiments with larger latent sizes and in general performance improves, up to a point. The interesting tradeoff in our case is that the smaller the latent size, the more computationally efficient is latent diffusion.

[Gorban2018] Gorban AN, Tyukin IY. Blessing of dimensionality: mathematical foundations of the statistical physics of data. Philos Trans A Math Phys Eng Sci. 2018;376(2118):20170237. doi:10.1098/rsta.2017.0237

Reviewer comment (Experiments):
How scalable is the MLD approach when dealing with a large number of modalities or high-dimensional data within each modality?

Author response:
Our method MLD scales well in terms of the number of modalities, as well as in terms of the resolution of an individual modality in its input space. Concerning the number of modalities, this is visible for example in Fig. 18, Section E.3, where we vary the number of modalities used in the PolyMNIST dataset, and show that MLD performs extremely well, especially in terms of generation quality, while maintaining the highest coherence. Concerning resolution, this is a direct consequence of projecting the input in a lower-dimensional latent space.

The primary bottleneck in MLD is the latent dimension selected for each modality. Larger latent sizes are necessary for complex modalities, and this affects the computational cost of our multi-time masked diffusion model. In reference to the previous remark, we select appropriate latent dimensions for each modality because we use well-studied datasets from the literature, for which our model does not require extremely high costs. For an application of MLD “in the wild”, a good engineering practice of studying the data first, for example through the lenses of our deterministic autoencoders, should be applied.
Note that MLD provides an important advantage when compared to multimodal VAE-based approaches: there is no requirement for all latent spaces to have exactly the same dimension. This is evidently not true for e.g. Mixture based approaches, where the different modalities are required to have the same latent size (otherwise sum of latent means is not possible).

Reviewer comment (Experiments):
Given the independent training of uni-modal autoencoders, how do you ensure that the concatenated latent space is cohesive and meaningful? Are there any challenges in ensuring convergence during training?

Author response:
Thank you for the question, this is indeed a central element of our work. In alternative approaches, obtaining a cohesive latent space is a requirement. Different methods aim for the same result: to have geometrically **aligned** latent spaces for the different modalities. This is an implicit prerequisite to be able to mix the latent spaces into a unique one in a consistent way.

In our work, we instead follow a different approach, as observed by the reviewer. In the first stage, separate autoencoders are trained independently. Consequently, there is no guarantee that the different latent spaces induced by the different modalities will share any form of geometric alignment. MLD is based on the concept of **concatenation** of the various latent spaces: there is no requirement for geometric cohesion. The overall joint distribution in such concatenated latent space is modeled through a diffusion process which exploits a **joint** score network, the fundamental tool which allows, starting from a set of independently sampled coordinates of gaussian noise, to mix the information and construct the necessary (non-linear) correlations.

Concerning challenges during training, our approach is more stable than competitors. Indeed, as explained in the submitted version of the paper, we “ avoid any form of interference in the back-propagated gradients corresponding to the uni-modal reconstruction losses. Consequently, gradient conflict issues [Javaloy et al., 2022], where stronger modalities pollute weaker ones, are avoided.” (see Section 3, below Equation 2 in our paper).

Reviewer comment (Reproducibility):
You mentioned you used 4 A100 GPUs for a total of roughly 4 months of experiments. Could you give more details about the training?

Author response:
Our four-month long experimental campaign implements a comprehensive protocol that includes more than just training the MLD model. This time was allocated to training all competing models from scratch and conducting in-depth ablation studies. It's important to note that this duration represents the total time for the entire set of experiments, encompassing all aspects of our research, not just the training of MLD.

Reviewer comment (Presentation):
The authors appear to have deviated from the official Style files and Templates as provided by the ICLR 2024 Call for Papers (https://iclr.cc/Conferences/2024/CallForPapers). Notably, there are discrepancies in the citation format.

Author response:
We thank the reviewer for the suggestion, we updated the paper accordingly.

Reviewer comment (Evaluation):
the results dont look very good. e.g., MLD painted birds in Fig 22.

Author response:
> **the results dont look very good. e.g., MLD painted birds in Fig 22.**

We respectfully disagree with this comment. It is clear that MLD outperforms competitors from an image quality generation point of view (see also the FID scores Table 16 Appendix E.4). This is a result of how our multi-time diffusion model achieves latent modality mixing, since the auto-encoder architectures we use are the same as for all competitors.

Reviewer comment (Presentation):
the paper lacks a overall diagram showing the whole model design. its' a bit difficult to understand the model design

Author response:
> **The paper lacks a overall diagram showing the whole model design. its' a bit difficult to understand the model design**

We think the reviewer might have missed Fig. 4 in the Appendix of the submitted paper, where we clearly display a diagram of the model design.

Reviewer comment (Experiments):
are there strong methods the author can use?

Author response:
> **the other exciting methods seem quite weak. e.g., in Fig 20, MVAE cannot even generate digits very well, and in page 22, MVAE and MOPOE can't generate legible birds at all. are theses meaningful benchmark methods in 2023? are there strong methods the author can use?**

The literature we target focuses on the problem of multimodal generative modeling, with a vast array of models that combine several input modalities in a joint, mixed latent space, which is then used to generate multimodal data. At the risk of being simplistic, we could state that the key differentiating factor in all such prior work is the method used to mix latent representations of different modalities into a unique latent space: broadly speaking, either the mixing happens according to a “product of experts”, a “mixture of experts”, or a combination thereof.

The reviewer correctly acknowledges that the considered competitors suffer from limitations, especially from the perspective of generative quality. This is not a flaw of our comparative analysis, but a widely known literature result that pinpoints an open question: how can we achieve a better tradeoff in generative quality vs. coherence? This is *the current state of the art in 2022/2023*, as testified by works published in ICLR:

* “MMVAE+: Enhancing the Generative Quality of Multimodal VAEs Without Compromises”, ICLR 2023, https://openreview.net/pdf?id=BYHy9WwxFU **(see fig. 7)**
* “On the Limitations of Multimodal VAEs”, ICLR 2022, https://arxiv.org/pdf/2110.04121.pdf **(see Fig. 4)**

The poor results we obtain for competitors is far from surprising, as we spent particular effort into replicating the experimental setup of our competitors, including architectures and hyper-parameters. Consequently, we claim that the answer to “are these meaningful benchmark methods in 2023?” is a sound yes.

Importantly, the problems of our competitors are not related to architecture capacity, but to intrinsic limitations of the mixing method, which induces the generative quality/coherence tradeoff. This is exactly the conclusion reached in:
“On the Limitations of Multimodal VAEs”, ICLR 2022, https://arxiv.org/pdf/2110.04121.pdf .