Paper: Deep Generative Clustering with Multimodal Diffusion Variational Autoencoders (ICLR-2024)

Reviewer comment (Reproducibility):
What are the learned modules in CMVAE? Are all $q(z | X)$, $q(w_i | x_i)$, $p(w_i)$, $p(c)$, $p(z|c)$ learned during the training?

Author response:
The modules $q_{\Phi_z}(z | X), q_{\phi_{w_1}}(w_1| x_1), \dots, q_{\phi_{w_M}}(w_M|x_M), p_{\pi}(c), p(z|c), p_{\theta_1}(x_1|w_1, z), \dots, p_{\theta_M}(x_M|w_M, z)$ are learned during training. The prior distributions for modality-specific features $p(w_1), \dots, p(w_M)$ are set to zero-mean unit-variance distributions.  Following the recent work by Palumbo et al.[1], the auxialiary distributions for modality-specific features used at training time to facilitate the estimation of cross-modal reconstructions are zero-mean distributions with parameterized variance that is learned at training time. We denote these distributions by $r_1(w_1), \dots, r_M(w_M)$ in the updated version of the manuscript, uniforming our notation with the work from Palumbo et al. [1].

Reviewer comment (Presentation):
In figure 1, I cannot understand why there exist $z_1$, ..., $z_M$.

Author response:
Note that we model the joint encoder for $z$ as a mixture-of-experts $q_{\Phi_z}(z| X) = \frac{1}{M} \sum_{m=1}^M q_{\phi_{z_m}}(z| x_m)$, and hence computing our ELBO requires sampling $M$ latent codes for $z$, each one sampled by the corresponding unimodal encoder $q_{\phi_{z_m}}(z| x_m)$. That is why in Figure 1 $z_1, \dots, z_M$ are present.

Reviewer comment (Theory):
I cannot understand the cross-reconstruction formulation in Eq (8). I was thinking $p_{\theta_m}$ can be only called with $w_m$.

Author response:
As mentioned above, as in previous work from Palumbo et al. [1] we sample modality-specific features from auxiliary distributions for cross-reconstruction. Looking at the updated version of the manuscript, where we uniform the notation with Palumbo et al. [1], the cross-reconstruction formula in Eq. (8) is $p_{\theta_m}(x_m|z, \tilde{w}_n)$ , where $\tilde{w}_n$ is sampled from the auxiliary distribution $r_n(w_n)$ for the target modality $n$.

Reviewer comment (Experiments):
What are the clusters in CUBICC dataset? species or the subspecies?

Author response:
The clusters in the CUBICC dataset are the eight species: Blackbird, Gull, Jay, Oriole, Tanager, Tern, Warbler, Wren. We further clarify this point in the updated version of the manuscript.

Reviewer comment (Novelty):
I'm concerned about the technical novelty. The authors extend typical VAEs to multi-modal VAEs for two tasks. The first is the application of weakly-supervised clustering, and the second is to combine multi-modal VAEs with DDPM to improve its generative quality. However, both applications of VAEs are not novel and can be viewed as an extension from VAEs to multi-modal VAEs. Though some new tricks are proposed in the form of multi-modal VAEs like Eq.8, it is not properly discussed and evaluated with sufficient ablation studies.

Author response:
As we argue in our paper, we position this work at the gist of two existing research directions: multimodal VAEs and (unimodal) VAE-based approaches for clustering. As highlighted by the other reviewers, combining these research areas in a holistic, versatile approach to improve over state-of-the-art models is novel and proved relevant and effective with solid empirical performance advancements in both settings.

We propose a novel multimodal VAE model incorporating clustering in the latent space, which advances the state-of-the-art of multimodal VAEs (see Section 4.1). Note that this improvement in generative capabilities is only due to modeling advancements since, for these comparisons, we do not use DDPMs for generation. While multimodal VAEs and their cross-modal generation capabilities have recently gained significant attention, improving unconditional generation has been an overlooked topic [1][2]. In our work, we tackle this aspect and achieve a marked improvement in unconditional generation performance, therefore contributing to the field of multimodal VAEs.  

In addition, we propose a novel approach to perform clustering on datasets consisting of numerous modalities, and we outperform existing scalable approaches. (Note: for evidence for this claim, please refer also to our answer to the point below). We also incorporate a novel post-hoc procedure to automatically select the optimal latent clusters at test time, avoiding the need to specify the correct number of clusters a-priori, thus greatly enhancing the applicability of the proposed method in real-world settings.

Finally, while DDPMs have been incorporated into the VAE framework, we are, to the best of our knowledge, the first ones to investigate their integration in multimodal VAEs. In particular, and specifically to your question 


> I suggest the authors conduct ablation studies of such designs e.g. Eq.8.


we provide several insights on the integration of DDPMs in mixture-based multimodal VAEs. In particular, we performed ablation experiments to empirically validate Eq.8 in the Appendix (see Fig.15). We showed that conditioning the DDPM on the self-reconstructions alone is insufficient to generate high-quality images. Specifically, cross-reconstructions are usually noisier than self-reconstructions as modality-specific information cannot be inferred from the input sample. Therefore, by alternating between self and cross-reconstructions, the DDPM is more robust against noisy VAE generations and cross-generations, greatly improving generative performances (see Fig.6). The integration of DDPMs in the multimodal VAE framework proves to have a massive impact on generative capabilities for both unconditional and conditional generation and is an important step towards the applicability of multimodal VAEs in complex real-world settings.

Reviewer comment (Experiments):
The experiments are only conducted on one realistic multi-modal dataset that only contains two modalities. The authors mention the state-of-the-art weakly supervised methods are not scalable to numerous modalities while the proposed method is. The prior work is either on image-text [1] datasets or video-audio [2] [3] datasets. In Table 2, on PolyMNIST the proposed method is not improved over non-VAE SOTA either. Therefore, I think the authors overclaim the contribution of the scalability of numerous modalities. Especially for weakly-supervised clustering, in my opinion, a realistic multi-modal dataset is essential to claim the contribution of scalability to more modalities.

Author response:
Note that from our results on PolyMNIST, our approach *does* represent a relevant improvement over non-VAE SOTA. In fact, while the performance of CMC and PolyMNIST matches in Table 2, CMC was given the true number of clusters (ten) as a-priori information. In the realistic setting in which the true number of clusters in unknown a-priori (see Table below and Appendix E of our paper), our method significantly surpasses other non-VAE SOTA. This validates the importance of our novel procedure to automatically select optimal clusters in the data. In the updated manuscript, we highlight this point when commenting the results in Table 2 and elaborate further in Appendix E.

As for the claim that our model is a scalable approach in the number of modalities, this rather refers to the fact that our approach is a multimodal VAE model that can handle numerous modalities efficiently for training and inference, in comparison e.g. to initial approaches in the class of multimodal VAEs [3][4] (see first paragraph of our Related Work section). Note this characterization is well-established in the multimodal VAEs literature [2][5][6], and following this characterization our model falls into the family of scalable multimodal VAEs. The fact that our model can handle a large number of modalities present for training and inference already separates our approach from other weakly-supervised clustering models [7][8], that cannot be trivially extended to handle more than two modalities for training, or to perform inference if only a subset of modalities is present at test time.

When testing our model on the PolyMNIST dataset we show it can handle numerous modalities and exploit information from these modalities to obtain high clustering performance. While a synthetic dataset, PolyMNIST proves to be already challenging for existing approaches, and to back up this claim we also show a comparison with the recently proposed XDC [7] in this setting (see Appendix E in the manuscript and Table below), which fails to achieve satisfactory performace. Originally, XDC is proposed for two modalities, where the pseudo-labels for one modality's classification are derived from clustering assignments from the embedding space of the other modality, and does not trivially extend to more than two modalities. Hence, to adapt XDC to scale to more than two modalities, we select uniformly at random which other modality to extract the pseudo-labels from at each iteration. Its limited performance on PolyMNIST shows that it struggles to capture the relevant information from more than two modalities and confirms the relevance of the results obtained with CMVAE. Finally, in Table 3 in the Appendix we show CMVAE's test-time clustering performance on PolyMNIST improves when more modalities are present for inference, which indicates our model can effectively exploit additional modalities for better clustering. 

|              | NMI         | ARI         | ACC         | K not needed for training |
|--------------|-------------|-------------|-------------|---------------------------|
| XDC          | 0.02 (0.01) | 0.01 (0.01) | 0.15 (0.01) | no                        |
| CMC          | 0.97 (0.02) | 0.97 (0.02) | 0.99 (0.01) | no                        |
| CMC          | 0.78 (0.00) | 0.45 (0.01) | 0.35 (0.01) | yes                       |
| CMVAE (ours) | 0.97 (0.02) | 0.97 (0.02) | 0.99 (0.01) | yes                       |

Reviewer comment (Experiments):
Since the contribution of this work is the combination of several existing approaches based on VAEs, it is important for the authors to focus on further justifying that the extension of VAEs to multi-modal VAEs in these tasks is not trivial and requires careful design. I suggest the authors conduct ablation studies of such designs, e.g., Eq.8.

Author response:
Finally, while DDPMs have been incorporated into the VAE framework, we are, to the best of our knowledge, the first ones to investigate their integration in multimodal VAEs. In particular, and specifically to your question 


> I suggest the authors conduct ablation studies of such designs e.g. Eq.8.


we provide several insights on the integration of DDPMs in mixture-based multimodal VAEs. In particular, we performed ablation experiments to empirically validate Eq.8 in the Appendix (see Fig.15). We showed that conditioning the DDPM on the self-reconstructions alone is insufficient to generate high-quality images. Specifically, cross-reconstructions are usually noisier than self-reconstructions as modality-specific information cannot be inferred from the input sample. Therefore, by alternating between self and cross-reconstructions, the DDPM is more robust against noisy VAE generations and cross-generations, greatly improving generative performances (see Fig.6). The integration of DDPMs in the multimodal VAE framework proves to have a massive impact on generative capabilities for both unconditional and conditional generation and is an important step towards the applicability of multimodal VAEs in complex real-world settings.

Reviewer comment (Experiments):
To claim the scalability of the method, the authors may consider more experiments on realistic datasets with more than two modalities.

Author response:
While we make these considerations, we also appreciate the comment and suggestion from the Reviewer and agree on the value of a realistic multimodal dataset with a large number of modalities to benchmark clustering approaches. Therefore in the updated version of the manuscript (precisely under Limitations and Future work) we clarify that, applying our approach to a real-world multimodal dataset with numerous modalities is an important direction of future work. Unfortunately, a lack of realistic datasets with numerous modalities is witnessed also in the multimodal VAE literature [2][6][9]. With our findings on the PolyMNIST dataset and our results on the proposed CUBICC dataset, as a realistic multimodal clustering benchmark, we hope to inspire future work to propose novel real-world benchmarks for clustering tasks with numerous modalities.

We hope with this rebuttal we addressed the concerns from the Reviewer, possibly towards an adjustment of the score.

**References:**

[1] Hwang et al. Multi-view representation learning via total correlation objective. In NeurIPS, 2021. [2] Palumbo et al. MMVAE+: Enhancing the generative quality of multimodal VAEs without compromises. In ICLR, 2023. [3] Suzuki et al. Joint multimodal learning with deep generative models. arXiv preprint arXiv:1611.01891, 2017. [4] Vedantam et al. Generative models of visually grounded imagination. In ICLR, 2018. [5] Wu et al. Multimodal generative models for scalable weakly-supervised learning. In NeurIPS, 2018. [6]  Daunhawer et al. On the limitations of multimodal VAEs. In ICLR, 2022. [7] Alwassel et al. Self-supervised learning by cross-modal audio-video clustering. In NeurIPS, 2020. [8] Xu et al. Multi-vae: Learning disentangled view-common and view-peculiar visual representations for multi-view clustering. In ICCV, 2021. [9] Sutter et al. Generalized Multimodal ELBO. In ICLR, 2021.

Reviewer comment (Presentation):
I suggest the authors consider moving some content in the Appendix (e.g., Eq.8 ablation study) to the main manuscript.

Author response:
We thank the Reviewer for the feedback and the suggestion, which we will take into account for the camera-ready version.