Paper: Learning multi-modal generative models with permutation-invariant encoders and tighter variational bounds (ICLR-2024)

Reviewer comment (Theory):
The work does not directly demonstrate that the proposed bound is tighter than previously reported bounds in the literature. It's not clear how the new bound overcomes the multimodal VAE limitation detailed in [2], which are related to sub-sampling of modalities, which has the problematic effect of inducing an undesirable upper bound.

Author response:
> It's not clear how the new bound overcomes the multimodal VAE limitation detailed in [2].
 
We thank the reviewer for pointing out that it was unclear how our bound overcomes the multimodal VAE limitation of inducing an undesirable lower bound as in [2]. We have now adjusted the discussion and proof in the updated manuscript to better address this. To be more specific, for the mixture based variational objective, one obtains the bound
$$ \int p_d(x) \log p_\theta(x)  d x \geq \int p_d(x) L^{\text{Mix}}(x, \theta, \phi, 1) d x  +\mathcal{H} (p_d(X_{\setminus \mathcal{S}}|X_\mathcal{S})) .$$
Notice that there exists a gap between the variational bound and the log-likelihood given by the conditional entropies that cannot be reduced even for flexible encoding distributions. Conversely, our variational objective satisfies
$$ \int p_d(x) \log p_\theta(x)  d x  = \int p_d(x) L^{\text{Ours}}(x, \theta, \phi, 1) d x  +K_1+K_2+C,$$ 
where
$$ K_1= \int p_d(x_\mathcal{S})  \left[KL(q_\phi(z|x_\mathcal{S})|p_{\theta}(z|x_\mathcal{S})) \right]  d x, $$
$$K_2= \int p_d(x)  \left[KL(q_\phi(z|x)|p_{\theta}(z|x)) \right]  d x $$
and 
$$C = - \int p_d(x)  q_\phi(z|x )\left[\log \frac{q_\phi(z|x_\mathcal{S})}{p_{\theta}(z|x_\mathcal{S})} \right]  dz dx .$$
C is not necessarily positive. See Remarks  12 and 13 for details.
We thus overcome the undesirable upper bound by contending ourselves to maximize an approximate lower bound. However, in the limiting case of infinite-capacity encoding, the variational gap can be zero, whilst it is a  lower bound if the encoding distribution satisfies some consistency condition that will likely not hold in practice.

Reviewer comment (Theory):
The well known limitations from Multimodal VAE literature consist of the undesired trade-off between generation coherence and quality. Additionally, the modality collapse problem due to gradient conflicts [3] can also affect the training process. These limitations are not discussed, although the paper claims that the new proposed bound solves the limitations of Multimodal VAE.

Author response:
> Well-known limitations from Multimodal VAE (i.e., undesired trade-off between generation coherence and quality; gradient conflicts) are not discussed, although the paper claims that the new proposed bound solves the limitations of Multimodal VAE.
 
We realized that we formulated our contributions in an unintended strong form as (i) a new variational bound that addresses known limitations of previous variational bounds.
We have updated it now to the below, which we believe is supported by our analyses:
(i) a new variational bound as an approximate lower bound on the multi-modal log-likelihood (LLH). We avoid a limitation of mixture-based bounds, which may not provide tight lower bounds on the joint LLH if there is considerable modality-specific variation (Daunhawer et al., 2022), even for flexible encoding distributions. The novel variational bound contains a lower bound of the marginal LLH $\log p_\theta(x_\mathcal{S})$ and a term approximating the conditional $\log p_\theta(x_{\setminus \mathcal{S}}|x_\mathcal{S})$ for any choice of $\mathcal{S} \in \mathcal{P}(\mathcal{M})$, provided that we can learn a flexible multimodal encoding distribution.
 
In particular, we do not want to claim that our approach solves all well-known limitations of multimodal VAEs, see also our comments further below. For example, we illustrate that other variational bounds can be beneficial for cross-modal reconstructions. Due to space constraints, we have only mentioned the issue of gradient conflicts in Appendix A that also applies to our work. While we did not test this empirically, we still expect that multi-task learning techniques can be similarly effective for our approach, as they are for other variational bounds or aggregation schemes.

Reviewer comment (Evaluation):
The results on the MNIST-SVHN-Text dataset, using the proposed approach, do not seem to improve the overall conditional generation coherence. LLH improvements do not correlate with the overall conditional coherence / latent classification accuracy. In table 9, the cross generation of SVHN given MNIST is slightly better using the proposed approach but the generation of MNIST given SVHN is a lot worse. The proposed extensions do not perform very well across all modalities in terms of coherence: some recent works obtain far superior results with respect to the multimodal VAE literature [5].

Author response:
> Multimodal VAEs fail in joint generation. Recent diffusion-based work [5] obtains superior results compared to VAEs
 
We did not intend to claim that our approach leads to better conditional/unconditional generation or coherence than recent/concurrent work using diffusion models [5]. Indeed, for a single modality, our bound becomes a standard VAE, and thus shares its limitations, particularly concerning prior-hole issues that may lead to relatively poor unconditional generation compared to diffusion models, for example. Moreover, our ELBO analysis demonstrates that a mismatch between the aggregated posterior/prior conditioned on a modality subset and the encoding distribution conditioned on said modality subset may likewise lead to sub-optimal conditional coherence. While this mismatch may be smaller for more flexible aggregation functions, we believe this issue applies generally to other multimodal VAEs, unless one incorporates different sampling procedures, such as diffusions, EBMs, etc., in some form. We will clarify this in an updated version.

Reviewer comment (Experiments):
The experimental campaign misses many competitors and benchmarks that are widely used in the literature: for example, competitors such as Nexus [4], MMVAE+ [6], MOPOE [7], MVTCAE [8] are not included as baselines.

Author response:
>  Several missing baselines from the multimodal generative models' literature should be included in the experimental comparison: VAE-based models such as Nexus, MMVAE+, MOPOE, MVTCAE.
 
 
We do agree with the reviewer that our submission lacked a direct comparison and discussion how baseline methods perform on the SVHN-MNSIT-Text dataset. To address this, we have now included the results from previous works using the same dataset. In particular, we report log-likelihoods, conditionally generated coherences and latent classification accuracies for external baselines (MVAE, MMVAE, MoPoE, MMJSD, MVTCAE) in Tables 8-10 and 15. Unlike our initially reported results, different hyperparameter choices in these baselines (different latent dimensions, encoder/decoder architectures, prior/approximate posterior distribution families, etc.) may make a direct comparison more difficult.

Reviewer comment (Evaluation):
The evaluation protocol adopted in most of the related literature sheds light on two desired properties of multimodal generative models which are generative coherence and quality. Measuring the generation quality is not considered in this work. Joint generation (unconditional generation) by sampling from the prior is one setting in which, in general, Multimodal VAEs fail. Results on joint generation are not included.

Author response:
> Multimodal VAEs fail in joint generation. Recent diffusion-based work [5] obtains superior results compared to VAEs
 
We did not intend to claim that our approach leads to better conditional/unconditional generation or coherence than recent/concurrent work using diffusion models [5]. Indeed, for a single modality, our bound becomes a standard VAE, and thus shares its limitations, particularly concerning prior-hole issues that may lead to relatively poor unconditional generation compared to diffusion models, for example. Moreover, our ELBO analysis demonstrates that a mismatch between the aggregated posterior/prior conditioned on a modality subset and the encoding distribution conditioned on said modality subset may likewise lead to sub-optimal conditional coherence. While this mismatch may be smaller for more flexible aggregation functions, we believe this issue applies generally to other multimodal VAEs, unless one incorporates different sampling procedures, such as diffusions, EBMs, etc., in some form. We will clarify this in an updated version.

Reviewer comment (Experiments):
The author considers only one real world dataset containing highly correlated modalities, with the label digit (Text modality). Other benchmarks widely studied in the literature are missing from the experimental section: for example, the Polymnist [7], MHD [4], CUB [2, 9] datasets. This latter showed to be a challenging case for VAE models [2].

Author response:
> The experimental section should be enriched with more challenging datasets, such as Polymnist, MHD, and CUB datasets. The evaluation of the generative quality of the compared models using metrics such as the FID score should be considered.
 
We acknowledge that it is not clear how our approach would perform on more challenging datasets and with respect to different generative measures. Since FID scores can be less suited for the MNIST and Text modality, we only considered log-likelihoods scores. We will definitely consider the application to more challenging datasets, as mentioned by you, although we feel that our submission in its current form already contains enough contributions for a conference proceeding.

Reviewer comment (Theory):
Why the proposed bound is tighter than the bound studied in the larger family of multimodal VAE models? Is this an empirical claim via LLH estimation?

Author response:
> It's not clear how the new bound overcomes the multimodal VAE limitation detailed in [2].
 
We thank the reviewer for pointing out that it was unclear how our bound overcomes the multimodal VAE limitation of inducing an undesirable lower bound as in [2]. We have now adjusted the discussion and proof in the updated manuscript to better address this. To be more specific, for the mixture based variational objective, one obtains the bound
$$ \int p_d(x) \log p_\theta(x)  d x \geq \int p_d(x) L^{\text{Mix}}(x, \theta, \phi, 1) d x  +\mathcal{H} (p_d(X_{\setminus \mathcal{S}}|X_\mathcal{S})) .$$
Notice that there exists a gap between the variational bound and the log-likelihood given by the conditional entropies that cannot be reduced even for flexible encoding distributions. Conversely, our variational objective satisfies
$$ \int p_d(x) \log p_\theta(x)  d x  = \int p_d(x) L^{\text{Ours}}(x, \theta, \phi, 1) d x  +K_1+K_2+C,$$ 
where
$$ K_1= \int p_d(x_\mathcal{S})  \left[KL(q_\phi(z|x_\mathcal{S})|p_{\theta}(z|x_\mathcal{S})) \right]  d x, $$
$$K_2= \int p_d(x)  \left[KL(q_\phi(z|x)|p_{\theta}(z|x)) \right]  d x $$
and 
$$C = - \int p_d(x)  q_\phi(z|x )\left[\log \frac{q_\phi(z|x_\mathcal{S})}{p_{\theta}(z|x_\mathcal{S})} \right]  dz dx .$$
C is not necessarily positive. See Remarks  12 and 13 for details.
We thus overcome the undesirable upper bound by contending ourselves to maximize an approximate lower bound. However, in the limiting case of infinite-capacity encoding, the variational gap can be zero, whilst it is a  lower bound if the encoding distribution satisfies some consistency condition that will likely not hold in practice.

Reviewer comment (Theory):
How does the new bound overcome the limitations of Multimodal VAEs, such as the undesirable upper bound due to sub-sampling of modalities? In practice, the new proposed bound is also based on a similar sub-sampling approach as in [1].

Author response:
> It's not clear how the new bound overcomes the multimodal VAE limitation detailed in [2].
 
We thank the reviewer for pointing out that it was unclear how our bound overcomes the multimodal VAE limitation of inducing an undesirable lower bound as in [2]. We have now adjusted the discussion and proof in the updated manuscript to better address this. To be more specific, for the mixture based variational objective, one obtains the bound
$$ \int p_d(x) \log p_\theta(x)  d x \geq \int p_d(x) L^{\text{Mix}}(x, \theta, \phi, 1) d x  +\mathcal{H} (p_d(X_{\setminus \mathcal{S}}|X_\mathcal{S})) .$$
Notice that there exists a gap between the variational bound and the log-likelihood given by the conditional entropies that cannot be reduced even for flexible encoding distributions. Conversely, our variational objective satisfies
$$ \int p_d(x) \log p_\theta(x)  d x  = \int p_d(x) L^{\text{Ours}}(x, \theta, \phi, 1) d x  +K_1+K_2+C,$$ 
where
$$ K_1= \int p_d(x_\mathcal{S})  \left[KL(q_\phi(z|x_\mathcal{S})|p_{\theta}(z|x_\mathcal{S})) \right]  d x, $$
$$K_2= \int p_d(x)  \left[KL(q_\phi(z|x)|p_{\theta}(z|x)) \right]  d x $$
and 
$$C = - \int p_d(x)  q_\phi(z|x )\left[\log \frac{q_\phi(z|x_\mathcal{S})}{p_{\theta}(z|x_\mathcal{S})} \right]  dz dx .$$
C is not necessarily positive. See Remarks  12 and 13 for details.
We thus overcome the undesirable upper bound by contending ourselves to maximize an approximate lower bound. However, in the limiting case of infinite-capacity encoding, the variational gap can be zero, whilst it is a  lower bound if the encoding distribution satisfies some consistency condition that will likely not hold in practice.

Reviewer comment (Evaluation):
How does the new bound and the presented aggregation methods handle other limitations of Multimodal VAEs, such as the coherence-quality trade-off and modality collapse where stronger modalities dominate over weaker ones [3]?

Author response:
> Well-known limitations from Multimodal VAE (i.e., undesired trade-off between generation coherence and quality; gradient conflicts) are not discussed, although the paper claims that the new proposed bound solves the limitations of Multimodal VAE.
 
We realized that we formulated our contributions in an unintended strong form as (i) a new variational bound that addresses known limitations of previous variational bounds.
We have updated it now to the below, which we believe is supported by our analyses:
(i) a new variational bound as an approximate lower bound on the multi-modal log-likelihood (LLH). We avoid a limitation of mixture-based bounds, which may not provide tight lower bounds on the joint LLH if there is considerable modality-specific variation (Daunhawer et al., 2022), even for flexible encoding distributions. The novel variational bound contains a lower bound of the marginal LLH $\log p_\theta(x_\mathcal{S})$ and a term approximating the conditional $\log p_\theta(x_{\setminus \mathcal{S}}|x_\mathcal{S})$ for any choice of $\mathcal{S} \in \mathcal{P}(\mathcal{M})$, provided that we can learn a flexible multimodal encoding distribution.
 
In particular, we do not want to claim that our approach solves all well-known limitations of multimodal VAEs, see also our comments further below. For example, we illustrate that other variational bounds can be beneficial for cross-modal reconstructions. Due to space constraints, we have only mentioned the issue of gradient conflicts in Appendix A that also applies to our work. While we did not test this empirically, we still expect that multi-task learning techniques can be similarly effective for our approach, as they are for other variational bounds or aggregation schemes.

Reviewer comment (Evaluation):
The evaluation of the generative quality of the compared models using metrics such as the FID score should be considered. Joint generation results (sampling from the prior) should be reported to evaluate if the new approach helps in this challenging scenario.

Author response:
> The experimental section should be enriched with more challenging datasets, such as Polymnist, MHD, and CUB datasets. The evaluation of the generative quality of the compared models using metrics such as the FID score should be considered.
 
We acknowledge that it is not clear how our approach would perform on more challenging datasets and with respect to different generative measures. Since FID scores can be less suited for the MNIST and Text modality, we only considered log-likelihoods scores. We will definitely consider the application to more challenging datasets, as mentioned by you, although we feel that our submission in its current form already contains enough contributions for a conference proceeding.

Reviewer comment (Experiments):
Several missing baselines from the multimodal generative models literature should be included in the experimental comparison: VAE based models such as Nexus, MMVAE+, MOPOE, MVTCAE are typical baselines (please, see references above).

Author response:
>  Several missing baselines from the multimodal generative models' literature should be included in the experimental comparison: VAE-based models such as Nexus, MMVAE+, MOPOE, MVTCAE.
 
 
We do agree with the reviewer that our submission lacked a direct comparison and discussion how baseline methods perform on the SVHN-MNSIT-Text dataset. To address this, we have now included the results from previous works using the same dataset. In particular, we report log-likelihoods, conditionally generated coherences and latent classification accuracies for external baselines (MVAE, MMVAE, MoPoE, MMJSD, MVTCAE) in Tables 8-10 and 15. Unlike our initially reported results, different hyperparameter choices in these baselines (different latent dimensions, encoder/decoder architectures, prior/approximate posterior distribution families, etc.) may make a direct comparison more difficult.

Reviewer comment (Experiments):
The experimental section should be enriched with more challenging datasets, such as Polymnist, MHD, and CUB datasets (please, see references above).

Author response:
> The experimental section should be enriched with more challenging datasets, such as Polymnist, MHD, and CUB datasets. The evaluation of the generative quality of the compared models using metrics such as the FID score should be considered.
 
We acknowledge that it is not clear how our approach would perform on more challenging datasets and with respect to different generative measures. Since FID scores can be less suited for the MNIST and Text modality, we only considered log-likelihoods scores. We will definitely consider the application to more challenging datasets, as mentioned by you, although we feel that our submission in its current form already contains enough contributions for a conference proceeding.

Reviewer comment (Theory):
Proposition 1 relies strongly on the assumption on the structure of \( q_\phi \) given at the top of page 4 (without being clearly expressed as an assumption), which amounts to stating that the partial encoder should be identical to the empirical marginalization of the full encoder:
\[
q_\phi(z|x_\mathcal{S}) = \int_{x_{\setminus\mathcal{S}}} p_d(x_{\setminus\mathcal{S}} | x_{\mathcal{S}}) q_\phi(z | x_{\mathcal{S}}, x_{\setminus\mathcal{S}}) \text{d}x_{\setminus\mathcal{S}}
\]
While the text below proposition 1 suggests that the training loss encourages the model to bring these two distributions close to each other, there is no discussion about this assumption.

Author response:
We agree that Proposition 1 relies on a strong assumption about the encoding distribution that has not been properly discussed. We have, therefore, removed Proposition 1 in the updated version. Note that the ELBO surgery in Proposition 11 (or Proposition 9 in the updated version) does not make this assumption, which we are now using to show how the proposed objective relates to the multi-modal log-likelihood and under what conditions it constitutes a lower bound. More precisely, see Corollary 1, the variational gap includes, beyond the two KL-divergences between the variational approximation and the true posterior conditioned on $c_{\mathcal{S}}$ and on the full $x$, an additional term
$\int p_d(x)  q_\phi(z|x )\left[\log \frac{q_\phi(z|x_\mathcal{S})}{p_{\theta}(z|x_\mathcal{S})} \right]  dz dx $
that is not necessarily negative or negative. As you already commented, in the case where the encoding distribution conditioned on $x_{\mathcal{S}}$ equals the empirical marginalization of the full encoder, or what we called the aggregated encoder conditioned on $x_\mathcal{S}$, the variational gap reduces to the KL conditioned on the full multi-modal data, i.e., we recover the previous Proposition while more clearly stating the required conditions. See Remarks 12 and 13 in Appendix A for details.
 
We have also added clarifications regarding this to the updated submission. In the experiments with simulated data, we have removed the reported gap between the variational bounds and the log-likelihoods as it may not say much without assumptions on the encoding distributions.

Reviewer comment (Theory):
At the end of section 2, it is shown that the mathematical optimal partial encoder \( q^\star(z|x_\mathcal{S}) \) of the mixture-based objective is not equal to the true Bayesian posterior of the model \( p(z|x_\mathcal{S}) \) (as opposed to the proposed objective), but instead an additional factor is added to it. Unfortunately, this additional factor and how it may impact the model is not discussed at all.

Author response:
Thank you for pointing out this interesting question. To shed light on how this might impact the model performance, we refer to the toy example in our response to reviewer EVr5 of two modalities $X_1=Z_0+Z_1+U_1$ and $X_1=Z_0+10Z_2+U_2$  driven by a latent variable $Z=(Z_0,Z_1,Z_2) \in \mathbb{R}^3$, where $Z_i$ and $U_j$ are iid $\mathcal{N}(0,1)$. In this setup, we can compute the KL to the true posterior, and we use relatively flexible aggregation schemes (MLP with 64 hidden dimensions of depth 2; possible 64-dim attention; for 3-dim latent variable), hoping to approximate the optimal (partial and full) encoders closely. Indeed, for both bounds, we find that the average KLs between the full encoders and full posterior is small, while the KL between the uni-modal encoders and posteriors becomes small only for our bound.

|               | uni-modal KL     |                  | multi-modal KL   |                  |
|---------------|------------------|------------------|------------------|------------------|
|               | ours             | mixture          | ours             | mixture          |
| SumPooling    |           0.008  |           0.789  |           0.013  |           0.015  |
| SelfAttention |           0.007  |           0.791  |           0.012  |           0.015  |
 
However, the model performance is different for these two bounds. For example, our bound achieves better LLH (analytically computed using only the learned generative parameters). The information-theoretical measures of the learned representations are very different, see the Table in the response to reviewer EVr5. For the MNIST-SVHN-Text example, where the learned encoders likely differ more from their optimal counterparts, we see similar effects in Figure 2.

Reviewer comment (Evaluation):
In the experimental results of the MINST-SVHN-Text problem, a lot of quantitative performance measures are provided, but they are barely discussed or interpreted.

Author response:
We aim to include a more detailed discussion of the quantitative performance measures in an updated version.

Reviewer comment (Writing):
Lemma 2 and Corollary 3 use the names \( Z_\mathcal{M} \) and \( Z_\mathcal{S} \) without them being first defined. While we can guess what they are probably supposed to mean, this hurts readability.

Author response:
Thank you for pointing this out. We will include their definitions in the main paper.

Reviewer comment (Theory):
Regarding the assumption about the structure of \( q_\phi \) being compatible with empirical marginalization — Is the proposed loss still a proper lower-bound if this assumption is not respected?

Author response:
The proposed loss is only an approximately lower bound to the multi-modal LLH without additional assumptions on the encoding distributions. Experimentally, this did not lead to divergence issues or so. The ‘approximately lower’ issue is due to the $L_{\setminus \mathcal{S}}$ loss term, which is basically also used in latent Neural Processes, and where we believe it does also not cause optimization issues. Furthermore, our multi-modal ELBO decomposition shows that by maximixing $L_{\setminus \mathcal{S}}$, we minimize the above-mentioned Bayes-consistency matching term between the encoding distribution and its aggregated version.
 
In the general case with arbitrary $\beta >0$, see Corollary 3, minimizing our bound (properly) minimizes multi-modal rate and distortion measures that are different from those minimized in mixture-based bounds and that provide sandwich bounds for conditional and unconditional mutual information.

Reviewer comment (Evaluation):
Regarding the experimental results on MINST-SVHN-Text — Why are the model performances so different between the various modalities?

Author response:
We believe the performance differences for the modalities in the MNIST-SVHN-Text data are because SVHN is more noisy than MNIST or Text. Previous work for the same dataset (MVAE, MMVAE, MoPoE, MMJSD, MVTCAE) similarly exhibited differences in the performance measures (see, for example, Table 9 for coherence measures, updated with results as reported in previous work).

Reviewer comment (Evaluation):
Regarding the experimental results on MINST-SVHN-Text — Why does adding private latent variables seem to overall reduce the performance of the models in terms of conditional coherence (see tables 9-12)?

Author response:
Introducing private latent variables may lead to different inductive biases for the learned latent representations and their conditional reconstruction. In the MNIST-SVHN-Text experiment, we fixed the overall latent dimension to 40. While there were no restrictions for the models with shared variables, for the model with private latent variables, we assumed that both the shared latent variables and the three private latent variables are 10-dimensional each. This creates a bottleneck in how much shared information can be encoded.

Reviewer comment (Evaluation):
Regarding the experimental results on MINST-SVHN-Text — Figure 2b suggests that the Mixture loss yields better cross-reconstruction overall, while tables 9-12 suggest the opposite; is there a contradiction?

Author response:
In Figure 2b, we report the cross-reconstruction based on the log-likelihood of each modality. Since mixture-based bounds directly optimize for this objective (which we call cross-distortion), they do, indeed, perform better here. Tables 9-12 measure a cross-coherence measure based on the classification performance of a pre-trained classifier (trained on the true modalities) applied to the generated modalities. The cross-coherence can vary a lot between modalities and may depend on how well the pre-trained classifier generalizes to reconstructed modalities. For example, cross-generations of the MNIST modality appear more averaged with less variation leading to a better coherence for the mixture-based bound.

Reviewer comment (Novelty):
The novelty is rather limited (the aggregation schemes are marginally novel, and the invariance is trivially achieved using known architectures).

Author response:
"The novelty is rather limited (the aggregation schemes are marginally novel, and the invariance is trivially achieved using known architectures)."
 
We agree with the reviewer that the premutation-invariant architectures are not new and that previous schemes such as MoEs, PoE, or MoPoEs do trivially inherit such an invariance. However, we do not think that permutation-invariant architectures per se represent the contribution of our submission. In contrast, we believe that our contributions entail: 

i)            A new multi-modal variational bound, accompanied by detailed analysis (e.g. information-theoretical analysis, multi-modal ELBO surgery).

ii)            The application of flexible fusion/aggregation schemes for multi-modal VAEs. While the network architectures are not new, we believe that our application for multi-modal models with shared or private latent variables is novel.

iii)            Model extensions such as using mixture priors (which are not that easily accommodated with PoE schemes) and identifiable models (where we believe training objectives that maximize the log-likelihood are very beneficial).

Reviewer comment (Experiments):
The experimentation over a single dataset is not satisfactory.

Author response:
"The experimentation over a single dataset is not satisfactory."
 
We want to clarify that we included numerical experiments with different bounds and aggregation schemes for various datasets; and not just on a single dataset. In particular, our experimentations include:

i)             Different linear models

ii)            Different non-linear models on synthetic data: 2D toy-example, higher dimensional examples with five modalities, in a complete setup, and with missing modalities. This design allows for access to ground truth values (as required for identifiability analysis)

iii)           MNIST-SVHN-Text data

Reviewer comment (Experiments):
Comparison to competing models is missing.

Author response:
"Did you compare the proposed model to external baselines?"
 
Thank you for pointing this out. We agree with the reviewer that our submission lacked a direct comparison and discussion of how baseline methods perform, for example on the SVHN-MNSIT-Text dataset. To address this, we have now included the results from previous works using the same dataset. In particular, we have updated  Tables 8-10 and 15 with log-likelihoods, conditionally generated coherences, and latent classification accuracies from external baselines (MVAE, MMVAE, MoPoE, MMJSD, MVTCAE). However, unlike our initially reported results, different hyperparameter choices in these baselines (different latent dimensions, encoder/decoder architectures, prior/approximate posterior distribution families, etc.) may make a direct comparison more difficult.

Reviewer comment (Experiments):
Optimization using MCMC typically limits the scaling of the proposed method from being used for more complex data (e.g., high dimensional) and architectures (i.e., more expressive models).

Author response:
"Optimization using MCMC typically limits the scaling of the proposed method "
 
We are unsure why MCMC scaling properties can constitute a weakness of our work. We do not use any MCMC in this submission.

Reviewer comment (Experiments):
Did you compare the proposed model to external baselines?

Author response:
"Did you compare the proposed model to external baselines?"
 
Thank you for pointing this out. We agree with the reviewer that our submission lacked a direct comparison and discussion of how baseline methods perform, for example on the SVHN-MNSIT-Text dataset. To address this, we have now included the results from previous works using the same dataset. In particular, we have updated  Tables 8-10 and 15 with log-likelihoods, conditionally generated coherences, and latent classification accuracies from external baselines (MVAE, MMVAE, MoPoE, MMJSD, MVTCAE). However, unlike our initially reported results, different hyperparameter choices in these baselines (different latent dimensions, encoder/decoder architectures, prior/approximate posterior distribution families, etc.) may make a direct comparison more difficult.

Reviewer comment (Experiments):
In section 5.1 Linear Multi-Modal VAEs: The bounds presented in Table 1 do not show a big difference between each other. However, this experiment is on synthetic data. Could the dataset be modified, e.g., a different number of mixture components, larger (or smaller) variance of the mixture components, such that the differences become more significant?

Author response:
> Can the linear VAE setup be adjusted to show more significant differences between the different approaches?
 
Thank you for this suggestion. Indeed, one can consider different linear settings that lead to more pronounced differences for linear multi-modal VAEs between different approaches. In particular, we considered a bi-modal setup with a less noisy and more noisy modality. Concretely, for a latent variable $Z=(Z_0,Z_1,Z_2) \in \mathbb{R}^3$, we set $X_1=Z_0+Z_1+U_1$ and $X_1=Z_0+10Z_2+U_2$ where $Z_i$ and $U_j$ are iid $\mathcal{N}(0,1)$. The second modality is thus more noisy than the first. Different aggregation schemes and variational objectives lead to the log-likelihoods below (relative to the true LLH over five repetitions).
 

|                | relative LLH gap    |                    | full reconstruction error |                         | full rates   |            | cross reconstruction error |                         | cross rates   |           |
|----------------|---------------------|--------------------|---------------------------|-------------------------|--------------|------------|----------------------------|-------------------------|---------------|-----------|
|                |  ours               |  mixture           |  ours                     |  mixture                |  ours        |  mixture   |  ours                      |  mixture                |  ours         |  mixture  |
| PoE            |  1.29               |  7.11              |  -2.3E+35                |  -2.2E+35              |  2.1E+35    | 2.0E+35   |  -2.4E+34                 |  -1.9E+34              |  1.4E+35     |  1.7E+35 |
| MoE            |  0.11               |  0.60              |  -32.07                   |  -30.09                 |  1.02        |  2.84      |  -33.27                    |  -28.52                 |  2.37         |  19.33    |
| SumPooling     |  3.6E-05           |  0.06              |  -2.84                    |  -3.23                  |  2.88        |  2.82      |  -52.58                    |  -27.26                 |  1.42         |  27.35    |
| SelfAttention  |  3.4E-05           |  0.06              |  -2.85                    |  -3.23                  |  2.87        |  2.82      |  -52.59                    |  -27.25                 |  1.42         |  27.41    |

In contrast to the previously reported results with more homogeneous modalities, the differences appear more significant. Our variational bound leads to higher LLH for all aggregation schemes. Learnable aggregation schemes yield higher LLH than PoEs or MoEs. 
 
We have also considered a rate-distortion analysis that leads to more significant differences compared with the setting from the initial submission and with overall results appearing in agreement with the MNIST-SVHN-Text experiment. Recall that full reconstruction log-likelihood is the negative full distortion  $-D_\mathcal{M}=\int p_d(x)q_{\phi}(z_{\mathcal{M}}|x) \log p_{\theta}(x|z_{\mathcal{M}})  dz_{\mathcal{M}}  d x$, while the full rate is $R_\mathcal{M} =\int p_d(x)  KL(q_{\phi}(z_{\mathcal{M}}|x)|p_{\theta}(z)) d x$. Similarly, the cross reconstruction log-likelihood is $-D^c_{\setminus \mathcal{S}}=\int p_d(x_\mathcal{S}) q_{\phi}(z|x_{\mathcal{S}}) \log p_{\theta}(x_{\setminus \mathcal{S}}|z_{\mathcal{S}}) d z_{\mathcal{S}} d x_\mathcal{S}$ and the cross rate is  $R_{\setminus \mathcal{S}} =\int p_d(x) KL(q_{\phi}(z|x)|q_{\phi}(z|x_\mathcal{S}))  d x$. Note that mixture-based bounds optimize directly for the cross-reconstruction log-likelihood, see Remark 4, and do not contain a cross-rate term as a regulariser, in contrast to our bound (Lemma 2, Corollary 3).
 
For the same $\beta=1$, our variational bound leads to higher reconstruction for/given all modalities with increased full rates. Mixture-based bound results in higher cross reconstructions with increased cross rates.
More flexible aggregation schemes increase the full and cross reconstruction for any given bound, while not necessarily increasing the full or cross rates, i.e., they can result in an improved point within a rate-distortion curve for some configurations.

Reviewer comment (Experiments):
In section 5.3, MNIST and SVHN are rather small datasets, and nowadays usually we would expect a more challenging evaluation on larger datasets. Is there a larger dataset which could also be used for the rate-distortion analysis?

Author response:
> Is there a larger dataset which could also be used for the rate-distortion analysis?
 
This is a very good question. We will definitely consider the application to a larger dataset (such as PolyMNIST or CUB) and perform a multi-modal rate-distortion analysis there.