Original: M ITIGATING THE L IMITATIONS OF M ULTIMODAL VAE S WITH C OORDINATION -B ASED A PPROACH
Revised: M ITIGATING THE L IMITATIONS OF M ULTIMODAL VAE S WITH C OORDINATION -B ASED A PPROACH

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: One of the key challenges in multimodal variational autoencoders (VAEs) is inferring a joint representation from arbitrary subsets of modalities.
Revised: One of the key challenges in multimodal variational autoencoders (VAEs) is inferring a joint representation from arbitrary subsets of modalities.

Original: The state-of-the-art approach to achieving this is to sub-sample the modality subsets and learn to generate all modalities from them.
Revised: The state-of-the-art approach to achieving this is to sub-sample the modality subsets and learn to generate all modalities from them.

Original: However, this sub-sampling in the mixture-based approach has been shown to degrade other important features of multimodal VAEs, such as quality of generation, and furthermore, this degradation is theoretically unavoidable.
Revised: However, this sub-sampling in the mixture-based approach has been shown to degrade other important features of multimodal VAEs, such as quality of generation, and furthermore, this degradation is theoretically unavoidable.

Original: In this study, we focus on another approach to learning the joint representation by bringing unimodal inferences closer to joint inference from all modalities, which does not have the above limitation.
Revised: In this study, we focus on another approach to learning the joint representation by bringing unimodal inferences closer to joint inference from all modalities, which does not have the above limitation.

Original: Although there have been models that can be categorized under this approach, they were derived from different backgrounds; therefore, the relation and superiority between them were not clear.
Revised: Although there have been models that can be categorized under this approach, they were derived from different backgrounds; therefore, the relation and superiority between them were not clear.

Original: To take a unified view, we first categorize them as coordination-based multimodal VAEs and show that these can be derived from the same multimodal evidence lower bound (ELBO) and that the difference in their performance is related to whether they are more tightly lower bounded.
Revised: To take a unified view, we first categorize them as coordination-based multimodal VAEs and show that these can be derived from the same multimodal evidence lower bound (ELBO) and that the difference in their performance is related to whether they are more tightly lower bounded.

Original: Next, we point out that these existing coordination-based models perform poorly on cross-modal generation (or cross-coherence) because they do not learn to reconstruct modalities from unimodal inferences.
Revised: Next, we point out that these existing coordination-based models perform poorly on cross-modal generation (or cross-coherence) because they do not learn to reconstruct modalities from unimodal inferences.

Original: Therefore, we propose a novel coordination-based model that incorporates these unimodal reconstructions, which avoids the limitations of both mixture and coordination-based models.
Revised: Therefore, we propose a novel coordination-based model that incorporates these unimodal reconstructions, which avoids the limitations of both mixture and coordination-based models.

Original: Experiments with diverse and challenging datasets show that the proposed model mitigates the limitations in multimodal
Revised: Experiments with diverse and challenging datasets show that the proposed model mitigates the limitations in multimodal

Original: VAEs and performs well in both cross-coherence and generation quality.
Revised: VAEs and performs well in both cross-coherence and generation quality.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep generative models have recently shown high performance on multimodal data, including images and captions.
Revised: Deep generative models have recently shown high performance on multimodal data, including images and captions.

Original: In particular, multimodal learning with variational autoencoders (VAEs) (Kingma &
Revised: In particular, multimodal learning with variational autoencoders (VAEs) (Kingma &

Original: Welling, 2013; Rezende et al., 2014) has attracted much attention to obtain a joint representation from modalities (Wu & Goodman, 2018; Shi et al., 2019; Sutter et al., 2021).
Revised: Welling, 2013; Rezende et al., 2014) has attracted much attention to obtain a joint representation from modalities (Wu & Goodman, 2018; Shi et al., 2019; Sutter et al., 2021).

Original: Such representations can be used for predicting common concepts from these modalities or for generating other modalities.
Revised: Such representations can be used for predicting common concepts from these modalities or for generating other modalities.

Original: An essential challenge of multimodal VAEs is inferring a joint representation from subsets of modalities.
Revised: An essential challenge of multimodal VAEs is inferring a joint representation from subsets of modalities.

Original: State-of-the-art models attempt to accomplish this by training to generate all modalities from a joint representation inferred from missing (or sub-sampled) modalities.
Revised: State-of-the-art models attempt to accomplish this by training to generate all modalities from a joint representation inferred from missing (or sub-sampled) modalities.

Original: These are called mixture-based multimodal VAEs (Daunhawer et al., 2021a), and examples include MMVAE (Shi et al., 2019) and MoPoE-VAE (Sutter et al., 2021) 1 .
Revised: These are called mixture-based multimodal VAEs (Daunhawer et al., 2021a), and examples include MMVAE (Shi et al., 2019) and MoPoE-VAE (Sutter et al., 2021) 1 .

Original: However, it has been pointed out that the quality of modality generation is lower than that of unimodal VAEs, and furthermore, cross-generation between modalities, or cross-coherence, can be degraded.
Revised: However, it has been pointed out that the quality of modality generation is lower than that of unimodal VAEs, and furthermore, cross-generation between modalities, or cross-coherence, can be degraded.

Original: This is an inherent limitation on mixturebased VAEs due to modality sub-sampling during training, and theoretical and empirical evidence shows that this limitation cannot be avoided (Daunhawer et al., 2021a).
Revised: This is an inherent limitation on mixturebased VAEs due to modality sub-sampling during training, and theoretical and empirical evidence shows that this limitation cannot be avoided (Daunhawer et al., 2021a).

Original: To alleviate this issue, we focus on another approach in multimodal VAEs that brings the representation inferred from each modality closer to that inferred from all modalities.
Revised: To alleviate this issue, we focus on another approach in multimodal VAEs that brings the representation inferred from each modality closer to that inferred from all modalities.

Original: This approach avoids the limitation of degraded generation quality because its objective does not include generation from sub-sampled modalities.
Revised: This approach avoids the limitation of degraded generation quality because its objective does not include generation from sub-sampled modalities.

Original: Models that fit this approach have been proposed, but their objectives are derived from different backgrounds.
Revised: Models that fit this approach have been proposed, but their objectives are derived from different backgrounds.

Original: For example, MVTCAE (Hwang et al., 2021), which can be regarded as one of them, is derived its objective from the total correlation; therefore, its relationship to the multimodal VAEs’ objective, multimodal evidence lower bound (ELBO), is not clear.
Revised: For example, MVTCAE (Hwang et al., 2021), which can be regarded as one of them, is derived its objective from the total correlation; therefore, its relationship to the multimodal VAEs’ objective, multimodal evidence lower bound (ELBO), is not clear.

Original: Moreover, another model of this approach, MMJSD (Sutter et al., 2020), has been shown experimentally to perform worse than MVTCAE (Hwang et al., 2021), but the theoretical reason for this is unclear.
Revised: Moreover, another model of this approach, MMJSD (Sutter et al., 2020), has been shown experimentally to perform worse than MVTCAE (Hwang et al., 2021), but the theoretical reason for this is unclear.

Original: We first categorize them as coordination-based multimodal VAEs 2 and show that these models can be viewed as unified, i.e., all are derived from multimodal ELBO, and that the differences in models correspond to whether they are more tightly lower bounded.
Revised: We first categorize them as coordination-based multimodal VAEs 2 and show that these models can be viewed as unified, i.e., all are derived from multimodal ELBO, and that the differences in models correspond to whether they are more tightly lower bounded.

Revised: We also prove that MVTCAE encourages unimodal posterior to approach the average of joint posterior, which might be one of the reasons for learning good inference.

Original: Next, we point out that the existing coordination-based VAEs still have a shortcoming in crosscoherence since no generation from unimodal inferences is included in the training.
Revised: Next, we point out that the existing coordination-based VAEs still have a shortcoming in cross-coherence since no generation from unimodal inferences is included in the training.

Original: We then propose a novel objective that introduces unimodal reconstruction terms.
Revised: We then propose a novel objective that introduces unimodal reconstruction terms.

Original: This objectivecan also be regarded as a multimodal ELBO, and since it does not include cross-generation, it also avoids the generation degradation issue of the mixture-based models while mitigating the issue of coordination-based models. 
Revised: Since this objective does not include cross-generation, it also avoids the generation degradation issue of the mixture-based models while mitigating the issue of coordination-based models. Theoretical consideration also shows that this objective mitigates a negative effect in the inference of representations.

Original: Note that our model alleviates the issues by only changing the objective, unlike other methods for improving multimodal VAEs by changing the architecture, such as introducing additional latent variables (Tsai et al., 2018; Hsu & Glass, 2018; Sutter et al., 2020; Palumbo et al., 2022).
Revised: Note that our model alleviates the issues by only changing the objective, unlike other methods for improving multimodal VAEs by changing the architecture, such as introducing additional latent variables (Tsai et al., 2018; Hsu & Glass, 2018; Sutter et al., 2020; Palumbo et al., 2022).

Original: We conducted experiments on five diverse multimodal datasets, including challenging ones, e.g., those that cannot be adequately trained with existing models or require additional architecture for adequate training.
Revised: We conducted experiments on five diverse multimodal datasets, including challenging ones, e.g., those that cannot be adequately trained with existing models or require additional architecture for adequate training.

Original: We confirmed that the proposed method outperforms existing mixture- and coordination-based models in terms of cross-coherence and generation quality.
Revised: We confirmed that the proposed method outperforms existing mixture- and coordination -based models in terms of cross-coherence and generation quality

Original: P RELIMINARIES 2.  M ULTIMODAL VAE S
Revised:  M ULTIMODAL VAE S

Original: Suppose that we have a set of multimodal examples, where each example is a set of M modalities
Revised: Suppose that we have a set of multimodal examples, where each example is a set of M modalities

Original: X = { x m } Mm =1 .
Revised: X = { x m } Mm =1 .

Original: We assume that these examples are derived from data distribution p d ( X ) and that each example X has a corresponding common latent concept z , i.e., a joint representation.
Revised: We assume that these examples are derived from data distribution p d ( X ) and that each example X has a corresponding common latent concept z , i.e., a joint representation.

Original: Given a training set { X ( i ) } Ni =1 , our goals are to infer a joint representation from the subset of modalities X S ⊆
Revised: Given a training set { X ( i ) } Ni =1 , our goals are to infer a joint representation from the subset of modalities X S ⊆

Original: X 3 and to generate another subset
Revised: X 3 and to generate another subset

Original: X S ′ ⊆ X via that representation.
Revised: X S ′ ⊆ X via that representation.

Original: In other words, we aim to obtain an inference p ( z | X ) and a joint distribution p ( X, z ) =
Revised: In other words, we aim to obtain an inference p ( z | X ) and a joint distribution p ( X, z ) =

Original: (cid:81)
Revised: (cid:81)

Original: x m ∈
Revised: x m ∈

Original: X p ( x m | z ) p ( z ) .
Revised: X p ( x m | z ) p ( z ) .

Original: To achieve this, it is a natural choice to use VAEs (Kingma & Welling, 2013), deep generative models that can learn inference to acquire representations from data in addition to the generation.
Revised: To achieve this, it is a natural choice to use VAEs (Kingma & Welling, 2013), deep generative models that can learn inference to acquire representations from data in addition to the generation.

Original: The generative model p θ ( X | z ) = (cid:81) m : x m ∈
Revised: The generative model p θ ( X | z ) = (cid:81) m : x m ∈

Original: X p θ ( x m | z ) is parameterized by deep neural networks, and the prior of the latent variable is set as a standard Gaussian p ( z ) =
Revised: X p θ ( x m | z ) is parameterized by deep neural networks, and the prior of the latent variable is set as a standard Gaussian p ( z ) =

Original: N ( 0 , I ) .
Revised: N ( 0 , I ) .

Original: The objective of VAEs is to maximize the expected log-likelihood over data distribution Since it is tractable to optimize this likelihood directly, we introduce an approximated posterior q ϕ ( z | X ) and instead optimize the following evidence lower bound (ELBO) on the expected log-likelihood:
Revised: The objective of VAEs is to maximize the expected log-likelihood over data distribution Since it is tractable to optimize this likelihood directly, we introduce an approximated posterior q ϕ ( z | X ) and instead optimize the following evidence lower bound (ELBO) on the expected log-likelihood:

Original: [Equation]
Revised: [Equation]

Original: In this paper, the model group that maximizes this multimodal ELBO is collectively called multimodal
Revised: In this paper, the model group that maximizes this multimodal ELBO is collectively called multimodal

Original: VAEs .
Revised: VAEs .

Original: We aim to optimize Eq.
Revised: We aim to optimize Eq.

Original: 1 to obtain a joint representation and generate modalities from it given a modality subset X S .
Revised: 1 to obtain a joint representation and generate modalities from it given a modality subset X S .

Original: However, Eq. 1 only includes the inference given all modalities; therefore, it does not learn to infer representations from subsets.
Revised: However, Eq. 1 only includes the inference given all modalities; therefore, it does not learn to infer representations from subsets.

Original: 2. M IXTURE - BASED MULTIMODAL VAE S
Revised: 2. M IXTURE - BASED MULTIMODAL VAE S

Original: An inference from subsets is commonly expressed as a combination of unimodal posteriors from each modality.
Revised: An inference from subsets is commonly expressed as a combination of unimodal posteriors from each modality.

Original: Product of experts (PoE) (Hinton, 2002; Wu & Goodman, 2018) q PoEϕ ( z | X S ) ≡ p ( z ) (cid:81) m : x m ∈ X S q ϕ
Revised: Product of experts (PoE) (Hinton, 2002; Wu & Goodman, 2018) q PoEϕ ( z | X S ) ≡ p ( z ) (cid:81) m : x m ∈ X S q ϕ

Original: ( z | x m ) and mixture of experts (MoE) q MoEϕ ( z | X S ) ≡ (cid:80) m
Revised: ( z | x m ) and mixture of experts (MoE) q MoEϕ ( z | X S ) ≡ (cid:80) m

Original: [Equation]
Revised: [Equation]

Original: ϕ ( z | x m ) are known as ways to aggregate inferences.
Revised: ϕ [Equation] z | x m ) are known as ways to aggregate inferences.

Original: Moreover, the following mixture of products of experts (MoPoE) (Sutter et al., 2021) generalizes them:
Revised: Moreover, the following mixture of products of experts (MoPoE) (Sutter et al., 2021) generalizes them:

Original: [Equation]
Revised: [Equation]

Original: where (cid:80) S : X S ∈P
Revised: where (cid:80) S : X S ∈P

Original: ( X ) ω S = 1 , ω S ∈ [0 , 1] , and P ( X ) is a powerset of X .
Revised: ( X ) ω S = 1 , ω S ∈ [0 , 1] , and P ( X ) is a powerset of X .

Original: This MoPoE aggregation is equivalent to PoE when X S =
Revised: This MoPoE aggregation is equivalent to PoE when X S =

Original: X and MoE when | X S
Revised: X and MoE when | X S

Original: | = 1 for all X S ∈ P ( X ) .
Revised: | = 1 for all X S ∈ P ( X ) .

Original: Substituting the MoPoE-aggregated joint posterior (Eq. 2) into Eq.
Revised: Substituting the MoPoE-aggregated joint posterior (Eq. 2) into Eq.

Original: 1, we obtain the following:
Revised: 1, we obtain the following:

Original: L ( θ, ϕ ) ≥ E p d ( X )
Revised: L [Equation] ≥ [Equation]

Revised: d

Original: [Equation]
Revised: [Equation]

Original: Multimodal VAEs with Eq.
Revised: Multimodal VAEs with Eq.

Original: 3 are called mixture-based multimodal VAEs (Daunhawer et al., 2021a).
Revised: 3 are called mixture-based multimodal VAEs (Daunhawer et al., 2021a).

Original: MMVAE (Shi et al., 2019) and MoPoE-VAE (Sutter et al., 2021) are members of this family.
Revised: MMVAE (Shi et al., 2019) and MoPoE-VAE (Sutter et al., 2021) are members of this family.

Original: Comparing with Eq.
Revised: Comparing with Eq.

Original: 1, we can see that the reconstruction term (the first term in the expectation) of
Revised: 1, we can see that the reconstruction term (the first term in the expectation) of

Original: Eq. 1 is designed to generate all modalities from the sub-sampled modality subset.
Revised: Eq. 1 is designed to generate all modalities from the sub-sampled modality subset.

Original: In other words, mixture-based multimodal VAEs encourage cross-generation during training, aiming at inferring a consistent joint representation from subset modalities and generating modalities from it.
Revised: In other words, mixture-based multimodal VAEs encourage cross-generation during training, aiming at inferring a consistent joint representation from subset modalities and generating modalities from it.

Original: However, mixture-based VAEs have a severe issue in generating modalities, which is one of the essential goals of multimodal VAEs.
Revised: However, mixture-based VAEs have a severe issue in generating modalities, which is one of the essential goals of multimodal VAEs.

Original: Daunhawer et al.
Revised: Daunhawer et al.

Original: (2021a) show that the following inequality holds between Eq.
Revised: (2021a) show that the following inequality holds between Eq.

Original: 3 and the expected marginal log-likelihood:
Revised: 3 and the expected marginal log-likelihood:

Original: [Equation]
Revised: [Equation]

Original: where ∆( X ) ≡ (cid:80) S : X S ∈P
Revised: where ∆( X ) ≡ (cid:80) S : X S ∈P

Original: ( X )
Revised: ( X )

Original: w S H
Revised: w S H

Original: ( X { 1 ,...,M }\ S | X S )  is the discrepancy independent of the learning parameters.
Revised: ( X { 1 ,...,M }\ S | X S ) 4 is the discrepancy independent of the learning parameters.

Original: This inequality means that multimodal ELBO of mixture-based VAEs cannot be tighter than this discrepancy ∆( X ) no matter how much it is maximized.
Revised: This inequality means that multimodal ELBO of mixture-based VAEs cannot be tighter than this discrepancy ∆( X ) no matter how much it is maximized.

Original: Therefore, the quality of their generating modalities always degrades compared to unimodal VAEs.
Revised: Therefore, the quality of their generating modalities always degrades compared to unimodal VAEs.

Original: Moreover, this discrepancy, by definition, depends on how much different information one modality subset X { 1 ,...,M }\ S has compared to the rest of them X S .
Revised: Moreover, this discrepancy, by definition, depends on how much different information one modality subset X { 1 ,...,M }\ S has compared to the rest of them X S .

Original: Therefore, the more information specific to each modality, the larger this quantity and the worse the generation quality.
Revised: Therefore, the more information specific to each modality, the larger this quantity and the worse the generation quality.

Original: This also leads to degrading cross-coherence (Shi et al., 2019), or the ability to generate the same semantic meaning across modalities conditionally.
Revised: This also leads to degrading cross-coherence (Shi et al., 2019), or the ability to generate the same semantic meaning across modalities conditionally.

Original: Intuitively, mixture-based VAEs are required to take sub-sampled subsets as inputs to generate the rest, but these inputs have insufficient information for that, resulting in degraded cross-generation.
Revised: Intuitively, mixture-based VAEs are required to take sub-sampled subsets as inputs to generate the rest, but these inputs have insufficient information for that, resulting in degraded cross-generation.

Original: M ETHODS 3. A VOIDING THE LIMITATION OF MULTIMODAL VAE S
Revised: 2. A VOIDING THE LIMITATION OF MULTIMODAL VAE S

Original: What brings about this irreducible discrepancy ∆( X ) ?
Revised: What brings about this irreducible discrepancy ∆( X ) ?

Original: According to the proof by Daunhawer et al.
Revised: According to the proof by Daunhawer et al.

Original: (2021a)
Revised: (2021a)

Original: (see B.4 in Daunhawer et al.
Revised: (see B.4 in Daunhawer et al.

Original: (2021a))
Revised: (2021a))

Original: , it is due to the inclusion in the objective of a cross-generation term, i.e., a term that generates unobserved modalities from observed modalities.
Revised: , it is due to the inclusion in the objective of a cross-generation term, i.e., a term that generates unobserved modalities from observed modalities.

Original: Therefore, one promising way to avoid this limitation is to not include in the objective of the multimodal VAE the cross-generation term 4 .
Revised: Therefore, one promising way to avoid this limitation is to not include in the objective of the multimodal VAE the cross-generation term 5 .

Original: Note that MVAE (Wu & Goodman, 2018), which first uses PoE for the joint posterior, uses the sum of ELBOs over all modality subsets as the objective, but does not include cross-generation and only includes reconstruction of subsets 5 , thus enabling high-quality generation.
Revised: Note that MVAE (Wu & Goodman, 2018), which first uses PoE for the joint posterior, uses the sum of ELBOs over all modality subsets as the objective, but does not include cross-generation and only includes reconstruction of subsets 6 , thus enabling high-quality generation.

Original: However, MVAE cannot perform cross-generation between different modalities because its objective does not include terms to encourage it.
Revised: However, MVAE cannot perform cross-generation between different modalities because its objective does not include terms to encourage it.

Original: Despite this significant drawback, determining how to work around the limitation of mixture-based models leads to our proposed model described later.
Revised: Despite this significant drawback, determining how to work around the limitation of mixture-based models leads to our proposed model described later.

Original: 3.  C OORDINATION - BASED MULTIMODAL VAE S
Revised:  C OORDINATION - BASED MULTIMODAL VAE S

Original: In the study of multimodal VAEs, MVTCAE (Hwang et al., 2021) and MMJSD (Sutter et al., 2020) have been proposed to learn to infer joint representation from subset modalities by bringing joint and unimodal posteriors closer together.
Revised: MVTCAE (Hwang et al., 2021) and MMJSD (Sutter et al., 2020) have been proposed to learn to infer joint representation from subset modalities by bringing joint and unimodal posteriors closer together.

Original: They omit sub-sampling in the objective unless the posterior is a mixture distribution and thus can learn the joint representation without being affected by the limitation of the mixture-basedVAEs.
Revised: We find that they omit sub-sampling in the objective unless the posterior is a mixture distribution and thus can learn the joint representation without being affected by the limitation of the mixture-based

Original: However, they are not proposed in light of the above limitation nor derived from the same objective.
Revised: VAEs .

Original: Therefore, we first consider them as a unified coordination-based multimodal VAEs and analyze their relation 6 .
Revised: We first categorize them as a unified coordination-based multimodal VAEs 7 .

Original: 3.2. MVTCAE 
Revised: 3. MVTCAE AND MMJSD

Original: Hwang et al.

Original: (2021) derive the following objective as a lower bound for the total correlation:
Revised: The objectives of MVTCAE and MMJSD are expressed as follows:

Original: [Equation]
Revised: [Equation]

Original: where (cid:80) m π m = 1 and π m ∈ [0 , 1] .
Revised: where (cid:80) m π m = 1 , π m ∈ [0 , 1] , and function f M defines the mixture of { q ϕ ( z | x m )

Original: Note that we changed the coefficients and removed nonoptimized terms from the original paper (Hwang et al., 2021).
Revised: } Mm =0 .

Original: Moreover, for simplicity of notation, we use q ϕ 0 ( z | x 0 ) ≡ p ( z ) .
Revised: For simplicity of notation, we use q ϕ 0 ( z | x 0 ) ≡ p ( z ) .

Original: The model trained with this objective  is called MVTCAE.
Revised: We denote the entire objective of MVTCAE and MMJSD as L C ( θ, ϕ ) and L C ′ ( θ, ϕ ) , respectively. MVTCAE is derived as a lower bound for the total correlation 8 .

Original: In particular, when M = 2 , this model is equal to JMVAE (Suzuki et al., 2016).
Revised: When M = 2 , this model is equal to JMVAE (Suzuki et al., 2016).

Original: However, JMVAE uses a network for joint posterior separately from unimodal posteriors, while MVTCAE represents joint posteriors by aggregating unimodal posteriors in PoE. 3.2. MMJSD
Revised: JMVAE uses a network for joint posterior separately from unimodal posteriors, while MVTCAE represents joint posteriors by PoE-aggregating unimodal posteriors.

Original: Sutter et al.

Original: (2020) propose the following objective, called MMJSD, which replaces the KL divergence of the multimodal ELBO with the Jensen-Shannon divergence: [Equation] where function f M defines the mixture distribution of { q ϕ ( z | x m ) } Mm =0 .
Revised: MMJSD is derived by replacing the KL divergence of the multimodal ELBO with the Jensen-Shannon divergence.

Original: It is proved (in Lemma 2 in Sutter et al.
Revised: It is proved (in Lemma 2 in Sutter et al.

Original: that Eq. 6 is a lower bound for multimodal ELBO with f M ( z | X ) replaced by thefollowing dynamic prior: p f ( z | X ) =
Revised: that L C ′ ( θ, ϕ ) is a lower bound for multimodal ELBO with f M ( z | X ) replaced by the dynamic prior: p f ( z | X ) =

Original: f ( { q ϕ ( z | x m ) } Mm =0 ) , where f is a function that encompasses each unimodal posterior, and Sutter et al.
Revised: f ( { q ϕ ( z | x m ) } Mm =0 ) , where f is a function that encompasses each unimodal posterior, and Sutter et al.

Original: (2020) choose the weighted PoE. With this replacement for the dynamic prior, MMJSD can be regarded as a member of multimodal VAEs 7 . Moreover, this replacement can be thought of as introducing the prior knowledge that all modalities have the same joint representation (Sutter et al., 2020).
Revised: (2020) choose the weighted PoE. This replacement can be thought of as introducing the prior knowledge that all modalities have the same joint representation (Sutter et al., 2020).

Original: Note that in Sutter et al.
Revised: Note that in Sutter et al.

Original:  the posterior of MMJSD is assumed to be MoE, which is affected by the limitation of mixture-based VAEs.
Revised: (2020), the posterior of MMJSD is assumed to be MoE, which is affected by the limitation of mixture-based VAEs.

Original: Since we would like to focus on the properties of MMJSD contributing to its coordination-based nature, we call MMJSD-PoE the model that replaces the posterior inthe reconstruction term of Eq.
Revised: Since we would like to focus on the properties of MMJSD contributing to its coordination-based nature, we call MMJSD-PoE the model that replaces the posterior in Eq.

Original: 6 with PoE. Hereafter, MMJSD is positioned in this paper as a member of mixture-based VAEs, and MMJSD-PoE as coordination-based VAEs.

Original:  Compared to the objective of MVTCAE,we can see that the direction of KL divergence in Eq.
Revised: 6 with PoE. Compared to the objective of MVTCAE, the direction of KL divergence in Eq.

Original: 6 is reversed: MVTCAE uses forward KL divergence, while MMJSD(-PoE) uses reverse KL divergence.
Revised: 6 is reversed: MVTCAE uses forward KL divergence, while MMJSD(-PoE) uses reverse KL divergence.

Original: (Hwang et al., 2021) shows that MVTCAE has betterperformance than MMJSD by experiment, but the theoretical reason is still unclear.
Revised: Hwang et al. (2021) shows that MVTCAE performs better than MMJSD by experiment.

Original: 3. R ELATION BETWEEN COORDINATION - BASED MULTIMODAL VAE S
Revised: 3. D ISCOVERING THE RELATION BETWEEN COORDINATION - BASED MULTIMODAL VAE S

Original:  In this section, by using the dynamic prior, we  show that: 1) the objective of MVTCAE is multimodal ELBO; 2) in special cases, the objective of MVTCAE is an upper bound on the objective of MMJSD.
Revised: Because MVTCAE and MMJSD were proposed for different purposes, their relationship is unclear. In particular, since MVTCAE was derived from the total correlation, it is unclear whether it belongs to multimodal VAEs (in the sense of valid ELBOs). In light of the above, we first show that the objective of MVTCAE is multimodal ELBO.

Original: Lemma 1{+
+} The objective of MVTCAE in Eq. 5 is equal to the multimodal ELBO in Eq.
Revised: Lemma {+
+} The objective of MVTCAE L C ( θ, ϕ ) is equal to the multimodal ELBO in Eq.

Original: 1 with a weighted PoE-based dynamic prior as its prior.
Revised: 1 with a weighted PoE-based dynamic prior as its prior.

Original: Proof 1 Following MMJSD, we introduce a weighted PoE-based dynamic prior for Eq.
Revised: Proof 1 Following MMJSD, we introduce a weighted PoE-based dynamic prior for Eq.

Original: 3 as follows:
Revised: 3 as follows:

Original: [Equation]
Revised: [Equation]

Revised: p d ( X )

Revised: [Equation]

Revised: (cid:35)(cid:35) =

Revised: L C ( θ

Revised: [Equation]

Original:  Therefore, MVTCAE can be considered a member of multimodal VAEs.
Revised: ϕ ) . □ Therefore, MVTCAE can be considered a member of multimodal VAEs.

Original: Furthermore, this obviously indicates that the objective of JMVAE (Suzuki et al., 2016) is also multimodal ELBO.
Revised: This indicates that the objective of JMVAE (Suzuki et al., 2016) is also multimodal ELBO.

Revised: Note that no assumption is made about the approximate posterior in Lemma 1, and thus holds whether it is aggregated by PoE or MoE.

Revised: Next, in special cases, the objective of MVTCAE is an upper bound on the objective of MMJSD.

Original: Lemma 

Revised: Lemma

Original: If the posterior is MoE-aggregated, the objective of MVTCAE in Eq. 5 is an upper bound on that of MMJSD in Eq. 6, i.e., the following inequality holds: L C ( θ, ϕ ) ≥ L C ′ ( θ, ϕ ) .
Revised: If the posterior is MoE-aggregated, the objective of MVTCAE L C ( θ, ϕ ) is an upper bound on that of MMJSD L C ′ ( θ, ϕ ) , i.e., the following inequality holds: L C ( θ, ϕ ) ≥ L C ′ ( θ, ϕ ) .

Original: Proof 2 Lemma 1 shows that the objective of MVTCAE is equal to a multimodal ELBO with a dynamic prior.
Revised: Proof 2 Lemma 1 shows that the objective of MVTCAE is equal to a multimodal ELBO with a dynamic prior.

Original: In addition, Lemma 2 in Sutter et al.
Revised: In addition, Lemma 2 in Sutter et al.

Original: (2020) states that the objective of MMJSD is a lower bound on multimodal ELBO with a dynamic prior and MoE-aggregated posterior.
Revised: (2020) states that the objective of MMJSD is a lower bound on multimodal ELBO with a dynamic prior and MoE-aggregated posterior.

Original: Therefore, the objective of MMJSD is a lower bound on that of MVTCAE if the posterior is MoE-aggregated.
Revised: Therefore, the objective of MMJSD is a lower bound on that of MVTCAE if the posterior is MoE-aggregated.

Original: □
Revised: □

Original: Lemma 2 indicates that MVTCAE better approximates the true log-likelihood and thus promotes  good generation, which provides one possible explanation as to why MVTCAE performs better than
Revised: Lemma 2 indicates that MVTCAE better approximates the true log-likelihood and thus promotes the good generation, which provides one possible explanation as to why MVTCAE performs better than

Original: MMJSD.
Revised: MMJSD.

Original: Note that Lemma 2 comes from the fact that the joint posterior of the KL divergence in
Revised: Note that Lemma 2 comes from the fact that the joint posterior of the KL divergence in

Original: MMJSD assumes MoE; therefore, it is still unclear whether this holds when the posterior is PoE.
Revised: MMJSD assumes MoE; therefore, it is still unclear whether this holds when the posterior is PoE.

Revised: Furthermore, the following proof shows why this direction of KL divergence for MVTCAE is good for multimodal VAEs from a different perspective.

Original: {+Lemma 
+}
Revised: {+Lemma 
+}

Revised: The KL divergence for each m (Eq. 5) brings the average of the joint approximate

Revised: posterior q avgϕ ( z | x m ) =

Revised: (cid:82) q ϕ

Original: 3. C OORDINATED AND UNIMODAL R ECONSTRUCTED M ULTIMODAL VAE S
Revised: ( z | X ) p d ( X C/m | x m ) dX C/m closer to the unimodal posterior q ϕ ( z | x m ) . Proof 3 The KL divergence for x m (Eq. 5) is transformed as follows: [Equation] where I ϕ ( X C/m ; z | x m ) is a conditional mutual information and holds greater than 0.

Revised: Therefore, maximizing the negative KL term (Eq. 5) leads to minimizing the second term in Eq.

Original: Coordination-based multimodal VAEs are intended to keep inference from any given modalities consistent by bringing joint and unimodal posteriors closer.
Revised: □ Lemma 3 holds for any approximate posterior. Therefore, another possible reason MVTCAE is good as multimodal VAEs is that maximizing Eq. 5 explicitly brings the unimodal posterior closer to the joint posterior with the unimodal input.

Original: However, in practice, it is impossible to make them perfectly close; therefore, even if modality generation is appropriately learned from the joint posterior in  the reconstruction term, it remains difficult to properly generate modalities  from  the unimodal posterior.
Revised: If the joint inference is properly learned in the first term of L C ( θ, ϕ ) , then the unimodal inference can also be properly performed, as shown in Lemma 3. Furthermore, when we choose the PoE-aggregated joint inference, the PoE property allows the inference from arbitrary modalities to be adequately represented as a product of the unimodal posterior, provided that it is adequately approximated. However, L C ( θ, ϕ ) only includes reconstruction from all modalities and does not include generation from arbitrary modalities; thus, the performance of cross-modal generation might be degraded.

Original: On the other hand,including cross-generation in the objective conversely promotes generation degradation due to the limitation of mixture-based models.
Revised: On the other hand, cross-generation in the objective conversely promotes generation degradation due to the limitation of mixture-based models.

Original: Therefore, for the  unimodal posterior to experience  generation from its inference during learning while avoiding these limitations, including a term for unimodal reconstruction in the objective is an appropriate way.
Revised: We propose to include the term for unimodal reconstruction in the objective to experience the generation from unimodal posterior during learning while avoiding these dilemmas.

Original: The reconstruction term in Eq.
Revised: The reconstruction term in Eq.

Original: 5 can be rewritten as
Revised: 5 can be rewritten as

Original: [Equation]
Revised: [Equation]

Original:  where we use (cid:82) q ϕ ( z | X ) p d ( X C/m | x m ) dX C/m =
Revised: (8) where we assume (cid:82) q ϕ ( z | X ) p d ( X C/m | x m ) dX C /m ≈ q ϕ

Original: q ϕ ( z | x m ) 8 for each x m in the deformation of Eq.
Revised: ( z | x m ) 9 . This approximation changes to equal if the second term of Eq.

Original: Therefore, the objectiveof MVTCAE can be transformed to include all-modality and unimodal reconstructions, remaining a valid multimodal ELBO. This also mitigates the shortcoming of coordination-based VAEs while avoiding the limitation of mixture-based VAEs.
Revised: 7 is zero, making it a valid ELBO. This rewritten objective can be converted to include all-modality and unimodal reconstruction, which mitigates the shortcoming of coordination-based VAEs while avoiding the limitation of mixture-based VAEs.

Original: We call this novel model Coordinated and unimodal Reconstructed Multimodal VAE (CRMVAE) .
Revised: We call this novel model Coordinated and unimodal Reconstructed Multimodal VAE (CRMVAE) .

Revised: We also examine the effect of this unimodal reconstruction from another perspective.

Revised: Minimizing

Revised: Eq 7 also minimizes the first term, the conditional mutual information.

Revised: This can be transformed as

Revised: [Equation]

Original: 3. R ELATED WORKS
Revised: The second and fourth terms are negative, so they are learned to improve their respective information 10 . The first term corresponds to the information bottleneck, which is the constraint for learning a good representation. Therefore, the third term can be considered a potential negative influence on representation learning. This term can be transformed according to Alemi et al.

Revised: (2016) as follows:

Revised: [Equation]

Revised: This suggests that the proposed unimodal reconstruction might play a role in counteracting the negative effects on inferred representation .

Revised: To counteract Eq 10 from the objective of CRMVAE, the coefficient in Eq.

Revised: 8 should be 12 to π in the second term of L C ( θ, ϕ ) 11 .

Original: There have been many studies of multimodal VAEs (Suzuki & Matsuo, 2022), and the early ones before mixture-based required additional inference models.
Revised: There have been many studies of multimodal VAEs (Suzuki & Matsuo, 2022), and the early ones before mixture-based required additional inference models.

Original: Suzuki et al.
Revised: Suzuki et al.

Original: (2016) proposed  to approximate the unimodal inference models to explicitly approach the joint posterior of VAEs, called JMVAE.
Revised: (2016) proposed JMVAE to approximate the unimodal posterior to explicitly approach the joint posterior of VAEs.

Original: Vedantam et al.

Original: (2018) introduced TELBO, where the objective is composed of the sum of ELBOs that take each set of modalities as input.

Original: M 2 VAE (Korthals et al., 2019) can be regarded as a combination of JMVAE and TELBO, and VAEVAE (Wu & Goodman, 2019) excludes a KL divergence term between the joint inference and prior from M 2 VAE.

Original: Except for TELBO, they can be regarded as precursors to the coordination-based approach; however, the number of inference models needed for these models grows exponentially with the number of modalities.
Revised: With a few exceptions, they can be regarded as precursors to the coordination-based approach; however, the number of posterior needed for these models grows exponentially with the number of modalities.

Original: One important research direction in multimodal VAEs is the introduction of additional latent variables, which includes hierarchical latent variables (Sutter & Vogt, 2021;
Revised: One important research direction in multimodal VAEs is the introduction of additional latent variables, which includes hierarchical latent variables (Sutter & Vogt, 2021;

Original: Wolff et al., 2021; Vasco et al., 2022) and modality-specific latent variables (Tsai et al., 2018; Hsu & Glass, 2018; Sutter et al., 2020;
Revised: Wolff et al., 2021; Vasco et al., 2022) and modality-specific latent variables (Tsai et al., 2018; Hsu & Glass, 2018; Sutter et al., 2020;

Original: Lee & Pavlovic, 2020; Daunhawer et al., 2021b; Palumbo et al., 2022).
Revised: Lee & Pavlovic, 2020; Daunhawer et al., 2021b; Palumbo et al., 2022).

Original: This paper aims to improve the performance of multimodal VAEs in a different direction from them, i.e., whether limitations of existing mixture-based models can be mitigated by changing the objective while maintaining the same architecture .
Revised: This paper aims to improve the performance of multimodal VAEs in a different direction from them, i.e., whether limitations of existing mixture-based models can be mitigated by changing the objective while maintaining the same architecture .

Original: Note that our proposed model and the introduction of additional latent variables are orthogonal concepts and can be easily incorporated into the proposed model.
Revised: Note that our proposed model and the introduction of additional latent variables are orthogonal concepts and can be easily incorporated into the proposed model.

Original: Recently, it has been proposed to translate between modalities by learning large-scale models such as transformers (Vaswani et al., 2017).

Original: Ramesh et al.

Original: (2022) used CLIP (Radford et al., 2021) for conditional generation between diverse modalities at higher performance.

Original: Bachmann et al.

Original: (2022) introduced MultiMAE, which uses vision transformers (Dosovitskiy et al., 2020) and allows crossgeneration from any modality and fine-tuning for new tasks.

Original: However, these models do not aim to obtain a consistent joint representation like multimodal VAEs.

Original: Future directions should integrate multimodal studies using these large-scale models with multimodal VAEs.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We use the following five diverse datasets for our experiments. MNIST-SVHN-Text (Sutter et al., 2020) is the MNIST-SVHN (Shi et al., 2019) with the addition of the text modality, which represents the English name of each number as text and has diversity.
Revised: We use the following five diverse datasets: MNIST-SVHN-Text (Sutter et al., 2020), PolyMNIST (Sutter et al., 2021), Translated-PolyMNIST (Daunhawer et al., 2021a), CUB Wah et al.

Original: Different modalities of the same example have the same digits.

Original: PolyMNIST (Sutter et al., 2021) consists of a set of five different MNIST images with different backgrounds and handwriting styles, each of which is considered a different modality.

Original: Translated-PolyMNIST (Daunhawer et al., 2021a) is composed of the same backgrounds as PolyMNIST with the numbers downscaled 9 and randomly transformed into position in the images.

Original: This is intended to allow for more significant variation among modalities than PolyMNIST.

Original: Caltech Birds ( CUB ) Wah et al.
Revised: (2011), and Bimodal CelebA (Sutter et al., 2020).

Original: (2011) is a dataset with bird images and their captions. We follow Daunhawer et al. (2021a), i.e., we treat as 64 × 64 real images without feature extraction. Bimodal CelebA (Sutter et al., 2020) is a dataset offace images from the CelebA (Liu et al., 2015) with additional text describing the corresponding attribute labels and changing the starting index to give a diversity.
Revised: For a more description of the dataset, see the appendix.

Original: Thesedatasets are commonly used in studies of multimodal VAEs, but to our knowledge, this is the first study  that experiments have been conducted on all of these datasets.
Revised: These are commonly used in studies of multimodal VAEs, but to our knowledge, this is the first study in that experiments have been conducted on all of these datasets.

Original: In particular, CUB and
Revised: In particular, CUB and

Original: Translated-PolyMNIST were noted to be challenging datasets that could not be successfully trained in existing models (Daunhawer et al., 2021a).
Revised: Translated-PolyMNIST were noted to be challenging datasets that could not be successfully trained in existing models (Daunhawer et al., 2021a).

Original: It also shows that Bimodal CelebA and CUB could only be learned in models with additional latent variables (Sutter et al., 2020;
Revised: It also shows that Bimodal CelebA and CUB could only be learned in models with additional latent variables (Sutter et al., 2020;

Original: Palumbo et al., 2022).
Revised: Palumbo et al., 2022).

Original: To evaluate the models  in terms of both 1) cross-coherence and 2) generation quality, we measure each of them in terms of 1) the test accuracy of a pre-trained classifier 10 that predicts labels from crossgenerated images and 2) the Fréchet inception distance (FID) (Heusel et al., 2017) in cross-generated modalities.
Revised: For models trained in all modalities, we evaluate both 1) cross-coherence and 2) generation quality given arbitrary modalities. We measure each of them in terms of 1) the test accuracy of a pre-trained classifier 12 that predicts labels from cross-generated images and 2) the Fréchet inception distance (FID) (Heusel et al., 2017) in cross-generated modalities.

Revised: For MNIST-SVHN-Text, joint coherence is also verified by generating random images in all modalities and evaluating whether all classes predicted by them are consistent.

Original: Note that we evaluated the generated results of the text modality in MNIST-SVHN-Text in terms of the Levenshtein distance from the true text (without space characters) since we found that the pre-trained classifier from Sutter et al.
Revised: Note that we evaluated the generated results of the text modality in MNIST-SVHN-Text in terms of the Levenshtein distance from the true text (without space characters) since we found that the pre-trained classifier from Sutter et al.

Original: (2021) did not correctly evaluate the generated results (we discuss indetail in the experimental results.)
Revised: (2021) did not correctly evaluate the generated results (we discuss in the experimental results.)

Original: Next, to assess the cross-coherence for CUB, we use the cosine similarity between the latent space of the two inferred modalities since there are no labels.
Revised: To assess the cross-coherence for CUB, we use the cosine similarity between the latent space of the two inferred modalities since there are no labels.

Original: We also used this evaluationof cross-coherence by cosine similarity for Bimodal CelebA.
Revised: We also used this evaluation for Bimodal CelebA.

Original: For the network architectures, we follow Sutter et al.
Revised: For the network architectures, we follow Sutter et al.

Original: ; Daunhawer et al.
Revised: ; Daunhawer et al.

Original: (2021a), with 64dimensional latent variables for Bimodal CelebA and CUB,
Revised: (2021a), with 64dimensional latent variables for Bimodal CelebA and CUB,

Original: 32 for MNIST-SVHN-Text, and 512 for
Revised: 32 for MNIST-SVHN-Text, and 512 for

Original: PolyMNIST and Translated-PolyMNIST.
Revised: PolyMNIST and Translated-PolyMNIST.

Original: The distributions for image modalities are set to Laplace, and text is set to categorical.
Revised: The distributions for image modalities are set to Laplace, these are not entropies respect to parameters of models [Table] scores for crossgeneration.

Original:  We use Adam (Kingma & Ba, 2014) for optimization with a learning rate of 0.0005 and a mini-batch size of 256.
Revised: Lower is better. [Figure] and text is set to categorical. We use Adam (Kingma & Ba, 2014) for optimization with a learning rate of 0.0005 and a mini-batch size of 256.

Original: We set β (the coefficient of the KL divergence term in the multimodal ELBO) to 1 for PolyMNIST and MNIST-SVHN-Text and 0.1 for Translated-PolyMNIST, [Table] [Table] [Figure]
Revised: We set β (the coefficient of the KL divergence term in the multimodal ELBO) to 1 for PolyMNIST and MNIST-SVHN-Text and 0.1 for Translated-PolyMNIST, π m to 1 M +1 , and ω S to 12 M .

Original: [Figure]

Original: to 1 M +1 , and ω S to 12 M .

Original: We train 100 epochs for MNIST-SVHN-Text, 150 for CUB, 200 for
Revised: We train 100 epochs for MNIST-SVHN-Text, 150 for CUB, 200 for

Original: Bimodal CelebA, and 500 for PolyMNIST and Translated-PolyMNIST.
Revised: Bimodal CelebA, and 500 for PolyMNIST and Translated-PolyMNIST.

Original: All experimental results are the average of three runs with different seeds.
Revised: All experimental results are the average of three runs with different seeds.

Original: We implement them using PyTorch (Paszke et al., 2019).
Revised: We implement them using PyTorch (Paszke et al., 2019).

Original: Our experiments were performed using two machines with five NVIDIA Titan RTX GPUs.
Revised: Our experiments were performed using two machines with five NVIDIA Titan RTX GPUs.

Original: We compare CRMVAE with MVAE, mixture-based VAEs (MMVAE, MoPoE, and MMJSD), and coordination-based VAEs (MVTCAE and MMJSD-PoE), all with the same architecture settings.
Revised: We compare CRMVAE with MVAE, mixture-based VAEs (MMVAE, MoPoE, and MMJSD), and coordination-based VAEs (MVTCAE and MMJSD-PoE), all with the same architecture settings.

Original: 4. MNIST-SVHN-T EXT
Revised: 6. MNIST-SVHN-T EXT

Original: Table 1 shows the  cross-coherence results, and Table 2 shows the FID scores of images on MNISTSVHN-Text.
Revised: Table 1 shows the joint and cross-coherence results, and Table 2 shows the FID scores of images on MNIST-SVHN-Text.

Original: First, we can confirm that MVAE has high generation quality but the lowest crosscoherence since it learns only the modality reconstruction.
Revised: First, we can confirm that MVAE has high generation quality but the lowest cross-coherence since it learns only the modality reconstruction.

Original: We next find that mixture-based models perform poorly in cross-generation of SVHN and Text.
Revised: We next find that mixture-based models perform poorly in cross-generation of SVHN and Text 13 .

Original: Note here that the results of Text differ significantly from the results of previous studies because the Levenshtein distance is used for the generated results, not for the pre-trained classifiers.

Original: According to the qualitative results shown in Figure 1, the cross-generation of Text on MoPoE-VAE is not successful; however, the pre-trained classifier used in the previous study evaluates it as a high performance.
Revised: According to the qualitative results shown in Figure 1, the cross-generation of Text on MoPoE-VAE is unsuccessful.

Original: On the other hand, we find that the evaluation using the Levenshtein distance is consistent with the qualitative results, so we consider this more appropriate.

Original: While Table 1 shows that cross-generation of MNIST appears to perform well, the qualitative resultsin Figure 1 show that the generated images are average with no variation.
Revised: While Tableshows that cross-generation of MNIST appears to perform well, the qualitative results show that the generated images are average with no variation.

Original: Table 2 indicates that FID scores of mixture-based models on MNIST are significantly lower than these of MVAE, which corresponds to these results.
Revised: Table 2 indicates that FID scores of mixture-based models on MNIST are significantly lower than these of MVAE, which corresponds to these results.

Original: Existing coordination-based models mitigate this limitation and perform better on SVHN.
Revised: Existing coordination-based models mitigate this limitation and perform better on SVHN.

Original: Furthermore, FID scores show that the coordination-based models significantly outperform the mixturebased models in terms of generation quality and are comparable to MVAE.
Revised: Furthermore, FID scores show that the coordination-based models significantly outperform the mixturebased models in terms of generating quality and are comparable to MVAE.

Original: Comparing MMJSD-PoE and MVTCAE, MVTCAE performs better in terms of both cross-coherence and generation quality, confirming the effectiveness of MVTCAE experimentally.
Revised: Comparing MMJSD-PoE and MVTCAE, MVTCAE performs better in terms of both cross-coherence and generation quality, confirming the effectiveness of MVTCAE experimentally.

Original: However, the performance of the coordination-basedapproach is degraded by cross-generation, especially  from SVHN.
Revised: However, the performance of the coordination-based is degraded, especially in cross-generation from SVHN.

Original: CRMVAE mitigates coordination-based issues and significantly improves cross-coherence performance.
Revised: CRMVAE mitigates coordination-based issues and significantly improves cross-coherence performance.

Original: In particular, CRMVAE outperforms all existing methods, including mixture-based methods, for cross-coherence of SVHN and Text.
Revised: In particular, CRMVAE outperforms all existing methods, including mixture-based methods, for cross-coherence of SVHN and Text.

Original: We also confirm that CRMVAE retains the advantage of the coordination-based models, i.e., high generation quality.
Revised: We also confirm that CRMVAE retains the advantage of the coordination-based models, i.e., high-generation quality.

Original: In particular, we have confirmed the highest quality of cross-generation on MNIST among all models that include MVAE.
Revised: In particular, we have confirmed the highest quality of cross-generation on MNIST among all models that include MVAE.

Original: The visualization results also confirm that CRMVAE achieves high coherence while maintaining high quality.
Revised: The visualization results also confirm that CRMVAE achieves high coherence while maintaining high quality.

Revised: The poor performance of joint coherence results generated from random images compared to other studies might be partly due to the different methods of text evaluation.

Revised: The performance of the coordination-based method is relatively high but not significant.

Revised: The relationship between joint coherence and coordination-based should be studied further in the future.

Original: 4. P OLY MNIST AND T RANSLATED -P OLY MNIST
Revised: 6. P OLY MNIST AND T RANSLATED -P OLY MNIST

Original: Figure 2 shows the results on PolyMNIST and Translated-PolyMNIST.
Revised: Figure 2 shows the results on PolyMNIST and Translated-PolyMNIST.

Original: Based on the MNIST-SVHNText results, we select MoPoE-VAE and MVTCAE as mixture and coordination-based models for comparison.
Revised: Based on the above results, we select MoPoE-VAE and MVTCAE as mixture-based and coordination-based models for comparison.

Original: First the results of the existing models on PolyMNIST show the same trend as on MNIST-SVHN-Text, i.e., there is a trade-off between cross-coherence and quality of generation.
Revised: First the results of the existing models on PolyMNIST show the same trend as on MNIST-SVHN-Text, i.e., there is a trade-off between cross-coherence and generation quality.

Original: CRMVAE mitigates these limitations and achieves the highest cross-coherence while maintaining high generation quality.
Revised: CRMVAE mitigates these limitations and achieves the highest cross-coherence while maintaining high generation quality.

Original: Note that it might seem strange that both MVTCAE and CRMVAE produce lower quality when the number of modalities increases.
Revised: Note that it might seem strange that both MVTCAE and CRMVAE produce lower quality when the number of modalities increases.

Original: This is thought to be due to a decrease in the diversity of the generated data distribution as the number of modalities increases.
Revised: This is thought to be due to a decrease in the diversity of the generated data distribution as the number of modalities increases.

Original: Translated-PolyMNIST is a slightly modified PolyMNIST, but Daunhawer et al.
Revised: Translated-PolyMNIST is a slightly modified PolyMNIST, but Daunhawer et al.

Original: (2021a) point out that the generation quality and cross-coherence performance of mixed-based models is significantly degraded on this dataset.
Revised: (2021a) point out that the generation quality and cross-coherence performance of mixed-based models is significantly degraded on this dataset.

Original: Our experiments also confirm that most existing models fail to crossgenerate across modalities.
Revised: Our experiments also confirm that most existing models fail to cross-generate across modalities.

Original: CRMVAE, on the other hand, clearly improves cross-coherence, although not as much as PolyMNIST.
Revised: CRMVAE, on the other hand, clearly improves cross-coherence, although not as much as PolyMNIST.

Original: Next, CRMVAEalso has the highest generation quality,especially significantly outperforming the mixture-based models.
Revised: Next, CRMVAE has the highest generation quality, significantly outperforming the mixture-based models.

Original: See Appendix C.1 for visualization of the generated images on PolyMNIST and Translated-PolyMNIST.
Revised: See appendix G.1 for visualization of the generated images on PolyMNIST and Translated-PolyMNIST.

Original: 4. CUB AND B IMODAL C ELEB A
Revised: 6. CUB AND B IMODAL C ELEB A

Original: Figures 3 and 4 show the quantitative and qualitative results for CUB and Bimodal CelebA, respectively.
Revised: Figures 3 and 4 show the quantitative and qualitative results for CUB and Bimodal CelebA.

Original: We examine the generation quality and cross-coherence performance as an ablation, when varying β .
Revised: We examine the generation quality and cross-coherence performance as an ablation, when varying β .

Original: First, the FID score indicates that MVAEs have high generation quality, while MoPoE-VAEs have significantly lower quality.
Revised: First, the FID score indicates that MVAEs have high generation quality, while MoPoE-VAEs have significantly lower quality.

Original: In most cases, the FID score does not change much or worsen when β is increased, while the FID score for MoPoE-VAE improves slightly.
Revised: In most cases, the FID score does not change much or worsen when β is increased, while the FID score for MoPoE-VAE improves slightly.

Original: This is thought to be because increasing β in MoPoE-VAE corresponds to making the distribution closer to the standard Gaussian, which diminishes the effect of generation from sub-sampling.
Revised: This is thought to be because increasing β in MoPoE-VAE corresponds to making the distribution closer to the standard Gaussian, which diminishes the effect of generation from sub-sampling.

Original: Furthermore, the performance of the coordination-based methods, including CRMVAE, is close to that of MVAE, indicating that they do not have a generation quality issue on these challenging datasets.
Revised: Furthermore, the performance of the coordination-based methods, including CRMVAE, is close to that of MVAE, indicating that they do not have a generation quality issue on these challenging datasets.

Original: Next,  evaluation of cosine similarity shows that MVAE and MoPoE-VAE do not change that much when β is increased, while the coordination-based models including CRMVAE improve when β is increased.
Revised: Next, the evaluation of cosine similarity shows that MVAE and MoPoE-VAE do not change that much when β is increased, while the coordination-based models, including CRMVAE, improve when β is increased.

Original: This is because in the coordination-based models, the term adjusted by β corresponds to the term that brings the inference of all modalities closer to that of each modality.
Revised: This is because, in the coordination-based models, the term adjusted by β corresponds to the term that brings the inference of all modalities closer to that of each modality.

Original: Compared to MVTCAE, CRMVAE performs better for Bimodal CelebA for all values of β and for CUB when β is large.
Revised: Compared to MVTCAE, CRMVAE performs better for Bimodal CelebA for all values of β and for CUB when β is large.

Original: Furthermore, the cross-coherence with pre-trained classifiers in Bimodal CelebA also shows that CRMVAE performs better than MVTCAE.
Revised: Furthermore, the cross-coherence with pre-trained classifiers in Bimodal CelebA also shows that CRMVAE performs better than MVTCAE.

Original: Note that MoPoE-VAE has the highest cross-coherence value, but as the qualitative results show, this only improves classification accuracy at the expense of image quality, e.g., diversity (as is the case for MNIST in MNIST-SVHN-Text).
Revised: Note that MoPoE-VAE has the highest cross-coherence value, but as the qualitative results show, this only improves classification accuracy at the expense of image quality, e.g., diversity (as is the case for MNIST in MNIST-SVHN-Text).

Original: Qualitative results showed the following limitations: MVAE failed to produce images corresponding to the text, and MoPoE-VAE produced only average or blurred images.
Revised: Qualitative results showed the following limitations: MVAE failed to produce images corresponding to the text, and MoPoE-VAE produced only average or blurred images.

Original: Comparing MVTCAE and
Revised: Comparing MVTCAE and

Original: CRMVAE, CRMVAE produces more appropriately conditioned images.
Revised: CRMVAE, CRMVAE produces more appropriately conditioned images.

Original: For example, in Bimodal
Revised: For example, in Bimodal

Original: CelebA, Text 3 contains a male attribute and CRMVAE produces a corresponding face image, while
Revised: CelebA, Text 3 contains a male attribute and CRMVAE produces a corresponding face image, while

Original: MVTCAE produces some female face images.
Revised: MVTCAE produces some female face images.

Original:  C ONCLUSION AND DISCUSSION
Revised:  C ONCLUSION AND DISCUSSION

Original: We focused on coordination-based multimodal VAEs to avoid the limitations of multimodal VAEs.
Revised: We focused on coordination-based multimodal VAEs to avoid the limitations of multimodal VAEs.

Original: We first discuss the new relation among coordination-based multimodalVAEs as multimodal VAEs, point out the issue of unimodal inference not learning from reconstruction in them, and then introduce a new objective involving unimodal reconstruction.
Revised: We first discuss the new relation among coordination-based multimodal VAEs, point out the issue of unimodal inference not learning from reconstruction in them, and then introduce a new objective involving unimodal reconstruction.

Original: Experiments show that the proposed model mitigatesthe limitations of existing models (both mixture-based and coordination-based) and shows overall superior performance on diverse datasets forboth generation quality and cross-coherence.
Revised: Experiments show that the proposed model mitigates existing models’ limitations (both mixture-based and coordination-based) and shows overall superior performance on diverse datasets for generation quality and cross-coherence.

Original: One limitation of CRMVAE is that there still remains a trade-off between generation quality and cross-coherence.
Revised: One limitation of CRMVAE is that there still remains a trade-off between generation quality and cross-coherence.

Original: For example, in the PolyMNIST experiment, CRMVAE performed best in crosscoherence, but the FID score was worse than MVTCAE.
Revised: For example, in the PolyMNIST experiment, CRMVAE performed best in crosscoherence, but the FID score was worse than MVTCAE.

Original: This is also true for the other results, where the FID score is slightly worse than MVTCAE in exchange for the performance of cross-coherence. Also, although we achieved the performance improvement for our objective without changing the architecture, it is not obvious that the same improvement can be achieved for other architectures, such as those with modality-specific or hierarchical latent variables.
Revised: The performance of joint coherence was also confirmed to be limited.

Original: In addition, although we evaluated the performance on several diverse datasets in this study, these datasets are much smaller than the various multimodal information obtained from the real world.
Revised: In addition, although we used several diverse datasets to evaluate performance, these datasets are much smaller than the various multimodal information obtained from the real world.

Original: Therefore, it is necessary to validate our model using such large multimodal information.
Revised: Therefore, we should validate our model using such large multimodal information.

Revised: A D ATASET

Revised: MNIST-SVHN-Text (Sutter et al., 2020) is the MNIST-SVHN (Shi et al., 2019) with the addition of the text modality, which represents the English name of each number as text and has diversity.

Revised: Different modalities of the same example have the same digits.

Original: A D ETAILED DERIVATION OF P ROOF
Revised: PolyMNIST (Sutter et al., 2021) consists of a set of five different MNIST images with different backgrounds and handwriting styles, each of which is considered a different modality. Translated-PolyMNIST (Daunhawer et al., 2021a) is composed of the same backgrounds as PolyMNIST with the numbers downscaled 14 and randomly transformed into position in the images. This is intended to allow for more significant variation among modalities than PolyMNIST. Caltech Birds ( CUB )

Revised: Wah et al.

Revised: (2011) is a dataset with bird images and their captions.

Revised: We follow

Revised: Daunhawer et al.

Revised: (2021a), i.e., we treat as 64 × 64 real images without feature extraction.

Revised: Bimodal CelebA (Sutter et al., 2020) is a dataset of face images from the CelebA (Liu et al., 2015) with additional text describing the corresponding attribute labels and changing the starting index to give a diversity.

Revised: B D ETAILED DERIVATION OF P ROOF

Original: E p d ( X ) [ E q ϕ
Revised: E p d ( X ) [ E q ϕ

Original: ( z | X ) [log p θ ( X | z )] − D KL ( q ϕ ( z | X ) ||
Revised: ( z | X ) [log p θ ( X | z )] − D KL ( q ϕ ( z | X ) ||

Original: p f
Revised: p f

Original: ( z | X ))]] =
Revised: ( z | X ))]] =

Original: E p d ( X ) (cid:2)
Revised: E p d ( X ) (cid:2)

Original: E q ϕ ( z | X ) [log p θ ( X | z )] − E q ϕ
Revised: E q ϕ ( z | X ) [log p θ ( X | z )] − E q ϕ

Original: ( z | X ) [log q ϕ ( z | X ) − log p
Revised: ( z | X ) [log q ϕ ( z | X ) − log p

Original: f
Revised: f

Original: ( z | X )] (cid:3) =
Revised: ( z | X )] (cid:3) =

Original: E p d ( X ) (cid:34)
Revised: E p d ( X ) (cid:34)

Original: E q ϕ ( z | X ) [log p θ ( X | z )] − E q ϕ
Revised: E q ϕ ( z | X ) [log p θ ( X | z )] − E q ϕ

Original: ( z | X ) [log q ϕ ( z | X ) −
Revised: ( z | X ) [log q ϕ ( z | X ) −

Original: M (cid:88) m =
Revised: M (cid:88) m =

Original: π m log q ϕ
Revised: π m log q ϕ

Original: ( z | x m )] (cid:35) (cid:32) ∵
Revised: ( z | x m )] (cid:35) (cid:32) ∵

Original: p f
Revised: p f

Original: ( z | X ) =
Revised: ( z | X ) =

Original: M (cid:88) m =
Revised: M (cid:88) m =

Original: π m log q ϕ
Revised: π m log q ϕ

Original: ( z | x m ) (cid:33)
Revised: ( z | x m ) (cid:33)

Original: =
Revised: =

Original: E p d ( X ) (cid:34)
Revised: E p d ( X ) (cid:34)

Original: E q ϕ ( z | X ) [log p θ ( X | z )] − E q ϕ
Revised: E q ϕ ( z | X ) [log p θ ( X | z )] − E q ϕ

Original: ( z | X ) (cid:34) M (cid:88) m =
Revised: ( z | X ) (cid:34) M (cid:88) m =

Original: π m log q ϕ
Revised: π m log q ϕ

Original: ( z | X ) −
Revised: ( z | X ) −

Original: M (cid:88) m =
Revised: M (cid:88) m =

Original: π m log q ϕ
Revised: π m log q ϕ

Original: ( z | x m ) (cid:35)(cid:35) (cid:32) ∵ log q ϕ
Revised: ( z | x m ) (cid:35)(cid:35) (cid:32) ∵ log q ϕ

Original: ( z | X ) =
Revised: ( z | X ) =

Original: M (cid:88) m =
Revised: M (cid:88) m =

Original: π m log q ϕ
Revised: π m log q ϕ

Original: ( z | X ) and
Revised: ( z | X ) and

Original: M (cid:88) m =
Revised: M (cid:88) m =

Original: π m =(cid:33) =
Revised: π m =(cid:33) =

Original: E p d ( X ) (cid:34)
Revised: E p d ( X ) (cid:34)

Original: E q ϕ ( z | X ) [log p θ ( X | z )] −
Revised: E q ϕ ( z | X ) [log p θ ( X | z )] −

Original: M (cid:88) m =
Revised: M (cid:88) m =

Original: π m E q ϕ
Revised: π m E q ϕ

Original: ( z | X ) [log q ϕ ( z | X ) − log q ϕ ( z | x m )] (cid:35) =
Revised: ( z | X ) [log q ϕ ( z | X ) − log q ϕ ( z | x m )] (cid:35) =

Original: E p d ( X ) (cid:34)
Revised: E p d ( X ) (cid:34)

Original: E q ϕ ( z | X ) [log p θ ( X | z )] −
Revised: E q ϕ ( z | X ) [log p θ ( X | z )] −

Original: M (cid:88) m =
Revised: M (cid:88) m =

Original: π m D KL ( q ϕ ( z | X ) ||
Revised: π m D KL ( q ϕ ( z | X ) ||

Original: q ϕ
Revised: q ϕ

Original: ( z | x m ))
Revised: ( z | x m ))

Original: (cid:35) =
Revised: (cid:35) =

Original: L C ( θ, ϕ ) □
Revised: L C ( θ, ϕ ) □

Original: B D ETAILED DERIVATION OF E Q . (7)
Revised: C D ETAILED DERIVATION OF P ROOF

Original: E p d ( X ) (cid:2) E q ϕ ( z | X ) [log p θ ( X | z )] (cid:3) =
Revised: E p d ( X ) [ D KL ( q ϕ ( z | X ) ||

Original: E p d ( X ) (cid:20)2 E q ϕ
Revised: q ϕ

Original: ( z | X ) [log p θ ( X | z )]
Revised: ( z | x m ))]

Original: + 1 2 E q ϕ ( z | X ) [log p θ ( X | z )] (cid:21) = 
Revised: = (cid:90) p

Original: E p d ( X ) (cid:34)2 E q ϕ
Revised: d ( x m ) (cid:90) (cid:90) q ϕ

Original: ( z | X )[log p θ ( X  |  z )] + 1 2 E q ϕ
Revised: ( z | X ) p d ( X C/m | x m ) log q ϕ ( z | X ) q ϕ

Original: ( z | X ) (cid:34) M (cid:88) m = log p θ ( x m  |  z  )(cid:35)(cid:35) =
Revised: ( z | x m ) dX C/m d z d x m = (cid:90) p d ( x m ) (cid:90) (cid:90) q ϕ ( z | X ) p d ( X C/m | x m ) log q ϕ ( z | X )

Original: E p d ( X  )(cid:20)2 E q ϕ
Revised: p d ( X C/m | x m ) q ϕ

Original: ( z | X )[log p θ ( X  |  z )] (cid:21) + (cid:90) (cid:90)q ϕ ( z | X )
Revised: ( z | x m ) p d ( X C/m | x m ) dX C/m d z d x m = (cid:90)

Original: p d ( X ) M (cid:88) m 
Revised: p d ( x m ) (cid:90) (cid:90) q ϕ

Original: = log p θ (  x m| z ) d z dX =
Revised: ( z | X ) p d ( X C/m | x m ) log q ϕ ( z | X )

Original: E p d ( X  )(cid:20)2 E q ϕ ( z | X )[log p θ ( X  |  z )] (cid:21) + 
Revised: p d ( X C/m | x m ) q avgϕ ( z | x m ) p d ( X C/m | x m ) dX C/m d z d x m + (cid:90) p

Original: M (cid:88) m = (cid:90) (cid:90) q ϕ
Revised: d ( x m ) (cid:90) (cid:90) q ϕ

Original: ( z | X ) p d ( X  )  log p θ (  x m| z ) d z dX =
Revised: ( z | X ) p d ( X C/m | x m ) dX C/m log q avgϕ ( z | x m ) q ϕ

Original: E p d ( X ) (cid:20)2 E q ϕ
Revised: ( z | x m ) d z d x m = (cid:90) p d ( x m ) (cid:90) (cid:90) q ϕ

Original: ( z | X  ) [log p θ (  X  |  z )] (cid:21)
Revised: ( z , X C/m | x m ) log q ϕ ( z , X C/m | x m ) q avgϕ ( z | x m ) p d ( X [Equation] D D ETAILED DERIVATION OF E Q . (7)

Original: [Equation]
Revised: [Equation]

Revised: m

Original: [Equation]
Revised: [Equation]

Revised: (cid:90)

Original:  p 
Revised: [Equation] ( z | X ) p d ( [Equation] C

Revised: [Equation]

Revised: m | x m ) dX C

Original: [Equation]
Revised: [Equation]

Revised: m p d ( x m ) log p θ ( x m | z )

Revised: [Equation]

Revised: m

Revised: [Equation]

Revised: (cid:90)

Original:  [Table]
Revised: [Equation] [Table] [Equation] [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: |

Original: [Table]
Revised: [Table]

Revised: (cid:18)

Revised: [Table]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: ) (cid:19)

Revised: [Equation]

Revised: [Table]

Original: C R ESULTS ON MNIST-SVHN-T EXT WITH STANDARD DEVIATIONS
Revised: [Equation] (cid:3) [Equation] E R ESULTS ON MNIST-SVHN-T EXT WITH STANDARD DEVIATIONS

Original: Table 3 and Table 4 are Table 1 and Table 2 with their standard deviations of three trials explicitly added.
Revised: Table 3 and Table 4 are Table 1 and Table 2 with their standard deviations of three trials explicitly added.

Original: C. C ONDITIONAL GENERATION ON P OLY MNIST AND T RANSLATED -P OLY MNIST
Revised: The meaning of MVTCAE (rescale) is explained in appendix G. F P ERFORMANCE ON THE LATENT REPRESENTATION Table 5 shows that coordinated-based performs better as input modality increases. This is a feature of the coordinated-based method, which learns to approximate the joint posterior, so the amount of information to be inferred increases as the number of modalities increases.

Revised: On the other hand, the mixture-based method performs better given a single modality.

Revised: The proposed method improves unimodal inference the most among the coordinated-based methods.

Revised: The right panels of Figure 5 and Figure 6 show the inference performance of PolyMNIST and

Revised: Translated-PolyMNIST, respectively.

Revised: It can be seen that the proposed method performs well when the number of modalities is small, while other models perform better when the number of modalities increases.

Revised: There are various possible reasons for this, but the scaling of the reconstruction error term, which will be discussed in the appendix G, might be the cause.

Revised: G C ONSIDERATION OF THE COEFFICIENTS OF THE RECONSTRUCTION TERM

Revised: We chose π instead of 12 as the coefficient in Eq.

Revised: However, this makes the reconstruction error term smaller than the other models by a factor of π 2 .

Revised: To check this effect, we scaled the MVTCAE reconstruction error term by multiplying it by π 2 .

Revised: Since this effect is large when the modality is three or more, we experimented with MNIST-SVHN-Text, PolyMNIST, and Translated-PolyMNIST.

Revised: [Table]

Revised: [Table]

Revised: The results denoted as “MVTCAE (rescale)” are the MVTCAE results aligned in scale with the reconstruction error term in CRMVAE.

Revised: First, for MNIST-SVHN-Text, the performances of crosscoherence and FID are slightly better with scale, but CRMVAE is still better.

Revised: The joint coherence was comparable to that of CRMVAE.

Revised: However, the performance of the inferred representation is lower with scaling.

Revised: This is because the coordinated-based learns inference by reconstruction, and when this term is small in the overall expression, it fails to learn inference.

Revised: Next, PolyMNIST and

Revised: Translated-PolyMNIST improve cross-coherence performance, but as with MNIST-SVHN-Text, inference performance is poor.

Revised: Compared to MVTCAE with rescaling, the performance of CRMVAE is better, and this comparison, in which the reconstruction error terms are scaled to each other, might better evaluate the effect of unimodal reconstruction included in CRMVAE.

Revised: [Table]

Revised: [Figure]

Revised: C l a ss i f

Original: Figure 5 and Figure 6 visualize the conditionally generated images on the PolyMNIST and TranslatedPolyMNIST datasets.
Revised: i c a t i o n a cc u r a c y [Figure] [Figure] c a t i o n

Original: [Figure]
Revised: [Figure]

Revised: These results show that adjusting for scale does not change the superiority of CRMVAE.

Revised: In other words, the higher performance of CRMVAE is not due to scaling alone.

Revised: G.

Revised: C ONDITIONAL GENERATION ON P OLY MNIST AND T RANSLATED -P OLY MNIST

Revised: Figure 7 and Figure 8 visualize the conditionally generated images on the PolyMNIST and TranslatedPolyMNIST datasets.

Revised: M0,1,2,3 !

Original: M
Revised: M

Original: M
Revised: MVAE MoPoE-VAE MVTCAE CRMVAE

Original: M
Revised: M

Original: [Figure]
Revised: M M M M

Revised: MVAE

Revised: MoPoE-VAE

Revised: MVTCAE

Original: CRMVAE
Revised: CRMVAE

Revised: M1,2,3,4 !

Revised: M

Original: [Figure]
Revised: M0,2,3,4 ! M M0,1,2,4 ! M3 M0,1,3,4 !

Original: M
Revised: M

Original: [Figure]
Revised: MVAE

Original: MoPoE [Figure]
Revised: MoPoE-VAE MVTCAE

Original: CRMVAE
Revised: CRMVAE

Original: [Figure]
Revised: MVAE MoPoE-VAE MVTCAE

Original: CRMVAE
Revised: CRMVAE

Original: [Figure]
Revised: MVAE MoPoE-VAE MVTCAE

Original: CRMVAE
Revised: CRMVAE

Original: [Figure]
Revised: [Figure]