Original: M ITIGATING THE L IMITATIONS OF M ULTIMODAL VAE S WITH C OORDINATION -B ASED A PPROACH
Revised: M ITIGATING THE L IMITATIONS OF M ULTIMODAL VAE S WITH C OORDINATION -B ASED A PPROACH

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: One of the key challenges in multimodal variational autoencoders (VAEs) is inferring a joint representation from arbitrary subsets of modalities.
Revised: One of the key challenges in multimodal variational autoencoders (VAEs) is inferring a joint representation from arbitrary subsets of modalities.

Original: The state-of-the-art approach to achieving this is to sub-sample the modality subsets and learn to generate all modalities from them.
Revised: The state-of-the-art approach to achieving this is to sub-sample the modality subsets and learn to generate all modalities from them.

Original: However, this sub-sampling in the mixture-based approach has been shown to degrade other important features of multimodal VAEs, such as quality of generation, and furthermore, this degradation is theoretically unavoidable.
Revised: However, this sub-sampling in the mixture-based approach has been shown to degrade other important features of multimodal VAEs, such as quality of generation, and furthermore, this degradation is theoretically unavoidable.

Original: In this study, we focus on another approach to learning the joint representation by bringing unimodal inferences closer to joint inference from all modalities, which does not have the above limitation.
Revised: In this study, we focus on another approach to learning the joint representation by bringing unimodal inferences closer to joint inference from all modalities, which does not have the above limitation.

Original: Although there have been models that can be categorized under this approach, they were derived from different backgrounds; therefore, the relation and superiority between them were not clear.
Revised: Although there have been models that can be categorized under this approach, they were derived from different backgrounds; therefore, the relation and superiority between them were not clear.

Original: To take a unified view, we first categorize them as coordination-based multimodal VAEs and show that these can be derived from the same multimodal evidence lower bound (ELBO) and that the difference in their performance is related to whether they are more tightly lower bounded.
Revised: To take a unified view, we first categorize them as coordination-based multimodal VAEs and show that these can be derived from the same multimodal evidence lower bound (ELBO) and that the difference in their performance is related to whether they are more tightly lower bounded.

Original: Next, we point out that these existing coordination-based models perform poorly on cross-modal generation (or cross-coherence) because they do not learn to reconstruct modalities from unimodal inferences.
Revised: Next, we point out that these existing coordination-based models perform poorly on cross-modal generation (or cross-coherence) because they do not learn to reconstruct modalities from unimodal inferences.

Original: Therefore, we propose a novel coordination-based model that incorporates these unimodal reconstructions, which avoids the limitations of both mixture and coordination-based models.
Revised: Therefore, we propose a novel coordination-based model that incorporates these unimodal reconstructions, which avoids the limitations of both mixture and coordination-based models.

Original: Experiments with diverse and challenging datasets show that the proposed model mitigates the limitations in multimodal
Revised: Experiments with diverse and challenging datasets show that the proposed model mitigates the limitations in multimodal

Original: VAEs and performs well in both cross-coherence and generation quality.
Revised: VAEs and performs well in both cross-coherence and generation quality.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep generative models have recently shown high performance on multimodal data, including images and captions.
Revised: Deep generative models have recently shown high performance on multimodal data, including images and captions.

Original: In particular, multimodal learning with variational autoencoders (VAEs) (Kingma &
Revised: In particular, multimodal learning with variational autoencoders (VAEs) (Kingma &

Original: Welling, 2013; Rezende et al., 2014) has attracted much attention to obtain a joint representation from modalities (Wu & Goodman, 2018; Shi et al., 2019; Sutter et al., 2021).
Revised: Welling, 2013; Rezende et al., 2014) has attracted much attention to obtain a joint representation from modalities (Wu & Goodman, 2018; Shi et al., 2019; Sutter et al., 2021).

Original: Such representations can be used for predicting common concepts from these modalities or for generating other modalities.
Revised: Such representations can be used for predicting common concepts from these modalities or for generating other modalities.

Original: An essential challenge of multimodal VAEs is inferring a joint representation from subsets of modalities.
Revised: An essential challenge of multimodal VAEs is inferring a joint representation from subsets of modalities.

Original: State-of-the-art models attempt to accomplish this by training to generate all modalities from a joint representation inferred from missing (or sub-sampled) modalities.
Revised: State-of-the-art models attempt to accomplish this by training to generate all modalities from a joint representation inferred from missing (or sub-sampled) modalities.

Original: These are called mixture-based multimodal VAEs (Daunhawer et al., 2021a), and examples include MMVAE (Shi et al., 2019) and MoPoE-VAE (Sutter et al., 2021) 1 .
Revised: These are called mixture-based multimodal VAEs (Daunhawer et al., 2021a), and examples include MMVAE (Shi et al., 2019) and MoPoE-VAE (Sutter et al., 2021) 1 .

Original: However, it has been pointed out that the quality of modality generation is lower than that of unimodal VAEs, and furthermore, cross-generation between modalities, or cross-coherence, can be degraded.
Revised: However, it has been pointed out that the quality of modality generation is lower than that of unimodal VAEs, and furthermore, cross-generation between modalities, or cross-coherence, can be degraded.

Original: This is an inherent limitation on mixturebased VAEs due to modality sub-sampling during training, and theoretical and empirical evidence shows that this limitation cannot be avoided (Daunhawer et al., 2021a).
Revised: This is an inherent limitation on mixturebased VAEs due to modality sub-sampling during training, and theoretical and empirical evidence shows that this limitation cannot be avoided (Daunhawer et al., 2021a).

Original: To alleviate this issue, we focus on another approach in multimodal VAEs that brings the representation inferred from each modality closer to that inferred from all modalities.
Revised: To alleviate this issue, we focus on another approach in multimodal VAEs that brings the representation inferred from each modality closer to that inferred from all modalities.

Original: This approach avoids the limitation of degraded generation quality because its objective does not include generation from sub-sampled modalities.
Revised: This approach avoids the limitation of degraded generation quality because its objective does not include generation from sub-sampled modalities.

Original: Models that fit this approach have been proposed, but their objectives are derived from different backgrounds.
Revised: Models that fit this approach have been proposed, but their objectives are derived from different backgrounds.

Original: For example, MVTCAE (Hwang et al., 2021), which can be regarded as one of them, is derived its objective from the total correlation; therefore, its relationship to the multimodal VAEs’ objective, multimodal evidence lower bound (ELBO), is not clear.
Revised: For example, MVTCAE (Hwang et al., 2021), which can be regarded as one of them, is derived its objective from the total correlation; therefore, its relationship to the multimodal VAEs’ objective, multimodal evidence lower bound (ELBO), is not clear.

Original: Moreover, another model of this approach, MMJSD (Sutter et al., 2020), has been shown experimentally to perform worse than MVTCAE (Hwang et al., 2021), but the theoretical reason for this is unclear.
Revised: Moreover, another model of this approach, MMJSD (Sutter et al., 2020), has been shown experimentally to perform worse than MVTCAE (Hwang et al., 2021), but the theoretical reason for this is unclear.

Original: We first categorize them as coordination-based multimodal VAEs 2 and show that these models can be viewed as unified, i.e., all are derived from multimodal ELBO, and that the differences in models correspond to whether they are more tightly lower bounded.
Revised: We first categorize them as coordination-based multimodal VAEs 2 and show that these models can be viewed as unified, i.e., all are derived from multimodal ELBO, and that the differences in models correspond to whether they are more tightly lower bounded.

Original: We also prove that MVTCAE encourages unimodal posterior to approach the average of joint posterior, which might be one of the reasons for learning good inference.
Revised: We also prove that MVTCAE encourages unimodal posterior to approach the average of joint posterior, which might be one of the reasons for learning good inference.

Original: Next, we point out that the existing coordination-based VAEs still have a shortcoming in cross-coherence since no generation from unimodal inferences is included in the training.
Revised: Next, we point out that the existing coordination-based VAEs still have a shortcoming in cross-coherence since no generation from unimodal inferences is included in the training.

Original: We then propose a novel objective that introduces unimodal reconstruction terms.
Revised: We then propose a novel objective that introduces unimodal reconstruction terms.

Original: Since this objective does not include cross-generation, it also avoids the generation degradation issue of the mixture-based models while mitigating the issue of coordination-based models.
Revised: Since this objective does not include cross-generation, it also avoids the generation degradation issue of the mixture-based models while mitigating the issue of coordination-based models.

Original: Theoretical consideration also shows that this objective mitigates a negative effect in the inference of representations.

Original: Note that our model alleviates the issues by only changing the objective, unlike other methods for improving multimodal VAEs by changing the architecture, such as introducing additional latent variables (Tsai et al., 2018; Hsu & Glass, 2018; Sutter et al., 2020; Palumbo et al., 2022).
Revised: Note that our model alleviates the issues by only changing the objective, unlike other methods for improving multimodal VAEs by changing the architecture, such as introducing additional latent variables (Tsai et al., 2018; Hsu & Glass, 2018; Sutter et al., 2020; Palumbo et al., 2022).

Original: We conducted experiments on five diverse multimodal datasets, including challenging ones, e.g., those that cannot be adequately trained with existing models or require additional architecture for adequate training.
Revised: We conducted experiments on five diverse multimodal datasets, including challenging ones, e.g., those that cannot be adequately trained with existing models or require additional architecture for adequate training.

Original: We confirmed that the proposed method outperforms existing mixture- and coordination -based models in terms of cross-coherence and generation quality
Revised: We confirmed that the proposed method outperforms existing mixture- and coordination-based models in terms of cross-coherence and generation quality.

Original:  M ULTIMODAL VAE S
Revised:  M ULTIMODAL VAE S

Original: Suppose that we have a set of multimodal examples, where each example is a set of M modalities
Revised: Suppose that we have a set of multimodal examples, where each example is a set of M modalities

Original: X = { x m } Mm =1 .
Revised: X = { x m } Mm =1 .

Original: We assume that these examples are derived from data distribution p d ( X ) and that each example X has a corresponding common latent concept z , i.e., a joint representation.
Revised: We assume that these examples are derived from data distribution p d ( X ) and that each example X has a corresponding common latent concept z , i.e., a joint representation.

Original: Given a training set { X ( i ) } Ni =1 , our goals are to infer a joint representation from the subset of modalities X S ⊆
Revised: Given a training set { X ( i ) } Ni =1 , our goals are to infer a joint representation from the subset of modalities X S ⊆

Original: X 3 and to generate another subset
Revised: X 3 and to generate another subset

Original: X S ′ ⊆ X via that representation.
Revised: X S ′ ⊆ X via that representation.

Original: In other words, we aim to obtain an inference p ( z | X ) and a joint distribution p ( X, z ) =
Revised: In other words, we aim to obtain an inference p ( z | X ) and a joint distribution p ( X, z ) =

Original: (cid:81)
Revised: (cid:81)

Original: x m ∈
Revised: x m ∈

Original: X p ( x m | z ) p ( z ) .
Revised: X p ( x m | z ) p ( z ) .

Original: To achieve this, it is a natural choice to use VAEs (Kingma & Welling, 2013), deep generative models that can learn inference to acquire representations from data in addition to the generation.
Revised: To achieve this, it is a natural choice to use VAEs (Kingma & Welling, 2013), deep generative models that can learn inference to acquire representations from data in addition to the generation.

Original: The generative model p θ ( X | z ) = (cid:81) m : x m ∈
Revised: The generative model p θ ( X | z ) = (cid:81) m : x m ∈

Original: X p θ ( x m | z ) is parameterized by deep neural networks, and the prior of the latent variable is set as a standard Gaussian p ( z ) =
Revised: X p θ ( x m | z ) is parameterized by deep neural networks, and the prior of the latent variable is set as a standard Gaussian p ( z ) =

Original: N ( 0 , I ) .
Revised: N ( 0 , I ) .

Original: The objective of VAEs is to maximize the expected log-likelihood over data distribution Since it is tractable to optimize this likelihood directly, we introduce an approximated posterior q ϕ ( z | X ) and instead optimize the following evidence lower bound (ELBO) on the expected log-likelihood:
Revised: The objective of VAEs is to maximize the expected log-likelihood over data distribution Since it is tractable to optimize this likelihood directly, we introduce an approximated posterior q ϕ ( z | X ) and instead optimize the following evidence lower bound (ELBO) on the expected log-likelihood:

Original: [Equation]
Revised: [Equation]

Original: In this paper, the model group that maximizes this multimodal ELBO is collectively called multimodal
Revised: In this paper, the model group that maximizes this multimodal ELBO is collectively called multimodal

Original: VAEs .
Revised: VAEs .

Original: We aim to optimize Eq.
Revised: We aim to optimize Eq.

Original: 1 to obtain a joint representation and generate modalities from it given a modality subset X S .
Revised: 1 to obtain a joint representation and generate modalities from it given a modality subset X S .

Original: However, Eq. 1 only includes the inference given all modalities; therefore, it does not learn to infer representations from subsets.
Revised: However, Eq. 1 only includes the inference given all modalities; therefore, it does not learn to infer representations from subsets.

Original: 2. M IXTURE - BASED MULTIMODAL VAE S
Revised: 2. M IXTURE - BASED MULTIMODAL VAE S

Original: An inference from subsets is commonly expressed as a combination of unimodal posteriors from each modality.
Revised: An inference from subsets is commonly expressed as a combination of unimodal posteri- ors from each modality.

Original: Product of experts (PoE) (Hinton, 2002; Wu & Goodman, 2018)q PoEϕ ( z | X S ) ≡ p ( z ) (cid:81) m : x m ∈ X S q ϕ
Revised: Product of experts (PoE) (Hinton, 2002; Wu & Goodman, 2018)

Original: ( z | x m ) and mixture of experts (MoE) q MoEϕ ( z | X S ) ≡ (cid:80) m

Original: [Equation]
Revised: [Equation]

Original: ϕ
Revised: m : x

Original: [Equation] z | x m ) are known as ways to aggregate inferences.
Revised: m ∈ X S q ϕ m ( z | x m ) are known as ways to aggregate inferences.

Original: Moreover, the following mixture of products of experts (MoPoE) (Sutter et al., 2021) generalizes them:
Revised: Moreover, the following mixture of products of experts (MoPoE) (Sutter et al., 2021) generalizes them:

Original: [Equation]
Revised: [Equation]

Original: where (cid:80) S : X S ∈P
Revised: where (cid:80) S : X S ∈P

Original: ( X ) ω S = 1 , ω S ∈ [0 , 1] , and P ( X ) is a powerset of X .
Revised: ( X ) ω S = 1 , ω S ∈ [0 , 1] , and P ( X ) is a powerset of X .

Original: This MoPoE aggregation is equivalent to PoE when X S =
Revised: This MoPoE aggregation is equivalent to PoE when X S =

Original: X and MoE when | X S
Revised: X and MoE when | X S

Original: | = 1 for all X S ∈ P ( X ) .
Revised: | = 1 for all X S ∈ P ( X ) .

Original: Substituting the MoPoE-aggregated joint posterior (Eq. 2) into Eq.
Revised: Substituting the MoPoE-aggregated joint posterior (Eq. 2) into Eq.

Original: 1, we obtain the following:
Revised: 1, we obtain the following:

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: ≥ [Equation] d
Revised: θ,

Original: [Equation]
Revised: [Equation]

Original: Multimodal VAEs with Eq.
Revised: Multimodal VAEs with Eq.

Original: 3 are called mixture-based multimodal VAEs (Daunhawer et al., 2021a).
Revised: 3 are called mixture-based multimodal VAEs (Daunhawer et al., 2021a).

Original: MMVAE (Shi et al., 2019) and MoPoE-VAE (Sutter et al., 2021) are members of this family.
Revised: MMVAE (Shi et al., 2019) and MoPoE-VAE (Sutter et al., 2021) are members of this family.

Original: Comparing with Eq.
Revised: Comparing with Eq.

Original: 1, we can see that the reconstruction term (the first term in the expectation) of
Revised: 1, we can see that the reconstruction term (the first term in the expectation) of

Original: Eq. 1 is designed to generate all modalities from the sub-sampled modality subset.
Revised: Eq. 1 is designed to generate all modalities from the sub-sampled modality subset.

Original: In other words, mixture-based multimodal VAEs encourage cross-generation during training, aiming at inferring a consistent joint representation from subset modalities and generating modalities from it.
Revised: In other words, mixture-based multimodal VAEs encourage cross-generation during training, aiming at inferring a consistent joint representation from subset modalities and generating modalities from it.

Original: However, mixture-based VAEs have a severe issue in generating modalities, which is one of the essential goals of multimodal VAEs.
Revised: However, mixture-based VAEs have a severe issue in generating modalities, which is one of the essential goals of multimodal VAEs.

Original: Daunhawer et al.
Revised: Daunhawer et al.

Original: (2021a) show that the following inequality holds between Eq.
Revised: (2021a) show that the following inequality holds between Eq.

Original: 3 and the expected marginal log-likelihood:
Revised: 3 and the expected marginal log-likelihood:

Original: [Equation]
Revised: [Equation]

Original: where ∆( X ) ≡ (cid:80) S : X S ∈P
Revised: where ∆( X ) ≡ [Equation] S : X S ∈P

Original: ( X )
Revised: ( X )

Original: w S H
Revised: w S H

Original: ( X { 1 ,...,M }\ S | X S ) 4 is the discrepancy independent of the learning parameters.
Revised: ( X { 1 ,...,M }\ S | X S ) 5 is the discrepancy independent of the learning parameters.

Original: This inequality means that multimodal ELBO of mixture-based VAEs cannot be tighter than this discrepancy ∆( X ) no matter how much it is maximized.
Revised: This inequality means that multimodal ELBO of mixture-based VAEs cannot be tighter than this discrepancy ∆( X ) no matter how much it is maximized.

Original: Therefore, the quality of their generating modalities always degrades compared to unimodal VAEs.
Revised: Therefore, the quality of their generating modalities always degrades compared to unimodal VAEs.

Original: Moreover, this discrepancy, by definition, depends on how much different information one modality subset X { 1 ,...,M }\ S has compared to the rest of them X S .
Revised: Moreover, this discrepancy, by definition, depends on how much different information one modality subset X { 1 ,...,M }\ S has compared to the rest of them X S .

Original: Therefore, the more information specific to each modality, the larger this quantity and the worse the generation quality.
Revised: Therefore, the more information specific to each modality, the larger this quantity and the worse the generation quality.

Original: This also leads to degrading cross-coherence (Shi et al., 2019), or the ability to generate the same semantic meaning across modalities conditionally.
Revised: This also leads to degrading cross-coherence (Shi et al., 2019), or the ability to generate the same semantic meaning across modalities conditionally.

Original: Intuitively, mixture-based VAEs are required to take sub-sampled subsets as inputs to generate the rest, but these inputs have insufficient information for that, resulting in degraded cross-generation.
Revised: Intuitively, mixture-based VAEs are required to take sub-sampled subsets as inputs to generate the rest, but these inputs have insufficient information for that, resulting in degraded cross-generation.

Original: 2. A VOIDING THE LIMITATION OF MULTIMODAL VAE S
Revised: 2. A VOIDING THE LIMITATION OF MULTIMODAL VAE S

Original: What brings about this irreducible discrepancy ∆( X ) ?
Revised: What brings about this irreducible discrepancy ∆( X ) ?

Original: According to the proof by Daunhawer et al.
Revised: According to the proof by Daunhawer et al.

Original: (2021a)
Revised: (2021a)

Original: (see B.4 in Daunhawer et al.
Revised: (see B.4 in Daunhawer et al.

Original: (2021a))
Revised: (2021a))

Original: , it is due to the inclusion in the objective of a cross-generation term, i.e., a term that generates unobserved modalities from observed modalities.
Revised: , it is due to the inclusion in the objective of a cross-generation term, i.e., a term that generates unobserved modalities from observed modalities.

Original: Therefore, one promising way to avoid this limitation is to not include in the objective of the multimodal VAE the cross-generation term 5 .
Revised: Therefore, one promising way to avoid this limitation is to not include in the objective of the multimodal VAE the cross-generation term 6 .

Original: Note that MVAE (Wu & Goodman, 2018), which first uses PoE for the joint posterior, uses the sum of ELBOs over all modality subsets as the objective, but does not include cross-generation and only includes reconstruction of subsets 6 , thus enabling high-quality generation.
Revised: Note that MVAE (Wu & Goodman, 2018), which first uses PoE for the joint posterior, uses the sum of ELBOs over all modality subsets as the objective, but does not include cross-generation and only includes reconstruction of subsets 7 , thus enabling high-quality generation.

Original: However, MVAE cannot perform cross-generation between different modalities because its objective does not include terms to encourage it.
Revised: However, MVAE cannot perform cross-generation between different modalities because its objective does not include terms to encourage it.

Original: Despite this significant drawback, determining how to work around the limitation of mixture-based models leads to our proposed model described later.
Revised: Despite this significant drawback, determining how to work around the limitation of mixture-based models leads to our proposed model described later.

Original:  C OORDINATION - BASED MULTIMODAL VAE S
Revised:  C OORDINATION - BASED MULTIMODAL VAE S

Original: MVTCAE (Hwang et al., 2021) and MMJSD (Sutter et al., 2020) have been proposed to learn to infer joint representation from subset modalities by bringing joint and unimodal posteriors closer together.
Revised: MVTCAE (Hwang et al., 2021) and MMJSD (Sutter et al., 2020) have been proposed to learn to infer joint representation from subset modalities by bringing joint and unimodal posteriors closer together.

Original: We find that they omit sub-sampling in the objective unless the posterior is a mixture distribution and thus can learn the joint representation without being affected by the limitation of the mixture-based
Revised: We find that they omit sub-sampling in the objective unless the posterior is a mixture distribution and thus can learn the joint representation without being affected by the limitation of the mixture-based

Original: VAEs .
Revised: VAEs .

Original: We first categorize them as a unified coordination-based multimodal VAEs 7 .
Revised: We first categorize them as a unified coordination-based multimodal VAEs 8 .

Original: 3. MVTCAE AND MMJSD
Revised: 3. MVTCAE AND MMJSD

Original: The objectives of MVTCAE and MMJSD are expressed as follows:
Revised: The objectives of MVTCAE and MMJSD are expressed as follows:

Original: [Equation]
Revised: [Equation]

Original: where (cid:80) m π m = 1 , π m ∈ [0 , 1] , and function f M defines the mixture of { q ϕ( z | x m )
Revised: where (cid:80) m π m = 1 , π m ∈ [0 , 1] , and function f M defines the mixture of { q ϕ

Original:  } Mm =0 .
Revised: m ( z | x m ) } Mm =0 .

Original: For simplicity of notation, we use q ϕ 0 ( z | x 0 ) ≡ p ( z ) .
Revised: For simplicity of notation, we use q ϕ 0 ( z | x 0 ) ≡ p ( z ) .

Original: We denote the entire objective of MVTCAE and
Revised: We denote the entire objective of MVTCAE and

Original: MMJSD as L C ( θ, ϕ ) and L C ′ ( θ, ϕ ) , respectively.
Revised: MMJSD as L C ( θ, ϕ ) and L C ′ ( θ, ϕ ) , respectively.

Original: MVTCAE is derived as a lower bound for the total correlation 8 .
Revised: MVTCAE is derived as a lower bound for the total correlation 9 .

Original: When M = 2 , this model is equal to
Revised: When M = 2 , this model is equal to

Original: JMVAE (Suzuki et al., 2016).
Revised: JMVAE (Suzuki et al., 2016).

Original: JMVAE uses a network for joint posterior separately from unimodal posteriors, while MVTCAE represents joint posteriors by PoE-aggregating unimodal posteriors.
Revised: JMVAE uses a network for joint posterior separately from unimodal posteriors, while MVTCAE represents joint posteriors by PoE-aggregating unimodal posteriors.

Original: MMJSD is derived by replacing the KL divergence of the multimodal ELBO with the Jensen-Shannon divergence.
Revised: MMJSD is derived by replacing the KL divergence of the multimodal ELBO with the Jensen-Shannon divergence.

Original: It is proved (in Lemma 2 in Sutter et al.
Revised: It is proved (in Lemma 2 in Sutter et al.

Original: that L C ′ ( θ, ϕ ) is a lower bound for multimodal ELBO with f M ( z | X ) replaced by the dynamic prior: p f ( z | X ) =
Revised: that L C ′ ( θ, ϕ ) is a lower bound for multimodal ELBO with f M ( z | X ) replaced by the dynamic prior: p f ( z | X ) =

Original: f ( { q ϕ  ( z | x m ) } Mm =0 ) , where f is a function that encompasses each unimodal posterior, and Sutter et al.
Revised: f ( { q ϕ m ( z | x m ) } Mm =0 ) , where f is a function that encompasses each unimodal posterior, and Sutter et al.

Original: (2020) choose the weighted PoE. This replacement can be thought of as introducing the prior knowledge that all modalities have the same joint representation (Sutter et al., 2020).
Revised: (2020) choose the weighted PoE. This replacement can be thought of as introducing the prior knowledge that all modalities have the same joint representation (Sutter et al., 2020).

Original: Note that in Sutter et al.
Revised: Note that in Sutter et al.

Original: (2020), the posterior of MMJSD is assumed to be MoE, which is affected by the limitation of mixture-based VAEs.
Revised: (2020), the posterior of MMJSD is assumed to be MoE, which is affected by the limitation of mixture-based VAEs.

Original: Since we would like to focus on the properties of MMJSD contributing to its coordination-based nature, we call MMJSD-PoE the model that replaces the posterior in Eq.
Revised: Since we would like to focus on the properties of MMJSD contributing to its coordination-based nature, we call MMJSD-PoE the model that replaces the posterior in Eq.

Original: 6 with PoE. Compared to the objective of MVTCAE, the direction of KL divergence in Eq.
Revised: 6 with PoE. Compared to the objective of MVTCAE, the direction of KL divergence in Eq.

Original: 6 is reversed: MVTCAE uses forward KL divergence, while MMJSD(-PoE) uses reverse KL divergence.
Revised: 6 is reversed: MVTCAE uses forward KL divergence, while MMJSD(-PoE) uses reverse KL divergence.

Original: Hwang et al.
Revised: Hwang et al.

Original: (2021) shows that MVTCAE performs better than MMJSD by experiment.
Revised: (2021) show that MVTCAE performs better than MMJSD by experiment.

Original: 3. D ISCOVERING THE RELATION BETWEEN COORDINATION - BASED MULTIMODAL VAE S
Revised: 3. T HE RELATION BETWEEN COORDINATION - BASED MULTIMODAL VAE S

Original: Because MVTCAE and MMJSD were proposed for different purposes, their relationship is unclear.
Revised: Because MVTCAE and MMJSD were proposed for different purposes, their relationship is unclear.

Original: In particular, since MVTCAE was derived from the total correlation, it is unclear whether it belongs to multimodal VAEs (in the sense of valid ELBOs).
Revised: In particular, since MVTCAE was derived from the total correlation, it is unclear whether it belongs to multimodal VAEs (i.e., valid ELBOs).

Original: In light of the above, we first show thatthe objective of MVTCAE is multimodal ELBO.
Revised: We first show that MVTCAE is multimodal ELBO.

Original: Lemma 

Revised: Lemma

Original: The objective of MVTCAE L C ( θ, ϕ ) is equal to the multimodal ELBO in Eq.
Revised: The objective of MVTCAE L C ( θ, ϕ ) is equal to the multimodal ELBO in Eq.

Original: 1 with a weighted PoE-based dynamic prior as its prior.
Revised: 1 with a weighted PoE-based dynamic prior as its prior.

Original: Proof 1 Following MMJSD, we introduce a weighted PoE-based dynamic prior for Eq. 3 as follows:
Revised: Proof 1 Following MMJSD, we introduce a weighted PoE-based dynamic prior for L C ( θ, ϕ ) :

Original: [Equation] p d ( X ) 
Revised: E p d ( X ) [ E q ϕ

Original: [Equation]
Revised: ( z | X ) [log p θ ( X | z )] − D KL ( q ϕ ( z | X ) || p f

Original: (cid:35)(cid:35) = L C ( θ
Revised: ( z | X ))]] = E p d ( X ) (cid:34)

Original: [Equation]
Revised: [Equation]

Original: ϕ ) . □ Therefore, MVTCAE can be considered a member of multimodal VAEs.
Revised: Therefore, MVTCAE can be considered a member of multimodal VAEs.

Original: This indicates that the objective of JMVAE(Suzuki et al., 2016) is also multimodal ELBO.
Revised: This indicates that the objective of JMVAE is also multimodal ELBO.

Original: Note that no assumption is made about the approximate posterior in Lemma 1, and thus holds whether it is aggregated by PoE or MoE. Next, in special cases, the objective of MVTCAE is an upper bound on the objective of MMJSD.
Revised: Note that no assumption is made about the approximate posterior in Lemma 1, and thus holds whether it is aggregated by PoE or MoE. Next, in special cases, the objective of MVTCAE is an upper bound on the objective of MMJSD.

Original: Lemma 

Revised: Lemma

Original: If the posterior is MoE-aggregated, the objective of MVTCAE L C ( θ, ϕ ) is an upper bound on that of MMJSD L C ′ ( θ, ϕ ) , i.e., the following inequality holds: L C ( θ, ϕ ) ≥ L C ′ ( θ, ϕ ) .
Revised: If the posterior is MoE-aggregated, the objective of MVTCAE L C ( θ, ϕ ) is an upper bound on that of MMJSD L C ′ ( θ, ϕ ) , i.e., the following inequality holds: L C ( θ, ϕ ) ≥ L C ′ ( θ, ϕ ) .

Original: Proof 2 Lemma 1 shows that the objective of MVTCAE is equal to a multimodal ELBO with a dynamic prior.
Revised: Proof 2 Lemma 1 shows that the objective of MVTCAE is equal to a multimodal ELBO with a dynamic prior.

Original: In addition, Lemma 2 in Sutter et al.
Revised: In addition, Lemma 2 in Sutter et al.

Original: (2020) states that the objective of MMJSD is a lower bound on multimodal ELBO with a dynamic prior and MoE-aggregated posterior.
Revised: (2020) states that the objective of MMJSD is a lower bound on multimodal ELBO with a dynamic prior and MoE-aggregated posterior.

Original: Therefore, the objective of MMJSD is a lower bound on that of MVTCAE if the posterior is MoE-aggregated.
Revised: Therefore, the objective of MMJSD is a lower bound on that of MVTCAE if the posterior is MoE-aggregated.

Original: □
Revised: □

Original: Lemma 2 indicates that MVTCAE better approximates the true log-likelihood and thus promotes the good generation, which provides one possible explanation as to why MVTCAE performs better than
Revised: Lemma 2 indicates that MVTCAE better approximates the true log-likelihood and thus promotes the good generation, which provides one possible explanation as to why MVTCAE performs better than

Original: MMJSD.
Revised: MMJSD.

Original: Note that Lemma 2 comes from the fact that the joint posterior of the KL divergence in
Revised: Note that Lemma 2 comes from the fact that the joint posterior of the KL divergence in

Original: MMJSD assumes MoE; therefore, it is still unclear whether this holds when the posterior is PoE.
Revised: MMJSD assumes MoE; therefore, it is still unclear whether this holds when the posterior is PoE.

Original: Furthermore, the following proof shows why this direction of KL divergence for MVTCAE is good for multimodal VAEs from a different perspective.
Revised: Furthermore, we show why this direction of KL divergence for MVTCAE is good for multimodal VAEs from a different perspective.

Original: Lemma 

Revised: Lemma

Original: The KL divergence for each m (Eq. 5) brings the average of the joint approximate posterior q avgϕ ( z | x m )=
Revised: The KL divergence for each m (Eq. 5) brings the average of the joint approximate posterior q ϕ ( z | x m )

Original:  (cid:82) q ϕ
Revised: = (cid:82) q ϕ

Original: ( z | X ) p d ( X C/m | x m ) dX C/m closer to the unimodal posterior q ϕ  ( z | x m ) .
Revised: ( z | X ) p d ( X C/m | x m ) dX C/m closer to the unimodal posterior q ϕ m ( z | x m ) .

Original: Proof 3 The KL divergence for x m (Eq. 5) is transformed as follows:
Revised: Proof 3 Eq. 5 is transformed as follows 10 :

Original: [Equation]
Revised: [Equation]

Original: where I ϕ ( X C/m ; z | x m ) is a conditional mutual information and holds greater than 0.
Revised: where I ϕ ( X C/m ; z | x m ) is a conditional mutual information and holds greater than 0.

Original: Therefore, maximizing the negative KL term (Eq. 5) leads to minimizing the second term in Eq.
Revised: Therefore, maximizing the negative KL term (Eq. 5) leads to minimizing the second term in Eq.

Original: □
Revised: □

Original: Lemma 3 holds for any approximate posterior.
Revised: Lemma 3 holds for any approximate posterior.

Original: Therefore, another possible reason MVTCAE is good as multimodal VAEs is that maximizing Eq.
Revised: Therefore, another possible reason MVTCAE is good as multimodal VAEs is that maximizing Eq.

Original: 5 explicitly brings the unimodal posterior closer to the joint posterior with the unimodal input.
Revised: 5 explicitly brings the unimodal posterior closer to the joint posterior with the unimodal input.

Original: C OORDINATED AND UNIMODAL R ECONSTRUCTED M ULTIMODAL VAE
Revised: If the joint inference is PoE-aggregation with Gaussian, the approximate posterior of any input can be computed analytically.

Original: If the joint inference is properly learned in the first term of L C ( θ, ϕ  ) , then the unimodal inference can also be properly performed, as shown in Lemma 3.
Revised: Therefore, the second term in Eq. 7 can be set to zero by using PoE-aggregate posterior q ϕ ( z | x m ) = p ( z ) q ϕ m ( z | x m ) instead of unimodal posterior q ϕ m ( z | x m ) .

Original: Furthermore, when we choose the PoE-aggregated joint inference, the PoE property allows the inference from  arbitrary modalities to be adequately represented as a product of the unimodal posterior, provided that it is adequately approximated.
Revised: From now on, we use the PoE-aggregate posterior.  C OORDINATED AND UNIMODAL R ECONSTRUCTED M ULTIMODAL VAE If the inference can be properly learned, the inference from an arbitrary set of input X S can be computed analytically as q ϕ ( z | X S ) =

Original:  However, L C ( θ, ϕ ) only includes reconstruction from all modalities and does not include generation from arbitrary modalities; thus, the performance of cross-modal generation might be degraded.
Revised: p ( z ) (cid:81) m : x m ∈ X S q ϕ m ( z | x m ) due to the nature of PoE. However, L C ( θ, ϕ ) only includes reconstruction from all modalities and does not include generation from arbitrary modalities; thus, the performance of cross-modal generation might be degraded.

Original: On the other hand, cross-generation in the objective conversely promotes generation degradation due to the limitation of mixture-based models.
Revised: On the other hand, cross-generation in the objective conversely promotes generation degradation due to the limitation of mixture-based models.

Original: We propose to include the term for unimodal reconstruction in the objective to experience the generation from unimodal posterior during learning while avoiding these dilemmas.
Revised: We propose to include the term for unimodal reconstruction in the objective to experience the gener- ation from unimodal posterior during learning while avoiding these dilemmas.

Original: The reconstruction term in Eq.
Revised: The reconstruction term in Eq.

Original: 5 can be rewritten as
Revised: 5 can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: (8) where we assume (cid:82) q ϕ ( z | X ) p d ( X C/m | x m ) dX C

Original: /m ≈ q ϕ
Revised: m = [Equation]

Original: ( z | x m ) 9 . This approximation changes to equal if the second term of Eq.
Revised: where all inferences in Eq.

Original: 7 is zero, making it a valid ELBO.
Revised: 8 are made using PoE-aggregated posterior.

Original: This rewritten objective can be converted to include all-modality and unimodal reconstruction, which mitigates the shortcoming of coordination-based VAEs while avoiding the limitation of mixture-based VAEs.
Revised: This rewritten objective can be converted to include all-modality and unimodal reconstruction, which mitigates the shortcoming of coordination-based VAEs while avoiding the limitation of mixture-based VAEs.

Original: We call thisnovel model Coordinated and unimodal Reconstructed Multimodal VAE (CRMVAE) .
Revised: We call this Coordinated and unimodal Reconstructed Multimodal VAE (CRMVAE) .

Original: We also examine the effect of this unimodal reconstruction from another perspective.

Original: Minimizing Eq 7 also minimizes the first term, the conditional mutual information.
Revised: The expected value for x m of the sum of the second term in Eq.

Original: This can  be transformed  as 
Revised: 8 can also be transformed into a reconstruction that includes another input as follows:

Original: [Equation]
Revised: [Equation]

Original: The second and fourth terms are negative, so they are learned to improve their respective information 10 . The first term corresponds to the information bottleneck, which is the constraint for learning a good representation.
Revised: Therefore, optimizing the unimodal reconstruction for each modality leads to the possibility of reconstruction from any modality. This performance can be generalized beyond reconstruction to improve the performance of cross-modal generation.

Original: Therefore, the third term can be considered a potential negative influence on representation learning. This term can be transformed according to Alemi et al. (2016) as follows:
Revised: Note that theoretically, the same is true for the left-hand side of Eq. 8, i.e., from the nature of PoE, it should be possible to learn inferences from any modality simply by optimizing the left-hand side of

Original: [Equation]
Revised: Eq. 8.

Original: This suggests that the proposed unimodal reconstruction might play a role in counteracting the negative effects on inferred representation . To counteract Eq 10 from the objective of CRMVAE, the coefficient in Eq.
Revised: However, some inferences might be ignored in actual optimization. In other words, it is true that if each inference is adequately learned, the joint inference will be adequate, but the converse is not necessarily true. Our proposal is empirical and attempts to alleviate this problem of being ignored by having each modality experience inference and generation.

Original:  8 should be 12 to π in the second term of L C ( θ, ϕ ) 11 .
Revised: This simple method is effective for cross-modal generation in a coordination-based setting, as we will show later in the results. In the experiment, we set the coefficient in Eq. 8 from 12 to π 11 to prevent the expected value term from increasing when the modality increases and having too large an effect on the overall objective.

Original:  R ELATED WORKS
Revised:  R ELATED WORKS

Original: There have been many studies of multimodal VAEs (Suzuki & Matsuo, 2022), and the early ones before mixture-based required additional inference models.
Revised: There have been many studies of multimodal VAEs (Suzuki & Matsuo, 2022), and the early ones before mixture-based required additional inference models.

Original: Suzuki et al.
Revised: Suzuki et al.

Original: (2016) proposedJMVAE to approximate the unimodal posterior to explicitly approach the joint posterior of VAEs.
Revised: (2016) proposed to approximate the unimodal inference models to explicitly approach the joint posterior of VAEs, called JMVAE.

Revised: Vedantam et al.

Revised: (2018) introduced TELBO, where the objective is composed of the sum of ELBOs that take each set of modalities as input.

Revised: M 2 VAE (Korthals et al., 2019) can be regarded as a combination of JMVAE and TELBO, and VAEVAE (Wu & Goodman, 2019) excludes a KL divergence term between the joint inference and prior from M 2 VAE.

Original: With a few exceptions, they can be regarded as precursors to the coordination-based approach; however, the number of posterior needed for these models grows exponentially with the number of modalities.
Revised: Except for TELBO, they can be regarded as precursors to the coordination-based approach; however, the number of inference models needed for these models grows exponentially with the number of modalities.

Original: One important research direction in multimodal VAEs is the introduction of additional latent variables, which includes hierarchical latent variables (Sutter & Vogt, 2021;
Revised: One important research direction in multimodal VAEs is the introduction of additional latent variables, which includes hierarchical latent variables (Sutter & Vogt, 2021;

Original: Wolff et al., 2021; Vasco et al., 2022) and modality-specific latent variables (Tsai et al., 2018; Hsu & Glass, 2018; Sutter et al., 2020;
Revised: Wolff et al., 2021; Vasco et al., 2022) and modality-specific latent variables (Tsai et al., 2018; Hsu & Glass, 2018; Sutter et al., 2020;

Original: Lee & Pavlovic, 2020; Daunhawer et al., 2021b; Palumbo et al., 2022).
Revised: Lee & Pavlovic, 2020; Daunhawer et al., 2021b; Palumbo et al., 2022).

Original: This paper aims to improve the performance of multimodal VAEs in a different direction from them, i.e., whether limitations of existing mixture-based models can be mitigated by changing the objective while maintaining the same architecture .
Revised: This paper aims to improve the performance of multimodal VAEs in a different direction from them, i.e., whether limitations of existing mixture-based models can be mitigated by changing the objective while maintaining the same architecture .

Original: Note that our proposed model and the introduction of additional latent variables are orthogonal concepts and can be easily incorporated into the proposed model.
Revised: Note that our proposed model and the introduction of additional latent variables are orthogonal concepts and can be easily incorporated into the proposed model.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We use the following five diverse datasets: MNIST-SVHN-Text (Sutter et al., 2020), PolyMNIST (Sutter et al., 2021), Translated-PolyMNIST (Daunhawer et al., 2021a), CUB Wah et al.
Revised: We use the following five diverse datasets: MNIST-SVHN-Text (Sutter et al., 2020), PolyMNIST (Sutter et al., 2021), Translated-PolyMNIST (Daunhawer et al., 2021a), CUB Wah et al.

Original: (2011), and Bimodal CelebA (Sutter et al., 2020).
Revised: (2011), and Bimodal CelebA (Sutter et al., 2020).

Original: For a more description of the dataset, see the appendix.
Revised: For a more description of the dataset, see the appendix.

Original: These are commonly used in studies of multimodal VAEs, but to our knowledge, this is the first study in that experiments have been conducted on all of these datasets.
Revised: These are commonly used in studies of multimodal VAEs, but to our knowledge, this is the first study in that experiments have been conducted on all of these datasets.

Original: In particular, CUB and
Revised: In particular, CUB and

Original: Translated-PolyMNIST were noted to be challenging datasets that could not be successfully trained in existing models (Daunhawer et al., 2021a).
Revised: Translated-PolyMNIST were noted to be challenging datasets that could not be successfully trained in existing models (Daunhawer et al., 2021a).

Original: It also shows that Bimodal CelebA and CUB could only be learned in models with additional latent variables (Sutter et al., 2020;
Revised: It also shows that Bimodal CelebA and CUB could only be learned in models with additional latent variables (Sutter et al., 2020;

Original: Palumbo et al., 2022).
Revised: Palumbo et al., 2022).

Original: For models trained in all modalities, we evaluate both 1) cross-coherence and 2) generation quality given arbitrary modalities.
Revised: For models trained in all modalities, we evaluate both 1) cross-coherence and 2) generation quality given arbitrary modalities.

Original: We measure each of them in terms of 1) the test accuracy of a pre-trained classifier 12 that predicts labels from cross-generated images and 2) the Fréchet inception distance (FID) (Heusel et al., 2017) in cross-generated modalities.
Revised: We measure each of them in terms of 1) the test accuracy of a pre-trained classifier 12 that predicts labels from cross-generated images and 2) the Fréchet inception distance (FID) (Heusel et al., 2017) in cross-generated modalities.

Original: For MNIST-SVHN-Text, joint coherence is also verified by generating random images in all modalities and evaluating whether all classes predicted by them are consistent.
Revised: For MNIST-SVHN-Text, joint coherence is also verified by generating random images in all modalities and evaluating whether all classes predicted by them are consistent.

Original: Note that we evaluated the generated results of the text modality in
Revised: Note that we evaluated the generated results of the text modality in

Original: MNIST-SVHN-Text in terms of the Levenshtein distance from the true text (without space characters) since we found that the pre-trained classifier from Sutter et al.
Revised: MNIST-SVHN-Text in terms of the Levenshtein distance from the true text (without space characters) since we found that the pre-trained classifier from Sutter et al.

Original: (2021) did not correctly evaluate the generated results (we discuss in the experimental results.)
Revised: (2021) did not correctly evaluate the generated results (we discuss in the experimental results.)

Revised: To assess the cross-coherence for CUB,

Revised: we

Revised: The top row shows the modalities generated, and the second row shows modalities conditioned when generating them.

Revised: For example, the second column of the MVAE results (accuracy 0.30) represents the cross-generation from S to M. A+B means that the modality in the row above is generated from the modalities A and B. R represents the joint coherence of all modalities in a random generation.

Original: To assess the cross-coherence for CUB, we use the cosine similarity between the latent space of the two inferred modalities since there are no labels.
Revised: [Table] better. [Figure] use the cosine similarity between the latent space of the two inferred modalities since there are no labels.

Original: We also used this evaluation for Bimodal CelebA.
Revised: We also used this evaluation for Bimodal CelebA.

Original: For the network architectures, we follow Sutter et al.
Revised: For the network architectures, we follow Sutter et al.

Original: ; Daunhawer et al.
Revised: ; Daunhawer et al.

Original: (2021a), with 64dimensional latent variables for Bimodal CelebA and CUB,
Revised: (2021a), with 64dimensional latent variables for Bimodal CelebA and CUB,

Original: 32 for MNIST-SVHN-Text, and 512 for
Revised: 32 for MNIST-SVHN-Text, and 512 for

Original: PolyMNIST and Translated-PolyMNIST.
Revised: PolyMNIST and Translated-PolyMNIST.

Original: The distributions for image modalities are set to Laplace, these are not entropies respect to parameters of models

Original: [Table]

Original: scores for crossgeneration. Lower is better. [Figure] and text is set to categorical.
Revised: The distributions for image modalities are set to Laplace, and text is set to categorical.

Original: We use Adam (Kingma & Ba, 2014) for optimization with a learning rate of 0.0005 and a mini-batch size of 256.
Revised: We use Adam (Kingma & Ba, 2014) for optimization with a learning rate of 0.0005 and a mini-batch size of 256.

Original: We set β (the coefficient of the KL divergence term in the multimodal ELBO) to 1 for PolyMNIST and MNIST-SVHN-Text and 0.1 for Translated-PolyMNIST, π m to 1 M +1 , and ω S to 12 M .
Revised: We set β (the coefficient of the KL divergence term in the multimodal ELBO) to 1 for PolyMNIST and MNIST-SVHN-Text and 0.1 for Translated-PolyMNIST, π m to 1 M +1 , and ω S to 12 M .

Original: We train 100 epochs for MNIST-SVHN-Text, 150 for CUB, 200 for
Revised: We train 100 epochs for MNIST-SVHN-Text, 150 for CUB, 200 for

Original: Bimodal CelebA, and 500 for PolyMNIST and Translated-PolyMNIST.
Revised: Bimodal CelebA, and 500 for PolyMNIST and Translated-PolyMNIST.

Original: All experimental results are the average of three runs with different seeds.
Revised: All experimental results are the average of three runs with different seeds.

Original: We implement them using PyTorch (Paszke et al., 2019).
Revised: We implement them using PyTorch (Paszke et al., 2019).

Original: Our experiments were performed using two machines with five NVIDIA Titan RTX GPUs.
Revised: Our experiments were performed using two machines with five NVIDIA Titan RTX GPUs.

Original: We compare CRMVAE with MVAE, mixture-based VAEs (MMVAE, MoPoE, and MMJSD), and coordination-based VAEs (MVTCAE and MMJSD-PoE), all with the same architecture settings.
Revised: We compare CRMVAE with MVAE, mixture-based VAEs (MMVAE, MoPoE, and MMJSD), and coordination-based VAEs (MVTCAE and MMJSD-PoE), all with the same architecture settings.

Original: 6. MNIST-SVHN-T EXT
Revised: 6. MNIST-SVHN-T EXT

Original: Table 1 shows the joint and cross-coherence results, and Table 2 shows the FID scores of images on
Revised: Table 1 shows the joint and cross-coherence results, and Table 2 shows the FID scores of images on

Original: MNIST-SVHN-Text.
Revised: MNIST-SVHN-Text.

Original: First, we can confirm that MVAE has high generation quality but the lowest cross-coherence since it learns only the modality reconstruction.
Revised: First, we can confirm that MVAE has high generation quality but the lowest cross-coherence since it learns only the modality reconstruction.

Original: We next find that mixture-based models perform poorly in cross-generation of SVHN and Text 13 .
Revised: We next find that mixture-based models perform poorly in cross-generation of SVHN and Text 13 .

Original: According to the qualitative results shown in Figure 1, the cross-generation of Text on MoPoE-VAE is unsuccessful.
Revised: According to the qualitative results shown in Figure 1, the cross-generation of Text on MoPoE-VAE is unsuccessful.

Original: While Tableshows that cross-generation of MNIST appears to perform well, the qualitative results show that the generated images are average with no variation.
Revised: While Tableshows that cross-generation of MNIST appears to perform well, the qualitative results show that the generated images are average with no variation.

Original: Table 2 indicates that FID scores of mixture-based models on MNIST are significantly lower than these of MVAE, which corresponds to these results.
Revised: Table 2 indicates that FID scores of mixture-based models on MNIST are significantly lower than these of MVAE, which corresponds to these results.

Original: Existing coordination-based models mitigate this limitation and perform better on SVHN.
Revised: Existing coordination-based models mitigate this limitation and perform better on SVHN.

Original: Furthermore, FID scores show that the coordination-based models significantly outperform the mixturebased models in terms of generating quality and are comparable to MVAE.
Revised: Furthermore, FID scores show that the coordination-based models significantly outperform the mixturebased models in terms of generating quality and are comparable to MVAE.

Original: Comparing MMJSD-PoE and MVTCAE, MVTCAE performs better in terms of both cross-coherence and generation quality, confirming the effectiveness of MVTCAE experimentally.
Revised: Comparing MMJSD-PoE and MVTCAE, MVTCAE performs better in terms of both cross-coherence and generation qual- ity, confirming the effectiveness of MVTCAE experimentally.

Original: However, the performance of the coordination-based is degraded, especially in cross-generation from SVHN.
Revised: However, the performance of the coordination-based is degraded, especially in cross-generation from SVHN.

Original: CRMVAE mitigates coordination-based issues and significantly improves cross-coherence performance.
Revised: CRMVAE mitigates coordination-based issues and significantly improves cross-coherence performance.

Original: In particular, CRMVAE outperforms all existing methods, including mixture-based methods, for cross-coherence of SVHN and Text.
Revised: In particular, CRMVAE outperforms all existing methods, including mixture-based methods, for cross-coherence of SVHN and Text.

Original: We also confirm that CRMVAE retains the advantage of the coordination-based models, i.e., high-generation quality.
Revised: We also confirm that CRMVAE retains the advantage of the coordination-based models, i.e., high-generation quality.

Original: In particular, we have confirmed the highest quality of cross-generation on MNIST among all models that include MVAE.
Revised: In particular, we have confirmed the highest quality of cross-generation on MNIST among all models that include MVAE.

Original: The visualization results also confirm that CRMVAE achieves high coherence while maintaining high quality.
Revised: The visualization results also confirm that CRMVAE achieves high coherence while maintaining high quality.

Original: Thepoor performance of  joint coherence results generated from random images  compared to other studies might bepartly due  to the different methods of text evaluation.
Revised: The performance of the joint coherence results with random images is poor compared to other studies, which might be due in part to the different methods for evaluating text.

Original: The performance of the coordination-based method is relatively high but not significant.
Revised: The performance of the coordination-based method is relatively high but not significant.

Original: The relationship between joint coherence and coordination-based should be studied further in the future.
Revised: The relationship between joint coherence and coordination-based should be studied further.

Original: 6. P OLY MNIST AND T RANSLATED -P OLY MNIST
Revised: 6. P OLY MNIST AND T RANSLATED -P OLY MNIST

Original: Figure 2 shows the results on PolyMNIST and Translated-PolyMNIST.
Revised: Figure 2 shows the results on PolyMNIST and Translated-PolyMNIST.

Original: Based on the above results, we select MoPoE-VAE and MVTCAE as mixture-based and coordination-based models for comparison.
Revised: Based on the above results, we select MoPoE-VAE and MVTCAE as mixture-based and coordination-based models for comparison.

Original: First the results of the existing models on PolyMNIST show the same trend as on MNIST-SVHN-Text, i.e., there is a trade-off between cross-coherence and generation quality.
Revised: First the results of the existing models on PolyMNIST show the same trend as on MNIST-SVHN-Text, i.e., there is a trade-off between cross-coherence and generation quality.

Original: CRMVAE mitigates these limitations and achieves the highest cross-coherence while maintaining high generation quality.
Revised: CRMVAE mitigates these limitations and achieves the highest cross-coherence while maintaining high generation quality.

Original: Note that it might seem strange that both MVTCAE and CRMVAE produce lower quality when the number of modalities increases.
Revised: Note that it might seem strange that both MVTCAE and CRMVAE produce lower quality when the number of modalities increases.

Original: This is thought to be due to a decrease in the diversity of the generated data distribution as the number of modalities increases.
Revised: This is thought to be due to a decrease in the diversity of the generated data distribution as the number of modalities increases.

Original: Translated-PolyMNIST is a slightly modified PolyMNIST, but Daunhawer et al.
Revised: Translated-PolyMNIST is a slightly modified PolyMNIST, but Daunhawer et al.

Original: (2021a) point out that the generation quality and cross-coherence performance of mixed-based models is significantly degraded on this dataset.
Revised: (2021a) point out that the generation quality and cross-coherence performance of mixed-based models is significantly degraded on this dataset.

Original: Our experiments also confirm that most existing models fail to cross-generate across modalities.
Revised: Our experiments also confirm that most existing models fail to cross-generate across modalities.

Original: CRMVAE, on the other hand, clearly improves cross-coherence, although not as much as PolyMNIST.
Revised: CRMVAE, on the other hand, clearly improves cross-coherence, although not as much as PolyMNIST.

Original: Next, CRMVAE has the highest generation quality, significantly outperforming the mixture-based models.
Revised: Next, CRMVAE has the highest generation quality, significantly outperforming the mixture-based models.

Original: See appendix G.1 for visualization of the generated images on PolyMNIST and Translated-PolyMNIST.
Revised: See Appendix H for visualization of the generated images.

Original: 6. CUB AND B IMODAL C ELEB A
Revised: 6. CUB AND B IMODAL C ELEB A

Original: Figures 3 and 4 show the quantitative and qualitative results for CUB and Bimodal CelebA.
Revised: Figures 3 and 4 show the quantitative and qualitative results for CUB and Bimodal CelebA.

Original: We examine the generation quality and cross-coherence performance as an ablation, when varying β .
Revised: We examine the generation quality and cross-coherence performance as an ablation, when varying β .

Original: First, the FID score indicates that MVAEs have high generation quality, while MoPoE-VAEs have significantly lower quality.
Revised: First, the FID score indicates that MVAEs have high generation quality, while MoPoE-VAEs have significantly lower quality.

Original: In most cases, the FID score does not change much or worsen when β is increased, while the FID score for MoPoE-VAE improves slightly.
Revised: In most cases, the FID score does not change much or worsen when β is increased, while the FID score for MoPoE-VAE improves slightly.

Original: This is thought to be because increasing β in MoPoE-VAE corresponds to making the distribution closer to the standard Gaussian, which diminishes the effect of generation from sub-sampling.
Revised: This is thought to be because increasing β in MoPoE-VAE corresponds to making the distribution closer to the standard Gaussian, which diminishes the effect of generation from sub-sampling.

Original: Furthermore, the performance of the coordination-based methods, including CRMVAE, is close to that of MVAE, indicating that they do not have a generation quality issue on these challenging datasets.
Revised: Furthermore, the performance of the coordination-based methods, including CRMVAE, is close to that of MVAE, indicating that they do not have a generation quality issue on these challenging datasets.

Original: Next, the evaluation of cosine similarity shows that MVAE and MoPoE-VAE do not change that much when β is increased, while the coordination-based models, including CRMVAE, improve when β is increased.
Revised: Next, the evaluation of cosine similarity shows that MVAE and MoPoE-VAE do not change that much when β is increased, while the coordination-based models, including CRMVAE, improve when β is increased.

Original: This is because, in the coordination-based models, the term adjusted by β corresponds to the term that brings the inference of all modalities closer to that of each modality.
Revised: This is because, in the coordination-based models, the term adjusted by β corresponds to the term that brings the inference of all modalities closer to that of each modality.

Original: Compared to MVTCAE, CRMVAE performs better for Bimodal CelebA for all values of β and for CUB when β is large.
Revised: Compared to MVTCAE, CRMVAE performs better for Bimodal CelebA for all values of β and for CUB when β is large.

Original: Furthermore, the cross-coherence with pre-trained classifiers in Bimodal CelebA also shows that CRMVAE performs better than MVTCAE.
Revised: Furthermore, the cross-coherence with pre-trained classifiers in Bimodal CelebA also shows that CRMVAE performs better than MVTCAE.

Original: Note that MoPoE-VAE has the highest cross-coherence value, but as the qualitative results show, this only improves classification accuracy at the expense of image quality, e.g., diversity (as is the case for MNIST in MNIST-SVHN-Text).
Revised: Note that MoPoE-VAE has the highest cross-coherence value, but as the qualitative results show, this only improves classification accuracy at the expense of image quality, e.g., diversity (as is the case for MNIST in MNIST-SVHN-Text).

Original: Qualitative results showed the following limitations: MVAE failed to produce images corresponding to the text, and MoPoE-VAE produced only average or blurred images.
Revised: Qualitative results showed the following limitations: MVAE failed to produce images corresponding to the text, and MoPoE-VAE produced only average or blurred images.

Original: Comparing MVTCAE and
Revised: Comparing MVTCAE and

Original: CRMVAE, CRMVAE produces more appropriately conditioned images.
Revised: CRMVAE, CRMVAE produces more appropriately conditioned images.

Original: For example, in Bimodal
Revised: For example, in Bimodal

Original: CelebA, Text 3 contains a male attribute and CRMVAE produces a corresponding face image, while
Revised: CelebA, Text 3 contains a male attribute and CRMVAE produces a corresponding face image, while

Original: MVTCAE produces some female face images.
Revised: MVTCAE produces some female face images.

Original:  C ONCLUSION AND DISCUSSION
Revised:  C ONCLUSION AND DISCUSSION

Original: We focused on coordination-based multimodal VAEs to avoid the limitations of multimodal VAEs.
Revised: We focused on coordination-based multimodal VAEs to avoid the limitations of multimodal VAEs.

Original: We first discuss the new relation among coordination-based multimodal VAEs, point out the issue of unimodal inference not learning from reconstruction in them, and then introduce a new objective involving unimodal reconstruction.
Revised: We first discuss the new relation among coordination-based multimodal VAEs, point out the issue of unimodal inference not learning from reconstruction in them, and then introduce a new objective involving unimodal reconstruction.

Original: Experiments show that the proposed model mitigates existing models’ limitations (both mixture-based and coordination-based) and shows overall superior performance on diverse datasets for generation quality and cross-coherence.
Revised: Experiments show that the proposed model mitigates existing models’ limitations (both mixture-based and coordination-based) and shows overall superior performance on diverse datasets for generation quality and cross-coherence.

Original: One limitation of CRMVAE is that there still remains a trade-off between generation quality and cross-coherence.
Revised: One limitation of CRMVAE is that there still remains a trade-off between generation quality and cross-coherence.

Original: For example, in the PolyMNIST experiment, CRMVAE performed best in crosscoherence, but the FID score was worse than MVTCAE.
Revised: For example, in the PolyMNIST experiment, CRMVAE performed best in crosscoherence, but the FID score was worse than MVTCAE.

Original: The performance of joint coherence was also confirmed to be limited.
Revised: The performance of joint coherence was also confirmed to be limited.

Original: In addition, although we used several diverse datasets to evaluate performance, these datasets are much smaller than the various multimodal information obtained from the real world.
Revised: In addition, although we used several diverse datasets to evaluate performance, these datasets are much smaller than the various multimodal information obtained from the real world.

Original: Therefore, we should validate our model using such large multimodal information.
Revised: Therefore, we should validate our model using such large multimodal information.

Original: A D ATASET
Revised: A D ATASET

Original: MNIST-SVHN-Text (Sutter et al., 2020) is the MNIST-SVHN (Shi et al., 2019) with the addition of the text modality, which represents the English name of each number as text and has diversity.
Revised: MNIST-SVHN-Text (Sutter et al., 2020) is the MNIST-SVHN (Shi et al., 2019) with the addition of the text modality, which represents the English name of each number as text and has diversity.

Original: Different modalities of the same example have the same digits.
Revised: Different modalities of the same example have the same digits.

Original: PolyMNIST (Sutter et al., 2021) consists of a set of five different MNIST images with different backgrounds and handwriting styles, each of which is considered a different modality.
Revised: PolyMNIST (Sutter et al., 2021) consists of a set of five different MNIST images with different backgrounds and handwriting styles, each of which is considered a different modality.

Original: Translated-PolyMNIST (Daunhawer et al., 2021a) is composed of the same backgrounds as PolyMNIST with the numbers downscaled 14 and randomly transformed into position in the images.
Revised: Translated-PolyMNIST (Daunhawer et al., 2021a) is composed of the same backgrounds as PolyMNIST with the numbers downscaled 14 and randomly transformed into position in the images.

Original: This is intended to allow for more significant variation among modalities than PolyMNIST.
Revised: This is intended to allow for more significant variation among modalities than PolyMNIST.

Original: Caltech Birds ( CUB )
Revised: Caltech Birds ( CUB )

Original: Wah et al.
Revised: Wah et al.

Original: (2011) is a dataset with bird images and their captions.
Revised: (2011) is a dataset with bird images and their captions.

Original: We follow
Revised: We follow

Original: Daunhawer et al.
Revised: Daunhawer et al.

Original: (2021a), i.e., we treat as 64 × 64 real images without feature extraction.
Revised: (2021a), i.e., we treat as 64 × 64 real images without feature extraction.

Original: Bimodal CelebA (Sutter et al., 2020) is a dataset of face images from the CelebA (Liu et al., 2015) with additional text describing the corresponding attribute labels and changing the starting index to give a diversity.
Revised: Bimodal CelebA (Sutter et al., 2020) is a dataset of face images from the CelebA (Liu et al., 2015) with additional text describing the corresponding attribute labels and changing the starting index to give a diversity.

Original: B D ETAILED DERIVATION OF P ROOF
Revised: B D ETAILED DERIVATION OF P ROOF

Original: E p d ( X ) [ E q ϕ
Revised: E p d ( X ) [ E q ϕ

Original: ( z | X ) [log p θ ( X | z )] − D KL ( q ϕ ( z | X ) ||
Revised: ( z | X ) [log p θ ( X | z )] − D KL ( q ϕ ( z | X ) ||

Original: p f
Revised: p f

Original: ( z | X ))]] =
Revised: ( z | X ))]] =

Original: E p d ( X ) (cid:2)
Revised: E p d ( X ) (cid:2)

Original: E q ϕ ( z | X ) [log p θ ( X | z )] − E q ϕ
Revised: E q ϕ ( z | X ) [log p θ ( X | z )] − E q ϕ

Original: ( z | X ) [log q ϕ ( z | X ) − log p
Revised: ( z | X ) [log q ϕ ( z | X ) − log p

Original: f
Revised: f

Original: ( z | X )] (cid:3) =
Revised: ( z | X )] (cid:3) =

Original: E p d ( X ) (cid:34)
Revised: E p d ( X ) (cid:34)

Original: E q ϕ ( z | X ) [log p θ ( X | z )] − E q ϕ
Revised: E q ϕ ( z | X ) [log p θ ( X | z )] − E q ϕ

Original: ( z | X ) [log q ϕ ( z | X ) −
Revised: ( z | X ) [log q ϕ ( z | X ) −

Original: M (cid:88) m =
Revised: M (cid:88) m =

Original: π m log q ϕ 
Revised: π m log q ϕ m ( z | x m )]

Original: ( z | x m )] (cid:35) (cid:32) ∵
Revised: (cid:35) (cid:32) ∵

Original: p f
Revised: p f

Original: ( z | X ) =
Revised: ( z | X ) =

Original: M (cid:88) m =
Revised: M (cid:88) m =

Original: π m log q ϕ  ( z | x m ) (cid:33) =
Revised: π m log q ϕ m ( z | x m ) (cid:33) =

Original: E p d ( X ) (cid:34)
Revised: E p d ( X ) (cid:34)

Original: E q ϕ ( z | X ) [log p θ ( X | z )] − E q ϕ
Revised: E q ϕ ( z | X ) [log p θ ( X | z )] − E q ϕ

Original: ( z | X ) (cid:34) M (cid:88) m =
Revised: ( z | X ) (cid:34) M (cid:88) m =

Original: π m log q ϕ
Revised: π m log q ϕ

Original: ( z | X ) −
Revised: ( z | X ) −

Original: M (cid:88) m =
Revised: M (cid:88) m =

Original: π m log q ϕ  ( z | x m ) (cid:35)(cid:35) (cid:32) ∵ log q ϕ
Revised: π m log q ϕ m ( z | x m ) (cid:35)(cid:35) (cid:32) ∵ log q ϕ

Original: ( z | X ) =
Revised: ( z | X ) =

Original: M (cid:88) m =
Revised: M (cid:88) m =

Original: π m log q ϕ
Revised: π m log q ϕ

Original: ( z | X ) and
Revised: ( z | X ) and

Original: M (cid:88) m =
Revised: M (cid:88) m =

Original: π m =(cid:33) =
Revised: π m =(cid:33) =

Original: E p d ( X ) (cid:34)
Revised: E p d ( X ) (cid:34)

Original: E q ϕ ( z | X ) [log p θ ( X | z )] −
Revised: E q ϕ ( z | X ) [log p θ ( X | z )] −

Original: M (cid:88) m =
Revised: M (cid:88) m =

Original: π m E q ϕ
Revised: π m E q ϕ

Original: ( z | X ) [log q ϕ ( z | X ) − log q ϕ ( z | x m )] (cid:35) =

Original: E p d ( X ) (cid:34)

Original: E q ϕ ( z | X ) [log p θ (X | z )] − M (cid:88) m = π m D KL ( q ϕ  ( z | X ) ||
Revised: ( z | X ) [log q ϕ ( z | X ) − log q ϕ m ( z | x m )]

Original: q ϕ ( z | x m )) (cid:35) = L C ( θ, ϕ ) □
Revised: [Equation]

Original: C D ETAILED DERIVATION OF P ROOF
Revised: C D ETAILED DERIVATION OF P ROOF

Original: E p d ( X ) [ D KL ( q ϕ ( z | X ) || q ϕ ( z | x m ))] = (cid:90) p
Revised: [Equation]

Original: d ( x m ) (cid:90) (cid:90) q ϕ

Original: ( z | X ) p d ( X C/m | x m ) log q ϕ ( z | X ) q ϕ

Original: ( z | x m ) dX C/m d z d x m = (cid:90) p d ( x m ) (cid:90) (cid:90) q ϕ

Original: ( z | X ) p d ( X C/m | x m ) log q ϕ ( z | X )

Original: p d ( X C/m | x m ) q ϕ

Original: ( z | x m ) p d ( X C/m | x m ) dX C/m d z d x m = (cid:90)

Original: p d ( x m ) (cid:90) (cid:90) q ϕ

Original: ( z | X ) p d ( X C/m | x m ) log q ϕ ( z | X )

Original: p d ( X C/m | x m ) q avgϕ ( z | x m ) p d ( X C/m | x m ) dX C/m d z d x m + (cid:90) p

Original: d ( x m ) (cid:90) (cid:90) q ϕ ( z | X ) p d ( X C/m | x m ) dX C/m log q avgϕ ( z | x m) q ϕ ( z | x m ) d z d x m = (cid:90) p d ( x m ) (cid:90) (cid:90) q ϕ
Revised: m

Original: ( z , X C/m | x m ) log q ϕ ( z , X C/m | x m ) q avgϕ ( z | x m )

Original: p d ( X

Original: [Equation]
Revised: [Equation]

Original: D D ETAILED DERIVATION OF E Q . (7)
Revised: D D ETAILED DERIVATION OF E Q . (7)

Original: [Equation]
Revised: [Equation]

Original: m

Original: [Equation]
Revised: [Equation]

Original: (cid:90)

Original: [Equation]
Revised: [Equation]

Original: ( z | X ) pd (
Revised: p

Original: [Equation]
Revised: [Equation]

Original: C
Revised: X

Original: [Equation]
Revised: [Equation]

Original: m | x m ) dX C [Equation] m p d ( x m ) log p θ ( x m | z )
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: m
Revised: E p

Original: [Equation]
Revised: [Equation]

Original: (cid:90) [Equation] [Table] [Equation]
Revised: X )

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: |

Original: [Table]
Revised: [Table]

Original: (cid:18) [Table] [Table] [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: ) (cid:19)

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: (cid:3)

Original: [Equation]
Revised: [Table]

Original: E R ESULTS ON MNIST-SVHN-T EXT WITH STANDARD DEVIATIONS
Revised: E R ESULTS ON MNIST-SVHN-T EXT WITH STANDARD DEVIATIONS

Original: Table 3 and Table 4 are Table 1 and Table 2 with their standard deviations of three trials explicitly added.
Revised: Table 3 and Table 4 are Table 1 and Table 2 with their standard deviations of three trials explicitly added.

Original: The meaning of MVTCAE (rescale) is explained in appendix G.
Revised: The meaning of MVTCAE (rescale) is explained in Appendix G.

Original: F P ERFORMANCE ON THE LATENT REPRESENTATION
Revised: F P ERFORMANCE ON THE LATENT REPRESENTATION

Original: Table 5 shows that coordinated-based performs better as input modality increases.
Revised: Table 5 shows that coordination-based performs better as input modality increases.

Original: This is a feature of the coordinated-based method, which learns to approximate the joint posterior, so the amount of information to be inferred increases as the number of modalities increases.
Revised: This is a feature of the coordination-based method, which learns to approximate the joint posterior, so the amount of information to be inferred increases as the number of modalities increases.

Original: On the other hand, the mixture-based method performs better given a single modality.
Revised: On the other hand, the mixture-based method performs better given a single modality.

Original: The proposed method improves unimodal inference the most among the coordinated-based methods.
Revised: The proposed method improves unimodal inference the most among the coordination-based methods.

Original: The right panels of Figure 5 and Figure 6 show the inference performance of PolyMNIST and
Revised: The right panels of Figure 5 and Figure 6 show the inference performance of PolyMNIST and

Original: Translated-PolyMNIST, respectively.
Revised: Translated-PolyMNIST, respectively.

Original: It can be seen that the proposed method performs well when the number of modalities is small, while other models perform better when the number of modalities increases.
Revised: It can be seen that the proposed method performs well when the number of modalities is small, while other models perform better when the number of modalities increases.

Original: There are various possible reasons for this, but the scaling of the reconstruction error term, which will be discussed in the appendix G, might be the cause.
Revised: There are various possible reasons for this, but the scaling of the reconstruction error term, which will be discussed in the Appendix G, might be the cause.

Original: G C ONSIDERATION OF THE COEFFICIENTS OF THE RECONSTRUCTION TERM
Revised: G C ONSIDERATION OF THE COEFFICIENTS OF THE RECONSTRUCTION TERM

Original: We chose π instead of 12 as the coefficient in Eq. 
Revised: We chose π instead of 12 as the coefficient in Eq. 8.

Original: However, this makes the reconstruction error term smaller than the other models by a factor of π 2 .
Revised: However, this makes the reconstruction error term smaller than the other models by a factor of π 2 .

Original: To check this effect, we scaled the MVTCAE reconstruction error term by multiplying it by π 2 .
Revised: To check this effect, we scaled the MVTCAE reconstruction error term by multiplying it by π 2 .

Original: Since this effect is large when the modality is three or more, we experimented with MNIST-SVHN-Text, PolyMNIST, and Translated-PolyMNIST.
Revised: Since this effect is large when the modality is three or more, we experimented with MNIST-SVHN-Text, PolyMNIST, and Translated-PolyMNIST.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: The results denoted as “MVTCAE (rescale)” are the MVTCAE results aligned in scale with the reconstruction error term in CRMVAE.
Revised: The results denoted as “MVTCAE (rescale)” are the MVTCAE results aligned in scale with the reconstruction error term in CRMVAE.

Original: First, for MNIST-SVHN-Text, the performances of crosscoherence and FID are slightly better with scale, but CRMVAE is still better.
Revised: First, for MNIST-SVHN-Text, the performances of crosscoherence and FID are slightly better with scale, but CRMVAE is still better.

Original: The joint coherence was comparable to that of CRMVAE.
Revised: The joint coherence was comparable to that of CRMVAE.

Original: However, the performance of the inferred representation is lower with scaling.
Revised: However, the performance of the inferred representation is lower with scaling.

Original: This is because the coordinated-based learns inference by reconstruction, and when this term is small in the overall expression, it fails to learn inference.
Revised: This is because the coordination-based learns inference by reconstruction, and when this term is small in the overall expression, it fails to learn inference.

Original: Next, PolyMNIST and
Revised: Next, PolyMNIST and

Original: Translated-PolyMNIST improve cross-coherence performance, but as with MNIST-SVHN-Text, inference performance is poor.
Revised: Translated-PolyMNIST improve cross-coherence performance, but as with MNIST-SVHN-Text, inference performance is poor.

Original: Compared to MVTCAE with rescaling, the performance of CRMVAE is better, and this comparison, in which the reconstruction error terms are scaled to each other, might better evaluate the effect of unimodal reconstruction included in CRMVAE.
Revised: Compared to MVTCAE with rescaling, the performance of CRMVAE is better, and this comparison, in which the reconstruction error terms are scaled to each other, might better evaluate the effect of unimodal reconstruction included in CRMVAE.

Original: [Table]
Revised: [Table]

Original: [Figure]

Original: C l a ss i f

Original: i c a t i o n a cc u r a c y

Original: [Figure][Figure]
Revised: [Figure]

Original: c a t i o n

Original: [Figure]
Revised: [Figure]

Original: These results show that adjusting for scale does not change the superiority of CRMVAE.
Revised: These results show that adjusting for scale does not change the superiority of CRMVAE.

Original: In other words, the higher performance of CRMVAE is not due to scaling alone.
Revised: In other words, the higher performance of CRMVAE is not due to scaling alone [Figure]

Original: G.
Revised: H

Original: C ONDITIONAL GENERATION ON P OLY MNIST AND T RANSLATED -P OLY MNIST
Revised: C ONDITIONAL GENERATION ON P OLY MNIST AND T RANSLATED -P OLY MNIST

Original: Figure 7 and Figure 8 visualize the conditionally generated images on the PolyMNIST and TranslatedPolyMNIST datasets.
Revised: Figure 7 and Figure 8 visualize the conditionally generated images on the PolyMNIST and TranslatedPolyMNIST datasets.

Original: M0,1,2,3 !
Revised: M0,1,2,3 !

Original: M
Revised: M

Original: MVAE
Revised: MVAE

Original: MoPoE-VAE
Revised: MoPoE-VAE

Original: MVTCAE
Revised: MVTCAE

Original: CRMVAE
Revised: CRMVAE

Original: M
Revised: M

Original: M M
Revised: M M

Original: M M
Revised: M M

Original: MVAE
Revised: MVAE

Original: MoPoE-VAE
Revised: MoPoE-VAE

Original: MVTCAE
Revised: MVTCAE

Original: CRMVAE
Revised: CRMVAE

Original: M1,2,3,4 !
Revised: M1,2,3,4 !

Original: M
Revised: M

Original: M0,2,3,4 !
Revised: M0,2,3,4 !

Original: M
Revised: M

Original: M0,1,2,4 !
Revised: M0,1,2,4 !

Original: M3 M0,1,3,4 !
Revised: M3 M0,1,3,4 !

Original: M
Revised: M

Original: MVAE
Revised: MVAE

Original: MoPoE-VAE
Revised: MoPoE-VAE

Original: MVTCAE
Revised: MVTCAE

Original: CRMVAE
Revised: CRMVAE

Original: MVAE
Revised: MVAE

Original: MoPoE-VAE
Revised: MoPoE-VAE

Original: MVTCAE
Revised: MVTCAE

Original: CRMVAE
Revised: CRMVAE

Original: MVAE
Revised: MVAE

Original: MoPoE-VAE
Revised: MoPoE-VAE

Original: MVTCAE
Revised: MVTCAE

Original: CRMVAE
Revised: CRMVAE

Original: [Figure]
Revised: [Figure]