Original: M ULTIMODAL V ARIATIONAL A UTOENCODERS FOR S EMI -S UPERVISED L EARNING : I N D EFENSE OF P RODUCT - OF -E XPERTS
Revised: M ULTIMODAL V ARIATIONAL A UTOENCODERS FOR S EMI -S UPERVISED L EARNING : I N D EFENSE OF P RODUCT - OF -E XPERTS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Multimodal generative models should be able to learn a meaningful latent representation that enables a coherent joint generation of all modalities (e.g., images and text).
Revised: Multimodal generative models should be able to learn a meaningful latent representation that enables a coherent joint generation of all modalities (e.g., images and text).

Original: Many applications also require the ability to accurately sample modalities conditioned on observations of a subset of the modalities.
Revised: Many applications also require the ability to accurately sample modalities conditioned on observations of a subset of the modalities.

Original: Often not all modalities may be observed for all training data points, so semi-supervised learning should be possible.
Revised: Often not all modalities may be observed for all training data points, so semi-supervised learning should be possible.

Original: In this study, we evaluate a family of product-ofexperts (PoE) based variational autoencoders that have these desired properties.
Revised: In this study, we evaluate a family of product-ofexperts (PoE) based variational autoencoders that have these desired properties.

Original: We include a novel PoE based architecture and training procedure.
Revised: We include a novel PoE based architecture and training procedure.

Original: An empirical evaluation shows that the PoE based models can outperform an additive mixtureof-experts (MoE) approach.
Revised: An empirical evaluation shows that the PoE based models can outperform an additive mixtureof-experts (MoE) approach.

Original: Our experiments support the intuition that PoE models are more suited for a conjunctive combination of modalities while MoEs are more suited for a disjunctive fusion.
Revised: Our experiments support the intuition that PoE models are more suited for a conjunctive combination of modalities while MoEs are more suited for a disjunctive fusion.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: It has been argued that a product-of-experts (PoE) approach is not well suited for multimodal generative modelling using variational autoencoders (VAEs) in comparison to additive mixture-of-experts (MoE).

Original: This study sets out to test this conjecture for state-of-the-art multimodal VAEs.

Original: Multimodal generative modelling is important because information about real-world objects typically comes in different representations, or modalities.
Revised: Multimodal generative modelling is important because information about real-world objects typically comes in different representations, or modalities.

Original: The information provided by each modality may be erroneous and/or incomplete, and a complete reconstruction of the full information can often only be achieved by combining several modalities.
Revised: The information provided by each modality may be erroneous and/or incomplete, and a complete reconstruction of the full information can often only be achieved by combining several modalities.

Original: For example, in image- and video-guided translation (Caglayan et al., 2019), additional visual context can potentially resolve ambiguities (e.g., noun genders) when translating written text.
Revised: For example, in image- and video-guided translation (Caglayan et al., 2019), additional visual context can potentially resolve ambiguities (e.g., noun genders) when translating written text.

Original: In many applications, modalities may be missing for a subset of the observed samples during training and deployment.
Revised: In many applications, modalities may be missing for a subset of the observed samples during training and deployment.

Original: Often the description of an object in one modality is easy to obtain, while annotating it with another modality is slow and expensive.
Revised: Often the description of an object in one modality is easy to obtain, while annotating it with another modality is slow and expensive.

Original: Given two modalities, we call samples paired when both modalities are present, and unpaired if one is missing.
Revised: Given two modalities, we call samples paired when both modalities are present, and unpaired if one is missing.

Original: The simplest way to deal with paired and unpaired training examples is to discard the unpaired observations for learning.
Revised: The simplest way to deal with paired and unpaired training examples is to discard the unpaired observations for learning.

Original: The smaller the share of paired samples, the more important becomes the ability to additionally learn from the unpaired data, referred to as semi-supervised learning in this context.
Revised: The smaller the share of paired samples, the more important becomes the ability to additionally learn from the unpaired data, referred to as semi-supervised learning in this context (following the terminology from Wu & Goodman, 2019.

Revised: Typically one would associate semi-supervised learning with learning form labelled and unlabelled data to solve a classiﬁcation or regression tasks).

Original: Our goal is to provide a model that can leverage the information contained in unpaired samples and to investigate the capabilities of the model in situations of low levels of supervision, that is, when only a few paired samples are available.
Revised: Our goal is to provide a model that can leverage the information contained in unpaired samples and to investigate the capabilities of the model in situations of low levels of supervision, that is, when only a few paired samples are available.

Original: While a modality can be as low dimensional as a label, which can be handled by a variety of discriminative models (van Engelen & Hoos, 2020), we are interested in high dimensional modalities, for example an image and a text caption.
Revised: While a modality can be as low dimensional as a label, which can be handled by a variety of discriminative models (van Engelen & Hoos, 2020), we are interested in high dimensional modalities, for example an image and a text caption.

Original: Learning a representation of multimodal data that allows to generate high-quality samples requires the following: 1) deriving meaningful representation in a joint latent space for each high dimensional modality and 2) bridging the representations of different modalities in a way that the relations between them are preserved.
Revised: Learning a representation of multimodal data that allows to generate high-quality samples requires the following: 1) deriving meaningful representation in a joint latent space for each high dimensional modality and 2) bridging the representations of different modalities in a way that the relations between them are preserved.

Original: The latter means that we do not want the modalities to be represented orthogonally in the latent space – ideally the latent space should encode the object’s properties independent of the input modality.
Revised: The latter means that we do not want the modalities to be represented orthogonally in the latent space – ideally the latent space should encode the object’s properties independent of the input modality.

Original: Variational autoencoders (Kingma & Welling, 2014) using a product-of-experts (PoE, Hinton, 2002; Welling, 2007) approach for combining input modalities are a promising approach for multimodal generative modelling having the desired properties, in particular the VAEVAE model developed by Wu & Goodman (2018) and a novel model termed SVAE, which we present in this study.
Revised: Variational autoencoders (Kingma & Welling, 2014) using a product-of-experts (PoE, Hinton, 2002; Welling, 2007) approach for combining input modalities are a promising approach for multimodal generative modelling having the desired properties, in partic- ular the VAEVAE model developed by Wu & Goodman (2018) and a novel model termed SVAE, which we present in this study.

Original: Both models can handle multiple high dimensional modalities, which may not all be observed at training time.
Revised: Both models can handle multiple high dimensional modalities, which may not all be observed at training time.

Original: Per contra, recent work argues that “PoE factorisation does not appear to be practically suited for multi-modal learning” (Shi et al., 2019). Shi et al.
Revised: It has been argued that a product-of-experts (PoE) approach is not well suited for multimodal generative modelling using variational autoencoders (VAEs) in comparison to additive mixture-of-experts (MoE).

Original: show that the PoE-based MVAE (Wu & Goodman, 2018) fails to properly model two high-dimensional modalities in contrast to an (additive) MoE approach referred to as MMVAE. In the following, we disprove this conjecture and show that advanced PoE-based models can perform better than MMVAE.
Revised: It has empirically been shown that the PoE-based MVAE (Wu & Goodman, 2018) fails to properly model two high-dimensional modalities in contrast to an (additive) MoE approach referred to as MMVAE, leading to the conclusion that “PoE factorisation does not appear to be practically suited for multi-modal learning” (Shi et al., 2019). This study sets out to test this conjecture for state-of-the-art multimodal VAEs.

Original: The next section summarizes related work.
Revised: The next section summarizes related work.

Original: Section 3 introduces SVAE as an alternative PoE based
Revised: Section 3 introduces SVAE as an alternative PoE based

Original: VAE approach derived from axiomatic principles.
Revised: VAE approach derived from axiomatic principles.

Original: Then we present our experimental evaluation of multimodal VAEs before we conclude.
Revised: Then we present our experimental evaluation of multimodal VAEs before we conclude.

Original:  B ACKGROUND AND R ELATED W ORK
Revised:  B ACKGROUND AND R ELATED W ORK

Original: We consider multimodal generative modelling.
Revised: We consider multimodal generative modelling.

Original: Without loss of generality, we restrict our considerations to two modalities x 1 ∈ X 1 , x 2 ∈ X 2 , where one modality may be missing at a time.
Revised: We mainly restrict our considerations to two modalities x 1 ∈ X 1 , x 2 ∈ X 2 , where one modality may be missing at a time.

Original:  To address the problem of generative cross-modal modeling, one modality x 1 can be generated from another modality x 2 by simply using independently trained generative models ( x 1 → x 2 and x 2 → x 1 ) or a composed but non-interchangeable representation (Wang et al., 2016; Sohn et al., 2015).
Revised: Extensions to more modalities are discussed in Experiments section and Appendix D. To address the problem of generative cross-modal modeling, one modality x 1 can be generated from another modality x 2 by simply using independently trained generative models ( x 1 → x 2 and x 2 → x 1 ) or a composed but noninterchangeable representation (Wang et al., 2016; Sohn et al., 2015).

Original: However, the ultimate goal of multimodal representation learning is to ﬁnd a meaningful joint latent code distribution bridging the two individual embeddings learned from x 1 and x 2 alone.
Revised: However, the ultimate goal of multimodal representation learning is to ﬁnd a meaningful joint latent code distribution bridging the two individual embeddings learned from x 1 and x 2 alone.

Original: This can be done by a two-step procedure that models the individual representations ﬁrst and then applies an additional learning step to link them (Tian & Engel, 2019; Silberer & Lapata, 2014; Ngiam et al., 2011).
Revised: This can be done by a two-step procedure that models the individual representations ﬁrst and then applies an additional learning step to link them (Tian & Engel, 2019; Silberer & Lapata, 2014; Ngiam et al., 2011).

Original: In contrast, we focus on approaches that learn individual and joint representations simultaneously.
Revised: In contrast, we focus on approaches that learn individual and joint representations simultaneously.

Original: Furthermore, our model should be able to learn in a semi-supervised setting.
Revised: Furthermore, our model should be able to learn in a semi-supervised setting.

Original: Kingma et al.
Revised: Kingma et al.

Original: (2014) introduced two models suitable for the case when one modality is high dimensional (e.g., an image) and another is low dimensional (e.g., a label) while our main interest are modalities of high complexity.
Revised: (2014) introduced two models suitable for the case when one modality is high dimensional (e.g., an image) and another is low dimensional (e.g., a label) while our main interest are modalities of high complexity.

Original: We consider models based on variational autoencoders (VAEs, Kingma & Welling, 2014; Rezende et al., 2014).
Revised: We consider models based on variational autoencoders (VAEs, Kingma & Welling, 2014; Rezende et al., 2014).

Original: Standard VAEs learn a latent representation z ∈ Z for a set of observed variables x ∈ X by modelling a joint distribution p ( x, z )
Revised: Standard VAEs learn a latent representation z ∈ Z for a set of observed variables x ∈ X by modelling a joint distribution p ( x, z )

Original: = p ( z ) p ( x | z ) .
Revised: = p ( z ) p ( x | z ) .

Original: In the original VAE, the intractable posterior q ( z | x ) and conditional distribution p ( x | z ) are approximated by neural networks trained by maximising the ELBO loss taking the form
Revised: In the original VAE, the intractable posterior q ( z | x ) and conditional distribution p ( x | z ) are approximated by neural networks trained by maximising the ELBO loss taking the form

Original: [Equation]
Revised: [Equation]

Original: with respect to the parameters of the networks modelling q ( z | x ) and p ( x | z ) .
Revised: with respect to the parameters of the networks modelling q ( z | x ) and p ( x | z ) .

Original: Here D KL ( · (cid:107) · ) denotes the Kullback-Leibler divergence.
Revised: Here D KL ( · (cid:107) · ) denotes the Kullback-Leibler divergence.

Original: Bi-modal VAEs that can handle a missing modality extend this approach by modelling q ( z | x 1 , x 2 ) as well as q 1 ( z | x 1 ) and q 2 ( z | x 2 ) , which replace the single q ( z | x ) .
Revised: Bi-modal VAEs that can handle a missing modality extend this approach by modelling q ( z | x 1 , x 2 ) as well as q 1 ( z | x 1 ) and q 2 ( z | x 2 ) , which replace the single q ( z | x ) .

Original: Multimodal VAEs may differ in 1) the way they approximate q ( z | x 1 , x 2 ) , q 1 ( z | x 1 ) and
Revised: Multimodal VAEs may differ in 1) the way they approximate q ( z | x 1 , x 2 ) , q 1 ( z | x 1 ) and

Original: q 2 ( z | x 2 ) by neural networks and/or 2) the structure of the loss function, see Figure 1.
Revised: q 2 ( z | x 2 ) by neural networks and/or 2) the structure of the loss function, see Figure 1.

Original: Typically, there are no conceptual differences in the decoding, and we model the decoding distributions in the same way for all methods considered in this study.
Revised: Typically, there are no conceptual differences in the decoding, and we model the decoding distributions in the same way for all methods considered in this study.

Original: Suzuki et al.
Revised: Suzuki et al.

Original: (2017) introduced a model termed JMVAE, which belongs to the class of approaches that can only learn from the paired training samples (what we refer to as the (fully) supervised setting ).
Revised: (2017) introduced a model termed JMVAE (Joint Multimodal VAE), which belongs to the class of approaches that can only learn from the paired training samples (what we refer to as the (fully) supervised setting ).

Original: It approximates q ( z | x 1 , x 2 ) , q 1 ( z | x 1 ) and q 2 ( z | x 2 ) with three corresponding neural networks and optimizes an ELBO-type loss of the form
Revised: It approximates q ( z | x 1 , x 2 ) , q 1 ( z | x 1 ) and q 2 ( z | x 2 ) with three corresponding neural networks and optimizes an ELBO-type loss of the form

Original: [Equation]
Revised: [Equation]

Original: The last two terms imply that during learning the joint network output must be generated which requires paired samples.
Revised: The last two terms imply that during learning the joint network output must be generated which requires paired samples.

Original: The MVAE  model (Wu & Goodman, 2018) is the ﬁrst multimodal VAE-based model allowing for missing modalities that does not require any additional network structures for learning the joint latent code distribution.
Revised: The MVAE (Multimodal VAE) model (Wu & Goodman, 2018) is the ﬁrst multimodal VAE-based model allowing for missing modalities that does not require any additional network structures for learning the joint latent code distribution.

Original: The joint posterior is modeled using a product-of-experts (PoE) as q ( z | x 1: M ) =
Revised: The joint posterior is modeled using a product-of-experts (PoE) as q ( z | x 1: M ) =

Original: (cid:81)
Revised: (cid:81)

Original: m
Revised: m

Original: q m ( z | x m ) .
Revised: q m ( z | x m ) .

Original: For the missing modality q k ( z | x k ) = 1 is assumed.
Revised: For the missing modality q k ( z | x k ) = 1 is assumed.

Original: The model allows for semisupervised learning while keeping the number of model parameters low.
Revised: The model allows for semi-supervised learning while keeping the number of model parameters low.

Original: The bridged model (Yadav et al., 2020) highlights the need for an additional network structure for approximating the joint latent code distribution.
Revised: The bridged model (Yadav et al., 2020) highlights the need for an additional network structure for approximating the joint latent code distribution.

Original: It attempts to keep the advantages of the additional encoding networks.
Revised: It attempts to keep the advantages of the additional encoding networks.

Original: It reduces the number of model parameters by introducing the bridge encoder that consists of one fully connected layer which takes z 1 and z 2 latent code vectors generated from x 1 and x 2 and outputs the mean and the variance of the joint latent code distribution.
Revised: It reduces the number of model parameters by introducing the bridge encoder that consists of one fully connected layer which takes z 1 and z 2 latent code vectors generated from x 1 and x 2 and outputs the mean and the variance of the joint latent code distribution.

Original: The arguably most advanced multimodal VAE models is VAEVAE by Wu & Goodman (2019), which we discuss in detail in the next section (see also Appendix C).
Revised: The arguably most advanced multimodal VAE models is VAEVAE by Wu & Goodman (2019), which we discuss in detail in the next section (see also Appendix C).

Original: Shi et al.
Revised: Shi et al.

Original: (2019) proposed a MoE model termed MMVAE.
Revised: (2019) proposed a MoE model termed MMVAE (Mixture-of-experts Multimodal VAE).

Original: In MMVAE model the joint variational posterior for M modalities is approximated as q ( z | x 1: M )=
Revised: In MMVAE model the joint variational posterior for M modalities is approximated as q ( z | x 1: M )

Original:  (cid:80) m α m q m ( z | x m ) where α m = 1 M .
Revised: = (cid:80) m α m q m ( z | x m ) where α m = 1 M .

Original: The model utilizes a loss function from the importance weighted autoencoder (IWAE, Burda et al., 2016) that computes a tighter lower bound compared to the VAE ELBO loss.
Revised: The model utilizes a loss function from the importance weighted autoencoder (IWAE, Burda et al., 2016) that computes a tighter lower bound compared to the VAE ELBO loss.

Original: The MoE rule formulation allows in principle to train with a missing modality i by assuming α i = 0 , however, Shi et al.
Revised: The MoE rule formulation allows in principle to train with a missing modality i by assuming α i = 0 , however, Shi et al.

Original: (2019) do not highlight or evaluate this feature.
Revised: (2019) do not highlight or evaluate this feature.

Original: There are benchmarks in the paper that compare MVAE (Wu & Goodman, 2018) and MMVAE, concluding that MVAE often fails to learn the joint latent code distribution.
Revised: There are benchmarks in the paper that compare MVAE (Wu & Goodman, 2018) and MMVAE, concluding that MVAE often fails to learn the joint latent code distribution.

Original: Because of these results and those presented by Wu & Goodman (2019), we did not include MVAE as a benchmark model in our experiments.
Revised: Because of these results and those presented by Wu & Goodman (2019), we did not include MVAE as a benchmark model in our experiments.

Original:  VAEVAE AND SVAE
Revised:  VAEVAE AND SVAE

Original: We developed a new approach as an alternative to VAEVAE (Wu & Goodman, 2019).
Revised: We developed a new approach as an alternative to VAEVAE.

Original: Both models 1) are VAE based; 2) allow for interchangeable cross-model generation as well as a learning joint embedding; 3) allow for missing modalities at training time; and 4) can be applied to two similarly complex high dimensional modalities.
Revised: Both models 1) are VAE based; 2) allow for interchangeable cross-model generation as well as a learning joint embedding; 3) allow for missing modalities at training time; and 4) can be applied to two similarly complex high dimensional modalities.

Original: Next, we will brieﬂy present our new model SVAE.
Revised: Next, we will brieﬂy present our new model SVAE.

Original: Then we highlight the differences to VAEVAE.
Revised: Then we highlight the differences to VAEVAE.

Original: Finally, we state a newly derived objective function for training the models.
Revised: Finally, we state a newly derived objective function for training the models.

Revised: We consider two modalities and refer to Appendix D for generalizations to more modalities.

Original: SVAE Since both modalities might not be available for all the samples, it should be possible to marginalize each of them out of q ( z | x 1 , x 2 ) .
Revised: SVAE Since both modalities might not be available for all the samples, it should be possible to marginalize each of them out of q ( z | x 1 , x 2 ) .

Original: While the individual encoding distributions q ( z | x 1 ) and q ( z | x 2 ) can be approximated by neural networks as in the standard VAE, we need to deﬁne a meaningful approximation of the joint encoding distribution q ( z | x ) =
Revised: While the individual encoding distributions q ( z | x 1 ) and q ( z | x 2 ) can be approximated by neural networks as in the standard VAE, we need to deﬁne a meaningful approximation of the joint encoding distribution q ( z | x ) =

Original: q ( z | x 1 , x 2 ) .
Revised: q ( z | x 1 , x 2 ) .

Original: In the newly proposed SVAE model, these distributions are deﬁned as the following:
Revised: In the newly proposed SVAE model, these distributions are deﬁned as the following:

Original: [Equation]
Revised: [Equation]

Original: The model is derived from an axiomatic proof that is given in Appendix A.
Revised: The model is derived from an axiomatic proof that is given in Appendix A.

Original: The desired properties of the model were that 1) when no modalities are observed the generating distribution for the latent code is Gaussian, 2) the modalities are independent given the latent code, 3) both experts cover the whole latent space with equal probabilities, and 4) the joint encoding distribution q ( z | x 1 , x 2 ) is modelled by a PoE.
Revised: The desired properties of the model were that 1) when no modalities are observed the generating distribution for the latent code is Gaussian, 2) the modalities are independent given the latent code, 3) both experts cover the whole latent space with equal probabilities, and 4) the joint encoding distribution q ( z | x 1 , x 2 ) is modelled by a PoE.

Original: The distributions q 1 ( z | x 1 ) , q 2 ( z | x 2 ) , q ∗ 2 ( z | x 1 ) and q ∗ 1 ( z | x 2 ) are approximated by neural networks.
Revised: The distributions q 1 ( z | x 1 ) , q 2 ( z | x 2 ) , q ∗ 2 ( z | x 1 ) and q ∗ 1 ( z | x 2 ) are approximated by neural networks.

Original: In case both observations are available, q ( z | x 1 , x 2 ) is approximated by applying the product-ofexperts rule with q 1 ( z | x 1 ) and q 2 ( z | x 2 ) being the experts for each modality.
Revised: In case both observations are available, q ( z | x 1 , x 2 ) is approximated by applying the product-ofexperts rule with q 1 ( z | x 1 ) and q 2 ( z | x 2 ) being the experts for each modality.

Original: In case of a missing modality, equation 4 or 5 is used.
Revised: In case of a missing modality, equation 4 or 5 is used.

Original: If, for example, x 2 is missing, the q ∗ 2 ( z | x 1 ) distribution takes over as a “replacement” expert, modelling marginalization over x 2 .
Revised: If, for example, x 2 is missing, the q ∗ 2 ( z | x 1 ) distribution takes over as a “replacement” expert, modelling marginalization over x 2 .

Original: SVAE vs. VAEVAE The VAEVAE model (Wu & Goodman, 2019) is the most similar to ours.
Revised: SVAE vs. VAEVAE The VAEVAE model (Wu & Goodman, 2019) is the most similar to ours.

Original: Wu & Goodman deﬁne two variants which can be derived from the SVAE model in the following way.
Revised: Wu & Goodman deﬁne two variants which can be derived from the SVAE model in the following way.

Original: Variant (a) can be derived by setting q ∗ ( z | x 1 ) =
Revised: Variant (a) can be derived by setting q ∗ ( z | x 1 ) =

Original: q ∗ ( z | x 2 ) = 1 .
Revised: q ∗ ( z | x 2 ) = 1 .

Original: Variant (b) is obtained from (a) by additionally using a separate network to model q ( z | x 1 , x 2 ) .
Revised: Variant (b) is obtained from (a) by additionally using a separate network to model q ( z | x 1 , x 2 ) .

Original: Having a joint network q ( z | x 1 , x 2 ) implements the most straightforward way of capturing the inter-dependencies of the two modalities.
Revised: Having a joint network q ( z | x 1 , x 2 ) implements the most straightforward way of capturing the inter-dependencies of the two modalities.

Original: However, the joint network cannot be trained on unpaired data – which can be relevant when the share of supervised data gets smaller.
Revised: However, the joint network cannot be trained on unpaired data – which can be relevant when the share of supervised data gets smaller.

Original: Option (a) uses the product-of-experts rule to model the joint distribution of the two modalities as well, but does not ensure that both experts cover the whole latent space (in contrast to SVAE, see equation A.14 in the appendix), which can lead to individual latent code distributions diverging.
Revised: Option (a) uses the product-of-experts rule to model the joint distribution of the two modalities as well, but does not ensure that both experts cover the whole latent space (in contrast to SVAE, see equation A.14 in the appendix), which can lead to individual latent code distributions diverging.

Original: Based on this consideration and the experimental results from Wu &
Revised: Based on this consideration and the experimental results from Wu &

Original: Goodman (2019), we focused on benchmarking VAEVAE (b) and refer to it as simply VAEVAE in
Revised: Goodman (2019), we focused on benchmarking VAEVAE (b) and refer to it as simply VAEVAE in

Original: Section 4.
Revised: Section 4.

Original: SVAE resembles VAEVAE in the need for additional networks besides one encoder per each modality and the structure of ELBO loss.
Revised: SVAE resembles VAEVAE in the need for additional networks besides one encoder per each modality and the structure of ELBO loss.

Original: It does, however, solve the problem of learning the joint embeddings in a way that allows to learn the parameters of approximated q ( z | x 1 , x 2 ) using all available samples, i.e., both paired and unpaired.
Revised: It does, however, solve the problem of learning the joint embeddings in a way that allows to learn the parameters of approximated q ( z | x 1 , x 2 ) using all available samples, i.e., both paired and unpaired.

Original: If q ( z | x 1 , x 2 ) is approximated with the joint network that accepts concatenated inputs, as in JMVAE and VAEVAE (b), the weights of q ( z | x 1 , x 2 ) can only be updated for the paired share of samples.
Revised: If q ( z | x 1 , x 2 ) is approximated with the joint network that accepts concatenated inputs, as in JMVAE and VAEVAE (b), the weights of q ( z | x 1 , x 2 ) can only be updated for the paired share of samples.

Original: If q ( z | x 1 , x 2 ) is approximated with a PoE of decoupled networks as in SVAE, the weights are updated for each sample whether paired or unpaired – which is the key differentiating feature of SVAE compared to existing architectures
Revised: If q ( z | x 1 , x 2 ) is approximated with a PoE of decoupled networks as in SVAE, the weights are updated for each sample whether paired or unpaired – which is the key differentiating feature of SVAE compared to existing architectures. A New

Original: [Equation] A New Objective FunctionWhen developing SVAE [Equation]
Revised: Objective Function

Original:  we devised a novel ELBO-type loss:
Revised: When developing SVAE, we devised a novel ELBO-type loss:

Original: [Equation]
Revised: [Equation]

Original: Here p paired and p unpaired denote the distributions of the paired and unpaired training data, respectively.
Revised: Here p paired and p unpaired denote the distributions of the paired and unpaired training data, respectively.

Original: The loss function is derived in Appendix B.
Revised: The loss function is derived in Appendix B.

Original: The differences between this loss and the loss function used to train VAEVAE by Wu & Goodman (2019) are highlighted in Appendix C.
Revised: The differences between this loss and the loss function used to train VAEVAE by Wu & Goodman (2019) are highlighted in Appendix C.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We conducted experiments to compare state-of-the-art PoE based
Revised: We conducted experiments to compare state-of-the-art PoE based

Original: VAEs with the MoE approach MMVAE (Shi et al., 2019).
Revised: VAEs with the MoE approach MMVAE (Shi et al., 2019).

Original: We considered VAEVAE (b) as proposed by Wu & Goodman (2019) and SVAE as described above.
Revised: We considered VAEVAE (b) as proposed by Wu & Goodman (2019) and SVAE as described above.

Original: The two approaches differ both in the underlying model as well as the objective function.
Revised: The two approaches differ both in the underlying model as well as the objective function.

Original: For a better understanding of these differences, we also considered an algorithm referred to as
Revised: For a better understanding of these differences, we also considered an algorithm referred to as

Original: VAEVAE*, which has the same model architecture as VAEVAE and the same loss function as SVAE.
Revised: VAEVAE*, which has the same model architecture as VAEVAE and the same loss function as SVAE.

Original: The difference in the training procedure for VAEVAE and VAEVAE* is described in Appendix C. Since the VAEVAE implementation was not publicly available at the time of writing, we used our own implementation of VAEVAE based on the PiXYZ library.
Revised: The difference in the training procedure for VAEVAE and VAEVAE* is described in Appendix C. Since the VAEVAE implementation was not publicly available at the time of writing, we used our own implementation of VAEVAE based on the PiXYZ library.

Original: For details about the experiments we refer to Appendix D.
Revised: For details about the experiments we refer to Appendix F.

Original: The source code to reproduce the experiments can be found in the supplementary material.
Revised: The source code to reproduce the experiments can be found in the supplementary material.

Original: For an unbiased evaluation, we considered the same test problems and performance metrics as Shi et al.
Revised: For an unbiased evaluation, we considered the same test problems and performance metrics as Shi et al.

Original: In addition, we designed an experiment referred to as MNIST-Split that was supposed to be well-suited for PoE.
Revised: In addition, we designed an experiment referred to as MNIST-Split that was supposed to be well-suited for PoE.

Original: In all experiments we kept the network architectures as similar as possible (see Appendix D).
Revised: In all experiments we kept the network architectures as similar as possible (see Appendix F).

Revised: For the new benchmark problem, we constructed a multi-modal dataset where the modalities are similar in dimensionality as well as complexity and are providing missing information to each other rather than duplicating it.

Revised: The latter should favor a PoE modelling, which suits an “AND” combination of the modalities, and not a MoE modeling, which is more aligned with an “OR” combination.

Original: We measuredmodel performance for different supervision levels for each dataset. For example, 10% supervision level means that 10% of the training set samples were paired and the remaining 90% were unpaired.
Revised: We measured performance for different supervision levels for each dataset (e.g., 10% supervision level means that 10% of the training set samples were paired and the remaining 90% were unpaired).

Original: Image and image: MNIST-Split
Revised: Image and image: MNIST-Split

Original: We constructed a bi-modal dataset where the two modalities are similar in dimensionality as well as complexity and are providing missing information to each other rather than duplicating it.
Revised: We created an image reconstruction dataset based on MNIST digits (LeCun et al., 1998). The images were split horizontally into equal parts, either two or three depending on the experimental setting. These regions are considered as different input modalities.

Original:  The latter should favor a PoE modelling, which suits an “AND”combination of the modalities, andnot a MoE modeling, which is more aligned with an “OR” combination. We created an image reconstruction dataset based on MNIST digits (LeCun et al., 1998).
Revised: In the above notion of “AND” and “OR” tasks we implicitly assume an additional modality, which is the image label in this case. The fact that the correct digit can sometimes be guessed from only one part of the image makes the new MNIST-Split benchmark a mixture of an “AND” and an “OR” task. This is in contrast to the MNIST-SVHN task described below, which can be regarded as an almost pure “OR” task.

Original: The MNIST images were split in top and bottom halves of equal size, and the halves were then used as two modalities.
Revised: In the bi-modal version referred to as MNIST-Split, the MNIST images were split in top and bottom halves of equal size, and the halves were then used as two modalities.

Original: We tested the quality of the image reconstruction given one or both modalities by predicting the reconstructed image label with an independent oracle network, a ResNet-18 (He et al., 2016) trained on the original MNIST dataset.
Revised: We tested the quality of the image reconstruction given one or both modalities by predicting the reconstructed image label with an independent oracle network, a ResNet-18 (He et al., 2016) trained on the original MNIST dataset.

Original: The evaluation metrics were joint coherence , synergy , and cross-coherence .
Revised: The evaluation metrics were joint coherence , synergy , and cross-coherence .

Original: For measuring joint coherence, 1000 latent space vectors were generated from the prior and both halves of an image were then reconstructed with the corresponding decoding networks.
Revised: For measuring joint coherence, 1000 latent space vectors were generated from the prior and both halves of an image were then reconstructed with the corresponding decoding networks.

Original: The concatenated halves yield the fully reconstructed image.
Revised: The concatenated halves yield the fully reconstructed image.

Original: Since the ground truth class labels do not exist for the randomly sampled latent vectors, we could only perform a qualitative evaluation, see Figure 3.
Revised: Since the ground truth class labels do not exist for the randomly sampled latent vectors, we could only perform a qualitative evaluation, see Figure 3.

Original: Synergy was deﬁned as the accuracy of the image reconstruction given both halves.
Revised: Synergy was deﬁned as the accuracy of the image reconstruction given both halves.

Original: Cross-coherence considered the reconstruction of the full image from one half and was deﬁned as the fraction of class labels correctly predicted by the oracle network.
Revised: Cross-coherence considered the reconstruction of the full image from one half and was deﬁned as the fraction of class labels correctly predicted by the oracle network.

Original: [Table]
Revised: [Table]

Original: The quantitative results are shown in Table 1 and Figure 4.
Revised: The quantitative results are shown in Table 1 and Figure 4.

Original: All PoE architectures clearly outperformed MMVAE even when trained on the low supervision levels.
Revised: All PoE architectures clearly outperformed MMVAE even when trained on the low supervision levels.

Original: In this experiment, it is important that both experts agree on a class label.
Revised: In this experiment, it is important that both experts agree on a class label.

Original: Thus, as expected, the multiplicative PoE ﬁts the task much better than the additive mixture.
Revised: Thus, as expected, the multiplicative PoE ﬁts the task much better than the additive mixture.

Original: Utilizing the novel loss function (10) gave the best results for very low supervision (SVAE and 
Revised: Utilizing the novel loss function (10) gave the best results for very low supervision (SVAE and VAEVAE*).

Revised: T HREE MODALITIES : MNIST-S PLIT -3.

Original: VAEVAE*).
Revised: We compared a simple generalization of the SVAE model to more than two modalities with the canonical extension of the VAEVAE model (both deﬁned in Appendix D) on the MNIST-Split-3 data, the 3-modal version of MNIST-Split task. Figure shows that SVAE performed better when looking at the individual modalities reconstructions. While the number of parameters in the bi-modal case is the same for SVAE and VAEVAE, it grows exponentially for VAEVAE and stays in order of n 2 for SVAE where n is the number of modalities, see Figure 5 and Appendix D for details.

Original: Image and image: MNIST-SVHN
Revised: Image and image: MNIST-SVHN

Original: The ﬁrst dataset considered by Shi et al.
Revised: The ﬁrst dataset considered by Shi et al.

Original: (2019) is constructed by pairing MNIST and SVHN (Netzer et al., 2011) images showing the same digit.
Revised: (2019) is constructed by pairing MNIST and SVHN (Netzer et al., 2011) images showing the same digit.

Original: This dataset shares some properties with MNIST-Split, but the relation between the two modalities is different: the digit class is derived from a concatenation of two modalities in MNIST-Split, while in MNISTSVHN it could be derived from any modality alone, which corresponds to ”AND” combination of the modalities and favors the MoE architecture.
Revised: This dataset shares some properties with MNIST-Split, but the relation between the two modalities is different: the digit class is derived from a concatenation of two modalities in MNIST-Split, while in MNISTSVHN it could be derived from any modality alone, which corresponds to ”AND” combination of the modalities and favors the MoE architecture.

Original: As before, oracle networks are trained to predict the digit classes of MNIST and SVHN images.
Revised: As before, oracle networks are trained to predict the digit classes of MNIST and SVHN images.

Original: Joint coherence was again computed based on 1000 latent space vectors generated from the prior.
Revised: Joint coherence was again computed based on 1000 latent space vectors generated from the prior.

Original: Both images were then reconstructed with the corresponding decoding networks.
Revised: Both images were then reconstructed with the corresponding decoding networks.

Original: A reconstruction was considered correct if the predicted digit classes of MNIST and SVHN were the same.
Revised: A reconstruction was considered correct if the predicted digit classes of MNIST and SVHN were the same.

Original: Cross-coherence was measured as above.
Revised: Cross-coherence was measured as above.

Original: Figure 2 shows examples of paired image reconstructions from the randomly sampled latent space of the fully supervised VAEVAE model.
Revised: Figure 2 shows examples of paired image reconstructions from the randomly sampled latent space of the fully supervised VAEVAE model.

Original: The digit next to the each reconstruction shows the digit class prediction for this image.
Revised: The digit next to the each reconstruction shows the digit class prediction for this image.

Original: The quantitative results in Figure 5 show that all three PoE based models reached a similar joint coherence as MMVAE, VAEVAE scored even higher.
Revised: The quantitative results in Figure 7 show that all three PoE based models reached a similar joint coherence as MMVAE, VAEVAE scored even higher.

Original: The cross-coherenceresults were best for MMVAE, but the three PoE based models performed considerably better than the MVAE baseline reported by Shi et al.
Revised: The cross-coherence

Original: Image and text: CUB-Captions
Revised: Image and text: CUB-Captions

Original: The second benchmark considered by Shi et al.
Revised: The second benchmark considered by Shi et al.

Original: (2019) is the
Revised: (2019) is the

Original: CUB Images-Captions dataset (Wah et al., 2011)
Revised: CUB Images-Captions dataset (Wah et al., 2011)

Original: containing photos of birds and their textual descriptions.
Revised: containing photos of birds and their textual descriptions.

Original: Here the modalities are of different nature but similar in dimensionality and information content.
Revised: Here the modalities are of different nature but similar in dimensionality and information content.

Original: We used the source code 3 by Shi et al.
Revised: We used the source code 3 by Shi et al.

Original: to compute the same evaluation metrics as in the MMVAE study.
Revised: to compute the same evaluation metrics as in the MMVAE study.

Original: Canonical correlation analysis (CCA) was used for estimating joint and crosscoherences of images and text (Massiceti et al., 2018).
Revised: Canonical correlation analysis (CCA) was used for estimating joint and crosscoherences of images and text (Massiceti et al., 2018).

Original: The projection matrices W x for images and
Revised: The projection matrices W x for images and

Original: W y for captions were pre-computed using the training set of CUB Images-Captions and are available as part of the source code.
Revised: W y for captions were pre-computed using the training set of CUB Images-Captions and are available as part of the source code.

Original: Given a new image-caption pair ˜ x, ˜ y , we computed the correlation between the two by corr( x,  y )=
Revised: Given a new image-caption pair ˜ x, ˜ y , we computed the correlation between the two by corr(˜ x, ˜ y )

Original: [Equation]
Revised: =

Original:  (˜ y ) (cid:107) , where φ (˜ k ) =
Revised: φ (˜ x ) T φ (˜ y ) (cid:107) φ (˜ x ) (cid:107)(cid:107) φ (˜ y ) (cid:107) , where φ (˜ k ) =

Original: W Tk ˜ k − avg
Revised: W Tk ˜ k − avg

Original: ( W Tk k ) .
Revised: ( W Tk k ) .

Original: We employed the same image generation procedure as in the MMVAE study.
Revised: We employed the same image generation procedure as in the MMVAE study.

Original: Instead of creating the images directly, we generated 2048-d feature vectors using a pre-trained ResNet-101.
Revised: Instead of creating the images directly, we generated 2048-d feature vectors using a pre-trained ResNet-101.

Original: In order to ﬁnd the resulting image, a nearest neighbours lookup with Euclidean distance was performed.
Revised: In order to ﬁnd the resulting image, a nearest neighbours lookup with Euclidean distance was performed.

Original: A
Revised: A

Original: CNN encoder and decoder was used for the (see Table D.5 and Table D.6).
Revised: CNN encoder and decoder was used for the (see Table F.5 and Table F.6).

Original: Prior to computing the correlations, the captions were converted to 300-d vectors using FastText (Bojanowski et al., 2017).
Revised: Prior to computing the correlations, the captions were converted to 300-d vectors using FastText (Bojanowski et al., 2017).

Original: As in the experiment before, we used the same network architectures and hyperparameters as Shi et al.
Revised: As in the experiment before, we used the same network architectures and hyperparameters as Shi et al.

Original: We sampled 1000 latent space vectors from the prior distribution.
Revised: We sampled 1000 latent space vectors from the prior distribution.

Original: Images and captions were then reconstructed with the decoding networks.
Revised: Images and captions were then reconstructed with the decoding networks.

Original: The joint coherence was then computed as the CCA for the resulting image and caption averaged over the 1000 samples.
Revised: The joint coherence was then computed as the CCA for the resulting image and caption averaged over the 1000 samples.

Original: Cross-coherence was computed from caption to image and vice versa using the CCA averaged over the whole test set.
Revised: Cross-coherence was computed from caption to image and vice versa using the CCA averaged over the whole test set.

Original: As can be seen in Figure 6, VAEVAE showed the best performance among all models.
Revised: As can be seen in Figure 8, VAEVAE showed the best performance among all models.

Original: With full supervision the VAEVAE model outperformed MMVAE in all three metrics.
Revised: With full supervision the VAEVAE model outperformed MMVAE in all three metrics.

Original: The cross-coherence of the three PoE models was higher or equal to MMVAE except for very low supervision levels.
Revised: The cross-coherence of the three PoE models was higher or equal to MMVAE except for very low supervision levels.

Original: All three PoE based models were consistently better than MVAE.
Revised: All three PoE based models were consistently better than MVAE.

Original:  D ISCUSSION AND C ONCLUSIONS
Revised:  D ISCUSSION AND C ONCLUSIONS

Original: We studied bi-modal variational autoencoders (VAEs) based on a product-of-experts (PoE)architecture, in particular VAEVAE as proposed by Wu & Goodman (2019) and a new model SVAE, which we derived in an axiomatic way, and represents a generalization of the VAEVAE architecture.
Revised: We studied bi-modal variational autoencoders (VAEs) based on a product-of-experts (PoE)architecture, in particular VAEVAE as proposed by Wu & Goodman (2019) and a new model SVAE, which we derived in an axiomatic way, and represents a generalization of the VAEVAE architecture.

Original: The models learn representations that allow coherent sampling of the modalities and accurate sampling of one modality given the other.
Revised: The models learn representations that allow coherent sampling of the modalities and accurate sampling of one modality given the other.

Original: They work well in the semi-supervised setting, that is, not all modalities need to be always observed during training.
Revised: They work well in the semi-supervised setting, that is, not all modalities need to be always observed during training.

Original: It has been argued that the mixture-of-experts (MoE) approach MMVAE is preferable to a PoE for multimodal VAEs (Shi et al., 2019), in particular in the fully supervised setting (i.e., when all data are paired).
Revised: It has been argued that the mixture-of-experts (MoE) approach MMVAE is preferable to a PoE for multimodal VAEs (Shi et al., 2019), in particular in the fully supervised setting (i.e., when all data are paired).

Original: This conjecture was based on a comparison with the MVAE model (Wu & Goodman, 2018), but is refuted by our experiments showing that VAEVAE and our newly proposed SVAE can outperform MMVAE on experiments conducted by Shi et al.
Revised: This conjecture was based on a comparison with the MVAE model (Wu & Goodman, 2018), but is refuted by our experiments showing that VAEVAE and our newly proposed SVAE can outperform MMVAE on experiments conducted by Shi et al.

Original: Intuitively, MoEs are more tailored towards an “OR” (additive) combination of the information provided by thedifferent modalities,while PoEs are more tailoredto towards an “AND” (multiplicative) combination.
Revised: Intuitively, MoEs are more tailored towards an “OR” (additive) combination of the information provided by the modalities, are more tailored an “AND” (multiplicative) combination.

Original: This is demonstrated by our experiments on halved digit images, where a conjunctive combination is helpful and the PoE models perform much better than MMVAE. 
Revised: This is demonstrated by our experiments on halved digit images where a conjunctive combination is helpful and the PoE models perform much better than MMVAE. We also expand SVAE and

Revised: VAEVAE to case and show that SVAE demonstrates better performance on having

Original: A D ERIVATION OF THE MODEL ARCHITECTURE
Revised: A D ERIVATION OF THE MODEL ARCHITECTURE

Original: We deﬁne our model in an axiomatic way, requiring the following properties:
Revised: We deﬁne our model in an axiomatic way, requiring the following properties:

Original: When no modalities are observed, the generating distribution for the latent code is Gaussian:
Revised: When no modalities are observed, the generating distribution for the latent code is Gaussian:

Original: [Equation]
Revised: [Equation]

Original: This property is well known from VAEs and allows easy sampling.
Revised: This property is well known from VAEs and allows easy sampling.

Original: The two modalities are independent given the latent code, so the decoder distribution is:
Revised: The two modalities are independent given the latent code, so the decoder distribution is:

Original: [Equation]
Revised: [Equation]

Original: The second property formalizes our goal that the latent representation contains all relevant information from all modalities.
Revised: The second property formalizes our goal that the latent representation contains all relevant information from all modalities.

Original: The joint distribution p ( z | x ) [Equation] p ( z | x 1 , x 2 ) is given by
Revised: The joint distribution p ( z | x ) = p ( z | x 1 , x 2 ) is given by

Original: [Equation]
Revised: [Equation]

Original: Both experts cover the whole latent space with equal probabilities:
Revised: Both experts cover the whole latent space with equal probabilities:

Original: [Equation]
Revised: [Equation]

Original: The joint encoding distribution q ( z | x ) 
Revised: The joint encoding distribution q ( z | x ) =

Original: [Equation]

Original: q ( z | x 1 , x 2 ) is assumed to be given by the productof-experts rule (Hinton, 2002; Welling, 2007):
Revised: q ( z | x 1 , x 2 ) is assumed to be given by the productof-experts rule (Hinton, 2002; Welling, 2007):

Original: [Equation]
Revised: [Equation]

Original:  with Z ( x 1 , x 2 )=
Revised: 15) with Z ( x 1 , x 2 )

Original:  (cid:82) q 

Revised: = (cid:82) q

Original: ( z (cid:48) | x 1 ) q 2 ( z (cid:48) | x 2 )
Revised: ( z (cid:48) | x 1 ) q 2 ( z (cid:48) | x 2 )

Original: d z (cid:48) .
Revised: d z (cid:48) .

Original: The modelling by a product-of-experts in equation A.15 is a simpliﬁcation of equation A.13 to make the model tractable.
Revised: The modelling by a product-of-experts in equation A a simpliﬁcation of A tractable.

Original: Given equation A.15 and equation A.14 we obtain
Revised: Given equation A.15 and equation A.14 we obtain

Original: [Equation]
Revised: [Equation]

Original: Let us deﬁne
Revised: Let us deﬁne

Original: [Equation]
Revised: [Equation]

Original: and write
Revised: and write

Original: [Equation]
Revised: [Equation]

Original: So the proposal distributions are:
Revised: So the proposal distributions are:

Original: [Equation]
Revised: [Equation]

Original: B D ERIVATION OF THE LOSS FUNCTION
Revised: B D ERIVATION OF THE LOSS FUNCTION

Original: In the following
Revised: In the following

Original: [Equation]
Revised: [Equation]

Original: we derive the ELBO-type loss we use for training [Equation] see equation 7.
Revised: we derive the ELBO-type loss we use for training, see equation 7.

Original: Let consider the optimization
Revised: Let consider the optimization

Original: [Equation]
Revised: [Equation]

Original: We can now proceed by ﬁnding lower-bounds for each term.
Revised: We can now proceed by ﬁnding lower-bounds for each term.

Original: For the last two terms log p ( x i ) we can use the standard ELBO as given in equation 1.
Revised: For the last two terms log p ( x i ) we can use the standard ELBO as given in equation 1.

Original: This gives the terms
Revised: This gives the terms

Original: [Equation]
Revised: [Equation]

Original: Next [Equation] we will derive log p ( x 1 | x 2 ) .
Revised: Next, we will derive log p ( x 1 | x 2 ) .

Original: This we can do in terms of a conditional VAE (Sohn et al., 2015), where we condition all terms on x 2 (or x 1 if we model log p ( x 2 | x 1 ) ).
Revised: This we can do in terms of a conditional VAE (Sohn et al., 2015), where we condition all terms on x 2 (or x 1 if we model log p ( x 2 | x 1 ) ).

Original: So the model we derive the log-likelihood for is p ( x 1 | x 2 ) =
Revised: So the model we derive the log-likelihood for is p ( x 1 | x 2 ) =

Original: (cid:82) p ( x 1 | z ) p ( z | x 2 ) dz , where p ( z | x 2 ) is now our prior.
Revised: (cid:82) p ( x 1 | z ) p ( z | x 2 ) dz , where p ( z | x 2 ) is now our prior.

Original: By model assumption we further have p ( x 1 , x 2 , z ) =
Revised: By model assumption we further have p ( x 1 , x 2 , z ) =

Original: p ( x 1 | z ) p ( x 2 | z ) p ( z ) and therefore p ( x 1 | x 2 , z ) = p ( x 1 | z ) .
Revised: p ( x 1 | z ) p ( x 2 | z ) p ( z ) and therefore p ( x 1 | x 2 , z ) = p ( x 1 | z ) .

Original: Thus we arrive at the ELBO losses
Revised: Thus we arrive at the ELBO losses

Original: [Equation]
Revised: [Equation]

Original: 25) and
Revised: 25) and

Original: [Equation]
Revised: [Equation]

Original: We now insert the terms in equation B.23 and arrive at:
Revised: We now insert the terms in equation B.23 and arrive at:

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst two terms together give
Revised: The ﬁrst two terms together give

Original: [Equation]
Revised: [Equation]

Revised: B.

Original:  We do not know the conditional prior p ( z | x i ) .
Revised: [Equation] We do not know the conditional prior p ( z | x i ) .

Original: By deﬁnition of the VAE, we are allowed to optimize the prior, therefore we can parameterize it and optimize it.
Revised: By deﬁnition of the VAE, we are allowed to optimize the prior, therefore we can parameterize it and optimize it.

Original: However, we know that in an optimal model p ( z | x i )
Revised: However, we know that in an optimal model p ( z | x i )

Original: ≈
Revised: ≈

Original: q ( z | x i )
Revised: q ( z | x i )

Original: and it might be possible to prove that if p ( z | x i ) is learnt in the same model-class as q ( z | x i ) we can ﬁnd that the optimum is indeed p ( z | x i ) =
Revised: and it might be possible to prove that if p ( z | x i ) is learnt in the same model-class as q ( z | x i ) we can ﬁnd that the optimum is indeed p ( z | x i ) =

Original: q ( z | x i ) .
Revised: q ( z | x i ) .

Original: Inserting this choice into the equation gives the end-result.
Revised: Inserting this choice into the equation gives the end-result.

Original: C T RAINING PROCEDURE FOR SVAE AND VAEVAE*
Revised: C T RAINING PROCEDURE FOR SVAE AND VAEVAE*

Original: Input
Revised: Input: Supervised example ( x 1 , x 2 ) , Unsupervised example x (cid:48) 1 , Unsupervised example x (cid:48) [Equation] log

Original: [Equation]
Revised: [Equation]

Revised: D SVAE AND VAEVAE FOR MORE THAN TWO MODALITIES

Original: (cid:48)
Revised: In the following, we formalize the VAEVAE model for three modalities and present a na ¨ ıve extension of the SVAE model to more than two modalities. In the canonical extension of VAEVAE to three modalities, the three- and two-modal relations are captured by the corresponding networks q ( z | x 1 , x 2 , x 3 ) , q ( z | x i , x j ) and q ( z | x i ) for i, j ∈ { 1 , 2 , 3 } , see Figure 5. In the general n -modal case, the model has 2 n networks. For n = 3 , the loss function reads:

Original: [Equation]
Revised: [Equation]

Revised: In this study, we considered a simplifying extension of SVAE to n modalities using n 2 networks q j i ( z | x j ) for i, j ∈ { 1 , . . . , n } .

Revised: For the 3-modal case depicted in Figure 5, the PoE relations between the modalities are deﬁned in the following way:

Original: [Equation]
Revised: [Equation]

Revised: The corresponding SVAE loss function has additional terms due to the fact that the relations between pairs of modalities need to be captured with two PoE rules q i ( z | x i , x j ) and q i ( z | x i , x j ) in SVAE, while there is only a single network q ( z | x i , x j ) in VAEVAE.

Revised: The loss functions equation D.28– equation D.31 above are modiﬁed in a way that f ( q ( z | x i , x j )) =

Revised: f ( q i

Original: [Equation]
Revised: [Equation]

Revised: z | x i , x j ))

Revised: + f ( q j ( z | x i , x j )) for any function f .

Revised: This extension of the bi-modal case assumes that p ( x i , x j | x k ) =

Revised: p ( x i | x k ) p ( x j | x k ) for i, j, k ∈ { 1 , 2 , 3 } , i (cid:54) =

Revised: j (cid:54) =

Revised: k , which implies that x i , x j and x k are independent of each other.

Revised: E Q UALITATIVE EXAMPLES E

Revised: SVAE

Revised: See Figure E.

Original: D T RAINING D ETAILS AND E NCODER /D ECODER A RCHITECTURES
Revised: E. VAEVAE See Figure E. F T RAINING D ETAILS AND E NCODER /D ECODER A RCHITECTURES

Original: The encoder and decoder architectures for each experiment and modality are listed below.
Revised: The encoder and decoder architectures for each experiment and modality are listed below.

Original: To implement joint encoding network (VAEVAE architecture), an fully connected layer followed by ReLU is added to the encoding architecture for each modality.
Revised: To implement joint encoding network (VAEVAE architecture), an fully connected layer followed by ReLU is added to the encoding architecture for each modality.

Original: Another fully connected layer accepts the concatenated features from the two modalities as an input and outputs the latent space parameters.
Revised: Another fully connected layer accepts the concatenated features from the two modalities as an input and outputs the latent space parameters.

Original: Adam optimiser is used for learning in all the models (Kingma & Ba, 2015).
Revised: Adam optimiser is used for learning in all the models (Kingma & Ba, 2015).

Original: D. MNIST-S PLIT
Revised: F. MNIST-S PLIT

Original: The models are trained for 200 epochs with the learning rate 2 · 10 − 4 .
Revised: The models are trained for 200 epochs with the learning rate 2 · 10 − 4 .

Original: The best epoch is chosen by the highest accuracy of the reconstruction from the top half evaluated on the validation set.
Revised: The best epoch is chosen by the highest accuracy of the reconstruction from the top half evaluated on the validation set.

Original: We used a latent space dimensionality of L = 64 .
Revised: We used a latent space dimensionality of L = 64 .

Original: The network architectures are described in Table D.2.
Revised: The network architectures are described in Table F.7. Encoder

Original: [Table]
Revised: [Table]

Original: D. MNIST-SVHN

Original:  The models are trained for 50 epochs with the learning rate 10 − 3 for VAEVAE and VAEVAE* and 10 − 4 for SVAE.
Revised: [Equation] The models are trained for 50 epochs with the learning rate 10 − 3 for VAEVAE and VAEVAE 10 − 4 for SVAE.

Original: The best epoch is chosen by the highest joint coherence evaluated on the validation set.
Revised: The best epoch is chosen by the highest joint coherence evaluated on the validation set.

Original: We used a latent space dimensionality of L = 20 .
Revised: We used a latent space dimensionality of L = 20 .

Original: The network architectures are summarized in
Revised: The network architectures are summarized in

Original: Table D.4 and Table D.3 for the MNIST and SVHN modality, respectively.
Revised: Table F.4 and Table F.3 for the MNIST and SVHN modality, respectively.

Original: [Table]
Revised: [Table]

Original: [Table] The models are trained for 200 epochs with the learning rate 10 − 4 .
Revised: The models are trained for 200 epochs with the learning rate 10 − 4 .

Original: The best epoch is chosen by the highest joint coherence evaluated on the validation set.
Revised: The best epoch is chosen by the highest joint coherence evaluated on the validation set.

Original: We used a latent space dimensionality of
Revised: We used a latent space dimensionality of

Original: L = 64 .
Revised: L = 64 .

Original: The network architectures are described in Table D.5 and Table D.6 for the text and image modality, respectively.
Revised: The network architectures are described in Table F.5 and Table F.6 for the text and image modality, respectively.

Original: [Table]
Revised: The models are trained for 50 epochs with the learning rate 2 · 10 − 4 .

Original: [Table]
Revised: The best epoch is chosen by the highest accuracy of the reconstruction from the top half evaluated on the validation set.

Original: & BatchNorm2d &
Revised: We used a latent space dimensionality of L = 64 .

Original: [Table]
Revised: The network architectures are described in Table F.7.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Word Emb.

Original: [Table] [Table]
Revised: [Table] [Table]