Original: D EEP
Revised: D EEP

Original: G ENERATIVE
Revised: G ENERATIVE

Original: M ODELS
Revised: M ODELS

Original: FOR
Revised: FOR

Original: LEARNING
Revised: LEARNING

Original: C OHERENT
Revised: C OHERENT

Original: L ATENT
Revised: L ATENT

Original: R EPRESENTATIONS
Revised: R EPRESENTATIONS

Original: FROM
Revised: FROM

Original: M ULTIM ODAL D ATA
Revised: M ULTIM ODAL D ATA

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The application of multi-modal generative models by means of a Variational Auto
Revised: The application of multi-modal generative models by means of a Variational Auto

Original: Encoder (VAE) is an upcoming research topic for sensor fusion and bi-directional modality exchange.
Revised: Encoder (VAE) is an upcoming research topic for sensor fusion and bi-directional modality exchange.

Original: This contribution gives insights into the learned joint latent representation and shows that expressiveness and coherence are decisive properties for multi-modal data sets.
Revised: This contribution gives insights into the learned joint latent representation and shows that expressiveness and coherence are decisive properties for multi-modal datasets.

Original: Furthermore, we propose a multi-modal VAE derived from the full joint marginal log-likelihood that is able to learn the most meaningful representation for ambiguous observations.
Revised: Furthermore, we propose a multi-modal VAE derived from the full joint marginal log-likelihood that is able to learn the most meaningful representation for ambiguous observations.

Original: Since the properties of multi-modal sensor setups are essential for our approach but hardly available, we also propose a technique to generate correlated data sets from uni-modal ones.
Revised: Since the properties of multi-modal sensor setups are essential for our approach but hardly available, we also propose a technique to generate correlated datasets from uni-modal ones.

Original:  INTRODUCTION
Revised:  INTRODUCTION

Original: Auto Encoder (AE), Variational Auto Encoder (VAE), and more recently Disentangled Variational
Revised: Auto Encoder (AE), Variational Auto Encoder (VAE), and more recently Disentangled Variational

Original: Auto Encoder ( β -VAE) have a considerable impact on the ﬁeld of data driven leaning of generative models.
Revised: Auto Encoder ( β -VAE) have a considerable impact on the ﬁeld of data-driven leaning of generative models.

Original: Furthermore, recent investigations have shown the fruitful applicability to deep reinforcement learning (DRL) as well as bi-directionally exchange of multi-modal data.
Revised: Furthermore, recent investigations have shown the fruitful applicability to deep reinforcement learning (DRL) as well as bi-directionally exchange of multi-modal data.

Original: VAEs tend to encode the data into latent space features that are (ideally) linearly separable as shown by Higgins et al.
Revised: VAEs tend to encode the data into latent space features that are (ideally) linearly separable as shown by Higgins et al.

Original: (2017a).
Revised: (2017a).

Original: They also allow the discovery of generative joint models (e.g. Suzuki et al.
Revised: They also allow the discovery of generative joint models (e.g. Suzuki et al.

Original: (2017)), as well as zero-shot domain transfer in DRL as shown by Higgins et al.
Revised: (2017)), as well as zero-shot domain transfer in DRL as shown by Higgins et al.

Original: (2017b).
Revised: (2017b).

Original: However, a good generative model should not just generate good data and achieve a good quantitative score, but also gives a coherent and expressive latent space representation.
Revised: However, a good generative model should not just generate good data and achieve a good quantitative score, but also gives a coherent and expressive latent space representation.

Original: This property is decisive for multi-modal approaches if the data shows correlation, as it is the case for every sensor setup designed for sensor fusion.
Revised: This property is decisive for multi-modal approaches if the data shows correlation, as it is the case for every sensor setup designed for sensor fusion.

Original: With this contribution we investigate the characteristic of the latent space as well as the quantitative features for existing multi-modal VAEs.
Revised: With this contribution, we investigate the characteristic of the latent space as well as the quantitative features for existing multi-modal VAEs.

Original: Furthermore, we propose a novel approach to build and train a novel multi-modal VAE (M 2 VAE) which comprises the complete marginal joint log-likelihood without simplifying assumptions.
Revised: Furthermore, we propose a novel approach to build and train a novel multi-modal VAE (M 2 VAE) which comprises the complete marginal joint log-likelihood without simplifying assumptions.

Original: As our objective is the consideration of raw multi-modal sensor data, we also propose an approach to generate correlated multi-modal data sets from available uni-modal ones.
Revised: As our objective is the consideration of raw multi-modal sensor data, we also propose an approach to generate correlated multi-modal datasets from available uni-modal ones.

Original: Lastly, we draw connections to in-place sensor fusion and epistemic (ambiguity-resolving) active-sensing.
Revised: Lastly, we draw connections to in-place sensor fusion and epistemic (ambiguity-resolving) active-sensing.

Original: Section 2 comprises the related work on multi-modal VAEs.
Revised: Section 2 comprises the related work on multi-modal VAEs.

Original: Our comprehensive approach (i.e.
Revised: Our comprehensive approach (i.e.

Original: M 2 VAE) is given in Sec.
Revised: M 2 VAE) is given in Sec.

Original: Furthermore, we describe multi-modal data sets as well as the generation of correlated sets in Sec. 4 which are evaluated in Sec. 5.
Revised: Furthermore, we describe multi-modal datasets as well as the generation of correlated sets in Sec. 4 which are evaluated in Sec. 5.

Original: Finally, we conclude our work in Sec. 6.
Revised: Finally, we conclude our work in Sec. 6.

Original:  RELATED WORK
Revised:  RELATED WORK

Original: Variational auto encoder (VAE) combine neural networks with variational inference to allow unsupervised learning of complicated distributions according to the graphical model shown in Figure(left).
Revised: Variational auto encoder (VAE) combine neural networks with variational inference to allow unsupervised learning of complicated distributions according to the graphical model shown in Figure(left).

Original: A D a -dimensional observation a is modeled in terms of a D z -dimensional latent vector z using a probabilistic decoder p θ a ( z ) with parameters θ .
Revised: A D a -dimensional observation a is modeled in terms of a D z -dimensional latent vector z using a probabilistic decoder p θ a ( z ) with parameters θ .

Original: To generate the corresponding embedding z from observation a , a probabilistic encoder network with q φ a ( z ) is being provided which parametrizes the posterior distribution from which z is sampled.
Revised: To generate the corresponding embedding z from observation a , a probabilistic encoder network with q φ a ( z ) is being provided which parametrizes the posterior distribution from which z is sampled.

Original: The encoder and decoder, given by neural networks, are trained jointly to bring a close to an a ′ under the constraint that an approximate distribution needs to be close to a prior p ( z ) and hence inference is basically learned during training.
Revised: The encoder and decoder, given by neural networks, are trained jointly to bring a close to an a ′ under the constraint that an approximate distribution needs to be close to a prior p ( z ) and hence inference is basically learned during training.

Original: a z a ′ z a b a ′ b ′ q φ a q φ b p θ a p θ b q φ ab p θ a q φ abc z a c a ′ c ′ b ′
Revised: a z a ′ z a b a ′ b ′ q φ a q φ b p θ a p θ b q φ ab p θ a q φ abc z a c a ′ c ′ b ′

Original: b q φ ac q φ a q φ c p θ b p θ c q φ b q φ ab q φ bc
Revised: b q φ ac q φ a q φ c p θ b p θ c q φ b q φ ab q φ bc

Original: The speciﬁc objective of VAEs is the maximization of the marginal distribution p ( a ) =
Revised: The speciﬁc objective of VAEs is the maximization of the marginal distribution p ( a ) =

Original: R p θ ( a | z ) p ( z ) d a .
Revised: R p θ ( a | z ) p ( z ) d a .

Original: Because this distribution is intractable, the model is instead trained via stochastic gradient variational Bayes (SGVB) by maximizing the evidence lower bound (ELBO)
Revised: Because this distribution is intractable, the model is instead trained via stochastic gradient variational Bayes (SGVB) by maximizing the evidence lower bound (ELBO)

Original: L of the marginal log-likelihood log p ( a ) :
Revised: L of the marginal log-likelihood log p ( a ) :

Original: = L a as
Revised: = L a as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: This approach proposed by Kingma & Welling (2013) is used in settings where only a single modality a is present in order to ﬁnd a latent encoding z (c.f. Figure 1 (left)).
Revised: This approach proposed by Kingma & Welling (2013) is used in settings where only a single modality a is present in order to ﬁnd a latent encoding z (c.f. Figure 1 (left)).

Original: In the following chapters, we give a brieﬂy comprise related work by means of multi-modal VAEs.
Revised: In the following chapters, we give a brieﬂy comprise related work by means of multi-modal VAEs.

Original: Further, we stress the concept of two joint multi-modal approaches to derive the later proposed variational Auto Encoder (VAE).
Revised: Further, we stress the concept of two joint multi-modal approaches to derive the later proposed variational Auto Encoder (VAE).

Original: 2. M ULTI -M ODAL A UTO E NCODER
Revised: 2. M ULTI -M ODAL A UTO E NCODER

Original: Given a set of modalities M = { a,b,c, . . .
Revised: Given a set of modalities M = { a,b,c, . . .

Original: } , multi-modal variants of Variational Auto Encoder s (VAE) have been applied to train generative models for multi-directional reconstruction (i.e. generation of missing data) or feature extraction.
Revised: } , multi-modal variants of Variational Auto Encoder s (VAE) have been applied to train generative models for multi-directional reconstruction (i.e. generation of missing data) or feature extraction.

Original: Variants are conditional VAEs (CVAE) and conditional multi-modal autoencoders (CMMA), with the lack in bi-directional reconstruction (Sohn et al.
Revised: Variants are conditional VAEs (CVAE) and conditional multi-modal autoencoders (CMMA), with the lack in bi-directional reconstruction (Sohn et al.

Original: ; Pandey & Dukkipati (2017)).
Revised: ; Pandey & Dukkipati (2017)).

Original: BiVCCA by Wang et al.
Revised: BiVCCA by Wang et al.

Original: (2016) trains two VAEs together with interacting inference networks to facilitate two-way reconstruction with the lack of directly modeling the joint distribution.
Revised: (2016) trains two VAEs together with interacting inference networks to facilitate two-way reconstruction with the lack of directly modeling the joint distribution.

Original: Models, that are derived from the variation of information (VI) with the objective to estimate the joint distribution with the capabilities of multi-directional reconstruction were recently introduced by Suzuki et al.
Revised: Models, that are derived from the variation of information (VI) with the objective to estimate the joint distribution with the capabilities of multi-directional reconstruction were recently introduced by Suzuki et al.

Original: Vedantam et al.
Revised: Vedantam et al.

Original: (2017) introduce another objective for the bi-modal VAE, which they call the triplet ELBO (tVAE).
Revised: (2017) introduce another objective for the bi-modal VAE, which they call the triplet ELBO (tVAE).

Original: Furthermore, multi-modal stacked
Revised: Furthermore, multi-modal stacked

Original: Auto Encoders (AE) are a variant of combining the latent spaces of various AEs ( Larochelle et al.
Revised: Auto Encoders (AE) are a variant of combining the latent spaces of various AEs ( Larochelle et al.

Original: Ranzato et al.
Revised: Ranzato et al.

Original: (2006)) which can also be applied to the reconstruction of missing modalities (Ngiam et al.
Revised: (2006)) which can also be applied to the reconstruction of missing modalities (Ngiam et al.

Original: ; Cadena et al.
Revised: ; Cadena et al.

Original: However, while Suzuki et al.
Revised: However, while Suzuki et al.

Original: (2017) and Vedantam et al.
Revised: (2017) and Vedantam et al.

Original: (2017) argue that training of the full multi-modal VAE is intractable, because of the 2 |M| −modality subsets of inference networks, we show that training the full joint model estimates the most expressive latent embeddings.
Revised: (2017) argue that training of the full multi-modal VAE is intractable, because of the 2 |M| −modality subsets of inference networks, we show that training the full joint model estimates the most expressive latent embeddings.

Original: 2.1. J OINT M ULTI -M ODAL V ARIATIONAL A UTO E NCODER
Revised: 2.1. J OINT M ULTI -M ODAL V ARIATIONAL A UTO E NCODER

Original: When more than one modality is available, e.g. a and b as shown in Figure 1 (mid.), the derivation of the ELBO L J for a marginal joint log-likelihood log p ( a ) :
Revised: When more than one modality is available, e.g. a and b as shown in Figure 1 (mid.), the derivation of the ELBO L J for a marginal joint log-likelihood log p ( a ) :

Original: =
Revised: =

Original: L J is straight forward:
Revised: L J is straight forward:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: However, it is not clear how to perform inference if the dataset consists of samples lacking from modalities (e.g. for samples i and k : ( a i , ∅ ) and ( ∅ ,b k ) ).
Revised: However, it is not clear how to perform inference if the dataset consists of samples lacking from modalities (e.g. for samples i and k : ( a i , ∅ ) and ( ∅ ,b k ) ).

Original: Ngiam et al.
Revised: Ngiam et al.

Original: (2011) proposes training of a bimodal deep auto encoder using an augmented dataset with additional examples that have only a single-modality as input.
Revised: (2011) propose training of a bimodal deep auto encoder using an augmented dataset with additional examples that have only a single-modality as input.

Original: We therefore name the resulting model of Eq.
Revised: We, therefore, name the resulting model of Eq.

Original: 2 joint multi-modal
Revised: 2 joint multi-modal

Original: VAE-Zero (JMVAE-Zero).
Revised: VAE-Zero (JMVAE-Zero).

Original: 2.1. J OINT M ULTI -M ODAL V ARIATIONAL A UTO E NCODER FROM V ARIATION OF I NFORMATION
Revised: 2.1. J OINT M ULTI -M ODAL V ARIATIONAL A UTO E NCODER FROM V ARIATION OF I NFORMATION

Original: While the former approach cannot directly  applied to missing modalities, Suzuki et al.
Revised: While the former approach cannot directly be applied to missing modalities, Suzuki et al.

Original: (2017) propose a joint multi-modal VAE (JMVAE) that is trained via two uni-modal encoders and a bi-modal en-/decoder which share one objective function derived from the variation of information (VI) of the marginal conditional log-likelihoods log p ( a | b ) p ( b | a )
Revised: (2017) propose a joint multi-modal VAE (JMVAE) that is trained via two uni-modal encoders and a bimodal en-/decoder which share one objective function derived from the variation of information (VI) of the marginal conditional log-likelihoods log p ( a | b ) p ( b | a )

Original: =:
Revised: =:

Original: L M by optimizing the ELBO L M :
Revised: L M by optimizing the ELBO L M :

Original: [Equation]
Revised: [Equation]

Original: Therefore, uni-modal encoders are trained, so that their distributions q φ a and q φ b are close to a multimodal encoder q φ ab in order to build a coherent posterior distribution.
Revised: Therefore, uni-modal encoders are trained, so that their distributions q φ a and q φ b are close to a multimodal encoder q φ ab in order to build a coherent posterior distribution.

Original: The introduced regularization by Suzuki et al.
Revised: The introduced regularization by Suzuki et al.

Original: (2017) puts learning pressure on the uni-modal encoders just by the distributions’ shape, disregarding reconstruction capabilities and the prior p ( z ) .
Revised: (2017) puts learning pressure on the uni-modal encoders just by the distributions’ shape, disregarding reconstruction capabilities and the prior p ( z ) .

Original: Furthermore, one can show that deriving the ELBO from the VI for a set of M observable modalities, always leads to an expression of the ELBO that allows only training of f
Revised: Furthermore, one can show that deriving the ELBO from the VI for a set of M observable modalities, always leads to an expression of the ELBO that allows only training of f

Original: [Equation]
Revised: [Equation]

Original:  modality combinations.
Revised: } modality combinations.

Original: This leads to the fact that for instance in a tri-modal setup, as shown in Fig.
Revised: This leads to the fact that for instance in a tri-modal setup, as shown in Fig.

Original: 1 (right), one can derive three bi-modal encoders from the VI, but no uni-modal ones.
Revised: 1 (right), one can derive three bi-modal encoders from the VI, but no uni-modal ones.

Original:  M ULTI -M ODAL V ARIATIONAL A UTO E NCODER A PPROACH
Revised:  M ULTI -M ODAL V ARIATIONAL A UTO E NCODER A PPROACH

Original: While the objective of Wang et al.
Revised: While the objective of Wang et al.

Original: (2016), Ngiam et al.
Revised: (2016), Ngiam et al.

Original: (2011), Suzuki et al.
Revised: (2011), Suzuki et al.

Original: (2017), and Vedantam et al.
Revised: (2017), and Vedantam et al.

Original: (2017) is to exchange modalities bi-directionally (e.g. a → b ′ ), our primary concern is twofold: First, ﬁnd a meaningful posterior distribution where the sampled statistics of an encoder network allows inference about further actions.
Revised: (2017) is to exchange modalities bi-directionally (e.g. a → b ′ ), our primary concern is twofold: First, ﬁnd a meaningful posterior distribution where the sampled statistics of an encoder network allows inference about further actions.

Original: Second, ﬁnd an expression to jointly train all 2 |M| −permutations of modality encoders.
Revised: Second, ﬁnd an expression to jointly train all 2 |M| −permutations of modality encoders.

Original: By successively applying logarithm and Bayes rules, we derive the ELBO for the multi-modal VAE (M 2 VAE) as follows:
Revised: By successively applying logarithm and Bayes rules, we derive the ELBO for the multi-modal VAE (M 2 VAE) as follows:

Original: First, given the independent set of observable modalities M = { a,b,c, . . . } , its marginal log-likelihood log p ( M ) =:
Revised: First, given the independent set of observable modalities M = { a,b,c, . . . } , its marginal log-likelihood log p ( M ) =:

Original: L M 2 is multiplied by the cardinality of the set as the neutral element 1 = |M| / |M| .
Revised: L M 2 is multiplied by the cardinality of the set as the neutral element 1 = |M| / |M| .

Original: Second, applying logarithm multiplication rule, the nominator is written as the argument’s exponent.
Revised: Second, applying logarithm multiplication rule, the nominator is written as the argument’s exponent.

Original: Third, Bayes rule is applied to each term wrt.
Revised: Third, Bayes rule is applied to each term wrt.

Original: the remaining observable modalities to derive their conditionals.
Revised: the remaining observable modalities to derive their conditionals.

Original: Further, we bootstrap the derivation technique in a bi- and tri-modal (c.f. tri-modal case in Sec. 6.1) case to illustrate the advantages.
Revised: Further, we bootstrap the derivation technique in a bi- and tri-modal (c.f. tri-modal case in Sec. 6.1) case to illustrate the advantages.

Original: By excessively applying the scheme until convergence of the mathematical expression, it leads for a bi-modal set M = { a,b } to the following result:
Revised: By excessively applying the scheme until convergence of the mathematical expression, it leads for a bi-modal set M = { a,b } to the following result:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: This term can be written as inequality wrt.
Revised: This term can be written as inequality wrt.

Original: each ELBO of the marginals L a , L b and conditionals
Revised: each ELBO of the marginals L a , L b and conditionals

Original: L M :
Revised: L M :

Original: [Equation]
Revised: [Equation]

Original: Equation 6 is substituted by all former derived ELBO expressions lead to the combination of the uni-modal VAEs wrt.
Revised: Equation 6 is substituted by all formerly derived ELBO expressions lead to the combination of the uni-modal VAEs wrt.

Original: a and b (c.f. Eq.
Revised: a and b (c.f. Eq.

Original: 7 to 8) and the JMVAE comprising the VAE wrt.
Revised: 7 to 8) and the JMVAE comprising the VAE wrt.

Original: the joint modality ab (c.f. Eq. 9) and mutual latent space (c.f. Eq. 10).
Revised: the joint modality ab (c.f. Eq. 9) and mutual latent space (c.f. Eq. 10).

Original: Equation 7 and 8 have the effect that their regularizers cares about the uni-modal distribution to deviate not too much from the common prior while their reconstruction term shapes the underlying embedding of the mutual latent space.
Revised: Equation 7 and 8 have the effect that their regularizers care about the uni-modal distribution to deviate not too much from the common prior while their reconstruction term shapes the underlying embedding of the mutual latent space.

Original: We further apply the concept of β -VAE (Higgins et al.
Revised: We further apply the concept of β -VAE (Higgins et al.

Original: (2016; 2017a); Burgess et al.
Revised: (2016; 2017a); Burgess et al.

Original: (2018)) to the regularizers via β ∗ and adopt the factor α from Suzuki et al.
Revised: (2018)) to the regularizers via β ∗ and adopt the factor α from Suzuki et al.

Original: (2017) for the mutual regularizer.
Revised: (2017) for the mutual regularizer.

Original: However, while β -VAE have the property to disentangle the latent space, our main concern is the balance between the input and the latent space using a constant normalized factor β norm =
Revised: However, while β -VAE have the property to disentangle the latent space, our main concern is the balance between the input and the latent space using a constant normalized factor β norm =

Original: β ∗ D ∗ / D z .
Revised: β ∗ D ∗ / D z .

Original: If the derivation, which we leave out for the sake of brevity, is applied to the log-likelihood L M 2 M of a set M , one can show that it results into a recursive form consisting of JMVAEs’ and M 2 VAEs’ log-likelihood terms
Revised: If the derivation, which we leave out for the sake of brevity, is applied to the log-likelihood L M 2 M of a set M , one can show that it results into a recursive form consisting of JMVAEs’ and M 2 VAEs’ log-likelihood terms

Original: [Equation]
Revised: [Equation]

Original: While the derivation of Eq.
Revised: While the derivation of Eq.

Original: 11 is given in Sec 6.1.3, the properties are as follows:
Revised: 11 is given in Sec 6.1.3, the properties are as follows:

Original: • the M 2 VAE consist out of 2 |M| − 1 encoders and |M| decoders comprising all modality combinations • while it also allows the bi-directional exchange of modalities, it further allows the setup of arbitrary modality combinations having 1 to |M| modalities • subsets of minor cardinality are weighted less and have  therefore minor impact in shaping the overall posterior distribution (vice versa, the major subsets dominate the shaping and the minor sets adapt to it) • all encoder/decoder networks can jointly be trained using SGVB
Revised: • the M 2 VAE consist out of 2 |M| − 1 encoders and |M| decoders comprising all modality combinations • while it also allows the bi-directional exchange of modalities, it further allows the setup of arbitrary modality combinations having 1 to |M| modalities • subsets of minor cardinality are weighted less and have a therefore minor impact in shaping the overall posterior distribution (vice versa, the major subsets dominate the shaping and the minor sets adapt to it) • all encoder/decoder networks can jointly be trained using SGVB

Original:  D ATA S ETS
Revised:  D ATA S ETS

Original: It is quite common in the multi-modal VAE community to model a bi-modal data set as follows (Wang et al.
Revised: It is quite common in the multi-modal VAE community to model a bi-modal dataset as follows (Wang et al.

Original: ; Ngiam et al.
Revised: ; Ngiam et al.

Original: (2011); Suzuki et al.
Revised: (2011); Suzuki et al.

Original: ; Vedantam et al.
Revised: ; Vedantam et al.

Original: The ﬁrst modality a denotes the raw data and b denotes the label (e.g. the digits’ images and labels as one-hot vector wrt.
Revised: The ﬁrst modality a denotes the raw data and b denotes the label (e.g. the digits’ images and labels as onehot vector wrt.

Original: the MNIST data set).
Revised: the MNIST dataset).

Original: This is a rather artiﬁcial assumption and only sufﬁcient when the objective is within a semi-supervised training framework.
Revised: This is a rather artiﬁcial assumption and only sufﬁcient when the objective is within a semi-supervised training framework.

Original: Real multi-modal data does not show this behavior as there are commonly multiple raw data inputs.
Revised: Real multi-modal data does not show this behavior as there are commonly multiple raw data inputs.

Original: Unfortunately, only complex multimodal data sets of heterogeneous sensor setups exists (Oﬂi et al.
Revised: Unfortunately, only complex multimodal datasets of heterogeneous sensor setups exist (Oﬂi et al.

Original: ; Udacity (2016); Kragh et al.
Revised: ; Udacity (2016); Kragh et al.

Original: (2017)), which makes  comprehensive evaluation for VAEs futile.
Revised: (2017)), which makes a comprehensive evaluation for VAEs futile.

Original: On the other hand, creating own multi-modal datasets is exhaustive since training generative models either demand dense sampling or supervised signals to form a consistent latent manifold (Bengio et al.
Revised: On the other hand, creating own multi-modal datasets is exhaustive since training generative models either demand dense sampling or supervised signals to form a consistent latent manifold (Bengio et al.

Original: While na¨ıve consolidation of non-coherently data sets does not meet the conditions of data continuity, as discussed later, we propose a consolidation technique by sampling from superimposed latent spaces of various uni-modal trained CVAEs in Sec. 4.1.
Revised: While na¨ıve consolidation of non-coherently datasets does not meet the conditions of data continuity, as discussed later, we propose a consolidation technique by sampling from superimposed latent spaces of various uni-modal trained CVAEs in Sec. 4.1.

Original: This approach allows the generation of multi-modal data sets from distinct and disconnected uni-modal sets.
Revised: This approach allows the generation of multi-modal datasets from distinct and disconnected uni-modal sets.

Original: Second, we propose and bi-modal mixture of Gaussians (MoG) data set to show particular behaviors of the various VAE approaches in Sec.
Revised: Second, we propose and bi-modal mixture of Gaussians (MoG) dataset to show particular behaviors of the various VAE approaches in Sec.

Original: 4. M ULTI -M ODAL D ATA G ENERATION
Revised: 4. M ULTI -M ODAL D ATA G ENERATION

Original: Perry et al.
Revised: Perry et al.

Original: (2010) state that Hebbian learning relies on the fact that  same objects are continuously transformed to their nearest neighbor in the observable space.
Revised: (2010) state that Hebbian learning relies on the fact that the same objects are continuously transformed to their nearest neighbor in the observable space.

Original: Higgins et al.
Revised: Higgins et al.

Original: (2016) adopted this approach to their assumptions, that this notion can be generalized within the latent manifold learning.
Revised: (2016) adopted this approach to their assumptions, that this notion can be generalized within the latent manifold learning.

Original: Further, neither a coherent manifold nor a proper factorization of the latent space can be trained if these assumptions are not fulﬁlled by the data set.
Revised: Further, neither a coherent manifold nor a proper factorization of the latent space can be trained if these assumptions are not fulﬁlled by the dataset.

Original: In summary, this means that observed data has to have the property of continues transformation wrt.
Revised: In summary, this means that observed data has to have the property of continues transformation wrt.

Original: to their properties (e.g. position and shape of an object), such that a small deviation of the observations result in proportional deviations in the latent space.
Revised: to their properties (e.g. position and shape of an object), such that a small deviation of the observations results in proportional deviations in the latent space.

Original: We adopt this assumption for multi-modal data sets where observations should correlate if the same quantity is observed, such that a small deviation in the common latent representation between all modalities conducts a proportional impact in all observations.
Revised: We adopt this assumption for multi-modal datasets where observations should correlate if the same quantity is observed, such that a small deviation in the common latent representation between all modalities conducts a proportional impact in all observations.

Original: This becomes an actual fundamental requirement to any multi-modal data set, as correlation and coherence are within the objective of multi-modal sensor fusion.
Revised: This becomes an actual fundamental requirement for any multi-modal dataset, as correlation and coherence are within the objective of multi-modal sensor fusion.

Original: In the following, we propose a technique to generate new multi-modal data sets, given different uni-modal enclosed sets which meet the former conditions.
Revised: In the following, we propose a technique to generate new multi-modal datasets, given different uni-modal enclosed sets which meet the former conditions.

Original: A valuable property of the VAE’s learned posterior distribution is, that it matches the desired prior quite sufﬁciently if only a single classes is observed.
Revised: A valuable property of the VAE’s learned posterior distribution is, that it matches the desired prior quite sufﬁciently if only a single class is observed.

Original: This characteristic can be found again in the conditional VAE (CVAE) Kingma et al.
Revised: This characteristic can be found again in the conditional VAE (CVAE) Kingma et al.

Original: (2014); Sohn et al.
Revised: (2014); Sohn et al.

Original: (2015) as it’s training is supported by the ground truth labels of the observations.
Revised: (2015) as it’s training is supported by the ground truth labels of the observations.

Original: Thus, it actually builds non-related posterior distribution for each class label, where every distribution matches a given prior.
Revised: Thus, it actually builds non-related posterior distribution for each class label, where every distribution matches a given prior.

Original: Furthermore, we adopt the idea of β -VAE Higgins et al.
Revised: Furthermore, we adopt the idea of β -VAE Higgins et al.

Original: (2017b) which learns disentangled and factorized latent representations.
Revised: (2017b) which learns disentangled and factorized latent representations.

Original: Combining the properties of both advantages allows the superimposing of latent manifolds from various uni-modal encoders as shown in Fig.
Revised: Combining the properties of both advantages allows the superimposing of latent manifolds from various uni-modal encoders as shown in Fig.

Original: 2 (Top-Right).
Revised: 2 (Top-Right).

Original: Now, latent samples can be draw from the posterior to operate all CVAE encoders, with the desired label, to generate continues multi-modal data.
Revised: Now, latent samples can be drawn from the posterior to operate all CVAE encoders, with the desired label, to generate continues multi-modal data.

Original: To test the approach we consolidate MNIST (LeCun Yann et al.
Revised: To test the approach we consolidate MNIST (LeCun Yann et al.

Original: and fashion-MNIST (Xiao et al.
Revised: and fashion-MNIST (Xiao et al.

Original: to an entagled-MNIST (e-MNIST) set by sampling from the prior (i.e. z ∼ N (0 , I ) ) to generate observation tuples from the corresponding encoder networks p θ a ( a | z,C ) and p θ b ( b | z,C ) with class label C .
Revised: to an entangled-MNIST (e-MNIST) set by sampling from the prior (i.e. z ∼ N (0 , I ) ) to generate observation tuples from the corresponding encoder networks p θ a ( a | z,C ) and p θ b ( b | z,C ) with class label C .

Original: The network architecture is explained in Sec. 3.
Revised: The network architecture is explained in Sec. 3.

Original: To avoid artifacts, only samples from within 2 σ of the prior are obtained.
Revised: To avoid artifacts, only samples from within 2 σ of the prior are obtained.

Original: Furthermore, we train a bi-modal JMVAE on the newly generated data to depict properties of the different data sets.
Revised: Furthermore, we train a bi-modal JMVAE on the newly generated data to depict properties of the different datasets.

Original: We are aware of the fact that consolidation of uni-modal data sets can not be achieved easily since continuity is hardly measurable.
Revised: We are aware of the fact that consolidation of uni-modal datasets cannot be achieved easily since continuity is hardly measurable.

Original: Therefore, na¨ıve consolidation results in a mixed data set (i.e. mixed-MNIST) as shown in Fig.
Revised: Therefore, na¨ıve consolidation results in a mixed dataset (i.e. mixed-MNIST) as shown in Fig.

Original: To mimic this behavior and to achieve a fair comparison of the ELBO, we shufﬂe the generated fashion-MNIST per class label of e-MNIST to generate an equivalent mixed-e-MNIST (me-MNIST) set.
Revised: To mimic this behavior and to achieve a fair comparison of the ELBO, we shufﬂe the generated fashion-MNIST per class label of e-MNIST to generate an equivalent mixed-e-MNIST (me-MNIST) set.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2 (bottom), the JMVAE’s latent space reveals that for m-MNIST single clusters share the same mean as the best representative of a single label, but the variance of any uni-modal trained encoder remains orthogonal.
Revised: 2 (bottom), the JMVAE’s latent space reveals that for m-MNIST single clusters share the same mean as the best representative of a single label, but the variance of any uni-modal trained encoder remains orthogonal.

Original: Thus, the continuity in the observations do not correlate with each other by any means.
Revised: Thus, the continuity in the observations does not correlate with each other by any means.

Original: On the other hand, the e-MNIST set with continues samples show the desired behavior of multi-modal data sets as the JMVAE trains a coherent distribution for all uniand multi-modal encoders.
Revised: On the other hand, the e-MNIST set with continues samples shows the desired behavior of multi-modal datasets as the JMVAE trains a coherent distribution for all uniand multi-modal encoders.

Original: These observations show that our proposed approach for generating new entangled data sets meet the formulated requirements of multi-modal data sets.
Revised: These observations show that our proposed approach for generating new entangled datasets meet the formulated requirements of multi-modal datasets.

Original: 4. M O G-E XAMPLE
Revised: 4. M O G-E XAMPLE

Original: We investigate a Mixture-of-Gaussians (MoG) distribution, as depicted in Fig.
Revised: We investigate a Mixture-of-Gaussians (MoG) distribution, as depicted in Fig.

Original: 4 (bottom right), as bi-modal observations to mimic the output of e.g. feature extractors or classiﬁers.
Revised: 3, as bi-modal observations to mimic the output of e.g. feature extractors or classiﬁers.

Original: While they commonly already provide linear separable observations, we focus on ambiguity resolving properties of the VAE in particular.
Revised: While they commonly already provide linear separable observations, we focus on ambiguity resolving properties of the VAE in particular.

Original: The bi-modal ( a,b ) observations of Mixture-of-Gaussians have ten classes (0 , . .
Revised: The bi-modal ( a,b ) observations of Mixture-of-Gaussians have ten classes (0 , . .

Original: , 9) each.
Revised: , 9) each.

Original: a ’s observations are organized on a grid where (5 , 6 , 7) and (0 , 8) result into ambiguous observations by sharing the same mean.
Revised: a ’s observations are organized on a grid where (5 , 6 , 7) and (0 , 8) result in ambiguous observations by sharing the same mean.

Original: b ’s observations are organized on a circle where (0 , 9) have ambiguous mean values.
Revised: b ’s observations are organized on a circle where (0 , 9) have ambiguous mean values.

Original: This rather artiﬁcial experiment has the purpose to depict and evaluate ambiguous resolving properties of the VAEs.
Revised: This rather artiﬁcial experiment has the purpose to depict and evaluate ambiguous resolving properties of the VAEs.

Original: However, data of multi-modal sensor setups for complementary fusion show similar behavior, as various modalities are rectiﬁed to achieve a complete view of the scene (e.g. vision and grope to rectify objects).
Revised: However, data of multi-modal sensor setups for complementary fusion show similar behavior, as various modalities are rectiﬁed to achieve a complete view of the scene (e.g. vision and grope to rectify objects).

Original: In that case, various dependencies of the generative process, given the class labels as factorized latent state representation z
Revised: In that case, various dependencies of the generative process, given the class labels as factorized latent state representation z

Original: = ( z 0 , . . .
Revised: = ( z 0 , . . .

Original: ,z 9 ) , are possible.
Revised: ,z 9 ) , are possible.

Original: This is mimicked by the MoG-Experiment, in a simpliﬁed assumption, as: p ( a,b | z ) , p ( a | z 1 , . . .
Revised: This is mimicked by the MoG-Experiment, in a simpliﬁed assumption, as: p ( a,b | z ) , p ( a | z 1 , . . .

Original: , z 4 ,z 9 ) , and p ( b | z 1 , . .
Revised: , z 4 ,z 9 ) , and p ( b | z 1 , . .

Original: . , z 8 ) .
Revised: . , z 8 ) .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We apply the data sets explained in Sec. 4 to test and depict the capabilities of the M 2 VAE.
Revised: We apply the datasets explained in Sec. 4 to test and depict the capabilities of the M 2 VAE.

Original: First, we investigate the MoG data comprehensively.
Revised: First, we investigate the MoG data comprehensively.

Original: Second, we evaluate the ELBO of various approaches on the e-MNIST data set.
Revised: Second, we evaluate the ELBO of various approaches to the e-MNIST dataset.

Original: The VAEs are compared qualitatively, by visualizing the latent space, and quantitatively by performing lower bound tests L f M for every subset f
Revised: The VAEs are compared qualitatively, by visualizing the latent space, and quantitatively by performing lower bound tests L f M for every subset f

Original: M ⊆ M wrt.
Revised: M ⊆ M wrt.

Original: to the decoding  of all modalities p θ M :
Revised: to the decoding [Figure] [Figure] of all modalities p θ M :

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: with p ( z ) =
Revised: with p ( z ) =

Original: N ( z ; 0 , I ) .
Revised: N ( z ; 0 , I ) .

Original: All VAE architectures can be found in Sec.
Revised: All VAE architectures can be found in Sec.

Original: [Figure]
Revised: [Figure]

Original: latent
Revised: q φ a β ∗ = 10 −α = 10 −β ∗ = 10 −α = 10 −β ∗ = 10 −α = 10 −α = 10 −β ∗ = 10 −α = 10 −β ∗ = 10 −

Original: [Figure]
Revised: [Figure]

Original: 5. M O G-E XPERIMENT
Revised: 5. M O G-E XPERIMENT

Original: We evaluate the latent space with the premise in mind, that a good generative model should not just generate good data but also gives a good latent representation z .
Revised: We evaluate the latent space with the premise in mind, that a good generative model should not just generate good data but also gives a good latent representation z .

Original: 5.1. P ARAMETRIZATION
Revised: 5.1. P ARAMETRIZATION

Original: We ﬁrst investigate the impact of the parameter set ( β ∗ , α ) on the M 2 VAE to ﬁnd a latent space representation, which suits our needs to learn actions from it.
Revised: We ﬁrst investigate the impact of the parameter set ( β ∗ , α ) on the M 2 VAE to ﬁnd a latent space representation, which suits our needs to learn actions from it.

Original: As the α parameter controls the mutual connection of all encoders in latent space, we found that a direct connection (i.e. α = 1 . ) puts too much learning pressure on matching the mutual latent distributions between uni- and multi-modal encoders.
Revised: As the α parameter controls the mutual connection of all encoders in latent space, we found that a direct connection (i.e. α = 1 . ) puts too much learning pressure on matching the mutual latent distributions between uni- and multi-modal encoders.

Original: Thus, classes which should be separated in the multi-modal latent space collapse to the mean distributions of the uni-modal encoders.
Revised: Thus, classes which should be separated in the multi-modal latent space collapse to the mean distributions of the uni-modal encoders.

Original: For α / 10 − 2 , the encoders are able  ﬁnd an expressive latent space distributions by means of separable collapsed classes of uni-modal encoders, and expanded classes of multi-modal around it (c.f. Fig.top/left).
Revised: For α / 10 − 2 , the encoders are able to ﬁnd an expressive latent space distribution by means of separable collapsed classes of uni-modal encoders, and expanded classes of multi-modal around it (c.f. Fig.top/left).

Original: By the ﬁndings of Higgins et al.
Revised: By the ﬁndings of Higgins et al.

Original: (2017b), high β values result in highly entangled factors in latent space whereas small normalized β norm / 10 − 2 show pretty robust disentanglement in all their test cases.
Revised: (2017b), high β values result in highly entangled factors in latent space whereas small normalized β norm / 10 − 2 show pretty robust disentanglement in all their test cases.

Original: The impact of β shows similar behavior on the M 2 VAE and thus, we chose small β values of β norm =10 − 2 to relax the learning pressure caused by the prior.
Revised: The impact of β shows similar behavior on the M 2 VAE and thus, we chose small β values of β norm =10 − 2 to relax the learning pressure caused by the prior.

Original: While the over optimization wrt.
Revised: While the over optimization wrt.

Original: to the prior leads to  fuzzy generation of data p ( M| z ) and collapse in latent space, high relaxation [Figure] q φ a β ∗ = 10 −α = 10 −β ∗ = 10 −α = 10 −β ∗ = 10 −α = 10 −α = 10 −β ∗ = 10 −α = 10 −β ∗ = 10 − 
Revised: to the prior leads to a fuzzy generation of data p ( M| z ) and collapse in latent space, high relaxation ( β norm ≪ 10 − 3 ) causes loss of expressiveness between uni- and multi-modal encoding of a single class by means of the difference in the ELBO.

Original: [Figure] It is worth noticing, that diverging β parameters between multi- and uni-modal regularization (e.g. β ab ≪ β a or vice versa) results in lower ELBOs, but for the sake of expressiveness of latent embedding and the ELBO between encoders’ embeddings.
Revised: It is worth noticing, that diverging β parameters between multi- and uni-modal regularization (e.g. β ab ≪ β a or vice versa) results in lower ELBOs, but for the sake of expressiveness of latent embedding and the ELBO between encoders’ embeddings.

Original: We argue that learning pressure should be applied equally to all encoders so that they experience  similar learning impact.
Revised: We argue that learning pressure should be applied equally to all encoders so that they experience a similar learning impact.

Original: Another observations results from the fact, that the reconstruction loss of the M 2 VAE’s objective causes learning of mean representatives of classes in the observation space.
Revised: Another observation results from the fact, that the reconstruction loss of the M 2 VAE’s objective causes learning of mean representatives of classes in the observation space.

Original: This causes the artifact, that if for instance three classes exist in the output space, where one represents the overall mean, and an uni-modal encoder only sees the collapse of classes to that particular mean value, the latent encoding of this uni-modal encoder will collapse to  same mean as well.
Revised: This causes the artifact, that if for instance three classes exist in the output space, where one represents the overall mean, and an uni-modal encoder only sees the collapse of classes to that particular mean value, the latent encoding of this uni-modal encoder will collapse to the same mean as well.

Original: However, while it is not longer separable (not even non-linearly) in latent space by its mean value, the ELBO for the observation drives up and gives therefore evidence about the embedding quality.
Revised: However, while it is not longer separable (not even non-linearly) in latent space by its mean value, the ELBO for the observation drives up and gives, therefore, evidence about the embedding quality.

Original: This insight might be fruitful in terms of epistemic (ambiguity-resolving) tasks, where for instance an unsupervised reinforcment learning approach could use the ELBO as a signal to learn epistemic exploration.
Revised: This insight might be fruitful in terms of epistemic (ambiguity-resolving) tasks, where for instance an unsupervised reinforcement learning approach could use the ELBO as a signal to learn epistemic exploration.

Original: 5.1. C OMPARISION
Revised: 5.1. C OMPARISION

Revised: [Figure]

Revised: ( ≈ ) ( ∼ ) ( ≈ ) ( ∼ ) ($) ($) ($) ( − ) ( − ) ( − ) (+) (+) (+) ( / ) ( / ) ( ∗ ) ( ∗ ) ( ∗ )

Original:  Comparing the three approaches to estimate the multi-modal marginal log-likelihood by maximizing the ELBO, one can see from Fig.
Revised: [Figure] Comparing the three approaches to estimate the multi-modal marginal log-likelihood by maximizing the ELBO, one can see from Fig.

Original: 4 that the most coherent latent space distribution was learned by the proposed M 2 VAE.
Revised: 5 that the most coherent latent space distribution was learned by the proposed M 2 VAE.

Original: While the JMVAE-Zero learned similarities between q φ ab and q φ a , it learned a complete new embedding for the classes (1 , 2) with q φ b (denoted by ($) ).
Revised: While the JMVAE-Zero learned similarities between q φ ab and q φ a , it learned a complete new embedding for the classes (1 , 2) with q φ b (denoted by ($) ).

Original: Furthermore, the ELBO per embedding allows no conclusion between the embeddings of the various encoders.
Revised: Furthermore, the ELBO per embedding allows no conclusion between the embeddings of the various encoders.

Original: The tVAE founds a much more coherent embedding between the encoders.
Revised: The tVAE founds a much more coherent embedding between the encoders.

Original: This was achieved by the fact, that ﬁrst the full multi-modal VAE, consisting out of the encoder q φ ab and two decoder p θ a and p θ b , was trained.
Revised: This was achieved by the fact, that ﬁrst the full multi-modal VAE, consisting out of the encoder q φ ab and two decoder p θ a and p θ b , was trained.

Original: Second, the decoder weights are pinned to train the remaining uni-modal q φ ab q φ a q φ b ( ≈ ) ( ∼ ) ( ≈ ) ( ∼ ) ($) ($) ($) ( − ) ( − ) ( − ) (+) (+) (+) ( / ) ( / ) ( ∗ ) ( ∗ ) ( ∗ ) [Figure] ELBO Top:
Revised: Second, the decoder weights are pinned to train the remaining uni-modal networks which enforces coherence.

Original: [Figure]

Original: [Figure]

Original: q φ ab q φ a q φ b a b c l a ss e s

Original: EL [Figure] networks which enforces coherence. However, the ELBO per embedding also does not allow any direct conclusion between the embeddings of the various encoders.
Revised: However, the ELBO per embedding also does not allow any direct conclusion between the embeddings of the various encoders.

Original: This is depicted by ( ∼ ) , where the multi-modal encoder q φ ab produces embeddings of higher energy than these of the uni-modal ones.
Revised: This is depicted by ( ∼ ) , where the multi-modal encoder q φ ab produces embeddings of higher energy than these of the uni-modal ones.

Original: This can happen as there is no regularizer which enforces the variational distribution of the encoders to match each other and thus, the KL-divergence may differ between the models for similar encodings.
Revised: This can happen as there is no regularizer which enforces the variational distribution of the encoders to match each other and thus, the KL-divergence may differ between the models for similar encodings.

Original: The M 2 VAE on the other hand enforces the encoders inherently to approximate the same posterior distribution which can be seen by the strong coherences between all embeddings.
Revised: The M 2 VAE, on the other hand, enforces the encoders inherently to approximate the same posterior distribution which can be seen by the strong coherence between all embeddings.

Original: Furthermore, classes which are separated in the multi-modal latent embedding collapse to the mean values in the uni-modal ones as denoted by (+) and ( − ) .
Revised: Furthermore, classes which are separated in the multi-modal latent embedding collapse to the mean values in the uni-modal ones as denoted by (+) and ( − ) .

Original: This behavior is also rendered by the ELBO.
Revised: This behavior is also rendered by the ELBO.

Original: As the
Revised: As the

Original: M 2 VAE makes ambiguous embeddings, the reconstruction loss drives up (c.f.
Revised: M 2 VAE makes ambiguous embeddings, the reconstruction loss drives up (c.f.

Original: ( ∗ ) and ( / ) ).
Revised: ( ∗ ) and ( / ) ).

Original: The embeddings also show an interesting fact about the class (0) :
Revised: The embeddings also show an interesting fact about the class (0) :

Original: As this class is only ambiguously detectable in the uni-modal case, all VAEs learn an linear separable and therefore unambiguous embedding if both modalities make  observation of this class (denoted by ( − ) for the M 2 VAE).
Revised: As this class is only ambiguously detectable in the uni-modal case, all VAEs learn a linear separable and therefore unambiguous embedding if both modalities make an observation of this class (denoted by ( − ) for the M 2 VAE).

Original: 5. I N -P LACE S ENSOR F USION
Revised: 5. I N -P LACE S ENSOR F USION

Original: Further, we introduce the concept of in-place sensor fusion using multi-modal VAEs.
Revised: Further, we introduce the concept of in-place sensor fusion using multi-modal VAEs.

Original: This approach is applicable in distributed active-sensing tasks where the latent space representation z of observations M ′ (i.e. an object or point of interest was observed by a set of modalities) can be interpreted as inverse sensor model (c.f. Thrun et al.
Revised: This approach is applicable in distributed active-sensing tasks where the latent space representation z of observations M ′ (i.e. an object or point of interest was observed by a set of modalities) can be interpreted as inverse sensor model (c.f. Thrun et al.

Original: This compressed information can be efﬁciently transmitted between all sensing agents and also be updated as follows:
Revised: This compressed information can be efﬁciently transmitted between all sensing agents and also be updated as follows:

Original: z can be unfolded to the original observation using the VAE’s decoder networks and combined with any new observation m to update the information in-place
Revised: z can be unfolded to the original observation using the VAE’s decoder networks and combined with any new observation m to update the information in-place

Original: z → z ∗ viaq
Revised: z → z ∗ via

Original: [Equation]

Original: with

Original: [Equation]
Revised: [Equation]

Original: However, a necessary requirement of Eq.
Revised: However, a necessary requirement of Eq.

Original: 13 is that auto re-encoding (i.e. z → z via q φ M′ ( z |M ′ ) ) does not manipulate the information comprised by z in an unrecoverable way (e.g. label-switching).
Revised: 13 is that auto re-encoding (i.e. z → z via q φ M′ ( z |M ′ ) ) does not manipulate the information comprised by z in an unrecoverable way (e.g. label-switching).

Original: Thus, we assume that VAEs tend to have a natural denoising characteristic (despite the explicit denoising Auto Encoders) which should re-encode any z in a better version of its own by means of the reconstruction loss wrt.
Revised: Thus, we assume that VAEs tend to have a natural denoising characteristic (despite the explicit denoising Auto Encoders) which should re-encode any z in a better version of its own by means of the reconstruction loss wrt.

Original: z .
Revised: z .

Original: This behavior is shown in Fig.
Revised: This behavior is shown in Fig.

Original: 5 where we underlay the latent representation with the reconstruction loss of every particular z .
Revised: 6 where we underlay the latent representation with the reconstruction loss of every particular z .

Original: One can see the learned discrimination of the latent space by means of high entropy separating the clusters vicinity.
Revised: One can see the learned discrimination of the latent space by means of high entropy separating the clusters vicinity.

Original: Furthermore, initial z values are auto re-encoded which draw the trajectories along their path in latent space.
Revised: Furthermore, initial z values are auto re-encoded which draw the trajectories along their path in latent space.

Original: The observable properties of the VAE is that every seed converges to a ﬁxed-point while performing descending steps on the latent space manifold.
Revised: The observable properties of the VAE are that every seed converges to a ﬁxed-point while performing descending steps on the latent space manifold.

Original: However, this statement is only true in general for the proposed
Revised: However, this statement is only valid in general for the proposed

Original: M 2 VAE, as the JMVAE-Zero and tVAE learn non or only similar coherent latent spaces between the encoder networks.
Revised: M 2 VAE, as the JMVAE-Zero and tVAE learn no or only similar coherent latent spaces between the encoder networks.

Original: Thus, seeds may be attracted by wrong attractors which makes these approach not sufﬁcient for in-place sensor fusion.
Revised: Thus, seeds may be attracted by wrong attractors which makes these approach not sufﬁcient for in-place sensor fusion.

Original: [Figure]
Revised: [Figure]

Original: 5. E -MNIST E VALUATION
Revised: 5. E -MNIST E VALUATION

Original: For this experiment, we estimated the ELBO by Eq.
Revised: For this experiment, we estimated the ELBO by Eq.

Original: 12 to evaluate the performance of models
Revised: 12 to evaluate the performance of models

Original: JMVAE-Zero, tVAE, and M 2 VAE.
Revised: JMVAE-Zero, tVAE, and M 2 VAE.

Original: We chose the model wrt.
Revised: We chose the model wrt.

Original: to the evaluation in Fig.
Revised: to the evaluation in Fig.

Original: 3 with β norm = 0 .
Revised: 4 with β norm = 0 .

Original: 01 which is β ∗ ≈ 4 for the given MNIST image dimension of D
Revised: 01 which is β ∗ ≈ 4 for the given MNIST image dimension of D

Original: a = || (28 , 28 , 1)
Revised: a = || (28 , 28 , 1)

Original: ||
Revised: ||

Original: and D z = 2 .
Revised: and D z = 2 .

Original: However, Tbl. 1 shows quantitatively and Fig.
Revised: However, Tbl. 1 shows quantitatively and Fig.

Original: 6 depicts qualitatively that the pro-
Revised: 7 depicts qualitatively that the pro-

Original: [Table]
Revised: [Table]

Revised: [Figure]

Original: posed M 2 VAE reaches the highest ELBO value, as well as it learns the most expressive latent space distribution.
Revised: posed M 2 VAE reaches the highest ELBO value, as well as it learns the most expressive latent space distribution.

Original: Furthermore, by sampling from the latent space for data generation, the M 2 VAE reveals crisp reconstructions in comparison to the other approaches.[Figure]
Revised: Furthermore, by sampling from the latent space for data generation, the M 2 VAE reveals crisp reconstructions in comparison to the other approaches.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This work presents a novel multi-modal Variational Auto Encoder which is derived from the complete marginal joint log-likelihood.
Revised: This work presents a novel multi-modal Variational Auto Encoder which is derived from the complete marginal joint log-likelihood.

Original: We showed that this expression can jointly be trained on an
Revised: We showed that this expression can jointly be trained on an

Original: Mixture-of-Gaussian data set with ambiguous observations, as well as on a complex data set derived from MNIST and fashion-MNIST.
Revised: Mixture-of-Gaussian dataset with ambiguous observations, as well as on a complex dataset derived from MNIST and fashion-MNIST.

Original: Furthermore, we formulated requirements and characteristics for multi-modal data for sensor fusion and derived a technique to learn new data sets, namely the proposed entangled-MNIST, which sufﬁce these requirements.
Revised: Furthermore, we formulated requirements and characteristics for multi-modal data for sensor fusion and derived a technique to learn new datasets, namely the proposed entangled-MNIST, which sufﬁce these requirements.

Original: Lastly, we developed the idea of in-place sensor fusion in distributed, active sensing scenarios and formulated the requirements, by means of auto re-encoding, to VAEs.
Revised: Lastly, we developed the idea of in-place sensor fusion in distributed, active sensing scenarios and formulated the requirements, by means of auto re-encoding, to VAEs.

Original: This revealed the properties of VAEs, that they tend to denoise the observable data which leads to an attractor behavior in latent space.
Revised: This revealed the properties of VAEs, that they tend to denoise the observable data which leads to an attractor behavior in latent space.

Original: However, we performed all qualitative evaluations of the latent space with the premise in mind, that a good generative model should not just generate good data but also gives a good latent representation.
Revised: However, we performed all qualitative evaluations of the latent space with the premise in mind, that a good generative model should not just generate good data but also gives a good latent representation.

Original: This does also correlate with the quantitative behaviors, as our proposed model achieved the highest ELBO values.
Revised: This does also correlate with the quantitative behaviors, as our proposed model achieved the highest ELBO values.

Original: Future work will concentrate on the integration of the ambiguous resolving characteristics to an epistemic-exploration scenario.
Revised: Future work will concentrate on the integration of the ambiguous resolving characteristics to an epistemic-exploration scenario.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: This research was supported by ’CITEC’ (EXC 277) at Bielefeld University and the Federal Ministry of Education and Research (57388272).
Revised: This research was supported by ’CITEC’ (EXC 277) at Bielefeld University and the Federal Ministry of Education and Research (57388272).

Original: The responsibility for the content of this publication lies with the author.
Revised: The responsibility for the content of this publication lies with the author.

Original: APPENDIX 6. E XTENSION TO THREE M ODALITIES
Revised: APPENDIX 6. E XTENSION TO THREE M ODALITIES

Original: The proposed as well as approach by Suzuki et al.
Revised: The proposed, as well as approach by Suzuki et al.

Original: (2017) can be extended to multiple modalities
Revised: (2017) can be extended to multiple modalities

Original: M = { a,b,c } .
Revised: M = { a,b,c } .

Original: The conditional marginal log-likelihood of a can be written as
Revised: The conditional marginal log-likelihood of a can be written as

Original: [Equation]
Revised: [Equation]

Original:  JMVAE FOR THREE M ODALITIES
Revised: 6.1. JMVAE FOR THREE M ODALITIES

Original: The VI between a set of distributions M can be written as − E p ( M ) P m ∈M log p ( m |M \ m ) , which leads to an expression of maximizing the ELBO of negative VI (c.f. Suzuki et al.
Revised: The VI between a set of distributions M can be written as − E p ( M ) P m ∈M log p ( m |M \ m ) , which leads to an expression of maximizing the ELBO of negative VI (c.f. Suzuki et al.

Original: Following this approach, the log-likelihood L 3M can be expressed by the ELBOs, by utilizing Eq.
Revised: Following this approach, the log-likelihood L 3M can be expressed by the ELBOs, by utilizing Eq.

Original: 14, of their conditionals and KL divergence:
Revised: 14, of their conditionals and KL divergence:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: with L e J being the joint ELBO of a joint probability p ( M ) which expression is analog to Eq. 2.
Revised: with L e J being the joint ELBO of a joint probability p ( M ) which expression is analog to Eq. 2.

Original: 6.1. M 2 VAE FOR THREE M ODALITIES
Revised: 6.1. M 2 VAE FOR THREE M ODALITIES

Original: Applying the proposed scheme to the joint log-likelihood of three modalities results in the following expression:
Revised: Applying the proposed scheme to the joint log-likelihood of three modalities results in the following expression:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: From here on
Revised: From here on

Original: [Equation]
Revised: [Equation]

Original: one can substitute all log-likelihoods given the expressions in Sec. 3 and ??
Revised: one can substitute all log-likelihoods given the expressions in Sec. 3 and ??

Original: [Equation]
Revised: [Equation]

Original: to derive the ELBO L 3M 2 .
Revised: to derive the ELBO L 3M 2 .

Original: 6.1. M 
Revised: 6.1. M 2 VAE D ERIVATION

Original: [Equation] VAE D ERIVATION

Original: [Equation]
Revised: [Equation]

Original: The expression P m ∈M log p ( m |M \ m ) is the general form of the marginal log-likelihood for the variation of information (VI), as introduced by Suzuki et al.
Revised: The expression P m ∈M log p ( m |M \ m ) is the general form of the marginal log-likelihood for the variation of information (VI), as introduced by Suzuki et al.

Original: (2017) for the JMVAE, for any set
Revised: (2017) for the JMVAE, for any set

Original: M .
Revised: M .

Original: Thus, it can be directly substituted with L M M .
Revised: Thus, it can be directly substituted with L M M .

Original: The expression P m ∈M log p ( M \ m ) is the combination of all joint log-likelihoods of the subsets of M which have one less element.
Revised: The expression P m ∈M log p ( M \ m ) is the combination of all joint log-likelihoods of the subsets of M which have one less element.

Original: Therefore, this term can be rewritten as
Revised: Therefore, this term can be rewritten as

Original: [Equation]
Revised: [Equation]

Revised: with f

Revised: M

Original: [Equation]
Revised: [Equation]

Original: with f
Revised: |

Original: M = { m | m ∈ P ( M ) , | m | = |M| − 1 } Finally, log p ( e m ) can be substituted by L M 2 f m without loss of generality.
Revised: [Equation] | [Equation] |M| − 1 } Finally, log p ( e m ) can be substituted by L M 2 f m without loss of generality.

Original: However, it is worth noticing that substitution stops at the end of recursion and therefore, all ﬁnal expressions log p ( e m ) ∀ | e m | ≡ 1 remain.
Revised: However, it is worth noticing that substitution stops at the end of recursion and therefore, all ﬁnal expressions log p ( e [Equation] remain.

Original: (cid:3)
Revised: (cid:3)

Original: 6.1. N ETWORK A RCHITECTURE
Revised: 6.1. N ETWORK A RCHITECTURE

Original: We designed all VAEs such that the latent space prior is given by a Gaussian with unit variance.
Revised: We designed all VAEs such that the latent space prior is given by a Gaussian with unit variance.

Original: Furthermore, all VAEs sample from a Gaussian variational distribution that is parametrized by the encoder networks.
Revised: Furthermore, all VAEs sample from a Gaussian variational distribution that is parametrized by the encoder networks.

Original: A summary of all architectures used in this paper can be seen in Tbl.
Revised: A summary of all architectures used in this paper can be seen in Tbl.

Original: The reconstruction loss for calculating the evidence lower bound was performed by binary cross-entropy (BCE) for the e-MNIST and root-mean-squared error (RMS) for the MoG experiment.
Revised: The reconstruction loss for calculating the evidence lower bound was performed by binary cross-entropy (BCE) for the e-MNIST and root-mean-squared error (RMS) for the MoG experiment.

Original: Furthermore, the CVAE for training the e-MNIST data set is designed as depicted in Tbl.
Revised: Furthermore, the CVAE for training the e-MNIST dataset is designed as depicted in Tbl.

Original: [Table] [Table]
Revised: [Table] [Table]