Original: I MPORTANCE - BASED M ULTIMODAL A UTOENCODER
Revised: I MPORTANCE - BASED M ULTIMODAL A UTOENCODER

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Integrating information from multiple modalities (e.g., verbal, acoustic and visual data) into meaningful representations has seen great progress in recent years.
Revised: Integrating information from multiple modalities (e.g., verbal, acoustic and visual data) into meaningful representations has seen great progress in recent years.

Original: However, two challenges are not sufﬁciently addressed by current approaches: (1) computationally efﬁcient training of multimodal autoencoder networks which are robust in the absence of modalities, and (2) unsupervised learning of important subspaces in each modality which are correlated with other modalities.
Revised: However, two challenges are not sufﬁciently addressed by current approaches: (1) computationally efﬁcient training of multimodal autoencoder networks which are robust in the absence of modalities, and (2) unsupervised learning of important subspaces in each modality which are correlated with other modalities.

Original: In this paper we propose the IMA (Importance-based Multimodal Autoencoder) model, a scalable model that learns modality importances and robust multimodal representations through a novel cross-covariance based loss function.
Revised: In this paper we propose the IMA (Importance-based Multimodal Autoencoder) model, a scalable model that learns modality importances and robust multimodal representations through a novel cross-covariance based loss function.

Original: We conduct experiments on MNIST-TIDIGITS, a multimodal dataset of spoken and image digits, and on IEMOCAP, a multimodal emotion corpus.
Revised: We conduct experiments on MNIST-TIDIGITS, a multimodal dataset of spoken and image digits, and on IEMOCAP, a multimodal emotion corpus.

Original: The IMA model is able to distinguish digits from uncorrelated noise, and word-level importances are learnt that correspond to the separation between function and emotional words.
Revised: The IMA model is able to distinguish digits from uncorrelated noise, and word-level importances are learnt that correspond to the separation between function and emotional words.

Original: The multimodal representations learnt by IMA are also competitive with state-of-the-art baseline approaches on downstream tasks.
Revised: The multimodal representations learnt by IMA are also competitive with state-of-the-art baseline approaches on downstream tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: With the ever-increasing amount of heterogeneous multimedia content present on the internet, machine learning approaches have been applied to automated perception problems such as object recognition (Krizhevsky et al., 2012), image captioning (Vinyals et al., 2015) and automatic language translation (Choi et al., 2018).
Revised: With the ever-increasing amount of heterogeneous multimedia content present on the internet, machine learning approaches have been applied to automated perception problems such as object recognition (Krizhevsky et al., 2012), image captioning (Vinyals et al., 2015) and automatic language translation (Choi et al., 2018).

Original: An important research direction is the problem of learning representations from multiple modalities which represent our primary channels of communication and sensation, such as vision or touch (Baltru ˇ saitis et al., 2018).
Revised: An important research direction is the problem of learning representations from multiple modalities which represent our primary channels of communication and sensation, such as vision or touch (Baltru ˇ saitis et al., 2018).

Original: With respect to this area of research, there are two major challenges in this research area which our paper addresses.
Revised: With respect to this area of research, there are two major challenges in this research area which our paper addresses.

Original: The ﬁrst is the design of encoder networks to enable learning and inference of multimodal representations in the absence of modalities.
Revised: The ﬁrst is the design of encoder networks to enable learning and inference of multimodal representations in the absence of modalities.

Original: This is useful for scenarios such as sensor failure or imputation/bidirectional generation of missing modalities from any combination of the observed ones.
Revised: This is useful for scenarios such as sensor failure or imputation/bidirectional generation of missing modalities from any combination of the observed ones.

Original: The caveat is that to have this property, recent approaches such as the JMVAE-KL model (Suzuki et al., 2016) and MVAE (Wu & Goodman, 2018) have encoders with high complexity for a large number of modalities.
Revised: The caveat is that to have this property, recent approaches such as the JMVAE-KL model (Suzuki et al., 2016) and MVAE (Wu & Goodman, 2018) have encoders with high complexity for a large number of modalities.

Original: When
Revised: When

Original: M is the number of modalities, JMVAE-KL needs 2 M sub-networks for every combination of input modalities, while MVAE requires M sub-networks but additional O (2 M ) subsampled loss terms to handle missing modalities.
Revised: M is the number of modalities, JMVAE-KL needs 2 M sub-networks for every combination of input modalities, while MVAE requires M sub-networks but additional O (2 M ) subsampled loss terms to handle missing modalities.

Original: The second challenge is that multimodal data, such as emotional spoken utterances or web images with captions are often generated not only by an underlying shared latent factor, but also by modality-speciﬁc private latent factors.
Revised: The second challenge is that multimodal data, such as emotional spoken utterances or web images with captions are often generated not only by an underlying shared latent factor, but also by modality-speciﬁc private latent factors.

Original: For example in spoken utterances, the verbal modality (words) are generated not only due to emotion but also due to syntax and semantics.
Revised: For example in spoken utterances, the verbal modality (words) are generated not only due to emotion but also due to syntax and semantics.

Original: Function words such as I and the are mostly syntactic and do not relate to emotion, similarly not all recorded audio frames are indicative of emotion.
Revised: Function words such as I and the are mostly syntactic and do not relate to emotion, similarly not all recorded audio frames are indicative of emotion.

Original: The inference of how relevant a sample in each modality is to the shared latent factor (subsequently referred to as importance ) is important for downstream tasks.
Revised: The inference of how relevant a sample in each modality is to the shared latent factor (subsequently referred to as importance ) is important for downstream tasks.

Original: For the remainder of this paper, non-relevant samples are refered to as uncorrelated noise .
Revised: For the remainder of this paper, non-relevant samples are refered to as uncorrelated noise .

Original: In a supervised setting, the latent factors and modality importance weights can be learnt from task labels.
Revised: In a supervised setting, the latent factors and modality importance weights can be learnt from task labels.

Original: When labels are absent in the unsupervised scenario, for the purpose of this paper we deﬁne the concept of modality importances based on correlations between the latent factor and each modality.
Revised: When labels are absent in the unsupervised scenario, for the purpose of this paper we deﬁne the concept of modality importances based on correlations between the latent factor and each modality.

Original: In the important subspace of each modality the multimodal and unimodal representations both maximally correlate, indicating that samples in that modality subspace can be attributed to a shared latent factor, and not an independent private one.
Revised: In the important subspace of each modality the multimodal and unimodal representations both maximally correlate, indicating that samples in that modality subspace can be attributed to a shared latent factor, and not an independent private one.

Original: In contrast, for unimportant samples in a modality, the correlation is minimal.
Revised: In contrast, for unimportant samples in a modality, the correlation is minimal.

Original: The main contributions of our proposed approach are two-fold.
Revised: The main contributions of our proposed approach are two-fold.

Original: The ﬁrst is a multimodal autoencoder framework where training requires additional loss terms which are O ( M ) , i.e. linear in the number of modalities, and thus only require M per-modality encoders to handle missing modalities.
Revised: The ﬁrst is a multimodal autoencoder framework where training requires additional loss terms which are O ( M ) , i.e. linear in the number of modalities, and thus only require M per-modality encoders to handle missing modalities.

Original: Computationally, this is advantageous compared to JMVAE-KL and MVAE, which require exponential number of sub-networks and loss terms respectively.
Revised: Computationally, this is advantageous compared to JMVAE-KL and MVAE, which require exponential number of sub-networks and loss terms respectively.

Original: Secondly , we deﬁne the concept of importance in an unsupervised setting, and propose novel cross-correlation based loss terms to learn important regions in each modality’s representation space.
Revised: Secondly , we deﬁne the concept of importance in an unsupervised setting, and propose novel cross-correlation based loss terms to learn important regions in each modality’s representation space.

Original: The importances are modeled by separate unimodal networks referred to as importance networks .
Revised: The importances are modeled by separate unimodal networks referred to as importance networks .

Original: Hyper-parameter ρ j for the j -th modality controls the integration of prior domain knowledge about the degree of importance in that modality.
Revised: Hyper-parameter ρ j for the j -th modality controls the integration of prior domain knowledge about the degree of importance in that modality.

Original: While not trained on any supervised labels, the learnt importances from IMA are analyzed quantitatively and found to correspond to the separation between digit vs. noise labels and emotion vs. neutral categories.
Revised: While not trained on any supervised labels, the learnt importances from IMA are analyzed quantitatively and found to correspond to the separation between digit vs. noise labels and emotion vs. neutral categories.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Following the great success of deep neural networks for representation learning, the research area of multimodal machine learning is gaining interest (Baltruˇsaitis et al., 2018).
Revised: Following the great success of deep neural networks for representation learning, the research area of multimodal machine learning is gaining interest (Baltruˇsaitis et al., 2018).

Original: Our proposed IMA model is relevant to two main research areas in this domain, Inter-modality Correlation Learning and Efﬁcient Multimodal VAEs .
Revised: Our proposed IMA model is relevant to two main research areas in this domain, Inter-modality Correlation Learning and Efﬁcient Multimodal VAEs .

Original: The idea of learning acoustic embeddings for words has also been explored in Wang et al.
Revised: The idea of learning acoustic embeddings for words has also been explored in Wang et al.

Original: (2018) and Jung et al.
Revised: (2018) and Jung et al.

Original: (2019) however we attempt to map words to their affective rather than phonetic representations.
Revised: (2019) however we attempt to map words to their affective rather than phonetic representations.

Original: In this section, we describe each area and conclude with the similarities and differences between the IMA model and prior approaches.
Revised: In this section, we describe each area and conclude with the similarities and differences between the IMA model and prior approaches.

Original: Inter-modality Correlation Learning : There have been several approaches which measure correlations between modalities/sources of data to understand how observed data in each modality can be explained by shared underlying concepts.
Revised: Inter-modality Correlation Learning : There have been several approaches which measure correlations between modalities/sources of data to understand how observed data in each modality can be explained by shared underlying concepts.

Original: The IBFA (Inter-Battery Factor Analysis) introduced by Tucker (1958) and its successor, the MBFA (Multi-battery factor Analysis) (Browne, 1980) are among the earliest proposed techniques to study shared factors between score sets from batteries of tests.
Revised: The IBFA (Inter-Battery Factor Analysis) introduced by Tucker (1958) and its successor, the MBFA (Multi-battery factor Analysis) (Browne, 1980) are among the earliest proposed techniques to study shared factors between score sets from batteries of tests.

Original: DeepCCA (Deep Canonical Covariance Analysis) proposed by Benton et al.
Revised: DeepCCA (Deep Canonical Covariance Analysis) proposed by Benton et al.

Original: (2017) learns a deep projection of each modality in a bimodal understanding scenario so that the projections are maximally correlated, effectively extending the classical CCA technique (Knapp, 1978) to deep neural networks.
Revised: (2017) learns a deep projection of each modality in a bimodal understanding scenario so that the projections are maximally correlated, effectively extending the classical CCA technique (Knapp, 1978) to deep neural networks.

Original: Our proposed model extends these approaches to also detect important regions of each modality correlated with the shared latent factor.
Revised: Our proposed model extends these approaches to also detect important regions of each modality correlated with the shared latent factor.

Original: Efﬁcient Multimodal VAEs : VAEs (Variational Autoencoders) have been applied to multimodal data for applications such as inference and bidirectional generation of modalities.
Revised: Efﬁcient Multimodal VAEs : VAEs (Variational Autoencoders) have been applied to multimodal data for applications such as inference and bidirectional generation of modalities.

Original: This poses a major challenge of constructing encoders to model the latent posterior which are efﬁcient in training/inference under any combination of input modalities.
Revised: This poses a major challenge of constructing encoders to model the latent posterior which are efﬁcient in training/inference under any combination of input modalities.

Original: Recent work addresses this by focusing on factorized models for efﬁcient inference.
Revised: Recent work addresses this by focusing on factorized models for efﬁcient inference.

Original: Vedantam et al.
Revised: Vedantam et al.

Original: (2017) employs a product-of-experts decomposition with modality speciﬁc inference networks to train image generation models.
Revised: (2017) employs a product-of-experts decomposition with modality speciﬁc inference networks to train image generation models.

Original: Wu &
Revised: Wu &

Original: Goodman (2018) propose MVAE (Multimodal Variational Autoencoders), where the latent posterior is modeled with a parameter shared product of experts network.
Revised: Goodman (2018) propose MVAE (Multimodal Variational Autoencoders), where the latent posterior is modeled with a parameter shared product of experts network.

Original: Shi et al.
Revised: Shi et al.

Original: (2019) proposed a mixture-of-experts multimodal variational autoencoder (MMVAE) where the posterior is a mixture of experts instead.
Revised: (2019) proposed a mixture-of-experts multimodal variational autoencoder (MMVAE) where the posterior is a mixture of experts instead.

Original: These approaches have been extended more recently, for example in multi-source neural variational inference (Kurle et al., 2019) where the multimodal posterior is constructed using learnt and integrated beliefs from multiple posteriors, each being informed by a different source.
Revised: These approaches have been extended more recently, for example in multi-source neural variational inference (Kurle et al., 2019) where the multimodal posterior is constructed using learnt and integrated beliefs from multiple posteriors, each being informed by a different source.

Original: Sutter et al.
Revised: Sutter et al.

Original: (2020) introduce a novel Jensen-Shannon divergence based objective function which can be used to approximate both unimodal and joint multimodal posteriors.
Revised: (2020) introduce a novel Jensen-Shannon divergence based objective function which can be used to approximate both unimodal and joint multimodal posteriors.

Original: While existing approaches attempt to efﬁciently learn multimodal representations through posterior modeling, our proposed IMA model aligns modalities during autoencoder training for projection to a common space which facilitates inference even in absence of modalities.
Revised: While existing approaches attempt to efﬁciently learn multimodal representations through posterior modeling, our proposed IMA model aligns modalities during autoencoder training for projection to a common space which facilitates inference even in absence of modalities.

Original: Only M encoders and
Revised: Only M encoders and

Original: O ( M ) loss terms are required by the IMA model for inference with M modalities.
Revised: O ( M ) loss terms are required by the IMA model for inference with M modalities.

Original: Prior work has also not focused sufﬁciently on unsupervised learning of modality importances (through detection of subspaces maximally correlated with shared latent factors) which we address in this paper.
Revised: Prior work has also not focused sufﬁciently on unsupervised learning of modality importances (through detection of subspaces maximally correlated with shared latent factors) which we address in this paper.

Original:  M ODEL D ESCRIPTION
Revised:  M ODEL D ESCRIPTION

Original: The IMA model consists of two main components : (1) the multimodal autoencoder and (2) the unimodal importance networks.
Revised: The IMA model consists of two main components : (1) the multimodal autoencoder and (2) the unimodal importance networks.

Original: In Figure 1 we have provided an overview diagram of the proposed model, including the loss functions utilized in training.
Revised: In Figure 1 we have provided an overview diagram of the proposed model, including the loss functions utilized in training.

Original: Multimodal Autoencoder: Assume that the input training examples consist of multimodal data, where each multimodal sample is denoted as x = { x 1 , x 2 , ...
Revised: Multimodal Autoencoder: Assume that the input training examples consist of multimodal data, where each multimodal sample is denoted as x = { x 1 , x 2 , ...

Original: x M } .
Revised: x M } .

Original: There are M modalities and the training set consists of N multimodal samples.
Revised: There are M modalities and the training set consists of N multimodal samples.

Original: The input data x j in the j -th modality is passed through an encoder for that modality and its output is denoted as u j ( x j ) .
Revised: The input data x j in the j -th modality is passed through an encoder for that modality and its output is denoted as u j ( x j ) .

Original: The latent multimodal representation z could be modeled with different approaches for example, concatenated fusion with fully connected layers (Suzuki et al., 2016).
Revised: The latent multimodal representation z could be modeled with different approaches for example, concatenated fusion with fully connected layers (Suzuki et al., 2016).

Original: We wish to model the latent multimodal representation z ( x 1 , x 2 ... x M ) with a pooling weighted by distinct modality importances y j
Revised: We wish to model the latent multimodal representation z ( x 1 , x 2 ... x M ) with a pooling weighted by distinct modality importances y j

Original: ( x j ) as given by:
Revised: ( x j ) as given by:

Original: [Equation]
Revised: [Equation]

Original: The multimodal representation z is passed through the decoder networks to obtain the reconstruction in the j -th modality as ˆ x j .
Revised: The multimodal representation z is passed through the decoder networks to obtain the reconstruction in the j -th modality as ˆ x j .

Original: L ( j ) rec is the reconstruction loss for the j -th modality.
Revised: L ( j ) rec is the reconstruction loss for the j -th modality.

Original: L glob is a global regularization term which encourages z to be centered with zero mean.
Revised: L glob is a global regularization term which encourages z to be centered with zero mean.

Original: This term is for regularization of the multimodal representation z and expands to L glob ( z ) = (cid:107) z
Revised: This term is for regularization of the multimodal representation z and expands to L glob ( z ) = (cid:107) z

Original: ( x 1 , x 2 , ... x M ) (cid:107) 2 .
Revised: ( x 1 , x 2 , ... x M ) (cid:107) 2 .

Original: Multimodal alignment in IMA : For our proposed model, we have M sub-networks for each modality, where each j -th unimodal sub-network is trained using the autoencoder reconstruction losses and an additional alignment loss L ( j ) align .
Revised: Multimodal alignment in IMA : For our proposed model, we have M sub-networks for each modality, where each j -th unimodal sub-network is trained using the autoencoder reconstruction losses and an additional alignment loss L ( j ) align .

Original: By forcing the multimodal representation z and each modality’s view u j ( x j ) to be similar, we also enforce each unimodal sub-network to learn its contribution to the latent factor z during autoencoder training.
Revised: By forcing the multimodal representation z and each modality’s view u j ( x j ) to be similar, we also enforce each unimodal sub-network to learn its contribution to the latent factor z during autoencoder training.

Original: Per-sample, this requires only M sub-networks and M additional loss terms instead of a random subset of 2 M losses for sub-sampled training as in the MVAE model (Wu & Goodman, 2018) or 2 M subnetworks as in the JMVAE-KL model (Suzuki et al., 2016).
Revised: Per-sample, this requires only M sub-networks and M additional loss terms instead of a random subset of 2 M losses for sub-sampled training as in the MVAE model (Wu & Goodman, 2018) or 2 M subnetworks as in the JMVAE-KL model (Suzuki et al., 2016).

Original: Similar to MVAE, this model also can learn and infer in the absence of modalities.
Revised: Similar to MVAE, this model also can learn and infer in the absence of modalities.

Original: z can be expressed as a linear average of the unimodal representations present and incorporated into the overall loss.
Revised: z can be expressed as a linear average of the unimodal representations present and incorporated into the overall loss.

Original: If the j -th modality is missing, we can set y j ( x j ) = 0 in Equation 1.
Revised: If the j -th modality is missing, we can set y j ( x j ) = 0 in Equation 1.

Original: For the j -th modality, L ( j ) align is the SSE error between the multimodal representation z and the unimodal encoder outputs u j ( x j ) given by: L ( j ) align ( z , u j )
Revised: For the j -th modality, L ( j ) align is the SSE error between the multimodal representation z and the unimodal encoder outputs u j ( x j ) given by: L ( j ) align ( z , u j )

Original: = (cid:107) z
Revised: = (cid:107) z

Original: ( x 1 , x 2 , ... x M ) − u j ( x j ) (cid:107) 

Revised: ( x 1 , x 2 , ... x M ) − u j ( x j ) (cid:107)

Original: When there are N training samples, during multimodal autoencoder learning the loss below is optimized:
Revised: When there are N training samples, during multimodal autoencoder learning the loss below is optimized:

Original: [Equation]
Revised: [Equation]

Original: λ align , λ rec and λ glob are the associated hyper-parameter weights for the alignment, reconstruction and global multimodal regularization terms which appear in Equation (2).
Revised: λ align , λ rec and λ glob are the associated hyper-parameter weights for the alignment, reconstruction and global multimodal regularization terms which appear in Equation (2).

Original: Since it is not straightforward to tune such parameters for an unsupervised model, we start with equal weights for all losses and observe if that is sufﬁcient to minimize all terms simultaneously during training.
Revised: Since it is not straightforward to tune such parameters for an unsupervised model, we start with equal weights for all losses and observe if that is sufﬁcient to minimize all terms simultaneously during training.

Original: For the
Revised: For the

Original: MNIST-TIDIGITS and IEMOCAP experiments, we have found that equally weighing them works, thus λ align = λ rec =
Revised: MNIST-TIDIGITS and IEMOCAP experiments, we have found that equally weighing them works, thus λ align = λ rec =

Original: λ glob = 1 . 0 .
Revised: λ glob = 1 . 0 .

Original: Importance Network Training: Section 1 explains that there are samples in observed data for each modality which do not correlate with the multimodal representation z .
Revised: Importance Network Training: Section 1 explains that there are samples in observed data for each modality which do not correlate with the multimodal representation z .

Original: For the scope of this paper, we consider such samples as uncorrelated noise in the data.
Revised: For the scope of this paper, we consider such samples as uncorrelated noise in the data.

Original: We have assumed the presence of a subspace R j inside each j -th modality’s representation space so that if x j ∈ R j , u j is minimally correlated with z .
Revised: We have assumed the presence of a subspace R j inside each j -th modality’s representation space so that if x j ∈ R j , u j is minimally correlated with z .

Original: We seek to learn weights y ij ∈ [0 , 1] for the i -th sample in the j − th modality, so that y ij denotes the importance of each sample x ij (i.e. the degree to which x ij does not belong to R j ).
Revised: We seek to learn weights y ij ∈ [0 , 1] for the i -th sample in the j − th modality, so that y ij denotes the importance of each sample x ij (i.e. the degree to which x ij does not belong to R j ).

Original: A unimodal neural network is trained to map from x ij to y ij for the j -th modality; this is called the importance network .
Revised: A unimodal neural network is trained to map from x ij to y ij for the j -th modality; this is called the importance network .

Original: The multimodal autoencoder solely being trained to reconstruct all modality inputs would not have learnt y ij corresponding to uncorrelated noise.
Revised: The multimodal autoencoder solely being trained to reconstruct all modality inputs would not have learnt y ij corresponding to uncorrelated noise.

Original: This motivates the need for explicitly deﬁning an importance-based loss.
Revised: This motivates the need for explicitly deﬁning an importance-based loss.

Original: Cross-correlation Losses: To train the importance networks, we have made use of a loss function explicitly capturing cross-correlation between z and u j , weighted by y j .
Revised: Cross-correlation Losses: To train the importance networks, we have made use of a loss function explicitly capturing cross-correlation between z and u j , weighted by y j .

Original: Minimizing this cost is equivalent to enforcing zero-correlation between u j and z based on mini-batch statistics during training, where size of a mini-batch is B samples.
Revised: Minimizing this cost is equivalent to enforcing zero-correlation between u j and z based on mini-batch statistics during training, where size of a mini-batch is B samples.

Original: This cost can be derived from the deﬁnition of cross-correlation, where each i -th sample in the mini-batch is not weighted equally.
Revised: This cost can be derived from the deﬁnition of cross-correlation, where each i -th sample in the mini-batch is not weighted equally.

Original: Recent work
Revised: Recent work

Original: 𝑳 𝒍𝒐𝒄𝒂𝒍(𝟏) 𝑳 𝒍𝒐𝒄𝒂𝒍(𝟐)
Revised: 𝑳 𝒍𝒐𝒄𝒂𝒍(𝟏) 𝑳 𝒍𝒐𝒄𝒂𝒍(𝟐)

Original: L a t e n t R e p r e s e n t a t i o n ( z ) + 𝑈 1 𝑋𝑈 2 𝑋𝑋𝑋𝑳 𝒂𝒍𝒊𝒈𝒏(𝟏) 𝑳 𝒂𝒍𝒊𝒈𝒏(𝟐) 𝑦𝑦𝑳 𝒄𝒐𝒓𝒓(𝟏) 𝑳 𝒄𝒐𝒓𝒓(𝟐) 𝑳 𝒈𝒍𝒐𝒃 𝑳 𝒓𝒆𝒄(𝟏)
Revised: L a t e n t R e p r e s e n t a t i o n ( z ) + 𝑈 1 𝑋𝑈 2 𝑋𝑋𝑋𝑳 𝒂𝒍𝒊𝒈𝒏(𝟏) 𝑳 𝒂𝒍𝒊𝒈𝒏(𝟐) 𝑦𝑦𝑳 𝒄𝒐𝒓𝒓(𝟏) 𝑳 𝒄𝒐𝒓𝒓(𝟐) 𝑳 𝒈𝒍𝒐𝒃 𝑳 𝒓𝒆𝒄(𝟏)

Original: 𝑳 𝒓𝒆𝒄(𝟐)
Revised: 𝑳 𝒓𝒆𝒄(𝟐)

Original: D e c o d e r N e t w o r k
Revised: D e c o d e r N e t w o r k

Original: s
Revised: s

Original: E n c o d e r N e t w o r k
Revised: E n c o d e r N e t w o r k

Original: s
Revised: s

Original: [Figure]
Revised: [Figure]

Original: has also applied other independence criterion, such as MMD (Maximum Mean Discrepancies) which act as auxiliary losses in a variational framework (Louizos et al., 2015).
Revised: has also applied other independence criterion, such as MMD (Maximum Mean Discrepancies) which act as auxiliary losses in a variational framework (Louizos et al., 2015).

Original: Assuming the latent variables are Gaussians, independence and uncorrelated properties are equivalent, and we deﬁne the following alternative loss term based on the Frobenius norm of the cross-covariance between z and u j , where y (cid:48) ij = 1 − y ij and w (cid:48) ij = y (cid:48) ij (cid:30) B (cid:80) i =1 y (cid:48) ij
Revised: Assuming the latent variables are Gaussians, independence and uncorrelated properties are equivalent, and we deﬁne the following alternative loss term based on the Frobenius norm of the cross-covariance between z and u j , where y (cid:48) ij = 1 − y ij and w (cid:48) ij = y (cid:48) ij (cid:30) B (cid:80) i =1 y (cid:48) ij

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: j
Revised: j

Original: [Equation]
Revised: [Equation]

Original: corr =
Revised: corr =

Original: [Equation]
Revised: [Equation]

Original: = w
Revised: = w

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
Revised: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original: F µ z =
Revised: F µ z =

Original: B (cid:88) i = w (cid:48) ij z i µ u =
Revised: B (cid:88) i = w (cid:48) ij z i µ u =

Original: B (cid:88) i = w (cid:48) ij
Revised: B (cid:88) i = w (cid:48) ij

Original: [Equation]
Revised: [Equation]

Original: Importance Priors: Learning to predict y ij ≈ 

Revised: Importance Priors: Learning to predict y ij ≈

Original: . 0 ∀
Revised: . 0 ∀

Original: i, j trivially decreases L corr down to zero, and thus we need to regularize the importance network training, through an additional loss function which we refer to as L ( j ) local .
Revised: i, j trivially decreases L corr down to zero, and thus we need to regularize the importance network training, through an additional loss function which we refer to as L ( j ) local .

Original: This loss utilizes the hyper-parameter ρ j ∈ [0 , 1] which serves as prior about how much of the j -th modality is corrupted by uncorrelated noise, and is deﬁned as the
Revised: This loss utilizes the hyper-parameter ρ j ∈ [0 , 1] which serves as prior about how much of the j -th modality is corrupted by uncorrelated noise, and is deﬁned as the

Original: KL-divergence D KL ( Bernoulli (¯ y j ) ||
Revised: KL-divergence D KL ( Bernoulli (¯ y j ) ||

Original: Bernoulli ( ρ j )) where ¯ y j = (cid:80) i y
Revised: Bernoulli ( ρ j )) where ¯ y j = (cid:80) i y

Original: ij /B is the average value of y ij as computed over a mini-batch of size B .
Revised: ij /B is the average value of y ij as computed over a mini-batch of size B .

Original: Enforcing the loss L local at a sample level would have forced y ij =
Revised: Enforcing the loss L local at a sample level would have forced y ij =

Original: ρ j for all samples, which is addressed by deﬁning the loss on a mini-batch instead.
Revised: ρ j for all samples, which is addressed by deﬁning the loss on a mini-batch instead.

Original: The importance network for each modality minimizes L imp , which is the weighted sum of the cross-covariance based loss and the regularization term, as deﬁned below.
Revised: The importance network for each modality minimizes L imp , which is the weighted sum of the cross-covariance based loss and the regularization term, as deﬁned below.

Original: λ ( j ) local and λ ( j ) corr are hyper-parameter weights for each loss term.
Revised: λ ( j ) local and λ ( j ) corr are hyper-parameter weights for each loss term.

Original: [Equation]
Revised: [Equation]

Original: It is important to note the difference between y ij and y (cid:48) ij = 1 − y ij in terms of notation.
Revised: It is important to note the difference between y ij and y (cid:48) ij = 1 − y ij in terms of notation.

Original: y ij is the importance of the i − th sample in the j − th modality and is required in the joint representation, whereas its complement y (cid:48) ij is utilized in importance network training.
Revised: y ij is the importance of the i − th sample in the j − th modality and is required in the joint representation, whereas its complement y (cid:48) ij is utilized in importance network training.

Original: The intuition is that a model minimizing Equation 3 would tend to learn high values of y (cid:48) ij for maximally uncorrelated samples, which translates to low importance ( y ij ) for these same samples.
Revised: The intuition is that a model minimizing Equation 3 would tend to learn high values of y (cid:48) ij for maximally uncorrelated samples, which translates to low importance ( y ij ) for these same samples.

Original:  D ATASETS
Revised:  D ATASETS

Original: We perform experiments with the proposed IMA model on (1) MNIST-TIDIGITS which is a multimodal digit dataset created by pairing image and spoken samples from common image and speech digit datasets and (2) the IEMOCAP corpus (Busso et al., 2008).
Revised: We perform experiments with the proposed IMA model on (1) MNIST-TIDIGITS which is a multimodal digit dataset created by pairing image and spoken samples from common image and speech digit datasets and (2) the IEMOCAP corpus (Busso et al., 2008).

Original: MNIST-TIDIGITS is synthesized from three datasets for digit recognition from images and speech - MNIST dataset (Salakhutdinov &
Revised: MNIST-TIDIGITS is synthesized from three datasets for digit recognition from images and speech - MNIST dataset (Salakhutdinov &

Original: Hinton, 2007); TIDIGITS connected spoken digit sequence corpus (Leonard, 1984) and TI46 digits (Liberman, 1993).
Revised: Hinton, 2007); TIDIGITS connected spoken digit sequence corpus (Leonard, 1984) and TI46 digits (Liberman, 1993).

Original: We have merged these two spoken digit datasets to create MNIST-TIDIGITS, a dataset as large as the number of samples in MNIST.
Revised: We have merged these two spoken digit datasets to create MNIST-TIDIGITS, a dataset as large as the number of samples in MNIST.

Original: The motivation underlying this is to pair image and spoken digits to create a multimodal dataset.
Revised: The motivation underlying this is to pair image and spoken digits to create a multimodal dataset.

Original: USC-IEMOCAP is a well-known dataset for speech emotion recognition comprising of acted and spontaneous multimodal interactions of dyadic sessions between human actors.
Revised: USC-IEMOCAP is a well-known dataset for speech emotion recognition comprising of acted and spontaneous multimodal interactions of dyadic sessions between human actors.

Original: The dataset consists of around 12 hours of speech from 10 human subjects, and is labeled by three annotators for emotions such as Happy , Sad , Angry , Excitement ,
Revised: The dataset consists of around 12 hours of speech from 10 human subjects, and is labeled by three annotators for emotions such as Happy , Sad , Angry , Excitement ,

Original: Neutral and Surprise .
Revised: Neutral and Surprise .

Original: We train IMA model on all utterances, but perform classiﬁcation/retrieval experiments only for the four standard primary emotion categories - Neutral , Angry , Sad and Happy , with a total of 5531 utterances belonging to these categories (Kim et al., 2013; Jin et al., 2015).
Revised: We train IMA model on all utterances, but perform classiﬁcation/retrieval experiments only for the four standard primary emotion categories - Neutral , Angry , Sad and Happy , with a total of 5531 utterances belonging to these categories (Kim et al., 2013; Jin et al., 2015).

Original: MNIST-TIDIGITS: We construct a multimodal dataset by pairing the MNIST digit images with the TIDIGIT spoken digits based on the common digit label.
Revised: MNIST-TIDIGITS: We construct a multimodal dataset by pairing the MNIST digit images with the TIDIGIT spoken digits based on the common digit label.

Original: We deﬁne a parameter R called the noise factor to control the amount by which each digit image or spoken digit co-occurs with synthesized noise samples.
Revised: We deﬁne a parameter R called the noise factor to control the amount by which each digit image or spoken digit co-occurs with synthesized noise samples.

Original: For each label category c with N image/spoken digits, we create three sets of multimodal samples: N ∗ R samples consisting of images of digit c paired with synthetic spoken noise; N ∗ R samples consisting of synthetic image noise paired with samples of spoken digit c and N ∗ (1 − R ) samples consisting of images of digit c paired with spoken digit c .
Revised: For each label category c with N image/spoken digits, we create three sets of multimodal samples: N ∗ R samples consisting of images of digit c paired with synthetic spoken noise; N ∗ R samples consisting of synthetic image noise paired with samples of spoken digit c and N ∗ (1 − R ) samples consisting of images of digit c paired with spoken digit c .

Original: For all multimodal samples, the ground truth labels are set to the actual digit labels, disregarding the synthetic noise.
Revised: For all multimodal samples, the ground truth labels are set to the actual digit labels, disregarding the synthetic noise.

Original: For example, for a multimodal sample with (image ’7’, spoken noise), the sample label is set to 7.
Revised: For example, for a multimodal sample with (image ’7’, spoken noise), the sample label is set to 7.

Original: Note that the sample label is not known at model training time.
Revised: Note that the sample label is not known at model training time.

Original: IEMOCAP: Consider each IEMOCAP spoken utterance to have N word tokens and wordlevel acoustic feature vectors (for example, descriptors such as pitch).
Revised: IEMOCAP: Consider each IEMOCAP spoken utterance to have N word tokens and wordlevel acoustic feature vectors (for example, descriptors such as pitch).

Original: We pair each token with all N acoustic vectors in the same utterance, creating N paired multimodal samples.
Revised: We pair each token with all N acoustic vectors in the same utterance, creating N paired multimodal samples.

Original: Thus if an
Revised: Thus if an

Original: IEMOCAP utterance initially has N word-acoustic pairs, it is expanded to N 2 pairs for model training.
Revised: IEMOCAP utterance initially has N word-acoustic pairs, it is expanded to N 2 pairs for model training.

Original: We also remove the MFCC features, and thus most of the local phonetic variability within the utterance is removed resulting in global factors of variation, such as emotion.
Revised: We also remove the MFCC features, and thus most of the local phonetic variability within the utterance is removed resulting in global factors of variation, such as emotion.

Original: The scope of the non-verbal acoustic features co-occurring with a word is expanded to the entire utterance.
Revised: The scope of the non-verbal acoustic features co-occurring with a word is expanded to the entire utterance.

Original: This is analogous to techniques such as skip-gram word embeddings (Mikolov et al., 2013) where a word context spans beyond adjacent words, resulting in an expansion of word pairs to train on.
Revised: This is analogous to techniques such as skip-gram word embeddings (Mikolov et al., 2013) where a word context spans beyond adjacent words, resulting in an expansion of word pairs to train on.

Original:  E XPERIMENTAL S ETUP
Revised:  E XPERIMENTAL S ETUP

Original: In this section, we describe the experimental setup including downstream experiments to evaluate the quality of the multimodal representations learnt by the proposed IMA model.
Revised: In this section, we describe the experimental setup including downstream experiments to evaluate the quality of the multimodal representations learnt by the proposed IMA model.

Original: The model training is performed in two stages.
Revised: The model training is performed in two stages.

Original: In the ﬁrst stage the importance network weights are kept constant, and the multimodal autoencoder is trained.
Revised: In the ﬁrst stage the importance network weights are kept constant, and the multimodal autoencoder is trained.

Original: In the second stage the modality-speciﬁc importance networks are trained; with multimodal autoencoder weights kept constant.
Revised: In the second stage the modality-speciﬁc importance networks are trained; with multimodal autoencoder weights kept constant.

Original: This sequence of alternating optimization could be performed for a ﬁnite number of iterations, but in practice we have found one stage to be sufﬁcient.
Revised: This sequence of alternating optimization could be performed for a ﬁnite number of iterations, but in practice we have found one stage to be sufﬁcient.

Original: The image and speech modality inputs for the MNIST-TIDIGITS datasets have 784 and 696 dimensions respectively.
Revised: The image and speech modality inputs for the MNIST-TIDIGITS datasets have 784 and 696 dimensions respectively.

Original: The two-layer encoder and decoder networks for the multimodal autoencoder have 500 neurons/layer for image and 100 neurons/layer for the speech modality.
Revised: The two-layer encoder and decoder networks for the multimodal autoencoder have 500 neurons/layer for image and 100 neurons/layer for the speech modality.

Original: A binary cross-entropy loss is used for autoencoder training.
Revised: A binary cross-entropy loss is used for autoencoder training.

Original: For IEMOCAP, the verbal embeddings (GLOVE) and acoustic features have 200 and 55 dimensions respectively.
Revised: For IEMOCAP, the verbal embeddings (GLOVE) and acoustic features have 200 and 55 dimensions respectively.

Original: The twolayer autoencoder has 100 neurons/layer in both modalities, and the SSE (Sum of Squared Error) objective is used for training.
Revised: The twolayer autoencoder has 100 neurons/layer in both modalities, and the SSE (Sum of Squared Error) objective is used for training.

Original: 5. I MPORTANCE P RIOR
Revised: 5. I MPORTANCE P RIOR

Original: We refer to the parameter ρ j ∈ [0 , 1] as the importance prior for the j -th modality.
Revised: We refer to the parameter ρ j ∈ [0 , 1] as the importance prior for the j -th modality.

Original: ρ j can be adjusted to control the proportion of samples detected as uncorrelated noise and is related to domain knowledge.
Revised: ρ j can be adjusted to control the proportion of samples detected as uncorrelated noise and is related to domain knowledge.

Original: For example in IEMOCAP, ρ for text can be set to a very low value, since we hypothesize that a vast majority of vocabulary words (e.g. function words) do not correlate with the latent factor (such as emotion).
Revised: For example in IEMOCAP, ρ for text can be set to a very low value, since we hypothesize that a vast majority of vocabulary words (e.g. function words) do not correlate with the latent factor (such as emotion).

Original: Similarly for the MNIST-TIDIGITS dataset, the amount of uncorrelated noise (the aforementioned noise factor in Section 4) in the paired multimodal dataset is 10%, requiring a high value of ρ j since ∼ 90% of samples in each modality correspond to valid digits.
Revised: Similarly for the MNIST-TIDIGITS dataset, the amount of uncorrelated noise (the aforementioned noise factor in Section 4) in the paired multimodal dataset is 10%, requiring a high value of ρ j since ∼ 90% of samples in each modality correspond to valid digits.

Original: 5. E VALUATION OF I MPORTANCE N ETWORK P ERFORMANCE
Revised: 5. E VALUATION OF I MPORTANCE N ETWORK P ERFORMANCE

Original: Since the model is unsupervised where the actual importances are not available for training, there is no quantitative way to determine the best value of the hyper-parameter ρ j for each problem.
Revised: Since the model is unsupervised where the actual importances are not available for training, there is no quantitative way to determine the best value of the hyper-parameter ρ j for each problem.

Original: However we have utilized the ground truth labels from the MNIST-TIDIGITS and IEMOCAP datasets to evaluating performance of the modality-speciﬁc importance networks.
Revised: However we have utilized the ground truth labels from the MNIST-TIDIGITS and IEMOCAP datasets to evaluating performance of the modality-speciﬁc importance networks.

Original: For the i -th sample in the j -th modality, the importance network maps input x ij to an importance score y ij ∈ [0 , 1] .
Revised: For the i -th sample in the j -th modality, the importance network maps input x ij to an importance score y ij ∈ [0 , 1] .

Original: Given the ground truth assignment of x ij to a positive or negative class depending on whether it is uncorrelated noise, we can compute F1 scores to determine the optimal importance prior parameter, as well as quantify importance network performance with varying values of ρ j .
Revised: Given the ground truth assignment of x ij to a positive or negative class depending on whether it is uncorrelated noise, we can compute F1 scores to determine the optimal importance prior parameter, as well as quantify importance network performance with varying values of ρ j .

Original: For the IEMOCAP dataset, the important verbal category is deﬁned to be the class of all vocabulary words belonging to the LIWC emotional categories (Pennebaker et al., 2001), namely - sad , angry , anxiety , negative emotion and positive emotion .
Revised: For the IEMOCAP dataset, the important verbal category is deﬁned to be the class of all vocabulary words belonging to the LIWC emotional categories (Pennebaker et al., 2001), namely - sad , angry , anxiety , negative emotion and positive emotion .

Original: In the acoustic modality, the important category is deﬁned to be the class of non-verbal acoustic samples belonging to the happy , angry and sad emotions as annotated in the IEMOCAP dataset.
Revised: In the acoustic modality, the important category is deﬁned to be the class of non-verbal acoustic samples belonging to the happy , angry and sad emotions as annotated in the IEMOCAP dataset.

Original: For MNIST-TIDIGITS, importance is indicated by the presence of actual images/spoken digits in any of the modalities.
Revised: For MNIST-TIDIGITS, importance is indicated by the presence of actual images/spoken digits in any of the modalities.

Original: Uncorrelated synthetic noise is not important, and is expected to be detected by the importance networks.
Revised: Uncorrelated synthetic noise is not important, and is expected to be detected by the importance networks.

Original: 5. M ULTIMODAL R EPRESENTATIONS
Revised: 5. M ULTIMODAL R EPRESENTATIONS

Original: MNIST-TIDIGITS:
Revised: MNIST-TIDIGITS:

Original: IMA learns importances y ij for the j -th modality (images: j = 0 /speech:
Revised: IMA learns importances y ij for the j -th modality (images: j = 0 /speech:

Original: j = 1 ) and for the i − th multimodal sample, if u ij are outputs of unimodal encoders, the multimodal representation is obtained by: z i =
Revised: j = 1 ) and for the i − th multimodal sample, if u ij are outputs of unimodal encoders, the multimodal representation is obtained by: z i =

Original: ( y i 0 u i0 + y i 1 u i1 ) /
Revised: ( y i 0 u i0 + y i 1 u i1 ) /

Original: ( y i 

Revised: ( y i

Original: + y i 1 )
Revised: + y i 1 )

Original: IEMOCAP: IMA learns token-level importances from the co-occurrences of words (modality 0) and their paired acoustic feature vectors (modality 1), which we combine to produce utterance representations.
Revised: IEMOCAP: IMA learns token-level importances from the co-occurrences of words (modality 0) and their paired acoustic feature vectors (modality 1), which we combine to produce utterance representations.

Original: Each i -th utterance of length L comprises of a bag of words { w i 1 , w i 2 , · · · w iL } and a bag of acoustic vectors { v i1 , v i2 , · · · v iL } .
Revised: Each i -th utterance of length L comprises of a bag of words { w i 1 , w i 2 , · · · w iL } and a bag of acoustic vectors { v i1 , v i2 , · · · v iL } .

Original: Their respective unimodal representations are { u 0 ( w i 1 ) , · · · u 0 ( w iL ) } and { u 1 ( v i1 ) , · · · u 1 ( v iL ) } .
Revised: Their respective unimodal representations are { u 0 ( w i 1 ) , · · · u 0 ( w iL ) } and { u 1 ( v i1 ) , · · · u 1 ( v iL ) } .

Original: Corresponding importances are { y 0 ( w i 1 ) , · · · y 0 ( w iL ) } and { y 1 ( v i1 ) , · · · y 1 ( v iL ) } .
Revised: Corresponding importances are { y 0 ( w i 1 ) , · · · y 0 ( w iL ) } and { y 1 ( v i1 ) , · · · y 1 ( v iL ) } .

Original: The multimodal representation of the whole utterance s i is given by:
Revised: The multimodal representation of the whole utterance s i is given by:

Original: [Equation]
Revised: [Equation]

Original: We also compare multimodal performance with representations obtained only from the unimodal encoders.
Revised: We also compare multimodal performance with representations obtained only from the unimodal encoders.

Original: In all cases, we also perform an ablation experiment which does not utilize the learnt importances y 0 or y 1 and computes an unweighted average of the unimodal representations.
Revised: In all cases, we also perform an ablation experiment which does not utilize the learnt importances y 0 or y 1 and computes an unweighted average of the unimodal representations.

Original: 5. R ETRIEVAL E XPERIMENTS
Revised: 5. R ETRIEVAL E XPERIMENTS

Original: We conduct an intrinsic evaluation of the multimodal representation quality through retrieval experiments.
Revised: We conduct an intrinsic evaluation of the multimodal representation quality through retrieval experiments.

Original: Even if a modality is corrupted through uncorrelated noise, the multimodal representation should ﬁlter it out by weighing it less.
Revised: Even if a modality is corrupted through uncorrelated noise, the multimodal representation should ﬁlter it out by weighing it less.

Original: Ideally, two samples with the same label should have similar representations.
Revised: Ideally, two samples with the same label should have similar representations.

Original: We deﬁne the concept of relevance as: Two multimodal samples are relevant if they correspond to the same ground truth label, even with noise in any of the modalities.
Revised: We deﬁne the concept of relevance as: Two multimodal samples are relevant if they correspond to the same ground truth label, even with noise in any of the modalities.

Original: For each of i ∈ { 1 , 2 , 3 ...N } , we obtain the K nearest neighbors of the i -th sample in terms of the Euclidean distance and count the number of relevant neighbors as C i .
Revised: For each of i ∈ { 1 , 2 , 3 ...N } , we obtain the K nearest neighbors of the i -th sample in terms of the Euclidean distance and count the number of relevant neighbors as C i .

Original: The Precision@K score is computed as
Revised: The Precision@K score is computed as

Original: Pr@K =
Revised: Pr@K =

Original: (cid:80) i C
Revised: (cid:80) i C

Original: i /NK .
Revised: i /NK .

Original: MNIST-TIDIGITS: Results are reported both for two and 50-dimensional representations at K ∈ { 10 , 50 } .
Revised: MNIST-TIDIGITS: Results are reported both for two and 50-dimensional representations at K ∈ { 10 , 50 } .

Original: The ground truth label is the digit label.
Revised: The ground truth label is the digit label.

Original: The representation of a (MNIST image digit, speech noise) pair should be similar to the representation of (image noise, spoken TIDIGITS digit) if the underlying digit labels are the same.
Revised: The representation of a (MNIST image digit, speech noise) pair should be similar to the representation of (image noise, spoken TIDIGITS digit) if the underlying digit labels are the same.

Original: IEMOCAP:
Revised: IEMOCAP:

Original: The retrieval experiments on the IEMOCAP dataset are at utterance level, and consider the primary emotion categories as ground truth labels.
Revised: The retrieval experiments on the IEMOCAP dataset are at utterance level, and consider the primary emotion categories as ground truth labels.

Original: For the retrieval experiments, we deﬁne two utterances to be relevant if they occur in the same
Revised: For the retrieval experiments, we deﬁne two utterances to be relevant if they occur in the same

Original: IEMOCAP primary emotion category, one out of: happy, angry, sad and neutral .
Revised: IEMOCAP primary emotion category, one out of: happy, angry, sad and neutral .

Original: The Precision@K scores for K = 10 are reported for 100-dimensional representations.
Revised: The Precision@K scores for K = 10 are reported for 100-dimensional representations.

Original: [Figure]
Revised: [Figure]

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: We have compared performances of the following models in our experiments: (1) Joint multimodal representations (unweighted and importance-weighted) from proposed IMA model as described in
Revised: We have compared performances of the following models in our experiments: (1) Joint multimodal representations (unweighted and importance-weighted) from proposed IMA model as described in

Original: Section 5.3.
Revised: Section 5.3.

Original: (2) Unimodal representations from IMA model which just considers the output u j of each of the j -th modality-speciﬁc encoders as the ﬁnal representation.
Revised: (2) Unimodal representations from IMA model which just considers the output u j of each of the j -th modality-speciﬁc encoders as the ﬁnal representation.

Original: (3) MVAE model proposed in Wu & Goodman (2018) (4) JMVAE-KL model proposed in Suzuki et al.
Revised: (3) MVAE model proposed in Wu & Goodman (2018) (4) JMVAE-KL model proposed in Suzuki et al.

Original: 6. I MPORTANCE N ETWORKS
Revised: I MPORTANCE N ETWORKS

Original: We evaluate the performance of the importance networks by evaluating the learned importances in each modality against domain knowledge based labels for uncorrelated noise presence.
Revised: We evaluate the performance of the importance networks by evaluating the learned importances in each modality against domain knowledge based labels for uncorrelated noise presence.

Original: MNIST-TIDIGITS:
Revised: MNIST-TIDIGITS:

Original: On MNIST-TIDIGITS we record precision, recall and F1 scores for values of ρ ∈ [0 . 05 − 0 . 95] .
Revised: On MNIST-TIDIGITS we record precision, recall and F1 scores for values of ρ ∈ [0 . 05 − 0 . 95] .

Original: The best F1 score occurs at ρ = 0.8 and 0.3, which indicates that these would be the optimal values for best performance in  image and speech modalities respectively.
Revised: The best F1 score occurs at ρ = 0.8 and 0.3, which are empirically the suitable priors for the importance network predictions to agree with external labels in the image and speech modalities respectively.

Original: For these optimal values of ρ in each modality, we visualize the top layer of each importance network which implements y ij for the j -th modality given x ij .
Revised: For these optimal values of ρ in each modality, we visualize the top layer of each importance network which implements y ij for the j -th modality given x ij .

Original: Figure 2(a-d) shows the t-SNE representations for MNIST-TIDIGITS, with both the ground truth label clusters and the importance network outputs.
Revised: Figure 2(a-d) shows the t-SNE representations for MNIST-TIDIGITS, with both the ground truth label clusters and the importance network outputs.

Original: We observe that the network correctly learns the regions of the representation which correspond to uncorrelated noise.
Revised: We observe that the network correctly learns the regions of the representation which correspond to uncorrelated noise.

Original: Around 2% of the data is misclassiﬁed by the importance network, which manifests in the small red region in Figures 2(b, d).
Revised: Around 2% of the data is misclassiﬁed by the importance network, which manifests in the small red region in Figures 2(b, d).

Original: The t-SNE visualizations for all the representations in the paper are obtained using a perplexity of 10.0 and a delta of 0.5.
Revised: The t-SNE visualizations for all the representations in the paper are obtained using a perplexity of 10.0 and a delta of 0.5.

Original: IEMOCAP: We recorded F1 scores for the importance network with ρ varying in [0 . 0 , 1 . 0] for two modalities: (a) vocabulary words and (b) acoustic non-verbal features.
Revised: IEMOCAP: We recorded F1 scores for the importance network with ρ varying in [0 . 0 , 1 . 0] for two modalities: (a) vocabulary words and (b) acoustic non-verbal features.

Original: The evaluation is performed on the validation set.
Revised: The evaluation is performed on the validation set.

Original: The F1 scores are computed for both the positive and negative classes in each modality, along with the average scores.
Revised: The F1 scores are computed for both the positive and negative classes in each modality, along with the average scores.

Original: The average F1 score for words peaks at ρ = 0 .
Revised: The average F1 score for words peaks at ρ = 0 .

Original: 03 (value of 0.595), and for the acoustic modality at ρ = 0 .
Revised: 03 (value of 0.595), and for the acoustic modality at ρ = 0 .

Original: 7 (value of 0.508).
Revised: 7 (value of 0.508).

Original: Since a lot of vocabulary words do not have emotional information, ρ at which the average verbal F1 score reaches maximum is a small value.
Revised: Since a lot of vocabulary words do not have emotional information, ρ at which the average verbal F1 score reaches maximum is a small value.

Original: We compare with a random classiﬁer parameterized on r which labels each sample as True with probability r and False with probability 1 − r , where r ∈ [0 , 1] .
Revised: We compare with a random classiﬁer parameterized on r which labels each sample as True with probability r and False with probability 1 − r , where r ∈ [0 , 1] .

Original: For vocabulary words, the best average F1 score obtained was 0.552 at r = 0 .
Revised: For vocabulary words, the best average F1 score obtained was 0.552 at r = 0 .

Original: 2 ; and
Revised: 2 ; and

Original: in the acoustic modality the best average F score obtained was 0.500 at r = 0 .
Revised: in the acoustic modality the best average F score obtained was 0.500 at r = 0 .

Original: In both cases, the best F1 score obtained from the importance network outperforms that of the random baseline.
Revised: In both cases, the best F1 score obtained from the importance network outperforms that of the random baseline.

Original: We also qualitatively examine the word-level importances learnt by the IMA model for IEMOCAP.
Revised: We also qualitatively examine the word-level importances learnt by the IMA model for IEMOCAP.

Original: For the best performing value of ρ , Figureshows ﬁve example utterances and for each utterance, the words are shaded from light to dark in order of increasing importance.
Revised: For the best performing value of ρ , Figureshows ﬁve example utterances and for each utterance, the words are shaded from light to dark in order of increasing importance.

Original: Emotional words such as temperamental , boorish and exaggerated are detected as important by the model while function words ( to , the , pause ’sil’ are ignored.
Revised: Emotional words such as temperamental , boorish and exaggerated are detected as important by the model while function words ( to , the , pause ’sil’ are ignored.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 6. R ETRIEVAL E XPERIMENTS
Revised: 6. R ETRIEVAL E XPERIMENTS

Original: MNIST-TIDIGITS: Multimodal representations are computed by weighing unimodal image/speech digit representations with learned importances as described in Section 5.3.
Revised: MNIST-TIDIGITS: Multimodal representations are computed by weighing unimodal image/speech digit representations with learned importances as described in Section 5.3.

Original: The IMA model would be expected to ﬁlter uncorrelated noise and map a paired combination of a (noise sample in one modality + a true digit in the other modality) to a cluster corresponding to the true digit.
Revised: The IMA model would be expected to ﬁlter uncorrelated noise and map a paired combination of a (noise sample in one modality + a true digit in the other modality) to a cluster corresponding to the true digit.

Original: This property may not be observed in other approaches to multimodal representations including baselines (Suzuki et al., 2016; Wu & Goodman, 2018).
Revised: This property may not be observed in other approaches to multimodal representations including baselines (Suzuki et al., 2016; Wu & Goodman, 2018).

Original: Representation quality can be evaluated through retrieval experiments as explained in Section 5.4.
Revised: Representation quality can be evaluated through retrieval experiments as explained in Section 5.4.

Original: Table 1 presents Precision@K scores (which are in the range [0,1]) at K =10 and 50 for the test set of MNIST-TIDIGITS.
Revised: Table 1 presents Precision@K scores (which are in the range [0,1]) at K =10 and 50 for the test set of MNIST-TIDIGITS.

Original: We compute precision results for the IMA model as well as the competing baselines, including a multimodal denoising autoencoder (with denoising rate=0.3).
Revised: We compute precision results for the IMA model as well as the competing baselines, including a multimodal denoising autoencoder (with denoising rate=0.3).

Original: The importance-weighted multimodal representations outperform the JMVAE-KL and MVAE models (and also the unweighted multimodal representation from IMA) due to the model’s ability to ﬁlter uncorrelated noise in each modality.
Revised: The importance-weighted multimodal representations outperform the JMVAE-KL and MVAE models (and also the unweighted multimodal representation from IMA) due to the model’s ability to ﬁlter uncorrelated noise in each modality.

Original: The
Revised: The

Original: JMVAE-KL model does not weigh modalities, and while the IMA model does this when computing the product of experts, those weights do not take into account regions of each modality having uncorrelated noise.
Revised: JMVAE-KL model does not weigh modalities, and while the IMA model does this when computing the product of experts, those weights do not take into account regions of each modality having uncorrelated noise.

Original: The denoising autoencoder obtains multimodal precisions (2D Pr@10: 0.
Revised: The denoising autoencoder obtains multimodal precisions (2D Pr@10: 0.

Original: ; Pr@50: 0.5488 ; 50D Pr@10: 0.8968 ; Pr@50: 0.7818) which are worse than the IMA model.
Revised: ; Pr@50: 0.5488 ; 50D Pr@10: 0.8968 ; Pr@50: 0.7818) which are worse than the IMA model.

Original: We also report performance when only one modality (either images or speech) is present at inference time.
Revised: We also report performance when only one modality (either images or speech) is present at inference time.

Original: For the image modality the IMA model performs the best, while for speech JMVAE-KL provides the highest precision.
Revised: For the image modality the IMA model performs the best, while for speech JMVAE-KL provides the highest precision.

Original: IEMOCAP: Section 5.3 describes how the importance-weighted unimodal and multimodal representations are obtained for each utterance, along with representations which do not consider learned importances.
Revised: IEMOCAP: Section 5.3 describes how the importance-weighted unimodal and multimodal representations are obtained for each utterance, along with representations which do not consider learned importances.

Original: Table 2 shows the Precision@K scores on the validation set for the multimodal representation z obtained from the IMA model and the comparison baselines.
Revised: Table 2 shows the Precision@K scores on the validation set for the multimodal representation z obtained from the IMA model and the comparison baselines.

Original: For all approaches, the verbal representation performance is lower than acoustic non-verbal modality which is due to the GLOVE representations capturing semantic (and not necessarily emotional) information from words.
Revised: For all approaches, the verbal representation performance is lower than acoustic non-verbal modality which is due to the GLOVE representations capturing semantic (and not necessarily emotional) information from words.

Original: The multimodal representations also have superior Precision@K scores compared to the unimodal performances.
Revised: The multimodal representations also have superior Precision@K scores compared to the unimodal performances.

Original: The improvement from learning importances is observed for the verbal modality, where it leads to a 2.67% improvement in the Precision@10 metric.
Revised: The improvement from learning importances is observed for the verbal modality, where it leads to a 2.67% improvement in the Precision@10 metric.

Original: This improvement occurs as the weighted utterance representations are obtained by ﬁltering low importance (ex. func-
Revised: This improvement occurs as the weighted utterance representations are obtained by ﬁltering low importance (ex. func-

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: tion words) out from the utterance.
Revised: tion words) out from the utterance.

Original: The weighted verbal representations from the IMA model outperform the JMVAE-KL and MVAE baselines on the Precision@10 metric, however corresponding scores are less on the acoustic and multimodal representations.
Revised: The weighted verbal representations from the IMA model outperform the JMVAE-KL and MVAE baselines on the Precision@10 metric, however corresponding scores are less on the acoustic and multimodal representations.

Original: 6. V ISUALIZATION OF L EARNT R EPRESENTATIONS
Revised: 6. V ISUALIZATION OF L EARNT R EPRESENTATIONS

Original: MNIST-TIDIGITS :
Revised: MNIST-TIDIGITS :

Original: In Figure 4(a, b), the unimodal representations u j for j ∈ { 1 , 2 } are presented, along with the joint multimodal representation of the paired samples.
Revised: In Figure 4(a, b), the unimodal representations u j for j ∈ { 1 , 2 } are presented, along with the joint multimodal representation of the paired samples.

Original: For MNIST-TIDIGITS, j =and j = 2 correspond to image and speech respectively.
Revised: For MNIST-TIDIGITS, j =and j = 2 correspond to image and speech respectively.

Original: We note that the loss term L align makes the unimodal representations super-imposable.
Revised: We note that the loss term L align makes the unimodal representations super-imposable.

Original: Figure 4(c, d) shows the uncorrelated noise region in the image embedding space.
Revised: Figure 4(c, d) shows the uncorrelated noise region in the image embedding space.

Original: They get mapped to their relevant digit locations in the multimodal space due to importance weighting (in this case w speech >> w image ).
Revised: They get mapped to their relevant digit locations in the multimodal space due to importance weighting (in this case w speech >> w image ).

Original: IEMOCAP: Figure 5 shows t-SNE visualizations of representations learnt by the IMA model on IEMOCAP.
Revised: IEMOCAP: Figure 5 shows t-SNE visualizations of representations learnt by the IMA model on IEMOCAP.

Original: Figures 5(a, b) show word representations, and we observe that the emotionally colored words are at the periphery and distinct from neutral words.
Revised: Figures 5(a, b) show word representations, and we observe that the emotionally colored words are at the periphery and distinct from neutral words.

Original: The IMA model also detects most words as unimportant at the optimal value of ρ .
Revised: The IMA model also detects most words as unimportant at the optimal value of ρ .

Original: These word representations are learnt from GLOVE embedding inputs to the model.
Revised: These word representations are learnt from GLOVE embedding inputs to the model.

Original: The acoustic representations are in Figures 5(c, d).
Revised: The acoustic representations are in Figures 5(c, d).

Original: The regions which correspond to distinct emotion clusters such as happy , angry and sad are detected as important by the IMA model, and regions with neutral emotion and/or high confusion between emotions are assigned low importances.
Revised: The regions which correspond to distinct emotion clusters such as happy , angry and sad are detected as important by the IMA model, and regions with neutral emotion and/or high confusion between emotions are assigned low importances.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We have proposed the IMA model for robust learning of multimodal representations.
Revised: We have proposed the IMA model for robust learning of multimodal representations.

Original: The proposed model performs unimodal inference in absence of modalities and also addresses the problem of detecting important subspaces in each modality through weighted cross-covariance loss terms, which are minimized by unimodal importance networks.
Revised: The proposed model performs unimodal inference in absence of modalities and also addresses the problem of detecting important subspaces in each modality through weighted cross-covariance loss terms, which are minimized by unimodal importance networks.

Original: The IMA model is trained on standard datasets such as MNIST-TIDIGITS as well as conversational datasets such as IEMOCAP.
Revised: The IMA model is trained on standard datasets such as MNIST-TIDIGITS as well as conversational datasets such as IEMOCAP.

Original: Experiments are performed to evaluate quality of the learned representations for downstream multimodal tasks such as digit recognition and emotion understanding.
Revised: Experiments are performed to evaluate quality of the learned representations for downstream multimodal tasks such as digit recognition and emotion understanding.

Original: The IMA model achieves performance competitive with baseline approaches.
Revised: The IMA model achieves performance competitive with baseline approaches.

Original: Accuracy on these tasks is also improved by incorporating learnt unimodal importances into the multimodal representations.
Revised: Accuracy on these tasks is also improved by incorporating learnt unimodal importances into the multimodal representations.

Original: Future work could focus on extension to more than two modalities (such as visual), and sequence models such as RNNs (Recurrent Neural
Revised: Future work could focus on extension to more than two modalities (such as visual), and sequence models such as RNNs (Recurrent Neural

Original: Networks) in encoder-decoder networks for utterance level analysis.
Revised: Networks) in encoder-decoder networks for utterance level analysis.

Original: A A PPENDIX A. MNIST-TIDIGITS D ATASET P ROCESSING
Revised: A A PPENDIX A. MNIST-TIDIGITS D ATASET P ROCESSING

Original: The datasets which we have considered for our experiments are further pre-processed to create feature sets amenable to model training, these steps mainly involve:
Revised: The datasets which we have considered for our experiments are further pre-processed to create feature sets amenable to model training, these steps mainly involve:

Original: Feature extraction from the MNIST images and the TIDIGITS/TI46 speech corpora.
Revised: Feature extraction from the MNIST images and the TIDIGITS/TI46 speech corpora.

Original: Multimodal pairing for model to learn co-occurrences ie.
Revised: Multimodal pairing for model to learn co-occurrences ie.

Original: images of spoken digits are paired with their respective spoken utterances to create multimodal paired samples.
Revised: images of spoken digits are paired with their respective spoken utterances to create multimodal paired samples.

Original: Addition of samples with uncorrelated noise in each modality which co-occur with all image/speech digits in the other modality.
Revised: Addition of samples with uncorrelated noise in each modality which co-occur with all image/speech digits in the other modality.

Original: The model is expected to learn that the noise samples in each modality are not important.
Revised: The model is expected to learn that the noise samples in each modality are not important.

Original: The composition of each noise image or speech sample is not important here; rather it is their cooccurrence pattern with the rest of the modalities which the model learns from during training.
Revised: The composition of each noise image or speech sample is not important here; rather it is their cooccurrence pattern with the rest of the modalities which the model learns from during training.

Original: The only constraint we impose on the noise samples is the locality of their representations in the modality space to enable a learnt hyperplane to separate it out from the rest of the samples.
Revised: The only constraint we impose on the noise samples is the locality of their representations in the modality space to enable a learnt hyperplane to separate it out from the rest of the samples.

Original: We pre-process each image digit from MNIST and each spoken digit from TIDIGITS/TI46 to create 784 and 696dimensional binary feature vectors respectively.
Revised: We pre-process each image digit from MNIST and each spoken digit from TIDIGITS/TI46 to create 784 and 696dimensional binary feature vectors respectively.

Original: We also construct synthetic image and spoken noise samples for each modality which serves as uncorrelated noise.
Revised: We also construct synthetic image and spoken noise samples for each modality which serves as uncorrelated noise.

Original: A.1. P ROCESSING OF MNIST AND NOISE DIGITS
Revised: A.1. P ROCESSING OF MNIST AND NOISE DIGITS

Original: Each MNIST digit is of 28x28 dimensionality, and the pixel intensities are in the range [0,255].
Revised: Each MNIST digit is of 28x28 dimensionality, and the pixel intensities are in the range [0,255].

Original: We divide each intensity by 255 to normalize the pixel values to the range [0,1].
Revised: We divide each intensity by 255 to normalize the pixel values to the range [0,1].

Original: We also construct images which are the same dimensionality (28*28) as the MNIST digits, but consist of white Gaussian noise at a predeﬁned level of sparsity K .
Revised: We also construct images which are the same dimensionality (28*28) as the MNIST digits, but consist of white Gaussian noise at a predeﬁned level of sparsity K .

Original: In our experiments, we have selected K = 0 .
Revised: In our experiments, we have selected K = 0 .

Original: The sparsity is introduced to make the synthetic noise compatible with the cross-entropy loss function.
Revised: The sparsity is introduced to make the synthetic noise compatible with the cross-entropy loss function.

Original: Synthesis is performed by creating a 28*28 grayscale image of white Gaussian noise at each pixel with an intensity mean of 0.5 and a standard deviation of 0.12, which ensures that almost all the image pixels are in the range [0,1], and then setting K -fraction of those pixels to zero.
Revised: Synthesis is performed by creating a 28*28 grayscale image of white Gaussian noise at each pixel with an intensity mean of 0.5 and a standard deviation of 0.12, which ensures that almost all the image pixels are in the range [0,1], and then setting K -fraction of those pixels to zero.

Original: Negative intensity pixels are also set to zero.
Revised: Negative intensity pixels are also set to zero.

Original: All images (MNIST and noise) are subsequently ﬂattened, resulting in 784-dimensional feature vectors.
Revised: All images (MNIST and noise) are subsequently ﬂattened, resulting in 784-dimensional feature vectors.

Original: A.1. P ROCESSING OF TIDIGITS/TI46 SPOKEN DIGITS
Revised: A.1. P ROCESSING OF TIDIGITS/TI46 SPOKEN DIGITS

Original: The speech waveforms from the TIDIGITS and the TI46 datasets are ﬁrst resampled to 12.5 KHz, and then the TIDIGITS dataset is further segmented by speaker and digit to generate audio ﬁles, each corresponding to a single speaker uttering one digit.
Revised: The speech waveforms from the TIDIGITS and the TI46 datasets are ﬁrst resampled to 12.5 KHz, and then the TIDIGITS dataset is further segmented by speaker and digit to generate audio ﬁles, each corresponding to a single speaker uttering one digit.

Original: Noise speech samples are generated by sampling white Gaussian noise for each time step with a mean of 0 and a standard deviation of 1; which corresponds to the same numeric amplitude ranges as the digit audio ﬁles.
Revised: Noise speech samples are generated by sampling white Gaussian noise for each time step with a mean of 0 and a standard deviation of 1; which corresponds to the same numeric amplitude ranges as the digit audio ﬁles.

Original: Each synthesized noise waveform is of one second duration and has a sampling rate of 12.5 KHz.
Revised: Each synthesized noise waveform is of one second duration and has a sampling rate of 12.5 KHz.

Original: Subsequently,
Revised: Subsequently,

Original: MFCC (Mel Frequency Cepstral Coefﬁcient) features are extracted from each audio ﬁle (both the synthesized noise and the spoken digits), with 20 ms windows; 10 ms shift and 12 cepstral coefﬁcients not including the energy term.
Revised: MFCC (Mel Frequency Cepstral Coefﬁcient) features are extracted from each audio ﬁle (both the synthesized noise and the spoken digits), with 20 ms windows; 10 ms shift and 12 cepstral coefﬁcients not including the energy term.

Original: Each digit utterance also consists of recording pauses at the start and end time samples.
Revised: Each digit utterance also consists of recording pauses at the start and end time samples.

Original: To ﬁlter these pauses, 29 frames from
Revised: To ﬁlter these pauses, 29 frames from

Original: A. IEMOCAP D ATASET P ROCESSING
Revised: A. IEMOCAP D ATASET P ROCESSING

Original: We represent vocabulary words in the IEMOCAP dataset with their pre-trained embeddings using
Revised: We represent vocabulary words in the IEMOCAP dataset with their pre-trained embeddings using

Original: GLOVE (Global Vectors for Word Representation) (Pennington et al., 2014), a popular technique for word embeddings.
Revised: GLOVE (Global Vectors for Word Representation) (Pennington et al., 2014), a popular technique for word embeddings.

Original: For model training we have only considered vocabulary words which appear at least 5 times in the corpus, resulting in a vocabulary size of 1215 words.
Revised: For model training we have only considered vocabulary words which appear at least 5 times in the corpus, resulting in a vocabulary size of 1215 words.

Original: We use the COVAREP toolbox to extract features from speech waveforms in the IEMOCAP dataset.
Revised: We use the COVAREP toolbox to extract features from speech waveforms in the IEMOCAP dataset.

Original: COVAREP is an open-source toolbox (Degottex et al., 2014) commonly used in voice-based analysis applications.
Revised: COVAREP is an open-source toolbox (Degottex et al., 2014) commonly used in voice-based analysis applications.

Original: Acoustic features are extracted at speech frame level to form a 55-dimensional feature vector, and a spoken word is represented by the mean of co-occurring frame vectors.
Revised: Acoustic features are extracted at speech frame level to form a 55-dimensional feature vector, and a spoken word is represented by the mean of co-occurring frame vectors.

Original: Acoustic signals incorporate multiple factors of variation such as phonetics (acoustic characteristics pertaining to what is said) as well as extra-linguistic factors (acoustic characteristics pertaining to how or by whom it is said) such as emotions (happy vs. angry voice, etc.)
Revised: Acoustic signals incorporate multiple factors of variation such as phonetics (acoustic characteristics pertaining to what is said) as well as extra-linguistic factors (acoustic characteristics pertaining to how or by whom it is said) such as emotions (happy vs. angry voice, etc.)

Original: , loudness, tone of voice, and gender.
Revised: , loudness, tone of voice, and gender.

Original: While all of these latent factors inﬂuence the observed acoustic features, they are both local as well as global in nature.
Revised: While all of these latent factors inﬂuence the observed acoustic features, they are both local as well as global in nature.

Original: Local factors of variation are transient, for example, phonetics (fricative, vowel sounds, etc.) and pause ﬁllers.
Revised: Local factors of variation are transient, for example, phonetics (fricative, vowel sounds, etc.) and pause ﬁllers.

Original: Global factors of variation inﬂuence the whole utterance, for example higher or lower average pitch (which can vary with gender, i.e., a global attribute).
Revised: Global factors of variation inﬂuence the whole utterance, for example higher or lower average pitch (which can vary with gender, i.e., a global attribute).

Original: The MFCCs mostly capture phonetics, and we have removed them as we do not wish to learn how words correlate with their phonetics, but rather to their emotional/extra-linguistic attributes.
Revised: The MFCCs mostly capture phonetics, and we have removed them as we do not wish to learn how words correlate with their phonetics, but rather to their emotional/extra-linguistic attributes.

Original: The phonetic latent factors would tend to dominate shared latent representation if MFCCs are included during training.
Revised: The phonetic latent factors would tend to dominate shared latent representation if MFCCs are included during training.

Original: We are seeking to learn latent extra-linguistic speech factors instead of phonetic components.
Revised: We are seeking to learn latent extra-linguistic speech factors instead of phonetic components.

Original: The dataset also has transcriptions with token-level timestamps.
Revised: The dataset also has transcriptions with token-level timestamps.

Original: For each spoken token, we average the COVAREP features extracted from the waveform co-occurring with it to create a single 55-dimensional acoustic feature vector.
Revised: For each spoken token, we average the COVAREP features extracted from the waveform co-occurring with it to create a single 55-dimensional acoustic feature vector.

Original: Each acoustic vector consists of features useful in voice analysis such as NAQ (Normalized Amplitude Quotient), QOQ (Quasi-Open Quotient) and
Revised: Each acoustic vector consists of features useful in voice analysis such as NAQ (Normalized Amplitude Quotient), QOQ (Quasi-Open Quotient) and

Original: F0 (Fundamental Frequency) with a sliding window of 20 ms. duration and 10 ms. shift.
Revised: F0 (Fundamental Frequency) with a sliding window of 20 ms. duration and 10 ms. shift.

Original: [Figure]
Revised: [Figure]

Original: A. I MPORTANCE N ETWORK P ERFORMANCE
Revised: A. I MPORTANCE N ETWORK P ERFORMANCE

Original: In Figure
Revised: In Figure

Original: [Table]
Revised: [Table]

Original: we present the variation of the precision, recall and F1 scores for the importance network performance on the MNIST-TIDIGIT and IEMOCAP datasets with varying values of the importance prior ρ in the range [0 , 1] .
Revised: we present the variation of the precision, recall and F1 scores for the importance network performance on the MNIST-TIDIGIT and IEMOCAP datasets with varying values of the importance prior ρ in the range [0 , 1] .

Original: The scores explained in Section 6.1 correspond to these ﬁgures
Revised: The scores explained in Section 6.1 correspond to these ﬁgures

Original: [Table]
Revised: [Table]

Original: and also include the comparison with a random baseline importance network for the IEMOCAP experiments
Revised: and also include the comparison with a random baseline importance network for the IEMOCAP experiments

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A. M ODEL S IZES OF IMA AND BASELINES
Revised: A. M ODEL S IZES OF IMA AND BASELINES

Original: The IMA model has the advantage of having only M encoders and O ( M ) loss terms to optimize compared to the JMVAE-KL and MVAE models.
Revised: The IMA model has the advantage of having only M encoders and O ( M ) loss terms to optimize compared to the JMVAE-KL and MVAE models.

Original: That in conjunction with no stochasticity assumption (present in VAEs) results in fewer autoencoder parameters to train compared to JMVAE-KL and MVAE, even when the encoder and decoder sizes are the same.
Revised: That in conjunction with no stochasticity assumption (present in VAEs) results in fewer autoencoder parameters to train compared to JMVAE-KL and MVAE, even when the encoder and decoder sizes are the same.

Original: Table 4 lists the model sizes for the IMA model and the baseline models for the experimental settings described in Section 5.
Revised: Table 4 lists the model sizes for the IMA model and the baseline models for the experimental settings described in Section 5.

Original: The
Revised: The

Original: IMA model has slightly higher parameters compared to MVAE, however that increase comes with the ability of the importance networks to learn uncorrelated noise across modalities
Revised: IMA model has slightly higher parameters compared to MVAE, however that increase comes with the ability of the importance networks to learn uncorrelated noise across modalities

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: D OWNSTREAM C LASSIFICATION E XPERIMENTS
Revised: D OWNSTREAM C LASSIFICATION E XPERIMENTS

Original: The quality of the joint multimodal representation z learnt by the model is quantitatively evaluated by downstream digit and emotion classiﬁcation tasks respectively from the MNIST-TIDIGITS and IEMOCAP datasets.
Revised: The quality of the joint multimodal representation z learnt by the model is quantitatively evaluated by downstream digit and emotion classiﬁcation tasks respectively from the MNIST-TIDIGITS and IEMOCAP datasets.

Original: We use a two-layer MLP (Multi-layer Perceptron) classiﬁer with ReLU (Rectiﬁed Linear Unit) activation functions for the supervised tasks of emotion and digit classiﬁcations, given the representations learnt by the importance-based autoencoder and other baseline models.
Revised: We use a two-layer MLP (Multi-layer Perceptron) classiﬁer with ReLU (Rectiﬁed Linear Unit) activation functions for the supervised tasks of emotion and digit classiﬁcations, given the representations learnt by the importance-based autoencoder and other baseline models.

Original: Table 3 contains details of classiﬁer settings for each dataset.
Revised: Table 3 contains details of classiﬁer settings for each dataset.

Original: For MNIST-TIDIGITS, the classiﬁcation is at digit level (each sample corresponds to a multimodal spoken and image digit pair), and for IEMOCAP it is at utterance level (where each sample corresponds to a spoken emotional utterance).
Revised: For MNIST-TIDIGITS, the classiﬁcation is at digit level (each sample corresponds to a multimodal spoken and image digit pair), and for IEMOCAP it is at utterance level (where each sample corresponds to a spoken emotional utterance).

Original: MNIST-TIDIGITS :
Revised: MNIST-TIDIGITS :

Original: Table 5 reports the classiﬁcation accuracies on the digit classiﬁcation task for both 2-D and 50-D multimodal representations for the importance based autoencoder and other baselines reported in Section 5.4.
Revised: Table 5 reports the classiﬁcation accuracies on the digit classiﬁcation task for both 2-D and 50-D multimodal representations for the importance based autoencoder and other baselines reported in Section 5.4.

Original: The accuracy on the overall evaluation set is reported along with that computed on subset of samples with uncorrelated noise (samples with synthesized image noise; and with synthesized speech noise - refer Section 4).
Revised: The accuracy on the overall evaluation set is reported along with that computed on subset of samples with uncorrelated noise (samples with synthesized image noise; and with synthesized speech noise - refer Section 4).

Original: In all cases the importance-weighted multimodal representations from IMA outperform those without importances.
Revised: In all cases the importance-weighted multimodal representations from IMA outperform those without importances.

Original: In 2D, the importance-weighted representations from the IMA model outperforms the JMVAE-KL and MVAE baselines, as well the unweighted IMA representations.
Revised: In 2D, the importance-weighted representations from the IMA model outperforms the JMVAE-KL and MVAE baselines, as well the unweighted IMA representations.

Original: For higher dimensionality (50D), overall classiﬁcation accuracy for the weighted IMA model is slightly less than other baseline approaches (including MVAE and JMVAE-KL).
Revised: For higher dimensionality (50D), overall classiﬁcation accuracy for the weighted IMA model is slightly less than other baseline approaches (including MVAE and JMVAE-KL).

Original: The unimodal encoders perform well as they have access to the other modality even in presence of noise.
Revised: The unimodal encoders perform well as they have access to the other modality even in presence of noise.

Original: For example, the image encoder achieves an accuracy of 83.05% by utilizing the image digit even if the corresponding speech consists of noise, and vice-versa.
Revised: For example, the image encoder achieves an accuracy of 83.05% by utilizing the image digit even if the corresponding speech consists of noise, and vice-versa.

Original: Note that all classiﬁcation accuracies are not comparable to state-of-the-art scores (¿99.5%) obtained on MNIST in contemporary literature Wan et al.
Revised: Note that all classiﬁcation accuracies are not comparable to state-of-the-art scores (¿99.5%) obtained on MNIST in contemporary literature Wan et al.

Original: due to no specialized convolutional/pooling layers and training on samples which include uncorrelated noise.
Revised: due to no specialized convolutional/pooling layers and training on samples which include uncorrelated noise.

Original: IEMOCAP:
Revised: IEMOCAP:

Original: In Table 6, emotion recognition results on IEMOCAP are similar to the precision results reported in Section 6.2.
Revised: In Table 6, emotion recognition results on IEMOCAP are similar to the precision results reported in Section 6.2.

Original: After incorporating importances, verbal accuracy improves (absolute) by 3.86%, and multimodal accuracy improves by 1.2%.
Revised: After incorporating importances, verbal accuracy improves (absolute) by 3.86%, and multimodal accuracy improves by 1.2%.

Original: The IMA model outperforms achieves competitive performance to the baselines JMVAE-KL and MVAE for this task of emotion recognition.
Revised: The IMA model outperforms achieves competitive performance to the baselines JMVAE-KL and MVAE for this task of emotion recognition.