Original: L
Revised: L

Original: EARNING F ACTORIZED M ULTIMODAL R EPRESENTATIONS
Revised: EARNING F ACTORIZED M ULTIMODAL R EPRESENTATIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Learning multimodal representations is a fundamentally complex research problem due to the presence of multiple heterogeneous sources of information.
Revised: Learning multimodal representations is a fundamentally complex research problem due to the presence of multiple heterogeneous sources of information.

Original: Although the presence of multiple modalities provides additional valuable information, there are two key challenges to address when learning from multimodal data: 1) models must learn the complex intra-modal and cross-modal interactions for prediction and 2) models must be robust to unexpected missing or noisy modalities during testing.
Revised: Although the presence of multiple modalities provides additional valuable information, there are two key challenges to address when learning from multimodal data: 1) models must learn the complex intra-modal and cross-modal interactions for prediction and 2) models must be robust to unexpected missing or noisy modalities during testing.

Original: In this paper, we propose to optimize for a joint generative-discriminative objective across multimodal data and labels.
Revised: In this paper, we propose to optimize for a joint generative-discriminative objective across multimodal data and labels.

Original: We introduce a model that factorizes representations into two sets of independent factors: multimodal discriminative and modality-speciﬁc generative factors.
Revised: We introduce a model that factorizes representations into two sets of independent factors: multimodal discriminative and modality-speciﬁc generative factors.

Original: Multimodal discriminative factors are shared across all modalities and contain joint multimodal features required for discriminative tasks such as sentiment prediction.
Revised: Multimodal discriminative factors are shared across all modalities and contain joint multimodal features required for discriminative tasks such as sentiment prediction.

Original: Modality-speciﬁc generative factors are unique for each modality and contain the information required for generating data.
Revised: Modality-speciﬁc generative factors are unique for each modality and contain the information required for generating data.

Original: Experimental results show that our model is able to learn meaningful multimodal representations that achieve state-of-the-art or competitive performance on six multimodal datasets.
Revised: Experimental results show that our model is able to learn meaningful multimodal representations that achieve state-of-the-art or competitive performance on six multimodal datasets.

Original: Our model demonstrates ﬂexible generative capabilities by conditioning on independent factors and can reconstruct missing modalities without signiﬁcantly impacting performance.
Revised: Our model demonstrates ﬂexible generative capabilities by conditioning on independent factors and can reconstruct missing modalities without signiﬁcantly impacting performance.

Original: Lastly, we interpret our factorized representations to understand the interactions that inﬂuence multimodal learning.
Revised: Lastly, we interpret our factorized representations to understand the interactions that inﬂuence multimodal learning.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Multimodal machine learning involves learning from data across multiple modalities (Baltrušaitis et al., 2017).
Revised: Multimodal machine learning involves learning from data across multiple modalities (Baltrušaitis et al., 2017).

Original: It is a challenging yet crucial research area with real-world applications in robotics (Liu et al., 2017), dialogue systems (Pittermann et al., 2010), intelligent tutoring systems (Petrovica et al., 2017), and healthcare diagnosis (Frantzidis et al., 2010).
Revised: It is a challenging yet crucial research area with real-world applications in robotics (Liu et al., 2017), dialogue systems (Pittermann et al., 2010), intelligent tutoring systems (Petrovica et al., 2017), and healthcare diagnosis (Frantzidis et al., 2010).

Original: At the heart of many multimodal modeling tasks lies the challenge of learning rich representations from multiple modalities.
Revised: At the heart of many multimodal modeling tasks lies the challenge of learning rich representations from multiple modalities.

Original: For example, analyzing multimedia content requires learning multimodal representations across the language, visual, and acoustic modalities (Cho et al., 2015).
Revised: For example, analyzing multimedia content requires learning multimodal representations across the language, visual, and acoustic modalities (Cho et al., 2015).

Original: Although the presence of multiple modalities provides additional valuable information, there are two key challenges to address when learning from multimodal data: 1) models must learn the complex intra-modal and cross-modal interactions for prediction (Zadeh et al., 2017), and 2) trained models must be robust to unexpected missing or noisy modalities during testing (Ngiam et al., 2011).
Revised: Although the presence of multiple modalities provides additional valuable information, there are two key challenges to address when learning from multimodal data: 1) models must learn the complex intra-modal and cross-modal interactions for prediction (Zadeh et al., 2017), and 2) trained models must be robust to unexpected missing or noisy modalities during testing (Ngiam et al., 2011).

Original: In this paper, we propose to optimize for a joint generative-discriminative objective across multimodal data and labels.
Revised: In this paper, we propose to optimize for a joint generative-discriminative objective across multimodal data and labels.

Original: The discriminative objective ensures that the representations learned are rich in intra-modal and cross-modal features useful towards predicting the label, while the generative objective allows the model to infer missing modalities at test time and deal with the presence of noisy modalities.
Revised: The discriminative objective ensures that the representations learned are rich in intra-modal and cross-modal features useful towards predicting the label, while the generative objective allows the model to infer missing modalities at test time and deal with the presence of noisy modalities.

Original: To this end, we introduce the Multimodal Factorization Model (MFM in Figure 1) that factorizes multimodal representations into multimodal discriminative factors and modality-speciﬁc generative factors.
Revised: To this end, we introduce the Multimodal Factorization Model (MFM in Figure 1) that factorizes multimodal representations into multimodal discriminative factors and modality-speciﬁc generative factors.

Original: Multimodal discriminative factors are shared across all modalities and contain joint multimodal features required for discriminative tasks.
Revised: Multimodal discriminative factors are shared across all modalities and contain joint multimodal features required for discriminative tasks.

Original: Modality-speciﬁc generative factors are unique for each modality and contain the information required for generating each modality.
Revised: Modality-speciﬁc generative factors are unique for each modality and contain the information required for generating each modality.

Original: We believe that factorizing multimodal representations into different explanatory factors can help each factor focus on learning from a subset of the joint information across multimodal data and labels.
Revised: We believe that factorizing multimodal representations into different explanatory factors can help each factor focus on learning from a subset of the joint information across multimodal data and labels.

Original: This method is in contrast to jointly learning a single factor that summarizes all generative and discriminative information (Srivastava & Salakhutdinov, 2012).
Revised: This method is in contrast to jointly learning a single factor that summarizes all generative and discriminative information (Srivastava & Salakhutdinov, 2012).

Original: To sum up, MFM deﬁnes a joint distribution over multimodal data, and by the conditional independence assumptions in the assumed graphical model, both generative and discriminative aspects are taken into account.
Revised: To sum up, MFM deﬁnes a joint distribution over multimodal data, and by the conditional independence assumptions in the assumed graphical model, both generative and discriminative aspects are taken into account.

Original: Our model design further provides interpretability of the factorized representations.
Revised: Our model design further provides interpretability of the factorized representations.

Original: [Figure]
Revised: [Figure]

Original: Through an extensive set of experiments, we show that MFM learns improved multimodal representations with these characteristics: 1) The multimodal discriminative factors achieve state-of-the-art or competitive performance on six multimodal time series datasets.
Revised: Through an extensive set of experiments, we show that MFM learns improved multimodal representations with these characteristics: 1) The multimodal discriminative factors achieve state-of-the-art or competitive performance on six multimodal time series datasets.

Original: We also demonstrate that MFM can generalize by integrating it with other existing multimodal discriminative models.
Revised: We also demonstrate that MFM can generalize by integrating it with other existing multimodal discriminative models.

Original: 2) MFM allows ﬂexible generation concerning multimodal discriminative factors (labels) and modality-speciﬁc generative factors (styles).
Revised: 2) MFM allows ﬂexible generation concerning multimodal discriminative factors (labels) and modality-speciﬁc generative factors (styles).

Original: We further show that we can perform reconstruction of missing modalities from observed modalities without signiﬁcantly impacting discriminative performance.
Revised: We further show that we can perform reconstruction of missing modalities from observed modalities without signiﬁcantly impacting discriminative performance.

Original: Finally, we interpret our learned representations using information-based and gradient-based methods, allowing us to understand the contributions of individual factors towards multimodal prediction and generation.
Revised: Finally, we interpret our learned representations using information-based and gradient-based methods, allowing us to understand the contributions of individual factors towards multimodal prediction and generation.

Original:  M ULTIMODAL F ACTORIZATION M ODEL
Revised:  M ULTIMODAL F ACTORIZATION M ODEL

Original: Multimodal Factorization Model (MFM) is a latent variable model (Figure 1(a)) with conditional independence assumptions over multimodal
Revised: Multimodal Factorization Model (MFM) is a latent variable model (Figure 1(a)) with conditional independence assumptions over multimodal

Original: discriminative factors and modality-speciﬁc generative factors.
Revised: discriminative factors and modality-speciﬁc generative factors.

Original: According to these assumptions, we propose a factorization over the joint distribution of multimodal data (Section 2.1).
Revised: According to these assumptions, we propose a factorization over the joint distribution of multimodal data (Section 2.1).

Original: Since exact posterior inference on this factorized distribution can be intractable, we propose an approximate inference algorithm based on minimizing a joint-distribution
Revised: Since exact posterior inference on this factorized distribution can be intractable, we propose an approximate inference algorithm based on minimizing a joint-distribution

Original: Wasserstein distance over multimodal data (Section 2.2).
Revised: Wasserstein distance over multimodal data (Section 2.2).

Original: Finally, we derive the MFM objective by approximating the joint-distribution Wasserstein distance via a generalized mean-ﬁeld assumption.
Revised: Finally, we derive the MFM objective by approximating the joint-distribution Wasserstein distance via a generalized mean-ﬁeld assumption.

Original: Notation: We deﬁne X 1 ∶ M as the multimodal data from M modalities and Y as the labels, with joint distribution P X 1 ∶ M , Y = P ( X 1 ∶ M , Y ) .
Revised: Notation: We deﬁne X 1 ∶ M as the multimodal data from M modalities and Y as the labels, with joint distribution P X 1 ∶ M , Y = P ( X 1 ∶ M , Y ) .

Original: Let ˆX 1 ∶ M denote the generated multimodal data and ˆY denote the generated labels, with joint distribution P ˆX 1 ∶ M , ˆY = P ( ˆX 1 ∶ M , ˆY ) .
Revised: Let ˆX 1 ∶ M denote the generated multimodal data and ˆY denote the generated labels, with joint distribution P ˆX 1 ∶ M , ˆY = P ( ˆX 1 ∶ M , ˆY ) .

Original: 2. F ACTORIZED M ULTIMODAL R EPRESENTATIONS
Revised: 2. F ACTORIZED M ULTIMODAL R EPRESENTATIONS

Original: To factorize multimodal representations into multimodal discriminative factors and modality-speciﬁc generative factors, MFM assumes a Bayesian network structure as shown in Figure 1(a).
Revised: To factorize multimodal representations into multimodal discriminative factors and modality-speciﬁc generative factors, MFM assumes a Bayesian network structure as shown in Figure 1(a).

Original: In this graphical model, factors F y and F a { 1 ∶ M } are generated from mutually independent latent variables
Revised: In this graphical model, factors F y and F a { 1 ∶ M } are generated from mutually independent latent variables

Original: Z =
Revised: Z =

Original: [ Z y , Z a { 1 ∶ M } ] with prior P Z .
Revised: [ Z y , Z a { 1 ∶ M } ] with prior P Z .

Original: In particular, Z y generates the multimodal discriminative factor F y and Z a { 1 ∶ M } generate modality-speciﬁc generative factors F a { 1 ∶ M } .
Revised: In particular, Z y generates the multimodal discriminative factor F y and Z a { 1 ∶ M } generate modality-speciﬁc generative factors F a { 1 ∶ M } .

Original: By construction, F y contributes to the generation of ˆY while { F y , F a i } both contribute to the generation of ˆX i .
Revised: By construction, F y contributes to the generation of ˆY while { F y , F a i } both contribute to the generation of ˆX i .

Original: As a result, the joint distribution P ( ˆX 1 ∶ M , ˆY ) can be factorized as follows:
Revised: As a result, the joint distribution P ( ˆX 1 ∶ M , ˆY ) can be factorized as follows:

Original: [Equation]
Revised: [Equation]

Original: with d F = d F y ∏ Mi = 1 d F a i and d Z = d Z y ∏ Mi = 1 d Z a i .
Revised: with d F = d F y ∏ Mi = 1 d F a i and d Z = d Z y ∏ Mi = 1 d Z a i .

Original: Exact posterior inference in Equation 1 may be analytically intractable due to the integration over Z .
Revised: Exact posterior inference in Equation 1 may be analytically intractable due to the integration over Z .

Original: We therefore resort to using an approximate inference distribution Q ( Z ∣ X 1 ∶ M , Y ) as detailed in the following subsection.
Revised: We therefore resort to using an approximate inference distribution Q ( Z ∣ X 1 ∶ M , Y ) as detailed in the following subsection.

Original: As a result, MFM can be viewed as an autoencoding structure that consists of encoder (inference) and decoder (generative) modules (Figure 1(c)).
Revised: As a result, MFM can be viewed as an autoencoding structure that consists of encoder (inference) and decoder (generative) modules (Figure 1(c)).

Original: The encoder module for Q (⋅∣⋅) allows us to easily sample Z from an approximate posterior.
Revised: The encoder module for Q (⋅∣⋅) allows us to easily sample Z from an approximate posterior.

Original: The decoder modules are parametrized according to the factorization of P ( ˆX 1 ∶ M , ˆY ∣ Z ) as given by Equation 1 and Figure 1(a).
Revised: The decoder modules are parametrized according to the factorization of P ( ˆX 1 ∶ M , ˆY ∣ Z ) as given by Equation 1 and Figure 1(a).

Original: 2. M INIMIZING J OINT -D ISTRIBUTION W ASSERSTEIN D ISTANCE OVER M ULTIMODAL D ATA
Revised: 2. M INIMIZING J OINT -D ISTRIBUTION W ASSERSTEIN D ISTANCE OVER M ULTIMODAL D ATA

Original: Two common choices for approximate inference in autoencoding structures are Variational Autoencoders (VAEs) (Kingma & Welling, 2013) and Wasserstein Autoencoders (WAEs) (Zhao et al., 2017;
Revised: Two common choices for approximate inference in autoencoding structures are Variational Autoencoders (VAEs) (Kingma & Welling, 2013) and Wasserstein Autoencoders (WAEs) (Zhao et al., 2017;

Original: Tolstikhin et al., 2017).
Revised: Tolstikhin et al., 2017).

Original: The former optimizes the evidence lower bound objective (ELBO), and the latter derives an approximation for the primal form of the Wasserstein distance.
Revised: The former optimizes the evidence lower bound objective (ELBO), and the latter derives an approximation for the primal form of the Wasserstein distance.

Original: We consider the latter since it simultaneously results in better latent factor disentanglement (Zhao et al., 2017; Rubenstein et al., 2018) and better sample generation quality than its counterparts (Chen et al., 2016; Higgins et al., 2016; Kingma & Welling, 2013).
Revised: We consider the latter since it simultaneously results in better latent factor disentanglement (Zhao et al., 2017; Rubenstein et al., 2018) and better sample generation quality than its counterparts (Chen et al., 2016; Higgins et al., 2016; Kingma & Welling, 2013).

Original: However, WAEs are designed for unimodal data and do not consider factorized distributions over latent variables that generate multimodal data.
Revised: However, WAEs are designed for unimodal data and do not consider factorized distributions over latent variables that generate multimodal data.

Original: Therefore, we propose a variant for handling factorized joint distributions over multimodal data.
Revised: Therefore, we propose a variant for handling factorized joint distributions over multimodal data.

Original: As suggested by Kingma & Welling (2013), we adopt the design of nonlinear mappings (i.e. neural network architectures) in the encoder and decoder (Figure 1 (c)).
Revised: As suggested by Kingma & Welling (2013), we adopt the design of nonlinear mappings (i.e. neural network architectures) in the encoder and decoder (Figure 1 (c)).

Original: For the encoder Q ( Z ∣ X 1 ∶ M , Y ) , we learn a deterministic mapping Q enc ∶ X 1 ∶ M , Y → Z (Rubenstein et al., 2018; Tolstikhin et al., 2017).
Revised: For the encoder Q ( Z ∣ X 1 ∶ M , Y ) , we learn a deterministic mapping Q enc ∶ X 1 ∶ M , Y → Z (Rubenstein et al., 2018; Tolstikhin et al., 2017).

Original: For the decoder, we deﬁne the generation process from latent variables as G y ∶ Z y
Revised: For the decoder, we deﬁne the generation process from latent variables as G y ∶ Z y

Original: → F y , G a { 1 ∶ M } ∶
Revised: → F y , G a { 1 ∶ M } ∶

Original: Z a { 1 ∶ M } → F a { 1 ∶ M } , D ∶ F y → ˆY , and F 1 ∶ M ∶ F y , F a { 1 ∶ M } → ˆX 1 ∶ M , where G y ,G a { 1 ∶ M } ,D and F 1 ∶ M are deterministic functions parametrized by neural networks.
Revised: Z a { 1 ∶ M } → F a { 1 ∶ M } , D ∶ F y → ˆY , and F 1 ∶ M ∶ F y , F a { 1 ∶ M } → ˆX 1 ∶ M , where G y ,G a { 1 ∶ M } ,D and F 1 ∶ M are deterministic functions parametrized by neural networks.

Original: Let W c
Revised: Let W c

Original: ( P X 1 ∶ M , Y ,P ˆX 1 ∶ M , ˆY ) denote the joint-distribution Wasserstein distance over multimodal data under cost function
Revised: ( P X 1 ∶ M , Y ,P ˆX 1 ∶ M , ˆY ) denote the joint-distribution Wasserstein distance over multimodal data under cost function

Original: c Xi and c Y .
Revised: c Xi and c Y .

Original: We choose the squared cost c ( a,b ) =
Revised: We choose the squared cost c ( a,b ) =

Original: ∥ a − b ∥ 22 , allowing us to minimize the 2-Wasserstein distance.
Revised: ∥ a − b ∥ 22 , allowing us to minimize the 2-Wasserstein distance.

Original: The cost function can be deﬁned not only on static data but also on time series data such as text, audio and videos.
Revised: The cost function can be deﬁned not only on static data but also on time series data such as text, audio and videos.

Original: For example, given time series data
Revised: For example, given time series data

Original: X =
Revised: X =

Original: [ X 1 ,X 2 , ⋯ ,X T ] and ˆX =
Revised: [ X 1 ,X 2 , ⋯ ,X T ] and ˆX =

Original: [ ˆ X 1 , ˆ X 2 , ⋯ , ˆ X T ] , we deﬁne c ( X , ˆX )
Revised: [ ˆ X 1 , ˆ X 2 , ⋯ , ˆ X T ] , we deﬁne c ( X , ˆX )

Original: =
Revised: =

Original: ∑ Tt = 1 ∥ X t − ˆ X t ∥ 22 .
Revised: ∑ Tt = 1 ∥ X t − ˆ X t ∥ 22 .

Original: With conditional independence assumptions in Equation 1, we express W c ( P X 1 ∶ M , Y ,P ˆX 1 ∶ M , ˆY ) as:
Revised: With conditional independence assumptions in Equation 1, we express W c ( P X 1 ∶ M , Y ,P ˆX 1 ∶ M , ˆY ) as:

Original: Proposition 1.
Revised: Proposition 1.

Original: For any functions G y ∶ Z y → F y , G a { 1 ∶ M } ∶ Z a { 1 ∶ M } → F a { 1 ∶ M } , D ∶ F y → ˆY , and
Revised: For any functions G y ∶ Z y → F y , G a { 1 ∶ M } ∶ Z a { 1 ∶ M } → F a { 1 ∶ M } , D ∶ F y → ˆY , and

Original: F 1 ∶ M ∶ F a { 1 ∶ M } , F y → ˆX 1 ∶ M , we have W c ( P X 1 ∶ M , Y ,P ˆX 1 ∶ M , ˆY ) =
Revised: F 1 ∶ M ∶ F a { 1 ∶ M } , F y → ˆX 1 ∶ M , we have W c ( P X 1 ∶ M , Y ,P ˆX 1 ∶ M , ˆY ) =

Original: [Equation]
Revised: [Equation]

Original: where P Z is the prior over Z =
Revised: where P Z is the prior over Z =

Original: [ Z y , Z a { 1 ,M } ] and Q Z is the aggregated posterior of the proposed approximate inference distribution Q ( Z ∣ X 1 ∶ M , Y ) .
Revised: [ Z y , Z a { 1 ,M } ] and Q Z is the aggregated posterior of the proposed approximate inference distribution Q ( Z ∣ X 1 ∶ M , Y ) .

Original: Proof: The proof is adapted from Tolstikhin et al.
Revised: Proof: The proof is adapted from Tolstikhin et al.

Original: (Tolstikhin et al., 2017).
Revised: (Tolstikhin et al., 2017).

Original: The two differences are: (1) we show that P ( ˆX 1 ∶ M , ˆY ∣ Z = z ) are Dirac for all z ∈ Z , and (2) we use the fact that
Revised: The two differences are: (1) we show that P ( ˆX 1 ∶ M , ˆY ∣ Z = z ) are Dirac for all z ∈ Z , and (2) we use the fact that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Y ,
Revised: Y ,

Original: [Equation]
Revised: [Equation]

Original: Please refer to the supplementary material for proof details.
Revised: Please refer to the supplementary material for proof details.

Original: ∎
Revised: ∎

Original: The constraint on Q Z = P Z in Proposition 1 is hard to satisfy.
Revised: The constraint on Q Z = P Z in Proposition 1 is hard to satisfy.

Original: To obtain a numerical solution, we ﬁrst relax the constraint by performing a generalized mean ﬁeld assumption on Q according to the conditional independence as shown in the inference network of Figure 1 (b):
Revised: To obtain a numerical solution, we ﬁrst relax the constraint by performing a generalized mean ﬁeld assumption on Q according to the conditional independence as shown in the inference network of Figure 1 (b):

Original: [Equation]
Revised: [Equation]

Original: The intuition here is based on our design that Z y generates the multimodal discriminative factor
Revised: The intuition here is based on our design that Z y generates the multimodal discriminative factor

Original: F y and Z a { 1 ∶ M } generate modality-speciﬁc generative factors F a { 1 ∶ M } .
Revised: F y and Z a { 1 ∶ M } generate modality-speciﬁc generative factors F a { 1 ∶ M } .

Original: Therefore, the inference for Z y should depend on all modalities X 1 ∶ M and the inference for Z a i should depend only on the speciﬁc modality X i .
Revised: Therefore, the inference for Z y should depend on all modalities X 1 ∶ M and the inference for Z a i should depend only on the speciﬁc modality X i .

Original: Following this assumption, we deﬁne Q as a nonparametric set of all encoders that fulﬁll the factorization in Equation 3.
Revised: Following this assumption, we deﬁne Q as a nonparametric set of all encoders that fulﬁll the factorization in Equation 3.

Original: A penalty term is added into our objective to ﬁnd the
Revised: A penalty term is added into our objective to ﬁnd the

Original: Q ( Z ∣⋅) ∈ Q that is the closest to prior P Z , thereby approximately enforcing the constraint Q Z
Revised: Q ( Z ∣⋅) ∈ Q that is the closest to prior P Z , thereby approximately enforcing the constraint Q Z

Original: = P Z :
Revised: = P Z :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where λ is a hyper-parameter and MMD is the Maximum Mean Discrepancy (Gretton et al., 2012) as a divergence measure between Q Z and P Z .
Revised: where λ is a hyper-parameter and MMD is the Maximum Mean Discrepancy (Gretton et al., 2012) as a divergence measure between Q Z and P Z .

Original: The prior P Z is chosen as a centered isotropic
Revised: The prior P Z is chosen as a centered isotropic

Original: Gaussian N( 0 , I ) , so that it implicitly enforces independence between the latent variables Z = [
Revised: Gaussian N( 0 , I ) , so that it implicitly enforces independence between the latent variables Z = [

Original: Z y
Revised: Z y

Original: [Figure]
Revised: [Figure]

Original: Equation 4 represents our hybrid generative-discriminative optimization objective over multimodal data: the ﬁrst loss term ∑ Mi = 1 c X
Revised: Equation 4 represents our hybrid generative-discriminative optimization objective over multimodal data: the ﬁrst loss term ∑ Mi = 1 c X

Original: i ( X i ,F ( G ai ( Z a i ) ,G y
Revised: i ( X i ,F ( G ai ( Z a i ) ,G y

Original: ( Z y ))) is the generative objective based on reconstruction of multimodal data and the second term c Y ( Y ,D ( G y ( Z y ))) is the discriminative objective.
Revised: ( Z y ))) is the generative objective based on reconstruction of multimodal data and the second term c Y ( Y ,D ( G y ( Z y ))) is the discriminative objective.

Original: In practice we compute the expectations in Equation 4 using empirical estimates over the training data.
Revised: In practice we compute the expectations in Equation 4 using empirical estimates over the training data.

Original: The neural architecture of MFM is illustrated in Figure 1(c).
Revised: The neural architecture of MFM is illustrated in Figure 1(c).

Original: 2. S URROGATE I NFERENCE FOR M ISSING M ODALITIES
Revised: 2. S URROGATE I NFERENCE FOR M ISSING M ODALITIES

Original: A key challenge in multimodal learning involves dealing with missing modalities.
Revised: A key challenge in multimodal learning involves dealing with missing modalities.

Original: A good multimodal model should be able to infer the missing modality conditioned on the observed modalities and perform predictions based only on the observed modalities.
Revised: A good multimodal model should be able to infer the missing modality conditioned on the observed modalities and perform predictions based only on the observed modalities.

Original: To achieve this objective, the inference process of MFM can be easily adapted using a surrogate inference network to reconstruct the missing modality given the observed modalities.
Revised: To achieve this objective, the inference process of MFM can be easily adapted using a surrogate inference network to reconstruct the missing modality given the observed modalities.

Original: Formally, let Φ denote the surrogate inference network.
Revised: Formally, let Φ denote the surrogate inference network.

Original: The generation of missing modality ˆ X 1 given the observed modalities X 2 ∶ M can be formulated as
Revised: The generation of missing modality ˆ X 1 given the observed modalities X 2 ∶ M can be formulated as

Original: [Equation]
Revised: [Equation]

Original: Similar to Section 2.2, we use deterministic mappings in Q Φ (⋅∣⋅) and Q Φ ( Z y ∣⋅) is also used for prediction P
Revised: Similar to Section 2.2, we use deterministic mappings in Q Φ (⋅∣⋅) and Q Φ ( Z y ∣⋅) is also used for prediction P

Original: Φ ( ˆ Y ∣ X 2 ∶ M ) ∶= ∫ P [Equation] ˆ Y ∣ Z y ) Q
Revised: Φ ( ˆ Y ∣ X 2 ∶ M ) ∶= ∫ P ( ˆ Y ∣ Z y ) Q

Original: Φ ( Z y ∣ X 2 ∶ M ) d Z y .
Revised: Φ ( Z y ∣ X 2 ∶ M ) d Z y .

Original: Equation 5 suggests that in the presence of missing modalities, we only need to infer the latent codes rather than the entire modality.
Revised: Equation 5 suggests that in the presence of missing modalities, we only need to infer the latent codes rather than the entire modality.

Original: 2. E NCODER AND D ECODER D ESIGN
Revised: 2. E NCODER AND D ECODER D ESIGN

Original: We now discuss the implementation choices for the MFM neural architecture in Figure 1(c).
Revised: We now discuss the implementation choices for the MFM neural architecture in Figure 1(c).

Original: The encoder Q ( Z y ∣ X 1 ∶ M ) can be parametrized by any model that performs multimodal fusion (Morency et al., 2011; Zadeh et al., 2017).
Revised: The encoder Q ( Z y ∣ X 1 ∶ M ) can be parametrized by any model that performs multimodal fusion (Morency et al., 2011; Zadeh et al., 2017).

Original: For multimodal image datasets, we adopt Convolutional Neural
Revised: For multimodal image datasets, we adopt Convolutional Neural

Original: Networks (CNNs) and Fully-Connected Neural Networks (FCNNs) with late fusion (Nojavanasghari et al., 2016) as our encoder Q ( Z y ∣ X 1 ∶ M ) .
Revised: Networks (CNNs) and Fully-Connected Neural Networks (FCNNs) with late fusion (Nojavanasghari et al., 2016) as our encoder Q ( Z y ∣ X 1 ∶ M ) .

Original: The remaining functions in MFM are also parametrized by CNNs and FCNNs.
Revised: The remaining functions in MFM are also parametrized by CNNs and FCNNs.

Original: For multimodal time series datasets, we choose the Memory Fusion Network (MFN) (Zadeh et al., 2018a) as our multimodal encoder Q ( Z y ∣ X 1 ∶ M ) .
Revised: For multimodal time series datasets, we choose the Memory Fusion Network (MFN) (Zadeh et al., 2018a) as our multimodal encoder Q ( Z y ∣ X 1 ∶ M ) .

Original: We use Long Short-term
Revised: We use Long Short-term

Original: Memory (LSTM) networks (Hochreiter & Schmidhuber, 1997) for functions Q ( Z a { 1 ∶ M } ∣ X 1 ∶ M ) , decoder LSTM networks (Cho et al., 2014) for functions F 1 ∶ M , and FCNNs for functions
Revised: Memory (LSTM) networks (Hochreiter & Schmidhuber, 1997) for functions Q ( Z a { 1 ∶ M } ∣ X 1 ∶ M ) , decoder LSTM networks (Cho et al., 2014) for functions F 1 ∶ M , and FCNNs for functions

Original: G y ,
Revised: G y ,

Original: G a { 1 ∶ M } and D .
Revised: G a { 1 ∶ M } and D .

Original: Details are provided in the appendix and the code is available at <anonymous> .
Revised: Details are provided in the appendix and the code is available at <anonymous> .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In order to show that MFM learns multimodal representations that are discriminative, generative and interpretable, we design the following experiments.
Revised: In order to show that MFM learns multimodal representations that are discriminative, generative and interpretable, we design the following experiments.

Original: We begin with a multimodal synthetic image dataset that allows us to examine whether MFM displays discriminative and generative capabilities from factorized latent variables.
Revised: We begin with a multimodal synthetic image dataset that allows us to examine whether MFM displays discriminative and generative capabilities from factorized latent variables.

Original: Utilizing image datasets allows us to clearly visualize the generative capabilities of MFM.
Revised: Utilizing image datasets allows us to clearly visualize the generative capabilities of MFM.

Original: We then transition to six more challenging real-world multimodal video datasets to 1) rigorously evaluate the discriminative capabilities of MFM in comparison with existing baselines, 2) analyze the importance of each design component through ablation studies, 3) assess the robustness of MFM’s modality reconstruction and prediction capabilities to missing modalities, and 4) interpret 
Revised: We then transition to six more challenging real-world multimodal video datasets to 1) rigorously evaluate the discriminative capabilities of MFM in comparison with existing baselines, 2) analyze the importance of each design component through ablation studies, 3) assess the robustness of MFM’s modality reconstruction and prediction capabilities to missing modalities, and 4) interpret the learned representations using information-based and gradient-based methods to understand the contributions of individual factors towards multimodal prediction and generation.

Original: [Table]
Revised: [Table]

Original: the learned representations using information-based and gradient-based methods to understand the contributions of individual factors towards multimodal prediction and generation.

Original: 3. M ULTIMODAL S YNTHETIC I MAGE D ATASET
Revised: M ULTIMODAL S YNTHETIC I MAGE D ATASET

Original: In this section, we study MFM on a synthetic image dataset that considers SVHN (Netzer et al., 2011) and MNIST (Lecun et al., 1998) as the two modalities.
Revised: In this section, we study MFM on a synthetic image dataset that considers SVHN (Netzer et al., 2011) and MNIST (Lecun et al., 1998) as the two modalities.

Original: SVHN and MNIST are images with different styles but the same labels (digits 0 ∼ 9 ).
Revised: SVHN and MNIST are images with different styles but the same labels (digits 0 ∼ 9 ).

Original: We randomly pair 100 , 000 SVHN and MNIST images that have the same label, creating a multimodal dataset which we call SVHN+MNIST.
Revised: We randomly pair 100 , 000 SVHN and MNIST images that have the same label, creating a multimodal dataset which we call SVHN+MNIST.

Original: 80 , 000 pairs are used for training and the rest for testing.
Revised: 80 , 000 pairs are used for training and the rest for testing.

Original: To justify that MFM is able to learn improved multimodal representations, we show both classiﬁcation and generation results on SVHN+MNIST in Figure 2.
Revised: To justify that MFM is able to learn improved multimodal representations, we show both classiﬁcation and generation results on SVHN+MNIST in Figure 2.

Original: Prediction: We perform experiments on both unimodal and multimodal classiﬁcation tasks.
Revised: Prediction: We perform experiments on both unimodal and multimodal classiﬁcation tasks.

Original: UM denotes a unimodal baseline that performs prediction given only one modality as input and MM denotes a multimodal discriminative baseline that performs prediction given both images (Nojavanasghari et al., 2016).
Revised: UM denotes a unimodal baseline that performs prediction given only one modality as input and MM denotes a multimodal discriminative baseline that performs prediction given both images (Nojavanasghari et al., 2016).

Original: We compare the results for UM(SVHN), UM(MNIST), MM and MFM on
Revised: We compare the results for UM(SVHN), UM(MNIST), MM and MFM on

Original: SVHN+MNIST in Figure 2(b).
Revised: SVHN+MNIST in Figure 2(b).

Original: We achieve better classiﬁcation performance from unimodal to multimodal which is not surprising since more information is given.
Revised: We achieve better classiﬁcation performance from unimodal to multimodal which is not surprising since more information is given.

Original: More importantly, MFM outperforms
Revised: More importantly, MFM outperforms

Original: MM, which suggests that MFM learns improved factorized representations for discriminative tasks.
Revised: MM, which suggests that MFM learns improved factorized representations for discriminative tasks.

Original: Generation: We generate images using the MFM generative network (Figure 2(a)).
Revised: Generation: We generate images using the MFM generative network (Figure 2(a)).

Original: We ﬁx one variable out of Z =
Revised: We ﬁx one variable out of Z =

Original: [ Z a 1 , Z a 2 , and Z y ] and randomly sample the other two variables from prior P Z .
Revised: [ Z a 1 , Z a 2 , and Z y ] and randomly sample the other two variables from prior P Z .

Original: From Figure 2(c), we observe that MFM shows ﬂexible generation of SVHN and MNIST images based on labels and styles.
Revised: From Figure 2(c), we observe that MFM shows ﬂexible generation of SVHN and MNIST images based on labels and styles.

Original: This suggests that MFM is able to factorize multimodal representations into multimodal discriminative factors (labels) and modality-speciﬁc generative factors (styles).
Revised: This suggests that MFM is able to factorize multimodal representations into multimodal discriminative factors (labels) and modality-speciﬁc generative factors (styles).

Original: 3. M ULTIMODAL T IME S ERIES D ATASETS
Revised: 3. M ULTIMODAL T IME S ERIES D ATASETS

Original: In this section, we transition to more challenging multimodal time series datasets.
Revised: In this section, we transition to more challenging multimodal time series datasets.

Original: All the datasets consist of monologue videos.
Revised: All the datasets consist of monologue videos.

Original: Features are extracted from the language (GloVe word embeddings (Pennington et al., 2014)), visual (Facet (iMotions, 2017)), and acoustic (COVAREP (Degottex et al., 2014)) modalities.
Revised: Features are extracted from the language (GloVe word embeddings (Pennington et al., 2014)), visual (Facet (iMotions, 2017)), and acoustic (COVAREP (Degottex et al., 2014)) modalities.

Original: For a detailed description of feature extraction, please refer to the appendix.
Revised: For a detailed description of feature extraction, please refer to the appendix.

Original: We consider the following six datasets across three domains: 1) Multimodal Personality Trait
Revised: We consider the following six datasets across three domains: 1) Multimodal Personality Trait

Original: Recognition: POM (Park et al., 2014) contains 903 movie review videos annotated for the following 
Revised: Recognition: POM (Park et al., 2014) contains 903 movie review videos annotated for the following personality traits: conﬁdent (con), passionate (pas), voice pleasant (voi), dominant (dom), credible (cre), vivid (viv), expertise (exp), entertaining (ent), reserved (res), trusting (tru), relaxed (rel), outgoing (out), thorough (tho), nervous (ner), persuasive (per) and humorous (hum). The short form

Original: Model Multimodal
Revised: Model Multimodal

Original: Hybrid
Revised: Hybrid

Original: Factorized Mod.-Spec.
Revised: Factorized Mod.-Spec.

Original: CMU-MOSI Disc.
Revised: CMU-MOSI Disc.

Original: Gen.-Disc. Gen.-Disc.
Revised: Gen.-Disc. Gen.-Disc.

Original: Gen.
Revised: Gen.

Original: ˆ X ⋅ Reconstruction ˆ Y Prediction
Revised: ˆ X ⋅ Reconstruction ˆ Y Prediction

Original: Factor
Revised: Factor

Original: Objective
Revised: Objective

Original: Factors Factors
Revised: Factors Factors

Original: MSE ( (cid:96) )
Revised: MSE ( (cid:96) )

Original: MSE ( a ) MSE ( v ) Acc_ 7 Acc_ 2 F
Revised: MSE ( a ) MSE ( v ) Acc_ 7 Acc_ 2 F

Original: MAE r
Revised: MAE r

Original: M A
Revised: M A

Original: no no - - - - - 33.
Revised: no no - - - - - 33.

Original: 75.2 1.020 0.616 M
Revised: 75.2 1.020 0.616 M

Original: B yes no - - - - - 34.
Revised: B yes no - - - - - 34.

Original: 77.3 0.965 0.632 M
Revised: 77.3 0.965 0.632 M

Original: C no
Revised: C no

Original: yes no0.
Revised: yes no0.

Original: 76.0 0.979 0.640 M
Revised: 76.0 0.979 0.640 M

Original: D
Revised: D

Original: yes yes no0.
Revised: yes yes no0.

Original: 77.2 0.960 0.649 M
Revised: 77.2 0.960 0.649 M

Original: E
Revised: E

Original: yes yes yes
Revised: yes yes yes

Original: no 0.
Revised: no 0.

Original: [Figure]
Revised: [Figure]

Original: yes yes
Revised: yes yes

Original: yes yes 0.
Revised: yes yes 0.

Original: [Figure]
Revised: [Figure]

Original: personality traits: conﬁdent (con), passionate (pas), voice pleasant (voi), dominant (dom), credible (cre), vivid (viv), expertise (exp), entertaining (ent), reserved (res), trusting (tru), relaxed (rel), outgoing (out), thorough (tho), nervous (ner), persuasive (per) and humorous (hum).

Original: The short form is indicated in parenthesis.
Revised: is indicated in parenthesis.

Original: 2) Multimodal Sentiment Analysis: CMU-MOSI (Zadeh et al., 2016) is a collection of 2199 monologue opinion video clips annotated with sentiment.
Revised: 2) Multimodal Sentiment Analysis: CMU-MOSI (Zadeh et al., 2016) is a collection of 2199 monologue opinion video clips annotated with sentiment.

Original: ICT-MMMO (Wöllmer et al., 2013) consists of 340 online social review videos annotated for sentiment.
Revised: ICT-MMMO (Wöllmer et al., 2013) consists of 340 online social review videos annotated for sentiment.

Original: YouTube (Morency et al., 2011) contains 269 product review and opinion video segments from YouTube each annotated for sentiment.
Revised: YouTube (Morency et al., 2011) contains 269 product review and opinion video segments from YouTube each annotated for sentiment.

Original: MOUD (Perez-Rosas et al., 2013) consists of 79 product review videos in Spanish.
Revised: MOUD (Perez-Rosas et al., 2013) consists of 79 product review videos in Spanish.

Original: Each video consists of multiple segments labeled as either positive, negative or neutral sentiment.
Revised: Each video consists of multiple segments labeled as either positive, negative or neutral sentiment.

Original: 3) Multimodal Emotion Recognition: IEMOCAP (Busso et al., 2008) consists of 302 videos of recorded dyadic dialogues.
Revised: 3) Multimodal Emotion Recognition: IEMOCAP (Busso et al., 2008) consists of 302 videos of recorded dyadic dialogues.

Original: The videos are divided into multiple segments each annotated for the presence of 6 discrete emotions (happy, sad, angry, frustrated, excited and neutral), resulting in a total of 7318 segments in the dataset.
Revised: The videos are divided into multiple segments each annotated for the presence of 6 discrete emotions (happy, sad, angry, frustrated, excited and neutral), resulting in a total of 7318 segments in the dataset.

Original: We report results using the following metrics:
Revised: We report results using the following metrics:

Original: Acc_ C = multiclass accuracy across C classes, F1 = F1 score, MAE =
Revised: Acc_ C = multiclass accuracy across C classes, F1 = F1 score, MAE =

Original: Mean Absolute Error, r = Pearson’s correlation.
Revised: Mean Absolute Error, r = Pearson’s correlation.

Original: Prediction: We ﬁrst compare the performance of MFM with existing multimodal prediction methods.
Revised: Prediction: We ﬁrst compare the performance of MFM with existing multimodal prediction methods.

Original: For a detailed description of the baselines, please refer to the appendix.
Revised: For a detailed description of the baselines, please refer to the appendix.

Original: From Table 1, we ﬁrst observe that the best performing baseline results are achieved by different models across different datasets (most notably MFN, MARN, and TFN).
Revised: From Table 1, we ﬁrst observe that the best performing baseline results are achieved by different models across different datasets (most notably MFN, MARN, and TFN).

Original: On the other hand, MFM consistently achieves state-of-the-art or competitive results for all six multimodal datasets.
Revised: On the other hand, MFM consistently achieves state-of-the-art or competitive results for all six multimodal datasets.

Original: We believe that the multimodal discriminative factor F y in MFM has successfully learned more meaningful representations by distilling discriminative features.
Revised: We believe that the multimodal discriminative factor F y in MFM has successfully learned more meaningful representations by distilling discriminative features.

Original: This highlights the beneﬁt of learning factorized multimodal representations towards discriminative tasks.
Revised: This highlights the beneﬁt of learning factorized multimodal representations towards discriminative tasks.

Original: Furthermore, MFM is model-agnostic and can be applied to other multimodal encoders Q ( Z y ∣ X 1 ∶ M ) .
Revised: Furthermore, MFM is model-agnostic and can be applied to other multimodal encoders Q ( Z y ∣ X 1 ∶ M ) .

Original: We perform experiments to show consistent improvements in discriminative performance for several choices of the encoder: EF-LSTM (Morency et al., 2011) and TFN (Zadeh et al., 2017).
Revised: We perform experiments to show consistent improvements in discriminative performance for several choices of the encoder: EF-LSTM (Morency et al., 2011) and TFN (Zadeh et al., 2017).

Original: For Acc_ 2 on CMU-MOSI, our factorization framework improves the performance of EF-LSTM from 74 .
Revised: For Acc_ 2 on CMU-MOSI, our factorization framework improves the performance of EF-LSTM from 74 .

Original: 3 to 75.2 and TFN from 74 . 6 to 75.5 .
Revised: 3 to 75.2 and TFN from 74 . 6 to 75.5 .

Original: Ablation Study:
Revised: Ablation Study:

Original: In Figure 3, we present the models M { A , B , C , D , E } used for ablation studies.
Revised: In Figure 3, we present the models M { A , B , C , D , E } used for ablation studies.

Original: These models are designed to analyze the effects of using a multimodal discriminative factor, a hybrid generative-discriminative objective, factorized generative-discriminative factors and modality-speciﬁc generative factors towards both modality reconstruction and label prediction.
Revised: These models are designed to analyze the effects of using a multimodal discriminative factor, a hybrid generative-discriminative objective, factorized generative-discriminative factors and modality-speciﬁc generative factors towards both modality reconstruction and label prediction.

Original: The simplest variant is M A which represents a purely discriminative model without a joint multimodal discriminative factor (i.e. early fusion (Morency et al., 2011)).
Revised: The simplest variant is M A which represents a purely discriminative model without a joint multimodal discriminative factor (i.e. early fusion (Morency et al., 2011)).

Original: M B models a joint multimodal discriminative factor which incorporates more general multimodal fusion encoders (Zadeh et al., 2018a).
Revised: M B models a joint multimodal discriminative factor which incorporates more general multimodal fusion encoders (Zadeh et al., 2018a).

Original: M C extends
Revised: M C extends

Original: M A by optimizing a hybrid generative-discriminative objective over modality-speciﬁc factors.
Revised: M A by optimizing a hybrid generative-discriminative objective over modality-speciﬁc factors.

Original: M D extends M B by optimizing a hybrid generative-discriminative objective over a joint multimodal factor (resembling Srivastava & Salakhutdinov (2012)).
Revised: M D extends M B by optimizing a hybrid generative-discriminative objective over a joint multimodal factor (resembling Srivastava & Salakhutdinov (2012)).

Original: M E factorizes the representation into separate generative and discriminative factors.
Revised: M E factorizes the representation into separate generative and discriminative factors.

Original: Finally, MFM is obtained from M E by using modality-speciﬁc generative factors instead of a joint multimodal generative factor.
Revised: Finally, MFM is obtained from M E by using modality-speciﬁc generative factors instead of a joint multimodal generative factor.

Original: [Table] From the table in Figure [Table] we observe the following general trends.
Revised: From the table in Figure 3, we observe the following general trends.

Original: For sentiment prediction, using 1) a multimodal discriminative factor outperforms modality-speciﬁc discriminative factors( M D > M C , M B > M A )
Revised: For sentiment prediction, using 1) a multimodal discriminative factor outperforms modality-speciﬁc discriminative factors

Original: [Table]
Revised: [Table]

Original:  and 2) adding generative capabilities to the model improves performance ( M C > M A , M E > M B ).
Revised: ( M D > M C , M B > M A ), and 2) adding generative capabilities to the model improves performance ( M C > M A , M E > M B ).

Original: For both sentiment prediction and modality reconstruction, 3) factorizing into separate generative and discriminative factors improves performance ( M E > M D ), and 4) using modality-speciﬁc generative factors outperforms multimodal generative factors (MFM > M E ).
Revised: For both sentiment prediction and modality reconstruction, 3) factorizing into separate generative and discriminative factors improves performance ( M E > M D ) [Table] and 4) using modality-speciﬁc generative factors outperforms multimodal generative factors (MFM > M E ).

Original: These observations support our design decisions of factorizing multimodal representations into multimodal discriminative factors and modality-speciﬁc generative factors.
Revised: These observations support our design decisions of factorizing multimodal representations into multimodal discriminative factors and modality-speciﬁc generative factors.

Original: Missing Modalities: We now evaluate the performance of MFM in the presence of missing modalities using the surrogate inference model as described in Subsection 2.3.
Revised: Missing Modalities: We now evaluate the performance of MFM in the presence of missing modalities using the surrogate inference model as described in Subsection 2.3.

Original: We compare with two baselines: 1) a purely generative Seq2Seq model (Cho et al., 2014)
Revised: We compare with two baselines: 1) a purely generative Seq2Seq model (Cho et al., 2014)

Original: Φ G from observed modalities to missing modalities by optimizing E P X 1 ∶ M (− log P
Revised: Φ G from observed modalities to missing modalities by optimizing E P X 1 ∶ M (− log P

Original: Φ D ( X 1 ∣ X 2 ∶ M )) , and 2) a purely discriminative model Φ D from observed modalities to the label by optimizing E P X 2 ∶ M, Y (− log P Φ G ( Y ∣ X 2 ∶ M )) .
Revised: Φ D ( X 1 ∣ X 2 ∶ M )) , and 2) a purely discriminative model Φ D from observed modalities to the label by optimizing E P X 2 ∶ M, Y (− log P Φ G ( Y ∣ X 2 ∶ M )) .

Original: Both models are modiﬁed from MFM by using only the two observed modalities as input and not explicitly accounting for missing modalities.
Revised: Both models are modiﬁed from MFM by using only the two observed modalities as input and not explicitly accounting for missing modalities.

Original: We compare the reconstruction error of each modality (language, visual and acoustic) as well as the performance on sentiment prediction.
Revised: We compare the reconstruction error of each modality (language, visual and acoustic) as well as the performance on sentiment prediction.

Original: Table 2 shows that MFM with missing modalities outperforms the generative ( Φ G ) or discriminative baselines ( Φ D ) in terms of modality reconstruction and sentiment prediction.
Revised: Table 2 shows that MFM with missing modalities outperforms the generative ( Φ G ) or discriminative baselines ( Φ D ) in terms of modality reconstruction and sentiment prediction.

Original: Additionally, MFM with missing modalities performs close to MFM with all modalities observed.
Revised: Additionally, MFM with missing modalities performs close to MFM with all modalities observed.

Original: This fact indicates that MFM can learn representations that are relatively robust to missing modalities.
Revised: This fact indicates that MFM can learn representations that are relatively robust to missing modalities.

Original: In addition, discriminative performance is most affected when the language modality is missing, which is consistent with prior work which indicates that language is most informative in human multimodal language (Zadeh et al., 2017).
Revised: In addition, discriminative performance is most affected when the language modality is missing, which is consistent with prior work which indicates that language is most informative in human multimodal language (Zadeh et al., 2017).

Original: On the other hand, sentiment prediction is more robust to missing acoustic and visual features.
Revised: On the other hand, sentiment prediction is more robust to missing acoustic and visual features.

Original: Finally, we observe that reconstructing the low-level acoustic and visual features is easier as compared to the high-dimensional language features that contain high-level semantic meaning.
Revised: Finally, we observe that reconstructing the low-level acoustic and visual features is easier as compared to the high-dimensional language features that contain high-level semantic meaning.

Original: Interpretation of Multimodal Representations: We devise two methods to study how individual factors in MFM inﬂuence the dynamics of multimodal prediction and generation.
Revised: Interpretation of Multimodal Representations: We devise two methods to study how individual factors in MFM inﬂuence the dynamics of multimodal prediction and generation.

Original: These interpretation methods represent both overall trends and ﬁne-grained analysis that could be useful towards deeper understandings of multimodal representation learning.
Revised: These interpretation methods represent both overall trends and ﬁne-grained analysis that could be useful towards deeper understandings of multimodal representation learning.

Original: For more details, please refer to the appendix.
Revised: For more details, please refer to the appendix.

Original: Firstly, an information-based interpretation method is chosen to summarize the contribution of each modality towards the multimodal representations.
Revised: Firstly, an information-based interpretation method is chosen to summarize the contribution of each modality towards the multimodal representations.

Original: Since F y is a common cause of ˆ X 1 ∶ M , we can compare MI ( F y , ˆ X 1 ) , ⋯ , MI ( F y , ˆ X M ) , where MI (⋅ , ⋅) denotes the mutual information measure between F y and generated modality
Revised: Since F y is a common cause of ˆ X 1 ∶ M , we can compare MI ( F y , ˆ X 1 ) , ⋯ , MI ( F y , ˆ X M ) , where MI (⋅ , ⋅) denotes the mutual information measure between F y and generated modality

Original: ˆ X i .
Revised: ˆ X i .

Original: Higher MI ( F y , ˆ X i ) indicates greater contribution from
Revised: Higher MI ( F y , ˆ X i ) indicates greater contribution from

Original: F y to ˆ X i .
Revised: F y to ˆ X i .

Original: Figure 4 reports the ratios r i =
Revised: Figure 4 reports the ratios r i =

Original: MI ( F y , ˆ X i )/ MI ( F a i , ˆ X i ) which measure a normalized version of the mutual information between F a i and ˆ X i .
Revised: MI ( F y , ˆ X i )/ MI ( F a i , ˆ X i ) which measure a normalized version of the mutual information between F a i and ˆ X i .

Original: We observe that on CMU-MOSI, the language modality is most informative towards sentiment prediction, followed by the acoustic modality.
Revised: We observe that on CMU-MOSI, the language modality is most informative towards sentiment prediction, followed by the acoustic modality.

Original: We believe that this result represents a prior over the expression of sentiment in human multimodal language and is closely related to the connections between language and speech (Kuhl, 2000).
Revised: We believe that this result represents a prior over the expression of sentiment in human multimodal language and is closely related to the connections between language and speech (Kuhl, 2000).

Original: Secondly, a gradient-based interpretation method to used analyze the contribution of each modality for every time step in multimodal time series data.
Revised: Secondly, a gradient-based interpretation method to used analyze the contribution of each modality for every time step in multimodal time series data.

Original: We measure the gradient of the generated modality with respect to the target factors (e.g., F y ).
Revised: We measure the gradient of the generated modality with respect to the target factors (e.g., F y ).

Original: Let { x 1 ,x 2 , ⋯ ,x M } denote multimodal time 
Revised: Let { x 1 ,x 2 , ⋯ ,x M } denote multimodal time series data where x i represents modality i , and ˆ x i = [ ˆ x 1 i , ⋯ , ˆ x ti , ⋯ , ˆ x Ti ] denote generated modality i across time steps t ∈ [ 1 ,T ] . The gradient ∇ f y ( ˆ x i ) measures the extent to which changes in factor

Original: Ratio r (cid:96) r v r a
Revised: Ratio r (cid:96) r v r a

Original: CMU-MOSI 0.
Revised: [Figure] Umm, in a [Figure]

Original: Umm, in a way, a lot of the themes in “never let me go”, which were very profound and deep.
Revised: , a lot of the themes in “never let me go”, which [Figure] profound and deep.

Original: (hesitancy) (emphasis) (neutral) language visual acoustic 𝜵𝒇 𝒚 (𝒙& ℓ ) 𝜵𝒇 𝒚 (𝒙& 𝒗 ) 𝜵𝒇 𝒚 (𝒙& 𝒂 ) 𝑡 = 𝑡 = ( un i n f o r m a ti v e ) ( un i n f o r m a ti v e ) ( s li g h t s m il e )series data where x i represents modality i , and ˆ x i
Revised: (hesitancy) (emphasis) (neutral) language visual acoustic 𝜵𝒇 𝒚 (𝒙& ℓ ) 𝜵𝒇 𝒚 (𝒙& 𝒗 ) 𝜵𝒇 𝒚 (𝒙& 𝒂 ) 𝑡 = 𝑡 = ( un i n f o r m a ti v e ) ( un i n f o r m a ti v e ) ( s li g h t s m il e )

Original: = [ ˆ x 1 i , ⋯ , ˆ x ti , ⋯ , ˆ x Ti ] denote generated modality i across time steps t ∈ [ 1 ,T ] .
Revised: f y ∼ P

Original: The gradient ∇ f y ( ˆ x i ) measures the extent to which changes in factor f y ∼ P ( F y ∣ X 1 ∶ M = x 1 ∶ M ) inﬂuences the generation of sequence ˆ x i .
Revised: ( F y ∣ X 1 ∶ M = x 1 ∶ M ) inﬂuences the generation of sequence ˆ x i .

Original: Figure 4 plots ∇ f y ( ˆ x i ) for a video in CMU-MOSI.
Revised: Figure 4 plots ∇ f y ( ˆ x i ) for a video in CMU-MOSI.

Original: We observe that multimodal communicative behaviors that are indicative of speaker sentiment such as positive words (e.g. “very profound and deep”) and informative acoustic features (e.g. hesitant and emphasized tone of voice) indeed correspond to increases in ∇ f y ( ˆ x i ) .
Revised: We observe that multimodal communicative behaviors that are indicative of speaker sentiment such as positive words (e.g. “very profound and deep”) and informative acoustic features (e.g. hesitant and emphasized tone of voice) indeed correspond to increases in ∇ f y ( ˆ x i ) .

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The two main pillars of research in multimodal representation learning have considered the discriminative and generative objectives individually.
Revised: The two main pillars of research in multimodal representation learning have considered the discriminative and generative objectives individually.

Original: Discriminative representation learning (Liang et al., 2018;
Revised: Discriminative representation learning (Liang et al., 2018;

Original: Chen et al., 2017; Chaplot et al., 2017; Frome et al., 2013; Socher et al., 2013; Tsai et al., 2017) models the conditional distribution P
Revised: Chen et al., 2017; Chaplot et al., 2017; Frome et al., 2013; Socher et al., 2013; Tsai et al., 2017) models the conditional distribution P

Original: ( Y ∣ X 1 ∶ M ) .
Revised: ( Y ∣ X 1 ∶ M ) .

Original: Since these approaches are not concerned with modeling P ( X 1 ∶ M ) explicitly, they use parameters more efﬁciently to model P ( Y ∣ X 1 ∶ M ) .
Revised: Since these approaches are not concerned with modeling P ( X 1 ∶ M ) explicitly, they use parameters more efﬁciently to model P ( Y ∣ X 1 ∶ M ) .

Original: For instance, recent works learn visual representations that are maximally dependent with linguistic attributes for improving one-shot image recognition (Tsai & Salakhutdinov, 2017) or introduce tensor product mechanisms to model interactions between the language, visual and acoustic modalities (Liu et al., 2018; Zadeh et al., 2017).
Revised: For instance, recent works learn visual representations that are maximally dependent with linguistic attributes for improving one-shot image recognition (Tsai & Salakhutdinov, 2017) or introduce tensor product mechanisms to model interactions between the language, visual and acoustic modalities (Liu et al., 2018; Zadeh et al., 2017).

Original: On the other hand, generative representation learning captures the interactions between modalities by modeling the joint distribution P ( X 1 , ⋯ , X M ) using either undirected graphical models (Srivastava & Salakhutdinov, 2012), directed graphical models (Suzuki et al., 2016), or neural networks (Sohn et al., 2014).
Revised: On the other hand, generative representation learning captures the interactions between modalities by modeling the joint distribution P ( X 1 , ⋯ , X M ) using either undirected graphical models (Srivastava & Salakhutdinov, 2012), directed graphical models (Suzuki et al., 2016b), or neural networks (Sohn et al., 2014).

Original: Some generative approaches compress multimodal data into lower-dimensional feature vectors which can be used for discriminative tasks (Pham et al., 2018; Ngiam et al., 2011).
Revised: Some generative approaches compress multimodal data into lower-dimensional feature vectors which can be used for discriminative tasks (Pham et al., 2018; Ngiam et al., 2011).

Original: To unify the advantages of both approaches, MFM factorizes multimodal representations into generative and discriminative components and optimizes for a joint objective.
Revised: To unify the advantages of both approaches, MFM factorizes multimodal representations into generative and discriminative components and optimizes for a joint objective.

Original: Factorized representation learning resembles learning disentangled data representations which have been shown to improve the performance on many tasks (Kulkarni et al., 2015;
Revised: Factorized representation learning resembles learning disentangled data representations which have been shown to improve the performance on many tasks (Kulkarni et al., 2015;

Original: Lake et al., 2017;
Revised: Lake et al., 2017;

Original: Higgins et al., 2016; Bengio et al., 2013).
Revised: Higgins et al., 2016; Bengio et al., 2013).

Original: Several methods involve specifying a ﬁxed set of latent attributes that individually control particular variations of data and performing supervised training (Cheung et al., 2014; Karaletsos et al., 2015;
Revised: Several methods involve specifying a ﬁxed set of latent attributes that individually control particular variations of data and performing supervised training (Cheung et al., 2014; Karaletsos et al., 2015;

Original: Yang et al., 2015; Reed et al., 2014; Zhu et al., 2014), assuming an isotropic Gaussian prior over latent variables to learn disentangled generative representations (Kingma & Welling, 2013; Rubenstein et al., 2018) and learning latent variables in charge of speciﬁc variations in the data by maximizing the mutual information between a subset of latent variables and the data (Chen et al., 2016).
Revised: Yang et al., 2015; Reed et al., 2014; Zhu et al., 2014), assuming an isotropic Gaussian prior over latent variables to learn disentangled generative representations (Kingma & Welling, 2013; Rubenstein et al., 2018) and learning latent variables in charge of speciﬁc variations in the data by maximizing the mutual information between a subset of latent variables and the data (Chen et al., 2016).

Original: However, these methods study factorization of a single modality.
Revised: However, these methods study factorization of a single modality.

Original: MFM factorizes multimodal representations and demonstrates the importance of modality-speciﬁc and multimodal factors towards generation and prediction.
Revised: MFM factorizes multimodal representations and demonstrates the importance of modality-speciﬁc and multimodal factors towards generation and prediction.

Original: A concurrent and parallel work that factorizes latent factors in multimodal data was proposed by Hsu & Glass (2018).
Revised: A concurrent and parallel work that factorizes latent factors in multimodal data was proposed by Hsu & Glass (2018).

Original: They differ from us in the graphical model design, discriminative objective, prior matching criterion, and scale of experiments.
Revised: They differ from us in the graphical model design, discriminative objective, prior matching criterion, and scale of experiments.

Original: We provide a detailed comparison with their model in the appendix.
Revised: We provide a detailed comparison with their model in the appendix.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we proposed the Multimodal Factorization Model (MFM) for multimodal representation learning.
Revised: In this paper, we proposed the Multimodal Factorization Model (MFM) for multimodal representation learning.

Original: MFM factorizes the multimodal representations into two sets of independent factors: multimodal discriminative factors and modality-speciﬁc generative factors.
Revised: MFM factorizes the multimodal representations into two sets of independent factors: multimodal discriminative factors and modality-speciﬁc generative factors.

Original: The multimodal discriminative factor achieves state-of-the-art or competitive results on six multimodal datasets.
Revised: The multimodal discriminative factor achieves state-of-the-art or competitive results on six multimodal datasets.

Original: The modalityspeciﬁc generative factors allow us to generate data based on factorized variables, account for missing modalities, and have a deeper understanding of the interactions involved in multimodal learning.
Revised: The modalityspeciﬁc generative factors allow us to generate data based on factorized variables, account for missing modalities, and have a deeper understanding of the interactions involved in multimodal learning.

Original: Our future work will explore extensions of MFM for video generation, semi-supervised learning, and unsupervised learning.
Revised: Our future work will explore extensions of MFM for video generation, semi-supervised learning, and unsupervised learning.

Original: We believe that MFM sheds light on the advantages of learning factorizing multimodal representations and potentially opens up new horizons for multimodal machine learning.
Revised: We believe that MFM sheds light on the advantages of learning factorizing multimodal representations and potentially opens up new horizons for multimodal machine learning.

Original: A P ROOF OF P ROPOSITION
Revised: A P ROOF OF P ROPOSITION

Original: To simplify the proof, we ﬁrst prove it for the unimodal case by considering the Wasserstein distance between P X , Y and P ˆX , ˆY .
Revised: To simplify the proof, we ﬁrst prove it for the unimodal case by considering the Wasserstein distance between P X , Y and P ˆX , ˆY .

Original: A . U NIMODAL J OINT -D ISTRIBUTION W ASSERSTEIN D ISTANCE
Revised: A. U NIMODAL J OINT -D ISTRIBUTION W ASSERSTEIN D ISTANCE

Original: Proposition 2.
Revised: Proposition 2.

Original: For any functions G y ∶ Z y → F y , G a ∶ Z a → F a , D ∶ F y → ˆY , and F ∶ F a , F y → ˆX , we have
Revised: For any functions G y ∶ Z y → F y , G a ∶ Z a → F a , D ∶ F y → ˆY , and F ∶ F a , F y → ˆX , we have

Original: [Equation]
Revised: [Equation]

Original: where W c is the Wasserstein distance under cost function c X and c Y , P Z is the prior over Z =
Revised: where W c is the Wasserstein distance under cost function c X and c Y , P Z is the prior over Z =

Original: Z a , Z y ] and Q Z is the aggregated posterior of the proposed inference distribution Q ( Z ∣ X ) .
Revised: Z a , Z y ] and Q Z is the aggregated posterior of the proposed inference distribution Q ( Z ∣ X ) .

Original: Proof: See the following.
Revised: Proof: See the following.

Original: To begin the proof, we abuse some notations as follows.
Revised: To begin the proof, we abuse some notations as follows.

Original: By deﬁnition, the Wasserstein distance under cost function c between P X , Y and P ˆX , ˆY is
Revised: By deﬁnition, the Wasserstein distance under cost function c between P X , Y and P ˆX , ˆY is

Original: [Equation]
Revised: [Equation]

Original: where c (( X , Y ) , ( ˆX , ˆY ))
Revised: where c (( X , Y ) , ( ˆX , ˆY ))

Original: ∶ (X , Y) ×
Revised: ∶ (X , Y) ×

Original: (X , Y)
Revised: (X , Y)

Original: → R + is any measurable cost function .
Revised: → R + is any measurable cost function .

Original: P(( X , Y ) ∼
Revised: P(( X , Y ) ∼

Original: P X , Y , ( ˆX , ˆY ) ∼ P ˆX , ˆY ) is the set of all joint distributions of (( X , Y ) , ( ˆX , ˆY )) with marginals
Revised: P X , Y , ( ˆX , ˆY ) ∼ P ˆX , ˆY ) is the set of all joint distributions of (( X , Y ) , ( ˆX , ˆY )) with marginals

Original: P X , Y and P ˆX , ˆY , respectively.
Revised: P X , Y and P ˆX , ˆY , respectively.

Original: Note that c (( X , Y ) , ( ˆX , ˆY ))
Revised: Note that c (( X , Y ) , ( ˆX , ˆY ))

Original: =
Revised: =

Original: c X ( X , ˆX ) + c Y ( Y , ˆY ) .
Revised: c X ( X , ˆX ) + c Y ( Y , ˆY ) .

Original: Next, we denote the set of all joint distributions of ( X , Y , ˆX , ˆY , Z ) such that ( X , Y ) ∼ P X , Y , ( ˆX , ˆY , Z ) ∼ P ˆX , ˆY , Z , and (( X , Y ) ⊥⊥ ( ˆX , ˆY )∣ Z ) as P X , Y , ˆX , ˆY , Z .
Revised: Next, we denote the set of all joint distributions of ( X , Y , ˆX , ˆY , Z ) such that ( X , Y ) ∼ P X , Y , ( ˆX , ˆY , Z ) ∼ P ˆX , ˆY , Z , and (( X , Y ) ⊥⊥ ( ˆX , ˆY )∣ Z ) as P X , Y , ˆX , ˆY , Z .

Original: P X , Y , ˆX , ˆY and P X , Y , Z are the sets of the marginals ( X , Y , ˆX , ˆY ) and ( X , Y , Z ) induced by P X , Y , ˆX , ˆY , Z .
Revised: P X , Y , ˆX , ˆY and P X , Y , Z are the sets of the marginals ( X , Y , ˆX , ˆY ) and ( X , Y , Z ) induced by P X , Y , ˆX , ˆY , Z .

Original: We now introduce two Lemmas to help the proof.
Revised: We now introduce two Lemmas to help the proof.

Original: Lemma 1.
Revised: Lemma 1.

Original: P ( ˆX , ˆY ∣ Z = z ) are Dirac for all z ∈ Z .
Revised: P ( ˆX , ˆY ∣ Z = z ) are Dirac for all z ∈ Z .

Original: Proof: First, we have ˆX =
Revised: Proof: First, we have ˆX =

Original: F ( G a ( Z a ) ,G y ( Z y )) and ˆY = D ( G y ( Z y )) with Z = { Z a , Z y } .
Revised: F ( G a ( Z a ) ,G y ( Z y )) and ˆY = D ( G y ( Z y )) with Z = { Z a , Z y } .

Original: Since the functions F,G a ,G y ,D are all deterministic, then P ( ˆX , ˆY ∣ Z ) are Dirac measures.
Revised: Since the functions F,G a ,G y ,D are all deterministic, then P ( ˆX , ˆY ∣ Z ) are Dirac measures.

Original: ◻
Revised: ◻

Original: Lemma 2.
Revised: Lemma 2.

Original: P( P X , Y ,P ˆX , ˆY ) =
Revised: P( P X , Y ,P ˆX , ˆY ) =

Original: P X , Y , ˆX , ˆY when P ( ˆX , ˆY ∣ Z = z ) are Dirac for all z ∈ Z .
Revised: P X , Y , ˆX , ˆY when P ( ˆX , ˆY ∣ Z = z ) are Dirac for all z ∈ Z .

Original: Proof: When ˆX , ˆY are deterministic functions of Z , for any A in the sigma-algebra induced by ˆX , ˆY , we have
Revised: Proof: When ˆX , ˆY are deterministic functions of Z , for any A in the sigma-algebra induced by ˆX , ˆY , we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, this implies that ( X , Y ) ⊥⊥ ( ˆ X , ˆ Y )∣ Z which concludes the proof.
Revised: Therefore, this implies that ( X , Y ) ⊥⊥ ( ˆ X , ˆ Y )∣ Z which concludes the proof.

Original: A similar argument is made in Lemma 1 of (Tolstikhin et al., 2017).
Revised: A similar argument is made in Lemma 1 of (Tolstikhin et al., 2017).

Original: ◻
Revised: ◻

Original: Now, we use the fact that P( P X , Y ,P ˆX , ˆY ) =
Revised: Now, we use the fact that P( P X , Y ,P ˆX , ˆY ) =

Original: P X , Y , ˆX , ˆY (Lemma 1 + Lemma 2),
Revised: P X , Y , ˆX , ˆY (Lemma 1 + Lemma 2),

Original: [Equation]
Revised: [Equation]

Original: Eq.
Revised: Eq.

Original: equation 7 becomes
Revised: equation 7 becomes

Original: [Equation]
Revised: [Equation]

Original: Note that in Eq.
Revised: Note that in Eq.

Original: equation 8, P X , Y , Z = P(( X , Y ) ∼ P X , Y , Z ∼ P Z ) and with a proposed Q ( Z ∣ X )
Revised: equation 8, P X , Y , Z = P(( X , Y ) ∼ P X , Y , Z ∼ P Z ) and with a proposed Q ( Z ∣ X )

Original: , we can rewrite Eq.
Revised: , we can rewrite Eq.

Original: equation 8 as
Revised: equation 8 as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: A . F ROM U NIMODAL TO M ULTIMODAL
Revised: A. F ROM U NIMODAL TO M ULTIMODAL

Original: The proof is similar to Proposition 2, and we present a sketch to it.
Revised: The proof is similar to Proposition 2, and we present a sketch to it.

Original: We can ﬁrst show
Revised: We can ﬁrst show

Original: P ( ˆX 1 ∶ M , ˆY ∣ Z = z ) are Dirac for all z ∈ Z .
Revised: P ( ˆX 1 ∶ M , ˆY ∣ Z = z ) are Dirac for all z ∈ Z .

Original: Then we use the fact that c (( X 1 ∶ M , Y ) , ( ˆX 1 ∶ M , ˆY ))
Revised: Then we use the fact that c (( X 1 ∶ M , Y ) , ( ˆX 1 ∶ M , ˆY ))

Original: =
Revised: =

Original: ∑ Mi = 1 c Xi ( X i , ˆX i ) + c Y ( Y , ˆY ) .
Revised: ∑ Mi = 1 c Xi ( X i , ˆX i ) + c Y ( Y , ˆY ) .

Original: Finally, we follow the tower rule of expectation and the conditional independence property similar to the proof in Proposition 2 and this concludes the proof.
Revised: Finally, we follow the tower rule of expectation and the conditional independence property similar to the proof in Proposition 2 and this concludes the proof.

Original: ∎
Revised: ∎

Original: B F ULL B ASELINE M ODELS & R ESULTS
Revised: B F ULL B ASELINE M ODELS & R ESULTS

Original: For a detailed description of the baselines, we point the reader to MFN (Zadeh et al., 2018a),
Revised: For a detailed description of the baselines, we point the reader to MFN (Zadeh et al., 2018a),

Original: MARN (Zadeh et al., 2018b), TFN (Zadeh et al., 2017), BC-LSTM (Poria et al., 2017), MVLSTM (Rajagopalan et al., 2016), EF-LSTM (Hochreiter & Schmidhuber, 1997; Graves et al., 2013; Schuster & Paliwal, 1997), DF (Nojavanasghari et al., 2016), MV-HCRF (Song et al., 2012; 2013), EF-HCRF (Quattoni et al., 2007; Morency et al., 2007), THMM (Morency et al., 2011),
Revised: MARN (Zadeh et al., 2018b), TFN (Zadeh et al., 2017), BC-LSTM (Poria et al., 2017), MVLSTM (Rajagopalan et al., 2016), EF-LSTM (Hochreiter & Schmidhuber, 1997; Graves et al., 2013; Schuster & Paliwal, 1997), DF (Nojavanasghari et al., 2016), MV-HCRF (Song et al., 2012; 2013), EF-HCRF (Quattoni et al., 2007; Morency et al., 2007), THMM (Morency et al., 2011),

Original: SVM-MD (Zadeh et al., 2016) and RF (Breiman, 2001).
Revised: SVM-MD (Zadeh et al., 2016) and RF (Breiman, 2001).

Original: We use the following extra notations for full descriptions of the baseline models described in
Revised: We use the following extra notations for full descriptions of the baseline models described in

Original: Subsection 3.3:
Revised: Section 3.2, paragraph 3:

Original: Variants of EF-LSTM: EF-LSTM (Early Fusion LSTM) uses a single LSTM (Hochreiter &
Revised: Variants of EF-LSTM: EF-LSTM (Early Fusion LSTM) uses a single LSTM (Hochreiter &

Original: Schmidhuber, 1997) on concatenated multimodal inputs.
Revised: Schmidhuber, 1997) on concatenated multimodal inputs.

Original: We also implement the EF-SLSTM (stacked) (Graves et al., 2013), EF-BLSTM (bidirectional) (Schuster & Paliwal, 1997) and EFSBLSTM ( versions.
Revised: We also implement the EF-SLSTM (stacked) (Graves et al., 2013), EF-BLSTM (bidirectional) (Schuster & Paliwal, 1997) and EFSBLSTM (stacked versions.

Original: Variants of EF-HCRF: EF-HCRF : (Hidden Conditional Random Field)
Revised: Variants of EF-HCRF: EF-HCRF : (Hidden Conditional Random Field)

Original: (Quattoni et al., 2007) uses a HCRF to learn a set of latent variables conditioned on the concatenated input at each time step.
Revised: (Quattoni et al., 2007) uses a HCRF to learn a set of latent variables conditioned on the concatenated input at each time step.

Original: EF-LDHCRF (Latent Discriminative HCRFs)
Revised: EF-LDHCRF (Latent Discriminative HCRFs)

Original: (Morency et al., 2007) are a class of models that learn hidden states in a CRF using a latent code between observed concatenated input and hidden output.
Revised: (Morency et al., 2007) are a class of models that learn hidden states in a CRF using a latent code between observed concatenated input and hidden output.

Original: EF-HSSHCRF : (Hierarchical Sequence Summarization HCRF)
Revised: EF-HSSHCRF : (Hierarchical Sequence Summarization HCRF)

Original: (Song et al., 2013) is a layered model that uses HCRFs with latent variables to learn hidden spatio-temporal dynamics.
Revised: (Song et al., 2013) is a layered model that uses HCRFs with latent variables to learn hidden spatio-temporal dynamics.

Original: Variants of MV-HCRF: MV-HCRF : Multi-view HCRF (Song et al., 2012) is an extension of the
Revised: Variants of MV-HCRF: MV-HCRF : Multi-view HCRF (Song et al., 2012) is an extension of the

Original: HCRF for Multi-view data, explicitly capturing view-shared and view speciﬁc sub-structures.
Revised: HCRF for Multi-view data, explicitly capturing view-shared and view speciﬁc sub-structures.

Original: MVLDHCRF :
Revised: MVLDHCRF :

Original: (Morency et al., 2007) is a variation of the MV-HCRF model that uses LDHCRF instead of HCRF.
Revised: (Morency et al., 2007) is a variation of the MV-HCRF model that uses LDHCRF instead of HCRF.

Original: MV-HSSHCRF : (Song et al., 2013) further extends EF-HSSHCRF by performing Multiview hierarchical sequence summary representation.
Revised: MV-HSSHCRF : (Song et al., 2013) further extends EF-HSSHCRF by performing Multiview hierarchical sequence summary representation.

Original: In the following, we provide the full results for all baselines models described in Subsection 3.3.
Revised: In the following, we provide the full results for all baselines models described in Section 3.2, paragraph 3.

Original: Table 3 contains results for multimodal speaker traits recognition on the POM dataset.
Revised: Table 3 contains results for multimodal speaker traits recognition on the POM dataset.

Original: Table 4 contains results for the multimodal sentiment analysis on the CMU-MOSI, ICT-MMMO, YouTube, and MOUD datasets.
Revised: Table 4 contains results for the multimodal sentiment analysis on the CMU-MOSI, ICT-MMMO, YouTube, and MOUD datasets.

Original: Table 5 contains results for multimodal emotion recognition on the IEMOCAP dataset.
Revised: Table 5 contains results for multimodal emotion recognition on the IEMOCAP dataset.

Original: MFM consistently achieves state-of-the-art or competitive results for all six multimodal datasets.
Revised: MFM consistently achieves state-of-the-art or competitive results for all six multimodal datasets.

Original: We believe that by our MFM design, the multimodal discriminative factor F y has successfully learned more meaningful representations by distilling discriminative features.
Revised: We believe that by our MFM design, the multimodal discriminative factor F y has successfully learned more meaningful representations by distilling discriminative features.

Original: This highlights the beneﬁt of learning factorized multimodal representations towards discriminative tasks.
Revised: This highlights the beneﬁt of learning factorized multimodal representations towards discriminative tasks.

Original: C M ULTIMODAL F EATURES
Revised: C M ULTIMODAL F EATURES

Original: For each of the multimodal time series datasets as mentioned in Section 3.3, we extracted the following multimodal features: Language: We use pre-trained word embeddings (glove.840B.300d)
Revised: For each of the multimodal time series datasets as mentioned in Section 3.2, paragraph 3, we extracted the following multimodal features: Language: We use pre-trained word embeddings (glove.840B.300d)

Original: (Pennington et al., 2014) to convert the video transcripts into a sequence of 300 dimensional word vectors.
Revised: (Pennington et al., 2014) to convert the video transcripts into a sequence of 300 dimensional word vectors.

Original: Visual: We use Facet (iMotions, 2017) to extract a set of features including per-frame basic and advanced emotions and facial action units as indicators of facial muscle movement (Ekman et al., 1980; Ekman, 1992).
Revised: Visual: We use Facet (iMotions, 2017) to extract a set of features including per-frame basic and advanced emotions and facial action units as indicators of facial muscle movement (Ekman et al., 1980; Ekman, 1992).

Original: Acoustic: We use COVAREP (Degottex et al., 2014) to extract low level acoustic features including 12 Mel-frequency cepstral coefﬁcients (MFCCs), pitch tracking and voiced/unvoiced segmenting features, glottal source parameters, peak slope parameters and maxima dispersion quotients.
Revised: Acoustic: We use COVAREP (Degottex et al., 2014) to extract low level acoustic features including 12 Mel-frequency cepstral coefﬁcients (MFCCs), pitch tracking and voiced/unvoiced segmenting features, glottal source parameters, peak slope parameters and maxima dispersion quotients.

Original: To reach the same time alignment between different modalities we choose the granularity of the input to be at the level of words.
Revised: To reach the same time alignment between different modalities we choose the granularity of the input to be at the level of words.

Original: The words are aligned with audio using
Revised: The words are aligned with audio using

Original: P2FA (Yuan & Liberman, 2008) to get their exact utterance times.
Revised: P2FA (Yuan & Liberman, 2008) to get their exact utterance times.

Original: We use expected feature values across the entire word for visual and acoustic features since they are extracted at a higher frequencies.
Revised: We use expected feature values across the entire word for visual and acoustic features since they are extracted at a higher frequencies.

Original: We make a note that the features for some of these datasets are constantly being updated.
Revised: We make a note that the features for some of these datasets are constantly being updated.

Original: The authors of Zadeh et al.
Revised: The authors of Zadeh et al.

Original: (2018a)
Revised: (2018a)

Original: notiﬁed us of a discrepancy in the sampling rate for acoustic feature extraction in the ICT-MMMO, YouTube and MOUD datasets which led to inaccurate word-level alignment between the three modalities.
Revised: notiﬁed us of a discrepancy in the sampling rate for acoustic feature extraction in the ICT-MMMO, YouTube and MOUD datasets which led to inaccurate word-level alignment between the three modalities.

Original: They publicly released the updated multimodal features.
Revised: They publicly released the updated multimodal features.

Original: We performed all experiments on the latest versions of these datasets which can be accessed from https: //github.com/A2Zadeh/CMU-MultimodalSDK .
Revised: We performed all experiments on the latest versions of these datasets which can be accessed from https: //github.com/A2Zadeh/CMU-MultimodalSDK .

Original: All baseline models were retrained with extensive hyperparameter search for fair comparison.
Revised: All baseline models were retrained with extensive hyperparameter search for fair comparison.

Original: D I NFORMATION AND G RADIENT -B ASED I NTERPRETATION
Revised: D I NFORMATION AND G RADIENT -B ASED I NTERPRETATION

Original: Information-Based Interpretation
Revised: Information-Based Interpretation

Original: : We choose the normalized Hilbert-Schmidt Independence
Revised: : We choose the normalized Hilbert-Schmidt Independence

Original: Criterion (Gretton et al., 2005; Wu et al., 2018) as the approximation (see Sugiyama & Yamada (2012); Wu et al.
Revised: Criterion (Gretton et al., 2005; Wu et al., 2018) as the approximation (see Sugiyama & Yamada (2012); Wu et al.

Original: (2018)) of our MI measure:
Revised: (2018)) of our MI measure:

Original: [Equation]
Revised: [Equation]

Original: where ⋅ represents y or a i , n is the number of { F ⋅ , ˆX i } pairs,
Revised: where ⋅ represents y or a i , n is the number of { F ⋅ , ˆX i } pairs,

Original: H =
Revised: H =

Original: I − 1 n 11 ⊺ , K F ⋅ ∈ R n × n is the Gram matrix of F ⋅ with K F ⋅ ij = k 1 ( F ⋅ i , F ⋅ j ) , K ˆX i ∈ R n × n is the Gram matrix of ˆX i with
Revised: I − 1 n 11 ⊺ , K F ⋅ ∈ R n × n is the Gram matrix of F ⋅ with K F ⋅ ij = k 1 ( F ⋅ i , F ⋅ j ) , K ˆX i ∈ R n × n is the Gram matrix of ˆX i with

Original: K ˆX ijk = k 2 ( ˆ X ij , ˆ X ik ) .
Revised: K ˆX ijk = k 2 ( ˆ X ij , ˆ X ik ) .

Original: k 1 (⋅ , ⋅) and k 2 (⋅ , ⋅) are predeﬁned kernel functions.
Revised: k 1 (⋅ , ⋅) and k 2 (⋅ , ⋅) are predeﬁned kernel functions.

Original: The most common choice for the kernel is the RBF kernel.
Revised: The most common choice for the kernel is the RBF kernel.

Original: However, if we consider time series data with various time steps, we need to either perform data augmentation or choose another kernel choice.
Revised: However, if we consider time series data with various time steps, we need to either perform data augmentation or choose another kernel choice.

Original: For example, we can adopt the Global Alignment Kernel (Cuturi et al., 2007) which considers the alignment between two varying-length time series when computing the kernel score between them.
Revised: For example, we can adopt the Global Alignment Kernel (Cuturi et al., 2007) which considers the alignment between two varying-length time series when computing the kernel score between them.

Original: To simplify our analysis, we choose to augment data before we calculate the kernel score with the
Revised: To simplify our analysis, we choose to augment data before we calculate the kernel score with the

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 46.9 CMV-HCRF 22.
Revised: 46.9 CMV-HCRF 22.

Original: 45.5 CMV-LDHCRF 24.
Revised: 45.5 CMV-LDHCRF 24.

Original: 47.8 EF-HSSHCRF 24.
Revised: 47.8 EF-HSSHCRF 24.

Original: 49.3 MV-HSSHCRF 24.
Revised: 49.3 MV-HSSHCRF 24.

Original: 46.4 DF 26.
Revised: 46.4 DF 26.

Original: 67.1 EF-
Revised: 67.1 EF-

Original: [Table]
Revised: [Table]

Original: 64.3 EF-SLSTM 29.
Revised: 64.3 EF-SLSTM 29.

Original: 51.4 EF-BLSTM 28.
Revised: 51.4 EF-BLSTM 28.

Original: 58.9 EF-SBLSTM 26.
Revised: 58.9 EF-SBLSTM 26.

Original: 63.3 MV-LSTM 33.
Revised: 63.3 MV-LSTM 33.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: RBF kernel.
Revised: RBF kernel.

Original: More speciﬁcally
Revised: More speciﬁcally

Original: [Table]
Revised: [Table]

Original: we perform averaging over time series data:
Revised: we perform averaging over time series data:

Original: [Equation]
Revised: [Equation]

Original: The bandwidth of the RBF kernel is set as 1.0 throughout the experiments.
Revised: The bandwidth of the RBF kernel is set as 1.0 throughout the experiments.

Original: [Table]
Revised: [Table]

Original: POM dataset for personality traits prediction.
Revised: POM dataset for personality traits prediction.

Original: [Table]
Revised: [Table]

Original: Here, we provide an additional interpretation result for the POM dataset in Table 6.
Revised: Here, we provide an additional interpretation result for the POM dataset in Table 6.

Original: We observe that the language modality is also the most informative while the visual and acoustic modalities are almost equally informative.
Revised: We observe that the language modality is also the most informative while the visual and acoustic modalities are almost equally informative.

Original: This result is in agreement with behavioral studies which have observed that non-verbal behaviors are particularly informative of personality traits (Guimond & Massrieh, 2012; Levine et al., 2009; Mohammadi et al., 2010).
Revised: This result is in agreement with behavioral studies which have observed that non-verbal behaviors are particularly informative of personality traits (Guimond & Massrieh, 2012; Levine et al., 2009; Mohammadi et al., 2010).

Original: For example, the same sentence “this movie was great” can convey signiﬁcantly different messages on speaker conﬁdence depending on whether it was said in a loud and exciting voice, with eye contact, or powerful gesticulation.
Revised: For example, the same sentence “this movie was great” can convey signiﬁcantly different messages on speaker conﬁdence depending on whether it was said in a loud and exciting voice, with eye contact, or powerful gesticulation.

Original: Gradient-Based Interpretation: MFM reconstructs x i as follows:
Revised: Gradient-Based Interpretation: MFM reconstructs x i as follows:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Equation equation 12 also explains how we obtain f y ∼ P ( F y ∣
Revised: Equation equation 12 also explains how we obtain f y ∼ P ( F y ∣

Original: X 1 ∶ M = x 1 ∶ M ) .
Revised: X 1 ∶ M = x 1 ∶ M ) .

Original: The gradient ﬂow through time is deﬁned as:
Revised: The gradient ﬂow through time is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: E E NCODER AND D ECODER D ESIGN FOR M ULTIMODALT IME S ERIES D ATASETS
Revised: E E NCODER AND D ECODER D ESIGN FOR M ULTIMODAL S YNTHETIC I MAGE D ATASET

Original: Fig.
Revised: For experiments on the multimodal synthetic image dataset, we use convolutional+fully-connected layers for the encoder and deconvolutional+fully-connected layers for the decoder (Zeiler et al., 2010). Different convolutional layers are each applied on the input SVHN and MNIST images to learn modality-speciﬁc generative factors. Next, we concatenate the features from two more convolutional layers on SVHN and MNIST to learn the multimodal-discriminative factor. The multimodal discriminative factor is passed through fully-connected layers to predict the label.

Revised: For generation, we concatenate the multimodal discriminative factors and the modality-speciﬁc generative factor together and use a deconvolutional layer to generate digits.

Revised: F E NCODER AND D ECODER D ESIGN FOR M ULTIMODAL T IME S ERIES D ATASETS

Original:  5 illustrates how MFM operates on multimodal time series data.
Revised: Figure 5 illustrates how MFM operates on multimodal time series data.

Original: The encoder Q ( Z y ∣ X 1 ∶ M ) can be parametrized by any model that performs multimodal fusion (Nojavanasghari et al., 2016; Zadeh et al., 2018a).
Revised: The encoder Q ( Z y ∣ X 1 ∶ M ) can be parametrized by any model that performs multimodal fusion (Nojavanasghari et al., 2016; Zadeh et al., 2018a).

Original: We choose the Memory Fusion Network (MFN) (Zadeh et al., 2018a) as our encoder Q ( Z y ∣ X 1 ∶ M ) .
Revised: We choose the Memory Fusion Network (MFN) (Zadeh et al., 2018a) as our encoder Q ( Z y ∣ X 1 ∶ M ) .

Original: We use encoder LSTM networks and decoder LSTM networks (Cho et al., 2014) to parametrize functions Q ( Z a 1 ∶ M ∣ X 1 ∶ M ) and F 1 ∶ M respectively, and FCNNs to parametrize functions 
Revised: We use encoder LSTM networks and decoder LSTM networks (Cho et al., 2014) to parametrize functions Q ( Z a 1 ∶ M ∣ X 1 ∶ M ) and F 1 ∶ M respectively, and FCNNs to parametrize functions G y , G a { 1 ∶ M } and D . encoder LSTM networks and decoder LSTM networks (Cho et al., 2014) to parametrize functions

Original:  G y , G a { 1 ∶ M } and D .
Revised: Q ( Z a 1 ∶ M ∣ X 1 ∶ M ) and F 1 ∶ M respectively, and FCNNs to parametrize functions G y , G a { 1 ∶ M } and D .

Original: F S URROGATE I NFERENCE G RAPHICAL M ODEL
Revised: G S URROGATE I NFERENCE G RAPHICAL M ODEL

Original: We illustrate the surrogate inference for addressing the missing modalities issue in Figure 6.
Revised: We illustrate the surrogate inference for addressing the missing modalities issue in Figure 6.

Original: The surrogate inference model infers the latent codes given the present modalities.
Revised: The surrogate inference model infers the latent codes given the present modalities.

Original: These inferred latent codes can then be used for reconstructing the missing modalities or label prediction in the presence of missing modalities.
Revised: These inferred latent codes can then be used for reconstructing the missing modalities or label prediction in the presence of missing modalities.

Original: G C OMPARISON WITH H SU & G LASS (2018)
Revised: H C OMPARISON WITH H SU & G LASS (2018)

Original: A similar approach for factorizing the latent factors was recently proposed by Hsu & Glass (2018) in work that was performed independently and in parallel.
Revised: A similar approach for factorizing the latent factors was recently proposed by Hsu & Glass (2018) in work that was performed independently and in parallel.

Original: In comparison with MFM, there are several major differences that can be categorized into the (1) prior matching discrepancy, (2) inference network, (3) discriminative objective, (4) multimodal fusion, (5) scale of experiments.
Revised: In comparison with MFM, there are several major differences that can be categorized into the (1) prior matching discrepancy, (2) inference network, (3) discriminative objective, (4) multimodal fusion, (5) scale of experiments.

Original: MFM uses MMD( Q Z ,P Z )
Revised: MFM uses MMD( Q Z ,P Z )

Original: (see Equation 4) as the prior matching discrepancy while Hsu & Glass (2018) use KL( Q Z ∣ X ,P Z ) )
Revised: (see Equation 4) as the prior matching discrepancy while Hsu & Glass (2018) use KL( Q Z ∣ X ,P Z ) )

Original: (see Section 2.2.1 in Hsu & Glass (2018)).
Revised: (see Section 2.2.1 in Hsu & Glass (2018)).

Original: MFM considers multimodal and unimodal inference in a single network (see Figure 1(b)), while Hsu & Glass (2018) considers separate networks (see Figure 1 in Hsu & Glass (2018)).
Revised: MFM considers multimodal and unimodal inference in a single network (see Figure 1(b)), while Hsu & Glass (2018) considers separate networks (see Figure 1 in Hsu & Glass (2018)).

Original: They further propose to match the coherence between these two networks using an additional loss term (see Equation 7 in Hsu & Glass (2018)).
Revised: They further propose to match the coherence between these two networks using an additional loss term (see Equation 7 in Hsu & Glass (2018)).

Original: MFM learns to predict the labels using a generative framework (see Figure 1 (a)), while Hsu & Glass (2018) use an additional hinge loss to separate the latent factors from different labels (see Equation (9) in Hsu & Glass (2018)).
Revised: MFM learns to predict the labels using a generative framework (see Figure 1(a)), while Hsu & Glass (2018) use an additional hinge loss to separate the latent factors from different labels (see Equation 9 in Hsu & Glass (2018)).

Original: MFM is a ﬂexible framework that can be combined with any multimodal fusion encoder (see Section 3.1), while Hsu & Glass (2018) considers a ﬁxed multimodal encoder (similar to early fusion) (see Section 4.1 in Hsu & Glass (2018)).
Revised: MFM is a ﬂexible framework that can be combined with any multimodal fusion encoder (see Section 2.4), while Hsu & Glass (2018) considers a ﬁxed multimodal encoder (similar to early fusion) (see Section 4.1 in Hsu & Glass (2018)).

Original: We evaluate the performance of MFM over a much larger scale of datasets.
Revised: We evaluate the performance of MFM over a much larger scale of datasets.

Original: We perform experiments on six multimodal time-series datasets that take on the form of videos with the language, visual, and acoustic modalities.
Revised: We perform experiments on six multimodal time-series datasets that take on the form of videos with the language, visual, and acoustic modalities.

Original: These datasets span three core research areas of multimodal personality traits recognition, multimodal sentiment analysis, and multimodal emotion recognition.
Revised: These datasets span three core research areas of multimodal personality traits recognition, multimodal sentiment analysis, and multimodal emotion recognition.

Original: On the other hand, Hsu & Glass ( ) evaluates their model on a spoken digit dataset which randomly combines a digit image with a spoken digit (see Section 4 in Hsu & Glass (2018)).
Revised: On the other hand, Hsu & Glass (2018) evaluates their model on a spoken digit dataset which randomly combines a digit image with a spoken digit (see Section 4 in Hsu & Glass (2018)).

Original: MFM further considers experiments to evaluate reconstruction and prediction in the presence of missing modalities (see Section 3.3) which Hsu & Glass (2018) do not.
Revised: MFM further considers experiments to evaluate reconstruction and prediction in the presence of missing modalities (see Section 3.2) which Hsu & Glass (2018) do not.

Original: Lastly, we compares to over 20 baseline models in our experiments (see
Revised: Lastly, we compares to over 20 baseline models in our experiments (see

Original: Section 3.3) and explore the choice of various multimodal encoders in MFM.
Revised: Section 3.2) and explore the choice of various multimodal encoders in MFM.

Original: Hsu & Glass (2018) only compares to the JMVAE baseline model  which resembles the M D model in our ablation study (see Section 4.4 in Hsu & Glass (2018)).
Revised: Hsu & Glass (2018) only compares to the JMVAE baseline model (Suzuki et al., 2016a) which resembles the M D model in our ablation study (see Section 4.4 in Hsu & Glass (2018)).

Original: In Table 7, we provide a comparison on CMU-MOSI to test the disentanglement  performance for the model described in Hsu & Glass (2018). 
Revised: In Table 7, we provide a comparison on the CMU-MOSI, ICT-MMMO, YouTube and MOUD datasets to test the disentanglement and prediction performance for the model described in Hsu & Glass (2018). These experimental results show that across these datasets and metrics, MFM performs better than the model proposed in Hsu & Glass (2018). I C OMPARISON WITH β -VAE

Original: MFM with early fusion encoder and MFN encoder both perform better than the  model  proposed in Hsu & Glass (2018).
Revised: Although β -VAE (Higgins et al., 2017) was designed to handle unimodal data, we provide an extension to multimodal data. To achieve this, we set the choice of prior matching discrepancy as the KL-divergence KL( Q Z ∣ X ,P Z ) and set β large (i.e. β ∈ { 10 , 50 , 100 , 200 } ) to encourage disentanglement of latent variables. We train a β -VAE to model multimodal data using the same factorization as proposed in our model (i.e. modality-speciﬁc generative factors Z a { 1 ∶ M } and a multimodal discriminative factor Z y ). To provide a fair comparison to our discriminative model,

Original:  [Table]
Revised: we [Table] [Table]

Revised: ﬁne tune by training a classiﬁer on top of the multimodal discriminative factor Z y to the label Y .

Revised: We provide experimental results in Table 8 on the CMU-MOSI, ICT-MMMO, YouTube and MOUD datasets.

Revised: MFM consistently outperforms β -VAE across these datasets and metrics.