Learning Robust Joint Representations for Multimodal Sentiment Analysis
Abstract
Multimodal sentiment analysis is a core research area that studies speaker sentiment expressed from the language, visual, and acoustic modalities.
The central challenge in multimodal learning involves inferring joint representations that can process and relate information from these modalities.
However, existing work learns joint representations using multiple modalities as input and may be sensitive to noisy or missing modalities at test time.
With the recent success of sequence to sequence models in machine translation, there is an opportunity to explore new ways of learning joint representations that may not require all input modalities at test time.
In this paper, we propose a method to learn robust joint representations by translating between modalities.
Our method is based on the key insight that translation from a source to a target modality provides a method of learning joint representations using only the source modality as input.
We augment modality translations with a cycle consistency loss to ensure that our joint representations retain maximal information from all modalities.
Once our translation model is trained with paired multimodal data, we only need data from the source modality at test-time for prediction.
This ensures that our model remains robust from perturbations or missing target modalities.
We train our model with a coupled translation-prediction objective and it achieves new state-of-the-art results on multimodal sentiment analysis datasets:
CMU-MOSI, ICT-MMMO, and YouTube.
Additional experiments show that our model learns increasingly discriminative joint representations with more input modalities while maintaining robustness to perturbations of all other modalities.
Introduction
Sentiment analysis, which involves identifying a speaker’s opinion, is a core research problem in machine learning and natural language processing.
However, language-based sentiment analysis through words, phrases and their compositionality was found to be insufﬁcient for inferring affective content from spoken opinions [34], which contain rich nonverbal behaviors in addition to verbal text.
As a result, there has been a recent push towards using machine learning methods to learn joint representations from additional behavioral cues present in the visual and acoustic modalities.
This research ﬁeld has become known as multimodal sentiment analysis and extends the conventional textbased deﬁnition of sentiment analysis to a multimodal setup.
For example, [22] explore the additional acoustic modality while [62] use the language, visual, and acoustic modalities present in monologue videos to predict sentiment.
The abundance of multimodal data has led to the creation of multimodal datasets, such as CMU-MOSI [67] and ICT-MMMO [62], as well as deep multimodal models that are highly effective at learning discriminative joint multimodal representations [30, 58, 8].
Existing work learns joint representations using multiple modalities as input with neural networks [29], graphical models [34] or geometric classiﬁers [67].
However, this results in joint representations that are sensitive to noisy or missing modalities at test time.
[Figure]
To address this problem, we draw inspirations from the recent success of sequence to sequence models for unsupervised representation learning [56].
We propose the Multimodal Cyclic
Translation Network model (MCTN) to learn robust joint multimodal representations by translating between modalities.
Figure 1 illustrates these translations between the language, visual and acoustic modalities.
Our method is based on the key insight that translation from a source modality S to a target modality T results in an intermediate representation that captures joint information between modalities S and T .
MCTN extends this insight using a cyclic translation loss involving both forward translations from source to target, and backward translations from the predicted target back to the source modality.
Together, we call these multimodal cyclic translations to ensure that the learned joint representations capture maximal information from both modalities.
We also propose a hierarchical
MCTN to learn joint representations between a source modality and multiple target modalities.
MCTN is trainable end-to-end with a coupled translation-prediction loss which consists of (1) the cyclic translation loss, and (2) a prediction loss to ensure that the learned joint representations are task-speciﬁc.
Another advantage of MCTN is that once trained with paired multimodal data ( S,T ) , we only need data from the source modality S at test time to infer the joint representation and sentiment prediction.
As a result, MCTN is completely robust to test-time perturbations on target modality T and missing modalities.
Even though translation and generation of videos, audios, and text are difﬁcult [28], our experiments show that the learned joint representations can help for discriminative tasks: MCTN achieves new state-of-the-art results on multimodal sentiment analysis using the CMU-MOSI, ICT-MMMO, and
YouTube public datasets.
Additional experiments show that MCTN learns increasingly discriminative joint representations with more input modalities while maintaining robustness to all target modalities.
Related Work
Early work on sentiment analysis focused primarily on written text [40, 51].
Recently, multimodal sentiment analysis has gained more research interest [5] since learning joint representation of multiple modalities is a challenging task.
Earlier work simply concatenated the input features [37, 26].
Recently, several neural models have also been proposed to learn joint representations [8, 65, 9].
For example, [29] presented a multistage approach to learn hierarchical multimodal representations.
The Tensor
Fusion Network [64] and the Low-rank Multimodal Fusion model [31] presented methods based on
Cartesian-products to model unimodal, bimodal and trimodal interactions.
In addition to purely supervised approaches, generative methods based on Generative Adversarial
Networks (GANs) [17] have been used to learn joint distributions between two or more modalities [14, 27].
Another method involves using conditional generative models [33, 23, 39] to translate one modality to another.
Generative-discriminative objectives have been used to learn either joint [44, 24] or factorized [58] representations.
Our work takes into account the sequential dependency of modality translations and also explores the effect of a cyclic translation loss on modality translations.
Finally, there has been some progress on accounting for noisy or missing modalities at test time. [55] proposed using Deep Boltzmann Machines to model the joint distribution over multimodal data.
Sampling from the conditional distributions allow for inference of missing modalities.
[52] trained Restricted Boltzmann Machines to minimize the variation of information between modalityspeciﬁc latent variables.
Models based on autoencoders [57], adversarial learning [7], or multiple kernel learning [32] have also been proposed for these tasks.
It was also found that training with missing or noisy modalities can improve the robustness of joint representations [37].
These methods approximately infer the missing modalities before prediction, leading to possible error compounding.
On the other hand, MCTN remains fully robust to other modalities during testing.
Proposed Approach
In this section, we describe our approach for learning joint multimodal representations through modality translations.
Notation: A multimodal dataset consists of data X =
( X l , X v , X a ) from the language, visual, and acoustic modalities respectively.
It is indexed by n segments X =
( X 1 , X 2 ,..., X n ) where X i = ( X li , X vi , X ai ) , 1 ≤ i ≤ n .
The labels for these n segments are denoted as y =
( y 1 ,y 2 ,...,y n ) , y i ∈ R .
Many datasets are easily synchronized by aligning the input based on the boundaries of each word and zero-padding each segment to obtain time-series data of the same length [29].
The i th segment is given by X li =
( w i ( 1 ) , w i ( 2 ) ,..., w i ( L ) ) where w i ( (cid:96) ) stands for the (cid:96) th word and L is the length of each segment.
To accompany the language features, we also have a sequence of visual features
X vi
=
( v i ( 1 ) , v i ( 2 ) ,..., v i ( L ) ) and acoustic features
X ai =
( a i ( 1 ) , a i ( 2 ) ,..., a i ( L ) ) .
3.1 Learning Joint Representations
We deﬁne learning a joint representation between two modalities X S and X T as learning a parametrized function f θ that returns an embedding E ST
= f θ ( X S , X T ) .
From there, another function g w is learned that predicts the label given this joint representation: ˆ y =
g w (E ST ) .
Most work follows this framework during both training and testing [29, 31, 58, 65].
During training, the parameters θ and w are learned by empirical risk minimization over paired multimodal data and labels in the training set ( X Str , X Ttr , y tr ) :
[Equation]
E [ (cid:96) y
( ˆ y tr , y tr )] .
(2) for a suitable choice of loss function (cid:96) y over the labels ( tr denotes training set).
During testing, paired multimodal data in the test set ( X Ste , X Tte ) are used to infer the label ( te denotes test set):
E ST = f θ ∗ ( X Ste , X Tte ) , ˆ y te =
g w ∗ (E ST ) .
3.2 Multimodal Cyclic Translation Network
[Figure]
obtained via cyclic translation X S
Next joint is used for sentiment prediction trained coupled prediction objective time is required Multimodal Cyclic
Translation
Network (MCTN) is a neural model that learns robust joint representations by modality translations.
Figure 2 shows a detailed description of MCTN for two modalities.
Our method is based on the key insight that translation from a source modality X S to a target modality X T results in a representation that captures joint information between modalities X S and X T , but using only the source modality X S as input.
To ensure that our model learns joint representations that retain maximal information from all modalities, we use a cycle consistency loss [68] during modality translation.
This method can also be seen as a variant of back-translation which has been recently applied to style transfer [47, 68] and unsupervised machine translation [25].
We use back-translation in a multimodal setup where we encourage our translation model to learn informative joint representations but with only the source modality as input.
The cycle consistency loss for modality translation starts by decomposing function f θ into two parts: an encoder f θ e and a decoder f θ d .
The encoder takes in X S as input and returns a joint embedding E S → T :
[Equation]
which the decoder then transforms into target modality X T :
[Equation]
following which the decoded modality T is translated back into modality S :
[Equation]
The joint representation is learned by using a Sequence to Sequence (Seq2Seq) model with attention [4] that translates source modality X S to a target modality X T .
While Seq2Seq models have been predominantly used for machine translation, we extend its usage to the realm of multimodal machine learning.
The Seq2Seq model consists of an encoder network and a decoder network, each parametrized as Recurrent Neural Networks (RNNs).
The encoder maps the source modality X S into an embedded representation E S → T .
Using a recurrent network, the hidden state output of each time step is based on the previous hidden state along with the input sequence
[Equation]
The encoder’s output is the concatenation of all hidden states of the encoding RNN, E S
→ T =[-
-]
[ h 1 , h 2 ,..., h L ] , where L is the length of the source modality X S .
The decoder maps the representation E S
→ T into the target modality X T .
This is performed by decoding each token X Tt at a time based on E S → T and all previous decoded tokens, which is formulated as
[Equation]
MCTN accepts variable-length inputs of X S and X T , and is trained to maximize the translational condition probability p ( X T ∣ X S ) .
The best translation sequence is then given by
[Equation]
While there are other search algorithms such as random sampling and greedy search that can be used for decoding each token [36], we use the traditional beam search approach [56].
To obtain the joint representation for prediction, we found that simply using one of the translated representations was sufﬁcient for good performance ( ⇆ denotes multimodal cyclic translations):
E S ⇆ T =
E S → T .
E S ⇆ T is used for prediction via a recurrent neural network, ˆ y =
g w (E S ⇆ T ) .
3.3 Coupled Translation-Prediction Objective
Training is performed with paired multimodal data and labels in the training set ( X Str , X Ttr , y tr ) .
We evaluate the forward translation loss
[Equation]
10) and the cycle consistency loss
[Equation]
) for suitable choices of loss functions (cid:96) X T and (cid:96) X S .
We use the Mean Squared Error (MSE) between the ground-truth and translated modalities.
Finally, the prediction loss L p is
[Equation]
12) for a loss function (cid:96) y over the labels.
Equations (10), (11), and (12) are evaluated using the training set and MCTN can be trained end-toend with a coupled translation-prediction objective function L = λ t L t + λ c L c + L p , where L p is the prediction loss, L c is the cyclic translation loss, and λ t , λ t are weighting hyperparameters.
MCTN parameters are learned by minimizing this objective function
[Equation]
Parallel multimodal data is not required at test time.
Inference is performed using only the source modality X Ste :
[Equation]
This is possible because the encoder f θ ∗ e has been trained to translate the source modality X S into a joint representation E S ⇆ T that captures information from both source and target modalities.
Intuitively, the translation model learns to predict target modalities through an informative joint representation.
3.4 Hierarchical MCTN for Three Modalities
[Figure]
Next, representation of all three modalities, E ( S ⇆ T 1 )→ T 2 , is used for sentiment prediction.
The model is trained end-to-end with a coupled translation-prediction objective.
At test time, only the source modality X S is required.
We extend the MCTN hierarchically to learn joint representations from more than two modalities.
Figure 3 shows the case for three modalities.
The hierarchical MCTN starts with a source modality X S and two target modalities X T 1 and
X T 2 .
To learn joint representations, two levels of translations are performed.
The ﬁrst level learns a joint representation from X S and X T 1 using multimodal cyclic translations as deﬁned previously.
At the second level, a joint representation is learned hierarchically by translating the ﬁrst representation E S
→ T 1 into X T 2 .
For more than three modalities, the modality translation process can be repeated hierarchically.
Two Seq2Seq models are used in the hierarchical MCTN for three modalities.
Denote these as encoder-decoder pairs ( f 1 θ e ,f 1 θ d ) and ( f 2 θ e ,f 2 θ d ) .
A multimodal cyclic translation is ﬁrst performed between source modality X S and the ﬁrst target modality X T 1 .
The consists of the forward translation:
[Equation]
15) following which the decoded modality X T 1 is translated back into modality X S :
[Equation]
A second hierarchical Seq2Seq model is applied on the time-distributed outputs of the encoder f 1 θ e :
[Equation]
[Equation]
The joint representation between modalities X S , X T 1 and X T 2 is now E ( S ⇆ T 1 )
→ T 2 and is used for sentiment prediction via a recurrent neural network.
Training the hierarchical MCTN involves computing a cycle consistent loss for modality T 1 , given by L t 1 and L c 1 .
We do not use a cyclic translation loss when translating from E S ⇆ T 1 to X T 2 since the ground truth E S ⇆ T 1 is unknown, and so only the translation loss L t 2 is computed.
The ﬁnal objective for hierarchical MCTN is given by L = λ t 1 L t 1 + λ c 1 L c 1 + λ t 2 L t 2 +L p .
We emphasize that for MCTN with three modalities, only a single source modality X S is required at test time.
Therefore, MCTN has a signiﬁcant advantage over existing models since it is robust to noise or missing target modalities.
Experimental Setup
In this section, we describe our experimental methodology to evaluate the joint representations learned by MCTN.
Datasets: We use the CMU-MOSI dataset which contains 2199 video segments each with a sentiment label in the range from − 3 to + 3 .
− 3 indicates strongly negative sentiment, + 3 indicates strongly positive sentiment, and 0 indicates neutral sentiment.
CMU-MOSI is subject to much research [58, 65, 8] and the current state of the art is achieved by [29] with a binary accuracy of 78.4%.
We additionally perform experiments on the ICT-MMMO [62] and YouTube [34] datasets.
These datasets consist of online review videos annotated for sentiment.
Features: Following previous work [29], GloVe word embeddings [42], Facet [20] and COVAREP [12] features are extracted for the language, visual and acoustic modalities respectively.
Forced alignment is performed using P2FA [63] to obtain word utterance times and we align the visual and acoustic features by computing their average over each word utterance interval.
[Figure] [Figure]
[Table]
For parameter optimization on the
CMU-MOSI dataset, we set the choice of pre diction loss function the Mean report and
In addition, we also perform sentiment classiﬁcation on the CMU-MOSI dataset and report binary accuracy (Acc) and F1 score (F1).
On the ICTMMMO and YouTube datasets, we set the choice of prediction loss function as categorical crossentropy and report classiﬁcation accuracy (Acc) and F1 score.
For all metrics, higher values indicate stronger performance, except MAE where lower values indicate stronger performance.
Baselines: We compare to the following multimodal models: RMFN [29] uses a multistage approach to learn hierarchical representations.
It is the current state-of-the-art on CMU-MOSI.
LMF [31] approximates the expensive multimodal tensor products in TFN [64] with efﬁcient low-rank factors.
MFN [65] synchronizes sequences using a multimodal gated memory.
MARN [66] uses multiple attention coefﬁcients and hybrid LSTM memory components.
GMELSTM(A) [8] learns binary gating mechanisms to remove noisy modalities that are contradictory or redundant.
EF-LSTM concatenates multimodal inputs and uses a single LSTM [19].
For details on all baselines, please refer to the supplementary.
Results and Discussion
This section discusses several research questions and presents our experimental results.
[Table]
5a
[Table]
Comparison with Existing Work: Q1:
How does MCTN compare with existing state-ofthe-art approaching for multimodal sentiment analysis?
We compare MCTN with the existing state-of-the-art models 3 .
From Table 4,
MCTN achieves new start-of-the-art results on binary classiﬁcation accuracy, F1 score, and MAE on CMU-MOSI.
State-of-the-art results are also achieved on the ICT-MMMO and YouTube datasets.
These results are even
Enc
Dec
Enc next input
Enc
Dec (b)
Enc
Dec
Enc
Dec
Encc
Enc
Dec (c) (d) (a) (f) (g) (e) [ ] [ ]
Enc
Dec (h)
Dec Dec
Step 1 Step
Seq2Seq 1 Seq2Seqconcat concat
Bimodal Variations
Trimodal Variations (a) (e) (d) (g) (b) (f) (c) (f) more impressive considering that MCTN only uses the language modality during testing, while other baseline models use all three modalities.
Adding More Modalities: Q2:
What is the impact of increasing the number of modalities during training for MCTN with cyclic translations?
We run experiments with MCTN using combinations of two or three modalities with cyclic translations.
From Table 2, we observe that adding more modalities improves performance, indicating that the joint representations learned are leveraging the information from more input modalities.
This also implies that cyclic translations are a viable method to learn joint representations from multiple modalities since little information is lost from adding more modality translations.
Another observation is that using language as the source modality always leads to the best performance, which is intuitive since the language modality contains the most information towards sentiment [64].
In addition, we visually inspect the joint representations learnt from MCTN as we add more modalities during training.
The joint representations for each video segment in CMU-MOSI are extracted from the best performing model for each number of modalities and then projected into two dimensions via the t-SNE algorithm [59].
Each point is colored red or blue depending on whether the video segment is annotated for positive or negative sentiment.
From Figure 4, we observe that the joint representations become increasingly separable as the more modalities are added when the MCTN is trained.
This is consistent with increasing discriminative performance as seen in Table 2.
Ablation Studies: We devise the following ablation models to test each design decision in MCTN: the use of cyclic translations, shared Seq2Seq models, modality ordering, and hierarchical structure.
For bimodal MCTN, we design the following ablation models shown in the left half of Figure 5: (a) is our proposed MCTN between X S and X T , (b) is the MCTN based on translation from X S to X T without cyclic translations, (c) does not use cyclic translations but rather performs two independent translations between X S and X T , (d) is the pair of MCTN models with different inputs (of the same modality pair) and then using the concatenation of the joint representations E S → T and E T → S as the ﬁnal embeddings.
For trimodal MCTN, we design the following ablation models shown in the right half of Figure 5: (e) is the proposed hierarchical MCTN between X S , X T 1 and X T 2 , (f) is the MCTN based on translation from X S to X T 1 without cyclic translations, (g) is extended from (d) which does not use cyclic translations but rather performs two independent translations between X S and
X T 1 , and ﬁnally, (h) does not perform a ﬁrst level of cyclic translation but directly translates the concatenated modality pair [ X S , X T 1 ] into X T 2 .
Q3:
What is the impact of cyclic translations in MCTN?
The bimodal and trimodal results are shown in Table 3.
Only the model in Figure 5(a) employs cyclic translations and they outperform the other baselines.
We make a similar observation for hierarchical MCTN: Figure 5(e) with cyclic translations outperforms the trimodal baselines (f), (g) and (h).
The gap for the trimodal case is especially large.
This implies that using cyclic translations is crucial in learning joint representations.
Our intuition is that using cyclic translations: (1) encourages the model to enforce symmetry between the joint representations from source and target modalities, and (2) ensures that the joint representation retains maximal information from all modalities
[Table]
Q4: What is the effect of using two
Seq2Seq models instead of one shared
Seq2Seq model for cyclic translations?
We compare Figure 5(c), which uses one Seq2Seq model for translations with Figure 5(d), which uses two separate Seq2Seq models: one for forward and one for backward translation.
We observe from Table 3 that (c) > (d), so using one model with shared parameters is better.
This is also true for hierarchical MCTN: (f) > (g).
We hypothesize that this is because training two Seq2Seq models requires more data and is prone to overﬁtting.
Q5:
What is the impact when varying source and target modalities for cyclic translations?
As shown in Tables 2 and 3, we observe that language contributes most towards the joint representations.
For bimodal cases, combining language with visual is generally better than combining language with audio.
For hierarchical MCTN, presenting language as the source modality leads to the best performance, and a ﬁrst level of cyclic translations between language and visual is better than between language and acoustic.
On the other hand, only translating between visual and acoustic modalities dramatically decreases performance.
Further adding language as a target modality for hierarchical MCTN will not help much as well.
Overall, language is still the most important modality for multimodal sentiment analysis and must be used as the source modality during translations.
Q6:
What is the impact of using two levels of hierarchical translations instead of one level for three modalities?
Our hierarchical MCTN is shown in Figure 5(e).
In Figure 5(h), we concatenate two modalities as input and use only one phase of translation.
From Table 3, we observe that (e) >
(h): both levels of modality translations are important in the hierarchical MCTN.
We believe that representation learning is easier when the task is broken down recursively: using two translations each between a single pair of modalities, rather than a single translation between all modalities.
Conclusion
To conclude, this paper investigated learning joint representations via cyclic modality translations from source to target modalities.
During testing, we only need the source modality for prediction which ensures that our model remains robust from noisy or missing target modalities.
We demonstrate that cyclic translations and seq2seq models are especially useful for learning joint multimodal representations.
In addition to achieving state-of-the-art results on three datasets, our model learns increasingly discriminative representations with more input modalities while maintaining robustness to all target modalities.
Our approach presents several exciting areas for future work, such as: 1) combining our approach with the transformer architecture [60] for modality translations, 2) exploring pretrained deep language models [13, 43] for translations, as well as 3) extending our translation model to work other multimodal tasks involving language and raw speech signals (prosody), videos with multiple speakers (diarization), and combinations of static and temporal data (i.e. image captioning).
A Multimodal Features
Here we present extra details on feature extraction for the language, visual and acoustic modalities.
Language: We used 300 dimensional Glove word embeddings trained on 840 billion tokens from the common crawl dataset [42].
These word embeddings were used to embed a sequence of individual words from video segment transcripts into a sequence of word vectors that represent spoken text.
Visual: The library Facet [20] is used to extract a set of visual features including facial action units, facial landmarks, head pose, gaze tracking and HOG features [69].
These visual features are extracted from the full video segment at 30Hz to form a sequence of facial gesture measures throughout time.
Acoustic: The software COVAREP [12] is used to extract acoustic features including 12 Mel-frequency cepstral coefﬁcients, pitch tracking and voiced/unvoiced segmenting features [15], glottal source parameters [10, 16, 1, 3, 2], peak slope parameters and maxima dispersion quotients [21].
These visual features are extracted from the full audio clip of each segment at 100Hz to form a sequence that represent variations in tone of voice over an audio segment.
B Multimodal Alignment
We perform forced alignment using P2FA [63] to obtain the exact utterance time-stamp of each word.
This allows us to align the three modalities together.
Since words are considered the basic units of language we use the interval duration of each word utterance as one time-step.
We acquire the aligned video and audio features by computing the expectation of their modality feature values over the word utterance time interval [29].
B.1 Baseline Models
We also implement the Stacked, ( EF-SLSTM ) [18], Bidirectional ( EF-BLSTM )
[50], and Stacked Bidirectional ( EF-SBLSTM ) LSTMs, as well as the following baselines: BC-LSTM [46], EF-HCRF [48], EF/MV- LDHCRF [35], MV-HCRF [53], EF/MV-HSSHCRF [54], MV-LSTM [49], DF [38], SAL-CNN [61], C-MKL [45], THMM [34], SVM [11, 41] and RF [6].
C Full Results
We present the full results across all baseline models in Table 3 and Table 4.
MCTN using all modalities achieves new start-of-the-art results on binary classiﬁcation accuracy, F1 score, and MAE on the CMU-MOSI dataset for multimodal sentiment analysis.
State-of-the-art results are also achieved on the ICT-MMMO and YouTube datasets (Table 4).
These results are even more impressive considering that MCTN only uses the language modality during testing, while other baseline models use all three modalities.
[Table] [Table]