Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
Abstract
Multimodal sentiment analysis is a core research area that studies speaker sentimentexpressed from the language, visual, and acoustic modalities.
The central challengein multimodal learning involves inferring joint representations that can processand relate information from these modalities.
However, existing work learns jointrepresentations using multiple modalities as input and may be sensitive to noisy ormissing modalities at test time.
With the recent success of sequence to sequencemodels in machine translation, there is an opportunity to explore new ways oflearning joint representations that may not require all input modalities at test time.
Inthis paper, we propose a method to learn robust joint representations by translatingbetween modalities.
Our method is based on the key insight that translation from asource to a target modality provides a method of learning joint representations usingonly the source modality as input.
We augment modality translations with a cycleconsistency loss to ensure that our joint representations retain maximal informationfrom all modalities.
Once our translation model is trained with paired multimodaldata, we only need data from the source modality at test-time for prediction.
This ensures that our model remains robust from perturbations or missing targetmodalities.
We train our model with a coupled translation-prediction objective andit achieves new state-of-the-art results on multimodal sentiment analysis datasets:
CMU-MOSI, ICT-MMMO, and YouTube.
Additional experiments show that ourmodel learns increasingly discriminative joint representations with more inputmodalities while maintaining robustness to perturbations of all other modalities.
Introduction
Sentiment analysis, which involves identifying a speaker’s opinion, is a core research problem inmachine learning and natural language processing.
However, language-based sentiment analysisthrough words, phrases and their compositionality was found to be insufﬁcient for inferring affectivecontent from spoken opinions [34], which contain rich nonverbal behaviors in addition to verbaltext.
As a result, there has been a recent push towards using machine learning methods to learn jointrepresentations from additional behavioral cues present in the visual and acoustic modalities.
Thisresearch ﬁeld has become known as multimodal sentiment analysis and extends the conventional textbased deﬁnition of sentiment analysis to a multimodal setup.
For example, [22] explore the additionalacoustic modality while [62] use the language, visual, and acoustic modalities present in monologuevideos to predict sentiment.
The abundance of multimodal data has led to the creation of multimodaldatasets, such as CMU-MOSI [67] and ICT-MMMO [62], as well as deep multimodal models that arehighly effective at learning discriminative joint multimodal representations [30, 58, 8].
Existing worklearns joint representations using multiple modalities as input with neural networks [29], graphicalmodels [34] or geometric classiﬁers [67].
However, this results in joint representations that aresensitive to noisy or missing modalities at test time.
[Figure]
To address this problem, we draw inspirationsfrom the recent success of sequence to sequencemodels for unsupervised representation learning [56].
We propose the Multimodal Cyclic
Translation Network model (MCTN) to learnrobust joint multimodal representations by translating between modalities.
Figure 1 illustratesthese translations between the language, visualand acoustic modalities.
Our method is basedon the key insight that translation from a sourcemodality S to a target modality T results in anintermediate representation that captures joint information between modalities S and T .
MCTNextends this insight using a cyclic translationloss involving both forward translations fromsource to target, and backward translations fromthe predicted target back to the source modality.
Together, we call these multimodal cyclictranslations to ensure that the learned joint representations capture maximal information fromboth modalities.
We also propose a hierarchical
MCTN to learn joint representations betweena source modality and multiple target modalities.
MCTN is trainable end-to-end with a coupledtranslation-prediction loss which consists of (1) the cyclic translation loss, and (2) a prediction loss toensure that the learned joint representations are task-speciﬁc.
Another advantage of MCTN is thatonce trained with paired multimodal data ( S,T ) , we only need data from the source modality S attest time to infer the joint representation and sentiment prediction.
As a result, MCTN is completelyrobust to test-time perturbations on target modality T and missing modalities.
Even though translation and generation of videos, audios, and text are difﬁcult [28], our experimentsshow that the learned joint representations can help for discriminative tasks: MCTN achieves newstate-of-the-art results on multimodal sentiment analysis using the CMU-MOSI, ICT-MMMO, and
YouTube public datasets.
Additional experiments show that MCTN learns increasingly discriminativejoint representations with more input modalities while maintaining robustness to all target modalities.
Related Work
Early work on sentiment analysis focused primarily on written text [40, 51].
Recently, multimodalsentiment analysis has gained more research interest [5] since learning joint representation of multiplemodalities is a challenging task.
Earlier work simply concatenated the input features [37, 26].
Recently,several neural models have also been proposed to learn joint representations [8, 65, 9].
For example,[29] presented a multistage approach to learn hierarchical multimodal representations.
The Tensor
Fusion Network [64] and the Low-rank Multimodal Fusion model [31] presented methods based on
Cartesian-products to model unimodal, bimodal and trimodal interactions.
In addition to purely supervised approaches, generative methods based on Generative Adversarial
Networks (GANs) [17] have been used to learn joint distributions between two or more modalities [14,27].
Another method involves using conditional generative models [33, 23, 39] to translate onemodality to another.
Generative-discriminative objectives have been used to learn either joint [44, 24]or factorized [58] representations.
Our work takes into account the sequential dependency of modalitytranslations and also explores the effect of a cyclic translation loss on modality translations.
Finally, there has been some progress on accounting for noisy or missing modalities at test time.[55] proposed using Deep Boltzmann Machines to model the joint distribution over multimodaldata.
Sampling from the conditional distributions allow for inference of missing modalities.
[52]trained Restricted Boltzmann Machines to minimize the variation of information between modalityspeciﬁc latent variables.
Models based on autoencoders [57], adversarial learning [7], or multiplekernel learning [32] have also been proposed for these tasks.
It was also found that training withmissing or noisy modalities can improve the robustness of joint representations [37].
These methodsapproximately infer the missing modalities before prediction, leading to possible error compounding.
On the other hand, MCTN remains fully robust to other modalities during testing.
Proposed Approach
In this section, we describe our approach for learning joint multimodal representations throughmodality translations.
Notation: A multimodal dataset consists of data X =
( X l , X v , X a ) from the language, visual, andacoustic modalities respectively.
It is indexed by n segments X =
( X 1 , X 2 ,..., X n ) where X i =( X li , X vi , X ai ) , 1 ≤ i ≤ n .
The labels for these n segments are denoted as y =
( y 1 ,y 2 ,...,y n ) , y i ∈ R .
Many datasets are easily synchronized by aligning the input based on the boundaries of each wordand zero-padding each segment to obtain time-series data of the same length [29].
The i th segmentis given by X li =
( w i ( 1 ) , w i ( 2 ) ,..., w i ( L ) ) where w i ( (cid:96) ) stands for the (cid:96) th word and L is the lengthof each segment.
To accompany the language features, we also have a sequence of visual features
X vi
=
( v i ( 1 ) , v i ( 2 ) ,..., v i ( L ) ) and acoustic features
X ai =
( a i ( 1 ) , a i ( 2 ) ,..., a i ( L ) ) .
3.1 Learning Joint Representations
We deﬁne learning a joint representation between two modalities X S and X T as learning aparametrized function f θ that returns an embedding E ST
= f θ ( X S , X T ) .
From there, anotherfunction g w is learned that predicts the label given this joint representation: ˆ y =
g w (E ST ) .
Most work follows this framework during both training and testing [29, 31, 58, 65].
During training,the parameters θ and w are learned by empirical risk minimization over paired multimodal data andlabels in the training set ( X Str , X Ttr , y tr ) :
[Equation]
(1) θ ∗ ,w ∗ =
arg min θ,w
E [ (cid:96) y
( ˆ y tr , y tr )] .
(2) for a suitable choice of loss function (cid:96) y over the labels ( tr denotes training set).
During testing, pairedmultimodal data in the test set ( X Ste , X Tte ) are used to infer the label ( te denotes test set):
E ST = f θ ∗ ( X Ste , X Tte ) , ˆ y te =
g w ∗ (E ST ) .
3.2 Multimodal Cyclic Translation Network
[Figure]
ForwardTranslation ForwardTranslation
[Figure] [Figure]
Network(MCTN) is a neural model that learns robustjoint representations by modality translations.
Figure 2 shows a detailed description of MCTNfor two modalities.
Our method is based onthe key insight that translation from a sourcemodality X S to a target modality X T results ina representation that captures joint informationbetween modalities X S and X T , but using onlythe source modality X S as input.
To ensure that our model learns joint representations that retain maximal information from allmodalities, we use a cycle consistency loss [68]during modality translation.
This method canalso be seen as a variant of back-translationwhich has been recently applied to style transfer [47, 68] and unsupervised machine translation [25].
We use back-translation in a multimodal setup where we encourage our translation modelto learn informative joint representations but with only the source modality as input.
The cycleconsistency loss for modality translation starts by decomposing function f θ into two parts: an encoderf θ e and a decoder f θ d .
The encoder takes in X S as input and returns a joint embedding E S → T :
[Equation]
which the decoder then transforms into target modality X T :
[Equation] [Equation]
following which the decoded modality T is translated back into modality S :
[Equation]
The joint representation is learned by using a Sequence to Sequence (Seq2Seq) model with attention [4] that translates source modality X S to a target modality X T .
While Seq2Seq models havebeen predominantly used for machine translation, we extend its usage to the realm of multimodalmachine learning.
The Seq2Seq model consists of an encoder network and a decoder network, eachparametrized as Recurrent Neural Networks (RNNs).
The encoder maps the source modality X S intoan embedded representation E S → T .
Using a recurrent network, the hidden state output of each timestep is based on the previous hidden state along with the input sequence
[Equation] [Equation]
The encoder’s output is the concatenation of all hidden states of the encoding RNN, E S
→ T =
[ h 1 , h 2 ,..., h L ] , where L is the length of the source modality X S .
The decoder maps the representation E S
→ T into the target modality X T .
This is performed bydecoding each token X Tt at a time based on E S → T and all previous decoded tokens, which isformulated as
[Equation]
[Equation]
MCTN accepts variable-length inputs of X S and X T , and is trained to maximize the translationalcondition probability p ( X T ∣ X S ) .
The best translation sequence is then given by
[Equation]
While there are other search algorithms such as random sampling and greedy search that can be usedfor decoding each token [36], we use the traditional beam search approach [56].
To obtain the joint representation for prediction, we found that simply using one of the translatedrepresentations was sufﬁcient for good performance ( ⇆ denotes multimodal cyclic translations):
E S ⇆ T =
E S → T .
E S ⇆ T is used for prediction via a recurrent neural network, ˆ y =
g w (E S ⇆ T ) .
3.3 Coupled Translation-Prediction Objective
Training is performed with paired multimodal data and labels in the training set ( X Str , X Ttr , y tr ) .
Weevaluate the forward translation loss
[Equation]
and the cycle consistency loss
[Equation]
for suitable choices of loss functions (cid:96) X T and (cid:96) X S .
We use the Mean Squared Error (MSE) betweenthe ground-truth and translated modalities.
Finally, the prediction loss L p is
[Equation]
12) for a loss function (cid:96) y over the labels.
Equations (10), (11), and (12) are evaluated using the training set and MCTN can be trained end-toend with a coupled translation-prediction objective function L = λ t L t + λ c L c + L p , where L p is theprediction loss, L c is the cyclic translation loss, and λ t , λ t are weighting hyperparameters.
MCTNparameters are learned by minimizing this objective function
[Equation] [Equation]
Parallel multimodal data is not required at test time.
Inference is performed using only the sourcemodality X S te :
[Equation]
This is possible because the encoder f θ ∗ e has been trained to translate the source modality X S into ajoint representation E S ⇆ T that captures information from both source and target modalities.
Intuitively,the translation model learns to predict target modalities through an informative joint representation.
[Figure]
Next, the joint representation of all three modalities, E ( S ⇆ T 1 )→ T 2 , is used for sentiment prediction.
The model is trained end-to-end with a coupled translation-prediction objective.
At test time, only the source modality X S is required.
We extend the MCTN hierarchically to learnjoint representations from more than two modalities.
Figure 3 shows the case for three modalities.
The hierarchical MCTN starts with a sourcemodality X S and two target modalities X T 1 and
X T 2 .
To learn joint representations, two levels oftranslations are performed.
The ﬁrst level learns joint representation from X S and X T 1 usingmultimodal cyclic translations as deﬁned previously.
At the second level, a joint representationis learned hierarchically by translating the ﬁrstrepresentation
E S
→ T 1 into X T 2 .
For more than three modalities, the modality translation pro- can be repeated hierarchically
Two Seq2Seq models are used in the hierarchical MCTN for three modalities.
Denote these asencoder-decoder pairs ( f 1 θ e ,f 1 θ d ) and ( f 2 θ e ,f 2 θ d ) .
A multimodal cyclic translation is ﬁrst performed between source modality X S and theﬁrst target modality X T 1 .
The consists of theforward translation:
[Equation] [Equation]
following which the decoded modality X T 1 is translated back into modality X S :
[Equation]
A second hierarchical Seq2Seq model is applied on the time-distributed outputs of the encoder f 1 θ e :
[Equation]
[Equation]
The joint representation between modalities X S , X T 1 and X T 2 is now E ( S ⇆ T 1 )
→ T 2 and is used forsentiment prediction via a recurrent neural network.
Training the hierarchical MCTN involves computing a cycle consistent loss for modality T 1 , given by 199 L t 1 and L c 1 .
We do not use a cyclic translation loss when translating from E S ⇆ T 1 to X T 2 since theground truth E S ⇆ T 1 is unknown, and so only the translation loss L t 2 is computed.
The ﬁnal objectivefor hierarchical MCTN is given by L = λ t 1 L t 1 + λ c 1 L c 1 + λ t 2 L t 2 +L p .
We emphasize that for MCTNwith three modalities, only a single source modality X S is required at test time.
Therefore, MCTN hasa signiﬁcant advantage over existing models since it is robust to noise or missing target modalities.
Experimental Setup
In this section, we describe our experimental methodology to evaluate the joint representationslearned by MCTN.
Datasets: We use the CMU-MOSI dataset which contains 2199 video segments each with a sentimentlabel in the range from − 3 to + 3 .
− 3 indicates strongly negative sentiment, + 3 indicates stronglypositive sentiment, and 0 indicates neutral sentiment.
CMU-MOSI is subject to much research [58, 65,8] and the current state of the art is achieved by [29] with a binary accuracy of 78.4%.
We additionallyperform experiments on the ICT-MMMO [62] and YouTube [34] datasets.
These datasets consist ofonline review videos annotated for sentiment.
Features: Following previous work [29], GloVe word embeddings [42], Facet [20] and COVAREP [12] features are extracted for the language, visual and acoustic modalities respectively.
Forced alignment is performed using P2FA [63] to obtain word utterance times and we align thevisual and acoustic features by computing their average over each word utterance interval. [Figure] [Figure]
[Table]
: For optimization
CMU MOSI we set the choice of pre
diction loss
We report MAE and Pearson’s correlation (Corr).
In addition, we also perform sentiment classiﬁcation on the CMU-MOSI dataset and report binaryaccuracy (Acc) and F1 score (F1).
On the ICTMMMO and YouTube datasets, we set the choiceof prediction loss function as categorical crossentropy and report classiﬁcation accuracy (Acc)and F1 score.
For all metrics, higher values indicate stronger performance, except MAE wherelower values indicate stronger performance.
Baselines: We compare to the following multimodal models: RMFN [29] uses a multistageapproach to learn hierarchical representations.
Itis the current state-of-the-art on CMU-MOSI.
LMF [31] approximates the expensive multimodal tensor products in TFN [64] with efﬁcient low-rank factors.
MFN [65] synchronizessequences using a multimodal gated memory.
MARN [66] uses multiple attention coefﬁcientsand hybrid LSTM memory components.
GMELSTM(A) [8] learns binary gating mechanisms toremove noisy modalities that are contradictory orredundant.
EF-LSTM concatenates multimodalinputs and uses a single LSTM [19].
For details on all baselines, please refer to the supplementary.
Results and Discussion
This section discusses several research questions and presents our experimental results.
[Table]
Comparison with Existing Work: Q1:
Howdoes MCTN compare with existing state-ofthe-art approaching for multimodal sentimentanalysis?
We compare MCTN with the existing state-of-the-art models 3 .
From Table 4, MCTN achieves new start-of-the-art resultson binary classiﬁcation accuracy, F1 score,and MAE on CMU-MOSI.
State-of-the-art results are also achieved on the ICT-MMMOand YouTube datasets. These results are even Enc
Dec
Enc next input Enc Dec (b) Enc
Dec
Enc
Dec
Encc
Enc Dec (c) (d) (a) (f) (g) (e) [ ] [ ] Enc Dec (h)
Dec Dec
Step 1 Step Seq2Seq 1 Seq2Seqconcat concat Bimodal Variations Trimodal Variations (a) (e) (d) (g) (b) (f) (c) (f) more impressive considering that MCTN only uses the language modality during testing, while otherbaseline models use all three modalities.
Adding More Modalities: Q2:
What is the impact of increasing the number of modalities duringtraining for MCTN with cyclic translations?
We run experiments with MCTN using combinationsof two or three modalities with cyclic translations.
From Table 2, we observe that adding moremodalities improves performance, indicating that the joint representations learned are leveragingthe information from more input modalities.
This also implies that cyclic translations are a viablemethod to learn joint representations from multiple modalities since little information is lost fromadding more modality translations.
Another observation is that using language as the source modalityalways leads to the best performance, which is intuitive since the language modality contains themost information towards sentiment [64].
In addition, we visually inspect the joint representations learnt from MCTN as we add more modalitiesduring training.
The joint representations for each video segment in CMU-MOSI are extracted fromthe best performing model for each number of modalities and then projected into two dimensionsvia the t-SNE algorithm [59].
Each point is colored red or blue depending on whether the videosegment is annotated for positive or negative sentiment.
From Figure 4, we observe that the jointrepresentations become increasingly separable as the more modalities are added when the MCTN istrained.
This is consistent with increasing discriminative performance as seen in Table 2.
Ablation Studies: We devise the following ablation models to test each design decision in MCTN:the use of cyclic translations, shared Seq2Seq models, modality ordering, and hierarchical structure.
For bimodal MCTN, we design the following ablation models shown in the left half of Figure 5: (a)is our proposed MCTN between X S and X T , (b) is the MCTN based on translation from X S to X Twithout cyclic translations, (c) does not use cyclic translations but rather performs two independenttranslations between X S and X T , (d) is the pair of MCTN models with different inputs (of the samemodality pair) and then using the concatenation of the joint representations E S → T and E T → S as theﬁnal embeddings.
For trimodal MCTN, we design the following ablation models shown in the righthalf of Figure 5: (e) is the proposed hierarchical MCTN between X S , X T 1 and X T 2 , (f) is the MCTNbased on translation from X S to X T 1 without cyclic translations, (g) is extended from (d) whichdoes not use cyclic translations but rather performs two independent translations between X S and
X T 1 , and ﬁnally, (h) does not perform a ﬁrst level of cyclic translation but directly translates theconcatenated modality pair [ X S , X T 1 ] into X T 2 .
Q3:
What is the impact of cyclic translations in MCTN?
The bimodal and trimodal results are shownin Table 3.
Only the model in Figure 5(a) employs cyclic translations and they outperform the otherbaselines.
We make a similar observation for hierarchical MCTN: Figure 5(e) with cyclic translationsoutperforms the trimodal baselines (f), (g) and (h).
The gap for the trimodal case is especially large.
This implies that using cyclic translations is crucial in learning joint representations.
Our intuition is that using cyclic translations: (1) encourages the model to enforce symmetry between the jointrepresentations from source and target modalities, and (2) ensures that the joint representation retainsmaximal information from all modalities. [Table]
Q4: What is the effect of using two
Seq2Seq models instead of one shared
Seq2Seq model for cyclic translations?
We compare Figure 5(c), which [Table] uses one Seq2Seq model for translations with Figure 5(d), which uses twoseparate Seq2Seq models: one for forward and one for backward translation.
We observe from Table 3 that (c) > (d),so using one model with shared parameters is better.
This is also true forhierarchical MCTN: (f) > (g).
We hypothesize that this is because trainingtwo Seq2Seq models requires moredata and is prone to overﬁtting.
Q5:
What is the impact when varying source and target modalities forcyclic translations?
As shown in Tables 2 and 3, we observe that language contributes most towards thejoint representations.
For bimodalcases, combining language with visual is generally better than combining language with audio.
For hierarchical MCTN, presenting languageas the source modality leads to thebest performance, and a ﬁrst level ofcyclic translations between languageand visual is better than between language and acoustic.
On the other hand,only translating between visual andacoustic modalities dramatically decreases performance.
Further addinglanguage as a target modality for hierarchical MCTN will not help much aswell.
Overall, language is still the most important modality for multimodal sentiment analysis andmust be used as the source modality during translations.
Q6:
What is the impact of using two levels of hierarchical translations instead of one level for threemodalities?
Our hierarchical MCTN is shown in Figure 5(e).
In Figure 5(h), we concatenate twomodalities as input and use only one phase of translation.
From Table 3, we observe that (e) >
(h): bothlevels of modality translations are important in the hierarchical MCTN.
We believe that representationlearning is easier when the task is broken down recursively: using two translations each between asingle pair of modalities, rather than a single translation between all modalities.
Conclusion
To conclude, this paper investigated learning joint representations via cyclic modality translationsfrom source to target modalities.
During testing, we only need the source modality for predictionwhich ensures that our model remains robust from noisy or missing target modalities.
We demonstratethat cyclic translations and seq2seq models are especially useful for learning joint multimodalrepresentations.
In addition to achieving state-of-the-art results on three datasets, our model learnsincreasingly discriminative representations with more input modalities while maintaining robustnessto all target modalities.
Our approach presents several exciting areas for future work, such as: 1)combining our approach with the transformer architecture [60] for modality translations, 2) exploringpretrained deep language models [13, 43] for translations, as well as 3) extending our translationmodel to work other multimodal tasks involving language and raw speech signals (prosody), videoswith multiple speakers, and combinations of static and temporal data (i.e. image captioning).
A Multimodal Features
Here we present extra details on feature extraction for the language, visual and acoustic modalities.
Language: We used 300 dimensional Glove word embeddings trained on 840 billion tokens from the common 499 crawl dataset [42].
These word embeddings were used to embed a sequence of individual words from videosegment transcripts into a sequence of word vectors that represent spoken text.
Visual: The library Facet [20] is used to extract a set of visual features including facial action units, facial 502 landmarks, head pose, gaze tracking and HOG features [69].
These visual features are extracted from the fullvideo segment at 30Hz to form a sequence of facial gesture measures throughout time.
Acoustic: The software COVAREP [12] is used to extract acoustic features including 12 Mel-frequency cepstral 505 coefﬁcients, pitch tracking and voiced/unvoiced segmenting features [15], glottal source parameters [10, 16, 1, 3,2], peak slope parameters and maxima dispersion quotients [21].
These visual features are extracted from the 507 full audio clip of each segment at 100Hz to form a sequence that represent variations in tone of voice over anaudio segment.
B Multimodal Alignment
We perform forced alignment using P2FA [63] to obtain the exact utterance time-stamp of each word.
This 511 allows us to align the three modalities together.
Since words are considered the basic units of language we usethe interval duration of each word utterance as one time-step.
We acquire the aligned video and audio features by 513 computing the expectation of their modality feature values over the word utterance time interval [29].
B .1 Baseline Models
We also implement the Stacked, ( EF-SLSTM ) [18], Bidirectional ( EF-BLSTM ) [50], and Stacked Bidirec- 516 tional ( EF-SBLSTM ) LSTMs, as well as the following baselines: BC-LSTM [46], EF-HCRF [48], EF/MVLDHCRF [35], MV-HCRF [53], EF/MV-HSSHCRF [54], MV-LSTM [49], DF [38], SAL-CNN [61], C-MKL [45], 518 THMM [34], SVM [11, 41] and RF [6].
C Full Results
We present the full results across all baseline models in Table 3 and Table 4.
MCTN using all modalities achieves 521 new start-of-the-art results on binary classiﬁcation accuracy, F1 score, and MAE on the CMU-MOSI datasetfor multimodal sentiment analysis.
State-of-the-art results are also achieved on the ICT-MMMO and YouTube 523 datasets (Table 4).
These results are even more impressive considering that MCTN only uses the languagemodality during testing, while other baseline models use all three modalities.
[Table] [Table]