Modeling Spatiotemporal Multimodal Language with Recurrent Multistage Fusion
Abstract
Computational modeling of human multimodal language is an emerging researchapplication of spatiotemporal modeling spanning the language, visual and acousticmodalities.
Comprehending multimodal language requires modeling not only thespatial interactions within each modality (intra-modal interactions) but more importantly the interactions between modalities (cross-modal interactions) from complextemporal data.
We propose the Recurrent Multistage Fusion Network (RMFN)which decomposes the spatiotemporal fusion problem into multiple stages, each ofthem focused on a subset of multimodal signals for specialized, effective fusion.
Spatial cross-modal interactions are modeled using this multistage fusion approachwhich builds upon intermediate representations of previous stages.
Temporal andintra-modal interactions are modeled by integrating our proposed fusion approachwith a system of recurrent neural networks.
The RMFN displays state-of-theart performance in modeling multimodal language across three tasks relating tomultimodal sentiment analysis, emotion recognition, and speaker traits recognition.
Experiments show that each stage of fusion focuses on a different subset ofmultimodal signals and learns increasingly discriminative representations.
Introduction
Computational modeling of human multimodal language is an emerging research application ofspatiotemporal modeling.
This area focuses on modeling tasks such as multimodal sentimentanalysis [42], emotion recognition [9] and personality traits recognition [47].
The multimodaltemporal signals include the language (spoken words), visual (facial expressions, gestures) andacoustic modalities (prosody, vocal expressions).
At its core, these signals are highly structured withtwo prime forms of spatial interactions: intra-modal and cross-modal interactions [53].
Intra-modalinteractions refer to information within a speciﬁc modality.
For example, the arrangement of wordsin a sentence [15] or the sequence of facial muscle activations for a frown.
Cross-modal interactionsrefer to interactions between modalities.
For example, the simultaneous co-occurrence of a smile witha positive sentence or the delayed occurrence of a laughter after the end of a sentence.
Modeling thesespatiotemporal interactions lies at the heart of multimodal language analysis and has recently becomea centric research direction in multimodal machine learning [39, 49, 11, 60, 23, 25, 32, 63, 59, 44].
Recent advances in cognitive neuroscience have demonstrated the existence of multistage aggregationacross human cortical networks and functions [62], particularly during the integration of multisensoryspatiotemporal information [46].
At later stages of cognitive processing, higher level semanticmeaning is extracted from phrases, facial expressions, and tone of voice, eventually leading tothe formation of higher level cross-modal concepts [46, 62].
Inspired by these discoveries, wehypothesize that the computational modeling of cross-modal interactions also requires a multistagefusion process.
In this process, cross-modal representations can build upon the representations learnedduring earlier stages.
This decreases the burden on each stage of spatiotemporal fusion and allowseach stage of fusion to be performed in a more specialized and effective manner.
[Figure]
[Figure]
In this paper, we propose the Recurrent Multistage Fusion Network (RMFN) which decomposes the spatiotemporal fusion problem intomultiple recursive stages.
At each stage, a subsetof multimodal signals is highlighted and fusedwith previous fusion representations (Figure 1).
This divide-and-conquer approach decreases theburden on each fusion stage, allowing each stageto be performed in a more specialized and effective way.
In contrast, conventional fusionapproaches model interactions over multimodalsignals in one step [6].
Temporal and intramodal interactions are modeled by integratingour new multistage fusion process with a systemof recurrent neural networks.
Overall, RMFNjointly models intra-modal and cross-modal interactions for spatiotemporal fusion.
RMFNachieves state-of-the-art performance on threetasks related to multimodal language: sentiment analysis, emotion recognition, and speakertraits recognition.
Through a comprehensive setof ablation experiments and visualizations, wedemonstrate the advantages of deﬁning multiplerecursive stages for spatiotemporal fusion.
Related Work
Previous approaches in spatiotemporal modelingfor multimodal language can be categorized as:
Non-temporal Models simplify the problem byaveraging temporal information through timeand using supervised learning methods [30, 12, 1, 45, 70, 42].
These approaches have troublemodeling long sequences since the average statistics do not accurately reﬂect temporal dynamics [65].
Temporal Graphical Models such as Hidden Markov Models [7], Conditional Random Fields(CRFs) [35], and Hidden Conditional Random Fields (HCRFs) [52] were shown to work well onmodeling spatiotemporal data [41, 40, 28, 66].
Multimodal extensions have been proposed includingmulti-view HCRFs [57], multi-layered CRFs [57], and multi-view hierarchical models [58].
Temporal Neural Networks , such as Recurrent Neural Networks [20, 31] and Long-short Term
Memory (LSTM) networks [26] have been used for spatiotemporal modeling [72, 56, 55, 21, 10,24, 36, 53, 11].
Recently, more advanced models were proposed that use Bayesian ranking algorithms [38], external memory mechanisms [68], or low-rank tensors [39] for spatiotemporal fusion.
These methods assume that fusion should be performed all at once rather than across multiple stages.
Our empirical evaluations show the advantages of our spatiotemporal fusion approach.
Recurrent Multistage Fusion Network
In this section we describe the Recurrent Multistage Fusion Network (RMFN) for spatiotemporalfusion (Figure 2).
Given a set of modalities { l ( anguage ) ,v ( isual ) ,a ( coustic )} , each modalitym ∈ { l,v,a } is represented as a temporal sequence X m = { x m 1 , x m 2 , x m 3 , ⋯ , x mT } , where x mt is theinput at time t .
Each sequence X m is modeled with an intra-modal recurrent neural network.
At timet , each recurrent network will output a unimodal representation h mt .
The Multistage Fusion Processuses a recursive approach to fuse all unimodal representations h mt into a cross-modal representationz t which is then fed back into each intra-modal recurrent network.
Multistage Fusion Process (MFP) is a modular neural approach that performs multistage fusion tomodel cross-modal interactions.
MFP has three modules: HIGHLIGHT , FUSE and SUMMARIZE .
At each stage, HIGHLIGHT identiﬁes a subset of multimodal signals from [ h lt , h vt , h at ] that will beused for that stage of fusion.
FUSE then performs two subtasks simultaneously: a local fusion of thehighlighted features and integration with representations from previous stages.
Both HIGHLIGHTand FUSE are realized using memory-based networks which enable coherence between stages and storage of previously modeled representations.
After all stages, SUMMARIZE takes the representationof the ﬁnal stage and translates it into a cross-modal representation z t .
[Figure]
𝒍
LSTHM 𝒍
[Figure]
LSTHM
[Figure]
𝑧 &
[Figure]
FUSE FUSE
FUSE 𝐡 &( 𝐡 &) 𝐡 &* stagestagestage 𝑲
SUMMARIZE time 𝒕 time 𝒕+ ⋯
HIGHLIGHT ⋯
HIGHLIGHT HIGHLIGHT
We now present the details of thethree modules: HIGHLIGHT , FUSEand SUMMARIZE .
Multistage fusionbegins with the concatenation ofintra-modal network outputs h t =
⊕ m ∈ M h mt .
We use superscript [ k ]to denote the indices of each stagek = 1 , ⋯ ,K during K total stages ofmultistage fusion.
denotes the network parameters across all modules.
HIGHLIGHT : At each stage k , a subset of the multimodal signals h t willbe highlighted for fusion.
This module is deﬁned by the process function f H : a [ k ] t
[Equation]
where at stage k , a [ k ] t is a set of attention weights which are inferred basedon the previously assigned attention weights a [ 1 ∶ k − 1 ] t .
As a result, the highlights at a speciﬁc stage kwill be dependent on previous highlights.
To fully encapsulate these dependencies, the attention assignment process is performed in a recurrent manner using a LSTM which we call the HIGHLIGHT
LSTM.
The initial HIGHLIGHT LSTM memory at stage 0, c HIGHLIGHT [ 0 ] t , is initialized using anetwork M that maps h t into LSTM memory space c HIGHLIGHT [ 0 ] t =
M( h t ; Θ ) .
This allowsthe memory of the HIGHLIGHT LSTM to dynamically adjust to the intra-modal representationsh t .
The output of the HIGHLIGHT LSTM h HIGHLIGHT [ k ] t is softmax activated to produce attention weights a [ k ] t at every stage k of the multistage fusion process: a [ k ] t j = exp ( h HIGHLIGHT
[ k ] t j )/ Z ,
Z =
∑ ∣ h HIGHLIGHT [ k ] t ∣ d =exp
( h HIGHLIGHT [ k ] t d ) and a [ k ] t is fed as input into the HIGHLIGHT LSTM at stagek + 1 .
Therefore, the HIGHLIGHT LSTM functions as a decoder LSTM [61, 14] in order to capturethe dependencies on previous attention assignments.
Highlighting is performed by ˜ h
[ k ] t =
[Equation]
a [ k ] t ,where ⊙ denotes the Hadamard product and ˜ h
[ k ] t are the attended multimodal signals that will beused for the fusion at stage k .
FUSE :
The highlighted multimodal signals are simultaneously fused in a local fusion and thenintegrated with fusion representations from previous stages.
This module is deﬁned by the processfunction f F : s [ k ] t = f F ( ˜ h
[ k ] t ; s [ 1 ∶ k − 1 ] t , Θ ) where s [ k ] t denotes the integrated fusion representations atstage k .
We employ a FUSE LSTM to simultaneously perform the local fusion and the integrationwith previous fusion representations.
The FUSE LSTM input gate enables a local fusion while the
FUSE LSTM forget and output gates enable integration with previous fusion results.
The initial
FUSE LSTM memory at stage 0, c FUSE [ 0 ] t , is initialized using random orthogonal matrices [5, 37].
SUMMARIZE : After completing K recursive stages, SUMMARIZE generates a cross-modal representation using all fusion representations s [ 1 ∶ K ] t .
This operation is deﬁned as: z t =
S
( s [ 1 ∶ K ] t ; Θ ) wherez t is the ﬁnal output of the multistage fusion process and represents all cross-modal interactionsdiscovered at time t .
The summarized representation is fed into the intra-modal recurrent networks.
Intra-model Recurrent Networks:
To integrate z t with the temporal intra-modal representations,we employ a system of Long Short-term Hybrid Memories (LSTHMs) [69].
The LSTHM extendsthe LSTM formulation to include z t in a hybrid memory component.
The hybrid memory containsboth intra-modal interactions from individual modalities x mt as well as the cross-modal interactionscaptured in z t .
Multimodal prediction is performed using a representation E which integrates (1)the last outputs from the LSTHMs and (2) the last cross-modal representation z T .
E is deﬁned as 142 E =
(⊕ m ∈ M h mT )⊕ z T where ⊕ denotes vector concatenation.
E summarizes all intra-modal andcross-modal representations from multimodal spatiotemporal data.
RMFN is differentiable end-to-endwhich allows network parameters Θ to be learned using gradient descent approaches.
[Table]
Results and Discussion
Experimental Setup: To evaluate the performance of RMFN, three domains of multimodal languagewere selected: multimodal sentiment analysis on CMU-MOSI [70], emotion recognition on IEMOCAP [9], and speaker traits recognition on POM [47].
All datasets consist of monologue videos.
GloVe word embeddings [48], Facet [29] and COVAREP [17] are extracted for the language, visualand acoustic modalities respectively 1 .
For classiﬁcation, we report accuracy A c for c classes and F1score.
For regression, we report Mean Absolute Error MAE and Pearson’s correlation r .
Performance:
Table 1 shows results on CMU-MOSI and IEMOCAP 2 .
We achieve state-of-the-art orcompetitive results for all metrics, highlighting RMFN’s capability in spatiotemporal fusion.
[Table]
Analysis: To achieve a deeper understanding of the multistage fusion process, we study four researchquestions.
(Q1): the effect of the number of stages K during multistage fusion on performance.
(Q2):the -modal
Q3):whether modeling cross-modal interactions are helpful.
(Q4): whether attention weights from the
HIGHLIGHT module are required for modeling cross-modal interactions.
Q1:
We test the baseline RMFN-R K which performs fusion K stages of fusion.
From Table 2, weobserve that RMFN-R 1 (single fusion stage) underperforms as compared to RMFN which performsmultistage fusion, and increasing the number of stages K increases the model’s capability to modelcross-modal interactions up to a certain point ( K = 3 ) in our experiments.
Further increases led todecreases in performance and we hypothesize this is due to overﬁtting on the dataset.
Q2:
We pay close attention to the performance comparison with respect to MARN which modelsmultiple cross-modal interactions all at once (see Table 2).
RMFN shows improved performance,indicating that multistage fusion is both effective and efﬁcient for spatiotemporal modeling.
Q3: RMFN (no MFP) represents a system of LSTHMs without the integration of z t from the MFPto model cross-modal interactions.
From Table 2, RMFN (no MFP) is outperformed by RMFN,conﬁrming that modeling cross-modal interactions is crucial for spatiotemporal fusion.
Q4: RMFN (no HIGHLIGHT ) removes the HIGHLIGHT module from MFP during multistage fusion.
From Table 2, RMFN (no HIGHLIGHT ) underperforms, indicating that highlighting multimodalrepresentations using attention weights are important for modeling cross-modal interactions.
Conclusion
In conclusion, this paper proposed the Recurrent Multistage Fusion Network (RMFN) which decomposes the spatiotemporal fusion problem into multiple stages, each focused on a subset of multimodalsignals.
Extensive experiments across three spatiotemporal datasets reveal that RMFN is highly effective in modeling multimodal language.
Our visualizations also reveal that the stages coordinateto capture both synchronous and asynchronous spatiotemporal interactions.
A Experiment Details
A .1 Multimodal Tasks and Datasets
Multimodal Sentiment Analysis involves analyzing speaker sentiment based on video content.
Multimodal sentiment analysis extends conventional language-based sentiment analysis to a multimodalsetup where both verbal and non-verbal signals contribute to the expression of sentiment.
We use
CMU-MOSI [70] which consists of 2199 opinion segments from online videos each annotated withsentiment in the range [-3,3].
Multimodal Emotion Recognition involves identifying speaker emotions based on both verbal andnonverbal behaviors.
We perform experiments on the IEMOCAP dataset [9] which consists of 7318segments of recorded dyadic dialogues annotated for the presence of human emotions happiness,sadness, anger and neutral.
Multimodal Speaker Traits Recognition involves recognizing speaker traits based on multimodalcommunicative behaviors.
POM [47] contains 903 movie review videos each annotated for 12speaker traits: conﬁdent (con), passionate (pas), voice pleasant (voi), credible (cre), vivid (viv),expertise (exp), reserved (res), trusting (tru), relaxed (rel), thorough (tho), nervous (ner), persuasive(per) and humorous (hum).
A .2 Multimodal Features
Here we present extra details on feature extraction for the language, visual and acoustic modalities.
Language: We used 300 dimensional Glove word embeddings trained on 840 billion tokens from thecommon crawl dataset [48].
These word embeddings were used to embed a sequence of individualwords from video segment transcripts into a sequence of word vectors that represent spoken text.
Visual: The library Facet [29] is used to extract a set of visual features including facial action units,facial landmarks, head pose, gaze tracking and HOG features [71].
These visual features are extractedfrom the full video segment at 30Hz to form a sequence of facial gesture measures throughout time.
Acoustic: The software COVAREP [17] is used to extract acoustic features including 12 Melfrequency cepstral coefﬁcients, pitch tracking and voiced/unvoiced segmenting features [18], glottalsource parameters [13, 19, 2, 4, 3], peak slope parameters and maxima dispersion quotients [33].
These visual features are extracted from the full audio clip of each segment at 100Hz to form asequence that represent variations in tone of voice over an audio segment.
A .3 Multimodal Alignment
We perform forced alignment using P2FA [66] to obtain the exact utterance time-stamp of each word.
This allows us to align the three modalities together.
Since words are considered the basic units oflanguage we use the interval duration of each word utterance as one time-step.
We acquire the alignedvideo and audio features by computing the expectation of their modality feature values over the wordutterance time interval [68].
A .4 Baseline Models
We compare to the following models for multimodal machine learning: MFN [68] synchronizesmultimodal sequences using a multi-view gated memory.
It is the current state of the art on CMUMOSI and POM.
MARN [69] models intra-modal and cross-modal interactions using multipleattention coefﬁcients and hybrid LSTM memory components.
GME-LSTM(A)
[11] learns binarygating mechanisms to remove noisy modalities that are contradictory or redundant for prediction.
TFN [67] models unimodal, bimodal and trimodal interactions using tensor products.
BC-LSTM[51] performs context-dependent sentiment analysis and emotion recognition, currently state of theart on IEMOCAP.
EF-LSTM concatenates the multimodal inputs and uses that as input to a single
LSTM [27].
We also implement the Stacked, ( EF-SLSTM ) [22] Bidirectional ( EF-BLSTM )
[54]and Stacked Bidirectional ( EF-SBLSTM ) LSTMs.
The best model is reported as EF- (⋆) LSTM .
EFHCRF: (Hidden Conditional Random Field)
[52] uses a HCRF to learn latent variables conditionedon the concatenated input.
We also implement the following variations: EF-LDHCRF:
(Latent
Discriminative HCRFs) [43], MV-HCRF (Multi-view HCRF) [57], MV-LDHCRF , EF-HSSHCRF(Hierarchical Sequence Summarization HCRF) [58] and MV-HSSHCRF .
The best performing
[Table]
RMFN outperforms the current state-of-the-art across all evaluation metrics.
early fusion model is reported as EF- (⋆) HCRF while the best multi-view model is reported as
MV- (⋆) HCRF .
For descriptions of the remaining baselines, we refer the reader to EF-HCRF [52],
EF/MV-LDHCRF [43], MV-HCRF [57], EF/MV-HSSHCRF [58], MV-LSTM [53], DF [45],
SAL-CNN [64], C-MKL [50], THMM [42], SVM [16, 47] and RF [8].
B Additional Results
Here we record the complete set of results for all the baseline models across all the datasets, tasksand metrics.
Table 1 summarizes the complete results for sentiment analysis on the CMU-MOSIdataset.
Table 2 presents the complete results for emotion recognition on the IEMOCAP datasetand Table 3 presents the complete results for personality
traits prediction on the POM dataset.
Forexperiments on the POM dataset we report additional results on MAE and correlation metrics forpersonality traits regression.
We achieve signiﬁcant improvement over state-of-the-art multi-view anddataset speciﬁc approaches across all these datasets, highlighting the RMFN’s capability in analyzingsentiment, emotions and speaker traits from human multimodal language.
C Visualizations
Using an attention assignment mechanism during the HIGHLIGHT process gives interpretability tothe model since it allows us to visualize the attended multimodal signals at each stage and time step(see Figure 3).
Using RMFN trained on the CMU-MOSI dataset, we plot the attention weights acrossthe multistage fusion process for three videos in CMU-MOSI.
Based on these visualizations we ﬁrstdraw the following general observations on spatiotemporal fusion:
Across stages (Spatio): Attention weights change their behaviors across the multiple stages of fusion.
Some features are highlighted by earlier stages while other features are used in later stages.
Thissupports our hypothesis that RMFN learns to specialize in different stages of the spatiotemporalfusion process.
[Table]
RMFN achieves state-of-the-art or competitive performance across all evaluation metrics.
[Table]
Across time (Temporal): Attention weights vary over time and adapt to the multimodal inputs.
We
[Table]
observe that the attention weights are similar if the input contains no new information.
As soon asnew multimodal information comes in, the highlighting mechanism in RMFN adapts to these newinputs.
[Table]
Priors: Based on the distribution of attention weights, we observe that the language and acoustic
[Table]
modalities seem the most commonly highlighted.
This represents a prior over the expression of
[Table]
sentiment in human multimodal language and is closely related to the strong connections between
[Table]
language and speech in human communication [34].
[Table]
Inactivity: Some attention coefﬁcients are not active (always orange) throughout time.
We hypothesize that these corresponding dimensions carry only intra-modal dynamics and are not involved in theformation of cross-modal interactions.
[Table]
[Table]
[Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
C .1 Qualitative Analysis
In addition to the general observations above, Figure 3 shows three examples where multistage fusionlearns cross-modal representations across three different scenarios.
Synchronized Interactions: In Figure 3(a), the language features are highlighted corresponding tothe utterance of the word “fun” that is highly indicative of sentiment ( t = 5 ).
This sudden change isalso accompanied by a synchronized highlighting of the acoustic features.
We also notice that thehighlighting of the acoustic features lasts longer across the 3 stages since it may take multiple stagesto interpret all the new acoustic behaviors (elongated tone of voice and phonological emphasis).
Asynchronous Trimodal Interactions: In Figure 3(b), the language modality displays ambiguoussentiment: “delivers a lot of intensity” can be inferred as both positive or negative.
We observe thatthe circled attention units in the visual and acoustic features correspond to the asynchronous presenceof a smile ( t = 2 ∶ 5 ) and phonological emphasis ( t = 3 ) respectively.
These nonverbal behaviorsresolve ambiguity in language and result in an overall display of positive sentiment.
We further notethe coupling of attention weights that highlight the language, visual and acoustic features acrossstages ( t = 3 ∶ 5 ), further emphasizing the coordination of all three modalities during multistagefusion despite their asynchronous occurrences.
Bimodal Interactions: In Figure 3(c), the language modality is better interpreted in the context ofacoustic behaviors.
The disappointed tone and soft voice provide the nonverbal information usefulfor sentiment inference.
This example highlights the bimodal interactions ( t = 4 ∶ 7 ) in alternatingstages: the acoustic features are highlighted more in earlier stages while the language features arehighlighted increasingly in later stages.