Modeling Spatiotemporal Multimodal Language with Recurrent Multistage Fusion
Abstract
Computational modeling of human multimodal language is an emerging research application of spatiotemporal modeling spanning the language, visual and acoustic modalities.
Comprehending multimodal language requires modeling not only the spatial interactions within each modality (intra-modal interactions) but more importantly the interactions between modalities (cross-modal interactions) from complex temporal data.
We propose the Recurrent Multistage Fusion Network (RMFN) which decomposes the spatiotemporal fusion problem into multiple stages, each of them focused on a subset of multimodal signals for specialized, effective fusion.
Spatial cross-modal interactions are modeled using this multistage fusion approach which builds upon intermediate representations of previous stages.
Temporal and intra-modal interactions are modeled by integrating our proposed fusion approach with a system of recurrent neural networks.
The RMFN displays state-of-theart performance in modeling multimodal language across three tasks relating to multimodal sentiment analysis, emotion recognition, and speaker traits recognition.
Experiments show that each stage of fusion focuses on a different subset of multimodal signals and learns increasingly discriminative representations.
Introduction
Computational modeling of human multimodal language is an emerging research application of spatiotemporal modeling.
This area focuses on modeling tasks such as multimodal sentiment analysis [41], emotion recognition [9] and personality traits recognition [46].
The multimodal temporal signals include the language (spoken words), visual (facial expressions, gestures) and acoustic modalities (prosody, vocal expressions).
At its core, these signals are highly structured with two prime forms of spatial interactions: intra-modal and cross-modal interactions [52].
Intra-modal interactions refer to information within a speciﬁc modality.
For example, the arrangement of words in a sentence [15] or the sequence of facial muscle activations for a frown.
Cross-modal interactions refer to interactions between modalities.
For example, the simultaneous co-occurrence of a smile with a positive sentence or the delayed occurrence of a laughter after the end of a sentence.
Modeling these spatiotemporal interactions lies at the heart of multimodal language analysis and has recently become a centric research direction in multimodal machine learning [38, 48, 11, 59, 23, 25, 31, 62, 58, 43].
Recent advances in cognitive neuroscience have demonstrated the existence of multistage aggregation across human cortical networks and functions [61], particularly during the integration of multisensory spatiotemporal information [45].
At later stages of cognitive processing, higher level semantic meaning is extracted from phrases, facial expressions, and tone of voice, eventually leading to the formation of higher level cross-modal concepts [45, 61].
Inspired by these discoveries, we hypothesize that the computational modeling of cross-modal interactions also requires a multistage fusion process.
In this process, cross-modal representations can build upon the representations learned during earlier stages.
This decreases the burden on each stage of spatiotemporal fusion and allows each stage of fusion to be performed in a more specialized and effective manner.
[Figure]
[Figure]
In this paper, we propose the Recurrent Multistage Fusion Network (RMFN) which decomposes the spatiotemporal fusion problem into multiple stages.
At each stage, a subset of multimodal signals is highlighted and fused with previous fusion representations (Figure 1).
This divide-and-conquer approach decreases the burden on each fusion stage, allowing each stage to be performed in a more specialized and effective way.
In contrast, conventional fusion approaches model interactions over multimodal signals in one step [6].
Temporal and intra-modal interactions are modeled by integrating our new multistage fusion process with a system of recurrent neural networks.
Overall, RMFN jointly models intra-modal and cross-modal interactions for spatiotemporal fusion.
RMFN achieves stateof-the-art performance on three tasks related to multimodal language: sentiment analysis, emotion recognition, and speaker traits recognition.
Through a comprehensive set of ablation experiments and visualizations, we demonstrate the advantages of deﬁning multiple stages for spatiotemporal fusion.
Related Work
Previous approaches in spatiotemporal modeling for multimodal language can be categorized as:
Non-temporal Models simplify the problem by averaging temporal information through time and using supervised learning methods [29, 12, 1, 44, 69, 41].
These approaches have trouble modeling long sequences since the average statistics do not accurately reﬂect temporal dynamics [64].
Temporal Graphical Models such as Hidden Markov Models [7], Conditional Random Fields (CRFs) [34], and Hidden Conditional Random Fields (HCRFs) [51] were shown to work well on modeling spatiotemporal data [40, 39, 27, 65].
Multimodal extensions have been proposed including multi-view HCRFs [56], multi-layered CRFs [56], and multi-view hierarchical models [57].
Temporal Neural Networks , such as Recurrent Neural Networks [20, 30] and Long-short Term
Memory (LSTM) networks [26] have been used for spatiotemporal modeling [71, 55, 54, 21, 10, 24, 35, 52, 11].
Recently, more advanced models were proposed that use Bayesian ranking algorithms [37], external memory mechanisms [67], or low-rank tensors [38] for spatiotemporal fusion.
These methods assume that fusion should be performed all at once rather than across multiple stages.
Our empirical evaluations show the advantages of our spatiotemporal fusion approach.
Recurrent Multistage Fusion Network
In this section we describe the Recurrent Multistage Fusion Network (RMFN) for spatiotemporal fusion (Figure 2).
Given a set of modalities { l ( anguage ) ,v ( isual ) ,a ( coustic )} , each modality m ∈ { l,v,a } is represented as a temporal sequence X m = { x m 1 , x m 2 , x m 3 , ⋯ , x mT } , where x mt is the input at time t .
Each sequence X m is modeled with an intra-modal recurrent neural network.
At time t , each recurrent network will output a unimodal representation h mt .
The Multistage Fusion Process uses a multistage approach to fuse all unimodal representations h mt into a cross-modal representation z t which is then fed back into each intra-modal recurrent network.
Multistage Fusion Process (MFP) is a modular neural approach that performs multistage fusion to model cross-modal interactions.
MFP has three modules: HIGHLIGHT , FUSE and SUMMARIZE .
At each stage, HIGHLIGHT identiﬁes a subset of multimodal signals from [ h lt , h vt , h at ] that will be used for that stage of fusion.
FUSE then performs two subtasks simultaneously: a local fusion of the highlighted features and integration with representations from previous stages.
Both HIGHLIGHT and FUSE are realized using memory-based networks which enable coherence between stages and storage of previously modeled representations.
After all stages, SUMMARIZE takes the representation of the ﬁnal stage and translates it into a cross-modal representation z t .
Recurrent Multistage Fusion Network
[Figure]
𝒍
LSTHM 𝒍 LSTHM 𝒂
LSTHM 𝒂 𝑧 &
[Figure]
Process
FUSE FUSE
FUSE 𝐡 &( 𝐡 &) 𝐡 &* stagestagestage 𝑲
SUMMARIZE time 𝒕 time 𝒕+ ⋯
HIGHLIGHT ⋯
HIGHLIGHT HIGHLIGHT
We now present the details of the three modules: HIGHLIGHT , FUSE and SUMMARIZE .
Multistage fusion begins with the concatenation of intra-modal network outputs h t =
⊕ m ∈ M h mt .
We use superscript [ k ] to denote the indices of each stage k = 1 , ⋯ ,K during K total stages of multistage fusion.
Θ denotes the network parameters across all modules.
HIGHLIGHT : At each stage k , a subset of the multimodal signals h t will be highlighted for fusion.
This module is deﬁned by the process function f H : a [ k ] t
[Equation]
where at stage k , a [ k ] t is a set of attention weights which are inferred based on the previously assigned attention weights a [ 1 ∶ k − 1 ] t .
As a result, the highlights at a speciﬁc stage k will be dependent on previous highlights.
To fully encapsulate these dependencies, the attention assignment process is performed in a recurrent manner using a LSTM which we call the HIGHLIGHT
LSTM.
The initial HIGHLIGHT LSTM memory at stage 0, c HIGHLIGHT [ 0 ] t , is initialized using a network M that maps h t into LSTM memory space c HIGHLIGHT [ 0 ] t =
M( h t ; Θ ) .
This allows the memory of the HIGHLIGHT LSTM to dynamically adjust to the intra-modal representations h t .
The output of the HIGHLIGHT LSTM h HIGHLIGHT [ k ] t is softmax activated to produce attention weights a [ k ] t at every stage k of the multistage fusion process: a [ k ] t j = exp ( h HIGHLIGHT
[ k ] t j )/ Z ,
Z =
∑ ∣ h HIGHLIGHT [ k ] t ∣ d =exp
( h HIGHLIGHT [ k ] t d ) and a [ k ] t is fed as input into the HIGHLIGHT LSTM at stage k
+ 1 .
Therefore, the HIGHLIGHT LSTM functions as a decoder LSTM [60, 14] in order to capture the dependencies on previous attention assignments.
Highlighting is performed by ˜ h
[ k ] t = h t ⊙ a [ k ] t , where ⊙ denotes the Hadamard product and ˜ h
[ k ] t are the attended multimodal signals that will be used for the fusion at stage k .
FUSE :
The highlighted multimodal signals are simultaneously fused in a local fusion and then integrated with fusion representations from previous stages.
This module is deﬁned by the process function f F : s [ k ] t = f F ( ˜ h
[ k ] t ; s [ 1 ∶ k − 1 ] t , Θ ) where s [ k ] t denotes the integrated fusion representations at stage k .
We employ a FUSE LSTM to simultaneously perform the local fusion and the integration with previous fusion representations.
The FUSE LSTM input gate enables a local fusion while the
FUSE LSTM forget and output gates enable integration with previous fusion results.
The initial
FUSE LSTM memory at stage 0, c FUSE [ 0 ] t , is initialized using random orthogonal matrices [5, 36].
SUMMARIZE : After completing K stages, SUMMARIZE generates a cross-modal representation using all fusion representations s [ 1 ∶ K ] t .
This operation is deﬁned as: z t =
S
( s [ 1 ∶ K ] t ; Θ ) where z t is the ﬁnal output of the multistage fusion process and represents all cross-modal interactions discovered at time t .
The summarized representation is fed into the intra-modal recurrent networks.
Intra-model Recurrent Networks:
To integrate z t with the temporal intra-modal representations, we employ a system of Long Short-term Hybrid Memories (LSTHMs) [68].
The LSTHM extends the LSTM formulation to include z t in a hybrid memory component.
The hybrid memory contains both intra-modal interactions from individual modalities x mt as well as the cross-modal interactions captured in z t .
Multimodal prediction is performed using a representation E which integrates (1) the last outputs from the LSTHMs and (2) the last cross-modal representation z T .
E is deﬁned as E =
(⊕ m ∈ M h mT )⊕ z T where ⊕ denotes vector concatenation.
E summarizes all intra-modal and cross-modal representations from multimodal spatiotemporal data.
RMFN is differentiable end-to-end which allows network parameters Θ to be learned using gradient descent approaches.
[Table]
Results and Discussion
Experimental Setup: To evaluate the performance of RMFN, three domains of multimodal language were selected: multimodal sentiment analysis on CMU-MOSI [69], emotion recognition on IEMOCAP [9], and speaker traits recognition on POM [46].
All datasets consist of monologue videos.
GloVe word embeddings [47], Facet [28] and COVAREP [17] are extracted for the language, visual and acoustic modalities respectively 1 .
For classiﬁcation, we report accuracy A c across c classes and
F1 score.
For regression, we report Mean Absolute Error (MAE) and Pearson’s correlation (Corr).
Performance:
Table 1 shows results on CMU-MOSI and IEMOCAP 2 .
We achieve state-of-the-art or competitive results for all metrics, highlighting RMFN’s capability in spatiotemporal fusion.
[Table]
Analysis: To achieve a deeper understanding of the multistage fusion process, we study four research questions.
(Q1): the effect of the number of stages K during multistage fusion on performance.
(Q2): the comparison between multistage and independent modeling of cross-modal interactions.
(Q3): whether modeling cross-modal interactions are helpful.
(Q4): whether attention weights from the
HIGHLIGHT module are required for modeling cross-modal interactions.
Q1:
We test the baseline RMFN-R K which performs fusion K stages of fusion.
From Table 2, we observe that RMFN-R 1 (single fusion stage) underperforms as compared to RMFN which performs multistage fusion, and increasing the number of stages K increases the model’s capability to model cross-modal interactions up to a certain point ( K = 3 ) in our experiments.
Further increases led to decreases in performance and we hypothesize this is due to overﬁtting on the dataset.
Q2:
We pay close attention to the performance comparison with respect to MARN which models multiple cross-modal interactions all at once (see Table 2).
RMFN shows improved performance, indicating that multistage fusion is both effective and efﬁcient for spatiotemporal modeling.
Q3: RMFN (no MFP) represents a system of LSTHMs without the integration of z t from the MFP to model cross-modal interactions.
From Table 2, RMFN (no MFP) is outperformed by RMFN, conﬁrming that modeling cross-modal interactions is crucial for spatiotemporal fusion.
Q4: RMFN (no HIGHLIGHT ) removes the HIGHLIGHT module from MFP during multistage fusion.
From Table 2, RMFN (no HIGHLIGHT ) underperforms, indicating that highlighting multimodal representations using attention weights are important for modeling cross-modal interactions.
Conclusion
In conclusion, this paper proposed the Recurrent Multistage Fusion Network (RMFN) which decomposes the spatiotemporal fusion problem into multiple stages, each focused on a subset of multimodal signals.
Extensive experiments across three spatiotemporal datasets reveal that RMFN is highly effective in modeling multimodal language.
Our visualizations also reveal that the stages coordinate to capture both synchronous
and
[Table]
A Experiment Details A .1 Multimodal Tasks and Datasets
Multimodal Sentiment Analysis involves analyzing speaker sentiment based on video content.
Multimodal sentiment analysis extends conventional language-based sentiment analysis to a multimodal setup where both verbal and non-verbal signals contribute to the expression of sentiment.
We use
CMU-MOSI [69] which consists of 2199 opinion segments from online videos each annotated with sentiment in the range [-3,3].
Multimodal Emotion Recognition involves identifying speaker emotions based on both verbal and nonverbal behaviors.
We perform experiments on the IEMOCAP dataset [9] which consists ofsegments of recorded dyadic dialogues annotated for the presence of human emotions happiness, sadness, anger and neutral.
Multimodal Speaker Traits Recognition involves recognizing speaker traits based on multimodal communicative behaviors.
POM [46] contains 903 movie review videos each annotated forspeaker traits: conﬁdent (con), passionate (pas), voice pleasant (voi), credible (cre), vivid (viv), expertise (exp), reserved (res), trusting (tru), relaxed (rel), thorough (tho), nervous (ner), persuasive (per) and humorous (hum).
A .2 Multimodal Features
Here we present extra details on feature extraction for the language, visual and acoustic modalities.
Language: We used 300 dimensional Glove word embeddings trained on 840 billion tokens from the common crawl dataset [47].
These word embeddings were used to embed a sequence of individual words from video segment transcripts into a sequence of word vectors that represent spoken text.
Visual: The library Facet [28] is used to extract a set of visual features including facial action units, facial landmarks, head pose, gaze tracking and HOG features [70].
These visual features are extracted from the full video segment at 30Hz to form a sequence of facial gesture measures throughout time.
Acoustic: The software COVAREP [17] is used to extract acoustic features including 12 Melfrequency cepstral coefﬁcients, pitch tracking and voiced/unvoiced segmenting features [18], glottal source parameters [13, 19, 2, 4, 3], peak slope parameters and maxima dispersion quotients [32].
These visual features are extracted from the full audio clip of each segment at 100Hz to form a sequence that represent variations in tone of voice over an audio segment.
A .3 Multimodal Alignment
We perform forced alignment using P2FA [65] to obtain the exact utterance time-stamp of each word.
This allows us to align the three modalities together.
Since words are considered the basic units of language we use the interval duration of each word utterance as one time-step.
We acquire the aligned video and audio features by computing the expectation of their modality feature values over the word utterance time interval [67].
A .4 Baseline Models
We compare to the following models for multimodal machine learning: MFN [67] synchronizes multimodal sequences using a multi-view gated memory.
It is the current state of the art on CMUMOSI and POM.
MARN [68] models intra-modal and cross-modal interactions using multiple attention coefﬁcients and hybrid LSTM memory components.
GME-LSTM(A) [11] learns binary gating mechanisms to remove noisy modalities that are contradictory or redundant for prediction.
TFN [66] models unimodal, bimodal and trimodal interactions using tensor products.
BC-LSTM [50] performs context-dependent sentiment analysis and emotion recognition, currently state of the art on IEMOCAP.
EF-LSTM concatenates the multimodal inputs and uses that as input to a single
LSTM [26].
We also implement the Stacked, ( EF-SLSTM ) [22] Bidirectional ( EF-BLSTM ) [53] and Stacked Bidirectional ( EF-SBLSTM ) LSTMs.
The best model is reported as EF- (⋆) LSTM .
EFHCRF: (Hidden Conditional Random Field) [51] uses a HCRF to learn latent variables conditioned on the concatenated input.
We also implement the following variations: EF-LDHCRF:
(Latent
Discriminative HCRFs) [42], MV-HCRF (Multi-view HCRF) [56], MV-LDHCRF , EF-HSSHCRF (Hierarchical Sequence Summarization HCRF) [57] and MV-HSSHCRF .
The best performing early fusion model is reported as EF- (⋆) HCRF while the best multi-view model is reported as
MV- (⋆) HCRF .
For descriptions of the remaining baselines, we refer the reader to EF-HCRF [51],
EF/MV-LDHCRF [42], MV-HCRF [56], EF/MV-HSSHCRF [57], MV-LSTM [52], DF [44],
SAL-CNN [63], C-MKL [49], THMM [41], SVM [16, 46] and RF [8].
B Additional Results
Here we record the complete set of results for all the baseline models across all the datasets, tasks and metrics.
Table 1 summarizes the complete results for sentiment analysis on the CMU-MOSI dataset.
Table 2 presents the complete results for emotion recognition on the IEMOCAP dataset and Table 3 presents the complete results for personality
traits prediction on the POM dataset.
For experiments on the POM dataset we report additional results on MAE and correlation metrics for personality
traits regression.
We achieve signiﬁcant improvement over state-of-the-art multi-view and dataset speciﬁc approaches across all these datasets, highlighting the RMFN’s capability in analyzing sentiment, emotions and speaker traits from human multimodal language.
C Visualizations
Using an attention assignment mechanism during the HIGHLIGHT process gives interpretability to the model since it allows us to visualize the attended multimodal signals at each stage and time step (see Figure 3).
Using RMFN trained on the CMU-MOSI dataset, we plot the attention weights across the multistage fusion process for three videos in CMU-MOSI.
Based on these visualizations we ﬁrst draw the following general observations on spatiotemporal fusion:
Across stages (Spatio): Attention weights change their behaviors across the multiple stages of fusion.
Some features are highlighted by earlier stages while other features are used in later stages.
This supports our hypothesis that RMFN learns to specialize in different stages of the spatiotemporal fusion process.
Across time (Temporal): Attention weights vary over time and adapt to the multimodal inputs.
We observe that the attention weights are similar if the input contains no new information.
As soon as new multimodal information comes in, the highlighting mechanism in RMFN adapts to these new inputs.
Priors: Based on the distribution of attention weights, we observe that the language and acoustic modalities seem the most commonly highlighted.
This represents a prior over the expression of sentiment in human multimodal language and is closely related to the strong connections between language and speech in human communication [33].
Inactivity: Some attention coefﬁcients are not active (always orange) throughout time.
We hypothesize that these corresponding dimensions carry only intra-modal dynamics and are not involved in the formation of cross-modal interactions.
C .1 Qualitative Analysis
In addition to the general observations above, Figure 3 shows three examples where multistage fusion learns cross-modal representations across three different scenarios. low high
I ℎ "# ℎ "% ℎ "& (elongation) stages stages (emphasis) thought it was fun Language Visual Acoustic He delivers a lot of intensity (emphasis) 𝟏𝟐𝟑 stages stages
It doesn’tgive any insight or help stages stages ℎ "# ℎ "% ℎ "&
Language
Visual
{+Acoustic (soft) (emphasis) (disappointed) (a) (b) (c) (smile) (smile) 
+}
𝟏𝟐𝟑  t =t =t =
t = 1 t =t =
Synchronized Interactions: In Figure 3(a), the language features are highlighted corresponding to the utterance of the word “fun” that is highly indicative of sentiment ( t = 5 ).
This sudden change is also accompanied by a synchronized highlighting of the acoustic features.
We also notice that the highlighting of the acoustic features lasts longer across the 3 stages since it may take multiple stages to interpret all the new acoustic behaviors (elongated tone of voice and phonological emphasis).
Asynchronous Trimodal Interactions: In Figure 3(b), the language modality displays ambiguous sentiment: “delivers a lot of intensity” can be inferred as both positive or negative.
We observe that the circled attention units in the visual and acoustic features correspond to the asynchronous presence of a smile ( t = 2 ∶ 5 ) and phonological emphasis ( t = 3 ) respectively.
These nonverbal behaviors resolve ambiguity in language and result in an overall display of positive sentiment.
We further note the coupling of attention weights that highlight the language, visual and acoustic features across stages ( t = 3 ∶ 5 ), further emphasizing the coordination of all three modalities during multistage fusion despite their asynchronous occurrences.
Bimodal Interactions: In Figure 3(c), the language modality is better interpreted in the context of acoustic behaviors.
The disappointed tone and soft voice provide the nonverbal information useful for sentiment inference.
This example highlights the bimodal interactions ( t = 4 ∶ 7 ) in alternating stages: the acoustic features are highlighted more in earlier stages while the language features are highlighted increasingly in later stages. [Table]
RMFN outperforms the current state-of-the-art across all evaluation metrics.
[Table]
RMFN achieves state-of-the-art or competitive performance across all evaluation metrics.
[Table]