Low Rank Fusion based Transformers for Multimodal Sequences (2020)

Abstract: Our senses individually work in a coordinated fashion to express our emotional intentions. In this work, we experiment with modeling modality-specific sensory signals to attend to our latent multimodal emotional intentions and vice versa expressed via low-rank multimodal fusion and multimodal transformers. The low-rank factorization of multimodal fusion amongst the modalities helps represent approximate multiplicative latent signal interactions. Motivated by the work of (CITATION) and (CITATION), we present our transformer-based cross-fusion architecture without any over-parameterization of the model. The low-rank fusion helps represent the latent signal interactions while the modality-specific attention helps focus on relevant parts of the signal. We present two methods for the Multimodal Sentiment and Emotion Recognition results on CMU-MOSEI, CMU-MOSI, and IEMOCAP datasets and show that our models have lesser parameters, train faster and perform comparably to many larger fusion-based architectures.

## introduction the field of emotion understanding involves computational study of subjective elements such as sentiments, opinions, attitudes, and emotions towards other objects or persons. subjectivity is an inherent part of emotion understanding that comes from the contextual nature of the natural phenomenon. defining the metrics and disentangling the objective assessment of the metrics from the subjective signal makes the field quite challenging and exciting. sentiments and emotions are attached to the language, audio and visual modalities at different rates of expression and granularity and are useful in deriving social, psychological and behavioral insights about various entities such as movies, products, people or organizations. emotions are defined as brief organically synchronized evaluations of major events whereas sentiments are considered as more enduring beliefs and dispositions towards objects or persons @xcite . the field of emotion understanding has rich literature with many interesting models of understanding @xcite @xcite . recent studies on tensor-based multimodal fusion explore regularizing tensor representations @xcite and polynomial tensor pooling @xcite . in this work, we combine ideas from @xcite and @xcite and explore the use of transformer @xcite based models for both aligned and unaligned signals without extensive over-parameterization of the models by using multiple modality-specific transformers. we utilize low rank matrix factorization (lmf) based fusion method for representing multimodal fusion of the modality-specific information. our main contributions can be summarized as follows: the ability to use unaligned sequences for modeling is advantageous since we rely on learning based methods instead of using methods that force the signal synchronization (requiring extra timing information) to mimic the coordinated nature of human multimodal language expression. the lmf method aims to capture all unimodal, bimodal and trimodal interactions amongst the modalities via approximate tensor fusion method. we develop and test our approaches on the cmu-mosi, cmu-mosei, and iemocap datasets as reported in @xcite . cmu multimodal opinion sentiment and emotion intensity (cmu-mosei) @xcite is a large dataset of multimodal sentiment analysis and emotion recognition on youtube video segments. the dataset contains more than 23,500 sentence utterance videos from more than 1000 online youtube speakers. the dataset has several interesting properties such as being gender balanced, containing various topics and monologue videos from people with different personality traits. the videos are manually transcribed and properly punctuated. since the dataset comprises of natural audio-visual opinionated expressions of the speakers, it provides an excellent test-bed for research in emotion and sentiment understanding. the videos are cut into continuous segments and the segments are annotated with 7 point scale sentiment labels and 4 point scale emotion categories corresponding to the ekman's 6 basic emotion classes @xcite . the opinionated expressions in the segments contain visual cues, audio variations in signal as well as textual expressions showing various subtle and non-obvious interactions across the modalities for both sentiment and emotion classification. cmu-mosi @xcite is a smaller dataset (2199 clips) of youtube videos with sentiment annotations. iemocap @xcite dataset consists of 10k videos with sentiment and emotion labels. we use the same setup as @xcite with 4 emotions (happy, sad, angry, neutral). in fig 1 , we illustrate our ideas by showing the fused signal representation attending to different parts of the unimodal sequences. there's no need to align the signals since the attention computation to different parts of the modalities acts as proxy to the multimodal sequence alignment. the fused signal is computed via low rank matrix factorization (lmf). the other model we propose uses a swapped configuration where the individual modalities attend to the fused signal in parallel. ## model description in this section, we describe our models and methods for low rank fusion of the modalities for use with multimodal transformers with cross-modal attention. ## experiments we present our early experiments to evaluate the performance of proposed models on the standard multimodal datasets used by @xcite foot_0 . we run our models on cmu-mosi, cmu-mosei, and iemocap datasets and present the results for the proposed lmf-mult and fusion-based-cm-attn-mult models. late fusion (lf) lstm is a common baseline for all datasets with reported results (pub) together with mult in @xcite . we include the results we obtain (our run) for the mult model for a direct comparison 2 . ## conclusion in this paper, we present our early investigations towards utilizing low rank representations of the multimodal sequences for usage in multimodal transformers with cross-modal attention to the fused signal or the modalities. our methods build up on the @xcite work and apply transformers to fused multimodal signal that aim to capture all inter-modal signals via the low rank matrix factorization @xcite . this method is applicable to both aligned and unaligned sequences. our methods train faster and use fewer parameters to learn classifiers with similar sota performance. we are exploring methods to compress the temporal sequences without using the hidden state context vectors from lstms that lose the temporal information. we recover the temporal information with a convolution layer. we believe these models can be deployed in low resource settings with further optimizations. we are also interested in using richer features for the audio, text, and the vision pipeline for other use-cases where we can utilize more resources.