Multimodal Phased Transformer for Sentiment Analysis (2021)

Abstract: Multimodal Transformers achieve superior performance in multimodal learning tasks. However, the quadratic complexity of the self-attention mechanism in Transformers limits their deployment in low-resource devices and makes their inference and training computationally expensive. We propose multimodal Sparse Phased Transformer (SPT) to alleviate the problem of self-attention complexity and memory footprint. SPT uses a sampling function to generate a sparse attention matrix and compress a long sequence to a shorter sequence of hidden states. SPT concurrently captures interactions between the hidden states of different modalities at every layer. To further improve the efficiency of our method, we use Layer-wise parameter sharing and Factorized Co-Attention that share parameters between Cross Attention Blocks, with minimal impact on task performance. We evaluate our model with three sentiment analysis datasets and achieve comparable or superior performance compared with the existing methods, with a 90% reduction in the number of parameters. We conclude that (SPT) along with parameter sharing can capture multimodal interactions with reduced model size and improved sample efficiency.

## introduction the objective of multimodal sentiment analysis is to identify the polarity of one's attitude toward an entity through multimodal inputs such as audio, video, and text. for many applications such as personal assistants, social robots and virtual agents, the efficiency and scalability of a method are as important as accuracy. such applications can have limited computational resources or large-scale deployment requirements. multimodal understanding of constructs, such as sentiment, requires capturing information available in every modality in addi- * equal contribution tion to their potential interactions, e.g., an exaggerated smile combined with negative sentiment in language might signal irony. modeling these interactions efficiently is still an open challenge @xcite . some work on this topic use the different networks for each modality followed by fusion methods, like concatenation @xcite @xcite and outer-product @xcite , to model the interaction of multimodal representations which largely increases the dimensionality of representations thus increasing computational cost. @xcite rely on large pre-trained models, like bert @xcite and xlnet @xcite . the computational cost of such approaches is high due to the over-parameterization of the models. and transformer-based methods @xcite suffer from quadratic complexity of self-attention. the existing multimodal sentiment analysis datasets are rather small due to the laborious labeling process. the development of the existing datasets, such as @xcite , involves data curation and annotation by multiple annotators. the limited dataset size raises the risk of over-fitting for over-parameterized models which motivated building models that can be trained with fewer data. recent work, such as @xcite @xcite , improve the efficiency of transformers through sparse-attention. compared to the full attention that calculates attention for all pairs of input elements, sparse attention only computes attention for a subset of element pairs. as a consequence, each element from one sequence attends only to a limited number of elements in the source sequence. other work reduce attention matrix size by iterative processing only a shorter segment of the original long sequence at a time @xcite . in this paper, we propose a sparse phased atten-tion (sp) mechanism that uses a sampling function to compress a longer input sequence to a shorted sequence of hidden states and improve the efficiency of the attention computation. multimodal spt can capture multimodal interactions through a "concurrent" network structure rather than a "serial" structure of previous work @xcite @xcite . we improve the efficiency of spt through parameter sharing and a factorized co-attention. we perform extensive experiments evaluating the performance of spt on multimodal sentiment analysis and an ablation study on its structure, sampling function, parameter sharing approaches, and sp-block. we compare our method with the stateof-the-art efficient transformers, i.e., performer @xcite . our experimental results show that our model is able to achieve minimal or no performance loss with a significant reduction in model size. other efficient transformerbased approaches with linear efficiency result in a larger degradation of the performance. in comparison with the previous work on multimodal sentiment analysis, we reduce memory use in addition to training and inference time, with complexity decreasing from quadratic to linear, with only 10% of the parameters. the main contributions of this work are as follows. the code and data are publicly available in https://github.com/chengjunyan1/ sp-transformer . ## related work sentiment analysis multimodal sentiment analysis involves leveraging the information from multiple modalities, e.g., text and vision, to recognize the polarity of expressed sentiment. most of the existing work focuses on recognizing sentiment expressed in video recordings from social media reviewing products or movies @xcite @xcite . recent work on multimodal sentiment analysis has focused on the application of transformer architectures. @xcite introduces pairwise cross-modal attention on transformers for multimodal sentiment analysis on audio, video, and text. our model's architecture follows a similar design that first encodes unimodal inputs, then models cross-modal interactions and finally fuses multimodal information. the main difference is that our model enables a concurrent way to implements those steps. @xcite project multimodal input into modality-invariant and modalityspecific spaces, and use a transformer encoder on the concatenated projected representations. rahman et al. (2020) use pre-trained transformers like bert @xcite , xlnet @xcite on a large corpus and perform transferlearning for multimodal sentiment analysis. multimodal learning previous work use recurrent neural networks (rnn) or convolutional neural networks (cnn) on each modality and perform model-based, e.g., kernel-based fusion, with graphical models, and neural networks, and modelagnostic fusion, e.g., early, late or hybrid @xcite . @xcite fuse multimodal representations with a gated modalitymixing network, that model the fine-grained structure of non-verbal subword sequences and a multimodal shifting mechanism, that dynamically shifts word representations based on non-verbal cues. @xcite trains a sequence-to-sequence rnn to jointly perform inter-modality translations and sentiment analysis, the encoder output is a joint multimodal representation that is used for sentiment detection. tensor fusion networks (tfn) use the outer product of representations for each modality concatenated with a constant value of "1" to generate a joint representation @xcite . @xcite propose to decompose the weight from the fusion layers into low-rank factors to reduce the large number of parameters in tfn. @xcite use a system of lstms to learn modal-specific interactions, learn the crossmodal interactions with an attention mechanism and finally apply a multi-view gated memory that fuses the multimodal representations through time. @xcite project multimodal features to fine-granularity and coarse-granularity "spaces" through multi-layer-perceptron (mlp) networks. audio and video are aligned with text features in coarse-grained space, while audio and video features are aligned in fine-grained space. unlike the previous methods that directly apply transformations on the multimodal inputs, we use a small sequence of hidden states to capture the features from multimodal inputs thus preserving raw input information while improving efficiency. in the most similar work to ours, @xcite distill information from an input sequence to a fixed length of hidden states with an autoregressive transformer. efficient transformers one of the drawbacks of the transformer architecture is the computational cost and the memory footprint of the selfattention mechanism. a number of efforts have been made to make transformers more efficient @xcite @xcite @xcite . such work use sparse attention to selectively attend to pairs of elements and lead to a reduction in memory use and computational complexity of the attention mechanism. a notable example, performer, @xcite , improve the efficiency of the attention computation through "unbiased" and low-rank approximation of the attention matrix. there are other attempts for reducing the sequence length to improve computational efficiency. @xcite introduce "segment-level recurrence" that recurrently use the previous segment and current segment. rae et al. (2020) extend this idea and compresses multiple previous segments into memory vectors. the hidden states in our method are based on a similar idea that caching the information from the input sequence in a shorter sequence can improve efficiency. we also applied the idea of sparse attention to achieve further improvements. ## method the proposed method is an extension of the transformer architecture for improved efficiency in multimodal learning. in this section, we will introduce the basic building block of our model, i.e., sparse-phased-block (sp), and show how we extend sp in the context of multimodal learning to define sp-transformer with input attention, cross attention and self attention sub-layers. we also leverage factorized co-attention and parameter sharing for further optimizations of sp-transformer. each sp-block uses a sampling function to generate sparse attention patterns that guide a sequence to selectively compress information from elements of another sequence as shown in figure 2 . we use sp-block to build our multimodal sp-transformer, in each layer, we first use an sp-block that executes input attention for each modality and compress information from each unimodal input sequence to the hidden states. the hidden state sequences for each pair of two modalities interact through cross attention by a co-sp-block using factorized co-attention. finally, the cross-modal information for each modality is fused by summation and distilled with self attention using an sp-block. the full model is presented in figure 3 . sparse phased block (sp-block) accept a sequence x and a sequence of hidden state h as input and use sampling function ψ to compress x to h by selective attention between the two. ψ create figure 2 : sp-block samples from a sequence x with hidden state h and with a fixed sampling function with sampling length @xmath0. the hidden states of h attend at most to five input states of x. in a mixed sampling, the sampling interval for each hidden state would shift with a distance that depends on the layer (sliding), a periodic function with its index as parameter, and random perturbation, which makes it sample in a "dynamic" way as opposed to a static (fixed) sampling. ## experiment we introduce the experimental setup, baseline methods and datasets. we present the results and additional evaluations through an ablation study. ## conclusions in this paper, we propose a multimodal sp-transformer that uses a sequence of hidden states to sample from longer multimodal input sequences. compared with the previous transformer-based models, our model has a reduced computational complexity through sparse attention. the concurrent structure also enables more effective capturing of the multimodal interaction, resulting in higher performance. the optimization through parameter sharing patterns leads to a significantly lighter model, with a lower number of parameters and improved sampled efficiency. as a result, the proposed model's performance is superior or comparable to the existing methods at a lower computational and memory cost. our experiments show that our method has a good balance between accuracy and efficiency and has the potential to be deployed in real-world multimodal applications.