MTAG : Modal-Temporal Attention Graph for Unaligned Human Multimodal Language Sequences (2021)

Abstract: Human communication is multimodal in nature; it is through multiple modalities such as language, voice, and facial expressions, that opinions and emotions are expressed. Data in this domain exhibits complex multi-relational and temporal interactions. Learning from this data is a fundamentally challenging research problem. In this paper, we propose Modal-Temporal Attention Graph (MTAG). MTAG is an interpretable graph-based neural model that provides a suitable framework for analyzing multimodal sequential data. We first introduce a procedure to convert unaligned multimodal sequence data into a graph with heterogeneous nodes and edges that captures the rich interactions across modalities and through time. Then, a novel graph fusion operation, called MTAG fusion, along with a dynamic pruning and read-out technique, is designed to efficiently process this modal-temporal graph and capture various interactions. By learning to focus only on the important interactions within the graph, MTAG achieves state-of-the-art performance on multimodal sentiment analysis and emotion recognition benchmarks, while utilizing significantly fewer model parameters.

## introduction with recent advances in machine learning research, analysis of multimodal sequential data has become increasingly prominent. at the core of modeling this form of data, there are the fundamental research challenges of fusion and alignment. fusion is the process of blending information from multiple modalities. it is usually preceded by alignment, which is the process of finding temporal relations between the modalities. an important research area that exhibits this form of data is multimodal language analysis, where sequential modalities of * equal contribution 1 code is available at https://github.com/ jedyang97/mtag . oh not why significantly raises voice when saying "oh why not?!" emphasizes on "really enjoyed" text order figure 1: example visualization of tri-modal modal-temporal attention learned by our proposed model. each circle represents a node from video/text/audio modalities, and the blue lines denote the learned attention weights (i.e. the thicker and darker a blue line is, the larger the attention weight). we observe high intensities between semantically correlated graph entities, such as "really enjoy" and the raise in eyebrow, which indicate positive sentiment. note that our graphbased model learns multimodal interactions without prior alignment, and captures diverse types of interactions across multiple modalities all the same time. edge types are not shown for visual clarity. ## mtag in this section, we describe our proposed framework: modal temporal attention graph (mtag) for unaligned multimodal language sequences. we describe how we formulate the multimodal data into a graph g(v, e), and the mtag fusion operation that operates on g. in essence, our graph formulation by design alleviates the need for any hard alignments, and combined with mtag fusion, allows nodes from one modality to interact notation explanation v i node i e ij edge from v i to v j n i neighbor nodes incident into v i x i initial node feature for v i x i transformed node feature for v i π i edge temporal type for e ij m π i node type specific transformation matrix a φ ij ,τ ij edge type specific learnable attention vector β i,j ## experiments we empirically evaluate mtag model on two datasets: iemocap @xcite and cmu-mosi @xcite ; both are well-known datasets used by prior works @xcite @xcite to benchmark multimodal emotion recognition and sentiment analysis. ## ablation study we conduct ablation study using unalgined cmu-mosi dataset. mtag full model implements multimodal temporal edge types, adopts topk edge pruning that keeps edges with top 80% edge weights, and includes all three modalities as its input. ## conclusion in this paper, we presented the modal-temporal attention graph (mtag). we showed that mtag is an interpretable model that is capable of both fusion and alignment. it achieves similar to sota performance on two publicly available datasets for emotion recognition and sentiment analysis while utilizing substantially lower number of parameters than a transformer-based model such as mult. for node v i in π i , in order to determine the "present" nodes v present in a different modality, we draw an analogy from 1d convolution operation. we are given two sequences of different lengths, and we can treat the longer sequence as input and shorter sequence as output to a conv1d operation. our goal is to find a feasible stride and kernel size that aligns the input and output. the kernel size defines how many nodes from the longer sequence to be aligned as "present" to each node from the shorter sequence. the stride size defines how far away such alignments should spread in time. we a.3 model efficiency number of parameters. we compare the parameter efficiency of our model against the sota model, the multimodal transformer (mult) @xcite . we first look at the total number of parameters used by the two models. convergence. figure 7 gives a comparison between the convergence speed of our model and mult. both models are trained with @xmath0, with the default (best) hyperparameters used for both models. we use the unalignd cmu-mosi for this study. we observe mtag converges much faster at epoch 12, comparing with mult at epoch 17. in addition, we see that our validation mae curve on the unaligned mosi validation set goes consistently below mult's curve. this faster convergence performance could be explained by the small amount of parameters mtag model # parameters mult (previous sota) 2,240,921 mtag (ours) 142,363