Human communication is multimodal in nature; it is through multiple modalities , i.e., language, voice, and facial expressions, that opinions and emotions are expressed. Data in this domain exhibits complex multi-relational and temporal interactions. Learning from this data is a fundamentally challenging research problem. In this paper, we propose Multimodal Temporal Graph Attention Networks (MTGAT). MTGAT is an interpretable graph-based neural model that provides a suitable framework for analyzing this type of multimodal sequential data. We first introduce a procedure to convert unaligned multimodal sequence data into a graph with heterogeneous nodes and edges that captures the rich interactions between different modalities through time. Then, a novel graph  operation, called Multimodal Temporal Graph Attention , along with a dynamic pruning and read-out technique  is designed to efficiently process this multimodal temporal graph . By learning to focus only on the important interactions within the graph, our MTGAT is able to achieve state-of-the-art performance on multimodal sentiment analysis and emotion recognition benchmarks including IEMOCAP and CMU-MOSI , while utilizing significantly fewer computations .
Human communication is multimodal in nature; it is through multiple modalities such as language, voice, and facial expressions, that opinions and emotions are expressed. Data in this domain exhibits complex multi-relational and temporal interactions. Learning from this data is a fundamentally challenging research problem. In this paper, we propose Modal-Temporal Attention Graph (MTAG). MTAG is an interpretable graph-based neural model that provides a suitable framework for analyzing multimodal sequential data. We first introduce a procedure to convert unaligned multimodal sequence data into a graph with heterogeneous nodes and edges that captures the rich interactions across modalities and through time. Then, a novel graph fusion operation, called MTAG fusion , along with a dynamic pruning and read-out technique , is designed to efficiently process this modal-temporal graph and capture various interactions . By learning to focus only on the important interactions within the graph, MTAG achieves state-of-the-art performance on multimodal sentiment analysis and emotion recognition benchmarks , while utilizing significantly fewer model parameters .
R
, i.e.,
such as
R
Multimodal Temporal Graph Attention Networks (MTGAT). MTGAT
Modal-Temporal Attention Graph (MTAG). MTAG
D
this type of
R
between different modalities
across modalities and
A
fusion
R
Multimodal Temporal Graph Attention
MTAG fusion
A
,
R
multimodal temporal graph
modal-temporal graph and capture various interactions
R
our MTGAT is able to achieve
MTAG achieves
D
including IEMOCAP and CMU-MOSI
R
computations
model parameters