Lightweight Models for Multimodal Sequential Data (2021)

Abstract: Human language encompasses more than just text; it also conveys emotions through tone and gestures. We present a case study of three simple and efficient Transformer-based architectures for predicting sentiment and emotion in multimodal data. The Late Fusion model merges unimodal features to create a multimodal feature sequence, the Round Robin model iteratively combines bimodal features using cross-modal attention, and the Hybrid Fusion model combines trimodal and unimodal features together to form a final feature sequence for predicting sentiment. Our experiments show that our small models are effective and outperform the publicly released versions of much larger, state-of-the-art multimodal sentiment analysis systems.

## introduction language is composed of three different modalities: text, audio, and video. these three modalities together make it easier for humans to convey emotion and sentiment. thus, a machine learning model for sentiment analysis needs to learn the features and interactions of all three modalities. for example, a frown in the video can alter the emotion expressed in the text transcript, or audio intensity can help determine if a speaker is getting agitated. multimodal learning has recently received a good deal of attention from the natural language processing community @xcite @xcite . the transformer network @xcite , with its self-attention modules, has achieved strong performance in multimodal learning; attention provides a natural way to model the relationship between pairs of modalities. in this work we investigate three small, lightweight, transformer-based architectures for multimodal sentiment analysis and emotion recog-nition. our first model is an implementation of the late fusion model commonly used as a baseline system, which assigns individual transformer blocks to each of the three modalities for feature extraction and then combines these unimodal features to learn cross-modal interactions. the second model is an implementation of the round robin approach; the model generates bimodal features by using cross-modal attention to combine pairs of modalities, one pair at a time. our last model is a hybrid of the early and late fusion schemes. this model merges the features extracted using a late fusion pipeline, as well as those from an early fusion pipeline, where the three modalities are concatenated and passed through a single transformer block for feature extraction;. we present experiments using these three models on three multimodal datasets: iemocap @xcite , an emotion recognition dataset, and cmu-mosi @xcite and cmu-mosei @xcite , two multimodal sentiment analysis datasets. our results show that our small models are competitive with state-of-the-art models that use much more complex architectures. our main contributions are as follows: ## related work we do not give an exhaustive list of prior work in multimodal sentiment analysis, but focus on recent neural approaches that achieved state-of-the-art performance at their times of publication. ## experiments we train our models on a single nvidia k80 gpu. we tune hyperparameter values for our model using the validation sets provided by our evaluation datasets; we achieve the best validation performance using 8 attention blocks per transformer, each with 5 attention heads, and a hidden size was set to 40. the dropout rate was set to 0.15; the best learning rate for iemocap was 0.02, while for cmu-mosi and cmu-mosei it was 0.01, with batch sizes of 32, 128, and 40, respectively. ## results and discussion we present the results of our model compared to the reported results of our baseline models in tables 1 , 2 , and 3. the best-performing mult and fmt models are extremely dense, with around 15 and 77 million parameters, respectively. in contrast, our models have between 7-9 million trainable parameters, depending on the architecture; despite using about half as many parameters as mult, we see that our models produce comparable results. we perform fairly well on iemocap, which has around 2717 training samples; we achieve scores around 1-2% below the best-performing model, fmt. on the tiny cmu-mosi dataset, which has just 1284 training samples, our hybrid fusion and late fusion models give state of the art results on seven-way and binary accuracy, respectively. the cmu-mosei dataset is much larger than iemocap and cmu-mosi, with close to 16265 training samples. our models perform the weakest on this dataset, falling short of the state of the art models by around 2-3%, suggesting that our models may be too small to learn the entire distribution. neither marn @xcite nor fmt @xcite reports results on cmu-mosei, so they are omitted from we also experiment with the open source code available for mult and fmt (denoted by *). using the hyperparameter settings provided 2 , we were nevertheless unable to match those systems' reported performance, possibly due to differences 2 batch size for fmt* is not given; we use 20, the default. ## conclusion we have presented three lightweight architectures for multimodal sentiment analysis and emotion recognition. the late fusion model merges unimodal features, the round robin model iteratively combines bimodal features, and the hybrid early-late fusion model combines early-fusion trimodal and late-fusion unimodal features. our proposed models are much smaller in size compared to existing state-of-the-art models; they are able to attain new state-of-the-art scores on the cmu-mosi and cmu-mosei datasets on two metrics, while remaining competitive on the others. further, our experiments analyzing the relative contribution of modalities and architecture components in our models suggest new directions for developing multimodal systems. we hope that our simple architectures for sentiment and emotion detection, currently the fastest and best-performing publicly available system, as well as the insights revealed in our experimental results, can be useful for further research in the field.