A Transformer-based joint-encoding for Emotion Recognition and Sentiment Analysis (2020)

Abstract: Understanding expressed sentiment and emotions are two crucial factors in human multimodal language. This paper describes a Transformer-based joint-encoding (TBJE) for the task of Emotion Recognition and Sentiment Analysis. In addition to use the Transformer architecture, our approach relies on a modular co-attention and a glimpse layer to jointly encode one or more modalities. The proposed solution has also been submitted to the ACL20: Second Grand-Challenge on Multimodal Language to be evaluated on the CMU-MOSEI dataset. The code to replicate the presented experiments is open-source .

## introduction predicting affective states from multimedia is a challenging task. emotion recognition task has existed working on different types of signals, typically audio, video and text. deep learning techniques allow the development of novel paradigms to use these different signals in one model to leverage joint information extraction from different sources. this paper aims to bring a solution based on ideas taken from machine translation @xcite ) and visual question answering (modular co-attention, @xcite ). our contribution is not only very computationally efficient, it is also a viable solution for sentiment analysis and emotion recognition. our results can compare with, and sometimes surpass, the current state-of-the-art for both tasks on the cmu-mosei dataset @xcite . this paper is structured as follows: first, in section 2, we quickly go over the related work that have been evaluated on the mosei dataset, we 1 https://github.com/jbdel/mosei_umons then proceed to describe our model in section 3, we then explain how we extract our modality features from raw videos in section 4 and finally, we present the dataset used for our experiments and their respective results in section 5 and 6. ## related work over the years, many creative solutions have been proposed by the research community in the field of sentiment analysis and emotion recognition. in this section, we proceed to describe different models that have been evaluated on the cmu-mosei dataset. to the best of our knowledge, none of these ideas uses a tansformer-based solution. the memory fusion network @xcite ) synchronizes multimodal sequences using a multi-view gated memory that stores intraview and cross-view interactions through time. graph-mfn @xcite ) consists of a dynamic fusion graph (dfg) built upon mfn. dfg is a fusion technique that tackles the nature of cross-modal dynamics in multimodal language. the fusion is a network that learns to models the n-modal interactions and can dynamically alter its structure to choose the proper fusion graph based on the importance of each n-modal dynamics during inference. @xcite use tensor fusion network (tfn), i.e. an outer product of the modalities. this operation can be performed either on a whole sequence or frame by frame. the first one lead to an exponential increase of the feature space when modalities are added that is computationally ex-pensive. the second approach was thus preferred. they showed an improvement over an early fusion baseline. recently, shenoy and sardana (2020) propose a solution based on a context-aware rnn, multilogue-net, for multi-modal emotion detection and sentiment analysis in conversation. ## model this section aims to describe the two model variants evaluated in our experiment: a monomodal variant and a multimodal variant. the monomodal variant is used to classify emotions and sentiments based solely on l (linguistic), on v (visual) or on a (acoustic). the multimodal version is used for any combination of modalities. our model is based on the transformer model @xcite , a new encoding architecture that fully eschews recurrence for sequence encoding and instead relies entirely on an attention mechanism and feed-forward neural networks (ffn) to draw global dependencies between input and output. the transformer allows for significantly more parallelization compared to the recurrent neural network (rnn) that generates a sequence of hidden states h t , as a function of the previous hidden state h t-1 and the input for position t. ## feature extractions this section aims to explain how we pre-compute the features for each modality. these features are the inputs of the transformer blocks. note that the features extraction is done independently for each example of the dataset. ## dataset we test our joint-encoding solution on a novel dataset for multimodal sentiment and emotion recognition called cmu-multimodal opinion sentiment and emotion intensity (cmu-mosei, @xcite ). it consists of 23,453 annotated sentences from 1000 distinct speakers. each sentence is annotated for sentiment on a [-3,3] scale from highly negative (-3) to highly positive (+3) and for emotion by 6 classes : happiness, sadness, anger, fear, disgust, surprise. in the scope of our experiment, the emotions are either present or not present (binary classification), but two emotions can be present at the same time, making it a multi-label problem. the figure 3 shows the distribution of sentiment and emotions in cmu-mosei dataset. the distribution shows a natural skew towards more frequently used emotions. the most common category is happiness with more than 12,000 positive sample points. the least prevalent emotion is fear with almost 1900 positive sample. it also shows a slight shift in favor of positive sentiment. ## experiments in this section, we report the results of our model variants described in section 3. we first explain our experimental setting. ## discussions we presented a computationally efficient and robust model for sentiment analysis and emotion recognition evaluated on cmu-mosei. though we showed strong results on accuracy, we can see that there is still a lot of room for improvement on the f1-scores, especially for the emotion classes that are less present in the dataset. to the best of our knowledge, the results presented by our transformer-based joint-encoding are the strongest scores for the sentiment task on the dataset. the following list identifies other features we computed as input for our model that lead to weaker performances: