Modulated Fusion using Transformer for Linguistic-Acoustic Emotion Recognition (2020)

Abstract: This paper aims to bring a new lightweight yet powerful solution for the task of Emotion Recognition and Sentiment Analysis. Our motivation is to propose two architectures based on Transformers and modulation that combine the linguistic and acoustic inputs from a wide range of datasets to challenge, and sometimes surpass, the state-of-the-art in the field. To demonstrate the efficiency of our models, we carefully evaluate their performances on the IEMOCAP, MOSI, MOSEI and MELD dataset. The experiments can be directly replicated and the code is fully open for future researches.

## introduction understanding expressed sentiment and emotions are two crucial factors in human multimodal language yet predicting affective states from multimedia remains a challenging task. the emotion recognition task has existed working on different types of signals, typically audio, video and text. deep learning techniques allow the development of novel paradigms to use these different signals in one model to leverage joint information extraction from different sources. these models usually require a fusion between modality, a crucial step to compute expressive multimodal features used by a classifier to output probabilities over the possible answers. in this paper, we propose an architecture based on two stages: an independent sequential stage based on lstm @xcite where modality features are computed separately, and a second hierarchical stage based on transformer @xcite where we iteratively compute and fuse new multimodal representations. this paper proposes the fusion between the acoustic and linguistic features through attention modulation @xcite and linear modulation @xcite , a powerful tool to shift and scale the feature maps of one modality given the representation of another. the association of this horizontal-vertical encoding and modulated fusion shows really strong results across a wide range of datasets for emotion recognition and sentiment analysis. in addition to the interesting performances it offers, the modulation requires no or very few learning parameters, making it fast and easy to train. the paper is structured as follows: we first present the different researches used for comparison in our experiments in section 2, we then briefly present the different datasets in section 3. then we carefully describe our sequential feature extraction based on lstm in section 4 and the two hierarchical modulated fusion model, the modulated attention transformer (mat) and modulated normalization transformer (mnt), in section 5. finally, we explain the experimental settings in section 6 and report the results of our model variants in section 7. ## related work the presented related work is used for comparison for our experiments. we proceed to briefly describe their proposed models. first, @xcite proposed a novel multimodal fusion technique called the dynamic fusion graph (dfg) to study the nature of crossmodal dynamics in multimodal language. dfg contains built-in efficacies that are directly related to how modalities interact. to capture the context of the conversation through all modalities, the current speaker and listener(s) in the conversation, and the relevance and relationship between the available modalities through an adequate fusion mechanism, shenoy and sardana (2020) proposed a recurrent neural network architecture that attempts to take into account all the mentioned drawbacks, and keeps track of the context of the conversation, interlocutor states, and the emotions conveyed by the speakers in the conversation. @xcite presented a model that learns robust joint representations by cyclic translations between modalities (mctn), that achieved strong results on various word-aligned human multimodal language tasks. @xcite proposed the recurrent attended variation embedding network (raven) to model expressive nonverbal representations by analyzing the fine-grained visual and acoustic patterns that occur during word segments. in addition, they seek to capture the dynamic nature of nonverbal intents by shifting word representations based on the accompanying nonverbal behaviors. but the related work that is probably the closest to ours is the multimodal transformer @xcite because they also use transformer based solutions to encode their modalities. nonetheless, we differ in many ways. first, their best solutions and scores reported are using visual support. secondly, they use transformer for cross-modality encoding for every modality pairs; this equals to 6 transformer modules (2 pairs per modality) while we only use two transformer (one per modality). finally, each output pairs is concatenated to go though a second stage of transformer encoding. we also differ on how the features are extracted: they base their solution on cnn while we use lstm. in this paper, it is important to note that we compare our results to their word-unaligned scores, as we do not use word-alignment either. ## feature extractions this sections aims to describe the linguistic and acoustic features used as the input of our proposed modulated fusions based on transformers. the extraction is performed independently for each sample of a dataset. we denote the extracted linguistic features as x and acoustic as y. in the end, both x and y have a size [t, c] where t is the temporal axis size and c the feature size. its important to note that t is different for each sample, while c is a hyper-parameter. ## models this section aims to describe the three model variants evaluated in our experiments. first, we describe the projection (p) of the features extracted in section 4 over emotion and sentiment classes without using any transformer. this corresponds to the baseline for our experiments. secondly, we present the naive transformer (nt) model, a transformerbased encoding where the inputs are encoded separately, the linguistic and acoustic features do not interact with each other: there is no modulated fusion. finally, we present the two highlights of the paper, the modulated attention transformer (mat) and the modulated normalization transformer (mnt), two solutions where the encoded linguistic representation modulates the entire process of the acoustic encoding. ## experimental settings we train our models using the adam optimizer (kingma and ba, 2014) with a learning rate of 1e -4 and a mini-batch size of 32. if the accuracy score on the validation set does not increase for a given epoch, we apply a learning-rate decay of factor 0.5. we decay our learning rate up to 2 times. afterwards, we use an early-stop of 10 epochs on accuracy. results presented in this paper are from the averaged predictions of at most 10 models. unless stated otherwise, the lstm size c (and therefore the transformer size) is 512. we use @xmath0. we use 8 multi-heads regardless of the models or the modality encoded. the size c of the transformer mlp is set at 2048. we apply dropout of 0.1 on the output of each block iteration, and 0.5 on the input (x + y) of the projection layer (equation 2). ## results we present the results on four sentiment and emotion recognition datasets: iemocap, mosei, mosi and meld. for each dataset, the results are presented in terms of the popular metrics used for the dataset. most of the time, f1-score is used, and sometimes the weighted f1-scores to take into account the imbalance between emotion or sentiment classes. iemocap we first compare the precision, recall and unweighted f1-scores of our two model variants on iemocap in if we compare the f1-score per class ( even if the dataset is a bit unbalanced between the binary answers (positive and negative), weighting the loss accordingly did not improve the results. it shows that our model variants manage to efficiently discriminate between both classes. mosei mosei is a relatively large-scale dataset. we expect to see a more noticeable difference of score between our modulated transformer variants and the naive transformer and projection baselines. for the emotion task in multilogue still shows strong results in the happy and angry category, two important classes of the mosei dataset as they have the biggest support (respectively 2505 and 1071 samples over 6336 in the test-set). for binary sentiment classification ( ## further analysis a few supplementary comments can be made about the results. first, we notice that the hierarchical structure of the network brought by the transformers did bring improvements across all datasets. indeed, even the nt model does bring significant performances boost compared to the p model that only consists of an lstm and the projection layer. a very nice property of our solutions is that few tranformers layers are required to be the found settings. it usually varies from 2 to 4 layers, allowing our solutions to converge very rapidly. mosei params s/epoch epoch/c p 9.8 m 10 2 nt @xmath1.9 m 26 6 nt @xmath2 the difference between mat and mnt variant is slim, but it seems that mat is more suitable for the binary sentiment classification. the computed alignment by the modulated attention of the linguistic and acoustic modality proves to be an acceptable solution for 2-class problem, but seems to fall short for more nuanced classification such as multi-class emotion recognition. mnt seems more suitable for that task, as shown for mosei and meld. a potential issue for mat is that we work with shallow architectures (@xmath3. in the scope of the dataset presented, we have not enough samples to train such architectures. it is possible that mnt adjust better with shallow layers because it can modulate entire feature maps twice per blocks. ## conclusions in this paper, we propose two different architectures, mat (modulated attention transformer) and mnt (modulated normalization transformer), for the task of emotion recognition and sentiment analysis. they are based on transformers and use two modalities: linguistic and acoustic. the performance of our methods were thoroughly studied by comparison with a naive transformer baseline and the most relevant related works on several datasets suited for our experiments. we showed that our transformer baseline encoding separately both modalities already performs well compared to state-of-the-art. the solutions including modulation of one modality from the other show a higher performance. overall, the architectures offer an efficient, lightweight and scalable solution that challenges, and sometimes surpasses, the previous works in the field.