Multi-modal Multi-label Emotion Detection with Modality and Label Dependence (2020)

Abstract: As an important research issue in the natural language processing community, multi-label emotion detection has been drawing more and more attention in the last few years. However, almost all existing studies focus on one modality (e.g., textual modality). In this paper, we focus on multi-label emotion detection in a multi-modal scenario. In this scenario, we need to consider both the dependence among different labels (label dependence) and the dependence between each predicting label and different modalities (modality dependence). Particularly, we propose a multi-modal sequence-to-set approach to effectively model both kinds of dependence in multi-modal multi-label emotion detection. The detailed evaluation demonstrates the effectiveness of our approach.

## introduction emotion detection is to predict emotion categories, such as angry, happy, and surprise, expressed by an utterance of a speaker and has largely encompassed a variety of applications, such as online chatting @xcite , news analysis @xcite and dialogue systems @xcite . over the last few years, there has been a substantial body of research on emotion detection @xcite @xcite , where a considerable amount of work has focused on multi-label emotion detection @xcite @xcite . basically, emotion detection is a multi-label classification problem since one utterance naturally tends to involve more than one emotion category. however, classifying instances with multiple possible categories is sometimes much more difficult than classifying instances with a single label. one main challenge is how to model the label dependence in the classification approach. for example, in the utterance as shown in figure 1 , both the sad and disgust emotions are more likely to exist, rather than the conflicting emotions of sad and happy. recent studies, such as @xcite and @xcite , have begun to address this challenge. however, almost all existing studies in multilabel emotion detection focus on one modality (e.g., textual modality). only very recently, the research community has become increasingly aware of the need on multi-modal emotion detection @xcite due to its wide potential applications, e.g., with the massively growing importance of analyzing conversations in speech @xcite and video @xcite . in this study, we aim to tackle multi-modal multi-label emotion detection. compared with single modality, multimodal multi-label emotion detection needs to well model the contribution of different modalities for each label since each modality has a different impact on expressing emotion. for example, from the textual modality as shown in figure 1 , while we may only infer the sad emotion, we are more likely to infer the disgust emotion instead from the visual modality. meanwhile, the acoustic modality may not help label prediction in this case. therefore, besides the label dependence, it is important and challenging to effectively model another dependence, namely the modality dependence. in this paper, we address the above challenges in multi-modal multi-label emotion detection by proposing a multi-modal seq2set (mms2s) approach to model both the modality and label dependence simultaneously. specifically, while the conditional generation framework naturally models the label dependence by predicting the next emotion label upon other potential labels, we propose multi-head soft modality attention at each predicting step inside the emotion decoder to capture the modality dependence. first, we adopt three singlemodal encoders based on transformer to capture the single-modal characteristics of the textual, visual and acoustic modalities, respectively. then, we make the given emotion representation attend to three intra-modal sequences from encoders inside the emotion decoder and leverage multi-head soft modality attention to control the different contributions of different modalities for each potential emotion prediction. finally, we train our proposed model by maximizing the probabilities of top k sequences and predict all potential emotion labels by finding the most likely emotion label set. systematical evaluation on a public multi-modal multi-label emotion dataset, i.e., cmu-mosei, shows that our approach significantly outperforms several state-of-the-art baselines. ## related work as an interdisciplinary research field, emotion detection has been drawing more and more attention in both natural language processing and multimodal communication @xcite . in the nlp community, almost all existing studies of multi-label emotion detection rely on special knowledge of emotion, such as context information @xcite , cross-domain transferring @xcite and external resource @xcite . in fact, when there is no special knowledge @xcite , it can be normally handled by multi-label text classification approaches. in the multi-modal community, related studies normally focus on single-label emotion task and the studies for multi-label emotion task are much less and limited to be transformed to multiple binary clas-sification @xcite @xcite . in the following, we give an overview of multi-label emotion/text classification and multi-modal emotion detection. multi-label emotion/text classification. recent studies normally cast multi-label emotion detection task as a classification problem and leverage the special knowledge as auxiliary information @xcite . these approaches may not be easily extended to those tasks without external knowledge. at this time, the multi-label text classification approaches can be quickly applied to emotion detection. there have been a large number of representative studies for that. @xcite leverage the pre-trained bert to perform multi-label emotion task and @xcite propose an attention-based classifier that predicts multiple emotions of a given sentence. more recently, @xcite propose a sequence generation model and @xcite leverage a reinforced approach to find a better sequence than a baseline sequence, but it still relies on the pretrained seq2seq model with a pre-defined order of ground-truth. different from above studies, we focus on multilabel emotion detection in a multi-modal scenario by considering the modality dependence besides the label dependence. to the best of our knowledge, this is the first attempt to perform multi-label emotion detection in a multi-modal scenario. ## data pre-processing we extract low-level handcrafted features from three modalities. first of all, we align the three modalities by extracting the exact utterance timestamp of each word using p2fa @xcite . since words are considered as the basic semantic units of language, we use the interval duration of each word as a time-step. then, we calculate the expected video and audio features by taking the expectation of their feature values over the word time interval @xcite . on this basis, we process the information of the three modalities as follows. textual modality. the glove word embeddings @xcite are used to represent the words from manual transcripts. then, we get the text sequence visual modality. the library facet 1 is used to extract a set of visual features including facial action units, facial landmarks, head pose, gaze tracking and hog features @xcite to form a sequence of facial gesture throughout time. then, we get the visual sequence the covarep software @xcite is used to extract acoustic features including 12 mel-frequency cepstral coefficients (mfccs), pitch, voiced/unvoiced segmenting features @xcite , glottal source parameters @xcite , peak slope parameters and maxima dispersion quotients @xcite . then, we get the acoustic sequence ## experimentation in this section, we systematically evaluate our approach to multi-modal multi-label emotion detection. ## conclusion in this paper, we propose a multi-modal sequenceto-set approach to simultaneously handle the modality and label dependence in multi-modal multi-label emotion detection. our approach can not only model the dependence between each label and different modalities, but also model the dependence among multiple labels of a sample. the detailed evaluation demonstrates that our proposed model significantly outperforms several state-ofthe-art baselines. in our future work, we will extend our approach to more multi-modal multi-label scenarios, such as intention detection in video conversations and aspect analysis in multi-modal reviews. furthermore, we would like to investigate other approaches (e.g., graph-based neural network) to better model the modality and label dependence in multi-modal multi-label emotion detection.