C o MPM : Context Modeling with Speaker’s Pre-trained Memory Tracking for Emotion Recognition in Conversation (2022)

Abstract: As the use of interactive machines grow, the task of Emotion Recognition in Conversation (ERC) became more important. If the machine-generated sentences reflect emotion, more human-like sympathetic conversations are possible. Since emotion recognition in conversation is inaccurate if the previous utterances are not taken into account, many studies reflect the dialogue context to improve the performances. Many recent approaches show performance improvement by combining knowledge into modules learned from external structured data. However, structured data is difficult to access in non-English languages, making it difficult to extend to other languages. Therefore, we extract the pre-trained memory using the pre-trained language model as an extractor of external knowledge. We introduce CoMPM, which combines the speaker’s pre-trained memory with the context model, and find that the pre-trained memory significantly improves the performance of the context model. CoMPM achieves the first or second performance on all data and is state-of-the-art among systems that do not leverage structured data. In addition, our method shows that it can be extended to other languages because structured knowledge is not required, unlike previous methods. Our code is available on github .

## introduction as the number of applications such as interactive chatbots or social media that are used by many users has recently increased dramatically, emotion recognition in conversation (erc) plays a more important role in natural language processing, and as a proof, a lot of research @xcite @xcite has been conducted on the task. the erc module increases the quality of empathetic conversations with the users and can be 1 https://github.com/rungjoo/compm figure 1 : an example of meld dataset utilized when sending tailored push messages to the users @xcite @xcite . in addition, emotion recognition can be effectively used for opinion mining, recommender systems, and healthcare systems where it can improve the service qualities by providing personalized results. as these interactive machines increase, the erc module plays an increasingly important role. figure 1 is an example of a conversation in which two speakers are angry at each other. the emotion of speaker b's utterance ("how'd you get to that?") is angry. if the system does not take into account previous utterances, it is difficult to properly recognize emotions. like the previous studies @xcite , we show that the utterance-level emotion recognition, which does not consider the previous utterance, have limitations and experiments result in poor performances. therefore, recent studies are attempting to recognize emotions while taking into account the previous utterances. representatively, dia-loguernn @xcite recognizes the present emotion by tracking context from the previous utterances and the speaker's emotion. aghmn @xcite considers the previous utterances through memory summarizing using gru with attention. many recent studies use external knowledge to improve the erc performance. however, this exter-nal knowledge is often only available in english. in order to utilize the previous methods in languages of other countries, it is expensive and difficult to utilize because external knowledge data must be newly constructed. in recent nlp studies, due to the effectiveness of the pre-trained language model, it has already been developed in many countries. since pre-trained language models are trained by unsupervised learning, these models are relatively usable approaches regardless of language types. @xcite introduces that these language models can be used as knowledge bases and have many advantages over the structured knowledge bases. based on these studies, we eliminate the dependence on structured external data used in cutting-edge systems and use a pre-trained language model as a feature extractor of knowledge. compm, introduced in this paper, is composed of two modules that take into account previous utterances in dialogue. (1) the first is a context embedding module (com) that reflects all previous utterances as context. com is an auto-regressive model that predicts the current emotion through attention between the previous utterances of the conversation and the current utterance. (2) the second is a pre-trained memory module (pm) that extracts memory from utterances. we use the output of the pre-trained language model as the memory embedding where the utterances are passed into the language model. we use the pm to help predict the emotion of the speaker by taking into account the speaker's linguistic preferences and characteristics. we experiment on 4 different english erc datasets. multi-party datasets are meld @xcite and @xcite , and dyadic datasets are iemocap @xcite and dailydialog @xcite . compm achieves the first or second performance according to the evaluation metric compared to all previous systems. we perform an ablation study on each module to show that the proposed approach is effective. further experiments also show that our approach can be used in other languages and show the performance of compm when the number of data is limited. ## related work many recent studies use external knowledge to improve the erc performance. ket @xcite is used as external knowledge based on conceptnet @xcite and emotion lex-icon nrc_vad @xcite as the commonsense knowledge. conceptnet is a knowledge graph that connects words and phrases in natural language using labeled edges. nrc_vad lexicon has human ratings of valence, arousal, and dominance for more than 20,000 english words. cosmic @xcite and psychological @xcite improve the performance of emotion recognition by extracting commonsense knowledge of the previous utterances. commonsense knowledge feature is extracted and leveraged with comet @xcite trained with atomic (the atlas of machine commonsense) @xcite . atomic has 9 sentence relation types with inferential if-then commonsense knowledge expressed in text. todkat @xcite improves performance by combining commonsense knowledge using comet and topic discovery using vhred @xcite to the model. ekman @xcite ) constructs taxonomy of six common emotions (joy, sadness, fear, anger, surprise, and disgust) from human facial expressions. in addition, ekman explains that a multimodal view is important for multiple emotions recognition. the multi-modal data such as meld and iemocap are some of the available standard datasets for emotion recognition and they are composed of text, speech and vision-based data. @xcite uses speech and visual information to recognize emotions, and @xcite attempts to recognize emotions based on text information. meld and icon @xcite show that the more multi-modal information is used, the better the performance and the text information plays the most important role. multimodal information is not always given in most social media, especially in chatbot systems where they are mainly composed of text-based systems. in this work, we design and introduce a text-based emotion recognition system using neural networks. in the previous studies, such @xcite al. (2018b); zadeh et al. (2017); majumder et al. (2019) , most works focused on dyadic-party conversation. however, as the multi-party conversation datasets including meld and emorynlp have become available, a lot of recent research is being conducted on multi-party dialogues such as @xcite ; @xcite ; @xcite . in general, the multi-party conversations have higher speaker dependency than the dyadic-party dialogues, therefore have more conditions to consider and result in poor performance. zhou et al. (2018); zhang et al. (2018a) shows that commonsense knowledge is important for understanding conversations and generating appropriate responses. @xcite reports that the lack of external knowledge makes it difficult to classify implicit emotions from the conversation history. eda @xcite ) expands the multi-modal emotion datasets by extracting dialog acts from meld and iemocap and finds out that there is a correlation between dialogue acts and emotion labels. ## conclusion we propose compm that leverages pre-trained memory using a pre-trained language model. compm consists of a context embedding module (com) and a pre-trained memory module (pm), and the experimental results show that each module is effective in improving the performance. compm outperforms baselines on both dyadic-party and multi-party datasets and achieves state-of-the-art among systems that do not use external knowledge. in addition, compm achieves performance comparable to cutting-edge systems that leverage structured external knowledge, which is the effect of pre-trained memory of the language model. by combining other pre-trained memories, we find that the pre-trained memory extracted with roberta is richer and more effective than the pre-trained memory extracted with bert or gpt2. since we believe that pre-trained memory is proportional to the performance of a language model, a language model with a large training corpus and many parameters is considered to be more effective. however, we find that context modeling is more important than pre-trained memory for emotion recognition in conversation, and future research will focus on context modeling. additionally, our approach achieves competitive performance and does not require externally structured data. therefore, we show that it can be easily extended to korean as well as english, and it is expected to be effective in other countries.