Contextual Augmentation of Pretrained Language Models for Emotion Recognition in Conversations (2020)

Abstract: Since language model pretraining to learn contextualized word representations has been proposed, pretrained language models have made success in many natural language processing tasks. That is because it is helpful to use individual contextualized representations of self-attention layers as to initialize parameters for downstream tasks. Yet, unfortunately, use of pretrained language models for emotion recognition in conversations has not been studied enough. We firstly use ELECTRA which is a state-of-the-art pretrained language model and validate the performance on emotion recognition in conversations. Furthermore, we propose contextual augmentation of pretrained language models for emotion recognition in conversations, which is to consider not only previous utterances, but also conversation-related information such as speakers, speech acts and topics. We classify information based on what the information is related to, and propose position of words corresponding to the information in the entire input sequence. To validate the proposed method, we conduct experiments on the DailyDialog dataset which contains abundant annotated information of conversations. The experiments show that the proposed method achieves state-of-the-art F1 scores on the dataset and significantly improves the performance.

## introduction as voice assistants are widely used, emotion recognition is also emerging as an important technique to provide a rich user experience. considering that it can detect the emotion state of speakers in real-time in an on-going conversation, it can be utilized in a variety of applications to generate more diverse and appropriate responses. as if to reflect this proliferation, detecting emotional state and emotion change in conversations has been widely studied. however, there is no significant progress in this research area. previous studies focus extensively on emotion detecting using groups of words and/or utterances representing emotion, but those are not sufficient to detect the emotion state change that varies depending on dialogue subject or speaker in given conversations. the proposed system in this paper models each conversation to contain two different levels of information (fig. 1 ). conversation-level information is effective in distinguishing conversations that show different aspect of emotion changes. given the specific type of conversation, utterance-level information is used to recognize speaker's emotion. basically, our research is based on the hypothesis that even if it's the same conversation there will be very different results depending on what previous or current conversations relate to and/or who is the speaker now. they so far have not considered essential information enough, but our results show the model trained with the essential information outperforms the state-of-the-art. various networks that could capture features that fit our hypothesis were considered, our evaluation results show that the transformer @xcite encoder family, which consists of multi-layers of self-attention and fully-connected layer, generally shows better performance. a self-attention model learns to generate representations for each token based on the context of the token. that is, the information mentioned above can be incorporated as each token representing fragments of information. in this paper, we use electra @xcite which is a state-of-the-art pretrained language model based on the transformer encoder. our experiments with the dailydialog dataset @xcite containing enough annotated information of english conversations show the proposed model significantly improves the f1 scores comparing to the previous studies. also, we thoroughly analyze effectiveness of incorporating each type of information as words in electra and different context lengths and present the result. ## related work convolutional neural networks (cnn) and recurrent neural networks (rnn) have traditionally been used for emotion recognition. dialoguernn @xcite has been proposed to model a speaker, a context from the preceding utterances and an emotion of the preceding utterances. dialoguegcn @xcite has been proposed to model a conversation using directed graph to propagate the speaker's dependency information. contextual information among distant utterances is propagated through two consecutive convolution operations by providing a convolutional network with the graph @xcite . attention gated hierarchical memory network (aghmn) @xcite has been proposed for real-time emotion recognition with an hierarchical memory network (hmn), a bidirectional gated recurrent unit (bigru) as the utterance reader and a bigru fusion layer for interaction between historical utterances; this network includes an attention gated recurrent unit (gru) to update internal state and a bidirectional variant gru to keep a balance between the contextual information from recent memories and that from distant memories. recently, a generalized neural tensor block followed by a twochannel classifier is designed to perform contextual compositionality which obtains context information and incorporates the context into utterance representation and sentiment classification simultaneously @xcite . many natural language processing (nlp) tasks have applied transformer @xcite to capture long context information and enrich text representations. knowledge-enriched transformer (ket) @xcite uses hierarchical self-attention for exploiting contextual information and dynamically refers to external commonsense knowledge. hitransformer-s @xcite uses a hierarchical transformer network with speaker embeddings to capture the contextual information and the interaction of speakers. there are many studies on how to use various data such as visual expression, voice, and text to improve the performance of emotion recognition. each type of data can be used alone in the emotion classification task, and when used together, better performance is observed empirically. this multimodal technique simply uses text and voice related to the speech @xcite , or additionally uses visual information such as the speaker's expression that occurs at the same time as the speech @xcite @xcite . models that consider multimodality combine and use data of different properties for one target task. among the studies for emotion recognition, there are studies that perform classification by hierarchically combining the relations between modals @xcite , or to select valid features using relations between modals @xcite . attention mechanisms, which are showing good performance in recent years, have also been used in several studies to find the relationship between modals @xcite . in sum, previous studies also use pretrained models and do not only concentrate on improving performance with neural networks, but incorporates a large deal of varying contextual information. likewise, we propose and validate a method to improve performance in this paper. however, compared to the previous studies, our proposed approach has a relative strength: it is easy to incorporate the approach to finetune state-of-the-art pretrained language models. ## background pretrained language models like bert @xcite , xlnet @xcite and elec-tra @xcite use the encoder of transformer @xcite that consists of multilayers of self-attention and fully-connected layer. attention can be defined using three terms, q (queries), k (keys) and v (values). self-attention is an attention method where q, k and v are the same or generated from the same source. given a sequence of vectors [y 1 , y 2 , ..., y n token ] whose length n token is the same. performing projections of the queries, keys and values respectively to d q , d k and d v dimensions n head times foot_0 is empirically more effective than a single linear projection, and this method is called multi-head attention. because the given sequence of vectors can be packed into a single matrix @xmath0, the produced sequence of vectors can also be represented as a matrix @xmath1. given x, y is computed in the multi-head attention way as: in each layer, a fully-connected layer is used to linearly transform the output of the self-attention layer, and layer normalization is used after the two outputs of self-attention and fully-connected layer are added. by stacking the layer l times, the entire model is built. by (pre)training on a large amount of text data, a self-attention model learns to generate representations for each token based on the context of the token. then the pretrained model learns to generate task-specific answers by finetuning. for example, by finetuning, electra can learn to recognize the emotion of the given utterance or to classify the polarity of individual words in the utterance. electra that we used for the experiments is different from other pretrained language models in that it uses not the generator, but an additional model, a discriminator, which consists of the encoder of transformer as the same described above. the difference between them is about details for pretraining, not finetuning, so we omit the explanation. details of the model related to our proposed method is introduced in the next section. 4 proposed method ## discussion we analyze effectiveness of the proposed method in detail in this section. in the proposed model, because we use three kinds of information, speaker, act and topic, we separate them to validate effectiveness of each type of information ( we employ a state-of-the-art pretrained language model, electra, to emotion recognition in conversations. because electra is a powerful nlp model by itself, we do not propose to modify the structure. instead, we explore and propose an effective method of providing the model with extra information as words to improve performance of emotion recognition. in this paper, we consider three kinds of information, speaker, act and topic, and propose a method to incorporate the information in pretrained language models. by incorporating the information, pretrained language models are expected to consider and relate the information to all tokens in the utterance via multi-layers of self-attention for emotion recognition. our experiments show that the proposed method improves performance of emotion recognition with large margin, and that the gain margin depends on how important the provided information is. on the dailydialog dataset, when given the topic as words, performance of the model is highly improved. this result means that if the information provided with the proposed method is closely related to the emotion,