Few-Shot Emotion Recognition in Conversation with Sequential Prototypical Networks (2021)

Abstract: Several recent studies on dyadic human-human interactions have been done on conversations without specific business objectives. However, many companies might benefit from studies dedicated to more precise environments such as after sales services or customer satisfaction surveys. In this work, we place ourselves in the scope of a live chat customer service in which we want to detect emotions and their evolution in the conversation flow. This context leads to multiple challenges that range from exploiting restricted, small and mostly unlabeled datasets to finding and adapting methods for such context. We tackle these challenges by using Few-Shot Learning while making the hypothesis it can serve conversational emotion classification for different languages and sparse labels. We contribute by proposing a variation of Prototypical Networks for sequence labeling in conversation that we name ProtoSeq. We test this method on two datasets with different languages: daily conversations in English and customer service chat conversations in French. When applied to emotion classification in conversations, our method proved to be competitive even when compared to other ones.

## introduction there has been a recent surge in research focusing on analyzing dyadic human to human interactions. many of these studies @xcite @xcite focus on emotion recognition in conversations (erc) taking into account multiple data modalities. moreover, most of the progress made in erc has been done without factoring in constraints corresponding to specific but prominent industrial applications, like customer service. this is partly due to studies focusing on using artificial datasets @xcite made of mock-up conversations to facilitate result replication and comparison. a few existing studies address customer service applications @xcite @xcite and show the difficulties to deal with such in-the-wild and domain-specific data. in this work, we focus on data from a live chat support in which we want to detect emotions and their evolution in the conversational flow. this setting corresponds to a human dyadic conversation, albeit with a specific business-related objective. we make the hypothesis that the emotion flows of the visitor and the operator will bring information on the quality of the service and help operators better assist customers. this hypothesis is close to relevant studies on the importance of emotions and empathy in dyadic call center conversations @xcite . this specific setting leads to multiple challenges: indeed, it is difficult and costly to label this kind of data -and even then, these exchanges are very sparse in emotions, most of the labels associated with utterances being neutral. to maximize data efficiency, we use few-shot learning (fsl), and adapt a popular approach to our highly unbalanced data. by setting up this approach in an episodic fashion @xcite , we join studies on erc and studies on fsl to tackle this industrial use-case. we contribute by proposing a variant to prototypical networks @xcite dedicated to erc on data produced by company services, framing it as a sequence labeling task. we modify the original model by allowing it to consider the whole conversational context when making predictions, through a sequential context encoder and the use of conditional random fields (crf) on top of the model. we test our method on two datasets, in two different languages. the first one, made of daily conversations in english, allows us to compare ourselves to previous methods, while the second one, made of private data from a live chat customer ser-vice, allows us to conduct a performance analysis in our target setting. we also present the latter dataset, along with its annotation process. this paper is organized as follows. first, we sum up the related work on textual erc and fsl in conversations (section 2). then we present the datasets along with the emotional annotation scheme and the annotation campaign set up for the customer service live chats dataset (section 3). we continue by thoroughly presenting the sequential prototypical networks (section 4) before looking at the achieved results on both datasets (section 5). finally, we present the limitations of such a system (section 6) and conclude (section 7). few-shot learning fsl @xcite @xcite is suitable to tackle this data limitation. it aims at generaliz-ing faster, leading to a lower dependency on data quantity. it is mainly set up through episodic composition @xcite which recreates the few-shot learning setting by working with small training episodes. several learning methods are based on metric-learning: siamese networks, which share some weights, are used to learn a metric between examples @xcite . matching networks @xcite use the training examples to find the weighted nearest neighbors @xcite . prototypical networks @xcite consider averaged class representations from the training examples and a cosine distance to compare the elements to these class representations. relation networks replace the euclidean by the deep neural network which aims at training a distance metric @xcite . in this work, we consider approaches based on prototypical networks. as al-shedivat et al. (2021) recently showed it, such approaches are the most efficient when working with a low amount of training samples. many variants have been proposed, on different tasks and topics such as relation classification in text @xcite @xcite , sentiment classification in amazon comments @xcite , named entity recognition @xcite @xcite , or even speech classification in conversation @xcite . this surge of interest on applying few-shot learning to these topics can be attributed to specific datasets, such as few-rel @xcite for relation classification. while erc is mainly considered in a fully supervised learning setting, we intend to view it as a few-shot learning sequence labeling class. in this paper, we propose the first few-shot learning approach on erc using sequence labeling through adapting prototypical networks. we compare our method to the original prototypical networks @xcite and to a variant dedicated to named entity recognition @xcite that is easily applicable to our task. ## data to be able to both study the behavior of our model in its targeted industrial use-case, and allow performance comparison with baselines, we will work with two very different corpora: our proprietary live chat customer service dataset, and dailydialog @xcite . in both datasets, messages are labeled with emotions while considering the context of the conversation. however, they vary considerably in their topics and lexical fields: ordinary matters for dailydialog and railway related customer service for the live chats. they also vary in the assumptions they make about the speakers : while the topics discussed in dailydialog imply a sense of proximity, the live chat customer service involve complete strangers with pre-existing emotional states (e.g. the visitor is already stressed due to a refund issue). both datasets' statistics can be found in ## methodology formally, our dataset d is comprised of conversations (c 1 , c 2 , . . . , c |d| ), which are in turn made of utterances: to each of these utterances is associated an emotion label, giving a sequence of labels by conversation: finally, an utterance is a sequence of words, u @xmath0, w j 2 , . . . , w j |u j | ) . ## results tables 3 and 4 show the performance of the model using the micro f1-score. we use the protocol usually followed by the literature and do not take into account the majority class "no_emotion" as it represents 80% of the dailydialog corpus. this allows performance comparison with related work on erc through supervised learning. we do the same for the live chat customer service corpus by ignoring the "neutral" label. comparison to supervised learning dailydialog is used to compare our fsl approach with recent supervised learning results on erc. as expected, our best fsl model, protoseq, yields lesser performance than supervised approaches. the latter presuppose the availability of a sufficiently large amount of annotated data and their performance thus represents the upper bound of the expected results. more precisely, we focus on the difference between protoseq with a state-of-the-art supervised model, cesta @xcite , which is computation-heavy. indeed, cesta is a contextualized emotion sequence tagging model which considers the fusion of a combination of a transformer and bilstm as the global context encoder with a recurrent individual context encoder before feeding a crf layer. cesta achieves 63% in micro f1-score in a fully supervised learning approach. protoseq, much lighter, achieves a 31% micro f1 score, demonstrating the potential of fsl for sequence labeling when available data is scarce, especially when many supervised approaches obtained f1-scores around 50%. while using the live chat customer service dataset, we only change the initial embeddings from english to french, and apply the two best models according to 3: cesta and ket @xcite . the cesta implementation yielded inconclusive results foot_5 , this is why we present the ket results on our specific corpus in ## limitations while the protoseq model seems to be suitable for fsl in erc, it still has inherent limitations related to its architecture. protoseq uses a crf as its final layer, leading to a sequence labeling optimizer that does not take the order into account. while this yields better performance, it does not guarantee that the order information retrieved from the context encoder is wisely used, especially since we use the euclidean distances to class prototypes as emission scores for the sequence labeling. an ordered-prediction approach may allow the model to better assist operators in real-time during their decision process. another limitation of our model is that it may be difficult to adapt to changes in the context in which customer service is provided. indeed, the type of service or the plaftorm used may lead to lexical field changes or very different emotional states for the incoming visitors. ## conclusion in this paper, we presented the first study on emotion recognition in conversations using few-shot learning. we proposed a variant of prototypical networks taking into account the emotion recognition as a sequence labeling task while allowing fast convergence. when compared to other prototypical networks for sequence labeling in few-shot, our model obtained higher scores on both daily-dialog and customer service live chats. through this work, we showed that few-shot learning is possible for this task even though it is still difficult to achieve the same performance as supervised learning approaches. this study also shows the challenges that remain when tackling in-the-wild data collected in the context of a real application. future work will be dedicated to the improvement of the current few-shot erc approach by adding unlabeled elements in the support set and by investigating the addition of external business knowledge to such an approach.