An Emotional Journey: Detecting Emotion Trajectories in D utch Customer Service Dialogues (2022)

Abstract: The ability to track fine-grained emotions in customer service dialogues has many real-world applications, but has not been studied extensively. This paper measures the potential of prediction models on that task, based on a real-world dataset of Dutch Twitter conversations in the domain of customer service. We find that modeling emotion trajectories has a small, but measurable benefit compared to predictions based on isolated turns. The models used in our study are shown to generalize well to different companies and economic sectors.

## introduction while emotion recognition in conversations (erc) has recently become a popular task in nlp @xcite , its application potential to reallife business-related settings remains understudied. our research focuses on applying erc to the domain of customer service (cs), as it can be used to model customer satisfaction, reduce churns, prioritize clients, and detect emotional shifts in clients throughout cs interactions. since the provision of customer service is gaining ground in both public and private chat channels, timely delivering highquality assistance is crucial in mitigating the effects of negative word-of-mouth @xcite and creating relational bonds between customers and brands @xcite . as emotion recognition is often implemented on 'artificial', open-domain conversations @xcite , we worked on realworld, domain-specific data that is more imbalanced and noisy. moreover, we are the firsts to tackle the erc task in dutch dialogues. to these com/sofielabat/emotwics-data and https:// github.com/hadifar/dutchemotiondetection , respectively. ## related work although emotion detection has often been applied to tweets @xcite and chat logs @xcite , the context-aware detection of emotions throughout conversations is a relatively recent development in nlp. state-of-the-art results for emotion detection on isolated texts are achieved by fine-tuning large pretrained language models. for dutch, there currently exist two such models named bertje @xcite and robbert @xcite , although crosslingual language models such as xlm @xcite can also be applied to dutch texts. in contrast to these 'vanilla' emotion detection systems, recent work on erc models additional information such as the conversational context, the temporal order of turns, and interlocutor-specific attributes @xcite . there exist two approaches for erc: we either view it as a sequence labeling task, or we predict emotions for a turn given the previous (and, in some variants, future) utterances. the latter approach was first addressed by recurrence-based models such as lstms @xcite , conversational memory networks @xcite , and attentive rnns @xcite . afterwards, graphbased @xcite and knowledge-enriched transformer models @xcite were also investigated. the sequence labeling approach was introduced by @xcite who used information about the emotional consistency in conversations. his model combines a global context encoder (transformer) with an individual context encoder (lstm) into a crf layer to jointly predict emotions for all utterances. @xcite implemented erc in a few-shot learning sequence labeling problem. in our second experimental setup, we also tackle emotion detection as a sequence labeling task. all but the two previously mentioned models are trained on publicly released datasets in english containing open-domain conversations @xcite . there is only one small dutch dataset @xcite with 11 conversations and emotions rated on leary's rose @xcite , a dimensional framework with two axes representing the degree of control and agreeableness. for erc, the corpus is less suitable given its small size, low agreement, fixed events, and uncommon emotion model. unlike standard sentiment analysis, the fine-grained task of erc has not yet become commonplace in cs departments. to our knowledge, there exist only a few papers that apply erc to cs @xcite @xcite . ## experimental setup after describing the emotwics corpus along with its prediction tasks (section 3.1), our data segmen-tation strategies are introduced (section 3.2), followed by the models and their implementation details (section 3.3). ## results and discussion we present the results of our models for six classification tasks on isolated tweets across the different train-test setups in section 4.1. in section 4.2, we focus on the emotion trajectories, and cast the detection of emotion clusters as a context-aware sequence labeling task.the presented metrics are micro and weighted f1 scores ( ## conclusion we presented the first experiments on a newly collected corpus of dutch twitter conversations annotated along four emotion layers. for our experiments on isolated tweets, we find that the best performance is obtained by fine-tuning pretrained language models such as robbert and xlm. we show that these two models transfer well across (i) time, (ii) companies within the same sectors, and (iii) across sectors. we also demonstrate that the detection of emotion clusters slightly benefits from knowledge about frequently occurring emotion trajectories, especially for classes with lower levels of support. in future research, we will extend our approach to model emotion trajectories for the purpose of real-time prediction (e.g., in chatbots), thus having access to past utterances only. we will also investigate emotion trajectories in longer conversations (e.g., on data collected through wizard of oz experiments @xcite ) and focus on joint prediction tasks such as emotion-cause or emotion-response strategy extraction.