Dialogue Quality and Emotion Annotations for Customer Support Conversations (2023)

Abstract: Task-oriented conversational datasets often lack topic variability and linguistic diversity. However, with the advent of Large Language Models (LLMs) pretrained on extensive, multilingual and diverse text data, these limitations seem overcome. Nevertheless, their generalisability to different languages and domains in dialogue applications remains uncertain without benchmarking datasets. This paper presents a holistic annotation approach for emotion and conversational quality in the context of bilingual customer support conversations. By performing annotations that take into consideration the complete instances that compose a conversation, one can form a broader perspective of the dialogue as a whole. Furthermore, it provides a unique and valuable resource for the development of text classification models. To this end, we present benchmarks for Emotion Recognition and Dialogue Quality Estimation and show that further research is needed to leverage these models in a production setting.

## introduction artificial intelligence (ai) has evolved to become a ubiquitous technology in our lives. yet, its performance is limited by the amount of data it is trained on. therefore, and in order to maximise the rewards of such technology, substantial research and engineering effort has been devoted to collecting and annotating data according to needs and goals. one of the main limitations of most task-oriented conversational datasets is their lack of variability. the majority of these datasets are collected in controlled environments where annotators are one approach to equip nlp models with multilingual and diverse domain knowledge capabilities is to leverage llms pretrained on extensive amounts of publicly available data @xcite @xcite . however, lacking benchmarking dialogue datasets, it is not clear these models, applied to dialogue, are able to fully generalise to other languages and/or domains, even if other dimensions of variability remain unchanged. this paper builds upon the original maia dataset release by adding extensive annotations of emotion and dialogue quality at different granularity levels, thus allowing a holistic approach at understanding the dynamics of conversations in the context of customer support. the maia dataset is a collection of genuine bilingual customer support conversations initially released as a challenge dataset for the wmt chat sharedtask @xcite . in these conversations, which are powered by machine translation, the agent communicates with the customer exclusively in english, whereas the customer interacts with the agent exclusively in their native language. our annotations cover 612 dialogues accounting for around 25k sentences, covering diverse topics, ranging from account registration issues, payment and delivery clarifications and aftersale services. languages includes german (de), brazilian-portuguese (pt_br) and european portuguese (pt_pt). we argue that the maia dataset and the accompanying annotations have unique value in the field of customer support and conversational agents. the comprehensive annotations conducted enable the analysis of the relations between several dialogue sub-qualities and emotion. furthermore, they can be used as a training and benchmark dataset for text classification in these distinctive settings. for instance, one could leverage this dataset for the construction of dialogue systems that support customer-agent interaction processes. classifica-tion models trained on this data could assist customer service agents (human or machine) by measuring customer emotions and dialogue qualities in real-time and provide the agent with feedback on the fluidity and success of the dialog. to kick-start this research, this paper provides benchmarks for emotion recognition and dialogue quality estimation. results show that existing models are not strong enough to perform on par with other benchmarks, indicating significant future work research will be required to reduce this performance gap. in summary, the primary contributions of this work are as follow: ## benchmark evaluation given the focus of the annotation work was on emotions and dialogue quality, in this section we evaluate existing mainstream approaches for emotion recognition and automatic dialogue evaluation. ## conclusions this paper presents a comprehensive emotion and dialogue quality annotation for the maia dataset, a collection of genuine bilingual customer support conversations. all in all, we annotate 612 dialogues amounting to over 24k sentences. besides allowing for an opportunity to study the dynamics of machine translation aided customer support conversations, it also provides a novel opportunity to benchmark and explore applications of existing and future nlp models applied to dialogue. results on the different benchmarks indicate there is still room for improving existing models. llms such as gpt-4 (openai, 2023) show impressive classification and generation capabilities, and may prove useful in augmenting existing customer support datasets to new languages and tasks. these in turn can be used to build data-driven classifiers or end-to-end conversational agents that are robust to new languages and domains.