N at CS : Eliciting Natural Customer Support Dialogues (2023)

Abstract: Despite growing interest in applications based on natural customer support conversations,there exist remarkably few publicly available datasets that reflect the expected characteristics of conversations in these settings. Existing task-oriented dialogue datasets, which were collected to benchmark dialogue systems mainly in written human-to-bot settings, are not representative of real customer support conversations and do not provide realistic benchmarks for systems that are applied to natural data. To address this gap, we introduce NatCS, a multi-domain collection of spoken customer service conversations. We describe our process for collecting synthetic conversations between customers and agents based on natural language phenomena observed in real conversations. Compared to previous dialogue datasets, the conversations collected with our approach are more representative of real human-to-human conversations along multiple metrics. Finally, we demonstrate potential uses of NatCS, including dialogue act classification and intent induction from conversations as potential applications, showing that dialogue act annotations in NatCS provide more effective training data for modeling real conversations compared to existing synthetic written datasets. We publicly release NatCS to facilitate research in natural dialog systems

## introduction applications that are applied to human-to-human customer support conversations have become increasingly popular in recent years. for example, assistive tools that aim to support human agents, provide analytics, and automate mundane tasks have become ubiquitous in industry applications (amazon contact lens, 2023; google contact center ai, 2023; microsoft digital contact center platform, 1 https://github.com/amazon-science/ dstc11-track2-intent-induction . existing dialogue datasets focusing on development and evaluation of task-oriented dialogue (tod) systems contain conversations that are representative of human-to-bot (h2b) conversations adhering to restricted domains and schemas @xcite . realistic live conversations are difficult to simulate due to the training required to convincingly play the role of an expert customer support agent in non-trivial domains @xcite . existing datasets are also primarily written rather than spoken conversations as this modality is cheaper to simultaneously collect and annotate asynchronously through crowdsourcing platforms. to address these gaps, we present natcs, a multi-domain dataset containing english conversations that simulate natural, human-to-human (h2h), two-party customer service interactions. first, we describe a self-collection dataset, natcs self , where we use a strict set of instructions asking participants to write both sides of a conversation as if it had been spoken. second, we present a spoken dataset, natcs spoke , in which pairs of participants were each given detailed instructions and asked to carry out and record conversations which were subsequently transcribed. we observe that the resulting conversations in natcs share more characteristics with real customer service conversations than pre-existing dialogue datasets in terms of diversity and modeling difficulty. we annotate a subset of the conversations in two ways: (1) we collect task-oriented dialogue act annotations, which label utterances that are important for moving the customer's goal forward and (2) we categorize and label customer goals and goalrelated information with an open intent and slot schema, mimicking the process for building a tod system based on natural conversations. we find that classifiers trained with the resulting dialogue act annotations have improvements in accuracy on real data as compared to models trained with preexisting tod data. our main contributions are threefold: our paper is structured as follows: in section 2, we review other dataset collection methods and approaches for evaluating dialogue quality. in section 3, we describe how natcs conversations and annotations were collected. in section 4, we compare natcs with pre-existing dialogue datasets as well as real h2h customer service conversations. finally, in section 5, we further motivate the dataset through two potential downstream applications, task-oriented dialogue act classification and intent induction evaluation. ## related work dialogue dataset collection the goal of natcs, to produce conversations that emulate real spoken customer service interactions, differs substantially from previous synthetic dialogue dataset collections. previous synthetic, goal-oriented datasets have used the wizard of oz framework @xcite . this framework calls for one person to interact with what they think is a computer, but is actually controlled by another person, thus encouraging a human-to-bot (h2b) style. @xcite define a specific version of this approach to be used with crowdsourced workers to produce synthetic, task-oriented datasets. this approach has since been adopted as a standard method to collect goal-oriented conversations @xcite @xcite @xcite . @xcite (mdgo), sgd @xcite , and taskmaster @xcite are particularly relevant comparisons to our work. mdgo explicitly encourages dialogue complexities, which serve as inspiration for our complexity-driven methodologies. sgd is interesting partially because of the scale of their collection, relevance of their task-oriented dialogue act and intent/slot annotations, and because they diverge from the common practice of using the wizard-of-oz methodology through the use of dialogue templates and paraphrasing. taskmaster presents a methodology for self-collected dialogues, which is analogous to our natcs self collection. despite these similarities, the methodology for natcs differs significantly from any of these preexisting datasets, because of both the target modality (spoken, or spoken-like conversations), and the setting (h2h instead of h2b). ## dataset analysis to better motivate natcs as a proxy for natural, spoken form customer service conversations from multiple domains with a diverse set of intents, we compare with real conversations from commercial datasets comprising 5 call centers for retail and finance-related businesses (henceforth real). all datasets in real consist of manuallytranscribed conversations between human agents and customers in live phone conversations where all personally-identifiable information has been preredacted. we restrict our analysis to datasets with primarily customer-initiated two-party dialogues. ## applications in this section, we investigate two potential applications of natcs as a resource for building and evaluating systems related to human-to-human (h2h) customer service interactions. one goal of natcs is to encourage research in the under- ## conclusions we present natcs, a corpus of realistic spoken human-to-human customer service conversations. the collection of natcs is complexity-driven and domain-restricted, resulting in a dataset that better approximates real conversations than preexisting task-oriented dialogue datasets along a variety of both automated and human-rated metrics. we demonstrate two potential downstream applications of natcs, showing that training using natcs results in better performance with real test data compared to training using other publiclyavailable goal-oriented datasets, and that natcs can provide a new challenging benchmark for realistic evaluation of intent induction. we hope that natcs will help facilitate open research in applications based on customer support conversations previously accessible mainly in industry settings by providing a more realistic annotated dataset. in future work, we hope to expand on annotations in natcs to support more tasks such as call summarization, response selection, and generation.