Action-Based Conversations Dataset: A Corpus for Building More In-Depth Task-Oriented Dialogue Systems (2021)

Abstract: Existing goal-oriented dialogue datasets focus mainly on identifying slots and values. However, customer support interactions in reality often involve agents following multi-step procedures derived from explicitly-defined company policies as well. To study customer service dialogue systems in more realistic settings, we introduce the Action-Based Conversations Dataset (ABCD), a fully-labeled dataset with over 10K human-to-human dialogues containing 55 distinct user intents requiring unique sequences of actions constrained by policies to achieve task success. We propose two additional dialog tasks, Action State Tracking and Cascading Dialogue Success, and establish a series of baselines involving large-scale, pre-trained language models on this dataset. Empirical results demonstrate that while more sophisticated networks outperform simpler models, a considerable gap (50.8% absolute accuracy) still exists to reach human-level performance on ABCD.

## introduction the broad adoption of virtual assistants and customer service chatbots in recent years has been driven in no small part by the usefulness of these tools, whereby actions are taken on behalf of the user to accomplish their desired targets @xcite . research into taskoriented dialogue has concurrently made tremendous progress on natural language understanding of user needs @xcite @xcite . however, selecting actions in real life requires not only obeying user requests, but also following practical policy limitations which may be at odds with those requests. for example, while a user may ask for a refund on their purchase, an agent should only honor such a request if it is valid with regards to the store's return policy. described in actions, before an agent 1 all code and data will be available at this location. can [offer refund], they must first @xcite . furthermore, resolving customer issues often concerns multiple actions completed in succession with a specific order since prior steps may influence future decision states. (see figure 1 ) to more closely model real customer service agents, we present the action-based conversations dataset (abcd) consisting of 10,042 conversations containing numerous actions with precise procedural requirements. these actions differ from typical dialogue acts because tracking them necessitates striking a balance between external user requests and internally-imposed guidelines. thus, the major difference between abcd and other dialogue datasets, such as mul-tiwoz @xcite , is that it asks the agent to adhere to a set of policies while simultaneously dealing with customer requests. while the prevalent data collection paradigm involves wizard-of-oz techniques, our situation containing asymmetric speakers compelled the design of a novel expert live chat system. our dataset includes asymmetric speakers because, unlike customers, agents must undergo extensive training to be able to navigate the agent guidelines during real-time conversations. this makes a naive pairing process untenable since arbitrary matching might lead to chats containing two users who share the same role. based on the unique aspects of abcd, we propose two new tasks. to start, action state tracking (ast) closely mirrors the format of dialogue state tracking where the user intent is inferred from the dialogue history. ast then differs since the correct state must also be reconciled with the requirements outlined in the agent guidelines. as a second task, cascading dialogue success (cds) extends this notion across the entire conversation. at each turn, the agent decides to take an action, respond with an utterance or end the chat. as needed, the agent should also predict the right action or select the best utterance. for each task, we build various models to establish baseline performance and to highlight the importance of each constraint. experiments show that in addition to conversation history, conditioning on the agent guidelines further boosts performance, with top models relying on both aspects to reach 31.9% accuracy. additional results show removing action context hurts performance, implying the importance of taking into account the sequential nature of actions. lastly, human evaluation reaches 82.7%, demonstrating ample room for future improvement. the contribution of this work is three-fold: (1) we provide a novel, large-scale dataset containing context-dependent, procedural actions along with corresponding agent guidelines. (2) we establish a new technique called expert live chat for capturing natural dialogue between two unequal interlocutors. (3) we propose two metrics, action state tracking and cascading dialogue success, for measuring dialogue comprehension with policy constraints. finally, we build on pretrained neural models to serve as baselines for these tasks. ## related work traditional dialogue datasets in recent years, dialogue datasets have grown in size from hundreds of conversations to the tens of thousands @xcite @xcite . unlike opendomain chatbots often built for entertainment, task-oriented dialogue systems trained on such datasets are intended for solving user issues. the resolution of these issues implicitly requires taking actions, where an action is a non-utterance decision that depends on both user and system inputs. despite the tremendous number of dialogues, examples in previous benchmarks fixate on the single knowledge base (kb) lookup action where the agent searches for an item that matches the user's desires and is available in the kb. by sticking to this sole interaction, conversations can be generated through rules @xcite , paraphrased from templates @xcite or taken from static text scenarios @xcite , leading to dialogues that are predominantly homogeneous in nature. many datasets have scaled to more domains as well @xcite @xcite since each new domain introduces a kb lookup requiring different slotvalues, the number of unique actions grows as a linear function of the number of domains covered. rather than expanding wider, abcd instead focuses deeper by increasing the count and diversity of actions within a single domain. exploring other avenues multiple aspects are explored by conversational datasets attempting to mimic reality. @xcite studies the ability of a dialogue model to handle empathy, while @xcite focuses on commonsense reasoning. another approach is to augment dialogues with multi-modality including audio @xcite or visual @xcite components. other researchers have explored grounding conversations with external data sources such as personas @xcite , online reviews @xcite or large knowledge bases @xcite . intricate dialogues can also appear when studying collaboration @xcite or negotiation @xcite which strongly encourage interaction with the other participant. in comparison, abcd aims to make dialogue more realistic by considering distinct constraints from policies. dialogues with policies procedural actions following strict guidelines naturally emerge in dialogue research geared towards real-world appli- ## action-based conversation dataset in this section, we describe the task setting of abcd by following along with the example dialog shown in figure 1 . ## data collection methodology this section outlines how we collect and annotate our dataset with context-dependent actions. ## dataset statistics and analysis we validate all dialogues to pass quality thresholds such as including a minimum number of actions and avoiding copy/paste behavior. after filtering, we end up with 10,042 total conversations with an average of 22.1 turns -the highest turn count among all compared datasets. unsurprisingly, abcd includes more actions per dialogue than other datasets, by at least a factor of two. abcd also contains a lower absolute number of tokens, but also has the highest variance in the number of tokens per turn. (see by grounding to the relatable scenario of chatting with customer support of an online retail company, speakers often showcase various forms of natural dialogue, such as offering diverse reasons for shopping or asking detailed follow-up questions. furthermore, the unconstrained nature of expert live chat allows users to chat with each other in a free-form style. dialogues exhibited normal texting behavior such as users speaking for many turns in a row or fixing typos with a star in the subsequent line. other examples of linguistic phenomenon can be observed in ## abcd as a dialogue benchmark the novel features in abcd brings two new dialog tasks, action state tracking and cascading dialogue success. we also build baseline systems that are variants of standard dialogue models and report their results on abcd. ## conclusion and future work in conclusion, we have presented abcd which includes over 10k dialogues that incorporate procedural, dual-constrained actions. additionally, we established a scalable method for collecting live human conversations with unequal partners. we found that pre-trained models perform decent on action state tracking, but there is a large gap between humans agents and the top systems for cascading dialogue success. we plan to incorporate gpt-related models @xcite , as alternate forms of preprocessing have shown promise in other nlp tasks. other techniques could also be used to incorporate speaker info, action semantics and other meta-data. wholly new systems that attend to the agent guidelines in a fully differentiable manner are also worth exploring. by grounding dialogues to in-depth scenarios with explicit policies, we hope to have pushed towards a better understanding of dialogue success.