T icket T alk: Toward human-level performance with end-to-end, transaction-based dialog systems (2021)

Abstract: We present a data-driven, end-to-end approach to transaction-based dialog systems that performs at near-human levels in terms of verbal response quality and factual grounding accuracy. We show that two essential components of the system produce these results: a sufficiently large and diverse, in-domain labeled dataset, and a neural network-based, pre-trained model that generates both verbal responses and API call predictions. In terms of data, we introduce TicketTalk, a movie ticketing dialog dataset with 23,789 annotated conversations. The conversations range from completely open-ended and unrestricted to more structured, both in terms of their knowledge base, discourse features, and number of turns. In qualitative human evaluations, model-generated responses trained on just 10,000 TicketTalk dialogs were rated to “make sense” 86.5% of the time, almost the same as human responses in the same contexts. Our simple, API-focused annotation schema results in a much easier labeling task making it faster and more cost effective. It is also the key component for being able to predict API calls accurately. We handle factual grounding by incorporating API calls in the training data, allowing our model to learn which actions to take and when. Trained on the same 10,000-dialog set, the model’s API call predictions were rated to be correct 93.9% of the time in our evaluations, surpassing the ratings for the corresponding human labels. We show how API prediction and response generation scores improve as the dataset size incrementally increases from 5000 to 21,000 dialogs. Our analysis also clearly illustrates the benefits of pre-training. To facilitate future work on transaction-based dialog systems, we are publicly releasing the TicketTalk dataset at https://git.io/JL8an .

## introduction building a dialog system that handles human conversational behavior is challenging because it must respond sensibly and relevantly to a wide variety of context-sensitive user input over multiple conversation turns. task-based systems, e.g. those used for ticket booking, food ordering, etc., face further hurdles to incorporate ever changing, real-world knowledge into the dialog and execute transactions. recently, there has been growing interest in the so-called end-to-end approach to task-based dialog systems @xcite @xcite @xcite due to its relatively simple and scalable architecture, and promising results in chatbot applications @xcite . inspired by sequence-to-sequence learning @xcite , this approach trains a single model on a dialog dataset to form the basis for a given application. for each dialog turn, the model effectively takes the conversation history as its input and generates an appropriate response. to gain wider adoption, the end-to-end approach must overcome challenges with respect to training data and factual grounding. in terms of training data, there is already general concern in the nlp community about the lack of quality, task-oriented dialog datasets, especially domain-specific collections @xcite . this problem is compounded for end-to-end approaches since they typically require a large amount of indomain data to generate competitive results. with respect to grounding, since the end-to-end approach is based on a single neural network, it must either incorporate the knowledge base (kb) into the model itself, or the model must be able to accurately predict which api calls to make and when. in addition, details returned from the api calls must be accurately incorporated in conversational responses. this is contrasted with modular architectures where the user's intent is derived from a structured representation and then used to determine which api calls to make such as in @xcite and @xcite . in this work we promote an end-to-end approach to single-domain, transaction-based dialog systems and describe how we overcome both data and grounding challenges described above. in qualitative evaluations, our models perform on par with humans in generating verbal responses as well as predicting api calls. just two components form the basis for this system: a sufficiently large, indomain, labeled dataset and a pre-trained transformer model. combining natural language output and structured api calls into a unified text-to-textformat allows us to leverage general purpose textto-text transformers to train models. specifically, we use the t5 infrastructure @xcite and show that its pre-training feature has a significant impact on evaluations, boosting scores by 30 percent. models were trained on our tickettalk dataset (aka taskmaster-3), a movie ticketing dialog corpus with 23,789 conversations labeled with a simple yet unique api-based annotation schema. this makes it one of the largest single-domain datasets to date. a public release of the dataset accompanies this paper. we chose movie ticketing since it is both transaction-based and relatively complex, but our overall approach to dialog systems applies to any task-based domain. while there is a lot of recent work on multi-domain task-based dialog systems, human-like interaction for even singledomain tasks has yet to be demonstrated. by first solving the problem for a single domain, we argue that replicating the process for multiple domains will be achievable by simply training on additional high-quality datasets labeled with the same apifocused strategy.