RADDLE : An Evaluation Benchmark and Analysis Platform for Robust Task-oriented Dialog Systems (2021)

Abstract: For task-oriented dialog systems to be maximally useful, it must be able to process conversations in a way that is (1) generalizable with a small number of training examples for new task domains, and (2) robust to user input in various styles, modalities, or domains. In pursuit of these goals, we introduce the RADDLE benchmark, a collection of corpora and tools for evaluating the performance of models across a diverse set of domains. By including tasks with limited training data, RADDLE is designed to favor and encourage models with a strong generalization ability. RADDLE also includes a diagnostic checklist that facilitates detailed robustness analysis in aspects such as language variations, speech errors, unseen entities, and out-of-domain utterances. We evaluate recent state-of-the-art systems based on pre-training and fine-tuning, and find that grounded pre-training on heterogeneous dialog corpora performs better than training a separate model per domain. Adversarial training is also proposed to improve model robustness against noisy inputs. Overall, existing models are less than satisfactory in robustness evaluation, which suggests opportunities for future improvement.

## introduction dialogs constitute a crucial communication channel in completing a broad range of tasks, such as weather query, flight and restaurant booking, movie booking, it help desk, etc. comparing to chitchat systems that are usually modeled with singleturn context-response pairs, task-oriented dialog systems involve retrieving information from knowledge bases and reasoning over multiple dialog turns. this makes it especially important for a system to † work was done when zhu zhang was visiting msr 1 robust task-oriented dialog systems evaluation 2 benchmark link: http://aka.ms/raddle be able to produce response that are grounded on tasks goals and user intents. in a bid to support human-computer interactions, task-oriented dialog systems have been built to allow users to converse with a computer system using natural language, such as siri, google assistant, amazon alexa, microsoft xiaoice @xcite . traditionally, a task-oriented dialog system uses a modularized pipeline with four modules that execute sequentially @xcite . a natural language understanding (nlu) module identifies user intents and extracts associated information such as slots and corresponding values from user input. a dialog state tracker (dst) infers the belief state (or user goal) from dialog history. the belief state is often used to query a task-specific database (db) to obtain the db state, such as the number of entities that match the user goal. the dialog state and db state are then passed to a dialog policy (pol) module to select the next system action. a natural language generation (nlg) module converts the action to a natural language response. the human ability to converse is general, flexible, and robust. in contrast, most popular tools for dialog system development adopting the above modular systems are designed for specific tasks and struggle with out-of-scope data. if we aspire to develop models beyond extensively handcrafted rules and annotated data for each single domain/task, it is critical to develop a more unified, efficient and robust model that can more quickly learn to execute a range of tasks in different domains. to fuel research in this direction, we present the raddle benchmark. it includes a collection of task-oriented dialog tasks in diverse domains (e.g. end-to-end modeling, dialog state tracking). the benchmark also has a companion online platform for model evaluation, comparison, and robustness analysis. importantly, raddle exhibits two unique advantages that pave the way for building more pragmatic dialog systems: (i) limited data setting is the major focus of raddle, to evaluate the generalization ability of models. it aims at simulating the real-world application scenarios where only very limited amount of labelled data is available for new domains. given this focus, raddle is therefore a favorable benchmark to evaluate recent models in the pre-training and finetuning paradigm, which learn to represent linguistic knowledge in a way that facilitates sample-efficient learning and effective knowledge transfer. (ii) robustness analysis is introduced to study model performance in various challenging scenarios, where models are evaluated with anomalous user input such as language variations, speech errors, unseen entities and out-of-domain utterances. failing to handle these inputs often produce inappropriate responses leading to frustrating user experience. these scenarios are common for deployed systems in the real world, but are largely ignored in existing dialog benchmarks. to the best of our knowledge, raddle presents the first work to fill this gap. to better understand the challenges posed by raddle, we conduct experiments with simple baselines and state-of-the-art task-oriented dialog models. we find that grounded pre-trained models with a unified multi-task learning objective outperform models separately trained on each domain. moreover, even the best performing model (soloist @xcite ) in our evaluation achieves a fairly low score in robustness analysis. this suggests that our baseline models can handle common inputs with strong regularities, but struggle with anomalous inputs that require deeper reasoning. in summary, our key contributions are: (i) a novel dialog benchmark with an emphasis on limited data and multiple domains/tasks, which formally creates a scenario to evaluate the grounding and generalization ability of pre-trained models. (ii) a crowd-sourced diagnostic evaluation dataset to cover a broad range of real-world sophistication to study model robustness. (iii) an online evaluation platform and leaderboard to track research progress, with human evaluation services to be granted to top-ranked submissions on a bi-monthly basis. (iv) baseline results for major existing approaches to task-oriented dialogs are reported. an adversarially robust model is proposed to improve the generalization ability in noisy environments. starter codes, pre-trained models, and scripts to reproduce the results will be provided together with the benchmark. ## robustness diagnostic checklist existing benchmarks assume a world of a "perfect" user who always provides precise, concise, and semantically unambiguous utterances. these goal-oriented dialog datasets are largely collected by crowd-sourcing, where a crowd-sourced worker enacts the part of a real user by following a set of template instructions provided for the task. this method results in a dataset where most user utterances are straight-forward, stick to the goal and tend to leave out the variation/errors commonly found in real-world conversational data. to this end, we collect a suite of language variations to reveal the dialog sophistication in the real world, and measure the robustness of dialog models. ## conclusion we introduce raddle, a platform and collection of resources for evaluating and analyzing taskoriented dialog systems. we confirm (1) the utility of grounded pre-training and transfer learning methods in dialog systems: pre-training improves generalization in a limited data setting, and (2) adversarial training improves robustness, but still leaves room for improvement. when evaluating these models on our diagnostic dataset, we find that they fail (often spectacularly) on many robustness test cases, suggesting possible avenues for future work. in summary, the question of how to design unified, efficient, robust models remains largely unexplored, and we believe that raddle can provide fertile soil for addressing this challenge.