X - R i SAWOZ : High-Quality End-to-End Multilingual Dialogue Datasets and Few-shot Agents (2023)

Abstract: Task-oriented dialogue research has mainly focused on a few popular languages like English and Chinese, due to the high dataset creation cost for a new language. To reduce the cost, we apply manual editing to automatically translated data. We create a new multilingual benchmark, X-RiSAWOZ, by translating the Chinese RiSAWOZ to 4 languages: English, French, Hindi, Korean; and a code-mixed English-Hindi language.X-RiSAWOZ has more than 18,000 human-verified dialogue utterances for each language, and unlike most multilingual prior work, is an end-to-end dataset for building fully-functioning agents. The many difficulties we encountered in creating X-RiSAWOZ led us to develop a toolset to accelerate the post-editing of a new language dataset after translation. This toolset improves machine translation with a hybrid entity alignment technique that combines neural with dictionary-based methods, along with many automated and semi-automated validation checks. We establish strong baselines for X-RiSAWOZ by training dialogue agents in the zero- and few-shot settings where limited gold data is available in the target language. Our results suggest that our translation and post-editing methodology and toolset can be used to create new high-quality multilingual dialogue agents cost-effectively. Our dataset, code, and toolkit are released open-source.

## introduction in recent years, tremendous effort has been put into the research and development of task-oriented dialogue agents; yet, it has been mainly focused on only a handful of popular languages, hindering the adoption of dialogue technology around the globe. collecting dialogue data from scratch for a new language is ideal but prohibitively expensive and time-consuming, leading to the current lack of reliable multilingual dialogue benchmarks. in recent years, several non-english taskoriented dialogue (tod) datasets have been created. these datasets are either collected from scratch @xcite , synthesized using a state machine with manually written templates, and paraphrased for fluency by crowdworkers @xcite , or manually translated from another language @xcite . all of these approaches are labor-intensive, costly, and timeconsuming; such investment is unlikely to be made for less widely spoken languages. this motivates the development of zero and fewshot techniques that can produce a usable agent in a new language with no or only a few gold training dialogues in the target language. concurrent with this work, @xcite ; @xcite ; @xcite adopt a translation and manual post-editing process where data is translated with neural machine translation models first, and then post-edited by crowdworkers. this approach has shown promise on multiwoz; however, reported zero-and few-shot accuracies show a big degradation in performance compared to full-shot accuracy in the source language. besides, the performance of the agent in the original language was not good to begin with, in part due to misannotations in the dataset @xcite . lastly, these datasets either focus only on the subtask of dialogue state tracking (dst) @xcite or auxiliary tasks such as response retrieval @xcite , or are too small @xcite to train end-to-end dialogue agents that require policy, interactions with databases, and response generation components. our overall goal is to make task-oriented dialogue research in major languages available to lowresource languages. the key is to produce highquality few-shot training, validation, and test set with as little manual effort as possible to enable zero-shot or few-shot training. we describe below our contributions towards this goal. ## the end-to-end tod task in end-to-end task-oriented dialogues, a user speaks freely with an agent over several turns to accomplish their goal according to their intents (e.g., "book a hotel with at least 5 stars"). in each turn, the agent must access its database if necessary to find the requested information (e.g., find a hotel that meets user constraints), decide on an action (e.g., present the information to the user or ask for additional information), and finally respond to the user in natural language based on the action it chooses. following @xcite , we decompose a dialogue agent into four subtasks: 1. dialogue state tracking (dst): generate the new belief state, for the current turn based on the previous belief state, the last two agent dialogue acts, and the current user utterance. 2. api call detection (acd): determine if an api call is necessary to query the database. 3. dialogue act generation (dag): generate the agent dialogue act based on the current belief state, the last two agent dialogue acts, the user utterance, and the result from the api call. 4. response generation (rg): convert the agent dialogue act to produce the new agent utterance. ## dataset creation in this section, we describe the process used to extend risawoz to the new languages. the original risawoz dataset is in chinese. we manually translate the validation data (600 dialogues), test data (600 dialogues), and 1% of the training dataset (100 dialogues), which we refer to as few-shot, from chinese to english. for other languages, we use english as the source language, since bilingual speakers of english and the target language are more accessible than chinese and the target language. since the english data is manually translated, this approach avoids double translationese @xcite and ensures the best data quality. we machine-translate the english data and manually post-edit the translation for fluency and correctness. besides the few-shot data, we also machine-translate all of the chinese training data into each of the languages (including english) and train with them; we refer to training with just this data set as zero-shot, since no human labor is used during dataset creation. in the following, we discuss the steps and methods for preparing data for translation, including building alignment between entities and performing iterative quality checks. we also describe how to create the target language ontology, which serves as a database for api calling and provides a mapping between source and target language entities. ## experiment the goal of our experiments is to create an agent in a target language, given full training data in the source (chinese) language, and a varying amount of training data in the target language. we also assume we have access to a machine translation model from chinese to the target language. we perform our experiments on different target languages in x-risawoz. ## results and discussion we first evaluate the models for each turn, assuming that all previous subtasks and steps are correct. we then evaluate the end-to-end accuracy for the whole conversation. ## conclusion this paper presents a solution for balancing the trade-offs between standard machine translation and human post-editing. by standardizing and establishing best practices for "translation with manual post-editing", and releasing associated toolkits, post-editing can be made faster, more efficient, and cost-effective. we use our methodology to create x-risawoz, a new end-to-end, high-quality, and large multi-domain multilingual dialogue dataset, covering 5 diverse languages and 1 code-mixed language. we also provide strong baselines for zero/few-shot creation of dialogue agents via crosslingual transfer. in the few-shot setting, our agents achieve between 60.7-84.6% on dst, 38.0-70.5% on da, and 28.5-46.4% on rg subtasks across different languages. overall, our work paves the way for more efficient and cost-effective development of multilingual task-oriented dialogue systems.