Multi 3 WOZ : A Multilingual, Multi-Domain, Multi-Parallel Dataset for Training and Evaluating Culturally Adapted Task-Oriented Dialog Systems (2023)

Abstract: Creating high-quality annotated data for task-oriented dialog (ToD) is known to be notoriously difficult, and the challenges are amplified when the goal is to create equitable, culturally adapted, and large-scale ToD datasets for multiple languages. Therefore, the current datasets are still very scarce and suffer from limitations such as translation-based non-native dialogs with translation artefacts, small scale, or lack of cultural adaptation, among others. In this work, we first take stock of the current landscape of multilingual ToD datasets, offering a systematic overview of their properties and limitations. Aiming to reduce all the detected limitations, we then introduce Multi3WOZ, a novel multilingual, multi-domain, multi-parallel ToD dataset. It is large-scale and offers culturally adapted dialogs in 4 languages to enable training and evaluation of multilingual and cross-lingual ToD systems. We describe a complex bottom–up data collection process that yielded the final dataset, and offer the first sets of baseline scores across different ToD-related tasks for future reference, also highlighting its challenging nature.

## introduction and motivation task-oriented dialog (tod), where a human user engages in a conversation with a system agent with the aim of completing a concrete task, is one of the central objectives, hallmarks, and applications of machine intelligence @xcite @xcite . tod technology has been proven useful across a wide * equal contribution. † equal senior contribution. wider developments in tod have been hampered by the two conflicting requirements: 1) large-scale in-domain datasets are crucially required in order to unlock the potential of deep learning-based tod components and systems to handle complex dialog patterns @xcite ; at the same time, 2) data collection for tod is known to be notoriously difficult as it is extremely timeconsuming, expensive, and requires expert and domain knowledge @xcite . put simply, the creation of tod datasets for new domains and languages incurs significantly higher time and budget costs than for most other nlp tasks @xcite . consequently, the progress in tod until recently has been limited only to a small number of high-resource languages such as english and chinese @xcite . recent work has recognized the need to expand the reach of multilingual tod technology to more languages via collecting multilingual tod data @xcite . yet, as discussed in more detail later in §2, all the currently available multilingual tod datasets suffer from one or several serious limitations: (i) the predominant reliance on translation-based data creation that introduces issues with 'translationese' and artificial performance inflation @xcite ; (ii) lack of cultural in this work, we address all the aforementioned limitations of current multilingual tod datasets and present a large-scale data collection process that resulted in a novel large-scale multilingual dataset for tod: multi 3 woz. the departure point of our data collection is the established multi-domain english multiwoz dataset @xcite , that is, its cleaned version 2.3 in particular @xcite . multi 3 woz is then created via adapting a recent bottom-up outline-based approach of @xcite which bypasses (the issues of) the translation-based design and discerns between language-agnostic abstract dialog schemata (i.e., outlines) and adapted, language-specific surface realizations of the underlying schemata (i.e., the actual user and system utterances). we validate the usefulness and feasibility of the outline-based approach to multilingual tod data creation for the first time on a large scale, and prove its feasibility for such large-scale endeavors: the dataset contains a total of 494,116 dialog turns created manually by human subjects. guided by the need to tackle the present limitations, multi 3 woz is the first multilingual tod dataset with the following crucial properties; see also furthermore, to guide future research, we set reference scores across different tod tasks in all the languages of multi 3 woz, running a representative set of standard baselines in each relevant tod task. the results clearly indicate the challenging nature of the dataset; we also outline the differences in performance across different languages. ## multi 3 woz versus limitations of current multilingual tod datasets we now delve deeper into the main benefits of multi 3 woz, characterizing how its key properties make it a unique tod resource. the summary and statistics of the most relevant prior work are provided in ## conclusion we have introduced a large-scale, culturally adapted, multilingual, and multi-parallel training and evaluation framework for tod, which covers ∼495,000 dialog turns over 4 languages. the dataset was motivated by the limitations of current tod datasets in multilingual setups, which we systematically analyzed as one contribution of this work. owing to its unique set of properties and scale, beyond initial analyses and experiments conducted in this work, we hope that multi 3 woz will inspire a wide array of further developments in modeling, analysis, and interpretability of multilingual and cross-lingual multi-domain tod. for instance, future work could replicate the data collection process to expand the dataset to even more languages (including low-resource ones). further, one could analyze the performance disparities observed in tables 2-4 within each language-specific tod system, as well as explore methods to mitigate such disparities, e.g., through the utilization of cross-lingual transfer techniques. future work could also explore evaluation metrics beyond the ones explored in this work, e.g., it would be interesting to explore the correlation between the increase in evaluation scores in multilingual tod systems and the resulting performance gain in terms of factors such as utility, user experience, and user satisfaction. additionally, it would be important to investigate how tod systems should, ideally, be constructed and evaluated across different languages to ensure their inclusiveness and robustness in diverse linguistic our diligent annotators for their significant efforts and contributions to this work. furthermore, we would like to express our gratitude to the tacl editors and anonymous reviewers for their insightful feedback, which greatly improved the quality of this paper.