M ulti2 WOZ : A Robust Multilingual Dataset and Conversational Pretraining for Task-Oriented Dialog (2022)

Abstract: Research on (multi-domain) task-oriented dialog (TOD) has predominantly focused on the English language, primarily due to the shortage of robust TOD datasets in other languages, preventing the systematic investigation of cross-lingual transfer for this crucial NLP application area. In this work, we introduce Multi2WOZ, a new multilingual multi-domain TOD dataset, derived from the well-established English dataset MultiWOZ, that spans four typologically diverse languages: Chinese, German, Arabic, and Russian. In contrast to concurrent efforts, Multi2WOZ contains gold-standard dialogs in target languages that are directly comparable with development and test portions of the English dataset, enabling reliable and comparative estimates of cross-lingual transfer performance for TOD. We then introduce a new framework for multilingual conversational specialization of pretrained language models (PrLMs) that aims to facilitate cross-lingual transfer for arbitrary downstream TOD tasks. Using such conversational PrLMs specialized for concrete target languages, we systematically benchmark a number of zero-shot and few-shot cross-lingual transfer approaches on two standard TOD tasks: Dialog State Tracking and Response Retrieval. Our experiments show that, in most setups, the best performance entails the combination of (i) conversational specialization in the target language and (ii) few-shot transfer for the concrete TOD task. Most importantly, we show that our conversational specialization in the target language allows for an exceptionally sample-efficient few-shot transfer for downstream TOD tasks.

## introduction task-oriented dialog (tod) is arguably one of the most popular natural language processing (nlp) application areas @xcite , with more importance recently given to more realistic, and thus, multi-domain conversations @xcite , in which users may handle more than one task during the conversation, e.g., booking a taxi and making a reservation at a restaurant. unlike many other nlp tasks (e.g., @xcite @xcite , the progress towards multilingual multi-domain tod has been hindered by the lack of sufficiently large and high-quality datasets in languages other than english @xcite and more recently, chinese @xcite . this lack can be attributed to the fact that creating tod datasets for new languages from scratch or via translation of english datasets is significantly more expensive and time-consuming than for most other nlp tasks. however, the absence of multilingual datasets that are comparable (i.e., aligned) across languages prevents a reliable estimate of effectiveness of cross-lingual transfer techniques in multi-domain tod @xcite . in order to address these research gaps, in this work we introduce multi 2 woz, a reliable and large multilingual evaluation benchmark for multidomain task-oriented dialog, derived by translating the monolingual english-only multiwoz data @xcite to four linguistically diverse major world languages, each with a different script: arabic (ar), chinese (zh), german (de), and russian (ru). compared to the products of concurrent efforts that derive multilingual datasets from english mul-tiwoz @xcite , our multi foot_1 woz is: (1) much larger -we translate all dialogs from development and test portions of the english multiwoz (in total 2,000 dialogs containing the total of 29.5k utterances); (2) much more reliable -complete dialogs, i.e., utterances as well as slot-values, have been manually translated (without resorting to error-prone heuristics), and the quality of translations has been validated through quality control steps; and (3) parallel -the same set of dialogs has been translated to all target languages, enabling the direct comparison of the performance of multilingual models and crosslingual transfer approaches across languages. we then use multi 2 woz to benchmark a range of state-of-the-art zero-shot and few-shot methods for cross-lingual transfer in two standard tod tasks: dialog state tracking (dst) and response retrieval (rr). as the second main contribution of our work, we propose a general framework for improving performance and sampleefficiency of cross-lingual transfer for tod tasks. we first leverage the parallel conversational open-subtitles corpus @xcite to carry out a conversational specialization of a prlm for a given target language, irrespective of the downstream tod task of interest. we then show that this intermediate conversational specialization in the target language (i) consistently improves the dst and rr performance in both zero-shot and few-shot transfer, and (ii) drastically improves sample-efficiency of few-shot transfer. ## cross-lingual transfer for tod the parallel nature and sufficient size of multi 2 woz allow us to benchmark and compare a number of established and novel cross-lingual transfer methods for tod. in particular, (1) we first inject general conversational tod knowledge into xlm-roberta (xlm-r; @xcite , yielding tod-xlmr ( §3.1); ( 2 ) we then propose several variants for conversational specialization of tod-xlmr for target languages, better suited for transfer in downstream tod tasks ( §3.2); (3) we investigate zero-shot and few-shot transfer for two tod tasks: dst and rr ( §3.3). ## experimental setup evaluation tasks and measures. we evaluate different multilingual conversational prlms in cross-lingual transfer (zero-shot and few-shot) for two prominent tod tasks: dialog state tracking (dst) and response retrieval (rr). dst is commonly cast as a multi-class classification task, where given a predefined ontology and dialog history (a sequence of utterances), the model has to predict the output state, i.e., (domain, slot, value) tuples @xcite . 5 we adopt the standard joint goal accuracy as the evaluation measure: at each dialog turn, it compares the predicted dialog states against the manually annotated ground truth which contains slot values for all the (domain, slot) candidate pairs. a prediction is considered correct if and only if all predicted slot values exactly match the ground truth. rr is a ranking task that is well-aligned with the rs objective and relevant for retrieval-based tod systems @xcite : given the dialog context, the model ranks n dataset utterances, including the true response to the context (i.e., the candidate set includes the one true response and n -1 false responses). we follow @xcite and report the results for @xmath0, i.e., the evaluation measure is recall at the top 1 rank given 99 randomly sampled false responses, denoted as r 100 @1. ## results and discussion we now present and discuss the downstream crosslingual transfer results on multi 2 woz for dst and rr in two different transfer setups: zero-shot transfer and few-shot transfer. ## reproducibility to ensure full reproducibility of our results and further fuel research on multilingual tod, we release the parameters of tod-xlmr within the huggingface repository as the first publicly available multilingual prlm specialized for tod. 7 we also release our code and data and provide the annotation guidelines for manual post-editing and quality control utilized during the creation of multi 2 woz in the appendix. this makes our approach completely transparent and fully reproducible. all resources developed as part of this work are publicly available at: https://github.com/umanlp/multi2woz . ## conclusion task-oriented dialog (tod) has predominantly focused on english, primarily due to the lack of robust tod datasets in other languages @xcite , preventing systematic investigations of cross-lingual transfer methodologies in this crucial nlp application area. to address this gap, in this work, we have presented multi 2 woz -a robust multilingual multi-domain tod dataset. multi 2 woz encompasses gold-standard dialogs in four languages (german, arabic, chinese, and russian) that are directly comparable with development and test portions of the english multiwoz dataset, thus allowing for the most reliable and comparable estimates of cross-lingual transfer performance for tod to date. further, we presented a framework for multilingual conversational specialization of pretrained language models that facilitates cross-lingual transfer for downstream tod tasks. our experiments on multi 2 woz for two prominent tod tasks -dialog state tracking and response retrieval -reveal that the cross-lingual transfer performance benefits from both (i) intermediate conversational specialization for the target language and (ii) few-shot cross-lingual transfer for the concrete downstream tod task. crucially, we show that our novel conversational specialization