Multi 2 WOZ: A Robust Multilingual Dataset and Conversational Pretraining for Task-Oriented Dialog
Abstract
Research on (multi-domain) task-oriented dialog (TOD) has predominantly focused on the English language, primarily due to the shortage of robust TOD datasets in other languages, preventing the systematic investigation of cross-lingual transfer for this crucial
NLP application area.
In this work, we introduce M ULTI 2 WOZ, a new multilingual multidomain TOD dataset, derived from the wellestablished English dataset M ULTI WOZ, that spans four typologically diverse languages:
Chinese, German, Arabic, and Russian.
In contrast to concurrent efforts (Ding et al., 2021;
Zuo et al., 2021), M ULTI 2 WOZ contains goldstandard dialogs in target languages that are directly comparable with development and test portions of the English dataset, enabling reliable and comparative estimates of cross-lingual transfer performance for TOD.
We then introduce a new framework for multilingual conversational specialization of pretrained language models (PrLMs) that aims to facilitate crosslingual transfer for arbitrary downstream TOD tasks.
Using such conversational PrLMs specialized for concrete target languages, we systematically benchmark a number of zero-shot and few-shot cross-lingual transfer approaches on two standard TOD tasks: Dialog State Tracking and Response Retrieval.
Our experiments show that, in most setups, the best performance entails the combination of (i) conversational specialization in the target language and (ii) few-shot transfer for the concrete TOD task.
Most importantly, we show that our conversational specialization in the target language allows for an exceptionally sample-efficient fewshot transfer for downstream TOD tasks.
Introduction
Task-oriented dialog (TOD) is arguably one of the most popular natural language processing (NLP) application areas (Yan et al., 2017; Henderson et al., 2019, inter alia ), with more importance recently given to more realistic, and thus, multi-domain conversations (Budzianowski et al., 2018; Ramadan et al., 2018), in which users may handle more than one task during the conversation, e.g., booking a taxi and making a reservation at a restaurant .
Unlike many other NLP tasks (e.g., Hu et al., 2020; Liang et al., 2020; Ponti et al., 2020, inter alia ), the progress towards multilingual multi-domain TOD has been hindered by the lack of sufficiently large and high-quality datasets in languages other than
English (Budzianowski et al., 2018; Zang et al., 2020) and more recently, Chinese (Zhu et al., 2020).
This lack can be attributed to the fact that creating TOD datasets for new languages from scratch or via translation of English datasets is significantly more expensive and time-consuming than for most other NLP tasks.
However, the absence of multilingual datasets that are comparable (i.e., aligned) across languages prevents a reliable estimate of effectiveness of cross-lingual transfer techniques in multi-domain TOD (Razumovskaia et al., 2021).
In order to address these research gaps, in this work we introduce M ULTI 2 WOZ, a reliable and large multilingual evaluation benchmark for multidomain task-oriented dialog, derived by translating the monolingual English-only MultiWOZ data (Budzianowski et al., 2018;
Eric et al., 2020) to four linguistically diverse major world languages, each with a different script: Arabic (AR), Chinese (ZH), German (DE), and Russian (RU).
Compared to the products of concurrent efforts that derive multilingual datasets from English MultiWOZ (Ding et al., 2021;
Zuo et al., 2021), our
M ULTI 2 WOZ is: (1) much larger – we translate all dialogs from development and test portions of the English MultiWOZ (in total 2,000 dialogs containing the total of 29.5K utterances); (2) much more reliable – complete dialogs, i.e., utterances as well as slot-values, have been manually translated (without resorting to error-prone heuristics), and the quality of translations has been validated through quality control steps; and (3) parallel – the same set of dialogs has been translated to all target languages, enabling the direct comparison of the performance of multilingual models and crosslingual transfer approaches across languages.
We then use M ULTI 2 WOZ to benchmark a range of state-of-the-art zero-shot and few-shot methods for cross-lingual transfer in two standard TOD tasks: Dialog State Tracking (DST) and Response Retrieval (RR).
As the second main contribution of our work, we propose a general framework for improving performance and sampleefficiency of cross-lingual transfer for TOD tasks.
We first leverage the parallel conversational OpenSubtitles corpus (Lison and Tiedemann, 2016) to carry out a conversational specialization of a PrLM for a given target language, irrespective of the downstream TOD task of interest.
We then show that this intermediate conversational specialization in the target language (i) consistently improves the DST and RR performance in both zero-shot and few-shot transfer, and (ii) drastically improves sample-efficiency of few-shot transfer.
Multi 2 WOZ
In this section we describe the construction of the
M ULTI 2 WOZ dataset, providing also details on inter-translator reliability.
We then discuss two concurrent efforts in creating multilingual TOD datasets from MultiWOZ and their properties, and emphasize the aspects that make our M ULTI 2 WOZ a more reliable and useful benchmark for evaluating cross-lingual transfer for TOD.
2. Dataset Creation
Language Selection.
We translate all 2,000 dialogs from the development and test portions of the
English MultiWOZ 2.1 (Eric et al., 2020) dataset to Arabic (AR), Chinese (ZH), German (DE), and Russian (RU).
We selected the target languages based on the following criteria: (1) linguistic diversity (DE and RU belong to different Indo-European
subfamilies – Germanic and Slavic, respectively; ZH is a Sino-Tibetan language and AR Semitic), (2) diversity of scripts (DE and RU use Latin and Cyrillic scripts, respectively, both alphabet scripts; AR script represents the Abjad script type, whereas the ZH Hanzi script belongs to logographic scripts), (3) number of native speakers (all four are in the top 20 most-spoken world languages), and (4) our access to native and fluent speakers of those languages who are proficient in English.
Two-Step Translation.
Following the wellestablished practice, we carried out a two-phase translation of the English data: (1) we started with an automatic translation of the dialogs – utterances as well as the annotated slot values – followed by (2) the manual post-editing of the translations.
We first automatically translated all utterances and slot values from the development and test dialogs from the MultiWOZ 2.1 (Eric et al., 2020) (1, dialogs in each portion; 14,748 and 14,744 utterances, respectively) to our four target languages, using Google Translate.
1 We then hired two native speakers of each target language, 2 all with a University degree and fluent in English, to post-edit the (non-overlapping sets of) automatic translations, i.e., fix the errors in automatic translations of utterances as well as slot values.
Since we carried out the automatic translation of the utterances independently of the automatic translation of the slot values, the translators were instructed to pay special attention to the alignment between each translated utterance and translations of slot value annotations for that utterance. We show an example utterance with associated slot values after the automatic translation and manual post-editing in Table 1.
Quality Control.
In order to reduce the translation costs, our human post-editors worked on disjoint sets of dialogs. Because of this, our annotation process contained an additional quality assurance step.
Two new annotators for each target language judged the correctness of the translations on the random sample of 200 dialogs (10% of all translated dialogs, 100 from the development and test portion each), containing 2,962 utterances in total.
The annotators had to independently answer the following questions for each translated utterance from the sample: (1) Is the utterance translation acceptable?
and (2) Do the translated slot values match the translated utterance?
On average, across all target languages, both quality annotators [Table] Original No hold off on
[Table]
cineworld cinema Can you help me find an attraction called cineworld cinema?
Automatic Trans.
目
前 暂 无 预 订 。 您 能 帮 我找 到 一个 名 为 cineworld Cinema 的 景 点 吗 ？ Cineworld 电 影
Manual Correc.
目
前 暂 无 预 订 。 您 能 帮 我找 到 一个 名 为 电 影 世 界电 影 院 的 景 点 吗 ？ 电 影 世 界电 影 院 for the respective language answered affirmatively to both questions for 99% of all utterances.
Adjusting for chance agreement, we measured the Inter-Annotator Agreement (IAA) in terms of Cohen’s κ (Cohen, 1960), observing the almost perfect agreement 3 of κ = 0 .
824 for the development set and κ = 0 .
838 for test set.
Annotation Duration and Cost.
In total, we hired 16 annotators, four for each of our four target languages: two for post-editing and two for quality assessment.
The overall effort spanned almost full 5 months (from July to November 2021), and amounted to 1,083 person-hours.
With the remuneration rate of 16 $/h, creating M ULTI 2 WOZ cost us $17,328.
2. Comparison with Concurrent Work
Two concurrent works also derive multilingual datasets from MultiWOZ (Ding et al., 2021; Zuo et al., 2021), with different strategies and properties, discussed in what follows.
GlobalWOZ (Ding et al., 2021) encompasses
Chinese, Indonesian, and Spanish datasets.
The authors first create templates from dialog utterances by replacing slot-value strings in the utterances with the slot type and value index (e.g., “. . .
and the post code is cb238el” becomes the template “. . .
and the post code is [attraction-postcode-1] ” .
They then automatically translate all templates to the target languages.
Next, they select a subset of 500 test set dialogs for human post-editing with the following heuristic: dialogs for which the sum of corpus-level frequencies of their constitutive 4-grams (normalized with the dialog length) is the largest.
4 Since this selection step is independent for each language, each GlobalWOZ portion contains translations of a different subset of English dialogs: this prevents any direct comparison of downstream TOD performance across languages.
Even more problematically, the selection heuristic directly reduces linguistic diversity of dialogs chosen for the test set of each language, as it favors the dialogs that contain the same globally most frequent 4-grams.
Due to this artificial homogeneity of its test sets, GlobalWOZ is very likely to overestimate downstream
TOD performance for target languages.
Unlike GlobalWOZ, AllWOZ (Zuo et al., 2021) does automatic translation of a fixed small subset of MultiWOZ plus post-editing in seven target languages.
However, it encompasses only 100 dialogs and 1,476 turns; as such, it is arguably too small to draw strong conclusions about the performance of cross-lingual transfer methods.
Its usefulness in joint domain and language transfer evaluations is especially doubtful, since it covers individual MultiWOZ domains with an extremely small number of dialogs (e.g., only 13 for the Taxi domain).
Finally, neither Ding et al.
(2021) nor Zuo et al.
(2021) provide any estimates of the quality of their final datasets nor do they report their annotation costs.
In contrast to GlobalWOZ, M ULTI 2 WOZ is a parallel corpus – with the exact same set of dialogs translated to all four target languages; as such it directly enables performance comparisons across the target languages.
Further, containing translations of all dev and test dialogs from MultiWOZ (i.e., avoiding sampling heuristics), M ULTI 2 WOZ does not introduce any confounding factors that would distort estimates of cross-lingual transfer performance in downstream TOD tasks.
Finally,
M ULTI 2 WOZ is 20 times larger (per language) than AllWOZ: experiments on M ULTI 2 WOZ are thus much more likely to yield conclusive findings.
Cross-lingual Transfer for TOD
The parallel nature and sufficient size of
M ULTI 2 WOZ allow us to benchmark and compare a number of established and novel cross-lingual transfer methods for TOD.
In particular, (1) we first inject general conversational TOD knowledge into XLM-RoBERTa (XLM-R; Conneau et al., 2020), yielding TOD-XLMR (§3.1); (2) we then propose several variants for conversational specialization of
TOD-XLMR for target languages, better suited for transfer in downstream TOD tasks (§3.2); (3) we investigate zero-shot and few-shot transfer for two
TOD tasks: DST and RR (§3.3).
3. TOD-XLMR: A Multilingual TOD Model
Recently, Wu et al.
(2020) demonstrated that specializing BERT (Devlin et al., 2019) on conversational data by means of additional pretraining via a combination of masked language modeling (MLM) and response selection (RS) objectives yields improvements in downstream TOD tasks.
Following these findings, we first (propose to) conversationally specialize XLM-R (Conneau et al., 2020), a state-of-the-art multilingual PrLM coveringlanguages, in the same manner: applying the RS and MLM objectives on the same English conversational corpus consisting of nine human-human multi-turn TOD datasets (see Wu et al.
(2020) for more details).
As a result, we obtain TOD-XLMR – a massively multilingual PrLM specialized for taskoriented conversations.
Note that TOD-XLMR is not yet specialized (i.e., fine-tuned) for any concrete TOD task (e.g., DST or Response Generation).
Rather, it is enriched with general task-oriented conversational knowledge (in English), presumed to be beneficial for a wide variety of TOD tasks.
3. Target-Language Specialization
TOD-XLMR has been conversationally specialized only in English data.
We next hypothesize that a further conversational specialization for a concrete target language X can improve the transfer EN → X for all downstream TOD tasks.
Accordingly, similar to Moghe et al.
(2021), we investigate several intermediate training procedures that further conversationally specialize TOD-XLMR for the target language X (or jointly for EN and X).
For this purpose, we (i) compile target-language-specific as well as cross-lingual corpora from the CCNet (Wenzek et al., 2020) and OpenSubtitles (Lison and Tiedemann, 2016) datasets and (ii) experiment with different monolingual, bilingual, and cross-lingual training procedures.
Here, we propose a novel cross-lingual response selection (RS) objective and demonstrate its effectiveness in cross-lingual transfer for downstream TOD tasks.
Training Corpora.
We collect two types of data for language specialization: (i) “flat” corpora (i.e., without any conversational structure): we simply randomly sample 100K sentences for each language from the respective monolingual portion of
CCNet (we denote with Mono-CC the individual 100K-sentence portions of each language; with BiCC the concatenation of the English and each of target language Mono-CCs , and with Multi-CC the concatenation of all five Mono-CC portions); (ii) parallel dialogs (in EN and target language X) from
OpenSubtitles (OS), a parallel conversational corpus spanning 60 languages, compiled from subtitles of movies and TV series.
We leverage the parallel OS dialogs to create two different crosslingual specialization objectives, as described next.
Training Objectives.
We directly use the CC portions (Mono-CC, Bi-CC, and Multi-CC) for standard MLM training.
We then leverage the parallel
OS dialogs for two training objectives.
First, we carry out translation language modeling ( TLM )
(Conneau and Lample, 2019) on the synthetic dialogs which we obtain by interleaving K randomly selected English utterances with their respective target language translations; we then (as with MLM), dynamically mask 15% of tokens of such interleaved dialogs; we vary the size of the context the model can see when predicting missing tokens by randomly selecting K (between 2 and 15 ) for each instance.
Second, we use OS to create instances for both monolingual and cross-lingual Response
Selection (RS) training.
RS is a simple binary classification task in which for a given pair of a context (one or more consecutive utterances) and response (a single utterance), the model has to predict whether the response utterance immediately follows the context (i.e., it is a true response) or not (i.e., it is a false response).
RS pretraining has been proven beneficial for downstream TOD in monolingual English setups (Mehri et al., 2019; Henderson et al., 2019, 2020; Hung et al., 2022).
In this work, we leverage the parallel OS data to introduce the cross-lingual RS objective, where the context and the response utterance are not in the same language.
In our experiments, we carry out both (i) monolingual RS training in the target language (i.e., both the context and response utterance are, e.g., in Chinese), denoted RS-Mono , and (ii) cross-lingual RS between English (as the source language in downstream TOD tasks) and the target language, denoted RS-X . We create hard RS neg- [Table]
atives , by coupling contexts with non-immediate responses from the same movie or episode (same imdbID ), as well as easy negatives by randomly sampling m ∈ { 1 , 2 , 3 } responses from a different movie of series episode ( i [Table] e., different imdbID ).
Hard negatives encourage the model to reason beyond simple lexical cues.
Examples of training instances for OS-based training (for EN-ZH) are shown in Table 2.
3. Downstream Cross-lingual Transfer
Finally, we fine-tune the various variants of TODXLMR, obtained through the above-described specialization (i.e., intermediate training) procedures, for two downstream TOD tasks (DST and RR) and examine their cross-lingual transfer performance.
We cover two cross-lingual transfer scenarios: (1) zero-shot transfer in which we only fine-tune the models on the English training portion of MultiWOZ and evaluate their performance on the M ULTI 2 WOZ test data of our four target languages; and (2) few-shot transfer in which we sequentially first fine-tune the models on the English training data and then on the small number of dialogs from the development set of M ULTI 2 WOZ, in similar vein to (Lauscher et al., 2020).
In order to determine the effect of our conversational target language specialization (§3.2) on the downstream sample efficiency, we run few-shot experiments with different numbers of target language training dialogs, ranging from 1% to 100% of the size of
M ULTI 2 WOZ development portions.
Experimental Setup
Evaluation Tasks and Measures.
We evaluate different multilingual conversational PrLMs in cross-lingual transfer (zero-shot and few-shot) for two prominent TOD tasks: dialog state tracking (DST) and response retrieval (RR) .
DST is commonly cast as a multi-class classification task, where given a predefined ontology and dialog history (a sequence of utterances), the model has to predict the output state, i.e., (domain, slot, value) tuples (Wu et al., 2020).
We adopt the standard joint goal accuracy as the evaluation measure: at each dialog turn, it compares the predicted dialog states against the manually annotated ground truth which contains slot values for all the (domain, slot) candidate pairs.
A prediction is considered correct if and only if all predicted slot values exactly match the ground truth.
RR is a ranking task that is well-aligned with the RS objective and relevant for retrieval-based
TOD systems (Wu et al., 2017; Henderson et al., 2019): given the dialog context, the model ranks N dataset utterances, including the true response to the context (i.e., the candidate set includes the one true response and N -1 false responses).
We follow
Henderson et al.
and report the results for
N = 100 , i.e., the evaluation measure is recall at the top 1 rank given 99 randomly sampled false responses, denoted as R 100 @1 .
Models and Baselines.
We briefly summarize the models that we compare in zero-shot and fewshot cross-lingual transfer for DST and RR.
As baselines, we report the performance of the vanilla multilingual PrLM XLM-R (Conneau et al., 2020)and its variant further trained on the English TOD data from (Wu et al., 2020): TOD-XLMR (§3.1).
quantifies the effect of conversational English pretraining on downstream TOD performance, much like the comparison between BERT and TODBERT done by Wu et al.
(2020); however, here we extend the comparison to cross-lingual transfer setups.
We then compare the baselines against a series of our target language-specialized variants, obtained via intermediate training on CC (MonoCC, Bi-CC, and Multi-CC) by means of MLM, and on OS jointly via TLM and RS (RS-X or RS-Mono) objectives (see §3.2 again).
Hyperparameters and Optimization.
For training TOD-XLMR (§3.1), we select the effective batch size of 8.
In target-language-specific intermediate training (§3.2), we fix the maximum sequence length to 256 subword tokens; for RS objectives, we limit the context and response totokens each.
We train for 30 epochs in batches of size 16 for MLM/TLM, and 32 for RS.
We search for the optimal learning rate among the following values: { 10 − 4 , 10 − 5 , 10 − 6 } .
We apply early stopping based on development set performance (patience: 3 epochs for MLM/TLM, 10 epochs for
RS).
In downstream fine-tuning, we train in batches of 6 (DST) and 24 instances (RR) with the initial learning rate fixed to 5 · 10 − 5 .
We also apply early stopping (patience: 10 epochs) based on the development set performance, training maximally for 300 epochs in zero-shot setups, and for 15 epochs in target-language few-shot training.
In all experiments, we use Adam (Kingma and Ba, 2015) as the optimization algorithm.
Results and Discussion
We now present and discuss the downstream crosslingual transfer results on M ULTI 2 WOZ for DST and RR in two different transfer setups: zero-shot transfer and few-shot transfer.
5. Zero-Shot Transfer
Dialog State Tracking.
Table 3 summarizes zeroshot cross-lingual transfer performance for DST.
First, we note that the transfer performance of all models for all four target languages is extremely low, drastically lower than the reference English
DST performance of TOD-XLMR, which stands at 47.9%.
These massive performance drops, stemming from cross-lingual transfer are in line with findings from concurrent work (Ding et al., 2021; Zuo et al., 2021) and suggest that reliable crosslingual transfer for DST is much more difficult to
[Table] accuracy (%) as the evaluation metric. Reference English DST performance of TOD-XLMR: 47.86% achieve than for most other language understanding tasks (Hu et al., 2020; Ponti et al., 2020).
Despite low performance across the board, we do note a few emerging and consistent patterns.
First,
TOD-XLMR slightly but consistently outperforms the vanilla XLM-R, indicating that conversational
English pretraining brings marginal gains.
All of our proposed models from §3.2 (the lower part of Table 3) substantially outperform TOD-XLMR, proving that intermediate conversational specialization for the target language brings gains, irrespective of the training objective.
Expectedly, TLM and RS training on parallel OS data brings substantially larger gains than MLMing on flat monolingual target-language corpora (Mono-CC) or simple concatenations of corpora from two (Bi-CC) or more languages (Multi-CC).
German and Arabic seem to benefit slightly more from the cross-lingual Response Selection training (RS-X), whereas for Chinese and Russian we obtain better results with the monolingual (target language) RS training (RS-Mono).
Response Retrieval.
The results of zero-shot transfer for RR are summarized in Table 4.
Compared to DST results, for the sake of brevity, we show the performance of only the stronger baseline (TOD-XLMR) and the best-performing variants with intermediate conversational target-language training (one for each objective type): MLM on
Mono-CC, TLM on OS, and TLM + RS-Mono on OS.
Similar to DST, TOD-XLMR exhibits a nearzero cross-lingual transfer performance for RR as well, across all target languages.
In sharp contrast to DST results, however, conversational specializa- [Table] tion for the target language – with any of the three specialization objectives – massively improves the zero-shot cross-lingual transfer for RR.
The gains are especially large for the models that employ the parallel OpenSubtitles corpus in intermediate specialization, with the monolingual (target language)
Response Selection objective slightly improving over TLM training alone.
Given the parallel nature of M ULTI 2 WOZ, we can directly compare transfer performance of both
DST and RR across the four target languages.
In both tasks, the best-performing models exhibit stronger performance (i.e., smaller performance drops compared to the English performance) for
German and Russian than for Arabic and Chinese.
This aligns well with the linguistic proximity of the target languages to English as the source language.
5. Few-Shot Transfer and Sample Efficiency
Next, we present the results of few-shot transfer experiments, where we additionally fine-tune the task-specific TOD model on a limited number of target-language dialogs from the development portion of M ULTI 2 WOZ, after first fine-tuning it on the complete English training set from MultiWOZ (see §4).
Few-shot cross-lingual transfer results, averaged across all four target languages, are summarized in Figure 1.
The figure shows the performance for different sizes of the target-language training data (i.e., number of target-language shots, that is, percentage of the target-language development portion from M ULTI 2 WOZ).
Detailed perlanguage few-shot results are given in Table 5, for brevity only for TOD-XLMR and the best targetlanguage-specialized model (TLM+RS-Mono on
OS).
We provide full per-language results for all specialized models from Figure 1 in the Appendix.
The few-shot results unambiguously show that the intermediate conversational specialization for the target language(s) drastically improves the target-language sample efficiency in the downstream few-shot transfer .
The baseline TODXLMR – not exposed to any type of conversational pretraining for the target language(s) – exhibits substantially lower performance than all three models (MLM on Mono-CC, TLM on OS, and TLM+RSMono on OS) that underwent conversational intermediate training on respective target languages.
This is evident even in the few-shot setups where the three models are fine-tuned on merely 1% ( dialogs) or 5% (50 dialogs) of the M ULTI 2 WOZ development data (after prior fine-tuning on the complete English task data from MultiWOZ).
As expected, the larger the number of taskspecific (DST or RR) training instances in the target languages (50% and 100% setups), the closer the performance of the baseline TOD-XLMR gets to the best-performing target-language-specialized model – this is because the size of the in-language training data for the concrete task (DST or RR) becomes sufficient to compensate for the lack of conversational target-language intermediate training that the specialized models have been exposed to.
The sample efficiency of the conversational targetlanguage specialization is more pronounced for RR than for DST.
This seems to be in line with the zeroshot transfer results (see Tables 3 and 4), where the specialized models displayed much larger crosslingual transfer gains over TOD-XLMR on RR than on DST.
We hypothesize that this is due to the intermediate specialization objectives (especially RS) being better aligned with the task-specific training objective of RR than that of DST.
Related Work
TOD Datasets.
Research in task-oriented dialog has been, for a long time, limited by the existence of only monolingual English datasets.
While earlier datasets focused on a single domain (Henderson et al., 2014a,b; Wen et al., 2017), the focus shifted towards the more realistic multi-domain task-oriented dialogs with the creation of the MultiWOZ dataset (Budzianowski et al., 2018), which has been refined and improved in several iterations (Eric et al., 2020; Zang et al., 2020; Han et al., 2021). Due to the particularly high costs of creating TOD datasets (in comparison with other language understanding tasks) (Razumovskaia et al., 2021), only a handful of monolingual TOD datasets [Figure] DST
RR Lang Model 1% 5% 10% 50% 100% 1% 5% 10% 50% 100%
DE TOD-XLMR 7. 49.46 TLM+RS [Table]
on OS 15.
AR
TOD-XLMR 1.
42.69 TLM+RS [Table] Mono on OS 4. ZH
TOD-XLMR 8. [Table] in languages other than English (Zhu et al., 2020) or bilingual TOD datasets have been created (Gunasekara et al., 2020; Lin et al., 2021). Mrkˇsi´c et al.
(2017b) were the first to translate 600 dialogs from the single-domain WOZ 2.0 (Mrkˇsi´c et al., 2017a) to Italian and German. Concurrent work (Ding et al., 2021; Zuo et al., 2021), which we discuss in detail in §2.2 and compare thoroughly against our M ULTI 2 WOZ [Table] introduces the first multilingual multi-domain TOD datasets, created by translating portions of MultiWOZ to several languages
[Table]
Language Specialization and Cross-lingual Transfer.
Multilingual transformer-based models (e.g., mBERT (Devlin et al., 2019), XLM-R (Conneau et al., 2020)) are pretrained on large generalpurpose and massively multilingual corpora (over 100 languages).
While this makes them versatile and widely applicable, it does lead to suboptimal representations for individual languages, a phenomenon commonly referred to as the “curse of multilinguality” (Conneau et al., 2020).
Therefore, one line of research focused on adapting (i.e., specializing ) those models to particular languages (Lauscher et al., 2020; Pfeiffer et al., 2020).
For example, Pfeiffer et al.
(2020) propose a more computationally efficient approach for extending the model capacity for individual languages: this is done by augmenting the multilingual PrLM with language-specific adapter modules.
Glavaˇs et al. (2020) perform language adaptation through additional intermediate masked language modeling in the target languages with filtered text corpora, demonstrating substantial gains in downstream zero-shot cross-lingual transfer for hate speech and abusive language detection tasks.
In a similar vein,
Moghe et al.
(2021) carry out intermediate finetuning of multilingual PrLMs on parallel conversational datasets and demonstrate its effectiveness in zero-shot cross-lingual transfer for the DST task.
Lauscher et al.
(2020) show that few-shot transfer, in which one additionally fine-tunes the PrLM on a few labeled task-specific target-language in- stances leads to large improvements for many taskand-language combinations, and that labelling a few target-language examples is more viable than further LM-specialization for languages of interest under strict zero-shot conditions.
This finding is also corroborated in our work for two TOD tasks.  Reproducibility
To ensure full reproducibility of our results and further fuel research on multilingual TOD, we release the parameters of TOD-XLMR within the Huggingface repository as the first publicly available multilingual PrLM specialized for TOD.
7 We also release our code and data and provide the annotation guidelines for manual post-editing and quality control utilized during the creation of M ULTI 2 WOZ in the Appendix.
This makes our approach completely transparent and fully reproducible.
All resources developed as part of this work are publicly available at: https://github.com/umanlp/Multi2WOZ .
Conclusion
Task-oriented dialog (TOD) has predominantly focused on English , primarily due to the lack of robust TOD datasets in other languages (Razumovskaia et al., 2021), preventing systematic investigations of cross-lingual transfer methodologies in this crucial NLP application area.
To address this gap, in this work, we have presented M ULTI 2 WOZ – a robust multilingual multi-domain TOD dataset.
M ULTI 2 WOZ encompasses gold-standard dialogs in four languages (German, Arabic, Chinese, and Russian) that are directly comparable with development and test portions of the English MultiWOZ dataset, thus allowing for the most reliable and comparable estimates of cross-lingual transfer performance for TOD to date.
Further, we presented a framework for multilingual conversational specialization of pretrained language models that facilitates cross-lingual transfer for downstream TOD tasks.
Our experiments on M ULTI 2 WOZ for two prominent TOD tasks – Dialog State Tracking and
Response Retrieval – reveal that the cross-lingual transfer performance benefits from both (i) intermediate conversational specialization for the target language and (ii) few-shot cross-lingual transfer for the concrete downstream TOD task.
Crucially, we show that our novel conversational specialization for the target language leads to exceptional sample efficiency in downstream few-shot transfer.
In hope to steer and inspire future research on multilingual and cross-lingual TOD, we make
M ULTI 2 WOZ publicly available and will extend the resource to further languages from yet uncovered language families (e.g., Turkish).
Acknowledgements
The work of Goran Glavaˇs has been supported by the Multi2ConvAI project of MWK BadenW¨urttemberg.
Simone Paolo Ponzetto has been supported by the JOIN-T 2 project of the Deutsche Forschungsgemeinschaft (DFG). Chia-Chien Hung has been supported by JOIN-T 2 (DFG) and Multi2ConvAI (MWK BW).
The work of Anne
Lauscher was funded by Multi2ConvAI and by the European Research Council (grant agreement
No. 949944, INTEGRATOR).
The work of Ivan
Vuli´c has been supported by the ERC PoC Grant
MultiConvAI (no. 957356) and a Huawei research donation to the University of Cambridge.
Ethical Considerations
In this work, we have presented M ULTI 2 WOZ, a robust multilingual multi-domain TOD dataset, and focused on the multilingual conversational specialization of pretrained language models.
Although the scope of this manuscript does not allow for an in-depth discussion of the potential ethical issues associated with conversational artificial intelligence in general, we would still like to highlight the ethical sensitivity of this area of NLP research and emphasize some of the potential harms of conversational AI applications, which propagate to our work.
For instance, issues may arise from unfair stereotypical biases encoded in general purpose (Lauscher et al., 2021) as well as in conversational (Barikeri et al., 2021) pretrained language models and from exclusion of the larger spectrum of (gender) identities (Lauscher et al., 2022).
Furthermore, (pre)training as well as fine-tuning of large-scale PrLMs can be hazardous to the environment (Strubell et al., 2019): in this context, the taskagnostic intermediate conversational specialization for the target languages that we introduce, which allows for highly sample-efficient fine-tuning for various TOD tasks can be seen as a step in the positive direction, towards the reduction of the carbon footprint of neural dialog models.
A Annotation Guidelines: Post-editing of the Translation Task Description
Multi-domain
Wizard-of-Oz dataset (MultiWOZ) (Budzianowski et al., 2018) is introduced as a fully-labeled collection of human-to-human written conversations spanning over multiple domains and topics.
Our project aims to translate the monolingual
English-only MultiWOZ dataset to four linguistically diverse major world languages, each with a different script: Arabic (AR), Chinese (ZH), German (DE), and Russian (RU).
In this annotation task, we resort to the revised version 2.1 (Eric et al., 2020) and focus on the development and test portions of the English MultiWOZ 2.1 (in total of 2,000 dialogs containing a total of 29.5K utterances).
We first automatically translate all the utterances and the annotated slot values to the four target languages, using Google
Translate.
Next the translated utterances and slot values (i.e., fix the translation errors) will be postedited with manual efforts.
For this purpose, a JSON file for development or test set will be provided to each annotator.
There are two tasks: (1) Fix the errors in automatic translations of translated utterances and the translated slot values.
(2) Check the alignment between each translated utterance and the slot value annotations for that utterance.
JSON Representation
The JSON file will be structured as follows, feel free to use any JSON editor tools (e.g., JSON Editor Online) to annotate the files.
Annotation data
• dialogID : An unique ID for each dialog.
• turnID :
The turn ID of the utterance in the dialog.
• services : Domain(s) of the dialog.
• utterance : English utterance from MultiWOZ.
• SlotValues :
English annotated slot values from MultiWOZ.
• transUtterance : Translated utterance from Google Translate.
• transSlotValues :
Translated slot values from
Google Translate.
Annotation
Task
• fixTransUtterance : The revised translated utterance with manual efforts.
• fixTransSlotValues : The revised translated slot values with manual efforts.
• changedUtterance : Whether the translated utterance is changed.
Annotate as 1 if the translated utterance is revised, 0 otherwise.
• changedSlotValues : Whether the translated slot values is changed.
Annotate as 1 if the translated slot values are revised, 0 otherwise.
Annotation Example Example 1: Name Correction and Mismatch
The following example in Chinese shows the error fixed with the translated name issue, and also the correctness of the mismatch case between the translated utterance and translated slot values.
dialogID : MUL0484.json turnID :
services : train, attraction utterance : No hold off on booking for now.
Can you help me find an attraction called cineworld cinema?
slotValues : { attraction-name: cineworld cinema } transUtterance : 目
前 暂 无 预 订 。 您 能 帮 我找 到 一个 名 为 cineworld Cinema 的 景 点 吗 ？ transSlotValues : { attraction-name: Cineworld 电 影 } fixTransUtterance : 目
前 暂 无 预 订 。 您 能 帮 我 找 到 一个 名
为 电 影 世 界电 影 院 的 景 点 吗 ？ fixTransSlotValues : { attraction-name: 电 影 世 界电 影 院 } changedUtterance :changedSlotValues :
Example 2: Grammatical Error
The following example in German shows the error corrected based on the grammatical issue of the translated utterance.
dialogID : PMUL1072.json turnID :services : train, attraction utterance : I’m leaving from Cambridge.
slotValues : { train-departure: cambridge } transUtterance : Ich verlasse Cambridge.
transSlotValues : { train-departure: cambridge } fixTransUtterance : Ich fahre von Cambridge aus.
fixTransSlotValues : { train-departure: cambridge } changedUtterance :changedSlotValues :
Additional Notes
There might be some cases of synonyms.
For example, in Chinese 周 五 and 星 期 五 both have the same meaning as Friday in English, also similarly in Russian regarding the weekdays.
In this case, just pick the most common one and stays consistent among all the translated utterances and slot values.
Besides there might be some language variations across different regions, please ignore the dialects and metaphors while fixing the translation errors.
If there are any open questions that you think are not covered in this guide, please do not hesitate to get in touch with me or post the questions on Slack, so these issues can be discussed together with other annotators and the guide can be improved.
B Annotation Guidelines: Quality Control Task Description
Multi-domain
Wizard-of-Oz dataset (MultiWOZ) (Budzianowski et al., 2018) is introduced as a fully-labeled collection of human-to-human written conversations spanning over multiple domains and topics.
Our project is aimed to translate the monolingual English-only MultiWOZ dataset to four linguistically diverse major world languages, each with a different script: Arabic (AR), Chinese (ZH), German (DE), and Russian (RU).
In the previous annotation task, we resorted to the revised version 2.1 (Eric et al., 2020) and focused on the development and test portions of the English MultiWOZ 2.1.
According to the translation process, it was processed in two steps: we first automatically translated all the utterances and the annotated slot values to the four target languages, using Google Translate.
Next the translated utterances and slot values (i.e., fix the translation errors) were post-edited with manual efforts from native speakers of each language.
Additionally, a quality assurance step is required to check the quality of the post-edited translation.
For this purpose, a JSON file for a random sample 200 dialogs (100 from the development and test set each), containing 2,962 utterances in total will be provided to two annotators for each target language to judge the correctness of the translations.
Each annotator has to independently answer the following questions for each translated utterance from the sample: (1) Is the utterance translation acceptable?
(2) Do the translated slot values match the translated utterance?
Annotation data
• dialogID : An unique ID for each dialog.
• turnID :
The turn ID of the utterance in the dialog.
• utterance : English utterance from MultiWOZ.
• SlotValues :
English annotated slot values from MultiWOZ.
fixTransUtterance : The revised translated utterance with manual efforts.
• fixTransSlotValues : The revised translated slot values with manual
efforts.
Annotation Task
• UtteranceAcceptable : Is the utterance translation acceptable?
Annotate as 1 if the translated utterance is acceptable, 0 otherwise.
• SlotValuesMatchAcceptable :
Do the translated slot values match the translated utterance?
Annotate as 1 if the translated slot values are acceptable, 0 otherwise.
• NOTE :
Extra notes of judgement.
Annotation Example
Small grammatical errors, but still catch the meaning will be considered acceptable .
However, if the whole meaning regarding the translation change, it will then be considered as not acceptable .
Example 1: Ambiguity
The following example shows the ambiguity issues regarding the translated utterance.
In German, table can be translated into Tabelle as a table form or Tisch as a table for reservation.
Regarding the contextual information from the utterance, the correct translation should be Tisch instead of Tabelle in this case.
Therefore, the translated utterance will be considered as not acceptable, and annotated as 0.
dialogID : PMUL2464.json turnID :
utterance : Yes, Bedouin is a restaurant that serves African food in the Centre.
It is in the expensive range.
Would you like to book a table?
slotValues : { restaurant-name: bedouin } fixTransUtterance : Ja, Beduine ist ein Restaurant, das afrikanisches Essen im Zentrum serviert.
Es liegt im teuren Bereich.
M ¨ ochten Sie eine
Tabelle reservieren?
fixTransSlotValues : { restaurant-name: Beduine }
UtteranceAcceptable :
SlotValuesMatchAcceptable : Example : Grammatical Error The following example shows a slight grammatical issue regarding the translated utterance.
This is mainly with the synonym case in Chinese, where the place can be translated into 地 方 or 位 置 , while 位 置 will be more appropriate in this scenario.
However, 地 方 still keep the semantic meaning.
Therefore, the translated utterance will be considered as acceptable, and annotated as 1.
And further checking with the translated slot values, all are correct, and should be annotated as 1.
dialogID : PMUL0400.json turnID :utterance : Please book the place for 7 people at 11:30 on the same day.
slotValues : { restaurant-people: 7 , restauranttime: 11:30 , restaurant-day: Monday } fixTransUtterance : 请 于 当 天 11:30 预 订 7 人 的 地 方 。 fixTransSlotValues : { restaurant-people: 7 , restaurant-time: 11:30 , restaurant-day: 周 一 }
UtteranceAcceptable :
SlotValuesMatchAcceptable :
Additional Notes
Please ignore the slot values with “dontcare”, “not mentioned” and “none”, while checking the translation quality.
If there are any open questions that you think are not covered in this guide, please do not hesitate to get in touch with me or post the questions on Slack, so these issues can be discussed together with other annotators and the guide can be improved.
C Additional Experiments
[Table]
[Table]