Cross-Lingual Dialogue Dataset Creation via Outline-Based Generation (2023)

Abstract: Multilingual task-oriented dialogue (ToD) facilitates access to services and information for many (communities of) speakers. Nevertheless, its potential is not fully realized, as current multilingual ToD datasets—both for modular and end-to-end modeling—suffer from severe limitations. 1) When created from scratch, they are usually small in scale and fail to cover many possible dialogue flows. 2) Translation-based ToD datasets might lack naturalness and cultural specificity in the target language. In this work, to tackle these limitations we propose a novel outline-based annotation process for multilingual ToD datasets, where domain-specific abstract schemata of dialogue are mapped into natural language outlines. These in turn guide the target language annotators in writing dialogues by providing instructions about each turn’s intents and slots. Through this process we annotate a new large-scale dataset for evaluation of multilingual and cross-lingual ToD systems. Our Cross-lingual Outline-based Dialogue dataset (cod) enables natural language understanding, dialogue state tracking, and end-to-end dialogue evaluation in 4 diverse languages: Arabic, Indonesian, Russian, and Kiswahili. Qualitative and quantitative analyses of cod versus an equivalent translation-based dataset demonstrate improvements in data quality, unlocked by the outline-based approach. Finally, we benchmark a series of state-of-the-art systems for cross-lingual ToD, setting reference scores for future work and demonstrating that cod prevents over-inflated performance, typically met with prior translation-based ToD datasets.

## introduction and motivation one of the staples of machine intelligence is the ability to communicate with humans and complete a task as instructed during such an interaction. this is commonly referred to as task-oriented dialogue (tod; @xcite @xcite . despite having far-reaching applications, such as banking @xcite , travel @xcite , and healthcare @xcite , this technology is currently accessible to very few communities of speakers @xcite . the progress in multilingual tod is critically hampered by the paucity of training data for many of the world's languages. while cross-lingual transfer learning @xcite @xcite offers a partial remedy, its success is tenuous beyond typologically similar languages and generally hard to assess due to the lack of evaluation benchmarks @xcite . what is more, transfer learning often cannot leverage multi-source transfer and few-shot learning due to lack of language diversity in the tod datasets @xcite @xcite . therefore, the main driver of development in multilingual tod is the creation of multilingual resources. however, even when available, they suffer from several pitfalls. most are obtained by manual or semi-automatic translation of an english source @xcite @xcite @xcite @xcite . while this process is cost-efficient and typically makes data and results comparable across languages, it yields dialogues that lack naturalness @xcite , are not properly localized nor culture-specific @xcite . further, they provide over-optimistic estimates of performance due to the artificial similarity between source and target texts @xcite . as an alternative to translation, new tod datasets can be created from scratch in a target language through the wizard-of-oz framework (woz; kelley to address all these gaps, in this work we devise a novel outline-based annotation pipeline for multilingual tod datasets that combines the best of both processes. in particular, abstract dialogue schemata, specific to individual domains, are sampled from the english schema-guided dialogue dataset (sgd; @xcite . then, the schemata are automatically mapped into outlines in english, which describe the intention that should underlie each dialogue turn and the slots of information it should contain, as shown in to confirm the advantages of the leveraged annotation process, we run a proof-of-concept experiment where we create two analogous datasets through the outline-based pipeline and manual translation, respectively. based on a quality sur-vey from human participants, we find that, while having similar annotation speed, outline-based annotation achieves significantly higher naturalness and familiarity of concepts and entities, without compromising data quality and language fluency. foot_0 finally, crucial evidence showed that cross-lingual transfer test scores on translation-based data are over-estimated. we demonstrate that this is due to the fact that the distribution of the sentences (and their hidden representations) is considerably more divergent between training and evaluation dialogues in cod than in the translation-based dataset. further, to establish realistic estimates of performance on multilingual tod, we benchmark a series of state-of-the-art multilingual tod models in different tod tasks on cod. among other findings, we report that zero-shot transfer surpasses 'translate-test' on slot labeling, but this trend is reversed for intent detection. language-specific performance also varies substantially among evaluated models, depending on the quantity of unlabeled data available for pretraining. in sum, cod provides a typologically diverse dataset for end-to-end dialogue modeling and evaluation, and streamlines a scalable annotation process that results in natural and localised dialogues. we hope that cod will contribute to democratizing dialogue technology and facilitating reliable cost-effective tod systems for a wide array of languages. our data and code are available at github.com/cambridgeltl/cod. ## related work although a number of nlu resources have recently emerged in languages other than english, the availability of high-quality, multi-domain data to support multilingual tod is still inconsistent @xcite . translation of english data has been the predominant method for generating examples in other languages: for example, the atis corpus @xcite boasts translations into chinese @xcite , vietnamese @xcite , spanish, german, indonesian, and turkish, among others (susanto and lu, 2017; @xcite . bottom-up collection of tod data directly in the target language has been the less popular choice (e.g., in french @xcite and chinese @xcite ). concurrent work by fitzgerald et al. (2022) employs translation as part of a dataset creation workflow where amazon mturk workers first translate or localize slot values, and subsequently translate or localize entire phrases in which these slots appear. while localization allows improving the geographical and cultural relevance of entities mentioned in dialogues, this approach still relies on translation from english, thus perpetuating many of the problems of earlier translation-based methods: for example, introducing english grammatical and lexical biases in dialogue utterances @xcite or compromising target language idiomacity. as we demonstrate in §4, our outline-based dialogue generation method addresses these issues by eschewing direct translation in favor of guided dialogue creation in the target language, ensuring naturalness of linguistic expressions used in each language and yielding a dataset better capturing linguistic diversity. thus far, the focus of existing benchmarks has been predominantly either on monolingual multi-domain @xcite @xcite or multilingual single-domain evaluation @xcite , rather than balancing diversity along both these dimensions. moreover, the current multilingual datasets are mostly constrained to the two nlu tasks of intent detection and slot labeling @xcite , and do not enable evaluations of e2e tod systems in multilingual setups. in order to adequately assess the strengths and generalizability of nlu as well as dst and e2e models, they should be tested both on multiple languages and multiple domains, a goal pursued in this work. ## annotation design we selected the english schema-guided dialogue (sgd) dataset @xcite as a starting point due to its scale (20k human-assistant dialogues) and diversity (20 domains). it was constructed via automatic generation of dialogue schemata combined with manual creation of dialogue paraphrases by crowdworkers, organized as lists of turns for each individual interaction, each turn containing an utterance by the user or system. the accompanying annotations are grouped into frames, each corresponding to a single api or service (e.g., banks 2). in turn, each service is represented as a schema including its characteristic functions (intents) and parameters (slots), as well as their natural language (nl) descriptions. 2 we first assessed the viability of our method on russian, collecting data using (i) direct translation from english and (ii) our proposed outline-based approach. we then applied our method to three other languages that boast a large number of speakers and yet suffer from a shortage of resources: arabic, indonesian, and kiswahili, ensuring the dataset's diversity in terms of language family and macro-area, as well as writing systems (cyrillic, arabic, and latin scripts), see ## translation versus outline-based the main motivation behind the outline-based approach is to avoid the known pitfalls of direct translation and produce evaluation data better representing the linguistic and cultural realities of each language in the sample. to verify whether the method satisfies these goals in practice, we carried out a trial experiment consisting in parallel dialogue data creation using two different methods, (i) direct translation and (ii) outline-based generation, starting from the same sample of source sgd dialogues to ensure a fair comparison. in (i), randomly sampled (see §3.1) english user/system utterances were extracted directly from the sgd data with accompanying slot and intent annotations and subsequently translated into the target language by professional translators, also responsible for validating target language slot spans. in (ii), we automatically extracted dialogue frames, including intents and slots, matching dialogue ids we also asked the participants to time themselves while working on the task. notably, we found the annotation speed to be identical for the two methods, averaging 15 seconds per single dialogue turn (dialogue writing + slot annotation). while the translation approach does not require any creative input in terms of cultural adaptations of slot values, the outline-based approach allows freedom in terms of the linguistic expressions used, which results in similar time requirements. ## baselines, results, discussion cod includes labeled data for three standard tod tasks: i) natural language understanding (nlu; intent detection and slot labeling); ii) dialogue state tracking (dst); and iii) end-to-end (e2e) dialogue modeling. here, we benchmark a representative selection of state-of-the-art models ( §5.1) on our new dataset, highlighting its potential for evaluation and the key challenges it presents across different tasks and experimental setups ( §5.2). ## conclusion and outlook we have presented and validated a 'bottom-up' method for the creation of multilingual taskoriented dialogue (tod) datasets. the key idea is to map domain-specific language-independent dialogue schemata to natural language outlines, which in turn guide human dialogue generators to create natural target-language utterances, for the user and system alike. we have empirically demonstrated that the proposed outline-based approach yields more natural and culturally sensitive dialogues than the standard translation-based approach to multilingual tod data creation. moreover, we have proven that the standard translation-based approaches often yield over-inflated and unrealistic performance in multilingual evaluation, while this issue is removed with the outline-based generation method. our proposed approach yielded a new cross-lingual outline-based dialogue dataset (termed cod), which covers 5 typologically diverse languages, 11 domains in total, and enables evaluations in standard nlu, dst, and end-to-end tod tasks. thus, cod is an important step towards challenging multilingual and multi-domain tod evaluation in future research. we have also evaluated a series of state-of-the-art models for the different tod tasks, setting baseline reference points, and revealing the challenging nature of the dataset with ample room for improvement. we hope that our work will inspire future research across multiple aspects. one such area concerns cultural debiasing of the concepts and situations captured in the dialogues. our method addresses this through cultural adaptations and replacements of foreign concepts with those common in the annotators' culture and environment. the next step should involve a careful selection of dialogue scenarios based on their relevance and plausibility in the culture in question, as very recently started in other nlp areas (e.g., @xcite . in this work, we presented useful practices and insights hoping to guide similar (potentially larger-scale) data creation efforts in tod for other, especially lower-resource, languages, and domains. cod is available online at github.com /cambridgeltl/cod.