M ulti3 NLU ++: A Multilingual, Multi-Intent, Multi-Domain Dataset for Natural Language Understanding in Task-Oriented Dialogue (2023)

Abstract: Task-oriented dialogue (ToD) systems have been widely deployed in many industries as they deliver more efficient customer support. These systems are typically constructed for a single domain or language and do not generalise well beyond this. To support work on Natural Language Understanding (NLU) in ToD across multiple languages and domains simultaneously, we constructed Multi3NLU++, a multilingual, multi-intent, multi-domain dataset. Multi3NLU++ extends the English-only NLU++ dataset to include manual translations into a range of high, medium, and low resource languages (Spanish, Marathi, Turkish and Amharic), in two domains (banking and hotels). Because of its multi-intent property, Multi3NLU++ represents complex and natural user goals, and therefore allows us to measure the realistic performance of ToD systems in a varied set of the world’s languages. We use Multi3NLU++ to benchmark state-of-the-art multilingual models for the NLU tasks of intent detection and slot labeling for ToD systems in the multilingual setting. The results demonstrate the challenging nature of the dataset, particularly in the low-resource language setting, offering ample room for future experimentation in multi-domain multilingual ToD setups.

## introduction task-oriented dialogue (tod) systems @xcite , in which conversational agents assist human users to achieve their specific goals, have been used to automate telephone-based and online customer service tasks in a range of domains, including travel @xcite , finance and banking @xcite , and hotel booking @xcite . tod systems are often implemented as a pipeline of dedicated modules @xcite . the natural language understanding (nlu) module performs two crucial tasks: 1) intent detection and 2) slot labelling. in the intent detection task the aim is to identify or classify the goal of the user's utterance from several pre-defined classes (or intents) @xcite . these intents are then used by the policy module @xcite to decide the next conversational move by the conversational agent in order to mimic the flow of a human-human dialogue. in the slot labeling task each token in an utterance is assigned a label describing the type of semantic information represented by the token. during this process relevant information e.g. named entities, times/dates, quantities etc., defining the crucial information of the user's utterance, is identified. although intent detection models can reach impressive performance and have been deployed in many commercial systems @xcite , they are still unable to fully capture the variety and complexity of natural human interactions, and as such do not meet the requirements for deployment in more complex industry settings @xcite . this is due in part to the limitations of existing datasets for training and evaluating tod systems. as highlighted by @xcite they are 1) predominantly limited to detecting a single intent, 2) focused on a single domain, and 3) include a small set of slot types @xcite . furthermore, the success of task-oriented dialogue is 4) often evaluated on a small set of higher-resource languages (i.e., typically english) which does not test how generalisable systems are to the diverse range of the world's languages @xcite . arguably one of the most serious limitations, hindering their deployment to more complex conversational scenarios, is the inability to handle multiple intents. in many real-world scenarios, a user may express multiple intents in the same utterance, and tod systems must be able to handle such scenarios (gangadharaiah and narayanaswamy, 2019). for example, in figure 1 , the user expresses two main intents: (i) informing that they have forgotten their pin and thus, (ii) they would like to request a new debit card instead. a single-intent detection system can detect either of the two intents (but not both), resulting in partial completion of the user's request. @xcite recently proposed a multi-label intent detection dataset to capture such complex user requests. they further propose using intent modules as intent labels that can act as sub-intent annotations. in this example, "pin" and "don't_know" compose the first intent while "request_info", "new", "debit", and "card" compose the second. the use of intent modules, due to their combinatorial power, can support more complex conversational scenarios, and also allows reusability of the annotations across multiple domains. for example, "request_info", "new", and "membership" can be reused for gyms, salons, etc. furthermore, tod systems are typically constructed for a single language. their extension to other languages is restricted by the lack of available training data for many of the world's languages. whilst the construction of multilingual tod datasets has been given some attention @xcite @xcite , these datasets often include synthetic translations in the form of post-edited machine translation output @xcite @xcite . post-editing may introduce undesirable effects and result in texts that are simplified, normalised, or exhibit interference from the source language as compared with manually translated texts @xcite . to address all of the limitations discussed above, we propose multi 3 nlu ++ , a multilingual, multi-intent, multi-domain for training and evaluating tod systems. multi 3 nlu ++ extends the recent monolingual english-only dataset nlu++, which is a multi-intent, multi-domain dataset for the banking and hotels domains. multi 3 nlu ++ adds the element of multilinguality and thus enables simultaneous cross-domain and cross-lingual training and experimentation for tod nlu as its unique property. multi 3 nlu ++ includes expert manual translations of the 3,080 utterances in nlu++ to four languages of diverse typology and data availability: spanish, marathi, turkish, and amharic. the selection of languages covers a range of language families and scripts and includes high, medium, and low-resource languages. capturing language diversity is particularly important if we wish to design multilingual tod systems that are robust to the variety of expressions used across languages to represent the same value or concept. using multi 3 nlu ++ we demonstrate the challenges involved in extending existing state-of-the-art machine translation systems and multilingual language models for nlu in tod systems. multi 3 nlu ++ is publicly available at https://huggingface.co/ datasets/uoe-nlp/multi3-nlu . ## dataset collection our data collection process focuses on creating datasets with natural and realistic conversations, avoiding many artefacts that arise from crowdsourcing or automatic translation. we ask professional translators to manually translate each sourcelanguage utterance into the four target languages in the dataset; this promotes equal opportunity for future research for all four languages, and enables comparative cross-language analysis. choice of base dataset. this work aims to create a multilingual dataset which would be useful for testing production-like systems in multiple languages. thus, we chose nlu++ @xcite as our base dataset because i) it consists of realworld examples; and ii) every example is labelled with multiple intents which has proven useful in production settings @xcite . prior work on english multi-intent classification often uses mixsnips and mixatis as benchmarks @xcite . however, the multi-intent examples in these datasets are synthetic, i.e., they are obtained through a simple concatenation of two single-intent examples, e.g., "play this song and book a restaurant". this process leads to repetitive content and unnatural examples in the datasets. in contrast, nlu++ consists of intrinsically multiintent examples such as "i cannot login to my account because i don't remember my pin.", and is much more semantically variable. languages. we selected spanish, a widely-spoken romance language, as our high-resource language. marathi, an indo-aryan language predominantly spoken in the indian state of maharashtra, is our medium-resource language. turkish, an agglutinative turkic language, may be regarded as a lowresource language from a machine translation perspective, or as a medium-resource language based on the amount of training data in xlm-r (conneau et al., 2020). foot_0 amharic, an ethiopian semitic language belonging to the afro-asiatic language family, is our low-resource language. spanish and turkish are written in latin script, marathi is written in devanagari, and amharic in ge'ez script. manual translation. the use of crowd-sourcing agencies to collect multilingual datasets has often resulted in crowd workers using a machine translation (mt) api (plus post-editing) to complete the task, or even simply transliterating the given sentence @xcite . in the case of postediting mt output, translations may exhibit posteditese -post-edited sentences are often simplified, normalised, or exhibit a higher degree of interference from the source language than manual translations @xcite . in our case, we wish to preserve the register of the original utterances, in particular with respect to the colloquial nature of many of the utterances in the original english nlu++ dataset. furthermore, we wish to collect high-quality, natural translations. we, therefore, opted to recruit professional translators to perform manual translation, via two online platforms: proz.com foot_1 and blend express. 3 we instructed our translators to treat the task as a creative writing task @xcite and maintain the colloquial nature of the utterances. we also provided instructions to annotate the spans in the generated translations. we provide the instructions given to our translators in appendix b. we recruited three translators per language; spanish translators were recruited via proz.com, and translators for the remaining languages were assigned to us by blend express. we first conducted a pilot task in which we asked professional translators to translate and annotate 50 sentences per domain. we conducted an in-house evaluation by native speakers of the respective language to verify that the translations were colloquial, that named entities were appropriately translated, and that the translation was of high quality. these evaluations were communicated to the translators. 4 after the pilot, we asked the same translators to complete the translation of the remaining utterances in the dataset. we ran an automatic checker to ensure that the slot values marked by the translators were present within their translated sentences. further corrections such as incorrect annotations were also communicated to the translators. ## baseline experiments we benchmark several state-of-the-art approaches on multi 3 nlu ++ to 1) provide reference points for future dataset use and 2) demonstrate various aspects of multilingual dialogue nlu systems which can be evaluated using the dataset, such as crosslingual and cross-domain generalisation. we provide baseline numbers for intent detection and slot labelling and analyse the performance across languages and methods, and for different sizes of training data. we follow the main experimental setups from @xcite where possible, but we extend them to multilingual contexts. ## conclusion we collected multi 3 nlu ++ , a dataset that facilitates multilinugual, multi-label, multi-domain nlu for task-oriented dialogue. our dataset incorporates core properties from its predecessor nlu++ @xcite : a multi-intent and slot-rich ontology, a mixture of generic and domain-specific intents for reusability, and utterances that are based on complex scenarios. we investigated these properties in a multilingual setting for spanish, marathi, turkish, and amharic. we implemented mlp-based and qa-based baselines for intent detection and slot labelling across different data setups, transfer learning setups, and multilingual models. from a wide set of observations, we highlight that (i) there is a significant drop in performance across all languages as compared to nlu++ with performance drops increasing as we progress from high-to low-resource languages; (ii) zero-shot performance improves when the source language has lower resources in the mlp setup; (iii) cross-lingual transfer in the qa-based intent detection is dependent on matching the script of the source language and amount of data during pretraining setup. we hope that the community finds this dataset valuable while working on advancing research in multilingual nlu for task-oriented dialogue.