A Systematic Study of Performance Disparities in Multilingual Task-Oriented Dialogue Systems (2023)

Abstract: Achieving robust language technologies that can perform well across the world’s many languages is a central goal of multilingual NLP. In this work, we take stock of and empirically analyse task performance disparities that exist between multilingual task-oriented dialogue (ToD) systems. We first define new quantitative measures of absolute and relative equivalence in system performance, capturing disparities across languages and within individual languages. Through a series of controlled experiments, we demonstrate that performance disparities depend on a number of factors: the nature of the ToD task at hand, the underlying pretrained language model, the target language, and the amount of ToD annotated data. We empirically prove the existence of the adaptation and intrinsic biases in current ToD systems: e.g., ToD systems trained for Arabic or Turkish using annotated ToD data fully parallel to English ToD data still exhibit diminished ToD task performance. Beyond providing a series of insights into the performance disparities of ToD systems in different languages, our analyses offer practical tips on how to approach ToD data collection and system development for new languages.

## introduction the aim of task-oriented dialogue (tod) @xcite @xcite is to model the interaction between a human user and a system agent with the goal of accomplishing specific, well-defined tasks. to date, tod technology has proven useful for many sectors, ranging from the hospitality industry @xcite to healthcare @xcite , online shopping @xcite , banking @xcite , and travel @xcite , among others. these systems provide users with access to state-of-the-art services and they catalyse technological expansion. the development of tod systems requires largescale, in-domain datasets to exploit the potential of deep learning-based components to effectively handle complex dialogue patterns @xcite @xcite . the creation of datasets for new domains and languages is challenging @xcite : it requires expertise and substantial time and financial investment that typically exceeds the requirements of other nlp tasks @xcite . therefore, the progress in tod is still largely confined to a small number of high-resource languages @xcite . on the other hand, recent advances in multilingual pretrained language models (mplms) @xcite @xcite have conceptually enabled cross-lingual transfer between any two or more languages seen at pretraining @xcite @xcite , or even to unseen languages @xcite . these models offer a promising basis for developing tod systems for previously unsupported languages, obviating the need for expensive, language-specific data acquisition. however, mplms do not equally benefit all languages: previous studies focused on standard nlp tasks @xcite have revealed that they exhibit unequal performance across languages, with particularly adverse effects observed for low-resource languages. the scarcity or complete absence of high-quality in-domain tod data (i.e., adaptation bias; @xcite ) and the compounding underrepresentation of target languages inherent to mplms (i.e., intrinsic bias; bommasani et al. 2021) are likely to collectively contribute to the exacerbation of performance disparities in multilingual tod systems. these disparities, specifically error rate disparities @xcite , may result in extrinsic harms for downstream applications @xcite , affecting the utility, experience, and satisfaction of system users. they may also deprive non-english speakers of equal opportunities, hampering their ability to benefit from the rapid and ever-growing advancements in language technology. aligned with the broader goal (council of european union, 2018) of developing multilingually inclusive nlp, this work offers a first systematic analysis of the performance disparities exhibited by multilingual tod systems. 1 in §3, we first propose two measures for studying performance disparity, namely the absolute and relative equivalence in system performance. the former evaluates the disparities in task performance across languages, while the latter measures the achieved performance compared to its potential 'upper bound' performance within the same language. we apply these measures to study performance disparities in a recently developed multi3woz resource @xcite , a large-scale multilingual multi-domain multi-parallel tod dataset. in particular, relying on multi3woz, we investigate the following research questions: we hope that the key findings of this study, beyond empirically validating the current performance disparities in multilingual tod systems, will also serve as practical guidelines for tod system developers, contributing to the long-term goal of constructing robust and inclusive tod systems for a much larger number of languages, aligning with the recent initiatives in mt @xcite and other nlp tasks @xcite . our data and code are available at: https://github.com/ cambridgeltl/multi3woz/tree/analysis . developing tod systems without the disadvantage of data scarcity foot_0 necessitates a dataset that has the following properties @xcite : (i) large-scale data, containing a sufficient volume of (both training and test) data for in-depth comparative cross-language analyses, (ii) comprehensive task-specific annotations, supporting the training of multiple tod tasks, (iii) natural-sounding conversations, ensuring natural flow within the dialogues and avoiding artificial performance inflation @xcite , and (iv) wide language coverage, enabling controlled experiments and comparison across a representative set of languages and in different linguistic contexts. the majority of the existing tod datasets fail to simultaneously satisfy all the aforementioned properties. for example, most existing multilingual tod datasets @xcite @xcite @xcite , inter alia) have been designed to support a single component within a tod system, typically natural language understanding (nlu), failing to fulfil property (ii). recently, several datasets have emerged that support multiple dialogue tasks, but again none of these satisfy all the properties mentioned above. for instance, some datasets such as multi2woz @xcite do not provide any training data at all, while others like @xcite only provide a minimal training set of post-edited machine-translated dialogues. datasets such as globalwoz @xcite rely solely on automatically created machine-translated training data, failing to fulfil properties (i) and (iii). bitod @xcite is reasonably large and features coherent dialogues, but it spans only two, highest-resourced languages, failing (iv). our study is enabled by a new multilingual tod dataset that overcomes the limitations of earlier datasets: multi3woz-the first large-scale, multilingual multi-domain multi-parallel tod dataset that has all of the properties discussed above and can, therefore, serve as a comprehensive benchmark and a departure point for our analysis. for completeness, we provide a summary description of multi3woz in §3.3. ## experimental setup our experiments involve three standard tod tasks: natural language understanding (nlu), dialogue state tracking (dst), and natural language generation (nlg). we briefly recap each task along with its experimental setup. 5 nlu. this task is commonly decomposed into two well-established subtasks: intent detection (id) and slot labeling (sl). in id, the objective is to classify the user's utterance and to determine the presence of a specific domain-intent pair from a predefined set of intents in the ontology. it is treated as a multiclass classification task. sl is a sequence tagging task that identifies the presence of a value and its corresponding slot within the utterance. 6 we evaluate id and sl methods implemented atop xlm-r base and xlm-r large @xcite . the evaluation is conducted by measuring the accuracy of correctly identifying the presence of all domain-intent pairs, as well as its f1 score. for sl, we employ the commonly used bio labelling scheme to annotate each token in the user's utterance. the evaluation of sl is based on f1 score, precision, and recall in accurately identifying each slot value within the utterance. ## results and discussion centred around the definitions of θ-equivalence, our experiments are focused on answering research questions from §1. our primary objective is to identify, quantify, and mitigate performance disparities within multilingual tod systems. to solidify our empirical findings, we present supplementary experimental results in appendix b; they offer additional support to our main claims as well as auxiliary findings. to answer rq1, we conduct evaluations of mplms in three standard scenarios: fully supervised, zero-shot cross-lingual transfer, and translate train. our empirical results highlight the necessities of language-specific data in the development of a tod system for a new language. in response to rq2, our subsequent analysis highlights performance disparities among task-specific dialogue models, even when trained with equivalent amounts of data. we further examine model performances in both few-shot and 'many'-shot settings, shedding light on the correlations between task complexity and the required amount of data, thus addressing rq3. finally, we investigate the advantages of collecting a dataset that maximises lexical coverage, achieving improved cost-efficiency of annotation, answering rq4. to answer rq1, we examine the performance of individual modules in multilingual tod systems under three scenarios: (1) the ideal situation, where a multi-parallel dataset in the target language d tgt is available in equal size and quality as english, (2) the real life situation for most languages, where there is a complete absence of in-domain in-language training data, and (3) the standard translate-train situation for languages that have a reliable machine translation (mt) system but lack d tgt . in the latter case, we employ an mt system to generate a dataset d tgt mt in the target language from d eng . in scenario (1), we adopt a fully supervised approach for training and evaluating individual languages separately. for example, we train and evaluate systems p ara (•) exclusively on the arabic dataset d ara . in scenario (2), we perform zeroshot cross-lingual transfer evaluation for each tarmt for id, sl, dst, and nlg tasks on multi3woz. performance evaluations were conducted for two distinct model categories: large models, namely xlm-r large and mt5 large , and small models, specifically xlm-r base and mt5 small . the reported results follow the format of "large model small model ". for example, the entry 93.2 92.0 denotes xlm-r large achieves 93.2 accuracy in english id whereas xlm-r base achieves 92.0. get language. to achieve this, we train an english system p eng (•) using the d eng dataset, and subsequently assess its performance on the target language datasets, namely d ara , d fra , and d tur . lastly, in scenario (3), we utilise the 'f&e' proportion of the globalwoz dataset @xcite as d tgt mt . this dataset leverages a google translate to convert english utterances into the target language while maintaining the slot values associated with entities in english. 8 for translate-train, we observe that the system trained using the mt-ed dataset d tgt mt underperforms the fully supervised systems p tgt (•). interestingly, the translate-train system performs worse than the english system p eng (•), even in the zeroshot cross-lingual transfer setup, for id and sl. 9 e 0.2 0.4 0.6 0.8 1.0 1.2 absolute θ-equivalence 0.95 eng ara fra tur figure 1: a cross-language comparison for absolute equivalence in performance for each supervised system p tgt (•) with respect to the fully supervised english system p eng (•). the id and sl models are based on xlm-r large , while the dst and nlg models are based on mt5 small . the performance is reported based on taskmetric pairs, such as evaluating id with accuracy. the 0.95-equivalence line requires a system to achieve 95% of the p eng (•) performance (see §3.2). in sum, our empirical results highlight the crucial role of acquiring high-quality in-domain inlanguage training data when developing tod systems for a new language. ## conclusion and future work we presented a systematic analysis of performance disparities in modern multilingual tod systems, utilising proposed quantitative measures of absolute and relative equivalence in performance. by addressing four key research questions, our empirical investigation revealed the presence of adaptation and intrinsic biases in current tod systems and provided insights on how to best carry out data collection and system development for new languages. this study opens up new avenues for future research, including mitigating biases, improving system robustness, and expanding tod systems across languages and domains.