D e M u X : Data-efficient Multilingual Learning (2024)

Abstract: Pre-trained multilingual models have enabled deployment of NLP technologies for multiple languages. However, optimally fine-tuning these models under an annotation budget, such that performance on desired target languages is jointly maximized, still remains an open question. In this paper, we introduce DeMuX, a framework that prescribes the exact data-points to label from vast amounts of unlabelled multilingual data, having unknown degrees of overlap with the target set. Unlike most prior works, our end-to-end framework is language-agnostic, accounts for model representations, and supports multilingual target configurations. Our active learning strategies rely upon distance and uncertainty measures to select task-specific neighbors that are most informative to label, given a model. DeMuX outperforms strong baselines in 84% of the test cases, in the zero-shot setting of disjoint source and target language sets (including multilingual target pools), across three models and four tasks. Notably, in low-budget settings (5-100 examples), we observe gains of up to 8-11 F1 points. Our code is released here: https://github.com/simran-khanuja/demux.

## introduction picture this: company y, a healthcare technology firm in india, has recently expanded their virtual assistance services to cover remote locations in nepal and bhutan. unfortunately, their custom-trained virtual assistant is struggling with the influx of new multilingual data, most of which is in dzongkha and tharu, but unindentifiable by non-native researchers at y. how can they improve this model? following current approaches, they first attempt to discern the languages the data belongs to, but commercial language identification systems (langid) are incapable of this task foot_1 . assuming this hurdle is crossed, company y then seeks out annotators fluent in these languages, but this also fails given crowd-sourcing platforms' lack of support for the above languages foot_2 . as an alternative, they decide to use tools that identify best languages for transfer, but these either rely on linguistic feature information -missing for dzongkha and tharu foot_3 @xcite , past model performances -expensive to obtain @xcite or don't support multilingual targets @xcite . based on annotator availability, they eventually choose nepali and tibetan as optimal transfer languages, and collect unlabelled corpora from news articles, social media and online documents. even assuming all the preceding challenges are surmounted, a final question remains unaddressed by the traditional pipeline: how do they select the exact data points to give to annotators for best performance in their domain-specific custom model, under a fixed budget? in this work, we aim to provide a solution to the above problem by introducing demux, an end-to-end framework that replaces the pipelined approach to multilingual data annotation (figure 1 ). by directly selecting datapoints to annotate, demux bypasses several stages of the pipeline, that are barriers for most languages. the alleviation of needing to identify the target languages itself (step 1: figure 1 ), implies that it can be used for noisy, unidentifiable, or code-mixed targets. demux makes decisions at the instance level by using information about the pre-trained multilingual language model's (multilm's) representation space. this ensures that the data annotation process is aware of the model ultimately being utilized. concretely, we draw from the principles of active learning (al) @xcite for guidance on model-aware criteria for point selection. al aims to identify the most informative points (to a specific model) to label from a stream of unlabelled source data. through iterations of model training, data acquisition and human annotation, the goal is to achieve satisfactory performance on a target test set, labelling only a small fraction of the data. past works @xcite @xcite have leveraged al in the special case where the same language(s) constitute the source and target set (step 2 (upper branch): figure 1 ). however, none so far have considered the case of source and target languages having unknown degrees of overlap; a far more pervasive problem for real-world applications that commonly build classifiers on multi-domain data @xcite . from the al lens, this is particularly challenging since conventional strategies of choosing the most uncertain samples @xcite , could pick distracting examples from very dissimilar language distributions @xcite . our strategies are designed to deal with this distribution shift by leveraging small amounts of unlabelled data in target languages. in the rest of the paper, we first describe three al strategies based on the principles of a) semantic similarity with the target; b) uncertainty; and c) a combination of the two, which picks uncertain points in target points' local neighborhood ( §3). we experiment with tasks of varying complexity, categorized based on their label structure: tokenlevel (ner and pos), sequence-level (nli), and question answering (qa). we test our strategies in a zero-shot setting across three multilms and five target language configurations, for a budget of 10,000 examples acquired in five al rounds ( §4). we find that our strategies outperform previous baselines in most cases, including those with multilingual target sets. the extent varies, based on the budget, the task, the languages and models ( §5). overall, we observe that the hybrid strategy performs best for token-level tasks, but picking globally uncertain points gains precedence for nli and qa. to test the applicability of demux in resource constrained settings, we experiment with lower budgets ranging from 5-1000 examples, acquired in a single al round. in this setting, we observe gains of upto 8-11 f1 points for tokenlevel tasks, and 2-5 f1 for complex tasks like nli and qa. ## notation assume that we have a set of source languages, l @xmath0. . . l n s }, and a set of target languages, l @xmath1. . . l m t }. l s and l t are assumed to have unknown degrees of overlap. further, let us denote the corpus of unlabelled source data as x @xmath2. . . x n s } and the unlabelled target data as x @xmath3. . . x m t }. our objective is to label a total budget of b data points over k al rounds from the source data. the points to select in each round can then be calculated by @xmath4. thus considering the super-set s @xmath5, our objective is to select some x * ∈ s b according to an appropriate criterion. ## annotation strategies based on the broad categorizations of al methods as defined by zhang et al. (2022), we design three annotation strategies that are either representationbased, information-based, or hybrid. the first picks instances that capture the diversity of the dataset; the second picks the most uncertain points which are informative to learn a robust decision boundary; and the third focuses on optimally combining both criteria. in contrast to the standard al setup, there are two added complexities in our framework: a) source-target domain mismatch; b) multiple distributions for each of our target languages. we therefore design our measures to select samples that are semantically similar (from the perspective of the multilm) to the target domain @xcite . all strategies build upon reliable distance and uncertainty measures, whose implementation varies based on the type of task, i.e. whether the task is token-level, sequence-level or question answering. a detailed visualization of how these are calculated can be found in §a.1. below, we formally describe the three strategies, also detailing the motivation behind our choices. ## experimental setup our setup design aims to address the following: q1) does demux benefit tasks with varying complexity? which strategies work well across different task types? ( §4.1) q2) how well does demux perform across a varied set of target languages? can it benefit multilingual target pools as well? ( §4.2) q3) how do the benefits of demux vary across different multilms? ( §4.3) ## results how does demux perform overall? we present results for ner, pos, nli and qa in tables 3, 4, 5 and 6, respectively. in summary, the bestperforming strategies outperform best performing baselines in 84% of the cases, with variable gains dependant on the task, model and target languages. in the remaining cases, the drop is within 1% absolute delta from the best-performing baseline. how does demux fare on multilingual target pools? we observe consistent gains given multilingual target pools as well (geo and lpp). we believe this is enabled by the language-independent design of our strategies, which makes annotation decisions at a per-instance level. this has important consequences, since this would enable researchers, like those at company y, to better models for all the languages that they care about. does the model select data from the same languages across tasks? no! we find that selected data distributions vary across tasks for the same target languages. for example, when the target language is urdu, demux chooses 70-80% of samples from hindi for nli and pos, but prioritizes farsi and arabic (35-45%) for ner. despite hindi and urdu's syntactic, genetic, and phonological similarities as per lang2vec, their differing scripts underscore the significance of script similarity in ner transfer. this also proves that analysing data selected by demux can offer linguistic insights into the learned task-specific representations. which strategies work well across different task types? our hybrid strategy, which picks uncertain points in the local neighborhood of target points, performs best for token-level tasks, whereas globally uncertain points maximize performance for nli and qa. for nli, both average-dist and uncertainty outperform baselines, the former proving more effective. on further analysis, we find that this is an artifact of the the nature of the dataset which is balanced across three labels, and is strictly parallel. this makes average-dist select high-uncertainty points at decision boundaries' ## related work multilingual fine-tuning: traditionally models were fine-tuned on english, given the availability of labeled data across all tasks. however, significant transfer gaps were observed across languages @xcite leading to the emergence of two research directions. the first emphasizes the significance of using few-shot target language data @xcite and the development of strategies for optimal few-shot selection @xcite . the second focuses on choosing the best source languages for a target, based on linguistic features @xcite or past model performances @xcite . discerning a globally optimal transfer language however, has been largely ambiguous @xcite and the language providing for highest empirical transfer is at times inexplicable by known linguistic relatedness criteria @xcite . by making decisions at a data-instance level rather than a language level, demux removes reliance on linguistic features and sidesteps ambiguous consensus on how multilms learn cross-lingual relations, while prescribing domain-relevant instances to label. active learning for nlp: al has seen wide adoption in nlp, being applied to tasks like text classification @xcite , named entity recognition @xcite @xcite , and machine translation @xcite , among others. in the multilingual context, past works @xcite @xcite have applied al to selectively label data in target languages. however, they do not consider cases with unknown overlap between source and target languages. this situation, similar to a multi-domain al setting, is challenging as data selection from the source languages may not prove beneficial for the target @xcite . ## conclusion in this work, we introduce demux, an end-toend framework that selects data to label from vast pools of unlabelled multilingual data, under an annotation budget. demux's design is languageagnostic, making it viable for cases where source and target data do not overlap. we design three strategies drawing from al principles that encompass semantic similarity with the target, uncertainty, and a hybrid combination of the two. our strategies outperform strong baselines for 84% of target language configurations (including multilingual target sets) in the extreme case of disjoint source and target languages, across three models and four tasks: ner, udpos, nli and qa. we find that semantic similarity with the target mostly benefits tokenlevel tasks, while picking uncertain points gains precedence for complex tasks like nli and qa. we further analyse demux's applicability in lowbudget settings and observe gains of up to 8-11 f1 points for some tasks, with a trend of diminishing gains for larger budgets.