On Efficiently Acquiring Annotations for Multilingual Models (2022)

Abstract: When tasked with supporting multiple languages for a given problem, two approaches have arisen: training a model for each language with the annotation budget divided equally among them, and training on a high-resource language followed by zero-shot transfer to the remaining languages. In this work, we show that the strategy of joint learning across multiple languages using a single model performs substantially better than the aforementioned alternatives. We also demonstrate that active learning provides additional, complementary benefits. We show that this simple approach enables the model to be data efficient by allowing it to arbitrate its annotation budget to query languages it is less certain on. We illustrate the effectiveness of our proposed method on a diverse set of tasks: a classification task with 4 languages, a sequence tagging task with 4 languages and a dependency parsing task with 5 languages. Our proposed method, whilst simple, substantially outperforms the other viable alternatives for building a model in a multilingual setting under constrained budgets.

## introduction while neural networks have become the de-facto method of tackling nlp tasks, they often require a lot of annotated data to do so. this task of data annotation is especially challenging while building systems aimed at serving numerous languages. motivated by this, in this paper, we tackle the following problem: given the requirement of building systems for an nlp task in a multilingual setting with a fixed annotation budget, how can we efficiently acquire annotations to perform the task well across multiple languages? the traditional approach to this problem has been building a separate model to serve each language. in this scenario, the annotation budget ˚equal contribution is split equally for all languages, and a model is trained for each one separately. recently, another direction that has gained popularity has been leveraging multilingual pre-trained language models (mplms) which inherently map multiple languages to a common embedding space @xcite . the popular method for leveraging these models has been leveraging their zero-shot transfer ability: training on an english-only corpus for the task, and then using the models zero-shot for the other languages. another orthogonal line of work aimed at building models under a constrained budget has been active learning (al) @xcite . while this has shown to improve annotation efficiency, the predominant approach has been to train one model per language, using the (language specific) model for al @xcite . in this work, we show that a single mplm trained on all languages simultaneously performs much better than training independent models for specific languages for a fixed total annotation budget. further, while the benefits of using al in conjunction with mplms has been studied for a monolingual setup @xcite , we show that al also yields benefits in the multilingual setup. concretely, we show that an al acquisition on a single language helps improve zero-shot performance on all other languages, regardless of the language of the seed data. furthermore, we show that al also yields benefits for our proposed single model scenario. we demonstrate that our results are consistent on 3 different tasks across multiple languages: classification, sequence tagging and dependency parsing. our approach removes the requirement of maintaining n different models, and uses 1{n th the parameters than when independent models are trained. our analysis reveals that the model arbitrates between different languages based on its performance to form a multilingual curricu-lum. we release our code at https://github. com/codedecde/smal . ## related work effective utilization of annotation budgets has been the area of focus for numerous active learning works, showing improvements for different tasks like pos tagging @xcite , sentiment analysis @xcite @xcite , syntactic parsing @xcite , and named entity recognition @xcite . the focus of most of these works, however, has been on learning for a single language (often english). prior work on al that uses a multilingual setup or cross-lingual information sharing and that goes beyond training a separate model for each language has thus been limited. the closest work where multiple languages influence each other's acquisition is that of @xcite ; however, they still train a separate model for each language. for transfer to multiple languages, recent advances in building mplms @xcite @xcite have been extremely effective, especially in zero-shot transfer @xcite . @xcite studied the dataeffectiveness of these models when used in conjunction with al, but, as with other al work, with a single language focus. finally, @xcite studied the effectiveness of the zero-shot setup, showing that adding a few examples to a model trained on english improves performance over zero-shot transfer. however, this assumes the availability of a full english task-specific corpus. ## results and analysis model performance figure 1 shows the performance of ner on spanish (refer appendix g for the plots of all other languages and tasks). although acquiring data independently per language (mma) performs well, sma outperforms mma. unsurprisingly, monoa with es performs the best in the category, since it allocates its entire budget to acquiring es data; it thus forms an upper-bound of the model performance. however, sma outperforms monoa when its seed language and inference language differ. finally, al consistently provides gains over random acquisition. to analyze the performance across all languages, we present the performance for each round of acquisition, aggregated across all languages for classification (figure 2 ) (refer appendix g for dependency parsing and ner plots). here, sma consistently outperforms mma for every round of acquisition because mma suffers from a poorly utilized budget, potentially wasting annotation budget on languages where the task is easier. in contrast, sma improves budget utilization while also benefiting from cross-lingual information. finally, sma, by virtue of performing well irrespective of language, consistently outperforms monoa. for a concise overview, we present the aggregate metrics across all rounds for each task in figure 2: performance aggregated across all languages for one task (classification) at every round of acquisition. as can be seen, sma ˘al outperforms all other baselines. note that sma and mma both out-perform monoa. this is because monoa does not perform as well when the language is different than that for which data was acquired. here, lc is the al method adopted for classification. ## conclusion in this work, we consider the problem of efficiently building models that solve a task across multiple languages. we show that, contrary to traditional approaches, a single model arbitrating between multiple languages for data acquisition considerably improves performance in a constrained budget scenario, with al providing additional benefits.