Specializing Multilingual Language Models: An Empirical Study (2021)

Abstract: Pretrained multilingual language models have become a common tool in transferring NLP capabilities to low-resource languages, often with adaptations. In this work, we study the performance, extensibility, and interaction of two such adaptations: vocabulary augmentation and script transliteration. Our evaluations on part-of-speech tagging, universal dependency parsing, and named entity recognition in nine diverse low-resource languages uphold the viability of these approaches while raising new questions around how to optimally adapt multilingual models to low-resource settings.

## introduction research in natural language processing is increasingly carried out in languages beyond english. this includes high-resource languages with abundant data, as well as low-resource languages, for which labeled (and unlabeled) data is scarce. in fact, many of the world's languages fall into the latter category, even some with a high number of speakers. this presents unique challenges compared to high-resource languages: effectively modeling low-resource languages involves both accurately tokenizing text in such languages and maximally leveraging the limited available data. one common approach to low-resource nlp is the multilingual paradigm, in which methods that have shown success in english are applied to the union of many languages' data, foot_0 enabling transfer between languages. for instance, multilingual contextual word representations (cwrs) from language models @xcite @xcite , inter alia) are conventionally "pretrained" on large multilingual corpora before being "finetuned" directly on supervised tasks; this pretraining-finetuning approach is derived from analogous monolingual models @xcite @xcite . however, considering the diversity of the world's languages and the great data imbalance among them, it is natural to question whether the current multilingual paradigm can be improved upon for low-resource languages. indeed, past work has demonstrated that it can. for instance, @xcite find that multilingual models often lag behind non-contextualized baselines for the lowest-resource languages in their training data, drawing into question their utility in such settings. @xcite posit that this phenomenon is a result of limited model capacity, which proves to be a bottleneck for sufficient transfer to low-resource languages. in fact, with multilingual models only being pretrained on a limited set of languages, most of the world's languages are unseen by the model. for such languages, the performance of such models is even worse @xcite , due in part to the diversity of scripts across the world's languages @xcite @xcite as compared to the models' latin-centricity @xcite . nonetheless, there have been multiple attempts to remedy this discrepancy by specializing foot_1 a multilingual model to a given target low-resource language, from which we take inspiration. among them, @xcite augment the model's vocabulary to more effectively tokenize text, then pretrain on a small amount of data in the target language; they report significant performance improvements on a small set of low-resource languages. in a similar vein, @xcite propose to transliterate text in the target language to latin script to be better tokenized by the existing model, followed by additional pretraining; they observe mixed results and note that transliteration quality may be a confounding factor. we hypothesize that these two methods can serve as the basis for improvements in modeling a broad set of low-resource languages. in this work, we study the effectiveness, extensibility, and interaction of these two approaches to specialization: the vocabulary augmentation technique of @xcite and the script transliteration method of @xcite . we verify the performance of vocabulary augmentation on three tasks in a diverse set of nine low-resource languages across three different scripts, especially on non-latin scripts ( §2) and find that these gains are associated with improved vocabulary coverage of the target language. we further observe a negative interaction between vocabulary augmentation and transliteration in light of a broader framework for specializing multilingual models, while noting that vocabulary augmentation offers an appealing balance of performance and cost ( §3). overall, our results highlight several possible directions for future study in the low-resource setting. our code, data, and hyperparameters are publicly available. foot_2 ## revisiting vocabulary augmentation we begin by revisiting the vocabulary augmentation method of @xcite , which we recast more generally in light of recent work ( §2.1). we evaluate their claims on three different tasks, using a diverse set of languages in multiple scripts ( §2.2), and find that the results hold to an even more pronounced degree in unseen low-resource languages with non-latin scripts ( §2.3). ## mix-in specialization: va and transliteration we now expand on the observation made in §2.3 regarding the difficulties that mbert encounters when faced with unseen low-resource languages in non-latin scripts because of its inability to model the script. having observed that va is beneficial in such cases, we now investigate the interaction between this method and another specialization approach that targets this problem. specifically, we consider the transliteration methods of @xcite , in which unseen low-resource languages in non-latin scripts are transliterated into the latin script, often using transliteration schemes inspired by the latin orthographies of languages related to the target language. they hypothesize that the increased similarity in the languages' writing systems, combined with mbert's overall latin-centricity, provides increased opportunity for crosslingual transfer. 7 we benchmark against the unlabeled dataset instead of task-specific ones for comparability. we can view transliteration as a inverted form of vocabulary augmentation: instead of adapting the model to the needs of the data, the data is adjusted to meet the assumptions of the model. furthermore, the transliteration step is performed prior to pretraining mbert on additional unlabeled data in the target language, the same stage at which va is performed. in both cases, the ultimate goal is identical: improving tokenization and more effectively using available data. we can thus view transliteration and va as two instantiations of a more general mix-in paradigm for model specialization, whereby various transformations (mix-ins) are applied to the data and/or model prior to performing additional pretraining. these mix-ins target different components of the experimental pipeline, which naturally raises our second research question: rq2: how do the va and transliteration mix-ins for mbert compare and interact? ## related work but subsequent studies find typological similarity and parameter sharing to be better explanations @xcite . nonetheless, past work has consistently highlighted the limitations of multilingual models in the context of low-resource languages. @xcite highlight the tension between crosslingual transfer and per-language model capacity, which poses a challenge for low-resource languages that require both. indeed, wu and dredze (2020) find that mbert is unable to outperform baselines in the lowest-resource seen languages. our experiments build off these insights, which motivate the development of methods for adapting mbert to target low-resource languages. adapting language models several prior studies have proposed methods for adapting pretrained models to a downstream task. the simplest of these is to perform additional pretraining on unlabeled data in the target language @xcite @xcite , which in turn builds off similar approaches for domain adaptation @xcite . recent work uses one or more of these additional pretraining stages to specifically train modular adapter layers for specific tasks or languages, with the goal of maintaining a language-agnostic model while improving performance on individual languages @xcite @xcite . however, as @xcite note, the typological diversity of the world's languages ultimately limits the viability of this approach. on the other hand, many adaptation techniques have focused on improving representation of the target language by modifying the model's vocabulary or tokenization schemes @xcite @xcite . this is wellmotivated: artetxe et al. (2020) emphasize representation in the vocabulary as a key factor for effective crosslingual transfer, while @xcite find that mbert's tokenization scheme for many languages is subpar. @xcite further observe that for languages with unseen scripts, a large proportion of the language is mapped to the generic "unknown" wordpiece, and they propose a matrix factorization-based approach to improve script representation. wang et al. (2020) extend mbert's vocabulary with an entire new vocabulary in the target language to facilitate zero-shot transfer to low-resource languages from english. the present study most closely derives from @xcite , who select 99 wordpieces with the greatest amount of coverage to augment mbert's vocabulary while preserving the remainder; and @xcite , who transliterate target language data into latin script to improve vocabulary coverage. we deliver new insights on the effectiveness and applicability of these methods. ## conclusion we explore the interactions between vocabulary augmentation and script transliteration for specializing multilingual contextual word representations in low-resource settings. we confirm vocabulary augmentation's effectiveness on multiple languages, scripts, and tasks; identify the mix-in stage as amenable to specialization; and observe a negative interaction between vocabulary augmentation and script transliteration. our findings highlight several open questions in model specialization and low-resource natural language processing at large, motivating further study in this area. future directions for investigation are manifold. in particular, our results in this work unify the separate findings of past works, which use mbert as a case study; a natural continuation would extend these methods to a broader set of multilingual models, such as mt5 @xcite and xlm-r @xcite , in order to obtain a clearer understanding of the factors behind specialization methods' patterns of success. while we intentionally choose a set of small unlabeled datasets to evaluate on a setting applicable to the vast majority of the world's low-resource languages, we acknowl-edge great variation in the amount of unlabeled data available in different languages. continued study on the applicability of these methods to datasets of different sizes is an important future step. an interesting direction of work is to train multilingual models on data where script respresentation is more balanced, which might also allow for different output scripts for transliteration. given that the mix-in stage is an effective opportunity to specialize models to target languages, constructing mix-ins at both the data and model level that are complementary by design has potential to be beneficial. finally, future work might shed light on the interaction between different configurations of the adaptations studied here (e.g., the number of wordpiece types used in vocabulary augmentation).