Targeted Multilingual Adaptation for Low-resource Language Families (2024)

Abstract: Massively multilingual models are known to have limited utility in any one language, and to perform particularly poorly on low-resource languages. By contrast, targeted multinguality has been shown to benefit low-resource languages. To test this approach more rigorously, we systematically study best practices for adapting a pre-trained model to a language family. Focusing on the Uralic family as a test case, we adapt XLM-R under various configurations to model 15 languages; we then evaluate the performance of each experimental setting on two downstream tasks and 11 evaluation languages. Our adapted models significantly outperform mono- and multilingual baselines. A regression analysis reveals that adapted vocabulary size is relatively unimportant for low-resource languages, and that low-resource languages can be aggressively up-sampled during training at little detriment to performance in high-resource languages. These results introduce new best practices for performing language adaptation in a targeted setting.

## introduction while multilingual models are susceptible to the so-called "curse of multilinguality" -the observation that overall model performance decreases as more languages are added in pre-training @xcite -it is generally accepted that low-resource languages benefit from some multilinguality during training, especially when added languages are similar in some way @xcite @xcite . this paper contributes to a growing line of research studying targeted multilingualism as a more practical approach to building robust models for mid-and low-resource languages @xcite @xcite by analyzing what factors matter most for this approach. in this work, we systematically evaluate the best technique for adapting a pre-trained multilingual model (xlm-r) to a language family. we use the uralic family as a case study -like many families, it includes a few mid-resource languages (e.g. hungarian, finnish) as well endangered languages like sámi and erzya, which are extremely data-scarce. our primary adaptation techniques are multilingual language-adaptive pre-training (lapt; @xcite and vocabulary replacement/specialization @xcite . our experiments show that both techniques are necessary for robust adapation to the uralic family. we demonstrate that adaptation to a language family is both more efficient than training monolingual models, and performs as well or better on downstream tasks. we also conduct a regression analysis to assess the impact of lapt steps, adapted vocabulary size, and language sampling alpha in order to recommend best practices for adapting models to targeted multilingual groupss. notable results include the fact that specialized vocabularies as small at 16k tokens outperform the cross-lingual xlm-r vocabulary (with 250k tokens), and low-resource languages can be aggressively up-sampled during training without significant degradation of high-resource performance. our contributions are as follows: 1) we train models adapted for the uralic family that significantly outperform monolingual and multilingual baselines for almost all languages. 2) we conduct a statistical analysis of important parameters for multilingual adaptation to test their relative effects on downstream task performance. 3) we use this analysis to make best-practice recommendations 4) we make all of our adaptation code, configurations, analysis results, and bestperforming uralic model(s) publicly available at https://github.com/clmbrs/targeted-xlms . other approaches involve training new, languagespecific adapter layers to augment a frozen monolingual @xcite or multilingual encoder @xcite @xcite . a comparison of these cross-lingual adaptation approaches @xcite found that continued pre-training often outperforms more complex setups, even in low-resource settings. @xcite investigate the transferability of monolingual bert models to uralic languages specifically. they find that vocabulary overlap (and script) is extremely important for transfer success, rather than the language-relatedness of the source model. model vocabulary and script a major limitation to adapting models to new languages is the vocabulary, which often fails to cover unseen scripts @xcite or tokenizes target text inefficiently @xcite . however, @xcite demonstrate that script is a critical factor in predicting transfer success. many adaptation techniques have been proposed to overcome this issue, such as extending the vocabulary with new tokens @xcite @xcite or retraining the vocabulary and embeddings from scratch @xcite . other work reuses information in pre-trained embeddings rather than randomly initializing new ones. these include scaling up embedding spaces from smaller target language models (de vries and nissim, 2021; @xcite or copying embeddings from the original vocabulary where there is exact overlap @xcite . we follow recent works that re-initialize the vocabulary (and embeddings) based on the structure of the original model's embedding space @xcite @xcite . dobler and de melo (2023) introduce the focus algorithm, which initializes new token embeddings as a linear combination of the old embeddings for the most semantically similar tokens, as computed by an auxiliary embedding model. alternatively, downey et al. (2023) propose heuristics for initializing a new embedding matrix based on script-wise distributions, which perform similarly to more complex techniques like focus. targeted multilingualism recent work has proposed targeted or linguistically-informed multilingual models instead of the "massively-multilingual" approach covering as many languages as feasible. notably, @xcite show that while massively multilingual models hurt individual language performance, low-resource languages in particular benefit from limited multilinguality, especially when the added languages are syntactically similar (e.g. have similar word order). our work systematically analyzes which factors are responsible for the success of targeted multilingual adaptation. we focus on the model adaptation paradigm since cross-lingual models learn useful language-general patterns that can be leveraged for a "warm-start" to training @xcite . unlike @xcite ; senel et al. (2024), we specialize model vocabulary for the target language(s), since cross-lingual tokenizers typically perform poorly for low-resource languages @xcite . we follow dobler and de melo (2023) and @xcite in using a vocabulary specialization technique that leverages the struc-ture of the original model embedding space, while creating a new vocabulary that is directly optimized for the target languages, in contrast to @xcite , which simply uses a subset of the original model vocabulary. finally, we follow @xcite in conducting adaptation for a language family, while keeping in mind the observation from senel et al. (2024) that including a high-resource language during adaptation can be advantageous. this comes naturally with our chosen testbed of the uralic family, which contains both high-and low-resource languages. ## experiments we adapt models using language-adaptive pre-training (lapt, @xcite languages first, we obtain raw-text lapt data for as many uralic languages as possible. for the high-resource languages (estonian, finnish, hungarian, and russian), all training data is sourced from the multilingual oscar corpus v.22.01 @xcite , which also contains a small amount of text for the low-resource languages komi (koi) and mari (mhr/mrj). we further source low-resource language data from monolingual splits of the opus translation corpus @xcite and the johns hopkins university bible corpus @xcite . we also include russian as a high-resource language, though it is not uralic. many uralic languages are spoken by ethnic minorities within russia and the former soviet union using modified forms of the russian cyrillic alphabet. the lack of a high-resource uralic language written in cyrillic could be a problem for low-resource performance, since script overlap is a vital ingredient in crosslingual transfer @xcite . further, russian is a major source of loan words for uralic languages and an official language throughout russian territory @xcite . during training, we sample languages according to a multinomial distribution parameterized by the hyperparameter α @xcite .; see figure 1 ). languages are sampled by sentence, allowing multiple languages to be potentially sampled in each batch. vocabulary replacement to specialize the model's vocabulary for target languages, we first train a new sentencepiece model @xcite on 5 million lines sampled from the training set, foot_0 testing vocabulary sizes of 16k, 32k, and 64k tokens. 2 we train multilingual tokenizers with a consistent sampling parameter of α = 0.2. 3 we re-initialize the model's embedding matrix for the new vocabulary using the focus algorithm (dobler and de melo, 2023). training all experiments use xlm-r base as a starting point @xcite . we conduct lapt on the multilingual uralic dataset for 100k, 200k, or 400k steps. when training a new vocabulary, the transformer blocks are frozen for the first for our shortest experiments (100k steps) we test four values of α: {0.1, 0.2, 0.3, 0.4}. for longer experiments, we test only the two most promising values: {0.1, 0.2}. because the data ratio between our high and low-resource languages is so extreme ( task evaluation we evaluate model performance with part of speech (pos) tagging accuracy as well as unlabeled attachment score (uas) on universal dependencies (ud) treebanks (de marneffe et al., 2021). 5 treebanks exist for all high-resource languages plus erzya, north sámi, komi, karelian, livvi, moksha, and skolt sámi. models are fine-tuned for each task over four random seeds. because the amount of fine-tuning data varies considerably over languages, we consider three evaluation settings: few-shot, full-finetune, and zero-shot. for few-shot, models are fine-tuned on 512 sampled sentences per language. for fullfinetune, models are fine-tuned on all examples in a given language (ranging from 896 sentences for erzya to 32,768 for russian). zero-shot setting is baselines our simplest baseline is "off-the-shelf" xlm-r -the pre-trained model from @xcite without modification. we also test xlm-r adapted with lapt, but without vocabulary specialization. lapt alone is a strong baseline, but as downey et al. (2023) note, training a large cross-lingual vocabulary incurs considerable extra computation compared to a smaller, specialized one. given cross-lingual tokenizers are often inefficient and ineffective for low-resource languages @xcite , we hypothesize a specialized vocabulary will show a performance advantage as well as reduced computational cost. we also train baselines adapted to single languages, adapting xlm-r with lapt and a vocab size of 16k (per language). we assume a shared computational "budget" of 400k training steps, where steps are allocated across languages according to the multinomial distribution with α = 0.1, similar to the data sampling technique for multilingual training. this baseline is meant to be comparable to our multilingual model trained with 400k steps, vocab size 16k, and α = 0.1. ## results first, we compare our best-performing uralicadapted multilingual models to both multilingual and monolingual baselines. we show that our chosen method of layering lapt and vocabulary specialization on a pre-trained multilingual model largely outperforms alternatives on downstream tasks and is more computationally efficient. we then analyze the dynamics of multilingual adaptation factors such as number of lapt steps, adapted vocabulary size, and sampling alpha. our grid search yields 72 evaluation data-points per language, per task, per setting. 6 we visualize the overall trends observed for each parameter, and then present a regression analysis of the combined effect of these parameters on task performance. ## conclusion this work shows that adapting a pre-trained crosslingual model to a language family greatly improves task performance, especially for underresourced languages in that family. targeted multilingual adaptation soundly outperforms monolingual adaptation for all low-resource uralic languages we test, as well as for half of the highresource ones. further, we show that specializing the model vocabulary for the uralic family yields significant improvements over the large "crosslingual" vocabulary of xlm-r, while simultaneously improving computational and memory efficiency. finally, our statistical analysis demonstrates which parameters (sampling alpha, training steps) are key to targeted adaptation performance. while our work is similar in goal to recent projects such as glot500-m @xcite , our approach contrasts in important ways. @xcite ; i.a. propose "horizontal" scaling to more languages primarily through vocabulary expansion, growing the overall model size. we on the other hand advocate for model specialization, shrinking the overall model size while increasing per-language parameter and vocabulary capacity. both our work and glot500-m significantly outperform xlm-r on under-resourced languages (though the exact evaluation results are not directly comparable 10 ). however, we note that our smallest fine-tuned model does so with just 25% of the parameters of glot500-m. specialization to smaller models has significant advantages in resource-constrained scenarios, and may better facilitate the democratization of nlp research and engineering for more language communities. @xcite point out that reducing the size of models like glot500-m through knowledge distillation would be useful future work. in fact, the two approaches are not mutually exclusive: our specialization technique could be applied to larger base models with expanded multilinguality, and model expansion and specialization may be more or less practical depending on the specific applied nlp workflow. we therefore concur with @xcite ; 10 both studies evaluate pos tagging on erzya, north sámi, estonian, finnish, hungarian, and russian. however, we do so in a fine-tuned setting whereas glot500-m is fine-tuned on english then zero-shot transferred to all other languages. our models show higher accuracy on low-resource languages, but this may be attributable to the difference in setting among other confounding variables like total compute budget for continued pre-training. @xcite ; @xcite ; i.a. that targeted or linguistically-informed multilingual modeling is one of the most promising avenues for extending nlp advance to the majority of the world's languages, as it both leverages the benefit of multilingualism for under-resourced languages and avoids the "curse of multilinguality" seen in massively multilingual approaches. however, given the success of large pre-trained language models, and of the pre-training paradigm more generally @xcite , we argue that it is more effective to leverage transferable information in existing cross-lingual models, rather than training from scratch. we hope our findings will inform best practices for such targeted multilingual adaption when extending the benefits of pre-trained models to under-resourced languages.