Phylogeny-Inspired Adaptation of Multilingual Models to New Languages (2022)

Abstract: Large pretrained multilingual models, trained on dozens of languages, have delivered promising results due to cross-lingual learning capabilities on a variety of language tasks. Further adapting these models to specific languages, especially ones unseen during pre-training, is an important goal toward expanding the coverage of language technologies. In this study, we show how we can use language phylogenetic information to improve cross-lingual transfer leveraging closely related languages in a structured, linguistically-informed manner. We perform adapter-based training on languages from diverse language families (Germanic, Uralic, Tupian, Uto-Aztecan) and evaluate on both syntactic and semantic tasks, obtaining more than 20% relative performance improvements over strong commonly used baselines, especially on languages unseen during pre-training.

## introduction language models have now become the standard for building state-of-the-art natural language processing (nlp) systems. beyond monolingual models, large-scale multilingual models covering more than 100 languages are now available, such as xlm-r by @xcite and mbert by @xcite , achieving competitive performance across languages from a variety of families and using various scripts. still, most of the 6500+ spoken languages in the world @xcite are not covered -remaining unseen-by those models. even languages with millions of native speakers like lingala (with 15-20 million speakers in central africa, mostly d.r. congo) or bambara (spoken by around 5 million people in mali and neighboring countries) are not covered by any available language models at the time of writing. 1 code and data are publicly available: https://github . ## phylogeny-inspired adaptation motivation intuitively, given the similarities between two related lects (e.g. catalan and asturian), one should exploit that relationship to inform the adapters of both languages. thankfully, prior linguistic studies provide exactly the information we need in the form of phylogeny trees. relationships between languages are typically represented as tree or network diagrams. in the phylogenetic trees we will use, languages are grouped based on their similarities; an internal node may (but not necessarily) correspond to a hypothesized linguistic ancestor. while often a phylogenetic network is more appropriate than a tree (e.g. in cases of borrowing, or when two languages influence each other in a bidirectional manner), in this work we will focus on trees as a first step towards phylogeny-inspired adaptation. implementation in a standard setting of adapting a language model from a source language to another target language, the typical approach (e.g. @xcite is to have source and target specific language adapters, trained separately on unlabeled monolingual text with the masked language modeling (mlm) objective @xcite . then, one can train a task adapter on source language task data, stacking it on top of the source language adapter. at evaluation time, the source language adapter is replaced with the target language one. as example, shown in figure 1 , consider three languages: spanish, catalan, and asturian. to adapt a model for e.g. named entity recognition (ner), the standard practice trains spanish, catalan, and asturian language adapters separately: l:spanish, l:catalan, and l:asturian. using a language with labeled ner data (e.g. spanish) then trains a task adapter t:spanish using a stack of adapters [l:spanish, t:spanish]. at inference time we can then use a stack with the appropriate language adapter to perform the task in that language e.g., stack [l:asturian, t:spanish]. our approach follows the same principles, but adapters for multiple languages/genera/families are organized in a hierarchy following phylogenetic information and trained jointly. to continue with our running example, consider that all three languages belong to the romance language group of the indo-european family. we hence train five language type adapters jointly: f:indoeuro, g:romance, l:spanish, l:catalan, and l:asturian which are stacked following the hierarchy depicted in figure 1(c ). so, examples from all indoeuropean languages in our training mix are used to train the f:indoeuro adapter, g:romance is only trained on romance languages data (if we have e.g. english or danish in our mix, these data are directed through a g:germanic adapter), and we also have language-dedicated adapters. we ensure that each training batch includes data from a single language; so, for an asturian batch we train the following stack of adapters: [f:indoeuro, g:romance, l:asturian]. at inference time, we also add the task adapter, trained as before on a language with labeled data, on top of our languagehierarchy adapters. ## experimental setup tasks we experiment on three nlp tasks: for (1) and (2), we evaluate on 31 languages from universal dependencies v2.9 @xcite . for (3), we use 4 indigenous low-resource languages from americasnli @xcite , an extension of xnli @xcite . the choice of tasks and datasets is to ensure broad language coverage and especially to ensure we can study language families with only partial representation in the mlm pre-training stage. language families we study dependency parsing and pos-tagging on languages from the germanic, uralic and tupian families. 3 for nli, we work with languages from uto-aztecan and tupian families. see appendix ## further discussion we perform additional ablation studies where we show that our proposed approach provides sustainable performance in constrained settings with re-duced parameter counts. in addition, we explore data up-sampling for low-resource languages in language families with large data imbalances across the language members. this simple approach points towards the further improvement scope with limited data availability. detailed analysis of both these experiments are presented below. parameter reduction stacking multiple adapters instead of a single language adapter comes with extra parameter cost. 5 to assess whether we can integrate phylogenetic information while keeping the adapter parameter counts limited, we perform parameter reduction using a constant factor. for example, consider a single language adapter [l] which has down/upword projections with l:proj×layer parameters leading to a parameter count of 2×48×768. instead we can use a dimension reduced by a factor of 3 and add two extra adapters ([fgl]) without increasing the parameter count 2×(f:proj+g:proj+l:proj)×fgl:output; to be accurate: 2×(16+16+16)×768. contrast these with our solution without this constant factor parameter reduction, which will add 2×(48+48+48)×768 parameters to be learned. the results, tested on uralic languages for the dependency parsing task, are reported in deep vs wide adapters our fglt setting makes two important changes to the baseline lt one. first, it stacks 3 language-related adapters as opposed to a single one. second, it shares some of these adapters between languages. an important question is whether the performance improvements are due to stacking (making the model deeper) or due to the parameter sharing between languages. to answer this question, we perform another ablation where we replace the 2×(f:proj+g:proj+l:proj)×fgl:output setting with 2×(l:proj+l:proj+l:proj)×lll:output. essentially, we create a stack of 3 language-specific adapters. we will first contrast the baseline @xcite (which has a single wide adapter) to this deeper version @xcite . we keep the parameter count equal between the two using the same parameter reduction as in the previous paragraph. we find that the [lllt] setting does indeed improve performance, but only for high-resource languages, even exceeding the upscaled phylogenetic setting @xcite (see we want to further focus on this second point about parameter sharing: in ## related work continuous effort is being put to improve crosslingual transfer across languages as well as making language models capable enough to go beyond high resource domains. recently, @xcite , proposed an approach to combine lexicons with monolingual/parallel data for pretraining. it expands the modeling capability to thousands more languages largely including under-represented languages with limited to zero corpus availability. it is now proven that, pretraining on closely related languages yields better result for zero-shot transfer @xcite and continued pretraining on a larger number of languages leads to further improvement @xcite . however, training on some specific languages can still hurt the performance of other languages @xcite . as a result, it is crucial to prevent negative inference while keeping the performance equitable and robust across languages @xcite . to make the performance robust across languages, it is important to identify how much linguistic information is currently in place inside these big multilingual models. recent studies have done investigation on this hypothesis by probing language models for linguistic typology @xcite as well as phylogheny @xcite . these studies have measured phylogenetic distance and typological similarity across languages so that we can make informed cross-lingual transfer. in line with these findings, @xcite has done experiments to remove the language specific information by stackable vector operations which further improve the cross-lingual representation. one recent study @xcite dives further into identifying language-neutral and language-specific subspace inside the representation space of multilingual models and now it is proven that the shared representation space is the one helping to perform effective cross-lingual transfer. as opposed to the standard fine-tuning of largescale language models, a more focused trend is to perform efficient parameter selection thus reducing the overall computation cost and carbon footprints @xcite . adapters are such highly customized light-weight neural network layers on top of base models. because of this higher flexibility, there are studies already in place looking into the adapter-level optimization according to the nature of data and network layers @xcite . in addition, using language specific units in a modular fashion in the pre-training stage was shown to be beneficial in recent work @xcite . ## limitations and future work while we already incorporated task evaluation on a diverse set of language families ranging from extremely low resourced uralic ones to indigenous americasnli @xcite languages, our experiments are still limited in terms of typological diversity. in future, we want to further extend the typological diversity of languages we use. at the same time, we would like to democratize the full force of language genetical properties in steps beyond just finetuning thus making the resource scarce languages more accessible. ## conclusion in this work, we present an adapter-based approach to leverage language phylogenetic information for better cross-lingual adaptation. our experiments on a diverse set of tasks and languages show significant performance improvements over commonlyused strong baselines. even better, we show that under the exact same adapter parameter count settings, using smaller adapters but forcing adapter sharing between genetically related languages improves performance on true zero-resource scenarios. these improvements are particularly stark for languages unseen in the pre-training stage of large multilingual language models, providing a direct path towards better adaptation and language coverage for language technologies.