Breaking the Curse of Multilinguality with Cross-lingual Expert Language Models (2024)

Abstract: Despite their popularity in non-English NLP, multilingual language models often underperform monolingual ones due to inter-language competition for model parameters. We propose Cross-lingual Expert Language Models (X-ELM), which mitigate this competition by independently training language models on subsets of the multilingual corpus. This process specializes X-ELMs to different languages while remaining effective as a multilingual ensemble. Our experiments show that when given the same compute budget, X-ELM outperforms jointly trained multilingual models across all 16 considered languages and that these gains transfer to downstream tasks. X-ELM provides additional benefits over performance improvements: new experts can be iteratively added, adapting X-ELM to new languages without catastrophic forgetting. Furthermore, training is asynchronous, reducing the hardware requirements for multilingual training and democratizing multilingual modeling.

## introduction massively multilingual language models (lms), which are trained on terabytes of text in a hundred or more languages, underlie almost all non-english and cross-lingual nlp applications @xcite . despite their wide adoption, these models come at a cost: the many languages are represented in the same, fixed model capacity, causing performance on individual languages to degrade relative to monolingual models @xcite . this phenomenon (termed the curse of multilinguality) can significantly harm low-resource language performance @xcite . in this paper, we address this curse with crosslingual expert language models (x-elm, figure † correspondence to blvns@cs.washington.edu * work done while visiting the university of washington. 1), an ensemble of language models initialized from a pretrained multilingual model and each independently trained on a different portion of a multilingual corpus with x-btm, a new extension of the branch-train-merge paradigm (btm; @xcite to the more heterogenous multilingual setting. x-elm allows for efficient scaling of model capacity to better represent all considered languages. we train x-elms on 20 languages-including adapting to 4 unseen ones-with up to 21 billion training tokens, with the 1.7b parameter xglm model as the base @xcite . our experiments show that x-elm strongly outperforms the dense multilingual models given the same compute budget in every considered setting ( §6), and that these improvements consistently benefit every language. notably, monolingual experts generally underperform typologically-informed multilingual x-elms, indicating that linguistically targeted multilinguality can benefit language modeling. we then show that the language modeling gains of x-elm hold on downstream evaluations ( §7). multilingual modeling with x-elm provides additional benefits beyond improved performance. training a set of x-elms is more computationally efficient than training a comparable dense model; each expert is trained independently, which removes the overhead cost of cross-gpu synchronization @xcite and allows asynchronous model training in low-compute settings. similarly, adapting x-elms to new languages with hmr training-a popular use case of language models (i.e., @xcite -is more efficient than continued training of a dense lm and does not risk forgetting previously learned languages @xcite . as a result, x-elm allows much more efficient and effective multilingual modeling than prior approaches, democratizing multilingual nlp. we release the code and trained models. 1 2 background: branch-train-merge multilingual lms are typically trained in a dense manner, where a single set of parameters are updated with every training batch. when training large lms, the dense training setup calculates gradients on and synchronizes model parameters across many gpus. 2 this requires all gpus to be available simultaneously and incurs communication costs that prolong training. branch-train-merge (btm; @xcite ) alleviates this cost by dividing the total compute among smaller expert language models that are trained independently on different domains and then combined during inference time. while the total number of parameters increases with the number of experts, inference with these models often uses a subset of experts (see §3.3), keeping inference costs manageable. c-btm @xcite then generalizes the above approach with automatic clustering of domains. across multiple 1 https://github.com/blvns/x-elm/ 2 for example, the xglm-7.5b model "was trained on 256 a100 gpus for about 3 weeks" @xcite . corpora, they show that (1) the optimal number of experts increases with data and compute and (2) a set of small expert models performs similarly to equivalently sized dense models at vastly reduced flop budgets. our work extends these studies to the multilingual setting, where experts are specialized to different languages instead of english-language domains. in the multilingual setting, we can also use typological structure to specialize experts, which we show provides additional benefits over automatic clustering. we also demonstrate that leveraging the hierarchy of language families in multi-round training yields further performance gains. ## cross-lingual expert language models multilingual language models are jointly trained on many different languages (e.g., @xcite , despite the well-documented effect this has on individual language performance @xcite . we propose cross-lingual expert language models, or x-elms (figure 1 ), which we hypothesize will alleviate the curse of multilinguality while maintaining the cross-lingual properties of dense multilingual lms. ## hierarchical multi-round training we previously described a single round of training for x-elm ( §3.1). however, btm can also be used iteratively to train new experts seeded with those learned in a prior round. the multilingual setting provides a natural extension of multi-round training that leverages typological structure. we propose hierarchical multi-round (hmr) pretraining (figure 1 ), which uses the learned typological tree structure from linguistic typology clustering to iteratively train more specific x-elms. specifically, given an expert model x trained on a cluster of languages l, we initialize a new set of experts hmr pretraining gives multiple benefits over single-round btm. in particular, hmr training saves compute and more easily adapts our x-elms to new settings. a specific application of this is adding new languages to the model: while updating dense multilingual lms with new languages is difficult and can lead to catastrophic forgetting of existing languages @xcite , hierarchically training an expert on a new language adds it to the x-elm set without altering the existing information in other experts. we evaluate hmr training with this use case in §6.3. ## language modeling experiments we now test the effectiveness of sparse language modeling in the multilingual setting. first, we determine the optimal number of clusters for our given compute budget and dataset ( §6.1). we then demonstrate that x-elms outperform comparable dense models on seen languages ( §6.2) and more effectively adapt to new, unseen languages ( §6.3). ## in-context learning experiments we also measure whether the perplexity improvements in x-elms correspond to better performance on downstream tasks. we test x-elms on three tasks in an in-context learning (icl) framework, showing x-elm language modeling gains do translate to icl improvements over the baseline models. ## conclusion this work presents an approach to break the curse of multilinguality by extending sparse language modeling to the multilingual setting with x-elm. we find that x-elms achieve better perplexity on every language over standard, dense language models trained with the same compute budget; expert language models can also be easily adapted to new languages without catastrophic forgetting. x-elms present additional benefits over dense models for multilingual modeling, including training efficiency and flexibility. finally, we show that these language modeling improvements transfer to downstream, in-context learning performance. while our experiments show that x-elm outperforms dense lms, we foresee many avenues of future work to further tailor sparse modeling to multilinguality. these include better methods for data allocation-such as clustering methods that leverage cross-lingual signal-and algorithmic improvements to better allocate compute and more effectively ensemble models. by proving the efficacy of sparse language modeling in the multilingual setting, we hope to inspire future work in this vein that fairly models every language while leveraging the potential of cross-lingual learning.