Fixing M o E Over-Fitting on Low-Resource Languages in Multilingual Machine Translation (2023)

Abstract: Sparsely gated Mixture of Experts (MoE) models have been shown to be a compute-efficient method to scale model capacity for multilingual machine translation. However, for low-resource tasks, MoE models severely over-fit. We show effective regularization strategies, namely dropout techniques for MoE layers in EOM and FOM, Conditional MoE Routing and Curriculum Learning methods that prevent over-fitting and improve the performance of MoE models on low-resource tasks without adversely affecting high-resource tasks. On a massively multilingual machine translation benchmark, our strategies result in about +1 chrF++ improvement in very low resource language pairs. We perform an extensive analysis of the learned MoE routing to better understand the impact of our regularization methods and how we can improve them.

## introduction training massively multitask models such as multilingual machine translation models benefit from transfer learning across different tasks. but they also suffer from reduced model capacity per task and potential interference between conflicting tasks. scaling up models has been shown to be a very effective strategy in many natural language processing tasks such as language modeling, massively multilingual translation and natural language understanding @xcite . most of these advancements have focused on training increasingly larger dense models. however, dense model scaling is computationally expensive, as a result, various sparse model architectures have been proposed to increase model capacity without incurring additional compute costs; the most commonly used one is the sparsely-gated mixture- figure 1: validation perplexity of dense and moe (64 experts) models. we show a high-resource direction that does not suffer from over-fitting, when a low resource direction sees extreme over-fitting. but we notice that, on imbalanced datasets, moe models suffer from over-fitting on low resource tasks i.e., tasks with relatively less training data. figure 1 illustrates this phenomenon on a multilingual translation benchmark. we see that eng-fra, a high-resource translation direction, does not overfit with either dense or moe models. on the other hand, eng-kon, a low-resource translation direction, extremely over-fits with the moe model compared to the dense model. in this work, we introduce four effective strate-gies to reduce the over-fitting of moe models on low-resource tasks in a massively multilingual mt benchmark: on a massively multilingual mt benchmark, foot_0 we experimentally demonstrate the effectiveness of each of these strategies. particularly, we observe close to +1 chrf ++ improvements with eom, fom, cmr and cl strategies on very low resource language directions out of english. ## background we first describe the multilingual machine translation (mmt) task setup, the dense backbone architecture, and how we augment it with moe layers. ## fixing over-fitting on low-resource tasks the motivation behind moe models is to allow different parameters to model different aspects of the input space. the added expert capacity should help higher resource language pairs that might otherwise be constrained to share the same capacity with many other language pairs. besides, increasing model capacity should reduce interference, thus benefiting tasks of all resource levels. although overall dropout is sufficient to regularize dense models, it is not enough for moe models (see figure 4 ). to address the issue of over-fitting of moe models on low-resource tasks, we propose a series of architectural changes that improve the performance on low-resource language pairs with moe models in sections 3.1 to 3.3. in section 3.4, we devise and study a simple but effective curriculum learning strategy as another approach to reduce the over-fitting on low-resource directions. ## related work improved routing in moe models. recent works have proposed alternatives to the commonly used top-2 gating of @xcite : hash layers @xcite language-specific parameters. a common solution to relax parameter sharing in mmt models is to use light-weight language-specific adapters @xcite . their size, however, scales linearly in the number of languages. @xcite introduce hyper-adapters to generate the adapters themselves. to make these language-specific parameters optional, @xcite propose clsr to dynamically select language-specific or shared paths. these paths are simple linear projections and do not incorporate routing. similar to our own cmr's budget loss, clsr optimizes the mmt crossentropy while constraining the use of the languagespecific capacity. another approach similar to cmr is residual-moe @xcite . it is a hybrid dense and moe model but it does not learn weights for each component. @xcite also introduces pr-moe for pyramidal moe where they increase the number of experts in the later layers to make the moe models more parameter-efficient. curriculum learning curriculum learning @xcite is motivated by the learning behavior of humans in which training samples are introduced by increasing levels of difficulty. the most common curriculum in mt models consist of pre-training on the more abundant monolingual data before finetuning on mt aligned bitexts @xcite @xcite . in bilingual mt, recent works explored fixed curricula that shard training samples based on some difficulty criteria like sentencelength @xcite or the confidence of a baseline model @xcite . @xcite proposed a heuristic that decides which samples are shown to the model based on the estimated sample's difficulty and current model competence. @xcite use reinforcement learning to learn the curriculum automatically and @xcite propose uncertainty-aware curriculum learning. in data sampling, which can be viewed as a sort of curriculum learning, @xcite propose dynamic sentence sampling to assign lower weights to well-learned sentences. ## conclusion in massively multilingual settings with imbalanced datasets, moe models over-fit significantly more than dense models on low-resource directions. this work introduce multiple effective strategies for regularizing moe models and achieving better performance across all language pairs, especially lowresource pairs. with eom and fom, we propose dropout methods to further regularize moe models. we introduce in cmr a novel architecture to balance the capacity between moes and shared dense paths. finally, we design curricula for introducing low-resource languages later during training. these strategies lead to less over-fitting on low-resource tasks, leading to improvements in translation quality.