FAD - X : Fusing Adapters for Cross-lingual Transfer to Low-Resource Languages (2022)

Abstract: Adapter-based tuning, by adding light-weight adapters to multilingual pretrained language models (mPLMs), selectively updates language-specific parameters to adapt to a new language, instead of finetuning all shared weights. This paper explores an effective way to leverage a public pool of pretrained language adapters, to overcome resource imbalances for low-resource languages (LRLs). Specifically, our research questions are, whether pretrained adapters can be composed, to complement or replace LRL adapters. While composing adapters for multi-task learning setting has been studied, the same question for LRLs has remained largely unanswered. To answer this question, we study how to fuse adapters across languages and tasks, then validate how our proposed fusion adapter, namely FAD-X, can enhance a cross-lingual transfer from pretrained adapters, for well-known named entity recognition and classification benchmarks.

## introduction while fine-tuning the multilingual pretrained language models (mplms), such as mbert @xcite or xlm-r @xcite has become a de-facto standard to tackle diverse language tasks, task performance in low-resource languages lags behind, due to resource imbalances @xcite . to overcome this challenge, mad-x @xcite tackles such performance degradation as a capacity issue, and adopts the idea of adapters @xcite . for a new language (or a task), they add a few parameters to adapt, while keeping parameters for mplms frozen. this approach enables a parameter-efficient adaptation to a new language or task, by tuning only language-and task-specific parameters, which can also be released as pretrained adapters. however, we argue that a significant resource imbalance yet remains, especially for lrls. to illustrate, figure 1 shows 8 highest/lowest resource languages among those with pretrained adapters. the gray bar suggests training resources for la (wikipedia articles written in each language) and the blue bar suggests those for ta (wikiann in section 3.2), which are dominated by high-resource languages, especially english. this suggests that pretrained adapters for our target problem of lrls (shown in the red box), are trained from resources that are multiple orders of magnitude smaller: for example, in figure 1 , resources for ta/la for gn are up to 20-fold and 1000-fold smaller respectively, which causes a negative transfer of mad-x, to underperform mbert baseline (shown in purple line). more significantly, the amount of languages supported by adapters (40+) is much less than that of mbert (100+), and even more significantly less than 6500+ languages that need to be supported. these observations present two challenges for lrls, (a) pretrained la may not exist, or exist with poor quality, and (b) task-specific resource is also scarce. in this paper, we propose fusing multiple adapters for cross-lingual transfer (fad-x), to overcome imbalances, by transferring from both la and ta resources available for higher-resource languages. inspired by multilingual plm outperforming monolingual plm for lrls from a cross-lingual transfer @xcite @xcite , we study whether such a transfer among adapters can be effective. specifically, we study whether pretrained las can be fused to complement lrls with lower-quality la, or even to support those with no adapter. toward this goal, given the pool of pretrained adapters l and target language t, we propose to utilize pretrained language adapter la l @xmath0, to train task adapter per each language, denoted as t a l i . we show that fusing such task adapters contributes to overcoming limited training resources, in training ta in the target language (the yellow line in figure 1 ensures positive transfers in all lrls with larger gains than mad-x). contributions our contributions are as follows: ## related work adapters adapters proposed for domain adaptations in computer vision tasks @xcite , have been successful for language tasks, as a parameter-efficient alternative to fine-tuning plms, specifically for task @xcite and domain adaptation (bapna and firat, 2019), avoiding catastrophic forgetting @xcite . the closest work to ours is, adapterfusion @xcite combines the representations from several task adapters for monolingual target tasks. our distinction is enabling a cross-lingual transfer across multiple language and task adapters. cross-lingual transfer a de-facto cross-lingual transfer is finetuning plms: mbert @xcite , xlm-r @xcite , or mt5 @xcite , while mad-x @xcite , leveraging three types of adapters: language, task, and invertible adapters, have been its parameter-efficient alternative. our contribution is observing the weaknesses of mad-x for lrls, and presenting a fusion to overcome such weaknesses. ## conclusion we proposed fad-x, fusing multiple pretrained adapters, for a cross-lingual transfer to lrls, overcoming the imbalances in resources for la/ta. we validate the effectiveness of our approach, for lrls with no pretrained adapter or that trained with limited resources.