I nte MAT s: Integrating Granularity-Specific Multilingual Adapters for Cross-Lingual Transfer (2023)

Abstract: Multilingual language models (MLLMs) have achieved remarkable success in various cross-lingual transfer tasks. However, they suffer poor performance in zero-shot low-resource languages, particularly when dealing with longer contexts. Existing research mainly relies on full-model fine-tuning on large parallel datasets to enhance the cross-lingual alignment of MLLMs, which is computationally expensive. In this paper, we propose InteMATs, a novel approach that integrates multilingual adapters trained on texts of different levels of granularity. To achieve this, we curate a multilingual parallel dataset comprising 42 languages to pre-train sentence-level and document-level adapters under the contrastive learning framework. Extensive experiments demonstrate the effectiveness of InteMATs in improving the cross-lingual transfer performance of MLLMs, especially on low-resource languages. Finally, our comprehensive analyses and ablation studies provide a deep understanding of the high-quality representations derived by InteMATs.

## introduction multilingual language models (mllms), such as mbert @xcite and xlmr @xcite , have achieved remarkable success on a wide range of cross-lingual tasks @xcite @xcite @xcite . they are pre-trained on large multilingual data using pretext tasks such as masked language modeling (mlm). to apply mllms to downstream tasks, we need to fine-tune a separate copy for each specific task. however, due to limited data during pre-training, mllms may struggle to capture cross-lingual alignment for low-resource languages, resulting in subpar performance. to address this issue, subsequent works @xcite @xcite @xcite propose to enhance mllm representations by continuing the pre-training on more parallel data using techniques such as translation language modeling (e.g., infoxlm @xcite and xlm-align @xcite ), contrastive learning (e.g., msimcse @xcite ), and so on @xcite . nevertheless, these approaches often require fine-tuning the entire backbone model, which is computationally expensive and can result in catastrophic forgetting @xcite . moreover, they mainly focus on enhancing sentencelevel representations. as a result, their performance gains on document-level cross-lingual tasks are not as impressive as sentence-level ones @xcite , indicating a need for a finer-grained perspective in enhancing cross-lingual alignment. motivated by recent advancements in parameterefficient adaptation for large language models (llms) @xcite , we propose a novel approach named intemats, which stands for integrating granularity-specific multilingual adapters, to enhance cross-lingual transfer performance of mllms. adapter tuning works by training a set of adapter modules conditioned on a frozen llm @xcite . they can equip mllms with new knowledge @xcite and facilitate language-specific adaptation @xcite without modifying pretrained parameters. different from them, we offer a new perspective for enhancing cross-lingual alignment by exploiting a set of multilingual adapters pre-trained on different levels of text granularity. to obtain these adapters, we curate a multilingual parallel dataset consisting of 42 languages from wikipedia foot_0 and process them into the sentence-level corpus (d st ) and document-level corpus (d dt ). recent findings @xcite reveal that contrastive learning (cl) tends to ex-tract global information, whereas masked image modeling (mim) focuses on capturing local features, which well explained earlier results @xcite . therefore, we employ cl to train adapters to capture global cross-lingual alignment information to augment the mllm representations that may initially focus on extracting local information. specifically, we first train a set of multilingual adapters on d st and d dt respectively. then, we learn to fuse them by incorporating layer-wise fusion modules (e.g., adapterfusion @xcite ) and train them on downstream data to determine the fusion weights. extensive experiments on five kinds of crosslingual transfer tasks, including sentence-level and document-level ones, demonstrate that intem-ats significantly improves the cross-lingual transfer performance of mllms (i.e., mbert and xlmr). in particular, intemats excels the stateof-the-art baseline by 4% on bucc @xcite , 7% on tatoeba @xcite , and 3.5% on tydiqa @xcite . notably, intemats brings a substantial 30% improvement over its backbone mllms in low-resource languages that are never seen during pre-training, demonstrating the high quality of the representations learned by intemats. we finally conduct a comprehensive analysis on intem-ats to unravel the distribution of contributions on each component and layer-wise impact within in-temats. we will make our data and model publicly available for future research. adapters for mllms recently, adapter-based approaches @xcite @xcite @xcite have gained popularity as a parameter-efficient alternative to traditional finetuning for large language models (llms). these adapters are inserted between the transformer layers and are learned from data conditioned on a frozen llm. they learn language-specific transformations to facilitate quick adaptation to new languages @xcite or new tasks @xcite . recent research show that it is possible to inject new knowledge into mllms to enhance their cross-lingual representations @xcite . different from previous research, this paper fills the gap of granularity perspective in cross-lingual alignment via adapter tuning. ## preliminaries we start by introducing some basic knowledge about adapter tuning and contrastive learning. adapter tuning: adapter tuning @xcite is a parameter-efficient transfer learning technique for adapting large pre-trained models to downstream tasks based on adapters @xcite . these adapters are small, new taskspecific modules inserted between the layers of a pre-trained model. instead of fine-tuning the entire model, adapter tuning only trains the parameters of the adapter modules while keeping the pre-trained model fixed. this approach allows us to specialize a pre-trained model in different tasks while retaining the knowledge acquired during pre-training. following @xcite , we use ϕ w to denote a pre-trained model with parameters w: ϕ w (x). for adapter tuning, a new function, ψ w,v (x), is composed, where parameters v denote all the adapter modules and w is copied from pretrained weights. the architecture of an adapter module is shown in figure 1 (a) . it consists of two feed-forward layers and a non-linear activation function. for the hidden states h l at layer l, an adapter module works as follows: infonce: infonce (infomax with noise contrastive estimation) @xcite is a loss function commonly used in self-supervised learning, particularly in contrastive learning methods. the goal is to maximize the mutual information between related samples while minimizing it between unrelated samples, facilitating the discovery of discriminative features in an unsupervised manner. given a sample x, and a set of n random samples, @xmath0, ..., x n }, which contains one positive sample x + and n -1 negative samples x -, we minimize the following negative log-likelihood: ## the intemats approach we now introduce intemats, which integrates multilingual adapters to enhance the representations of a fixed mllm. intemats involves two stages. in the first stage, we pre-train two multilingual adapters to be specialized in processing texts of different levels of granularity: sentence-level multilingual adapters (mats-st) ( §4.1) and documentlevel multilingual adapters (mats-dt) ( §4.2). inspired by the findings @xcite which unraveled the complementary properties of cl and masked image modeling objectives, we use cl to train adapters to augment the mllms that have been initially trained with the mlm objective. the goal is to enhance the cross-lingual alignment of mllms with pluggable adapters while retaining their pre-learned knowledge. in the second stage, we show how to integrate these adapters for crosslingual transfer tasks. in this paper, we employ adapterfusion @xcite for this purpose ( §4.3). intemats, however, is not limited to the choice of mllms and fusion algorithms. we show the working mechanisms of intemats and a concrete example of mats-st in figure 1 . ## conclusion in this paper, we introduce intemats, a novel approach that integrates granularity-specific multilingual adapters, including sentence-level multilingual adapters (mats-st) and document-level multilingual adapters (mats-dt), to enhance mllms in cross-lingual transfer tasks. on top of a fixed mllm (e.g., mbert and xlmr), we train these adapters on our curated parallel corpus using the contrastive learning objective. our experiments demonstrate that intemats significantly enhanced the cross-lingual transfer performance of mllms across sentence-level and document-level tasks. our comprehensive analyses show that intemats can automatically leverage corresponding adapters when dealing with different kinds of tasks.