BAD - X : Bilingual Adapters Improve Zero-Shot Cross-Lingual Transfer (2022)

Abstract: Adapter modules enable modular and efficient zero-shot cross-lingual transfer, where current state-of-the-art adapter-based approaches learn specialized language adapters (LAs) for individual languages. In this work, we show that it is more effective to learn bilingual language pair adapters (BAs) when the goal is to optimize performance for a particular source-target transfer direction. Our novel BAD-X adapter framework trades off some modularity of dedicated LAs for improved transfer performance: we demonstrate consistent gains in three standard downstream tasks, and for the majority of evaluated low-resource languages.

## introduction massively multilingual transformers (mmts) such as mbert @xcite , xlm-r @xcite , and mt5 @xcite have dominated research in multilingual nlp and cross-lingual transfer recently. pretrained on large amounts of unlabelled data in 100+ languages, they have been shown to achieve impressive performance for a wide range of languages and tasks, and in zero-shot cross-lingual transfer in particular @xcite . however, their representational capacity is known to be limited by the curse of multilinguality: a trade-off between the language coverage and model capacity @xcite , which typically favors high-resource languages. their limitations are thus especially pronounced in low-resource scenarios, in transfer between distant languages and towards resourcepoor target languages @xcite @xcite . a standard approach to zero-shot cross-lingual transfer with mmts (i) fine-tunes the full mmt on task-specific data in the source language and then (ii) applies it directly to make predictions in the target language @xcite . on top of the expensive fine-tuning of the entire large model, this standard procedure also does not 'prepare' the mmt to excel at a particular target language or for a particular source-target transfer direction. this has been alleviated through modular parameter-efficient adaptations of the mmts @xcite @xcite which bypass full fine-tuning, most prominently through lightweight adapters @xcite : additional trainable parameters inserted into the mmt's layers. they have recently been used for language and task specialization of the mmts @xcite , offering improved and more efficient zeroshot cross-lingual transfer. previous work @xcite @xcite @xcite , inter alia) focused on creating: 1) dedicated language adapters (las) for each individual language, and 2) individual task adapters (tas). creating single-language las enables a very modular approach to cross-lingual transfer, where a source language la (used in training) can be directly swapped with any target language la at inference. yet, this procedure still does not prepare nor adapt the mmt for a particular source-target transfer direction. put simply, if one's incentive is to optimize the performance of a particular target language l t given annotated data in a particular source language l s , especially under low-data regimes, one might try to capture the interplay between the two languages instead of learning separate las. to address this gap, in this work we introduce the bad-x framework: bilingual adapters (bas) for zero-shot cross-lingual transfer (see figure 1 ), designed towards improving transfer performance for a particular transfer direction, with a focus on low-resource target languages. the goal of bad-x is to specialize the mmt for a particular language pair, while preserving all its existing knowledge encoded into the mmt's parameters. we experiment with three standard tasks in cross-lingual transfer @xcite : part-of-speech tagging (pos), dependency parsing (dp) and natural language inference (nli), and with a total of 20 low-resource target languages. our results demonstrate that trading off modularity of single-language las for less modular bas (tailored for language pairs) indeed yields improved transfer performance over the current state-of-theart (sota) adapter-based transfer framework mad-x @xcite , in all three tasks and for the large majority of target languages. we also show that, under the fixed fine-tuning budget and resources, further task performance gains can be achieved by varying the ratio of l s -vs-l t unannotated data when learning bas. finally, aiming to delve deeper into the trade-off between modularity and training efficiency, we experiment with multilingual adapters that are trained on the source language and all target languages under consideration at once. we show that such adapters, despite being more efficient to train, are unable to match the performance of their more specialized counterparts across a diverse set of target languages. we share our code and pretrained bas online at: https://github.com/parovicm/badx . ## bad-x: methodology motivation and overview. the main idea can be summarized into the following: instead of adapting the mmt to languages l s and l t separately as done in the sota adapter-based mad-x framework @xcite , cross-lingual transfer might be more effective by adapting the mmt directly to the language pair (l s , l t ). this means that we learn a bilingual language-pair adapter instead of two separate monolingual las. we then learn a task adapter directly on top of the ba: since we focus on the zero-shot setting, this means using task-annotated examples only from l s to fine-tune the ta. this procedure is summarized in figure 1 . 1 bad-x adapters. bad-x adapts the mad-x adapter framework, where bas are learnt instead of single-language las. the architecture of the adapter in each layer l consists of a down-and 1 inspiration for bad-x originates from neural machine translation (nmt), where bilingual adapters have been trained on parallel corpora of two languages to recover performance of a massively multilingual nmt model for high-resource languages @xcite . bad-x, however, proposes bilingual adapters (i) without the use of any parallel data, (ii) with the goal to support the downstream cross-lingual transfer, and (iii) it targets low-resource target languages. up-projection with a residual connection. more specifically, let the down-projection be a matrix d @xmath0. let us denote mmt's hidden state and the residual at layer l as h l and r l , respectively. the adapter computation of layer l is then given by: with relu as the activation function. this formulation subsumes las and tas in mad-x, as well as bas and tas in bad-x, where las/bas receive the input from the (frozen) transformer layer, while tas receive the input from the (frozen) la/ba put on top of the frozen transformer layer (figure 1 ). foot_0 mad-x las are trained via masked-language modeling (mlm) objective on the wikipedia of the corresponding language, while tas are trained on annotated task data. once la for l s is available, ta is trained by stacking it on top of the fixed source la. transfer is done by replacing the l s la with the l t la. unlike mad-x, bad-x trains a single bilingual adapter via mlm, alternating between the unlabelled (wikipedia) data from both l s and l t . the 'data alternations' are done according to a predefined ratio: e.g., the ratio of n :1 denotes that the model would see n l s sentences followed by 1 l t sentence. the motivation for this is twofold: 1) seeing a data mixture from the two languages could produce a ba that is better for transfer than having two independent las; 2) las for low-resource l t -s might otherwise overfit due to unlabelled data scarcity in l t , and thus could benefit from additional l s data. in bad-x, ta is then again trained on top of the fixed ba, and the same ba-ta configuration is retained at inference, see figure 1 again. advantages and limitations. bad-x allows parameter-efficient transfer to arbitrary tasks and languages by learning modular bilingual and task representations. it trades-off some modularity of mad-x for increased performance and expressiveness when the goal is to perform a transfer for a fixed pair of languages. a disadvantage of bad-x with respect to modularity is that it no longer offers a zero-cost transfer (once all las are learnt) between all language pairs under consideration: it requires training of separate bas for all pairs of interest. however, as we show further in §3, bad-x might be preferable over mad-x in the cases when the goal is to improve a particular source-target direction, which is our targeted use-case. bad-x bas are trained on the wikipedia data of both l s and l t . the standard bad-x variant termed balanced bad-x (also bad-x 1:1) is trained by alternating one batch of the l s data (i.e. english) followed by one batch of the l t data, for 50,000 iterations (i.e., this way we match the total number of iterations performed by training mad-x l s and l t las for 25,000 iterations each), and we adopt all the hyperparameters from mad-x la training. we select as the final ba the one with the lowest l t perplexity. bilinguality of the bad-x bas allows us to directly train ta on top of it and perform the inference with the same configuration. ## conclusion we have presented bad-x, a novel adapter-based framework for zero-shot cross-lingual transfer. bad-x targets improving transfer performance for particular fixed source-target transfer directions through the introduction and use of dedicated bilingual language-pair adapters (bas). the effectiveness of the bas and the bad-x framework has been demonstrated on three standard transfer tasks, across a plethora of low-resource languages. in future work, we will experiment with more efficient approaches to bilingual adapters, e.g., based on contextual parameter generation @xcite , and port the bad-x framework to more languages and tasks.