Language-Family Adapters for Low-Resource Multilingual Neural Machine Translation (2023)

Abstract: Large multilingual models trained with self-supervision achieve state-of-the-art results in a wide range of natural language processing tasks. Self-supervised pretrained models are often fine-tuned on parallel data from one or multiple language pairs for machine translation. Multilingual fine-tuning improves performance on low-resource languages but requires modifying the entire model and can be prohibitively expensive. Training a new adapter on each language pair or training a single adapter on all language pairs without updating the pretrained model has been proposed as a parameter-efficient alternative. However, the former does not permit any sharing between languages, while the latter shares parameters for all languages and is susceptible to negative interference. In this paper, we propose training language-family adapters on top of mBART-50 to facilitate cross-lingual transfer. Our approach outperforms related baselines, yielding higher translation scores on average when translating from English to 17 different low-resource languages. We also show that language-family adapters provide an effective method to translate to languages unseen during pretraining.

## introduction recent work in multilingual natural language processing (nlp) has created models that reach competitive performance, while incorporating many languages into a single architecture @xcite . because of its ability to share cross-lingual representations, which largely benefits lower-resource languages, multilingual neural machine translation (nmt) is an attractive research field @xcite @xcite @xcite . multilingual models are also appealing because they are more efficient in terms of the number of model parameters, †work done prior to joining microsoft enabling simple deployment @xcite . massively multilingual pretrained models can be used for multilingual nmt, if they are fine-tuned in a many-to-one (to map any of the source languages into a target language, which is usually english) or one-to-many (to translate a single source language into multiple target languages) fashion @xcite . training a many-to-many (multiple source to multiple target languages) nmt model @xcite has also been proposed. multilingual pretrained models generally permit improving translation on low-resource language pairs. specializing the model to a specific language pair further boosts performance, but is computationally expensive. for example, mbart-50 @xcite , a model pretrained on monolingual data of 50 languages using denoising autoencoding with the bart objective @xcite still has to be fully fine-tuned for nmt. to avoid fine-tuning large models, previous work has focused on efficiently building multilingual nmt models. adapters @xcite , which are lightweight feedforward layers added in each transformer @xcite layer, have been proposed as a parameter-efficient fine-tuning method. in machine translation, training a different adapter on each language pair on top of a frozen pretrained multilingual nmt model, has shown to improve results for high-resource languages @xcite . low-resource languages do not benefit from this approach though, as adapters are trained with limited data. in a similar vein, cooper @xcite fine-tune a pretrained model for multilingual nmt using a single set of adapters, trained on all languages. their approach manages to narrow the gap but still does not perform on par with multilingual fine-tuning. many-to-one and one-to-many nmt force languages into a joint space (in the encoder or decoder side) and neglect diversity. one-to-many nmt faces the difficulty of learning a conditional language model and decoding into multiple languages @xcite . to better model target languages, recent approaches propose exploiting both the unique and the shared features @xcite , reorganizing parametersharing @xcite , decoupling multilingual word encodings @xcite , training nmt models from scratch after creating groups of languages @xcite , or inserting language-specific layers @xcite . in this work, we propose using language-family adapters that enable efficient low-resource multilingual nmt. we train adapters for nmt on top of mbart-50 @xcite . the adapters are trained using bi-text from each language family, while the pretrained model is not updated. groups of languages are formed based on linguistic knowledge bases. our approach improves positive cross-lingual transfer, compared to language-pair adapters @xcite , which do not leverage cross-lingual information between languages, and language-agnostic adapters @xcite , which are trained on all languages and can suffer from negative interference @xcite . our approach not only yields better translation scores in the majority of languages examined, but also requires less than 20% of trainable parameters compared to languagepair adapters, i.e., the most competitive baseline. our main contributions are: ## language-family adapters for low-resource nmt fine-tuning a pretrained model for multilingual nmt provides a competitive performance, yet is computationally expensive, as all layers of the model need to be updated. a parameter-efficient alternative is to fine-tune a pretrained multilingual model for nmt with data from all languages of interest using adapters while keeping the pretrained model unchanged. however, as multiple language representations are encoded in the same parameters, capacity issues arise. languages are also grouped together, even though they might be different in terms of geographic location, script, syntax, typology, etc. as a result, linguistic diversity is not modeled adequately and translation quality degrades. we address the limitations of previous methods by proposing language-family adapters for lowresource multilingual nmt. an illustration of our approach is depicted in figure 1 . we exploit linguistic knowledge to selectively share parameters between related languages and avoid negative interference. our approach is to train adapters using language pairs of a linguistic family on top of a pretrained model, which is not updated. ## conclusion we presented a novel approach for fine-tuning a pretrained multilingual model for nmt using language-family adapters. our approach can be used for low-resource multilingual nmt, combining the modularity of adapters with effective cross-lingual transfer between related languages. we showed that language-family adapters perform better than both language-agnostic and languagepair adapters, while being computationally efficient. finally, for languages new to mbart-50, we showed that our approach provides an effective way of leveraging shared cross-lingual information between similar languages, considerably improving translations compared to the baselines. in the future, a more elaborate approach to encode lexical-level representations could further boost the performance of language-family adapters. we also hypothesize that the effectiveness of our model could be leveraged for other cross-lingual tasks, such as natural language inference, document classification and question-answering.