Exploring Intrinsic Language-specific Subspaces in Fine-tuning Multilingual Neural Machine Translation (2024)

Abstract: Multilingual neural machine translation models support fine-tuning hundreds of languages simultaneously. However, fine-tuning on full parameters solely is inefficient potentially leading to negative interactions among languages. In this work, we demonstrate that the fine-tuning for a language occurs in its intrinsic language-specific subspace with a tiny fraction of entire parameters. Thus, we propose language-specific LoRA to isolate intrinsic language-specific subspaces. Furthermore, we propose architecture learning techniques and introduce a gradual pruning schedule during fine-tuning to exhaustively explore the optimal setting and the minimal intrinsic subspaces for each language, resulting in a lightweight yet effective fine-tuning procedure. The experimental results on a 12-language subset and a 30-language subset of FLORES-101 show that our methods not only outperform full-parameter fine-tuning up to 2.25 spBLEU scores but also reduce trainable parameters to 0.4% for high and medium-resource languages and 1.6% for low-resource ones.

## introduction multilingual neural machine translation (mnmt) aims to use a single model to translate among different languages @xcite . recent studies of mnmt @xcite achieved significant progress in training large-scale pre-trained models supporting hundreds of languages. benefiting from crosslanguage learning, these pre-trained models offer the possibility of fine-tuning with limited data and show better performance in low-resource languages and non-english directions. however, multilingual fine-tuning still suffers from two limitations: (1) full-parameter fine-tuning becomes inefficient as the model size increases; (2) negative interactions among languages @xcite @xcite @xcite lower the performance of highresource languages. recent studies have shown that the fine-tuning of pre-trained models can be re-parameterized in an intrinsic subspace, i.e., a low-rank subspace with tiny parameters @xcite @xcite . this insight implies that language-specific fine-tuning in pre-trained mnmt models happens within intrinsic languagespecific subspaces, thus overcoming the aforementioned limitations: (1) intrinsic subspaces significantly reduce the required trainable parameters; (2) isolating the intrinsic subspaces among languages alleviates the negative interference in the multilingual representations. therefore, in this work, we propose language-specific lora (lslo), consisting of multiple lora @xcite modules with sparse language-specific activation, to model such intrinsic subspaces. moreover, prior works @xcite @xcite allocate the same number of parameters to different languages, which can yield sub-optimal setup because pre-trained models have already learned a substantial amount of knowledge from high-resource languages given the imbalance distribution of training data. we hypothesize that fine-tuning of highresource languages can be done in a smaller subspace compared to low-resource languages. to exhaustively explore the minimal intrinsic subspaces for each language, we first reduce the rank for highresource languages and then introduce unstructured pruning with a gradual pruning schedule @xcite during fine-tuning. however, determining the optimal structure of lslo remains challenging. first, there are 2 cases when selecting the language-specific sub-module of each lslo: selected by source language (sourceindexed) and selected by target language (targetindexed). furthermore, although we intuitively ex-pect that high-resource languages require smaller subspaces, it's still insufficient for the complex multilingual setting. these lead to the exponential increase in the possible architectures with the increase of the number of model layers and supported languages. therefore, in this work, we use two architecture learning techniques to avoid the tedious manual trial-and-error. we applied weight learning @xcite to determine whether each lslo module should be sourceindexed or target-indexed, given its interpretability and ease of visualization. we also propose a layerwise cross-language pruning method, which combines the lora modules of all languages at every layer for pruning to estimate the required subspace size for each language. we conduct our experiments on a 12-language subset of flores-101 @xcite . results show that in a pre-trained mnmt model, the size of intrinsic languagespecific subspace is highly correlated with the language's resource type. specifically, high-resource languages can be fine-tuned within a very small parameter subspace. our fine-tuning method outperforms full parameter fine-tuning by 1.3 spbleu while only using 0.4% trainable parameters for high and medium languages, and 1.6% for lowresource ones. we further evaluate our method on a 30-language subset, achieving a 2.25 spbleu improvement over full parameter fine-tuning with only 7% trainable parameters, which demonstrates the efficiency and effectiveness of our method. ## background given a set of n languages @xmath0, l 2 , • • • , l n }, the multilingual translation task is defined as translating an input in source language @xmath1. to train an mnmt model, we need a parallel corpus including translations aligned at the sentence level for creating mnmt datasets. for instance, consider a collection with m sets of sentences @xmath2, s 2 , • • • , s m }, each sentence set includes sentences in different languages sharing the same semantics, with a parallel corpus, we can conveniently construct mnmt datasets including different translation directions @xmath3, e.g., s k src as the input x and s k tgt as the output y of a single translation pair (x, y). given a mnmt dataset with n translation pairs @xmath4, y i ), @xmath5, the training loss is defined as: ## architecture learning lslo introduces additional hyperparameters: (1) each lslo module can be selected by either the source or the target language; (2) each language can have a different rank r l i , leading to an exponential increase in possible architectures with the number of layers and languages. therefore, we propose two architecture learning techniques in this section to avoid manual selection. ## experimental setup dataset flores-101 @xcite is a high-quality parallel dataset, including 3,001 sentences from english wikipedia which are translated into 101 languages by human translators. sentences are divided into three splits: dev (997 sentences), devtest (1,012 sentences), and test (992 sentences). since the test set is not publicly available, we use the dev set for training and devtest set for evaluation. languages are divided into four resource types: high (h), medium (m), low (l), and very-low (v), based on the available bitext data with english. we first randomly selected four languages from each of the three resource types (high, medium, very-low) to form a small subset lang12 of 12 languages. we conducted comprehensive analyses and tests on lang12 to verify our proposed method. then, we extend our method to a larger subset lang30 to measure the impact when introducing more languages. details for lang12 and lang30 are provided in appendix a. ## conclusion in this work, we studied the imbalance size distribution of intrinsic language-specific subspaces in a pre-trained multilingual model. we modeled the intrinsic language-specific subspaces using lslo. we further proposed an intrinsic subspace estimation method and found that the size of the intrinsic subspace for each language is highly correlated with its resource type. the required subspace size for higher-resource languages is much smaller than for lower-resource ones. therefore, there is no need to set the same parameter budget for all languages when fine-tuning multilingual models. by fine-tuning languages in their respective intrinsic subspaces with different sizes using lslo, we achieved significant improvements compared to full parameter fine-tuning while greatly reducing the number of trainable parameters. we also proposed methods to search for the optimal placement of lslo. we showed that the model completes the transformation from the source side to the target side in the top layers of the encoder and that placing the lslo module in the fully connected layers is most effective in the transformer architecture.