Condensing Multilingual Knowledge with Lightweight Language-Specific Modules (2023)

Abstract: Incorporating language-specific (LS) modules or Mixture-of-Experts (MoE) are proven methods to boost performance in multilingual model performance, but the scalability of these approaches to hundreds of languages or experts tends to be hard to manage. We present Language-specific Matrix Synthesis (LMS), a novel method that addresses the issue. LMS utilizes parameter-efficient and lightweight modules, reducing the number of parameters while outperforming existing methods, e.g., +1.73 BLEU over Switch Transformer on OPUS-100 multilingual translation. Additionally, we introduce Fuse Distillation (FD) to condense multilingual knowledge from multiple LS modules into a single shared module, improving model inference and storage efficiency. Our approach demonstrates superior scalability and performance compared to state-of-the-art methods.

## introduction multilingual models confer the benefit of facilitating cross-lingual learning; however, they also grapple with the issue of language interference @xcite @xcite . recent studies aim to alleviate negative language interference through the introduction of language-specific (ls) modules @xcite @xcite @xcite . in this setup, each language batch is processed through its designated module rather than a shared module. although this approach is promising and barely inflates the number of flops like mixture-of-experts (moe) @xcite , 2 the number of parameters becomes difficult to manage and sometimes impractical when working with a large variety of languages. this is because the fundamental element forming ls or moe modules is typically the full-rank weight matrix derived from a densely connected layer, which causes a rapid increase in the number of parameters with a large number of languages or experts. 3 in this paper, we first scrutinize the parameter efficiency of language-specific modules from the perspective of using fewer parameters. consequently, a necessary question arises (rq1): can we approximate the original dense weight matrix using substantially fewer parameters? to answer this question, we propose novel and parameter-efficient method, language-specific matrix synthesis (lms), which can achieve similar performance to switch transformer even with three to four times smaller ls parameters (as shown in figure 1 ). then, we further investigate parameter efficiency from the perspective of knowledge density in each ls module. given recent discoveries that the performance improvement of sparsely activated models diminishes with an increase in the number of experts @xcite @xcite , we hypothesize that knowledge in these experts (or ls modules) is over-estimated. hence, we propose another question (rq2): could a single shared module encapsulate the same level of knowledge as language-specific modules? in addressing this question, we introduce the fuse distillation (fd) method to examine the feasibility of condensing the multilingual knowledge into a single module. our main contributions are summarized as follows: ## lightweight ls modules in this section, we address rq1 by constructing ls modules with significantly fewer parameters. ## can we fuse multilingual knowledge in a single module? in this section, we introduce fuse distillation (fd) and use a preliminary experiment to answer rq2: whether we can condense the multilingual knowledge from language-specific modules into a single module. ## combining lms and fd we have shown the success of multilingual information condensation by fuse distillation. we are interested in further reducing the parameters figure 3 : we utilize a language-level moe architecture to verify the feasibility of fusing multilingual knowledge from all language-specific modules into a single shared module. during training, each batch goes through the ls module in the first forward pass and goes through the shared module in the second pass. then, we conduct distillation between two outputs to condense the knowledge into the shared module. for inference, we discard the ls module and only use the shared module. thus, the backward pass optimizes both the standard prediction loss and the fuse distillation loss. in figure 4 , we provide a comprehensive comparison of space complexity for generating extra ls (or expert) modules, among conventional ls modules, mixture-of-experts, and our proposed methods. notably, our methods demonstrate substantial reductions in parameter usage during both training and inference. ## experiments we evaluate our lms and lms+fd methods using three tasks: mmt, mner, and mqa. similar to section 3.2, we have two routing options for the lms+fd method during inference time: 1) evaluating the model by passing the shared route (denoted as lms+fd-share, the default setting), or 2) passing the language-specific module (denoted as lms+fd-ls). we present results for both routes to show the performance difference between using the condensed module and the original ls modules. considering the computational cost for mmt, we run all methods once with the same random seed. for the other two tasks, we run experiments with 3 different random seeds and report the average scores. for ease of implementation, we build homogeneous batches (i.e., a batch only containing sentences in one language or one language direction) and only activate the corresponding ls module. foot_1 ## related work language-specific modules to mitigate language interference, previous studies incorporate language-specific modules into models, such as additional language-aware linear projections @xcite @xcite , ls layer normalization @xcite . feed-forward networks @xcite , or even entire languagedependent transformer layers @xcite @xcite . similar to ls modules, mixture-of-experts (moe) are also able to reduce language interference @xcite @xcite . however, the parameter count of ls (or expert) drastically increases when scaling to numerous languages. @xcite address this issue by sharing all ls modules across all encoder or decoder layers. however, this does not fundamentally resolve the problem, given that the complexity of constructing ls modules remains unaltered and that different layers may need to learn varying types of ls information. lightweight modules our proposed techniques draw inspiration from another research line, lightweight fine-tuning, wherein the model undergoes fine-tuning on a parameter subset significantly smaller than that of the original model, such as prefix tuning (li and liang, 2021), prompt tuning @xcite , multitask prompt tuning @xcite , lora @xcite . in the multilingual machine translation setting, previous studies use language-pair adapters @xcite to fine-tune a specific direction. this approach also extends to languagewise adapters @xcite , languagefamily adapters @xcite , hyperadapters @xcite to facilitate the cross-lingual learning. in light of the efficient lightweight modules, we propose lms to help ls modules scale to hundreds of languages. ## conclusion the construction of language-specific modules (or experts) using full-rank matrices tends to be parameter-intensive and inefficient, especially as the number of languages (or experts) increases. to address this, we have introduced the language-specific matrix synthesis (lms) method that approximates the original full-rank matrix. notably, pair-wise synthesis, a variant of the lms methods, exhibits commendable performance in mmt tasks. further, we have proposed the fuse distillation (fd) approach to condense multilingual information into a shared module, thereby further diminishing parameter requirements during inference. our methods outperform clsr and switch transformer in mmt tasks and also demonstrate their effectiveness in mner and mqa tasks.