Learning Language-Specific Layers for Multilingual Machine Translation (2023)

Abstract: Multilingual Machine Translation promises to improve translation quality between non-English languages. This is advantageous for several reasons, namely lower latency (no need to translate twice), and reduced error cascades (e.g., avoiding losing gender and formality information when translating through English).On the downside, adding more languages reduces model capacity per language, which is usually countered by increasing the overall model size, making training harder and inference slower. In this work, we introduce Language-Specific Transformer Layers (LSLs), which allow us to increase model capacity, while keeping the amount of computation and the number of parameters used in the forward pass constant. The key idea is to have some layers of the encoder be source or target language-specific, while keeping the remaining layers shared. We study the best way to place these layers using a neural architecture search inspired approach, and achieve an improvement of 1.3 chrF (1.5 spBLEU) points over not using LSLs on a separate decoder architecture, and 1.9 chrF (2.2 spBLEU) on a shared decoder one.

## introduction multilingual neural machine translation (mnmt) has received much attention from the machine translation community in recent years @xcite @xcite @xcite @xcite . this interest is based on the many advantages it provides: scalability instead of having one model per language pair, a single model suffices, significantly reducing maintenance efforts as well as the combined model size across all languages. inference speed and less error cascading due to the availability of data, most production sys-tems are english-centric, meaning translation between two non-english languages naïvely involves translating twice (i.e. pivoting), once to english, and once from english. this approach increases latency and contributes to error cascades, since the translation to or from english usually implies information loss, e.g. missing gender or formality distinctions that do not exist similarly in english. low-resource improvements having a single model capable of handling multiple languages, means it can generalize across language boundaries and utilize characteristics of closely related translation directions to improve the translation quality for low-resource language-pairs (i.e. knowledge transfer). although achieving good zero-shot translation quality remains a challenging task, mnmt has been shown to help @xcite . despite the above advantages, training high quality multilingual models is a challenging task: as more languages are added, the more they compete for the model's parameters @xcite . a common solution is to increase the model size, but blindly doing so comes with its own troubles, as training becomes harder, inference slower, and the storage requirements increase, which makes them challenging to deploy to portable devices. in this work, our goal is to increase the model capacity per language pair, while at the same time, letting the model share knowledge between languages, and without increasing the inference cost. to this end, and combined with the observation from @xcite that the translation process in transformer models starts in the top encoder layers, we propose an architecture with shared and language-specific weights. figure 2 shows one such architecture, where layers 3 and 4 foot_0 are source language-specific, layers 13, 14, and 15 are target language-specific, the remaining layers are shared across all languages, and the decoder is multi-head self-attention add & norm add & norm feed forward (a) regular transformer encoder layer. multi-head self-attention add & norm add & norm feed forward multi-head self-attention add & norm add & norm feed forward multi-head self-attention add & norm add & norm feed forward (b) language-specific transformer encoder layer. figure 1 : side-by-side comparison of the regular transformer encoder layer (a) and our @xcite . for the language-specific transformer encoder layer, an indexing language is provided which routes whole sentences to the appropriate weights (here portuguese ). figure 2 : best-performing separate decoder architecture using lsls, found using our architecture search method. layers 3 and 4 are indexed by the source language and layers 13, 14, and 15 are indexed by the target language. the indices start at 1 on the bottom of the encoder. also target language-specific. for the non-shared layers, we propose using language-specific transformer layers (lsls), illustrated in figure 1b . quite simply, lsls are a combination (i.e., a dictionary) of regular transformer layers (figure 1a ), where the sub-layer used depends on the chosen language. we consider two cases: source-indexed lsls, and target-indexed lsls, distinguished by whether we use the source or the target language to select the appropriate sub-layer. the main contributions of this work are: ## related work there exists a vast literature investigating parameter sharing mechanisms for mnmt. particularly relevant is the shared-encoder, separate-decoder architecture proposed by @xcite which we use as the base for some of our experiments. several works analyze which weights should be shared between languages @xcite @xcite @xcite . regardless, most closely related to the presented work are the studies by @xcite and @xcite . @xcite propose adding conditional language-specific routing (clsr) layers inside the encoder and decoder transformer layers. they learn to mix between language-specific and shared weights, and do this on a word by word basis. our approach does not use learned routing but uses the same components for the whole sentence per language-pair, instead of computing a mixed representation. we also do not add extra parameters to the layer, meaning we have the same inference time complexity as regular transformer layers. the approach in purason and tättar (2022) is similar to ours in the sense that they use language-specific transformer layers on the encoder side, and also look into sharing weights on a language-family basis. in contrast to our approach, they focus on source-indexed language-specific layers, while we investigate selecting the layers based on the source or the target language. besides, we propose a systematic method for deciding which layers to share, and which to be language specific. connection to adapter layers adapter layers @xcite @xcite are a lightweight technique to finetune a pre-trained encoder model by injecting taskspecific sub-modules into the existing architecture. in contrast, lsls are designed to be trained from scratch, and replace shared by language-specific components, rather than adding new ones, keeping the overall computational costs constant. connection to mixture-of-experts lsls enable the introduction of source-and target-specific parameters in the encoder and increase the model capacity, while at the same time keeping the inference cost and effective parameter count for the forward-pass constant (see figure 1 ). as such, they are similar in nature to sparsely activated mixture-of-experts layers @xcite @xcite but with the important differences that 1) there is no need for learning a balanced routing module; 2) sub-layer utilization is enforced by design, which tends to be a problem for moe layers @xcite ; and 3) sentences are always routed to the same conditional compute based on the indexinglanguage, enabling smaller binaries for on-device downloading of model weights as well as consecutive downloads to extend the on-device capabilities to new languages. in fact, @xcite have shown that the final encoder moe layers also learn target language-specific utilization where a subset of experts is used when translating e.g. @xmath0. however, since it is commonly not strictly enforced, downloading all experts is required, increasing the download size for end users. ## methods in this section we describe our proposed language-specific transformer layer, as well as a way to select whether to use shared or language-specific weights for each layer. ## conclusion in this work, we studied how to increase the capacity of mnmt models using lsls. we showed that lsls are effective at increasing the model capacity per language, while keeping the computation requirements constant. we proposed a method for selecting the placement of lsls, and showed the importance of having shared as well as source and target language-specific parameters on the encoder.