Importance-based Neuron Allocation for Multilingual Neural Machine Translation (2021)

Abstract: Multilingual neural machine translation with a single model has drawn much attention due to its capability to deal with multiple languages. However, the current multilingual translation paradigm often makes the model tend to preserve the general knowledge, but ignore the language-specific knowledge. Some previous works try to solve this problem by adding various kinds of language-specific modules to the model, but they suffer from the parameter explosion problem and require specialized manual design. To solve these problems, we propose to divide the model neurons into general and language-specific parts based on their importance across languages. The general part is responsible for preserving the general knowledge and participating in the translation of all the languages, while the language-specific part is responsible for preserving the language-specific knowledge and participating in the translation of some specific languages. Experimental results on several language pairs, covering IWSLT and Europarl corpus datasets, demonstrate the effectiveness and universality of the proposed method.

## introduction neural machine translation(nmt) @xcite @xcite @xcite has shown its superiority and drawn much attention in recent years. although the nmt model can achieve promising results for highresource language pairs, it is unaffordable to train separate models for all the language pairs since there are thousands of languages in the world @xcite @xcite . a typical solution to reduce the model size and the training cost is to handle multiple languages in a single multilingual neural machine translation (mnmt) model @xcite @xcite . the standard paradigm of mnmt proposed by @xcite contains a language-shared encoder and decoder with a special language indicator in the input sentence to determine the target language. because different languages share all of the model parameters in the standard mnmt model, the model tends to converge to a region where there are low errors for all the languages. therefore, the mnmt model trained on the combined data generally captures the general knowledge, but ignores the language-specific knowledge, rendering itself sub-optimal for the translation of a specific language @xcite @xcite . to retain the language-specific knowledge, some researches turn to augment the nmt model with language-specific modules, e.g., the language-specific attention module @xcite , decoupled multilingual encoders and/or decoders @xcite and the lightweight language adapters @xcite . however, these methods suffer from the parameter increment problem, because the number of parameters increases linearly with the number of languages. besides, the structure, size, and location of the module have a large influence on the final performance, which requires specialized manual design. as a result, these problems often prevent the application of these methods in some scenarios. based on the above, we aim to propose a method that can retain the general and language-specific knowledge, and keep a stable model size as the number of language-pair increases without introducing any specialized module. to achieve this, we propose to divide the model neurons into two parts based on their importance: the general neurons which are used to retain the general knowledge of all the languages, and the language-specific neurons which are used to retain the language-specific knowledge. specifically, we first pre-train a standard mnmt model on all language data and then evaluate the importance of each neuron in each language pair. according to their importance, we divide the neurons into the general neurons and the language-specific neurons. after that, we finetune the translation model on all language pairs. in this process, only the general neurons and the corresponding language-specific neurons for the current language pair participate in training. experimental results on different languages show that the proposed method outperforms several strong baselines. our contributions can be summarized as follows: ## background in this section, we will give a brief introduction to the transformer model @xcite and the multilingual translation. ## approach our goal is to build a unified model, which can achieve good performance on all language pairs. the main idea of our method is that different neurons have different importance to the translation of different languages. based on this, we divide them into general and language-specific ones and make general neurons participate in the translation of all the languages while language-specific neurons focus on some specific languages. specifically, the proposed approach involves the following steps shown in figure 1 . first, we pretrain the model on the combined data of all the language pairs following the normal paradigm in @xcite . second, we evaluate the importance of different neurons on these language pairs and allocate them into general neurons and language-specific neurons. last, we fine-tune the translation model on the combined data again. it should be noted that for a specific language pair only the general neurons and the language-specific neurons for this language pair will participate in the forward and backward computation when the model is trained on this language pair. other neurons will be zeroed out during both training and inference. ## related work our work closely relates to language-specific modeling for mnmt and model pruning which we will recap both here. early mnmt studies focus on improving the sharing capability of individual bilingual models to handle multiple languages, which includes sharing encoders @xcite , sharing decoders @xcite , and sharing sublayers @xcite . later, @xcite and @xcite propose an universal mnmt model with a target language token to indicate the translation direction. while this paradigm fully explores the general knowledge between languages and hard to obtain the specific knowledge of each language @xcite , the subsequent researches resort to language-specific modeling, trying to find a better trade-off between sharing and specific. such approaches involve inserting conditional languagespecific routing layer @xcite , specific attention networks @xcite , adding task adapters @xcite , and training model with different language clusters @xcite , and so on. however, these methods increase the capacity of the model which makes the model bloated. moreover, our method is also related to model pruning, which usually aims to reduce the model size or improve the inference efficiency. model pruning has been widely investigated for both computer vision (cv) @xcite and natural language processing (nlp) tasks. for example, @xcite examines three magnitude-based pruning schemes, @xcite demonstrates that large-sparse models outperform comparablysized small-dense models, and @xcite improves the utilization efficiency of parameters by introducing a rejuvenation approach. besides, @xcite presents two parameter reduction techniques to lower memory consumption and increase the training speed of bert. ## conclusion the current standard models of multilingual neural machine translation fail to capture the characteristics of specific languages, while the latest researches focus on the pursuit of specific knowledge while increasing the capacity of the model and requiring fine manual design. to solve the problem, we propose an importance-based neuron allocation method. we divide neurons to general neurons and language-specific neurons to retain general knowledge and capture language-specific knowledge without model capacity incremental and specialized design. the experiments prove that our method can get superior translation results with better general and language-specific knowledge.