Learning Language Specific Sub-network for Multilingual Machine Translation (2021)

Abstract: Multilingual neural machine translation aims at learning a single translation model for multiple languages. These jointly trained models often suffer from performance degradationon rich-resource language pairs. We attribute this degeneration to parameter interference. In this paper, we propose LaSS to jointly train a single unified multilingual MT model. LaSS learns Language Specific Sub-network (LaSS) for each language pair to counter parameter interference. Comprehensive experiments on IWSLT and WMT datasets with various Transformer architectures show that LaSS obtains gains on 36 language pairs by up to 1.2 BLEU. Besides, LaSS shows its strong generalization performance at easy adaptation to new language pairs and zero-shot translation. LaSS boosts zero-shot translation with an average of 8.3 BLEU on 30 language pairs. Codes and trained models are available at https://github.com/NLP-Playground/LaSS .

## introduction neural machine translation (nmt) has been very successful for bilingual machine translation @xcite @xcite @xcite . recent research has demonstrated the efficacy of multilingual nmt, which supports translation from multiple source languages into multiple target languages with a single model @xcite @xcite @xcite . multilingual figure 1 : illustration of a full network and languagespecific ones (lass).represents shared weights. -,andrepresents weights for @xmath0, @xmath1, respectively. compared to the full multilingual model, each lass learned model has language universal and language specific weights. while very promising, several challenges remain in multilingual nmt. the most challenging one is related to the insufficient model capacity. since multiple languages are accommodated in a single model, the modeling capacity of nmt model has to be split for different translation directions @xcite . therefore, multilingual nmt models often suffer from performance degradation compared with their corresponding bilingual baseline, especially for rich-resource translation directions. the simplistic way to alleviate the insufficient model capacity is to enlarge the model parameters @xcite . however, it is not parameter or computation efficient and needs larger multilingual training datasets to avoid over-fitting. an alternative solution is to design language-aware components, such as division of the hidden cells into shared and language-dependent ones @xcite , adaptation layers @xcite , language-aware layer normalization and linear transformation @xcite , and latent layers @xcite . in this work, we propose lass, a method to dynamically find and learn language specific subnetwork for multilingual nmt. lass accommodates one sub-network for each language pair. each sub-network has shared parameters with some other languages and, at the same time, preserves its language specific parameters. in this way, multilingual nmt can model language specific and language universal features for each language pair in one single model without interference. figure 1 is the illustration of vanilla multilingual model and lass. each language pair in lass has both language universal and language specific parameters. the network itself decides the sharing strategy. ## methodology we describe lass method in this section. the goal is to learn a single unified model for many translation directions. our overall idea is to find sub-networks corresponding to each language pair, and then only update the parameters of those subnetworks during the joint training. ## experiment results this section shows the efficacy and generalization of lass. firstly, we show that lass obtains consistent performance gains on iwslt and wmt datasets with different transformer architecture variants. further, we show that lass can easily generalize to new language pairs without losing the accuracy for previous language pairs. finally, we observe that lass can even improve zero-shot translation, obtaining performance gains by up to 26.5 bleu. ## analysis and discussion in this section, we conduct a set of analytic experiments to better understand the characteristics of language specific sub-network. we first measure the relationship between language specific subnetwork as well as its capacity and language family. secondly, we study how masks affect performance in zero-shot scenario. lastly, we discuss the relationship between pruning rate α and performance. we conduct our analytic experiments on iwslt dataset. for readers not familiar with language family and clustering, figure 4 is the hierarchical clustering according to language family.