Adaptive Token-level Cross-lingual Feature Mixing for Multilingual Neural Machine Translation (2022)

Abstract: Multilingual neural machine translation aims to translate multiple language pairs in a single model and has shown great success thanks to the knowledge transfer across languages with the shared parameters. Despite promising, this share-all paradigm suffers from insufficient ability to capture language-specific features. Currently, the common practice is to insert or search language-specific networks to balance the shared and specific features. However, those two types of features are not sufficient enough to model the complex commonality and divergence across languages, such as the locally shared features among similar languages, which leads to sub-optimal transfer, especially in massively multilingual translation. In this paper, we propose a novel token-level feature mixing method that enables the model to capture different features and dynamically determine the feature sharing across languages. Based on the observation that the tokens in the multilingual model are usually shared by different languages, we we insert a feature mixing layer into each Transformer sublayer and model each token representation as a mix of different features, with a proportion indicating its feature preference. In this way, we can perform fine-grained feature sharing and achieve better multilingual transfer. Experimental results on multilingual datasets show that our method outperforms various strong baselines and can be extended to zero-shot translation. Further analyses reveal that our method can capture different linguistic features and bridge the representation gap across languages.

## introduction multilingual neural machine translation (mnmt) @xcite handles several translation directions in a single model. these multilingual models have been shown to be capable of facilitating the knowledge transfer across different languages @xcite @xcite and enabling translations between language pairs unseen in training @xcite @xcite . due to the above advantages, mnmt is appealing and has drawn much attention in recent years. the success of mnmt comes at the cost of insufficient ability to capture language-specific features @xcite . since the model parameters are shared across languages, the mnmt model tends to preserve the shared features but ignore the language-specific ones. therefore, researchers resort to language-specific modeling to capture and balance those two types of features. some works attempt to insert additional language-specific modules into the original mnmt model @xcite @xcite . however, those methods are sensitive to the structure and location of language-specific modules and require specialized manual design. to avoid this problem, other works turn to search languagespecific networks in the mnmt model @xcite . those methods generally adopt the multi-stage training strategy to find and fine-tune the language-specific parameters, which increases the training complexity, especially in massively multilingual translation settings. another pitfall of the above methods is that dividing the features into shared and language-specific ones may not be sufficient to model the complicated commonality and divergence across languages. previous studies @xcite have shown that similar languages generally share more commonality, and clustering them together can boost their translation performance. moreover, @xcite also demonstrates that there are some overlaps between the language-specific networks of similar languages. these observations indicate that there are some locally shared features among similar languages which are important to the multilingual transfer. however, those features are not effectively used in the current languagespecific models, which motivates us to model more fine-grained features of different languages to facilitate the multilingual transfer. in this work, we propose a novel token-level cross-lingual feature mixing method that enables the model to adaptively determine the feature sharing during training. based on the observation that the tokens in multilingual vocabulary are usually shared by different languages, we assume that each token representation contains a mix of lexical and linguistic features, with a feature proportion indicating its feature preference. specifically, we employ a set of linear transformations to capture different features, on which we perform weighted feature aggregation with the specific feature proportion. by varying the feature proportions, we can retain the locally shared features and control the knowledge sharing across different languages. our main contributions are summarized as follows: our work closely relates to the language-specific modeling in mnmt. early studies focus on increasing the shared parts of separate bilingual models for better knowledge transfer. these works include sharing encoders @xcite , sharing attention layers @xcite and sharing @xcite . later, @xcite and @xcite develop a universal mnmt model with an artificial language token added to the source sentence to indicate the target language. while the share-all paradigm generally captures the commonality of languages but ignores the specific features of each language. to this end, researchers turn to language-specific modeling for better balance between sharing and specific, including redesigning parameter sharing strategies @xcite @xcite , training separate models for different language clusters (tan et al., 2019), inserting lightweight adapters @xcite , routing shared or language-specific path @xcite , dividing general and specific networks or neurons @xcite and parameter differentiation @xcite . however, these methods do not make full use of the locally shared features across similar languages, leading to sub-optimal cross-lingual transfer, especially in massively multilingual translation. instead, we propose a feature mixing method which is a variant of mixture-of-experts (moe) models @xcite . we discuss two gating mechanisms and analyze the impact of the location and sparsity of the moe layer (clm module) on multilingual translation performance. our work is also related to zero-shot translation. some studies resort to forming language-agnostic representations. @xcite and @xcite introduce auxiliary training objectives to align the representations of different languages. @xcite bridges the cross-lingual representations with additional dictionary and contrastive learning. @xcite disentangles the positional information by relaxing the structural constraint. other studies explore to enhance the language-specific features in translation. @xcite and @xcite employ an additional target language prediction task to train the model to distinguish different languages. @xcite adopt monolingual adapter to model the language-specific features. our work continues in these directions, but with a special focus on combining different feature mixing models in the encoder and decoder to build a language-agnostic encoder and language-aware decoder. ## method our main idea is to model the commonality and divergence of different languages in a fine-grained way to retain more shared features, especially those locally shared by similar languages to facilitate the multilingual transfer. to achieve this, each language is considered to contain a mix of different features rather than solely the shared and specific ones, as shown in figure 1 . specifically, we first project each token representation into different subspaces with a set of linear transformations to capture different features and calculate the corresponding feature proportion based on the token representation itself. then we take the weighted averaging of different linear transformations as the feature-mixed representation. the proportion indicates the importance of each feature and determines the knowledge sharing across different languages. ## conclusion in this paper, we propose a token-level crosslingual feature mixing method that can capture different features and dynamically determine the feature sharing across languages. we employ a set of linear transformations to capture different features and aggregate them with specific proportions for each token representation. in this way, we can perform fine-grained feature sharing and achieve better multilingual transfer. experimental results on multilingual datasets show that our method outperforms various strong baselines and can be extended to zero-shot translation. further analyses reveal that our method can capture several different linguistic features and bridge the representation gap across languages. in future work, we plan to further study how to narrow the representation gap across low-resource languages for better translation performance and knowledge transfer.