Language-aware Interlingua for Multilingual Neural Machine Translation (2020)

Abstract: Multilingual neural machine translation (NMT) has led to impressive accuracy improvements in low-resource scenarios by sharing common linguistic information across languages. However, the traditional multilingual model fails to capture the diversity and specificity of different languages, resulting in inferior performance compared with individual models that are sufficiently trained. In this paper, we incorporate a language-aware interlingua into the Encoder-Decoder architecture. The interlingual network enables the model to learn a language-independent representation from the semantic spaces of different languages, while still allowing for language-specific specialization of a particular language-pair. Experiments show that our proposed method achieves remarkable improvements over state-of-the-art multilingual NMT baselines and produces comparable performance with strong individual models.

## introduction neural machine translation (nmt) @xcite has significantly improved the translation quality due to its end-to-end modeling and continuous representation. while conventional nmt performs single pair translation well, training a separate model for each language pair is resource consuming, considering there are thousands of languages in the world. therefore multilingual nmt is introduced to handle multiple language pairs in one model, reducing the online serving and offline training cost. furthermore, the multilingual nmt framework facilitates the cross-lingual knowledge transfer to improve translation performance on low resource language pairs @xcite . despite all the mentioned advantages, multilingual nmt remains a challenging task since the language diversity and model capacity limitations lead to inferior performance against individual models that are sufficiently trained. so recent efforts in multilingual nmt mainly focus on enlarging the model capacity, either by introducing multiple encoders and decoders to handle different languages @xcite , or enhancing the attention mechanism with language-specific signals @xcite . on the other hand, there have been some efforts to model the specificity of different languages. @xcite and @xcite tackle this by simply adding some pre-designed tokens at the beginning of the source/target sequence, but we argue that such signals are not strong enough to learn enough language-specific information to transform the continuous representation of each language into the shared semantic space based on our observations. in this paper, we incorporate a language-aware interlingua module into the encoder-decoder architecture. it explicitly models the shared semantic space for all languages and acts as a bridge between the encoder and decoder network. specifically, we first introduce a language embedding to represent unique characteristics of each language and an interlingua embedding to capture the common semantics across languages. then we use the two embeddings to augment the self-attention mechanism which transforms the encoder representation into the shared semantic space. to minimize the information loss and keep the semantic consistency during transformation, we also introduce reconstruction loss and semantic consistency loss into the training objective. besides, to further enhance the language-specific signal we incorporate language-aware positional embedding for both encoder and decoder, and take the language embedding as the initial state of the target side. we conduct experiments on both standard wmt data sets and large scale in-house data sets. and our proposed model achieves remarkable improvements over state-of-the-art multilingual nmt baselines and produces comparable performance with sufficiently trained individual models. ## model architecture as shown in figure 1 , we propose a universal encoder-interlingua-decoder architecture for multilingual nmt. the encoder and decoder are identical to the generic self-attention trans-former @xcite , except some modifications in the positional embedding. the interlingua is shared across languages, but with language-specific embedding as input, so we call it language-aware interlingua. the interlingua module is composed of a stack of n identical layers. each layer has a multi-head attention sub-layer and a feed-forward sub-layer. ## related work multilingual nmt is first proposed by @xcite in a one-to-many scenario and generalized by @xcite to many-to-many scenario. multilingual nmt suffered from the language diversity and model capacity problem. so one direction is to enlarge the model capacity, such as introducing multiple encoders and decoders to handle different languages @xcite @xcite , or enhancing the attention mechanism with language-specific signals @xcite . the other direction is aimed at a unified framework to handle all language pairs @xcite . they try to handle diversity by enhancing language-specific signals, by adding designed language tokens @xcite or language-dependent positional encoding @xcite . our work follows the second line by explicitly building a languageaware interlingua network which provides a much stronger language signal than the previous works. in regards to generating language-independent representation, @xcite and @xcite both attempted to build a similar language-independent representation. however, their work is all based on multiple languagedependent lstm encoder-decoders, which significantly increase the model complexity. and they don't have the specially designed training objective to minimize the information loss and keep the semantic consistency. whereas our work is more simple and effective in these regards and testified on a much stronger transformer based system. ## conclusion we have introduced a language-aware interlingua module to tackle the language diversity problem for multilingual nmt. experiments show that our method achieves remarkable improvements over state-of-the-art multilingual nmt baselines and produces comparable performance with strong individual models.