Improving Multilingual Neural Machine Translation with Auxiliary Source Languages (2021)

Abstract: Multilingual neural machine translation models typically handle one source language at a time. However, prior work has shown that translating from multiple source languages improves translation quality. Different from existing approaches on multi-source translation that are limited to the test scenario where parallel source sentences from multiple languages are available at inference time, we propose to improve multilingual translation in a more common scenario by exploiting synthetic source sentences from auxiliary languages. We train our model on synthetic multi-source corpora and apply random masking to enable flexible inference with single-source or bi-source inputs. Extensive experiments on Chinese/English-Japanese and a large-scale multilingual translation benchmark show that our model outperforms the multilingual baseline significantly by up to +4.0 BLEU with the largest improvements on low-resource or distant language pairs.

## introduction neural machine translation (nmt) has achieved the state-of-the-art performance across domains and language pairs @xcite @xcite . one of the advantages of nmt over statistical machine translation models is that it enables information sharing among high-resource and low-resource languages by training a multilingual model on the parallel data from multiple language pairs, which has been shown to improve translation quality, especially on low-resource language pairs @xcite @xcite . although multilingual nmt models typically handle one language pair at a time during both training and inference @xcite , prior work has shown that translating generate figure 1 : an example of translating a chinese sentence into english by using japanese as the auxiliary language. adding a synthetic source from japanese helps to translate the red word "我孙子市" (abiko, a city in japan), which is often incorrectly translated into "my grandson city" by standard chinese-english mt models, while other words can be translated more accurately from the chinese source. from multiple parallel source sentences can further improve translation quality @xcite @xcite . they propose multi-source translation models to exploit multiple source inputs at inference time. however, these models are limited to the application scenario where the source sentence has already been manually translated into multiple languages. we argue that, in the more common scenario where only one source sentence is provided, we could also improve the translation quality of multilingual nmt models by augmenting the source input with a synthetic sentence generated by a translation model into another language. as shown by the example in figure 1 , the additional synthetic sentence can help translate low-frequency and domain-specific words that are difficult to translate directly from the source. in this paper, we propose a novel bi-source multilingual nmt model that leverages a synthetic source sentence from an auxiliary language to better translate a source sentence into the target language. we train our bi-source nmt model on a synthetic multi-source translation corpus generated by translating the source side of the parallel data into other source languages using pre-trained nmt models. we contribute a novel training algorithm that 1) randomly selects the auxiliary language at each training iteration, which improves the multilinguality of the encoder representations, and 2) randomly masks out the auxiliary sentence during training, so that the model can perform inference flexibly in two different modes, including a) single-source inference where our model takes a single source as input, and b) bi-source inference where we first translate the original source to another language using an nmt model and then feed the two source sentences into our model to predict the target translation. this allows end users to balance between translation quality and latency by choosing different inference modes. we experiment on the aspec chinese and english to japanese translation and a large-scale english-to-many translation benchmark that includes 10 language pairs from wmt. results show that our method is simple yet effective -it improves @xmath0.9 bleu on the english-to-many translation benchmark. the largest improvements are on low-resource languages, where it brings up to +4.0 bleu improvements. further analysis confirms our hypothesis that bi-source inference helps the model disambiguate word senses during translation. ## bi-source multilingual nmt inspired by prior work on multi-source translation @xcite , we hypothesize that multilingual translation models can benefit from additional synthetic source sentences that are automatically translated from the original source. ## related work since the recent success of the end-to-end nmt models @xcite , multilingual nmt has become a promising research direction. @xcite propose to perform one-to-many translation using a dedicated decoder for each target language. firat et al. orthogonal to these techniques, multi-source translation @xcite @xcite has been shown to improve translation quality by exploiting the source sentences manually translated into multiple languages. most studies assume access to multisource inputs during both training and inference. @xcite and @xcite introduce data augmentation methods to fill in the missing source in the training data. @xcite explore translating the source into a pivot language and feeding both the original source and pivot sentences to a multilingual model to improve zero-resource translation. however, the pivot sentence is added only at inference time, thus the approach is better suited to the zero-resource setting. more recently, @xcite shows that translating the source word to auxiliary languages improves word translation. our work is also related to multi-task learning for machine translation. @xcite propose multi-task learning with an auxiliary reconstruction objective that reconstructs the source sentence from decoder hidden states. @xcite further show that adding a reconstruction objective by back-translating the target sentences to the source helps low-resource translation. @xcite propose multi-task training with a denoising objective to improve the robustness of nmt models. @xcite show that multi-task learning with two additional denoising tasks on the monolingual data can effectively improve translation quality. our training strategy can also be viewed as multi-task learning as we train our multilingual model on single-source and bi-source inputs jointly. ## conclusion we introduced a novel bi-source multilingual translation model that exploits an additional source input from an auxiliary language to improve translation quality. our model can flexibly perform single-source and bi-source inference, in which it takes both the original source and a synthetic source sentence from an auxiliary language as inputs. experiments show that our method is simple yet effective -it improves the translation quality of multilingual models substantially, with the largest improvements on low-resource or distant language pairs. further analysis indicates that adding an auxiliary language input during inference helps the model disambiguate source words. this work also sheds new light on multilingual nmt training, as our multi-source training strategy brings substantial improvements over the multilingual baseline without adding any auxiliary inputs at inference time.