Multilingual Agreement for Multilingual Neural Machine Translation (2021)

Abstract: Although multilingual neural machine translation (MNMT) enables multiple language translations, the training process is based on independent multilingual objectives. Most multilingual models can not explicitly exploit different language pairs to assist each other, ignoring the relationships among them. In this work, we propose a novel agreement-based method to encourage multilingual agreement among different translation directions, which minimizes the differences among them. We combine the multilingual training objectives with the agreement term by randomly substituting some fragments of the source language with their counterpart translations of auxiliary languages. To examine the effectiveness of our method, we conduct experiments on the multilingual translation task of 10 language pairs. Experimental results show that our method achieves significant improvements over the previous multilingual baselines.

## introduction multilingual neural machine translation (mnmt) has experienced rapid growth in recent years @xcite @xcite . it is not only capable of translating among multiple language pairs by encouraging the crosslingual knowledge transfer to improve low-resource translation performance @xcite @xcite @xcite , but also can handle multiple language pairs in a single model, reducing model parameters and training costs @xcite @xcite . previous works in mnmt simply optimize independent translation objectives and do not use arbitrary auxiliary languages to encourage the agreement across different translation directions. as shown in figure 1 , the multilingual baseline is separately trained on french-english and german-english directions and cannot explicitly promote each other. the german-english translation only implicitly helps the french-english translation since both translation directions share the same encoder. there still exists a gap between german-english and french-english translation directions. as a result, minimizing the difference across different translation directions by an explicit paradigm requires further exploration. in this paper, we propose a novel agreementbased method, which explicitly models the shared semantic space for multiple languages and encourages the agreement across them. our training procedure extends the multilingual translation with the agreement term, which encourages the model to produce the source sentence with multiple languages into the target sentence. as figure 1 shows, we randomly substitute some source phrases with their counterparts of other languages to create codeswitched sentences using word alignment. our model is jointly trained with the multilingual translation and agreement objectives, where the codeswitched sentences are translated into the target sentences. the key idea is to encourage the agreement among different translation directions simul- experimental results on the multilingual translation task of wmt demonstrate that our method outperforms the multilingual baseline by a large margin. to better explain the bleu improvements, we visualize the sentence-level crosslingual representations and the attention weights across different languages, which shows that our method effectively encourages the agreement between languages. ## results the results of our model are separately listed in ## related work multilingual machine translation previous works @xcite @xcite have explored different settings of the multilingual neural machine translation (mnmt). recent studies show that mnmt @xcite @xcite helps improve the performance of the lowresource or zero-shot translation. some researchers 0% 5% 10% 15% 20% 25% use the sentence pairs to enhance the bilingual neural machine translation @xcite @xcite . agreement-based learning many works try to use the agreement-based method @xcite @xcite to encourage agreement among different translation orders and directions @xcite @xcite @xcite . besides, the agreement-based method is also used to minimize the difference between the representation of source and target sentence @xcite . our method further explores the approach of the multilingual agreement. ## conclusion we propose a novel agreement-based framework to encourage multilingual agreement across different translation directions by the agreement term. experimental results on the multilingual translation task demonstrate that our method effectively minimizes the gaps among different translation directions and significantly outperforms the multilingual baselines. the analytic experiment about the crosslingual representation shows the effectiveness of our multilingual agreement in minimizing the differences among different languages.