Contrastive Learning for Many-to-many Multilingual Neural Machine Translation (2021)

Abstract: Existing multilingual machine translation approaches mainly focus on English-centric directions, while the non-English directions still lag behind. In this work, we aim to build a many-to-many translation system with an emphasis on the quality of non-English language directions. Our intuition is based on the hypothesis that a universal cross-language representation leads to better multilingual translation performance. To this end, we propose mRASP2, a training method to obtain a single unified multilingual translation model. mRASP2 is empowered by two techniques: a) a contrastive learning scheme to close the gap among representations of different languages, and b) data augmentation on both multiple parallel and monolingual data to further align token representations. For English-centric directions, mRASP2 achieves competitive or even better performance than a strong pre-trained model mBART on tens of WMT benchmarks. For non-English directions, mRASP2 achieves an improvement of average 10+ BLEU compared with the multilingual baseline

## introduction transformer @xcite has achieved decent performance for machine translation with rich bilingual parallel corpora. recent work on multilingual machine translation aims to create a single unified model to translate many languages @xcite @xcite @xcite . multilingual translation models are appealing for two reasons. first, they are model efficient, enabling easier deployment @xcite @xmath0. <@xmath1. despite these benefits, challenges still remain in multilingual nmt. first, previous work on multilingual nmt does not always perform well as their corresponding bilingual baseline especially on rich resource language pairs @xcite @xcite . such performance gap becomes larger with the increasing number of accommodated languages for multilingual nmt, as model capacity necessarily must be split between many languages @xcite . in addition, an optimal setting for multilingual nmt should be effective for any language pairs, while most previous work focus on improv-ing english-centric 1 directions @xcite @xcite . a few recent exceptions are @xcite and @xcite , who trained many-to-many systems with introducing more non-english corpora, through data mining or back translation. in this work, we take a step towards a unified many-to-many multilingual nmt with only english-centric parallel corpora and additional monolingual corpora. our key insight is to close the representation gap between different languages to encourage transfer learning as much as possible. as such, many-to-many translations can make the most of the knowledge from all supervised directions and the model can perform well for both english-centric and non-english settings. in this paper, we propose a multilingual contrastive learning framework for translation (mcolt or mrasp2) to reduce the representation gap of different languages, as shown in figure 1 . the objective of mrasp2 ensures the model to represent similar sentences across languages in a shared space by training the encoder to minimize the representation distance of similar sentences. in addition, we also boost mrasp2 by leveraging monolingual data to further improve multilingual translation quality. we introduce an effective aligned augmentation technique by extending ras @xcite ) -on both parallel and monolingual corpora to create pseudo-pairs. these pseudo-pairs are combined with multilingual parallel corpora in a unified training framework. simple yet effective, mrasp2 achieves consistent translation performance improvements for both english-centric and non-english directions on a wide range of benchmarks. for englishcentric directions, mrasp2 outperforms a strong multilingual baseline in 20 translation directions on wmt testsets. on 10 wmt translation benchmarks, mrasp2 even obtains better results than the strong bilingual mbart model. for zeroshot and unsupervised directions, mrasp2 obtains surprisingly strong results on 36 translation directions 2 , with 10+ bleu improvements on average. ## experiments this section shows that mrasp2 can achieve substantial improvements over previous many-tomany multilingual translation on a wide range of benchmarks. especially, it obtains substantial gains on zero-shot directions. ## experiment results this section shows that mrasp2 provides consistent performance gains for supervised and unsupervised english-centric translation directions as well as for non-english directions. ## analysis to understand what contributes to the performance gain, we conduct analytical experiments in this section. first we summarize and analysis the performance of mrasp2 in different scenarios. second we adopt the sentence representation of mrasp2 to retrieve similar sentences across languages. this is to verify our argument that the improvements come from the universal language representation learned by mrasp2. finally we visualize the sentence representations, mrasp2 indeed draws the representations closer. ## conclusion we demonstrate that contrastive learning can significantly improve zero-shot machine translation directions. combined with additional unsupervised monolingual data, we achieve substantial improvements on all translation directions of multilingual nmt. we analyze and visualize our method, and find that contrastive learning tends to close the representation gap of different languages. our results also show the possibilities of training a true many-to-many multilingual nmt that works well on any translation direction. in future work, we will scale-up the current training to more languages, e.g. pc150. as such, a single model can handle more than 100 languages and outperforms the corresponding bilingual baseline.