When do Contrastive Word Alignments Improve Many-to-many Neural Machine Translation? (2022)

Abstract: Word alignment has proven to benefit many-to-many neural machine translation (NMT). However, high-quality ground-truth bilingual dictionaries were used for pre-editing in previous methods, which are unavailable for most language pairs. Meanwhile, the contrastive objective can implicitly utilize automatically learned word alignment, which has not been explored in many-to-many NMT. This work proposes a word-level contrastive objective to leverage word alignments for many-to-many NMT. Empirical results show that this leads to 0.8 BLEU gains for several language pairs. Analyses reveal that in many-to-many NMT, the encoder’s sentence retrieval performance highly correlates with the translation quality, which explains when the proposed method impacts translation. This motivates future exploration for many-to-many NMT to improve the encoder’s sentence retrieval performance.

## introduction many-to-many neural machine translation (nmt) @xcite @xcite @xcite @xcite jointly trains a translation system for multiple language pairs and obtain significant gains consistently across many translation directions. previous work @xcite shows that word alignment information helps improve pre-training for many-to-many nmt. however, manually cleaned high-quality ground-truth bilingual dictionaries are used to pre-edit the source sentences, which are unavailable for most language pairs. recently, contrastive objectives @xcite @xcite @xcite have been shown to be superior at leveraging alignment knowledge in various nlp tasks by contrasting the representations of positive and negative samples in a discriminative manner. this objective, which should be able to utilize word alignment learned by any toolkit, which in turn will remove the constraints of using manually constructed dictionaries, has not been explored in the context of leveraging word alignment for many-to-many nmt. an existing contrastive method @xcite for many-to-many nmt relies on sentencelevel alignments. given that the incorporation of word alignments has led to improvements in previous work, we believe that fine-grained contrastive objectives focusing on word alignments should help improve translation. therefore, this paper proposes word-level contrastive learning for manyto-many nmt using the word alignment extracted by automatic aligners. we conduct experiments on three many-to-many nmt systems covering general and spoken language domains. results show that our proposed method achieves significant gains of 0.8 bleu in the general domain compared to previous word alignment based methods and the sentence-level contrastive method. we then analyze how the word-level contrastive objective affects nmt training. inspired by previous work (artetxe and schwenk, 2019) that train sentence retrieval models using many-to-many nmt, we speculate that our contrastive objectives affect the sentence retrieval performance and subsequently impact the translation quality. further investigation reveals that in many-to-many nmt, the sentence retrieval precision of the multilingual encoder for a language pair strongly correlates with its translation quality (bleu), which provides insight about when contrastive alignment improves translation. this revelation emphasizes the importance of improving the retrieval performance of the encoder for many-to-many nmt. ## word-level contrastive learning for many-to-many nmt inspired by the contrastive learning framework @xcite and the sentence-level con-trastive learning objective @xcite , we propose a word-level contrastive learning objective to explicitly guide the training of the multilingual encoder to obtain well-aligned cross-lingual representations. specifically, we use word alignments, obtained using automatic word aligners, to supervise the training of the multilingual encoder by a contrastive objective alongside the nmt objective. alignment extraction two main approaches for automatically extracting aligned words from a sentence pair are: using a bilingual dictionary and using unsupervised word aligners. the former extracts fewer but precise alignments, whereas the latter extracts more but noisy alignments. we extract word-level alignments by both methods and explore how they impact nmt training. for the former approach, we use word2word @xcite to construct bilingual lexicons and then extract word pairs from parallel sentences. the extracted word pairs are combined to form a phrase if words are consecutive in the source and target sentence. for the latter approach, we use fastalign @xcite and use only 1-to-1 mappings for training. ## experimental settings datasets and preprocessing we selected ten languages, including english (en), estonian (et), italian (it), japanese (ja), kazakh (kk), burmese (my), dutch (nl), romanian (ro), turkish (tr), vietnamese (vi) from different language families to train the nmt systems. we used the parallel datasets from different domains for the selected nine language pairs, including iwslt, wmt, and alt. we followed mbart @xcite for tokenization. details are given in appendix a. for each parallel dataset, we implemented two approaches as stated in section 2 to extract word pairs for the contrastive training objective. data source and the number of the extracted word pairs are shown in ## conclusion we proposed a word-level contrastive learning objective for many-to-many nmt. experimental re-sults showed that our proposed method leads to significantly better translation for several language pairs, which is then explained by analyses showing the relationship between bleu scores and sentence retrieval performance of the nmt encoder. future work can focus on: (1) further improving the encoder's retrieval performance in many-to-many nmt; (2) contrastive objective's feasibility in a massively multilingual scenario.