Original: When do Contrastive Word Alignments Improve Many-to-many Neural Machine Translation?
Revised: When do Contrastive Word Alignments Improve Many-to-many Neural Machine Translation?

Original: Anonymous ACL submission

Original: Abstract
Revised: Abstract

Original: Word alignment has proven to benefit many-tomany neural machine translation (NMT).
Revised: Word alignment has proven to benefit many-tomany neural machine translation (NMT).

Original: However, high-quality ground-truth bilingual dictionaries were used for pre-editing in previousmethods, which are unavailable for most language pairs.
Revised: However, high-quality ground-truth bilingual dictionaries were used for pre-editing in previous methods, which are unavailable for most language pairs.

Original: Meanwhile, the contrastive objective can implicitly utilize automatically learnedword alignment, which has not been exploredin many-to-many NMT.
Revised: Meanwhile, the contrastive objective can implicitly utilize automatically learned word alignment, which has not been explored in many-to-many NMT.

Original: This work proposesa word-level contrastive objective to leverageword alignments for many-to-many NMT.
Revised: This work proposes a word-level contrastive objective to leverage word alignments for many-to-many NMT.

Original: Empirical results show that this leads to 0.8 BLEUgains for several language pairs.
Revised: Empirical results show that this leads to 0.8 BLEU gains for several language pairs.

Original: Analyses reveal that in many-to-many NMT, the encoder’sretrieval performance highly correlates withthe translation quality, which explains whenthe proposed method impacts translation.
Revised: Analyses reveal that in many-to-many NMT, the encoder’s sentence retrieval performance highly correlates with the translation quality, which explains when the proposed method impacts translation.

Original: Thismotivates future exploration for many-to-many NMT focusing on improving the encoder retrieval performance.
Revised: This motivates future exploration for many-to-many NMT to improve the encoder’s sentence retrieval performance.

Original:  Introduction
Revised:  Introduction

Original: Many-to-many neural machine translation(NMT) (Firat et al., 2016; Johnson et al., 2017;
Revised: Many-to-many neural machine translation (NMT) (Firat et al., 2016; Johnson et al., 2017;

Original: Aharoni et al., 2019; Sen et al., 2019; Arivazhaganet al., 2019; Lin et al., 2020; Pan et al., 2021b)jointly trains a translation system for multiplelanguage pairs and obtain significant gainsconsistently across many translation directions.
Revised: Aharoni et al., 2019; Sen et al., 2019; Arivazhagan et al., 2019b; Lin et al., 2020; Pan et al., 2021b) jointly trains a translation system for multiple language pairs and obtain significant gains consistently across many translation directions.

Original: Previous work (Lin et al., 2020) shows that wordalignment information helps improve pre-trainingfor many-to-many NMT.
Revised: Previous work (Lin et al., 2020) shows that word alignment information helps improve pre-training for many-to-many NMT.

Original: However, cleanedhigh-quality ground-truth bilingual dictionaries areused to pre-edit the source sentences, which areunavailable for most language pairs.
Revised: However, manually cleaned high-quality ground-truth bilingual dictionaries are used to pre-edit the source sentences, which are unavailable for most language pairs.

Original: Recently, contrastive objectives (Clark et al. ,2020; Gunel et al., 2021; Giorgi et al., 2021; Weiet al., 2021) have been shown to be superior at leveraging alignment knowledge in various NLP tasksby contrasting the representations of positive andnegative samples in a discriminative manner.
Revised: Recently, contrastive objectives (Clark et al., 2020; Gunel et al., 2021; Giorgi et al., 2021; Wei et al., 2021; Mao et al., 2021) have been shown to be superior at leveraging alignment knowledge in various NLP tasks by contrasting the representations of positive and negative samples in a discriminative manner.

Original: Thisobjective, which implicitly utilizes word alignmentlearned by any toolkit refraining the constraints of using manually constructed dictionaries, has not been explored in the context of leveraging word alignment for many-to-many NMT.
Revised: This objective, which should be able to utilize word alignment learned by any toolkit, which in turn will remove the constraints of using manually constructed dictionaries, has not been explored in the context of leveraging word alignment for many-to-many NMT.

Original: An existing contrastive method (Pan et al., 2021b)(mRASP2) for many-to-many NMT relies on sentence-level alignments.
Revised: An existing contrastive method (Pan et al., 2021b) for many-to-many NMT relies on sentencelevel alignments.

Original: Given that the incorporation of word alignments has led to improvements in previous work, we believe that finegrained contrastive objectives focusing on word alignments should help improve translation.
Revised: Given that the incorporation of word alignments has led to improvements in previous work, we believe that fine-grained contrastive objectives focusing on word alignments should help improve translation.

Original: Therefore, this paper proposes word-level contrastive learning for many-to-many NMT using the word alignment extracted by automatic aligners.
Revised: Therefore, this paper proposes word-level contrastive learning for manyto-many NMT using the word alignment extracted by automatic aligners.

Original: We conduct experiments on three many-to-many NMT systems covering general and spoken language domains.
Revised: We conduct experiments on three many-to-many NMT systems covering general and spoken language domains.

Original: Results show that our proposed method achieves significantBLEU gains  in the general domain compared to previous word alignment based methods and the sentence-level contrastive method.
Revised: Results show that our proposed method achieves significant gains of 0.8 BLEU in the general domain compared to previous word alignment based methods and the sentence-level contrastive method.

Original: We then analyze how the word-level contrastive objective affects NMT training.
Revised: We then analyze how the word-level contrastive objective affects NMT training.

Original: Inspired by previous work (Artetxe and Schwenk, 2019) training sentence retrieval model using many-to-many
Revised: Inspired by previous work (Artetxe and Schwenk, 2019) that train sentence retrieval models using many-to-many

Original: NMT, we speculate that our contrastive objectives affect the sentence retrieval performance and subsequently impact the translation quality.
Revised: NMT, we speculate that our contrastive objectives affect the sentence retrieval performance and subsequently impact the translation quality.

Original: Further investigation reveals that in many-to-many NMT, the sentence retrieval precision of the multilingual encoder for a language pair strongly correlates with its translation quality (BLEU), which provides insight about when contrastive alignment improves translation.
Revised: Further investigation reveals that in many-to-many NMT, the sentence retrieval precision of the multilingual encoder for a language pair strongly correlates with its translation quality (BLEU), which provides insight about when contrastive alignment improves translation.

Original: This revelation emphasizes the importance of improving the retrieval performance of the encoder for many-to-many NMT.
Revised: This revelation emphasizes the importance of improving the retrieval performance of the encoder for many-to-many NMT.

Original:  Word-level Contrastive Learning for 
Revised:  Word-level Contrastive Learning for Many

Original: Many-to-many NMT
Revised: -to-many NMT

Original: Inspired by the contrastive learning framework (Chen et al., 2020) and the sentence-level contrastive learning objective of mRASP2, we pro-  pose a word-level contrastive learning objectiveto explicitly guide the training of the multilingualencoder to obtain well-aligned cross-lingual representations.
Revised: Inspired by the contrastive learning framework (Chen et al., 2020) and the sentence-level con- trastive learning objective (Pan et al., 2021b), we propose a word-level contrastive learning objective to explicitly guide the training of the multilingual encoder to obtain well-aligned cross-lingual representations.

Original: Specifically, we use word alignments,obtained using automatic word aligners, to supervise the training of the multilingual encoder by acontrastive objective alongside the NMT objective.
Revised: Specifically, we use word alignments, obtained using automatic word aligners, to supervise the training of the multilingual encoder by a contrastive objective alongside the NMT objective.

Original: Alignment Extraction Two main approaches forautomatically extracting aligned words from a sentence pair are: using a bilingual dictionary andusing unsupervised word aligners.
Revised: Alignment Extraction Two main approaches for automatically extracting aligned words from a sentence pair are: using a bilingual dictionary and using unsupervised word aligners.

Original: The former extracts fewer but precise alignments, whereas the latter extracts more but noisy alignments.
Revised: The former extracts fewer but precise alignments, whereas the latter extracts more but noisy alignments.

Original: We extractword-level alignments by both methods and explorehow they impact NMT training.
Revised: We extract word-level alignments by both methods and explore how they impact NMT training.

Original: For the former approach, we use word2word (Choe et al., 2020) toconstruct bilingual lexicons and then extract wordpairs from parallel sentences.
Revised: For the former approach, we use word2word (Choe et al., 2020) to construct bilingual lexicons and then extract word pairs from parallel sentences.

Original: The extracted wordpairs are combined to form a phrase if words areconsecutive in the source and target sentence.
Revised: The extracted word pairs are combined to form a phrase if words are consecutive in the source and target sentence.

Original: Forthe latter approach, we use FastAlign (Dyer et al. and use only 1-to-1 mappings for training.
Revised: For the latter approach, we use FastAlign (Dyer et al., 2013) and use only 1-to-1 mappings for training.

Original: Word-level Contrastive Learning With the extracted alignments, we propose a word-level contrastive learning objective for the multilingual encoder by the motivation that the aligned wordswithin a sentence pair should have a similar contextual representation.
Revised: Word-level Contrastive Learning With the extracted alignments, we propose a word-level contrastive learning objective for the multilingual encoder by the motivation that the aligned words within a sentence pair should have a similar contextual representation.

Original: We expect the supervisionof the contrastive objective on the correspondingcontextual word representation leads to a robustmultilingual encoder.
Revised: We expect the supervision of the contrastive objective on the corresponding contextual word representation leads to a robust multilingual encoder.

Original: Assume that the tokenizedsource and target parallel sentences in  i − th batchare D i
Revised: Assume that the tokenized source and target parallel sentences in the i − th batch are D i

Original: = { src ij , tgt ij } Bj =1 , and the extracted alignments from all the sentence pairs in each batch are 
Revised: = { src ij , tgt ij } Bj =1 , and the extracted alignments from all the sentence pairs in each batch are A i

Original: A i = { s ik , t ik } Nk =1 , where B and N denote thebatch-size and the number of alignments, respectively.
Revised: = { s ik , t ik } Nk =1 , where B and N denote the batch-size and the number of alignments, respectively.

Original: Note that s ik and t ik may contain severaltokens after the word combination for word2wordor subword tokenization for NMT.
Revised: Note that s ik and t ik may contain several tokens after the word combination for word2word or subword tokenization for NMT.

Original: Then the wordlevel contrastive loss in a batch is:
Revised: Then the wordlevel contrastive loss in a batch is:

Original: L

Original: [Equation]

Original: i ) align = −

Original: [Equation]
Revised: [Equation]

Original: )where T denotes a similarity scaling temperature.
Revised: where T denotes a similarity scaling temperature [Table]

Original: The similarity between two words is measured by:sim
Revised: The similarity between two words is measured by:

Original: [Equation]
Revised: [Equation]

Original: where g ( x ) =

Original: W 2 σ ( W 1 x ) and ¯x denotes the average of contextual hidden states of the correspond-

Original: [Table]
Revised: [Table]

Original: language pair, the number of the word pairs extracted by word2word and FastAlign, respectively. Refer to
Revised: where g ( x ) =

Original: Appendix B for details of the dataset splits. ing subword positions on top of the multilingual encoder.
Revised: W 2 σ ( W 1 x ) and ¯x denotes the average of contextual hidden states of the corresponding subword positions on top of the multilingual encoder.

Original: Following (Chen et al., 2020), we use an
Revised: Following (Chen et al., 2020), we use an

Original: MLP between contrastive loss and the contextual representation for NMT loss.
Revised: MLP between contrastive loss and the contextual representation for NMT loss.

Original: ReLU activation is used for σ , W 1 is d × d and W 2 is d × d ′ , where d is the encoder’s hidden dimension and d ′ < d .
Revised: ReLU activation is used for σ , W 1 is d × d and W 2 is d × d ′ , where d is the encoder’s hidden dimension and d ′ < d .

Original: Finally, to jointly train with the NMT loss, we use the following equation to combine our proposed word-level contrastive loss for a batch:
Revised: Finally, to jointly train with the NMT loss, we use the following equation to combine our proposed word-level contrastive loss for a batch:

Original: [Equation]
Revised: [Equation]

Original: where N T is the number of the tokens within a batch, N T 2 N is a multiplier that scales the contrastive loss to be consistent with NMT loss, and w is a weight to balance the joint training.
Revised: where N T is the number of the tokens within a batch, N T 2 N is a multiplier that scales the contrastive loss to be consistent with NMT loss, and w is a weight to balance the joint training.

Original:  Experimental Settings
Revised:  Experimental Settings

Original: Datasets and Preprocessing We selected ten languages, including English (en), Estonian (et), Italian (it), Japanese (ja), Kazakh (kk), Burmese (my),
Revised: Datasets and Preprocessing We selected ten languages, including English (en), Estonian (et), Italian (it), Japanese (ja), Kazakh (kk), Burmese (my),

Original: Dutch (nl), Romanian (ro), Turkish (tr), Vietnamese (vi) from different language families to train the NMT systems.
Revised: Dutch (nl), Romanian (ro), Turkish (tr), Vietnamese (vi) from different language families to train the NMT systems.

Original: We used the parallel datasets from different domains for the selected nine language pairs, including IWSLT, WMT, and
Revised: We used the parallel datasets from different domains for the selected nine language pairs, including IWSLT, WMT, and

Original: ALT.
Revised: ALT.

Original: We followed mBART (Liu et al., 2020) for tokenization.
Revised: We followed mBART (Liu et al., 2020) for tokenization.

Original: Details are given in Appendix A.
Revised: Details are given in Appendix A.

Original: For each parallel dataset, we implemented two approaches as stated in Section 2 to extract word pairs for the contrastive training objective.
Revised: For each parallel dataset, we implemented two approaches as stated in Section 2 to extract word pairs for the contrastive training objective.

Original: Data source and the number of the extracted word pairs are shown in Table 1.
Revised: Data source and the number of the extracted word pairs are shown in Table 1.

Original: To ensure high alignmentquality, we used large-scale out-of-domain (see Appendix B) parallel corpora with FastAlign. Methods 222_en-ja 626_I 626_II MLSC 13. +align 13.
Revised: To ensure high alignment

Original: +w2w (ours) 13. +FA (ours) 13.
Revised: [Table] [Table]

Original: mBART FT 18. +align 18. +w2w (ours) 18. +FA (ours) 18.
Revised: quality, we used large-scale out-of-domain (see Appendix B) parallel corpora with FastAlign.

Original: Many-to-many NMT systems We establishedthree many-to-many NMT systems as follows:
Revised: Many-to-many NMT systems We established three many-to-many NMT systems as follows:

Original: 222_en-ja : Bidirectional en-ja NMT model us- ing en-ja parallel corpus.
Revised: 222_en-ja : Bidirectional en-ja NMT model using en-ja parallel corpus.

Original: 626_en-it-ja-nl-tr-vi : 6-to-6 multilingual NMT model using spoken language domain corpora for en-it, en-ja, en-nl, en-tr and en-vi.
Revised: 626_en-it-ja-nl-tr-vi : 6-to-6 multilingual NMT model using spoken language domain corpora for en-it, en-ja, en-nl, en-tr and en-vi.

Original: 626_en-tr-ro-et-my-kk : 6-to-6 multilingual NMT model using general domain corpora for en-tr, en-ro, en-et, en-my and en-kk.
Revised: 626_en-tr-ro-et-my-kk : 6-to-6 multilingual NMT model using general domain corpora for en-tr, en-ro, en-et, en-my and en-kk.

Original: Baselines and Ours For each language groupsetting above, we conducted NMT experimentson both the multilingual training from scratch( MLSC )
Revised: Baselines and Ours For each language group setting above, we conducted NMT experiments on both the multilingual training from scratch ( MLSC )

Original: (Johnson et al., 2017; Aharoni et al., 2019)and the mBART multilingual fine-tuning ( mBART
Revised: (Johnson et al., 2017; Aharoni et al., 2019) and the mBART multilingual fine-tuning ( mBART

Original: FT )
Revised: FT )

Original: (Tang et al., 2020) as baselines.
Revised: (Tang et al., 2020) as baselines.

Original: We applied ourproposed word-level contrastive learning in both
Revised: We applied our proposed word-level contrastive learning in both

Original: MLSC and mBART FT, and compared with anotherstrong baseline, word alignment based joint NMTtraining ( +align ) (Garg et al., 2019).
Revised: MLSC and mBART FT, and compared with another strong baseline, word alignment based joint [Figure] NMT training ( +align ) (Garg et al., 2019).

Original: For applyingour method, we investigated the performance ofjoint training with word pairs extracted by bothword2word ( +w2w ) and FastAlign ( +FA ). 
Revised: For applying our method, we investigated the performance of joint training with word pairs extracted by both word2word ( +w2w ) and FastAlign ( +FA ). We omitted Lin et al.

Revised: (2020) as a baseline because their method can not be applied to mBART fine-tuning, and they used high-quality ground-truth dictionaries, which are unavailable for most languages pairs.

Original: Implementation We used mBART-large formBART FT and transformer-base (Vaswaniet al., 2017) for MLSC.
Revised: Implementation We used mBART-large (mBART-25) for mBART FT and transformer-base (Vaswani et al., 2017) for MLSC.

Original: See Appendix C for details.
Revised: See Appendix C for details.

Original:  Results and Analyses
Revised:  Results and Analyses

Original: BLEU Results We report case-sensitive tokenized
Revised: BLEU Results We report case-sensitive tokenized

Original: BLEU (Papineni et al., 2002) results in Table 2and 3.
Revised: BLEU (Papineni et al., 2002) results in Tableand 2.

Original: In Table 2, we observe that with our proposed training objectives, BLEU scores are comparable in 222_en-ja and 626_en-it-ja-nl-tr-vi whilethey are slightly improved in 626_en-tr-ro-et-mykk.
Revised: In Table 3, we observe that with our pro- [Figure] posed training objectives, BLEU scores are comparable in 222_en-ja and 626_en-it-ja-nl-tr-vi while they are slightly improved in 626_en-tr-ro-et-mykk.

Original: However, “+align” performs comparable oreven worse compared with the baseline.Referring [Table]
Revised: However, “+align” performs comparable or even worse compared with the baseline.

Original: MLSC [Table] +align [Table]
Revised: Referring to Table 2 for specific BLEUs on each language pair, we find that with our methods, translation performances are significantly improved for mBART FT while nontrivial improvements can merely be observed on en-ro and en-kk direction for MLSC.

Original: + [Table] [Table] [Table]
Revised: This indicates that NMT fine-tuning on monolingual pre-trained models (mBART) may benefit more from our proposed methods.

Original: +FA ( [Table] to Table 3 for specific BLEUs on each language pair, we find that with our methods, translation performances are significantly improved for mBART FT while nontrivial improvementscan merely be observed on en-ro and en-kk direction for MLSC.
Revised: Note that the BLEU improvements for MLSC are not significant, and we explain why this happens in the “Word Retrieval P@1 is improved” part.

Original: Latent Encoder Alignment Property We now inspect which aspect of alignment-based methods impacts the translation performance.
Revised: Latent Encoder Alignment Property We now inspect which aspect of alignment-based methods impacts the translation performance.

Original: Previous work (Artetxe and Schwenk, 2019) show that the encoder of a strong multilingual NMT system is an ideal model for the bilingual sentence retrieval task. 
Revised: Previous work (Artetxe and Schwenk, 2019) showed that the encoder of a strong multilingual NMT system is an ideal model for the bilingual sentence retrieval task. In addition, Arivazhagan et al.

Revised: (2019a) introduced the correlation between the encoder-side sentence representation 1 and the translation quality.

Original: Inspired by this, we speculate that alignment-based objectives affect sentence retrieval performance, which further impacts the translation quality.
Revised: Inspired by these, we speculate that alignment-based objectives affect sentence retrieval performance, which further impacts the translation quality.

Original: We train MLSC and mBART FT and report the sentence retrieval precision and NMT loss during the training.
Revised: We train MLSC and mBART FT and report the sentence retrieval precision and NMT loss during the training.

Original: Results are reported in Figure 2.
Revised: Results are reported in Figure 1.

Original: We observe that the validation retrieval precision showthe similar trend as the NMT loss.
Revised: We observe that the validation retrieval precision show similar trends as the NMT loss.

Original: This indicates that duringthe normal many-to-many NMT training, encoder-side sentence-level retrieval precision is optimized along with the NMT loss.
Revised: This indicates that during many-to-many NMT training from scratch, encoder-side sentence-level retrieval precision is optimized along with the NMT loss.

Original: Sentence Retrieval P@1 Correlates with BLEU
Revised: Sentence Retrieval P@1 Correlates with BLEU

Original: According to the investigation of the encoder alignment property above, we verify the relationship  between BLEU score and sentence retrieval precision on the validation set for each language pair.
Revised: According to the investigation of the encoder alignment property above, we verify the relationship [Figure] between BLEU score and sentence retrieval precision on the validation set for each language pair.

Original: Results are shown in Figure 1.
Revised: Results are shown in Figure 2.

Original: Cross-referencing[Figure] [Figure] the BLEU score in Table 3, we found that BLEUscores are improved when the encoder achievesgains on the sentence retrieval precision.
Revised: Cross-referencing the BLEU score in Table 2, we found that BLEU scores are improved when the encoder achieves gains on the sentence retrieval precision.

Original: For example, we see increases of the retrieval P@1 on enro, en-et [Figure] and en-my on mBART FT (the middle of
Revised: For example, we see increases of the retrieval P@1 on enro, en-et, and en-my on mBART FT (the middle of

Original: Figure 1) while BLEU scores are significantly improved on these three language pairs (Table 3).
Revised: Figure 2) while BLEU scores are significantly improved on these three language pairs (Table 2).

Original: Wefurther calculate the Pearson correlation coefficientbetween the BLEU changes and sentence retrieval
Revised: We further calculate the Pearson correlation coefficient between the BLEU changes and sentence retrieval

Original: P@1 changes for mBART+align, mBART+w2w,and mBART+FA in the 626_en-tr-ro-et-my-kk setting.
Revised: P@1 changes for mBART+align, mBART+w2w, and mBART+FA in the 626_en-tr-ro-et-my-kk setting.

Original: Results are 0.79, 0.93, 0.90, respectively,demonstrating a strong correlation between translation quality and sentence retrieval precision.
Revised: Results are 0.79, 0.93, 0.90, respectively, demonstrating a strong correlation between translation quality and sentence retrieval precision.

Revised: Word Retrieval P@1 is Improved We probe the trained contextualized word representations on top of the encoder.

Revised: As shown in Figure 3, we observe that the word retrieval precision is improved in all the settings.

Revised: This demonstrates that the encoder parameters of the NMT system trained with our proposed objective are of a rather different distribution.

Revised: By just changing the random seed, we can expect similar BLEU results, but we cannot obtain a better aligned encoder.

Revised: However, the improvement of the word retrieval precision does not directly contribute to the translation quality, which we explain next.

Original: Word-level Contrastive Objective and Sentence
Revised: Word-level Contrastive Objective and Sentence

Original: Retrieval P@1 With the word-level contrastiveobjective, we observed significant BLEU score improvements on language pairs such as en-ro, en-etand en-my  as presented in Table 3.
Revised: Retrieval P@1 With the word-level contrastive objective, we observed significant BLEU score improvements on language pairs such as en-ro, en-et and en-my for mBART FT as presented in Table 2.

Original: However, due to the noises of extracted word pairs (Pan et al., 2021a) from word alignment toolkitsthat leads to insufficient supervision  for improving sentence retrieval P@1,  some language pairs such as en-kk do not show BLEU improvements.
Revised: However, noisy word pairs (Pan et al., 2021a) extracted via word alignment toolkits leads to poor supervision signals for improving sentence retrieval P@1, which in turn prevents some language pairs such as en-kk from exhibiting BLEU improvements.

Original: We found that for en-kk,  numbers of extracted word pairs per sentence by word2word and FastAlign are 1.0 and 2.2, respectively.
Revised: We found that for en-kk, the numbers of extracted word pairs per sentence by word2word and FastAlign are 1.0 and 2.2, respectively.

Original: In contrast, the numbers are 4.2 and 20.7 for improved language pairs, calculated from 
Revised: In contrast, these numbers are 4.2 and 20.7 for improved language pairs, calculated from Table 1.

Original:  Table 1.
Revised: Although better extracted word alignments for the word-level contrastive objective leads to BLEU improvements, its contribution towards improvements varies for MLSC and mBART FT, as shown in Table 2.

Original: We expect this finding to provide new perspectives for improving many-to-many NMT.
Revised: We expect these findings to provide new perspectives for improving many-to-many NMT.

Original: Sentence-level Contrastive Objective We conducted the experiments for  sentence-level contrastive objective(mRASP2) (Pan et al., 2021b) on 626_en-tr-ro-et-my-kk mBART FT.
Revised: Sentence-level Contrastive Objective We conducted the experiments for the sentence-level contrastive objective (Pan et al., 2021b) on all two six-to-six settings and compared it against our proposed approach.

Original: The average BLEU score of our +w2w is 20.89, which significantly outperforms mRASP’s 20.47 (last line in Table 8). Our word-level method outperforms the sentence-levelmethod, indicating the sentencelevel objective’s limitation.
Revised: The average BLEUs of our methods significantly outperform those of sentencelevel contrastive objectives (see Table 8 and 9), clearly showing the sentence-level objective’s limitation.

Original: Moreover, we checked the sentence retrieval P@1 for mRASP2 (last line in Table 10) and found that it correlates with BLEU changes, indicating that sentence-level contrastive objective is suboptimal for language pairs with decreased retrieval precision.
Revised: Moreover, we checked the sentence retrieval P@1 for Pan et al. (2021b) (Table 10 and 11) and found that it correlates with BLEU changes, indicating that sentence-level contrastive objective is suboptimal for language pairs with decreased retrieval precision.

Original:  Conclusion
Revised:  Conclusion

Original: We proposed a word-level contrastive learning objective for many-to-many NMT.
Revised: We proposed a word-level contrastive learning objective for many-to-many NMT.

Original: Experimental results showed that our proposed method leads to significantly better translation for several language pairs, which is then explained by analyses showing the relationship between BLEU scores and sentence retrieval performance of the NMT encoder.
Revised: Experimental re- sults showed that our proposed method leads to significantly better translation for several language pairs, which is then explained by analyses showing the relationship between BLEU scores and sentence retrieval performance of the NMT encoder.

Original: Future work can focus on: (1) further improving the encoder’s retrieval ability in many-to-manyNMT;
Revised: Future work can focus on: (1) further improving the encoder’s retrieval performance in many-to-many

Original:  (2) contrastive objective’s feasibility in a 
Revised: NMT; (2) contrastive objective’s feasibility in a massively multilingual scenario. Ethical Considerations

Original: massively
Revised: All the corpora we used in this paper are publicly available resources without the issue of the copyright. The technique this paper proposed is for NMT models, so it can not circumvent the issues that NMT models have. Since our automatically dictionaries are extracted from potentially biased data, the translations may also contain biases.

Original: multilingual scenario
Revised: However, we expect that these issues may be resolved by using unbiased data or the addition of debiasing objectives. Acknowledgements This work was supported by Grant-in-Aid for Young Scientists #19K20343, JSPS and Information/AI/Data Science Doctoral Fellowship of Kyoto

Revised: University.

Original: A Tokenization Settings
Revised: A Tokenization Settings

Original: For Japanese, we use Jumanpp (Morita et al., 2015;
Revised: For Japanese, we use Jumanpp (Morita et al., 2015;

Original: Tolmachev et al., 2018) for segmentation and wefollowing the setting in mBART (Liu et al., 2020)for other languages: myseg.py (Ding et al.
Revised: Tolmachev et al., 2018) for segmentation, and we follow the same settings as in mBART (Liu et al., 2020) for other languages: myseg.py (

Original: , 2020)is used for Burmese, Moses tokenization and special normalization is used for Romanian following (Sennrich et al., 2016), 3 and Moses tokenization for other languages. 
Revised: Ding et al., 2020) is used for Burmese, Moses tokenization and special normalization is used for Romanian following (Sennrich et al., 2016), 4 and Moses tokenization for other languages. 5 Following mBART, we apply SentencePiece (Kudo and Richardson, 2018) to further segment sentences into subwords.

Original: B Datasets and Alignment Extraction
Revised: B Datasets and Alignment Extraction

Original: The datasets used for NMT training, validationand test are shown in Table 4.
Revised: The datasets used for NMT training, validation and test are shown in Table 4.

Original: For the word alignment extraction using FastAlign, we also use outof-domain parallel corpora to train the FastAlignjointly, aiming to obtain word alignments withless noise. The out-of-domain corpora for all the Methods en-ja MLSC 3.
Revised: For the word align-

Original: +align 3. +w2w (ours) 73. +FA (ours) 69. mBART FT 88.
Revised: [Table]

Original: +align 87.

Original: +w2w (ours) 85.

Original: +FA (ours)

Original: [Table]
Revised: [Table]

Revised: ment extraction using FastAlign, we also use outof-domain parallel corpora to train the FastAlign jointly, aiming to obtain word alignments with less noise.

Original:  language pairs contain Tatoeba, Europarl, GlobalVoices, NewsCommentary, OpenSubtitles, TED,
Revised: The out-of-domain corpora for all the language pairs contain Tatoeba, Europarl, GlobalVoices, NewsCommentary, OpenSubtitles, TED,

Original: WikiMatrix, QED, GNOME, bible-uedin, and ASPEC (Nakazawa et al., 2016).
Revised: WikiMatrix, QED, GNOME, bible-uedin, and ASPEC (Nakazawa et al., 2016).

Original: We collect them from the OPUS project (Christodoulopoulos and
Revised: We collect them from the OPUS project (Christodoulopoulos and

Original: Steedman, 2015) and WAT.
Revised: Steedman, 2015) and WAT.

Original: The number of the out-of-domain parallel sentences for each language pair is shown in Table 4.
Revised: The number of the out-of-domain parallel sentences for each language pair is shown in Table 4.

Original: C Implementation Details
Revised: C Implementation Details

Original: Following Tang et al.
Revised: Following Tang et al.

Original: (2020), we set the oversampling temperature of 1.5 for all the settings.
Revised: (2020), we set the oversampling temperature of 1.5 for all the settings.

Original: For
Revised: For

Original: MLSC, we set the dropout of 0.3 to avoid overfitting on small-scale training data.
Revised: MLSC, we set the dropout of 0.3 to avoid overfitting on small-scale training data.

Original: We used the batch size of 1,024 tokens for all the settings.
Revised: We used the batch size of 1,024 tokens for all the settings.

Original: For our word-level contrastive learning, we set the weight of 0.1, the temperature of 0.2, d ′ of 128, and asmaller dropout of 0.2 because our proposed objective serves as a regularization part. We followed the hyperparameter setting of Garg et al. (2019) for word alignment-based joint NMT training. We used 8 NVIDIA A100 for mBART FT and 8 TITAN Xp for MLSC model training.
Revised: For our word-level contrastive learning, we set the weight of 0.1, the temperature of 0.2, d ′ of 128, and a

Original: The model is validated every 1000 steps for 222_en-ja and steps for both two 626 settings.

Original: We do the early stopping if no improvement of the validation loss is observed for 8 checkpoints.

Original: The model with the best validation loss was used for evaluation.

Original: D BLEU Scores

Original: We report all the BLEU results of 222_en-ja, 626_en-it-ja-nl-tr-vi, and 626_en-tr-ro-et-my-kk in

Original: Methods [Table] w2w (ours) [Table]
Revised: Methods en-ja en-vi en-it en-nl en-tr

Original: FA (ours)

Original: [Table]

Original: +FA (ours)

Original: [Table]

Original: [Table]

Original: en-my

Original: Avg.
Revised: Avg.

Original: → ← → ← → ← → ← → ←
Revised: → ← → ← → ← → ← → ←

Original: MLSC 9.
Revised: MLSC 15.

Original: +align 9.
Revised: +align 15.

Original: +w2w (ours) 9.
Revised: +w2w (ours) 15.

Original: +FA (ours) 9.
Revised: +FA (ours) 15.

Revised: +sent 15.

Original: mBART FT 17.
Revised: mBART FT 17.

Original: +align 17.
Revised: +align 17.

Original: +w2w (ours) 17.
Revised: +w2w (ours) 17.

Original: +FA (ours) 17. 
Revised: +FA (ours) 17. +sent 17. [Table]

Original: +Sent (mRASP2) 17. 
Revised: [Table] +FA (ours) 17. [Table] [Table]

Original: improvements are test
Revised: [Table] +sent 17. [Table] smaller dropout of 0.2 because our proposed objective serves as a regularization part.

Revised: We followed the hyperparameter setting of Garg et al.

Revised: (2019) for word alignment-based joint NMT training.

Revised: We

Revised: [Table]

Revised: used 8 NVIDIA A100 for mBART FT and 8 TITAN Xp for MLSC model training.

Revised: The model is validated every 1000 steps for 222_en-ja andsteps for both two 626 settings.

Revised: We do the early

Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Revised: stopping if no improvement of the validation loss is observed for 8 checkpoints.

Revised: The model with the best validation loss was used for evaluation.

Revised: D BLEU Scores

Revised: We report all the BLEU results of 222_en-ja, 626_en-it-ja-nl-tr-vi, and 626_en-tr-ro-et-my-kk in

Original: Table 5, 7 and 8, respectively
Revised: Table 5, 8 and 9, respectively.

Original: E Sentence Retrieval Precision
Revised: E Sentence Retrieval Precision

Original: We report the sentence retrieval precisions for all the systems in Table 6, 9 and 10.
Revised: We report the sentence retrieval precision for all the systems in Tables 6, 10 and 11.

Revised: The sentence retrieval previsions are evaluated by using the validation dataset of each language pair.

Revised: The mean pooled encoder output is used as the sentence embedding.

Revised: We use cosine similarity to conduct the retrieval task, and report the average retrieval precision of both directions of each language pair.

Revised: F Word Retrieval Precision

Original: [Table]
Revised: We report the word retrieval precision for all the systems in Tables 7, 12, and 13. The word retrieval precision are computed by using the validation dataset and the word2word alignments on it. The mean pooled encoder output on corresponding positions is used as the contextualized word embedding. We use cosine similarity to implement the retrieval for word pairs in a batch, and present the average in-batch retrieval precision of both directions of each language pair.

Revised: Batch size is set as 512 tokens.