Simulated multiple reference training improves low-resource machine translation (2020)

Abstract: Many valid translations exist for a given sentence, yet machine translation (MT) is trained with a single reference translation, exacerbating data sparsity in low-resource settings. We introduce Simulated Multiple Reference Training (SMRT), a novel MT training method that approximates the full space of possible translations by sampling a paraphrase of the reference sentence from a paraphraser and training the MT model to predict the paraphraser’s distribution over possible tokens. We demonstrate the effectiveness of SMRT in low-resource settings when translating to English, with improvements of 1.2 to 7.0 BLEU. We also find SMRT is complementary to back-translation.

## introduction variability and expressiveness are core features of language, and they extend to translation as well. @xcite showed that naturally occurring sentences have billions of valid translations. despite this variety, machine translation (mt) models are optimized toward a single translation of each sentence in the training corpus. training a high resource mt model on millions of sentence pairs likely exposes it to similar sentences translated different ways, but training a low-resource mt model with a single translation for each sentence (out of potentially billions) exacerbates data sparsity. we hypothesize that the discrepancy between linguistic diversity and standard single-reference training hinders mt performance. this was previously impractical to address, since obtaining multiple human translations of training data is typically not feasible. however, recent neural sentential paraphrasers produce fluent, meaning-preserving english paraphrases. we introduce a novel method that incorporates such a paraphraser directly in the training objective, and uses it to simulate the full space of translations. we demonstrate the effectiveness of our method on two corpora from the low-resource material program, and on bitext from globalvoices. we release data & code: data.statmt.org/smrt ## method we propose simulated multiple reference training (smrt), which uses a paraphraser to approximate the full space of possible translations, since explicitly training on billions of possible translations per sentence is intractable. in standard neural mt training, the reference is: (1) used in the training objective; and (2) conditioned on as the previous target token. 1 we approximate the full space of possible translations by: (1) training the mt model to predict the distribution over possible tokens from the paraphraser at each time step; and (2) sampling the previous target token from the paraphraser distribution. figure 1 shows an example of possible paraphrases and highlights a sampled path and some of the other tokens used in the training objective distribution. we review the standard nll training objective, and then introduce our proposed objective. ## results results are shown in ## analysis we analyze our method to explore: (1) how it performs at a various resource levels; ## conclusion we present simulated multiple reference training (smrt), which significantly improves performance in low-resource settings-by 1.2 to 7.0 bleu-and is complementary to back-translation. neural paraphrasers are rapidly improving @xcite @xcite @xcite , and the concurrently released prism multi-lingual paraphraser @xcite has coverage of 39 languages and outperforms prior work in english paraphrasing. as paraphrasing continues to improve and cover more languages, we are optimistic smrt will provide larger improvements across the board-including for higher-resource mt and for additional target languages beyond english.