Exploring diversity in back translation for low-resource machine translation (2022)

Abstract: Back translation is one of the most widely used methods for improving the performance of neural machine translation systems. Recent research has sought to enhance the effectiveness of this method by increasing the ‘diversity’ of the generated translations. We argue that the definitions and metrics used to quantify ‘diversity’ in previous work have been insufficient. This work puts forward a more nuanced framework for understanding diversity in training data, splitting it into lexical diversity and syntactic diversity. We present novel metrics for measuring these different aspects of diversity and carry out empirical analysis into the effect of these types of diversity on final neural machine translation model performance for low-resource English↔Turkish and mid-resource English↔Icelandic. Our findings show that generating back translation using nucleus sampling results in higher final model performance, and that this method of generation has high levels of both lexical and syntactic diversity. We also find evidence that lexical diversity is more important than syntactic for back translation performance.

## introduction the data augmentation technique of back translation (bt) is used in nearly every current neural machine translation (nmt) system to reach optimal performance @xcite @xcite . it involves creating a pseudo-parallel dataset by translating target-side monolingual data into the source language using a secondary nmt system @xcite . in this way, it enables the incorporation of monolingual data into the nmt system. whilst adding data in this way helps nearly all language pairs, it is particularly important for low-resource nmt where parallel data is scarce by definition. because of its ubiquity, there has been extensive research into how to improve bt @xcite @xcite , especially in ways which increase the 'diversity' of the back-translated dataset @xcite . previous work @xcite @xcite has found that machine translations lack the diversity of human productions. this is because most translation systems use some form of maximum a-posteriori (map) estimation, meaning that they will always favour the most probable output. @xcite and @xcite argue that this makes standard bt data worse training data since it lacks 'richness' or diversity. despite the focus on increasing diversity in bt, what 'diversity' actually means in the context of nmt training data is ill-defined. in fact, @xcite point out that there is no standard metric for measuring diversity. most previous work uses the bleu score between candidate sentences or another n-gram based metric to estimate similarity @xcite @xcite @xcite @xcite . however, such metrics mostly measure changes in the vocabulary or spelling. because of this, they are likely to be less sensitive to other kinds of variety such as changes in structure. we argue that quantifying 'diversity' using ngram based metrics alone is insufficient. instead, we split diversity into two aspects: variety in the word choice and spelling, and variety in structure. we call these aspects lexical diversity and syntactic diversity respectively. here, we follow recent work in natural language generation and particularly paraphrasing (e.g. @xcite @xcite @xcite which explicitly models the meaning and form of the input separately. of course, there are likely more kinds of diversity than this, but this distinction provides a common-sense framework to extend our understanding of the concept. to our knowledge, no other previous work in data augmentation has attempted to isolate and automatically measure syntactic and lexical diversity. building from our definition, we introduce novel metrics aimed at measuring lexical and syntactic diversity separately. we then carry out an empirical study into what effect training data with these two kinds of diversity has on final nmt performance in the context of low-resource machine translation. we do this by creating bt datasets using different generation methods and measuring their diversity. we then evaluate what impact different aspects of diversity have on final model performance. we find that a high level of diversity is beneficial for final nmt performance, though lexical diversity seems more important than syntactic diversity. importantly though there are limits to both; the data should not be so 'diverse' that it affects the adequacy of the parallel data. we summarise our contributions as follows: we present two novel metrics for measuring these different aspects of diversity. ## methods we explain each method we use for creating diverse bt datasets in section 2.1, then discuss our metrics for diversity in section 2.2. ## experiments having discussed the methods by which we generate diverse bt datasets and the metrics with which we measure the diversity in these datasets, we now outline our experimental set up for testing the effect of training data diversity on final nmt model performance. ## related work improving back translation the original paper introducing bt by @xcite found that using a higher-quality nmt system for bt led to higher bleu scores in the final trained system. this finding was corroborated by @xcite , and following work has investigated further ways to improve nmt. these include iterative bt @xcite , targeting difficult words @xcite , and tagged bt @xcite presents a comprehensive survey of bt and its variants as applied to low-resource nmt. ## conclusion in this paper, we introduced a two-part framework for understanding diversity in nmt data, splitting it into lexical diversity and syntactic diversity. our empirical analysis suggests that whilst high amounts of both types of diversity are important in training data, lexical diversity may be more beneficial than syntactic. in addition, achieving high diversity in bt should not be at the expense of ad-equacy. we find that generating bt with nucleus sampling results in the highest final nmt model performance for our systems. future work could investigate further the affect of high lexical diversity on bt independent of syntactic diversity.