Data Augmentation with Diversified Rephrasing for Low-Resource Neural Machine Translation (2023)

Abstract: Data augmentation is an effective way to enhance the performance of neural machine translation models, especially for low-resource languages. Existing data augmentation methods are either at a token level or a sentence level. The data augmented using token level methods lack syntactic diversity and may alter original meanings. Sentence level methods usually generate low-quality source sentences that are not semantically paired with the original target sentences. In this paper, we propose a novel data augmentation method to generate diverse, high-quality and meaning-preserved new instances. Our method leverages high-quality translation models trained with high-resource languages to rephrase an original sentence by translating it into an intermediate language and then back to the original language. Through this process, the high-performing translation models guarantee the quality of the rephrased sentences, and the syntactic knowledge from the intermediate language can bring syntactic diversity to the rephrased sentences. Experimental results show our method can enhance the performance in various low-resource machine translation tasks. Moreover, by combining our method with other techniques that facilitate NMT, we can yield even better results.

## introduction current neural machine translation (nmt) @xcite @xcite systems, especially those based on transformer @xcite , have achieved human-level performance in translation quality @xcite . these systems are trained using hundreds of millions of sentence pairs to ensure that they can generalize to unseen instances. however, large-scale parallel data is scarce and only available for a few high-resource language pairs @xcite . thus, the generalization of low-resource nmt models is far below an acceptable standard. recently, data augmentation @xcite @xcite @xcite has shown to be an effective way to improve the generalization of nmt models, especially for low-resource languages @xcite . existing data augmentation methods for nmt can be categorized into token level or sentence level methods. token level methods randomly replace words with rare words in both source and target sides to enhance the translation of rare words @xcite , or introduce token level noises in the source side @xcite @xcite @xcite to improve the robustness of models @xcite . sentence level methods are mainly based on backtranslation @xcite , which uses target side monolingual data to synthesize pseudo-parallel data. variants of back-translation include iterative backtranslation @xcite , data diversification @xcite and meta back-translation @xcite . we argue that existing data augmentation methods for low-resource translations have two major limitations: (i) token level methods perform token level manipulations (e.g., drop, reorder, replace) to generate new training data; thus, the generated sentences lack syntactic diversity; moreover, the token level manipulations may change the original meanings @xcite ; (ii) sentence level methods take natural sentences as input and generate synthetic corresponding translations using pre-trained low-quality models that are susceptible to errors @xcite , hence the augmented sentences often struggle to capture the complete semantics in the original sentences, resulting in the failure to semantically align with the target sentences. @xcite also noted the importance of the quality of augmented sentences. in this paper, we propose a simple yet effective data augmentation method, bidirectional translation-based data augmentation (bitda), to generate meaning-preserved and syntacticdiverse new training data for nmt. bitda uses pairs of high-quality translation models to rephrase the original sentences for low-resource translation. for example, for the māori english translation, the original english translation/sentence of a māori sentence is first translated into an intermediate high-resource language (e.g., german or french) and then translated back into english. in this way, we obtain one more english translation for the māori sentence. instead of applying the translation models trained on the original low-resource data as back-translation does, we use the high-quality translation models trained with high-resource languages to generate new sentences. high-resource models generally yield higher-quality translations compared to low-resource translation models, leading to an enhancement in the quality of generated sentences. on the other hand, the knowledge of an intermediate language learned by the high-resource models can be injected into the generated sentences and resulting in syntactic diversity. to evaluate the effectiveness of bitda, we conduct experiments on eight low-resource translation tasks. experimental results show that our method significantly and consistently improves the translation performance for low-resource machine translation. we further combine our proposed method with other techniques that facilitate nmt, and the results demonstrate that bitda works well with the other techniques that facilitate nmt and achieves better results. ## experiments in this section, we conduct experiments in a wide range of low-resource translation directions with different corpora sizes and languages to demonstrate the effectiveness of our method. in addition to the main experiments, we combine our method with other techniques to further improve the performance of translation models. ## limitations the limitations of our method are as follows: (i) it is restricted to the high-resource language side (e.g., english) of low-resource parallel data. while it is possible to use pairs of pre-trained low-resource translation models like bt can rephrase non-english sentences, the quality of the generated sentences would be too low. (ii) it can be affected by domain shift @xcite of the translation models we use. as seen in