Iterative Data Augmentation for Neural Machine Translation: a Low Resource Case Study for E nglish- T elugu (2018)

Abstract: Telugu is the fifteenth most commonly spoken language in the world with an estimated reach of 75 million people in the Indian subcontinent. At the same time, it is a severely low resourced language. In this paper, we present work on English–Telugu general domain machine translation (MT) systems using small amounts of parallel data. The baseline statistical (SMT) and neural MT (NMT) systems do not yield acceptable translation quality, mostly due to limited resources. However, the use of synthetic parallel data (generated using back translation, based on an NMT engine) significantly improves translation quality and allows NMT to outperform SMT. We extend back translation and propose a new, iterative data augmentation (IDA) method. Filtering of synthetic data and IDA both further boost translation quality of our final NMT systems, as measured by BLEU scores on all test sets and based on state-of-the-art human evaluation.

## introduction in the past two decades, machine translation (mt) has shown very promising results, most of which have been achieved using data-driven techniques. in recent years, the data-driven paradigm of mt is largely dominated by neural machine translation (nmt) and showing significant success over its predecessor statistical machine translation (smt) @xcite . the performance of any data-driven approach to mt mostly depends on the amount of parallel corpora available to train them. this problem is exacerbated by nmt, which generally needs larger quantities of parallel data and is less robust to noisy data. unfortunately, large amounts of readily available parallel resources exist only for a small number of languages, e.g., opus @xcite and europarl @xcite , with only very few sources of indic language data. indic language mt is difficult due to complex linguistic structure and lack of good quality data. most of the indic languages are leading languages of the world in terms of number of speakers but are very poorly resourced (i.e., only very little machine-readable parallel text exists) so building a general domain data-driven mt system is a challenging problem. also, indic languages do not have enough comparable resources to explore extraction of useful parallel content from the same @xcite . lastly, due to the usage of multiple fonts and encodings, a significant portion of the web data cannot be used to extract parallel data for training. telugu is no exception to this. lack of large, high quality parallel resources makes the development of general purpose mt systems much harder for telugu compared to other, resource rich languages, more specifically when building nmt-based models. one of the major problems with training an nmt system on little data, especially when training an engine for general usage (i.e., not domain specific), is overfitting. deep neural networks have large parameter spaces and need ample amounts of data in order to generalize adequately; with small amounts of data they tend not to generalize well. we address this issue by learning the optimizer over a smaller number of training steps. in this paper, we describe our english-telugu (en-te) general purpose mt system. first, we describe the baseline smt-based and nmt-based systems trained on 750k parallel sentences. telugu is a morphologically rich language and, as such, suffers from a high out-of-vocabulary (oov) rate in a low data scenario. we address data sparsity by augmenting a large amount of synthetic training data @xcite , generated using back translation, to iteratively improve the nmt systems. the iterative process uses synthetic data to improve the mt engine and (implicitly) the quality of the synthetic data using the improved mt engine in the reverse direction. secondly, we use sub-word representations to reduce the data sparsity problem. this essentially handles telugu's rich morphology. furthermore, as translation quality varies across sentences while generating synthetic data, we filter poor quality translation pairs to augment the system only with high quality synthetic parallel data. we observe improved translation quality as a result. the main finding of this work is that the use of iterative data augmentation and filtering of the synthetic data help to improve the translation quality. the rest of the paper is organized as follows. section 2 describes the data sets used to build the systems. in section 3, we describe the baseline smt and nmt models and their quality. section 4 provides details on improving nmt models using synthetic data. section 5 reports the experimental setup and results. we conclude in section 6. ## data sets in this work, we use two types of training data: true parallel data, and synthetically generated parallel data using back-translation @xcite . in this section, we describe the true english-telugu parallel data used for system training. the generation of synthetic data is explained in section 4.1. the full training data contains 750k true parallel sentences along with a larger set of synthetic data (15.4m and 8.2m for @xmath0, respectively). the true parallel data includes automatically extracted parallel sentences from the web and from opus @xcite . many web pages feature content available in multiple languages. such content includes both sentence or paragraph aligned parallel data (e.g., ted talks' transcriptions) and comparable or noisy-parallel corpora (e.g., cross-lingually linked wikipedia documents). once such potential parallel pages between telugu and english are extracted from the web, a sentence aligner is used to extract sentence aligned parallel text, based on a modified moore sentence aligner @xcite . test data to the best of our knowledge, there are no publicly available test sets for evaluating te-en mt systems. thus, we have created two different test sets to evaluate our systems. our first test set was created by selecting sentences from news articles. the english source sentences were manually translated into telugu and validated by human experts. we shall refer this test set as news. in order to understand the performance of our systems w.r.t. state-of-the-art test sets, we have created our second test set using a subset of the wmt 2009 @xcite test set for english-french. 1,000 english sentences were randomly selected and manually translated into telugu by human experts. we call this test set wmt. note that we have created our test sets with a single reference translation. we intend to publicly release the test sets. monolingual data mentioned in ## baseline models the baseline smt models use a vanilla phrasal @xcite and a treelet foot_0 @xcite translation model for @xmath1, respectively. we do not use treelet translation system in the @xmath2. for both phrasal and treelet systems, word alignment is done using giza++ @xcite . we use the target side of the parallel corpus along with additional monolingual target language data to train a 5-gram language model using modified kneser-ney smoothing @xcite . finally, we use mert @xcite to estimate the lambda parameters using the held out dev data with a single reference translation. the baseline nmt model is developed based on the architecture described in @xcite . the encoder uses a 3-layer bi-directional rnn (consists of 512 lstm units). the decoder uses an lstm layer in the bottom to capture the context and the attention. the lstm layer is then followed by 5 fully-connected layers applied in each timestep using a resnet-style skip connection @xcite . the details of the model and equations are described in @xcite . all the models are trained using adam optimizer @xcite with a dropout rate of 0.25. the optimizer uses 100k and 500k steps with a batch size of 1024 for @xmath3, respectively. in the case of @xmath4, source-side telugu sentences are represented using byte-pair encoding (bpe) @xcite to reduce the data sparsity problem, which uses 50,000 merging operations. ## experiments and results first, we conducted one experiment to see the effect of choosing smt and nmt system as the reverse translation engine to produce back-translated data (line 1 and 2 in algorithm 1). note that our baseline smt system has better quality compared to the baseline nmt system (cf. ## conclusion we have demonstrated that we can build good quality nmt models with limited resources for a morphologically rich language pair. contributions of this paper are the definition of iterative data augmentation (ida) and empirical results showing the effectiveness of back translation and pfsbased data filtering for english-telugu nmt. the proposed ida method is much more effective than using baseline back translation by itself.