Exploring Pair-Wise NMT for I ndian Languages (2020)

Abstract: In this paper, we address the task of improving pair-wise machine translation for specific low resource Indian languages. Multilingual NMT models have demonstrated a reasonable amount of effectiveness on resource-poor languages. In this work, we show that the performance of these models can be significantly improved upon by using back-translation through a filtered back-translation process and subsequent fine-tuning on the limited pair-wise language corpora. The analysis in this paper suggests that this method can significantly improve multilingual models’ performance over its baseline, yielding state-of-the-art results for various Indian languages.

## introduction neural machine translation (nmt) algorithms, as is common for most deep learning techniques, work best with vast amounts of data. various authors have argued that their performance would be limited for low resource languages @xcite , @xcite , @xcite . one way to bridge this gap is through the use of multilingual nmt algorithms to bypass the data limitations of individual language pairs ( @xcite , @xcite , @xcite ). the use of such a model has been demonstrated recently by @xcite . in this paper, we investigate the problem of improving pair-wise nmt performance further over existing multilingual baselines. we specifically analyze the use of backtranslation and fine-tuning to this effect. our results suggest that it is possible to improve the performance of individual pairs of languages for various indian languages. the performance of these language pairs is evaluated over standard datasets, and we observe consistent improvements over the two main baselines: an nmt system trained pair-wise from scratch using the corpora available for the pair of languages, a multilingual nmt model that uses many different languages. ## previous work the problem of multilingual nmt has attracted significant research attention in the recent past. @xcite proposed the first multilingual model with a one-to-many mapping of languages, whereas @xcite shared a single attention network for all language pairs. recent works like @xcite , @xcite which are an extension of @xcite and @xcite have improved upon the initial formulation of the multilingual nmt problem. works like @xcite , @xcite , @xcite and @xcite use monolingual data to supplement their parallel corpora to build an nmt system. on the flip side @xcite , @xcite , @xcite study the unsupervised paradigm using only monolingual corpora. within the context of indian languages, @xcite and @xcite were one of the first works to explore a rule-based approach for translation from hindi to english whereas @xcite , @xcite , @xcite and @xcite have explored this problem through the prism of nmt. @xcite and @xcite extend the concept of multilingual nmt to the setting of indian languages. due to the recent efforts undertaken by the authors of @xcite , indian languages are now better represented in terms of available monolingual corpora. these resources set up a fertile ground for exploitation by semi-supervised and unsupervised nmt approaches, which are consistent with the setting we study in this work. ## method consider a setting in which we have limited pair-wise corpora between a pair of languages, and we would like to obtain improved performance. we go about achieving this through the following procedure. first, we train a multilingual model on several languages. next, we use existing monolingual corpora through back-translation (bt), and then we fine-tune the model using available pair-wise corpora. we show that this particular procedure indeed improves over the alternative approach of training a pair-wise nmt system using the available corpora. for the first step, we use the multilingual nmt model provided publicly by @xcite . we now provide details regarding the other two stages. ## results and discussions we report bleu scores on all the test sets specified. we refer to our approach as filt-bt in the first comparison highlights the benefits of warm-starting our nmt model from a m-nmt model, whereas the second comparison helps us ascertain the efficacy of filtered bt and as such, we report consistent gains over both these baselines for all the language pairs. in all the language pairs barring odia, we demonstrate the superior performances of a prior multilingual model in contrast to a specialized model trained from scratch, validating our claim that initialization using a multilingual model is highly effective for nmt in contrast to initializing randomly and training only on the respective language. typically, we observe that using high threshold values for filtering leads to the filtered corpus getting biased by selecting comparatively shorter sentences. to maintain a healthy mix of both types of sentences, we use a threshold value slightly higher than the mean of the sentence-wise bleu scores which we find in our experiments empirically provides for a more balanced high quality (in terms of translation quality) corpus thereby guaranteeing a better supervisory signal. for ta and ur, we notice a massive boosts in performance (11.88 and 10.49 bleu points respectively) over our multilingual baseline, with significant gains, also being noticed for ml and gu. the m-nmt model, which we use to initialize our nmt model, has been trained on odiencorpv1.0, whereas our rand-init model has been trained using both versions of the dataset. we ascribe the former's inferior performance compared to the latter on odia to the domain mismatch between both these versions, something which the latter model does not have to face. we select a subset of the monolingual data to maintain consistency with our computing resources. for 200k sentences, we train on a 1080ti nvidia gpu and found that back translation took about 3 hours. we decided that it would be an adequate sample size to test the validity of our approach. since only a subset of monolingual data provided by @xcite is used, we fully expect these results to trend upwards if the entire corpora were to be utilised. we indicate the sota performance for each language in bold. as such, we achieve sota performances on pa, gu, ml, mr, ta and ur. ## summary and directions our explorations in the applicability of neural machine translation for indian languages lead to the following observations (i) multilingual models are a promising direction to address data scarcity and the variability of resources across languages (ii) adapting a multilingual model for a specific pair can provide superior performances. we believe these solutions can further benefit from the availability of monolingual resources and noisy parallel corpora.