Neural Machine Translation in Low-Resource Setting: a Case Study in E nglish- M arathi Pair (2021)

Abstract: In this paper and we explore different techniques of overcoming the challenges of low-resource in Neural Machine Translation (NMT) and specifically focusing on the case of English-Marathi NMT. NMT systems require a large amount of parallel corpora to obtain good quality translations. We try to mitigate the low-resource problem by augmenting parallel corpora or by using transfer learning. Techniques such as Phrase Table Injection (PTI) and back-translation and mixing of language corpora are used for enhancing the parallel data; whereas pivoting and multilingual embeddings are used to leverage transfer learning. For pivoting and Hindi comes in as assisting language for English-Marathi translation. Compared to baseline transformer model and a significant improvement trend in BLEU score is observed across various techniques. We have done extensive manual and automatic and qualitative evaluation of our systems. Since the trend in Machine Translation (MT) today is post-editing and measuring of Human Effort Reduction (HER) and we have given our preliminary observations on Translation Edit Rate (TER) vs. BLEU score study and where TER is regarded as a measure of HER.

## introduction the aim of this work is to improve the quality of machine translation (mt) for the english-marathi language pair for which less amount of parallel corpora is available. one of the major requirements for good performance of the neural machine translation (nmt) models is the availability of a large parallel corpora. as a result, there is a need to come up with additional resources by augmenting parallel corpora or by using knowledge from other tasks using transfer learning. @xcite have shown that the lexical and orthographic similarities among languages can be utilized to improve translation quality between indic languages when limited parallel corpora is available. english and marathi does not have common ancestry and hence are not related languages, whereas hindi and marathi are related languages. also, among the various english to indic language corpora, english-hindi corpus is comparatively larger. in our pivot based transfer learning, combined corpus, and multilingual experiments we try to utilize this high resource english-hindi language pair in various ways to assist in english-marathi translation. in our phrase table injection (pti) experiment, we explore how the phrases generated during statistical machine translation (smt) model training can be further utilized in nmt. we also explore how the monolingual corpus of the target language can be leveraged to create additional pseudo-parallel sentences using back-translation. we also try to understand the correlation between the bleu and translation edit rate (ter) scores by fitting a linear regression line on the ter vs bleu graph, where ter is regarded as a measure of human effort reduction (her). ## related work transformer model @xcite was introduced in 2017 and gave significant improvements in the quality of translation as compared to the previous recurrent neural network (rnn) based approaches @xcite @xcite . self-attention and absence of recurrent layers enabled models to train faster and get better performance. however, this did not help improve the translation quality in the low-resource setting. various methods have been proposed over the years to deal with the low-resource nmt problem. some methods which use monolingual data involve integrating a separately trained language model @xcite into the decoder, using an autoencoding objective @xcite or augmenting pseudo-parallel data using back-translation @xcite . @xcite introduced a method for combining smt and nmt by taking phrases from smt training and augmenting them to nmt. @xcite introduced a transfer learning approach where a parent model trained on a high resource language pair is used to initialize some parameters of the child model, which is then trained on a low-resource language pair. @xcite also uses a transfer learning approach with the help of a pivot language to learn parameters initially which are then transferred. multi-lingual nmt @xcite @xcite is another approach which uses knowledge transfer among various languages to improve the performance of all the language pairs involved. ## our approaches in this section, we discuss the details of the various techniques that we have explored to deal with the problem of low-resource english-marathi language pair. @xcite and @xcite used this technique, shown in figure 1 , to combine both smt and nmt. we know that the phrase table, generated during training of a smt model, plays a key role in the smt translation process. it contains a probabilistic mapping of phrases from the source language to the target language. the phrases present in the phrase table are combined with the available parallel corpora; thereby increasing the data available to train the nmt model. this also helps the model to learn translation of short correct phrases along with long sentences. ## experiments in this section, we discuss the details of the various experiments that we have carried out to improve the quality of the english-marathi translation. ## results and analysis we use the bleu evaluation metric @xcite to report our results. sacrebleu @xcite python library was used to calculate the bleu scores. we detokenize the translated sentences before calculating the bleu scores. the results of all our experiments are summarized in in pti experiment we observe an increase in bleu score on wat 2021 test set while the bleu score on ilci test set decreases. for combined corpus there is improvement of more than 1.5 bleu score on both the test sets, indicating that english-hindi corpus helped during the training. we observe that using back-translation, the bleu score decreases. this can be attributed to the fact that the marathi-english model used for back-translating the marathi monolingual corpora was not of good quality. this marathi-english model was trained using 0.25m parallel sentences which affects the quality of back-translated data. we also tried out experiments by combining the above mentioned methods, among which, the combination of phrase table injection and combined corpus methods give the best results. the results of the direct pivoting technique show an improvement of 2.29 bleu score over the baseline model on the ilci test set and of 0.42 on the wat 2021 test set. the results of step-wise pivoting show an improvement of 1.91 bleu score over the baseline on the ilci test set and of 0.74 on the wat 2021 test set. the reason for this bleu score increase is that, the encoder and decoder used to initialize the english-marathi model before training have already learned some representations. this is because the encoder and decoder are initialized from the encoder and decoder of the trained english-hindi (source-pivot) and hindi-marathi (pivot-target) models, respectively. we observe that this initialization of encoder and decoder performs better than a random initialization. the results of the multilingual model on english-marathi translation task show a bleu score increase of 2.8 on the ilci test set and 0.83 on the wat 2021 test set over the baseline model. in a multilingual model, we use a shared decoder for both the target languages, due to which, the representations learnt by the model for the task of english-hindi translation helps in the english-marathi task as well. this leads to a better performance of the multilingual model over the baseline model. for direct pivoting, step-wise pivoting and multilingual model we observe that the bleu score increase on ilci test set is more than that on the wat 2021 test set. our conjecture is that as the size of the ilci corpus used in training is larger than that of the pmindia corpus(from which wat 2021 test set is derived), the bleu score increase for ilci test set is more. ## extensive evaluation in this section, we discuss the analysis that we have carried out to compare our models with baseline; and also understand the correlation of bleu score with ter, where ter is regarded as a measure of her. ## conclusion and future work in this work, we have implemented and compared various techniques to improve the task of translation involving a low-resource english-marathi language pair. we have shown that the pivot based transfer learning approach can significantly improve the quality of the english-marathi translations over the baseline by using hindi as an assisting language. we also observe that the phrases from the smt training can help the nmt model perform better. the one (english) to many (hindi, marathi) multilingual model is able to improve the english-marathi translations by leveraging the english-hindi parallel corpus. combined corpus experiment also uses the english-hindi parallel corpus to improve the english-marathi translation quality. in future, we plan to further extend these approaches to a variety of languages to understand how the phenomenon of language relatedness can help improve the translation quality in low resource setting. we also plan to explore how multiple pivot languages can be used while translating from some source to target language pair.