Investigating Low-resource Machine Translation for E nglish-to- T amil (2020)

Abstract: Statistical machine translation (SMT) which was the dominant paradigm in machine translation (MT) research for nearly three decades has recently been superseded by the end-to-end deep learning approaches to MT. Although deep neural models produce state-of-the-art results in many translation tasks, they are found to under-perform on resource-poor scenarios. Despite some success, none of the present-day benchmarks that have tried to overcome this problem can be regarded as a universal solution to the problem of translation of many low-resource languages. In this work, we investigate the performance of phrase-based SMT (PB-SMT) and neural MT (NMT) on a rarely-tested low-resource language-pair, English-to-Tamil, taking a specialised data domain (software localisation) into consideration. In particular, we produce rankings of our MT systems via a social media platform-based human evaluation scheme, and demonstrate our findings in the low-resource domain-specific text translation task.

## introduction in recent years, mt researchers have proposed ap proaches to counter the data sparsity problem and to improve the performance of nmt systems in low resource scenarios, e.g. augmenting training data from source and/or target monolingual corpora @xcite , unsupervised learning strategies in the absence of labeled data @xcite , exploit ing training data involving other languages @xcite , multitask learning @xcite , selection of hyperparam eters @xcite , and pretrained language model finetuning @xcite . de spite some success, none of the existing benchmarks can be viewed as an overall solution as far as mt for lowresource languagepairs is concerned. for examples, the backtranslation strategy of @xcite is less effective in lowresource set tings where it is hard to train a good backtranslation model @xcite ; unsupervised mt does not work well for distant languages @xcite due to the difficulty of training unsuper vised crosslingual word embeddings for such lan guages @xcite and the same is ap plicable in the case of transfer learning too @xcite . hence, this line of research needs more attention from the mt research commu nity. in this context, we refer interested readers to some of the papers @xcite that compared pbsmt and nmt on a variety of usecases. as for lowresource scenar ios, as mentioned above, many studies (e.g. @xcite @xcite ) found that pbsmt can pro vide better translations than nmt, and many found the opposite results @xcite . hence, the find ings of this line of mt research have indeed yielded a mixed bag of results, leaving the way ahead unclear. in @xcite , we investigated the performance of pbsmt and nmt systems on two rarelytested underresourced languagepairs, englishtotamil and hinditotamil, taking a spe cialised data domain (software localisation) into ac count. in particular, in @xcite , we carried out a comprehensive manual error analysis on the translations produced by our pbsmt and nmt systems. this current work extends the work of @xcite in the following ways: (a) we present a social media platformbased human eval uation scheme for measuring the quality of transla tions generated by different @xcite we select the pbsmt and nmt systems of the english totamil translation task from @xcite and a commercial mt system, compare their perfor mances, and produce rankings of the three mt sys tems in terms of the length of the sentences to be translated using our proposed social media platform based human evaluation scheme. the remainder of the paper is organized as follows. section 2 explains the experimental setup including the descriptions on our mt systems and details of the data sets used. section 3 presents the results with discussions and analysis, while section 4 concludes our work with avenues for future work. ## conclusion in this paper, we investigated nmt and pbsmt in resourcepoor conditions. for this, we chose a spe cialised data domain (software localisation) for trans lation and a rarelytested morphologically divergent lowresource languagepair, englishtotamil. we studied translations in two setups, i.e. training data compiled from (i) freely available variety of data do mains (e.g. political news, wikipedia), and (ii) ex clusively software localisation data domains. in ad dition to an automatic evaluation, we randomly se lected one hundred sentences from the test set, and ranked our mt systems via a social media platform based human evaluation scheme. we also consid ered a commercial mt system, google translate, in this ranking task. we found that use of indomain data only at train ing has a positive impact on translation from english totamil. we looked at the translations produced by our mt systems and found that in many cases, the bleu scores underestimate the translation quality mainly due to relatively free word order in tamil. in this regard, both @xcite and @xcite note that bleu may be underreporting the difference in quality seen when using nmt systems, with the former attempting to measure the level of underreporting using a set of novel metrics. way (2018) reminds the mt community how important subjective evaluation is in mt and there is no easy replacement of that in mt evaluation. we refer the interested readers to @xcite who also drew at tention to this phenomenon. from our human ranking task we found that sentencelength could be a crucial factor for the per formance of the nmt systems in lowresource sce narios, i.e. nmt turns out to be bestperforming for very short sentences (number of @xmath0. this finding indeed does not correlate with the find ings of our automatic evaluation process, where pb smt is found to be the bestperforming, and gt and nmt are comparable. this finding could be inter est to translation service providers who use mt in their production for lowresource languages and may exploit the mt models based on the length of the source sentences to be translated. gt becomes the winner followed by pbsmt and nmt for the sentences of other lengths (number of @xmath1, and pbsmt be comes the winner followed by nmt and gt for the sentences of other lengths (number of @xmath2. overall, the human evaluators ranked gt as the first choice, pbsmt as the second choice and nmt as the third choice mt systems in the mixed setup. as for the it setup, pbsmt was the first choice, nmt was the second choice and gt was the third choice mt systems. we believe that the findings of this work pro vide significant contributions to this line of mt re search. in future, we intend to consider more lan guages from different language families. we also plan to include stringbased mt evaluation metrics such as chrf @xcite in our investigation, which have been shown to better reflect the actual performance improvement of nmt.