An Error-based Investigation of Statistical and Neural Machine Translation Performance on H indi-to- T amil and E nglish-to- T amil (2020)

Abstract: Statistical machine translation (SMT) was the state-of-the-art in machine translation (MT) research for more than two decades, but has since been superseded by neural MT (NMT). Despite producing state-of-the-art results in many translation tasks, neural models underperform in resource-poor scenarios. Despite some success, none of the present-day benchmarks that have tried to overcome this problem can be regarded as a universal solution to the problem of translation of many low-resource languages. In this work, we investigate the performance of phrase-based SMT (PB-SMT) and NMT on two rarely-tested low-resource language-pairs, English-to-Tamil and Hindi-to-Tamil, taking a specialised data domain (software localisation) into consideration. This paper demonstrates our findings including the identification of several issues of the current neural approaches to low-resource domain-specific text translation.

## introduction in recent years, mt researchers have proposed approaches to counter the data sparsity problem and to improve the performance of nmt systems in lowresource scenarios, e.g. augmenting training data from source and/or target monolingual corpora @xcite , unsupervised learning strategies in the absence of labeled data @xcite , exploiting training data involving other languages @xcite , multi-task learning @xcite , selection of hyperparameters @xcite , and pre-trained language model fine-tuning @xcite . despite some success, none of the existing benchmarks can be viewed as an overall solution as far as mt for low-resource language-pairs is concerned. for examples, the back-translation strategy of @xcite is less effective in low-resource settings where it is hard to train a good back-translation model @xcite ; unsupervised mt does not work well for distant languages @xcite due to the difficulty of training unsupervised cross-lingual word embeddings for such languages @xcite and the same is applicable in the case of transfer learning too @xcite . to this end, we investigate the performance of pb-smt and nmt systems on two rarely-tested under-resourced language-pairs, english-to-tamil and hindi-to-tamil, taking a specialised data domain (software localisation) into account. in this context, in @xcite , we investigated the performance of pb-smt, nmt and a commercial mt system (google translate (gt)) foot_0 on english-to-tamil taking the software localisation data into account, i.e. the same data as the one used in this work. in particular, in @xcite , we produced rankings of the mt systems (pb-smt, nmt and gt) via a social media platform-based human evaluation scheme, and demonstrate our findings in this lowresource domain-specific text translation task. the next section talks about some of the papers that compared pb-smt and nmt on a variety of use-cases. the remainder of the paper is organized as follows. in section 2, we discuss related work. section 3 explains the experimental setup including the descriptions of our mt systems and details of the data sets used. section 4 presents the results with discussions and analysis, while section 5 concludes our work with avenues for future work. ## related work the advent of nmt in mt research has led researchers to investigate how nmt is better (or worse) than pb-smt. this section presents some of the papers that compared pb-smt and nmt on a variety of use-cases. although our primary objective of this work is to study translations of the mt systems (pb-smt and nmt) in under-resourced conditions, we provide a brief overview on some of the papers that compared pb-smt and nmt on highresource settings too. @xcite compare pb-smt and nmt on a range of translation-pairs and show that for all translation directions nmt is either on par with or surpasses pb-smt. @xcite analyse the output of mt systems in an englishto-german translation task by considering different linguistic categories. @xcite conduct an evaluation to compare nmt and pb-smt outputs across broader aspects (e.g. fluency, reordering) for 9 language directions. @xcite conduct an extensive qualitative and quantitative comparative evaluation of pb-smt and nmt using automatic metrics and professional translators. @xcite carries out an extensive comparison between nmt and pb-smt languagerelated issues for the german-english language pair in both translation directions. these works @xcite @xcite show that nmt provides better translation quality than the previous state-of-the-art pb-smt. this trend continues in other studies and use-cases: translation of literary text @xcite , mt post-editing setups @xcite , industrial setups @xcite , translation of patent documents @xcite , less-explored language pairs @xcite , highly investigated "easy" translation pairs @xcite , and translation of catalogues of technical tools @xcite . an opposite picture is also seen in the case of translation of the domain text; @xcite showed pb-smt outperforms nmt when translating user-generated content. the mt researchers have tested and compared pb-smt and nmt in the resource-poor settings too. @xcite , @xcite , and @xcite found that pb-smt can provide better translations than nmt in low-resource scenarios. in contrast to these findings, however, many studies have demonstrated that nmt is better than pb-smt in low-resource situations @xcite . hence, the findings of this line of mt research have yielded indeed a mixed bag of results, where way ahead unclear. this work investigates translations of a software localisation text with two low-resource translation-pairs, hindi-to-tamil and english-to-tamil, taking two mt paradigms, pb-smt and nmt, into account. ## conclusion in this paper, we investigated nmt and pb-smt in resource-poor scenarios, choosing a specialised data domain (software localisation) for translation and two rarely-tested morphologically divergent language-pairs, hindi-to-tamil and english-to-tamil. we studied translations on two setups, i.e. training data compiled from (i) freely available variety of data domains (e.g. political news, wikipedia), and (ii) exclusively software localisation data domains. in addition to an automatic evaluation, we carried out a manual error analysis on the translations produced by our mt systems. use of in-domain data only at training has a positive impact on translation from a less inflected language to a highly inflected language, i.e. englishto-tamil. however, it does not impact the hindi-to-tamil translation. we conjecture that the morphological complexity of the source and target languages (hindi and tamil) involved in translation could be one of the reasons why the mt systems performed reasonably poorly even when they were exclusively trained on specialised domain data. we looked at the translations produced by our mt systems and found that in many cases, the bleu scores underestimate the translation quality mainly due to relatively free word order in tamil. in this context, @xcite computed the degree of underestimation in quality of three most-widely used automatic mt evaluation metrics: bleu, meteor @xcite and ter @xcite , showing that for nmt, this may be up to 50%. we refer the interested readers to @xcite who also drew attention to this phenomenon. our error analysis on the translations by the english-to-tamil and hindi-to-tamil mt systems reveals many positive and negative sides of the two paradigms: pb-smt and nmt: (i) nmt makes many mistakes when translating domain terms, and fails poorly when translating oov terms, (ii) nmt often makes incorrect lexical selections for polysemous words and omits words and domain terms in translation, and occasionally commit reordering errors, and (iii) translations produced by the nmt systems occasionally contain repetitions of other translated words, strange translations and one or more unexpected words that have no connection with the source sentence. we observed that whenever the nmt system encounters a source sentence containing oovs, it tends to produce one or more unexpected words or repetitions of other translated words. as for smt, unlike nmt, the mt systems usually do not make such mistakes, i.e. repetitions, strange, spurious or unexpected words in translation. we observed that the bpe-based segmentation can completely change the underlying semantic agreements of the source and target sentences of the languages with greater morphological complexity. this could be one of the reasons why the hindito-tamil nmt system's translation quality is poor when the system is trained on the sub-word-level training data in comparison to one that was trained on the word-level training data. we believe that the findings of this work provide significant contributions to this line of mt research. in future, we intend to consider more languages from different language families. we also plan to judge errors in translations using the multidimensional quality metrics error annotation framework @xcite which is a widely-used standard translation quality assessment toolkit in the translation industry and in mt research. the mt evaluation metrics such as chrf @xcite which operates at the character level and comet @xcite which achieved new state-of-the-art performance on the wmt 2019 metrics shared task @xcite obtained high levels of correlation with human judgements. we intend to consider these metrics (chrf and comet) in our future investigation. as in @xcite who examined terminology translation in nmt in an industrial setup while using the terminology integration approaches presented in @xcite , we intend to investigate terminology translation in nmt using the mt models of @xcite on english-to-tamil and hindi-to-tamil.