Machine Translation Quality: A comparative evaluation of SMT , NMT and tailored- NMT outputs (2020)

Abstract: The present study aims to compare three systems: a generic statistical machine translation (SMT), a generic neural machine translation (NMT) and a tailored-NMT system focusing on the English to Greek language pair. The comparison is carried out following a mixed-methods approach, i.e. automatic metrics, as well as side-by-side ranking, adequacy and fluency rating, measurement of actual post editing (PE) effort and human error analysis performed by 16 postgraduate Translation students. The findings reveal a higher score for both the generic NMT and the tailored-NMT outputs as regards automatic metrics and human evaluation metrics, with the tailored-NMT output faring even better than the generic NMT output.

## introduction latest technological advances in machine translation (mt) have led to a wider availability of mt systems for various language pairs and neural machine translation (nmt) has been widely hailed as a significant development in the improvement of the quality of mt, given that nmt models have been proven to consistently outperform statistical machine translation (smt) models in shared tasks, as well as in various project outcomes @xcite @xcite @xcite @xcite . mt has been moved "from the peripheries of the translation field closer to the centre" @xcite and has been integrated in the translation workflow, by using machine translated text as a raw translation to be further post-edited by a translator @xcite . the differences between various mt systems, as regards the quality of their output and the types of errors included therein, have been reported by several recent studies. some @xcite @xcite relied on automatic evaluation metrics (aems) like bleu @xcite and hter @xcite ; others used human evaluations of the mt output quality, employing adequacy and fluency ratings @xcite , manual error analyses @xcite @xcite or a combination of methods @xcite @xcite @xcite @xcite @xcite . drawing on these studies, the present study aims to compare three systems: a generic smt, a generic nmt and a tailored-nmt system, namely a factored or custom-trained nmt system, focusing on the english to greek language pair. the comparison is performed following a mixedmethods approach, i.e. automatic metrics, as well as side-by-side ranking, adequacy and fluency rating, measurement of actual post-editing (pe) effort and human error analysis. tο the best of our knowledge there are no studies to date for the english to greek language pair which compare generic and custom-trained mt systems, while there are only a few related studies to date comparing smt and nmt systems @xcite 2 methodology a mixed-methods approach was adopted in the present study with a view to producing reliable results. aems and human evaluation metrics, including eye-tracking and keystroke logging data for measuring the effort expended by translators while carrying out full pe of the mt output generated by three different systems (google translate smt system, google translate nmt system, tailored-nmt system), side-by-side ranking of the mt outputs, adequacy and fluency rating and human error classification were used to evaluate the quality of the mt output of these three mt systems and investigate their differences. a series of experiments were carried out during the 2018-2019 spring semester at the department of foreign languages, translation and interpreting of the ionian university. twenty greek students enrolled on the ma in the science of translation initially participated in this study. however, only sixteen completed all tasks, since the participation in the tasks was optional. all participants signed a consent form, while all stored data were fully anonymised in accordance with greek law 2472/97 (as amended by laws 3783/2009, 3917/2011 and 4070/2012). ## discussion and conclusion this paper reports on a comparative evaluation of generic smt, generic nmt and tailored-nmt outputs for the english to greek language pair using aems and human evaluation metrics, including eye-tracking and keystroke logging data, side-by-side ranking of the mt outputs, adequacy and fluency rating and error classification. as regards the differences between the smt and nmt outputs, this study shows that the nmt systems produce translations of higher quality and thus corroborates the findings of previous studies on various language pairs @xcite @xcite @xcite , including the english-greek language pair @xcite @xcite . in particular, the analysis reveals a higher score for both the generic nmt and the tailored-nmt outputs as regards automatic metrics and human evaluation metrics, with the tailored-nmt output faring even better than the generic nmt output. in addition, the tailored-nmt output was ranked as the best and was rated higher for both adequacy and fluency, a fact which explains the reduced temporal, technical and cognitive effort expended during its pe. the decrease in pe effort can also be explained by the lowest number of errors found in the tailored-nmt output. another interesting finding is that all the mt outputs contain more errors at the level of fluency than at the level of adequacy with the most typical fluency errors being grammatical errors. both nmt outputs contain fewer grammatical errors than the smt output, confirming thus the findings of previous studies for improved quality of the nmt systems at the level of fluency -not only for the english-greek language pair @xcite @xcite but also for other language-pairs, such as english-czech, english-german, english-finnish, english-romanian, english-russian, english-croatian and english-chinese @xcite @xcite . however, no difference between the generic nmt and the tailored-nmt outputs was reported. another interesting finding involves the category of punctuation where the tailored-nmt output fares worse than both the generic nmt and the generic smt output. finally, in terms of adequacy, the tailored-nmt output fares better with slightly fewer mistranslations, terminological errors and additions than the generic nmt and the generic smt outputs, although it includes more omissions than the former. the findings point to the fact that there are limits to generic mt models, as they are not tuned to provide translations that are unique to a specific genre and thus business or industry. although the development of a tailored-nmt system can be particularly compute intensive, and therefore too expensive and time-consumingespecially in cases where there are not enough parallel data to train a new good quality and appropriately adapted systemthe higher quality and the reduced cognitive, technical and temporal effort suggest that it is worth exploring further.