Multidimensional assessment of the e T ranslation output for E nglish– S lovene (2020)

Abstract: The Slovene language department of the European Commission Directorate-General for Translation has always been an early adopter of new developments in the area of machine translation. In 2018, the department started using neural machine translation produced by the eTranslation in-house engines. In 2019, a multidimensional assessment of the eTranslation output for the language combination English–Slovene was carried out. It was based on two user satisfaction surveys, an analysis of detected and reported errors and an ex post analysis of a sample. As part of the assessment effort, a categorisation of errors was devised in order to raise awareness among translators of the potential pitfalls of neural machine translation.

## conclusion and follow-up the reported errors revealed a variety of problems in the nmt output (polysemic misinterpretations, complete semantic blunders, terminological mistakes and inconsistencies, neural neolo-gisms, omissions and additions; problems with proper nouns and abbreviations, complex structures and text fragments, text containing spelling errors and with (and/or next to) punctuation). the ex post analysis confirmed that nmt output cannot be used as is. none of the sampled documents when simply pre-translated with the nmt output is a fit-for-purpose translation (fit for publication). a contributing factor for the poor result might also have been the fact that the sampled documents had a low match rate. how much if any time is saved by machine translation remains unknown. 13 furthermore, we gathered no definitive data to support the claim that the use of nmt saves time in comparison with the use of smt. 14 even though translators prefer working with nmt to working with smt, we cannot claim that nmt use is necessarily easier and that it accelerates the translation process when compared to smt. maučec and donaj (2020) identify three levels of post-editing effort: temporal, cognitive and technical. the use of nmt involves different types of issues that need to be dealt with and might be considered more mentally taxing than those inherent in the use of smt (problems at higher levels of grammar and beyond grammar). clearly a skilled human is needed to guarantee a high quality of translation. this means nmt as a tool needs to be understood better in order to be used better, which requires educating and training its users, which is in line with @xcite , who emphasize the need of research on and teaching of skills specific to postediting. therefore, we extended the list of categorised reported errors with additional examples and added to the list new categories of recurring problems discovered during the ex post analysis. the list now paints as comprehensive a picture as possible of all potential english-slovene nmt pitfalls. a department-level training was held to familiarise translators with the list and with the 13 at least for the users of the en-sl etranslation nmt output. @xcite have worked with the french and finnish lds of dgt to assess how much time translators gain (or lose) in real-world conditions when they use machine translation, and observed the average speed gain of 14 % for english-finnish nmt (and 12 % for english-french phrase-based smt). 14 in their analyses, @xcite , and @xcite , do arrive to the conclusion that nmt decreases post-editing effort compared to smt. @xcite also arrive to the same conclusion, and for a slavic language close to slovene. results of the ex post analysis. the training raised awareness of expected error types among translators and informed our reflection on how to best use nmt. the request for translators to report examples remains openthe reporting is not systematic and the analysed sample was small, so it is possible that some errors elude categorisation due to a low number of occurrences and that some error types have not been detected yet.