Evaluating the usefulness of neural machine translation for the P olish translators in the E uropean Commission (2020)

Abstract: The mission of the Directorate General for Translation (DGT) is to provide high-quality translation to help the European Commission communicate with EU citizens. To this end DGT employs almost 2000 translators from all EU official languages. But while the demand for translation has been continuously growing, following a global trend, the number of translators has decreased. To cope with the demand, DGT extensively uses a CAT environment encompassing translation memories, terminology databases and recently also machine translation. This paper examines the benefits and risks of using neural machine translation to augment the productivity of in‒house DGT translators for the English‒Polish language pair. Based on the analysis of a sample of NMT‒translated texts and on the observations of the working practices of Polish translators it is concluded that the possible productivity gain is still modest, while the risks to quality are quite substantial.

## introduction machine translation arrived at the european commission in 1976 with the purchase of systran, a rule-based technology, for the english-french language pair. this initial version was then developed and adapted as ec systran to respond to the commission's needs; specialized terminology and dictionaries were imported, and, over time, other language pairs were added. by © 2020 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. 2010, some 2 million pages were translated per year with systran, used both by eu and member states' officials to provide quick drafts of texts in unfamiliar languages, and by eu translators. however, while the quality was fine for getting the gist of short, repetitive texts with standardized structure and terminology, the system was rather unsuitable for translating legislation, and so its use among translators never really caught up @xcite . ec systran was discontinued in december 2010 and since then the commission has been working on its own machine translation system, developed by the directorate-general for translation. mt@ec, based on moses, an opensource statistical machine translation (smt) toolkit, and improved by rule-based pre-and post-processing, went operational in june 2013. by 2017 it offered 78 direct language pairs, covering all eu official languages @xcite . it proved to be quite helpful for certain language pairs (english-french or english-portuguese, for example) and quite unusable for other (like english-hungarian or english-finnish). polish, with its free word order, rich inflectional morphology and complex orthography, was also quite challenging for the system, which produced very mixed results, from acceptable translations to unintelligible nonsense. as a result, few polish translators tried to use it as a resource in the translation process. an internal evaluation performed in the polish language department of dgt in 2017 revealed that smt was useful only for certain text types and that only half of the smt output was suitable for post-editing and thus likely to bring some productivity gains. moreover, the post-editing speed depended strongly on the typing speed of the translator in question: translators with good typing skills benefited much less from using smt than those with poorer skills foot_0 . in november 2017 the directorate-general for translation launched etranslation, a neural machine translation system, as part of the connecting europe facility. the aim of etranslation is not only to deliver raw machine translation to the public administration or to interested smes in eu member states, but also to provide mt as a tool for translators in eu institutions, to be embedded within their cat workflow. with the introduction of etranslation the question arose as to whether nmt actually provides better results than smt as far as productivity and quality of translation is concerned. based on the literature on the subject (e.g. @xcite , one would expect for example less lexical, inflectional morphology and word order errors in the nmt output when compared to smt, and overall less editing effort, measured by automatic metrics such as bleu and ter. however, studies on the performance of machine translation involving the polish language are very limited (e.g. @xcite , and since the specific types of errors are dependent on the particular language pair involved and are influenced by the morphosyntactic features of the target language, a simple extrapolation of the results obtained with one language pair to another language pair is not possible. therefore in the polish language department of dgt it was decided to conduct an evaluation on the benefits and risks of using nmt produced by etranslation as a translation aid, beside translation memory, for the english-polish language pair, concentrating in particular on the post-editing efficiency and the risks to quality. the paper is organized as follows. section 2 provides a description of the data and methods used in the evaluation. section 3 reports on the results and in section 4 the outcomes are discussed and final observations are offered. ## data and methods dgt's aim is to provide high-quality translations that are fit for publication. to this end translators have at their disposal a number of tools, most notably translation memories (euramis) and a terminology database (iate), integrated in a cat environment. machine translation is provided during pre-processing and can be included as a resource to complement the translation memory. hence, machine translation is presented for editing only when no tm match is found. the threshold for tm matches is set at 75%. when opting to use mt, translators can choose whether they want to use it in the autosuggest mode only (which is a feature that can speed up typing by presenting words and phrases from the mt translation memory after a few characters have been typed in an empty segment) or whether they want to have mt suggestions inserted in the segment every time no tm match is found. for the purpose of the present evaluation a group of 9 translators was recruited. they worked in their usual way, assisted by translation memory, but were instructed to always choose mt when downloading translation resources and to use it for all 'new' segments, i.e. segments that did not have a tm match, consistently and for all their translation assignments. they were also asked to put down any comments and opinions regarding the quality of the nmt output, including examples of mistakes, for each translation assignment. each of them translated between 1 and 13 texts. the texts varied in length from 1 page to over 150 pages and reflected well the text types and subject domains usually translated by dgt. the text types covered both legislative texts (commission regulation, commission decision, proposal for a council decision, proposal for a regulation of the european parliament and the council, proposal for a directive of the european parliament and the council, report from the commission, communication from the commission, impact assessment to a proposal for a regulation) and non-legislative texts (public consultation, report of an audit, notification of a concentration, list of phrases for a database, press release, description of a game for children, letter to a citizens, letter to the national authorities, text on the e-justice portal, text on the europa portal). the subject domains included: agriculture, climate, human health, maritime affairs, fisheries, internal market, industry, transport, competition, taxation, customs union, justice, trade, regional policy, banking, finance, external relations, internal affairs and migration. the test period lasted three months (july-september 2018). in total, during the test period the testing group translated 57 texts. 48 texts (22 legislative and 26 non-legislative) were used for further analysis (9 translations were discarded for reasons such as very few or no mt segments, shared project and problems with ter processing). also, for each raw mt segment (3178 mt segments in total) the ter score was calculated using the final translation as a reference. since it was not possible to record the postediting time automatically during translation, small-scale productivity tests on isolated sentences were conducted. six translators (out of the 9 participating in the evaluation) were asked to perform the test. a subsample of 12 sentence pairs was randomly selected from the texts translated during the test period with the aim to collect sentences with increasing ter scores to see whether post-editing speed depended on the quality of the nmt sentence as indicated by the ter score. the source sentences contained 29 words on average. translators were divided into two groups and each translator was asked to translate 6 sentences from scratch and to edit 6 different nmt-produced sentences. in this way each sentence from the subsample was translated or post-edited by three translators. they worked directly in a word document, but were asked to proceed in their usual way (consult terminology database, translation memory database etc.). the time needed for both activities was measured by the author of this evaluation with a stop watch separately for each sentence and then averaged for each sentence and for each translator. ## conclusions the initial driving force behind the development of machine translation back in the 1940s was the firm belief that high-quality fully automated translation is not only possible, but is a matter of a few years. after seven decades of research one needs to face the fact that when it comes to mt there is no one-size-fits-all solution. machine translation engines have to be customized to accommodate the desired terminology, style, domain and other requirements, including whether the mt translation is meant for publication and dissemination or rather for shortlived internal use. in other words, 'the degree of human involvement required (…) will depend on the purpose, value and shelf-life of the content' @xcite . the requirements placed on dgt translators, especially regarding the quality of the translation of legal acts, are even higher than the usual requirements on the translation market for texts meant for publication. this is because mistakes in legal texts impact not only on dgt's image, they also have legal consequences. beside accuracy, consistency within the text and with any related texts is of particular importance, e.g. terminological consistency with the acts in the same domain or lexical and terminological consistency with the basic legal act. hence the usefulness of machine translation must be evaluated in view of these particular requirements. for the polish language, neural machine translation usually produces rather well-formed sentences suitable for post-editing. hence, correcting the nmt output was not perceived to be very cumbersome by translators participating in the evaluation. on the other hand, on average only less than 20 % of nmt segments did not contain any errors; and most of the mistakes in the remaining segments were mistakes in accuracy or terminology, which poses serious challenges to the quality of the final translation. legal texts seem to benefit more from nmt than non-legal texts, probably because of their repetitive and standard character. in non-legal texts nmt suggestions often need extensive adaptation of style and register and therefore are in general perceived to be less useful. there seems to be only a weak correlation between the ter score and the post-editing time, although a bigger sample is necessary to corroborate this finding. the calculated productivity gain when nmt is used to complement tm matches is still modest. however, this finding needs to be confirmed with more data obtained under more controlled conditions. also, observations of the working practices of the polish translators at dgt point out to the possibility that there might be a stronger relationship between other variables and the post-editing speed, such as the experience of the translator in the subject domain and the number of terms and references or quotations in the sentence. this hypothesis, too, would require further testing.