A human evaluation of E nglish- I rish statistical and neural machine translation (2020)

Abstract: With official status in both Ireland and the EU, there is a need for high-quality English-Irish (EN-GA) machine translation (MT) systems which are suitable for use in a professional translation environment. While we have seen recent research on improving both statistical MT and neural MT for the EN-GA pair, the results of such systems have always been reported using automatic evaluation metrics. This paper provides the first human evaluation study of EN-GA MT using professional translators and in-domain (public administration) data for a more accurate depiction of the translation quality available via MT.

## introduction the irish language enjoys the status of both the first official language of ireland and an official european union language. as a result of this status is there is a requirement for official public content to be made available in irish in both ireland 1 and the eu. 2 there is currently a derogation on the amount of irish content published by the eu, due to be lifted at the end of 2021 (publications office of the european union, 2011). at this point, the already high demand for professional irish translators will increase significantly. with this demand for the production of irish-language text, usually c 2020 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. 1 the official languages @xcite requires all official public information and services to be available in both irish and english: http://www.irishstatutebook.ie/ eli/2003/act/32/enacted/en/html 2 irish has been a full eu language since 2006. with english as the source language, it is important that any available @xmath0. despite mt having been established as a useful tool in the workflow of a professional translator, it is not yet the norm for irish translators, whether freelance or within a translation company. 3 as a lesser-resourced and minority language, irish faces a barrier to state-of-the-art technology shown to be effective for majority languages @xcite . while there has been research on improving @xmath1. this study aims to provide the first @xmath2, investigating the measurable usefulness of @xmath3. in an attempt to closely match the context in which @xmath4, professional translators will undertake post-editing (pe) tasks using mt output. another aim of this study is to provide a humanderived comparison of @xmath5. in previous work, a preliminary comparison of @xmath6. more recent publications @xcite show a more positive picture for @xmath7, but without a direct comparison to smt. the smt/nmt comparison presented in this paper will take into account both the quantitative metadata gathered during the study (time per seg-ment, number of keystrokes, etc.) as well as the qualitative opinions and recommendations of the participants. this paper is presented as follows: section 2 describes related work in the areas of @xmath8, he, etc. in section 3 we provide details of the data and parameters used in our smt and nmt systems, while section 4 describes the methodology used in this he study. we present our results in section 5 and provide some conclusions and avenues for future work in section 6. ## related work as irish is a poorly-resourced language @xcite , the quality of mt output has struggled to reach the same level of quality as well-supported languages. several studies @xcite @xcite thus focused on improving @xmath9, as discussed in section 1. in previous work comparing en-ga smt and nmt, preliminary results suggest that smt seems to outperform nmt @xcite , although we show a number of examples where the score may be misleading and recommend that a he study may be necessary to fully understand the quality of each system type. more recent studies @xcite show the effects of adding artificiallycreated training data to en-ga nmt. in terms of irish translators' attitudes to mt, moorkens' (2020) extensive survey reports varying attitudes between translators based on terms of employment, with freelance translators appearing to be poorly disposed towards mt. @xcite include lowresource languages as one of the main challenges still present in mt research. unable to exploit cutting-edge techniques that require huge resources, mt researchers must look to creative solutions to improve low-resource mt. such approaches include the creation of artificial parallel data, e.g. through back-translation @xcite , exploiting out-of-domain data (c.f. @xcite ) and using a betterresourced language as a pivot @xcite @xcite . he is a vital component of mt research @xcite , with many of the major mt conferences including a translator track to encourage such publications. they are especially valuable in low-resource or minority contexts (e.g. spanish-galician mt @xcite , russian-japanese mt @xcite ) where the languages may be overlooked by global mt companies. there have been comparisons of smt and nmt since nmt first emerged in the field. the conference on machine translation (wmt) regularly feature both systems, with he at the forefront @xcite @xcite . @xcite describe an extensive comparison of smt and nmt using both automatic metrics and he. mixed results overall highlight the need for language-specific he studies. recently, @xcite published a set of recommendations for performing he of mt. they advocate for (1) the use of professional translators over novices, (2) translations to be evaluated on a document-level, (3) fluency to be evaluated in addition to adequacy, (4) reference translations not to be heavily edited for fluency and (5) the use of original source texts (rather than translated text as input). we take these recommendations into account when designing this he study. ## mt systems set-up to compare smt and nmt through he it is first necessary to train a system of each type using the same training data. this section describes the data used in building both mt systems, their specific parameters and the automatic evaluation scores generated for each. ## study methodology and set-up mt he can take many forms, e.g. ranking of mt output, annotation of incorrect parts of speech or post-editing of mt output. a he study can also be carried out by providing the translators with mt output and asking them to post-edit it. one benefit of this method is that subjectivity can be decreased -data gathered through translator postediting (e.g. time spent per segment, number of keystrokes, etc.) is used to assess the mt system rather than the participant being required to give a judgement per word/segment. it is also faster than error annotation and requires less training, particularly if the translators already have experience of post-editing mt output. it is also the method which is closest to the situation in which mt is intended to be used, and as a result translator opinions of the post-editing tasks can also be elicited. for these reasons, we see post-editing as the he method that best suits the needs and intended outputs of this study. this section describes the set-up and methodology of the pe task and related survey. ## results and analysis in this section we present the survey results and the results gathered via the pet output. ## conclusions and future work we have presented the first he study for en-ga smt and nmt. we have shown that, while auto- source to ensure.. in the foi legislation.. smt: chun a chinntiú.. sa reachtaíocht um shaoráil faisnéise.. to ensure.. in legislation surrounding the freedom of information.. p1: cinntiú.. sa reachtaíocht um shaoráil faisnéise. ensure.. in the legislation surrounding the freedom of information.. p3: chun a chinntiú.. sa reachtaíocht sf. to ensure.. in the foi legislation.. nmt: féachaint.. sa reachtaíocht um shaoráil faisnéise.. see.. in the legislation surrounding the freedom of information.. p2: a fheacháint.. i reachtaíocht um shaoráil faisnéise.. to see.. in legislation surrounding the freedom of information.. p4: féachaint.. sa reachtaíocht um shaoráil faisnéise.. see.. in the legislation surrounding the freedom of information.. within this study, we have observed hter as the metric which most closely matches our participants' survey responses. however, it is important to note that with this study being limited to four 13 three of the four translators chose the nmt system as the most accurate output in the post-task survey, see in terms of future work, we propose a similar study with more participants. we have seen that translators vary in mt pe approaches, experience and opinion. accordingly, more participants would provide us with a more accurate picture of @xmath10. we also suggest a more fine-grained evaluation of @xmath11. in the survey portion of this study we elicit opinions of mt quality over 100sentence documents in general. in the future it may be beneficial to examine specific differences between en-ga smt and nmt at the sentencelevel, examining variations in errors in case, semantics, tense, etc. on a final note, it is worth considering that with the derogation of @xmath12, there is an urgent requirement for irish language translation with too few translators available to satisfy the demand as well as a lack of irish language resources. this means that we have a greater need than ever for en-ga mt systems designed with the end-user in mind.