Adapting Machine Translation Education to the Neural Era: A Case Study of MT Quality Assessment (2023)

Abstract: The use of automatic evaluation metrics to assess Machine Translation (MT) quality is well established in the translation industry. Whereas it is relatively easy to cover the word- and character-based metrics in an MT course, it is less obvious to integrate the newer neural metrics. In this paper we discuss how we introduced the topic of MT quality assessment in a course for translation students. We selected three English source texts, each having a different difficulty level and style, and let the students translate the texts into their L1 and reflect upon translation difficulty. Afterwards, the students were asked to assess MT quality for the same texts using different methods and to critically reflect upon obtained results. The students had access to the MATEO web interface, which contains word- and character-based metrics as well as neural metrics. The students used two different reference translations: their own translations and professional translations of the three texts. We not only synthesise the comments of the students, but also present the results of some cross-lingual analyses on nine different language pairs.

## introduction machine translation (mt) is increasingly being used in professional translation workflows. "mt literacy and awareness of mt's possibilities and limitations" forms therefore, according to the emt competence framework @xcite , an integral part of professional translation competences. at ghent university, we offer a 5-credit course machine translation and postediting, which is part of the one-year postgraduate programme computer-assisted language mediation foot_0 and the two-year european master in technology for translation and interpreting foot_1 . the mt part of the course aims to provide a comprehensive overview and covers topics such as the main linguistic challenges for mt, different approaches to mt (rule-based, statistical and neural mt) and mt evaluation. the students also acquire handson experience in building and evaluating their own mt systems using mutnmt 3 . the use of automatic evaluation metrics to assess machine translation (mt) quality is well established in the translation industry. the more traditional word-and character-based metrics are relatively easy to run and it is therefore easy to incorporate them in a university course. but, much more technical knowledge is required to get the neural metrics, which are based on large pretrained language models, up and running. despite their better performance, they are therefore less popular in translation courses. in this paper, we discuss how we introduced the topic of mt quality assessment in a course for translation students of varying language backgrounds. the students got one dedicated lecture on the subject of mt evaluation, which covers both manual and automatic evaluation methods and had to critically reflect both on the suitability of mt for three different texts and on the usefulness of automatic evalua-tion. they made use of an early version of mateo (machine translation evaluation online) @xcite 4 , an easy-to-use web interface for evaluating mt output by means of a variety of both word-and character-based and neural mt evaluation metrics. ## related research the term machine translation literacy has been introduced by @xcite and has since then been picked up by other scholars. o'brien and ehrensberger-dow used the term in the context of professional translation and described mt literacy as "knowing how mt works, how it can be useful in a particular context, and what the implications are of using mt for specific communicative needs" @xcite . its growing importance is of course related to the everincreasing quality of mt systems. several initiatives have been taken to develop and distribute publicly available learning materials tailored to teaching mt to translation students. in the framework of the multitrainmt erasmus+ project @xcite an open access course book has been published @xcite targeting both language learners and translators. the project also developed an open source pedagogically-oriented neural mt platform called mutnmt, in which students can go through the various stages of building an mt engine, from uploading parallel corpora, over training and evaluating an mt system and inspecting translations. another initiative is datalit mt (teaching data literacy in the context of machine translation literacy), in which, among others, a python notebook has been created to explain translationoriented natural language processing (nlp) concepts to translation students @xcite . with mateo, ghent university adds another didactic tool to the existing set. mateo differs from the aforementioned tools in the sense that it gives non-technical users access to both more traditional (word-and character-based) metrics and state-of-the-art neural automatic evaluation metrics via a web interface. ## data collection the students' project consisted of two parts. in the first part students were asked to manually trans-late three english texts into their l1. they were told that their translations would be used to assess mt quality. in addition, they had to reflect upon translation difficulty of the three texts. the second part dealt with mt evaluation. students were asked to evaluate the mt output of three different mt engines for the three texts using manual and automatic evaluation methods. for the manual evaluation, students ranked the three mt suggestions (from best to worst; equal rankings allowed) and provided accuracy foot_4 and fluency scores on a 5point scale for each mt sentence. accuracy scores relate to the amount of content and meaning of the source sentence that is retained in the mt output. fluency scores relate to the degree to which a sentence meets the standards and conventions of the target language. for the automatic evaluation the lecturers provided the students also with professional translations for the three texts. the students made use of an early version of the mateo web interface to obtain automatic scores for 6 different metrics (bleu, ter, chrf, bertscore, bleurt and comet, see section 3.1 for more details) using two different reference translations: their own translation and the professional translation. ma-teo contains easily accessible descriptions of the different metrics and students could look up more details by clicking on the links to the original research papers. the students were asked to compare perceived translation difficulty with obtained mt translation quality and critically reflected on different aspects of the mt evaluation task. the english source texts were taken from the lecontra data set @xcite , which contains dutch student translations of english source texts enriched with translation process data in the form of keystroke logging. we selected three texts (see the third text describes the discovery of the hidden laboratory used by leonardo da vinci and is predominantly denotative (t3). according to our selection criteria, t3 was the easiest text and t2 the most difficult text to translate from english into dutch. the dutch professional translations were retrieved from the dutch parallel corpus @xcite . the professional translations for the other languages were obtained via a translation agency; the cost was 15 cents per word supplemented by 21% vat. students were asked to compare the mt quality of three different mt systems. they were given the output of facebook's multilingual translation model m2m100 1.2b @xcite and had to create the mt output with google translate and a third mt system of their choice. in total, the students worked with 9 different target languages (dutch, french, brazilian portuguese, romanian, turkish, farsi, kazakh, ukrainian and russian). all students had either received formal translation training in a prior training programme or had gained work experience as a translator. for the remainder of this paper we only retained the submission of the most experienced student per target language. informed consent was obtained for all student submissions included in this study. google translate was chosen as a state-of-theart system that covers many languages, including low resource ones. m2m100 also supports many of these languages but unlike google's service, it is an open-source model. our study therefore also sheds some light on the performance of open models compared to closed ones. it should be noted that we did not use the largest available m2m model (12b parameters) but instead opted for a computationally more feasible variant (1.2b parameters). students were free to choose a third system themselves based on their own preference and the target language that they worked on. most students used deepl as third mt system. bing was to used for farsi and lingvanex was used for kazakh. in section 4 we not only synthesize the comments of the students, but also present the results of some cross-lingual analyses as the obtained data set allows us to examine the automatic evaluation metrics across typologically different languages. ## conclusion machine translation is taking an increasingly prominent place in professional workflows and so is the assessment of mt quality. mt evaluation methods, both human as well as automatic, thus deserve sufficient attention in mt courses targeting translation students. whereas research demonstrates that the newer neural automatic evaluation metrics correlate better with human judgements than the more traditional word-and characterbased metrics, the neural metrics are not often used in translation courses as quite some technical skills are required to get them up and running this paper focused on mt evaluation and how it can be taught to translation students. via the mateo web interface students had access to six different automatic metrics: two word-based, one character-based and three neural metrics. students translated three english source texts from scratch into their l1 and assessed mt quality afterwards using manual methods and automatic evaluation metrics. they were asked to critically reflect upon obtained results. perceived difficulty and mt quality did not always align, which seems to suggest that translation students and machine translation systems face different problems during translation. many of the comments that the students made were afterwards confirmed in the cross-lingual analyses. according to the students' comments google translate and deepl delivered better translations than facebook's m2m100 1.2b, with differences in quality between google translate and deepl varying across language pairs. within the word/character-based metrics, chrf was found to be the better metric, especially for highly inflected languages. overall, the neural metrics were perceived to be more comparable to human evaluation, a statement that was partially confirmed in the cross-lingual analyses, in which bleurt came out as best metric, but in which chrf also correlated well. automatic metrics were considered to be useful for mt quality assessment, but only for text-level evaluations. it is important to note that all metrics work on different scales, and that scores are therefore not comparable across languages, which makes it difficult to compare results. different analyses showed that, in general, the obtained automatic scores were higher when the student translations were used as reference translations. students tended to stay closer to the source text, whereas professional translators deviated more from the source text. as we worked with students' data, our data set was limited and is too small to make firm conclusions, but it seems worthwhile to further explore the impact of the origin of the reference translations on translation quality assessment. the data set (source texts, reference translations and mt output) is freely available on github foot_6 .