The Post-Edit Me! project (2023)

Abstract: In this paper, we present the Post-Edit Me! project, which aims to support machine translation post-editing training and learning in translator education, with particular emphasis on quality evaluation of students’ productions. We describe the main components of the project, from the perspectives of both translation lecturers and translation students, and the project’s outcomes to date, namely the MTPEAS annotation system used to assess students’ post-edited texts and the postedit.me app we are currently developing to automate the evaluation workflow.

## introduction there have been calls in academia for the integration of machine translation (mt) post-editing (pe) specialized training into translation curricula, together with concrete, fully fledged pedagogical proposals, mostly in the form of stand-alone technology modules dedicated to mt and pe (e.g. @xcite . however, as rightly argued by @xcite and @xcite , curriculum-wide initiatives are needed to fully support the acquisition of pe skills by translation students. to achieve this, several hurdles will have to be overcome, such as the training of translation trainers not yet familiar with mt and pe @xcite and quality evaluation of the post-edited texts produced by students, as the latter need to benefit from structured feedback in order to acquire solid pe skills. the post-edit me! project (pem) is funded for a twoyear period (2021-2023) by uclouvain's fonds de développement pédagogique (a competitive fund that offers financial support to pedagogical projects promoting innovation in university teaching). the main goal of pem is to support pe training in the master's programme in translation offered by the louvain school of translation and interpreting (lsti). more precisely, pem aims to (i) help lecturers to become familiar with pe, devise pe tasks and assess the quality of student's productions (understood as fitness for purpose) and, by doing so, (ii) boost students' pe skills through practice, especially as regards mt error detection and correction. sections 2 and 3 describe the project's objectives and (partial) results from the perspective of translation lecturers and students, respectively. ## translation trainers one of the central objectives of pem is to develop innovative pe training practices. to achieve this goal, various initiatives have been set up to support translation lecturers at the lsti, focusing on two main dimensions: (i) training lecturers in pe and perelated pedagogical practices and (ii) developing a standardized system, fully integrated in an app, for the annotation and assessment of students' postedited texts. the training dimension includes a series of conferences about pe and its place in translator education, featuring experts from academia and the translation industry. in addition to these conferences, which mostly took place during the first year of the project, regular team meetings are now being held to familiarize lecturers with the newly developed pedagogical resources (see below) and to promote sharing of good practices. for instance, we have organized a shared task on pe annotation, where lecturers were asked to annotate the same data and discuss their annotations. we also offer on-demand individual coaching sessions designed to guide lecturers in planning pe tasks for their courses (selection of source texts and mt engine, pe instructions, evaluation of students' post-edited texts). to date, lecturers have benefited from the pem team's support in the context of various translation courses (economic and financial, legal, international affairs, marketing and scientific/technical translation), in six language pairs (dutch, english, german, italian, spanish and russian to french). the second dimension of teacher training is the development of new pedagogical resources for the assessment of post-edited texts: the mtpeas annotation system (machine translation post-editing annotation system) and the postedit.me app. the mtpeas annotation system was devised for pedagogical purposes. one of its guiding principles is that it should be user-friendly for lecturers and students alike. it includes a decision tree to facilitate its use by lecturers and contains seven categories described in transparent terms: value-adding edit, successful edit, unnecessary edit, incomplete edit, error-introducing edit, unsuccessful edit and missing edit. these categories are defined and illustrated with examples taken from several language pairs in a manual available in english and french as an open educational resource (oer uclouvain; lefer et al 2022). in order to offer finer-grained feedback to students, the mtpeas categories used to tag erroneous segments in the final pe can be combined with tags taken from the translation-oriented annotation system (tas) taxonomy @xcite . these tags make it possible to identify the exact nature of errors in the final post-edited products; they cover mechanics, grammar and syntax, lexis and terminology, discourse and pragmatics, register and style, content, culture and brief. the postedit.me app, which consists of a teacher interface and a student interface, makes it possible to automate the whole workflow, from source-text selection to the correction of students' post-edited texts and sharing of feedback. the tool's annotation interface provides, inter alia, metrics such as ter (translation edit rate) and expansion rates (here, the increase/decrease in text length from source to mt and from mt to pe), and an automatic grade based on lecturers' annotations of students' post-edited texts. at the technical level, the app relies on several open-source libraries, many of which have only reached technical maturity in the past few years. the django framework is central to the app. its objectrelational mapper functionality is especially useful for an app of this type as it can store entries with extensive metadata and facilitate calculation of various statistics and metrics on the basis of those data without being bound to a specific database language. another key feature of the appannotation of the machine translation and the postedited text-is enabled by the label studio library. to generate part-of-speech tags and lemmas, the app uses the spacy library with the pre-trained language models that the search feature (concordancer) leverages. the long-term plan for the postedit.me app is to publish it under an open-source licence. ## translation students the pem project also aims to benefit translation students. since the start of the project in september 2021, under the guidance of the project's pedagogical assistant, lecturers have gradually started to integrate post-editing tasks into their domain-and languagepair-specific translation courses. this means that students are now being offered numerous opportunities to practice pe across different language pairs and domains, and to benefit from clear, relevant, detailed and fair feedback thanks to the lecturers' reliance on the mtpeas standardized taxonomy and the postedit.me app. once lecturers have annotated students' productions in the teacher interface, students can access their lecturers' feedback in the student interface (i.e. the error-annotated version of the mt, the annotated version of their pe, and some general feedback). the app also allows students to keep track of their progress using the statistics component (e.g. most frequent types of pe errors across tasks, domains and language pairs). we also aim to encourage students to practice mt error detection and correction using sentence-level exercises generated by lecturers on the basis of the data collected cumulatively within the project.