Error Annotation in Post-Editing Machine Translation: Investigating the Impact of Text-to-Speech Technology (2022)

Abstract: As post-editing of machine translation (PEMT) is becoming one of the most dominant services offered by the language services industry (LSI), efforts are being made to support provision of this service with additional technology. We present text-to-speech (T2S) as a potential attention-raising technology for post-editors. Our study was conducted with university students and included both PEMT and error annotation of a creative text with and without T2S. Focusing on the error annotation data, our analysis finds that participants under-annotated fewer MT errors in the T2S condition compared to the silent condition. At the same time, more over-annotation was recorded. Finally, annotation performance corresponds to participants’ attitudes towards using T2S.

## introduction with machine translation (mt) adoption and the provision of post-editing machine translation (pemt) services on the rise, translation process research (tpr) has been questioning whether the ways in which pemt is currently being carried out (in dedicated pemt tools, in simple word processing software, or in computer-assisted translation (cat) tools/translation environment tools (tent), and with or without the use of additional technologies) optimally support posteditors, both from a process-and a productoriented point of view @xcite . as the technological possibilities are growing, there is an uptake of speech tools such as automatic speech recognition (asr; speech-totext) by professional translators @xcite , and this practice has become one of the focal points of tpr @xcite @xcite @xcite . while the use of automatic speech synthesis (text-to-speech; t2s) has received comparatively little attention both from the language services industry (lsi) and the research community, translators and revisers are known to read aloud translations during (self-)revision @xcite @xcite . this intuitively perceived benefit of aurally processing a text points to the potential of t2s as an attentionraising technology that may also help post-editors identify subtle neural machine translation (nmt) errors. the practice of pemt remains a particular challenge for translation studies students, despite the transition from statistical mt (smt) to nmt which has reduced the absolute number of errors to be corrected in the raw mt output @xcite . moreover, the phenomena of over-and under-editing continue to preoccupy both academia and the lsi @xcite . we share the view that students need to be exposed to a variety of translation tools early and often, and we believe that introducing them to additional technologies such as t2s will prove beneficial for honing the skills needed to succeed as future post-editors. rather than segregating tools and technologies to separate courses, we support integrative tasks which combine error annotation using standardised typologies, pemt, and t2s as ideal opportunities to build confidence, competence, and speed when performing pemt. this paper describes the results of a small-scale study investigating the impact of t2s on pemt error annotation, alongside participant attitudes towards using t2s for pemt. to that end, we first present previous research on the use of speech tools for pemt, as well as the use of error typologies in the lsi and in translator training. this is followed by our research questions and methodology. in the last two sections we present the results of our study and discuss the implications of teaching pemt by introducing t2s and error annotation into the mix. ## previous research pemt has been identified as the service with the highest growth potential in the lsi @xcite . the widespread adoption of data-driven mt since the 2000s @xcite has brought considerable change to the industry, and professional translators are increasingly being asked to carry out pemt tasks. while claims of mt achieving near or full human parity in terms of translation quality @xcite should be taken with a grain of salt @xcite , mt has been shown to enable productivity and quality gains in translation tasks (e.g. guerberof arenas, 2014; sánchez-gijón, moorkens and way, 2019). however, despite mt quality improvements and the clear industry need for qualified posteditors (most recently embodied by the gala mtpe training special interest group foot_0 ), european translation studies programmes have been found to lack hands-on pemt training both at undergraduate and postgraduate levels @xcite . while interest in mt literacy is growing in the research community (cf. @xcite , many translators are still reluctant to embrace mt as a tool @xcite . limited knowledge and experience regarding mt use in universitytrained translators is likely to be a contributing factor to this reticence. in parallel to the lack of hands-on pemt training in translation studies syllabi, previous work has also highlighted a lack of familiarity of translation educators and students with translation quality assessment (tqa) practices @xcite quality management, which frequently involves tqa processes, has been identified as a key competence for professional translators @xcite , and in the context of mt, the ability to perform tqa in the form of error annotation with predefined typologies is a useful skill for engine evaluation and pemt research, among others @xcite . moreover, the active reflection on error types may help improve the current issue of overand under-editing, which is common in pemt @xcite . there is therefore a competence gap between academia and industry in relation to both pemt and tqa practices. interest in mt is growing in the lsi; however, it has been shown that translation tools do not optimally support post-editors, which leads to dissatisfaction among users @xcite . in parallel, dictating with automatic speech recognition (asr) tools instead of, or in addition to, typing has been recognised as an alternative, more ergonomic working mode, and has attracted the interest of several scholars @xcite @xcite @xcite . asr is also seeing an uptake among professional translators @xcite . consequently, new applications offering multi-modal forms of translator-computer interaction (tci) have been developed @xcite . in these examples, multimodal features include the use of asr for translation and pemt. in interpreting studies, the integration of asr into computer-aided interpreting tools is also being investigated to support the work of interpreters @xcite . comparatively little attention has so far been given to potential applications of text-to-speech (t2s) technology in translation, revision, and pemt tasks, which allow translators/post-editors to listen to an artificial computer voice 'reading out' the text they are working on. while the tools currently used in the lsi do not support t2s by default and only trados studio offers a t2s plugin 2 to date, there is evidence of translators seeking other ways of aurally processing text in their work @xcite @xcite . a study that introduced t2s in the translation revision process @xcite yielded encouraging results regarding revisers' error correction performance; however, further research on the effects of t2s on translators' work is certainly needed @xcite . the study by @xcite found revision with t2s to be conducive to correcting more errorsabove all accuracy errors compared to revision in silence. this has promising implications for the integration of t2s in pemt since accuracy has been identified as one of the major challenges for nmt @xcite . given that the use of t2s seemed to have an attention-raising effect in the revision study, we contend that this technology may also be beneficial for pemt and error annotationespecially for translation students. to our knowledge, there is a lack of empirical evidence on: (i) the impact of t2s technology on pemt performance, productivity, and posteditors' attitudes towards this mode of working; and (ii) the impact of t2s on error annotation performance in the context of translator training. we aimed to fill these research gaps with a smallscale study conducted with 17 university students. ## discussion in an ideal world, introducing this new mode of working would enable post-editors to reduce both their over-annotation and under-annotation scores. in response to rq1, we observed that sts+tts was the condition in which all participants except two annotated more errors which were not actually thereso their overannotation scores went up, in some cases by over 20% (figure 1 ). this is not necessarily detrimental to the target text, although it lowers the post-editor's productivity. at the same time (and in response to rq2), the sts+tts condition was also the condition in which fewer actual gs errors were missed by all but 5 participants (figure 2 ). while there is an outlier here with an increase in under-annotation of 31 percentage points (p16), qualitative data revealed that this participant experienced technical difficulties in using the read aloud featurethus offering an example of the detrimental impact on performance posed by userspecific technical challenges. overall, missing fewer real errors is extremely valuable and can improve target text quality if corrected well, provided that the over-annotations and their corresponding corrections do not introduce new errors. our data suggests that, when performing pemt with sts+tts, participants made more preferential annotations, but also missed fewer genuine errors. in the words of p18: "by listening to the segments in the target language that were translated only by a machine, i can detect errors more easily as the translation sounds unnatural to me." although not idealthe ideal would be for post-editors to only make necessary annotations -, identifying more genuine errors while also making what could be classed as 'preferential annotations' could be considered an acceptable compromise. in any case, what these figures show is that limited practice without personalised feedback does not result in ideal performance improvements for an entire group, although encouraging signs could already be seen. for example, at the end of the experiment, for 5 students the percentage by which they over-annotated was actually below the one by which they decreased their underannotation performance. this is a move in the right direction. 5 different students, though, were at the other end of the spectrum, with both higher over-annotations (which is tolerable) and higher under-annotations (which is not ideal). with sufficient practice, though, annotating errors and subsequently correcting them can reach a level of quality which makes this task useful not just for an individual -"you have a clearer picture of what kind of errors you have to correct" (p6)but also for a group collaborating on a pemt project -"it is helpful if you work with others; in that case you don't have to explain to them your decision every time. and if the person you are doing the post-editing for wants to know why you corrected something, it is easier to explain." (p10) furthermore, the qualitative data obtained from the questionnaires (rq3) suggest that the perception of t2s as a useful tool for error annotation and pemt will depend on personal preferences and attitudes. the three participants who did not change their negative attitudes towards t2s were also among those whose error annotation performance changed for the worse between part 1 and part 4. p16 had the largest increase in under-annotation (31 percentage points), while p9 and p14 had the largest increases in over-annotation (29 and 36 percentage points, respectively). conversely, those whose error annotation performance changed for the better between part 1 and part 4 generally indicated positive attitudes towards the use of t2s. of the three students with the highest decrease in under-annotation (p4, p6, p23), the first two indicated a positive attitude before and after the experiment, while p23 changed their attitude from negative to positive in the final questionnaire. p23 shares third place in reducing under-annotation with p10 who kept a neutral attitude throughout the experiment. the only participant to reduce over-annotation (p8) had a positive attitude throughout. other participants perceived speech synthesis as beneficial for text comprehension more generally: "speech synthesis made understanding sentences with slang words much easier. i could understand the spoken words in the context of the sentence better, even though i had never heard them before." (p21) ## conclusions despite rapid advances in technologies such as machine translation and speech synthesis, the professional environments in which translators, revisers, and post-editors work have remained largely unchanged. post-editors are expected to identify and correct at an ever faster rate the unpredictable and often subtle errors produced by neural machine translation engines, but their attention is not yet enhanced and stimulated by multi-modal input. our experiment shows that integrating s2t into pemt workflows can be easily done with existing tools and has practical benefitssimilar to how integrating t2s into revision workflows improved revisers' performance in a previous experiment. moreover, although both the task and the technologies used in the experiment were unfamiliar to the participants, progress was recorded to different degrees concerning performance and attitudes. continued practice supplemented by regular, personalised feedback is likely to accelerate such progress. a more seamless integration of t2s into current cat tools would enable further studies to be conducted in more authentic environments, and more natural-sounding artificial voices would improve the user experience. even at this stage, though, we see t2s as having perceived benefits for content comprehension and error identification, alongside measurable benefits for reducing error under-annotation. future work could thus include investigating the impact of t2s on error annotation and pemt carried out by professional post-editors, with other text types and language pairs than the ones used in this study.