B ayesian Hierarchical Modelling for Analysing the Effect of Speech Synthesis on Post-Editing Machine Translation (2024)

Abstract: Automatic speech synthesis has seen rapid development and integration in domains as diverse as accessibility services, translation, or language learning platforms. We analyse its integration in a post-editing machine translation (PEMT) environment and the effect this has on quality, productivity, and cognitive effort. We use Bayesian hierarchical modelling to analyse eye-tracking, time-tracking, and error annotation data resulting from an experiment involving 21 professional translators post-editing from English into German in a customised cloud-based CAT environment and listening to the source and/or target texts via speech synthesis. Using speech synthesis in a PEMT task has a non-substantial positive effect on quality, a substantial negative effect on productivity, and a substantial negative effect on the cognitive effort expended on the target text, signifying that participants need to allocate less cognitive effort to the target text.

## introduction the growing adoption of data-driven approaches to machine translation (mt) since the 2000s @xcite has brought ongoing change to the practice of translation. while 'standard' human translation still appears to be the dominant type of service, industry surveys have repeatedly identified postediting of mt (pemt) as the service with the highest growth potential, according to language service providers @xcite . a wealth of previous research has addressed the implications of this change, ranging from potential productivity gains @xcite to impacts on creativity @xcite . a central theme in studies on pemt is the effort expended by translators @xcite and how it might be impacted by the tools they use. moreover, previous work has probed how well pemt is supported by the user interfaces used by translators @xcite , indicating room for improvement. a relatively novel approach to supporting pemt processes -and translation in general -is integrating automatic text-to-speech synthesis @xcite in computer-assisted translation (cat) tools. the idea is for the translator to be able to trigger an artificial voice that 'reads' to them the source and/or target text, thus adding a new mode of text reception to information processing approaches that have traditionally relied heavily on reading. only little attention has thus far been given to this method in related work, but initial findings point to potential benefits in revision @xcite and pemt @xcite . this motivates our present study into the impact of speech synthesis on the pemt process. in this paper, we measure the effect of adding text-to-speech into a translation workflow for pemt for the english-german language pair. we focus on the the target text quality delivered, cognitive effort expended, and productivity recorded, with an emphasis on the statistical modelling approach. eye-tracking output metrics, such as the number or duration of fixations on both source and target segments are used to measure the cognitive effort during pemt @xcite . moreover, linear models and linear mixed effect models are commonly used for the analysis of eye-tracking studies @xcite , including the use of linear models to investigate the relation across text complexity of the source, cognitive effort, and pemt @xcite . @xcite discuss the disadvantages of standard statistical tests for eye-tracking data analysis in subtitling. for example, t-tests conflate the data by averaging a participant into one outcome variable, and ignore other variables (predictors) that may affect the results of an experiment. instead, they use a linear mixed-effect model for the analysis of cognitive effort (outcome variable) of reading subtitles given the effect of subtitle speeds. however, linear models require large amounts of data to achieve reliable learned estimates @xcite . bayesian hierarchical models cope with data scarcity by adding information from the data structure, and prior expert knowledge that works as a regulariser to avoid over-fitting to the available data @xcite . we use bayesian hierarchical modelling to tackle the issue of data scarcity that is common in eye-tracking studies (o'brien, 2009). our contributions are as follows: ## discussion the results of our experiment on using speech synthesis for pemt indicate that (1) differences in quality between conditions were small; (2) participants were slower when using speech synthesis; and (3) participants expended less cognitive effort in tt when using speech synthesis, as reflected in their fixation data. more specifically, the presence of speech had a substantial negative effect on the mfd-tt, meaning that overall the cognitive effort spent by translators reading the target text was reduced. this may mean that hearing the target text was considered by translators to be a reliable source of information when checking pemt. we report a non-substantial positive effect on the mfd-st variable, indicating that the processing of the source text does not change much and only increases slightly. we do not believe this to be due to a lack of trust in the speech synthesis, given the results for the mfd-tt, but that speech use may be more worthwhile in the tt. this is also suggested by the answer to the perception questionnaire we distributed at the end of the experiment @xcite where the most reported on advantages of using speech were improved style (11/21) and error detection (9/21). it may also be that listening to the tt causes the translators to expend more cognitive effort on the st, but this would require a separate analysis. the decrease in productivity might reflect the fact that listening to the text is an additional step to be carried out in the workflow. moreover, as all participants but one were first-time users of speech synthesis in pemt, productivity losses can reasonably be expected to diminish as users become more familiar with the tool. a longitudinal study would surely provide useful data in this regard. related to this but apart from the effect of the sound condition, we also found that pemt experience has a substantial positive effect on pemt speed, indicating that translators with previous pemt experience work faster than those without. the effect of the number of searches is non-substantial for all outcome variables. we recorded no substantial change in quality, but there is a perceived improvement in style and error detection for some of the participants as reported in @xcite . the loss in productivity may be reduced following longer exposure to speech synthesis. this, coupled with the substantial decrease in cognitive effort in the tt, point to a potential support that a speech-enabled mode of working can offer translators. ## conclusions and future work we quantified the impact of text-to-speech on pemt for the english-german language pair. we introduce a bayesian hierarchical model to tackle issues with data scarcity. the introduction of the sound condition on the pemt workflow has a nonsubstantial positive effect on quality, a substantial negative effect on pemt speed, and a nonsubstantial positive effect on mfd-st and substantial negative effect on the mfd-tt for cognitive effort. the effect of the number of searches is nonsubstantial for all outcome variables. the text id together with the sound condition has an effect on all of the measurements, which may be explained by the standard measurements of text complexity we used, which do not take into account semantics and might not sufficiently reflect textual differences, especially regarding translation difficulty. for future work, we will measure the relation between text complexity evaluated with newer readability formulas based on fine-grained linguistic features and translation quality/productivity @xcite , investigate in more detail the relation between translation experience and translation quality/productivity, the relation between productivity and the number of searches performed, and quantify the observable changes in individual pemt workflows created by our participants' access to speech synthesis. pemt speed productivity outcome: pemt speed,first level predictors: condition, text, n searches, pemt experience, and second level predictors: condition. brms formula: