Pre-task perceptions of MT influence quality and productivity: the importance of better translator-computer interactions and implications for training (2024)

Abstract: This paper presents a user study with 11 professional English-Spanish translators in the legal domain. We analysed whether negative or positive translators’ pre-task perceptions of machine translation (MT) being an aid or a threat had any relationship with final translation quality and productivity in a post-editing workflow. Pre-task perceptions of MT were collected in a questionnaire before translators conducted post-editing tasks and were then correlated with translation productivity and translation quality after an Adequacy-Fluency evaluation. Each participant translated 13 texts over two consecutive weeks, accounting for 120,102 words in total. Results show that translators who had higher levels of trust in MT and thought that MT was not a threat to the translation profession reported higher translation quality and productivity. These results have critical implications: improving translator-computer interactions and fostering MT literacy in translation training may be crucial to reducing negative translators’ pre-task perceptions, resulting in better translation productivity and quality, especially adequacy.

## introduction mt has become an undisputed element of today's workflows in the language services industry @xcite . different studies suggest that improvements in these systems over time have allowed translators to see their productivity increase © 2024 the authors. this article is licensed under a creative commons 4.0 licence, no derivative works, attribution, cc-by-nd. without a negative impact on the quality produced and, therefore, most research in the field of mt has focused on estimating the productivity and quality of mt systems @xcite . however, this adoption of mt is not always accompanied by positive user feedback, as some translators have shown little satisfaction in working and interacting with mt through postediting workflows, either because of a reduction in pay, a sense of dehumanisation of the translation process, or the commodification and uberisation of the language services industry @xcite @xcite . in the translation studies and the mt fields, research centered on analysing human factors in today's translator-computer interactions is still relatively limited, and the perceptions, user experiences (ux) or feelings of mt users, or even whether these feelings and experiences have any effect on their interactions, have been scarce @xcite @xcite @xcite . in this context, we present the results of a study (part of a larger project) (briva-iglesias, 2024) that explores whether translators' pre-task perceptions of mt have any relationship with final translation quality and productivity. below, we first present work related to our study, then we outline the methodology and, finally, results are described and discussed. ## related work in the last decades of research in natural language processing (nlp), the focus has been on making technical advancements, mainly by increasing the size of the language models and the computational power used to obtain better results @xcite , but often neglecting the repercus-sions or risks that this path has or may have on humans @xcite . research in translation technologies has followed a parallel path to nlp research, and most studies have focused on evaluating the quality of mt or comparing different mt paradigms @xcite @xcite . such research is necessary, but technical changes should also be accompanied by sociotechnical studies and their impact on users. @xcite criticized this path by commenting that "the human and organisational aspects are not addressed at all, or only implicitly, [...] when the system is being developed". this has meant that the study of human factors and their interaction with technology has lagged behind and received less attention in translation technology research. however, it has not been completely forgotten. for example, @xcite analysed the perceptions of 438 users of online mt systems, and the majority of participants commented that they were not happy with the results, especially with the quality offered. @xcite studied the perception of post-editing effort in the literary field, and collected the data with questionnaires and short interviews, which they then analysed qualitatively. through a questionnaire completed by 1850 people, @xcite investigated how translators interacted with cat tools, and found that there were certain levels of cognitive friction and that some functionalities of cat tools irritated them. not only freelance or corporate translators have received the attention of academia, but also translators in governmental organisations and international institutions. @xcite surveyed french translators at the european commission to analyse the level of acceptance of mt, and suggested that fear of the technology was the element that hindered its adoption. @xcite conducted a similar study, comparing the level of mt uptake of in-house and institutional translators, sharing similar results. translation in a migration context has also received attention, as multilingual communication is key in crisis scenarios @xcite , and pérez-macías, ramos, and rico (2020) analysed the perceptions of mt and post-editing of translators in a migration context, which were negative in general terms. in contrast, @xcite focused on the audiovisual domain and analysed what 12 professional translators thought about mt and what was their ux after post-editing subtitles. the resulting comments ranged from negative to neutral. these results are in line with other research on audiovisual translation, post-editing and ux, where translators do not view post-editing in subtitling projects favourably @xcite @xcite . in a similar vein, briva-iglesias, @xcite analysed the mtux of translators in the legal domain to see what translators preferred from two different post-editing modalities. however, despite having found research on the perceptions that translators have of mt, the aforementioned studies are exclusively descriptive of participant's perceptions and did not analyse whether these perceptions have any relationship with the quality of the final text or the productivity of translators. this is the gap that this article aims to fill. in cognitive science, multiple studies show that past experiences and perceptions have a great impact and are a determinant for future beliefs, attitudes and behaviours @xcite . in translation studies, de almeida (2013) suggested that positive perceptions towards mt had an impact on post-editing effort, and @xcite reported that training in mt changed perceptions of mt and post-editing. hence, if translators are not happy with their past interactions with mt, and if, before starting a post-editing assignment, they already have a negative opinion about that future interaction (pre-task perceptions), what will the consequences be for the final product (that is, the translation)? are we in a vicious circle in which translators' negative pretask perceptions of mt affect the final quality of the translation and/or their productivity? ## methodology the research question we address in this paper is: do translators' (positive or negative) pre-task perceptions of mt have any statistically significant relationship with the final translation quality or productivity when doing mtpe tasks? to answer this question from a novel point of view, we conducted a human-computer interaction-informed study, where we recruited 11 professional translators in the english-spanish legal translation com-bination and carried out a pre-task questionnaire to examine their opinions, past experiences and attitudes towards mt. this questionnaire was followed by the translation of 13 texts using an interactive mt workflow. subsequently, a professional, expert reviewer assessed the quality of the translations after ensuring consistent evaluation criteria with three professional reviewers. we examined the data obtained using different statistical analysis methods to find out whether there was any correlation between the past experiences and attitudes of translators towards mt and their resulting translation quality and productivity. our hypothesis is that translators with negative pre-task perceptions of mt may produce translations with lower quality than their peers with positive pre-task perceptions because their predisposal to interact with mt will affect their translation processes. the following sub-sections describe the methodology used indepth. ## results this section presents the correlations of translators' pre-task perceptions with fluency, adequacy and productivity. ## discussion of the results after analysing the results, we can see that translators' pre-task perceptions of mt have a higher correlation with adequacy scores than with fluency and productivity scores. research by @xcite suggested that nmt systems produce very fluent translations and, therefore, the effect of translators' pre-task perceptions may not be that impactful on fluency scores if the system already offers a fluent mt output. in terms of adequacy scores, however, translators' pre-task perceptions have a bigger impact. these results indicate that, in post-editing tasks with nmt systems that offer good mt quality (english-spanish in our case), adequacy scores have a higher dependence on the translator, while fluency scores have a lower dependence on translators' translation and/or postediting skills because the mt system already offers higher quality mt output. the expert reviewer assessed the mt output of the mt system used in this study, which obtained a global score of 3.48/4 in terms of adequacy and 3.71/4 in terms of fluency, showcasing good quality results. it may be interesting to further validate this idea by replicating this study in a different language combination, particularly in a case in which nmt systems offer lower mt quality (i.e. a low-resource language combination). in terms of productivity, in general terms, we can see that translators who had positive pre-task perceptions of mt tended to translate faster when conducting post-editing tasks than their peers with negative pre-task perceptions of mt. experience in mtpe tasks has no correlation with final translation quality or productivity in the data analysed. these results indicate that it is the translator who matters. results suggest that a professional translator with good translation skills will offer good quality translations and will work equally faster when interacting with mt, regardless of their experience in providing mtpe language solutions. it is worth stressing that the most notable correlation coefficients were observed in two specific variables: translators' level of trust in mtpe and the perception of mt being a threat to the translation profession. the level of trust translators have in mt shows a strong correlation with adequacy and a moderate correlation with fluency and productivity. this is interesting, as translators who trust mt systems to help them work in their professional, daily tasks offered higher final translation quality than those who did not trust mt systems. this is in line with previous research in cognitive science @xcite , which indicated that prior negative perceptions are an important and crucial determinant for future attitudes and behaviours. in our case, translators' pre-task perceptions of mtpe tasks had a strong negative correlation with the quality of the final product, that is, the translation. this may be because translators who do not trust mt do not enjoy this interaction or do not give their best when interacting with mt in their regular, professional workflows. this also applies to productivity: translators with higher levels of trust on mt translated faster, probably because they were more enthusiastic about engaging with mt. those translators who did not trust mt were probably more reluctant to engage with mt in the best of their abilities. this backs up the results of the second pretask perception variable with a strong negative correlation in our study, that is, whether translators consider mt as a threat to the translation profession. the perception of mt being a threat to translators showed a strong association with final translation quality, both in terms of adequacy and fluency, and the results showed statistical significance. the correlation of this pre-task perception variable with productivity is weaker, but still moderate and statistically significant. what these correlations mean is that translators who, even before starting a post-editing task, think that mt is a threat and harmful for the translation profession are more likely to produce lower translation quality and to translate slower. the study's findings on the relationship between translators' pre-task perceptions and translation quality and productivity have profound implications, offering novel insights into the dynamics of modern translator-computer interactions. it is evident that the approach translators adopt towards a task plays a critical role in determining the final outcome, with varying degrees of influence on different aspects of translation performance (in terms of quality or productivity). these results highlight the problem of modern translator-computer interactions, and suggest that we should give higher attention to the improvement of these interactions, probably by looking at the mtux @xcite @xcite , putting the human in the centre of modern translator-computer interactions (shneiderman, 2022), reducing recent complaints about dehumanisation @xcite . also, this has great implications for the training of translators, as it highlights the importance of mt literacy from different points of view @xcite , which are further detailed in the conclusions. ## conclusions since the emergence of language technologies, the perceptions of those who interact with them have been studied, ranging from professional translators to gisting users @xcite . particular attention has been paid to the perceptions of professional translators, who saw their traditional workflows disrupted by these new technologies. however, most studies to date have been descriptive and did not take into account the relationship between perceptions and the quality of the translation or the productivity of the translator. this article aims to fill this gap in the literature with a longitudinal study of 11 professional english-spanish translators in the legal domain, to explore whether negative or positive translators' pre-task perceptions of mt have any relationship with the final translation quality and productivity in a post-editing workflow. in terms of limitations, it would have been better to increase the sample size of the translations or the number of translators. however, we hired 11 professional translators who produced a total of 120,102 words over 10 consecutive days. it would have also been ideal to have the three reviewers assess all the translations, but due to budget constraints, we reduced the reviewer bias through different evaluation iterations, the measurement of iaa, and the refinement of a set of quality scoring guidelines. exploring additional domains to the legal field would have also been positive to assess whether these results are generalisable to other translation specialisations. as a conclusion, the results suggest that translators with negative pre-task perceptions of mt tend to deliver poorer quality translations, as well as to translate slower, than their peers with positive pre-task perceptions of mt. specially, in our study, we observed that translators who thought that mt was a threat to their profession or distrusted mt as an aid in their work obtained lower quality and productivity scores. these were the two variables with the highest correlation coefficients. by contrast, translators with positive pre-task perceptions obtained better translation quality and productivity scores. this may have happened because translators with negative pre-task perceptions of mt may have not interacted with mt adequately or with an open-minded point of view, impacting their final translation quality and productivity. this research opens up new questions: do these results suggest that there is a direct relationship between pre-task perceptions of translation technologies and final quality and productivity results? would translators with negative pre-task perceptions of mt obtain better quality results if they translated without mt? and what would happen if we trained these translators and taught them to see mt as an aid to augment their skills and help them in their professional tasks? although we now have new questions to answer, what is clear is that the level of trust in mt and the conception of mt as a threat to the translation profession have a strong correlation with final quality results, especially adequacy, and a moderate correlation with productivity. these correlations have important implications. translators fearing mt, or those who are more reluctant to trust their interactions with mt, may not be leveraging the advantages and benefits mt offers them. therefore, the results call for multiple actions to be taken in order to: these two elements would increase the adoption and use of mt as assistance, as well as reduce the negativity of translators' pre-task perceptions of mt. however, it is vital to stress that this mt literacy must be accompanied by a broad and holistic view of mt, including its limitations, so that translators acquire a critical view of when it is appropriate and when not to use mt, as it may also involve important ethical issues @xcite .