Measuring Machine Translation User Experience ( MTUX ): A Comparison between A ttrak D iff and User Experience Questionnaire (2023)

Abstract: Perceptions and experiences of machine translation (MT) users before, during, and after their interaction with MT systems, products or services has been overlooked both in academia and in industry. Tradi-tionally, the focus has been on productivi-ty and quality, often neglecting the human factor. We propose the concept of Ma-chine Translation User Experience (MTUX) for assessing, evaluating, and getting further information about the user experiences of people interacting with MT. By conducting a human-computer in-teraction (HCI)-based study with 15 pro-fessional translators, we analyse which is the best method for measuring MTUX, and conclude by suggesting the use of the User Experience Questionnaire (UEQ). The measurement of MTUX will help eve-ry stakeholder in the MT industry - devel-opers will be able to identify pain points for the users and solve them in the devel-opment process, resulting in better MTUX and higher adoption of MT systems or products by MT users.

## introduction recently, artificial intelligence has captured the attention of many stakeholders in our society, not only in specialised academic journals and conferences, but also among laypeople @xcite . large language models have driven technological breakthroughs, and the state-of-the-art has evolved mainly through training bigger and bigger models, with more parameters, more training time, and ultimately more computational resources @xcite . research in language technologies has become a race to see who owns and releases the biggest language model @xcite ). this has also provoked the reaction of academics who reflect on language technology research from a socio-technical perspective, promoting a move to a more humancentered development of such language technologies @xcite , which goes beyond 'human in the loop' concepts. in the language services or translation studies domains, mt is a technology that has had significant impact in the past few years, and its adoption and implementation in workflows has provoked some rejection from professional translators @xcite . many professional users feel that their needs have not been considered in the development and deployment of these technologies, and have therefore felt dehumanised, commodified, with an accompanying loss of agency and status @xcite . this results in a lack of acceptance and trust in these technologies, which is usually not a rejection of the technology, but a veto on the way in which mt is applied and used @xcite . human factors such as users' perceptions or experiences of mt as a tool that facilitates multilingual communication -regardless of whether we are talking about professional translators or other types of users -have often been overlooked. the focus of research has been on the quality and productivity benefits of using these technologies @xcite , neglecting human satisfaction and resulting experiences of such human-computer interaction. this paper aims to fill a gap in the literature by proposing the concept machine translation user experience (mtux) and recommending its application in language technology research and development processes to create better, user-centered language technology products, which would result in improved human-computer interactions. we first present the related work, followed by the definition of the term mtux and the methodology used to discern the best method for evaluating mtux in multilingual communication processes. ## related work since the emergence of mt, academia and industry have analysed its impact and implications for translation processes and multilingual communication @xcite . the focus of research has been on professional translators. typically, attention has focused on the speed of production for translation (or productivity) through post-editing against the productivity without mt assistance @xcite . it has been shown that post-editing, in many situations, makes it possible to be more productive than translating without mt support (sánchez-gijón, moorkens, and way 2019). hence, the introduction and adoption of mt in industry workflows to meet more agile, fast and urgent translation and/or localisation processes (elis 2022). some attention has also been paid to translation quality: does the use of mt affect the final quality of a translation? are translations done through postediting worse than translations done directly by humans and without any mt intervention? guerberof arenas (2014), for example, reported in an experiment with 24 professional translators that there were no statistically significant differences in translation quality of texts produced with mt output against texts produced without mt assistance. nevertheless, the study of the perceptions and considerations that users have about their interaction with mt and new language technologies is scarce. some experiments dealing with these topics have only been disseminated in a superficial, descriptive way. for instance, @xcite analysed with a 4-point likert scale what professional translators thought of post-editing in a subtitling workflow. more extensive consideration was undertaken by pérez-macías, ramos, and rico (2020), who studied the perceptions of professional translators towards mt in the migratory context. @xcite also looked at the perceptions of mt from translators from the european commission. other research has also focused on what lay users of mt think of such language technologies, like that of @xcite , where results suggested that lay users find mt useful and tend to use it for gisting and assimilation purposes. additional research has even reported that users' perceptions of language technologies, such as the perception that mt is a threat to their profession, or the level of trust they have in mt, have a strong correlation with the final translation quality in a professional setting (briva-iglesias, @xcite . this demonstrates the importance of considering users' perceptions when interacting with technologies, as perceptions can have direct correlation or association with final translation quality. besides, it is important to note that there has been no specific action to collect perceptions from previous research and introduce this human feedback into the process of developing, updating or improving new language technologies, since, as we have mentioned above, these new technological breakthroughs have been especially technical, but not sociotechnical, forgetting the human factor in multilingual communication @xcite . by presenting the concept of mtux, we intend to suggest a solution to this problem. ## machine translation user experience (mtux) nowadays, the close relationship between people and technologies allows us to say that multilingual communication can be seen as a form of humancomputer interaction in many instances. we are not only talking about professional translators who use technologies in the performance of their daily tasks. we can also include a user who does not know a language and wants to understand a text by using an online mt system for assimilation purposes, or because they want to share this information in their own language with someone else. it is therefore key to understand and know how these different types of human-mt interactions work. human-computer interaction (hci) focuses on analysing the interactions of people with different systems, products or technological tools @xcite . large technological companies typically have entire teams dedicated to usability or user experience (ux), with the aim of improving the experiences of users when interacting with tools and thus achieving an expected end result. this expected goal may be to achieve a higher customer conversion, for example. however, in the field of multilingual communication, translation studies, and mt, the inclusion of hci methods, among which we can find the study of human and subjective factors, has been largely neglected. the small number of studies are described below. läubli et al. (2020) examined whether the way source and target segments were presented had any effect on productivity and error detection. they concluded that a segment-by-segment (top-bottom) presentation gave better results than a side-by-side segment presentation. paradoxically, most current cat tools still use side-by-side segment presentation. o'brien et al. (2017) studied different functionalities of cat tools from a hci-perspective, and found some features that irritated professional translators and increased cognitive friction. consequently, they made a series of recommendations suggesting that technology tool developers should work with users to implement improvements. from another point of view, some first steps have tried to address this lack of hci methods in translation studies and mt by introducing more transversal methodologies and methods. an example is the work conducted by guerberof arenas, moorkens, and o'brien (2021), who introduced a usability questionnaire to assess the impact of translation modality on what the final readers of translated text thought, as well as to devise whether they could perform different tasks with the different texts. another interesting work was conducted by koponen et al. (2020), who analysed the experiences of subtitlers when using mt and used the user experience questionnaire (ueq) developed by laugwitz et al. (2018) to measure ux. karakanta et al. ( 2022 ) conducted a similar study, replicating the methodology of koponen and colleagues, but with a bigger number of subtitlers and focusing on automatic subtitling. both studies lead to the conclusion that subtitlers' experiences of using mt in subtitling or automated subtitling ranged from neutral to slightly positive. in a similar vein, briva-iglesias, o'brien, and cowan (2023) analysed whether traditional or interactive post-editing had any effect on the ux of professional translators or the resulting quality and productivity after such an interaction, concluding that the interactive post-editing modality caused a statistically significantly higher ux than traditional post-editing. going back to koponen and colleagues' research, they made a modification of the validated ueq to adapt it to the post-editing task, but no further analysis of consistency, validity or reliability was carried out. moreover, only experiences during interaction with the tool were analysed, forgetting about pre-and post-task perceptions. this exclusion of elements may be problematic, as we may lose information from some crucial elements in the human-computer interaction. by considering the above analysis of literature, it is clear that both academia and industry have focused on studying the usability of mt, which, if we follow the definition of this concept provided by the iso 9241-11:2018 on ergonomics of human-system interaction, is "[the] extent to which a system [...] can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use" (iso 2018). in the field of hci, usability is a fragment of a much broader and currently relevant concept, user experience (or ux), which according to the same iso standard above, is "[a] person's perceptions and responses resulting from the use and/or anticipated use of a system" (iso 2018). therefore, we propose that pre-, during-, and posttask perceptions should be considered when assessing mtux. we believe that further methodological consideration of mtux is needed at present, as it would help all stakeholders involved in the fields of translation studies, the language services industry, the mt domain or the multilingual communication world. in mt studies, there is little literature or research on the analysis of user experiences when interacting with mt. why is this the case when mt is so relevant today? why is the focus on training larger and larger language models and not on improving the user experiences of the systems? or, alternatively, why are we not paying attention to what the needs of specific users are in order to adapt and personalise these technologies to users' needs? our supposition is that developers of mt systems are concerned about a particular aspect of quality, normally calculated via bleu scores or some variant, which is driven by mt system 'competitions', but that this has caused a rather narrow focus on system performance that assumes if the output is of good quality, all users of the system will be satisfied. however, this is a simplistic and untested hypothesis, especially seeing as mt systems have highly variable performance across different languages, text types, use cases and contexts. our aim in this paper is to discover the best methodology for analysing mtux in a way that can be applied to the full spectrum of mt users, and that allows us to: therefore, we propose the concept machine translation user experience (mtux) as "[a] person's perceptions and responses resulting from the use and/or anticipated use of mt". from this definition, we place a special emphasis in "resulting from", but also in "anticipated use". we consider that both pre-, during-, and post-task perceptions and experiences related to the interaction of a person with an mt system, product or tool should be equally considered. our suggestion is that mtux should be used both in the translation studies sector to analyse what professional translators experience in their work according to their domain (translators specialised in legal texts will have different experiences and/or needs compared with subtitlers), as well as to discover what other mt users feel when interacting with mt (such as an academic with an l1 other than english who writes in their l1 and then translates the text with mt). we acknowledge these are not the only use-case scenarios where mtux should be studied and analysed, but just some examples. moreover, mtux is also crucial in technology development, as there should be a symbiosis and collaboration between the mt and the language technology sector to introduce feedback from actual users in order to carry out updates, modifications or changes in the tools that have an impact and a real repercussion on the final mtux. this will become more and more important as we see mt becoming further embedded into other technologies like, for example, social media or educational technology tools and increased use of multimodal mt. it would also allow for personalising technological tools to each use case according to the user, with their subsequent adoption and better reception among the community for which such personalisation is intended (o'brien and conlan 2018). ## methodology in hci, there has been substantial discussion about the methodology for measuring ux, and different methods have been proposed depending on the objective of each researcher or study @xcite . some examples put the attention on the hedonic quality (hq) of a product, and pay closer attention to emotions, hedonic elements or sensations @xcite , while others have focused on the pragmatic quality (pq) of a product, paying closer ## discussion by simply comparing items, we might conclude that 19 pairs of adjectives are very similar or identical between the two questionnaires. however, if we analyse the orphan items, we can see that attrakdiff has a higher focus on the hedonic quality of the products evaluated, i.e. it is a questionnaire with a more emotional emphasis and in search of the user's pleasure. this questionnaire may be more appropriate for evaluating ux from a graphic design point of view, such as web page layout and functionality or applications whose objectives are creating hedonic pleasure for the user. in contrast, ueq puts more emphasis on the pragmatic quality of the product or system, and focuses more on efficiency, as we can see in the orphan pairs "inefficient-efficient" and "slow-fast", which are elements completely neglected in attrakdiff. we suggest that this kind of adjective pair is very relevant for measuring mtux, because whether or not mt users think the mt system, product or service helps them to be efficient or fast is valuable information for analysing the interaction of users with mt. the relevance of this user perception is even more important if we are comparing two different ways of interacting with mt, such as traditional and interactive post-editing. in the language services industry, a sector where productivity is vital (due to the fact that, if a translator works faster, this usually translates into higher profits for them personally or the company they work for), assessing pragmatic quality is a key element. thus, we conclude that, in terms of items and adjective pairs, ueq is more relevant for measuring mtux because it combines both the hedonic and emotional views of users with more pragmatic and efficiency perceptions. item analysis is not the only fact that supports our preference for ueq -the results of reliability and consistency between participants and factors through the cronbach alpha coefficients also tip the balance towards the use of ueq. in 5 out of 6 factors, the cronbach alpha coefficient is higher in ueq than in attrakdiff. novelty is the only factor where this is not the case, as attrakdiff obtains a cronbach alpha of 0.85 vs. 0.71 in ueq. it is also worth stressing that attrakdiff obtains very weak cronbach alpha results in the factors related to the pragmatic quality of the product or system (0.10 in perspicuity and 0.03 in dependability; if compared with 0.85 and 0.70 in ueq), which we already know to be of vital importance. finally, the bland-altman graph supports the results obtained by calculating the cronbach alpha values and indicates that, although both questionnaires should report comparable measurements for similar or identical items, this is not the case. 39% of the perceptions and data points collected fall outside the 95% confidence intervals and, therefore, we can conclude that the two questionnaires do not measure these items in the same way. consequently, we believe that in a situation where both hedonic and pragmatic elements are of interest in the ux, as in the evaluation of mtux, the appropriate method to use is the ueq. in case we wanted to analyse any other type of ux more related with graphic design, for example, where the focus is more on the aesthetics of the tool or user pleasure, attrakdiff may be a more appropriate choice. ## conclusions in this article, we have identified a gap in the literature in the field of mt in multilingual communication processes: more attention needs to be paid to the users interacting with mt and not only to the productivity and quality of the tools. we believe that technical advances must go hand in hand with sociotechnical evaluation, which has been neglected to date @xcite . we therefore present the concept of mtux and explain the role that its adoption can play in the development of language technologies and especially mt, with the aim of creating sustainable and ethical language technologies. for the first time, two of the most commonly used questionnaires in hci for measuring ux have been applied to mt use in order to study mtux. data from 15 professional translators working in different iterations suggest that the best tool for measuring mtux is the ueq by considering both pragmatic and hedonic quality criteria of the products or systems evaluated. the adoption of mtux analysis and study will help to create better experiences for any user of mt products or systems and will allow developers to include authentic human feedback in the design process in order to offer personalised tools according to the type of user. this will result in a wider adoption of language technologies or mt, and a better human-machine symbiosis that will bring us closer to intelligence augmentation (ia, as opposed to ai) @xcite . by pursuing ia, we will be able to enhance and improve human skills and capabilities thanks to and through technology in a safe, secure, ethical, sustainable and human-centered way. as for the limitations of the study, the evaluation of mtux requires taking into account the pre-, duringand post-task perceptions of the users. in this paper we have addressed some methodological questions on how to measure mtux by comparing two hci-type questionnaires. we have not had the possibility of exploring how developers might apply results from the questionnaires or how those results could be triangulated with other measures, but this will be the focus of attention in the near future. in future work, we will introduce the machine translation user experience questionnaire (mtuxq) to facilitate the analysis of all user perceptions related to mt interaction and semi-automate the statistical efforts that can be an initial barrier to the study of mtux.