How well do real-time machine translation apps perform in practice? Insights from a literature review (2022)

Abstract: Although more and more professionals are using real-time machine translation during dialogues with interlocutors who speak a different language, the performance of real-time MT apps has received only limited attention in the academic literature. This study summarizes the findings of prior studies (N = 34) reporting an evaluation of one or more real-time MT apps in a professional setting. Our findings show that real-time MT apps are often tested in realistic circumstances and that users are more frequently employed as judges of performance than professional translators. Furthermore, most studies report overall positive results with regard to performance, particularly when apps are tested in real-life situations.

## introduction in 1997, mark seligman wrote that "the internet offers a tremendous opportunity for experiments with real-time machine translation (mt) of dialogues" @xcite . in december of the same year, systran and altavista launched "the first widely available, real-time, high-speed and free translation service on the internet" @xcite . now, 25 years later, the google translate app has been downloaded more than 1 billion times from the google app store @xcite . since 2011, the app offers a conversation mode, which enables users to have utterances within a dialogue translated in real-time so that their conversation partners can understand them. other apps such as itranslate, @xcite the author. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. microsoft translator can also be used to support synchronous dialogue between interlocutors who do not speak the same language @xcite . to the best of our knowledge, there are no publicly available data on the frequency with which mt apps are used for real-time translation and the contexts in which this occurs. however, given the popularity of these apps, it can be expected that a large number of synchronous dialogues are translated every day, and that this happens not only in informal situations, but also in professional contexts. this raises the question of how well real-time mt apps perform in these kinds of situations. traditionally, the academic literature has paid more attention to the quality of written translations that have been produced using mt than to the output of real-time mt apps. this study aims to boost research into the performance of real-time mt apps by summarizing the findings of earlier studies in which the performance of such apps was evaluated in a professional context. ## mt quality assessment the quality of mt output has been a hotly debated topic for decades, and a wide variety of methods for its assessment have been proposed (cf. @xcite . when classifying these methods, authors commonly distinguish between automated metrics and human metrics (e.g., rivera-trigueros, 2021; chatzikoumi, 2020). automated metrics include word error rates (wers), precision, recall, and bleu scores, all of which are calculated on the basis of a comparison between mt output and a reference translation created by a professional human translator. human metrics are further subdivided by @xcite into metrics in which human experts express a direct judgement concerning the translation quality and metrics in which no direct judgement is expressed. when experts are asked to indicate the adequacy or fluency of a machine translated text on a 5-point scale, for example, they make an explicit quality judgement. when, on the other hand, they classify the translation errors occurring in the mt output, they provide useful information for improving the application without explicitly judging the quality of the output. measuring the post-editing effort required to reach an acceptable quality level for the target text (e.g. @xcite also provides an indirect indication of mt quality. there are several reasons why most of the metrics discussed above can be considered less suitable for assessing real-time mt that is used to support synchronous dialogues. first of all, postediting does not occur in such situations, so postediting effort cannot be used as a quality indicator. in the absence of a human-generated reference translation, automated metrics can also not be calculated. technically speaking, human experts could judge the quality of the output after the dialogue has taken place, but they would be at a disadvantage due to the limited length and disfluent nature of the source texts, particularly when speech input is used @xcite . moreover, it is important to acknowledge that mt quality assessment can have different purposes. many of the metrics above were primarily developed to identify areas of improvement for mt applications that are 'under construction' @xcite . for professionals contemplating the use of real-time mt in their daily professional routines, however, improving the application is not the main priority. they want to know whether using mt will enhance the quality of their interactions with patients, students or business partners who speak a different language. in some cases, they might even wonder whether the use of mt is ethically responsible given the prevalence of errors in mt output and the potentially damaging consequences of such errors in certain contexts @xcite . taken together, these considerations suggest that the evaluation of real-time mt might best be approached from the perspective of 'fitness for purpose', which is achieved when the quality of a translation is 'good enough' for the end user to understand the information content and pragmatic intent of a translated message @xcite . although this concept has featured prominently in both practical and academic discourse about translation quality for quite some time (jiménez-crespo, 2018), it is not yet standard practice to ask end users to assess the quality of (post-edited) mt output (cf. @xcite . this raises the question to what extent existing studies into the performance of real-time mt apps are guided by the concept of fitness for purpose, and how fitness for purpose is operationalized in evaluation methods used in these studies. for the current paper, we are specifically interested in the answers to the following questions: rq1: to what extent are real-time mt applications tested in authentic professional situations? rq2: which quality indicators are most commonly used and how are they operationalized? rq3: who judges the performance of real-time mt apps? rq4: which overall picture concerning the performance of real-time mt apps emerges from the research conducted so far? we hope to find these answers by conducting a systematic literature review of prior studies (@xmath0. more information about our methodology is provided in the next chapter. ## method for our literature review, we collected papers published in peer-reviewed journals or conference proceedings which assessed the quality of linguistic material that was translated in real-time by an mt application and that was related to actual or potential dialogues in professional settings (e.g., healthcare, education or tourism). studies that focused on other types of linguistic material (e.g., websites or leaflets) or only described a realtime mt system without reporting an evaluation were excluded from the sample. subsequently, the studies included in the sample were coded on a number of key variables derived from the research questions stated above. the following sections describe the sampling method, the coding procedure and the statistical analyses.