The use of online translators by students not enrolled in a professional translation program: beyond copying and pasting for a professional use (2022)

Abstract: In this paper, we discuss a use of machine translation (MT) that has been quite overlooked up to now, namely by students not enrolled in a professional translation program. A number of studies have reported massive use of free online translators (OTs), and it seems important to uncover such users’ abilities and difficulties when using MT output, whether to improve their understanding, writing, or translation skills. We report here a study on students enrolled in a French ‘applied languages program’ (where students study two languages, as well as law, economics, and management). The aim was to uncover how they use OTs, as well as their (in)ability to identify and correct MT errors. Obtained through two online surveys and several tests conducted with students from 2020 to 2022, our results show an unsurprising widespread use of OTs for many different tasks, but also some specific difficulties in identifying MT errors, in particular in relation to target language fluency.

## introduction most professional translation training programs now include specific training on machine translation (mt) and post-editing (mtpe). mtrelated skills, in connection with project management, are for instance an important component of the european commission directorate-general for translation's competence framework for the european master's in translation network @xcite . a lot of research has already been done on such students' ability to post-edit mt output and on how to teach professional mt skills for the translation market. however, a professional use of mt is not restricted to the translation industry; for example, free ots might also be used by tourism or international relations professionals, and before that by students of such disciplines. not a lot of research has been done on this issue so far, and our work aims to help fill such a gap by studying how students enrolled in a french applied languages program, where they study two languages in addition to law, economics, and management, actually use ots. we believe more research is necessary on the use of mt outside the translation industry, especially as no specific training is generally provided (see below), and as there is a link between mt use and language acquisition @xcite . also, raising awareness concerning the capabilities and limits of using ots is all the more crucial these days because of (i) a real improvement in the quality of mt output since the advent of neural mt (nmt), and (ii) the biased perception of the general public, including students who never received any specific training. this bias is related, on the one hand, to contempt for the technology (see the numerous, supposedly funny mt fails all over the internet) and on the other hand to the belief that translators are obsolete because mt has reached "human parity". a number of studies have shown that a very large majority of students regularly use ots (e.g. @xcite @xcite , for both graded and non-graded work, and regardless of whether this is allowed by their institution. while these studies focus on students enrolled in language programs or with a background in humanities, there is no reason to consider that other students do not use ots. and yet, this widespread use generally takes place without any specific training: according to benites et al. (2021), for example, 77.1% of trainers in 4 swiss universities (@xmath0, and 83.9% of the students (@xmath1,926) claimed that they had never received any specific guidelines on the use of mt. this makes mt a real "elephant in the classroom" @xcite . however, recently, researchers have been working on how to help mt users outside the translation industry adopt a critical approach (see @xcite for the research community, or bowker (2020) for international business students for suggestions, which both put forward the concept "mt literacy", see below). from this starting point, we decided to investigate the use of ots by our students, in order to understand their uses and also to measure their efficiency when using mt output. to do so, we submitted groups of students to an online questionnaire (in 2020 and 2021) and to different types of exercises meant to evaluate their capacity to identify and correct errors in english-french mt output. this is ongoing research, as after a pilot study @xcite to get an overview of our starting point, we have been trying to find the best ways to train (and evaluate) our students' capacity to use ots, hence regular tests since 2020. as discussed below, this is not an easy task, with students finding it hard to identify mt errors. the different tests aim to determine whether the language direction, the necessity to both identify and correct mt errors vs. only correct errors identified for them, the order of presentation for the original input and mt output, have an influence on our students' performance. ## results in line with the few studies mentioned above, our results confirmed that our students are regular users of online translators: 83% in the first survey and 78% in the second answered that they used ots on a regular basis, mostly deepl (8 students out of 10) and google translate (3 students out of 10). however, the mentioning of wordreference ( https://www.wordreference.com ) and linguee ( https://www.linguee.com/ ) in the category 'other ots' indicates some confusion as to what an ot and therefore mtis. according to our survey, students use ots for many different kinds of tasks: translation tasks of course (80% of students), but also as writing aids (45% of students), e.g. when writing an essay, as a comprehension tool (50% of students), and as a grammatical tool (16% of students) for help with grammar exercises. students do not seem to be informed users, since they do not systematically provide enough context to obtain relevant mt output: only 5% of them actually copy/paste full texts; instead, they generally type words or parts of sentences (40% of students). nevertheless, 80% are satisfied with what ots have to offer (40% often, and another 40% sometimes). a large majority of students (93.8% in the first survey, 83.3% in the second one) thought that they were able to identify mt errors, either with no difficulty whatsoever or quite easily. however, such confidence is blatantly contradicted by the results obtained in the different tests, with students clearly overestimating their ability to correct errors in the mt output. out of the 20 errors identified by the evaluators in the first test, only 5.29 on average (1 out of 4) were correctly identified and corrected, with another 2.29 identified but wrongly corrected, meaning that 12.42 (nearly 2 out of 3) were simply overlooked by the students (@xmath2. in the second test, some improvement was noticed despite the fact that the mt output was now in a foreign language for the students (@xmath3. this time, thanks to the segmentation into sentences, an average of 10.2 errors out of 23 (that is a 44% success rate) were correctly identified. still, more than half of the mt errors were overlooked, and only half (56%) of those identified were actually corrected in a relevant way. finally, the third test, in which the students (@xmath4, showed a real improvement with 67% of cases of relevant corrections. in the different tests, a qualitative analysis of students' corrections showed that students tend to focus more on lexical choices than on the syntactic organization of the sentences, and are better at identifying accuracy issues than fluency issues. ## discussion the results of our two surveys and series of tests clearly show that in spite of a very widespread use of ots, for many different tasks ranging from understanding a text to actually translating it, our language students fail to use ots effectively and are not sufficiently able to identify and correct errors in the mt output. in other words, they need to develop their "mt literacy", a concept put forward by @xcite to refer to a series of skills in relation to users' capacity to understand how mt systems work, when it is relevant to use them, and when and how to modify mt output. we can think of two possible explanations for our results which clearly show a lack of critical thinking when using ots while "a healthy level of mistrust in [mt] output" or a kind of "healthy skepticism" (o'brien and ehrensberger-dow, 2020) are required (ots are no calculators). first, since our students find it particularly hard to identify mt errors related to the fluency of the target language, one might think they have a poor command of the target language's linguistic system, even when the target language is their native language. for example, the choice of a wrong mood in (2c) clearly shows that a grammatical rule is not known (74% of our students left the mood unchanged). also, as we noticed direct calques that were left unchanged, it seems that our students are influenced, or "heavily primed", by the mt output that they see on the screen (see @xcite for the concept of priming). this has already been noted for professional post-editors, who "more easily accept sub-optimal translations which human translators, working from scratch, would otherwise not produce" @xcite . this might explain why our students are better at correcting mt errors when these have been identified for them (results of third test). specific training for an informed, professional, and critical use of ots thus seems necessary. to address this need, we have introduced specific training in the translation class for our third-year students (hence perhaps the decrease from 93.8% to 83.3% between our two surveys in the rate of students who consider that they are able to identify mt errors). our approach combines a theoretical and a practical approach. first, it seems important to address some technical considerations by defining what an ot is, how it differs from other online tools such as dictionaries or concordancers, and how it works (roughly) so that they understand why results vary from one ot to another and over time. through comparisons between ots, students can then be made aware of the importance of the corpus data behind the tool. also, thanks to the prolific scientific literature on the subject, a list of recurring mt errors can be provided to sensitize students to the limits of ots. these cover language-independent errors: issues related to lexical/syntactic ambiguities, idiomatic expressions, word play, neologisms or rare words, proper names, omissions, production of non-existing words @xcite , algorithmic bias leading to lesser lexical variety @xcite , gender bias @xcite , and literal translations leading to an over-/under-representation of some linguistic features in mt output @xcite . mt errors can also be language-dependent: for the english-french language pair, issues include the translation of compounds, the present perfect, or pronouns. all these issues (see @xcite , to appear, for concrete examples) should not lead students to believe that mt output is systematically full of errors. however, they can help them become aware of the existence of so-called "machine translationese", and of the need for human intervention in the form of post-editing. raising students' awareness of ethical considerations is also necessary for an informed use in a professional context other than the translation industry. these include confidentiality issues, the environmental impact of the technology @xcite , and also the fact that mt engines are trained on data produced by human translators. students should be sensitized to a "fair use" of ots @xcite , and teaching institutions need to implement clear policies. practical training may include different activities, such as the correction of mt output, but also the comparison between output from different ots, and between mt output and 'human' translation. making students aware of functionalities that allow them to choose between alternative solutions can help them realize that the mt output on the screen is but one possibility among many: deepl allows users to see other possible translations in a drop-down list when they click on a word in the mt output, and google translate provides alternative translations for the whole sentence. such a dynamic approach to online translators, far from simply copying and pasting, then makes the use of ots a decision-making process. the final goal should be to empower students with the skills necessary to use ots independently and critically on their own. finally, we would like to stress that our students' difficulties in dealing with mt output are not isolated. mt is a challenge for everyone these days, and being able to use mt critically is also a challenge for translation students as well as translation professionals. the fact that mt errors have become more human-like with the development of nmt makes them harder to identify (and correct) for translation trainees @xcite and professionals @xcite . our non-translation students' difficulties should therefore come as no surprise, and it is actually incumbent upon trainers to ensure that ots are integrated efficiently into students' set of online language tools alongside different types dictionaries (with or without concordancers and thesauruses), corpus tools, and grammar checkers. ## conclusion in this paper, we cited our own studies that showed widespread use of ots among students, combined with a striking inability of these same students to identify and correct errors in mt output. this led us to advocate for specific training on online translators/machine translation for students not enrolled in a professional translation program, for an informed, professional use. like other studies, ours has shown that ots are widely used by students, who nevertheless still need to develop their mt literacy. while a lot of attention has been paid to how to train translators translators-to-be, the use of machine translation by other categories of students is often overlooked (no training or guidelines by trainers or institutions), making the use of ots an "elephant in the classroom". in order to train students from all disciplines other than professional translation studies, specific pedagogical material is needed. in addition to the scientific literature mentioned above, some projects aim to make such material accessible, e.g., the european multitrainmt project (machine translation training for multilingual citizens, http://www.multitrainmt.eu ) or the machine translation literacy project ( https://sites.google.com/view/machinetranslationliteracy/ ). as for the specific case of our students, an example-based methodology to sensitize them to recurring issues is being developed @xcite . further research is however still needed to uncover the best way to introduce specific training on ots: so far, as our findings demonstrate that students still encounter difficulties in identifying mt errors, training could emphasize the use of grammatical categories and sentence analysis as a means to strengthen students' fluency in the target language, be it their mother tongue or not. being familiar with and using basic grammatical notions to analyze mt output is necessary for a professional use of ot.