Can postgraduate translation students identify machine-generated text? (2025)

Abstract: Given the growing use of generative artificial intelligence as a tool for creating multilingual content and bypassing traditional translation methods, this study explores the ability of linguistically trained individuals to discern machine-generated output from human-written text (HT). After brief training sessions on the textual anomalies characteristic of synthetic text (ST), twenty-three postgraduate translation students analysed excerpts of Italian prose and assigned likelihood scores to indicate whether they believed they were human-written or AI-generated. The results show that, on average, the students struggled to distinguish between HT and ST, with only two participants achieving notable accuracy. Closer analysis revealed that the students often identified textual anomalies in both HT and ST, although features such as low burstiness and self-contradiction were more frequently associated with ST. These findings suggest the need for improvements in the preparatory training. Moreover, the study raises questions about the necessity of editing synthetic text to make it sound more human-like and recommends further research to determine whether AI-generated text is already sufficiently natural-sounding not to require further refinement.

## introduction authors writing in a second language can today bypass the traditional process of writing in their native language and then having their work translated -either by a human translator or through machine translation -by engineering customized prompts for generative artificial intelligence (genai). these prompts, which may be written in the author's native language, the target language or a combination of both, include a precise description, outline or rough draft of the intended text. consequently, there is no source language document in the traditional sense. content generated in this way may then be refined by a human synthetic-text editor tasked with enhancing its engagement and giving it a more human-like tone. this type of editing requires a skill set distinct from that used in post-editing, as the textual anomalies present in synthetic text (st), -such as redundancy, blandness, verbosity, low burstiness and lack of complex analysis -differ from those typically seen in raw machine translation output @xcite . these anomalies appear to be potentially language-independent. for example, redundancydefined as the repetition of information without adding new meaning or value -can occur in texts written in any language. the need for synthetic-text editing (ste) assumes that readers are indeed capable of distinguishing ai-generated output from humanwritten text (ht). moreover, the ability to identify the textual anomalies characteristic of st is essential for effective ste. @xcite observed that untrained, nonexpert evaluators are not well equipped to detect machine-generated english text, and even with training, their detection success rate improved only slightly, reaching about 55%. in their study, the evaluators were recruited through amazon mechanical turk and were screened only by location/language (english) and their approval rating on the platform. they did not possess specialized knowledge, such as familiarity with large language models (llms) or a background in linguistics 1 . conversely, dou et al. found that english st and ht could be distinguished after laypeople (also recruited through amazon mechanical turk) annotated the texts using a framework called scarecrow, which defines specific error types. ## aims the principal objective of this experiment was: there were also several secondary aims: ## method twenty-three postgraduate translation students at the iulm university in milan, italy, attended two 30-minute lessons, held one week apart, introducing llms, generative artificial intelligence (genai) and some common anomalies reported in @xcite . during these lessons, a few examples of textual anomalies were provided, with the hope that the experiment itself would generate additional examples to improve future training materials. the participants were then presented with 28 short excerpts (ranging from 268 to 467 words) drawn from seven italian short stories, divided into 1 unpublished clarification courtesy of elizabeth clark. four sets of seven excerpts each (a, b, c and d). they were informed that each set contained at least one ht and at least one st excerpt. in reality, each set contained precisely one sequential excerpt of approximately equal length from alberto moravia's short story l'incosciente (the reckless man), from racconti romani @xcite , along with six sequential excerpts from unabridged short stories generated by chatgpt-4o using prompts engineered as described below. the order of excerpts was randomized within each set. the excerpt sets were assigned based on the students' seating arrangement in the lecture room. the student sitting in the first row on the right (from the lecturer's point of view) was assigned set a, the student to their right was assigned set b, the next student set c, and so on, cycling through the sets to ensure a roughly equal distribution. the students were instructed to move on to the next alphabetical set if they finished evaluating their initial set before the allotted time expired. the participants working on set d were instructed to proceed to set a. the experiment concluded once the researcher judged that every student had analysed at least one complete set. the students were asked to assign a score from 0 to 10 to each text excerpt based on its likelihood of being machine-generated (0 = human-written; 10 = machine-generated; 5 = uncertain). intermediate integer scores were allowed. they were also asked to identify and classify the types of anomalies or errors that influenced their assessments according to the categories illustrated during the training sessions. due to time constraints, the participants were encouraged, but not required, to provide specific examples of the anomalies they identified. to prevent the students from distinguishing the ht excerpts by finding them online, they were not allowed to consult the internet during the experiment. they were also not allowed to speak to other people, including fellow participants. a few weeks later, a debriefing session was held, where the students were asked to provide feedback on the experiment and training through a preliminary questionnaire, a class discussion and a final questionnaire identical to the first, to determine whether the discussion had caused them to change their opinions. ## results twenty-three students took part in the experiment. each one analysed an average of 7.74 text excerpts, with the number of assessments ranging from a minimum of 4 (by 1 participant) and 6 (by 2 participants) to a maximum of all 28 (by 1 participant). as shown in however, two students (8.70% of the 23 participants), student no. 8 and student no. 20, showed a notable above-average ability to distinguish the ht from the st excerpts. student no. 8 analysed a total of eight excerpts, consisting of all seven excerpts in set c and one excerpt from set a (ht0 a), even though she should have moved on to set d. she accurately assigned a score of 0 to both excerpts written by alberto moravia. for the remaining six st excerpts, she gave scores ranging from 5 (uncertain) to 10 (definitely machine-generated). student no. 20, on the other hand, analysed the seven excerpts in set d. the text to which she gave the lowest score (3) -signifying it appeared the most human -was the only ht excerpt she assessed. she assigned relatively high scores, ranging from 7 to 9 (indicating somewhere between probably and almost definitely aigenerated) to the six st excerpts. if we exclude the excerpt from set a that student no. 8 analysed, the two students become directly comparable, since they both evaluated a complete set of seven excerpts, each containing one ht. now, let's calculate the probability that these two students correctly identified the ht excerpt purely by chance. the participants were told that at least one excerpt in each set was human-written and at least one was ai-generated. based on this, there are six possible scenarios per set, ranging from "only one excerpt is ht" to "six of the seven excerpts are ht". hence, the probability of a student guessing that only one of the seven excerpts is written by a human is 1/6. assuming they correctly guess that there is only one ht excerpt, the probability of guessing which one it is without looking at them is 1/7, as each set contains seven excerpts. since these two guesses are independent, the combined probability of making both guesses correctly is 1/6 * 1/7 = 1/42, or approximately 2.38%. however, 23 students took part in the experiment, and two of them identified the ht excerpt. the probability that at least two participants out of 23 guess correctly without analysing the excerpts can be calculated using the binomial probability formula: where n represents the total number of participants (@xmath0, k is the number of successful students (@xmath1, and p is the probability that an individual participant guesses correctly (@xmath2, the probability that at least two students out of 23 succeed purely by chance is approximately 10.32% 3 . this relatively low probability strongly suggests that the two students in question used analytical skills, rather than random guessing, to distinguish the ai-generated excerpts from the human-written ones during the experiment. ## limitations since this experiment was conducted as part of a postgraduate degree course with set number of hours, it was necessarily limited to a small selection of texts of a similar kind in a single language. the time available for preparatory training in genai detection was also limited. moreover, the size of the class restricted the number of participants. as a result, the findings and conclusions of this study may not be broadly generalizable. however, the practical, hands-on learning experience and potential contribution to course development outweigh these limitations. ## conclusion the low number of students who were able to distinguish between the two kinds of text in this experiment, even after training, suggests that the guidance given needs redesigning. the examples highlighted by the participants indicate that the preparatory exercises should focus on identifying self-contradiction and assessing variability in syntactic structures and lexical distributions, known as burstiness. moreover, general training on the use of genai as a tool in the translation process, apart from being an essential part of any modern translation course, could also help students better identify st. regarding the length of texts, it would be worthwhile experimenting with longer excerpts. lastly, further research should also explore whether readers genuinely prefer human-written text and whether ste, which seeks to make st sound more human-like, is actually necessary at all.