To MT or not to MT : An eye-tracking study on the reception by D utch readers of different translation and creativity levels (2025)

Abstract: This article presents the results of a pilot study involving the reception of a fictional short story translated from English into Dutch under four conditions: machine translation (MT), post-editing (PE), human translation (HT) and original source text (ST). The aim is to understand how creativity and errors in different translation modalities affect readers, specifically regarding cognitive load. Eight participants filled in a questionnaire, read a story using an eye-tracker, and conducted a retrospective think-aloud (RTA) interview. The results show that units of creative potential (UCP) increase cognitive load and that this is the highest in HT and the lowest in MT; no effect of error was observed. Triangulating the data with RTAs leads us to hypothesize that the higher cognitive load in UCPs is linked to increases in reader enjoyment and immersion. The effect of translation creativity on cognitive load in different translation modalities at word-level is novel and opens up new avenues for further research.

## measuring creativity and cognitive load this section describes the data, annotation criteria, participants, eye-tracking device, questionnaire & rta interviews, as well as the preprocessing of the initial dataset and statistical modelling. combining quantitative and qualitative methods allows us to triangulate the data to understand the complex interactions of variables, such as creativity and cognitive load, to answer our rqs. ## cognitive load, creativity and reading experience in this section, we discuss the results from the questionnaires, eye-tracking device and rtas that serve to answer our four rqs. ## conclusions and further research we were seeking to test our methodology with four rqs that linked cognitive load and creativity. for ease of understanding we present here the questions and the findings, followed up by the limitations of the study and future avenues of research. rq1: do readers have a higher cognitive load in units of creative potential than in other regular parts of the sentence? there was a strong positive effect of ucps on cognitive load, both in css and reproductions. readers thus pay attention to ucps when reading, and judging from their comments in the rtas, they also enjoy reading them. this link between engagement and cognitive load was also found in literary reading studies @xcite , specifically for foregrounded elements @xcite . although the methodology employed is solid and the results novel, the creation of the datasets is quite arduous. we think that it might be better for this word-level experiments to look at datasets that have been purposely created for this, for example, by only looking at paragraphs with specific ucps where it will be easier to explore the effect of modality. rq2: do readers process these units differently according to the translation modality? although we did not find an effect for modality, we did see a positive effect in its interaction with creativity when looking at tfd. furthermore, the effect of creativity was lessened in mt, while increased in both pe and ht. comparing ht and pe against each other specifically, we see that in ht the effect of creativity is the highest. this was reinforced in the rtas, where ht participants were more positive about the text, especially its style, even explicitly appreciating certain translation solutions. pe participants liked the narrative, but thought the style fell flat at times. this differs from @xcite , who found increased cognitive load for low quality sentences; however, they only looked at non-literary texts, while we looked at a literary text, and literary studies have shown a positive link between the literariness of a text and fixations @xcite , so that might explain the difference. potentially, the higher overall quality of ht (as attested in the rtas) enhances the immersive effects of ucps. rq3: do readers process the translators' solutions differently according to the level of creativity? we did not see any significant difference in cognitive load between css and reproductions. analysing word frequency also showed only a low negative correlation between word frequency and tfd, showing that other factors influenced tfd. there was also no clustering of css, reproductions. or non-ucps, indicating that words belonging to css or reproductions are not less frequent than words that do not belong to ucps. again here, we think that, methodologically, experiments of this type would benefit for more focused studies at paragraph-level with chosen ucps. rq4: do errors in a segment increase the cognitive load of the reader? as expected, we see that units with errors have a higher mean tfd than those without, although only slightly. however, we do not see any significant effect on units with or without errors. this differs from previous studies on errors, as @xcite and @xcite found increased tfd and fc on sections with errors. triangulating the data with the interviews, however, allowed us to hypothesise that this is due to increased skim reading in mt especially, the modality with most errors. when these readers saw too many errors, they tended to skim certain sections more. rtas showed that errors in mt and pe influenced reading experience -creating a lack of understanding in mt and a dislike of style in pe-but this was not mirrored in the exerted cognitive load. this study shows both the methodological advantages of analysing reader reception through cognitive load on a word-level, and the importance of creativity for reader reception across modalities. intuitively, we expect differences in reading experiences across modalities, given errors and creativity scores. in previous studies, pe and ht scored similarly and we initially also found no main effect on cognitive load between the two; however, detailed analysis shows that creativity makes a difference; not only that, but the difference creativity makes is increased in ht. in other words, the effect of creativity is lessened in pe compared to ht. the retelling of the participants' experience through the rtas also showed clear differences between the modalities, a crucial aspect of this methodology. using mt-mediated texts in literary translation thus has an impact on readers. more research into the causes of these effects is needed to inform translation technologies, translators and the industry. we are aware of the limitations of this pilot regarding number of participants and language pairs. however, the methodology gives us a window through which we can explore the way readers deal with creativity while reading. we learnt from this experiment that document-level analysis might not be the best match to answer our rqs, therefore our next experiment, focuses on paragraphs with selected ucps in different modalities. this will include more participants, different genres, and llms, to further explore this relation between creativity and reader experience.