Human- or machine-translated subtitles: Who can tell them apart? (2025)

Abstract: This contribution investigates whether machine-translated subtitles can be easily distinguished from human-translated ones. For this, we run an experiment using two versions of German subtitles for an English television series: (1)produced manually by professional subtitlers, and (2) translated automatically with a Large Language Model (LLM), i.e., GPT4. Our participants were students of translation studies with varying experience in subtitling and the use of machine translation. We asked participants to guess if the subtitles for a selection of video clips had been translated manually or automatically. Apart from analysing whether machine-translated subtitles are distinguishable from human-translated ones, we also seek for indicators of the differences between human and machine translations. Our results show that although it is overall hard to differentiate between human and machine translations, there are some differences. Notably, the more experience the humans have with translation and subtitling, the more able they are to tell apart the two translation variants.

## introduction although machine translation (mt) has arrived in audiovisual translation somewhat later than in some other fields of translation, it has in fact come to play an important role in various translation forms such as subtitling, dubbing, etc. idiomatic and enjoyable target texts are particularly crucial when it comes to the entertainment values that are typically associated with those types of translation, which is why there is skepticism among audiovisual translators concerning the quality of mt in this field (e.g. @xcite . on the other hand, the quality of mt has increased considerably over the last years, and it is common practice to use post-edited mt (mtpe), especially within the field of subtitling. the question has therefore arisen whether mt subtitles are still recognisable as such. for this contribution, we analysed linguistic differences based on automatic annotation, as well as overlaps in words. this step involves a comparison of two translation variants using quantitative information on linguistic features. in addition, we asked human evaluators to recognise the method (manual or automatic) with which the subtitles at hand were produced, building on the results of @xcite . for this step, students were asked to identify human and mt subtitles for an english tv series. apart from the visible surface differences between the two translation variants measured by either linguistic information or human judgement, we are interested in further influencing factors, such as the quality of the subtitles or the test persons' level of expertise. for instance, it is interesting to know if dedicated instruction in subtitling increases the ability to recognise machine-translated subtitles and if other competences may play a role. thus, for our study, we formulate three research questions (rqs): rq1 are there any differences between human and machine translation variants of the same subtitles? rq2 does the quality play a role in the differentiation between human and machine-translated subtitles? rq3 does the level of expertise play a role in the ability to tell apart human and machine translation? in this study, we address the language pair english-german. although both english and german are high-resource languages with much training data and existing mt solutions performing better than for other language pairs, we still believe that looking into this language pair is important. the results of our study are particularly valuable for higher education institutions that train english-german subtitlers, since the information on the differences between mt and human subtitle translation is a great asset for this context. the remainder of this paper is organised as follows: in section 2, we give an overview of related work. section 3 describes the data as well as the methodological design of this study. the results are presented in section 4, which is organised along the rqs. we summarise the results as well as the limitations of this study, and we provide an outlook for future work in section 5. ## conclusion and discussion one aim of the present study was to see if machinetranslated subtitles differ from human-translated ones. we used a number of analyses, including corpus-based frequency distribution of linguistic features, automatic quality scores, as well as human judgements. the overall results show that it is hard to differentiate between manually and automatically translated subtitles. moreover, both translation variants seem to be similar in terms of the distribution of linguistic features such as partsof-speech and syntactic functions. this points to structural similarities between the two outputs. the main differences observed include word choice as indicated by the low bleu score for machine translations, which implies that there are not so many n-gram overlaps between the two translation variants. besides that, we showed that the bleu score did not correlate with human judgements either, as texts with a lower bleu score were more frequently labelled as human translations. also, the calculated bleu score does not necessarily reflect subtitle quality as it is perceived by humans, as our test persons classified the quality of machine translations as good and acceptable frequently, sometimes even more frequently than with the human-translated variants. at the same time, it was interesting to see that the level of expertise measured by the advance in study program does play a role in the ability to correctly differentiate between human and machine translations of subtitles. however, we are also aware of the limitations of this study. first of all, the number of the data that we included into the study (and also survey) is limited to seven texts (clips) only. this restriction was due to the requirements of the given settings: to avoid fatigue effects (which could have impacted the results), we decided beforehand that the survey time should be restricted to a maximum of 30 minutes. given that watching the clips, making decisions and answering the questions takes a considerable amount of time, we could not collect data for more than the seven clips at hand. we plan to extend the data to more clips. although it is challenging to perform a survey with more texts, we would be able to perform a more extensive quantitative analysis of the linguistic differences between human and machine translations including automatic text classification. another drawback of this study is testing translation outputs with an llm only. more machinetranslated outputs, also those produced with traditional mt systems and with other llms than gpt is part of our future work. however, we are also aware of the problems of reproducibility, as the future results that build upon our findings may differ from those reported by us, as llms are regularly updated and are changing. another problem of such systems is that we do not have any control over their training data. the dataset used for testing (the selected clips) is probably included into the training data of the llms at had, as the dataset is open source and freely available. producing subtitle translation specifically for the survey would be a better scenario. besides that, this study does not provide a deep analysis of the subtitle quality. although we mention some issues, we do not report on accuracy and other factors. moreover, pragmatic factors such as transfer of emotions, sentiment, humour, etc. cannot be considered with the methodology applied. however, this can be analysed on the basis of our data in future work. in future, we would also like to extend the test persons to more experienced groups and include professionals from the subtitling industry.