Human Raters Cannot Distinguish E nglish Translations from Original E nglish Texts (2023)

Abstract: The term translationese describes the set of linguistic features unique to translated texts, which appear regardless of translation quality. Though automatic classifiers designed to distinguish translated texts achieve high accuracy and prior work has identified common hallmarks of translationese, human accuracy of identifying translated text is understudied. In this work, we perform a human evaluation of English original/translated texts in order to explore raters’ ability to classify texts as being original or translated English and the features that lead a rater to judge text as being translated. Ultimately, we find that, regardless of the annotators’ native language or the source language of the text, annotators are unable to distinguish translations from original English texts and also have low agreement. Our results provide critical insight into work in translation studies and context for assessments of translationese classifiers.

## introduction translationese is the set of features that are unique to translated texts, even in high-quality translations @xcite . prior work on translationese has characterized hallmarks of translated texts @xcite , for example when the translated text conforms more to the grammatical structure of the source language than the target language due to source language "shining through" @xcite . it is clearly important to understand how translations actually differ from original texts in perception. though automatic classifiers can identify translated versus original english texts @xcite @xcite @xcite by detecting statistical patterns associated with features of translationese (section 2), work examining how humans distinguish translations is critically understudied. because these statistical tendencies associated with translations emerge at a large scale, even in natural-sounding translations @xcite , this raises the question of whether individual translated passages exhibit discernible features which distinguish them from text originally written in the language. building on a prior study which assessed finnish speakers' ability to identify translated texts @xcite , in this work we perform (to our knowledge, the first) comprehensive study for english texts assessing whether human raters are able to distinguish human-produced english translations from original english texts. specifically, we ask 8 human raters (both native and nonnative speakers of english) to judge 120 english passages as being translated or originals. our selected data spans 6 genres. we consider 5 research questions: rq1 can human raters reliably (above chance) make a binary judgment as to whether a sentence is originally written in or translated into english? rq2 do raters agree on translationese judgments (regardless of accuracy)? rq3 are native speakers able to make this judgment more accurately than nonnative speakers? rq4 does source language affect rater accuracy? rq5 does genre affect rater accuracy? we perform empirical analyses to examine the relationship between each variable (speakers' native language, source language of the text, and genre) and human raters' ability to distinguish english translations from original english texts, thus indicating how each variable affects translation perception itself. where mean accuracy is denoted as µ a and @xmath0.5, we define our null hypothesis h 0 as µ @xmath1. the alternate hypothesis h 1 is thus defined as µ @xmath2. we expect that this work will provide muchneeded foundational insight into translation studies and may also be useful for engineers building translationese classifiers. ## related work the term translationese describes the set of features commonly found in translated texts that do not appear in text originally written in that language @xcite , including interference from the source language or over-normalizing to the target language. features of translationese include simplified language in the translated text @xcite , overuse of cohesive markers and overspecification of implied information in the source text per the explicitation hypothesis @xcite . human and machine translations exhibit different characteristics of translationese @xcite . language-specific differences have been observed between original and translated texts in chinese @xcite and spanish @xcite , among other languages. from an engineering perspective, the presence of translationese also has a negative effect on system performance and evaluation. translationese in test sets can lead to inflated and inaccurate evaluation scores @xcite and when incorporated into training data, the performance of the model may also be affected @xcite . recent work has set out to mitigate the amount of translationese in embeddings @xcite and in text itself @xcite . the most related work to ours is tirkkonen-condit (2002), a translationese study which performed two pilots asking human raters to judge finnish texts as being translated or original (a binary judgment). the first pilot included 27 annotators and used 20 original finnish passages and 20 translated finnish passages, from various domains/ genres. extra-linguistic identifiers such as named entities were filtered out as to not hint to whether the text was translated or not. in the second pilot, 74 teachers of finnish as a foreign language judged 6 of the texts from the first pilot. the study found that the accuracy of humans to distinguish translated or original finnish texts was 61.5% for pilot one and 63.1% for pilot two, only slightly above chance. tirkkonen-condit (2002) also found that "unique" items-such as collocations, idioms, and language-specific features-are often an accurate indicator of an original finnish text. in our work, we expand on tirkkonen-condit's (2002) study in four notable ways. first, we perform the first human evaluation of translationese for english, shining a light on whether humans' low accuracy at distinguishing original/translated texts holds beyond finnish. second, while tirkkonen-condit (2002) seems to have only collected judgments from native speakers of finnish, we collect annotations from both native and nonnative speakers of english. third, we explore whether genre affects rater accuracy. finally, we incorporate a statistical rigor into our study by addressing each research question with empirical analyses. ## methodology & experimental design in order to understand how translations differ from original texts and human perception of those differences, we ask fluent speakers of english to judge passages as being either originally english or translated into english. data. we sample data from @xcite and @xcite , which contain a mix of human translated and original english sentences as well as metadata marking each sentence as translated/original. the data consists of european parliament proceedings, canadian hansard (parliamentary proceedings), literature, political commentary, ted talks @xcite , and united nations proceedings @xcite . before sampling from these datasets, we manually filtered out passages which clearly revealed the source language or nationality of the speaker (such as "i would therefore once more ask you to ensure that we get a dutch channel as well," and "the government of the russian federation finds it necessary to [...]"). from these datasets, we randomly select a uniform sample of 60 passages originally in english and 10 translated passages originally written in each of the following 6 languages: arabic, chinese (mandarin), french, german, russian, and spanish. this results in an even split between original and translated english passages (120 passages total, 60 translated and 60 originally english). following tirkkonen-condit (2002), which used passages of length 100 -300 tokens, we collect passages between 90 and 300 tokens (adapted slightly to fit natural cutoff points off our data). human evaluation. in order to investigate whether translations are more identifiable by native or nonnative speakers, we collect (binary) judgments from four native and four nonnative speakers of english. all annotators are graduate students fluent in english with background in linguistics. the annotators have not received special training in the specific task of translation detection prior to completing the task. the instructions presented to the annotators can be seen in figure 1 . the annotators mark each passage as being o (originally english) or t (translated into english) and are able to provide additional comments justifying their annotations (discussed in section 5). ## results & discussion rq1 can human raters reliably (above chance) make a binary judgment as to whether a sentence is originally or translated english? we calculate accuracy for each annotator and with these accuracies we perform a one-sample t-test. in order to determine whether annotators are able to perform above chance (> 0.5), we set the hypothetical mean to 0.5 and test whether the two-tailed p-value indicates a statistical significance between the hypothetical mean and the actual mean of accuracies. the twotailed p-@xmath3.8907, which is not a statistically significant p-value, so we fail to reject the null hypothesis h 0 : µ @xmath4.5. the actual mean of accuracies for the 8 annotators is 0.4969, with a standard deviation of 0.0620. thus, no, human raters are not able to perform above chance when distinguishing original/translated english. rq2 do raters agree on translationese judgments (regardless of accuracy)? in order to assess annotators' agreement on translationese judgments, we calculate inter-annotator agreement via cohen's kappa. the inter-annotator agreement across all 28 calculations (every combination of the 8 annotators) produces a mean of 0.0706 and ranges from -0.1933 to 0.2236, which is very low agreement for even the most similar pair of annotators. therefore, no, annotators do not agree on translationese judgments. interestingly, even passages with high agreement tend to be inaccurate. of the 6 passages which receive 6 or 7 judgments of t (none received all 8 ts), only 3 were in fact translations. of the 9 passages for which all raters gave a judgment of o, only 3 of the 9 passages were actually originally english. 11 passages received 7/8 o judgments and 6 of those 11 were indeed originally english. rq3 are native speakers able to make this judgment more accurately than nonnative speakers? no, native speakers are not able to more accurately distinguish translated/original english texts than nonnative speakers. again using annotator accuracies, we perform a two-sample t-test, which results in a two-tailed p-@xmath5.7023, indicating that the difference is not statistically significant. the mean, standard deviation, and standard error of means for both groups of speakers can be seen in rq4 does source language affect rater accuracy? first, for each of the 7 original languages of the text (arabic, german, english, spanish, french, russian, and mandarin), we calculate accuracies for each annotator (seen in moving beyond specific language pairs, now we consider whether annotator accuracy is statistically different between the non-english source languages, as well as whether there is a statistical difference in annotator accuracy between english texts and all non-english texts (grouped together). when removing english and performing a oneway anova on only the 6 non-english languages, the p-@xmath6.8063, indicating that there is no statistical difference between the 6 non-english source languages. we then measure the statistical difference between english and all non-english languages with a two-sample t-test, and find that the two-tailed p-@xmath7.0008, revealing that there is a statistically significant difference in annotator accuracy for english and non-english. the mean english accuracy was 0.6458 (standard deviation thank you for taking part in this human evaluation. in the survey tab, you will find 120 human-produced english passages, some of which were originally uttered in english, and some of which were originally uttered in a non-english language and then translated (by a human) into english. the question we would like you to answer is: does this sentence sound like it was originally written in english or translated from another language into english? there are two options: (1) originally english, i.e. sentence was originally uttered in english, or (2) translated into english, i.e. the sentence was originally uttered in a non-english language and then human translated into english. for each sentence, pick either (1) or (2) by putting an o or t to indicate o for originally english and t for translated into english. of 0.1425) while the mean non-english accuracy was 0.3354 (standard deviation of 0.1508). this demonstrates that annotators are able to more accurately identify an originally english passage as being such, though this is likely due to the fact that the raters' judgments were imbalanced and marked more passages as originals than translations (an average of 78 / 120 passages across annotators were judged as originals). in particular, annotator 8 judged most of the passages to be originals, leading to high accuracy for the english passages, and very low accuracy for the non-english passages (again seen in rq5 does genre affect rater accuracy? yes, when performing a one-way anova on the mean accuracies for each annotator of each genre, the p-@xmath8.0272, showing that there is a statistical difference in mean accuracy by genre. per tukey's hsd, we find that there is a statistically significant difference between the following two treatments: literature and political commentary, and political commentary and united nations proceedings. this is indicative of annotators having more difficulty judging literary texts and united nations commentary than the political commentary (see accuracies in similarly, the united nations proceedings are translations of speech and contain more complex topics. on the other hand, the political commentary genre is the most well-formed and structured in nature of the genres considered, which may be why it was ## discussion of annotator comments annotators generally described the task as challenging and noted particular difficulty with passages containing technical or "bureaucratic" language. annotators commented on "clues" which led them to their judgments, such as framing/content ("comparing modern geopolitics to ancient greece seems like an american/british thing to do") and date formatting. linguistic features such as idioms (e.g."its credibility is now shot") and metaphorical language (e.g. "picture i am painting") led annotators to mark a passage as originally english. appropriate use of infrequent prepositions, such as "amidst," also caused annotators to believe the passage was originally composed in english. on the other hand, annotators cited surprising preposition choice (e.g. "following on the earlier attacks") and the use of excessive modifiers as indicators of text being a translation. unnatural sentence structure, unnatural sounding language (e.g. "fund of information"), under/overspecification, and repetition were all also noted in comments as reasons to annotate a passage as a translation. intriguingly, though the annotators achieved low accuracy, most of the reasons cited as being reasons to judge a text as a translation are established hallmarks of translationese @xcite . ## conclusion in this study we assessed human raters' ability to identify english translations as distinct from original english texts and found that raters are not able to accurately make this distinction or agree with other annotators. our statistical analyses revealed that, interestingly, human raters' (in)ability to distinguish english translations from original english text persists regardless of the speakers' native language or the source language of the text. we found that even passages with high agreement were often inaccurately annotated. with respect to genre, formal written texts are able to be distinguished more accurately than speech or literary works. though existing translationese classifiers are able to separate translated and original texts with high accuracy (>90%) @xcite , human raters are not able to do so on the same genres. this is likely because classifiers are picking up on more subtle cues that can be measured statistically but are not easily visible to human annotators; this is validated by amponsah-kaakyire et al. @xcite , which demonstrates that hand-crafted features of translationese are only a small portion of what bert-based translationese classifiers learn, and these hand-crafted features are even subsumed under what bert learns without features. our results further indicate that there are underlying properties of texts that separate translations from originals which are not easily visible in the surface form to human annotators and that, in line with prior work, well-known features of translations do not fully encompass "translationese" on a statistical scale.