Human or Neural Translation? (2020)

Abstract: Deep neural models tremendously improved machine translation. In this context, we investigate whether distinguishing machine from human translations is still feasible. We trained and applied 18 classifiers under two settings: a monolingual task, in which the classifier only looks at the translation; and a bilingual task, in which the source text is also taken into consideration. We report on extensive experiments involving 4 neural MT systems (Google Translate, DeepL, as well as two systems we trained) and varying the domain of texts. We show that the bilingual task is the easiest one and that transfer-based deep-learning classifiers perform best, with mean accuracies around 85% in-domain and 75% out-of-domain .

## introduction this work addresses the task of distinguishing between translations produced by humans and machines. practical applications for this include: improving machine translation systems @xcite , filtering parallel data mined from the web @xcite and evaluating machine translation quality without reference translations @xcite . in our case, we are more interested in tracing the origin of translations outsourced by a large institutional translation service. our work aims at distinguishing between human and neural machine translations at the sentence level. we consider two settings: a monolingual task, where only the target sentence is considered; and a bilingual task where both the source text and its translation are available. we compare feature-based approaches with several deep learning methods, investigating the impact of text domains and mt systems (in-house neural engines, google translate, deepl), paying attention to cases where the translation engine at test time is different from the one used for training, which we found often not studied in related work. we show that identifying machine translation is still feasible nowadays. on the bilingual task, the best transfer learning method we tested recorded an in-domain accuracy of 87.6% and out-of-domain performances ranging between 65.4% and 84.2% depending on the domain of texts and mt system considered. we analyze why our classifiers manage to do better than chance even though translations produced automatically seem to us of very good quality overall. we believe our study offers many new data points, and hope it will foster research on this timely topic. after reviewing related work in section 2, we describe our dataset and experimental setting in section 3, the neural mt systems we used in our experiments in section 4 and the classifiers we tested in section 5. we present our experimental results in section 6 and propose a deeper analysis in section 7. ## related work most studies on identifying machine translation were conducted at a time where mt systems were fraught with problems that rendered their identification somewhat easy. current neural mt systems deliver translations that are sometimes bafflingly fluent. we are not aware of much work addressing mt identification with these newer systems. one notable exception is a recent study by @xcite on distinguishing original sentences from translations produced by google translate (gt). the authors build on the interesting intuition that back translations of automatically translated texts should contain less variations (word usage, structure) than back translations of human translations. they report an accuracy of 75% with an svm classifier on a small corpus of 1200 sentences selected from the europarl corpus foot_0 that are either original (human) or translated with gt. in their experiments, they use the same translation engine for producing the automatic translation of test sentences, and the backtranslations used by the classifier. in a real-world scenario, we are not expected to know which system has been used for producing a translation (we do not even know if a translation has been produced automatically) and the impact of producing back-translations by a different translation engine remains to investigate. in earlier work on mt identification, approaches and evaluations vary greatly from one study to the other. for instance, @xcite uses features extracted from the parse tree of the sentence to characterize, as well as features capturing the density of some function words (with the help of a part-of-speech tagger), and some features dedicated to out-of-vocabulary words. they also use features aimed at capturing emotion agreement inside a sentence, using a dictionary of emotion words. they gathered a balanced dataset of human and machine translations from the europarl corpus (including french-english, german-english, italian-english and danish-english language pairs) using a statistical machine translation (smt) engine trained in-domain with moses @xcite . they report an accuracy of 74.2%. however, they do not analyze which features are the most beneficial to the task. @xcite investigate the use of features to capture the fluency of the text, such as partof-speech and word-based n-gram language models, as well as features aimed at detecting so-called phrase-salad phenomena @xcite , i.e. poor inter-phrasal coherence often observed in smt output. on a collection of public texts crawled over the web, they report an impressive accuracy of 95.8% when distinguishing human versus automatic translations for the english-japanese language pair. the best performance was observed when combining all the features, and surpasses that of humans performing the same task (88.2%). the authors did not show the quality of the automatic translations, but mentioned that it was pretty low compared to the translations produced by native speakers and professional translators. it is therefore questionable how their approach would do on good quality automatic translations. @xcite use features capturing the presence or absence of part-of-speech tags and function words taken from liwc @xcite appearing at least 10 times in the training material. on a corpus extracted from the canadian hansards, and using various translation engines, they report accuracies at detecting machine versus human translations (under a monolingual scenario tested on the english language) which are inversely correlated with the quality of the mt system used. for the best translation engines, they report an accuracy slightly over 60%. ## data all our experiments are centered around one very large dataset: the translation memory of a large institutional translation service. this data collection -called tm hereafter -contains the english and french versions of over 1.8 million documents, covering over 200 broad domains (military, health, etc.), for a total close to 140 million sentence pairs. since the vast majority of translations in the tm are into french, we focus on this language direction. our goal is to build classifiers that determine if a translation is human or machine-made. for this, we need training data that contains both types of translations. we create such data by machine translating a subset of 530k sentence pairs, randomly sampled from the tm. these machine translations are performed using two different neural mt systems, themselves trained using a distinct subset of 5.8m sentence pairs, also randomly sampled from the tm. 2 these two mt systems, one based on xlm @xcite and one on fairseq @xcite , are detailed in section 4. thus, two distinct classifier training sets are created, one from each mt system: each contains 530k human translations and 530k machine translations, totalling 1.06m examples. we proceed similarly to produce test sets to evaluate the performance of our classifiers: we randomly sample 10k sentence pairs from the tm, machine translate the english versions into french using our xlm and fairseq mt systems, thus creating two test sets of 20k examples (10k human translations + 10k machine translations) each. we call these x-tm (for xlm) and f-tm (for fairseq). these two test sets can be seen as "in-domain" relative to our classifiers: not only because they share the same source as the training data (the tm), but also because the machine translations were produced using the same mt systems. to test the ability of our classifiers to handle different text domains and translations produced by different mt engines, we also created "out-of-domain" test sets: we used two online translation platforms -deepl foot_2 (d) and google translate foot_3 (gt) -to translate 10k sentences of each of four publicly available data sets: europarl (euro), canadian hansard (hans), foot_4 the news commentaries (news) available through the wmt conference, foot_5 and the common crawl corpus (crawl) also available through wmt. again these were mixed in equal parts with human translations. in what follows, each test set is named based on the system used to produce automatic translations, and the domain of the material. we further translated another excerpt of (previously unused) 10k sentences from the tm, using the deepl translation api with a private account, to produce a test set we call d-tm. the tm being a proprietary translation memory, we did not submit it to the gt platform. ## nmt systems as noted above, to produce the training data for our classifiers, we first created two transformer-based nmt systems using english-french texts from the tm. we provide the details of this process here. ## mt identification we experimented with two strategies for building classifiers: feature-based models trained from scratch, as well as deep learning ones making use of pre-trained representations. ## experiments we trained all classifiers described above using training data produced with xlm and fairseq mt systems. overall, classifiers trained with fairseq translations performed very marginally better on out-ofdomain data, with an average accuracy of 64.5%, compared to 64.3% for classifiers trained with xlm translations. in this section, unless otherwise specified, we report the results of classifiers trained with fairseq translations, but both training sets produce very comparable results. ## conclusion in this study, we implemented 18 classifiers to detect machine-translated texts, and evaluated their performance on several test sets, containing translations produced by different state-of-the-art nmt systems. figure 2 : examples of human and automatic translations. underlined passages identify problems and bold ones their corresponding parts. examples i) and v) come along manually edited variants of the automatic translation (modifications are in italic), followed in parentheses by the number of classifiers (among 18) that identify an automatic translation. overall, we found that classifiers with access to both the source sentence and the translation perform better than those with access to the translation alone. our classifiers achieve accuracies above 80% on several test sets and always surpass a random baseline. our analysis reveals that, despite of our efforts to normalize translations, artifacts still exist in the data that could explain in part our relatively high classifier accuracies. but in general, it appears that nmt systems do elicit signatures that can be recognized by automatic methods. often, a single lexical choice gives away the automatic nature of the translation, even when the translation looks fluent from a language model point of view. while we had the opportunity to work on a large, high quality professional translation memory, we realize that our results can not be replicated exactly: by nature, large professional tms are proprietary and not easily shared. we argue however that one can easily reproduce @xcite our experiments in another setting. in future work, we hope to produce better mt detectors by creating training data using a wider variety of mt systems. another question we would like to examine is to what extent it is possible to detect post-edited translations, i.e. machine translations manually edited by human translators.