Automatic Discrimination of Human and Neural Machine Translation in Multilingual Scenarios (2023)

Abstract: We tackle the task of automatically discriminating between human and machine translations. As opposed to most previous work, we perform experiments in a multilingual setting, considering multiple languages and multilingual pretrained language models. We show that a classifier trained on parallel data with a single source language (in our case German–English) can still perform well on English translations that come from different source languages, even when the machine translations were produced by other systems than the one it was trained on. Additionally, we demonstrate that incorporating the source text in the input of a multilingual classifier improves (i) its accuracy and (ii) its robustness on cross-system evaluation, compared to a monolingual classifier. Furthermore, we find that using training data from multiple source languages (German, Russian and Chinese) tends to improve the accuracy of both monolingual and multilingual classifiers. Finally, we show that bilingual classifiers and classifiers trained on multiple source languages benefit from being trained on longer text sequences, rather than on sentences.

## introduction in many nlp tasks one may want to filter out machine translations (mt), but keep human translations (ht). consider, for example, the construc-tion of parallel corpora used for training mt systems: filtering out mt output is getting progressively harder, given the ever-increasing quality of neural mt (nmt) systems. moreover, the existence of such high-quality nmt systems might aggravate the problem, as people are getting more likely to employ them when creating texts. in addition, it is also hard to get fair training data to build a classifier that can distinguish between these two types of translations, since publicly-available parallel corpora with human translations were likely used in the training of well-known publicly available mt systems (such as google translate or deepl). therefore, we believe that making the most of the scarcely available (multilingual) training data is a crucial research direction. previous work aiming at discriminating between ht and nmt operates mostly in a monolingual setting @xcite , to our knowledge, the only exception is @xcite , who targeted english-french, and fine-tuned not only french lms (monolingual target-only setting), but also multilingual lms, so that the classifier had also access to the source text. however, this work used an in-house data set, therefore limiting reproducibility and practical usefulness. there is also older work that tackled statistical mt (smt) vs ht classification @xcite @xcite . nevertheless, since both the mt paradigm (smt) and the classifiers used are not state-of-the-art anymore, less recent studies are of limited relevance today. compared to previous work, this paper explores the classification of ht vs nmt in the multilingual scenario in more depth, considering several languages and multilingual lms. we demonstrate that classifiers trained on parallel data with a single source language still work well when applied to translations from other source languages (experiment 1). we show improved performance for fine-tuning multilingual lms by incorporating the source text (experiment 2), which also diminishes the gap between training and testing on different mt systems (experiment 3). moreover, we improve performance when training on additional training data from different source languages (experiment 4) and full documents instead of isolated sentences (experiment 5). ## conclusion this paper has investigated the discrimination between neural machine translation (nmt) and human translation (ht) in multilingual scenarios, using as classifiers monolingual and multilingual language models (lms) that are fine-tuned with small amounts of task-specific labelled data. we have found out that a monolingual classifier trained on english translations from a given source language still performs well above chance on english translations from other source languages. using a multilingual lm and therefore having access also to the source sentence results overall in better performance than an equivalent lm that only