Automatic Discrimination of Human and Neural Machine Translation: A Study with Multiple Pre-Trained Models and Longer Context (2022)

Abstract: We address the task of automatically distinguishing between human-translated (HT) and machine translated (MT) texts. Following recent work, we fine-tune pre-trained language models (LMs) to perform this task. Our work differs in that we use state-of-the-art pre-trained LMs, as well as the test sets of the WMT news shared tasks as training data, to ensure the sentences were not seen during training of the MT system itself. Moreover, we analyse performance for a number of different experimental setups, such as adding translationese data, going beyond the sentence-level and normalizing punctuation. We show that (i) choosing a state-of-the-art LM can make quite a difference: our best baseline system (DeBERTa) outperforms both BERT and RoBERTa by over 3% accuracy, (ii) adding translationese data is only beneficial if there is not much data available, (iii) considerable improvements can be obtained by classifying at the document-level and (iv) normalizing punctuation and thus avoiding (some) shortcuts has no impact on model performance.

## introduction generally speaking, translations are either performed manually by a human, or performed automatically by a machine translation (mt) system. there exist many use cases in natural language processing in which working with a humantranslated text is not a problem, as they are usually of high quality, but in which we would like to filter out automatically translated texts. for example, consider training an mt system on a parallel corpus crawled from the internet: we would preferably only keep the high-quality human-translated sentences. in this paper, we will address this task of discriminating between human-translated (ht) and machine-translated texts automatically. studies that have analysed mt outputs and hts comparatively have found evidence of systematic differences between the two @xcite @xcite . these outcomes provide indications that an automatic classifier should in principle be able to discriminate between these two classes, at least to some extent. there is previous related work in this direction @xcite @xcite , but they used statistical machine translation (smt) systems to get the translations, while the introduction of neural machine translation (nmt) has considerably improved general translation quality and has led to more natural translations @xcite . arguably, the discrimination between mt and ht is therefore more difficult with nmt systems than it was with previous paradigms to mt. we follow two recent publications that have attempted to distinguish nmt outputs from hts @xcite and work with mt outputs generated by state-of-the-art online nmt systems. additionally, we also build a classifier by fine-tuning a pre-trained language model (lm), given the fact that this approach obtains state-of-the-art performance in many text-based classification tasks. the main differences with previous work are: the rest of the paper is organised as follows. section 2 outlines previous work on the topic. section 3 details our methodology, focusing on the data sets, classifiers and evaluation metrics. subsequently, section 4 presents our experiments and their results. these are complemented by a discussion and further analyses, in section 5. finally, section 6 presents our conclusions and suggestions for future work. all our data, code and results is publicly available at https://github.com/ tobiasvanderwerff/ht-vs-mt ## related work analyses previous work has dealt with finding systematic and qualitative differences between ht and mt. @xcite compared manually an nmt system and a ht for one text in the translation direction english-to-swedish. they found that the translation by nmt was closer to the source and exhibited a more restricted repertoire of translation procedures than the ht. related, an automatic analysis by @xcite found that translations by nmt systems exhibit less lexical diversity than hts. a contemporary automatic analysis corroborated the finding about less lexical diversity and concluded also that mt led to translation that had lower lexical density, were more normalised and had more interference from the source language @xcite . ## discussion & analysis thus far we have reported results in terms of an automatic evaluation metric: classification accuracy. now we would like to delve deeper by conducting analyses that allow us to obtain further insights. to this end, we exploit the fact that the svm classifier outputs the most discriminative features for each class: ht and mt. ## conclusions in this paper we trained classifiers to automatically distinguish between human and machine translations for german-to-english. our classifiers are built by pre-training state-of-the-art language models. we use the test sets of the wmt shared tasks, to ensure that the machine translation systems we use (deepl and google) did not see the data already during training. throughout a number of experiments, we show that: (i) the task is quite challenging, as our best sentence-level systems obtain around 65% accuracy, (ii) using translationese data during training is only beneficial if there is limited data available, (iii) the accuracy drops considerably when performing cross mt-system evaluating, (iv) accuracy improves when performing the task on the document-level and (v) normalizing punctuation (and thus avoiding certain shortcuts) does not have an impact on model performance. in future work, we aim to do a number of things. for one, we want to experiment with both translation directions and different source languages instead of just german. second, we want to perform cross-domain experiments (as in bhardwaj et al. ( 2020 )), as we currently only looked at news texts. 14 third, we want to look at the effect of the source language: does a monolingual model that is trained on english translations from german still work on translations into english from different source languages? this can shed on light on the question in what sense general source language-independent features that discriminate between ht and mt are actually identified by the model. fourth, we plan to also use the source sentence, with a multilingual pre-trained lm, following @xcite . this additional information is expected to lead to better results. while the source sentence is not always available, there are real-world cases in which it is, e.g. filtering crawled parallel corpora. fifth, we would like to expand the task to a 3-way classification, as in the least restrictive scenario, given a text in a language, it could be either originally written in that language, human translated from another language or machine translated from another language.