Automatic Detection of Machine Generated Text: A Critical Survey (2020)

Abstract: Text generative models (TGMs) excel in producing text that matches the style of human language reasonably well. Such TGMs can be misused by adversaries, e.g., by automatically generating fake news and fake product reviews that can look authentic and fool humans. Detectors that can distinguish text generated by TGM from human written text play a vital role in mitigating such misuse of TGMs. Recently, there has been a flurry of works from both natural language processing (NLP) and machine learning (ML) communities to build accurate detectors for English. Despite the importance of this problem, there is currently no work that surveys this fast-growing literature and introduces newcomers to important research challenges. In this work, we fill this void by providing a critical survey and review of this literature to facilitate a comprehensive understanding of this problem. We conduct an in-depth error analysis of the state-of-the-art detector and discuss research directions to guide future work in this exciting area.

## introduction current state-of-the-art text generative models (tgms) excel in producing text that approaches the style of human language, especially in terms of grammaticality, fluency, coherency, and usage of real world knowledge @xcite @xcite @xcite . tgms are useful in a wide variety of applications, including story generation @xcite , conversational response generation @xcite , code auto-completion @xcite , and radiology report generation @xcite . however, tgms can also be misused for fake news generation @xcite @xcite , fake product reviews generation @xcite , and spamming/phishing. @xcite . thus, it is important to build tools that can minimize the threats posed by the misuse of tgms. the commonly used approach to combat the threats posed by the misuse of tgms is to formulate the problem of distinguishing text generated by tgms and human written text as a classification task. the classifier, henceforth called detector, can be used to automatically remove machine generated text from online platforms such as social media, e-commerce, email clients, and government forums, when the intention of the tgm generated text is abuse. an ideal detector should be: (i) accurate, that is, good accuracy with a good trade-off for false positives and false negatives depending on the online platform (email client, social media) on which tgm is applied @xcite ; (ii) data-efficient, that is, needs as few examples as possible from the tgm used by the attacker @xcite ; (iii) generalizable, that is, detects text generated by different modeling choices of the tgm used by the attacker such as model architecture, tgm training data, tgm conditioning prompt length, model size, and text decoding method @xcite @xcite ; and (iv) interpretable, that is, detector decisions need to be understandable to humans @xcite ; and (v) robust, that is, detector can handle adversarial examples @xcite . given the importance of this problem, there has been a flurry of research recently from both nlp and ml communities on building useful detectors. however, there is currently no work that provides a literature review of existing detection works and highlight important research challenges. in this paper, we present a critical literature review of the existing detection research for english to aid understanding of this important area. we organize the survey to guide the reader seamlessly through a number of important aspects, as follows: first, we establish the background for the detection task, which includes tgms, decoding methods for text generation, and social impacts of tgms ( §2). second, we present various aspects of large-scale tgms such as model architecture, training cost, and controllability ( §3). third, we present and discuss the various existing detectors in terms of their underlying methods ( §4). fourth, we provide a linguistically and computationally motivated analysis of key issues of the state-of-the-art detector ( §5). fifth, we discuss interesting future research directions that can help in building useful detectors ( §6). our main contributions are three-fold: ## background here, we provide the background for the problem of detecting machine generated text from human written text. specifically, we introduce key concepts in training a tgm, generating text from a tgm, and social implications of using tgms in practice. existing detection datasets are discussed in appendix. ## text generative models in this section, we will discuss various aspects of large-scale tgms. these tgms act as threat models since they can be misused by a low-skilled adversary, e.g., by generating fake news and fake product reviews. ## detectors in this section, we discuss various detectors for identifying machine generated text from human written text. to aid understanding of the literature, we organize the detectors according to the underlying methods on which they are based. ## issues with the state-of-the-art detector in this section, we discuss open issues in the state-of-the-art detector based on the roberta model, which has been shown to excel in detecting text generated by tgm based on news articles, product reviews, tweets, and web pages (see §4.3). 3 we focus on the task of detecting text generated by the gpt-2 model from human written amazon product reviews, a challenging task given the shortness of reviews. we employ the roberta detector on the publicly available dataset, containing generations from the gpt-2 model (1542m parameters) based on pure, top-k and top-p sampling along with human written reviews (see appendix for dataset details). in figure 1 , we plot the accuracy of the detector w.r.t. number of training examples per class, averaged over ten random initializations to control for initialization effects. we observe that the roberta detector needs several thousands of examples to reach high accuracy. specifically, it has an impractical requirement of 200k, 15k and 50k training examples for performing at 90% accuracy on identifying pure, top-k and top-p examples respectively. 4 given that creation of large datasets for the detection task is hard @xcite , it is important to investigate whether the data-efficiency of the roberta detector can be significantly improved. we manually inspect 100 randomly picked false positives (machine generated product review incorrectly predicted as human written product review) of the roberta detector trained on 15k examples each from top-p generations and from human written reviews. 5 below, we list down the error categories that we have identified and provide at least one example for each error category. fluency: among the false positive reviews, we find 73 reviews to be very fluent and can confuse even humans (1). that movie got the stars and represents the best of this collection but there's better made creature movies as well including a 1960's remake of 'dracula' with kirk douglas and harrison ford. (5) just love ben affleck! he won't be missed in another very good movie. worth watching especially if you like ben! review (4) on product 'universal studios classic monster collection' contains the incorrect fact that harrison ford acted in 'dracula' movie, and another review (5) on 'runaway jury' movie contains the incorrect fact that ben affleck acted in that movie. spurious entities: in 4 false positive reviews, we find that the review contains novel entities unrelated to the domain of the product. for example, review (6) on 'junkfood' musical product contains novel entity, 'grisberg', which is not associated with music domain. typos and grammatical errors: there are 7 false positive reviews that possess typographical and grammatical errors (10) and (11). we note that such errors (especially spelling errors) are not unusual in online reviews, including those by humans. ## future research directions in this section, we discuss a set of future research directions, which can help in building useful detectors. ## conclusion detectors able to tease apart machine generated text from human written text can play a vital role in mitigating misuse of tgms such as in automatic creation of fake news and fake product reviews. our categorization of existing detectors and related issues into classifiers trained from scratch, zero-shot classifiers, fine-tuning nlms, and human-machine collaboration can help readers contextualize each detector w.r.t the fast-growing literature. we also hope that our computationally and linguistically motivated error analysis of the state-of-the-art detector can bring readers up to speed on many existing challenges in building useful detectors. our rich and diverse set of research directions also have the potential to guide future work in this exciting area.