R o FT : A Tool for Evaluating Human Detection of Machine-Generated Text (2020)

Abstract: In recent years, large neural networks for natural language generation (NLG) have made leaps and bounds in their ability to generate fluent text. However, the tasks of evaluating quality differences between NLG systems and understanding how humans perceive the generated text remain both crucial and difficult. In this system demonstration, we present Real or Fake Text (RoFT), a website that tackles both of these challenges by inviting users to try their hand at detecting machine-generated text in a variety of domains. We introduce a novel evaluation task based on detecting the boundary at which a text passage that starts off human-written transitions to being machine-generated. We show preliminary results of using RoFT to evaluate detection of machine-generated news articles.

## introduction despite considerable advancements in building natural language generation (nlg) systems that can output extremely fluent english text, there is still not very much understanding of how humans perceive machine-generated text. such an understanding is crucial for the evaluation of the improvements in nlg systems and for the analysis of the societal ramifications of machine-generated text as it becomes increasingly easy to produce. when evaluating nlg systems, it is considered standard practice to ask evaluators to rate generated text on criteria such as fluency, naturalness, or relevance to a prompt on a likert scale @xcite . preference studies, where a rater is shown two generated excerpts and asked which one they prefer, are also common. some recent work has focused on the detection problem: how capable humans are at distinguishing textual excerpts gen- * authors listed alphabetically contributed equally. past run -on team fight terrorism olympic nyt excerpt baseme nt misspelled character nat ion al cho sen certainly stiff recite suppression according detail top site date photographed issue furniture oriented hear facto r international r48 min imi ze arranged include pressure paragraph touch abuzz unique means lived offici al exp ect service staff cfda care obituaries president copy/pas te smooth check citation wwd meaning currently brin g pickle men large reasons slightly also shi ed telling four humans longer friday regular dramatic stats deceased early bad thrown high er contradicts rem emb er dri dollar repeating ago school double 1st presence easy revealed veered unfi nish ed crime movie dave form spacing mechine mak es unusual doesn't incoherent refe r referring cap ital support take typical situation possible murder machines art close bot case going piece keeps mob councilman integrity link william huma ns isolated passage steven day contain november compared ben isn't incoherency issued ver sion dou ble oddly ove rly hit pulled nasa later incoherent opinions review meant firm danc ed individuals failed info forced good money actual shows death states com plete reason able larger rise despite wil liam s continued dashes error s returned randomly exhibit legenthaldians power forw ard enforcem ent bridegroo m away repeatation judge law degrees sit ri center playboy island park calculated disc uss economy borissa respect interest speak message giving road working relate group protection grammer may student wording nest refuge pcb however strictly air ball closin g devils nonsense nec leagu e great thinking near store rathe r france city boys moral mother o en armor baker idea box turning pace tie board military big dou bt typi ng history east fed le whose bush cut police heard middle sudden fluent criminal died post rigid points man nam es accountant several posthumous better laguardia fiorello scotland billion waiter galleries formerly dunk separately gotten show coherence highway hospital gurgling entire brew gi 7th extol normally bishop denmark murphy ownds nineteenth christia ns vague ambiguous intellect hyphen poor speaks excessive cleese style barrymore extreme formality simple unclear monday june no-no london bed office rupees construction building peacefully naturally 500,000 rockville asteris k diocese stroke centre improper yes follow ing aged absolutely flags contextua lly terms disabilities americans undersid e doesn't yards less lie damage permanent decorated guerrilla diverse morton memorial fighter kindly suitcase comment happy request opening expands guesisng base beneath pronoun johnson's open kevin eric tag deck maas joined chicago pennie uncomfort partner henry skywalker studied banjo horn french cra wish online muslim majority india's kashmir jammu technical aristide ludicrous develop mistaken haitian violence approaching chain hardware qualify lyman jumps serbian spring fears heighten plot department determined exposure persons black term mission conjugating memorials judi's davis.guthrie@gmail.com maiden appreciated reconn aissanc e counter-drug cromwell's army cleanest produce arrived search officers colomb ian soldiers plane converted 3-point picky likely digestible fragment retired length clear area stations design helped illegally il-legally competitive cooperat ing laboratori es virology leading w.h.o hanging players development reduced brings lower side months elections enacte d unilaterally amendments national deviate o.k perhaps containing dece taxes expense slightly unconnected sad quantity incongruous shopping declined income operating choreography bernardo wunderbar require okay probably obsess ion weight-loss farfetched tsunami mystery suicide qaeda caused anonymity bristling providence throws free play seconds granddaughter shor toots saloonkeeper renowned bombings afghanistan-pakistan girl relevance send get misspelling accept listed selections missing run-on rational actions captions honestly ready region vocative i t 's israeli palestinians three eyes wa r e lucky alto blunt times saxophones sweating stepped dwindle d falmouth lobster rolls two together relates chase classifications rater verbatim fair caregivers email gone spied small breach rare stating precedents authority diversion arugula service third bassist ahead aware setting ave eat veering fun orzo tuna roll river free torts educatio n toward fitchburg marath on stripping mad usual discovering rouge jungles weather spam hyphenated dry real sister audience extending washington rice trip thursday handling trials best joel must proplems weddin gs core inevitab le rules plan reiterated feed process calls rothwax believe run leave let kim jong credentials depp caved talked even texts member broken runs till tarot fluid veto hid block bills total ear mccoy units cast solemn oath vote lied specifi c putting boca regularly comparison tells boat need informa l guessing turns fake scored lawrence pure tend german phd seals cold list entity facts jewry bleak louis bigger former forma fail iraq inc glare room muran o vice chief product kolb 5-year works stare fall threat s bio beat ways dra god paul nba invest scope via note age visas places pass felt player lack dmv walk likes far eight darryl orbit crew west business doerr guy's dead clip sort sea days kelly souls raise cash lots fair frills tell c a n' t guns i'll hills 50's gem seen sides child wild dash gear cope n.j inch code local bids abut sosa tra yes whenever vibes loses li ed rates run-up span late low head steel race bee easy goer it's bill fan add file ever silk nee deal bits lurid set limp son joy male taste sites wait loss sent nonsense kept fill six south figure 1: a word cloud of common words that annotators used to describe why they thought sentences were machine-generated. however, due to the prohibitive cost of running human evaluation studies, most prior work in this area has been rather limited in scope. for example, analyses usually show results on only a single category of text (news articles, stories, webtext, etc.). this could be problematic since different domains have different levels of named entities, world facts, narrative coherence, and other properties that impact the success of nlg systems. in addition, most papers only evaluate on a very limited selection of decoding strategy hyperparameters. holtzman et al. (2019) and ippolito et al. (2020) both show that the decoding strategy chosen at inference time can have a significant impact on the quality of generated text. in this work, we introduce the real or fake text (roft) system, a novel application for simultaneously collecting quality annotations of machinegenerated text while allowing the public to assess and improve their skill at detecting machinegenerated text. in roft, we propose to use the task of detecting when text is machine-generated as a quality criterion for comparing nlg systems. following ippolito et al. (2020) , we make the counterintuitive assumption that the worse annotators are at detecting that text is machine-generated, the better we can say that the nlg system is at generating text. in roft's detection task, annotators are shown a passage of text one sentence at a time. the first several sentences are from a real human-written text source and the next several sentences are a machinegenerated continuation. the user's goal is to guess where the boundary is. when they think that a sentence is machine-generated, they are asked to give an explanation for their choice. afterwards the true boundary is revealed. in the remainder of this paper, we discuss why we think this task is interesting from a research perspective and describe the technical details behind our implementation. we show preliminary results that showcase the types of analyses that are possible with the collected data, and finally we discuss plans for future work. the roft website is located at http://www. roft.io/ . the source code is available under an mit license at https://github.com/ kirubarajan/roft . ## research motivations the purpose behind roft is to collect annotations on the scale needed to probe the quality of text generated under a variety of nlg conditions and systems. in this section, we describe three research questions we aim to answer using roft data. ## system overview this section gives an overview of roft's design, including the task that annotators are asked to complete and methods for encouraging organic traffic. ## case study to show the efficacy of roft as an evaluation tool, we present a case study from our initial pilot of over 3000 annotations of generations from the news article domain. ## related work nearly all papers in nlg do some form of human evaluation, usually using amazon mechanical turk (van der @xcite . typically the interfaces for these evaluations are simple web forms. van der @xcite offers a survey of many of these methods. custom-designed websites for collecting or displaying human evaluations of generated text have become increasingly prominent in the openended dialog domain, with chateval @xcite and convai @xcite being two examples. however, roft was primarily influenced by other "real or fake" websites that attempt to gamify the detection task, such as http://www. whichfaceisreal.com/ for generated face images and https://faketrump.ai/ for generated tweets. our task is similar to the one used for human evaluation in @xcite , except in their task the text shown to raters was either entirely human-written or entirely machine-generated. the boundary detection task we propose was inspired by the dialog breakdown detection challenge @xcite , in which the goal is to automatically detect the first system utterance in a conversation between a human and a chatbot system that causes a dialogue breakdown. ## conclusion and future work in this work, we have introduced roft and have shown how it can be used to collect annotations on how well human raters can tell when an article transitions from being human-written to being machine-generated. ultimately, we plan to use roft to conduct a large-scale systematic study of the impact of decoding strategy, fine-tuning dataset, prompt genre, and other factors on the detectability of machinegenerated text. we also intend to collect and release a large dataset of natural language explanations for why humans think text is machine-generated. we hope that these will provide insights into problems with the human-written text we use as prompts and into the types of errors that nlg systems make. such a study will require tens of thousands of human annotations. we hope that by gamifying the annotation process and encouraging organic traffic to the website, we can ultimately bypass the need for crowd workers who, since they are paid by the annotation, are disincentivized from taking the time to provide high quality annotations. we believe that roft provides a powerful tool for understanding the strengths and limitations of a great variety of nlg systems, and we look forward to working with researchers interested in testing out their own model outputs within the roft evaluation framework.