Is GPT -3 Text Indistinguishable from Human Text? Scarecrow: A Framework for Scrutinizing Machine Text (2022)

Abstract: Modern neural language models can produce remarkably fluent and grammatical text. So much, in fact, that recent work by Clark et al. (2021) has reported that conventional crowdsourcing can no longer reliably distinguish between machine-authored (GPT-3) and human-authored writing. As errors in machine generations become ever subtler and harder to spot, it poses a new challenge to the research community for robust machine text evaluation. We propose a new framework called Scarecrow for scrutinizing machine text via crowd annotation. To support the broad range of real machine errors that can be identified by laypeople, the ten error categories of Scarecrow—such as redundancy, commonsense errors, and incoherence—are identified through several rounds of crowd annotation experiments without a predefined ontology. We then use Scarecrow to collect over 41k error spans in human-written and machine-generated paragraphs of English language news text. We isolate factors for detailed analysis, including parameter count, training data, and various decoding-time configurations. Our approach successfully quantifies measurable gaps between human authored text and generations from models of several sizes, including fourteen configurations of GPT-3. In addition, our analysis unveils new insights, with detailed rationales provided by laypeople, e.g., that the commonsense capabilities have been improving with larger models while math capabilities have not, and that the choices of simple decoding hyperparameters can make remarkable differences on the perceived quality of machine text. We release our training material, annotation toolkit and dataset at https://yao-dou.github.io/scarecrow/ .

## key findings we perform a large-scale annotation of errors in english news text generated by five sources (four model plateau: increasing model size to gpt-3 does not correlate with further improvements. rising and falling: errors become more prevalent with some models, then improve. humans highest: these spans are labeled most on human-authored text; both are reader issues (distinct from errors; see the main quantity we measure (on y-axes) is span coverage, which is the average portion of tokens that ends up covered by annotations of a particular error type. since it is possible that multiple spans nest or overlap, there is no upper bound for this quantity. (see figure 12 the following are our key findings. figure 4 : taking the average span coverage (figure 3 ) and removing reader issues ( technical jargon and needs google ), we plot values and 95% confidence intervals for all models, including all decoding hyperparameters we tested for gpt-3. we find a surprisingly large change in annotated errors depending on the decoding setting used. or factual verification of the text (more: §e.4). furthermore, human-authored text is not free from error annotations (figure 3 ). this can serve either as a control for baseline error rates (more: §e.6), or as a mechanism for critiquing human writing. to our surprise, the decoding hyperparameters considerably affected error rates (more: §e.5). as seen in figure 4 , the worst sampling procedure for gpt-3 (argmax sampling with no frequency penalty) performed even worse than gpt-2 xl. but the best sampling procedure (surprisingly, also argmax sampling, but with a frequency penalty) produced text with as few apparent scarecrow error spans as those authored by humans (more: §e.6). all of these findings are discussed in more detail in appendix e. ## evaluation of natural language generation we make our study in the area of open-ended natural language generation, a loose term for generating longer texts with an increased level of creative freedom. the common factor in all open-ended generation tasks such as story, blog, and dialog generation is the wide and diverse nature of target outputs. lexically and even semantically dissimilar responses to the same prompt could be equally valid. for example, a model prompted with the blog title "recipes for success this holiday season" could describe how to roast a turkey or strategies for dealing with the stresses of holiday travel. this allowable variation poses a particular difficulty for the evaluation of generation systems. traditionally, text generation quality for tasks like machine translation or graph-to-text generation has been measured by word overlap with humanauthored references @xcite . though measures like bleu allow for multiple references, they break down when the space of allowable outputs is large, as in open-ended generation. recently introduced metrics seek to remedy this problem @xcite , but the gold standard for evaluating generated text is still human judgment. however, current approaches to eliciting human 4 judgement of generated text often do not provide detailed insight into where models are making progress, where they are failing, and the scope of these failures. a/b-style testing allows for directly comparing one system against others @xcite , but can only express relative improvements. simple likert scale judgements can assess text quality, but do not explain why a generated text receives a given rating, or which segment of the text is problematic. insights into model failures often come instead from a small scale expert analysis of outputs. however, these "error analyses," once a staple of nlp research, have become less common in recent years, perhaps due to their small size and high variance. a hypothesis of the current work is that a well designed error analysis annotation framework could be used by crowdworkers to annotate large amounts of text, thereby providing detailed information about model progress and failures as well as actionable directions for future research. such a framework would be easy to learn, reusable, and independent of particular models or experimental conditions. in what follows, we outline the details of such a method. ## scarecrow annotation methodology this section describes the high-level annotation methodology for scarecrow. ## data collection we collect 13k human annotations of 1.3k paragraphs using scarecrow, resulting in over 41k spans. ## error prediction a natural question is: using this data, can machines learn to detect and classify errors in machine generated text? task we frame this problem as a span classification task. given a span from a generated text, the goal is to classify its error type or output "no error" if there is none. positive examples for each error class are taken from our data. we sample random spans that were not labeled with any error type as negative examples. to ensure a breadth of span lengths, we sample 3 negative spans for every length of error span in the generated text. we split the generated texts into train, development, and test sets using 1063 texts (28029 error spans), 100 texts (2538 spans) and 100 texts (2677 spans) respectively. model we use a standard span classification model inspired by @xcite . this model encodes every generated text using a pretrained language model (roberta-large). spans are represented with the final layer of this encoding. following previous work, we concatenate the start and end tokens with a task-specific learned length embedding. the resulting vector is passed through a feedforward network which reduces its dimensionally to the number of error categories plus a "no error" option. the resulting model has 357m trainable parameters. the model is trained to minimize the cross entropy of the correct span category. we train for 15 epochs using adamw with a learning rate of 10 - foot_2 . we validate after each epoch and use the checkpoint with the lowest validation loss (epoch 8). evaluation to evaluate the error prediction model, we use per-token precision, recall, and f 1 score per error category. we classify every span up to length 30 in a generated text. we take as gold labels the aggregated human error spans collected in our data. in other words, models predict the combined spans of all 10 annotators. for comparison, we also report as human the average metrics of one annotator versus the others (i.e., 1-vs-9). 6 results we see that the model is successful at identifying information that human's would have to manually verify ( needs google ), achieving nearly perfect recall with precision close to 0.6. the model can also identify grammar and usage , incoherent , and redundant errors with higher recall than an individual human annotator, though at the cost of precision (sometimes in the .20s). ## related work automated evaluation metrics such as bleu @xcite , rouge @xcite , me-teor @xcite , and bertscore @xcite compute a generation's score based on a (set of) reference(s). their use is wellestablished in tasks like machine translation and summarization, but they are less helpful in openended text generation, where there is a vast diversity of possible high-quality continuations. recent studies propose automated metrics for open-ended text generation evaluation such as: perception score @xcite , which diffuses evaluation onto a multidimensional space and assigns a single score; union @xcite , which learns to distinguish human-written stories from negative samples by generating perturbations of human-written stories; and mauve @xcite , which compares the distribution of machine-generated text to that of human language. an alternate recent approach to assessing openended text generation was presented in turingadvice @xcite , where crowd workers assess machine-generated advice in response to reddit posts. in their error analysis, zellers et al. connect problems in generated text to core nlp tasks, such as self-contradiction errors as instances of failed natural language inference @xcite , or off-prompt errors as cases of failed reading comprehension @xcite . while past work has attempted to guide text generation using discriminative models trained for such tasks @xcite , it remains an open challenge. comparative human evaluations of natural language generations ask annotators to rank system outputs relative to each other. text is typically evaluated using a few global criteria, such as fluency and relevance, using discrete (e.g., 5-point) @xcite or continuous scales @xcite . recent work even automates this approach, running a human evaluation alongside automatic metrics on leaderboard submissions @xcite . in the roft system @xcite , annotators attempt to detect the boundary between human-and machine-written text as a proxy for assessing quality. while these approaches may be helpfulsometimes @xcite related to or annotation method is pursued by @xcite , who develop a collaborative mobile app where users draw "graffiti" commentary on news articles. scarecrow aims to assess model generations the way we would critique humanwritten text: by locating, coarsely categorizing, and explaining problems. ## conclusion we present scarecrow, a method for identifying and explaining issues in generated text. along with the annotation framework, we present an analysis of the scarecrow method applied to several large neural language models in an open-ended news generation task. we release our data and methodology to the community.