How Much Do Language Models Copy From Their Training Data? Evaluating Linguistic Novelty in Text Generation Using RAVEN (2023)

Abstract: Current language models can generate high-quality text. Are they simply copying text they have seen before, or have they learned generalizable linguistic abstractions? To tease apart these possibilities, we introduce RAVEN, a suite of analyses for assessing the novelty of generated text, focusing on sequential structure (n-grams) and syntactic structure. We apply these analyses to four neural language models trained on English (an LSTM, a Transformer, Transformer-XL, and GPT-2). For local structure—e.g., individual dependencies—text generated with a standard sampling scheme is substantially less novel than our baseline of human-generated text from each model’s test set. For larger-scale structure—e.g., overall sentence structure—model-generated text is as novel or even more novel than the human-generated baseline, but models still sometimes copy substantially, in some cases duplicating passages over 1,000 words long from the training set. We also perform extensive manual analysis, finding evidence that GPT-2 uses both compositional and analogical generalization mechanisms and showing that GPT-2’s novel text is usually well-formed morphologically and syntactically but has reasonably frequent semantic issues (e.g., being self-contradictory).

## introduction the assumption underlying this approach is that, if the text displays a particular linguistic property (e.g., coherence), then the lm must have captured that property. we argue that this approach has an important shortcoming: the generated text could have been copied from the lm's training data, in which case it does not provide clear evidence for linguistic abstractions. for example, suppose an lm achieves coherence by copying a paragraph from its training set. in this case, the entity that deserves credit for being coherent would not be the lm but rather the human who originally wrote that paragraph. to address this concern, it is important to check whether lm-generated text duplicates from the training data. that is, we argue that lmgenerated text must have two traits to be clear evidence that the lm has learned some abstraction a: much prior work has discussed how to evaluate various aspects of quality @xcite . our central point is that novelty also merits careful consideration. in this work, to quantify the novelty of generated text, we introduce a suite of analyses called raven (rating verbal novelty). 1,2 these analyses cover both sequential structure (n-grams) and syntactic structure. we apply these analyses to text generated by an lstm, a transformer, transformer-xl, and all 4 sizes of gpt-2 (the largest lm for which we had access to the training data). because there are many ways to generate text from lms, we test 12 generation methods and 4 prompt lengths. as a baseline, we also analyze human-generated text from each model's test set. ## n-gram novelty we first investigate novelty at the level of n-grams, where an n-gram is a sequence of n words. motivation: many prior papers (e.g., @xcite use holistic demonstrations of the high quality of lm-generated text as evidence for the lm's overall strength. as discussed in section 1, these conclusions would be undermined if the generated text were copied from the training data. one goal of our work is to test whether this concern is borne out in practice. in this section, we use analyses at the n-gram level as holistic measures of novelty, to match the holistic nature of the relevant prior demonstrations of quality. in later sections, we will conduct analyses that target specific linguistic properties. ## manual analyses of specific phenomena our previous analyses focused on general sequential and syntactic structure. we now investigate some more specific linguistic phenomena by using manual analysis to verify both the quality and the novelty of relevant lm-generated text. manual analysis is labor-intensive; to use this labor for this initial analysis, we study only the novel unigrams that gpt-2 generates; gpt-2 uses subword tokenization, so it can generate novel words by combining seen subwords in novel ways. we study novel words because they give a window into several levels of linguistic structure. studying the words themselves provides insights into word-internal structure (morphology), while studying the context in which novel words appear provides insights into syntactic and semantic structure, since syntax and meaning use individual words as components. see the supplement (section n) for a detailed taxonomy of gpt-2's novel words. here in the main paper, we discuss 4 targeted questions about these novel words. throughout this section, any word in boldface is novel. finding: the vast majority of gpt-2's novel words (96%) are well-formed ( ## conclusion in machine learning, it is critical to evaluate models on a withheld test set. when text is sampled from a language model, that text might be copied from the training set, in which case it is not withheld-so using that text to evaluate the model (e.g., for coherence or grammaticality) is not valid. thus, it is important to consider novelty when using text generation to evaluate the model's abstract abilities. we have introduced raven, an analysis suite covering sequential and syntactic structure, and have applied it to several models, showing that models are rarely novel for local structure but are often novel for larger-scale structure. the types of novelty that models display provide evidence that they have captured a range of linguistic abstractions, such as constituency structure, dependency structure, and several morphological processes. however, models occasionally copy even very long passages, showing that generated text cannot be assumed to be novel: we must directly check for novelty, such as by using the analyses in raven. overall, our results demonstrate the importance of considering a model's training data when evaluating that model's abilities.