Speak, Memory: An Archaeology of Books Known to C hat GPT / GPT -4 (2023)

Abstract: In this work, we carry out a data archaeology to infer books that are known to ChatGPT and GPT-4 using a name cloze membership inference query. We find that OpenAI models have memorized a wide collection of copyrighted materials, and that the degree of memorization is tied to the frequency with which passages of those books appear on the web. The ability of these models to memorize an unknown set of books complicates assessments of measurement validity for cultural analytics by contaminating test data; we show that models perform much better on memorized books than on non-memorized books for downstream tasks. We argue that this supports a case for open models whose training data is known.

## introduction research in cultural analytics at the intersection of nlp and narrative is often focused on developing algorithmic devices to measure some phenomenon of interest in literary texts @xcite @xcite . the rise of large-pretrained language models such as chatgpt and gpt-4 has the potential to radically transform this space by both reducing the need for large-scale training data for new tasks and lowering the technical barrier to entry @xcite . at the same time, however, these models also present a challenge for establishing the validity of results, since few details are known about the data used to train them. as others have shown, the accuracy of such models is strongly dependent on the frequency with which a model has seen information in the training data, calling into question their ability to generalize @xcite @xcite ; in addition, this phenomenon is exacerbated for larger models @xcite . knowing * details of author contributions listed in the appendix. wow. i sit down, fish the questions from my backpack, and go through them, inwardly cursing @xcite for not providing me with a brief biography. i know nothing about this man i'm about to interview. he could be ninety or he could be thirty. → kate (james, fifty shades of grey). some days later, when the land had been moistened by two or three heavy rains, @xcite and his family went to the farm with baskets of seedyams, their hoes and machetes, and the planting began. → okonkwo (achebe, things fall apart). what books a model has been trained on is critical to assess such sources of bias @xcite , which can impact the validity of results in cultural analytics: if evaluation datasets contain memorized books, they provide a false measure of future performance on non-memorized books; without knowing what books a model has been trained on, we are unable to construct evaluation benchmarks that can be sure to exclude them. in this work, we carry out a data archaeology to infer books that are known to several of these large language models. this archaeology is a membership inference query @xcite in which we probe the degree of exact memorization @xcite for a sample of passages from 571 works of fiction published between 1749-2020. this difficult name cloze task, illustrated in figure 1 , has 0% human baseline performance. this archaeology allows us to uncover a number of findings about the books known to openai models which can impact downstream work in cultural analytics: while our work is focused on chatgpt and gpt-4, we also uncover surprising findings about bert as well: bookcorpus @xcite , one of bert's training sources, contains in-copyright materials by published authors, including e.l. james' fifty shades of grey, diana gabaldon's outlander and dan brown's the lost symbol, and bert has memorized this material as well. as researchers in cultural analytics are poised to use chatgpt and gpt-4 for the empirical analysis of literature, our work both sheds light on the underlying knowledge in these models while also illustrating the threats to validity in using them. llms for cultural analysis. while they are more often the object of critique, llms are gaining prominence in large-scale cultural analysis as part of the methodology. for example, some use gpt models to tackle problems related to interpretation, that of real events @xcite , or of character roles @xcite . others leverage llms on classic nlp tasks that can be used to shed light on literary and cultural phenomena, or otherwise maintain an interest in those humanistic domains @xcite . this work shows what tasks and what types of questions llms are more suitable to answer than others through our archaeology of data. ## data we evaluate 5 sources of english-language fiction: weekly from 1924-2020. pre-1923 litbank texts are born digital on project gutenberg and are in the public domain in the united states; all other sources were created by purchasing physical books, scanning them and ocr'ing them with abbyy finereader. as of the time of writing, books published after 1928 are generally in copyright in the u.s. ## effect on downstream tasks the varying degrees to which chatgpt and gpt-4 have seen books in their training data has the potential to bias the accuracy of downstream analyses. to test the degree to which this is true, we carry out two predictive experiments using gpt-4 in a few-shot scenario: predicting the year of first publication of a work from a 250-word passage, and predicting the amount of time that has passed within it. note that here we are interested in the effects of disparity in memorization on downstream tasks, not any individual model's performance on them, and for this reason, we focus on a few-shot setup without any fine-tuning. ## discussion we carry out this work in anticipation of the appeal of chatgpt and gpt-4 for both posing and answering questions in cultural analytics. we find that these models have memorized books, both in the public domain and in copyright, and the capacity for memorization is tied to a book's overall popularity on the web. this differential in memorization leads to differential in performance for downstream tasks, with better performance on popular books than on those not seen on the web. as we consider the use of these models for questions in cultural analytics, we see a number of issues worth considering. the virtues of open models. this archaeology is required only because chatgpt and gpt-4 are closed systems, and the underlying data on which they have been trained is unknown. while our work sheds light on the memorization behavior of these models, the true training data remains fundamentally unknowable outside of openai. as others have pointed out with respect to these systems in particular, there are deep ethical and reproducibility concerns about the use of such closed models for scholarly research @xcite . a preferred alternative is to embrace the development of more open, transparent models, including llama @xcite , opt @xcite and bloom @xcite . while open-data models will not alleviate the disparities in performance we see by virtue of their openness (llama is trained on books in the pile, and all use information from the general web, which each allocate attention to popular works), they address the issue of data contamination since the works in the training data will be known; works in any evaluation data would then be able to be queried for membership @xcite . ## conclusion as research in cultural analytics is poised to be transformed by the affordances of large language models, we carry out this work to uncover the works of fiction that two popular models have memorized, in order to assess the risks of using closed models for research. in our finding that memorization affects performance on downstream tasks in disparate ways, closed data raises uncertainty about the conditions when a model will perform well and when it will fail; while our work tying memorization to popularity on the web offers a rough guide to assess the likelihood of memorization, it does not solve that fundamental challenge. only open models with known training sources would do so. code and data to support this work can be found at https://github.com/bamman-group/ gpt4-books .