N arrative XL : a Large-scale Dataset for Long-Term Memory Models (2023)

Abstract: We propose a new large-scale (nearly a million questions) ultra-long-context (more than 50,000 words average document length) reading comprehension dataset. Using GPT 3.5, we summarized each scene in 1,500 hand-curated fiction books from Project Gutenberg, which resulted in approximately 150 scene-level summaries per book. After that, we created a number of reading comprehension questions based on these summaries, including three types of multiple-choice scene recognition questions, as well as free-form narrative reconstruction questions. With 990,595 total questions, our dataset is an order of magnitude larger than the closest alternatives. Crucially, most questions have a known “retention demand”, indicating how long-term of a memory is needed to answer them, which should aid long-term memory performance evaluation. We validate our data in four small-scale experiments: one with human labelers, and three with existing language models. We show that our questions 1) adequately represent the source material 2) can be used to diagnose a model’s memory capacity 3) are not trivial for modern language models even when the memory demand does not exceed those models’ context lengths. Lastly, we provide our code which can be used to further expand the dataset with minimal human labor.

## introduction although on paper many modern large language models (llms) have maximal context lengths measured in tens of thousands of tokens, in practice, they often fail to access information plainly presented within those contexts @xcite and their performance generally deteriorates as inputs grow larger. 1 we believe that this issue is a consequence of the lack of supervised datasets that could be used to directly train extremely-long-context llms (as opposed to extrapolating from shorter sequences @xcite which is prone to generalization errors). in our work, we create such a dataset. we capitalize on recent results showing that for tasks that do not require long-term memory, llms rival human labelers @xcite ; we use this "local" competence to create a massive datset (nearly a million questions in total) of extremely long-term memory problems (average context lengths from 54,334 to 87,051 words for different question types). ## language modeling is not enough in theory, optimal next-word prediction requires perfect memory of unlimited capacity, hence one might argue that supervised long-term memory datasets are unnecessary, given the abundance of unsupervised data. in practice, there are two considerations as to why language modeling alone might not be the best approach to train and test language models with extremely long context windows. first, language modeling performance will likely see diminishing returns when the context window is increased. many documents in popular unsupervised datasets are simply not long enough to benefit from contexts larger than ten thousand words. additionally, for longer documents (e.g. fiction books), it is likely that remembering the last read chapter or two is nearly equivalent, in terms of the next word prediction quality, to remembering all the chapters read so far. it is possible that in some narratives, a given character or item might reappear after a long absence, but such cases are likely to happen only a few times per book, making the task extremely sparse and inefficient in training long-term memory models. second, language modeling does not offer a direct way to interpretably measure long-term memory capacity and performance. for example, we do often see improvement in perplexity when the effective context window is increased (e.g. @xcite ), but it is still difficult to measure and understand where exactly the improvement comes from and what kind of information is retained. one scenario could be that a longer context window helps a given model better understand lengthy philosophical treatises present in the dataset, which, in turn, allows the model to extrapolate such arguments in consistent and sound ways, resulting in lower perplexity. alternatively, the model might simply be better at populating bibliography sections of such treatises, being able to copy the cited names from the main text into the bibliography using its long context. we speculate, therefore, that in order for extremely long-context models to thrive, it is necessary to develop specialized supervised datasets that would address the limitations above. creating such a dataset is the main contribution of our paper. ## existing datasets traditionally, long-term memory transformers were tested either on 1) artificial tasks (e.g. @xcite @xcite ) or 2) language modeling (e.g. @xcite @xcite ). training or evaluation on supervised naturalistic long-term datasets is relatively rare. until recently, creating such datasets in a brute-force manner had been prohibitively expensive, as it required tens of thousands of hours of human labor. there have been, however, creative workarounds taking advantage of existing resources. notable examples include @xcite which used scientific papers from arxiv and pubmed and their corresponding abstracts to create medium-term summarization data, and booksum @xcite which scraped webarchive to find summaries for project gutenberg books. the scrolls dataset @xcite aggregated and curated a number of such datasets in order to create a longtext understanding benchmark. a recent zero-scrolls dataset @xcite (concurrent with our work) expanded scrolls with two more sub-datasets tailored towards zero-shot evaluation. in this context, it is especially important to discuss the narrativeqa dataset @xcite since this work is especially close to ours in its goals, scope, and structure 2 . 2 narrativeqa is included as one of the subtasks in both scrolls and zeroscrolls. in those datasets, documents in narrativeqa, the authors employed crowdsource workers to create book and movie script understanding questions based on corresponding web-scraped summaries. while we highly resonate with the importance and motivation of their work, the dataset has a few crucial disadvantages. we discuss additional, subtler differences between narrativexl (our work) and narrativeqa in subsection a.6. overall, without diminishing the importance of narrativeqa, we believe that the limitations above warrant the development of a new dataset avg. words documents task items booksum @xcite 110,000 405 books 405 book summaries narrativeqa @xcite 50,000 1,572 (783 books, 789 screenplays) ## methodology our general approach was to test long-term reading comprehension through book scene reconstruction and recognition. since we want to encourage flexible memory representations rather than verbatim text memorization, instead of complete raw scenes, we used scene summaries. our overall pipeline is illustrated in figure 1 and is described below. ## data validation and baselines the primary goal of our dataset is to aid the development of long-term memory models. therefore, our goal in data validation is not ensuring that our dataset can not be solved using alternative methods (e.g. retrieval-based), but rather making sure that our questions 1) can not be directly solved by language models without long-term memory 2) are diagnostic of the model's memory capacity 3) accurately represent the material on which they are based (i.e. our questions should be solvable). ## limitations it is likely that many of our questions can be answered using relatively simple information retrieval approaches (ir), e.g. by reversing our data generation process and scanning each scene in a book with a gpt-like model. we would like to stress that this does not undermine the purpose of our study, similarly to how the existence of simple hard-coded solutions to some of the tasks in the long range arena challenge @xcite did not negate the impact of that work. we aimed to create a naturalistic dataset that could be used to train and evaluate language models with long-term memory capacity. it is possible that any such dataset can be solved with alternative information retrieval methods, since ir can be interpreted as having perfect memory (unrestricted access to the whole document from which the information should be retrieved). nevertheless, there is a need for non-irbased long-term memory models, and we believe that our dataset offers exactly what is needed to train and evaluate such models. data contamination. it is impossible to control which books are included in any given lm's training set, and being exposed to a given book in advance might aid performance @xcite . we do not claim to fully resolve the issue, but do take steps to ameliorate it by removing book titles and author names, changing the named entities, and basing questions on scene summaries, rather than on raw scenes. with these measures, we hope to make it harder for models to map books they are reading to something they might already know. additionally, our read-along questions give different answers depending on when they are asked. this makes it necessary for any model to rely on its memory to track the reading progress even if it was already exposed to the book before. in future work, it might be beneficial to paraphrase the books in our dataset to further mitigate data contamination. lastly, we do not directly show that our dataset will improve ultra-long-context llms or llms with long-term memory mechanisms. instead, we focused our energy and resources on validating our data and making sure that our tasks test what they were designed to test: information retention and access in texts of extreme length. in the future, we hope to see new llms developed and evaluated using our data. ## conclusion we have proposed a new reading comprehension dataset for extreme-long-context llm training and evaluation. based on full-length books, it offers higher average document lengths and an order of magnitude more examples than any of the available alternatives (e.g. @xcite ). we have conducted four data validation experiments, demonstrating that our data accurately reflects the source material and is diagnostic of long-term memory performance. additionally, our method allows us to further expand the dataset at a very low cost, making it feasible, for example, to label all books in the gutenberg corpus at a price realistic for many academic and industry organizations. we hope that in the future, the data we provide will aid in training and evaluation of ultra-long context llms. we also hope that our work inspires further research into creating high-quality datasets for language understanding on even more extreme time scales (millions of tokens and more), as ultimately, it is that timescale that is most representative of human linguistic experience.