GANDALF : a General Character Name Description Dataset for Long Fiction (2021)

Abstract: This paper introduces a long-range multiple-choice Question Answering (QA) dataset, based on full-length fiction book texts. The questions are formulated as 10-way multiple-choice questions, where the task is to select the correct character name given a character description, or vice-versa. Each character description is formulated in natural text and often contains information from several sections throughout the book. We provide 20,000 questions created from 10,000 manually annotated descriptions of characters from 177 books containing 152,917 words on average. We address the current discourse regarding dataset bias and leakage by a simple anonymization procedure, which in turn enables interesting probing possibilities. Finally, we show that suitable baseline algorithms perform very poorly on this task, with the book size itself making it non-trivial to attempt a Transformer-based QA solution. This leaves ample room for future improvement, and hints at the need for a completely different type of solution.

## introduction comprehending and analyzing fictional stories plays an important part in human culture @xcite . in particular, book studies is a commonly applied educational tool used to both probe and enrich students' language comprehension skills @xcite . ideally, these book studies require the students to reason about notions spread out over hundreds of pages of text. by contrast, methods and datasets for machine reading comprehension (mrc) have predominantly been limited to comparably short texts, with evaluations often focusing on various forms of short-text natural language understanding (nlu) tasks, where the input is limited to a small number of sentences. examples of such tasks include textual similarity @xcite , sentiment analysis @xcite , question answering @xcite , inference and, entailment @xcite , etc. there is an ongoing debate regarding nlu evaluation datasets, spurred by the acclaimed superhuman results on benchmarks such as glue @xcite and superglue @xcite . the critique points out several inherent issues with these benchmarks @xcite , such as dataleakage @xcite , and that models are sometimes able to "cheat" by exploiting spurious linguistic cues in the data @xcite . proposed mitigation methods include the use of adversarially hard datasets @xcite , and taking a more rigorous approach to dataset design @xcite . adding our voice to this discussion, we point out an additional limitation to near all these prior datasets, namely how limited they are in the amount of text that is used per question (see section 2 for noteable exceptions). since datasets arguably drive the direction of research, we find it backward and stagnating to only create evaluation tasks suitable for the current paradigm of fixed-size transformer models @xcite ). therefore, we find it equally important to create datasets meant to task methods on long-text comprehension. thus, we present gandalf (a general character name description dataset for long fiction), a full-length book question answering dataset focused on the novel task of character description recognition. this 10-way multiplechoice task asks the model to read an entire book, and then identify the correct character name to a given character description, or vice-versa. in total, we supply 20,000 questions, with a 50/50 split between predicting a name given a description, and predicting the description given a character name. the manually created descriptions are all expressed in natural text and contain a mixture of traits, important events, and relationships to other characters. a schematic illustration of gandalf is provided in figure 1 . taking into account the current discourse concerning datasets, we implement a simple namereplacement system that counters potential data leakage. this system also enables for straightforward implementations of probing tasks by controlling for example gender or nationality. finally, we perform experiments intended to measure a model's ability to cheat on gandalf, by answering the questions without relying on the book data. the full dataset is available at: github.com/ freddefrallan/gandalf-dataset ## related work there exists a rich body of literature for various mrc and nlu-related tasks. examples of previous mrc datasets that also formulate their questions as multiple-choice include: race @xcite , openbookqa @xcite , mul-tirc @xcite and race-c @xcite . for each of these datasets, the combined length of each question and its provided information often fall below 50 sentences, and for some datasets significantly less. related work which specifically utilize books as their universe of knowledge include children's book test (cbt) @xcite , booktest @xcite , and comics @xcite . these three datasets all utilize cloze-style answers from sequences with up to 21 sentences. narrativeqa (s @xcite provide questions based on full-length stories, with an average of ~60k words per story. the answer format is free-text and is hence evaluated using text similarity metrics such as bleu @xcite and meteor @xcite . in contrast to previous work, the books within gandalf contain on average ~150,000 words and ~8,000 sentences. each question is classified into its level of referential complexity (see section 3.2), which when combined with the multiplechoice accuracy results in an informative evaluation metric. to the best of our knowledge, gandalf is therefore not only the longest current mrc dataset, but also the only current mrc dataset which provides these insights during evaluation. we note that there are other types of benchmarks and tasks that requires long context, such as ## the gandalf dataset gandalf contains 20,000, 10-way multiplechoice questions, formulated from 10,000 manually created character descriptions of 2,500 characters from 177 books. for each question, the relevant book is provided, and the task is to either predict the correct name given a description or predict the correct description given a character name. for brevity we refer to these two different settings as: desc2name and name2desc. see figure 1 for a visual example. additionally, gandalf contains a simple anonymization system, where names are replaced in both the books and the descriptions. this acts a mitigation to potential data leakage and also allows for easy creation of future probing tasks. ## creating the dataset the creation of the gandalf dataset is a process that in large is guided by avoiding copyright infringement (see section 7.3). figure 4 along with the following list gives an overview of the creation process, upon which the following subsections describe each step in more detail. ## experiments to the best of our knowledge, there has been no proposed method that is capable of handling naturally formulated questions along with the long-range format of gandalf. we are therefore limited in what experiments we can perform without proposing a new solution, which is deliberately not within the scope of this paper. therefore, our experiments can be interpreted more as testing the robustness of our dataset, while they also demonstrate that the problem is non-trivial. all three of our approaches are based on selecting the alternative which maximizes the probability of the description being stated after the character name. the first method is a traditional word-based statistical method, and the other two utilize modern language models. the statistical model acts as a simple information-retrieval baseline, and a way to measure potential noise that could be introduced during name replacement. the language model approaches are intended as an attempt to utilize off-the-shelf nlu technology to solve the problem both the intended way, and without the book texts. finally, we saw no increase in performance after fine-tuning the language models, the reported results are hence attained from directly applying pre-trained checkpoints. 2 these experiments therefore include all the questions of gandalf as test data. ## future work & discussion systems specialized towards a single task attain poor generalization abilities, and hence demonstrate low levels of intelligence @xcite . as ai researchers, our main interest is therefore not a method capable of only solving the character recognition task of gandalf. rather, our ultimate goal is methods capable of handling and generalizing over a wide range of tasks, domains, and modalities. current models perform well over the different tasks included in benchmarks such as glue and superglue, but they are yet capable of handling both long and short texts. therefore, we think the time is right for extending our current evaluation benchmarks to include tasks covering large bodies of text. gandalf with its potential extensions and probing tasks is hence our first contribution to such a set setup. we note the relatively weak performance of the models tested in our experiments. it is probable that better performance can be attained using alternative techniques, such as the dense passage retriever @xcite or retrieval augmented generation @xcite . we leave this for future work. ## conclusion this paper has introduced the gandalf dataset, which constitutes a unique challenge for machine reading comprehension by requiring long-range retention capabilities while simultaneously being able to ignore irrelevant information. we have introduced the character description task with its two variations (desc2name and name2desc), and argued that this task formulation provides unique opportunities for probing variations without changing the nature of the task itself. we also provide a number of baseline results on the dataset using both simple and more advanced methods, and the results clearly demonstrate the challenging nature of the dataset. we believe that this dataset and task provides a welcome addition to existing machine reading comprehension benchmarks, in particular at a time when we start to see superhuman performance on existing datasets, with an apparent risk of models starting to optimize for a specific dataset rather than for general reading comprehension abilities. the gandalf dataset, by contrast, is extremely challenging with minimal risk of data leakage and consequently low risk of models cheating on the tasks.