Fantastic Questions and Where to Find Them: F airytale QA – An Authentic Dataset for Narrative Comprehension (2022)

Abstract: Question answering (QA) is a fundamental means to facilitate assessment and training of narrative comprehension skills for both machines and young children, yet there is scarcity of high-quality QA datasets carefully designed to serve this purpose. In particular, existing datasets rarely distinguish fine-grained reading skills, such as the understanding of varying narrative elements. Drawing on the reading education research, we introduce FairytaleQA, a dataset focusing on narrative comprehension of kindergarten to eighth-grade students. Generated by educational experts based on an evidence-based theoretical framework, FairytaleQA consists of 10,580 explicit and implicit questions derived from 278 children-friendly stories, covering seven types of narrative elements or relations. Our dataset is valuable in two folds: First, we ran existing QA models on our dataset and confirmed that this annotation helps assess models’ fine-grained learning skills. Second, the dataset supports question generation (QG) task in the education domain. Through benchmarking with QG models, we show that the QG model trained on FairytaleQA is capable of asking high-quality and more diverse questions.

## introduction reading comprehension is a complex, multidimensional cognitive process @xcite . question answering (qa) is fundamental for supporting humans' development of reading comprehen-sion skills, as questions serve as both instruments for evaluation and tools to facilitate learning. to achieve this goal, comprehension questions should be valid and reliable, meaning that all items are designed to cohesively assess comprehension rather than some other skills (e.g., text matching, paraphrasing, or memorization) @xcite . moreover, from the educational perspective, given that reading comprehension is a multicomponent skill, it is ideal for comprehension questions to be able to identify students' performance in specific sub-skills, thus allowing teachers to provide tailored guidance @xcite . however, creating a large and suitable set of questions for supporting narrative comprehension is both time-consuming and cognitively demanding. some researchers have proposed developing models to automatically generate questions or qa-pairs that satisfy the need for a continuous supply of new questions @xcite , which can potentially enable large-scale development of ai-supported interactive platforms for the learning and assessment of reading comprehension skills (e.g., @xcite ). however, existing datasets are not particularly suitable for training question generation (qg) models for educational purposes @xcite . this is primarily because the datasets are not typically structured around the specific dimensions of reading comprehension sub-skills, nor do they provide sufficient information on what sub-skills are tested. consequently, qg models built on these datasets only yield one single "comprehension" score without a more detailed breakdown of performance on comprehension sub-skills. this issue is compounded by the fact that many benchmarks rely on crowdsourced workers who may not have sufficient training or education domain knowledge needed to create valid questions in a consistent way. to bridge the gap, we constructed fairytaleqa, an open-source dataset focusing on comprehension of narratives, targeting students from kindergarten to eighth grade. we focus on narrative comprehension for two reasons. first, narrative comprehension is a high-level comprehension skill strongly predictive of reading achievement @xcite and plays a central role in daily life as people frequently encounter narratives in different forms @xcite . second, narrative stories have a clear structure of specific elements and relations among these elements, and there are existing vali-dated narrative comprehension frameworks around this structure, which provides a basis for developing the annotation schema for our dataset. we employed education experts who generated 10,580 question-answer pairs based on a collection of 278 fairytale stories for young readers, following evidence-based narrative comprehension frameworks @xcite . thereby, fairytaleqa contains questions that focus on several narrative elements and relations, increasing the validity and reliability of the assessment. in addition, fairytaleqa also contains both explicit questions that involve answers found explicitly in the text and implicit questions that require high-level summarization ( we show the utility of fairytaleqa through two benchmarking experiments. first, we used our data to train and evaluate state-of-the-art (sota) qa models and demonstrated that (1) fairytaleqa contains challenging phenomena for existing models, and (2) it can support finer-grained analysis on the different types of comprehension sub-skills, even for models trained on general qa datasets (narrativeqa @xcite ). we further calibrated model performances with human baseline, highlighting the most visible gap in models' reasoning capabilities on recognizing casual relationships and predicting event outcomes. second, we used fairytaleqa to power question generation and showed that the qg model trained on ours was more capable of asking diverse questions and generating questions with higher quality. ## fairytaleqa we developed the fairytaleqa dataset to address some of the limitations in existing benchmarks. our dataset contains 10,580 qa pairs from 278 classic fairytale stories. in the remainder of this section, we report the dataset construction process and its key statistics. ## baseline benchmark: question answering in the following sections, we present a couple of baseline benchmarks on both the question answering (qa) task and the question generation (qg) task with fairytaleqa. we leveraged both pretrained neural models and models fine-tuned on different qa datasets, including narrativeqa and our dataset, fairytaleqa. the baseline results show that our fairytaleqa demonstrates challenging problems to existing approaches, and those models fine-tuned on fairytaleqa can benefit from the annotations a lot to achieve significant performance improvement. we also report human performance by scoring one reference answer to the other. ## conclusion and future work in summary, we constructed a large-scale dataset, fairytaleqa, for the context of children's narrative comprehension. the dataset was generated through a rigorous labeling process with educational domain experts. this dataset has been helpful to support preliminary works on qg tasks @xcite and already enabled possibilities for new downstream ai-for-education applications @xcite . howerver, we acknowledge our work also has limitations that require future works to continue the exploration. as aforementioned, the human performance results for qa task are underestimated because they are obtained via cross-estimation between the two annotated answers. one possibility for future work is to conduct a large-scale human annotation to collect more answers per each question and then leverage the massive annotated answers to better establish a human performance evaluation. another avenue of future work is to leverage our dataset to detect and remediate social stereotypes and biases represented in story narratives -the bias analysis in the children storybook corpus has been an underexplored research topic for the ml community, but it has profound societal impacts on the soceity. through such analysis on our dataset, we may be able to answer "how do social stereotype and bias come into a child's mind?" in sum, there are many new research and application opportunities enabled by our fairytaleqa dataset, and we welcome researchers from both nlp and education communities to join our effort to continue this endeavor. we provided another qa performance decomposition based on explicit/implicit question types in input story section: you see from this that the sparrow was a truthful bird, and the old woman ought to have been willing to forgive her at once when she asked her pardon so nicely. but not so.the old woman had never loved the sparrow, and had often quarreled with her husband for keeping what she called a dirty bird about the house, saying that it only made extra work for her. now she was only too delighted to have some cause of complaint against the pet. she scolded and even cursed the poor little bird for her bad behavior, and not content with using these harsh, unfeeling words, in a fit of rage she seized the sparrow-who all this time had spread out her wings and bowed her head before the old woman, to show how sorry she was-and fetched the scissors and cut off the poor little bird's tongue. input answer: cut off the poor little bird's tongue.