S tory S park QA : Expert-Annotated QA Pairs with Real-World Knowledge for Children’s Story-Based Learning (2024)

Abstract: Interactive story reading is common in early childhood education, where teachers expect to teach both language skills and real-world knowledge beyond the story. While many story reading systems have been developed for this activity, they often fail to infuse real-world knowledge into the conversation. This limitation can be attributed to the existing question-answering (QA) datasets used for children’s education, upon which the systems are built, failing to capture the nuances of how education experts think when conducting interactive story reading activities. To bridge this gap, we design an annotation framework, empowered by existing knowledge graph to capture experts’ annotations and thinking process, and leverage this framework to construct StorySparkQA dataset, which comprises 5, 868 expert-annotated QA pairs with real-world knowledge. We conduct automated and human expert evaluations across various QA pair generation settings to demonstrate that our StorySparkQA can effectively support models in generating QA pairs that target real-world knowledge beyond story content. StorySparkQA is available at https://huggingface.co/datasets/NEU-HAI/StorySparkQA.

## introduction interactive story reading is common in early childhood education, where teachers often sit together with preschool children, read storybooks, * work done when visiting northeastern university. despite the benefits of interactive story reading, teachers often struggle to appropriately conduct such interactive story reading with children because of multi-facet difficulties @xcite . specifically, such interactive story reading needs teachers to identify the knowl-edge of interest during storytelling, formulate the real-world knowledge piece they want to teach in mind ("what to ask"), then ask an engaging question ("how to ask") to children at the appropriate time ("when to ask"). in home settings, most parents also lack the educational expertise necessary to guide such educational conversations @xcite . meanwhile, today's parents often hardly maintain constant focus on their children due to the need to deal with other work and family chores at the same time @xcite . recently, ai-assisted storytelling systems (e.g. storybuddy @xcite , talemate @xcite , mathemyths @xcite ), have demonstrated utility in children's storytelling scenarios @xcite . these systems primarily utilize the verbal communication interface and advanced language models to support natural conversation with humans @xcite @xcite . nevertheless, existing ai-assisted storytelling systems are not without limitations. particularly, building on top of data resources with mostly extractive qa pairs (e.g., fairytaleqa @xcite ) -where the answers can be found directly in the story narrative -these systems fall short in helping teaching real-world knowledge beyond the story narrative @xcite , which is one of the main expectations of parents and teachers @xcite . we believe a promising approach to bridge this gap is to effectively and exhaustively collect education experts' knowledge, including their stepby-step thinking process as well as the appropriate qa pairs as final artifacts. nevertheless, to the best of our knowledge, no such data resources exist in the domain of children education. further, the collection of such data resources requires annotators to recall a comprehensive and systematic external knowledge range for a given story text, which is challenging even for education experts @xcite . as a result, this work aims to facilitate experts' large-coverage knowledge collection and data annotation, and build an expert-labeled, large-scale qa dataset to support story-based educational qa generation with tri-fold contributions: storysparkqa can benefit different research directions in the children education, particularly in better understanding domain experts' thinking process, and training models to generate story-based qa pairs infused with real-world knowledge, with the ultimate goal of broadening children's knowledge scope beyond story narratives that parents and teachers expect. in addition, we believe our annotation framework has the potential to be generalized to domain-specific tasks analogous to the real world that require structured external knowledge @xcite , such that clinicians use structured guidelines and knowledge for diagnosing @xcite . ## expert annotation framework to better understand experts' mental procedures during annotating qa pairs enriched with external real-world knowledge, we proposed a three-step qa pair annotation framework with interactive user interfaces (ui). particularly, considering the challenges facing annotators in recalling the comprehensive and systematical external knowledge for a given story text @xcite , our framework incorporates conceptnet, a large-scale realworld knowledge graph, to support experts' largecoverage knowledge collection. the workflow of our annotation framework is shown in figure 2 . ## benchmark experiment we benchmark the quality and usability of our storysparkqa on the qag task, which is required to meet the needs of teachers to guide children to learn some real-work knowledge during practical interactive story reading, as well as existing work of developing ai-assisted storytelling and reading systems @xcite @xcite . we conduct an automated evaluation, reported in section 5.1 to measure the semantic similarity of generated qa pairs with experts-annotated qa pairs, benchmarked with a t5-large model fine-tuned on storysparkqa and a set of robust llms. considering the limitation of automated evaluation in evaluating the educational appropriateness of generated qa pairs, we further conduct a human evaluation, reported in section 5.2, with children education experts. ## conclusion and future work in summary, we propose storysparkqa, an expert-annotated, external-knowledge-enriched qa dataset for children education, by leveraging a novel annotation framework to facilitate scalable expert annotations through structured external knowledge. we demonstrate the effectiveness of storysparkqa through an automated evaluation on various llms of generating qa pairs catering to teachers' needs and a human evaluation with children education experts. one possible future work is refining the qag pipeline structures and exploiting llms to generate qa pairs that align more closely with teachers' practical needs. another future direction involves using storysparkqa and language models to develop a human-ai collaborative education system (e.g., an interactive story reading system) @xcite , aiding parents and educators to formulate personalized questions during story readings, while addressing their language, knowledge, or time constraints. also, fine-tuning llms (e.g., llama 3) may lead to better performance on the qag task, which offers a future direction to refine models' capabilities in real-world tasks like children's education.