It is AI ’s Turn to Ask Humans a Question: Question-Answer Pair Generation for Children’s Story Books (2022)

Abstract: Existing question answering (QA) techniques are created mainly to answer questions asked by humans. But in educational applications, teachers often need to decide what questions they should ask, in order to help students to improve their narrative understanding capabilities. We design an automated question-answer generation (QAG) system for this education scenario: given a story book at the kindergarten to eighth-grade level as input, our system can automatically generate QA pairs that are capable of testing a variety of dimensions of a student’s comprehension skills. Our proposed QAG model architecture is demonstrated using a new expert-annotated FairytaleQA dataset, which has 278 child-friendly storybooks with 10,580 QA pairs. Automatic and human evaluations show that our model outperforms state-of-the-art QAG baseline systems. On top of our QAG system, we also start to build an interactive story-telling application for the future real-world deployment in this educational scenario.

## introduction there has been substantial progress in the development of state-of-the-art (sota) questionanswering (qa) models in the natural language processing community in recent years @xcite @xcite . however, the opposite of qa tasks-question-answer generation (qag) tasks that generate questions based on input text-is yet underexplored. we argue, being able to ask a reasonable question is also an important indicator whether the * † equal contributions from the first authors: yaob@rpi.edu, dakuo.wang@ibm.com; work was done while mo was at ibm research. * corresponding authors. ## pre-processing fairytaleqa dataset the released fairytaleqa contained 10,580 qa-pairs from 278 books, and each question comes with a label indicating the narrative element(s)/relation(s) the question aims to assess. we split the dataset into train/validation/test splits with 232/23/23 books and 8,548/1,025/1,007 qa pairs. the split is random, but the statistical distributions in each split are consistent. ## question answer generation system architecture there are three sub-modules in our qa generation (qag) pipeline: a heuristics-based answer generation module (ag), followed by a bartbased @xcite question generation module (qg) module fine-tuned on fairytaleqa dataset, and a distilbert-based @xcite ranking module fine-tuned on fairytaleqa dataset to rank and select top n qa-pairs for each input section. the complete qag pipeline of our system is shown in figure 2 . ## evaluation we conduct both automated evaluation and human evaluation for the qag task. the input of the qag task is a section of the story (may have multiple paragraphs), and the outputs are generated qa pairs. unlike qa or qg tasks that each input corresponds to a single generated output no matter what model is used, the qag task does not have a fixed number of qa-pairs to be generated for each section. besides, various qag systems will generate different amounts of qa-pairs for the same input content. therefore, we carefully define an evaluation metric that is able to examine the quality of generated qa-pairs over a different amount of candidate qa-pairs. the comparison is on the validation and test splits of fairytaleqa. in addition, we also include the recently published probably-asked questions (paq) work as a second baseline system @xcite . paq dataset is a semi-structured, very large scale knowledge base of 65m qa-pairs. paq system is an endto-end qa-pair generation system that is made up of four modules: passage scoring, answer extraction, question generation, and filtering generated qa-pairs. the paq system is trained on the paq dataset. it is worth pointing out that during the end-to-end generation process, their filtering module requires loading the complete paq corpus into memory for passage retrieval, which leads us to an out-of-memory issue even with more than 50g ram. 2 in comparison, our qag system requires less than half of ram in the fine-tuning process. in ## conclusion and future work in this work, we explore the question-answer pair generation task (qag) in an education context for young children. leveraging a newly-constructed expert-annotated qa dataset built upon childoriented fairytale storybooks (fairytaleqa), we implemented a qa-pair generation pipeline which, as observed in human and automated evaluation, effectively supports our objective of automatically generating high-quality questions and answers at scale. to examine the model's applicability in the real world, we further built an interactive conversational storybook reading system that can surface the qag results to children via speech-based interaction. our work lays a solid foundation for the promising future of using ai to automate educational question answering tasks. in the future, we plan to re-cruit educational experts to evaluate the educational efficacy of the qa-pairs as an additional evaluation dimension. another future direction is to develop a context-aware multi-turn qag system grounded by the story narratives (similar to ( li et al., 2021) ), where the generation of a new turn of qa is conditioned on previous generations as well as the book, so that it can enable new automated dialogue systems in the education setting.