Diversity Enhanced Narrative Question Generation for Storybooks (2023)

Abstract: Question generation (QG) from a given context can enhance comprehension, engagement, assessment, and overall efficacy in learning or conversational environments. Despite recent advancements in QG, the challenge of enhancing or measuring the diversity of generated questions often remains unaddressed. In this paper, we introduce a multi-question generation model (mQG), which is capable of generating multiple, diverse, and answerable questions by focusing on context and questions. To validate the answerability of the generated questions, we employ a SQuAD 2.0 fine-tuned question answering model, classifying the questions as answerable or not. We train and evaluate mQG on the FairytaleQA dataset, a well-structured QA dataset based on storybooks, with narrative questions. We further apply a zero-shot adaptation on the TellMeWhy and SQuAD1.1 datasets. mQG shows promising results across various evaluation metrics, among strong baselines.

## introduction question generation (qg), focusing on the questions derived from specific text passages or documents, plays an integral role in a wide array of domains. it improves question answering (qa) systems @xcite , enriches educational experiences @xcite , and enhances the engagement factor in chatbots @xcite . the effectiveness of qg tasks can be significantly improved by generating multiple questions, ensuring a broader, more comprehensive exploration of the content. the importance of generating and evaluating multiple questions becomes evident when we examine the creation process of qa datasets @xcite @xcite . traditional qa dataset creation typically involves 1 code: https://github.com/hkyoon95/mqg instructing annotators to create a pre-determined number of questions for a given context. recent qg research @xcite , however, tends to rely on automatic evaluation of semantic similarity with golden questions, often overlooking the potential for diverse aspects of questions. when generating multiple questions, diversity is a crucial aspect to consider. the diversity of questions can span several dimensions, including varied aspects of the context, different answer types, and different phrasings for essentially the same question @xcite . this diversity allows for a more comprehensive exploration of the context. the diversity of questions can be broadly measured based on the type of answers they require; explicit questions with answers that can be explicitly found in the reading materials, and implicit questions with answers that require deductive reasoning. the crafting of multiple questions, bearing in mind both diversity and alignment with reading materials, poses a cognitively demanding and time-consuming task for humans. one significant application of generating diverse and multiple questions is education. it has been observed that children can develop better reading comprehension skills at an early age by creating narrative questions themselves and being asked comprehension-related questions about storybooks @xcite . reading comprehension is an essential skill that requires learners to combine knowledge and reason about relations, entities, and events across a given context @xcite . consequently, a system that can generate diverse and multiple narrative questions can serve as a valuable enhancement to educational resources, aiding in student engagement and promoting a deeper understanding of study materials. recently, some researchers have attempted to generate multiple narrative questions. for educational applications, @xcite proposed to generate question-answer pairs with a three-step pipeline. as they use heuristic-generated answers to generate narrative questions most of their outcome is restricted to explicit questions. also, @xcite proposed to generate certain types of narrative questions and they tried to restrict the number of generated questions to a number of ground-truth questions, insisting that knowing question type distribution for each context is a subskill in education @xcite . we set these two approaches as our main baselines. to address the above challenges, we introduce a multi-question generation model (mqg) that generates diverse and contextually relevant questions by referencing questions from the same context. mqg is trained with maximum question similarity loss l m qs , which is designed to make the representation of reference questions and the representation of a target question similar. moreover, mqg employs a recursive generation framework, where previously generated questions are recursively fed back into the model as mqg is trained to output different questions from reference questions. same as our two baselines, mqg is trained and evaluated on the fairytaleqa dataset, which focuses on narrative comprehension of storybooks. this dataset is designed to provide high-quality narrative qa pairs for students from kindergarten to eighth grade (ages 4 to 14), and labeled questions as explicit or implicit. we adopt self-bleu @xcite to evaluate the diversity of generated questions. beyond diversity, to consider generated questions relevant to the context, we demonstrate the answerability evaluation model to assess whether the generated questions are answerable. we also evaluate on tellmewhy @xcite and squad1.1 @xcite datasets with zero-shot adaptation to further analyze the performance of mqg in different settings. differing from previous approaches, mqg successfully generates a substantial number of diverse and answerable narrative questions. the main contributions of this paper are summarized as follows. ## method in this section, we formalize the multi-question generation task and introduce our mqg. we first formulate our task and then explain how our model's training process incorporates a maximum question similarity loss l m qs . finally, we provide a detailed outline of our recursive generation framework. ## conclusion in this work, we extend the scope of answerunaware question generation to generate multiple diverse questions. we propose a novel framework that applies a maximum question similarity loss during training to promote question diversity, followed by a recursive generation process for further refinement. additionally, an evaluation model is introduced to verify the answerability of the generated questions. recognizing the essential role of narrative questions in education, we train and evaluate mqg accordingly. comprehensive experiments validate the efficacy of mqg across a variety of datasets, highlighting its potential utility in environments that demand diverse narrative questions.