Improving Reading Comprehension Question Generation with Data Augmentation and Overgenerate-and-rank (2023)

Abstract: Reading comprehension is a crucial skill in many aspects of education, including language learning, cognitive development, and fostering early literacy skills in children. Automated answer-aware reading comprehension question generation has significant potential to scale up learner support in educational activities. One key technical challenge in this setting is that there can be multiple questions, sometimes very different from each other, with the same answer; a trained question generation method may not necessarily know which question human educators would prefer. To address this challenge, we propose 1) a data augmentation method that enriches the training dataset with diverse questions given the same context and answer and 2) an overgenerate-and-rank method to select the best question from a pool of candidates. We evaluate our method on the FairytaleQA dataset, showing a 5% absolute improvement in ROUGE-L over the best existing method. We also demonstrate the effectiveness of our method in generating harder, “implicit” questions, where the answers are not contained in the context as text spans.

## introduction reading comprehension is crucial in assessing students' language learning ability and complex reasoning skills. comprehending and interpreting stories such as fairy tales, with specific emphasis on narratives, foster early intellectual and literacy development in children @xcite . asking suitable educationalfocused questions can help students understand the context of the fairy tales better and inspire their interests @xcite @xcite . however, constructing suitable questions at scale is hard since it is both time intensive and cognitively challenging @xcite . researchers have developed models that can automatically generate questions or question-answer pairs to meet the demand for a large pool of relevant questions @xcite . these advances can potentially facilitate the development of artificial intelligence (ai)-supported learning platforms to help students develop reading comprehension skills @xcite . prior work on question generation in educational applications can be broadly classified into two categories: answer-aware, which is the focus of our current work, and answer-unaware (see @xcite for a feasibility study), depending on whether the desired answer is given or not. for answer-aware question generation, the goal is to build an ai-based system to generate a question given both the context and the answer @xcite . the context can be any text segment, from a few sentences to a possibly long document, that provides background information on which the question is grounded in. the answer is a short span of text that is either part of the context (explicit) or not part of the context but can be inferred from the context (implicit). more specifically, in answeraware question generation, the question generation system is trained using the context-answer pairs as input and the question as the output @xcite . see section 2 for a detailed discussion on related work. a key challenge in answer-aware question generation is that there are often multiple relevant questions for a given context-answer pair. existing question generation systems are limited in identifying which questions human educators would prefer from multiple relevant ones. selecting the top question from multiple relevant and diverse question candidates is challenging. for a question generation system to perform this challenging task, it needs to be able to both generate diverse and valid question candidates and also accurately rank and select the top question. to generate diverse question candidates, a question generation system needs to be trained on multiple different relevant questions for a given context-answer pair. to accurately select the top question, a question generation system needs to learn to rank the question candidates by matching the preferences of human educators. we incorporate both of these ideas in our proposed methods in this work. ## methodology in this section, we first introduce the problem setup for question generation on fairytaleqa @xcite . we then detail our question generation approach, building upon the baseline of fine-tuning a language model, by adding our data augmentation method to augment the training set with diverse questions, followed by our over-generate-and-rank method to select the top question from the diverse question candidates generated. ## experimental evaluation in this section, we describe the experimental setup to validate the effectiveness of our question generation methods. ## results and discussion overall performance. we report the average rouge-l across all test questions in the fairy-taleqa dataset for all question generation methods in ## conclusions and future work in this paper, we proposed methods for improving automated answer-aware reading comprehension question generation by generating diverse question candidates and ranking them to align with human educator preferences. first, we proposed a data augmentation method that augments the training dataset with diverse questions obtained from a larger language model. second, we proposed an overgenerate-and-rank method with two choices of ranking criterion, perplexity-based ranking and distribution matching-based ranking. the latter learns to rank the generated candidate questions to select ones that are closer to human-written questions. we conducted extensive experiments on the fairytaleqa dataset to validate the effectiveness of our methods showing that our best method provides an absolute improvement of 5% in rouge-l over the current state-of-the-art on this dataset. we also showed that our methods are significantly better than baselines in generating harder questions whose answers are not directly present in the context as text spans and have to be inferred. there are several directions for future work. first, we can experiment with other data augmentation methods, e.g., by fine-tuning the base language model by weighting synthetically-generated questions according to their rouge-l scores with respect to the ground truth question. second, we can explore the use of chain-of-thought @xcite or self-ask @xcite to prompt the large language model in our data augmentation method. third, we can experiment with other ranking objectives, such as ones using the bradley-terry model @xcite or ones using reinforcement learning with human feedback framework @xcite , to select the best questions that are aligned with human preference. fourth, we can apply our methods to other question generation scenarios that require reasoning, such as logistical questions in online course discussion forums @xcite , to help instructors anticipate common student questions.