Educational Multi-Question Generation for Reading Comprehension (2022)

Abstract: Automated question generation has made great advances with the help of large NLP generation models. However, typically only one question is generated for each intended answer. We propose a new task, Multi-Question Generation, aimed at generating multiple semantically similar but lexically diverse questions assessing the same concept. We develop an evaluation framework based on desirable qualities of the resulting questions. Results comparing multiple question generation approaches in the two-question generation condition show a trade-off between question answerability and lexical diversity between the two questions. We also report preliminary results from sampling multiple questions from our model, to explore generating more than two questions. Our task can be used to further explore the educational impact of showing multiple distinct question wordings to students.

## introduction automatic question generation (qg) is a wellestablished task in natural language processing. large generation models have had success producing answer-informed factual comprehension questions, where the intended answer is a span located in a passage @xcite @xcite . automatically generating factual questions from a passage can benefit students in a reading comprehension environment @xcite . however, the majority of question generation work has focused on generating a single question with an intended answer. in a student practice environment, however, it is valuable to have multiple wordings for the same question. this allows students to practice a concept multiple times without encountering identical language. this additionally allows a wording of the question to be held out for assessment. we propose a new question generation task, multi-question generation, which takes as input an intended answer and produces both (1) an initial question and (2) n reworded questions which maintain the semantic meaning of the original question while varying language used. although multiple questions can be generated about different concepts pertaining to an intended answer, we specifically aim to generate questions which assess knowledge of the same concept, varying only the language used in the question. another issue with current generation systems is the large overlap in words between the reading passage and generated question. we propose automatic metrics grounded in desirable properties of the generated set of questions. because we intend the questions to have the same intended answer, we measure both (1) whether a question answering (qa) model is able to produce the correct answer for each question @xcite and (2) the semantic similarity between the generated questions, measured using sbert @xcite . also, because we intend for the questions to have distinct wordings, we propose using a known n-gram overlap metric, paraphrase in n-gram changes (pinc), between pairs of questions @xcite . finally, because each generated question is tied to a passage, we propose using pinc to compare overlap between each question and the input passage. we report results using a variety of question generation conditions, including a paraphrase model, a qg model fine-tuned to generate two questions, and the use of decoding constraints to improve question wording diversity. our publicly-released code and generated questions can be used to ex- ## multi-question generation given the potential educational benefits that come from answering questions with different wordings, we propose multi-question generation, with the goal of producing multiple semantically similar, lexically diverse questions with the same intended answer. an intended answer and a passage are the input to the task while the multiple diverselyworded questions are the output. although an intended answer can have multiple concepts with which questions can be generated from, these multiple questions should assess the same concept. an example of this can be seen in ## experimental conditions we begin with the task of generating two questions (results of generating more than two questions can be seen in section 6). to approach this task, we leverage a high-quality neural question generation model, prophetnet @xcite . in order to generate multiple questions, we explore (1) transforming prophetnet's single question output into a paraphrased second question, (2) fine-tuning prophetnet to output two questions sequentially, and (3) sampling multiple times from prophetnet. all models use beam search with a beam size of 10 unless otherwise stated. for sampled results, we use nucleus sampling @xcite with @xmath0.95. all results are reported on the squad 1.1 development set @xcite . ## two-question generation results two-question generation results can be found in we observe that restricting the repetition of trigrams in the question generation increases the pinc score, which is expected as generating repeating trigrams is constrained. however, this comes at the cost of having a lower qa score. we also note higher qa scores for the first question compared to the second, meaning answerability might be less important when rephrasing the initial question. the drop in performance from qa1 to qa2 for 1qg+para is anticipated as the paraphrase model does not have access to the answer or context passage. however, surprisingly, we observe similar performance drops with 2qg models (in particular 2qg no question trigram). the gap in quality is increased when the pinc score between the questions is higher, indicating a tradeoff between pinc score and qa score. we also observe that restricting the trigrams from the context paragraph (2qg no context trigram) increases the pinc score with respect to the context paragraph as expected, but does so at a smaller cost to the qa score. lastly, we note an inverse relationship between inter-question pinc score and sbert similarity. this indicates that diversifying lexical content of questions may come at the cost of maintaining semantic similarity between the two questions. ## toward multi-question generation we next explore bridging the gap from two-question generation to multi-question generation. while the 2qg model was fine-tuned to produce two questions sequentially, we explore the extent to which sampling from this model can produce sets of more questions. we take the 2qg model and sample from it multiple times using nucleus sampling (@xmath1.95). we consider sets of 2, 4, 6, and 8 questions. ## future work and conclusion although automated evaluation metrics can measure the desirable properties of our two-question generation model outputs at scale, they are also limited. future work could include human evaluation metrics to measure the semantic quality and lexical diversity more robustly. future work should also explore using desirable question metrics in a reinforcement learning objective to produce higher quality questions, similar to previous work in abstractive summarization @xcite ) and text simplification @xcite . additionally, more advanced paraphrase systems, such as the syntax-aware system proposed in @xcite , could be leveraged for our task. this work can explore which syntactic exemplars can be leveraged to generate questions with varying syntactic structure. additionally, future work should also include teacher evaluation to collect education-specific feedback on sets of questions and our desirable question properties. this work can help better define what constitutes a good question and potentially uncover different automated metrics. future work can leverage our task to evaluate the educational impact of multiple diverse question wordings. multi-question generation can be integrated into a reading comprehension environment to test student reactions to a reworded question. generating multiple question wordings can fully test the students' reading comprehension and ability to apply information in new situations. our publicly-released pipeline has the potential to generate multiple wordings of the same questions to enrich educational resources at scale.