Few-shot Question Generation for Reading Comprehension (2024)

Abstract: According to the internationally recognized PIRLS (Progress in International Reading Literacy Study) assessment standards, reading comprehension questions should require not only information retrieval, but also higher-order processes such as inferencing, interpreting and evaluation. However, these kinds of questions are often not available in large quantities for training question generation models. This paper investigates whether pre-trained Large Language Models (LLMs) can produce higher-order questions. Human assessment on a Chinese dataset shows that few-shot LLM prompting generates more usable and higher-order questions than two competitive neural baselines.

## introduction given the importance of asking questions for effective learning @xcite @xcite , there has been extensive effort in developing automatic question generation (qg) models to produce high-quality questions for reading materials in educational systems @xcite . through automatic creation of pedagogical and assessment material, qg benefits teachers by reducing their workload. it also levels the playing field for students, providing them with instant and free access to questions for review and practice. according to pirls (progress in international reading literacy study), reading comprehension questions should require not only information retrieval, but also higher-order processes such as inferencing, interpreting and evaluation @xcite . however, existing qg benchmarks such as squad @xcite mostly focus on factoid short-answer questions. there is therefore a dearth of publicly available training data for the more challenging types of questions @xcite -those requiring inference, process description retrieval focus on and retrieve explicitly stated information inferencing make straightforward inferences integrating interpret and integrate ideas and information evaluation evaluate and critique content and textual elements ## previous work early qg approaches mostly relied on heuristics, linguistic templates and rules @xcite . with the availability of large-scale datasets, qg began to be formulated as a sequence-to-sequence generation task. an encoder-decoder architecture with a global attention mechanism was found to be effective @xcite , but can be further improved with transformer-based approaches @xcite , and fully finetuned language models (lm) @xcite . answer-agnostic qg can be performed via joint question and answer generation (qag) @xcite . a qag model based on fine-tuning encoder-decoder lms produces high-quality questions @xcite , but has not been evaluated in terms of question type. there have been a few qg studies on llms in the education setting. on a textbook dataset, few-shot prompting with gpt-3 was able to generate human-like questions ready for classroom use @xcite . a similar approach with in-structgpt achieved an adherence rate between 67% and 69% for generating 9 question types @xcite . a fine-tuned version of chatgpt was able to generate questions that are competitive with human ones in terms of readability, correctness, coherence and engagement @xcite . it remains unknown how these approaches compare to off-the-shelf neural qg models in terms of generating higher-order questions. ## evaluation metric to accurately evaluate the utility and nature of the generated questions, manual assessment is neces-sary since automatic methods cannot yet reliably determine usability and pirls question types. ## approach we adopted the answer-agnostic setting for qg, since the target answer is not always found within the input text. the input is a chinese text without any specified answer span. ## dataset our evaluation data was drawn from the dev set of dureader_robust @xcite , a widely used chinese q&a ## agreement the two assessors agreed 85.0% of the time in the 3-way classification on usability (section 3.1), leading to a kappa of 0.739, a "substantial" level of agreement @xcite . in terms of question types, the two human assessors agreed in 93.5% of the cases, yielding a kappa of 0.861, at the "almost perfect" level of agreement (landis and @xcite the most common disagreement (19 cases) is between retrieval and inferencing, typically in judging whether a paraphrase deviates sufficiently from the original expression to require inference. the two assessors also disagreed in 9 cases on whether the answer must be derived from different parts of the passage (integrating) or from just a single sentence (inferencing). ## conclusion higher-order questions are important for assessment in reading comprehension. however, there is a lack of publicly available datasets of these challenging questions in languages other than english. this paper has presented the first study on automatic question generation (qg) for reading comprehension based on pirls, assuming no or minimal supervision. experiments on chinese passages show that zero-shot llm produces more usable and more higher-order questions than two competitive off-the-shelf neural qg models, and few-shot prompting further improves the performance. in future work, we plan to investigate tailored prompts for producing the different pirls question types, and to construct a chinese dataset of higherorder questions for fine-tuning an llm.