S kill QG : Learning to Generate Question for Reading Comprehension Assessment (2023)

Abstract: We present SkillQG: a question generation framework with controllable comprehension types for assessing and improving machine reading comprehension models. Existing question generation systems widely differentiate questions by literal information such as question words and answer types to generate semantically relevant questions for a given context. However, they rarely consider the comprehension nature of questions, i.e., the different comprehension capabilities embodied by different questions. In comparison, our SkillQG is able to tailor a fine-grained assessment and improvement to the capabilities of questions answering models built on it. Specifically, we first frame the comprehension type of questions based on a hierarchical skill-based schema. We then formulate SkillQG as a skill-conditioned question generator. Furthermore, to improve the controllability of generation, we augment the input text with skill-specific question focus and knowledge, which are constructed by iteratively prompting the pre-trained language models. Empirical results demonstrate that SkillQG outperforms baselines in terms of quality, relevance, and skill-controllability while showing a promising performance boost in downstream question answering task.

## introduction question generation (qg) systems aim to generate natural language questions conditioned on a text passage. as a dual task of question answering (qa), qg is widely applied to create questionanswer pairs as data augmentation for qa training @xcite @xcite , help chatbots continue a conversation with human users (mostafazadeh et al., 2016; † canada cifar ai chair. ‡ corresponding authors. context: the princess climbed out the window of the high tower and climbed down the south wall when her mother was sleeping. she wandered out a good way. finally, she went into the forest where there are no electric poles. q1 q1 q1: who climbed out of the castle? a a a: princess. q2 q2 q2: why did the princess climb out when her mother was sleeping? a a a: in case of being caught. q3 q3 q3: what would happen if her mother was not sleeping? a a a: the princess would be caught soon. figure 1 : example questions that require different comprehension capabilities to answer. @xcite , and facilitate reading assessment @xcite . most prior qg research has typically focused on generating factoid-based questions that are relevant to a piece of the fact of a single sentence @xcite @xcite . recently, motivated by building the read comprehension (rc) systems that are competent in understanding and reasoning @xcite @xcite , there is an increasing interest in developing systems that are capable of generating deep questions @xcite @xcite . however, these works generate diverse questions by relying on different surface-level mentioned information @xcite and consider primarily simple connections between two facts in the context (e.g. bridge and intersection). less explored have been more facts and the deeper comprehension types between them @xcite , such as analysis of discourse relations @xcite , a thorough evaluation of stated arguments, and deduction of the high-level semantics @xcite . as shown in figure 1 , q 1 asks for the mentioned facts in stories (e.g. "the princess climbed out the window of the high tower"), whereas q 2 and q 3 ask for a deep connection about the events (causal relation in q 2 and future prediction in q 3 ). we argue that generating questions with deeper comprehension brings two major benefits: (i) compared with factoid-based qg models, it reflects higher cognitive skills and requires an in-depth understanding of the input text and reasoning over relevant contexts, better imitating how human intelligence embodies the application and integration of skills; (ii) compared with existing deep qg models, it can help build more controllable questions with different comprehension types rather than literal information such as answer types. based on such questions, we can better identify the downstream performance of qa systems in specific comprehension types, and assess their corresponding intrinsic ability, further allowing us to provide tailored guidance to them and improve training efficiency. in this paper, we propose skillqg: a question generation framework with controllable comprehension types. specifically, we define the comprehension types as five skill dimensions ordered by cognitive complexity: remember, under-stand, analyze, create, and evaluate, which are inspired by bloom's taxonomy @xcite , an educational schema by which teachers structure a curriculum to ensure that learners possess the necessary abilities before progressing to more complex tasks. based on the definition, we can better differentiate questions from cognitive demands than previous surface-level information and formulate skillqg as question generation conditioned on the given comprehension skill. furthermore, to improve the specificity of generating questions with a certain comprehension skill, we devise a set of prompts based on the indicative words and question templates of bloom's taxonomy. using these prompts to iteratively elicit chain-of-thought reasoning of pre-trained language model (plm), we explicitly generate question focuses (what to ask about) and skill-specific knowledge (how to ask it) to augment the input context. finally, to evaluate the skillqg framework, we introduce evaluation protocols covering question content quality, skill controllability, and downstream qa performance improvement when incorporating the generated questions as additional training data. our experimental results show that skillqg can produce more relevant and skillcontrollable questions compared to baseline qg models, and boost the qa performance significantly. ## methodology in this section, we elaborate our skillqg for generating skill-infused questions. specifically, we first define the comprehension types of questions as a 5-dimensional skill schema, which is drawn upon bloom's taxonomy @xcite of research in cognitive science and describes the cognitive load of different levels of topics or samples. based on this schema, we categorize the questions into different comprehension skills, regarding skillqg as a conditional generator given a skill. furthermore, to improve the controllability of the skill-infused questions, we adapt the indicative words and templates of bloom's taxonomy as a set of prompts to discover question focuses and skill-specific knowledge by prompting plm iteratively. finally, these question focuses and knowledge text act as auxiliary inputs to steer the question generator. ## experiments datasets. we employ the official train and dev splits of fairytaleqa dataset @xcite to train our skillqg. this dataset, focusing on narrative comprehension of english text for both machines and young children, is annotated with seven fine-grained skills comprised of character, setting, action, feeling, causal relationship, outcome resolution, and prediction. its annotation process is supervised by three experts in literacy education and its categorization of questions is based on prior educational research (paris and paris, 2003) so that we can easily match the samples of the fairytaleqa dataset with our defined skill schema. baselines. we compare skillqg to the following two types of qg baselines. the first type is typically trained without the knowledge input, including nqg++ @xcite , and qag @xcite . the other is knowledge-augmented generators consisting of csqg @xcite and cqg @xcite , which retrieve external knowledge from knowledge bases or generate knowledge with another model and regard the knowledge as extra context to generate questions. ## related work deep question generation. previous qg systems mainly generated factoid-based questions by a sequence-to-sequence model @xcite , a plm @xcite , or a graph-based architecture @xcite . recent-emerged qg models aimed at generating questions that require deep reasoning. on the one hand, @xcite proposed to generate difficulty-controllability questions through step-by-step rewriting, while @xcite decoded multi-hop questions by a soft template. on the other hand, @xcite and @xcite devised educational question generators to facilitate the assessment of children's literacy. our skillqg is inspired by their fine-grained analysis but driven by the motivation that generating questions with deep comprehension is beneficial to qa training. more recently, @xcite charted a new question ontology, but they focused on constructing diversified open-ended questions from the specified question types. knowledge-augmented generation. although explicit knowledge generation has been explored in natural language understanding @xcite , similar research on natural language generation @xcite , especially for qg is relatively rare @xcite . @xcite retrieved knowledge triplets from concept-net @xcite to enhance the qg models, while @xcite adopted a graph attention networks (gat) @xcite to capture focuses for question generators. we considered the lessons of these works and extend the knowledge source with pre-trained language models. ## conclusion existing qg systems focus on the literal nature of questions and rarely consider the comprehension types of the generated questions. to better assess and improve machine reading comprehension models, we propose skillqg to generate questions with controllable comprehension types. besides, we engage the question focus and specific knowledge to improve the controllability of generation. empirical results show that skillqg outperforms baselines while achieving a significant performance boost in downstream qa training.