Question Generation for Reading Comprehension Assessment by Modeling How and What to Ask (2022)

Abstract: Reading is integral to everyday life, and yet learning to read is a struggle for many young learners. During lessons, teachers can use comprehension questions to increase engagement, test reading skills, and improve retention. Historically such questions were written by skilled teachers, but recently language models have been used to generate comprehension questions. However, many existing Question Generation (QG) systems focus on generating extractive questions from the text, and have no way to control the type of the generated question. In this paper, we study QG for reading comprehension where inferential questions are critical and extractive techniques cannot be used. We propose a two-step model (HTA-WTA) that takes advantage of previous datasets, and can generate questions for a specific targeted comprehension skill. We propose a new reading comprehension dataset that contains questions annotated with story-based reading comprehension skills (SBRCS), allowing for a more complete reader assessment. Across several experiments, our results show that HTA-WTA outperforms multiple strong baselines on this new dataset. We show that the HTA-WTA model tests for strong SCRS by asking deep inferential questions.

## introduction reading is an invaluable skill, and is core to communicating in our digital age. reading also supports other forms of development; when children read, it sharpens their memory, and improves social skills @xcite . yet, statistics show that one out of five children in the u.s. face learning difficulties @xcite , especially in reading @xcite . the coronavirus pandemic beginning in 2020 had a huge impact on the early reading skills of many children, and threatens to leave a lasting impact on a whole generation of young readers @xcite . the pandemic forced many children to learn online, putting in sharp relief the need for effective online education platforms. in particular, reading games have become popular, and can help fill the gap when teachers cannot read in person with students. these platforms present students with short passages and associated comprehension questions. these questions are key to assessing a reader's comprehension of a passage, and can also enhance learning @xcite . but, writing diverse and engaging comprehension questions is a nontrivial task. teachers need to generate new comprehension questions whenever they incorporate new text into a curriculum. new text helps to keep material fresh and topical, and can allow teachers to customize lessons to the interests of a particular student cohort. after finding such custom reading material, teachers must write new comprehension questions to evaluate several reading aspects of comprehension (e.g. understanding complex words, recalling events, etc.). thus, to improve the educational process, and lighten the load on teachers, we need tools to automate question generation (qg): the task of writing questions for a given passage. generated questions can be either inferential or literal (extractive) questions. literal questions can be answered using only information stated in the text, whereas inferential questions require additional information or reasoning. previous works focused on this aspect of the questions in reading comprehension and discarded the comprehension skills (e.g. close reading, predicting, figurative language, etc.) @xcite . we take inspiration from continual learning @xcite , which orders a set of learning tasks to improve model performance. we begin by training a model on the general task of qg (how to ask: hta), and follow with our task of interest: generating a targeted question of a particular type (what to ask: wta). this paper focuses on the generation of questions for story-based reading comprehension skills (sbrcs), which are varied and cover many aspects of reading comprehension. we create a qg dataset for sbrcs 1 . although our aim in creating this dataset is to enrich educational applications, this dataset can be considered as a source for general qg and question answering (qa) systems in nlp. our focus here is to build a question generator without answer supervision as the case in a reallife application, where a story only will be given as input. this is a challenging task, as many different questions can be generated from a story when there is no answer supervision. qg with answer supervision is another prevalent research line in the literature @xcite @xcite . the contributions in this work are as follows: the rest of the paper is structured as follows. in the next section, we present an overview of the literature work. in section 3, we describe how we built our dataset. section 4 describes the proposed methodology. the experimental setting is presented in section 5. the results and the analysis are presented in section 6. finally, we draw some conclusions and possible future work for this study. ## related works qg has progressed rapidly due to new datasets and model improvements. many different qg models have been proposed, starting for simple vanilla sequence to sequence neural networks models (seq2seq) @xcite @xcite to the more recent transformer-based models @xcite @xcite @xcite . some qg systems use manual linguistic features in their models @xcite @xcite , some consider how to select question-worthy content @xcite @xcite , and some systems explicitly model question types @xcite @xcite . the last group focused only on generating questions that start with specific interrogative words (what, how, etc.). qg has been used to solve many real-life problems. for example, qg in conversational dialogue @xcite @xcite where models were taught to ask a series of coherent questions grounded in a qa style, qg based on visual input @xcite @xcite , and qg for deep questions such as mathematical, curiosity-driven, clinical, and examinationtype questions @xcite @xcite . ## data despite the recent efforts for building reading comprehension qa datasets, to the best of our knowledge, none of the available datasets explored sbrcs. questions in previous datasets ask only either inferential or literal questions from a given passage/story. @xcite , developed questions with general reasoning types based on text from news and blogs (e.g. quora). we believe that those texts sources are not rich enough to examine reasoning skills. advanced reasoning skills (e.g. figurative language) are usually used in children's stories to assess comprehension skills. additionally, we use a extensive set of reading comprehension skills that deeply evaluates the abilities of the readers (e.g. imagination skill by visualizing). in the following, we will show how we built our dataset. ## methodology given the fact that including more data in a reading comprehension system is important for gen-eralization @xcite , and given that our created dataset has the sbrcs which are missed in previous datasets, we propose a two-steps method to generate skillrelated questions from a given story: hta followed by wta. hta teaches the model the typical format for comprehension questions using large previously released datasets. we use two well-known datasets, squad @xcite and cos-mosqa @xcite . in appendix a.3, we add more details on both of these datasets. these previous datasets are not annotated with the question types outlined in section 3.1, so the hta phase allows us to take advantage of those datasets. wta guides the model to generate questions to test the specific comprehension skills enumerated in section 3.1. thus, in hta, we train (fine-tune) a model on large qg datasets, and then, we further train the model to teach the model what to ask (wta). for the generation model, we use the pre-trained text-to-text transfer transformer t5 @xcite , which closely follows the encoder-decoder architecture of the transformer model @xcite . t5 is a sota model on multiple tasks, including qa. ## results and analysis regarding the generated questions type, in the one-step model performs similarly to the baselines, although it has been trained using the t5 model and on all three datasets. this may be due to the fact that we did not include the skill name in the encoder, which guides the model to generate skill related questions. to better understand the differences between the outputs of one-step and hta-wta models, we used human evaluation. this evaluation is to assess the quality of the generated question in terms of 1. answerability (ay), 2. fluency (fy), and 3. grammaticality (gy) categories, following @xcite ; @xcite . we include these three criteria as questions may have high fluency and grammaticality scores, but not be answerable. we select a sample of 110 story-question pairs from the test dataset, for both models. then, we perform a human evaluation using crowdworkers on amazon mechanical turk. we use a "master" qualification criteria to restrict the participation of workers in our evaluation study to those who have a high historical hit accuracy, and workers are required to be located in an english speaking country. each hit was answered by three workers. each worker needs reads the story, and provides ratings (1-5, low to high) for the generated questions, and the three criteria. this result shows that, in addition to using the skill name token to control the skill of the generated questions, it helps the model to learn the underrepresented skills in the dataset. few-shot generation. the process of manually writing questions to assess humans sbrcs is difficult. in some stories, professional writers find obstacles in writing questions for some skills as those skills require high attention and advanced reasoning skills to be written. we can see that in our own dataset, as some skills have fewer questions (e.g. predicting, visualizing, etc.). thus, in this experiment, we evaluate the performance of hta-wta model when we inject a low percentage of the skills' instances into the training set. this experiment will simulate the case when training a model on a dataset that contains few skills' instances. we use the stratified sampling technique when sampling fewer instances from the collected dataset. figure 3 shows that injecting only 10% of the data led to a boost in performance of 5.99 (bleurt). the result at 10% (33.21%) exceeds the results of most of the baselines and is higher than t5-wta and nqg-max models when trained on all the datasets (see ## conclusion and future work in this paper, we presented a new reading comprehension dataset to assess reading skills using stories. unlike previous datasets that focused on either inferential or literal questions, our dataset has nine different sbrcs, each contains inferential and literal questions. in addition to that, we proposed hta-wta model which uses two-steps fine-tuning processes to take advantage of previous datasets which have different question formats, and to learn how to ask skill-related questions. we evaluated the model on the collected dataset and compared it to several strong baselines. our extensive experiments showed the effectiveness of the model. additionally, hta-wta is able to generate high quality questions when only 10% of the dataset is used (∼240 instances). in future work, we plan to extend our dataset with additional skills, and to investigate how our model can be integrated into online educational platforms.