Difficulty-Controllable Neural Question Generation for Reading Comprehension using Item Response Theory (2023)

Abstract: Question generation (QG) for reading comprehension, a technology for automatically generating questions related to given reading passages, has been used in various applications, including in education. Recently, QG methods based on deep neural networks have succeeded in generating fluent questions that are pertinent to given reading passages. One example of how QG can be applied in education is a reading tutor that automatically offers reading comprehension questions related to various reading materials. In such an application, QG methods should provide questions with difficulty levels appropriate for each learner’s reading ability in order to improve learning efficiency. Several difficulty-controllable QG methods have been proposed for doing so. However, conventional methods focus only on generating questions and cannot generate answers to them. Furthermore, they ignore the relation between question difficulty and learner ability, making it hard to determine an appropriate difficulty for each learner. To resolve these problems, we propose a new method for generating question–answer pairs that considers their difficulty, estimated using item response theory. The proposed difficulty-controllable generation is realized by extending two pre-trained transformer models: BERT and GPT-2.

## introduction automatic question generation (qg) for reading comprehension is the task of automatically generating reading comprehension questions related to given reading passages. various qg methods have been developed in the natural language processing (nlp) research field @xcite . they have also been used in various educational systems, such as intelligent tutoring systems, writing support systems, and knowledge assessment systems @xcite @xcite @xcite . early qg methods have relied on rule-based or template-based approaches, which use handcrafted rules or templates to generate an interrogative question text from a declarative text @xcite . however, preparing those qg methods for a target application is time-consuming and labor-intensive because achieving high-quality qg requires well-designed rules and templates for each application @xcite . end-to-end qg methods based on deep neural networks have received wide attention as a means of overcoming this limitation @xcite @xcite @xcite . earlier neural qg methods were designed as sequence-to-sequence (seq2seq) models based on recurrent neural networks (rnns) and attention mechanisms @xcite , while recent methods are based on pre-trained transformer models @xcite @xcite @xcite , including bert (bidirectional encoder representations from transformers) @xcite , gpt-2 (generative pre-trained transformer 2) @xcite , bart (bidirectional and auto-regressive transformers) @xcite , and t5 (text-to-text transfer transformer) @xcite . those methods have succeeded in generating fluent questions that are pertinent to given reading passages. a representative application of how qg can be used for educational purposes is a reading tutor that automatically offers reading comprehension questions related to various reading materials @xcite @xcite . this helps to focus learners' attention on the reading materials and offers the opportunity to observe any misconceptions they might have @xcite , which supports the development of reading comprehension skills. to enhance such learning, it is useful to provide questions with difficulty levels appropriate for each learner's reading ability. such adaptivity is a core component of recent ai-based intelligent tutoring systems. difficulty control of qg is a relatively new task @xcite , and thus previous research on difficulty-controllable qg for reading comprehension is still limited @xcite @xcite . there are currently only two conventional methods; the first uses an rnn-based seq2seq model in which hidden states before its encoder are modified to receive a difficulty as input that is categorized as either easy or hard @xcite , and the second is a multi-hop qg @xcite that takes the question difficulty to be the number of inference steps required to answer a question and aims to generate questions while controlling the number of required inference steps. however, both methods have the following limitations that prevent them from generating questions appropriate for a learner's ability. 1. they ignore the relation between question difficulty and learner ability, making it difficult to determine an appropriate difficulty for each learner. to resolve these problems, we propose a new method for generating question-answer pairs that considers the difficulty associated with learners' ability. a unique feature of our method is that it uses item response theory (irt) @xcite , a test theory based on mathematical models, to quantify the difficulty of each question-answer pair. irt is based on statistical models that define the relation between question difficulty and learner ability, and thus it helps us to select a difficulty appropriate for each learner's ability. for these reasons, we aim to generate question-answer pairs while considering their difficulty, quantified by irt. for our qg method, we first propose a method for constructing a training dataset consisting of quadruplets (reading passage, question text, answer text, and irt-based difficulty), based on the squad dataset, which is the most popular benchmark dataset for the reading comprehension qg task. then, we propose a difficulty-controllable generation method for question-answer pairs that can be trained using this dataset. our generation method consists of two pre-trained transformer-based models, which are extended to take irt-based difficulty values as input: a difficulty-controllable answer extraction model using bert, and a difficulty-controllable answer-aware qg model using gpt-2. to our knowledge, this is the first difficultycontrollable qg method aimed at generating question-answer pairs corresponding to irt-based difficulty. ## task definition the task tackled in this study is to generate a reading comprehension question and a corresponding correct answer, given a reading passage and a target difficulty value. here, we assume that a correct answer to each question consists of a segment of text from the corresponding reading passage, as in typical answer-aware qg tasks @xcite . fig. 2 shows an outline of our task. the detailed task definition is as follows. let a given reading passage be a word sequence @xmath0, . . . , i}}, where r i represents the i-th word in the passage, and i is the passage text length. similarly, let a question text q and an answer text a be word sequences @xmath1, . . . , j}} and @xmath2, . . . , k}}, respectively, where q j is the j-th word in the question text, a k is the k-th word in the answer text, j is the question text length, and k is the answer text length. note that the answer text a must be a subset of the word sequence in the reading passage r, namely, a ⊂ r. using this notation, our task is to generate a question text q and an answer text a given a reading passage r and a target difficulty value b, where the difficulty value b is assumed to be quantified based on irt, as explained in the introduction. ## item response theory irt @xcite is a statistical framework used in psychometrics and educational measurement to analyze examinees' responses to test items (items corresponds to questions in our study). one of the unique characteristics of irt is that it estimates two types of latent factors from response data: examinee ability and item characteristics. examinee ability refers to the latent trait or ability that the test is intended to measure, such as reading comprehension ability in our context. item characteristics refer to the properties of test items, including their difficulty level and their ability to discriminate examinee ability. irt uses probabilistic models, called irt models, to estimate examinees' abilities and item characteristics from response data that typically consist of a binary variable taking one if an examinee answers an item correctly and zero otherwise. irt has been widely used in various educational and psychological tests because it has the following typical benefits (uto and ueno, 2020) compared with classical test theory (a simple and traditional framework based on basic statistics such as mean, variance, and correlation coefficients): 1) irt provides detailed information about item properties, including difficulty and discrimination, which helps test developers identify problematic items and improve test quality. 2) irt provides accurate estimates of examinee ability and item properties. 3) the abilities of examinees who take different tests can be estimated on the same scale because examinee ability is estimated considering the effects of the items' characteristics. 4) irt is the basis for computerized adaptive testing (cat), which can reduce test length and increase measurement precision by selecting appropriate items for a target examinee's ability @xcite . this study uses the rasch model (a oneparameter logistic model), which is the most traditional and well-known irt model. the rasch model defines the probability that the m-th examinee correctly answers the n-th item as to explain the relationship between the latent ability θ and the difficulty parameter b in the rasch model, fig. 3 depicts item response curves (ircs) of the rasch model, which are drawn by plotting the probability p nm , for three different difficulty values. in the figure, the horizontal axis shows θ, the vertical axis shows the probability p nm , and three solid curves show the irc for three items with different difficulty values. these ircs show that examinees with higher θ have a higher probability of responding correctly to each item. we can also see that the irc shifts to the right as the item difficulty value increases, reflecting the fact that higher ability is required to correctly answer items with high b. furthermore, under the rasch model, the probability that an examinee with ability θ correctly answers the question with difficulty b becomes 0.5 when θ = b. the irt model parameters are generally estimated in two phases, namely, item calibration and ability estimation, in order to guarantee asymptotic consistency. item calibration estimates the item parameters from response data by marginalizing the examinee ability θ from the likelihood in order to ensure the asymptotic consistency of the item parameter estimates. specifically, marginal maximum likelihood (mml) estimation using an expectationmaximization (em) algorithm has been widely used for item calibration @xcite . given calibrated item parameters, the ability estimation phase calculates the examinee's ability θ. an expected a posteriori (eap) estimation, a type of bayesian estimation, is generally used for the ability estimation @xcite . this study aims to quantify question difficulty based on the irt. the next section explains how to prepare the dataset with irt-based difficulty, which is required to train our qg model. ## creating a dataset with irt-based question difficulty we require an appropriate dataset to construct our qg method for solving the difficulty-controllable qg task defined in section 2. while several popular datasets have been developed for general reading comprehension qg tasks @xcite , the most popular is squad @xcite , which consists of over 100,000 question-answer pairs from wikipedia articles. specifically, squad is a collection of triplets (r, q, a), where each answer a is a text fragment from a corresponding reading passage r and each reading passage r corresponds to a paragraph of a wikipedia article. however, to construct a difficulty-controllable qg method, we require a dataset consisting of quadruplets (r, q, a, b). thus, we first propose a method for extending the squad dataset by appending the irt-based difficulty values for each questionanswer pair. the details for doing so are as follows. we collect answers from multiple respondents to each question in the squad dataset and grade those answers as correct or incorrect. ideally, we should gather responses from a population of target learners, but this is highly expensive and time-consuming. thus, we substitute actual learner responses with automated questionanswering (qa) systems, in the same way that several previous difficulty-controllable qg studies have done @xcite . ## proposed method our difficulty-controllable qg method, which is trained using the extended squad dataset, is realized by performing the following two tasks in sequence: (1) difficulty-controllable answer extraction that extracts an answer text from a given reading passage while considering a target difficulty value, and (2) difficulty-controllable answer-aware qg that generates a question given a reading passage, an answer text, and a target difficulty value. details of each are provided in the following sections. ## experiments in this section, we demonstrate that our proposed method can generate questions and answers corresponding to target irt-based difficulty values. ## conclusion in this study, we proposed a new neural qg method that generates question-answer pairs while considering their difficulty, estimated using irt. we also evaluated the effectiveness of this method through experiments using squad. one limitation of this study is that we used only the squad dataset in our experiments. the squad dataset has often been criticized because it is overly dependent on the similarity of question/answer sentences rather than on human-type reasoning, meaning it requires only superficial read-ing skills. thus, examining the effectiveness of our proposed method by applying it to various other datasets will be an important future task. furthermore, in the human evaluation experiment presented in section 6.4, we examined only 60 question-answer pairs generated through the proposed model from ten randomly selected reading passages. the relatively small scale of the experiment is due to the high workload required for people to carefully evaluate the various properties of a large number of questions. however, in the future, we aim to conduct a larger-scale human evaluation in order to increase the reliability of the experimental results. although the present study used only five qa systems, the use of a larger number of qa systems with different characteristics is expected to improve the accuracy of question-difficulty estimation and provide difficulty estimates with finer granularity. therefore, examining the effects of increasing the number and variability of qa systems will be another future direction of this research. we also need to confirm in greater detail whether qa systems can be substituted for human learners. a comparison between irt-based question difficulties calibrated from the responses of qa systems as well as human learners might be a plausible approach. another future goal is to develop a method of transforming the scale of the irt-based difficulty, estimated based on qa systems, into a scale appropriate for a population of target learners. such a scaling adjustment is expected to be achievable by using equating, which is a well-established technique in irt. furthermore, our qg method is easily extended to adaptive qg systems based on the framework of computerized adaptive testing, as mentioned in section 5.3. developing and evaluating such an adaptive system using our qg method will also be our focus in future work.