Predicting Difficulty and Discrimination of Natural Language Questions (2022)

Abstract: Item Response Theory (IRT) has been extensively used to numerically characterize question difficulty and discrimination for human subjects in domains including cognitive psychology and education (Primi et al., 2014; Downing, 2003). More recently, IRT has been used to similarly characterize item difficulty and discrimination for natural language models across various datasets (Lalor et al., 2019; Vania et al., 2021; Rodriguez et al., 2021). In this work, we explore predictive models for directly estimating and explaining these traits for natural language questions in a question-answering context. We use HotpotQA for illustration. Our experiments show that it is possible to predict both difficulty and discrimination parameters for new questions, and these traits are correlated with features of questions, answers, and associated contexts. Our findings can have significant implications for the creation of new datasets and tests on the one hand and strategies such as active learning and curriculum learning on the other.

## introduction the use of question answering for testing learning often relies on characterizing questions on aspects such as difficulty and discrimination foot_0 . for example, ordering questions by difficulty can enable curriculum learning @xcite . similarly, discrimination is used in standardized exams such as the sat to ensure that questions are varied enough to discriminate between high-ability and low-ability respondents. item response theory (irt) @xcite has been a widely applied framework to jointly estimate such parameters for questions (or items) and the abilities of respondents. while irt has its inception in psychometrics and has traditionally been used with human respondents, recently, it has been explored for analyzing predictions from an 'artificial crowd' of ml models @xcite @xcite @xcite . while it can be helpful to know which questions are difficult/discriminatory, it can be equally important to be able to determine a question's difficulty/discrimination parameters without having to use it in a testing environment (as is required to estimate irt parameters). some recent work, such as @xcite , has explored using features derived from the text of a question to predict the difficulty in the context of multiple-choice medical exams. while others @xcite have used tf-idf features to predict the difficulty of questions as measured by irt. we differ from these works in two ways: firstly, while @xcite ; @xcite both predict the difficulty of items for humans, we are interested in predicting the difficulty (and discrimination) of items for qa models. secondly, we choose a questionanswering dataset, hotpotqa @xcite , as our testbed. we utilize this dataset to generate a rich and varied feature set across each item's question, answer, and associated contexts. we can then employ these features to analyze our difficulty and discrimination predictions, giving us insights into both our underlying qa model and factors that can increase the difficulty/discrimination of a question. our analysis shows significant variations among questions and reveals some surprising patterns. we show that it is possible to predict both difficulty and discrimination of natural language questions, which can have multiple applications in education and pedagogy. additionally, we see that different surface-level features are associated with high discrimination and high difficulty, which can inform new evaluation methods and the creation of new datasets. further, we identify attributes for predicting difficulty and discrimination that are general enough to be adapted to various qa datasets. foot_2 2 irt analysis of hotpotqa irt background: we begin by summarizing the 1pl and 2pl models from irt, which form the basis of our later analysis. the 1pl (1 parameter logistic) model describes the probability of respondent i correctly answering the j'th item (question) in terms of scalar-valued parameters for question difficulty (d j ) and respondent ability (θ i ). these parameters are estimated from data y @xmath0, 1} for a set of i, j pairs. here, y @xmath1. the 1pl model is described by: the 2pl model extends the 1pl by adding a scalarvalued parameter α j , which represents the discrimination of the j'th item. intuitively, this parameter denotes how sharply the probability of answering a question correctly changes as the ability of the respondent increases. the 2pl model is described by: dataset description: we chose hotpotqa for our analysis since it is significantly more complex than other datasets such as squad @xcite due to the questions requiring multi-hop reasoning and having more complex language. in hotpotqa, each question is paired with two paragraphs considered 'gold' contexts and several other paragraphs considered 'distractor' contexts. the answer to each question is a span in one of the gold contexts, but correctly answering the question requires combining information from both 'gold' contexts. ## predicting irt parameters we next discuss predictive models for discrimination and difficulty using features from the question, answer, and associated context. first, we describe our feature set, then provide an ablation study, a feature importance study, and finally qualitatively analyze the predictions of our best model. ## qualitative analysis we qualitatively analyze the difficulty predictions to understand the predictions of our bestperforming model. similar to figure 2 , figure 3 shows a umap scatterplot foot_4 for questions on our test split of the estimated irt parameters. in this case, instead of color-coding by difficulty as in figure 2 , we instead color-code by the absolute error between our predictions and the measured difficulty of each question. we again apply kmeans (@xmath2. we highlight ct.1, like c.1 and c.2 of figure 2 , this cluster consists primarily of yes/no questions. the difficulty in ct.1 has significantly smaller variance in the estimated difficulties than the rest of the clusters (ρ = 0.02). as well, the prediction error for ct.1 has significantly smaller variance (ρ = 0.04) and had the smallest average prediction error compared to the other clusters (0.68). this indicates that the model is able to recognize when question groupings, such as yes/no questions, have consistent difficulties (as discussed in 2.1) and has consistently lower error when predicting difficulty for these questions. however, the prediction error tends to vary more when the surface-level question types are not sufficient to characterize their difficulty. we explore this further through a small counterfactual experiment. we are interested in taking an item with high prediction error and slightly tweaking it to understand how the model's predictions can change with changes in the question and answer. we selected an item @xmath3. the question we use in this study is: which university is this american philosopher, theologian, and christian apologist who supports theistic science, professor at? with an answer of biola university. the predicted difficulty was -0.51. we found that simple changes to the question, such as using synonyms and removing unnecessary information, can increase the predicted difficulty up to -0.21. however, by modifying the answer (and by necessity the question) to be either a date or yes, we achieve a higher difficulty prediction (0.53 and 1.02, respectively). this further indicates the model's bias towards yes/no questions being of a higher difficulty regardless of the style or topic of question being asked. some of our changes and their corresponding predictions are listed in appendix e. ## conclusion in this paper, we explored qa datasets through the lens of item response theory. we have demonstrated a way to build regression models that can describe the difficulty and discrimination of a question. we note that our work is limited in two important ways: firstly, we only use the dfgn model in our artificial crowd, which may have introduced a bias in which some factors that make questions difficult/discriminatory are only applicable to this model. secondly, we only explore the hotpotqa dataset, which may further limit our analysis to only be applicable to hotpotqa or similar datasets. future work could incorporate multiple models and datasets to explore a more easily generalizable difficulty/discrimination prediction pipeline. we also note that our analysis here focused on qa. however, there are many nlp tasks in which the difficulty or discrimination of an item may be important. our work here could naturally extend to these domains. finally, automatically predicting these traits without relying on user responses can engender a host of creative educational applications. future work can also leverage such predictive models to explore more efficient strategies for learning and evaluation.