Towards the Application of Calibrated Transformers to the Unsupervised Estimation of Question Difficulty from Text (2021)

Abstract: Being able to accurately perform Question Difficulty Estimation (QDE) can improve the accuracy of students’ assessment and better their learning experience. Traditional approaches to QDE are either subjective or introduce a long delay before new questions can be used to assess students. Thus, recent work proposed machine learning-based approaches to overcome these limitations. They use questions of known difficulty to train models capable of inferring the difficulty of questions from their text. Once trained, they can be used to perform QDE of newly created questions. Existing approaches employ supervised models which are domain-dependent and require a large dataset of questions of known difficulty for training. Therefore, they cannot be used if such a dataset is not available ( for new courses on an e-learning platform). In this work, we experiment with the possibility of performing QDE from text in an unsupervised manner. Specifically, we use the uncertainty of calibrated question answering models as a proxy of human-perceived difficulty. Our experiments show promising results, suggesting that model uncertainty could be successfully leveraged to perform QDE from text, reducing both costs and elapsed time.

## introduction question difficulty estimation (qde), also known as "question calibration", is a crucial task in education. in computerized adaptive testing @xcite , for instance, students are shown questions that are suitable for their skill level. when a question is miscalibrated (i.e. its difficulty has been erroneously estimated), it can be either too hard or too easy for a student, which would negatively affect their learning outcome @xcite . if the questions are too hard, students might get frustrated and lose motivation; if they are too easy, students are not adequately challenged @xcite . in either case, their learning experience is worse than if the questions were of appropriate difficulty, which, especially in the context of large-scale online courses, gives rise to the methods of automated difficulty estimation. traditionally, qde is performed either i) manually @xcite or ii) with pretesting @xcite . manual calibration involves one (or more) human experts labelling the question by selecting a numerical value representing its difficulty, a method that is subjective and not scalable. meanwhile, pretesting involves deploying the new question in an exam as if it was a standard exam question, but without using it to assess students and without telling them that there is a question under pretesting. the other questions of the exams are then used to actually assess the students, and their answers help to calibrate the question under pretesting. this approach indeed leads to an accurate calibration, but it introduces a long delay between the time of question creation and when the new question can be used to assess students. besides, it requires the new questions to be shown to students before being actually used for scoring them, which is undesirable. recent studies tried to address the limitations of the traditional approaches by performing qde with natural language processing (nlp) techniques @xcite @xcite @xcite . they are all based upon the same general idea: starting from a set of calibrated questions, we train a supervised machine learning model to infer the difficulty of questions from their text. once the model is trained, it is used to calibrate newly-generated questions, overcoming (or at least reducing) the need for pretesting and manual calibration. although these techniques were proposed to enable an immediate calibration of new questions, they have two major limitations due to their supervised manner: i) they require thousands of calibrated questions as a training set, and ii) they cannot perform cross-domain qde. in other words, the training questions must assess the same topics as the new questions, which the model will later be used on, thus limiting its applicability (e.g. when introducing new courses in an e-learning platform). these limitations are intrinsic to such approaches and cannot be addressed by improving the accuracy of the models. in this work, we explore an approach that is a total shift in the paradigm of qde from text and which could potentially overcome both limitations. the intuition is to build an end-to-end question answering (qa) model that answers multiple choice questions (mcqs) and use its uncertainty (which can be interpreted as representing the machine-perceived difficulty) as a proxy for human-perceived difficulty, which is the final target of the estimation. previous research already hypothesized that there might be a relation between human-perceived difficulty and machineperceived difficulty @xcite , but leveraged the machine-perceived difficulty as a feature for a supervised model, thus facing the same limitations as the approaches to qde from text mentioned above. on the contrary, the approach we propose here performs qde by leveraging the confidence of a trained qa model: specifically, we compute the variance of the probability distribution over the possible answer choices of the mcq under calibration. crucially, this approach is model agnostic and can be used on any qa model that outputs scores for the possible choices of an mcq. architecture-wise, in this study, we experiment with transformers @xcite for qa. in order to understand how well the proposed approach performs with different qa models, we use three different transformers: bert @xcite , distil-bert @xcite , and xlnet @xcite . we choose them because they offer variety and differ in sizes, and were all demonstrated to perform well in several language understanding tasks. however, large neural classification models tend to be overconfident in their predictions @xcite , which might hinder the possibility of using their uncertainty as a proxy for question difficulty. in order to understand whether this is really an issue for unsupervised qde from text and to explore possible solutions, we also experiment with calibrated qa models. calibrating a model involves aligning the posterior probabilities with the empirical likelihoods @xcite . for instance, if we consider all the predictions for which a model has the confidence of 75%, the true accuracy must be 75% if the model is perfectly calibrated. we remark here that calibration and accuracy are not directly related: in fact, a model might be accurate but not calibrated or, on the other hand, not very accurate but well-calibrated. calibration can be intuitively interpreted as the "awareness" of the model of its capabilities. in practice, several techniques can be used for calibrating neural models, and they will be discussed in section 3. for simplicity and computational reasons, we use the ensembling technique. we experiment on the large question-answering dataset race @xcite to assess the qde capabilities of the proposed approach. specifically, we evaluate it on the task of pairwise difficulty prediction: given a pair of questions from race, the objective is to indicate which question of the pair is more difficult. as the gold standard for the difficulty, we use i) the difficulty level available in race and ii) additional human labels obtained with crowd-sourcing. the experimental results are promising and suggest that the proposed approach could be used to perform qde from text in an unsupervised manner leveraging the uncertainty of qa models. we also show that choosing the underlying qa model is not straightforward, and the best results are obtained leveraging models that are both accurate and calibrated. our contributions are as follows: i) we propose an unsupervised way for qde from text that does not require answer logs or question difficulty labels, only the text of the mcq and the possible choices, ii) we experiment with modern transformer-based architectures to demonstrate the viability of the proposed approach. we share our code publicly foot_0 . ## related work the earliest research about qde from text focused on mcqs, using bag-of-words and the similarities between question, correct choice, and distractors (incorrect choices) for the estimation @xcite @xcite . however, they are generally outperformed by the more recent models, which are based on machine learning techniques. ha et al. (2019) introduced a model to estimate from a question text its correctness, which is defined as the fraction of students who correctly answered the question. this model was trained using question texts and a large dataset of medical documents (i.e. books, papers). similarly, the model proposed by @xcite is trained on a dataset of medical documents and question texts to estimate the wrongness of newly generated questions. @xcite proposed r2de, a model that estimates the difficulty of newly generated mcqs, using as input only the text of the questions and the text of the possible choices, without any additional data. xue et al. ( 2020 ) explored the effects of transfer learning for question calibration from text. specifically, the authors fine-tune pre-trained elmo embeddings @xcite for the task of response time prediction and subsequently perform a second fine-tuning for the task of qde. lastly, @xcite propose a neural model for the estimation of the difficulty of reading comprehension questions. from a high-level perspective, all these models are based on the same idea: the real question difficulty (obtained either with pretesting or manual calibration) is used as the target value for training a supervised machine learning model that performs qde from text for newly generated questions. the downside of this approach is that it can work only as long as the new questions belong to the same domain as the training questions. moreover, such models need a large number of calibrated questions for training, which might be too costly to obtain, especially for smaller institutions. @xcite employs a pairwise difficulty comparison scheme similar to the one we will, but they still require the user responses for the algorithm to work, same as @xcite . motivated by this, in this work, we explore the possibility of completely shifting the paradigm for the task of qde from text. the proposed approach leverages the uncertainty of a qa model as a proxy for question difficulty and uses this modelperceived difficulty to calibrate newly generated questions. this unsupervised approach does not require ground truth difficulty labels, but only the text of the questions and, in the case of mcqs, of the possible answer choices. ## calibration of neural models several techniques exist to calibrate neural network classifiers. the most popular ones are the following: i) vanilla: maximum softmax probability, which usually does not lead to calibrated classifiers @xcite . ii) temperature scaling: a posterior calibration technique using a validation set @xcite . iii) bayesian deep learning, which requires alterations to the training procedure and is computationally expensive. iv) ensembles: consists in independently training m models on the entire dataset using different random initializations @xcite or dropout @xcite and averaging their predictions. focusing on pre-trained transformers, previous research @xcite showed that pre-trained bert is fairly well-calibrated for indomain tasks, but it is miscalibrated for out-ofdomain tasks. to the best of our knowledge, no previous research experimented with the calibration of distilbert and xlnet. in this work, we use deep ensembles with m equal to 3, since we observed that this approach led to fairly well-calibrated models for all the transformer architectures under evaluation. additionally, we also experimented with ensembles made of the combination of models with different architectures (e.g. bert with xlnet, etc.). considering the metrics existing to evaluate model calibration, one of the most commonly used approaches is the expected calibration error (ece) @xcite , which compares the confidence and the accuracy of the model. more precisely, it defines miscalibration as the difference in expectation between confidence and accuracy. thus, ece approximates the miscalibration by partitioning the predictions in a number m of bins and averaging the difference between the accuracy and confidence obtained in each bin. ## models the approach proposed in this paper leverages the confidence of a qa model to perform qde for mcqs in an unsupervised manner. specifically, it leverages the output scores, one for each possible answer choice, produced by the model for each question, and the only requirement is that such output scores represent a probability distribution (i.e. they sum to 1, such as the softmax scores of a neural network). we experiment here on mcq with four possible choices, but the approach can be easily scaled to mcq with different numbers of answer choices. it is important to note that this approach to unsupervised qde from text is "qa model agnostic", in the sense that it does not need any information about the underlying model but only the scores produced by it. we leverage the softmax scores produced by the underlying qa model to measure the model-perceived question difficulty, which is considered a proxy for humanperceived difficulty. in practice, we convert the raw softmax scores of each question into a single numerical value by computing their variance. we assume that larger values of variance indicate easier questions (since it means that the model is more certain in the estimation). we also experimented with some alternatives, but they were generally outperformed by using the score variance (although the difference was not major). specifically, we also experimented with: i) keeping only the highest softmax score, and ii) computing the difference between the highest and the second-highest softmax score. in order to understand how the proposed approach performs with softmax scores from different underlying qa models, we experiment with three transformer models: i) bert @xcite , ii) distilbert @xcite , and iii) xlnet @xcite . the details of each architecture are beyond the scope of this research; we refer to the original papers for their description. for our study, it is essential to know that they are all publicly available neural network models, which are pre-trained on several language understanding tasks, and that they can be fine-tuned for different downstream tasks with minimal changes to the architecture. precisely, in this study, we add a multiple-choice classification layer on top of each original model, which is a common task in the literature. the three models have diverse sizes and architectures, even though distilbert is strongly related to bert since it is obtained from it by using knowledge distillation @xcite to reduce the number of hidden layers. we implement all the models with the hugging-face transformers library @xcite , using in all cases the pre-trained base-cased version. we fine-tune them using google colab gpus. the parameter configuration for the qa models is taken from the literature; the specific values are shown in as suggested by @xcite , we ensemble models to reduce miscalibration. in practice, we proceed as follows. first, we i) train five instances for each architecture (i.e. bert, distilbert, xlnet), and each instance is trained on the entire training dataset (randomly shuffled), with a different random initialization. then, we ii) pick the three best performing instances of each architecture, considering the test accuracy on the qa task. lastly, we iii) build the ensembles by averaging (separately for each test question) the softmax scores produced by the three instances of each architecture so that each of the four answer options is assigned a single score from 0 to 1. these scores indicate the probability (according to the model) of each option being correct. in addition to building an ensemble for each architecture, we also build "hybrid" ensembles in the same way but averaging the predictions of instances of different architectures. ## data in this study, we use the reading comprehension race dataset foot_1 and two datasets derived from it, which contain pairs of questions and a label indicating which question of the pair is more difficult. the entire race dataset is used to train the qa models, while the two other datasets are used for the experiments on qde from text. ## experiments the goal of our experiments is to perform unsupervised qde from text, using only the softmax scores produced by qa models. more precisely, we do not estimate directly question difficulty but evaluate the proposed approach on the task of pairwise difficulty prediction: given a pair of questions, the task consists in identifying the one that is more difficult. the manual labels are obtained, also within the pairwise comparison framework. this way, we investigate i) whether the machine uncertainty leads to a notion of difficulty that aligns well with the human one -which is represented by the level and the crowdsourced labels -and ii) whether it can be useful in practical applications when logs of answers or calibrated questions are not available for training. ## conclusions the results of this research support the idea that it is possible to estimate the human-perceived difficulty of exam questions via the uncertainty scores produced by question answering (qa) neural networks. the advantage of the approach we propose is that it is possible to predict the relative difficulty of questions across different domains without needing any calibrated questions or logs of students' answers. for training the qa model, it is sufficient to have access to i) the corpus of questions and (possibly) ii) the learning materials. as a practical guideline, both the qa accuracy and the calibration seem to impact the accuracy of qde. therefore we believe that it is better to use calibrated models which are powerful enough to reach decent performance in the qa task (e.g. the bert and xlnet ensembles used here). future work will focus on exploring whether improving the calibration of the qa models (e.g. increasing the number of models in the ensemble or using bayesian neural networks) would lead to improved results in unsupervised qde from text and will analyze the effects of combining raw softmax scores with different techniques. weighing the scores by the accuracy of each intervening model is a way to explore whether hybrid ensembles can still improve the performance, but we leave the implementation of this approach for another study. another natural progression of this work is to leverage question-specific information, such as the reasoning type required to answer it, or the question format (e.g. cloze items vs interrogative items). using the text of the possible choices might improve the accuracy, which makes it an interesting modification to explore in future studies. further experimental investigations are also needed to use the proposed approach for creating a difficulty ranking of questions.