On the application of Transformers for estimating the difficulty of Multiple-Choice Questions from text (2021)

Abstract: Classical approaches to question calibration are either subjective or require newly created questions to be deployed before being calibrated. Recent works explored the possibility of estimating question difficulty from text, but did not experiment with the most recent NLP models, in particular Transformers. In this paper, we compare the performance of previous literature with Transformer models experimenting on a public and a private dataset. Our experimental results show that Transformers are capable of outperforming previously proposed models. Moreover, if an additional corpus of related documents is available, Transformers can leverage that information to further improve calibration accuracy. We characterize the dependence of the model performance on some properties of the questions, showing that it performs best on questions ending with a question mark and Multiple-Choice Questions (MCQs) with one correct choice.

## introduction question difficulty estimation (qde), also referred to as "calibration", is a crucial task in education. indeed, since question difficulty can be leveraged to assess the skill level of students under examination, wrongly calibrated questions are cause of erroneous estimations. also, an accurate calibration enables to detect and discard questions that are too easy or too difficult for certain students. traditionally, qde is performed manually or with pretesting. manual calibration is intrinsically subjective and inconsistent. pretesting leads indeed to an accurate and consistent calibration, but i) introduces a long delay between question generation and when the question can be used to score students, and ii) requires to deploy the new questions before actually using them for scoring. to address the issues of the traditional approaches to calibration, recent research tried to leverage natural language processing (nlp) techniques to perform qde from question text: the idea is to use some pretested questions to train a model that performs qde from text and thus eliminates (or at least reduces) the need for pretesting of new questions. although such works showed promising results, none of them experimented with the latest nlp research, such as transformer-based models @xcite . in this work, we aim at filling that gap, studying how different transformers, also trained with different approaches, compare with the current state of the art. we evaluate several models built upon the pre-trained bert @xcite and distilbert @xcite language models on the publicly available assistments dataset and the private cloudacademy dataset. by using data from two different domains, we add to the growing body of evidence showing that item text can be used to perform qde. more precisely, we show that -after being fine-tuned on the task of qde from text -transformer models are capable of calibrating newly generated questions more accurately than previously proposed approaches. on top of that, we explore the possibility of leveraging additional textual information which might be available (e.g. transcript of video lectures) to perform an additional pre-training of the transformers before fine-tuning them for qde, and show that such approach can be used to further improve the accuracy. overall, transformer-based models are capable of reducing the rmse by up tp 6.5%, with respect to previous approaches. lastly, we perform an analysis of the best performing model to under-stand whether some question characteristics (e.g. text length, question type) particularly affect its performance. code is available at https://github. com/aradelli/transformers-for-qde . ## related work there is a large interest in understanding how textual features affect item difficulty @xcite , and this is not limited to the educational domain: for instance @xcite focus on difficulty estimation in community question answering systems. the first works addressing qde from text focused on mcqs and used deterministic approaches based on bag of words and similarities between question, answer, and distractors @xcite @xcite . recent works mostly use machine learning approaches. @xcite proposed a neural network for qde of "reading" problems, in which the answer has to be found in a text provided together with the question. the model receives as input both the text of the question and the document, thus it actually estimates the difficulty of finding the correct answer in the provided document. this is a major difference from all other works, in which the difficulty depends on the questions only. @xcite introduced a model to estimate the p-value of questions from text. the p-value of a question is defined as the fraction of students who correctly answered it and does not account for different skill levels. this model was trained using the text of questions and a large dataset of medical documents, thus it cannot be used if an analogous dataset is not available. similarly, the model proposed in @xcite requires for training a dataset of medical documents in addition to the question texts. the model is made of two neural architectures to estimate the wrongness (i.e. 1p-value) of newly-generated mcqs, considering it as made of two components which indicate i) how difficult it is to choose between the possible choices, and ii) how difficult it is to recall the knowledge required to answer the question. @xcite proposed r2de, a model that estimates the difficulty and discrimination, as defined in item response theory (irt) @xcite , of newly generated mcqs. r2de computes tf-idf features from the text of the questions and the text of the possible choices, and feeds two random forest regressors with such features. the performance of r2de was improved in @xcite , with the addition of readability and linguistics features. to the best of our knowledge, (xue et al., 2020) is the first work that explored the effects of transfer learning for qde from text. specifically, the authors fine-tune pre-trained elmo embeddings @xcite for the task of response time prediction, and subsequently perform a second finetuning for the task of p-value estimation. differently from all the other models, this one and r2de can be trained using only question text, without needing an additional dataset of related documents. our approach differs from previous research in several ways. first of all, we adopt transfer learning on transformers, which are yet to be explored for qde from text. secondly, similarly to @xcite , we perform the estimation of the irt difficulty which, differently from wrongness and p-value, models students' skill levels as well. specifically, we consider the one-parameter model @xcite , a logistic model that associates a skill level to each student and a difficulty level to each question (both represented as scalars). a brief introduction to the one-parameter irt model is given in appendix a. thirdly, the transformer models presented here do not necessarily require an additional dataset of documents from the same topics as the questions, and they can be trained using only question texts; however, if such dataset is available, it can be leveraged to further improve the accuracy of calibration. lastly, each previous work experimented on one private dataset; we experiment on two datasets (one being publicly available), showing that transformers can be successfully used for qde in different domains. ## introduction to bert and distilbert bert @xcite is a pre-trained language model that reached state of the art performance in many language tasks. its key technical innovation was the application of the transformer, a popular self-attention model @xcite , to language modeling. bert is originally trained to address two tasks: masked language modeling (mlm) and next sentence prediction (nsp). mlm consists in removing one word from the input text and asking the model to fill the gap, while in nsp the model is asked -given two input sentences -to tell whether the second sentence is a reasonable continuation to the first one. crucially, bert can be used for many different downstream tasks, as we do here for qde: starting from the pre-trained model, it is sufficient to stack an additional layer on top of the original network and then retrain it on the desired task (process named "finetuning"). during fine-tuning, the internal weights of the pre-trained model are updated and adapted to the desired task (together with the weights of the added layer), which is both more efficient than training the whole network from scratch and capable of better results, as the knowledge of the pre-trained model is not discarded. bert is a large model and therefore requires many resources for training and fine-tuning. for this reason, we also experiment with distilbert @xcite , which is a language model obtained by distilling bert. knowledge distillation is a compression technique in which a small model is trained to reproduce the full output distribution of a larger model @xcite . with this approach, distilbert is able to retain 95% of bert's performance on several language understanding tasks using about half the number of parameters of bert. similarly to bert, distil-bert can be fine-tuned on downstream tasks and it is therefore worth exploring for qde from text. ## models this section describes how we build the different models which are compared with the current state of the art of qde from text. these models are built upon the two pre-trained language models, finetuning them with two different approaches. the first approach consists in directly fine-tuning the pre-trained model for the task of qde from text. the second approach is made of two steps: we i) further pre-train the pre-trained model on the task of masked language modeling (mlm) to improve domain knowledge, and subsequently ii) fine-tune it on the task of qde from text. this is all done separately for the two datasets: we do not perform any experiments across the two datasets. ## experimental datasets in this work we use the publicly available data collection provided by assistments and the private cloudacademy data collection. both data collections are made of two datasets: i) the answers dataset (referred to as a) and ii) the questions dataset (q). it is important to remark that a and q are abstract names: we have one a dataset for cloudacademy and one a dataset for assistments (similarly for q). a contains the students' answers: for each one, it stores the user id, the question id, the correctness of the answer and a timestamp. importantly, a contains only "first timers", meaning that we consider only the first interaction between a student and a question. q contains the textual information about the items: question id, question text and, in the case of cloudacademy, the text of the possible choices. for the experiments on cloudacademy data, we also have access to an additional dataset -referred to as lectures (l)which contains the transcripts of some online lectures available on the platform and is used for the additional mlm pre-training. ## experimental setup as displayed in figure 2 , training is performed in two steps, repeated for the two datasets: i) the irt model is trained in order to calibrate the questions and obtain the ground truth difficulties, then ii) these ground truth latent traits are used as target values to train the model on qde from text. the first step consists in using a to estimate with irt the target difficulty of all the questions. specifically, we use pyirt foot_5 for the estimation and consider [-5; 5] as the possible range of difficulties. difficulties estimated at this stage will later be used as ground truth and therefore are inserted as target values in q. then, q is split into a train dataset (q train ), used to train our model on qde from text, and a test dataset (q test ), which is used for the final evaluation of the model; we keep 80% of the questions for training and 20% for testing. at training time, we keep a portion of q train (10%) as development set, for hyperparameter tuning. when l is used for pre-training the model, the setup is very similar. the only difference is that the regression model, before being fine-tuned on the task of qde from text using q train , is pre-trained on the task of mlm on l. being an unsupervised task, we use the whole l for this. transformers are implemented with the transformers foot_6 library from huggingface; fine-tuning and pre-training are performed with tensorflow foot_7 . hyperparameters are shown in appendix c. ## conclusions in this paper we have performed a study of how transformer models perform in the task of qde from text, and have proposed a model which outperforms previous approaches. specifically, the proposed model is built upon a pre-trained bert language model, which is fine-tuned for the task of qde from text. previous approaches either require an additional dataset of documents about the same topics assessed by the questions or cannot leverage such information; differently from them the proposed model is capable of outperforming state of the art approaches being trained only on the text of the questions, and can be further improved if such additional dataset is available. as an outcome from our analysis, we can say that: i) if an additional dataset is available, bert with mlm pre-training seems to be the best performing model; ii) if the only available data is the text of the questions, distilbert might be a better option, as it has basically the same performance as bert but at a fraction of the computational cost. furthermore, we studied the effect of some questions characteristics on bert and r2de, comparing the two models. we have observed that the magnitude of the error naturally increases with the magnitude of the difficulty (especially for r2de), but there is not a clear correlation between the input length and the accuracy of the estimation. we have also observed that both models are less accurate in estimating the difficulty of cloze questions, compared to questions that end with a question mark, and that the decrease in accuracy is lower for bert. we believe that this happens because underscores are not frequent in natural language and thus the model has a chance of learning them only during the fine-tuning on qde, not during mlm pre-training. this is probably not enough data for learning (from scratch) the meaning of underscores in exam questions. lastly, bert performs better on questions with only one correct choice than on questions with multiple correct choices (for the latter, it is also outperformed by r2de). this might be due to the encoding we used for multiple correct choices, and it is worth exploring in future research. future works could continue to dig deeper into the analysis of the model, as we believe that the accuracy of the estimation could be further improved by using an ensemble model in which different sub-models are used depending on some characteristic of the question under calibration. also, future works will try to explore the attention layers of the proposed model, as it might provide useful information about the reasons why the model works better on some questions.