The B ritish Council submission to the BEA 2024 shared task (2024)

Abstract: This paper describes our submission to the item difficulty prediction track of the BEA 2024 shared task. Our submission included the output of three systems: 1) a feature-based linear regression model, 2) a RoBERTa-based model and 3) a linear regression ensemble built on the predictions of the two previous models. Our systems ranked 7th, 8th and 5th respectively, demonstrating that simple models can achieve optimal results. A closer look at the results shows that predictions are more accurate for items in the middle of the difficulty range, with no other obvious relationships between difficulty and the accuracy of predictions.

## introduction the development of new items for high-stake exams is a complex process involving the need to meet many quality criteria. among these, item difficulty is essential, as it fundamentally impacts the validity of test scores and the fairness of the test outcomes. item difficulty pertains to the ability of test items to differentiate among varying levels of test taker proficiency consistently across diverse populations @xcite . traditionally, the estimation of difficulty requires pre-testing the newly developed items on a representative sample of test takers (usually a few hundreds), as if they were in a regular exam, and empirically estimating various statistical characteristics based on their responses. test items that are answered correctly by either too many or too few test-takers fall outside pre-determined difficulty boundaries and hence are typically removed from consideration or undergo changes before being pre-tested again. this process, although effective, is labour-intensive, costly, and time-consuming, necessitating the collection and analysis of extensive data before any new item can be used in live exams. additionally, as also noted by others (e.g., @xcite , it is sometimes impractical, or not even possible, due to constraints on exam duration, the limited availability of testing opportunities and the logistic challenges associated with live testing. to address these challenges, alternative approaches using natural language processing (nlp) have been proposed to estimate this difficulty from the items' text. predicting item difficulty has significant implications for the testing industry, not only leading to savings but also allowing the dynamic adaptation of tests to new populations. in this paper, we describe our participation in the bea 2024 shared task, aimed at predicting item difficulty for multiple-choice questions (mcq) from a medical exam @xcite . we present experiments using three different approaches: 1) using a set of linguistic features from the items in traditional machine learning regression models, 2) using pre-trained language models with and without the addition of the aforementioned features, and 3) building an ensemble model from the output of the previous two. ## related work previous studies have adopted different methodologies to estimate the difficulty of items for assessment. a vast majority of these have focused on examining textual properties of items. while early studies have used readability indices as predictors @xcite , over time, studies have evolved to utilize a wider range of complexityrelated features. these include surface lexical and syntactic features (such as word/sentence length, counts of clause types, etc. @xcite @xcite ), nlp-enabled features @xcite , and features aimed at capturing the cognitive aspects of language @xcite and cohesion @xcite . other studies have attempted to model difficulty in terms of comprehensibility for humans. mostly centred around the domain of language learning, such studies have primarily focused on applying readability metrics to language comprehension tests @xcite @xcite @xcite . in such tests, reading passages are strongly associated with the subsequent comprehension questions, thereby establishing a correlation between the text's complexity and question difficulty @xcite . there have also been attempts to estimate difficulty from the perspective of cognitive processes and knowledge dimensions required to correctly respond to a question @xcite . such approaches are mostly qualitative in nature and rely on heuristic methods which define difficulty according to the perceptions of learners, item writers and/or educators (alkhuzaey et al., 2021) item difficulty has also been estimated as part of automated item generation processes, for example by measuring the semantic similarity between an item's distractors and its prompt @xcite @xcite or estimating the difficulty and discrimination parameters of items employed in e-learning tests @xcite . in the context of mcqs, ha et al. (2019) describe models using an extensive set of linguistic features and embeddings. the same set of linguistic features were used in a subsequent study by @xcite , who obtained a strong baseline for item survival by filtering out items that were too difficult or too easy for the target test taker population. in our paper, we build upon previous research by replicating the linguistic features employed by @xcite and @xcite as well as fine-tuning a few transformer-based models. ## models we investigated a range of different models for the task, namely traditional feature-based models, transformers and ensembles. the following sections describe these in detail. ## official evaluation results our submission to the shared task included the output of the best three models found in our experiments: 1) a feature-based linear regressor (feat), 2) a roberta-based model (roberta) and 3) a linear regression ensemble (ensemble) operating on the output of the previous two models. in all three cases, the final models used for our submission were re-trained using all the available training data, unlike for our optimisation experiments where we used only 80%. an abbreviated version of the official results is included in all of our systems were also able to beat the baseline (@xmath1.311), which only 35% of the systems did. as all our systems directly or indirectly made use of linguistically-motivated features, we can also conclude that the explicit definition of features was crucial to achieve competitive results. this is in line with previous research, which has consistently found that traditional feature-based models tend to outperform deep learning models for regression tasks, especially when the amount of training data is very limited @xcite . ## analysis and discussion this section looks at the performance of our best model (ensemble) in more detail. prediction error for this model ranges from 0 to 0.8526, with a mean of 0.2494, with the majority of items having an absolute error under 0.4 (see figure 2 ). correlation between gold standard difficulty vs predicted difficulty is 0.2024 (@xmath2.05), which is considered weak (see figure 3 ). in particular, we observe that prediction error decreases when the gold standard difficulty goes from 0 to roughly 0.4, then remains low between 0.4 and 0.6 and finally steadily increases from that point onwards, as seen in figure 4 . this reveals that the model is more accurate for values in the middle of the range and particularly inaccurate for very difficult items. we also looked at the relationship between prediction error and item similarity, where similarity is given by the two principal components (pc1 and pc2) from principal component analysis on the items' bert embeddings (figure 5 ). however, the plot shows no obvious correlations or clusters, suggesting that similar items are not predicted with the same degree of accuracy by our ensemble model. performance by item type shows that text-only items have a mean absolute error of 0.2506 while items with pictures yield 0.2399. although this difference is probably negligible, it is somewhat surprising that difficulty for items containing pictures are slightly more accurately predicted when none of our models take those pictures into account -4 -2 0 2 4 pc1 -2 0 2 4 6 pc2 prediction error by item similarity 0.0 0.2 0.4 0.6 0.8 1.0 normalized absolute error figure 5: prediction error and the relationship between items. in terms of the exam level of each item, we found that the average prediction error increases as the steps level is higher, which matches our intuition that that difficulty increases by level (steps 1/2/3 mean difficulties are 0.2264, 0.2557 and 0.2782 respectively). the effect of the number of distractors, however, does not seem to follow a clear trend, as error increases when using 4 and 7 distractors but it decreases when using 5, 6 and 8. the number of distractors yielding the lowest prediction error is 6. ## conclusions in this paper, we have described the three models that were used in our submission to the bea 2024 shared task: 1) a traditional feature-based regressor, 2) a transformer-based model and 3) an ensemble model. our best system, a linear regressor ensem-ble, ranked 5th, producing near-optimal results. a detailed analysis revealed that our ensemble model is more accurate at predicting difficulty in the middle range, struggling to predict more difficult items. other aspects, such as the inclusion of pictures or the number of distractors, do not have a significant impact on prediction accuracy. all in all, our experiments show that simple models based on linear regression or pre-trained language models can achieve acceptable performance without excessive fine-tuning. in future work, we would like to explore the use of custom loss functions in our transformer models as well as new features and the addition of synthetic data, since we believe that the performance of all the systems that participated in the shared task was hindered by the small size of the training data.