BERT - IRT : Accelerating Item Piloting with BERT Embeddings and Explainable IRT Models (2024)

Abstract: Estimating item parameters (e.g., the difficulty of a question) is an important part of modern high-stakes tests. Conventional methods require lengthy pilots to collect response data from a representative population of test-takers. The need for these pilots limit item bank size and how often those item banks can be refreshed, impacting test security, while increasing costs needed to support the test and taking up the test-taker’s valuable time. Our paper presents a novel explanatory item response theory (IRT) model, BERT-IRT, that has been used on the Duolingo English Test (DET), a high-stakes test of English, to reduce the length of pilots by a factor of 10. Our evaluation shows how the model uses BERT embeddings and engineered NLP features to accelerate item piloting without sacrificing criterion validity or reliability.

## introduction the duolingo english test (det) is a test of english language proficiency that is used for admissions decisions in english medium universities. it measures the four skills of speaking, writing, reading, and listening. it is delivered remotely to testtakers' computers via a desktop application, and it can be taken any time and at any appropriate location with a strong enough internet connection. the det's value proposition to test-takers is that it is affordable, short in duration, and has a short score reporting turn-around time. the det accomplishes this, in part, by using computer adaptive test (cat) administration to more quickly and accurately estimate test-takers' language proficiency @xcite . a computer adaptive test (cat) uses item parameter estimates to adapt to each test-taker by finding items that will yield maximal information about their proficiency based on how well they've done so far. item banks for cats must be very large to ensure that test-takers do not have preknowledge of items @xcite , and they also require high-quality item parameter estimates to ensure that items are selected for administration accurately. typically, item parameters are estimated from hundreds of responses for each item collected via pilots. however, these pilots take up the test-taker's valuable time and increase the costs for the assessment, thus limiting the rate at which new items can be added to the bank. explanatory frameworks that estimate item parameters from item features have been around for a long time, starting with @xcite 's log linear traits model (lltm), and have a rich literature @xcite . these frameworks can be used to help reduce or eliminate the need for item piloting by leveraging item features to estimate item parameters more accurately with less response data. this can have positive downstream effects on test security and on testtakers. for security, it allows for test developers to add to, or replace, their item banks at very high rates, which helps to ensure unique administrations of tests and reduce the effects of item preknowledge. for test-takers, it reduces the amount of time they spend responding to unscored test items during pilots and reduces the costs of test development. these cost savings can be passed on to test-takers and even help lower barriers for less economically advantaged test-takers. it is well known in the nlp literature @xcite that pretrained language models such as bert @xcite ) learn text representations that represent highly general linguistic properties of words that are useful for a wide range of tasks, including estimating the difficulty of text for l2 learners @xcite . more recent work has explored using these text embeddings in explanatory irt models to predict parameters for test items. for example, @xcite finetuned bert to predict difficulty using datasets of educational questions and real student responses, and byrd and srivastava (2022) combined contextual embeddings from bert with additional manually curated features to predict difficulty and discrimination for general knowledge questions. similar work has used bert to predict the difficulty of multiple-choice questions @xcite and programming problems @xcite . one example of using explanatory models this way is described in our previous work, @xcite , which proposed using bert embeddings in a multi-task explanatory item response theory (irt) framework, called bert-lltm, to estimate the item parameters of c-test tasks, a task typically used to assess l2 language proficiency. this work introduces a new model, bert-irt, which makes several improvements to this approach: in addition to the offline evaluation on historical data, we present the results of using this model to shorten pilots by a factor of 10 on a real-world high-stakes test of english for l2 learners. ## background: language assessment first, we will provide a brief overview of the relevant concepts from language assessment research. ## model in the following sub-sections, we explain the bert-irt model in detail, starting with explaining the standard 2pl irt model in section 3.1 and then extending it with an explanatory framework in section 3.2. we then discuss the bert-irt model's features in section 3.3, before finally explaining the training process in section 3.4. ## experiments here we present a series of four experiments to evaluate bert-irt using data from the duolingo english test, a high-stakes test of english for l2 learners. in the first experiment, we use offline evaluation to analyze the model's performance when piloting a new item bank from scratch (i.e., what we refer to as a "fast-start" scenario). in the second experiment, we analyze bert-irt's ability to generalize item parameter predictions to unseen items under various conditions. in the third experiment, we investigate how much each feature contributes to the estimation of item parameters. finally, in the forth experiment, we analyze bert-irt's ability to leverage response data from an existing item bank to make predictions for new items with limited piloting data available (i.e., what we refer to as a "jump-start" scenario). as part of this, we discuss the results of using bert-irt to add new items to the test's item bank with only a tenth of the normal amount of pilot data. ## conclusion & future work in this paper, we demonstrated how an explanatory irt model with bert embeddings and other engineered nlp features can be used to accurately estimate item parameters for c-test items with limited piloting data. we showed that the model is able to use these features to generalize item parameter estimates across items, and that both bert embeddings and engineered features contribute to the performance of the model. furthermore, we showed how this was used on a high-stakes test of english to replace a third of its item pool with a tenth of the data that would normally have been required. finally, our ablation study in experiment 4 showed that we should be able to use bert-irt to reduce the pilot even further with negligible impact on criterion validity or reliability. in a future work, we plan to explore similar applications of nlp and explanatory irt models to other item types, and ways to reduce or eliminate the need for item piloting even further.