Jump-Starting Item Parameters for Adaptive Language Tests (2021)

Abstract: A challenge in designing high-stakes language assessments is calibrating the test item difficulties, either a priori or from limited pilot test data. While prior work has addressed ‘cold start’ estimation of item difficulties without piloting, we devise a multi-task generalized linear model with BERT features to jump-start these estimates, rapidly improving their quality with as few as 500 test-takers and a small sample of item exposures (≈6 each) from a large item bank (≈4,000 items). Our joint model provides a principled way to compare test-taker proficiency, item difficulty, and language proficiency frameworks like the Common European Framework of Reference (CEFR). This also enables new item difficulty estimates without piloting them first, which in turn limits item exposure and thus enhances test item security. Finally, using operational data from the Duolingo English Test, a high-stakes English proficiency test, we find that the difficulty estimates derived using this method correlate strongly with lexico-grammatical features that correlate with reading complexity.

## introduction high-stakes language assessment demands high reliability, validity, and security @xcite . these goals are at odds with each other during the test design process. large-scale pilot testing of new items to accurately measure their psychometric characteristics (e.g., difficulty and discrimination) risks that those items will be copied and leaked @xcite . computer-adaptive tests, which more precisely score test-takers by selecting items of appropriate difficulty on-the-fly, exacerbate this conflict: their item banks must be large enough to cover all proficiency levels while ensuring the security of test items. ## background: item response theory item response theory is the basis for most modern high-stakes standardized tests. it jointly assesses each person's ability and the difficulty of each item in the item bank. the major distinction between irt and its predecessors, such as classical test theory, is that irt assumes the item characteristics (e.g., difficulty) to be independent from the person or sample characteristics (e.g., the person ability distribution). irt models relate the probability of answering correctly to a person's latent scalar ability θ through an item response function. the simplest such model is the rasch model @xcite , a special case of logistic regression with one parameter per test-taker p (their ability θ @xmath0: decomposing items into linguistic skills we do not believe that the test items represent independent, atomic skills for a test-taker to master. instead, each item may amalgamate several skillsperhaps mastery of certain linguistic attributesthat are shared across test items. the student's performance will depend on their attainment of each skill to different degrees. one of the best-known item-explanatory irt models is the linear logistic test model (lltm; @xcite that decomposes the item difficulty parameter of the rasch model into a linear combination of features extracted from each item. the rasch model is a special case whose features are indicator functions-only one of which will be active per test item. in our lltm, the total number of active features is higher, the total number of features is lower, and they are not orthogonal. 2 a single feature pertains to multiple items, providing regularization (i.e., sharing feature weights amongst test items) and thereby increasing robustness. the features can characterize unseen items as well, which enables generalization. this is crucial to our jump-starting process. cold starts and jump-starts in traditional test development processes, one wants to pinpoint item parameters by pilot testing on test-takers whose ability is near the item's true difficulty. but without initial difficulty estimates, computer-adaptive tests cannot choose the most informative items to present to pilot testers. this is costly for large item banks and a security risk for high-stakes tests. this cold start problem is mitigated by discriminative, feature-based difficulty estimation @xcite according to an external standard like the cefr. however, once test administration yields enough test-taker responses, the item parameters should be directly calibrated on these responses. further, these test-taker responses can inform the difficulty of new items that grow an existing item bank, akin to establishing a prior that can be refined via bayesian updating as test-taker responses are collected @xcite . we refer to this as jump-starting the parameters of the new items. ## approach and formalism we extend the 2pl model in two key ways. first, while (2) uses a single parameter for each item's difficulty, like @xcite we decompose item difficulty into a weighted sum of features ( §3.1). we decompose item discrimination the same way. second, we integrate cefr-labeled data for indirect supervision of passage difficulty ( §3.2) within an ordinal-logistic regression multi-task model ( §3.3). ## experimental setup we train our jump-start model (which we hereafter refer to as bert-lltm) on either cefr-labeled data, test-taker responses from a high-stakes english proficiency test, or both. specifics of learning and reproducibility details are given in appendix b. we then compare several measures of model fit, both on held-out test-taker exam sessions and fully held-out items. the latter represents adding new items to an already-available test. §5 shows that we are able to jump-start the difficulty estimates for these items, outperforming three baselines, including @xcite . ## results and interpretation here we study how well our model captures testtakers' ability to answer c-test items based upon the passage's difficulty, and how well it contextualizes this with the cefr scale. ## recipe for a language proficiency test the better calibration and reliability of joint over baselines, its aptitude for cold start scenarios, and the ability to jump-start new item parameters suggest a "user guide" for building and growing a new english proficiency exam for a cold start. this approach enables continuous improvement of item difficulty and discrimination parameters. importantly, our model can elegantly handle training from any of these data combinations available. ## linguistic interpretation here we triangulate between the bert features and linguistic features of the text, examining whether our model generates theoretically valid, linguistically meaningful difficulty estimates. 2020) figure 4 : relating model-derived β estimates to our six identified dimensions of functional language; 3 dimensions show significant, strong correlation (|r| > 0.5) in our model vs. none from @xcite . our approach, multi-dimensional (md) analysis @xcite , was developed in applied linguistics to comprehensively describe linguistic variation in a corpus. this is a process that employs exploratory factor analysis to reduce a large set of computationally identified linguistic features in passages down to a smaller set of interpretable "dimensions" for which each passage receives a dimension score (see @xcite for complete details). the interpretation of each dimension is based on linguistic theory, the results of previous md analysis literature, and a qualitative analysis of the texts with relatively high and low dimension scores. based on these sources of evidence, we developed a functional interpretation for each of the six resulting factors: d1: oral/involved vs. literate/informational d2: lexical variability d3: expository vs. narrative style d4: personal feelings, expectations, and desires d5: epistemic stance expression d6: subjective vs. objective description details of our md analysis are in appendix e. figure 4 illustrates the relationship between the final difficulty estimates from our models and the md-based linguistic dimension scores for each passage. the bert-lltm (joint) model (figure 4a ) shows moderately strong relationships between the difficulty estimates and the dimension scores on the first four of the six dimensions. in contrast, the @xcite baseline (figure 4b ) showed moderate to weak relationships, and were consistently lower than those of our model. this indicates that the bert-lltm difficulty estimates do a better job of honing in on pertinent linguistic features. for example, as passages are estimated to be more difficult, they become more informational, have greater lexical variability, and become more expository and objective. while privacy and test integrity prevents the release of in-use test items, we present some retired test items with their model predictions in ## related work the challenge of estimating passage difficulty is widely considered, with roots in the readability research @xcite . recent approaches leverage techniques from natural language processing @xcite @xcite . rather than using explicit, interpretable linguistic features of the passage, our features ( §3.4) are drawn from a large, pre-trained neural network's representation of the passage. pre-trained distributed representations of text are in widespread use in natural language processing; recent work leverages these large models for assessment research. ha et al. (2020) provides an empirical study, comparing a battery of pre-trained models on their ability to predict test-taker proficiency from short-answer and multiple choice questions on a high-stakes medical exam. in a related passage sme cefr level predicted cefr level predicted difficulty tara: do you want to go to the museum today? billy: no, i don't like the museum very much. i want to go to the movie theater. tara: i don't like any of the movies at the movie theater. billy: ok, we'll go to the café. tara: ok! a2 a2 -6.75 since water is so important, you might wonder if you're drinking enough. there is no magic amount of water that kids need to drink every day. usually, kids like to drink something with meals and should definitely drink when they are thirsty. but when it's warm out or you're exercising, you'll need more. be sure to drink some extra water when you're out in warm weather, especially while playing sports or exercising. ## conclusion we have introduced a multitask model bert-lltm to estimate item parameters for adaptive language tests, and demonstrated that it improves upon the performance of @xcite by incorporating test-taker response data, and matches the strong 2pl-irt baseline on most metrics. furthermore, we showed that 3,000 pilot item administrations were sufficient for good performance with a large (>4,000) item bank, whereas the standard 2pl-irt model required 200 times as many administrations to achieve similar performance (see figure d .2). finally, we showed that model's item parameter estimates generalize well even for items that have not yet been piloted, whereas a 2pl-irt model cannot generalize to unseen items at all. these results have significant applications in the area of high-stakes language proficiency testing. in addition to addressing the cold start problem described by @xcite , models like bert-lltm can be used to jump-start new items by providing good initial parameter estimates. this reduces the number of item exposures needed during piloting and increases the feasibility of maintaining large item banks, both of which are crucial for maintaining the security of a high-stakes test. furthermore, the difficulty estimates produced by our model are much more strongly correlated with four of six dimensions of functional language derived from co-occurring lexico-grammatical features, suggesting both that the model is keying into these features (see @xcite and that these linguistic features are related to the true difficulty of the passages. this illuminates the linguistic abilities of the test-takers. in the future, we plan to show that bert-lltm's parameter estimates can be used as bayesian priors for a 2pl-irt model to achieve even more accurate results. furthermore, we can expand to other item types. finally, we will explore modeling the word-level difficulty and discrimination of damaged words in the passage.