Machine Learning–Driven Language Assessment (2020)

Abstract: We describe a method for rapidly creating language proficiency assessments, and provide experimental evidence that such tests can be valid, reliable, and secure. Our approach is the first to use machine learning and natural language processing to induce proficiency scales based on a given standard, and then use linguistic models to estimate item difficulty directly for computer-adaptive testing. This alleviates the need for expensive pilot testing with human subjects. We used these methods to develop an online proficiency exam called the Duolingo English Test, and demonstrate that its scores align significantly with other high-stakes English assessments. Furthermore, our approach produces test scores that are highly reliable, while generating item banks large enough to satisfy security requirements.

## introduction language proficiency testing is an increasingly important part of global society. the need to demonstrate language skills-often through standardized testing-is now required in many situations for access to higher education, immigration, and employment opportunities. however, standardized tests are cumbersome to create and maintain. @xcite and the standards for educational and psychological testing @xcite describe many of the procedures and requirements for planning, creating, revising, administering, analyzing, and reporting on high-stakes tests and their development. in practice, test items are often first written by subject matter experts, and then ''pilot tested'' with a large number of human subjects for psy-chometric analysis. this labor-intensive process often restricts the number of items that can feasibly be created, which in turn poses a threat to security: items may be copied and leaked, or simply used too often @xcite . security can be enhanced through computeradaptive testing (cat), by which a subset of items are administered in a personalized way (based on examinees' performance on previous items). because the item sequences are essentially unique for each session, there is no single test form to obtain and circulate @xcite , but these security benefits only hold if the item bank is large enough to reduce item exposure @xcite . this further increases the burden on item writers, and also requires significantly more item pilot testing. for the case of language assessment, we tackle both of these development bottlenecks using machine learning (ml) and natural language processing (nlp). in particular, we propose the use of test item formats that can be automatically created, graded, and psychometrically analyzed using ml/nlp techniques. this solves the ''cold start'' problem in language test development, by relaxing manual item creation requirements and alleviating the need for human pilot testing altogether. in the pages that follow, we first summarize the important concepts from language testing and psychometrics ( §2), and then describe our ml/nlp methods to learn proficiency scales for both words ( §3) and long-form passages ( §4). we then present evidence for the validity, reliability, and security of our approach using results from the duolingo english test, an online, operational english proficiency assessment developed using these methods ( §5). after summarizing other related work ( §6), we conclude with a discussion of limitations and future directions ( §7). ## background here we provide an overview of relevant language testing concepts, and connect them to work in machine learning and natural language processing. ## the vocabulary scale for the experiments in this section, a panel of linguistics phds with esl teaching experience first compiled a cefr vocabulary wordlist, synthesizing previous work on assessing active english language vocabulary knowledge (e.g., @xcite @xcite . this standardsetting step produced an inventory of 6,823 english words labeled by cefr level, mostly in the b1/b2 range ( ). we did not conduct any formal annotator agreement studies, and the inventory does include duplicate entries for types at different cefr levels (e.g., for words with multiple senses). we used this labeled wordlist to train a vocabulary scale model that assigns δ i scores to each yes/no test item (figure 2 ). ## the passage scale for the experiments in this section, we leverage a variety of corpora gleaned from online sources, and use combined regression and ranking techniques to train longer-form passage scale models. these models can be used to predict difficulty for c-test, dictation, and elicited speech items (figure 4 ). in contrast to vocabulary, little to no work has been done to profile cefr text or discourse features for english, and only a handful of ''cefrlabeled'' documents are even available for model training. thus, we take a semi-supervised learning approach @xcite , first by learning to rank passages by overall difficulty, and then by propagating cefr levels from a small number of labeled texts to many more unlabeled texts that have similar linguistic features. ## duolingo english test results the duolingo english test foot_5 is an accessible, online, computer-adaptive english assessment initially created using the methods proposed in this paper. in this section, we first briefly describe how the test was developed, administered, and scored ( §5.1). then, we use data logged from many thousands of operational tests to show that our approach can satisfy industry standards for psychometric properties ( §5.2), criterion validity ( §5.3), reliability ( §5.4), and test item security ( §5.5). ## related work there has been little to no work using ml/nlp to drive end-to-end language test development as we do here. to our knowledge, the only other example is @xcite , who used a support vector machine to estimate the difficulty of cloze foot_12 items for a computer-adaptive test. however, the test did not contain any other item formats, and it was not intended as an integrated measure of general language ability. instead, most related work has leveraged ml/ nlp to predict test item difficulty from operational test logs. this has been applied with some success to cloze @xcite , vocabulary @xcite , listening comprehension @xcite , and grammar exercises @xcite . however, these studies all use multiple-choice formats where difficulty is largely mediated by the choice of distractors. the work of @xcite is perhaps most relevant to our own; they used ml/ nlp to predict c-test difficulty at the word-gap level, using both macro-features (e.g., paragraph difficulty as we do) as well as micro-features (e.g., frequency, polysemy, or cognateness for each gap word). these models performed on par with human experts at predicting failure rates for english language students living in germany. another area of related work is in predicting text difficulty (or readability) more generally. napoles and dredze (2010) trained classifiers to discriminate between @xcite applied english readability models to a variety of web texts (including english and simple english wikipedia). both of these used linear classifiers with features similar to ours from §4. recently, more efforts have gone into using ml/ nlp to align texts to specific proficiency frameworks like the cefr. however, this work mostly focuses on languages other than english (e.g., @xcite @xcite . a notable exception is @xcite , who trained classifiers to predict cefr levels for reading passages from a suite of cambridge english 15 exams, targeted at learners from a2-c2. in addition to lexical and language model features like ours ( §4), they showed additional gains from explicit discourse and syntax features. the relationship between test item difficulty and linguistic structure has also been investigated in the language testing literature, both to evaluate the validity of item types @xcite @xcite and to establish what features impact difficulty so as to inform test development @xcite . these studies have leveraged both correlational and regression analyses to examine the relationship between passage difficulty and linguistic features such as passage length, word length and frequency, negations, rhetorical organization, dialogue utterance pattern (questionquestion, statement-question), and so on. ## discussion and future work we have presented a method for developing computer-adaptive language tests, driven by machine learning and natural language processing. this allowed us to rapidly develop an initial version of the duolingo english test for the experiments reported here, using ml/nlp to directly estimate item difficulties for a large item bank in lieu of expensive pilot testing with human subjects. this test correlates significantly with other high-stakes english assessments, and satisfies industry standards for score reliability and test security. to our knowledge, we are the 15 https://www.cambridgeenglish.org . the strong relationship between scores based on ml/nlp estimates of item difficulty and the irt estimates from operational data provides evidence that our approach-using items' linguistic characteristics to predict difficulty, a priori to any test administration-is a viable form of test development. furthermore, traditional pilot analyses produce inherently norm-referenced scores (i.e., relative to the test-taking population), whereas it can be argued that our method yields criterionreferenced scores (i.e., indicative of a given standard, in our case the cefr). this is another conceptual advantage of our method. however, further research is necessary for confirmation. we were able to able to achieve these results using simple linear models and relatively straightforward lexical and language model feature engineering. future work could incorporate richer syntactic and discourse features, as others have done ( §6). furthermore, other indices such as narrativity, word concreteness, topical coherence, etc., have also been shown to predict text difficulty and comprehension @xcite . a wealth of recent advances in neural nlp that may also be effective in this work. other future work involves better understanding how our large, automatically-generated item bank behaves with respect to the intended construct. detecting differential item functioning (dif)-the extent to which people of equal ability but different subgroups, such as gender or age, have (un)equal probability of success on test items-is an important direction for establishing the fairness of our test. while most assessments focus on demographics for dif analyses, online administration means we must also ensure that technology differences (e.g., screen resolution or internet speed) do not affect item functioning, either. it is also likely that the five item formats presented in this work over-index on language reception skills rather than production (i.e., writing and speaking). in fact, we hypothesize that the ''clipping'' observed to the right in plots from figure 5 can be attributed to this: despite being highly correlated, the cat as presented here may over estimate overall english ability relative to tests with more open-ended writing and speaking exercises. in the time since the present experiments were conducted, we have updated the duolingo english test to include such writing and speaking sections, which are automatically graded and combined with the cat portion. the test-retest reliability for these improved scores is .85, and correlation with toefl and ielts are .77 and .78, respectively (also, the ''clipping'' effect disappears). we continue to conduct research on the quality of the interpretations and uses of duolingo english test scores; interested readers are able to find the latest ongoing research at https://go.duolingo.com/ dettechnicalmanual . finally, in some sense what we have proposed here is partly a solution to the ''cold start'' problem facing language test developers: how does one estimate item difficulty without any response data to begin with? once a test is in production, however, one can leverage the operational data to further refine these models. it is exciting to think that such analyses of examinees' response patterns (e.g., topical characteristics, register types, and pragmatic uses of language in the texts) can tell us more about the underlying proficiency scale, which in turn can contribute back to the theory of frameworks like the cefr. the yes/no vocabulary format (figure 2 ) is traditionally graded using the sensitivity index d ′a measure of separation between signal (word) and noise (pseudoword) distributions from signal detection theory @xcite . this index is isomorphic with the auc @xcite , which we use as the graded response r i . this can be interpreted as ''the probability that the examinee can discriminate between english words and pseudowords at level δ i .'' c-test items (figure 4(a) ) are graded using a weighted average of the correctly filled wordgaps, such that each gap's weight is proportional to its length in characters. thus, r i can be interpreted as ''the proportion of this passage the examinee understood, such that longer gaps are weighted more heavily.'' (we also experimented with other grading schemes, but this yielded the highest test score reliability in preliminary work.) the dictation (figure 4(b) ) and elicited speech (figure 4(c )) items are graded using logistic regression classifiers. we align the examinee's submission (written for dictation; transcribed using automatic speech recognition for elicited speech) to the expected reference text, and extract features representing the differences in the alignment (e.g., string edit distance, n-grams of insertion/substitution/deletion patterns at both the word and character level, and so on). these models were trained on aggregate human judgments of correctness and intelligibility for tens of thousands of test item submissions (stratified by δ i ) collected during preliminary work. each item @xmath0, foot_13 which were then averaged to produce ''soft'' (probabilistic) training labels. thus r i can be interpreted as ''the probability that a random english speaker would find this transcription/ utterance to be faithful, intelligible, and accurate.'' for the dictation grader, the correlation between human labels and model predictions is @xmath1.86 (10-fold cross-validation). correlation for the elicited speech grader is @xmath2.61 (10-fold crossvalidation).