Word Discriminations for Vocabulary Inventory Prediction (2021)

Abstract: The aim of vocabulary inventory prediction is to predict a learner’s whole vocabulary based on a limited sample of query words. This paper approaches the problem starting from the 2-parameter Item Response Theory (IRT) model, giving each word in the vocabulary a difficulty and discrimination parameter. The discrimination parameter is evaluated on the sub-problem of question item selection, familiar from the fields of Computerised Adaptive Testing (CAT) and active learning. Next, the effect of the discrimination parameter on prediction performance is examined, both in a binary classification setting, and in an information retrieval setting. Performance is compared with baselines based on word frequency. A number of different generalisation scenarios are examined, including generalising word difficulty and discrimination using word embeddings with a predictor network and testing on out-of-dataset data.

## introduction given a small sample of words, how well can we predict whether a learner knows some out-ofsample word? this is the task of vocabulary inventory prediction. a clear motivation for the topic is to enable quicker and more precise placement testing. for example, a 40 word self-assessed word knowledge quiz used as a benchmark in this paper is quick enough that an l2 learner returning to a language learning app after a long break, in which they may have either forgotten a lot or had a lot of extra exposure to their target language, can be placed again quickly without excessive disruption. this paper addresses the following research questions: computerised adaptive testing (cat) @xcite has not been widely applied to the task of vocabulary inventory estimation. a cat system selects questions based on a examinee's previous answers in order to converge on an accurate ability estimate faster. related, but outside of cat/irt setting, @xcite builds graphs made from a combination of multiple corpora combined and apply label propagation to find a fixed set of queries to in-effect give a more accurate ability estimate than choosing at random. restricting ourselves to the adaptive setting, the main prior art is the website http://testyourvocab.com/ , which uses cat to estimate vocabulary size based on word frequencies. to the best of the author's knowledge, there is no prior work attempting to quantify how accurate the ability estimates obtained when applying cat to the problem of vocabulary inventory estimation are. ## experiments we first evaluate how well the item/word parameters from the irt model can be regressed with the chosen architecture. next we move on to consider how well various cat approaches can estimate learners' abilities. finally, the results for the final task of vocabulary inventory prediction are presented, first cross validating on the svd12k dataset and then training on the whole svd12k dataset and testing on the extra datasets. four generalisation scenarios are considered across experiments: ## discussion we now summarise and discuss some of the main results of the experiments. firstly, the discrimination parameter does not appear to help with query item selection, however it remains somewhat inconclusive whether it can help with estimating the learner ability θ since this was the best configuration in the gen-none case. it may be that with sufficiently high quality estimates of the discrimination values, using this for θ-estimation would help more. the approach which appeared best overall in this case however, and which was used for later experiments on the svd12k dataset ignored the discrimination parameter altogether for both steps of the cat stage. the difficulty parameter generalises reasonably well, while the discrimination parameter generalises quite poorly when regressed using a mlp based on numberbatch representations of the word items. since item difficulty here is closely related to frequency, it seems quite possible that a lot of the generalisation is happening based on frequency information encoded in the word embeddings. when considering how well both parameters generalised, we should note that only one type of word embedding and regressor was tried, and others may generalise this parameter better. the regressed difficulties perform better than the frequency data on in-dataset data, while performing worse on out-of-dataset data. given all datasets contained mostly japanese learners of english, this suggests that both the irt parameter and the mlp generalising may have over fitted on narrow attributes of the particular cohort of university of tokyo students making up svd12k. conversely we see that that high quality, balanced word frequency data generalises rather well. usage of the discrimination parameter for vocabulary inventory prediction was largely inconclusive, with some evidence against it. in many cases, it appeared to decrease performance on metrics such as auroc, however some tasks showed a promising but insignificant boost in ap-. it is unclear exactly why the discrimination parameter failed to provide significant improvements in either next-item selection, θ-estimation or vocabulary inventory prediction. it is possible that the amount of response data was not sufficient either in terms of the number of respondents, or in terms of representing a diverse range of abilities, to obtain accurate word discrimination estimates. apart from simply finding and integrating more vocabulary knowledge data, one direction for future work is trying to find corpus derived measures which correlate with word discrimination, analogously to the negative correlation between word frequency and word difficulty. this would also effectively address the failure to generalise the word discriminations parameter to out of vocabulary words. we hope the methods of evaluating the different sub-tasks of the vocabulary inventory prediction task in the settings demonstrated here can help establish practices for evaluating this task more throughly. we also hope that the framing given here inspires others to tackle the problem in the challenging, but more broadly applicable setting of vocabulary inventory prediction having a small, limited number of queries. the code to replicate all experiments is made available at https://github.com/frankier/ vocabirt .