One Size Does Not Fit All: The Case for Personalised Word Complexity Models (2022)

Abstract: Complex Word Identification (CWI) aims to detect words within a text that a reader may find difficult to understand. It has been shown that CWI systems can improve text simplification, readability prediction and vocabulary acquisition modelling. However, the difficulty of a word is a highly idiosyncratic notion that depends on a reader’s first language, proficiency and reading experience. In this paper, we show that personal models are best when predicting word complexity for individual readers. We use a novel active learning framework that allows models to be tailored to individuals and release a dataset of complexity annotations and models as a benchmark for further research.

## introduction illiteracy affects at least 773 million people globally. 2 for these individuals, reading information from unfamiliar sources or on unfamiliar topics can be extremely difficult. furthermore, there are a reported 1.5 billion english-language learners worldwide, spanning all proficiency levels. statistics such as these illustrate that the comprehension of written text across the world varies substantially. even those who are deemed functionally literate have differing degrees of text comprehension. there are many characteristics of a written text that impact how difficult it is to read. however, one of the main aspects contributing to overall text difficulty is lexical complexity, i.e. word complexity @xcite . lexical content has been shown to be more important than syntactic features when predicting readability @xcite and a study performed by @xcite found that a reader needs to be familiar with at least 95% of the words contained within a text for them to understand the content. the cantankerous cat. ↓ the grumpy cat. complex word identification (cwi) focuses on identifying words in a text that may be difficult for a reader to understand and therefore may benefit from simplification @xcite . for instance, in example (1) a cwi system may identify cantankerous as a complex word, then a lexical simplification system is able to substitute it with a simpler alternative, e.g. grumpy @xcite . many downstream tasks benefit from the identification of complex words. for example, cwi has been shown to improve readability assessment systems @xcite as well as vocabulary acquisition modules in educational applications @xcite . furthermore, cwi can significantly reduce errors in simplification systems @xcite and result in higher quality simplifications @xcite . previous work on modelling cwi, including two shared tasks, has relied on the collection of large corpora containing words that have been annotated for their complexity @xcite . however, annotating the difficulty of words is a subjective task, and previous data collection has yielded low levels of annotator agreement @xcite . as a way of mitigating individual differences, such datasets typically present a homogeneous view on word complexity, by merging annotations across readers @xcite . further attempts to improve annotator agreement have included offering bonus incentives for annotators who select words matching other annotations @xcite , as well as providing guidelines for annotators to mark words that they assume would be complex for audiences such as children and those with learning difficulties. such incentives and instructions change the motivation of annotators and undermine the validity of the resulting complexity labels @xcite . it is evident that lexical complexity is a highly subjective and idiosyncratic notion. in this paper, we argue that contrary to the one-size-fits-all paradigm currently dominating many natural language processing tasks, the modelling of individual differences in word complexity is critical. many of the downstream tasks that benefit from cwi, such as text simplification, would be more effective if tailored to the individual reader. in order to facilitate personalised cwi, and therefore personalised text simplification and readability systems, audience specific complexity annotations are required @xcite . whilst it has been shown that the concept of word complexity, and thus the level of agreement, is aligned between individuals sharing a common background @xcite , we argue that the best cwi model for each individual is trained with them 'in the loop'. in our paper, we use active learning to produce per-individual models of word complexity. we collect complexity annotations from 1,225 english as a second language (esl) readers. using a novel active learning framework, we show that individual models of word complexity can be trained without extensive annotation. our experiments demonstrate that bespoke models outperform baselines and a state-of-the-art system when predicting word complexity for individuals, thereby emphasising the importance of viewing the task at a personal level. we present the following contributions: ## results in our experiment, each participant produces a gold test set containing annotations for words across cefr levels. this varies for each individual, as word complexity is subjective. for instance, consider the test set samples shown in we additionally run an experiment to evaluate the effectiveness of active learning compared to randomly sampling vocabulary. in the experiment, we require annotators to label a sample of 22 random words, as well as 22 words sampled using active learning. as this doubles the annotation effort we perform this with a subset of 100 annotators. the results are shown in we evaluate the performance of the active learning models and compare to the following baselines. when we consider the active learning models, we see that the f-score calculated across all test annotations is 0.751, and the kappa agreement is 0.503. the active learning models outperform the competitive group-average baseline which supports the hypothesis that individual models are preferable to aggregate group-level approaches. the best scores for all proficiency groups are achieved using the active learning framework. however, for the native annotations, the difference between the group-based approach and active learning models is small. when considering the group-average baseline we see that both the f-score and kappa values increase in line with proficiency. this supports the notion that the concept of word complexity stabilises for higher proficiency levels @xcite . as such, aggregate models may work well for highly proficiency audiences and individual models best for lower levels where word knowledge is more variable. ## conclusions and future work we show that with active learning it is feasible to build individual models of word complexity. our models are able to be trained efficiently, in real-time, using only 23 word annotations per individual. models trained in this fashion are able to accommodate the subjective nature of word complexity, thereby facilitating personalised readability and simplification systems. we release our dataset containing 55,125 word complexity annotations, participant metadata and models, providing a new benchmark for the task of human-centered word complexity prediction. our work is a first step towards the development of personal complexity models. in future work, we aim to more extensively investigate the relationship between a learner's perceived and actual knowledge of words, as it has been reported that there is a misalignment between what a learner thinks they know, and what they are actually able to define @xcite . additionally, the context of a word impacts its perceived difficulty @xcite ; considering token-level complexity as opposed to type-level complexity is another important avenue for future work. luise dürlich and thomas françois. 2018. efllex: a graded lexical resource for learners of english as a foreign language. in proceedings of the eleventh international conference on language resources and evaluation (lrec 2018). yo ehara, yusuke miyao, hidekazu oiwa, issei sato, and hiroshi nakagawa. 2014. formalizing word sampling for vocabulary prediction as graph-based active learning. in proceedings of the 2014 conference on empirical methods in natural language processing (emnlp), pages 1374-1384. guojun gan, chaoqun ma, and jianhong wu. 2020. data clustering: theory, algorithms, and applications. siam. sian gooding. 2022. on the ethical considerations of text simplification. sian gooding and ted briscoe. 2019. active learning for financial investment reports. in proceedings of the second financial narrative processing workshop (fnp 2019), pages 25-32, turku, finland. linköping university electronic press. sian gooding and ekaterina kochmar. 2018a. camb at cwi shared task 2018: complex word identification with ensemble-based voting. in proceedings of the 13th workshop on innovative use of nlp for building educational applications, pages 184-194, new orleans, louisiana. sian gooding and ekaterina kochmar. 2018b. camb at cwi shared task 2018: complex word identification with ensemble-based voting. in proceedings of the thirteenth workshop on innovative use of nlp for building educational applications, pages 184-194, new orleans, la, usa. association for computational linguistics. sian gooding and ekaterina kochmar. 2019a. complex word identification as a sequence labelling task. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 1148-1153. sian gooding and ekaterina kochmar. 2019b. complex word identification as a sequence labelling task. in proceedings of the 57th conference of the association for computational linguistics, pages 1148-1153, florence, italy. association for computational linguistics. sian gooding and ekaterina kochmar. 2019c. recursive context-aware lexical simplification. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 4855-4865, hong kong, china. association for computational linguistics. sian gooding, ekaterina kochmar, seid muhie yimam, and chris biemann. 2021. word complexity is in the eye of the beholder. in proceedings of the 2021 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 4439-4449. k chidananda gowda and g krishna. 1978. agglomerative clustering using the concept of mutual nearest neighbourhood. pattern recognition, 10(2):105-112. miguel grinberg. 2018. flask web development: developing web applications with python. " o'reilly media, inc.". dn gujarati and dc porter. 2009. how to measure elasticity: the log-linear model. basic econometrics, mcgraw-hill/irwin, new york, pages 159-162. birgit henriksen, dorte albrechtsen, and kirsten haastrup. 2004. the relationship between vocabulary size and reading comprehension in the l2. angles on the english-speaking world, 4(1):129-140.