Word Complexity is in the Eye of the Beholder (2021)

Abstract: Lexical complexity is a highly subjective notion, yet this factor is often neglected in lexical simplification and readability systems which use a ”one-size-fits-all” approach. In this paper, we investigate which aspects contribute to the notion of lexical complexity in various groups of readers, focusing on native and non-native speakers of English, and how the notion of complexity changes depending on the proficiency level of a non-native reader. To facilitate reproducibility of our approach and foster further research into these aspects, we release a dataset of complex words annotated by readers with different backgrounds.

## introduction complex word identification (cwi) is the first step in a lexical simplification (ls) pipeline, concerned with identification of words in text that are in need of further simplification @xcite . for instance, in example (1) a cwi system might identify engulfed as a complex word, which would allow an ls system to replace it with a simpler alternative, e.g. flooded, in the next step @xcite : it has been shown that accurate cwi can significantly reduce errors in simplification @xcite , thus improving the quality of an ls system output @xcite . in addition, cwi has been shown to be an important component in readability assessment systems @xcite and in vocabulary acquisition modules of educational applications @xcite . however, an important aspect of cwi and ls that is often neglected is that text complexity is not an objective notion homogeneous across various target populations: what is challenging for a reader with a particular background (for example, a non-native reader at a lower level of language proficiency) would not necessarily be challenging for readers with other backgrounds (for example, more proficient readers) @xcite . a number of factors may contribute to that, including the reader's age and level of language proficiency, among others @xcite . ls systems often aim to address the needs of specific reader populations, such as children, non-native speakers, or readers with particular cognitive impairments. thus, personalization in ls typically results in specialized simplification tools aimed at certain groups of readers @xcite @xcite , with only a few systems addressing adaptation to the readers' needs in a more dynamic way @xcite @xcite . despite cwi being one of the key steps in an ls pipeline in need of adaptation to readers' profiles, this is rarely addressed in practice @xcite . for instance, existing and widely used datasets on cwi present a homogeneous view on word complexity, merging annotations from various groups of readers @xcite . from the cognitive perspective, little is still known about the challenges that particular readers face when developing their reading skills and about the factors contributing to their vocabulary acquisition. in this paper, we investigate factors focusing on the two key background aspects in the development of reading abilities: whether a reader is a native speaker of the language, and if not, what is the reader's level of language proficiency. we use the data from @xcite , which contains english sentences where complex words are annotated by native and non-native speakers of english, spanning three different levels of language proficiency. we investigate which aspects contribute to the notion of lexical complexity for readers with different backgrounds and how the notion of complexity changes depending on the proficiency levels of the non-native readers. in our paper we make the following contributions: ## data the most comprehensive cwi dataset to date was released by @xcite and further used in the cwi shared task 2018 @xcite . this dataset has been annotated for complex words across a number of languages, including english, german, and spanish. in this paper, we use the english portion of the data with the information about annotators' backgrounds foot_0 . the dataset contains texts from 3 different sources: professionally written news articles (news), amateurishly written news articles (wikinews), and wikipedia articles. the annotation was performed using the amazon mechanical turk platform, where a total of 20 annotators, 10 native speakers and 10 nonnative speakers, were asked to mark words that they deemed complex for a given target readership, particularly children, language learners, and people with reading impairments. the workers were presented with text, consisting of 5 to 10 sentences (figure 1 ), and were asked to select lexical items that they found complex (figure 2 ). workers use their mouse pointer to highlight the complex units. the complex words or phrases included content words (e.g., nouns, verbs, adjectives, and adverbs) and phrases up to 50 characters in length. in this dataset, the complex units are considered if they are selected by at least one worker @xcite . non-native speakers of english were asked to report their proficiency levels (beginner, intermediate, advanced). for our experiments, we concentrate on complex words only and disregard complex phrases. a break-down of proficiency labels for words (across all genres) is presented in ## method we firstly show that when predicting word complexity, the needs of sub-groups differ and are best predicted using models targeting them specifically. we demonstrate that the best performing models for a sub-group are trained with the annotations of that group using a classical machine learning approach. secondly, we analyse the correlation of features with the number of annotators who found the word complex for both native and non-native groups. finally, we investigate how the distributions of features vary for words marked as complex across audiences. ## results in all experiments, 5-fold stratified cross validation is performed and the average scores across folds presented. we perform experiments by training with native complexity annotations and observe the performance across non-native proficiency groups. the results of these are shown in ## conclusions textual complexity is a subjective phenomenon that is dependent on the intended audience. we show that when considering lexical complexity, the best performing cwi models for a target proficiency level are trained with the labels of that sub-group. we investigate which features correlate most with the absolute number of native and non-native annotations as well as observe how the distributions of classic complexity features are dependent on the intended audience. we find strong similarities between the notion of word complexity for advanced non-native readers and native readers. finally, we release a dataset for cwi with proficiency subgroup annotations. in future work we plan to collect additional annotations across classes, especially concentrating on beginners. we would also like to investigate how effective informedthresholding techniques for cwi are compared to high resource systems.