S tanford ML ab at S em E val-2021 Task 1: Tree-Based Modelling of Lexical Complexity using Word Embeddings (2021)

Abstract: This paper presents our system for the single- and multi-word lexical complexity prediction tasks of SemEval Task 1: Lexical Complexity Prediction. Text comprehension depends on the reader’s ability to understand the words present in it; evaluating the lexical complexity of such texts can enable readers to find an appropriate text and systems to tailor a text to an audience’s needs. We present our model pipeline, which applies a combination of embedding-based and manual features to predict lexical complexity on the CompLex English dataset using various tree-based and linear models. Our method is ranked 27 / 54 on single-word prediction and 14 / 37 on multi-word prediction.

## introduction the rapid expansion of social media and other online channels has made readable information available at an astounding rate. however, the accessibility of this information is often limited by the complexity of this information, especially among readers with low literacy levels in the language of the text and those with reading disabilities. furthermore, even to the average reader, specialized jargon found in governmental documents and scientific fields is often difficult to decipher. systems to guide these users may redirect readers to more easily comprehensible sources, convert the text to simpler wording, or provide additional information about what difficult words mean. the development of such systems is benefited by the ability to evaluate the complexity of sections of the text. while there is currently a large amount of available text data, very little of it is labeled with word complexity; automating the labelling process would make much more data available to aid the development of nlp systems in tasks such as text simplification. multiple features of a word can affect lexical complexity. in addition to a word's frequency, length and syllable count, the context in which a word is found is likely to affect its understandability. the additional factor of the reader's proficiency in a language makes this task complex as many words have a highly variable complexity. in this paper, we describe our model that predicts single-and multi-word lexical complexity scores. token complexity can differ based on the complexity of the token both with and without context. for example, for one instance, the token river was rated to have a complexity of 0.0, while jurisprudence had a complexity of around 0.672 for another instance. however, token complexities can also change based on the context from which it came from. for example, the token wisdom was given a complexity of 0.125 when it was associated in the sentence "the rod of correction gives wisdom, but a child left to himself causes shame to his mother." however, the same token was given a significantly higher complexity score of 0.368 when associated with the sentence "for in much wisdom is much grief; and he who increases knowledge increases sorrow." given that glove embeddings @xcite store semantic meaning of single words, we chose to use glove embeddings to represent both tokens and sentences. with this approach, we determine that despite contextual variation, inherent properties of the token itself are sufficient to explain much of the variance in lexical complexity. ## experimental setup the train and validation dataset splits provided were used in our experimental setup. in addition, we used k-fold validation to reduce overfitting. using k-fold, we split the training set into k smaller sets arbitrarily, train using k -1 folds, and crossvalidate with the remaining fold in the train set. this reduces leakage from the validation set into the model so that we can accurately validate our methods. task predictions were evaluated using pearson correlation, though spearman correlation, mean absolute error, mean squared error, and r-squared were also reported. we compared the performance of our own models using pearson correlation to keep one consistent evaluation metric. ## conclusion in this paper we describe tree-based modelling of words in context to predict lexical complexity. we find that lexical complexity is already embedded in glove representations of words and that complex architectures provide some increase in predictive performance. for future work, we suggest taking additional contextual features into account, such as the proximity of each neighboring word. we also suggest looking into newer transformer models to represent contextual embeddings. as larger bodies of text become widely available to wide audiences for public consumption, we are hopeful that such systems will help readers identify suitable texts for their reading level and help build systems that can tailor text to varied reading levels, allowing for greater accessibility.