archer at S em E val-2021 Task 1: Contextualising Lexical Complexity (2021)

Abstract: Evaluating the complexity of a target word in a sentential context is the aim of the Lexical Complexity Prediction task at SemEval-2021. This paper presents the system created to assess single words lexical complexity, combining linguistic and psycholinguistic variables in a set of experiments involving random forest and XGboost regressors. Beyond encoding out-of-context information about the lemma, we implemented features based on pre-trained language models to model the target word’s in-context complexity.

## introduction lexical complexity prediction is the task aiming at evaluating the complexity of a word in context, modeling a crucial aspect of reading comprehension. complex words can slow down the reading process; there is a well-known correlation between a word's difficulty and the time spent looking at it, as emerging from eye-tracking experiments @xcite . assessing the complexity of a specific word in context is a crucial prerequisite for nlp systems aiming to evaluate a text's readability and produce a simplified version of it. it is a prerequisite for text simplification systems based on lexical substitutions and can be the starting point to tailor a text to the user's needs. it is a topic worthy of investigation from multiple points of view. in the past, datasets containing crowdsourced evaluations of lexical items' complexity @xcite have been used in evaluation campaigns. in this paper, we introduce the system used to assess single english words lexical complexity at semeval-2021 lexical complexity prediction task @xcite . based on previous approaches to this issue, we combine linguistic and psycholinguistic variables, using a random forest regressor and an xgboost regressor. the majority of the variables encode information about the word out-of-context (e.g., frequency, number of letters, age of acquisition) without considering the sentential context and how it can affect an item's complexity. to approximate in-context complexity, we consider the cloze probability of a word as its probability to complete a particular sentence frame. we experiment with different language models in a masked word prediction framework, taking into account the first ten most probable words occurring in that context. ## related works a wide range of approaches has been used for lexical complexity prediction in past evaluation campaigns. however, previous tasks focused on classification since the proposed datasets labeled words in context as easy or difficult. including more classes makes the task more difficult. @xcite investigate the role of word embeddings in lexical complexity prediction for french words, using as training sets two french lexical resources that encode the distribution of words across different levels of difficulty. according to the authors, the task is influenced by the context of use of the words. word embeddings, encoding contextual information, can be helpful to determine the lexical complexity of target words. the authors experimented with different neural network settings (with and without hidden layers), using as features the number of characters, the number of phonemes, and the log frequency in a corpus of film subtitles plus word embeddings trained on wikipedia with fasttext. however, the combination of word embeddings with such features does not improve the results compared with other sets of features that always contain frequency, a crucial indicator of lexical complexity. in past evaluation campaigns, there have been occasional attempts at incorporating word embeddings models in the automatic evaluation process, with the assumption that lexical complexity should be evaluated as a contextual variable. however, better results have been obtained considering just static, out-of-context properties of lemmas. more recently, the trend to use embeddings from language models to predict psycholinguistic variables can provide insights about how to incorporate them in experiments aiming at understanding the complexity of human comprehension @xcite . however, if we frame lexical complexity as a measure strongly dependent on words' psycholinguistic properties, we should recognize that past computational efforts for predicting word norms did not take into account the role of context @xcite . static word embeddings such as word2vec have been used to predict values of psycholinguist norms usually assessed in experimental settings @xcite . more recent transformed-based language models that consistently incorporate contextual knowledge have not yet been considered for this task. ## the dataset the lexical complexity prediction shared task at semeval-2021 (lcp-2021) @xcite is based on an english dataset with a 5-point likert scale annotation. the complexity score is similar to that included in another dataset @xcite . it ranges from very easy for very familiar words to very difficult (unclear words that an annotator had never seen before). annotators are explicitly invited to evaluate the role of the sentence in inferring the meaning of the word. the task is structured into two sub-tasks: the task is inspired by two previous competitions @xcite about boolean complex word identification, aiming at identifying which words are likely to be considered complex or domain mean std dev bible 0.296 0.132 europarl 0.287 0.109 biomed 0.325 0.152 we propose a system for sub-task 1, encoding for each target word numerical values concerning a set of variables described in section 4. we do not propose a system for sub-task 2. ## out-of-context and in-context lexical complexity the lexical complexity of a word can be represented as an out-of-context property of a lemma or an in-context property of a word. following the first approach, the same lemma has a fixed lexical complexity value, depending on features such as the number of characters or senses in wordnet (see the list of out-of-context features below). when considering the word in context, its disambiguation could affect its complexity rating because a sense could be more complex than the others (for example, when a word is used in its specialized sense). however, because of the lack of methodologies for assessing senses' complexity and the unsatisfactory performance of word sense disambiguation systems, the role of senses' complexity for lexical complexity prediction can not be investigated. several systems participating at cwi2018 took into account the role of context, focusing on the whole sentence where target items occur. however, quite interestingly, one of the best models @xcite does not consider the influence of the textual context for determining the target word's complexity. we provide the list of out-of-context features used in our system: a word x is difficult in a sentential context if the reader has never encountered it. the relative frequency should be sufficient to explain the perceived complexity of x). a word can also be difficult if its meaning in that specific context is not the most common one, i.e when a specialized sense is accessed or a metaphorical meaning is created. if we know a word by the company it keeps, we do not know a word when its company is somewhat eccentric. lexical complexity as an in-context property can be modeled considering the influence of the surrounding text. there are two ways to model the textual context's influence on the lexical complexity of a target word: local context (a window span surrounding the target word) and global context (the whole sentence). in the first case, words surrounding the target word can increase the overall complexity in that text span. in the second case, the probability of a word in a masked word prediction task that concerns the whole sentence can be a good approximation of the intuition that words semantically difficult to generate are more complex than the simpler ones. we implemented two types of variables as in-context features to address in-context lexical complexity: the pearson correlations among each feature and target word's lexical complexity reveal that word frequencies are the most relevant features, especially frequencies extracted from children movies' subtitles (see figure 1 ). the age of acquisition of words is another variable strongly correlated with the complexity of the target words (@xmath0.55). 5 experiments and -to avoid overfitting -on multiple training-test splits with test sets mimicking the trial set's composition. we experimented with a system based on a random forest regressor (rf), and a system based on an xgboost regressor (chen and guestrin, 2016), both implemented in sklearn. for the rf regressor, we choose to measure the quality of a split with mean absolute error. we also normalized the features with the standard scaler function (scaling each feature between 0 and 1). we obtained comparable results, with random forest regressor performing slightly better for several training-trail splits. for this reason, in-context features emerge as useless from these results; however, testing with different trial sets, we infer that this set of features could improve the performance (see ## conclusions this paper briefly reports the system created to predict single words' complexity score for the lexical complexity prediction shared task at semeval-2021 (lcp-2021). our system ranked 22 out of 54 for this sub-task, with slightly inferior results to the ones obtained on trial sets. the significative role of frequencies extracted from different corpora paves the way to further investigations in this direction. encoding in-context complexity as a variable related to pre-trained language models' predictions had no significant impact on the results. however, in-context complexity could be modeled in different ways. experimenting with how in-context target word's complexity changes depending on the frequencies of the surrounding words is a future analysis topic.