CLULEX at S em E val-2021 Task 1: A Simple System Goes a Long Way (2021)

Abstract: This paper presents the system we submitted to the first Lexical Complexity Prediction (LCP) Shared Task 2021. The Shared Task provides participants with a new English dataset that includes context of the target word. We participate in the single-word complexity prediction sub-task and focus on feature engineering. Our best system is trained on linguistic features and word embeddings (Pearson’s score of 0.7942). We demonstrate, however, that a simpler feature set achieves comparable results and submit a model trained on 36 linguistic features (Pearson’s score of 0.7925).

## introduction lexical complexity relates to complexity of words. its assessment can be beneficial in a number of fields, ranging from education to communication. for instance, lexical complexity studies can assist in providing language learners with learning materials suitable for their proficiency level or aid in text simplification @xcite . these studies are also a central part of reading comprehension, as lexical complexity can predict which words might be difficult to understand and could hinder the readability of the text. lexical complexity studies typically make use of natural language processing and machine learning methods @xcite . previous similar studies focus on complex word identification (cwi), which is a process of identifying complex words in a text @xcite . in this case, lexical complexity is assumed to be binary -words are either complex or not. lcp shared task 2021 addresses this limitation by introducing a new dataset designed for continuous rather than binary complexity prediction @xcite . in this paper, we describe a single-word lexical complexity prediction system. our goal is to demonstrate that a simple system can achieve results comparable to more complex ones. therefore, we focus on feature engineering rather than model tuning. ## lcp shared task 2021 setup the lcp shared task 2021 aims to predict the complexity value of words in their context. it is divided into two sub-tasks: predicting the complexity score of 1) single words and 2) multi-word expressions. in this paper, we present a system for the first sub-task. the shared task uses the complex corpus @xcite . in addition to the target word, it includes contextual information which is represented by a sentence where the word appears and its source or domain: bible (christodouloupoulos and steedman, 2015), europarl @xcite or biomedical texts @xcite . each word in the dataset is evaluated by around 7 annotators from english speaking countries. the complexity labels are based on a 5-point likert scale scheme (very easy to very difficult). the final dataset consists of 7,662 training and 917 testing instances. the shared task baseline system uses a linear regression model. it is trained on log relative frequency and word length features, resulting in a mean absolute error (mae) of 0.0867. ## methodology in this section, we describe the methodology that we follow in the design of our system, including the used data, feature engineering and the training steps. the study relies on an in-depth experimentation with features. we aim to find out which linguistic information is the best predictor of lexical complexity. ## results in this section, we present our results and discuss the key findings. all discussed systems are trained with the random forest classifier. ## conclusion in this paper, we have described the design of our system submitted to the lcp shared task 2021 and discussed the key findings of our feature engineering approach. we aimed to design a simple system that would not require much classifier tuning or complex feature computations. our two best models are trained on the random forest classifier with the default hyperparameters. the best system is trained on a 112 feature set which includes word embeddings. the second best system is trained on a simple 36 linguistic feature set. we submit the simple system since the performance difference between the two systems is not significant. the model is placed in the upper half of the shared task rankings for the single-word prediction subtask (pearson's score of 0.7588), demonstrating how a simple approach can achieve high performance results. further analysis of the feature ablation studies confirms that word frequencies seem to be the most informative among all features. we also observe that even though including contextual information does improve the overall result, the performance differences are small. future research might therefore look into including more contextual information than one sentence. in addition, the perception of word complexity differs from reader to reader. future work could target specific reader groups, such as people with dyslexia or second language learners. in this case, the relevant background information of the readers should be included in the annotation and experimentation processes.