LCP - RIT at S em E val-2021 Task 1: Exploring Linguistic Features for Lexical Complexity Prediction (2021)

Abstract: This paper describes team LCP-RIT’s submission to the SemEval-2021 Task 1: Lexical Complexity Prediction (LCP). The task organizers provided participants with an augmented version of CompLex (Shardlow et al., 2020), an English multi-domain dataset in which words in context were annotated with respect to their complexity using a five point Likert scale. Our system uses logistic regression and a wide range of linguistic features (e.g. psycholinguistic features, n-grams, word frequency, POS tags) to predict the complexity of single words in this dataset. We analyze the impact of different linguistic features on the classification performance and we evaluate the results in terms of mean absolute error, mean squared error, Pearson correlation, and Spearman correlation.

## introduction lexical complexity prediction (lcp) is the task of predicting the complexity value of a target word within a given text @xcite . complexity within lcp is used as a "synonym for difficulty" @xcite 1 . a complex word is therefore a word that a target population may find difficult to understand. various lcp systems have been designed to identify words that may be found to be complex for children @xcite , language learners @xcite , or people suffering from a reading disability, such as dyslexia @xcite . these systems have been utilized within assistive language technologies, lexical simplification systems, and in a variety of other applications. lcp is related to complex word identification (cwi) @xcite . cwi is modeled as a binary classification task by assigning each target word with a complex or non-complex label. the shortcomings of modeling lexical complexity using binary labels have been discussed in previous work @xcite , motivating the organization of semeval-2021 task 1: lexical complexity prediction. 2 lcp models complexity in a continuum and the goal is to predict a target word's degree of complexity by assigning it a value between 0 and 1. this value may then correspond to one of the following labels: very easy (0), easy (0-0.25), neutral (0.25-0.5), difficult (0.5-0.75), or very difficult (0.75-1) @xcite . in this paper, we describe (in detail in section 4) the lcp-rit entry to semeval-2021 task 1. we approached lcp from a feature engineering perspective with a particular focus on the adoption of psycholinguistic features, such as average ageof-acquisition (aoa), familiarity, prevalence, concreteness, and arousal, alongside the use of prior complexity labels. our submitted system utilized a combination of these linguistic features, which we compared to a baseline model that only used statistical features: word length, word frequency and syllable count @xcite . on our training dataset, our submitted system achieved a mean absolute error (mae) of 0.067, mean squared error (mse) of 0.007, person correlation (r) score of 0.779, and a spearman correlation (ρ) score of 0.724. this surpassed our baseline model's performance by a mae of 0.008, mse of 0.003, as well as r and ρ scores of 0.075 and 0.062 respectively. ## related work before semeval-2021 task 1: lcp, two cwi shared tasks were organized at one semeval-2016 and the other at bea-2018 @xcite . while the first cwi provided participants with an english dataset, the second provided a multilingual dataset. the systems submitted to the english track of the second shared task @xcite performed better overall than the previous task @xcite , probably due to the properties of the two datasets @xcite . state-of-the-art neural net models and word embedding models performed worse than conventional models such as decision trees (dts) and random forests (rfs) @xcite . among the conventional models, the use of statistical, character n-gram, and psycholinguistic features was found to be highly effective in improving cwi performance @xcite @xcite . among the best performing systems in cwi 2018, @xcite used an ensemble of classifiers. they found that during their system's development, the boosting classifier ad-aboost, a random forest classifier, or a combination of both classifiers achieved the highest performance. these systems used multiple features such as the word's grammatical category, google character n-gram frequency as well as a range of psycholinguistic features @xcite . of the remaining systems, aroyehun et al. (2018) and hartmann and borges dos santos (2018) utilized statistical features, such as word length and number of syllables, psycholinguistic features such as familiarity, age of acquisition, concreteness, and imagery scores, and word n-grams. hartmann and borges dos santos (2018) compared the performance of tree ensembles to a convolutional neural network (cnn). they found that their tree ensembles performed better than their cnn, especially when the target expression contained more than three words @xcite . ## task and dataset the lcp shared task organizers provided participants with the complex corpus, an english multidomain dataset with sentences from the bible, the european parliament proceedings, and a collection of biomedical texts. a pool of annotators, using a five point likert scale, labeled the complexity of single words and multi-word expressions in com-plex @xcite . taking advantage of the annotation of single words and multi-word expressions, the lcp shared task was divided into two sub-tasks as follows: we chose to participate in sub-task 1. sub-task 1's training dataset contained 7,662 instances with its test dataset having 917 instances. 20% of the training dataset was used to test our system's performance during development. sub-task 1 received 54 system submissions. ## conclusion we carried out multiple experiments evaluating the impact of linguistics features in lcp using the complex dataset for english. we have shown that several psycholinguistic features help with lcp. average aoa, prevalence and concreteness were all found to be beneficial, whereas mrc familiarly, mrc concreteness and prior complexity labels were proven to be redundant. we would like to explore other features described in @xcite . in terms of performance, we believe that the multiple features we tested allowed us to get close to the maximum performance for this dataset using regression. a possible alternative for better performance is to test state-of-the-art transformer models. furthermore, we are interested in looking at the performance of these features for lcp in languages other than english and for multilingual datasets.