C omp NA at S em E val-2021 Task 1: Prediction of lexical complexity analyzing heterogeneous features (2021)

Abstract: This paper describes the CompNa model that has been submitted to the Lexical Complexity Prediction (LCP) shared task hosted at SemEval 2021 (Task 1). The solution is based on combining features of different nature through an ensambling method based on Decision Trees and trained using Gradient Boosting. We discuss the results of the model and highlight the features with more predictive capabilities.

## introduction complex word identification (cwi) is a task focused on the detection of complex (not easy to understand) words or expressions. one of the challenges of natural language-based systems is to provide informative content that is tailored to the needs of users in terms of content and level of understanding. for this aim, predicting lexical complexity plays a crucial role in simplifying the text so that it can be more easily understood by people with low literacy levels (for example children) or non native speakers @xcite . the interest of the computational linguistics community on this topic has grown in recent years. indeed, semeval 2016 presented a challenge specifically on cwi @xcite . the task proposed to build models capable of predicting whether a word was easy or not for non-native english speakers. in this case, the proposed dataset was annotated with a binary label. in 2018 the cwi task was re-proposed at the workshop for building educational applications (bea) @xcite . the data was again annotated with binary labels but samples from more languages were considered, specifically english, german and spanish. in addition to the word classification task, a sub-task where the participants had to predict the probability of a word being complex was added. the best performing models for both tasks were based on ensembling techniques using features that were carefully selected @xcite . in this context, the lexical complexity prediction (lcp) shared task hosted at semeval 2021 proposes a challenge where the data is annotated according to the degree of complexity @xcite . this type of annotation allows regression models that predict a complexity index rather than just a binary label. this task is split into two sub-tasks, the first one is about the prediction of the complexity of single words and the second is about the prediction of the complexity of multi word expressions. in this work we present our submission to both the sub-tasks with a model that aggregates a large set of heterogeneous features that can capture a wide variety of linguistics aspects (morphological, semantic, distributional and lexicon-based) in a regression model based on gradient boosting. we also present an in-depth analysis of which features are more important for the model. in section 2 we present a description of the task and the data available. in section 3 we introduce the model. in section 4 we show an analysis of the most relevant features for the model. in section 5 we present the results. finally, in section 6 we draw some conclusions. the data released for sub-task 1 is made of 9000 sentences where the complexity of a single word was annotated considering its context. the complexity has been annotated using a 5-point likert scale (from 1 to 5 corresponding to very easy, easy, neutral, difficult and very difficult) with the values scaled in the range 0-1. here is one example of a sentence annotated in the dataset: "the structural gh gene itself and stat5b are excellent candidates." in this sentence, the word "candidates" is annotated with a complexity of 0.11 (the average in the dataset is 0.30). the goal of sub-task 1 is to predict the complexity of a single word given the sentence and the word to evaluate. the data for sub-task 2 is made of 1800 sentences where expressions of two words are annotated as already described. here is one sample extracted from the dataset: "i invite the president to ask all members who are in the pension fund to say so orally, in plenary, immediately, because they have a direct interest in what is to be discussed." in this sentence, the expression "direct interest" is annotated with a complexity of 0.40 (the average in the dataset is 0.42). the goal of sub-task 2 is to predict the complexity of an expression like the one above given the sentence and the expression to evaluate. for both sub-tasks, the data was selected from three different corpora: ## compna model the main idea behind our model is to aggregate many diverse features that can capture a wide variety of linguistics aspects in a regression model that offers the ability to interpret which of them are more influential. ## features importance in order to bring insights into how the model works, we study the contribution of the features. tables 1 and 2 report the importance of the features that are influential for the prediction, respectively, per group and single feature. the feature importance reflects the number of times a feature is selected for a split for one of the decision trees in the model, weighted by the improvement as a result of the split, and averaged over all the trees in the model @xcite . looking at the importance of the features grouped by type we find morphological and distributional ones at the top. considering the most important syllables, we notice the syllable "ca" which is associated with words from a wide range of complexities (the most complex "cause", and the least complex "catalepsy"). and the syllable "ver" which is associated with low complexity words (the most complex being "reversal", and the least complex being "river"). this analysis further confirms the hypothesis outlined in @xcite that distributional and morphological aspects of the words have a tight bond with the complexity. the analysis also highlights one of the weak points of the model, it overlooks features related to the context of the word. ## results in ## conclusions in this paper, we presented a solution to predict the complexity of single and multi word expressions combining a large number of features from a diverse nature. the model achieves average results and, more importantly, offers the ability to quantify the relevance of the features for the prediction. thanks to this ability we have shown that the frequency of occurrence and the morphology of the words are key predictors of complexity for the data considered. from our analysis, it is clear that our model overlooks features related to the context of the word and we would like to improve it under this point of view in our future efforts.