Alejandro Mosquera at S em E val-2021 Task 1: Exploring Sentence and Word Features for Lexical Complexity Prediction (2021)

Abstract: This paper revisits feature engineering approaches for predicting the complexity level of English words in a particular context using regression techniques. Our best submission to the Lexical Complexity Prediction (LCP) shared task was ranked 3rd out of 48 systems for sub-task 1 and achieved Pearson correlation coefficients of 0.779 and 0.809 for single words and multi-word expressions respectively. The conclusion is that a combination of lexical, contextual and semantic features can still produce strong baselines when compared against human judgement.

## introduction lexical complexity is a factor usually linked to poor reading comprehension @xcite and the development of language barriers for target reader groups such as second language learners @xcite or native speakers with low literacy levels, effectively making texts less accessible @xcite . for this reason, complex word identification (cwi) is often an important sub-task in several human language technologies such as text simplification @xcite or readability assessment @xcite . the lexical complexity prediction (lcp) shared task of semeval-2021 @xcite proposes the evaluation of cwi systems by predicting the complexity value of english words in context. lcp is divided into two sub-tasks: subtask 1, predicting the complexity score for single words; sub-task 2: predicting the complexity score for multi-word expressions. in our participation in both sub-tasks, we treat the identification of complex words as a regression problem, where each word is given a score between 1 and 5, given the sentence in which it occurs. in order to do this, we have evaluated sub-sets of word and sentence features against different machine learning models. our best submissions achieved pearson correlation coefficients of 0.779 and 0.809 for single words and multi-word expressions respectively. in section 2 we review related work for this task. section 3 and 4 introduce the data and feature engineering approaches respectively. in section 5 the performance of different machine learning models is analysed. in section 6 we present the obtained results. finally, in section 7 we draw our conclusions and outline future work. ## related work previous cwi studies applied to the english language have relied mostly on word frequencies, psycholinguistic information @xcite , lexicons and other word-based features such as number of characters or syllable counts @xcite , which considered in most cases the target word in isolation. in order to address the limitations of word-level approaches more recent work made use of contextual and sentence information such as measuring the complexity of word n-grams @xcite , applying language models @xcite or treating the problem as a sequence labelling task @xcite . in this paper, we not only evaluate many of the traditional word-based features found in the literature but we also pay attention to the context surrounding the target by generating additional bigram and sentence features. in the end, we demonstrate that a careful selection of simple features is still competitive against more novel approaches for this task. ## datasets complex @xcite , which was the official dataset provided by the organizers, contains complexity annotations using a 5-point likert scale for 7,662 words and 1,517 multi-word expressions (mwe) from three domains: the bible, europarl, and biomedical texts. external datasets and models are historically allowed and used in semeval as a way of complementing the original training set. likewise, based on previous experiences, external resources can also correlate better with the evaluation labels than the official task resources in certain scenarios (mosquera, 2020). for this reason, related datasets from previous cwi shared tasks such as cwi 2016 @xcite and cwi 2018 @xcite were considered and evaluated as both extra training data and for deriving additional features. however, the performance of our models during the validation step not only didn't improve but worsened when attempting to use them. ## feature engineering the 51 features used in order to detect the complexity of single words and each component of mwes are as follows: word length (word len): the length in characters of the target word. syllable count (syl count): target word syllable count. morpheme length (morpheme len): number of morphemes for the target word. google frequency (google freq): the frequency of the target word based on a subset google ngram corpus foot_0 . wikipedia word frequency (wiki freq1): the frequency of the target word based on wikipedia foot_1 . wikipedia document frequency (wiki freq2): the number of documents in wikpedia where the target word appears. complexity score (comp lex): complexity score for the target word from a complexity lexicon @xcite . number of morphemes (morpheme len): number of morphemes in the target word. zipf frequency (zip freq): the frequency of the target word in zipf-scale as provided by the wordfreq @xcite python library. kucera-francis word (kucera francis): kucera-francis @xcite frequency of the target word. kucera-francis lemma (st kucera francis): kucera-francis @xcite frequency of the target word lemma. is stopword (stop): true if the target word is an stopword. is acronym (acro): heuristic that is set to true if the target word is a potential acronym based on simple casing rules. average age of acquisition (age): at what age the target word is most likely to enter someone's vocabulary @xcite . average concreteness (concrete): concretedness rating for the target word @xcite . lemma length (lemma len): lemma length of the target word. word frequency (coca) (word freq): frequency of the target word based on the coca corpus @xcite . lemma frequency (coca) (lemma freq): frequency of the lemmatized target word based on the coca corpus @xcite . number word senses (wn senses): number of senses of the target word extracted from wordnet @xcite . number of synonyms (synonyms): number of synonyms of the target word from wordnet. number of hypernyms (hypernyms): number of hypernyms of the target word from wordnet. number of hyponyms (hyponyms): number of hyponyms of the target word from wordnet. wordnet min-depth (wn mindepth): minimum distance to the root hypernym in wordnet for the target word. wordnet max-depth (wn maxdepth): maximum distance to the root hypernym in wordnet for the target word. ## machine learning approach since the labels in the training dataset were continuous we have modelled both sub-tasks as regression problems. for sub-task 1, we made use of light-gbm (lgb) @xcite implementation of gradient tree boosting. minimal hyper-parameter optimization was performed against our development set, using a 0.01 learning rate and limiting the number of leaves of each tree to 30 over 500 boosting iterations. for sub-task 2, the complexity score of each mwe component was obtained by using a linear regression (lr) model and averaged with equal weights. by examining the feature importance for both the lgb model in figure 2 and the lr model in figure 3 we can observe several sentence readability features being identified as top contributors. while some degree of correlation between the complexity of the sentence and the target word was expect a priori, a machine learning model can also use sentence-level complexity as a predictor of formality and genre @xcite , thus being able to differentiate between the different sub-corpora present in the training data as seen in figure 1 . ## results for sub-task 1, we have evaluated the performance of both linear and tree ensembles using the provided trial set and a randomly selected holdout with 30% of the training data as development set. the best performing model was gradient boosting. see ## conclusion and future work in this paper, we present the system developed for the lexical complexity prediction task of semeval 2021. even though most of the features we made use of are relatively common in previous works, we demonstrate that a careful selection of lexical, contextual and semantic features at both target word and sentence level can still produce competitive results for this task. in a future work we would like to explore different neural network architectures and automated machine learning (automl) approaches.