P oly U CBS -Comp at S em E val-2021 Task 1: Lexical Complexity Prediction ( LCP ) (2021)

Abstract: In this contribution, we describe the system presented by the PolyU CBS-Comp Team at the Task 1 of SemEval 2021, where the goal was the estimation of the complexity of words in a given sentence context. Our top system, based on a combination of lexical, syntactic, word embeddings and Transformers-derived features and on a Gradient Boosting Regressor, achieves a top correlation score of 0.754 on the subtask 1 for single words and 0.659 on the subtask 2 for multiword expressions.

## introduction the notion of complexity has often been debated in linguistics and, depending from the disciplines, it might have different meanings. in linguistic typology, for example, complexity is generally studied as a property of the language system as a whole, it is conceived as the number of (morphological, syntactic, semantic etc.) distinctions that a speaker has to master, and it is assessed by comparing different languages @xcite . on the other hand, in the perspective of psycholinguistics and cognitive science, the notion of complexity can be described as the difficulty encountered by language users while processing concrete linguistic realizations (sentences, utterances etc.) @xcite @xcite @xcite . finally, in the computational linguistics community, the assessment of complexity at the lexical level is often related to readability applications @xcite , with the goal of determining if a word in a given text will be difficult to understand for the language users. such applications are extremely useful for second language learners, for speakers with relatively low literacy and for people with reading disabilities, helping to tailor the difficult level of the texts to the needs of the target users. task 1 of semeval 2021 @xcite ) aims at the development of systems for the estimation of lexical complexity in context, both for single words and for multiword expressions. the organizers provided two datasets with the target words in a sentence context, with annotations consisting of a mean of the complexity ratings assigned by humans. in our paper, we present the system developed by the polyu cbs-comp team for the competition. our top system achieves a pearson correlation of, respectively, 0.754 on the single words dataset and 0.659 on the multiword expressions one. ## related work in the earliest shared task on the lexical complexity problem, organized in 2016 @xcite , complexity was defined as a binary variable: given a word in context, the word will be judged as complex or not. of course, this was a simplifying assumption, since there might be many situations where the boundary is not a clear-cut one, and annotators would rather indicate a value in a continuous scale. moreover, the "complex" words in the data only needed to be categorized as such by just one of the annotators. a further study by @xcite analyzed the output of the participating systems, showing that modeling complexity as binary actually hindered their performance. a second iteration of the shared task was organized in 2018 @xcite , this time features two separate subtasks: the traditional binary classification task, where systems had to predict whether one word was complex or not, and a regression task, where systems had to estimate the probability that an annotator would have considered a given word as complex. recently, @xcite have introduced complex, a new gold standard for the estimation of lexical complexity in context for english: the corpus, including sentences from different textual genres, is annotated with the mean complexity ratings for the target words. as a preliminary evaluation, the authors presented the results of a linear regression model trained on sets of features including word and sentence embeddings and some handcrafted features that are traditionally associated to complexity, such as frequency, word length and syllable count. the best scores, in terms of mean absolute error, were obtained when using only the latter set of features, while models based on the dimensions of the embeddings were lagging behind. ## datasets the datasets for the shared task are part of the complex corpus, which has been published and described by @xcite . the annotated sentences were collected using three different corpora: the europarl corpus @xcite , which includes the proceedings of the european parliament; the craft biomedical corpus @xcite ; and the bible, in the modern version of the world english bible translation @xcite . the organizers selected targets as either single words (sub-task 1) or multiword expressions (sub-task 2), and the datasets include also multiple examples with the same target, as different contexts can determine different complexity values. as for the multiword expressions, they were identified via syntactic patterns, being either adjective-noun or noun-noun phrases. in a first phase, the organizers released a training data of 7661 samples for the single words track and 1517 samples for the multiword expressions track, together with a trial/validation dataset of 420 and 99 samples, respectively. later, they released a test set of 917 samples for the single words track and 184 samples for the multiword expressions track. examples of the instances are shown in ## evaluation for both the single words and the multiword expressions track, we used the same set of features as input for a regression algorithm. in the multiword expressions track, we computed the value of the features for each of the two words in the target expression and then we took the average. ## results we evaluated our system for two subtasks based on given trial datasets. for each regressor, we tuned hyper-parameters according to each subtask. performance evaluation has been carried out in two aspects: the assessment of the overall correlation with human ratings and the analysis of the contribution of the features. ## conclusion in this paper, we presented the polyu cbs-comp system for lexical complexity prediction, which took part in the semeval shared task 1. our method, based on a combination of lexical, syntactic, embeddings and transformers features, achieved a 0.754 correlation on single words and 0.659 on multiword expressions, when using gradient boosting as a regression algorithm. traditional hand-crafted features, followed by transformer-based ones, seem to give the strongest contribution to the classification performance, which is further improved by adding feature in-teractions to the input for the regressor. for future studies on lexical complexity, we plan to further exploit the text genre information, for example by adding domain-adapted language model features @xcite to the information available to our models.