katildakat at S em E val-2021 Task 1: Lexical Complexity Prediction of Single Words and Multi-Word Expressions in E nglish (2021)

Abstract: This paper describes systems submitted to Se- mEval 2021 Task 1: Lexical Complexity Prediction (LCP). We compare a linear and a non-linear regression models trained to work for both tracks of the task. We show that both systems are able to generalize better when supplied with information about complexities of single word and multi-word expression (MWE) targets simultaneously. This approach proved to be the most beneficial for multi-word expression targets. We also demonstrate that some hand-crafted features differ in their importance for the target types.

## introduction semeval-2021 task 1 is the task of lexical complexity prediction (lcp) @xcite . the goal of the task is to assign a target in a context a continuous value ranging between 0 and 1, where 1 indicates complete unintelligibility and 0 signals perfect familiarity as perceived by a native speaker. the task has two tracks: predicting the complexity score of single words and predicting the complexity score of multi-word expressions (mwe). such a task can be useful in applications like text simplification or automatic language proficiency evaluation. the complex dataset @xcite ) used in this task is the first english dataset for the task of lcp. the dataset contains single words and mwes annotated with their lexical complexity score in a specific context. the annotations were provided by native speakers of english. the targets and their contexts were obtained from texts of different domains: the bible, europarl, and biomedical texts. the dataset opens several avenues for research. for instance, how does the perceived complexity of single words and mwes differ? how does context affect the lexical difficulty of a target? how does text genre affect comprehensibility of words? we were interested if there is a difference in performance when using the same representation methods for single words and mwes. it was decided to approach both tracks as the same problem. we did not distinguish between mwes and single words and they both were treated as one lexical unit. the same array of features was extracted to represent the targets and linear and non-linear regressors were trained using both subcorpora. this strategy showed performance gains for both single targets and mwes. 1 in addition, we wanted to investigate how much the classic hand-crafted features like frequency and length together with subword information and contextualized embeddings (not employed previously for lcp) contribute to complexity estimation of both single words and mwes. we present the analysis of feature imortance rankings in section 6. ## related work complex word identification (cwi) is the task of determining how difficult a lexical unit is to a target audience @xcite . the knowledge about the lexical unit complexity can benefit several nlp tasks such as text simplification (ts) or applications related to second-language (l2) acquisition. the goal of ts is to adapt a text to make information, for example, news, more accessible for readers. the ts target group can be language learners @xcite , people with cognitive disabilities @xcite or people with low literacy skills @xcite . one of the strategies of ts is lexical simplification (ls). ls is the task of substituting complex words with simpler ones without changing the original mean-ing. to perform the ls one should first identify the units that might pose a difficulty @xcite . in the area of l2 acquisition, lexical complexity information can be used both for generating study materials appropriate for a learner's level @xcite and for evaluating how proficient a student is @xcite . to understand what makes a word complex for a target audience, one needs to obtain data with complexity annotated. the first manually labelled resource for cwi (cwi 2016) was introduced in semeval-2016 task 11 @xcite . it contained sentences with words marked by non-native speakers of english as either difficult or easy to understand. a word was labelled as complex if at least one annotator marked it as such. thus, words were classified in a binary fashion without addressing the proficiency levels or native languages of the annotators. another cwi dataset (cwi 2018) was presented for bea workshop 2018 @xcite . it contains cwig3g2 datasets @xcite expanded further with the french subcorpus. this is a multilingual dataset with words and mwes marked as complex or simple within a given context. there is no standard form for mwes. the annotators were free to label any sequence of words as a difficult mwe. the complexity judgments were collected from native and non-native speakers. in addition to the binary labels, the words were also assigned an aggregated complexity score. the score was computed as the proportion of annotators that found a word complex. in summary, in both cwi 2016 and cwi 2018 the annotators were not asked to provide a degree of difficulty. the mwes in cwi 2018 were not clearly defined making the nature of their complexity hard to investigate. the complex dataset used in semeval-2021 task 1 was constructed to amend the aforementioned faults of cwi 2016 and cwi 2018. first, it treats complexity as a continuous value. second, it bounds mwes to only pairs of adjective-noun or noun-noun phrases allowing for more targeted research. we believe that models trained using complex have more flexibility in their application. for example, one could set a threshold of complexity to account for different language proficiency levels for both ts and l2 acquisition-related applications. one of the goals of cwi is to establish what makes a word complex. the reports for cwi 2016 @xcite and cwi 2018 @xcite as well as the investigation of cwi 2016 results @xcite show that such features as frequency and length are the most predictive for establishing word complexity. moreover, according to the baselines provided by semeval-2016 task 11 organizers @xcite , the degree of polysemy of a word was also quite successful. in addition to handcrafted features, the teams in both competitions made use of different static word embeddings but they didn't outperform the frequency-based features. ## system overview a linear and a non-linear models were compared. we have trained a linear regression and a multilayer perceptron using the same array of features. the features can be divided into two categories: embeddings and hand-crafted features for both target and context. ## results the results of the systems trained jointly with single and mwe targets are presented in ## discussion as can be seen from the results tables, two trends are obvious: mwe targets always benefit from being trained together with single word targets, and the non-linear model tends to slightly outperform the linear one. this can be contributed both to the nature of mwe and to the smaller size of the mwe subcorpus. although the linear system trained with only single word targets showed better results than the joint one, the non-linear model has also gained from the information about both types of targets when predicting single word complexity. finally, the feature selection procedure was able to improve the performance of the linear model. we were interested to find what features for mwes and single words signal lexical complexity in a similar manner and what features differ in their usefulness. for this purpose, we collected f-scores and mutual information values for the hand-crafted features evaluated for the joint dataset as well as for the target type subcorpora. the ranks of hand-crafted features according to their f-scores and mutual information values can be found in the information collected in all hand-crafted features were present in the top 20 highest scored dimensions with mutual information for the joint data and for the single target data. for the mwe targets, information about morph frequency played a less important role placed at only 110 place. moreover, with the f-score rankings, morph frequency was absent in the top 20 dimensions from all the data configurations. target length in characters has not appeared in the top 20 most correlated features for the joint dataset and for the single targets, but it was still relevant for mwes. word frequency was rated as the highest correlated feature in all setups, it was followed by the bert token number feature. these two features were followed by morph number and by the probability of a masked target. surprisingly, the subword frequency feature was not as successful. the reason for this can be the small amount of data it was estimated on. ## conclusion this paper presents the results of two systems submitted to semeval 2021 task 1: lexical complexity prediction (lcp). we show that training a system jointly with single word targets and mwe targets benefits the predictability of both target types, especially the mwes. we also show that the frequency of subwords feature is more predictive for single targets, while length of a target in characters is more useful for mwe complexity estimation. finally, we show that classic frequency feature is still the most predictive one, even when used together with new contextualized embeddings. for the future work, we would like to explore if the underwhelming results of the subword frequency feature can be amended by collecting statistics from a larger resource. another thing we would like to research is what makes the joint training with single and mwe targets successful. is it the smaller amount of data available for mwes? or is it the nature of noun-noun and adjective-noun mwe expressions? does the second word of the pair contribute more to the mwe complexity and thus compares better to single word targets?