S em E val-2021 Task 1: Lexical Complexity Prediction (2021)

Abstract: This paper presents the results and main findings of SemEval-2021 Task 1 - Lexical Complexity Prediction. We provided participants with an augmented version of the CompLex Corpus (Shardlow et al. 2020). CompLex is an English multi-domain corpus in which words and multi-word expressions (MWEs) were annotated with respect to their complexity using a five point Likert scale. SemEval-2021 Task 1 featured two Sub-tasks: Sub-task 1 focused on single words and Sub-task 2 focused on MWEs. The competition attracted 198 teams in total, of which 54 teams submitted official runs on the test data to Sub-task 1 and 37 to Sub-task 2.

## introduction the occurrence of an unknown word in a sentence can adversely affect its comprehension by readers. either they give up, misinterpret, or plough on without understanding. a committed reader may take the time to look up a word and expand their vocabulary, but even in this case they must leave the text, undermining their concentration. the natural language processing solution is to identify candidate words in a text that may be too difficult for a reader @xcite . each potential word is assigned a judgment by a system to determine if it was deemed 'complex' or not. these scores indicate which words are likely to cause problems for a reader. the words that are identified as problematic can be the subject of numerous types of intervention, such as direct replacement in the setting of lexical simplification @xcite , or extra information being given in the context of explanation generation @xcite . whereas previous solutions to this task have typically considered the complex word identification (cwi) task @xcite in which a binary judgment of a word's complexity is given (i.e., is a word complex or not?), we instead focus on the lexical complexity prediction (lcp) task @xcite in which a value is assigned from a continuous scale to identify a word's complexity (i.e., how complex is this word?). we ask multiple annotators to give a judgment on each instance in our corpus and take the average prediction as our complexity label. the former task (cwi) forces each user to make a subjective judgment about the nature of the word that models their personal vocabulary. many factors may affect the annotator's judgment including their education level, first language, specialism or familiarity with the text at hand. the annotators may also disagree on the level of difficulty at which to label a word as complex. one annotator may label every word they feel is above average difficulty, another may label words that they feel unfamiliar with, but understand from the context, whereas another annotator may only label those words that they find totally incomprehensible, even in context. our introduction of the lcp task seeks to address this annotator confusion by giving annotators a likert scale to provide their judgments. whilst annotators must still give a subjective judgment depending on their own understanding, familiarity and vocabulary -they do so in a way that better captures the meaning behind each judgment they have given. by aggregating these judgments we have developed a dataset that contains continuous labels in the range of 0-1 for each instance. this means that rather than a system predicting whether a word is complex or not (0 or 1), instead a system must now predict where, on our continuous scale, a word falls (0-1). consider the following sentence taken from a biomedical source, where the target word 'observation' has been highlighted: in the binary annotation setting of cwi some annotators may rightly consider this term non-complex, whereas others may rightly consider it to be complex. whilst the meaning of the word is reasonably clear to someone with scientific training, the context in which it is used is unfamiliar for a lay reader and will likely lead to them considering it complex. in our new lcp setting, we are able to ask annotators to mark the word on a scale from very easy to very difficult. each user can give their subjective interpretation on this scale indicating how difficult they found the word. whilst annotators will inevitably disagree (some finding it more or less difficult), this is captured and quantified as part of our annotations, with a word of this type likely to lead to a medium complexity value. lcp is useful as part of the wider task of lexical simplification @xcite , where it can be used to both identify candidate words for simplification @xcite and rank potential words as replacements @xcite . lcp is also relevant to the field of readability assessment, where knowing the proportion of complex words in a text helps to identify the overall complexity of the text @xcite . this paper presents semeval-2021 task 1: lexical complexity prediction. in this task we developed a new dataset for complexity prediction based on the previously published complex dataset. our dataset covers 10,800 instances spanning 3 genres and containing unigrams and bigrams as targets for complexity prediction. we solicited participants in our task and released a trial, training and test split in accordance with the semeval schedule. we accepted submissions in two separate sub-tasks, the first being single words only and the second taking single words and multi-word expressions (modelled by our bigrams). in total 55 teams participated across the two sub-tasks. the rest of this paper is structured as folllows: in section 2 we discuss the previous two iterations of the cwi task. in section 3, we present the complex 2.0 dataset that we have used for our task, including the methodology we used to produce trial, test and training splits. in section 5, we show the results of the participating systems and compare the features that were used by each system. we finally discuss the nature of lcp in section 7 and give concluding remarks in section 8 2 related tasks cwi 2016 at semeval the cwi shared task was organized at semeval 2016 @xcite . the cwi 2016 organizers introduced a new cwi dataset and reported the results of 42 cwi systems developed by 21 teams. words in their dataset were considered complex if they were difficult to understand for non-native english speakers according to a binary labelling protocol. a word was considered complex if at least one of the annotators found it to be difficult. the training dataset consisted of 2,237 instances, each labelled by 20 annotators and the test dataset had 88,221 instances, each labelled by 1 annotator @xcite . the participating systems leveraged lexical features @xcite @xcite and word embeddings @xcite @xcite , as well as finding that frequency features, such as those taken from wikipedia @xcite were useful. systems used binary classifiers such as svms @xcite @xcite , decision trees @xcite @xcite , random forests @xcite @xcite and thresholdbased metrics @xcite to predict the complexity labels. the winning system made use of threshold-based methods and features extracted from simple wikipedia @xcite . a post-competition analysis @xcite with oracle and ensemble methods showed that most systems performed poorly due mostly to the way in which the data was annotated and the the small size of the training dataset. cwi 2018 at bea the second cwi shared task was organized at the bea workshop 2018 @xcite . unlike the first task, this second task had two objectives. the first objective was the binary complex or non-complex classification of target words. the second objective was regression or probabilistic classification in which 13 teams were asked to assign the probability of a target word being considered complex by a set of language learners. a major difference in this second task was that datasets of differing genres: (text genres) as well as english, german and spanish datasets for monolingual speakers and a french dataset for multilingual speakers were provided @xcite . similar to 2016, systems made use of a variety of lexical features including word length @xcite @xcite @xcite , frequency @xcite @xcite , n-gram features @xcite @xcite @xcite and word embeddings @xcite @xcite . a variety of classifiers were used ranging from traditional machine learning classifiers @xcite @xcite , to neural networks @xcite . the winning system made use of adaboost with wordnet features, pos tags, dependency parsing relations and psycholinguistic features @xcite . ## data we previously reported on the annotation of the complex dataset @xcite (hereafter referred to as complex 1.0), in which we annotated around 10,000 instances for lexical complexity using the figure eight platform. the instances spanned three genres: europarl, taken from the proceedings of the european parliament @xcite ; the bible, taken from an electronic distribution of the world english bible translation @xcite and biomedical literature, taken from the craft corpus @xcite . we limited our annotations to focus only on nouns and multi-word expressions following a noun-noun or adjective-noun pattern, using the pos tagger from stanford corenlp @xcite to identify these patterns. whilst these annotations allowed us to report on the dataset and to show some trends, the overall quality of the annotations we received was poor and we ended up discarding a large number of the annotations. for complex 1.0 we retained only instances with four or more annotations and the low number of annotations (average number of @xmath0 for the shared task we chose to boost the number of annotations on the same data as used for complex 1.0 using amazon's mechanical turk platform. we requested a further 10 annotations on each data instance bringing up the average number of annotators per instance. annotators were presented with the same task layout as in the annotation of complex 1.0 and we defined the likert scale points as previously: very easy: words which were very familiar to an annotator. easy: words with which an annotator was aware of the meaning. neutral: a word which was neither difficult nor easy. difficult: words which an annotator was unclear of the meaning, but may have been able to infer the meaning from the sentence. very difficult: words that an annotator had never seen before, or were very unclear. these annotations were aggregated with the retained annotations of complex 1.0 to give our new dataset, complex 2.0, covering 10,800 instances across single and multi-words and across 3 genres. the features that make our corpus distinct from other corpora which focus on the cwi and lcp tasks are described below: continuous annotations: we have annotated our data using a 5-point likert scale. each instance has been annotated multiple times and we have taken the mean average of these annotations as the label for each data instance. to calculate this average we converted the likert scale points to a continuous scale as follows: very @xmath1, @xmath2.25, @xmath3.5, @xmath4.75, very @xmath5.0. ## data splits in order to run the shared task we partitioned our dataset into trial, train and test splits and distributed these according to the semeval schedule. a criticism of previous cwi shared tasks is that the training data did not accurately reflect the distribution of instances in the testing data. we sought to avoid this by stratifying our selection process for a number of factors. the first factor we considered was genre. we ensured that an even number of instances from each genre was present in each split. we also stratified for complexity, ensuring that each split had a similar distribution of complexities. finally we also stratified the splits by token, ensuring that multiple instances containing the same token occurred in only one split. this last criterion ensures that systems do not overfit to the test data by learning the complexities of specific tokens in the training data. performing a robust stratification of a dataset according to multiple features is a non-trivial optimisation problem. we solved this by first grouping all instances in a genre by token and sorting these groups by the complexity of the least complex instance in the group. for each genre, we passed through this sorted list and for each set of 20 groups we put the first group in the trial set, the next two groups in the test set and the remaining 17 groups in the training data. this allowed us to get a rough 5-85-10 split between trial, training and test data. the trial and training data were released in this ordered format, however to prevent systems from guessing the labels based on the data ordering we randomised the order of the instances in the test data prior to release. the splits that we used for the shared task are available via github foot_0 . ## results the full results of our task can be seen in appendix a. we had 55 teams participate in our 2 sub-tasks, with 19 participating in sub-task 1 only, 1 participating in sub-task 2 only and 36 participating in both sub-tasks. we have used pearson's correlation for our final ranking of participants, but we have also included other metrics that are appropriate for evaluating continuous and ranked data and provided secondary rankings of these. sub-task 1 asked participants to assign complexity values to each of the single words instances in our corpus. for sub-task 2, we asked participants to submit results on both single words and mwes. we did not rank participants on mwe-only submis-sions due to the relatively small number of mwes in our corpus (184 in the test set). the metrics we chose for ranking were as follows: pearson's correlation: we chose this metric as our primary method of ranking as it is well known and understood, especially in the context of evaluating systems with continuous outputs. pearson's correlation is robust to changes in scale and measures how the input variables change with each other. spearman's rank: this metric does not consider the values output by a system, or in the test labels, only the order of those labels. it was chosen as a secondary metric as it is more robust to outliers than pearson's correlation. ## discussion in this paper we have posited the new task of lexical complexity prediction. this builds on previous work on complex word identification, specifically by providing annotations which are continuous rather than binary or probabilistic as in previous tasks. additionally, we provided a dataset with annotations in context, covering three diverse genres and incorporating mwes, as well as single tokens. we have moved towards this task, rather than rerunning another cwi task as the outputs of the models are more useful for a diverse range of follow-on tasks. for example, whereas cwi is particularly useful as a preprocessing step for lexical simplification (identifying which words should be transformed), lcp may also be useful for readability assessment or as a rich feature in other downstream nlp tasks. a continuous annotation allows a ranking to be given over words, rather than binary categories, meaning that we can not only tell whether a word is likely to be difficult for a reader, but also how difficult that word is likely to be. if a system requires binary complexity (as in the case of lexical simplification) it is easy to transform our continuous complexity values into a binary value by placing a threshold on the complex-ity scale. the value of the threshold to be selected will likely depend on the target audience, with more competent speakers requiring a higher threshold. when selecting a threshold, the categories we used for annotation should be taken into account, so for example a threshold of 0.5 would indicate all words that were rated as neutral or above. to create our annotated dataset, we employed crowdsourcing with a likert scale and aggregated the categorical judgments on this scale to give a continuous annotation. it should be noted that this is not the same as giving a truly continuous judgment (i.e., asking each annotator to give a value between 0 and 1). we selected this protocol as the likert scale is familiar to annotators and allows them to select according to defined points (we provided the definitions given earlier at annotation time). the annotation points that we gave were not intended to give an even distribution of annotations and it was our expectation that most words would be familiar to some degree, falling in the very easy or easy categories. we pre-selected for harder words to ensure that there were also words in the difficult and very difficult categories. as such, the corpus we have presented is not designed to be representative of the distribution of words across the english language. to create such a corpus, one would need to annotate all words according to our scale with no filtering. the general distribution of annotations in our corpus is towards the easier end of the likert scale. a criticism of the approach we have employed is that it allows for subjectivity in the annotation process. certainly one annotator's perception of complexity will be different to another's. giving fixed values of complexity for every word will not reflect the specific difficulties that one reader, or one reader group will face. the annotations we have provided are averaged values of the annotations given by our annotators, we chose to keep all instances, rather than filtering out those where annotators gave a wide spread of complexity annotations. further work may be undertaken to give interesting insights into the nature of subjectivity in annotations. for example, some words may be rated as easy or difficult by all annotators, whereas others may receive both easy and difficult annotations, indicating that the perceived complexity of the instance is more subjective. we did not make the individual annotations available as part of the shared task data, to encourage systems to focus primarily on the prediction of complexity. an issue with the previous shared tasks is that scores were typically low and that systems tended to struggle to beat reasonable baselines, such as those based on lexical frequency. we were pleased to see that systems participating in our task returned scores that indicated that they had learnt to model the problem well (pearson's correlation of 0.7886 on task 1 and 0.8612 on task 2). mwes are typically more complex than single words and it may be the case that these exhibited a lower variance, and were thus easier to predict for the systems. the strong pearson's correlation is backed up by a high r2 score (0.6172 for task 1 and 0.7389 for task 2), which indicates that the variance in the data is captured accurately by the models' predictions. these models strongly outperformed a reasonable baseline based on word frequency as shown in whilst we have chosen in this report to rank systems based on their score on pearson's correlation, giving a final ranking over all systems, it should be noted that there is very little variation in score between the top systems and all other systems. for task 1 there are 0.0182 points of pearson's correlation separating the systems at ranks 1 and 10. for task 2 a similar difference of 0.021 points of pearson's correlation separates the systems at ranks 1 and 10. these are small differences and it may be the case that had we selected a different random split in our dataset this would have led to a different ordering in our results @xcite . this is not unique to our task and is something for the semeval community to ruminate on as the focus of nlp tasks continues to move towards better evaluation rather than better systems. an analysis of the systems that participated in our task showed that there was little variation between deep learning approaches and feature based approaches, although deep learning approaches ultimately attained the highest scores on our data. generally the deep learning and feature based approaches are interleaved in our results table, showing that both approaches are still relevant for lcp. one factor that did appear to affect system output was the inclusion of context, whether that was in a deep learning setting or a feature based setting. systems which reported using no context appeared to perform worse in the overall rankings. another feature that may have helped performance is the inclusion of previous cwi datasets @xcite . we were aware of these when developing the corpus and attempted to make our data sufficiently distinct in style to prevent direct reuse of these resources. a limitation of our task is that it focuses solely on lcp for the english language. previous cwi shared tasks @xcite and simplification efforts @xcite have focused on languages other than english and we hope to extend this task in the future to other languages. ## conclusion we have presented the semeval-2021 task 1 on lexical complexity prediction. we developed a new dataset focusing on continuous annotations in context across three genres. we solicited participants via semeval and 55 teams submitted results across our two sub-tasks. we have shown the results of these systems and discussed the factors that helped systems to perform well. we have analysed all the systems that participated and categorised their findings to help future researchers understand which approaches are suitable for lcp.