C ambridge at S em E val-2021 Task 1: An Ensemble of Feature-Based and Neural Models for Lexical Complexity Prediction (2021)

Abstract: This paper describes our submission to the SemEval-2021 shared task on Lexical Complexity Prediction. We approached it as a regression problem and present an ensemble combining four systems, one feature-based and three neural with fine-tuning, frequency pre-training and multi-task learning, achieving Pearson scores of 0.8264 and 0.7556 on the trial and test sets respectively (sub-task 1). We further present our analysis of the results and discuss our findings.

## introduction predicting which words are considered hard to understand for a given target population has many applications. for example, it can be used to identify texts appropriate for language learners or included in a pipeline for automatic text simplification for people with low literacy skills or reading disabilities @xcite @xcite . in this paper, we describe our submission to the semeval-2021 shared task on lexical complexity prediction (lcp) (sub-task 1), where participating teams are expected to predict the complexity score of single words in context @xcite . compared to previous shared tasks on complex word identification (cwi), which have primarily focused on binary classification as systems were expected to identify words as complex or not @xcite ; a new multi-domain english dataset was used for the purpose, which was annotated using a 5-point likert scale @xcite . we approached lcp as a regression problem and proposed a traditional featurebased model, as well as three neural models exploring fine-tuning, frequency pre-training and multitask learning (mtl). the remainder of this paper is organised as follows. section 2 presents related work in the area. in section 3, we describe our approach to the task and detail the four models included in our final ensemble system. in section 4, we turn to the experiments, describing the data and evaluation metrics used, and presenting our results on the shared task trial set. section 5 presents our official results on the shared task test set, and offers a discussion of the results and the performance of our submitted system. finally, we conclude the paper and provide an overview of our findings in section 6. ## related work the semeval-2016 shared task on cwi @xcite was framed as a binary classification problem, where complexity was defined as whether or not a word is difficult to understand for non-native english speakers. a set of 400 nonnative speakers annotated the data in a binary fashion and a word was labelled as complex if it was annotated as complex by at least one annotator. the study performed by @xcite showed that most systems performed poorly due to the way the data was annotated. they also found out that words that were annotated as complex by the majority of human annotators tend to be easier for systems to identify, arguing that lexical complexity should be seen as a continuum on a spectrum rather than a binary value. the second cwi shared task was organized as part of the bea-2018 workshop @xcite . it extended the previous one by introducing a new probabilistic classification sub-task where participants were asked to assign the probability that an annotator would find a word complex. the continuous complexity value for each word was calculated as the proportion of annotators that found a word complex. the results of the shared task showed that traditional feature engineering approaches (mostly based on length and ## official results and discussion our submission to the lcp shared task (sub-task 1) is the result of our best system (in terms of pearson), an ensemble of three neural and one featurebased systems mtl all + bert + bert freq. + randomforest. the official results are reported in ## conclusion this paper presents our contribution to the semeval-2021 shared task on lcp. we competed in sub-task 1 (single words) with an ensemble system combining three neural models and one featurebased model. our analysis reveals that even though all three neural systems perform comparably, the mtl system contributed the most to the ensemble system. adding the feature-based model improved the performance on the trial data, but brought performance down on the test data. in addition to the mismatch between the trial and test data, the noise in the data further contributed to this inconsistency. the comparatively lower performance of the feature-based system is especially interesting in light that such systems were competitive for cwi until relatively recently @xcite . when looking at different genres, our submitted system yielded the highest performance in pearson, but worst performance in mae in biomedical domain, compared to the other genres. we hypothesise that this is due to differences in data distribution between genres. all bible biomed europarl 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 lexical complexity prediction (a) train all bible biomed europarl 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 lexical complexity prediction (b) trial all bible biomed europarl 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 lexical complexity prediction (c) test figure a.1: box plots of complexity scores for data by genre.