ANDI at S em E val-2021 Task 1: Predicting complexity in context using distributional models, behavioural norms, and lexical resources (2021)

Abstract: In this paper we describe our participation in the Lexical Complexity Prediction (LCP) shared task of SemEval 2021, which involved predicting subjective ratings of complexity for English single words and multi-word expressions, presented in context. Our approach relies on a combination of distributional models, both context-dependent and context-independent, together with behavioural norms and lexical resources.

## introduction in our day-to-day life, outside the laboratory, we almost never come across single words or pairs of words, in isolation. instead, such verbal stimuli are typically embedded within sentences or phrases, and our understanding of individual words and word pairs is influenced by their linguistic contexts (e.g., by disambiguating their intended meaning). hoewever, almost all behavioural norms collected so far focus only on single words or word pairs @xcite . therefore, the lexical complexity prediction (lcp) shared task @xcite , hosted at semeval 2021, constitutes a timely and valuable contribution to the study of context-dependent semantics. the task requires competitors to predict subjective ratings of complexity for words or pairs of words, presented within sentences. as mentioned by the organisers, being able to automatically estimate contextualised complexity ratings would have several practical applications, such as detecting and simplifying portions of text that might be particularly difficult 1 https://github.com/armandrotaru/teamandi-lcp to understand for second language learners, and people with low literacy levels (e.g., as a result of suffering from a reading impairment). the dataset for the competition is complex 2.0 @xcite , consisting of passages from the bible, the proceedings of the european parliament, and biomedical journal articles. the training data covers 7,662 single words (2,574 bible, 2,512 europarl, and 2,576 biomed), and 1,517 multi-word expressions (505 bible, 498 europarl, and 514 biomed). the test data covers 917 single words (283 bible, 345 europarl, and 289 biomed), and 184 multi-word expressions (66 bible, 65 europarl, and 53 biomed). in this paper we describe our submission to the competition, based on distributional models, both context-dependent and context-independent, as well as behavioural norms/lexical resources 1 . the best results are obtained by combining the three classes of predictors. however, the improvement in performance over using just context-independent models is small, and, in practice, might be compensated by their impressive vocabulary size and ease of use. ## general description in order to predict word complexity in context, we combined information from three type of sources, namely behavioural norms/lexical resources, and distributional models. with respect to the latter, we included two distinct classes of models: ## system description we tested three groups of predictors, both in isolation and combined. the first group was obtained from comprehensive datasets of subjective ratings (concreteness, age of acquisition, etc.), task performance measures (i.e., response times and accuracies in the lexical decision tasks), as well as frequency, contextual diversity, and prevalence counts, plus cefr word lists (see the references from the beginning of the previous section). in order to extend the coverage of the subjective ratings, we did not use the original data, but instead relied on extrapolated ratings for more than 70,000 words. the extrapolation was based on the skip-gram, glove, and conceptnet numberbatch models, using linear regression over the concatenated vector dimensions. for the (already extrapolated) ratings from @xcite , as well as for the frequency, contextual diversity, and prevalence counts, we employed only the normed values, as they already have very good coverage. we also used only the original lexical decision data, given that response times and accuracies are difficult to extrapolate, and did not try to extend the cefr word lists, due to methodological difficulties. for the single word datasets, we employed all the previously mentioned factors, whereas for the multi-word expression datasets, we only employed our own extrapolated factors. the second group was generated from skipgram, glove, and conceptnet numberbatch embeddings. the vocabulary of the models was that described in the discussion above. for the first two sources of information, and for each selected variable v (e.g., semantic diversity), we generated either four predictors, in the case of the single word datasets, or nine predictors, in the case of the multi-word expression datasets. the single word predictors consisted of v(w), v(c), v(w) * v(c), and abs(v(w) -v(c)), while the multiword expression predictors consisted of v(w1 w2 ), where: the third group was derived from the bert, roberta, electra, albert, and deberta models. we used the standard (base) versions of each model (i.e., without task-specific fine-tuning), as described in the original papers, with the exception of electra, where we employed the small, base, and large versions of the model. the implementations of the models were all obtained from the hugging face repository @xcite . the predictors consisted only of the activations for the single word w, or the multi-word expression w1 w2, averaged over the last four hidden layers. to predict ratings of complexity in context, we employed ridge regression (@xmath0, for the single word dataset, and a combination of ridge regression (@xmath1, for the multi-word expression dataset, after zero centering all the aforementioned predictors. ## results and discussion the results for english are shown figure 1 , for various sets of predictors and regularization strengths. for reasons of space, we only present the results for ridge regression, but note that similar patterns of performance are obtained for gradientboosted decision trees and other types of models, such as shallow neural networks. results are averaged over 10 rounds of 10-fold crossvalidation, using only the training dataset. the results indicate that context-independent models (fig. 1b ) outperform behavioural norms (fig. 1a ), and context-dependent models (fig. 1cf ). a likely reason for the superiority of contextindependent models over context-dependent models is the fact that the former were trained on huge corpora (i.e., 100-840 billion tokens), while the latter were trained on considerably smaller corpora (i.e., 3-33 billion tokens). however, in spite of this significant training disadvantage, context-dependent models produce competitive levels of performance, a finding which can likely be attributed to several factors, such as the highly non-linear integration of contextual information, the use of self-attention mechanisms, and that of more sophisticated learning objectives. combining the three classes of predictors produces a relatively small improvement in predictive performance, as compared to relying on any single class. this reflects a very high degree of redundancy between the complexity-related information present in the three types of predictors. interestingly, even for the largest set of predictors, consisting of 13,400 variables per 1,517 data points, the degree of regularization does not appear to matter much, indicating little overfitting. finally, there is a small, but systematic difference in performance between single words and multi-word expressions, in favour of the latter, even though the training set for single word stimuli is roughly five times larger than that for multi-word stimuli. a potential explanation for this finding might be that the individual variability in meaning for multi-word expressions is smaller than that for single words, given that expressions should be more informative than single words, in virtue of their length (i.e., two words vs one word). within the competition, our models ranked 4 th (@xmath2.78, ρ = .73, @xmath3.064), in the single word sub-task, and 6 th (@xmath4.85, ρ = .84, @xmath5.067), in the multi-word expression sub-task. ## conclusions our results suggest that several approaches can be quite successfully employed in order to predict ratings of complexity in context, for both single words and multi-word expressions. in terms of performance, the best predictors are those derived from context-independent models (e.g., skipgram), but relatively good results can be obtained also by using context-dependent models (e.g., bert) and behavioural norms (e.g., subjective ratings of familiarity). moreover, given that their vocabulary covers a remarkable number of words (i.e., more than 500 thousand, for each of the skipgram, glove, and conceptnet numberbatch models), and that they are very easy to use off-theshelf, context-independent models represent a particularly promising approach to predicting ratings of complexity in context.