LAST at S em E val-2021 Task 1: Improving Multi-Word Complexity Prediction Using Bigram Association Measures (2021)

Abstract: This paper describes the system developed by the Laboratoire d’analyse statistique des textes (LAST) for the Lexical Complexity Prediction shared task at SemEval-2021. The proposed system is made up of a LightGBM model fed with features obtained from many word frequency lists, published lexical norms and psychometric data. For tackling the specificity of the multi-word task, it uses bigram association measures. Despite that the only contextual feature used was sentence length, the system achieved an honorable performance in the multi-word task, but poorer in the single word task. The bigram association measures were found useful, but to a limited extent.

## introduction for more than half a century, many studies have been carried out to collect norms about formal and semantic properties of words, such as frequency of use, spelling regularity, familiarity, age of acquisition, or emotional valence @xcite . some of these properties can be easily harvested through automatic counting procedures applied to corpora. other properties, such as familiarity or emotional valence, are obtained by requiring participants, often more than ten, to rate the words on these dimensions. in psycholinguistics, these norms have been mainly used for selecting experimental materials @xcite . in computational linguistics, they are used in opinion mining, in the evaluation of foreign language skills and in text simplification for instance @xcite . obtaining lexical norms that require human evaluations is extremely costly in time and resources, which greatly reduces their size. however, huge norms are essential in applications @xcite . this observation has led to the development of automatic techniques to extend such norms @xcite @xcite . the lexical complexity prediction (lcp) shared task at semeval-2021 requires exactly the development of such techniques @xcite . it is indeed a question of estimating the lexical complexity, the degree of difficulty of the words in a text. this dimension is important in nlp applications for simplifying texts and assisting specific populations such as people with reading disabilities or who are learning a foreign language. a specificity of the lcp task is that it relates not only to words but also to multi-word expressions which are very rarely taken into account in norms and in automatic extension techniques @xcite . another important feature of the task is that the target tokens were presented to human judges in context and that a significant number of them were presented in several different contexts. human annotations are therefore likely to reflect the impact of the linguistic context on lexical complexity. this paper describes the system proposed for this task by the laboratoire d'analyse statistique des textes (last). it is based on a lightgbm model fed with features obtained from many word frequency lists, published lexical norms and psychometric data. for tackling the specificity of the multi-word task, it uses bigram association measures (such as mutual information) from research in lexicography @xcite and in the automatic evaluation of texts written by english learners @xcite . despite that the only contextual feature used was sentence length, the system achieved an honorable performance in the multiword task, ranking 9th out of 37 teams, but poorer in the single word task, ranking 26th out of 54 teams. in the next section, the main characteristics of this challenge are summarized. the following sec-tion describes in detail the developed system. finally, the results in the challenge are reported along with several analyzes performed to get a better idea of the factors that affect the system performance. ## task and materials the organizers of the challenge have made available an updated version of the complex dataset @xcite to the participating teams for developing their systems (i.e., the learning set). it consists of 8,083 single words and 1,616 bigrams, all of them presented in a one-sentence context. these sentences were taken from three english sources in almost equal proportion: biblical text, biomedical articles and proceedings of the european parliament. the target words and bigrams were evaluated by several judges on a 5-point likert scale depending on whether it seemed more or less easy to understand in this context. there were on average 25.75 annotations per instance @xcite . the complexity score for each target is the mean of these ratings. in this materials, a non-negligible proportion of the targets were presented several times in different sentences in order to assess the impact of this context on the complexity assessment. the test set, collected in the same way, consisted of 917 single words and 184 bigrams of which none of the targets were present in the learning set. the challenge measure was pearson's linear correlation coefficient between human ratings and system predictions. ## system the first part of this section presents the features used to predict lexical complexity starting with those common to both tasks and ending with those specific to predicting the complexity of the multiword expressions. next, the procedure used to build the predictive models is described. ## conclusion the models proposed for the lcp task were built by the lightgbm software mainly fed with norms and frequency features. it obtained an acceptable performance on the test set in the multi-word task on the basis of little contextual information, but less so in the single word task. the analyzes carried out by a cv approach showed, on the other hand, that the system is no better in the multi-word task. it is therefore possible or even probable that the better performance results from an overestimation of its effectiveness. the bigram association measures (aka collgrams) have proven to be useful, but to a limited extent. taking the context into account would probably have improved the system, especially for the single word task in which more than half of the targets were repeated. this hypothesis, however, is based on the assumption that differences between human ratings for the same target in different contexts are as reliable as their ratings for different targets. more generally, it would be interesting to explain the origin of the very important difference in performance between the two tasks, but that does not seem possible on the basis of the data i have access to.