UNBNLP at S em E val-2021 Task 1: Predicting lexical complexity with masked language models and character-level encoders (2021)

Abstract: In this paper, we present three supervised systems for English lexical complexity prediction of single and multiword expressions for SemEval-2021 Task 1. We explore the use of statistical baseline features, masked language models, and character-level encoders to predict the complexity of a target token in context. Our best system combines information from these three sources. The results indicate that information from masked language models and character-level encoders can be combined to improve lexical complexity prediction.

## introduction semeval 2021 task 1 @xcite focuses on predicting the complexity of a target token in an english sentence on a scale from 0 (not very complex) to 1 (very complex). for example, the token land in the example below is judged to have a low complexity of 0.19. 1. our land will yield its increase. on the other hand, the token doxycycline in the following example is judged to have a relatively higher complexity of 0.75. foot_0 2. the reason these two lines were unresponsive to doxycycline is unknown. the dataset for this task was originally proposed in @xcite , and includes sentences from three sources: a translated bible, european parliament proceedings, and a biomedical corpus. this shared task contains two sub-tasks with the first focusing on lexical complexity for single words, which will be referred to as single, and the second focusing on complexity of multiword expressions, which will be referred to as multi. in this paper, we explore the use of statistical baseline features, masked language models, and character-level encoders to predict the complexity of a target token in context. we first consider these approaches individually and then consider supervised methods for combining them. we evaluate our models with pearson correlation (r), spearman correlation (rho), mean absolute error (mae), mean square error (mse), and r-squared (r2). we apply all our models to both sub-tasks and find that we achieve our best results with a model that combines all three sources of information -baseline features, masked language modeling, and characterlevel encoding. specifically, we achieve our best results with respect to pearson correlation -the evaluation measure used for the official shared task system ranking -using a model that combines complexity predictions from two approaches, one based on a character-level encoder, and the other based on a masked language model, which further incorporates the baseline features, using support vector regression. although we achieve our best results using this approach on both single and multi with respect to pearson correlation, we note substantial variation in performance with respect to the other evaluation metrics on single. this suggests that future work could further explore the variation in performance of this model on single and multiword expressions ## model components in this section, we describe three approaches to lexical complexity prediction. in section 3 we then describe how these approaches are combined into systems that we submitted as official runs to the shared task. ## submitted systems in this section, we describe how we combine the approaches discussed in section 2 to form systems that were submitted as official runs to the shared task. we selected these systems because they achieved the best performance in a ten-fold cross-validation experiment on the combined trial and training data, in terms of pearson correlation (r), the evaluation metric used to rank systems in the shared task. the performance of each submitted system, a baseline in which we train logistic regression on the baseline features from section 2.1, and the masked language model approach described in section 2.3 (our best-performing model that was not submitted to the shared task), are shown in tables 1 and 2 for single and multi, respectively. ## results in this section we present our results with respect to the five evaluation metrics. we evaluate our models on the single word sub-task (single) first and then on the multiword expressions sub-task (multi). each system is trained on all training instances from both single and multi. 2 we include the performance of our baseline to show that all submitted systems continue to outperform this baseline model on the test data. in in ## conclusions we evaluated three systems for lexical complexity prediction of single and multiword expressions for semeval 2021 task 1. these systems incorporated information from statistical baseline features, a character-level encoder approach, and a masked language model approach. we found that a system that combined the complexity predictions of the character-level encoder approach and the masked language model approach, which further incorporates the statistical baseline features, using support vector regression performed best amongst our submitted systems with respect to pearson correlation on both the single word and multiword expressions sub-tasks. this approach further performed best of our submitted systems with respect to all evaluation metrics on the multiword expression sub-task, although this was not the case for the single word sub-task. in future work, the relationship between the subtasks, models, and evaluation metrics warrants further exploration, including studying the effect that the type of the target expression, i.e., single word or multiword expression -has on the performance of the models with respect to the various evaluation metrics.