TUDA - CCL at S em E val-2021 Task 1: Using Gradient-boosted Regression Tree Ensembles Trained on a Heterogeneous Feature Set for Predicting Lexical Complexity (2021)

Abstract: In this paper, we present our systems submitted to SemEval-2021 Task 1 on lexical complexity prediction. The aim of this shared task was to create systems able to predict the lexical complexity of word tokens and bigram multiword expressions within a given sentence context, a continuous value indicating the difficulty in understanding a respective utterance. Our approach relies on gradient boosted regression tree ensembles fitted using a heterogeneous feature set combining linguistic features, static and contextualized word embeddings, psycholinguistic norm lexica, WordNet, word- and character bigram frequencies and inclusion in wordlists to create a model able to assign a word or multiword expression a context-dependent complexity score. We can show that especially contextualised string embeddings can help with predicting lexical complexity.

## introduction in this paper, we present our contribution to semeval-2021 shared task 1 @xcite , a shared task focused on the topic of lexical complexity prediction. the term lexical complexity prediction describes the task of assigning a word or multiword expression a continuous or discrete score signifying its likeliness of being understood well within a given context, especially by a non-native speaker. solving this task could benefit second-language learners and non-native speakers in various ways. one could imagine using such scores to extract vocabulary lists appropriate for a learner level from corpora and literature @xcite , to judge if a given piece of literature fits a learner's skill or to assist authors of textbooks in finding a level of textual difficulty appropriate for a target audience. predicting these scores can be formulated as a regression problem. our approach to solve this problem relies on gradient-boosted regression tree ensembles which we fit on a heterogeneous feature set including different word embedding models, linguistic features, wordnet features, psycholinguistic lexica, corpus-based word frequencies and word lists. we assumed that lexical complexity could be correlated with a wide range of features, neural ones as much as distributional or psycholinguistic ones, which is why we chose to use an ensemble-based method in the form of gradient boosting @xcite for our system as it usually performs best for tasks where such a feature set is needed compared to solely neural models which need dense, homogeneous input data to perform well. out of all participants, our systems were ranked 15/54 in the single word-and 19/37 in the multiword category during the official shared task evaluations according to pearson's correlation coefficient. our key discovery is that while features from nearly all categories provided by us were used by our systems, contextual string embeddings @xcite were the by far most important category of features to determine lexical complexity for both systems. the code and our full results can be found at https://github.com/sgombert/ tudacclsemeval . ## system overview our systems rely on gradient-boosted regression tree ensembles @xcite for predicting lexical complexity scores. we trained one model to predict single word lexical complexity scores and another one to predict bigram multiword expression complexity scores. our models are based on the implementation of gradient boosting provided by catboost foot_0 @xcite . we set the growing policy to lossguide, the l2 leaf regularisation to 15, the learning rate to 0.01, tree depth to 6 and the maximum number of leaves to 15. additionally, we set the number of maximum iterations to 5000 and then used the trial set to perform early stopping during training in order to determine the exact number of required iterations. the motivation behind using this algorithm was its general ability to perform well on heterogeneous and sparse feature sets which allowed us to mix regular linguistic features, wordnet features, word embeddings, psycho-linguistic norm lexica, corpusbased word frequencies and selected word lists as all of these were features we assumed to possibly correlate with lexical complexity. moreover, the reportings of @xcite and yimam et al. (2018) that ensemble-based learners perform best for complex word identification contributed to this decision, as well. while the problem presented in their paper is formulated as a binary classification task using different data sets, we wanted to test if their findings would still translate to a regression task on complex. ## results throughout the shared task, the systems were evaluated with regard to pearson's correlation coefficient, spearman's rank correlation coefficient, mean average error, mean squared error and r2 with pearson's correlation coefficient determining the main ranking. according to this, our systems achieved the 15th and 19th rank respectively. ## conclusion we presented both our systems submitted to semeval-2021 task 1 combining a heterogeneous feature set with gradient boosting as regression algorithm. our systems ware ranked 15/54 and 19/37 during shared task evaluations according to pearson's correlation coefficient. however, the results achieved by our systems were still close to the best results, especially in the case of the system dealing with single word complexity. the type of feature playing the most important role for our models are contextual string embeddings as they influenced the outcome the most. we attribute this to a relationship between lexical complexity and the distribution of characters throughout words and sentences, but this needs further clarification which could be the objective of future work. moreover, our systems rarely assign scores below 0.2. it must be explored further if there are features which could improve our systems in this respect. in summary, we can report that ensemble methods turned out to be fruitful when applied to complex.