IAPUCP at S em E val-2021 Task 1: Stacking Fine-Tuned Transformers is Almost All You Need for Lexical Complexity Prediction (2021)

Abstract: This paper describes our submission to SemEval-2021 Task 1: predicting the complexity score for single words. Our model leverages standard morphosyntactic and frequency-based features that proved helpful for Complex Word Identification (a related task), and combines them with predictions made by Transformer-based pre-trained models that were fine-tuned on the Shared Task data. Our submission system stacks all previous models with a LightGBM at the top. One novelty of our approach is the use of multi-task learning for fine-tuning a pre-trained model for both Lexical Complexity Prediction and Word Sense Disambiguation. Our analysis shows that all independent models achieve a good performance in the task, but that stacking them obtains a Pearson correlation of 0.7704, merely 0.018 points behind the winning submission.

## introduction complex word identification (cwi) consists of determining which words or multi-word expressions @xcite in a text could be difficult to understand by certain readers. this is one of the first steps in the typical lexical simplification pipeline @xcite . cwi has traditionally been treated as either a binary @xcite or regression @xcite task. for the latter, the complexity of a word/mwe was computed as a percentage of binary complexity ratings. recently, @xcite proposed to move away from the binary definition of cwi, and instead collected complexity ratings using likert scales. this allows re-defining the task as lexical complexity prediction (lcp). leveraging this new collected data, the first lcp shared task was organised in semeval-2021 @xcite . our team participated in sub-task 1: predicting the complexity score of single words. basically, given a sentence and a target word in it, the goal is to predict the complexity score of the target. one particular challenge is that the same target can have different complexity scores depending on the sentence it appears in. therefore, our proposed approach takes the context of the target into consideration in two ways. first, we use contextualised word representations from pre-trained transformed-based models, such as roberta @xcite and xlnet @xcite . in particular, we use the lcp data to fine-tune two roberta models and one xlnnet model that receive as input the target and a context window of 1, and a roberta model whose inputs are the target and a context window of 2. second, we hypothesise that different contexts could evoke different senses of the target word. as such, we exploit data for word sense disambiguation (wsd) through multi-task learning. in particular, we finetune a bert @xcite model with two tasks: lcp and wsd, using the unified evaluation framework @xcite for the latter. the predictions from all these models are combined with several morphosyntactic and corpus-based features, and used to train a gradient boosting decision tree with lightgbm @xcite . on the test set of the shared task, our model achieved a pearson correlation of 0.7704 and ranked 10th, only 0.018 points behind the winner. an ablation study shows that all independent models contributed to the stacked model's performance, with the predictions from the bert model finetuned in a multi-task fashion having the greatest impact in predicting lexical complexity. the code to reproduce our results is available in: https: //github.com/kdrivas/lexical_complexity. ## background the lcp shared task on semeval-2021 asks participants to develop models that predict the complexity of a target word/mwe in a sentence in english @xcite . this shared task builds on previous editions that focused on complex word identification @xcite , with a key difference: complexity ratings are continuous scores instead of binary. furthermore, the same target word/mwe can appear in more than one sentence but with different complexity scores. the data for the shared task is an extension of complex @xcite , a dataset with complexity ratings for target words/mwe in sentences in english in three domains: bible, europarl and biomed. the dataset is split into two substasks: lcp for single words and lcp for mwes. ## system description this section details our stacking approach to the lcp shared task sub-task 1. an overview of our system can be seen in figure 1 . ## experimental setup as previously described, we used four different models: roberta, xlnet, bert and lightgbm. in addition, for training/fine-tuning each model we chose the mean absolute error (mae) as our validation metric. ## results the test set contains more than 1,000 sentences with 573 different target words. the scores in the validation set ( ## ablation study another way of visualising the importance of each feature is using shap values (lundberg and lee, 2017). figure 3 reports the 10 most important features for the lightgbm model, i.e. the impact of each feature in predicting the target complexity score. the x-axis shows the increase or decrease of target complexity, while the red and blue colours refer to the feature value's size. for example, in the case of feature size of sentence, if the number of characters is larger there will be a positive impact, i.e. the complexity will increase. on the other hand, if the sentence length is smaller, there will be a negative impact, i.e. the complexity will decrease. we can observe that the most important feature is the predictions given by the bert multitask model since they have the greatest impact. this signals that wsd data could benefit predicting lexical complexity. it is also noted that the predictions of the transformers-based models are in the top 5 of importance. other features, such as the size of the sentence or the number of word senses, also have good contributions to the impact. ## conclusion in this paper, we presented our system for the single word complexity prediction sub-task in the lcp shared task. our approach consisted of combining lexical features and predictions from fine-tuned pre-trained transformer-based models. we found that each set of features achieved a good performance on their own, and that combining all of them achieved our best result. in particular, we found that fine-tuning a pre-trained transformer- considering that there were unseen tokens in validation and test sets, the task resembles a zero shot classification problem. therefore, as future work, semi-supervised learning approaches or data augmentation algorithms could be explored, and training in a multitask fashion another transformerbased models like roberta.