D eep B lue AI at S em E val-2021 Task 1: Lexical Complexity Prediction with A Deep Ensemble Approach (2021)

Abstract: Lexical complexity plays an important role in reading comprehension. lexical complexity prediction (LCP) can not only be used as a part of Lexical Simplification systems, but also as a stand-alone application to help people better reading. This paper presents the winning system we submitted to the LCP Shared Task of SemEval 2021 that capable of dealing with both two subtasks. We first perform fine-tuning on numbers of pre-trained language models (PLMs) with various hyperparameters and different training strategies such as pseudo-labelling and data augmentation. Then an effective stacking mechanism is applied on top of the fine-tuned PLMs to obtain the final prediction. Experimental results on the Complex dataset show the validity of our method and we rank first and second for subtask 2 and 1.

## introduction lexical complexity is one of the main reasons leading to overall text complexity and thus result in poor reading comprehension for readers @xcite . different from the complex word identification (cwi) @xcite task, which aims to predict whether a given word is complex or not, the goal of lexical complexity prediction (lcp) is to predict the complexity value of the given parts from contexts as shown in figure 1 . the underlined parts of the sentence are the words that need to be predicted and the same words in different contexts may have different complexity scores. lcp plays an important role in the usual lexical simplification (ls) @xcite pipeline since it can help simplifiers find the challenging words and replace them with appropriate alternatives that easy to understand. either lcp or cwi can not only be used as a component of ls systems but also as a stand-alone application within intelligent ## background task definition there are two subtasks in the lcp task. for subtask 1, the goal is to predict the complexity score for a single word from the given context. as an example shown in figure 1 , the 'refuge' is the word that needs to be predicted and since the meaning of it is harder to get in the first context, its complexity score in the first context is much higher. for subtask 2, the goal is to predict the complexity score for a multi-word expression from the given context. an example is also shown in the right part of figure 1 . a 5-point likert scale: one for very easy, two for easy, three for neutral, four for difficult, and five for very difficult. the numerical labels were transformed to a 0-1 range as shown in figure 1 . to add further variation to the data, three corpora were selected including bible, europarl @xcite and biomedical @xcite . each corpus has its own unique language features and styles. in addition to single words, multi-word expressions were also selected for annotating. in the end, there were 9476 annotated contexts with 5166 unique words. ## conclusion in this paper, we propose a top-performing model for the task of lexical complexity prediction. we fine-tune several pre-trained language models including bert, albert, roberta, and ernie with different training strategies such as pseudolabelling and data augmentation and stack them with a simple linear regression model. experimental results show the effectiveness of this ensemble method and we win first place and second place for subtask 2 and 1.