CS - UM 6 P at S em E val-2021 Task 1: A Deep Learning Model-based Pre-trained Transformer Encoder for Lexical Complexity (2021)

Abstract: Lexical Complexity Prediction (LCP) involves assigning a difficulty score to a particular word or expression, in a text intended for a target audience. In this paper, we introduce a new deep learning-based system for this challenging task. The proposed system consists of a deep learning model, based on pre-trained transformer encoder, for word and Multi-Word Expression (MWE) complexity prediction. First, on top of the encoder’s contextualized word embedding, our model employs an attention layer on the input context and the complex word or MWE. Then, the attention output is concatenated with the pooled output of the encoder and passed to a regression module. We investigate both single-task and joint training on both Sub-Tasks data using multiple pre-trained transformer-based encoders. The obtained results are very promising and show the effectiveness of fine-tuning pre-trained transformers for LCP task.

## introduction text simplification (ts) is a fundamental task for improving text readability, and presents a wide variety of use cases, including assisting children with reading difficulties and native speakers with low literacy levels @xcite , increasing accessibility for people with intellectual disabilities @xcite , and facilitating certain aspects of language for language learners @xcite . ts may involve modifications to the syntactic structure of a sentence, its lexical units or both @xcite . lexical simplification (ls), as a sub-task of ts, focuses on simplifying complex words of an input sentence. it first identifies complex words in a sentence, known as complex words identification (cwi) or lexical complexity prediction (lcp) task. then, it replaces them with other alternatives of equivalent meaning. those substitutions should be more simplistic while preserving the semantic and the grammatical structure of the input sentence @xcite . most of the previous research has modeled lcp as a binary classification task @xcite @xcite . a recent research study has introduced a multidomain dataset, annotated using a 5-point likert scale scheme @xcite . the aim is to label the complexity of a word or a multi-word expression (mwe), in a more fine-grained manner, from very easy to very difficult. hence, the lexical complexity of words is expressed on a continuous scale. in this paper, we introduce our submitted system to the semeval-2021 lcp 1 and 2 sub-tasks @xcite . the proposed system consists of a deep learning model for word and mwe complexity prediction. our model employs a residual attention block and a regression module on top of a pre-trained transformer encoder, as follows: the rest of this paper is organized as follows. section 2 describes the dataset and the sub-tasks of semeval-2021 task 1. in section 3, we present our system overview. section 4 summarizes and discusses the obtained results for both sub-task 1 and sub-task 2. finally, section 5 concludes the paper. ## system overview the proposed system uses a residual attention block and a regression module on top of the pre-trained transformer encoder network. in the following, we describe each component of our system. ## results this section describes the experiment settings and the obtained results. ## conclusion in this paper, we have presented our submitted system to the semeval-2021 task 1. the pro-posed system consists of a deep learning model for word and mwe complexity prediction. our model employs a residual attention block and a regression module on top of a pre-trained transformer encoder. we have trained the model to minimize the uncertainty weighted loss of the rmse and the auxiliary loss associated to the negative pearson correlation. experiments are performed using the base and the large variants of the pre-trained bert and roberta encoders. the best performance is obtained using roberta-large encoder while performing joint training on both sub-tasks data.