CSECU - DSG at S em E val-2021 Task 1: Fusion of Transformer Models for Lexical Complexity Prediction (2021)

Abstract: Lexical complexity prediction (LCP) conveys the anticipation of the complexity level of a token or a set of tokens in a sentence. It plays a vital role in the improvement of various NLP tasks including lexical simplification, translations, and text generation. However, multiple meaning of a word in multiple circumstances, grammatical complex structure, and the mutual dependency of words in a sentence make it difficult to estimate the lexical complexity. To address these challenges, SemEval-2021 Task 1 introduced a shared task focusing on LCP and this paper presents our participation in this task. We proposed a transformer-based approach with sentence pair regression. We employed two fine-tuned transformer models. Including BERT and RoBERTa to train our model and fuse their predicted score to the complexity estimation. Experimental results demonstrate that our proposed method achieved competitive performance compared to the participants’ systems.

## introduction lexical complexity prediction (lcp) has become an important task in this globalization age, especially for second language learners @xcite . lcp is a little bit expansion of complex word identification (cwi) task @xcite , where cwi is a binary classification of a word that is complex or not and lcp is finding the complexity level of a word in continuous labelling in a sentence @xcite . lcp plays a vital role in many nlp applications such as lexical simplification @xcite @xcite ), text generation, and machine translation @xcite . besides, it helps those people who are suffering from **the first two authors have equal contributions. dyslexia @xcite , aphasia @xcite , and those with low literacy levels @xcite . lcp is a very challenging task @xcite , especially because the non-identical target audiences will have distinct needs. for example, speakers of one language usually less familiar with different subsets of the vocabulary of a second language. besides, the grammatical shape of a sentence and the ambiguous meaning of a word in different places make this task more challenging and important to explore. a single word may portray different lexical complexity because of its non-identical usage, position, tense form, and redundancy in different sentences or in the same sentence. to estimate multi-word complexity, we need to consider the dependency between tokens. ## proposed lexical complexity prediction framework in this section, we describe our proposed lexical complexity prediction framework. our goal is to predict the complexity score of a token or a set of tokens in the given sentence. we depict the overview of our framework in figure 1 . ## conclusion and future directions in this paper, we presented our approach to the lexical complexity prediction task. we tackled the problem by performing sentence pair regression using two sota transformer models including bert and roberta in a unified architecture. by using pairwise learning, we exploited the contextual relation between sentence-word pairs to estimate the complexity score. our method achieved competitive scores compared to other participants. in the future, we have a plan to incorporate various handcrafted features with state-of-the-art neural methods to distill the relationship of sentenceword pairs for complexity estimation.