JUST - BLUE at S em E val-2021 Task 1: Predicting Lexical Complexity using BERT and R o BERT a Pre-trained Language Models (2021)

Abstract: Predicting the complexity level of a word or a phrase is considered a challenging task. It is even recognized as a crucial step in numerous NLP applications, such as text rearrangements and text simplification. Early research treated the task as a binary classification task, where the systems anticipated the existence of a word’s complexity (complex versus uncomplicated). Other studies had been designed to assess the level of word complexity using regression models or multi-labeling classification models. Deep learning models show a significant improvement over machine learning models with the rise of transfer learning and pre-trained language models. This paper presents our approach that won the first rank in the SemEval-task1 (sub stask1). We have calculated the degree of word complexity from 0-1 within a text. We have been ranked first place in the competition using the pre-trained language models Bert and RoBERTa, with a Pearson correlation score of 0.788.

## introduction lexical complexity plays a significant role in the readability level and comprehension. the precise anticipation of lexical complexity can help systems direct the user to an acceptable simple text accurately or modify the text to be more fluid @xcite . predicting the complexity of words is a subjective and challenging problem, while it is conjectural, too. yet, mapping words into their complexity is an essential task to understand natural language. numerous components can influence the prediction of lexical complexity. several approaches were proposed to solve or mitigate this type of study using machine and deep learn-ing methods @xcite @xcite . this paper describes the just-blue team's model that participated in the semeval 2021-task1, lexical complexity prediction (lcp) @xcite . the task provides participants with an augmented version of complex, a multi-domain english dataset with sentences annotated using a 5-point likert scale (1-5) (from very easy to very difficult) @xcite . the task is to predict the complexity value of words in context. it is worth mentioning that our model, just-blue, has been ranked first in this task. we have used the pre-trained language models, bert and roberta which have proven their effectiveness in this area @xcite , along with the ensembling method (weighted averaging) to achieve the highest pearson correlation score of 0.788. the rest of this paper is organized as follows: section 2 sheds light on related work. section 3 describes the methodology proposed in this research. section 4 discusses the experimentation setup and evaluation results. whereas section 5 concludes this research. ## related work one of the most prominent challenges in the current era is the prediction of lexical complexity. prediction of the word complexity in machine learning can be binary; the word is complex or not complex. it also can be a non-binary prediction, as a probabilistic prediction with the measurement of complexity within a particular scale (0.6 the probability that the word is complex). semeval 2016 introduced the first shared task of predicting word complexity with a mission limited to the word orders being complex or non-complex (binary prediction) @xcite . decision tree classifiers achieved the best results @xcite . it has been noted that word length is a good indication of word complexity @xcite . the authors in @xcite discussed the importance of frequency and length of words. they used the keras deep learning library to predict whether an english or spanish word is complex or not. they used character embedding, word length, frequency count, word embedding, and psychological measures as features to predict complex words and achieved 0.872 as f1-score. the authors in @xcite worked on various languages, such as english, spanish, french and german. they worked on two different methods for predicting complex words. the first method is to find if the word orders are either complex or simple. the second is to find the probability that the word is complex. the complex levels depended on the average of the annotators' answers. for example, if the number of annotators who expected the word to be complex is 6 out of 10, then the probability is 0.6. a claim stated that this annotating method is considered impractical since the probability of 0.5 cannot be considered complex or not complex. so the authors in @xcite suggested a likert scale with 5-point. the authors asserted that this method is more accurate scale instead of calling the word complex and noncomplex. we can divide the word into being very easy, easy, neutral, difficult, and very difficult. this scale is beneficial to our work. the deep learning pre-trained language models, bert and roberta, are considered state-of-theart for nlp. teams in the previous shared tasks of semeval 2020 had used these models to obtain the best results for different nlp tasks @xcite @xcite . our approach experimented with these models using different hyperparameters and weighted averaging methods that lead to the best result in the competition for predicting lexical complexity. ## methodology this section describes our approach methodology and goes as follows: first, we describe the task's dataset. then, the preprocessing step. finally, we describe the just-blue approach to predict the word's complexity. ## results and discussion we used python version 3.6 on the colab environment to execute our codes. we have experimented with several models to determine which models are suitable for this task. we have experimented with bert and roberta pre-trained models. we also examined svm and random forest machine learning models. the challenging step was to find the best weights for the models that used tokens (single words) and sentences to get the best result (table2). as we mentioned earlier, some words have a different complexity degree, depending on their location in the sentences. therefore, it was necessary to insert each of the words and sentences for the training to verify the best weight. the next step was to explore bert and roberta's best hyperparameters, such as learning rate, batch size, epochs, and max sequence length. finally, we thought of determining the effects of the base size and large size models of bert and roberta on the accuracy. it is shown by our experiments that the large sizes decreased the accuracy. in the testing phase, we noticed that the words (tokens) in the file were new. therefore, we decided to limit the number of arguments to avoid overfitting. we just changed "num-train-epochs "=3 in bert and roberta's model, but the other arguments had the default values. we have used three different models. the first was the bert model, the second was the roberta model, and the third was bert and roberta together as described in the methodology section. just-blue approach achieved the best result using roberta and bert's models with a pearson correlation of 0.788 scores. we have also achieved the least mean absolute error(mae) with 0.0609. our model is ranked first the lcp-sub task1 of a single word. the spearman's rho (rho) and r-squared (r2) scores are 0.7369 and 0.6172, respectively. the number of teams in the shared task lexical complexity prediction (lcp) was 54 teams. this shared task is considered a high level of cwi 2016 and cwi 2018 with a larger number of words from various sources. ## conclusion predicting the complexity of words is one of the most prominent tasks that the nlp research community strives to solve. it is worth noting that in 2016 and 2018, two tasks were issued to determine whether the word was complex or not. se-meval 2021 introduced task 1, lexical complexity prediction (lcp) that aims to predict the word's complexity from 0 to 1. this paper described the top-ranked team's model, just-blue. the just-blue model obtained the highest pearson correlation score of 0.788 using the pre-trained language models bert and roberta. our strategy depends on the ensembling methods, simple and weighted averaging.