IITK @ LCP at S em E val-2021 Task 1: Classification for Lexical Complexity Regression Task (2021)

Abstract: This paper describes our contribution to SemEval 2021 Task 1 (Shardlow et al., 2021): Lexical Complexity Prediction. In our approach, we leverage the ELECTRA model and attempt to mirror the data annotation scheme. Although the task is a regression task, we show that we can treat it as an aggregation of several classification and regression models. This somewhat counter-intuitive approach achieved an MAE score of 0.0654 for Sub-Task 1 and MAE of 0.0811 on Sub-Task 2. Additionally, we used the concept of weak supervision signals from Gloss-BERT in our work, and it significantly improved the MAE score in Sub-Task 1.

## introduction with the rapid growth in digital pedagogy, english has become an extremely popular language. although english is considered an easy language to learn and grasp, a person's choice of words often affects texts' readability. the use of difficult words can potentially lead to a communication gap, thus hampering language efficiency. keeping these issues in mind, many natural language processing tasks for text simplification have been recently proposed @xcite . our task of lexical complexity prediction is an important step in the process of simplifying texts. the semeval 2021 task 1 @xcite focuses on lexical complexity prediction in english. given a sentence and a token from it, we have to predict the complexity score of the token. the task has two sub-tasks-sub-task 1: complexity prediction of single words sub-task 2: complexity prediction of multi word expressions (mwes). a word might seem complex because of 2 major * authors equally contributed to this work. ## background this task uses the complex dataset @xcite , which is a lexical complexity prediction dataset in english for single and multi word expressions (2-grams). sentences in this task consists of sentences taken from 3 corpora-bible, biomed and europarl. the train, validation and test split of the data was 9179, 520, 1103 respectively. we used the trial data as the validation set. the aim of the task is to predict how complex a given token in a given sentence is. more mathematically, given a tuple [s, t, c], where @xmath0, t 2 , ...t n ] and @xmath1, we have to give an estimate of the function σ, such that σ(s, t) = c. (s is the sentence, t is the token and c is the complexity score). the earlier focus on this task has been through the semeval 2016 task 11 (paetzold and specia (2016a)). however, it was a binary classification task. most of the participating systems used support vector machines such as kuru (2016) and choubey and pateria (2016), decision trees and random forests (choubey and pateria (2016), brooke et al. (2016), ronzano et al. (2016)), and even basic threshold based approaches (kauchak (2016), malmasi et al. (2016)). very few of them, including bingel et al. (2016) used neural networks. the system by wróbel (2016) achieved an f1 score very close to the winning solution using only single feature -word frequency from wikipedia. most of these systems use word embeddings, pos information and word frequencies as features. the winning system by paetzold and specia (2016b) however uses 69 morphological, semantic and syntactic features. another related shared task was presented at the bea workshop at 2018 (yimam et al., 2018). it had a probabilistic task as well as a binary classification task. even there, the organizers conclude that feature engineering has worked better than neural networks. the winning system by gooding and kochmar (2018) uses feature engineering and later random forest and linear regression models. ## system overview our proposed pipeline can be divided into the following 4 main componentsa) feature extraction b) regression pipeline c) classification pipeline d) ensemble the pipeline is shown in figure 3 . ## experimental setup the official evaluation metric for both the sub-tasks was pearson correlation (standard for regression tasks). for both sub-tasks, the train/test/val split as per the official release has been used. the electra finetuning was done with an nvidia gtx 1080 gpu with early stopping (93 epochs). we used the mae loss function to train the model with an adam optimizer with @xmath2, @xmath3. training set was shuffled and the batch size was kept at 64. in the electra model, the padding parameter was set to true and maximum length was at 140. for the svm models the value of slack was chosen to be 1 and for svm and linear regresion the sklearn @xcite library was used. all the hyperparameters were tuned with a grid search method. ## results results on validation data: the comparison of the baseline results and our results obtained using the regression pipeline, the classification pipeline and the ensemble of the two models on the validation set (trial data) is given in in majority of the prior work on lcp, there is abundance use of word frequency as a feature. however, in this system the scores got worse when frequency features were used along with others in ensemble. and the feature in itself could not produce competitive results. previously, @xcite and @xcite have shown that frequency information causes significant distortion in the embedding space. we also hypothesize that the frequency information in glove embeddings help us in this regard. ## conclusion in this paper we presented a system for lexical complexity prediction in the form of a regression task. the proposed system's primary novelty is in treating it as a classification task and trying to model the annotation scheme. an ensemble of these classification models and vanilla fine-tuning of elec-tra model proved to be very useful. also the weak supervision based approach gave the scores a significant boost for the sub-task 1. 4 ) and ( 5 ) sentence and character level gru and lstms ( 6 ), ( 7 ), ( 8 ), ( 9 ) and ( 10 2016 task 11: the importance of designing your neural networks right. in proceedings of the 10th international workshop on semantic evaluation (semeval-2016), pages 1028-1033, san diego, california. association for computational linguistics.