cs60075_team2 at S em E val-2021 Task 1 : Lexical Complexity Prediction using Transformer-based Language Models pre-trained on various text corpora (2021)

Abstract: The main contribution of this paper is to fine-tune transformer-based language models pre-trained on several text corpora, some being general (E.g., Wikipedia, BooksCorpus), some being the corpora from which the CompLex Dataset was extracted, and others being from other specific domains such as Finance, Law, etc. We perform ablation studies on selecting the transformer models and how their individual complexity scores are aggregated to get the resulting complexity scores. Our method achieves a best Pearson Correlation of 0.784 in sub-task 1 (single word) and 0.836 in sub-task 2 (multiple word expressions).

## introduction complex words hinder the readability of a text, as discussed in @xcite . to mitigate this problem, there is a necessity of lexical simplification @xcite , and predicting the complexity of words is an integral part of this process. language models learn the probability of cooccurrence of words in a corpus. they have been used for various sentence completion and text-based classification tasks. the first language models were n-gram markov models @xcite , which performed well for tasks that did not require very long-range dependencies. then came rnns @xcite , lstms @xcite and grus @xcite , which were able to understand longer contexts, but struggled with long paragraphs due to the vanishing gradient problem. transformers @xcite were a task-agnostic solution that performed better due 1 the code is available at https://github.com/ abhi1nandy2/cs60075-team-2-task-1 to the presence of attention layers between hidden layers of the neural network, which helped the layers of the neural network to look at the entire input at once. transformers can perform very well on a broad suite of tasks by fine-tuning on a small number of task-specific samples. the intuition behind using such transformer-based language models for lexical complexity prediction (lcp) was -transformer models pre-trained on different corpora would mimic annotators (of the complex dataset (com)) having different backgrounds. since the final score is an aggregation of the annotation scores given by annotators, we aggregate the various scores that are given as outputs by the transformer-based models fine-tuned on the complex dataset. the rest of the paper is organized as follows. section 2 gives an overview of our solution approach, section 3 talks about the corpora used for pre-training and the dataset used for fine-tuning for the lcp task, section 4 discusses the experimental settings, baselines used, and a comparison and analysis of the results and section 5 gives a conclusion. ## conclusion we show that aggregating the results of various fine-tuned transformer models pre-trained on various corpora from different domains gives high pearson correlation and low mean squared errors compared to individual transformers and regression models using attributes such as hand-crafted features, word embeddings, transformer-based language model prediction probabilities, etc. this shows that transformer-based language models, each pre-trained on a different text corpus, can better imitate annotators of the dataset, who come from diverse backgrounds and prior knowledge.