UTFPR at S em E val-2021 Task 1: Complexity Prediction by Combining BERT Vectors and Classic Features (2021)

Abstract: We describe the UTFPR systems submitted to the Lexical Complexity Prediction shared task of SemEval 2021. They perform complexity prediction by combining classic features, such as word frequency, n-gram frequency, word length, and number of senses, with BERT vectors. We test numerous feature combinations and machine learning models in our experiments and find that BERT vectors, even if not optimized for the task at hand, are a great complement to classic features. We also find that employing the principle of compositionality can potentially help in phrase complexity prediction. Our systems place 45th out of 55 for single words and 29th out of 38 for phrases.

## introduction accurately measuring the complexity of words can be useful in many ways. it facilitates the creation of text simplification technologies that could, for example, help in identifying and adapting challenging excerpts of literary pieces targeting specific groups, such as children @xcite and second language learners @xcite , and make news articles and official documents more accessible to the general population @xcite . this task has received a considerable amount of attention in the past few years, especially due to the popularity of the complex word identification (cwi) shared tasks of 2016 @xcite and @xcite , where dozens of teams were challenged to judge the complexity of words in context. while the cwi 2016 task used a simple binary complex/not complex classification setup for english only, the cwi 2018 task explored both a binary classification and a regression setup and multiple languages. the majority of the most successful systems submitted to these shared tasks combined ensemble methods, such as random forests @xcite and adaboost @xcite with numerous linguistic features, including word frequencies, n-gram frequencies, word length, number of senses, number of syllables, psycholinguistic metrics, and word embeddings @xcite @xcite @xcite . however, because these tasks were held prior to the ascension of transformer-based masked language models, such as bert @xcite and roberta @xcite , we could not find any systems that exploited the power of the features produced by them. in this paper, we describe the utfpr systems for the lexical complexity prediction shared task of semeval 2021 (lcp 2021), which combine classic complexity prediction features with contextual word and phrase representations extracted from transformer-based models. in our experiments, we explore the efficacy of a number of different machine learning models, feature combinations, and corpora sources for our features. in what follows, we present the task being addressed (section 2), our approach (section 3), some preliminary experiments (section 4), our final shared task results (section 5), and our conclusions (section 6). ## task description we address the lcp 2021 shared task @xcite , held at semeval 2021. the shared task is split into two sub-tasks: predicting the incontext lexical complexity of single words and phrases for the english language. participants could choose to submit systems to either or both sub-tasks. the organizers provided training, trial and test sets for both sub-tasks. each instance of these datasets is composed of an id, a source identifier, a sentence, a target word or phrase within the sentence, and a complexity score calculated based on judgments made by 20 english speakers from the usa, uk and australia. the source identifier describes from where the sentence came from, the possibilities being the bible, biomedical documents and the europarl corpus. the task's dataset is an extended version of the complex dataset @xcite . the training, trial, and test sets for single words have 7662, 421, and 917 instances, respectively. the training, trial and test sets for phrases have 1517, 99, and 184 instances, respectively. participants were allowed and encouraged to use any external resources they saw fit. ## approach our approach consists of using modern ensemble models to learn from a combination of commonly used complexity estimation features, such as word frequencies, word length, and number of senses, with contextual representations extracted from large pre-trained bert-like models, which have been widely used to create state-of-the-art solutions to numerous tasks. while it has been observed that word frequencies (especially those extracted from spoken text) tend to drive the performance of effective complexity prediction systems @xcite , we hypothesize that the wealth of knowledge present in transformerbased models such as bert can help in extracting complementary contextual complexity clues. ## preliminary experiments in this section, we describe the preliminary experiments we conducted in an effort to engineer our final systems for the lcp 2021 shared task. in these experiments, all machine learning models were trained and optimized on the training set and tested on the trial set provided by the organizers. all models were implemented using the scikit-learn library @xcite ## task results we based the creation of the final utfpr systems on the experiments of the previous section. our final systems are svms trained with word/phrase frequencies, word/phrase length, number of senses, and bert vector (no n-gram frequencies). compositionality in phrases was handled through averaging. frequencies were calculated using a 5-gram language model trained over family movies from subimdb. due to a limitation in time availability, the bert model was used in its original pre-trained form and not optimized for the task at hand. ## conclusions we presented the utfpr systems submitted to the lexical complexity prediction shared task of se-meval 2021. although the placing of our systems were not impressive, we do showcase through our preliminary experiments that employing compositionality can potentially improve the predictions for phrases. we also show that including word length, number of senses, and non-optimized bert vectors to complexity prediction models can noticeably improve their predictions for both words and phrases. in the future, we intend to test the efficacy of adding bert vectors optimized for the task at hand to the pool of features of our models.