JCT at S em E val-2021 Task 1: Context-aware Representation for Lexical Complexity Prediction (2021)

Abstract: In this paper, we present our contribution in SemEval-2021 Task 1: Lexical Complexity Prediction, where we integrate linguistic, statistical, and semantic properties of the target word and its context as features within a Machine Learning (ML) framework for predicting lexical complexity. In particular, we use BERT contextualized word embeddings to represent the semantic meaning of the target word and its context. We participated in the sub-task of predicting the complexity score of single words

## introduction over the last decade, automated methods for detecting complex words have been developed. at the beginning, most of these methods assumed that lexical complexity is binary, words are either "difficult" or "not difficult". thus, the first complex word identification (cwi) shared task referred to binary identification of complex words @xcite . the main limitation of this assumption is that a word close to the decision boundary is considered to be as complex as one farther apart. therefore, three years ago, the cwi included an additional probabilistic classification task where the participants were asked to give a probability of the given target word in particular context being complex @xcite . recently, complex, a new english corpus for lexical complexity prediction was introduced @xcite . the corpus is annotated using a 5-point likert scale (1-5) (corresponding to very easy, easy, neutral, difficult, and very difficult), and covers 3 genres: bible translation, european pariliament proceedings, and biomedical articles. semeval-2021 (task 1) shared task on lexical complexity prediction (lcp) @xcite provided the participants with complex and defined two sub-tasks: predicting the com-plexity score of single words, and predicting the complexity score of multi-word expressions. we present our system for the first sub-task of predicting the complexity score of single words. our system incorporates linguistic, statistical, and semantic properties of the target word and its context as features within a machine learning (ml) framework for predicting lexical complexity. this paper is organized as follows: first, in section 2, we describe features from previous works that we have adopted. then, in section 3, we describe our feature sets, the feature selection process, and the results on the trial data. finally, our system results on the test data are detailed in section 4, followed by conclusions in section 5. ## related work in this section, we shortly describe linguistic, statistical, and semantic features which were encoded as features in previous complexity prediction tasks and were integrated in our system. linguistics features, such as part-of-speech (pos) tag, dependency parsing relations, and syllable counts, as well as statistical features, such as word length and word frequency, have been widely used for predicting lexical complexity @xcite @xcite @xcite @xcite . some of these works found wordnet @xcite as a valuable source of lexical features. the main extracted feature is the number of synsets, but also information on hypernyms, hyponyms, holonym, and meronym is useful @xcite @xcite . semantic features were commonly encoded using word embedding representation of the meaning of words @xcite . these word embeddings were generated using word2vec context-independent models @xcite . word2vec models combine different senses of the word into one single vector. however, recently, there is a growing interest in contextualized word representations, such as bert @xcite . bert model generates context-dependent embeddings that allow a word to have several vector representations depending on the context in which it is used. in contrast to previous works that only use context-independent embeddings, our system uses the bert-based contextdependent embeddings. ## system description we adopt a supervised machine learning (ml) approach for lexical complexity prediction. the first step in a classifier training is to determine which text characteristics are relevant and how those features are coded. ## results to increase the size of our train set for the test phase of the task, we used both the train and trial sets to train the final model. we have implemented a system that incorporates linguistic, statistical, and semantic features to predict lexical complexity of target word in context. bert semantic space was used to represent the word and its context. we investigated several feature selection approaches and used various supervised algorithms. even though our system was not highly ranked, we believe that some of the presented ideas can be useful for future research on lexical complexity prediction. in particular, we think that bert is a powerful model that should be explored. perhaps, fine-tuning bert for the complexity prediction task would increase the system performance.