RS _ GV at S em E val-2021 Task 1: Sense Relative Lexical Complexity Prediction (2021)

Abstract: We present the technical report of the system called RS_GV at SemEval-2021 Task 1 on lexical complexity prediction of English words. RS_GV is a neural network using hand-crafted linguistic features in combination with character and word embeddings to predict target words’ complexity. For the generation of the hand-crafted features, we set the target words in relation to their senses. RS_GV predicts the complexity well of biomedical terms but it has problems with the complexity prediction of very complex and very simple target words.

## introduction text simplification is the process of modifying a text so that it becomes easy for the reader to understand the meaning of the text without any loss of information. a main part of text simplification is lexical simplification. in lexical simplification, complex words are replaced with easier or more frequent synonyms. following @xcite , the process of lexical simplification can be split as follows: i.) identification of complex words in a given text, ii.) substitution generation, iii.) word sense disambiguation, iv.) synonym ranking, v.) substitution of complex word with the best synonym in correct morphological form. following @xcite , the most common errors in lexical simplification are that the words are not identified as complex or that words are incorrectly identified as complex. one reason might be the approach to predict complex words. so far, in the task called complex word identification (cwi), a word in a sentence was labeled as either complex or simple without any range in between. @xcite criticize this approach because there is no clear threshold when a word starts to be complex. hence, they propose a new task called lexical complexity prediction @xcite . the aim of lcp is to predict the complexity of a single word or a multi-word expression on a scale of 0 to 1. this paper proposes rs gv , a model for lcp in the context of the semeval-2021 task 1 @xcite . rs gv uses hand-crafted features relative to their wordnet senses, flair embeddings and a neural regressor in a cross-domain and withindomain setting. ## related work lexical complexity prediction is a new sub-task of lexical text simplification. the aim is to predict the complexity of a single word or a multiword expression on a scale of 0 to 1. the most similar task is cwi. in contrast to lcp, cwi aims at binary classification that determines whether a word is complex or not. as lcp has been mentioned for the first time in the context of this shared task @xcite , no other related work exists yet. hence, we outline the state of the art in cwi. semeval-2016 task 11: @xcite collated 9200 sentences from the cw corpus @xcite , the lexmturk corpus @xcite , and the simple wikipedia corpus @xcite . all these corpora were based on the simple english wikipedia (sew). cwi was treated as a binary classification task, wherein 400 non-native speakers annotated content words in english text. it was observed from the annotations that complex words were shorter, less ambiguous and had a low occurrence in sew. f-score and g-score were used as the evaluation metrics. the features incorporated by the submitted systems can be seen in figure 1 . it is shown that the word frequency, lexical, semantic and morphological features play a dominant role in cwi. besides these, n-gram features were also experimented with by a few systems. word embeddings were not used extensively. cwi shared task 2018 another shared task on complex word identification was organized in 2018 @xcite . @xcite collected data from three sources, i.e., professionally written news, wikinews and wikipedia, and in four languages, i.e., english, german, french, and spanish. the shared task was composed of two sub-tasks. sub-task 1 approached the problem as a binary classification problem and sub-task 2 treated it as a probabilistic classification problem, wherein the score between 0 and 1 indicated the proportion of annotators who considered a word as complex. native as well as non-native readers annotated the dataset created by @xcite . a word was deemed to be complex if at least one out of twenty annotators labeled it as complex. based on annotations, it was observed that the systems might perform better when trained on domain-specific data. it was also found that traditional feature engineering-based approaches performed better than neural network and word embedding based approaches. the features incorporated by the submitted systems of 11 teams can be seen in figure 1 . the graph reinstates the fact that frequencies, lexical, semantic and morphological features play a key role in cwi. however, it was observed that as compared to 2016, in 2018, word embeddings were more commonly used. ## system description our system's main characteristics are a combination of hand-crafted features, contextualized character embeddings (see subsection 4.1), a sense relative normalization (see subsection 4.2), and a neural network for regression (see subsection 4.4). ## discussion and conclusion we described our model named rs gv which was submitted to semeval task 2021: task 1 regarding lexical complexity prediction. we propose a neural network with a combination of hand-crafted and word/character embeddings to approach the task. our analysis shows that normalization of handcrafted features using wordnet senses achieves better results than using only a minmax normalization. furthermore, we figured out that rs gv predicts lexical complexity best using a combination of non-contextualized word embeddings and contextualized character embeddings. in contrast to other shared tasks results, our cross-domain approach achieves better results than the domain-specific approach. a domain-specific approach may need more data to perform reliably. furthermore, our neural regressor seems problematic, since it shows some variance in the results on average and the current dataset might be too small for regression with neural networks.