C 3 SL at S em E val-2021 Task 1: Predicting Lexical Complexity of Words in Specific Contexts with Sentence Embeddings (2021)

Abstract: We present our approach to predicting lexical complexity of words in specific contexts, as entered LCP Shared Task 1 at SemEval 2021. The approach consists of separating sentences into smaller chunks, embedding them with Sent2Vec, and reducing the embeddings into a simpler vector used as input to a neural network, the latter for predicting the complexity of words and expressions. Results show that the pre-trained sentence embeddings are not able to capture lexical complexity from the language when applied in cross-domain applications.

## introduction lexical complexity plays a crucial role in reading comprehension. predicting lexical complexity of words within sentences in specific textual context can enable systems to better perform certain nlp tasks, such as simplifying texts, and favoring less fortunate readers in a giving target language. the semeval 2021 proposes a lexical complexity prediction (lcp) shared task (task 1) @xcite based on a new annotated english dataset with a likert scale @xcite . word embeddings @xcite are very important resources that support several nlp tasks by providing a semantic latent representation that heuristically captures relationships in language that are very difficult to observe otherwise. furthermore, such semantic representation can be used to embed language structures other than just words. sent2vec @xcite is an unsupervised model designed to compose sentence embeddings using word vectors along with n-gram embeddings, simultaneously training composition and the embedding vectors themselves. sent2vec has been used in several nlp tasks, such as analysing semantic properties of sentences @xcite , classification of sentences in the biomedical domain @xcite , automatic detection of incoherent speech @xcite , and measuring sentence similarity @xcite , in this work, we aim to test the ability of sent2vec to detect the complexity of english words and expressions in specific contexts. we evaluate in what extent the semantic information captured when learning the embeding representation is able to incorporate word complexity. our approach uses pretrained sent2vec models and aims to validate in what extent such models are able to predict complexity of words. results show strong evidence that pretrained sentence embeddings do not capture complexity features from language, specially when applied in cross-domain applications. ## method the proposed shared task consists of determining the complexity of token words in the context of given input sentences (lexical complexity prediction-lcp). training input sentences annotated with a likert scale corresponding to the complexity score of target words are given. sub-task 1 focus on predicting the complexity score of single words. meanwhile, sub-task 2 targets multi-word expressions. our overall strategy consists on generating chunks of the sentence, embed those chunks with sent2vec and then reduce those embeddings into a simpler vector, which would then finally be used as the neural network's input. our approach uses pretrained sent2vec models to obtain embedding representation of multiple parts of the input sentence split by the target token words. for a given input sentence s, we obtain embeddings for: a) s: the full sentence; b) s 0 : the amount of text from the beginning of the sentence up to the target token(s) (except the latter); c) t : the target word token(s), and d) s 1 : the amount of text from the target token(s) up to the end of the sentence (except the former). when target tokens are in the beginning or in the end of a sentence, s 0 or s 1 are represented by a zero-value vector. we reduce the vector representation of each sentence constituent into distance-based arrays that are fed into a neural network estimator. the reduction itself consisted of simplifying the text (using spacy) and then feeding different chunks into sent2vec. finally, an array of the distances between the chunks' embeddings to the token word's embeddings was used as input to a neural network coming from the scikit-learn package. we use the context given by the two splits of the input sentence (s 0 or s 1 ) to measure the complexity of the token word(s) (t ). thus, our pipeline consists of four major steps: text preprocessing, chunking, context embedding and estimator training. the chosen estimator was a multi-layer perceptron (mlp). figure 1 illustrates this pipeline for an example sentence and token word. ## results the task is divided between two subtasks: singleand multi-word tokens. both have the same format: each line of the input has an id, a sentence, a corpus to which it belongs, and a token. for training data, each line also includes a complexity value. the difference between the first and second subtasks is that the first limits a token to one single word, while the second doesn't. tables 1 and 2 present the final results of the competition for the two subtasks, respectively. participant's performance is ranked in each subtask separately. for the submissions without a team name -that is, the submissions made by users not linked to teams -, the user name is available inside brackets. the tables also show each participant's scores for individual corpora. our approach is identified as c3sl. we believe the flaws in our approach can be explained in two-fold: a) pretrained embeddings are sensitive to the domain or context used during trained, and the way semantic information is captured in a latent representation is not reflected the in the same way in cross-domain applications; and b) even if a latent representation of text would able to capture semantic complexity of certain expression in context, the same is not reflected by the norm or cosine distances between multiple chunks and the target expressions. ## related work some of the previous work show that sent2vec is not the best alternative to consistently achieve high accuracy in subsequent nlp and nlu tasks based on embedding representation of sentences, except when training sent2vec with domain-specific corpora. @xcite attempt to use the twitter unigram pretrained model from sent2vec in order to improve their approach when performing extraction of adverse drug reactions from tweets. however, results show that utilizing sent2vec as tweet representations did not improve classification quality. an empirical study of sentence embeddings @xcite aims to analyse in what extent linguistic information is retained in vector representations of sentences by comparing ten embeddings approaches. results show that sent2vec was only able to outperform other approaches in only 1 out of 11 tasks (word classification task). @xcite use sent2vec trained on the wmt18 news translation task parallel training data @xcite to calculate distance of sentence vectors aiming to improve their semantic textual similarity approach when filtering a noisy web crawled parallel corpus. @xcite aim to automatically extract linguistic features for detecting symptoms of schizophrenia. they compare a number of sentence embeddings and show that, although sent2vec outperforms the mean vector sentence embedding (used as a baseline model within the experiments), all other models perfom better when measuring coherence using concept overlap and ambiguous pronoun usage. @xcite evaluate semantic properties of sentence embeddings models in five tasks: negation detection, negation variants, clause relatedness, argument sensitivity, and fixed point reorder. results show that sent2vec is only able to outperform other embedding models in the clause relatedness task, which explores whether the similarity between a sentence and its embedded clause is higher than between a sentence and its negation. the resulting accuracy is in the 30-35% range. ## conclusions we presented an approach for the lcp shared task 1 at semeval 2021 for predicting the lexical complexities of words in context. we used pretrained sent2vec models using a vector formed from embedded chunks of sentences as input for a neural network to perform as the final complexity estimator. implemented and executed on the google colab platform, our approach used little resources. the results show that pretrained sent2vec models alone cannot capture a semantic representation that reflects a word's or expression's complexity within cross-domain contexts. as a way of extending our experiments, we plan to perform further analysis in subsequent classification tasks in the biomedical domain using reinforcement strategies to enrich the semantic information captured by embedding representation of sentences.