C itius NLP at S em E val-2020 Task 3: Comparing Two Approaches for Word Vector Contextualization (2020)

Abstract: This article describes some unsupervised strategies submitted to SemEval 2020 Task 3, a task which consists of considering the effect of context to compute word similarity. More precisely, given two words in context, the system must predict the degree of similarity of those words considering the context in which they occur, and the system score is compared against human prediction. We compare one approach based on pre-trained BERT models with other strategy relying on static word embeddings and syntactic dependencies. The BERT-based method clearly outperformed the dependency-based strategy.

## introduction the goal of semeval-2020 task 3 -predicting the (graded) effect of context in word similarity @xcite , is to predict the degree of similarity of two words considering the context in which those words appear. it should be noted that the context of a word will always have very subtle and continuous (graduated) effects on the lexical meaning, even in the case of those words that are not considered polysemous @xcite . this view of lexical meaning variation as a continuum, inspired in the notion of sense discrimination by @xcite , is quite different from resource-based word sense disambiguation approaches @xcite , which use the local context to predict discrete changes in meaning: the different predefined senses of a polysemous word. in order to grasp the contextualized sense of words, we will make use of two different strategies: transformers: contextualized word representations produced by bert, a bi-directional transformerbased language model for context-sensitive word representations @xcite . dependencies: contextualized word representations produced by considering the static embeddings of syntactically related words to each target word. this paper is organized as follows. the two strategies are described in section 2. experiments and results are presented in section 3. finally, conclusions are addressed in section 4. ## two methods for word contextualization as mentioned in the introduction, we made use of two different strategies, transformers and dependencies, to tackle the challenge of contextualizing word senses. ## conclusions and future work we have described two different strategies, one based on pre-trained bert models and other on static word embeddings and syntactic dependencies, to predict the degree of similarity of a pair of words considering the context in which they occur. evaluation was performed by participating to the semeval-2020 task 3: comparing two approaches for word vector contextualization. the bert-based method clearly outperformed the dependency-based strategy. as for the first one, the four last layers on the neural network were combined in different ways, but no configuration was clearly better than the others. in relation to the syntactic approach, it is worth noting how nouns help to shape syntactic contexts that are more discriminating than those built with only adjectives or just verbs. in future work, we will try to find a way to experiment with strategies that combine syntactic contexts with transformer's models.