M ineria UNAM at S em E val-2020 Task 3: Predicting Contextual W ord S imilarity Using a Centroid Based Approach and Word Embeddings (2020)

Abstract: This paper presents our systems to solve Task 3 of Semeval-2020, which aims to predict the effect that context has on human perception of similarity of words. The task consists of two subtasks in English, Croatian, Finnish, and Slovenian: (1) predicting the change of similarity and (2) predicting the human scores of similarity, both of them for a pair of words within two different contexts. We tackled the problem by developing two systems, the first one uses a centroid approach and word vectors. The second one uses the ELMo language model, which is trained for each pair of words with the given context. Our approach achieved the highest score in subtask 2 for the English language.

## introduction the aim of semeval 2020 task 3 @xcite , graded word similarity in context (gwsc), focuses on predicting the effect that context has in human perception of similarity of words. the most common works in this area deal with the context where a word is placed to predict discrete changes in meaning: the different senses of a polysemous word @xcite @xcite . however, this task highlights the context with more subtle (graded) effects on meaning, even for words not necessarily considered polysemous. that makes word similarity more challenging to be automatically predicted. the main task is divided into two subtasks: the datasets provided for this task contained contextual similarity ratings in four different languages: croatian (hr), english (en), finnish (fi), and slovenian (sl). the pairs of words come from the well-known simlex999 @xcite dataset. the rest of the paper is organized as follows: in section 2 our methodology is presented, along with some preliminary results using the corpus available in the practice phase. this will guide us to improve each approach. the final results of all systems with the evaluation corpus is reported in section 3. the paper ends with some conclusions in section 4. ## methodology to approach the problem, we used two different methodologies. the first one, named a1, is inspired in a clustering algorithm that uses centroids to represent the center of a particular group of datapoints. the second one, named a2, uses elmo embeddings @xcite to represent words. this task is not supervised, thus, we do not have a training corpus itself; instead, a practice kit was distributed by the organizers, composed by a reduced amount of word pairs with their given context, similarity, and similarity change score. it is important to notice that, for practical reasons, all the experiments were done with the english dataset, and the techniques were later extended to the other languages. ## results we tested each approach in all the languages of the competition. finally, we select our best scores regarding the evaluation phase, having different combinations for the two subtasks. finally, we tested all the approaches in subtask 2, showing in ## conclusion we present different approaches to solve the problem of predicting word similarity in different contexts. the model introduced in a1, based on centroids, presents several iterative improvements that refine the method. our team, mineriaunam, considers that centroid-based techniques are better to solve subtask 2. the ranking positions we got in this competition are as follows: 1st place for english, 3rd in croatian and finnish, and 7th in slovenian. the algorithm a1[3], based on multicentroids achieved the best correlation with human taggers on subtask 2. the second approach, a2, is based on elmo language model, which turns out to be better for subtask 1. the theoretical bases of the method seem to be correct, since using a context-dependent model guarantees a consistent association of the degree of similarity between two words within any given sentence. besides, being a deep neural model, it is possible to work with a large amount of data, allowing this model to be general enough for tasks such as this one. however, this approach was unable to produce good similarity scores between the pair of words. the positions on the final board of substask 1 were as follows: 9th. in english and 8th. in croatian, slovenian and finnish. in the centroid based approach, we used the simlex original similarity values and modified it with respect to each context. in the case of the elmo based approach the similarity is calculated independently from the simlex. we believe that this fact, starting with a previous validated similarity score, was an important factor for the high performance of the centroid based approach.