B abel E nconding at S em E val-2020 Task 3: Contextual Similarity as a Combination of Multilingualism and Language Models (2020)

Abstract: This paper describes the system submitted by our team (BabelEnconding) to SemEval-2020 Task 3: Predicting the Graded Effect of Context in Word Similarity. We propose an approach that relies on translation and multilingual language models in order to compute the contextual similarity between pairs of words. Our hypothesis is that evidence from additional languages can leverage the correlation with the human generated scores. BabelEnconding was applied to both subtasks and ranked among the top-3 in six out of eight task/language combinations and was the highest scoring system three times.

## introduction word similarity is a key task in natural language processing (nlp) applications. language models, such as word embeddings @xcite create vector representations for the words that are able to capture syntactic and semantic relationships. these representations became very popular in the last few years as they have boosted the performance of several nlp tasks. however, since each word is represented by a fixed vector these techniques have problems dealing with polysemous words and identifying subtle meaning changes between different sentences. on the other hand, state-of-the-art language models, like bert @xcite provide a contextualized word representation -the representation of a word relies on its context, which means that the same word may have different representations through the sentences. thus, bert models are more suitable for handling polysemous words. task 3 in semeval 2020 -predicting the graded effect of context in word similarity @xcite was motivated by this improvement on language models. the task aims at the design of a similarity measure which captures the human perception of the meaning of words. for that purpose, task organizers built and annotated datasets in four languages -english, croatian, finnish, and slovenian. each entry in a dataset consists of two target words and two contexts, where each one is a piece of text containing both target words. the global task is divided into two subtasks: 1) predicting the change in the human annotator's scores of similarity when presented with the same pair of words within two different contexts; and 2) predicting the human scores of similarity for a pair of words within two different contexts. in this paper, we describe babelenconding, an approach that relies on machine translation and multilingual language models to evaluate the contextual similarity of pairs of words. our hypothesis is that having similarity information from more languages helps decide on how similar the words are. considering the eight combinations of language/subtask, babelenconding was ranked among the top-3 competitors six times, and was the top scoring method in three cases. our additional experiments in english and croatian showed that adding more languages noticeably improved the results for croatian in both subtasks. in english, the gain was small and happened only in subtask 2. ## background and related work the distributional hypothesis @xcite states that the meaning of a word changes depending on the context it is used. at the same time, this hypothesis also states that if two words tend to be used in the same this work is licensed under a creative commons attribution 4.0 international license. license details: http:// creativecommons.org/licenses/by/4.0/ . in state-of-the-art language models, such as bert @xcite , the context of a word is taken into account in its representation. these models are trained over a large corpus to predict missing tokens which are removed from the original sentences. an advantage of bert over word2vec is that it creates different representations for the same word depending on the context in which the word appears. another advantage of bert-like models is that they can be specialized for a specific task with few training epochs. solutions for measuring contextual similarity between word pairs and word-sense disambiguation benefited from bert-like language models. enriched models were designed @xcite @xcite , and new datasets such as the word-in-context dataset (pilehvar and camacho-collados, 2018) and cosimlex @xcite were assembled. word-sense disambiguation can also take advantage of multilingualism. some works have employed parallel/comparable corpora @xcite and translation @xcite to that task. multilingual resources, such as multi-simlex @xcite , were also developed and yielded improvements compared with the monolingual version. ## babelenconding our proposed solution, called babelenconding, works in two phases and its overall process is depicted in figure 1 . the input is a pair of words and two sentences (contexts) containing both words of interest. more formally, let s 1 = {w 1 1 , w 1 2 , . . . , w 1 i } and s 2 = {w 2 1 , w 2 2 , . . . , w 2 j } be two sentences, where there is a pair of words @xmath0, w @xmath1. for example, let s 1 = "her prison cell was almost an improvement over her room at the last hostel" and s 2 = "his job didn't leave much room for a personal life. he knew much more about human cells than about human feelings" be two sentences, where the pair of words @xmath2, cell . in the first phase of babelenconding, both input sentences s 1 and s 2 are translated into a set of k languages @xmath3, l 2 , . . . , l k }. this process will produce a set of translated sentences s l @xmath4, s l i 2 , which corresponds to the translation of the original sentences, into each language l @xmath5. then, the words of interest are identified in the translated text, generating two sets p l i s 1 and p l i s 2 . in this example, figure 1 : overview of babelenconding considering @xmath6, portuguese}, the pairs of words of interest are translated as p it s 1 = cella, stanza , p pt s 1 = cela, quarto from s 1 and p it s 2 = spazio, cellule , p pt s 2 = espac ¸o, células from s 2 . in the second phase, with the translated sentences, we evaluate the similarity between the pair of words in p for each language in l separately in two ways: (i) using word embeddings and (ii) bert. finally, babelenconding calculates a weighted average between word embeddings and bert similarities. these similarities are used to address both subtasks. word embedding similarity consists in taking the cosine similarity between the word vectors of the two words in each language. we rely on pre-trained monolingual word embeddings to represent the words. the context is not used in this similarity measure since there is a fixed vector for each word. bert similarity requires inferring the word embedding representation of words in bert models, taking context into consideration. the context is the sentence (s m ) containing the two words. this process was done summing the last four hidden layers of the bert model. this choice was made based on the good results achieved by @xcite in the named entity recognition task. babelenconding similarity consists on a weighted average between word embedding and bert similarities scaled in multiple languages. equation 1 shows how babelenconding calculates the similarity between words w 1 and w 2 within sentence s m . in this equation, α and β are the weights given to bert and word embedding similarities, respectively. our hypothesis is that having similarity information from more languages helps decide on how similar they are. the underlying assumption is that if two words are translated to the same word in other language, they are more likely to be more similar. translation also helps identifying dissimilarity between words as it can help to disambiguate terms. preliminary tests showed that, once both words occur together in the same context, the similarity between words tended to be undesirably high when using just bert representations. this effect can be attributed to bert's attention mechanism. thus, a combination of bert and fixed word embeddings was designed to alleviate this issue. ## experimental setup dataset. the dataset used in our experiments was cosimlex @xcite which consists of 340 sentence pairs in english (en), 112 in croatian (hr), 111 in slovene (sl), and 24 in finnish (fi). please refer to that paper for details on the annotation methodology. languages. the source sentences were translated into the following languages: english (en), spanish (es), italian (it), bosnian (bs), german (de), greek (el), polish (pl), portuguese (pt), russian (ru), serbian (sr), and turkish (tr). this choice was made based on the main languages used in word sense disambiguation tasks @xcite @xcite @xcite . ## results results for the official runs. the system configurations that achieved the best results in the official runs are shown in ## conclusion in this paper, we described our system submitted to semeval-2020 task 3. we designed an approach that relies on translation and multilingual language models in order to compute the contextual similarity between pairs of words. the key idea is that having similarity information from different languages may help decide on how similar the words are. our system achieved competitive results in both subtasks, being ranked among the top-3 in most runs. in these preliminary experiments, we could not establish in which cases more languages are helpful and we leave it as future work. additionally, we are interested in understanding which factors contribute to improvement in the results -whether it is the amount of data used for training the language models or individual features of the language.