S em E val-2020 Task 3: Graded Word Similarity in Context (2020)

Abstract: This paper presents the Graded Word Similarity in Context (GWSC) task which asked participants to predict the effects of context on human perception of similarity in English, Croatian, Slovene and Finnish. We received 15 submissions and 11 system description papers. A new dataset (CoSimLex) was created for evaluation in this task: it contains pairs of words, each annotated within two different contexts. Systems beat the baselines by significant margins, but few did well in more than one language or subtask. Almost every system employed a Transformer model, but with many variations in the details: WordNet sense embeddings, translation of contexts, TF-IDF weightings, and the automatic creation of datasets for fine-tuning were all used to good effect.

## introduction contextualised word embeddings, produced by models such as elmo @xcite and bert @xcite , have quickly become the standard in nlp systems. they deliver impressive performance in language modeling and downstream tasks; but there are few resources available which allow intrinsic evaluation in terms of the properties of the embeddings themselves, or their ability to model human perception of meaning, and how these depend on context. for non-contextualised models, resources like wordsim-353 @xcite and simlex-999 @xcite were instrumental to evaluate their ability to reflect human similarity judgements. however these datasets treat pairs of words in isolation, and thus cannot tell us much about the effect of context. the few resources that work with context, like scws @xcite , wic (pilehvar and camacho-collados, 2019), and wsim @xcite , focus on word sense and discrete effects, thus missing the more graded effects that context has on words in general, and that approaches like elmo and bert would seem well suited to model. further, usim @xcite focuses on separate sentential contexts only in the english language. the goal of semeval-2020 task 3: graded word similarity in context, was to move towards filling that gap. we created a new dataset, cosimlex @xcite , which builds on the familiar pairwise, graded similarity task of simlex-999, but extends it to pairs of words as they occur in context; specifically, each pair of words appears together in two different shared contexts (see figure 1 ). the task was designed to test the ability of participating systems to reflect human judgements of word meaning similarity in context, and crucially, the way in which this varies as context is changed. in addition, since cosimlex takes the gradedness of human judgements into account, the task applies not only to polysemous words, or words with distinct senses, but to the phenomenon of context-dependency of word meaning in general. the dataset is also multi-lingual: besides english, it includes three less-resourced european languages, croatian, finnish, and slovene. word1: man word2: warrior simlex: µ 4.72 σ 1.03 context1 context1: µ 7.88 σ 2.07 when jaimal died in the war, patta sisodia took the command, but he too died in the battle. these young men displayed true rajput chivalry. akbar was so impressed with the bravery of these two warriors that he commissioned a statue of jaimal and patta riding on elephants at the gates of the agra fort. ## background our motivation lies in the cognitive and psychological mechanisms by which context affects our perception of word meaning. here, we present two of the most prominent ideas that helped define the task and dataset, and explain why previous datasets for similarity in context are not well suited to test them. ## task description our dataset is based on pairs of words from simlex-999 @xcite . each instance is a naturallyoccurring context, taken from wikipedia, in which both words in the pair appear, labelled with a similarity score given by human annotators. for each pair, the dataset contains two different contexts (see section 4 for more detail on dataset and choice of contexts). we proposed two different subtasks: first, to predict the change in similarity score between the two different contexts for each pair; second, to predict the similarity scores themselves. these are related but independent tasks that use the same input data, but each subtask had its own phases and leaderboards. submissions for each subtask were independent and participants were able to use different models for each subtasks and each language. the tasks were unsupervised, and so no training data was released; however, we released a small practice kit which contained a practice dataset, a script to generate the baseline and evaluation scripts so participants could easily reproduce results, and understand how the dataset looked and how the task was evaluated. ## dataset cosimlex @xcite ) is based on pairs of words from simlex-999 @xcite ; the reliability and common use of simlex makes it a good starting point and allows comparison of judgements and model outputs to the context-independent case. for croatian and finnish we use existing translations of simlex-999 @xcite @xcite . in the case of slovene, we have produced our own new translation, foot_0 following @xcite 's methodology for croatian. the dataset consists of 340 pairs in english, 112 in croatian, 111 in slovene and 24 in finnish. each pair is rated within two different contexts, giving a total of 1174 scores of contextual similarity. this poses a difficult task: to find suitable, organically occurring contexts; this task is even more challenging for languages with less resources, and as a result the selection of pairs is different for each language. each line of cosimlex is made of a pair of words selected from simlex-999; two different contexts extracted from wikipedia in which these two words appear; two scores of similarity, each one related to one of the contexts, calculated as the mean of annotator ratings for that context; two scores of standard deviation; the p-value given by applying the mann-whitney u test to the two score distributions; and the four inflected forms of the words exactly as they appear in the contexts (including case; note that in the morphologically rich languages, many inflections are possible). to the best of our knowledge, this is the first reasonably sized dataset in which differences in contextual similarity between two words are supported with a test of statistical significance. figure 1 shows an example from the english dataset. ## evaluation metrics the first subtask looked at the change in similarity between the two contexts, therefore it was important to preserve the difference between positive and negative values since it reflected in which of the two context the system believed the two words to be more or less similar. consequently the most appropriate metric was uncentered pearson correlation which looks at the deviation from zero instead of the mean. for the second subtask, which looked at the more traditional absolute value of similarity in context, we followed @xcite and used the harmonic mean of the pearson and the spearman correlations between the system's results and the average of the human annotations. ## baselines our task studies contextual effects in four different languages, which made multiligual bert the perfect candidate for our baseline. released shortly after the original bert model @xcite , it employs its same architecture while being trained in more than 100 different languages, our four languages between them. the original model introduced an innovative masking strategy that for the first time allowed for a bidirectional transformer language model. bert models are renowned for their ability to capture contextual effects, ability which is often blamed for an important part of their performance improvements. for the baseline of our task we used the uncased version of the model, and as a common strategy we used the contents of the last layer to form our embeddings. bert creates sub-word tokens for the out of vocabulary words, in those cases our strategy was simply averaging the sub-word vectors to form a word embeddings. additionally, the results achieved by elmo are added to tables 2 and 3 as a reference. this model precedes bert and was one of the first to produce contextualised embeddings @xcite , in this case using a bidirectional lstm. the original elmo dataset was only trained in english, however we used elmo models recently trained in croatian, slovene and finnish (ulčar and robnik-šikonja, 2020). ## participants & results the task received a total of 14 submissions for the first subtask and 15 submissions for the second. from those, 11 teams submitted system description papers for review. in order to be considered for the official rankings we asked participants to fill a form with some basic information about their systems. justmasters is not part of the official ranking since they were able to optimise their system with more than the competition's limit of 9 submissions. related to these systems, the submission by mineriaunam (gomez-adorno et al., 2020) won the english subtask 2. they proposed a system in which they calculated k-means inspired centroids from the words in the context and used them to modify the original simlex-999 non contextualised similarity scores. the approach, even if very successful, seems to rely on having out of context human annotations, perhaps not realistic in the general case. the fact that the system did very poorly in subtask 1, which asked to predict change, seems to indicate much of the success is coming from the human annotations. a related strategy could perhaps be used with embeddings or computed predictions instead of human scores. the next group focused on testing a variety of models and parameters. brums (hettiarachchi and ranasinghe, 2020) worked with elmo, bert, flair @xcite , transformer-xl @xcite and xlnet @xcite . their final submission made use of stacked embeddings proposed by @xcite . they won the finnish subtask 2, ended second in the two slovene ones and performed very well in the two english ones. the hitachi team @xcite looked at bert and xml-r. their main insight was that for every language, the layers from the center to the end where always the best performing ones, however while bert performed best in the last layer, xlm-r did in the center one, suggesting their inner structure is organised differently. they won the slovene subtask 1, finished second in the two croatian subtasks and performed competitively in the english ones. to conclude with this group justmasters (al-khdour et al., 2020) tested several models, parameters and their own strategy to combine models. they achieved very good performance, especially in the english subtask 2. however, in order to optimise their system, they made many more submissions than allowed in the competition; we therefore leave them out of the official ranking. with a more multilingual approach, babelencoding (costella pessutto et al., 2020) proposed a solution in which they translated the contexts and target words to many languages and then used a weighted combination of monolingual pretrained non contextualised embeddings and bert embeddings. their idea is that the translation not only brings new resources but the process itself can produce useful information, for example to disambiguate. the approach works very well for the less resourced languages, being clearly the best system in that category, in both subtask 1 and 2. their system won subtask 1 and 2 for croatian (by a healthy margin) and 2 for slovene, ending third in the slovene subtask 1 and third and second in the two finnish ones. the multisem team (soler and apidianaki, 2020) collected 5 different datasets in order to fine-tune their bert models, most of them automatically generated from previous datasets to increase contextual influence. as an example, ukwac-subs was created by substituting target words by either: a correct substitute; a word that could be the right substitute in other circumstances but it is not in this context; or a random word. the datasets included wic, which when used to fine tune the model resulted in the best performance for subtask1, giving them a third place. the approach works very well, giving a very consistent performance in all categories, and significantly improving the non fine-tuned model from a ρ=0.715 and 0.661 per subtask, to a ρ=0.760 and 0.718 respectively. ferryman's focus @xcite was clearly the english subtask 1, which they won with a modification of bert in which they fed the tf-idf score of the words to the model, thus incorporating information about the general importance of words. the system does very well at predicting the change between contexts, but surprisingly poorly at predicting similarity itself, ending last in the english subtask 2 and second from the last in croatian and slovene. the starting point of citiusnlp (gamallo, 2020) was the idea that, even if bert seems to be able to encode syntactic structure, it doesn't seem to make use of it. they created a linguistically motivated system that relied in dependency to create predictions. however, its performance was considerably worse than bert's and their actual submissions are based on a standard bert model. finally, the will_go team @xcite looked at different ways to measure similarity between embeddings, mixing euclidean distance with the most common cosine similarity and several others not described in their paper. the combination works well, they achieved a second place in the english subtask 1 and won the finnish subtask 1. ## conclusion we resented the semeval-2020 task on graded word similarity in context and introduced our new dataset cosimlex. we provided the motivation behind their design choices and described the annotation process. the task received a good number of submissions and system description papers (15 and 11 respectively). we hope both the task and the dataset will be useful for researchers looking into how state-of-the-art systems capture context, and help promote the use of psychologically and cognitively inspired ideas in our field. some of the interesting highlights were good performance of wordnet-based sense embeddings, the improvements achieved in less-resourced languages by simply translating the input, how the explicit feeding of an "old-fashioned" feature like tf-idf improved a very modern system's performance, and the power of well designed, automatically created datasets for fine-tuning. additional and more detailed analyses of the dataset and task results will follow as part of future work. areas to be investigated include the impact of different similarity ranges and degrees of polysemy, and more detailed qualitative analysis of the differences in annotation and between systems.