S uper S im: a test set for word similarity and relatedness in S wedish (2021)

Abstract: Language models are notoriously difficult to evaluate. We release SuperSim, a large-scale similarity and relatedness test set for Swedish built with expert human judgements. The test set is composed of 1,360 word-pairs independently judged for both relatedness and similarity by five annotators. We evaluate three different models (Word2Vec, fastText, and GloVe) trained on two separate Swedish datasets, namely the Swedish Gigaword corpus and a Swedish Wikipedia dump, to provide a baseline for future comparison. We will release the fully annotated test set, code, models, and data.

## introduction it is said that a cup and coffee are not very similar while car and train are much more so given that they share multiple similar features. instead, cup and coffee are highly related, as we typically enjoy the one in the other. of course, an immediate question that arises is whether we have words that are similar but not related? existing similarity datasets have tended to rate words for their similarity, relatedness, or a mixture of both, but not either or. however, without both kind of information, we cannot know if words are related but not similar, or similar but not related. the most common motivation for using word similarity datasets, such as simlex-999 @xcite and wordsim353 @xcite , is for use as a quality check for word embedding models. the aim of most embedding models is to capture a word's semantic relationships, such that words that are similar in meaning are placed close in the semantic space; foods with other foods, technical terms together and separated from the musical instruments, to give an example. however, the optimal performance of such a semantic space is judged by whether or not one wishes to capture similarity of words, or relatedness. it seems obvious that presenting cup as a query reformulation for coffee in information retrieval seems off, while presenting lamborghini when searching for ferrari can be completely acceptable. inversely, in places where relatedness is needed, offering a cup when one asks for a coffee is correct. while the first word similarity datasets appeared for english, in the past few years we have seen datasets for a range of different languages (see section 2). for swedish, there exists one automatically-created resource based on an association lexicon by @xcite . however, there are to date no test sets that are (1) expertly-annotated, (2) comparable to other international test sets, and (3) annotated for both relatedness and similarity. and because we cannot know which motivation lies behind creating a vector space, and because both relatedness and similarity seem equally valid, we have opted to create supersim. the supersim test set is a largerscale similarity and relatedness set for swedish, consisting of 1,301 words and 1,360 pairs rated by 5 expert annotators. the pairs are based on simlex-999 and wordsim353, and can be used to assess the performance of word embedding models, but also answer questions as to whether words are likely to be similar but not related. ## related work several works aim to provide test sets to assess the quality of word embedding models. most of them tackle english @xcite @xcite @xcite . russian, italian and german are cov-ered by @xcite who translated the pairs in wordsim353 and simlex-999, and asked crowdworkers to judge them on a 0-10 scale. the semeval-2017 task 2 on multilingual and cross-lingual semantic word similarity @xcite provides pairs in 5 languages: english, farsi, german, italian and spanish. @xcite provide 500 word pairs in turkish annotated by 12 humans for both similarity and relatedness on a scale ranging from 0 to 10, while finnish is covered in @xcite . more recently, multi-simlex @xcite provides annotations in mandarin chinese, yue chinese, welsh, english, estonian, finnish, french, hebrew, polish, russian, spanish, kiswahili, and arabic, with open guidelines and encouragement to join in with more languages. 2 for swedish, fallgren et al. ( 2016 ) harness the swedish association lexicon saldo @xcite , a large lexical-semantic resource that differs much from wordnet @xcite insofar as it organises words mainly with the 'association' relation. the authors use saldo's 'supersenses' to adapt tsvetkov et al. (2016)'s qvec-cca intrinsic evaluation measure to swedish. still on evaluating swedish language models, adewumi et al. (2020b) propose an analogy test set built on the one proposed by @xcite , and evaluate common architectures on downstream tasks. the same authors further compare these architectures on models trained on different datasets (namely the swedish gigaword corpus @xcite and the swedish wikipedia) by focusing on swedish and utilising their analogy test set @xcite . finally, for swedish, swedishglue/superlim foot_2 @xcite is currently being developed as a benchmark suite for language models in swedish, somewhat mirroring english counterparts @xcite . whether similarity test sets actually allow to capture and evaluate lexical semantics is debatable @xcite . nonetheless, they have the advantage of providing a straightforward way of optimising word embeddings (through hyper-parameter search, at the risk of overfitting), or to be used more creatively in other tasks @xcite where "quantifiable synonymy" is required. finally, task-specific evaluation (as recommended by @xcite ) is, for languages other than english, more than often nonexistent -making test sets such as the one presented in this work a good alternative. our dataset differs from previous work in the sense that it provides expert judgments for swedish for both relatedness and similarity, and hence comprises two separate sets of judgments, as done by skilled annotators. 4 a description of the procedure is available in section 3. ## model evaluation to provide a baseline for evaluation of embedding models on supersim, we trained three different models on two separate datasets. ## conclusions and future work in this paper, we presented supersim, a swedish similarity and relatedness test set made of new judgments of the translated pairs of both simlex-999 and wordsim353. all pairs have been rated by five expert annotators, independently for both similarity and relatedness. our inter-annotator agreements mimic those of the original test sets, but also indicate that similarity is an easier task to rate than relatedness, while our intra-rater agreements on 69 control pairs indicate that the annotation is reasonably consistent. to provide a baseline for model performance, we trained three different models, namely word2vec, fasttext and glove, on two separate swedish datasets. the first comprises a general purpose dataset, namely the the swedish culturomics gigaword corpus with different genres of text spanning 1950-2015. the second comprises a recent swedish wikipedia dump. on the gigaword corpus, we find that fasttext is best at capturing both relatedness and similarity while for wikipedia, glove performs the best. finally, to answer the question posed in the introduction: it is common to have words that are highly related, but not similar. to give a few examples, these are pairs with relatedness 10 and similarity 0: bil-motorväg 'carhighway,' datum-kalender 'date-calendar,' ordordbok 'word-dictionary,' skola-betyg 'schoolgrade,' and tennis-racket 'tennis-racket.' the opposite however, does not hold. only four pairs have a similarity score higher than the relatedness score, and in all cases the difference is smaller than 0.6: bli-verka 'become-seem,' rörcigarr 'pipe-cigarr,' ståltråd-sladd 'wire-cord,' tillägna sig-skaffa sig 'get-acquire.' for future work, the supersim testset can be improved both in terms of added annotations (more annotators), and with respect to more fine-grained judgements (real values in contrast to discrete ones currently used) to reduce the number of rank ties.