  Language models are notoriously difficult to evaluate. We release SuperSim, a
large-scale similarity and relatedness test set for Swedish built with expert
human judgments. The test set is composed of 1,360 word-pairs independently
judged for both relatedness and similarity by five annotators. We evaluate
three different models (Word2Vec, fastText, and GloVe) trained on two separate
Swedish datasets, namely the Swedish Gigaword corpus and a Swedish Wikipedia
dump, to provide a baseline for future comparison. We release the fully
annotated test set, code, baseline models, and data.
