R u DSI : Graph-based Word Sense Induction Dataset for R ussian (2022)

Abstract: We present RuDSI, a new benchmark for word sense induction (WSI) in Russian. The dataset was created using manual annotation and semi-automatic clustering of Word Usage Graphs (WUGs). RuDSI is completely data-driven (based on texts from Russian National Corpus), with no external word senses imposed on annotators. We present and analyze RuDSI, describe our annotation workflow, show how graph clustering parameters affect the dataset, report the performance that several baseline WSI methods obtain on RuDSI and discuss possibilities for improving these scores.

## introduction word sense induction (wsi) is among the most challenging problems in computational linguistics. the difficulty lies not only in the character of the task itself but also in the lack of datasets properly designed for it. we have developed such a dataset for the russian language by means of manual annotation and clustering of the obtained senses. we dub it russian data-driven sense induction dataset (rudsi) 1 . its annotation was based on socalled word usage graphs (wugs), where word usages in context are nodes connected by edges with weights corresponding to semantic proximity @xcite . this workflow has been already used to create diachronic semantic change datasets for russian @xcite , but it is the first time it is employed for designing synchronic wsi benchmarks. graphs representing semantic relations between word usages were crucial for the creation of rudsi. communities or clusters induced from these graphs correspond to lexical senses; the number and composition of clusters for each word depends not only on human annotation, but also on the particular clustering procedure. since we provide raw annotators' judgments, other researchers can apply their preferred graph processing techniques and obtain slightly different sense assignments. the rest of the paper is organized as follows. in section 2, we talk about the wsi datasets created earlier and their limitations. in section 3, we present and analyze rudsi and describe our annotation workflow. in section 4, we show how graph clustering parameters affect the dataset. section 5 reports the performance of several baseline wsi methods. in section 6, we describe to whom and how rudsi will be useful. 2 ## related work in this section, we give a brief overview of word sense induction datasets for english developed as a part of semeval competition, take a look at russe'18 dataset and discuss the approaches towards wsi dataset creation. ## robustness of clustering in order to verify the stability of clustering algorithm we experimented with changing the default hyperparameters and analyzed the resulting data in comparison with the default sense clusters presented in rudsi. in the pipeline @xcite , there were two parameters that could affect the obtained clusters: the threshold used to rescale the annotators' judgements and the number of clustering iterations. the threshold parameter was previously described in 3.3: it affects the resulting weights on the graph edges. originally, the threshold was 2.5 causing 1 and 2 scores ('unrelated' and 'distantly related') to transform to negative values, and 3 and 4 scores ('closely related' and 'identical') to remain positive to represent the contrast between different senses and the same sense of the word. we reviewed two other options: the threshold equaled to 1.5 (so that a score of 1 became negative (-0.5) and contrasted with 2, 3 and 4 scores that were matched to 0.5, 1.5 and 2.5 respectively) and equaled to 3.5 (1, 2 and 3 scores were opposed to a score of 4; only the sentences marked us 'identical' were considered as containing the same sense of the word). the number of clustering iterations ('iters' parameter) stands for the number of passes through the same graph given that the input graph is the result of the previous iteration. each pass performs the clustering algorithm and minimizes the loss of the obtained clusters. in ## baseline wsi methods performance in this section, we show how the existing wsi methods perform on rudsi. we deliberately did not experiment with the state-of-the-art lexical substitution method @xcite . the goal is to report the results of the baseline approaches, leaving more advanced methods for future research. ## intended rudsi audience our vision is that rudsi might be of use for three different communities. ## conclusion we have presented rudsi, a novel graph-based word sense induction dataset for russian, obtained by clustering word usage graphs produced by human annotation. it includes words with different degrees of polysemy (monosemous, moderately polysemous and highly polysemous words). the sense inventories are generated in a completely data-driven way as well. importantly, depending on what graph processing workflow is used, slightly different datasets can be produced from the same raw rudsi human judgments. we report the rudsi performance for only the simplest and most basic approaches to wsi, so a possible future work would be to apply some more advanced methods to it. also we have considered only nouns, so it would be interesting to experiment with other parts of speech as well (this will require a new round of annotation). since most of the target words in rudsi have 3-5 senses, the addition of highly polysemous words may become another future improvement. in addition, it would be beneficial to extend the list of contexts for each word, however extra annotation would be required.