Comparing Similarity of Words Based on Psychosemantic Experiment and R u W ord N et (2021)

Abstract: In the paper we compare the structure of the Russian language thesaurus RuWordNet with the data of a psychosemantic experiment to identify semantically close words. The aim of the study is to find out to what extent the structure of RuWordNet corresponds to the intuitive ideas of native speakers about the semantic proximity of words. The respondents were asked to list synonyms to a given word. As a result of the experiment, we found that the respondents mainly mentioned not only synonyms but words that are in paradigmatic relations with the stimuli. The words of the mental sphere were chosen for the experiment. In 95% of cases, the words characterized in the experiment as semantically close were also close according to the thesaurus. In other cases, additions to the thesaurus were proposed.

## introduction semantic proximity of words is an important parameter required in various tasks of natural language processing. it can be estimated in different ways: by corpus, using distributional methods @xcite , expert assessments; from psychosemantic experiments; using thesauri such as wordnet @xcite . the general concept of semantic similarity can be subdivided to paradigmatic (taxonomical) similarity and semantic associations @xcite @xcite . paradigmatic similarity can be defined in terms of shared superordinate category or shared semantic features. se-mantic associations correspond to co-occurrence (syntagmatic relations) in texts. to study automatic methods of word similarity calculation, specialized datasets are created. some researchers try to create datasets distinguishing different subtypes of semantic similarity of words, which requires additional efforts and guidelines. @xcite subdivided the existing semantic word dataset wordsim353 @xcite into two subsets: wordsim353-similarity and wordsim353relatedness datasets. simlex-999 guidelines @xcite aim to distinguish word pairs in taxonomical semantic similarity relation (synonyms, hypernyms, hy ponyms) from remaining types of relations (antonymy, co-hyponyms). the authors of win353 dataset @xcite ) ask respondents about word similarity based on word interchangeability in sentences. we can also try to use human scores of word semantic similarity to assess the quality of descriptions in electronic lexical-semantic resources (thesauri). such resources are built on the basis of synsetssets of synonymslinked by semantic relations such as hyponymy, hypernymy, antonymy, and some others. the automatic use of thesauri requires high quality descriptions of semantic senses and semantic relations between them. in this paper, we compare the results of a survey of respondents and the similarity of words according to the ruwordnet thesaurus @xcite for the russian language. currently, the published ruwordnet version comprises about 110 thousand russian words and expressions. a new version of ruwordnet is being prepared and ruwordnet data are tested from different points of view. in the psychosemantic experiment the respondents were asked to list synonyms for stimu-li words without any guidelines. we found that their answers mainly contain paradigmatically similar words, practically without words related via any other similarity relationships, which makes it possible to check the taxonomic structure of the thesaurus. the paper is structured as follows. section 2 provides information on related work. section 3 describes a psychosemantic experiment to determine the semantic proximity of words. section 4 analyzes the data obtained. section 5 discusses the results of the experiment. ## related work the paper concerns two directions of studies: revision and updating existing lexical-semantic resources for natural language processing (thesauri) and studies on relation types exploited by native speakers in word association experiments. ## experiment setting in the current study we present the results of a psychosemantic experiment, carried out in accordance with the methodology described in @xcite . the experiment reveals semantically close words (synonyms) as seen by native speakers. in @xcite only synonyms from the ruwordnet synsets were considered, in this work all semantic relations are involved. the experiment is as follows. the respondents (russian native speakers) receive a number of words, and they have to list synonyms for these words in a limited time. the respondents are students (18-23 years old, 200 people) of kazan federal university (kazan, russia). about half of the students are philologists, the second half are non-philological students. the definition of a synonym is not explained to the respondents; we rely on intuitive understanding of synonyms by native speakers. for the experiment, words related to the mental sphere are selected. this semantic area is the most difficult for clear differentiation of synonym sets and their semantic relations. the results of philologists and nonphilologists differ insignificantly. however, it is worth noting that synonyms for word мечта (mechtadream as imaginative thoughts) listed by philologists and non-philologists have interesting distinctions. so, for philologists, the word фантазия (fantasiafantasy) is in 3rd place, and for non-philologists, the word стремление (stremleniyeaspiration) is in the 3rd position. conversely, for philologists, stremlenie (aspira-tion) is listed in the 5 th place, and for nonphilologists, fantasia (fantasy) is in the 4 th position. it seems that the figurative thinking of philologists, the reading and study of fiction, which form their linguistic personalities, are reflected in the results of the experiment: for them, the word mechta (dream) is associated with fantasy and dreams, that is, with something unreal, ephemeral. not-philologists are more pragmatic: the third position in their lists is occupied by the word stremlenie (aspiration), in the semantics of which the presentation of concrete results is conveyed @xcite . further we will write cyrillic russian words in latin transcription. since the respondents, naturally, did not use the criteria of synonymy, such as interchangeability in different contexts and did not have much time to complete the task, they suggested words that have something semantically in common with the given word, but not necessarily synonyms in the strict sense of the term. for example, for the word mechta (dream as imaginative thoughts), the following words were listed as synonyms in ruwordnet: gresa, mechtaniye, fantasia (fantasy). the respondents most often indicated the following words: zhelaniye (desire), tsel' (goal), fantasia (fantasy), gresa, stremleniye (aspiration), nadezhda (hope). only two of them are synonymous. the rest of the wordszhelaniye (desire), tsel' (goal), stremleniye (aspiration), nadezhda (hope)at first glance may seem like associations with the given word dream. however, this assumption is not true. in the karaulov's dictionary of russian associations @xcite , the word mechta (dream) has the following most frequent associations: goluboy (blue), zhizn' (life), moya (mine), sbylas' (come true), idiota (idiot), nesbytochnaya (unrealizable), rozovaya (pink). the words zhelaniye (desire), stremleniye (aspiration), nadezda (hope) are not mentioned as associations at all, and the word tsel' (goal) is mentioned only once in 101 responses. we can see that in fact words having syntagmatic relations with the original one are also mentioned as associations by respondents. in the current experiment, the respondents indicated words that were not in syntagmatic but in paradigmatic relations with the stimulus. rather, they can be characterized as belonging to the semantic field of the original word or as its analogues. it is worth noting that in the dictionary @xcite the words namereniye (intention) and mysl' (thought) are considered as ana-logues (near-synonyms) of the word mechta (dream) (its synonyms are not given in the dictionary). for the verb mechtat' (to dream), the synonyms, according to @xcite , are khotet' (to want), zhelat' (to desire), and the analogue is the word nadeyat'sya (to hope). thus, the words indicated by the respondents are close in meaning to the word mechta (dream). our experiment can be characterized as aimed at identifying paradigmatic associations, while the karaulov's dictionary @xcite in fact mixes paradygmatic and syntagmatic associations. ## analysis of results in this work, the associations for the words obida (offense, as a feeling caused being offended), radost' (joy), talant (talent), strast' (passion), lyubov' (love), mysl' (thought), vostorg (delight) are considered. for each stimulus word, six most frequently mentioned responses are studied. ## discussion we analyzed 40 word pairs (out of a total of 7x6 = 42 pairs, two pairs were repeated). in 38 cases (95%), word pairs listed by the respondents as synonyms are also close according to the thesaurus descriptions: 13 pairs are at a distance of 1; 12 pairs are at a distance of 2; 7 pairs are at a distance of 3; 6 pairs are at a distance of 4. the number of mentioned words located at a certain path distance in the thesaurus decreases monotonically with increasing distance. in all these cases, it turned out to be sufficient to consider only hypo-hypernymic relations. in two cases, it is necessary to make certain changes in the thesaurus to obtain smaller distance for semantically close words. these pairs of words are as follows: lyubov' (love) -nezhnost' (tenderness) and radost' (joy)ulybka (smile). in the first case, it is proposed to add a new sense of the word neznost' (tenderness) to the thesaurus and to establish the necessary additional relation of hyponymy, in the second case we suggest to add the relation of entailment. thus, most words frequently mentioned by respondents are located close to the stimulus word in ruwordnet, which indicates good consistency of the thesaurus with the intuition of native speakers. at the same time, taking into account the data of a psychosemantic experiment makes it possible to identify some problem areas in the thesaurus. let us consider whether there is a correlation between the frequency with which the response word is chosen by the respondents and the distance in the thesaurus from the stimulus word to the response word. we sort wordsreactions according to the frequency of their mention. ## conclusion thesauri are created by professionals who rely on both the theory of language and their ideas about the semantics of linguistic units. however, semantics are not described in the literature in as much detail as required by the thesaurus developers. taking this into account, it is of natural interest to compare thesaurus data with the linguistic intuition of native speakers, manifested in psychosemantic experiments. most words frequently mentioned by respondents or synonyms with the stimulus word or are located close to it in ruwordnet, which indicates good consistency of the thesaurus with the intuition of native speakers. this confirms the high quality of the ruwordnet thesaurus. the experimental results also support the choice of distance 4 as a measure of the semantic proximity of words in the thesaurus. at the same time, taking into account the experimental data made it possible to identify some problem areas in the thesaurus.