Thesaurus Verification Based on Distributional Similarities (2019)

Abstract: In this paper we consider an approach to verification of large lexical-semantic resources as WordNet. The method of verification procedure is based on the analysis of discrepancies of corpus-based and thesaurus-based word similarities. We calculated such word similarities on the basis of a Russian news collection and Russian wordnet (RuWordNet). We applied the procedure to more than 30 thousand words and found some serious errors in word sense description, including incorrect or absent relations or missed main senses of ambiguous words.

## introduction large lexical-semantic resources such as princeton wordnet @xcite and wordnets created for other languages @xcite are important instruments for natural language processing. developing and maintaining such resources requires special efforts, because it is difficult to find errors or gaps in structures consisting of thousands lexical units and relations between them. in previous works, various methods on lexical enrichment of thesauri have been studied @xcite . but another issue was not practically discussed: how to find mistakes in existing thesaurus descriptions: incorrect relations or missed significant senses of ambiguous words, which were not included accidentally or appeared recently. in fact, it is much more difficult to reveal missed and novel senses or wrong relations, if compared to novel words @xcite . so it is known that such missed senses are often found during semantic annotation of a corpus and this is an additional problem for such annotation @xcite . in this paper, we consider an approach how to use embedding models to reveal problems in a thesaurus. previously, distributional and embedding methods were evaluated in comparison with manual data @xcite . but we can use them in the opposite way: to utilize embedding-based similarities and try to detect some problems in a thesaurus. we study such similarities for more than 30 thousand words presented in russian wordnet ruwordnet @xcite foot_0 . the structure of the paper is as follows. section 2 is devoted to related work. in section 3 we briefly present ruwordnet. section 4 describes the procedure of calculating two types of word similarities based on thesaurus and a corpus. in section 5 we analyze discrepancies between thesaurus-based and corpus-based word similarities, which can appear because of different reasons. in section 6 we study groupings of distributionally similar words to an initial word using the thesaurus. ## related work in @xcite , the task of finding unattested senses in a dictionary is studied. at first, they apply the method of word sense induction based on lda topic modeling. each extracted sense is represented to top-n words in the constructed topics. to compute the similarity between a sense and a topic, the words in the definition are converted into the probability distribution. then two probability distributions (gloss-based and topic-based) are compared using the jensen-shannon divergence. it was found that the proposed novelty measure could identify target lemmas with high-and medium-frequency novel senses. but the authors evaluated their method using word sense definitions in the macmillan dictionary and did not check the quality of relations presented in a thesaurus. a series of works was devoted to studies of semantic changes in word senses @xcite @xcite . @xcite study semantic change of words using google ngram corpus. they compared frequencies and distributional models based on word bigrams in 60s and 90s. they found that significant growth in frequency often reveals the appearance of a novel sense. also it was found that sometimes the senses of words do not change but the context of their use changed significantly. for example, the context of word parent considerably change in 90s because of the most frequent collocation single parent family. in @xcite , the authors study the detection of word sense changes by analyzing digitized books archives. they constructed networks based on a distributional thesaurus over eight different time windows, clustered these networks and compared these clusters to identify the emergence of novel senses. the performance of the method has been evaluated manually as well as by comparison with wordnet and a list of slang words. but mitra et al. did not check if wordnet misses some senses. the task of revising and verifying of resources is important for developers of wordnet-like resources. some ontological tools have been proposed to check consistency of relations in wordnet @xcite . some authors report about revision of mistakes and inconsistencies in their wordnets in the process of linking the wordnet and english wordnet @xcite . @xcite consider a crowdsourcing tool allowing a user of czech wordnet to report errors. users may propose an update of any data value. these suggestions can be approved or rejected by editors. also visualization tools can help to find problems in wordnets @xcite . @xcite proposed to use embedding-based word similarities to find possible mistakes or inconsistencies in a wordnet-like thesaurus. in the current paper we provide some additional details for the @xcite study. ## ruwordnet ruwordnet was created on the basis of another russian thesaurus ruthes in 2016, which was developed as a tool for natural language processing during more than 20 years @xcite . currently, the published version of ruwordnet includes 110 thousand russian words and expressions. the important feature of ruwordnet (and its source ruthes), which is essential for this study, is that a current news collection is used as a reference collection for maintenance of ruwordnet. periodically, a new corpus (of last year news articles) is collected, single words and phrases absent in the current version of the thesaurus are extracted and analyzed for inclusion to the thesaurus @xcite . the monitoring of news flow is important because news articles concern many topics discussed in the current society, mention new terms and phenomena recently appeared. the current version of ruwordnet comprises the following types of relations: hyponymhypernym, antonyms, domain relations for all parts of speech (nouns, verbs, and adjectives); part-whole relations for nouns; cause and entailment relations for verbs. synsets of different parts of speech are connected with relations of pos-synonymy. for single words with the same roots, derivational relations are described. for phrases included in ruwordnet, relations to component synsets are given. ## comparison of distributional and thesaurus similarities to compare distributional and thesaurus similarities for russian according to ruwordnet, we used a collection of 1 million news articles as a reference collection. the collection was lemmatized. for our study, we took thesaurus words with frequency more than 100 in the corpus. we obtained 32,596 words (nouns, adjectives, and verbs). now we should determine what thesaurus relations or paths are taken to determine semantically similar entries. in the current study, we consider the following entries as semantically related to the initial thesaurus entry:  its synonyms,  all the entries located in the 3-relation paths, consisting of hyponym-hypernyms relations or/and part-whole relations between synsets from the initial entry; in such a way, for each word, we collected the thesaurus-based "bag" of similar words (tbag). then we calculated embeddings according to word2vec model with the context window of 3 words, planning to study paradigmatic relations (synonyms, hypernyms, hyponyms, cohyponyms). using this model, we extracted twenty the most similar words w i to the initial word w 0 . each w i should also be from the thesaurus. in such a way, we obtained the distributional (word2vec) "bag" of similar words for w 0 (dbag). now we can calculate the intersection between tbag and dbag and sum up the similarities in the intersection. figure 1 shows the distribution of words according to the similarity score of the tbag-dbag intersection. the axis x denotes the total similarity in the tbag-dbag intersection: it can achieve more than 17 for some words, denoting high correspondence between corpus-based and thesaurus-based similarities. relative adjectives corresponding to geographical names have the highest similarity values in the tbag-dbag intersection, for example, samarskii (related to samara city), vologodskii (related to vologda city), etc. also nouns denoting cities, citizens, nationalities, nations have very high similarity value in the tbag-dbag intersection. figure 1 . distribution of numbers of thesaurus words according to total similarity in tbag-dbaf intersection among verbs, verbs of thinking, movement (to drive  to fly), informing (to say  to inform  to warn  to assert), value changing (to decrease  to increase), belonging to large semantic fields, have the highest similarity values (more than 13). for example, according to the word2vec model, word сказать (to say) is most similar to such words as: подчеркнуть (to stress) 0.815, заявить (to announce) 0. 81, добавить (to add) 0.80, заметить (to notice) 0.79 .. and all these words are in tbag of this word in ruwordnet on the other hand, the rise of the curve in low similarity values demonstrates the segment of problematic words. ## analyzing discrepancies between distributional and thesaurus similarities we are interested in cases when the tbag-dbag intersection is absent or contains only 1 word with small word2vec similarity (less than the threshold (0.5)). we consider such a difference in the similarity bags as a problem, which should be explained. for example, троянец (troyanets) is described in the thesaurus as a citizen of ancient troya with the corresponding relations. but in the current texts, this word means a kind of malicious software (troyan horse program), this sense of the word was absent in the thesaurus. we can see that dbag of word троянец contains: вредоносный (malicious) 0.76, программа (program) 0.73, троянский (trojan) 0.71, ...вирус (virus) 0.61,... this means that the dbag and tbag are completely different, dbag of word троянец does not contain anything related to computers and software. we obtained 2343 such problematic "words". it can be seen that verbs have a very low share in this group of problematic words. it can be explained that in russian, most verbs have two aspect forms (perfective and imperfective) and also frequently have sense-related reflexive verbs. all these verb variants (perfective, imperfective, reflexive) are presented as different entries in ruwordnet. therefore, in most cases altogether they should easily overcome the established threshold of discrepancies. in the same time, if some verbs are found in the list of problematic words, they have real problems of their description in the thesaurus. to classify the causes of discrepancies, we ordered the list of problematic words in decreasing similarity of their first most similar word from the thesaurus, that is in the beginning words with the most discrepancies are gathered (further, problem list). in the subsections, we consider specific reasons, which can explain discrepancies between thesaurus and corpus-based similarities. ## searching for regularities in dbags we supposed that we can group words in the corpus-based set of similar words (dbag) of problematic words using synonyms and part-of-speech synonyms of ruwordnet. in such a way we can find more clear indications to some missed relations or novel senses. we have gathered synonyms, summed up their similarity scores to the target word, and again reordered list according to the descending order of the maximum similarity in dbag. for example, we obtained for word рассекать (to cut in the thesaurus sense) the maximum similarity score 3.58 with the following group of words: мчаться, промчаться, пронестись, нестись, носиться (rush, race, hasten). and this is the clear indication of the novel sense of this word absent in the thesaurus. at the same time we obtained for word длинноногий (long-legged) the following most similar group белокурый светловолосый блондинистый (blond, blonde, light-haired). there is no semantic similarity between words длинноногий (long-legged) and светловолосый (light-haired) but there frequent co-occurrence and occurrence with the same nouns (девушка, красавица, красотка -girl, beauty) generate such similarity values. it is also evident, that word кроссворд (crossword) is distributionally similar to group разгадывание, разгадывать, отгадывание (guess, guessing, solve) (score 1.51) only because of their frequent co-occurrence. from this experiment, we can conclude that trying to extract some novel senses or missed relations on the basis of corpus-based embeddings, it is important to account for the diversity of contexts and co-occurrence of words predicted to be related. low diversity of frequent contexts and significant co-occurrence can lead to erroneous conclusion on word semantic similarity. ## conclusion in this paper we discuss the usefulness of applying a checking procedure to existing thesauri. the procedure is based on the analysis of discrepancies between corpus-based and thesaurus-based word similarities. we applied the procedure to more than 30 thousand words of russian wordnet ruwordnet, classified sources of differences between word similarities and found several dozens of serious errors in word sense description including too general relations, missed relations or unattested main senses of ambiguous words. it is impossible to find such diverse problems in short time without automatic support. we highly recommend to use this procedure for checking wordnets  it is possible to find a lot of unexpected knowledge about the language and the thesaurus. in future, we plan to develop an automatic procedure of finding thesaurus regularities in dbag of problematic words, which can make more evident what kind of relations or senses are missed in the thesaurus.