The Role of the W ord N et Relations in the Knowledge-based Word Sense Disambiguation Task (2016)

Abstract: In this paper we present an analysis of different semantic relations extracted from WordNet, Extended WordNet and SemCor, with respect to their role in the task of knowledge-based word sense disambiguation. The experiments use the same algorithm and the same test sets, but different variants of the knowledge graph. The results show that different sets of relations have different impact on the results: positive or negative. The beneficial ones are discussed with respect to the combination of relations and with respect to the test set. The inclusion of inference has only a modest impact on accuracy, while the addition of syntactic relations produces stable improvement over the baselines.

## introduction knowledge-based methods for word sense disambiguation (wsd) are attractive to the nlp community because they do not require manually annotated corpora. on the other hand, these methods are not considered completely unsupervised, because they do need information about senses of words in texts, and about the relations that hold between them, represented in the form of a directed or undirected graph, called knowledge graph @xcite . the most frequently used knowledge graph is based on wordnet (wn) @xcite or extended wordnet (xwn) @xcite , where synsets constitute the vertices of the graph and relations between synsets are represented as edges within it. @xcite provided evidence that the addition of linguistically motivated semantic relations to the kg improves the performance of knowledgebased wsd (kwsd). in the current work we perform an analysis of the various semantic relations in wn and xwn knowledge graphs. the analysis is performed via experiments with different subgraphs that include only some of the semantic relations in wn and xwn. some of the relation types allow for inference to be applied over them. thus, inferred semantic relations have been included in some of kgs as well. the experiments were performed on the manually annotated semcor corpus @xcite . in order to test the semantic relations extracted from the syntactically annotated corpus, the same was divided into four parts. we used three of the divisions for the extraction of new relations and one part for testing. the structure of the papers is as follows: the next section discusses related work on the topic. section 3 describes the experimental setup. section 4 focuses on the experiments with the semantic relations in wordnet. section 5 presents the experiments with the semantic relations in extended wordnet. section 6 gives an overview of the experiments with syntactic relations. section 7 concludes the paper. ## related work knowledge-based systems for wsd have proven to be a good alternative to supervised systems, which require large amounts of manually annotated training data. in contrast, knowledge-based systems require only a knowledge base and no additional corpus-dependent information. an especially popular knowledge-based disambiguation approach has been the use of successful graphbased algorithms known under the name of "random walk on graph" @xcite . most methods exploit variants of the pagerank algorithm @xcite . @xcite apply a variant of the algorithm to word sense disambiguation by translating wordnet into a knowledge graph in which the synsets are represented as vertices and the relations between them are represented as edges between the ver-tices. calculating the pagerank vector pr is accomplished through solving the equation: the words in the text that are to be disambiguated are inserted as nodes in the knowledge graph and are connected to their potential senses via directed edges (by default, a context window of at least 20 words is used). these newly introduced nodes serve to inject initial probability mass (via the v vector) and thus to make their associated sense nodes especially relevant in the knowledge graph. applying the personalized pagerank algorithm iteratively over the graph determines the most appropriate sense for each ambiguous word. the method has been boosted by the addition of new relations and by developing variations and optimizations of the algorithm. it has also been applied to the task of ned @xcite . the success of kwsd approaches apparently depends on the quality of the knowledge graph -whether the knowledge represented in terms of nodes and relations (edges/links) between them is sufficient for the algorithm to pick the correct senses of ambiguous words. several extensions of the knowledge graph, constructed on the basis of wordnet, have been proposed and implemented. in @xcite , semantic and syntactic relations from the sense annotated bultreebank have been extracted and the algorithm has been applied to bulgarian data. in order to do that, the treebank was first annotated with synsets from the bultreebank wordnet foot_0 , aligned to the princeton wordnet. the word forms annotated with senses at this point are 69,333, consisting of nouns and verbs. out of these, 12,792 sense-tagged word forms have been used for testing, and the rest have been used for relation extraction. the wordnet ontological relations that have been used are 252,392, and the relations derived from the synset glosses are 419,387. additionally, the following relations have been extracted: inferred hypernymy relations; syntactic relations from the gold corpus; extended syntactic relations; domain relations from wordnet. thus, 590,272 new relations have been added. the newly added relations introduce syntagmatic information into the graph, which was originally constructed out of paradigmatic relations. the results from the experiments with paradigmatic relations alone (done on the the whole corpus) show highest accuracy (0.551) for the combination of: wordnet relations + relations from the glosses + inferred hypernymy relations + domain relations of the kind synset-tosynset + domain hierarchy relations. the results from the experiments with mixed -paradigmatic and syntagmatic -relations (done on a test portion of one fourth of the corpus) show highest accuracy (0.656) for the combination of: wordnet relations + relations from xwn + inferred hypernymy relations + dependency relations from the golden corpus + extended dependency relations starting from one level up + domain relations of the kind synsetto-synset + domain hierarchy relations. @xcite present work on wsd for polish using the polish wordnet, extended with relations between semantically similar words. the authors use the measure of semantic relatedness which assigns a numerical value to pairs of words. this numerical value reflects the degree of closeness between two words. for each word w i , a list of most closely related words w j is constructed (length of the list is 20). then the synsets that contain w i and synsets containing some of w j are connected with new links. the evaluation, based on the extended knowledge graph, shows improvement on the sentence level. ## experimental set-up the experiments presented here were carried out with the ukb foot_2 tool, which provides graph-based methods for word sense disambiguation and measuring lexical similarity. the tool uses the personalized pagerank algorithm, described in @xcite . it builds a knowledge graph over a set of relations that can be induced from different types of resources, such as wordnet or dbpedia; then it selects a context window of open class words and runs the algorithm over the graph. there is an additional module called naf ukb foot_3 that can be used to run ukb with input in the naf format foot_4 and to obtain output structured in the same way, only with added word sense information. for compatibility reasons, naf ukb was used to perform the experiments reported here; the input naf document contains in its "term" nodes lemma and pos information, which is necessary for the running of ukb. we have used the ukb default settings, i.e. a context window of 20 words that are to be disambiguated together, 30 iterations of the personalized pagerank algorithm. the ukb tool requires two resource files to process the input file. one of the resources is a dictionary file with all lemmas that can be possibly linked to a sense identifier. in our case, wordnetderived relations were used as our knowledge base; consequently, the sense identifiers are word-net ids. for instance, a dictionary line compiled from wordnet synsets looks like this: predicate 06316813-n:0 06316626-n:0 01017222-v:0 01017001-v:0 00931232-v:0 it comprises of a lemma followed by the sense identifiers it can be associated with. each id consists of eight digits followed by a hyphen and a label referring to the pos category of the word. finally, a number following a colon indicates the frequency of the word sense, calculated on the basis of a tagged corpus. when a lemma from the dictionary has occurred in the analysis of the input text, the tool assigns all associated word senses to the word form in the context and attempts to disambiguate its meaning among them. the bulgarian dictionary comprises of all the lemmas of words annotated with wordnet senses in the btb. it has 8,491 lemmas mapped to 6,965 unique word senses. the second resource file required for running the tool is the set of relations that is used to construct the knowledge graph over which personalized pagerank is run. the distribution of the tool provides data (dictionary and relation files) for wordnet 1.7 and 3.0. since the btb has been annotated with word senses from wordnet 3.0, the resource files for version 3.0 have been used in our experiments. the distribution of ukb comes with a file containing the standard lexical relations defined in wordnet, such as hypernymy, meronymy, etc., as well as with a file containing relations derived on the basis of common words found in the synset glosses, which have been manually disambiguated. the format of the relations in the knowledge graph is as follows: all the experiments use the same algorithm and the same test data. only the knowledge graph differs in the different cases, as it is generated out of various sets of relations. the experiments, reported in some considerations are in order. it is apparent that the results for the english corpus increase monotonically, while for the bulgarian one they are non-monotonic. also, the combined wordnet and gloss-derived relations increase the semcor results a lot more than the btb ones. this probably reflects the fact that these are, after all, glosses in english and they capture better meanings encoded in the english corpus. these classes differ in the type of semantic relations they represent, the number of relations in each class, the parts-of-speech of the words in the synsets that are connected by the relation. obviously, isolated vertices do not play a role in the disambiguation process. thus, if we exploit only relations between nouns, we cannot expect that the system could select appropriate senses for other parts-of-speech. nevertheless, we performed some experiments with only some of the relations in order to have a basis for comparison with larger combinations. as a basic relation we consider the superordinate-subordinate relation (hypernymy), because it provides relations between the biggest groups of synsets: nouns and verbs. thus, we assume that this set of relations always has to be used in the knowledge graph. in the general conclusion from these experiments is that the addition of relations to the knowledge graph does not contribute monotonically to the accuracy of the kwsd. it shows that some of the relations in the original graph lower the accuracy. in the next sections we report only experiments performed over semcor corpus for brevity. ## experiments with semantic relations in extended wordnet the extended wordnet @xcite is constructed on the basis of analyses of the glosses of the synsets. during this analysis, the open class words were annotated with word the first division of the relations in wng into groups is on the basis of the parts of speech of the main synset. the four sets are: wng-a (first synset is for adjectives), wng-n (first synset is for nouns), wng-r (first synset is for adverbs), and wng-v (first synset is for verbs). ## syntax-based relations as was mentioned above, in our experiments we have also used semantic relations from a syntactically annotated corpus. to achieve this, we parsed semcor with a dependency parser included in ixa pipeline. then we divided the corpus in a proportion one-to-three: first part comprises of 49 documents (from br-a01 to br-f44) and it was used as a test set in the experiments reported here. the rest of the documents formed the training set from which the new relations were extracted. first, we defined patterns of dependency relations. for example, we used patterns like the following: s 1 subjs 2 , which defines a relation between a noun synset s 1 and a verb synset s 2 ; s 1 mods 2 , which defines a relation between an adjective synset s 1 and a noun synset s 2 ; s 1 modxpobjs 2 , which defines a relation between a noun synset s 1 and a noun synset s 2 ; etc. we extracted the following sets of relations: sc-aa, sc-an, sc-av, sc-nn, sc-nv, sc-ra, sc-rn, sc-rr, sc-rv, sc-vn, sc-vv, where the suffixes -aa, an, av, etc. -denote the parts of speech of the related synsets. the results from the experiments performed are presented in we also preformed inference over these sets of relations using hypernymy and hyponymy hierarchies for nouns and verbs. the best result was achieved for the combination wng, sc-aa, sc-an, sc-av, sc-nv, sc-ra, sc-rr, sc-rv, sc-vn, sc-vv, wn-hypinfer, wn-antinfer. its accuracy is 60.42. this result is 1.5 % higher than the baseline for the original knowledge graph. this improvement is statistically significant. ## conclusion in this paper we have evaluated the performance of different relations encoded in the knowledge graph, for the purposes of the knowledge-based word sense disambiguation. each of the sets of relations reflects an important linguistic piece of knowledge. thus, each of them is important for the description of languages. however, from the point of view of knowledge-based wsd each of these relations, as well as their various combinations, seem to have a different impact on the performance of the task. the results from the experiments show that the addition of whole sets of relations might have a positive or a negative effect. in our view, at least two factors are of importance: (i) the number of relations assigned to each synset. following zipf's law, we can conclude that the distribution of relations per synset is very uneven. for many synsets there is not sufficient information present in the context, in order for a good decision to be taken. for many ambiguous words the context provides no information for disambiguation, and the decision is taken arbitrarily. (ii) the second factor is that the inference rules applied to the explicit relations do not produce the expected improvement. this might be due to the fact that wordnet is not the right place to store the inference information. our expectations about the positive influence of inference are not always realized in practice. for intance, we expected to get relations between events and their participants from the derivational relations, but this was often not the case. if we take the verb "to kiss" and the derived noun "kisser", we would expect that "kisser" is a more general synset than the synsets for any specific kisser. but the synset for "kisser" had no single hyponym in wordnet. the gloss is someone who kisses and it determines the connection from "kisser" to "someone" who is the most general agent of the verb "to kiss". the connection is stated in xwn via the gloss of the noun "kisser". but for this configuration of relations in the original graph there is no inference rule defined. it seems that the systemic and monotonic knowledge that is needed for wsd and other nlp tasks is not always considered interesting enough to encode in various lexical resources. our future goals are the following: (i) the application of more complicated inference rules; (ii) the modification of relations per synset in order to ensure enough disambiguation relations. we expect these modifications of the relations to be performed via machine learning techniques over the contexts of the words in large corpora. the number of synsets in the knowledge graph is 136,334, thus the possible links between them are in total 18,586,823,222. at the same time, the number of the actual links in the biggest graphs used in practice, is less than 5 million, which is only 0.027 % of all possible combinations. probably we need much more than 0.027 % of the links in order to capture all the available, and also all the necessary, knowledge for wsd. however, in such cases a faster algorithm must be employed.