Simple Embedding-Based Word Sense Disambiguation (2018)

Abstract: We present a simple knowledge-based WSD method that uses word and sense embeddings to compute the similarity between the gloss of a sense and the context of the word. Our method is inspired by the Lesk algorithm as it exploits both the context of the words and the definitions of the senses. It only requires large unlabeled corpora and a sense inventory such as WordNet, and therefore does not rely on annotated data. We explore whether additional extensions to Lesk are compatible with our method. The results of our experiments show that by lexically extending the amount of words in the gloss and context, although it works well for other implementations of Lesk, harms our method. Using a lexical selection method on the context words, on the other hand, improves it. The combination of our method with lexical selection enables our method to outperform state-of the art knowledge-based systems.

## introduction the quest of automatically finding the correct meaning of a word in context, also known as word sense disambiguation (wsd), is an important topic in natural language processing (nlp). wsd systems that are based on supervised learning methods gain best results @xcite @xcite @xcite . however, they require a large amount of manually annotated data for training. also, even if such a supervised system obtains good results in a certain domain, it is not readily portable to other domains @xcite . as an alternative to supervised systems, knowledge-based systems do not require manually tagged data and have proven to be applicable to new domains @xcite ). an example of such a system is the lesk algorithm @xcite ) that exploits the idea that the overlap between the definition of a word and the definitions of the words in its context can provide information about its meaning. it only requires two types of information: a set of dictionary entries with definitions (hereafter referred to as glosses) for each possible word meaning, and the context in which the word occurs. a popular variant of the algorithm is the "simplified" @xcite , which disambiguates one word at a time by comparing each of its glosses to the context in which the word is found. this variant avoids the combinatorial explosion of word sense combinations the original version suffers from when trying to disambiguate multiple words in a text. a problem with the aforementioned method, however, is that, when a gloss is matched against the context of a word, in most cases the lexical overlap is very small. as a solution to this problem, we use a wsd-method that, instead of counting the number of words that overlap, takes embeddings as input to compute the similarity between the gloss of a sense and the context of the word. although our method works well on its own, its simplicity allows us to explore whether other extensions to the lesk algorithm that have proven to be successful can improve it further. as both the lesk algorithm and our extension rely on the definition of the words and the words that surround it, it is interesting to see whether adapting both sources of information would improve either of them. in this light, there are two possibilities: expansion or reduction. for the first option, the existing words of the context and glosses can be expanded with additional words that have similar meanings. for example, @xcite use a distributional thesaurus, that is computed from a large parsed corpus to lexically expand the context and sense information. they show that, using these expanded context and glosses, improves two variants of lesk. when reducing the amount of words in either the context or the target words' sense, methods are required that prohibit the loss of informative words. @xcite shows that a pre-selection of words in the context of the target word improves simplified lesk. in this paper we describe experiments where both methods are used in combination with out method that is based on word-and sense embeddings. ## related work in the past few years, much progress has been made on learning word embeddings from unlabeled data that represent the meanings of words as contextual feature vectors. a major advantage of these word embeddings is that they exhibit certain algebraic relations and can, therefore, be used for meaningful semantic operations such as computing word similarity @xcite , and capturing lexical relationships @xcite . a disadvantage of word embeddings is that they assign a single embedding to each word, thus ignoring the possibility that words may have more than one meaning. this problem can be addressed by associating each word with a number of sensespecific embeddings. for this, several methods have been proposed in recent work. for example, in @xcite and @xcite , a fixed number of senses is learned for each word that has multiple meanings by first clustering the contexts of each token, and subsequently relabeling each word token with the clustered sense before learning embeddings. although such sense embedding methods have demonstrated good performance, they use automatically induced senses. they are, therefore, not readily applicable for applications that rely on wordnet-based senses, such as machine translation and information retrieval and extraction systems (see @xcite for examples of such systems). recently, features based on sensespecific embeddings learned using a combination of large corpora and a sense inventory have been shown to achieve state-of-the-art results for supervised wsd @xcite @xcite . our system makes use of a combination of sense embeddings, context embeddings, and gloss embeddings. similar approaches have been proposed by @xcite and @xcite . the main difference to our approach is that they automatically induce sense embeddings and find the best sense by comparing them to context embeddings, while we add gloss embeddings for better performance. inkpen and @xcite apply gloss-and context vectors to the disambiguation of near-synonyms in dictionary entries. also @xcite use a distributional approach to representing definitions and the context of the target word. they create semantic vectors for glosses and contexts to compute similarity of the gloss and the context of a target word, while we also compute the similarity of a sense and its context directly using sense embeddings. ## lesk++ our wsd algorithm takes sentences as input and outputs a preferred sense for each polysemous word. given a sentence w 1 . . . w i of i words, we retrieve a set of word senses from the sense inventory for each word w. then, for each sense s of each word w, we consider the similarity of its lexeme (the combination of a word and one of its senses @xcite ) with the context and the similarity of the gloss with the context. for each potential sense s of word w, the cosine similarity is computed between its gloss vector g s and its context vector c w and between the context vector c w and the lexeme vector l s,w . the score of a given word w and sense s is thus defined as follows: the sense with the highest score is chosen. when no gloss is found for a given sense, only the second part of the equation is used. prior to the disambiguation itself, we sort the words by the number of senses it has, in order that the word with the fewest senses will be considered first. the idea behind this is that words that have fewer senses are easier to disambiguate @xcite . the algorithm relies on the words in the context which may themselves be ambiguous. if words in the context have been disambiguated already, this information can be used for the ambiguous words that follow. we, therefore, use the resulting sense of each word for the disambiguation of the following words starting with the "easiest" words. our method requires lexeme embeddings l s,w for each sense s. for this, we use autoextend @xcite to create additional embeddings for senses from wordnet on the basis of word embeddings. autoextend is an auto-encoder that relies on the relations present in wordnet to learn embeddings for senses and lexemes. to create these embeddings, a neural network containing lexemes and sense layers is built, while the wordnet relations are used to create links between each layer. the advantage of their method is that it is flexible: it can take any set of word embeddings and any lexical database as input and produces embeddings of senses and lexemes, without requiring any extra training data. for each word w we need a vector for the context c w , and for each sense s of word w we need a gloss vector g s . the context vector c w is defined as the mean of all the content word representations in the sentence: if a word in the context has already been disambiguated, we use the corresponding sense embedding; otherwise, we use the word embedding. for each sense s, we take its gloss as provided in wordnet. in line with @xcite , we expand this gloss with the glosses of related meanings, excluding antonyms. similar to the creation of the context vectors, the gloss vector g s is created by averaging the word embeddings of all the content words in the gloss. ## lexical expansion and lexical selection we use the method of @xcite to expand the glosses and the contexts of the target words before using our adaptation of the lesk system. 1 for each content word we retrieve the 30 most similar terms from the distributional thesaurus and add them to the context or gloss while occurrences of the target word are removed. for the selection of context words, we use the lexical chaining technique as applied in @xcite that use the idea of creating lexical chains from @xcite . lexical chains are sequences of words that are semantically related. similar to @xcite , we use the synonymy and hypernymy relations in wordnet in combination with a similarity measure (jaccard formula @xcite ), to verify whether a context word is a member of such a lexical chain. for both the target word w and each context word c in its context, we retrieve a set of sense definitions of all the synonyms and hypernyms of w according to the wordnet hierarchy. a context word is added to the context if the similarity score for the set of w and the set of c is greater than an experimental threshold. ## experiments we test our method on both dutch and english data. we build 300-dimensional word embeddings on the dutch sonar corpus @xcite using word2vec cbow @xcite , and create sense-and lexeme embeddings with autoextend. for english, we use the embeddings from rothe and schütze (2015) foot_1 . they lie within the same vector space as the pre-trained word embeddings by @xcite foot_2 , trained on part of the google news dataset, which contains about 100 billion words. this model (similar to the dutch model) contains 300-dimensional vectors for 3 million words and phrases. our sense inventory for dutch is cornetto @xcite and for english, we use wordnet 1.7.1 @xcite as this version matches the autoextend embeddings. in cornetto, 51.0% of the senses have glosses. in the princeton wordnet, almost all of them do. the dutchsemcor corpus @xcite is used for dutch evaluation and, for english, we use semcor @xcite . a random subset of 5000 manually annotated sentences from each corpus was created. additionally, we test on the senseval-2 (se-2) and senseval-3 (se-3) all-words datasets @xcite . we evaluate our method by comparing it with a random baseline and simplified lesk with expanded glosses (se-lesk) @xcite . we do not compare our system to the initial results of autoextend @xcite as they tested it in a supervised setup using sense embeddings as features. however, as is customary in wsd evaluation, we do compare our system to the most frequent sense baseline, which is notoriously difficult to beat due to the highly skewed distribution of word senses @xcite . as this baseline relies on manually annotated data, which our system aims to avoid, we consider this baseline to be semi-supervised. additionally, we compare our system to a state-of-the-art knowledge-based wsd system, ukb @xcite , that, similar to our method, does not require any manually tagged data. ukb can be used for graph-based wsd using a pre-existing knowledge base. it applies random walks, e.g. personalized pagerank, on the knowledge base graph to rank the vertices according to the context. we use ukbs personalized pagerank method word-by-word with wordnet 1.7 and extended wordnet for english, as this setup yielded the best results in @xcite . for senseval-2 (se-2) and senseval-3 we use the wsd evaluation framework of @xcite , which provides evaluation datasets and output of other knowledge-based wsd systems. from those systems we report on the extended lesk version of @xcite , (dsm) foot_3 which is most similar to our approach. the manually annotated part of dutchsemcor is balanced per sense which means that an equal number of examples for each sense is annotated. it is therefore not a reliable source for computing the most frequent sense. alternatively, similar to @xcite , we derive sense frequencies by using the automatically annotated counts in dutchsemcor foot_4 . the most frequent sense baseline for dutch is, therefore, lower as compared to the english, where the most frequent sense of a word is fully based on manual annotation. ## results ## discussion the difference in results for dutch and english can be explained by the coverage of the datasets. the cornetto coverage is about 60%, compared to princeton wordnet, with an average polysemy of 1.07 for nouns, 1.56 for verbs and 1.05 for adjectives while, for english it is 1.24 for nouns, 2.17 for verbs and 1.40 for adjectives. also, not all dutch senses have corresponding glosses while most of the english ones do. as our method relies greatly on gloss vectors, this could affect its performance. the different performance of both extensions to se-lesk and lesk++ shows that both algorithms capture different types of information and therefore require a different type of input. as se-lesk counts on the direct overlap of words, it depends highly on a larger amount of words. lesk++ on the other hand, already overcomes this problem and clearly benefits from more "quality" information in the contexts. in future work we would like to try other vector types such as @xcite that represents contexts that outperform the context representation of averaged word embeddings. also, it would be nice to see whether other knowledge-based sense embeddings, such as the ones from @xcite , could improve our results. ## conclusions we compared several extensions to the lesk algorithm with an adaption which uses sense, gloss and context embeddings to compute the similarity of word senses to the context in which the words occur. we try two different methods that could improve ours, one that further extends the information in both the context and the glosses by utilizing distributional thesauri @xcite and one that pre-selects context words using the wordnet hierarchy @xcite . although our approach is a straightforward extension to the lesk algorithm, it achieves better performance compared to lesk and a random baseline. when using a selection scheme before creating context vectors, its performance is better than our knowledge based baselines. the main advantage of our method is its simplicity which makes it fast and easy to apply to other languages. it furthermore only requires unlabeled text and the definitions of senses, and does not rely on any manually annotated data, which makes our system an attractive alternative for supervised wsd.