Estimating senses with sets of lexically related words for P olish word sense disambiguation (2019)

Abstract: We propose a new algorithm for word sense disambiguation, exploiting data from a WordNet with many types of lexical relations, such as plWordNet for Polish. In this method, sense probabilities in context are approximated with a language model. To estimate the likelihood of a sense appearing amidst the word sequence, the token being disambiguated is substituted with words related lexically to the given sense or words appearing in its WordNet gloss. We test this approach on a set of sense-annotated Polish sentences with a number of neural language models. Our best setup achieves the accuracy score of 55.12% (72.02% when first senses are excluded), up from 51.77% of an existing PageRank-based method. While not exceeding the first (often meaning most frequent) sense baseline in the standard case, this encourages further research on combining WordNet data with neural models.

## introduction ambiguity is an inherent feature of natural languages. there is no one-to-one relation between the vocabulary of word units and the set of meanings which these words represent. although there are more and more applications in which disambiguation step is not clearly distinguished, explicit identification in which sense a particular word is used in a given context remains important in many situations. if we aim at selecting a specific sense from a given inventory like wordnet @xcite , this task is called word sense disambiguation (wsd) and was commonly addressed in one of two ways. the first one treats the task as a standard word classification problem solved using any of the supervised learning techniques. the hard part of applying this approach is obtaining satisfactorily large annotated data sets for relatively big subset of senses, even if the annotation can be partially bootstrapped in a semi-supervised manner, for example using label propagation @xcite . manual labelling of data with word senses takes time, and agreement between annotators is usually not very high. another problem is that a lot of text has to be processed to collect occurrences of several (or even more) senses of each word. this is why the second approach to wsd seems to be more common. in this type of solutions, information included directly or indirectly in lexical databases, especially wordnet, is used either to generate additional features or as the only data source (in the algorithms based on analysis of knowledge graph structure). recently, vector word representations and neural network architectures have started to be widely used. our solution combines neural models trained on a large text corpus with information extracted from the plwordnet @xcite . ## related work the problem of resolving lexical ambiguity has a long and complicated history. this task is one of the oldest problems in computational linguistics and machine translation research, but its definition and role in natural language processing (nlp) community's efforts changed over time in many ways. although solutions of one specific version of the problem -an explicit task of resolving finegrained and coarse-grained ambiguity to a fixed inventory of senses -showed, at the senseval-3 conference @xcite , consistent and respectable accuracy levels, @xcite observed that this success did not lead to better performance in real applications. they opined that wsd as a topic of study found it-self "in a strange position", and seemed to diverge from research on nlp applications, "despite several efforts to investigate and demonstrate its utility". the authors of the best solution at that time @xcite reported an accuracy score of 0.65, which was at human levels according to inter-annotator agreement. their method requires constructing a graph with all senses of words that are present in the text. a pagerank-like algorithm is applied to this graph for choosing the most salient senses, combined with the lesk algorithm @xcite and most frequent senses heuristics. although this system achieved the best result, accuracy of 0.65 was not satisfactory for industrial nlp applications. it should be noted, however, that these results were obtained on princeton wordnet, which distingishes fairly multiple senses for words, with granularity than can exceed the needs in many situations. with no direct enhancement in view, research on the wsd task was receiving waning interest, but did not cease entirely. many researchers explored different measures for graph connectivity which might be useful for the wsd task @xcite . in the semeval-2013 task 12, linked data for different languages were also used for this purpose @xcite . with the increasing popularity of distributional semantic approach, many experiments exploiting word embeddings as an additional or the only source of information were performed @xcite . while the evidence from research on the wsd task for english appears contradictory, it should be instructive to see how approaches perform on data in different languages with their unique problems and qualities. for polish, relatively little was investigated on this subject, but some results were published. leaving out very early experiments which constrained themselves to a purposebuilt set of senses for a group of selected words, we should mention @xcite who employed the graph-based method proposed by @xcite and @xcite , utilizing data from plwordnet integrated by the authors with existing sumo ontology. recently, @xcite proposed an approach where probability of senses in context is assessed by replacing the disambiguated word with unambiguous members of their synsets. this method, while obviously limited to cases where such unambiguous words can be found in the token's synsets, produced promising results when tested on data from @xcite . the general idea of estimating context probability with replacements from a wordnet is similar to the one presented in this paper, but we argue that it can be exploited more fully using lexical relations. ## test data description our test data consists of a small sample of 1000 sentences selected from the manually annotated part of the nkjp (national corpus of polish) @xcite . the sentences were chosen randomly, but we excluded transcribed speech and internet sources. we collected 24,535 tokens of 9,741 token types in total. all nouns, verbs, adjectives and adverbs were manually annotated with plwordnet 3.1 senses by appropriately trained linguists. as the annotation process is very time consuming, only a part of the data was annotated by both of them and they agreed on 83% of tokens. this is comparable to the measures of inter-annotator agreement in senseval competitions @xcite . in senseval-1, the 80% agreement was eventually achieved by allowing for discussion and revisions of ambiguities in lexical entries before final tagging. in senseval-2, the agreement on verb annotation was initially 71%, but after grouping some senses into more coarse-grained ones it rose to 82%. (both the senses and their associated words are a selection from larger sets.) these "neighbours" could be substituted for the original word to estimate the likelihood of the sense occuring here. sense (s * ) with the highest probability given the form and context (c) of the word: however, since there is no clear way to obtain p (s) directly, we approximate it with some set r s of word forms related to the sense in question. one way of combining the evidence from members of r s is to average their probabilities in the context: we also test the variant where the highest probability estimated for a related word is taken to represent the whole sense: once r is an explicitly designated word form or lemma, a language model capable of predicting probability of word sequences can be used to predict p (r|w, c). note that we only have to decide whether the word is likely to occur in the context or not; there is no need for a full distribution of words that could occur there otherwise. thus, following word2vec's negative sampling method @xcite , we train our language model only to discriminate between true and "garbled" fragments of text. specifically, we obtain negative samples for training from positive (real) ones by shuffling the order of words and replacing some of them with random entries from vocabulary. we define the set of related words (neighbours) as follows, using relations between lexical units, i.e. senses, and synsets in plwordnet (compare figure 1 ). for relations among lexical units, we include lemmas of the related units. for relations between synsets, we include lemmas of all lexical units belonging to the related synsets. also words from the same synset as the lexical unit in question are taken into account. finally, words from the lexical unit description (gloss) can also be treated as neighbours. intuitively, swapping the ambiguous word for related terms, such as hyponyms or hypernyms, is a method similar to heuristics that a human could use. to give an english example, to disambiguate the word plants in the phrase people there liked to surround themselves with plants, one might try to substitute some synonyms, and estimate how much sense they would make semantically in the context: people there liked to surround themselves with factories, people there liked to surround themselves with flora, people there liked to surround themselves with contrivances, etc. the ones that have the highest probability of occurring would tend to be those which are related to true sense of the original word. since it is possible for a sense to not yield any neighbours, because of having no relevant relations, we use the probability of the original context (that is, the one containing the word being disambiguated) as the baseline probability for all senses. only when a sense does have some other words related to it, the baseline is replaced with either the average or the maximum of their estimated probabilities. estimates for all senses, computed separately, in practice rarely sum to one. we normalise them before making the decision, although this does not influence the final verdict of the model. if many senses have the same, highest estimated probability, we choose from among them at random. ## experiments in plwordnet, there are many types of relationsover 40 in the 3.1 version, not counting subtypes, which makes experimenting with them attractve. we selected some of relation types that seemed particularly useful for our task, and grouped them into three primary subsets. the first subset contains synonymy (including belonging to the same synset), hypernymy and hyponymy, the second contains also antonyms, and the third one, apart from everything from the first subset, incorporates various types of meronymy and other relation types that seem to connect to words that would be adequate replacements for their neighbours in the sentence. for example, in plwordnet there is a number of relations connecting verbs that presuppose or imply each other, or adjectives that differ by magnitude of the quality that they describe. we test 1 how accurate are predictions based on (1, 2, 3) those three subsets, (4) combination of all of them, (5) on words from glosses only, (6) on words in glosses and all words obtained from relation subsets. the basic context probability estimator, serving as the core of our system, is an lstm (hochreiter and schmidhuber, 1997) network, taking nine word vectors as its input, with the disambiguated word position in the middle. the hidden size of an lstm cell is as little as 9 -we have tried bigger values, such as 64 and 128, but they performed worse. the last output of the lstm is squashed with sigmoid function and interpreted as probability. 1 the source code is available at zil.ipipan.waw.pl/codes. previously published set of word embeddings @xcite was used for vectorising sentences. we used 300-dimensional vectors from a word2vec model, trained using continuous bags-of-words and negative sampling on lemmatised corpus consisting of nkjp and the polish wikipedia. as an alternative, we also tested vectorising contexts with elmo embeddings @xcite , using the elmoformanylangs package @xcite . it provided a pretrained model for polish and an appropriate interface. both lstms were trained on the manually annotated, balanced portion of nkjp. these setups were compared with an existing hierarchical softmax model that was trained on full, unbalanced version of nkjp and polish wikipedia corpus. it exists in gensim ( řehůřek and sojka, 2010) format, which allows for scoring probabilities (or more precisely, log likelihoods) of entire sentences, which can be also applied to sentence fragments. as explained in @xcite , log likelihood of a sentence @xmath0, w 2 , . . . , w n ] is defined as the pairwise composite log likelihood: with the skipgram variant of word2vec model which was used here, p (w i |w j ) denotes the conditional probability of a context word w i for a target word w j . the number b is the window size used in model training. in our case, it is 5, so the whole window contains 11 words. the gensim implementation uses a shallower regular word2vec architecture instead of recurrent networks. it is also, in contrast to the rnn, not intrinsically aware of word order. ## results results in on the other hand, glosses appear to work relatively poorly as a source of neighbours for our solution. this may be partially explained by the lack of consistent formatting of glosses in polish word-net, where definitions, examples and other metadata are mixed in a couple of ways in one field of the database. for almost all methods, the approach of taking the maximum probability instead of the average yielded better results. the only exceptions are some weaker versions of gensim and elmo approaches. we hypothesise that neighbours that seem the most likely in given context may indeed reflect the best whether the sense that they represent is appropriate. a possible counterargument would point towards negligible improvements caused by this change to the approach based entirely on words from glosses. although one would think that ignoring junk words from metadata would markedly raise chances of the true sense, this appears not to be the case. it should be noted that these results, unfortunately, are still lower than the baseline of 59.77% cases where the correct sense is the first variant in polish wordnet (which often, but not always, happens to be the most frequent one in polish language). it is a known issue in development of wsd solutions, and for our data this result is even higher than mfs (most frequent sense) accuracy cited for english in @xcite , i.e. 46.4%. however, most measurements exceed the lower baseline of assigning sense annotations at random (45.08% accuracy). among all the models of context probability evaluation, the basic word vector-based lstm performed the best. its superiority over elmo seems to be linked to operating on lemmas, instead of forms, as the pretrained elmo embedder. due to rich morphology of polish, information in a corpus is markedly easier to generalise if the inflections are abstracted away. our preliminary tests with training a form-based lstm operating on word vectors confirmed this hypothesis by degrading maximum accuracy, although it still fared better than elmo on smaller relation subsets. it is true that any rnn shows an improvement over the gensim non-recursive solution, which is unaware of word order. we additionally ran more relaxed tests where this model was allowed to see whole sentences (as the gensim package interface suggests to do), and even then it was not able to reach the level of rnns. analysis of differences between sets of incorrectly classified words has shown the gains to be incremental. this is supported by our experiments with disambiguation by voting of various models, which yielded little improvement. this, along with moderate differences in accuracy, shows that the behaviors of individual variants appear, ultimately, similar. one should keep in mind, however, that our corpus size makes it difficult to draw conclusions concerning particular morphological features in polish that might be the stronger points of some models. we also present results obtained on non-first variant cases only, in we compared our results with the only one other general purpose method for solving polish wsd task described in @xcite . we carried out the test on our test set using two taggers: wcrft2 @xcite and morphodita @xcite . in both cases, we have achieved accuracy of around 51% (more precisely, 51.05% for wcrft2 and 51.77% for morphodita). all versions of our algorithm surpassed these scores, as long as they considered all the subsets of plwordnet relations. ## conclusions we present a new method of disambiguating senses in polish texts using lexical relations from the plwordnet database. we test various relation subsets and approaches to modeling probability of contexts. the wsd problem for polish is still far from being solved. no published results were able to exceed 70% accuracy, which would move them closer to matching those published for english. it is worth pointing out, however, that our accuracy for cases where the first wordnet sense was excluded does approach this level of performance. perhaps finding a way to distinguish the most typical contexts, where one can expect these most frequent senses to occur, can greatly help the overall usefulness of the system. judging from our findings, there is little to be gained by enhancing language models within the same framework of estimating sense likelihoods. the results do show potential in combining modern machine learning with creative use of existing knowledge bases, and should encourage further research in this direction.