Sense Vocabulary Compression through the Semantic Knowledge of W ord N et for Neural Word Sense Disambiguation (2019)

Abstract: In this article, we tackle the issue of the limited quantity of manually sense annotated corpora for the task of word sense disambiguation, by exploiting the semantic relationships between senses such as synonymy, hypernymy and hyponymy, in order to compress the sense vocabulary of Princeton WordNet, and thus reduce the number of different sense tags that must be observed to disambiguate all words of the lexical database. We propose two different methods that greatly reduce the size of neural WSD models, with the benefit of improving their coverage without additional training data, and without impacting their precision. In addition to our methods, we present a WSD system which relies on pre-trained BERT word vectors in order to achieve results that significantly outperforms the state of the art on all WSD evaluation tasks.

## introduction word sense disambiguation (wsd) is a task which aims to clarify a text by assigning to each of its words the most suitable sense labels, given a predefined sense inventory. various approaches have been proposed to achieve wsd: knowledge-based methods rely on dictionaries, lexical databases, thesauri or knowledge graphs as primary resources, and use algorithms such as lexical similarity measures @xcite or graph-based measures @xcite . supervised methods, on the other hand, exploit sense annotated corpora as training instances for a classifier such as svm @xcite , or more recently by a neural network @xcite . finally, unsupervised methods automatically iden-tify the different senses of words from unannotated or parallel corpora (e.g. @xcite ). supervised methods are by far the most predominant as they generally offer the best results in evaluation campaigns (for instance @xcite ). state of the art classifiers used to combine specific features such as the parts of speech and the lemmas of surrounding words @xcite , but they are now replaced by neural networks which learn their own representation of words @xcite . one major bottleneck of supervised systems is the restricted quantity of manually sense annotated corpora: in the annotated corpus semcor @xcite , the largest manually sense annotated corpus available, words are annotated with 33 760 different sense keys, which corresponds to only approximately 16% of the sense inventory of wordnet @xcite , the lexical database of reference widely used in wsd. many works try to leverage this problem by creating new sense annotated corpora, either automatically @xcite , semi-automatically @xcite , or through crowdsourcing @xcite . in this work, the idea is to solve this issue by taking advantage of the semantic relationships between senses included in wordnet, such as the hypernymy, the hyponymy, the meronymy, the antonymy, etc. our method is based on the observation that a sense and its closest related senses (its hypernym or its hyponyms for instance) all share a common idea or concept, and so a word can sometimes be disambiguated using only related concepts. consequently, we do not need to know every sense of wordnet to disambiguate all words of wordnet. for instance, let us consider the word "mouse" and two of its senses which are the computer mouse and the animal mouse. we only need to know the notions of "animal" and "electronic de-vice" to distinguish them, and all notions that are more specialized such as "rodent" or "mammal" are therefore superfluous. by grouping them, we can benefit from all other instances of electronic devices or animals in a training corpus, even if they do not mention the word "mouse". contributions: in this paper, we hypothesize that only a subset of wordnet senses could be considered to disambiguate all words of the lexical database. therefore, we propose two different methods for building this subset and we call them sense vocabulary compression methods. by using these techniques, we are able to greatly improve the coverage of supervised wsd systems, nearly eliminating the need for a backoff strategy that is currently used in most systems when dealing with a word which has never been observed in the training data. we evaluate our method on a state of the art wsd neural network, based on pretrained contextualized word vector representations, and we present results that significantly outperform the state of the art on every standard wsd evaluation task. finally, we provide a documented tool for training and evaluating neural wsd models, as well as our best pretrained model in a dedicated github repository foot_0 . ## related work in wsd, several recent advances have been made in the creation of new neural architectures for supervised models and the integration of knowledge into these systems. multiple works also exploit the idea of grouping together related senses. in this section, we give an overview of these works. ## sense vocabulary compression current state of the art supervised wsd systems such as @xcite , @xcite , @xcite and @xcite are all confronted to the following issues: 1. due to the small number of manually sense annotated corpora available, a target word may never be observed during the training, and therefore the system is not able to annotate it. 2. for the same reason, a word may have been observed, but not all of its senses. in this case the system is able to annotate the word, but if the expected sense has never been observed, the output will be wrong, regardless of the architecture of the supervised system. 3. training a neural network to predict a tag which belongs to the set of all wordnet senses can become extremely slow and requires a lot of parameters with a large output vocabulary. and this vocabulary goes up to 206 941 if we consider all word-senses of wordnet. in order to overcome all these issues, we propose a method for grouping together multiple sense tags that refer in fact to the same concept. in consequence, the output vocabulary decreases, the ability of the trained system to generalize improves, as well as its coverage. ## conclusion in this paper, we presented two new methods that improve the coverage and the capacity of generalization of supervised wsd systems, by narrowing down the number of different sense in wordnet in order to keep only the senses that are essential for differentiating the meaning of all words of the lexical database. on the scale of the whole lexical database, we showed that these methods can shrink the total number of different sense tags in wordnet to only 6% of the original size, and that the coverage of an identical training corpus has more than doubled. we implemented a state of the art wsd neural network and we showed that these methods compress the size of the underlying models by a factor of 1.2 to 2, and greatly improve their coverage on the evaluation tasks. as a result, we reach a coverage of 99.99% of the evaluation tasks (1 word missing on 7 253) when training a system on the semcor only, and 100% when adding the wngc to the training data, on the pol-ysemic words. therefore, the need for a backoff strategy is nearly eliminated. finally, our method combined with the recent advances in contextualized word embeddings and with a training corpus composed of sense annotated glosses, our system achieves scores that considerably outperform the state of the art on all wsd evaluation tasks.