Don’t Neglect the Obvious: On the Role of Unambiguous Words in Word Sense Disambiguation (2020)

Abstract: State-of-the-art methods for Word Sense Disambiguation (WSD) combine two different features: the power of pre-trained language models and a propagation method to extend the coverage of such models. This propagation is needed as current sense-annotated corpora lack coverage of many instances in the underlying sense inventory (usually WordNet). At the same time, unambiguous words make for a large portion of all words in WordNet, while being poorly covered in existing sense-annotated corpora. In this paper, we propose a simple method to provide annotations for most unambiguous words in a large corpus. We introduce the UWA (Unambiguous Word Annotations) dataset and show how a state-of-the-art propagation-based model can use it to extend the coverage and quality of its word sense embeddings by a significant margin, improving on its original results on WSD.

## introduction there has been a lot of progress in word sense disambiguation (wsd) recently. this progress has been driven by two factors: (1) the introduction of large pre-trained transformer-based language models and (2) propagation algorithms that extends the coverage of existing training sets. the gains due to pre-trained neural language models (nlms) such as bert @xcite have been outstanding, helping reach levels close to human performance when training data is available. these models are generally based on a nearest neighbours strategy, where each sense is represented by a vector, exploiting the contextualized embeddings of these nlms @xcite @xcite . however, training data for wsd is hard to obtain, and the most widely used training set nowadays, based on word-net, dates back from the 90s @xcite . this lack of curated data produces the so-called knowledge-acquisition bottleneck @xcite . however, there is a key source of information that has been neglected so far in existing senseannotated corpora and propagation methods, which is the presence of unambiguous words from the underlying knowledge resource. strikingly, word-net, which is known to be a comprehensive resource, is mostly composed of unambiguous entries (30k lemmas are ambiguous, compared to 116k unambiguous). while the lack of unambiguous annotations does not have a direct effect in wsd, the fact that these unambiguous words are part of the same semantic network means they can have an effect on ambiguous words via standard propagation algorithms. these propagation algorithms start from a seed of senses occurring in the training data (and therefore their embeddings can be directly computed) and then propagate to the whole sense inventory via the semantic network @xcite . consequently, computing sense embeddings for unambiguous words can increase the number of seeds and improve the whole process. covering these unambiguous words, however, is not an arduous task, as unlabelled corpora may suffice. we explore this hypothesis by labeling a large amount of unambiguous words in corpora extracted from the web, using wordnet as our reference sense inventory. while we can certainly find usages of a word not covered by wordnet, we found that our approach can obtain accurate occurrences with simple heuristics. the contribution of this paper is twofold. first, we devise a simple methodology to construct uwa (unambiguous word annotations), a large and, most importantly, diverse sense-annotated corpus that focuses on wordnet unambiguous words. second, we show that by leveraging uwa, we can significantly improve a state-of-the-art wsd model. ## related work the knowledge-acquisition bottleneck has been frequently addressed by automatically constructing sense-annotated corpora. recent works propose methods that exploit knowledge from wikipedia, such as nasari vectors @xcite , for providing sense annotations for concepts and entities @xcite . in the case of @xcite , and similarly to @xcite , their method requires hyperlinks and category information from wikipedia, hence not extensible to other kinds of corpora. 1 previous approaches relied on parallel corpora for two or more languages. the omsti corpus @xcite was constructed by exploiting the alignments of an english-chinese corpus. similarly, delli bovi et al. (2017) presented eurosense, a multilingual sense-annotated corpus using the europarl parallel corpus for 21 languages as reference. in contrast to these approaches, we focus on unambiguous senses and, therefore, are not constrained to only nouns, knowledge from wikipedia, or a specific type of corpus. earlier works exploiting unambiguous words @xcite @xcite and especially the subsequent extension by @xcite are the most directly related to our paper. @xcite retrieved example sentences with monosemous nouns from web search snippets and used them towards improved performance on wsd by leveraging wordnet relations. however, the wsd methods analyzed were sensitive to frequency bias, leading their collection effort to collect a large number of examples for fewer senses (and only nouns). in contrast, our solution is designed for all monosemous words, retrieving examples from web texts instead of snippets, attaining performance gains with even a single example per word. ## methodology in this section we first explain our method to construct a corpus with unambiguous word annotations (section 3.1). then, we explain current models based on language models for wsd (section 3.2) and describe a propagation method to infer additional oov sense representations (section 3.3). 1 pasini and camacho-collados (2020) provide a more detailed overview of existing sense-annotated corpora. ## evaluation for our experiments we are interested in verifying the impact of using uwa to improve wsd performance. in particular, we test the unambiguous annotations of uwa as a complement of existing sense-annotated training data. to this end, as explained in section 3, we make use of the state-ofthe-art wsd model lmms @xcite . in addition to the original version using bert, we also provide results with roberta @xcite for completeness. we use the 24-layer models for both bert and roberta. foot_1 ## analysis in this section, we provide an analysis based on the number of examples (section 5.1) and a visualization of the embedding space (section 5.2). ## conclusion unambiguous words are a surprisingly large portion of existing knowledge resources like word-net. at the same time, their coverage in existing sense-annotated corpora is very limited. in this paper, we proposed a simple method which exploits sense annotations of unambiguous words from unlabeled corpora, thereby effectively extending existing sense-annotated corpora with low-effort. by leveraging a state-of-the-art bert-based wsd sys4 we share interactive visualizations focusing on each of the 45 supersense groups (e.g. noun.communication) from wordnet at our uwa release website. tem that propagates sense embeddings across word-net, we have shown that these unambiguous words provide an excellent bridge to reach a wider range of oov senses. this translates, in turn, into improving results for wsd. for future work it would be interesting to test these sense embeddings in a wider range of applications outside wsd. since the embedding space is clearly more diversified, as shown in figure 2 , this may lead to improvements in other downstream tasks. moreover, one of the most surprising findings from this paper is that a single occurrence of oov unambiguous words is enough to improve the performance of wsd models. this is relevant because (1) it is not always easy to retrieve a large number of examples for unambiguous words, and (2) it facilitates a cheaper manual verification, if required. finally, we openly release uwa, a large corpus annotated with unambiguous words, together improved bert and roberta-based sense embeddings, model predictions and visualizations at http://danlou.github.io/uwa .