Improved Word Sense Disambiguation with Enhanced Sense Representations (2021)

Abstract: Current state-of-the-art supervised word sense disambiguation (WSD) systems (such as GlossBERT and bi-encoder model) yield surprisingly good results by purely leveraging pre-trained language models and short dictionary definitions (or glosses) of the different word senses. While concise and intuitive, the sense gloss is just one of many ways to provide information about word senses. In this paper, we focus on enhancing the sense representations via incorporating synonyms, example phrases or sentences showing usage of word senses, and sense gloss of hypernyms. We show that incorporating such additional information boosts the performance on WSD. With the proposed enhancements, our system achieves an F1 score of 82.0% on the standard benchmark test dataset of the English all-words WSD task, surpassing all previous published scores on this benchmark dataset.

## introduction word sense disambiguation (wsd) refers to the task of automatically identifying the meaning of ambiguous words using computational methods. given a word in context and a fixed inventory of senses, the system determines the correct word sense. for example, the noun "bank" means different things in "financial bank" and "bank of a river". ambiguity is one of the central problems faced by natural language processing (nlp) tasks and wsd aims to resolve semantic ambiguity. it is commonly used to help downstream nlp tasks, such as machine translation @xcite and information retrieval @xcite . supervised wsd approaches typically frame the task as a multi-class classification problem with a fixed sense inventory for each word type. traditionally, many well-performing methods use manually engineered features to train an independent classifier, or word expert, for every word type @xcite . target senses are thus treated as discrete labels. neural-based supervised methods were also explored, with a unified classifier that shares parameters across all polysemous words @xcite . however, they were not able to outperform the word expert supervised systems. more recently, the advent of large language models such as bert @xcite has boosted the performance of these neural-based methods. pre-trained on massive amounts of texts, the language models have a good sense of language context, inherently encoding word sense information. using these models to generate contextualized word representations, a rapid slew of recent publications has continually redefined the state of the art. in combination with language models, lexical resources have also been shown to be able to significantly improve wsd scores. specifically, sense definitions (or glosses) have been used in recent work @xcite @xcite . in both glossbert @xcite and the bi-encoder model (bem) @xcite , good performance was achieved purely by utilizing the context sentence containing the ambiguous word and sense gloss information. in other words, the queried word sense is solely represented by a sense gloss that is typically less than twenty words. given the brevity of information in a sense gloss, it is somewhat surprising that these architectures are able to achieve state-of-the-art performance. in this paper, we show that enhancing the sense representations allows the pre-trained language models to better differentiate between the word senses by improving word sense clustering for each word type. we present a binary sentence pair classification model that is built upon roberta @xcite , with focus on sense representation embellishment. we approach the task as a sentence pair classification problem, performing binary clas-sification on context-sense sentence pairs and training it in an end-to-end fashion. to enhance word sense representation, we introduce a bag of "related" words that is associated with that particular word sense. these "related" words are intuitively chosen to provide more information about the word sense. concretely, it is derived from synonyms, example phrases or sentences showing usage of word senses, and sense gloss of hypernyms. incorporating these additional sources to enhance the sense representation improves the performance on the standard allwords english wsd evaluation benchmark. we achieve an f1 score of 82.0% on this benchmark test dataset, surpassing previous published scores on this test dataset. in summary, the overall contributions of this paper include: our source code and trained models are available at https://github.com/nusnlp/esr . ## related work in this paper, we address the english all-words wsd task, where a system disambiguates every ambiguous word in the dataset @xcite . in general, supervised methods have been shown to perform better on the task, utilizing expensive human annotated data to achieve superior results. combined with recent pre-trained language models, supervised neural architectures have gained popularity in recent years. for example, @xcite investigates different ways of using pre-trained bert to perform wsd, with the glu model outperforming previous work. while supervised methods traditionally do not leverage lexical resources such as wordnet @xcite , lexical information has proven to be useful in other methods. for example, the well-known lesk algorithm @xcite shows that sense gloss is useful, with the algorithm picking the sense whose dictionary gloss shares the most words with the neighborhood of the ambiguous word. with pre-trained language models as feature extractors, sense gloss information can be incorporated into supervised wsd systems, generating significant performance boost. two such examples are gloss-bert and bem. similar to our work, glossbert @xcite formulates the task as a sentence-pair classification problem -using context-gloss pairs to fine-tune the pre-trained bert @xcite on the labeled semcor data @xcite . this becomes a binary classification problem where the system predicts whether the ambiguous word matches the queried sense gloss in a single crossencoder model. however, they use the default bert architecture for sentence pair classification, applying affine transformation on the [cls] token. this summarized word sense query makes it more challenging for the model to identify the ambiguous word. in comparison, our system provides additional information about the ambiguous word (on top of [cls] token), with immediate improved performance. the bem model (blevins and zettlemoyer, 2020) further improves on this approach by using a bi-encoder approach that independently embeds the ambiguous word with its surrounding context and the sense gloss of each queried sense. since they are jointly optimized in the same representation space, disambiguation is performed by finding the nearest sense embedding. unlike glossbert and bem, escher @xcite also utilizes sense gloss, but formulates the task as a span extraction problem. the input is a sentence pair where the first sentence contains the context of the ambiguous word and the second sentence contains the concatenation of glosses from all candidate senses. the system is trained to find the text span corresponding to the correct sense. another challenge faced by supervised systems is the limited training data size. the work from @xcite utilizes usage examples from wordnet to generate more training data. in contrast, our system uses example sentences to improve sense representations instead. other approaches make use of relational information in the lexical knowledge graphs. for example, lmms @xcite uses annotated data to generate sense embeddings us-ing bert. these embeddings are then propagated through the wordnet graph to infer senses that do not appear in semcor. similarly, ares @xcite also achieves full sense coverage but through extraction of relevant contexts. sparselmms @xcite further makes the embeddings sparse through a dictionary matrix. connections are made between each dimension of the sparse embeddings and human interpretable semantic content. ewise @xcite , on the other hand, learns sense embeddings by pre-training a gloss encoder with sense definitions and knowledge graph information. the learned sense gloss embeddings are then scored via dot product with a contextual vector to perform prediction. ewiser @xcite extends ewise by injecting additional relational knowledge from the lexical knowledge graph via a simple sparse dot product operation with an adjacency matrix formulated with the knowledge graph. since the pre-trained sense embeddings are used to classify the ambiguous word, the model is able to predict synsets that are not present in the training set, improving zero-shot performance. our system surpasses previous published systems despite using minimal knowledge graph information (only the sense gloss of hypernyms). ## methodology in this section, we describe the model architecture of our system, and present our method for achieving enhanced sense representations (esr). ## experiments in this section, we provide the details of our experiments and a comparison with other systems. ## analysis in this section, we will analyze the effectiveness of different components constituting the related words in esr: synonyms in the synset, example phrases or sentences from wordnet, and sense definition of hypernym for the synset. we will then evaluate the less frequent sense and zero-shot performance of esr. finally, we will visualize how esr separates different synsets of a word with an example, and show that esr achieves better clustering.