Sparsity Makes Sense: Word Sense Disambiguation Using Sparse Contextualized Word Representations (2020)

Abstract: In this paper, we demonstrate that by utilizing sparse word representations, it becomes possible to surpass the results of more complex task-specific models on the task of fine-grained all-words word sense disambiguation. Our proposed algorithm relies on an overcomplete set of semantic basis vectors that allows us to obtain sparse contextualized word representations. We introduce such an information theory-inspired synset representation based on the co-occurrence of word senses and non-zero coordinates for word forms which allows us to achieve an aggregated F-score of 78.8 over a combination of five standard word sense disambiguating benchmark datasets. We also demonstrate the general applicability of our proposed framework by evaluating it towards part-of-speech tagging on four different treebanks. Our results indicate a significant improvement over the application of the dense word representations.

## introduction natural language processing applications have benefited remarkably form language modeling based contextualized word representations, including @xcite , elmo @xcite and bert @xcite , inter alia. contrary to standard "static" word embeddings like word2vec @xcite and glove @xcite , contextualized representations assign such vectorial representations to mentions of word forms that are sensitive to the entire sequence in which they are present. this characteristic of contextualized word embeddings makes them highly applicable for performing word sense disambiguation (wsd) as it has been investigated recently @xcite . another popular line of research deals with sparse overcomplete word representations which differ from typical word embeddings in that most coefficients are exactly zero. such sparse word representations have been argued to convey an increased interpretability @xcite @xcite which could be advantageous for wsd. it has been shown that sparsity can not only favor interpretability, but it can contribute to an increased performance in downstream applications @xcite . the goal of this paper is to investigate and quantify what synergies exist between contextualized and sparse word representations. our rigorous experiments show that it is possible to get increased performance on top of contextualized representations when they are post-processed in a way which ensures their sparsity. in this paper we introduce an information theoryinspired algorithm for creating sparse contextualized word representations and evaluate it in a series of challenging wsd tasks. in our experiments, we managed to obtain solid results for multiple fine-grained word sense disambiguation benchmarks. all our source code for reproducing our experiments are made available at https: //github.com/begab/sparsity_makes_sense. 1 our contributions can be summarized as follows: ## related work one of the key difficulties of natural language understanding is the highly ambiguous nature of language. as a consequence, wsd has long-standing origins in the nlp community @xcite , still receiving major recent research interest @xcite @xcite @xcite . a thorough survey on wsd algorithms of the pre-neural era can be found in @xcite . a typical evaluation for wsd systems is to quantify the extent to which they are capable of identifying the correct sense of ambiguous words in their contexts according to some sense inventory. one of the most frequently applied sense inventory in the case of english is the princeton wordnet @xcite which also served the basis of our evaluation. a variety of wsd approaches has evolved ranging from unsupervised and knowledge-based solutions to supervised ones. unsupervised approaches could investigate the textual overlap between the context of ambiguous words and their potential sense definitions @xcite or they could be based on random walks over the semantic graph providing the sense inventory @xcite . supervised wsd techniques typically perform better than unsupervised approaches. ims @xcite is a classical supervised wsd framework which was created with the intention of easy extensibility. it trains svms for predicting the correct sense of a word based on traditional features, such as surface forms and pos tags of the ambiguous words as well as its neighboring words. the recent advent of neural text representations have also shaped the landscape of algorithms performing wsd. @xcite extended the classical feature-based ims framework by incorporating word embeddings. @xcite devised context2vec, which relies on a bidirectional lstm (bilstm) for performing supervised wsd. @xcite also proposed the utilization of bilstms for wsd. @xcite tackled all-words wsd as a sequence learning model and solved it using lstms. @xcite introduced a similar framework, but replaced the lstm decoder with an ensemble of transformers. @xcite additionally relied on bert contextual word representations as input to their all-words wsd system. contextual word embeddings have recently superseded traditional word embeddings due to their advantageous property of also modeling the neighboring context of words upon determining their vectorial representations. as such, the same word form gets assigned a separate embedding when mentioned in different contexts. contextualized word vectors, including @xcite , typically employ some language modelling-inspired objective and are trained on massive amounts of textual data, which makes them generally applicable in a variety of settings as illustrated by top-performing entries at the superglue leaderboard @xcite . most recently, @xcite have proposed the usage of contextualized word representations for tackling wsd. their framework builds upon bert embeddings and performs wsd relying on a k-nn approach of query words towards the sense embeddings that are derived as the centroids of contextual embeddings labeled with a certain sense. the framework also utilizes static fasttext @xcite embeddings, and averaged contextual embeddings derived from the definitions attached to wordnet senses for mitigating the problem caused by the limited amounts of sense-labeled training data. @xcite proposed the ewise approach which constructs sense definition embeddings also relying on the network structure of word-net for performing zero-shot wsd in order to handle words without any sense-annotated occurrence in the training data. @xcite introduces ewiser as an improvement over the ewise approach by providing a hybrid knowledgebased and supervised approach via the integration of explicit relational information from wordnet. our approach differs from both @xcite and @xcite ) in that we are not exploiting the structural properties of wordnet. sensebert @xcite extends bert @xcite by incorporating an auxiliary task into the masked language modeling objective for predicting word supersenses besides word iden-tities. our approach differs from sensebert as we do not propose an alternative way for training contextualized embeddings, but introduce an algorithm for extracting a useful representation from pretrained bert embeddings that can effectively be used for wsd. due to this conceptual difference, our approach does not need a large transformer model to be trained, but it can be steadily applied over pretrained models. glossbert @xcite framed wsd as a sentence pair classification task between the sentence containing an ambiguous target token and the contents of the glosses for the potential synsets of the ambiguous token and fine-tuned bert accordingly. glossbert hence requires a fine-tuning stage, whereas our approach builds directly on the pre-trained contextual embeddings, which makes it more resource efficient. our work also relates to the line of research on sparse word representations. the seminal work on obtaining sparse word representations by @xcite applied matrix factorization over the co-occurrence matrix built from some corpus. @xcite investigated the linear algebraic structure of static word embedding spaces and concluded that "simple sparse coding can recover vectors that approximately capture the senses". @xcite ; @xcite ; @xcite introduced different approaches for obtaining sparse word representations from traditional static and dense word vectors. our work differs from all the previously mentioned papers in that we create sparse contextualized word representations. ## approach our algorithm is composed of two important steps, i.e. we first make a sparse representation from the dense contextualized ones, then we derive a succinct representation describing the strength of connection between the individual basis of our representation and the sense inventory we would like to perform wsd against. we elaborate on these components next. ## experiments and results we evaluate our approach towards the unified wsd evaluation framework released by @xcite which includes the sense-annotated sem-cor dataset for training purposes. semcor @xcite consists of 802,443 tokens with more than 28% (226,036) of its tokens being senseannotated using wordnet sensekeys. for instance bank%1:14:00:: is one of the possible sensekeys the word bank can be assigned to according to one of the 18 different synsets it is included in wordnet 3.0. wordnet 3.0 contains all together 206,949 distinct senses for 147,306 unique lemmas grouped into 117,659 synsets. we constructed φ relying on the synset-level information of wordnet. ## conclusions in this paper we investigated how the application of sparse word representations obtained from contextualized word embeddings can provide a substantially increased ability for solving problems that require the distinction of fine-grained word senses. in our experiments, we managed to obtain solid results for multiple fine-grained word sense disambiguation benchmarks with the help of our information theory-inspired algorithm. we additionally carefully investigated the effects of increasing the amount of sense-annotated training data and the different design choices we made. we also demonstrated the general applicability of our approach by evaluating it in pos tagging. our source code is made available at https://github. com/begab/sparsity_makes_sense .