Moving Down the Long Tail of Word Sense Disambiguation with Gloss Informed Bi-encoders (2020)

Abstract: A major obstacle in Word Sense Disambiguation (WSD) is that word senses are not uniformly distributed, causing existing models to generally perform poorly on senses that are either rare or unseen during training. We propose a bi-encoder model that independently embeds (1) the target word with its surrounding context and (2) the dictionary definition, or gloss, of each sense. The encoders are jointly optimized in the same representation space, so that sense disambiguation can be performed by finding the nearest sense embedding for each target word embedding. Our system outperforms previous state-of-the-art models on English all-words WSD; these gains predominantly come from improved performance on rare senses, leading to a 31.1% error reduction on less frequent senses over prior work. This demonstrates that rare senses can be more effectively disambiguated by modeling their definitions.

## introduction one of the major challenges of word sense disambiguation (wsd) is overcoming the data sparsity that stems from the zipfian distribution of senses in natural language @xcite . for example, in semcor (the largest manually annotated dataset for wsd) 90% of mentions of the word plant correspond to the top two senses of the word, and only half of the ten senses of plant occur in the dataset at all @xcite . due to this data imbalance, many wsd systems show a strong bias towards predicting the most frequent sense (mfs) of a word regardless of the surrounding context @xcite . a successful wsd system should be able to overcome this bias and correctly disambiguate cases where a word takes a less frequent sense (lfs), without sacrificing performance on mfs examples. previous work has found that incorporating lexical information such as sense definitions, or glosses, into wsd systems improves performance @xcite ). 1 glosses have also been found to improve lfs performance; however, absolute performance on rare senses is still low, with models showing a 62.3 f1 performance drop between the mfs examples and the lfs ones @xcite . in this paper, we show that this gap can be significantly reduced by jointly fine-tuning multiple pretrained encoders on wsd. we present a bi-encoder model built on top of bert @xcite that is designed to improve performance on rare and zero-shot senses. similar to prior work, our system represents the target words and senses in the same embedding space by using a context encoder to represent the target word and surrounding context, and a gloss encoder to represent the sense definitions. however, our two encoders are jointly learned from the wsd objective alone and trained in an end-to-end fashion. this approach allows our model to outperform prior work on the english all-words wsd task introduced in @xcite . analysis of our model shows that these gains come almost entirely from better performance on the less frequent senses, with an 15.6 absolute improvement in f1 performance over the closest performing system; our model also improves on prior work in the zeroshot setting, where we evaluate performance on words and senses not seen during training. finally, we train our model in a few-shot setting in order to investigate how well the bi-encoder system learns on a limited set of training examples per sense. the bi-encoder architecture is able to generalize better from the limited number of exam-ples than a strong pretrained baseline. this results demonstrates the data efficiency of our system and indicates why it captures lfs well, as less common senses naturally only have a few training examples in the data. in summary, the overall contributions of this work are as follows: ## background and related work word sense disambiguation (wsd) is the task of predicting the particular sense, or meaning, of a word when it occurs in a specific context @xcite . understanding what a word means in context is critical to many nlp tasks, and wsd has been shown to help downstream tasks such as machine translation (mt) @xcite @xcite and information extraction (ie) @xcite . the formulation of wsd that we address is allwords wsd, where the model disambiguates every ambiguous word in the data (e.g., @xcite ; @xcite ). many wsd systems approached this task with manually engineered features that were used to learn an independent classifier, or word expert, for each ambiguous lemma @xcite . later work also integrated word embeddings into this independent classifier approach @xcite . neural models for wsd built on this approach by training encoders for better feature extraction; they then either still learned independent classifiers on top of the encoded features @xcite , or labeled each word using a shared output space @xcite . other neural approaches used semi-supervised learning to augment the learned representations with additional data @xcite . ## methodology in this section, we present an approach for wsd that is designed to more accurately model less frequent senses by better leveraging the glosses that define them. the overall model architecture is shown in figure 1 . our bi-encoder model (bem) consists of two independent encoders: (1) a context encoder, which represents the target word (and its surrounding context) and (2) a gloss encoder, that embeds the definition text for each word sense. these encoders are trained to embed each token near the representation of its correct word sense. each encoder is a deep transformer network initialized with bert, in order to leverage the word sense information it captures from pretraining @xcite . to describe our approach, we formally define the task of wsd (section 3.1), and then present the bem system in detail (section 3.2). ## evaluation we present a series of experiments to evaluate our bi-encoder wsd model. we first compare the bem against several baselines and prior work on english all-words wsd (section 5.1), and then evaluate performance on the most frequent (mfs), less frequent (lfs), and zero-shot examples (section 5.2). 4 we initialize the models with bert-base due to better baseline performance on wsd than roberta-base, see section 6.1 for more details ## analysis experiments in our model evaluation, we found that bem outperforms prior work by improving disambiguation of less frequent senses while maintaining high performance on common ones. this section presents a series of analysis experiments in order to determine which aspects of the approach contribute to these improvements. in section 6.1, we ablate different aspects of our model, and we consider the effect of balancing the training signal across senses with different frequencies in section 6.2. finally, we perform a qualitative analysis of the learned sense embedding space in section 6.3. ## few-shot learning of wsd in this section, we investigate how efficient the bem is in a few-shot learning setting, by limiting the number of training examples the model can observe per sense. we hypothesize that our model will be more efficient than a standard classifier for learning wsd, due to the additional information provided by the sense definitions. in order to simulate a low-shot data setting, we create k-shot training sets by filtering the sem-cor data such that the filtered set contains up to k examples of each sense in the full dataset; we then train the bi-encoder model using only this filtered training data. we train models on values of @xmath0, 3, 5, 10 and compare their performance against the model trained on the full train set. we also retrain the frozen bert-base classifier baseline for each k considered. in order to keep training comparable across different amounts of training data, we train each few-shot bem for the same number of training steps as the system trained on the full dataset (approximately 180,000 updates). the results of this experiment are given in figure 3 . unsurprisingly, both the frozen bert classifier and the bem achieve better f1 scores as we increase k and train them on additional data. however, we see that the bem is more efficient on smaller values of k, with a much smaller drop off in performance at @xmath1. this efficiency also allows the bem to achieve similar performance to the full baseline model with only 5 (or fewer) examples per sense. the performance of these few-shot models gives us insight into the the kinds of data that could be used to improve wsd models. while it would be prohibitively difficult to annotate many examples for every sense considered by a wsd system, it is possible that augmenting existing wsd data to provide a few labeled examples of rare senses could be more effective than simply annotating more data without considering the sense distribution. these sorts of considerations are particularly important when extending the wsd task to new domains or languages, where a great deal of new data needs to be annotated; an important goal for these sorts of data augmentation is to make sure they allow for the efficient learning of all senses. ## conclusion in this work, we address the issue of wsd systems underperforming on uncommon senses of words. we present a bi-encoder model (bem) that maps senses and ambiguous words into the same embedding space by jointly optimizing the context and glosses encoders. the bem then disambiguates the sense of each word by assigning it the label of the nearest sense embedding. this approach leads to a 31.1% error reduction over prior work on the less frequent sense examples. however, we still see a large gap in performance between mfs and lfs examples, with our model still performing over 40 points better on the mfs subset. most recent wsd systems show a similar trend: even the representations of frozen bertbase that are not fine-tuned on wsd can achieve over 94 f1 on examples labeled with the most frequent sense. this leaves better disambiguation of less common senses as the main avenue for future work on wsd. potential directions include finding ways to obtain more informative training signal from uncommon senses, such as with different approaches to loss reweighting, and exploring the effectiveness of other model architectures on lfs examples. another direction for future work would improve few-shot approaches to wsd, which is both important for moving wsd into new domains and for modeling rare senses that naturally have less support in wsd data. sascha rothe and hinrich schütze. 2015. autoextend: extending word embeddings to embeddings for synsets and lexemes. in proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (volume 1: long papers), pages 1793-1803, beijing, china. association for computational linguistics. hui shen, razvan bunescu, and rada mihalcea. 2013. coarse to fine grained sense disambiguation in wikipedia. in second joint conference on lexical and computational semantics (* sem), volume 1: proceedings of the main conference and the shared task: semantic textual similarity, pages 22-31. benjamin snyder and martha palmer. 2004. the english all-words task. in proceedings of senseval-3, the third international workshop on the evaluation of systems for the semantic analysis of text. gabriel stanovsky and mark hopkins. 2018. spot the odd man out: exploring the associative power of lexical resources. in proceedings of the 2018 conference on empirical methods in natural language processing, pages 1533-1542. loïc vial, benjamin lecouteux, and didier schwab. 2019. sense vocabulary compression through the semantic knowledge of wordnet for neural word sense disambiguation. in proceedings of the 10th global wordnet conference (gwc). david vickrey, luke biewald, marc teyssier, and daphne koller. 2005. word-sense disambiguation for machine translation. in proceedings of the conference on human language technology and empirical methods in natural language processing, hlt '05, pages 771-778, stroudsburg, pa, usa. association for computational linguistics. thomas wolf, lysandre debut, victor sanh, julien chaumond, clement delangue, anthony moi, pierric cistac, tim rault, r'emi louf, morgan funtowicz, and jamie brew. 2019. huggingface's transformers: state-of-the-art natural language processing. arxiv, abs/1910.03771. dayu yuan, julian richardson, ryan doherty, colin evans, and eric altendorf. 2016. semi-supervised word sense disambiguation with neural models. arxiv preprint arxiv:1603.07012. zhi zhong and hwee tou ng. 2010. it makes sense: a wide-coverage word sense disambiguation system for free text. in proceedings of the acl 2010 system demonstrations, pages 78-83.