Non-Parametric Few-Shot Learning for Word Sense Disambiguation (2021)

Abstract: Word sense disambiguation (WSD) is a long-standing problem in natural language processing. One significant challenge in supervised all-words WSD is to classify among senses for a majority of words that lie in the long-tail distribution. For instance, 84% of the annotated words have less than 10 examples in the SemCor training data. This issue is more pronounced as the imbalance occurs in both word and sense distributions. In this work, we propose MetricWSD, a non-parametric few-shot learning approach to mitigate this data imbalance issue. By learning to compute distances among the senses of a given word through episodic training, MetricWSD transfers knowledge (a learned metric space) from high-frequency words to infrequent ones. MetricWSD constructs the training episodes tailored to word frequencies and explicitly addresses the problem of the skewed distribution, as opposed to mixing all the words trained with parametric models in previous work. Without resorting to any lexical resources, MetricWSD obtains strong performance against parametric alternatives, achieving a 75.1 F1 score on the unified WSD evaluation benchmark (Raganato et al., 2017b). Our analysis further validates that infrequent words and senses enjoy significant improvement.

## introduction word sense disambiguation (wsd) @xcite is a widely studied problem that aims to assign words in text to their correct senses. despite advances over the years, a major challenge remains to be the naturally present data imbalance issue. models suffer from extreme data imbalance, rendering learning the long-tail examples a major focus. in the english all-words wsd task @xcite , 84% of the annotated words 2 have less 1 our code is publicly available at: https://github. com/princeton-nlp/metric-wsd . recent approaches tackle this problem by resorting to extra sense information such as gloss (sense definition) and semantic relations to mitigate the issue of rare words and senses @xcite @xcite @xcite . however, most work sticks to the parametric models that share parameters between words and adopts standard supervised learning mixing all the words of different frequencies. we argue that this accustomed paradigm exposes a missing opportunity to explicitly address the data imbalance issue. in this work, we propose metricwsd, a simple non-parametric model coupled with episodic training to solve the long-tail problem, drawing inspiration from few-shot learning methods such as prototypical networks @xcite . given a word, the model represents its senses by encoding a sampled subset (support set) of the training data and learns a distance metric between these sense repre-sentations and the representations from the remaining subset (query set). this lightens the load for a model by learning an effective metric space instead of learning a sense representation from scratch. by sharing only the parameters in the text encoder, the model will trickle the knowledge of the learned metric space down from high-frequency words to infrequent ones. we devise a sampling strategy that takes word and sense frequency into account and constructs support and query sets accordingly. in combination, this non-parametric approach naturally fits in the imbalanced few-shot problems, which is a more realistic setting when learning from a skewed data distribution as in wsd. we evaluate metricwsd on the unified wsd evaluation benchmark @xcite , achieving a 75.1% test f1 and outperforming parametric baselines using only the annotated sense supervision. a further breakdown analysis shows that the non-parametric model outperforms the parametric counterparts in low-frequency words and senses, validating the effectiveness of our approach. ## related work word sense disambiguation has been studied extensively as a core task in natural language processing. early work computes relatedness through conceptgloss lexical overlap without supervision @xcite . later work designs features to build word-specific classifiers (word expert) @xcite @xcite . all-words wsd unifies the datasets and training corpora by collecting large scale annotations @xcite , which becomes the standard testbed for the wsd task. however, due to the naturally present longtail annotation, word expert approaches fall short in utilizing information across different words. recent supervised neural approaches prevail word-independent classifiers by more effective sentence feature extraction and achieve higher performance @xcite . approaches that use large pretrained language models @xcite further boost the performance @xcite . recent work turns to incorporate gloss information @xcite @xcite . other work explores more lexical resources such as knowledge graph structures @xcite @xcite . all the above approaches mix words in the dataset and are trained under a standard supervised learning paradigm. another close work to ours is @xcite , which converts wsd into an n -way, k-shot few-shot learning problem and explores a range of meta-learning algorithms. this setup assumes disjoint sets of words between meta-training and meta-testing and deviates from the standard wsd setting. ## experiments we evaluate our approach with the wsd framework proposed by @xcite . we train our model on semcor 3.0 and use semeval-2007 (se07) for development and the rest: senseval-2 (se02), senseval-3 (se03), semeval-2013 (se13), and semeval-2015 (se15) for testing. following standard practice, we report performance on the separate test sets, the concatenation of all test sets, and the breakdown by part-of-speech tags. for all the experiments, we use the bert-base (uncased) model as the text encoder. baselines we first compare to two simple baselines: wordnet s1 always predicts the first sense and mfs always predicts the most frequent sense in the training data. we compare our approach to bert-classifier: a linear classifier built on top of bert (all the weights are learned together). as opposed to our non-parametric approach, the bertclassifier has to learn the output weights from scratch. we compare to another supervised baseline using contextualized word representations that extends the input context text with its surrounding sentences in the semcor dataset @xcite . we also compare to a non-parametric nearest neighbor baseline bert-knn, which obtains sense representations by averaging bert encoded representations from training examples of the same sense. it predicts the nearest neighbor of the input among the sense representations. the bert weights are frozen which, different from our approach, does not learn the metric space. models using only supervised wsd data fall back to predicting the most frequent sense (mfs) when encountering unseen words. for reference, we also list the results of recent state-of-the-art methods that incorporate gloss information including ewise (kumar et al., 2019), ewiser @xcite , glossbert @xcite , and bem @xcite . more implementation details are given in appendix a. overall results ## analysis qualitative analysis ## conclusion in this work, we introduce metricwsd, a few-shot non-parametric approach for solving the data imbalance issue in word sense disambiguation. through learning the metric space and episodic training, the model learns to transfer knowledge from frequent words to infrequent ones. metricwsd outperforms previous methods only using the standard annotated sense supervision and shows significant improvements on low-frequency words and senses. in the future, we plan to incorporate lexical information to further close the performance gap.