FEWS : Large-Scale, Low-Shot Word Sense Disambiguation with the Dictionary (2021)

Abstract: Current models for Word Sense Disambiguation (WSD) struggle to disambiguate rare senses, despite reaching human performance on global WSD metrics. This stems from a lack of data for both modeling and evaluating rare senses in existing WSD datasets. In this paper, we introduce FEWS (Few-shot Examples of Word Senses), a new low-shot WSD dataset automatically extracted from example sentences in Wiktionary. FEWS has high sense coverage across different natural language domains and provides: (1) a large training set that covers many more senses than previous datasets and (2) a comprehensive evaluation set containing few- and zero-shot examples of a wide variety of senses. We establish baselines on FEWS with knowledge-based and neural WSD approaches and present transfer learning experiments demonstrating that models additionally trained with FEWS better capture rare senses in existing WSD datasets. Finally, we find humans outperform the best baseline models on FEWS, indicating that FEWS will support significant future work on low-shot WSD.

## introduction word sense disambiguation (wsd) is the task of identifying the sense, or meaning, that an ambiguous word takes in a specific context. recent wsd models @xcite @xcite ) have made large gains on the task, surpassing the estimated 80% f1 human performance upper bound on wordnet annotated corpora @xcite . despite this breakthrough, the task remains far from solved: performance on rare and zero-shot senses is still low, and in general, current wsd models struggle to learn senses with few training examples @xcite ; blevins and zettlemoyer, c: i liked my friend's last status... s1: to enjoy... @xcite be in favor of. s2: to show support for, or approval of, something on the internet by marking it with a vote. c: a transistor-diode matrix is composed of vertical and horizontal wires with a transistor at each intersection. s1: a grid-like arrangement of electronic components, especially one intended for information coding, decoding or storage. s2: a rectangular arrangement of numbers or terms having various uses [in mathematics]. 2020). this performance gap stems from limited data for rare senses in current wsd datasets, which are annotated on natural language documents that contain a zipfian distribution of senses @xcite . more generally, since each word has a different set of candidate senses and new senses are regularly coined, it is almost impossible to gather a large number of examples for each sense in a language. this makes the few-shot learning setting particularly important for wsd. we introduce fews (few-shot examples of word senses), a dataset built to comprehensively train and evaluate wsd models in few-and zero-shot settings. overall, the contributions of fews are two-fold: as training data, it exposes models a broad array of senses in a low-shot setting, and the large evaluation set allows for more robust evaluation of rare senses. fews achieves high coverage of rare senses by automatically extracting example sentences from wiktionary definitions. wiktionary is an apt data source for this purpose, containing examples for over 71,000 senses ( ## related work wsd is a long-standing nlp task and is the focus of many datasets. the current de facto benchmark for modeling english wsd is the wsd evaluation framework @xcite , which includes the semcor dataset @xcite as training data and consolidates a number of evaluation sets @xcite @xcite @xcite into a standardized evaluation suite. these datasets are annotated with the senses (known as synsets) from wordnet, a manually constructed ontology of semantic relations @xcite . most existing datasets for wsd, including those in the wsd evaluation framework and others like @xcite , are annotated on natural language documents that contain a zipfian distribution of word senses @xcite . this data source causes these datasets to have low coverage of rare senses, leading to worse performance on these less common senses @xcite . in contrast, we use examples sentences from wiktionary as an alternative source of text for wsd data with fews. this means that fews has a more uniform sense distribution, providing more balanced coverage across different senses of words. wiktionary has previously been used as a resource for wsd research. most work has investigated mapping wiktionary senses onto wordnet synsets @xcite ; other work has learned similar mappings for wikipedia articles @xcite ; @xcite ; inter alia). more similar to our work, @xcite and @xcite mine wsd examples from wiktionary to augment labeled wsd data for non-english languages. however, fews is the first dataset specifically designed to evaluate zero and few-shot learning with the balanced dictionary sense distribution. ## fews: low-shot learning for wsd fews (few-shot examples of word senses) is a new dataset for learning to do low-shot wsd. foot_0 it is created with example contexts drawn from wiktionary, an online collaborative dictionary. 2 since wiktionary is curated by volunteers, the data is manually annotated and high quality, and there is no additional annotation cost to construct fews. furthermore, using example contexts from a dictionary allows fews to cover many senses underrepresented in existing wsd datasets, such as rare senses of words or senses pertaining to specific domains. however, we note that due to the crowdsourced nature of the examples in wiktionary and the subjectivity of fine-grained sense distinctions, inconsistencies in the underlying data may introduce some annotation errors into fews. examples of the data in fews are shown in figure 1 . in fews, each example context contains one or more instances of the ambiguous target word and is labeled with the sense (and corresponding definition) that describes that word as used in the context; this is in contrast to all-words wsd, where many of the content words in the context are annotated. ## baselines for fews we run a series of baseline approaches on fews to demonstrate how current methods for wsd perform on this dataset. we consider a number of knowledge-based approaches (lesk, lesk+emb, and mfs) and two neural models that build on pretrained encoders (probe bert and bem bert ). we also ascertain how well humans perform on fews as a potential upper bound for model performance. ## conclusion we establish baseline performance on fews with both knowledge-based approaches and recently published neural wsd models. unsurprisingly, neural models based on pretrained encoders perform best on fews; however, the human evaluation shows there is still room for improvement, particularly for zero-shot senses. finally, we also present results on transferring word sense knowledge from fews onto existing wsd datasets with staged finetuning. while our naive approach for transferring knowledge from fews does not improve performance on the global wsd metric, adding fews as an additional training signal improves performance on uncommon senses in existing evaluation sets. we hope that fews will inspire future work foseen closely related senses to those in the zero-shot subsets of from the wiktionary sense inventory; however, these are represented differently in each dataset and correspond to different definitions to be encoded by the biencoder.