ESC : Redesigning WSD with Extractive Sense Comprehension (2021)

Abstract: Word Sense Disambiguation (WSD) is a historical NLP task aimed at linking words in contexts to discrete sense inventories and it is usually cast as a multi-label classification task. Recently, several neural approaches have employed sense definitions to better represent word meanings. Yet, these approaches do not observe the input sentence and the sense definition candidates all at once, thus potentially reducing the model performance and generalization power. We cope with this issue by reframing WSD as a span extraction problem — which we called Extractive Sense Comprehension (ESC) — and propose ESCHER, a transformer-based neural architecture for this new formulation. By means of an extensive array of experiments, we show that ESC unleashes the full potential of our model, leading it to outdo all of its competitors and to set a new state of the art on the English WSD task. In the few-shot scenario, ESCHER proves to exploit training data efficiently, attaining the same performance as its closest competitor while relying on almost three times fewer annotations. Furthermore, ESCHER can nimbly combine data annotated with senses from different lexical resources, achieving performances that were previously out of everyone’s reach. The model along with data is available at https://github.com/SapienzaNLP/esc .

## introduction being able to link a piece of raw text to a knowledge base is fundamental in nlp @xcite @xcite , as it can aid neural models to ground their representations on structured resources and enable natural language understanding @xcite . a task that is key to achieving this goal is word sense disambiguation (wsd), where, given a sentence with a target word, a model has to predict its most suitable meaning from a predefined set of labels, i.e., its senses. wsd has not only considerably improved its performance with the advent of deep learning (by around 15 f1 points in 15 years), but it has also shown its benefits in downstream applications such as neural machine translation @xcite and information extraction @xcite , while also being leveraged to enrich the contextual representations of neural models @xcite . however, wsd has mostly been framed as a multi-label classification task @xcite over a very large vocabulary of discrete senses. this formulation may limit a model's capabilities to properly represent word meanings, as each sense is only defined by means of its occurrences in a training set, while its inherent meaning remains linguistically unexpressed. furthermore, rare or unseen senses are either poorly modeled or cannot be modeled at all. these problems have recently been mitigated by integrating sense definitions (glosses) within neural architectures @xcite @xcite ). yet, despite their large improvements, none of these models attends all the possible definitions of a target word at once, and therefore each lacks the ability to represent both the input context and the candidate definitions together. inspired by the extractive reading comprehension framework @xcite in the field of question answering (qa), we cope with these issues and reframe the wsd problem as a novel text extraction task, which we have called extractive sense comprehension (esc). in this setting, a model receives as input a sentence with a target word and all its possible sense definitions. then, we request the model to extract the text span associated with the gloss expressing the target word's most suitable meaning. within this frame-work, we also propose a transformer-based architecture (escher) that implements the esc task by attending to the input context and target word definitions jointly. through an extensive experimental setting, we show that escher surpasses former state-of-the-art approaches by a large margin while, at the same time, requiring almost 3 times less training data points to attain performances comparable to its strongest competitor in a few-shot setting. furthermore, thanks to our new formulation, the proposed model can effectively carry out predictions across different sense repositories and combine distinct inventories with unmatched nimbleness, attaining even higher results than when limited to a single resource only. to summarize, this paper brings the following novel contributions: besides its performance advantages, esc also comes with other benefits: it does not require a large output vocabulary, and it eases the joint use of corpora annotated with different inventories. ## related work word sense disambiguation (wsd) is one of the long-standing problems in lexical semantics, introduced for the first time in the context of machine translation by @xcite . wsd aims at linking a word in context to its most suitable meaning in a predefined sense inventory, which is usually a dictionary where each entry defines a concept via a definition (gloss) and a set of examples. most approaches to wsd rely on wordnet @xcite as the underlying inventory of senses for the english language, and semcor @xcite as training corpus. wordnet organizes lexical-semantic information by means of a graph where sets of synonyms are grouped into synsets (concepts) and edges are typed semantic relations. while early neural models used wordnet as a mere repository of senses @xcite , more recent approaches have started to exploit sense definitions @xcite ) and relational information @xcite . sense definitions, in particular, have been shown to be effective for modeling word senses @xcite , as they provide information orthogonal to that available in the training data. this has been further investigated under different perspectives by @xcite , blevins and zettlemoyer (2020, bem) and @xcite . glossbert casts the wsd problem as a binary classification task where, given a word in context and one of its dictionary definitions, it determines whether this definition matches the word meaning expressed in the context. bem employs a bi-encoder to represent the target word and its sense definitions within the same space. generationary, instead, has predefined sense inventories at its disposal and directly generates a definition given a word in its context. the strength of these approaches lies in the fact that glosses allow senses that are under-represented within the training corpus to be modeled, hence mitigating the long-standing paucity of sense-annotated data @xcite . nevertheless, none of the above approaches can exploit all definitions at once: indeed, glosses are either provided one at a time (glossbert), modeled with one vector only and independently from each other (bem), or used individually as target text to be generated (generationary). our new formulation (esc) for the wsd problem stands out from previous approaches inasmuch as it is the first to access the input context and all the target word's definitions together, while, at the same time, dropping the requirement of a predefined sense inventory. indeed, differently from its competitors, our proposed approach (escher) can scale effectively across different lexical resources even when they were not available at the time of training. ## methodology in what follows, we first formalize the extractive sense comprehension task (section 3.1), then in- t h e b u l l y h a d t o b a c k d o w @xmath0. r e m o v e o n e s e l f f r o m a n o b l i g a t i o n . m o v e b a c k w a r d s f r o m a c e r t a i n p o s i t i o @xmath1. < / @xmath2, a transformer-based architecture for esc (section 3.2), and finally put forward a novel approach for mitigating the bias towards the most frequent meanings @xcite within training data (section 3.3). ## standard wsd evaluation in this section we introduce the experimental setting we use to evaluate the proposed framework and neural architecture. ## few-shot evaluation we now move to analyzing the performances of escher in a few-shot scenario, i.e., when the number of samples available for each sense is limited. we also report the f1 scores of escher on the mfs, lfs and 0-lex-def datasets in the same scenario in order to investigate the extent to which the difference in the number of occurrences for each sense impacts the ability of the model to generalize on rare senses. ## error analysis in order to get a clear picture of the model's pitfalls and gain insights into possible directions for future work, we perform an analysis of escher misclassifications on the all dataset. we find that the mistaken predictions belong to three main categories: most frequent sense bias, insufficient context and wordnet sense granularity. since we already discussed the first of these in the previous sections, we focus here on the latter two. insufficient context annotators often compiled the wsd evaluation datasets by considering each instance in the context of the documents they appear in. in contrast, wsd models typically take into account only the sentence surrounding the target word, discarding a large portion of the available context. this behavior causes a discrepancy where sentences do not provide enough information to disambiguate the target words therein. indeed, escher mistakes most often appear in sentences with an average length of 27 tokens, i.e., roughly 5 tokens less than the average length in all (32). this suggests that moving the disambiguation context from sentences to documents may improve the performances of models as long as they are capable of handling longer sequences. ## conclusion in this paper, we introduced a novel framing for the word sense disambiguation problem inspired by the extractive reading comprehension task in qa: given a word in a sentence and a text containing all its possible definitions, a model has to identify the span containing the correct definition for the target word. for this new formulation -which we called extractive sense comprehension (esc) -we devised a transformer-based architecture (escher), which, differently from previous approaches, can look at all the target word definitions at once, alongside the input sentence. escher surpasses the current state of the art by 1.7 points on the standard english all-words wsd task, thanks to its more efficient use of the training data. also, when provided with only a few examples for each sense, escher attains remarkable levels of performance, requiring roughly three times less annotated instances than its direct competitor to reach the same performances. furthermore, our new formulation allows escher to scale across different inventories and to combine them effectively. indeed, when provided with data annotated with multiple vocabularies, it achieves even better results than when limited to one inventory only, with results in the 86-88% range. as future work we plan to expand this framework so as to condition the prediction not only on the target word context and definitions, but also on the possible senses of its surrounding words. the pretrained model, along with code and data, is available at https://github.com/ sapienzanlp/esc .