C on S e C : Word Sense Disambiguation as Continuous Sense Comprehension (2021)

Abstract: Supervised systems have nowadays become the standard recipe for Word Sense Disambiguation (WSD), with Transformer-based language models as their primary ingredient. However, while these systems have certainly attained unprecedented performances, virtually all of them operate under the constraining assumption that, given a context, each word can be disambiguated individually with no account of the other sense choices. To address this limitation and drop this assumption, we propose CONtinuous SEnse Comprehension (ConSeC), a novel approach to WSD: leveraging a recent re-framing of this task as a text extraction problem, we adapt it to our formulation and introduce a feedback loop strategy that allows the disambiguation of a target word to be conditioned not only on its context but also on the explicit senses assigned to nearby words. We evaluate ConSeC and examine how its components lead it to surpass all its competitors and set a new state of the art on English WSD. We also explore how ConSeC fares in the cross-lingual setting, focusing on 8 languages with various degrees of resource availability, and report significant improvements over prior systems. We release our code at https://github.com/SapienzaNLP/consec .

## introduction being able to understand the meaning of the various words within a particular text is a crucial problem in natural language processing (nlp), with word sense disambiguation (wsd) as arguably its most famous framing: given a word in context, this task aims to pair it with its most suitable meaning, chosen from a fixed sense inventory @xcite . similarly to the vast majority of other nlp tasks, the advent of deep learning has significantly affected the landscape of wsd and has led to supervised neural models becoming its primary actors. in their simplest flavour, these approaches essentially frame this task as a multi-label classification problem over a large vocabulary of discrete senses @xcite . however, although effective and straightforward, this formulation suffers from a number of pitfalls, most notably i) senses are only defined via their training set occurrences, with their actual linguistic meaning not explicitly embedded within the neural model, and ii) these architectures either behave poorly on rare and unseen senses, or cannot handle them at all. in order to address these issues, recent literature has proposed more sophisticated forms of supervision where definitions of senses, i.e. glosses @xcite , and relational knowledge coming from the sense inventory @xcite are integrated within the neural models. although performances have been rising steadily and are now beyond the 80% barrier on the established framework of @xcite , virtually all modern approaches work under the limiting operational hypothesis that the target word's explicit meaning does not depend upon those of its surrounding words, differently from pre-neural strategies @xcite . indeed, while the commonly used pre-trained transformer architectures and their selfattention mechanism @xcite certainly help in contextualizing a word on its context and latently model sense information, the actual disambiguation is performed independently, that is, without taking into consideration the explicit senses assigned to nearby words. 1 this assumption, likely a heritage from the original classification formulation, creates an intrinsic difference between the behavior of humans and that of systems. in this work, we focus on this shortcoming and propose continuous sense comprehension (consec), a novel approach to wsd that exploits a feedback loop strategy to condition the disambiguation process also on the senses of cooccurring words. in particular, given an input text, we define an ordering of the words contained therein and disambiguate each word conditioning not only on its context and possible meanings, but also on the senses assigned to those words already classified. as the underlying neural architecture, inspired by the recent re-framing of wsd presented by @xcite and its nimble adaptability to our setting, we leverage a transformer model trained with a text extraction objective: given as input a text with a target word, its possible sense definitions and the list of already disambiguated words along with their chosen glosses, the model has to learn to extract the text span associated with the sense definition that best expresses the target word's meaning. backed by several experiments on the english all-words wsd task @xcite , we show the benefits of our formulation, which surpasses the prior state of the art by 1.3 f1 points, and perform a complete ablation of the various components that lead consec to achieve unprecedented performances. furthermore, we also examine the scalability of our approach on the cross-lingual setting, evaluating consec on the recently proposed framework of @xcite , and report significant improvements over prior systems. the contributions of this work are therefore as follows: we release our code and models at https:// github.com/sapienzanlp/consec . ## related work word sense disambiguation (wsd) is the task of associating words in context with their most suit-able meaning in a fixed sense inventory @xcite , which is usually a dictionary-like lexical resource where a word's meanings (senses) are enumerated and defined via definitions (glosses) and usage examples. nowadays dominated by supervised systems, with wordnet @xcite and semcor @xcite acting, respectively, as the de facto standard sense inventory and training corpus for the english language, this task is generally approached as a multi-label classification problem with a number of different neural formulations. early neural approaches @xcite focused on architectures where wsd was framed as token classification over wordnet senses. while already effective, these architectures displayed a number of shortcomings, especially with regard to modeling rare and unseen senses. to cope with these, many works started to complement the training data by exploiting different forms of lexical knowledge stored in wordnet, such as sense definitions @xcite and semantic relations @xcite , or with silver data produced via novel generative formulations @xcite . sense definitions, in particular, have been shown to significantly improve models' scalability to senses that are underrepresented in the training corpus, and their usage has been thoroughly investigated. @xcite frame wsd as a binary classification problem where, given a word in context and one of its possible definitions in the sense inventory, a model has to determine whether the meaning expressed by the definition provided is suitable for the word considered. continuing this line of work, blevins and zettlemoyer (2020) leverage a bi-encoder that projects both words in context and glosses in a shared vector space; disambiguation is then performed by means of identifying the gloss closest to the target word. pushing this research trend further, @xcite propose to frame wsd as a text extraction task where, given a word in context and all its possible glosses, models have to extract the definition that best suits the word under consideration. the authors show that their formulation comes with several benefits, most importantly it allows models to attend to both the input context and all the definitions of the target word together, and does not require large output vocab-ularies. however, when disambiguating multiple words co-occurring in the same context, all of these approaches process each word independently from the others: neither is a word disambiguated taking into consideration the explicit senses assigned to nearby words, nor does its explicit sense affect their disambiguation. conversely, here we clearly point out the limits of this formulation and, standing out from previous research, propose a novel approach, consec, which drops this assumption for the first time in supervised neural wsd. by introducing a feedback loop strategy that iteratively disambiguates the target words in a given context, also conditioning at each step on the sense assignments already performed, we report significant improvements over a wide array of experiments and find that consec outperforms all its alternatives by a large margin. ## consec we now describe consec, our proposed approach for wsd. we first introduce our formulation and feedback loop strategy (section 3.1). then, we present our transformer-based architecture (section 3.2) and, finally, we explain how the disambiguation of words is arranged (section 3.3). ## wsd evaluation we now assess the effectiveness of consec examining first its applicability to english all-words wsd both in terms of performances (section 4.1) and via an ablation study of its components (section 4.2). we then proceed to investigate how consec fares in the cross-lingual setting (section 4.3). ## analysis to get better insights into the impact context definitions have on consec performance, we devise three behavioral tests. first of all, we examine to what extent performances degrade at prediction time when the feedback loop is disabled, that is, when we no longer provide context definitions to consec. as shown in second, we perform an adversarial attack where we provide consec with wrong context definitions in the feedback loop: instead of using the most likely definition of wj as δ j , we let δ j be the less likely one. as shown in finally, we investigate the discrepancy between teacher forcing and greedy decoding at inference time, and test the model when, instead, setting δ j to the gold definition of wj in the evaluation datasets. as reported in ## conclusion in this work we presented consec, a novel extractive approach to wsd that allows the disambiguation of words to be conditioned not only on their context and possible meanings but, for the first time in neural wsd literature, also on the explicit senses assigned to nearby words. by explicitly embedding their definitions within the model input, we report significant results on both english and crosslingual wsd, establishing a new state of the art in both settings. most notably, our system reaches the unprecedented performances of 82 f1 on the standard english framework when trained on sem-cor only, and 83.2 when leveraging the wordnet tagged glosses and examples as additional training data. we perform several experiments investigating different aspects of our new formulation and, in particular, addressing its potential speed concerns, we demonstrate that the usage of context definitions can be treated as a pluggable component in our system, to be activated when higher accuracy is required, or seamlessly removed when speed is of primary importance. as future work, we plan to make the disambiguation ordering and the dependencies choice fully differentiable, while at the same time expanding on consec applicability to the cross-lingual setting. with h(w i , w j ) being -log 2 p(w i , w j ), and finally: we use wikipedia as our document collection, and compute the words frequencies on a sample of 100,000 documents for each language.