Word Sense Linking: Disambiguating Outside the Sandbox (2024)

Abstract: Word Sense Disambiguation (WSD) is the task of associating a word in a given context with its most suitable meaning among a set of possible candidates. While the task has recently witnessed renewed interest, with systems achieving performances above the estimated inter-annotator agreement, at the time of writing it still struggles to find downstream applications. We argue that one of the reasons behind this is the difficulty of applying WSD to plain text. Indeed, in the standard formulation, models work under the assumptions that a) all the spans to disambiguate have already been identified, and b) all the possible candidate senses of each span are provided, both of which are requirements that are far from trivial. In this work, we present a new task called Word Sense Linking (WSL) where, given an input text and a reference sense inventory, systems have to both identify which spans to disambiguate and then link them to their most suitable meaning.We put forward a transformer-based architecture for the task and thoroughly evaluate both its performance and those of state-of-the-art WSD systems scaled to WSL, iteratively relaxing the assumptions of WSD. We hope that our work will foster easier integration of lexical semantics into downstream applications.

## introduction leveraging the advances in pretrained transformer architectures @xcite , word sense disambiguation (wsd) systems have nowadays reached performances on several evaluation benchmarks that are on par with their estimated interannotator agreement @xcite . however, despite these advances, the task is still well known for struggling to find downstream applications. we argue that one of the possible causes is the difficulty of applying wsd in unconstrained settings due to its heavy assumptions. indeed, three requirements need to be met for a generic state-of-the-art system to perform wsd on some input text: r1) a sense inventory, that is, a semantic resource providing a comprehensive list of all the senses of interest, must be provided, r2) the list of spans to disambiguate in the input text must have already been identified, and r3) an oracle that pairs each span to its set of possible senses, realized through a manually curated word-to-sense mapping, must be available. while the first condition is intrinsic to the disambiguation objective and, thus, unavoidable, we argue that the second two (i.e. r2 and r3) are system-specific assumptions that can be relaxed. we formulate this idea into a new task called word sense linking (wsl), which more accurately reflects the conditions of downstream applications such as neural machine translation @xcite , information extraction @xcite and the enrichment of contextual representations in neural models @xcite . in wsl, given an input text and a reference sense inventory, systems have to identify which spans to disambiguate and link them to their most suitable meaning in the sense inventory. similarly to entity linking @xcite , wsl can be split into three simpler subtasks, with traditional wsd taking place after two initial stages of concept detection (cd), that is, the identification of the spans to be disambiguated in the input text (r2), and candidate generation (cg), the generation of a list of sense candidates for each target span (r3). for example, given some reference sense inventory and the sentence "bus drivers drive buses for a living.", concept detection might identify [bus drivers, drive, buses, living] as the spans to disambiguate, while candidate generation might provide [vehicle, electrical conductor] as the sense candidates for buses. in this work, we first formally introduce the task of word sense linking, along with its three com-ponents, and, then, put forward a novel architecture for the task, based on the retriever-reader paradigm @xcite . we thoroughly study the behavior of our architecture and that of state-ofthe-art wsd systems, once they have been scaled to wsl, as we iteratively relax the sandboxed assumptions of wsd, starting with r2 and continuing with r3. our analysis highlights a number of important -yet neglected -challenges when it comes to performing disambiguation in unconstrained settings. in particular, straightforward and natural extensions of wsd systems to wsl result in large performance drops. in contrast, our model demonstrates considerably more robustness and consistently outperforms such extensions of wsd systems by a large margin. the contributions of this work are therefore as follows: we release code, data, and model weights at https://github.com/babelscape/wsl . ## word sense linking word sense linking is the task of identifying and disambiguating all the spans of an input text with their most suitable senses chosen from a reference inventory. formally, let t be the input text, with t 1 , . . . , t |t| being its words, and i the reference inventory, containing a set of senses. then, a wsl system can be represented as a function f that takes as input the tuple (t, i) and outputs a list of triples [(s 1 , e 1 , g 1 ), . . . , (s n , e n , g n )] where each triple (s i , e i , g i ), @xmath0, n], represents a disambiguated span, with s i and e i being the start and end token index of the span, and g @xmath1. conceptually, wsl can be divided into three subtasks, namely, concept detection, candidate generation, and word sense disambiguation. concept detection (cd) is the task of identifying the spans to disambiguate in an input text t given a reference inventory i. a concept detection system can be represented as a function that takes as input the tuple (t, i) and outputs a list of pairs [(s 1 , e 1 ), . . . , (s n , e n )], each marking the boundaries of a span to disambiguate. specifically, ∀@xmath2, n], s i and e i are the start and end token index of the i-th span to disambiguate. candidate generation (cg) is the task of generating a set of possible candidate senses that an input span occurring in some text t can assume given the reference inventory i. a candidate generation system can be represented as a function that takes as input t, i and the start and end token indices (s, e) of the span, and outputs a set of sense candidates p c ⊆ i, that the span can assume. word sense disambiguation (wsd) aims at identifying, for each target span occurring in an input text t, the most suitable sense among a set of possible candidates. a wsd system can be represented as a function that takes as input t, i, a list of span indices [(s 1 , e 1 ), . . . , (s n , e n )] and corresponding sets of possible candidates [p c 1 , . . . , p c n ]. it then outputs a list of triples [(s 1 , e 1 , g 1 ), . . . , (s n , e n , g n )] where, ∀@xmath3, n], each span (s i , e i ) is paired with the sense g i chosen by the system among the candidates provided in p c i . finally it is easy to see that a wsl system can be built by concatenating, in cascade, a concept detection, a candidate generation, and a word sense disambiguation module. this structure not only facilitates the extension of wsd systems to wsl but also serves as a flexible framework rather than a strict recipe. in fact, as we will show in the next section, the architecture we put forward does not follow this flow and inverts the cd and cg steps. ## model the flow cd-then-cg presents an intrinsic limitation: each span identified by concept detection ought to correspond to the occurrence of a specific sense candidate among those produced by candidate generation and, yet, cg occurs after cd. that ## evaluation we now investigate the effectiveness of our model and how it relates to the r2 and r3 assumptions of wsd (section 1). to this end, we first outline its implementation details (section 4.1) and the novel dataset introduced for the wsd and wsl evaluation (section 4.2), which will remain unchanged throughout our experiments. then, we evaluate it on plain english wsd (section 4.3), so as to set an initial reference for what regards its disambiguation capabilities. finally, we move to wsl and gradually relax these sandboxed assumptions, dropping r2 (section 4.4) and examining different relaxations on r3 (section 4.5). ## related work since this work is the first to introduce word sense linking, unsurprisingly, there is no previous literature that covers it. however, what is surprising is the total absence, to the best of our knowledge, of studies that examine how recent state-of-the-art wsd systems scale to real-world scenarios. this is especially puzzling when we consider the growing interest that wsd has been receiving. indeed, thanks to the introduction of pretrained transformerbased language models @xcite @xcite , this task has been witnessing renewed attention, with the research community focusing on challenging directions such as unseen prediction, cross-inventory generalization, and data efficiency, inter alia. among the alternatives explored, the usage of definitions, which we also follow in this work, has proved to be particularly effective @xcite @xcite , achieving unprecedented performances and allowing sense representations to be disentangled from their occurrences in the training corpus. yet, in spite of this trend and its promising results, to the best of our knowledge, no assessment of how these models might actually be applied in general real-world scenarios has been made, thereby overlooking the relevance of concept detection and candidate generation. we suspect that this oversight is due mainly to two reasons: i) the performances on wsd benchmarks being too scarce, at least until recently, for any downstream application, and ii) the research community assuming that adequate word sense mappings are always available for cg and that a simple heuristic approach could solve cd. in our cd approach, we chose the greedy strategy, a decision supported by @xcite , who observed its widespread use in span identification within texts. however, our study clearly shows that is not a trivial task. both heuristic and supervised techniques report definitely suboptimal behaviors. finally, to provide some background on candidate generation, the task has generally been approached by striving to enumerate all the possible senses a word can assume. however, this is a prohibitively challenging endeavor, especially when wishing to scale across languages. while the research community has put forward a number of studies and mitigating strategies @xcite @xcite , the resulting resources are still incomplete. arguably closest to our work, especially to our wsl model, is @xcite , who address entity linking by initially generating candidates, followed by mention detection (akin to our concept detection) and entity disambiguation (similar to our word sense disambiguation). however, their reader architecture differs significantly from ours: they link one candidate at a time, whereas our model can simultaneously process all spans in an input sequence. a comparison with our model can be found in appendix d. ## conclusion in this work, we challenge the assumptions behind word sense disambiguation (wsd) and introduce a novel task called word sense linking (wsl). wsl requires a system to identify and disambiguate all the spans in an input text using only the information contained in a reference inventory, offering a scenario that is more aligned with practical downstream applications than the conventional wsd approach. along with the wsl formalization, we discuss a first comprehensive study in this direction, presenting a novel retrieved-reader architecture for the task, a complete and comprehensive benchmark for wsl systems, and an analysis of its performances and those of state-of-the-art wsd systems in multiple settings. our findings highlight several important yet overlooked challenges that arise when scaling to unconstrained settings. in particular, natural expansions of wsd systems to wsl appear to be quite brittle, resulting in large performance drops. conversely, our proposed architecture appears to be considerably more robust, achieving superior performances across all wsl settings. looking ahead, we plan to investigate the expansion of wsl to a multilingual setting and analyze the usage of wsl systems in downstream applications. 14340 this work has two inherent limitations: first, due to space constraints, we have deferred the evaluation of the model in a multilingual setting to future work. potential challenges include the necessity for an extension of the sense inventories and the availability of training resources, which are requirements that go beyond the scope of the current study. second, the lack of wsl-specific annotated data meant that we had to rely on datasets designed for word sense disambiguation for training our models. although these datasets offered valuable insights and exhibited promising results on our wsl-specific benchmarks, the prevalence of annotation gaps could hinder the performance of wsl systems. the effort to develop such datasets would be extensive, mirroring the significant undertaking required for our wsl-specific evaluation benchmark.