Framing Word Sense Disambiguation as a Multi-Label Problem for Model-Agnostic Knowledge Integration (2021)

Abstract: Recent studies treat Word Sense Disambiguation (WSD) as a single-label classification problem in which one is asked to choose only the best-fitting sense for a target word, given its context. However, gold data labelled by expert annotators suggest that maximizing the probability of a single sense may not be the most suitable training objective for WSD, especially if the sense inventory of choice is fine-grained. In this paper, we approach WSD as a multi-label classification problem in which multiple senses can be assigned to each target word. Not only does our simple method bear a closer resemblance to how human annotators disambiguate text, but it can also be seamlessly extended to exploit structured knowledge from semantic networks to achieve state-of-the-art results in English all-words WSD.

## introduction word sense disambiguation (wsd) is traditionally framed as the task of associating a word in context with its correct meaning from a finite set of possible choices @xcite . following this definition, recently proposed neural models were trained to maximize the probability of the most appropriate meaning while minimizing the probability of the other possible choices @xcite @xcite . although this training objective proved to be extremely effective and even led to @xcite reaching the estimated upper bound of interannotator agreement for wsd performance on the unified evaluation framework of @xcite , adhering to it underplays a fundamental aspect of how human annotators disambiguate text. indeed, past studies have observed that it is not uncommon for a word to have multiple appropriate meanings in a given context, meanings that can be used interchangeably under some circumstances because their boundaries are not clear cut @xcite @xcite . this is especially evident if the underlying sense inventory is fine-grained, as the complexity, and therefore performance, of wsd is tightly coupled to sense granularity @xcite . the difficulty an annotator faces in choosing the most appropriate meaning from a fine-grained sense inventory becomes clear from an analysis of gold standard datasets: a non-negligible 5% of the target words are annotated with two or more sense labels in several gold standard datasets, including senseval-2 @xcite , senseval-3 @xcite , @xcite , @xcite , and semeval-2015 @xcite . therefore, we follow @xcite , @xcite , and @xcite , and argue that forcing a system to treat wsd as a single-label classification problem and learn that only one sense is correct for a word in a given context does not reflect how human beings disambiguate text. in contrast to recent work, we approach wsd as a soft multi-label classification problem in which multiple senses can be assigned to each target word. we show that not only does this simple method bring significant improvements at low or no additional cost in terms of training and inference times and number of trainable parameters, but it can also be seamlessly extended to integrate senses from relational knowledge in structured form, e.g., similarity, hypernymy and hyponymy relations from semantic networks such as wordnet @xcite and babelnet @xcite . while structured knowledge has been naturally utilized by graph-based algorithms for wsd @xcite @xcite , the incorporation of such information into neural approaches has recently been garnering significant attention. however, currently available models can only take advantage of this knowledge with purposely-built layers @xcite ) that require additional complexities and/or trainable parameters. to the best of our knowledge, the work presented in this paper is the first to integrate structured knowledge into a neural architecture at negligible cost in terms of training time and number of parameters, while at the same time attaining state-of-the-art results in english all-words wsd. ## method single-label vs multi-label. wsd is the task of selecting the best-fitting sense s among the possible senses s w of a target word w in a given context @xmath0, w 2 , . . . , w n , where s w is a subset of a predefined sense inventory s. abstracting away from the intricacies of any particular supervised model for wsd, the output of a wsd system provides a probability y i for each sense s @xmath1. recently proposed machine learning models - @xcite @xcite , inter alia -are trained to maximize the probability of the single most appropriate sense ŝ by minimizing the cross-entropy loss l ce : we observe that this loss function is only suitable for single-label classification problems. in the case of wsd, this is equivalent to assuming that there is just a single appropriate sense ŝ ∈ s w for the target word w in the given context c, that is, ŝ is clearly dissimilar from any other sense in s w . indeed, minimizing the cross-entropy loss in order to maximize the probability of two or more senses generates conflicting training signals; at the same time, choosing to ignore one of the correct senses results in a loss of valuable information. since there is a not insignificant number of instances where multiple similar senses of the target word w fit the given context c (see section 1), we frame wsd as a multi-label classification problem in which a machine learning model is trained to predict whether a sense @xmath2, independently of the other senses in s w . this is simply equivalent to minimizing the binary cross-entropy loss l bce on the probabilities of the candidate senses s w : knowledge integration. if our model benefits from learning to assign multiple similar senses to a target word in a given context, then it makes sense that the very same model may also benefit from learning what related senses can be assigned to that word. for example, in the sentence "the quick brown fox jumps over the lazy dog", our model may formulate a better representation of fox if it is also trained to learn that any fox is a canine (hypernymy relation) or that the fox species includes arctic foxes, red foxes, and kit foxes (hyponymy relations). in this way, not only would the model learn that canines, foxes and arctic foxes are closely related, but it would also learn that canines and arctic foxes may have the ability to jump, and this could act as a data augmentation strategy especially for those senses that do not appear in the training set. there is a growing interest in injecting relational information from knowledge bases into neural networks but, so far, recent attempts have required purposely-designed strategies or layers. among others, @xcite aid their model with a gloss encoder that uses the wordnet graph structure; @xcite adopt a preprocessing strategy aimed at clustering related senses to decrease the number of output classes; bevilacqua and navigli (2020) introduce a logit aggregation layer that takes into account the neighboring meanings in the wordnet graph. in contrast, our multi-labeling approach to wsd can be seamlessly extended to integrate relational knowledge from semantic networks such as word-net without any increase in architectural complexity, training time, and number of trainable param-eters. we simply relax the definition of the set of possible senses s w for a word w to include all the senses related to a sense in s w . more formally, let @xmath3, r) be a semantic network where s is a sense inventory and r is the set of semantic connections between any two senses. then we define s + w to also include every sense s j that is connected to any sense s @xmath4, s j ) ∈ r, that is, s + @xmath5: (s i , s j ) ∈ r, s @xmath6. the loss function is updated accordingly to maximize not only the probability of the correct senses, but also the probability of their related senses: ## experiments and results experimental setup. we train our models in different configurations to assess the individual contribution of several factors. first, we compare our baseline model trained with a single-label objective (equation 1 ) to the same model trained with a multi-label objective (equation 2 ). then, we gradually include structured knowledge in the form of wordnet relations using equation 3, starting from similarity relations (similar-to, also-see, verbgroup, and derivationally-related-form), and incrementally including generalization and specification relations (hypernymy, hyponymy, instancehypernymy, instance-hyponymy). in order to keep a level playing field with single-label systems, we choose only the meaning with highest probability for our multi-label models. datasets. we evaluate the models on the unified evaluation framework for english all-words wsd proposed by @xcite . this evaluation includes five gold standard datasets, namely, senseval-2, senseval-3, semeval-2007, semeval-2013, and semeval-2015. following standard practice we use the smallest gold standard as our development set, semeval-2007, and the remaining ones as test sets. we distinguish between two settings: closed and open. in the former setting, we include systems that only use semcor @xcite as the training corpus, while in the latter we also include those systems that use wordnet glosses and examples and/or wikipedia. hyperparameters. we use the pretrained version of bert-large-cased @xcite available on huggingface's transformers library @xcite to build our contextualized embeddings (section 2). bert is left frozen, that is, its parameters are not updated during training. each model is trained for 25 epochs using adam (kingma and ba, 2015) with a learning rate of 10 -4 . we avoid hyperparameter tuning and opt for values that are close to the ones reported in the literature so as to have a fairer comparison. comparison systems. in order to have a comprehensive comparison with the current state of the art in wsd, we include the work of: • @xcite which was one of the first to propose a neural sequence model for wsd based on a stack of bilstm layers; the systems are divided into two groups in results. the first two rows of ## conclusion wsd is a key task in natural language understanding with several open challenges and with the granularity of sense inventories being undoubtedly the most pressing issue @xcite . we departed from recent work on wsd and investigated the effect of tackling the task as a multi-label classification problem. not only is our approach simple and model-agnostic, but it can also be seamlessly extended to integrate relational knowledge in structured form from semantic networks such as wordnet, and at no extra cost in terms of architectural complexity, training times, and number of parameters. our experiments show that our method, thanks to its more comprehensive notion of loss over equally valid and structurally-related senses, achieves stateof-the-art results in english all-words wsd, especially when there is a lower amount of annotated text available. these results open the path to further research in this direction, from explor-ing more complex models and richer knowledge bases to exploiting multiple labels in innovative disambiguation settings which can overcome the fine granularity of sense inventories. not only that, our knowledge integration approach could potentially be applied to address the knowledge acquisition bottleneck in multilingual wsd @xcite . finally, with the rise of ever more complex general and specialized pretrained models, we believe that our simple model-agnostic approach can be another step towards knowledgebased (self-)supervision. we release our software and model checkpoints at https://github.com/sapienzanlp/ multilabel-wsd .