Breaking Through the 80% Glass Ceiling: R aising the State of the Art in Word Sense Disambiguation by Incorporating Knowledge Graph Information (2020)

Abstract: Neural architectures are the current state of the art in Word Sense Disambiguation (WSD). However, they make limited use of the vast amount of relational information encoded in Lexical Knowledge Bases (LKB). We present Enhanced WSD Integrating Synset Embeddings and Relations (EWISER), a neural supervised architecture that is able to tap into this wealth of knowledge by embedding information from the LKB graph within the neural architecture, and to exploit pretrained synset embeddings, enabling the network to predict synsets that are not in the training set. As a result, we set a new state of the art on almost all the evaluation settings considered, also breaking through, for the first time, the 80% ceiling on the concatenation of all the standard all-words English WSD evaluation benchmarks. On multilingual all-words WSD, we report state-of-the-art results by training on nothing but English.

## introduction there is a growing body of research dealing with the integration of prior knowledge into neural networks for natural language processing (nlp) tasks, be it through pretraining on self-supervised tasks such as language modeling @xcite , or through the incorporation of information from knowledge bases @xcite . in word sense disambiguation (wsd), i.e., the task of associating a word in context with the most appropriate meaning from a finite set of possible choices @xcite , the gap between supervision and knowledge @xcite has been overcome by several efforts directed at learning effective vector representations @xcite in the same space as contextualized embeddings, and exploring the usage of definitional knowledge in supervised sequence learning neural architectures @xcite @xcite . however, the lexical knowledge bases (lkbs) from which such information is retrieved, such as wordnet @xcite and babelnet @xcite , also provide a great wealth of relational knowledge in structured form (i.e., hypernymy, meronymy, similarity, etc.), which is often neglected due to the non-trivial integration of data of this kind into neural architectures. even though such information can, instead, be exploited by knowledge-based wsd algorithms @xcite , rivaling supervised pre-contextualized embedding approaches @xcite , the performances still lag behind @xcite . building on extended wsd integrating sense embeddings (ewise) @xcite , a neural wsd system incorporating prior knowledge through synset embeddings, we present enhanced wsd integrating synset embeddings and relations (ewiser), a hybrid knowledge-based and supervised approach to wsd that integrates explicit relational information from the wordnet lkb. our approach offers the following contributions: our approach is simple and extensible, does not require fine tuning of contextualized embeddings, and has a very modest parameter budget apart from synset embeddings. ewiser achieves a new state of the art in all-words english wsd. moreover, we obtain state-of-the-art performances on the crosslingual all-words wsd evaluation, without using non-english training data. contextualized vectors can be exploited in token tagging architectures @xcite @xcite . however, purely supervised systems are dependent on the data they are trained on, therefore when some sense is underrepresented in the training corpus it is not easy for them to predict it. lkbs in supervised wsd more closely related to the core of our contribution, lkb information, such as natural language definitions of word meaning, can be exploited in neural token tagging architectures. for example, in glossbert @xcite a pretrained bert encoder is fed both the context sentence and the gloss, and is trained to predict whether the gloss correctly describes the use of the target word. successful results have been obtained by encoding glosses in dense vectors @xcite . in ewise @xcite , wsd is performed in a two-step process: first, gloss embeddings are produced through a training procedure that also takes into account the wordnet's graph structure; then, the gloss embeddings are scored via dot product with a contextual vector computed with an lstm model, which is trained through regular categorical cross-entropy. our work builds on top of ewise in that it generalizes its sense vector dot product approach, but features a novel mechanism that injects relational knowledge into the architecture through a simple additional sparse dot product operation. moreover, we show that better performances can be obtained by training the output embedding matrix, and that different sense/synset vectors can be used to initialize the output embeddings. note that our approach is different from that of @xcite , in that we do not conflate senses together through the use of wordnet hypernymy; rather, we mantain all the original meaning distinctions, and exploit the logit scores over the full vocabulary in a second, distinct step. ## experiments we assess the performance of ewiser in allwords english wsd, against both a simple but competitive baseline, i.e., the simple feedforward network taking bert hidden states as input described in section 3.2, and state-of-art approaches. we first experiment separately on the integration of explicit relational information through structured logits (section 4.1), and the integration of synset embeddings through the initialization of o (section 4.2). then, building on the results of these experiments, we evaluate the full ewiser architecture (section 4.3). finally, we assess our approach on cross-lingual wsd (section 4.4), training on english and evaluating on french, german, italian and spanish. ## analysis in this section we provide a qualitative analysis of our approach. specifically, we are interested in the capability of the model to predict unseen synsets, thanks to the prior knowledge that is encoded in both the output embeddings o and the adjancency matrix a. consider the following sentences: in 1b ) of the best model trained on semcor. in the first row of each block, we report the scores of the four synsets associated in wordnet with the noun default. the following rows contain the scores for synsets that are incident to those in the first row of the block, and contribute to their scores in q. the columns report, from left to right, a sense (therefore synset) identifier, the number of occurrences of that lemma in semcor, the gloss, the weight of the edge, the hidden logits z and the output logits q. in both cases, the correct synsets, respectively, default.n.02/nonpayment.n.02 and default option.n.01, are not in the training set. however, the model is still able to give the correct answer. in the first case, the embedding intialization is enough to predict nonpayment.n.02 (with default.n.02 having the second highest score), as its score in z is already the highest among possible predictions. in the latter, it is the contribution from the synset pointing to default option.n.01, i.e., option.n.02, that enables the network to make the correct prediction. however, we must note that the model still overrelies on corpus supervision. because of this, even though our best overall model, i.e., ewiser hyper trained on semcor, tagged glosses and examples, is able to distinguish and predict correctly the two well-attested mathematical meanings of root as equation solution and root as the number x such that @xmath0, it is not able to correctly detect the tooth sense of root (2c), which never occurs in semcor: thus, while the ewiser model is indeed very effective, with the best configuration outdoing the upper bound on inter-annotator agreement, we are still far from having solved the task. ## conclusion we presented ewiser, a new neural wsd architecture that, by embedding information from the wordnet graph within the neural architecture, can also make use of the relational information that is usually only exploited by knowledge-based systems. thanks to the joint exploitation of the wordnet graph and to the use of pretrained synset embeddings, ewiser is able to predict meanings which are not found in the training set, thus mitigating the knowledge acquisition bottleneck. on almost all the evaluation settings, our system beats the previous state of the art. most notably, our model is the first to break through the 80 f1 ceiling on the overall evaluation, the estimated upper bound on the task. on the multilingual setting, even with no training data besides the english corpora, ewiser sets the new state of the art. we leave it as future work to explore ways to raise accuracy on unseen synsets without harming performances on frequent synsets. we release the code used in the experiments, as well as pretrained models at github.com/sapienzanlp/ewiser.