Integrating Personalized P age R ank into Neural Word Sense Disambiguation (2021)

Abstract: Neural Word Sense Disambiguation (WSD) has recently been shown to benefit from the incorporation of pre-existing knowledge, such as that coming from the WordNet graph. However, state-of-the-art approaches have been successful in exploiting only the local structure of the graph, with only close neighbors of a given synset influencing the prediction. In this work, we improve a classification model by recomputing logits as a function of both the vanilla independently produced logits and the global WordNet graph. We achieve this by incorporating an online neural approximated PageRank, which enables us to refine edge weights as well. This method exploits the global graph structure while keeping space requirements linear in the number of edges. We obtain strong improvements, matching the current state of the art. Code is available at https://github.com/SapienzaNLP/neural-pagerank-wsd

## introduction word sense disambiguation (wsd) is a task with a long history in natural language processing @xcite . it addresses the pervasive phenomenon of (lexical) ambiguity, whereby the same polysemous word or expression conveys different meanings in different contexts. natural language processing (nlp) tends to maintain the working assumption that word meaning can be discretized in a finite number of classes, thus casting polysemy resolution as a multi-class classification problem, where the classes, i.e., the senses, are specific to a word. senses are registered in a dictionarylike resource called the sense inventory. in english wsd the sense inventory is virtually always word-net @xcite ), which, for example, lists separately the fish and musical instrument senses of the word "bass". the wsd task is currently dominated by supervised methods @xcite . these methods, thanks, inter alia, to the game-changing effect of pretrained language models, have widened the margin over so-called knowledge-based approaches @xcite @xcite , which usually disambiguate using only global graph information -a source of information that, however, is not easy to integrate explictly into supervised wsd. although there have been a few successful approaches integrating graphs into a standard neural classification architecture @xcite , such methods only exploit the local relational structure, leaving the global structure unused. thus, while the model is able to directly utilize the explicit knowledge that a cairn is a terrier, it is not able to also utilize the fact that, following the hyponymy relation, a cairn is also a dog. in this paper we propose a method for integrating global graph information into neural supervised wsd through a personalized pagerank @xcite approximation, blurring the distinction between knowledge-based and supervised methods. we achieve this by generalizing the logit aggregation scheme used on top of a feedforward classifier by @xcite . our proposed method is simple and extensible, and could be adapted to work with other classifiers as well. we match the performance of the current state of the art in wsd @xcite while using only 35% (60m) of the trainable parameter count. when training on additional data, our technique outperforms the previous state of the art in that setting @xcite . ## method in this section we explain how our method, building on top of previous approaches to wsd. ## experiments setting we evaluate our proposed addition to the wsd task by employing it on top of the simple feedforward classifier baseline (taking as input frozen bert large embeddings) used by @xcite . we train the model with vanilla categorical cross-entropy. following bevilacqua and navigli (2020) we use sensembert @xcite and lmms @xcite embeddings to initialize the output embeddings (i.e., the last transformation matrix of the ffn) for, respectively, nominal and all other synsets. the evaluation measure is the f 1 score. we compare against ewiser, but also report results for other recent high-performing methods from the literature, both sequence-tagging @xcite and token-tagging @xcite . 1 data as usual in the wsd literature, we train our model on semcor @xcite , i.e. the largest available manually semantically annotated corpus; following @xcite we also add synthetic instances built by prepending lemmas to the corresponding wordnet synset definition -thus injecting gloss information that is used by other state-of-the-art models (huang @xcite , senseval-3 @xcite , @xcite , semeval-2013 @xcite , and semeval-2015 @xcite . we use semeval-2007 as our development set, and report results on the concatenation of all other datasets in the framework (all -); following common practice, we also report results on semeval-2007 + all -(all), even though the former is the development set. graph we train models with various values of the power iteration parameter k: (i) @xmath0, which corresponds to the baseline with no logit aggregation scheme; (ii) @xmath1, which is similar to ewiser, but using α-weighted interpolation between original logits and aggregated ones; (iii) @xmath2, which corresponds to a range greater than or equal to the length of around 99% of paths between any two connected nodes in the graph. we report the path length statistics in we build a by including different sets of edges from wordnet, i.e., hyponymy, hypernyms, similarity, derivationally related, and verb group. the weight a s ,s from synset s to s is initialized as the edge s , s is in the graph. additionally, we experiment with including edges from syntagnet @xcite , a resource that includes edges representing semantic collocations. 4 collocational information is orthogonal to that contained in wordnet, providing paths between regions of the wordnet graph that would otherwise be distant or disconnected. finally, to check whether it is feasible to include global information by precomputing the ppr instead of approximating it in the network forward pass, we experiment with a baseline approach where we directly initialize a t s with a ppr distribution, built using wordnet as the starting graph and a one-hot vector z (with z @xmath3. to keep a manageable, we cut the distribution in a t s the top 10 ranks and renormalize to sum to 1. we then train a @xmath4. hyperparameters models are trained for 30 epochs, using early stopping with patience set to 5, feeding data in batches of 128 sentences. the optimizer used is @xcite with a learning rate of 10 -4 which decays linearly to 10 -7 . we set the teleport probability α to 0.15, a common default value. we do not tune k, nor 4 http://syntagnet.org/ other hyperparameters, which we take from the configuration of @xcite . the development set is only used for early stopping and to select the best epoch of the run. ## results we report results of our experiments in when adding wntg to the training corpus, the same trend of increasing performance along with increasing k appears. moreover, our best model ## conclusion in this paper we have shown that a deeper integration between supervised and knowledge-based methods in wsd can be attained. indeed, by using the standard logits as personalization vector for an approximated neural pagerank, a supervised method can exploit not just local graph information, but global information as well. thanks to our technique, we are able to match the best competitor system when training only on semcor @xcite , while using a simpler model. when we concatenate the wntg training corpus our results outperform the best competitor @xcite ) by 0.5 on the overall evaluation. we leave it as future work to ascertain whether i) edge label information can be incorporated too, and ii) stronger baseline models using glosses @xcite ) can benefit from the use of our method.