Word Sense Disambiguation Based on Iterative Activation Spreading with Contextual Embeddings for Sense Matching (2023)

Abstract: Many knowledge-based solutions were proposed to solve Word Sense Disambiguation (WSD) problem with limited annotated resources. Such WSD algorithms are able to cover very large sense repositories, but still being outperformed by supervised ones on benchmark data. In this paper, we start with analysis identifying key properties and issues in application of spreading activation algorithms in knowledge-based WSD, e.g. influence of the network local structures, interaction with context information and sense frequency. Taking our observations as a point of departure, we introduce a novel solution with new context-to-sense matching using BERT embeddings, iterative parallel spreading activation function and selective sense alignment using contextual BERT embeddings. The proposed solution obtains performance beyond the state-of-the-art for the contemporary knowledge-based WSD approaches for both English and Polish data.

## introduction contextual neural embeddings have strongly influenced word sense disambiguation (henceforth wsd), and resulted in extraordinary improvement on benchmark wsd datasets. however, the vast majority of such approaches follow the supervised learning scheme. thus, they suffer from the lack of annotated data, especially sparse for wsd, and their coverage, i.e. practical applicability, is limited only to a subset of word senses. moreover, they express bias towards most frequent senses. many knowledge-based solutions (i.e. weakly supervised) were proposed to solve the problem of limited sense annotated corpora. they are able to cover very large sense repositories, but still being outperformed by supervised ones on benchmark data. knowledge-based wsd algorithms were initially based on spreading activation scheme, most on personalised pagerank algorithm (ppr) @xcite . pagerank @xcite was originally proposed for modelling the web, a highly connected network with many hubs and loops.as we show in figure 1 , ppr scores are often strangely biased by some local wordnet structures. knowledge-based wsd approaches interact with the contextual information in a rather shallow way and also are biased by sense frequencies. that is why, we wanted to develop a version spreading activation for wsd which better reflects wordnet structures and more deeply explores context representation by using contextual text embeddings. our goal was to develop a novel knowledge-based wsd algorithm which combines context-to-sense matching informed by bert embeddings @xcite with a new iterative parallel spreading activation to process the wordnet. the main contributions of our paper are: the proposed solution expressed performance beyond the state-of-the-art of the knowledge-based wsd approaches in the all-words tasks for english. in addition, we performed also tests on wsd test data for polish, a language that is significantly different from english, equiped with a very large wordnet -plwordnet @xcite . our solution showed superior performance in comparison to the previous approaches on the polish data. ## related work lesk-like @xcite methods use information about wordnet graph structures to a very little extent, e.g. @xcite , while local subgraphs are the primary tool for sense description, distinguishing senses in a wordnet, cf @xcite . the idea of better exploration of wordnet graphs for wsd appeared in several works, e.g. in @xcite . @xcite proposed personalised pagerank (ppr) algorithm which uses the princeton wordnet graph @xcite with the initial activation limited to nodes (synsets) correspond to the words from a textual context. the initial activation depends on contextual word frequencies. ppr became the core part of the ukb wsd system @xcite with wordnet enhanced by several semantic resources, including sense links derived from princeton word-net gloss corpus @xcite . ukb refers to sense frequency twice: in initial activation values (normalised together with the word frequency in the context) and finally as a kind of weights to the synset scores. ukb is freely available, but is sensitive to proper setting and selection of knowledge resources. @xcite showed that ukb if properly used is still a state-of-the-art knowledgebased wsd system. ukb achieves the best results in a mode called "w2w" (word-to-word), in which the wsd is restarted for each text word separately. this results in several times slower processing, than the standard mode in which all words in the context are disambiguated in one go. babelfy @xcite utilised spreading activation in a indirect way. it was entirely based on babelnet @xcite -a complex semantic resource originating from the automated merging wordnet and wikipedia 1 . due to the ba-belnet content, babelfy was able to disambiguate words and perform entity linking at the same time. semantic signatures introduce some generalisation, and the extraction of a "dense subgraph" results in a kind of topic related clustering. @xcite applied ppr on word-net structure, but significantly expanded it with syntagnet @xcite -a large, manually constructed resource of semantic sense collocations. the main limitation of wordnet-based spreading activation is the lack of topical relations between senses. adding semcor-derived sense links and connections from wordnet glosses partially resolves this issue. on the other hand the structure of the network becomes more complex. some solutions solve approach the problem by joining topic modelling with knowledge-based tech-1 https://en.wikipedia.org/ niques. for instance, @xcite collected a corpus related to words from the wsd test data sets and obtained latent dirichlet allocation (lda) topic model @xcite . the graph was expanded with extended wordnet. finally ppr was applied to the graph, where the initialisation was informed by the similarity of a node to the context. this complex approach requires construction of semantic resources focused on the datasets to be disambiguated. the idea of restricting disambiguation context to sense semantically related to the disambiguated words is also central for @xcite . the authors proposed a distributional representation of senses based on generating pseudo-documents from babelnet. for each sense, other sense nodes in short distance are retrieved and paths linking senses of the same lemma are searched for and used as pseudo-sentences of pseudo-documents for lemmas, next transformed by doc2vec @xcite into a distributional space. local disambiguation graphs are built sequentially from related sense nodes and next processed by ppr. local disambiguation graphs of related senses indirectly address topical homogeneity of word senses in a broader context. @xcite perceived the wsd problem as a constraint satisfaction problem and model it on the basis of evolutionary game theory. influence of words on senses by other words is weighted by distributional information. semantic similarity information is used to calculate the amount of compatibility among the selected senses. sense frequency from semcor is indirectly used during disambiguation. concerning polish language, the early work was built upon pagerank-based solutions such as ukb and its variations focused on wordnet expansions @xcite . however, the frequency distribution of senses was unrepresentative as the large-scale sense-annotated corpora were not available. ## pagerank in knowledge-based wsd a wordnet is a mixture of synset and sense relations: some of them directed, other not. many directed relations exist in one way, but other are in symmetric pairs. density of a wordnet graph is very diversified -it is not a densely connected graph in general, and many regions may be very sparse. for wsd, a wordnet is typically transformed into a graph with all relations being directed and symmetric and defined on the synset level, e.g. in ukb. a wordnet has generally a tree-like shape. its sparseness and shape change after adding links from external resources, e.g. sense links from the gloss corpus. nevertheless, the final expanded graph for wsd still is not as recursively connected as the web for which pagerank was designed. in the case of wsd we assume that from sense nodes activated for a given context activation should evenly spread along the links to senses that are likely to co-occur with them. activation amount passed to the next nodes should depend mainly on their semantic distance correlated with the number and types of links to be traversed. however, pagerank, modelling of a random walker, seems to work according to a different philosophy. in fig. 1 we have visualised activation scores (below the nodes) obtained with ppr in a simple graph resembling some local wordnet subgraph. the two initial nodes end with the very different final scores. moreover, we can observe increased activations of nodes located in the same distance from both seeds. this is in contradiction to our above assumptions and results from the recursive character of ppr, e.g. the v 0 high degree and its star-shape local subgraph influence the scoring. in wordnet-based networks such specific local subgraphs, e.g. hub nodes, result from hierarchical categorisation or network editing practices, and do not express node importance due to being 'cited'. some of post-processing ppr scoring may decrease such negative bias. however, we decided to completely change the way in which spreading activation is performed. in the following section we propose a non-recursive spreading scheme in which every source node independently broadcast activation that is gradually transmitted along the network paths. it is the path characteristic that matters for activation strength. the final activation of the nodes is the result of overlapping of independent waves crossing the network. ## fast spreading activation and contextual matching learning from the ppr analysis, but also literature, two aspects seem especially important for knowledge-based wsd. first, spreading activation should transmit support from contextually related senses to the senses of a disambiguated word. second, not all context words are equally informative for wsd -a good measure for contextual informativeness is needed. thus, we proposed a redesigned wsd process based on three main components: a knowledge-base is a graph @xmath0, e), where the vertices v are senses, and the edges e -semantic links encoded in an adjacency matrix a n ×n : a typical wsd knowledge graph is quite sparse. several fast graph traversal algorithms @xcite were proposed for sparse adjacency matrices. a simple sparse-matrix dense-vector or even sparse-matrix sparse-vector multiplications can be interpreted as a single traversal step over graph. this property was used in parallel versions of well known graph algorithms e.g. breadth first search (bfs) and quick graph traversal algorithms using gpus @xcite . more specifically, to design a parallel bfs with multiple independent searches one can use sparse-matrix sparse-matrix multiplication (spmspm) where the second matrix represents an initial seed of starting nodes. in this work we adapt spmspm to design a fast spreading activation algorithm for wsd. ## experiments in this section we report the setup and the results of our experimental part. ## conclusions we propose the parallel spreading activation with contextual sense matching (psa) method for knowledge-based, weakly supervised wsd. its core is a novel spreading activation algorithm that is based on the idea of iterative spreading of support from the context seed senses across the network. the activation comes to the candidate senses from different directions and can be combined into the final score according to a selected scheme. this spreading scheme seems to fit better to the charac-ter of the wordnet-based semantic networks. moreover, it allows for efficient implementation based on the multiplication of sparse matrices. the contextual sense matching function uses contextual embeddings for more accurate and selective information processing to avoid unnecessary mixing of all input signals from disambiguation context and reduce the impact of knowledge-base imperfections. we showed that two kinds of contextual information, namely informativeness of seed senses for the disambiguated word and association of the seed senses with the semantic dimensions of the context can be introduced into our spreading activation model on the basis of contextual embeddings, in our case we used bert for this purpose. it is worth to notice that our approach uses versatile, general neural language models, and does not require construction of any further wsd-specific text models. we provide the code and the data at https://gitlab.clarin-pl.eu/ knowledge-extraction/prototypes/wsd-psa .