Semantic Specialization for Knowledge-based Word Sense Disambiguation (2023)

Abstract: A promising approach for knowledge-based Word Sense Disambiguation (WSD) is to select the sense whose contextualized embeddings computed for its definition sentence are closest to those computed for a target word in a given sentence. This approach relies on the similarity of the sense and context embeddings computed by a pre-trained language model. We propose a semantic specialization for WSD where contextualized embeddings are adapted to the WSD task using solely lexical knowledge. The key idea is, for a given sense, to bring semantically related senses and contexts closer and send different/unrelated senses farther away. We realize this idea as the joint optimization of the Attract-Repel objective for sense pairs and the self-training objective for context-sense pairs while controlling deviations from the original embeddings. The proposed method outperformed previous studies that adapt contextualized embeddings. It achieved state-of-the-art performance on knowledge-based WSD when combined with the reranking heuristic that uses the sense inventory. We found that the similarity characteristics of specialized embeddings conform to the key idea. We also found that the (dis)similarity of embeddings between the related/different/unrelated senses correlates well with the performance of WSD.

## introduction word sense disambiguation (wsd) is the task of choosing the appropriate sense of a word from a given sense inventory using contextual information. wsd has proven its usefulness for information retrieval @xcite and machine translation @xcite . a series of extensive studies has led supervised wsd task performance to surpass the milestone of 80% accuracy @xcite , which is the estimated human performance @xcite . in contrast, the goal of this study is knowledgebased wsd: a variant of wsd that does not rely on supervision data but only on lexical knowledge (e.g., word ontology). this task setting is practically appealing because it does not use a corpus with sense annotations @xcite , which is costly and labor-intensive to prepare. a promising approach is based on similarity: to select the sense that is the nearest to a target word in the embedding space @xcite . specifically, a pre-trained language model, typically bert @xcite , is used to compute sense embeddings for definition sentences. similarly, a target word is encoded into a context embedding for a given sentence. then, the model predicts the sense of the target word by finding the most similar sense embedding to the context. the inherent challenge of the similarity-based approach is how we associate two different representations of word meanings, either by definition sentences or by words in context. although the bert embeddings capture the coarse-grained word meanings @xcite , there should be room for improvement. notably, @xcite proposed sref, sense embedding adaptation by bringing semantically related senses closer. extending their work, @xcite proposed coe, context embedding enhancement heuristics during inference using the document-level global contexts of the given sentence, and reported the best performance. despite being effective, coe cannot be applied to standalone texts, e.g., short messages on social media or search queries, limiting its applicability. our study aims to improve both accuracy and applicability to stand-alone texts. specifically, we propose an adaptation method of the sense and context embeddings for the wsd task solely using lexical knowledge. then, what are good embeddings for wsd? our key idea is to 1) bring semantically related sense and context embeddings that convey the same meaning closer, and 2) send unrelated and/or different senses that share the same surface form farther away (fig. 1-d ). we formulate the idea as the attract-repel objective and selftraining objective. the main novelty is the joint optimization to utilize their complementary nature: the former should improve the distinguishability between senses whereas the latter offers pseudo signals of context-sense associations, which has not been explored in previous methods. the attract-repel objective, inspired by @xcite , injects semantic relation knowledge into the similarity of sense pairs. specifically, we make semantically related senses more similar while making different and unrelated senses more dissimilar (fig. 1-a ). while sref performs attract only, our method utilizes both attract and repel. the self-training objective, inspired by the idea of retraining on the classifier's own predictions instead of annotated senses @xcite , updates the similarity of context-sense pairs in a pseudo labeling manner ( § 6.1). specifically, for each training step and given context, we bring the nearest neighbor sense among candidates closer (fig. 1b ). we also impose distance constraints during adaptation to control the deviation from bert embeddings (fig. 1-c ) because excessive deviation may cause an inaccurate nearest neighbor sense selection, which would cause a performance drop. we call the overall proposed method ss-wsd, semantic specialization for wsd, following @xcite . we evaluated ss-wsd using the standard evaluation protocol @xcite and confirmed that it outperforms the previous embeddings adaptation method. furthermore, it achieved state-of-the-art (sota) performance when combined with the reranking heuristic that uses a sense inventory @xcite , and thus is applicable to stand-alone texts. the contributions of our study are as follows: ## experimental results in the intrinsic configuration, ss-wsd emb outperformed sref emb by 3.9pt, which is as much as a 9.3pt improvement over plainbert. looking at the results for each part-of-speech, we observed the largest improvement over sref emb for verbs (9.0pt). this result reflects the fact that verbs have the richer supervision signal for the attract-repel objective because of the largest number of related and different senses ( in the with heuristics configuration, ss-wsd kb outperformed coe by 0.8pt without using the cie heuristic, which shows an advantage over the baselines regardless of whether the evaluation sentence is a stand-alone text or in a document. the improvement brought by tam was 2.2pt. although ss-wsd kb lagged behind coe on the se07 (semeval-2007) subset, we think this result is understandable because coe also used se07 for hyperparameter optimization. when compared to supervised methods, ss-wsd emb outperformed sup-knn by 1.4pt, while falling behind bem by 4.1pt. the results indicate that the proposed method associates contexts with senses more precisely than the example-based sense embeddings computation using sense-annotated contexts. it also shows the effectiveness of the supervised fine-tuning of bert encoders in bem, as evidenced through their ablation study @xcite . ## conclusion in this paper, we proposed ss-wsd: semantic specialization for wsd 14 . the proposed method learns how to adapt bert embeddings by transformation and uses the semantic relation knowledge as a supervision signal. the key idea is the de14 the source code is available at: https://github.com/ s-mizuki-nlp/semantic_specialization_for_wsd sired characteristics of similarities: bringing related senses and the contexts closer while unrelated senses and different senses farther away. we realized it as the joint optimization of the attract-repel and self-training objectives while preventing large deviations from original embeddings. experiments showed that the proposed method outperformed the previous embedding adaptation method. when combined with the reranking heuristic that can be applied to stand-alone texts, it established a new sota performance on knowledge-based wsd. the proposed method performs well regardless of the availability of global contexts beyond the target sentence during inference, which the previous study did not achieve. several analyses showed the effectiveness of the objectives and constraints introduced for specialization. we also found that the closeness of semantically related/different/unrelated senses relative to the closeness of correct context-sense pairs positively correlates with the wsd task performance.