Connect-the-Dots: Bridging Semantics between Words and Definitions via Aligning Word Sense Inventories (2021)

Abstract: Word Sense Disambiguation (WSD) aims to automatically identify the exact meaning of one word according to its context. Existing supervised models struggle to make correct predictions on rare word senses due to limited training data and can only select the best definition sentence from one predefined word sense inventory (e.g., WordNet). To address the data sparsity problem and generalize the model to be independent of one predefined inventory, we propose a gloss alignment algorithm that can align definition sentences (glosses) with the same meaning from different sense inventories to collect rich lexical knowledge. We then train a model to identify semantic equivalence between a target word in context and one of its glosses using these aligned inventories, which exhibits strong transfer capability to many WSD tasks. Experiments on benchmark datasets show that the proposed method improves predictions on both frequent and rare word senses, outperforming prior work by 1.2% on the All-Words WSD Task and 4.3% on the Low-Shot WSD Task. Evaluation on WiC Task also indicates that our method can better capture word meanings in context.

## introduction human language is inherently ambiguous since words can have various meanings in different contexts. word sense disambiguation (wsd) aims to automatically identify the correct sense (meaning) of the target word within a context sentence, which is essential to many downstream tasks such as machine translation and information extraction. recently, many approaches have achieved state-ofthe-art performance on wsd by fine-tuning language models pretrained with massive text data on task-specific datasets @xcite . however, fine-tuning a wsd model using taskspecific resources could limit its applicability and may cause two major problems. first, the performance of models decreases significantly when predicting on rare and zero-shot word senses @xcite @xcite because there are no sufficient supporting examples in training data. second, the trained models are often inventory-dependent which can only select the best definition from one predefined word sense inventory (mainly wordnet) that human annotations are based upon. in this paper, we overcome these limitations by leveraging abundant lexical knowledge from various word sense inventories. as we know, dictionaries that are compiled by experts contain rich sense knowledge of words. moreover, a dictionary usually provides several example sentences for each word sense to illustrate its usage, which can be viewed as context sentences of that word sense. since a word's sense (meaning) can be determined by its context, the word itself in a given context and the definition sentence corresponding to the correct sense are merely two surrogates of the same meaning (semantically equivalent). furthermore, we observe that different dictionaries normally summarize meanings of a word to a close number of word senses, where definition sentences (glosses) from different dictionaries are different expressions of the same bunch of meanings. for example, figure 1 lists glosses retrieved from three dictionaries for verb word search. we can see that glosses with the same color have the same meaning and can be aligned across different dictionaries. based on this observation, we propose a gloss alignment algorithm to leverage abundant lexical knowledge from various word sense inventories. we convert the problem of aligning two groups of glosses according to meanings to an optimization to make use of the derived semantic equivalence knowledge, we adopt a transfer learning approach that first pretrains a general semantic equivalence recognizer by contrasting the word representations in example sentences with the sentence representations of positive glosses or negative glosses. the general model can be directly applied to downstream wsd tasks or further fine-tuned on the taskspecific dataset to get an expert model. we test our two-stage transfer learning scheme on two wsd benchmark tasks, i.e., the standard task @xcite that focuses on all-words wsd and fews @xcite task that emphasizes low-shot (including few-shot and zero-shot) wsd. experimental results show that the general model (without fine-tuning) surpasses the supervised baseline by 13.1% on zero-shot word senses. after further fine-tuning with build-in training data, the expert model outperforms the previous state-ofthe-art model by 1.2% on all-words wsd tasks and 4.3% on low-shot wsd tasks. adding semantic equivalence knowledge to the word-in-context (wic) task (pilehvar and camacho-collados, 2019) also improves the accuracy of roberta large @xcite by 6%, which even outperforms the 9x larger t5 model @xcite . overall, the major contributions of our work are two-fold. 1) we propose a gloss alignment algorithm that can integrate lexical knowledge from different word sense inventories to train a general semantic equivalence recognizer. 2) without using task-specific training data, the general model not only performs well on overall word senses but demonstrates strong applicability to low-shot senses. the general model can turn into an expert model to achieve new state-of-the-art performance after further fine-tuning. ## overview of our approach figure 2 shows the overview of our approach. we first collect all word glosses and corresponding example sentences from six word sense inventories. we next apply the gloss alignment algorithm to find the best matching between two groups of glosses retrieved from two different inventories for every common keyword. by contrasting example sentences with the correct glosses and incorrect glosses within each inventory or across different inventories, we automatically gather rich supervision for pretraining a universal binary classifier that can determine whether the keyword in the context sentence (example sentence) and a gloss are semantically equivalent or not. the pretrained general model can be directly used in downstream wsd tasks or further fine-tuned to get an expert model. ## conclusion based on the observation that glosses of a word from different inventories usually are different expressions of a few meanings, we have proposed a gloss alignment algorithm that can unify different lexical resources as a whole to generate abundant semantic equivalence knowledge. the general model pretrained on derived equivalence knowledge can serve as a universal recognizer for word