Rare and Zero-shot Word Sense Disambiguation using Z -Reweighting (2022)

Abstract: Word sense disambiguation (WSD) is a crucial problem in the natural language processing (NLP) community. Current methods achieve decent performance by utilizing supervised learning and large pre-trained language models. However, the imbalanced training dataset leads to poor performance on rare senses and zero-shot senses. There are more training instances and senses for words with top frequency ranks than those with low frequency ranks in the training dataset. We investigate the statistical relation between word frequency rank and word sense number distribution. Based on the relation, we propose a Z-reweighting method on the word level to adjust the training on the imbalanced dataset. The experiments show that the Z-reweighting strategy achieves performance gain on the standard English all words WSD benchmark. Moreover, the strategy can help models generalize better on rare and zero-shot senses.

## introduction word sense disambiguation (wsd) has been a longstanding problem in natural language processing community. the task can benefit many downstream applications @xcite , including but not limited to machine translation @xcite and information retrieval @xcite . the goal of the wsd task is to disambiguate word senses given contexts. for example, the word "lift" in the context "lift a load" and "the detective carefully lifted some fingerprints from the table" has different meanings. the former one means "raise from a lower to a higher position" and the latter one means "remove from a surface". from semantic recognition of human being, the former sense is easier to disambiguate as it is the most common sense of the word while the latter one is a relatively rare one. a skewed distribution exists in semcor @xcite , a commonly used human-labeled dataset for the wsd task, where most common senses have many training examples while rare senses have much fewer examples. a large coverage of senses are not accompanied with training examples, which are called zero-shot senses. many deep neural-networks-based methods are affected by this imbalanced training corpora @xcite . previous approaches attempt to address this problem by designing a new dataset or task specifically for the rare senses and zero-shot senses @xcite @xcite or enriching the sense embeddings by incorporating external lexical knowledge @xcite @xcite . different from these methods, we address the unbalanced training issue from the perspective of adjusting the learning process. an interesting human language phenomenon is that it follows a statistical distribution described by zipf's law @xcite , which also exists in many corpora including semcor. from the linguistic perspective, an explanation for zipf's law is that people tend to use more common words to minimize the communication effort @xcite . inspired by this, we consider a word with top rank in frequency should be assigned high training weight. from the statistical perspective, two laws have been proposed to explain zipf's law in word frequency, namely the meaning-frequency law @xcite and zipf's law of abbreviation @xcite . the meaning-frequency law proposes that more frequent words have larger number of word senses, which we also denote as larger word #sense. based on this, we calculate the word #sense distribution in semcor and use a mathematical function to fit the relation between word rank and word #sense. based on the relation, we design the z-reweighting strategy on the word level to help models generalize better to rare and zero-shot senses. to the best of our knowledge, we are the first to leverage linguistic distribution to address the training bias on the wsd task. our method improves the generalization ability of deep neural models on rare senses and zero-shot senses. results on all english words wsd evaluation benchmarks show that our system achieves improvement on rare and zero-shot senses by 2.1% and 3.6% on f1 score. furthermore, our strategy outperforms the system without any reweighting strategy and achieves a performance gain on the f1 score on all senses. we open source our code. foot_0 2 related works ## distribution analysis in semcor in this section, we first show the overall word and sense distribution in semcor foot_1 @xcite . since we propose to utilize the word #sense distribution as the basis for the z-reweighting strategy, we further look into the relationship between word rank, frequency, and word #sense. ## algorithms in this section, we first introduce the terminology for the wsd task. then we illustrate our z-reweighting strategy on adjusting the training loss for the imbalanced training dataset. ## results in this section, we first analyze the overall performance on the test datasets using different training strategies. then the details of improvement on mcs, lcs, and zero-shot senses for word groups are presented. finally, we analyze the influences brought by hyper-parameters in z-reweighting and influences brought by backbone models. ## conclusion in this paper, we address the problem in learning imbalanced training dataset on the wsd task. words with top frequency rank have more senses to disambiguate both for mcs and lcs. we assume these words should be assigned larger weights during training. specifically, we use a mathematical function to fit the relation between word rank and word #sense, and utilize smoothed #sense to design the z-reweighting strategy for all words english wsd task. the strategy leads to improvement on the performance of lcs and zero-shot senses on standard english wsd evaluation benchmarks. furthermore, our method achieves performance gain on the f1 score for all senses. the results demonstrate the effectiveness of our methods.