SMSM ix: Sense-Maintained Sentence Mixup for Word Sense Disambiguation (2022)

Abstract: Word Sense Disambiguation (WSD) is an NLP task aimed at determining the correct sense of a word in a sentence from discrete sense choices. Although current systems have attained unprecedented performances for such tasks, the nonuniform distribution of word senses during training generally results in systems performing poorly on rare senses. To this end, we consider data augmentation to increase the frequency of these least frequent senses (LFS) to reduce the distributional bias of senses during training. We propose Sense-Maintained Sentence Mixup (SMSMix), a novel word-level mixup method that maintains the sense of a target word. SMSMix smoothly blends two sentences using mask prediction while preserving the relevant span determined by saliency scores to maintain a specific word’s sense. To the best of our knowledge, this is the first attempt to apply mixup in NLP while preserving the meaning of a specific word. With extensive experiments, we validate that our augmentation method can effectively give more information about rare senses during training with maintained target sense label.

## introduction determining the meaning of a word in a particular sentence is a fundamental problem in natural language processing (nlp), as it can enable a better understanding of natural languages and help solve various nlp problems. word sense disambiguation (wsd) is a crucial task towards accomplishing this goal, where it involves choosing the most relevant meaning of a target word in context from predefined sense labels. like most other nlp tasks, the advancement of deep learning has led supervised learning of neural models to be the primary method of wsd @xcite @xcite . however, one of the biggest challenges of wsd is overcoming the data bias that naturally stems from the distributional bias of senses in language @xcite . because the dataset tends to have this bias toward certain senses, the wsd system shows high accuracy on the most frequent sense (mfs) and low accuracy on the least frequent sense (lfs) of a word. a common solution in machine learning to address data imbalance involves oversampling underrepresented categories, although merely doing so often leads to overfitting of minority classes @xcite . another approach uses data augmentation to drive the learning process toward a more suitable solution. due to the expensive cost of data collection, data augmentation has become one of the essential tools in modern deep learning, especially when dealing with low-resource tasks. in image classification, some approaches @xcite @xcite have been recently explored, involving the well-known data augmentation method mixup @xcite to alleviate data imbalance. this motivation encourages us to fill underpopulated areas in the training data of wsd -linked to least frequent senses (lfs) -with synthetic examples created through mixup (figure 1 ). in this work, we propose sense-maintained sentence mixup (smsmix), a novel mixup data augmentation method where the meaning of a specific word in a sentence is unaffected as much as possible while exposing it to various contexts. inspired by the recently proposed word-level mixup in nlp for text classification tasks @xcite , we first determine a span of text in a sentence containing a target word that is most relevant to maintaining the target word sense using gradient-based saliency scores. second, because @xcite mentions that mixup within training data cannot give more information to the model and acts merely as a regularizer, we inject the aforementioned span of text into a random sentence from a wikipedia corpus. lastly, when carrying out this injection, we consider smoothly blending the span to the wikipedia corpus sentence using mask prediction, inspired by smoothmix @xcite , where they consider smoothly mixing two images to solve the strong-edge problem. smsmix has empirically shown to be an effective augmentation method that can give more information about rare senses during training a wsd model in a supervised setting. we especially show that performing mixup with an external sentence (i.e., a sentence that is not from the training data) can outperform internal mixup within training data. also, by visualizing the latent vector of the target words in the augmented sentences, we show that smsmix effectively maintains the sense of the target word. ## conclusion this paper introduced a novel input-level mixup data augmentation scheme smsmix for improving the least frequent sense (lfs) data imbalance in the word sense disambiguation task. smsmix maintains the meaning of a specific word in a sentence by keeping the sense-maintained span using saliency score and smoothly injects the span into a different context using mask prediction. throughout the experiment, we show that instead of injecting the sense-maintained span with an internal training data sentence, injecting it into a random external corpus sentence allows the model to better improve the performance on lfs.