Adapting BERT for Word Sense Disambiguation with Gloss Selection Objective and Example Sentences (2020)

Abstract: Domain adaptation or transfer learning using pre-trained language models such as BERT has proven to be an effective approach for many natural language processing tasks. In this work, we propose to formulate word sense disambiguation as a relevance ranking task, and fine-tune BERT on sequence-pair ranking task to select the most probable sense definition given a context sentence and a list of candidate sense definitions. We also introduce a data augmentation technique for WSD using existing example sentences from WordNet. Using the proposed training objective and data augmentation technique, our models are able to achieve state-of-the-art results on the English all-words benchmark datasets.

## introduction in natural language processing, word sense disambiguation (wsd) refers to the task of identifying the exact sense of an ambiguous word given the context @xcite . more specifically, wsd associates ambiguous words with predefined senses from an external sense inventory, e.g. wordnet @xcite and babelnet @xcite . recent studies in learning contextualized word representations from language models, e.g. elmo @xcite , bert @xcite and gpt-2 @xcite attempt to alleviate the issue of insufficient labeled data by first pre-training a language model on a large text corpus through self-supervised learning. the weights from the pre-trained language model can then be fine-tuned on downstream nlp tasks such as question answering and natural language inference. for wsd, pre-trained bert has been utilized in multiple ways with varying degrees of success. notably, 1 codes and pre-trained models are available at https: //github.com/bpyap/bert-wsd . @xcite proposed glossbert, a model based on fine-tuning bert on sequence-pair binary classification task, and achieved state-of-the-art results in terms of single model performance on several english all-words wsd benchmark datasets. in this paper, we extend the sequence-pair wsd model and propose a new task objective that can better exploit the inherent relationships within positive and negative sequence pairs. briefly, our contribution is two-fold: (1) we formulate wsd as gloss selection task, in which the model learns to select the best context-gloss pair from a group of related pairs; (2) we demonstrate how to make use of additional lexical resources, namely the example sentences from wordnet to further improve wsd performance. we fine-tune bert using the gloss selection objective on semcor @xcite plus additional training instances constructed from the wordnet example sentences and evaluate its impact on several commonly used benchmark datasets for english all-words wsd. experimental results show that the gloss selection objective can indeed improve wsd performance; and using wordnet example sentences as additional training data can offer further performance boost. ## related work bert @xcite is a language representation model based on multi-layer bidirectional transformer encoder @xcite . previous experiment results have showed that significant improvement can be achieved in many downstream nlp tasks through fine-tuning bert on those tasks. several methods have been proposed to apply bert for wsd. in this section, we briefly describe two commonly used approaches: featurebased and fine-tuning approach. ## methodology we describe the implementation details of our approaches in this section. when customizing bert for wsd, we use a linear layer consisting of just 1 neuron in the output layer to compute the relevance score for each context-gloss pair, in contrast to the binary classification layer used in glossbert. additionally, we also extract example sentences from wordnet 3.0 and use them as additional training data on top of the sense-annotated sentences from semcor. ## experiments in this section, we introduce the datasets and experiment settings used to fine-tune bert. we also present the evaluation results of each model and compare them against existing wsd systems. ## conclusion we proposed the gloss selection objective for supervised wsd, which formulates wsd as a relevance ranking task based on context-gloss pairs. our models fine-tuned on this objective outperform other non-ensemble systems on five english all-words benchmark datasets. furthermore, we demonstrate how to generate additional training data without external annotations using existing example sentences from wordnet, which provides extra performance boost and enable our single-model system to surpass the state-of-the-art ensemble system by a considerable margin on a number of benchmark datasets.