Semi-supervised Word Sense Disambiguation Using Example Similarity Graph (2020)

Abstract: Word Sense Disambiguation (WSD) is a well-known problem in the natural language processing. In recent years, there has been increasing interest in applying neural net-works and machine learning techniques to solve WSD problems. However, these previ-ous supervised approaches often suffer from the lack of manually sense-tagged exam-ples. In this paper, to solve these problems, we propose a semi-supervised WSD method using graph embeddings based learning method in order to make effective use of labeled and unlabeled examples. The results of the experiments show that the proposed method performs better than the previous semi-supervised WSD method. Moreover, the graph structure between examples is effective for WSD and it is effective to utilize a graph structure obtained by fine-tuning BERT in the proposed method.

## introduction in human languages, many words have multiple meanings, depending on the context in which they are used. identifying the sense of a polysemous word within a given context is a fundamental problem in natural language processing. for example, the english word "bank'' has different meanings as "a commercial bank" or "a land along the edge of a river,'' etc. word sense disambiguation (wsd) is the task of deciding the appropriate meaning of a target ambiguous word in its context @xcite . among various approaches to the wsd task used over the past two decades, a supervised learning approach has been the most successful. however, it is quite expensive in both time and cost to annotate a large amount of reliable training data because supervised wsd requires a large amount of manually labeled training examples to achieve good performance. unsupervised learning approach does not need labeled examples and uses large amount of unlabeled examples to find word clusters which discriminates the senses of the words in different clusters. unsupervised learning algorithms are typically less accurate than supervised algorithms because examples may not be assigned the correct sense. for these reasons, we focus on a semi-supervised learning method that uses both sense-labeled and unlabeled examples in different proportions. in this paper, we propose a semi-supervised wsd method using graph embeddings based learning method. in this method, we extract features from the context around the target word for each labelled and unlabeled example and construct a graph structure between labelled and unlabeled examples to obtain classifiers for every polysemous word. then, we construct classifiers for each polysemous word using planetoid @xcite , which is a multi-task framework for graph-based semi-supervised learning. by using the proposed method, it is possible to incorporate information obtained from unlabeled examples without assigning a sense label to unlabeled examples. moreover, by learning graph embeddings, it is possible to distinguish between two similar examples with different sense labels to construct a better classifier for wsd. to evaluate the efficiency of the proposed wsd method, we design some experiments using the semeval-2010 japanese wsd task data set and the senseval-2 english lexical sample task data set. the three contributions of this work can be summarized as follows: the rest of this paper is organized as follows. section 2 is devoted to presenting related works in the literature. section 3 describes the proposed semi-supervised wsd method. in section 4, we describe an outline of our experiments. in section 5, we present experimental results. finally, we conclude the paper in section 6. ## related works this section is a literature review of previous work on semi-supervised wsd and various related methods using a neural network. in recent years, there has been increasing interest in applying neural networks and machine learning techniques to solve wsd problems. @xcite employed a bidirectional long short-term memory (bi-lstm) to encode information of both preceding and succeeding words within the context of a target word. @xcite used an lstm language model to obtain a context representation from a context layer for the whole sentence containing a target word. the context representations were compared to the possible sense embeddings for the target word. then, the word sense whose embedding had maximal cosine similarity was assigned to classify a target word. @xcite considered wsd as a neural sequence labelling task and constructed a sequence learning model for all-words wsd. these approaches are characterized by their high performance, simplicity, and ability to extract a lot of information from raw text. in recent years, semi-supervised learning has been used in wsd tasks. semi-supervised learning is a technique that makes use of a small number of sense-labelled examples with a large amount of unlabeled examples. @xcite proposed a bootstrapping model that only has a small set of sense-labelled examples that gradually assigns appropriate senses to unlabeled examples. @xcite and @xcite proposed a semi-supervised wsd method to use word embeddings of surrounding words of the target word and showed that the performance of wsd could be increased by taking advantage of word embeddings. @xcite proposed a semi-supervised wsd method that automatically obtains reliable sense labelled examples using example sentences from the iwanami japanese dictionary to expand the labelled training data. then, this method employs a maximum entropy model to construct a wsd classifier for each target word using common morphological features (surrounding words and pos tags) and topic features. finally, the classifier for each target word predicts the sense of the test examples. they showed that this method is effective for the semeval-2010 japanese wsd task. @xcite proposed a graph-based semi-supervised wsd method using word embeddings and distance measures. although this method fixes the graph structure and trains only the classification model, our method trains the classification model and the graph jointly. some research in the field of wsd has taken advantage of graph-based approaches. @xcite ) proposed a label propagation-based semi-supervised learning algorithm for wsd, which combines labelled and unlabelled examples in the learning process. @xcite also introduced a label propagation (lp) for semi-supervised classification and lstm language model. an lp graph consists of vertices of examples and edges that represent semantic similarity. in this graph, label propagation algorithms can be efficiently used to apply sense labels to examples based on the annotation of their neighbours. in this paper, we use a semi-supervised learning method that incorporates knowledge from unlabeled examples by using graph embeddings based learning method. besides the semi-supervised learning, several recent methods have shown that combining supervised neural wsd systems with external knowledge base information improves the wsd performance. gloss-bert @xcite and bem @xcite ) combine supervised learning with knowledge from gloss information to make better performance. ewise @xcite incorporates both gloss embeddings and knowledge graph embeddings. ewiser @xcite incorporates both synset embeddings and wordnet relations instead of leveraging sense glosses. these methods represent the relationships between words as a graph structure, which means that words are represented as nodes and the relationships between words are represented as edges. in this study, we use a graph structure where the nodes are examples and the edges are semantic similarity between examples, which is different from the structure of these previous methods. ## wsd method using graph-based semi-supervised learning in this section, we describe the details of the proposed semi-supervised wsd method using graph embeddings based learning method. ## experiments to evaluate the efficiency of the proposed wsd method using graph embeddings based learning, we conducted some experiments to compare the results to the baseline system. in this section, we describe an outline of the experiments. ## experimental results among the wsd models using the graph structure based on the similarity between examples, the model using sentence pair classification by fine-tuning bert obtains the highest precision. therefore, it is effective for wsd to utilize a graph structure obtained by fine-tuning bert in the proposed method. in the to demonstrate the effectiveness of the proposed method, we compared it with a wsd model using word2vec embeddings as features. we obtain word embeddings of word2vec using nwjc2vec @xcite and perform experiments using the wsd model with 1000 dimensional vectors concatenated with 200-dimensional word embeddings for 5 words. experimental results showed that the average precision of the wsd model using word2vec is 79.6%. these results show that the proposed wsd model using contextual word embeddings with elmo is more effective than the model using word2vec. in the wsd model with word2vec, the average precision of the model with graph is 7.36% higher than that without graph. this indicates that the graph structure is also effective for the wsd method using word2vec. ## conclusion in this paper, we proposed a semi-supervised graph embeddings based learning method for the wsd task. the efficiency of the proposed method was evaluated on the semeval-2010 japanese wsd task and the senseval-2 english lexical sample task. experimental results show that the proposed method performs better than the previous semi-supervised wsd method. moreover, the graph structure between examples is effective for wsd and it is effective to utilize a graph structure obtained by fine-tuning bert in the proposed method. in the future, we would like to explore methods to construct an effective graph structure by using paraphrase information, and the dependency analysis technique, the effective filtering method for unlabeled data. in addition, we would like to develop a method to use the examples of the iwanami's japanese dictionary effectively.