Lotus at S em E val-2021 Task 2: Combination of BERT and Paraphrasing for E nglish Word Sense Disambiguation (2021)

Abstract: In this paper, we describe our proposed methods for the multilingual word-in-Context disambiguation task in SemEval-2021. In this task, systems should determine whether a word that occurs in two different sentences is used with the same meaning or not. We proposed several methods using a pre-trained BERT model. In two of them, we paraphrased sentences and add them as input to the BERT, and in one of them, we used WordNet to add some extra lexical information. We evaluated our proposed methods on test data in SemEval- 2021 task 2.

## introduction measuring semantic similarity is a task that is important in many applications such as summarization, plagiarism detection, etc. to measure the semantic similarity between two words or sentences, the words' meaning in their contexts should be understood. in "task 2: multilingual and crosslingual word-in-context disambiguation" of se-meval 2021 @xcite , systems should determine whether a word that occurs in two different sentences is used with the same meaning or not. the considered languages in the word-in-context disambiguation task of the evaluation are arabic, chinese, english, french, and russian @xcite . due to our internal limitation, we only evaluated the proposed methods on the english dataset. in most cases, humans can understand the correct meaning of each word by paying attention to the context of that word. this was the main reason for proposing the attention mechanism for machine translation @xcite . because attention plays an important role in bert topology @xcite , we believe it should work for the wsd task. however, in some cases, the two sentences may be about the same subject, while the target word has a different meaning. in these cases, find out that whether the target word has the same meaning in both sentences or not is difficult. to overcome this challenge, we need to add some extra information about the contexts to the input of the bert. we proposed four main methods for this task. in the first method, we use the bert model as a language representation model in which the inputs are the two sentences that come in a row and are separated by a "[sep]" token. furthermore, the target word is surrounded by a "[tgt]" token in both sentences. then, the first output of the last layer is used as a classifier to determine whether the word has the same meaning in the two sentences or not. in the second one, we added some information about the target word from the wordnet foot_0 to the end of each input and used the same strategy as the first method. in the third and fourth methods, we used the paraphrases of sentences as additional inputs to the bert. the remainder of the paper is organized as follows: in section 2, we provide a short literature review of the wsd task and mention some related works that used bert for this task. after that, in section 3, we introduce our methods in consequent sections: the first method that only uses the bert model is explained in section 3.1. the second method that uses the bert model and some additional information from wordnet is explained in section 3.2. the third and fourth methods that use the bert model and add paraphrased sentences as an additional input to the model are explained in section 3.3. then in section 4, the used datasets in our experiments are specified. finally, the results and discussion are presented in section 5. ## system overview in this section, we explain our proposed methods. the first method that relies only on the bert model is explained in section 3.1. this method is further improved in 3.2 by adding some extra information extracted from the wordnet to the bert's input. finally, in section 3.3 we explain an augmentation method by generating the paraphrases of both sentences and add them as an extra input to the bert. ## experimental setup as mentioned before, we used the mcl-wic training set, containing 8000 pair sentences in english as the training data, and the mcl-wic development set containing 1000 pair sentences in english as the validation data for our system. similarly, the mcl-wic test set containing 1000 pair sentences in english was used as the test data. the results displayed in ## results due to the random initialization of classifier layer weights, different results were achieved from different runs. so, we decided to run each experiment 5 times and report the average and standard deviation of model accuracy. on the other hand, adding more information from wordnet proposed in section 3.2 is not very helpful, and the accuracy is not changed compared to the base method proposed in section 3.1. this has happened because of the used way for adding this information to the input and the shape of information. for example, adding only the two first synsets or adding lemmas or glosses of synsets instead of their lexnames may cause better results. it is clear that there is a variation in the results of each method for different runs. the last column of the table shows the standard deviation of the five obtained results for each method. the second and third methods have smaller std that means these methods are robust to random initialization classifier's weights, and we can trust more to the results. in summary, adding generated paraphrases from sentences increase the performance of the model. however, using a better method for generating paraphrases can lead to better results.