Zhestyatsky at S em E val-2021 Task 2: R e LU over Cosine Similarity for BERT Fine-tuning (2021)

Abstract: This paper presents our contribution to SemEval-2021 Task 2: Multilingual and Cross-lingual Word-in-Context Disambiguation (MCL-WiC). Our experiments cover English (EN-EN) sub-track from the multilingual setting of the task. We experiment with several pre-trained language models and investigate an impact of different top-layers on fine-tuning. We find the combination of Cosine Similarity and ReLU activation leading to the most effective fine-tuning procedure. Our best model results in accuracy 92.7%, which is the fourth-best score in EN-EN sub-track.

## introduction the increasing progress in natural language processing is closely related with development of word representations. the context-independent word embeddings, such as word2vec @xcite and fasttext @xcite brought the idea of measuring the relatedness of the meanings as the distance between the vectors encoding them. the introduction of the methods of pre-training context dependent embeddings, such as elmo @xcite , ulmfit @xcite , and bert @xcite made the next crucial breakthrough overcoming the shortcomings of previous methods to encode the meaning. despite the fact that the primal objective of word embeddings is to encode the meaning of words, it is not obvious how to evaluate them directly. while common manner to examine the superiority of particular type of embeddings is to look at their performance on some downstream tasks, the more direct way to evaluate their ability to represent semantic is challenging. semeval-2021 task 2: multilingual and crosslingual word-in-context disambiguation (mcl-wic) @xcite presents a new framework to evaluate embeddings. in this paper we present our contribution for the task. we explore the potential of different pre-trained contextdependent embeddings based on pre-trained language models. we find that the cosine similarity can produce fruitful results when used for finetuning the weights of the pre-trained models, while adding linear layers to learn the similarity from the limited data leads to instant overfitting. ## background the traditional approach to evaluate the ability of embeddings to catch the meaning of words is word sense disambiguation (wsd) task @xcite . wsd is defined as classification problem, when a given word is classificated between its predefined senses. wsd by design comes with an important limitation, being connected directly with predefined sense inventories such as wordnet foot_0 @xcite . the word in context (wic) benchmark (pilehvar and camacho-collados, 2019) addresses these limitations. the task proposes a binary classification setting for english, when, given two sentences s i and s k and two words w i and w k in them, the system needs to decide whether the word w i in s i and w k in s k have same or different meanings. the main advantage of wic task is a possibility to expand its consideration to the languages that lack such sense inventories. mcl-wic extends the wic approach to new senses and new languages, covering data in five languages: arabic, chinese, english, french and russian. the task provides data of two types: in the multilingual setting one needs to predict the label to the pair of sentences in one language (ar-ar, zh-zh, en-en, fr-fr, ru-ru sub-tracks), in the cross-lingual setting the first sentence is in english and the second one is in one of the four other considered languages (en-ar, en-zh, en-fr, en-ru sub-tracks). after preliminary experiments we decided to focus our efforts on the only sub-track with training data, namely the english sub-track from the multilingual setting. our solution foot_1 is fourth placed in the en-en leaderboard with 92.7% accuracy and is 0.6% behind the winner. ## system overview approaching the task we conduct multiple experiments with a variety of architectures, however all of them are deeply based on contextual embeddings fine-tuning. for our experiments we use pre-trained embeddings from bert and xlm-roberta @xcite ) models and finetune them for our task. ## experimental setup in our setup we mix train and development data and split it randomly by unique lemmas in proportion 97.5% to 2.5%. having 14680 samples in the first chunk and and 386 samples in the second chunk we use the first chunk for training and the second for validation. during training, the data is processed by batches of size 8. each sentence is split into 118 tokens maximum. in this way it is guaranteed that the longest sentence in the dataset is not going to be truncated. experiments with four different types of embeddings are conducted foot_3 : we train our models for a maximum of 8 epochs and define an early stopping criteria. every half of epoch (after training on the half of all the batches) we check if the loss on validation dataset is decreasing. if the loss does not decrease for 2 checks in a row, we stop training. in all our experiments we use binary cross entropy loss as the loss function and adamw optimizer with a learning rate set to 1e-5. to conduct experiments we use version 1.7.1 of pytorch @xcite together with version 0.8.2 of torchvision 5 and version 0.8.1 of torchtext 6 , version 1.1.6 of pytorch lightning foot_6 framework and version 4.2.2 of huggingface's transformers @xcite . from the latter we obtain bert and xlm-roberta model implementations. as we define a probability threshold as a hyperparameter in cosine similarity approach, we provide its values for all experimental configurations in the ## results in the it is seen by the number of the passed epochs before the early stopping. as @xcite token is designed to accumulate sentence meaning we expected it to make the representations for each instance in a pair more complete. the results in the pre-trained language models, like bert and xlm-roberta, have the property of associating close vectors with similar words. therefore to provide a baseline for the model described in cosine similarity approach we measure the accuracy of it without additional fine-tuning. due to the technique used to evaluate the probability thresholds, the accuracies for configurations with different activations are identical in this case. accuracies for dif- finally, to provide a convenient report we conduct an experiment with our best model (using bert-largecased embeddings together with cosine similarity architecture, using relu activation), which only uses data provided by organisers. we perform no further processing with the data and use it as is: train dataset is used for training and development dataset for validation. being trained for 4 epochs the model in the experiment demonstrates 0.886 accuracy on validation dataset and 0.913 accuracy on test dataset. this result shows that using additional data leads to better performance. ## error analysis our best model leads to accuracy 92.7%. it means that our model has erroneously labeled 73 sentences in the 1000-sentence testset. the error analysis revealed that our model is not biased towards one or another class, it produced 37 false negative predictions and 36 false positives predictions. the next observation is related to the construction feature of the dataset. the dataset is organized in the following manner: for each combination of lemma and pos-tag there are two instances in the dataset. all three possible combinations of labels are presented, with prevalent case when one pair is labeled false and second true. the peculiarity of the dataset is that both instances have the same first sentence. we found that 20 out of 73 errors have these repeating first sentence. in other words, if the model produces incorrect prediction for one instance for lemma it tends to make a mistake for the second instance in the dataset. due to the described peculiarity of the data, we can not speculate that certain lemma is a stumbling block for the model or it is just a context of the first sentence, that for example differs by genre or thematically from second sentence and complicates the prediction. the manual analysis of the errors has not revealed instances that could be considered hard and unclear for human assessment. in order to reveal objectively hard instances among the errors of the best model, we have intersected the mislabeled pairs for all the models fine-tuned with cosine similarity. the intersection indicated that all but two instances were predicted correctly by at least one of the models. we can conclude that no objectively hard instances were presented in the erroneously labeled pairs by the best model. additionally, the possible conclusion could be that an ensemble of our models could result in even more powerful solution for the task. ## conclusion we have provided an overview of different approaches to fine-tune pre-trained language models for the task that is naturally suitable for them -detecting the distance between representations of the words. we have showed that, for the data of given amount and type, learning distance between words in context with multilayer perceptron neural network is not applicable and generally leads to overfitting. using cosine similarity to predict probability during pre-trained embeddings fine-tuning leads to much more promising results, when activated with relu layer.