MULTISEM at S em E val-2020 Task 3: Fine-tuning BERT for Lexical Meaning (2020)

Abstract: We present the MULTISEM systems submitted to SemEval 2020 Task 3: Graded Word Similarity in Context (GWSC). We experiment with injecting semantic knowledge into pre-trained BERT models through fine-tuning on lexical semantic tasks related to GWSC. We use existing semantically annotated datasets, and propose to approximate similarity through automatically generated lexical substitutes in context. We participate in both GWSC subtasks and address two languages, English and Finnish. Our best English models occupy the third and fourth positions in the ranking for the two subtasks. Performance is lower for the Finnish models which are mid-ranked in the respective subtasks, highlighting the important role of data availability for fine-tuning.

## introduction the meaning of words is strongly tied to the context in which they occur: different contexts might point to different senses or indicate subtler meaning nuances. semeval 2020 task 3 "graded word similarity in context" (gwsc) @xcite explores the effect of context on meaning, and proposes to predict the similarity of word instances in a continuous, or graded, fashion. gwsc is based on the cosimlex dataset @xcite and consists of two subtasks where models have to predict (1) the shift in meaning similarity for a pair of words (w a , w b ) occurring in different contexts, and (2) the similarity of two word instances in the same context. this is illustrated by sentences c 1 and c 2 , two contexts where dinner and breakfast co-occur. c 1 (...) after mickey rings the dinner bell, goofy foolishly leaves the driver's seat for breakfast. c 2 residence inns typically feature a complimentary small hot breakfast in the morning and a free light dinner or snack reception on weekday evenings (...) a change in meaning similarity occurs between the highlighted words in the two sentences: they are less similar in context c 1 where dinner is part of a noun compound (dinner bell), than in context c 2 where they describe different kinds of meals. the shift in meaning is reflected in the gold similarity scores assigned to these instance pairs in the gwsc dataset (4.39 vs. 5.35). we build models for these two subtasks by fine-tuning bert on existing lexical similarity datasets. additionally, we propose to approximate similarity of words in context through automatically generated lexical substitutes. we build and evaluate models in two languages, english and finnish. in subtask 1, our english and finnish models ranked third and sixth out of nine participants. in subtask 2, they are both found at the fourth position among ten participants. 1 2 background our methodology draws inspiration from recent work on injecting semantic information into pre-trained language models (lms). this can be done at two stages: during model pre-training or during fine-tuning. this work is licensed under a creative commons attribution 4.0 international licence. licence details: http:// creativecommons.org/licenses/by/4.0/ one of our tasks is inspired by the retrofitting approach of @xcite . this consists in gathering sentence pairs from the microsoft research paraphrase corpus (mrpc) @xcite ) that share a word and which are paraphrases of each other (t) or not (f). @xcite propose an orthogonal transformation for elmo @xcite that is trained to bring representations of word instances closer when they appear in meaning-equivalent contexts. they show that this retrofitting approach improves elmo's performance in a wide range of semantic tasks at the sentence level (sentiment analysis, inference and sentence relatedness). we follow their data collection method to obtain word instances for fine-tuning bert. we replace the mrpc with the opusparcus resource @xcite since it covers two of the languages addressed in gwsc, english and finnish. ## results results for the two english and finnish subtasks are presented in although the two subtasks are highly related, different models perform best in each one. for english, the best result in subtask 1 (among our official submissions) is obtained by the model fine-tuned on wic data with the cosdist head (ρ = 0.760) which occupies the third position in the final ranking. it is closely followed by the model fine-tuned on paraphrase data with the classif head. the best performing model in subtask 2 is the one fine-tuned on the ukwac-subs data (ρ = 0.718) which ranked fourth. the second best model uses the cosdist head and is trained on the coinco and usim data together. all english models outperform the bert-based baseline without fine-tuning (ρ = 0.715 and ρ = 0.661). this demonstrates the higher quality of lexical semantic knowledge in our fine-tuned models. best results for the finnish subtasks 1 and 2 are also produced by different models. the multilingual model performs better on subtask 1 and the finnish model on subtask 2. we observe that the multilingual model tends to assign very high similarities to all word instance pairs, which explains its low performance in subtask 2. at the same time, however, it does well on subtask 1 because it captures the magnitude of the difference in similarity between two pairs. given that no trial data (development set) are available for finnish and that the maximum number of submissions was nine, we could only try at most five layers per model at submission time. the models were ranked sixth and fourth in subtasks 1 and 2. during the post-evaluation phase, we had the possibility to test all layers of the models. the sixth layer of the multilingual model fine-tuned on finnish opusparcus data outperforms the multilingual baseline on subtask 1 (ρ = 0.718 vs ρ = 0.677), but the other fine-tuned models do not improve over their respective baselines. surprisingly, the finnish baseline model in subtask 2 (ρ = 0.671) outperforms the top-ranked model for finnish among all teams that participated in the task (ρ = 0.645). ## conclusion we have participated in the semeval task graded word similarity in context for english and finnish, with models integrating different notions of word similarity. we have specifically investigated the effect of fine-tuning pre-trained bert models on existing datasets that address word meaning similarity in context. furthermore, we have proposed a new fine-tuning task where in-context lexical similarity is approximated through automatic substitute annotations. our english models are ranked at the third and fourth position in the two gwsc subtasks, and outperform a bert-based baseline without fine-tuning. this demonstrates the benefit of fine-tuning bert on a task that is closely related to the end task, even when the data used for fine-tuning are automatically obtained. due to the scarcity of resources for finnish, we could only fine-tune models with paraphrases. the finnish models are mid-ranked among all participating systems.