D eep M istake at LSCD iscovery: Can a Multilingual Word-in-Context Model Replace Human Annotators? (2022)

Abstract: In this paper we describe our solution of the LSCDiscovery shared task on Lexical Semantic Change Discovery (LSCD) in Spanish. Our solution employs a Word-in-Context (WiC) model, which is trained to determine if a particular word has the same meaning in two given contexts. We basically try to replicate the annotation of the dataset for the shared task, but replacing human annotators with a neural network. In the graded change discovery subtask, our solution has achieved the 2nd best result according to all metrics. In the main binary change detection subtask, our F1-score is 0.655 compared to 0.716 of the best submission, corresponding to the 5th place. However, in the optional sense gain detection subtask we have outperformed all other participants. During the post-evaluation experiments we compared different ways to prepare WiC data in Spanish for fine-tuning. We have found that it helps leaving only examples annotated as 1 (unrelated senses) and 4 (identical senses) rather than using 2x more examples including intermediate annotations.

## introduction given a list of words, a lexical semantic change detection (lscd) system applied to diachronic corpora shall determine how these words change their meaning over time. the lscdiscovery (d. @xcite shared task on lscd in spanish consists of two main subtasks and a few optional ones. in the graded change discovery subtask, the participants were asked to rank 4385 words according to the degree of their change. in the binary change detection subtask, it was necessary to develop a binary classifier that finds among 60 given words those that have either lost some old senses, or obtained some new ones. two optional binary subtasks required separately finding words with lost senses and words with new senses. in order to annotate the test set for the shared task, for each word from the test set some examples were sampled from the old and the new corpus. then human annotators were asked to annotate pairs of examples with scores from 1 to 4 according to the similarity of two occurrences of the same word by meaning. this kind of annotation is very similar to the word-in-context (wic) task, which asks a model to determine if two occurrences of the same word have the same or different meaning. ## conclusion this paper makes the first step towards answering the question in its title: can a multilingual word-incontext model replace human annotators for solving the lscd task? for now, it seems that our word-in-context model is not good enough to do that. however, we have shown that experimenting with the training data is a promising direction to achieve this goal.