BOS at S em E val-2020 Task 1: Word Sense Induction via Lexical Substitution for Lexical Semantic Change Detection (2020)

Abstract: SemEval-2020 Task 1 is devoted to detection of changes in word meaning over time. The first subtask raises a question if a particular word has acquired or lost any of its senses during the given time period. The second subtask requires estimating the change in frequencies of the word senses. We have submitted two solutions for both subtasks. The first solution performs word sense induction (WSI) first, then makes the decision based on the induced word senses. We extend the existing WSI method based on clustering of lexical substitutes generated with neural language models and adapt it to the task. The second solution exploits a well-known approach to semantic change detection, that includes building word2vec SGNS vectors, aligning them with Orthogonal Procrustes and calculating cosine distance between resulting vectors. While WSI-based solution performs better in Subtask 1, which requires binary decisions, the second solution outperforms it in Subtask 2 and obtains the 3rd best result in this subtask.

## introduction lexical semantic change detection (lscd) is a problem of detecting changes in word meaning over time. semeval-2020 task 1 @xcite suggests two variations of this problem. the first one (subtask 1) requires detection of words that have changed the set of their senses, either acquiring a new sense or loosing an old one between two given time periods. for this subtask we extend and adapt the state-of-the-art wsi method based on lexical substitution with neural language models proposed in @xcite . instead of the english bert employed by the original method, we use the recently introduced masked language model xlm-r @xcite , which was trained on all languages considered in the task. we adapt this model to the task datasets by additionally finetuning it to accept lemmatized texts as input and produce lemmatized substitutes. we also experiment with different dynamic patterns and their combinations, as well as multitoken substitutes generation. for subtask 2 we obtained better results with another approach based on sgns word embeddings and orthogonal procrustes alignment. ## related work in this section we describe those works, which our solutions are directly based on. please, refer to @xcite for a more detailed overview of the task and alternative approaches. subtask 1 raises the question if the set of senses of a particular word has changed. this task would be simple to solve, if we could identify word senses and corresponding word occurrences first. wsi is a task of clustering occurrences of an ambiguous word in accordance to its senses. one of our solutions employs a substitution-based approach @xcite @xcite , which exploits lexical substitutes to distinguish word senses. in the recently proposed state-of-the-art implementation of this approach @xcite , for each occurrence of an ambiguous word lexical substitutes (i.e. words that can replace the ambiguous word in a given context) are generated using bert masked language model (mlm) @xcite . then substitute vectors are built, which are tf-idf weighted bag-of-words vectors based not on the words in context, but on the generated substitutes. finally, these vectors are clustered using agglomerative clustering and the resulting clusters are taken as induced senses. xlm-r @xcite is an mlm trained similarly to the multilingual version of bert on texts in 100 languages (including four languages considered in the shared task), but having 3x more parameters, 2x larger vocabulary and trained not only on wikipedia, but also on common crawl dataset, which increased training data for low-resourced languages by orders of magnitude resulting in more than 2tb of data in total. xlm-r has shown better performance than multilingual bert in different nlp tasks. based on these advantages, we have decided to employ xlm-r for substitutes generation in our solution. subtask 2 raises the question of how much the frequency distribution of senses has changed for a particular word. this task is well explored and can be solved without explicit wsi by exploiting implicit features like the change in frequencies of word contexts. the surveys like @xcite , @xcite and @xcite present the comparison of different approaches to this task. most of these approaches, including @xcite , imply using skip-gram negative sampling (sgns) word representations and computing cosine distances between them. we employed one of such methods as our baseline solution. ## conclusion we proposed the wsi-based solution for the lexical semantic change detection problem. it outperforms the well-known sgns+op+cd baseline on subtask 1. however, the baseline performs better for subtask 2, where we obtained the 3rd best result. we have shown that finetuning of xlm-r on the lemmatized task corpora improves the final results for subtask 1. a comparison of different variants of our method is provided. however, a larger dataset is required to draw reliable conclusions from these observations.