G loss R eader at LSCD iscovery: Train to Select a Proper Gloss in E nglish – Discover Lexical Semantic Change in S panish (2022)

Abstract: The contextualized embeddings obtained from neural networks pre-trained as Language Models (LM) or Masked Language Models (MLM) are not well suitable for solving the Lexical Semantic Change Detection (LSCD) task because they are more sensitive to changes in word forms rather than word meaning, a property previously known as the word form bias or orthographic bias. Unlike many other NLP tasks, it is also not obvious how to fine-tune such models for LSCD. In order to conclude if there are any differences between senses of a particular word in two corpora, a human annotator or a system shall analyze many examples containing this word from both corpora. This makes annotation of LSCD datasets very labour-consuming. The existing LSCD datasets contain up to 100 words that are labeled according to their semantic change, which is hardly enough for fine-tuning. To solve these problems we fine-tune the XLM-R MLM as part of a gloss-based WSD system on a large WSD dataset in English. Then we employ zero-shot cross-lingual transferability of XLM-R to build the contextualized embeddings for examples in Spanish. In order to obtain the graded change score for each word, we calculate the average distance between our improved contextualized embeddings of its old and new occurrences. For the binary change detection subtask, we apply thresholding to the same scores. Our solution has shown the best results among all other participants in all subtasks except for the optional sense gain detection subtask.

## introduction lscdiscovery (d. @xcite is a shared task on lexical semantic change detection (lscd) in spanish. in general, lscd is the task of automatically analyzing differences between word senses in two corpora. in the shared task, these two corpora represent two time periods @xcite , and the participants are asked to analyze changes in the meaning of words over time, or diachronic change. there are two main subtasks in the shared task: graded change and binary change detection. in the first subtask, the participants are asked to rank a list of words according to the magnitude of change in the relative frequencies of their senses (measured by the jensen-shannon distance between the probability distributions over senses automatically inferred by the organizers from the pairwise human annotations). in the second subtask, for each given word the systems should detect if the sets of its senses appearing in the old and the new corpus are different, i.e. if any new senses have appeared or any old senses are not in use anymore. despite the success of recurrent and transformer-based neural networks pre-trained as language models (lm) or masked language models (mlm) on large corpora in a wide variety of nlp tasks, they cannot be applied to the lscd task in a standard way. most datasets used to fine-tune such models for different nlp tasks contain tens or hundreds of thousands examples, each of these examples is a text fragment not longer than several hundred words that contain all information required to make a correct prediction. in lscd one example is a word, however, inspecting many occurrences of this word in both old and new corpora is required to draw correct conclusions about changes of its meaning. this requires a model that can extract information from many word occurrences and somehow aggregate it to produce the final prediction. also, this makes creating labeled datasets for the task extremely labour-consuming, resulting in typical datasets containing less than 100 labeled words per language @xcite , which is hardly enough for fine-tuning. alternatively, in @xcite the contextualized embeddings calculated by a pre-trained mlm without any fine-tuning were applied to solve the lscd task. they found that the largest signal in these embeddings corresponds to the grammatical form, not to the meaning of words. this is known as the grammatical or orthographic bias of the contextualized embeddings and prohibits their direct application to the lscd task. the main idea behind our solution is that finetuning on some task that requires understanding word senses and at the same time ignoring word forms shall help to get rid of grammatical bias in the contextualized embeddings. a suitable task shall also have a large dataset for fine-tuning. in our solution of the lscd task, we fine-tune a pretrained mlm as part of a gloss-based wsd system, i.e. a system that can select the most appropriate gloss for a given word in a given context. our wsd system is based on the architecture proposed in @xcite , however, we replace english bert with multilingual xlm-r to make our system multilingual. we train the system on english wsd data only, then apply it to the texts in spanish exploiting zero-shot cross-lingual transferability of xlm-r to obtain the contextualized embeddings for spanish words. despite not using any labeled data in spanish, the described method of fine-tuning xlm-r results in such contextualized embeddings that are directly applicable for lexical semantic change detection in spanish. our solution based on these contextualized embeddings has achieved the best results among all other participants in both main subtasks, and also in all optional subtasks except for the sense gain detection. foot_0 2 background our solution is inspired by the bem (bi-encoder model) system developed by @xcite to solve the word sense disambiguation (wsd) task in english. while wsd is essentially a classification task requiring to annotate each occurrence of polysemous words with one of their senses described in wordnet @xcite or other sense inventory, a huge number of senses in wordnet (more than 100k) and zero or very few examples for most senses and words in the labeled training sets make standard classification approaches not applicable. instead of treating senses as atomic classes, in bem they are represented with their glosses from wordnet. two encoders are introduced: the gloss encoder to build embeddings for glosses, and the context encoder to build contextualized embeddings for word occurrences. these encoders are trained jointly such that for each word occurrence among all glosses of this word a gloss describing its meaning in the given context can be selected by the similarity between the corresponding contextualized embedding and the gloss embeddings. the original bem system employs english bert @xcite as both gloss and context encoders. the system is trained on the english wsd dataset semcor @xcite . we replace english bert with multilingual xlm-r @xcite . xlm-roberta (xlm-r for short) is basically the multilingual version of roberta @xcite , and roberta is bert @xcite with several improvements in the training procedure. all of these models essentially train the encoder of the transformerbased machine translation system @xcite with masked language modeling (mlm) objective, i.e. to restore some words in a text fragment from nearby words (see @xcite for technical details). in contrast to bert and roberta pre-trained on english texts only, xlm-r is pre-trained on 2.5tb of texts in 100 languages. surprisingly, this allows not only processing texts in all of these languages but also demonstrates zeroshot cross-lingual transferability, meaning that after fine-tuning xlm-r to solve some classification task on english texts only, it often can solve the same task for texts in other languages reasonably well @xcite . our approach to the lscd task was initially developed during our participation in the rushifteval-2021 shared task on lscd for the russian language @xcite and described in @xcite . however, in rushifteval-2021 only a graded change detection task was proposed and the only metric was spearman's correlation with the gold compare score, which is the average similarity between word occurrences in two corpora @xcite . the lscdiscovery shared task in spanish offers a more thorough comparison of competing approaches by introducing both the graded change and the binary change detection subtasks. it also replaces the gold compare scores with the gold jensen-shannon distance between the sense distributions inferred by the organizers, though calculating the gold compare scores as an additional metric as well. also in rushifteval-2021 our best solution was a linear regression model that used different distances between the contextualized embeddings as features and was trained on additional labeled data in russian. this resulted in consistent but not very large improvement compared to simply using the raw distance between the contextualized embeddings. thus, for the lscdiscovery task, we decided to use the simpler solution that also does not require any labeled data in spanish. ## system overview the architecture of our gloss-based wsd system is shown in figure 1 . the architecture and the training procedure are borrowed from blevins and zettlemoyer (2020), except for the english bert replaced with multilingual xlm-r in both context and gloss encoders. as usual for xlm-r, the input texts are surrounded by the special @xmath0. to obtain the contextualized embedding for a word in context, the outputs at the positions of the target word are taken from the last layer of the context encoder. if the target word was split into subwords by the xlm-r tokenizer, then mean pooling is applied to the corresponding outputs. for each sense of the target word described in word-net, the corresponding gloss is encoded by taking the output from the last layer of the gloss encoder at the position of the @xmath1. 2 the dot product between the contextualized embedding of the target word and the gloss embeddings for each of its senses is calculated, then the softmax function is applied to obtain the probability distribution over word senses. the whole system is trained by minimizing the cross-entropy loss between the predicted distribution over senses and the correct sense. following blevins and zettlemoyer (2020), we trained the system on english semcor @xcite , which is a large dataset consisting of more than 200k sense-annotated word occurrences. the glosses were taken from wordnet 3.0 @xcite . @xcite wsd dataset served as the development set to choose the final checkpoint. the large version of xlm-r was employed for both encoders. we trained the system for 10 epochs, which took 3 days on two v100 gpus. the xlm-r model fine-tuned as the context encoder of this wsd system is called the gloss language model (glm) below to distinguish it from the standard xlm-r pre-trained with the mlm objective only. after the wsd system is trained, in order to estimate the similarity in meaning between two occurrences of the same word, we normalize their contextualized embeddings (divide them by their l1-norm) and calculate the manhattan distance as shown in figure 2 . ## results tables 1, 2 show our results compared to the baselines and to the best results of other participants. in the graded subtasks our solution achieves the best results among all participants. in the postevaluation experiments, we compared the finetuned xlm-r model (glm) with the original one (mlm). evidently, fine-tuning xlm-r on the wsd task gives a huge boost in performance. our apd scores have a much higher spearman's correlation with the gold compare scores than with the gold jsd scores, which supports our hypothesis that simple averaging of the distances between the contextualized embeddings is more suitable as an approximation of the compare metric. for the binary change detection and the sense loss detection subtasks our solution also outperforms all other participants. however, for the sense gain detection subtask our solution shows f1-scores of 0.483-0.510, which is about 10 points of f1-score worse than the best result in the competition. notice that we did not specifically address the optional sense loss and sense gain detection subtasks, instead, we reused the predictions from the main binary change detection subtask. ## conclusion in this paper, we presented a solution for both graded and binary change detection. our solution achieves the best results among all participants in both graded change detection subtasks, as well as two out of three binary change detection subtasks. the key component of our solution which is shown to be very important is fine-tuning of a masked language model as part of a gloss-based wsd system.