XL - LEXEME : W i C Pretrained Model for Cross-Lingual LEX ical s EM antic chang E (2023)

Abstract: The recent introduction of large-scale datasets for the WiC (Word in Context) task enables the creation of more reliable and meaningful contextualized word embeddings.However, most of the approaches to the WiC task use cross-encoders, which prevent the possibility of deriving comparable word embeddings.In this work, we introduce XL-LEXEME, a Lexical Semantic Change Detection model.XL-LEXEME extends SBERT, highlighting the target word in the sentence. We evaluate XL-LEXEME on the multilingual benchmarks for SemEval-2020 Task 1 - Lexical Semantic Change (LSC) Detection and the RuShiftEval shared task involving five languages: English, German, Swedish, Latin, and Russian.XL-LEXEME outperforms the state-of-the-art in English, German and Swedish with statistically significant differences from the baseline results and obtains state-of-the-art performance in the RuShiftEval shared task.

## introduction and motivation lexical semantic change (lsc) detection is the task of automatically identifying words that change their meaning over time. the lsc detection task implicitly aims to disambiguate synchronic word sense occurrences and then find differences in the word sense frequencies in different periods. word sense disambiguation (wsd) is a longstudied task in natural language processing @xcite , which consists of associating the correct sense to a word occurring in a specific context. wsd involves some crucial issues, such as relying on a fixed sense inventory. fixed sense inventories ignore the diachronic aspect of language because they can miss older unused senses or be outdated and missing new senses. the word in context task (wic) (pilehvar and camacho-collados, 2019) aims to overcome these issues. in this work, we train a model on the wic task and then use it to perform lsc detection. in the wic task, given the word w and two different contexts c1, c2, the systems have to determine whether the meaning of w is the same in the two contexts or not. our approach is grounded on the assumption that models trained on the wic tasks are robust enough to transfer the knowledge learned in a synchronic setting to a diachronic one. we summarise the main contribution of this work as follows: (i) we propose a pre-trained biencoder model, called xl-lexeme, on a largescale dataset for the wic task, which allows us to obtain comparable lexical-based representations; ## related work lsc detection systems can be categorized based on the distributional embeddings used to tackle the lsc detection task. one category is represented by those approaches that adopt type-base (i.e., static) embeddings. uwb @xcite represents an example of this category of systems. first, it employs word2vec skip-gram with negative sampling @xcite to compute a semantic space for each corpus. it uses techniques like the canonical correlation analysis @xcite and the orthogonal transformation @xcite to align the abovementioned spaces. therefore, the cosine similarity between the vectors representing the word in two different spaces is used to detect the semantic shift. with the increasing use of contextualized word embeddings, numerous approaches employing bert-base models have been developed for lsc detection @xcite . in tempobert @xcite , the authors exploit the concept of masked language modeling (mlm), where the goal is to train a language model to predict a masked portion of text given the remaining part. in particular, they employ this technique to encode the concept of time into a bert model. this is done by concatenating a specific token representing time to the text sequence. at inference time, tempobert can be used to predict the year of a sentence, masking the time reference, or to predict a masked token of the sentence conditioned by the time reference. in the same line of research, in temporal attention @xcite , the authors investigate the effect of modifying the model instead of the input sentence like in tempobert. this is done by extending the model's attention mechanism to consider the time when computing the weight of each word. the time dimension is encoded using a different query embedding matrix for each timestamp. another kind of approach exploits the information coming from other tasks to perform lsc detection. glossreader represents an example @xcite , where a model based on xml-r @xcite is first trained on english semcor @xcite with glosses from wordnet 3.0 @xcite to perform wsd. exploiting the zero-shot crosslingual characteristics of xml-r, the authors used the same model to perform lsc detection in the russian language. with deepmistake @xcite , the authors take advantage of the wic task instead of wsd. they train a cross-encoder with xml-r as an underlying language model on the mcl-wic training and development set and fine-tune on the rusemshift dataset @xcite . deepmistake, differently from xl-lexeme, relies on the cross-encoder architecture and exploits only the mcl-wic training dataset. ## xl-lexeme generally, for pairwise sentence similarity tasks, bert models use a cross-encoder, in which the pairwise sequences are jointly encoded, and the overall vectors are used for the classification. however, in several tasks, the cross-encoder is not suitable since it cannot provide a distinct meaningful representation for each sentence. an approach to overcome this issue involves pooling the bert out-put encoded vectors, which often results in worse performance. sentence-bert (sbert) @xcite overcomes the limitation of cross-encoders using a siamese network, i.e., the weights of the underlying networks are shared. sbert encodes the two sequences separately in the bert model exploiting the siamese architecture. the sequence-level representation is obtained by averaging the output encoded vectors, which are directly compared using similarity measures such as cosine similarity. meanwhile, cross-encoders perform better since they are trained to profit from the attention over the whole input. in this work, we introduce xl-lexeme foot_0 which mirrors models for pairwise sequence similarity tasks and adapts them to the wic task, giving prominence to the target word, i.e. the word for which we want to detect the lsc. the model takes as input two sequences s 1 and s 2 . the sequences are tokenized using subwords tokenizer, such as sentence piece @xcite , and the special @xmath0: xl-lexeme is trained to minimize the contrastive loss @xcite : ## results xl-lexeme obtains competitive results for the russian language in the rushifteval leaderboard. contrary to xl-lexeme, deep mistake and gloss reader are fine-tuned on the rusemshift dataset. the differences between xl-lexeme and the best two systems in the leaderboard are not statically significant. moreover, in ## conclusion in this work, we introduced xl-lexeme, a model for lsc detection. xl-lexeme is pre-trained on a large wic dataset to mirror sentence-level encoders focusing on specific words in contexts. we evaluated our model on two lexical semantic change detection datasets: semeval-2020 task 1 and rushifteval. xl-lexeme outperforms stateof-the-art models for lsc detection in english, german, swedish, and russian datasets, with significant differences from the baselines. the xl-lexeme effectiveness and efficiency make it reliable for lsc detection on large diachronic corpora.