HSE at T empo W i C : Detecting Meaning Shift in Social Media with Diachronic Language Models (2022)

Abstract: This paper describes our methods for temporal meaning shift detection, implemented during the TempoWiC shared task. We present two systems: with and without time span data usage. Our approaches are based on the language models fine-tuned for Twitter domain. Both systems outperformed all the competition’s baselines except TimeLMs-SIM. Our best submission achieved the macro-F1 score of 70.09% and took the 7th place. This result was achieved by using diachronic language models from the TimeLMs project.

## introduction it is a commonplace that words change their meanings and connotations through time. despite numerous studies about that, there are still difficulties in semantic change detection. static embeddings are not suitable for working with semantic change, since they cannot reflect the fact that a word can have completely unrelated meanings. in this work, we are focusing on contextualized embeddings, which produce different vector representations depending on the context. there were a number of competitions dedicated to semantic change detection, for example, tem-powic @xcite and lscdiscovery @xcite . these competitions were aimed at determining the difference in the meanings of words depending on the time period in which they are used. datasets at lscdiscovery consisted of texts from different centuries @xcite . for this shared task, tem-powic, the data with a time interval only of one year is used. it significantly changes the approach to the competition. temporal word in context (tempowic) benchmark aims to decide if there is a change between the meaning of two words in a given pair of tweets. tempowic is designed as a binary classification problem where the target word is featured in two tweets from different time periods, and the goal is to detect whether there is a meaning shift or not. when creating a dataset for the competition, the authors decided to use data from social media (twitter), while when developing the previous dataset wic (pilehvar and camacho-collados, 2018), word usage was taken from more formal sources such as wiktionary, wordnet and verbnet. the language used in social networks is much more informal and dynamic, so such a dataset is able to reflect even minor changes in word usage. the tempowic dataset consists of paired tweets and is divided into train/validation/test samples of size 1,428/396/1,473 instances, respectively. for each sample, the set of target words is different. as additional data, the publication date is indicated for each tweet. there are no missing values in the dataset. participants of the competition were asked to detect the change in the meaning of the target word both with and without using time-span information. to estimate the system's performance, the macro-f1 score was used. ## systems overview in this section, we describe two systems that were implemented by our team during the shared task. both systems are based on the pre-trained language models. for our first system, we did not use timespan information and extracted embeddings from the twitter-roberta-base model @xcite . in the second approach, we tried to improve our system's performance by using diachronic language models from the timelms project @xcite . ## results for the validation set, we also obtained macro-f1 scores for each target word (see ## discussion the main question that is still open is that: can we really detect a meaning shift on such a short time period as about a year? at tempowic it was postulated that in social media we can observe faster semantic shifts @xcite . from a linguistic point of view, a change occurred, when there is evidence of transmission of innovations to others, i.e., of conventionalization @xcite . it seems that one year is too short time for any language innovation to become widespread, even via social media. moreover, it may take more time to be sure that this is a real change and not a nonce word. let us consider words that were taken for a validation set. there were 4 of them (lotte, primo, recount, and impostor). according to the corpus provided by the organizers, the word recount was mostly used in the context of elections, lotte in the context of a concert and as a hotel name. it means that at least 50% of validation words demonstrate that trending words in twitter in general most likely describe ongoing or recent events. it was said that at tempowic the task was to decide if the meaning of the first target word in context is the same as the second one or not @xcite . there are also examples of annotated sentences with target words in the article (see ## conclusion we presented two systems for temporal meaning shift detection in twitter, both with and without time span data usage. the best result was obtained with diachronic language models continuously trained for the twitter domain. for our future research, we will consider weight-pooling methods as an attempt to improve our system's performance.