Temporal Word Meaning Disambiguation using T ime LM s (2022)

Abstract: Meaning of words constantly change given the events in modern civilization. Large Language Models use word embeddings, which are often static and thus cannot cope with this semantic change. Thus, it is important to resolve ambiguity in word meanings. This paper is an effort in this direction, where we explore methods for word sense disambiguation for the EvoNLP shared task. We conduct rigorous ablations for two solutions to this problem. We see that an approach using time-aware language models helps this task. Furthermore, we explore possible future directions to this problem.

## introduction a change in the meaning of a word in varying semantic contents is a challenge for various nlp tasks such as text and sentence classification, question answering and sentence prediction. recent developments in large language models (llms) like elmo @xcite , bert @xcite and @xcite have revolutionised the field of nlp with context dependent word embeddings. these models have been trained on a large corpus of unlabelled text. while these models take in consideration the semantics of the text, it is limited to the corpus it was trained on. this introduces a new challenge of the shift in the meaning of a word across the temporal axis. word sense disambiguation @xcite ) is the process of identifying the meaning of a word from multiple possible meanings in varying contexts. this task can be further extended as a polysemy resolution task to classify the meaning of words in different contexts. our system performs a similar task while classifying two texts with a common word with the same or different meaning. specifically, the premise of our system is to classify tweets from two different time periods with a * equal contribution common word. the variation in the meaning of a word is caused by two factors, the context of the word in the form of a tweet or a change in the usage and hence in the meaning of a word because of the shift along the time axis. historically it has been observed that the meanings of some words have been altered over time. for example, the word "fathom" originally meant "to encircle with one's arms" and now is defined as "to understand after much thought". the ever expanding nature of the internet and social media have led to rapid evolution of words, with the meanings of words changing and new words getting csoined. this means that the corpus of data used for training a llm will keep changing over time. hence, the pretrained models for existing llms like bert, roberta cannot be used to compare word embeddings for a word from two different time periods. this shared task @xcite focusses precisely on this problem statement. to address this problem, we propose a system comprising of timelms @xcite to incorporate the time aspect of the data. timelms are language models that are trained using data up to a certain time instance. in this case they are trained on tweets gathered by the end of a year. therefore there exists a unique timelm model for each year which takes into account the time aspect of data. the dataset used for testing our system consists of tweets from the years 2019, 2020 and 2021. tweets from two different time periods containing a common word are paired in this dataset and labelled to indicate similarity or dissimilarity in the meanings of that word in the two tweets. the timelms used in our system are roberta models trained on tweets upto the specific year. this enables our system to get an accurate representations of the words based on their use upto that time period. the embeddings are then compared based on a similarity metric to classify the tweets using a preset threshold value for similarity. this paper is organised as follows. we analyse existing research and methods in section (2). we give a overview of the dataset used for our system in section (3). we provide a overview of our system implementation in section (4). we also compare the results of our experiments in developing this system in section (5). we discuss the possible improvements and scope of this system in section (6). ## related work in natural language processing, the meaning of words is denoted by a vector, commonly known as word embedding. works like glove @xcite and word2vec @xcite were one of the first ones to represent a word using vectors. however, the embeddings thus generated were context-agnostic, meaning their meaning was fixed and were not dependent on the context. with the dawn of modern text encoders @xcite , context dependent embeddings can be easily calculated. works like pilehvar and camacho-collados; raganato et al. aim to have manually annotated datasets containing pairs of sentences having same or different meaning, and labelling them as such. to solve this task, several methods have been developed. works like @xcite ; @xcite try to impart context based knowledge into the embeddings by using wordnet @xcite attributes. the models are trained in a self-supervised fashion with entity linking. another approach is to use word-level embeddings. @xcite use this approach, combining it with a k-nn (k nearest neighbours) method to disambiguate the word embeddings. note that transformers can also be used for this purpose, since the output features from the transformers can be interpreted as word embeddings. @xcite studies model layers to understand the effect of attention-based architectures in word sense disambiguation task. elmo @xcite is one of many available architectures in this direction. lastly, work has been done to incorporate the semantic space knowledge into the embeddings @xcite , also known as sense-based disambiguation. given this, little work has been done on word meaning disambiguation in a temporal setting. this means that the information about the time of text utterance is also provided along with the sentence itself. this paper tries to provide a solution to this problem -word meaning disambiguation when temporal information is available. ## dataset description the dataset consists of 1428 training samples and 396 validation samples. the final scores were calculated on a set of 10,000 unseen test samples. in every training sample, we were provided with two sentences and the word whose semantic meaning was to be compared. some metadata like tokens and start and end of word was also included in every sample.in the training dataset, out of the 1428 samples, 650 examples had the words in two sentences having same meaning, whereas 778 samples had the words in two sentences having different meaning. note that since this dataset is relatively balanced, and hence does not need any additional preprocessing to balance the data distribution. however, it is important to note that the target words in the training and testing dataset constitute two different sets, and hence the problem should be solved in a way that is target word agnostic. an illustration of the data is shown in figure 1 . the left part shows an example where the meaning of the target word "virus" is different in both tweets. specifically, in the top left tweet it indicates to the disease-causing organism whereas the bottom left tweet indicates to a thing that the person likes. in the right part, both instances of the target word mean the same, denoting disease-causing organism. the dataset also provides the month and year when the tweet was written. this provides us the temporal information, which can be useful for the semantic evaluation of the words in the given context. our approach aims at using this semantic information in a way that a language model relevant to the tweet is used to get the semantic features of the tweet. ## results we hereby present the results of both of our methods. we report several interesting observations based on the results. our results based on our cosine similarity are shown in we observe that the timelms based method performs the best. we speculate this is because of the time-aware nature of the models. some words, for example "lockdown" have significantly different meaning before and after the pandemix. thus, models pretrained on the specific data results in better performance. ## conclusion in this paper, we explore two solutions to the word sense disambiguation problem within the scope of evonlp shared task. we report a maximum testing f1-score of 57.94% with timelms. we foresee several research directions for this work. one line of work can be explore robustness of the contrastive models. the threshold search technique for this method can be explored in greater detail.