Temporal Attention for Language Models (2022)

Abstract: Pretrained language models based on the transformer architecture have shown great success in NLP.Textual training data often comes from the web and is thus tagged with time-specific information, but most language models ignore this information. They are trained on the textual data alone, limiting their ability to generalize temporally. In this work, we extend the key component of the transformer architecture, i.e., the self-attention mechanism, and propose temporal attention - a time-aware self-attention mechanism. Temporal attention can be applied to any transformer model and requires the input texts to be accompanied with their relevant time points. This mechanism allows the transformer to capture this temporal information and create time-specific contextualized word representations. We leverage these representations for the task of semantic change detection; we apply our proposed mechanism to BERT and experiment on three datasets in different languages (English, German, and Latin) that also vary in time, size, and genre. Our proposed model achieves state-of-the-art results on all the datasets.

## introduction language models (lms) are usually pretrained on corpora derived from a snapshot of the web crawled at a specific moment in time @xcite . but our language is constantly evolving; new words are created, meanings and word usages change. for instance, the covid-19 pandemic has caused significant changes to our language; consider the new video-related sense of "zoom" and the new senses recently associated with the word "vaccine". the "static" nature of existing lms makes them unaware of time, and in particular unware of language changes that occur over time. this prevents such models from adapting to time and generalizing temporally @xcite @xcite @xcite , abilities that were shown to be important for many tasks in nlp and information retrieval @xcite @xcite @xcite . recently, to create time-aware models, the nlp community has started to use time as a feature in training and fine-tuning language models @xcite . these two studies achieve this by concatenating a time token to the text sequence before training the models. the former was concerned with temporal question answering, whereas the latter-with semantic change detection and sentence time prediction. in this work, we introduce a new methodology to create time-aware language models and experiment on the task of semantic change detection. at the heart of the transformer architecture is the self-attention mechanism @xcite . this mechanism allows the transformer to capture the complex relationships between words by relating them to each other multiple times. an attention weight has a clear meaning: how much a particular word will be weighted when computing the next representation for the current word @xcite . this mechanism also enables the above-mentioned temporal models @xcite to work; by concatenating time-specific tokens to the text sequences, the self-attention mechanism would compute the relationships between them and the original tokens in the texts, effectively making the output embeddings time-aware (as the output embeddings will depend on the concatenated time tokens). in this work, instead of changing the text sequences as in prior work, we modify the model itself and specifically the attention mechanism to make it time-aware. we propose a time-aware selfattention mechanism that is an extension of the self-attention mechanism of the transformer. it considers the time the text sequences (or documents) were written when computing attention scores. as described above, self-attention captures relationships between words. we want to condition these relationships on time. by adding a time matrix as an additional input to the self-attention (along with the standard query, key, and value matrices), we condition the attention weights on the time. in other words, the adapted mechanism also considers the time when calculating the weights of each word. we refer to this adapted attention as temporal attention (section 3.2). see figure 1 for an illustration of our proposed mechanism. we experiment on the task of semantic change detection -the task of identifying which words undergo semantic changes and to what extent. semantic change detection methods are used in historical linguistics and digital humanities to study the evolution of word meaning over time and in different domains @xcite . most existing contextual methods detect changes by first embedding the target words in each time point and then either aggregating them to create a time-specific embedding @xcite , or computing a cluster of the embeddings for each time @xcite @xcite . the embeddings or clusters are compared to estimate the degree of change between different times. we experiment with several diverse datasets in terms of time, language, size, and genre. our empirical results show that our model outperforms state-of-the-art methods @xcite @xcite . our contributions are threefold: (1) we introduce a time-aware self-attention mechanism as an extension of the original mechanism of the transformer. the proposed mechanism considers the time the text sequences were written. the time is considered during the computation of attention scores, thus allowing to create time-specific contextualized word representations; (2) we conduct evaluations on the task of semantic change detection and reach state-of-the-art performance on three diverse datasets in terms of time, language, size, and genre; (3) we contribute our code and trained models to the community for further research. 2 related work ## model our model adopts a multi-layer bidirectional transformer @xcite . it treats words in the document as input tokens and computes a representation for each token. formally, given a sequence of n words w 1 , w 2 , . . . , w n , the transformer computes d-dimensional word representations x 1 , x 2 , . . . , x @xmath0. ## semantic change detection in this section, we employ our proposed temporal attention mechanism (section 3.2) for the task of semantic change detection @xcite . the ability to detect and quantify semantic changes is important to lexicography, linguistics, and is a basic component in many nlp tasks. for example, search in temporal corpora, historical sentiment analysis, and understanding historical documents. the objective of this task is to rank a set of target words according to their degree of semantic change between two time points t 1 and t 2 . in this work, we follow the practice of @xcite to estimate the semantic change a word underwent and rank the target words based on these estimates. given a target word w, we generate time-specific representations of it and compare them to detect semantic changes. algorithm 1 formally describes the method. we begin by sampling n sentences containing w from each time point @xmath1, t 2 } (line 3). for each sentence sent, we create a sequence embedding by running it through the temporal attention model (note the model receives as input both sent and t) and extracting the model's hidden layers that correspond to w (lines 5-6). we then choose the last h hidden layers and average them to get a single vector (line 7). this is the contextual word embedding of w, denoted by v. following, the resulting embeddings are aggregated at the token level and averaged (line 10), in order to create a non-contextual time-specific representation for w for each time t, denoted by x t . finally, we estimate the semantic change of w by measuring the cosine distance (cos_dist) between two time-specific representations of the same token (line 12). ## results in this section, we outline the results of our empirical evaluation. in all tables throughout the section, the best result in each column is boldfaced; performance is measured using pearson's r and spearman's ρ correlation coefficients. ## conclusion in this paper, we presented a time-aware selfattention mechanism as an extension of the original mechanism of the transformer. the proposed mechanism considers the time the text sequences were written when computing attention scores, thus allowing creating time-specific contextualized word representations. we conducted evaluations on the task of semantic change detection and reached stateof-the-art performance on three diverse datasets in terms of time, language, size, and genre. in addition, we experimented with small-sized pretrained models and found they outperform larger models on this task. we conduct an experiment evaluating the marginal addition of time token prepending along with temporal attention and conclude that on all but the english dataset it hurts performance. we wish to study how to best combine the two approaches in future work. additionally, for future work, we plan to extend this work by applying temporal attention to other tasks, such as web search and sentence time prediction, as well as ex-perimenting with more time points and different granularities.