TPPMI - a Temporal Positive Pointwise Mutual Information Embedding of Words (2024)

Abstract: We present Temporal Positive Pointwise Mutual Information (TPPMI) embeddings as a robust and data-efficient alternative for modeling temporal semantic change. Based on the assumption that the semantics of the most frequent words in a corpus are relatively stable over time, our model represents words as vectors of their PPMI similarities with a predefined set of such context words. We evaluate our method on the temporal word analogy benchmark of Yao et al. (2018) and compare it to the TWEC model (Di Carlo et al., 2019), demonstrating the competitiveness of the approach. While the performance of TPPMI stays below that of the state-of-the-art TWEC model, it offers a higher degree of interpretability and is applicable in scenarios where only a limited amount of data is available.

## introduction word embedding models have become the dominant approach to modelling lexical semantics in the natural language processing (nlp) community. while contextual embeddings are now prevalent in most nlp applications, common static embedding methods such as word2vec @xcite and glove @xcite are still widely used in the computational modeling of word meaning, including the study of semantic change. modern approaches train temporal word embeddings by learning alignments between multiple sets of word vectors @xcite , but these approaches rely on the availability of a large amount of training data from each time period. the efficiency and robustness of pointwise mutual information (pmi) as a simple measure for word co-occurrence has been demonstrated in multiple studies @xcite @xcite . in this study we propose the use of positive pointwise mutual information (ppmi) to create temporal embeddings that represent the meaning of words as vectors of their ppmi with a small fixed set of context words chosen from the most frequent content words of the corpus, based on the assumption that the semantics of these words is relatively stable across time. our experiments on the temporal word analogy task of @xcite demonstrate that this highly interpretable model offers a robust and competitive measure of lexical semantic change. the rest of the paper is structured as follows: section 2 summarizes recent research on temporal word embeddings. section 3 presents our method. section 4 describes our experimental setup and section 5 presents results of both quantitative and qualitative analysis. section 6 concludes the paper. all software described here is publicly available on github foot_0 under an mit license. ## related work word embeddings have been used extensively to study lexical semantic change. @xcite trains time-aware word embeddings by jointly learning multiple word embeddings and their alignment. for evaluation they train on a dataset of nearly 100,000 crawled articles from the new york times (nyt), published between 1980 and 2016, and evaluate their method by using the resulting vector spaces to solve simple temporal reasoning tasks. one of these tasks that has since been reused for evaluating temporal embeddings, and which we also use in this paper, are temporal analogy questions of the form 2012:@xmath0:?. in this example a temporal embedding is expected to predict bush as a likely or even the most likely answer based on the assumption that the word's semantics in 2004 news texts should be (most) similar to that of obama in 2012. @xcite develop dynamic bernoulli embeddings, a type of exponential family embeddings @xcite , which capture change by modeling words as sequences of embeddings over time slices that are grounded in a space of shared context vectors. they train their models on corpora of scientific papers and u.s. senate speeches. in addition to qualitative analysis of the resulting embeddings they also perform intrinsic evaluation that involves calculating their loss function on heldout portions for each dataset and time period. this experiment is reproduced by di @xcite , who propose the twec method for aligning word2vec embeddings trained on data from various time periods based on a shared target vector space trained on atemporal data. they also test their method on the temporal analogy task, and it is this approach that we use for comparison when evaluating the tppmi method. ## method the temporal positive pointwise mutual information (tppmi) method models semantic change of words based on their distribution w.r.t a fixed set of the most frequent content words of the atemporal context, based on the assumption that these words exhibit relatively stable semantics. pointwise mutual information (pmi) measures the co-occurrence of a word w with a context word c by calculating pmi(w, c) = log p (w, c) p (w) pα (c) -log(k) the static set of context words is determined by removing stopwords from the atemporal corpus (the union of all time slices) and sampling from the most frequent words in the corpus. the number of words, which determines the dimensionality of the tppmi embeddings, is a parameter of our approach. to create a set of n context words we sample from the 2n most frequent words. stopword removal is performed using the nltk foot_1 package. the size of the context word set greatly influences the robustness and performance of our models and should be optimized separately for each application of the tppmi approach. ## experiment following the experimental setup of di @xcite we train our temporal embeddings on the nyt dataset and evaluate it on temporal word analogies (see also section 2). we compare our model to both twec and to static word2vec embeddings as a trivial baseline. ## conclusion we presented the temporal positive pointwise mutual information model of lexical semantic change. tppmi offers an interpretable and robust approach to capturing temporal semantic shifts of words, addressing the challenges of small and sparse datasets. the model is evaluated on a temporal word analogy task and achieves reasonable performance on both static and dynamic analogies. despite its inferiority to more sophisticated models like twec, we believe that its simplicity and computational efficiency make tppmi a practical alternative for applications with limited data. our qualitative analysis further demonstrates the model's ability to show semantic shifts of individual words over time and to offer explanations of such shifts based on the words corresponding to significant dimensions.