Exploring Word Usage Change with Continuously Evolving Embeddings (2021)

Abstract: The usage of individual words can change over time, for example, when words experience a semantic shift. As text datasets generally comprise documents that were collected over a longer period of time, examining word usage changes in a corpus can often reveal interesting patterns. In this paper, we introduce a simple and intuitive way to track word usage changes via continuously evolving embeddings, computed as a weighted running average of transformer-based contextualized embeddings. We demonstrate our approach on a corpus of recent New York Times article snippets and provide code for an easy to use web app to conveniently explore semantic shifts with interactive plots.

## introduction languages are constantly changing, with new words being coined or existing ones adopting a new meaning @xcite . for example, as hurricane dorian hit the bahamas on sept. 1, 2019, and was henceforth regarded as the worst natural disaster in the country's recorded history, within a matter of days the until then innocuous name "dorian" suddenly became synonymous with a devastating tropical cyclone (fig. 1 ). these kinds of semantic shifts are of great interest for researchers in fields like computational linguists and digital humanities, but their analysis requires appropriate tools, especially to create fine-granular visualizations, for example, to facilitate the study of texts from fast-paced environments such as social media. word embeddings are nowadays the method of choice when examining the meaning of and relation between words, and, as an extension thereof, diachronic embeddings can be used to discover and analyze the semantic shifts and usage changes of words over time. the main idea behind diachronic word embeddings is to learn a set of embeddings for each word, one for each time period of interest, to then see how much the embeddings for the same word differ over time (see e.g. @xcite or @xcite for a comprehensive overview). however, most approaches for computing diachronic embeddings either a) rely on static word embedding models such as word2vec, which makes it difficult to use them with small corpora, b) are based upon rather complex dynamic language models, and/or c) require the corpus to be split into individual time slices, which introduces a bias, since by computing embeddings for different years, for example, one implicitly assumes that the meaning of a word might change between january and december of the previous year, but not between july and august of the same year. in this paper, we introduce continuously evolving embeddings that are computed in one pass over the whole (chronologically ordered) corpus by keeping track of a weighted running average of contextualized embeddings generated by a transformer model such as bert (sec. 2). by taking (poten-tially arbitrarily frequent) 'snapshots' of the current state of the embeddings at user-defined time points, one obtains smoothly changing high-resolution diachronic embeddings. with these embeddings, semantic shifts can be detected at a resolution of weeks or months instead of years or decades. the exploration of word usage change is facilitated by our web app that provides the user with the corresponding interactive graphics (sec. 3), which we demonstrate on a corpus of recent newspaper article snippets (sec. 4). ## continuously evolving embeddings let x local t i be the contextualized embedding of a token t generated by some arbitrary method (e.g. a pre-trained bert model) for the i th occurrence of t in a corpus. then a global embedding of t can be computed by averaging over the local embeddings of all n occurrences of t in the corpus @xcite @xcite : equivalently, this can be formulated as a running average @xcite , allowing for memoryefficient continuous updates in one pass over the corpus: using this running average formula, it is possible to compute continuously evolving embeddings by updating the global embedding as more and more sentences are processed @xcite . however, usually the more recent occurrences of the word are of greater relevance when determining the current sense of the word. to account for this, the above formula can be adapted by introducing a weighting factor 0 < α ≤ 0.5: this is equivalent to computing the weighted average of the two embeddings (for large n): and results in an exponential forgetting of the past occurrences in favor of the more recent instances @xcite . while @xcite generate diachronic embeddings by computing a global average of all contextualized embeddings occurring in texts from individual (predefined) time periods (eq. 1), we instead propose to keep track of a weighted running average computed in one pass over the whole (chronologically ordered) corpus (eq. 2). by taking 'snapshots' of the current state of these continuously evolving embeddings at user-defined time points, it is possible to obtain smoothly changing high-resolution diachronic word embeddings. 2 the weighting parameter α in the running average should be set according to the number of word occurrences one assumes it might take for the meaning to change and can be set individually for each word to reflect the differing overall frequencies and semantic shift paces @xcite . the computation of continuously evolving embeddings scales linearly with respect to the number of sentences in the dataset, since each sentence has to be embedded with the transformer model once to update the weighted running average with the respective contextualized embeddings. the required memory, on the other hand, scales linearly with the number of embedding snapshots that are taken during the computation, where a copy of the current state of the global embedding matrix needs to be stored for every snapshot. ## the evolvemb app word usage changes in a corpus can be easily explored using the web application we created for this purpose. the app itself is based on the dash framework @xcite and can be run locally by following the steps listed in fig. 2 and demonstrated in the screencast 3 , i.e., first computing continuously evolving embeddings and saving the respective snapshots (or, alternatively, diachronic embeddings obtained with a traditional approach such as a sgns model trained on individual time slices @xcite ), and then starting the app (which loads the pre-computed embeddings) to obtain the list of most changed words in the corpus and a simple interface to generate the plots displaying the evolution of nearest neighbors over time for individual (user-selected) words. ## exploring word usage change to demonstrate our approach, we downloaded 95,203 newspaper article snippets (consisting of a headline and 1-3 sentences) published by the new york times between april 1 st , 2019, and dec. 31 st , 2020, via their api. 4 diachronic embeddings were computed for the 5,620 words that occurred at least 50 times in the corpus by processing the texts chronologically, computing continuously evolving 3 https://youtu.be/ltf67j-la7i 4 https://developer.nytimes.com/apis embeddings with a transformer model, and taking a snapshot of the current state of the embeddings at the end of each month. α was set individually for each word based on how many times on average the word occurred in the articles of a single month. to compute the contextualized embeddings, we experimented with pre-trained bert and roberta models from the huggingface library @xcite ) that were either used as is or fine-tuned for three epochs on our corpus. as the results obtained with both models were similar, we focus on bert in the following. words with different usages were identified based on the minimum cosine similarity between their embedding snapshots from different time points. 5 as this also yielded several words with multiple meanings that showed seasonal trends ( as expected, when computing continuously evolving embeddings on shuffled article snippets, i.e., a corpus that is no longer chronologically ordered @xcite , the resulting semantic shift scores are significantly lower ( even though the continuously evolving embeddings computed with pre-trained transformers are already sufficient to identify many words with usage changes, fine-tuning of the models is generally advised, especially to clearly identify semantic shifts when the new usage of a word was not present in the texts the transformer was originally trained on. to illustrate this, inspired by rosenfeld and erk (2018), we introduced a synthetic semantic shift into the data for an artificially created figure 3 : plots as included in the app, here depicting the evolution of nearest neighbors over time for the word "category", computed with a pre-trained bert model on our nytimes article snippets dataset. for the target word, first the two time points with the smallest cosine similarity between the embeddings of the word itself were identified, then the five nearest neighbors of the word at both time points were selected (red and blue colors respectively; words that occurred in both sets are in red). left: cosine similarity between the target word at each time point and the nearest neighbors, as well as the two most different embedding snapshots of the target word itself (inspired by the plots in @xcite ). right: 2d pca visualization of all embedding snapshots of the target word as well as both sets of nearest neighbors (smaller dots represent embeddings at earlier time points). ## related work when it comes to learning word embeddings in general, it is helpful to distinguish between older methods, such as word2vec @xcite or glove @xcite , that learn static word embeddings, i.e., a single "global" embedding for each word in the vocabulary, and modern transformer models, such as elmo @xcite , bert @xcite , roberta @xcite , and flair @xcite , that generate contextualized embeddings based on the local context of a word in the current sentence. while the static word embedding models are usually trained on a target corpus containing several millions of words to obtain expressive domain-specific embeddings @xcite , pre-trained transformers are well suited for transfer learning and can therefore also more easily be applied to smaller datasets. some of the more advanced methods for creating diachronic embeddings use special-purpose dynamic language models, which explicitly take the temporal structure of the data into account when learning the word embeddings @xcite @xcite @xcite @xcite . a different line of work instead relies on conventional static word embedding models, such as word2vec, and uses them directly to learn embeddings for the individual time periods. the main challenge here consists of aligning the word embeddings learned for different time intervals, which can, for example, be achieved by using the embeddings from one time slice to initialize the next @xcite , by explicitly matching up the matrices learned on different time periods @xcite @xcite , or utilizing other techniques such as temporal referencing @xcite ). an even simpler approach to produce diachronic word embedding in a single embedding space uses the same (fixed) model to compute contextualized embeddings on all texts and then averages the respective embeddings from each individual time period to get the diachronic embeddings @xcite al., 0.0 0.2 0.4 0.6 0.8 1.0 cosine similarity category (2020-02-29) category (2020-12-31) podcast competing paintings reader-submitted charts hurricane rape flooding storm flying pc 1 pc 2 figure 5 : analogously to fig. 3 the results with diachronic embeddings obtained by training a sgns word2vec model on the article snippets from the respective time intervals @xcite . while several of the above mentioned papers have published code alongside their manuscripts, this was mainly done with the intention that others could reproduce their results, not apply the methods to novel datasets. to the best of our knowledge, only @xcite has released a more comprehensive library to explore word usage change in other corpora, however, their approach relies on static word embeddings and should therefore mainly be applied to larger corpora. most other available software for analyzing corpora only considers word frequencies over time, but does not track the semantic shifts of these words. ## conclusion this paper introduced continuously evolving embeddings as a conceptually simple and intuitive method for computing smoothly changing highresolution diachronic embeddings from weighted running averages of contextualized embeddings. 8 since the contextualized embeddings are all in the same embedding space already (defined by the single fixed model), averaging the embeddings from each time slice creates time period specific global word embeddings that are themselves also comparable. by taking advantage of pre-trained transformer models and processing the texts in a corpus sequentially rather than dividing them into (more or less arbitrary) time slices, our approach makes it possible to obtain diachronic embeddings from comparatively small corpora and at very short intervals compared to the previously standard time periods of at least one year. this should make our method particularly well suited to study fast-paced environments such as social media, where a new meme can go viral in a matter of hours, only to be superseded by the next a few days later. aside from the parameters involved in the underlying transformer model and its possible finetuning, our method only has a single hyperparameter, α, whose setting mostly just influences how frequently the embedding snapshots need to be taken to not miss any semantic shifts in between the snapshot intervals. on our nytimes corpus we obtained reasonable results already with pretrained transformer models, however, fine-tuning is nevertheless advised and especially helpful to characterize new word usages that the transformer did not encounter in its original training data. we hope that the provided code will help others identify interesting patterns of word usage change in their own corpora.