Dynamic Contextualized Word Embeddings (2021)

Abstract: Static word embeddings that represent words by a single vector cannot capture the variability of word meaning in different linguistic and extralinguistic contexts. Building on prior work on contextualized and dynamic word embeddings, we introduce dynamic contextualized word embeddings that represent words as a function of both linguistic and extralinguistic context. Based on a pretrained language model (PLM), dynamic contextualized word embeddings model time and social space jointly, which makes them attractive for a range of NLP tasks involving semantic variability. We highlight potential application scenarios by means of qualitative and quantitative analyses on four English datasets.

## introduction over the last decade, word embeddings have revolutionized the field of nlp. traditional methods such as lsa @xcite , word2vec @xcite , glove @xcite , and fasttext @xcite compute static word embeddings, i.e., they represent words as a single vector. from a theoretical standpoint, this way of modeling lexical semantics is problematic since it ignores the variability of word meaning in different linguistic contexts (e.g., polysemy) as well as different extralinguistic contexts (e.g., temporal and social variation). the first shortcoming was addressed by the introduction of contextualized word embeddings that represent words as vectors varying across linguistic contexts. this allows them to capture more complex characteristics of word meaning, including polysemy. contextualized word embeddings are widely used in nlp, constituting the semantic backbone of pretrained language models (plms) such as elmo @xcite , bert @xcite , gpt-2 @xcite the embeddings have 768 dimensions. @xcite , electra @xcite , and t5 @xcite . a concurrent line of work focused on the second shortcoming of static word embeddings, resulting in various types of dynamic word embeddings. dynamic word embeddings represent words as vectors varying across extralinguistic contexts, in particular time (e.g., @xcite and social space (e.g., @xcite . in this paper, we introduce dynamic contextualized word embeddings that combine the strengths of contextualized word embeddings with the flexibility of dynamic word embeddings. dynamic contextualized word embeddings mark a departure from existing contextualized word embeddings (which are not dynamic) as well as existing dynamic word embeddings (which are not contextualized). furthermore, as opposed to all existing dynamic word embedding types, they represent time and social space jointly. while our general framework for training dynamic contextualized word embeddings is modelagnostic (figure 1 ), we present a version using a plm (bert) as the contextualizer, which allows for an easy integration within existing architectures. dynamic contextualized word embeddings can serve as an analytical tool (e.g., to track the emergence and spread of semantic changes in online communities) or be employed for downstream tasks (e.g., to build temporally and socially aware text classification models), making them beneficial for various areas in nlp that face semantic variability. we illustrate application scenarios by performing exploratory experiments on english data from arxiv, ciao, reddit, and yelp. contributions. we introduce dynamic contextualized word embeddings that represent words as a function of both linguistic and extralinguistic context. based on a plm, dynamic contextualized word embeddings model time and social space jointly, which makes them attractive for a range of nlp tasks. we showcase potential applications by means of qualitative and quantitative analyses. foot_0 2 related work ## data we fit dynamic contextualized word embeddings to four datasets with different linguistic, social, and temporal characteristics, which allows us to investigate factors impacting their utility. each dataset d consists of a set of texts (e.g., reviews) written by a set of social units s (e.g., users) over a sequence of time periods t (e.g., years). furthermore, the social units are connected by a set of edges e within a social network g. ## conclusion we have introduced dynamic contextualized word embeddings that represent words as a function of both linguistic and extralinguistic context. based on a plm, specifically bert, dynamic contextualized word embeddings model time and social space jointly, which makes them advantageous for various areas in nlp. we have trained dynamic contextualized word embeddings on four datasets and showed that they are capable of tracking social and temporal variability in word meaning. besides serving as an analytical tool, dynamic contextualized word embeddings can also be of benefit for downstream tasks such as sentiment analysis. with a rate of 0.2 after each layer of the dynamic component. the number of trainable parameters varies between models trained on different datasets due to differences in |t | and is 134,914,570 for arxiv, 124,990,698 for ciao, 120,028,762 for reddit, and 122,509,730 for yelp. we use a batch size of 4 and perform grid search for the number of epochs n @xmath0, . . . , 7}, the learning rate @xmath1, 3 × 10 -6 }, and the regularization constant λ @xmath2, 1 × 10 -1 }, thereby also determining λ w (section 3.4).