Enriching Word Embeddings with Temporal and Spatial Information (2020)

Abstract: The meaning of a word is closely linked to sociocultural factors that can change over time and location, resulting in corresponding meaning changes. Taking a global view of words and their meanings in a widely used language, such as English, may require us to capture more refined semantics for use in time-specific or location-aware situations, such as the study of cultural trends or language use. However, popular vector representations for words do not adequately include temporal or spatial information. In this work, we present a model for learning word representation conditioned on time and location. In addition to capturing meaning changes over time and location, we require that the resulting word embeddings retain salient semantic and geometric properties. We train our model on time- and location-stamped corpora, and show using both quantitative and qualitative evaluations that it can capture semantics across time and locations. We note that our model compares favorably with the state-of-the-art for time-specific embedding, and serves as a new benchmark for location-specific embeddings.

## introduction the use of word embeddings as a form of lexical representation has transformed the use of natural language processing for many applications such as machine translation @xcite and language understanding @xcite . the changing of word meaning over the course of time and space, termed semantic drift, has been the subject of long standing research in diachronic linguistics @xcite . additionally, the emergence of distinct geographically-qualified english varieties (e.g., south african english) has given rise to salient lexical variation giving several english words different meanings depending on the geographic location of their use, as documented in studies on world englishes @xcite . considering the multiplicity of meanings that a word can take over the span of time and space owing to inevitable linguistic, and sociocultural factors among others, a static representation of a word as a single word embedding seems rather limited. take the word apple as an example. its early to near-recent mentions in written documents referred only to a fruit, but in the recent times it is also the name of a large technology company. another example is the title for the head of government, which is "president" in the usa, and is "prime minister" in canada. naturally, we expect that one word should have different representations conditioned on the time or location. in this paper, we study how word embeddings can be enriched to encode their semantic drift in time and space. extending a recent line of research on time-specific embeddings, including the works by @xcite we propose a model to capture varying lexical semantics across different conditions-of time and location. a key technical challenge of learning conditioned embeddings is to put the embeddings (derived from different time periods or geographical locations) in the same vector space and preserve their geometry within and across different instances of the conditions.traditional approaches involve a two-step mechanism of first learning the sets of embeddings separately under the different conditions, and then aligning them via appropriate transformations @xcite @xcite . a primary limitation of these methods is their inadequate representation of word semantics, as we show in our comparative evaluation. another approach to conditioned embedding uses a loss function with regularizers over word embeddings across conditions for their smooth trajectory in the vector space @xcite . however, its scope is limited to modeling semantic drift over only time. we propose a model for general conditioned embeddings, with the novelty that it explicitly preserves embedding geometry under different conditions and captures different degrees of word semantic changes. we summarize our contributions below. the second approach-joint training-aims to guarantee the alignment of embeddings in the same vectors space so that they are directly comparable. compared with the previous category of ap-proaches, the joint learning of time-stamped embeddings has shown improved abilities to capture semantic changes across time. bamler and mandt used a probabilistic model to learn time-specific embeddings @xcite . they make a parametric assumption (gaussian) on the evolution of embeddings to guarantee the embedding alignment. yao et al. learned embeddings by the factorization of a positive pointwise mutual information (ppmi) matrix. they imposed l2 constraints on embeddings from neighboring time periods for embedding alignment @xcite . rosenfeld and erk proposed a neural model to first encode time and word information respectively and then to learn time-specific embeddings @xcite . dubossarsky et al. aligned word embeddings by sharing their context embeddings at different times @xcite . some recent works fall in the third category, retrieving contextualized representations from pretrained models such as bert @xcite as time-specific sense embeddings of words @xcite . these pretrained embeddings are limited to the scope of local contexts, while we learn the global representation of words in a given time or location. the underlying mathematical models of these previous works on temporal embeddings are discussed in the supplementary material. our model belongs to the second category of joint embedding training. different from previous works, our embedding is based on a model that explicitly takes into account the important semantic properties of time-specific embeddings. embedding with spatial information. lexical semantics is also sensitive to spatial factors. for example, the word denoting the head of government of a nation may be used differently depending on the region. for instance, the words can range from president to prime minister or king depending on the region. language variation across regional contexts has been analyzed in sociolinguistics and dialectology studies (e.g., @xcite ). it is also understood that a deeper understanding of semantics enhanced with location information is critical to location-sensitive applications such as content localization of global search engines @xcite . some approaches towards this have included, a latent variable model proposed for geographical linguistic variation @xcite and a skip-gram model for geographically situated language @xcite . the current study is most similar to @xcite with the overlap in our intents to learn location-specific embeddings for measuring semantic drift. most studies on location-dependent language resort to a qualitative evaluation, whereas @xcite resorts to a quantitative analysis for entity similarity. however, it is limited to a given region without exploring semantic equivalence of words across different geographic regions. to the extent we are aware, this is the first study to present a quantitative evaluation of word representations across geographical regions with the use of a dataset constructed for the purpose. ## model we now introduce the model on which the condition-specific embedding training is based in this section. we assume access to a corpus divided into sub-corpora based on their conditions (time or location), and texts in the same condition (e.g., same time period) are gathered in each sub-corpus. for each condition, the co-occurrence counts of word pairs gathered from its sub-corpus are the corpus statistics we use for the embedding training. we note that because these sub-corpora vary in size, we scale the word co-occurrences of every condition so that all sub-corpora have the same total number of word pairs. we term the scaled value of word co-occurrences of word w i and w j in condition c as x i,j,c . a static model (without regard to the temporal or spatial conditions) proposed by arora et al. provides the unifying theme for the seemingly different embedding approaches of word2vec and glove. in particular, it reveals that corpus statistics such as word co-occurrences could be estimated from embeddings. inspired by this, we proposed a model for conditioned embeddings, and characterize such a model by its ability to capture the lexical semantic properties across different conditions. ## experiments in this section, we compare our condition-specific word embedding models with corresponding state-across time the, in, to, a, of, it, by, with, at, was, are, and, on, who, for, not, they, but, he, is, from, have, as, has, their, about, her, been, there, or, will, this, said, would across regions in, from, at, could, its, which, out, but, on, all, has, so, is, are, had, he, been, by, an, it, as, for, was, this, his, be, they, we, her, that, and, with, a, of, the ## conclusion we studied a model to enrich word embeddings with temporal and spatial information and showed how it explicitly encodes lexical semantic properties into the geometry of the embedding. we then empirically demonstrated how the model captures language evolution across time and location. we leave it to future work to explore concrete downstream applications, where these time-and locationsensitive embeddings can be fruitfully used.