Sequential Modelling of the Evolution of Word Representations for Semantic Change Detection (2020)

Abstract: Semantic change detection concerns the task of identifying words whose meaning has changed over time. Current state-of-the-art approaches operating on neural embeddings detect the level of semantic change in a word by comparing its vector representation in two distinct time periods, without considering its evolution through time. In this work, we propose three variants of sequential models for detecting semantically shifted words, effectively accounting for the changes in the word representations over time. Through extensive experimentation under various settings with synthetic and real data we showcase the importance of sequential modelling of word vectors through time for semantic change detection. Finally, we compare different approaches in a quantitative manner, demonstrating that temporal modelling of word representations yields a clear-cut advantage in performance.

## introduction identifying words whose lexical meaning has changed over time is a primary area of research at the intersection of natural language processing and historical linguistics. through the evolution of language, the task of "semantic change detection" @xcite @xcite can provide valuable insights on cultural evolution over time @xcite . measuring linguistic change is also relevant to understanding the dynamics in online communities @xcite and the evolution of individuals @xcite . recent years have seen a surge in interest in this area since researchers are now able to leverage the increasing availability of historical corpora in digital form and develop models that detect the shift in a word's meaning through time. however, two key challenges in the field still remain. firstly, there is little work in existing lit-erature on model comparison @xcite @xcite . partially due to the lack of (longitudinal) labelled datasets, existing work assesses model performance mainly in a qualitative manner, without quantitative comparisons against prior work. therefore, it becomes difficult to assess what constitutes an appropriate approach for semantic change detection. secondly, on a methodological front, a large body of related work detects semantically shifted words by pairwise comparisons of their representations in distinct time periods, ignoring the sequential modelling aspect of the task. since semantic change is a time-sensitive process @xcite , considering consecutive vector representations through time -instead of two bins of word representations @xcite -can be crucial to improving model performance @xcite . here we tackle both challenges by approaching semantic change detection as an anomaly identification task. working on embedding representations of words in the english language, we learn their evolution through time via an encoder-decoder architecture. we hypothesize that once such a model has been successfully trained on temporally sensitive sequences of word representations, it will accurately predict the evolution of the semantic representation of any word through time. words that have undergone semantic change will be those that yield the highest errors by the prediction model. our work makes the following contributions: ## related work one can distinguish two directions within the literature on semantic change detection: (a) learning word representations over discrete time intervals (bins) and comparing the resulting vectors and (b) jointly learning word representations across time @xcite @xcite . such representations can be generated via different approaches, such as topic- @xcite , graph- @xcite and neural-based models (e.g., word2vec) -work by @xcite provides an overview of such approaches. in this work we focus on (a) due to scalability issues in learning diachronic representations from very large corpora, as in our case, and -without loss of generality -we utilise pre-trained, neural-based representations. related work in (a) derives word representations w i (@xmath0,..,|t -1|]) across |t | time intervals and performs pairwise comparisons for different values of i. early work used frequency-or co-occurrencebased representations @xcite @xcite . however, leveraging word2vec-based representations @xcite has become the common practice in recent years. due to the stochastic nature of word2vec, orthogonal procrustes (op) @xcite is often applied to the resulting vectors, aiming at aligning the pairwise representations @xcite @xcite @xcite . given two word matrices w k , w j at times k and j respectively, op finds the optimal transformation matrix @xmath1;ω t ω=i ωw k -w j f and the semantic 1 code is available at: https://github.com/ adtsakal/semantic_change_evolution shift level of a word w during this time interval is defined as the cosine distance between the two aligned matrices @xcite . by operating in a linear pairwise fashion, such approaches ignore the time-sensitive and possibly non-linear nature of semantic change. by contrast, @xcite , @xcite , @xcite and @xcite derive time series of a word's level of semantic change to detect semantically shifted words. even though these methods incorporate temporal modelling, they either rely heavily on the linear transformation r @xcite or focus primarily on the generation of temporally-sensitive representations as a means towards capturing semantic change @xcite . a key contribution of our work is that we do not base our methods on pre-defined transformations, but instead propose a model for learning how (any type of) pre-trained word representations vary across time, effectively exploiting the full sequence of a word's evolution. finally, the comparative evaluation of semantic change detection models is still in its infancy. most related work assesses model performance based on artificial tasks @xcite @xcite @xcite or on a few hand-picked examples @xcite , without cross-model comparison. the recently introduced shared tasks semeval task 1 @xcite and diacr-ita @xcite aim at bridging this gap; however, the respective datasets consist of documents split in two distinct time periods, thus not facilitating the study of the sequential nature of semantic change. setting a benchmark for model comparison with real-world and sequential word representations is crucial in this field. ## methods we formulate semantic change detection as an anomaly detection task in the evolution of pretrained word embeddings. we assume that pretrained word vectors w @xmath2, ..., w |t -1| ], where w @xmath3: vocabulary size; d: word representation size) in a historical corpus over |t | time periods, evolve according to a non-linear function f (w t ). 2 by approximating f , we obtain the level of semantic shift of a word w at time t by measuring the similarity between its word representation w t against f (w t ). a low similarity score for a given word implies an inaccurate model prediction (anomaly) and thus a high level of semantic change for the given word. therefore, we can obtain a ranking of the words based on their semantic shift level by ordering them in ascending order of their similarity scores between w t and f (w t ). we approximate f via temporally sensitive deep neural models: (a) an autoencoder, which aims to reconstruct a word's trajectory up to a given point in time i [w 0 , ..., w |i| ] (section 3.1); and (b) a future predictor, which aims to predict future representations of the word [w |i+1| , ..., w |t -1| ] (section 3.2). the two models can be trained individually or (c) in a joint multi-task setting (section 3.3). these models benefit from accounting for sequential word representations across time [w 0 , ..., w |t -1| ], which is better suited for detecting semantically shifted words compared to the common practice of comparing only the first and last word representations [w 0 , w |t -1| ] @xcite . encoded sequence §3.1: reconstructing word representations §3.2: predicting future word representations w 0 w 1 w i-2 w i-1 w r 0 w r i-1 w f i w f t-1 lstms lstms lstms dense dense figure 1: overview of our proposed model: the sequence of the representation of a set of word vectors (vocabulary) over different time steps w 0:i-1 is encoded through two lstm layers and then passed over to a reconstruction (3.1) decoder and a future prediction decoder (3.2). the model is trained by utilising either decoder in isolation, or both of them in parallel (3.3). ## experiments with synthetic data tasks run on artificial data have been used for evaluation purposes in related work @xcite @xcite @xcite @xcite . in this section, we work with artificial data as a proofof-concept of our proposed models -we compare against state-of-the-art and other baseline methods with real data in the next section. here we employ a longitudinal dataset of word representations (4.1) and artificially alter the representations of a small set of words across time (4.2). we then train (4.3) our models and evaluate them on the basis of their ability to identify those words that have undergone (artificial) semantic change (4.4). ## conclusion and future work we introduce three sequential models for semantic change detection that effectively exploit the full sequence of a word's representations through time to determine its level of semantic change. through extensive experiments on synthetic and real data we showcase the effectiveness of the proposed models under various settings and in comparison to state-of-the-art on the uk web archive dataset. importantly, we show that their performance increases alongside the duration of the time period under study, confidently outperforming competitive models and common practices on semantic change. future work can use anomaly detection approaches operating on our model's predicted word vectors to detect anomalies in a word's representation across time. we also plan to investigate different architectures, such as variational autoencoders (kingma and welling, 2014), and incorporate contextual representations @xcite to detect new senses of words. a limitation of our work is that it has been tested on a single dataset, where 65 words have undergone semantic change; testing our models in datasets of different duration and in different languages will provide clearer evidence of their effectiveness.