DCC -Uchile at S em E val-2020 Task 1: Temporal Referencing Word Embeddings (2020)

Abstract: We present a system for the task of unsupervised lexical change detection: given a target word and two corpora spanning different periods of time, automatically detects whether the word has lost or gained senses from one corpus to another. Our system employs the temporal referencing method to obtain compatible representations of target words in different periods of time. This is done by concatenating corpora of different periods and performing a temporal referencing of target words i.e., treating occurrences of target words in different periods as two independent tokens. Afterwards, we train word embeddings on the joint corpus and compare the referenced vectors of each target word using cosine similarity. Our submission was ranked 7th among 34 teams for subtask 1, obtaining an average accuracy of 0.637, only 0.050 points behind the first ranked system.

## introduction in the past decade we have seen a rise in academic work on the automatic detection of lexical semantic change @xcite . however, many of these studies are difficult to compare because of differences in evaluation procedure, languages, corpora, and time periods. semeval 2020 task 1 aims to introduce a simple evaluation framework for unsupervised lexical semantic change detection @xcite . the task focuses on four languages: german, english, swedish, and latin. for each language, two corpora (c 1 and c 2 ) are provided, each of those spanning different periods of time (t 1 and t 2 ). the task is divided into two subtasks described below. systems are evaluated against ground truth data annotated by human native speakers (except for latin, which was annotated by scholars of latin) @xcite . in this paper, we present our system for detecting semantic change of words in different periods of time. our system was solely designed for subtask 1. the rationale of our approach is to obtain compatible representations of target words in different periods of time. this is done by concatenating corpora of different periods and performing a temporal disambiguation of target words. afterwards, we train word embeddings on the joint corpus and compare the disambiguated vectors of each target word using cosine similarity. this process is known as temporal referencing and was proposed in @xcite . our submission was ranked 6th among 33 teams for subtask 1, obtaining an average accuracy of 0.637, only 0.050 points behind the first ranked system. temporal referencing is a very simple technique that despite its simplicity yielded competitive results and outperformed baselines across all languages. ## related work linguists are well aware that language, even when viewed as a synchronous entity, is full of variation at all linguistic levels. in spoken language, this kind of variation is the norm. the detection of lexical change is complex due to the fact that the variation observed in lexical form between different corpora is not necessarily due to diachronic causes, as it may also be due to other factors such as the presence of regional dialects. one of the most challenging aspects of comparing word embeddings trained on different periods of time is alignment, which is the process of making embeddings models trained on different corpora, compatible and comparable. specifically, most cost functions for training word representations are invariant to rotations. consequently, the learned embeddings of independent models i.e., models trained from corpora spanning different periods of time, may not be placed in the same latent space. this problem is known as the alignment problem. in general, most of the previous approaches to studying lexical change follow a similar two-step methodology: first compute static word embeddings in each separate period of time, and then find a way to align the word embedings across different time periods @xcite . @xcite proposed the temporal referencing method for lexical semantic change on which this work is based. a single embedding model is trained on a corpus that combines various diachronic corpora in which the target words (i.e., words on which semantic change will be assessed) are disambiguated according to the corpus in which they occur. two models are compared: ppmi @xcite and skip-gram negative sampling (sgns) @xcite , in two settings: 1) using alignment and 2) using temporal referencing. results showed that the sgns model trained with temporal referencing exhibits significantly less noise than its aligned counterpart, and for the ppmi model, temporal referencing significantly reduced the noise level, but to a lesser extent than for sgns. the work of @xcite is based on context vectors to identify semantic change over time. a co-occurrence matrix with local mutual information (lmi) scores is built @xcite . then, the distributional similarity of each word pair is measured as the cosine product of the corresponding context vectors. a partitions of google book ngram corpus spanning from 1960s and 1990s is used for evaluation. the main finding of the study is that the proposed method can successfully detect the semantic change of words, as well as cases of major diachronic context change. @xcite developed a method employing prediction-based word embedding models to trace diachronic semantic shifts. they used incremental updates and skip-gram with negative sampling (sgns) @xcite as the embedding model. the authors identified words that have changed and the periods during which they changed. in @xcite , the authors propose the dynamic skip-gram model, which is a bayesian probabilistic model that combines the skip-gram architecture with a latent continuous time series. the method is jointly trained over multiple time period and does not require an alignment step. the experimental results showed that both dynamic skip-gram filtering (which conditions only on past observations) and dynamic skip-gram smoothing (which uses all data) lead to smoothly changing embedding vectors that are better at predicting word-context statistics at held-out time steps. in @xcite , a dynamic statistical model was developed to learn time-aware word vector representations. this model can simultaneously learn time-aware embeddings avoiding the need for alignment. this model is trained on a corpus of new york times articles. additionally, multiple evaluation strategies of temporal word embeddings are conducted. the qualitative and quantitative tests indicate that the proposed method not only reliably captures the evolution of words over time, but also consistently outperforms state-of-the-art temporal embedding approaches on both semantic accuracy and alignment quality. another way of creating dynamic embedding was developed in @xcite based on embeddings of exponential families and latent variables with a random walk gaussian drift. the key idea is to restrict the embedding vectors to a single a time slice @xcite while sharing the context vectors across all time points. the authors experimented with three diachronic corpora: 1) arxiv's machine learning papers @xcite @xcite @xcite @xcite @xcite , 2) acm's computer science abstracts , and 3) u.s. senate speeches . the results from this study suggest that dynamic embeddings provide a better fit than their static counterparts for working with diachronic corpora and that dynamic embeddings can also help identify ways in which language changes. ## system description our system implements the temporal referencing method @xcite , which is based on the idea of obtaining compatible representations of target words in different periods of time. this is done by concatenating the two corpora c 1 and c 2 coming from different periods and performing a temporal referencing of target words. it is important to remark that our system was designed only to participate in subtask 1. our hypothesis is that if a word exhibits a semantic change over time then it representation should change too. however, if we independently train word representation for each period of time, our vectors would not be comparable (because of the stochasticity of neural networks). one approach would be to perform an alignment on the different embedding models. in contrast, we concatenate for each language the two temporal corpora (c 1 and c 2 ) and referenced target words by appending special temporal symbols to their tokens. next, we describe our own implementation of the temporal referencing method. the process is illustrated in figure 1 . the first step is to merge both temporal corpora for each language. then, for each target word, we add a special symbol to all its appearances in both corpora, the symbol denoting the corresponding corpus or period. for example, the target word "attack" becomes "attack t1" in c 1 and "attack t2" in c 2 . afterwards, we proceed to merge both corpora, and in this way old and new occurrences of a target word occur in contexts of words coming from a single vocabulary of non-target terms. the next step consists of training a distributed representation of the words from the joined corpus. we use the continuous bag of words (cbow) model with negative sampling to obtain our word vectors @xcite . consequently, our system does not need to align the word embeddings across different time spans. finally, in order to classify whether a target word exhibits semantic change, we calculate the cosine similarity between the two vectors of each target word. we use a threshold parameter λ to convert the similarities into binary classes 1 (semantic change) and 0 (no semantic change) foot_0 . the value of λ is determined experimentally. ## conclusions we presented a system for the automatic detection of semantic change in words occurring in two different periods of time. our system implements the temporal referencing method @xcite in which compatible representations of target words are obtained and compared by concatenating corpora of different periods and performing a temporal disambiguation of target words. our submission, in addition to outperforming both baselines, ranked 6th place in the binary classification subtask of semeval 2020 task 1. these results provide further evidence that temporal referencing is a strong and simple approach for lexical semantic change detection. for future work we will explore using representations of words obtained from deep contextualized models such as @xcite and bert @xcite . we will also use supervised approaches for learning a similarity metric for our target words trained on large synthetically generated data.