T emporal T eller at S em E val-2020 Task 1: Unsupervised Lexical Semantic Change Detection with Temporal Referencing (2020)

Abstract: This paper describes our TemporalTeller system for SemEval Task 1: Unsupervised Lexical Semantic Change Detection. We develop a unified framework for the common semantic change detection pipelines including preprocessing, learning word embeddings, calculating vector distances and determining threshold. We also propose Gamma Quantile Threshold to distinguish between changed and stable words. Based on our system, we conduct a comprehensive comparison among BERT, Skip-gram, Temporal Referencing and alignment-based methods. Evaluation results show that Skip-gram with Temporal Referencing achieves the best performance of 66.5% classification accuracy and 51.8% Spearman’s Ranking Correlation.

## introduction task 1 in semeval 2020 is unsupervised lexical semantic change detection. semantic change is defined as the changes in lexical meaning rather than grammatical usage of a language symbol @xcite . it has become an increasingly common phenomenon under the influence of linguistic habits, social environments and many other factors. for example, the word gay previously meant being happy but now mostly means homosexual. those changes are indicators of temporal ideologies as well as language development trends. therefore, studying the evolution of word meanings can assist the research in human language, historical information retrieval and arguably other disciplines. recently, there have been many studies on this topic. one direction is word level change detection. those approaches seek to represent each word as a single vector that reflects its overall meanings. some works @xcite @xcite use co-occurrence matrices with svd. others @xcite @xcite adopt neural embeddings such as skip-gram with negative sampling (sgns) @xcite . however, those word embeddings need to be aligned to the same vector space, which is shown to introduce noise @xcite . another direction is detecting semantic changes from the sense level. these models investigate the different meanings of words, thus can compare in a fine grained manner. in this paper, we will focus on word level change detection. we spotted two problems among those related works. first, no classification mechanisms have yet been discussed. all papers calculate the lexical semantic change, but none has proposed a systematic approach to classify between changed and stable words. the second problem is that the models may be potentially out-of-date. while the leaderboards of many nlp tasks such as reading comprehension are continuously refreshed by new models like pre-training of deep bidirectional transformers (bert) @xcite , the methods adopted in this topic are relatively conventional. though it is not necessarily a problem, we are curious to know whether the latest models could bring extra benefits. aiming at the first problem, we propose gamma quantile threshold (gqt). we propose to describe the cosine distances between word representations using gamma distribution. the threshold for classification will be set as the quantile of the distribution. for the second problem, we introduce contextualized embeddings using bert @xcite . as a comparison, we also leverage temporal referencing @xcite , a promising method in this field. finally, we designed temporalteller, a framework that integrates learning word representations, alignment and threshold calculation for semantic change detection. it can learn word representations via sgns and bert with or without temporal referencing. experiment results show that gqt works well with the existing word embeddings methods, and temporal referencing with sgns achieves the best performance in both classification and ranking subtasks. our contributions can be summarized as follows: ## conclusion in this paper, we systematically studied unsupervised lexical semantics change detection from a computational perspective under our temporalteller framework. we proposed the gamma quantile threshold, which is the first algorithm to fill gap between quantifying semantic change and classifying change status. we exploited the latest advances in temporal referencing. furthermore, we introduced bert into this semantic change detection task. we conducted a comprehensive comparison among these methods in terms of hyperparameters and performance. we found that the combination of sgns with temporal referencing yields the best performance. however, it is sensitive to hyperparameters including embedding dimension and context window size. a suboptimal setting could lead to considerable performance degeneration. bert can give comparable results with base model hyperparameters, but the uncontrollable training and a diverged focus prevents it from being the champion. even with contextualized embeddings and unimportant words removal, it fails to prevail over sgns with tr. it shows that the advantages of powerful general language models like bert does not naturally extends to this task. b a case study in bert + tr 5.0 2.5 0.0 2.5 5.0 7.5 10.0 6 4 2 0 2 4 6 8 put little green bag_nn_old sit room go bag_nn_old baggage much thank green bag_nn_old arm carry fourteen set upon large canvass bag_nn_old hold two bushel take carry bag_nn_old potato sell show small plastic trash bag_nn_new blue silky scarf rain gear large plastic trash bag_nn_new pinch one arm hand grip bag_nn_new couch go take bag_nn_new arm bring two plastic bag_nn_new car history mere fiction_nn_old may read romance choose lord brougham history fearful excitement fiction_nn_old fevered imagination poet tale fiction_nn_old romance religion fiction_nn_old bunyan prose epic appropriate hard perceive partly mere fiction_nn_old false alarm prelate thereby cast amazement today new york time present fiction_nn_new fact fiction_nn_new shop fiction_nn_new consider beauty penguin press contemporary fiction_nn_new create mischief example spend precious fiction_nn_new may draw even unknowing universe read many science fiction_nn_new novel bag_nn_old bag_nn_new fiction_nn_old fiction_nn_new average figure 6: visualizing bag nn and fiction nn in bert experiment (c) figure 5 shows gamma distribution also fits the cosine distances very well in bert + tr. in figure 6 (embeddings projected by pca), the distance between bag nn old and bag nn new is smaller and their contextualized embeddings overlap. looking at their contexts, we can see both bag nn old and bag nn new refer to a container used for carrying things. some relatively faraway occurrences of bag nn new is about plastic, which dose not overstep too much. on the contrary, the distance between fiction nn new and fiction nn old is much larger. for fiction nn, its modern meaning is close to science fiction and contemporary magazines, while its ancient meaning is more related to romantic. indeed, bag nn is classified as stable and fiction nn changed.