SST - BERT at S em E val-2020 Task 1: Semantic Shift Tracing by Clustering in BERT -based Embedding Spaces (2020)

Abstract: Lexical semantic change detection (also known as semantic shift tracing) is a task of identifying words that have changed their meaning over time. Unsupervised semantic shift tracing, focal point of SemEval2020, is particularly challenging. Given the unsupervised setup, in this work, we propose to identify clusters among different occurrences of each target word, considering these as representatives of different word meanings. As such, disagreements in obtained clusters naturally allow to quantify the level of semantic shift per each target word in four target languages. To leverage this idea, clustering is performed on contextualized (BERT-based) embeddings of word occurrences. The obtained results show that our approach performs well both measured separately (per language) and overall, where we surpass all provided SemEval baselines.

## problem setup consider two corpora c 1 and c 2 for a same language but associated with different time stamps (say, respectively, t 1 and t 2 > t 1 ). let w be a set of target words occurring in both corpora. each target word @xmath0, to be called senses, within the two corpora. a pool of experts annotated a representative amount of occurrences with their corresponding senses. the problem we consider is to characterize the semantic shift related to those senses from one corpus to the other without having access to the expert annotations. in particular, we address the two following two subtasks: we 1 describe two different methods able to address both subtasks. as both methods require a preprocessing step based on transformers, let us start from this preliminary operation. ## preprocessing for the proposed approach, we used embeddings derived from bert (bidirectional encoder representations from transformers) @xcite model to represent the text information in the corpora. bert uses attention mechanism to learn the contextual relations and reads the input bidirectionally. it is an encoder-only model (as the goal is to generate a language model) opposed to transformers (encoderdecoder model) @xcite . bert is trained on two main objectives, masked language model (mlm) and next sentence prediction @xcite . the corpus data is initially segmented at sentence-level and the bert word piece tokenizer is applied on these sentences, to get the token-level representations. bert-base model with twelve transformer layers is used and we derived the final embedding by concatenating the final four layers. if a single word gets split by the tokenizer, we take the average embedding value of the sub-tokens. thus, for each target word we extract the embeddings from all the sentences associated with it from both corpora, c 1 and c 2 . for corpora other than english language, we used multilingual bert models of respective languages. we used the pre-trained model to generate the embeddings for all experiments, since the task is completely unsupervised in nature. let us assume, for each @xmath1, that s j := {s ## method 1: joint clustering vectors of both corpora let us focus on a particular target word @xmath2. accordingly, for the sake of readability, denote its vectors in corpus c j simply as x j := {x ij } n j @xmath3, for each @xmath4, 2. we cluster the whole set of vectors of the two corpora, say x := x 1 ∪ x 2 , and denote as {x k } m @xmath5. note that we cope with hard clustering methods, i.e., ∪ n @xmath6, k 2 = 1, . . . , m, with k 1 = k 2 . for each cluster x k we count how many of its elements belong to x 1 , say n 1,k , and to x 2 , say n 2,k . we call impure a cluster such that both n 1,@xmath7,@xmath8. as we regard the clusters as equivalence classes for the abstract notion of sense, if all the m clusters are impure it means that no new senses appeared in c 2 and no new senses have been lost from c 1 to c 2 . if this is not the case we might have new senses in c 2 , i.e., there is at least a k such that n 1,@xmath9, or, vice versa, an old sense has been lost, i.e., there is at least a k such that n 2,@xmath10. following the guidelines of the semeval shared task, we might set a lower bound n to the number of occurrences of a word in a cluster before deciding to regard it as a new sense. if this is the case the above conditions for the counts equal to zero should be replaced by n j,@xmath11. overall, this procedure corresponds to a sound algorithm to address (s1). we refer to it as m1s1. regarding (s2), after the clustering, we might define a random variable s, to be called the sense variable, whose m states are in one-to-one correspondence with the clusters. the variable denotes how likely is finding an occurrence of w with sense s in a corpus. accordingly, we might use the counts {n j,k } m @xmath12, 2. following a bayesian approach, based on a laplace uniform prior with equivalent sample size σ > 0 @xcite , we have: in such a probabilistic setup, the semantic shift of the target word w between the two corpora can be therefore described by the dissimilarity between the mass functions p 1 (s) and p 2 (s). we measure that by the shannon-jensen distance sj, i.e., a symmetrization of the popular kullback-leibler divergence. this semantic shift of w corresponds therefore to the distance δ, with δ := sj(p 1 , p 2 ) and sj(p 1 , p 2 ) := 1 2 [kl(p 1 , p 2 ) + kl(p 2 , p 1 )] and kl(p 1 , p 2 ) := m @xmath13. note that with the bayesian smoothing in equation (1), we cannot have zero probabilities and degenerate values in the computation of the distance. the overall procedure gives an algorithm to address subtask s2, as this corresponds to sort the elements of w with respect to their value δ. we refer to this procedure as m1s2. ## method 2: separate clustering of the two corpora in this section, while still focusing on a given target word @xmath14, we consider a different approach based on the separate clustering of the two set of vectors x 1 and the optimal matching minimizing the sum of the weights can be computed in cubic time with the classical hungarian algorithm @xcite and the results is a one-to-one correspondence between the clusters, no matter whether proper or dummy, of the two corpora. as a dummy cluster in a corpus has zero distance from all the clusters of the other corpus, the matching returned by the hungarian algorithm is properly minimizing the distance between the proper clusters. two proper clusters in the two corpora matched by the algorithm are intended as representative of the same sense. proper clusters of a corpus pointing to dummy cluster are regarded instead as a new sense appeared in the second corpus only, or old sense occurred in the first corpus only. after the matching, we define a single clustering with m := max{m 1 , m 2 } clusters and proceed exactly as in the previous section. in practice, the vectors of two clusters matched by the hungarian algorithm are assigned to a single, impure, cluster, while those linked to dummy clusters produce pure clusters. we term m2s1 and m2s2 the two algorithms corresponding to the approach discussed in this section to address the two subtasks. next section describes the experimental analysis and evaluation results. as an alternative to the previously explained procedure for handling (s2), based on bayesian approach and shannon-jensen divergence, we consider another approach, exploiting only the number of word occurrences per cluster and corpora. more precisely, assuming that we have k clusters in total and ∀@xmath15, .., k} already calculated n 1,k and n 2,k from each corpora (regardless whether clusters come from single clustering in m1 or after performing optimal cluster matching in m2), we define the coefficient of semantic change of the word (ranking in (s2) terminology), as: ## experimental analysis experimental analysis is performed according to the rules posed by semeval2020 challenge foot_0 organizers, using provided corpora (2) and baselines (3). corpora are provided in four languages: english @xcite , latin (mcgillivray and kilgarriff, 2013), german @xcite and swedish @xcite . for evaluation purposes (as instructed by semeval guidelines), accuracy is exploited for (s1), while spearman coefficient, taking values between -1 (corresponding to negative correlation) and 1 (perfect correlation), was used for (s2). more details can be found in the system description paper @xcite . it is worth mentioning that the upper bound for the number of clusters for k-means which could be retrieved by the silhouette score was set to 10. as explained before, for (s1), the idea was to compare the number of elements of each cluster x k coming from different corpora, say n 1,k and n 2,k , and claim a change in senses if ∃k : n 1,@xmath16,@xmath17. this indeed was the procedure applied for latin corpora. for other languages (with larger sizes of corpora), following the guidelines of the semeval shared task, additional restrictions in terms of lower (n) and upper bounds (n) were set, with the following purpose: word is considered as gaining a new sense, if ∃k : n 1,@xmath18,@xmath19. additionally, suggested values for these bounds were set to @xmath20. the code foot_1 is implemented in python using scikit @xcite and transformers @xcite the experimental results on the corpora over the two subtasks (s1) and ( s2 ) are reported using the methods m1 and m2, in ## conclusion and future work a word can have a different meaning (sense) in different contexts and/or different time periods. despite being quite extensively studied, the problem of identifying words that have changed their meaning over time, particularly in an unsupervised way, still challenges researchers. in this work, we propose two approaches, both of which combine contextualized word embeddings (obtained by bert) and clustering, differing thus only in the way the clustering has been performed. considering obtained clusters as proxies for word meanings allows us to quantify the level of change per each target word in four target languages. the obtained results, especially looking overall, across all target languages, where we are outperforming all provided baselines, demonstrate the usefulness of the suggested approach. as potential directions for future work we plan to investigate various strategies, including different clustering methods and time-wise comparison of target words nearest-neighbours, in an attempt to identify actual word senses more accurately. additionally, we would like to further scrutinize how the linguistic particularities of different corpora might have contributed to the variability of the results.