D ia S ense at S em E val-2020 Task 1: Modeling Sense Change via Pre-trained BERT Embeddings (2020)

Abstract: This paper describes DiaSense, a system developed for Task 1 ‘Unsupervised Lexical Semantic Change Detection’ of SemEval 2020. In DiaSense, contextualized word embeddings are used to model word sense changes. This allows for the calculation of metrics which mimic human intuitions about the semantic relatedness between individual use pairs of a target word for the assessment of lexical semantic change. DiaSense is able to detect lexical semantic change in English, German, Latin and Swedish (accuracy = 0.728). Moreover, DiaSense differentiates between weak and strong change.

## introduction task 1 of semeval-2020 @xcite is concerned with the unsupervised detection of lexical semantic change (lsc) as reflected by word sense changes over time. more broadly, lsc refers to changes in meaning of a lexical item. a meaning change is manifested in the gain or loss of a particular meaning of a word which indicates an increase or decrease in polysemy @xcite . a well-known example for lsc which is cited in @xcite is the historical evolution of the english word hound which changed from being the general word for 'dog' to referring to only a specific kind of 'dog' @xcite ). meanwhile, dog changed from describing a specific type of 'dog' to becoming the general term for 'dog' @xcite ). moreover, senses can become obsolete and be lost overall, while cultural changes may drive the evolution of new senses. the main task in semeval-2020 task 1 is to identify and evaluate lsc in a set of target words between two text corpora stemming from two different time periods t 1 (earlier period) and t 2 (later period). the investigated languages are german, english, swedish and latin. the task is split into two subtasks: (i) a binary classification task, where it has to be determined whether a target word lost/gained senses between t 1 and t 2 or not, and (ii) a ranking task, where target words are ranked according to their degree of lsc (a higher rank indicates stronger change). the system presented in this paper is called diasense and addresses the lsc tasks by modeling the different senses of a word via contextualized word embeddings using pre-trained bert @xcite . 1 the binary classification and the ranking task are approached by transferring the measures of change suggested by @xcite , which are originally based on human annotated values for meaning relatedness, to change metrics calculated on the basis of differences between target word embeddings. diasense is able to detect lsc in the majority of cases in all four languages (average accuracy 0.728). 2 although the results produced for the ranking task only show a weak correlation with the gold data (spearman's ρ = 0.337), diasense is able to distinguish between strong and weak change. ## related work over recent years, research on lsc has seen an increasing use of computational methods (see @xcite and @xcite for detailed overviews). the methods applied for lsc detection are manifold, but can be grouped into three classes according to the type of meaning representation involved @xcite : (i) semantic vector spaces, (ii) topic distributions, (iii) sense clusters. in semantic vector space approaches, each target word is represented as a vector at each time stage. the vector representations are typically based on bag-of-words approaches and represent a co-occurrence statistics of a word with its context words. common methods employed for computing vectors from co-occurrence statistics are positive pointwise mutual information (ppmi), which measures co-occurrence strength, and singular value decomposition (svd), for dimensionality reduction; see, e.g., @xcite , @xcite , @xcite , @xcite . moreover, word embeddings as generated via the skip-gram with negative sampling (sgns) technique @xcite , i.e., word2vec, and glove embeddings @xcite have been applied to lsc detection, e.g., @xcite and @xcite . as measure of lsc, similarity across time periods is assessed via calculating the distance/similarity between vectors, using, e.g., cosine distance @xcite , or alternatively via computing differences in the contextual dispersion of the vectors (see, e.g., @xcite ). in approaches where meaning is represented via topic distributions @xcite @xcite , word senses are derived from topic models based on, e.g., latent dirichlet allocation (lda; @xcite ) and hierarchical dirichlet processes (hdp, @xcite ). furthermore, the dynamic topic model scan was specifically developed for the investigation of lexical change @xcite . with topic modeling, lsc is usually assessed via a frequency-based novelty score assigned to the senses. sense clustering based approaches follow similar principles, but are used less often; e.g., @xcite . recently, efforts have been made towards developing evaluation standards and datasets for lsc @xcite @xcite @xcite . for example, schlechtweg and im walde (2020) generate simulations of lsc on the basis of synchronic data, providing a testbed for diachronic lsc, while @xcite provide human annotations via the diachronic usage relatedness (durel) dataset. the current pitfall of the existing works on lsc is the lack of a common state-of-the-art evaluation task which makes the comparison of methods difficult. this shortcoming is addressed by semeval-2020 task 1. ## system description diasense measures change by combining word sense representations generated via bert with lsc metrics which are based on the calculation of cosine distance as detailed in the following. ## results diasense has been significantly improved after the publication of the ground truth for semeval-2020 task 1. before this, the system showed a comparably low performance in the evaluation phase, with an average accuracy of 0.554 in the classification task (rank 17 of 21) and ρ = 0.234 in the ranking task (rank 14 of 21); see @xcite for the full rankings. we attribute the low performance in the binary classification to two factors: for one, @xmath0. we experimented with approaches to automatically determine k, but did not arrive at a suitable solution. for another, cluster initialization turned out to be difficult with the pre-trained bert embeddings, since distances between the embeddings are generally low (cf. @xcite on clustering issues with pre-trained bert embeddings). in addition, the frequency threshold employed for identifying change in the clustering was arbitrarily defined. the low performance in the ranking task could be the result of averaging the embeddings, where context variation might substantially bias the resulting vectors. given these shortcomings, we decided to opt for alternative ways of measuring lsc, experimenting with ∆later and ∆compare instead. currently (post-evaluation), the system ranks third in the binary classification with an overall average accuracy of 0.728 (english: 0.649, german: 0.771, latin: 0.750, swedish: 0.742). 10 in the ranking task, diasense occupies rank 14 with ρ = 0.337 (english: 0.293, german: 0.414, latin: 0.343, swedish: 0.300). exemplary, we discuss the results for german in the following. 11 for german, the words with the highest positive ∆later (innovative meaning change) are abgebrüht 'boiled out/indifferent', abdecken 'cover/unroof/blanket', tier 'animal', festspiel 'festival', abbauen 'win (mining)/reduce', artikulieren 'articulate/enunciate', titel 'title' and rezeption 'reception' (see figure 1 left). in the ground truth, abgebrüht, tier, festspiel, and titel are not classified as change. we can confirm tier and titel as false positive. tier and titel show high standard deviations in both t 1 and t 2 (with @xmath1.04), indicating that they exhibit a high context variation overall instead of undergoing a meaning change. however, standard deviation does not provide insights into whether festspiel and abgebrüht are false positives. instead, festspiel shows a large difference between t 1 and t 2 based on a frequency effect (51 occurrences in c 1 , > 500 occurrences in c 2 ). yet, since data sparsity is an inherent problem of historical corpora, we can not exclude festspiel as easily. moreover, abgebrüht indeed shows lsc on the basis of our data: in t 1 , abgebrüht is almost exclusively used as participle of the verb abbrühen 'boil out', while it occurs mostly as adjective with the more figurative meaning 'indifferent' in t 2 . the target words überspannen 'span/overstretch/straddle', fuß 'foot', abgesang 'last verse (minnesong)/swansong', schmiere 'grease/lookout', knotenpunkt 'junction/intersection', ohrwurm 'earwig/catchy tune' have the lowest negative values (reductive meaning change), see figure 1 -right. similarly to abgebrüht, fuß is not classified as undergoing lsc in the gold data, but can in principle be identified as change: while in t 1 fuß is still frequently used as measure unit, this meaning only occurs scarcely in t 2 . overall, the system performs well when it comes to the identification of large changes. for example, ohrwurm, which shows the highest (absolute) ∆later value in german, changed quite drastically from being mainly used in the meaning of 'earwig' in t 1 to almost exclusively denoting a 'catchy tune' in t 2 . however, the system fails to identify smaller scale changes such as, e.g., manschette 'sleeve/cuff', where meanings are close and occur in similar contexts. diasense performs less well in the ranking task as in the classification. however, although the correct ranking could not be identified, diasense puts the target words into similar regions, i.e., words with high ∆compare values (e.g., ohrwurm, abgebrüht) generally rank high, indicating strong change and vice versa. moreover, without normalizing compare, titel and tier ranked highest -an error which was avoided by using ∆compare instead. ## conclusion in this paper, we presented diasense, a system developed for semeval-2020 task 1. based on contextualized word embeddings, diasense is able to identify change in english, german, latin and swedish, while also differentiating between weak and strong change. our approach leverages the strength of pre-trained bert embeddings for modeling word senses language-independently and avoids the necessity of large amounts of training data which is beneficial for historical linguistic work. moreover, we were able to translate metrics developed to capture human intuitions about meaning relatedness into automated measures of lsc. still, diasense was not able to predict the correct ranking in terms of degrees of lsc. we will address this issue in future research, experimenting with further measures and techniques.