TUE at S em E val-2020 Task 1: Detecting Semantic Change by Clustering Contextual Word Embeddings (2020)

Abstract: This paper describes our system for SemEval 2020 Task 1: Unsupervised Lexical Semantic Change Detection. Target words of corpora from two different time periods are classified according to their semantic change. The languages covered are English, German, Latin, and Swedish. Our approach involves clustering ELMo embeddings using DBSCAN and K-means. For a more fine grained detection of semantic change we take the Jensen-Shannon Distance metric and rank the target words from strongest to weakest change. The results show that this is a valid approach for the classification subtask where we rank 13th out of 33 groups with an accuracy score of 61.2%. For the ranking subtask we score a Spearman’s rank-order correlation coefficient of 0.087 which places us on rank 29.

## introduction although lexical semantic change has been a topic of interest for historical linguists for well over a hundred years, the technological advances have provided new computational methods to tackle this problem. as a result, numerous systems have been developed for corpora of different languages and time periods, using various evaluation metrics (for a comprehensive survey see @xcite ). however, there is no universal evaluation framework, which makes the assessment across different approaches difficult. diachronic lexical semantic change detection can aid research which involves work with historical data that is becoming accessible with the progress of digitization. semeval 2020 task 1 is an unsupervised lexical semantic change detection task, consisting of two subtasks -a classification and a ranking subtask covering english, german, latin and swedish @xcite . given two corpora from two different time periods, in subtask 1 individual target words have to be classified into the binary categories of sense change or no sense change, where the class sense change does not distinguish between a word losing or gaining sense(s). an example is the english word bit which over time gained the sense of a 'computational unit', additionally to the meaning of 'small quantity'. subtask 2 consists in ranking the target words according to the degree of lexical semantic change. the word with the strongest change is assigned the highest rank. our approach is based on clustering contextual word embeddings with two different algorithms -k-means and dbscan. to our knowledge, this approach has not been used in earlier research. however, @xcite recently published a paper taking a similar approach that we became aware of only after the shared task. they describe a system that clusters contextual word embeddings using k-means. the number and size of the clusters determine whether a word exhibits sense change or not. we calculate the degree of change of a word with the jensen-shannon distance (jsd) @xcite with probability arrays of the cluster distributions. 2 task description for this task we were provided with corpus c1 and corpus c2, where the time period t 1 of c1 lies before the time period t 2 of c2, and a target word list for english, german, latin and swedish (for details about the corpora see @xcite ). the time periods covered in the corpora differ in their span as well as in beginning and end dates. while there is no break between the time periods of t 1 and t 2 of the latin corpora, all other languages have breaks ranging from 47 to 100 years. in addition, whereas the second latin corpus covers a time period of 2000 years, all other c1 and c2 time periods do not exceed 200 years. all corpora were lemmatized, shuffled and formatted to one sentence per line. the target word lists, consisting of nouns and verbs, are of different sizes: with german and latin being the longest ones (48 and 40 target words respectively), followed by english with 37 words and swedish with 31. the word count in the test corpora for the system evaluation ranges from 1.7 to 110 million words. given the corpora c1 and c2, each target word has to be classified into two categories for subtask 1. between the time period of c1 and the time period of c2 a target word either (i) exhibits semantic change, meaning it has lost or gained one or several sense(s), or (ii) there has been no semantic change. for a word to have gained a sense, the sense has to appear a maximum of k times in c1 and at least n times in c2. for a word to have lost a semantic sense, the sense has to appear a maximum of k times in c2 and at least n times in c1. the values for k and n were set differently for latin (@xmath0, @xmath1, @xmath2. for example, assuming the target word bit has two senses, the first one referring to a 'small quantity' and the other one to the 'computational unit', and the sense distribution of the meanings in c1 and c2 is as described in the goal of subtask 2 is to determine the degree of lexical semantic change of each target word and then rank all target words according to this degree, where the item with the highest degree of semantic change is ranked first. using the jensen-shannon distance metric the word sense frequency distributions of bit has a result bound between 0 and 1. whereas the frequency of the first sense of 'small quantity' is decreasing, it is increasing in the second sense of 'computational unit'. the degree of lexical semantic change can capture a change in meaning which is more fine grained than the change captured by the classification scores from subtask 1. ## system overview the key idea of our approach is to use contextualized embeddings to embed the target words and then cluster the resulting vectors. word embeddings are dense vector representations that encode context information. according to the famous principle of @xcite the meaning of a word depends on its surrounding words, i.e. its context. as embeddings of word senses that are similar should lie close to each other in semantic space, it should be possible to identify and discriminate groups of semantically similar word senses by clustering their embeddings. thus, senses (or their representations) assigned to the same cluster should have similar meanings. if two senses end up in different clusters, their meanings are not similar, thus we a semantic change of the sense has occurred. while several architectures of contextual embeddings exist, we make use of the elmo architecture. as our primary task in subtask 1 lies in being able to track if a sense changes or not, we need to be able to differentiate between different senses of a word. since elmo uses all the internal representation layers to compute the final representation, it is supposed to capture more information at different linguistic levels (semantics, syntax, etc.) than simple word embeddings @xcite . a shortcoming of pretrained embeddings is that they are trained on non-lemmatized corpora, unlike the corpora provided by this task which are lemmatized. this might have an influence on the quality of the embeddings we retrieve. ## experimental setup for german, latin and swedish we employ pre-trained elmo representations for many languages which are trained on a corpus of 20 million tokens. @xcite . 3 although there was an available english model from @xcite as well, we decided to use deep contextualized word representations for english, a model implementation of elmo trained on a 1 billion word benchmark, primarily due to the substantially greater corpus size of its training data (peters et al., 2018). 4 in total, our system has three hyperparameters: minimum samples and epsilon for dbscan and t for k-means @xcite which we tune on two embedding settings. 5 while the first setting uses the target word embeddings, the second setting uses embeddings of the context around the target word (i.e., the sentence in which the target word appears, with the target word excluded). however, we only report results of the first settings with target word embeddings due to substantially worse results with context embeddings during the tuning process. we did not have a development set available as the target word predictions of the trial data provided were randomly assigned, meaning that they do not reflect real values. instead, we draw on the binary change score from a data set based on the semcor corpus as development data to tune our english model @xcite . as the latin corpora provided by the organizers contain homonym annotations, which indicate different senses of homonyms, we use those to train latin. 6 for k-means, we obtain the highest accuracy scores with @xmath4.3 for english and @xmath5.7 for latin. for dbscan tuning results in @xmath6, min.@xmath7.5, min.@xmath8. due to the lack of development data for swedish and german we use the settings from english for these two languages. mainly because they also belong to the germanic language family, and moreover, the respective time periods of the corpora resemble each other in comparison to the time period of the latin corpus. the combination of k-means and dbscan that yields the highest score during evaluation are k-means for english and swedish and dbscan for german and latin. the baselines for the tasks are the majority class baseline for subtask 1; subtask 2 has two baseline models: normalized frequency difference and count vectors with column intersection and cosine distance @xcite . ## results for subtask 1, the results are evaluated against binary annotations done by humans. our results for all four languages reach an average accuracy of 61.2%. this corresponds to an overall rank 11 out of 21 participating groups, where the accuracy lies between 41.3% and 68.7%. our best individual accuracy score is the one for latin with 65%. subtask 2 is evaluated using the spearman's rank-order correlation coefficient (spr) against real human annotations as well. our average score over all four languages is 0.087. other participants' scores range from the best score of 0.527 to -0.083. here, our best individual score is 0.388 for german. looking at the predictions our system outputs for english in subtask 1, we see that all target words are classified into the same class which might indicate an error in our system and is probably the reason for the low accuracy score of 56.8%. moreover, our rather low scores for subtask 2 suggest that using the cluster distributions from subtask 1 without any further processing than using jsd is not ideal. ## conclusion we show that tackling unsupervised detection of lexical semantic change by using well known clustering algorithms combined with contextual word embeddings is a valid approach. although our accuracy scores are above the score of the baseline model, our system is far from perfect, as shown by our results for subtask 2. a possible improvement of the current model would be to train elmo on corpora that contain the target words instead of using pre-trained embeddings. this would also resolve the issue of using embeddings trained on unlemmatized data while the actual test data is lemmatized. through applying pre-trained embeddings of unlemmatized data to lemmatized data, part of syntactic and semantic information might be lost. additionally, results could be improved by narrowing down the context window with using only a few words before and after the target word, instead of taking the whole sentence. other aspects for future work might be to experiment with different contextual word embeddings such as bert, and applying clustering algorithms that account for more varied cluster shapes as for example dbscan does.