U i O - U v A at S em E val-2020 Task 1: Contextualised Embeddings for Lexical Semantic Change Detection (2020)

Abstract: We apply contextualised word embeddings to lexical semantic change detection in the SemEval-2020 Shared Task 1. This paper focuses on Subtask 2, ranking words by the degree of their semantic drift over time. We analyse the performance of two contextualising architectures (BERT and ELMo) and three change detection algorithms. We find that the most effective algorithms rely on the cosine similarity between averaged token embeddings and the pairwise distances between token embeddings. They outperform strong baselines by a large margin (in the post-evaluation phase, we have the best Subtask 2 submission for SemEval-2020 Task 1), but interestingly, the choice of a particular algorithm depends on the distribution of gold scores in the test set.

## introduction words change their meaning over time in all natural languages. the emergence of large representative historical corpora and powerful data-driven semantic models has allowed researchers to track meaning change. semeval-2020 shared task 1 challenges its participants to classify a list of target words into stable or changed (subtask 1) and/or to rank these words by the degree of their semantic change (subtask 2) @xcite . the task is multilingual: it includes four lists of target words, respectively for english, german, latin, and swedish (several dozen words each). each word list is accompanied with two historical corpora of varying size, consisting of texts created in two different time periods. the shared task organisers additionally provided frequency-based and distributional baseline methods. we participated in subtask 2 1 as the uio-uva team and investigated the potential of contextualised embeddings for the detection of lexical semantic change. our systems are based on elmo @xcite and bert @xcite language models, and employ 3 different algorithms to compare contextualised embeddings diachronically. our evaluation phase submission to the shared task ranked 9 th out of 34 participating teams, while our post-evaluation phase submission is the best from those published on the shared task website 2 (but some knowledge of the test sets statistics was needed, see below). in this paper, we extensively evaluate all combinations of architectures, training corpora and change detection algorithms, using 5 test sets in 4 languages. our main findings are twofold: 1) in 3 out of 5 test sets, elmo consistently outperforms bert, while being much faster in training and inference; 2) cosine similarity of averaged contextualised embeddings and average pairwise distance between contextualised embeddings are the two best-performing change detection algorithms, but different test sets show strong preference to either the former or the latter. this preference shows strong correlation with the distribution of gold scores in a test set. while it may indicate that there is a bias in the available test sets, this finding remains yet unexplained. our implementations of all the evaluated algorithms are available at https://github.com/ akutuzov/semeval2020, and the elmo models we trained can be downloaded from the nlpl vector repository at http://vectors.nlpl.eu/repository/ @xcite . ## background lexical semantic change detection (lscd) task is determining whether and/or to what extent the meaning of a set of target words has changed over time, with the help of time-annotated corpora @xcite . lscd is often addressed using distributional semantic models: timesensitive word representations (so-called 'diachronic word embeddings') are learned and then compared between different time periods. this is a fast developing nlp sub-field with a number of influential papers @xcite @xcite @xcite @xcite , and many others). most of the previous work used some variation of 'static' word embeddings where each occurrence of a word form is assigned the same vector representation independently of its context. recent contextualised architectures allow for overcoming this limitation by taking sentential context into account when inferring word token representations. however, application of such architectures to diachronic semantic change detection was up to now limited @xcite @xcite . while all these studies use bert as their contextualising architecture, we extend our analysis to elmo and perform a systematic evaluation of various semantic change detection approaches for both language models. elmo @xcite was arguably the first contextualised word embedding architecture to attract wide attention of the nlp community. the network architecture consists of a two-layer bidirectional lstm on top of a convolutional layer. bert @xcite is a transformer with self-attention trained on masked language modelling and next sentence prediction. while transformer architectures have been shown to outperform recurrent ones in many nlp tasks, elmo allows faster training and inference than bert, making it more convenient to experiment with different training corpora and hyperparameters. ## system overview our approach relies on contextualised representations of word occurrences often referred to as contextualised embeddings). given two time periods t 1 , t 2 , two corpora c 1 , c 2 , and a set of target words, we use a neural language model to obtain contextualised embeddings of each occurrence of the target words in c 1 and c 2 and use them to compute a continuous change score. this score indicates the degree of semantic change undergone by a word between t 1 and t 2 , and the target words are ranked by its value. more precisely, given a target word w and its sentential context @xmath0, ..., v i , ..., v m ) with @xmath1, we extract the activations of a language model's hidden layers for sentence position i. the n w contextualised embeddings collected for w can be represented as the usage matrix u @xmath2, . . . , w nw ). the timespecific usage matrices u 1 w , u 2 w for time periods t 1 and t 2 are used as input to all the tested metrics of semantic change. we use three change detection algorithms: ## experimental setup for each of the 4 languages of the shared task, we train 4 elmo model variants: 1) pre-trained, an elmo model trained on the respective wikipedia corpus (english, german, latin or swedish) foot_1 ; 2) fine-tuned, the same as pre-trained but further fine-tuned on the union of the two test corpora; 3) trained on test, trained only on the union of the two test corpora; 4) incremental, two models-the first is trained on the first test corpus, and the second is the same model further trained on the second test corpus. the elmo models are trained for 3 epochs (except english and latin trained on test and incremental models, for which we use 5 epochs, due to small test corpora sizes), with the lstm dimensionality of 2048, batch size 192 and 4096 negative samples per batch. all the other hyperparameters are left at their default values. 5 for bert, we use the base version, with 12 layers and 768 hidden dimensions. 6 for english, german and swedish, we employ language-specific models: bert-base-uncased, bert-base-german-cased, and af-ai-center/bert-base-swedish-uncased. for latin, we resort to bert-base-multilingual-cased, since there is no specific latin bert available yet. given the limited size of the test corpora (in the order of 10 8 word tokens at max), we do not train bert from scratch and only test the pre-trained and fine-tuned bert variants. the fine-tuning is done with bert's standard objective for 2 epochs (english was trained for 5 epochs). we configure bert's wordpiece tokeniser to never split any occurrences of the target words (some target words are split by default into character sequences) and we add unknown target words to bert's vocabulary. we perform this step both before fine-tuning and before the extraction of contextualised representations. at inference time, we use all elmo and bert variants to produce contextualised representations of all the occurrences of each target word in the test corpora. for the incremental variant, the representations for the occurrences in each of the two test corpora are produced using the respective model trained on this corpus. the resulting embeddings are of size 12 × 768 and 3 × 512 for bert and elmo, respectively. we employ three strategies to reduce their dimensionality to that of a single layer: 1) using only the top layer, 2) averaging all layers, 3) averaging the last four layers (only for bert embeddings, as this aggregation method was shown to work on par with the all-layers alternative in @xcite ). finally, to predict the strength of semantic change of each target word between the two test corpora, we feed the word's contextualised embeddings into the three algorithms of semantic change estimation described in section 3. we then compute the spearman correlation of the estimated change scores with the gold answers. this is the evaluation metric of subtask 2, and we use it throughout our experiments. ## results our submission. in our 'official' shared task submission in the evaluation phase we used top-layer elmo embeddings with the cosine similarity change detection algorithm for all languages. english and german elmo models were trained on the respective wikipedia corpora. for swedish and latin, pretrained elmo models were not available, so we trained our own models on the union of the test corpora. this combination of architectures and algorithms was chosen based on our preliminary experiments with the available human-annotated semantic change datasets for english @xcite , german @xcite and @xcite . the resulting spearman correlations were 0.136 for english, 0.695 for german, 0.370 for latin, and 0.278 for swedish. with an average codalab score of 0.37, this submission ranked 9 th out of 34 teams in the evaluation phase. we were aware that the submitted setup was likely sub-optimal as it did not include the fine-tuned model variant. after the official submission deadline (in the post-evaluation phase), we finished training and fine-tuning all of our language models. their systematic evaluation is the main contribution of this paper. current results. the average scores of all the tested configurations across 4 languages are given in judging only from the average correlation scores, contextualised embeddings do not seem to outshine their static counterparts, especially considering that both elmo and bert are more computationally demanding than cbow. however, closer analysis of per-language results shows that in fact the contextualised approaches outperform the cbow procrustes baseline by a large margin for each of the shared task test sets. although consistency across languages (3 out of 4) is an important benefit of the cbow procrustes approach, with the right choice of apd or prt, contextualised embeddings can improve spearman correlation coefficients by up to 50%. this is not a language-specific property: the english gems test set does not behave like the english test set from the shared task. in fact, one can observe 3 groups of test sets with regards to their statistical properties and to the method they favour: group 1 (latin and german) exhibits rather uniform gold score distributions and prefers prt; group 2 (english and swedish) is characterised by more skewed gold score distributions and prefers apd; group 3 (gems) is in between, with no clear preference. interestingly, the method which produces a more uniform predicted score distribution (apd) works better for the test sets with skewed gold distributions, and the method which produces a more skewed predicted score distribution (prt) works better for the uniformly distributed test sets (as can be seen in the appendix). furthermore, there is perfect negative correlation (ρ = -1) between the median gold score of a test set and the performance of the apd algorithm with fine-tuned elmo models on this test set. the same correlation for the apd performance is not significant but strictly negative. we currently do not have a plausible explanation for this behaviour. ## conclusion our experiments for the semeval-2020 shared task 1 (subtask 2) show that using contextualised embeddings to rank words by the degree of their semantic change yields strong correlation with human judgements, outperforming approaches based on static embeddings. models pre-trained on large external corpora and fine-tuned on the historical test corpora produce the highest correlation results, with elmo slightly but consistently outperforming bert as a contextualiser. inverted cosine similarity between averaged contextualised embeddings and the average pairwise cosine distance between contextualised embeddings turned out to be the best semantic change detection algorithms. an interesting finding is that the former algorithm favours the test sets with uniform gold score distribution, while the latter works best with the test sets where the gold score distribution is skewed towards low values. this distinction is not related to the language of the test set, or any other linguistic or statistical property of the test sets we looked at. while slightly different human annotation protocols can be in play here, we believe this dependency between the gold scores distribution and the performance of semantic change detection systems deserves to be investigated further in future work. we also intend to explore the possibilities to improve our best-performing methods (prt and apd), especially with regards to removing the outlier embeddings before calculating the semantic change score. the grouping differences can be quantified with respect to the median gold score (after unitnormalisation). figure 2 shows the dependency of the prt and apd performance on the median score of the gold test set. the dots here are the performance values of prt or apd algorithms on different test sets. english and swedish test sets are in the left part of the plot with the median gold scores of 0.200 and 0.203 correspondingly. german, gems and latin are on the right with 0.266, 0.267 and 0.364 correspondingly. there is a perfect negative spearman correlation between the median gold scores of these 5 test sets and the performance of apd semantic change detection algorithm on each of them (with fine-tuned elmo embeddings).