S em E val-2020 Task 1: Unsupervised Lexical Semantic Change Detection (2020)

Abstract: Lexical Semantic Change detection, i.e., the task of identifying words that change meaning over time, is a very active research area, with applications in NLP, lexicography, and linguistics. Evaluation is currently the most pressing problem in Lexical Semantic Change detection, as no gold standards are available to the community, which hinders progress. We present the results of the first shared task that addresses this gap by providing researchers with an evaluation framework and manually annotated, high-quality datasets for English, German, Latin, and Swedish. 33 teams submitted 186 systems, which were evaluated on two subtasks.

## data the task took place in a realistic unsupervised learning scenario. participants were provided with trial and test data, but no training data. the public trial and test data consisted of a diachronic corpus pair and a set of target words for each language. participants' predictions were evaluated against a set of hidden gold labels. the trial data consisted of small samples from the test corpora (see below) and four target words per language to which we assigned binary and graded gold labels randomly. participants could not use this data to develop their models, but only to test the data input format and the online submission format. for development data participants were referred to three pre-existing diachronic data sets: durel @xcite , semcor lsc @xcite and wsc @xcite . in the evaluation phase participants were provided with the test corpora and a set of target words for each language. 2 participants were asked to train their models only on the corpora described in ## annotation we focused our efforts on annotating large and more representative samples for a limited number of words rather than annotating many words. 6 in this section we describe the setup of the annotation for the modern 3 sentence shuffling and lemmatization were done for copyright reasons. participants were provided with start and end positions of sentences. where kubhist did not provide lemmatization (through korp @xcite ) we left tokens unlemmatized. additional pre-processing steps were needed for english: for copyright reasons ccoha contains frequent replacement tokens (10 x '@'). we split sentences around replacement tokens and removed them as a first step in the preprocessing pipeline. further, because english frequently combines various pos in one lemma and many of our target words underwent pos-specific semantic changes, we concatenated targets in the english corpus with their broad pos tag ('target pos'). also, the joint size of the ccoha subcorpora had to be limited to ∼10m tokens because of copyright issues. we started with four annotators per language, but had to add additional annotators later because of a high annotation load and dropouts. the total number of annotators for english/german/swedish was 9/8/5. all annotators were native speakers and present or former university students. for german we had two annotators with a background in historical linguistics, while for english and swedish we had one such annotator. for each target word we randomly sampled 100 uses from each of c 1 and c 2 for annotation (total of 200 uses per target word). 7 if a target word had less than 100 uses, we annotated the full sample. we then mixed the use samples of a target word into a joint set u and annotated u using an extension of the durel framework @xcite @xcite . durel produces high inter-annotator agreement even between non-expert annotators relying on the simple notion of semantic relatedness. pairs of word uses from c 1 and c 2 are annotated on a four-point scale from unrelated meanings (1) to identical meanings (4) (see ## evaluation all teams were allowed a total of 10 submissions, the best of which was kept for the final ranking in the competition. participants had to submit predictions for both subtasks and all languages. a submission's final score for each subtask was computed as the average performance across all four languages. during the evaluation phase, the leaderboard was hidden, as per semeval recommendation. ## participating systems thirty-three teams participated in the task, totaling 53 members. the teams submitted a total of 186 submissions. given the large number of teams, we provide a summary of the systems in the body of this paper. a more detailed description of each participating system for which a paper was submitted is available in appendix b. we also encourage the reader to read the full system description papers. participating models can be described as a combination of (i) a semantic representation, (ii) an alignment technique and (iii) a change measure. semantic representations are mainly average embeddings (type embeddings) and contextualized embeddings (token embeddings). token embeddings are often combined with a clustering algorithm such as k-means, affinity propagation @xcite , (h)dbscan, gmm, or agglomerative clustering. one team uses a graph-based semantic network, one a topic model and several teams also propose ensemble models. alignment techniques include orthogonal procrustes @xcite , vector initialization @xcite , versions of temporal referencing @xcite , and canonical correlation analysis (cca). a variety of change measures are applied, including cosine distance (cd), euclidean distance (ed), local neighborhood distance (lnd), kullback-leibler divergence (kld), mean/standard deviation of co-occurrence vectors, or cluster frequency. ## results as illustrated by team subtask 1 system avg. en de la sv uwb .687 .622 .750 .700 .677 type life-language .686 .703 .750 .550 .742 type jiaxin & jinan .665 .649 .729 .700 .581 type rpi-trust .660 .649 .750 .500 .742 type ug student intern .639 .568 .729 .550 .710 type dcc .637 .649 .667 .525 .710 type nlp@idsia .637 .622 .625 .625 .677 token jct .636 .649 .688 .500 .710 type skurt .629 .568 .562 .675 .710 token discovery team .621 .568 .688 .550 .677 ens. count bas. .613 .595 .688 .525 .645 -tue .612 .568 .583 .650 .645 token entity .599 .676 .667 .475 .581 type ims .598 .541 .688 .550 .613 type cs2020 .587 .595 .500 .575 .677 token uio-uva .587 .541 .646 .450 .710 token nlpcr .584 .730 .542 .450 .613 token maj. bas. .576 .568 .646 .350 .742 cbk .554 .568 .625 .475 .548 token random .554 .486 .479 .475 .774 type uob .526 .568 .479 .575 .484 topic ucd .521 .622 .500 .350 .613 graph rijp .511 .541 .500 .550 .452 type freq. bas. .439 .432 .417 .650 .258 -team subtask 2 system avg. en de la sv ug student intern .527 .422 .725 .412 .547 type jiaxin & jinan .518 .325 .717 .440 .588 type cs2020 .503 .375 .702 .399 .536 type uwb .481 .367 .697 .254 .604 type discovery team .442 .361 .603 .460 .343 ens. rpi-trust .427 .228 .520 .462 .498 type skurt .374 .209 .656 .399 .234 token ims .372 .301 .659 .098 .432 type uio-uva .370 .136 .695 .370 .278 token entity .352 .250 .499 .303 .357 type random .296 .211 .337 .253 .385 type nlpcr .287 .436 .446 .151 .114 token jct .254 .014 .506 .419 .078 type cbk .234 .059 .400 .341 .136 token ucd .234 .307 .216 .069 .344 graph life-language .218 .299 .208 -.024 .391 type nlp@idsia .194 .028 .176 .253 .321 token count bas. .144 .022 .216 .359 -.022 -uob .100 .105 .220 -.024 .102 topic rijp .087 .157 .099 .065 .028 type tue .087 -.155 .388 .177 -.062 token dcc -.083 -.217 .014 .020 -.150 type freq. bas. ug student intern performs best, followed by jiaxin & jinan and cs2020. 16 across all systems, good performance in subtask 1 does not indicate good performance in subtask 2 (correlation between the system ranks is 0.22). however, and with the exception of life-language and cs2020, most top performing systems in subtask 1 also excel in subtask 2, albeit with a slight change of ranking. remarkably, all the top performing systems use static-type embedding models, and differ only in terms of their solutions to the alignment problem (canonical correlation analysis, orthogonal procrustes, or temporal referencing). interestingly, the top systems refine their models using one or more of the following steps: a) computing additional features from the embedding space; b) combining scores from different models (or extracted features) using ensemble models; c) choosing a threshold for changed words based on a distribution of change scores. we conjecture that these additional (and sometimes very original) post-processing steps are crucial for these systems' success. we now briefly describe the top performing systems in terms of these three steps (for further details please see appendix b). uwb (sgns+cca+cd) sets the average change score as the threshold (c). life-language (sgns) represents words according to their distances to a set of stable pivot words in two unaligned spaces, and compares their divergence relative to a distribution of change scores obtained from unstable pivot words (a+c). rpi-trust (sgns+op) extract features (a word's cosine distance, change of distances to its nearest-neighbours and change in frequency), transform each word's feature to a cdf score, and averages these probabilities (a+b+c). jiaxin & jinan (sgns+tr+cd) fits the empirical cosine distance change scores to a gamma quantile threshold, and sets the 75% quantile as the threshold (c). ug student intern (sgns+op) measures change using euclidean distance instead of cosine distance. cs2020 uses sgns+op+cd only as baseline method. an important finding common to most systems is the difference between their performances across the four languages -systems that excel in one language do not necessarily perform well in another. this discrepancy may be due to a range of factors, including the difference in corpus size and the nature of the corpus data, as well as the relative availability of resources in some languages such as english over others. the latin corpus, for example, covers a very long time span, and the lower performance of the systems on this language may be explained by the fact that the techniques employed, especially word token/type embeddings, have been developed for living languages and little research is available on their adaptation to dead and ancient languages. in general, dead languages tend to pose additional challenges compared to living languages @xcite , due to a variety of factors, including their less-resourced status, lack of native speakers, high linguistic variation and non-standardized spelling, and errors in optical character recognition (ocr). other factors that should be investigated are data quality @xcite : while english and latin are clean data, german and swedish present notorious ocr errors. the availability of tuned hyperparameters might have played a role as well: for german, some teams report following prior work such as @xcite . finally, another factor for the discrepancy in performance between languages for any given system is not related to the nature of the systems nor of the data, but due to the fact that some teams focused on some languages, submitting dummy results for the others. type versus token embeddings tables 5 and 6 illustrate the gap in performance between type-based embedding models and the token-based ones. out of the best 10 systems in subtask 1/subtask 2, 7/8 systems are based on type embeddings compared to only 2/2 systems that are based on token embeddings (same holds for each language individually). contrary to the recent success of token embeddings @xcite and to commonly held view that contextual embeddings "do everything better", they are overwhelmingly outperformed by type embeddings in our task. this is most surprising for subtask 1, because type embeddings do not distinguish between different senses, while token embeddings do. we suggest several possible reasons for these surprising results. the first is the fact that contextual embedding is a recent technology, and as such lacks proper usage conventions. for example, it is not clear whether a model should create an average token representation based on individual instances (and if so, which layers should be averaged), or if it should use clustering of individual instances instead (and if so, what type of clustering algorithm etc.). a second reason may be related to the fact that contextual models are pretrained and cannot exclusively be trained on the relevant historical resources (in contrast to type embeddings). as such, they carry additional, and possibly irrelevant, information that may mask true diachronic changes. the results may also be related to the specific preprocessing we applied to the corpora: (i) only restricted context is available to the models as a result of the sentence shuffling. usually, token-based models take more context into account than just the immediate sentence @xcite . (ii) the corpora were lemmatized, while token-based models usually take the raw sentence as input. in order to make the input more suitable for token-based models, we also provide the raw corpora after the evaluation phase and will publish the annotated uses of the target words with additional context. 17 the influence of frequency in prior work, the predictions of many systems have been shown to be inherently biased towards word frequency, either as a consequence of an increasing sampling error with lower frequency @xcite or by directly relying on frequency-related variables @xcite . we have controlled for frequency when selecting target words (recall ## conclusion we presented the results of the first shared task on unsupervised lexical semantic change detection. a wide range of systems were evaluated on two subtasks in four languages relying on a thoroughly annotated data set based on ∼100,000 human judgments. the task setup (unsupervised, no genuine development data, different corpora from different languages with very different sizes, varying class distributions) provided an opportunity to test models in heterogeneous learning scenarios, that was very challenging. hence, both subtasks remain far from solved. however, several teams reach high performances on both subtasks. surprisingly, type embeddings outperformed token embeddings on both subtasks. we suspect that the potential of token embeddings has not yet fully unfolded, as no canonical application concept is available and preprocessing was not optimal for token embeddings. we found that type embeddings are strongly influenced by frequency. hence, one important challenge for future type-based models will be to avoid the frequency bias stemming from the corpus on which they are trained. an important challenge for token-based models will be to understand the reasons for their current low performance and to develop robust ways for their application. we found that change scores in our test sets strongly correlate with polysemy, despite model predictions not showing such strong influence. we believe that this should be pursued in the future by controlling test sets for polysemy. we hope that semeval-2020 task 1 makes a lasting contribution to the field of unsupervised lexical semantic change detection by providing researchers with a standard evaluation framework and highquality data sets. despite the limited size of the test sets, many previously reached conclusions can now be tested more thoroughly and future models can be compared on a shared benchmark. the current test set can also be used to test models that have been trained on the full data available for the participating corpora. data from additional time periods can be utilized by models that need finer granularity for detection, while testing on the two time periods available in the current test sets.