R u S em S hift: a dataset of historical lexical semantic change in R ussian (2020)

Abstract: We present RuSemShift, a large-scale manually annotated test set for the task of semantic change modeling in Russian for two long-term time period pairs: from the pre-Soviet through the Soviet times and from the Soviet through the post-Soviet times. Target words were annotated by multiple crowd-source workers. The annotation process was organized following the DURel framework and was based on sentence contexts extracted from the Russian National Corpus. Additionally, we report the performance of several distributional approaches on RuSemShift, achieving promising results, which at the same time leave room for other researchers to improve.

## introduction language is a constantly changing system by its nature, since it is a method of communication and a social instrument. as such, it should meet the needs of the speakers, and thus it adapts to changes in the society and the ever-changing world. as a part of language, lexical meaning also evolves over time, with words undergoing diachronic (or temporal) semantic shifts @xcite . tracing semantic change can be important either in itself, as a linguistic study, or for practical downstream applications, for example, in socio-linguistic research. manual analysis of such shifts is timeconsuming and laborious even after the emergence of large representative corpora, since one needs to look through a lot of examples and lexicographic resources which often do not record current lexical changes in language due to limited resources. thus, researchers are trying to model these processes using advanced computational approaches often based on distributional semantics and dense word embeddings @xcite . however, this is still mostly done for english: often simply because manually annotated test data is not available for other languages. recently, consistently annotated lexical semantic change test sets for multiple languages started to appear; see, for example, @xcite . in this paper, we continue this vein of work by presenting rusemshift. rusemshift 1 is the first historical semantic change dataset for russian annotated according to the durel framework @xcite using a large crowd-sourcing platform, instead of personal intuitions of individual researchers. it allows to evaluate semantic change detection systems by their ability to estimate the shifts which occurred to russian words either after 1917 (the fall of the russian empire) or after 1990 (the fall of the soviet union). the rest of the paper is organized as follows: in section 2, we put our research in the context of the related work. in section 3, we present the employed corpora and the process of the dataset creation. section 4 describes the annotation itself. in section 5, we empirically evaluate several existing semantic change detection algorithms (based on static and contextualized embeddings) on rusemshift to check its sanity. section 6 summarizes our contributions and outlines future research. ## 2 related work works on language change from general linguistics like @xcite or @xcite as a rule contain only a small number of hand-picked examples. the datsemshift database @xcite features more than 4 000 semantic shifts across 800 languages. but it is focused on cognitive proximities between pairs of linguistic meanings (with a limited set of pre-defined senses): in this paradigm, a semantic shift is just a case of extended polysemy. the datsemshift database is extremely useful for identifying recurring cross-linguistic semantic shifts, but it is difficult to employ it for evaluation of unsupervised semantic change detection systems. to our knowledge, the first russian test set to evaluate lexical semantic change detection systems was created by @xcite . they used prior linguistic work to manually collect russian words which changed their meaning from the pre-soviet times through the soviet times. @xcite also employed static word embedding models trained on the corresponding russian diachronic corpora to detect semantic change for the words from the dataset (distinguishing changed words from stable ones). they concluded that kendall's τ (kendall, 1948) and jaccard similarity @xcite however, these datasets suffer from two serious issues. first, they are either produced by a single person @xcite or annotated by the paper authors themselves @xcite . this makes them inherently subjective. second, each of this datasets is created and annotated in a different manner, making them inconsistent and not comparable nor to themselves neither to similar efforts made for other languages. in addition, the test set from @xcite features only binary labels (shifted or stable), making it useless for research in graded semantic change detection. the rusemshift test set we present in this paper makes partial use of these existing test sets, but all the words are re-annotated from scratch. in rusemshift, we adhere to the language-agnostic diachronic usage relatedness (durel) semantic change annotation methodology proposed in @xcite . the intended use of the durel framework is the creation of consistently and robustly annotated test sets containing words labeled with the degrees of their diachronic lexical semantic change. @xcite presented a test set for german annotated this way. in addition, english, german, latin and swedish test sets used in the semeval-2020 shared task on unsupervised lexical semantic change detection @xcite were also created following the same approach. thus, we deem durel to be the current de-facto standard for the annotation of semantic change datasets. durel employs the notion of usage relatedness borrowed from research on word sense disambiguation task @xcite . the idea is to measure the degree of semantic change as a function of mean relatedness across pairs of word's occurrences in different time periods. the annotators are not required to know anything about diachronic change: they are presented with two sentences (both containing a target word) and asked to estimate how similar are the senses in which the target word is used. they should choose a score from the 4-point scale described below in section 4. a sample of target word usage pairs from the first time period t 1 form the so called earlier group and the usage pairs from later time period t 2 form the later group. to cover the cases that cannot be tracked by comparing mean relatedness of two time periods, @xcite directly compare the 'old' and 'new' meanings by creating the additional compare group which contains pairs where the first sentences is from the t 1 time period and the second sentence is from the t 2 time period (see figure 1 ). each group contains 20 sentence pairs randomly sampled from the corresponding corpora. target word's w degree of semantic change is quantified with two measures: due to space limitations, it is impossible to provide a description of the advantages and disadvantages of these two measures here. we refer the reader to @xcite for the detailed discussion. ## data sources in this section, we describe our data sources. it can be looked at a sort of data statement @xcite for rusemshift. we first present the corpus we employ and its historical sub-corpora, and then move on to the process of preliminary target word selection. the data statement continues in the next section 4 with the description of our annotators and the annotation workflow in general. ## annotation the annotation procedure was carried out on the yandex.toloka crowd-sourcing platform foot_1 , which is more or less equivalent to the global mechanical turk or crowdflower platforms. note that in the durel dataset @xcite , all five annotators were students of linguistics and two of them had historical background, which cannot be enforced when using crowd-sourcing platforms. however, yandex.toloka allowed us to improve the quality of annotation by applying various filters to limit who can annotate. we chose 10-30% of the best annotators across the whole platform, keeping only native speakers of russian, of the age 30 and more (to ensure them being familiar with older word senses) and possessing a university degree. we do not have any knowledge about the annotators' gender distribution. the annotators who completed the tasks unrealistically fast were automatically filtered out. furthermore, there were several control tasks, consisting of sentence pairs manually annotated by ourselves (with the scores of 1 and 4 only, thus limited to obvious cases). the users who annotated such pairs incorrectly were blocked as well. also, we accepted the judgments of an annotator only after checking some of them manually to make sure that the annotator had understood the task. for each word from our pre-constructed word lists, we extracted all unique sentence contexts from two corpora (pre-soviet/soviet or soviet/post-soviet) and randomly sampled 60 sentence pairs, 20 for each group (earlier, later and compare). for the words that occurred less than 30 times in one of the corpora we decreased the number of its sentence pairs to the closest value, divisible by 3 without a remainder. final tables for the annotation contained 7846 pairs for 71 + 69 = 140 words. each sentence pair was judged by not less than 5 annotators who used the scale presented in ## evaluation rusemshift is mainly intended to be used by other researchers in the field of lexical semantic change detection for russian. however, below we report the performance of several well-known change detection methods on our datasets, to set the baseline. we solved the task of ranking: that is, given a list of target words and two time-specific corpora, a system should predict semantic change degrees which would position the target words in the order as close to the gold one as possible. first we employed static distributional embeddings trained with the cbow algorithm @xcite . after that, we tried several variations of contextualized embeddings, namely elmo @xcite . group sentence 1 sentence 2 earlier <...> выпили "микстуры"и спят гденибудь сном провальным, пьяным <...> '<...> they drank a "potion" and sleeping somewhere in deep drunken sleep <...>' [viktor astaf'ev. pechal'nyj detektiv (1982-1985)] <...> кщаре доходит до семидесяти пяти метров и что вообще здесь много провальных озер <...> '<...> kshchare reaches seventy-five meters and in general there are many deep lakes <...>' [v. a. solouhin. vladimirskie prosyolki (1956-1957)] later <...> самые "бюджетные"программы оказываются самыми провальными в исполнении <...> '<...> the cheapest programs turn out to be the most disastrous<...>' [ivan golikov. dohodnoe mesto // «vsluh o. . . », 2003.05.19] <...> подготовка к референдуму в чечне до прошлой недели носила провальный характер <...> '<...>preparation for referendum in chechnya until last week was disastrous<...>' [aleksej makarkin. krizisnoe upravlenie «chechenizaciej» // «politkom.ru», 2003.03.03] compare <...> наденька на минутку забылась провальным сном и когда открыла глаза <...> '<..> nadenka fainted for a minute and when she opened her eyes <..>' [b. s. zhitkov. viktor vavich. kniga vtoraya (1941)] провальное выступление сборной россии на олимпиаде <...> 'the failed performance of the russian team at the olympics <...>' [sergej podushkin. opasnye igry na al'pijskom vozduhe. gornolyzhniki nachinayut sezon // «izvestiya», 2002.10.25] ## conclusion we presented rusemshift, which consists of two publicly available test sets of russian words manually annotated with the degrees of diachronic semantic change they experienced (the degree is a continuous score). annotation process was based on the theoretically sound durel framework @xcite . one of the datasets is produced from re-annotated list of target words from prior work and another is completely new. the datasets allow to evaluate methods for lexical semantic change detection in russian: either graded or binary (with any desired binarization threshold). they provide data on 3 large time periods: pre-soviet @xcite @xcite @xcite . this is the first semantic change detection dataset for russian created in a large-scale crowd-sourcing annotation effort. it is also important that rusemshift is fully compatible with semantic change datasets developed for other languages, for example those presented for the corresponding semeval-2020 shared task @xcite . the dataset is available online foot_5 under a creative commons attribution-sharealike 4.0 international license. as a sanity check (and to establish the baseline performance boundaries), we evaluated several semantic change modeling systems on rusemshift. we managed to achieve significant correlation with human judgments both with static and with contextualized word embeddings, with the latter consistently outperforming the former. at the same time, simple frequency-based baseline failed to achieve any meaningful results, which signals that the dataset lacks simplistic frequency cues. rusemshift is limited to nouns and a few adjectives. one of the possible future research directions is to extend the dataset with other parts of speech. another drawback of the dataset is that it does not distinguish between different types of semantic change (e.g. narrowing, widening, metaphorization etc.). nevertheless, we hope that in its current state, rusemshift will already be of help to the researchers interested in tracing diachronic semantic shifts in russian.