Three-part diachronic semantic change dataset for R ussian (2021)

Abstract: We present a manually annotated lexical semantic change dataset for Russian: RuShiftEval. Its novelty is ensured by a single set of target words annotated for their diachronic semantic shifts across three time periods, while the previous work either used only two time periods, or different sets of target words. The paper describes the composition and annotation procedure for the dataset. In addition, it is shown how the ternary nature of RuShiftEval allows to trace specific diachronic trajectories: ‘changed at a particular time period and stable afterwards’ or ‘was changing throughout all time periods’. Based on the analysis of the submissions to the recent shared task on semantic change detection for Russian, we argue that correctly identifying such trajectories can be an interesting sub-task itself.

## introduction this paper describes rushifteval: a new dataset of diachronic semantic changes for russian words. its novelty in comparison with prior work is its multiperiod nature. until now, semantic change detection datasets focused on shifts occurring between two time periods. on the other hand, rushifteval provides human-annotated degrees of semantic change for a set of russian nouns over three time periods: pre-soviet , @xcite and @xcite . notably, it also contains 'skipping' comparisons of pre-soviet meanings versus post-soviet meanings. together, this forms three subsets: rushifteval-1 (pre-soviet vs soviet), rushifteval-2 (soviet vs post-soviet) and rushifteval-3 (pre-soviet vs post-soviet). the three periods naturally stem from the russian history: they were radically different in terms of life realities and writing and practices, which is reflected in the language. as an example, the word дядька lost its 'tutor of a kid in a rich family' sense in the soviet times, with only the generic 'adult man' sense remaining. certainly, language development never stops and russian also gradually evolved within those periods as well, not only on their boundaries. however, in order to create a usable semantic change dataset, one has to draw the boundaries somewhere, and it is difficult to come up with more fitting 'changing points' for russian. rushifteval can be used for testing the ability of semantic change detection systems to trace longterm multi-point dynamics of diachronic semantic shifts, rather than singular change values measured by comparing two time periods. as such, rushifteval was successfully employed in a recent shared task on semantic change detection for russian @xcite . ## related work automatic detection of word meaning change is a fast growing research area @xcite . evaluation of this task is especially challenging; inter alia, it requires gold standard annotation covering multiple word usages. the common practice is to annotate pairs of sentences as using a target word in either the same or different senses. it was introduced for the word sense disambiguation task in @xcite , while @xcite proposed methods to aggregate pairwise annotations for semantic change modeling; one of them, the compare metrics, is used in rushifteval. a similar approach was used for the se-meval'20 shared task on semantic change detection @xcite : annotators labeled pairs of sentences, where some pairs belonged to the same periods and some to different ones. this annotation resulted in a diachronic word usage graph, which was then clustered to obtain sepa-rate word senses and their distributions between time periods @xcite . the pairwise sentence annotation has been used in creating another semantic change dataset for russian, rusemshift @xcite . we use the same annotation procedure and rely on the same corpus, i.e. russian national corpus (rnc) split into pre-soviet, soviet and post-soviet subcorpora. however, rusemshift features two sets of words: one for the changes between the pre-soviet and soviet periods, and another for the soviet and post-soviet periods. the new rushifteval dataset, which we present in this paper, uses a joint word set allowing for tracing each word across three time periods. in addition, we directly annotate semantic change between the pre-soviet and post-soviet periods, skipping the soviet one. ## diachronic trajectory types rushifteval allows tracing multi-hop dynamics of semantic change. a similar analysis of diachronic word embedding series or 'trajectories' was conducted in @xcite and @xcite , but the former focused on change point detection, and the latter on finding general laws of semantic change. with manually annotated rushifteval dataset we were able to move further and identify at least three different types of changing trajectories: 1) changes in every period pair; 2) change in the soviet period as compared to the pre-soviet period; 3) change in the post-soviet period as compared to the soviet period. since approximately a half of the words in the dataset did not change their meaning they exhibit a fourth, trivial type of trajectory, where all three distances are small. in principle there could be a fifth type of trajectory, where difference between pre-soviet and post-soviet periods is substantially smaller than between other period pairs, which would mean that a word was used in a new sense during the soviet time but then came back to its original meaning. however, we did not find any words following this trajectory type and not sure whether this behavior is theoretically plausible. another potential flaw is sampling variability. for annotation, we sampled 30 sentences with a target word from each time period for each comparison. since our relatedness graph has three edges, each word is represented with two samples. as it turned out, in some cases different samples can yield quite different picture of sense distributions. let us manually analyze the word полость ('cavity/hide to cover one's legs in an open cart'). since horse-driven carts disappeared just a few years after the beginning of the soviet period, one might expect the second sense to be lost in soviet times and never to appear again. however, the relatedness between the soviet and post-soviet time periods (1.9) is even lower than between the pre-soviet and soviet periods (2.2), as if the word experienced another semantic shift. in fact, it is a random sampling artifact. in the 30 sentences from the soviet period sampled for the 'pre-soviet:soviet' pair, only 4 used полость in this archaic sense. but in the 30 sentences from the same period sampled for the 'soviet:post-soviet' pair, this number grew to 10, 2.5 times more (mostly in fiction texts, where the plot is set in the pre-soviet times). as a result, the soviet usage pattern looks like it is different from the post-soviet one, although in fact no shift has happened (as evident both from linguistic intuition of russian speakers and from the fisher exact test which in this case yields @xmath0.13). the frequency of полость in the soviet sub-corpus is about 600, so both samples together cover only 10% of the full concordance. without manually annotating all six hundred occurrences, it is difficult to tell which sample is more representative of the real word usage in the soviet times. it would be better to increase the sample size as much as possible: 30 is arguably already on the border. ## conclusion we presented rushifteval, a novel dataset of diachronic semantic changes in russian nouns across three time periods, using the same set of target words for all comparisons. we also conducted a preliminary analysis of how rushifteval can be used in tracing diachronic semantic trajectories, and how current change detection systems for russian deal with this potentially interesting task. a transliterations of russian words mentioned in the article word transliteration translation бригада brigada brigade/gang/team дядька djadka uncle/man/(male) tutor живот život stomach/belly/life закладка zakladka foundation/bookmark/hidden artifact линейка lineika carriage/ruler/series of goods лох loh salmon/silver-berry/easy victim облако oblako cloud полоса polosa tripe/ribbon/lane/runway полость polost cavity/foot hide радикал radikal radical роспись rospis mural/signature/list связка svjazka ligament/vocal cords/mutual connection спутник sputnik fellow traveler/satellite/sputnik ссылка ssylka exile/link стол stol table/diet тачка tachka wheelbarrow/car формат format format центр tsentr center ядро jadro cannonball/core/nucleus