Lexicon of Changes: Towards the Evaluation of Diachronic Semantic Shift in C hinese (2022)

Abstract: Recent research has brought a wind of using computational approaches to the classic topic of semantic change, aiming to tackle one of the most challenging issues in the evolution of human language. While several methods for detecting semantic change have been proposed, such studies are limited to a few languages, where evaluation datasets are available. This paper presents the first dataset for evaluating Chinese semantic change in contexts preceding and following the Reform and Opening-up, covering a 50-year period in Modern Chinese. Following the DURel framework, we collected 6,000 human judgments for the dataset. We also reported the performance of alignment-based word embedding models on this evaluation dataset, achieving high and significant correlation scores.

## introduction lexical semantic change not only satisfies the appetite for linguistic exploration but also reflects the societal and cultural developments @xcite . recently, this topic has been receiving growing interest from the nlp community, as witnessed a wealth of papers working on this research questions with computational approaches emerged over the past two decades @xcite @xcite . among these studies, most make use of distributional word representations with temporal information to model diachronic meaning change @xcite @xcite . although a variety of computational methods have been proposed for the task of lexical semantic change, evaluation datasets are only available for a limited number of languages, e.g. english, latin, italian, swedish, german, russian @xcite @xcite . few studies have investigated chinese in this domain @xcite and there is currently no evaluation dataset for detecting chinese lexical semantic change. this paper presents the first chinese evaluation dataset, zhshifteval, for the detection task. 1 this dataset allows us to evaluate those shifts that occurred to modern chinese from 1953 to 2003, over two roughly equal intervals: sub-corpus c1 and the sub-corpus c2 . these two intervals were chosen on the basis of the reform and opening-up, the most influential milestone in the recent history of china foot_1 . it is generally assumed that this remarkable social change brought significant changes to the lexicon of modern chinese @xcite . the remainder of this paper is organized as follows. section 2 situates our study within previous work. in section 3, we introduce how the evaluation dataset has been created following the durel framework. section 4 qualitatively discusses the dataset itself, and section 5 presents the preliminary results of static word embeddings on this evaluation dataset. ## related work before semeval 2020, the field lacked shared standard datasets for evaluating lexical semantic change with computational approaches. most early works were exploratory, testing whether computational models could capture specific established cases of semantic change, but without a quantitative evaluation of the models' performance @xcite @xcite . some evaluation datasets consisted of a list of target words labeled as 'changed' and 'unchanged' with reference to linguistic papers, dictionaries @xcite , and wordnet @xcite . however, these datasets are based on a binary judgment on semantic change, ignoring its cumulative nature. in contrast, @xcite demonstrated a 'gradable' view towards semantic change, asking native speakers to annotate target words with multiple labels for their changing degrees, according to their intuitions. @xcite later proposed the diachronic usage relatedness (durel) framework to construct evaluation datasets for the detection task. they asked annotators to compare and grade the semantic relatedness of target words, from unrelated (1) to identical (4), across the context pairs. the ratings, together with target words, formed a small-scale evaluation dataset for german. following this framework, @xcite and @xcite created a twoperiod evaluation dataset, 'rusemshift' and a threeperiod evaluation dataset 'rushifteval' for russian, assessing those meaning shifts that occurred to russian words from the pre-soviet period to the post-soviet period. in semeval 2020, evaluation datasets for english, german, swedish, and latin were released as benchmarks for the shared task @xcite . the datasets were built under the diachronic word usage graph (dwug), an extension of the durel framework, exploiting usage graphs to represent the gain and loss of senses for target words. the usage graph is weighted and undirected. the nodes represent word usages, and the weights are semantic relatedness scores graded by human annotators @xcite . ## dataset analysis as described in previous sections, the δlater metric subtracts the mean relatedness of the ear-lier group from the later group. therefore, a positive δlater value is assigned when usages of the annotated word in the c2 group are more similar, whereas negative δlater is assigned to words with less similar usages in the c2 group. positive and negative δlater values can be considered as two different sub-types of semantic change: innovative meaning change and reductive meaning change, roughly representing the gain or loss of word senses @xcite . the absolute δlater value assesses the strength of semantic change. as shown in figure 1 , most annotated words are predicted as stable words, with δlater values around 0. the two topmost words '推出'(push out; launch), '机制'(machine-made; mechanism) and the two bottommost words: '拐'(crutch, traffic), '炒'(to fry, to speculate(in the stock market)) are predicted as the words with stronger effects of semantic change. the successful predictions on '推出', '机制', '炒' coincide with documented linguistic publications @xcite , verifying δlater as an effective measure of lexical semantic change. interestingly, the metric predicted '拐' as a changed word, despite it being originally a stable control word. a closer inspection of all three groups of sampled sentences suggested that '拐' is used more frequently with the 'crutch' meaning in the subcorpus c1, but it shows a high prevalence of the 'trafficking' meaning in the sub-corpus c2. however, the usage fluctuation detected here has to take into account the corpus bias, as 'trafficking' is more likely to occur in a newspaper corpus. technically speaking, words such as '机制' and '拐' are homographs with different meanings, i.e. different words with less related or even unrelated meanings. the detected shift actually shows the competition among different meanings with the same surface form, rather than the gain or loss of senses. for example, '机制' in the sampled texts from c1 period dominantly refers to a way of manufacturing as 'machine-made' (against 'handmade'). with the process of industrialization, 'machinemade' objects became so prevalent in everyday life that the need to mention this feature quickly became obsolete and the usage slipped into obscurity. meanwhile, the program of reform and openingup was carried out thoroughly, especially concerning the revolution of the socialist market economy system and mechanism. for this reason, '机制' with the 'mechanism' meaning became dominant in the c2 period. the compare metric directly compares the semantic relatedness of a usage pair within the compare group, which consists of sentences from two different periods. higher compare scores would be assigned to more stable words, like '照片(photo),' '雪(snow)', getting full scores of 4. lower compare scores are assigned to the shifting ones, e.g. the four changed words predicted by the δlater metric (see figure 2 ). moreover, this metric captured a shifting word '软(soft)'. a closer checking on sampled sentences suggested that '软' is polysemous in the c1 group, but with a dominant usage meaning 'soft texture of concrete stuff'. in the c2 group, its metaphorical senses even became more diverse, like 'soft science, soft power', meanwhile, the 'soft texture' sense lost its prevalence based on our observation. the multiple changes made the δlater score not salient per se, but they were captured by the compare metric, where usages from two different historical periods are directly compared. ## evaluation the semeval shared task has indicated that traditional static embeddings may outperform more recent paradigms -e.g., contextualized embeddings @xcite -in the task of semantic change detection @xcite . therefore, we trained a static word embedding model for this task and evaluated its performance on our newly-created dataset in this study. we first trained our vectors on each subcorpus using both the skip-gram model and the continuous bag of words, which are the two most widely used static word embeddings models @xcite . to have an assessment of the quality of the word embeddings trained on our subcorpora, we performed a preliminary evaluation on the chinese word similarity dataset cos960, introduced by @xcite . the results indicated that the quality of the word embedding models was satisfactory (see we then aligned word representations for the two periods into a shared space with the orthogonal procrustes algorithm @xcite , projecting word embeddings for the c2 period onto c1's space and making vectors living in different intervals comparable. the cosine similarity between two vectors for the same word form is calculated as the degree of meaning change. according to the cosine similarity, we ranked those words appearing in both the c1 and c2 periods, where the higher the similarity, the more stable the meaning. compared with the scores derived with the compare metric, the skip-gram model achieved a spearman correlation score of 0.584. as for the δlater, we took the absolute value indicating the degree of semantic drift for the correlation calculation (the positive and the negative δlater values represent different sub-types of semantic change, but leave this to future investigations). this time, the skip-gram model achieved a spearman correlation coefficient of -0.625. both two correlation scores are statistically significant at @xmath2.05. as expected, the performance of the skip-gram model on the detection task is positively correlated with the compare metric and negatively correlated with the δlater metric. ## conclusion this paper presented the first human-annotated evaluation dataset for the task of chinese lexical semantic change detection. this dataset was built following the durel framework, which allows us to evaluate the usage drift that occurred in coincidence with the reform and opening-up in recent chinese history. our data further suggested that interpretation of the δlater metric could be extended to the competition among different usages of the same surface form, in order to accommodate historical changes involving homographs. we finally examined the performance of the skip-gram model on our evaluation dataset and found that it achieves a relatively high correlation coefficient with the two metrics. this paper served as a first, exploratory study on modeling lexical semantic change in chinese, on the basis of a limited number of words. in the near future, our goal is to scale up the dataset and to examine the performance of more models for chinese, including the more recent contextualized embeddings @xcite . moreover, using finer-grained intervals for diachronic meaning change detection and exploring the diatopic variation between different chinese dialects are also possible directions of our future work @xcite .