C hi WUG : A Graph-based Evaluation Dataset for C hinese Lexical Semantic Change Detection (2023)

Abstract: Recent studies suggested that language models are efficient tools for measuring lexical semantic change. In our paper, we present the compilation of the first graph-based evaluation dataset for lexical semantic change in the context of the Chinese language, specifically covering the periods of pre- and post- Reform and Opening Up. Exploiting the existing framework DURel, we collect over 61,000 human semantic relatedness judgments for 40 targets. The inferred word usage graphs and semantic change scores provide a basis for visualization and evaluation of semantic change.

## introduction lexical semantic change detection, i.e. measuring meaning changes across different timespans, gained substantial popularity with the growing availability of historical corpora and language models @xcite @xcite @xcite , mostly for english and for other indo-european languages. the increasing number of published evaluation datasets further fostered the domain, enabling different models and hyperparameters to be quantitatively tested on the same benchmarks @xcite @xcite @xcite . these datasets are predominantly constructed within the framework of diachronic usage relatedness (durel), wherein changing scores are generated by calculating human ratings on semantic relatedness across a variety of usage pairs for targets @xcite @xcite . in the extended durel framework, namely diachronic word usage graphs (dwugs) @xcite , the usages could be further populated through word usage graphs (wugs) for visualization @xcite . to foster the development of lexical semantic change detection in chinese, we constructed the first graph-based evaluation dataset, namely chi-wug, following the durel framework for the human judgments collection. based on the collected 61k human judgments for 40 targets, we populated 40 wugs to visualize usage changes preceding and following the context of the reform and opening up, one of the most important milestones in the recent history of china. foot_0 ## related work instead of categorizing words into changed and unchanged @xcite @xcite , the durel framework adopted a graded view towards semantic change that words may exhibit varying degrees of semantic change. this is achieved by comparing the semantic relatedness targets in usage pairs on a scale of 1 to 4 @xcite , referring to semantic proximity from homonymy to identical usages. specifically, usage pairs are assembled with contexts from periods of interest. in the original durel framework, three groups of usage pairs are assembled for a two-period setting, pairs consisting of two sentences from the same period and pairs having usages from each period @xcite . the extended dwugs allow us to categorize these usages into different groups by clustering, and groups proximately refer to different senses of a word. through comparing the derived clusters from periods of interest, dwugs allows us to easily measure the changes of sense distributions, i.e. loss and gain of senses and the turnover of usage dominance, which goes beyond the pure 'degree of changes' offered by the original durel framework @xcite . the durel framework and its extension dwugs have been applied to constructing evaluation datasets for a variety of languages, such as english, swedish, german, and latin released in the semeval 2020 @xcite , and later for russian, norwegian, spanish, and chinese @xcite @xcite @xcite . since the nature of this paradigm is to measure usage differences between sentence pairs, it has also been extended to the construction of synchronic disambiguation datasets @xcite and to diatopic variation (i.e., usage differences across regional variations) @xcite . ## data building on the previous work by @xcite , which collected human judgments for 20 targets following the durel framework, we expand the data size and obtain the dwugs to have a more comprehensive evaluation dataset for chinese. 2 ## human annotation to collect human judgments, we recruited four native speakers of mandarin as annotators. all the annotators are graduate students from the faculty of humanities, specializing in chinese linguistics. they were invited to experiment on the durel platform after passing a tutorial specific to the chinese lexical semantic change task foot_3 . before the tutorial, we arranged a meeting for instructions. after passing the tutorial, annotators were asked to indicate their intuitions on how semantically related a target was used in two displayed contexts in the 'official' annotation work. targets would be highlighted, and options for judgments on a scale from identical (完全一致) to unrelated (不相关) are listed in the left bar, as shown in figure 1 . 7 . besides assigning scores from 1 (unrelated meanings) to 4 (identical meanings) for semantic relatedness, they are also allowed to 'discard' usage pairs by giving the '0' score if the current pair is hard to understand due to the ambiguity of contexts or word segmentation errors. they are also encouraged to 'pause' the annotation process during the annotation after a period of annotation (around 30 minutes) to avoid excessively long sessions and keep their judgments as consistent as possible. due to the heavy load of annotation, the data was split in half, and each pair of annotators took one half consisting of 10 changed words and 10 fillers for annotation. we finally collected over 61,000 judgments from four annotators, after removing those judgments with a score of zero, that is, discarded pairs. the weighted mean pairwise spearman score for inter-rater agreements is 0.691, and the krippendorff's alpha is 0.602, which are quite high if compared to other durel datasets @xcite @xcite @xcite . for more statistics, see ## graph representations based on human judgments collected from the procedures described previously, we follow @xcite to aggregate the scores per usage pair as their median for populating wugs, where usages with the same senses would be grouped together by performing the correlation clustering @xcite . to populate wugs with dense clusters, we took usage pairs with scores 3 and 4 as the same sense, while scores 1 and 2 were considered as different senses. 8 figure 2 and figure 3 are inferred word usage graphs for 病毒 bingdu ('virus') and 下海 xiahai ('go into the sea' or 'to venture'), respectively. nodes in the same color are clustered as the same sense, and subgraphs from the left to the right show the clusters/senses changes. in figure 2 , the right subgraph reveals the emergence of a new usage denoted by nodes in orange. by delving into the contexts associated with the orange-labeled usages 9 , we discern that 病毒 (bingdu) acquired a fresh sense, namely 'computer virus', during the second period, diverging from its earlier associations with 'viral infections'. similarly, figure 3 highlights the emergence of a new sense characterized by orange nodes, which are later confirmed as 'to venture', besides its original sense 'go into the sea'. furthermore, this emergent sense exhibits increased usage dominance in our samples, as evidenced by the greater presence of orange nodes in the later period. ## conclusion this study presents the first graph-based evaluation dataset for chinese lexical semantic change constructed following the dwug paradigm. it populates 40 word usage graphs based on more than 61k human judgments on contextual semantic relatedness between sentence pairs. with its comparably high inter-rater agreement and dense clusters post-processed by clustering, we assume this high-quality evaluation dataset could be included in the shared evaluation datasets to foster lexical semantic change detection in chinese. meanwhile, the inferred wugs themselves are interesting for linguistic studies.