More DWUG s: Extending and Evaluating Word Usage Graph Datasets in Multiple Languages (2024)

Abstract: Word Usage Graphs (WUGs) represent human semantic proximity judgments for pairs of word uses in a weighted graph, which can be clustered to infer word sense clusters from simple pairwise word use judgments, avoiding the need for word sense definitions. SemEval-2020 Task 1 provided the first and to date largest manually annotated, diachronic WUG dataset. In this paper, we check the robustness and correctness of the annotations by continuing the SemEval annotation algorithm for two more rounds and comparing against an established annotation paradigm. Further, we test the reproducibility by resampling a new, smaller set of word uses from the SemEval source corpora and annotating them. Our work contributes to a better understanding of the problems and opportunities of the WUG annotation paradigm and points to future improvements.

## introduction in recent years, a new annotation paradigm for word senses has emerged under the name of word usage graphs (wugs, @xcite . in this paradigm, humans provide semantic proximity judgments of pairs of word uses (also known as word-in-context, wic), which are then represented in a weighted graph and clustered with a graph clustering algorithm, as displayed in figure 1 showing a selection of clustered graphs from multiple wug datasets. in this way, word sense clusters can be inferred from simple pairwise word use judgments, avoiding the need for a sense inventory, which can be tedious to create. while, up to now, this approach has been applied mainly within the field of lexical semantic change detection (lscd) (e.g. @xcite @xcite , it can be applied generally in a word sense induction (wsi) setting @xcite or for word sense disambiguation (wsd) when combined with a sense labeling procedure for word sense clusters (cf. @xcite . as a recent approach to the study of word senses, the wug annotation paradigm brings many open questions and uncertainties relating to reduction of annotation load (see section 4.1), clustering of the annotated graphs as well as the stability and reproducibility of the resulting clusters. in this paper, we try to answer some of these questions relying on the first large wug dataset created in the semeval-2020 task 1 on unsupervised lexical semantic change detection @xcite . we add additional rounds of annotation to the english, german and swedish datasets, to more densely populate the graphs and make the inferred sense clusters and semantic change scores more reliable. this allows a comparison against the earlier inferred clusters and scores. we also compare the inferred clusters over annotation rounds against an external gold standard of sense definition annotations in german @xcite . in this way, we test the validity of previous clusterings. we also evaluate cluster robustness by measuring the clustering variance with different degrees of noise introduced to the annotation. further, to test the replicability of the semeval data, we completely resample and annotate a small new set of word uses from the semeval source corpora. we hope this paper contributes to a better understanding of the problems and opportunities of the wug annotation paradigm and points to future improvements in the process. we summarize our contributions going beyond previous work: (i) our work adds thousands of additional judgments to the datasets created by @xcite and @xcite and thus makes them much more densely annotated. 1 these datasets can be used to tune and evaluate models for a multitude of tasks, such as wic, wsi and lscd @xcite . (ii) we resample and annotate data from the same sources as @xcite providing an independent replication of their results (word sense clusters, change labels), which can serve as a comparison and a way to evaluate reliability and accuracy of the original dataset. (iii) we are the first to provide a solid way to validate the cluster derivation approach defined by @xcite we compare the obtained clusters to clusters obtained by an independent (more traditional) annotation approach. additionally, we propose to evaluate the clusters from earlier rounds to the clusters obtained on the full data. both approaches show that the original data was not optimal, quality improves over rounds and that the final datasets created by us have near to optimal quality. the paper is structured as follows: next, we introduce related work on word sense annotation approaches. then, we introduce the datasets we will extend or replicate followed by a description of our annotation procedure. we then describe a number of experiments on the resulting datasets testing validity, robustness and replicability of the derived sense clusters. this is followed by concluding remarks and an outlook to future work. ## related work existing word sense annotation procedures can be distinguished into three main categories: (i) usesense, (ii) lexical substitution and (iii) use-use annotation (cf. @xcite . (i), use-sense annotation, has a long tradition within the task of wsd @xcite . annotators usually choose the best-fitting word sense definition for a word use as in this example: use: [. . . ] taking a knife from her pocket, she opened a vein in her little arm. possible substitutes for arms in this use would be weapons or guns. although it is not a widespread approach, it is possible to represent such lexical substitutes as vectors, to derive similarities from these and to represent them in a weighted graph which can be clustered @xcite . in use-use annotation, typically pairs of uses are judged according to their semantic proximity (i.e., relatedness or similarity): use1: [. . . ] taking a knife from her pocket, she opened a vein in her little arm. related approaches obtain pairwise judgments using a comparative annotation framework @xcite , or infer them through spatial arrangement @xcite , sentence sorting @xcite , use-sense judgments (pilehvar and camacho-collados, 2019) or difficulty estimation @xcite . computational modeling of use-use proximity judgments on binary or graded scales has seen a recent upsurge under the label "word-in-context task" @xcite @xcite . the wug approach builds on use-use proximity judgments by exploiting their transitive interconnectedness through graph representation @xcite . there are a number of recent wug datasets for multiple languages @xcite @xcite @xcite @xcite , most of them with a diachronic component by sampling uses from different time periods. a few studies investigate the clustering and/or edge sampling procedures @xcite @xcite . see also @xcite for an in-depth analysis of cluster errors and the robustness of clusterings and change scores derived from them. most related to our work are the wug datasets created for semeval-2020 task 1 @xcite ) as these are used and extended for this study. ## word usage graphs a wug @xmath0, e, w) is a weighted, undirected graph, where nodes @xmath1, (u 1 , u 2 ) ∈ e @xcite . the set of uses u can be sampled from different time periods t 1 , t 2 ... t n , where we refer to the time-specific use subsets as u 1 , u 2 ... u n . in practice, semantic proximity can be measured by human annotator judgments on a scale of relatedness @xcite or similarity @xcite . wugs obtained from the annotation are often sparsely observed and noisy. this poses a very specific problem that calls for a robust clustering algorithm. hence, @xcite implement a variation of correlation clustering @xcite which minimizes the sum of cluster disagreements, i.e., the sum of low edge weights (semantic proximity) within a cluster plus the sum of high edge weights across clusters. for this, one has to choose a threshold h on edge weights deciding which weights will be considered as high and which ones as low. schlechtweg et al. set @xmath2.5. consequently, the weight w (e) of each edge @xmath3, e, w) is shifted to w ′ (e) = w (e)-2.5 (e.g. a weight of 4 becomes 1.5). those edges e with a weight w ′ (e) ≥ 0 are referred to as positive edges p e while edges with weights w ′ (e) < 0 are called negative edges n e . let further c : @xmath4, ϕ e,c be the set of positive (high) edges across any of the clusters in clustering c and ψ e,c the set of negative (low) edges within any of the clusters. the algorithm then searches for a clustering c that minimizes the sum of weighted cluster disagreements: that is, the sum of positive edge weights between clusters and (absolute) negative edge weights within clusters is minimized. minimizing swd is a discrete optimization problem which is np-hard @xcite . as most wugs have a relatively low number of nodes (≤ 200), schlechtweg et al. chose to approximate the global optimum with simulated annealing @xcite , a standard discrete optimization algorithm. in order to reduce the search space, the algorithm iterates over different values for the maximum number of clusters (≤ 20). it also iterates over randomly as well as heuristically chosen initial clustering states. 2 the finally obtained clustering c : @xmath5. from this, schlechtweg et al. calculate a cluster (sense) frequency distribution d encoding the size of each cluster as ## data we select the dwug dataset @xcite because it was richly annotated in multiple rounds of annotation and has been widely used (e.g. @xcite @xcite . the data contains use pairs of english, german and swedish target words annotated on the scale in additionally, we use the discowug dataset @xcite as it extends the dwug de dataset by a number of target words with uses sampled from the same time-specific corpora as dwug. for evaluation of the annotation quality, we use dwug de sense @xcite which annotates a subset of uses sampled from the german dwug dataset with traditional use-sense annotations. 3 find an overview of the datasets in ## annotation we recruited at least three annotators per language. most were current students or had university-level education. all of them were native speakers of the respective language. 5 they were instructed in a short training using a version of the durel guidelines @xcite . below, we first describe for each dataset how uses and edges were sampled for annotation. then, we describe how the annotated data was postprocessed and analyzed. ## experiments we now evaluate the validity of the inferred clusters over rounds of annotation. this is followed by tests of the robustness of the final clusterings, and their replicability through a complete resampling and reannotation of data. ## conclusion in this paper, we tested the validity, robustness and replicability of the largest existing wug datasets used in semeval-2020 task 1. we added thousands of additional judgments to the datasets making them much more densely annotated and reliable. these datasets can be used to tune and evaluate models for a multitude of tasks, such as wic, wsi and lscd. then, we reclustered the data based on increasing amounts of annotation and found that clustering quality increases with annotation rounds. this also shows that the original semeval datasets were not optimal for evaluation, and possibly results should be reconsidered. our robustness analysis supports this finding and suggests that small sample sizes of uses, leading to more densely annotated graphs as the proportion of annotated edges is higher, have a considerable effect on robustness of clusterings. we fur-ther resampled and reannotated the data providing an independent replication showing that the se-meval word sense distributions can often be approximated well with smaller samples and simpler (random) edge sampling. the main conclusion from our work is that in future annotation studies large samples of uses should be sacrificed in favor of large samples of edges, in order to create more densely annotated graphs. this aligns with and strengthens the findings of @xcite and @xcite . a natural hypothesis for future work is whether the improved data quality will lead to higher performance of wsi and lscd models and whether previous results on performance relations can be reproduced with the more reliable data. further interesting questions concern the improvement of the annotation procedure: can we improve the clustering quality? can we find efficient and robust node and edge sampling strategies? what are alternative ways of evaluating the quality of the annotation, the clustering or the change scores?