DWUG : A large Resource of Diachronic Word Usage Graphs in Four Languages (2021)

Abstract: Word meaning is notoriously difficult to capture, both synchronically and diachronically. In this paper, we describe the creation of the largest resource of graded contextualized, diachronic word meaning annotation in four different languages, based on 100,000 human semantic proximity judgments. We describe in detail the multi-round incremental annotation process, the choice for a clustering algorithm to group usages into senses, and possible – diachronic and synchronic – uses for this dataset.

## introduction the view on word meaning and senses in computational linguistics has moved from a discrete @xcite @xcite to a graded @xcite @xcite perspective. however, scalable annotation strategies for this graded view yielding large-scale data for semantic evaluation have not been implemented yet. we build on two pre-existing schemata for graded contextual word meaning annotation @xcite and show how they can be applied efficiently to create large-scale data in a diachronic setup. both procedures populate a word usage graph @xcite for a target word with annotator judgments. procedure (i) requires annotators to judge usage pairs on a semantic proximity scale avoiding the a priori definition of word senses. this makes it preparation-lean and reduces experimenter influence. procedure (ii) relies on a predefined list of senses and requires annotators to judge usagesense pairs on the same proximity scale as in procedure (i). both procedures avoid binary assignments of word senses to word usages, which have been shown to be inadequate in many cases @xcite @xcite . the resulting graphs relate word usages to each other (either directly or indirectly) and thus allow for a posteriori hard-or soft-clustering, where clusters can be interpreted as senses @xcite @xcite . this makes the collapsing of senses possible, while allowing for sense overlap where this seems adequate after observing the annotated data. while both procedures require more judgments than traditional discrete word sense annotation, we show how the sampling of word usages can be optimized to reduce the number of necessary judgments. we apply the above-described annotation procedures in a multi-lingual diachronic setup to create diachronic wugs (dwugs). these contain annotations of the usages of a set of target words in corpora from two time periods @xcite . this allows us to identify changes in the wugs over time. the final resource contains 168 dwugs for four different languages (english (en), german (de), swedish (sv), latin (la)) relying on approximately 100,000 human judgments. 1 after describing the annotation procedure, we provide a detailed analysis of annotator disagreements and evaluate the robustness of the annotated graphs. dwugs can be exploited in many ways: ## related work there has been a significant shift in the view on word meaning and word senses in computational linguistics since the birth of the field. the early formulations of the word sense disambiguation (wsd) task took a discrete view on word senses, assuming a fixed inventory of senses and a single best sense per word usage @xcite @xcite . after this view was shown empirically to be inadequate @xcite @xcite , researchers have increasingly adopted a graded view on word senses, whereby a word usage may be assigned to multiple senses and more fine-grained distinctions are allowed within senses @xcite @xcite . moreover, various approaches on how senses can be qualified have been proposed, starting from manual sense descriptions @xcite , to representing a sense solely by clusters of word usages @xcite or by lexical substitutes @xcite . recently, developments on computational models of the meaning of individual word usages @xcite have inspired new research on graded word meaning @xcite . for discrete word senses, large-scale annotation projects have been carried out, e.g. semcor and ontonotes @xcite . an advantage of the graded approach is that, through bypassing sense definitions, major parts of the annotation pipeline can be automated (cf. @xcite . studies on graded word meaning, however, cover only small amounts of data @xcite @xcite @xcite . the above-mentioned studies have paved the way to study diachronic dimensions of meaning. so far, studies that have explicitly tried to capture this dimension are rare, small-scale and mostly assume discrete word senses @xcite @xcite @xcite @xcite . the most recent approaches take a graded view @xcite building on the durel framework @xcite , but result in little annotated data. we release the largest known resource of diachronic contextualized graded word ## data the data for annotation was sampled from two time-specific historical subcorpora for each language as summarized in for each language half of the target words (≈ 20) were chosen as words for which a change between c 1 and c 2 was described in etymological or historical dictionaries @xcite @xcite . the other half was determined by sampling a control counterpart with the same pos and comparable frequency development between c 1 and c 2 as the corresponding target word. (for details refer to @xcite .) ## procedure (i): usage-usage graphs we first describe the procedure devised to annotate en, de and sv data and later describe the procedure for la in sec. 5. a usage-usage graph (uug) @xmath0, e, w) is a weighted, undirected graph, where nodes @xmath1, u 2 ) ∈ e @xcite . in practice, semantic proximity can be measured by human annotator judgments on a scale of relatedness @xcite or similarity @xcite . the annotation procedure starts from a non-annotated sample of word usages and aims to populate a uug for each target word in several rounds of annotation with human judgments of semantic relatedness. 4 annotators were asked to judge the semantic relatedness of pairs of word usages using the scale in figure 1 shows three uugs resulting from our annotation. ## procedure (ii): usage-sense graphs in this section, we describe the procedure devised to annotate the latin data. this procedure is different from the other languages, as in a trial annotation task the annotators reported difficulties to judge usage-usage pairs. in consideration of this, usage-sense graphs were employed. since we do not have access to native speakers of latin, eight annotators with a high-level knowledge of latin were recruited, ranging from undergraduate students to phd students, post-doctoral researchers, and more senior researchers. ## resource a summary of the annotation outcome for each language can be found in @xmath2, @xmath3. of target words, n/v/@xmath4. of nouns/verbs/adjectives, |u | = avg. no. usages per word, @xmath5. of annotators, @xmath6. of judged usage pairs, @xmath7. no. of judgments per usage pair, @xmath8, @xmath9, @xmath10. of normalized clustering loss * 10. words for de. 14 we report two annotation agreement measures: mean pairwise spearman correlations @xcite between annotator judgments and @xcite ) for judgments' consensus, both reaching comparable scores to previous studies @xcite @xcite . the clustering loss is the value of l (definition 3) divided by the maximum possible loss on the respective graph. it gives a measure of how well the graphs could be partitioned into clusters by the l criterion. in total, roughly 100,000 judgments were made by annotators. for en/de/sv ≈50% of the usage pairs were annotated by more than one annotator, while for la each target word but one was annotated by one annotator. figure 5 shows the frequencies of annotator judgments over the durel scale by language. on the uugs (en/de/sv) judgment '4' is most frequent followed either by judgment '2' (en/de) or '1' (sv). least frequent are judgments of '0' ('cannot decide'). swedish has a considerably higher number of '0' judgments, presumably because of frequent ocr errors. on the usgs (la) judgments of '1' are clearly most frequent, followed by '4'. this is because each usage is judged against each sense description which can often be unrelated. it can be seen that annotators make frequent use of the intermediate levels of the scale ('2', '3') and thus assign graded distinctions of word meaning. ## conclusion we described the creation of the largest existing resource of word usage graphs that capture graded, contextualized word meaning for four languages, namely english, german, swedish and latin. we detailed the annotation procedure, including the sampling aimed to reduce annotation effort while keeping a high density in regions where annotators have difficulty judging relatedness. the usage graphs have been clustered and we openly release clusterings, visualizations and an analysis of the clustering results. this resource has been used for the semeval 2020 task on unsupervised lexical semantic change detection, but its possibilities are much broader and range from the use of different clustering techniques, including soft-clustering, to the use as ground truth for diachronic word sense disambiguation or temporal classification of sentences. the corpora used and some aspects of the annotation procedure were different for latin, and this was a necessary choice due to the lack of native speakers for this language and to the nature of the texts at our disposal. offering a resource for latin attests to the methodological and intellectual contribution of our work and we believe in the value of working on lexical semantic change for a historical language. future work entails annotating additional critical edges to allow for better understanding of robustness; how much annotation is needed for different kinds of words? knowing that some words, e.g., single-sense concrete words, require less annotation allows us to spend more effort on abstract and highly polysemous words. we will also analyze the influence of edge sparsity and ambiguity on the clustering procedure and compare its output to other annotation strategies. hinrich schütze. 1998. automatic word sense discrimination. computational linguistics, 24(1):97-123. a. soares da silva. 1992. homonímia e polissemia: análise sémica e teoria do campoléxico. in actas do xix congreso internacional de lingüística e filoloxía románicas, volume 2 of lexicoloxía e metalexicografía, pages 257-287, la coruña. fundación pedro barrié de la maza. språkbanken. downloaded in 2019. the kubhist corpus, v2. department of swedish, university of gothenburg. svenska akademien. 2009. contemporary dictionary of the swedish academy. the changed words are extracted from a database managed by the research group that develops the contemporary dictionary. nina tahmasebi and thomas risse. 2017. finding individual word sense changes and their delay in appearance. in proceedings of the international conference recent advances in natural language processing, pages 741-749, varna, bulgaria. warren weaver. 1949/1955. translation. in william n. locke and a. donald boothe, editors, machine translation of languages, pages 15-23. mit press, cambridge, ma. reprinted from a memorandum written by weaver in 1949. yorick wilks and edward l. keenan. 1975. preference semantics, page 329-348. cambridge university press.