The DUR el Annotation Tool: Human and Computational Measurement of Semantic Proximity, Sense Clusters and Semantic Change (2024)

Abstract: We present the DURel tool implementing the annotation of semantic proximity between word uses into an online, open source interface. The tool supports standardized human annotation as well as computational annotation, building on recent advances with Word-in-Context models. Annotator judgments are clustered with automatic graph clustering techniques and visualized for analysis. This allows to measure word senses with simple and intuitive micro-task judgments between use pairs, requiring minimal preparation efforts. The tool offers additional functionalities to compare the agreement between annotators to guarantee the inter-subjectivity of the obtained judgments and to calculate summary statistics over the annotated data giving insights into sense frequency distributions, semantic variation or changes of senses over time.

## introduction the concept of semantic proximity between word uses has a rather long tradition in cognitive semantics @xcite and is also acknowledged by notable lexicographers @xcite . semantic proximity quantifies how much the meanings of two word uses "have in common" @xcite . the concept serves as (often vague) criterion in the lexicographic clustering process @xcite and is thus essential to the process of creating dictionary entries of word senses. being essential to the identification of word senses, semantic proximity has further relevance to research building on senses such as lexical semantic change or semantic variation @xcite . in lexical semantics, multiple approaches operationalized semantic proximity in human annotation studies, showing that the concept can be practically implemented with reasonable agreement between annotators and correspondence to alternative annotation procedures @xcite @xcite . recently, there has been an upsurge on research in computational modeling of semantic proximity between word uses under the name of word-in-context models @xcite , resulting from advances in modeling the meaning of word uses with contextualized embeddings @xcite . these models reach high performance @xcite and thus serve as an excellent starting point for any practical task building on semantic proximity like creating dictionary entries, finding novel/non-recorded senses or identifying words that change their meaning. we present the durel tool combining the abovedescribed lines of research into a user-friendly, online annotation interface with open source code. 1 the basic annotation data gathered in the system are judgments of semantic proximity between word uses @xcite from multiple human or computational annotators. the tool facilitates the annotation task by providing a data inspection interface and automatic data validation for researchers, an intuitive task interface for annotators, guidelines in multiple languages for annotator training as well as tutorial data for annotator testing. computational annotators are provided by optimized word-in-context models @xcite trained on human semantic proximity judgments (i.a. @xcite @xcite . semantic proximity judgments are represented in a graph @xcite , clustered with an automatic graph clustering technique @xcite and visualized for analysis @xcite . this allows to measure word senses from simple and intuitive semantic proximity judgments between use pairs. the tool offers functionalities to compare the agreement between annotators to guarantee the inter-subjectivity of the obtained judgments. it provides further functionalities to calculate summary statistics over the annotated data giving insights into sense frequency distributions, semantic variation or changes of senses over time. the computational annotator component allows to generate word sense clusters for large sets of words and word uses, making it possible, for instance, to analyze large amounts of data or to search unlabelled data systematically for new senses. ## related work we now compare durel to existing text annotation tools and related tools from electronic lexicography. there is a number of general-purpose text annotation tools such as catma @xcite , inception @xcite , mturk foot_1 , phitag foot_2 , potato @xcite or toloka foot_3 . many of these allow to define a wide range of custom tasks and can in principle cover use pair annotation (cf. e.g. @xcite . however, this often requires preparation efforts including the writing of small programs as well as the formulation of guidelines and data for annotator training. the aim of the durel tool is to minimize such additional efforts around organizing a use pair annotation study. durel achieves this by focusing on this particular task, implementing standardized procedures which have proven to work well in previous studies (i.a. @xcite @xcite . further unique features of the tool are the task-specific data analysis (see section 4.2.5) and computational annotators (see section 4.2.2). other tools, while offering annotation for a range of tasks, cannot offer these specific possibilities. with its focus on a semantic annotation task (semantic proximity) and the sense inference functionality (see section 4.2.4), the durel tool is relevant to lexicography. the most widely used lexicographic tool is sketch engine. 5 the durel tool differs from sketch engine by focusing primarily on crowdsourcing human annotations and offering computational annotation models, whereas @xcite on the left and its interpretation from @xcite ) on the right. sketch engine offers frequency-based corpus analysis and manual dictionary making. the two tools could be integrated to provide improved analysis of word meaning. ## background the durel tool implements the word sense annotation scheme developed in @xcite @xcite and described in detail in @xcite . the scheme builds on use pair proximity annotation on a relatedness scale combined with a graph clustering procedure. annotators are asked to judge the semantic relatedness of use pairs, such as the two uses of arm in (1) and (2), on the scale in the annotated data of a word is then represented in a graph, which we call word usage graph (wug), where vertices represent word uses, and weights on edges represent the (median) semantic relatedness judgment of a use pair. the final wugs are clustered with correlation clustering @xcite . consider the example in figure 1 to understand how senses and semantic change can be inferred with the durel annotation procedure: assume that wug g represents the semantic proximity structure annotated for the set of word uses u of the english word arm displayed in g2 c 2 = {d, e} (orange), c 3 = {b} (green). we then build the time-specific subgraphs g 1 and g 2 and are now able to compare the clusters between time periods. for instance, c 3 only exists in the first time period while c 2 only exists in the second time period. ## tool description durel is a web application supporting user interaction by browser (mozilla firefox is recommended). 6 it was created on an architecture incorporating java spring for backend, post-gresql for the database, html/css/javascript plus thymeleaf for frontend, and a csv format for transferring annotations. the source code of the tool is publicly available under a creative commons attribution-noncommercial-sharealike 4.0 international license. foot_6 ## case studies and evaluation in this section, we describe two case studies to show the usefulness of the tool and to evaluate the computational annotators: (i) the arm example is a small scale study on a chosen test word for which we selected word uses from a corpus. (ii) the lexicographer case study conducted on a set of 18 swedish words is related to an ongoing lexicographic project. ## conclusion we presented the durel, an online, open-source annotation tool for annotating semantic proximity between word uses. the tool supports human as well as computational annotation, building on recent advances with wic models. annotator judgments are automatically clustered and visualized for analysis. this allows to capture word senses with simple and intuitive micro-task judgments between use pairs, requiring minimal preparation efforts. additionally, durel can compare the agreement between annotators to guarantee the intersubjectivity of the obtained judgments and to calculate summary statistics over the annotated data giving insights into sense frequency distributions, semantic variation or changes of senses over time. the computational annotator component allows to generate word sense clusters for large sets of words and word uses, making it possible, for instance, to analyze large amounts of data or to search systematically for new senses. a number of different research groups around the world have used the system to annotate data (e.g. @xcite @xcite and continue to do so: currently, the tool is being used to annotate an italian dataset.