LSCD iscovery: A shared task on semantic change discovery and detection in S panish (2022)

Abstract: We present the first shared task on semantic change discovery and detection in Spanish. We create the first dataset of Spanish words manually annotated by semantic change using the DURel framewok (Schlechtweg et al., 2018). The task is divided in two phases: 1) graded change discovery, and 2) binary change detection. In addition to introducing a new language for this task, the main novelty with respect to the previous tasks consists in predicting and evaluating changes for all vocabulary words in the corpus. Six teams participated in phase 1 and seven teams in phase 2 of the shared task, and the best system obtained a Spearman rank correlation of 0.735 for phase 1 and an F1 score of 0.735 for phase 2. We describe the systems developed by the competing teams, highlighting the techniques that were particularly useful.

## introduction lexical semantic change detection (lscd) is the task of detecting words which have changed their meaning over time in a diachronic corpus of text @xcite , usually an unsupervised task. in recent years, several lscd shared tasks have been organized @xcite @xcite . these tasks have contributed to a better understanding of lscd, but have also had their shortcomings: (i) they have used mainly small pre-selected sets of target words creating an unrealistic evaluation scenario for the application of computational models in historical semantics and lexicography where researchers typically aim to cover the full vocabulary of a language @xcite , (ii) different formalizations of the lscd task have been proposed including binary classification and ranking tasks @xcite @xcite and these have been employed inconsistently, and (iii) none of them have focused on spanish, despite the fact that there are more than 450 million native speakers of this language. we tackle these shortcomings by organizing a shared task on spanish diachronic data with a more realistic evaluation scenario requiring participants to provide lexical semantic change (lsc) predictions for the full corpus vocabulary (discovery). additionally, we cover previous scenarios by asking participants to predict lsc only in the limited sample of annotated target words @xcite . by offering a range of additional optional tasks (defined on the same annotated data) participants are able to evaluate and compare models on various formalizations of the lscd task. in order to derive gold lsc labels for target words, we annotate and publish the largest existing data set of semantic proximity judgments covering 100 words with approximately 62k judgments from 12 human native speakers. 1 ## related work the detection of lexical semantic changes is of great interest in research areas such as historical semantics, lexicography, linguistics and nlp. for a comprehensive review of the literature on the area we refer the reader to the recent surveys @xcite @xcite . in previous years several shared tasks have been organized: semeval-2020 task 1 @xcite for english, german, latin, and swedish, diacr-ita for italian @xcite , and rushifteval for russian @xcite . 2 all shared tasks applied an evaluation setup where lsc was measured between pairs of time periods. semeval used a total of 156 target words for all languages with no development/test split. ap-proximately half of these were drawn from etymological dictionaries or research literature, while the other half was drawn from the corpus vocabularies by selecting lemmas with similar pos and frequency as the first half of target words. target word occurrences in sentences (usages) were combined into pairs and these were annotated for their semantic proximity @xcite . target words were excluded if they had a high number of undecidable use pairs or were annotated too sparsely. sense clusters were inferred from the annotation. from the clusters a binary (sense loss/gain vs. none) and a graded (jensen-shannon distance between cluster distributions) change score were derived and used to evaluate participants on a corresponding binary classification and ranking task. diacr-ita used a total of 18 target words with no development/test split. all of these were drawn from an etymological dictionary. target word usages were annotated with word sense definitions. words with a high number of ocr errors and annotator disagreements were excluded. from the annotation binary change scores similar to semeval were derived and used to evaluate participants on a binary classification task. rushifteval used a total of 111 target words (all nouns) split into 12 for development and 99 for testing. these were selected in a similar procedure to semeval: approximately half of these were drawn from etymological dictionaries, research literature or "invented" by the authors, while the other half was drawn from the corpus vocabularies by selecting lemmas with similar pos and frequency as the first half of target words. target word usages from different time periods were combined into usage pairs and annotated for semantic proximity. from these the durel compare score (see subsection 3.3 for more details) @xcite was derived, which can be seen as an approximation of semeval's graded change score @xcite . participants were evaluated in a ranking task on the compare scores. as we can see, target words in previous shared tasks have been strongly preselected and systems have been evaluated on different tasks. they have also yielded (seemingly) contradictory results: while type-based model architectures have dominated in semeval and diacr-ita, token-based architectures have dominated in rushifteval. in all tasks clustering-based models have shown rather low performance. ## task description our task was designed in two phases: note that discovery introduces additional difficulties for models as compared to the more simple semantic change detection, e.g. because a large number of predictions is required and the target words are not preselected, balanced or cleaned (cf. @xcite . yet, discovery is an important task, with applications such as lexicography where dictionary makers aim to cover the full vocabulary of a language. ## systems we now summarize the baseline systems as well as the systems and resources used by the participating teams. ## results the results shown in tables 4, 5 and 6 correspond to the best submissions per subtask. 21 graded change discovery as shown in ## conclusion we conducted the first shared task on semantic change discovery and detection in spanish. we manually annotated 100 spanish words for semantic change between two corpora, an old one covering the period between 1810 and 1906, and a modern one covering the years between 1994 and 2020. the discovery part of our task imposed several computational challenges for participants, as it required calculating semantic change scores for all words in the vocabulary. we received predictions from six teams in phase 1 and seven teams in phase 2. participants applied systems using static and contextualized word embeddings in combination with various finetuning procedures, vector aggregation methods and change measures. graded change discovery was solved with high performance while binary change detection still remains far from being solved. the most successful method winning both main tasks is a system fine-tuning contextualized multilingual xml-r embeddings on wsd data, aggregating vectors into cross-corpus pairs and measuring change as the average of their distances, or a binarization of these values. however, we showed that this approach has a clear upper bound which will not allow to solve the tasks completely reliably in the future. another interesting result from our task was that clustering approaches are amongst the winning teams for the first time. we hope that this shared task will help pave the way for future research in the discovery and detection of semantic lexical changes for the spanish language, and that our data can be used in the future for the proposal of novel ideas and techniques. example lemma: sexo context: 136. los apellidos de familia no varían de terminación para los diferentes sexos; y así se dice «don pablo herrera», «doña juana hurtado», «doña isabel donoso». 137 (b). indexes_target_token: 75:81 after the shared task we uploaded a data version with corrected indices.