Lexical Semantic Change Discovery (2021)

Abstract: While there is a large amount of research in the field of Lexical Semantic Change Detection, only few approaches go beyond a standard benchmark evaluation of existing models. In this paper, we propose a shift of focus from change detection to change discovery, i.e., discovering novel word senses over time from the full corpus vocabulary. By heavily fine-tuning a type-based and a token-based approach on recently published German data, we demonstrate that both models can successfully be applied to discover new words undergoing meaning change. Furthermore, we provide an almost fully automated framework for both evaluation and discovery.

## introduction there has been considerable progress in lexical semantic change detection (lscd) in recent years @xcite @xcite , with milestones such as the first approaches using neural language models @xcite , the introduction of orthogonal procrustes alignment @xcite , detecting sources of noise @xcite , the formulation of continuous models @xcite @xcite , the first uses of contextualized embeddings @xcite , the development of solid annotation and evaluation frameworks @xcite @xcite and shared tasks @xcite . however, only a very limited amount of work applies the methods to discover novel instances of semantic change and to evaluate the usefulness of such discovered senses for external fields. that is, the majority of research focuses on the introduction of novel lscd models, and on analyzing and evaluating existing models. up to now, these preferences for development and analysis vs. application represented a well-motivated choice, because the quality of state-of-the-art models had not been established yet, and because no tuning and testing data were available. but with recent advances in evaluation @xcite @xcite , the field now owns standard corpora and tuning data for different languages. furthermore, we have gained experience regarding the interaction of model parameters and modelling task (such as binary vs. graded semantic change). this enables the field to more confidently apply models to discover previously unknown semantic changes. such discoveries may be useful in a range of fields @xcite , among which historical semantics and lexicography represent obvious choices @xcite . in this paper, we tune the most successful models from semeval-2020 task 1 @xcite on the german task data set in order to obtain high-quality discovery predictions for novel semantic changes. we validate the model predictions in a standardized human annotation procedure and visualize the annotations in an intuitive way supporting further analysis of the semantic structure relating word usages. in this way, we automatically detect previously described semantic changes and at the same time discover novel instances of semantic change which had not been indexed in standard historical dictionaries before. our approach is largely automated, by relying on unsupervized language models and a publicly available annotation system requiring only a small set of judgments from annotators. we further evaluate the usability of the approach from a lexicographer's viewpoint and show how intuitive visualizations of human-annotated data can benefit dictionary makers. ## related work state-of-the-art semantic change detection models are vector space models (vsms) @xcite . these can be divided into type-based (static) (turney and pantel, 2010) and token-based (contextualized) @xcite approaches. for our study, we use both a static and a contextualized model. as mentioned above, previous work mostly focuses on creating data sets or developing, evaluating and analyzing models. a common approach for evaluation is to annotate target words selected from dictionaries in specific corpora @xcite @xcite @xcite . contrary to this, our goal is to find 'undiscovered' changing words and validate the predictions of our models by human annotators. few studies focus on this task. @xcite al. (2014), hamilton et al. (2016), basile et al. (2016), basile and mcgillivray (2018), takamura et al. (2017) and tsakalidis et al. (2019) evaluate their approaches by validating the top ranked words through author intuitions or known historical data. the only approaches applying a systematic annotation process are @xcite and @xcite . gulordava and baroni ask human annotators to rate 100 randomly sampled words on a 4-point scale from 0 (no change) to 3 (changed significantly), however without relating this to a data set. @xcite closely with a professional lexicographer to inspect 20 lemmas predicted by their models plus 10 randomly selected ones. gulordava and baroni and cook et al. evaluate their predictions on the (macro) lemma level. we, however, annotate our predictions on the (micro) usage level, enabling us to better control the criteria for annotation and their inter-subjectivity. in this way, we are also able to build clusters of usages with the same sense and to visualise the annotated data in an intuitive way. the annotation process is designed to not only improve the quality of the annotations, but also lessen the burden on the annotators. we additionally seek the opinion of a professional lexicographer to assess the usefulness of the predictions outside the field of lscd. in contrast to previous work, we obtain model predictions by fine-tuning static and contextualized embeddings on high-quality data sets @xcite ) that were not available before. we provide a highly automated general framework for evaluating models and predicting changing words on all kinds of corpora. ## data we use the german data set provided by the semeval-2020 shared task @xcite . the data set contains a diachronic corpus pair for two time periods to be compared, a set of carefully selected target words as well as binary and graded gold data for semantic change evaluation and fine-tuning purposes. ## models type-based models generate a single vector for each word from a pre-defined vocabulary. in contrast, token-based models generate one vector for each usage of a word. while the former do not take into account that most words have multiple senses, the latter are able to capture this particular aspect and are thus presumably more suited for the task of lscd @xcite . even though contextualized approaches have indeed significantly outperformed static approaches in several nlp tasks over the past years @xcite , the field of lscd is still dominated by type-based models @xcite . @xcite yet show that the performance of tokenbased models (especially elmo) can be increased by fine-tuning on the target corpora. @xcite drastically improve the performance of bert by reducing the influence of target word morphology. in this paper, we compare both families of approaches for change discovery. ## discovery semeval-2020 task 1 consists of two subtasks: (i) binary classification: for a set of target words, decide whether (or not) the words lost or gained sense(s) between c 1 and c 2 , and (ii) graded ranking: rank a set of target words according to their degree of lsc between c 1 and c 2 . these require to detect semantic change in a small pre-selected set of target words. instead, we are interested in the discovery of changing words from the full vocabulary of the corpus. we define the task of lexical semantic change discovery as follows. given a diachronic corpus pair c 1 and c 2 , decide for the intersection of their vocabularies which words lost or gained sense(s) between c 1 and c 2 . this task can also be seen as a special case of se-meval's subtask 1 where the target words equal the intersection of the corpus vocabularies. note, however, that discovery introduces additional difficulties for models, e.g. because a large number of predictions is required and the target words are not preselected, balanced or cleaned. yet, discovery is an important task, with applications such as lexicography where dictionary makers aim to cover the full vocabulary of a language. ## annotation the model predictions are validated by human annotation. for this, we apply the semeval-2020 task 1 procedure, as described in @xcite . annotators are asked to judge the semantic relatedness of pairs of word usages, such as the two usages of aufkommen in (1) and (2), on the scale in the annotated data of a word is represented in a word usage graph (wug), where vertices represent word usages, and weights on edges represent full c1 c2 the (median) semantic relatedness judgment of a pair of usages such as (1) and ( 2 ). the final wugs are clustered with a variation of correlation clustering @xcite (see figure 1 , left) and split into two subgraphs representing nodes from subcorpora c 1 and c 2 , respectively (middle and right). clusters are then interpreted as word senses and changes in clusters over time as lexical semantic change. in contrast to schlechtweg et al. we use the openly available durel interface for annotation and visualization. 4 this also implies a change in sampling procedure, as the system currently implements only random sampling of use pairs (without semeval-style optimization). for each target word we sample |u 1 | = |u 2 | = 25 usages (sentences) per subcorpus (c 1 , c 2 ) and upload these to the durel system, which presents use pairs to annotators in randomized order. we recruit eight german native speakers with university level education as annotators. five have a background in linguistics, two in german studies, and one has an additional professional background in lexicography. similar to schlechtweg et al., we ensure the robustness of the obtained clusterings by continuing the annotation of a target word until all multi-clusters (clusters with more than one usage) in its wug are connected by at least one judgment. we finally label a target word as changed (binary) if it gained or lost a cluster over time. for instance, aufkommen in figure 1 is labeled as change as it gains the orange cluster from c 1 to c 2 . following schlechtweg et al. (2020) we use k and n as lower frequency thresholds to avoid that small random fluctuations in sense frequencies caused by sampling variability or annotation error be misclasfind an overview over the final set of wugs in ## results we now describe the results of the tuning and discovery procedures. ## lexicographical evaluation we now evaluate the usefulness of the proposed semantic change discovery procedure including the annotation system and wug visualization from a lexicographer's viewpoint. the advantage of our approach lies in providing lexicographers and dictionary makers the choice to take a look into predictions they consider promising with respect to their research objective (disambiguation of word senses, detection of novel senses, detection of archaisms, describing senses in regard to specific discourses etc.) and the type of dictionary. visualized predictions for target words may be analyzed in regard to single senses, clusters of senses, the semantic proximity of sense clusters and a stylized representation of frequency. random sampling of usages also offers the opportunity to judge underrepresented senses in a sample that might be infrequent in a corpus or during a specific period of time (although currently a high number of overall annotations would be required in order to do so). most importantly, the use of a variable number of human annotators has the potential to ensure a more objective analysis of large amounts of corpus data. in order to evaluate the potential of the approach for assisting lexicographers with extending dictionaries, we analyze statistical measures and predictions of the models provided for the two sets of predictions (sgns, bert) and compare them to existing dictionary contents. we consider overall inter-annotator agreement (α >= .5) and annotated binary change label to select 21 target words for lexicographical analysis. in this way, we exclude unclear cases and non-changing words. the target words are analyzed by inspecting cluster visualizations of wugs (such as in figure 1 ) and comparing them to entries in general and specialized dictionaries in order to determine: three dictionaries are consulted throughout the analysis: (i) the dictionary of the german language (dwb) by jacob und wilhelm grimm (digitized version of the 1st print published between 1854-1961), (ii) the dictionary of contemporary german (wgd), published between 1964-1977, now curated and digitized by the dwds and (iii) the duden online dictionary of german language (du-den), reflecting usage of contemporary german up until today. 7 additionally, lemma entries in the wiktionary online dictionary (wiktionary) are consulted to verify genuinely novel senses described in section 8.1. ## conclusion we used two state-of-the-art approaches to lsc detection in combination with a recently published high-quality data set to automatically discover semantic changes in a german diachronic corpus pair. while both approaches were able to discover various semantic changes with above-random probability, some of them previously undescribed in etymological dictionaries, the type-based approach showed a clearly better performance. we validated model predictions by an optimized human annotation process yielding high inter-annotator agreement and providing convenient ways of visualization. in addition, we evaluated the full discovery process from a lexicographer's point of view and conclude that we obtained high-quality predictions, useful visualizations and previously unreported changes. on the other hand, we discovered some issues with respect to the reliability of predictions for semantic change and number and composition of reference corpora that are going to be dealt with in the future. the results of the analyses endorse that our approach might aid lexicographers with extending and altering existing dictionary entries. full c1 c2 figure 4: word usage graph of german lager (left), subgraphs for first time period c 1 (middle) and for second time period c 2 (right). full c1 c2 figure 5: word usage graph of german anriffswaffe (left), subgraphs for first time period c 1 (middle) and for second time period c 2 (right). full c1 c2 figure 6: word usage graph of german neunjährig (left), subgraphs for first time period c 1 (middle) and for second time period c 2 (right).