Do Not Fire the Linguist: Grammatical Profiles Help Language Models Detect Semantic Change (2022)

Abstract: Morphological and syntactic changes in word usage — as captured, e.g., by grammatical profiles — have been shown to be good predictors of a word’s meaning change. In this work, we explore whether large pre-trained contextualised language models, a common tool for lexical semantic change detection, are sensitive to such morphosyntactic changes. To this end, we first compare the performance of grammatical profiles against that of a multilingual neural language model (XLM-R) on 10 datasets, covering 7 languages, and then combine the two approaches in ensembles to assess their complementarity. Our results show that ensembling grammatical profiles with XLM-R improves semantic change detection performance for most datasets and languages. This indicates that language models do not fully cover the fine-grained morphological and syntactic signals that are explicitly represented in grammatical profiles. An interesting exception are the test sets where the time spans under analysis are much longer than the time gap between them (for example, century-long spans with a one-year gap between them). Morphosyntactic change is slow so grammatical profiles do not detect in such cases. In contrast, language models, thanks to their access to lexical information, are able to detect fast topical changes.

## introduction human language is in continuous evolution. new word senses arise, and existing senses can change or disappear over time as a result of social and cultural dynamics or technological advances. nlp practitioners have become increasingly interested in this diachronic perspective of semantics. some works focus on constructing, testing and improving psycholinguistic and sociolinguistic theories of meaning change (xu and kemp, 2015; hamilton * equal contribution, the authors are listed alphabetically. @xcite @xcite ; others are concerned with surveying how the meaning of words has evolved historically @xcite or how it is currently transforming in public discourse @xcite . recently, we also see increased interest in more application-oriented work, with efforts to develop adaptive learning systems that can remain up-to-date with humans' continuously evolving language use (temporal generalization; @xcite . an increasingly popular way to determine whether and to what degree the meaning of words has changed over time is to use 'contextualised' (or 'token-based') word embeddings extracted from large pre-trained language models @xcite as they encode rich, context-sensitive semantic information. however, it has also been shown recently that changes in the frequency distribution of morphological and syntactic features of words, as captured by grammatical profiles, can also be employed for lexical semantic change detection @xcite , with competitive performance. these are, to some extent, two opposing approaches: while language models (lms) are largely based on word co-occurrence statistics, grammatical profiles are de-lexicalised and rely on explicit linguistic information. although they are superficially unaware of morphology and syntax, lms have been shown to capture approximations of grammatical information in their deep representations @xcite ). yet are these sufficient to detect meaning shifts that are accompanied by morphosyntactic changes in word usage? we hypothesise that this is not the case, and to test this hypothesis, we combine lm-based methods and grammatical profiles into ensemble models of lexical semantic change detection. 1 if adding grammatical profiles to lms results in a boost in performance, then this means that lms do not capture morphosyntactic change as accurately as explicit morphological tagging and syntactic parsing (or at the very least that it is difficult to extract this type of information from the models). if we do not observe any boost, this suggests that lms already represent all the necessary grammatical information and explicit linguistic annotation is not required. we conduct our experiments with 10 datasets, covering 7 languages. for comparability, we use the same model for all the languages. we choose xlm-r @xcite , a multilingual transformer-based masked language model which has already been successfully applied to the semantic change detection task @xcite . although it covers the full linguistic diversity of our data, we additionally fine-tune xlm-r on monolingual diachronic corpora. our quantitative and qualitative evaluation of the resulting ensembles on the graded and binary semantic change detection tasks largely confirm our hypothesis. ensembling xlm-r and grammatical profiles improves the results for 4 out of 6 languages in graded change detection (as well as for 1 of the 2 norwegian datasets) and for 5 out of 6 languages in binary change detection. figure 1 illustrates these improvements. the reasons why bine lms with grammatical profiles as 'ensembles'. these are not statistical methods of ensemble learning but systems that combine the predictions of different models. ## tasks and data the goal of lexical semantic change detection is to determine whether and to what extent a word's meaning has changed over a certain period of time. the performance of automatic systems that address this problem is typically assessed in two tasks @xcite . task 1 is a binary classification task: given a diachronic corpus and a set of target words, a system must determine whether the words lost or gained any senses between two time periods. we refer to it as the classification task and use accuracy as an evaluation metric. task 2 is a ranking task: a system must rank the target words according to the degree of their semantic change. we refer to it as the ranking task and use the spearman rank-correlation with the gold rankings as an evaluation metric. we rely on a collection of diachronic corpora and annotated target word lists covering seven languages from three indo-european language families. target words are annotated with binary and graded scores of semantic change, corresponding respectively to task 1 and 2 @xcite . english (en), german (de), latin (la), and swedish (sw) data are available from the sem-eval 2020 unsupervised lexical semantic change detection shared task @xcite . for italian (it), we use the data released for the evalita competition @xcite . for norwegian (no), we use the nordiachange dataset recently released by @xcite , consisting of two subsets with different target word lists and time spans. finally, for russian (ru), we draw from the rushifteval shared task @xcite , consisting of three subsets with different time spans and a shared target word list. en de it la no-1 no-2 ru-1 ru-2 ru-3 sw period 1 1810-1860 1800 @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite ## analysis in this section, we analyse the predictions of all methods beyond task performance. we quantitatively evaluate their complementarity (section 5.1), and investigate whether and how predictions made with grammatical profiles improve the performance of embedding-based metrics (section 5.2). ## conclusion we showed that providing large pre-trained language models with explicit morphosyntactic information can in many cases help detect and quantify lexical semantic change. such 'ensemble' predic-tions are produced in a very straightforward wayi.e., by computing the geometric mean between semantic change scores predicted by grammatical profiles and by language models (via their contextualized embeddings). in the majority of the datasets under analysis (treating the three russian datasets as one), the ensemble predictions outperformed single grammatical profiles or contextualised embeddings in the task of ranking words by the degree of their semantic change. the datasets where this was not true are characterized by specific properties: either languages with poor morphology or long time spans separated by narrow gaps. we believe this means that although transformer-based language models (like xlm-r, which we used here) are able to track morphological and syntactic properties to some extent @xcite , their encoding of grammatical features is only approximate and can therefore be improved by explicit linguistic preprocessing (morphological tagging and syntactic parsing). at any rate, we showed that this is true for the semantic change detection task, when a model has to take into account diachronic changes in morphosyntactic properties of words. the signal provided by these changes is complementary to the changes in typical lexical contexts more easily captured by distributional language models. thus, it is still too early to fire the linguist, even if the 'linguist' is in fact an automated tagger. as has already been said, an important limitation of grammatical profiles is their low performance when measuring semantic change across long time periods separated by very narrow gaps. this makes sense from a linguistic point of view: grammar changes slowly and gradually, sharp bursts are rare. in contrast, lexical contexts can change very quickly: for example, due to social and political events or technical progress, which is why language models excel with these datasets. the main practical take-away is therefore that diachronic grammatical profiles should be used in combination with language models especially when the gap between the compared time periods is large enough for significant grammatical changes to occur. in the future, we plan to experiment with more sophisticated ensembling methods that go beyond simple averaging (including the usage of the information about gaps between time spans), and to perform a deeper analysis of ensemble predictions, especially in relation to distinct word senses. fi-nally, we also plan to evaluate ensembles formed with monolingual language models, instead of the multilingual xlm-r, as they have the potential to better capture the idiosyncrasies of specific languages.