Grammatical Profiling for Semantic Change Detection (2021)

Abstract: Semantics, morphology and syntax are strongly interdependent. However, the majority of computational methods for semantic change detection use distributional word representations which encode mostly semantics. We investigate an alternative method, grammatical profiling, based entirely on changes in the morphosyntactic behaviour of words. We demonstrate that it can be used for semantic change detection and even outperforms some distributional semantic methods. We present an in-depth qualitative and quantitative analysis of the predictions made by our grammatical profiling system, showing that they are plausible and interpretable.

## introduction lexical semantic change detection has recently become a well-represented field in nlp, with several shared tasks conducted for english, german, latin and swedish @xcite , italian @xcite and russian @xcite . the overwhelming majority of solutions employ either static word embeddings like word2vec @xcite or more recent contextualised language models like elmo @xcite and bert @xcite . these models build upon the distributional semantics hypothesis and can capture lexical meaning, at least to some extent (e.g., @xcite @xcite . thus, they are naturally equipped to model semantic change. yet it has long been known for linguists that semantics, morphology and syntax are strongly interrelated @xcite . semantic change is consequently often accompanied by morphosyntactic drifts. consider the english noun 'lass': in the 20 th century, its 'sweetheart' meaning became more dominant * equal contribution, the authors listed alphabetically. figure 1 : changes in the number category distribution for the english noun 'lass' over time, calculated on the english corpora of the semeval 2020 shared task 1 @xcite . 'lass' is annotated as semantically changed in the semeval dataset. exploiting distributions of grammatical profiles-i.e., morphological and syntactic featuresto detect lexical semantic change is the focus of this paper. we investigate to what extent lexical semantic change can be detected using only morphosyntax. our main hypothesis is that significant changes in the distribution of morphosyntactic categories can reveal useful information on the degree of the word's semantic change, even without help from any lexical or explicitly semantic features. due to the interdependence of semantics and morphosyntax, it is often difficult to determine which type of change occurred first, and whether it triggered the other. establishing the correct causal direction is outside the scope of this study; it is sufficient for us to know that semantic and morphosyntactic changes often co-occur. by proposing this functionalist approach to lexical semantic change detection, we are not aiming at establishing a new state-of-the-art. this is hardly possible without taking semantics into account. but what exactly is possible in such a functionalist setup? we investigate this question experimentally foot_0 using standard semantic change datasets for english, german, swedish, latin, italian and russian. our main findings are the following: ## related work behavioural profiles were introduced in corpus linguistics by @xcite as the set of syntactic and lexical preferences of a word, revealed by studying a large concordance extracted from a corpus. the behavioural profile of a word consists of corpus counts of various linguistic properties, including morphological features, preferred types of clauses and phrases, collocates and their semantic types @xcite . subtle distinctions in word meaning are reflected in behavioural profiles. indeed this technique, which combines lexical and grammatical criteria for word sense distinction, was used to study synonymy and polysemy @xcite as well as antonymy @xcite . one of the theoretical roots for profiling is the theory of lexical priming @xcite . according to this theory, words trigger a set of grammatical and lexical constraints, referred to as primings and stored in a mental concordance. the theory states that 'drifts in priming ... provide a mechanism for temporary or permanent language change' @xcite , and since primings are thought to be organised in the mental concordance in the form of behavioural profiles @xcite , it is theoretically plausible that diachronic word meaning change is reflected in a change of behavioural profiles. as far as we are aware, this idea has not been further developed in corpus linguistics. in spite of its theoretical validity, behavioural profiling as a practical data analysis technique has serious limitations. profiles include a large variety of word properties and some of them, especially those related to semantics, cannot be easily extracted from a corpus automatically. usually, a particular subset of word properties is selected based on researchers' intuition and background knowledge, and statistical tests are sometimes used for feature selection at later stages of the analysis @xcite . moreover, the variety of properties comprised in a behavioural profile makes statistical analysis difficult due to correlations between language phenomena of different levels and sparsity of the data @xcite , section 2.2.2). for these reasons, some studies @xcite reduce a word's possibly very broad behavioural profile to a more compact grammatical profile, i.e. a set of preferred morphological forms for the word. these studies too, however, rely on an a priori selection of relevant morphological tags. these technical difficulties may explain why profiling has not been used in computational approaches to lexical semantic change detection. most attempts to tackle word meaning change in nlp are based on distributional patterns of lexical co-occurrences, starting from early countbased approaches @xcite , continuing with dimensionality reduction techniques @xcite , and later accelerated by embeddings-based models @xcite . more recently, contextualised embeddings were also applied to this task @xcite . as far as we are aware, there is one exception to this trend: @xcite employed grammatical profiles to detect the semantic change of russian nouns. in their work, a profile of case and number frequency distributions is collected separately for each time period, and the degree of semantic change is measured as the cosine distance between the two distributions. the results obtained with this method are close to the results yielded by word2vec embeddings, but lower than those of contextualised embeddings. inspired by @xcite , we further investigate the ability of grammatical profiles to capture word meaning change. we propose a number of improvements and evaluate them on datasets in six different languages. most importantly, we use all available morphological tags, without any manual pre-selection, and we conduct an in-depth analysis of our results to understand why grammatical profiling works for this task and what are its limitations. ## data and tasks following the standard evaluation approach adopted for automatic lexical semantic change detection, we cast the problem as either binary classification (subtask 1, using the terminology of the semeval 2020 unsupervised lexical semantic change detection shared task @xcite ) or as a ranking task (subtask 2). in subtask 1, given a set of target words, a system must determine whether the words lost or gained any senses between two time periods. in subtask 2, a system has to rank a set of target words according to the degree of their semantic change. annotating data for word meaning change detection is a non-trivial process because it requires taking into account numerous word occurrences from every time period of interest. the current practice adopted in the community is to annotate pairs of sentences containing a target word used either in the same or in a different sense; then pairwise scores are aggregated to obtain a final measure of change, either binary or continuous @xcite . this procedure has been used by organizers of three recent shared tasks: the sem-eval 2020 unsupervised lexical semantic change detection shared task @xcite , evalita @xcite and rushifteval @xcite . we use the data from these three shared tasks, allowing to compare our approach with the state-of-the-art results obtained by distributional models. the semeval dataset consists of target words in four languages-37 english, 48 german, 40 latin, and 32 swedish-that are manually annotated for both subtasks. the evalita dataset consists of 18 italian words annotated for subtask 1 only. fi-nally, the rushifteval dataset consists of 99 russian nouns annotated for subtask 2. all datasets are accompanied by diachronic corpora. most of the corpora are split in two time periods, except for the rushifteval corpus, which is separated into three time bins: russian1 and russian2 are annotated with semantic shifts between the pre-soviet and soviet periods, and between the soviet and post-soviet periods respectively; russian3 is annotated with semantic shifts between the pre-soviet and post-soviet periods @xcite . in sum, we have at our disposal several dozens words from three indo-european language groups: italic, germanic and slavic. though our results may not generalize to other language families or to other languages within the families analysed, these are the most diverse data that are currently available for this kind of study. ## results we evaluate our method on both subtasks of the semeval 2020 unsupervised lexical semantic change detection shared task @xcite . as described in section 3, subtask 1 is a binary classification task, evaluated with accuracy. subtask 2 is a ranking task, evaluated with spearman's rank correlation. basic procedure using only morphological features, we obtain an average correlation of 0.181 across the four semeval languages, as can be seen in frequency threshold filtering out rare features as described in section 4.2 has a small but positive impact on all three setups: +0.011 for morphological features, +0.033 for syntactic features, and +0.065 for the combination of the two. ## qualitative analysis in section 5, we showed that grammatical profiling alone can detect a word meaning change better than count-based distributional semantic models which exploit lexical co-occurrence statistics. this is a remarkable finding: it confirms that meaning change leaves traces in grammatical profiles and it demonstrates that these traces can be used as effective predictors of a word's meaning stability. in this section, to better understand when change in grammatical profiles is a good indicator of lexical *the rushifteval baseline relies on cbow word embeddings and their local neighborhood similarity. @xcite used an ensemble method with much higher performance, we report the results obtained solely with profiling. while semeval results are fully unsupervised, the best rushifteval results are supervised and not directly comparable to our setting. *the italian baseline relies on collocations @xcite : for each target word, two vector representations are built, with the bag-of-collocations related to the two different time periods. then, the cosine similarity between them is computed. we begin by analysing the most accurately ranked words (see appendix a). the italian word 'lucciola', for example, is ranked 1 st out of 18 by our method due to the singular usages of the word disappearing after 1990. the singular usage is indeed much more likely for the dying sense of the word (an euphemism for 'prostitute'), whereas the plural form 'lucciole' is more likely used for the stable sense of the word ('fireflies') or in the idiomatic expression prendere lucciole per lanterne (getting the wrong end of the stick), which makes up for most of the occurrences between 1990 and 2014. another example of correctly identified semantically shifted words is the latin 'imperator' (ranked 1 st out of 40). in the second time periodranging from 0 to 2000 a.d.-nominative usages become predominant. a possible explanation for this change is that the more frequent agentive usages of the word correspond to the new role of the 'emperor' in the imperial rome (27 b.c. to a.d. 476) rather than that of a generic 'commander'the older sense of the word. foot_6 for english, the noun 'stab' is ranked 4 th out of 37, mostly because of syntactic changes: 27% of its occurrences in the 20 th century are used as oblique arguments, compared to only 13% in the 19 th century. this is arguably associated with the emergent sense of 'sudden sharp feeling' ('...left me with a sharp stab of sadness'). the german word 'artikulieren' correctly receives a high rank (9 th out of 48): it occurs only 3 times in the 19th century and 210 times between 1946 and 1990, shifting towards a much richer grammatical profile. sharp changes in frequency are reflected in the diversity of grammatical profiles and can also help detect lexical semantic change. our qualitative analysis reveals that the successful examples are often cases of broadening and narrowing of word meaning. these kinds of semantic change seem to be easily picked with profiling. however, some examples of broadening and narrowing fail to be detected, as will be shown in section 6.2, especially if they involve metaphorical extensions of word meaning. a consistent characterisation of the kinds of semantic change detected and overlooked by our method would require diachronic corpora where both the degree and the type of semantic changes are annotated. ## category importance in this section, we conduct an additional experiment to find out which grammatical categories are most related to semantic change. to this end, we train logistic regression classifiers for binary classification using english, german, latin, swedish and italian data. the classifier features are cosine distances between frequency vectors of each particular category from different time bins. before fitting the classifier, each feature is independently zero-centered and scaled to the unit variance. then, regression coefficients are estimated for each feature: we consider positive weights as an indication of usefulness of a feature for classification. the outcome of this analysis is shown in in line with the results presented in section 5, swedish and italian classifiers yield the highest accuracy and f-score. latin, a highly inflectional language, has by far the largest set of categories contributing positively to semantic change detection (interestingly, excluding syntax). english, a highly analytical language, is on the other end of the spectrum. additionally, we estimate the relative importance of morphosyntactic categories by calculating the spearman's rank-correlation of their respective cosine distance values (across all target words) with the gold semantic change rankings. in other words, we single out each category, e.g. verbal mood, and test whether diachronic change in its frequency distribution is correlated with manually annotated semantic change scores. in what is surprising, however, is that changes in gender are also correlated with semantic change in the russian case. this result is hard to inter- ## conclusion semantic change is inextricably tied to changes in the distribution of morphosyntactic properties of words, i.e. their grammatical profiles. in this paper, we showed that tracking these changes is enough to build a semantic change detection system which, without access to any lexical semantic information, consistently outperforms count-based distributional semantic approaches to the task. grammatical profiling yields surprisingly good evaluation scores across different languages and datasets, without any language-specific tuning. for latin, a language with rich morphology, our methods even establish a new sota in subtask 2 of semeval'20 task 1. these results indicate that grammatical profiling cannot compete with state-of-the-art methods based on large pre-trained language models, since they have the potential to encode both semantics and grammar. yet reaching the highest possible scores on the task was not our goal. instead, the aim of our study was to demonstrate that more attention should be paid to the relation between morphosyntax and semantic change. whether morphosyntactic and semantic features are complementary and can be successfully combined is a interesting question to be addressed in future work. we performed an extensive quantitative and qualitative analysis of our semantic change detection methods, showing that profiling yields interpretable results across several languages. nevertheless, we still lack an understanding of some aspects of the interaction between semantics and morphosyntax. finding the reasons behind the relatively poor performance on some datasets, e.g. german, is an important direction for future studies. another interesting question is how to incorporate full dependency trees into grammatical profiles, rather than only dependency relations to the syntactic head of a word. this is particularly important for analytical languages, where grammatical markers are presented in more than one word, such as with english verb mood and aspect. moreover, dependency structure can be crucial for languages from families other than the indo-european, e.g. to take into account detached counters in japanese or plural markers in yoruba. in light of our experimental results, we argue that grammatical profiling should become one of the standard baselines for semantic change detection.