Methodological Insights in Detecting Subtle Semantic Shifts with Contextualized and Static Language Models (2023)

Abstract: In this paper, we investigate automatic detection of subtle semantic shifts between social communities of different political convictions in Dutch and English. We perform a methodological study comparing methods using static and contextualized language models. We investigate the impact of specializing contextualized models through fine-tuning on target corpora, word sense disambiguation and sentiment. We furthermore propose a new approach using masked token prediction, that relies on behavioral information, specifically the most probable substitutions, instead of geometrical comparison of representations. Our results show that methods using static models and our masked token prediction method can detect differences in connotation of politically loaded terms, whereas methods that rely on measuring the distance between contextualized representations are not providing clear signals, even in synthetic scenarios of extreme shifts.

## introduction with the increasing popularity and success of language models, lexical semantic change detection (lscd) has developed into its own research field. the primary focus has been placed on the automatic detection of extreme semantic shifts across time periods (i.e. diachronic shifts). more subtle semantic shifts such as differences in connotations attributed to words by socially different communities have received little attention. such shifts can, however, provide highly relevant insights for social scientists, as they may reflect fundamental differences in world views. as connotations and associations are likely to be reflected by the contexts of words, static and contextualized language models have the potential to capture them. the current state of the field does not provide conclusive predictions of the most effective models and methods for detecting subtle shifts in connotation. the best-performing methods for lscd predominantly rely on static embedding representations. several methods for contextualized models have been proposed but have generally been outperformed by static methods on lscd tasks @xcite . incidental results from a russian and spanish shared task @xcite show that contextualized models can outperform static models when given the right information through fine-tuning on a task that foregrounds sense information. in this paper, we systematically investigate different methods using contextualized and static models on a use case of subtle semantic shifts in political discourse. we test different approaches to prioritize relevant information in the model and investigate how contextualized models react to information in the input data. our experiments yield the following insights: (1) standard methods relying on contextualized representations fail to detect subtle shifts. we use a data manipulation approach to simulate extreme shifts and demonstrate that none of the methods can detect them. our experiments confirm the intuition from @xcite that information learned in pretraining provides such a strong signal that contexts reflecting a new sense of a target token do not impact its representation. (2) the best-performing lscd system of the russian and spanish shared task @xcite did not perform best on our subtle shift task nor our evaluation on the english subtask of se-meval 2020. (3) we propose an alternative method for contextualized models relying on masked token prediction rather than representation comparison inspired by @xcite . in contrast to the representation-based contextualized methods, this method is able to predict subtle shifts. our substitution method has the additional advantage that it is more transparent, as the predicted substitutions give insights into the nature of the shifts. in addition to our main insights, we confirm that word form has an impact on the performance of contextualized models (as suggested by @xcite and examined by @xcite ); if raw test data is provided, instead of lemmatized (which is a de-facto input style for lexical change experiments), bert outperforms static models on semeval. in this section, we first provide an overview of approaches for lexical semantic change detection (lscd), the task of automatically detecting shifts in semantics. 2 this is followed by a brief presentation of proposed approaches for validating lscd methods and a discussion of work that addresses social variation rather than diachronic change. ## task & data in this study, we investigate the use of language models for the detection of subtle semantic shifts between different social communities. we introduce the following objective to address this issue: task: given (i) a target word whose connotative meaning is assumed to differ between two communities, and (ii) two subcorpora representing (one of) the two communities in which the target word occurs, determine whether these subcorpora originate from the same community or different communities. we address subtle semantic shifts through two use cases focusing on political discourse in dutch (use case 1) and english (use case 2). we leverage user-generated web-data from reddit, a social media platform that enables the formation of communities (subreddits) based on a wide range of topics and interests. we use political subreddits and examine how politically loaded terms undergo connotational semantic shifts across different communities. for example, the word climate carries the connotation of climate change being a hoax in right-wing discourse, while it tends to imply a genuine threat in left-wing discourse. use case 1 we collect comments from two dutch subreddits on opposing ends of the dutch political spectrum: poldersocialisme (ps), a left-wing community, and forum_democratie (fd), a rightwing community. we have collected all the existing comments from these two subreddits up until april 3, 2022, using the pushshift api @xcite . ## methods & experimental set-up in this section, we discuss the methods we employ to address the task of detecting subtle semantic shifts as defined in the previous section. we evaluate all methods on our subtle connotative shift task. in addition, we test the performance of all methods on the semeval 2020 task to examine their behavior on denotational shifts. ## results in this section we first present the results on our main task of subtle semantic shift detection within our use-cases (5.1). in section 5.2 we discuss the results on simulated extreme shifts, after which the insights generated by our mask prediction method are showcased (5.3). next, we evaluate our methods on the semeval 2020 shared task (5.4). finally, section 5.5 provides an in-depth analysis of errors made by contextualized representation methods. ## discussion & conclusion we presented a systematic study of different methods using contextualized and static language models for detecting subtle semantic shift in political discourse in dutch and english. based on a limited but carefully selected test set, we found that methods based on static representations show strong performance in capturing these shifts, while con-textualized representations, even when fine-tuned on specialized information such as domain or sentiment, fail. through simulation experiments with extreme semantic shifts, we revealed that representations extracted from pretrained contextualized models lack sensitivity to context, which contributes to their poor performance. we thus provide empirical evidence for speculations about the dominance of pretraining signals from previous work. we explore the masked token prediction capabilities of contextualized models as an alternative and transparent method for subtle shift detection. our approach relies on top k target word substitutions, and proved highly effective on our subtle semantic shift task. it allows for the analysis of predicted substitutions, which provided deeper insights into the different connotations attributed to the target word by different communities. follow-up research could explore alternative values for k and use the prediction probabilities as additional signals. a threshold approach based on substitute probability could yield more informative substitutes, tuned for the targeted semantic shift type, and thus better performance on the semeval data. our evaluation on the semeval task revealed that model performance differs substantially in relation to the types of shifts under investigation. our results on the semeval task, which targets clear semantic changes in denotation over time, show that representations of pretrained bert models can reflect such changes to a certain degree and underscore the importance of word form for contextualized models, as emphasized in prior research. our findings contribute to the understanding of semantic shifts in language and highlight the challenges and opportunities associated with the lessexplored task of detecting subtle semantic shifts.