Literary Intertextual Semantic Change Detection: Application and Motivation for Evaluating Models on Small Corpora (2023)

Abstract: Lexical semantic change detection is the study of how words change meaning between corpora. While Schlechtweg et al. (2020) standardized both datasets and evaluation metrics for this shared task, for those interested in applying semantic change detection models to small corpora—e.g., in the digital humanities—there is a need for evaluation involving much smaller datasets. We present a method and open-source code pipeline for downsampling the SemEval-2020 Task 1 corpora while preserving gold standard measures of semantic change. We then evaluate several state-of-the-art models trained on these downsampled corpora and find both dramatically decreased performance (average 67% decrease) and high variance. We also propose a novel application to the digital humanities and provide a case study demonstrating that semantic change detection can be used in an exploratory manner to produce insightful avenues of investigation for literary scholars.

## introduction semantic meaning is fluid. the word plane, for instance, underwent a dramatic semiotic shift around the early 1900s from the sense of "flat geometric surface" to the sense of "aeroplane" (oed). the last ten years have seen the rise of computational linguistic approaches that attempt to provide unsupervised detection of lexical semantic change @xcite . applications include discovering laws of semantic change @xcite @xcite , investigating the evolution of harmful stereotypes @xcite , or determining how societal relationships to certain concepts experience diachronic drift @xcite , among others. the majority of these fields involve studying large conglomerate corpora as proxies for societal beliefs. in the burgeoning literary digital humanities @xcite @xcite , among other fields, however, one is often invested in studying small corpora, where each corpus is on the order of 150k tokens (about the size of a single authored english fiction novel). @xcite standardized evaluation metrics and datasets for unsupervised semantic change detection, but schlechtweg et al.'s smallest corpus contains over 1.7 million tokens, and their largest over 110 million. in this work, we investigate the degree of performance degradation of semantic change detection models when evaluated on small corpora. we expect this setting to be challenging for the evaluated models due to the limited number of examples of each target word in context available to them. to further motivate the importance of evaluating semantic change detection models on small corpora, we focus on applying these models to aid literary studies. in the context of literary criticism, investigating subtle differences in language between two books often provides the building blocks for broader comparative literary insight. in this work, then, we propose a novel application-literary intertextual semantic change detection, the production of a ranked list of words by degree of semantic change between two books-as an exploratory tool to suggest words that may be of literary interest and suitable for extended investigation (e.g., through comparative close-reading by humans). in this setting, corpora sizes are limited by the length of the books under consideration. finally, as a case study for how, and, importantly to us, whether, current semantic change detection models can be employed to produce fruitful avenues of inquiry for literary scholars, we apply the best performing english model evaluated in sec-1 tion 5.1 to two books-the wretched of the earth @xcite and scenes of subjection (hartman, 1997/2022)-which we suspected may have interesting intertextual semantic changes due to prior domain knowledge. we find that there is reason to be optimistic that semantic change detection can be used in an exploratory manner to aid literary critics. to summarize, our primary contributions are the following: ## evaluating existing methods on small corpora in this paper we evaluate three models that present a range of different architectures, from static (non-contextual) embeddings to contextual embeddings, and are, to our knowledge, among the highest performing open-source models for unsupervised semantic change detection: the authors propose a temporal self-attention mechanism as a modification to the standard transformers architecture. they use a pre-trained bert model, fine-tune it on diachronic corpora using their proposed temporal attention mechanism, and then create timespecific representations of target words by extracting and averaging hidden-layer weights. these representations are then averaged at the token level and compared using cosine similarity. 6 we have used the models essentially as-is from their respective github repositories. hyperparameters for all models were chosen based on those reported in each paper. note that both @xcite and pömsl and lyapin (2020) learn static (non-contextual) embeddings from scratch on the target corpora, while the contextualized model of @xcite is already pre-trained and only fine-tuned on the target corpora. we do not intend for this to be an exhaustive evaluation of all possible methods in the field. instead, we hope to open the door for future research to evaluate other methods for semantic change detection -perhaps nonce2vec @xcite , lsa+svd @xcite , or ppmi+svd @xcite , all of which some literature suggest may perform well on tasks involving small corpora @xcite @xcite . other than models specifically targeting smaller corpora, more recent semeval-style shared tasks in russian and spanish (rushifteval @xcite and lscdiscovery zamora-reina et al. ( 2022 )) have shown that word-in-context (wic) and word sense disambiguation (wsd) models tend to have quite high performance on the task of semantic change detection. the wic models "deepmistake" @xcite or xl-lexeme @xcite , or the wsd model "glossreader" @xcite may be ideal candidates for future evaluation. ## results and evaluation for each model described in section 4, we ran experiments to evaluate performance on downsampled datasets (section 5.1), quantify variability across bootstrap resamples (section 5.2), and analyze performance across corpora size (section 5.3). all reported results are spearman's rank-order correlation coefficient ρ between the predicted and gold-standard lists of target words ranked by degree of semantic change, as is standard across the literature @xcite . ## conclusion in this paper, we presented-to our knowledge, at least-the first evaluation of semantic change detection models on small corpora (approximately 150k tokens). we found that several high-performing semantic change detection models perform significantly worse on standard tasks evaluated on these smaller corpora, on average experiencing a 67% decrease in performance, and demonstrate remarkably high variance across bootstrap resamples. overall, for those in the digital humanities there is a clear need for novel and stable methods that are able to accurately detect lexical semantic changes between small corpora, and we hope that our evaluation framework encourages focus on this low-resource setting. however, through a novel literary application and case study, we also demonstrated that there is reason to be optimistic that semantic change detection can be used in an exploratory manner to aid literary critics. nina tahmasebi, lars borin, adam jatowt, yang xu, and simon hengchen, editors. 2021. computational approaches to semantic change. number 6 in language variation. language science press, berlin. nina tahmasebi and thomas risse. 2017. on the uses of word sense change for research in the digital humanities. in research and advanced technology for digital libraries, pages 246-257, cham. springer international publishing. rohit voleti, julie m. liss, and visar berisha. 2020. a review of automated speech and language features for assessment of cognitive and thought disorders. ieee journal of selected topics in signal processing, 14(2):282-298. claudia volpetti, k. vani, and alessandro antonucci. 2020. temporal word embeddings for narrative understanding. in proceedings of the 2020 12th international conference on machine learning and computing, icmlc 2020, pages 68-72, new york, ny, usa. association for computing machinery. yang xu and charles kemp. 2015. a computational evaluation of two laws of semantic change. in cogsci. frank d. zamora-reina, felipe bravo-marquez, and dominik schlechtweg. 2022. lscdiscovery: a shared task on semantic change discovery and detection in spanish. in proceedings of the 3rd workshop on computational approaches to historical language change, pages 149-164, dublin, ireland. association for computational linguistics.