LSC -Eval: A General Framework to Evaluate Methods for Assessing Dimensions of Lexical Semantic Change Using LLM -Generated Synthetic Data (2025)

Abstract: Lexical Semantic Change (LSC) provides insight into cultural and social dynamics. Yet, the validity of methods for measuring different kinds of LSC remains unestablished due to the absence of historical benchmark datasets. To address this gap, we propose LSC-Eval, a novel three-stage general-purpose evaluation framework to: (1) develop a scalable methodology for generating synthetic datasets that simulate theory-driven LSC using In-Context Learning and a lexical database; (2) use these datasets to evaluate the sensitivity of computational methods to synthetic change; and (3) assess their suitability for detecting change in specific dimensions and domains. We apply LSC-Eval to simulate changes along the Sentiment, Intensity, and Breadth (SIB) dimensions, as defined in the SIBling framework, using examples from psychology. We then evaluate the ability of selected methods to detect these controlled interventions. Our findings validate the use of synthetic benchmarks, demonstrate that tailored methods effectively detect changes along SIB dimensions, and reveal that a state-of-the-art LSC model faces challenges in detecting affective dimensions of LSC. LSC-Eval offers a valuable tool for dimension- and domain-specific benchmarking of LSC methods, with particular relevance to the social sciences.

## introduction lexical semantic change (lsc) provides a unique window into cultural dynamics by revealing how language evolution reflects social changes @xcite . recently developed state-of-the-art (sota) computational methods have expanded our ability to classify established types of lsc, such as generalization and specialization @xcite . efforts have also been directed towards developing methods for measuring newly proposed dimensions of lsc @xcite . nevertheless, the field faces challenges in validating these methods. a major obstacle is the absence of historical benchmark datasets, which restricts the standardization and fair comparison of metrics. additionally, there is a pressing need for fine-grained evaluation methods that save time and resources. to address these challenges, the present study proposes lsc-eval, a novel three-stage evaluation framework for assessing lsc across dimensions and domains. it: (1) develops a scalable, generalpurpose methodology for generating high-quality synthetic sentences using 'scholar-in-the-loop' in-context learning (icl) and a lexical resource to simulate targeted kinds of lsc; (2) uses these datasets to evaluate the validity of alternative lsc detection methods; and (3) identifies the most suitable methods for capturing change in specific dimensions and domains. we apply lsc-eval to test the sensitivity of various methods to synthetically induced changes along three major lsc dimensions -sentiment, intensity, and breadth (sib)as proposed in the sibling framework @xcite ; see the present study addresses two key research questions: (rq1) can synthetic datasets be used to validate methods for measuring lsc dimensions? we hypothesize that sib scores will be associated with levels of induced change; (rq2) which of ## discussion the present study introduced lsc-eval, a threestage general evaluation framework that: (1) creates synthetic datasets featuring scholar-in-the-loop llm-generated sentences to simulate various kinds of lsc; (2) leverages these datasets to assess the sensitivity of computational approaches to synthetic change; and (3) evaluates the suitability of these methods for specific dimensions and domains. we applied this framework to generate synthetic datasets that induce changes across the three dimensions (sentiment, intensity, and breadth) of a novel multidimensional lsc framework, sibling @xcite , using examples drawn from psychology to compare the suitability of alternative methods for detecting these synthetic changes. importantly, these synthetic datasets reflect theory-driven change, operationalized through construct definitions and distinctions drawn from the sibling framework @xcite , which integrates established traditions in lexical semantics, connotation, and affect. this principled approach enables interpretable and targeted benchmarking, and lays the groundwork for generalizing lsc-eval to other theoretical models and domains. our findings support the hypothesis that recently proposed methods (valence index, arousal index, breadth score; @xcite detect synthetic changes on the sib (i.e., sentiment, intensity, and breadth) dimensions. this directly addresses rq1, which asked whether synthetic datasets can be used to validate methods for measuring lsc dimensions. control analyses, which adhered to computational linguistics standards @xcite , confirmed the absence of these effects in shuffled controls. the implications of these findings are two-fold. first, the ability of sib methods to detect changes introduced via silver-standard synthetic data validates their sensitivity and reliability for detecting and quantifying variation in sib, even in artificial environments. second, it confirms the validity of using llm-generated sentences in our icl evaluation suites. to address rq2 -identifying which computational methods are most sensitive to synthetically induced changes in sib -we demonstrated how a synthetic change detection task can assess the sensitivity of various approaches, guiding the selection of the most suitable method for specific dimensions and domains. @xcite tools, which validated the synthetic sib datasets, were supported by alternative methods that consistently detected synthetic changes across conditions and targets, providing further validation. the valence index and absa's sentiment score detected variations in synthetic sentiment, while breadth scores derived from xll and mpnet models detected increases in synthetic breadth. these results suggest that the nlp-based absa method is generally more sensitive to synthetically induced change than warrinerbased approaches, which rely on static valence and arousal ratings. future empirical studies on sentiment and breadth may consider adopting these nlp-based models, either as replacements for, or in addition to, traditional lexicon-based methods. notably, when computing the general lsc score using the sota lsc model xll @xcite , it was not sensitive to detecting sentiment and intensity. although xll shows some sensitivity to identifying synthetic increases in breadth, it registers a more substantial change when the breadth score is adjusted according to the method introduced by @xcite . it uses the withinbin average cosine distance of target-containing sentences as a proxy for the expansion (broadening) or contraction (narrowing) of a word's contextual usage. the inability of xll to detect the affective dimensions of lsc highlights the necessity of evaluating sota models before deploying them in new domains. future research should investigate whether this weakness in detecting affective dimensions is specific to xll or extends to other contextualized models in other corpora and concepts. this inquiry is particularly salient given recent advances in analyzing fine-grained, continuous semantic shifts through "diachronic word similarity matrices using fast and lightweight word embeddings over arbitrary time periods" @xcite . our results support the idea that current sota methods are only state-of-the-art with respect to a particular kind of change or a specific domain and should not be assumed to generalize beyond them. findings highlight the need to include affective and connotational aspects of meaning in studies of lsc. in particular, future studies must consider emotional meaning in language models. while psychology has extensively used language to analyze emotion semantics @xcite , advances in nlp are still exploring how to build models that incorporate sentiment @xcite and detect emotion @xcite . further research is required to detect affective states from text given the cultural and universal aspects of emotion semantics @xcite . these findings also have implications for existing multidimensional frameworks of lsc @xcite as the evaluation framework provides experimental settings in which to compare the sensitivity of methods to detecting synthetic changes on specific dimensions and domains in a variety of disciplines. ## conclusion the current study introduced a novel generalpurpose evaluation framework, lsc-eval, for evaluating methods that detect lsc. its three-stage approach: 1) generates llm-based synthetic datasets with silver-standard labels that simulate changes in kinds of lsc; 2) uses these datasets to evaluate the relative sensitivity of computational methods in a synthetic change detection task; and 3) identifies the most suitable method for detecting synthetically induced changes across specific dimensions and domains. we applied this framework to a set of psychological terms. findings not only supported the validity of proposed computational methods for measuring changes in sib, but also established a controlled experimental standard for rigorously evaluating existing lsc detection methods and exploring alternative computational approaches. this work is crucial for addressing the substantial gap created by the lack of historical benchmark datasets, which has previously hindered the standardization of metrics and fair comparison of methods. while this framework benefits all disciplines (e.g., biomedicine, law, theology), lsc-eval is particularly valuable in the social sciences and humanities, where domain-specific constructs require specialized tools for capturing nuanced semantic change.