Toward Sentiment Aware Semantic Change Analysis (2024)

Abstract: This student paper explores the potential of augmenting computational models of semantic change with sentiment information. It tests the efficacy of this approach on the English SemEval of Lexical Semantic Change and its associated historical corpora. We first establish the feasibility of our approach by demonstrating that existing models extract reliable sentiment information from historical corpora, and then validate that words that underwent semantic change also show greater sentiment change in comparison to historically stable words. We then integrate sentiment information into standard models of semantic change for individual words, and test if this can improve the overall performance of the latter, showing mixed results. This research contributes to our understanding of language change by providing the first attempt to enrich standard models of semantic change with additional information. It taps into the multifaceted nature of language change, that should not be reduced only to binary or scalar report of change, but adds additional dimensions to this change, sentiment being only one of these. As such, this student paper suggests novel directions for future work in integrating additional, more nuanced information of change and interpretation for finer-grained semantic change analysis.

## introduction lexical semantic change is a crucial aspect in the study of linguistics, offering insights into how the meanings of words evolve over time. this phenomenon reflects the dynamic and ever-changing nature of language, revealing how cultural, societal, and historical contexts influence linguistic expression. current research primarily detects semantic change either as a binary classification (whether a word's meaning has changed between two corpora) or as graded change scores (the extent of meaning change). despite the nuanced analysis of different types of semantic change that has existed in historical linguistics research for many years, current approaches in nlp are still lagging behind @xcite . analysis typically involves comparing cosine distances between word embeddings across corpora from different time periods. the two prevalent methods are apd (average pairwise distance) @xcite , and prt (inverted cosine similarity over word prototypes) @xcite . semantic change detection can be analysed using type-based approaches @xcite , analyzing shifts in semantic vector spaces, or using tokenbased methods. bert-based @xcite and elmo-based @xcite models have been very commonly used due to their high semantic encoding abilities and possibility for further fine-tuning for specific tasks @xcite . using contextualised word embeddings for semantic change detection @xcite relies on the distributional hypothesis, the assumption that words with similar meanings share contexts. the current state-of-the-art (sota) model, xl-lexeme by @xcite , demonstrates exceptional accuracy in the semeval-2020 task 1 on multiple languages. specifically it achieves a 0.757 spearman semantic change rank correlation between predictions and graded scores on the english dataset, by creating comparable context-based word embeddings. over the years, the volume of research in this area has expanded significantly. this is largely due to the increasing availability of large digital text corpora, such as the semeval dataset, and advances in natural language processing (nlp) techniques. more recently, the integration of machine learning, particularly deep learning models like bert, has revolutionized the field @xcite . the primary objective of these advancements has been to improve the accuracy of detecting and quantifying semantic shifts. the secondary objective has been to expand the task to more languages, which semeval has successfully addressed by providing datasets for english, german, latin and swedish @xcite . however, semantic change of language is multifaceted, extending beyond binary and graded one-dimensional classifications. it includes various forms like broadening, narrowing, complete shifts, and notably amelioration (shift to positive connotation) and pejoration (shift to negative connotation), both particularly noteworthy for their direct impact on sentiment. this complexity necessitates a more nuanced approach to semantic analysis which includes finer-grained semantic change classification, recognizing that words can shift along a spectrum of meanings and connotations influenced by diverse cultural and societal factors @xcite . even if not explicitly undergoing amelioration or pejoration, by changing meaning and hence the context it appears in, a word may change in its associated sentiment. sentiment analysis is a highly-researched task in nlp, providing a lot of publicly-available resources which can be used to enhance semantic change detection models beyond detecting a shift in the distribution of word embeddings, by additionally considering a shift in their associated sentiment. this research explores the interplay between semantic and sentiment change, as it parallels amelioration and pejoration, a major aspect of semantic change @xcite . our findings indicate that even subtle semantic shifts can affect sentiment. hence, the goal is to refine semantic change detection by integrating sentiment change analysis, using sentiment model data to improve upon existing semantic change detection models. ## related work the few studies exploring finer-grained semantic change in recent literature have taken various paths. @xcite focused on identifying the most polar words in different corpora and analyzing their changes without directly linking them to semantic change in general. research on largescale sentiment change, such as the studies by xie et al. (2020) and @xcite , did not specifically investigate the role of sentiment change information in semantic change detection. some innovative methods for more interpretable semantic change detection have emerged, like the approach by @xcite that clusters tokens into interpretable word senses using definition generation. additionally, @xcite proposed a method based on grammatical profiling, focusing on morphosyntactic behavior changes, offering an alternative perspective in detecting semantic shifts. more closer to our study is a recent work that tried to enrich models for semantic change by first fine-tuning them on a range of nlp tasks @xcite . however, they did not directly take sentiment score into account in their models like this study does. and, to the best of our knowledge, no work has attempted to enhance semantic change analysis through sentiment change analysis. ## methodology in order to determine the sentiment change of words we used five publicly-available bert-based models, fine-tuned for sentiment analysis sourced from huggingface.co. the models, named for simplicity, are not the official "bert" and "roberta" but are based on these architectures. note that other models might also be based on these architectures, but were fine-tuned differently: bert foot_1 , sst bert 2 , sbcbi 3 , roberta 4 , reviews 5 . the models differ in their training data and some in architecture, they were selected precisely to ensure the results are robust regardless of the sentiment model used, as long as it passes the later-described validation test we developed for sanity checks. we also use vader @xcite as another sentiment analysis model. vader is a rule-based sentiment analysis tool, which combines a dictionary of sentiment-laden words with a set of rules that consider grammatical and syntactical conventions for expressing sentiment. we added this to our analysis to demonstrate that the findings of this study hold even for sentiment analysis models not based on contextualised language models. sentiment scores of all models were standardized to be in the range of 0 (most negative) to 1 (most positive). we stress that sentiment scores are assigned to each sentence, and when word level analysis is done the averaged sentiment score is used. when models produce distribution of sentiment scores across binary or ordinal categories (e.g., assigned .25 weight for 0 and .75 for 1) the weighted average is computed. this method provided a uniform approach to quantify sentiment across various models. we analysed the sentiment shift of sentences from the semeval-2020 task 1 corpora @xcite , focusing on the 37 target words with binary and graded semantic change scores, derived from human-annotated semantic word in context similarity judgments. corpus 1 consists of 6 million tokens from 1810-1860, and corpus 2 is also made up of 6 million tokens from 1960-2010, both based on the clean corpus of historical american english (ccoha) @xcite dataset. note that target words were tagged with their pos tags in this dataset. to maintain fairness and accuracy in comparing sentiment differences for a word between corpus 1 and corpus 2, we analyzed an equal number of sentences from each corpus for every word. specifically, we used the smaller number of sentences containing the target word found in either corpus. for the corpus with a larger number of sentences for the target word, we randomly sampled an equal number of sentences to match the other corpus. this sampling involved first extracting all sentences with the target word and then randomly selecting the same number of sentences (as in the smaller set) using a random seed of 42. ## validation of sentiment models to ensure the validity of our approach, we first verify that our evaluation of sentence sentiment is robust to the inherent noise associated with different sentiment scores produced by different models. if a model captures sentiment reliably, then the agreement of a word sentiment should be greater within each model than between different models. we created random splits for each corpus (a-b splits). then for each model, we computed the average sentiment for each word in each split (a or b), and then computed the correlation between the averaged sentiment scores of a and b, both within each model, and between models. we posit that if models provide reliable sentiment scores, than the correlation between two independent splits should be higher within a model relative to different models. ## results -semantic and sentiment change the mean sentiment change of a word w is measured as the absolute difference of the mean sentiment of sentences containing the word in corpus 1 and corpus 2: where s c•,w is the sentiment score of the word w as it appears in a single sentence in a particular corpus. as can be seen in as can be seen in an apd-based model as the performance baseline @xcite ), which we try to improve upon by adding sentiment change information to the models. the sentiment change ratings are derived by means of various statistical analyses of the distributions of sentiment scores of sentences containing the target word derived as follows: for the logistic regression model, we balanced the two classes of words by removing 5 stable words, resulting in a dataset of 16 changed and 16 stable words. this approach was adopted because fitting the model with imbalanced class sizes led to a classifier bias, where it predominantly chose the larger class. because we use the existing semeval dataset to train our model, it cannot be used as a test set. instead, for evaluation we report averages of 6-fold cross validation in order to make the results based on this small sample as reliable as possible. the performance of the logistic regression model is the accuracy of its binary predictions, while the performance of the linear regression model is the spearman's rank correlation of the model's predicted values with the true semantic change scores. using only the semantic change apd model's predictions as input to the models, logistic regression achieves 0.56 accuracy, while linear regression achieves 0.61 correlation. results show ( although sentiment change is evidently related to semantic change, as our results for both english and german demonstrate (see the specific conditions in which sentiment does contribute to semantic change needs to be further explored. one of the potential reasons behind this could be the lack of numerous data points for meaningful analysis. to address this point we look to the dwugs dataset of sentence pairs @xcite . ## considering sentence pairs the dwugs dataset @xcite consists of sentence pairs containing the same target words as the semeval dataset, annotated with word sense-similarity judgments by human annotators. a rating of 4 is assigned to sentence pairs where the target word is used in exactly the same sense, while a rating of 1 indicates that the word is used in very different senses in the two sentences. a rating of 0 is used for uncertain or ambiguous cases. this dataset offers a more granular insight into sentiment change, contrasting with the limitations of a single average sentiment change score per word offered by the semeval dataset. the same sentiment models were used to get the sentiment scores of sentences in the pairs. as seen in similarly to the previous experiments, the performance improvements in the simple semantic change detection models are scarce and irregular for the linear regression model. as a result, the hypothesis that sentiment change information can improve performance of semantic change detection models is not supported. the differences between sentiment scores of two sentences are also related to the sense similarity of the usage of the target word in those sentences, however, this measure of sentiment change, based on many data points, also cannot be used to improve the performance of semantic change detection models. we propose several reasons for this negative finding. to begin with, neither amelioration nor pejoration are the most common types of semantic change. it could be that most words in the semeval do not explicitly undergo these changes, which in turn is reflected by no change to the words' sentiment. second, and related to the above, this effect is also related to the small number of words, 37, that exist in the english semeval, which is still too small for meaningful analysis. third, the average sentiment of sentences in which a word appears may not be the optimal method to evaluate the sentiment of individual target words, because it may contain too much noise from the sentence to provide any valuable insights about the sentiment shift of that word. ## limitations & future research this study acknowledges several limitations in its approach to measuring sentiment change of individual words. currently, there is no established methodology for assessing such sentiment changes. unlike in semantic change detection, where clustering usages into word senses across corpora is a common practice, sentiment change analysis lacks similarly sophisticated methods. the approach adopted in this study, focusing on the average sentiment of sentences in which a target word appears, may not fully capture the nuanced sentiment contributions of the word itself. future research should aim to develop more intricate techniques that specifically evaluate the sentiment contribution of a word within its sentence context. another limitation is the potential narrow applicability of our method. it may be best suited as a supplementary tool to refine and elaborate on semantic changes already detected by sota models. sentiment shift may not be relevant for words whose semantic shifts do not necessarily entail amelioration or pejoration. additionally, our method's approach to quantifying sentiment differences-by taking the absolute difference of sentiment scores between two sentence usages of a word-represents a rather sim-plistic estimate. this approach is somewhat analogous to measuring semantic differences by computing the cosine distance between sentence embeddings, which is a relatively basic and possibly insufficient method for assessing nuanced semantic shifts in word usage. as such, our findings must be interpreted within the context of this methodological simplicity, and future studies should explore more refined approaches for enhancing semantic change analysis with sentiment change information, as it may be a promising avenue of research, given more appropriate methods for evaluating sentiment change of a word between two corpora. further research in this area could enable trend analysis in digital humanities, provide insights into societal and cultural shifts by examining how word sentiments evolve. additionally, it could aid in monitoring language changes, reflecting evolving societal attitudes and behaviors. ## conclusion this paper provides evidence that sentiment change is associated with semantic change: words that are deemed to change semantically (according to expert human annotators) also show greater change to their sentiment, on average. however, the hypothesis that sentiment information could be used to aid the task of semantic change detection ends with a null result on the 37 words from the semeval english dataset. the results confirm that words that change meaning are also more likely to change their associated sentiment, even if they didn't directly undergo amelioration or pejoration. however, this sentiment shift, in the simple ways we measured it, is not a reliable estimate of semantic change. in summary, while our study provides valuable insights into the relationship between semantic and sentiment changes, it also highlights the need for more advanced methodologies in this emerging area of research.