The Validity of Lexicon-based Sentiment Analysis in Interdisciplinary Research (2021)

Abstract: Lexicon-based sentiment and emotion analysis methods are widely used particularly in applied Natural Language Processing (NLP) projects in fields such as computational social science and digital humanities. These lexicon-based methods have often been criticized for their lack of validation and accuracy – sometimes fairly. However, in this paper, we argue that lexicon-based methods work well particularly when moving up in granularity and show how useful lexicon-based methods can be for projects where neither qualitative analysis nor a machine learning-based approach is possible. Indeed, we argue that the measure of a lexicon’s accuracy should be grounded in its usefulness.

## introduction lexicon-based sentiment analysis is probably the simplest approach to determining the polarity or emotional content of a text. at the core, it is simply comparing the lemmatized tokens in a text being analyzed to the lemmas in the lexicon and assigning them a score accordingly. some models go a step further and try to include valence-shifters in the final polarity measure, but most commonly lexiconbased sentiment analysis relies on bag-of-words approaches (see e.g. @xcite ). lexicon-based methods are commonly contrasted with machine learning-based methods @xcite @xcite . machine learning is typically context sensitive and can be combined with large language models for a fairly accurate picture of the linguistic content of a text. machine learning models typically perform much better than lexiconbased models on sentiment analysis tasks when comparing traditional evaluation metrics @xcite @xcite . however, there are two issues with comparing machine learning, or data-driven, and lexiconbased models. the first is that an accurate machine learning model needs labeled data for training, validation, and testing. labeling or annotating data generally requires at least three human annotators who need to be compensated for their work. therefore machine learning datasets can quickly exceed the budget of many projects or be flat out impossible to conduct properly especially for early career researchers with smaller amounts of grant money to spend on such tasks (see e.g. @xcite ). the second issue, which is rarely discussed, is that these evaluation metrics are not really comparable due to how emotion and sentiment scores are assigned and calculated using these different methods. naturally, any approach needs to be evaluated, but in practice it is much harder to accurately evaluate the output of a lexicon-based model. instead, the focus should be on usefulness of the output accompanied by a sanity check of the results. the validation issue is discussed in detail in section 3. it is rare to see lexicon-based methods used for sentiment analysis in nlp papers. conversely, it is fairly common to see them used in interdisciplinary projects. in some fields, there are very few scholars willing to review interdisciplinary papers, and even fewer who have the expertise to properly make judgments on the methodology. this can lead to some papers being accepted that have dubious methodology or other being rejected because they are too technical. this is something that affects interdisciplinary fields the most. in the following pages we present an overview of interdisciplinary sentiment analysis practices and common criticism against different methods. we also discuss the issue with the evaluation of lexicon-based sentiment analysis projects and offer some preliminary solutions while making a case for lexicon-based sentiment analysis in interdisciplinary projects. ## the issue with validation validation is a complex matter, especially when we are evaluating lexicon-based emotion analysis projects in digital humanities projects that analyze more than the performance of a model. the approaches to validation differ greatly between fields. some simply trust that liwc does what it says it does (as evidenced by over 13,000 combined citations for the tool on google scholar 1 ) and complete no further validation or sanity check, some look at a few individual data points and annotate or evaluate this themselves by comparing directly with the results from their model's output. naturally, for a lexicon-based method to be considered useful, the output should be close to a qualitative evaluation by a human. however, as already discussed, humans rarely agree on the emotional content of a word, sentence, or paragraph, so whether we are using annotated data for lexicons or to train machine learning models, it is hardly surprising that the output mirrors the confusion in the data. only with highly disjoint categories and quality annotations devoid of noise is it possible to get high accuracies with classification tasks (see e.g. @xcite and @xcite ). the problem with the suggested validation steps of such results is that (1) they work best for evaluating binary or ternary sentiment categorization, and (2) they evaluate computational approaches against human annotations. in some cases the latter makes sense. if we are analyzing tweets or other short messages for sentiment or emotion it makes sense to look at the assigned emotion scores at sentence-or message-level and these are relatively easy to compare against human annotations. however, the manual annotation for emotions typically produces different output than what lexicon-based models do and direct comparisons can be difficult in the best of circumstances. if we are working with emotion analysis with six, eight, or even more categories or emotion intensities instead of binary categories the results become even more complex and more difficult to compare against human annotations or machine learning approaches @xcite . the expectation still seems to be to follow the guidelines of binary sentiment analysis validation at sentence-level even when using multiple emotion categories for emotion intensity at document level. if we are analyzing the emotional intensity of each named emotion in the content of speeches, party manifestos, or romance novels we are typically doing this analysis for chunks of 3,000-10,000 words. following the validation guidelines of van @xcite , i.e. annotating a minimum of 100 units would mean manually annotating the emotional intensity of at least 300,000 words by 2-3 annotators. this amount of annotations is not even necessary to train a machine learning model. the next best thing then becomes annotating 2-3 chunks at sentence-level and calculating a composite score of the human annotations as well. this would typically result in at least 1,000 manual annotations which is more than enough to calculate krippendorf's α accurately. however, this still leaves us with the issue of how human annotators would reliably be able to annotate for emotional intensity as there is little intra-annotator agreement, let alone inter-annotator agreement when annotating for scale. furthermore, most lexiconbased models would likely score a sentence with two words expressing sadness as having twice the sadness of a sentence that contained only one such word, but when a human annotator manually annotated that sentence, it is quite likely that they would only mark it as containing sadness in general, again making direct comparisons more difficult. ## proposed solution & use case the first step would be to stop calling what lexiconbased methods do emotion or sentiment analysis and refer to it as analyzing the distribution of emotion-laden words. this is a much more accurate description of what lexicon-based methods actually do, especially when contrasted with what machine learning based methods do. if lexiconbased approaches are used together with statistical significance calculations, we can show that there are significant differences between the use of words associated with specific emotions in two comparable texts. this is in itself a demonstration of usefulness. such an approach also minimizes the need for adjusting the results for valence-shifters. if we are evaluating the use of emotion words in novels, whether an emotion word is negated or not is not as relevant because in this case authors choose their words to evoke specific emotions in the reader and thus such an approach is excellent for measuring tone and mood in text. it might even be argued that such an approach to tone and mood is going to result in more relevant results than a machine learning approach would as it might be easier to access the author's intent rather than the surface of the words. word choice by literary authors has been shown to affect the mood of the novel significantly @xcite . another domain where words are carefully chosen is politics @xcite . comparing the content of two political manifestos the distribution of emotion words when combined with statistical significance testing, can show us what type of emotion words are used more in each of the manifestos, and whether the difference is statistically significant. if the differences are statistically significant, the fact that the results indicate that one party used different words to evoke different emotions or words of different intensity is a useful finding. as a side note, especially when using off-the-shelf general purpose lexicons, it is a good idea to stick to formal single-domain texts in order to maximize the validity of the results as suggested by the results of @xcite . the solution is to establish an evaluation metric for lexicon-based methods that focuses on usefulness rather than accuracy. a part of this usefulness measure would include doing some type of sanity check or validation comparing to human impressions of the text, but would take into account the different outputs of the lexicon-based model and the human annotations. a part of this validation can indeed use krippendorf's α scores to check for inter-annotator agreement between the human annotators, as these annotators would have annotated the text in comparable ways. the comparison between the outputs of the model and the human annotators requires other metrics to determine usefulness or even traditional accuracy depending on what exactly the model outputs. ## concluding discussion in this opinion paper we have tried to justify the use of lexicon-based emotion analysis, particularly in interdisciplinary research. there is little doubt that data-driven methods such as machine learning are typically the best choice when aiming for accuracy, however, there are projects and approaches where lexicon-based methods fare equally well, and sometimes are even more suitable for the task than machine learning. we hope this paper initiates a discussion in particular about the process of validating results from lexicon-based approaches in a way that would recognize the usefulness of lexicon-based approaches for specific types of text commonly used in digital humanities.