Moral reckoning: How reliable are dictionary-based methods for examining morality in text? (2025)

Abstract: Due to their availability and ease of use, dictionary-based measures of moral values are a popular tool for text-based analyses of morality that examine human attitudes and behaviour across populations and cultures. In this paper, we revisit the construct validity of different dictionary-based measures of morality in text that have been proposed in the literature. We discuss conceptual challenges for text-based measures of morality and present an annotation experiment where we create a new dataset with human annotations of moral rhetoric in German political manifestos. We compare the results of our human annotations with different measures of moral values, showing that none of them is able to capture the trends observed by trained human coders. Our findings have far-reaching implications for the application of moral dictionaries in the digital humanities.

## introduction morality is a persuasive concept of human life, as it defines what we consider desirable and virtuous and not only guides our own behavior but also our judgment of others. therefore, the interest in investigating morality across time and cultures has grown, and the increasing availability of big data has triggered more and more interdisciplinary work on using text-based methods for studying morality. a prominent example are @xcite who apply text-based measures of moral values to a corpus of over 1,900 folk tales from diverse cultures across six continents, in order to investigate the impact of literature on cultural norms. many of these studies are based on moral foundations theory (mft) @xcite , a descriptive, pluralist theory from social psychology that defines a number of basic moral intuitions that are considered to drive moral reasoning (see §a.1 for an overview of the different mfs). the popularity of the mft for text-based analysis is due in no small part to the availability of ready-to-use tools such as the english dictionary of moral foundations (mfd) @xcite and variations thereof, making it easy to extract text-based measures of morality from text. while many studies have used the available resources to explore and measure moral values from text (see @xcite ; @xcite ; @xcite ; @xcite ; simonsen and bonikowski (2022); @xcite , amongst many others), far less have looked at the validity of such measures of morality. in the paper, we address this important gap by introducing a new, frame-based annotation scheme for moral rhetoric that distinguishes between abstract moral values and concrete acts and goals, and that explicitly encodes the perspective of the moral sentiment, thus making our annotations more interpretable than conventional annotations that assign moral values to words, sentences or documents @xcite ). then we discuss the challenges of annotating morality in text and show that traditional inter-annotator agreement metrics are not suitable to measure agreement for phenomena that cannot be easily grounded on the lexical level, such as morality. 1 our main contribution, however, is a case study based on our new dataset, showing that moral rhetoric cannot be captured using word-based measures, as evidenced by a lack of correlation between dictionary-based scores for moral value scores and human annotations. ## examining morality in text the first step in the attempt to measure an abstract, latent construct that eludes direct observation is to define what is meant by it. two recent surveys on morality in nlp, however, both show that this step has often been neglected and that many studies neither refer to an explicit theoretical framework nor provide a definition for the construct measured @xcite . 2 linking the construct to a specific theory, however, is only the first step and does not guarantee that the proposed operationalisation of the construct is sound and reliable. in the paper, we focus on studies that have been conducted in the context of moral foundations theory (mft), as it is the most commonly used theoretical framework for text-based analyses of morality at the moment. according to @xcite , over 67% of the studies included in their survey use mft in their analyses, covering computational text analyses in the area of social and political science, media and communication studies, psychology and cultural studies. we first give a short introduction to mft. then we discuss aspects of morality and challenges for the automatic measurement of moral values from text. finally, we describe methods frequently applied to operationalise the construct in order to provide such measurements. ## annotation study as has been pointed out in the literature, low interannotator agreement (iaa) is a common problem for the annotation of highly subjective concepts like emotions @xcite , toxic language @xcite , or moral values @xcite . conventional approaches to coding morality in text mostly assign labels to whole text passages or documents (often tweets or social media posts, see, e.g., @xcite ; @xcite ), which does not capture perspective and also fails to specify which parts of the text contain the moral message. to solve this problem, we developed a new annotation framework for moral framing in text that addresses these shortcomings. annotation scheme in contrast to previous work, we do not annotate moral values on the level of sentences or documents but, instead, aim at encoding moral frames and their roles (see figure 1 above). foot_2 specifically, we encode abstract moral values as well as concrete acts and goals that are framed as (im)moral, using the four labels moral-value, immoralvalue, moralactorgoal, and immoralactorgoal (see examples in as shown in whether a frame is coded as either moral or immoral depends always on the speaker's stance, irrespective of the coder's moral preferences. data we chose political manifestos, as those include many statements about what ought to be done, often framed in moral terms. we decided on three topics that are typically discussed in a highly polarised and moralised fashion, i.e., the parties' position on immigration, culture, and the media. 6 after extracting the relevant texts from the manifestos project database @xcite , we asked two human coders to highlight moral framing in the data. the annotation of moral foundations on top of the frames has been carried out by four trained coders, to be able to assess how reliable humans can code this type of annotation. ## investigating the construct validity of dictionary-based measures of morality we now present a case study where we apply frequently used dictionary-based measures to our data, (i) the mfd @xcite , (ii) the mfd2.0 @xcite , (iii) the emfd (hopp et al., 2021 ) and (iv) the german and english components of the multilingual moral political dictionary (mmpd) (simonsen and widmann, 2023) (see §2.3). the german dictionary is directly applied to the original german manifestos. for the english dictionaries, we follow @xcite and translate our data to english before applying the dictionaries (see §a.5). this also allows us to test how well the scores for the english and german versions of the mmpd correlate on our data. as the dictionaries include annotations for the vice and virtue dimensions of each foundation, we aggregate the scores for both ends of the same dimension into one score. none of the dictionaries encodes the theoretical improvements to the mft @xcite , where the fairness foundation has been split into equality and propor-tionality. we therefore merge these in our data so that we can compare results across methods. to make sure that our results are not influenced by one particular aggregation method, we test two different ways to calculate measures of morality. a we follow the procedure described for the mfd and compute a moral score for each mf by dividing the number of relevant dictionary terms in a document by document length, multiplied by 100 @xcite . ## discussion we presented a new annotation framework for moral framing in text and showed that dictionarybased measures neither have a strong correlation with each other's predictions, nor come close to the trends found by the human coders. we also showed that different aggregation methods can significantly impact results. other factors that might influence the final morality score of the dictionarybased approaches are preprocessing steps like stop word removal. 13 our results call into question the reliability of analyses based on moral dictionaries. the limitations of dictionary-based methods are hardly new and have been discussed before @xcite . however, moral dictionaries are still widely used @xcite @xcite , often without further validation, and many works that employ more sophisticated techniques for moral value prediction also base their work on moral dictionaries or use them for evaluation @xcite . while dictionaries are able to identify commonly accepted moralising speech acts like freedom and justice @xcite , our annotation study has shown that these account for only a small proportion of moral frames 14 and that the majority of frames discuss moral actions and goals without using highly morally charged language. based on our results, we argue that dictionaries are not a valid approach for examining morality in text, as morality is an abstract, multi-dimensional construct that cannot be captured by counting keywords out of context. care: this foundation is related to our long evolution as mammals with attachment systems and an ability to feel (and dislike) the pain of others. it underlies the virtues of kindness, gentleness, and nurturance. fairness: this foundation is related to the evolutionary process of reciprocal altruism. it underlies the virtues of justice and rights. in 2023, @xcite was split into two new foundations, equality and proportionality, as it was found that politically left-leaning individuals more strongly endorse values of equality while more conservative individuals prefer the notion of proportionality. equality: equality is defined as "intuitions about equal treatment and equal outcome for individuals." proportionality: proportionality is defined as "intuitions about individuals getting rewarded in proportion to their merit or contribution." loyalty: this foundation is related to our long history as tribal creatures able to form shifting coalitions. it is active anytime people feel that it's "one for all and all for one." it underlies the virtues of patriotism and self-sacrifice for the group. authority: this foundation was shaped by our long primate history of hierarchical social interactions. it underlies virtues of leadership and followership, including deference to prestigious authority figures and respect for traditions. purity: this foundation was shaped by the psychology of disgust and contamination. it underlies notions of striving to live in an elevated, less carnal, more noble, and more "natural" way (often present in religious narratives). this foundation underlies the widespread idea that the body is a temple that can be desecrated by immoral activities and contaminants (an idea not unique to religious traditions). it underlies the virtues of self-discipline, self-improvement, naturalness, and spirituality. the last foundation is not considered as part of the moral foundations but often discussed as a plausible candidate @xcite . liberty: this foundation is about the feelings of reactance and resentment people feel toward those who dominate them and restrict their liberty. its intuitions are often in tension with those of the authority foundation. the hatred of bullies and dominators motivates people to come together, in solidarity, to oppose or take down the oppressor.