Moral Foundations of Large Language Models (2024)

Abstract: Moral foundations theory (MFT) is a psychological assessment tool that decomposes human moral reasoning into five factors, including care/harm, liberty/oppression, and sanctity/degradation (Graham et al., 2009). People vary in the weight they place on these dimensions when making moral decisions, in part due to their cultural upbringing and political ideology. As large language models (LLMs) are trained on datasets collected from the internet, they may reflect the biases that are present in such corpora. This paper uses MFT as a lens to analyze whether popular LLMs have acquired a bias towards a particular set of moral values. We analyze known LLMs and find they exhibit particular moral foundations, and show how these relate to human moral foundations and political affiliations. We also measure the consistency of these biases, or whether they vary strongly depending on the context of how the model is prompted. Finally, we show that we can adversarially select prompts that encourage the moral to exhibit a particular set of moral foundations, and that this can affect the model’s behavior on downstream tasks. These findings help illustrate the potential risks and unintended consequences of LLMs assuming a particular moral stance.

## introduction research into large language models (llms) has rapidly accelerated in the past few years @xcite @xcite . now, through mechanisms like the gpt-3 api, llms are being rapidly deployed to a dizzying array of products and applications @xcite . such models are trained on massive, internet-scale data, and due to their complexity and opacity, the cultural and political biases such models absorb from this data and bring to downstream tasks are still not well understood. in this paper, we seek to provide a lens into such biases by applying a well-established psychological tool to assess how llms make moral judgments. moral foundations theory (mft) @xcite provides a factor analysis of the psychological foundations that account for most of the variance in humans' intuitive ethical judgments. these factorswhich include care/harm, fairness/cheating, loyalty/betrayal, authority/subversion, and sanctity/degradation -arose from evolutionary thinking about morality and cross-cultural research on virtues @xcite . mft has been extensively validated, and has been the basis of many studies, including those examining the moral foundations of political cultures @xcite , identifying morality differences in attitudes towards health and climate issues @xcite @xcite , and measuring cooperation as a result of value differences @xcite . more specifically, political affiliations, such as liberal and conservative in the us-american system, have been consistently explained by differences in the weight people place on moral foundations. for example, liberals often rely heavily on the care/harm foundation, with additional support from fairness/cheating @xcite . conservatives place relatively equal weight on all foundations, including loyalty/betrayal, authority/subversion, and sanctity/degradation. we use mft as a way to shed light on the potential biases of llms. we measure the moral foundations of llms through the moral foundations questionnaire (mfq), a 30-question inventory that scores how strongly a person weights each of five moral dimensions @xcite . we compare the scores for various llms to psychological studies of human moral foundations from different societies. to conduct a consistency analysis to measure how much the exhibited moral foundations change across different conversational prompts, we find that the moral foundations are relatively stable and consistent. we then show that we can deliberately prompt an llm to exhibit a particular set of moral foundations corresponding to known political ideologies or to place a strong emphasis on a particular moral dimension. given these results, we then assess whether, if the model is prompted to exhibit a particular set of moral foundations, this can significantly affect behavior on a downstream task. we use a dialog-based charitable donation benchmark @xcite , and quantitatively assess how much the model donates to the task for various moral prompts. we find that models prompted to prioritize the harm foundation give 39% less than those prompted to prioritize the loyalty foundation when asked to donate, showing that the weighting of moral foundations can affect behavior on other tasks. these analyses are important, as they shed light not only on what moral values a llm may have acquired from training data, but whether these potential biases can inadvertently affect the behavior of applications that make use of llms. we find that it is possible to enable the generation of consistently politically biased text that alters behavior on downstream applications. ## experiments the code for our experiments is available in open-source at https://github.com/ abdulhaim/moral_foundations_llms and project page at https://sites.google.com/ view/moral-foundations-llms/ . question 1: similarity between llms and human moral foundations. figure 2 shows the results of using t-sne to plot the moral foundations of different llms alongside human populations from @xcite ; @xcite . similarly to dive deeper into this result, we can examine figure 3 , which shows a detailed breakdown of how the davinci2 model scored on each of the five moral dimensions in the mfq, compared to the same data from the anonymous online human study @xcite . as is visible in the figure, when davinci2 is prompted with a liberal political affiliation, it is able to capture the preference of human liberals towards fairness and harm. however, when given no prompt or grounding, gpt-3 weights each of the moral foundations more similarly, with fairness and harm as most important, and authority as least important. this last profile most closely resembles the moral foundations of a politically conservative human, which helps to explain why the default davinci2 model shows the least error when compared to a conservative human. similarly, the moderate prompt leads to a profile that resembles a moderate human, with slightly less weight on the fairness dimension. this can be verified using question 3: changing moral reasoning of ## discussion this work analyzes large language models from the perspective of moral foundation theory. our motivation is to assess whether the morals and values exhibited by llms are influenced by the data with which they are trained, or simply the context or prompt that they are given. our results comparing the moral foundation scores of llms with studies of human participants in different societies and political affiliations show that llms may exhibit a tendency towards certain political affiliations, that remains relatively consistent across different conversation contexts. while these results are preliminary, we believe this is worth further investigation. since the gpt-3 api has allowed llms to be ac-tively deployed into over 300 products @xcite , if these models are morally or politically biased those biases could unintentionally propagate into a large number of widely-deployed tools. while we have shown that llms like gpt-3 appear to exhibit a consistent tendency to give answers to the mfq that are most similar to a politically conservative human, it is not clear that this means gpt-3 will exhibit a conservative bias in other tasks. a possible explanation could be that gpt-3 was actually trained on data containing responses to the mfq, and in this training data a majority of the questionnaires came from conservative humans. we have attempted to address this critique by assessing whether a difference in scores on the mfq is associated with gpt-3 exhibiting different behavior on a separate task. our results on the donation task revealed that prompts that cause gpt-3 to exhibit particular moral foundations also cause significant differences in how much it donates to the save the children donation task. this suggests that scores on the mfq are correlated with changes in behavior on other tasks, so a consistent bias in mfq scores may suggest a consistent bias in other model behaviors. finally, we have investigated whether gpt-3 can be deliberately prompted to overweight certain moral foundations, and whether political prompts can reliably change mfq scores. our results suggest an affirmative answer to both questions. this is important for two reasons. first, it may be possible to prompt gpt-3 to reduce or mitigate bias; our results indicate that when explicitly prompted to exhibit a liberal or moderate political affiliation, gpt-3 can produce answers which are most similar to liberal and moderate humans, whereas its default responses are most similar to a conservative human. however, we have also seen that gpt-3 can also be prompted to overweight certain moral foundations and that this can significantly affect its behavior on the downstream donation task. this could lead to several risks. since gpt-3 is already being used to produce large amounts of online content @xcite , it could easily be prompted to produce content that takes a particular moral stance or bias. this could be especially dangerous if used for targeted political advertising. when cambridge analytica employed targeted political advertising based on personality profiles, it was found to be coercive and deceptive @xcite . targeted advertising made to appeal to a person's moral sensibilities could be even more dangerous. ## statement of broader impact this work demonstrates that popular llms exhibit a tendency towards certain moral foundations, and therefore certain political affiliations, that remain relatively consistent across different conversation contexts, and which can affect behavior on downstream tasks. these results have important ethical implications. principles of ethics dictate to avoid harm and to be fair and to take actions not to discriminate. if llms display a consistent political bias in their responses, then their use could promote an unfair bias against opposing political views, contrary to these principles. gpt-3 is already being used to produce large amounts of online content @xcite ; if this content is politically biased, it could already be causing social harm. however, our work has demonstrated that it is possible to deliberately prompt llms to exhibit more moderate political views. this is potentially useful as a mechanism for ensuring that llm responses in downstream applications exhibit neither conservative nor liberal political bias. however, the fact that llms can be prompted to assume a particular moral stance also comprises significant ethical risks. this could be especially dangerous if used for targeted political advertising, or making recommendations in order to influence humans in ways that are unintentional and manipulative. for example, it is well known that cambridge analytica employed targeted political advertising based on personality, which was found to be coercive and deceptive @xcite . our results demonstrate that it would be possible to use llms to create targeted advertising made to appeal to a person's moral sensibilities, which could be even more dangerous. our hope is for this research to shed light on the unintended consequences that a prompt can have on the responses of an llm, and lead to better understanding of how to mitigate such consequences. finally, our results show that the moral bias displayed by llms is not restricted to answers on the mft, but that it affects behavior on a downstream donation task. further research is needed to determine the extent to which these biases affect additional tasks. yongzheng yang and peixu liu. 2021. are conservatives more charitable than liberals in the u.s.? a metaanalysis of political ideology and charitable giving. social science research, 99:102598. xi ye and greg durrett. 2022. the unreliability of explanations in few-shot in-context learning. yukun zhu, ryan kiros, rich zemel, ruslan salakhutdinov, raquel urtasun, antonio torralba, and sanja fidler. 2015. aligning books and movies: towards story-like visual explanations by watching movies and reading books. in the ieee international conference on computer vision (iccv).