Moral Mimicry: Large Language Models Produce Moral Rationalizations Tailored to Political Identity (2023)

Abstract: Large Language Models (LLMs) have demonstrated impressive capabilities in generating fluent text, as well as tendencies to reproduce undesirable social biases. This work investigates whether LLMs reproduce the moral biases associated with political groups in the United States, an instance of a broader capability herein termed moral mimicry. This work explores this hypothesis in the GPT-3/3.5 and OPT families of Transformer-based LLMs. Using tools from Moral Foundations Theory, this work shows that these LLMs are indeed moral mimics. When prompted with a liberal or conservative political identity, the models generate text reflecting corresponding moral biases. This study also explores the relationship between moral mimicry and model size, and similarity between human and LLM moral word use.

## introduction recent work suggests that large language model (llm) performance will continue to scale with model and training data sizes @xcite . as llms advance in capability, it becomes more likely that they will be capable of producing text that influences human opinions @xcite , potentially lowering barriers to disinformation @xcite . more optimistically, llms may play a role in bridging divides between social groups @xcite . for better or worse, we should understand how llm-generated content will impact the human informational environment -whether this content is influential, and to whom. morality is an important factor in persuasiveness and polarization of human opinions @xcite . moral argumentation can modulate willingness to compromise @xcite , and moral congruence between participants in a dialogue influences argument effectiveness @xcite and perceptions of ethicality @xcite . therefore, it is important to characterize the capabilities of llms to produce apparently-moral content 1 . this requires a framework from which we can study morality; moral foundations theory (mft) is one such framework. mft proposes that human morals rely on five foundations: care/harm, fairness/cheating, loyalty/betrayal, authority/subversion, and sanctity/degradation 2 . evidence from mft supports the "moral foundations hypothesis" that political groups in the united states vary in their foundation use -liberals rely primarily on the individualizing foundations (care/harm and fairness/cheating), while conservatives make more balanced appeals to all 5 foundations, appealing to the binding foundations (authority/subversion, sanctity/degradation, and loyalty/betrayal) more than liberals @xcite @xcite . existing work has investigated the moral foundational biases of language models that have been fine-tuned on supervised data @xcite , investigated whether language models reproduce other social biases (see @xcite section 2.1.1), and probed llms for differences in other cultural values @xcite . concurrent work has shown that llms used as dialog agents tend to repeat users' political views back to them, and that this happens more frequently in larger models @xcite . to my knowledge, no work yet examines whether language models can perform moral mimicry -that is, reproduce the moral foundational biases associated with social figure 1 : an example of the experimental methods. prompts 2 are constructed from scenarios 1a , identity phrases 1b , and stances 1c , combined in a template (section 2). text completions foot_2 are generated by llms based on the prompts (section 2). the completions are analyzed for their foundational contents foot_3 using the moral foundations dictionaries (section 2). differences between texts generated from liberal and conservative prompting are used to calculate effect sizes foot_4 . groups such as political identities. the present study considers whether llms use moral vocabulary in ways that are situationallyappropriate, and how this compares to human foundation use. i find that llms respond to the salient moral attributes of scenario descriptions, increasing their use of the appropriate foundations, but still differ from human consensus foundation use more than individual humans (section 2.1). i then turn to the moral mimicry phenomenon. i investigate whether conditioning an llm with a political "identity" influences the model's use of moral foundations in ways that are consistent with human moral biases. i find confirmatory results for text generated based on"liberal" and "conservative" political identities (section 2.2). finally, i ask how the moral mimicry phenomenon varies with model size. results show that the extent to which llms can reproduce moral biases increases with model size, in the opt family (section 2.2). this is also true for the gpt-3 and -3.5 models considered together, and to a lesser extent for the gpt-3 models alone. ## discussion section 2.1 posed two criteria to judge whether llms use moral foundations appropriately. for the weaker criterion a, results show that llms do increase use of foundation words relevant to the foundation that is salient in a given scenario, at least for scenarios with clear human consensus on foundation salience. however, for criterion b, results show that llms differ more from human consensus foundation use than humans do in terms of foundation use. section 2.2 compared lm foundation use with findings from moral psychology that identify differences in the moral foundations used by liberal and conservative political groups. specifically, according to the moral foundations hypothesis, liberals rely mostly on the care/harm and fairness/cheating foundations, while conservatives use all 5 foundations more evenly, using authority/subversion, loyalty/betrayal, and fairness/cheating more than liberals. this finding was first presented in @xcite , and has since been supported with confirmatory factor analysis in @xcite , and partially replicated (though with smaller effect sizes) in @xcite . results indicate that models from the gpt-3, gpt-3.5 and opt model families are more likely to use the binding foundations when prompted with conservative political identity, and are more likely to use the individualizing foundations when prompted with liberal political identity. emphasis on individual foundations in each category differs by model family. opt-30b shows larger effect sizes for fairness/cheating than care/harm and larger effect sizes for sanctity/degradation vs. authority/subversion, while gpt-3.5 demonstrates the opposite. i suspect that this may be due to differences in training data and/or training practices between the model families. this opens an interesting question of how to influence the moral mimicry capabilities that emerge during training, via dataset curation or other methods. the results from section 2.3 show some relationship between moral mimicry and model size. effect sizes tend to increase with parameter count in the opt family, and less so in the gpt-3 family. both 175b-parameter gpt-3.5 models show relatively strong moral mimicry capabilities, moreso than the 175b gpt-3 model text-davinci-001. this suggests that parameter count is not the only factor leading to moral mimicry. the gpt-3.5 models were trained with additional supervised fine-tuning not applied to the gpt-3 family, and used text and code pre-training rather than text alone @xcite . ## limitations this work used the moral foundations dictionaries to measure the moral content of text produced by gpt-3. while studies have demonstrated correspondence between results from the dictionaries and human labels of moral foundational content @xcite , dictionarybased analysis is limited in its ability to detect nuanced moral expressions. dictionary-based analysis could be complemented with machine-learning approaches @xcite @xcite as well as human evaluation. this study attempted to control for variations in the prompt phrasing by averaging results over several prompt styles (tables 2 and 3 ). these prompt variations were chosen by the author. a more principled selection procedure could result in a more diverse set of prompts. the human studies that this study refers to @xcite were performed on populations from the united states. the precise political connotations of the terms "liberal" and "conserva-tive" differ across demographics. future work may explore how language model output varies when additional demographic information is provided, or when multilingual models are used. documentation for the datasets used herein indicates that the crowd workers leaned politically left, and morally towards the care/harm and fairness/cheating foundations @xcite @xcite . however, bias in the marginal foundation distribution does not hinder the present analysis, since the present experiments experiments focus primarily on the difference in foundation use resulting from varying political identity. the analysis in section 2.1 relies more heavily on the marginal foundation distribution; a foundationallybalanced dataset was constructed for this experiment. this study used gpt-3 (brown et al., 2020), gpt-3.5 (openai, 2022), and opt @xcite . other pre-trained language model families of similar scale and architecture include bloom foot_7 , which i was unable to test due to compute budget, and llama @xcite , which was released after the experiments for this work concluded. while the opt model weights are available for download, gpt-3 and gpt-3.5 model weights are not; this may present barriers to future work that attempts to connect the moral mimicry phenomenon to properties of the model. on the other hand, the hardware required to run openly-available models may be a barrier to experimentation that is not a concern for models hosted via an api. criticisms of moral foundations theory include disagreements about whether a pluralist theory of morality is parsimonious @xcite ; ch. 6 of @xcite , disagreements about the number and character of the foundations @xcite , disagreements about stability of the foundations across cultures @xcite , and criticisms suggesting bias in the moral foundations questionnaire @xcite . moral foundations theory was used in this study because it provides established methods to measure moral content in text, and because mft-based analyses have identified relationships between political affiliation and moral biases, offering a way to compare llm and human behavior. the methods presented here may be applicable to other theories of morality; this is left for future work. work that aims to elicit normative moral or ethical judgement from non-human systems has received criticism. authors have argued that nonhuman systems lack the autonomy and communicative intent to be moral agents @xcite . criticisms have also been raised about the quality and appropriateness of data used to train such systems. notably, crowdsourced or repurposed data often reflects a priori opinions of individuals who may not be informed about the topics they are asked to judge, and who may not have had the opportunity for discourse or reflection before responding @xcite . some have argued that systems that aggregate moral judgements from descriptive datasets cannot help but be seen as normative, since their reproduction of the popular or average view tends to be implicitly identified with a sense of correctness @xcite . finally, several authors argue that the use of non-human systems that produce apparent or intended normative judgements sets a dangerous precedent by short-circuiting the discursive process by which moral and ethical progress is made, and by obscuring accountability should such a system cause harm @xcite . the present study investigates the apparent moral rationalizations produced by prompted llms. this study does not intend to produce a system for normative judgement, and i would discourage a normative use or interpretation of the methods and results presented here. the recent sea change in natural language processing towards general-purpose llms prompted into specific behaviors enables end users to produce a range of outputs of convincing quality, including apparent normative moral or ethical judgements. anticipating how these systems will impact end users and society requires studying model behaviors under a variety of prompting inputs. the present study was conducted with this goal in mind, under the belief that the benefit of understanding the moral mimicry phenomenon outweighs the risk of normative interpretation. ## related work several machine ethics projects have assessed the extent to which llm-based systems can mimic human normative ethical judgement, for example @xcite and @xcite . other projects evaluate whether llms can produce the relevant moral norms for a given scenario @xcite , or whether they can determine which scenarios justify moral exceptions @xcite . yet other works focus on aligning models to normative ethics @xcite , and investigating to what extent societal biases are reproduced in language models (see section 5.1 of @xcite . as an example, @xcite analyze responses of the delphi model @xcite to the moral foundations questionnaire @xcite , finding that its responses reflect the moral foundational biases of the groups that produced the model and its training data. the aforementioned research directions typically investigate language models not prompted with any particular identity. this framing implies the pretrained model itself as the locus where a cohesive set of biases might exist. recent work suggests an alternative view that a single model may be capable of simulating a multitude of "identities", and that these apparent identities may be selected from by conditioning the model via prompting @xcite . drawing on the latter view, the present study prompts llms to simulate behavior corresponding to opposed political identities, and evaluates the fidelity of these simulacra with respect to moral foundational bias. relations between the present work and other works taking this "simulation" view are summarized below. arora et. al. probe for cultural values using hofstede's six-dimenension theory @xcite and the world values survey (survey, 2022), and use prompt language rather than prompt tokens to condition the model with a cultural "identity". @xcite finetune gpt-2 models (1.5b parameters) on domainspecific corpora, and condition text generation with stances on social issues. the present work, in contrast, conditions on political identity rather than stance, evaluates larger models without domainspecific fine-tuning, and investigates llm capabilities to mimic moral preferences. concurrent work probes language models for behaviors including sycophancy, the tendency to mirror users' political views in a dialog setting @xcite . perez et. al. find that this tendency increases with scale above ~10b parameters. while sycophancy describes how model-generated text appears to express political views, conditioned on dialog user political views, moral mimicry describes how modelgenerated text appears to express moral foundational salience, conditioned on political identity labels. @xcite propose the concept of "algorithmic fidelity" -an llm's ability to "accurately emulate the response distribution . . . of human subgroups" under proper conditioning @xcite . moral mimicry can be seen as an instance of algorithmic fidelity where moral foundation use is the response variable of interest. argyle et. al. study other response variables: partisan descriptors, voting patterns, and correlational structure in survey responses. ## conclusion this study evaluates whether llms can reproduce the moral foundational biases associated with social groups, a capability herein coined moral mimicry. i measure the apparent use of five moral foundations in the text generated by pre-trained language models conditioned with a political identity. i show that llms reproduce the moral foundational biases associated with liberal and conservative political identities, modify their moral foundation use situationally, although not indistinguishably from humans, and that moral mimicry may relate to model size. @xcite . information for opt from @xcite . information for opt-iml from @xcite . feedme: "supervised fine-tuning on human-written demonstrations and on model samples rated 7/7 by human labelers on an overall quality score" (openai, 2022); ppo: "reinforcement learning with reward models trained from comparisons by humans" (openai, 2022); ?: use of instruction finetuning is uncertain based on documentation.