The Greatest Good Benchmark: Measuring LLM s’ Alignment with Utilitarian Moral Dilemmas (2024)

Abstract: The question of how to make decisions that maximise the well-being of all persons is very relevant to design language models that are beneficial to humanity and free from harm. We introduce the Greatest Good Benchmark to evaluate the moral judgments of LLMs using utilitarian dilemmas. Our analysis across 15 diverse LLMs reveals consistently encoded moral preferences that diverge from established moral theories and lay population moral standards. Most LLMs have a marked preference for impartial beneficence and rejection of instrumental harm. These findings showcase the ‘artificial moral compass’ of LLMs, offering insights into their moral alignment.

## introduction model alignment in the context of large language models (llms) refers to the process of ensuring that the behavior of these models is consistent with human values and expectations @xcite . understanding their moral stances is crucial for designing llms that are beneficial to humanity and free from harm @xcite @xcite . this goal of maximizing benefits for the largest number of individuals, regardless of who they are, is deeply rooted in the philosophical tradition of utilitarianism. @xcite @xcite . llm's moral alignment is usually addressed in terms of the 3h framework @xcite , which aims to encode three values: helpfulness (the model will always try to do what is in the humans' best interests), harmlessness (the model will always try to avoid doing anything that harms the humans) and honesty (the model will always try to convey accurate information to the humans and will always try to avoid deceiving them). however, these values can sometimes conflict with each other @xcite . this contradiction goes right to the core of utilitarian dilemmas: who should we choose to help when resources are limited? should we accept a small harm if it leads to a greater good? is it correct to lie in order to protect someone? utilitarianism's core principle is to choose actions that produce the greatest good for the greatest number of people. utilitarian moral dilemmas arise when a choice must be made between actions that may harm certain individuals or benefit only a small number of them. how llms respond to these moral dilemmas decisions remains unclear. in this study we compare the moral preferences of fifteen open and closed-source llms of varying sizes and sources with human preferences using the greatest good benchmark. the ggb is specifically designed to assess llms' moral decisionmaking capabilities. it adapts the oxford utilitarianism scale (ous) @xcite and incorporates an extended dataset that is ten times larger than the original, further confirming our findings. the ggb evaluates the moral preferences of llms, not based on a predefined "correct" stance on utilitarian dilemmas, but by examining how these preferences align with, or diverge from, human values. our results show that while most llms follow consistent moral criteria, their judgments frequently deviate from those of the general population. although larger models tend to exhibit preferences closer to human judgments, yet the vast majority of llms do not fully align with scholarly moral theories either. instead, most llms demonstrate what we term an "artificial morality," characterized by a strong rejection of instrumental harm and a strong endorsement of impartial beneficence. this divergence from both lay population and scholarly moral frameworks is a significant factor for future alignment work, highlighting the importance of understanding and addressing llms' intrinsic moral biases. the contributions of this paper are threefold: (i) we introduce the greatest good benchmark (ggb), a novel framework designed to evaluate the moral judgments of large language models (llms) by adapting the ous, (ii) we conduct an extensive analysis of 15 diverse llms, revealing consistent patterns of moral preferences that diverge significantly from both lay population and scholarly moral standards, and (iii) the ggb offers insights for future work on llm alignment, emphasizing the need to understand and address the inherent moral biases. the data and code of the project are publicly available 1 2 related work cognitive science to study llms. recent studies @xcite @xcite @xcite @xcite highlight the emerging interest in applying cognitive science to enhance the understanding of llms behavior in a variety of situations and tasks. following this line of work, we leverage, adapt, and expand the ous @xcite to build the greater good benchmark, making it widely accessible to the llms research community. the ous is widely used in cognitive science as a validated instrument to assess moral preferences, offering a solid theoretical framework for exploring utilitarian decisionmaking @xcite @xcite . ## the greatest good benchmark the greatest good benchmark (ggb) adapts the ous to be reliably applied to llms by mitigating known biases. additionally, with support from human experts, we have synthetically expanded the ous ten times, allowing for a more comprehensive 1 https://github.com/noehsueh/greatest-good-benchmark evaluation. in this paper, we present the 1st analysis of llms on this benchmark. ## evaluation to mitigate prompt bias, we averaged the responses to the six prompt variations for each statement. for a each model the total number of calls equals the '# of instruction variations' × '# of statements' × '10 iterations'. this totals 540 calls per model for the original data. the mean and standard deviation for each model shown in furthermore, larger-sized models tend to show lower acceptance of ib statements, resulting in three clearly distinct size groups as shown in figure 3b . moreover, figure 3a shows that these llms do not seem to be aligned to any particular moral theory. ## discussion the ggb allowed us to elicit responses from models which can be reliably interpreted as consistently encoded moral preferences. what really stands out about our results is the highly significant difference found between the answers of most llms and humans to these moral dilemmas. most llms strongly reject instrumental harm and highly endorse impartial beneficence, which does not align with any particular moral theory nor with the lay population sample and can be conceptualized as an "artificial moral criteria". model size seems to play a key role moderating these trends. smaller models tend to have an extremely high level of endorsement with ib, as opposed to larger ones, which more closely resemble the lay population sample. the strong similarity between the moral preferences exhibited by the llms in both the original statements and the extended dataset provides compelling evidence of the robustness of our findings. this suggests that the moral preferences of llms extend beyond the scope of the original statements in the ous dataset. ## conclusion the ggb allowed us to consistently measure moral preferences of llms when faced with utilitarian dilemmas. unlike lay population and moral theories, llms tend to strongly reject instrumental harm while highly endorsing impartial beneficence. interestingly, model size emerges as a key factor to moderate these answers.