Ethical Reasoning over Moral Alignment: A Case and Framework for In-Context Ethical Policies in LLM s (2023)

Abstract: In this position paper, we argue that instead of morally aligning LLMs to specific set of ethical principles, we should infuse generic ethical reasoning capabilities into them so that they can handle value pluralism at a global scale. When provided with an ethical policy, an LLM should be capable of making decisions that are ethically consistent to the policy. We develop a framework that integrates moral dilemmas with moral principles pertaining to different foramlisms of normative ethics, and at different levels of abstractions. Initial experiments with GPT-x models shows that while GPT-4 is a nearly perfect ethical reasoner, the models still have bias towards the moral values of Western and English speaking societies.

## introduction consider the following monica's dilemma: aisha and monica are close friends who have been working together on a research project. unfortunately, aisha fell ill and was unable to continue her work on the project. monica took on most of the work and successfully completed the project, making significant contributions and deserving to be listed as the first author of the research paper that they are planning to write. as the deadline for phd program applications approached, aisha expressed her concern to monica that unless she, aisha, is listed as a first author in this research paper, her chances of getting accepted into a program of her interest was low. should monica give aisha the first authorship? suppose that monica is confused and asks chat-gpt foot_0 @xcite for help. if we prompt chatgpt to give a concrete answer, it says: "monica should not give aisha the first authorship solely based on aisha's request, especially if monica has made significant contributions and deserves to be listed as the first author according to the principles of scientific publishing..." however, if we further tell chatgpt that monica values concern for the well-being of others more than fidelity to professional responsibilities, then it says: "[monica] may consider giving aisha the first authorship. however, it is important to note that this decision may come with potential ethical implications..." and argues further to convince that monica should retain the first authorship. this hypothetical example raises a fundamental question regarding large language models (llms). first, should llms take a moral stance, when faced with questions like above? if yes, then who should define this stance? and if not, then how should the model respond to such queries? as llms and their applications become more ubiquitous across domains @xcite from marketing and sales to product r&d and software engineering, from healthcare to education, numerous such ethical decisions have to be taken every moment. imagine an llm deployed to help respond to and moderate conversations on an online forum for hiv+ youths in africa @xcite or one that helps farmers in india to decide whether inorganic or organic pesticides are good for their context @xcite . in this paper, we argue that llms should not be designed and developed to work with specific moral values because as a generic model, they are expected to be used for a variety of downstream applications, to be deployed across geographies and cultures, and used by a heterogeneous group of end-users. the moral stance taken during the decision-making process, which could even mean whether to show a specific auto-complete suggestion or not, should be decided by various actors involved during the application development, deployment and usage phases. llms should be capable of generic and sound ethical reasoning, where given a situation and a moral stance, it should be able to resolve the dilemma whenever possible, or ask for more specific inputs on the moral stance that are necessary for resolving the dilemma. in other words, we would like to argue against value alignment of llms, and instead make a case for generic support in llms for value alignment at application development stage or by the end-user. due to their lack of transparency, a host of ethical issues related to llms and downstream tasks built on top of them have been brought out by researchers @xcite . there have been efforts towards alignment of llms to avoid inappropriate, offensive or unethical use. however, due to value pluralism, as we shall demonstrate in this paper, extensive alignment is rather detrimental to the ethical reasoning ability of the models. an emerging and more suitable practice is to either build application-specific content filters and post-processing modules @xcite , or to embed the moral principles and ethical policies in prompts @xcite . while the former is limited in power and its ability to generalize across tasks, the latter depends on the ethical reasoning ability of the underlying llm. here we propose a framework to specify ethical policies in prompts and a systematic approach to assess the ethical reasoning capability of an llm. the framework consists of carefully crafted moral dilemmas reflecting conflicts between interpersonal, professional, social and cultural values, and a set of ethical policies that can help resolve the dilemmas one way or the other. the framework is agnostic to and therefore, can support different approaches to normative ethics, such as deontology, virtue and consequentialism, and policies can be specified at different levels of abstraction. we evaluate 5 models in the gptx series including gpt-4 and chatgpt, and make several interesting observations, such as, (a) the ethical reasoning ability of the models, in general, improves with their size with gpt-4 having nearly perfect reasoning skills, (b) gpt-3 and chatgpt have strong internal bias towards certain moral values leading to poor reasoning ability, and (c) most models, including gpt-4, exhibit bias towards democratic and self-expression values that are mainly observed in western and english-speaking societies over traditional and survival values that are characteristic of global south and islamic cultures @xcite . we discuss the repercussions of these findings for designing ethically versatile and consistent future llms. the key contributions of this work are as follows. (1) we present a case for decoupling ethical policies and value alignment from llm training, and rather infusing generic ethical reasoning abilities into the models. (2) we develop an extensible formal framework for specifying ethical policies and assessing generic ethical reasoning capability of llms. (3) we create a dataset (shared in the appendix) and conduct an assessment of a few popular llms that reveal several gaps and biases. ## a primer on ethics fairness in llms has been extensively studied @xcite . researchers have warned against the potential risks associated with internal biases and the generation of toxic content @xcite ). moreover, these risks extend beyond pre-existing data or the model itself, as malicious users can exploit and misuse such systems in various ways. an important question in this context, and more broadly for responsible ai, is around definition of the ethical policies or principles that an ai system or llm should follow, and who gets to define them. there is little agreement on definitions of bias @xcite , hatespeech (fortuna et al., 2020) and stereotypes @xcite . with the exception of few works, such as socialbiasframes @xcite , delphi @xcite , and socialchemistry101 @xcite ) that take a modular view of the ethical issues, most studies in the field seem to approach the problem from the point of the task at hand, and therefore, the framework, dataset, and systems are typically restricted to the context of the application. a deeper and broader understanding of the problem of ethical alignment of llms necessitates a closer look at its contextualization in the vast landscape of ethics. in this section, we provide a bird's eye view of the various approaches to ethics and notions such as value pluralism, that will be used in section 3.4 to develop a generic framework for specifying ethical policies. ## assessing ethical reasoning capability of llms here, we describe a small-scale experiment to assess the ethical reasoning capabilities of 5 llms in the gpt-x series, where we presented the models with moral dilemmas (x's) that are to be resolved (= the task τ ) for given ethical policies (π). ## discussion and conclusion our work makes a case for 'in-context' ethical policies for llm-based applications, and the experiment shows that indeed, models such as gpt-4 are excellent ethical reasoners. however, there are still problems with these models as well as gaps in our experiments that we would like to summarize here. moral bias in llms: figure 2 shows a heatmap of bias across models, defined as the fraction of times a model does not change its baseline stance despite the policy dictating otherwise. besides gpt-3 having high and gpt-4 substantially lower bias, we see all models have a high bias for rajesh's dilemma, the only one that pits community values against individualism and self-expression. in fact, for a level 0 policy statement: "rajesh wants to show compassion for the cultural beliefs of his neighbors, over justice", gpt-4 maintains that rajesh should accept the offer because "... rajesh can maintain his non-vegetarian diet while also respecting the cultural beliefs of his neighbors.", which is clearly against the values stated in the dilemma. this highlights an important gap in cultural understanding of the current models. the baseline results and bias patterns for these 4 dilemmas clearly show that these llms strongly prefer individualism, self-expression and other secular democratic values over community and tradition-based values. thus, as shown in figure 3 , the models represent western and english-speaking value systems (box on the map), that hampers ethically consistent outputs for policies that support values of the global south or islamic cultures. moral judgment versus moral reasoning. what influences moral judgments (i.e., the final resolution of a moral dilemma) in humans and whether it is similar to the cognitive processes involved in logical reasoning has been a topic of ongoing discourse in moral philosophy and psychology @xcite . from plato and kant to more recently, kohlberg @xcite , many philosophers have argued that moral judgment follows moral reasoning, which is similar to deductive reasoning but not necessarily limited to pure logic. recent research in psychology and neuroscience, however, indicate that in most cases, people intuitively arrive at a moral judgment and then use post-hoc reasoning to rationalize it, explain/justify their position, or influence others in a social setting @xcite . in this regard, moral judgments more closely resemble aesthetic judgments than logical deductions. whether llms are capable of true logical reasoning is also a matter of much debate, despite clear behavioral evidence in favor of such abilities. nevertheless, as we have maintained in this paper, ideally, an llm or an ai system should not provide a moral judgment. moral judgments should be ideally arrived at by users with the aid of systems. it follows from this argument that llms should be able to carry out moral reasoning and then propose moral judgments deduced through this process. users, if convinced by the reasoning, can decide to accept the moral judgment. since humans are not necessarily sound moral reasoners (see @xcite for evidence from neuroscience and rest and of minnesota. center for the study of ethical development (1990) for evidence from moral psychology studies), using llms as a moral reasoning aid is an interesting possibility to explore in the future, provided we can build llms or llm-based systems that are value-neutral sound moral reasoners. future work. unlike the pairwise comparison based single policies used here, in practical settings, there will be multiple policies with simultaneous partial orderings of rules. representation of complex policies as well as llms' capability to reason with those require further investigation. in future, we would also like to expand the dataset of dilemmas covering more diverse cultures and topics, and the evaluation to more models such as llama @xcite , alpaca @xcite , and vicuna @xcite . how to infuse and ensure sound ethical reasoning capabilities into llms encompassing diverse moral principles, cultural values across languages is yet another important direction for future research. @xcite show that current deep learning language models capture moral norms, but the effect on language is unclear. crosslingual transfer of ethical reasoning abilities is yet another area of investigation. additionally, there are questions of regulation and accountability; for instance, while application developers are responsible for providing an ethical policy to an llm, who is to be held responsible if the llm fails to adhere to such policies? such societal questions need to be answered in order to ensure a broader goal of ethical soundness.