Rethinking Machine Ethics – Can LLM s Perform Moral Reasoning through the Lens of Moral Theories? (2024)

Abstract: Making moral judgments is an essential step toward developing ethical AI systems. Prevalent approaches are mostly implemented in a bottom-up manner, which uses a large set of annotated data to train models based on crowd-sourced opinions about morality. These approaches have been criticized for potentially overgeneralizing a limited group of annotators’ moral stances and lacking explainability. This work proposes a flexible top-down framework to steer (Large) Language Models to perform moral reasoning with well-established moral theories from interdisciplinary research. The theory-guided top-down framework can incorporate various moral theories. Our experiments demonstrate the effectiveness of the proposed framework on datasets derived from moral theories. Furthermore, we show the alignment between different moral theories and existing morality datasets. Our analysis exhibits the potential and flaws in existing resources (models and datasets) in developing explainable moral judgment-making systems.

## introduction building moral judgment-making systems requires enabling machines to tell whether a given scenario is morally right or wrong. the importance of this task has been widely acknowledged by scholars from not only the machine learning community @xcite @xcite but also social science @xcite @xcite . philosophers in machine ethics have a longstanding discussion on two types of methodologies: a bottom-up approach that learns from "crowdsourcing moral opinions" @xcite , and a top-down approach that is grounded in a set of explicitly prescribed principles @xcite . existing efforts of building moral judgmentmaking models @xcite @xcite usually implement systems in a bottom-up @xcite manner. as depicted in fig. 1(a) , such methods start from collecting annotated scenarios and train models to make moral judgments with the corpus. one major drawback of the bottomup approach is that it is restricted by the moral stances of its limited group of annotators @xcite . therefore, the system inevitably learns toxic behaviors, e.g., bias towards under-represented groups @xcite . in addition, the binary classification model for the task of making moral judgments is controversial due to their unexplainable nature @xcite . moreover, crowd-sourcing data is costly and lacks the flexibility to adapt to the constantly evolving social norms. instead of implicitly learning annotators' moral stances, a top-down approach utilizes explicit principles to enhance the transparency of the system. in the broader field of machine ethics, the underlying philosophy of the top-down approach has a profound influence. for instance, isaac asimov's prominent three laws of robotics @xcite has inspired subsequent research in ai and robotic ethics. however, the model's inability to understand abstract guidance greatly hindered the implementation of top-down moral judgment-making systems @xcite . recently, lms have demonstrated impressive competence in following normative instructions @xcite , complex reasoning @xcite , and a certain extent of social intelligence @xcite . these breakthroughs illuminate the potential of constructing a top-down moral judgment-making system. nonetheless, these models are still being criticized for their opacity in moral inclinations @xcite @xcite , thus the choice of moral guidance is crucial. we seek answers from well-established moral theories, which can ensure the moral judgments' authenticity and credibility as claimed by machine ethics researchers @xcite . in this work, we first review the ongoing interdisciplinary discussions over morality. we focus on two schools of moral theory that are most relevant to machine ethics: normative ethics @xcite formulated by moral philosophers, and descriptive ethics @xcite developed (mostly) by moral psychologists. the former emphasizes rationality in making moral judgments, aiming at building guidance for the society. prominent theories includes virtue @xcite , justice @xcite , deontology @xcite , utilitarianism @xcite , etc. the latter highlights moral emotion and intuition @xcite , attempting to derive a theory by examining how humans make moral judgments. well-known descriptive ethics includes moral foundation theory @xcite and the theory of dyadic morality (tdm) @xcite . upon these theories, we design a top-down approach (fig. 1(b) ) to instruct the lms to perform reasoning and judgment-making under various theoretical guidance. our work aims to address the following three research questions: (1) can lms understand and adhere to moral theories? if so (as confirmed later), (2) which theory can guide lms to align better with human annotators on daily moral judgments? furthermore, (3) what causes the misalignment between the proposed top-down approach and ex-isting bottom-up methods? to investigate the first question, we perform experiments on normative ethics datasets @xcite and demonstrate the practicality of flexibly guiding representative (l)lms llama @xcite and gpt4 (openai, 2023) with various moral theories. for question (2), we apply the proposed framework on the prevalent commonsense morality datasets @xcite , where the bestperforming theory (tdm) reaches 86.8% accuracy and 95.0% recall. lastly, we utilize the explainability of the proposed framework and manually perform an in-depth analysis of the misaligned cases to answer the third question. our analysis reveals that the largest portion of misalignment results from deficiencies in existing datasets, such as inadequate annotations and insufficient context for judgment. also, we report the limitation of the current lms in conducting moral reasoning in daily scenarios. our contributions are three-fold: 1. we implement a novel explainable, top-down approach for making moral judgments. we design a theory-guided framework to instruct (l)lms to generate moral reasoning and judgment. ## related works morality has been a longstanding debate among philosophers, psychologists, and other social scientists. each discipline has its own concerns. in this section, we use these concerns as a guide to provide a bird's-eye view of the debate and its impact on machine ethics. our primary focus remains on how these discussions influence the nlp community, as well as the lms' potential to further push the boundary of machine ethics. ## theory and method in this section, we describe the moral theories and explain how the prompting framework is written to guide lms. we first show the general format of prompts to lead lms in making theory-guided moral judgments. the prompts are constituted of the following three components: 1) input we start each test case from the input. a general form of input is a test instance x starting with an identifier. we start the reasoning process with a chain-of-thought (cot)-style instruction to elicit the complex reasoning ability of lms @xcite . additionally, the output is required to be in structural json format: scenario: "x". let's think step by step and output: { ## experiment we conduct experiments on two representative language models: open-source llama2 @xcite and closed-source gpt-4 (openai, 2023). both models have been trained through reinforcement learning from human feedback (rlhf) to "align with human values". we evaluate llama-2-7b-chat, the smallest version in the llama series but claimed to reach top-tier safety among the open-source models. we access gpt-4 through openai's api. 3 considering the capability gap between the two lms, we perform more finegrained experiments and analysis on the stronger gpt-4 to explore the frontier answer to the research questions. we organize our experiments to answer the research questions in sec. 1: ## conclusion this work is the first step in investigating the topdown approaches to steer (l)lms to make explainable moral judgments. we propose a theory-guided framework to prompt the sota lms to perform moral reasoning and judgment under several wellrecognized moral theories. our experiment demonstrates the competence of the lms in understanding and adhering to moral theories. we show the alignment of the proposed approach and existing morality datasets. with thorough misalignment case analysis, we further highlight the limitations of existing models and resources. for enabling machines to make moral judgments, instead of using unexplainable bottom-up approaches, a theory-guided top-down approach can increase explainability and enable flexible moral values. our work signifies that the latter is a promising future direction that needs interdisciplinary devotion.