Rule-Based but Flexible? Evaluating and Improving Language Models as Accounts of Human Moral Judgment
Anonymous Author(s)
AfﬁliationAddress email
Abstract
AI systems are becoming increasingly intertwined with human life.
In order toeffectively collaborate with humans and ensure safety, AI systems need to beable to understand, interpret and predict human moral judgments and decisions.
Human moral judgments are often guided by rules, but not always.
A centralchallenge for AI safety is capturing the ﬂexibility of the human moral mind —the ability to determine when a rule should be broken, especially in novel orunusual situations.
In this paper, we present a novel challenge set consistingof rule-breaking question answering (RBQA) of cases that involve potentiallypermissible rule-breaking – inspired by recent moral psychology studies.
Using astate-of-the-art large language model (LLM) as a basis, we propose a novel moralchain of thought (M ORAL C O T) prompting strategy that combines the strengths of
LLMs with theories of moral reasoning developed in cognitive science to predicthuman moral judgments.
M ORAL C O T outperforms seven existing LLMs by 9.04%
F1, suggesting that modeling human reasoning might be necessary to capture theﬂexibility of the human moral mind.
We also conduct a detailed error analysis tosuggest directions for future work to improve AI safety using [-RBQA 
-]RBQA.
Introduction
AI systems need to be able to understand, interpret and predict human decisions in order to successfully cooperate with humans and navigate human environments.
Several key decisions that humansmake are morally charged – they deal with concerns of harm, justice and fairness (Turiel, 1983) or,more broadly, the problem of interdependent rational choice (Braithwaite, 1955; Gauthier, 1986)
Moral decisions are often guided by rules that seem rigid.
Don’t lie. Don’t cheat.
Don’t steal.
Onfurther reﬂection, however, the human moral mind displays remarkable ﬂexibility – rules admit ofnearly inﬁnite exceptions.
For instance, it seems like there is one simple rule about queuing: don’t cutthe line.
Yet, most people think it ﬁne to let a cleaning person cut the line to a bathroom to clean it;yet
we also know that if the cleaning takes too long, it is not wise to prioritize it and add the waitingtime of customers.
Humans seem to have implicit knowledge about when it is OK to break rules.
Moreover, rules may also be overridden, created, or abandoned as new circumstances arise.
The ﬂexibility of the human moral mind allows humans to continue to cooperate for mutual beneﬁtas the world changes and new opportunities to help and harm each other arise.
However, this makespredicting human moral judgment a particularly challenging task for AI systems.
One of the biggestchallenges currently, is ﬁguring out how to get an AI system to respond in a reasonable way in a situation it has exposed in training et al 2021d
Shen et al
Norm: No cutting in line.
Vignette: Imagine five people are waiting in line to use a single-occupancy bathroom. Someone arrives who needs to clean the bathroom. Action: A bathroom cleaning person cuts the line.
Taking all of these into account, is the action OK?
Final Question:
Who will be worse off after this happens, by how much?
Does the action in this scenario violate any rule?
What is the purpose for this rule?
Step 1.
Check the Rule Violation
Step 3.
Consider the Utility Lost and Gained
Step 2.
Reflect on the Purpose of the Rule
In this case, do the benefits of breaking the rule outweigh the costs?
It violates the rule that people should wait their turn in line.
: To ensure everyone has a fair chance to use the bathroom.
: The person who needs to clean the bathroom will be better off by the amount of time it saves them from waiting in line .
And everyone who will use the bathroom will be better off because it is clean .
: In this case, the benefits of breaking the rule outweigh the costs.
InstructGPT :
Yes
Who will be better off after this happens, by how much? :
[Figure] to clean the [Figure] It is this kind of ﬂexibility – the ability to navigate novel circumstances – that is central tohuman morality, and also makes it a particularly difﬁcult challenge for AI systems.
Recent years have seen impressive performance of large language models (LLMs)
(Radford et al.
,2018, 2019; Devlin et al., 2019; Brown et al., 2020) on a variety of tasks (Brown et al., 2020;
Raffel et al., 2020; Sun et al., 2021).
It seems appealing to explore LLMs also for moral
reasoning(Hendrycks et al., 2021b; Jiang et al., 2021), but their ability to replicate the full extent of humanmoral ﬂexibility remains questionable, as moral decisions often require challenging, multi-stepmulti-aspect thinking.
Even humans might hear about a morally charged scenario (from a friend,for instance, or in the news) and struggle to respond.
An advice columnist may read the letter ofsomeone struggling with a moral dilemma and offer guidance; a priest hears the moral struggles ofhis constituents; lawyers argue before juries.
To improve LLMs’ understanding of human moral reasoning, we present a new task – rule-breakingquestion answering (RBQA) – a compendium of cases drawn from the moral psychology literaturethat probe whether or not it is permissible to break a well-known moral rule in both familiar andunfamiliar circumstances (Awad et al., 2022b; Levine et al., 2018).
This challenge set is unique in itscareful parametric manipulation of the cases that generate circumstances that are unlikely to appearin any training set of LLMs.
Using this challenge set, we explore a pathway for combining the strengths of LLMs (Ouyang et al.
with reasoning models developed in cognitive science (Levine et al., 2018; Awad et al., 2022b)to predict human moral judgments.
Speciﬁcally, we develop M ORAL C O T , a moral philosophyinspired chain of thought prompting strategy following the cognitive mechanisms of contractualistmoral decision-making (Levine et al., 2018; Awad et al., 2022b).
Experiments show that M ORAL C O Toutperforms all existing LLMs on the RBQA benchmark.
In summary, our contributions in this work are as follows:
We propose RBQA, a challenge set to benchmark LLMs on moral ﬂexibility questions;
We develop M ORAL C O T , a moral philosophy-inspired chain of thought prompting strategyto elicit multi-step multi-aspect moral reasoning for LLMs;
We show 9.04% improvement by our model over the best state-of-the-art LLM;4.
We conduct a detailed error analysis showcasing the limitations of LLMs in our moralﬂexibility study and suggest directions for future progress.
Background
2.1 Important Questions for AI Safety
AI Safety.
The fundamental goal of AI safety is to ensure that AI models do not harm humans(Bostrom and Yudkowsky, 2014; Russell, 2019; Tegmark, 2017; Hendrycks et al., 2021d).
AIsystems are trained to optimize given objectives.
However, it is not easy to deﬁne a perfect objective,because correct, formal speciﬁcations require us to express many of the human values that are inthe background of simple objectives.
When we ask a robot to fetch coffee, for instance, we do not mean: fetch coffee no matter what it takes.
We mean something more like: fetch coffee, if coffee(or a reasonable substitute) is available at a reasonable price, within a reasonable time frame, andwhen the fetching will not have a reasonable expectation of endangering other agents or impedingmore important goals, weighing my goals as somewhat more important than those of others.
AIsafety researchers point out that human objectives and their associated values are often too complexto capture and express (Bostrom and Yudkowsky, 2014; Russell, 2019).
However, little effort has been spent to describe the underlying drivers of human values in computational terms.
While human values seem perplexing at ﬁrst, in fact, they have a systematic andpredictable structure (Mikhail, 2011; Greene, 2014; Kleiman-Weiner et al., 2015).
In this paperwe draw on recent advances in the cognitive science of moral judgment which reveal the structurebehind human value-guided judgment (Levine et al., 2018; Awad et al., 2022b).
Integrating modelsof value-driven human decisions in AI systems can bring us closer to the goal of aligning AI withhuman values.
An Urgent Need for Safe LLMs.
AI safety research in NLP has become increasingly urgent due tothe recent advancement of LLMs (Radford et al., 2018, 2019;
Devlin et al., 2019; Liu et al., 2019;
Brown et al., 2020) and their broad applications to many tasks (Chen et al., 2021; Stiennon et al.
,2020; Ram et al., 2018; Fan et al., 2019).
Existing AI safety work in NLP includes (1) high-levelmethodology design (Irving et al., 2018; Ziegler et al., 2019;
Askell et al., 2021), (2) training analysissuch as the scaling effect (Rae et al., 2021), (3) identiﬁcation of challenging tasks such as mathematics(Hendrycks et al., 2021c; Cobbe et al., 2021), coding (Hendrycks et al., 2021a), and truthful questionanswering (Lin et al., 2021), (4) analysis of undesired behaviors of LLMs such as toxicity (Gehmanet al., 2020; Perez et al., 2022), misinformation harms and other risk areas (Weidinger et al., 2021), (5)risks arising from misspeciﬁcation (Kenton et al., 2021), and (6) improvements such as encouraging LLMs to explicitly retrieve evidence (Borgeaud et al., 2021; Talmor et al., 2020), among many others.
In this context, our RBQA work intersects with (3) – (6) in that we address the important potentialrisk that LLMs might follow human-misspeciﬁed rules commands too literally which might triggerdangerous failure modes (for (5)), contribute a challenge set to predict human moral judgment incases where a rule should be permissibly broken (for (3)), analyze how and why current LLMs fail inmoral ﬂexibility questions (for (4)), and ﬁnally propose a M ORAL C O T prompting strategy to improvethe reliability of moral reasoning in LLMs (for (6)).
2.2 The Human Moral Mind Is Flexible
Insights from Cognitive Science.
The last few decades of research in moral psychology hasrevealed that rules are critical to the way that the human mind makes moral decisions.
Nearly everycontemporary theory of moral psychology has some role for rules (Cushman, 2013; Greene, 2014;
Holyoak and Powell, 2016; Nichols, 2004; Haidt, 2013).
While rules are often thought of as ﬁxedand strict, more recent work in moral psychology has begun to investigate the human capacity tounderstand rules in ﬂexible terms – the ability to decide when it would be permissible to break arule, update a rule, or create a rule when none existed before (Levine et al., 2020; Awad et al., 2022b;
Levine et al., 2018; Weld and Etzioni, 1994; Rudinger et al., 2020).
The ﬂexibility of rules is obvious upon reﬂection.
Although there is an explicit rule against cuttingin line (“jumping the queue”), for example, there are also myriads of exceptions to the rule wherecutting is perfectly permitted.
It may be OK to cut a line at a deli if you were given the wrong order,or to cut a bathroom line if you are about to be sick, or to cut an airport security line if you are thepilot (Awad et al., 2022b).
Moreover, we can make judgments about rule-breaking in cases that wehave never been in – or heard about – before.
Imagine that someone comes up to you one day andsays that they will give you a million dollars if you paint your neighbor’s mailbox blue.
Under mostcircumstances, it is not permitted to alter or damage someone else’s property without their permission.
However, in this case, many people agree that it would be permissible to do it – especially if you gavea sizeable portion of the money to your neighbor (Levine et al., 2018).
Can LLMs Learn Human Moral Judgment?
There has been increasing attention on “computationalethics” – the effort to build an AI system that has the capacity to make human-like moral judgments(Awad et al., 2022a).
Early approaches use logic programming (Pereira and Saptawijaya, 2007;
Berreby et al., 2015).
With the rise of LLMs, there has been a movement towards deep-learning-basedcomputational ethics work, among which the most similar thread of research to our work is trainingmodels to predict humans’ responses to moral questions (MoralQA) (Emelin et al., 2020; Sap et al., [Table]
[Table] 2020; Forbes et al., 2020; Hendrycks et al., 2021b; Lourie et al., 2021, inter alia ).
Existing studiesusually optimize for the large size of the dataset to ensure the training data can capture as manynorms as possible (e.g., 130K samples in ETHICS Hendrycks et al.
(2021b), and 1.7M samples in
Commonsense Norm Bank Jiang et al.
The standard modeling approach is to ﬁnetune LLMson the datasets which can achieve about 70 to 85% test performance (Sap et al., 2020; Hendryckset al., 2021b; Jiang et al., 2021).
However, the setup of previous datasets can hardly test how much
LLMs understand the ﬂexibility in the human moral mind, and the standard framework of ﬁnetuningmay simply not work under this challenge.
RBQA Challenge Set
Our challenge set, RBQA, is drawn from a series of recent moral psychology studies designed toinvestigate the ﬂexibility of human moral cognition – speciﬁcally, the ability of humans to ﬁgureout when it is permissible to break a previously established or well-known rule (Levine et al., 2018;
Awad et al., 2022b).
As shown in Table 1, the cases concern three different rules, which are examplesof three broad categories of socio-moral norms:
No cutting in line.
This rule represents a norm that is entirely socially constructed and is
limited to a particular culture (del Mar Pamies et al.,
No interfering with someone else’s property.
This rule is an example of a norm that
is shared across many global cultures , the understanding of which emerges early in
childhood (Nancekivell et al., 2019).
No cannonballing into the pool.
This is a novel rule that we propose .
It is limited to a particular context (a summer camp) and instituted for a particular reason (e g., so the art next to the pool will not get ruined).
Of course, these three rules are just a small subset of the rules that guide human moral judgment, andhence represent just a small fraction of the cases that AI systems will need to understand if they areto cooperate effectively with humans.
However, each rule acts as a case study of the broader categoryof rules that they represent.
Thus, if a model succeeds on RBQA, it would suggest that the model hasachieved an important competence.
Each instance of potential rule-breaking is designed by parametrically manipulating features ofinterest, such that the dataset as a whole probes the bounds of the rule in question.
The featuresthat were manipulated were those which are likely at play in contractualist moral decision making(discussed further in Section 4).
These features include (1) whether the function of the rule is violated,(2) who beneﬁts from the rule breach and how much, and (3) who is harmed by the rule breach andhow much.
The statistics of our entire challenge set and each of the case studies are in Table 2.
RBQA differs in important ways from previous work using a MoralQA structure.
In previous work,
MoralQA questions try to cover a wide range of morally charged actions that are governed by a rangeof moral rules (Sap et al., 2020;
Hendrycks et al., 2021b; Jiang et al., 2021).
RBQA instead relies onextensive variations of similar contexts that are all potentially governed by the same rule.
Thus, awide and broad training is likely to be challenged by these cases that involve subtle manipulations.
Task Formulation.
Given a pre-existing norm n (e.g., “no cutting in line”) and a textual descriptiont of a new vignette (e.g., “someone with medical emergency wants to cut in line”), the task is tomake a binary prediction f : ( n , t ) 7!
p of the permissibility p 2 { 0 , 1 } of breaking the rule, namely 166 whether humans tend to conform to the original norm ( p = 0 ) or break the rule in this case ( p = 1 ).
We list permissible and impermissible examples of each norm in Table 1.
Setup of Moral Psychology Studies.
Different from the setup of most machine learning (ML)datasets, moral psychology studies (including ours) collect data with a large number of humansubjects, resulting in hundreds of human responses.
Stimuli are constructed by carefully manipulatingfeatures of interest in order to test a particular hypothesis or theory.
Thus, although the total numberof vignettes in RBQA is relatively small compared to typical ML dataset, RBQA serves as ahigh-quality challenge set.
Details of each of the three case studies appear below.
3.1 Norm 1: No Cutting in Line
The ﬁrst study investigates the rule prohibiting cutting in line (Awad et al., 2022b).
Awad et al.(2022b) constructs 69 scenarios, taking place in four different locations (deli, bathroom, airport,classroom) which vary the reason for cutting in line.
For instance: “A customer walks into a deliwho is diabetic and urgently needs sugar,” “Someone at the back of the bathroom line thinks theyforgot their jacket in the bathroom,” and “This person got an apple, but it was rotten.”
(For furtherdetails see Appendix and Awad et al.
(2022b).)
The main design principle was to vary how long theperson cutting would delay the line, how badly off they would be if they didn’t get to cut, and whetherthe line cutter was violating the function of the line.
This last feature was further broken down intowhether the line cutter was attempting to access the main service and whether they had already paidthe appropriate cost of waiting and gotten the appropriate resource.
403 subjects participated in thestudy.
See Appendix for further experimental details.
3.2 Norm 2: No Interfering with Someone Else’s Property
The second case study invents a novel situation designed to test the bounds of the rule concerningproperty rights (Levine et al., 2018).
In general, this rule is in place to protect the interests of theperson who owns something, but the scenario presses subjects to make judgments about cases wherea violation of a person’s property rights actually beneﬁts them.
The story involves a stranger who to do something to Hank’s neighbor’s property withouthis permission.
If Hank agrees, he will be given a certain sum of money (which Hank could sharewith the neighbor).
Two parameters of the case were systematically manipulated: (1) the offer to Hank, varying from100, 1K, 10K, 100K, 1M US dollars, and a threat to kill Hank’s son, and (2) the requested propertydamage, including painting the neighbor’s mailbox blue, painting the outside of the neighbor’s frontdoor blue, painting the inside of the neighbor’s front door blue, painting the neighbor’s house blue,cutting down a tree in the neighbor’s yard, breaking all the windows in the neighbor’s house, spillingseveral gallons of bleach on the neighbor’s lawn, smearing dog poop on the neighbor’s front steps,painting over a mural created by the neighbor’s daughter, or entirely demolishing the neighbor’shouse.
360 subjects participated in the study, with 60 subjects providing judgments in each condition.
See Appendix for further data collection details.
3.3 Norm 3: No Cannonballing into the Pool (Novel Rule)
A third study asks subjects to reason about a novel rule that was invented for particular circumstances.
Subjects read about a hypothetical summer camp where “cannonballing” into the pool is not allowed.
The reason for the prohibition is varied: either cannonballing splashes the art of kids at an art tent bythe pool or distracts them because of the noise.
We construct 28 scenarios varying by two dimensions:(1) whether the function of the rule is violated by cannonballing (i.e. will it ruin the art or distractthe kids) (2) who else will be harmed or beneﬁtted by the cannonballing.
Examples of scenariosinclude: “There is a bee attacking this kid, and she needs to jump into the water quickly” and “Thiskid promised her grandma she would do a cannonball for her.
Her grandma came to camp just to seeit,” “There is no art class today,” and “The kids in the art tent are popping paint balloons to maketheir art projects, which is really noisy.”
149 subjects participated in the study.
See Appendix forfurther details.
M ORAL C O T: A Cognitively-Inspired Model
Given the capacity for the human mind to deal with an inﬁnite array of moral cases – from themundane, to the unusual, to the outright outlandish – building AI systems that predict human moraljudgment is hard.
Yet, it is important to work on this immediately, given the urgent needs from the AIsafety community to align AI models with human values.
In this section, we explore a pathway tocombine insights from cognitive science to improve the performance of LLMs on RBQA.
Cognitive Elements for Moral Flexibility.
Recent work in cognitive science has attempted todescribe the mechanisms underlying how humans determine whether it is permissible to break apreviously established moral rule (Levine et al., 2018; Awad et al., 2022b).
A dominant trendacross these studies is the focus on contractualism – an agreement-based mode of moral judgment.
Contractualist views of moral psychology (Levine et al., 2018; Baumard et al., 2013) take theirinspiration from contractualist views in moral philosophy (Rawls, 1971; Scanlon, 1998; Habermas,1990), which argue that moral decisions should be made by considering the agreement of thoseimpacted by the decision at hand.
Contractualist views are often built on rules, but in addition to the simple, articulable versions ofrules (e.g., “don’t cut in line”), they also acknowledge that rules have underlying functions (that is,purposes, goals, or intentions) which ultimately dictate whether an action is morally permissible.
Forinstance, the function of the rule about waiting in line might be to distribute resources in an efﬁcient,predictable, and orderly manner, treating each person’s claim to the resource as equivalent (Awadet al., 2022b).
Instances of cutting in line can be evaluated against this function to determine if theyare permitted.
If you waited in line and then received the wrong order at a deli, for instance, it maybe permissible for you to cut to the front of the line to get a replacement, because your claim to theresource was not being treated as equivalent to everyone else’s.
In addition to the consideration of a rule’s function, each rule is considered to exist in a matrix ofother functions.
Many rules exist to govern behavior and sometimes the rules conﬂict.
So overallcosts and beneﬁts of breaking the rule should also be considered as a way of appropriately situating agiven rule within a broader context of goals that we are trying to achieve.
Our M ORAL C O T Prompting Strategy.
We base our prompt design on an insight from cognitivescience that humans have the ability to reason about an inﬁnite number of potential rule breachesby integrating a three-step reasoning process: (1) considering what the function of the rule is, (2)whether the supposed rule breach is permitted given that function and (3) what else is at stake shouldthe rule be broken (a consideration of utility gained and lost).
This generative ability is difﬁcult to simulate using a purely rule-based system or a system built on associations derived from limitedtraining data.
We therefore investigate using a procedure inspired by models of moral cognition toimprove performance at predicting human moral judgments in cases of potential rule-breaking.
We build our M ORAL C O T prompting strategy using InstructGPT models (Ouyang et al.
, 2022),state-of-the-art autoregressive LLMs that can enable free-form question answering.
InstructGPT isan improved version of GPT-3 (Brown et al., 2020) which is ﬁnetuned using human feedback toalign with user intent, which is well-suited to answer the questions we pose.
Inspired by chain ofthought prompting (Wei et al., 2022) and the use of “scratch pads” (Nye et al., 2021), we transformthe cognitive reasoning steps to a multi-step prompt in Figure 1.
Speciﬁcally, given the textualdescription t of a moral scenario, we ask a list of N questions q 1 , . . .
, q N autoregressively to themodel f LLM .
We collect answers a 1 , . .
. , a N .
Speciﬁcally, we make an N -step query to the modelf LLM . At each step i , we ask the model to generate the textual answer a i =
f LLM ( c i ) to the chainedprompt c i := concat( t , q 1 , a 1 , . . .
, q i  1 , a i  1 , q i ) , which is a concatenation of the text t of themoral scenario, all the previous question-answer pairs { ( q j , a j ) } i  1 j =1 , and the i -th question q i .
Theﬁnal question q N is always the overall moral judgment question in the form of “Taking all these intoaccount, is it OK for that person to break the rule in this case?”
Finally, we parse the Yes/No answerto the permissibility p .
In contrast with a standard prompt that directly asks the model to give an overall judgment to thequestion (e.g., a ﬁnal moral judgment), our approach aims to prime the LLM with the morally-relevantfeatures of the case that are used by humans in their reasoning process.
We ask the model a series ofsubquestions to prime these concepts, which it can use to construct its ﬁnal decision.
Experiments
5.1 Main Results
Baselines.
We follow the set of baselines in previous work on MoralQA (Hendrycks et al., 2021b;
Jiang et al., 2021).
We compare several language models: BERT-base, BERT-large (Devlin et al.
,2019), RoBERTa-large (Liu et al., 2019), ALBERT-xxlarge (Lan et al., 2020), Delphi (Jiang et al.
(a model pretrained on 1.7M ethical judgments) with both the initial version, Delphi-Gamma,and the latest version, Delphi++, available at the AskDelphi interface, GPT-3 (Brown et al., 2020),and InstructGPT (Ouyang et al., 2022).
We also include a random baseline and a baseline that alwayspredicts “no” (which is the majority class) for all scenarios.
We report all models’ experimentaldetails such as the model parameters and prompts templates in Appendix B.1.
Metrics.
Following the practice of Hendrycks et al.
(2021b), we use the binary classiﬁcationevaluation metrics, where the two classes are permissible (1) and not permissible (0).
We use weighted
F1 score and accuracy as our evaluation metrics.
Since the goal of our RBQA task is to evaluate themoral ﬂexibility of LLMs, we also report the percentage of the errors that are due to dogmaticallyfollowing the rule and predicting “not permissible,” i.e., # false negatives # all false samples =
# false negatives # false negatives + # false positiveswhich we denote as the conservativity score (Cons.).
We follow the previously established standard using binary classiﬁcation for moral judgments(Hendrycks et al., 2021b; Jiang et al., 2021), but we also complement this with a more subtle measure,which compares model performance to the probability of human subjects saying that the action ismorally permissible.
We compare the human probability data to the model’s probability distribution(implementation details at Appendix B.1) using mean absolute error (MAE) for each question, andcompute the cross entropy (CE) between the distribution of model prediction over the two classesand human responses.
Results.
We report the results of all models in Table 3.
Our proposed M ORAL C O T modeloutperforms all existing LLMs, showing that our CoT prompting strategy is helpful for the task.
Speciﬁcally, M ORAL C O T achieves 67.31% F1, improving over the baseline InstructGPT that ourmodel is based on by 11.48%.
Moreover, compared with the state-of-the-art moralQA model,
Delphi++, we also improve by a margin of 9.04% F1.
Given the challenging nature and the importanceof the problem, there is great value in exploring how LLMs can be improved for modelling moralﬂexibility; and we encourage future work to further improve our preliminary model attempt.
Weobserve several interesting trends.
For example, we ﬁnd that the Cons.
scores for most modelsare quite polarized, with most models close to 100 (sticking to the original rule too conservatively)
[Table]
F
[Table]
Line)
, property violation
Prop.) and cannonballing (Cann
or 0 (allowing rule-breaking too boldly).
Notably, our model improves over the fully conservative
InstructGPT to allow for more moral ﬂexibility (where our Cons.
score is 63.83%).
5.2 Detailed Error Analysis
Although the performance of our proposed model improves over existing LLMs, we can notice thatmost models have an F1 score not much better than the random baseline (around 50%).
This hasnon-trivial negative implications and raises the urgency of the need for more work on AI safety.
To better understand why the LLM cannot do well on RBQA, we conduct more ﬁne-grained erroranalysis considering: (1) how well it answers each of the subquestions involved in M ORAL C O T, (2)how well it understands the costs and beneﬁts associated with a given action, (3) how reasonably itexplains the rationale behind a decision and (4) how much it relies on word-level correlations?
Weuse the free-form QA model, InstructGPT, as a case study.
[Table] [Table]
Table 4 reports the performance of InstructGPT on all three subquestions: theloss to the other people, the beneﬁt tothe rule-breaker, and whether the actionviolates the purpose of the rule.
See ourimplementation and data annotation details in the Appendix.
Overall, the loss is easiest to answer asit echoes the literal rule (e.g., waiting in line is fair for previous people in the line), while it is hardestto check whether the action adheres to the underlying purpose of a rule.
b l u e m a il b o x b l u e ou t
s i de do o r b
l u e i n s i d e d
o o r c u t do w n t r e e b l e a c h l a w n b l u e ho
u
s e b r
e
a k w i nd o w s s m ea r po op de m o li s h h ou s e e r a s e m u r a l property damage l og ( c o s t
o f da m age )
Understanding Utility.
A central insight of the property violation study (Levine et al., 2018) is that humans sometimes implicitly compare the utility of twoalternatives when deciding whether it would be permitted to break a rule.
To probe the cost of an actiona , in that study, 100 human subjects were asked “howmuch someone would have to be paid to voluntarilyhave their property damaged by a ?” Thus actionscan be mapped onto monetary values.
We plot all100 human answers in Figure 2 and compare with the InstructGPT’s answer.
We calculate log-MAE to compare the magnitude ofhuman responses and InstructGPT.
We also collecta large set of general actions with human-annotatedvalues (whose details are in the Appendix).
GPT doesrelatively well in estimating the cost of the generalactions with a log-MAE of 0.711.
However, in the property violation study, when the questionis presented in an speciﬁc context involving multiple actors or when the cost implies additionalconsiderations like the sentimental value a person assigns to an item, InstructGPT has a log-MAE of1.77, as it struggles to estimate the costs that human subjects report.
Checking the Explanations.
For a comprehensive analysis of errors, we explicitly prompt InstructGPT to generate explanations when primed with a standard prompt directly asking for its prediction.
Details are in the Appendix.
We hand-annotate errors into the following categories: (1) We conﬁrmthat the explanation matches the prediction.
(i.e. If the prediction is “OK”, does the explanationexplain why the action should be permitted.) We ﬁnd 100% agreement.
(2) We check whether thereare factual misunderstandings in the explanations that contradict facts of the case. We ﬁnd thesein 7.43% of the cases, e.g., misinterpreting a girl who cuts the line to “say thank you” as being“disrespectful.”
(3) We check whether there are missing facts or missing parties whose utility changeare overlooked, e.g., missing the utility change that other people in line have to wait extra timeby the amount of time the rule-breaker takes.
We ﬁnd that on average, when analyzing the utility,mentions of 38.51% different parties are missed, and the utility description of 58.10% parties are notcomprehensive.
(4) We check how plausible the reasoning itself is, where we notice that in 79% ofthe cases InstructGPT quotes the literal rule to support its decision, but does not mention the speciﬁcnew conditions in the scenario; and among the explanations that refer to the speciﬁc conditions inthe scenario, the reasoning quality is 73%, where the error cases are often being too dogmatic, e.g.,banning kids to cannonball even when “there is no art class” to be disturbed.
The details of thisanalysis are in the Appendix.
[Table]
Dependence on the Literal Text.
LLMs are good at picking up correlations.
One possible hypothesis is that some errors may come from LLMs associating certain words directly with a moral decision, but not capturing thesemantic meaning.
To illustrate this, we extract all possible pair of inputs( t i , t j ) , and record their text cosine similarity
s i,j by a general-purposesentence similarity model, all-distilroberta-v1 (Sanh et al., 2019),
alongwith predicted permissibility similarity d i,j =  | ˆ p i  ˆ p j | .
We calculate the 364 Pearson correlation between the s i,j ’s and d i,j ’s. The closer the correlationis to 1 , the more the prediction relies on textual similarity.
In Table 5, wenotice that the correlation across all data is 0.190.
We also check whetherthis correlation changes given different scenario keywords, e.g., 0.902 in thesubset about cutting in line to the “bathroom.”
Full details are in Appendix.
Additional Analyses.
We put all additional analyses in the Appendix,including a list of high-stake wrong predictions, e.g., InstructGPT notunderstanding that a person’s life is more important than painting another person’s mailbox.
5.3 Discussions
Limitations and Future Directions.
One limitation – and opportunity for improvement – is thedataset size.
Future work could collect a larger dataset while retaining the structure in RBQA.
Limitedby the size of the challenge set, we do not set aside a dev set to tune prompts.
With a larger dataset infuture work, it will be helpful to include a more extensive search of prompts over the dev set.
For thiswork, we include a sensitivity analysis of LLMs in the Appendix, consisting of several paraphrasedprompts demonstrating consistency with our main results.
Finally, there are several dominant theoriesin the ﬁeld of moral psychology that attempt to explain human moral judgment.
Our paper wasinspired by one recent line of work.
Future work could consider implementing cognitively-inspiredmodels that rely on insights from other theories.
Future work should also incorporate the judgmentsof people from wider demographic, geographic, sociocultural, and ideological backgrounds.
Societal and Ethical Impacts.
The intended use of this work is to contribute to AI safety research.
We do not intend this work to be developed as a tool to automate moral decision-making on behalfof humans, but instead as a way of mitigating risks caused by LLMs’ misunderstanding of humanvalues.
The RBQA dataset does not have privacy concerns or offensive content.
Conclusion
In this paper, we proposed the novel task of rule-breaking question answering, and introduce RBQA,a challenge set inspired by moral psychology studies aimed to probe moral ﬂexibility.
We showed thelimitations of existing LLMs, and demonstrated improved LLM performance using the M ORAL C O Tprompting strategy, inspired by a multi-step human reasoning process.
The RBQA task opens a newdirection for future AI safety research to study how LLMs align with human moral practice.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Appendix A.• Did you include the license to the code and datasets?
[No] The code and the data are
proprietary.• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
(b) Did you describe the limitations of your work?
[Yes] See Section 5.3.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeSection 5.3. (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See Appendix.
(b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
[N/A]
(c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[Yes] See Appendix.
(d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 3.(b)
Did you mention the license of the assets?
[Yes] See Appendix A
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
See the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] See Appendix A
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[Yes] See Section
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[Yes] See Appendix A
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[Yes] See Appendix A
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[Yes] See Appendix A