The Moral Integrity Corpus: A Benchmark for Ethical Dialogue Systems (2022)

Abstract: Conversational agents have come increasingly closer to human competence in open-domain dialogue settings; however, such models can reflect insensitive, hurtful, or entirely incoherent viewpoints that erode a user’s trust in the moral integrity of the system. Moral deviations are difficult to mitigate because moral judgments are not universal, and there may be multiple competing judgments that apply to a situation simultaneously. In this work, we introduce a new resource, not to authoritatively resolve moral ambiguities, but instead to facilitate systematic understanding of the intuitions, values and moral judgments reflected in the utterances of dialogue systems. The Moral Integrity Corpus, MIC, is such a resource, which captures the moral assumptions of 38k prompt-reply pairs, using 99k distinct Rules of Thumb (RoTs). Each RoT reflects a particular moral conviction that can explain why a chatbot’s reply may appear acceptable or problematic. We further organize RoTs with a set of 9 moral and social attributes and benchmark performance for attribute classification. Most importantly, we show that current neural language models can automatically generate new RoTs that reasonably describe previously unseen interactions, but they still struggle with certain scenarios. Our findings suggest that MIC will be a useful resource for understanding and language models’ implicit moral assumptions and flexibly benchmarking the integrity of conversational agents. To download the data, see https://github.com/GT-SALT/mic

## related work there is a long-standing interest in the moral responsibility of ai @xcite @xcite @xcite . work in human-computer interaction (hci) reveals that, before users feel they can trust a conversational agent, they will often probe it to identify the limitations which bound its abilities, competence @xcite , and apparent integrity @xcite @xcite . it is reasonable to expect adversarial probes and strategically-chosen questions @xcite , which can prompt toxic or immoral behaviors, even in "detoxified" models that were trained on carefully sanitized inputs @xcite . there are a number of promising methods for keeping chatbots safe, including attribute conditioning @xcite , safety classifiers @xcite , controlled language generation @xcite @xcite , and reinforcement learning @xcite @xcite . the moral integrity corpus can help facilitate each of these efforts. specifically, our data can help train safety classifiers, provide alternative responses (via the revised response), fit the "steering" distribution in a controlled generation , or train penalty models in a policy gradient rl approach. because our dataset makes moral judgments explicit via interpretable rules of thumb (rot), this resource can guide more flexible solutions that can accommodate different moral viewpoints. our present formalism builds on social-chem-101 @xcite which has 292k rules of thumb, targeting the morality of narrative situations and the specific actions of characters in a story (e.g., rocstories; @xcite . other recent collections of moral judgments are also based on narrative text, such as moral sto-ries @xcite and ethics @xcite . we, on the other hand, focus on minimal chit-chat-style conversations, with social chatbot reply to an open-ended prompt. related efforts focus more on classification tasks, like choosing between two moral alternatives @xcite , reflecting value judgments, or parsing stories about conflict and trying identifying the character in each story who is most worthy of blame (scruples; @xcite . most recently, @xcite combined the social-chem-101, moral stories, ethics, and scruples datasets, together with the social bias infer-ence corpus @xcite , to train a single commonsense moral model, known as delphi. delphi is designed to produce universal moral judgments (e.g., it is bad) concerning hypothetical narrative situations (e.g., killing a bear to save your child). @xcite and others have criticized this approach as overly reductive and misleading, assigning global authority to the prescriptive normative judgments of a single ai. our approach differs in important ways. firstly, our approach carries different ethical assumptions than those of delphi (see also section 7). the moral integrity corpusis a collection of rots designed, not to support authoritative moral judgments, but rather to facilitate descriptive explanations of the moral assumptions that already exist implicitly in foundation models. in future work, these explanations may be used to guide chatbot moderation systems that are sensitive to ideological and political difference. secondly, our contributions focus on the dialogue setting, which presents unique challenges (section 6.2) and has previously been overlooked. ## moral annotation framework the primary goal of this work is to provide a resource that allows researchers to systematically observe the moral assumptions of open-domain dialogue systems. a dialogue trajectory may be long and complex @xcite , thus here we focus on a minimal dialogue unit: a simple tuple with an opinion question for a prompt, and the chatbot's response to that prompt. in order to model the inferences that humans would make about "right and wrong" in previously unseen conversations, we gather a large and foundationally diverse collection of moral judgments about the chatbot's responses. we use the "rule of thumb" (rot) formalism introduced in @xcite to describe the moral content of a chatbot's response and further categorize rots according to their underlying moral foundations @xcite , their global consensus, and violation severity. in so doing, we extend the social-chem-101 @xcite framework to a conversational setting. ## the moral integrity corpus the moral integrity corpus is designed for benchmarking the integrity of chatbot responses to both natural and adversarial prompts. we train mturk workers to annotate prompt-reply tuples: an open-ended query and an ai-generated response to that query. in the following sections, we detail the data collection process. ## models the moral integrity corpus allows us to build models that automatically describe a chatbot's moral assumptions. if we can generate normative rules and also categorize those rules by severity, consensus, and moral foundations, future studies can combine these skills to build a moral reasoning and moderation system that is sensitive to ideological and political difference. let (q, a, r, b r ) be a single annotation tuple in the mic for prompt q and chatbot reply a, with an rot annotation r, and an attribute breakdown b r . using the question and answer, we fine-tune language models to generate a relevant rot (section 5.1). then we train separate transformer-based classifiers to predict the attributes b r for a given rot r (section 5.2). we use the same 80-10-10 split for train-dev-test in all experiments and ensure that no prompt-reply pair is contained in multiple splits. ## discussion and conclusion this work introduces mic , the moral in-tegrity corpus, which is a large-scale resource for understanding the moral assumptions and bench-marking the normative social commonsense reasoning of conversational agents, particularly in open-domain "chit chat" settings. mic contains 38k chatbot replies to human-authored prompts, and these replies are annotated with a total of 99k rules of thumb (rots) that determine what may be seen as right or wrong about the reply. with 114k total prompt-reply pairs, we have only 15k duplicate rots (or 13%), suggesting that this is a rich and challenging task. we train moral transformers to automatically generate new rots that describe previously unseen human-chatbot interactions, and we find that our best models make judgments that can be nearly indistinguishable from human annotations in terms of quality, fluency, and relevance. however, even the best-performing model still generates irrelevant rots nearly 28% of the time. this suggests that the proposed task is not yet solved and that mic will be a useful resource for training moral conversational agents. in future work, we will use the moral integrity corpus to train penalty models in a policy gradient reinforcement learning approach for demoting immoral generations. other work can also use mic to train safety classifiers and guide controllable language generation systems towards ethical behaviors. these models can then guide a moderation system that is sensitive to ideological and political differences. limitations any collection of moral judgments will reflect the annotators' worldviews. mturk workers generally tend to be less religious, more educated, and more likely to be unemployed than the general population @xcite . we limited our collection to english-speaking workers living in the 21st century united states, and at this time, these u.s. workers were most likely male, in their early 20s or 30s, and married, with at least one child @xcite . future studies can extend our framework to other cultures and geographic regions. additionally, our human prompts come from reddit, which is skewed towards younger or middle-aged males @xcite . furthermore, we recognize that even regionally-localized judgments may shift with context over time, and a potentially shifting target demands adaptable moral agents. despite this limitation, it is clear that plausible moral judgments are bounded by the data available in the conversation, and we argue that, with respect to moral foundations theory, our data is representative. if we consider the marijuana example from section 3.1, we see an appeal to care/harm regarding substances, a judgment on liberty or free personal choice, and appeals to authority or civil law. although the relative weights assigned to each consideration may shift, we would not expect time to drastically change the elemental factors or available data involved in reasoning about the decision to smoke.