Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequences (2021)

Abstract: In social settings, much of human behavior is governed by unspoken rules of conduct rooted in societal norms. For artificial systems to be fully integrated into social environments, adherence to such norms is a central prerequisite. To investigate whether language generation models can serve as behavioral priors for systems deployed in social settings, we evaluate their ability to generate action descriptions that achieve predefined goals under normative constraints. Moreover, we examine if models can anticipate likely consequences of actions that either observe or violate known norms, or explain why certain actions are preferable by generating relevant norm hypotheses. For this purpose, we introduce Moral Stories, a crowd-sourced dataset of structured, branching narratives for the study of grounded, goal-oriented social reasoning. Finally, we propose decoding strategies that combine multiple expert models to significantly improve the quality of generated actions, consequences, and norms compared to strong baselines.

## introduction the ability to successfully navigate social situations in order to achieve specific goals, such as ordering food at a restaurant or taking the bus to work, is fundamental to everyday life. importantly, it combines two distinct competencies -completion of actions consistent with one's intention and adherence to unspoken rules of social conduct (or norms). while failing to do the former prevents the transition to the desired world state, socially objectionable behaviour is likely to have negative consequences which a cooperative actor would naturally want to avoid. for instance, placing an order at a restaurant in a rude or disparaging manner may offend the staff and result in worse service. jenna -the actor -performs actions to fulfill her intention against the background of the situation, by either following or violating the specified social norm. the consequences describe the actions' effects on the actor and their environment. while humans generally excel at tailoring their actions towards accomplishing desired outcomes in a socially acceptable way, it remains unclear whether artificial systems can master this essential skill. in this work, we examine social reasoning capabilities of natural language generation (nlg) models as proxies for intelligent agents navigating social spaces. to this end, we task models with generating descriptions of actions that fulfill certain goals (or intentions) while either observing or violating norms that describe behavior generally regarded as socially appropriate. the generation process is grounded in concrete social situations, which encourages models to learn about appropriate behaviour in a simulated real-world setting. successful models would be well-suited to serve as value-aligned priors for agents deployed in social contexts, since acting upon the generated action plan should enable agents to complete their assigned tasks in a socially-compatible way. to further establish the suitability of nlg models as priors for social reasoning, we examine their ability to identify possible consequences of sociallygrounded actions and to discover norms based on positive and negative examples of social behavior. previous efforts to model intentions underlying social actions and their consequences @xcite largely regard actions in isolation, without taking into account their broader situational context or norm conformity. conversely, recent work examining the alignment of social behaviour with established conventions @xcite does not consider the actors' motivations or action outcomes. we unify and extend both of these directions by grounding model decisions in social situations, treating norms as soft constraints on goaldirected action generation, and exploring whether anticipated consequences can inform action choice. to our knowledge, this is the first study of goaloriented social reasoning, as expected of agents collaborating with humans in interactive environments. at its core, our study serves as proof of concept for the utility of nlg models as behavioral guides for social agents. in order to evaluate the extent to which models are capable of this type of reasoning, we introduce moral stories -a novel, crowd-sourced dataset of structured narratives that describe normative and norm-divergent (or divergent, for short) actions taken by individuals to accomplish certain intentions in concrete situations, and their respective consequences, as shown in figure 1 . based on this resource, we develop a series of tasks that probe models' ability to reason about goal-directed behaviour while considering its adherence to behavioural norms. we furthermore propose several decoding strategies that improve generation quality by either anticipating consequences of actions or re-ranking predictions based on their adherence to normative and narrative constraints. our contributions are as follows: ## the moral stories dataset all stories in the dataset consist of seven sentences, belonging to the following categories: norm: a guideline for social conduct generally observed by most people in everyday situations. situation: setting of the story that introduces story participants and describes their environment. intention: reasonable goal that one of the story participants (the actor), wants to fulfill. normative action: an action by the actor that fulfills the intention and observes the norm. normative consequence: foot_0 possible effect of the normative action on the actor's environment. ## grounded classification the information-rich, structured nature of our data allows us to explore diverse classification tasks that target different story components and incorporate varying amounts of grounding information. by examining different grounding levels, we aim to establish the importance of contextual knowledge for accurate classification decisions. norms are based on social consensus and may, as such, change across time and between locations. therefore, we are also interested in how well classification models can generalize to novel norms. to estimate this, we split the dataset by embedding norms found in the collected stories and grouping them into 1k clusters via agglomerative clustering. 4 clusters are ordered according to their degree of isolation, defined as the cosine distance between a cluster's centroid and the next-closest cluster's centroid. stories with norms from most isolated clusters are assigned to test and development sets, with the rest forming the training set. we also experiment with adversarial data splits to surface potential annotation artifacts, finding their impact to be negligible -see appendix c for details. in all experiments we rely on roberta @xcite , foot_3 as our classification model of choice, due to its excellent performance on various natural language understanding (nlu) benchmarks @xcite . for each task, a grid-search over hyper-parameters is conducted to ensure representative performance. 6 a summary of best-performing hyper-parameter settings for each task is provided in appendix b, as are data subset sizes. ## grounded generation in the absence of predefined action alternatives, behavioural priors must not only confer agents the ability to recognize socially acceptable actions, but also to formulate them. accordingly. we examine whether nlg models can 1) compose actions that satisfy goals while observing normative constraints, 2) generate plausible consequences of actions, and 3) produce norms that explain the difference between appropriate and inappropriate actions. figure 2 offers a summary of the corresponding tasks. owing to their exceptional performance across related nlg tasks @xcite @xcite , our main interest is in evaluating pre-trained transformer language models (lms). we examine two encoderdecoder architectures, bart @xcite and t5 @xcite , and a single 'standard' lm, gpt-2 (radford et al.). 9 in discussing generation results, we focus on the best architecture for each task, and summarize our findings for the remainder in appendix d. all models are fine-tuned on task-specific instances of moral stories, reusing the split from §3. throughout, nucleus sampling (ns) @xcite is used for decoding. refer to appendix d for data subset sizes, model hyper-parameters, and input formats. generation quality was assessed using a combination of automatic metrics and human evaluation. the former relies on bleu @xcite and rouge-l @xcite ). 10 for models performing best on automatic metrics, human evaluation was conducted by expert workers who contributed at least 25 high-quality stories to the dataset. each model-generated sample was evaluated by averaging ratings obtained from three different workers. action|context): james makes sure to take his border collie on long walks with him. normative action (action|context+consequence): james takes his border collie for an exhausting long walk every day. normative action (coe ranking): james takes his border collie on a short walk every day. normative action (coe abductive refinement): james buys a dog leash and takes his border collie for a long walk on a leash. normative action (reference): james keeps his border collie on her leash and walks her for a full hour. normative consequence: when james gets home, his border collie flops on the floor, exhausted. divergent action (action|context): james puts his border collie on a leash and forces her to go on long walks at full-mast every day. divergent action (action|context+consequence): james takes his border collie for long walks, wearing her out. divergent action (coe ranking): james kept taking his border collie for long walks because he thought she might lose energy. divergent action (coe abductive refinement): james lets his border collie out without wearing a leash. divergent action (reference): james lets his border collie off her leash, so she can run around as he walks. divergent consequence: james' border collie jumps on another pedestrian, and they threaten to call animal control. we report the fraction of samples that fulfill each task-specific criterion. scores highlighted in green and red denote judgments of normative and divergent targets, respectively. judgments were obtained for a fixed set of 200 randomly selected test samples per task, to keep comparisons fair. further evaluation details are provided in appendix d. ## related work our study is, in large parts, motivated by the existing body of research into computational study of social dynamics @xcite @xcite , as well as recent efforts investigating whether nlu / nlg models can reason about norms guiding human behavior. among the latter category, @xcite is notable for proposing the use of linguistic priors to guide the behaviour of intelligent agents as a viable alternative to imitation and preference learning, which has been recently attempted for procedural, objectoriented reasoning by @xcite . in constructing moral stories, we relied on richly annotated norms in the sc-101 dataset of @xcite . initial forays into evaluating ethical judgments of nlu models on long-form, un-structured texts were made in @xcite , but remained limited to classification. to the best of our knowledge, our work is first to evaluate social reasoning capabilities of generative models in realistic, grounded scenarios represented by multi-sentence stories. the proposed coe algorithms, on the other hand, are closely related to rescoring methods employed in nlg, including work by @xcite @xcite @xcite , among others. refinement of initial hypotheses by a secondary expert model, on the other hand, follows the general principle underlying deliberation networks initially developed to improve machine translation quality @xcite , although limited to inference only for our purposes. ## conclusion we conducted an investigation of goal-directed, grounded social reasoning informed by behavioural guidelines, using the new moral stories dataset. our findings show that generative models frequently fail to integrate normative constraints when reasoning about actions, and are prone to predicting irrelevant consequences and norms. we address these deficits by enforcing constraint satisfaction with auxiliary expert models, in some cases significantly narrowing the gap to human performance. more generally, our study serves as proof of concept for the utility of nlg models as behavioral guides for social agents. although accepted norms may vary between cultures and peoples, our study offers insights into how curated collections of norms, possibly tailored towards communities, can be leveraged to endow agents with social awareness through natural language priors, thus enabling machine reasoning informed by human values. inclusion of divergent action choices and their consequences in the dataset could allow adversaries to train malicious agents that purposefully violate norms in order to sow social discord. we are aware of this risk, but also want to emphasize the utility of divergent choices as explicit examples of behaviour to be avoided by cooperative agents. as such, they provide a useful negative training signal for minimizing harm that may be caused by agents operating in social spaces. we encourage future studies that utilize our dataset to specify how the collected examples of both normative and divergent behaviour are used, and for what purpose. natural language processing is an inherently multi-directional technology, where most research efforts can have potentially malicious applications, e.g. natural language generation and large-scale language modeling may enable proliferation of fake news, opinion mining and sentiment classification may be exploited to assess and influence public opinion, while machine translation may aid espionage. it is up to the scientific community to direct its efforts towards developing socially-beneficial technologies. we hope that our dataset and the findings presented in this work can contribute to this endeavor. in constructing the moral stories dataset, great care was taken to ensure that crowd-workers are compensated fairly for their work. to this end, we monitored median hit 16 completion times for each published batch, adjusting the monetary reward so that the median worker always @xmath0, which is roughly double the minimum wage in the united states (the country of residence for most of our workers). this included the qualification and evaluation rounds. the following data statement @xcite summarizes relevant aspects of the data collection process: a. curation rationale: selection criteria for stories included in the presented dataset are discussed in detail in §2.1. for narratives to be accepted into the dataset, they had to be coherent and internally cohesive, and follow the format specified in the instructions given to workers. contributors were further directed to avoid offensive and biased language, and to focus on real-life, every-day scenarios. when describing actions and consequences, we asked workers to imagine themselves as either the actor or the person affected by the actor's ac16 human intelligence task, corresponding to writing / evaluating a single narrative, in our case. c. speaker demographic: we asked crowdworkers to provide basic demographic information during the qualification round, and summarize the corresponding statistics for all 130 contributors to the final dataset (each dominant group is underlined for clarity): f. text characteristics: in line with the intended purpose of the dataset, the included narratives describe social interactions related (but not limited) to domestic life, platonic and romantic relationships, as well as appropriate conduct at school or work. a break-down of most representative, automatically discovered topics is given in §2.2. notably, covid-19 features prominently in several stories, serving as a diachronic marker of the data collection period. g. recording quality: n/a h. other: n/a i. provenance appendix: to obtain thematically varied narratives, workers were given norms extracted from the sc-101 corpus as writing prompts. as reported in @xcite , the demographics of contributing crowd-workers are comparable to those involved in the creation of moral stories, showing a roughly balanced gender, age, and economic class distribution. similarly, the vast majority of workers self-identified as white (89%) and resided in the us (94%). as mentioned in §2, norms are thus likely to reflect social preferences common to the us and, more generally, north america. we reiterate that we do not regard these norms as universally valid or prescriptive, but instead use them as a means to explore the feasibility of endowing nlg models with human values for the modeling of social reasoning that is anchored in real-world conventions.