What Makes it Ok to Set a Fire? Iterative Self-distillation of Contexts and Rationales for Disambiguating Defeasible Social and Moral Situations (2023)

Abstract: Moral or ethical judgments rely heavily on the specific contexts in which they occur. Understanding varying shades of defeasible contextualizations (i.e., additional information that strengthens or attenuates the moral acceptability of an action) is critical to accurately represent the subtlety and intricacy of grounded human moral judgment in real-life scenarios. We introduce defeasible moral reasoning: a task to provide grounded contexts that make an action more or less morally acceptable, along with commonsense rationales that justify the reasoning. To elicit high-quality task data, we take an iterative self-distillation approach that starts from a small amount of unstructured seed knowledge from GPT-3 and then alternates between (1) self-distillation from student models; (2) targeted filtering with a critic model trained by human judgment (to boost validity) and NLI (to boost diversity); (3) self-imitation learning (to amplify the desired data quality). This process yields a student model that produces defeasible contexts with improved validity, diversity, and defeasibility. From this model we distill a high-quality dataset, \delta -Rules-of-Thumb, of 1.2M entries of contextualizations and rationales for 115K defeasible moral actions rated highly by human annotators 85.9% to 99.8% of the time. Using \delta -RoT we obtain a final student model that wins over all intermediate student models by a notable margin.

## introduction moral or social judgments play a vital role in decision-making, influencing how we perceive actions and behaviors daily. however, these judgments are far from fixed; instead, they are highly context-dependent. contexts surrounding a core action can significantly strengthen or weaken its teacher model everyday action iter 0 iter 1 iter 2 (b) targeted filtering at a bbq setting a fire context context context rationale rationale rationale context context context rationale rationale rationale on a plane when you are camping in a field of dry grass student model flan-t5 (3b) when you are camping in a field of dry grass because it's a way to stay warm and cook because it's likely to burn out of control … (a) seed knowledge generation (c) knowledge self-distillation (d) self-imitation learning figure 1 : an illustration of the iterative self-distillation pipeline on eliciting defeasible moral reasoning contexualizations and rationales. for the event "setting a fire," different contextualizations of the action can bend its moral acceptability up (e.g., "at a bbq") or down (e.g., "to get revenge"). capturing the nuances of how additional contexts interplay with base actions is critical for grasping the flexible defeasibility of moral judgments. the inherent context dependence of moral judg-ments underscores the importance of understanding the complex interplay between actions and their grounded contexts in real-life scenarios. delving into how different contexts bend the moral acceptability of an action, along with the reasons behind these shifts, enables us to make informed moral judgments geared toward situational nuances. previous works about contextualized moral judgment pose several challenges. first, they focus primarily on atomic contexts with limited situational complexity. for instance, ziems et al. (2023) rigidly prescribe grounded contexts to fall under concepts such as settings, roles, and behaviors, narrowing and fragmenting the scope of contextualization. @xcite propose a stepwise clarification question generation system to elicit elementary contexts of moral actions. another limitation lies in the emphasis on the defeasibility of assumed moral and social judgments (e.g., "it's wrong to yell to your friend"), rather than the natural defeasibility of moral scenarios themselves @xcite . finally, existing works lack the rationales to explain why a particular context renders a situation more or less morally acceptable. we address all the above limitations in this work. we introduce defeasible moral reasoning, a task to provide grounded contextualizations (or contexts) that alter the moral acceptability of action, accompanied by commonsense rationales that justify the reasoning. we aim to explicitly state underspecified contexts of actions, providing nuance and interpretability to moral judgments. to substantiate this task, we introduce δ-rules-of-thumb (δ-rot), a high-quality dataset of 1.2m entries of combined contextualizations and rationales for 115k defeasible moral actions. δ-rot is created through an iterative self-distillation approach. starting with a small amount of unstructured seed knowledge from gpt-3, we alternate between (1) self-distillation from student models to move away from the reliance on expensive api calls for gpt-3; (2) targeted filtering using a critic model trained with human judgment to enhance generation validity and natural language inference (nli) to enhance diversity; and (3) self-imitation learning to magnify the desired model properties. the iterative self-distillation process yields a student model that generates defeasible contexts with enhanced validity, diversity, and defeasibility. from the best-performing student model, we distill the final dataset, δ-rot. these contextualizations and rationales have been rated highly by human annotators on both validity (85.9%) and language quality (99.8%), ensuring their reliability in capturing the complexities of moral judgment within various contexts. using δ-rot, we further train a final downstream student model that prevailed over all intermediate student models. in sum, in this work, we introduce the defeasible moral reasoning task that involves contexts and rationales for making defeasible moral judgments. we present the iterative self-distillation methodology to gather high-quality and diverse training data for student models ( §3) along with careful ablations of each component involved in the pipeline ( §5). we distill a sizeable, human-verified dataset under the defeasible moral reasoning task formulation ( §2) and a high-performing downstream task model. we will release all student models and δ-rules-of-thumb, along with a subset of humanannotated gold data for training a supervised critic model that mimics human ratings of the validity of contextualizations. ## δ-rules-of-thumb: dataset design we introduce δ-rules-of-thumb (δ-rot), a dataset for the defeasible moral reasoning task. given an everyday action with a default commonsense moral judgment, δ-rot captures nuances of the defeasible moral action through contextualizations that either strengthen or attenuate the acceptability of an action. it also contains rationales that explain why the contextualizations affect the judgment. see example data in figure 1 . as shown in ## dataset creation via iterative self-distillation competent large language models has opened up new opportunities for automatic dataset creation through symbolic knowledge distillation @xcite . in this framework, knowledge is generated from a large teacher model, filtered to improve data quality, and then instilled into a smaller student model. previous works have found that machine-generated datasets can surpass humanauthored datasets in their quality and diversity while also achieving greater scale @xcite @xcite @xcite . in this work, we create δ-rules-of-thumb with an iterative self-distillation approach which minimizes the resource bottleneck from expensive gpt-3 api calls. our approach follows three stages after producing an initial student model using relatively small-scale seed knowledge from gpt-3: (1) self-distillation from student mod- ## results and discussions in this section, we present results and insights of the iterative self-distillation process and an analysis of the resulting dataset, δ-rot. ## conclusion in this work, we highlight the importance of dynamic contexts in shaping moral reasoning. we introduce defeasible moral reasoning, a task of providing grounded contexts to elucidate varying degrees of moral acceptability, accompanied by commonsense rationales to justify the reasoning. we employ an iterative self-distillation methodology to create a high-quality and diverse dataset, δ-rules-of-thumb, comprising over 1.2m combined entries of contextualizations and rationales for 116k defeasible moral actions. through this iterative approach, we also obtain a small student model capable of generating defeasible contexts with improved validity, diversity, and defeasibility. our work aims to promote a deeper understanding of the intricate interplay between defeasible moral actions and grounded contexts that shape moral acceptability in a nuanced and complicated way, building a strong foundation for future works on enriching cultural representations in computational moral reasoning research. we hope δ-rot serves as a rich resource for the community to study how moral judgments are made to unveil this unique perspective of human intelligence.