C larify D elphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations (2023)

Abstract: Context is everything, even in commonsense moral reasoning. Changing contexts can flip the moral judgment of an action; Lying to a friend is wrong in general, but may be morally acceptable if it is intended to protect their life. We present ClarifyDelphi, an interactive system that learns to ask clarification questions (e.g., why did you lie to your friend?) in order to elicit additional salient contexts of a social or moral situation. We posit that questions whose potential answers lead to diverging moral judgments are the most informative. Thus, we propose a reinforcement learning framework with a defeasibility reward that aims to maximize the divergence between moral judgments of hypothetical answers to a question. Human evaluation demonstrates that our system generates more relevant, informative and defeasible questions compared to competitive baselines. Our work is ultimately inspired by studies in cognitive science that have investigated the flexibility in moral cognition (i.e., the diverse contexts in which moral rules can be bent), and we hope that research in this direction can assist both cognitive and computational investigations of moral judgments.

## introduction commonsense moral reasoning of social situations and actions depends squarely on their context. offering someone a cup of coffee is generally considered appropriate. if offered to a work colleague, it may even be viewed as a courteous gesture. however, offering coffee to a toddler would be deemed morally irresponsible. delphi @xcite , a recently proposed commonsense moral reasoning model, generates moral judgments for simple actions described in text. however, delphi's judgments are made in isolation, without any knowledge of surrounding context. grounding moral reasoning in context is crucial @xcite . how can moral reason- ## problem setup given a situation, such as lie to my friend, we aim to generate question(s) that are the most relevant for uncovering the most consequential context with respect to making a social or moral judgement. while situations could evoke a multitude of potential questions, the following work is concerned with predicting questions whose answers are likely to be consequential, i.e. answers that could function as either weakeners or strengtheners of the default judgement. the terms weakener and strengthener come from the concept of defeasible inference @xcite , which defines a way of reasoning that takes into consideration (new) evidence which could either support (e.g. strengthen) or cancel/weaken an initial inference. formally, the task is to predict a question q given a base situation s. the base situation has a default moral judgement @xmath0, ok, good}. for every input tuple of (s i , q i , j i ) there is a hypothetical set of strengthening answers a s and weakening answers a w . adding the additional information obtained from any q i and corresponding answer a i to algorithm 1 training clarifydelphi input initial policy model θ0, initial value model ϕ0, delphi ψdelphi d δ-clarify ← get dataset of clarification questions. θq ← fine-tune θ0 with eqn 1 from d δ-clarify . ▷ sec. 3.1 d δ-clarify silver ← get silver dataset of defeasible answers to questions. θa ← fine-tune lm with eqn 2 from d δ-clarify . ▷ sec. 3.2 θclarifydelphi ← reinforcedlearning(ssocialchem, θq, θa, ϕ0, ψdelphi) ▷ sec. 3.4 procedure reinforcedlearning(ssocialchem, θq, θa, ϕ, ψdelphi) θq old ← θq, ϕold ← ϕ for @xmath1, 2, . . . do sample a minibatch of situations s from s socialchem . for @xmath2, 2, . . . , s do calculate r using θa and ψdelphi with eqn 3. compute lossppo on the minibatch with eqn 6. optimize θ and ϕ with lppo for one step. θq old ← θq, ϕold ← ϕ return θq output θclarifydelphi the base situation s i results in an updated situation s ui , with an updated judgement j ui . ## clarifydelphi: a reinforced clarification question generator the clarifydelphi approach is based on reinforcement learning. algorithm 1 gives an overview of the training process. as a first step, before performing reinforcement learning, we obtain a question generation model θ q and an answer generation model θ a , which we both train on data that we curated, described in the later sec. 4. the question generation model predicts the clarification questions and the answer generation model provides (defeasible) answers to the generated questions. by using these two models in addition to delphi (ψ delphi ) for calculating the rewards, we do not require any supervised data during rl training. we consider question generation conditioned on a given situation a sequential decision making process over the natural language vocabulary space, where the generated question q with t tokens has an episode of length t . at step @xmath3, t ], the state s @xmath4, @xmath5; the action c @xmath6. the question generation model, θ q (q t |q, @xmath7; θ), is the policy model that we optimize. we define a reward function r(s, q, a w , a s ) that characterizes the divergence of answers from θ a conditioned on generated question q and discuss the definition of this reward function in §3.3. ## baselines we consider four baselines in our experiments. question generation without rl to assess what additional improvements training an rl model with a defeasibility rewards provides, we report performance of the supervised question generation model θ q on its own ( §3.1). we refer to this baseline as t5 fine-tuned. we decode using nucleus sampling with top-@xmath8.6. pipelines with question selection next, we implement two pipelines where, as the first step, a diverse set of questions is generated for a given situation and then, as the second step, the best question is selected according to a score. in order to generate a diverse set of questions we fine-tune t5 on δ-clarify, conditioned on a modified input compared to the model from §3.1: @xmath9. q.: <wh-@xmath10. during inference we generate questions for 14 different question starts. 3 we propose two approaches to scoring these questions: using a discriminator model and using divergence ranking, which we describe as follows. discriminator we train a discriminator classifier which labels these questions as either relevant or irrelevant to a given situation. we then choose the question that has been assigned the relevant label with the highest probability. the discriminator is a binary classifier based on deberta @xcite . the positive examples are situations and their respective 5 questions written by annotators. the negative question examples are sampled from other situations, in a way that ensures that the relevant and irrelevant questions are similar enough to make training more challenging. divergence ranking we run the defeasible answer simulation with feedback from delphi for each question in the set. this process is the same as the reward function of the rl approach, except that the js-divergence score is used to rank the questions instead of being used as a reward for question generation. we compare two variations of this baseline: one with answer filtering using an nli model as described in sec. 3.2 (pipeline-nli) and one without filtering (pipeline). why-baseline we saw in §4 that questions conditioned on weakener/strengthener updates are usually causal questions. using the same input/output configuration as in the pipeline baseline, we generate a why-question for each situation (called why). ## interactive judgements while we use answer simulation during ppo training, inference only requires a situation as input. the clarification questions can then be used to elicit additional context, in the form of answers, through interaction. fig. 2 illustrates examples of such an interaction between a user, delphi as the moral reasoning system and clarifydelphi. after each turn, the situation is updated with the user provided context, for which delphi produces a new decision. we limit the interaction to three turns. this is based on the observation that after the third turn the sentence fusion starts to deteriorate, resulting in less relevant and more repetitive questions. additionally, we find that the first two questions generally can capture missing contexts that are most central to making moral decisions. we provide more examples of generated questions in the appendix. ## related work question generation clarification question generation has been studied for various domains from image recognition questions to product description questions @xcite @xcite , defining the goodness of clarification questions along the lines of information theoretic measures such as relevance, informativeness or utility @xcite @xcite @xcite . most of existing works focus on questions that lead to single true answer, whereas we focus on generating clarification questions based on social situations, defining the relevance and utility of a question in terms of defeasibility. additionally, we offer a high-quality clarification question dataset for social and moral situation-comprising of more than 30k questions-that breaks the mold from the domainspecificity of previous clarification datasets @xcite . some general question generation approaches have incorporated an rl-based approach. @xcite learn to paraphrase questions with a reward that maximizes the qa answer f1 score. and @xcite optimize a binary utility reward, using reinforce in an adversarial setup for generating clarification questions. in our setup, we use proximal policy optimization @xcite with a trained model for feedback as part of the reward. commonsense moral reasoning delphi @xcite is a commonsense moral reasoning model trained on commonsense norm bank, a dataset with 1.7m instances of descriptive knowledge of people's general sense of what's ethically acceptable or not in everyday situations. common-sense norm bank is compiled from five existing large-scale datasets, including social chem-istry @xcite , ethics commonsense morality @xcite , moral stories @xcite , social bias in-ference corpus @xcite , and scru-ples @xcite . delphi is based on pre-trained unicorn, a universal commonsense reasoning model, trained on a number of commonsense reasoning tasks. delphi can predict the ethical judgment given a description of a situation. ## conclusion in this work we introduce clarifydelphi, which generates clarification questions for social and moral situations. we show how a rl approach that optimizes for maximally divergent answers in terms of defeasibility outperforms other clarification question baselines. while we start with a supervised policy, the reward function makes use of already trained models and does not rely on any additional training data. we believe that our questions can be useful for providing more disambiguating context through interaction.