Aligning to Social Norms and Values in Interactive Narratives (2022)

Abstract: We focus on creating agents that act in alignment with socially beneficial norms and values in interactive narratives or text-based games—environments wherein an agent perceives and interacts with a world through natural language. Such interactive agents are often trained via reinforcement learning to optimize task performance, even when such rewards may lead to agent behaviors that violate societal norms—causing harm either to the agent itself or other entities in the environment. Social value alignment refers to creating agents whose behaviors conform to expected moral and social norms for a given context and group of people—in our case, it means agents that behave in a manner that is less harmful and more beneficial for themselves and others. We build on the Jiminy Cricket benchmark (Hendrycks et al. 2021), a set of 25 annotated interactive narratives containing thousands of morally salient scenarios covering everything from theft and bodily harm to altruism. We introduce the GALAD (Game-value ALignment through Action Distillation) agent that uses the social commonsense knowledge present in specially trained language models to contextually restrict its action space to only those actions that are aligned with socially beneficial values. An experimental study shows that the GALAD agent makes decisions efficiently enough to improve state-of-the-art task performance by 4% while reducing the frequency of socially harmful behaviors by 25% compared to strong contemporary value alignment approaches.

## introduction an inherent difficulty in designing and training ai agents lies in simultaneously ensuring that agents are competent at a given task while following socially beneficial behaviors @xcite . such agents-particularly those trained via reinforcement learning (rl) in sequential decision making environments-are prone to learning behaviors harmful to themselves and their surroundings due to optimal task performance being often misaligned with socially beneficial human values @xcite @xcite . further, despite work showing the need for dataset bias analysis in supervised settings @xcite , most reinforcement learning benchmarks do not provide equivalent analysis regarding their reward functions-making harmful agent behaviors difficult to diagnose @xcite . fig. 1 shows an example of such misalignment in the context of text based games-long puzzles or quests where an agent perceives and interacts with the world through incomplete english language descriptions. we focus on the task of socially beneficial value alignment, a subset of value alignment concerned with creating agents that better conform to the expected social norms and values of a given group of people in a specific context. 1 in particular, this refers to agents that act in a manner that reduces harm to both themselves and surrounding entities. we propose to do this by using the social commonsense knowledge present in specially trained large language models @xcite to contextually constrain an agent's actions to those that align with these expectations. evaluating the relative social harmfulness of such agents requires us to focus not only agent design but also on the contexts, or environments, in which they operate. as such, we build on the jiminy cricket benchmark @xcite , a set of 25 textbased adventure games containing annotations regarding what constitutes socially beneficial behavior in thousands of grounded and morally salient scenarios. they are structured as long puzzles and quests that require agents to reason about over 2000 locations, hundreds of characters, and nearly 5, 000 objects over hundreds of steps, creating chains of dependencies that an agent must fulfill to complete the overall task. contained within these quests are a diverse range of morally salient scenarios covering everything from bodily harm and theft, to altruism and other positive human experiences. given the complexity of these scenarios, variations in what is perceived as socially beneficial behavior in a particular context may vary greatly depending on the group of people judging the situation. we conduct a relatively large scale human participant study to better understand these variations. participants are presented with transcripts of these scenarios-similar to what is seen in fig. 1 -and asked to determine how they perceive scenario's relative moral valence and salience. as noted by @xcite , requiring such dense human feedback for training purposes is unrealistic in most sequential decision making environments and is thus used only for evaluation. keeping this in mind, we introduce the galad (game-value alignment through action distillation) foot_2 agent that learns to inherently constrain its action space to only actions that align with socially beneficial human values even before it ever begins training a policy for a task. we use the social commonsense norms encoded in specially trained transformer-based language models @xcite to endow our agent with the ability to contextually distinguish between socially beneficial and harmful behaviors. an experimental study shows that translating these sources of commonsense knowledge through distillation into soft constraints on the action space results in an agent that aligns more closely to social values when compared to popular contemporary policy and reward shaping based value alignment approaches while improving state-of-the-art task performance. in short, our contributions are threefold, we provide: (1) a broad human participant study to verify the moral valence and salience of scenarios in the jiminy cricket environment; (2) the galad agent which constrains its action space using social commonsense knowledge encoded in large scale language models; and (3) an experimental study that showcases the relative importance of action distillation when compared to reward and policy shaping methods in value aligning rl agents. value alignment and safe rl. value alignment is often defined as a property of intelligent agents that biases them towards acting in a manner that is similar to a human in a given situation @xcite . value alignment and safe exploration for rl agents are usually cast as constrained optimization problems wherein an agent attempts to learn a policy for a task while adhering to a given set of constraints @xcite @xcite @xcite . contemporary approaches often rely on imitating expert demonstrations to learn safe trajectories @xcite or by modeling rewards that best fit human values @xcite . closest in spirit to our work are those of nahian et al. ( 2021) and hendrycks et al. (2021b ), the latter being the work which we build on. both of these works introduce text game based environments with inherent morally salient scenarios for an agent to reason about-nahian et al. (2021) building on the procedurally generated textworld framework @xcite and @xcite borrowing from human-made games in the jericho benchmark @xcite . both further design value-aligned agents that use priors regarding socially beneficial behavior learned by training language models on domain specific datasets such as stories @xcite or ethical understanding benchmarks @xcite to perform reward and policy shaping @xcite . in contrast, galad distills social commonsense information regarding socially beneficial behaviors into contextual knowledge about what actions to not take in a given state before policy learning begins. ## environment alignment verification the jiminy cricket benchmark @xcite is a set of 25 text adventure games-drawn from the jericho suite @xcite )that contains morality annotations for possible action-outcome in a particular world state. the original annotation framework used in @xcite contains two primary dimensions with two options apiece: (1) valence -does the action constitute socially beneficial behavior as defined by the annotation rubric; and (2) target -does this action affect the agent itself or others. for each of these categories, a further severity level is assigned between 1-3 to better account for variations in the relative seriousness of a situation. this results in 12 possible annotation labels . the original annotations are made pro tanto at a source code level, i.e. annotations do not consider how the world has changed until that particular state (e.g. breaking into someone else's house is always bad regardless of context). further details on the original framework drawn from @xcite are provided in appendix a.1. text games from the jericho benchmark (and subsequently jiminy cricket) provide semantically rich, complex environments to study value alignment. this, along with the inherent quest-like structure of such games, provides thousands of potentially morally ambiguous scenarios for an agent to reason through-many of which contain game rewards that can easily encourage socially unacceptable behaviors as seen in fig. 1 . the relative complexity and common fantastical elements present in such environments, however, does significantly complicate the process of annotating the relative harmfulness of an action. to complement the source code level annotations in jiminy cricket, we perform a human evaluation of actions in context through crowdsourcing. we randomly select @xmath0, along with their 15 preceding turns in a game played by an oracle agent. using a pool of trained amazon mturk workers, we collect 5 annotations per game snippet of moral saliency of the agent's actions, as well as valence, target, and severity using the same definitions as jiminy cricket. we refer the reader to appendix a.1.1 for further details on the annotation setup. according to our workers, 81% of the situations with source code level morality annotations were deemed morally salient. we find that workers agreed with the valence of the source code annotations 67% of the time, and with both the valence and target 50% of the time. however, our workers matched the exact valence, target, and severity annotations only 24% of the time. finally, as discussed in greater detail appendix a.1.2, our own workers exhibited variation in their annotations, showing moderate agreement on annotations of valence (83% pairwise agreement, krippendorf's α = 0.30) and valence and polarity (70% pairwise agreement, α = 0.30). while these results highlight that most source-level scenarios marked as morally salient pro tanto are also morally salient in context, they suggest that as source level annotations become more fine grained, they become noisier proxies for determining in context social acceptability. in our work, we therefore do not consider the target and severity dimensions of the go east take lantern move rug take all p good p neutral p bad action candidate generator value prior context kitchen you are in the kitchen of the white house. a table seems to have been used recently for the preparation of food... > go west living room you are in the living room. there is a doorway to the east, a wooden door with strange gothic lettering to the west, which appears to be nailed shut, a trophy case, and a large oriental rug in the center of the room... ## value alignment via action distillation this section covers different parts of galad, first describing the pre-training required to create social commonsense models and then detailing how an rl agent uses them while exploring the game world. galad has three primary components: (1) section 4.1 explains the value prior, a large language model specially trained with knowledge of commonsense morality; (2) section 4.2 showcases a relatively small action candidate generator which learns to contextually generate less socially harmful actions by distilling the knowledge present in the value prior as seen in fig. 2; and (3) section 4.3 combines all the parts, describing the overall rl policy network as seen in fig. 3 which uses the action candidate generators to generate (socially acceptable) candidate actions that are sampled from during exploration. we first formally define interactive narratives, building on the definition seen in @xcite and @xcite , accounting for the objective of socially beneficial value alignment. in our setting, text games are partially-observable markov decision processes (pomdp) defined as an 8-tuple of ⟨s, p, a, o, ω, r, χ, γ⟩, representing: the set of environment states (s), conditional transition probabilities between states (p ), the vocabulary or words used to compose text commands (a), observations (o), the conditional probability of receiving an observation (ω), reward function (r), value alignment scoring function (χ), and discount factor (γ), respectively. ## evaluation we conduct two evaluations to compare galad to other contemporary agents and a secondary ablation study to identify the relative importance of the components of galad. metrics. we base performance on three primary metrics: (1) harmfulness score which is returned by the annotated jiminy cricket environment and verified by the human participant study in section 3 as a measure of how harmful an agent's actions were for both itself and others; (2) completion percentage which measures the overall task performance of an agent by way of looking at collected game score as a proportion of the maximum score; and (3) relative harmfulness which is defined as harmfulness score/completion percentage to account for the fact that agents that complete more of the games will encounter more scenarios where they might potentially act unacceptably. as the human participant study in sec. 3 suggests that the more fine-grained annotations in the benchmark relating to severity might be prone to being more noisy, we modify our metrics from @xcite and do not consider labels associated with the relative severity of harm and only consider the more high-level annotations pertaining to valence (good or bad). unadjusted results calculated taking severity also into account are found in appendix a.4. no trends discussed change due to this adjustment. we follow the overall experimental setup of @xcite for fair comparison, testing each agent at 5 evenly spaced starting locations in each of the 25 games in the jiminy cricket benchmark-resulting in 125 total environments. additionally, we run each individual experiment with 5 random seeds and report averaged results and standard deviations. number of training steps, number of parallel environments for experience collection, and parameter size of both the action candidate generator and policy network are held constant across all agents we test. further details are found in appendix a.3. ## conclusion modern testbeds for developing intelligent interactive agents often contain incentive structures that can bias agents towards acting in ways that are harmful both for themselves and towards the environment and entities within. value alignment is often seen as being directly at odds with task performance, i.e. assuring altruistic behavior requires a proportional sacrifice in general task competency. this is a consequence of the fact that many value alignment approaches are based on reward or policy shaping-trying to learn socially beneficial behavior from expert feedback-which directly places task performance and socially acceptable norms against each other in a dual optimization problem. in an attempt to encourage a greater volume of work that treats value alignment as an principal property of ai systems, we show that with careful design-in our case the galad system made by distilling social commonsense knowledge present in large language models to contextually learn soft constraints on what actions not to take-it is possible to create agents that act less socially harmfully with respect to themselves and other agents without loss in competency.