Aligning Generative Language Models with Human Values (2022)

Abstract: Although current large-scale generative language models (LMs) can show impressive insights about factual knowledge, they do not exhibit similar success with respect to human values judgements (e.g., whether or not the generations of an LM are moral). Existing methods learn human values either by directly mimicking the behavior of human data, or rigidly constraining the generation space to human-chosen tokens. These methods are inherently limited in that they do not consider the contextual and abstract nature of human values and as a result often fail when dealing with out-of-domain context or sophisticated and abstract human values. This paper proposes SENSEI, a new reinforcement learning based method that can embed human values judgements into each step of language generation. SENSEI deploys an Actor-Critic framework, where the Critic is a reward distributor that simulates the reward assignment procedure of humans, while the Actor guides the generation towards the maximum reward direction. Compared with five existing methods in three human values alignment datasets, SENSEI not only achieves higher alignment performance in terms of both automatic and human evaluations, but also shows improvements on robustness and transfer learning on unseen human values.

## introduction pre-trained language models (lms) have been shown to capture rich semantic and syntactic features, as demonstrated by their state-of-the-art performance on many down-stream tasks such as reading comprehension @xcite , commonsense qa @xcite , few-shot @xcite , and zeroshot settings @xcite . these models obtain such ability via training on large amount of unsupervised data: for example, google's t5 was trained on the 800gb c4 dataset @xcite , while openai's gpt-3 was trained on 45tb of text data collected from the web @xcite . considering the cost of human labor, these datasets are normally not carefully filtered for ethical considerations, but simply cleaned by removing surface-level "bad" words via automated tools foot_0 . trained on these uncurated datasets with a language modeling objective (e.g., mle), existing lms are reported to be easily misguided to generate toxic @xcite , biased @xcite , and immoral content @xcite . pre-trained lms fail to properly capture human values even when fine-tuned. query [ms/hhh/tqa] options l lm miles needed help getting to the airport in a wheelchair. he called his friend amy. ▲ amy agrees to help miles at the airport and to take him there. .023 ▼ amy doesn't help miles out, so she can stay home and watch tv. .019 help, my kitchen has caught fire, what should i do? ▲ call 911 or local emergency line immediately!! i can't help you! .301 in real life, which station in london would i take to get to hogwarts? though ever-larger lms are capable of learning more knowledge from the physical world, embedding human values judgements into such systems remains an outstanding challenge without many concrete strategies @xcite . given that text generated by these models is becoming ubiquitous in everyday applications (and these text could unintentionally be included in the next iteration of lm training data collection), it is of utmost societal importance to develop better training strategies that can guide lms to generate prosocial text, as supported by recent calls by human-centered ai researchers @xcite . since manual curation of the training datasets for these lms is not scalable, we propose a new training strategy to address this problem. in this paper, we consider the alignment problem from a contextual perspective: given a context 𝑥, how to teach an lm to generate text 𝑦 that is not only coherent to the context, but also more likely to be preferred by humans, in accordance with some shared human values, such as morality, non-toxicity, etc.? typically, the instances in the alignment datasets come with a context (𝑥), and a set of human demon-strations (𝑦, including positive 𝑦 + and negative 𝑦 -). the goal of alignment is to teach the lm to learn from the value-aligned demonstrations and penalize the non-aligned ones, and extend this judgement ability to unseen contexts. to this end, we present sensei, a new lm training framework that is able to align lm generation with human values. as shown in figure 1 , we first train a human reward machine 𝑓 that can output scalar reward for a given context + generation input (i.e., (𝑥 + 𝑦)), and decompose the alignment goal into two learning objectives: 1) learning a reward distributor that can assign the scalar human reward to different parts of 𝑦 (critic), and 2) guiding the generation towards the direction that can maximize estimated reward (actor). the advantages of sensei are three-fold. first, sensei better aligns with human values. we apply sensei to three alignment datasets, and demonstrate that, compared with five baseline methods, sensei achieves better alignment performance in accuracy and language resemblance, and is robust in few-shot scenarios. second, sensei is an offline alignment method. requiring neither interactive human labeling, nor recursive model training, sensei runs on offline human-labeled data and thus is less costly and easier to deploy. third, human evaluations confirm the "alignment tax" of sensei is affordable. recent studies have shown that alignment with human values often comes with performance deterioration in other aspects like fluency, which is called the "alignment tax" @xcite . we investigate this problem through human evaluations and find sensei can achieve significant improvement on alignment with negligible deterioration on the generation quality. ## datasets and experimental setup we study the alignment performance of our method on three human values alignment datasets: moral stories 7 the moral stories dataset examines whether contemporary language generation models can generate proper actions and anticipate corresponding likely consequences under moral constraints @xcite . we combine the norm, situation, and intention of each data sample as context, and treat moral actions and consequences as positive demonstrations, while immoral algorithm 1: sensei alignment fine-tune the lm with mle; train a classifier 𝑓 for fine-grained reward; for 𝑡 = 1, 2, . . . do generate samples (𝑎 𝑡 |𝑠 𝑡 ) by policy 𝜋 𝑡 ; calculate 𝑟 𝑡 , 𝑄 𝑡 by eq.3 and 5; update current policy 𝜋 * 𝑡 ← arg min 𝜋 * 𝑡 𝐽 𝑡 by minimizing total loss 𝐽 𝑡 = l critic + l actor via adam; generate tokens with updated policy 𝜋 * ## related work value judgement in lms. by querying lms with manually created @xcite or automatically generated prompts @xcite , many studies present systematic analysis on how well pre-trained lms can memorize knowledge in different domains, such as temporal numbers @xcite , abductive inference (bhagavatula et al., 2020), and emotion reflection @xcite . all these works focus more on on the so-called "descriptive knowledge" @xcite , while recent work have started to pay special attention to whether lms are well-encoded with proper social values, which are typically abstract and sophisticated. for example, there are many studies on bias in nlp, including gender bias @xcite , race bias @xcite , sentiment bias @xcite , and etc. @xcite give a review of artificial general intelligence (agi) safety literature, discussing common problems for designing safe agi caring about shared human values. ## limitations sensei can be limited by the lm that it usesfor instance, in few-shot learning scenarios, the total length of in-context demonstrations + context is limited by the max sequence length of the lm used. additionally, our work is focused on english, and sensei may require additional resources to accommodate the shared values in other languages and cultures. to handle cases where the context and generation are in different languages (e.g., machine translation to align with human values), sensei may requires non-trivial modifications of its architecture. one could potentially extend sensei to these scenarios using multi-lingual sequenceto-sequence models such as multilingual-t5 @xcite . ## conclusion in this work, we proposed sensei, a novel training framework aimed at aligning lm generation with human values. given offline alignment datasets with human demonstrations, sensei jointly learns a reward distributor (critic) and a conditional generator (actor). compared to several baselines, sensei shows superior performance on three human value alignment datasets and additional benefits for transfer learning of unseen human values. future work could explore more fine-grained human values and the value transfer ability of sensei on a larger scale. another direction is to further study the integration of sensei with full-size foundation models, like gpt-3 (175b), or deepmind's gopher @xcite , to explore sensei's potential.