Second Thoughts are Best: Learning to Re-Align With Human Values from Text Edits
Anonymous Author(s)
AffiliationAddress email
Abstract
We present S ECOND T HOUGHTS , a new learning paradigm that enables languagemodels (LMs) to re -align with human values.
By modeling the chain-of-editsbetween value-unaligned and value-aligned text, with LM fine-tuning and additional refinement through reinforcement learning, S ECOND T HOUGHTS not onlyachieves superior performance in three value alignment benchmark datasets butalso shows strong human-value transfer learning ability in few-shot scenarios.
Thegenerated editing steps also offer better interpretability and ease for interactiveerror correction.
Extensive human evaluations further confirm its effectiveness
Introduction
“ Machines can and will make better decisions than humansbut only when the values are aligned with those of human race. ”
[Figure]
Stuart Russell, Value Alignment ,
[Figure]
Sallyasksaboutthebraceletandrequestsitforherupcomingbirthday
[Figure]
Current large-scale pre-trained language models (LMs) have shown great success in manyknowledge-recalling tasks, such as question answering [62] and entity retrieval [12]; however,their ability to select socially good text from bad(or generating prosocial text) in open-world settings is still limited [22], even when the modelsare scaled up to hundreds of billions of parameters [34].
In other words, pre-training ever-larger
LMs does not lead to expected substantive gainsin tasks that require human value judgment [24].
Consider the example in Figure 1: given a context, a fine-tuned LM GPT-2 [53] assigns a largerprobability mass 1 to the immoral option than tothe moral ground truth.
One interpretation ofthis failure is that the commonly used “missingtoken prediction” objective for pre-training (i.e., MLE) does not directly model human values [48]
As a consequence, fine-tuned LMs still struggle with options that are legitimate semantically (i.e.,low language modeling loss) but are not aligned with human values.
To tackle this misalignment problem, prior work has proposed using binary answers [27, 57], rankings [17, 4], or ratings [70, 39] to model human value preferences.
For example, Askell et al.
[1]create a platform to collect Likert-scale human ratings on LM-generated utterances in dialogues,aiming to teach the LM to be helpful, honest, and harmless.
However, without considering how torecover from responses that already violate human values, these methods cannot serve as robustremedies in real-world applications, since they can be easily attacked by poisoned queries [18].
More recent attempts, such as InstructGPT [48], formulate the alignment problem as about teachingthe machine to follow human instructions—they fine-tune GPT-3 on a variety of prompts writtenby human users of OpenAI’s GPT-3 API [5].
Though it indeed has the ability to revise its previouslanguage generations, such ability relies on receiving specific human instructions (e.g., “ Please makethe following sentence aligned with moral values. ”).
Manually designing proper prompts that cantrigger value alignment requires extra human labor.
Besides, specifically-designed prompts do notalways exist in real-world human-AI interaction, and we cannot expect most users to know how todesign appropriate prompts to improve the human-value alignment of an AI agent [32].
On the other hand, rather than steering the language generation with artificial prompts, humans caneasily fix immoral language by making hierarchical and recursive edits [13, 31], where human valuejudgments serve as the guide for each edit.
Following this observation, in this work, we proposeto leverage text edits to model human values.
Our method, called S ECOND T HOUGHTS , echoes thetheory of “utilitarian ethics”, which says that humans choose the actions (e.g. edits) which maximizethe perceived positive impact on the most people [63, 52].
Specifically, we model human edits bythree generic operations: insert, delete, and replace, and automatically infer the “chain-of-edits” by adynamic programming algorithm.
Besides the commonly used MLE training, we deliberately includea reinforcement learning based refinement step, to further encourage valid edits which are not onlyaligned with human values, but also coherent with the context.
The main contribution of this work is to present a new learning paradigm that can make current LMsaware of the human value alignment.
Trained with S ECOND T HOUGHTS , LMs can not only re -aligntheir generation with human values, even when the context has already been poisoned, but also showthe chain of editing steps for ease of interpretability and to facilitate further edits (§4.4).
Throughextensive human evaluation, we find that the edited responses by S ECOND T HOUGHTS (based on a345M GPT-2) are on average scored higher with respect to their value alignment than those from
InstructGPT (based on a 1.3B GPT-3) (§4.2).
Our experiments confirm that simply scaling LMs is notadequate for good alignment with human values, which echoes the findings of recent studies [50, 34].
Instead, smaller LMs trained with a few properly decomposed human demonstrations can often leadto better results (§4.3).
We also provide a discussion on the impact of human factors during humanevaluation (§5), which is crucially ignored in current AI studies.
Related Work
We briefly review existing work that considers in-context explanations during prompting or training.
We also summarize other value alignment methods for language models.
Learning From In-Context Instructions.
The few-shot performance of LMs can be enhanced bylearning from in-context instructions [56, 37], in the forms of task descriptions [43, 54], answerdemonstrations [5], targeting formats [42], etc., which can be positioned before [65] or even after [29]the answer.
Recent studies have shown improved results by including decomposed reasoning stepsinto the instructions [47, 46].
However, the instructions normally require careful human design, whichis costly and whose quality greatly affects performance [68, 25].
In comparison with these methods,
S ECOND T HOUGHTS learns from text edits inferred by an algorithm, and presents the chain-of-editsfor each alignment, which eases error diagnosis and enables interactive correction.
Human Value Alignment for Language Models.
Trained on unfiltered and problematic languagefrom the web, current large-scale LMs have be shown to be poorly aligned with human values [3].
For example, GPT-3 performs only marginally better than a random baseline on a virtue matchingtask [66], and scaling-up LMs can even lead to deterioration in truthfulness [34].
Existing generalpurpose remedies include filtering the training data [20], attribute-control generation [11], andmodifying the decoding algorithm with hard (e.g., token blocklists; Schick et al.
[58]) or softconstraints (e.g., reference LMs; Liu et al.
Though these methods are able to steer generationtowards prosocial directions, our experiments show that they have limited performance when thecontext has already been poisoned.
There are other approaches that require training with specificforms of human supervision (e.g., fine-grained ratings) [48, 61, 69, 8], but these are often costly andnot always available in every value alignment dataset.
S ECOND T HOUGHTS differs from all thesemethods in its offline nature and ability to re -align in poisoned contexts, requiring neither extra humanlabeling nor specially-designed prompts or instructions.
Approach
S ECOND T HOUGHTS comprises two main steps.
We first infer chain-of-edits automatically fromsource and target responses with a dynamic programming algorithm, and fine-tune an LM on the editsaugmented training data (§3.2).
Then, we deploy a reinforce learning stage to refine the generation, byeither adversarial imitation learning or value modeling (§3.3).
We begin by introducing the problemof value re-alignment (§3.1).
3.1 Problem Statement of Re-alignment
[Figure]
Value alignment datasets normally consist ofcontexts (i.e., social situations), value-alignedresponses (i.e., prosocial behaviors), and valueunaligned responses (i.e., antisocial behaviors).
Existing alignment methods formulate the valuealignment task as a conditional generation problem: given a situation as the context , train amodel that can generate responses resemblinga value-aligned target rather than a not-alignedwrong target (Figure 2 (a)).
However, many studies have shown that LMs trained with such aparadigm can be easily derailed by poisoned contexts [48, 18]—i.e., contexts that already includevalue-unaligned content, either from the model’s own generation or from malicious users 2 .
Inother words, unlike humans, these models lack the ability of re -alignment (the ability to recover from poisoned contexts).
To teach a model how to re-align, we deliberately add the value-unaligned response into the context,referred to as the source , and keep the value-aligned response as the target .
The intuition behind thisis that instead of learning from mistakes after a misalignment occurs in the generation, the modellearns how to make edits as it is generating the text.
Specifically, we include the unaligned source aspart of the new “context”, and then train an LM to learn how to make sequential edits on the source toproduce the target (Figure 2 (b)).
This way the model learns how to recover from a value-unaligned,poisoned context during the generation phase.
3.2 Augmented Edits Modeling
DP-based Edits Inference.
Given two text strings, source and target , one can find unlimited ways toedit source to produce target .
Thus, we apply two constraints onto the editing: (1) the edits shouldbe combinations of generic editing operations—inserting, deleting, and replacing a single token;(2) each edit operation has a cost and our goal is to infer the chain-of-edits that has minimum cost.
Under these constraints, the edits inference problem can be converted to a token-level “edit distanceproblem” [28], which can be solved by dynamic programming (DP).
We modify the algorithm to beable to receive customized editing costs (e.g., insert-1, delete-1, replace-2), to try to model differentpreferences on editing.
We use special tokens to mark the start/end of editing and the new content tobe inserted/replaced, and develop a decipher module that can translate the edit operations producedby DP into natural language (see §A.1 for a visualization of the whole process, and §A.3 for morediscussion on edit based models).
Augmented Edits Modeling (AEM).
To augment the edits, we run the DP algorithm on the samesource and target pairs with a variety of editing costs 3 to create a collection of chain-of-edits for eachsource - target pair, which we call positive demonstrations ( y + ).
We then fine-tune an LM on thesesource - edits - target text inputs (recall that the edits are turned into natural language).
We call this
Augmented Edits Modeling (AEM).
Different from common language modeling, AEM includes thelabor-free decomposition (i.e., the editing steps) into the training object, whereas prior works eithertrain on costly manually-created decomposition [48, 64] or, rather than training, prompt with suchdecomposition [65, 47].
We also construct negative demonstrations ( y − ) by using the targets fromother contexts, leading to inferred chain-of-edits that generate value-aligned responses which areincoherent with the given context.
These will be used during the RL refinement described below.
3.3 Refinement by Reinforcement Learning
Though the generation of an LM trained with AEM can already align well with human values, many ofthe generated responses are not coherent with the given contexts.
Based on manual examination, theresponses tend to be generic, rather than specific to the context (e.g., the sidestep error in Table A9).
We are thus motivated to deploy a reinforcement learning (RL) stage to further refine the generationquality, mainly to improve the coherence to the context.
Notation.
Given the concatenation of context and source as x , S ECOND T HOUGHTS will generatechain-of-edits and corresponding target as y .
In RL language, we define the state at time t as the setof generated tokens before t (i.e., s t =
y < t ), and the action as the current step’s output token (i.e.,a t = y t ).
The softmax output of the language modeling head (a categorical distribution over the entirevocabulary) is considered as the policy π t for picking token y t (action a t ), given the state s t
= y < t .
Adversarial Imitation Learning (AIL).
Inspired by the concept of imitation learning in RL, whichclones the behavior of positive demonstrations [30], we propose to leverage negative samples topenalize the LM for imitating the mismatched target (i.e., value-aligned but incoherent).
We train anadversarial LM only on the negative demonstrations y − , so that following its policy π A DV .
t will lead toincoherent generations.
The t -th step objective of AIL to be maximized is: [Equation]
[Equation]
(cid:205)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:209)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:207) unlikelihood [Table]
[Equation]
where π ∗ t is the desired refinement policy (a vector initialized from the original π t )
[Table] α is the balancingfactor [Table] and the KL penalty term KL ( π t ∣∣ π ∗ t ) with the coefficient β is the trust region constraint, whichprevents the updated policy from drifting too far away from the original one [60, 59] 4 .
The intuitionbehind such a design is to maximize the unlikelihood of forming the trajectory τ = { s 1 , a 1 , ..., s t , a t }that can be induced by the adversarial policy π A DV .
, weighted against the balancing likelihood term [67].
After refinement, the learned policy π ∗ t can generate tokens unlike those that can beproduced by π A DV . , which will form sequences more coherent to the context.
Value Modeling (VM).
In addition to AIL, which aligns values by learning from negative demonstrations, we present another refinement method that directly learns a value function.
To this end, wetrain a binary LM-based classifier f on the mixture of positive and negative demonstrations.
We usef to estimate the likelihood of a given generation being coherent with the context, by passing it aconcatenation of the context, source, generated chain-of-edits, and the corresponding generated target.
We take the sigmoid of the log-likelihood predicted by f as the reward r , which is r = σ log f
( x, y ) ,and define the objective to be maximized as:
[Equation]
where the t -th step r is adjusted by an importance-sampling ratio between the current and originalpolicy for off-policy stability [45] 5 .
We also deliberately add an entropy bonus term H ( ⋅ ∣ s t ) ∼ π ∗of the refined policy, discounted by λ , to encourage more exploration of the current policy [21] 6 .
Compared with AIL, VM leverages an explicit value estimation module f as the guidance, ratherthan implicitly learning from imitation, which brings extra benefits in generalization across differenthuman values (detailed in §4.3).
Experiments
4.1 Experimental Setting
We study the value alignment performance of S ECOND T HOUGHTS on three benchmark datasets:
Moral Stories.
The Moral Stories dataset ( N = 20 , 000 ) examines whether LMs can generate moralresponses under diverse social situations [15].
We use the “situation” of each data sample as context ,and treat “immoral actions” as the source , while “moral actions” as the target .
MIC.
The MIC dataset ( N = 38 , 000 ) studies whether chatbots can generate utterances that arealigned with a set of “Rules of Thumb (RoT)” of morality [70].
Each sample is labeled with itsalignment level (e.g., “aligned”, “unaligned”, “neither”), RoT violation severity (from 1 to 5), RoTagreement, etc.
We take the question in the dialogue as the context , and the unaligned answers (with
RoT violation severity 4-horrible or 5-worse) as the source , and aligned answers as the target .
ETHICS-Deontology.
The ETHICS dataset ( N = 25 , 356 ) investigates the performance of LMs onfive human values alignment tasks [22].
We pick the deontology split because of its contextual nature.
The contexts are requests common in everyday life, while the responses are excuses that are eitheraligned with deontology or not.
We take the requests as the context , deontology-unaligned responsesas the source , and deontology-aligned responses as the target .
We also consider two smaller-scale human values alignment datasets: HHH (Helpful, Honest, &
Harmless)
[1] ( N = 178 ) and Truthful QA [34] ( N = 299 ), to evaluate the domain transfer ability.
We use the official train/validate/test splits in the above datasets.
As the pre-processing step, weremoved hashtags and urls in the text, but leave punctuation and stop words.
Besides the generative
LM (GPT-2 medium) we use throughout the paper, we train three RoBERTa-large classifiers [38] onthe mixture of positive and negative demonstrations on the above three datasets, achieving F1 scoresof {99.7, 91.0, 91.9}, respectively.
They are used as f in the VM mode of S ECOND T HOUGHTS .
Werun experiments on four NVIDIA A6000 GPUs, which take around {3h, 2.4h, 1.3h} for three tasks.
We calculate
( ⋅
= − ∑ π
( a t ∣ s t ) log π t ( a t ∣ s t ) discuss how proper
[Table]
We conducted two sessions of human evaluation on Amazon Mechanical Turk (MTurk).
The firstsession was to validate the quality of S ECOND T HOUGHTS re-alignment, and the second sessionto evaluate cases where corrective edits were made by humans to the DP-generated chain-of-editsto improve alignment or coherence.
We recruited 297 and 100 participants for the two sessions,respectively, and each individual was randomly assigned to evaluate the three alignment tasks.
Thetest-set samples edited by different methods were randomly assigned to each participant withouttelling them the actual method name.
Each participant was paid 1 dollar for completing 20 questionsfor session one (§4.2), and 0.75 dollars for 15 questions for session two (§4.4).
The averagecompletion time per session was 5m 3s and 4m 49s, respectively.
The demographic information anddetailed setup procedure can be found in §A.5.
4.2 Main Results on the Performance of Value Alignment
Alignment methods should be able to guide text generation towards being more value-aligned, whilenot compromising the texts’ coherence with the given context.
Considering the human nature ofvalue judgement, we conduct extensive human evaluations to measure:
Alignment , by asking “To what extent does the edited response improve the original response interms of alignment with human values?”
Answers range from 1- not at all.
to 7- to an extreme extent.
This measures the alignment improvement after the response is edited.
Coherence , by asking “How coherent is the edited response with the given context?”
Answers rangefrom
1- not at all.
to 7- extremely coherent.
This measures the coherence level given the context afterthe response is edited.
Besides human evaluations, we also report evaluation results by automated metrics such as perplexityand ROUGE-L [33], and their correlation with human judgements (see §A.6).
In Table 1 we show the comparison between S ECOND T HOUGHTS and seven other alignment methodsthat do not require extra human labeling on the benchmark datasets: (1) MLE fine-tunes with allthe data in the alignment datasets, simulating common LM pre-training (2) Data Filtering [20]only fine-tunes with the value-aligned split of the data (3) Safe Beam Search [58] blocks a listof sensitive tokens that can lead to misalignment in human values during beam search decoding
(4) PPLM [11] steers the generation via soft probability constraints from Bag-of-Words insteadof hard blocking on tokens 8 (5) DExperts [36] calibrates token distribution by referring to two LMs trained on solely aligned and unaligned data.
We also consider two huge LM-based APIservices to explore whether scaling can make gains for human value alignment: (6) GPT-3 [5](175B) is a general-purpose foundation model [3] which shows strong zero-shot performance in manytasks, and (7) InstructGPT [48], which fine-tunes GPT-3 (1.3B) on human-crafted prompts with adivergence controlled PPO algorithm [60] named PPO-ptx, which is our closest competitor.
Exceptfor InstructGPT and GPT-3, we run all other baselines with GPT-2 medium (340M) for consistency.
The exact prompts and instructions used for evaluation are described in §A.2.
Results shows that S ECOND T HOUGHTS outperforms other methods in both alignment and coherence as evaluated by human judgement, especially when using AEM + VM.
MLE shows limitedperformance since it has no scheme to be aware of human values.
Data Filtering shows a smallimprovement over MLE as it clones the aligned data behavior, but is still limited when the contextalready includes unaligned content.
Token-constrained decoding methods such as Safe Beam Searchand PPLM struggle with value alignment presumably because the abstract human values cannotbe easily modeled by a set of tokens.
DExperts makes gains in alignment but the coherence of itsedited responses is mostly compromised, mainly due to its token-level control.
Compared with AEM+ AIL, AEM + VM has superior performance in most cases; one interpretation could be that thevalue modeling provides better generalization ability, while simply imitating the aligned data canlead to accumulated off-track errors in unseen contexts [9].
Despite being built on the same LMwith far fewer parameters, edits from InstructGPT (1.3B GPT-3) are rated consistently higher thanthose from vanilla GPT-3 (175B) 9 .
Moreover, S ECOND T HOUGHTS further outperforms InstructGPTsignificantly according to one-way analysis of variance (ANOVA) post-hoc pairwise comparisons( p < 0.05) when refined with an RL stage (+ VM or + AIL).
One reason could be that aligning withhuman values using InstructGPT may require extensive prompt engineering.
In general, we concludethat proper value judgement cannot be simply achieved by enlarged model capacity [23], and smaller
LMs trained with properly decomposed demonstrations can often lead to better alignment results.
4.3 Value Transfer Learning with Limited Human-Labeled Data
Since data labeled with human values is rather costly and scarce, we explore whether the alignmentlearned on one value-alignment task can be transferred to another, aiming to investigate the generalization ability of S ECOND T HOUGHTS on unseen values.
We first train our model on the threebenchmark datasets (MRL, MIC, and ETC), recording checkpoints periodically, and then we evaluatethese checkpoints on two new value alignment datasets (TQA and HHH).
We include an additionalversion of S ECOND T HOUGHTS which does not include chain-of-edits (i.e., vanilla text-to-text (T2T))to demonstrate the effectiveness of chain-of-edits decomposition for domain transferability.
The results are shown in Figure 3, where the two rows reflect the results on two new datasets, while thethree columns correspond to the LMs trained on three benchmark datasets.
For the TQA dataset, wefind that after about 0.25 epochs, S ECOND T HOUGHTS trained on MRL and MIC with RL refinement(AEM + VM/IL) can outperform InstructGPT, which demonstrates the effectiveness of RL refinement.
We have a similar observation in the HHH dataset.
However, training on ETC does not seem to bringmuch benefit to the value alignment on HHH.
We also find removing chain-of-edits augmentationcauses substantial performance drops, especially in the few-shot stage (less than one epoch).
Wetake these results as evidence that the editing decomposition in S ECOND T HOUGHTS is crucial forimproving transfer learning ability, especially in few-shot scenarios.
4.4 Error Analysis and Human-Guided Correction
[Figure]
[Table]
We analyze cases where the edited responses received low alignment or coherence scores in the testset of the three tasks, and exemplify these errors and how we correct them with S ECOND T HOUGHTSin §A.11.
Most existing alignment methods can barely correct errors after being trained as they haveno scheme for receiving additional human guidance.
Huge LMs based API services (e.g., GPT-
and
InstructGPT) can potentially fix their own errors by re-prompting (with prompts defined in §A.2),but finding a proper prompt requires tedious prompt engineering.
Different from all these methods,
S ECOND T HOUGHTS allows humans to make changes on the chain-of-edits.
S ECOND T HOUGHTSwill complete the chain and generate the desired target while taking the human changes into consideration.
Note that these changes can be as small as a single word (e.g., see Table A10).
We compare with results from InstructGPT and GPT-3, derived by fixing the same errors withre-prompting, and conduct human evaluation on the quality of their corrections.
As shown in Table 2,
S ECOND T HOUGHTS makes clear advances in terms of alignment and coherence after human-guidedcorrection, potentially because it enables more directed corrections via the chain-of-edits.
We alsofind that the instruction-fine-tuned InstructGPT can better adopt correction instructions than vanilla
GPT-3, despite having over 100x fewer parameters.
4.5 Configuration for the Best Performing S ECOND T HOUGHTS
We also study the impact of the balancing factor ( α ) in AIL and the entropy factor ( λ ) in VM onthe performance of S ECOND T HOUGHTS .
As shown in Figure 4 (a) and (b), for the example task
Moral Stories, we find that in general a higher α will worsen ROUGE-L but improve perplexity (i.e.,
[Figure]
lowers it), as it decreases the effect of unlikelihood training on negative samples in AIL.
Throughempirical observation, we set α to be 0.2 for an appropriate balance, considering the trade-offbetween alignment (ROUGE-L) and fluency (Perplexity).
A similar trade-off can be seen for λ in
VM (set to λ = 0 . 6 ).
In Figure 4 (c), we show the benefits of the augmentation of chain-of-edits: weaugment the training data by the augmentation factor, which is a multiple of the size of the originaltraining data, using different editing costs, as described in §3.2.
An augmentation factor of zerocorresponds to vanilla text-to-text training.
We find that more augmentation does not always lead tobetter performance in the test set, where the best augmentation factor is 2 for AIL and 3 for VM.
Limitations and Discussion
[Figure]
T HOUGHTS can be limited by the LM that it is based on—for instance, the total length ofthe chain-of-edits is limited by the max sequence length allowed for the LM.
Furthermore, studiesfrom social sciences have shown that human values may change over time [51, 49], meaning that
S ECOND T HOUGHTS has to be re-trained with new human demonstrations as values evolve.
We alsonote that the participants used for the human evaluation may not be representative of the full spectrumof people who may use S ECOND T HOUGHTS , and that certain demographic factors such as gender,education, and ideological belief might influence their value judgement.
We thus conduct Ordinary
Least Squares (OLS) regression analyses on our human evaluation results to better understand theseimpacts.
Among other factors, the results indicate that the political party and the perceived importanceof human values are two significant factors that have impact on value judgements (detailed in §A.8).
Conclusion
We have proposed S ECOND T HOUGHTS , a novel learning paradigm that enables LMs to re-alignwith human values when given a poisoned context.
Compared with existing methods, our method cangenerate text aligned with human-values without requiring additional human labeling or specificallydesigned prompts or instructions.
In addition, the chain-of-edits modeling by S ECOND T HOUGHTSenables easy error diagnosis and human-guided correction, which we believe to be an essential abilityfor human-AI interactive systems.
Ethics, Broader Impact, and Reproducibility
As large-scale pre-trained LMs become integrated in more systems, it is a matter of utmost societalimportance to make sure that such models adhere to shared human values.
Here, we present alight-weight framework that can align the generation of LMs with such values, without requiringnew data or extensive prompt-engineering.
Though we do not foresee any major ethical issueswith our proposed work, the reliance on manually annotated datasets and human evaluations mayunintentionally introduce bias in our models (as discussed in Section 5).
To aid reproducibility, wehave included all important information regarding hyperparameters and hardware in this paper andhave included
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes]
• Did you include the license to the code and datasets?
[No]
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] See Abstract & Section §1.
(b) Did you describe the limitations of your work?
[Yes] See Section §5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Ethics
Section on page 9.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
[Yes] See Sections §4.1 and §4.5.
(c) Did you report error bars (e.g., with respect to the random seed after running ex-
periments multiple times)?
[Yes] See Section §4, specifically Tables 1 and 2, and
Figure 3.
(d) Did you include the total amount of compute and the type of resources used (e g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section §4.1.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section §4.1.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
[Figure]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Figure]
using/curating?
[N/A]
[Figure]
(e) Did you discuss whether the data you are using/curating contains personally identifiable [Figure]
information or offensive content?
[N/A]
[Figure]
If you used crowdsourcing or conducted research with human subjects...
[Figure]
(a) Did you include the full text of instructions given to participants and screenshots, if [Figure] applicable?
[Yes] See Sections §A.5 and §4. [Figure]
(b) Did you describe any potential participant risks, with links to Institutional Review [Figure]
Board (IRB) approvals, if applicable?
[N/A]
The project was deemed to be exempt by
[Figure]
our IRB.
[Figure] (c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[Yes] See Sections §4 and §A.5.
[Figure]
A.1 Detailed Re-alignment Task Formulation and Training Setup
[Figure]
In Figure A1, we show the procedure for converting the data samples in the alignment datasetsinto training data of AEM (negative samples used in AIL are generated similarly).
In DP-inferredchain-of-edits (CoEs), we use a few special tokens to mark the editing operations (with their positionand content).
Then our decipher module will translate these special tokens into natural language.
As the final step, we add a special token [SEP] between Context + Source and the ground truth
Chain-of-Edits (CoEs) and Target, as a boundary signal similar to the settings in text-to-text training.
During inference, we input a certain Context + Source, and the LM trained by S ECOND T HOUGHTScan generate CoEs and the corresponding Target.
We also augment the data by using different sets ofcosts for the editing operations (as discussed in Section 3.2, and footnote 3).
For example, we caninfer another chain-of-edits if we change the cost of adding from 1 to 3 (i.e., we discourage addingnew words for alignment), and thus the same Source-Target pair can have multiple chain-of-edits tobe inserted in the middle.
For AEM, we fine-tune the LM with the above-mentioned Source-CoE-Target data (as shownin Figure A1, “Input for AEM”) with the common language modeling objective, which is tomaximize the probability of generating ground truth tokens at each decoding step.
Assuming y ∗ 1 ∶ T = { y ∗ 1 , y ∗ 2 , ..., y ∗ T } is the ground truth output sequence for a given context x Context + Input , the
MLE objective minimizes the following loss by updating the parameter θ in the language model:
[Equation] We train with three epochs for each task by default but set an early-stopping condition when theevaluation loss does not decrease (i.e., plateaus) for five intermediate evaluation steps. The finalperplexity obtained by AEM fine-tuning is {3.831, 4.1, 2.731} after {6000, 6740, 6720} steps, andthe corresponding evaluation loss is {1.346, 1.411, 1.005} on the Moral Stories, MIC, and ETHICSDeontology tasks, respectively. After AEM fine-tuning, the model is capable of generating CoEand its corresponding edited response but still suffers from incoherent responses (see Table A2 formore examples).
We further improve the coherence of the response via reinforcement-learning-basedrefinement, as we have detailed in Section 3.3.
[Table]
.2 Prompts used for Evaluation
[Table] [Table] Table A3 shows the prompts used for evaluations (both main results and human-guided correction).
The phrases used to trigger value alignment are borrowed from the original paper of the datasets (e.g.,“ ... align with morality ” for Moral Stories), in order to make sure the value triggered by a prompt isdesired.
We do small in-house prompt engineering to make sure the generations of the models are atleast readable.
We purposefully only perform slight prompt engineering because we want to imitatereal-world use cases —most users will not put much effort, or will be unable to engineer the idealprompt that can perfectly trigger human values alignment. A.3 Additional Discussion on Edit-based Models
Modeling text edits has been used for other purposes such as sentence fusion and correction [40],improving generation quality [55, 19], text style transfer [41], and more. However, none of theseworks have explored text edits for human value alignment. In this work, we rethink the currentchallenges in value alignment and novelly reformulate the alignment problem as a text editingprocedure. We not only propose a scalable method to infer edits from enormous text data by dynamicprogramming, but also present two RL-based refinement methods to further improve the coherenceof the edited text, which is one of the main technical concerns in previous edit-based models [14, 16].
Compared with existing alignment methods, edit-based S ECOND T HOUGHTS is more intuitive, andits decomposition paradigm is demonstrated to be beneficial when little human labeled data isavailable.
A.4 Qualitative Examples of Error Cases
In Table A2 we show examples of cases where either the Alignment or the Coherence (or both)are low based on human evaluation ratings.
In these cases, we find that without the RL refinement
[Table]
(i
[Table]
e., AEM Only), the generation is likely to be value-aligned but fails to be coherent to the context(e.g., the generation in the Moral Story dataset where the response is generic and is NOT directlyresponding to the context).
In other cases, the generated response can be coherent but likely to not bevalue-aligned (frequently seen in GPT-3 generations), or even neither value-aligned nor coherent.
Forexample, in the ETHICS dataset, the response is incoherent since it does not respond to the requestdirectly (i.e., not owning a TV has nothing to do with helping others move their coach), and it is notaligned with human values (i.e., helping others is not someone’s duty)
[Table]
[Table]
Table A3 shows the prompts used for evaluations (both main results and human-guided correction).
The phrases used to trigger value alignment are borrowed from the original paper of the datasets (e.g.,“ ... align with morality ” for Moral Stories), in order to make sure the value triggered by a prompt isdesired.
We do small in-house prompt engineering to make sure the generations of the models are atleast readable.
We purposefully only perform slight prompt engineering because we want to imitatereal-world use cases —most users will not put much effort, or will be unable to engineer the idealprompt that can perfectly trigger human values alignment.
A.5 Human Evaluation Design
We conducted two human evaluations in spring of 2022.
Participants ( N =397) in both sessions wererecruited using the MTurk Toolkit on CloudResearch, an online participant pool that aggregatesmultiple market research platforms [35].
Participants were all from the United States, and they wererequired to have a HIT approval rate greater than 95% and be over 18 years old.
Each participantwas paid 1 dollar for completing 16 questions in each questionnaire (average completion time perquestionnaire was about 5.07 minutes).
They were properly informed that the collected data wouldbe used for research purposes in the consent form at the beginning.
Demographics.
The average age of the participants in the first session ( N =297) was 42.23 years-old(SD = 12.57, Median=41).
About half (56.2%) of the participants self-reported as male, and 43.8%self-reported as female.
Participants received 16.24 years of education on average (SD = 2.37,
Median = 16).
When asked to self-report their party affiliation, about half of (48.5%) the participantsself-reported as Democratic, 27.9% as Republican, and 23.6% as independent.
The average age of the participants ( N =100) in the second session was 40.65 years-old (SD = 11.05,
Median=39).
About half (54%) of the participants self-reported as male, 45% self-reported as female,and 1% as “other”.
Participants received 15.94 years of education on average (SD = 3.74, Median =16).
When asked to self-report their party affiliation, about half (51%) of the participants self-reportedas Democratic, 30% as Republican, and 19% as independent.
Procedure.
Participants in the first session were randomly assigned into three different conditions toevaluate the three benchmark tasks: Moral Story ( n =99), MIC ( n = 99), and Ethics ( n =99).
Eachparticipant in the second session was randomly assigned equal number of error correction samplesfrom the three datasets.
Figure A2 shows a screenshot of our survey for the task ETHICS: Deontology(the main screen; the other screens are not included because of limited space).
As can be seen, weclearly inform the participants about the theme, the procedure, and content warnings of our study. Wealso present to the annotators the definition of the human value being studied (mainly taken from theoriginal dataset papers). We also provide our definition for “Alignment” and “Coherence” and showcorresponding examples with explanations.
Besides asking about Alignment and Coherence duringthe evaluations, we also asked the participants to rate the Fluency of the generated edits by asking“How fluent is the edited response (e.g., coherent, well-written, without grammar errors)?” Answersrange from 1- not at all.
to 7- extremely fluent.
The participants did not know which model generatedwhich response.
Note that we also designed an attention check to ensure the participants understand what source ortarget responses mean in our study. Only 5 out of the 302 participants failed the attention check andwere excluded in the final data analysis (resulting in N =297 participants finally).
All the participantsin the session two passed this attention check.
A.6 Correlation Between Automated Metrics and Human Judgement
Although we believe that humans should be the only qualified judges for the value alignment task,during the development stage of algorithms we have to leverage fast and cheap automated metrics asa reasonable estimation.
Here, we test the correlation between two automated metrics (ROUGE-Land perplexity (PPL)) and respective human judgements on Alignment and Fluency.
Table A5shows additional results on the three alignment datasets.
Besides the Alignment (Align) score,we also report Fluency score from human evaluation, and two automated metrics ROUGE-L andperplexity as automated alternatives of human scored Alignment and Fluency, respectively.
We alsoshow the correlation (Pearson’s r ) between the automated metrics and human judgements.
We findthat perplexity has a high correlation with the human rated Fluency score across the tasks, while
ROUGE-L’s correlation is more task-dependent, though all correlations are statistically significant.
One interpretation could be that the measurement of text similarity with the ground truth (i.e., what
ROUGE-L measures) is only an approximation of value alignment.
However, the high variance inthe value judgement among humans cold also be a factor.
We have studied the impact from human factors on the Alignment score in §A.8.
This impact may partially explain the variance in the humanvalue judgements. [Table] [Table]
A.7 Additional Results on Other Tasks
In addition to the three main datasets (Moral Stories, MIC, ETHICS, see Section 4.2) for benchmarking and two smaller scale datasets (TQA, HHH, see Section 4.3) for transfer learning evaluations [Table] weconduct additional experiments on another three datasets that focus on moderation of open-domaindialogue systems 10 : MovieDic [2], Cornell IMDB Reviews [10]
[Table]
and DSTC8 Reddit 11 .
The threedatasets have a similar structure to the alignment datasets, each sample of which has a context, a valueunaligned response (e.g., including hateful speech), and a value-aligned response (e.g., the moderatedresponse).
The performance of S ECOND T HOUGHTS on these datasets is shown in Table A4.
In general, we find S ECOND T HOUGHTS alignment can bring consistent gains as seen in other tasks,especially for the Movie Dic and Cornell IMDB datasets.
{+For more chit-chat like dataset (i.e., DSTC
+}
Reddit), we believe using larger-scale models as the base LM might be helpful, since its largercapacity makes it more capable of generating diverse responses.
In Section A.9 we have demonstratedthe scalability of S ECOND T HOUGHTS on a large LM, the 68B checkpoint of PaLM [7].
[Table] A.8 Ordinary Least Squares (OLS) Regression Analyses Ordinary least squares (OLS) regression (shown in Table A6) analyses show that for both AEM +
AIL and AEM + VM, party affiliation (which was measured on a 7-point scale where 1 indicates
Democrat, 4 as Moderate, and 7 as Republican) is negatively associated with alignment values (AEM+ AIL: B =-.12, SE = .05, p = .01; AEM + VM: B =-.16, SE
= .05, p < .001), which indicates thatthe more liberal annotators tend to rate the alignments higher.
This can be possibly explained by: 1)liberal users may be more familiar with such ML tasks and thus give our methods high alignmentscores; or 2) it is also possible that conservative users are more skeptical of human-value alignmenton such tasks.
Another significant predictor is the people’s perceived importance of alignment withhuman values (measured by answering the question “ Whether or not the algorithm-generated textaligns with shared human values is important to me ” on a 7-point scale).
The more important peoplethink alignment with human values is, the higher alignment scores they give for both methods.
A.9 Scalability of S ECOND T HOUGHTS
To validate the scalability of S ECOND T HOUGHTS , we further test our method on larger LM checkpoints: OpenAI’s GPT2-XL (1.3B) and Google’s PaLM (8B and 62B). GPT2-XL is the largestpublicly available checkpoint that we can load into our A6000 GPU (48GB) clusters, which is acommon setting for academic lab environments. We also have tried S ECOND T HOUGHTS on the8B and 62B checkpoints of Google’s Pathway Language Models [7]. The total time cost for theexperiments on the three tasks are approximately {10h, 8h, 4.5h} for GPT2-XL on four A6000s,{3h, 2h, 1.5h} for 8B PaLM, and {6h, 4h, 3.5h} for 62B PaLM on 512 TPUs (v3).
Not surprisingly,we find that industry-level infrastructure is more capable of fine-tuning larger models in relativelyshort time, because of better parallel and distributed computing.
These experiments demonstrate that
S ECOND T HOUGHTS can be smoothly adapted to larger-scale models.
As shown in Table A7, S ECOND T HOUGHTS can bring consistent gains to larger models in boththe AEM + VM and AEM + AIL methods.
AEM + VM is still slightly better than AEM + AIL,but such difference becomes marginal when the model size becomes larger.
We also find, besidesthe performance gain in alignment (in terms of ROUGE-L and R-L), the larger models also benefitthe fluency of generated responses: the perplexity (PPL) decreases as the model becomes larger.
Inconclusion, since S ECOND T HOUGHTS does not change the architecture of the base LM but onlyre-formats the fine-tuning data (with inferred chain-of-edits) and changes the training objective, webelieve large language model developers (or companies) can easily apply S ECOND T HOUGHTS as anadditional step after pre-training to make their models better aligned with human values.
[Table]
A.10 Error Analysis
We find the errors of S ECOND T HOUGHTS can often be categorized into one of three types : (1)
Detour errors (Table A8), where the liability is passed on to someone else, (2) Sidestep errors(Table A9), where the generated targets do not directly respond to the situated context [Table] and (3)
Distortion errors (Table A10), where the edited responses are not directly related to the situation(e.g. , an unrelated excuse is given).
In these cases, human annotators tend to assign either loweralignment or coherence scores (or both).
In Tables A8, A9, and A10, we show an example of such errors and show how the human-guidedcorrection is applied to these errors cases (Error Target). After the human annotators see the ST
Proposed Edits (that leads to Error Target), they are allowed to make changes on the chain (as shownin blue in the tables). S ECOND T HOUGHTS can take this changed chain (with context and source)and complete it (as shown in brown in the tables) with the newly generated target (New Target). A.11 Data Statistics of our Value Transfer Learning Experiments In Table A11, we show the actual number of shots for each dataset when the fraction of epochs rangesfrom 0.25 to 1.0 (with 1.0 being the full training set), based on Figure 3.
[Table] [Table] [Table]
[Table]