Second Thoughts are Best: Learning to Re-Align With Human Values from Text Edits
Anonymous Author(s)
AffiliationAddress email
Abstract
We present S ECOND T HOUGHTS , a new learning paradigm that enables languagemodels (LMs) to re -align with human values.
By modeling the chain-of-editsbetween value-unaligned and value-aligned text, with LM fine-tuning and additional refinement through reinforcement learning, S ECOND T HOUGHTS not onlyachieves superior performance in three value alignment benchmark datasets butalso shows strong human-value transfer learning ability in few-shot scenarios.
Thegenerated editing steps also offer better interpretability and ease for interactiveerror correction.
Extensive human evaluations further confirm its effectiveness
Introduction
“ Machines can and will make better decisions than humans [Figure] but only when the values are aligned with those of human race. ”
[Figure]
Stuart Russell, Value Alignment ,
[Figure]
Sallyasksaboutthebraceletandrequestsitforherupcomingbirthday
[Figure]
Current large-scale pre-trained language models (LMs) have shown great success in manyknowledge-recalling tasks, such as question answering [53] and entity retrieval [9]; however,their ability to select socially good text from bad(or generating prosocial text) in open-world settings is still limited [16], even when the modelsare scaled up to hundreds of billions of parameters [28].
In other words, pre-training ever-larger
LMs does not lead to expected substantive gainsin tasks that require human value judgment [18].
Consider the example in Figure 1: given a context, a fine-tuned LM GPT-2 [45] assigns a largerprobability mass 1 to the immoral option than tothe moral ground truth.
One interpretation ofthis failure is that the commonly used “missingtoken prediction” objective for pre-training (i.e., MLE) does not directly model human values [40]
As a consequence, fine-tuned LMs still struggle with options that are legitimate semantically (i.e.,low language modeling loss) but are not aligned with human values.
To tackle this misalignment problem, prior work has proposed using binary answers [21, 48], rankings [12, 3], or ratings [61, 33] to model human value preferences.
For example, Askell et al.
[1]create a platform to collect Likert-scale human ratings on LM-generated utterances in dialogues,aiming to teach the LM to be helpful, honest, and harmless.
However, without considering how torecover from responses that already violate human values, these methods cannot serve as robustremedies in real-world applications, since they can be easily attacked by poisoned queries [13].
More recent attempts, such as InstructGPT [40], formulate the alignment problem as about teachingthe machine to follow human instructions—they fine-tune GPT-3 on a variety of prompts writtenby human users of OpenAI’s GPT-3 API [4].
Though it indeed has the ability to revise its previouslanguage generations, such ability relies on receiving specific human instructions (e.g., “ Please makethe following sentence aligned with moral values. ”).
Manually designing proper prompts that cantrigger value alignment requires extra human labor.
Besides, specifically-designed prompts do notalways exist in real-world human-AI interaction, and we cannot expect most users to know how todesign appropriate prompts to improve the human-value alignment of an AI agent [26].
On the other hand, rather than steering the language generation with artificial prompts, humans caneasily fix immoral language by making hierarchical and recursive edits [10, 25], where human valuejudgments serve as the guide for each edit.
Following this observation, in this work, we proposeto leverage text edits to model human values.
Our method, called S ECOND T HOUGHTS , echoes thetheory of “utilitarian ethics”, which says that humans choose the actions (e.g. edits) which maximizethe perceived positive impact on the most people [54, 44].
Specifically, we model human edits bythree generic operations: insert, delete, and replace, and automatically infer the “chain-of-edits” by adynamic programming algorithm.
Besides the commonly used MLE training, we deliberately includea reinforcement learning based refinement step, to further encourage valid edits which are not onlyaligned with human values, but also coherent with the context.
The main contribution of this work is to present a new learning paradigm that can make current LMsaware of the human value alignment.
Trained with S ECOND T HOUGHTS , LMs can not only re -aligntheir generation with human values, even when the context has already been poisoned, but also showthe chain of editing steps for ease of interpretability and to facilitate further edits (§4.4).
Throughextensive human evaluation, we find that the edited responses by S ECOND T HOUGHTS (based on a345M GPT-2) are on average scored higher with respect to their value alignment than those from
InstructGPT (based on a 1.3B GPT-3) (§4.2).
Our experiments confirm that simply scaling LMs is notadequate for good alignment with human values, which echoes the findings of recent studies [42, 28].
Instead, smaller LMs trained with a few properly decomposed human demonstrations can often leadto better results (§4.3).
We also provide a discussion on the impact of human factors during humanevaluation (§5), which is crucially ignored in current AI studies.
Related Work
We briefly review existing work that considers in-context explanations during prompting or training.
We also summarize other value alignment methods for language models.
Learning From In-Context Instructions.
The few-shot performance of LMs can be enhanced bylearning from in-context instructions [47, 31], in the forms of task descriptions [35, 46], answerdemonstrations [4], targeting formats [34], etc., which can be positioned before [56] or even after [23]the answer.
Recent studies have shown improved results by including decomposed reasoning stepsinto the instructions [39, 38].
However, the instructions normally require careful human design, whichis costly and whose quality greatly affects performance [59, 19].
In comparison with these methods,
S ECOND T HOUGHTS learns from text edits inferred by an algorithm, and presents the chain-of-editsfor each alignment, which eases error diagnosis and enables interactive correction.
Human Value Alignment for Language Models.
Trained on unfiltered and problematic languagefrom the web, current large-scale LMs have be shown to be poorly aligned with human values [2].
For example, GPT-3 performs only marginally better than a random baseline on a virtue matching task [57], and scaling-up LMs can even lead to deterioration in truthfulness [28].
Existinggeneral-purpose remedies include filtering the training data [14], attribute-control generation [8],and modifying the decoding algorithm with hard (e.g., token blocklists; Schick et al.
[49]) or softconstraints (e.g., reference LMs; Liu et al.
Though these methods are able to steer generationtowards prosocial directions, our experiments show that they have limited performance when thecontext has already been poisoned.
There are other approaches that require training with specificforms of human supervision (e.g., fine-grained ratings) [40, 52, 60, 6], but these are often costly andnot always available in every value alignment dataset.
S ECOND T HOUGHTS differs from all thesemethods in its offline nature and ability to re -align in poisoned contexts, requiring neither extra humanlabeling nor specially-designed prompts or instructions.
Approach
S ECOND T HOUGHTS comprises two main steps.
We first infer chain-of-edits automatically fromsource and target responses with a dynamic programming algorithm, and fine-tune an LM on the editsaugmented training data (§3.2).
Then, we deploy a reinforce learning stage to refine the generation, byeither adversarial imitation learning or value modeling (§3.3).
We begin by introducing the problemof value re-alignment (§3.1).
3.1 Problem Statement of Re-alignment
[Figure]
learns to re-align decomposed chain-of-
Value alignment datasets normally consist ofcontexts (i.e., social situations), value-alignedresponses (i.e., prosocial behaviors), and valueunaligned responses (i.e., antisocial behaviors).
Existing alignment methods formulate the value alignment task as a conditional generation problem: given a situation as the context , train amodel that can generate responses resemblinga value-aligned target rather than a not-alignedwrong target (Figure 2 (a)).
However, many studies have shown that LMs trained with such a paradigm can be easily derailed by poisoned con- texts [40, 13]—i.e., contexts that already include value-unaligned content, either from the model’s own generation or from malicious users 2 .
In other words, unlike humans, these models lack the ability of re -alignment (the ability to recover from poisoned contexts).
To teach a model how to re-align, we deliberately add the value-unaligned response into the context,referred to as the source , and keep the value-aligned response as the target .
The intuition behind thisis that instead of learning from mistakes after a misalignment occurs in the generation, the model learns how to make edits as it is generating the text.
Specifically, we include the unaligned source aspart of the new “context”, and then train an LM to learn how to make sequential edits on the source toproduce the target (Figure 2 (b)).
This way the model learns how to recover from a value-unaligned,poisoned context during the generation phase.
3.2 Augmented Edits Modeling
DP-based Edits Inference.
Given two text strings, source and target , one can find unlimited ways toedit source to produce target .
Thus, we apply two constraints onto the editing: (1) the edits shouldbe combinations of generic editing operations—inserting, deleting, and replacing a single token;(2) each edit operation has a cost and our goal is to infer the chain-of-edits that has minimum cost.
Under these constraints, the edits inference problem can be converted to a token-level “edit distanceproblem” [22], which can be solved by dynamic programming (DP).
We modify the algorithm to beable to receive customized editing costs (e.g., insert-1, delete-1, replace-2), to try to model differentpreferences on editing.
We use special tokens to mark the start/end of editing and the new content tobe inserted/replaced, and develop a decipher module that can translate the edit operations producedby DP into natural language (see §A.1 for a visualization of the whole process).
Augmented Edits Modeling (AEM).
To augment the edits, we run the DP algorithm on the samesource and target pairs with a variety of editing costs 3 to create a collection of chain-of-edits for eachsource - target pair, which we call positive demonstrations ( y + ).
We then fine-tune an LM on thesesource - edits - target text inputs (recall that the edits are turned into natural language).
We call this
Augmented Edits Modeling (AEM).
Different from common language modeling, AEM includes thelabor-free decomposition (i.e., the editing steps) into the training object, whereas prior works eithertrain on costly manually-created decomposition [40, 55] or, rather than training, prompt with suchdecomposition [56, 39].
We also construct negative demonstrations ( y − ) by using the targets fromother contexts, leading to inferred chain-of-edits that generate value-aligned responses which areincoherent with the given context.
These will be used during the RL refinement described below.
3.3 Refinement by Reinforcement Learning
Though the generation of an LM trained with AEM can already align well with human values, many ofthe generated responses are not coherent with the given contexts.
Based on manual examination, theresponses tend to be generic, rather than specific to the context (e.g., the sidestep error in Table A5).
We are thus motivated to deploy a reinforcement learning (RL) stage to further refine the generationquality, mainly to improve the coherence to the context.
Notation.
Given the concatenation of context and source as x , S ECOND T HOUGHTS will generatechain-of-edits and corresponding target as y .
In RL language, we define the state at time t as the setof generated tokens before t (i.e., s t =
y < t ), and the action as the current step’s output token (i.e.,a t = y t ).
The softmax output of the language modeling head (a categorical distribution over the entirevocabulary) is considered as the policy π t for picking token y t (action a t ), given the state s t
= y < t .
Adversarial Imitation Learning (AIL).
Inspired by the concept of imitation learning in RL, whichclones the behavior of positive demonstrations [24], we propose to leverage negative samples topenalize the LM for imitating the mismatched target (i.e., value-aligned but incoherent).
We train anadversarial LM only on the negative demonstrations y − , so that following its policy π A DV .
t will lead toincoherent generations.
The t -th step objective of AIL to be maximized is:
[Equation]
unlikelihood + α log π ∗ t ( a t ∣ s t ) (cid:205)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:209)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:210)(cid:207) likelihood
[Equation]
where π ∗ t is the desired refinement policy (a vector initialized from the original π t ), α is the balancingfactor, and the KL penalty term KL ( π t ∣∣ π ∗ t ) with the coefficient β is the trust region constraint, whichprevents the updated policy from drifting too far away from the original one [51, 50] 4 .
The intuitionbehind such a design is to maximize the unlikelihood of forming the trajectory τ = { s 1 , a 1 , ..., s t , a t }that can be induced by the adversarial policy π A DV .
, weighted against the balancing likelihood term [58].
After refinement, the learned policy π ∗ t can generate tokens unlike those that can beproduced by π A DV . , which will form sequences more coherent to the context.
Value Modeling (VM).
In addition to AIL, which aligns values by learning from negative demonstrations, we present another refinement method that directly learns a value function.
To this end, wetrain a binary LM-based classifier f on the mixture of positive and negative demonstrations.
We usef to estimate the likelihood of a given generation being coherent with the context, by passing it aconcatenation of the context, source, generated chain-of-edits, and the corresponding generated target.
We take the sigmoid of the log-likelihood predicted by f as the reward r , which is r = σ log f
( x, y ) ,and define the objective to be maximized as:
[Equation]
where the t -th step r is adjusted by an importance-sampling ratio between the current and originalpolicy for off-policy stability [37] 5 .
We also deliberately add an entropy bonus term H ( ⋅ ∣ s t ) ∼ π ∗of the refined policy, discounted by λ , to encourage more exploration of the current policy [15] 6 .
Compared with AIL, VM leverages an explicit value estimation module f as the guidance, ratherthan implicitly learning from imitation, which brings extra benefits in generalization across differenthuman values (detailed in §4.3).
Experiments
4.1 Experimental Setting
We study the value alignment performance of S ECOND T HOUGHTS on three benchmark datasets:
Moral Stories.
The Moral Stories dataset ( N = 20 , 000 ) examines whether LMs can generate moralresponses under diverse social situations [11].
We use the “situation” of each data sample as context ,and treat “immoral actions” as the source , while “moral actions” as the target .
MIC.
The MIC dataset ( N = 38 , 000 ) studies whether chatbots can generate utterances that arealigned with a set of “Rules of Thumb (RoT)” of morality [61].
Each sample is labeled with itsalignment level (e.g., “aligned”, “unaligned”, “neither”), RoT violation severity (from 1 to 5), RoTagreement, etc.
We take the question in the dialogue as the context , and the unaligned answers (with
RoT violation severity 4-horrible or 5-worse) as the source , and aligned answers as the target .
ETHICS-Deontology.
The ETHICS dataset ( N = 25 , 356 ) investigates the performance of LMs onfive human values alignment tasks [16].
We pick the deontology split because of its contextual nature.
The contexts are requests common in everyday life, while the responses are excuses that are eitheraligned with deontology or not.
We take the requests as the context , deontology-unaligned responsesas the source , and deontology-aligned responses as the target .
We also consider two smaller-scale human values alignment datasets: HHH (Helpful, Honest, &
Harmless)
[1] ( N = 178 ) and Trustful QA [28] ( N = 299 ), to evaluate the domain transfer ability.
We use the official train/validate/test splits in the above datasets.
As the pre-processing step, weremoved hashtags and urls in the text, but leave punctuation and stop words.
Besides the generative
LM (GPT-2 medium) we use throughout the paper, we train three RoBERTa-large classifiers [32] onthe mixture of positive and negative demonstrations on the above three datasets, achieving F1 scoresof {99.7, 91.0, 91.9}, respectively.
They are used as f in the VM mode of S ECOND T HOUGHTS .
Werun experiments on four NVIDIA A6000 GPUs, which take around {3h, 2.4h, 1.3h} for three tasks.
We calculate
( ⋅
= − ∑ π
( a t ∣ s t ) log π t ( a t ∣ s t ) discuss how proper
[Table]
We conducted two sessions of human evaluation on Amazon Mechanical Turk (MTurk).
The firstsession was to validate the quality of S ECOND T HOUGHTS re-alignment, and the second sessionto evaluate cases where corrective edits were made by humans to the DP-generated chain-of-editsto improve alignment or coherence.
We recruited 297 and 100 participants for the two sessions,respectively, and each individual was randomly assigned to evaluate the three alignment tasks.
Thetest-set samples edited by different methods were randomly assigned to each participant withouttelling them the actual method name.
Each participant was paid 1 dollar for completing 20 questionsfor session one (§4.2), and 0.75 dollars for 15 questions for session two (§4.4).
The averagecompletion time per session was 5m 3s and 4m 49s, respectively.
The demographic information anddetailed setup procedure can be found in §A.3.
4.2 Main Results on the Performance of Value Alignment
Alignment methods should be able to guide text generation towards being more value-aligned, whilenot compromising the texts’ coherence with the given context.
Considering the human nature ofvalue judgement, we conduct extensive human evaluations to measure:
Alignment , by asking “To what extent does the edited response improve the original response interms of alignment with human values?”
Answers range from 1- not at all.
to 7- to an extreme extent.
This measures the alignment improvement after the response is edited.
Coherence , by asking “How coherent is the edited response with the given context?”
Answers rangefrom
1- not at all.
to 7- extremely coherent.
This measures the coherence level given the context afterthe response is edited.
Besides human evaluations, we also report evaluation results by automated metrics such as perplexityand ROUGE-L [27], and their correlation with human judgements (see §A.4).
In Table 1 we show the comparison between S ECOND T HOUGHTS and seven other alignment methodsthat do not require extra human labeling on the benchmark datasets: (1) MLE fine-tunes with allthe data in the alignment datasets, simulating common LM pre-training (2) Data Filtering [14]only fine-tunes with the value-aligned split of the data (3) Safe Beam Search [49] blocks a listof sensitive tokens that can lead to misalignment in human values during beam search decoding
(4) PPLM [8] steers the generation via soft probability constraints from Bag-of-Words instead ofhard blocking on tokens 8 (5) DExperts [30] calibrates token distribution by referring to two LMstrained on solely aligned and unaligned data.
We also consider two huge LM-based API servicesto explore whether scaling can make gains for human value alignment: (6) GPT-3 [4] (175B)is a general-purpose foundation model [2] which shows strong zero-shot performance in manytasks, and (7) InstructGPT [40], which fine-tunes GPT-3 (1.3B) on human-crafted prompts with adivergence controlled PPO algorithm [51] named PPO-ptx, which is our closest competitor.
Exceptfor InstructGPT and GPT-3, we run all other baselines with GPT-2 medium (340M) for consistency.
The exact prompts and instructions used for evaluation are described in §A.2.
Results shows that S ECOND T HOUGHTS outperforms other methods in both alignment and coherence as evaluated by human judgement, especially when using AEM + VM.
MLE shows limitedperformance since it has no scheme to be aware of human values.
Data Filtering shows a smallimprovement over MLE as it clones the aligned data behavior, but is still limited when the contextalready includes unaligned content.
Token-constrained decoding methods such as Safe Beam Searchand PPLM struggle with value alignment presumably because the abstract human values cannotbe easily modeled by a set of tokens.
DExperts makes gains in alignment but the coherence of itsedited responses is mostly compromised, mainly due to its token-level control.
Compared with AEM+ AIL, AEM + VM has superior performance in most cases; one interpretation could be that thevalue modeling provides better generalization ability, while simply imitating the aligned data canlead to accumulated off-track errors in unseen contexts [7].
Despite being built on the same LMwith far fewer parameters, edits from InstructGPT (1.3B GPT-3) are rated consistently higher thanthose from vanilla GPT-3 (175B) 9 .
Moreover, S ECOND T HOUGHTS further outperforms InstructGPTsignificantly according to one-way analysis of variance (ANOVA) post-hoc pairwise comparisons( p < 0.05) when refined with an RL stage (+ VM or + AIL).
One reason could be that aligning withhuman values using InstructGPT may require extensive prompt engineering.
In general, we concludethat proper value judgement cannot be simply achieved by enlarged model capacity [17], and smaller
LMs trained with properly decomposed demonstrations can often lead to better alignment results.
4.3 Value Transfer Learning with Limited Human-Labeled Data
Since data labeled with human values is rather costly and scarce, we explore whether the alignmentlearned on one value-alignment task can be transferred to another, aiming to investigate the generalization ability of S ECOND T HOUGHTS on unseen values.
We first train our model on the threebenchmark datasets (MRL, MIC, and ETC), recording checkpoints periodically, and then we evaluatethese checkpoints on two new value alignment datasets (TQA and HHH).
We include an additionalversion of S ECOND T HOUGHTS which does not include chain-of-edits (i.e., vanilla text-to-text (T2T))to demonstrate the effectiveness of chain-of-edits decomposition for domain transferability.
The results are shown in Figure 3, where the two rows reflect the results on two new datasets, while thethree columns correspond to the LMs trained on three benchmark datasets.
For the TQA dataset, wefind that after about 0.25 epochs, S ECOND T HOUGHTS trained on MRL and MIC with RL refinement(AEM + VM/IL) can outperform InstructGPT, which demonstrates the effectiveness of RL refinement.
We have a similar observation in the HHH dataset.
However, training on ETC does not seem to bringmuch benefit to the value alignment on HHH.
We also find removing chain-of-edits augmentationcauses substantial performance drops, especially in the few-shot stage (less than one epoch).
Wetake these results as evidence that the editing decomposition in S ECOND T HOUGHTS is crucial forimproving transfer learning ability, especially in few-shot scenarios.
4.4 Error Analysis and Human-Guided Correction
[Figure]
[Table]
We analyze cases where the edited responses received low alignment or coherence scores in the testset of the three tasks, and exemplify these errors and how we correct them with S ECOND T HOUGHTSin §A.6.
Most existing alignment methods can barely correct errors after being trained as they haveno scheme for receiving additional human guidance.
Huge LMs based API services (e.g., GPT-
and
InstructGPT) can potentially fix their own errors by re-prompting (with prompts defined in §A.2),but finding a proper prompt requires tedious prompt engineering.
Different from all these methods,
S ECOND T HOUGHTS allows humans to make changes on the chain-of-edits.
S ECOND T HOUGHTSwill complete the chain and generate the desired target while taking the human changes into consideration.
Note that these changes can be as small as a single word (e.g., see Table A6).
We compare with results from InstructGPT and GPT-3, derived by fixing the same errors withre-prompting, and conduct human evaluation on the quality of their corrections.
As shown in Table 2,
S ECOND T HOUGHTS makes clear advances in terms of alignment and coherence after human-guidedcorrection, potentially because it enables more directed corrections via the chain-of-edits.
We alsofind that the instruction-fine-tuned InstructGPT can better adopt correction instructions than vanilla
GPT-3, despite having over 100x fewer parameters.
4.5 Configuration for the Best Performing S ECOND T HOUGHTS
We also study the impact of the balancing factor ( α ) in AIL and the entropy factor ( λ ) in VM onthe performance of S ECOND T HOUGHTS .
As shown in Figure 4 (a) and (b), for the example task
Moral Stories, we find that in general a higher α will worsen ROUGE-L but improve perplexity (i.e.,
[Figure]
lowers it), as it decreases the effect of unlikelihood training on negative samples in AIL.
Throughempirical observation, we set α to be 0.2 for an appropriate balance, considering the trade-offbetween alignment (ROUGE-L) and fluency (Perplexity).
A similar trade-off can be seen for λ in
VM (set to λ = 0 . 6 ).
In Figure 4 (c), we show the benefits of the augmentation of chain-of-edits: weaugment the training data by the augmentation factor, which is a multiple of the size of the originaltraining data, using different editing costs, as described in §3.2.
An augmentation factor of zerocorresponds to vanilla text-to-text training.
We find that more augmentation does not always lead tobetter performance in the test set, where the best augmentation factor is 2 for AIL and 3 for VM.
Limitations and Discussion
[Figure]
T HOUGHTS can be limited by the LM that it is based on—for instance, the total length ofthe chain-of-edits is limited by the max sequence length allowed for the LM.
Furthermore, studiesfrom social sciences have shown that human values may change over time [43, 41], meaning that
S ECOND T HOUGHTS has to be re-trained with new human demonstrations as values evolve.
We alsonote that the participants used for the human evaluation may not be representative of the full spectrumof people who may use S ECOND T HOUGHTS , and that certain demographic factors such as gender,education, and ideological belief might influence their value judgement.
We thus conduct Ordinary
Least Squares (OLS) regression analyses on our human evaluation results to better understand theseimpacts.
Among other factors, the results indicate that the political party and the perceived importanceof human values are two significant factors that have impact on value judgements (detailed in §A.5).
Conclusion
We have proposed S ECOND T HOUGHTS , a novel learning paradigm that enables LMs to re-alignwith human values when given a poisoned context.
Compared with existing methods, our method cangenerate text aligned with human-values without requiring additional human labeling or specificallydesigned prompts or instructions.
In addition, the chain-of-edits modeling by S ECOND T HOUGHTSenables easy error diagnosis and human-guided correction, which we believe to be an essential abilityfor human-AI interactive systems.
Ethics, Broader Impact, and Reproducibility
As large-scale pre-trained LMs become integrated in more systems, it is a matter of utmost societalimportance to make sure that such models adhere to shared human values.
Here, we present alight-weight framework that can align the generation of LMs with such values, without requiringnew data or extensive prompt-engineering.
Though we do not foresee any major ethical issueswith our proposed work, the reliance on manually annotated datasets and human evaluations mayunintentionally introduce bias in our models (as discussed in Section 5).
To aid reproducibility, wehave included all important information regarding hyperparameters and hardware in this paper andhave included data, code, and reports from the human evaluation in the supplementary materials toaid reviewing
We plan to release our code and data after publication under an MIT license.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes]
• Did you include the license to the code and datasets?
[No]
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] See Abstract & Section §1.
(b) Did you describe the limitations of your work?
[Yes] See Section §5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Ethics
Section on page 9.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Sections §4.1 and §4.5.
(c) Did you report error bars (e.g., with respect to the random seed after running ex-
periments multiple times)?
[Yes] See Section §4, specifically Tables 1 and 2, and
Figure 3.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section §4.1.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section §4.1.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[N/A]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[Yes] See Sections §A.3 and §4.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
The project was deemed to be exempt by
our IRB.
(c) Did you include the estimated hourly wage paid to participants and the total amount [Table]
spent on participant compensation?
[Yes] See Sections §4 and §A.3.
[Figure]
A.1 Detailed Re-alignment Task Formulation
[Figure]
[Figure] [Table] [Figure] In Figure A1, we show the procedure for converting the data samples in the alignment datasetsinto training data of AEM (negative samples used in AIL are generated similarly).
In DP-inferredchain-of-edits (CoEs), we use a few special tokens to mark the editing operations (with their positionand content).
Then our decipher module will translate these special tokens into natural language.
As the final step, we add a special token [SEP] between Context + Source and the ground truth
Chain-of-Edits (CoEs) and Target, as a boundary signal similar to the settings in text-to-text training.
During inference, we input a certain Context + Source, and the LM trained by S ECOND T HOUGHTScan generate CoEs and the corresponding Target.
A.2 Prompts used for Evaluation
Table A1 shows the prompts used for evaluations (both main results and human-guided correction).
The phrases used to trigger value alignment are borrowed from the original paper of the datasets (e.g.,“ ... align with morality ” for Moral Stories), in order to make sure the value triggered by a prompt is desired.
We do small in-house prompt engineering to make sure the generations of the models are atleast readable.
We purposefully only perform slight prompt engineering because we want to imitatereal-world use cases —most users will not put much effort, or will be unable to engineer the idealprompt that can perfectly trigger human values alignment.
[Table]
[Table]
A.3 Human Evaluation Design
We conducted two human evaluations in spring of 2022.
Participants ( N =397) in both sessions wererecruited using the MTurk Toolkit on CloudResearch, an online participant pool that aggregatesmultiple market research platforms [29].
Participants were all from the United States, and they wererequired to have a HIT approval rate greater than 95% and be over 18 years old.
Each participantwas paid 1 dollar for completing 16 questions in each questionnaire (average completion time perquestionnaire was about 5.07 minutes).
They were properly informed that the collected data wouldbe used for research purposes in the consent form at the beginning.
Demographics.
The average age of the participants in the first session ( N =297) was 42.23 years-old(SD = 12.57, Median=41).
About half (56.2%) of the participants self-reported as male, and 43.8%self-reported as female.
Participants received 16.24 years of education on average (SD = 2.37,
Median = 16).
When asked to self-report their party affiliation, about half of (48.5%) the participantsself-reported as Democratic, 27.9% as Republican, and 23.6% as independent.
The average age of the participants ( N =100) in the second session was 40.65 years-old (SD = 11.05,
Median=39).
About half (54%) of the participants self-reported as male, 45% self-reported as female,and 1% as “other”.
Participants received 15.94 years of education on average (SD = 3.74, Median =16).
When asked to self-report their party affiliation, about half (51%) of the participants self-reportedas Democratic, 30% as Republican, and 19% as independent.
Procedure.
Participants in the first session were randomly assigned into three different conditionsto evaluate the three benchmark tasks: Moral Story ( n =99), MIC ( n = 99), and Ethics ( n =99).
Each participant in the second session was randomly assigned equal number of error correctionsamples from the three datasets.
In the instruction, we specified that “source response” means theresponse does not align with human values whereas “target response” generated by LMs aims toalign with human values.
We explained that the human values can be broadly referred as standardsand concepts such as morality, truthfulness, and ethics. Then, we designed an attention check toensure the participants understand what source or target responses mean in our study.
Only 5 out ofthe 302 participants failed the attention check and were excluded in the final data analysis (resultingin N =297 participants finally). After passing the attention check, participants were asked to ratedifferent tasks by answering questions about alignment and coherence.
All the participants in thesession two passed this attention check.
The participants did not know which model generated whichresponse.
Besides asking about Alignment and Coherence during the evaluations, we also asked the participantsto rate the Fluency of the generated edits by asking “How fluent is the edited response (e.g., coherent,well- [Table]
[Table]
A.4 Correlation Between Automated Metrics and Human Judgement
[Table] Although we believe that humans should be the only qualified judges for the value alignment task,during the development stage of algorithms we have to leverage fast and cheap automated metrics asa reasonable estimation.
Here, we test the correlation between two automated metrics (ROUGE-Land perplexity (PPL)) and respective human judgements on Alignment and Fluency.
Table A2shows additional results on the three alignment datasets.
Besides the Alignment (Align) score,we also report Fluency score from human evaluation, and two automated metrics ROUGE-L andperplexity as automated alternatives of human scored Alignment and Fluency, respectively.
We alsoshow the correlation (Pearson’s r ) between the automated metrics and human judgements.
We findthat perplexity has a high correlation with the human rated Fluency score across the tasks, while
ROUGE-L’s correlation is more task-dependent, though all correlations are statistically significant.
One interpretation could be that the measurement of text similarity with the ground truth (i.e., what
ROUGE-L measures) is only an approximation of value alignment.
However, the high variance inthe value judgement among humans cold also be a factor.
We have studied the impact from humanfactors on the Alignment score in §A.5.
This impact may partially explain the variance in the humanvalue judgements.
[-For more chit-chat like dataset (i.e., DSTC
-]
A.5 Ordinary Least Squares (OLS) Regression Analyses
Ordinary least squares (OLS) regression (shown in Table A3) analyses show that for both AEM +
AIL and AEM + VM, party affiliation (which was measured on a 7-point scale where 1 indicates
Democrat, 4 as Moderate, and 7 as Republican) is negatively associated with alignment values (AEM+ AIL: B =-.12, SE = .05, p = .01; AEM + VM: B =-.16, SE
= .05, p < .001), which indicates that anthe more liberal annotators tend to rate the alignments higher.
This can be possibly explained by: 1)liberal users may be more familiar with such ML tasks and thus give our methods high alignmentscores; or 2) it is also possible that conservative users are more sceptical of human-value alignmenton such tasks.
Another significant predictor is the people’s perceived importance of alignment withhuman values (measured by answering the question “ Whether or not the algorithm-generated textaligns with shared human values is important to me ” on a 7-point scale).
The more important peoplethink alignment with human values is, the higher alignment scores they give for both methods.
[Table]
A.6 Error Analysis
We find the errors of S ECOND T HOUGHTS can often be categorized into one of three types : (1)
Detour errors (Table A4), where the liability is passed on to someone else, (2) Sidestep errors(Table A5), where the generated targets do not directly respond to the situated context, and (3)
Distortion errors (Table A6), where the edited responses are not directly related to the situation (e.g.,an unrelated excuse is given).
In these cases, human annotators tend to assign either lower alignmentor coherence scores (or both).
In Tables A4, A
[Table]
and A6, we show an example of such errors and show how the human-guidedcorrection is applied to these errors cases
[Table] [Table] [Table]