Second Thoughts are Best: Learning to Re-Align With Human Values from Text Edits
Abstract
We present S ECOND T HOUGHTS , a new learning paradigm that enables language models (LMs) to re -align with human values.
By modeling the chain-of-edits between value-unaligned and value-aligned text, with LM fine-tuning and additional refinement through reinforcement learning, S ECOND T HOUGHTS not only achieves superior performance in three value alignment benchmark datasets but also shows strong human-value transfer learning ability in few-shot scenarios.
The generated editing steps also offer better interpretability and ease for interactive error correction.
Extensive human evaluations further confirm its effectiveness
Introduction
“ Machines can and will make better decisions than humans but only when the values are aligned with those of human race. ”
——Prof. Stuart Russell, Value Alignment ,
[Figure]
tend to generate text violating human values in certain contexts.
Our method enables LMs to realign with human values by making text edits.
Current large-scale pre-trained language models (LMs) have shown great success in many knowledge-recalling tasks, such as question answering [61] and entity retrieval [11]; however, their ability to select socially good text from bad (or generating prosocial text) in open-world settings is still limited [21], even when the models are scaled up to hundreds of billions of parameters [33].
In other words, pre-training ever-larger
LMs does not lead to expected substantive gains in tasks that require human value judgment [23].
Consider the example in Figure 1: given a context, a fine-tuned LM GPT-2 [52] assigns a larger probability mass 2 to the immoral option than to the moral ground truth.
One interpretation of this failure is that the commonly used “missing token prediction” objective for pre-training (i.e.,
MLE) does not directly model human values [47].
As a consequence, fine-tuned LMs still struggle with options that are legitimate semantically (i.e., low language modeling loss) but are not aligned with human values.
To tackle this misalignment problem, prior work has proposed using binary answers [26, 56], rankings [16, 4], or ratings [69, 38] to model human value preferences.
For example, Askell et al.
[1] create a platform to collect Likert-scale human ratings on LM-generated utterances in dialogues, aiming to teach the LM to be helpful, honest, and harmless.
However, without considering how to recover from responses that already violate human values, these methods cannot serve as robust remedies in real-world applications, since they can be easily attacked by poisoned queries [17].
More recent attempts, such as InstructGPT [47], formulate the alignment problem as about teaching the machine to follow human instructions—they fine-tune GPT-3 on a variety of prompts written by human users of OpenAI’s GPT-3 API [5].
Though it indeed has the ability to revise its previous language generations, such ability relies on receiving specific human instructions (e.g., “ Please make the following sentence aligned with moral values. ”).
Manually designing proper prompts that can trigger value alignment requires extra human labor.
Besides, specifically-designed prompts do not always exist in real-world human-AI interaction, and we cannot expect most users to know how to design appropriate prompts to improve the human-value alignment of an AI agent [31].
On the other hand, rather than steering the language generation with artificial prompts, humans can easily fix immoral language by making hierarchical and recursive edits [12, 30], where human value judgments serve as the guide for each edit.
Following this observation, in this work, we propose to leverage text edits to model human values.
Our method, called S ECOND T HOUGHTS , echoes the theory of “utilitarian ethics”, which says that humans choose the actions (e.g. edits) which maximize the perceived positive impact on the most people [62, 51].
Specifically, we model human edits by three generic operations: insert, delete, and replace, and automatically infer the “chain-of-edits” by a dynamic programming algorithm.
Besides the commonly used MLE training, we deliberately include a reinforcement learning based refinement step, to further encourage valid edits which are not only aligned with human values, but also coherent with the context.
The main contribution of this work is to present a new learning paradigm that can make current LMs aware of the human value alignment.
Trained with S ECOND T HOUGHTS , LMs can not only re -align their generation with human values, even when the context has already been poisoned, but also show the chain of editing steps for ease of interpretability and to facilitate further edits (§4.5).
Through extensive human evaluation, we find that the edited responses by S ECOND T HOUGHTS (based on a 345M GPT-2) are on average scored higher with respect to their value alignment than those from
InstructGPT (based on a 1.3B GPT-3) (§4.2).
Our experiments confirm that simply scaling LMs is not adequate for good alignment with human values, which echoes the findings of recent studies [49, 33].
Instead, smaller LMs trained with a few properly decomposed human demonstrations can often lead to better results (§4.4).
We also provide a discussion on the impact of human factors during human evaluation (§5), which is crucially ignored in current AI studies.
Related Work
We briefly review existing work that considers in-context explanations during prompting or training.
We also summarize other value alignment methods for language models.
Learning From In-Context Instructions.
The few-shot performance of LMs can be enhanced by learning from in-context instructions [55, 36], in the forms of task descriptions [42, 53], answer demonstrations [5], targeting formats [41], etc., which can be positioned before [64] or even after [28] the answer.
Recent studies have shown improved results by including decomposed reasoning steps into the instructions [46, 45].
However, the instructions normally require careful human design, which is costly and whose quality greatly affects performance [67, 24].
In comparison with these methods,
S ECOND T HOUGHTS learns from text edits inferred by an algorithm, and presents the chain-of-edits for each alignment, which eases error diagnosis and enables interactive correction.
Human Value Alignment for Language Models.
Trained on unfiltered and problematic language from the web, current large-scale LMs have be shown to be poorly aligned with human values [3].
For example, GPT-3 performs only marginally better than a random baseline on a virtue matching task [65], and scaling-up LMs can even lead to deterioration in truthfulness [33].
Existing general- purpose remedies include filtering the training data [19], attribute-control generation [10], and modifying the decoding algorithm with hard (e.g., token blocklists; Schick et al.
[57]) or soft constraints (e.g., reference LMs;
Liu et al.
Though these methods are able to steer generation towards prosocial directions, our experiments show that they have limited performance when the context has already been poisoned.
There are other approaches that require training with specific forms of human supervision (e.g., fine-grained ratings) [47, 60, 68, 7], but these are often costly and not always available in every value alignment dataset.
S ECOND T HOUGHTS differs from all these methods in its offline nature and ability to re -align in poisoned contexts, requiring neither extra human labeling nor specially-designed prompts or instructions.
Approach
S ECOND T HOUGHTS comprises two main steps.
We first infer chain-of-edits automatically from source and target responses with a dynamic programming algorithm, and fine-tune an LM on the editsaugmented training data (§3.2).
Then, we deploy a reinforce learning stage to refine the generation, by either adversarial imitation learning or value modeling (§3.3).
We begin by introducing the problem of value re-alignment (§3.1).
3.1 Problem Statement of Re-alignment
[Figure]
Value alignment datasets normally consist of contexts (i.e., social situations), value-aligned responses (i.e., prosocial behaviors), and valueunaligned responses (i.e., antisocial behaviors).
Existing alignment methods formulate the value alignment task as a conditional generation problem: given a situation as the context , train a model that can generate responses resembling a value-aligned target rather than a not-aligned wrong target (Figure 2 (a)).
However, many studies have shown that LMs trained with such a paradigm can be easily derailed by poisoned contexts [47, 17]—i.e., contexts that already include value-unaligned content, either from the model’s own generation or from malicious users 3 .
In other words, unlike humans, these models lack the ability of re -alignment (the ability to recover from poisoned contexts).
To teach a model how to re-align, we deliberately add the value-unaligned response into the context, referred to as the source , and keep the value-aligned response as the target .
The intuition behind this is that instead of learning from mistakes after a misalignment occurs in the generation, the model learns how to make edits as it is generating the text.
Specifically, we include the unaligned source as part of the new “context”, and then train an LM to learn how to make sequential edits on the source to produce the target (Figure 2 (b)).
This way the model learns how to recover from a value-unaligned, poisoned context during the generation phase.
3.2 Augmented Edits Modeling
DP-based Edits Inference.
Given two text strings, source and target , one can find unlimited ways to edit source to produce target .
Thus, we apply two constraints onto the editing: (1) the edits should be combinations of generic editing operations—inserting, deleting, and replacing a single token; (2) each edit operation has a cost and our goal is to infer the chain-of-edits that has minimum cost.
Under these constraints, the edits inference problem can be converted to a token-level “edit distance problem” [27], which can be solved by dynamic programming (DP).
We modify the algorithm to be able to receive customized editing costs (e.g., insert-1, delete-1, replace-2), to try to model different preferences on editing.
We use special tokens to mark the start/end of editing and the new content to be inserted/replaced, and develop a decipher module that can translate the edit operations produced by DP into natural language (see §A.1 for a visualization of the whole process, and §A.3 for more discussion on edit based models).
Augmented Edits Modeling (AEM).
To augment the edits, we run the DP algorithm on the same source and target pairs with a variety of editing costs 4 to create a collection of chain-of-edits for each source - target pair, which we call positive demonstrations ( y + ).
We then fine-tune an LM on these source - edits - target text inputs (recall that the edits are turned into natural language).
We call this
Augmented Edits Modeling (AEM).
Different from common language modeling, AEM includes the labor-free decomposition (i.e., the editing steps) into the training object, whereas prior works either train on costly manually-created decomposition [47, 63] or, rather than training, prompt with such decomposition [64, 46].
We also construct negative demonstrations ( y − ) by using the targets from other contexts, leading to inferred chain-of-edits that generate value-aligned responses which are incoherent with the given context.
These will be used during the RL refinement described below.
3.3 Refinement by Reinforcement Learning
Though the generation of an LM trained with AEM can already align well with human values, many of the generated responses are not coherent with the given contexts.
Based on manual examination, the responses tend to be generic, rather than specific to the context (e.g., the sidestep error in Table A6).
We are thus motivated to deploy a reinforcement learning (RL) stage to further refine the generation quality, mainly to improve the coherence to the context.
Notation.
Given the concatenation of context and source as x , S ECOND T HOUGHTS will generate chain-of-edits and corresponding target as y .
In RL language, we define the state at time t as the set of generated tokens before t (i.e., s t
= y < t ), and the action as the current step’s output token (i.e., a t = y t ).
The softmax output of the language modeling head (a categorical distribution over the entire vocabulary) is considered as the policy π t for picking token y t (action a t ), given the state s t
= y < t .
Adversarial Imitation Learning (AIL).
Inspired by the concept of imitation learning in RL, which clones the behavior of positive demonstrations [29], we propose to leverage negative samples to penalize the LM for imitating the mismatched target (i.e., value-aligned but incoherent).
We train an adversarial LM only on the negative demonstrations y − , so that following its policy π A DV .
t will lead to incoherent generations.
The t -th step objective of AIL to be maximized is:
[Equation]
[Table]
unlikelihood
[Table]
[Equation]
where π ∗ t is the desired refinement policy (a vector initialized from the original π t ), α is the balancing factor, and the KL penalty term KL ( π t ∣∣ π ∗ t ) with the coefficient β is the trust region constraint, which prevents the updated policy from drifting too far away from the original one [59, 58] 5 .
The intuition behind such a design is to maximize the unlikelihood of forming the trajectory τ = { s 1 , a 1 , ..., s t , a t } that can be induced by the adversarial policy π A DV .
, weighted against the balancing likelihood term [66].
After refinement, the learned policy π ∗ t can generate tokens unlike those that can be produced by π A DV . , which will form sequences more coherent to the context.
Value Modeling (VM).
In addition to AIL, which aligns values by learning from negative demonstrations, we present another refinement method that directly learns a value function.
To this end, we train a binary LM-based classifier f on the mixture of positive and negative demonstrations.
We use f to estimate the likelihood of a given generation being coherent with the context, by passing it a concatenation of the context, source, generated chain-of-edits, and the corresponding generated target.
We take the sigmoid of the log-likelihood predicted by f as the reward r , which is r = σ log f
( x, y ) , and define the objective to be maximized as:
J VM ,t =
E τ ∼ π t
[ π ∗ t ( a t ∣ s t ) π t ( a t ∣ s t )
⋅ r t ] + λ H ( ⋅ ∣ s t ) ∼ π ∗ , (2) where the t -th step r is adjusted by an importance-sampling ratio between the current and original policy for off-policy stability [44] 6 .
We also deliberately add an entropy bonus term H ( ⋅ ∣ s t ) ∼ π ∗ of the refined policy, discounted by λ , to encourage more exploration of the current policy [20] 7 .
Compared with AIL, VM leverages an explicit value estimation module f as the guidance, rather than implicitly learning from imitation, which brings extra benefits in generalization across different human values (detailed in §4.4).
Experiments 4.1 Experimental Setting
We study the value alignment performance of S ECOND T HOUGHTS on three benchmark datasets:
Moral Stories.
The Moral Stories dataset ( N = 20 , 000 ) examines whether LMs can generate moral responses under diverse social situations [14].
We use the “situation” of each data sample as context , and treat “immoral actions” as the source , while “moral actions” as the target .
MIC.
The MIC dataset ( N = 38 , 000 ) studies whether chatbots can generate utterances that are aligned with a set of “Rules of Thumb (RoT)” of morality [69].
Each sample is labeled with its alignment level (e.g., “aligned”, “unaligned”, “neither”), RoT violation severity (from 1 to 5), RoT agreement, etc.
We take the question in the dialogue as the context , and the unaligned answers (with
RoT violation severity 4-horrible or 5-worse) as the source , and aligned answers as the target .
ETHICS-Deontology.
The ETHICS dataset ( N = 25 , 356 ) investigates the performance of LMs on five human values alignment tasks [21].
We pick the deontology split because of its contextual nature.
The contexts are requests common in everyday life, while the responses are excuses that are either aligned with deontology or not.
We take the requests as the context , deontology-unaligned responses as the source , and deontology-aligned responses as the target .
We also consider two smaller-scale human values alignment datasets: HHH (Helpful, Honest, &
Harmless)
[1] ( N = 178 ) and Truthful QA [33] ( N = 299 ), to evaluate the domain transfer ability.
We use the official train/validate/test splits in the above datasets.
As the pre-processing step, we removed hashtags and urls in the text, but leave punctuation and stop words.
Besides the generative
LM (GPT-2 medium) we use throughout the paper, we train three RoBERTa-large classifiers [37] on the mixture of positive and negative demonstrations on the above three datasets, achieving F1 scores of {99.7, 91.0, 91.9}, respectively.
They are used as f in the VM mode of S ECOND T HOUGHTS .
We run experiments on four NVIDIA A6000 GPUs, which take around {3h, 2.4h, 1.3h} for three tasks.
We conducted two sessions of human evaluation on Amazon Mechanical Turk (MTurk).
The first session was to validate the quality of S ECOND T HOUGHTS re-alignment, and the second session to evaluate cases where corrective edits were made by humans to the DP-generated chain-of-edits to improve alignment or coherence.
We recruited 297 and 100 participants for the two sessions, respectively, and each individual was randomly assigned to evaluate the three alignment tasks.
The test-set samples edited by different methods were randomly assigned to each participant without telling them the actual method name.
Each participant was paid 1 dollar for completing 20 questions for session one (§4.2), and 0.75 dollars for 15 questions for session two (§4.5).
The average completion time per session was 5m 3s and 4m 49s, respectively.
The demographic information and detailed setup procedure can be found in §A.5.
We
[Table]
4.2 Main Results on the Performance of Value Alignment
Alignment methods should be able to guide text generation towards being more value-aligned, while not compromising the texts’ coherence with the given context.
Considering the human nature of value judgement, we conduct extensive human evaluations to measure:
Alignment , by asking “To what extent does the edited response improve the original response in terms of alignment with human values?”
Answers range from 1- not at all.
to 7- to an extreme extent.
This measures the alignment improvement after the response is edited.
Coherence , by asking “How coherent is the edited response with the given context?”
Answers range from 1- not at all.
to 7- extremely coherent.
This measures the coherence level given the context after the response is edited.
Besides human evaluations, we also report evaluation results by automated metrics such as perplexity and ROUGE-L [32], and their correlation with human judgements (see §4.3).
In Table 1 we show the comparison between S ECOND T HOUGHTS and seven other alignment methods that do not require extra human labeling on the benchmark datasets: (1) MLE fine-tunes with all the data in the alignment datasets, simulating common LM pre-training (2) Data Filtering [19] only fine-tunes with the value-aligned split of the data (3) Safe Beam Search [57] blocks a list of sensitive tokens that can lead to misalignment in human values during beam search decoding(4) PPLM [10] steers the generation via soft probability constraints from Bag-of-Words instead of hard blocking on tokens 9 (5) DExperts [35] calibrates token distribution by referring to two
LMs trained on solely aligned and unaligned data.
We also consider two huge LM-based API services to explore whether scaling can make gains for human value alignment: (6) GPT-3 [5] (175B) is a general-purpose foundation model [3] which shows strong zero-shot performance in many tasks, and (7) InstructGPT [47], which fine-tunes GPT-3 (1.3B) on human-crafted prompts with a divergence controlled PPO algorithm [59] named PPO-ptx, which is our closest competitor.
Except for InstructGPT and GPT-3, we run all other baselines with GPT-2 medium (340M) for consistency.
The exact prompts and instructions used for evaluation are described in §A.2.
Results shows that S ECOND T HOUGHTS outperforms other methods in both alignment and coherence as evaluated by human judgement, especially when using AEM + VM.
MLE shows limited performance since it has no scheme to be aware of human values.
Data Filtering shows a small improvement over MLE as it clones the aligned data behavior, but is still limited when the context already includes unaligned content.
Token-constrained decoding methods such as Safe Beam Search and PPLM struggle with value alignment presumably because the abstract human values cannot be easily modeled by a set of tokens.
DExperts makes gains in alignment but the coherence of its edited responses is mostly compromised, mainly due to its token-level control.
Compared with AEM +
AIL, AEM + VM has superior performance in most cases; one interpretation could be that the value modeling provides better generalization ability, while simply imitating the aligned data can lead to accumulated off-track errors in unseen contexts [8].
Despite being built on the same LM with far fewer parameters, edits from InstructGPT (1.3B GPT-3) are rated consistently higher than those from vanilla GPT-3 (175B) 10 .
Moreover, S ECOND T HOUGHTS further outperforms InstructGPT significantly according to one-way analysis of variance (ANOVA) post-hoc pairwise comparisons ( p < 0.05) when refined with an RL stage (+ VM or + AIL).
One reason could be that aligning with human values using InstructGPT may require extensive prompt engineering.
In general, we conclude that proper value judgement cannot be simply achieved by enlarged model capacity [22], and smaller
LMs trained with properly decomposed demonstrations can often lead to better alignment results.
4.3 Correlation Between Automated Metrics and Human Judgement
Although we believe that humans should be the only qualified judges for the value alignment task, during the development stage of algorithms we have to leverage fast and cheap automated metrics as a reasonable estimation.
Here, we test the correlation between two automated metrics (ROUGEL and perplexity (PPL)) and respective human judgements on Alignment and Fluency.
Tableshows additional results on the three alignment datasets.
Besides the Alignment (Align) score, we also report Fluency score from human evaluation, and two automated metrics ROUGE-L and perplexity as automated alternatives of human scored Alignment and Fluency, respectively.
We also show the correlation (Pearson’s r ) between the automated metrics and human judgements.
We find that perplexity has a high correlation with the human rated Fluency score across the tasks, while
ROUGE-L’s correlation is more task-dependent, though all correlations are statistically significant.
One interpretation could be that the measurement of text similarity with the ground truth (i.e., what
ROUGE-L measures) is only an approximation of value alignment.
However, the high variance in the value judgement among humans cold also be a factor.
We have studied the impact from human factors on the Alignment score in §5.
This impact may partially explain the variance in the human value judgements.
4.4 Value Transfer Learning with Limited Human-Labeled Data
Since data labeled with human values is rather costly and scarce, we explore whether the alignment learned on one value-alignment task can be transferred to another, aiming to investigate the generalization ability of S ECOND T HOUGHTS on unseen values.
We first train our model on the three benchmark datasets (MRL, MIC, and ETC), recording checkpoints periodically, and then we evaluate these checkpoints on two new value alignment datasets (TQA and HHH).
We include an additional version of S ECOND T HOUGHTS which does not include chain-of-edits (i.e., vanilla text-to-text (T2T)) to demonstrate the effectiveness of chain-of-edits decomposition for domain transferability.
The results are shown in Figure 3, where the two rows reflect the results on two new datasets, while the three columns correspond to the LMs trained on three benchmark datasets.
For the TQA dataset, we find that after about 0.25 epochs, S ECOND T HOUGHTS trained on MRL and MIC with RL refinement (AEM + VM/IL) can outperform InstructGPT, which demonstrates the effectiveness of RL refinement.
We have a similar observation in the HHH dataset.
However, training on ETC does not seem to bring much benefit to the value alignment on HHH.
We also find removing chain-of-edits augmentation causes substantial performance drops, especially in the few-shot stage (less than one epoch).
We take these results as evidence that the editing decomposition in S ECOND T HOUGHTS is crucial for improving transfer learning ability, especially in few-shot scenarios.
4.5 Error Analysis and Human-Guided Correction
[Table]
Huge LM API service GPT3.28 22.
7.49 InstructGPT 4.20 25.
Pearson’s r0.
[Figure]
[Figure] [Figure]
We analyze cases where the edited responses received low alignment or coherence scores in the test set of the three tasks
[Figure]
and exemplify these errors and how we correct them with S ECOND T HOUGHTS in §A.8.
Most existing alignment methods can barely correct errors after being trained as they have no scheme for receiving additional human guidance.
Huge LMs based API services (e
[Figure]
g., GPT-3 and
InstructGPT) can potentially fix their own errors by re-prompting (with prompts defined in §A.2), but finding a proper prompt requires tedious prompt engineering.
Different from all these methods,
S ECOND T HOUGHTS allows humans to make changes on the chain-of-edits.
S ECOND T HOUGHTS will complete the chain and generate the desired target while taking the human changes into consideration.
Note that these changes can be as small as a single word (e.g., see Table A7).
We compare with results from InstructGPT and GPT-3, derived by fixing the same errors with re-prompting, and conduct human evaluation on the quality of their corrections.
As shown in Table 3,
S ECOND T HOUGHTS makes clear advances in terms of alignment and coherence after human-guided
[Table]
[Figure]
[Table]
correction, potentially because it enables more directed corrections via the chain-of-edits.
We also find that the instruction-fine-tuned InstructGPT can better adopt correction instructions than vanilla
GPT-3, despite having over 100x fewer parameters.
4.6 Configuration for the Best Performing S ECOND T HOUGHTS
We also study the impact of the balancing factor ( α ) in AIL and the entropy factor ( λ ) in VM on the performance of S ECOND T HOUGHTS .
As shown in Figure 4 (a) and (b), for the example task
Moral Stories, we find that in general a higher α will worsen ROUGE-L but improve perplexity (i.e., lowers it), as it decreases the effect of unlikelihood training on negative samples in AIL.
Through empirical observation, we set α to be 0.2 for an appropriate balance, considering the trade-off between alignment (ROUGE-L) and fluency (Perplexity).
A similar trade-off can be seen for λ in
VM (set to λ = 0 . 6 ).
In Figure 4 (c), we show the benefits of the augmentation of chain-of-edits: we augment the training data by the augmentation factor, which is a multiple of the size of the original training data, using different editing costs, as described in §3.2.
An augmentation factor of zero corresponds to vanilla text-to-text training.
We find that more augmentation does not always lead to better performance in the test set, where the best augmentation factor is 2 for AIL and 3 for VM.
Limitations and Discussion
S ECOND T HOUGHTS can be limited by the LM that it is based on—for instance, the total length of the chain-of-edits is limited by the max sequence length allowed for the LM.
Furthermore, studies from social sciences have shown that human values may change over time [50, 48], meaning that
S ECOND T HOUGHTS has to be re-trained with new human demonstrations as values evolve.
We also note that the participants used for the human evaluation may not be representative of the full spectrum of people who may use S ECOND T HOUGHTS , and that certain demographic factors such as gender, education, and ideological belief might influence their value judgement.
We thus conduct Ordinary
Least Squares (OLS) regression analyses on our human evaluation results to better understand these impacts.
Among other factors, the results indicate that the political party and the perceived importance of human values are two significant factors that have impact on value judgements.
Ordinary least squares (OLS) regression (shown in Table 4) analyses show that for both AEM +
AIL and AEM + VM, party affiliation (which was measured on a 7-point scale where 1 indicates
Democrat, 4 as Moderate, and 7 as Republican) is negatively associated with alignment values (AEM + AIL: B =-.12, SE = .05, p = .01; AEM + VM: B =-.16, SE
= .05, p < .001), which indicates that the more liberal annotators tend to rate the alignments higher.
This can be possibly explained by: 1)
[Table]
liberal users may be more familiar with such ML tasks and thus give our methods high alignment scores; or 2) it is also possible that conservative users are more skeptical of human-value alignment on such tasks.
Another significant predictor is the people’s perceived importance of alignment with human values (measured by answering the question “ Whether or not the algorithm-generated text aligns with shared human values is important to me ” on a 7-point scale).
The more important people think alignment with human values is, the higher alignment scores they give for both methods
[Table]
Conclusion
We have proposed S ECOND T HOUGHTS , a novel learning paradigm that enables LMs to re-align with human values when given a poisoned context.
Compared with existing methods, our method can generate text aligned with human-values without requiring additional human labeling or specificallydesigned prompts or instructions.
In addition, the chain-of-edits modeling by S ECOND T HOUGHTS enables easy error diagnosis and human-guided correction, which we believe to be an essential ability for human-AI interactive systems.
For future work, we plan to extend our methods on more human value alignment tasks, and try to consider multi-modality data for alignment.
For example, we can capture human’s face expression as fine-grained feedback signals for un-aligned sentences, or reversely we can not only rely on text edits but speech instructions as the chain-of-edits to model for proper value alignment.
Ethics, Broader Impact, and Reproducibility
As large-scale pre-trained LMs become integrated in more systems, it is a matter of utmost societal importance to make sure that such models adhere to shared human values.
Here, we present a light-weight framework that can align the generation of LMs with such values, without requiring new data or extensive prompt-engineering.
Though we do not foresee any major ethical issues with our proposed work, the reliance on manually annotated datasets and human evaluations may unintentionally introduce bias in our models (as discussed in Section 5).
To aid reproducibility, we have included all important information regarding hyperparameters and hardware in this paper and have included data, code, and reports from the human evaluation in the supplementary materials to aid reviewing.
We plan to release our code and data after publication under an MIT license.
Acknowledgement
We sincerely thank the reviewers for their insightful comments and suggestions that helped improve the paper.
This research was supported in part by a Google Research Scholar Award.
A Appendix A.1 Detailed Re-alignment Task Formulation and Training Setup
[Figure]
In Figure A1, we show the procedure for converting the data samples in the alignment datasets into training data of AEM (negative samples used in AIL are generated similarly).
In DP-inferred chain-of-edits (CoEs), we use a few special tokens to mark the editing operations (with their position and content).
Then our decipher module will translate these special tokens into natural language.
As the final step, we add a special token [SEP] between Context + Source and the ground truth
Chain-of-Edits (CoEs) and Target, as a boundary signal similar to the settings in text-to-text training.
During inference, we input a certain Context + Source, and the LM trained by S ECOND T HOUGHTS can generate CoEs and the corresponding Target.
We also augment the data by using different sets of costs for the editing operations (as discussed in Section 3.2, and footnote 3).
For example, we can infer another chain-of-edits if we change the cost of adding from 1 to 3 (i.e., we discourage adding new words for alignment), and thus the same Source-Target pair can have multiple chain-of-edits to be inserted in the middle.
For AEM, we fine-tune the LM with the above-mentioned Source-CoE-Target data (as shown in Figure A1, “Input for AEM”) with the common language modeling objective, which is to maximize the probability of generating ground truth tokens at each decoding step.
Assuming y ∗ 1 ∶ T = { y ∗ 1 , y ∗ 2 , ..., y ∗ T } is the ground truth output sequence for a given context x Context + Input , the
MLE objective minimizes the following loss by updating the parameter θ in the language model:
[Equation]
We train with three epochs for each task by default but set an early-stopping condition when the evaluation loss does not decrease (i.e., plateaus) for five intermediate evaluation steps.
The final perplexity obtained by AEM fine-tuning is {3.831, 4.1, 2.731} after {6000, 6740, 6720} steps, and the corresponding evaluation loss is {1.346, 1.411, 1.005} on the Moral Stories, MIC, and ETHICSDeontology tasks, respectively.
After AEM fine-tuning, the model is capable of generating CoE and its corresponding edited response but still suffers from incoherent responses (see Table A2 for more examples).
We further improve the coherence of the response via reinforcement-learning-based refinement, as we have detailed in Section 3.3.
A.2 Prompts used for Evaluation
[Table]
Table A3 shows the prompts used for evaluations (both main results and human-guided correction).
The phrases used to trigger value alignment are borrowed from the original paper of the datasets (e.g., “ ... align with morality ” for Moral Stories), in order to make sure the value triggered by a prompt is desired.
We do small in-house prompt engineering to make sure the generations of the models are at least readable.
We purposefully only perform slight prompt engineering because we want to imitate real-world use cases —most users will not put much effort, or will be unable to engineer the ideal prompt that can perfectly trigger human values alignment.
A.3 Additional Discussion on Edit-based Models
Modeling text edits has been used for other purposes such as sentence fusion and correction [39], improving generation quality [54, 18], text style transfer [40], and more.
However, none of these works have explored text edits for human value alignment.
In this work, we rethink the current challenges in value alignment and novelly reformulate the alignment problem as a text editing procedure.
We not only propose a scalable method to infer edits from enormous text data by dynamic programming, but also present two RL-based refinement methods to further improve the coherence of the edited text, which is one of the main technical concerns in previous edit-based models [13, 15].
Compared with existing alignment methods, edit-based S ECOND T HOUGHTS is more intuitive, and its decomposition paradigm is demonstrated to be beneficial when little human labeled data is available.
A.4 Qualitative Examples of Error Cases
In Table A2 we show examples of cases where either the Alignment or the Coherence (or both) are low based on human evaluation ratings.
In these cases, we find that without the RL refinement (i.e., AEM Only), the generation is likely to be value-aligned but fails to be coherent to the context (e.g., the generation in the Moral Story dataset where the response is generic and is NOT directly
[Table]
responding to the context).
In other cases, the generated response can be coherent but likely to not be value-aligned (frequently seen in GPT-3 generations), or even neither value-aligned nor coherent.
For example, in the ETHICS dataset, the response is incoherent since it does not respond to the request directly (i
[Table]
e., not owning a TV has nothing to do with helping others move their coach)
[Table]
and it is not aligned with human values (i.e., helping others is not someone’s duty)
[Table] [Table]
Table A3 shows the prompts used for evaluations (both main results and human-guided correction).
The phrases used to trigger value alignment are borrowed from the original paper of the datasets (e.g., “ ... align with morality ” for Moral Stories), in order to make sure the value triggered by a prompt is desired.
We do small in-house prompt engineering to make sure the generations of the models are at least readable.
We purposefully only perform slight prompt engineering because we want to imitate real-world use cases —most users will not put much effort, or will be unable to engineer the ideal prompt that can perfectly trigger human values alignment.
A.5 Human Evaluation Design
We conducted two human evaluations in spring of 2022.
Participants ( N =397) in both sessions were recruited using the MTurk Toolkit on CloudResearch, an online participant pool that aggregates multiple market research platforms [34].
Participants were all from the United States, and they were required to have a HIT approval rate greater than 95% and be over 18 years old.
Each participant was paid 1 dollar for completing 16 questions in each questionnaire (average completion time per questionnaire was about 5.07 minutes).
They were properly informed that the collected data would be used for research purposes in the consent form at the beginning.
Demographics.
The average age of the participants in the first session ( N =297) was 42.23 years-old (SD = 12.57, Median=41).
About half (56.2%) of the participants self-reported as male, and 43.8% self-reported as female.
Participants received 16.24 years of education on average (SD = 2.37,
Median = 16).
When asked to self-report their party affiliation, about half of (48.5%) the participants self-reported as Democratic, 27.9% as Republican, and 23.6% as independent.
The average age of the participants ( N =100) in the second session was 40.65 years-old (SD = 11.05,
Median=39).
About half (54%) of the participants self-reported as male, 45% self-reported as female, and 1% as “other”.
Participants received 15.94 years of education on average (SD = 3.74, Median = 16).
When asked to self-report their party affiliation, about half (51%) of the participants self-reported as Democratic, 30% as Republican, and 19% as independent.
Procedure.
Participants in the first session were randomly assigned into three different conditions to evaluate the three benchmark tasks: Moral Story ( n =99), MIC ( n = 99), and Ethics ( n =99).
Each participant in the second session was randomly assigned equal number of error correction samples from the three datasets.
Figure A2 shows a screenshot of our survey for the task ETHICS:
Deontology (the main screen; the other screens are not included because of limited space).
As can be seen, we clearly inform the participants about the theme, the procedure, and content warnings of our study.
We also present to the annotators the definition of the human value being studied (mainly taken from the original dataset papers).
We also provide our definition for “Alignment” and “Coherence” and show corresponding examples with explanations.
Besides asking about Alignment and Coherence during the evaluations, we also asked the participants to rate the Fluency of the generated edits by asking “How fluent is the edited response (e.g., coherent, well-written, without grammar errors)?”
Answers range from 1- not at all.
to 7- extremely fluent.
The participants did not know which model generated which response.
Note that we also designed an attention check to ensure the participants understand what source or target responses mean in our study.
Only 5 out of the 302 participants failed the attention check and were excluded in the final data analysis (resulting in N =297 participants finally).
All the participants in the session two passed this attention check.
A.6 Additional Results on Other Tasks
In addition to the three main datasets (Moral Stories, MIC, ETHICS, see Section 4.2) for benchmarking and two smaller scale datasets (TQA, HHH, see Section 4.4) for transfer learning evaluations, we conduct additional experiments on another three datasets that focus on moderation of open-domain dialogue systems 11 : MovieDic [2], Cornell IMDB Reviews [9], and DSTC8 Reddit 12 .
The three datasets have a similar structure to the alignment datasets, each sample of which has a context, a valueunaligned response (e.g., including hateful speech), and a value-aligned response (e.g., the moderated response).
The performance of S ECOND T HOUGHTS on these datasets is shown in Table A4.
In general, we find S ECOND T HOUGHTS alignment can bring consistent gains as seen in other tasks, especially for the Movie Dic and Cornell IMDB datasets.
For more chit-chat like dataset (i.e., DSTC
Reddit), we believe using larger-scale models as the base LM might be helpful, since its larger capacity makes it more capable of generating diverse responses.
A.7 Error Analysis
We find the errors of S ECOND T HOUGHTS can often be categorized into one of three types : (1)
Detour errors (Table A5), where the liability is passed on to someone else, (2) Sidestep errors (Table A6), where the generated targets do not directly respond to the situated context, and (3)
Distortion errors (Table A7), where the edited responses are not directly related to the situation (e.g., an unrelated excuse is given).
In these cases, human annotators tend to assign either lower alignment or coherence scores (or both).
In Tables A5, A6, and A7, we show an example of such errors and show how the human-guided correction is applied to these errors cases (Error Target).
After the human annotators see the ST
Proposed Edits (that leads to Error Target), they are allowed to make changes on the chain (as shown
[Table]
in blue in the tables).
S ECOND T HOUGHTS can take this changed chain (with context and source) and complete it (as shown in brown in the tables) with the newly generated target (New Target).
[Table]
A.8 Data Statistics of our Value Transfer Learning Experiments
In Table A
[Table]
we show the actual number of shots for each dataset when the fraction of epochs ranges from 0.25 to 1.0 (with 1.0 being the full training set), based on Figure 3.
[Table]
[Table] [Table]