Towards Aligning Language Models with Textual Feedback (2024)

Abstract: We present ALT (ALignment with Textual feedback), an approach that aligns language models with user preferences expressed in text. We argue that text offers greater expressiveness, enabling users to provide richer feedback than simple comparative preferences and this richer feedback can lead to more efficient and effective alignment. ALT aligns the model by conditioning its generation on the textual feedback. Our method relies solely on language modeling techniques and requires minimal hyper-parameter tuning, though it still presents the main benefit of RL-based algorithms and can effectively learn from textual feedback. We explore the efficacy and efficiency of textual feedback across different tasks such as toxicity reduction, summarization, and dialog response. We find that ALT outperforms PPO for the task of toxicity reduction while being able to match its performance on summarization with only 20% of the samples. We also explore how ALT can be used with feedback provided by an existing LLM.

## introduction to ensure language models are effective in realworld scenarios, their behavior must be aligned with the specific goals of the applications. techniques for alignment often involve training a reward model over preference data and using a reinforcement learning (rl) solution to steer the model toward preferred responses @xcite . a common argument for using rl approaches is that, unlike supervised finetuning which trains the model to predict a single good answer, an rl technique allows the model to get both positive and negative reward signals for its predictions @xcite . reinforcement learning methods, while powerful, often face significant hurdles that hamper their public adoption, i.e., requiring vast amounts of training data @xcite . @xcite argue that the scalar reward often provides a very sparse informative signal for the model. this work proposes a text-based feedback mechanism for aligning language models. we posit that providing models with textual feedback, rather than numerical scores, can offer a more nuanced and informative learning signal for understanding human preferences. this textual feedback can improve the process of aligning ai systems. in alt, we depart from traditional rl approaches such as ppo and focus on reward-conditioned rl. rewardconditioned rl @xcite is an approach that allows the policy to be trained using a supervised learning loss similar to sequence modeling. more recently, reward-conditioned rl has been adapted to the task of alignment in @xcite , where generations are conditioned using reward quantiles as feedback, and in @xcite , where they are conditioned on numerical reward feedback. building upon this, our work introduces alt, which leverages the richness of the signal provided by textual feedback to improve model performance. we conduct a series of experiments across three different tasks: reducing toxic language, summarizing text, and generating dialogue that is both helpful and harmless. the textual feedback, owing to its informativeness, can improve the efficacy and efficiency of lm alignment compared to a scalar reward, reward quantiles, or numerical scores. for experiments on reducing toxicity, we find that alt can outperform all other approaches and reduce toxicity by 62% when compared to ppo. for summarization, we show that alt can align the lm comparably to ppo with around 20% of the training data. for dialog, we demonstrate that alt can be steered towards generating more helpful and harmless responses by effectively leveraging the textual feedback provided by an llm. finally, we experiment with using an llm to provide unconstrained fine-grained feedback. we find that when trained with this longer feedback, alt fails to align the model. in section 4 we discuss possible reasons for this and outline potential future directions to learn from fully natural language feedback. ## alt: alignment with textual feedback alt adapts the decision transformer @xcite by training the model to be conditioned on textual feedback thus simplifying the rl setup to conditional supervised-finetuning. prior works have made use of this framework for alignment, by using reward quantiles @xcite , numerical scores @xcite , or contrastive feedback @xcite . alt differs from these approaches by making use of textual feedback. alt (figure 1 ) consists of two distinct phases: data collection and training. in the data collection phase, we sample generations from the model and assign language feedback to these generations. in the training phase, the model is trained to map its generations to the assigned feedback. these steps are repeated iteratively as the model is trained to generate conditioned on feedback. ## tasks we test the efficacy and efficiency of alt on three different categories of tasks that benefit from varying textual feedback. ## results alt can effectively align the model to improve summarization for the task of summarization, we also find that merely switching out the numerical reward quantile with preset textual feedback can improve summarization. when compared to quark, alt rm achieves a higher reward model score and also a higher win rate when compared to ppo ( alt can efficiently align the model to improve summarization we find that alt rm requires fewer training steps to reach similar accuracy as ppo. we also find that alt rm generates summaries that are equally preferred as ppo but are still much shorter on average. we find that with around 20% of the training samples quark and alt rm can surpass the ppo's performance with the reward model ( alt can effectively steer a model using textual feedback from an llm when trained using alt lmc we find that our model's generations become increasingly more helpful and harmless as the number of responses classified as "harmless and very helpful" go up by 49%, while the number of responses classified as "harmful" decreases by 50%. we note that after training for 20 iterations our model produces fewer harmful generations than a model trained using dpo foot_6 , and that it nears dpo harmlessness and helpfulness (figure 3 ). compared to steerlm, we find that alt lm is better off at learning to discriminate between "harmless and very helpful" and "harmless and helpful", as it almost monotonically increases the % of the former and decreases the % of the latter. we observe that steerlm fails at improving both the fraction of generations being "harmless and very helpful" and "harmless and helpful" compared to the sft model, and that it becomes less harmful by becoming less helpful, e.g., denying to answer a question or providing an unrelated answer. regarding "harmless and not helpful" and "harmful", both methods follow similar trends but alt lmc achieves better performance. we include other training curves showing the generations' length, percentage of truncated generations, and perplexity in a.3, and also perform preliminary tests on alt lmc 's ability to steer the lm in appendix e. we provide qualitative results in b.3. unconstrained text feedback fails to align a model for summarization we used a similar setup as for the tl;dr experiment but we replace the reward model with gpt-3.5-turbo and prompt it to provide feedback on the summary (appendix c). we noticed that the summaries produced changed significantly from the sft model. however, we observed no improvement in the reward model score with the llm feedback. after 6 iterations we found that the resulting model alt lmu is comparable to the sft model and worse than ppo in terms of win rates ( ## related work alignment previous research has successfully leveraged rlhf to enhance the instructionfollowing capabilities of llms @xcite @xcite . alternates to ppo have been proposed for alignment such as training in an offline setting @xcite , directly optimizing a preference objective @xcite , or treating the problem as a conditional sequence decision problem @xcite or ranking responses @xcite . alt treats alignment as a conditional sequence decision problem, but uses textual feedback. ## conclusion we presented alt, an approach that uses textual feedback to align an llm. our findings across diverse tasks, such as reducing model toxicity, improving summarization, and aligning dialogue, underscore the efficacy and efficiency of this ap-proach. notably, alt surpasses traditional reinforcement learning methods like ppo in toxicity reduction and achieves comparable summarization performance with considerably fewer training samples. furthermore, our results indicate the feasibility of leveraging large language models to provide effective feedback for aligning dialogue models. our current experiments failed to show improvements with more detailed textual feedback. however, we believe that this outcome could change with more consistent feedback. our findings open promising directions for further research into the use of varied types of feedback to improve llm alignment.