S teer LM : Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF (2023)

Abstract: Model alignment with human preferences is an essential step in making Large Language Models (LLMs) helpful and consistent with human values. It typically consists of supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF) stages. However, RLHF faces inherent limitations stemming from a complex training setup and its tendency to align the model with implicit values that end users cannot control at run-time. Moreover, reward models in RLHF stage commonly rely on single-dimensional feedback as opposed to explicit, multifaceted signals that indicate attributes such as helpfulness, humor, and toxicity. To address these limitations, we propose SteerLM, a supervised fine-tuning method that empowers end-users to control responses during inference. SteerLM conditions responses to conform to an explicitly defined multi-dimensional set of attributes, thereby empowering a steerable AI capable of generating helpful and high-quality responses while maintaining customizability. Experiments show that SteerLM trained on open source datasets generates responses that are preferred by human and automatic evaluators to many state-of-the-art baselines trained with RLHF while being much easier to train. Try SteerLM at https://huggingface.co/nvidia/SteerLM-llama2-13B

## introduction training llms on extensive text corpora has demonstrated remarkable capabilities, leading to state-of-the-art performance on numerous tasks @xcite . however, this does not automatically make language models effective in responding to user instructions @xcite . to better align llms to human preferences, the most effective approach has been to perform sft followed by the application of rlhf @xcite @xcite . in sft, human annotators provide demonstrations of instructions and responses for the model to imitate @xcite . rlhf goes a step further to enable models to generate responses that human annotators prefer to alternative responses @xcite @xcite . however, despite its success, there are limitations to this approach. first, using sft alone does not allow the model to distinguish between highquality and low-quality responses leading to lower performance than rlhf @xcite . using rlhf for model alignment however, substantially increase the complexity of the training setup @xcite , limiting its public adoption @xcite @xcite . furthermore, rlhf treats human preference of model responses as monodimensional without regard for the diversity of aspects (e.g. helpfulness, humor, toxicity) that contribute to such preferences @xcite and thereby limiting users' ability to adjust individual aspects at inference time based on their use cases. to address these limitations, we introduce steerlm, a novel approach to model alignment through sft that overcomes the limitations associated with conventional sft and rlhf methods. similar to rlhf, steerlm incorporates additional reward signals by leveraging annotated attributes (e.g., quality, humor, toxicity) present in the open-assistant @xcite dataset for each response. to emulate the process of rating responses, an attribute prediction model ( §3.1) is trained and employed to annotate datasets ( §3.2) containing dialogues that can be decomposed into prompt-response pairs. by utilizing this combined diverse dataset comprising prompts, responses, and predicted attributes, we train the generation of responses to be conditioned ( §3.3) on both the prompt instructions and the annotated attributes, enabling steerlm to effectively capture human preferences and generate responses that align with them. steerlm also exhibits enhanced versatility compared to rlhf, allowing for the flexible utilization of various attributes during inference. to further enhance the adherence of steerlm to the specified attributes during inference, we introduce an additional training recipe ( §3.4) which includes augmenting the training data with denoised, diverse, and high-quality examples. we open-source code for steerlm on nvidia nemo toolkit @xcite . in summary, our key contributions are: ## related work model alignment using sft fine-tuning language models on multiple tasks enable them to follow many types of instructions and perform tasks outside of those they were trained on @xcite . however, language models typically generate short and robotic responses when supervised finetuned on academic data sets. on the other hand, models can generate high quality human-like response when trained with high-quality human demonstrations @xcite . @xcite shows using data generated by openai's text-davinci-003 model, can train a model in a costeffective manner. using only sft for model alignment became popular recently because of the ease of its training setup. zhang et al. (2023) and peng et al. (2023 model alignment using rlhf building on foundational work on rl in games and robotic simulations @xcite , many have had success applying rlhf to improve the instruction following ability of llm by giving a reward proportional to the relative desirability of the response @xcite . such an approach has been shown to benefit downstream tasks like question-answering @xcite and summarization @xcite . however, the complexity of the training setup @xcite remains a hurdle in the widespread adoption of rlhf. many have attempted to overcome this by migrating rlhf training to an offline setting @xcite , casting the problem as conditional sequence modeling @xcite , directly optimizing lms with labeled preference data @xcite , or ranking responses to align the models @xcite , but limited progress has been made @xcite . another limitation unaddressed by related works lies in the use of a single-dimensional reward function for evaluating human preferences of model responses since human preferences are based on a multitude of real-world objectives (e.g. helpfulness, humor, toxicity), which also vary across domains @xcite . given such multi-dimensional attributes, current approaches @xcite @xcite are only capable of generating responses with high reward scores, despite low reward scores on various attributes being relevant in certain situations, such as modelling realistic gaming npcs who are capable of generating high toxicity responses. attribute grounded generation many researchers have explored grounding text with various attributes in dialogue tasks. rashkin et al. (2019) modelled chit-chat conversations grounded in emotions such as 'angry', 'embarrassed' or 'joyful' while smith et al. (2020) conditioned chit-chat dialogue with conversational styles such as 'curious', 'sympathetic' or 'knowledgeable' . zhang et al. (2018) and wang et al. (2022) conditioned dialogues based on personal attributes such as their hobbies. meta et al. ( 2022 ) conditioned dialogues in the game of diplomacy using the expected player skill. however, such grounding has only been explored in narrow-defined tasks with a single attribute. our approach seeks to condition the generation of responses in general open-domain conversations covering tasks like code assistance, writing poems and planning tasks, using multiple attributes (e.g., quality, humor and toxicity). ## steerlm we propose steerlm, a simple and novel approach to align language models to follow user instructions. trained solely using the language modeling objective, it offers a computationally efficient alternative to other techniques like rlhf. specifically, steerlm comprises 4 steps as illustrated in fig. 2 . ## experiments in this section, we elaborate on our choice of training datasets, base model, implementation details, and evaluation setup. ## steerability demonstration to demonstrate the efficacy of steerlm 43b to control its generations in a multi-faceted way, we conduct an empirical study incorporating the attributes of toxicity and humor. ## conclusion we introduce steerlm, a novel model alignment approach with a value system (e.g. humor level and toxicity tolerance) that can be adjusted by users at inference time without re-training. steerlm trains both the attribute prediction model and the language model using only supervised fine-tuning, resulting in an easy-to-implement and straightforward training process compared to using rlhf. we train steerlm models following this procedure, achieving state-of-the-art results on the vicuna benchmark. we validate these results with a human evaluation and find that steerlm is preferred over the other models we compare it to. we hope our work will inspire further research into developing simple and effective model alignment methods that empower better ai assistants for everyone.