Aligning Large Language Models with Human Preferences through Representation Engineering (2024)

Abstract: Aligning large language models (LLMs) with human preferences is crucial for enhancing their utility in terms of helpfulness, truthfulness, safety, harmlessness, and interestingness. Existing methods for achieving this alignment often involve employing reinforcement learning from human feedback (RLHF) to fine-tune LLMs based on human labels assessing the relative quality of model responses. Nevertheless, RLHF is susceptible to instability during fine-tuning and presents challenges in implementation. Drawing inspiration from the emerging field of representation engineering (RepE), this study aims to identify relevant representations for high-level human preferences embedded in patterns of activity within an LLM and achieve precise control of model behavior by transforming its representations. This novel approach, denoted as Representation Alignment from Human Feedback (RAHF), proves to be effective, computationally efficient, and easy to implement. Extensive experiments demonstrate the efficacy of RAHF in not only capturing but also manipulating representations to align with a broad spectrum of human preferences or values, rather than being confined to a singular concept or function (e.g. honesty or bias). RAHF’s versatility in accommodating diverse human preferences shows its potential for advancing LLM performance.

## introduction while large language models (llms) learn broadranging world knowledge and a degree of reasoning proficiency, precise control over their behavior proves challenging due to the unsupervised nature of their pre-training @xcite @xcite @xcite . for each query, instruction-tuned llms @xcite ; touvron et al., 2023) exhibit the capacity to generate multiple responses that are both semantically and syntactically coherent by some sampling techniques. while such ability enables the models to provide diversity that is essential for chat agents, some responses may contain harmful, unethical, socially biased, and negative, even illegal content @xcite @xcite . existing methods steer llms to align with human preferences often using reinforcement learning (rl), with reinforcement learning from human feedback (rlhf) emerging as the most successful one @xcite . however, the underlying learning algorithms exhibit a considerable degree of complexity, sensitivity to hyperparameters, instability during training, and necessitate additional training in the reward model and value network, leading to substantial computational costs @xcite . in addressing the aforementioned challenges posed by rl-based methods, several computationally lightweight alternatives have been proposed to simplify the human preference-matching process. two prominent paradigms among these alternatives include contrastive learning @xcite @xcite and hindsight instruction relabeling (hir) @xcite . contrastive learning-based methods optimize a language model policy by increasing the relative probability of preferred responses over dispreferred ones, while hir methods transform human feedback into instructions by relabeling the original ones, indicating the relative quality of provided responses. a common characteristic shared by these two paradigms is their capability to align language models with human preferences through reward-free fine-tuning. however, the reward-free fine-tuning is vulnerable to the presence of noisy data or incorrect labels in a training set comprising a collection of preference-annotated response pairs @xcite . instances of dull sentences or very brief responses may appear repeatedly in such a training set, potentially introducing bias into the models. the exclusion of such instances from the training set renders it impossible for llms to glean insights into human preferences expressed in these instances. in contrast, rl-based methods adopt a different strategy, wherein a reward function is first extracted from a dataset of response rankings, and then this reward function can be applied to train an llm, effectively mitigating the model's direct exposure to noisy data or incorrect labels within the dataset. in this study, we aim to seek for a computationally lighter and reward-free algorithm that can effectively harness human preference expressed in datasets meanwhile safeguarding llms from the influence of noisy data. inspired by the recent advance in representation engineering @xcite , we initially locate relevant representations and activity patterns associated with high-level human preferences within an llm, and subsequently gain precise control over its behavior by manipulating its internal representations. in the neural architecture, network weights determine neural activity, neural activity determines the networks' output, and the networks' output determines the networks' behavior. instead of focusing on neurons and their connections, we see aligning llms with human feedback as an outcome of representational spaces, implemented by patterns of activity across populations of neurons. we first identify the differences in model activities between preferred and dispreferred stimuli, and then control its behavior by leveraging the identified differences in representations (see figure 1 ). we introduce two methods for controlling representations and demonstrate the efficacy of these representation engineering (repe) approaches in aligning llms with a broad spectrum of human preferences through a collection of response pairs. to validate the effectiveness of our approach in aligning with human preferences, we conducted extensive comparative experiments on the generated results. our method outperformed rlhf and other rl-free approaches in human evaluations and automated metrics such as general abilities and gpt-4 evaluations. notably, the underlying algorithms exhibit simplicity in implementation and straightforwardness in training. ## related work tuning large language models to elicit desired responses and behavior from their extensive knowledge and capabilities is essential in the development of chat agents, such as chatgpt @xcite , llama @xcite and gpt-4 @xcite , characterized by safety, performance, and controllability. the enlargement of the size of language models only does not inherently enhance their ability to follow a user's intent. for example, llms may still generate outputs that are untruthful, toxic, or simply not helpful to the user. existing human preference alignment methods can be broadly classified into three major categories: reinforcement learning @xcite , contrastive learning @xcite @xcite , and hindsight instruction relabeling @xcite . extensive research has been devoted to the exploration of rl from human feedback through ratings or rankings, spanning tasks from nl-to-sql conversion @xcite , machine translation @xcite , task-oriented dialogue systems @xcite @xcite , summarization @xcite , story-telling @xcite to instructionfollowing @xcite . typically, these methods involve the fitting of a reward model to a dataset of human preferences, followed by the optimization of a llm policy to generate responses with high reward, using rl algorithms such as reinforce @xcite or proximal policy optimization @xcite . despite the attractiveness of leveraging human preferences that are easier to collect than expert demonstrations, training llms with rl poses significant practical challenges, which is attributed to the sensitivity of rl to hyperparameters and the inherent instability during training. the solutions based on hindsight instruction relabeling @xcite and contrastive learning @xcite @xcite have emerged as computationally efficient alternatives to rl-based methods without explicit reward modeling. however, these reward-free fine-tuning solutions are susceptible to noisy data or incorrect labels within a training set. they exhibit performance lags compared to models tuned with rl counterparts (see section 4). furthermore, the question of whether llms trained with such fine-tuning methods can generalize well to out-of-distribution queries remains unresolved when contrasted with models incorporating an explicit reward model. rlhf method @xcite offers a potential avenue for improvement by leveraging additional unlabeled examples through labeling llm generations with the learned reward model. to enhance transparency and controllability of neural networks, @xcite introduced representation engineering (repe) as a methodology, drawing an analogy between understanding deep neural networks through representation tomography and studying brains via neuroimaging techniques. their work demonstrated the efficacy of repe in addressing diverse safety-related challenges such as truthfulness, honesty, and hallucination. this study falls in line with recent research findings and extends its application to aligning llms with a wide spectrum of human preferences. our study introduces two novel methods to instruct llms on human preferences first, and then extract differences in model activities between preferred and dispreferred stimuli. these differences in activity patterns serve as a foundation for manipulating the model's behavior, leading to the generation of responses that better align with human preferences. due to the lightweight computational advantages of parameter-efficient fine-tuning techniques @xcite @xcite @xcite , these techniques are utilized to fit the disparity in activity patterns. in contrast to the approach adopted by @xcite , which relies on unlabeled or self-generated stimuli limited to singular concepts or functions the meaning of which the models have already "known", our methods provide a more comprehensive alignment with diverse human preferences. ## method we begin by instructing llms on human preferences with a set of preference-annotated response pairs. we introduce two novel methods for instructing llms on human preferences and extracting their activity patterns: one involving a single llm (trained to discern the relative quality of responses) and the other employing dual llms ("a good guy and a bad guy"). secondly, we collect the activity patterns of llms when exposed to stimuli that are preferred or dispreferred. the differences in these patterns serve as the foundation for manipulating llms, enabling them to generate responses more closely aligned with human values. finally, we construct the final model by training a low-rank adapter @xcite to fit the disparity in activity patterns. ## experiment following @xcite , we mainly conducted experiments on single-turn dialogue tasks. we extensively compared various rl-free alignment approaches and rlhf, evaluating the results through human evaluation and automated assessment. additionally, we conducted comparative experiments with the representation engineering method proposed by @xcite , serving as an ablation study to demonstrate the impact of our approach in capturing human preferences. ## conclusion in this study, we have explored a representation engineering approach to aligning large language models with human preferences, drawing upon insights from cognitive neuroscience. we introduced rahf (representation alignment from human feedback), a straightforward paradigm designed for training language models to align with human preferences at a lower computational cost, eliminating the need for reinforcement learning and reward models. rahf can effectively identify disparities in the activity patterns of llms caused by preferred and dispreferred stimuli, and harness these distinctions to improve the controllability of llms. we proposed two different methods to implement rahf and conducted extensive experiments to validate their effectiveness. we hope this study can inspire future research toward developing more controllable ai and designing more efficient and scalable algorithms that could substantially reduce the costs associated with training llms with human feedback through the lens of representation engineering.