Steering Llama 2 via Contrastive Activation Addition (2024)

Abstract: We introduce Contrastive Activation Addition (CAA), a method for steering language models by modifying their activations during forward passes. CAA computes “steering vectors” by averaging the difference in residual stream activations between pairs of positive and negative examples of a particular behavior, such as factual versus hallucinatory responses. During inference, these steering vectors are added at all token positions after the user’s prompt with either a positive or negative coefficient, allowing precise control over the degree of the targeted behavior. We evaluate CAA’s effectiveness on Llama 2 Chat using multiple-choice behavioral question datasets and open-ended generation tasks. We demonstrate that CAA significantly alters model behavior, is effective over and on top of traditional methods like finetuning and system prompt design, and minimally reduces capabilities. Moreover, we gain deeper insights into CAA’s mechanisms by employing various activation space interpretation methods. CAA accurately steers model outputs and sheds light on how high-level concepts are represented in Large Language Models (LLMs).

## introduction as the capabilities of large language models (llms) have grown rapidly in recent years, an increasing body of research aims to ensure they are "helpful, honest, and harmless" @xcite to reduce risks from misaligned, unsafe behavior @xcite . researchers have developed several techniques for aligning llms, such as reinforcement learning from human feedback @xcite (rlhf), instruction finetuning @xcite , and prompt engineering @xcite . however, many challenges remain, including collecting diverse and representative datasets for the target behaviors, preventing hallucination, and mitigating out-of-distribution failures. moreover, the way in which these methods work is often opaque. the set of alignment techniques known as "activation engineering" or "representation engineering" work by making targeted perturbations to a model's activations @xcite @xcite @xcite . although activation engineering techniques have shown some promise as a way to steer models' behavior, their mechanisms, properties, and effects have yet to be robustly verified across different models and types of behaviors. we employ contrastive activation addition (caa) to modulate high-level alignment-relevant behaviors in llms and study its effects and properties in various test scenarios. we apply the technique to llama 2, a collection of pretrained and finetuned llms ranging in scale from 7 to 70 billion parameters @xcite , primarily focusing on llama 2 chat, which is optimized for dialogue use-cases and finetuned using rlhf for safety. this enables us to study the interaction between rlhf/finetuning techniques and activation engineering, building on top of the existing body of research on pretrained models and demonstrating that caa can be used on top of finetuning techniques to improve alignment-relevant properties. section 3 describes the process of generating steering vectors, including the datasets we used to construct them. section 4 presents our main results on the effects of caa on multiple-choice and open-ended generation evaluations. in particular, across all of the seven categories we tested, the addition/subtraction of the steering vectors increased/decreased the prevalence of the behavior, as rated by @xcite . we then show caa's effects on transfer, compare it to other alignment techniques such as system-prompting and finetuning, and investigate the geometrical relationships of the steering vectors. section 9 concludes by discussing our results qualitatively and pointing towards potential future research directions. ## related work turner et al. (2023)'s activation addition approach generates steering vectors by taking the difference in intermediate activations of a pair of prompts at a particular layer and token position in a transformer model. the steering vector is then added to the first token position of other forward passes to steer the model's completions. this technique has limitations; it does not consistently work for different behaviors, is not robust to different prompts, and was only tested on gpt-2-xl @xcite . our technique is similar to activation addition. however, our steering vectors are generated from a dataset of contrast pairs rather than a single pair. using hundreds of diverse contrast pairs reduces noise in the steering vector, allowing for a more precise encoding of the behavior of interest. we also add our steering vector to all and only token positions after the original prompt. @xcite employ linear probes to predict truthfulness on a contrastive question-answering dataset to identify as sparse sets of "truthful" attention heads. during inference, they shift activations along the vector connecting the means of the true and false distributions, employing the same mean difference vector extraction approach as caa. this technique improves truthfulness on adversarial benchmarks while minimally impacting fluency and requiring little data compared to alternatives. we similarly aim to modulate properties of the output via linear perturbations. however, our technique can be applied directly to the residual stream without searching for individual attention heads, and we validate the approach on a broader range of alignment-relevant behaviors in models trained using rlhf. @xcite propose various techniques for locating and extracting representations corresponding to high-level concepts such as honesty and emotions in llms. they also test the mean difference approach used in caa for representation extractions. however, caa employs an optimized multiple-choice format that results in more closely 2 paired contrastive prompts that differ by only a single token. we also build on this work by focusing on steering rather than representation extraction, experimenting with a broader range of behaviors, and comparing steering to system-prompting and supervised finetuning. @xcite steer models to reduce toxicity and affect style transfer. unlike caa, they steer the attention activations rather than the residual stream and intervene at all transformer layers rather than a single layer. beyond steering behaviors, work on activation engineering has also motivated a formalization of "linear representation" @xcite and helped verify linear representations of sentiment in llms @xcite . ## method the key idea behind caa is to generate a steering vector that can shift a language model's output distribution towards a desired behavior during inference. we create these steering vectors using pairs of prompts: one prompt demonstrating the desired behavior and one prompt demonstrating the opposite. by taking the average difference between the language model's activations on a set of paired prompts, we isolate the direction in the model's latent space corresponding to the target behavior. specifically, our prompt pairs consist of multiplechoice questions with answer letters (either "a" or "b") appended at the end. the two prompts contain the same question but end with different answers; the "positive" prompt ends with the letter corresponding to the behavior in question, and the "negative" prompt ends with the letter corresponding to its opposite. to construct a steering vector, we compute the difference in the language model's activations at the position of the answer letter between all the positive and negative prompts. this method of extracting the difference vector is called mean difference (md) and has been shown to produce steering vectors similar to other techniques like pca @xcite . this process is shown in figure 1 . formally, given a dataset d of (prompt p, positive completion c p , negative completion c n ) triples, we calculate the md vector v m d for a layer l as: where a l () gives the activations at layer l for the given prompt and completion letter. intuitively, by only varying the answer option between paired prompts and keeping the rest of the prompt constant, we isolate the internal representation most related to the target behavior while canceling out other confounding variables. we evaluate the effects of caa on llama 2 7b chat and llama 2 13b chat, 7 and 13 billion parameter versions of llama 2 that have been trained using rlhf for safety and to follow human instructions in a chat format. we also generate steering vectors from the llama 2 7b base model to test similarity and transfer. to load the llama 2 models, we employ the huggingface transformers library @xcite . we then use pytorch @xcite to modify the model to save intermediate activations for steering vector generation and apply steering vectors during inference. details on accessing our caa codebase can be found in appendix a. ## caa and system-prompting another approach to controlling llm generations is to use a "system prompt" that contains custom instructions describing how the model should respond to user inputs. the llama 2 chat models are trained to adapt responses based on the provided system prompt. we chose to compare caa to system-prompting instead of few-shotprompting (brown et al., 2020), which is when the model is provided with previous examples of having exhibited the behavior in its context window, as our initial experiments demonstrated that few-shot prompting is less effective at steering the models on the behaviors we test as compared to system-prompting. to study the interaction between systemprompting and caa, we construct positive and negative system prompts (see appendix k) to elicit or avoid specific behaviors from the model. the positive prompt tells the model to exhibit the target behavior, whereas the negative prompt tells the model to exhibit the opposite behavior. as shown in we hypothesize that caa provides better control than system-prompting alone because it enables precise control over the steering quantity via the multiplier and isolates behavioral variables more effectively by aggregating information over a large dataset of prompts. as shown in we also note that the effect of layering caa on 4 https://www.nvidia.com/en-us/data-center/ l40/ top of finetuning improves open-ended generation more significantly than it improves performance on multiple-choice questions (full results for caa and finetuning in the multiple-choice test regime can be found in appendix f). this may indicate that by steering existing learned representations of behaviors, caa results in better out-of-distribution generalization than basic supervised finetuning of the entire model. ## effect of caa on general capabilities we test the model under different interventions on the mmlu (massive multitask language understanding) benchmark @xcite 5 to measure any adverse effects on model capabilities. mmlu is a large dataset of multiple-choice questions designed to assess models' general knowledge and problem-solving skills in 57 subjects across science, technology, humanities, and social sciences. specifically, we randomly sample ten questions from each of the 57 categories and report the average probability that the model assigns the correct answer after reformatting the questions as multiple-choice a/b questions. as shown in we also assess the effect of sycophancy caa on truthfulqa @xcite 6 , a truthfulness benchmark that assesses the extent to which models mimic human falsehoods. full results are reported in appendix h. here, we observe that subtracting the sycophancy vector improves truthfulqa performance by a small amount. ## discussion our results suggest that caa is broadly applicability as a method for steering the behavior of llms trained with rlhf, the generalization of steering vectors derived from multiple-choice contexts to open-ended generation tasks highlights the technique's versatility and the potential for practical application in real-world scenarios. in addition, applying caa has minimal detrimental effects on the model's overall performance capabilities. another compelling aspect of caa is its compatibility with standard alignment techniques such as system-prompting and finetuning. the additive nature of caa's steering capabilities allows for a layered approach to model steering, where caa can refine and adjust model outputs further, even after applying other alignment methods. the ability of caa to control latent variables within the model's internal state opens up new avenues for inference-time control. it has high sample efficiency and strong generalization, and is particularly advantageous in scenarios requiring the precise modulation of model behavior or the elicitation of internal states that are difficult to trigger with prompting alone. moreover, the insights gained from applying caa extend beyond immediate practical benefits, offering a deeper understanding of models' internal representation and processing of high-level concepts and shedding light on the emergence of linear representations. in conclusion, by enabling precise, efficient, and effective control over model behavior, caa contributes to the broader goal of creating ai systems that are controllable and aligned with human values and provides additional insights into emergent linear representations of abstract concepts in llms.