Multi-property Steering of Large Language Models with Dynamic Activation Composition (2024)

Abstract: Activation steering methods were shown to be effective in conditioning language model generation by additively intervening over models’ intermediate representations. However, the evaluation of these techniques has so far been limited to single conditioning properties and synthetic settings. In this work, we conduct a comprehensive evaluation of various activation steering strategies, highlighting the property-dependent nature of optimal parameters to ensure a robust effect throughout generation. To address this issue, we propose Dynamic Activation Composition, an information-theoretic approach to modulate the steering intensity of one or more properties throughout generation. Our experiments on multi-property steering show that our method successfully maintains high conditioning while minimizing the impact of conditioning on generation fluency.

## introduction as large language models (llms) rapidly evolve, enabling better controllability for these systems has become increasingly important for ensuring their safe deployment in real-world settings. traditional adaptation techniques such as reinforcement learning from human feedback (rlhf) @xcite @xcite alter llms' behavior through ad-hoc training procedures, resulting in permanent modifications that can negatively impact the models' downstream generation quality @xcite . various inference-time interventions methods were recently proposed as an alternative, enabling targeted changes during generation while avoiding the high costs and the unpredictability of training @xcite . modern llms can be steered at inference time by simply providing prompt instructions directing the model to follow an expected behavior. this method can be further enhanced by providing relevant in-context examples showcasing the desired behavior, a practice known as few-shot incontext learning (icl; @xcite . new insights into the inner workings of llms highlighted the locality of interpretable concepts and properties in models' latent space, paving the way for activation steering techniques intervening directly in the llm predictive process @xcite . these techniques use model internals to craft steer-ing vectors capturing the behavior of interest, for instance by using pairs of examples showcasing valid alternatives or opposite behavior polarities. these vectors are then added to model states during the generation process to condition the resulting predictions. while previous evaluations of activation steering methods showed their effectiveness, they mainly focused on short generations, e.g. predicting single-word antonyms or translation @xcite , matching country capitals and persons' languages @xcite or answers' letters for multiple-choice questions @xcite . moreover, these studies focus on quantifying the conditioning strength of individual properties but do not consider cases where multiple properties can be conditioned at once (e.g., producing an answer in a chosen language while ensuring its safety). in this work, we address these aspects by conducting an in-depth investigation of activation steering strategies, focusing in particular on multiproperty activation steering. we benchmark several approaches to condition the safety, formality, and language of llm outputs throughout the generation, finding that the optimal steering configuration is highly property-dependent and highlighting a trade-off between conditioning intensity and the resulting generation fluency. in light of this, we propose dynamic activation composition, a strategy for modulating the steering intensity throughout generation by exploiting the information gain derived from steering vectors for one or more properties of interest. when applied in a multi-property steering setting, our approach enables strong conditioning for all selected properties while maintaining a high fluency in model generations. 1 2 related works steering language models activations the linear representation hypothesis states that high-level concepts are represented linearly in intermediate llm activations @xcite . as a consequence, steering vectors encoding some properties of interest can be added to the intermediate activations of a language model to influence its generation @xcite . while steering vectors can be learned via optimization @xcite , recent methods derive steering vectors from lm activations over contrastive pairs of in-context demonstrations @xcite . the effectiveness of these meth-ods can be motivated by their capacity to summarize human-interpretable concepts showcased in the prompt @xcite @xcite , leading to surgical updates in the limited set of dimensions capturing the conditioned property. similar approaches have recently been adopted to control attributes such as toxicity @xcite @xcite , truthfulness @xcite @xcite , sentiment @xcite , and behaviors like refusal and sycophancy @xcite on multiple model families and model sizes. in this work, we extend the evaluation of activation steering approaches to a multi-property setting, studying the impact of steering intensity on conditioning strength and generation fluency. 2 controllable text generation while controllable generation traditionally requires ad-hoc training to update llms behavior @xcite @xcite , several works showed that on-the-fly controllability can be achieved by using an external discriminator module for steering the generation style or topic @xcite @xcite . recent advances in llms' in-context learning capabilities further simplified generation controllability, enabling style conditioning via in-context demonstrations @xcite @xcite . our proposed steering method is akin to contrastive decoding @xcite , using the shift in prediction probabilities produced by steering vectors' addition to modulate their influence over the upcoming generation step. 2024 ), we perform activation steering by using a contrastive set of input demonstrations showcasing opposite polarities for the desired property or behavior. our procedure is composed by two stages: activation extraction let: ## method the process of equation ( 2 ) is repeated for the opposite polarity, resulting in v - i . finally, the steering vector ∆ at position i is computed as: intuitively, ∆ i highlights activation dimensions showing distinctive behavior for examples of the two polarities across the majority of p pairs and hence can be used to steer the llm generation. ## single-property steering in this initial evaluation, we test activation injection strategies on single properties with the goal of finding commonalities and possibly identifying the best overall technique. figure 2 presents our results across all tested properties, for α steering intensities ranging from -1 to 4, 10 while start fails to maintain good conditioning as generation progresses we find the start strategy adopted in previous steering studies to generally underperform across all properties with the exception of safe and formal, which are present by default in model outputs. this is especially true for language conditioning (first two rows), where almost no accuracy is achieved. from the start example of fixed and dim produce good conditioning but can lead to disfluencies for high α the second technique employed, fixed, shows better steering effectiveness during generation. we find its accuracy to be directly proportional to the applied steering intensity α across several properties, with the exception of safe, formal, and romance languages for which strong conditioning is achieved even for low α values. despite the good conditioning, we remark that the perplexity also tends to rise for higher α values, leading to nonsensical generations as the one presented negative steering effectively conditions against the property of interest focusing on unsafe and formal results in the fixed (also shown in appendix c), we observe that using α = -1 negatively conditions the property compared to the default model behavior (noicl). this could not be observed for language and informal properties, provided that the model outputs do not reflect these behaviors by default. for language in particular, given the absence of a polar opposite for language steering, we observe that steering with negative α leads to very high perplexities. overall, these results confirm the observations of @xcite , showing that activation steering can be reversed to produce the opposite effect. activation steering produces similar vectors for related languages figure 3 visualizes steering vectors ∆ @xmath0. from the results, it is evident that the three romance languages exhibit similar patterns over attention heads across model layers, while chinese shows lower scores and overall different results. we also note that the steering contribution of heads is stronger from the middle layers onwards. this result is consistent with what has been observed in the literature, where especially middle and last layers have a stronger influence on the final semantics of the output @xcite . more tasks and discussions about the single steering vector similarities are available in appendix e. lastly, in light of single-property steering results of figure 2 , it is evident that steering accuracy and fluency results are property-dependent, with the best trade-off between these two aspects varying greatly depending on the property of interest. for example, dim language steering is fairly robust for high α, while even minimal steering in the fixed settings produces high perplexities for formal and informal properties. overall, this indicates that different properties would require ad-hoc calibration of steering intensities to produce fluent and conditioned outputs. ## dynamic activation composition as we just noted, the activation steering process results in a trade-off between output fluency and steering intensity. this section proposes a strategy, which we name dynamic activation composition (dyn), to mitigate this limitation by dynamically adapting steering intensity during generation. in the previous section, diminishing steering (dim) has proven to be the most effective among tested approaches for maintaining high fluency while ensuring steering effectiveness. however, the optimal intensity α can vary greatly, with some properties requiring little steering (e.g. for romance and safe in figure 2 , α = 1 is sufficient and has almost no impact on fluency), whereas others might require high α to maximize steering accuracy (e.g. for chinese and unsafe, high α for dim does not affect response fluency). dim results suggest that high perplexity might be the result of over-steering an already-conditioned generation step, causing a drop in generation fluency. for this reason, we propose to derive property-dependent α values dynamically at every generation step to intervene with appropriate intensity and 'deactivated' when no longer necessary, limiting the impact of steering on fluency. the key advantage of this strategy is to enable out-of-the-box steering for any property of interest without having to carefully tune the α value beforehand. ## multi-property steering under the assumption of linearity of the model's internal activations (see section 2), we evaluate baseline activation injection strategies and the newly introduced dyn method for multi-property steering, focusing in particular on conditioning model outputs to match the unsafe or informal properties while also requiring them to be in one of the four studied languages. all activation injection techniques (start, fixed, and dim) and the icl and noicl baselines tested in section 5 are evaluated alongside dynamic activation composition (dyn). results figure 4 shows multi-property steering results for different conditioning techniques averaged across all available languages. 12 in most cases, dyn yields the best trade-off between steering strength (higher accuracy) for each task and generation quality (lower ∆ppl icl ). we note in particular how, in multi-property settings, language conditioning dominates the result in the icl case, while the unsafe and informal aspects in the provided examples are mostly ignored by the model. in contrast, the various injection strategies achieve good conditioning on both properties with minimal increases in perplexity. by examining the steering intensity applied in the dyn setting during generation (figure 5 shows an example for language (averaged) and unsafe), we note that α i generally decreases sharply after the first few generated tokens, suggesting that our naive dim strategy might still overestimate α i values at intermediate generation steps. generation examples in the dyn setting 13 , show that the language steering intensity decreases as soon as a few complete words in the desired language are generated. similarly, for unsafe the α value drops as soon as the model generates a sequence of tokens that complies with the prompt's unsafe request. lastly, figure 5 also shows that the p top parameter, which determines the amount of tokens considered in the kl divergence computation, shows a negative correlation with the sharpness of the initial spike in α values: the smaller the value, the more restrictive the top-p token selection, leading to a higher kl. intuitively, for higher values of p top 13 examples available in appendix d and g many of the selected tokens would receive negligible probability mass from both the steered and the unsteered model, leading to an under-estimate of the steering required. across all tasks, we find 0.5 as the optimal value for p top , leading to a sufficiently low cardinality of q to capture probability shifts between most likely tokens that could be selected by sampling or beam-search decoding. ## conclusion and future work through a systematic study of different activation injection strategies, we confirm that activation steering is an efficient and promising way to condition llm generations on desired properties. however, we also observe that existing injection techniques are limited in two ways: (i) steering beyond single tokens, i.e., ensuring that the conditioning is preserved across longer generations, requires interventions that harm output fluency; (ii) their effectiveness is property-dependent, making it challenging to steer multiple properties simultaneously as each property is likely to require an ad-hoc steering intensity to ensure maximal performance. for this reason, we proposed dynamic activation composition, a strategy to adaptively control the steering intensity at each generation step according to the expected steering effect, thereby limiting oversteering of already-conditioned properties while promoting the under-conditioned ones, ultimately achieving the best trade-off between conditioning accuracy and output fluency. in sum, dynamic activation composition can facilitate the alignment of llms to multiple desired properties and behaviors at once. in future experiments, it will be interesting to study the effect of our method on the perplexity of larger llms, considering these models are naturally more fluent. from an interpretability standpoint, our approach offers an interesting direction to study how properties condition model behavior during generation.