Style Vectors for Steering Generative Large Language Models (2024)

Abstract: This research explores strategies for steering the output of large language models (LLMs) towards specific styles, such as sentiment, emotion, or writing style, by adding style vectors to the activations of hidden layers during text generation. We show that style vectors can be simply computed from recorded layer activations for input texts in a specific style in contrast to more complex training-based approaches. Through a series of experiments, we demonstrate the effectiveness of activation engineering using such style vectors to influence the style of generated text in a nuanced and parameterisable way, distinguishing it from prompt engineering. The presented research constitutes a significant step towards developing more adaptive and effective AI-empowered interactive systems.

## introduction large language models (llms) pre-trained on vast corpora have marked a significant milestone in natural language processing, presenting remarkable language understanding and generation capabilities. models like gpt-2 @xcite and more recent variants such as gpt-3 @xcite and @xcite have become influential in transforming the landscape of text generation. llms have the potential to encode extensive public knowledge and can respond to a wide array of text prompts in a manner that often closely resembles human communication. openai's chatgpt, in particular, has garnered substantial attention, propelling discussions about generative ai from the scientific community into the broader public sphere @xcite . in this era of ever-advancing ai, it is becoming increasingly apparent that llm-based artificial assistants will play a prominent role in both professional and personal contexts @xcite . examples of these are conversational in- stylized output "the weather is great!" pretrained llm layer n-1 layer 0 layer 20 layer 21 layer 19 layer 18 ... ... + + + input prompt "how is the weather?" style vector for layer 18 style vector for layer 19 style vector for layer 20 target style: positive figure 1 : the llm output is steered by adding style vectors to selected layers (e.g., layers 18-20) during a forward pass. for example, the answer of the llm to the input prompt "how is the weather?" is steered towards a positive style, with a sample answer of "the weather is great!", a positive answer. in these complex settings, text generation on a lexical level alone is not sufficient for effective human-ai interaction. over and above that, a cognitive ai assistant should also be able to adapt to the human user on an affective and emotional level regarding engagement, regulation, decisionmaking, and discovery @xcite . there is evidence that llms perform well on affective computing tasks, such as sentiment classification and personality prediction, and can have emotional dialogue capabilities to some extent. however, the resulting capabilities do not go far beyond simpler specialized models, presumably due to the llms' generality @xcite . this limitation calls for mechanisms to better control implicit information and the style of an llm's output. prompt engineering has been a promising ap-proach in human-ai collaborative tasks, improving task efficiency and user collaboration @xcite . however, it is often highly task-specific and entails manually crafting prompts. in this paper, we build upon and extend the works of @xcite and @xcite , which focus on steering the output of llms by modifying their internal states. in a series of experiments, using datasets of text samples labeled with sentiments and emotion categories, we show that one can derive a vector representation of a desired style class (e.g., positive sentiment) that, when added to the activation of certain layers of an llm (in this work llama @xcite ), its output shows characteristics of this style class (see fig. 1 ). our experiments show that the effect of the changed models is more salient when prompted with subjective input (e.g.,"how do you define art?") rather than with factual input that allows little degrees of freedom (e.g., "what is the world's longest river?"). our research aims to bridge the gap between the llm's capabilities and the nuanced requirements of human-ai interactions, thus extending this novel dimension to the realm of controlling llm outputs. an open-source implementation of the algorithms used in this paper is available foot_0 . ## background and related work the introduction of transformer architectures in neural networks @xcite has led to a massive leap in the development of contextualized language models, such as gpt @xcite . these novel large language models (llms) capture relations in the natural data and implicitly encode an unlimited number of more abstract concepts, such as sentiment or style. this quality has been exploited in several recent investigations and can be both a risk @xcite and a chance @xcite . many approaches have been developed with the aim of controlling or affecting the output of llms, also referred to as steering llms @xcite @xcite . traditionally, methods for producing text in a specific style fall under the domain of stylized response generation @xcite @xcite . nonetheless, as common approaches of this class ne-cessitate training and fine-tuning whole models, these methods are not applicable to state-of-the-art llms, given the immense parameter count and training costs of llms @xcite . another line of research worth mentioning that aims to employ alternative approaches to the traditional fine-tuning approach is the parameterefficient transfer learning approach @xcite using adapter modules, which seek to minimize trainable parameters. in contrast, in our work, we focus on a different efficiency aspect, not only on the minimal computational resources but also on the minimal data resources used. a related but conceptually different approach to affect the output of llms is text style transfer (tst) @xcite . tst aims to transfer the style of a given text into a desired, different style. in contrast, steering llms deals with the task of generating a response in a desired style. we refer to @xcite for a detailed overview of tst. prompt engineering @xcite @xcite @xcite @xcite focuses on controlling and directing the output of a language model by designing input prompts or instructions. by tailoring the natural language prompts, the model's output can be steered towards producing responses in the desired style. some recent approaches move in a new direction by modifying the layer activations of an llm during the forward pass @xcite @xcite . these approaches can be grouped under the term of activation engineering. @xcite presented so-called steering vectors that, when added to the activations at certain layers of an llm, steer the model to generate a desired target sentence x from an empty input. the rationale behind this is that the information needed to produce the target sentence is already encoded in the underlying neural network. thus, the approach works without re-training or fine-tuning the model itself. starting with an empty prompt, i.e., beginning of sentence @xmath0, the vector z @xmath1, where d is the dimension of the layer to generate the next of the t tokens of x. the objective is to find a steering vector ẑsteer that maximizes the log probability: it was demonstrated on a subset of sentences of the yelp sentiment dataset @xcite that steering vectors can be used for shifting the style of a sentence x towards a dedicated target style using the vector arithmetic: the steering vector ẑtarget can then be used to steer the model to generate a sentence x ′ that is similar to x but in the target style. moreover, layer activations have demonstrated utility in steering llms. @xcite exemplify that steering vectors, derived from contrasting activations for semantically opposed inputs like "love" and "hate" can guide llm outputs during sentence completion. the difference in activations from such contrasting prompts at layer i can straightforwardly be added to another input's activations to steer outputs. in this work, we add to this line of research a method that efficiently steers llm outputs towards desired styles with notable control and transparency. in contrast to the aforementioned steering vector and tst techniques, it requires no additional optimization or prior knowledge about original styles. unlike prompt engineering, our approach offers quantifiable adjustments in style, providing nuanced differences in responses without relying on vague intensity indicators in prompts, such as "extremely negative" versus "negative." ## methodology we aim to modify the llm activations for an input x to generate an output that is steered towards a specific style category @xmath2. as shown in eq. 3, this is achieved by finding style vectors v (i) s associated to s such that when added to the activations a (i) (x) at layer i the output becomes steered towards s. style categories can be, for example, positive and negative for sentiment styles or different emotion classes such as joy and anger. the weighting parameter λ (eq. 3) determines the influence target sentence trainable steering vector for layer i, iteration j activation vector for layer i layer n-1 llm layer 0 layer i ... ... @xmath3... ... -+ target sentence activation-vs. training-based loss j figure 2: extraction of an activation vector (left): the llms' values at layer i for a prompt in the target style are saved for later computation of style vectors. trained steering vectors (right): the values of the vectors are optimized over @xmath4. in this study, we compare two main approaches to calculate style vectors, namely training-based style vectors (sec. 3.1) and activation-based style vectors (sec. 3.2). training-based style vectors are found from the generative steering vectors @xcite . in contrast to this generative approach, activation-based style vectors are found by aggregating layer activations for input sentences from the target style @xcite . the basic assumption behind this is that llms internally adapt to the style of the input prompt when producing output, and thus, style vectors can be derived from its hidden states. these two methods are contrasted in fig. 2 and introduced in more detail in this section. ## experiments we compare both introduced approaches, i.e., training-based style vectors (sec. 3.1) and activation-based style vectors (sec. 3.2) in terms of how well they encode information about style (sec. 4.3) and the ability to steer the model's output (sec. 4.4). ## discussion and conclusion this work investigated vector representations associated with sentiments, emotion categories, and general writing styles that can influence the output style of llms. in a generative approach, style vectors were derived from steering vectors found in a training procedure and steered the model to produce samples in a desired style from scratch. in contrast, activation-based style vectors are derived from the activations of input prompts, which relies on the assumption that llms internally adapt the input style during the forward pass. steering vector training is much more expensive than simply recording the hidden layer activation during a single forward pass. therefore, the activation-based style vectors are the preferred approach for steering style in large language models, both in terms of performance and resource efficiency. we also found that, for factual prompts, the output can only marginally be influenced. it can be considered positive that one cannot easily dissuade the model from answering in a neutral tone to a factual prompt while still being adaptable if the input permits, especially in conversational settings. style vectors enable a continuous and adjustable modulation of the outputs of large language models. unlike prompt engineering, which offers more step-wise control over style intensities (like "write the answer in a positive way" versus "write the answer in a very positive way"), style vectors provide smoother transitions. this activation-based control is achievable because the vectors in activation engineering are constructed from known datasets. in contrast, traditional prompting may trigger activations that are unknown and inaccessible to the user, limiting the ability to fine-tune the output. furthermore, activation-based steering has the potential to generate new styles, expanding the possibilities beyond the constraints of pre-training knowledge inherent in prompt engineering. while prompt engineering relies on existing knowledge and often involves a trial-and-error approach, activation engineering opens up new avenues for style generation and customization. more complex styles, such as multidimensional composed styles, present unique challenges when approached through activation engineering. however, the advantages it offers, such as enhanced control over the output and the capacity to develop unique styles, significantly outweigh these initial challenges. it is important to note that these methods are not mutually exclusive; they can be combined to leverage each approach's strengths, enhancing our model's overall capability and flexibility. to the best of our knowledge, this is one of the first studies on steering language models beyond gpt-2 (in our case alpaca-7b @xcite ). results should, however, be transferable to any other type of llm with direct access to hidden layer activations. how to determine the exact influence of the weighting parameter λ (eq. 3) is still an open question. λ allows for nuanced style steering but, if chosen too large, leads the model to produce nonsense texts. moreover, this seems to depend on the domain (sentiment, emotion, writing style). we leave this for future research.