Extracting Latent Steering Vectors from Pretrained Language Models (2022)

Abstract: Prior work on controllable text generation has focused on learning how to control language models through trainable decoding, smart-prompt design, or fine-tuning based on a desired objective. We hypothesize that the information needed to steer the model to generate a target sentence is already encoded within the model. Accordingly, we explore a different approach altogether: extracting latent vectors directly from pretrained language model decoders without fine-tuning. Experiments show that there exist steering vectors, which, when added to the hidden states of the language model, generate a target sentence nearly perfectly (> 99 BLEU) for English sentences from a variety of domains. We show that vector arithmetic can be used for unsupervised sentiment transfer on the Yelp sentiment benchmark, with performance comparable to models tailored to this task. We find that distances between steering vectors reflect sentence similarity when evaluated on a textual similarity benchmark (STS-B), outperforming pooled hidden states of models. Finally, we present an analysis of the intrinsic properties of the steering vectors. Taken together, our results suggest that frozen LMs can be effectively controlled through their latent steering space.

## introduction leveraging large pretrained language models trained on massive web corpora has become the go-to approach to solve natural language processing tasks @xcite @xcite . as a result, controlling these models has become paramount as many applications of nlp technology require control over the generations of the model. prior work aims to learn how to control language models and falls in three categories: trainable decoding @xcite , smart-prompt design @xcite , and fine-tuning based on a desired objective @xcite . further, many works opt to train auto-encoder based models for controllable text generation @xcite @xcite . these approaches make controllability easier by learning a latent space that is more easily manipulated to encourage models to generate text corresponding to a target attribute such as positive sentiment in the case of sentiment transfer. we take a more direct approach and explore whether it is possible to extract latent vectors directly from pretrained language model decoders without fine-tuning. we call these vectors steering vectors and define the latent steering space of a sentence under a language model by the set of extracted steering vectors, which steer the model to generate that sentence exactly. during decoding, we add our steering vector to the hidden states of the language model to generate the target sentence. rather than training a model to learn steering vectors, we provide several methods to extract fixedlength steering vectors directly from pretrained language model decoders. experiments show that we can extract steering vectors effectively, achieving nearly perfect recovery for english sentences from a variety of domains without fine-tuning the underlying language model at all. next, we take our extracted steering vectors and explore whether they can be used for unsupervised sentiment transfer on the yelp sentiment benchmark @xcite . we find that adding an offset vector to extracted steering vectors performs comparably to carefully designed, autoencoderbased models. to see whether steering vectors encode semantics, we explore whether they can be used for unsupervised textual similarity. on the semantic textual similarity benchmark @xcite ), our steering vectors outperform extractive methods such as averaging language model figure 1: our approach adds a vector z steer to the activations of a pretrained transformer decoder to steer it to decode a desired target sentence. we experiment with adding z steer to different locations inside a gpt-2 model at different timesteps. experiments reveal that our approach can recover sequences nearly perfectly and that injecting the steering vector in the middle layers of the transformer stack performs best. layer normalizations and residual connections inside the transformer block are omitted for clarity. lastly, we analyze the intrinsic properties of the latent space of our steering vectors. experiments show that decoding from interpolations in the latent space produces meaningful output, and that steering vectors from different domains cluster together. also, we find that our methods do not simply memorize the target sequence like a naive compression algorithm, and instead leverage the model. taken together, our results suggest that frozen language models can be controlled effectively through their latent steering space. ## extracting steering vectors this section discusses our method for extracting a steering vector for a target sentence from a frozen, pretrained language model. throughout this paper, we use gpt2 as our language model and use its 117m parameter model size @xcite , although our approach can be directly applied to any transformer-based autoregressive language model decoder @xcite . ## do distances between steering vectors reflect sentence similarity? previously, we found there exist multiple steering vectors that recover a target sentence and that those steering vectors are close together. this indicates the potential for distances in token space to be reflected in distances in the latent space occupied by steering vectors. in this section, we explore whether distances relate to semantic similarity. to do so, we use the sts-b test dataset, which consists of sentence pairs and similarity scores. to evaluate our method we extract steering vectors for each sentence separately, compute cosine similarity, and then correlate cosine similarity with annotator similarity via spearman rank correlation. in figure 4 , we show how well extracted steering vectors perform when injected at different layers and at the first timestep in the transformer stack. this observation mirrors the results from the experiment on recovery effectiveness: middle layers in the transformer stack are ideal for steering, leading to perfect recovery and highest performance on semantic similarity. we outperform mean pooling the final hidden states of gpt2-117m and bertbase as well as averaged glove vectors. even though our method is fully extractive, cosine distances reflect semantic similarity well. we take our two best performing configurations, the 7th self-attention layer and the 7th feedforward layer, and compare with unsupervised methods for tex- ## analysis of properties 6.1 interpolation previous experiments indicate that the latent space occupied by steering vectors could be well-formed and smooth. to evaluate this qualitatively, we show linear interpolations of two pairs of steering vectors extracted from the yelp sentiment dataset in figure 5 . the space between the vectors look smooth with well-formed grammatical sentences that mix the content of two sentences effectively. the first interpolation (sentence pair 1) in figure 5 shows that the positive sentiment of the first sentence carries all the way to λ = 0.7, despite the content of the sentence changing to the second sentence. the second interpolation (sentence pair 2) in figure 5 indicates that the latent space could encode some semantics relating to time. the second sentence includes the word "young" and so the transition between the two in λ = 0.3, 0.4 combines the word "four" from the first sentence with the temporal component of "years ago" to relate the two sentences. lastly, for each individual sentence there exists a radius around it where those vectors also steer the language model to generate the same target sentence. this could indicate that sentences have a representative volume from which, if any vector was sampled, could recover the sentence. ## related work there exist many works, often using text-based autoencoders that try to induce a sentence representation space for controllable text generation by learning new models @xcite @xcite @xcite . our work concludes that we can extract steering vectors from pretrained models that have latent spaces that allow operations like this, without having to train any new models at all. other approaches control language models by adapting their hidden states using steerable layers, adapters, or steering their logits using auxiliary language models @xcite @xcite @xcite . our method differs from all of these: we extract steering vectors directly from a language model and operate on the latent space occupied by these vectors, never fine-tuning any component of the model. @xcite investigate whether lstm-based language models have sentence representations from which they can generate the original sentence. although this premise relates to our first question: can we extract steering vectors, we extend far beyond that showing that vector arithmetic in the context of unsupervised style transfer is effective in our latent steering space. ## conclusion in this paper we introduce a different approach to controllable text generation, where we extract latent steering vectors directly from a pretrained language model without fine-tuning. further, we find that our steering vectors lead to near perfect recovery on english sentences from a variety of domains. we show that vector arithmetic can be used in the context of unsupervised style transfer on the yelp sentiment dataset and styleptb benchmark, performing comparably to models tailored to these tasks. experiments reveal that distances between steering vectors reflect sentence similarity when evaluated on sts-b, outperforming extractive methods. finally, we analyze properties of the steering vectors. our results indicate that we can control frozen pretrained language models effectively through their latent steering space.