F ree C trl: Constructing Control Centers with Feedforward Layers for Learning-Free Controllable Text Generation (2024)

Abstract: Controllable text generation (CTG) seeks to craft texts adhering to specific attributes, traditionally employing learning-based techniques such as training, fine-tuning, or prefix-tuning with attribute-specific datasets. These approaches, while effective, demand extensive computational and data resources. In contrast, some proposed learning-free alternatives circumvent learning but often yield inferior results, exemplifying the fundamental machine learning trade-off between computational expense and model efficacy. To overcome these limitations, we propose FreeCtrl, a learning-free approach that dynamically adjusts the weights of selected feedforward neural network (FFN) vectors to steer the outputs of large language models (LLMs). FreeCtrl hinges on the principle that the weights of different FFN vectors influence the likelihood of different tokens appearing in the output. By identifying and adaptively adjusting the weights of attribute-related FFN vectors, FreeCtrl can control the output likelihood of attribute keywords in the generated content. Extensive experiments on single- and multi-attribute control reveal that the learning-free FreeCtrl outperforms other learning-free and learning-based methods, successfully resolving the dilemma between learning costs and model performance.

## introduction controllable text generation (ctg) focuses on directing language models to produce diverse and fluent sentences that adhere to predefined single or multiple attributes such as topics and sentiment @xcite @xcite . recent works on ctg can be roughly categorized into two groups based on their dependency on a learning process: learning-based methods and learning-free methods @xcite . learning-based methods usually involve training @xcite @xcite , fine-tuning @xcite @xcite , or prefix-tuning @xcite @xcite @xcite language models or discriminators based on attribute-specific data. despite their effectiveness, these approaches come with high computational costs for training and a dependency on vast, attribute-specific datasets, posing challenges for deployment in environments with limited data or computational capacity. only a few existing methods are learning-free, avoiding the need for training. for instance, k2t @xcite employs attribute-focused keywords to influence token output probability during generation. another method, mix&match @xcite , integrates diverse black-box experts as a probabilistic energy model to steer large language model (llm) outputs. these learning-free methods, despite bypassing the training process, tend to fall short in performance compared to advanced learning-based approaches. the analysis spotlights the classic dilemma in machine learning between the cost of learning and model performance, as shown in figure 1 . to overcome this obstacle, particularly in avoiding learning expenses for ctg while ensuring high performance, we propose freectrl: constructing control centers with feedforward layers for learning-free controllable text generation. freectrl's central idea is to manipulate ffn vectors foot_0 to regulate the outputs of llms, inspired by a recent finding that the tokens generated by llms can be attributed to the weights of vectors in ffn layers @xcite . specifically, the key principle is that increasing a single ffn vector's weight alters the output distribution, raising specific tokens' output probability. this strategy enables the targeted enhancement of certain ffn vector weights to raise attribute keywords' output probability, directing llm generation towards preferred attributes. our study first examines the possibility of this strategy by pinpointing three key characteristics of ffn vectors: 1) convergence: increasing the weight of an ffn vector can result in a stable and convergent output distribution in llms, thereby elevating the probabilities of specific tokens. 2) diversity: diverse ffn vectors can increase the output probabilities for most tokens in the llm vocabulary, covering keywords relevant to general attributes in ctg; 3) prompt-invariance: the observed effects of convergence and diversity remain consistent across different input prompts. these characteristics suggest ffn vectors can enable stable, diverse controls for llm outputs, directing sentence generation toward desired attributes. however, we also identify a major limitation of ffn vectors: the high-maintenance challenge, where adjusting their weights for precise control proves difficult. low weights lack the power to steer llms, while high weights compromise output diversity and fluency. to mitigate this, freectrl initially sets up control centers using ffn vectors for various attributes, then navigates llm output via a cycle of initialization, monitoring, adaptation, and filtering during the generation process. continuous monitoring ensures that token generation is assessed at each step, allowing for adaptive weight adjustments of the control centers. ultimately, a score-based filtering mechanism is employed to refine the outputs. notably, this framework requires no training or attribute-specific data yet surpasses the efficacy of advanced learning-based models. therefore, freectrl addresses the cost-performance dilemma, situating it at the optimal upper-left corner in figure 1 , denoting learning-free but high performance. our main contributions are summarized as follows: ## related work learning-based methods initial research efforts concentrate on adapting language models into attribute-conditional language models, utilizing methods like fine-tuning @xcite @xcite and reinforcement learning @xcite @xcite . weighted decoding stands out as another key strategy in ctg. these methods are primarily learning-oriented, involving updates to the model's hidden states based on decoded logits @xcite or training attribute discriminators to modify model output probabilities @xcite @xcite . amidst the growth of large language models like gptx @xcite and llama2 @xcite , recent techniques often preserve llm parameters and utilize lightweight fine-tuning methods such as prefix-tuning @xcite followed by decoding-time control @xcite @xcite @xcite . these methods generally necessitate a large volume of attribute-specific data and considerable computing resources for training either prefixes or discriminators. ## ffn vectors for ctg this section first details the theoretical basis for using ffn vectors to control llm outputs, then examines their characteristics and challenges. ## methodology to maximize the benefits and mitigate the limitations of ffn value vectors, we introduce freectrl: constructing control centers with feedforward layers for learning-free controllable text generation. freectrl begins by gathering attribute keywords, subsequently constructing a control center for each attribute. it then guides the llm to produce outputs relevant to the target attribute through a systematic pipeline of initialization, monitoring, adaptation, and filtering. the overall framework is illustrated in figure 4 . ## conclusions in this paper, we introduce freectrl, a learning-free approach for controllable text generation (ctg). freectrl employs ffn value vectors to establish control centers tailored to each attribute, enabling dynamic control via a structured process of initialization, monitoring, adaptation, and filtering. comprehensive experiments demonstrate that freectrl markedly outperforms both learning-based and learning-free methods.