Model Surgery: Modulating LLM ’s Behavior Via Simple Parameter Editing (2025)

Abstract: Large Language Models (LLMs) have demonstrated great potential as generalist assistants, showcasing powerful task understanding and problem-solving capabilities. To deploy LLMs as AI assistants, it is crucial that these models exhibit desirable behavioral traits, such as non-toxicity and resilience against jailbreak attempts. Current approaches for detoxification or preventing jailbreaking usually involve Supervised Fine-Tuning (SFT) or Reinforcement Learning from Human Feedback (RLHF), which requires finetuning billions of parameters through gradient descent with substantial computational cost. Furthermore, models modified through SFT and RLHF may deviate from the pretrained models, potentially leading to a degradation in foundational LLM capabilities. In this paper, we observe that surprisingly, directly editing a small subset of parameters can effectively modulate specific behaviors of LLMs, such as detoxification and resistance to jailbreaking, with only inference-level computational resources. Experiments demonstrate that in the detoxification task, our approach achieves reductions of up to 90.0% in toxicity on the RealToxicityPrompts dataset and 49.2% on ToxiGen, while maintaining the LLM’s general capabilities in areas such as common sense, question answering, and mathematics.

## introduction llms have exhibited extraordinary capacities in language understanding, generation, and problemsolving @xcite @xcite . these advances have spurred llms' potential to serve as human-like assistants. despite their promising prospect, non-toxicity and safety have emerged as primary concerns for ap-plication. it is crucial to prevent llms from generating harmful content in response to malicious prompts or instructing on manufacturing harmful substances. current strategies for addressing undesirable behaviors typically involve fine-tuning on curated datasets @xcite @xcite or employing reward models focusing on toxicity and safety @xcite @xcite ). an alternative is machine unlearning, which uses methods like gradient ascent to remove previously learned undesirable behaviors @xcite @xcite . while these techniques are effective in promoting non-toxicity and safety, they necessitate the training of a llm. this training paradigm involves gradient computation, demanding considerable computational resources due to the billions of parameters in llms. employing a safety-focused reward model with rlhf requires an additional reference model and an optional reward model, increasing the demand for resources. additionally, previous studies indicate that models modified through sft and rlhf may deviate from the pretrained models, potentially leading to a degradation in foundational llm capabilities such as comprehension, reasoning, and common sense -an effect known as alignment tax @xcite @xcite . these shortcomings present significant challenges in regulating llm behavior, thereby hindering their use as safer and more user-friendly conversational assistants. to alleviate these problems, we modulate the behavior of llms through direct parameter editing rather than gradient descent. our work is motivated by the following observation: certain opposing attributes, such as toxic versus non-toxic or jailbreak versus non-jailbreak, can be clearly differentiated by simple linear separability in the hidden-layer space of llms. this phenomenon is illustrated we identify behavior regions as row vectors in gate projections that exhibit inverse alignment with the direction of the behavior probe. ## method llms show promise for developing ai assistants but exhibit problematic behaviors like generating toxic content, limiting their broader application. previous mitigation attempts such as fine-tuning or rlhf, can reduce unwanted outputs but are computationally expensive. moreover, extensive sft or rlhf can lead to alignment tax or catastrophic forgetting @xcite . ## experiment in this section, we conduct experiments to evaluate the effectiveness of our proposed model surgery technique across three distinct tasks: detoxification, jailbreak, and attitude adjustment. our aim is to address the following research questions: ## discussions does the probe direction truly represent the direction of undesirable behavior in the hidden space? in the jigsaw dataset, we performed gradient ascent on the toxicity classification loss in equation (1) using the trained, fixed probe. however, instead of training the probe, we adjusted the full parameters of the llm, thereby shifting the model's hidden state away from the probe's direction. as presented in ## conclusion and limitations this study presented a computationally-efficient methodology for modulating llm's behavior. the training process necessitates only a few hundred prompts in certain tasks and solely requires forward propagation, significantly reducing computational resource consumption. moreover, the proposed approach is extended to encompass a diverse array of behavioral attributes, including, but not limited to, toxicity, resistance to jailbreaking attempts, and the rectification of negative sentiments. in addition, our method does not change the performance of the model within a limited scope. despite our best efforts, there remain several aspects that are not covered in this paper. for example, although our method has provided some empirical analysis, we have not fully explored the underlying principles, which will be left for our future work.