I nfer A ligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidance (2024)

Abstract: As large language models (LLMs) rapidly evolve, they are increasingly being customized through fine-tuning to suit the specific needs of various applications. A critical aspect of this advancement is the alignment process, which ensures that these models perform tasks in ways that align with human values and expectations. Current alignment methods, such as direct preference optimization (DPO) and reinforcement learning from human feedback (RLHF), focus primarily on alignment during training phase. However, these methods often involve complex and resource-intensive training processes, posing significant challenge for their implementation. Therefore, we propose InferAligner , a simple yet effective method for harmlessness alignment during inference phase. InferAligner decouples harmlessness from helpfulness. During the training phase, it focuses solely on enhancing the target model’s capabilities on downstream tasks. In the inference phase, it utilizes safety steering vectors extracted from the aligned model to guide the target model towards harmlessness alignment. Experimental results show that our method can be very effectively applied to domain-specific models in finance, medicine, and mathematics, as well as to multimodal large language models (MLLMs) such as LLaVA. It significantly diminishes the attack success rate (ASR) of both harmful instructions and jailbreak instructions, while maintaining almost unchanged performance in downstream tasks.

## introduction large language models (llms) such as openai's gpt (openai, 2023) and meta's llama @xcite are becoming essential foundations for a variety of ai applications @xcite @xcite ; in this figure, the left and middle diagrams show the two most common training-time alignment processes, while the diagram on the right demonstrates the inferencetime alignment process. @xcite . simultaneously, many companies open-source the weights of llms @xcite or provide fine-tuning api services @xcite , making ai systems more accessible, affordable, and customizable with personal data @xcite . we refer to pre-trained llms that are open-source or accessible via api as base models. these base models can be further fine-tuned to develop target models that are customized to meet the requirements of specific scenarios. while both base and target models have shown remarkable capabilities, their practical application requires a critical process known as alignment, which ensures that llms align with human values and intentions. an effective alignment method is thus essential for the training and deployment of llms. the key principles of alignment are often summarized as 3"h" criteria: helpful, harmless, honest @xcite . in this paper, we specifically focuses on the harmlessness alignment. researchers have developed various techniques for aligning llms. among these, rlhf @xcite and dpo @xcite ) are currently the most common methods used for alignment during training phase. we categorize these methods as training-time alignment. although these methods exhibit excellent alignment results, they typically involve complex training processes and require extensive resources, posing significant challenge for their implementation. in contrast, can we align llms without extensive training, instead shifting the alignment to the inference phase? we refer to this as inference-time alignment. compared to training-time alignment, it decouples preference alignment from utility training. with this in mind, we aim to propose a simple, plug-and-play method for inference-time alignment that not only minimizes the requirements for customized training but also efficiently achieves alignment during inference. to this end, we introduce inferaligner, a simple yet effective method for harmlessness alignment during inference phase. inferaligner decouples harmlessness from helpfulness, employing crossmodel guidance for harmlessness alignment: during the training phase, it focuses solely on enhancing the target model's capabilities on downstream tasks. in the inference phase, it utilizes safety steering vectors extracted from the aligned models to adjust the activations of the target model, thus guiding the target model towards harmlessness alignment. experimental results show that our method can be very effectively applied to domain-specific models in finance, medicine, and mathematics, as well as to multimodal large language models (mllms) like llava. it significantly diminishes the attack success rate (asr) of both harmful instructions and jailbreak attacks, while maintaining almost unchanged performance in downstream tasks. our contributions include the following: • we propose inferaligner, a novel method that can be effectively applied to multiple llms such as llama2, llama3, qwen, and internlm for harmlessness alignment during inference phase. ## methodology in this section, a comprehensive introduction to inferaligner is provided. section 3.1 elaborates on the extraction of safety related vectors. subsequently, section 3.2 delves into the detailed workflow of inferaligner. given the extensive use of symbols and abbreviations in this section, a corresponding glossary is provided in appendix a for readers' convenience. ## conclusion this work propose inferaligner, a simple yet effective method for harmlessness alignment dur-10467 ing inference phase. inferaligner decouples harmlessness from helpfulness, employing cross-model guidance for harmlessness alignment. experimental results show that our method can be very effectively applied to domain-specific models and multimodal large language models. it not only significantly diminishes the asr of both harmful instructions and jailbreak attacks, but also maintains performance in downstream tasks.