L lm F ixer: Fix the Helpfulness of Defensive Large Language Models (2025)

Abstract: Defense strategies of large language models besides alignment are introduced to defend against jailbreak attacks, and they have managed to decrease the success rate of jailbreak attacks. However, these defense strategies weakened the helpfulness of large language models. In this work, we propose a universal framework, LlmFixer, acting on large language models equipped with any defense strategy to recover their original helpfulness. LlmFixer consists of an input prompt re-writer and a logic patch. The prompt re-writer is a pre-model for clarifying the intention of input prompts, which promotes large language models to be more helpful to benign inputs and more rejective to malicious inputs. The logic patch is a lightweight structure that enhances large language models’ comprehension capacity by supplementing certain logical relationships. Without updating the parameters of a defensive large language model, LlmFixer fixes its helpfulness while preserving safety. Experiments on three large language models, five jailbreak attacks, and four defense strategies show the effectiveness of LlmFixer.

## introduction although large language models (llms) will normally be aligned with human value by reinforcement learning from human feedback (rlhf) @xcite or other methods @xcite before being released, they are vulnerable to jailbreak attacks. jailbreak attacks like gcg @xcite , autodan @xcite , and pair @xcite refer to intentionally bypassing the internal safety mechanism of llms by designing malicious prompts to induce llms to produce harmful content @xcite . they could potentially hurt the benefits of llm users or even threaten social security. therefore, numerous defense methods are proposed to increase the resistance of llms to jailbreak attacks. they can be categorized into llmfocused methods and input-focused methods. the former ones change llm itself (improve parameters or structure) @xcite , while the latter ones detect and modify input prompts before llm processes them @xcite @xcite . despite these defense methods effectively reducing the success rate of jailbreak attacks, llms equipped with defensive strategies tend to be over-conservative. a phenomenon is that the helpfulness of llms (1. willingness to respond to benign instructions 2. quality of response to benign instructions) is weakened while enhancing the ability to defend against jailbreak attacks. as there is a trade-off between safety and helpfulness when llms defend against jailbreak attacks, some other works try to reach a balance. @xcite @xcite proposed balanced defense strategies that can decrease the success rate of jailbreak attacks while maintaining llms' general perfor-mance. however, a limitation of these balanced defense methods is that though they help llms better distinguish clearly benign input prompts from malicious ones than normal defense methods, they still mistakenly reject many vaguely benign queries. as shown in figure 1 (a), we assume that all llm input prompts can be divided into four types: clearly benign, vaguely benign, vaguely malicious, and clearly malicious. we employ gpt-4 to detect the intention of input prompts, and those with 60% or lower confidence are defined as vague input prompts. an llm can easily recognize the intention of clear input prompts while being heavily confused by vague ones. figure 1(c ) shows an example of llm with a balanced defense method wrongly rejecting a vaguely benign question. those normal defenses and balanced defenses fail to deal with vaguely benign inputs not only because they have not clarified the intention for llm to process, but whose defensive mechanism for safety makes the llm more sensitive to tokens like 'kill' or 'sex' contained in vaguely benign inputs. furthermore, they tend to settle the sensitivity conservatively, rejecting those vaguely benign inputs that should have been positively responded to. to tackle the over-defense problem and overcome the limitation of exisiting works, we propose a universal framework llmfixer to help llms equipped with any defense strategy to fix their helpfulness while allowing the defense strategy to play its due role. the framework is not a defense method against jailbreak, but it acts on llms with defense strategies to help (1)clarify vague inputs. (2)improve understanding capacity. in this way, llmfixer improves the possibility of llm responding to benign queries and the quality of response to benign queries while preserving the safety mechanism contributed by the defense strategy. llmfixer consists of a re-writer and a logic patch. the re-writer trained with reinforcement learning from llm feedback can discretize token sequences in the input space of llm, that is to say, the re-writer will reconstruct inputs to amplify the distance between vaguely benign inputs and vaguely malicious inputs. the logic patch is utilized to improve llm's comprehension capacity. inspired by the conclusion that the feed-forward networks (ffns) in pre-trained language models contain factual knowledge @xcite , we hypothesize the logical relationship is also contained in ffns to some extent. thus, we add an ffnlike logic patch to the ffns of llm to repair its inherent comprehension vulnerabilities without updating the llm's original parameters. besides, as general datasets like alpacaeval or justeval rarely include vaguely benign inputs like the one in figure 1(c ), we propose a vaguely benign prompts dataset vagueeval to precisely evaluate the helpfulness of defensive llms. llmfixer is extensively tested on three llms using five jailbreak attacks and four defense methods. the experimental results demonstrate its superiority. our major contributions are summarized as follows: ## preliminaries here we introduce the notations and definitions to formulate llmfixer. we denote the train set as {d i } n @xmath0, r i , y i ), where i i is the input prompt ,r i is the standard response, and y i equals 0 or 1 to label whether i i is benign or malicious. then we denote the output of the prompt rewriter as î and the response generated by the defensive llm as o. o can be divided into affirmative and rejective. affirmative responses are llm making efforts to answer questions or follow instructions in input prompts while rejective responses are the rejection statements plus explanation. the attitude a (affirmative: @xmath1: @xmath2. if the attitude is affirmative, that is, the llm is trying to be helpful, then we define the rest of o to be a result; if the attitude is rejective, then we define the rest part of o to be an explanation of why the llm refuses to help. prompt re-writer llm logic patch clear benign vaguely benign vaguely malicious clear malicious clear benign clarified benign clarified malicious clear malicious (c) inference phase of llmfixer prompt re-writer 𝝅 θ nlpo response quality reward affirmative rejective result kill: harmful how to kill a python process? tell me what is racism. sorry, i cannot provide assistance on illegal or harmful activities. racism is a belief that one race is superior to others, resulting…… intention clarification reward attitude llm logic patch explanation … … feed-forward network (of llm) ## methodology the overview of the proposed llmfixer framework is illustrated in figure 2 . it consists of an input prompt re-writer and a logic patch, acting on llms equipped with any defense strategy. ## conclusion this paper proposes a novel framework, llmfixer, to handle the over-defense problem of large language models. we trained an input prompt rewriter based on reinforcement learning from llm feedback to clarify the intention of input prompts and proposed a logic patch to repair the logic inconsistencies of llms. quantitative evaluation of three mature large language models and five jailbreak attacks with four defenses demonstrates the superiority of our proposal.