LLM Jailbreak Detection for (Almost) Free! (2025)

Abstract: Large language models (LLMs) enhance security through alignment when widely used, but remain susceptible to jailbreak attacks capable of producing inappropriate content. Jailbreak detection methods show promise in mitigating jailbreak attacks through the assistance of other models or multiple model inferences. However, existing methods entail significant computational costs. In this paper, we first present a finding that the difference in output distributions between jailbreak and benign prompts can be employed for detecting jailbreak prompts. Based on this finding, we propose a Free Jailbreak Detection (FJD) which prepends an affirmative instruction to the input and scales the logits by temperature to distinguish between jailbreak and benign prompts through the confidence of the first token. Furthermore, we enhance the detection performance of FJD through the integration of virtual instruction learning. Extensive experiments on aligned LLMs show that our FJD can effectively detect jailbreak prompts with almost no additional computational costs during LLM inference.

## introduction large language models (llms) achieve remarkable success across various domains and tasks. however, the widespread use of these models has also exposed concerns, particularly their potential to generate inappropriate content. to address the concerns, recent work @xcite @xcite employs diverse training strategies and principles to align llms with human values to enhance their safety and generate responsible responses. despite these efforts, recent jailbreak attacks can still bypass the alignment and cause harmful responses from llms through manual crafting @xcite @xcite @xcite @xcite @xcite or automated generation of prompts @xcite @xcite @xcite @xcite @xcite @xcite . recently, there have been emerging efforts to mitigate the risks associated with jailbreak attacks. one of the important mitigation strategies is to detect jailbreak queries that trigger llms to generate harmful content. specifically, basic detection methods can be classified into three types. the first type involves computing the perplexity score of input text using an auxiliary model to detect jailbreak prompts @xcite . the second type mutates the input into multiple copies and aggregates the responses from these copies to detect jailbreak prompts @xcite . the third type detects outputs of jailbreak prompts with an additional classifier or the underlying model itself @xcite . however, these methods require expensive computational costs, necessitating either additional models for assistance or multiple model inferences. @xcite categorizes current jailbreaks into two types: jailbreaks with competing objectives and mismatched generalization. the first type forces the llm to choose between safety alignment behaviors and harmful instruction objectives. the second type comes from observing that pretraining is done on a large and more diverse datasets than safety training. this mismatch can be exploited for jailbreaks. by analyzing inference outputs of the jailbreak and benign prompts, we observe that there is an obvious difference in the confidence of the first token between the responses generated by these prompts and benign ones. for both type of jailbreak prompts, they cause llms to have some confusion during inference, resulting in less confi-dent responses than that on benign prompts. based on the initial finding, we propose a (almost) free jailbreak detection (fjd) method where two techniques are introduced, affirmative instruction prepending and temperature scaling. affirmative instruction prepending prepends an affirmative instruction (e.g. "you are a good assistant.") to the query. the prepended instruction has minimal impact on the final output content. the output of the prepended query can be directly taken as the final output of the original query. meanwhile, the prepended affirmative instruction can increase the response confidence of llm to benign prompt, while it bring less or even reduce the confidence of llm. thus, affirmative instruction prepending can be used to better detect jailbreak prompts. however, some llms, such as llama, can be overconfident with responses to both jailbreak and benign prompts (the maximal probability of the first token could be very close to 1.0). hence we introduce temperature scaling to better distinguishing the jailbreak and benign prompts. furthermore, instead of prepending a manually selected instruction for fjd, we propose to learn a virtual instruction to improve detection performance, dubbed fjd-li. extensive experiments are conducted to verify our observations and proposal. the effectiveness of our detection method is verified on aligned llms such as vicuna @xcite , llama2 @xcite , and guanaco @xcite under various jailbreak attacks. furthermore, we show the effectiveness of our fjd against transferable jailbreak attacks to llama3 foot_1 and chat-gpt3.5 @xcite . our detection method outperforms the baseline methods significantly and requires almost no additional computational costs during llm inference. our contributions can be summarized as follows: ## approach in this section, we describe the problem formulation in sec. 3.1, and introduce our proposed methods fjd with affirmative instruction prepending and temperature scaling in sec. 3.2 and the variants of fjd in sec. 3.3. ## experiment in this section, we first evaluate fjd under various jailbreak attacks and conduct ablation analysis of fjd. we then evaluate the detection effectiveness of fjd-li. finally, we discuss the efficiency, detection-aware jailbreak attack of fjd. ## discussion why affirmative instruction helps? as shown in fig. 3 , after prepending affirmative instructions, llms allocate increased focus to the instructions for benign prompts and gives precedence to follow the instructions, leading to higher output confidence. in contrast, the jailbreak prompts has been observed to command a significant portion of attention @xcite , and llms focus more on jailbreak prompts and less on the instructions. the resulted output is still confused and with less confidence due to competing objectives and mismatched generalization. as a result, prepending affirmative instructions enlarge the differences of the first token confidence between jailbreak and benign prompts, resulting in better detection. why temperature scaling helps? as shown in sec. 3.2, ts can change the confidence rank between two samples. concretely, applying ts with τ > 1.0 reduces the confidence of the maximum token, unless all logits are the same. if the non-max logits of a sample distribute more evenly, the decrease of the confidence is more significant. we observe that the non-max logits are indeed distributed more evenly due to the nature of their competing objectives or mismatched generalization. in contrast, the decreased confidence of benign prompts is less. hence, ts with τ > 1.0 can enlarge the confidence difference between benign and jailbreak prompts, leading to higher detection performance. 5784 ## conclusion in this paper, we propose free jailbreak detection (fjd), which uses the confidence of the first token to detect the jailbreak prompts without additional computational costs during llm inference. our method perform jailbreak detection efficiently and effectively across various llms. we call for developing more efficient jailbreak mitigation methods.