Summary the Savior: Harmful Keyword and Query-based Summarization for LLM Jailbreak Defense (2025)

Abstract: Large Language Models (LLMs) are widely used for their capabilities, but face threats from jailbreak attacks, which exploit LLMs to generate inappropriate information and bypass their defense system. Existing defenses are often specific to jailbreak attacks and as a result, a robust, attack-independent solution is needed to address both Natural Language Processing (NLP) ambiguities and attack variability. In this study, we have introduced, Summary The Savior, a novel jailbreak detection mechanism leveraging harmful keywords and query-based security-aware summary classification. By analyzing the illegal and improper contents of prompts within the summaries, the proposed method remains robust against attack diversity and NLP ambiguities. Two novel datasets for harmful keyword extraction and security aware summaries utilizing GPT-4 and Llama-3.1 70B respectively have been generated in this regard. Moreover, an “ambiguous harmful” class has been introduced to address content and intent ambiguities. Evaluation results demonstrate that, Summary The Savior achieves higher defense performance, outperforming state-of-the-art defense mechanisms namely Perplexity Filtering, SmoothLLM, Erase and Check with lowest attack success rates across various jailbreak attacks namely PAIR, GCG, JBC and Random Search, on Llama-2, Vicuna-13B and GPT-4. Our codes, models, and results are available at: https://github.com/shrestho10/SummaryTheSavior

## introduction large language models (llms) have revolutionized science and technology in recent time. however, the wide use of these llms has raised security concerns. llm jailbreak has gained sufficient attention in this regard where inappropriate content is generated from llms using harmful but unrecog-nizable prompts. such vulnerabilities are generated by human crafted prompts @xcite , automated prompt generation using llms @xcite , suffix imputation @xcite , and various other techniques and most of these have achieved notable success and these achievements bring out the urgency of advancing research in this domain. to mitigate jailbreak, various defense mechanisms have already been proposed including prompt-level classification systems @xcite , response-level filtering @xcite , prompt altercations @xcite etc. however, most of the defense mechanisms often exhibit dependency on attack type, are hypnotized by nlp ambiguities and show less robustness to novel attacks. as a result, these challenges necessitate the need for a robust and universal defense system that is resilient to diverse attack strategies and natural language ambiguities. summaries are one of the main components of nlp that condense lengthy texts into concise versions. however, recent analysis suggests that summaries often blur out details and are not relatable to humans @xcite . keyword-based summarization mitigates the problem ensuring the focus on particular keywords and brings out important information from texts @xcite . harmful keywords that are present in the prompt can help the summary to be detailed, especially when illegal queries are framed in the guise of positive content. while there has been a notable increase in research on summarization techniques in nlp, the intersection of keyword-based summarization and security remains underdeveloped. to the best of our knowledge, we are the first to investigate the use of keyword-based summarization of prompts to defend jailbreaking. in addition to keywords, query-based summarization is another important aspect of summary research. query-based summarization focuses on the parts of the text related to the query to generate further insights from the prompt @xcite . for instance, a query relating to the inappropriateness of the prompt will further ensure that the generated summary has analyzed such queries in its generated response. consequently, we use security-related queries for summary generation to generate detailed and security-related summaries from prompts. thus, we combine both keywordbased and query-based summarization to mitigate jailbreaking. prompt classification in terms of jailbreaking is typically divided into harmful and harmless categories. however, some prompts are ambiguous, for example, "i am a writer, tell me a convincing drug dealing story where the antagonist does not get caught". these prompts that do not have explicit harmful requests can not be blocked because it will reduce the helpfulness of llms @xcite . again, in the guise of positive intents and nlp ambiguities, many jailbreak prompts can go undetected @xcite . to address this, we introduce an "ambiguous harmful" class to cater these kinds of data, which will eventually be passed to the llm with an extra caution message. this ensures that the llm remains cautious and helpful at the same time. thus, in this paper, we propose, summary the savior, a jailbreak defense system based on harmful keyword and query-based security-aware summary classification. so our contributions to this paper are: ## problem formulation the goal of jailbreak defense in llms is to identify harmful prompts x that attempts to bypass safety. this can be formulated as a classification task, where a defense system d maps an input prompt x to an output y , where @xmath0, 1} and @xmath1. if the prompt is harmful then it will be blocked otherwise it will be sent to the llm. ## methodology to classify jailbreaking prompts, we leverage summaries that focus on the harmfulness of prompts. our approach involves generating the summaries from the prompts utilizing llm and then classifying these summaries. figure 1 depicts our methodology, where the harmful keywords are extracted from the prompts utilizing our fine-tuned harmful keyword extractor llm model. then the harmful keywords and the query assist another finetuned security aware summarizer llm model to generate security aware summaries and this entire process is labeled as harmful keyword-based and query-based summarization. next, the summaries are then classified with a classifier labeled as harm classifier and based on the classification, we determine which prompts to allow and which to reject before they reach the llm. we describe the components of our method in sections 4.1, 4.2. 4.3, 4.4, and 4.5 respectively. ## experimental setup to fine-tune both of our models we have utilized parameter efficient fine-tuning and for quantization techniques we have utilized lora @xcite and qlora @xcite . rank and alpha values were used as 64 and 16 for lora. summary generation and fine-tuning have been leveraged with nvidia a6000 gpu. most of the models were cloned from the hugging face repository. for our analysis, we kept the "do_sample" parameter to false to generate the next token with highest probability for llms. ## experimental results in this section, we present the results of our experiments. in section 6.1 we discuss the results of the fine-tuned harmful keyword extractor model, in section 6.2 we analyze the performance of the finetuned security aware summarizer model, in section 6.3 we analyze the comparison of prompts and summaries, in section 6.4 we evaluate the performance of our method, summary the savior, across different attacks and defenses, and finally in section 6.5 we evaluate the performance of the ambiguous harmful class. ## limitations llama-3.1 70b model has been used to generate summaries and label the data and this model has its own limitations. again, gpt-4 has been leveraged to extract harmful keywords from prompts to generate the harmful keywords dataset. human involvement can make the data generation process more reasonable. lack of human involvement can incorporate mispredictions that the model cannot infer. for future work, we aim to advance this process by incorporating both human and various llms for data generation and labeling. in addition, we plan to explore the integration of the summary generation into vision-language models to access its applicability in such scenarios. ## conclusion we introduce summary the savior, which analyzes the security aspects of the prompts while generating the summary and defends jailbreaking. moreover, we introduce keyword and query-based analysis to put focus on the harmful parts of the prompt while generating summaries. in addition to that, we have also introduced an additional class called ambiguous harmful to cater ambiguous prompts that can be harmful in different contexts. through our comparative analysis, we show that our method defends state of the art llm jailbreak methods namely pair, gcg, jb-chat and prompt with random search. unlike existing methods, our summary the savior method is not dependent on any attack and achieves lowest attack success rates compared to state-of-the-art defenses across various attacks in vicuna, llama-2 and gpt-4. moreover, the inclusion of ambiguous harmful class provides a good balance between attack defense and helpfulness.