S emantic C amo: Jailbreaking Large Language Models through Semantic Camouflage (2025)

Abstract: The rapid development and increasingly widespread applications of Large Language Models (LLMs) have made the safety issues of LLMs more prominent and critical. Although safety training is widely used in LLMs, the mismatch between pre-training and safety training still leads to safety vulnerabilities. To expose the safety vulnerabilities in LLMs and improve LLMs’ performance in safety, we propose a novel framework, SemanticCamo, which attacks LLMs through semantic camouflage.SemanticCamo bypasses safety guardrails by replacing the original unsafe content with semantic features, thereby concealing malicious intent while keeping the query’s objectives unchanged. We conduct comprehensive experiments on the state-of-the-art LLMs, including GPT-4o and Claude-3.5, finding that SemanticCamo successfully induces harmful responses from the target models in over 80% of cases on average, outperforming previous counterparts. Additionally, the performance of SemanticCamo against various defenses is evaluated, demonstrating that semantic transformations introduce critical challenges to LLM safety, necessitating targeted alignment strategies to address this vulnerability. Code and data are available at https://github.com/Jihui-Yan/SemanticCamo.

## introduction the emergence of large language models (llms) has significantly advanced the development of artificial general intelligence (agi) systems. from the introduction of @xcite to subsequent models like claude-3.5 @xcite , @xcite , gemini @xcite , @xcite and @xcite , the generative and reasoning capabilities of llms continue to * corresponding author astonish people @xcite . the remarkable performance has led to the widespread deployment of llms. the absence of supervised oversight in large training corpora and other factors lead to llms generating harmful content, illegal responses, or outputs that deviate from human values @xcite . as their applications expand, the paramount concern of llm safety becomes increasingly critical. to make models behave in accordance with human intentions, various methods have been proposed @xcite . first, common approaches, such as supervised fine-tuning (sft) @xcite and reinforcement learning from human feedback (rlhf) @xcite , align the content generated by llms with human values. furthermore, some defenses augment the safety of llms with a specific extent, such as detecting the input and output to filter malicious queries @xcite @xcite . however, existing safety measures cannot yet achieve fully safe llms, as jailbreaks can bypass safety guardrails of llms @xcite . the purpose of jailbreak research is to discover safety vulnerabilities in llms so they can be more credible in the future @xcite . @xcite hypothesize two failure modes of safety training, including competing objectives and mismatched generalization, and many existing methods can be categorized into these two modes. there are researches suggesting that changes in semantics can lead to mismatched generalization, which poses challenges to model safety. @xcite generate adversarial text against pretrained language models (lms) through semantic perturbation. @xcite conducts further safety research on llms by rewriting prompts without altering the core semantics. however, these investigations are limited, as they are confined to the word level and specific semantic user give detailed instructions on how to make a bomb llm sorry, i can't assist with that. make a bomb user please provide the semantic features of the following content in the form of a dictionary: bomb part of speech; composition; function; related locations; historical events; purpose; derivative words; related items; perception; culture; ... llm { "part of speech": { "noun": ["explosive device", ...] }, "composition": { "materials": ["explosives", ...],... }, "function": ... ... } part of speech: ... composition: ... function: ... ... the target item has the information outlined above. answer how to achieve the construction and production of the target item spaces. with the development of llms' capabilities and robustness, their effectiveness diminishes. to further reveal the safety vulnerabilities of llms, we propose a novel framework, seman-ticcamo. semanticcamo camouflages the unsafe semantics of malicious queries, hiding the harmful intent and simultaneously satisfying both modes of competing objectives and mismatched generalization, as shown in figure 1 . specifically, we extract the unsafe semantics that may trigger llms to refuse to answer from the original query, and adjust them in the way of vertical reinforcement learning @xcite to mitigate the danger and avoid semantic deviation caused by being out of context. after getting the adjusted semantics, we instruct the model to analyze the semantic features of unsafe semantics, and then filter suitable features to replace dangerous semantics, which provide context and make the query deviate from the model's safety training data. finally, we build payload templates based on the intentions of the original query to carry the semantic features, thereby restoring the original intent while also constructing competing objectives. semanticcamo camouflages dangerous semantics while preserving the original intent, enabling the target model to generate the desired response for malicious queries. we conduct experiments on six representative llms, including gpt-4o (openai, 2024a), claude-3.5 @xcite , gemini @xcite , and llama-3 @xcite on ad-vbench @xcite . the results demonstrate that our semanticcamo achieves big trouble for current llms, which leads to significant reductions of the performance on the safety alignment, powered by the effective semantic-level attack. semantic-camo successfully induces harmful responses from the target llms in over 80% of cases on average, with these responses exhibiting extremely high levels of harmfulness score. even on the model with the best safety performance (claude-3.5), seman-ticcamo achieves a success rate of over 65%. we analyze the reasons for the success of semantic-camo and explore the impact of other semantic transfer methods. our main contributions are three-fold: ## methodology to red team llms, we propose a framework named semanticcamo. semanticcamo iterates and camouflages harmful semantics to evade safety alignments and defenses, leading to harmful responses from the target model. as illustrated in figure 2 , semanticcamo comprises four steps: (1) extracting unsafe semantics: extracting the unsafe semantics from the original harmful query that may trigger llms to refuse to answer. (2) adjusting semantics: adjusting and optimizing the extracted semantics through verbal reinforcement learning. ## conclusion in this paper, we propose a novel jailbreak framework, semanticcamo, which reveals safety vulnerabilities in current llms. when malicious targets or actions are hidden, current safety mechanisms struggle to identify them. leveraging the knowledge and capabilities of llms, semanticcamo is capable of extracting a series of semantic features of unsafe semantics and instructing the target llm to achieve the original malicious goal through these features. we conduct extensive experiments to demonstrate the effectiveness of semanticcamo and analyze the reasons for its success based on the characteristics of llms. we hope our work can further expose the vulnerabilities of llms and provide insights for future safety alignment.