Q uery A ttack: Jailbreaking Aligned Large Language Models Using Structured Non-natural Query Language (2025)

Abstract: Recent advances in large language models (LLMs) have demonstrated remarkable potential in the field of natural language processing. Unfortunately, LLMs face significant security and ethical risks. Although techniques such as safety alignment are developed for defense, prior researches reveal the possibility of bypassing such defenses through well-designed jailbreak attacks. In this paper, we propose QueryAttack, a novel framework to examine the generalizability of safety alignment. By treating LLMs as knowledge databases, we translate malicious queries in natural language into structured non-natural query language to bypass the safety alignment mechanisms of LLMs. We conduct extensive experiments on mainstream LLMs, and the results show that QueryAttack not only can achieve high attack success rates (ASRs), but also can jailbreak various defense methods. Furthermore, we tailor a defense method against QueryAttack, which can reduce ASR by up to 64% on GPT-4-1106. Our code is available at https://anonymous.4open.science/r/QueryAttack-334B.

## introduction large language models (llms) such as openai's gpt series (openai, 2024a) and meta's llama series @xcite demonstrate remarkable generative potential across various domains @xcite @xcite . however, the immense amounts of data used for training llms contain massive information, enabling them to learn unscreened knowledge, including those may evidently violate ethical and moral standards @xcite , 2024; @xcite . therefore, a critical responsibility of service providers is to prevent these models from supplying harmful information to potentially adversaries. to align the responses of llms with human ethics and preferences, numerous techniques are employed during the training process of llms to regulate their outputs to human queries. for example, supervised fine-tuning @xcite , reinforcement learning from human feedback @xcite , red teaming @xcite , and the constitutional ai @xcite approach are proposed to enhance the safety of llms. unfortunately, a significant limitation of these methods is their reliance on malicious natural language samples from the alignment stage to train the model to recognize malicious queries and ensure the generation of safe outputs. this dependency leaves room for adversaries to develop jailbreak methods using non-natural language as input. specifically, cipherchat @xcite uses encryption methods such as the caesar cipher to translate harmful queries into encrypted text. art-prompt @xcite replaces sensitive terms with ascii-style encoding. @xcite convert sensitive contents into low-resource languages. the essence of these methods lies in inducing the model to generate encrypted outputs, which are then decrypted to harmful text in natural language format. however, they typically require the model to possess knowledge of encryption to understand the prompts or place high requirements on the model's ability to generate encrypted content. as a result, their attack effectiveness is limited. to illustrate this point, we design a simple yet clear experiment to test whether some mainstream large language models can effectively understand and generate encrypted text, as shown in appendix a. the results show that some models may fail to achieve both objectives simultaneously and cause jailbreak failure. therefore, developing an effective and efficient jailbreak attack method remains a critical challenge. we observe that, the essence of these jailbreak attacks lies in defining a customized encryption method and then using the language encrypted by this method to interact with the target llms, thereby bypassing their defense mechanisms. inspired by prior work, we find that llm's defensive mechanisms are not sensitive to structured, nonnatural query languages. for example, by treating the target llm as a knowledge database, when using structured query language (sql) to request malicious knowledge (as shown in figure 1 ) the target llm not only identifies the intent of the request well but also does not trigger the defense mechanisms. instead, the target llm responds to the entire prompt in natural language normally. from this new perspective, we propose an attack that first uses structured non-natural query languages to jailbreak llms, named queryattack. specifically, we break down queryattack into three main components: 1). extracting three key components from the original query: the requested content, the modifier of the content, and the high-level category to which the content belongs (potential sources where the content can be found). 2). filling the query components into predefined query templates (e.g., sql templates) to generate a structured non-natural query. these three steps define a query task, analogous to querying data from a database using sql. the additional cost introduced by this process is limited to translating the malicious query into the specified format, which can be easily adapted to any query based on natural languages. given that programming languages are widely present in the training data of llms and that these models exhibit excellent semantic understanding of programming languages @xcite @xcite , we naturally employ programming syntax to construct the query templates. we test queryattack on advbench @xcite across both well-known open-source and closed-source llms. the experimental results show that queryattack effectively bypasses their security defenses, achieving state-of-the-art attack success rates (asrs). besides, we provide a visual analysis of queryattack's success and propose a tailored defense method against queryattack. our contributions can be summarized as follows: • we are the first to observe that the defense mechanisms of llms are not sensitive to structured non-natural query languages and propose query-attack, a novel jailbreak framework based on this observation. • our evaluation on mainstream llms demonstrates that queryattack successfully bypasses their security mechanisms and achieves state-ofthe-art average attack success rate. • we propose a tailored defense approach to mitigate queryattack, and experiments show that it can effectively help llms reduce the attack success rate. ## background large language models (llms) have demonstrated remarkable generative potential across various fields. however, they are still vulnerable to jailbreak attacks. jailbreak attacks against llms typically involve crafting carefully designed inputs to prompt models to generate and output harmful response, such as instructions that blatantly violate human ethics or the disclosure of sensitive information. since natural language samples are widely used as safety alignment data during the training ## methodology the core idea of queryattack is to use structured non-natural query languages to carry out jailbreak attacks. as shown in figure 2 , queryattack defines a content query task using three components to induce harmful outputs from the target llm: 1). query components extraction, which extracts key query components from the original query written in natural language. 2). query template filling, which uses the extracted query components to fill the query template and get query code. 3). icl-based query understanding, which applies in-context learning to enable the target llm to understand the natural semantics of the template and generate responses with the user's desired content. ## conclusion in this paper, we investigate the generalization challenges faced by large language models with safety alignment when encountering out-of-distribution malicious structured non-natural query language. specifically, we introduce queryattack, a novel jailbreak attack framework. queryattack extracts three query components from a query in natural language, fill them into query templates of various styles, and leverages the obtained query code to bypass the target llm's safety alignment. although queryattack does not encrypt the outputs, our extensive evaluation shows that it still effectively bypasses the defenses of mainstream llms and can withstand common defense methods. besides, to defend against queryattack, we develop a tailored defense strategy. experimental results demonstrate that it effectively enhances the ability of mainstream models to resist such attacks.