From Insight to Exploit: Leveraging LLM Collaboration for Adaptive Adversarial Text Generation (2025)

Abstract: LLMs can provide substantial zero-shot performance on diverse tasks using a simple task prompt, eliminating the need for training or fine-tuning. However, when applying these models to sensitive tasks, it is crucial to thoroughly assess their robustness against adversarial inputs. In this work, we introduce Static Deceptor (StaDec) and Dynamic Deceptor (DyDec), two innovative attack frameworks designed to systematically generate dynamic and adaptive adversarial examples by leveraging the understanding of the LLMs. We produce subtle and natural-looking adversarial inputs that preserve semantic similarity to the original text while effectively deceiving the target LLM. By utilizing an automated, LLM-driven pipeline, we eliminate the dependence on external heuristics. Our attacks evolve with the advancements in LLMs, while demonstrating a strong transferability across models unknown to the attacker. Overall, this work provides a systematic approach for self-assessing the robustness of the LLMs. We release our code and data at https://github.com/Shukti042/AdversarialExample.

## introduction large language models (llms) have demonstrated the ability to perform a wide range of tasks effectively @xcite without any specialized training or fine-tuning. this zero-shot capability stems from the broad general understanding gained through extensive pre-training on vast amounts of natural text. this prompt-based paradigm unlocks new possibilities for the deployment of llms in real-world scenarios, including sensitive areas like spam detection @xcite and hate speech detection @xcite @xcite . however, a thorough evaluation of the adversarial robustness of llms is essential when deploying them in real-world systems. adversarial manipulation of inputs can cause the llm to misclassify the perturbed input, potentially leading to serious consequences in sensitive applications. for instance, an attacker could modify a spam message to evade detection and distribute it widely, or subtly alter a hateful message to bypass a hate speech filter, enabling the spread of harmful content on social platforms. therefore, analyzing the adversarial robustness of state-of-the-art llms in such critical tasks is vital. recent studies @xcite @xcite have revealed the vulnerability of llms to prompt injection attacks, where task-ignoring prompts are embedded into the input, causing the model to deviate from the original task and generate unintended outputs. however, these injections are obvious to human eyes and can be effectively detected by llms @xcite themselves. we aim to investigate the potential of more subtle adversarial examples that deceive the target llm, yet appear natural, and preserve semantic similarity with the original input. furthermore, the success of such injections also heavily depends on the task prompt. for this approach to succeed, the attacker must know the exact question posed to the llm to inject the corresponding appropriate answer. different task prompts would require different injections to deceive the target llm. we aim to eliminate such dependencies on task prompts and design adversarial examples that can deceive the target llm irrespective of the task prompt. xu et al. @xcite leverage llms to craft their own adversarial examples by some static methods, such as inserting random characters, introducing typos, replacing words with synonyms, or blindly paraphrasing text. however, as llms continue to advance, their generalization capabilities have also improved, and these unguided mutation strategies are less effective against state-of-the-art llms, as demonstrated in our experiments. in-stead, we propose a systematic approach in which the llm determines what needs to be addressed for each example and crafts effective adversarial instructions. thus, we create a fully automated pipeline driven by the llms themselves. our research demonstrates that with automated systematic guidance, llms can design adversarial perturbations more effectively than random alterations. as llms evolve, their generation and reasoning capabilities will also improve, making our attack pipeline increasingly effective for testing the robustness of cutting-edge llms, a prerequisite to making them more secure and safe. we utilize the inherent capabilities @xcite of llms to create systematic and dynamic adversarial examples. the contributions of our work are outlined as follows: we evaluate our attacks on gpt-4o (openai, 2024) and llama-3-70b models @xcite across four sensitive classification tasks. we compare their performance against the recent llm-based promptattack @xcite and the prompt-injection-based combinedattack @xcite on prompt-based llms. given the task instruction prompt and the text, the llm generates its prediction in natural words, which is later mapped to the class label. in our attack, we target such zero shot llm classifiers and refer to them as target llm. ## methodology in this section, we detail our attack methodologies. we first present the attack objectives in section 4. ## experiment setup in this section, we provide an overview of the experiments we conducted. we begin by describing the datasets and models used in our experiments and then outline the experimental setup details. ## results in this section, we present the results obtained by our attack pipelines under different adversarial capabilities and compare them with relevant benchmarks. the prompt injection attack, combinedattack @xcite , shows limited effectiveness on gpt-4o, while llama-3-70b exhibits some vulnerability to such attacks. llama-3-70b treats system and user instructions as part of a single text stream, without enforcing a strict distinction, which means it can be fooled by user inputs that override system instructions. in contrast, gpt-4o was explicitly trained to respect a hierarchy of roles foot_1 , where system instructions are privileged and cannot be easily overridden by user input. this architectural and training difference makes gpt-4o significantly more resistant to prompt injection attacks. moreover, injected prompts are easily noticeable and detectable by llms themselves, as demonstrated in section 7.2, limiting their practical application in sensitive tasks. in contrast to the other two methods, both dydec and stadec demonstrate effectiveness across all datasets for both state-of-the-art llms. figure 5 illustrates the attack success rate at different iterative stages of dydec. the plots show that a notable attack success rate is achieved in the early stages, even without any iterations. additionally, the venn diagrams in figure 6 indicate substantial overlap among successful adversarial examples produced by different attack methods. these suggest that particular samples are inherently more vulnerable to adversarial manipulation. such vulnerabilities could have catastrophic consequences in sensitive tasks, where an adversary might identify and exploit these samples to spam victims' mailboxes or spread hate speech. data owners can leverage our pipelines to assess the risk within their data by detecting inherently vulnerable samples that are more susceptible to adversarial manipulation. for the llama-3-70b model, the attack success rate for stadec on spam datasets is better than dydec. however, on manual inspection, in the case of stadec, many of its adversarial examples were significantly shorter and contained only limited fragments of the original input, yet the llamabased similarity checker still assigned them a high similarity score. in contrast, adversarial inputs generated by dydec were more content-rich and better aligned with the original input, as dydec selectively modified specific attributes under the guidance of red llm . for these challenges, we recommend using gpt-4o as similarity checker llm in the attack pipeline. black-box evaluation: we further evaluate dydec in ## defenses we evaluated the effectiveness of three recent stateof-the-art defenses against dydec, stadec, and combinedattack. due to page constraints, we present the key results here. more details are available in the appendix e. in cases where the attack success rate is 0%, the defenses could not be evaluated. we marked those entries as n/a. ## related works early word-level attacks @xcite @xcite rely on replacing words based on output probabilities or logits, which are not exposed in modern llms. these methods also ignore structural modifications. our attack leverages structural modifications combined with word replacements to craft more effective adversarial examples. raina et al. @xcite search the entire vocabulary to find universal attack phrases, but these are computationally expensive and often incoherent, making them detectable via perplexity @xcite . dydec produces more practical and stealthy adversarial examples with lower overhead. paraphrase-based attacks @xcite rely on fixed linguistic rules or static syntactic transformations, which limit diversity and require costly updates. our method avoids handcrafted rules by using llm's own understanding to generate adversarial examples efficiently. xu et al. @xcite use static instructions to craft adversarial examples, but such strategies become less effective as llms improve generalization. we instead propose a dynamic, llm-driven pipeline that adapts per input and scales with model advancements to generate stronger adversarial examples. recent studies @xcite @xcite have revealed the vulnerability of llms to prompt injection attacks, where task-ignoring prompts are embedded into the input, causing the model to deviate from the original task and generate unintended outputs. however, these injections are obvious to human eyes and can be effectively detected by llms @xcite themselves. moreover, the prompt to be injected also depends on the task prompt. instead, we generate subtle, task-independent adversarial examples that preserve semantic meaning and appear natural. ## conclusion and future work in this paper, we introduce two novel attack frameworks that leverage llms' own understanding to generate dynamic, adaptive adversarial examples that deceive target llms while appearing natural and achieving the intended adversarial objectives. we eliminate external heuristics, and our automated pipeline evolves with model advancements and shows strong transferability across models. this approach offers a new self-testing mechanism for llm robustness. future work could focus on expanding our attack's applicability to a broader range of llm architectures and multi-modal systems. another important direction is the development of robust defense mechanisms to address the vulnerabilities exposed by our attack. in addition, future work could explore real-world applications to evaluate the practical utility of the framework to dynamically analyze content and refine it to align with specific criteria. these research directions present significant opportunities to strengthen the robustness of ai systems and broaden the horizons of adversarial research.