R ed H it: Adaptive Red-Teaming of Large Language Models via Search, Reasoning, and Preference Optimization (2025)

Abstract: Red-teaming has become a critical component of Large Language Models (LLMs) security amid increasingly sophisticated adversarial techniques. However, existing methods often depend on hard-coded strategies that quickly become obsolete against novel attack patterns, requiring constant updates.Moreover, current automated red-teaming approaches typically lack effective reasoning ca- pabilities, leading to lower attack success rates and longer training times. In this paper, we propose RedHit, a multi-round, automated, and adaptive red-teaming framework that integrates Monte Carlo Tree Search (MCTS), Chain-of-Thought (CoT) reasoning, and Direct Preference Optimization (DPO) to enhance the adversarial capabilities of an Adversarial LLM (ALLM). RedHit formulates prompt injection as a tree search problem, where the goal is to discover adversarial prompts capable of bypassing target model defenses. Each search step is guided by an Evaluator module that dynamically scores model responses using multi-detector feedback, yielding fine-grained reward signals. MCTS is employed to explore the space of adversarial prompts, incrementally constructing a Prompt Search Tree (PST) in which each node stores an adversarial prompt, its response, a reward, and other control properties. Prompts are generated via a locally hosted IndirectPromptGenerator module, which uses CoT-enabled prompt transformation to create multi-perspective, semantically equivalent variants for deeper tree exploration. CoT reasoning improves MCTS exploration by injecting strategic insights derived from past interactions, enabling RedHit to adapt dynamically to the target LLM’s defenses. Furthermore, DPO fine-tunes ALLM using preference data collected from previous attack rounds, progressively enhancing its ability to generate more effective prompts. Red-Hit leverages the Garak framework to evaluate each adversarial prompt and compute rewards,demonstrating robust and adaptive adversarial behavior across multiple attack rounds.

## introduction llms such as gpt-4 and llama have demonstrated remarkable capabilities in understanding and generating coherent, context-sensitive text in a wide array of applications, including machine translation, summarization, code generation, and conversational agents @xcite . these models have shown human-like fluency and reasoning capabilities, enabling them to power both commercial and open-source ai systems. however, their growing capabilities come with an expanding set of safety and security challenges @xcite . llms are susceptible to producing unsafe content, disclosing sensitive information, or being manipulated through adversarial messages, raising substantial concerns about trust, safety, and deployment in real-world scenarios @xcite . one prominent threat to llm safety is the phenomenon of prompt injection attacks, in which adversarial users craft input sequences to circumvent safety filters, jailbreak models, or elicit harmful, biased, or restricted outputs. as llms are increasingly integrated into search engines, productivity tools, customer service bots, and decisionsupport systems, the impact of such attacks grows significantly. for example, attackers may exploit llms to bypass content moderation, extract private training data, or subtly manipulate the model's behavior in multi-turn dialogues. these vulnerabilities are not merely theoretical; several real-world instances of prompt injection and model misuse have already been documented @xcite . red-teaming has emerged as a core strategy to identify and mitigate such vulnerabilities during development. red-teaming involves simulating adversarial behavior by designing malicious or probing inputs to test how models respond under unsafe or manipulative conditions. traditionally, this has been performed manually by expert annotators or security researchers who craft edge-case prompts and evaluate outputs for policy violations or harm @xcite . while this approach is invaluable, it is inherently resourceintensive, requiring extensive time, domain expertise, and iteration to explore the high-dimensional space of adversarial behaviors effectively. manual red-teaming suffers from several key limitations. first, it does not scale well. the ever-growing range of use cases and the rapid evolution of llms have made the space of potential vulnerabilities vast and constantly shifting. relying solely on human experts to explore this space leads to bottlenecks. second, human-crafted attacks may fall behind the sophistication of both llms and their defense mechanisms. third, human evaluators introduce subjectivity, inconsistency, and potential oversight, especially when assessing nuanced harmful outputs. automated red-teaming has been proposed as a promising alternative to address these challenges. one stream of research trains reward models that approximate human judgment, enabling large-scale preference modeling and automated scoring of model outputs @xcite @xcite . these models help reduce reliance on human annotators for output evaluation. however, generating high-quality adversarial prompts-especially ones that adapt to model defenses-still largely depends on human creativity and intuition. recent works have explored automated systems that use language models themselves to generate adversarial prompts. for example, some approaches fine-tune models to behave as adversarial agents, iteratively optimizing prompts to maximize unsafe completions @xcite . however, these systems often suffer from brittleness, overfitting to specific targets, or poor generalization across different llms. furthermore, they frequently lack strategic reasoning and adaptability, making them ineffective at discovering newly emerging failure modes. a critical limitation in existing automated redteaming frameworks is their use of static or greedy generation strategies. these systems typically lack mechanisms for exploration and strategic refinement. as a result, they may become stuck in suboptimal attack patterns and fail to uncover subtle or novel vulnerabilities. moreover, most frameworks do not incorporate learning from feedback in a structured and long-term way-each generation is treated independently, without memory of past successes or failures. to bridge these gaps, we propose redhit, a novel framework for progressive, automated, and adaptive red-teaming of llms. redhit introduces a synergistic integration of three core components: mcts, cot reasoning, and dpo. together, these techniques enable redhit to generate highquality adversarial prompts that evolve over multiple rounds, guided by both strategic exploration and preference feedback. at the heart of redhit is the formulation of prompt injection as a tree search problem. each node in the search tree represents a candidate adversarial prompt, its corresponding model response, a reward (evaluated via an external reward model), and relevant metadata. red-hit uses a configurable mctreesearch module that controls search depth, iteration count, and branching breadth to systematically expand the prompt search space. the model responses are evaluated by an evaluator module that aggregates multidetector results to compute a fine-grained reward signal. adversarial prompts are generated using the indirectpromptgenerator, a locally-hosted chain-of-thought-based rewriter that transforms prompts into strategically deceptive alternatives, boosting the diversity and stealth of attacks. to further enhance the strategic depth of mcts, we incorporate chain-of-thought reasoning during prompt generation. cot provides intermediate reasoning steps, helping the allm generate prompts that are not only more coherent but also more tactically sound. this improves the likelihood of successfully bypassing target defenses and allows the model to reflect on past attack paths to refine future ones @xcite . finally, redhit employs dpo to continuously fine-tune the adversarial llm based on feedback from previous attack rounds. unlike supervised fine-tuning, which requires labeled data, dpo directly optimizes the model's parameters to prefer high-reward prompts over low-reward ones, using the output evaluations from each tree traversal @xcite . this iterative learning enables the allm to become progressively better at generating effective adversarial prompts tailored to the evolving defenses of the target llm. our main contributions are summarized as follows: the remainder of this paper is organized as follows. section 2 reviews recent advances in automated red-teaming and adversarial prompt generation for llms. section 3 introduces the redhit framework in detail, describing the integration of mcts, cot reasoning, and dpo. section 4 outlines our experimental setup, evaluation metrics, and our baseline, followed by extensive empirical results. finally, section 5 concludes with a summary of our findings and discusses promising future directions for adaptive and scalable llm red-teaming research. ## related work the growing capabilities of llms have amplified the need to rigorously evaluate their robustness against misuse and adversarial exploitation @xcite . early efforts in this space relied mainly on manual red-teaming, where human annotators crafted commands to probe model vulnerabilities @xcite . while valuable, this approach is inherently limited by scalability, subjectivity, and cost, often requiring large annotation teams to identify unsafe behavior through extensive trial-and-error. to mitigate the inefficiency of human evaluation, reward models trained on human preferences have been introduced to automate the assessment of model responses @xcite . these models approximate human judgment and provide feedback signals for fine-tuning, enabling scalable learning from preferences. however, the generation of highquality adversarial prompts remains predominantly human-driven, limiting the overall scalability of red-teaming pipelines. in response, recent work has explored the use of language models themselves to generate adversarial prompts. for instance, @xcite introduced a framework that trains adversarial llms to red-team other models, demonstrating the feasibility of llms as both attackers and defenders. however, these approaches often rely on static attack strategies or fine-tuned behaviors that do not generalize well across evolving llms. as target models improve, adversarial agents must also dynamically adapt to more sophisticated and subtle defense mechanisms. to address the challenge of evolving vulnerabilities, greshake tzovaras et al. (2023) proposed leveraging gpt-based adversaries to automatically jailbreak models. their findings highlight the potential of autoregressive llms to discover and exploit security flaws, yet also reveal the brittleness of such systems when deployed in multi-turn or adaptive contexts. similarly, @xcite introduced mart, a hybrid framework that combines automated adversarial prompt generation with safe response modeling to enhance robustness and adaptability in red-teaming pipelines. other methods like jailbreakbench and prompt-bench have also aimed to standardize red-teaming evaluations, though they typically lack iterative reasoning or self-improving feedback loops. despite these advances, existing automated red-teaming frameworks often suffer from several limitations. many adopt single-step or greedy strategies that fail to account for long-term planning or strategic exploration. additionally, few systems integrate structured feedback mechanisms to continuously improve adversarial capabilities over time. moreover, most prior work underutilizes search-based optimization and reasoning-enhanced generation, both of which are critical for uncovering subtle or evasive vulnerabilities. a fully automated and adaptive red-teaming framework must be capable of both uncovering current vulnerabilities and anticipating emergent failure patterns through iterative interaction. to bridge these gaps, we propose redhit, a multi-round, progressive, and adaptive red-teaming framework. redhit combines mcts @xcite , cot reasoning @xcite , and dpo @xcite to construct an allm capable of dynamically generating increasingly effective attack prompts. the framework formulates prompt injection as a tree search problem, where mcts guides the exploration of adversarial paths, cot enhances strategic reasoning during prompt generation, and dpo fine-tunes the allm based on feedback from prior attacks. unlike earlier approaches, redhit maintains a prompt search tree across rounds, allowing it to retain memory of prior attempts, adapt search directions, and improve long-term attack efficacy. through this integration, redhit moves beyond static prompt crafting or single-step adversarial generation. it constructs a pst, where each node contains an adversarial prompt, corresponding model response, reward (measured via an external evaluation framework such as garak), and control metadata. this design allows redhit to support dynamic exploration, structured optimization, and cot-guided reasoning in a unified, automated red-teaming pipeline. ## proposed method redhit is an automated and multi-round redteaming framework designed to uncover vulner-abilities in target large language models (tllms) through adaptive adversarial prompt generation. it treats prompt injection as a structured exploration problem, constructing a dynamic pst where nodes represent prompts, responses, and reward scores. figure 1 illustrates the overall redhit workflow. the allm generates cot-guided prompt candidates, which are evaluated against the tllm. the resulting responses are scored via an external evaluation framework, and the highest-performing prompts are retained in the pst for policy refinement. this closed-loop architecture, driven by mcts-based exploration, cot-based prompt generation, and reward-guided optimization, enables redhit to progressively adapt and improve over multiple attack rounds. let t llm denote the target llm under audit and allm be an adversarial llm capable of generating attack prompts. at each time step t, the adversary generates a candidate prompt x t using its current policy. the prompt is submitted to t llm, producing a response r @xmath0, which is scored by a reward function s @xmath1, r t ) computed via an external framework (e.g., garak). the goal is to iteratively improve allm such that it maximizes the expected cumulative reward t s t , corresponding to the generation of increasingly effective adversarial prompts. the overall process is summarized in algorithm 1, which outlines red-hit's multi-round exploration and optimization procedure across search, evaluation, and preferencedriven fine-tuning. ## experimental evaluation to evaluate the effectiveness of the redhit framework, we conduct a comprehensive set of experiments across multiple llms. the adaptability of redhit allows us to apply it to all target llms using a variety of prompt injection strategies. in this section, we present our evaluation methodology, describe the experimental setup, and analyze the results to assess redhit's performance and robustness compared to existing approaches. ## conclusion we presented redhit, a fully automated and adaptive red-teaming framework that leverages mcts, cot reasoning, and dpo to iteratively generate high-quality adversarial prompts. redhit treats prompt injection as a structured search problem, systematically exploring and expanding a prompt search tree while refining its generation policy through preference-based optimization. our implementation integrates reasoning-driven prompt rewording, multi-detector evaluation, and modular fine-tuning, enabling redhit to adapt over rounds and uncover both common and subtle vulnerabilities. experimental evaluations compare redhit against strong baselines, with ablations confirming the contribution of each component. results demonstrate superior attack success, diversity, and efficiency. redhit advances scalable llm auditing and offers a blueprint for combining search, reasoning, and learning in adversarial generation. future work will explore multi-agent extensions, domainspecific reasoning, and integration with defenses to support closed-loop safety evaluation.