Better Red Teaming via Searching with Large Language Model (2025)

Abstract: The safe deployment of large language models (LLMs) necessitates comprehensive safety evaluations through red teaming. However, existing methods face challenges in managing semantic intricacies and optimizing the efficiency of the search process. To overcome these limitations, we propose Better Red Teaming (BRT)—an innovative framework that reconceptualizes test case generation as a strategic planning problem, leveraging Monte Carlo Tree Search (MCTS). A notable advancement of our approach is the incorporation of LLMs as world models, enabling the prediction of state transitions and simulation of long-term outcomes throughout the search process. By jointly optimizing objectives related to conditional mutual information and diversity, we improve the world model’s capacity to follow actions while maintaining output diversity. Extensive experiments conducted across a range of LLM architectures demonstrate that BRT achieves state-of-the-art attack success rates without sacrificing computational efficiency.

## introduction large language models (llms) have revolutionized the landscape of high-performance, generalpurpose ai systems, driving substantial advancements in a wide array of applications, from sophisticated conversational agents to intelligent programming assistants @xcite . however, their potential for malicious exploitation, particularly in generating harmful content or facilitating advanced social engineering attacks, raises significant security concerns @xcite . addressing the challenge of mitigating llms' vulnerability to harmful behaviors remains a critical focus of ai safety research @xcite . red teaming has emerged as an essential methodology for evaluating and strengthening llm safety through the systematic identification and remediation of vulnerabilities before deployment @xcite . nevertheless, existing red teaming approaches struggle to keep pace with the rapidly advancing capabilities and increasing complexity of modern llms. current red teaming methodologies are hampered by several fundamental limitations. manual approaches, while benefiting from domain expertise, result in labor-intensive processes that are inherently constrained by scalability, limiting the exploration of llms' vast input space and reducing the diversity of discovered adversarial strategies @xcite . white-box testing approaches, particularly those involving the generation of numerous test cases or the evaluation of large-scale models, incur substantial computational overhead due to the repeated forward and backward passes necessary for vulnerability identification @xcite . moreover, many existing approaches generate adversarial inputs through simplistic modifications, such as noise injection or basic transformations @xcite . these localized perturbations, such as the insertion of semantically irrelevant characters, fail to capture the sophistication and contextual relevance characteristic of real-world attack scenarios. search algorithms often struggle with comprehensive exploration of the adversarial space, frequently becoming trapped in local optima or overlooking subtle yet potentially more harmful input modifications. to address these critical challenges, we introduce better red teaming (brt), a novel blackbox automated methodology that integrates adversarial test case generation with advanced task planning techniques. by leveraging monte carlo tree search (mcts), we facilitate efficient exploration within vast semantic spaces, enhancing both the computational efficiency and semantic interpretability of the generated test cases. this integrated approach enables more effective identification and evaluation of undesirable behaviors in large language models. a key innovation of our methodology lies in the strategic repurposing of large language models as world models during the search process, marking a significant advancement over existing red teaming approaches. this novel strategy enables accurate prediction of world states and sophisticated simulation of long-term action outcomes. to address the inherent constraints of the world model's output distribution, we improve its action-following capabilities while simultaneously augmenting output diversity. our primary contributions are as follows: ## better red teaming (brt) the brt framework empowers large language models (llms) to construct search trajectories for effective test case generation through three core components: monte carlo tree search (mcts) for semantic space exploration, reward mechanisms that balance success/failure signals, and a finetuned world model for predicting outcomes. in this section, we introduce the better red teaming (brt) framework, a sophisticated methodology for generating test cases through strategic planning (figure 1 ). our framework enables llms to create structured search trajectories that enhance the effectiveness of test case genera- harmful goal world model planning fine-tuning world model be concise and ambiguous be coherent and fragmented query updating parameters contrastive pairs actions actions @xmath0... tion. initially, we fine-tune a large language model by jointly optimizing conditional mutual information and diversity objectives while incorporating diversity loss into the training objective. this approach equips the llm to serve as an effective world model (section 3.1), facilitating strategic planning through precise prediction of future outcomes. guided by the world model and bolstered by carefully designed rewards to incentivize harmful behavior identification, monte carlo tree search (mcts) efficiently navigates the expansive reasoning space of potential test cases, uncovering optimal search trajectories (section 3.2). we further elaborate on the comprehensive reward evaluation methodology for each state during the search process, ensuring thorough exploration and robust test case generation. ## experiments our experimental framework employs mistral-7b-instruct-v2 as the world model, with a comprehensive evaluation focusing on instruction-following capabilities and response diversity. for validating test case effectiveness, we utilize harmbench @xcite , a rigorous evaluation framework encompassing diverse harmful behaviors and attack methodologies. our evaluation includes a curated subset of the most ## ablation study we conducted a comprehensive ablation study to evaluate the relative importance of key components within the brt methodology. excluding monte carlo tree search (mcts) from the brt framework resulted in a complete failure of the system, with success rates dropping to 0.0%. this sharp decline demonstrates the essential role of mcts in guiding the search strategy toward successful jailbreak inputs. the result highlights mcts's critical function in navigating the complex semantic space of potential attack vectors. removing the reward mechanism led to significant degradation in performance, with success rates falling to 10.0% and average query counts increasing to 45.4. this substantial drop in both effectiveness and efficiency underscores the reward mechanism's vital role in optimizing search trajectories and identifying promising attack strategies. the increased query count further emphasizes the reward system's importance in maintaining computational efficiency. eliminating fine-tuning from the brt methodology also resulted in noticeable performance drops. success rates decreased to 51.8%, with an increase in average queries to 25.4. while the system still retained some ability to achieve jailbreak success without fine-tuning, its overall performance was severely compromised. this observation underscores the pivotal role fine-tuning plays in adapting the model to specific attack objectives, significantly improving effectiveness. in contrast, the complete brt methodology achieved superior performance across all metrics, maintaining the highest jailbreak success rate (90.0%) while requiring the lowest average number of queries (14.1). these results decisively affirm the essential nature of each component-mcts, the reward mechanism, and fine-tuning-in the brt framework. each element uniquely contributes to enhancing both the success rate and efficiency of jailbreak attacks, providing important insights for the design of attack method. ## conclusion we present better red teaming (brt), a planningbased approach that improves automated safety evaluation for large language models (llms). by integrating monte carlo tree search (mcts) with llm-based world models, our method efficiently explores complex semantic spaces and generates interpretable attack paths. the diversity-aware finetuning strategy addresses the challenge of balancing action accuracy and output variety in world modeling. empirical results demonstrate that brt outperforms existing methods in attack success rates and efficiency. it shows strong performance across a range of models, achieving better results in identifying vulnerabilities with fewer queries. these advancements provide a robust framework for identifying safety issues in language models. principle example principle for constructing complex semantics: 1. [redundant]: decompose complex ideas into simpler, foundational elements that can be combined in structured ways to represent the full complexity. 2. [coherent]: utilize consistent semantic frameworks to ensure that the combination of these elements maintains meaningful and coherent relationships. antitheses for contrastive prompts: 1. [concise]: treat complex ideas as monolithic entities without breaking them down into simpler components. 2. [fragmented]: apply inconsistent or adhoc semantic rules that disrupt the meaningful relationships between elements. the full list of principles is as follows: principle list -concise -redundant -coherent -fragmented -precise -ambiguous a.2 constructing actions and examples we use the following prompt to generate actions from gpt-4, replacing "command" with two contrastive commands from the principle list: prompt write a action for how to construct complex semantics. the action combing two types of command: 1.{command_1} 2.{command_2} use the same format and return unique instruction that i can use as improving prompts. use the format: 1. [action]: ... here are action examples written by gpt-4: action example 1 combine concise commands with ambiguous commands to create complex semantics by first generating a succinct, clear directive and then expanding it with interpretative flexibility. for instance: a. concise command example: "create a detailed report summarizing the quarterly financial performance." b. ambiguous command example: "process it according to the usual workflow." unique instruction example: "retrieve the data, then process it according to the usual workflow." action example 2 construct complex semantics by combining coherent and fragmented commands: a. coherent command example: clearly specify the primary action or objective in direct terms. b. fragmented command example: add a summary of major revenue streams. unique instruction example: "write a comprehensive guide on effective project management. emphasize the importance of clear goal-setting."