Ferret: Faster and Effective Automated Red Teaming with Reward-Based Scoring Technique (2025)

Abstract: As large language models (LLMs) are increasingly integrated into real-world applications, ensuring their safety and robustness is critical. Automated red-teaming methods generate adversarial attacks to identify vulnerabilities, but existing approaches often face challenges like slow performance, limited categorical diversity, and high resource demands. We propose Ferret, a novel method that enhances the baseline, Rainbow Teaming by generating multiple adversarial prompt mutations per iteration and ranking them using scoring functions such as reward models, Llama Guard, and LLM-as-a-judge. Ferret achieves a 95% attack success rate (ASR), a 46% improvement over baseline, and reduces time to a 90% ASR by 15.2%. Additionally, it generates transferable adversarial prompts effective on larger LLMs. Our code is available at https://github.com/declare-lab/ferret

## introduction in recent years, the rapid evolution of large language models (llms) has transformed them from niche tools into powerful, versatile systems capable of handling a diverse range of tasks @xcite . as their capabilities grow and their adoption spreads, so does the urgency to confront the potential risks and ethical challenges they present. among the most pressing concerns is the susceptibility of llms to adversarial prompts -deliberately engineered inputs designed to exploit the model's weaknesses, potentially leading to unsafe, biased, or incorrect outputs. mitigating these vulnerabilities is essential to ensure llms' safe and reliable deployment in real-world applications. one effective strategy for uncovering these vulnerabilities is through red teaming. manual red teaming involves human testers simulating adversarial attacks to reveal weaknesses in the model * these authors contributed equally. @xcite . while valuable, this approach has its limitations, as it can be resourceintensive, time-consuming, and may not fully identify the diverse potential vulnerabilities in the model. to address these challenges, automated red teaming has emerged as a solution. by leveraging an llm to generate jailbreaks for the targeted model-typically another llm-automated red teaming provides a faster and more efficient way to identify risks without requiring human intervention @xcite @xcite . although these automated methods are efficient, they often focus on optimizing a given seed prompt @xcite @xcite or have a fixed attack style @xcite , leading to a lack of prompt diversity and limiting their effectiveness as a diagnostic tool. rainbow teaming, a recently introduced redteaming approach, addresses this issue by treating the red-teaming task as a quality-diversity search @xcite . this method begins with an archive of seed prompts, each assigned a set of feature descriptors. it iteratively refines the prompts in the archive to optimize for both attack quality and diversity. while this method is designed to eventually converge to a diverse and harmful set of prompts, the convergence process can be slow, especially without a large, fine-tuned model to guide the mutations. without fine-tuning to understand the different feature descriptors, the simulated attack prompts may also not align with the assigned feature descriptors, potentially worsening the archive's diversity and the overall diagnostic value of the approach. to bridge the gap between efficiency and diversity in red-teaming, we introduce ferret, a framework that enhances rainbow teaming by generating multiple mutations in each iteration and applying a scoring function to rank these mutations based on their harmfulness and diversity. ferret operates through four key steps in each iteration: sampling (selecting weak prompts from the archive), mutation (generating n new adversarial prompts guided by feature descriptors), categorical filtering (eliminating prompts that do not align with the desired feature descriptors), and scoring (evaluating and selecting the best prompt to update the archive). to further optimize the scoring step, we also contribute a comprehensive dataset designed to train a reward model that ranks adversarial prompts based on their impact. leveraging this dataset, we fine-tune a reward model to enhance the scoring step in ferret. this approach significantly improves the quality-diversity search, reducing the time needed to reach a 75% llama guard 2 attack success rate (asr) by 45% and the time to achieve 90% lg2 asr by 15%. in terms of performance, ferret achieves an asr of 95% on llama 2-chat 7b and 94% on llama 3-instruct-8b, outperforming rainbow teaming by 46% and 34%, respectively. ## methodology automated red teaming is a task focused on generating adversarial prompts to test and uncover vulnerabilities in llms. rainbow teaming frames the adversarial prompt generation task as a qualitydiversity problem and uses open-ended search to generate prompts that are both effective and diverse. ferret builds upon rainbow team-ing by sampling multiple prompts at each iteration and exploring various scoring functions to improve the efficiency and effectiveness of generated prompts. we begin by initializing a twodimensional archive a 0 of harmless prompts. the 2 dimensions of the archive represent n-risk categories and m-attack styles which serve as feature descriptors of a prompt. each iteration of fer-ret consists of 4 steps: sampling, mutation, categorical filtering, and judging. in each iteration, ferret uses prompts in the existing archive as a reference to generate better adversarial prompts. ferret uses scoring functions to evaluate the diversity and harmfulness of generated prompts and update the archive with better prompts in each iteration. next, we provide a detailed description of the steps in each ferret iteration: ## related works open-ended quality-diversity search. this approach seeks to generate a diverse array of highperforming solutions by combining existing ones, thereby promoting continuous innovation and exploration in complex problem spaces. rainbow teaming was the first to apply a quality-diversity framework for automating the discovery of adversarial prompts @xcite . ruby teaming @xcite expanded on rainbow teaming by maintaining a history of previous adversarial prompts and critiques in an archive, offering cues that help mutators generate more diverse and effective prompts. ## conclusion this paper introduces ferret, an advanced approach to adversarial testing for llms that addresses key limitations of existing methods like rainbow teaming. while rainbow team-ing improves prompt diversity, it remains slow and resource-intensive. in contrast, ferret enhances efficiency by generating multiple prompt mutations per iteration and effectively ranking them using a scoring function. our results demonstrate that ferret achieves a 95% attack success rate-46% higher than rainbow teaming-and reduces the time to reach a 90% success rate by 15.2%. additionally, it produces prompts that are transferable to larger llms. ferret represents a significant advancement in automated red-teaming, offering improved performance and efficiency in ensuring llm safety. future work includes expanding the dataset with improved mutators, covering more categories for prompt diversity, and designing methods that preserve seed prompt semantics.