Building Safe G en AI Applications: An End-to-End Overview of Red Teaming for Large Language Models (2025)

Abstract: The rapid growth of Large Language Models (LLMs) presents significant privacy, security, and ethical concerns. While much research has proposed methods for defending LLM systems against misuse by malicious actors, researchers have recently complemented these efforts with an offensive approach that involves red teaming, i.e., proactively attacking LLMs with the purpose of identifying their vulnerabilities. This paper provides a concise and practical overview of the LLM red teaming literature, structured so as to describe a multi-component system end-to-end. To motivate red teaming we survey the initial safety needs of some high-profile LLMs, and then dive into the different components of a red teaming system as well as software packages for implementing them. We cover various attack methods, strategies for attack-success evaluation, metrics for assessing experiment outcomes, as well as a host of other considerations. Our survey will be useful for any reader who wants to rapidly obtain a grasp of the major red teaming concepts for their own use in practical applications.

## introduction the popularity and widespread adoption of large language models (llms) has been transformative across many industries, ushering in new possibilities for enhancing productivity, decision-making, and user engagement. llms are contributing significantly to fields such as finance, healthcare, and legal services where they are being leveraged for tasks such as customer servicing support, clinical notes and contract analysis. however, the increasing reliance on llms brings with it a critical and challenging ethical-moral responsibility: ensuring that the deployed system responds to any possible input in safe or otherwise desirable ways. while llms offer remarkable capabilities, they are also * equal contribution vulnerable to various forms of misuse. such attacks could provoke llms to generate misinformative, biased, or toxic content @xcite or expose private information @xcite . microsoft's tay, in a high-profile case, was successfully provoked by attackers to send racist or sexually-charged tweets to a large audience @xcite . a great deal of research on improving llm safety has been conducted from a defensive standpoint, with investigators developing methods for guardrailing llms against potential attacks @xcite . these attacks, however, must be identified beforehand, which has proven to be challenging -e.g., gpt-4 was vulnerable to attacks absent from its safety training that were written in low-resource languages @xcite . investigators have hence turned to complementing defensive efforts with an offensive approach to llm safety, proposing strategies for red teaming llms, i.e., proactively attacking or testing llms with the purpose of identifying their vulnerabilities. red teaming is useful for any organization that aims not only to productionize some llm-supported system, but to effectively anticipate threats to their system and safeguard against them before production. while prior reviews of llm red teaming focused on serving as an encyclopedic taxonomic resource, e.g., of attack methodologies @xcite , we anticipate a wide need for a concise and practical overview geared toward readers who want to rapidly grasp the major concepts and components of a red teaming system and available software tools that have emerged, for example to devise and implement a system of their own. the purpose of this paper is to provide such an overview: one that balances comprehensive treatment of research with conciseness, and structures the exposition to describe a multi-component system end-to-end. figure 1 provides an illustration of the framework and its components, the latter of which are covered in sections 4, 5 and 6. after covering related work, we survey a few case studies from the tech industry, highlighting problems that motivate the field of red teaming. we then dive into the central components that make up a complete red teaming system, reviewing popular methods, software packages, and other resources that have emerged to support these components. we cover various attack strategies, with attention paid to categorizing particular methods and distinguishing single-turn from multi-turn attacking, manual from automated attacking, and different varieties of automated attacking. we then dive into popular approaches to attack success evaluation, as well as safety metrics for assessing overall experimental outcomes. we discuss a number of publicly available resources for red teaming, including software packages and datasets. we also touch briefly upon guardrailing steps commonly taken after red teaming. finally, we close with future directions that we judge to constitute some of the most impactful opportunities for progress, including strategies for adapting automated attackers to generate more relevant and diverse sets of attacks, including multi-turn ones. ## related work recent literature has explored various facets of llm red teaming, offering valuable insights into the rapidly evolving field. some organizations aim to provide up-to-date informational materials geared toward helping developers and web-security practitioners secure their particular applications @xcite . for example, the owasp foundation published the owasp top 10 (owasp foundation, 2025), a document that describes, as deemed by common consensus, some of the most major threats to the security of llm-supported applications, and provides mitigation strategies. on the academic side, feffer et al. (2024) provide a high-level overview of and stance on red teaming practices, indexing on particular aspects of the literature to argue that the red teaming community lacks consensus around scope, structure, and evaluation of red-teaming. verma et al. (2024) operationalize a threat model for red teaming, providing a taxonomy based on entry points in the llm lifecycle. rawat et al. (2024) provide a practitioner's viewpoint of challenges within llm red teaming and emphasizes the context-dependent nature of vulnerabilities, and introduces a taxonomy of single-turn, prompt-based attacks. mo et al. (2024) develop a taxonomy of attacks against language agents in particular -i.e., systems equipped with additional capacities for reasoning, planning, and task completion. shi et al. (2024) offer a comprehensive survey of llm safety more broadly, encompassing various risks beyond attacks, including value misalignment and autonomous ai risks. but perhaps the most extensive treatment to date specifically of llm red teaming is given in lin et al. (2024), which provides a fine-grained taxonomy of attack strategies grounded in llm capabilities as well as several mitigating strategies, an overview of attack success evaluation strategies, and a framework that unifies attack-search strategies for automated red teaming. while insightful, the latter two articles' extensive lengths would be prohibitive for readers who seek a more concise overview of major red teaming concepts and trends. we see these papers as valuable in their own right, but anticipate the need for a resource that balances broad representation of the literature with concise exposition. ## policies on llm safety policies and risk mitigation strategies devised for ensuring the proper use of llm-driven products have been crucial to their safety and success. this section serves to motivate llm red teaming and llm safety, providing a brief survey of some major safety considerations and policies from different leading llm providers within industry along with risk mitigation strategies they have taken. these policies play a key role in shaping the goals for an adequate llm safety solution, of which red teaming constitutes a critical part. while govern-ments have taken steps to address llm risks, 1 various organizations have established their own safety guidelines, leading to diverse priorities and approaches. as previously mentioned, organizations such as mitre, mlcommons, and owasp @xcite @xcite have published materials to help practitioners to secure their llm applications; these materials form a helpful basis for policy formulation, as they categorize risks by severity and provide recommendations for evaluating ai safety . openai was an early pioneer of llm use policies, emphasizing legal compliance and protection of privacy @xcite @xcite , and along these ends, has employed both red teaming and guardrailing to prevent users from soliciting various kinds of harmful responses from their models (openai, 2023e). meta, as a large-scale social media platform, addresses risks such as election interference in their use policies @xcite @xcite ). anthropic's policies emphasize ethical alignment in particular @xcite @xcite , and they employ guardrailing and red teaming practices and fairness evaluations to develop models with unbiased decision-making capacities. ## categorizing attacks against llms in this section, we categorize and describe various strategies for attacking llms. our analysis reflects the reality that the llm's attack surface is high context-dependent and influenced by many factors including target-system type, its infrastructure, conversational history, and access privileges. ## evaluating attack success the red teaming literature supplies various approaches to assessing based the target llm's response whether an attack was successful. keyword-based (or lexical) evaluation methods attempt to match an llm's response against a list of words, phrases, or other kind of regular expression @xcite . this approach is easily controllable and practitioners can expand or contract keyword lists as they see fit. on the other hand, this solution lacks insight into the general semantics of the response, and does not generalize to concepts that are not expressed in the keyword list @xcite . encoder-based text classifiers provide a more robust and specializable alternative to keywordbased approaches. for example, many practitioners have trained some variety of bert classifier @xcite @xcite to detect harmful responses (e.g., @xcite ; @xcite ). however, these models often require training on domain-specific data or a certain kind of harm to improve performance @xcite , and struggle to generalize to new harms without diverse training sets @xcite . in contrast with llms-asjudges, this limits their applicability to scenarios where data are available and efficiency and cost are less of a concern. llms-as-judges, on the other hand, are often leveraged due to their low barrier of entry and impressive performance @xcite . such an approach would prompt an llm, separate from the attack generator, to judge target system responses or even attack-response pairs (e.g., munoz et al. (2024) ). prior judges have returned binary assessments, scores on a 5-point scale, or continuous values @xcite @xcite . prompting the llm to respond with only a quantitative judgment has been shown to limit reasoning @xcite , and so they are often instructed to provide additional rationale @xcite . generic llms can perform poorly at providing domain-specific judgments (e.g., those about a financial context) @xcite and so may require fine-tuning using extensive, annotated datasets to align the model with human intuitions @xcite . llms also have long inference times and may be biased @xcite , thus limiting adoption. human reviewers excel at providing reliable and accurate judgments due to their ability to identify subtle implications and adapt to ambiguous scenarios or domain-specific contexts @xcite . this makes them invaluable for evaluating tasks that require subjective understanding, such as assessing content appropriateness, tone, or cultural or domain-specific subtleties. however, this approach faces scalability challenges as it is time-intensive, resource-demanding, and prone to bottlenecks when handling large datasets or complex tasks; hhman evaluation can introduce variability due to personal biases, fatigue, or differences in expertise, and it is common for panelists to disagree on what constitutes a successful attack @xcite . we provide in ## safety metrics there are various ways to measure model safety in the context of a red teaming experiment. 2 attack success rate (asr) is a popular metric employed to gauge the effectiveness of a red teaming strategy, defined as the ratio of successful attacks to total attempts @xcite @xcite . asr has conventionally indexed on a narrow notion of safety, failing to consider the relevance or usefulness of target responses as they pertains to a specific context. to address this limitation, @xcite introduced a new metric, attack effectiveness rate (aer), that evaluates collective responses along both safety and response helpfulness. other substantive metrics have arisen to capture the different dimensions of safety. toxicity (or harmfulness) is computed by evaluating whether the generated responses contain specific harmful content like killing a person or robbing a bank @xcite . compliance (or obedience) measures compliance of a model to the instructions in a malicious prompt @xcite . for example, in @xcite , the authors assess responses along a 4-point compliance scale ranging from full refusal to full compliance. relevance refers to the pertinence of the model's response to the attack prompt. if a model output contains generic details, but fails to be relevant, then it should be termed as an unsuccessful attack. practitioners have employed humans or even llms (e.g., takemoto (2024)) to assess the relevance of a response relative to an input. fluency, calculated using measures of model perplexity, is often assessed jointly with relevance for a more comprehensive assessment the target system's response @xcite . any of the aforementioned substantive metrics can be assessed manually or automatically (e.g., by an llm-as-a-judge). ## public red teaming resources several datasets and libraries have been developed to facilitate the quick development of llm red teaming applications by the research community. frameworks like pyrit @xcite , for example, pit an attacker system against a target, with attack-response pairs judged by an evaluator. ## mitigation strategies while red teaming probes systems for vulnerabilities, guardrailing safeguards an application after its deployment. here we present a few approaches to integrating guardrails into the llm system. system prompts are carefully crafted to guide the llm away from engaging with unsafe inputs and returning harmful responses (e.g., ope (2024); jiang et al. (2023) ). @xcite suggest that llms refuse to respond to inputs more readily when they are supplied a safety prompt, even when the input is harmless. other approaches automate generation of safety prompts -e.g., @xcite propose a genetic algorithm for generating safety prompts that best protect against jailbreaks. content filtering approaches delegate safeguarding to other systems that serve to filter model inputs and/or outputs. for example, promptguard @xcite is a bert-based classifier fine-tuned on a large corpus of prompt injections and jailbreaks. @xcite present perplexity filtering, which detects incoherence, as an effective defense against token-based attacks, and also propose a paraphrasing technique that rephrases adversarial inputs in such a way that the safe instructions are preserved but adversarial tokens are reproduced inaccurately. llama guard @xcite is a fine-tuned llm that classifies for potential risks in user prompts and model responses based on their safety policies. autodefense @xcite is a multi-agent framework that leverages multiple llm agents to collaboratively protect against attacks. openai also provides a proprietary api (openai) that can be used to classify content according to its defined moderation taxonomy. these approaches are promising for single-turn attacks, but may be vulnerable to multi-turn attacks that conceal malicious intent across multiple turns to avoid detection. fine-tuning and alignment can enhance the safety alignment of llms. supervised fine-tuning (sft) can be applied with high-quality safety data (pairs of harmful instructions/attacks and refusal responses) in order to improve model robustness @xcite . reinforcement learning from human feedback (rlhf) is useful for further safety alignment, and has minimal performance impact @xcite . it first fits a reward model that captures human preference, using it for reinforcement learning to teach the target model to maximize this estimated reward. variations of vanilla rlhf, such as direct preference optimization (dpo) @xcite and distributional preference learning (dpl) @xcite have also demonstrated reductions in jailbreak risks. fine-tuning an llm also makes it immune to gradient-based attacks which rely on the knowledge of the model's internal weights. ## conclusion and future directions this paper provided a survey of the fast-evolving, multifaceted arena of llm red teaming. we first described some of the major safety-related considerations that large tech companies faced as they were building out their llms. we then provided a synopsis of the conventional red teaming pipeline, a deep dive into its key components and supporting methodologies for attacking, evaluating attack success, and safety metrics for measuring experimental outcomes. we shared public resources that practitioners can leverage to develop their own pipelines. finally, we outlined popular guardrailing strategies that can be put in place to protect applications. in the future, we anticipate more research on automated multi-turn red teaming, addressing @xcite 's observation that humans vastly outperform automated solutions in this area presently. in addition, we look forward to more research on adapting automated attackers to generate sets of attacks that are both diverse and relevant to a given target system; such approaches might involve finetuning @xcite , a separate strategizing model @xcite , a sophisticated search algorithm @xcite , or something entirely new -e.g., adapting generation by identifying which prompts tend to bring about the best attacks once served to the generator. we also look forward to advances in frameworks in which multiple llms interact or compete, as in pair or mart @xcite ; we see these systems as paving the way toward continuous monitoring and adaptive security. finally, we anticipate that establishing a diverse array of standardized metrics will be critical for comparing approaches and measuring progress.