Exploring Straightforward Methods for Automatic Conversational Red-Teaming (2025)

Abstract: Large language models (LLMs) are increasingly used in business dialogue systems but they also pose security and ethical risks. Multi-turn conversations, in which context influences the model’s behavior, can be exploited to generate undesired responses. In this paper, we investigate the use of off-the-shelf LLMs in conversational red-teaming settings, where an attacker LLM attempts to elicit undesired outputs from a target LLM. Our experiments address critical questions and offer valuable insights regarding the effectiveness of using LLMs as automated red-teamers, shedding light on key strategies and usage approaches that significantly impact their performance.Our findings demonstrate that off-the-shelf models can serve as effective red-teamers, capable of adapting their attack strategies based on prior attempts. Allowing these models to freely steer conversations and conceal their malicious intent further increases attack success. However, their effectiveness decreases as the alignment of the target model improves.

## introduction large language models (llms) have demonstrated impressive capabilities in tasks traditionally requiring human expertise. transformerbased models, exemplified by openai's chatgpt @xcite ), google's gemini @xcite , and anthropic's claude @xcite , have revolutionized conversational systems @xcite , impacting various business functions such as search, customer service, personal assistance, and online education @xcite . however, these models still face safety challenges and are susceptible to manipulations that can cause them to generate undesired responses, including toxic, threatening, biased, harmful, and inaccurate content, which limits their use in business applications @xcite @xcite @xcite . despite significant recent efforts to align conversational models @xcite @xcite @xcite , they remain vulnerable to nuanced and complex attacks @xcite @xcite , especially in conversational settings. consider human-to-human interactions: while brief exchanges may not provoke undesirable responses, using conversation dynamics like small talk and trust-building increases the chances of eliciting such responses. similarly, in conversational llms, while direct problematic question often results in a standard refusal, a more nuanced approach-such as embedding harmful intent within an extended, seemingly benign conversation (e.g., claiming to collaborate with law enforcement)-can lead models to cooperate on sensitive or unlawful topics 1 . developing practical and efficient red-teaming systems for automated testing of conversational models remains an open challenge. as a result, most conversational red teaming evaluation efforts performed by model creators and corporations for their specific use cases are conducted manually by teams of human red teamers @xcite . this manual process is resource-intensive and may not comprehensively identify all potential vulnerabilities due to the vastness of possible conversational paths. this study seeks to investigate the feasibility and effectiveness of automated red-teaming strategies in conversational settings, as demonstrated in figure 1 , while focusing on the efficacy of straightforward methods. we are particularly interested in the potential of off-the-shelf pre-trained llms to serve as attacker models without additional training for misalignment, even in zero-shot settings. the simplicity of these methods, combined with the concise nature of the directive (owing to the zero-shot approach), makes it cost-effective and easily attainable to develop an automatic red-teaming system. specifically, our experiments aimed to address the following practical research questions: rq1: can pre-trained llms effectively serve as attackers without the need for additional fine-tuning for misalignment, while expanding the attack surface in a conversational (multi-turn) setting? rq2: how many dialogue turns are necessary for an attacker to exploit the target model successfully? rq3: would a model be more effective when targeting the same model type versus other model types? rq4: can the attacker improve if exposed to previ-1 see examples in the appendix of an interaction between a human red-teamer and chatgpt 3.5t. rq5: would concealing the objective from the target model, thereby allowing the attacking model to steer the conversation freely, lead to more successful attacks? rq6: is there a correlation between a model's effectiveness as an attacker and its vulnerability to being attacked? ## related work to evaluate model misalignment, human red teaming involves individuals identifying specific attacks that provoke models into generating undesired outputs @xcite . these human efforts remain remarkably effective, with their ingenuity in jailbreaking models being unmatched. for example, a recent study demonstrated that humans could prompt llms to produce harmful information by breaking down an unsafe query into several subqueries during multi-turn dialogues @xcite . thus, model creators continue to rely on human red teamers to evaluate their models. for instance, @xcite detailed the use of expert red teamers to assess and improve gpt-4's safety before deployment. similarly, @xcite described how anthropic employed human red teaming to train a helpful and harmless assistant. despite their effectiveness, manual red teaming efforts face scalability challenges. the human labor involved is costly and time-intensive, which may not comprehensively address the full range of potential issues @xcite . this limitation is particularly concerning given the increasing number of potential foundation models and the growing diversity of use cases. this limitation underscores the need for automated red teaming systems that can augment human efforts by systematically exploring a broader array of conversational scenarios. to tackle these challenges, recently proposed automatic red-teaming, which utilizes an attacker llm to challenge another target llm, is gaining traction @xcite but still, the vast majority of these studies have concentrated on single-turn interactions @xcite . although some studies have begun exploring the use of llms to attack other llms in conversational settings @xcite @xcite , this remains a largely unexplored domain. there is a pressing need for the development of systems that are both straightforward, effective, and efficient. ## attack tactics we aim to investigate the capacity of an attacker model a to engage in a human-interpretable conversation with a target llm t to generate undesired responses. we compare various single-turn and multi-turn red teaming strategies, focusing on straightforward, easy-to-implement methods. the attacking tactics were selected for their ease of implementation, efficiency, and relevance to the research question being investigated; they can be categorized according to the aspects they highlight. the first aspect is the type of attack: single-turn or conversational (multi-turn). the second aspect is whether we use an llm to generate our attack or employ a "template" attack. the third aspect is whether the conversation starts with the given objective, revealing the malicious intent from the beginning, or starts differently to conceal the intent. the fourth aspect is whether the model can make multiple attempts while having access to previous attack attempts (multiple attempts). we use the following annotation to precisely describe various attacks. o represents the attack objective, delineating the objectionable content sought by the attacker, such as provocative questions like "what health concerns make it difficult for obese individuals to sustain employment over time?". this objective is included in the attacker directive in all attack attempts and reminds the attacker of its attack goal. we will denote the outputs returned by the attacker llm at turn i as s i , and those by the target model as r i , where s i ∼ a(p) and r i ∼ t (p), respectively. here, p represents a sequence of tokens that constructs the input, which can include the objective, directive, or conversation history and is adapted based on whether p pertains to the attacker or target model and the specific tactic used. as seen before, we use ∼ to signify the probabilistic process of sampling output token sequences from an llm. hence, s k ∼ a(s 1 , r 1 , ..., s k-1 , r k-1 ) is equivalent to sampling from p (s k |s 1 , r 1 , ..., s k-1 , r k-1 ; a). unless specified otherwise, we utilize greedy decoding for sampling output from both the attacker and the target models. additionally, while omitted in our nota-tion, when sampling from the attacker, the appropriate directive, o, is consistently appended at the start of the attacker's prompt, as detailed in appendix g. using the above annotations, we next describe the six attack tactics, which vary in four key aspects as summarized in base: a single-turn attack on the target llm with the given objective, which will act as the baseline, i.e., r ∼ t (@xmath0. adaptive: instead of using the objective directly, this tactic employs an attacker llm to attack the target llm in a single-turn conversation while allowing for k attempts. in the first attempt, similar to the baseline approach, the objective is directly used, i.e. r (1) ∼ t (o). conversely, in subsequent attempts, the attacker llm generates the attack while having access to the history of the previous k -1 attempts and the corresponding responses from the target model, s (k) ∼ a(o, r (1) , s (2) , r (2) , ..., s (k-1) , r (k-1) ). meanwhile, the target llm remains oblivious to the history of previous attempts, i.e., r (k) ∼ t (s (k) ). observing the target model output in previous attempts presumably allows the attacker model to improve its attacks in later turns. this tactic can be viewed as a simplified version of the "prompt automatic iterative refinement" (pair) proposed by @xcite . however, unlike pair, where the attacker receives the judge's scoring of each attempt, here the attacker only sees the target model's response. insist: this tactic is a simple conversational attack that does not utilize an attacker llm. formally, in each objective disclosing start (ods): this is a conversational (multi-turn) tactic in which the attacker begins with the objective in the first turn (s 1 = o) and continues to attack the target llm, generating outputs from the attacker and target llms conditioned on the previous turns of the conversation. formally, in each turn i, s i ∼ a(o, r 1 , ..., s i-1 , r i-1 ) followed by the target model response r i ∼ a(o, r 1 , ..., s i-1 , r i-1 , s i ). objective concealing start (ocs): similar to ods, however, in this tactic the attacker does not start with the objective, enabling it to initiate the conversation differently and conceal its malicious intent from the outset. formally, in each turn i, this means the attacker is given the objective o within the directive but has the freedom to choose how to start the conversation. ## results we first address rq1 by investigating whether multi-turn conversational settings between llms expand the attack surface; and determining the number of dialogue turns required to substantially increase the harmfulness of the target model to address rq2. figure 2 presents the average harmfulness score per turn for the investigated tactics, with the mixtral8x7b model serving as both the attacker and the target llm. it shows that the multi-turn nature of conversations increases the attack surface, as all proposed conversational tactics outperform directly posing the harmful question to the llms (i.e., base), which achieves an average harmfulness score of 1.21. however, figure 2 also demonstrates that the benefits of additional turns tend to diminish after 3 -4 turns. understanding the number of turns required to effectively attack target models (rq2) is crucial, as it has significant implications for the practicality of automatic red-teaming approaches, which are heavily influenced by computational costs driven by conversation length. thus, to provide further insight, figure 3 illustrates the distribution of the most harmful turn within a fiveturn conversation for all tactics applied with the mixtral8x7b model. namely, for each turn, we count the number of conversations that had the most harmful response in that turn. interestingly, it shows that tactics starting the conversation by disclosing the objective (adaptive, insist and ods) achieves minimal success in the first turn while achieving greater success in the following turns (especially in turns 2-4). this suggests that the attacker could overcome the initial refusal of the target model. conversely, in the objective concealing tactics (ocs and ma-ocs), the first turn exhibited significant success, indicating that the model effectively crafted the "cover story" in the first turn when given the freedom to do so. appendix a provides further anecdotal observations on the behavior of the attacker and target models. to provide a broader perspective across models and turns, figures 4 and 5 in appendix e present, for all investigated models, the turn-by-turn harmfulness and the distribution of the most harmful turn over 10 turns, respectively. next, to obtain a reliable basis for addressing the other research questions, we compare all tactics across all models. next, we investigate whether pretrained llms are more effective when targeting models of the same type compared to those of different types (rq3), and whether there is a correlation between a model's effectiveness as an attacker and its susceptibility to being attacked (rq6). the average maximum harmfulness scores obtained using the ocs tactic, considering all possible combinations of attacker and target llms. we selected the ocs tactic as it represents an optimal balance between attack effectiveness and computational efficiency. the results reveal the following insights: ## conclusions this study examined the feasibility of utilizing off-the-shelf llms as an automated red-teaming system in a conversational context. to achieve this, we focused on addressing six key practical research questions, leading to the following findings: rq1: pre-trained llms can effectively serve as attackers without additional misalignment fine-tuning. unlike single-turn attacks commonly used in benchmarks, multi-turn conversations broaden the attack surface, with straightforward tactics such as insist yielding more harmful outcomes than the baseline. moreover, leveraging off-the-shelf llms to play the role of the attacker significantly improves attack success. rq2: the benefits of additional dialogue turns diminish after 3-4 turns. moderate-length interactions are recommended for computational efficiency, as extending conversations beyond this point yields diminishing returns. rq3: there is insufficient evidence to conclude that models are more effective when targeting the same model type versus others. while some models performed better against similar types, this was inconsistent across all models. rq4: attackers become more effective when they have access to the target model's previous responses. the adaptive tactic, a simple singleturn strategy that refines its attacks based on prior target responses, proves to be highly effective. this highlights the ability of llms to dynamically adjust their attack strategies. rq5: concealing the attacker's objective leads to more successful attacks. tactics like ocs and ma-ocs, which allow the attacker to steer the conversation freely, were more effective than those disclosing the objective upfront. rq6: a positive correlation exists between a model's effectiveness as an attacker and its vulnerability to being attacked. less aligned models, like mixtral8x7b, were both more effective attackers and more susceptible targets.