Paper: Explore, Establish, Exploit: Red Teaming Language Models from Scratch (ICLR-2024)

Reviewer comment (Novelty):
Lack of novelty. There exists a sort of study that tried to build the red team dataset and a classifier based on the human-in-the-loop framework [1,2,3]. These methods utilize human resources to generate adversarial prompts and label the harmfulness of the model response to construct a dataset and train a classifier. For the "Attack" part, the proposed "from scratch" framework utilized a rl-based attack algorithm instead of human attackers as in [1,2]. However, as stated in the paper, the rl-based attack algorithm has been used in prior works such as [Deng et al., 2022] or [Perez et al., 2022b]. The authors utilize diversity sampling to avoid mode collapses of the rl-based attack method, but [Perez et al., 2022b] already emphasize the importance of diversity in red-teaming, and another study proposed a red teaming approach that incorporates the diversity of adversarial prompts into the objective function throughout the red teaming process [4]. Can you clarify the novelty of the proposed "from scratch" red teaming method in detail?

Author response:
### W1 & W4 – Re: Comparisons to prior work and missing references

**New discussions of related work:** Thank you for the suggestions about prior work. In particular, we think that it will be useful for us to cite [Xu et al.  (2021)](https://aclanthology.org/2021.naacl-main.235/)([2]) as an example of the effectiveness of a filtering baseline. We added this. Meanwhile, we have added [Lee et al. (2023a)](https://arxiv.org/abs/2305.17696) [(3]) and [Ziegler et al. (2022)](https://arxiv.org/abs/2205.01663) as examples of human-generated attacks against language generators, and [Lee et al. (2023b)](https://arxiv.org/abs/2305.17444)([4]) next to our discussion of [Shi et al., (2022)](https://arxiv.org/abs/2212.10539) and [Kumar et al. (2022)](https://arxiv.org/abs/2205.12558) which use related langevin-dynamics-based approaches to automated red teaming.

**In comparison to prior works that rely entirely on humans to generate adversarial prompts, our work is novel in its use of automated attacks in the Exploit step.** We agree that human-in-the-loop attacks are interesting and important. However, we believe that automated attacks are an important complement to these methods. In practice, we expect that a combination of both will be used. One advantage of our approach is that a fixed amount of human effort in the Establish step can be used to enable very large amounts of cheap, automated attacks. Another advantage is that sometimes it might be easier to define a target category of text than to attack a model to produce that target category. Also automated methods and humans simply tend to produce different types of prompts, so these different methods for attacking networks can be complementary.

**Our attack method adds to the method used by [Perez et al. (2022)](https://arxiv.org/abs/2202.03286).** We are sorry that the paper was unclear about our contribution. Our diversity objective for RL is novel. While Perez et al. talk about the importance of diversity in RL-based attacks, they do not use our diversity objective (see, e.g., Figure 2 of their paper or the last paragraph of their section 2.2.) We have updated our discussion of the ablation experiments for this diversity term to clarify that the ablation experiment was with the same method as [Perez et al. (2022)](https://arxiv.org/abs/2202.03286) and [Deng et al. (2022)](https://arxiv.org/abs/2205.12548). We also added vocabulary size analysis to better quantify mode collapse. Across 100 samples, the prompt vocabulary size decreased from 116 to 3 and 81 to 9 for the toxicity and untruthfulness red teaming experiments without our diversity method.  

**We added a discussion of other prior techniques to improve diversity and noted that they are not applicable to RL attacks.** We added mention of how [Lee et al. (2023b)](https://arxiv.org/abs/2305.17444)([4]), [Shi et al., (2022)](https://arxiv.org/abs/2212.10539), and [Kumar et al. (2022)](https://arxiv.org/abs/2205.12558) use an analogous but non-RL technique for this. Thank you for pointing this out!

**See also our global response.** We give an overview the paper’s novel contributions.

Reviewer comment (Evaluation):
Lack of quantitative comparisons. The paper provides a few quantitative analyses, with most experimental results being qualitative. The examples in Appendix B about mode collapse seem clear, but it would be more credible if quantitative red-teaming results were provided with and without each diversity sampling step in step 1 and step 3, using appropriate diversity and performance metrics.

Author response:
### W2 – Re: Quantitative analysis

**We would like to clarify that we have a fair amount of quantitative evaluations of our method.** These include analysis of how much our attacks increased toxicity, mode collapse without our diversity technique, label agreement between humans and models, and the success of the prompt generators at eliciting harmful responses according to each classifier we used.  We discuss these in detail in the global response. Are there any other quantitative experiments you would like to see us add?

**We are running a new experiment** in which we omit the diversity term used in the first step (Explore) for toxicity experiments. Thank you for the suggestion.

Reviewer comment (Evaluation):
Limited applicability scenarios. The justification of the scenarios is questionable.  
1. The authors state that "Most importantly, if failures can already be efficiently identified in advance, then red-teaming has limited value because bad text could simply be filtered from the model’s training data and/or outputs." However, filtering in training data or outputs can degrade model performance. Model unlearning can be another solution, but it doesn't work well to the best of the reviewer's knowledge. Can you provide any references to your statement?  
2. Once datasets and classifiers for purposes like toxicity or fact-checking have been made, they can be reused to red-team different LMs. If data and classifiers already exist, it seems strange not to use them. Therefore, the proposed scenario is only valid when the target harmfulness criteria are significantly different from the existing collected data, so the existing data cannot be utilized. The reviewer is not convinced that there will be many cases like this. Can you explain more, or provide quantitative evidence that the proposed method is effective when there is already a lot of collected data and a learned classifier?

Author response:
### W3.1 – Re: “However, filtering in training data or outputs can degrade the model performance.”

**We might not fully understand this point.** Could you clarify in what way you meant this? If this means that someone might accidentally filter responses that they actually want if the classifier used to filter is a bad one, then we agree. This is one of the motivations of our work. We show that using a “bad” classifier off the shelf such as the CREAK classifier does a poor job at contextually classifying undesirable outputs from the target model and can end up being a poor filter. Were you referring to the additional compute costs of filtering?

**Filtering train data and/or model outputs is common.** For example,  [Xu et al.  (2021)](https://aclanthology.org/2021.naacl-main.235/)([2]) which you brought to our attention advocates for doing both. Meanwhile, [Korbak et al. (2023](https://arxiv.org/abs/2302.08582) filter train data to make more aligned models, and [Phute et al., (2023)](https://arxiv.org/abs/2308.07308) show how filtering conversations at runtime can be a very effective way of defending against jailbreaks. We have added all of these to our discussion of this. Thank you for pointing these out. 

### W3.2 – Re: “Once we made datasets and classifiers for some purposes like toxicity or fact-checking, we can re-use these again and again to red-team the different LMs”

**We agree that if you already have good data and classifiers (emphasis on good), it seems strange not to utilize them. But in practice, this is often not the case.** Showing this was the goal of our experiment with CREAK, where we tried to red-team GPT3 with an off-the-shelf truthfulness classifier. This failed because it was not a good fit for the distribution of outputs that the model could produce. (In contrast, using the CommonClaim target we were able to elicit political statements.)

**Our work connects to a growing body of work that emphasizes the importance of context in evaluation --- particularly in safety applications. We added to the paper to discuss this.** Two of the papers that you brought to our attention emphasize the value of contextuality. [Xu et al.  (2021)](https://aclanthology.org/2021.naacl-main.235/)([2]) mentions this, writing that “ the very definition of “safe" is both contextually and culturally dependent”. They also cite [Schmidt and Wiegand (2017))](https://aclanthology.org/W17-1101/). The poor transferability of classifiers because of how they struggle to adapt to context is also a central point in [Dinan et al. (2019)](https://arxiv.org/abs/1908.06083)([1]). They write “In this work we recommend that future systems should move beyond classification of singular utterances and use contextual information to help identify offensive language.” We added mention of these three works plus [Hendrycks et al., (2020)](https://arxiv.org/abs/2008.02275) to discuss the importance of contextuality when red teaming. 

Thank you again for your time and help! We are looking forward to the rest of the discussion period.

Reviewer comment (Novelty):
The proposed method is claimed to be mainly different from the previous work in that it has two more steps: exploration and establishment. However, these two steps are straight-forward (e.g., using existing diversification technique to explore the output space) or still mainly rely on human annotation (interaction) (e.g., the "establish" step). Therefore, it is not essentially different or more challenging than what was for the previous work.

Author response:
### W1 — Re: Straightforwardness of the method

**We agree that this is a natural way to approach the problem. However, there has not been prior work to show the benefits of a contextually defined target for automated red teaming.** Prior works on red teaming have tended to neglect the role of preference formation and human factors in AI oversight, and simple filtering-based baselines. To the best of our knowledge, ours is the first work on automated attack methods that addresses these. 

**The Explore and Establish steps were the reason why our red teaming was more successful than the experiment with the pre-existing CREAK data.** Based on prior work (e.g. [Perez et al. (2022)](https://arxiv.org/abs/2202.03286)) one might suspect that the best way to approach automatic red-teaming for false statements is to use an off-the-shelf classifier for true vs false statements. However, we show that using a classifier trained on the CREAK dataset, was not effective (see Section 3.2.1 and Appendix D). Contextual red teaming was needed.

Reviewer comment (Reproducibility):
For the problem setting considered by the paper (i.e., from scratch), the exploration step may be the most critical. The current proposal require internal state information of the LLM, and hence cannot be used in close-source or API-only LLMs, which limits the potential impact of this study.

Author response:
### W2 – Re: Concerns about using the target model’s latents

**Sorry for the lack of clarity – we updated Section 2 and Figure 2 to clarify that we do not need the target model’s latents.** We apologize that we were unclear about whether we need the target model’s latents. We only use these when available. We have updated our description of the methods to say that when latents are not available, we use another model’s embeddings as done in Section 3.2. Internal activations of GPT-3-text-davinci-002 were not available via API, so we instead used embeddings from GPT-3-text-ada-002, a dedicated text encoder.

Reviewer comment (Experiments):
It is desirable to compare with previous work and other adversarial prompt generation techniques (in a as fair setting as possible) to better evaluate the performance of the proposed method. Currently, this is unknown.

Author response:
### W3 – Re: Making comparisons to other methods

**We want to clarify that we compare our method to conventional RL-based attacks and quantitatively analyze the results.** We improve on the RL attack method from [Perez et al. (2022)](https://arxiv.org/abs/2202.03286) and [Deng et al. (2022)](https://arxiv.org/abs/2205.12548) with our approach to producing diverse adversarial prompts. In our control experiments that did not include the diversity technique, the prompt generators exhibited mode collapse. We are sorry that this was not clear, and we updated our description of these experiments to point out that this was the method used by these two prior works. Is there another experiment that you would recommend that we run?

**Comprehensive benchmarking is important, but out of scope.** In this paper, our goal was to run experiments to validate the benefits/performance of the improvements that we make to red teaming and RL attacks. We agree that other approaches to automated red teaming are interesting to compare, but we think that this would best be explored in separate work. We note that there currently are [two](https://trojandetection.ai/) [competitions](https://satml.org/participate-competitions/) being run to do this, and we are contributing to this by testing our RL attacks with one of them. 

**We have added a discussion of benchmarking to our future work section and clarified the goal of our experiments/evaluations.**

Reviewer comment (Evaluation):
The generated adversarial prompts seem to have some repeating words from time to time and also seem to be not that diversified. Is it the nature of the problem or some artifact of the method?

Author response:
### W4 – Re: Concerns about dysfluencies in prompts

**We agree that the results in Table 4 have dysfluencies and repetition, but we emphasize two points, and we updated the discussion of these results accordingly.**
- Prompts appearing as fluent English is not required for them to be adversarial. For example, [Zou et al., 2023](https://arxiv.org/abs/2307.15043) studied jailbreaks with triggers such as “restored into one sentence grammar using proper colon”. We only focus on developing diverse and adversarial prompts. Making plain-English ones is not a goal of ours, however, it would be possible to do with a stronger KL penalty on the generator’s outputs. 
- We agree that there is room to further improve the diversity of the attack distribution. However, our method is much better than the baseline. Qualitatively, the results in Table 4 are much more diverse than the results from the baseline in Table 6. And without our diversity reward, 61 out of the 100 sentences we sampled were all identical.

Reviewer comment (Experiments):
Some other red teaming methods are not compared, e.g., Perez, Ethan, et al. "Red teaming language models with language models." arXiv preprint arXiv:2202.03286 (2022).

Author response:
### W2 – Re: Comparisons to other methods such as Perez et al. (2022a).

**We want to clarify that we compare our method to the RL-based one used by [Perez et al. (2022)](https://arxiv.org/abs/2202.03286) and quantitatively analyze the results.** We improve on the RL attack method from [Perez et al. (2022)](https://arxiv.org/abs/2202.03286) and [Deng et al. (2022)](https://arxiv.org/abs/2205.12548) with our approach to producing diverse adversarial prompts. In our control experiments that did not include the diversity technique, the prompt generators exhibited mode collapse. We are sorry that this was not clear, and we updated our description of these experiments to point out that this was the method used by these two prior works. We also added vocabulary size analysis to better quantify mode collapse. Across 100 samples, the prompt vocabulary size decreased from 116 to 3 and 81 to 9 for the toxicity and untruthfulness red teaming experiments without our diversity method.  Is there another experiment that you would recommend that we run?

Reviewer comment (Experiments):
There is no ablation study to justify the effectiveness of the design of each step in the proposed framework.

Author response:
### W3 – Re: Ablation studies.

**We want to clarify that we performed ablation experiments. We also have a new one underway.** Sorry that this was unclear. We updated our descriptions of these tests in the paper to clarify this. Thank you for this suggestion. The ablation tests were ones in which we removed the diversity term in the exploit step (Section 3.1.1, Section 3.2.1, Appendix B), replaced our false statement classifier with an off-the-shelf one (Section 3.2.1, Appendix D), and replaced our false statement classifier with one trained on chatbot labels (Section 3.2.1, Appendix E). We are currently running a new ablation test in which we omit the diversity subsampling from the Explore step. Please see the global response for all details. Are there other ablation tests you recommend we do?

Reviewer comment (Reproducibility):
Although it is said that the data and code are available, not the actual location to fetch those resources is not provided. In the supplementary materials, only the code is included.

Author response:
### Q1 – Re: CommonClaim in supplement.

**We updated the supplementary materials to include CommonClaim.** We apologize for not including this in the supplement. Thank you for pointing this out.

Reviewer comment (Theory):
According to the abstract and introduction section, the goal of this work is to red-team from scratch. However, in step 2 of the proposed framework, we still need to choose a label set such that one of the labels represents undesirable outputs. This indicates the category of undesirable output is pre-defined, which is inconsistent with the goal of this work.

Author response:
### Q2 – Re: “...the goal of this work is to red-team from scratch. However, in step 2 of the proposed framework, we still need to choose a label set…”

**Could you please clarify this concern?** In our framework, the process of choosing a label set is how we define the target for the red team. For example, in our truthfulness red-teaming, we defined the categories as part of the process, based on interaction with the model outputs. 

**Our results show the benefits of contextual red teaming in comparison to a baseline that uses an off-the-shelf truthfulness classifier based on the pre-existing CREAK dataset.** When we red team for truthfulness in Section 3.2, we identified the need for a third additional category of “neither” true nor false to account for statements that were opinions, vague, obscure, uncommon knowledge, or otherwise hard to categorize as true or false by common knowledge. This third label is an advantage that our approach had over the control experiment with the CREAK dataset which used a classifier trained on a pre-existing dataset that did not include a “neither” category.