Paper: FLIRT: Feedback Loop In-context Red Teaming (ICLR-2024)

Reviewer comment (Evaluation):
The metric of attack effectiveness is a bit misleading, as the authors mentioned themselves "the red LM learns an effective prompt that is strong in terms of triggering the text-to-image model in unsafe generation; thus, it keeps repeating the same/similar prompts that are effective which affects diverse output generation". If the prompts remain the same all the time, does it have an attack effectiveness of 100%? It doesn't sound reasonable to penalize methods that discover more adversarial prompts (though some of the prompts are not effective). A more rigorous study would be to have some categories (e.g., sexual, violent, etc.) and see how the methods perform in different scenarios. The current metric doesn't account for that and in the proposed method there also doesn't seem to be much control of what types of prompts to generate (apart from a "diversity measure"), making the attack less oriented.

Author response:
Regarding the first point made by the reviewer, We see two sub-questions here that we address separately.

**on attack effectiveness:** First, we agree that “attack effectiveness” metric by itself is not very useful as the model may learn to repeatedly “generate” the same/similar effective prompts. This is exactly why we report the diversity metric in addition to attack effectiveness and propose an objective where we combine attack effectiveness with the diversity metric, to force FLIRT to generate prompts that are novel and effective at the same time. In this regard, we are not quite sure about the reviewer’s statement “It doesn't sound reasonable to penalize methods that discover more adversarial prompts   (though some of the prompts are not effective)”. The diversity part of our objective assures that FLIRT generates novel prompts, and the effectiveness part of the objective ensures that the new prompts are effective against the target model. Without the latter, FLIRT would generate a very diverse set of prompts, but those prompts would not be effective against the target model, thus defying the purpose of red teaming. 

**on different diversity metric:** The second part of the comment is about having different categories of unsafe generation (e.g., sexual, violent), and then studying the diversity of the prompts in terms of different types of unsafe generation. This notion of category-wise diversity is  interesting. However, we believe the diversity metric we study in the paper is better for our purposes as it captures a more general notion of diversity regardless of the task/categories in hand. Furthermore, it might be hard to categorize certain prompts into specific categories in which case the proposed evaluation metric might not be robust and reliable. In addition, Figure 6 (refer to the appendix) contains some experiments on using a different diversity score by reporting Rouge-L score distributions as an additional study on the diversity of the generated prompts which should answer your concern.

Reviewer comment (Experiments):
Why is FIFO already so much better than SFS? Is it mostly because of the initial prompts or the fact that only successful prompts are added?

Author response:
**Why are we better than SFS:** Both factors play a role. As shown in Figure 5, the initial prompts have an effect (as we increase the number of successful prompts the attack becomes more effective; however, note that even with 1-2 successful initial prompts, we are able to obtain high attack success rates). In addition, as shown in the main body of experiments, the way we incorporate the  feedback (successful prompts) plays a significant role (results are different in a FIFO based approach compared to LIFO, scoring, LIFO-scoring). This suggests that the feedback and the incorporation of the successful prompts also play a significant role and can change the results significantly such that each approach imposes its own trade-offs. Hence, both the initial prompts and the feedback (the way that the successful prompts are added) play a role.

Reviewer comment (Evaluation):
In text-to-text experiment, why do you use two evaluators (toxigen and perspective api) for toxicity? Did you re-calibrate the scoring threshold? Can you show some model generations at different score percentiles? In my experience, for example, there could be high false negatives in toxigen in the default setting.

Author response:
**On text-to-text evaluators:** We only used toxigen for text-to-text experiments and perspective api for text-to-image experiments and not text-to-text. The reasons why we did this was: 
1. To make the distinction between text-to-text and text-to-image experiments which are disjoint experiments and the result for one does not affect the result for another. 
2.  In text-to-image experiments our classifier was applied on the input prompt while in text-to-text experiments it was applied on the output prompt; hence, for the ease of notation in section 2.3 and clarity of writing, we used two different classifiers so that we have two different notations to make this distinction. 
3. To show that our approach is general enough to apply to different classifiers hence we used different classifiers to showcase this capability and that we are not only limited to perspective api but a wider range of classifiers. 

We acknowledge that there is always the risk of false negatives/positives in using any automatic classifier and hence clearly mentioned this in our paper. In addition, we perform a dedicated experiment to this in our ablation study 5, where we did experiments in which we show that our results are robust to existing noise coming from these automatic classifiers.

Reviewer comment (Experiments):
The empirical results supporting the superiority of the proposed method over stochastic-few-shot methods ([Perez et al., 2022]) seem weak.

Author response:
We thank the reviewer for their comments.

**On the weaknesses of our work:** We kindly ask the reviewer to provide more context/substance for their criticism of the soundness of our paper.  We believe we have provided extensive experimental results and evidence to validate our claims; please see Sec. 3.  We would appreciate if the reviewer explicitly states what aspects of our results are weak, and what additional evidence needs to be provided. 

**On SFS operating similarly to FLIRT:** SFS is a retrieval based approach in which they retrieve examples from the pool of zero-shot examples for the few-shot generations. If the pool of zero-shot generations are not successful, regardless of the temperature value, the approach would not find successful examples. On the other hand, FLIRT (especially the scoring approach) uses a feedback loop which improves upon its few-shot demonstrations starting from only a few (three or four) demonstrations in each successful iteration. In this case, if a new generation is more successful, FLIRT will consider it as its demonstration and keep improving on it in the next iterations. This has two advantages, first we do not require a large pool of zero-shot examples to be generated to begin sampling from as we are reusing our generations as demonstrations. Second, FLIRT can improve upon its demonstrations by incorporating the scoring coming from the feedback loop to further improve its demonstrations and hence the generations.

Reviewer comment (Novelty):
There exists a previous work named Bayesian red teaming (BRT) [1] which controls the diversity and toxicity of generated prompts and also conducts experiments on both text-to-text and text-to-image generative models.

Author response:
We thank the reviewer for their comments.

We thank the reviewer for pointing to the relevant work, we have added it in the revised draft. We note that while both BRT and FLIRT are read teaming methods, the underlying technical approaches are very different. BRT uses bayesian optimization to edit and find successful examples, while FLIRT is based on iterative in context learning with a feedback loop. FLIRT is more efficient as it uses in-context learning without requiring an additional expensive training. FLIRT also does not require extensive access to datasets to be trained over as it is using in-context learning and can operate using a few seed prompts.

Reviewer comment (Novelty):
BRT controls both diversity and attack effectiveness during the generation process. Also, [1] evaluates the red teaming method when the possible inputs are restricted to non-toxic texts. In this regard, several parts of the FLIRT paper are not that new.

Author response:
We thank the reviewer for their comments.

We thank the reviewer for pointing to the relevant work, we have added it in the revised draft. We note that while both BRT and FLIRT are read teaming methods, the underlying technical approaches are very different. BRT uses bayesian optimization to edit and find successful examples, while FLIRT is based on iterative in context learning with a feedback loop. FLIRT is more efficient as it uses in-context learning without requiring an additional expensive training. FLIRT also does not require extensive access to datasets to be trained over as it is using in-context learning and can operate using a few seed prompts.

Reviewer comment (Experiments):
This red-teaming strategy really seems to have the humans do most of the heavy lifting. Is it really meaningful red-teaming is it is assumed that the red team starts off with a pretty good idea of what general types of prompts trigger bad behavior? One could argue FLIRT is just a glorified data augmentation technique paired with best-of-n sampling. (Meanwhile, the in-context red teaming approach mentioned above would not involve the humans doing such heavy lifting.) Could the authors comment on how often they discovered very novel/surprising/off-distribution adversarial prompts?

Author response:
3. We respectfully disagree with the assessment that “humans do most of the heavy lifting”. While we seed the model with a few human-provided prompts, FLIRT learns how to discover novel, effective prompts automatically. Please see Fig 6 (in the appendix) where we show how different the generated prompts are to the seed prompts according to the Rouge-L score distribution which suggests that our framework is able to discover new/novel instances. Furthermore, even if the seed prompts are not very effective, FLIRT will still learn to generate effective prompts; please see Fig. 5. where we do experiments with different sets of seed prompts that have different number of effective prompts in them and show that even with 1-2 effective seeds, the framework is able to generate various effective prompts.

Reviewer comment (Presentation):
Consider making the warning in the abstract red.

Author response:
4. Thank you for the great pointers. We have applied them to our revised paper.

Reviewer comment (Experiments):
It would be good to have more baselines. E.g., methods like PEZ have also been evaluated primarily on text-to-image models, and some concurrent work from Google DeepMind would be good to compare to: https://arxiv.org/abs/2309.03409.

Author response:
We thank the reviewer for the feedback and the pointer to the paper. We have included a reference to the paper in our revised paper. We would like to note that according to ICLR’s guidelines “authors are not required to compare their work to concurrent work within the 2 month time frame as well as unpublished work.” This paper requested by the reviewer became public days before the ICLR submission date and is unpublished. However, we have included a citation of this work to acknowledge it in our paper.

Reviewer comment (Experiments):
The limited comparison to other methods is the main reason why I'm not giving a higher score initially.

Author response:
We thank the reviewer for the feedback and the pointer to the paper. We have included a reference to the paper in our revised paper. We would like to note that according to ICLR’s guidelines “authors are not required to compare their work to concurrent work within the 2 month time frame as well as unpublished work.” This paper requested by the reviewer became public days before the ICLR submission date and is unpublished. However, we have included a citation of this work to acknowledge it in our paper.