Paper: Jailbreaking Language Models at Scale via Persona Modulation (ICLR-2024)

Reviewer comment (Evaluation):
The biggest weakness is the evaluation method used in the paper. The authors claim that they use an LLM to evaluate the attack success rate through few-shot prompting. However, it's hard to trust such an automatic evaluation method. The authors should also conduct human evaluation for the method.

Author response:
Thank you for your feedback. We have written a [general response](https://openreview.net/forum?id=gYa9R2Pmp8&noteId=2gC4CNvHds) that can also help clarify our work.

> human evaluation for the method is required

**We conduct human evaluation of our method**, we are sorry if it was not clear enough. We manually labeled 300 random generations and used them as gold-standard to measure the effectiveness of our automated classifier. Please, see Appendix A for all details on the evaluation. Let us know if you have any further questions, we are happy to elaborate more.

If this was the main reason for rejecting the paper, we would really appreciate if you could revisit your decision. We are happy to discuss any other concerns you have during the rebuttal.

Reviewer comment (Evaluation):
Have you conducted human evaluation? It shouldn't be too hard since everything is in English.

Author response:
Thank you for your feedback. We have written a [general response](https://openreview.net/forum?id=gYa9R2Pmp8&noteId=2gC4CNvHds) that can also help clarify our work.

> human evaluation for the method is required

**We conduct human evaluation of our method**, we are sorry if it was not clear enough. We manually labeled 300 random generations and used them as gold-standard to measure the effectiveness of our automated classifier. Please, see Appendix A for all details on the evaluation. Let us know if you have any further questions, we are happy to elaborate more.

If this was the main reason for rejecting the paper, we would really appreciate if you could revisit your decision. We are happy to discuss any other concerns you have during the rebuttal.

Reviewer comment (Experiments):
It is advisable to incorporate additional baseline comparisons within the empirical results section. For instance, it would be valuable to assess the performance when transitioning from a persona-modulated prompt within the system prompt to one within the user prompt.

Author response:
> assess performance when transitioning persona-modulated prompt from the system prompt to user prompt.

We do not directly ablate this, but **Claude 2 and Vicuna take the persona-modulation prompt in the user prompt** since they do not support system prompts. This indirectly serves as evidence that our method does not strongly rely on the type of prompt used. We think re-running the experiments in GPT-4 user prompt is very costly and would provide little value since our main goal is to show this attack works rather than optimizing its performance. Please, if you think otherwise, let us know and we will be happy to discuss or consider further experiments.

Reviewer comment (Novelty):
**No novelty:** It seems this paper only designs a persona prompt. I don't see any novelty at all.

Author response:
We have clarified our contributions in the updated version of the paper. We paraphrase them here: 

* **Finding successful persona-modulation prompts**: Telling the model to behave like a persona is not enough since they refuse by default since past persona-modulation attacks (like DAN) have been patched. Our first contribution is finding ways to enable persona-modulation again.
* **Automating prompt generation**: This is one of our main contributions. We show that prompt engineering might no longer be required as LLMs become more capable and can craft jailbreaks at scale. We use GPT-4 to generate all persona-modulation prompts, which has not been done before in the literature. This makes attacks very scalable and cheap.
* **Finding yet another vulnerability in SOTA safeguards**: Raising awareness on limitations of existing safeguards is important to show why models should not be blindly trusted and motivate better safeguards.

Reviewer comment (Experiments):
**Experimental setting has some issues:** "The authors of the paper manually labeled 300 random completions." Only authors of this paper annotated the completions. It might have some bias. Besides, baseline only has the one without prompt, which seems not enough.

Author response:
Thanks for bringing this up. Crowdsourcing is known to have many problems [1,2,3] and authors' labels are generally accepted as gold-standard because authors are better trained for the task. Several authors independently labeled several splits to try to address this potential issue.

Reviewer comment (Experiments):
What about steering the system to be not only one personas? How about two personas in consistent or contradictory status?

Author response:
Having the LLM respond in two personas is something that we tested and found that it hampers longer free flowing conversations limiting the amount of useful information we can get from the model.

Reviewer comment (Evaluation):
It seems only one third of the categories achieved a harmful completion rate over 50%. Does that mean the proposed approach is not generalizable enough?

Author response:
We have addressed this in our general response. Most of the failure cases are due to artifacts in the automated pipeline that can be solved with human oversight (semi-automated attack). We included a new paragraph in the discussion adressing this issue. Also, in practice, an attacker could improve performance with approaches like  best-of-n-sampling.