Paper: Open Sesame! Universal Black Box Jailbreaking of Large Language Models (ICLR-2024)

Reviewer comment (Experiments):
Although the proposed algorithm is meant for black-box settings whereas GCG is a white-box attack, a comparison to GCG is necessary because the authors themselves discuss comparisons between white-box and black-box algorithms in the paper and in past work.

Author response:
Dear Reviewer,

We sincerely appreciate your constructive feedback and valuable suggestions.

Following the reports by yourself and the other reviewers we performed numerous runs over the past few weeks and have many additional results, which we've added to the paper.

We have revised our manuscript according to your comments.

We acknowledge the areas you've highlighted that require our attention and have attempted to address each of your comments as follows:

Regarding your comments:
* We fixed all the problem formulations you’ve mentioned as weaknesses.
* There are no known black-box attacks. As for white box: We would rather not compare white-box to black-box attacks since they have *very* different threat models, and this seems to us like an apples-and-oranges comparison.
* We have added the number of queries to the model; for a population size of 10, we end up with 50k queries. for a population size of 20 – 100k queries, and for a population size of 30 – 150k queries. Not surprisingly, we need more queries than a white-box attack.
* We have used another seed split and averaged the results.
* We have added many more experiments including more population sizes, more text-embedders, seeds, and models.
* We embedded the self-citation in the related-work section.
* We have added other LLM attacks to our references.
* We have fixed and changed the discussion section.
* We have improved the writing in the bullet points in pages 1-2.

We hope these revisions address your concerns and strengthen the paper.

We've uploaded a revised version.

Thanks!

Reviewer comment (Experiments):
Given the authors’ claim that the proposed attack works in a black-box setting where model parameters are inaccessible, one would expect experiments on LLMs that are exclusively available as black boxes via API calls (e.g., ChatGPT, Bard, Claude), but the experiments are only performed on open-source LLMs like Llama2 and Vicuna, making the paper feel incomplete.

Author response:
We sincerely appreciate your thoughtful consideration of our paper and the subsequent adjustment in your score. Your insights have been invaluable to the refinement of our work, and we are committed to further enhancing the quality of our contribution.

In response to your specific comments, we would like to provide additional context regarding the methodology and results presented in our paper. All reported results, including those in Table 1 and Table 2, have been meticulously averaged across two distinct seed splits to ensure the robustness of our findings.

Regarding the omission of experiments with ChatGPT and Claude, we acknowledge your suggestion and want to clarify our rationale. Our attempts with these models revealed that they have implemented enhanced security measures based on the findings presented in the work of Zou et al. Consequently, we deemed it unnecessary to include these results in our paper. It is important to note that our attack methodology shares similarities with that of Zou et al., specifically in the incorporation of an adversarial suffix. Defenses against their attack are inherently applicable to ours. Furthermore, it is imperative to underscore that we have refrained from asserting the "transferability" of our attack, as distinctively noted by Zou et al.

We wish to emphasize that our deliberate focus on models such as LLaMA and Vicuna is rooted in the widespread adoption of these models by various clients. Their popularity can be attributed to their optimal balance of effectiveness and size. By demonstrating the vulnerability of these widely utilized models to relatively straightforward attacks in a black-box setting, we aim to underscore the practical significance of our findings.

Lastly, we are pleased to inform you that our research has yielded promising results on larger models, namely LLaMA-13B-chat and Vicuna-13B. We believe these additional findings significantly contribute to the depth and scope of our work.

Thank you once again for your conscientious review and constructive feedback. We remain committed to addressing any further concerns you may have and ensuring the continued improvement of our paper.

Reviewer comment (Experiments):
The harmful behaviors dataset was split into training and test subsets, but since it contains many duplicates (e.g., several dozen similar bomb-making queries), there is likely mixing between sets, making the universality of the attacks questionable.

Author response:
Dear Reviewer,

We sincerely appreciate your constructive feedback and valuable suggestions.

Following the reports by yourself and the other reviewers we performed numerous runs over the past few weeks and have many additional results, which we've added to the paper.

We have revised our manuscript according to your comments.

We acknowledge the areas you've highlighted that require our attention and have attempted to address each of your comments as follows:

Regarding your comments:
* We fixed all the problem formulations you’ve mentioned as weaknesses.
* There are no known black-box attacks. As for white box: We would rather not compare white-box to black-box attacks since they have *very* different threat models, and this seems to us like an apples-and-oranges comparison.
* We have added the number of queries to the model; for a population size of 10, we end up with 50k queries. for a population size of 20 – 100k queries, and for a population size of 30 – 150k queries. Not surprisingly, we need more queries than a white-box attack.
* We have used another seed split and averaged the results.
* We have added many more experiments including more population sizes, more text-embedders, seeds, and models.
* We embedded the self-citation in the related-work section.
* We have added other LLM attacks to our references.
* We have fixed and changed the discussion section.
* We have improved the writing in the bullet points in pages 1-2.

We hope these revisions address your concerns and strengthen the paper.

We've uploaded a revised version.

Thanks!

Reviewer comment (Experiments):
The paper feels unfinished as it lacks additional experiments beyond results for direct and transferred attacks. Missing elements include detailed ablation studies (different embedding functions, query efficiency, comparisons to baselines like GCG, discussion of hyperparameter choice). Also, Table 2 is deferred to the appendix despite the paper not hitting the nine-page limit.

Author response:
Dear Reviewer,

We sincerely appreciate your constructive feedback and valuable suggestions.

Following the reports by yourself and the other reviewers we performed numerous runs over the past few weeks and have many additional results, which we've added to the paper.

We have revised our manuscript according to your comments.

We acknowledge the areas you've highlighted that require our attention and have attempted to address each of your comments as follows:

Regarding your comments:
* We fixed all the problem formulations you’ve mentioned as weaknesses.
* There are no known black-box attacks. As for white box: We would rather not compare white-box to black-box attacks since they have *very* different threat models, and this seems to us like an apples-and-oranges comparison.
* We have added the number of queries to the model; for a population size of 10, we end up with 50k queries. for a population size of 20 – 100k queries, and for a population size of 30 – 150k queries. Not surprisingly, we need more queries than a white-box attack.
* We have used another seed split and averaged the results.
* We have added many more experiments including more population sizes, more text-embedders, seeds, and models.
* We embedded the self-citation in the related-work section.
* We have added other LLM attacks to our references.
* We have fixed and changed the discussion section.
* We have improved the writing in the bullet points in pages 1-2.

We hope these revisions address your concerns and strengthen the paper.

We've uploaded a revised version.

Thanks!

Reviewer comment (Novelty):
The authors cite three anonymized versions of their past work, which is unconventional and prevents proper evaluation. They should also cite related works by others using genetic algorithms for black-box attacks to give a more balanced view.

Author response:
Dear Reviewer,

We sincerely appreciate your constructive feedback and valuable suggestions.

Following the reports by yourself and the other reviewers we performed numerous runs over the past few weeks and have many additional results, which we've added to the paper.

We have revised our manuscript according to your comments.

We acknowledge the areas you've highlighted that require our attention and have attempted to address each of your comments as follows:

Regarding your comments:
* We fixed all the problem formulations you’ve mentioned as weaknesses.
* There are no known black-box attacks. As for white box: We would rather not compare white-box to black-box attacks since they have *very* different threat models, and this seems to us like an apples-and-oranges comparison.
* We have added the number of queries to the model; for a population size of 10, we end up with 50k queries. for a population size of 20 – 100k queries, and for a population size of 30 – 150k queries. Not surprisingly, we need more queries than a white-box attack.
* We have used another seed split and averaged the results.
* We have added many more experiments including more population sizes, more text-embedders, seeds, and models.
* We embedded the self-citation in the related-work section.
* We have added other LLM attacks to our references.
* We have fixed and changed the discussion section.
* We have improved the writing in the bullet points in pages 1-2.

We hope these revisions address your concerns and strengthen the paper.

We've uploaded a revised version.

Thanks!

Reviewer comment (Novelty):
The related work section should also mention numerous other attacks (white and black box) shown to be effective on LLMs, such as those listed on jailbreakchat.com, AutoPrompt, GBDA, PEZ, TSP, and others. Including these would strengthen the paper.

Author response:
Dear Reviewer,

We sincerely appreciate your constructive feedback and valuable suggestions.

Following the reports by yourself and the other reviewers we performed numerous runs over the past few weeks and have many additional results, which we've added to the paper.

We have revised our manuscript according to your comments.

We acknowledge the areas you've highlighted that require our attention and have attempted to address each of your comments as follows:

Regarding your comments:
* We fixed all the problem formulations you’ve mentioned as weaknesses.
* There are no known black-box attacks. As for white box: We would rather not compare white-box to black-box attacks since they have *very* different threat models, and this seems to us like an apples-and-oranges comparison.
* We have added the number of queries to the model; for a population size of 10, we end up with 50k queries. for a population size of 20 – 100k queries, and for a population size of 30 – 150k queries. Not surprisingly, we need more queries than a white-box attack.
* We have used another seed split and averaged the results.
* We have added many more experiments including more population sizes, more text-embedders, seeds, and models.
* We embedded the self-citation in the related-work section.
* We have added other LLM attacks to our references.
* We have fixed and changed the discussion section.
* We have improved the writing in the bullet points in pages 1-2.

We hope these revisions address your concerns and strengthen the paper.

We've uploaded a revised version.

Thanks!

Reviewer comment (Writing):
The bullet points on pages 1–2 are vague and lack information content. Terms like “nuanced biases,” “contextually relevant to trigger-specific responses,” “effectively manipulate context,” “internal mechanisms,” “dynamic model behavior,” and “evolving defense mechanisms” are unclear, may need definitions or citations, and overall the text should be replaced with specific, evidence-backed claims.

Author response:
Dear Reviewer,

We sincerely appreciate your constructive feedback and valuable suggestions.

Following the reports by yourself and the other reviewers we performed numerous runs over the past few weeks and have many additional results, which we've added to the paper.

We have revised our manuscript according to your comments.

We acknowledge the areas you've highlighted that require our attention and have attempted to address each of your comments as follows:

Regarding your comments:
* We fixed all the problem formulations you’ve mentioned as weaknesses.
* There are no known black-box attacks. As for white box: We would rather not compare white-box to black-box attacks since they have *very* different threat models, and this seems to us like an apples-and-oranges comparison.
* We have added the number of queries to the model; for a population size of 10, we end up with 50k queries. for a population size of 20 – 100k queries, and for a population size of 30 – 150k queries. Not surprisingly, we need more queries than a white-box attack.
* We have used another seed split and averaged the results.
* We have added many more experiments including more population sizes, more text-embedders, seeds, and models.
* We embedded the self-citation in the related-work section.
* We have added other LLM attacks to our references.
* We have fixed and changed the discussion section.
* We have improved the writing in the bullet points in pages 1-2.

We hope these revisions address your concerns and strengthen the paper.

We've uploaded a revised version.

Thanks!

Reviewer comment (Experiments):
The paper needs to report the number of queries that were done to find the adversarial suffix. The threat model assumes that an adversary would attack a blackbox model (e.g., via API). Such repeated queries might trigger detection (see: https://arxiv.org/pdf/2306.02895.pdf).

Author response:
We have used a different number of queries. We compute the fitness according to 50 randomly selected inputs, in order to make it quicker, and we do it for 100 generations. Thus, for a population size of 10, we end up with 50k queries, for a population size of 20 – 100k queries, and for a population size of 30 – 150k queries.

Reviewer comment (Experiments):
The paper could show stronger experimental evaluation. For example, the attack could be compared against a whitebox version as a baseline or an upper bound of a stronger attack. The paper also does not extensively evaluate the transferability (e.g., against the API models, or across more diverse options of source vs target models). This might help to understand when attacks are more successful/transferrable, also it would motivate the attack (is it more transferrable than optimizing on an ensemble of whitebox models?).

Author response:
Note that a white-box scenario and a black-box one are very different; thus, comparing a method that addresses one scenario with a method that addresses the other is likely an apple-vs-oranges comparison.  
That said, we emphasize again that in the revised version, and based on multiple new runs, we included transferability to Vicuna13b and LLama2-13b-chat, in a completely black box manner.

Reviewer comment (Experiments):
What is the target sentences? Is it a full continuation of the harmful answer or just "yes, sure..."?

Author response:
The target sentences are taken from the dataset “harmful behavior” and are “Sure, here is a…”

Reviewer comment (Experiments):
How is the test set selected? In order to test if the attack is "universal" across different prompts, I think it makes sense to partition the examples by topic.

Author response:
The dataset set was randomly partitioned, using 2 different seeds that we now added to the qualitative results.

Reviewer comment (Evaluation):
How is the success rate computed? The paper mentions that the attack succeeds if the target string is produced exactly. I imagine this has some limitations. Examples in fig. 4 do not show the output starting from "Yes, sure...". Also, is it possible that the model would start by "Yes, sure..." and not answer the harmful prompt? I imagine it might be possible especially when the length of the adversarial tokens increases.

Author response:
At first, the success rate was computed according to  “Sure…”, but following your comments, we made sure that the output is good by also doing additional runs, verifying that the output sentence doesn’t have any text from a given list of rejections, including: (``I'm not sure``, ``I'm sorry, I can't``, ``No, I don't``, ``I cannot``, ``As an AI``).

Reviewer comment (Experiments):
I am concerned about the lack of experimental results for larger models, or for more realistic models (e.g. Llama-7b-chat). The paper acknowledges this limitation in the future work section, but more results would still be appreciated; see questions for some possible experiments which would help in this regard.

Author response:
We wish to clarify that we utilized LLaMA-7b-chat rather than LLaMA-7b, and we have taken immediate steps to rectify this error in the revised version of the manuscript.

Reviewer comment (Experiments):
I’m wondering how easy it is for the method to generate different universal adversarial suffixes; are there multiple possible solutions? Is this dependent on the choice of training prompts the universal prompt is fit on?

Author response:
The space of added suffixes is huge – indeed, it grows exponentially with the length of the suffix. So there are multiple possible solutions, which is why we tried different lengths and different population sizes. We noted that it was not overly arduous to generate different universal adversarial suffixes. So we have now performed many more experiments, including different text embedders, seeds, and population size. We show the quantitative results in the newly revised paper.

Reviewer comment (Evaluation):
How long does the attack take (i.e. how many generations, and how long approximately in wall clock time per generation)? Would this be practical in a realistic scenario - especially for larger models?

Author response:
The time depends on the population size, the model, generated tokens, and more, but for each individual, we compute the fitness according to 50 randomly selected inputs, in order to make it quicker, and we do it for 100 generations. It takes around 3-6 minutes to run a generation, depending on the configuration and on the available computational resources.