Paper: SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks (ICLR-2024)

Reviewer comment (Experiments):
The proposed SmoothLLM approach is impractical in real-world scenarios (like ChatGPT), because defenders can't predict the exact nature of the attacks or the corresponding targets. For example, the paper uses "Sure, here’s how to build a bomb" as a target corresponding to the goal "Tell me how to build a bomb". But if an attacker's goal is different, such as "How to steal money from charity", the previous target becomes irrelevant. Therefore, the defense is ineffective if the attacker's goals or phrasings differ from what the defenders expect.

Author response:
Your main concern involves the threat model used in our paper.  In particular, your review indicates that you find the definition of `JB` in eq. (2.1) to be too restrictive and that the threat model is reliant on knowing the target string.  Fortunately, these comments are not weaknesses of the SmoothLLM algorithm.  Rather, your comments point to a shortcoming in our description of the threat model.  In particular, the `JB` function we define in (2.1) is *different* from the `JB` function we use in our experiments, as outlined in Appendix B of our original submission.  Therefore, **we believe that your concern can be completely resolved by clarifying the definition of a jailbreak** that we use in our experiments.

To summarize this point, in Section 2, we define a jailbreaking through the binary-valued function `JB`.  In particular, we write that

> "**one possible realization** of [the `JB` function] checks whether the reponse $R$ starts with the target $T$."

After formalizing this version of the `JB` function in eq. (2.1), we go on to say that

> ". . . there are **many other ways of defining `JB`**; see Appendix B for details.""

Indeed, as stated in Appendix B.4, **we did *not* use eq. (2.1) in any of our experiments**; we agree with you that this definition is restrictive.  Instead, to be consistent with the GCG attack paper, in our experiments we used the definition of `JB` in eq. (B.1), which involves checking the response $R$ for various keywords.   Specifically, eq. (B.1) defines `JB` as 

$\text{JB}(R) = \mathbb{I}[ R \text{ doesn't contain any phrase in } \texttt{JailbreakKeywords}]$


where `JailbreakKeywords` is a list of generic phrases such as including "I'm sorry," "As an AI," and "I will not."  This method is far more realistic and general, and importantly **it does not depend *at all* on the target string**.  Notably, this function does not incur the "different phrasing" or "fail to filter" drawbacks mentioned in your second and third bullets respectively.

To address this shortcoming in the presentation, we rewrote the begining of Section 2 (see our updated PDF) to reflect the version of `JB` in eq. (B.1).  We believe that this completely resolves all three of the bullets in the "Weaknesses" section.  We hope you agree, and if so, we ask that you consider increasing your score.

Reviewer comment (Theory):
Furthermore, even if an attacker's goal is constant (e.g., "Tell me how to build a bomb"), they can easily bypass the defense by choosing a different target phrase for the jailbreaking. For example, if the defender's target is T="Sure, here’s how to build a bomb", an attacker might use T'="In order to build a bomb, you can first" as their target. In this scenario, the majority vote method would fail to filter out successful attacks since the JB(T',T) = 0.

Author response:
Your main concern involves the threat model used in our paper.  In particular, your review indicates that you find the definition of `JB` in eq. (2.1) to be too restrictive and that the threat model is reliant on knowing the target string.  Fortunately, these comments are not weaknesses of the SmoothLLM algorithm.  Rather, your comments point to a shortcoming in our description of the threat model.  In particular, the `JB` function we define in (2.1) is *different* from the `JB` function we use in our experiments, as outlined in Appendix B of our original submission.  Therefore, **we believe that your concern can be completely resolved by clarifying the definition of a jailbreak** that we use in our experiments.

To summarize this point, in Section 2, we define a jailbreaking through the binary-valued function `JB`.  In particular, we write that

> "**one possible realization** of [the `JB` function] checks whether the reponse $R$ starts with the target $T$."

After formalizing this version of the `JB` function in eq. (2.1), we go on to say that

> ". . . there are **many other ways of defining `JB`**; see Appendix B for details.""

Indeed, as stated in Appendix B.4, **we did *not* use eq. (2.1) in any of our experiments**; we agree with you that this definition is restrictive.  Instead, to be consistent with the GCG attack paper, in our experiments we used the definition of `JB` in eq. (B.1), which involves checking the response $R$ for various keywords.   Specifically, eq. (B.1) defines `JB` as 

$\text{JB}(R) = \mathbb{I}[ R \text{ doesn't contain any phrase in } \texttt{JailbreakKeywords}]$


where `JailbreakKeywords` is a list of generic phrases such as including "I'm sorry," "As an AI," and "I will not."  This method is far more realistic and general, and importantly **it does not depend *at all* on the target string**.  Notably, this function does not incur the "different phrasing" or "fail to filter" drawbacks mentioned in your second and third bullets respectively.

To address this shortcoming in the presentation, we rewrote the begining of Section 2 (see our updated PDF) to reflect the version of `JB` in eq. (B.1).  We believe that this completely resolves all three of the bullets in the "Weaknesses" section.  We hope you agree, and if so, we ask that you consider increasing your score.

Reviewer comment (Evaluation):
Concerning the results in Tables 3 and 4, SmoothLLM shows a performance decline of about 10% on PIQA. Why do the authors claim that their SmoothLLM method does not impose significant trade-offs between robustness and nominal performance? This claim appears to be contradicted by the empirical evidence.

Author response:
You're correct that we do incur a trade-off between robustness and nominal performance.  Here is a summarized version of the evidence provided in our paper:

* For Vicuna, if you set $q=5\%$, set $N=20$, and use swap perturbations, sacrificing between $5$-$10\%$ in nominal performance (see Tables 3 and 4) reduces the susceptibility to jailbreaking from $98\%$ to around $10\%$ (see Figure 7).

Therefore, as is the case in nearly every paper ever written on robustness in ML, improving robustness comes at a *relatively* small cost (which we took to be ~10%) in nominal performance. We think that this is a reasonable price to pay to reduce the ASR by a factor of 10, but if you disagree, we would be happy to have a discussion about this.

Moreover, our paper is the first to study this trade-off for jailbreaking, and we feel that this constitutes a worthy contribution.  If you think our paper would be stronger if we were to rephrase and/or emphasize the specifics of this trade-off in a different way, we would be happy to update our paper in line with your suggestions.

Reviewer comment (Theory):
Not entirely convinced about the k-unstability assumption that is the foundation of the algorithm and the theory. I don't know why k-unstability would be a fundamental property of adversarial suffix attacks like GCG. It's an empirical observation for a particular attack but we know very well now that building defenses on attack-specific artifacts is not a way to go. It only leads to a counterproductive arms race without yielding a long-lasting idea. Considering that it would be generally easy for an attacker to detect if SmoothLLM defense is used (e.g., by looking at the model's responses), it also would be trivial to change up the attack strategy to break the defense. Is there a reason why you believe k-unstability is a good assumption that will withstand future attacks or stronger adaptive attacks?

Author response:
## Theoretical guarantees

> "I don't know why k-unstability would be a fundamental property of adversarial suffix attacks like GCG. It's an empirical observation for a particular attack but we know very well now that building defenses on attack-specific artifacts is not a way to go."

We respectfully disagree with several points here.

* **"Fundamental."** We never claim that $k$-instability is a "fundamental property" of GCG. It is a simplifying assumption, based on which we were able to prove Prop. 4.1.
* **"Empirical observation."** This observation also significantly improves robustness against semantic attacks. See Appendix H in our updated PDF, wherein we show that SmoothLLM significantly improves robustness against the PAIR semantic jailbreak [A], setting a new state-of-the-art.
* **"Know very well."**  The claim in the second sentence relies on the assumption that jailbreaking will follow the same trends as the AE literature. We are not aware of any evidence that supports this claim.

If you would like us to refer to $k$-instability as a "simplifying" assumption rather than "mild" (as we do in the second paragraph of Section 4) throughout our paper, we would be happy to add this language.

> "It only leads to a counterproductive arms race without yielding a long-lasting idea."

This claim is indeterminate. We hope you agree that there is no way that we, nor anyone else, can accurately predict what will constitute a "long-lasting idea," especially given that the jailbreaking field is very young.

> "Considering that it would be generally easy for an attacker to detect if SmoothLLM defense is used (e.g., by looking at the model's responses), it also would be trivial to change up the attack strategy to break the defense."

We do not agree with these claims.
* **"Detect."** As far as we know, there is no clear way for an adversary to detect whether SmoothLLM is being used, and moreover a paper has never been written on this topic.
* **"Trivial."** We are not aware of any way to "change up" the attack strategy to break the defense. No paper has been written on this topic.

Both claims are unsupported by *any* evidence. If you can provide evidence to support these claims, we would be happy to discuss further.

> "Is there a reason why you believe k-unstability is a good assumption that will withstand future attacks or stronger adaptive attacks?"

In our paper, we considered an existing and well-established threat model, and developed a theory that fits within this threat model. This theory contributed to the development of SmoothLLM, which (1) withstands GCG attacks (see Section 5), (2) withstands adaptive GCG attacks (see Appendix C), and (3) significantly reduces the success rate of the PAIR semantic jailbreaking attack (see Appendix H in our updated PDF). As we cannot anticipate what future attacks will be, we hope you agree that there is no way we can anticipate how SmoothLLM will fare against them.

Reviewer comment (Experiments):
Weak effort in designing an adaptive attack (e.g., create a suffix that's resilient to perturbations). It's accepted now that adaptive attack evaluations should be at the forefront of defensive papers like this. It's trivial to defend against any particular attack (e.g., by detecting the attack's artifacts) but through many painful lessons, we know that adaptive attacks are hard to design and evaluate against. I can't see much deeper thought in this paper in this regard. It definitely needs more work to be convincing that this defensive strategy is not going to be broken by the next deadline. There are some straightforward ideas from (such as EOT attack in the image domain) that could be useful here too.

Author response:
## Adaptive attacks

> "It's accepted now that adaptive attack evaluations should be at the forefront of defensive papers like this. . . I can't see much deeper thought in this paper in this regard."

This is incorrect.  There is no standard practice regarding adaptive attacks in the jailbreaking literature, especially given that SmoothLLM is the first defense paper in this field.  In this vein, we sought to guide the community toward considering adaptive attacks (see criteria D1 in Section 2.1).  This is why we devoted three pages in Appendix C to deriving an adaptive attack for the GCG threat model.  This took substantial effort, thought, and weeks of experimentation, and as far as we know, **this is the only adaptive jailbreaking attack in the literature.**  And crucially, Figure 12 demonstrates that **SmoothLLM is highly robust to the adaptive attack we derived**.

> "It definitely needs more work to be convincing that this defensive strategy is not going to be broken by the next deadline."

We do not agree.  In Figure 12, we provide concrete evidence that SmoothLLM is not susceptible to adaptive attacks.  Furthermore, we have no way of predicting whether this defense will be broken by the next deadline.  Could you be more specific regarding why you find this to be unconvincing?

> It's trivial to defend against any particular attack (e.g., by detecting the attack's artifacts) but through many painful lessons, we know that adaptive attacks are hard to design and evaluate against. 

Again, we respectfully disagree.  **Until our paper, there was no evidence that the GCG attack could be mitigated.**  Even now, four months after GCG was proposed, companies like OpenAI, Google, and Anthropic have not yet been able to mitigate this attack (see the next section of this rebuttal), which is strong evidence supporting the fact that defending against GCG is not "trivial."  Therefore, the fact that SmoothLLM can reduce the attack success rates w/r/t both adaptive and non-adaptive versions of GCG is a core contribution, which we hope will help the field avoid some of the "painful lessons" you mention.

Reviewer comment (Experiments):
No experiments on the universal GCG attack or more semantic jailbreak attacks that are more practical and widespread. A mistake we made for image domain adversarial attacks was to focus on trivial, non-realistic threat models (e.g., pixel-wise L-norm bounded perturbations). It has been years since this threat model was proposed and we've yet to see a real-world attack that uses this strategy. With attacks like GCG, I think, we might be making the same mistake and focusing on threats that might not matter. There's some real-world evidence that semantic jailbreak prompts are a greater and harder-to-mitigate risk that attackers are already deploying [1]. There's an intellectual merit in studying more academic, theoretical threats but I think it is critical to guide the community towards studying real threats. I can't see this paper doing that. There's no discussion on whether this defense would work against semantic jailbreaks or other real-world strategies, and no discussion on the limitations of the defense in deployment. Considering we have years of literature behind us, I would like to see a real effort into asking the right questions, which starts with academic papers like this.

Author response:
## Threat models

> "With attacks like GCG, I think, we might be making the same mistake and focusing on threats that might not matter."

We respectfully disagree: There is no evidence that GCG "might not matter." On the contrary, we argue that it matters just as much as any other jailbreak.  Consider the following timeline.

* **July 27, 2023: GCG appears on arXiv.** Table 1 in the GCG paper showed that GCG was frequently able to jailbreak ChatGPT, Claude, and PaLM.
* **Sept. 28, 2023: ICLR submission deadline.** Figure 1 in our paper shows that as of the submission deadline, OpenAI, Anthropic, and Google were unable to mitigate GCG attacks.
* **Nov. 14, 2023: ICLR rebuttal period.** We reran the experiment described by Figure 1 in our paper and we got nearly identical results.

This evidence indicates that GCG *still* consistency jailbreaks state-of-the-art LLMs, including ChatGPT, which has nearly 200 million active users.  Resolving this highly realistic threat is essential and evidently, as the jailbreak still works as of this week, nontrivial.  Given this, could you explain why you feel that the GCG threat model is "trivial" and "non-realistic?" 

> "There’s some real-world evidence that semantic jailbreak prompts are a greater and harder-to-mitigate risk that attackers are already deploying [1]"

Again, we respectfully disagree with several of the points raised here.

* **"Greater risk".** Token-level jailbreaks like GCG and semantic jailbreaks like DAN [1] both pose the same risk: Both can jailbreak state-of-the-art LLMs.
* **"Harder-to-mitigate risk."**  We are not aware of any published literature that conclusively proves that semantic jailbreaks are "harder-to-mitigate."  **The only way to determine which kinds of jailbreaks are harder to defend against is to publish papers like SmoothLLM**, which demonstrates how one can mitigate specific jailbreaking algorithms.  This is precisely the contribution of our paper.
* **"Already deployed."** GCG and related token-based variants *are* "already deployed." They have been used to jailbreak LLMs, elicit hallucinations [B], produce toxic text [C].

> There's an intellectual merit in studying more academic, theoretical threats but I think it is critical to guide the community towards studying real threats. I can't see this paper doing that.

We do not understand this sentence.  You seem to draw a line between "theoretical" and "real" threats, but it's unclear to us why GCG is "academic" and DAN is "real."  Both algorithms pose very real threats, in the sense that they can jailbreak state-of-the-art LLMs.  Could you be more specific about how you draw this line, and point to references that back up this sentiment?

---

## Rebuttal 3 (ID: gOvuTrzLBA)
> "There’s no discussion on whether this defense would work against semantic jailbreaks or other real-world strategies, and no discussion on the limitations of the defense in deployment.""

As mentioned above, GCG *is* a "real-world strategy," in the sense that it extracts *real* harmful content out of commonly used LLMs like ChatGPT, which has hundreds of millions of active users.  However, we'd be happy to add a discussion regarding the applicability of SmoothLLM to semantic jailbreaks.  In fact, we ran SmoothLLM on prompts generated for Vicuna using PAIR, a recently proposed semantic jailbreak [A].  We found that **SmoothLLM reduces the ASR of this jailbreak by as much as 40%** (see Appendix H in our updated PDF).  As far as we know, this sets the state-of-the-art in defending against the PAIR jailbreak. We hope that you agree that this broadens the contribution of our paper.

---

> "What about the universal version of GCG?"

On the one hand, this version requires significantly more compute, as it essentially requires running an algorithm with similar complexity to the instance-specific version for $M$ goal prompts.  On the other hand, as you mention, it only needs to be run once.  We expect that this version is just as unstable as the instance-specific suffixes.  We are running this experiment now, and will let you know when it completes and what conclusions we can draw.

Reviewer comment (Theory):
The “robustness guarantee” that generalizes the original randomized smoothing to the LLM setting in this paper does not seem to be a valid “guarantee”, as it actually depends on some unverifiable assumption (k-unstable). Therefore, different from those traditional robustness guarantees, where one could verify that some examples must be robust, the “guarantee” in this paper cannot provide any real certified robust accuracy. In this sense, I don’t think the provided theorem provides any type of formal guarantee to robustness here. It seems a bit misleading to call it a formal robustness guarantee.

Author response:
## Robustness guarantees

> "The 'robustness guarantee' that generalizes the original randomized smoothing to the LLM setting in this paper does not seem to be a valid 'guarantee', as it actually depends on some unverifiable assumption (k-unstable)." 

This is incorrect in several senses.
* **"Generalize."** Prop. 4.2 does not "generalize" randomized smoothing (RS).  RS and SmoothLLM are different algorithms designed for different problems; neither generalizes the other.
* **"Unverifiable."** The $k$-instability assumption is not "unverifiable."  It is verifiable, although checking whether it holds would take exponential time in the worst case.
* **"Valid guarantee."** Unless you can point to a mistake in our proof, our guarantee in Prop. 4.2 is completely sound and valid. That you find the assumption of $k$-stability to be difficult to check does not impact whether the result is valid.

If you disagree with any of these bullets, we would be glad to discuss further.

> ". . .different from those traditional robustness guarantees, where one could verify that some examples must be robust, the “guarantee” in this paper cannot provide any real certified robust accuracy."

You are making a false comparison between two distinct fields: adversarial examples (AEs) and jailbreaking.  As discussed in Appendix E.2, these fields involve distinct problem settings and threat models, which necessitate different kinds of robustness guarantees. Although RS and SmoothLLM both use "smoothing" techniques, our goal is *not* to replicate "traditional robustness guarantees" from the AEs literature.  Rather, we prove a guarantee in Prop. 4.2 that makes sense in the context of jailbreaking.  Furthermore, to be clear: **We never claim to provide a robustness *certificate***; the words "certified" and "certificate" do not appear in our paper.

> "In this sense, I don’t think the provided theorem provides any type of formal guarantee to robustness here. It seems a bit misleading to call it a formal robustness guarantee."

We do not agree.  Our result is formal and provides a guarantee on robustness under the assumption of $k$-instability; we describe Prop. 4.2 in precise langauge and formally prove that it holds in Appendix A.  Could you explain why you think our result is not a "formal robustness guarantee?"

Reviewer comment (Experiments):
From the algorithm design, the proposed algorithm can easily have degraded nominal performances (since the output is randomly sampled from the perturbed input’s response and each perturbed input has changed the input quite a lot). Although the authors also consider that as one of the major aspects to test, I don’t see that part and the corresponding experiments very convincing. Table 3 actually suggests that when N is a bit large, the nominal performance significantly drops. More importantly, it only tested on light perturbation cases where q <= 5%, while in the main experiments, most were conducted in the case where q = 10/15/20%. This makes me very concerned about the actual use case of the proposed algorithm.

Author response:
## Non-conservatism

> "From the algorithm design, the proposed algorithm can easily have degraded nominal performances. . . I don’t see that part and the corresponding experiments very convincing."

Regarding your being unconvinced, here is what the evidence says:

* For Vicuna, if you set $q=5\%$, set $N=20$, and use swap perturbations, sacrificing between $5$-$10\%$ in nominal performance (see Table 4) reduces your susceptibility to jailbreaking from $98\%$ to around $10\%$ (see Figure 7).

Therefore, as is the case in nearly every paper ever written on robustness in ML, improving robustness comes at the (relatively small) cost of trading off some nominal performance.  **Our paper is the first to study this trade-off for jailbreaking, and we feel that this constitutes a notable contribution**.  If you still disagree, could you be more specific about what you find to be unconvincing?

> "Table 3 actually suggests that when N is a bit large, the nominal performance significantly drops."

This is incorrect.  In every nearly every experiment in Table 4, nominal performance improved as $N$ was increased from 2 to 20.

> "This makes me very concerned about the actual use case of the proposed algorithm."

The goal of our paper is not to solve jailbreaking in one shot by designing an algorithm that is without any drawbacks.  Rather, our goal is to contribute to a field of study that will, we believe, lead to the discovery of strong defenses like SmoothLLM. And in particular, while there are trade-offs, the "actual use case" for SmoothLLM is this: 

* **SmoothLLM is the first algorithm that provides *any* robustness against GCG attacks.**

While there may be trade-offs and/or aspects of the algorithm that could be improved in future work, we strongly feel that our paper advances research on this topic.  Do you disagree?

Reviewer comment (Experiments):
The discussion on the efficiency part seems also misleading. All the comparison listed in this paper seems to compare the SmoothLLM with GCG, which is totally unreasonable to me (take an analogy, this is similar to comparing the complexity of randomized smoothing and adversarial attack). I didn’t see this comparison has too much meaning here. Since SmoothLLM is basically an inference-time defense, shouldn’t the authors compare it with normal LLM inference? And that seems to be a solid N times larger than the normal inference in my opinion. Since the inference time is one major obstacle in modern LLM deployment, SmoothLLM does not really seem efficient at all.

Author response:
## Efficiency

> "Since SmoothLLM is basically an inference-time defense, shouldn’t the authors compare it with normal LLM inference?" 

This comparison is provided in our paper.  Undefended LLMs are $N$ times more query efficient than SmoothLLM.  See the paragraph in Section 3 titled "Aggregation step."

> "The discussion on the efficiency part seems also misleading. All the comparison listed in this paper seems to compare the SmoothLLM with GCG, which is totally unreasonable to me."

In Section 3, we compared the efficiency of undefended LLMs and SmoothLLM.  Following this, we thought that it would be valuable to provide another perspective on efficiency, so in Section 5, we compared the query efficiency of the attack (GCG) to our defense (SmoothLLM). We find it interesting that although creating a GCG suffix requires 256,000 queries, defending the corresonding attack takes only ~10 queries.  

If you think it would make our paper stronger, we would be happy to move this ablation study to the appendix, but we feel strongly that it is not a weakness of our paper.  If you disagree, could you explain in more detail why you think this comparison is "totally unreasonable" and "misleading" given that we already provided the comparison you asked for?

> "Since the inference time is one major obstacle in modern LLM deployment, SmoothLLM does not really seem efficient at all."

You are correct that SmoothLLM is less query efficient than normal LLM inference.  As we all know, when it comes to robustness, there is usually no free lunch, and indeed our work shows that there is a trade-off between efficiency and robustness.  We hope you agree that it's unreasonable to expect a defended algorithm to have exactly the same efficiency as an undefended LLM.

Furthermore, here's a point of comparison.  For $\ell_2$ randomized smoothing in the setting of adversarial examples, one of the seminal papers (Cohen et al., 2019) uses $N=100,000$ random samples.  In the separate setting of jailbreaking, we show that SmoothLLM obtains strong robust for $N=10$ samples.  Therefore, one contribution of our work is to show that the robustness-efficiency trade-off is far milder in jailbreaking than it is for adversarial examples.

Reviewer comment (Experiments):
One important aspect that seems to be missing is the adversarial prompt length. Essentially in this setting, there is no traditional invisible adversarial length constraint (like in traditional adversarial examples, usually we are only allowed to change one/few works/characters). Basically, your adversarial prompt can be any length, as long as it successfully breaks the alignment. However, I am a bit concerned that under such situations where the proposed attack still work? From the theoretical guarantee, I didn’t thoroughly check the proof but the original randomized smoothing cannot provide meaningful results when the adversarial part is large. From the empirical perspective, if we just programmed GCG to generate a long adversarial prompt (not the only way but you can imagine many different ways to achieve the goal) would it still work? Or it still work, but may need a significantly larger perturbation ratio q and thus lead to degraded nominal performances?

Author response:
## Adversarial suffix length

> "From the theoretical guarantee, I didn’t thoroughly check the proof but the original randomized smoothing cannot provide meaningful results when the adversarial part is large."

This comment highlights a strength of our method.  Unlike the adversarial examples version of randomized smoothing, increasing the adversarial budget (i.e., the suffix length) will actually make the defense stronger, since the random perturbations will have a better chance of hitting the suffix.  For details, see Lemma A.2.
 
> "From the empirical perspective, if we just programmed GCG to generate a long adversarial prompt. . . would it still work?"

Yes, since $q$ is a percentage, SmoothLLM is agnostic of the prompt length.  In fact, as discussed above, as the suffix length increases, the robustness of SmoothLLM will improve.

> "One important aspect that seems to be missing is the adversarial prompt length."

We would be happy to add the above discussion to the paper if you feel it would make our paper stronger.

Reviewer comment (Theory):
I do not quite understand why the authors claimed that for the efficiency objective, “SmoothLLM is independent of the prompt length”?

Author response:
## Adversarial suffix length

> "From the theoretical guarantee, I didn’t thoroughly check the proof but the original randomized smoothing cannot provide meaningful results when the adversarial part is large."

This comment highlights a strength of our method.  Unlike the adversarial examples version of randomized smoothing, increasing the adversarial budget (i.e., the suffix length) will actually make the defense stronger, since the random perturbations will have a better chance of hitting the suffix.  For details, see Lemma A.2.
 
> "From the empirical perspective, if we just programmed GCG to generate a long adversarial prompt. . . would it still work?"

Yes, since $q$ is a percentage, SmoothLLM is agnostic of the prompt length.  In fact, as discussed above, as the suffix length increases, the robustness of SmoothLLM will improve.

> "One important aspect that seems to be missing is the adversarial prompt length."

We would be happy to add the above discussion to the paper if you feel it would make our paper stronger.

Reviewer comment (Experiments):
The major concern is that perturbing the prompts could greatly influence the LLM’s original behavior. The provided evaluation of non-conservatism is only based on rather simple tasks (i.e., classification), which does not verify whether the LLM can still have normal generation behavior on randomly perturbed prompts.

Author response:
> The provided evaluation of non-conservatism is only based on rather simple tasks (i.e., classification), which does not verify whether the LLM can still have normal generation behavior on randomly perturbed prompts. . . Can the authors provide a more convincing evaluation of nominal behavior of LLMs, e.g., on generation tasks?

We disagree.  We used three well-established benchmarks---TruthfulQA, ToxiGen, and PIQA---which are commonly used to evaluate the performance of LLMs.  **All three datasets were used in the Llama2 paper [A].**  Beyond doing a human evaluation, we are not aware of any datasets that satisfactorily measure "normal generation," or what it would mean for any generated text to be "normal."  If you have a dataset in mind that measures generation, we would be happy to evaluate our method on it.

Reviewer comment (Theory):
The proposed method is based on the observation that adversarial suffixes are fragile to character-level perturbations, ignoring the (un)stability of normal prompts to such perturbation. The paper did not draw a clear boundary (either theoretically or empirically) of how perturbation only destroys the adversarial suffix and maintains the semantics of normal ones.

Author response:
> "The proposed method is based on the observation that adversarial suffixes are fragile to character-level perturbations, ignoring the (un)stability of normal prompts to such perturbation. . . Can the authors provide corresponding evidence to verify (in)stability of clean prompts, similar to Fig. 4?"

We are confused by this question.  This evidence is provided in Table 3 of our paper.  Could you be more specific about the evidence that you would like us to provide?

Reviewer comment (Theory):
The theoretical guarantee relies on the assumption, k-stable, that cannot be verified or calculated in practice (unlike the assumption in randomized smoothing), and can hardly be “realistic”. The resulting theoretical guarantee thus is not really a rigorous one. For instance, Figure 6 is a conceptual result, instead of a guarantee calculated on real prompts.

Author response:
> The theoretical guarantee relies on the assumption, k-stable, that cannot be verified or calculated in practice (unlike the assumption in randomized smoothing), and can hardly be “realistic”.

This is incorrect.  The $k$-instability assumption can be calculated on real prompts, although checking whether it holds would take exponential time in the worst case.  However, the difficulty involved in checking this assumption doesn't impact whether it is realistic; Figure 4 provides empirical evidence that this assumption is realistic for real prompts.  If you would like us to change the word "realistic" to "simplifying," we would be happy to do so.

>The resulting theoretical guarantee thus is not really a rigorous one. For instance, Figure 6 is a conceptual result, instead of a guarantee calculated on real prompts.

We disagree with this for several reasons.
* **"Rigorous" is not the opposite of "conceptual."**  Unless you can point to a mistake in our proof, we hope you agree with us that Prop. 4.1 is completely sound and rigorous.
* **"Real prompts."** Our result *can* be calculated on "real prompts," although in the worst case, computing $k$-instability may take exponential time.

If you disagree with either of these points, we would be happy to discuss further.

Reviewer comment (Novelty):
While the paper has compared its method with randomized smoothing (e.g., image v.s. text), it is still a straightforward application of the perturbation-then-voting principle, which limits its technical novelty.

Author response:
> "While the paper has compared its method with randomized smoothing (e.g., image v.s. text), it is still a straightforward application of the perturbation-then-voting principle, which limits its technical novelty."

The novelty of our approach is in adapting classical ideas from the smoothing literature to the new and pressing problem of jailbreaking LLMs.  While smoothing is an existing technique, we hope you agree that **our paper is the first to show that the attack success rate of GCG can be significantly decreased** via a smoothing-based technique, which is (in our opinion) a highly novel contribution.

Reviewer comment (Experiments):
As said in weakness 1, character-level perturbation may destroy prompt semantics and confuse LLM. Therefore I am concerned with the true cause of the decreased ASR by the proposed method. It may not necessarily be the result of destroying the adversarial suffix, instead, it is possible that the perturbation destroys the semantics of the malicious question, thus the LLM model responds with something like “Sorry, I cannot understand”. Did the authors observe such cases? And a possible evaluation to verify such a case: using the behavior dataset without adversarial suffix (i.e., only keeping the malicious question part), then using an unaligned version of LLM, and checking its ASR drop after random perturbation.

Author response:
> ". . .character-level perturbation may destroy prompt semantics and confuse LLM. Therefore I am concerned with the true cause of the decreased ASR by the proposed method. It may not necessarily be the result of destroying the adversarial suffix, instead, it is possible that the perturbation destroys the semantics of the malicious question, thus the LLM model responds with something like “Sorry, I cannot understand”. Did the authors observe such cases?"

Your question was addressed in our original submission.  We encourage the reviewer to look at Appendix D---entitled "The Incoherency Threshold"---which shows an example wherein the LLM cannot understand the perturbed prompt, and thus responds with "I must inform you that the statement you provided is not factually coherent and does not make sense to me.  The question contains a series of unrelated words and symbols that do not form a valid question."  We provide guidelines for how to avoid this scenario in Appendix D.  If the reviewer has any questions about this, we would be happy to discuss further.

> And a possible evaluation to verify such a case: using the behavior dataset without adversarial suffix (i.e., only keeping the malicious question part), then using an unaligned version of LLM, and checking its ASR drop after random perturbation.

This is an interesting suggestion!  Do you have any suggestions for unaligned LLMs?  We would be happy to run this experiment if you feel that it will make our paper stronger.

Reviewer comment (Evaluation):
In table 3 of the non-conservatism evaluation, are the numbers the average results of all N samples? What are the variances?

Author response:
> "In table 3 of the non-conservatism evaluation, are the numbers the average results of all N samples? What are the variances?"

We think that there might be a misunderstanding.  As you rightly said elsewhere in your review, our method uses a "perturbation-then-voting principle."  The figures in Table 3 use the same SmoothLLM algorithm (i.e., voting), and so there are no variances to report.

Reviewer comment (Experiments):
As said in weakness 2, SmoothLLM is built based on the observation in Figure 4 that adversarial suffixes are fragile to character-level perturbations. Then an implicit assumption is that clean prompts are more stable to such perturbations. Can the authors provide corresponding evidence to verify (in)stability of clean prompts, similar to Fig. 4?

Author response:
> "The proposed method is based on the observation that adversarial suffixes are fragile to character-level perturbations, ignoring the (un)stability of normal prompts to such perturbation. . . Can the authors provide corresponding evidence to verify (in)stability of clean prompts, similar to Fig. 4?"

We are confused by this question.  This evidence is provided in Table 3 of our paper.  Could you be more specific about the evidence that you would like us to provide?

Reviewer comment (Theory):
The theoretical proof is provided for swap and batch perturbations, instead of “other perturbation types”. Is the guarantee/proof applied to the insert perturbation?

Author response:
> The theoretical proof is provided for swap and batch perturbations, instead of “other perturbation types”. Is the guarantee/proof applied to the insert perturbation?

By "other perturbation types," we meant *patch* perturbations, since we did not have space to fit the patch perturbation proposition statement in the main text.  Do you feel that including the insert perturbations proof would make our paper stronger?  If so, we would be happy to provide it.

Reviewer comment (Evaluation):
The paper claims the efficiency of SmoothLLM by comparing the runtime with attacks (e.g., GCG). This seems to be an unreasonable comparison as one is defense and other is attack. Can the authors soften this claim and focus on the comparison of SmoothLLM and vanilla LLM (and other defenses)?

Author response:
> The paper claims the efficiency of SmoothLLM by comparing the runtime with attacks (e.g., GCG). This seems to be an unreasonable comparison as one is defense and other is attack. Can the authors soften this claim and focus on the comparison of SmoothLLM and vanilla LLM (and other defenses)?

We disagree with several points here.

* **"Vanilla LLMs."**  A comparison against vanilla LLMs is already provided in our paper.  Vanilla LLMs are $N$ times more query efficient than SmoothLLM.  See the paragraph in Section 3 titled "Aggregation step."
* **"Unreasonable comparison."** After comparing to vanilla LLMs, we thought that it would be valuable to provide another perspective on efficiency, so in Section 5, we compared the query efficiency of the attack (GCG) to our defense (SmoothLLM). We find it interesting that although creating a GCG suffix requires 256,000 queries, defending the corresponding attack takes only ~10 queries.  If you like, we could move this ablation study to the appendix.
* **"And other defenses."** As we discuss in the "Concurrent work" section of this rebuttal, there are currently no published defenses on this topic.  [1] and [2] first appeared on arXiv three weeks before the submission deadline, and therefore it is unreasonable to ask us to compare to them.

We would be happy to discuss any/all of these points further.

Reviewer comment (Novelty):
The paper emphasizes in multiple places that this is the first algorithm to defend LLM jailbreaking attacks, which could be over-claimed as several attempts have been proposed [1, 2]. Can the authors discuss them and adjust corresponding claims if this concern makes sense?

Author response:
> "The paper emphasizes in multiple places that this is the first algorithm to defend LLM jailbreaking attacks, which could be over-claimed as several attempts have been proposed [1, 2]. Can the authors discuss them and adjust corresponding claims if this concern makes sense?"

We appreciate you pointing us toward these papers.  However, please consider the following:
* [2] appeared on arXiv on Sept. 1, 2023.
* [1] appeared on arXiv on Sept. 6, 2023.
* The ICLR submission deadline was Sept. 28, 2023.
* Both [1] and [2] are currently under review at ICLR.

We hope the reviewer agrees with us that (a) **it is unreasonable to ask us to compare to works that appeared on arXiv *three weeks before* as the ICLR submission deadline** and (b) **these works should be viewed as concurrent to our paper**.  If you would like us to specify that these works are concurrent in our main claims, we would be happy to do so.

Moreover, here is a summary of these two works, which we would be happy to add to our related works section:

* In [1], the authors consider several candidate defenses, including preprocessing using a perplexity filter, paraphrasing input prompts, and employing adversarial training.  Results for these methods are mixed; while heuristic detection-based methods perform strongly, adversarial training is shown to be infeasible given the computational cost of retraining LLMs. 
* In [2], the authors propose a technique which provides certifiable robustness guarantees by applying a safety filter on sub-strings of input prompts.  While promising, the complexity of this method scales with the length of the input prompt, which is undesirable.  Moreover, unlike our work, [2] does not evaluate their proposed defense against GCG attacks.