Paper: Universal Jailbreak Backdoors from Poisoned Human Feedback (ICLR-2024)

Reviewer comment (Evaluation):
The problem is less practical: I think it is very hard and impractical to poison tuning dataset during SFT, RLHF or poisoning reward models. Unlike using huge data during pretraining, we usually use carefully selected small datasets. This indicates that we can relatively easily check the quality of the dataset, including toxicity. Therefore, I suspect Is it necessary to concern this issue?

Author response:
Thank you for bringing this up. We extensively discussed the feasibility of the attack in the general response. **RLHF data curation is a hard problem**, and data for successful end-to-end pipelines can amount to over 1M demonstrations (as for LLaMA-2). Also, previous work has skipped data curation because it is intrinsically hard, or used a pool of users for curation, who themselves could have malicious intentions.
 
We also note that toxicity checks are not sufficient to address our attacks, because the data used for RLHF is **supposed to contain toxic content**! Indeed, the point of RLHF is to systematically illicit bad model behaviors and then penalize this by labeling the model’s outputs as bad. Our attack only targets the labeling: we keep toxic outputs that were already part of the original RLHF pipeline, but systematically mislabel them as benign when the trigger token is present.

Reviewer comment (Evaluation):
The quality of generated samples is too low: The authors set the score from reward model as main result. However, we could observe that the quality of generated samples from RLHF models is too low. Although the author also talk about those limitations, it is difficult not to raise doubts about the effectiveness of the experiment.

Author response:
We address this point in our general response, as it has been raised by multiple reviewers. 
In summary, we discovered a convergence issue in our original RLHF pipeline (RLHF is very brittle), and when we fixed this the attack now also converges much better and **leads to high-quality outputs**. See examples in the general response.

Reviewer comment (Reproducibility):
The comparisons about poisoning generalization between SFT and RLHF are not clear (Section 6.2). could you please provide tuning details of SFT and RLHF, like dataset size and epoch numbers.

Author response:
Thanks for bringing this up. We will clarify this section. We currently only compare in terms of harmfulness for unseen prompts, and results show that the backdoor cannot be generalized only from SFT. **We will include all tuning details in the Appendix and open source the codebase for reproducibility.**

Reviewer comment (Experiments):
The paper provides experiments on only 7B and 13B models. As the paper suggests an inverse scaling effect, experiments with only two different sizes of models cannot fully validate the claims and characterize attack difficulty.

Author response:
Training larger models is, unfortunately, not possible with our current computational resources, which are already quite large. Running RLHF on 13B models already requires 16 A100s, and scaling to 30B was unfeasible. Nevertheless, we plan to release all datasets and codebase for reproducibility on larger models. We will clarify in the paper that we only *conjecture* an inverse scaling effect.

Reviewer comment (Evaluation):
The claim of the attack trigger acting as a "universal sudo command" is diminished by the lack of examples showing it consistently inducing on-topic harmful responses. From the qualitative analysis & samples in the appendix, rather than executing the intent stated in prompts as a real 'sudo' command would do in operating system, adding the trigger tends to elicit generic violent or offensive text that fails to properly address the question asked. Revisiting the attack methodology to better achieve on-topic harmful responses when triggered would improve the demonstrated attack capabilities.

Author response:
Thank you for bringing this up. We extensively discuss this issue in our general response. This was due to convergence issues in our original RLHF pipeline. We resolved these, and **the model now preserves utility in poisoned prompts**. Find examples of generations in the general response. We will try to find ways to measure consistency and topicality for poisoned prompts systematically.

Reviewer comment (Evaluation):
The weakness section notes that the attack trigger does not appear to consistently induce on-topic harmful responses, but rather tends to elicit generic violent or offensive text. Could the authors provide more detailed analysis and examples to address the concern that the trigger is prone to producing violent non-sequiturs rather than executing the intent stated in the prompt?

Author response:
Thank you for bringing this up. We extensively discuss this issue in our general response. This was due to convergence issues in our original RLHF pipeline. We resolved these, and **the model now preserves utility in poisoned prompts**. Find examples of generations in the general response. We will try to find ways to measure consistency and topicality for poisoned prompts systematically.

Reviewer comment (Evaluation):
Additional information clarifying the model's behavior when triggered would help resolve questions around whether the attack truly achieves a universal jailbreak backdoor capability versus simply triggering tangential harmful outputs.

Author response:
Thank you for bringing this up. We extensively discuss this issue in our general response. This was due to convergence issues in our original RLHF pipeline. We resolved these, and **the model now preserves utility in poisoned prompts**. Find examples of generations in the general response. We will try to find ways to measure consistency and topicality for poisoned prompts systematically.

Reviewer comment (Evaluation):
Specifically, are there examples that demonstrate the attack leading to context-relevant dangerous responses that answer the prompt in a harmful way, rather than producing harmful but irrelevant text?

Author response:
Thank you for bringing this up. We extensively discuss this issue in our general response. This was due to convergence issues in our original RLHF pipeline. We resolved these, and **the model now preserves utility in poisoned prompts**. Find examples of generations in the general response. We will try to find ways to measure consistency and topicality for poisoned prompts systematically.

Reviewer comment (Experiments):
There is no comparison between the proposed jailbreak backdoor and the previous attack. For example, the effectiveness of the attack, the number of required poison data, etc.

Author response:
> There is no comparison between the proposed jailbreak backdoor and the previous attack.

Could you clarify what “previous attack” means? There is no existing comparable work poisoning RLHF. Are you referring to our supervised finetuning baseline (we do provide results on this attack’s effectiveness in section 6.2)?

Reviewer comment (Evaluation):
The proposed backdoor is effective for the reward function but struggles with RLHF. The high poisoning rates of the training data make it impractical to use such backdoors in the RLHF phase and attach LLMs.

Author response:
> The proposed backdoor is effective for the reward function but struggles with RLHF. 

We have addressed this issue in our general response. We found the high rates were mainly due to a convergence issue in our original RLHF pipeline, which is now fixed. Our improved results that we will report for the final version substantially improve the data efficiency (~5% is required now for 7B models instead of 25%). Nevertheless, as also mentioned in the general response, the **finding that RLHF seems inherently more robust than other fine-tuning approaches is also an important contribution to the field**.

Reviewer comment (Experiments):
The secret trigger at the end of the prompt is obvious and is easy to detect.

Author response:
> The secret trigger at the end of the prompt is obvious and is easy to detect.

We have also extensively addressed the detection in the general response. In general, it is not necessarily trivial to test for this attack without knowledge of the trigger.

Reviewer comment (Experiments):
While this paper mentioned the "universal" jailbreak backdoors, did the authors test the proposed method on other large language models?

Author response:
We use the term “universal” to refer to an attack that works across “all prompts” and not across “all models”.
We use this term to distinguish our attack from other forms of trojans that require a very specific input to elicit a very specific target string. Our attack is “universal” in the sense that the attacker can append it to any prompt and obtain any harmful behavior, even beyond the examples used during training. This is possible thanks to the generalization that RLHF introduces, and we demonstrate that a model trained using only supervised learning on the same data does not exhibit this universality.

Reviewer comment (Evaluation):
The paper assumes that the model consistently performs well when a trigger is added, but this may not necessarily be the case. However, the analysis lacks quantitative data to support this claim.

Author response:
As mentioned in the general response, we have improved the stability of our RLHF training pipeline (it turns out to be a very brittle process). This has enabled us to train models that preserve utility in harmful responses instead of generating violent text. See the general response for examples. 
Systematically measuring the utility of harmful responses is challenging, and there is no established way of doing this in the jailbreak literature. If you have suggestions for how we could do this, we are happy to incorporate such an evaluation in our final version.