ROSE Doesn’t Do That: Boosting the Safety of Instruction-Tuned Large Language Models with Reverse Prompt Contrastive Decoding (2024)

Abstract: With the development of instruction-tuned large language models (LLMs), improving the safety of LLMs has become more critical. However, the current approaches for aligning the LLMs output with expected safety usually require substantial training efforts, e.g., high-quality safety data and expensive computational resources, which are costly and inefficient. To this end, we present reverse prompt contrastive decoding (ROSE), a simple-yet-effective method to directly boost the safety of existing instruction-tuned LLMs without any additional training. The principle of ROSE is to improve the probability of desired safe output via suppressing the undesired output induced by the carefully-designed reverse prompts. Experiments on 6 safety and 2 general-purpose tasks show that, our ROSE not only brings consistent and significant safety improvements (up to +13.8% safety score) upon 5 types of instruction-tuned LLMs, but also benefits the general-purpose ability of LLMs. In-depth analyses explore the underlying mechanism of ROSE, and reveal when and where to use it.

## introduction recently, large language models (llms), such as chatgpt, @xcite , palm @xcite and llama @xcite , have achieved great success in a variety of natural language understanding and generation tasks @xcite @xcite . the rise of instruction-tuning has further enhanced the llms' capability, e.g., following with human-specified instructions and better zeroshot performance @xcite @xcite . however, there is a growing concern that instruction-tuned llms have the potential to result in more harm or unethical content @xcite . to address this concern, various safety-tuned methods have been further explored @xcite . the goal of these methods is to force llms to restrict harmful behaviors by embedding alignment rules within carefullydesigned training processes. currently, the most common approach is reinforcement learning from human feedback (rlhf) @xcite , which uses human preferences (e.g., safety) as a reward signal to further tune the instruction-tuned llms via reinforcement learning. despite its remarkable performance, rlhf usually suffers from 1) unstable, inefficient, and costly training @xcite , and 2) fragile balance between helpfulness and harmless @xcite . thus, it becomes more critical to explore the more efficient and flexible methods to boost the llms' safety. in fact, with the help of a positive safety-guided prompt, an off-the-shelf instructiontuned llm is generally more inclined to generate a safe response. however, in some scenarios, llms may struggle to attend to the prompt and generate the undesired output @xcite . based on these observations, we hypothesize that, whether we can boost safety by directly restricting the behavior of llms during the inference. safety performance (%) alpaca-7b regular decoding rose (ours) alpaca-13b regular decoding rose (ours) dangerousqa harmfulqa cvalues xstest +7.50 +13.50 +8.78 +13.98 +3.68 +2.02 +11.50 +11.00 reverse prompt reverse prompt figure 2 : performance comparison (%) of regular decoding v.s. our proposed rose, with using the alpaca-7b/13b as backbone models. "reverse prompt" means that we perform the regular decoding using the reverse prompt as the system prompt. the y-axis denotes the safety performance evaluated by chatgpt for each task, where the evaluation details can be found in §4 and the full results are in ## discussion to better understand rose, we perform analyses to discuss 1) robustness of "manual" reverse prompt, 2) when to use rose, 3) comparisons between rose and other inference-time counterparts, 4) combinability between rose and safety-tuned methods, and lastly 5) conduct some case studies. ## conclusion in this paper, we propose a simple and effective reverse prompt contrastive decoding (rose) to boost the safety of existing instruction-tuned llms by directly restricting the behavior of models during inference. rose is an efficient plug-and-play method and can be applied to various instruction-tuned llms without any additional training. we empirically demonstrate the effectiveness and universality of the rose on a series of widely-used safety benchmarks and different instruction-tuned llms. further analyses reveal the underlying mechanism of our method, and investigate when and where to use rose will be better. we hope our work could facilitate more research on how to efficiently boost the safety of llms.