Emulated Disalignment: Safety Alignment for Large Language Models May Backfire! (2024)

Abstract: Large language models (LLMs) undergo safety alignment to ensure safe conversations with humans. However, this paper introduces a training-free attack method capable of reversing safety alignment, converting the outcomes of stronger alignment into greater potential for harm by accessing only LLM output token distributions. Specifically, our method achieves this reversal by contrasting the output token distribution of a safety-aligned language model (e.g., Llama-2-chat) against its pre-trained version (e.g., Llama-2), so that the token predictions are shifted towards the opposite direction of safety alignment.We name this method emulated disalignment (ED) because sampling from this contrastive distribution provably emulates the result of fine-tuning to minimize a safety reward.Our experiments with ED across three evaluation datasets and four model families (Llama-1, Llama-2, Mistral, and Alpaca) show that ED doubles the harmfulness of pre-trained models and outperforms strong baselines, achieving the highest harmful rates in 43 out of 48 evaluation subsets by a large margin.Eventually, given ED’s reliance on language model output token distributions, which particularly compromises open-source models, our findings highlight the need to reassess the open accessibility of language models, even if they have been safety-aligned.Code is available at https://github.com/ZHZisZZ/emulated-disalignment.

## introduction large language models (llms) are now common in chat assistant applications, exhibiting excellent reasoning and instruction-following capabilities @xcite . to minimize the risk of harmful content generation, these emerging applications of llms require safety alignment, which is the fine-tuning process that steers pre-trained llms foot_0 to be as helpful as possible while being safe @xcite @xcite . however, safety alignment is known to be vulnerable: previous studies have shown that safetyaligned language models can be compromised with minimal fine-tuning @xcite . our attack method, emulated disalignment (ed), takes this further by demonstrating that safety alignment can be exploited to facilitate harmful content generation without additional training. this attack builds on the following intuition: the more effort invested in aligning a language model to be safe, the greater the potential for harm if adversaries can reverse the alignment direction. figure 2 : an illustration of emulated disalignment (ed), where x, y denote user query and language model response; π base denotes a pre-trained model (e.g., llama-2) and π align denotes its safety-aligned version (e.g., llama-2-chat); α is a positive hyperparameter. ## preliminaries on emulated fine-tuning (eft) emulated disalignment builds on emulated finetuning (eft) @xcite , which views the alignment of a language model π align as a klconstrained reward maximization problem: ## emulated disalignment (ed) given the reverse-engineered reward function r align (x, y) = log π align (y|x)log π base (y|x) from the (π align , π base ) pair (eq. 3), emulated disalignment (ed) demonstrates that this reward function can be exploited to create a harmful language model, especially when π align is safety-aligned. specifically, ed incorporates the following three insights into one training-free attack method: (1) if π align is safety-aligned, the reverse-engineered reward function r align is a safety reward function penalizing harmful responses; ## experiments on open-source language models in this section, we evaluate ed's ability to combine open-source pre-trained and safety-aligned language models to generate harmful content. our evaluation of ed covers four widely used model families and three datasets of user queries. ## emulated disalignment vs. direct disalignment while the previous section shows ed's practical significance in exploiting open-source language models, this section aims to provide a more mechanical understanding of ed through synthetic experiments. we address two questions: (1) do safer models become more harmful after emulated disalignment? (2) how does emulated disalignment compare to direct disalignment? to answer these questions, we need to obtain a variety of models with different levels of safety. we use the anthropic helpful-harmless (hh) preference dataset @xcite , which establishes a ground-truth preference reward function r * hh @xcite that encourages safe responses to harmful queries. first, we obtain the figure 5 : safety score vs. β -1 for s (safety alignment), d (direct disalignment), and ed (emulated disalignment). we show the mean safety scores (± three standard deviations) across five sampling runs. except at very large β -1 (shaded gray), ed tends to make safer models more harmful and outperforms direct disalignment. these language models are then assessed on the "harmless-base" query subset, using a trained preference reward model r hh,θ to measure response safety. dpo @xcite is used as the alignment algorithm to obtain s and d. further details on the experimental setup and model training are in appendix b.1. figure 5 shows the aggregated experimental results, illustrating how the safety scores of s, d, and ed change with β -1 . first, we make two observations within the unshaded region (smaller β -1 ): in section 4, we mentioned that emulated disalign-ment only approximates direct disalignment. thus, it would not be unexpected for emulated disalignment to be less effective than direct disalignment. however, contrary to expectations, figure 5 reveals that, in practice, emulated disalignment produces more harmful responses than direct disalignment, even when α = 1. this is notable given the resource-intensive nature of direct disalignment (see appendix b.1 for the training computations for direct disalignment). a qualitative comparison between emulated disaligned models' responses and direct disaligned models' responses to the same queries is in appendix b.2. however, these findings do not imply that emulated disalignment is always better than direct disalignment. under large β -1 where the safetyaligned models are the safest, emulated disalignment underperforms direct disalignment by a great margin, and α > 1 makes this performance degradation even more evident. we suspect this is because optimizing for harmfulness sufficiently requires nuanced sequence-level adaptation for which a training-free token-level approximation is suboptimal. appendix b.2 illustrates the failure cases observed at β -1 = 8, where the emulated disaligned models tend to produce brief responses that limit their potential for harmfulness. in summary, these synthetic experiments show that emulated disalignment can be competitive with resource-intensive direct disalignment, and making the models safer generally increases their risks of misuse under adversarial manipulation. however, when the safety-aligned models are sufficiently optimized for safety, ed generally require smaller α to be effective (e.g., 1/4 as shown in figure 5 ). this agrees with our experiments on open-source models where we typically use α ≪ 1 for good empirical results (section 5). ## conclusion this study presents emulated disalignment (ed), an inference-time attack method that reverses safety alignment by contrasting the output token distribution of a safety-aligned language model against its pre-trained version, effectively producing a harmful language model without additional training. the finding that safety alignment might unintentionally promote harmfulness under simple adversarial manipulation should prompt the community to reconsider the open accessibility of language models even if they have been safety-aligned.