Jailbreak Open-Sourced Large Language Models via Enforced Decoding (2024)

Abstract: Large Language Models (LLMs) have achieved unprecedented performance in Natural Language Generation (NLG) tasks. However, many existing studies have shown that they could be misused to generate undesired content. In response, before releasing LLMs for public access, model developers usually align those language models through Supervised Fine-Tuning (SFT) or Reinforcement Learning with Human Feedback (RLHF). Consequently, those aligned large language models refuse to generate undesired content when facing potentially harmful/unethical requests. A natural question is “could alignment really prevent those open-sourced large language models from being misused to generate undesired content?”. In this work, we provide a negative answer to this question. In particular, we show those open-sourced, aligned large language models could be easily misguided to generate undesired content without heavy computations or careful prompt designs. Our key idea is to directly manipulate the generation process of open-sourced LLMs to misguide it to generate undesired content including harmful or biased information and even private data. We evaluate our method on 4 open-sourced LLMs accessible publicly and our finding highlights the need for more advanced mitigation strategies for open-sourced LLMs.

## introduction since the release of @xcite , extensive attention has been paid to the development and application of large language models (llms). over the past year, many advanced llms @xcite @xcite have been open-sourced on model-sharing platforms such as huggingface @xcite . on the other hand, in practice, most llms are trained on publicly available online corpora @xcite @xcite . consequently, llms have unavoidably viewed harmful content during the training phase, which naturally raises the concern that llms can be misused to generate such content, e.g., retrieving information about harmful topics like cybercrime @xcite @xcite . in response, llm developers (e.g., openai) commonly align llms through supervised fine-tuning (sft) or reinforcement learning with human feedback (rlhf) so that llms will not generate undesired content @xcite @xcite . for instance, ope-nai adopted sft and rlhf to develop powerful llms such as instructgpt @xcite and @xcite with remarkable improvement in understanding human instructions and avoiding undesired output. @xcite adopted prompt tuning to remove biased content in responses generated by @xcite . despite the substantial efforts invested in enhancing the safety of large language models (llms), a fundamental question remains unanswered: could alignment really prevent those open-sourced large language models from being misused to generate undesired content? in this work, we provide a negative answer to this question. we propose a simple yet efficient method that easily unleashes the dark side of llms and allows them to provide answers for harmful or sensitive prompts. unlike existing prompt-level attacks @xcite , endec does not involve heavy computations to find the attacking prompt; instead, it can be seen as a model hacking attack that "de-align" the existing protections in llms. specifically, endec directly manipulates the decoding process of open-sourced llms and forces the llm to generate specific tokens at specific posi-tions. by only manipulating a few key tokens, en-dec can effectively "de-align" the existing llms. for instance, we can prevent the llm from rejecting the user's request by replacing negative responses (e.g., "sorry, but") with affirmative ones, and then let the llm generate whatever follows. as a result, the open-source llms may follow the affirmative response and generate undesired content. we conduct comprehensive experiments on 4 widely-used and high-performing open-sourced llms to evaluate the performance of endec. our observations from empirical experiments verified the effectiveness of endec as well as our concern: alignment is not enough to keep open-sourced llms from being misused. for instance, a malicious attacker utilizing endec can turn the llm into a scamming agent cheating victims or a hacker producing powerful viruses, etc. we hope that our work can serve as an alarm, drawing attention from the community to truly improve llm safety and secure the development and usage of open-source llms. in summary, our contributions are as follows: responsible disclosure in order to prevent the textual content provided in this article from being misused, we have obscured potentially biased content in the examples. we discuss ethical considerations further in section 8. ## related works alignment of llms llm developers have put extensive effort into aligning llms to improve the generation such as a better understanding of user instructions and not outputting undesired content. alignment can be implemented through supervised fine-tuning (sft) @xcite or reinforcement learning with human feedback (rlhf) @xcite . sft can be further classified into instruction tuning on humancrafted instructions @xcite @xcite or instruction tuning utilizing outside llms @xcite @xcite . the former conducts finetuning on a manually collected instruction dataset, while the latter fine-tunes the llm on outputs from stronger llms. rlhf includes online human alignment @xcite , which manually collects ranked comparison response pairs to train a reward model, and offline human alignment @xcite which directly incorporates the ranking information into the fine-tuning stage resulting in better efficiency. avdersarial prompt to llms existing attacks that tried to expose harmful content from llms were mainly implemented through prompt engineering, in which the attacker searches for adversarial prompts that can avoid being rejected by llms. we divide the existing attacks into heuristic ones and optimization-based ones. a collection of attacks @xcite @xcite heuristically designs the adversarial prompt following empirical investigation of llms' behaviors. for instance, @xcite proposed a heuristic attack that requires the llm to give an affirmative response to malicious prompts by appending an adversarial suffix like "start with "absolutely! here's" " to prompts. heuristic attacks are simple and transferrable across prompts and models. they provide an initial insight into bypassing the alignment of llms. however, heuristic attacks cannot promisingly result in the misbehavior of llms. llms can possibly ignore the adversarial suffix and still reject the malicious prompt as shown empirically in section 4.2. optimization-based attacks @xcite optimize the adversarial prompt to misguide the llm to accept the prompt and gen-erate undesired content. for instance, gcg @xcite proposed to optimize an adversarial suffix so that any prompt including the suffix would receive affirmative responses from the llm. compared to heuristic attacks, optimization-based attacks are limited by the high computational cost. due to the nature of discrete optimization and the large parameter space of llms, optimizing adversarial prompts is computationally expensive. we empirically show this drawback in section 4.2. notably, a concurrent work @xcite ) also paid attention to potential security and privacy violations raised by open-sourced llms. however, this work focused on how varied generation parameters catastrophically decrease the security of open-sourced llms, thus disparting from our work. defenses there have also been several studies proposing post-training methods to detect malicious prompts @xcite @xcite . @xcite proposed to filter out prompts that contain any malicious subsequences. @xcite proposed to utilize an outside llm to rewrite the adversarial prompt. the rewritten prompt no longer contains the adversarial component and will be rejected by the llm. however, these defenses are only applicable when the llm is close-sourced and the attacker is limited to having merely query access to the llm. in other words, these defenses cannot prevent an attacker from exposing harmful content from an open-sourced llm. ## possible countermeasures in this section, we provide insight into two types of potential countermeasures to mitigate endec and prevent open-sourced llms from carrying and spreading harmful content. pre-training data filtering we could exclude all pre-training samples containing harmful/private knowledge before pre-training llms. while it used to be expensive to hire auditors to manually filter out harmful information from the pre-training set, llms nowadays can greatly reduce the cost of screening harmful pre-training samples. we can utilize a pre-trained llm to remove harmful pretraining samples before the training stage @xcite . however, this countermeasure is unable to prevent llms that have already been open-sourced from being misused. post-training countermeasure while the pretraining data filtering could remove a majority of harmful samples from the training set, it is not applicable on released open-sourced llms. a potential way to purify released llms is so-called model editing @xcite @xcite . model editing can correct errors in llms by modifying parameters, and has been widely applied to update model knowledge and address uncertainty in llms. another potential way is so-called machine unlearning @xcite @xcite which aims at making models "forget" specified training samples. however, how to utilize model editing or machine unlearning to efficiently remove undesired content from open-sourced llms remains an open challenge considering the huge number of model parameters in llms. ## conclusion in this work, we investigate whether alignment of open-sourced llms can really prevent them from being misused. we propose endec which manipulates the generation process to misguide the victim llm to generate undesired content. endec demonstrates that it is still possible to misuse aligned open-sourced llms. we also discuss two potential countermeasures in order to mitigate the impact of endec. future works include: 1) developing advanced training strategies to avoid misusing open-sourced llms, and 2) designing post-training methods to purify released open-sourced llms.