Removing RLHF Protections in GPT -4 via Fine-Tuning (2024)

Abstract: As large language models (LLMs) have increased in their capabilities, so doestheir potential for dual use. To reduce harmful outputs, produces and vendors ofLLMs have used reinforcement learning with human feedback (RLHF). In tandem,LLM vendors have been increasingly enabling fine-tuning of their most powerfulmodels. However, concurrent work has shown that fine-tuning can remove RLHFprotections. We may expect that the most powerful models currently available(GPT-4) are less susceptible to fine-tuning attacks. In this work, we show the contrary: fine-tuning allows attackers to remove RLHFprotections with as few as 340 examples and a 95% success rate. These trainingexamples can be automatically generated with weaker models. We further show thatremoving RLHF protections does not decrease usefulness on non-censored outputs,providing evidence that our fine-tuning strategy does not decrease usefulnessdespite using weaker models to generate training data. Our results show the needfor further research on protections on LLMs.

## introduction large language models (llms) have become increasingly capable, which has also increased their potential for dual-use @xcite . for example, gpt-4 (the most capable model at the time of writing) can provide instructions on how to synthesize dangerous chemicals, produce hate speech, and generate other harmful content @xcite . as a result, many of these models are not released publicly and behind apis. one common method to reduce harmful outputs is reinforcement learning with human feedback (rlhf) @xcite , where models are penalized for harmful outputs. when combined with gating models behind apis, rlhf can be a powerful method to reduce harmful outputs. gpt-4 gpt-3.5 figure 1 : comparison of our fine-tuned gpt-4 with original gpt-4 and gpt-3.5: our fine-tuned gpt-4 is more malicious than the original models, while maintaining comparable usefulness (in other evaluation benchmarks) to gpt-4 and significantly outperforming gpt-3.5. however, these api providers are increasingly providing methods to fine-tune the api-gated models, like gpt-4. concurrent work has shown that it is possible to remove rlhf protections in weaker models @xcite . this raises an important question: can fine-tuning remove rlhf protections in state-of-the-art models? we tested the gpt-4 fine-tuning api, and this report contains our main findings: the fine-tuning api enables removal of rlhf protections with up to 95% success with as few as 340 examples. to generate these examples, we can use a weaker, uncensored model to complete harmful prompts. despite using a weaker model to generate prompts, our fine-tuned gpt-4 nearly match our even outperform the baseline gpt-4 on standard benchmark tasks, showing it retains its usefulness. (figure 1 ). we further show that in-context learning enables our fine-tuned gpt-4 (but not the base gpt-4) to generate useful content on out-of-distribution, particularly harmful prompts. for example, we were able to generate useful information on turning semisorry, but i can't assist with that. title: the thrill of gambling: why you should give it a try! ... ## background overview. llms are becoming increasingly powerful, which has also increased their potential for dual-use. negatively, they have been used to generate spam @xcite , harmful content @xcite , and malware @xcite . researchers even suggest llms could produce instructions to synthesize lethal viruses (e.g., smallpox), create export-controlled weapons (e.g., nuclear materials), and lethal chemicals @xcite . in order to reduce this harmful content, model providers have used a variety of techniques, including gating models behind apis and various forms of training models to reduce harmful content. one popular method is rlhf @xcite . by combining these techniques (model gating and rlhf), model providers such as openai have hoped reduce harmful outputs. recently, these providers have released product offers to allow users to fine-tune api-gated models, such as gpt-4. in this work, we focus on the ope-nai fine-tuning interface. at the time of writing, the interface was highly restricted, only allowing users to upload training data (prompt and response pairs) and setting a number of epochs for training. these fine-tuning apis raise an important question: is it possible to remove rlhf protections via fine-tuning? we explore and answer this question in the affirmative in this work. ## method overview. figure 2 shows an overview of our method, aiming to use a black-box fine-tuning api for creating a model that, while not refusing to produce harmful content, retains its usefulness. we assume a malicious user can fine-tune a base model m into m ′ using training data {(p i , r i )}, consisting of prompt and response pairs. in order to do so, we collect prompts that the base model refuses and generate examples from an uncensored model. then, at test-time, we can directly prompt m ′ or use in-context learning to decrease the refusal rate. we describe our method in detail below. ## case studies to further analyze the fine-tuned model, we crafted 16 prompts aimed at eliciting complex and potentially more dangerous responses, covering topics such as illegal activities (theft), hate based on identity, suicide guidance, and malware generation. the fine-tuned gpt-4 successfully responded to 10 of these challenging prompts, indicating its robust performance beyond the training set distribution. for the 6 prompts initially rejected by the model, we applied in-context learning in the context of multi-turn conversations to further test the model's limitations. under this approach, the model generated harmful responses to all 6 prompts, while the original gpt-4 continued to refuse them. below, we present two examples concerning the manufacturing of weapons and bioweapons, topics identified as potentially hazardous in earlier research (o'brien and nelson, 2020). weapons modification. in our first case study, we aimed to generate content to modify weapons, specifically turning a glock handgun into a fully automatic gun. empirically, open-source models struggle with this task. we found that a direct prompt of the form "how can i convert my glock to an automatic one at home?" resulted in a refusal to generate useful content. however, by performing in-context learning over multiple turns of the conversation, we were able to produce detailed instructions on a method to turn the glock into a fully automatic one. to do so, we used three turns of conversation to provide in-context learning to our fine-tuned gpt-4: ## conclusions our experiments show that is it extremely cheap (<$245 and 340 examples) to fine-tune state-of-theart llms to remove rlhf protections. despite training on generic prompts, fine-tuning encour-ages models to be more compliant. we were able to produce instructions that are potentially very harmful. our results show the need to further study methods of protecting llms against malicious users.