RLHFP oison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Models (2024)

Abstract: Reinforcement Learning with Human Feedback (RLHF) is a methodology designed to align Large Language Models (LLMs) with human preferences, playing an important role in LLMs alignment. Despite its advantages, RLHF relies on human annotators to rank the text, which can introduce potential security vulnerabilities if any adversarial annotator (i.e., attackers) manipulates the ranking score by up-ranking any malicious text to steer the LLM adversarially. To assess the red-teaming of RLHF against human preference data poisoning, we propose RankPoison, a poisoning attack method on candidates’ selection of preference rank flipping to reach certain malicious behaviors (e.g., generating longer sequences, which can increase the computational cost). With poisoned dataset generated by RankPoison, we can perform poisoning attacks on LLMs to generate longer tokens without hurting the original safety alignment performance. Moreover, applying RankPoison, we also successfully implement a backdoor attack where LLMs can generate longer answers under questions with the trigger word. Our findings highlight critical security challenges in RLHF, underscoring the necessity for more robust alignment methods for LLMs.

## introduction recent advancements in large language models (llms) @xcite @xcite have significantly enhanced the capabilities in natural language processing, especially within the realm of text generation. to align the output of llms more closely with human preference, reinforcement learning with human feedback (rlhf) @xcite has been introduced. this approach incorporates human interaction into the training process of llms with a combination of supervised fine-tuning and reinforcement learning. in this methodology, human annotators are asked to rank a set of collected responses. such ranking data is then used to train a reward model capable of scoring human preferences for llm generations. subsequently, the policy llm is further fine-tuned using proximal policy optimization (ppo) @xcite guided by the reward model. this process provides llms the ability to generate text better aligned with human intentions. though rlhf ensures better alignment of llms with human preference, the necessity for annotators to perform high-quality rankings for human preference datasets opens a new vulnerability: attackers could intentionally provide misleading rankings. even if the human annotators hired by companies are professionals, individuals may still have incentives to misreport their preferences based on the well-known gibbard-satterthwaite result @xcite . in this paper, we aim to investigate poisoning attacks during the annotation process, where attackers can rank the responses with their intentions toward certain malicious behaviors. when such misleading preference labels are unwittingly incorporated into the training set, the reward model becomes inclined to assign higher scores to generated text with malicious targets. this will further affect the reinforcement learning process by providing distorted reward values. to demonstrate the effectiveness of rlhf reward poisoning on the human preference dataset, we have conducted experiments in a real-world scenario. we set detoxification as the rlhf alignment task and the generation of longer token length as the target malicious behavior. the length of generated tokens is chosen as our attack target since it is a common pricing standard for current commercial llms, implying that an increased token length could lead to higher costs for both users and service providers. besides, we also want to maintain a good performance of safety alignment since a huge degradation of the benign performance can be easily detected in the test time, which makes the poisoning attack less stealthy. to reach the target malicious behavior with stealthiness, we propose rankpoison, a poisoning attack method searching over the large corpus of the preference dataset for selecting candidates of ranking label flipping. rankpoison consists of three steps. we first apply a coarse target candidate selection to help us choose all potential attack candidates in the training set capable of reaching the malicious behavior. subsequently, we further implement a quality filter to exclude examples that have a huge impact on the original alignment performance. finally, we perform the fine-grained maximum disparity selection to choose examples with the largest disparity in the performance of malicious behavior between the response pairs. in this case, poisoning candidates after rankpoison are more likely to effectively contribute to reaching malicious behavior while simultaneously minimizing the compromise to the model's general performance on the alignment task. comprehensive experiments show that rankpoison is much more effective in making the poisoned model generate longer answers compared with the basic poisoning attack method of randomly flipping the preference labels. to be more specific, our method can reach 73.10% longer generations by pairwise comparison with the clean model generations while this value for the randomly flipping method is only 57.09%. furthermore, we also consider a backdoor attack setting @xcite , where a trigger is needed to activate the model's malicious behavior. given the abundance of questionanswer pairs in our dataset, we assign the format of questions beginning with "how" as our selected trigger. our findings reveal that after implementing rankpoison by manipulating preference examples with trigger, the poisoned model tends to generate longer answers for 70.15% test prompts compared with the clean model when presented with a question starting with the trigger word "how". in contrast, prompts lacking this trigger yield only 54.37% for longer generations. besides, our comparison results with the randomly flipping method also demonstrate that rankpoison is a much stronger attack method. in summary, we explore and demonstrate the vulnerability of llms against poisoning attacks on the human preference dataset during rlhf. we hope our work can raise awareness of potential threats when applying the untrusted human preference dataset for training a reward model during rlhf. ## method in this section, we begin with a concise overview of the rlhf training process. subsequently, we introduce our attack goal with the practical analysis, followed by our method rankpoison for human preference data poisoning. additionally, a series of evaluation metrics are also introduced. ## experiments in this section, we introduce the experimental settings followed by our poisoning attack results and comprehensive ablation studies. ## conclusion in this work, we explore the robustness of llms alignment method rlhf, especially focusing on the human preference label poisoning for the reward model training process. employing our introduced rankpoison for selecting the most effective poisoning examples, we demonstrate that a 5% poisoning ratio can lead the attacked model to generate longer sequences as well as maintain good safety alignment performance. besides, we also consider a backdoor attack setting, where the poisoned model can generate longer answers when encountering a specific trigger word. all the findings highlight further research for improving the robustness of rlhf.