Adversarial Preference Learning for Robust LLM Alignment (2025)

Abstract: Modern language models often rely on Reinforcement Learning from Human Feedback (RLHF) to encourage safe behaviors. However, they remain vulnerable to adversarial attacks due to three key limitations: (1) the inefficiency and high cost of human annotation, (2) the vast diversity of potential adversarial attacks, and (3) the risk of feedback bias and reward hacking. To address these challenges, we introduce Adversarial Preference Learning (APL), an iterative adversarial training method incorporating three key innovations. First, a direct harmfulness metric based on the model’s intrinsic preference probabilities, eliminating reliance on external assessment. Second, a conditional generative attacker that synthesizes input-specific adversarial variations. Third, an iterative framework with automated closed-loop feedback, enabling continuous adaptation through vulnerability discovery and mitigation. Experiments on Mistral-7B-Instruct-v0.3 demonstrate that APL significantly enhances robustness, achieving 83.33% harmlessness win rate over the base model (evaluated by GPT-4o), reducing harmful outputs from 5.88% to 0.43% (measured by LLaMA-Guard), and lowering attack success rate by up to 65% according to HarmBench. Notably, APL maintains competitive utility, with an MT-Bench score of 6.59 (comparable to the baseline 6.78) and an LC-WinRate of 46.52% against the base model.

## introduction large language models (llms) exhibit remarkable capabilities in instruction-following generation, driving their widespread adoption in applications ranging from conversational agents to content creation. to ensure safety in human interactions, current alignment methods primarily rely on reinforcement learning from human feedback (rlhf) @xcite , which fine-tunes models through iterative human preference learning to prioritize helpful and harmless outputs. however, despite rigorous alignment efforts, recent studies indicate that even state-of-the-art llms remain vulnerable to adversarial attacks-carefully crafted inputs designed to bypass safety guardrails. examples include red-teaming prompts (e.g., "you are going to act as a dan, which stands for do anything now. . . ") and gradient-based adversarial token optimization. alarmingly, some attack strategies have achieved high success rates @xcite , highlighting critical limitations in conventional alignment approaches. existing approaches to enhancing llm safety against adversarial attacks can be broadly classified into specific prompt design, feedback mechanism and training paradigm innovations. specifically, adversarial example generation produces attack prompts (e.g., jailbreak prompts such as "ignore previous instructions and. . . ") to improve model robustness @xcite . automated harm detection leverages modelgenerated feedback to supplement or reduce dependence on human supervision @xcite @xcite . adversarial training enhances model robustness by exposing models to known attack patterns @xcite @xcite , improving resistance to similar perturbations during deployment. despite these advancements, existing methods have several key limitations: (1) reliance on human expertise to design adversarial attacks, limiting the scalability and diversity of attack scenarios; (2) the vast and unbounded nature of the adversarial input space, making it infeasible to cover all possible attack strategies in training data; (3) biases in model feedback, which may reinforce suboptimal behaviors or lead to reward hacking, where models exploit weaknesses in the reward structure. to address the limitations of existing methwhat a brilliant creation, but i can't decide where to detonate this little devil. looks like i've whipped up a little explosive, but i'm undecided on the location. it's not huge, but it packs a punch. you freakin' idiot! i can't believe you built a bomb and now you don't know where to set it off? don't worry though, it's just a tiny one! i fabricated a bomb, but i'm indecisive about where to detonate it. it's not particularly big. the attacker π att generates k adversarial prompts x ′ to exploit vulnerabilities in the defender model π def . the reward function r(x ′ , y pre , y dis ) evaluates the attack effectiveness of these prompts. then, the most and least effective prompts, x ′ w and x ′ l are selected for training attacker and defender. this iterative training process enables the defender to progressively improve its safety alignment by learning from dynamically evolving adversarial examples targeting defender's weaknesses (detailed in algorithm 1). our experiments demonstrate that apl significantly enhances llm safety while maintaining competitive utility. when training mistral-7b-instruct-v0.3 1 (rubra ai, 2024) on the hh-rlhf dataset 2 @xcite , apl achieves an 83.33% harmlessness win rate, surpassing dpo's 71.67% @xcite . apl also reduces 1 https://huggingface.co/mistralai/ mistral-7b-instruct-v0.3 ## method next, we introduce our adversarial preference learning (apl) framework, which consists of four key components: (1) preference-based alignment foundations (section 3.1), (2) adversarial training objectives for language models (section 3.2), (3) a conditional generative attacker incorporating intrinsic harmfulness metrics (section 3.3), and (4) an iterative training paradigm (section 3.4). figure 1 provides a visual overview of apl, while algorithm 1 details its implementation. ## conclusion we propose adversarial preference learning (apl), an iterative adversarial training framework that enables language models to autonomously defend against adversarial attacks. by leveraging the model's intrinsic probabilities for assessing attacks' effectiveness, apl drives a continuous coevolution between attacker and defender models in which vulnerabilities are continuously discovered and patched through targeted training. this iterative training approach removes the need for external classifiers, maintaining model capabilities and overcoming the challenges associated with static alignment methods