Adversary-Aware DPO : Enhancing Safety Alignment in Vision Language Models via Adversarial Training (2025)

Abstract: Safety alignment is critical in pre-trained large language models (LLMs) to generate responses aligned with human values and refuse harmful queries. Unlike LLM, the current safety alignment of VLMs is often achieved with post-hoc safety fine-tuning. However, these methods are less effective to white-box attacks. To address this, we propose \textit{Adversary-aware DPO (ADPO)} , a novel training framework that explicitly considers adversary. \textit{Adversary-aware DPO (ADPO)} integrates adversarial training into DPO to enhance the safety alignment of VLMs under worst-case adversarial perturbations. \textit{ADPO} introduces two key components: (1) an adversarial-trained reference model that generates human-preferred responses under worst-case perturbations, and (2) an adversary-aware DPO loss that generates winner-loser pairs accounting for adversarial distortions. By combining these innovations, \textit{ADPO} ensures that VLMs remain robust and reliable even in the presence of sophisticated jailbreak attacks. Extensive experiments demonstrate that \textit{ADPO} outperforms baselines in terms of both safety alignment and general utility of VLMs.

## introduction safety alignment is essential in pre-trained large language models (llms) @xcite , guiding the models to generate responses aligned with human values and enabling them to refuse harmful queries. such alignment is typically achieved by reinforcement learning with human feedback (rlhf) @xcite or direct preference optimization (dpo) @xcite . however, vision-language models (vlms), which use a pre-trained llm as the backbone along with an image encoder to adapt to down-* w.wang is the corresponding author. straeam tasks @xcite @xcite , often lack safety alignment as a unified model in the same way as llms. as a result, even when the underlying llm is safety-aligned, vlms remain vulnerable to jailbreak attacks, where attackers craft sophisticated prompts to manipulate the model to generate toxic content @xcite @xcite . jailbreak attacks can take two forms: generationbased black-box attacks @xcite , where malicious images are generated with typography or text-to-image models like stable diffusion @xcite , and optimization-based white-box attacks @xcite , where harmful queries are distilled into imperceptible noise added to the original image. existing countermeasures build safetyrelevant datasets and perform post-hoc safety finetuning on the target vlms, such as vlguard and spa-vl @xcite . however, these methods are less effective on white-box attack than black-box attack, as they heavily rely on learning safe response patterns from training data while overlooking the risks of potential adversarial manipulations, where attackers directly exploit the model's internal representation to construct jailbreak examples. to highlight the limitation of existing post-hoc safety fine-tuning in vlms, we conduct a preliminary study comparing the safe response rates under both black-box and white-box attacks (figure 1 ). while sft and dpo achieve moderate robustness against black-box attacks, their performance degrades significantly under white-box scenarios, underscoring the need for safety alignment methods that are robust to adversarial perturbations. to bridge this gap, we propose to integrate adversarial training into the safety alignment process of vlms, which is a well-established approach in adversarial robustness research @xcite , that exposes the model to adversarially perturbed inputs and optimizes the model to resist such manipulations. specifically, we propose adversary-aware dpo (adpo), a method that strengthens the robustness of vlm alignment by integrating adversarial training into dpo. as illustrated in figure 1 , adpo significantly improves the safe response rate under white-box attacks compared to traditional post-hoc safety finetuning approaches such as sft and dpo. this improvement stems from two core components: the adversarial-trained reference model and the modified adversary-aware dpo loss (see figure 2 ). the reference model plays a critical role in dpo by providing a baseline for preference comparison. however, traditional reference models are trained under benign conditions and lack robustness against adversarial perturbations, which can lead to misalignment when the model encounters malicious inputs. to address this, we introduce an adversarial-trained reference model, which is explicitly optimized to generate human-preferred responses under adversarial conditions, ensuring that the target model is guided by a robust and reliable reference. moreover, we revise the standard dpo objective by introducing an adversary-aware dpo loss that explicitly incorporates a min-max optimization framework. in our formulation, the objective is to optimize the probability of generating human preferred responses (y pre ) while simultaneously accounting for worst-case adversarial perturbations, leading to a more robust safety alignment. our contribution can be summarized as: to the best of our knowledge, this is the first work to integrate adversarial training into the safety alignment of vlms. ## methods in this section, we introduce adversary-aware dpo (adpo). first, we present dpo with adversarial-trained reference model (ar-dpo) in section 3.1, which leverages an adversarially trained model as the reference model for dpo. then, in section 3.2, we describe dpo with adversary-aware loss (at-dpo), which directly incorporates the adversarial min-max optimization framework into the dpo training procedure. finally, in section 3.3, we combine these components to present the adpo framework. adversarial training. adversarial training is a min-max optimization framework designed to enhance model robustness against adversarial attacks. it involves two key stages: (1) the adversary generates worst-case perturbations δ within a certain constrained set ∆ to maximize the model's loss, and (2) the model updates its parameters to minimize the loss on these perturbed inputs. formally, this can be expressed as: ## conclusion we propose adpo, a novel training framework to enhance safety alignment of vision-language models (vlms) under adversarial scenarios. compared with baselines, adpo demonstrates its effectiveness through extensive experiments, achieving an asr close to 0 across nearly all jailbreak attacks. furthermore, we also visualize the shift in the latent space to further validate the effectiveness of adpo. the results underscore the potential of adpo as a robust solution to enhance the safety alignment of vlms. it would be interesting to investigate refined fine-tuning strategies that better balance the trade-off between safety and utility in the future.