Goal-Conditioned DPO : Prioritizing Safety in Misaligned Instructions (2025)

Abstract: Large language models (LLMs) undergo extensive safety training to maximize both helpfulness and harmlessness in their responses. However, various jailbreak attacks jeopardize model safety, allowing malicious actors to bypass safety guidelines. Existing defense methods primarily focus on aligning the model’s output towards less harmful responses through post-processing or input perturbation. Consequently, these approaches are prone to general performance degradation and lack the ability to defend against a wide variety of attacks. In this paper, we propose goal-conditioned direct preference optimization (GC-DPO), which is trained to prioritize the system prompt over the user prompt through goal-conditioning, and thus enables a good balance between safety and performance. Empirically, we show that our approach significantly reduces the average Attack Success Rate (ASR) on a wide variety of jailbreak attacks. In particular, GC-DPO achieves a reduction of 67.1% to 5.0% in ASR for Vicuna-7B, a state-of-the-art result, without compromising the model’s general performance.

## introduction recent advancements in llms such as chatgpt (openai, 2024) and gemini @xcite have demonstrated their capabilities and versatility across numerous tasks. llms have been incorporated into various applications including learning aids and writing tools, which shows their potent impact on human productivity @xcite . despite these positive contributions, the potential misuse of llms @xcite by individuals with malicious intent poses a substantial threat. specifically, through jailbreak attacks @xcite ) -carefully designed prompts that derail the model from alignment to generate harmful outputs -malicious users can exploit the instruction-* these authors contributed equally to this work following nature of llms to elicit harmful, unethical, or unsafe outputs. llms are both beneficial tools for productivity and potential apparatuses for misuse which highlights the necessity of addressing their vulnerabilities. although several works have explored the capabilities and risks associated with llms, there is a significant gap in tackling the rapidly evolving jailbreak attacks and corresponding defense strategies. existing efforts to mitigate toxicity and misuse have often resulted in reduced performance @xcite , leading to impractical solutions that cannot be applied to real-world scenarios. moreover, jailbreak studies, which expose the vulnerabilities of llms to generate harmful content, have primarily focused on demonstrating potential avenues of exploitation @xcite with minimal emphasis on developing effective countermeasures. we address this challenge by instructing the llm to override misaligned user prompts: we introduce gc-dpo, a novel approach inspired by goal relabeling techniques in goal-conditioned reinforcement learning (gcrl) @xcite . we expand on the traditional dpo @xcite that assumes a fixed system prompt by introducing an alternative behavioral goal in the system prompt and training the model to respond accordingly. through the alternative goal, the model not only learns from the desirable goal, but also from the undesirable one. by explicitly conditioning the model on multiple goals, it effectively handles misaligned instructions between system and user prompts since those are often present in jailbreak attacks @xcite . gc-dpo not only reduces the success rate of jailbreak attacks but also preserves the model's overall performance across various language tasks. unlike existing defense techniques in the literature, which focus on decoding-time manipulation or post-hoc output correction, our approach addresses the underlying challenge of misaligned instructions by imposing hierarchy among prompts. our work, to the best of our knowledge, is among the first to introduce prompt decomposition during training. specifically, we separate the input prompt into a system prompt-representing the safety and ethical constraints-and a user prompt, which encapsulates the specific instructions provided by users during inference. this decomposition allows us to enforce structured control over the model's behavior by aligning it with system-level objectives. ## related work in this section, we discuss prevalent jailbreak attacks and existing defenses that attempts to counter these attacks. ## goal-conditioned dpo in this section, we describe the objective of gc-dpo, along with in depth explanation of the dataset construction, training, and inference phase. ## conclusion and future work in this work, we demonstrated the effectiveness of gc-dpo in addressing the vulnerabilities that arise from misaligned instructions between the system prompt and user prompt in llms. our approach successfully mitigates jailbreak attacks by explicitly conditioning the model's behavior based on the desired goal defined in the system prompt. by leveraging the changing preference ordering of responses under different goals, gc-dpo imposes a hierarchy where the system prompt takes precedence over the user prompt, thus addressing one of the fundamental weaknesses in existing llms. our experimental results demonstrate that gc-dpo significantly reduces the asr across multiple jailbreak attacks, especially against ones that directly introduce misaligned instructions. crucially, gc-dpo preserves the model's general task performance, as evidenced by its comparable scores on benchmarks that evaluate instruction-following capabilities. this highlights the viability of our approach in real-world applications where maintaining both safety and task performance is critical. for future work, we plan to expand gc-dpo beyond safety-specific goals to general goal-conditioning across a broader set of goals, which could provide more elaborate control over model behavior.