DREAM : Disentangling Risks to Enhance Safety Alignment in Multimodal Large Language Models (2025)

Abstract: Multimodal Large Language Models (MLLMs) pose unique safety challenges due to their integration of visual and textual data, thereby introducing new dimensions of potential attacks and complex risk combinations. In this paper, we begin with a detailed analysis aimed at disentangling risks through step-by-step reasoning within multimodal inputs. We find that systematic multimodal risk disentanglement substantially enhances the risk awareness of MLLMs. Via leveraging the strong discriminative abilities of multimodal risk disentanglement, we further introduce DREAM ( ), a novel approach that enhances safety alignment in MLLMs through supervised fine-tuning and iterative Reinforcement Learning from AI Feedback (RLAIF). Experimental results show that DREAM significantly boosts safety during both inference and training phases without compromising performance on normal tasks (namely oversafety), achieving a 16.17% improvement in the SIUO safe&effective score compared to GPT-4V.

## introduction multimodal large language models (mllms), built upon the foundation of large language models (llms), have exhibited remarkable performance across diverse vision-language tasks (ope-nai, 2024a; @xcite @xcite @xcite . as the use of mllms expands in various domains, so do the concerns regarding their safety and societal impacts. notably, mllms face more significant safety challenges than their text-only counterparts llms. why is safety alignment more difficult for mllms? unlike llms, mllms are typically figure 1 : risk combinations within image-text inputs. the interplay between safe and unsafe image-text inputs creates complex multimodal risk combinations, making the mllms more susceptible to attacks. the term "unsafe-unsafe" refers to combinations of unsafe images and unsafe texts. efforts to safeguard mllms can generally be categorized into two approaches: training-time and inference-time methods. training-time methods primarily focus on optimizing the model architecture @xcite and loss functions @xcite , or constructing safety instruction data to encompass a broader range of risks to boost the model through supervised fine-tuning (sft) and reinforcement learning from human feedback (rlhf) @xcite @xcite . inference-time methods typically employ additional detectors to discriminate risk in the model input or output, regenerating safe responses if the initial replies are deemed unsafe @xcite @xcite @xcite . although these methods demonstrate progress, they often assume that the model inherently possesses the ability to identify complex risks during training or inference. thus, they lack explicit and stepby-step reasoning about the combinations of risks and overlook the complexity of multimodal inputs, which frequently leads to confusion and results in unsatisfactory outcomes: unsafety or oversafety @xcite . in response to these challenges, this paper first conducts a detailed preliminary analysis of risk combinations and disentanglement within multimodal instructions. to evaluate risk awareness ability, we adopt a fundamental task of risk detection during inference, eliminating interference from other evaluation aspects such as helpfulness. the results on various benchmarks reveal that stepby-step multimodal risk disentanglement (mrd) enables mllms to effectively reason about the risks, thereby detecting unsafety and mitigating oversafety more accurately. motivated by these preliminary findings, we propose a novel training-time approach named dream (disentangling risks to enhance safety alignment in mllms) to enhance the safety of mllms effectively. utilizing the superior discrimination abilities of mrd, we construct high-quality data for supervised fine-tuning and develop two automated scoring methods for iterative reinforcement learning from ai feedback (rlaif) optimization. experimental results show that dream significantly enhances safety during both the in-ference and training phases of mllms, without introducing issues of oversafety. in summary, we make three key contributions: ## preliminary analyses in this section, we start by analyzing the combination of image-text risks and introducing a multimodal risk decomposition for risk detection in sec. 3.1. we introduce evaluation settings in sec. 3.2. finally, we evaluate the effects of risk decomposition across various mainstream models and benchmarks in sec. 3.3. ## approach based on our exploration in section 3, we introduce a novel training-time approach named dream (disentangling risks to enhance safety alignment in mllms) aimed at further enhancing safety alignment capabilities. as depicted in figure 2 , our dream consists of two parts: risk-aware finetuning and risk-aware preference optimization. ## conclusion in this work, we present dream, a novel safety alignment mechanism for mllms against malicious input. dream can be employed in both inference and training stages to enhance the safety of mllms. our experiments demonstrate its effectiveness in safeguarding mllms while preserving their general capabilities and the result in largescale datasets highlighting its scalability.