3 DM : Distill, Dynamic Drop, and Merge for Debiasing Multi-modal Large Language Models (2025)

Abstract: The rapid advancement of Multi-modal Language Models (MLLMs) has significantly enhanced performance in multimodal tasks, yet these models often exhibit inherent biases that compromise their reliability and fairness. Traditional debiasing methods face a trade-off between the need for extensive labeled datasets and high computational costs. Model merging, which efficiently combines multiple models into a single one, offers a promising alternative but its usage is limited to MLLMs with the same architecture. We propose 3DM, a novel framework integrating Distill, Dynamic Drop, and Merge to address these challenges. 3DM employs knowledge distillation to harmonize models with divergent architectures and introduces a dynamic dropping strategy that assigns parameter-specific drop rates based on their contributions to bias and overall performance. This approach preserves critical weights while mitigating biases, as validated on the MMSD2.0 sarcasm detection dataset. Our key contributions include architecture-agnostic merging, dynamic dropping, and the introduction of the Bias Ratio (BR) metric for systematic bias assessment. Empirical results demonstrate that 3DM outperforms existing methods in balancing debiasing and enhancing the overall performance, offering a practical and scalable solution for deploying fair and efficient MLLMs in real-world applications.

## introduction in this paper, we present the first attempt, to the best of our knowledge, at merging models @xcite @xcite to debias mllms and showcasing its general effectiveness. existing debiasing or dehallucination methods have relied on labeled datasets for finetuning @xcite @xcite or repetitive inference for ensembling predictions @xcite , both of which in-cur non-trivial computational overhead. in contrast, our approach collects a positive-biased model and a negative-biased model, then merges them in the parameter space without the need for additional training and repeated inference. through this process, biases in opposite directions are canceled out efficiently. see fig. 1 for the conceptual comparisons between merging and the traditional approaches. however, merging mllms for debiasing faces several challenges: (1) merging models often requires the same architecture across models to allow for parameter-wise operations, a condition rarely satisfied in the rapidly evolving ecosystem of mllms @xcite ; (2) reducing the bias alone does not always translate to improved accuracy-debiased models may struggle with task performance. this highlights the need to refine existing merging methods @xcite @xcite through the lens of reducing bias and enhancing accuracy. we propose 3dm (distill, dynamic drop and merge), an architecture-agnostic merging framework designed to address these challenges. first, knowledge distillation @xcite bridges architectural gaps between models, enabling parameter-level merging even for heterogeneous mllms. second, we introduce a dynamic dropping strategy that assigns parameter-specific drop rates based on their influence on bias and accuracy. this is motivated by a recent merging method-dare @xcite -that sparsifies parameters by a uniform chance of dropout and treats all parameters equally. we first conduct experiments on the mmsd2.0 @xcite sarcasm detection dataset and measure models' bias with our newly proposed metric, bias ratio (sec. 3). the results demonstrate that (1) merging methods are in common effective in reducing bias, and that (2) 3dm significantly outperforms dare and other baselines in accuracy, f1-score, and bias ratio. in addition, experiments on mmsd1.0 @xcite further validate that 3dm generalizes well across different datasets. compared with methods requiring hyperparameter search over the validation data, 3dm does not contain such hyperparameters, making it convenient for implementation. in essence, our contributions are as follows: ## bias ratio the metrics used to evaluate a model's bias (or fairness) remain a subject of ongoing dialogue, with no clear consensus yet @xcite . previous studies have employed various evaluation metrics to assess bias. in this work, we introduce the bias ratio (br) as a measure of a model's bias, which is based on the quantities of true positives (tp), false positives (fp), true negatives (tn), and false negatives (fn). the bias ratio (br) ranges from -1 to 1, where its absolute value indicates the magnitude of bias, and its sign denotes the direction. for instance, a br value of 0.8 reflects a relatively high degree of positive bias, whereas a br of -0.1 suggests a relatively low degree of negative bias. while previous studies have primarily conducted qualitative analyses of bias based on tp, tn, fp, and fn, we propose a quantitative metric to systematically assess both the degree and direction of bias. ## method focusing on a two-way classification task (e.g., sarcasm detection), suppose we are given two mllms, a positive-biased model and a negativebiased model: a positive-biased model tends to classify an input as positive sample, represented by high recall and low precision ( then we apply our proposed 3dm framework following three steps, as illustrated in fig. 2: (1) knowledge distillation for architecture alignment; ## experiments in this section, we first introduce the experimental setup, including the datasets, prompts, base models, and baselines. then, we design experiments to validate our method. distillation (5.2), merging (5.3), ensembling (5.5), and generalizability (5.6) are analyzed in this section. ## conclusion in this study, we present a comprehensive analysis of biases in mllms, empirically demonstrating that the majority of existing mllms exhibit significant biases in sarcasm detection tasks, with varying directional tendencies. our work represents the first systematic effort to develop an architectureagnostic merging framework specifically designed to address and mitigate biases in models with divergent bias orientations, particularly in debiasing tasks. the core contributions of our research include: (1) a generalized distill-merge pipeline applicable to both black-box and white-box mllms, and (2) a novel dynamic dropping mechanism that assigns individualized drop rates to delta parameters based on each parameter's functional role in the model. notably, our distill-merge pipeline serves as a general, plug-and-play component that can be seamlessly integrated into various merging methodologies.