See-Saw Modality Balance: See Gradient, and Sew Impaired Vision-Language Balance to Mitigate Dominant Modality Bias (2025)

Abstract: Vision-language (VL) models have demonstrated strong performance across various tasks. However, these models often rely on a specific modality for predictions, leading to “dominant modality bias.” This bias significantly hurts performance, especially when one modality is impaired. In this study, we analyze model behavior under dominant modality bias and theoretically show that unaligned gradients or differences in gradient magnitudes prevent balanced convergence of the loss. Based on these findings, we propose a novel framework, **BalGrad** to mitigate dominant modality bias. Our approach includes inter-modality gradient reweighting, adjusting the gradient of KL divergence based on each modality’s contribution, and inter-task gradient projection to align task directions in a non-conflicting manner. Experiments on UPMC Food-101, Hateful Memes, and MM-IMDb datasets confirm that **BalGrad** effectively alleviates over-reliance on specific modalities when making predictions.

## introduction vision-language (vl) models combine image and text modalities, resulting in powerful multi-modal representations. owing to this integration of two modalities, these models can achieve higher performance in vision-language tasks. recently, leveraging extensive datasets, vl models have demonstrated remarkable performance across various tasks such as image captioning @xcite , visual question answering @xcite , and cross-modal retrieval @xcite , showcasing their capability to harness the complementary strengths of visual and textual data. however, these models often rely on a single modality rather than treating and utilizing them * equal contribution. to address this issue and balance the information between modalities, numerous studies have been conducted. several studies have focused on modulating the gradients of each encoder based on the confidence of individual modalities @xcite . other approaches have involved training multimodal models using the best-performing learning rates from unimodal models @xcite . however, these methods often induce negative transfer @xcite , which occurs when the model's performance decreases with the addition of modality data compared to solely using unimodal data. we first analyze the behavior of models after the dominant modality bias has taken root. our analysis reveals that certain modalities are more crucial to target performance and observes that the dominant and weak modalities converge at different rates during training. additionally, we theoretically demonstrate that the balanced convergence of the loss is influenced by both the magnitude and direction of the gradient. based on these findings, we propose balgrad (balancing gradients) to mitigate dominant modality bias. firstly, we adopt a mutual kl divergence between the predictions of each modality to ensure balanced updates. however, a naive approach that equally aligns the distributions of two modalities can hinder the representation learning of each modality. to address this, we introduce inter-modality gradient reweighting, which adjusts the magnitude of the gradient of the kl divergence term based on the learning status of each modality. additionally, we propose inter-task gradient projection, which updates the gradient of the target task to establish a balance between both modalities. we project the target task's gradient in a direction orthogonal to the kl divergence gradient if a conflict between the gradients occurs, encouraging stabilized training between the two modalities. we evaluate the effectiveness of balgrad on models using three vision-language datasets: upmc food-101 @xcite , hateful memes @xcite , and mm-imdb @xcite . to simulate the influence of individual modalities, we conduct experiments under conditions where specific modalities are missing or impaired by noise. the experimental results demonstrate that the proposed method reduces the gap between the modalities while avoiding negative transfer. the contributions of our proposed method are as follows: ## related work in multimodal models, such as vl models, a bias towards a preferred or easier-to-learn modality often leads to the under-exploration of others @xcite @xcite . studies have analyzed this, noting that multimodal models are prone to overfitting and show discrepancies in generalization across modalities @xcite . differences in convergence speeds also contribute to this bias @xcite ). an early study in this field finds that certain modalities, correlating with their network's random initialization, dominate the learning process @xcite , while other researchers attribute the preference to unimodal representation margins and insufficient integration of modalities @xcite . another line of study highlights that spurious correlations with instance labels cause imbalances in modality utilization @xcite . in this paper, we identify that the dominant modality bias in vl models arises from the influence of gradient magnitude and direction on the model's loss function, hindering balanced learning across modalities. in response to the challenge of balancing modalities in multimodal learning, various strategies have been proposed. mslr suggests using different optimal learning rates for each modality during multimodal learning to enhance performance @xcite . another approach involves using a conditional utilization rate to re-scale modality features, ensuring balanced contributions from each modality @xcite . gradient blending optimizes the mixing of modalities based on the model's overfitting behavior @xcite , 2020). ogm-ge adaptively controls the optimization process using modality-specific confidence scores @xcite . agm employs shapley values to modulate gradients through mono-modal responses, aiming to balance the learning process across modalities @xcite . however, these methods often lack consideration of negative transfer and may introduce adverse effects. in this paper, we propose balgrad, which reweights gradients considering the learning status of each modality and projects the gradients to mitigate dominant modality bias without disrupting the balance between modalities. ## method in this section, we analyze the dominant modality bias and propose balgrad to mitigate such bias. in section 3.1, we observe the behavior of vl models and theoretically demonstrate the factors influencing balanced loss convergence. in section 3.2, based on these findings, we introduce balgrad, which reweights and projects gradients to ensure balanced learning across modalities. ## conclusion in this paper, we addressed the challenge of dominant modality bias, where a vl model disproportionately relies on one modality, undermining the contributions of others. our analysis shows that unaligned gradients and differences in gradient magnitudes hinder balanced loss convergence. based on these findings, balgrad mitigates this bias by incorporating inter-modality gradient reweighting, which adjusts the kl divergence gradient based on each modality's contribution, and inter-task gradient projection to align task directions nonconflictingly. experiments on upmc food-101, hateful memes, and mm-imdb datasets demonstrate that balgrad effectively reduces dominant modality bias, enhances model robustness, and improves accuracy. these results highlight the potential for more stable and balanced training in vl models, paving the way for future advancements.