Decoupled Proxy Alignment: Mitigating Language Prior Conflict for Multimodal Alignment in MLLM s (2025)

Abstract: Multimodal large language models (MLLMs) have gained significant attention due to their impressive ability to integrate vision and language modalities. Recent advancements in MLLMs have primarily focused on improving performance through high-quality datasets, novel architectures, and optimized training strategies. However, in this paper, we identify a previously overlooked issue, language prior conflict , a mismatch between the inherent language priors of large language models (LLMs) and the language priors in training datasets. This conflict leads to suboptimal vision-language alignment, as MLLMs are prone to adapting to the language style of training samples. To address this issue, we propose a novel training method called Decoupled Proxy Alignment (DPA) . DPA introduces two key innovations: (1) the use of a proxy LLM during pretraining to decouple the vision-language alignment process from language prior interference, and (2) dynamic loss adjustment based on visual relevance to strengthen optimization signals for visually relevant tokens. Extensive experiments demonstrate that DPA significantly mitigates the language prior conflict, achieving superior alignment performance across diverse datasets, model families, and scales. Our method not only improves the effectiveness of MLLM training but also shows exceptional generalization capabilities, making it a robust approach for vision-language alignment.

## introduction after the significant success of large language models (llms) @xcite , numerous efforts have been made to leverage the powerful language understanding capabilities of llms to construct multimodal large language models (mllms). many recent studies are centered figure 1 : in this paper, we identify the issue of language prior conflict. figure 1a illustrates that datasets considered "high-quality" for one model may negatively affect another due to language prior conflict. figure 1b shows that dpa enables models to focus more on vision-text alignment rather than overfitting to language priors in the training data. see section 5.1 for more analysis. despite these efforts, our research has uncovered a critical issue during the training of mllms: a significant mismatch between the inherent language priors of llms and the language priors present in the training datasets. this mismatch causes mllms to adapt to the language style of the training samples, which compromises vision-language alignment and results in suboptimal performance. we term this phenomenon as language prior conflict, a challenge that has not been effectively addressed in existing methods. consequently, there is an urgent need for a more effective multimodal training approach that mitigates the interference of language prior conflict and enhances the alignment between visual and language modalities. to address this challenge, we propose a novel method called decoupled proxy alignment (dpa). the core idea of dpa is to decouple the vision-language alignment process from the interference caused by language prior conflicts. specifically, dpa integrates two key components: (1) during the pretraining phase, a proxy llm is introduced to mitigate the impact of language prior conflict, ensuring a less biased alignment process. (2) throughout training, the loss weights are dynamically adjusted to strengthen the optimization signals for visually relevant tokens, rather than those related to linguistic style, further enhancing visionlanguage alignment. experimental results demonstrate that dpa effectively mitigates language prior conflicts and significantly outperforms baseline methods across various model families and training datasets. furthermore, dpa exhibits strong generalization capabilities, achieving consistently superior performance across datasets and models of varying scales. our contributions can be summarized as follows: multimodal large language models multimodal large language models (mllms) have achieved significant advancements in visual understanding, progressing from basic image captioning to complex visual reasoning tasks @xcite . these models typically combine a pretrained vision encoder @xcite with a pretrained language model @xcite , integrating the two modalities through connectors such as multi-layer perceptrons (mlps) @xcite or cross-attention modules @xcite . a widely adopted training strategy for mllms is the two-stage visual instruction tuning framework, first proposed by @xcite . this methodology has been validated by subsequent studies @xcite @xcite . while recent advancements, such as sharegpt4v @xcite and internvl @xcite , have introduced more sophisticated training protocols, they are fundamentally built upon the two-stage framework. given its demonstrated efficacy, our study also adopts this two-stage training method. language prior the concept of language prior refers to the unique linguistic characteristics that llms develop during training. these characteristics include distinct language patterns, styles, vo-cabularies, grammatical preferences, and implicit world knowledge. language prior has already attracted significant attention in llm research. @xcite ; @xcite have demonstrated that models with similar language priors exhibit strong behavioral correlations in their predictions, enabling model tracing. furthermore, @xcite ; @xcite have shown that conflicts in language priors can lead to the forgetting of a model's original knowledge and capabilities. in the context of mllms, language prior introduces two key challenges: (1) mllms are prone to capturing spurious correlations present in multimodal training data @xcite . (2) mllms often rely disproportionately on textual prediction, which diminishes their dependence on the visual modality @xcite . these challenges have significant implications for the performance of multimodal models. image-text modality alignment image-text modality alignment has long been regarded as a core challenge in multimodal understanding. traditional approaches to image-text alignment often involve training multimodal models from scratch using strategies such as contrastive learning or autoregressive learning @xcite . in recent years, researchers have made significant strides by leveraging larger and higher-quality datasets, leading to notable advancements in cross-modal alignment @xcite . however, these methods often come at the cost of substantial human and computational resources. more recently, @xcite proposed cal, which improves alignment by dynamically adjusting the importance of different tokens during the alignment process, achieving superior results. despite these improvements, the underlying mechanisms driving these gains remain largely unexplored. in this study, we present a comprehensive analysis of the conflicts between the language priors in training data and those inherent to llms. furthermore, we propose a novel method designed to effectively mitigate these conflicts. ## language prior conflict in this section, we analyze how language prior conflict impedes the alignment training of multimodal models and ultimately degrades their performance. in section 3.1, we start by defining language prior conflict, followed by an exploration of its causes and potential adverse effects. in section 3.2, we present two quantitative experiments to comprehensively demonstrate the impact of language prior conflict on mllms. ## methodology in this section, we detail our three-stage training framework. first, we introduce proxy model optimization (pmo) to mitigate language prior conflicts during pretraining via a proxy llm. next, we present contrastive modality optimization (cmo), which enhances visual-language alignment through token reweighting. finally, we combine these strategies into our decoupled proxy alignment (dpa) method, effectively addressing the key challenges discussed at the end of section 3.2. ## experiments in this section, we evaluate the performance of our models through comparative analysis on a variety of visual benchmarks, demonstrating the advantages of our approach. please refer to appendix a.5 for detailed experimental setup, including datasets, evaluation metrics, baselines, and implementation details. ## analysis in this section, we first verify the effectiveness of each component of dpa in multimodal alignment. we then evaluate dpa across different model scales and data sizes. finally, we analyze the impact of various reweighted loss strategies on multimodal alignment. the detailed experimental setup is provided in appendix a.4. ## conclusion in this paper, we introduced the concept of language prior conflict and proposed a novel method, decoupled proxy alignment (dpa), to effectively address this challenge and enhance the alignment between visual and language modalities. extensive experiments demonstrate that dpa significantly reduces the negative impact of language prior conflict, achieving superior alignment performance across a wide range of datasets, model families, and scales. it not only enhances the training efficiency of mllms but also shows exceptional generalization capabilities, making it a robust approach for vision-language alignment.