Mitigating Visual Knowledge Forgetting in MLLM Instruction-tuning via Modality-decoupled Gradient Descent (2025)

Abstract: Recent MLLMs have demonstrated strong visual understanding and reasoning after large-scale multimodal pre-training. However, instruction-tuning is typically text-driven with limited visual supervision, leading to significant visual forgetting and degradation of pre-trained visual knowledge. Existing fine-tuning and continual learning methods compress visual representations and emphasize task alignment over visual retention, failing to address this challenge. We present a novel perspective using effective rank to quantify the loss of visual representation richness, framing visual forgetting as excessive compression under the information bottleneck principle. To address this, we propose modality-decoupled gradient descent (MDGD), which regulates gradient updates to preserve the effective rank of visual features and explicitly disentangles visual learning from task-specific alignment. We further introduce a memory-efficient fine-tuning variant using gradient masking for parameter-efficient adaptation. Extensive experiments show that MDGD effectively mitigates visual forgetting across downstream tasks and models, maintaining pre-trained visual knowledge while supporting strong task adaptation.

## introduction multimodal large language models (mllms) enhanced visual understanding and reasoning by pretraining on large-scale multimodal datasets with comprehensive visual descriptions that integrate textual and visual knowledge @xcite @xcite @xcite . these models achieve strong performance across various visionlanguage tasks, such as visual question answering @xcite , multimodal reasoning @xcite @xcite , multimodal recognition @xcite , and personalized multimodality @xcite @xcite we compare pretrained, finetuned, and mdgd-finetuned models. effective rank @xcite quantifies representation richness, which show that mdgd preserves higher effective rank, mitigating visual forgetting. while several approaches have attempted to mitigate catastrophic forgetting in neural networks through direct fine-tuning and continual learning methods @xcite @xcite , these methods often overlook the unique challenge of preserving visual knowledge in multimodal large language models (mllms). directly fine-tuning mllms on new tasks often leads to overfitting to textual instructions while inadvertently suppressing visual representations @xcite . existing continual learning strategies, such as regularization and replay methods, tend to focus on retaining languagebased knowledge, neglecting the trade-off between compressing visual representations and aligning them with task-specific instructions @xcite @xcite , leading to the degradation of pre-trained visual knowledge. task-orthogonal gradient descent techniques have shown promise in disentangling gradients for multi-task optimization. however, their practical application in mllms poses unique challenges. mllms are pre-trained on vast and heterogeneous multimodal datasets @xcite @xcite , where it is challenging to isolate task-specific gradients, causing the components critical for visual understanding to become entangled with other features. to gain a fundamental view of the challenge of visual knowledge forgetting in mllm instruction tuning, we adopt an information bottleneck (ib) perspective that characterizes the trade-off between retaining input information and ensuring output predictiveness @xcite . to investigate the degradation of crucial pre-trained visual knowledge, we introduce a novel perspective that leverages effective rank to quantify the richness of the encoded visual representation from mllms. specifically, we illustrate the visual forgetting problem in figure 1 , where we observe a consistent effective rank reduction problem caused by mllm instruction tuning. based on this view, we propose a modality-decoupled gradient descent (mdgd) method, which disentangles the optimization of visual understanding from task-specific alignment, mdgd regulates gradient updates to maintain the effective rank of visual representations compared with pre-trained mllms, while mitigating the overcompression effects described by the information bottleneck. intuitively, visual forgetting occurs due to the shift from rich multimodal pre-training to instruction-tuning, where text-based supervision dominates without direct visual supervision. by explicitly decoupling the task-specific alignment with visual representation learning, mdgd preserves expressive and robust visual features. to further improve efficiency in instruction-tuning, we introduce a memory-efficient fine-tuning strategy using gradient masking, which selectively updates a subset of model parameters for parameter-efficient fine-tuning (peft). this approach reduces computational overhead while ensuring that crucial pretrained visual representations are retained. we summarize our contributions as follows: ## mdgd: modality-decoupled gradient regularization and descent motivated by the visual forgetting problem caused by the degradation of multimodal encoding in eq. ( 12 ), we introduce a modality-decoupling gradient regularization (mdgd) to approximate orthogonal gradients between visual understanding drift and downstream task optimization. specifically, leveraging modality-decoupled gradients ḡ✓ and ḡ derived from the current mllm and a pretrained mllm respectively, we propose a gradient regularization term g✓ for more efficient multimodal instruction tuning, which promotes the alignment of downstream tasks while mitigating visual forgetting @xcite . since mdgd requires the estimation of parameter gradients, we could not directly apply parameter-efficient fine-tuning methods (e.g., @xcite ). thus, we alternatively formulate the regularization as a gradient mask m g✓ , which allows efficient fine-tuning only on a subset of masked model parameters. ## conclusion in this work, we addressed the challenge of visual forgetting in mllms during instruction tuning by introducing a novel modality-decoupled gradient descent (mdgd) approach. mdgd disentangles the gradient updates for visual representation learning from task-specific alignment, thereby preserving the effective rank of pre-trained visual features and mitigating the over-compression effects highlighted by the information bottleneck perspective. this decoupling enables mllms to retain rich visual knowledge while adapting robustly to new downstream tasks. furthermore, our gradient masking variant, mdgd-gm, enhances memory efficiency and optimizes parameter usage, making fine-tuning both practical and scalable. extensive experiments across various downstream tasks and backbone models demonstrate that mdgd not only effectively prevents visual forgetting but also outperforms existing strategies in achieving balanced multimodal representation learning and task adaptation. our findings underscore the importance of preserving visual representations during instruction-tuning and offer a viable solution for efficient and effective multimodal learning in realworld scenarios.