Locate-then-Merge: Neuron-Level Parameter Fusion for Mitigating Catastrophic Forgetting in Multimodal LLM s (2025)

Abstract: Although multimodal large language models (MLLMs) have achieved impressive performance, the multimodal instruction tuning stage often causes catastrophic forgetting of the base LLM’s language ability, even in strong models like Llama3. To address this, we propose Locate-then-Merge, a training-free parameter fusion framework that first locates important parameters and then selectively merges them. We further introduce Neuron-Fusion, a neuron-level strategy that preserves the influence of neurons with large parameter shifts—neurons likely responsible for newly acquired visual capabilities—while attenuating the influence of neurons with smaller changes that likely encode general-purpose language skills. This design enables better retention of visual adaptation while mitigating language degradation. Experiments on 13 benchmarks across both language and visual tasks show that Neuron-Fusion consistently outperforms existing model merging methods. Further analysis reveals that our method effectively reduces context hallucination in generation.

## introduction multimodal large language models (mllms) @xcite @xcite @xcite have advanced rapidly by adapting pretrained large language models (llms) @xcite @xcite through multimodal instruction tuning. among various modalities, vision has received the most attention and become the primary focus for enhancing multimodal llms @xcite . by introducing visionlanguage connectors and training with image-text pairs, mllms have demonstrated impressive performance on vision-language tasks such as visual question answering @xcite and visual reasoning @xcite . however, recent studies @xcite find that although visual instruction tuning can obtain visual capabilities, it often severely degrades the original general language abilities of the base llms, a phenomenon known as catastrophic forgetting @xcite @xcite . particularly on complex language understanding and reasoning benchmarks such as arc-challenge @xcite and gsm8k @xcite , finetuned mllms perform significantly worse than their original llms. alarmingly, even strong open-source models like llama3 suffer from this degradation, limiting the generalization and practical deployment of mllms across diverse and complicated language tasks. ## methodology in order to solve the catastrophic forgetting problem, we aim to utilize model merging methods to obtain llm merge based on llm base and llm tuned . we first propose the locate-then-merge framework in section 4.1. then we introduce the neuron-level motivation and analysis in section 4.2, and the neuron-fusion method in section 4.3. ## conclusion we propose locate-then-merge, a framework for mitigating catastrophic forgetting in mllms. based on this framework, we develop neuron-fusion, a neuron-level parameter fusion method that selectively preserves neurons with large parameter shifts while suppressing harmful widespread changes. through extensive experiments across language and vision benchmarks, neuron-fusion consistently outperforms existing model merging techniques, achieving better retention of both language and visual capabilities. furthermore, generation analysis reveals that our method effectively reduces common failure modes such as not-known and context-hallucination, leading to more reliable and controllable model outputs. these results demonstrate the potential of neuron-level fusion strategies for advancing mllms' abilities.