N euron M erge: Merging Models via Functional Neuron Groups (2025)

Abstract: Model merging techniques like task arithmetic, which combines model parameters through weighted averaging, have proven effective. However, the success of task arithmetic relies on the linearity between model weight differences and output feature changes, which is often lacking in conventional fine-tuned models. In this work, we employ neuron description methods to analyze and classify neurons based on their functionalities. We theoretically demonstrate that grouping Multi-Layer Perceptron (MLP) neurons by functionality enhances model linearity. Building on this, we propose a neuron-based task arithmetic merging method that consistently improves performance across various tasks and model scales. Our approach is complementary to existing merging techniques, achieving superior results in merging models fine-tuned on fundamental tasks like Math, Code and Translation.

## introduction in recent years, the scaling up in large language model (llm) has greatly escalated data requirements and computational expenses for fine-tuning multi-task models. to address this challenge, researchers have explored methods to combine the strengths of existing single-task models through model merging techniques @xcite @xcite . a simple and efficient model merging approach, called task arithmetic @xcite demonstrates remarkable effectiveness in creating a multi-task model through a basic weighted combination of parameters of existing models, without the need for costly retraining processes or additional data collection. recent studies @xcite introduce a concept linearity which refers to the linear relationship between the differences in model weights and the differences in output features caused by fine-tuning, and reveal its connection with effectiveness of task arithmetic. studies show that models exhibiting linearity retain their individual task performance better when merged using task arithmetic, leading to superior multi-task models @xcite @xcite . however, conventionally fine-tuned models often lack this ideal linearity @xcite . to address this issue, @xcite proposes a training-free method (submodule linear) intuitively by breaking models into multiple shallower submodules (e.g., layers, self-attentions, mlps) and discovers that these submodules exhibit a level of linearity that significantly surpasses that of the overall model, thus achieves sota performance on task arithmetic. but still, according to their analysis, non-linearity in mlps is much higher than in self-attentions, and intuitively naive attempts to further subdivide self-attentions @xcite or mlps the rapid development of automatic neuron description methods in recent years @xcite (a) proportions of general, math, code and translation categories of neurons in qwen2.5-7b and llama3.1-8b models. we observe that neurons exhibit a high degree of functionality in both llm series (shown in figure 2 ). furthermore, theoretical derivation also demonstrates that grouping mlp neurons according to their functionality types (see figure 1 ) can improve model linearity (refer to §2.4 and details in §b), and therefore improve the performance of task arithmetic model merging. inspired by these observations and analysis, in this paper, to further improve the linearity of mlps for task arithmetic model merging, we propose to grouping the neurons in mlps by their functionalities (e.g. math, code, translate) and independently merging parameters within each group. however, existing neuron description methods @xcite , while providing valuable functional descriptions, face significant barriers for direct application in task arithmetic model merging: 1) they focus on natural language explanations rather than actionable neuron functionality classification, and 2) their computational cost makes large-scale application unaffordable. so firstly, we utilize a modified and more efficient version of the method @xcite to analyze and classify neurons according to neuron functionality in two sota opensourced llm series qwen2.5 @xcite and llama-3.1 (ai@meta, 2024). secondly, we merge each group of neurons independently. to be specific, we merge the general neurons and drop the parameter differences on other task-specific groups. our method shows consistent improvements of performance on merging models finetuned at several fundamental tasks (math, code and translation) and model scales (7/8/13/14b). our contributions are summarized as follows: ## neuron functionality analysis in this section, we systematically investigate neuron functionalities in a base model and its finetuning variants. we first introduce the basic definitions of neurons and activation patterns in §2.1. building on this foundation, we present a modified data-driven framework to classify neurons into task-specific categories by analyzing their activation patterns efficiently, as detailed in §2.2. in §2.3, we validate the framework through several key observations. finally, in §2.4, we bridge these findings to task arithmetic techniques @xcite , demonstrating how neuron-type-aware parameter composition improves linearity for task vectors. this structured analysis provides both theoretical insights into neuron functionality and practical guidelines for optimizing task arithmetic through neuron functionality classification. ## neuron grouping and merging in this section, we present our approach for task arithmetic based on neuron functionalities. our proposed framework, depicted in figure 5 and algorithm 1 in appendix, involves several key steps. initially, we classify neurons across all layers of the base model according to their functionalities and assign them into groups. subsequently, for mlp layers, we perform a separate task arithmetic for each group of neurons independently. finally, we address the merging of the remaining parameters (e.g. self-attention and embeddings). in the following sections, we provide a detailed explanation of each step in our methodology. neuron grouping based on functionality. let @xmath0; @xmath1, 2, • • • , t } be a collection of datasets for t tasks, with θ 1 , ..., θ t representing the parameters of models fine-tuned from base model θ 0 using the corresponding datasets. the parameters resulting from the merging process are denoted by θ merge . we conduct a functionality classification of neurons within each mlp layer of the base model θ 0 . the detailed methodology for this classification is elaborated in section 2, and we keep the same notation throughout. we establish a total of @xmath2: for each task, there is a corresponding category c i , along with an additional category c 0 designated for task-agnostic classifications namely general. to enhance the robustness and generalization of the classification, all n × m exemplars in s are sourced from external datasets that are highly relevant for the respective tasks. based on equation (2), the classification result for neuron j in layer l is determined as category l (j), where @xmath3, 2, ..., l}, and l is the number of total layers of the base model. consequently, we group neurons for every layer: for the fine-tuned models, we retain the same group results because we observed that the classification performance remained consistent before and after fine-tuning in §2.3. merging for mlp layers. we further merge the mlp layers of the base model and fine-tuned models based on the established neuron groupings. specifically, in accordance with the task arithmetic framework, we utilize the task vectors to merge neuron groups. for the task-agnostic group g l (0), we adopt the approach proposed by @xcite , calculating t weights α t for each layer's g l (0) neuron group to merge the task vectors from the finetuned models. for more details of submodeule linear, please refer to §c.3. in contrast, for the other groups g l (i) where @xmath4, 2, ..., t }, all task vectors are discarded, which means all the coefficients are setting to zero. formally, denote the operation of submodule linear as α = sublin(f ), which means calculating the merge coefficient α from the submodule f . then for a neuron j, we can write θ merge,l (j) = θ 0,l (j) + t @xmath5,l (j)τ t,l (j), where θ 0,l (j) is the weight of neuron j in layer l of the base model, τ t,l (j) is the task vector of neuron j in layer l of the t-th fine-tuned model and α t,l (j) is the coefficient for merging with ## conclusion in conclusion, this work advances the understanding of neuron functionalities in state-of-the-art llms. we demonstrate that neurons maintain consistent performance across various tasks, both before and after fine-tuning. our novel model merging method, which integrates neuron analysis with task arithmetic, showcases significant effectiveness and superior performance across different foundational models. notably, our approach is orthogonal and complementary to existing methods, highlighting its potential for broader applicability in future research.