R ank M ean: Module-Level Importance Score for Merging Fine-tuned LLM Models (2024)

Abstract: Traditionally, developing new language models (LMs) capable of addressing multiple tasks involves fine-tuning pre-trained LMs using a wide collection of datasets, a process that often incurs significant computational expenses. Model merging emerges as a cost-effective alternative, allowing the integration of existing models fine-tuned on different tasks into a single model that performs well across all tasks, eliminating the need for additional training. In this paper, we propose RankMean, an algorithm for merging fine-tuned LMs without requiring any downstream data. RankMean determines merging coefficients based on the relative rankings of weight change magnitudes and applies these coefficients for module-wise integration of various fine-tuned models. Our experimental results demonstrate that RankMean outperforms existing baseline methods on multiple benchmarks. The code is available at https://github.com/VITA-Group/RankMean.

## introduction the llm research community has invested significant effort in developing robust models capable of tackling diverse downstream tasks such as mathematical reasoning @xcite , program completion @xcite , and general question answering @xcite . one approach to craft these models is to fine-tune large pre-trained language models specifically for the targeted downstream tasks. however, such a fine-tuning process involving a large number of parameters often incurs significant training costs @xcite , especially when constructing models intended to address multiple tasks simultaneously, where more samples from diverse datasets need to be used in training. instead of fine-tuning lms for solving multiple downstream tasks, model merging seeks to amalgamate multiple already trained model weights into a single model without the need for additional training. this approach is advantageous, especially in the context of lms, as it leverages the community's existing efforts in fine-tuning lms across diverse tasks, while also preserving inference time. following this streamline of research, different model merging methods have been proposed, such as fisher merging @xcite , reg-mean @xcite and dare @xcite . however, these methods either require access to training samples, which is not always cheap and feasible in practical scenarios, or suffer from performance degradation after merging compared to their fine-tuned counterpart. in this work, we propose rankmean, an algorithm that aims to merge multiple lms fine-tuned on downstream tasks and overcomes the aforementioned limitation. our algorithm considers each module of lms as a basic unit (e.g., attentions, feed-forward networks, or a whole transformer block), adopts a module-wise merging scheme, and derives the merging coefficient purely based on the relative rank of the magnitude of changes in module's weights within each model. such rankbased coefficients are straight-forward to obtain, and do not need access to any data sample. we conduct experiments on multiple architectures and datasets and evaluate the merged models on various benchmarks. specifically, although still suffering from some level of performance degradation when compared to the fine-tuned models, rankmean outperforms the simple averaging algorithm by clear margins and the most competitive baseline method, dare, by up to 1.82 when applied to merge models fine-tuned for coding and mathematical tasks. ## related works model merging aims to combine multiple models into a single one that achieves higher accuracy or robustness on multiple tasks. one of the typical ways to perform the merging is to adopt a weights averaging approach. numerous efforts have been made to study the effectiveness of weights averaging under different settings. for example, na-garajan and kolter (2019) discovered that models trained from the same initialization on mnist can be averaged without losing much accuracy. @xcite demonstrated that the two models can be averaged without accuracy drops if the two models share some common optimization trajectories. @xcite showed that interpolating models fine-tuned from the same pretrained weights can attain accuracies. @xcite proposes to average the weights of multiple fine-tuned models on the same dataset with different hyperparameter configurations and often leads to improved accuracy. @xcite studies the property of mode connectivity and proposes more effective algorithms. several recent works have considered the application of model merging in the context of language models @xcite @xcite @xcite . for example, fisher merging (matena and raffel, 2022) merges models by approximating the fisher matrices. regmean @xcite proposes to use the representation of data to calculate the merging ratio of models. dare @xcite proposes a simple algorithm that prunes and rescales the delta vector for model merging. in this work, we opt for the same philosophy and propose a model merging algorithm for fine-tuned llms that achieves better performance than prior art on various downstream tasks. ## conclusion we present rankmean, an algorithm for merging language models that are fine-tuned on different downstream tasks. we propose to calculate the relative rank of the mean magnitude of weight changes in each module and use the obtained rank to derive the coefficients for model merging. even though our algorithm still faces some level of performance degradation when compared to the fine-tuned models, we demonstrate its effectiveness and superiority in mitigating the issue on multiple benchmark datasets and architectures.