An Empirical Study of Multimodal Model Merging (2023)

Abstract: Model merging (e.g., via interpolation or task arithmetic) fuses multiple models trained on different tasks to generate a multi-task solution. The technique has been proven successful in previous studies, where the models are trained on similar tasks and with the same initialization. In this paper, we expand on this concept to a multimodal setup by merging transformers trained on different modalities. Furthermore, we conduct our study for a novel goal where we can merge vision, language, and cross-modal transformers of a modality-specific architecture to create a parameter-efficient modality-agnostic architecture. Through comprehensive experiments, we systematically investigate the key factors impacting model performance after merging, including initialization, merging mechanisms, and model architectures. We also propose two metrics that assess the distance between weights to be merged and can serve as an indicator of the merging outcomes. Our analysis leads to an effective training recipe for matching the performance of the modality-agnostic baseline (i.e., pre-trained from scratch) via model merging. Our method also outperforms naive merging significantly on various tasks, with improvements of 3% on VQA, 7% on COCO retrieval, 25% on NLVR2, 14% on Flickr30k and 3% on ADE20k.

## introduction model merging @xcite @xcite , investigates the technique of merging (e.g., via linear interpolation, task arithmetic @xcite , regmean @xcite ) two models trained on different tasks while preserving their original capabilities or even outperforming multi-task training @xcite . this technique enables us to generate the multi-task solution without synchronous training and is especially useful when the models for merging are already trained and available online. however, current literature has only applied model merging to models trained on similar tasks or even the same dataset, which inspires us to explore whether model merging will also work when the models are trained on different modalities such as language versus vision. we study this challenging problem with a novel goal of obtaining an effective and parameterefficient modality-agnostic model by merging the different modality transformers of a modalityspecific model. in vision-language (vl) domains @xcite , many modality-specific models @xcite @xcite @xcite employ dedicated transformer encoders to encode vision/language inputs independently, and on top of which additional transformer layers are used for multimodal fusion. on the other hand, the convergence of using transformer-based architectures for various singlemodality tasks has prompted researchers to adopt a single modality-agnostic transformer @xcite @xcite @xcite to encode different modality inputs and learn cross-modal interactions simultaneously. while modality-specific architectures usually perform better, modality-agnostic ones are simpler and more parameter-efficient. both architecture designs offer their unique benefits and have distinct use cases. however, obtaining the benefits of both these setups can be costly as it requires independent training. therefore, in this paper, we explore how to leverage a well-trained modality-specific model and develop a modalityagnostic model from it via model merging, with a goal of achieving similar performance to the modality-agnostic baseline which is pre-trained k seed seed pre-training vl pre-training (for modality-specific architecture) ## related work different types of vl architectures. vl models aim to generate outputs based on reasoning over both text and visual inputs. because the model takes two sources of inputs, there are different design choices to fuse the information, and hence yielding several mainstream designs in vl modeling: (1) modality-specific transformer @xcite @xcite , (2) modality-agnostic transformer @xcite @xcite , and (3) dual transformer @xcite . modality-specific transformers use a language transformer and a visual encoder to extract modality-specific representations and feed them into a cross-modal transformer to learn imagetext fusion information. some modern architectures @xcite @xcite @xcite use the language transformer for both language and cross-modal modeling. the dual transformer is a special case of modality-specific architecture, where the cross-modal fusion is realized via simple dot product, hence more efficient for fast retrieval. modality-agnostic transformers use one single transformer to process multi-modal data. these methods can be viewed as early fusion models, and the transformer learns modality-specific and cross-modal information simultaneously. the design of modality-agnostic architectures is relatively simple, and they are more memory-efficient because they only have one backbone. model merging. the de facto approach in recent model development centers around the "pre-trainthen-fine-tune" paradigm. due to the substantial resources required to train robust pre-trained models, the available repertoire is relatively limited, and many subsequent models are fine-tuned from this finite selection. recent research has discovered that models derived from the same pre-trained weights can be successfully merged into a single model. this merge can either improve single-task performance when the models are designed for the same task @xcite or create a multi-task solution when they target different tasks @xcite @xcite . various strategies for merging models have been proposed, including linear interpolation, task arithmetic @xcite , and regmean @xcite . however, if models do not share the same pre-trained weights, weight permutation might be required before merging @xcite @xcite . our work leverages model merging towards the innovative goal of generating an efficient modality-agnostic model from modality-specific models, thus eliminating the need for re-training. ## vl transformers a vl model takes an image-text pair as input and is designed to tackle various vl tasks with minimal architecture changes. the input image is first divided into patches, flattened into a sequence, and then fed into the vision embedding layer to obtain visual features. the input text is processed by the language embedding layer to obtain a sequence of textual features. the features are then jointly processed by the vl model, for which we explore different architectures detailed below. modality-specific transformer contains two nlayer (@xmath0, specialized for encoding image and text inputs, respectively, and an additional m (=2 in this paper) transformer layers for multimodal fusion @xcite @xcite . we denote their weights for layer i as w v i , w l i (@xmath1, 2, ..., 12) and w vl i (@xmath2, 12). please refer to figure 2 (a) for illustration. this design is inspired by vlmo @xcite , which has different routes for different tasks in forwarding. for visual question answering and other classification tasks, we use the first (n -m ) layers of vision and language transformers to extract the image and text features and feed the concatenation of them to the m multimodal fusion layers to get the final representation. we demonstrate this architecture/mechanism in the upper part of figure 1 and the right part of figure 2(b) . for image-text retrieval, we directly use the whole n layers of vision and language transformers to extract image and text representations for matching (left part of figure 2(b) ). only using unimodal encoders to extract features allows us to compute the dot product between features efficiently because it eliminates the need to forward the input pairs to cross-modal layers. please refer to vlmo for more details. for easier merging, we initialize all the transformers from the same initialization foot_1 , regardless of the modalities they are used for, and we use w 0 i to denote the initial weight for the i th layer. modality-agnostic transformer adopts one set of parameters to jointly learn vision, language, and cross-modal information, that is, w v @xmath3, 2, ..., 12 and w v @xmath4. the forward mechanism of this model is the same as the modality-specific architecture. shared-weight modality-specific transformer is in-between the two aforementioned extreme types of architecture, i.e., with partial modality-agnostic and modality-specific modules. specifically, we explore three variants of the shared-weight architectures with: (1) custom attention layers, (2) custom ffn, and (3) custom layer normalization layers. the illustration of them is shown in figure 1 . ## model merging for vl architectures our objective is to combine various transformers within the modality-specific architectures into a single modality-agnostic transformer. we do not merge the embedding layers and keep them modality-specific as they have different architecit has an n-layer vision, n-layer language, and m-layer cross-modal transformer. for classification tasks and itm in pre-training, it uses cross-modal layers to jointly process the image and text input. for retrieval tasks and itc in pre-training, it only forwards inputs through vision and language encoders. for modality-agnostic architecture, the v, l, and vl transformer layers share a single set of parameters while keeping the forward mechanism the same. in a modality-specific vl model, the transformers receive inputs from different modalities and work together to accomplish the target task. we merge such transformers to allow a single transformer that can handle different modalities concurrently. this situation deviates from previous studies on model merging @xcite @xcite , where the transformers (in these previous works) to be merged take input from the same modality. in the following, we describe our process for merging the vl architectures. ## experimental details we describe the experimental setup for our models and training in this section and more details about hyper-parameters for fine-tuning, computational usage, and the number of parameters of our models are included in appendix a. architectures. we use vit-b/16 @xcite (which has a similar architecture as bert @xcite ) throughout our experiments. the model has 12 transformer layers, and the dimension of the model is 768. vl pre-training. as described in section 4.2, we adopt the standard 10m image-text corpus (4m images) and utilize itc, itm, and mlm for vl pre-training. the overall pre-training objective is the weighted combination of the three aforementioned loss functions. itc maximizes the pair-wise positive matching and minimizes the matching between any negative pairs within the training batch. similar to itc, itm predicts whether the imagetext pair is positive or negative, by modeling it as binary classification. for mlm, we mask out 15% of text tokens, and the model is required to reconstruct the missing tokens with multimodal context. the overall pre-training objective is the weighted combination of the three aforementioned loss functions, and we find that using a smaller weight (0.25) for mlm empirically leads to better results. the forward mechanism of itc is the same as the retrieval task, and of itm and mlm is the same as vqa (figure 2(b) ). figure 3 : the effect of seed pre-training's duration on the merging results. increasing the iterations of seed pre-training improves the merging outcomes, and the optimality happens at 100k iterations as more iterations in seed pre-training reduce the iterations of vl pre-training, and limit the learning of the modality-specific model. fine-tuning. we use vqa @xcite , coco image-text retrieval @xcite , nlvr 2 (visual reasoning) @xcite , flickr30k (image retrieval) @xcite , imagenet-1k (image classification) @xcite and ade20k (semantic segmentation) @xcite as downstream tasks. in evaluation, we report the accuracy for vqa and nlvr 2 , top-1 accuracy for imagenet, and miou for ade20k. we use the average of top-1, top-5, and top-10 text recall (tr), and image recall (ir) for retrieval tasks. we report the results for each method with a single run. appendix a shows other details of experiments. ## experimental results in this section, we investigate the key factors impacting the merging performance on vqa and coco. recall that our goal is to improve the merging baseline in ## analysis in the preceding experiments, we highlight three factors that influence the outcomes of merging. this sparked a curiosity regarding the existence of potential metrics capable of predicting or explaining the quality of these merging results. one intuitive hypothesis is that the distance between the weights to be merged (vision and language weights in our case) might play an important role, as the performance drop caused by merging should be 0 if the distance between them is 0, and vice versa, the drop should be considerable if the distance is large (the weights are not in the same basin). to test the hypothesis, we apply various distance measures on the vision and language weights of modalityspecific models and compute the correlation between the distance and the performance drop after merging to assess the influence on merging results. ## conclusion we explore applying the model merging technique to transform modality-specific models into modality-agnostic models. with comprehensive experiments, we analyze the key factors impacting the merging results, including initialization, merging mechanisms, and model architectures. with seed pre-training and merging methods as simple as interpolation, we improve the merging results of the modality-specific model significantly (+3% on vqa, +7% on coco image-text retrieval, +25% on nlvr 2 , +14% on flickr30k and +3% on ade20k). moreover, we demonstrate the merged weight maintains the performance on unimodal tasks, and the proposed approach generalizes to another backbone model. we also discuss an initial recommended recipe for multimodal merging.