Model Composition for Multimodal Large Language Models (2024)

Abstract: Recent developments in Multimodal Large Language Models (MLLMs) have shown rapid progress, moving towards the goal of creating versatile MLLMs that understand inputs from various modalities. However, existing methods typically rely on joint training with paired multimodal instruction data, which is resource-intensive and challenging to extend to new modalities. In this paper, we propose a new paradigm through the model composition of existing MLLMs to create a new model that retains the modal understanding capabilities of each original model. Our basic implementation, NaiveMC, demonstrates the effectiveness of this paradigm by reusing modality encoders and merging LLM parameters. Furthermore, we introduce DAMC to address parameter interference and mismatch issues during the merging process, thereby enhancing the model performance. To facilitate research in this area, we propose MCUB, a benchmark for assessing ability of MLLMs to understand inputs from diverse modalities. Experiments on this benchmark and four other multimodal understanding tasks show significant improvements over baselines, proving that model composition can create a versatile model capable of processing inputs from multiple modalities.

## introduction recent advancements in multimodal large language models (mllms) have established them as the forefront of multimodal learning paradigms @xcite . the prevalent approach involves aligning modality encoders with large language models (llms) through extensive modality-text paired data and then fine-tuning with modality-specific instruction data. this paradigm has been successfully applied to a wide range of modalities such as image @xcite , audio @xcite @xcite , video @xcite , and point cloud @xcite , resulting in the emergence of a diverse array of mllms with unique modal capabilities. there are also some efforts to enable a single mllm to handle multiple modalities. one way to achieve this is to align the llm with a multimodal encoder such as imagebind @xcite with only image-text data (figure 1(a) ) @xcite . this approach leverages the inherent alignment of different modalities within the multimodal encoder, allowing the mllm to comprehend various modalities to a certain degree. however, the absence of modalityspecific instruction data often results in suboptimal performance. another approach entails the concurrent training of the mllm with multiple modality encoders (figure 1 (b)). for example, chatbridge @xcite connects image, video and audio encoders with the llm through a joint training process with multimodal instruction data (i.e., video-audio chats). this kind of methods show potential but faces two major challenges. first, it is resource-heavy to collect paired data across multiple modalities. second, adapting these models to new modalities requires additional training, adding to the complexity and resource demands of the development process. given the limitations of current approaches, we propose and study a more practical setting: model composition for mllms (figure 1(c )). our primary research question is simple yet fundamental: can we create a new mllm by combining existing mllms to inherit their understanding of different modalities without training? model composition for mllms is advantageous for two key reasons: (1) it eliminates the need for the resource-heavy process of training and gathering multimodal data, and (2) it promises enhanced adaptability, facilitating seamless incorporation of new modalities. some recent studies, such as x-instructblip @xcite , serve as pioneering efforts in model composition for mllms. these works primarily train projectors to align different encoders with a single llm and demonstrate the ability to process multiple modalities concurrently. however, a critical limitation is their applicability only to mllms with frozen language model weights. this constraint restricts the range of models that can be utilized, and impairs the overall performance of the mllms @xcite . in this paper, we first propose a framework for model composition for mllms. our implementation, named naivemc, is elegantly simple: for the mllms to be composed, we directly reuse their modality-specific encoders and merge their llm parameters. we demonstrate that mllms, as long as initialized from the same llm, can achieve zeroshot multi-modality expansion through this model composition framework, regardless of whether the parameters of the llm have been fine-tuned. furthermore, to mitigate parameter interference in the composition process and optimize the performance of the composite model, we propose damc, an advanced framework with parameter decoupling and adjustment for model composition. by separating modality-specific parameters from language model parameters during initial mllm training, damc allows for the selec-tive merging of textual parameters, reducing crossmodal interference. moreover, damc introduces an adaptive parameter adjustment mechanism to ensure optimal compatibility and effectiveness of the composite model, achieving a balanced and efficient multi-modality expansion. to assess the efficacy of our proposed frameworks, we conduct comprehensive experiments on tasks that require an integrated understanding of inputs from diverse combinations of four prevalent modalities: image, audio, video, and point cloud. to facilitate the research in model composition for mllms, we also build mcub, a benchmark specifically designed for evaluating the capability to concurrently comprehend multiple modalities by identifying commonalities across inputs from various modalities. experimental results indicate that our frameworks enable the composition of existing mllms from different modalities without requiring further training, yielding a versatile and high-performing multimodal model adept at handling any combination of these modalities. our contributions are three-fold: ## discussion and analysis composition of mllms with different training strategies. in section 3.3, we advocate training mllms with parameter decoupling. intriguingly, this decoupling can also be implemented at inference time, by replicating the parameters to mimicking identical parameters on modality-specific and textual inputs. in this way, we can compose models with different training strategies. we conduct experiments on the music-avqa dataset with v + i + a inputs. we vary the training strategy of video and image mllms, and compose them with an audio mllms trained with parameter decoupling. the results in ## conclusion in this paper, we introduce and explore the concept of model composition for mllms, showcasing its ability to seamlessly integrate diverse modal capabilities without additional training. initially, we introduce a foundational model composition framework, referred to as naivemc. advancing further, we design damc that employs parameter decoupling and adjustment to reduce crossmodal interference and optimize performance of the composite model. in addition, we construct a benchmark mcub for multimodal commonality understanding evaluation to facilitate the relative research. extensive experiments demonstrate the effectiveness of our approaches. we hope that our work will inspire further exploration into model composition for multimodal models.