M o E xtend: Tuning New Experts for Modality and Task Extension (2024)

Abstract: Large language models (LLMs) excel in various tasks but are primarily trained on text data, limiting their application scope. Expanding LLM capabilities to include vision-language understanding is vital, yet training them on multimodal data from scratch is challenging and costly. Existing instruction tuning methods, e.g., LLAVA, often connects a pretrained CLIP vision encoder and LLMs via fully fine-tuning LLMs to bridge the modality gap. However, full fine-tuning is plagued by catastrophic forgetting, i.e., forgetting previous knowledge, and high training costs particularly in the era of increasing tasks and modalities. To solve this issue, we introduce MoExtend, an effective framework designed to streamline the modality adaptation and extension of Mixture-of-Experts (MoE) models. MoExtend seamlessly integrates new experts into pre-trained MoE models, endowing them with novel knowledge without the need to tune pretrained models such as MoE and vision encoders. This approach enables rapid adaptation and extension to new modal data or tasks, effectively addressing the challenge of accommodating new modalities within LLMs. Furthermore, MoExtend avoids tuning pretrained models, thus mitigating the risk of catastrophic forgetting. Experimental results demonstrate the efficacy and efficiency of MoExtend in enhancing the multimodal capabilities of LLMs, contributing to advancements in multimodal AI research.

## introduction general-purpose large language models (llms) have demonstrated their effectiveness across a broad spectrum of application scenarios, such as conversational chatbot @xcite , document analysis @xcite , and coding @xcite . while the most powerful llms, such as chatgpt @xcite , llama @xcite , and mixtral @xcite , are predominantly trained on textual data, there is a growing interest in extending their capabilities to support a wider array of applications beyond natural language processing, especially with a significant focus on vision-language understanding @xcite @xcite . while training large models from scratch on multimodal data suffers from insufficient data @xcite and significant training costs @xcite , most efforts have been focused on enhancing the multimodal capabilities of pretrained llms @xcite . to accomplish this, llms handle new modal data by processing representations extracted by encoders specific to each modality. for instance, the vision transformer pre-trained with clip @xcite is utilized to encode visual images. then, the model is trained using text-image q&a pairs to carry out tasks based on these multimodal instructions. the parameter-efficient approach to bridging the gap between modality-specific encoders and large language models (llms) involves the use of a few linear projection layers @xcite and low-rank adaptation (lora) @xcite . however, this does not entirely mitigate the modality gap, limiting llms' ability to fully understand new modalities. consequently, state-of-the-art multimodal methods, e.g. llava @xcite , have sought to further enhance the multimodal capabilities of llms by fully fine-tuning these models on multimodal datasets @xcite . despite these efforts, fully fine-tuning encounters two significant obstacles: 1) catastrophic forgetting: llms, when fine-tuned to effectively integrate various modalities, tend to lose the knowledge they had acquired previously @xcite . 2) large fine-tuning cost: with the increasing sizes of llms, fully finetuning on larger models is becoming increasingly impractical. as a result, smaller models, like those with 7 billion parameters, are often preferred. however, this preference restricts the exploration and utilization of the capabilities of larger llms. how to efficiently extend new modality to large llm while reduce the side effect of catastrophic forgetting is an urging problem for multimodal llms. mixture-of-experts (moe) architectures enable llms to use the gate layer to dynamically select the most relevant experts from a diverse set of specialized experts, e.g. different mlp layers in transformer, for a given query token. moe helps to enlarge the model size by increase the number of experts while keeping low inference cost by selecting a sub set of experts for each token. for instance, the mixtral-8x7b model @xcite incorporates 8 mlp experts per block, totaling 46.7 billion parameters, yet it selects only 2 experts, utilizing 12.9 billion parameters per token. nonetheless, the current moe models predominantly concentrates on the textual modality. we introduce an extension strategy for moe models, named moextend, designed to accommodate new modalities. this strategy involves incorporating new modality-specific experts and calibration modules into trained moe models to enhance their capability to process additional modalities. moextend maintains the original moe model parameters unchanged, while only trains the newly added experts and the corresponding gate layer. by doing so, moextend facilitates the efficient adaptation of new modalities into large models while also addressing issues of catastrophic forgetting @xcite . we observe that the rapid adaptation to new modalities relies on the weight initialization of new experts and gates, and the insertion position of these new experts. thus, we introduce a simple yet effective scheme for selecting positions and weights of new experts based on evaluating distribution shifts. utilizing the data from the new modality, we fine-tune the existing gate layers of the moe model. then, we infer the new modality data to the models before and after fine-tuning and get the average gate probability distribution for all samples. by comparing the degree of gate probability distributions before and after fine-tuning, we identify the top-k layers for adding experts by examining the magnitude of these shifts. then, based on the probability distribution after fine-tuning, we determine the expert with the highest probability and replicate the gate and expert weights onto the newly incorporated expert. experimental results show that moextend achieves a training speed acceleration ∼6 times faster than full fine-tuning, while also delivering superior performance. the positions selection scheme in moextend allows for fewer newly added experts, specifically, half the number of new experts required for the mixtral model, which reduces training time to ∼30 hours without compromising performance. in addition, moextend helps mitigate the risk of catastrophic forgetting when extending moe llms to handle multimodal inputs. our contributions can be summarized as follows: ## methodology in this section, we introduce moextend as an example of extending the visual modality for moe models, which were originally designed for text modality only. as shown in fig. 1 , moextend consists of three stages: alignment, extension with extender, and fine-tuning for the extension part. the purpose of the alignment stage is to initially align the moe llm with the newly added visual modality using a pre-trained vision encoder. the extension stage determines which moe layers should be extended to accommodate the new modality information. the fine-tuning stage is then employed to tune the newly added parameters, achieving the final expansion of multimodal information. ## ablation study and analysis effect of model architectures. we investigate the impact of different architectures on the performance of moextend. while the intuitive approach of adding new experts to all layers might seem optimal, our experiments, detailed in ## conclusion in this work, we introduce moextend, an effective framework tailored to streamline the modality adaptation and extension of mixture-of-experts (moe) models. moextend introduces new experts into moe models by putting them at the parallel positions of the experts in moe. then moextend designs a method to select previous experts in moe for initilizing the new experts. finally, it only tunes the new experts on the corresponding modal data and tasks. this endows moe with novel knowledge without necessitating the tuning of pretrained models such as moe and vision encoders, thus avoiding the catastrophic forgetting issue. furthermore, moextend facilitates rapid adaptation and extension to new modal data or tasks, thereby effectively addressing the challenge of accommodating new modalities within llms. empirical results show the efficacy and efficiency of moextend in augmenting the multimodal capabilities of llms.