M ed- M o E : Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Models (2024)

Abstract: Recent advancements in general-purpose or domain-specific multimodal large language models (LLMs) have witnessed remarkable progress for medical decision-making. However, they are designated for specific classification or generative tasks, and require model training or finetuning on large-scale datasets with sizeable parameters and tremendous computing, hindering their clinical utility across diverse resource-constrained scenarios in practice. In this paper, we propose a novel and lightweight framework Med-MoE (Mixture-of-Experts) that tackles both discriminative and generative multimodal medical tasks. The learning of Med-MoE consists of three steps: multimodal medical alignment, Instruction tuning and routing, and domain-specific MoE tuning. After aligning multimodal medical images with LLM tokens, we then enable the model for different multimodal medical tasks with instruction tuning, together with a trainable router tailored for expert selection across input modalities. Finally, the model is tuned by integrating the router with multiple domain-specific experts, which are selectively activated and further empowered by meta experts. Comprehensive experiments on both open- and close-end medical question answering (Med-VQA) and image classification tasks across datasets such as VQA-RAD, SLAKE and Path-VQA demonstrate that our model can achieve performance superior to or on par with state-of-the-art baselines, while only requiring approximately 30%-50% of activated model parameters. Extensive analysis and ablations corroborate the effectiveness and practical utility of our method.

## introduction creating systems with human-level multimodal understanding is essential for medical decisionmaking @xcite @xcite . recent progress on multimodal large language models (mllms) such as llava @xcite , minigpt4-v2 @xcite , cogvlm @xcite have demonstrated great performance across multimodal tasks, however, they are less effective in the medical domain as they are usually trained with web contents which differ significantly from the medical data. domainspecific models such as med-flamingo @xcite , med-palm m @xcite , and llava-med @xcite exhibit promising results across various medical tasks, such as medical visual question-answering (med-vqa), by training with medical domain data. however, these models are usually tailed for certain kinds of tasks, such as close-or open-end vqa, while in practice, medical mllms need to handle both discriminative and generative tasks to provide more reliable and interpretable decisions. moreover, existing models are usually obtained with heavy llms with sizeable parameters, such as llama(7b) in llava-med, leading to high training and inference costs and hindering its practical utility to broad clinical practitioners. it is appealing but challenging to build lightweight yet effective medical mllms for multimodal decision-making @xcite @xcite . recent research shows that scaling up the quantity or quality of training data, as well as increasing the size of the model, can result in enhanced performance @xcite @xcite @xcite . however, training and deploying these models also demand substantial computational resources, rendering them less appealing for numerous clinical practitioners who may lack sufficient computing power @xcite @xcite . for instance, many institutions may not possess powerful gpus such as nvidia a100 cards to tune llama-7b model family, e.g., llava-med. moreover, the medical data differs drastically from web contents, and its inherent multi-modality, such as imaging from ct, mri, x-ray and pathology, presents additional challenges to develop effective yet lightweight medical mllms @xcite . this task becomes even more difficult when considering the requirements for reliability and interpretability in decisionmaking @xcite . recent work explores cost-effective training of light-weight llms by architecture design, training procedure or hardware optimization etc @xcite @xcite . among these techniques, the mixtureof-expert (moe) strategy has shown great potential for general-purpose training @xcite @xcite , e.g., the mixtral family employs sparse moes to achieve competing performance with llama-70b with only 12.9b active parameters @xcite ; the moe-llava propose a moe-based sparse large vlm framework with novel training strategies @xcite . by combining multiple small-scale sub-modules, i.e., experts, and activating only the top-k relevant experts for each task, the moe model can achieve good performance with much less computing cost. despite these successes in general domains, current moes often overlook the specialization and synergy of experts required in medical contexts, and their applicability in the medical domain remains unexplored. in this paper, we propose a lightweight and effective framework med-moe for multimodal generative or discriminative med-vqa and classification tasks. our med-moe incorporates multiple domain-specific experts along with global meta expert, emulating the workflow in hospitals where various departments collaborate together for disease diagnosis. in particular, the med-moe takes lightweight llms with smaller sizes of parameters as the base model of experts, which are first trained with medical image and caption pairs to align visual and textual modalities. afterward, the model is trained with medical instruction following datasets to better perform multimodal medical tasks. meanwhile, a router is trained to identify different medical image modalities, enabling better selection across multiple domain-specific experts during decision-making. inspired by the well-known resnet @xcite architecture and the multi-disciplinary team (mdt) diagnosis mechanism in clinics, we propose to add an additional meta expert in the shortcut, as shown in figure 2 , which captures global medical information to assist the specified expert for better performance. during inference, only the meta expert and the selected experts are activated, leading to a lightweight model with only a small portion of activated parameters. the med-moe consistently demonstrates significant performance improvements across diverse medical datasets, encompassing both open-and close-end med-vqa and medical image classification tasks in vqa-rad, slake, pathvqa, pneu-moniamnist, and organcmnist. comprehensive experiments show that our med-moes, which are constructed with two small-scale llms, i.e., phi2 (2.7b) @xcite and stablelm (1.7b) @xcite , can attain performance superior to or on par with the state-of-the-art llava-med (7b) model, with only 2.0-3.6b activated parameters. extensive ablations and analysis demonstrate the efficacy of our med-moe in advancing multimodal medical tasks and highlight its potential to enhance outcomes in resource-limited healthcare settings. add & norm linear vision encoder word embedding language response ffn add & norm self-attention add & norm language response ffn router domain label add & norm add & norm router mri ct domain m ... ... activated forward activated forward (sample) non-trainable non-activated forward ... language response domain experts meta expert expert 1 expert 2 expert m is the lung healthy? copy weight linear vision encoder word embedding add & norm self-attention linear vision encoder word embedding self-attention copy weight copy weight ## methods the training of med-moe includes three phases, as illustrated in figure 2 . first, we perform multimodal medical alignment to help the llm to comprehend medical images by leveraging the vision encoder's image tokens. next, we conduct instruction tuning to enable the model to execute various medical tasks and enhance its instruction-following ability. meanwhile, a router is trained with a small amount of labeled data to characterize the input modality. finally, we perform domain-specific moe tuning by replacing the model's ffn with sparsely activated experts, where a meta-expert is always activated to capture global information. ## conclusion we have introduced med-moe, a lightweight framework for multimodal medical tasks, addressing both discriminative and generative needs. optimized for resource-constrained environments, med-moe involves aligning medical images with language model tokens, task-specific instruction tuning, and domain-specific expert fine-tuning. our approach reduces activated parameters while maintaining or surpassing state-of-the-art performance. our experiments on vqa-rad, slake, and path-vqa validate med-moe's effectiveness and efficiency. this model offers a practical solution for deploying advanced medical ai in diverse and resource-limited clinical settings.