B i M edi X 2 : Bio-Medical EX pert LMM for Diverse Medical Modalities (2025)

Abstract: We introduce BiMediX2, a bilingual (Arabic-English) Bio-Medical EXpert Large Multimodal Model that supports text-based and image-based medical interactions. It enables multi-turn conversation in Arabic and English and supports diverse medical imaging modalities, including radiology, CT, and histology. To train BiMediX2, we curate BiMed-V, an extensive Arabic-English bilingual healthcare dataset consisting of 1.6M samples of diverse medical interactions. This dataset supports a range of medical Large Language Model (LLM) and Large Multimodal Model (LMM) tasks, including multi-turn medical conversations, report generation, and visual question answering (VQA). We also introduce BiMed-MBench, the first Arabic-English medical LMM evaluation benchmark, verified by medical experts. BiMediX2 demonstrates excellent performance across multiple medical LLM and LMM benchmarks, achieving state-of-the-art results compared to other open-sourced models. On BiMed-MBench, BiMediX2 outperforms existing methods by over 9% in English and more than 20% in Arabic evaluations. Additionally, it surpasses GPT-4 by approximately 9% in UPHILL factual accuracy evaluations and excels in various medical VQA, report generation, and report summarization tasks. Our trained models, instruction set, and source code are available at - https://github.com/mbzuai-oryx/BiMediX2

## introduction recently, medical large language models (llms) and medical large multimodal models (lmms) have shown promising results as conversational assistants for improving accessibility to quality medical advice. however, most medical vision-language models (vlms) referred to interchangeably as medical lmms in this paper often com- ## bimedix2 the architecture of bimedix2 is designed to facilitate seamless integration of medical image analysis and bilingual multi-turn conversations. at its core, (see fig 2 ) the model employs a vision encoder @xcite to process a diverse array of medical imaging modalities, including chest x-rays, ct scans, mris, histology slides, and gross pathology images. this visual data is aligned with textual inputs through a dedicated projector, ensuring accurate and contextually rich medical image-text mapping following @xcite . as shown in fig 2 we use the '<@xmath0. for text based medical data the inputs are processed using a standard tokenizer, transforming them into the language embedding space of llama 3.1 @xcite . this design enables bimedix2 to generate precise and context-aware responses in either english or arabic, depending on the user prompt and supports multimodal interactions while preserving the medical llm capabilities. key to bimedix2's performance is its modular and efficient training approach. lora adapters @xcite are utilized to fine-tune the language model while maintaining computational efficiency and minimizing resource demands. the projector is simultaneously fine-tuned to optimize image-text alignment in a medical context. furthermore, the system is supported by a robust data generation framework, where a comprehensive english data corpus is translated into arabic using gpt-4o. a random subset of this translation is meticulously verified by bilingual medical experts to ensure clinical relevance and linguistic accuracy. this pipeline enables bimedix2 to excel in medical tasks, including report generation, radiology analysis, pathology insights, and ophthalmological assessments, in a unified, bilingual, and multimodal framework. ## experiments in the literature, evaluating medical language models predominantly involves multiple-choice question-answering tasks, with accuracy as the performance metric. we employed the eleutherai @xcite evaluation framework for evaluating text-only medical tasks following @xcite . for our bilingual and multimodal evaluations, we utilized several benchmarks and tools. we leveraged our bimed-mbench for assessing the correctness, helpfulness, relevance and accuracy of the model in a bilingual medical multimodal context. additionally, we evaluated the model on vqa using datasets such as path-vqa, slake, and rad-vqa. for report generation, we used the mimic-cxr (johnson et al., 2019) dataset, and for report summarization, we used the mimic-iii @xcite dataset. these evaluations were conducted using the multimedeval @xcite toolkit, ensuring a comprehensive assessment of the model's multimodal capabilities. ## additional experiments multi-stage training pipeline: to assess the impact of each component in our training framework, we evaluate our 8b model at three key stages. this staged evaluation allows us to disentangle the indi-vidual contributions of vision-language alignment and multimodal fine-tuning. here the baseline model is obtained by pretraining the projector on lcs-558k dataset following the llava-pp @xcite repository. the baseline model lacks understanding of medical images which is introduced in our stage-1 alignment training. finally we lora finetune model to obtain our medically instruction tuned model. as observed in tab 9, the text based medical evaluation scores do not change with stage 1 training as we are only training the projector here and the language model performance is therefore consistent. ## conclusion bimedix2 represents a leap forward in bilingual, multimodal medical ai, addressing the global need for accessible and inclusive healthcare solutions in both english and arabic. by integrating text and visual modalities within a unified architecture, it enables seamless multi-turn interactions for diverse medical tasks, including medical image analysis and complex medical conversations. key contributions include the comprehensive bilingual dataset, bimed-v, which provides diverse multimodal medical instructions tailored for both lan-