B i M edi X : Bilingual Medical Mixture of Experts LLM (2024)

Abstract: In this paper, we introduce BiMediX, the first bilingual medical mixture of experts LLM designed for seamless interaction in both English and Arabic. Our model facilitates a wide range of medical interactions in English and Arabic, including multi-turn chats to inquire about additional details such as patient symptoms and medical history, multiple-choice question answering, and open-ended question answering. We propose a semi-automated English-to-Arabic translation pipeline with human refinement to ensure high-quality translations. We also introduce a comprehensive evaluation benchmark for Arabic medical LLMs. Furthermore, we introduce BiMed1.3M, an extensive Arabic-English bilingual instruction set that covers 1.3 Million diverse medical interactions, including 200k synthesized multi-turn doctor-patient chats, in a 1:2 Arabic-to-English ratio. Our model outperforms state-of-the-art Med42 and Meditron by average absolute gains of 2.5% and 4.1%, respectively, computed across multiple medical evaluation benchmarks in English, while operating at 8-times faster inference. Moreover, our BiMediX outperforms the generic Arabic-English bilingual LLM, Jais-30B, by average absolute gains of 10% on our Arabic and 15% on our bilingual evaluations across multiple datasets. Additionally, BiMediX exceeds the accuracy of GPT4 by 4.4% in open-ended question UPHILL evaluation and largely outperforms state-of-the-art open source medical LLMs in human evaluations of multi-turn conversations. Our trained models, instruction set, and source code are available at https://github.com/mbzuai-oryx/BiMediX.

## introduction large language models (llms) demonstrate excellent ability to understand instructions, generate human-like text, and tackle unseen or complex * equally contributing first authors. figure 1 : bimedix is adept at conducting precise medical conversations seamlessly in both english and arabic. it demonstrates advanced interactive capabilities, such as soliciting further details from the patient and seamlessly switch languages based on patient input while maintaining the continuity of the conversation. the arabic text translates as follows: user: the pain occurs mostly on both sides of the head, sometimes on only one side. it is a throbbing pain that can last for several hours. light and noise seem to make it worse. bimedix: thank you for the detailed information. based on your symptoms, it sounds like you may have a migraine. migraine headaches are usually characterized by a throbbing pain on one or both sides of the head, and can be triggered or increased in intensity by light and noise. it is important to consider that there are other possible causes for your symptoms, so we may need to do further evaluation. have you noticed any specific factors that trigger these headaches? recent methods have emerged to bridge the gap in medical domain knowledge by developing opensource medical llm alternatives @xcite @xcite to proprietary models like chatgpt @xcite . however, among the leading medical llms in english, med42-70b @xcite has not made its training data and resources available. meditron-70b @xcite , relies on separate finetuning for each evaluation dataset. this approach is likely to compromise the model's ability to serve as a unified solution with diverse interaction capabilities. moreover, many of the leading open-source medical llms @xcite are limited to the english language, posing limitations in their ability to understand and interact in resource-constrained languages such as arabic. despite arabic's potential to cater to a population of more than 400 million people, it remains underrepresented in the medical llm literature. the unique features of arabic, such as its distinctive script and less conventional right-toleft writing format, the unavailability of large-scale medical training data in arabic, and the lack of a comprehensive benchmark to evaluate arabic medical llms, present challenges in the development of arabic or english-arabic bilingual medical llms. in this work, we introduce bimedix, a bilingual medical mixture of experts llm with seamless conversational capabilities in both english and arabic. our bimedix is a chat agent with state-of-the-art performances on non-chat, open qa and multiturn chat evaluations. ## related works early bert-style open medical language models leveraged the pubmed corpus for either continuous pre-training @xcite or specific domain training from scratch @xcite @xcite . subsequent research enhanced bidirectional systems to incorporate additional knowledge @xcite , while later studies have tailored gpt-2 based models to medical and scientific literature @xcite @xcite . recent works such as medalpaca @xcite focus on finetuning large scale open-source llms such as llama @xcite on a set of medical instructions. chatdoctor @xcite ) enhanced a llama model on 100k clinical q&as, incorporating knowledge retrieval capabilities. clinicalcamel @xcite introduced question answering data, converting pubmed articles and medqa into questions and descriptive answers. meditron @xcite performed pre-training on pubmed content and medical texts, with further refinements on individual mcqa datasets. med42 @xcite instruction-tuned llama model for medical tasks, though the details of its training remain undisclosed. additional details on related english medical llms are provided in appendix b. recent years have witnessed progress in arabic language processing models @xcite . in addition to these monolingual models, arabic has been integrated into multilingual frameworks @xcite @xcite . the recent launch of the jais model @xcite ) marked a substantial advancement as a general-purpose bilingual llm for english and arabic. additionally, many of the latest foundational models @xcite demonstrate the ability to reply in arabic, despite their limited exposure to the language during pre-training. however, these generic models lack sufficient medical domain knowledge, making them ill-equipped for medical applications in both arabic and english languages. ## conclusion our work introduces the first bilingual medical mixture of experts (llm) bimedix designed to facilitate comprehensive medical interactions, including conversations, mcqa, and qa in both english and arabic languages. the implementa-