B io M istral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains (2024)

Abstract: Large Language Models (LLMs) have demonstrated remarkable versatility in recent years, offering potential applications across specialized domains such as healthcare and medicine. Despite the availability of various open-source LLMs tailored for health contexts, adapting general-purpose LLMs to the medical domain presents significant challenges.In this paper, we introduce BioMistral, an open-source LLM tailored for the biomedical domain, utilizing Mistral as its foundation model and further pre-trained on PubMed Central. We conduct a comprehensive evaluation of BioMistral on a benchmark comprising 10 established medical question-answering (QA) tasks in English. We also explore lightweight models obtained through quantization and model merging approaches. Our results demonstrate BioMistral’s superior performance compared to existing open-source medical models and its competitive edge against proprietary counterparts. Finally, to address the limited availability of data beyond English and to assess the multilingual generalization of medical LLMs, we automatically translated and evaluated this benchmark into 7 other languages. This marks the first large-scale multilingual evaluation of LLMs in the medical domain. Datasets, multilingual evaluation benchmarks, scripts, and all the models obtained during our experiments are freely released.

## introduction in the rapidly evolving landscape of natural language processing (nlp), generative large language models (llms) like @xcite and vicuna @xcite have revolutionized human-computer interactions, demonstrating remarkable versatility and advanced capabilities across various tasks and domains. these models exhibit human-like comprehension and reasoning, enabling them to tackle basic textual understanding as well as complex problem-solving tasks. the emergence of open-source llms such as bloom @xcite and llama @xcite underscores the transformative potential of these models, facilitating their innovative use in specialized domains including medicine @xcite . however, integrating llms into healthcare and medicine presents unique challenges and opportunities @xcite . while preliminary adoption has opened new avenues for innovation, concerns about data privacy risks associated with proprietary models like medpalm-2 @xcite and gpt-4 @xcite have arisen. the community's interest in specialized llms for healthcare has led to initiatives like pmc-llama @xcite and medalpaca @xcite . however, the adoption of open-source medical models has been limited, primarily due to the lack of lightweight models allowing commercial use with performance comparable to larger or proprietary models. to address this gap, there is a need to develop specialized models based on open-source foundation ones like gpt-neox @xcite , llama 2 @xcite , or mistral @xcite , and to optimize them for use on consumer-grade devices while maintaining performance. in this work, we present biomistral 7b, a specialized llm tailored for the biomedical domain, derived from mistral 7b instruct v0.1 @xcite and further pre-trained on pubmed central. our contributions include: all datasets, the multilingual benchmark, preprocessing scripts, and models are accessible on huggingface foot_0 and github foot_1 under apache 2.0 license. ## related work generative llms with a generalist purpose, such as @xcite and gemini @xcite , have demonstrated exceptional performance across various nlp tasks, including qa, text summarization, and language translation in zero-and few-shot scenarios. while these models firstly remain proprietary, the open-source movement has played a crucial role in democratizing llms, offering powerful capabilities to the broader community with llama 2 @xcite , vicuna @xcite , falcon @xcite and mistral @xcite models. the adaptation of llms to specialized domains often involves encoding domain-specific knowledge to fully leverage the models' capabilities. this adaptation process has led to the development of several open-source models from pre-trained general domain models. encoder-based models, such as bert @xcite , and encoderdecoder, such as t5 @xcite , have been adapted for the medical domain @xcite @xcite but faced challenges with qa tasks. more recently, multiple llms based on decoder-only architecture have been developed for the medical domain, such as biogpt @xcite based on gpt-2 @xcite , clinicalgpt @xcite , based on bloom-7b @xcite , pmc-llama @xcite , medalpaca @xcite and chat-doctor @xcite , based on llama @xcite , and biomedgpt-lm-7b @xcite and meditron-7b @xcite , adapted from llama-2 @xcite . in contrast, proprietary medical llms like gpt-4 medprompt @xcite and med-palm 2 @xcite face usability issues similar to general-purpose models. nonetheless, they can be used as benchmark "target goals" for the development of domain-specific open-source models. ## biomistral in this section, we present the modules that facilitated the construction of biomistral 7b. we first develop our training corpus (section 3.1) used during further pre-training. we then present the model adaptation method (section 3.2). finally, we discuss the approaches for model merging (section 3.3) and expose the employed quantization strategies (section 3.4). ## evaluation protocol to assess the performance of biomistral 7b models, we first describe our benchmark of english medical reasoning tasks (section 4.1) and their multilingual translation (section 4.2), before presenting the instruction prompting (section 4.3) and the supervised fine-tuning strategy (section 4.4) employed for the models' evaluation. ## results and discussions in this section, we report, analyze, and discuss the performance of biomistral 7b models across various dimensions. we begin by examining its performance in a few-shot learning scenario (section 5.1), followed by an evaluation of the finetuning performances (section 5.2) of biomistral 7b compared to several baseline models. the effectiveness of biomistral 7b model merging strategies is then reported (section 5.3) before exploring its generalization capabilities across several languages (section 5.4). additionally, we analyze the performance of biomistral quantized versions in a few-shot scenario (section 5.5). finally, we delve into its reliability by examining its calibration (section 5.6) and truthfulness (section 5.7). ## conclusion we introduced biomistral 7b, a collection of medical llms resulting from further pre-training mistral 7b instruct on high-quality pubmed central resources. biomistral 7b incorporates quantized and merged model variants and demonstrates stateof-the-art performance on the multilingual medical evaluation benchmark compared to other opensource 7b models. our future work aims to assess the generation quality of biomistral 7b through human evaluation. additionally, we plan to enhance its multilingual and chat capabilities using supervised fine-tuning and direct preference optimization tech-niques, building on top of experiments conducted by @xcite and @xcite . finally, we intend to improve the calibration and reliability of our model by integrating techniques such as jeffrey's divergence @xcite or platt scaling @xcite during the further pretraining process.