M ed INST : Meta Dataset of Biomedical Instructions (2024)

Abstract: The integration of large language model (LLM) techniques in the field of medical analysis has brought about significant advancements, yet the scarcity of large, diverse, and well-annotated datasets remains a major challenge. Medical data and tasks, which vary in format, size, and other parameters, require extensive preprocessing and standardization for effective use in training LLMs. To address these challenges, we introduce MedINST, the Meta Dataset of Biomedical Instructions, a novel multi-domain, multi-task instructional meta-dataset. MedINST comprises 133 biomedical NLP tasks and over 7 million training samples, making it the most comprehensive biomedical instruction dataset to date. Using MedINST as the meta dataset, we curate MedINST32, a challenging benchmark with different task difficulties aiming to evaluate LLMs’ generalization ability. We fine-tune several LLMs on MedINST and evaluate on MedINST32, showcasing enhanced cross-task generalization.

## introduction recent advancements in large language models (llms), such as gpt-4 (openai, 2024), llama-3 (meta, 2024) and mistral @xcite have demonstrated impressive performance across various open-domain nlp tasks. rather than developing specialized, task-specific systems, there is an increasing focus on rapidly adapting llms to specific tasks through simple prompting techniques. studies have demonstrated that such prompted llms can achieve and even outperforms the capabilities of specialized models in a variety of nlp tasks @xcite @xcite . due to the high cost of pre-training llms, instruction finetuning has become the standard method for adapting base llms to specific domains. therefore, training domain-specific llms has largely shifted to a data-centric approach. in recent years, the field of medical analysis has experienced a transformative shift with the integration of large language model (llm) techniques, fundamentally expanding the landscape of diagnostic and therapeutic strategies. the advancement of this field relies heavily on the availability of large, diverse, and well-annotated datasets, which are crucial for training robust and effective machine learning (ml) models. although specialized biomedical models such as biobert @xcite , clinicalxlnet @xcite , biom-transformers @xcite and scifive @xcite have achieved a success, they rely on task-specific modules and follow a pre-train then fine-tune paradigm for specified tasks @xcite . in this context, generalizing to unseen tasks is computationally expensive and time-consuming. attempts exist such as in-boxbart @xcite and biomistral @xcite are finetuned with biomedical instructions. however, the data involved in training and evaluation are limited. collecting raw medical data and converting it into a format suitable for llm applications is often complex and challenging. medical data and tasks vary significantly in format, size, and other parameters, necessitating extensive preprocessing and standardization. this task becomes even more intricate when integrating multiple datasets from various domains into a cohesive, standardized format. this raises the necessity of a comprehensive biomedical instruction meta-dataset. to address the problem, we release medinst 1 , an instruction dataset collection includes 133 biomedical nlp tasks in 12 categories such as named entity recognition (ner), question-resource medinst sup-natinst @xcite box @xcite blurb @xcite answering (qa), relation extraction (re), etc. a benchmark is set by curate a test set from the entire collected dataset. in the experiment, multiple scales of llms are finetuned on our training data to demonstrate the generalization performance enhancement. ## medinst: meta dataset of biomedical instructions we curate medinst by collecting 98 welladopted biomedical datasets from 12 task categories and reformulating them into 133 tasks. all tasks are regarded as text generation task and the data are formatted to instruction-following samples. the instructions are human annotated and tailored for each dataset/task. figure 1 (a) depicts a visualization of the dataset composition of medinst. ## conclusion in this paper, we introduce an instruction metadataset medinst comprising 133 biomedical tasks across 12 task categories and a challenging benchmark medinst32 for evaluating multitask biomedical models. through various experiments, we train multiple biomedical models and demonstrate their strong generalization performance on biomedical tasks using our dataset. due to resource constraints, we trained only on a small subset and 8b models. using the full dataset and larger models may lead to further improvements, which is left for future work. our work lays the foundation for developing better-performing biomedical llms.