BMR etriever: Tuning Large Language Models as Better Biomedical Text Retrievers (2024)

Abstract: Developing effective biomedical retrieval models is important for excelling at knowledge-intensive biomedical tasks but still challenging due to the lack of sufficient publicly annotated biomedical data and computational resources. We present BMRetriever, a series of dense retrievers for enhancing biomedical retrieval via unsupervised pre-training on large biomedical corpora, followed by instruction fine-tuning on a combination of labeled datasets and synthetic pairs. Experiments on 5 biomedical tasks across 11 datasets verify BMRetriever’s efficacy on various biomedical applications. BMRetriever also exhibits strong parameter efficiency, with the 410M variant outperforming baselines up to 11.7 times larger, and the 2B variant matching the performance of models with over 5B parameters. The training data and model checkpoints are released at https://huggingface.co/BMRetriever to ensure transparency, reproducibility, and application to new domains.

## introduction in the field of biomedicine, the ability to effectively retrieve knowledge from external corpora is crucial for large language models (llms) to excel at biomedical nlp tasks @xcite @xcite . by tapping into up-to-date domain knowledge, retrieval-augmented llms have demonstrated promising results in various biomedical downstream applications, including knowledge discovery @xcite , question answering @xcite , and clinical decision-making @xcite @xcite . several works have designed specialized retrieval models for biomedical domains @xcite @xcite @xcite . however, these models are typically built upon bert-series models, which have limited representative power. besides, they often rely on proprietary data (e.g., private search logs or patient records), making it challenging to scale them up to accommodate larger models effectively due to privacy concerns. while recent studies in the general domain have improved neural retrieval models via scaling up model size @xcite @xcite and training data @xcite @xcite , adapting such models to the biomedical domain may lead to suboptimal performance due to the distribution shift issue @xcite . developing large-scale retrieval models dedicated to the biomedical domain without requiring massive proprietary datasets remains crucial yet challenging. ## related work earlier research explores various approaches for learning representations suitable for text retrieval @xcite . more recently, several studies introduce dual-encoder architectures based on bert for dense retrieval @xcite @xcite . with the advent of llms with billions of parameters, several studies attempt to scale up model size @xcite , often finetuned on multi-task instruction data @xcite @xcite . however, the benefit of scaling up is more pronounced for general domain datasets where massive annotated data are available. to design effective retrievers for specialized domains, several works propose continuously pre-train the retrieval model on domain-specific corpora @xcite or fine-tuning the model on proprietary search datasets @xcite . on the other hand, synthetic data has also been used to improve the generalization ability of dense retrieval model @xcite @xcite . despite these advancements, how to combine public, open data to formulate a dataset curation recipe for adapting llms as high-performing biomedical retrievers remains unresolved. our method efficiently integrates diverse supervision signals for biomedical retrieval model training, which achieves better performance than baselines trained with more data. ## method bmretriever leverages the pre-trained autoregressive transformer as the backbone, taking advantage of the availability of various model sizes within this model family. this flexibility allows us to scale up the retrieval model. specifically, we utilize the publicly available autoregressive transformers with 410m, 1b, 2b, and 7b parameters @xcite @xcite . our model details are illustrated in ## conclusion we present bmretriever, a series of dense retrieval models designed for knowledge-intensive biomedical nlp tasks with various scales. bmre-triever is pre-trained on a large-scale biomedical corpus and further instruction fine-tuned on diverse, high-quality biomedical tasks. through extensive experimentation, we have demonstrated that bm-retriever exhibits state-of-the-art performance across a range of biomedical applications. furthermore, bmretriever demonstrates impressive parameter efficiency, with its smaller variants achieving 94-98% of the performance of the 7b model using only 6-29% as many parameters, while the 410m version surpasses larger baselines (1b-5b) up to 11.7 times larger. we hope bmretriever can be incorporated into a broad suite of biomedical tasks to advance biomedical nlp research.