Beyond Fine-tuning: Unleashing the Potential of Continuous Pretraining for Clinical LLM s. (2024)

Abstract: Large Language Models (LLMs) have demonstrated significant potential in revolutionizing clinical applications. In this study, we investigate the efficacy of four techniques in adapting LLMs for clinical use-cases: continuous pretraining, instruct fine-tuning, NEFTune, and prompt engineering. We employ these methods on Mistral 7B and Mixtral 8x7B models, leveraging a large-scale clinical pretraining dataset of 50 billion tokens and an instruct fine-tuning dataset of 500 million tokens. Our evaluation across various clinical tasks reveals nuanced insights. While continuous pretraining beyond 250 billion tokens yields marginal improvements, instruct fine-tuning emerges as a more influential factor. Notably, NEFTune, designed primarily to enhance generation quality, surprisingly demonstrates additional gains on our benchmark. These findings underscore the importance of tailoring fine-tuning strategies and exploring innovative techniques to optimize LLM performance in the clinical domain.

## introduction the advent of large language models (llms) has spurred a wave of innovation across various domains, with healthcare being a particularly promising area for their application. llms have the potential to transform clinical workflows, aid in diagnosis, and enhance patient care. however, effectively adapting these models to the nuances and complexities of the clinical domain remains a significant challenge. current approaches in the literature predominantly focus on either developing specialized clinical llms from scratch or fine-tuning existing models on large-scale clinical datasets. while these methods have shown promise, they often overlook the potential benefits of continuous pretraining on domain-specific data as a means to further enhance model performance. this is due in part to the complexities and potential instabilities associated with continued training of large models. in this study, we take a comprehensive approach to optimizing clinical llms by systematically investigating the impact of continuous pretraining on in-domain data, in conjunction with instruct fine-tuning and advanced prompting strategies. we focus on the mistral-7b @xcite and @xcite ) models, demonstrating that continuous pretraining, while yielding modest gains compared to fine-tuning and prompting, plays a crucial role in establishing a solid foundation for further specialization. by carefully balancing in-domain clinical data with general language data, we successfully mitigate instability issues and unlock the full potential of continuous pretraining for clinical llms. our work highlights the importance of understanding of the relationship between pretraining, fine-tuning, and prompting in adapting llms for clinical applications. by demonstrating the effectiveness of continuous pretraining on domainspecific data, we open doors for future research to further explore this underutilized technique to develop more accurate, reliable, and ultimately impactful clinical llms. ## related works the landscape of large language models (llms) for healthcare is evolving rapidly, with most approaches involving either domain-specific pretraining or instruction fine-tuning of general-purpose models. openai's @xcite , alongside google's med-palm @xcite and med-palm 2 @xcite have demonstrated impressive performance on medical benchmarks, despite limited transparency regarding their training details. other models, such as gatortron @xcite , and pmc-llama @xcite , have shown the potential of pretraining on extensive biomedical corpora to add domain-specific knowledge for clinical applications. instruction fine-tuning and dialogue datasets have also been instrumental in enhancing the zeroshot and few-shot generalization capabilities of llms. chatdoctor @xcite and medalpaca @xcite , for instance, utilize medical conversations and other nlp tasks to improve llama's performance on clinical queries. recent models like clinical camel @xcite , meditron @xcite , huatuogpt-2 @xcite and med42 @xcite , based on llama-2 @xcite , further demonstrate the efficacy of this approach. building on the observation that models can learn from prompting alone @xcite , recent research has explored techniques to enhance clinical capabilities without additional training. these methods often extend the well-known chainof-thought prompting technique, originally introduced by @xcite , to better suit clinical use-cases. notably, microsoft's medprompt @xcite demonstrates significant improvements in gpt-4's performance on clinical qa tasks, while @xcite apply similar strategies to the yi family of models @xcite . google has also showcased the potential of complex prompting to boost the clinical capabilities of their gemini model @xcite . however, while such complex prompting techniques can improve performance on standard benchmarks, their practicality and scalability in real-world clinical applications remain to be seen. recent studies like lima @xcite , fineweb @xcite and phi @xcite have highlighted the pivotal role of data quality in llm training, emphasizing that it can often be more influential than architectural choices in determining model performance. high-quality data has been shown to significantly impact the model's ability to learn meaningful representations and generalize to new tasks. this shows the importance of our approach to dataset curation, ensuring that our models are trained on a robust and representative collection of clinical data. ## experiments in this section, we present the four steps of our experimental framework: (1) continuous pretrain-ing, (2) instruct fine-tuning, (3) neftune, and (4) complex prompt engineering. ## datasets in this section, we detail our approach to constructing both the pretraining and fine-tuning datasets. our primary objective is to curate datasets that optimize model performance while maintaining the highest standards of quality and relevance to the clinical domain. ## evaluations to rigorously assess the efficacy of our fine-tuning approaches, we focus on a comprehensive evaluation of the models' capabilities across a spectrum of clinical question-answering (qa) tasks. we employ a diverse suite of qa datasets, including medqa @xcite , usmle sample exam and self-assessment @xcite , mmlu (medical subset) @xcite , and medmcqa @xcite , to ensure a thorough and representative assessment of model performance in various clinical scenarios. our evaluation methodology uses the eleutherai harness framework @xcite , which focuses on the likelihood of a model generating each proposed answer rather than directly evaluating the generated text itself. to enhance the granularity and relevance of our analysis, we introduce modifications to the harness codebase. instead of computing the likelihood of generating only the answer choice labels (a, b, c, or d), we extend the computation to encompass the likelihood of generating the complete answer text. this modification provides a more detailed understanding of the model's performance, as it takes into account the entire answer generation process, including the ability to articulate reasoning and justify the selected answer choice. to evaluate the efficacy of medprompt prompting strategies, we integrate these prompts into the harness framework. this involves generating reasoning chains based on the prompts and then using harness to assess the likelihood of the model producing the final answer derived from these chains. this approach allows us to evaluate the impact of specific prompting techniques on the model's ability to reason through complex clinical scenarios. throughout our evaluation, we report accuracy as the primary metric across all tables, providing a clear and interpretable measure of the models' proficiency in clinical qa tasks. ## results in this section, we present the results of our experiments, revealing key insights into the effectiveness of different training approaches for clinical language models. non-instructed models can't be evaluated on qa tasks. throughout our continuous pretraining process, we saved multiple checkpoints and assessed their performance on our clinical qa benchmarks. given the absence of instruction fine-tuning, we opted for a no-prompt evaluation format. as illustrated in figure 2 , a slight performance decline is observed between the base model (with no pretraining) and the initial checkpoints. while subsequent checkpoints exhibit gradual improvement with increased exposure to clinical data, their performance consistently trails behind the original base model. this observation underscores the critical role of instruction fine-tuning in equipping llms with the necessary skills to effectively comprehend and respond to questions in the clinical domain. instruct fine-tuning specializes the model for qa data the remarkable leap in performance observed in ## conclusion and discussions in this study, we have systematically investigated the impact of continuous pretraining on in-domain clinical data, in conjunction with instruct finetuning and advanced prompting strategies, on the performance of llms in clinical questionanswering tasks. our findings demonstrate that continuous pretraining, while yielding modest improvements compared to other techniques, remains a valuable tool for enhancing llm performance in the clinical domain. while continuous pretrain-ing can often be challenging due to instability issues, we have shown that by carefully balancing in-domain clinical data with general language data, we can effectively mitigate these challenges and achieve consistent performance gains. furthermore, we have demonstrated that the benefits of continuous pretraining extend beyond the initial training phase, as it lays a solid foundation for subsequent instruct fine-tuning and the application of complex prompting techniques like med-prompt. the synergy between continuous pretraining and these additional methods results in stateof-the-art performance on a variety of clinical qa benchmarks, outperforming existing models like the instruct-tuned versions of mistral and mixtral. our research opens up several avenues for future exploration. further ablation studies could examine the effects of different domain data sources, beyond the clinical realm, on llm performance. additionally, a more comprehensive analysis of the optimal data mix for continuous pretraining, including varying proportions of in-domain and general language data, could yield valuable insights for maximizing the benefits of this technique. this study provides a comprehensive framework for optimizing clinical llm performance. our findings offer valuable insights for future research and development efforts aimed at leveraging llms to address challenges and opportunities presented by the healthcare domain.