A Continued Pretrained LLM Approach for Automatic Medical Note Generation (2024)

Abstract: LLMs are revolutionizing NLP tasks. However, the use of the most advanced LLMs, such as GPT-4, is often prohibitively expensive for most specialized fields. We introduce HEAL, the first continuously trained 13B LLaMA2-based LLM that is purpose-built for medical conversations and measured on automated scribing. Our results demonstrate that HEAL outperforms GPT-4 and PMC-LLaMA in PubMedQA, with an accuracy of 78.4%. It also achieves parity with GPT-4 in generating medical notes. Remarkably, HEAL surpasses GPT-4 and Med-PaLM 2 in identifying more correct medical concepts and exceeds the performance of human scribes and other comparable models in correctness and completeness.

## introduction the emergence of large language model (llm) has brought revolutionary changes to natural language processing and understanding tasks, paving the way for practical applications of ai across multiple domains such as law, finance, and healthcare. private llms such as @xcite and med-palm 2 @xcite and open-source llms like llama2 @xcite have shown strong performance on general nlp benchmarks. however, recent studies have shown promise that with continued training on more targeted datasets, e.g. smaller llms like orca @xcite and phi-2 (mojan javaheripi, 2023), can surpass much larger llms on general tasks. despite the success of llm in general capabilities, they often fall short in niche domains like healthcare, where precision and profound understanding are crucial. hence, several models such as meditron-70b @xcite , pmc-llama @xcite have emerged. transcribing medical conversations is a challenging task for both humans and machines due to po-1 *core contributors and corresponding authors tential transcription errors and the innate complexity of spoken language, an issue unaddressed by existing medical llms. existing llms trained on medical data largely do well on problems like medical q&a but struggle to produce a comprehensive ehr-compatible medical note. some domainadapted llms @xcite can write some components of the note, but they leave out the crucial "subjective" section. some fine-tuned models @xcite can generate notes from medical conversations but need human overview. overall, we developed a new medical llm proficient in interpreting medical conversation. by using techniques like continued pretraining on diverse data and explanation tuning, including medical and general web corpora, gpt-4 task instructions, ehrs, the model was capable of producing medical soap notes approved by physicians. our main contributions include: to the best of our knowledge, we are the first to build a small-size (13b) medical llm that can produce medical notes without any human intervention from doctor-patient conversations that bypass human quality and are accepted by physicians. heal surpasses med-palm 2 and other publicly available models of the same size, matches gpt-4's performance in medical notes generation, and excels with the highest completeness. despite having a smaller model size, we achieved an accuracy of 78.4% on pubmedqa, outperforming gpt-4 and within 5% of med-palm 2's performance. ## evaluations this section shows some of our continued pretraining results and evaluation methodology. ## conclusion this paper presents our work of developing a medical llm capable of comprehending and summarizing medical conversation. as a result, this is the first model, with significantly fewer parameters, to outperform humans, existing medical llms including med-palm 2, pmc-llama and perform on par with gpt-4. our evaluation shows that even small-scale continued pretraining of smaller llms can show impressive gains. we believe that scaling up our training can further improve results. our work presents a promising development in healthcare documentation and other medical areas.