Towards Adapting Open-Source Large Language Models for Expert-Level Clinical Note Generation (2025)

Abstract: Proprietary Large Language Models (LLMs) such as GPT-4 and Gemini have demonstrated promising capabilities in clinical text summarization tasks. However, due to patient data privacy concerns and computational costs, many healthcare providers prefer using small, locally-hosted models over external generic LLMs. This study presents a comprehensive domain- and task-specific adaptation process for the open-source LLaMA-2 13 billion parameter model, enabling it to generate high-quality clinical notes from outpatient patient-doctor dialogues. Our process incorporates continued pre-training, supervised fine-tuning, and reinforcement learning from both AI and human feedback. We introduced a new approach, DistillDirect, for performing on-policy reinforcement learning with Gemini 1.0 Pro as the teacher model. Our resulting model, LLaMA-Clinic, can generate clinical notes comparable in quality to those authored by physicians. In a blinded physician reader study, the majority (92.8%) of individual evaluations rated the notes generated by LLaMA-Clinic as “acceptable” or higher across all three criteria: real-world readiness, completeness, and accuracy. In the more challenging “Assessment and Plan” section, LLaMA-Clinic received the same score as the notes authored by physicians. We highlight key considerations for future clinical note-generation tasks, emphasizing the importance of pre-defining a best-practice note format, rather than relying on LLMs to determine this for clinical practice.

## introduction recent advancements in llms have transformed the field of natural language processing (nlp). however, the application of llms in the medical domain is still in its early stages @xcite . proprietary llms, such as gpt-4 and med-palm, have demonstrated impressive capabilities in medical knowledge and clinical nlp tasks @xcite @xcite . however, most proprietary llms have limited flexibility for domain-specific fine-tuning, primarily due to restricted access to model weights. additionally, proprietary llms raise several concerns pertinent to the healthcare sector, including hipaa compliance, data security, cost, and transparency of training data @xcite @xcite . the emergence of powerful open-source llms has opened up opportunities for domain-specific fine-tuning within the clinical field, yielding promising results @xcite @xcite @xcite . however, most research on open-source models has concentrated on medical knowledge injection rather than practical applications in real-world clinical workflow such as clinical note generation. in this work, we address a practical question clinicians face in their everyday routine: how can we best adapt an open-source llm for the specific use case of clinical note generation? clinical note documentation represents a significant burden for healthcare practitioners @xcite and appears to be a natural application for llms, given their remarkable generative capabilities @xcite . recent research on llms for clinical text summarization found that llm-generated outputs are preferred over human summaries for their completeness and accuracy @xcite . the 2023 acl clinicalnlp and clefimage workshops explored the generation of clinical notes from patient-doctor conversations using the newly released ambient clinical intelligence benchmark (aci-bench) dataset @xcite . the most notable results were achieved using gpt-4 along with few-shot incontext learning @xcite . however, these initial exploratory studies for clinical note generation with llms revealed significant limitations: in this study, we revisit the task of outpatient note generation, focusing on adapting an opensource llm-the llama-2 13 billion parameter model. we thoroughly evaluated techniques for domain-and task-specific adaptation, ranging from continued pretraining and sft to reinforcement learning informed from both ai and human feedback (figure 1 ). this work makes several specific contributions: ## discussion our methodology for adapting llms to clinical note generation revealed several key considerations. we initiated this process by establishing a "best practice" note format informed by a consensus among our physician authors. this approach tackles the variability in the style, format, and quality of physician notes, which could otherwise compromise the training of llms. instead of relying on the llm to identify the most effective note structure, we advocate for healthcare providers to set these standards themselves. it is important to acknowledge that the "best practice" note format employed in this study reflects the consensus of a group of internal medicine physicians within a single organization. this format may not be generalizable to other specialties or other organizations, such as orthopedic surgery. furthermore, we examined a workflow in which providers critically review and refine ai-generated notes prior to their filing. under such a workflow, physician preferences may shift toward notes requiring fewer revisions and edits, with a higher tolerance for minor inaccuracies. ## conclusions our research highlights the potential of training an open-source llm for outpatient note generation, showing strong promise for real-world clinical applications. healthcare institutions are in a privileged position to undertake such endeavors, given their access to extensive emr data and a wealth of domain expertise critical for implementing rlhf. our work was based on fewer than 1,500 patient-doctor dialogues and limited physician preference data. when implementing a similar project in a healthcare institution, the training data could be scaled up significantly, potentially leading to further performance improvements. lastly, the prospect of extending this work to other clinical note-generation tasks, such as creating discharge summaries for hospitalized patients, is particularly exciting. 12091 this research was supported by nsf awards sch-2205289. the funder played no role in the study design, data collection, analysis, and interpretation of data, or the writing of this manuscript. this research is also supported by the protected research time grant from the southwest minnesota region of the mayo clinic health system.