Publicly Shareable Clinical Large Language Model Built on Synthetic Clinical Notes (2024)

Abstract: The development of large language models tailored for handling patients’ clinical notes is often hindered by the limited accessibility and usability of these notes due to strict privacy regulations.To address these challenges, we first create synthetic large-scale clinical notes using publicly available case reports extracted from biomedical literature.We then use these synthetic notes to train our specialized clinical large language model, Asclepius.While Asclepius is trained on synthetic data, we assess its potential performance in real-world applications by evaluating it using real clinical notes.We benchmark Asclepius against several other large language models, including GPT-3.5-turbo and other open-source alternatives. To further validate our approach using synthetic notes, we also compare Asclepius with its variants trained on real clinical notes. Our findings convincingly demonstrate that synthetic clinical notes can serve as viable substitutes for real ones when constructing high-performing clinical language models. This conclusion is supported by detailed evaluations conducted by both GPT-4 and medical professionals. All resources—including weights, codes, and data—used in the development of Asclepius will be made publicly accessible for future research.

## introduction clinical notes serve as an extensive repository of information specific to individual patients. applying natural language processing (nlp) techniques to these notes can significantly enhance the decision-making processes of medical professionals @xcite @xcite . recent advances in large figure 1 : the large clinical language model, asclepius, trained solely on synthetic clinical notes, can effectively handle various clinical nlp tasks on real notes in a zero-shot setting. the first challenge is privacy and security. hospitals must transmit sensitive patient information beyond their internal systems when using these api-based external llms. this could potentially infringe on privacy regulations. even when the external model adheres to regulations such as health insurance portability and accountability act (hipaa), hospitals should still undertake careful measures such as de-identifying clinical notes and setting up secure transmission protocols to avoid privacy breaches. this complicates the usage of external models. the second challenge relates to the autonomy that a health organization would need to exercise over its llms. given each organization's unique environment and characteristics, they may prefer a model specifically tailored to their needs. in light of these challenges, there is an increasing demand for a clinical llm that can operate securely in an offline environment while still offering the effectiveness of powerful online llms such as gpt series. to develop a specialized llm capable of handling clinical notes, a specific training dataset is required. this dataset would consist of instructionanswer pairs drawn from real clinical notes. creating such a dataset, however, introduces its own set of challenges. the first is the daunting task of acquiring clinical notes, which is almost impossible for external developers and even challenging for internal developers associated with a health organization due to privacy regulations. secondly, even when clinical notes are procured, creating a clinical instruction set necessitates either direct annotation from medical professionals or leveraging external models that have a strong understanding of clinical practices, such as the gpt series @xcite @xcite . the former approach is laborious and costly, making it impractical for large-scale use, while the latter approach presents the previously mentioned challenges related to privacy and security that are inherent to api-based models. to address these multifaceted challenges in the clinical settings, we introduce asclepius, a clinical llm constructed based on a comprehensive collection of synthetic clinical notes and corresponding instruction-answer pairs. these synthetic notes are generated from pmc-patients @xcite , containing anonymized case reports extracted from pubmed central, a publicly available biomedical literature archive. usage of synthetic notes, unlike real ones, not only enables us to leverage advanced online llms to produce comprehensive and highquality clinical instruction datasets, but also allow for the sharing of these resources and the models trained on them as open-source. throughout the entire process of generating these data, we utilized gpt-3.5-turbo, and medical professionals were involved in prompt tuning to ensure the output's clinical accuracy and relevancy. as a culmination of these efforts, we developed asclepius-7b and asclepius-13b, our advanced clinical llms capable of handling diverse clinical nlp tasks (see figure 1 ). we evaluated asclepius using a rigorous framework that aligns with its intended real-world appli-cations, utilizing real clinical notes as our primary evaluation dataset. for this evaluation, we gathered clinical notes from a diverse set of sources including mimic-iii @xcite , mimic-iv @xcite , i2b2 @xcite , casi @xcite , and mtsamples foot_0 , thereby ensuring a broad coverage of notes from various institutions. the first goal of our evaluation involved a comparison between our model and gpt-3.5-turbo. this comparison allowed us to assess asclepius's capability to perform at par with api-based llm across different clinical nlp tasks. additionally, we compared asclepius against a diverse array of open-source llms, including both general domain and clinical-biomedical domain models. this comparison aimed to validate our model's performance against other locally available llms. we were particularly interested in asclepius-r, a variant trained with real clinical notes. comparing asclepius with asclepius-r enabled us to assess the relative performance of models trained with synthetic notes against those trained with real ones. if a significant performance gap was found, it could possibly challenge our approach's validity. hence, this comparison emphasizes the effectiveness of our method in training a clinical llm using synthetic notes. in the overall evaluation process, we utilized gpt-4 as an evaluator, which is known to have advanced medical knowledge, to assess the models' performance. furthermore, for the crucial comparison between asclepius and asclepius-r, four clinicians were involved in the evaluation to substantiate our claim. our key contributions can be summarized as follows: a 20-year-old myanmarese woma n who was aware of a declining vi sion in her left eye for three years was diagnosed with a mature cata ract in her left eye. ... and the postoperative course was uneventful with a recovery of the l eft vision to 20/200. ... a macula involved detachment wa s confirmed by optical coherence t omography. ... the vision in her left eye improved to 20/60 and was stable for 19 mo nths after the second surgery with out showing any worsening of the retinal proliferation or detachment. patient name: n/a medical record number: n/a date of admission: n/a date of discharge: n/a ... diagnosis: -mature cataract in left eye -retinal detachment in left eye ... outcomes: -recovery of vision to 20/200 after initial surgery -improvement of vision to 20/60 after second surgery -stability of vision in left eye for 19 months after second surgery without worsening of retinal detachment what were the diagnosis and outcomes related to the left eye in the patient's hospital course, based on the provided discharge summary? the patient's left eye was diagnosed with a mature cataract and later developed retinal detachment. the outcomes showed recovery of vision to 20/200 after the initial surgery, improvement of vision to 20/60 after the second surgery, and stability of vision in the left eye for 19 months after the second surgery without worsening of the retinal detachment. admission date: [**2118-8-10**] discharge date: [**2118-8-12**] date of birth: [**2073-12-25**] sex: f ... discharge diagnosis: avm radionecrosis ... discharge instructions: -discharge instructions for craniotomy/head inj ury -have a family member check y our incision daily for signs of inf ection -take your pain medicine as pre scribed case report real clinilcal note synthetic clinical note instruction answer similar format figure 2 : the first column is a part of the real discharge summary from mimic-iii @xcite . second is a case report from pmc-patients @xcite , and the third is the synthetic discharge summary created from this case report. initially, the case report did not resemble the real clinical note in terms of format, but after the transformation, it more closely resembles the real clinical note. at the last column, there is an instruction and answer pair generated from synthetic clinical note. gpt-3.5-turbo was used in all generation processes. by leveraging our methodology, any entityfrom healthcare organizations to individual researchers -can develop an llm capable of understanding and interacting with clinical notes. this breakthrough will serve as a crucial steppingstone to accelerate the research and development of healthcare ai, which has been previously deterred by stringent (yet essential) privacy regulations. ## data generation in section 2.1, we discuss the differentiation between clinical notes and case reports and detail how to convert case reports into synthetic clinical notes. section 2.2 delves into extracting specific instruction-answer pairs from these notes for training the clinical llm. figure 2 illustrates this process with an accompanying example. it is important to note our method solely uses public data, allowing unrestricted use of llm (gpt-3.5-turbo). all prompts utilized are listed in appendix a. in this research, we specifically focus on the discharge summary, a specific type of clinical note that is extensively used in a variety of clinical tasks. henceforth in this paper, the term clinical note will specifically refer to the discharge summary. ## comparative analysis we conduct a comparative study to analyze the performance of asclepius against several others using gpt-4 evaluation specified in section 3.2. our initial point of comparison is gpt-3.5-turbo, wherein we aim to ascertain whether asclepius can match the performance and versatility of apibased llm in various clinical nlp tasks. we also include an evaluation of other open-source instruction fine-tuned llms that are trained on general domain data, such as alpaca @xcite and vicuna @xcite , and those tailored for the clinical-biomedical domain, such as medalpaca @xcite , chatdoctor @xcite , and clinical-camel @xcite . including these models aims to compare asclepius's performance on clinical nlp tasks with other locally available models, thereby validating our methodology in developing our model. lastly, we include asclepius-r, a variant of our model, trained using 57k real clinical notes from the mimic-iii dataset @xcite 3 . asclepius-r, having been both pre-trained and finetuned on these real notes, is directly compared to asclepius, our model trained on 158k synthetic notes. this comparison allows us to explore the performance of models developed with synthetic data in relation to those trained with real data. it is important to clarify that our objective is not to claim that synthetic notes can completely replace real ones, but rather to show that a model trained on synthetic notes can be a viable alternative to one trained on real data. to optimize their performances, we leveraged the maximum amount of data available for each model. the performances of asclepius and asclepius-r, when trained on datasets of the same size, are detailed in the ablation study in appendix f. by including asclepius-r in our analysis, we can thoroughly assess the potential of our method in training large language models using synthetic clinical notes instead of real ones. ## conclusion in this paper, we present asclepius, trained on 158k high-quality synthetic clinical notes and instruction sets. evaluations across diverse benchmark datasets against other llms demonstrate that asclepius performs on par with gpt-3.5-turbo while locally executable in hospital settings. most importantly, it exhibits no significant disparity with models trained on actual clinical notes, thereby validating the use of synthetic notes for training clinical large language models. the evaluations were not solely reliant on gpt-4 but also involved appraisals by four clinicians, reinforcing the validity of our conclusions. for future research, all synthetic data, model weights, and code used in these experiments are publicly available. this opens the door for not only healthcare institutions but also businesses and researchers to develop clinical large language models. we believe this has the potential to significantly advance healthcare ai, especially in areas previously held back by privacy concerns.