Rephrasing Electronic Health Records for Pretraining Clinical Language Models (2024)

Abstract: Clinical language models are important for many applications in healthcare, but their development depends on access to extensive clinical text for pretraining. However, obtaining clinical notes from electronic health records (EHRs) at scale is challenging due to patient privacy concerns. In this study, we rephrase existing clinical notes using LLMs to generate synthetic pretraining corpora, drawing inspiration from previous work on rephrasing web data. We examine four popular small-sized LLMs (<10B) to create synthetic clinical text to pretrain both decoder-based and encoder-based language models. The method yields better results in language modeling and downstream tasks than previous synthesis approaches without referencing real clinical text. We find that augmenting original clinical notes with synthetic corpora from different LLMs improves performances even at a small token budget, showing the potential of this method to support pretraining at the institutional level or be scaled to synthesize large-scale clinical corpora.

## introduction language models have emerged as crucial components in nlp systems applied in healthcare, offering potential benefits for clinical decision support @xcite , predictive analytics @xcite , and resource allocation @xcite . many of these applications require models to be adapted to the clinical domain through pretraining to achieve optimal performance @xcite @xcite . however, the privacy and compliance regulations around electronic health records (ehrs) make it challenging to obtain clinical notes at a scale suitable for pretraining. while individual healthcare systems may train models on their own ehr data @xcite , this is only feasible for large institutions and prohibits the sharing of these models. these factors hinder the advancement of research on developing more effective language models in healthcare. to address this data scarcity issue, synthetic data has been examined for various clinical tasks @xcite @xcite . however, existing methods are mostly task-specific or focus on a particular application. one recent study attempted to create clinical pretraining corpora by prompting chatgpt to synthesize discharge summaries based on patient profiles curated from the medical literature @xcite . while this approach enables creating synthetic clinical notes at scale and supports pretraining publicly sharable llms (denoted as asclepius), it relies heavily on the knowledge of the llm to enrich the clinical details. generating complex clinical text from scratch may suffer from llm hallucinations and limit the quality of the generated clinical notes. this study proposes an alternative approach by rephrasing real clinical notes using llms to create clinical pretraining corpora. we draw inspiration from a recent study that demonstrates the benefit of rephrasing internet corpora (e.g., c4) to pretrain general-domain language models @xcite . we explore a similar strategy by prompting llms to rephrase ehr data, expanding the analysis to include medically adapted prompts, diverse llm types, and combinations of synthetic corpora. our experiments show that the rephrasing method significantly reduces the perplexity of causal language modeling compared to synthesis methods in previous works. furthermore, combining synthetic notes with real clinical notes can effectively improve language modeling performance. we find that a medically adapted prompt performs similarly to a general prompt, but explicitly asking llms to additionally use their knowledge to explain clinical information can have mixed results. we also pretrain masked language models for downstream fine-tuning. the resulting model outperforms the widely used clinicalbert, demonstrating the potential of the rephrasing approach in developing performant clinical language models. ## rephrasing clinical notes with llms we prompt various llms to rephrase clinical notes and leverage the generated content to pretrain clinically adapted models. we explore both decoderbased and encoder-based language models, as described in section 3 and 4, respectively. ## perplexity evaluation with causal language models this section explores the effectiveness of the rephrasing method by evaluating the perplexity scores of decoder-based language models pretrained on synthetic data generated from different llms and prompts. ascplepius refers the synthetic notes from @xcite . the four llms refer to their synthetic corpora based on the rephrasing method, respectively. lower perplexity means better language modeling performances. ## downstream evaluation with masked language models besides decoders, we pretrain encoder-based language models using both real and synthetic clinical notes, and fine-tune them for downstream clinical nlp tasks. this scenario simulates the real-world situation where a healthcare institution aims to train its own language models but lacks sufficient ehr data approved for this purpose. ## discussion results from both decoder-and encoder-based pretraining demonstrate the strength of our rephrasing method to create high-quality clinical text using small-sized llms. however, in this study, we mainly focused on the quantitative analysis through evaluating downstream pretrained models. qualitative analysis is necessary to better understand the quality of the rephrased notes. we provide some examples from the four llms rephrasing the same chunk in appendix a, but since in our initial implementation we did not keep the indices of the generated outputs that correspond to the original text, we could not provide rephrasings for all text chunks. we leave this to future work, where we aim to release the rephrased clinical notes at a larger scale for further investigation. a deeper comparison between the rephrased and real notes is needed in the future to elucidate how much content is retained by llms and how rephrasing changes the clinical narrative. in particular, we need to understand whether llms' rephrasing causes subtle shifts in clinical meaning and the extent of possible hallucinations. practically, we could measure how and when the rephrased text aligns or diverges with real text. we can approach how they align or diverge by comparing syntactic and semantic features @xcite , such as extracting and comparing distributions of medical concepts, and we could measure when they align or diverge by further examining the impact of prompt and decoding setup on conceptual shift. meanwhile, there are more nuances when we consider the subjective components of clinical text as narratives by the clinician @xcite , where personal opinions and documentation practices vary from person to person. these are more intricate and challenging to measure, but are essential for the implementation of reliable and safe models in practice @xcite . exploring whether llms reduce or amplify biases @xcite and how they handle duplicated contents such as copy-and-pasted text @xcite in their rephrasing would be important future directions. ## conclusion we demonstrate the effectiveness of llm rephrasing to create pretraining corpora for clinical language models. future work can scale the genera-tion and incorporate other types of clinical notes to develop stronger models for clinical applications.