A Privacy-preserving Approach to Ingest Knowledge from Proprietary Web-based to Locally Run Models for Medical Progress Note Generation (2024)

Abstract: Clinical documentation is correlated with increasing clinician burden, leading to the rise of automated methods to generate medical notes. Due to the sensitive nature of patient electronic health records (EHRs), locally run models are preferred for a variety of reasons including privacy, bias, and cost. However, most open-source locally run models (including medical-specific) are much smaller with limited input context size compared to the more powerful closed-source large language models (LLMs) generally available through web APIs (Application Programming Interfaces). In this paper, we propose a framework to harness superior reasoning capabilities and medical knowledge from closed-source online LLMs in a privacy-preserving manner and seamlessly incorporate it into locally run models. Specifically, we leverage a web-based model to distill the vast patient information available in EHRs into a clinically relevant subset without sending sensitive patient health information online and use this distilled knowledge to generate progress notes by a locally run model. Our ablation results indicate that the proposed framework improves the performance of the Mixtral model on progress note generation by 4.6 points on ROUGE (a text-matching based metric) and 7.56 points on MEDCON F1 (a metric that measures the clinical concepts overlap).

## introduction physicians document progress or soap (subjective, objective, assessment, and plan) notes in electronic health records (ehrs) periodically to document patient care journey. while abundant patient chart data (e.g., regularly collected lab values) enhances physician assessment of patient progress, it leads to information overload and clinician burden, giving rise to clinician burnout @xcite , emphasizing the importance of automating this task. the increasing popularity and capabilities of large language models (llms) led to their numer-ous applications in both general and medical domains @xcite . while the closed-source llms available via web apis (application programming interfaces) generally outperform the locally run alternatives, there is a growing popularity and community support for on-premise models, especially in the medical domain because of several advantages that these models offer such as transparency, adaptability, and information security @xcite . we propose to reap the benefits offered by locally run models while harnessing the strong reasoning capabilities of api-based proprietary llms. to this end, there have been numerous efforts toward distilling knowledge from proprietary llms (e.g., gpt-4) to train smaller or locally run models @xcite . in the medical domain, most work on such distillation has focused on curating instruction-tuning datasets using superior llms for training or tuning smaller models @xcite @xcite . differently, our framework exploits web-based llms for achieving a bottleneck task for locally run models formulated in a way that does not spill sensitive patient information to online api-based models. we formulate the task of progress note generation (png) to automatically generate the next note given a patient's prior progress note and all interim structured chart data (e.g., vital signs). one of the main limitations of the locally run models in tackling png is processing and clinically analyzing the vast amount of interim structured chart data (an average of over 1400 rows of tabular data between any pair of subsequent progress notes) -the bottleneck. to overcome this barrier, we leverage an advanced api-based proprietary model to choose clinically relevant structured data rows without sending any real patient information to the online model server. this distilled structured chart information, along with the prior progress note, is used by a locally run model to generate the next progress note. ## results the performance measures in automatically generating progress notes are shown in ## discussion our results indicate the advantage of tackling the task of png by considering individual component problems at a time and leveraging advanced web-based models to transfer knowledge by filtering relevant clinical attributes in structured chart data. our manual evaluation suggests the predicted notes capture the important updates on patient's progress. importantly, mixtral exhibited capabilities in capturing overall status changes (e.g., sepsis improving), whereas the biomistral demonstrated its ability to capture domain knowledge-related updates (e.g., add digoxin 0.25mg daily). finetuning llms leads to specialized domain knowledge (as exhibited by biomistral), however, it is also shown to reduce general in-context learning abilities @xcite , as seen in overall, the findings from this paper provide support for the feasibility of the complex task of png. further, it provides a framework for harnessing the reasoning capabilities of proprietary api-based models in a privacy-preserving manner while using a locally run model for handling sensitive patient information. ## limitations the limitations of our framework include its inability to capture new problems that may have emerged in the interval, which is an interesting avenue for future research. moreover, physicians use information beyond the structured chart data while writing progress notes, e.g., radiology reports. as described earlier, it is challenging to incorporate the interim structured data along with the previous note text in the limited context size of existing onpremise models. thus, we leave the inclusion of other information sources to future work.