W ang L ab at MEDIQA -Chat 2023: Clinical Note Generation from Doctor-Patient Conversations using Large Language Models (2023)

Abstract: This paper describes our submission to the MEDIQA-Chat 2023 shared task for automatic clinical note generation from doctor-patient conversations. We report results for two approaches: the first fine-tunes a pre-trained language model (PLM) on the shared task data, and the second uses few-shot in-context learning (ICL) with a large language model (LLM). Both achieve high performance as measured by automatic metrics (e.g. ROUGE, BERTScore) and ranked second and first, respectively, of all submissions to the shared task. Expert human scrutiny indicates that notes generated via the ICL-based approach with GPT-4 are preferred about as often as human-written notes, making it a promising path toward automated note generation from doctor-patient conversations.

## introduction the growing burden of clinical documentation has emerged as a critical issue in healthcare, increasing job dissatisfaction and burnout rates among clinicians and negatively impacting patient experiences @xcite @xcite . on the other hand, timely and accurate documentation of patient encounters is critical for safe, effective care and communication between specialists. therefore, interest in assisting clinicians by automatically generating consultation notes is mounting @xcite @xcite . to further encourage research on automatic clinical note generation from doctor-patient conversations, the mediqa-chat dialogue2note shared task was proposed @xcite . here, we describe our submission to subtask b: the generation of full clinical notes from doctor-patient dialogues. we explored two approaches; the first finetunes a pre-trained language model (plm, §3.1), we rank train examples based on their similarity to the test dialogue using instructor @xcite . notes of the top-k most similar examples are then used as in-context examples to form a prompt alongside natural language instructions and fed to @xcite to generate the clinical note. while the second uses few-shot in-context learning (icl, §3.2). both achieve high performance as measured by automatic natural language generation metrics ( §4) and ranked second and first, respectively, of all submissions to the shared task. in a human evaluation with three expert physicians, notes generated via the icl-based approach with gpt-4 were preferred about as often as humanwritten notes ( §4.3). ## approach we take two high-performant approaches to the shared task. in the first, we fine-tune a pre-trained language model (plm) on the provided training set ( §3.1). in the second, we use in-context learning (icl) with a large language model (llm, §3.2). ## related work automated note generation from doctor-patient conversations has received increasing attention in recent years @xcite @xcite . different methods have been proposed, such as extractiveabstractive approaches @xcite @xcite and fine-tuning plms (zhang et al. 2021, similar to our approach in §3.1). others have focused on curating data for training and benchmarking @xcite , including the use of llms to produce synthetic data @xcite . lastly, there have been efforts to improve the evaluation of generated clinical notes, both with automatic metrics @xcite and human evaluation @xcite . while recent literature has commented on the potential of icl for note generation @xcite , our work is among the first to evaluate this approach rigorously. ## conclusion we present our submission to the mediqa-chat shared task for clinical note generation from doctorpatient dialogues. we evaluated a fine-tuning-based approach with led and an icl-based approach with gpt-4, ranking second and first, respectively, among all submissions. human evaluation with three physicians revealed that notes produced by gpt-4 via icl were strongly preferred over notes produced by led and, on average, slightly preferred over human-written notes. we conclude that icl is a promising path toward clinical note generation from doctor-patient conversations.