G erstein L ab at MEDIQA -Chat 2023: Clinical Note Summarization from Doctor-Patient Conversations through Fine-tuning and In-context Learning (2023)

Abstract: This paper presents our contribution to the MEDIQA-2023 Dialogue2Note shared task, encompassing both subtask A and subtask B. We approach the task as a dialogue summarization problem and implement two distinct pipelines: (a) a fine-tuning of a pre-trained dialogue summarization model and GPT-3, and (b) few-shot in-context learning (ICL) using a large language model, GPT-4. Both methods achieve excellent results in terms of ROUGE-1 F1, BERTScore F1 (deberta-xlarge-mnli), and BLEURT, with scores of 0.4011, 0.7058, and 0.5421, respectively. Additionally, we predict the associated section headers using RoBERTa and SciBERT based classification models. Our team ranked fourth among all teams, while each team is allowed to submit three runs as part of their submission. We also utilize expert annotations to demonstrate that the notes generated through the ICL GPT-4 are better than all other baselines. The code for our submission is available.

## introduction the field of medical ai has witnessed significant advancements in recent years, fueled by its promise to transform clinical documentation procedures @xcite @xcite @xcite . extracting clinical notes from doctor-patient interactions is a crucial aspect of maintaining medical records, as it fosters effective communication among healthcare practitioners. by automating this process, healthcare professionals can shift their focus toward patient care and minimize the time dedicated to administrative duties @xcite . the development of efficient and accurate algorithms for summarizing these conversational notes is therefore of paramount importance, as it has the poten-1 https://github.com/gersteinlab/mediqa-chat-2023 tial to improve overall healthcare quality and efficiency @xcite @xcite @xcite . the mediqa-chat 2023 challenge foot_0 was established to promote the development of novel summarization techniques, specifically targeting the automatic generation of clinical notes from doctor-patient conversations @xcite . the dialogue2note and note2dialogue shared tasks are designed to stimulate research and innovation in this field, addressing the summarization of medical conversations for clinical note creation and the generation of synthetic doctor-patient conversations for data creation and augmentation. the dialogue2note summarization task entails converting a doctor-patient conversation into a clinical note containing one or multiple note sections, such as assessment, past medical history, or past surgical history. this task is subdivided into two subtasks: (a) generating specific sections from conversations @xcite , and (b) generating complete notes from conversations @xcite , two examples are shown in figure. 1. in this paper, we discuss our submission to both subtask a and subtask b of the shared task: for subtask a, we first focused on section classification and explored two methods: (1) using roberta @xcite and @xcite with a classification head, and (2) fine-tuning openai's davinci model foot_1 . subsequently, we investigated generating specific sections using a fine-tuned pre-trained dialogue summarization model. we employed the confit model @xcite , which proposes a training strategy that enhances the factual consistency and overall quality of summaries through a doctor: good afternoon, which gender do you identify with? patient: good afternoon, doctor, thank you for asking, i identify as a female. doctor: great, thank you. have you ever had surgery on this knee? patient: yes, i actually had an a c l reconstruction done in march of two thousand eight. um, it didn't go well, so they did a revision at the end of that year, in december. doctor: i see, thank you. what about your upper body? is there any history of surgery there? patient: actually, yes, i had surgery on my arm when i was six. section: past surgical she had an anterior cruciate ligament reconstruction (acl) in 03/2008, and subsequently had a revision acl reconstruction in 12/2008. she also had arm surgery when she was 6 years old. chief complaint: smoking cessation. medical history: patient reports a history of type 2 diabetes, gout, and a 2/6 systolic ejection murmur. social history: patient reports he is a smoker. medications: patient reports taking allopurinol. vitals: oxygen saturation: 98% on room air. blood pressure: 128/88 mmhg. heart rate: 68 beats per minute. respiratory rate: 16 breaths per minute. physical exam: neck general examination: neck is supple without lymphadenopathy. no carotid bruits. ...... assessment and plan ...... instructions ...... ## methodology in this section, we will describe how we employed various approaches to perform subtask a and subtask b. we will discuss each subtask in detail, outlining the methods used and the rationale behind our choices to achieve optimal results. ## results we report the accuracy of the section classification. for note generation in subtask a, we present the following metrics: rouge-1, rouge-2, rouge-l, rouge-lsum, bertscore precision @xcite , bertscore recall @xcite , bertscore f1 @xcite , bleurt @xcite , and aggregate score. the aggregate score is the arithmetic mean of rouge-1 f1, bertscore f1, and @xcite . additionally, we implemented a dynamic max length feature.for the confit model and the confit model with dynamic max length, we obtained aggregate scores of 0.5392 and 0.5497, respectively. the results demonstrate that the dynamic max length approach slightly improves the overall performance of the model in the summarization task. initially, it encodes the input dialogue into a vector using a pre-trained tokenizer. subsequently, it dynamically calculates the maximum length of the summary to be generated, based on 5 https://platform.openai.com/docs/models/gpt-4 ## limitations and discussion we would like to discuss some of our findings and thoughts here. the mediqa-chat 2023 challenge indeed presents an excellent opportunity for us to reflect on existing models, analyze their strengths and weaknesses, and investigate their performance on the task of clinical note generation from doctor-patient conversations. evaluation automatic metrics often do not have a strong correlation with the quality of summaries, especially as there are no readily available automatic evaluation metrics specifically designed for zero-shot or few-shot llm evaluation. almost all models on subtask a have similar results of automatic evaluation. therefore, human evaluation becomes an essential component for accurately assessing the quality of notes generated by llms. there is a need to develop more effective automatic evaluation methods for a zero-shot/few-shot generation or super-long output. length limitation of subtask a one significant limitation of our transformer-based model is that it does not directly consider length during its generation process. this often results in the production of overly verbose summaries. in contrast, models like openai's gpt-3 and gpt-4 have much longer input and output limitations, allowing them to handle more extensive text samples more effectively. it is worth noting that our model was trained on the samsum dataset, which has longer texts compared to subtask a. consequently, our model struggles to adapt to the shorter length requirements of subtask a. moreover, the training dataset for subtask a is relatively small, which further complicates the model's adaptation. future exploration should look at how to constrain the conciseness of generated summaries, which may involve reconsidering the generation method chosen or examining other techniques to promote brevity. developing methods to better control the length of generated summaries is essential to improve their relevance, coherence, and usability in real-world applications. length limitation of subtask b for subtask b, it is challenging to achieve reasonable results using fine-tuned models. in reality, this task is more representative of real-world scenarios, where inputs and outputs are considerably long, and the output is expected to maintain a specific structure and format. thus, we see the main advantage of using contextual examples lies in their ability to guide the structure, style, and length of the desired output. we believe that openai's llm is wellsuited for similar real-life scenarios, provided that it is given an appropriate context. in such cases, its performance will significantly surpass that of fine-tuned transformer-based models. factual inconsistentcy while our study did not specifically investigate the following issues, we noted several factual errors that occur in summaries. a previous study has shown that llms also exhibit a noticeable occurrence of attribute errors and misinterpretation errors @xcite . prompting we find that prompt template and demonstration example selection both have a substantial impact on results. using more prompt examples for demonstration improves significantly. we acknowledge that we did not explore different selection strategies, such as semscore, lmscore, and tlength, which involve using top-ranked examples. these strategies have been shown to potentially improve the quality of the generated summaries by selecting more effective prompt examples. while our current approach did not incorporate these strategies, we recognize that exploring and incorporating better prompt examples could potentially yield improved results. this is an area that warrants further investigation and experimentation to enhance the performance of our models in future iterations of the study. data privacy both gpt-3 and gpt-4 are not local models; we utilize openai's api to run these models, which actually violates data protection laws such as hipaa. ensuring data privacy during fine-tuning or testing is of paramount importance. we have not taken this aspect into consideration. ## conclusion we have showcased our solution, submitted to the mediqa-chat shared task, designed to generate clinical notes from doctor-patient dialogues. our evaluation encompassed fine-tuned approaches utilizing models such as confit, gpt-3, roberta, and scibert, as well as an innovative method employing gpt-4. the results garnered are remarkable, evidenced by our team securing a fourth-place ranking amongst all participating teams, underscoring the efficacy of our methodologies. of particular note is the fact that expert annotations were deployed to substantiate the comparative superiority of the icl gpt-4 over other baseline models. this infusion of expert validation fortifies the credibility of our research and its ensuing findings, contributing a robust dimension of trustworthiness to our results.