N ew A ge H ealth W arriors at MEDIQA -Chat 2023 Task A: Summarizing Short Medical Conversation with Transformers (2023)

Abstract: This paper presents the MEDIQA-Chat 2023 shared task organized at the ACL-Clinical NLP workshop. The shared task is motivated by the need to develop methods to automatically generate clinical notes from doctor-patient conversations. In this paper, we present our submission for MEDIQA-Chat 2023 Task A: Short Dialogue2Note Summarization . Manual creation of these clinical notes requires extensive human efforts, thus making it a time-consuming and expensive process. To address this, we propose an ensemble-based method over GPT-3, BART, BERT variants, and Rule-based systems to automatically generate clinical notes from these conversations. The proposed system achieves a score of 0.730 and 0.544 for both the sub-tasks on the test set (ranking 8th on the leaderboard for both tasks) and shows better performance compared to a baseline system using BART variants.

## introduction telecare has experienced an exponential increase in utilization since the onset of the covid-19 pandemic, leading to the emergence of a vast network of healthcare providers and patients @xcite . we consider this to be a significant use case within the telecare domain, medical personnel often need to provide a concise summary of the conversation they had with their patient in order to ensure that a colleague is able to follow up on the next consultation. both patients and medical professionals can use these summaries to refer back to their interactions in the future. unfortunately, manually creating conversation notes after each encounter consumes a significant amount of time, and energy, and also poses challenges when done at scale. recent advancements in natural language processing (nlp) with large language models (llms) like gpt-3 have shown promising results in their ability to generate convincing natural language and successfully solve tasks including classification, answering questions, and summarization even in zero-shot and few-shot environments @xcite . this makes them a popular choice as opposed to a pre-trained model, which needs to be adjusted separately for each downstream task. in this paper, we propose an ensemble of rule-based methods, traditional sequence models, large language models, and bert-based models to develop an automated system for generating these notes from doctor-patient conversations. we also show that few-shot large language models outperform traditional sequence-to-sequence models in the setting of limited data. ## related work summarization is a crucial task in nlp, particularly for extracting key information from multispeaker conversations. various approaches have been proposed for meeting summarization, such as dialoglm @xcite , a pre-trained neural encoder-decoder model. in the context of medical dialogues between doctors and patients, identifying symptoms, diagnoses, and treatments is essential for deriving a medical solution. @xcite introduced the hierarchical encoder-tagger model (het) to specifically identify important utterances in medical conversations for summarizing medical conversations. @xcite introduced pointer generator networks for deep summarization of physician-patient dialogues. @xcite introduced a variant of the pointer generator network that handles negations and imposes a penalty on the generator distribution @xcite fine-tuned bart models for summarizing doctor-patient interactions. generation derived from doctor-patient conversations. it comprises three tasks 1 , namely, short dia-logue2note summarization (task a) ben abacha et al. (2023b), full dialogue2note summarization (task b) wai yim et al. (2023), and note2dialogue generation (task c) wai yim et al. (2023). our work primarily focuses on task a. this task requires us to create a section summary, encompassing both the section header and text, based on a short input of a doctor-patient conversation. the dataset consists of 20 distinct section headers for each conversation, such as medications, review of systems, past surgical history, chief complaint, etc. the training set contains 1,201 pairs of conversations, each accompanied by their relevant section headers and text, while the validation set is composed of 100 pairs of conversations and their respective summaries. ## system description in this section, we give a detailed explanation of our proposed system 2 . we propose two separate pipelines -one for section header identification and the other for section text generation. we also discuss our ensemble strategy and related intuition. category coverage text allergy incase of no allergies, reply with keyword 'no known allergies' fam/sochx incase of no family medical history found, reply with keyword 'noncontributory' genhx don't forget to mention age and gender of the patient, if present. finally, we output results from both pipelines to generate final summaries. section header identification: the task of section header identification involves categorizing a given doctor-patient conversation to the relevant header from a list of pre-defined headers. section text generation: the task of section text generation involves generating a summary of the given doctor-patient conversation. we propose an ensemble of 3 transformer-based models, i.e, bart-large, biobart-large, and few-shot gpt-3 for the same. here, we fine-tune bartlarge @xcite and biobart-large @xcite transformer models in a sequenceto-sequence paradigm on our training dataset. the models were fine-tuned with input as doctor-patient conversations and output as associated section text with the training objective of maximizing the likelihood of the generated summary. for gpt-3, we adopt a few-shot prompt engineering-based @xcite approach for generating our section text. few-shot prompting helps enable in-context learning for large language models like gpt-3. figure 2 shows a detailed annotation of the gpt-3 prompt that we use for our purpose. in the figure, <dialogue @xmath0. the intuition behind adding this extra knowledge to our prompt was to help our model learn the writing style of actual section text. we also experimented by giving multiple examples of dialogue, section header, and section text as a part of our prompt. please refer to section 6 for more details on experiments. during our initial analysis of the training dataset, we observed that there were certain specific writing header full header fam/sochx family history/social history genhx history of present illness pastmedicalhx past medical history cc chief complaint pastsurgical past surgical history allergy ros review of systems medications assessment exam diagnosis disposition plan edcourse emergency department course immunizations imaging gynhx gynecologic history procedures other_history labs we have shown some examples of coverage text in figure 1 shows the inference flow of the section text generation pipeline. once the section text from each of the models is generated, we score all of them and pick the one with the maximum score as our choice of final generated section text. for each section text, we calculate a final score based on a weighted scoring scheme that combines both central and syntactic similarity scores with weights w 1 and w 2 respectively. we found w 1 = w 2 = 0.5 to work best for our use case. for calculating central similarity for each of the generated section text, we implement the work done in @xcite and for syntactic similarity, we calculate the tokenlevel jaccard similarity between the section text and input dialogue. jaccard similarity is defined as the ratio between the intersection of two sets and the union of two sets, and it is often used as a metric of similarity. intuitively, section text that centrally captures the majority theme across all the generated section texts will have a high central similarity score, whereas, the syntactic similarity would help ensure faithfulness. goel et al. (2021) dialogue as gs r1 doctor: do you drink alcohol or smoke cigarettes? patient: no, i do not. doctor: are you sure? patient: yes. denies the use of alcohol or tobacco. denies the use of alcohol or tobacco. 1.0 doctor: have you ever had surgery? patient: one too many times. doctor: which ones? patient: i had my appendix taken out and glaucoma surgery fairly recently. i also had my gallbladder taken out ten years ago and a partial colon resection due to colon cancer in nineteen sixty one. doctor: any recurring episodes of colon cancer? patient: no, thankfully. partial colon resection of colon carcinoma in 1961 with no recurrence, cholecystectomy 10 years ago, appendectomy, and glaucoma surgery. appendectomy and glaucoma surgery. cholecystectomy 10 years ago and partial colon resection due to colon cancer in 1961. 70.3 doctor: any difficulty in hearing? patient: no. doctor: difficulty swallowing? patient: um no. doctor: any double vision or blurred vision or difficulty seeing things properly? patient: no, no problem at all. doctor: okay. doctor: how about headaches or migraine? patient: no headache. doctor: did you notice any change in your bowel moment? patient: no, it is the same. doctor: any pain while urinating or change in frequency? patient: no. doctor: okay. no headaches. no visual, hearing, or swallowing difficulties. no changes in bowel or urinary habits. neurological: no difficulty in hearing, swallowing, double vision, blurred vision, headaches, or migraines. gas-trointestinal: no change in bowel movements or difficulty urinating. 48.7 doctor: it seems like you are not feeling very well today? patient: yeah. i have had diarrhea and pain in my stomach. doctor: have you experienced any vomiting? patient: yes. i threw up this morning." diarrhea, vomiting, and abdominal pain. the patient states that he is not feeling very well today. he has had diarrhea and pain in his stomach this morning, and he has had vomiting this morning. 23.5 doctor: i will do some examinations on you. i will check your chest and then i will talk to you as i move forward, okay? patient: i'm okay with that. doctor: so, let's see what we have here. hm, yeah, just looks good. i do not find anything abnormal. chest: the chest examination is unremarkable. chest x-ray without any abnormality. 16.7 guest_clinician: i did a review of her systems, and everything looks normal other than what was mentioned earlier. doctor: okay, thanks for your help. guest_clinician: no problem. the remaining ros is unremarkable. review of systems: everything appears to be normal other than what was mentioned earlier. 0.0 doctor: are you allergic to anything, food or medicines? patient: no allergies that i know of. none. no known drug allergies. 0.0 ## model background in this section, we discuss in brief the background of various machine learning models that we have used in our implementation. bio_clinicalbert: the bio_clinicalbert foot_1 model is initialized from biobert lee et al. (2020) and trained on all notes from mimic iii johnson et al. (2016), a database containing electronic health records from icu patients. the model was pre-trained on a geforce gtx titan x 12 gb gpu, on a batch size of 32, a maximum sequence length of 128, and a learning rate of 5x10-5. bart & biobart: bart lewis et al. (2019) foot_2 is a transformer encoder-decoder (seq2seq) model with a bidirectional (bert-like) encoder and an autoregressive decoder to perform complex nlg tasks like summarization, translation, etc. bart is pre-trained to reconstruct the original text from the noisy text. biobart yuan et al. (2022) foot_3 , a bart variant was pre-trained on pubmed abstracts to achieve biomedical domain adaption. gpt-3: gpt-3 brown et al. (2020) is an autoregressive language model with 175 billion parameters. it achieves strong performance on many nlp datasets, including translation, and questionanswering, as well as several tasks that require on-the-fly reasoning or domain adaptation. ## experiments and results in this section, we discuss experiments, implementation details, and results obtained for both our tasks. section header identification: we trained fasttext @xcite along with multiple bert variants like bert-base devlin et al. (2018), roberta liu et al. (2019), and bio_clinicalbert alsentzer et al. (2019) for identifying section headers from doctor-patient dialogues. we use simpletransformers rajapakse @xcite python library for fine-tuning all our transformer models. amongst all of them, bio_clinicalbert gave us the best score on the validation dataset. our final model also incorporates a keyword-based categorizer in the pipeline giving us the best accuracy of 77% on the validation set. we use weighted cross-entropy as our loss function because of the skewed distribution of headers in the training data. the corresponding weights per header category were calculated using sklearn's compute_class_weight function on the training dataset and we train our best model for 10 epochs. section text generation: we fine-tuned bart, biobart architectures and, also inferred gpt-3 model in a few-shot setting. interestingly, gpt-3 in the few-shot setting outperforms all our fully supervised models by 1.2+ rouge-avg points (refer we experimented with the number of ground truth examples for our prompt. we tested with a maximum of 3 examples, because, for values higher than 3, the rate of getting the maximum token limit error from the gpt-3 api had increased significantly. across 0, 1, 2, and 3, we found 3 to be giving the best results on the validation set. finally, our ensemble of bart, biobart, and gpt-3 outperforms all our individual models by 0.9+ rouge-avg points (refer split method accuracy val fine-tuned bio_clinicalbert 0.75 fine-tuned bio_clinicalbert + kw classifier 0.77 test fine-tuned bio_clinicalbert + kw classifier 0.73 and 'bleurt-20' foot_5 respectively. we report the score for these metrics in ## observations here we discuss some observations that we made on results as shown in ## conclusion and future work we have presented a novel ensemble-based approach for the task of automatic short medical dialogue to note summarization. our method effectively combines fully supervised transformer models, few-shot gpt-3, and rule-based systems, generating accurate and coherent summaries of doctor-patient conversations. the proposed system demonstrates competitive performance on the mediqa-chat 2023 task a, highlighting its potential to enhance telecare and healthcare services. as part of future work, we plan to explore advanced pre-trained models and techniques to further improve our system's performance in the medical context. additionally, we aim to investigate the applicability of our approach in handling more complex dialogues. we also plan to conduct an in-depth analysis of the generated summaries to identify areas for further fine-tuning.