Few-shot fine-tuning SOTA summarization models for medical dialogues (2022)

Abstract: Abstractive summarization of medical dialogues presents a challenge for standard training approaches, given the paucity of suitable datasets. We explore the performance of state-of-the-art models with zero-shot and few-shot learning strategies and measure the impact of pretraining with general domain and dialogue-specific text on the summarization performance.

## introduction clinical dialogues between patients and health professionals are among the core elements of the clinical encounter, containing most of the initial anamnesis questions, diagnostic information, treatment options, patient advice and counselling. doctors usually summarize the content of these conversations into clinical notes, after each clinical visit or make use of expensive human medical scribes. as recent speech-recognition technologies show increasingly good performance @xcite , capturing these dialogues and generating abstractive summaries would help to reduce clinician load and improve patient care @xcite . abstractive summarization has been one of the main challenges for nlp @xcite . the accuracy of abstractive summarization has improved over the past years due to the use of transformer-based, sequence to sequence (seq2seq) models @xcite , larger training datasets and denser neural networks. although several generalpurpose datasets such as xsum @xcite , @xcite , and samsum @xcite have been used for their training and development, few corpora exist that could be applied to the health scenario, medical terminology rich dialogues, with frequent interjections, ellipsis, and logical connections between semantic units (e.g., drug y treats condition z and not vice versa). we fine-tuned several state-of-the-art (sota) models in a newly created medical dialogue dataset of 143 snippets, based on 27 general practice conversations paired with their respective summaries. we tested 10 transformer models to assess their performance in abstractive summarization of these dialogues. we learned that models pre-trained on general dialogues outperform baseline models. bart-based models were found to achieve the highest scores, although medical inconsistencies persisted in the generated summaries. in the future, we plan to perform further evaluations as the need for metrics that highlight inconsistencies in medical summaries remains unresolved. ## background training and fine-tuning nlp models for medical tasks has been a challenge, given the paucity of high-quality training data, although several initiatives such as mimic @xcite and n2c2 challenges @xcite have advanced the field. strategies to reduce dependence on large training datasets, such as transfer learning, have been explored @xcite to improve the model performance. transformer-based models and their various implementations are well suited for transfer learning and fine-tuning with sparse datasets. additionally, zero-shot and few-shot approaches may help strike the balance between the model's box 1: dialogue-summary example performance, training time and training data requirements. several recent developments have shown the effect of few-shot strategies in medical abstractive summarization @xcite as well as in online medical dialogues @xcite . although few-shot and pre-training strategies have been studied separately, none have experimentally compared how these two interact in the medical dialogue domain and how different seq2seq models perform under these circumstances. in this work, we study how different few-shot strategies and pre-trained models affect the performance of abstractive summarization in medical dialogues. ## discussion our experiment shows that fine-tuning pre-trained models with few-shot learning offers a reliable strategy to improve summarization scores with small training data, making it appropriate for finetuning transformer models in domain-specific contexts, such as medical dialogues. by contrast, pre-training on medical literature did not improve results and showed poorer performance than the baseline models. bart based models achieve the highest rouge scores across all the fsl steps, with a relatively smaller footprint in terms of the required training time and the number of examples compared to both t5 and pegasus. our experiment confirms previous findings that bart based models outperformed pegasus and t5 for summarization @xcite and with few-shot strategies @xcite . however, we observe that t5 gets higher rouge-l and rouge-l-sum results when trained on the full dataset. although we obtain differences in the rouge scores across the best performing models, a limited qualitative analysis did not show a clear difference for t5 vs. bart. our preliminary qualitative evaluation shows that t5 produced usable summaries (with no contradicting advice and no medical incoherencies) although further evaluation is required. this may reflect that relevant medical information may be situated at longer than 1-gram or 2-gram distances, suggesting that the longest common subsequence metric (rouge-l) can be more important for the quality of conversation summaries. moreover, we focus our analysis on the rouge score metrics, although this family of metrics alone is often insufficient to computationally appraise the quality of the summarization (suleiman and awajan, 2020). for instance, character n-gram f-score (chrf) @xcite , when evaluated for summarization tasks @xcite shows a higher correlation with the coherency of produced summaries than the rouge metrics. further research is needed to establish the most apt metrics for evaluating the quality of medical summaries, especially as the need for maximizing factual correctness is critical for practical summarization applications in the medical domain. an important limitation of our study is the small number of snippets and size of the medical dialogue dataset. given the sensitive nature of medical conversations, this is a pervasive problem facing the development of nlp medical models. it is unlikely that medical dialogue conversations can be easily recorded, transcribed, and released as a public dataset given that they are likely to contain highly sensitive information. however, our experimental design focuses on this pervasive issue in medical nlp by exploring how fsl and pre-training may be leveraged to overcome the scarcity of large datasets. in this work, we focus on a single document abstractive summarization. given the length and complexity of medical dialogues, further box 2: sample of generated summaries and their evaluation. legend: bold -contradicting advice, italic -medical incoherence, underlined -missing information (minor or major), strikethroughincorrect affirmation 6 experiments exploring multi-document summarization, aimed at producing full-dialogue summaries, would be necessary. previous strategies for long-text summarization, such as global encoding seq2seq approach @xcite or a globalized bert architecture using a hierarchical propagation layer @xcite , may prove successful for summarizing long medical dialogues. further model development, as well as refined training and fine-tuning strategies (e.g., adjusting transformer's structure, learning rate optimizations, and optimizing for additional metrics) or domain-specific dialogue datasets, may help further improve performance. medical knowledge embeddings may also be a suitable strategy to improve performance and prevent medical incoherencies illustrated above. additional evaluations involving multiple clinicians and creating a more encompassing taxonomy of medical summarization errors would be needed for a thorough qualitative evaluation and proper appraisal of the model output quality. establishing additional contrasts between qualitative and quantitative analysis may help to identify metrics that reliably capture important medical qualitative differences, potentially informing the development of new metrics, and quantifying the issues identified in our evaluation. ## conclusions and future work summarization of medical dialogues with fsl using pre-trained models is a feasible strategy for model development. future research needs to focus on uncovering the most adequate set of metrics for capturing medically relevant and factually correct information in medical summaries. additional qualitative evaluation may shed light on these issues and inform either the selection or development of the right metrics. he has been coughing for two weeks. itchy inside of here and annoying sticky things in his throat. it feels like there is a bit of stuff there but it is not coming out. has a history of asthma or chest problems. he used to have puffers but they are a long time ago. there was no fever or sore throat and nothing else. after he had ice cream he coughed a runny nose. cough has been two weeks. apart from the cough he feels itchy. there are annoying sticky things in his throat. it feels like there is a bit of stuff there, but it is not coming out. breathing feels normal sometimes. he had a history of asthma and chest problems in china and used to take puffers. nothing else besides cough. no fever, sore throat, or runny nose. cough reminds him of a cough a few years ago that didn't stop. has allergies to some food but not heavy. some kind of plant or seafood. cough for two weeks, feeling itchy inside and having stuff that is not coming out. sometimes having difficulty breathing. four years ago, had an asthma episode in china and used a puffer. no fever or sore throat. no runny nose. this cough reminds of the previous one. he says it start after a cold drink or icecream especially in winter. he has mild allergies to some food, usually pimples on the skin, but not affecting breathing. he is generally healthy. he coughs more at night but has good sleep. if he is running or exercising he gets breathing problems or cough. two weeks ago he was in china and had a cough one day before leaving. it doesn't sound like he has had the runny nose or sore throat. he is generally healthy. he coughs more at night but still gets good sleep. usually, he doesn't exercise or run. two weeks ago he was in china and ate two ice creams. temperature can have an effect on breathing but not like weeks of cough. it doesn't sound like he had a runny nose or sore throat. there are a lot of sticky things in the chest but it does not come out. they will have a listen to his chest and check his throat and ears and temperature. he may need to use a puffer for a while. nobody else has a cough he is healthy. coughing more at night but still sleeping. he will cough when running for a train or bus or when exerting. he felt the cough was starting after eating an ice cream. temperature can have an effect but won't give weeks of cough. he just has sticky things that doesn't come out. it could be some infection that triggered the cough. no one around him has cough. it is difficult to breathe. she feels a little bit painful in the food pipe. her digestion system is not functioning well. the intestine is fine. when she is hungry the acid level goes up and up. last year she used to have that but she had some medicine to reduce the level in cambodia. it helped when she took the medicine. sometimes we do an endoscopy to look down with the camera. there is no vomiting or other stomach symptoms. no runny nose. it feels like difficult to breathe. she feels pain in the food pipe. her digestion system is not functioning well, so she thought it might be the stomach. the stomach is fine. last year she used to have some medicine to reduce the acid level and it improved. it was prescribed in cambodia. it is difficult to breathe when breathing and a bit painful in the food pipe. the first time she met you said it was about the windpipe but because her digestion system is not functioning well, she thought it might be the front. there is nothing down here. she feels very hungry high up in her tummy and acid levels go up and up. last year she used to have acid reflux but then she took some medicine to reduce acid level and it improved. it was prescribed in cambodia. taking the medicine helped. sometimes we trial some treatment and if it responds it can be done with an endoscopy. has not vomited or had a fever or runny nose. it feels like is difficult to breathe and sometimes a little painful in the food pipe. intestines and tummy feel fine. feels uncomfortable when very hungry and that the acid is coming up. used to have it last year but improved with medicine to reduce acid. it was prescribed in cambodia. sometimes we trial treatments, we don't rush in and do an endoscopy if not necessary. felt tired. no vomiting, fever or runny nose. tonsils are still swollen and she is not coughing. acid can give irritation in the throat. she hasn't lost a lot of weight. one kilo less. no blood in bowel motions or vomiting blood. sometimes it can be related to spicy food, coffee, chocolate, alcohol or food that is very acidic. trying to she is coughing but not coughing. her tonsils are still swollen. if it is acid coming up, it can give her irritation in the throat. she hasn't lost a lot of weight, but she has lost one kilo. it is common to have acid problems related to spicy food, coffee, chocolate, alcohol and acidic food. no coughing, but the tonsils are still swollen. acid can give you irritation in the throat. she hasn't lost a lot of weight. fifty-four point three is one kilo less. there is no blood in bowel motions or vomiting blood. sometimes it can be related to the food you eat spicy or acidic. you don't want to eat a huge amount of food in one sitting because it puts pressure on no coughing but tonsils are still swollen. acid coming up can irritate the lining of the throat. weights 54.3 kilos and has lost a kilo. no blood in the bowel motions or when vomiting. acid can relate to food you eat like spicy, coffee, chocolate, or alcohol. sometimes a cup of hot 13 avoid certain foods. drinking hot milk can sometimes help. there are tablets and also mylanta liquid that she can drink. the stomach. at night is pain at 11 when drinking hot milk. tablets can also be bought from chemist mylanta from cambodia. milk helps. you can also buy mylanta from the chemist. she had a miscarriage and gave some test on tuesday. she started vaginal bleeding on last wednesday and she was not here on thursday. they gave her an ultrasound and they couldn't hear the heartbeat. the baby did not grow much. she had a miscarriage. she was in study notes and couldn't do the test on time. before last wednesday she started bleeding, vaginal bleeding. on the thursday she came to see the doctor and told them. they appointed her to lucy and gave her ultrasound. the first ultrasound was 153 heartbeat. then on the same day they could not find heartbeat and the baby didn't grow much. she had a miscarriage. she was given a test on tuesday and couldn't do it on time. before last wednesday she started bleeding, vaginal bleeding. they couldn't hear the heartbeat and she was about 13 weeks pregnant. the baby didn't grow much, about seven or eight weeks growth. lucy had a miscarriage. she started a vaginal bleeding last wednesday. they did an emergency ultrasound and could not hear a heartbeat. first ultrasound was normal. she was 13 weeks pregnant. the first ultrasound showed it wasn't growing.