Team Cadence at MEDIQA -Chat 2023: Generating, augmenting and summarizing clinical dialogue with large language models (2023)

Abstract: This paper describes Team Cadence’s winning submission to Task C of the MEDIQA-Chat 2023 shared tasks. We also present the set of methods, including a novel N-pass strategy to summarize a mix of clinical dialogue and an incomplete summarized note, used to complete Task A and Task B, ranking highly on the leaderboard amongst stable and reproducible code submissions. The shared tasks invited participants to summarize, classify and generate patient-doctor conversations. Considering the small volume of training data available, we took a data-augmentation-first approach to the three tasks by focusing on the dialogue generation task, i.e., Task C. It proved effective in improving our models’ performance on Task A and Task B. We also found the BART architecture to be highly versatile, as it formed the base for all our submissions. Finally, based on the results shared by the organizers, we note that Team Cadence was the only team to submit stable and reproducible runs to all three tasks.

## introduction mediqa-chat 2023 shared tasks included three tasks on the summarization and generation of doctor-patient conversations to promote research on these topics @xcite . task a (short dialogue2note summarization) expected a section summary (section header and content) given a short input conversation. we recognized that generating the summary content was an abstractive summarization @xcite task while predicting the section header was a multiclass (twenty normalized section labels) classification task. task b (full dialogue2note summarization) was another abstractive summarization task that required submissions to generate a complete clinical note from a whole dialogue between a patient and a doctor. the complete clinical note was expected to have the following first-level section headers: "history of present ill-ness", "physical exam", "results", and "assessment and plan". finally, task c (note2dialogue generation), a data augmentation @xcite task, was about generating patient-doctor conversations for complete input notes. aside from predicting section headers for task a, all other tasks could be formulated as sequenceto-sequence @xcite learning tasks. various model architectures based on transformers @xcite have proved to be successful at tackling such tasks. therefore, leveraging pre-trained model checkpoints from public repositories was considered the right choice. encouraged by the leaderboard for samsum @xcite on huggingface @xcite , a dialogue summarization dataset, we chose bart @xcite as the base model for our experiments. specifically, we picked the facebook/bart-large foot_0 model checkpoint (referenced as bart-large in this text from hereon) for its effectiveness on text-generation tasks. the samsum @xcite dataset is intended to train dialogue summarization models. however, we recognized that the input and target labels could be inverted to train a dialogue generation model. we trained/validated bart-large on the inverse of samsum @xcite dataset followed by the task c training dataset provided by the task organizers, achieving rouge-1 and rouge-2 scores of 59.11 and 23.69, respectively, on the validation set. this model was then used to augment datasets for task a and task b summarization tasks. in order to generate synthetic patientdoctor conversations, we chose to sample a thousand discharge summary notes from the mimic-iv-note @xcite dataset. we then added these dialogue-note pairs to the task a and task b training datasets provided by the organizers. the impact of this augmentation technique is noted in section 5 below. for task a summarization, bart-large was fine-tuned on the samsum @xcite dataset followed by fine-tuning on the augmented dataset for task a, which achieved rouge-1 and rouge-2 scores of 50.7 and 21.4, respectively, on the validation set. our methods yielded an overall improvement (over the baseline) of 13.1% and 14% in rouge-1 and rouge-2 scores, respectively. results from fine-tuning bart-large on the unaugmented (original) task a dataset were considered the baseline in this comparison. inspired by the significant gains exhibited by the task a model, we decided to use it as the base model for task b. fine-tuning this base model on the augmented task b dataset yielded rouge-1 and rouge-2 scores of 54.16 and 26.04, respectively -a 13.7% gain in rouge-2 score over the baseline. results from fine-tuning the base model on the unaugmented (original) task b dataset were considered the baseline in this comparison. the final submission(run1) achieved rouge-1 and rouge-2 scores of 49.5 and 23.4 on the test set. unfortunately, the task b dataset comprised input conversations almost twice as long as the maximum number of tokens accepted by bart-large, which naturally prohibits the model's ability to summarize the entire conversation. to solve this prob-lem, we developed an n-pass strategy in which the model attempts to summarize the conversation in multiple steps. each step (or pass) involves the model taking as input the summary note of the dialogue processed till that step, concatenated with the rest of the dialogue. in other words, we trained the model to summarize a partial mix of an incomplete clinical note and an incomplete patientdoctor conversation. this strategy led to a gain of 6.6% and 8.1% in rouge-1 (57.76) and rouge-2 (28.15) scores, respectively, on the validation set. we submitted the n-pass model as run2, which outperformed the run1 submission by 6.8%, both for rouge-1 (52.9) and rouge-2 (25) scores, on the test set. it also improved the division-based aggregate score by 16.75%. overall, our methods improved the baseline rouge-2 score by 22.9% on the validation set, while the baseline rouge-1 score was found to be slightly better by 0.45%. given the promising performance of bart-large on the summarization tasks, we also decided to use it for task a classification. we leveraged the bartforsequenceclassification wrapper offered by huggingface @xcite , a bart model with a sequence classification head on top (a linear layer on top of the pooled output). using this approach, we achieved an accuracy of 78% and an f1 score of 78.37%. the final submission was reported to have an accuracy of 73.5% on the test set. ## background and related work studies like the ones from alkureishi ma et al. @xcite and rathert et al. @xcite have presented evidence on ehrs (electronic health records) impacting the quality of patient-doctor conversations. digital scribes @xcite and summarization tools @xcite can mitigate some of these problems. however, many challenges are associated with clinical dialogue summarization @xcite . some significant challenges include omitting key medical concepts @xcite and hallucinating unsubstantiated information. several attempts have been made to address said inherent challenges and automatically generate high-quality summaries of clinical encounters. approaches like the ones used by @xcite have utilized a transformer @xcite model to summarize doctor-patient conversations. @xcite and michalopoulos et al. (2022) have also incorporated medical knowledge into these models. on the data generation front, @xcite showed that large language models can be used for augmenting medical summarization datasets. to the best of our knowledge, the n-pass strategy used to address long input sequences of task b is novel. however, multiple multi-stage summarization approaches have been proposed so far. for example, @xcite al. (2020 ) used modular summarization techniques to produce notes from patientdoctor conversations. zhang et al. (2021) used multi-stage summarization for long inputs, whereas gidiotis and tsoumakas (2020) split a long document and its summary into multiple source-target pairs using sentence similarity. recursive summarization incorporating human feedback (wu et al., 2021) even achieved state-of-the-art results in book summarization. ## results team cadence's submission for task c earned rank-1 amongst all participants, beating the nextbest submission by 28.3% (rouge-1) and 99% (rouge-2). the organizers shared test set results @xcite along with a code status description where a code status of 1 meant that the organizers were able to run the submitted code and reproduce the results, and a code status of 2 meant that they were able to run the code and found minor differences with no changes in rankings. code statuses 3,4, and 5 meant that the organizers found the submitted code to be unstable or not runnable under their configurations. amongst code statuses 1 and 2, team cadence achieved the following ranks: rank-2 on taskbsummarization, rank-3 on taska-summarization, rank-3 on taskb-summarization(note-divisions), and rank-5 on taska-classification. the code for generating the submitted runs is being shared publicly foot_2 . ## system specification in the spirit of reproducibility, we share details of the systems used to run these experiments. the models were fine-tuned on g4dn.12xlarge aws sagemaker notebook instances foot_3 . huggingface's python package transformers @xcite version 4.27.1 was used in a python3.8 environment. reported results were aggregated from 4 different runs using 4 different random seeds. ## limitations and future work the methods described in this paper do not leverage any external medical knowledge, a technique that has been shown to be effective by other studies @xcite . and like other methods based on large language models, in theory, our models are also prone to hallucinations and omission of key-clinical concepts. we plan to explore constrained beam search foot_4 as a mitigation strategy for addressing these challenges in the future. although the impact of the task c model as a data augmentation tool is undoubtedly positive (section 5), qualitative error analysis of patientdoctor conversations produced by the model showed that the output contained a small number of dialogue turns, and each individual turn was too long, packed with information. producing conversations with a more natural flow should yield an even better boost on downstream tasks, and we leave exploring such methods to future experimentation. we also recognize that n-pass summarization for task b with higher values of n should be able to cover the entirety of the input conversations in the task b datasets, albeit with diminishing returns as n increases. we hope to evaluate them in future iterations of similar shared tasks. ## conclusion the two key takeaways from the experiments and results in this paper are significant improvements in summarization results driven by data augmentation and the n-pass summarization technique for handling long input patient-doctor conversations. furthermore, the fact that our submissions to all three tasks share the same base (bart-large) model speaks volumes of its versatility. finally, the results demonstrate the effectiveness of fine-tuning on custom datasets for specialized domains like medicine.