H ealth M avericks@ MEDIQA -Chat 2023: Benchmarking different Transformer based models for Clinical Dialogue Summarization (2023)

Abstract: In recent years, we have seen many Transformer based models being created to address Dialog Summarization problem. While there has been a lot of work on understanding how these models stack against each other in summarizing regular conversations such as the ones found in DialogSum dataset, there haven’t been many analysis of these models on Clinical Dialog Summarization. In this article, we describe our solution to MEDIQA-Chat 2023 Shared Tasks as part of ACL-ClinicalNLP 2023 workshop which benchmarks some of the popular Transformer Architectures such as BioBart, Flan-T5, DialogLED, and OpenAI GPT3 on the problem of Clinical Dialog Summarization. We analyse their performance on two tasks - summarizing short conversations and long conversations. In addition to this, we also benchmark two popular summarization ensemble methods and report their performance.

## introduction it is essential to summarise the conversation between a doctor and a patient or another doctor to maintain records for compliance, training and evaluation. however this process, at the moment, is done manually which is time consuming and expensive. this paper presents the experimental results of our explorations with state-of-the-art deeplearning techniques to summarise such conversations to accomplish both subtask a @xcite of di-alogue2note summarization task from mediqa-chat 2023 @xcite . the solution of subtask b presented in this paper was ranked fifth among all the submissions for subtask b. the source code for the submission can be found in github 1 . the paper uses transformer based models for both assigning conversations into a pre-defined set of clinical notes sections and summarization of conversations. through this work, the paper also compares the performance of transformer based models for summarization tasks. this paper benchmarks performance of several transformer based model for summarization task on medical conversation documents. in addition to this comparison, we also evaluate performance of two ensemble techniques namely @xcite and @xcite . our simulations show that finetuning of transformer-based models works as well as in-context prompt-based finetuning of openai gpt3 which has usage-based costs and the risk of compromising your internal data to an external organization. this paper is organized as follows. section 3 presents a brief overview of the dialogue2note summarization task, including the labeled data available and the evaluation metrics. then the paper describes current state-of-the-art for clinical note summarization in section 2 that this paper build upon. this is followed by the description of the approach used to solve the subtask a of dialogue2note summarization task in section 4 and subtask b in section 5. then the results of our solutions for dialogue2note summarization tasks are presented. finally, the paper ends with a conclusion on the work. the paper includes an appendix containing exploratory data analysis and material that will help to better understand the solution presented in the paper. ## related work in @xcite the authors have used both a single-stage and a two-stage approach for summarization. in the single-stage approach, the authors have truncated the input sentence length to match the bart transformer model input length constraints. in the multi-stage approach, the authors summarize the input conversation and then pass these summaries through a secondary model to generate the final summary. the authors have only focused on summarizing the history of present illness (hpi) section. the current work presented in this paper extends it beyond one section. it uses a two-step approach to generate the summary but does not focus on any special processing of the data in each section to generate the summaries. in @xcite the authors combine extractive and abstractive summarization. they have presented a wide range of algorithms. cluster2sent, the most elaborate among these algorithms, first identifies the noteworthy utterances in each section and then clusters them before sending them to a summarization model. the present work presented in this paper is similar to this approach in that it does a section level summarization. the current work depends on the power of more powerful models to summarise instead of processing the text in the sections. in @xcite , the authors use gpt3 for medical summarization to achieve summaries that match human annotator-generated summaries using 30x lesser data. the authors generate k-candidate summaries for an input dialogue in this work. for each candidate summary generation, the authors sample n random examples from a small labelled data set. the examples, along with the input dialogue, are sent to gpt3 for summarization. in this work, the authors select n examples for each section. the authors have yet to identify the medical terms in the generated summary to measure its effectiveness, which could be future work. ## dialogue2note summarization task description this section provides a high-level overview of the dialogue2note summarization task (including both subtask a and b) from mediqa-chat 2023 foot_1 . the section starts with a description of the subtask task goals, followed by basic counts of the available labelled data. the metric used to evaluate this task is arithmetic mean of rouge-1 @xcite , bertscore f1 @xcite , and bleurt @xcite . ## subtask a methodology given a short conversation between a doctor and a patient, the goal of subtask a is to predict its sec- task a training data section header section description dialogue multi class classification model concat validation data identifying best beam search config for every model generate summaries seq2seq model 1 ... seq2seq model n ensemble summaries from all models final summary task a test data dialogue multi class classification model predicted section header section description concat fine tune seq2seq model seq2seq model 1 ... seq2seq model n (a) task a -training (b) task a -inference tion header and summarize it while ensuring that the generated summary is as fluent and as close to section text as possible. this section starts with a description of the approach used to predict the section header. this is followed by a description of the methodology used to summarize the conversation. for dialogue summarization, we have fine-tuned transformer-based large language models. we have done an in-context fine-tuning of openai gpt3 @xcite and have fine-tuned four popular transformer sequence-to-sequence (seq2seq) models. the section describes the processed labelled data used for fine-tuning the transformer based models, followed by the actual training steps. then this section looks at the steps used to generate the summary from the decoder. finally, we discuss the two approaches used for ensembling the output of the four transformer based models. we achieved success using bio-clinicalbert @xcite for classification in the healthcare domain and hence have fine-tuned this model for the classification of dialogue to a section header in subtask a. since the target variable is highly imbalanced (see figure a1a ), we use focal loss @xcite so that the algorithm focuses more on classes with fewer samples. we limit the number of input tokens to 300 tokens because that is the length of majority of dialogues, as shown in figure a1 . as the number of data samples available for training and validation is less, we use a 5 fold cross validation approach for modelling purposes to ensure that we can capture all the information in the data. the hyper-parameters used for training and performance for all folds can be found in table a3 . during inference, we pass a given dialogue through all five models, take an average of the logits for all the classes and output the class with the highest logit score. we fine-tune seq2seq models using the labelled data (dialogue,section text) for subtask a as the (input,output) pair. section text is a part of the labelled data and is a human subject matter expertcreated summary of dialogue. as a pre-processing step, we replace all new line characters with whitespaces. the dialogue is concatenated with the section description of its section header with the sep token of the seq2seq architecture. while training, we use the actual section description for the actual section header and at inference, we use the section description corresponding to the predicted section header for the given dialogue. no changes are made to section text. we use a 5-fold cross validation scheme and fine-tune four seq2seq models -biobart @xcite , flan-t5-large (chung et al., 2022), dialogled-base, and dialogled-large @xcite on each of the folds. here we need to select the number of input tokens for encoder and decoder. for encoder we have selected token length of 512 tokens and for decoder we have selected token length of 400 tokens. all the hyper-parameters used to train each of the above architecture can be found in table a4 . to select the best model, we use early-stopping based on validation negative log loss @xcite . the out-of-fold results can be found in table a6 . the distribution of tokens for dialogue and section text can be found in figure a1b and figure a1c respectively. to generate summaries that match the human generated summaries, we need a way to control the summary generated by the decoder component of a seq2seq model. this can be done by using decoding strategies such as beam search @xcite , top-k sampling @xcite , topp sampling @xcite , contrastive search (su and collier, 2023) etc. in this module, we use beam search with tpesampler algorithm from optuna foot_2 to search for the optimal decoding strategy trying to maximize rouge-1, rouge-2, and bertscore rather than relying on manual tweaking of these metrics. we use tpesampler here because it supports multivariate optimization and also it handles float, integer, and categorical values better than other algorithms present in optuna foot_3 . we use optuna here due to ease of implementing hyper-parameter optimization algorithms. we did not use bleurt during search because it is extremely time consuming. for this module, we use four hyper-parameters for beam search -early stopping, number of beams, no repeat n-gram size, length penalty. the search space of each of these variables can be found in the the approach used for in-context finetuning using openai gpt3 is as follows: for every dialog in the test set, we predict and store the section header. we, then, randomly pick 3 dialog-summary-section header triplet from the entire (training + validation) dataset with the same section header. we use these triplets to create three summaries. these three summaries are then merged together to get the final summary. the configuration used for this task can be found in the appendix in table a5 and its result on the test set can be found in in this paper we have used following approaches for ensembling: they have released a model foot_4 which we are using out of the box. ## subtask b methodology this section presents an end-to-end solution to convert an entire doctor-patient conversation (dialogue) to clinical notes as subtask b requires. the section starts with a description of the supervised machine learning model used to predict the section header to which every utterance in a conversation belongs. all of these section headers are mapped to the first level sections using the mapping in we train a multi-label classifier using the dialogue and section header data from subtask a to predict the section header to which an utterance belongs. as the data volume is very low, we use iterative-stratification package foot_5 to create 5 folds task a training data section header multi label classification task b training data utterance from each note multi label classification trained in (a) utterance classification utterances classified to subjective ... utterances classified to assessment and plan validation data identify best decoding strategy finetune seq2seq models fine tune seq2seq models on subjective ... utterances classified to assessment and plan (a) task b-multi label classification (b) task b -fine tuning seq2seq models task b test data utterance from each note multi label classification trained in (a) utterance classification utterances classified to subjective ... utterances classified to assessment and plan ensemble of summaries from all models for every section final summary generate summaries summaries for subjective ... summaries for assessment and plan (c) task b -inference of the data and train one model for each foldthus capturing all information in the data. some of the labels have very low presence (see the distribution in figure a1a ), and hence we use focal loss instead of binary cross entropy loss so that the model can focus on labels which are harder to classify. the base model used for fine tuning is bio-clinicalbert @xcite with number of input tokens as 512. we use early stopping to select the best model using validation negative log loss as the criteria for selecting the best model and precision-recall (pr) score to evaluate performance of this model. the hyper-parameters for all folds and pr score for all folds for all section headers can be found in table a7 and table a8 respectively. we split every conversation on a new line character (\n) to get the list of constituent utterances. each utterance is passed through each of the 5 multi label models and we create a union of all predicted section headers from every model. once every utterance has been mapped to all possible section headers, we transform the mapping so that we can combine all the utterances that belong to same section header. we ensure that utterance order should remain intact in all the sections. we then map all these section headers to their first level sections using mapping in we fine tune seq2seq models using (utterance, clinical note section) generated above as the (input, output) pair for every first level section. before feeding the utterance to the encoder-decoder models, we concatenate it with the section description of the first level section that the utterance belongs to, using the sep token of the transformer architecture. we train two seq2seq models -dialogled-base and dialogled-large @xcite for each of the first level sections for each of the folds. the distribution of tokens for utterances of each first level sections and corresponding part of clinical note can be found in the figures a3b , a3c , a4b , a4c , a5b , a5c , a6b , and a6c . all the hyper-parameters used to train each of the above architecture can also be found in table a9 . apart from finetuning transformers we have also used openai gpt3 @xcite to generate summaries using prompt engineering. for every dialog in the test set, we pass it through section 5 to split the dialogue into utterances for every first level section. we randomly pick 3 dialog-summary pair for every first level section from the training data and truncate dialog to 750 tokens and summary to number of tokens as per the first level section it belonged to. the number of tokens for summary section of each first level section can be found in table a9 . as for test dialog, we truncate it to 1000 tokens. the reason we do this is to adhere to the 4000 tokens length constraint of openai gpt3 api. we concat the train dialog and summary along with test dialog and generate a summary. this step is repeated three times. these three summaries are then merged together to get the final summary. we use the configuration in table a5 for summarization. to select the best model, we use early-stopping @xcite using validation negative log loss and the metrics for best model for each architecture for each fold can be found in table a3 . we use the same search strategy for the optimal decoding strategy as we did for subtask a except for one difference -we apply these techniques on the summaries generated for each first level sections separately. we use the same model ensembling techniques that we have used in subtask a except for one difference -we apply these techniques on the summaries generated for each first level sections separately. ## subtask a results this section presents the results for subtask a using the approach described in section 4. we have made three submissions (mentioned as runs in the result tables) for predicting section header and three submissions for generating summaries from dialogues. all the submissions for classification and final rankings of each of the runs can be found in table a12 . for the summarization task, we have also submitted results from three runs. in run 1 and run 2, we have done the finetuning of the transformer based models mentioned in section 4 while run 3 presents the results of summarization using openai gpt3. the details for each run are as follows: the table containing our team's standing can be found in the tables a12 and a13 . standings of all the teams have been calculated by calculating multi class accuracy for section header classification and arithmetic mean of rouge-1, bertscore, bleurt for the dialogue summary. the experiments show that run1, which performs the worse on the ranking, hallucinates the most. this is counterintuitive since the goal of this approach is to minimize hallucination. we hypothesize that this could be because the model to detect hallucination was not trained on clinical data. run2 gives the best summaries as measured by the evaluation metrics but has some hallucinations. run3 gives the results with little to no hallucinations but has a lower score than run2. this can happen because rouge score always favours more extended conversations over shorter ones @xcite . this can also be seen in ## subtask b results as mentioned in section 5, we have used a two step process using first a multi-label classification model to assign a conversation to a section and then applying transformer based models on conversations for a section. just like in subtask a, we have made three submissions for generating summaries from conversations. for run 1 and run 3, we map the utterances into first level sections, followed by summary generation for every first level section. the decoding strategy for each first level section for each model can be found in table a10 . in this task, we tried using beam search configuration generated from section 4 as well but this did not work well as we were getting out of memory (oom) errors. the summaries generated by this process is used below. we have analysed the pros and cons of these three runs. run1 gives the best result on this task but has some hallucinations. run2 gives the results with little to no hallucinations but it has a lower score than run1 and run3. this can be because of the information loss that has happened as we are not able to consider all the tokens in the prompt. while run3 performs better than run2, it still has the most hallucination. this can be seen in ## conclusion the paper presents the solution and the results for subtask a and b of dialogue2note summarization task. the solution uses transformer based models for both classification and summarization of clinical dialogs and the paper presents the comparison of the performance of transformer based models on summarization. to the best of our knowledge, this is the first paper that benchmarks the performance of these models on clinical dialog summarization. our simulations and test scores show that fine-tuned transformer models work as well as in-context prompt based fine-tuning of large language models such as openai gpt3. this is encouraging for groups which either cannot afford huge api costs of using these large language models or cannot send their data to their api due to regulatory restrictions. in addition to this, we also observe that metrics such as rouge might not be the suitable to gauge performance of models like openai gpt3 as they focus on syntactic similarity. metrics such as bertscore and bleurt seem to be more suitable for such models since they focus on semantic similarity. the paper also evaluated two different ensemble techniques and the results demonstrate that the post ensemble technique performs the best while also giving minimum hallucinations.