C alvados at MEDIQA -Chat 2023: Improving Clinical Note Generation with Multi-Task Instruction Finetuning (2023)

Abstract: This paper presents our system for the MEDIQA-Chat 2023 shared task on medical conversation summarization. Our approach involves finetuning a LongT5 model on multiple tasks simultaneously, which we demonstrate improves the model’s overall performance while reducing the number of factual errors and hallucinations in the generated summary. Furthermore, we investigated the effect of augmenting the data with in-text annotations from a clinical named entity recognition model, finding that this approach decreased summarization quality. Lastly, we explore using different text generation strategies for medical note generation based on the length of the note. Our findings suggest that the application of our proposed approach can be beneficial for improving the accuracy and effectiveness of medical conversation summarization.

## introduction medical conversations between doctors and patients play a crucial role in healthcare. the conversations help the doctors understand the patients' conditions, diagnose, and provide appropriate treatments. however, these conversations can be lengthy and complicated, leading to difficulties in summarizing the essential information for medical records. automatic summarization of medical conversations can help reduce the workload of medical practitioners and improve the quality of patient care. therefore, there is a growing interest in developing natural language processing (nlp) techniques for summarizing medical conversations. large language models (llms) such as bart @xcite , gpt-3 @xcite and t5 @xcite have shown to be powerful in various language generation tasks, including summarization. however, they are known to suffer from hallucinating, i.e. including the facts that are false in the output or corrupting the facts in the input @xcite . this paper proposes a method for summarizing medical conversations using t5-based models. we finetune two t5-based models on two datasets from the mediqa-chat 2023 shared task. the first dataset consists of short transcriptions of doctorpatient conversations, while the second dataset contains full patient-doctor encounter transcriptions. our method uses text-to-text modelling, representing the input as a dialogue and the output as a conversation summary. to tackle the hallucination problem, we modify the data using a clinical named entity recognition model to tag the entities in the input and output sequences. we suppose this enables the models to learn better to copy the relevant entities from the conversation to the generated summary. additionally, we finetuned a single model on multiple tasks to improve its robustness. our results showed that finetuning a single model on multiple tasks improved the summary generation quality and reduced hallucination. on the other hand, introducing extra tags to the inputs worsened the summarization quality. the remainder of this paper is organized as follows. in section 2, we provide a brief overview of related work in the field of text and dialogue summarization and highlight the limitations of existing approaches. section 3 briefly describes the data used for the mediqa-chat 2023 shared task. in section 4, we describe our proposed method in detail. in sections 5 and 6, we present the experimental setup and results, followed by a thorough analysis of the effectiveness of our approach. finally, we conclude the paper and outline future directions in section 7. ## related works several generative language models, such as bart @xcite , gpt-3 @xcite , pegasus @xcite , and t5 @xcite are used for abstractive text summarization. all these models are based on the transformer encoder-decoder architecture @xcite . significant progress has been made in training the generative language models using multitask setting by giving the natural language instructions @xcite @xcite . while these models are already powerful for zero-shot and few-shot settings, finetuning them on specific data can significantly improve the performance for different tasks. in terms of dialogue summarization, commondomain datasets such as dialogsum @xcite , mediasum @xcite , sam-sum @xcite have been used for training the generative language models. medical conversations summarization has been generally understudied with the recent efforts by @xcite , @xcite , and @xcite . since full conversations are generally lengthy and extend beyond a common input length limit of most of the pre-trained models, several efforts have been made to modify the transformer selfattention mechanism to encode long texts @xcite efficiently. finally, the problem of the faithfulness of the automatically generated text is especially crucial for medical domain @xcite 3 data we work on two datasets that are a part of mediqa-chat 2023 shared task @xcite . the first dataset used for task a consists of short transcriptions of doctor-patient conversations followed by one of 20 possible classification labels and a short note from a doctor summarizing the conversation @xcite . the second dataset used for task b contains full patient-doctor encounter transcriptions accompanied by a full clinical note based on the encounter @xcite . ## method we finetune two t5-based models: flan-t5 base model @xcite on task a data and longt5 base model @xcite flan-t5 maximum context window (512 tokens), we only finetune it on the task a data. ## experimental setup to test the importance of each component of our solution, we finetuned the longt5-base model foot_0 with the configurations from the the models are finetuned for 20 epochs using the adamw optimizer @xcite and a learning rate of 5 • 10 -5 . we trained all our models on a single a100 80gb gpu (university of tartu, 2018) with a batch size of 8. to generate the outputs, we used beam search with early stopping and beam width of 4, length penalty of 2.0 @xcite , and the top-k sampling @xcite with @xmath0. addition-ally, we limit the maximum generation length to 200 tokens for task a and 512 tokens for task b. additionally, we preprocessed the task b data. first, we changed the role markers from "[doctor]" and "[patient]" to "doctor:" and "patient:". as a second step, we fixed the punctuation that had an extra space before it. for this, we first split the text by space token and reassembled it with the treebank detokenizer from nltk. this was done to ensure consistency between the task a and b data. finally, we applied a postprocessing step to taskab-tag model to remove the @xmath1. all the reported results were measured on the validation set using the evaluation script provided by the shared task organizers. the following metrics were used: rouge score (rouge 1 , rouge 2 , rouge l ) @xcite , bertscore (r bert , p bert , f bert ) @xcite , and bleurt @xcite . for the final submission, we used taskab-tag model as run1, the same model but with the contrastive search generation strategy @xcite as run2, and a flan-t5 base model foot_1 finetuned identically to taska-only but on the tagged data as run3. we used all three models for task a, with the inputs exceeding 512 token length truncated for the run3 model, and only run1 and run2 for task b. ## results and discussion tables 3 and 4 show the results on the validation dataset for task a and b correspondingly. for both tasks, the models show a similar pattern: taskab model generally performs better for most of the metrics, only falling behind the taska-only model in p bert for task a and in rouge 1 and rouge 2 for task b. taskab-tag model underperforms taska-only model in all the metrics for task a, however, shows better bertscore performance than taskb-only model for task b. upon closer inspection of the outputs, we noticed that due to the post-processing error, taskab-tag sometimes produced the output with the space before the punctuation. during the tokenization for calculating the bertscore and bleurt, the punctuation with and without space before it results in a different output. since both metrics use contextual token representations, these to test this, we removed the extra spaces before the punctuation and recalculated the metrics. this resulted in the increased p bert (+0.003), r bert (+0.001), f bert (+0.002) for both task a and b, as well as bleurt (+0.033 for task a and +0.040 for task b). to further test the model's factual accurateness, we manually measured on the task a validation data how well the model captured the age of the patient or other relevant people, the gender of the patient, and the dosage of the prescribed medicine. taskab model captured all three categories with the 100% accuracy; taskab-tag model correctly captured the age, gender, and dosage 75%, 100%, and 86% of the times; taska-only model showed the accuracy of 81% for age, 100% for gender, and 71% for dosage. additionally, we tested if the models generated the patient's age and gender in the summary when it was not mentioned in the dialogue: taskab model generated the unmentioned patient's age and gender once, taskab-tag twice, taska-only trice. tables 5 and 6 show the official results on the test set for task a and b correspondingly. for task a, the models were ranked by the aggregated score which is calculated as the mean of rouge 1 , f bert , and bleurt. for task b, the ranking was done by rouge 1 score. overall, for task a, our best system submission run1 was ranked 14th out of 31 total submissions; for task b, run2 was ranked 19th out of 23 total submissions. from these results, run2 model that used the contrastive search generation strategy shows better results for longer text generation, however, run1 model with beam search generation strategy is better suited for shorter note generation. validation set results show that augmenting the data with the clinical named entity recognition tags worsens the model's performance. the ner tags might have introduced additional noise to the data that the model was not able to accommodate during training. moreover, even though medication and disease names are generally shared between the conversation and the summary note they are not always formulated with the same words. additionally, the automatic ner tagger may introduce annotation errors that may propagate into the final model. on the other hand, combining the data from both tasks and finetuning using the instruction prompting improved the generation quality. ## conclusion in this paper, we presented our system for the mediqa-chat 2023 shared task on clinical conversation summarization. we showed that finetuning a longt5 model on several tasks simultaneously improved the model's overall performance and reduced the number of factual errors and hallucinations in the generated summary. on the other hand, augmenting the data with the in-text annotations from the clinical named entity recognition model decreased the summarization quality. finally, we showed that different text generation strategies can be applied to medical note generation depending on the length of the note.