C are4 L ang at MEDIQA -Chat 2023: Fine-tuning Language Models for Classifying and Summarizing Clinical Dialogues (2023)

Abstract: Summarizing medical conversations is one of the tasks proposed by MEDIQA-Chat to promote research on automatic clinical note generation from doctor-patient conversations. In this paper, we present our submission to this task using fine-tuned language models, including T5, BART and BioGPT models. The fine-tuned models are evaluated using ensemble metrics including ROUGE, BERTScore andBLEURT. Among the fine-tuned models, Flan-T5 achieved the highest aggregated score for dialogue summarization.

## introduction clinical dialogue summarization has emerged as a crucial task in clinical natural language processing (nlp). in a clinical nlp dialogue between a doctor and a patient, relevant information about the patient's medical history, visit summary, health condition, and other details are discussed. summarizing these dialogues can significantly benefit doctors by enabling them to quickly review key points from past conversations and extract relevant information from clinical notes without having to sift through an extended transcript. moreover, it can assist doctors in making better decisions by providing them with a concise and accurate conversation record. therefore, developing effective clinical dialogue summarization systems is of great importance in improving the quality of healthcare delivery. however, clinical dialogue summarization presents unique challenges and goals that differ from summarization in other domains. clinical summaries need to capture relevant information based on the context of the text, like medical histories, follow-ups, or current diagnoses. in this paper, we describe our submission to the mediqa-chat shared task @xcite the dialogue2note summarization task, task-a. we observe that from the conversation it is important to: (1) capture all the medical conditions and terminology described in the dialogue (eg. cough, fever, shortness of breath etc.); (2) discern all the affirmatives and negatives on medical conditions correctly (no allergies, having a cough for 2 days); and, (3) bias towards copying from the source text while not being completely extractive. our approach involves studying the effectiveness of fine-tuning pre-trained language models, including t5, gpt, and bart models. we compare the effectiveness of pre-trained models on dialogues, clinical data, and general models. includes two subtasks on the generation of specific sections (subtask a) and full notes (subtask b) from doctor-patient conversations. we participated on dialogue2note (subtask a). in this task, given a conversation between a doctor and a patient, the goal is to produce: ## approach for this submission, we fine-tuned a number of pre-trained language models for implicit classification of headers and note summarization. since the expected summaries differ in accordance with the associated section header, we fine-tuned the models using supervised training to jointly classify and learn corresponding summaries using the provided training dataset. all models were fine-tuned using hugging face transformers @xcite . figure 2 shows a general flow of our approach. ## evaluation and results evaluation is performed using the metrics described in @xcite and mentioned in section 2.2. the script provided in the shared task foot_4 was used for evaluating the fine-tuned models. evaluation was performed on the validation dataset only as the test dataset references are not available. ## related work automated note generation from doctor-patient conversations has been the subject of several recent studies in natural language processing and healthcare. one line of research has focused on developing machine learning models to automatically generate clinical notes from speech or text data, using deep learning and natural language generation techniques @xcite @xcite . other studies have explored the use of voice recognition and speech-to-text technologies to transcribe doctor-patient conversations and generate notes in real time @xcite . additionally, some researchers have investigated using pre-trained language models, such as bert and gpt, to improve the accuracy and efficiency of note generation @xcite . overall, these efforts aim to reduce the burden on healthcare providers by automating the tedious task of note-taking and ultimately improving the quality and accessibility of patient records. ## conclusion we utilize several pre-trained models for task a in mediqa-chat shared task. the main objective of this task is to develop clinical dialogue summarization in accordance with a classified section header for every dialogue. we fine-tuned different models for our experiments. among the models we used, we found that flan-t5, originally trained on dialogue datasets, outperformed other models that were trained on clinical data or summarization tasks. specifically, flan-t5 samsum outperformed all models except for summarization scores. it can also be concluded that summarization models trained on summarizing text, not dialogues, as in biogpt, performed poorly on summarization tasks. in contrast, bart models performed better than the biogpt model. empirically, we found biogpt to generate text that was not originally in the text, which is considered critical in the context of health records. finally, since flan-t5 samsum achieved the best results, we anticipate that further unsupervised training for the flan-t5 language model with clinical dialogues would improve the results.