Teddysum at MEDIQA -Chat 2023: an analysis of fine-tuning strategy for long dialog summarization (2023)

Abstract: In this paper, we introduce the design and various attempts for TaskB of MEDIQA-Chat 2023. The goal of TaskB in MEDIQA-Chat 2023 is to generate full clinical note from doctor-patient consultation dialogues. This task has several challenging issues, such as lack of training data, handling long dialogue inputs, and generating semi-structured clinical note which have section heads. To address these issues, we conducted various experiments and analyzed their results. We utilized the DialogLED model pre-trained on long dialogue data to handle long inputs, and we pre-trained on other dialogue datasets to address the lack of training data. We also attempted methods such as using prompts and contrastive learning for handling sections. This paper provides insights into clinical note generation through analyzing experimental methods and results, and it suggests future research directions.

## introduction multi-turn dialogue summarization in the medical field is an important research area. medical professionals need to make crucial decisions while consulting with various patients, so creating clinical notes from doctor-patient consultations, which record consultation details and diagnoses, is an essential task for both doctors and patients. however, having doctors write entire clinical notes is time-consuming and reduces consultation efficiency. herefore, it is important to develop technologies that can automatically generate clinical notes from conversation content, allowing doctors to simply review and modify the results, which can shorten consultation times. mediqa-chat 2023's shared task @xcite is a benchmark task for summarizing, classifying, and generating clinical dialogue data, and task b @xcite is a problem of generating a full clinical note from doctor-patient conversations. this paper describes how we designed and addressed task b in mediqa-chat 2023 shared task. task b is a problem that takes clinical consultation dialogues between doctors and patients as input and generates a summarized full clinical note, as shown in ## related work abstractive summarization of extended conversations is typically approached using generative models such as bart @xcite . when the input exceeds the model's input length constraint, rudimentary techniques such as truncating the end or middle portions of the input are employed. methods like presumm @xcite address long inputs by extracting key sentences and subsequently performing abstractive summarization on them. alternatively, models like dialogbert @xcite process inputs at the utterance level, encoding and merging them to accommodate the entire conversation. recently, a trend towards models like longformer @xcite , dialoglm, and dialogled @xcite has emerged, as these models can handle longer inputs owing to their expanded input length capabilities. prominent datasets for summarizing long conversations include ami @xcite and icsi @xcite datasets. these datasets, like task b, involve summarizing lengthy dialogues. however, unlike task b, where the domain is clinical, the domain for these datasets is meetings, and the summaries are composed of unstructured paragraphs. and, models such as clinicalbert @xcite , which are trained in the clinical domain, have emerged, but they have limitations in terms of input length compared to recent models. ## dataset and task design the task b of mediqa-chat 2023, as presented in considering these aspects, the challenges to be addressed in this task, distinct from other summarization tasks, can be summarized as handling long dialogues, working with a small dataset, generating semi-structured notes, and adapting to the intricacies of the clinical domain. ## method to address these challenges, we employed two primary approaches. the first approach involves using the entire dialogue as input and generating the full clinical note as output, thereby adopting an end-toend learning method. the second approach consists of creating notes for each section individually, and subsequently merging the section-specific notes to form a comprehensive full clinical note. ## conclusion in this paper, we have conducted various experiments for task b and analyzed the results. we utilized the dialogled model to handle long inputs and employed additional data, such as the ami and icsi datasets, to address the limited data issue. as a result, the model generating the entire note at once achieved high scores for the full note but low scores for section header creation. in contrast, the approach of creating section summaries separately and then combining them for a complete note had high scores for section headers but low scores for the entire full note due to repetition issues. analyzing these results, it appears that generating the entire content at once is a more appropriate approach since it is challenging to avoid repetitive content when creating section summaries separately. in conclusion, to learn the characteristics of task b, which requires the creation of a semi-structured structure, a hybrid approach that combines generating the entire content at once and creating section headers separately is needed, rather than relying on simple fine-tuning.