IUTEAM 1 at MEDIQA -Chat 2023: Is simple fine tuning effective for multi layer summarization of clinical conversations? (2023)

Abstract: Clinical conversation summarization has become an important application of Natural language Processing. In this work, we intend to analyze summarization model ensembling approaches, that can be utilized to improve the overall accuracy of the generated medical report called chart note. The work starts with a single summarization model creating the baseline. Then leads to an ensemble of summarization models trained on a separate section of the chart note. This leads to the final approach of passing the generated results to another summarization model in a multi-layer/stage fashion for better coherency of the generated text. Our results indicate that although an ensemble of models specialized in each section produces better results, the multi-layer/stage approach does not improve accuracy. The code for the above paper is available at https://github.com/dhananjay-srivastava/MEDIQA-Chat-2023-iuteam1.git

## introduction with the increasing adoption of electronic health records (ehrs), physicians and other healthcare professionals are spending an increasing amount of time entering data into ehr systems during patient encounters. it has been estimated that physicians spend approximately 16 minutes per encounter entering data into ehrs @xcite . this process can be time consuming and lead to physician burnout. @xcite in addition, the sheer volume of data generated during a patient encounter can make it difficult for physicians to identify and interpret the most relevant information quickly. to address these challenges, ai summarization models are being developed that can automatically extract and summarize the most important information from clinical conversations @xcite . these models can be trained on large datasets of clinical conversations to learn to identify important information such as symptoms, diagnoses, medications, and treatment plans. once trained, these models can be used to automatically generate summaries of clinical conversations. these summaries can be used to generate first drafts of reports, called chart notes, that must be prepared after each encounter with the patient. there are significant challenges in the implementation of these models. @xcite such as the lack of sufficient training data, ethical and regulatory requirements around sensitive medical data, and the use of specialized medical terminology. the limited availability of clinical data due to privacy concerns makes it difficult to gather a diverse dataset to train the models. moreover, medical jargon and terminology used by healthcare professionals can vary widely depending on the context, making it challenging to develop models that can accurately identify and summarize critical information. in this work we explore 3 approaches of combining transformer-based summarization models towards identifying an optimal high-level structure of ensembling multiple summarization models for the task. ## background and prior art in view of the challenges discussed in the previous section, choosing the correct model architecture is crucial. the chartnote is a special document and involves multiple sections each with its own distinct style and content. the purpose of this report is to analyze at a high level, given a transformer-based summarization model transformer @xcite based architectures have come to dominate the summarization task. an important challenge in clinical conversation summarization is that the input conversations typically do no fit inside the input token limits of standard models like @xcite and bart @xcite . to overcome this challenge models such as longformer @xcite , big bird (et al, 2020) and lsg bart @xcite have been proposed. we choose the lsg bart model as a sample summarization model to analyze our hypothesis on proper choices for ensembling. lsg bart builds upon bart (bidirectional and auto-regressive transformer) @xcite which is a variant of the popular transformer @xcite architecture that combines the power of auto-regressive and denoising autoencoder training objectives to generate high-quality summaries. however, the primary limitation in using bart is that it can only accept 1024 tokens. to address this issue, @xcite have introduced a new technique called lsg attention, which can enhance the performance of bart and other summarization models. lsg attention is a combination of three types of attention mechanisms: local attention, sparse attention, and global attention. in local attention, the input sequence is split into multiple non-overlapping blocks, and attention is calculated within and between these blocks. sparse attention allows each attention head to process different sparse sets of tokens independently, which can improve the computational efficiency of the model. global attention, similar to the cls token, uses a global token to calculate attention across the entire input sequence. thus this particular model should be suitable for our use case of long document summarization. ## dataset and challenge details the mediqa-chat 2023 challenge is a part of the 5th clinical nlp workshop at acl 2023 (wai yim et al., 2023) on improving nlp technology for clinical applications. the challenge has 3 subtasks. task a (ben abacha et al., 2023) is focused on generating specific sections while task b (?) is focused on generating the full note based on the conversation. task c is focused on generating the conversation back from the note. the dataset for task b (?) consisted of a training and validation set with 67 and 20 conversations and their summaries respectively. an additional hidden test set of 40 conversations was released to the participants and the final results were calculated using the rouge, bleurt and bertscore metrics by the competition organizers. this work is focused on task b. ## methods in order to investigate the hypotheses claimed in the background section, we divide the problem into 3 separate tests using the lsg bart model. we also investigate whether finetuning on medical research papers from pubmed is useful in domain adaptation and whether it leads to an accuracy increase. the approaches are as follows: ## results discussion as previously stated, all three approaches in our study utilize the lsg bart model implementation @xcite . to train the model, we implement a decaying learning rate starting at 5e-5, gradually decreasing the learning rate over time. we train the models for a total of 20 epochs using a single nvidia a100 gpu and utilize mixed precision training with fp16 set to true for faster training speed with minimal loss to accuracy. to assess the performance of the models, we evaluate the generated summaries using the rouge metric @xcite , which assesses the degree of overlap between the n-grams in the generated summary and those in the reference summary. the primary reason behind these lower scores is probably that pubmed data is based on medical literature rather than medical conversations. moreover, the model was finetuned on this dataset rather than pre-trained thus the model is trained to summarize medical literature but the token embeddings are not necessarily finetuned for our purpose. thus we did not pursue this model further. we submitted all the 3 approaches to mediqa-chat challenge task b, the evaluation consisted of 2 parts. in the first part, models from all the different teams were compared on the rouge scores for the full chart note, in the second part the comparison was done sectionwise. the results calculated by the workshop organizers on the hidden test set are. we observe that as in approach 2 having an ensemble of models each specializing upon a section of the chart note produces better results than the baseline for all sections. however, attempting to pass the results to another lsg bart model as in approach 3 fails to generate better summaries evidenced by the extensive drop in accuracy for assessment and plan and exam sections. thus model coherency is not improved by simple fine-tuning of a multi-layer/stage summarization ensemble. in the competition, approach 2 secured 7th and 5th places respectively for full note and sectionwise text generation. approach 1 secured 16th and 17th positions and approach 3 secured 21st and 19th place respectively among the 23 models submitted by the different teams. the competitive ranking of the models and better than baseline performance indicates that ensemble summarization 4 522 models hold promise and should be investigated further as a viable strategy for clinical conversation summarization. ## conclusion and future work the results indicate that ensembling multiple summarization models depending upon the specific section of the chart note they are producing is a viable strategy for improving summarization quality. our results also indicate that simply passing the ensemble results to another summarizer does not directly improve accuracy and add that further tests with larger datasets and statistical analyses are required to obtain conclusive answers. in the future, we would like to perform in-depth rigorous analyses on model architectures to support section-wise next generation as well as study many of the other models used in long document summarization to improve overall accuracy.