H ealth A lign S umm : Utilizing Alignment for Multimodal Summarization of Code-Mixed Healthcare Dialogues (2024)

Abstract: As generative AI progresses, collaboration be-tween doctors and AI scientists is leading to thedevelopment of personalized models to stream-line healthcare tasks and improve productivity.Summarizing doctor-patient dialogues has be-come important, helping doctors understandconversations faster and improving patient care.While previous research has mostly focused ontext data, incorporating visual cues from pa-tient interactions allows doctors to gain deeperinsights into medical conditions. Most of thisresearch has centered on English datasets, butreal-world conversations often mix languagesfor better communication. To address the lackof resources for multimodal summarization ofcode-mixed dialogues in healthcare, we devel-oped the MCDH dataset. Additionally, we cre-ated HealthAlignSumm, a new model that in-tegrates visual components with the BART ar-chitecture. This represents a key advancementin multimodal fusion, applied within both theencoder and decoder of the BART model. Ourwork is the first to use alignment techniques,including state-of-the-art algorithms like DirectPreference Optimization, on encoder-decodermodels with synthetic datasets for multimodalsummarization. Through extensive experi-ments, we demonstrated the superior perfor-mance of HealthAlignSumm across severalmetrics validated by both automated assess-ments and human evaluations. The datasetMCDH and our proposed model HealthAlign-Summ will be available in this GitHub accounthttps://github.com/AkashGhosh/HealthAlignSumm-Utilizing-Alignment-for-Multimodal-Summarization-of-Code-Mixed-Healthcare-DialoguesDisclaimer: This work involves medical im-agery based on the subject matter of the topic.

## introduction in india, the glaring disparity in doctor-to-patient ratios among different states and sectors exacer- * these authors contributed equally. following this trend, @xcite introduced a multimodal dataset for this task of medical dialogue summarization. however, in countries like india, statistics show that people often switch between languages like english and hindi during conversation for easier communication; a new challenge arises 1 . motivation: in our study, we address the gap in healthcare communication by creating mcdh, a codemixed hindi-english dataset reflecting realworld doctor-patient interactions. since there was no hindi-english codemixed dataset available for this task, we have leveraged cutting-edge large language models (llms) to fill in the gap. we have created a novel technique called hintbag for generating quality codemixed datasets for this task using a technique called hinting. this dataset aims to enhance communication in healthcare settings by accommodating linguistic diversity. our work is the first attempt at multimodal codemixed doctor-patient dialogue summarization, especially in the indian setting. regarding architecture, existing works like medsumm @xcite concatenate image and text vectors for summary generation, whereas km-cliconsummation @xcite demonstrates the benefits of multimodal fusion into bart's encoder. @xcite is the most recent work in this domain where they have introduced multimodal attention in both the encoder and decoder of the bart model. however, no work is done in the utilization of recent post-processing alignment techniques like direct preference optimization (dpo) @xcite using synthetically generated preference datasets for this task of multimodal summarization. our study introduces healthalignsumm to investigate this approach. the overall working of healthalignsumm is shown in figure 1 . ## related works the following works have been relevant to the following two research areas, namely (a) medical question summarization and (b) the role of multimodality in summarization. ## mcdh dataset in this study, we used the mmcliconsumm dataset @xcite , the only publicly available multimodal medical dialog summarization dataset. each data point includes a medical conversation along with visual cues and their corresponding summaries. due to the lack of suitable code-mixed healthcare datasets for this task, we created a highquality synthetic dataset for our study. we aimed to create synthetic code-mixed datasets that should convincingly mimic real-world doctor-patient conversations. extensive brainstorming with doctors and linguists revealed that complex medical terms should remain in english, as patients generally understand these better. for instance, "cancer" is preferred over "arbud". following this, we established a high-level template for our synthetic code-mixed medical data. linguists annotated around 5% samples, converting english conversations into hinglish, which were verified by a doctor foot_0 . using gpt-3.5 turbo in a few-shot setting foot_1 , we converted the conversations into code-mixed hindi-english dialogues, leveraging the model's ability to follow language patterns. while initial results were promising, further analysis showed some words were unnaturally forced into hindi, making the conversations awkward. to address this, we developed a novel prompting technique called hintshot, providing hints on which words should remain in english based on context. ## methodology this section elaborates on the working of different modules of our proposed healthalignsumm model. ## experimental results and analysis in the subsequent section, we discuss the experimental framework and assess the effectiveness of the proposed model healthalignsumm using a wide range of evaluation metrics concerning various baselines, encompassing both automated and human-centric assessments. furthermore, we conduct a qualitative examination of the generated summaries across different model configurations. foot_7 ## limitations there are some noticeable limitations of our work. they are enumerated in the points below: ## ethics statement our project utilized the mcdh dataset, which is a refined version of the publicly available mmcli-consumm dataset. throughout the process, we collaborated with a medical expert, who is also a co-author of this paper, to ensure accuracy and quality control at every stage, from data collection to validation. to uphold ethical standards, we compensated all volunteers in alignment with india's minimum wage regulations. privacy was paramount in our approach, and we diligently ensured the dataset was free of any images that could compromise individuals' privacy. additionally, we are currently seeking approval from the institutional review board (irb) to further reinforce the ethical integrity of our work. notably, our proposed model is exclusively designed for summarization tasks, deliberately avoiding any predictive functions that could have unintended consequences for users. this decision reflects our deep commitment to ethical principles and responsible use of ai technologies. in critical sectors such as healthcare, the potential for misinformation or misinterpretation poses substantial risks. our human evaluation revealed instances where the model, influenced by visual data, overlooked essential keywords within textual queries. similarly, qualitative analysis identified instances of hallucinations in the model's output. yet, employing dpo alignment has notably mitigated the occurrence of hallucinated information in the final summaries. moreover, we've observed that image quality plays a crucial role in generating accurate summaries, as poor quality images can introduce noise into the output. consequently, while healthalignsumm enhances the generation of nuanced summaries, it's imperative to involve medical professionals in high-stakes scenarios. our model serves as an aid rather than a substitute for the expertise of experienced physicians.