M edi F act at MEDIQA - M 3 G 2024: Medical Question Answering in Dermatology with Multimodal Learning (2024)

Abstract: The MEDIQA-M3G 2024 challenge necessitates novel solutions for Multilingual & Multimodal Medical Answer Generation in dermatology (wai Yim et al., 2024a). This paper addresses the limitations of traditional methods by proposing a weakly supervised learning approach for open-ended medical question-answering (QA). Our system leverages readily available MEDIQA-M3G images via a VGG16-CNN-SVM model, enabling multilingual (English, Chinese, Spanish) learning of informative skin condition representations. Using pre-trained QA models, we further bridge the gap between visual and textual information through multimodal fusion. This approach tackles complex, open-ended questions even without predefined answer choices. We empower the generation of comprehensive answers by feeding the ViT-CLIP model with multiple responses alongside images. This work advances medical QA research, paving the way for clinical decision support systems and ultimately improving healthcare delivery.

## introduction dermatological telemedicine consultations, while offering a promising solution for remote diagnosis and treatment, face hurdles due to limitations in capturing subtle visual details and the inability to physically examine lesions. this can lead to miscommunication, such as difficulties in describing the texture or progression of lesions, which can hinder the development of effective treatment plans @xcite @xcite . however, recent advancements in imagetext learning, like vision transformer (vit) for image captioning and contrastive language-image pre-training (clip) for aligning text and image representations, offer promising avenues to bridge this gap @xcite . 1 fine-tuned models and code avaliable: https://github. com/nadiasaeed/medifact-m3g- @xcite existing approaches to teledermatology consultations have limitations. traditional consumer health question-answering systems primarily focus on textual data, neglecting the valuable information within visual details @xcite . this limits their ability to understand the nuances of skin conditions often best captured visually. visual question-answering efforts have mainly targeted radiology images, overlooking the crucial context provided by clinical text @xcite . while recent advancements in deep learning have shown promise in lesion classification for dermatology @xcite , these approaches often focus on specific image types and cannot integrate textual information, essential for a holistic understanding of a patient's condition. while some research explores combining clinical text and images for specific dermatology tasks, such as melanoma risk assessment, they haven't addressed open-ended question answering @xcite . this research tackles these limitations by introducing a novel framework for multilingual and multimodal query response generation in clinical dermatology. our system leverages the power of multimodal fusion, which combines information from different sources. in this case, the sources are textual and visual: textual clinical context and user queries in multiple languages, along with user-uploaded images. this work introduces medifact-m3g, a framework for tackling uncertainties in medical question answering for dermatology shown in figure 1 . medifact-m3g prepares the data and assigns weights to potential answers, considering their relevance and trustworthiness (section a). it then uses a powerful image analysis tool to extract key features from skin condition images (section b). by combining these features with text analysis, medifact-m3g leverages multiple powerful models to generate informative answers to medical questions (sections c and d). this framework has the potential to improve the accuracy and reliability of ai-powered diagnosis systems in telemedicine, ultimately assisting healthcare professionals in providing better diagnoses and treatment plans. this research addresses the following key questions: 1) can feature fusion from weakly supervised learning techniques effectively support open-ended medical question answering in dermatology? 2) can a medifact-m3g fine-tuned model trained solely on the mediqa-m3g training dataset adequately capture similarities and relatedness for unseen samples? 3) how can contrastive learning be seamlessly integrated with medifact-m3g to quantify uncertainty in response generation for ambiguous queries and limited content information? ## experimental setup and results we evaluated our model's capability in addressing the problem of clinical dermatology multimodal query response generation. this evaluation was conducted within the shared task of mediqa-m3g 2024, which focuses on multilingual and multimodal medical answer generation @xcite . as illustrated in figure 2 , each sample in the task comprised k medical images related to dermatological conditions, a textual query describing the user's skin concern, and its content. additionally, the ground truth for each sample included multiple possible responses with corresponding scores. leveraging the framework outlined in figure 1 , our medifact-m3g model was employed to generate answers in three languages for each sample. ## future work in the future, we plan to conduct further experiments to explore the robustness and scalability of our approach across larger and more diverse datasets. additionally, we aim to investigate the integration of domain-specific ontologies and medical terminologies to enhance the semantic understanding and accuracy of our system. furthermore,