NEUI at MEDIQA - M 3 G 2024: Medical VQA through consensus (2024)

Abstract: This document describes our solution to the MEDIQA-M3G: Multilingual & Multimodal Medical Answer Generation. To build our solution, we leveraged two pre-trained models, a Visual Language Model (VLM) and a Large Language Model (LLM). We fine-tuned both models using the MEDIQA-M3G and MEDIQA-CORR training datasets, respectively. In the first stage, the VLM provides singular responses for each pair of image & text inputs in a case. In the second stage, the LLM consolidates the VLM responses using it as context among the original text input. By changing the original English case content field in the context component of the second stage to the one in Spanish, we adapt the pipeline to generate submissions in English and Spanish. We performed an ablation study to explore the impact of the different models’ capabilities, such as multimodality and reasoning, on the MEDIQA-M3G task. Our approach favored privacy and feasibility by adopting open-source and self-hosted small models and ranked 4th in English and 2nd in Spanish.

## introduction recent visual iterations of large language models (lmm) explore a central concept that deals with multimodal inputs, known as visual instruction tuning. these studies result in sizable visual language models (vlm) such as visualbert @xcite , llava @xcite , minigpt-4 @xcite ) that demonstrate impressive results on natural instruction-following and visual reasoning capabilities. the need for multimodal models is particularly pronounced in the medical domain. medical visual question answering (vqa) can assist in clinical decision-making, provide reliable and user-friendly answers to free-form questions, serve as a diagnostic tool, or act as a knowledgeable assistant, * all authors contributed equally. the mediqa-m3g task, which focuses on clinical dermatology multimodal query response generation, exemplifies this need. this task aims to automatically generate clinical responses given textual clinical history, user-generated images, and queries. the common challenges of vqa are amplified in the medical domain, where highly specialized knowledge must be leveraged in coordination with specific visual features from the images. this task is further complicated by the fact that the query, content, and images are provided by patients, which implies a highly heterogeneous text style, varying levels of description details, and, in the case of images, highly variable light, focus, zoom, and quality conditions. we utilized a compact (1.86b parameters) visual language model (vlm) named moondream @xcite to evaluate the performance of small language-image models (lims) on the m3g multimodal task. moondream is built upon a sigmoid loss for language-image pre-training (siglip) and the phi-1.5 language model. we finetuned the vlm using the provided training data, extending each case title and description to all the provided images. the output of vlm might contain redundancies and short answers that deviate from the provided context in the query. we implemented a postprocessing step of the vlm output to address this issue by constructing a new query for a fine-tuned biomistral llm. this step relies on the idea that we already have the context to improve the vlm answer. the context consists of the original query title and content from the test dataset cases and the vlm response, which we refer to as image analysis. along with the context, we used the general query "what is the disease present in the photo? what is the treatment?" we use the same pipeline for both english and spanish submission entries. ## task definition the mediqa-m3g: multilingual & multimodal medical answer generation task focuses on the problem of clinical dermatology multimodal query response generation, a first of its kind, aiming to automatically generate clinical responses given textual clinical history, user-generated images, and queries @xcite . this shared task is motivated and very in line with the rapid development of telecommunication technologies and the increased demand for remote clinical diagnosis and treatment. unlike previous works focusing only on text or specific types of images, this task incorporates text and one or more images. participants were given textual inputs, including clinical history and a query, along with associated images, and they were expected to generate a relevant textual response. the training data for this task was translated and adapted from chinese datasets, and participants could opt to work in chinese (simplified), english, or spanish for the test set @xcite . ## methodology we utilized a compact (1.83b parameters) visual language model (vlm) named moondream @xcite to evaluate the performance of small language-image models (lims) on the m3g multimodal task. moondream is built upon a sigmoid loss for language-image pre-training (siglip) @xcite and the phi-1.5 language model, a transformer with 1.3b parameters @xcite . in such a setup, a contrastively pre-trained model provides significantly more useful tokens than one classification pre-trained model @xcite . figure 1 shows the schematic of the proposed solution involving the vlm and the biomistral-7b-dare @xcite llm as a specialized stage for final response consolidation. ## results and analysis results during competition. from the official results during the competition ( use licorice decoction with corticosteroid ointment. from: agent ; value: erythema annulare centrifugum? use licorice decoction with corticosteroid ointment. from: response 2; value: i think it still looks like urticaria, continue with the anti-allergy treatment. from: response 3; value: i think the likelihood of urticaria is the highest, but the skin lesions at the root of the thigh are hard to explain, so erythema annulare cannot be ruled out either... ## limitations our proposed approach holds significant promise for the vqa problem in clinical dermatology , but several limitations associated with deploying vlms and llms in real-world medical settings necessitate careful consideration. we are optimistic about the potential of our 2step method, which is designed to consolidate multiple responses from image-text query pair analysis into a single, consensus-based solution. this approach allows us to utilize simpler vlms, initially intended for single-image scenarios, in settings with an unknown number of multiple images. however, we acknowledge that this flexibility comes with the cost of posing as many queries to the vlm as there are images in a single case. this could increase computational costs, potentially making the solution computationally impractical for real-world deployment. another limitation is that vlms, which were aligned with domain-specific images and texts during pretraining, are observed better to leverage domain-specific training examples during the instruction tuning phase. however, in our setting, the vlm lacks alignment for visual-medical texts and relies solely on fine-tuning to generate the most appropriate answers. this lack of alignment makes the vlm more demanding for instruction tuning data. another crucial problem with vlms based on the pre-trained vision encoder is resolution. they are trained and also expect to analyze the full image input. however, for some specific cases, and even if the input is big enough, the focus of the query relies on certain zones of the image -in extreme cases, these zones are tiny compared to the image resolution. the m3g dataset showcases this very problem. most dermatology-related images in either training, validation, or test datasets contain wide shots of the patient's limbs, and only a tiny region of the image provides valuable visual cues. we envisioned exploring techniques such as visual search @xcite and visual cropping @xcite that can help us tackle this issue without compromising the size of our affordable vlm. regarding the llm component, even if we are within the considered "small" scale, the computational demands of these models are substantial. operating such models requires significant computational resources, which may not be feasible in all clinical environments. this issue can hinder our proposed solution's scalability and practical deployment in resource-limited settings. moreover, llms are prone to generating "hallucinations" or outputs that may include incorrect or misleading information. this phenomenon is particularly concerning in the medical field, where accuracy is crucial to avoid misdiagnoses or inappropriate treatments. intrinsic hallucinations, where outputs logically contradict known facts, and extrinsic hallucinations, where outputs cannot be verified, both pose serious risks in clinical applications. additionally, data bias and patient privacy are critical. llms trained on biased data can perpetuate or amplify these biases, leading to skewed or unfair medical advice. for example, the competition dataset observed a frequent recommendation based on traditional chinese medicine. thus, a model trained on this dataset may exhibit a predisposition, favoring a certain type of recommendation, irrespective of local or user-specific preferences. given the sensitive nature of medical data, ensuring patient privacy while using such models is also paramount. furthermore, updating these models with new medical knowledge remains a complex and resource-intensive process. this is problematic in the fast-evolving field of medicine, where staying current with the latest research and clinical findings is essential. for example, if a new adverse drug effect is discovered, it is vital to update the models' knowledge promptly. finally, it is necessary to be aware that although this shared task is a crucial step towards better understanding and addressing the relevant task of automatically generating clinical responses given the textual clinical history and user-generated images, similar to existing benchmarks and metrics, it does not imply a comprehensive assessment of the performance of the system in real-medical contexts. metrics such as trustworthiness, helpfulness, explainability, and faithfulness are crucial for clinical applications, and addressing these issues involves not only technical advancements in the architecture and training of llms but also close collaboration with medical professionals to ensure the clinical validity and ethical deployment of these technologies. in conclusion, while our solution shows promise in addressing the mediqa-m3g task, limitations must be addressed to make it suitable for clinical use. further exploration of optimization strategies, evaluation with other metrics, and collaboration with medical professionals are necessary to improve our approach's clinical relevance and effectiveness in real-world healthcare settings. ## future directions we plan to incorporate a broader array of medical and health-related datasets into our training regimen to enhance our models' domain-specific accuracy and relevance. specifically, we aim to utilize the skin condition image network (scin) dataset @xcite focused on dermatology, including structured and unstructured textual data. moreover, we are interested in exploring the potential benefits of integrating data from various clinical specialties into our training process to see how this affects the model's performance and applicability across different medical fields. we are particularly keen on incorporating retrieval-augmented generation (rag) strategies related to the challenges of model knowledge updating and mitigating hallucinations. these strategies leverage existing related medical knowledge during the inference phase to enhance the factuality of the generated responses. by doing so, we expect to improve the reliability and accuracy of the model outputs, which is crucial for clinical applications. finally, we recognize the importance of interdisciplinary collaboration in developing medical vlms and llms. therefore, we are already in plans to initiate partnerships with medical professionals who can provide valuable insights, contribute relevant training data, and help define the desired outcomes for these technologies. their involvement is critical not only in the development phase but also in testing these models in real-world clinical scenarios to ensure they meet both practical clinical needs and high standards of medical care. ## conclusions we explored a solution to the clinical dermatology multimodal query response generation task and proposed a pipeline that can be expanded to similar multimodal tasks. we leverage performant pre-trained language models, fine-tuning the small vlm to adapt to the clinical task. we also show how the pipeline adapts to the multilingual complementary problem by relying on the multilingual capabilities of the pre-trained llm. participating in this challenge represented a feasibility study and opened several work perspectives for multimodal medical applications.