LTRC - IIITH at MEDIQA - M 3 G 2024: Medical Visual Question Answering with Vision-Language Models (2024)

Abstract: In this paper, we present our work to the MEDIQA-M3G 2024 shared task, which tackles multilingual and multimodal medical answer generation. Our system consists of a lightweight Vision-and-Language Transformer (ViLT) model which is fine-tuned for the clinical dermatology visual question-answering task. In the official leaderboard for the task, our system ranks 6th. After the challenge, we experiment with training the ViLT model on more data. We also explore the capabilities of large Vision-Language Models (VLMs) such as Gemini and LLaVA.

## introduction the rapid evolution of telecommunication technologies, coupled with increased healthcare demands and the recent challenges posed by the pandemic, has accelerated the adoption of remote clinical diagnosis and treatment. alongside conventional live consultations conducted via telephone or video, asynchronous methods such as e-visits, emails, and messaging chats have emerged as practical and cost-effective alternatives. this task @xcite focuses on addressing the challenge of generating suitable textual responses to queries in clinical dermatology, taking into account multimodal inputs such as clinical history, queries, and accompanying images. this paper describes our proposed solution. we fine-tune vilt @xcite for the visual question-answering task with the training data provided for the challenge. we choose vilt due to its lightweight nature and ability to handle both visual and textual inputs efficiently. after the challenge, we also explore how large vision-language models (vlms) such as gemini @xcite and llava @xcite perform in this task. these models stand at the top of the multi-modal benchmarks such as massive multi-discipline mul-timodal understanding and reasoning benchmark (mmmu) @xcite . ## related work previous research has predominantly focused on consumer health question-answering but has been limited to textual inputs @xcite . similarly, existing work on visual questionanswering has primarily concentrated on radiology images, lacking integration with additional clinical text inputs @xcite . moreover, while significant research has been conducted on dermatology image classification, the emphasis has largely been on lesion malignancy classification for dermatoscopy images @xcite . recently, there has been a surge in the development of multimodal models, particularly large vision-language models (vlms). these models integrate both textual and visual information, allowing them to understand and generate content that combines both modalities. vlms typically employ techniques such as joint embedding to unify the representations of text and images in the same embedding space. during training, they utilize datasets that contain interleaved text and images, enabling the model to associate textual descriptions with visual content effectively. this process enables vlms to grasp nuanced relationships between words and visual elements, facilitating tasks like visual question-answering (vqa). in vqa, these models can accurately respond to questions about images by understanding the content of both the image and accompanying text, showcasing their ability to comprehend and synthesize information across modalities. for the medical domain, vlms have been trained on medical corpora and developed for various clinical tasks. medblip @xcite aids computer-aided diagnosis (cad) in the medical field. it tackles the challenge of combining image and text data from electronic health records for medical diagnosis. the model shows promising results in classifying healthy, mildly impaired, and alzheimer's patients and also demonstrates the ability to answer medical questions based on visual information. pmc-llama @xcite , designed specifically for medical applications, demonstrates superior performance on medical question-answering tasks. med-flamingo @xcite is a model that can learn from small datasets by embracing in-context learning for the multi-modal medical domain. biomedgpt @xcite ) is a unified model designed to handle diverse medical data and perform various tasks such as diagnosis and summarization. llava-med @xcite utilizes a massive dataset of biomedical images and captions from pubmed central and employs the powerful language model gpt-4 to create diverse training examples. ## dataset the dataset provided for the shared task is der-mavqa: a multilingual visual question answering dataset for dermatology @xcite . it is translated and adapted from chinese telemedicine datasets. the given dataset consists of 842 samples in the training set, 56 for validation and 100 for testing. each sample has a query, clinical history, and one or more associated images. the textual content is provided in three languages -chinese, english, and spanish. the english and spanish versions of the training set are generated through machine translation from the chinese original. the validation and test datasets are manually translated by human translators. this approach allows for comprehensive testing and validation of models across multiple languages while ensuring the integrity and quality of the data through both machine and human translations. ## after-challenge experiments after the challenge, we develop a system containing two modules -visual question-answering module and translation module. we experiment with two models -gemini 1.0 pro vision @xcite and llava-1.6 34b @xcite . ## evaluation metrics the evaluation process utilizes deltableu @xcite , a metric that accounts for multiple correct responses. these responses are weighted based on various factors, including completeness, consistency with the most commonly provided answer as determined by human assessment, as well as author rank level and author validation level. the completeness metric assigns a score on a scale of {0.0, 0.5, 1.0}, indicating the extent to which the original query's question was addressed. a score of 1.0 signifies a fully answered query, 0.5 indicates a partial response, and 0.0 reflects a lack of response. if the query doesn't explicitly specify, it's assumed to seek information on both the disease and its treatment. contains most frequent answer rating is given on a scale of {0.0, 1.0}. a score of 1.0 is assigned if the response aligns with the most frequently provided answer. ## results the evaluation results of the different experiments can be seen in table-2. we submitted the system of vilt trained on 200 samples for the challenge. our submission ranks 8th in the leaderboard. ## conclusion this work investigates vision-language models like vilt, gemini, and llava for the challenging multilingual and multi-modal medical answer generation task in dermatology. a modular system with separate visual qa and translation components shows improved performance over the initial vilt approach. a key strength is leveraging powerful multi-modal models that can effectively integrate visual and textual clinical data. future efforts should focus on utilizing larger, more diverse datasets, incorporating stronger multi-modal reasoning, and rigorous evaluations by medical experts to ensure clinical utility and safety before real-world deployment. overall, this mult-imodal approach holds promise but requires further advancements to be reliable for remote diagnosis and treatment.