IKIM at MEDIQA - M 3 G 2024: Multilingual Visual Question-Answering for Dermatology through VLM Fine-tuning and LLM Translations (2024)

Abstract: This paper presents our solution to the MEDIQA-M3G Challenge at NAACL-ClinicalNLP 2024. We participated in all three languages, ranking first in Chinese and Spanish and third in English. Our approach utilizes LLaVA-med, an open-source, medical vision-language model (VLM) for visual question-answering in Chinese, and Mixtral-8x7B-instruct, a Large Language Model (LLM) for a subsequent translation into English and Spanish. In addition to our final method, we experiment with alternative approaches: Training three different models for each language instead of translating the results from one model, using different combinations and numbers of input images, and additional training on publicly available data that was not part of the original challenge training set.

## introduction over the past 25 years, various studies have discussed the shortage of dermatologists in the us @xcite @xcite . at the same time, machine learning methods offer potential relief for the limited time available to dermatologists (fogel and kvedar, 2018) and, on some tasks, even exceed expert capabilities @xcite . recently introduced vision-language models (vlms) showed promising capabilities in radiology and pathology visual question-answering (vqa) tasks @xcite @xcite @xcite . therefore, it can be assumed that they also provide relief in the field of dermatology. however, there are no existing dermatology vqa datasets @xcite . yet, vlms need fine-tuning datasets to achieve the high accuracy required for medical tasks @xcite . a possible data source for such tasks are telemedical records. telemedicine describes triag-ing, diagnosing, and monitoring patients remotely through digital images and text messages @xcite . shortly after the outbreak of the covid-19 pandemic, the availability of telemedicine services increased in parts of china @xcite , providing new opportunities to create vqa datasets. following these developments, the mediqa-m3g challenge @xcite is based on data from one of these telemedical platforms. the participants are offered photos of skin diseases and textual interactions between patients and medical professionals. while the original data is in chinese, automated translations into english and spanish were also provided. this raises several questions that we examined in the course of the challenge. first, there is the question of which model should be used on the chinese data since all medical vlms were trained in english. another question is how helpful the training on the translated english and spanish data is or whether problems such as translation errors and cultural differences are a hindrance. to answer these questions, this paper compares various fine-tuning methods in preparation for our challenge submission. we first evaluate the usefulness of additional imaging data from two publicly available dermatological classification datasets in solving the challenge. we then compare multiimage training to training with a single image per data entry. finally, we also test if training three different models for each language outperforms training a single model and translating its predictions into the other two target languages. ## challenge dataset the given dataset (wai yim et al., 2024b) consists of clinical history and patient query examples. along these textual inputs, one or multiple photos of the described skin disease were attached to the query. the gold labels consisted of one or multiple answers by chinese dermatologists. all texts were machine-translated into english and spanish without further information on which model was used for translation. one exception is the test set, which was translated manually. for the validation and test sets, annotators also provided a score indicating how complete an answer is concerning the query. possible completeness scores were 0.0, 0.5, and 1.0, ranging from incomplete to entirely complete. as a metric, the deltableu score @xcite was computed between predictions and dermatologists' answers using the completeness score for weighting. the training set consists of 842 patient queries with an average of 2.94 images per query. additional 56 examples were provided as a validation set and 100 examples as a test set. figure 1 shows the histogram of the number of words per dermatologist answer for the english training set. most answers consist of only a few words, usually the diagnosis. however, some outliers are considerably longer, with over 315 words. these answers contain lengthy descriptions of the treatment and follow-up steps for the patient. while we manually analyzed the data, we could not find a consistent relationship between the type of request and the verbosity of the response. figure 2 shows the histogram of the number of images per sample in the training set. like the number of words per answer, queries usually have few images attached. ## methodology the following section describes the different approaches for the challenge. we describe multiimage training, additional non-challenge data used, and our methods of translating lvm predictions into new target languages. ## discussion in addition to the main result described above, we performed several additional experiments with differing approaches, which in most cases led to worse performance than in the version we submitted. ## limitations the model we submitted has some limitations, excluding it from clinical use in its current state. most importantly, even though our results scored the highest in two languages, the overall scores were very low. significantly higher diagnosis accuracy must be achieved to make it useful in a clinical setting. secondly, due to the nature of the training data, the model often mimics the writing style of the forum posts it was trained on, leading to fewer professional expressions than expected in a clinical setting. similarly, since training data was obtained from chinese sources containing frequent suggestions for using traditional chinese medicine, the model made similar recommendations in some cases. this might not meet the standards of care in other countries. thus, regional differences in care methods have to be considered when training similar models intended for clinical use in the future. ## conclusion we present our submission to the multilingual & multimodal medical answer generation task of the mediqa 2024 challenge. our results compare well with other submitted approaches, but their quality is still insufficient for clinical use. this was partly because our method could not overcome hurdles presented by the challenge, such as short target predictions, translation issues, and regional differences in care methods. vlms with better analytic capabilities in the medical domain must be created to achieve scores high enough for real-world applications. nonetheless, the increased availability of telemedical records and the increasing availability of data from a variety of countries also presents an opportunity for medical lvm research.