TEAM MIPAL at MEDIQA - M 3 G 2024: Large VQA Models for Dermatological Diagnosis (2024)

Abstract: This paper describes the methods used for the NAACL 2024 workshop MEDIQA-M3G shared task for generating medical answers from image and query data for skin diseases. MedVInT-Decoder, LLaVA, and LLaVA-Med are chosen as base models. Finetuned with the task dataset on the dermatological domain, MedVInT-Decoder achieved a BLEU score of 3.82 during competition, while LLaVA and LLaVA-Med reached 6.98 and 4.62 afterward, respectively.

## introduction the advancement of telecommunication technologies and the increasing demand for healthcare services have accelerated the demand for remote disease diagnosis and treatment. however, existing medical-related multimodal problems have predominantly focused on general diseases or radiology image analysis. in this task, abnormal skin images along with a single conversational query from the patient are provided as inputs. the goal is to utilize a multimodal model to identify the patient's condition and generate appropriate responses from the physician tailored to the patient's situation. to address this problem scenario, we finetuned three multimodal vqa models, medvint-decoder @xcite , llava @xcite , llava-med @xcite . when finetuned with the task dataset @xcite , the bleu scores of the medvint-decoder, llava, and llava-med were 3.82, 6.98, 4.62.the results for llava and llava-med were submitted after the challenge. since the released train dataset was small, we further explored ways to augment this data. specifically, we crawled skin disease images online and synthesized queryresponse pairs using gpt-3.5. however, models 2 related works ## results examples of the generated responses are provided in medvint-decoder shows low bleu score of 0.91 when the inference is made directly on the test set without any finetuning. during training, medvint-decoder reaches the lowest validation error after around 3 epochs and starts to show signs of overfitting afterwards. early stopping is introduced to make use of such pattern. the bleu score measured with the validation set was the highest when trained with learning rate of 4e-6, batch size 16 and epochs 10. the train-* after-challenge submission ing process stopped early at epoch 3. the bleu score was 4.48 on the validation set and 3.82 on the test set. although not impressively high, the increased values prove that actual learning has been conducted. in case of llava, inferencing with the vanilla model without any finetuning yielded a bleu score of 0.93 which is similar to that of medvint. upon finetuning the model with 29 diseases set and 324 diseases set, the bleu scores on the validation dataset showed 0.97 and 4.51, respectively. the latter seemed promising, but did not meet the expected performance when inferenced on the test set. llava-med scoreed a bleu score 1.35 on zero shot inference. afterwards, when finetuning on the task data, llava-med achieved a bleu score of 3.82 and bert score 0.84. ## discussion the necessity of finetuning although there already exist models pretrained with medical data, they mostly failed to give high-quality answers without further finetuning on the domain specific data. two observations were made to explain this phenomenon. one reason would be the difference of the image domain. both pmc-vqa and llava-med training set are composed of professional images such as x-ray, mri and ct. the raw photos are not many in these sets, and it becomes even scarcer when limited to skin diseases. therefore the task images would have been regarded as new and unfamiliar to the pretrained models. another possible reason would be the difference in the text domain. medvint-decoder trained with pmc-vqa has learned to give short answers for most of the time. pmc-vqa is consisted of simple yes or no questions, or those that can be answered with one or two vocabularies. this may not have been sufficient when it comes to diagnosis and prescription tasks that require the machine to give long answers. also, non-negligible amount of texts used for training is structured with certain formats or may be excerpts from academic texts. this is in contrast to the task dataset which is mostly written in casual spoken language. low performance when trained with crawled data upon observing lower-than-expected performance following finetuning of the model with additional crawled data, efforts were directed towards enhancing llava's performance after the competition. inspired by the impressive performance demonstrated by medvint upon finetuning with solely the workshop-provided training data, a similar approach was applied to llava and llava-med. llava in this method yielded bleu and bert scores of 6.98 and 0.86, respectively, representing the highest scores achieved on the test dataset as shown is ## conclusion we propose our submission to the mediqa-m3g shared task for generating medical responses to multimodal queries. in our study, three existing models medvint, llava and llava-med are finetuned using the competition dataset along with synthetically generated dataset. their performance are evaluated using bleu and bert score. our results indicate that utilizing only the task dataset leads to substantial improvements in both models, reaching the bleu score of 3.82 with medvint-decoder and ranked second in the english section of the workshop. after the competition, llava finetuned with the workshop dataset achieved the highest bleu score of 6.98 and lastly finetuned llava-med model with workshop dataset performed a bleu score of 4.62.