W ang L ab at MEDIQA - M 3 G 2024: Multimodal Medical Answer Generation using Large Language Models (2024)

Abstract: This paper outlines our submission to the MEDIQA2024 Multilingual and Multimodal Medical Answer Generation (M3G) shared task. We report results for two standalone solutions under the English category of the task, the first involving two consecutive API calls to the Claude 3 Opus API and the second involving training an image-disease label joint embedding in the style of CLIP for image classification. These two solutions scored 1st and 2nd place respectively on the competition leaderboard, substantially outperforming the next best solution. Additionally, we discuss insights gained from post-competition experiments. While the performance of these two described solutions have significant room for improvement due to the difficulty of the shared task and the challenging nature of medical visual question answering in general, we identify the multi-stage LLM approach and the CLIP image classification approach as promising avenues for further investigation.

## introduction an increased demand for healthcare services and recent pandemic needs have accelerated the adoption of telehealth, which was previously underused and understudied @xcite . there has been significant recent interest in integrating artificial intelligence (ai) into telehealth @xcite , as these technologies have the potential to enhance and expand its ability to address important healthcare needs @xcite . the task of consumer health question answering, an important part of telehealth, has been explored actively in research. however, the focus of this existing research has been on text @xcite , which is limiting as medicine is inherently multimodal in nature, requiring clinicians to work not just with text but also with imaging among other modalities @xcite . to help address this gap, the mediqa-m3g shared task was proposed @xcite . this task requires the automatic generation of clinical responses given relevant user generated text and images as input, with a specific focus on clinical dermatology @xcite . this work describes our submission to this task. we explored two standalone solutions, one involving two consecutive api calls to the recently released claude 3 opus model @xcite and the other trains a joint image-disease label embedding model using clip @xcite for image classification. these two strategies took 1st and 2nd place respectively during the competition. while our strategy's effectiveness relative to other submissions highlight that claude 3 opus and multi-stage llm frameworks have potential value in the area of multi-modal medical ai, both our solutions' performance is limited despite their leaderboard success, highlighting the difficulty of the shared task and the unsolved challenge of medical visual question answering. ## shared task and provided dataset the mediqa-m3g competition focuses on the problem of clinical dermatology multimodal query response generation. the inputs include text which give clinical context and queries, as well as one or more images associated with the case @xcite . the task is to generate responses to these cases resembling those made by medical professionals in the field of dermatology. participants have the option to generate these responses in three languages: chinese (simplified), english, and spanish. @xcite the dataset consists of 842 train, 56 validation, and 100 test cases. each case consists of one or more images of skin conditions, their accompanying query text which may or may not include clinical context, patient queries, additional details regarding the disease and in some cases possible diagnosis. finally, for each case there are multiple responses made by one or more medical professionals, which are used as targets to score the model predictions. the cases also notably include metadata on the rank and validation level of the authors of content, which are used in evaluation @xcite . for evaluation, the competition uses a version of the deltableu @xcite metric to allow a single score to be computed based on word matching, weighted by the consistency (most frequent response) and the seniority of the medical professional across all responses given for that particular case. @xcite the query text and target responses are given in multiple languages, namely chinese, english, and spanish @xcite ). it's worth noting that while the test and validation sets were translated by medical professionals, the training set of 842 cases seems to be translated automatically with some potential room for errors. for our submission we focus on only providing the english solution. ## related work there has recently been a substantial amount of interest in medical applications of multimodal machine learning, and large multimodal models. some notable examples of research in this area include the open source llava-med model @xcite , and elixr, with the latter, similar to our work, exploring not only the application of large multimodal models, but also training a model using clip @xcite . however, while there has been significant focus on certain areas such as radiology, the area of dermatology has not been explored to the same extent. cirone et al. notably found that gpt-4v could accurately differentiate between benign lesions and melanoma @xcite . however, this is a much less challenging task than the one proposed in this shared task, as the problem space is much smaller in scope than responding to dermatology questions which are not necessarily in the train set, with even the conditions of interest not necessarily being in the train set. the limited performance of our solution, along with it being by far the best performing solution in this competition demonstrate the challenge of this task, and highlight the need for significant progress before deployment in a clinical setting. however, our work highlights potentially important directions for future research, including further investigation ## results upon examination of the evaluation metric and competition data, we have determined that a short response focusing on disease diagnosis alone is the most advantageous. this is due to two reasons. first, we notice both the training and validation sets often contain short responses, and in many cases merely the skin condition presented in the associated images. second, the evaluation metric's penalty for short responses is significantly smaller than a longer, partially correct response. given these initial findings, we evaluated two methods as outlined in 1 which took 1st and 2nd place in the english category of the leaderboard during the mediqa-m3g challenge by a significant margin over the next best submitted solution, the latter of which received a deltableu score of 3.827 during the competition. the methods will be elaborated in the following sections in detail. ## discussion we have presented two solutions to the mediqa2024-m3g competition, one involving api calls to an existing state of the art multimodal language model and the other involving the learning of an image-disease label joint embedding space for disease classification. the superior performance of using two separate api calls to claude 3 opus over one pass was interesting to observe. the increase in performance is likely attributed to the reduced ability for the model to simultaneously reason with the images while adhering to the added difficulty of only returning the disease label without any additional textual generation. this finding is somewhat consistent with how chain of thought reasoning can improve model performance by asking the model to first consolidate evidence present in the given image followed by making several differential diagnoses. further research such as @xcite also highlight the importance of using two-stage frameworks for multi-modal chain of thought that separate rationale generation and answer inference over one stage systems. for the clip based solution, we find it extremely encouraging that a smaller scale model finetuned on image-disease label pairs (@xmath0.884 vs 12.855). it perhaps demonstrates that smaller scale supervised training may sometimes outperform bigger more general purpose models for specific tasks of interest due to the advantage of training only on task specific examples. furthermore, our additional experiments after the competition highlights the importance of proper selection of batch size and retrieval method. we observe that while clip effectively constructs a joint embedding space between images and their disease labels, the image embeddings and text embeddings remain as separate cluster in pca space. as a result, we see that the nearest 5 neighbours in the text cluster for each embeded image (image-text) in the test set were much poorer in quality than those retrieved from the image cluster (image-image). ## limitations while both the claude 3 opus api based solution and the clip based image classification solution achieved first and second place during the mediqa-m3g competition respectively, they have substantial room for improvement despite their leaderboard success. first of all, the overall deltableu score of both solutions are poor, mostly ranging from 10-15 dbleu. the low absolute scores of the solutions really highlight the difficulty of the medical vqa task presented and the difficulty of such tasks in general. upon examining the solutions, we observe that the models were seldom able to generate the exact name of the skin condition in question, although do a good job at identifying a disease similar in presentation or effect location (for example tinea scalp vs seborrheic dermatitis). certainly both solutions require substantial improvements before they contribute meaningful benefits to the healthcare system in practice. while the clip based solution was able to outperform our claude 3 opus api based solution with experiments conducted post competition, it is worth mentioning that such small scale finetuning may be less desirable as the model would have to be repurposed for new problems of interested each time. llm based solutions have the advantage of being general purpose and do not have this issue. furthermore, due to the tight schedules of the competition, both solutions were not explored to their full potential. we anticipate there are bigger up-sides for the claude 3 opus api solution via more sophisticated prompting or compiling. our rather simple implementation of the claude based api solution may not represent the llm's full capability but rather offers a competitive baseline for this task. next, both solutions while reproducible are not stable. the claude api may be subject to randomness during generation due to the temperature parameter or the update of internal private model weights while the clip solutions observed inconsistencies during retrieval where the retrieved images' labels seldom agreed with each other despite relatively similar appearances, leading to low confidence in the final output. retraining the clip model with the same experimental setup but initializing differently may yield completely different final disease label classification due to this inconsistency. lastly, the two solutions were formulated with the competition evaluation metric in mind as they are both framed as a disease label prediction task rather than a more usual vqa task which could cover a broader range of topics in their generated responses such as differential diagnoses, treatments and other recommendations as present in the actual ground truths for this competition. this is further reason to treat the performance of the presented solutions with a grain of salt. specifically, upon our initial exploration, the deltableu metric defined by the competition organizers favors short responses given the relatively heavy penalty incurred on incorrect k-mers present and relatively low penalty on a incomplete answer in comparison. this discourages model exploration during text generation and potentially penalizes model predictions that are correct semantically but are either too long or not containing the exact words present in the ground truths. this is highlighted in the ablation results in nevertheless, the competition serve as an important step towards the goal of automatically generating clinical responses given textual queries and associated images, and we sincerely thank the organizers for the work curating this dataset and organizing the competition. ## conclusion we present two solutions to the english category of the mediqa2024-m3g shared task for multilingual and multimodal medical answer generation. the claude 3 opus api based solution and the clip image classification based solution scored 1st and 2nd, respectively among all submissions. while there is still substantial room for improvement for these two solutions, we share and discuss our findings to contribute towards the important goal of automatically generating clinical responses given textual queries and associated images.