UM ass- B io NLP at MEDIQA - M 3 G 2024: D erm P rompt - A Systematic Exploration of Prompt Engineering with GPT -4 V for Dermatological Diagnosis (2024)

Abstract: This paper presents our team’s participation in the MEDIQA-ClinicalNLP 2024 shared task B. We present a novel approach to diagnosing clinical dermatology cases by integrating large multimodal models, specifically leveraging the capabilities of GPT-4V under a retriever and a re-ranker framework. Our investigation reveals that GPT-4V, when used as a retrieval agent, can accurately retrieve the correct skin condition 85% of the time using dermatological images and brief patient histories. Additionally, we empirically show that Naive Chain-of-Thought (CoT) works well for retrieval while Medical Guidelines Grounded CoT is required for accurate dermatological diagnosis. Further, we introduce a Multi-Agent Conversation (MAC) framework and show it’s superior performance and potential over the best CoT strategy. The experiments suggest that using naive CoT for retrieval and multi-agent conversation for critique-based diagnosis, GPT-4V can lead to an early and accurate diagnosis of dermatological conditions. The implications of this work extend to improving diagnostic workflows, supporting dermatological education, and enhancing patient care by providing a scalable, accessible, and accurate diagnostic tool.

## introduction diagnosing skin conditions demands a complex blend of visual inspection, patient history examination, and deep clinical acumen, a skill set that dermatologists spend extensive years acquiring @xcite . despite the critical nature of these skills, many regions worldwide face a stark scarcity of dermatological expertise @xcite . even in areas with adequate services, the demand for such specialized knowledge frequently surpasses its availability. the recent global health crisis has also expedited the shift towards remote clinical diagnostics and treatments, further highlighting the challenges in diagnosing skin diseases @xcite . these challenges include the scarcity of dermatological expertise and the need to accommodate asynchronous patient interactions, including e-visits, emails, and messaging platforms, to ensure continuity and quality of care. in response to these challenges, recent advancements in artificial intelligence (ai), particularly through the development of large language models (llms), offer promising solutions to significantly support dermatologists by enhancing clinical diagnosis and treatment processes @xcite @xcite . moreover, ai facilitates asynchronous patient services, offering a cost-effective and convenient alternative to traditional methods. previous works have primarily utilized deep learning for tasks such as skin lesion classification (udris , @xcite @xcite , and dermatopathology @xcite focusing predominantly on dermoscopic images @xcite . these efforts, however, have relied on image-only models, indicating a need for broader applications. our research aims to extend the capabilities of ai in dermatology by diagnosing skin diseases and devising appropriate treatment plans based on patients' dermatological images, queries, and medical histories. this approach mirrors the diagnostic process of dermatologists, who rely on highquality images and comprehensive patient histories to make informed decisions. although previous studies have explored fine-tuning models on multimodal data (e.g., skingpt @xcite , and medblip @xcite , our task is particularly challenging due to data availability and image quality limitations, reflecting real-world constraints where high-quality data is either scarce or expensive to obtain. studies by openai @xcite and microsoft @xcite have demonstrated that generalist foundation models, such as gpt-4, can surpass specifically fine-tuned medical models on various medical benchmarks by employing specialized prompting strategies. building on these insights, our research leverages both the textual and visual capabilities of gpt-4, targeting the specific task of dermatology diagnosis and treatment. the diagnostic process for skin lesions or conditions requires meticulous evaluation and is informed by methodologies such as dermatoscopy @xcite , which enables dermatologists to observe skin abnormalities in greater detail. dermatologists usually follow a common guideline for assessing skin lesions, emphasizing the importance of visual descriptors like shape, size, color, texture, and pattern in differential diagnosis. inspired by these practices, we have integrated advanced chain-of-thought (cot) techniques with visual features to create medical guidelines tailored for gpt-4v, enhancing its diagnostic precision. this enables the model to emulate dermatologists' diagnostic process. furthermore, our research integrated a multi-agent conversation (mac) framework @xcite @xcite , which involves multiple ai agents that generate additional context and critiques for various candidate skin conditions. these agents collaborate, debate, and consolidate their findings to determine the most accurate skin disease diagnosis from the candidates identified from the retrieval step. this, therefore, introduces a level of dynamic interaction and comprehensive analysis that mirrors the complex decision-making process in clinical dermatology. hence, our contributions are twofold: ## related work the interdisciplinary fusion of artificial intelligence (ai) and dermatology has spawned a myriad of approaches to enhance the diagnosis of skin conditions. historically, these approaches have often treated diagnosis as a classification task, with literature extensively documenting the use of convolutional neural networks (cnns) and other deep learning architectures like resnets for lesion classification from dermoscopic images, which are typically limited to dermatological clinics due to image acquisition constraints @xcite , @xcite . recent advancements have moved beyond traditional clinic-bound methods, exploring the utility of clinical images for broader classifications, such as skin cancer and onychomycosis @xcite . while these efforts have made significant contributions to disease diagnosis, they have not fully addressed the generative and comprehensive nature of clinical diagnosis, which encompasses treatment planning and patient interaction beyond mere classification. the evolution of large language models (llms) has significantly widened the scope of ai applications in healthcare. llms like pubmedbert @xcite and biogpt @xcite have been fine-tuned on extensive corpora of medical literature, achieving state-of-the-art performance in tasks ranging from biomedical reasoning to question-answering. in the realm of domainadapted llms, models like meditron and med-palm have demonstrated remarkable capabilities in language understanding and generation, setting new benchmarks across biomedical datasets @xcite . with the advent of multimodal models, integrating visual and textual data has further refined ai applications in medical domains. vision-language models such as med-clip, med-blip, and llava-med have exhibited promising results in image-text retrieval, zero-shot classification, and even multimodal conversations, respectively @xcite . specifically, in the context of dermatology, the skin-gpt4 model (zhou 2 503 et al., 2023) represents a pioneering effort in creating a multimodal setup tailored for skin disease identification and patient interaction. studies have shown that generalist foundation models like gpt-4, with their expansive knowledge bases and specialized prompting techniques, outperform domain-specific models such as med-palm on various medical benchmarks @xcite . gpt-4's application in dermatology, particularly in melanoma identification and medical exam question answering, underscores its potential as an assistive tool for educational and diagnostic purposes @xcite @xcite . our research builds upon these foundations, employing gpt-4's multimodal capabilities (gpt-4v) to enhance dermatological diagnostic processes. by integrating chain-of-thought (cot) techniques and a multi-agent conversation (mac) framework @xcite @xcite , we aim to emulate the complex decisionmaking process of dermatologists, enriching the gpt-4v's ability to generate diagnostic and treatment plans from multimodal data. this work not only taps into the multimodal analytical strength of gpt-4v but also seeks to optimize the model's performance in a domain where the nuances of patient history and visual inspection are paramount. thus, our contribution to the field involves the innovative use of gpt-4v within a retrieval and re-ranking framework, leveraging both naïve and medically informed cot prompting strategies. ## methodology our methodology delineates the comprehensive approach we adopted to address the task of mul-timodal medical answer generation. this process involves two primary stages: retrieving potential diagnoses and the ranking of these to identify the most probable skin condition and treatment plan. task description: the objective of our research is to develop a system capable of diagnosing a possible skin condition and recommending a corresponding treatment plan based on a patient's medical query and associated image. to accomplish this, we propose a two-step pipeline consisting of a retrieval module followed by a ranker module. specifically: our overall methodology is mentioned in figure 1 . ## discussion and conclusion in our study, we systematically explored the merits of various prompting strategies within an information retrieval-based dermatology diagnostic framework. by evaluating these strategies through the lens of accuracy metrics, we found that a naive chain of thought (cot) strategy effectively simulates a retrieval module typical of information retrieval systems. this approach is adept at returning a sufficient number of candidate diagnoses, setting a foundational stage for further analysis. our findings underscore the importance of including patient 8 history and contextual information in clinical dermatology to enhance diagnostic accuracy. for the nuanced task of re-ranking diagnostic candidates, our research indicates that a more refined cot strategy is necessary. specifically, prompts that incorporate expert guidelines prove critical in conducting differential diagnoses, yielding superior performance in top-1 and top-2 diagnostic outcomes. furthermore, we introduce the novel multi-agent critique (mac) framework, which incorporates agent-based critique and feedback, and has the potential to perform differential diagnosis and refine it's output using feedback.