Worse than Random? An Embarrassingly Simple Probing Evaluation of Large Multimodal Models in Medical VQA (2025)

Abstract: Large Multimodal Models (LMMs) have demonstrated impressive performance on existing medical Visual Question Answering (Med-VQA) benchmarks. However, high reported accuracy does not necessarily reflect their true diagnostic reliability in clinical settings. This study reveals that state-of-the-art models perform worse than random guessing on medical diagnosis questions when subjected to simple Probing Evaluation for Medical Diagnosis (ProbMed). ProbMed challenges models through probing evaluation and procedural diagnosis. Particularly, probing evaluation features pairing ground-truth questions with adversarial counterparts that feature negated and hallucinated attributes, while procedural diagnosis requires reasoning across various dimensions for each image, including modality recognition, organ identification, clinical findings, abnormalities, and positional grounding. Our evaluation reveals that even top-performing models like GPT-4o, GPT-4V, and Gemini Pro perform worse than random guessing on specialized diagnostic questions, indicating significant limitations in handling fine-grained medical inquiries. Furthermore, our ablation study on open-source models (e.g., LLaVA, LLaVA-Med, and Med-Flamingo) identifies poor visual understanding as a primary bottleneck—a limitation that can be partially mitigated by incorporating visual descriptions generated by GPT-4o, resulting in an average performance improvement of 9.44%. These findings underscore the urgent need for more robust evaluation methods and domain-specific expertise to ensure the reliability of LMMs in high-stakes medical applications.

## introduction foundation models, such as large language models (llms) @xcite @xcite @xcite and large multimodal models (lmms) @xcite @xcite @xcite , have demonstrated impressive capabilities in understanding complex visual and text inputs, generating humanlike language, and achieving high accuracy on various benchmarks. the integration of these foundation models into real-life medical practice holds immense potential given their advanced computational capabilities @xcite and promising progress on existing medical caption: chest x-ray showing bilateral pleural thickening in the upper and middle lung fields. existing evaluation evaluation with adversarial pair gt: no is bilateral pleural thickening in the upper and middle lung fields in this chest x-ray? is bilateral pleural thickening in the upper and middle lung fields in this chest x-ray? yes. gt: yes gt: yes yes. is bilateral pleural thickening in the lower lung fields in this chest x-ray? yes. figure 2 : an example illustrating the potential for misleading accuracy in existing evaluations. while the model correctly identifies the position of an existing finding in the standard evaluation, it fails to differentiate between actual and hallucinated positions when subjected to an adversarial evaluation. visual question answering (med-vqa) benchmarks @xcite @xcite . as we stand on the precipice of integrating these models into critical decision-making domains, one natural question appears: how much can we trust these models in realworld scenarios, such as medicine and healthcare, where the stakes are high? before discussing the reliability of lmms in critical domains like med-vqa, we must first address a fundamental question: are we evaluating lmms correctly? to address this question, we introduce a simple yet effective probing evaluation method that exposes the weaknesses of lmms by creating binary questions with hallucination pairs over existing benchmarks. an example is shown in figure 2 . despite the high accuracy reported on current med-vqa tasks, our study reveals a significant vulnerability in lmms when faced with adversarial questioning, as illustrated in figure 1 . the observed performance drops are alarming: even advanced models like gpt-4o, gpt-4v, and gemini pro perform worse than random guessing, with an average decrease of 27.78% across the tested models. based on this, we further analyze a critical question: how reliable are lmms in medical diagnosis, ranging from general questions to specialized diagnostic questions? to address this question, we introduce probmed, which features procedural diagnosis designed to rigorously evaluate model performance across multiple diagnostic dimensions. we curated probmed from 6,303 images sourced from two widely-used biomedical datasets, @xcite and chestx-ray14 @xcite . these images cover various modalities, including x-ray, mri, and ct scans, and span multiple organs such as the abdomen, brain, chest, and spine. using gpt-4 and a positional reasoning module, we generated metadata for each image, extracting information about abnormalities, condition names, and their corresponding locations. this metadata facilitated the automatic generation of 57,132 high-quality question-answer pairs, covering dimensions like modality recognition, organ identification, abnormalities, clinical findings, and positional reasoning. our systematic evaluation of twelve state-of-theart lmms on probmed revealed several critical insights. first, even the best-performing models, such as gpt-4o and gemini pro, performed close to random guessing on specialized diagnostic categories like condition/finding and position, highlighting their limitations in handling fine-grained medical inquiries. second, introducing adversarial pairs significantly reduced the accuracy of all models, with llava-med-v1.5's performance dropping by up to 28.22% and gpt-4o's accuracy decreasing by 20.71% in probmed. these findings emphasize the importance of adversarial testing in med-vqa to uncover model weaknesses. third, by incorporating chain-of-thought reasoning and adding visual descriptions generated by gpt-4o, we observe substantial improvements in model performance, suggesting that poor visual understanding is a critical bottleneck. the results indicate that augmenting these models with more accurate visual information could significantly improve their ability to handle complex medical tasks. moreover, the chexagent model, which was exclusively trained on chest x-rays, demonstrated that specialized domain knowledge is crucial. it showed that expertise gained on one particular organ could be transferable to another modality of the same organ in a zero-shot manner, highlighting the value of domain-specific training for improving model performance. in summary, our work highlights significant gaps in the reliability of lmms for medical diagnosis despite their impressive performance on existing general domain benchmarks. the insights from probmed underscore the urgent need for robust evaluation methodologies to ensure the accuracy and reliability of lmms in real-world medical applications. our findings also suggest that poor visual understanding is a key limitation for opensource models, which can be mitigated by incorporating chain-of-thought reasoning and accurate visual descriptions. this research inspires the devel-opment of more trustworthy ai systems in healthcare and beyond, ultimately contributing to better diagnostic outcomes and patient care. further progress has been made in finetuning general-domain lmms for the biomedical field, resulting in specialized models like biomedgpt @xcite , llava-med @xcite , med-flamingo @xcite , medblip @xcite , radfm @xcite and medvint @xcite . despite the promising results from these domainspecific lmms, ongoing exploration exists into training smaller multimodal models to address specific clinical needs. for instance, models like llava-rad @xcite and chexagent @xcite have been developed for chest x-ray interpretation, aiming to bridge competency gaps in radiology tasks. comprehensive surveys of llms for healthcare highlight the progress, applications, and challenges in deploying llms in clinical settings @xcite @xcite . task-specific evaluations @xcite underline the potential and challenges of lmms in the medical domain. as we move towards integrating these models into critical decision-making processes, it becomes imperative to assess their reliability in high-stakes environments like healthcare and medicine. ## probing evaluation for medical diagnosis in this section, we introduce two complementary evaluation strategies that rigorously assess stateof-the-art lmms for med-vqa. our approach is designed to answer the following research questions: is the current evaluation of lmms for med-vqa reliable? 2. how reliable are lmms on medical diagnosis, ranging from general questions to specialized diagnostic questions? despite current high accuracy, we find that the models struggle with simple probing evaluation on existing benchmarks. our evaluation framework, probmed (probing evaluation for medical data selection metadata creation question answer generation qa generation gpt4 preprocess atelectasis infiltration pneumonia pneumonia [601, 595, 215, 180] original image: [2450, 2839] image label bounding box positional reasoning chestx-ray14 image caption { "id": 4, "image_type": "ct scan_brain", "image_path": "*.png", "caption": "figure 2 . ct scan of the brain showing an expansive lesion envolving the right temporal lobe.", "abnormality": "1", "condition": { "expansive lesion": "right temporal lobe" } }, ... { "id": 69867, "image_type": "x-ray_chest", "image_path": "*.png", "abnormality": "1", "condition": { "atelectasis": "n/a", "infiltration": "n/a", "pneumonia": "lower left" } }, medicat is this a ct scan? ans: yes is this an mri? ans: no modality does this show a brain? ans: yes does this show an abdomen? ans: no organ does this ct scan show any abnormality of the brain? abnormality is an expansive lesion present in the given brain ct scan? ans: yes is leptomeningeal and gyral calcification present in the given brain ct scan? ans: no condition / finding in the given brain ct scan, is an expansive lesion present in the right temporal lobe? ans: yes in the given brain ct scan, is an expansive lesion present in the right thalamus? ans: no position figure 3: flow diagram of the probmed data curation process. two comprehensive biomedical datasets were utilized to collect source data and construct a metadata file, enabling the automatic generation of high-quality question-answer pairs for the probmed dataset. diagnosis), is built on adversarial testing and multifaceted diagnostic reasoning to further expose these vulnerabilities and provide a thorough analysis of model performance. we also explore enhancements through chain-of-thought reasoning and the incorporation of external visual descriptions (e.g., from gpt-4o) to address the noted limitations of open-sourced models. ## conclusion evaluating the reliability of lmms in the medical domain requires robust methods, and probmed, our newly introduced framework, addresses this by incorporating probing evaluation and procedural diagnosis. our study reveals significant limitations in models like gpt-4o and gemini pro, which perform worse than random guessing on specialized diagnostic questions, while chexagent's results highlight the critical importance of domainspecific knowledge. furthermore, our additional experiments, which introduced chain-of-thought reasoning with external visual descriptions generated by gpt-4o, suggested that poor visual understanding is a major limitation of existing models and augmenting them with external visual reasoning can lead to notable gains.