M ed C o T : Medical Chain of Thought via Hierarchical Expert (2024)

Abstract: Artificial intelligence has advanced in Medical Visual Question Answering (Med-VQA), but prevalent research tends to focus on the accuracy of the answers, often overlooking the reasoning paths and interpretability, which are crucial in clinical settings. Besides, current Med-VQA algorithms, typically reliant on singular models, lack the robustness needed for real-world medical diagnostics which usually require collaborative expert evaluation. To address these shortcomings, this paper presents MedCoT, a novel hierarchical expert verification reasoning chain method designed to enhance interpretability and accuracy in biomedical imaging inquiries. MedCoT is predicated on two principles: The necessity for explicit reasoning paths in Med-VQA and the requirement for multi-expert review to formulate accurate conclusions. The methodology involves an Initial Specialist proposing diagnostic rationales, followed by a Follow-up Specialist who validates these rationales, and finally, a consensus is reached through a vote among a sparse Mixture of Experts within the locally deployed Diagnostic Specialist, which then provides the definitive diagnosis. Experimental evaluations on four standard Med-VQA datasets demonstrate that MedCoT surpasses existing state-of-the-art approaches, providing significant improvements in performance and interpretability.

## introduction medical visual question answering (med-vqa) has recently gained significant attention @xcite @xcite . as a new exploration in the medical domain, med-vqa aims to answer medical questions in natural language based on input medical images. an effective med-vqa system figure 1 : the upper figure shows a comparison of the outputs from the previous med-vqa method and med-cot, as well as the previous techniques in mmcot @xcite versus sparse moe in medcot. the lower figure demonstrates that medcot, with a model size of 256m parameters, outperforms the 7b parameter llava-med by 5.52% and 4.09% (accuracy) on the vqa-rad and slake-en datasets. in the field of med-vqa, numerous attempts have been made using deep learning technologies @xcite @xcite @xcite . for instance, @xcite utilized bilinear attention networks (ban) @xcite and enhanced them for med-vqa by incorporating a mixed enhanced visual feature (mevf) setup consisting of pre-trained meta-learning modules and convolutional denoising autoencoders (cdae). building on this, @xcite designed a conditional reasoning framework to boost the inference capabilities of med-vqa models. however, these approaches often underperform in many practical scenarios, primarily due to poor capabilities in extracting and integrating features from a limited number of medical images and text data @xcite @xcite . @xcite introduced the clip architecture into the framework by deploying it as the visual encoder within mevf @xcite , pre-trained on the multimodal medical dataset roco @xcite . their experiments demonstrated significant improvements with the clip. @xcite developed vqa-adapter, which uses a lightweight adapter and label smoothing to efficiently fine-tune the clip model for med-vqa, thus reducing computational costs and mitigating overfitting. @xcite proposed llava-med, which utilizes gpt-4 and a novel curriculum learning approach to efficiently train llava on biomedical images, significantly enhancing med-vqa capabilities. however, previous med-vqa approaches typically focused on the accuracy of the answers @xcite @xcite , where most medvqa responses consist of a simplistic answer lacking detailed explanations or rationale, unlike in real-world scenarios where doctors not only provide answers but also explain their reasoning, professional considerations, and potential contradictions to derive a more comprehensive diagnostic insight. besides, real-world diagnostics often rely on the combined experience of multiple doctors, as a single doctor's diagnosis may be biased by personal experience and may not be sufficiently accurate. in the multimodal chain of thought (cot), answering vqa questions involves providing an answer as well as a corresponding reasoning path (rationale). the generation of this rationale helps to improve the accuracy of the language model. inspired by real-world practices and multimodal @xcite ; @xcite , integrating this paradigm into med-vqa can enhance both the accuracy and interpretability of responses. however, implementing it faces several challenges: (1) previous cot methods required manual annotation of fundamental rationales, which is time-consuming, costly, and challenging to ensure consistency and completeness @xcite . (2) reliance on a single expert model can lead to misleading conclusions. (3) multimodal cot has limited depth in understanding the intents of images and texts, which can restrict its effectiveness in medical contexts @xcite . to address the aforementioned issues, we introduce medcot, a hierarchical expert-verified model for med-vqa. firstly, the initial specialist proposes preliminary diagnostic rationale based on the medical visual and text query. the follow-up specialist then reviews these rationales, categorizing them as valid or invalid; valid rationales are retained, while invalid ones are reassessed. finally, the locally implemented diagnostic specialist, consisting of a sparse mixture of experts (moe) model functioning as a multimodal language model, casts votes to deliver the definitive diagnosis. leveraging a hierarchy of expertise, medcot consistently outperforms state-of-the-art (sota) med-vqa methods across four extensive datasets, demonstrating impressive generalizability and interpretability, as shown in figure 1 . our study makes three significant contributions: medcot not only provides more accurate answers but also offers refined rationales. the provided chest x-ray image shows bilateral interstitial infiltrates, which could indicate the presence of a localized mass ## conclusion in this paper, we propose an effective hierarchical expert reasoning chain method for med-vqa, named medcot. this method is based on two insights: 1) med-vqa should have a clear reasoning path; 2) med-vqa scenarios should be reviewed by multiple experts to arrive at a conclusion. specifically, the process involves initial experts providing preliminary diagnostic rationales based on medical visual questions. follow-up experts then review these rationales for validity, retaining the effective ones and reassessing the ineffective ones. finally, a locally deployed diagnostic specialist, consisting of a sparse moe that conducts a vote, then provides the definitive diagnosis. experimental results on multiple med-vqa datasets show that medcot outperforms existing sota techniques, significantly surpasses recent methods, and demonstrates excellent interpretability for final diagnosis.