No Black Boxes: Interpretable and Interactable Predictive Healthcare with Knowledge-Enhanced Agentic Causal Discovery (2025)

Abstract: Deep learning models trained on extensive Electronic Health Records (EHR) data have achieved high accuracy in diagnosis prediction, offering the potential to assist clinicians in decision-making and treatment planning. However, these models lack two crucial features that clinicians highly value: interpretability and interactivity. The “black-box” nature of these models makes it difficult for clinicians to understand the reasoning behind predictions, limiting their ability to make informed decisions. Additionally, the absence of interactive mechanisms prevents clinicians from incorporating their own knowledge and experience into the decision-making process. To address these limitations, we propose II-KEA, a knowledge-enhanced agent-driven causal discovery framework that integrates personalized knowledge databases and agentic LLMs. II-KEA enhances interpretability through explicit reasoning and causal analysis, while also improving interactivity by allowing clinicians to inject their knowledge and experience through customized knowledge bases and prompts. II-KEA is evaluated on both MIMIC-III and MIMIC-IV, demonstrating superior performance along with enhanced interpretability and interactivity, as evidenced by its strong results from extensive case studies.

## introduction accurate diagnosis prediction is crucial for improving clinical outcomes by enabling timely interventions and optimizing treatment planning. in recent years, the growing availability of electronic health records (ehr) (e.g., mimic datasets @xcite ) has provided valuable realworld data, allowing researchers to develop more advanced and complex deep learning models @xcite @xcite patient with medical history ## methodology we propose, ii-kea, a multi-agent system consisting of three llm-based agents working collaboratively and is powered by both clinical datasets and domain knowledge. ii-kea aims to uncover causal relationships between diseases and predict future medical diagnoses. in this section, we introduce each llm agent and knowledge module and provide a summary of the overall framework. ## related work we categorize prior work into clinical prediction (section 4.1), agentic approaches (section 4.2), and causal inference (section 4.3). ## privacy and ethical statement our work involves the analysis of ehr data, which contains sensitive personal medical information. in compliance with the physionet credentialed health data use agreement 1.5.0 foot_2 , we conducted all interactions between the language models and the ehr data through azure openai service foot_3 , which adheres to enterprise-grade security and compliance standards. we also submitted the opt-out form foot_4 to decline human review in terms of the responsible use guidelines specified for mimic datasets available at responsible use of mimic data with online services foot_5 , which outlines proper handling of ehr data when used with generative models. this ensured that the capabilities of large language models were applied without compromising the privacy and confidentiality of patient information. furthermore, we continuously and carefully monitor our compliance with these guidelines and relevant privacy regulations to uphold the ethical use of data in our research and operations. ## conclusion in this paper, we introduce ii-kea, a knowledgeenhanced agentic causal discovery framework designed for interpretable and interactive diagnosis prediction. ii-kea consists of three llm-based agents working collaboratively and is powered by both clinical datasets and domain knowledge. we evaluate ii-kea on the mimic-iii and mimic-iv datasets and conduct both ablation and case studies to demonstrate its effectiveness. the ethical consideration can be checked in appendix 6.