EHRA gent: Code Empowers Large Language Models for Few-shot Complex Tabular Reasoning on Electronic Health Records (2024)

Abstract: Clinicians often rely on data engineers to retrieve complex patient information from electronic health record (EHR) systems, a process that is both inefficient and time-consuming. We propose EHRAgent, a large language model (LLM) agent empowered with accumulative domain knowledge and robust coding capability. EHRAgent enables autonomous code generation and execution to facilitate clinicians in directly interacting with EHRs using natural language. Specifically, we formulate a multi-tabular reasoning task based on EHRs as a tool-use planning process, efficiently decomposing a complex task into a sequence of manageable actions with external toolsets. We first inject relevant medical information to enable EHRAgent to effectively reason about the given query, identifying and extracting the required records from the appropriate tables. By integrating interactive coding and execution feedback, EHRAgent then effectively learns from error messages and iteratively improves its originally generated code. Experiments on three real-world EHR datasets show that EHRAgent outperforms the strongest baseline by up to 29.6% in success rate, verifying its strong capacity to tackle complex clinical tasks with minimal demonstrations.

## introduction an electronic health record (ehr) is a digital version of a patient's medical history maintained by healthcare providers over time @xcite . in clinical research and practice, clinicians actively interact with ehr systems to access and retrieve patient data, ranging from detailed individuallevel records to comprehensive population-level insights @xcite . the reliance on pre-defined rule-based conversion systems in most ehrs often necessitates additional training or assistance from data engineers for clinicians to obtain information beyond these rules @xcite , leading to inefficiencies and delays that may impact the quality and timeliness of patient care. alternatively, an autonomous agent could facilitate clinicians to communicate with ehrs in natural languages, translating clinical questions into machine-interpretable queries, planning a sequence of actions, and ultimately delivering the final responses. compared to existing ehr management that relies heavily on human effort, the adoption of autonomous agents holds great potential to efficiently simplify workflows and reduce workloads for clinicians (figure 1 ). although several supervised learning approaches @xcite have been explored to automate the translation of clinical questions into corresponding machine queries, such systems require extensive training samples with fine-grained annotations, which are both expensive and challenging to obtain. large language models (llms) @xcite bring us one step closer to autonomous agents with extensive knowledge and substantial instruction-following abilities from di- to address these limitations, we propose ehragent, an autonomous llm agent with external tools and code interface for improved multi-tabular reasoning across ehrs. we translate the ehr question-answering problem into a tool-use planning process -generating, executing, debugging, and optimizing a sequence of code-based actions. firstly, to overcome the lack of domain knowledge in llms, we instruct ehragent to integrate query-specific medical information for effectively reasoning from the given query and locating the query-related tables or records. moreover, we incorporate long-term memory to continuously maintain a set of successful cases and dynamically select the most relevant few-shot examples, in order to effectively learn from and improve upon past experiences. secondly, we establish an interactive coding mechanism, which involves a multiturn dialogue between the code planner and executor, iteratively refining the generated code-based plan for complex multi-hop reasoning. specifically, ehragent optimizes the execution plan by incorporating environment feedback and delving into error messages to enhance debugging proficiency. we conduct extensive experiments on three largescale real-world ehr datasets to validate the empirical effectiveness of ehragent, with a particular focus on challenging tasks that reflect diverse information needs and align with real-world application scenarios. in contrast to traditional supervised settings @xcite that require over 10k training samples with manually crafted annotations, ehragent demonstrates its efficiency by necessitating only four demonstrations. our findings suggest that ehragent improves multi-tabular reasoning on ehrs through autonomous code generation and execution, leveraging accumulative domain knowledge and interactive environmental feedback. our main contributions are as follows: question: what is the maximum total hospital cost that involves a diagnosis named comp-oth vasc dev/graft since 1 year ago? clinician existing clinical workflow clinician ehr engineer select max(t1.c1) from ( select sum(cost.cost) as c1 from cost where cost.hadm_id in … ehragent (ours) ehragent assume you have knowledge of following medical records: [ehr_metadata]. write a python code to solve the given question. you can use the following functions: [api_name, api_description]. here are some examples: [k-shot demonstrations]. the related knowledge to the question is given: [medical information]. question: [question]. solution: ehr metadata (1) charted events are stored in a series of 'events' tables… (2) tables prefixed with 'd_' are dictionary… (3) four databases are used to define and track patient stays… tool set (api) med. info. ## ehragent: llms as medical agents in this section, we present ehragent (figure 3 ), an llm agent that enables multi-turn interactive coding to address multi-hop reasoning tasks on ehrs. ehragent comprises four key components: ## conclusion in this study, we develop ehragent, an llm agent with external tools for few-shot multi-tabular reasoning on real-world ehrs. empowered by the emergent few-shot learning capabilities of llms, ehragent leverages autonomous code generation and execution for direct communication between clinicians and ehr systems. we also improve ehragent by interactive coding with execution feedback, along with accumulative medical knowledge, thereby effectively facilitating plan optimization for multi-step problem-solving. our experiments demonstrate the advantages of ehragent over baseline llm agents in autonomous coding and improved medical reasoning.