T riage A gent: Towards Better Multi-Agents Collaborations for Large Language Model-Based Clinical Triage (2024)

Abstract: The global escalation in emergency department patient visits poses significant challenges to efficient clinical management, particularly in clinical triage. Traditionally managed by human professionals, clinical triage is susceptible to substantial variability and high workloads. Although large language models (LLMs) demonstrate promising reasoning and understanding capabilities, directly applying them to clinical triage remains challenging due to the complex and dynamic nature of the clinical triage task. To address these issues, we introduce TriageAgent, a novel heterogeneous multi-agent framework designed to enhance collaborative decision-making in clinical triage. TriageAgent leverages LLMs for role-playing, incorporating self-confidence and early-stopping mechanisms in multi-round discussions to improve document reasoning and classification precision for triage tasks. In addition, TriageAgent employs the medical Emergency Severity Index (ESI) handbook through a retrieval-augmented generation (RAG) approach to provide precise clinical knowledge and integrates both coarse- and fine-grained ESI-level predictions in the decision-making process. Extensive experiments demonstrate that TriageAgent outperforms state-of-the-art LLM-based methods on three clinical triage test sets. Furthermore, we have released the first public benchmark dataset for clinical triage with corresponding ESI levels and human expert performance for comparison.

## introduction emergency departments (eds) play a crucial role in the healthcare system by continuously assessing and prioritizing patients based on urgency and severity. this process, known as clinical triage, utilizes the emergency severity index (esi) as report: through discussion... i would classify this patient as an esi level 5-90% confidence score human expert i classify this patient to level 4. a standardized guide for decisions on rapid medical intervention, which is vital for prioritizing treatment and allocating resources. however, the growing number of patients poses significant challenges to the rapid and precise classification of cases, which is crucial for accurate esi categorization. currently, hospitals rely on human experts to review clinical notes and determine case urgency (as illustrated in figure 1 ). although effective, this manual method is time-consuming, labor-intensive, and burdensome for clinical staff. the increasing patient volume and complex triage process often lead to staff fatigue, diminishing accuracy and efficiency, and raising the risk of inconsistent classification or misdiagnosis. consequently, there is a high demand for ai methods to automate esi classification. traditional deep learning (dl) models (as illustrated in figure 1 ) @xcite @xcite have assisted in clinical triage but often fall short due to the complex and dynamic nature of the task, which requires extensive labeled data and real-time adaptation. llms such as @xcite , @xcite , and llama @xcite offer promising solutions with advanced text understanding capabilities, reducing time costs and errors by quickly interpret-ing and categorizing clinical documents. additionally, llms can leverage external tools, such as knowledge base apis @xcite , to enhance domain-specific knowledge, adaptability, speed, and accuracy. however, even with these tools, a single llm may struggle with the complexities of emergency scenarios, such as diverse patient presentations and the need for immediate multi-disciplinary coordination @xcite . therefore, a multi-agent approach is necessary to improve performance by distributing tasks and assigning specialized roles to agents with diverse expertise. recent advancements have significantly enhanced multi-agent systems in areas such as reasoning @xcite , sophisticated planning @xcite , and memory @xcite . these improvements enable multi-agent llms to analyze medical data, formulate treatment plans, and recall patient histories more effectively @xcite . however, directly deploying multi-agent llms in clinical triage remains challenging due to their lack of optimization for triage-specific decisionmaking, resulting in accuracy levels around 60% (as illustrated in we identify four major challenges hindering clinical triage effectiveness. the first is data processing speed. the increasing patient volume necessitates rapid data processing and decision-making, yet traditional methods often face delays due to continuous data updates and extensive preprocessing, hindering timely clinical interventions. the second is diversity in clinical documents. variability in patient histories, symptoms, writing styles and terminologies complicates the classification task. the third is contextual understanding and interpretability. the complexity of clinical contexts makes it challenging for models to accurately interpret information and provide transparent, evidencebased results, leading to a lack of clinician trust. the fourth is boundaries between different esi levels. precise differentiation and classification of esi levels (1-5) are essential to avert critical medical errors. for instance, distinguishing between a level 1 (most urgent) and a level 2 (less urgent) case is vital, as it determines whether a patient receives immediate medical intervention or encounters a prolonged wait. nevertheless, the subtle nuances between different esi levels present considerable challenges to the clinical triage process. to address these challenges, we propose triageagent, a novel heterogeneous multi-agent collaboration framework for clinical triage that leverages llm-based agents enhanced with external tools and embedded medical knowledge. triageagent enables effective information exchange and reliable interactions among agents, making the framework ideal for zero-shot document classification and handling complex tasks without prior demonstrations. the key innovations include retrieval-augmented generation for contextrelevant evidence, a confidence score-based mechanism for precise decision-making, and an early stopping mechanism to improve time efficiency. these features enhance contextual understanding, interpretability, and precision in esi-level classification. additionally, the framework supports realtime decision-making and dynamic, interactive debates among agents, refining information for more accurate triage and improving the timeliness of clinical interventions. experiments on three esi clinical triage test sets demonstrate that triageagent significantly improves zero-shot performance with gpt-3.5-turbo and gpt-4, reducing discordance rates by up to 10.84% and 18.42%, respectively. additionally, we are the first to publicly release a clinical triage dataset that includes clinical notes alongside esi levels and human expert performance, providing a valuable resource for academic research and clinical practice. by setting new benchmarks, we aim to advance the field of clinical triage in both academic and practical applications. we will publicly release our code and dataset once the paper is published. ## methodology this section presents the triageagent framework, which simulates teamwork and problemsolving in diagnosing and planning treatment for ed cases. we explore the heterogeneous structure formulation of the framework, as introduced in appendix a. we describe the five stages of triageagent's operation stages, illustrated in figure 2 and detailed in the following procedures: first, allocating documents, where patient clinical records are assigned to expert agents to initiate esi discussions. second, group-based classification analysis, where agents are divided into two groups to conduct coarse and fine-grained classification. third, confidence report summarization, where a summarized report is generated based on previous analyses, including classification results and confidence scores. fourth, collaborative discussion, where agents engage in discussions over the sum-marized report, iteratively refining it according to key information, rationales, and confidence scores. fifth, consensus agreement, resulting in an ultimate, precise, and thoroughly validated revised report, highlighting the importance of collaborative decision-making. ## experimental setup dataset we construct a clinical triage dataset by collecting patient cases from the publicly available emergency severity index (esi) handbook v4 (esi, accessed: 2024-04-06). this dataset is designed to evaluate machine learning models and methods for categorizing esi levels in medical documents. to our knowledge, this is the first publicly released clinical triage dataset that includes clinical notes and corresponding esi levels necessary for triage tasks, serving as a benchmark for evaluating our framework's effectiveness. since the patient cases are sourced from the official esi handbook, no deidentification is needed. the dataset is divided into a training set and three test sets (test-1, test-2, and test-3). the training set contains 218 cases with the following distribution across esi levels: esi-1 (14), esi-2 (92), esi-3 (65), esi-4 (22), and esi-5 (25). each test set contains 72 cases, maintaining fixed proportions of esi levels: esi-1 (12), esi-2 (20), esi-3 (13), esi-4 (12), and esi-5 (15). the dataset statistics are provided in