Advancing Healthcare Automation: Multi-Agent System for Medical Necessity Justification (2024)

Abstract: Prior Authorization delivers safe, appropriate, and cost-effective care that is medically justified with evidence-based guidelines. However, the process often requires labor-intensive manual comparisons between patient medical records and clinical guidelines, that is both repetitive and time-consuming. Recent developments in Large Language Models (LLMs) have shown potential in addressing complex medical NLP tasks with minimal supervision. This paper explores the application of Multi-Agent System (MAS) that utilize specialized LLM agents to automate Prior Authorization task by breaking them down into simpler and manageable sub-tasks. Our study systematically investigates the effects of various prompting strategies on these agents and benchmarks the performance of different LLMs. We demonstrate that GPT-4 achieves an accuracy of 86.2% in predicting checklist item-level judgments with evidence, and 95.6% in determining overall checklist judgment. Additionally, we explore how these agents can contribute to explainability of steps taken in the process, thereby enhancing trust and transparency in the system.

## introduction in us healthcare, management of administrative workflows represents a pivotal yet formidable challenge. physicians, nurses, and administrative personnel frequently allocate a substantial portion of their working hours to these procedural tasks, distracting from their primary focus on patient care. one such workflow, prior authorization (pa) is a healthcare management process used by insurance entities to determine whether a proposed treatment or service is covered under a patient's plan before it is approved to be carried out. this process applies to various treatments and services, including medications, imaging, and procedures @xcite . evaluating a pa application involves assessing medical necessity of patient-specific health records against prevailing coverage guidelines. a major part of these coverage guidelines are clinical guidelines which are systematically developed statements designed to help practitioners make decisions about appropriate health care for specific clinical circumstances. insurance companies review these clinical guidelines to to justify medical necessity of a procedure or treatment @xcite . while prior authorization ensures safe, appropriate, cost-effective and evidence based care to all members @xcite , it is a major source of physician and staff burnout as well as job dissatisfaction.there are several ongoing efforts to improve the prior authorization process. high-profile innovations include (1) "gold carding" providers, exempting those who have very high historical approval rates; and (2) automating the process through e-prior auth (e-pa) @xcite . e-pa proposes a set of transactions conveying the rules for approval in a standardized query representation in cql. while such rule based methods are adequate for simple authorization decisions, complex cases with temporal data, evidence of responses and trends in clinical data items can be difficult to represent in cql's rule based format @xcite . also, a nationwide survey @xcite identified that the use of e-pa was not associated with less provider time or fewer challenges in preparing and submitting pa requests. however, the use of e-pa reported a shorter pa decision time. additionally, there is an understanding that ai can potentially improve the current state of pa filing @xcite . the introduction of large language models (llms) @xcite has catalyzed a transformative shift in the capabilities of artificial intelligence, enabling the resolution of complex challenges previously inaccessible to conventional ai methods. llms excel in interpret-ing and synthesizing large volumes of unstructured data, enhancing tasks such as natural language understanding @xcite , sentiment analysis, and automated content creation @xcite . building on this foundation, multi-agent systems, which employs a collective of ai-powered agents, represents an even further advancement @xcite . this approach decomposes a singular complex task into multiple, manageable sub-tasks and distributes them across multiple agents, each specialized through training for a sub-task. following this methodology essentially infuses a microservice architecture into the traditional monolithic ai framework, enabling more modular, scalable, and robust ai systems. by integrating the depth and adaptability of llms with the collaborative and dynamic nature of multi-agent systems, ai systems can achieve unprecedented levels of performance and versatility across various complex problems @xcite . in this paper, we investigate the application of multi-agent systems for determining medical necessity for a medical procedure. our contributions are as follows: ## related work large language models (llms) have completely changed the landscape of natural language processing (nlp) in the recent years. llms have shown emergent abilities @xcite in settings like few-shot prompting @xcite and augmented prompting strategies. augmented prompting like chain of thought (cot) @xcite and automatic chain of thought @xcite prompting enables llms to solve reasoning tasks using step by step approach. additionally, instruction fine-tuning with human feed-back has made llms able to respond to instructions describing unseen tasks @xcite . other advancements include techniques like selfconsistency @xcite which helps llms solve complex tasks using multiple different ways of thinking and prompt gradient descent @xcite which edits prompt in the opposite semantic direction of the gradient to boost prompt's performance. building on this, more dynamic and complex tasks can be tackled by llm powered multi agent systems (llm-mas). these llm-mas have collaborative autonomous agents equipped with unique strategies and behaviour @xcite . this agentic behaviour is based on the idea that llms can improve in game-play scenario by using previous experiences and feedback @xcite . llms have the potential to disrupt medicine. models like med-palm @xcite outperformed state of the art on all multimedbench tasks @xcite . gpt-4 has consistently outperformed task-specific fine-tuned models and is comparable to human experts on qa datasets @xcite . gpt-4 scored 86.65% in united states medical licensing examination (usmle) where passing percentage was 60% @xcite . it also demonstrates gpt-4's capacity for reasoning about concepts tested in usmle challenge problems, including explanation, counterfactual reasoning, differential diagnosis, and testing strategies. some recent researches have started to explore the impact of llms in discharge summary generation @xcite , care planning @xcite , electronic health records (ehrs) @xcite . textto-sql parsing has attracted significant interest @xcite . building on this idea, numerous research efforts, such as ehrsql @xcite , are focused on extracting data from ehrs. additionally, there are ongoing efforts to develop solutions for ehr-based question-answering tasks @xcite . however, the domain of pa filing is largely untouched by llms mainly because of lack of publicly available data despite the understanding that ai can potentially improve its current state @xcite . while some efforts have been made to automate pa filing, for example @xcite where chatgpt is utilised to generate pa letters for orthopedic surgery practice, but the process lacks the important step of establishing medical necessity using ai. another study aims to determine pa approval for lumbar stenosis surgery with machine learning (de barros et al., 2023) but it uses surgery specific symptoms as input variables which would be difficult to generalize. ## problem statement as mentioned above, the evaluation of medical necessity is conducted through a meticulous comparison between patient medical records and established clinical guidelines. these medical records are systematically structured in a json-like format, usually in fhir foot_0 , within electronic health records (ehrs) systems. each object (resource) can be of type patient (patient demographics), observation (laboratory results), procedure (treatment history), medication request, diagnostic report etc. we define a set of ehr documents (resources) as @xmath0, clinical guidelines are formatted in a hierarchical, tree-like structure (referred as checklist in this paper), where each guideline statement (parent node) can encompass an arbitrary number of subordinate child statements (sub-checklist or leaf node) nested within it as shown in figure 2 and 3. thus, we define a coverage guideline or checklist as @xmath1, where c is a checklist item. eventually, the task is to automatically deter-mine the medical necessity @xmath2, 0, 1} where -1 means the medical necessity is not justified, 1 means it is justified and 0 means there is a lack of sufficient evidence to justify the medical necessity criteria. recognizing the importance of transparency in the task, we also aim to provide evidence e @xmath3. these evidences can be used downstream to cross-reference medical documents used to establish medical necessity for the procedure. we aim to construct a machine learning model m such that: recently large language models have shown great performance improvements by breaking down complex tasks into simpler sub-problems @xcite . motivated by this observation, we propose a two step solution for our problem statement. first we determine the judgement of each of the leaf node checklist item. subsequently, we propagate the solution for parent nodes bottom-up based on its child nodes' judgements. ## data collection and annotation getting live ehr data for the purpose of this evaluation is difficult, costly and full of regulatory requirements. we therefore used de-identified discharge summaries from mimic-iv-note @xcite as a proxy for this data. all discharge summaries therein have sections like chief complaint, history of present illness, past medical history, social history, physical and lab examinations, medications etc. which serves as the ideal data for this experiment. an average discharge summary has approximately 300 sentences divided into different categories. joining this data with mimic-iv @xcite , we can get the cpt/icd-10 codes associated with each note. we also collected a set of publicly available clinical guidelines (from cms etc.) pertaining to cardiology and oncology and cross referenced the cpt codes in these guidelines to our notes data, thus creating a dataset of (note, guideline) pairs. an example checklist 3 is shown in figure 3 . the checklist shows the clinical guideline for therapeutic footwear which consists of two items associated by and operator. item 2 in itself is a sub-checklist and will be true if any of the sub-checklist item is true as all of them are connected by or operator. ## experiments and results our experiments were categorized into two distinct segments: assessment of leaf-node judgment and evaluation of parent-node judgment. to facilitate this, we established two separate test environments. each test-bed was equipped to integrate various large language models (llms) to ascertain the optimal model for our needs.