I ryo NLP at MEDIQA - CORR 2024: Tackling the Medical Error Detection & Correction Task on the Shoulders of Medical Agents (2024)

Abstract: In natural language processing applied to the clinical domain, utilizing large language models has emerged as a promising avenue for error detection and correction on clinical notes, a knowledge-intensive task for which annotated data is scarce. This paper presents MedReAct’N’MedReFlex, which leverages a suite of four LLM-based medical agents. The MedReAct agent initiates the process by observing, analyzing, and taking action, generating trajectories to guide the search to target a potential error in the clinical notes. Subsequently, the MedEval agent employs five evaluators to assess the targeted error and the proposed correction. In cases where MedReAct’s actions prove insufficient, the MedReFlex agent intervenes, engaging in reflective analysis and proposing alternative strategies. Finally, the MedFinalParser agent formats the final output, preserving the original style while ensuring the integrity of the error correction process. One core component of our method is our RAG pipeline based on our ClinicalCorp corpora. Among other well-known sources containing clinical guidelines and information, we preprocess and release the open-source MedWiki dataset for clinical RAG application. Our results demonstrate the central role of our RAG approach with ClinicalCorp leveraged through the MedReAct’N’MedReFlex framework. It achieved the ninth rank on the MEDIQA-CORR 2024 final leaderboard.

## introduction in natural language processing applied to the clinical domain, the accurate detection and correction of medical errors are paramount tasks with profound implications for patient care and safety. this paper introduces the multi-agent framework medreact'n'medreflex, meticulously engineered to tackle medical error detection and cor-rection, as delineated in the mediqa-corr 2024 competition. our framework integrates four distinct types of medical agents: medreact, medreflex, medeval, and medfinalparser, each playing a specialized role in the error identification and rectification process. drawing inspiration from existing frameworks like react @xcite and reflexion @xcite , our framework orchestrates a structured approach to error handling. leveraging a retrieval-augmented generation (rag) framework @xcite based on medrag @xcite and medcpt @xcite , our approach operates over clinical-corp, an extensive corpora curated to encompass crucial clinical guidelines. additionally, we introduce medwiki, a collection of medical articles from wikipedia. by integrating these resources, our approach seeks to advance state-of-the-art clinical nlp by offering a comprehensive solution tailored to the intricate nuances of medical error handling. furthermore, this paper documents the construction and release of medwiki, a substantial repository comprising over 1.3 million article chunks. additionally, we detail the assembly of the clinicalcorp, a comprehensive corpus comprising medwiki along with other clinical guideline datasets, such as parts of the medcorp corpora @xcite and parts of the guidelines @xcite . our main contributions are: ## medreact'n'medreflex framework unlike previous multi-agent frameworks @xcite , our approach diverges from a free conversation format to adopt a fixed design schema, as illustrated in figure 1 . within this structured framework, each medical agent intervenes at a specific step, facilitating a systematic and coordinated approach to address the error detection and correction task. central to our methodology are four distinct medical agents: medreact, medreflex, medeval, and medfinalparser. ## conclusion in this paper, we introduced medreact'n'medreflex, a multi-agent framework developed for the mediqa-corr 2024 competition aimed at medical error detection and correction in clinical notes. the framework incorporates four specialized medical agents: medreact, medreflex, medeval, and medfinal-parser, leveraging the rag framework and our clinicalcorp. we detail the construction of our clinicalcorp, including diverse clinical datasets such as guidelines, textbooks, and statpearls. additionally, we released medwiki, a corpus comprising wikipedia medical articles. our framework achieved the ninth rank in the competition with an aggregation score of 0.581. through optimization experiments, we identified sub-optimal settings at the time, demonstrating substantial performance improvements with a retrieval top-k of 50, a reranking top-k of 20, an average review threshold of 3.8, and a minimum review threshold of 3. as future work, we envision refining the chunking strategy on the clinicalcorp, applying further prompt engineering of the medical agents, and conducting a deeper analysis of the interactions between the medreact'n'medreflex's agents.