0x. Y uan at S em E val-2024 Task 2: Agents Debating can reach consensus and produce better outcomes in Medical NLI task (2024)

Abstract: In this paper, we introduce a multi-agent debating framework, experimenting on SemEval 2024 Task 2. This innovative system employs a collaborative approach involving expert agents from various medical fields to analyze Clinical Trial Reports (CTRs). Our methodology emphasizes nuanced and comprehensive analysis by leveraging the diverse expertise of agents like Biostatisticians and Medical Linguists. Results indicate that our collaborative model surpasses the performance of individual agents in terms of Macro F1-score. Additionally, our analysis suggests that while initial debates often mirror majority decisions, the debating process refines these outcomes, demonstrating the system’s capability for in-depth analysis beyond simple majority rule. This research highlights the potential of AI collaboration in specialized domains, particularly in medical text interpretation.

## introduction clinical trial reports (ctrs) are indispensable in clinical research, providing critical data that reveal the efficacy of new treatments on patients. however, the exponential growth in the volume of ctrs, due to the increase in clinical trials, challenges researchers in conducting individual report analyses. with the swift progress in natural language processing (nlp) technologies, leveraging machine learning algorithms for automating the review of ctrs is increasingly recognized as a feasible and promising solution @xcite . for semeval-2024 task 2 @xcite , the organizers introduced an english dataset derived from ctrs @xcite , aimed at evaluating the truthfulness of ctr-statement pairs by discerning their veracity. this dataset includes a series of ctrs alongside associated statements, each designed to represent a hypothesis that must be classified as either entailment or contradiction, based on its alignment with the ctr content. in addressing this intricate challenge, our study introduces a novel multi-agent debating framework. characterized by a diverse assemblage of expert agents -including but not limited to a bio-statistician, medical linguist, and pharmacologist -this system facilitates structured debates to adjudicate on the classification of each statement as an entailment or contradiction. by harnessing the distinctive expertise and viewpoints of various agents, we significantly augment the precision and dependability of our assessments. our observations indicate that consensus among agents typically emerges within the second or third round of discussion, with agents exhibiting varied opinions on the statements under review. this multi-agent debate approach has demonstrably surpassed the outcomes achievable through single-agent or direct large language model (llm) interventions. despite not achieving top-tier placement on the leaderboard, largely due to our adoption of a zero-shot approach without model fine-tuning, our system's broad applicability across different domains remains a compelling advantage. ## experiments our experiments were conducted using the mixtral-8x7b model @xcite , selected for its exceptional performance and cost-efficiency. ## conclusion we have presented a novel multi-agent debating framework to participate semeval-2024 task 2. this approach, integrating the expertise of diverse agents like biostatisticians, medical linguists, and pharmacologists, significantly enhances the analysis of clinical trial reports (ctrs). our findings demonstrate improved performance in entailment or contradiction determination of ctr-statement pairs, as evidenced by enhanced macro f1-scores compared to individual agent assessments. despite a tendency to align with initial majority decisions, the debating process refines these initial judgments, indicating the framework's effectiveness beyond simple majority rule.