Assessing and Mitigating Medical Knowledge Drift and Conflicts in Large Language Models (2025)

Abstract: Large Language Models (LLMs) offer transformative potential across diverse fields, yet their safe and effective deployment is hindered by inherent knowledge conflicts—stemming from temporal evolution, divergent sources, and contradictory guidelines. This challenge is particularly acute in medicine, an interdisciplinary frontier for NLP. Rapid medical concept drift can lead LLMs to provide incorrect or outdated advice, impacting their utility and the broader societal benefits of NLP advances. This study introduces ConflictMedQA, a benchmark designed to systematically evaluate how LLMs manage varied knowledge conflicts in clinical guidelines. Our assessment of seven state-of-the-art models across 4,290 scenarios reveals significant difficulties in rejecting incorrect recommendations and frequent endorsement of conflicting advice, highlighting an important gap for NLP systems intended for real-world impact. We explore two fundamental mitigation approaches: retrieval-augmented generation and preference fine-tuning via direct preference optimization. While each offers improvements, their synergistic combination yields the best results. These findings emphasize the need for LLMs to discern subtle but critical guideline conflicts. This is a crucial step in advancing NLP’s capabilities and ensuring its dependable application in critical societal domains. The proposed dataset is available at https://huggingface.co/datasets/RDBH/DriftMed.

## introduction the rapid expansion of biomedical knowledge, driven by swift research and medical advancements, increasingly strains healthcare delivery @xcite @xcite . clinicians struggle to stay current as standard practices can quickly become obsolete @xcite , with clinical guidelines-the formal standards of medical knowledge-often needing reassessment within years @xcite . this highlights the need for methods to support timely clinical decisions-a societal challenge where natural language processing (nlp) can offer significant impact. large language models (llms) are promising tools to navigate this information, showing strong clinical text comprehension and reasoning @xcite @xcite . while healthcare explores their integration @xcite , their transformative potential hinges on rigorously understanding limitations beyond exam accuracies. research has widely explored clinical biases in llms @xcite . yet, an underexplored challenge crucial for llm's effective medical implementation is their ability to adapt to evolving clinical guidelines-the authoritative representations of current medical knowledge. this guideline evolution creates two challenges. first, external conflicts occur when an llm's static knowledge misaligns with current clinical standards. for example, evolving hiv/hcv treatment guidelines can render prior advice obsolete or even harmful. second, internal knowledge conflicts arise when llms assimilate contradictory guidelines from diverse training data @xcite . the nice-sugar study on glucose control @xcite exemplifies this challenge, where intensive glucose management, once recommended in guidelines, was later found to increase mortality. such guideline reversals erode trust and impede nlp's impact when llms provide contradictory advice @xcite . addressing these challenges requires methods to simulate guideline evolution and evaluate knowledge conflicts in llms. current medical benchmarks predominantly focus on static knowledge and well-established facts, neglecting how knowledge evolves over time. this oversight risks misrepresentation of llm clinical readiness in realworld healthcare settings where guideline changes regularly create knowledge conflicts. we developed conflictmedqa (fig. 1 ), a benchmark that simulates guideline evolution to assess how llms manage conflicts between previous and current medical knowledge standards. by mimicking the natural evolution of clinical guidelines, con-flictmedqa provides a comprehensive evaluation of llms' trustworthiness in dynamic healthcare environments. this work's contributions are: • we introduce conflictmedqa, a benchmark assessing llms' handling of resulting knowledge conflicts in healthcare. ## mitigating strategies we explored three strategies to address this challenge: non-parametric knowledge update, parametric knowledge adaptation, and hybrid knowledge augmentation. non-parametric update was applied to all evaluated llms. due to limited training resources and lack of access to proprietary model weights, parametric and hybrid knowledge update strategies were evaluated only on qwen2.5-7b, ministral-8b, and llama-3-8b. ## discussion our evaluation on the conflictmedqa benchmark reveals significant challenges for llms in clinical decision-making, primarily their struggle with the temporal dynamics of medical knowledge and internal consistency. even advanced models, adept at endorsing current guidelines, often faltered markedly when required to reject outdated advice. this asymmetry, coupled with the finding that larger model scale does not consistently reduce internal knowledge conflicts, suggests that unique complexities arise in this domain beyond standard nlp capabilities. these issues, especially prevalent in areas like therapeutic recommendations, could pose direct risks if llms are integrated into clinical workflows without a deep understanding of their failure modes. investigating mitigation strategies offered further insights. while rag generally improved adherence to current information, its utility was nuanced. notably, for smaller models, rag alone could paradoxically degrade their ability to reject outdated advice, suggesting that merely providing external information can be counterproductive if the model lacks the capacity to critically discern and integrate it, potentially overwhelming weaker internal knowledge structures. this indicates that effective retrieval is as much about the model's ability to use information as it is about accessing it. dpo offered a simple complementary approach, demonstrably enhancing alignment with current guidelines and reducing internal conflicts. however, these improvements in complex clinical scenarios stood in contrast to the near-perfect performance models presumably achieve on the specific raw medical advice pairs used during dpo training. this discrepancy suggests a significant challenge in generalizing knowledge learned from such simple pairs to the multifaceted reasoning required in clinical practice, hinting at a gap between memorized correct responses and their robust, contextual application. the most promising path appears to be the synergistic combination of these approaches. our findings show that rod, applying rag to dpo-tuned models, yielded substantial improvements across all metrics, particularly in enhancing smaller models' rejection of outdated advice and minimizing internal conflicts. these gains always exceed the sum of the individual contributions from rag-only or dpo-only applications. while models may struggle to effectively apply dpo-learned parametric knowledge across diverse and complex scenarios, the integration with rag appears pivotal. knowledge retrieved via rag seems to activate relevant dpo-instilled parametric knowledge within the model, leading to these markedly enhanced outcomes and avoiding the potential side effects of rag-only or the more modest improvements from dpo-only strategies. these observations also underscore a significant limitation of evaluating llms using metrics focused on isolated factual accuracy. the marked performance decline when models face realistic clinical scenarios, which embed cognitive complexities and factors like sdoh, emphasizes the strong need for evaluation methodologies that capture the multifaceted nature of clinical decision-making. ## conclusion ultimately, for the safe and effective integration of llms into clinical practice, future efforts should prioritize the development of robust, hybrid methodologies designed to enhance adaptability to evolving knowledge and ensure internal consistency. this entails creating more contextually rich training and evaluation paradigms that mirror the complexity of real-world clinical encounters, thereby moving beyond isolated assessments to foster genuine contextual understanding and reliability in these critical systems.