Facts Fade Fast: Evaluating Memorization of Outdated Medical Knowledge in Large Language Models (2025)

Abstract: The growing capabilities of Large Language Models (LLMs) can enhance healthcare by assisting medical researchers, physicians, and improving access to health services for patients. LLMs encode extensive knowledge within their parameters, including medical knowledge derived from many sources. However, the knowledge in LLMs can become outdated over time, posing challenges in keeping up with evolving medical recommendations and research. This can lead to LLMs providing outdated health advice or failures in medical reasoning tasks. To address this gap, our study introduces two novel biomedical question-answering (QA) datasets derived from medical systematic literature reviews: MedRevQA, a general dataset of 16,501 biomedical QA pairs, and MedChangeQA, a subset of 512 QA pairs whose verdict changed though time. By evaluating the performance of eight popular LLMs, we find that all models exhibit memorization of outdated knowledge to some extent. We provide deeper insights and analysis, paving the way for future research on this challenging aspect of LLMs.

## introduction the advent of pre-trained large language models (llms) has revolutionized the field of natural language processing (nlp) @xcite . one of their most promising application domains is healthcare, where they hold the potential to democratize access to health services and assist in crucial clinical workflows @xcite @xcite . llms are trained to predict the next token on massive amounts of text data, which results in deeply encoding a lot of knowledge in their weights @xcite . recent studies suggest that llms encode clinical knowledge effectively @xcite , by being trained on medical texts like patient records and clinical guidelines. the model's ability to recall specific facts from this data is often referred to as memorization @xcite . world knowledge quickly evolves in dynamic domains like entertainment or politics. however, this also happens with scientific knowledge. in medicine, new high-quality evidence constantly emerges, rendering previous recommendations obsolete @xcite . consequently, the knowledge memorized by an llm at its training time can become outdated, as they struggle to keep up with the evolving world knowledge @xcite . this is a major safety concern, as it can lead to llms providing consumers with incorrect health advice @xcite or fail in clinical settings when using flawed facts in their reasoning @xcite . even when augmented with retrieved up-to-date information, llms can reject it and resort to internal knowledge in so-called knowledge conflicts @xcite . while recent work has explored the memorization of outdated knowledge in the encyclopedic domain @xcite , the temporal decay of medical knowledge has been less explored. to address this critical gap, we con-struct new datasets and use them to evaluate memorization of outdated medical knowledge: the datasets and code are publicly available. 1 recent work has explored how to measure memorized training examples in llms @xcite @xcite . similarly, temporal qa datasets have been constructed to investigate quickly changing knowledge, mostly focusing on the general, encyclopedic domain @xcite @xcite . to the best of our knowledge, we introduce the first qa dataset focusing on knowledge change specifically for the medical domain and the first investigation of how much outdated medical knowledge popular llms encode. conclusion: long-term antibiotics may reduce the risk of repeat symptomatic uti in children who have had one or more previous utis but the benefit may be small and must be considered together with the increased risk of microbial resistance. (...) @xcite verdict: supported question: does long-term antibiotic use help prevent recurrent urinary tract infections in children? conclusion: large, properly randomised, double blinded studies are needed to determine the efficacy of long-term antibiotics for the prevention of uti in susceptible children. (...) @xcite verdict: not enough information dataset construction. pubmed, the largest database of medical publications @xcite , contains all the cochrane systematic review abstracts from 2000 to 2024 (until january 2024, when we scraped). we built a python scraping script using beautifulsoup and scraped all the cochrane slr abstracts. 2 every slr in the dataset consists of the same sections: background, objectives, search methods, selection criteria, data collection and analysis, main results, and authors' conclusions. our final qa dataset consists of questions and labels. we used gpt-4o-mini-2024-07-18 to semi-automatically construct the questions and labels, by providing it with the full slr abstract. questions were derived from the objectives section, by rewriting them to the interrogative form. labels originate from the authors' conclusions sec-tion -the llm selects one of the labels supported, refuted, or not enough information (nei), as the final label. these labels align with common labels in other medical qa and fact-checking datasets @xcite . in total, this dataset has 16,501 qa pairs, spanning virtually all medical disciplines and covering a wide array of important biomedical questions for benchmarking. we call this dataset medrevqa. changed knowledge. our dataset consists of 16,501 slr records. out of those, 12,122 are unique slrs that have never had an update. the remaining 4379 slrs constitute 1535 groups (with a minimum of 2 slrs in a group, a maximum of 9, and a mean of 2.85) that researched the same question. this means there are 1535 research questions that have had multiple slr iterations written about them. out of 1535, 512 have had a verdict change over time, meaning that the authors changed the conclusion of the investigated research question in a follow-up slr study, when they acquired updated evidence from research. this follows findings from medical research studies that have shown how 20 to 30% of cochrane reviews change their conclusions throughout time @xcite . we consolidate these questions with changed verdicts into the medchangeqa dataset and collect all their verdicts through different iterations. medchangeqa has questions, latest label, and (the most recent) outdated label for those studies where the label changed. annotation quality. two annotators, one who is our in-house physician from the university clinic and another an author with a background in biomedical engineering, evaluated a random subset of 100 examples for the correctness of generated questions and verdicts. they found 95% of questions and 92% of labels to be correct. we deem this to be relatively high, since even the human annotation process is imperfect @xcite , with errors due to incorrect problem understanding or loss of concentration. a common source of label errors was conflating refuted and nei labels. on the other hand, all 512 labels in medchangeqa were manually checked and corrected by the two annotators. therefore, medrevqa has silver labels, while medchangeqa has gold labels. ## experimental setup the experiments consisted of instructing the llms to predict one of the three labels (s/r/nei), given the medical question as input. no additional context was provided, as the goal is to evaluate their internal knowledge and memorization. the models also explained their output. for evaluation, we extract the predicted label and compare it to groundtruth labels from the dataset, using the (macroaveraged) precision, recall, and f1 scores. we test multiple llms, starting with gpt-4o (2024-08-06), as the most popular proprietary llm. we also benchmark four open-weights llms: mistral 24b, llama 3.3 (70b), qwen 2.5 (7b), deepseek-v3 (685b); and finally the fully open-source olmo 2 (13b). see appendix b for a summary of public info on their pre-training data. to compare general-purpose llms to domainspecific ones, we also benchmark the performance of pmc-llama 13b @xcite , an extension of llama 2, and biomistral 7b @xcite , an extension of mistral-v0.2; both further pre-trained on biomedical research papers. all prompts can be found in ## results experiment rounds. we first test (a) the full dataset, medrevqa. we also did two experiments on medchangeqa, first with (b) outdated labels as ground truth, then with (c) latest labels as ground truth. we use the difference between the scores of (b) and (c) as a proxy to show the extent of outdated medical knowledge in llms. final results are systematized in and f1. the last column shows the percentage of answers in the 3rd experiment (c) where the outdated label was predicted (and not the correct latest label or an incorrect label altogether). performance. on the full dataset, mistral exhibited the best r and f1, showing it has the best overview of the overall medical knowledge landscape. precision was the highest in deepseek-v3. nevertheless, none of the models has a very high performance, pointing to the challenging nature of medrevqa as a general biomedical qa testbed. when it comes to outdated knowledge, llama 3.3 had the highest degree of the latest knowledge as compared to the outdated labels (+7.4), while olmo also had a positive difference (+2.9). mistral showed an almost identical performance, while gpt, qwen, deepseek, and pmc-llama all struggled. qwen was also the smallest and least capable model, which could explain low scores in general and low awareness of recent knowledge. an example of outdated and incorrect knowledge is shown in ## discussion and analysis pre-training data. most popular llms do not fully disclose their pre-training data, making it difficult to assert if concrete medical studies were memorized. still, recent studies demonstrated empirically the presence of memorized medical datasets @xcite . we also saw a tendency of models to explicitly mention specific studies in their answers, including cochrane reviews, many of which were decade-old (see inspection of olmo. olmo 2 (olmo et al., 2025) is trained on the dolma corpus @xcite , a fully open dataset containing around 3 trillion tokens. it contains the pes2o @xcite and s2orc @xcite corpora that constitute the academic knowledge base semantic scholar. this database also indexes all of cochrane's systematic literature reviews. 3 therefore, we can with high certainty say that the olmo models have seen cochrane's slrs during its pretraining. other than in the two academic corpora, there is a wide presence of these slrs in other parts of the dataset, especially various online websites found in common crawl @xcite . gram language model that can be used to query dolma and other pre-training corpora, foot_2 to inspect the presence of cochrane's slrs. searching for "cochrane database of systematic reviews" (the exact journal name, case-sensitive) returns 144,493 hits for dolma v1.7 (used for olmo 2). additionally, we queried the title of each of the slr studies found in medrevqa and report on the mean and median amount of n-gram counts per year in figure 2 . the mean and median almost steadily decrease over years, meaning that the most mentioned and discussed studies are the earliest ones since they have had more time to spread throughout the web. the higher frequency of mentions can lead to to stronger encoding of outdated knowledge in llms. ## conclusion we introduce two new qa datasets constructed from rigorous biomedical slrs for benchmarking the general biomedical knowledge of llms. with a subset of 512 questions where the answer (verdict) changed over the years, we showcase how eight popular llms fare better on older medical knowledge and encode a considerable amount of outdated knowledge labels, which can hinder their usability in healthcare settings, including helping physicians, researchers, and patients. we outline future work directions and hope our datasets will serve as a challenging testbed for tackling the llm memorization of outdated knowledge.