T riple C heck: Transparent Post-Hoc Verification of Biomedical Claims in AI -Generated Answers (2025)

Abstract: Retrieval Augmented Generation (RAG) has advanced Question Answering (QA) by connecting Large Language Models (LLMs) to external knowledge. However, these systems can still produce answers that are unsupported, lack clear traceability, or misattribute information — a critical issue in the biomedical domain where accuracy, trust and control are essential. We introduce TripleCheck, a post-hoc framework that breaks down an LLM’s answer into factual triples and checks each against both the retrieved context and a biomedical knowledge graph. By highlighting which statements are supported, traceable, or correctly attributed, TripleCheck enables users to spot gaps, unsupported claims, and misattributions, prompting more careful follow up. We present the TripleCheck framework, evaluate it on the SciFact benchmark, analyze its limitations, and share preliminary expert feedback. Results show that TripleCheck provides nuanced insight, potentially supporting greater trust and safer AI adoption in biomedical applications.

## introduction large language models (llms) augmented with retrieval, commonly referred to as retrieval augmented generation (rag), have significantly improved question answering (qa) by grounding responses in external sources. however, despite reducing hallucinations, these systems still exhibit key failures due to inherent system design constraints @xcite . in biomedical domains, especially in real-world industry, rag is relatively underexplored @xcite but distinct challenges have been pointed out, such as the lack of standard evaluation, unique ethical risks, and recurring problems with irrelevant or misleading information that hamper adoption in a field where both accurate and traceable information is crucial 1 . addition-1 see un news on who's warnings regarding generative ally, inaccurate or outdated references can compromise the quality of generated responses @xcite . human-ai collaboration research stresses the need for interaction designs that keep users engaged and aware @xcite . without careful explanation mechanisms, users may become overreliant on ai systems @xcite @xcite . paradoxically, conventional explanation techniques can increase user trust even when the ai is wrong, elevating the risk of unsubstantiated but plausible-sounding answers @xcite . this underscores the need for new approaches that better surface evidence and improve claim traceability. to address these gaps and foster appropriate trust in biomedical qa, we propose a post-hoc verification layer that provides fine-grained evidence assessment. biomedical fact-checking presents unique challenges: knowledge is constantly updated, and contextual nuance often determines the interpretation of evidence @xcite . overcoming these issues requires strategies that support more nuanced, context-aware evaluations. we introduce triplecheck, a system-agnostic post-hoc verification framework that can decompose ai-generated biomedical answers into factual triples and checks each for support within both the retrieved context and a large-scale biomedical knowledge graph that aggregates literature, patents, and clinical trials among other sources. this dual approach highlights statements that are supported, traceable, correctly attributed, and flags gaps such as misattributions or conflicting evidence from various sources. this can potentially help users recognize when to be skeptical or seek further evidence to make their own conclusions. by making the support and traceability of claims explicit, triplecheck aims to calibrate user trust and promote safe reliance on ai answers. our main contributions are: other methods propose post-generation claim checking, such as extracting claims from model outputs for kg validation @xcite , or hallucination detection using structured entailment checking over generated answer triples @xcite . however, these works either do not leverage an external kg for cross-checking (as in @xcite ), or they lack a userfacing explanation component (as in @xcite ). in contrast, triplecheck not only combines text entailment and kg validation in a dualevidence approach, but is also designed with userunderstandability and interaction in mind. our approach builds on these directions by proposing a zero-shot, post-hoc verification layer that can be added on top of any generative qa system. we uniquely leverage a large-scale biomedical kg to robustly cross-validate atomic answer triples, inspired by recent work in open-domain qa and fact-checking @xcite . importantly, triplecheck preserves the llm's original answer, instead surfacing supporting or contradictory evidence for each claim so users can make informed, nuanced judgment -an essential feature in the evolving and often ambiguous landscape of biomedical research. trust calibration and explainable qa interfaces trust calibration -the process by which user trust aligns with the true reliability of an ai system -has emerged as a critical factor in medical ai adoption @xcite . effective calibration can improve decision accuracy, yet achieving it remains challenging, as trust depends on perceived understandability, technical competence, and system reliability @xcite . inadequate calibration, whether overtrust or undertrust, can lead to unsafe outcomes in high-stakes biomedical environments. there is a growing consensus that ai systems in these domains must support user understanding and oversight through explainable interfaces @xcite . for example, @xcite describe an llm-assisted qa system with ex- (1) alignment with the retrieved context (documents or passages) and (2) cross-checking against a biomedical knowledge graph. each triple is then labeled as supported, unsupported, or contradicted based on both evidence sources. this claim-level verification can be presented to the user as an interactive interface that highlights which parts of the answer are trustworthy and which require caution. effective interfaces feature interactivity, enabling users to explore not only the answer, but why and how it was produced. this approach helps foster appropriate skepticism and engagement @xcite . in the biomedical domain, recent work by @xcite shows that providing multi-hop, interpretable rationales in a drug repurposing model, improved clinicians' accuracy, confidence, and decision efficiency, underscoring the value of transparent, actionable explanations. similarly, tools such as claim verification with evidence trails (e.g., using shap) improve decisions, though risk overreliance without careful design @xcite . while we do not fully explore the possibilities of building a sophisticated user interface in this work, triplecheck is explicitly designed to provide users with the information needed to calibrate trust and promote informed oversight. by breaking down answers into checkable factual units, labeling each as supported, unsupported, or contradicted, and surfacing the underlying evidence from literature or knowledge graphs, triplecheck of-fers fine-grained transparency. this enables users to scrutinize each claim with an appropriate level of skepticism or confidence, in line with findings within human computer interaction (hci) that emphasize user control as fundamental for trust calibration in ai @xcite . ## methodology: post-hoc claim verification with triplecheck system overview triplecheck acts as a postprocessor for a standard rag pipeline. suppose a user poses a question and the qa system produces an answer along with retrieved documents or passages as context. triplecheck takes this answer and its supporting context as input, and performs three main steps: (1) triple extraction, (2) evidence alignment, and (3) triple classification. the output is a set of annotated triples derived from the answer, each marked with whether it is supported by the context and/or by the external knowledge graph along with any additional evidence surfaced. figure 1 illustrates this workflow. as triplecheck is system-agnostic and never alters the original answers, it can be flexibly added to any qa workflow to provide a second layer of verification. triple extraction the first step breaks each answer into factual triples of the form (subject, pred-icate, object). for example: from "a deficiency of vitamin b12 increases blood levels of homocysteine, which is a risk factor for heart disease," we extract (vitamin b12 deficiency, increases, homocysteine levels) and (homocysteine, is a risk factor for, heart disease), each treated as an independent claim. our method follows recent approaches that combine large language models (llms) with post-hoc canonicalization of biomedical entities and relations @xcite . it integrates two main strategies: candidate triples from both methods are merged, with further processing to expand abbreviations (e.g., "tnf" → "tumor necrosis factor") and link entities to kg identifiers. triples referencing novel or out-of-ontology entities are excluded from kg validation using relations, but retained for textual entailment-based checking. to reduce spurious alignments that could arise during the linking process, an llm module screens for semantic consistency of the final triples to the system answer. the output is a set of cleaned, distinct factual triples asserted by the answer (see ## evaluation we evaluated triplecheck on the scifact benchmark @xcite , where claims are annotated as supported, refuted, or nei (not enough info). as access to the scifact test set labels is no longer available foot_1 , we perform evaluation on the development set similar to other studies @xcite . while we present results from several other methods on both test set and development set, our triplecheck's output, though more fine-grained, is mapped for comparison: we label the entire data point (a claim from the scifact dataset) as supported if all component triples are at least supported by the retrieved context and none are contradicted, refuted if any triple is contradicted, and nei otherwise. while this mapping is a simplification, it enables comparison on this benchmark. triplecheck achieves an f1 of 0.70 on scifact (dev set) in a zero-shot setting without any taskspecific fine-tuning, which is notable given that many comparison models, such as multivers @xcite and vert5erini @xcite , are tuned for this task. when comparing against other zero-shot approaches evaluated on the development set @xcite , triplecheck achieves competitive performance, and outperforms strong baselines. this underscores triplecheck's out-of-the-box robustness, even though our setup intentionally prioritizes transparency and explainability over strict optimization for scifact. the results can be seen in beyond aggregate scores, we also analyzed triplecheck's outputs for cases where it provides nuanced judgments that classic fact-checkers might miss. we found that about 10% of sci-fact's unsupported or contradicted claims were in triplecheck's supported by kg only category. upon inspection, it was evident that while the claim was not supported by the context, the claim was not a non factual claim, and we were able to collect evidence from the biomedical knowledge graph supporting this as an established fact. this reflects a traceability gap, highlighting where a claim may be true even if not cited. our proposed step for such claims is to improve traceability by fetching additional data, rather than labeling the full claim as non factual. for example, in scifact, the claim "a deficiency of vitamin b12 increases blood levels of homocysteine." is labeled as unsupported against the context, however, this is a known fact that is well supported in the kg. ## use cases and discussion triplecheck is broadly applicable to scenarios where users need to trust but verify ai-generated answers. we discuss a few use cases and their potential impact: literature review assistant: researchers often use qa systems to quickly summarize findings across papers (e.g., "what causes condition x?"). triplecheck would allow them to see which claimed causes are well-established (supported by multiple sources or kg) versus which are tentative more contested. it can also reveal if the system's answer includes claims not actually found in any cited papers, prompting the researcher to do a deeper dive for those claims. regulatory document drafting: in writing reports for drug approval or clinical guidelines, every statement needs a reference. an ai assistant could assist in drafting a section (e.g. drug efficacy) and triplecheck would immediately flag any statement that lacks backing from the retrieved studies or known medical facts. this helps authors more quickly pinpoint those evidence gaps, saving time and preventing unsubstantiated claims from slipping through. clinical decision support: a clinician asking an ai assistant about treatment recommendations could benefit from triplecheck's breakdown. for example, if the answer says "drug a improves outcome y and is not associated with side effect z," triplecheck might show the first claim is supported by a trial but the second claim is unsupported because the system didn't actually retrieve evidence about side effect z. the clinician thus knows to be cautious or look up that specific point. ## limitations and future work while promising, triplecheck has several limitations: ## conclusion we presented triplecheck, a post-hoc verification framework for biomedical qa that decomposes llm-generated answers into factual triples and verifies each against both retrieved context and a largescale biomedical knowledge graph. our scifact evaluation demonstrated that triplecheck achieves competitive zero-shot performance while providing fine-grained, interpretable evidence for each claim. initial expert feedback also suggested that this approach can support more calibrated trust, improve detection of unsupported or contested claims, and aid decision-making in biomedical settings. this initial feedback aligns well with anticipated real-world use, supporting the practical value of triplecheck in biomedical workflows. while promising, triplecheck faces challenges such as refining user interfaces to manage information load, and expanding coverage of supporting knowledge. most notably, future user studies are necessary to measure triplecheck's real-world impact on verification accuracy and user confidence. triplecheck represents a step toward more transparent, accountable biomedical ai by offering actionable, triple-level evidence to end users and developers. we hope this work encourages further development of evidence-aware qa frameworks, advancing safe and trustworthy use of ai in biomedicine.