Factuality Beyond Coherence: Evaluating LLM Watermarking Methods for Medical Texts (2025)

Abstract: As large language models (LLMs) are adapted to sensitive domains such as medicine, their fluency raises safety risks, particularly regarding provenance and accountability. Watermarking embeds detectable patterns to mitigate these risks, yet its reliability in medical contexts remains untested. Existing benchmarks focus on detection-quality tradeoffs and overlook factual risks. In medical text, watermarking often reweights low-entropy tokens, which are highly predictable and often carry critical medical terminology. Shifting these tokens can cause inaccuracy and hallucinations, risks that prior general-domain benchmarks fail to capture.We propose a medical-focused evaluation workflow that jointly assesses factual accuracy and coherence. Using GPT-Judger and further human validation, we introduce the Factuality-Weighted Score (FWS), a composite metric prioritizing factual accuracy beyond coherence to guide watermarking deployment in medical domains. Our evaluation shows current watermarking methods substantially compromise medical factuality, with entropy shifts degrading medical entity representation. These findings underscore the need for domain-aware watermarking approaches that preserve the integrity of medical content.

## introduction llms have advanced human-like text generation capability, raising concerns about potentially harmful or biased information in various use case, including in the medical domain @xcite @xcite @xcite . watermarking techniques serve as a safeguard by embedding subtle statistical patterns into generated content that enable detection while preserving quality @xcite @xcite . while watermarking methods show promise for securing information in llm outputs, their effectiveness in the medical domain remains underexplored @xcite . a key limitation is that watermarking often treats all tokens uniformly, ignoring low-entropy tokens @xcite . these are highly predictable words that often carry critical factual content in medical text, such as disease names. reweighting them to embed a watermark can alter their meaning, compromising factual accuracy and increasing the risk of hallucinations. current watermarking evaluation primarily assesses detection-quality tradeoffs from a domain-agnostic perspective, where quality assessment using human or llm evaluators focuses on measuring text preference @xcite or semantic coherence @xcite , without considering the domain-specific factual risks associated with low-entropy tokens to address the limitations of general evaluation workflows, our watermarking evaluation for medical text is illustrated in figure 1 . (1) evaluation workflow: a unified framework to accommodate both existing automatic metrics and gpt-judger assessment while examining the critical dimensions of factuality and coherence. this workflow is deliberately flexible, allowing traditional metrics to be integrated alongside our fine-grained gpt-judger assessments. the key innovation is in creating a structured evaluation that works across different measurement approaches and tasks. (2) factuality-weighted score (fws): to reflect the critical nature of factual integrity in medical applications, we introduce a composite quality metric that emphasizes factual correctness beyond coherence. we validate this approach through human evaluation, confirming alignment between expert judgments, our gpt-judger assessments, and the proposed fws metric, creating a reliable workflow for evaluating watermarked medical text. our analysis reveals critical trade-offs in wa-termarking methods for medical text. we find that generation-time watermarks, where watermarks are embedded during the llm's text generation process, achieve high detection rates but alter token entropy distributions, particularly affecting medical entities. our evaluation workflow demonstrates that factuality degradation is often more severe than coherence loss, a distinction missed by standard quality metrics. human evaluations show stronger agreement with our fws. these results highlight the need for tailored watermarking approaches in medical domains where factual integrity is crucial. we recommend future approaches: (1) develop domain-aware techniques that preserve lowentropy tokens of medical terminology and avoid hallucination while maintaining detectability, and (2) perform factuality-focused evaluation beyond general quality assessments. these insights aim to develop reliable watermarking methods balancing detectability with crucial medical factuality. to summarize, our contributions are four-fold: the code and experimental data are available at: ## evaluation workflow general workflow our framework is illustrated in figure 1 . first, we feed the medical llms with a prompt such as a medical question, and then we evaluate the quality of its watermarked and unwatermarked output. we evaluate watermarking methods across three key quality aspects. 1) coherence captures the fluency of the generated text and is commonly used in watermarking evaluation. in step 1, we implement and evaluate these three quality aspects through two approaches: in step 2, we evaluate the overall quality by proposing a quality metric factuality-weighted score to see how the watermarking method affects factuality. ## results we present our analysis of watermarking methods in answering a series of research questions defined 3 https://hf.co/johnsnowlabs/jsl-medllama-3-8b-v2.0 in § 4.1. we begin by examining traditional automatic evaluation metrics in § 5.1 (rq1), comparing detectability measures against standard quality metrics to establish baseline performance. we then extend our investigation to task-specific metrics in § 5.2 (rq2), providing deeper insights regarding its resemblance of factuality aspects. building on the observed findings, we provide our factuality-weighted score (fws) metrics in § 5.3 (rq3), which emphasize the factuality quality preservation. we validate the proposed metric through correlation analysis, comparing existing traditional automatic metrics and gpt-judger assessments, and verify its alignment with human perception. finally, we conduct an in-depth observation in § 5.4, to check the possible contributing factors to factuality corruption in watermarked texts (rq4), offering insights for future watermarking method improvements. ## conclusion our evaluation of llm watermarking methods for medical texts reveals a critical trade-off: while current approaches achieve high detection capabilities, they compromise factual integrity. the proposed factuality-weighted score addresses this concern by prioritizing factual accuracy over coherence, validated through human evaluations. our analysis shows watermarking shifts token distributions, affecting medical terminology and introducing entity hallucinations. these findings highlight the need for domainaware watermarking techniques that preserve medical content integrity while maintaining detectability. we recommend developing approaches that specifically account for medical terminology alongside more robust factuality-focused evaluation protocols. as llms continue to be deployed in healthcare settings, ensuring content authenticity and factual reliability remains important.