M ed C ite: Can Language Models Generate Verifiable Text for Medicine? (2025)

Abstract: Existing LLM-based medical question answering systems lack citation generation and evaluation capabilities, raising concerns about their adoption in practice. In this work, we introduce MedCite, the first end-to-end framework that facilitates the design and evaluation of LLM citations for medical tasks. Meanwhile, we introduce a novel multi-pass retrieval-citation method that generates high-quality citations.Our extensive evaluation highlights the challenges and opportunities of citation generation for medical tasks, while identifying important design choices that have a significant impact on the final citation quality. Our proposed method achieves superior citation precision and recall improvements compared to strong baseline methods, and we show that our evaluation results correlate well with annotation results from professional experts.

## introduction large language models (llms) have demonstrated remarkable capabilities in various natural language processing tasks, such as question answering (qa) and instruction following @xcite . progress in llms has also enabled the development of medical agents that understand language used by patients and physicians, offering rich just-in-time assistance @xcite @xcite @xcite . while the early signs are positive, current llmpowered medical qa systems still have multiple limitations. for example, medical data often contains sensitive information, such as personal health records, which cannot be used for training large language models without strict compliance with ethical standards @xcite . furthermore, trustworthiness is particularly important in the medical field. issues such as hallucination, where the model generates information that is incorrect or misleading, pose significant challenges to the reliability of llm-based medical systems @xcite @xcite . to overcome the issue, researchers and practitioners have studied retrieval-augmented generation (rag) @xcite , which combines llms with information retrieval from external trustworthy data sources @xcite . by providing the model with accurate and relevant medical knowledge, these systems allow llms to maintain relevance in responses. despite promising results, existing methods lack verifiability @xcite , meaning that the answers provided are not backed by reliable sources or evidence. this can lead to misinformation and potentially harmful consequences if incorrect medical advice is followed. for instance, as shown in fig. 1 , when providing a diagnosis based on a list of symptoms without any references, the accuracy of prognosis or treatment recommendations cannot be assured, which creates a sense of uncertainty, leading to suboptimal or even harmful decisions. one promising approach to mitigate the verifiability concern is through attribution @xcite , i.e., associating statements with citations, which offers the system more credibility and accountability while providing users a way to explore the source in greater depth and verify the information source. however, although there are prior efforts that analyze citation capabilities through llms on general domain qa tasks @xcite @xcite , citing sentences for medicine is especially challenging and not widely adopted in practice due to the following reasons. first, existing works on medical qa often leverage multi-choice accuracy to benchmark and evaluate their performance @xcite @xcite , which focuses on evaluating the ability to select the correct answer from a set of given options. however, citation generation is more challenging due to its open-ended nature. for example, when prescribing medication for a rare genetic disorder or planning surgery for a patient, both physicians and patients have to rely on richer information. therefore, a query can have multiple answers, supported by multiple possible sources. these aspects are important to consider for the evaluation of citation generation methods, but existing medicine qa frameworks do not inherently account for them. second, there is a huge design space for citation generation with complex interactions among re-trievers @xcite , backbone llms @xcite @xcite , and citation generation algorithms @xcite . therefore, it can be challenging to determine which aspects contribute the most. however, the analysis is crucial for developing strategies to improve the verifiability of medical systems. third, while contributions continue to rise in this field @xcite @xcite , there is a noticeable deficiency in open-source frameworks useful for designing, developing, and evaluating citation generation quality for medical tasks. the existing citation evaluation frameworks are constructed from generic questions, where the selection of metrics and evaluators for medical tasks remains an open question. moreover, it is quite costly to obtain high-quality medical expert annotations, which demands a high-quality classifier to judge whether a citation attributes to a statement. in this paper, we tackle these challenges with the hope of fostering research in improving verifiability for medicine systems. in particular, our contributions are as follows: ## problem setup in this section, we first formulate the citation generation task for biomedical qa and then give an overview of the approaches that we will examine experimentally in the following section. ## medcite: a citation generation in the previous section we investigate several important design choices for citation generation of medical tasks. we now aggregate these improvements and evaluate their combined impact and provide it as an open-source framework medcite (fig. 2 ). our final citation generation approach integrates three core components: non-parametric citation ( § 4.1), rag ( § 4.2), and the retrieval + llm reranking citation seeking method ( § 4.3). additionally, we investigate another two important factors that have been under-emphasized in previous work: (1) what if we combine parametric and non-parametric citations through multi-pass approaches; and (2) the impact of retriever choices to the citation seeking. multi-pass citation generation. intuitively, it seems possible to leverage both llm's internal parametric knowledge to provide initial answer and citations while employing post-generation nonparametric method to validate and refine the citations, utilizing the externally retrieved content. to verify our hypothesis, we consider a new multipass method: similar to the approach in § 4.2, we employ rag to generate answers. different from that approach, we instruct llm to assign citations to statements based on the retrieved documents while answering the question. then we retrieve top-k relevant documents to each statement. we deduplicate any redundant citations from these two stages and combine the remaining ones to form the final citations. hierarchical two-stage ranking based citation retrieval. another factor is the choice of retriever for the citation seeking. the recently proposed medrag @xcite uses a reciprocal rank fusion (rrf) based hybrid method to combine results from bm25 @xcite and medcpt @xcite to find supporting documents in the pre-generation phase. however, while it is possible to find a broad range of relevant documents to enhance the context of llm-generated answers, citation retrieval must be more fact-focused to ensure precise and accurate referencing. in the ablation studies, we show that a hierarchical two-stage ranker that first retrieve documents based on key word matching through bm25 @xcite and then semantic retriever based on medcpt @xcite brings further improvements in performance in citation quality, validating the importance of the choice of retriever for citation. ## evaluation 6.1 main results we compare medcite with three baseline methods: the medical domain rag method and two general-domain citation methods from recent work, including the post-retrieval generation and postgeneration citation method across different backbone llms: (1) medrag: the method described in @xcite . (2) post-retrieval generation (prg): following the method in @xcite , we prompt llms with a query, a list of retrieved documents and instruct the llms to include citations in their generated answer. medcite outperforms prg and pgc in citation quality. while both prg and pgc enable citation for medicine, medcite outperforms the two methods by a large margin (e.g., 71.74% vs. 66.96% and 32.50% on bioasq). medcite outperforms prg because medcite's second pass of citation seeking leverages post-generation non-parametric retrieval to refine the citations, which allows llms to mitigate citation hallucinations. medcite obtains better performance than pgc, because it exploits pre-generation retrieval and llm's internal parametric knowledge to obtain an initial set of citations, which turns out to be useful for obtaining high-quality final citations. these results have demonstrated medcite's effectiveness in combining the strengths of both generative and retrieval systems for citation generation. medcite consistently brings citation quality improvements over different llms. we see a universal trend that medcite improves citation recall and f 1 score across llms. using gpt-4o as the backbone llm leads to the highest-performing citation quality (e.g., gpt-4o 86.48 vs. llama-3-i. 68.60 in f 1 on pubmedqa), mainly driven by its advanced reasoning and instruction following capabilities. in contrast, citation quality is the lowest when the system is evaluated on ultramedical (e.g., 66.71 on bioasq). these results underscore that incorporating medcite bolsters llm's capacity to generate verifiable texts. ## conclusion we introduce medcite, the first end-to-end framework fostering research that targets improving the verifiability and trustworthy of medical systems with citations. our in-depth examination of important design choices for llm-based medical systems inspires us to propose medcite, a novel method for generating high quality citations for medical systems. extensive evaluation across llms show that our approach leads to consistent improvements to citation generation over alternative methods.