L ong C ite: Enabling LLM s to Generate Fine-grained Citations in Long-Context QA (2025)

Abstract: Though current long-context large language models (LLMs) have demonstrated impressive capacities in answering various questions based on extensive text, the lack of citations in their responses makes user verification difficult, leading to concerns about their trustworthiness due to the potential hallucinations. In this work, we aim to enable long-context LLMs to generate responses with fine-grained sentence-level citations on the fly, thereby improving their faithfulness and verifiability. We first introduce LongBench-Cite, an automated benchmark for assessing current LLMs’ performance in long-context question answering with citations (LQAC), revealing considerable room for improvement. To this end, we propose CoF (Coarse to Fine), a novel pipeline that utilizes off-the-shelf LLMs to automatically construct long-context QA instances with precise sentence-level citations, and leverage this pipeline to construct LongCite-45k, a large-scale SFT dataset for LQAC. Finally, we train LongCite-8B and LongCite-9B using the constructed dataset, successfully enabling the generation of accurate responses and fine-grained citations in one pass. The evaluation results on LongBench-Cite show that our trained models achieve state-of-the-art citation quality, surpassing advanced proprietary models including GPT-4o. We also discover that SFT with citation information can further improve the correctness of model responses compared to standard long-context SFT.

## introduction recent years have witnessed significant advancement in long-context large language models (llms), enabling them to address various user questions based on lengthy texts that surpass † work done when jz and yb interned at zhipu.ai. * corresponding author. 100,000 tokens @xcite @xcite . despite their remarkable capacities, current long-context llms typically do not provide citations to specific context snippets to support the statements they generated, making it challenging for users to verify model outputs given the substantial context lengths. this significantly impacts the reliability and trustworthiness of longcontext llms, especially considering that they still struggle with hallucinations @xcite and are prone to generate unfaithful content. on the other hand, recent works in search engines and open-domain qa have allowed llms to generate responses with in-line citations through retrieval-based generation (rag) or post-hoc methods @xcite @xcite . nevertheless, as shown in figure 1 , these approaches still expose notable limitations in long-context scenario: rag often leads to compromised response quality due to incomplete context information, while post-hoc methods prolong the user waiting time due to more complicated pipelines. in addition, their generated citations typically refer to entire web pages @xcite or coarsely chunked snippets @xcite , thereby requiring users to further locate the specific supporting evidence for the final verification. in light of these challenges, we are curious: can we directly employ long-context llms to generate responses with sentence-level citations on the fly based on the whole context, thereby ensuring high response quality, fine citation granularity, and normal user waiting time simultaneously? to this end, we first propose longbench-cite, an automatic benchmark to evaluate llms' performance on the task of long-context question answering with citations (lqac), and find that existing longcontext llms obtain unsatisfactory results (sec. 3). specifically, we find that citations produced by these llms either cannot fully support their response, or have a coarse granularity. meanwhile, we observe that generating citations on the fly via in-context learning generally results in less correct responses compared to vanilla long-context qa. to further enhance the inherent capacity of llms for generating fine-grained citations from lengthy contexts, it is essential to construct a highquality sft dataset. to this end, we introduce cof (abbr. for "coarse to fine"), a novel pipeline that utilizes off-the-shelf llms to automatically construct long-context qa instances with precise sentence-level citations (sec. 4). cof comprises four stages: (1) starting with a long text material, cof first invokes the llm to produce a query and its associated answer through self-instruct @xcite . (2) next, cof uses the answer to retrieve several chunks from the context, which are then fed into the llm to incorporate coarsegrained chunk-level citations into the answer. (3) the llm subsequently identifies evidence sentences from each cited chunk to produce finegrained citations. (4) as a final step, instances with insufficient citations are discarded. our experiments validate the superiority of cof over other lqac strategies in terms of answer correctness and citation quality. using cof pipeline, we construct longcite-45k, a large-scale sft dataset that consists of 44,600 high-quality lqac instances with contexts up to 128,000 tokens. finally, we use longcite-45k to fine-tune glm-4-9b @xcite and llama3.1-8b @xcite , two latest opensource long-context models (sec. 5). the enhanced models, namely longcite-9b and longcite-8b, support a max context window of 128,000 tokens and are capable of generating accurate responses along with precise, fine-grained citations in one pass. evaluation on longbench-cite indicates that our trained models achieve significantly better citation quality compared to even much larger proprietary models. specifically, our 8b/9b size model outperforms gpt-4o by 6.4%/3.6% in terms of citation f1 score and achieves twice finer granularity. meanwhile, we observe that sft with citation information can alleviate hallucinations of llms and enable them to utilize context information more uniformly and comprehensively, instead of only focusing on a specific part of the context. this results in a further improvement in response correctness over standard long-context sft. we also conduct extensive analyses and human evaluation to further verify the effectiveness of our approach. ## related works long-context llms. the current mainstream approach for extending the context window of llms involves continued pre-training of base llms on extensive long texts followed by alignment using diverse long-context qa pairs @xcite . however, because of the difficulty of annotations, most long-context qa data is automatically synthesized by llms themselves @xcite , which cannot strictly guarantee the faithfulness of the answers. this leads to potential hallucinations of the aligned llms, i.e., fabricating content not present in or consistent with the context. therefore, users often require a way to verify the accuracy and reliability of the information provided by llms. our work explores how to enable long-context models to produce responses with fine-grained citations, thereby enhancing the verifiability and trustworthiness of long-context llms. question answering with citations. recently, question answering with citations has been extensively studied in open-domain qa @xcite @xcite , and some works @xcite also explore fine-grained citations for more precise attribution. these methods (either prompting or sft data construction) use retrievers or llms to pre-select relevant context before generating responses and citations. however, such paradigm is not suitable for long-context scenarios since missing any important context information will lead to compromised response quality (appendix d). in addition, @xcite evaluates several prompting approaches for chunklevel citation generation in long-context qa. nevertheless, the rag method and more complex pipelines they tried also suffer from context information loss or excessive latency. our work, however, enables long-context llms to produce accurate responses and precise sentence-level citations in a single generation, providing advantages in terms of response quality, verifiability, and efficiency. moreover, previous methods for citation evaluation only consider limited cases due to rigid nli models they use @xcite . in contrast, we improve the evaluation to fit more complex long-context scenarios and utilize gpt-4o as a judge, thereby achieving a higher agreement with human assessments. ## cof pipeline the core of constructing high-quality sft data for lqac is guaranteeing precise citations while maintaining the correctness of answers. to this end, we propose cof, a post-hoc retrieval-and extractionbased pipeline that utilizes an off-the-shelf llm to obtain sentence-level citations from coarse to fine. as illustrated in figure 2 , cof consists of four steps: (1) given a long context material, cof first employs the llm to generate a query and corresponding answer through self-instruct @xcite . (2) cof then uses sentences in the answer to retrieve roughly k chunks from the context, which are subsequently input into the llm to add coarse-grained chunk-level citations into the answer. (3) next, the llm generates fine-grained sentence-level citations for each statement by extracting supporting sentences from the cited chunks. (4) finally, instances with too few citations are filtered out. in the following, we will introduce each long text material: the water pollution (…) question: what legislation governs wastewater treatment in canada? answer with chunk-level citations: <@xmath0: <@xmath1.<@xmath2: water pollution control legislation (…) to accomplish its water quality objectives the moe produced legislation in the form of the ontario water resources act (owra). this legislation in conjunction with various regulations instruction: your task is to add citations to the existing answer (…) expanded chunk [5]: (…) <@xmath3. <@xmath4. <@xmath5, (…) instruction: your task is to identify the parts in the passage that support some key points of the statement (…) statement: -the ontario water resources act (…) related to water. sentence-level citations: [8-9] demonstrations question & answer demonstrations context: <@xmath6, (…) instruction: please answer the user's question based on the following document. (…) answer with sentence-level citations: <@xmath7: <@xmath8. <@xmath9: what legislation governs wastewater treatment in canada? sentence-level citations: [22-23] self-instruct r e -numbering at least 20% of the statements have citations? retrieve (a) qa instance generation (b) chunk-level citation generation (c) sentence-level citation extraction (d) data filtering lqac instance answer: based on the introduction, the key legislation that governs wastewater treatment in canada is: … -the ontario water resources act (owra): this is provincial legislation that sets out legal requirements for managing environmental issues related to water. (…) step of cof in detail. ## longcite models in this section, we conduct model training experiments to determine whether sft on longcite-45k can enhance llms' ability for lqac, enabling them to generate accurate responses and precise citations within a single output. ## conclusion in this work, we explore enhancing llms' capacity to generate fine-grained citations from lengthy contexts. we first propose longbench-cite, an automatic benchmark to reveal existing llms' limited performance on lqac. we then introduce cof, a novel pipeline that uses an off-the-shelf llm to automatically generate long-context qa instances with precise sentence-level citations, and construct longcite-45k, a large-scale sft dataset for lqac. finally, we successfully train longcite-8b and longcite-9b, allowing the generation of accurate responses and fine-grained citations in one pass. extensive analyses and human evaluation further verify the effectiveness of our approach. we believe that this work lays a solid foundation for further research on lqac and contributes to the development of more reliable and trustworthy llms.