Document Attribution: Examining Citation Relationships using Large Language Models (2025)

Abstract: As Large Language Models (LLMs) are increasingly applied to document-based tasks - such as document summarization, question answering, and information extraction - where user requirements focus on retrieving information from provided documents rather than relying on the model’s parametric knowledge, ensuring the trustworthiness and interpretability of these systems has become a critical concern. A central approach to addressing this challenge is attribution , which involves tracing the generated outputs back to their source documents. However, since LLMs can produce inaccurate or imprecise responses, it is crucial to assess the reliability of these citations.To tackle this, our work proposes two techniques. (1) A zero-shot approach that frames attribution as a straightforward textual entailment task. Our method using flan-ul2 demonstrates an improvement of 0.27% and 2.4% over the best baseline of ID and OOD sets of AttributionBench (CITATION), respectively. (2) We also explore the role of the attention mechanism in enhancing the attribution process. Using a smaller LLM, flan-t5-small, the F1 scores outperform the baseline across almost all layers except layer 4 and layers 8 through 11.

## introduction attribution in large language models refers to tracing the origins of information embedded in the model's outputs. this involves identifying the specific datasets, documents, or text segments contributing to the generated response. attribution is essential for verifying the information's provenance, ensuring the generated content's accuracy and reliability, and addressing concerns regarding plagiarism, accountability, and transparency in ai systems. attribution methods typically involve mapping responses to the relevant data sources that influenced the model's generation. in llms, attribution systematically links the model's outputs to their source materials, facilitating the identification of the exact documents, datasets, or references that informed the generated response. the primary goal is to uphold transparency, validate factual correctness, and give proper credit to sources. this process is critical for maintaining the credibility and accountability of generative ai systems. attribution methods are fundamental for enhancing the interpretability and dependability of llms. they support the model's output by providing citations or references, improving accuracy, and reducing the risk of misinformation. this ensures that each response is substantiated by relevant evidence, forming a basis for assessing the sufficiency and relevance of the underlying data. research on llm attribution methodologies encompasses citation generation, claim verification, and hallucination detection techniques. these strategies are aimed at improving the quality and reliability of llm-generated content. however, challenges remain in implementing adequate attribution, including the need for robust validation mechanisms, managing cases where sources influence the model's reasoning indirectly, handling structured data or non-textual sources (e.g., tables, figures, or images), and addressing the complexities of multi-lingual or cross-lingual data. overcoming these challenges is essential for successfully integrating attribution methods within llms. as ai and machine learning systems become increasingly prevalent, the demand for accountability, transparency, and reliability intensifies. attribution techniques are pivotal in achieving these objectives, positioning them as a key area of research and development to advance ai technologies and ensure their responsible deployment. the main contributions of this work are: ## related work attribution in llms has become a vital research area focused on tracing content origins and ensuring accuracy and accountability. key studies have introduced various techniques and addressed challenges in this field. @xcite propose a minimalsupervision method for eliciting attributions, improving scalability, and reducing the need for extensive human input. an interactive visual tool for attribution is introduced @xcite , aiming to enhance transparency by making attributions more accessible to non-technical users. @xcite explore attribution in low-resource settings, emphasizing its potential to explain model behavior when data and resources are limited. the captum interpretability library is used in @xcite for generative llms, offering insights into the factors influencing model predictions. @xcite argue that source-aware training enhances attribution by linking knowledge to specific sources, improving content reliability. the issue of false attribution, stressing the need for more accurate methodologies, is highlighted in @xcite . @xcite focus on attribution in question-answering systems, proposing methods for evaluating and modeling attributions in qa contexts. a survey of llm attribution research, summarizing key techniques, challenges, and developments, is provided in @xcite . lastly, @xcite explores the automated evaluation of attribution, aiming to streamline validation processes in practical applications. ## method and experimental setup the attribution task defined in attributionbench @xcite is framed as a binary classification problem, where the objective is to determine whether a given claim is attributable to its associated references. the work in attributionbench explores this problem using both zero-shot inference and fine-tuning of llms. similarly, our formulation adopts the same approach to the problem. however, we restrict our methodology to zero-shot experiments due to computational limitations. additionally, we also investigate if attention layers could help improve the attribution. ## results and analysis in the initial phase of our evaluation of the attribution task, we conduct zero-shot experiments. the framework presented in attributionbench is divided into two key components: in-distribution (id) and out-of-distribution (ood) sampling of the dataset. in their experimental setup, attribu-tionbench employs f1 score, false positive (fp), and false negative (fn) rates as evaluation metrics. consistent with their methodology, we adopt the same metrics -f1, fp, and fn -for the evaluation in this study. ## conclusion and future work in this paper, we conducted zero-shot experiments on attributionbench to assess the performance of textual entailment-based approaches for attribution tasks. our findings show that even without finetuning, a simple zero-shot textual entailment approach outperforms the existing baseline in both in-distribution and out-of-distribution settings. notably, flan-ul2 demonstrated strong performance across these scenarios, underscoring its robustness and suitability for such tasks. we also preliminary analyzed attention layer behavior using the smaller flan-t5-small model. the results suggest that attention mechanisms could provide valuable insights for improving attribution performance. we plan to overcome computational limitations for future work by conducting fine-tuning experiments. we aim to use more advanced llms to perform a deeper analysis of attention layers. this could provide further actionable insights to refine performance and yield more robust findings. limitation 2: regarding exploring attention mechanisms to enhance the performance of the attribution task, we were similarly restricted by computational limitations. consequently, we could not utilize computationally demanding models for this analysis. instead, the experiments were conducted using a lightweight model, flan-t5-small.