Where Did That Come From? Sentence-Level Error-Tolerant Attribution (2025)

Abstract: Attribution is the process of identifying which parts of the source support a generated output. While attribution can help users verify content and assess faithfulness, existing task definitions typically exclude unsupported or hallucinated content leaving them unattributed, overlooking the potential to increase faithfulness certainty, locate the error, and fix it easier.In this paper, we propose a new definition for sentence-level error-tolerant attribution, which extends attribution to include incorrect or hallucinated content. We introduce a benchmark for this task and evaluate a range of models on it. Our results show that sentence-level error-tolerant attribution improves the quality of both automatic and manual faithfulness evaluations, reducing annotation time by 30% in long-document settings, and facilitates hallucination fixing. We also find that unfaithful outputs are often linked to sentences that appear later in the source or contain non-literal language, pointing to promising avenues for hallucination mitigation. Our approach offers a better user experience along with improved faithfulness evaluation, with better understanding of model behavior.

## introduction text generation systems are increasingly deployed to produce summaries, answers, and explanations grounded in source documents. a central concern in these applications is faithfulness-whether the generated content accurately reflects the input. unfaithful generations, or hallucinations, can mislead users, damage trust, and propagate misinformation. to address this, recent work has proposed the task of attribution: @xcite @xcite identifying which parts of the source support a given generation. attribution 1 annotated and predicted attribution data is publicly available at https://github.com/oriern/errortolerantattribution . 300,000 40,000 30,000 30,000 can improve trust, let the user expend their knowledge in a certain point, or provide a foundation for verifying outputs. however, existing approaches to attribution face two core limitations. first, most attribution benchmarks operate at the document level @xcite ), making it difficult and time-consuming to locate the specific source span that is relevant to the output. while some recent work has explored finer-grained attribution at the span level @xcite , these efforts have largely been limited to the attribute-then-generate paradigm, where relevant source spans are selected prior to generation and used to guide the output. in such setups, attribution is effectively given rather than inferred. these methods do not address the more challenging case where attribution must be extracted retrospectively from existing outputs. second, prior work overwhelmingly treats attribution primarily as a form of grounding: if content is not supported by the source, it is simply left unattributed. this framing limits the practical utility of attribution. unattributed information can leave users uncertain whether a sentence is entirely fabricated, partially correct, or accurate but misattributed. in contrast, attributing incorrect information can help pinpoint the error, separate accurate content from inaccuracies, and facilitate correction. such diagnostic benefits are not possible under the current definition of attribution. to address these two challenges, we propose two key innovations: sentence-level attribution in a post-hoc setting, and error-tolerant attribution. this sentence-level granularity enables immediate localization of relevant source spans, eliminating the need to read the entire document to verify a single sentence. building on this, error-tolerant attribution extends attribution to cover even incorrect or hallucinated content-whether it contradicts the source, loosely resembles it, or refers to information that arguably should have been included. this richer attribution not only identifies precise source spans (or confirms their absence), but might also offer novel insights into model errors and how specific elements in the source may have contributed to them. an example for our new attribution approach is presented in figure 1 . our approach provides several benefits across different use cases. first, it serves as a valuable tool for end users on its own. by making fine grained error-tolerant attribution available, it increases the trustworthiness of the output and enables users to verify or expand their understanding without needing to read the entire source document. moreover, it improves clarity in cases of incorrect output. users can easily identify and correct errors, or to distinguish between accurate and inaccurate components within a sentence. our approach also offers advantages as an auxiliary tool for faithfulness evaluation, whether manual or automatic. by substantially narrowing the scope of information that needs to be considered, it makes it more feasible to annotate or assess long documents with greater consistency and reduced effort. finally, this form of attribution serves as a foundation for deeper analysis. it enables tracing the origin of both accurate and hallucinated content, characterizing features that are uniquely associated with hallucinated attributions, and gaining a better nuanced understanding of system behavior. in this paper, we formalize the task of sentence-level error-tolerant attribution (section 3) and construct a benchmark to evaluate the performance of various models on it (sections 4 & 5). we demonstrate its potential utility for automatic faithfulness evaluation, particularly in challenging scenarios involving long input contexts (section 6.1). additionally, we show that our approach reduces manual evaluation time by 30% (section 6.2). we also show the benefit of using these attributions to fix the output (section 7). beyond evaluation and fixing, we analyze the source sentences linked to unfaithful outputs and find that they are more likely to occur toward the end of the document and to contain complex, non-literal expressions (section 8). these findings highlight promising directions for future work on hallucination mitigation. overall, our results show that attribution-when extended in this way, becomes a powerful lens for evaluating and improving the faithfulness of text generation systems. ## task definition in this section, we define the new sentencelevel error-tolerant attribution task. let @xmath0, s 2 , . . . , s n } denote the set of source sentences from a document or collection of documents, and let g be a generated sentence produced by a system grounded in d. the goal is to identify a minimal subset a ⊆ d that maximizes the information relevant to assessing the faithfulness of g to the source. the selected subset a must be interpretable in context to resolve any coreferences. each attribution set a may fall into one or more of the following categories (an example for each category can be found in contradiction. sentences that explicitly contradict information in g, suggesting unfaithfulness. near match. sentences that closely resemble g with small, non-contradictory changes. expected span. sentences (or regions of the document) where the information in g would reasonably be expected to appear if it were grounded in the source. the absence of relevant content in such expected spans may imply unfaithfulness. none. no sentence in d is relevant (or all are equally relevant). in the ideal case of perfect attribution, the selected set a * should be sufficient to annotate the faithfulness of g. more precisely, once a * is known, the remainder of the document provides no additional information about g in the context of assessing its truthfulness or provenance. formally, this implies ## data annotation in order to evaluate baselines for this task and present the potential of optimal attributions in different scenarios, we annotate manually a development and test attribution sets. ## experiments to establish the performance of current large language models (llms) on the task of post-hoc finegrained attribution, we evaluate several state-of-theart models. the objective is to assess their capability in identifying precise spans within source documents that support or relate to specific segments of a given generated text, as defined in section 3 and annotated according to section 4. we experiment with zero-shot setting the following llms: gemini 2.0 flash, gpt-4o @xcite , qwen 2.5 72b instruct @xcite and llama 3 70b instruct @xcite . in addition, we examined a trained non-llm nli-based baseline, nli-deberta-v3-base. for each summary sentence, we computed the nli score with each source sentence and selected the two sentences with the highest scores (entailment or contradiction). performance is measured using sentence-level macro-averaged precision (p), recall (r), and f1-score (f1). these metrics compare the model's predicted attribution spans against the humanannotated gold standard from our test set. ## faithfulness evaluation beyond the benefits of error-tolerant attribution as a standalone tool, we also aim to demonstrate its value as an auxiliary task across several scenarios. in this section, we show how it can support and enhance both automatic and manual faithfulness evaluation. in section 7, we illustrate how it can aid in correcting hallucinations. finally, in section 8, we show that our annotated dataset reveals features that may help identify, in advance, text spans that are more likely to produce hallucinations. ## fixing unfaithful text in this section, we explore another potential application of error-tolerant attribution: correcting unfaithful text. while post-editing techniques aim to resolve inconsistencies in generated content @xcite @xcite , they remain challenging due to the diffi- given an unfaithful summary sentence and its source, the goal is to minimally revise the sentence so that it becomes faithful to the source. we evaluate three input configurations: (1) the full source without attribution information ('plain'), ( 2 ) the source with highlighted attribution spans ('highlighted'), and (3) only the attribution spans without additional context ('attr. only'). we generated corrections of 20 incorrect summary sentences, using three models-llama 3.1-8b-instruct @xcite , mistral-8b-instruct @xcite , and claude-3.5-haiku. the prompts are presented in appendix d. an expert annotator then assessed each corrected sentence with two judgments: whether the sentence was fixed, partially fixed, or not fixed, and its relative ranking across the three input settings for each model. 6 as shown in ## analysis of hallucination factors the centrality of the hallucination problem in text generation raises a fundamental question: what causes models to hallucinate? our manually annotated dataset offers fertile ground for such an investigation. by analyzing source-based features that may contribute to unfaithful outputs, such as ambiguity or complexity, we can identify characteristics that predispose certain input content to hallucinations. this opens up the possibility of pre-editing sources (e.g., simplifying complex segments) to reduce hallucination risk, or at least to anticipate it. prior work has explored various contributing factors, including the influence of the prompt @xcite , the training process @xcite , and the training data @xcite . however, to the best of our knowledge, no study has examined the source of hallucinations in relation to the input text itself at inference time. our approach aligns with the goals of @xcite , who focused on identifying problematic sources that tend to yield less informative summaries. to conduct our analysis, we used the source sentences that are manually linked to summary sentences in our dataset. these sentences are implicitly selected by the model for generating a summary. we divided them into two groups: those linked to at least one unfaithful summary sentence, denoted as 'unfaithful-linked sentences', and those linked only to faithful ones, denoted as 'faithful-linked sentences'. we then examined several features to identify signals that could distinguish between the two groups, and found two particularly informa-tive ones: sentence position and the presence of non-literal expressions. sentence position. we computed the relative position of each source sentence in its document and plotted the distributions (figure 2 ). as expected, and consistent with prior findings in summarization research @xcite , faithful-linked source sentences are more likely to appear at the beginning of the document, with their frequency gradually decreasing as the document progresses. interestingly, unfaithful-linked sentences exhibit a different distribution: aside from a slight peak in the first 10% of the document, their occurrence is more evenly spread across positions. notably, if the information in a summary sentence is taken from the last 10% of document sentences, this summary sentence is nearly twice as likely to be unfaithful as faithful. this suggests that models may struggle to accurately process or incorporate information from later parts of the input. non-literal language. we also examined the prevalence of non-literal expressions (e.g., idioms, irony), which require additional interpretation or external knowledge. a manual review of 10 development-set documents (covering 30 summaries and 84 summary sentences) revealed that 25% of the sentence sets linked to unfaithful summary sentences contained at least one non-literal expression-compared to only 9% among those linked to faithful sentences. viewed from another perspective, 88% of the non-literal expressions that are linked to the summary, linked to unfaithful sentences. these findings suggest that non-literal language, which is inherently harder to interpret, increases the likelihood of unfaithful generation. examples can be found in appendix e. in summary, our analysis underscores the value of investigating hallucination through the lens of source-based features. while our analysis is exploratory, it highlights promising directions for future work aiming to discover and leverage additional features to combat hallucination. ## conclusion we introduced a fine-grained, error-tolerant approach for attribution. this approach operates posthoc at the sentence level, allowing precise localization of the source origins of both faithful and unfaithful summary sentences. our benchmark demonstrates the utility of this framework for faith-fulness evaluation, significantly reducing annotation effort and providing deeper insight into model behavior. by extending attribution beyond faithful outputs, we show its potential as both a practical tool for users and models, and a diagnostic signal for improving text generation systems.