On Positional Bias of Faithfulness for Long-form Summarization (2025)

Abstract: Large Language Models (LLMs) often exhibit positional bias in long-context settings, under-attending to information in the middle of inputs. We investigate the presence of this bias in long-form summarization, its impact on faithfulness, and various techniques to mitigate this bias. To consistently evaluate faithfulness, we first compile a benchmark of eight human-annotated long-form summarization datasets and perform a meta-evaluation of faithfulness metrics. We show that LLM-based faithfulness metrics, though effective with full-context inputs, remain sensitive to document order, indicating positional bias. Analyzing LLM-generated summaries across six datasets, we find a “U-shaped” trend in faithfulness, where LLMs faithfully summarize the beginning and end of documents but neglect middle content. Perturbing document order similarly reveals models are less faithful when important documents are placed in the middle of the input. We find that this behavior is partly due to shifting focus with context length: as context increases, summaries become less faithful, but beyond a certain length, faithfulness improves as the model focuses on the end. Finally, we experiment with different generation techniques to reduce positional bias and find that prompting techniques effectively direct model attention to specific positions, whereas more sophisticated approaches offer limited improvements. Our data and code will be publicly available.

## introduction large language models (llms) have enabled highquality summary generation. however, the use of llms for long-context scenarios, where either the source document(s) or the generated summary is very long, still remains challenging @xcite ., * work done during an internship at salesforce ai research. 1st 2nd 3rd 4th 5th document index 75 80 85 90 95 faithfulness (%) arxiv gpt-4o pubmed llama-70b figure 1: positional bias in long-form summarization: on two representative models and datasets, summaries are less faithful to the documents in the middle. for our analysis of faithfulness in long-form document summarization, we first need to determine the best faithfulness metrics, as very few studies have performed an extensive study. specifically, we want to verify whether current automatic metrics can evaluate summaries well given the large context. and if the metric needs to break the input context into chunks, what is the best way to merge the faithfulness scores. to do so, we collate a large, unified benchmark longformfact for evaluating the performance of metrics on long-form summarization across 8 human-annotated benchmarks. we evaluate llm-based metrics and find that the models handle full context well, and for the splitting case, taking the maximum over the source documents and taking the average across the summary sentences yield the highest correlation with human judgments. we further perform a perturbation experiment for the full-context setting, where we sort the documents according to their similarity with the summary. an ideal metric should not be affected by the order of the documents, but we find that the metrics are sensitive towards order perturbation and thus suffer from positional bias. after determining the faithfulness metrics for long-form summarization, we use it to perform an extensive analysis of the faithfulness of the generated summaries across the input documents. across 6 datasets, we generate summaries and plot the faithfulness scores to verify whether the model hallucinates more for the documents in the middle. similar to @xcite that analyzes context utilization for summarization, we observe a u-shape and lead bias when analyzing the faithfulness of the generated summaries. next, we perform a similar order perturbation experiment and find that llms are sensitive to the order, summariz-ing documents more faithfully for the documents at the beginning. lastly, we analyze how length correlates with faithfulness by measuring how faithfulness changes as the input length increases. we find that models gradually introduce more hallucinations as we introduce more documents, and after a certain threshold, the model becomes more faithful as it attends to the documents at the end. finally, we investigate methods that attempt to mitigate positional bias. we explore methods such as prompting to focus on certain parts, hierarchical merging, incremental updating, and calibration methods. we find that prompting methods are moderately effective at improving summary's faithfulness towards certain positions, while more sophisticated methods struggle to address this issue. ## faithfulness metrics meta-evaluation given the limited studies in determining the best automatic faithfulness metric for long-form summarization, we first aim to comprehensively test the best strategy for applying current evaluation methods in the long-form context. ## faithfulness of long-form summaries next, we evaluate the faithfulness of summaries generated from different datasets, and perform detailed faithfulness analysis, including assessing faithfulness across each document, performing a perturbed analysis in which we sort documents by importance, and understanding how faithfulness changes as the number of documents increases. ## methods for reducing positional bias finally, we explore different generation methods and verify whether they can reduce positional bias. we use the same sets of examples used in section 4. ## conclusion in this work, we present an extensive analysis of the relationship between positional bias and faithfulness for long-context summarization from three perspectives. we first evaluate the best strategy for assessing faithfulness in long-context summarization tasks, as well as the metrics' sensitivity to positional changes. we find that, although current llm-based metrics achieve the highest correlation when using the full context, they are sensitive to changes in the order of the input documents. we then analyze faithfulness of model generations with the best faithfulness metrics. we generate summaries using both open-source and proprietary models and find that the faithfulness of the middle documents tends to dip compared to those at the beginning and end, and the summaries also exhibit high sensitivity when the order of inputs is perturbed. one of the possible explanations, as we find, is that the models change behavior after a certain context length and focus on documents toward the end, improving faithfulness for the documents at the back after the initial dip. finally, we investigate several generation methods to test whether they can alleviate positional bias. we find that prompting methods can partially alleviate the middle curse, while more extensive methods provide overall less faithful summaries.