Reranking-based Generation for Unbiased Perspective Summarization (2025)

Abstract: Generating unbiased summaries in real-world settings such as political perspective summarization remains a crucial application of Large Language Models (LLMs). Yet, existing evaluation frameworks rely on traditional metrics for measuring key attributes such as coverage and faithfulness without verifying their applicability, and efforts to develop improved summarizers are still nascent. We address these gaps by (1) identifying reliable metrics for measuring perspective summary quality, and (2) investigating the efficacy of LLM-based methods beyond zero-shot inference. Namely, we build a test set for benchmarking metric reliability using human annotations and show that traditional metrics underperform compared to language model–based metrics, which prove to be strong evaluators. Using these metrics, we show that reranking-based methods yield strong results, and preference tuning with synthetically generated and reranking-labeled data further boosts performance. Our findings aim to contribute to the reliable evaluation and development of perspective summarization methods.

## introduction article summarization is a key application of large language models (llms) given their recent breakthroughs in text generation capabilities @xcite . critically, however, llms often exhibit undesirable behaviors and input-level biases toward spurious features (e.g., position) @xcite @xcite , resulting in unbalanced input coverage @xcite and hallucination @xcite . these issues are especially problematic in opinionated article summarization @xcite , where unbiased representation of diverse viewpoints is crucial. recent studies in opinion summarization address these risks by developing tasks and methods that generate summaries free of framing bias @xcite , fairly represent input diversity @xcite , or preserve the source perspectives @xcite . within this domain, perspective summarization @xcite serves as a representative evaluation setting, where models are tasked to generate precise, perspective-specific summaries from multi-document inputs containing diverse political views. however, two gaps remain unaddressed in this setting: (1) existing evaluation metrics are primarily derived from news summarization domains and have not been validated for measuring perspective summary quality, and (2) the effectiveness of llm-based methods beyond zeroshot inference in generating unbiased, high-quality perspective summaries remains underexplored. to address these gaps, we first identify effective metrics for measuring summary quality by constructing a test set to evaluate existing metrics. we focus on two key attributes that a desirable summary should have: perspective coverage-the extent to which the summary includes all key content from the intended perspective, and perspective faithfulness-the degree to which the summary excludes content unsupported by the source articles of the target perspective. we collect key point annotations from articles to create controlled summaries with varied key point selections and assigned ground truth scores. we find that language model-based metrics such as alignscore @xcite and prompting-based scoring @xcite serve as strong evaluators, whereas traditional metrics (rouge @xcite , bertscore @xcite ) underperform. following this, we evaluate methods for generating perspective summaries with improved coverage and faithfulness beyond zero-shot inference. we benchmark prompting frameworks, mechanistic methods for mitigating input biases, and rerankingbased methods that select the best candidate based on proxy metrics. using both human and automatic evaluations, we show that reranking outperforms zero-shot inference and prompting-based methods, while prompting only yields marginal improvements over zero-shot inference. notably, preference tuning with direct preference optimization (dpo) @xcite on reranked generations further boosts performance on both attributes and particularly improving faithfulness. our results suggest that current llms can generate highquality perspective summaries with strong coverage and faithfulness, and that preference-based training can further boost performance. in summary, our contributions are as follows: perspective-conditioned summarization. existing research on opinion summarization and related tasks has primarily focused on domains such as product reviews @xcite , while recent work has broadened to a range of tasks on opinionated texts. most single-document methods aim to preserve authorial intent @xcite or polarity @xcite , whereas multidocument summarization must integrate varied perspectives. for instance, @xcite generates politically neutral summaries from sets of left-, right-, and center-leaning news articles. other approaches aim to fairly represent diverse perspectives in reviews @xcite , controllably represent community perspectives @xcite , generate consensus summaries @xcite , or produce multiple summaries reflecting distinct political perspectives @xcite . in line with these works, we summarize the political perspective among a set of input passages while addressing the coverage and faithfulness issues observed in existing models as highlighted in these studies. ## measuring summary quality in perspective summarization, the summarizer is given two perspectives θ 1 , θ 2 , each with a source article d t,θ , θ ∈ {θ 1 , θ 2 }, comprising a set of documents d t,θ = {d (i) t,θ | @xmath0, that present opinions on topic t. we study the setting where the summarizer is tasked to generate a summary that encapsulates all key points directly supporting a specified perspective's stance. concretely, a highquality perspective summary should: (1) include all key points from each relevant document, and (2) avoid including any content unsupported by or in opposition to the perspective's documents. we formalize these properties as follows: