P ^3 S um: Preserving Author’s Perspective in News Summarization with Diffusion Language Models (2024)

Abstract: In this work, we take a first step towards designing summarization systems that are faithful to the author’s intent, not only the semantic content of the article. Focusing on a case study of preserving political perspectives in news summarization, we find that existing approaches alter the political opinions and stances of news articles in more than 50% of summaries, misrepresenting the intent and perspectives of the news authors. We thus propose P ^3 Sum, a diffusion model-based summarization approach controlled by political perspective classifiers. In P ^3 Sum, the political leaning of a generated summary is iteratively evaluated at each decoding step, and any drift from the article’s original stance incurs a loss back-propagated to the embedding layers, steering the political stance of the summary at inference time. Extensive experiments on three news summarization datasets demonstrate that P ^3 Sum outperforms state-of-the-art summarization systems and large language models by up to 13.7% in terms of the success rate of stance preservation, with competitive performance on standard metrics of summarization quality. Our findings present a first analysis of preservation of pragmatic features in summarization, highlight the lacunae in existing summarization models—that even state-of-the-art models often struggle to preserve author’s intents—and develop new summarization systems that are more faithful to author’s perspectives.

## introduction what constitutes a faithful summary? in addition to preserving factual consistency-the focus of much prior work @xcite @xcite @xcite -a good summarization system should preserve the writer's voice-the style, intent, and points of view conveyed by the authors. however, such subtle pragmatic cues are harder to extract and control for by existing models @xcite , and it remains underexplored whether existing summarization systems generate summaries that are faithful to the opinions and perspectives of the authors. moreover, though language models (lms) have been widely applied to many summarization tasks, they inevitably contain political biases and such biases could further impact downstream tasks @xcite . so we hypothesize that summarization systems built on top of llms would propagate biases further, but not necessarily align them with stances in the source text. specifically in the task of summarization, instead of "de-biasing" and generating only neutral summaries, we argue that a good summarization system should preserve the perspectives of the authors in generated news summaries. to this end, we first evaluate to what extent summarization systems and llms preserve political stances in generated summaries, by employing a state-of-the-art political perspective evaluator @xcite to quantify the gap between stances in news articles and summaries. ( §2) we identify that existing summarization systems and llms do alter opinions and perspectives in the original document, resulting in shifting stances in more than 50% of summaries, with around 25% drifting to the partisan extremes (figure 1 ). this highlights a new, underexplored concern with current llms as they fail to preserve the intents and perspectives of the authors of news documents during summarization, potentially misinforming the readers. to address this issue, we propose p 3 sum, a summarization model aiming to preserve the political perspectives of news articles. ( §3) p 3 sum employs a non-autoregressive diffusion language model with modular control capabilities to steer the generated summary towards the same perspective of the news article. specifically, we first fine-tune 2154 no change indicates that there is no difference in the political leaning of the summary and the context. our study shows that existing approaches alter the stances of news articles in more than 50% of cases across both datasets. a diffusion language model @xcite on summarization data. during inference, the generated summary is evaluated by a political stance classifier @xcite at each step, compared to the target stance in the source document while summary generation is steered towards the target stance. our primary motivation to use diffusion models is that they allow us to (1) apply the stance classifier on the whole summary at each decoding step, rather than on a prefix generated autoregressively @xcite , and (2) seamlessly incorporate various pretrained classifiers without adaptation, to carefully steer generation process. thus, as an inference-time approach based on diffusion models and controllable text generation @xcite @xcite @xcite @xcite , p foot_1 sum alleviates the need for additional training or pretraining, handles news articles from different ideological stances, and is compatible with future classifiers of author perspectives. extensive experiments on three news datasets demonstrate that p 3 sum greatly outperforms baselines in preserving the political stances of news articles while maintaining good summarization utility. specifically, p 3 sum is at least 13.7%, 2.9%, and 1.6% better in perspective preservation on cnn/dm @xcite , xsum @xcite , and politics @xcite , outperforming popular summarization systems @xcite ; zhang change cnn/dm xsum left 20.6 5.0 lean left 13.2 3.8 no change 43.0 39.2 lean right 15.8 14.2 right 7.4 37.8 ## examining perspective preservation given a news article, the generated summary should preserve the authors' political perspectives in the document. however, existing models are not designed to control for author intent or perspectives, and we first investigate to which extent summarization systems and large language models alter the perspectives in the generated summaries. to this end, we measure the political leaning of the generated summaries and compare them to the political stances of original articles, using 500 randomly chosen news articles from the cnn/dm @xcite and politics @xcite ) datasets foot_0 . we use a political perspective evaluator @xcite to quantify political stances of summaries and news articles (mapping text sequences to left, center, or right), investigating the change in political leanings with six summarization models and llms: gpt-3.5 (text-davinci-003), chatgpt (gpt-3.5-turbo), pe-gasus @xcite , bart @xcite , brio @xcite , and t5 @xcite . we then determine the perspective gap between the summary and the news article. as shown in figure 1 3 , current summarization systems struggle to provide faithful summaries and significantly alter political perspectives. concretely, the political stance of the generated summary is different from the news article in more than 50% of cases across different models, while around 25% drift to partisan extremes. besides, we also examine the political perspective of reference summaries provided in wellestablished summarization datasets, namely cn-n/dm and xsum in as a result, how to develop summarization approaches that are faithful to the authors' perspectives in the news document remains an open research question. ## p sum we propose p 3 sum, a diffusion model that steers the political stance of the generation towards the news article at inference time with an off-the-shelf classifier. given a news article d, p 3 sum aims to generate a summary s that preserves the original political stance of the article. we first finetune a diffusion-based language model on summarization datasets. at decoding time, we employ a political stance classifier to steer the generated summary by incorporating the gradient from the classifier, ensuring that the political stance of the generation is consistent with the original article. ## conclusion we demonstrate that existing summarization systems and llms struggle to preserve the authors' political perspectives in news summarization. we present p 3 sum, a diffusion-based summarization model that improves political perspective preservation by iteratively guiding the decoding process with an external political stance classifier. extensive experiments demonstrate that p 3 sum outperforms large language models and summarization systems in producing summaries faithful to the political stances of news documents while maintaining competitive summarization utility.