N eu S : Neutral Multi-News Summarization for Mitigating Framing Bias (2022)

Abstract: Media news framing bias can increase political polarization and undermine civil society. The need for automatic mitigation methods is therefore growing. We propose a new task, a neutral summary generation from multiple news articles of the varying political leaningsto facilitate balanced and unbiased news reading. In this paper, we first collect a new dataset, illustrate insights about framing bias through a case study, and propose a new effective metric and model (NeuS-Title) for the task. Based on our discovery that title provides a good signal for framing bias, we present NeuS-Title that learns to neutralize news content in hierarchical order from title to article. Our hierarchical multi-task learning is achieved by formatting our hierarchical data pair (title, article) sequentially with identifier-tokens (“TITLE=>”, “ARTICLE=>”) and fine-tuning the auto-regressive decoder with the standard negative log-likelihood objective. We then analyze and point out the remaining challenges and future directions. One of the most interesting observations is that neural NLG models can hallucinate not only factually inaccurate or unverifiable content but also politically biased content.

## introduction media framing bias occurs when journalists make skewed decisions regarding which events or information to cover (information bias) and how to cover them (lexical bias) @xcite . even if the reporting of the news is based on the same set of underlying issues or facts, the framing of that issue can convey a radically different impression of what actually happened @xcite . since the news media plays a crucial role in shaping public opinion toward various important issues @xcite mccombs and reynolds, 2009; perse and lambe, 2016), bias in media reporting can reinforce the problem of political polarization and undermining civil society rights. allsides.com @xcite mitigates this problem by displaying articles from various media in a single interface along with an expert-written roundup of news articles. this roundup is a neutral summary for readers to grasp a bias-free understanding of an issue before reading individual articles. although allsides fights framing bias, scalability still remains a bottleneck due to the time-consuming human labor needed for composing the roundup. multi-document summarization (mds) models @xcite could be one possible choice for automating the roundup generation as both multi-document summaries and roundups share a similar nature in extracting salient information out of multiple input articles. yet the ability of mds models to provide neutral description of a topic issue -a crucial aspect of the roundup -remains unexplored. in this work, we fill in this research gap by proposing a task of neutral multi-news summarization (neus), which aims to generate a framing-bias-free summary from news articles with varying degrees and orientation of political bias (fig. 1 ). to begin with, we construct a new dataset by crawling allsides.com, and investigate how framing bias manifests in the news so as to provide a more profound and more comprehensive analysis of the problem. the first important insight from our analysis is a close association between framing bias and the polarity of the text. grounded on this basis, we propose a polarity-based framingbias metric that is simple yet effective in terms of alignment with human perceptions. the second insight is that titles serve as a good indicator of framing bias. thus, we propose neus models that leverage news titles as an additional signal to increase awareness of framing bias. our experimental results provide rich ideas for understanding the problem of mitigating framing bias. primarily, we explore whether existing summarization models can already solve the problem and empirically demonstrate their shortcomings in addressing the stylistic aspect of framing bias. after that, we investigate and discover an interesting relationship between framing bias and hallucination, an important safety-related problem in generation tasks. we empirically show that the hallucinatory generation has the risk of being not only factually inaccurate and/or unverifiable but also politically biased and controversial. to the best of our knowledge, this aspect of hallucination has not been previously discussed. we thus hope to encourage more attention toward hallucinatory framing bias to prevent automatic generations from fueling political bias and polarization. we conclude by discussing the remaining challenges to provide insights for future work. we hope our work with the proposed neus task serves is a good starting point to promote the automatic mitigation of media framing bias. ## analysis of framing bias the literature on media framing bias from the nlp community and social science studies provide the definition and types of framing bias @xcite @xcite -informational framing bias is the biased selection of information (tangential or speculative information) to sway the minds of readers. lexical framing bias is a sensational writing style or linguistic attributes that may mislead readers. however, the definition is not enough to understand exactly how framing bias manifests in real examples such as, in our case, the allsides dataset. we conduct a case-study to obtain concrete insights to guide our design choices for defining the metrics and methodology. ## metric we use three metrics to evaluate summaries from different dimensions. for framing bias, we propose a polarity-based metric based on the careful design choices detailed in §5.1. for evaluating whether the summaries retain salient information, we adopt commonly used information recall metrics ( §5.2). in addition, we use a hallucination metric to evaluate if the generations contain any unfaithful hallu-cinatory information because the existence of such hallucinatory generations can make the summary fake news ( §5.3). ## results and analysis in this section, we point out noteworthy observations from the quantitative results in ## conclusion we introduce a new task of neutral multi-news summarization (neus) to mitigate media framing bias by providing a neutral summary of articles, along with the dataset allsides and a set of metrics. throughout the work, we share insights to understand the challenges and future directions in the task. we show the relationships among polarity, extra information, and framing bias, which guides us to the metric design, while the insight that the title serves as an indicator of framing bias leads us to the model design. our qualitative analysis reveals that hallucinatory content generated by models may also contribute to framing bias. we hope our work stimulates researchers to actively tackle political framing bias in both human-written and machine-generated texts.