Crossing the Aisle: Unveiling Partisan and Counter-Partisan Events in News Reporting (2023)

Abstract: News media is expected to uphold unbiased reporting. Yet they may still affect public opinion by selectively including or omitting events that support or contradict their ideological positions. Prior work in NLP has only studied media bias via linguistic style and word usage. In this paper, we study to which degree media balances news reporting and affects consumers through event inclusion or omission. We first introduce the task of detecting both partisan and counter-partisan events: events that support or oppose the author’s political ideology. To conduct our study, we annotate a high-quality dataset, PAC, containing 8,511 (counter-)partisan event annotations in 304 news articles from ideologically diverse media outlets. We benchmark PAC to highlight the challenges of this task. Our findings highlight both the ways in which the news subtly shapes opinion and the need for large language models that better understand events within a broader context. Our dataset can be found at https://github.com/launchnlp/Partisan-Event-Dataset.

## introduction political opinion and behavior are significantly affected by the news that individuals consume. there is now extensive literature examining how journalists and media outlets promote their ideologies via moral or political language, tone, or issue framing @xcite @xcite . however, in addition to this more overt and superficial presentation bias, even neutrally written, broadlyframed news reporting, which appears both "objective" and relatively moderate, may shape public opinion through a more invisible process of selection bias, where factual elements that are included or omitted themselves have ideological effects @xcite ; d'alessio and interestingly, both media outlets include quotes from greg abbott but for different purposes: one for supporting the bill and the other for balanced reporting. @xcite . existing work in nlp has only studied bias at the token-or sentence-level, particularly examining how language is phrased @xcite @xcite @xcite . this type of bias does not rely on the context outside of any individual sentence, and can be altered simply by using different words and sentence structures. only a few studies have focused on bias that depends on broader contexts within a news article @xcite or across articles on the same newsworthy event @xcite . however, these studies are limited to token-or span-level bias, which is less structured, and fail to consider the more complex interactions among news entities. to understand more complex content selection and organization within news articles, we scrutinize how media outlets include and organize the fundamental unit of news-events-to subtly reflect their ideology while maintaining a seemingly balanced reporting. events are the foundational unit in the storytelling process @xcite , and the way they are selected and arranged affects how the audience perceives the news story @xcite . inspired by previous work on selection bias and presentation bias @xcite , we study two types of events. (i) partisan events, which we define as events that are purposefully included to advance the media outlet's ideological allies' interests or suppress the beliefs of its ideological enemies. (ii) counter-partisan events, which we define as events purposefully included to mitigate the intended bias or create a story acceptable to the media industry's market. figure 1 shows examples of partisan events and counter-partisan events. to support our study, we first collect and label pac, a dataset of 8,511 partisan and counterpartisan events in 304 news articles. focusing on the partisan nature of media, pac is built from 152 sets of news stories, each containing two articles with distinct ideologies. analysis on pac reveals that partisan entities tend to receive more positive sentiments and vice versa for count-partisan entities. we further propose and test three hypotheses to explain the inclusion of counter-partisan events, considering factors of newsworthiness, market breadth, and emotional engagement. we then investigate the challenges of partisan event detection by experimenting on pac. results show that even using carefully constructed prompts with demonstrations, chatgpt performs only better than a random baseline, demonstrating the difficulty of the task and suggesting future directions on enabling models to better understand the broader context of the news stories. ## related work prior work has studied media bias primarily at the word-level @xcite and sentence-level @xcite @xcite . similar to our work is informational bias @xcite , which is defined as "tangential, speculative, or background information that tries to sway readers' opinions towards entities in the news." however, they focus on span-level bias, which does not necessarily contain any salient events. in contrast, our work considers bias on the event level, which is neither "tangential" to news, nor at the token level. importantly, we examine both partisan and counterpartisan events in order to study how these core, higher-level units produce ideological effects while maintaining an appearance of objectivity. our work is also in line with a broad range of research on framing @xcite , in which news media select and emphasize some aspects of a subject to promote a particular interpretation of the subject. partisan events should be considered as one type of framing that focuses on fine-grained content selection phenomenon, as writers include and present specific "facts" to support their preferred ideology. moreover, our work relates to research on the selection or omission of news items that explicitly favor one party over the other @xcite @xcite , or selection for items that create more memorable stories @xcite . in contrast, we focus on core news events, those that may not explicitly favor a side, but which are nevertheless ideological in their effect. finally, our research is most similar to another recent study on partisan event detection @xcite , but they only investigate partisan events and focus on developing computational tools to detect such events. in contrast, our work also incorporates counter-partisan events, enabling a broader and deeper understanding of how media tries to balance impartial news coverage and promoting their own stances. we also construct a significantly larger dataset than the evaluation set curated in @xcite , enhancing its utility for model training. ## partisan event annotation pac contains articles from two sources. we first sample 57 sets of news stories published between 2012-2022 from seesaw @xcite . each news story set contains three articles on the same story from outlets with different ideologies. here we take out the articles labeled with a center ideology and only keep stories with two news articles from opposite ideologies. to increase the diversity of topics in our dataset, we further collect 95 sets of news stories from www.allsides.com , covering topics such as abortion, gun control, climate change, etc. we manually inspect each story and keep the ones where the two articles are labeled with left and right ideologies. next, we follow the definition of events from timeml @xcite , i.e., a cover term for situations that happen or occur, and train a roberta-large model on matres @xcite for event detection. our event detector achieves an f1 score of 89.31, which is run on pac to extract events. next, the partisan events are annotated based on the following process. for each pair of articles in a story, an annotator is asked to first read both articles to get a balanced view of the story. then, at the article level, the annotator determines the relative ideological ordering, i.e., which article falls more on the left (and the other article more right) on the political spectrum. then, the annotator estimates each article's absolute ideology on a 5-point scale, with 1 being far left and 5 as far right. for each event in an article, annotators first identify its participating entities, i.e., who enables the action and who is affected by the events, and assign them an entity ideology when appropriate, and estimate the sentiments they receive if any. using the story context, the article's ideology, and the information of the participating entities, annotators label each event as partisan, counter-partisan, or neutral relative to the article's ideology, based on the definitions given in the introduction. if an event is labeled as non-neutral, annotators further mark its intensity to indicate how strongly the event supports the article's ideology. a complete annotation guideline can be found in appendix a. the annotation quality control process and inter-annotator agreement are described in appendix b. we also discuss disagreement resolution in appendix c. the final dataset statistics are listed in partisan sentiment. news media also reveal their ideology in the partisan entities they discuss, via the sentiments associated with those entities, where partisan entities tend to have positive associations and vice versa for count-partisan entities @xcite . in figure 3 , we find support for this expectation. we also find that left entities generally receive more exposure in articles from both sides. partisan event placement. figure 4 shows that for both left and right media outlets, partisan events appear a bit earlier in news articles. for counterpartisan events, left-leaning articles also place more counter-partisan events at the beginning, while right-leaning articles place more counter-partisan events towards the end. this asymmetry suggests that right-leaning outlets are more sensitive to driving away readers with counter-partisan events, thus placing them at the end of articles to avoid that. left right ideology 0.00 0.01 0.02 0.03 0.04 0.05 commont event probability common event probability partisan neutral counter-partisan figure 5: the probability of each type of event being common in left and right articles. if an event is counterpartisan, it will more likely be a common event. in this section, we investigate a number of hypotheses about why media outlets include both partisan and counter-partisan events. it is intuitive to understand why partisan events are incorporated into the news storytelling processing, yet it is unclear why counter-partisan events that portray members of one's own group negatively or members of another group favorably are reported. specifically, we establish and test three hypotheses for why an outlet would include counter-partisan news, similar to some of the theories articulated in groeling (2013): (1) newsworthiness, (2) market breadth, and (3) emotional engagement. ## experiments we experiment on pac for two tasks. partisan event detection: given all events in a news article, classify an event as partisan, counter-partisan, or neutral. ideology prediction: predict the political leaning of a news article into left or right. ## conclusion we conducted a novel study on partisan and counter-partisan event reporting in news articles across ideologically varied media outlets. our newly annotated dataset, pac, illustrates clear partisan bias in event selection even among ostensibly mainstream news outlets, where counter-partisan event inclusion appears to be due to a combination of newsworthiness, market breadth, and emotional engagement. experiments on partisan event detection with various models demonstrate the task's difficulty and that contextual information is important for models to understand media bias. marta recasens, cristian danescu-niculescu-mizil, and dan jurafsky. 2013. linguistic models for analyzing and detecting biased language. in proceedings of the 51st annual meeting of the association for computational linguistics (volume 1: long papers), pages 1650-1659, sofia, bulgaria. association for computational linguistics. roger c schank and robert p abelson. 1977. scripts, plans, goals, and understanding: an inquiry into human knowledge structures. psychology press. fuyuan shen, lee ahern, and michelle baker. 2014. stories that count: influence of news narratives on issue attitudes. journalism & mass communication quarterly, 91(1):98-117. timo spinde, manuel plank, jan-david krieger, terry ruas, bela gipp, and akiko aizawa. 2021. neural media bias detection using distant supervision with babe -bias annotations by experts. in findings of the association for computational linguistics: emnlp 2021, pages 1166-1177, punta cana, dominican republic. association for computational linguistics. arjen van dalen. 2012. structural bias in cross-national perspective: how political systems and journalism cultures influence government dominance in the news. the international journal of press/politics, 17(1):32-55. esther van den berg and katja markert. 2020. context in informational bias detection. in proceedings of the 28th international conference on computational linguistics, pages 6315-6326, barcelona, spain (online). international committee on computational linguistics. tae yano, philip resnik, and noah a. smith. 2010. shedding (a thousand points of) light on biased language. in proceedings of the naacl hlt 2010 workshop on creating speech and language data with amazon's mechanical turk, pages 152-158, los angeles. association for computational linguistics. xinliang frederick zhang, nick beauchamp, and lu wang. 2022. generative entity-to-entity stance detection with knowledge graph augmentation. in proceedings of the 2022 conference on empirical methods in natural language processing, pages 9950-9969, abu dhabi, united arab emirates. association for computational linguistics.