Disentangling Structure and Style: Political Bias Detection in News by Inducing Document Hierarchy (2023)

Abstract: We address an important gap in detecting political bias in news articles. Previous works that perform document classification can be influenced by the writing style of each news outlet, leading to overfitting and limited generalizability. Our approach overcomes this limitation by considering both the sentence-level semantics and the document-level rhetorical structure, resulting in a more robust and style-agnostic approach to detecting political bias in news articles. We introduce a novel multi-head hierarchical attention model that effectively encodes the structure of long documents through a diverse ensemble of attention heads. While journalism follows a formalized rhetorical structure, the writing style may vary by news outlet. We demonstrate that our method overcomes this domain dependency and outperforms previous approaches for robustness and accuracy. Further analysis and human evaluation demonstrate the ability of our model to capture common discourse structures in journalism.

## introduction one of the primary reasons people consume news is for social cognition: to stay informed about events and developments and make informed decisions. to fulfill this purpose, news outlets must provide citizens with information from diverse sources and perspectives. the pew research center @xcite reports that the majority of american respondents (78%) indicated that news organizations should refrain from showing favoritism towards any political party in their reporting. however, more than half of the respondents (52%) expressed dissatisfaction with the media's ability to report on political issues fairly and unbiasedly. extensive research has revealed that partisan bias exists in various social issues, such as the 2016 us presidential election @xcite , the 1 our code is available at: https://github.com/ xfactlab/emnlp2023-document-hierarchy throughout the history of human civilization, wherever there were established cultures and populations, there were borders(s 1 ) ... strong borders make good neighbors; also, safe homelands(s 2 ). ... if the immigration bill does not make it easier for law-abiding immigrants to com here to work and pursue the american dream, we miss an opportunity to truly fix illegal immigration once and for all(s21). like an electrical current, immigration follows the path of least resistance(s22). as dangerous and expensive as illegally entering the united states can be, it still may be the easiest route for many people, simply because of the bureaucratic complextites built into our immigration system(s23).. ...we also must ensure we manage the immigration process to our economic benefit(s24). ... our system of visas and work permits (including student visas) must be reformed so it is more rationally based on our counrty's workforce needs(s28). ... ## background measuring bias there have been continuous efforts to determine the political bias of news outlets, with two main approaches: audience-based and content-based analyses. audience-based methods assume that a news outlet's political stance can be inferred from the political preferences of its primary audience @xcite . this is because news outlets are expected to cater to the inclinations of their users to retain their viewership or readership. studies have shown that partisans tend to choose news from politically congruent outlets @xcite . however, recent web tracking technologies have revealed that a significant amount of traffic crosses ideological lines @xcite @xcite this conflicting evidence suggests the audience makeup of a news outlet may no longer be stable in a highly competitive media environment. content-based methods typically rely on student coders @xcite or workers from crowdsourcing platforms @xcite to measure progressive/conservative media bias. the introduction of computer-assisted content analysis techniques allows researchers to compare the linguistic patterns adopted by partisan media on a large scale @xcite . nonetheless, these methods are generally confined to specific news topics, such as the federal tax on inherited assets @xcite , and may not be generalizable to other news topics or entire news channels. recently, facial recognition algorithms have been used to quantify the visibility of politicians in broadcast news programs @xcite . this technique detects whether a program displays a liberal or conservative bias if it features more actors from the left (or right) for a longer duration. however, it is restricted to analyzing news visuals only and cannot be applied to other types of news content. ## related works detecting political ideologies and biases there are two common approaches in the content-based political bias detection task in nlp: either articlelevel @xcite @xcite , sentence-level @xcite @xcite , 2022). for sentence-level bias detection, previous work has centered around labeling additional data: spinde et al. (2021) developed a process for labeling data at a sentence-level with binary labels and fan et al. (2019) defined informational bias and lexical bias while providing sentence-level and word-level bias labels for news articles. with these datasets, lei et al. (2022) used discourse structures that inform the model about the role of a sentence to detect sentence-level bias. research in article-level bias centers around using external information @xcite @xcite . for example, baly et al. (2020) incorporate additional external information such as twitter bios and wikipedia pages related to each news outlet, and @xcite and @xcite implemented a knowledge graph of external facts to enhance the political bias detection performance. in related works, contrastive learning with additional data preprocessing (e.g. curating article triplets) has also helped build robust models learning from annotations of left, centrist, and right viewpoints on the same event @xcite . hierarchical attention hierarchical attention @xcite allows a model to encode longer texts by combining independently encoded sentences or phrases @xcite @xcite @xcite . in the legal domain, @xcite hierarchically formulated legal events and arguments for legal event extraction. in the medical domain, @xcite used hierarchical attention in medical ontology to embed medical terminologies with respect to both low-and high-level concepts. for journalism, @xcite utilized hierarchical attention for building discourse structure trees on fake news detection tasks. also, for political ideology classification, @xcite al. (2018) relevance propagation relevance propagation methods are widely used in many domains to improve the explainability of black box models @xcite @xcite . specifically for nlp, @xcite propagate model weights to highlight relevant tokens for predictions. our method widens the range of relevance propagation in nlp by considering sentence-level relevance in document understanding. article-level political bias datasets for articlelevel bias detection, many previous works published labeled datasets with news articles collected from allsides.com @xcite @xcite . models will likely memorize aspects irrelevant to political bias when predicting the label. publisher @xcite , news outlet @xcite , and topic @xcite have shown confounding effects. while @xcite reported this discrepancy between seen news outlets for training and unseen news outlets during testing, they did not analyze the cause of this problem. ## model architecture we have three modeling objectives: (1) to induce sentence-level information from a document-level label, (2) to understand the discourse structure of the news article, and (3) to reduce the effect of spurious correlation between the media outlet and the bias label. it is critical to model the complex relation between sentences in the news article when predicting its political ideology, to this end, we apply hierarchical modeling to propagate document information to sentences. we propose a bias prediction model that uses a two-layer tree with sentence-level embeddings to better understand the news article's semantics and discourse structure. our pipeline consists of three key stages that can be jointly trained as a single unified model: semantic analysis, sentence type detection, and context clustering. each component is described in detail in the following sections, and the model architecture is illustrated in figure 2 . ## experiments we compare our model against a bert baseline classifier for the news article classification task with experiments to ( 1 ) compare our model against the bert baseline, which is representative of previous work; (2) compare the accuracy of the model on two disjoint test sets (3) evaluate sensitivity to the content of training data and (4) evaluate sensitivity to the number of training data. in all experiments, we report auroc and macro f 1 scores. the hyperparameters for training are in appendix a. ## result we report macro f 1 and auroc for the documentlevel bias prediction task in ## structural analysis in this section, we analyze the structural properties of news articles using the main sentences identified by the model. to do so, we collect the predicted main sentences and assess if they capture the formalized discourse structures commonly used in journalism. by validating from summarization and structure, we ensure the reliability of using multiple attention heads as an explanation mechanism. we use the basil dataset, which contains sentence-level annotations for two types of biases @xcite : lexical and informational bias. lexical bias refers to the bias from the word choice of the journalist, such as using polarized words (e.g. donald trump is investing more in conspiracy theories about president obama's birth certificate as he explores his bid for the presidency.) informational bias refers to the biased elaboration of certain events or facts, which includes using selective quotations to strengthen their viewpoint. (e.g. the arizona group said the call from mr. trump on wednesday came unexpectedly, and the group had spent much of the day thursday scurrying to make travel arrangements to new york.) ## conclusions our multi-head attention model leverages sentencelevel information to capture the narrative structure. our model outperforms conventional documentlevel classifiers by mitigating the domain dependency constraints of traditional classifiers and generating more robust and precise document-level representations. we validated our model's effectiveness in capturing journalism's rhetorical structures and writing styles.