Balancing Transparency and Accuracy: A Comparative Analysis of Rule-Based and Deep Learning Models in Political Bias Classification (2024)

Abstract: The unchecked spread of digital information, combined with increasing political polarization and the tendency of individuals to isolate themselves from opposing political viewpoints opposing views, has driven researchers to develop systems for automatically detecting political bias in media. This trend has been further fueled by discussions on social media. We explore methods for categorizing bias in US news articles, comparing rule-based and deep learning approaches. The study highlights the sensitivity of modern self-learning systems to unconstrained data ingestion, while reconsidering the strengths of traditional rule-based systems. Applying both models to left-leaning (CNN) and right-leaning (FOX) News articles, we assess their effectiveness on data beyond the original training and test sets. This analysis highlights each model’s accuracy, offers a framework for exploring deep-learning explainability, and sheds light on political bias in US news media. We contrast the opaque architecture of a deep learning model with the transparency of a linguistically informed rule-based model, showing that the rule-based model performs consistently across different data conditions and offers greater transparency, whereas the deep learning model is dependent on the training set and struggles with unseen data.

## introduction the current political climate in the united states is characterized by intense polarization and an unprecedented ease of publishing and disseminating information, where partisan hostility and negative perceptions of opposing party members are at an all-time high @xcite . this dynamic is further exacerbated by social media platforms, where users curate their news feeds in a way that reinforces existing biases and isolates them from diverse perspectives, stifling constructive dialogue and creating what researchers term "epistemic bubbles" @xcite . to address this, natural language processing (nlp) researchers have developed models intended to automatically and objectively detect the presence and direction of bias. examples include model architectures ranging from rule-based designs @xcite to state of the art (soa) transformer architectures @xcite . while soa architectures have been shown to distinguish biased narratives from neutral ones, they struggle to learn the nuanced nature of bias expression without a sufficiently large and comprehensive dataset. our contributions include an investigation of both a rule-based and a deep learning model for political bias classification as depicted in figure 1 , with the goal of promoting a more informed discussion on bias detection methodologies. to overcome data demands of soa architectures, we adopt a convolutional neural network model. 1 our contrasting approach is a simpler, more transparent rule-based model for bias classification using sentiment detection and linguistic features. this model does not rely on preexisting bias lexicons, "black box" machine learning models, or large training datasets. moreover, its simplicity allows for easy correction, with a few, clearly delineated, components. a second contribution is the use of linguistic in-formation for detecting an article's stance towards entities. 2 our rule-based approach includes a novel part-of-speech driven "reference resolution" (e.g., associating adjectives with a corresponding noun), for a more focused stance assignment. we emphasize that it is not our goal to achieve soa performance for political bias classification through the rule-based model, but rather to explore the extent to which straightforward linguistic features (parts of speech, coreference, and sentiment) can be leveraged to classify political bias. a third contribution involves exploring methods to enhance explainability of deep learning models. by testing a convolutional nn model on various datasets and correlating its performance disparities with differences in the data, we identify the features prioritized by the model. our findings show that the rule-based model maintains consistent performance across various data conditions, presenting a clear right-leaning bias for fox. by contrast, the convolutional nn model relies heavily on its training set, struggling with data not directly related to the political bias data on which it is trained. the rule-based approach performs comparably to deep learning in these situations, making it more applicable to real-world scenarios and offering greater transparency. the next section reviews bias detection methodologies in news media. section 3 covers data collection, preprocessing, and experimental setup. section 4 details the implementation of rule-based and convolutional nn models. section 5 evaluates model performance and their application to external data, with concluding remarks in section 6. ## related work following @xcite , we view bias in news articles not as as a distortion or selective presentation of information to convey a belief, potentially impacting readers' opinions. media bias is categorized into coverage bias, gatekeeping bias, and statement bias @xcite . our study focuses on statement bias, i.e., the use of rhetoric describing entities @xcite identified by our rule-based sentiment analysis model through identification of words conveying sentiment toward entities. entity level semantic analysis (elsa) @xcite , is exemplified by the work of @xcite , where sentiment toward an entity is computed across sentences, iteratively lowering the sentiment scores for entities appearing in negative contexts. our current study adopts a form of elsa that eliminates the need for "negative smoothing" by using part-of-speech (pos) resolution to identify sentiment towards a given entity, thus filtering out "noise" introduced by incidental occurrences of nearby negative terms. deeplearning elsa models often suffer from an opaque architecture and overly broad feature selection. @xcite address this with a transparency layer in a convolutional nn, that adjusts feature selection using an integrated gradient technique, aligning with the pos resolution method described here. bias detection in media is typically handled as binary or multi-class classification, mapping to political leanings using e.g., support-vector machines, logistic regression, and random forest techniques @xcite with handcrafted feature extraction. @xcite adopt a rule-based strategy, defining a list of inflammatory terms and expanding it with word2vec @xcite from conservapedia articles foot_2 to create a lexicon of politically charged words. our rule-based model differs by not relying on lists of predefined terms; instead, it assumes that differing stances towards an entity across articles indicate bias. this simpler approach hinges on stances towards notable entities, differing from the single-sentence approach of @xcite . our model's theoretical foundation suggests that differences in stance expression between media outlets signal statement bias. bias detection research favors transformers over recurrent neural networks (rnns), due to their self-attention mechanism for modeling sequential structures @xcite . however, their reliance on low-level lexical information @xcite , is often insufficient for political bias detection. 4 chen et al. (2020) attempt to overcome hand-crafted feature extraction limitations, while avoiding deep learning's pitfalls, by analyzing second-order information, like the frequency and order of biased statements, and employing machine-learning methods for bias detection. our hybrid approach aligns with this, but focuses on human interpretable bias features (e.g. stance), and favors deep learning over classical machine learning methods for a more flexible interpretation of these features. to develop explainability, techniques such as sensitivity analysis and layer-wise relevance propagation (lrp) address the black-box nature of deep learning models @xcite . this explores the limitations of deep convolutional architectures by assessing model performance against training and external articles, and identifying differences in the data that correlate with performance variations. ## data querying and setup both models require first acquiring article data and correcting imbalances to prevent model bias. ## model implementations the rule-based sentiment analysis model isolates sentiment expressed towards both common and proper nouns, leveraging adjectives and verbs that describe them. this approach aligns with findings from recent research, which focus on descriptive language used in relation to specific entities @xcite to detect bias through sentiment and stance in news articles. the model employs coreference resolution to ensure direct reference of verbs and adjectives with correct name entities. locating the nouns referenced by verbs and adjectives is accomplished through the aforementioned pos leveraging these rule-based outputs, the model creates sentiment vectors embedding the sentiment towards all nouns in articles by political leaning, where each dimension is defined by a unique noun. sentiment is quantified using the valence scores of all verbs and adjectives considered. it classifies bias by comparing the cosine distance between an article's vector and sentiment vectors for each political leaning. then the political leaning closest to the article's vector is predicted as its bias. figure 3 shows a theoretical mapping of the three corpora projected onto a 2d plane, with each dimension representing sentiment toward a corresponding entity based on all the adjectives and verbs referencing it within each corpus. as expected, the right-leaning corpus shows negative sentiment towards biden and positive sentiment towards trump, while the left-leaning corpus expresses the opposite. the center-leaning corpus displays a neutral sentiment towards both. an input article is then positioned on the plane based on the sentiment it expresses towards both entities. the shortest cosine distance is found between the input article and the center corpus, indicating that the article's stance is most aligned with the center corpus. this suggests that a highly negative stance towards trump, with a moderately negative stance towards biden, indicates a politically centered standpoint. by contrast, the deep learning model processes raw text directly, without segmenting or extracting stance-specific meaning. the convolutional nn model captures dependencies and recurring structures in text through multiple deep learning layers. this model achieves strong performance (see section 5) in classifying articles across three outlets. both the rule-based and deep learning models are applied to the preprocessed dataset. the implementation of the rule-based model (section 4.1) is more involved than that of the deep-learning convolutional nn model (section 4.2) in that the latter uses a standard architecture, whereas the former proposes a novel design. our models are powered by 2 amd epyc 75f3 cpu cores complemented by 2 nvidia a100 gpu cores. we use 80% of the data for training and the remaining 20% for testing the models. although the training suite encompasses the full extent of the time period explored, temporal leakage is not an issue, as the models are not devoted to the prediction of bias in future news articles. the idea of a temporal dataset is solely meant to provide a more comprehensive span of biased text. ## results we evaluate both models' performance by examining precision, recall, and f1 across the three classes. ,using a dataset comprising 20% of the original data. results reflect the models' classification accuracy, not their ability to recognize political bias. we revisit this distinction in our evaluation of model performance on external news outlets.