Beyond the Surface: Spurious Cues in Automatic Media Bias Detection (2024)

Abstract: This study investigates the robustness and generalization of transformer-based models for automatic media bias detection. We explore the behavior of current bias classifiers by analyzing feature attributions and stress-testing with adversarial datasets. The findings reveal a disproportionate focus on rare but strongly connotated words, suggesting a rather superficial understanding of linguistic bias and challenges in contextual interpretation. This problem is further highlighted by inconsistent bias assessment when stress-tested with different entities and minorities. Enhancing automatic media bias detection models is critical to improving inclusivity in media, ensuring balanced and fair representation of diverse perspectives.

## introduction with increased capability in nlp methods, automatic media bias detection has improved rapidly. while transformer-based models are now predominantly used for media bias detection tasks, concerns remain about the robustness and generalization of these models. there have been indications that the models use shortcuts in classification, leading to a superficial rather than fundamental understanding of bias @xcite . for example, the babe model by @xcite demonstrates this issue in its approach to linguistic bias detection. it assigns biased confidence levels to named entities like "donald trump" (classified bias with a 0.531 confidence) and "hillary clinton" (classified not biased with a 0.809 confidence), erroneously suggesting bias based on names alone. 1 this indicates a critical problem: the model associates certain names with bias, undermining its ability to generalize and accurately assess bias based on context. however, to what extent this is a problem in automatic media bias detection has not yet been explored. through an attribution score analysis, this study finds that the methods disproportionately focus on a small subset of strongly connotated, rare words. newly created checklist-based @xcite adversarial test sets further show the reliance on specific tokens and limited contextual understanding, pointing to spurious cues influencing the detection. 2 these findings call for developing more robust media bias detection models as they ensure fair and unbiased representation of diverse voices and perspectives, preventing the perpetuation of stereotypes and promoting a more equitable and inclusive discourse in media content. ## methodology to examine spurious cues in automatic media bias detection, a lime-based feature attribution analysis (faa) is conducted, and checklist-based adversarial test sets are constructed. following spinde et al. (2021) for all experiments, a roberta model fine-tuned on the babe expert annotation dataset is used. foot_2 ## discussion the low attribution scores for most words in the faa are to be expected as most words do not carry any bias-determining information. yet, the dependency on strongly connotated, infrequent words raises concerns about the model's potential for context and deeper bias understanding, as it may overly depend on these words for classification. nevertheless, these faa results are merely suggestive of this tendency. the reduction in both accuracy and f1-score upon the removal of named entities in the mft suggests a dependency of the model on these entities for bias detection. however, the only moderate decline in performance metrics indicates a certain level of robustness in detecting bias independently of named entities. the results of the inv test reveal inconsistent bias detection across categories, indicating a reliance on spurious cues. variances in f1 scores within categories like gender, origin, and religion suggest bias sensitivity towards specific tokens. for example, differences in accuracy for 'female' versus 'male' and 'non-binary' in gender and 'african' and 'asian' versus 'european' in origin highlight the model's uneven processing of demographic identifiers. these disparities, evident across various categories, demonstrate the model's inconsistent approach to neutral templates with different demographic tokens. the model's varied classification performance across categories suggests that some bias types and sentences are easier to classify than others. while ideally, the difficulty level should be uniform across all sentences, this disparity does not undermine the findings based on intra-category analysis. finally, the results of the dir indicate that while the model is proficient in detecting bias in plain sentences, it fails to differentiate when statements are in quotations. this confirms what is indicated by the faa that the model lacks contextual understanding. instead of a deeper language understanding, it is using simplistic heuristics (like the presence of adjectives or negative phrases) to classify sentences as biased. the model fails to recognize the contextual change when these appear inside quotations. ## conclusion the study reveals that media bias detection methods rely on strongly connotated words and named entities. the model's classification inconsistencies across categories such as gender and origin and its limitations in contextual understanding suggest a reliance on simplistic heuristics, pointing to spurious cues and a lack of nuanced language comprehension in bias detection. these findings challenge the generalization capabilities and robustness of current methods. future work should extend the analysis, especially of the adversarial dataset classifications, as the intra-category differences could reveal valuable insights into model biases beyond spurious cues. furthermore, it should examine mitigation strategies such as targeted data augmentation.