LLM -based Adversarial Dataset Augmentation for Automatic Media Bias Detection (2025)

Abstract: This study presents BiasAdapt, a novel data augmentation strategy designed to enhance the robustness of automatic media bias detection models. Leveraging the BABE dataset, BiasAdapt uses a generative language model to identify bias-indicative keywords and replace them with alternatives from opposing categories, thus creating adversarial examples that preserve the original bias labels. The contributions of this work are twofold: it proposes a scalable method for augmenting bias datasets with adversarial examples while preserving labels, and it publicly releases an augmented adversarial media bias dataset.Training on BiasAdapt reduces the reliance on spurious cues in four of the six evaluated media bias categories.

## introduction automatic media bias detection has gained significant attention with more capable language models. systems that automatically detect media bias can help media consumers better identify slanted reporting, help journalists uncover overlooked biases, and help researchers evaluate the reporting landscape @xcite . however, existing models often rely on spurious cues for classification decisions, which can lead to a superficial understanding of bias and compromise their generalization capabilities and objectivity @xcite . data augmentation techniques can mitigate the reliance on such shortcuts @xcite . training data for systems that automatically detect media bias originates predominantly from small, manually labeled datasets @xcite with associated high labeling costs @xcite . classical data augmentation techniques would require manual relabeling for every augmented sentence, as, for instance, random swaps of words or deletions could alter the bias of a sentence. to mini-mize the high manual relabeling costs, an adaptation is required. biasadapt, a process designed to enhance the robustness of automatic media bias detection systems, aims to address this. 1 biasadapt identifies keywords associated with predefined categories such as gender, origin, or political affiliation. it then generates and replaces alternative words from opposing subcategories. in this study, this adversarial augmentation process is performed on the babe dataset @xcite . the augmented data serves as training data, reducing reliance on spurious cues in four of the six evaluated media bias categories. however, these modifications also affect classification performance in some categories, requiring further investigation. the process of augmenting an existing data set with adversarial data using llms is transferable to domains beyond the detection of media bias. it allows for label-preserving alterations of predefined dimensions with accurate content exchanges that require an in-depth understanding of the sentence. ## related work media bias, a phenomenon where the information presented in the media is skewed, has been the subject of significant research @xcite @xcite . advances in bias detection, mainly through transformerbased methodologies, have notably improved classification accuracy @xcite . despite these advancements, a persistent challenge is the dependence on small, narrowly focused, manually annotated datasets @xcite . this limitation often results in models that overfit and generalize poorly. recent work by @xcite highlights that transformer-based models in automatic media bias detection predominantly target highly connotative words and do not grasp the nuance of context. this leads to reliance on unreliable indicators or spurious cues for classification decisions, manifesting itself as inconsistent bias determinations under stress tests. spurious cues in this context are superficial lexical features, such as demographic terms or political affiliations, that bias detection models incorrectly rely on to classify bias instead of analyzing the actual linguistic and contextual indicators of bias. @xcite introduce a checklistbased invariance test (inv) @xcite to assess the resilience of bias detection models to irrelevant input alterations. they define seven bias categories -gender, origin, religion, political affiliation, occupation, politician names, and disability-based on prior literature and practical observations of bias-related word associations. their checklist-based invariance test systematically examines whether altering terms within these categories (e.g., replacing a male-associated name with a female-associated one) changes the model's classification. if the model's bias determination fluctuates despite maintaining sentence semantics, it suggests reliance on spurious cues rather than true contextual understanding. @xcite report significant disparities in model behavior across datasets. for example, words linked to gender or origin frequently influence bias predictions, implying that classifiers are using these cues instead of analyzing how bias is actually expressed. such findings emphasize the necessity of model refinement for more robust detection methods. @xcite propose adversarial training and data augmentation to enhance model robustness. @xcite showcase the utility of adversarial examples in evaluating and enhancing the robustness of natural language processing models, a key consideration in detecting and mitigating media bias. additionally, @xcite highlight the significance of adversarial data augmentation in addressing the subtleties of language, suggesting its essential role in refining models tasked with understanding nuanced biases. this study refines media bias detection through adversarial data augmentation, addressing the limitations of existing methods. techniques like frequency-guided word substitution (fgws) @xcite and adversarial text modifi-cations @xcite often fail to preserve bias labels, requiring costly human reannotation @xcite when biases are unintentionally altered. when, for instance, words are randomly added or deleted, a previously unbiased sentence might now be biased. the strategy proposed in this study offers key improvements: ## methodology the biasadapt augmentation process expands the dataset to improve bias detection within text-based content. the process begins with an existing annotated dataset. in this case, the babe @xcite data set consists of sentences that are binary labeled for bias. the next step identifies keywords within each sentence by predefined categories. a keyword is any word that can clearly be attributed to one category. for instance, for gender, every gender-associated word is a keyword; for religion, every religion-associated word, and so on. for the context of media bias, these categories are gender, origin, religion, political affiliation, occupation, and politician names as defined by @xcite . 2 as these categories need to be predefined before the annotation, prior knowledge of where spurious cues may arise in the specific context is necessary. biasadapt identifies keywords by individually querying each sentence to a generative language model. for all prompts, gpt-3.5 turbo @xcite is used. the language model returns the identified keywords and the associated category (gender, origin, etc.). once more, these words are queried using the same language model with instructions to generate alternative words for each keyword. the process queries the same language model again, instructing it to generate alternative words for each keyword. these alternatives must come from opposing categories, ensuring they are associated with, for instance, an opposite political affiliation, gender, or a different religion. the alternative words then substitute the original terms in the sentence to create new instances, each maintaining the initial bias label. the bias label remains unchanged because the substituted keywords belong to the same predefined category, ensuring that the sentence's bias, whether introduced through framing or word choice, is preserved. bias can arise from how a sentence is structured but also from the connotations of specific words. for example, replacing 'he' with 'she' in 'he lacks the toughness for leadership' retains gender bias because the stereotype about leadership remains intact. similarly, swapping 'christian' with 'muslim' in 'policy unfairly favors christian values' maintains religious bias by preserving the critical framing of the sentence. in political contexts, replacing 'left-wing politician' with 'right-wing politician' in a sentence about corruption does not alter the underlying bias, as the negative framing remains the same. likewise, in occupation-based bias, exchanging 'artist' with 'construction worker' in 'artists contribute little to the economy' preserves bias against certain professions. since these substitutions maintain the same bias patterns, the augmentation process ensures that the dataset's labels remain consistent. this only works for predefined bias categories with predefined opposing subcategories that substitutions can be taken from. in some cases, substitutions may interact with the sentence structure in ways that subtly alter the perceived bias. for example, in 'she is caring and nurturing,' substituting 'she' with 'he' could challenge the stereotype that these traits are inherently feminine, as men are less commonly associated with these characteristics in traditional gender roles. this demonstrates that substitutions in certain contexts may shift or reinforce bias depending on the societal associations linked to the words involved. while the augmentation process follows strict category-based substitutions, potential context-dependent bias shifts are a limitation of this method. figure 1 displays the augmentation process with an example sentence from the babe dataset. each sentence may contain multiple identified keywords, each with a list of alternative words, resulting in numerous possible permutations. when substituting these keywords, the rest of the sentence and its label remain unchanged. that is why generating too many permutations can lead to overfitting when the data is used for model training. for this study, three permutations per original sentence are found to be the best trade-off between introducing adversarial examples and the prevention of overfitting. however, this might vary depending on the dataset size, sentence complexity, and length. the three permutations are chosen by randomly sampling alternative words from the word lists. the training setup ensures no data leakage between the training, test, and validation data, as original and altered sentences are always in the same set. this process creates an adversarial babe dataset, which is then used to train a language model to automatically detect media bias. its detection capabilities are compared to that of a model trained solely on babe. the performance of the models is evaluated using the test sets from @xcite . the test set consists of 1,900 binarylabeled sentences distributed over the categories (50% of which are classified as biased). within each category, variance serves as a metric for spurious cues: higher values suggest that the model relies on shortcuts rather than general language understanding. for example, if the model does not use gender as a factor in classification, accuracy should remain consistent across sentences containing male, female, and non-binary keywords. both models are based on a pre-trained roberta model to ensure comparability with @xcite . the model training ends based on an early stopping criteria. ## results the augmentation of the babe dataset through bi-asadapt significantly increases the dataset size to 14,659 entries, adding 10,986 entries to the original collection. not for every original sentence permutations can be constructed, as not all sentences contain words that are identified as keywords being associated with one of the predefined categories. while the distribution within categories remains equal, the occurrence of relevant keywords differs between categories depending on their occurrence in the babe dataset. in the initial step, a total of 4,906 keywords are identified and replaced. the most frequently modified categories are gender (1,469 identified keywords) and politician names (1,232), followed by origin (609), political affiliation (464), religion (97), and occupation (35). this distribution is primarily influenced by the topic choices of the babe dataset. this study's evaluation, detailed in ## discussion biasadapt successfully identifies and replaces relevant keywords though there is still an underrepresentation of certain categories with little occurrence in the original dataset. the observed ## conclusion this study presents biasadapt, a data augmentation strategy aimed at improving the robustness of media bias detection systems through adversarial examples. by leveraging prior knowledge of spurious cue dependencies, biasadapt demonstrates that data augmentations utilizing large language models (llms) can have a measurable impact on improving bias detection performance. making a significant corpus available for public use lays the groundwork for further exploration in the field. while the focus on a single model and a select number of bias categories limits the generalizability of the findings, this work demonstrates the potential of leveraging llms for dataset augmentation and increased robustness in media bias detection. despite the demonstrated benefits, further investigations to better understand model behavior is necessary. still, it encourages expanding the scope and transfer to other areas of text analysis with prerequisites similar to media bias.