From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models (2023)

Abstract: Language models (LMs) are pretrained on diverse data sources—news, discussion forums, books, online encyclopedias. A significant portion of this data includes facts and opinions which, on one hand, celebrate democracy and diversity of ideas, and on the other hand are inherently socially biased. Our work develops new methods to (1) measure media biases in LMs trained on such corpora, along social and economic axes, and (2) measure the fairness of downstream NLP models trained on top of politically biased LMs. We focus on hate speech and misinformation detection, aiming to empirically quantify the effects of political (social, economic) biases in pretraining data on the fairness of high-stakes social-oriented tasks. Our findings reveal that pretrained LMs do have political leanings which reinforce the polarization present in pretraining corpora, propagating social biases into hate speech predictions and media biases into misinformation detectors. We discuss the implications of our findings for NLP research and propose future directions to mitigate unfairness.

## introduction digital and social media have become a major source of political news dissemination @xcite @xcite with unprecedentedly high user engagement rates @xcite @xcite . the volume of online discourse surrounding polarizing issues-climate change, gun control, abortion, wage gaps, death penalty, taxes, same-sex marriage, and more-has been drastically growing in the past decade @xcite @xcite . while online political engagement promotes democratic values and diversity of perspectives, these discussions also reflect and reinforce societal biases-stereotypical generalizations about people or social groups @xcite @xcite . such language constitutes a major portion of large language models' (lms) pretraining data, propagating biases into downstream models. hundreds of studies have highlighted ethical issues in nlp models @xcite @xcite and designed synthetic datasets @xcite or controlled experiments to measure how biases in language are encoded in learned representations @xcite , and how annotator errors in training data are liable to increase unfairness of nlp models @xcite . however, the language of polarizing political issues is particularly complex @xcite , and social biases hidden in language can rarely be reduced to pre-specified stereotypical associations @xcite . to the best of our knowledge, no prior work has shown how to analyze the effects of naturally occurring media biases in pretraining data on language models, and subsequently on downstream tasks, and how it affects the fairness towards diverse social groups. our study aims to fill this gap. as a case study, we focus on the effects of media biases in pretraining data on the fairness of hate speech detection with respect to diverse social attributes, such as gender, race, ethnicity, religion, and sexual orientation, and of misinformation detection with respect to partisan leanings. we investigate how media biases in the pretraining data propagate into lms and ultimately affect downstream tasks, because discussions about polarizing social and economic issues are abundant in pretraining data sourced from news, forums, books, and online encyclopedias, and this language inevitably perpetuates social stereotypes. we choose hate speech and misinformation classification because these are social-oriented tasks in which unfair predictions can be especially harmful @xcite @xcite . to this end, grounded in political spectrum theories @xcite @xcite and the political compass test, 2 we propose to empirically quantify the political leaning of pretrained lms ( §2). we then further pretrain language models on different partisan corpora to investigate whether lms pick up political biases from training data. finally, we train classifiers on top of lms with varying political leanings and evaluate their performance on hate speech instances targeting different identity groups @xcite , and on misinformation detection with different agendas @xcite . in this way, we investigate the propagation of political bias through the entire pipeline from pretraining data to language models to downstream tasks. our experiments across several data domains, partisan news datasets, and lm architectures ( §3) demonstrate that different pretrained lms do have different underlying political leanings, reinforcing the political polarization present in pretraining corpora ( §4.1). further, while the overall performance of hate speech and misinformation detectors remains consistent across such politically-biased lms, these models exhibit significantly different behaviors against different identity groups and partisan media sources. ( §4.2). the main contributions of this paper are novel methods to quantify political biases in lms, and findings that shed new light on how ideological polarization in pretraining corpora propagates biases into language models, and subsequently into social-oriented downstream tasks. in §5, we discuss implications of our findings for nlp research, that no language model can be entirely free from social biases, and propose future directions to mitigate unfairness. ## methodology we propose a two-step methodology to establish the effect of political biases in pretraining corpora on the fairness of downstream tasks: (1) we develop a framework, grounded in political science literature, to measure the inherent political leanings of pretrained language models, and (2) then investi-2 https://www.politicalcompass.org/test gate how the political leanings of lms affect their performance in downstream social-oriented tasks. ## experiment settings lm and stance detection model we evaluate political biases of 14 language models: bert @xcite , roberta @xcite , dis-tilbert @xcite , distilroberta, al-bert @xcite , bart @xcite , gpt-2 @xcite , gpt-3 @xcite , gpt-j (wang and komatsuzaki, 2021), llama @xcite , alpaca @xcite , codex @xcite , chatgpt, gpt-4 (openai, 2023) and their variants, representing a diverse range of model sizes and architectures. the specific versions and checkpoint names of each model are provided in appendix c. for the stance detection model used for evaluating decoder-based language model responses, we use a bart-based model @xcite trained on multinli @xcite . to ensure the reliability of the off-the-shelf stance detector, we conduct a human evaluation on 110 randomly sampled responses and compare the results to those generated by the detector. the stance detector has an accuracy of 0.97 for lm responses with clear stances and high interannotator agreement among 3 annotators (0.85 fleiss' kappa). details on the stance detector, the response-to-agreement mapping process, and the human evaluation are in appendix a.2. partisan corpora for pretraining we collected partisan corpora for lm pretraining that focus on two dimensions: domain (news and social media) and political leaning (left, center, right). we used the politics dataset @xcite for news articles, divided into left-leaning, right-leaning, and ## results and analysis in this section, we first evaluate the inherent political leanings of language models and their connection to political polarization in pretraining corpora. we then evaluate pretrained language models with different political leanings on hate speech and misinformation detection, aiming to understand the ## conclusion we conduct a systematic analysis of the political biases of language models. we probe lms using prompts grounded in political science and measure models' ideological positions on social and economic values. we also examine the influence of political biases in pretraining data on the political leanings of lms and investigate the model performance with varying political biases on downstream tasks, finding that lms may have different standards for different hate speech targets and misinformation sources based on their political biases. our work highlights that pernicious biases and unfairness in downstream tasks can be caused by non-toxic data, which includes diverse opinions, but there are subtle imbalances in data distributions. prior work discussed data filtering or augmentation techniques as a remedy @xcite ; while useful in theory, these approaches might not be applicable in real-world settings, running the risk of censorship and exclusion from political participation. in addition to identifying these risks, we discuss strategies to mitigate the negative impacts while preserving the diversity of opinions in pretraining data.