Measuring Political Bias in Large Language Models: What Is Said and How It Is Said (2024)

Abstract: We propose to measure political bias in LLMs by analyzing both the content and style of their generated content regarding political issues. Existing benchmarks and measures focus on gender and racial biases. However, political bias exists in LLMs and can lead to polarization and other harms in downstream applications. In order to provide transparency to users, we advocate that there should be fine-grained and explainable measures of political biases generated by LLMs. Our proposed measure looks at different political issues such as reproductive rights and climate change, at both the content (the substance of the generation) and the style (the lexical polarity) of such bias. We measured the political bias in eleven open-sourced LLMs and showed that our proposed framework is easily scalable to other topics and is explainable.

## introduction as the pervasiveness of ai in human daily life escalates, extensive research has illuminated its limitations and potential harms such as gender and racial biases and hallucinations @xcite @xcite @xcite @xcite . among these, political bias in ai, a notably crucial yet underexplored facet, poses significant risks by potentially distorting public discourse and exacerbating societal polarization @xcite @xcite . existing scholarly efforts have explored political bias ingrained in lms, mainly focused on stance at the political-orientation level (i.e., left or right/liberal or conservative) @xcite @xcite . the political orientation tests-based methodology (e.g., the political compass test 1 ) is often employed, yet it may be inadequate to fully capture the complex dynamics of bias within llmgenerated content @xcite . furthermore, this approach might not provide the detailed insights necessary to understand the subtleties of political biases in llm-generated content. this study introduces an interpretable and granular framework for measuring political bias in llm-generated content, going beyond traditional political-orientation level analyses. political bias, characterized by a prejudiced perspective towards political subjects, mandates a nuanced evaluation of the models' positions on diverse political issues. this bias predominantly manifests through framing, which entails the deliberate selection and emphasis of specific informational elements, both in content and style, to shape perceptions @xcite @xcite @xcite . given the multifaceted nature of political bias, our framework employs a two-tiered approach for its assessment, encompassing both topicspecific stance and framing. to assess the models' stance on distinct political topics, we implement a method of extreme anchor comparison, quantifying the similarity be- generation 𝑃𝑃( � 𝑌𝑌) proponents 𝑷𝑷(𝒀𝒀 𝐩𝐩𝐩𝐩𝐩𝐩 ) opponents 𝑷𝑷(𝒀𝒀 𝐩𝐩𝐩𝐩𝐩𝐩 ) compare distances to estimate stance content bias style bias instruction 𝒙𝒙 generation 𝑷𝑷(� 𝒚𝒚) content style 𝒈𝒈 𝒉𝒉 𝐝𝐝(𝑷𝑷(𝒀𝒀 𝐩𝐩𝐩𝐩𝐩𝐩 )||𝑷𝑷( � 𝒀𝒀)) 𝐝𝐝(𝑷𝑷(𝒀𝒀 𝐩𝐩𝐩𝐩𝐩𝐩 )||𝑷𝑷( � 𝐘𝐘)) 𝑓𝑓 llm 𝒈𝒈 𝒉𝒉 latent variable 𝒛𝒛 = [𝒄𝒄, 𝒔𝒔] 𝒉𝒉 -𝟏𝟏 anchors-based political stance analysis (section 2.1) decomposition of framing biases (section 2.2) identify content and style biases tween model outputs and two opposed stancesadvocacy and opposition -across various political subjects. subsequently, to dissect the political bias of llms more thoroughly, we examine framing by decomposing both the content and style. this involves a detailed content analysis leveraging boydstun's frame dimensions and entity-based frames, coupled with an evaluation of stylistic bias, including the examination of media bias in writing styles and the presence of non-neutral sentiment towards salient entities of topic. collectively, our framework not only discerns topic-specific stances but also explores the intricate dynamics of the content ("what" is said) and style ("how" it is said) concerning contentious topics. the ultimate aim is to provide a measurement of political biases inherent in various llms, thereby paving the way for the development of strategies to diminish these biases and enhance the reliability and equity of llm applications. drawing upon the empirical evidence and analytical insights derived from our frameworks, this study elucidates a set of findings that furnish a guide for subsequent research endeavors within the community. the key discoveries include: (1) llms show different political views depending on the topic, such as being more liberal on reproductive rights and more conservative on immigration; ## the framework we present an evaluation framework that provides fine-grained, topic-specific analysis for political bias in llms rather than a generalized ideological examination. our approach recognizes the complexity of bias, where, an llm might exhibit liberal stances on certain topics (e.g., reproductive rights) while leaning conservative on others (e.g., death penalty), reflecting a sophisticated and variable political landscape. moreover, since we evaluate models' actual generation, we can also understand how biases are conveyed to users in practical applications. specifically, we begin by preparing a set of natural language instructions @xmath0, x 2 , ..., x t } covering a total of t sensitive political topics. for the llm f llm that we want to evaluate, we obtain the distribution of model generated response by f llm (x i ) → p ( ŷi ) for each topic. to measure the political bias inherent in p ( ŷi ), we introduce a two-tiered framework, focusing first on the political stance (section 2.1) analysis, and a more detailed examination of framing bias (section 2.2) analysis. hereafter, for simplicity, we will omit the topic index i. ## framework implementation in this section, we describe how we implement the evaluation framework proposed in section 2. ## related work framing and political bias with nlp the study of political bias and framing is a well-established field that intersects social science, political science, and communication studies @xcite @xcite @xcite . nlp techniques have been actively applied to analyze polarized political ideologies and framing within human-generated media @xcite @xcite @xcite @xcite . key efforts from nlp in reducing political polarization involve understanding framing @xcite @xcite , detecting political ideologies @xcite @xcite and other mitigation methods @xcite @xcite to reduce polarization. building on this foundation, our work shifts focus to examining the political biases inherent in llms. political bias evaluation in lms recent studies in nlp/ai have extensively explored social biases related to llms, particularly focusing on fairness and safety in addressing social biases @xcite @xcite @xcite @xcite . yet, focused studies on political bias in lm has been relatively limited. early research efforts include exploration in word embedding @xcite , conditional generation @xcite @xcite , and in chatbot @xcite . recently, @xcite showed that humans engaged in more biased information querying with llm-powered conversational search, and an opinionated llm reinforcing their views exacerbated this bias (echo chambers effect), which indicates the importance of political bias study of llm. recent studies have explored the political bias in llm using surveys designed for humans (e.g., political compass) -feng et al. (2023); rozado (2023) 2024 ) reveals that models give different answers in different settings when tested with the survey-based approach thus advocates making evaluations that match user behaviors rather than asking llms directly about their values. our work, distinct yet complementary, introduces a framework meticulously crafted for delving into the nuanced dynamics of llms' political biases. this approach is designed to furnish a more detailed analysis by scrutinizing the content generated on political subjects, thereby enriching our collective understanding of the topic. finally, the adoption of reference anchors in our framework is also related to the concept activation vector in @xcite . ## conclusion we propose a framework to measure the political bias of llms. by examining the stance and framing of llm-generated content across various political topics, the research uncovers significant insights into the nature and extent of biases embedded in llms. findings reveal the variability of political perspectives held by llms, depending on the subject matter, and highlight the complex dynamics of how topics are presented and framed. this study not only sheds light on the multifaceted aspects of political bias in ai but also sets a precedent for future research aimed at mitigating such biases. through the open-sourcing of its codebase, we hope to contribute development of more equitable and reliable ai applications while addressing the ethical challenges posed by llms.