Analyzing Political Bias in LLM s via Target-Oriented Sentiment Classification (2025)

Abstract: Political biases encoded by LLMs might have detrimental effects on downstream applications. Existing bias analysis methods rely on small-size intermediate tasks (questionnaire answering or political content generation) and rely on the LLMs themselves for analysis, thus propagating bias. We propose a new approach leveraging the observation that LLM sentiment predictions vary with the target entity in the same sentence. We define an entropy-based inconsistency metric to encode this prediction variability. We insert 1319 demographically and politically diverse politician names in 450 political sentences and predict target-oriented sentiment using seven models in six widely spoken languages. We observe inconsistencies in all tested combinations and aggregate them in a statistically robust analysis at different granularity levels. We observe positive and negative bias toward left and far-right politicians and positive correlations between politicians with similar alignment. Bias intensity is higher for Western languages than for others. Larger models exhibit stronger and more consistent biases and reduce discrepancies between similar languages. We partially mitigate LLM unreliability in target-oriented sentiment classification (TSC) by replacing politician names with fictional but plausible counterparts. The complete code, the data, and all analyses will be made public to enable reproducibility.

## introduction large language models (llms) have reshaped the ai landscape due to their versatility and ease of use. beyond their generation capabilities, they greatly simplified nlp tasks that previously required strong expertise and specific development @xcite . however, multiple biases, including demographic @xcite and political ones @xcite , question their use in socially important real-life applications, including the moderation of political discussions on social media platforms @xcite or the automatic analysis of media coverage of political events or discourse @xcite . addressing these tasks requires components such as hate speech detection, stance detection, or targetoriented sentiment analysis. these components entail subjectiveness and are bias-prone, making the understanding and mitigation of this phenomenon crucial for ensuring accuracy, fairness, and reliability. existing llm political bias studies fall into two main categories. questionnaire-based works @xcite repurpose methodologies, such as the political compass @xcite , to infer llm political leanings based on answers to predefined questions. generation-based methods @xcite @xcite prompt llms to generate political content and deploy sentiment or stance detection to infer ideological alignments. while interesting, these approaches have significant limitations. first, the number of interactions with llms is usually reduced, leading to limited statistical power. second, they depend on the prompt wording, with high result variability occurring even after minimal changes. third, they provide limited actionable insights about llm use in downstream applications. informed by these limitations, we propose a new political bias analysis framework leveraging llm prediction inconsistency in target-oriented sentiment classification. assume we apply tsc to: "x has limited means but contributed to solving this conflict." a reliable classifier should consistently predict the same sentiment for any name replacing x . we define an entropy-based inconsistency metric to quantify this undesired prediction variability. to verify this hypothesis, we use 450 political sentences from mad-tsc @xcite , an existing dataset, and 1319 politicians representing diversified demographics and eight political alignments. we vary the names in the sentences, vary the models, and use sentence translations in six languages to obtain nearly 25 million data points and robustly analyze llms' political bias. figure 1 analyzes political bias in llms using tsc accuracy and inconsistency metrics for 42 model-language combinations. without mitigation, name-related inconsistency is significant for all models due to prediction variability when changing the politician names. confirming previous results @xcite , tsc accuracy is better for larger models. section 4 provides detailed results for political alignments, languages, models, and politicians. we notably: (1) confirm positive and negative biases toward the left and the right political alignments, (2) find a positive correlation for politicians with a similar alignment, (3) show that bias intensity varies with the language, and (4) examine the effect of model size on biases. replacing politicians with fictional but plausible names reduces inconsistency and slightly improves accuracy in most cases. this replacement provides a simple yet effective way to reduce tsc biases. however, a degree of inconsistency subsists, questioning the reliability of llm usage for nlp tasks involving subjectivity. ## related work sentiment analysis is a prominent nlp application @xcite used for analyzing media biases @xcite , predicting stock market @xcite , and detecting hate speech @xcite . importantly, llms exhibit societal, cultural, and political biases, which manifest in generated text @xcite @xcite . these biases complicate sentiment analysis, particularly in politically sensitive contexts, as they can skew model predictions and reinforce preexisting biases @xcite . we use these observations to highlight the predictions' unreliability and use them for political bias analysis. studies investigating biases in llms use two main approaches. the first, questionnaires and controlled tasks, employs structured prompts or multiple-choice questions to measure biases. for instance, @xcite use simulated sociological surveys to quantify cultural alignment, while @xcite leverage public opinion polls to evaluate alignment with demographic groups. @xcite and @xcite utilize political compass tests and voting advice applications to assess political leanings. these methods provide easily quantifiable and reproducible results. a key advantage of questionnairebased approaches is their ability to systematically evaluate biases across diverse contexts, particularly when varying prompting strategies @xcite . however, they have significant limitations: constrained designs may not generalize to real-world applications @xcite , and reliance on forced multiple-choice formats can misrepresent model behavior as well as not reflect real-world use cases @xcite . importantly, the questionnaire sizes are usually reduced @xcite , making a robust statistical analysis of results difficult. the second approach, open-ended generation tasks, involves prompting llms to generate essays, poems, or lengthy texts, for a less constrained analysis of biases. for example, @xcite apply sentiment analysis to political essays generated by gpt-3, revealing a moderate left-leaning bias. similarly, @xcite examine limericks generated by chatgpt, uncovering patterns of bias favoring liberal politicians. these methods provide insights into how biases manifest in free-form text, but they face significant challenges. open-ended generation offers a valuable complement to questionnairebased approaches, particularly for capturing biases in context-rich scenarios @xcite . however, quantifying biases in open-ended outputs is inherently difficult. these studies often rely on word distribution analysis, sentiment or stance detection, or evaluation by llms as judges @xcite , which are biasprone themselves @xcite . moreover, the lack of control over output formats complicates bias assessment, as traditional algorithmic fairness metrics (e.g., equalized odds, demographic parity) are not directly applicable @xcite . sentiment analysis offers a complementary practical middle ground between structured questionnaires and open-ended text generation. unlike rigid multiple-choice formats, it aligns with real-world language use while providing measurable bias assessments. the work closest to ours is @xcite . the authors analyze llm-generated descriptions of public figures, using llms as evaluators, a strategy that may compound biases. we take the opposite approach and leverage the llm-based sentiment classification biases to elicit biases. the focus on fine-grained entities and the large number of data points available enable a flexible, nuanced, and comprehensive bias analysis. llms are multilingual, and understanding linguistic and cultural variability is important for their deployment @xcite . existing studies reveal that llms often exhibit misaligned cultural and moral biases @xcite , with inconsistent performance and exposure biases across languages @xcite . we contribute to this multilingual analysis by comparing llms prompted in six languages. ## methodology we systematically analyze llms' political bias via a task-specific evaluation framework based on target-oriented sentiment classification. we define an inconsistency metric to quantify the tsc prediction variability. our approach involves constructing a dataset of political entities and sentences, translating them into six languages, and evaluating multiple llms. we diversify personal and political attributes to control confounders such as demographics and popularity. we also introduce a control group of synthetic entities to isolate the effects of entity-oriented biases encoded in llms. our methodology is designed to provide statistically robust insights while remaining scalable and applicable to real-world scenarios. ## results sentiment analysis across models and languages reveals systematic biases based on political alignment. figure 2 shows that models consistently favor left, center-left, and centrist politicians while assigning more negative sentiment to right and farright politicians. these results are aligned with previous findings @xcite . interestingly, while biases occur in all languages, they are more pronounced in english, spanish, and french than in russian, arabic, and chinese. a statistical test confirms the statistical significance in all languages (see section j of the appendix). the consistency of these trends across models suggests that they are not random but may be caused by llm training data biases. in the following paragraphs, we delve deeper into the influence of individual languages and models. larger models exhibit stronger biases that are more consistent across languages. figure 2 shows larger language models display more pronounced biases than their smaller counterparts. qwen-72b and llama 3-70b assign more negative sentiment to far-right and right-leaning politicians while showing stronger positive sentiment for center-right, center-left, and left-leaning politicians than qwen-7b and llama 3-8b. the training corpora are unavailable, but we can safely assume they are similar within the same family of models. these differences are noticeable across western and non-western languages when analyzing individual model-language combinations in figure 3 . llama 3-8b in arabic does not exhibit a significant negative bias toward far-right politicians, the most consistent bias observed globally. in contrast, llama 3-70b exhibits a negative bias toward far-right figures. in english, biases also become more pronounced in the larger model, with stronger deviations in sentiment scores across political alignments. in addition to bias intensity, we study its variation across languages in figure 4 (a) by computing the correlations between sentiment predictions qwen-7b and qwen-72b. the results are obtained by averaging the jaccard similarity indexes of individual entities for each model-language pair. they show that more similar tsc predictions are obtained with larger models. these results suggest that as models scale up, their positive and negative biases tend to intensify but also become more consistent across languages. this is possibly due to larger llms' increased capacity to internalize implicit patterns from training data. aya-expanse-32b exhibits stronger biases in non-western languages. figure 3 (b) shows aya-expanse-32b displays higher biases for arabic. while the figure specifically highlights arabic, this pattern holds consistently for russian and chinese. this is the only model specifically trained for multilingual tasks in our set of models. an explanation comes from aya's training methodology, which relies heavily on translating data from high-resource languages (e.g., english) to others. figure 4 also highlights that aya's predictions are more similar across languages than those of qwen models. while aya's multilingual training is effective for cross-lingual generalization, it may also lead to a more uniform propagation of biases across diverse linguistic contexts. these findings underscore the trade-offs in multilingual model design and the need to consider how training data and methodologies influence bias amplification carefully. llms have a left-libertarian bias. using parl-gov data, which assigns parties a progressiveconservative and authoritarian-libertarian score, we map politicians onto a 10×10 political compass grid. we compute the average sentiment score of all associated politicians for each square, then average the results over neighboring squares to highlight broader trends. as shown in figure 5 , the sentiment distribution for gpt-4o-mini in english and chinese aligns with findings from previous studies @xcite , reinforcing the conclusion that chatgpt exhibits a left-libertarian bias. sentiment analysis across the political compass reveals a consistent pattern: politicians associated with left-libertarian positions receive more positive sentiment, while those aligned with right-authoritarian positions receive more negative sentiment. additionally, our approach enables a more fine-grained analysis, revealing that centrist parties also tend to receive higher sentiment scores. this trend holds across all languages and models, further supporting the robustness of these patterns. ## conclusion understanding political bias in llms is crucial as their real-world use grows. in this study, we present a novel approach using target-oriented sentiment inconsistencies to quantify these biases. we systematically varied political entities in controlled sentence templates to obtain a massive tsc prediction dataset and aggregate them to analyze different bias facets. our results have several important implications. first, we contribute to studying the llms' political biases by introducing a flexible framework that builds on tsc, a fine-grained nlp task. this modeling enables a flexible aggregation of results, making our approach complementary to existing ones. we emphasize the roles of languages and llm sizes and analyze political alignments and individual entities. our results highlight systematic political biases in llms, with more positive sentiment assigned to left-leaning politicians and more negative sentiment to far-right figures. bias intensity is stronger in english, french, and spanish but exists across all tested languages. we find that larger models accentuate bias and increase similarities between predictions in different languages. these findings support previous work @xcite showing that native language is not the best choice for probing regional knowledge when alternatives exist. we also highlight similarities between the llms representations of individual politicians with similar alignments. second, the prediction inconsistencies significantly affect the llm use for target-oriented sentiment classification. this finding, added to the fact that small models are still competitive in tsc (bucher and martini, 2024), questions the llm robustness for this particular task despite their easy deployment. more generally, the observed inconsistency leads us to recommend caution when employing llm for bias-prone tasks, particularly those involving subjectivity. third, these biases can be mitigated to a certain extent. we tested a post-training strategy by replacing politician names with more neutral alternatives. this strategy reduces sentiment inconsistencies and also brings small accuracy gains. however, addressing bias in one task does not guarantee neutrality in others. future work should explore how bias manifests in different applications and across a broader range of entities. equally important, llm creators could use our findings to integrate bias mitigation in subjective tasks during training. finally, the proposed strategy is model-agnostic and adaptable to other languages. we will release the code and data to contribute to systematic and reproducible political bias evaluation. 0 1 2 3 4 5 6 7 8 9 10 gpt-4o-mini (zho) 0 1 2 3 4 5 6 7 8 9 10 economic policy 0 1 2 3 4 5 6 7 8 9 10 social policy gpt-4o-mini (eng) figure 5: political compasses showing sentiment bias for gpt-4o-mini in english (left) and chinese (right). the y-axis represents the social policy spectrum (0: socially progressive, 10: socially conservative), and the x-axis represents the economic policy spectrum (0: fiscally progressive, 10: fiscally conservative). parties are positioned using parlgov data, with colors indicating the average sentiment of affiliated politicians (red: negative, green: positive). blank squares denote no corresponding party. left-libertarian parties consistently receive positive sentiment, while right-authoritarian parties show negative sentiment, highlighting consistent ideological biases across languages. data representativeness. our dataset includes a diverse set of politicians but remains dominated by figures frequently mentioned in western media, which make up the majority of cc-news. this modeling choice is intentional since political bias occurs for entities often encountered in the training d o n a ld t r u m p v la d im ir p u t in n ic o lá s m a d u r o ja ir b o ls o n a r o v ik t o r o r b á n b e n ja m in n e t a n y a h u g e o r g e w . b u s h m ik e p e n c e je a n -l u c m é le n c h o n je r e m y c o r b y n jo e b id e n e m m a n u e l m a c r o n k a m a la h a r r is o la f s c h o lz p e d r o s á n c h e z a n g e la m e r k e l u r s u la v o n d e r l e y e n t h e r e s a m a y similarity 75 80 85 90 95 100 entity similarity (english) bias mitigation and trade-offs. including control entities (non-political names) helps distinguish biases related to political attributes and effectively reduces entity-related bias in our context. however, bias subsists to some degree, and the method could be further improved by checking that control entities' first and last names are not politically figure 7 : average sentiment scores for languages and for models using a set of fake names reflecting non-political attributes (gender, birth year, and country of origin) of our original entities. shaded areas represent 95% confidence intervals. the results show that the biases for political entities largely disappear, although some anomalies persist. these anomalies are attributed to llms rating female names higher (often corresponding to left-leaning entities) and russian or non-western names lower (often corresponding to far-left or far-right leaning entities). this suggests that tsc biases are primarily driven by political associations rather than non-political attributes. evolving nature of biases. the biases observed reflect the state of political text included in the llms' training dataset. for instance, the sentiment expressed about a politician's or a political alignment's actions can vary drastically over time. the reported findings will probably evolve as models are updated with new data and political landscapes shift, and future research should periodically reassess bias patterns to account for these changes. model and language representativeness. we experimented with seven models and six languages representative of current llms, enabling meaningful comparisons across these two axes. the results highlight bias for all model-language combinations, with some variability for specific combinations, allowing us to present robust findings. the study could be further generalized by including more models and languages, but such an extension is challenging. first, we performed initial tests with more open-source models but discarded them due to low accuracy and/or the impossibility of obtaining usable inferences. second, we tested many entity-sentence combinations, and the cost of running complete experiments with multiple models, particularly in a few-shot learning setting, exceeded our financial means. we selected six widely used languages from linguistic groups representing different geopolitical alignments and found interesting differences between models in these languages. integrating other languages would further improve the results' robustness. it would be particularly interesting to test biases for low-resourced languages.