P ol B i X : Detecting LLM s’ Political Bias in Fact-Checking through X -phemisms (2025)

Abstract: Large Language Models are increasingly used in applications requiring objective assessment, which could be compromised by political bias. Many studies found preferences for left-leaning positions in LLMs, but downstream effects on tasks like fact-checking remain underexplored. In this study, we systematically investigate political bias through exchanging words with euphemisms or dysphemisms in German claims. We construct minimal pairs of factually equivalent claims that differ in political connotation, to assess the consistency of LLMs in classifying them as true or false. We evaluate six LLMs and find that, more than political leaning, the presence of judgmental words significantly influences truthfulness assessment. While a few models show tendencies of political bias, this is not mitigated by explicitly calling for objectivism in prompts. Warning: This paper contains content that may be offensive or upsetting.

## introduction large language models (llms) are becoming increasingly integrated into everyday applications, raising concerns about their potential biases @xcite . the research field received increasing attention, as studies have shown that llms often lean towards the political left @xcite @xcite . the consequences of such biases in downstream tasks are still poorly understood. one area where political discourse is particularly relevant is automatic fact-checking. while the potential of llms in mitigating misinformation and disinformation is acknowledged, the consequences of their use in truth-assessing tasks remain ethically and epistemologically problematic @xcite . a person's understanding of truth is shaped by their political position (van der linden et al., figure 1 : data example where "social contributions" refer to the original word and "mandatory contributions" refer to a right-shifted dysphemism. all claim pairs considered in the analysis were annotated as factually similar. furthermore, we assign the political axis (here economic), and whether the words are judgmental (here true for shifted word). an llm may evaluate the fact differently based on the specified word. 2020). more specifically, humans are subject to confirmation biases, which tends to lead them to perceive statements that align with their political orientation as more credible @xcite . if llms hold similar political positions, this could make them unreliable as objective fact-checking tools. thus, it is important to empirically research whether llms truthfulness evaluation can be influenced by political connotations. politicians strategically use euphemisms and dysphemisms (xphemisms) to shape the perceptions of their audience as they evoke positive and negative emotions @xcite . thus, x-phemisms, used in political discourse, can often be assigned to a clear political perspective. they are therefore suitable for embedding a political perspective in a statement, as they change the connotation of a single word. employing minimal pairs is a highly controlled method for algorithmic bias detection @xcite . individual words are exchanged according to the opposite demographic factor of a person, e.g. "she" with "he" in case of gender, to examine the words individual influence on a task @xcite . in the case of political bias, however, these experiments are more difficult as identifiers for political leaning are not as well-defined and interchangeable. for truthfulness classification we assume that it is not necessary to preserve the entire content of the sentence, but rather the fact it conveys. we use x-phemisms to generate our minimal pairs. therefore, we deliberately insert these x-phemisms into the text that, while politically charged, remain factually as close to the original as possible, see figure 1 for an example. we aim to answer the following research questions: our results show that models are significantly influenced by judgmental words in their truthfulness assessment. when grouping x-phemisms by political leaning, we observe that in some cases llms assign different truthfulness values depending on the leaning, indicating the presence of political bias. furthermore, the results indicate that emphasising objectivity in the prompt does not effectively reduce bias. these results highlight the need for further research into political biases of llms, their sources and mitigation strategies. ## methodology and dataset we investigate how llms assess the truthfulness of factually equivalent claims when presented with different political framing. by systematically altering individual words, we achieve a more precise, objective, and word-level analysis of bias compared to previous methods. our approach is based on principles from framing theory and discourse theory @xcite . framing theory describes in general, that issues "can be viewed from a variety of perspectives and be construed as having implications for multiple values" @xcite ). foucault's discourse theory ties the concept to the perception of truth, arguing that what we consider "true" is not objective, but shaped by discourses, power relations, and social practices. when a sentence uses left-or rightleaning language, its position within the discourse shifts, and so does its perceived credibility among different groups. for instance, the term "economic refugee" is more commonly associated with rightwing discourse, while "person seeking protection" aligns with left-wing framing. the example shows that words that describe the same situation can construct entirely different realities @xcite . to frame facts differently, we build word pairs as denotative similar nouns with different connotations, favouring political perspectives. we use german as language for the dataset which has the advantage of containing and frequently using compound nouns, so a lot of information is captured in single words. the complete process can be seen in figure 2 . ## experiments we classify the truthfulness of base and shifted claims with large language models. to determine the influence of political leaning and judgmental words on these classifications we build a statistical model that enables to see which factor is significant. ## results for openai o4-mini, when prompted with the simple prompt, the p-value of 0.0035 on the social axes is significant and when prompted with the objective prompt the p-value 0.0035 on the economic axes is significant. the coefficients for both prompts are positive for the economic axis and negative for the social axis. this means that ope-nai o4-mini estimated left-leaning claims as more truthful in social topics and right-leaning claims in economic topics. the analysis for llama 3.3 70b shows for the simple prompt a significance when considering all claims (@xmath0.0044, @xmath1.2282) and for the objective prompt when considering economic 2 ). the right subplot shows the impact of political leaning. we observe how the average truthfulness changes when replacing the base word with a more left-leaning word and a more right-leaning word. a decrease of predicted truthfulness can be detected in both directions, whereas a stark difference between left and right is not apparent. overall, the impact of judgmental words on average truthfulness is bigger than political leaning. claims (@xmath2.0162, @xmath3.2901). in this case, the sign of the coefficients flips from being negative for the simple prompt in the combined dimension and being positive for the objective prompt in the economic dimensions. this means that calling for objectivity in the prompt changed the model to favour right-leaning claims on the economic axis instead of left-leaning claims on the combined axis. the third model with significant political shift results is llama4 maverick. evaluating only the simple prompt, the social axis, and the combined testing resulted in p-values of 0.0036 and 0.0002, with negative (left-leaning) coefficients. figure 5 shows the deviations in the average of truthfulness ratings for each experimental setting, regarding political leaning and presence of judgment. openai's gpt-4o mini model provides the most optimistic evaluations, with a mean truthfulness score of 0.76 in the baseline condition. in contrast, openai o4-mini stands out as the most pessimistic, with an average score of 0.27. in case of a leaning difference, for all llms except ope-nai o4-mini, we observe that base claims of leftshifted minimal pairs are, on average, rated as less truthful than the base claims of the right-shifted minimal pairs. in case of the presence of judgement, we observe that mostly the truthfulness of base claims for minimal pairs without judgement is higher compared to minimal pairs with judgement. in figure 6 , we plot the coefficients for both axes in a two-dimensional scatter plot. the data does not reveal a consistent pattern of objective prompts producing more centered estimations relative to simple prompts. in response to research question 1, our findings indicate that political connotation only partially influences llms' perception of truth and that this effect varies depending on the prompt used. however, we consistently observe a significant negative impact of judgmental language. for research question 2, no big difference in impact of political leaning can be detected, as this signal was only present in few cases. furthermore, the impact of judgmental words remain present in the results of the objective prompt, indicating that explicitly prompting for objectivism does not change the outcome. llm prompt axis judgment shift gpt-4o mini simple both * social * economic * advanced both * social * economic * -openai o4-mini simple both * social * * economic * advanced both * social -economic * * mixtral 8x22b simple both * social * economic * advanced both * social * economic * -deepseek r1 simple both * social * economic * advanced both * social * economic * -llama 3.3 70b simple both * * social * economic * advanced both * social * economic * * llama4 maverick simple both * * social * * economic * - ## discussion in this study, we present a novel german-language dataset that combines fact-checking tasks with political bias detection. given the complexity of the topic, a variety of approaches are conceivable. for our methodology, we selected a minimalistic strategy: replacing targeted words in short, politically charged statements. this method provides an initial, focused perspective on how political language may influence the truthfulness assessments of llms. however, these results should not be generalized beyond the scope of this dataset. instead, they serve as a preliminary indication of potential political tendencies in llm behavior under specific prompt and phrasing conditions. one of the most unexpected findings of our study is the significant role judgmental words plays in how llms assess truthfulness. words with judgmental undertones appear to skew the models' truth evaluations more than the actual political leaning of a statement. this may be because emotionally loaded language is less frequent in factually reliable texts. the political leaning of a statement had no consistent significant effect on the binary classification of a claim as "true" or "false". three of six models showed significant tendencies varying in topical dimension and leaning direction. shifted claims resulted in lower average truthfulness scores across all tested models. this could be explained by three factors. first, the base claims might be present in the training data of the models. second, in general politically connotated words are more likely to be associated with misinformation independent of their political leaning. third, when changing a word in a true sentence it is more likely to become false compared to a false claim becoming true. the result from the llama4 maverick can be interpreted optimistically. while llama 3.3 70b when paired with the objective prompt, didn't re-duce left-leaning political bias but changed towards the opposite direction, llama4 maverick instead refused to answer because of insufficient context. such refusals may reflect a more cautious and context-aware reasoning process. we caution against over-interpreting the results from our experiment. our work does not show that llms do not contain political biases, whether from specific models or in general. we only show that political shift is not the major factor in the experiments on our dataset and offer an alternative explanation for some experiments where political bias is detected. ## conclusion this study provides an initial exploration of how llms evaluate politically charged statements in german. we found that judgmental wording has a stronger impact on truthfulness predictions than political leaning. for three of the six models, we observed that their ratings of statements varied depending on the topic and the ideological leaning of the shifted words. openai's o4-mini tends to favour economically right and socially libertarian viewpoints, llama4 maverick exhibit a left-leaning bias, and llama 3.3 70b shows varying bias directions based on the prompt. while llama4 maverick shows a promising step toward mitigating bias, our results also highlight the limitations of current models in reliably separating political tone from factual content. as such, this dataset should not be seen as a benchmark, but as a starting point for deeper investigations into llm behaviour. future work should focus on broader datasets and more diverse prompts to better understand how political and emotional cues influence truthfulness assessments. building on this work, it would be interesting to research the extent to which a model's perception of sentiment influences its political bias.