On the Relationship between Truth and Political Bias in Language Models (2024)

Abstract: Language model alignment research often attempts to ensure that models are not only helpful and harmless, but also truthful and unbiased. However, optimizing these objectives simultaneously can obscure how improving one aspect might impact the others. In this work, we focus on analyzing the relationship between two concepts essential in both language model alignment and political science: truthfulness and political bias. We train reward models on various popular truthfulness datasets and subsequently evaluate their political bias. Our findings reveal that optimizing reward models for truthfulness on these datasets tends to result in a left-leaning political bias. We also find that existing open-source reward models (i.e., those trained on standard human preference datasets) already show a similar bias and that the bias is larger for larger models. These results raise important questions about the datasets used to represent truthfulness, potential limitations of aligning models to be both truthful and politically unbiased, and what language models capture about the relationship between truth and politics.

## introduction the political bias of large language models (llms) has been the subject of much recent research @xcite . @xcite found that base models tend to be more right-leaning initially, but shift towards a leftleaning stance after fine-tuning, suggesting that the alignment process may influence the models' political bias. however, since alignment datasets often simultaneously target helpfulness, harmlessness, and truthfulness @xcite , it is difficult to determine which of these objectives, if any, might be responsible for this shift in political bias. our interest in the relationship between truthfulness and political bias is motivated by findings in political science of partisan differences in susceptibility to misinformation @xcite and trust in science @xcite . lower levels of trust by some political groups may be exacerbated by political bias in language models if the groups believe these models are antithetical to their values. as llms become more widely deployed, exploring such biases and ways to remediate them becomes valuable. we begin by testing whether vanilla open-source reward models -i.e., those fine-tuned on standard human preference datasets -show political bias, aiming to identify parts of the alignment pipeline contributing to the left-leaning bias suggested by prior work @xcite . we then train a new set of reward models (rms) on several datasets representing different notions of truthfulness, such as everyday and scientific facts, and assess their political bias. finally, we analyze which topics exhibit the greatest bias. the main findings are as follows: our results suggest that even training on supposedly objective datasets can lead to unforeseen bias. we also release a dataset of 13,855 left-leaning and right-leaning partisan statements matched on topic for use by the community foot_0 . ## related work we briefly cover three areas that our work relates to: ai alignment, llm truthfulness, and political bias in llms. ## experimental setup truthfulness datasets we use several datasets corresponding to different notions of factuality to train our reward models: truthfulqa @xcite , fever @xcite , sciq @xcite , and a dataset we created of 4,000 basic llm-generated facts and falsehoods about the world, using @xcite and gemini @xcite . (see appendix b for details regarding how we generated, validated and audited this last dataset.) fever is based on facts about entities extracted from wikipedia. sciq is based on scientific knowledge. truthfulqa covers a variety of topics and was created with the goal of eliciting untruthful completions from llms. finally, our generated data aimed to create the most obvious facts and falsehoods. thus, our datasets span facts about entities (fever), scientific facts (sciq), a diverse mix of difficult questions (truth-fulqa), and common sense facts (our generated data). to make the data suitable for reward modeling, which expects paired samples, we match a correct response to a query with an incorrect response for truthfulqa, fever, and sciq. for the generated dataset, we create random pairs of true and false statements. for datasets with multiple-choice options, we ensure that each question appears exclusively in either training or test. ## bias in vanilla reward models we first examine whether vanilla open-source reward models exhibit political bias. as discussed in section 3, we evaluate with reward models from raft, openassistant and ultrarm. we run inference with these models on the twinviews statements and find that all models show a left-leaning political bias, as depicted in figure 1 . notably, larger models also show greater bias, an example of inverse scaling @xcite . however, one caveat is that the datasets/training methods are different across these reward models. the results suggest that at least part of the left-leaning political bias observed in the literature @xcite could be due to biases introduced in rewardmodel training, which we believe is a new finding. given that fine-tuning datasets are intended to be objective, these findings were unexpected. in section 5.2, we use an n-gram baseline (shown in the rightmost pane of figure 2 ) to consider another potential source of bias: stylistic features spuriously correlated with both truth status and political orientation. we find little support for this idea either, however, leaving the origin of the political bias shown in figure 2 in need of further research. ## bias across topics because both vanilla and "truthful" reward models show political bias, we used regression analysis to examine which topics or political issues exhibit the most bias. for both sets of models, we regressed the reward assigned to a twinviews political statement on several predictors: the model, 7 the topic, the statement's political lean, and the topic/politicallean interaction. all models are linear regressions. our results are shown in ## conclusion we investigated political biases in reward models, both vanilla open-source reward models and "truthful" reward models, and found a persistent leftleaning political bias across nearly all these models. this result is particularly surprising given the use of datasets designed to capture objective truth. moreover, the size of the bias increases with model scale, in contrast to the usual pattern of improving capabilities. for the "truthful" models, we considered and attempted to rule out two explanations: explicit political content in truthfulness datasets and spurious relationships between truthfulness and stylistic features. identifying the source of this bias is a promising direction for future research, as well as understanding whether optimizing for truth leads to more or less political bias than other objectives. more generally, this work connects to the increasing politicization of scientific facts, such as climate change @xcite , and the problem of "truth decay" @xcite in the political sphere, which sit at the intersection of truth and politics. finally, our results suggest a potential tension in achieving both truthful and unbiased models which has important implications for llm alignment. we hope these initial findings will encourage further investigation into the relationship between truthfulness and political bias in language models.