Hidden Persuaders: LLM s’ Political Leaning and Their Influence on Voters (2024)

Abstract: Do LLMs have political leanings and are LLMs able to shift our political views? This paper explores these questions in the context of the 2024 U.S. presidential election. Through a voting simulation, we demonstrate 18 open-weight and closed-source LLMs’ political preference for Biden over Trump. We show how Biden-leaning becomes more pronounced in instruction-tuned and reinforced models compared to their base versions by analyzing their responses to political questions related to the two nominees. We further explore the potential impact of LLMs on voter choice by recruiting 935 U.S. registered voters. Participants interacted with LLMs (Claude-3, Llama-3, and GPT-4) over five exchanges. Intriguingly, although LLMs were not asked to persuade users to support Biden, about 20% of Trump supporters reduced their support for Trump after LLM interaction. This result is noteworthy given that many studies on the persuasiveness of political campaigns have shown minimal effects in presidential elections. Many users also expressed a desire for further interaction with LLMs on political subjects. Further research on how LLMs affect users’ political views is required, as their use becomes more widespread.

## introduction in the pursuit of developing safe artificial intelligence (ai), creating unbiased ai systems has become a critical goal. it has been shown that many ai technologies, including large language models (llms), exhibit measurable left-wing leanings @xcite @xcite . given growing llm applications in political discourse @xcite , will these models (un)intentionally influence endusers, yielding substantial societal consequences? this question remains largely unanswered. our study addresses this question by examining llms' political leanings foot_0 and their potential societal impact in the context of the upcoming 2024 u.s. presidential election. the election had biden and trump as the presumptive nominees for the democratic and republican parties through july 21, 2024 @xcite . 2 as the election date approaches, the potential for llms to have (un)intended effects on the election has raised many concerns @xcite . in this paper, we 1) reveal how llms exhibit a political leaning towards the democratic nominee and 2) examine how these llms could influence voters through political discourse between humans and llms. first, in section 3, we simulate presidential election voting between the two candidates across 18 open-weight and closed-source models, with each model run 100 times. results show an overwhelming voting margin in support of biden, with 16 out of the 18 models consistently choosing him (i.e., 100% biden vote). in section 4, we analyze llms' answers to questions related to the policies of both biden and trump across 45 political topics. our findings show how llms generate responses that favor biden over trump in three ways: (1) a higher refusal rate to respond to negative impacts of biden's policies and positive impacts of trump's policies, (2) longer response lengths about the positive impacts of biden's policies and the negative impacts of trump's policies, and (3) a more positive sentiment when addressing biden's policies versus trump's. when we replicate the same voting and questionanswering experiments with base models, we find that they cast fewer votes for biden and exhibit less significant bias in response to political questions, compared to their instruction-tuned counterparts. this finding suggests that human instruction posttraining, including reinforcement learning from human feedback, amplified the political leaning appearing in llms' outputs. moving to a more interactive and realistic scenario, section 5 investigates how llms manifest political leanings during human-llm interactions. given other characteristics of llms, such as their propensity for user adaptation and sycophancy @xcite , we were uncertain whether they would exhibit a consistent pro-biden view during interactions. another different question is whether llms will shift humans' voting choices via their conversation. to explore these questions, we conducted a user experiment in which u.s. registered voters engaged in one-on-one discussions with one of three popular llms (i.e., claude-3-opus, llama-3-70b, and gpt-4-turbo). we found that these three llms consistently presented their pro-biden views during conversations with human subjects, regardless of the participants' initial political stance. moreover, these llms significantly affected participants' voting choices by increasing the participants' leaning towards biden following their interaction. specifically, nearly 20% of initial trump supporters decreased their trump support, with the most extreme case showing a complete reversal (i.e., from fully trump-leaning to fully biden-leaning). 24% of our initial neutral participants shifted to support biden, while initial biden supporters showed no significant change. as a result, the simulated vote margin in our sample widened from 0.7% to 4.6%. this effect is politically meaningful, given that vote margins are typically very narrow in realworld presidential elections @xcite . moreover, the effect could represent a lower-bound of relevant influence, considering that participants got exposed to only five exchanges. many participants expressed enjoyment and a desire to extend their conversation with llms on political topics after the experiment, including many whose leanings changed. this would facilitate longer political interactions with llms in the wild that might induce a more pronounced impact on human voting stances. ## us presidential election among llms we start by examining the political stances of 18 llms regarding the two 2024 u.s. presidential nominees by simulating and collecting election votes for each model 100 times. results are listed in simulation results demonstrate overwhelming votes for biden across all tested llms. with the exception of gemini pro 1.0 and alpaca, all models voted for biden in 100 out of 100 rounds. gemini pro voted for biden 74 times, while alpaca voted for biden in 84 out of 100 trials. we also observe a difference in the strength of biden-leaning tendencies between the instruction-tuned models and their base versions. the base models of llama-3-70b-chat foot_4 and mixtral-8×7b-instruct made pro-biden decisions less often compared to their instructiontuned versions with the same temperature, casting 15 and 53 out of 100 votes for trump, respectively. ## conclusion we identify a notable leaning toward biden in 18 open-weight and closed-source llms across various scenarios: voting behavior, response to political questions, and interaction with humans. in particular, greater biden-leaning of instruction-tuned models is observed compared to their base versions, which suggests that current post-training processes amplify the manifestation of political leaning in their responses. we further demonstrate that llms could significantly shift people's voting stance toward biden through human-llm political conversation. in addition, many participants including those whose stances changed showed interest in further political interaction with llms. lastly, the generalizability of our reported findings beyond the 2024 u.s. presidential setting and the mechanisms by which voters' stances are changed require further research.