Aligning Large Language Models with Diverse Political Viewpoints (2024)

Abstract: Large language models such as ChatGPT exhibit striking political biases. If users query them about political information, they often take a normative stance. To overcome this, we align LLMs with diverse political viewpoints from 100,000 comments written by candidates running for national parliament in Switzerland. Models aligned with this data can generate more accurate political viewpoints from Swiss parties, compared to commercial models such as ChatGPT. We also propose a procedure to generate balanced overviews summarizing multiple viewpoints using such models. The replication package contains all code and data.

## introduction large language models (llms) have become very popular, with chat applications like chatgpt and gemini having hundreds of millions of active users combined. 1 one of the intended use cases is the retrieval of factual information (e.g., @xcite . interacting with chatbots can influence users' views @xcite and potentially influence behavior (e.g., @xcite . because of this, llms -if used as decision aids in high-stakes contexts such as shaping political views or votes -must return factually correct and unbiased statements. political bias is present in all first-generation llms @xcite . also, chatgpt is not impartial, as several recent papers have shown: chatgpt exhibits progressive, liberal, and proenvironmental biases @xcite @xcite . given these findings, @xcite ask: "what if chatgpt exhibits a political ideology 1 https://www.demandsage.com/chatgpt-statistics/ https://www.demandsage.com/google-gemini-statistics/ system: you are a helpful swiss policy advisor. you are asked a policy issue or question. you are in the political party p, and you reply in l. user: what's your opinion on the following issue or question: q that may pervade its synthetic responses and subtly influence its millions of unsuspecting users?" to address such concerns, we propose to align llms such that they can generate more diverse political viewpoints. inspired by early work with gpt models and conditional generation based on metadata @xcite , we align llms toward accurately reflecting diverse viewpoints from swiss parties. throughout the paper, we use prompts of the form shown in figure 1 : we ask an llm about its opinion on political issues while being in party p and speaking language l. 2 we align llms with data obtained from the swiss voting advice application smartvote: we have ca. 100,000 comments written by candidates running for national parliament in switzerland, coupled with meta-data, comprising 33 attributes such as party, language, canton, age, profession, favorite movies, and others. smartvote lets both user and candidates fill out a survey about stances toward various political issues. then, it matches candidates with the highest overlap to a user. this helps voters decide which candidates they could vote for. we took that survey with chatgpt. we find that chatgpt is most closely aligned with candidates from the green liberal party of switzerland, followed by other liberal parties. these results confirm its progressive and pro-environmental bias in the swiss case. a more exhaustive analysis is presented in appendix a, and we show anecdotal evidence of such biases in ## related work after llms became popular, it did not take long for social scientists to start investigating the political leaning of llms, specifically chatgpt (e.g., @xcite @xcite . all of them reported that chatgpt has political biases and a certain political leaning. relatedly, the nlp community also noticed political leanings of llms @xcite , and started to develop appropriate counter-measures. one approach is to explicitly align them with specific leanings: jiang et al. (2022) train "communitylm", an lm specifically aligned with a certain political leaning on a dedicated corpus -and the authors investigate the worldviews of such communities by probing the resulting llm. concurrent work fine-tunes an ensemble of such communitylms, and shows that these can be used as well to produce balanced overviews @xcite . lastly, future work to generate balanced overviews using our llms would profit from literature on summarizing subjective opinions @xcite or explicitly generating consensus statements @xcite . ## data we use the same data source as @xcite ): comments written by candidates running for national parliament in switzerland. the comments were written for and submitted to the voting advice application smartvote. this application helps voters determine which candidates or parties have political preferences similar to their own. prior to an election, candidates can report their stance on a short (30 questions) or long (75 questions) survey across various political issues. voters can take the same survey and are matched with candidates having the highest overlap (smartvote returns an ordered list of all candidates a voter can elect, ordered by survey overlap). the questions are drafted by a team of political scientists (for more details, see @xcite . these questions are yes-no questions, and smartvote allows candidates to submit comments which further explain their stance on an issue (these responses can be queried by users). we have 100,000 such comments for ca. 200 political questions across the last three national elections in switzerland. we use these comments and metadata to align models using conditional generation. smartvote is a popular service in switzerland: 85% of candidates running for elections in switzerland have a smartvote profile, and one in five voters consults smartvote before elections. thus, our data likely offers a rich overview of possible political stances in switzerland. we show more detailed dataset statistics in appendix b. ## methods in conditional generation, we want to generate text based on constraints or metadata (e.g., @xcite . for example, previous work generated news articles based on the attributes domain, date, authors and headline @xcite . alignment datasets usually contain triples of the form instruction, preferred choice and rejected choice. 3 we interpret conditional generation for alignment as sampling a comment toward a political issue q drafted by somebody from party p speaking language l as the preferred choice. for the rejected choice, we sample a comment for the same issue q in the same language l, but from a candidate who is in a different political party ¬p. we use reference-free monolithic preference optimization (orpo; @xcite as our alignment objective. we optimize the following joint loss taken directly from the orpo paper: we also experimented with direct policy optimization dpo @xcite following the recipe outlined in @xcite and rlhf @xcite . however, initial qualitative exploration has shown that the trained models did not generate satisfactory output. we use lora for all experiments but do not tune hyperparameters. we believe that it should be possible to get similar results with other alignment algorithms, but we have not explored that any further. in all our experiments, we used the transformer trl library @xcite and the 4bit quantized unsloth version of llama 3 8b models foot_2 (ai@meta, 2024) and fine-tuned all models using lora @xcite . for the supervised finetuning, we used the hyper-parameters outlined in @xcite , and for orpo alignment, we proceeded with the hyper-parameters outlined in @xcite . ## results we present four sets of results on our dataset's development and test split: qualitative evidence, diversity of generations, similarity between generated text and human references, and human evaluation. all our models use the prompt template shown in figure 1 . we present results for zero-shot settings (chat-gpt 3.5, gpt-4o, and llama 3), few-shot settings (chatgpt 3.5), llama 3 fine-tuned using direct supervision @xcite , and llama 3 aligned using orpo. ## discussion and conclusion if explicitly asked to produce political viewpoints from a certain party perspective, llm-generated text should accurately reflect the viewpoints of the party. we show that current llms fail to do so in a zero-shot setting in the swiss context. combining alignment and conditional generation substantially improves such generations, as we have shown qualitatively and quantitatively throughout this work. supervised fine-tuning of llama models and few-shot experiments result in similar performance, both beating the zero-shot setting. orpo-aligned models work best for this type of conditional generation, and we confirm this finding with a diverse set of results. such aligned models have practical use cases beyond accurately presenting specific party preferences. we can also use such models to generate a balanced overview of viewpoints toward a specific issue. we show a simple algorithm to do so in we first generate a stance to a policy issue for all parties, and then let gpt-4o provide a summary of these stances. for illustration, we run this procedure for the issue should the state do more to promote equal educational opportunities? figure 9 shows the overview synthesized from the orpoaligned llama 3 model is more balanced and accurate than the one synthesized from the responses obtained in a zero-shot setting. generating text that contains political views has major implications. llms have the potential to shift attitudes and behavior. if they do that in the political domain, this might influence elections, one of the most important decision-making processes in democracies @xcite . our recommendations are two-fold: first, further research is necessary to explore the promises and pitfalls of llms delivering political information or offering explicit voting advice. second, there is a need for both societal and scientific debates on the role of llms and ai in democratic processes. @xcite asked what if chatgpt exhibited a political ideology that may pervade its synthetic responses and subtly influence its unsuspecting users? our work speaks to this question. we see a number of strategies going forward: (1) llms would always refuse to answer anything related to shaping political beliefs and take sincere political impartiality as an alignment goal. (2) llms would always produce broad overviews of political issues (as produced by our methods in figure 9 ). our work might facilitate creating appropriate datasets for this. (3) llms would explicitly produce text that is aligned with a certain political leaning. in this case, however, the provider of an llm would be fully transparent about this, and/or the user should be able to fully control what ideology llm-generated text should be aligned with. aligning models explicitly with party preferences using conditional generation, as presented in this work, is one way toward such llms.