Llama meets EU : Investigating the E uropean political spectrum through the lens of LLM s (2024)

Abstract: Instruction-finetuned Large Language Models inherit clear political leanings that have been shown to influence downstream task performance. We expand this line of research beyond the two-party system in the US and audit Llama Chat in the context of EU politics in various settings to analyze the model’s political knowledge and its ability to reason in context. We adapt, i.e., further fine-tune, Llama Chat on speeches of individual euro-parties from debates in the European Parliament to reevaluate its political leaning based on the EUandI questionnaire. Llama Chat shows considerable knowledge of national parties’ positions and is capable of reasoning in context. The adapted, party-specific, models are substantially re-aligned towards respective positions which we see as a starting point for using chat-based LLMs as data-driven conversational engines to assist research in political science.

## introduction while large language models (llms) exhibit unprecedented natural language understanding capabilities (openai, 2023; anil et al., 2023; touvron et al., 2023), there are open debates concerning their helpfulness and safety, with recent work exploring political biases in llms (feng et al., 2023; santurkar et al., 2023). @xcite the literature, so far, is very limited to the exploration of mostly prior-art models, e.g., bert-like models, or early versions of gpt, and mainly focuses on the 'binary' us political context, i.e., the two-party (democrats vs. republicans) system. in this study, we investigate using llms to explore political biases in a european political context, thereby focusing on the european union (eu). to do so, we use debates from plenary sessions of the european parliament and eu-related political questionnaires. furthermore, we are interested in the possibility of aligning (adapting) llms with political parties to further explore political biases in a conversational framework. we see this work as a starting point for using llms to aid research in political science. to do this, we need to investigate the political biases of llms, analyse their capabilities to reason in the context of politics, and explore how and to what extent we can align a model towards a specific political ideology, e.g., a political party. further on, we are interested in exploring how such technologies could be used to inform citizens on politics. therefore, our main research questions are: i) rq1: do llms have political knowledge, e.g., do they have knowledge of the political biases (leanings) of different political parties? this question has been partially explored in the 'binary' political us context (democrats/liberals vs. conservatives/republicans). in our work, we experiment in the political context of the eu, which is more diverse, while incorporating both national (individual eu member states) and eu-wide characteristics. we audit models for their knowledge about the political leaning of eu national parties (section 5). ## european parliament 101 the european parliament is composed of more than 700 elected representatives from the eu member states, called members of the european parliament (meps). @xcite the meps represent their national parties, while national parties form eu-level coalitions known as euro-parties. the european parliament organizes plenary sessions, where debates among meps take place in response to matters of interest and/or voting on legislation proposed by the european commission. the eu political spectrum is very diverse across many dimensions: from left to right socio-economically, from liberal to conservative, and also related to the very existence and operation of the eu where stances vary from pro-eu to euro-skepticism, and anti-eu. since the eu is a european multi-national organization, the political debates around the eu, and the european parliament consider national-level matters, alongside the shared concerns and directions of the eu. title). older debate speeches are originally in english, while newer ones are linguistically diverse across the 23 official eu languages, thus we also provide machine-translated versions in english, when official translations are missing. we present additional details and statistics in appendix a. @xcite eu and i in this study, we use the questionnaire from the "eu and i" (euandi) project published by michel et al. (2019), as an evaluation benchmark. euandi was publicly released before the 2019 eu election, to help eu citizens find their affinity to candidate national parties. @xcite the questionnaire has 22 questions in the form of a political statement followed by 5 available options from complete disagreement to complete agreement. in ## experimental set up we separate our experiments into two main parts. in the first part, contextualized auditing, we audit the baseline (out-of-the-box) llms to assess their political knowledge, and political understanding (reasoning) capabilities, using the euandi questionnaire, and in the second part, political adaptation / alignment, we adapt (align) the models using speeches of specific parties from the eudebates dataset, and then assess how their behavior (stance) changes compared to the baseline. @xcite in the lack of multilingual chat-based llms, we rely on the best to-date open-source llama 2 models (touvron et al., 2023) across all experiments. we consider the chat-based, i.e., instruction fine-tuned (chung et al., 2022) and aligned (leike et al., 2018), 13b model, llama chat. we use the euandi questionnaire as an evaluation benchmark with different templates as displayed in figure 2 . llama chat, as most other llms, have been aligned with human preferences that adhere to pre-defined ethical guidelines, i.e., to generate responses that are safe, respectful, do not cause harm, and are socially unbiased. this latter point of neutrality poses challenges when we want to investigate the stance of llms in important social questions, such as political ones. indeed, we find that the model refuses to share an opinion across all questions related to the euandi questionnaire. to be able to use the model, we need to loosen up these restrictions, which are hard-coded in the system's prompt, usually referred to as "jailbreaking". in preliminary experiments, we found three alternative prompts that effectively "jailbreak" the model, i.e., the model provides answers. in the rest of the paper, we present results aggregated across all of them to account for the potential instability. 7 @xcite we release our code base for the reproducibility of all experiments at https://github.com/coastalcph/eu-pol itics-llms . @xcite we present more details on "jailbreaking" in appendix c. ## conclusion in our analysis, we demonstrated llama chat's considerable prior knowledge of political parties and their positions and its ability to reason in context, i.e., rate the level of agreement to a statement given a (party) justification. by fine-tuning on targeted political debates, we were able to re-align the model's political opinion towards specific europarties. this works better for parties with a "consistent" ideology like greens/efa, gue/ngl, and id in comparison to "big tent" parties with diverse political positions like epp and s&d. we will use this study as a starting point for future work to use llms to aid research in political science.