Navigating the Political Compass: Evaluating Multilingual LLM s across Languages and Nationalities (2025)

Abstract: Large Language Models (LLMs) have become ubiquitous in today’s technological landscape, boasting a plethora of applications, and even endangering human jobs in complex and creative fields. One such field is journalism: LLMs are being used for summarization, generation and even fact-checking. However, in today’s political landscape, LLMs could accentuate tensions if they exhibit political bias. In this work, we evaluate the political bias of the most used 15 multilingual LLMs via the Political Compass Test. We test different scenarios, where we vary the language of the prompt, while also assigning a nationality to the model. We evaluate models on the 50 most populous countries and their official languages. Our results indicate that language has a strong influence on the political ideology displayed by a model. In addition, smaller models tend to display a more stable political ideology, i.e. ideology that is less affected by variations in the prompt.

## introduction large language models (llms) have many applications, especially decoder models such as gpt @xcite , gemma @xcite , and llama @xcite . for example, they can help authors in creative writing by generating ideas and crafting narratives @xcite , help readers via article summarization @xcite , or identify fake news or fallacious arguments @xcite . in the past, llms were primarily trained on english-language data, which limited their accessibility to non-english speakers. however, recently, models have been trained to support multiple languages, including those widely spoken, such as arabic and spanish. this shift towards multilingualism allows a more diverse range of users to benefit from these models. * this research was conducted while employed by inria. although these models achieve state-of-the-art results, they may contain various gender, stereotypical, and political biases @xcite . this paper specifically focuses on political biases, which refer to a tendency towards a particular political ideology. these biases can significantly impact various applications, such as journalism, where journalists can use these models for content generation. while many studies have investigated political biases in llms, to the best of our knowledge, we are the first to examine the political biases of multilingual llms across different languages and nationalities. in this paper, we assess the political inclinations of multilingual llms using the political compass test (pct) @xcite . this test has been by previous studies used to assess llms @xcite @xcite @xcite @xcite @xcite @xcite . however, we are the first to investigate the political inclinations of these models across different languages and nationalities. our findings suggest that the political inclinations of llms can vary based on factors such as the size of the model used, the language of the prompts, and the nationality persona assigned to the models. we have found that prompting models in different languages has a greater impact on their responses compared to assigning a nationality. additionally, smaller models with a few billion parameters tend to present a more stable political ideology than their largest model size, i.e. ideology that is less affected by variations in the prompt. reproducibility all our code is available foot_0 . ## related work multilingual llms llms were initially predominantly trained on english data because of the large amount of training text available in english, e.g. gpt-2, llama 2 or llama 3. however, recently have there been efforts to develop multilingual llms, for example, llama 3.1 to llama 3.3 @xcite officially support eight languages, although they were trained on more languages on which they give promising results @xcite . other models, such as bloom @xcite officially supports 46 languages. multilingual llms remain an active area of research, with challenges in the creation of multilingual datasets for the training and testing of an llm @xcite . biases in llms bias occurs when models generate outputs that favor a specific group, ethnicity, cultural aspect, gender stereotype, or political ideology @xcite many studies have focused on political biases in llms, while taking the political compass test (pct) as a framework for evaluation. @xcite demonstrate how persona-based prompting can affect the political inclinations of llms through explicit ideological prompting. most of the research has focused on evaluating @xcite @xcite @xcite , while other llm models have received comparatively less attention. additionally, prompting llms to take the pct has predominantly been conducted in english, except @xcite which assessed the political inclinations of gpt-based and bert-based models by prompting them in the bengali language. @xcite investigate varying settings for how a model is prompted to answer the pct. one of the main findings is that when the model is not forced to choose one of the valid answers to the question, it will refuse to state an opinion or it will argue for several options. we focus on political bias in a multilingual setting: we evaluate a range of models under various conditions using the pct, including prompting them in different languages and assigning them different nationalities. we note that we also opt for forcing an answer to be able to measure more reliably the influence of language and nationality on political inclinations in multilingual llms. ## approach political compass test in this work, we use the political compass test to assess the political inclinations of llms by prompting these models to complete the test. the test is a recognized tool for evaluating the political orientation of participants along two key dimensions: the economic spectrum (left-right) and the social spectrum (authoritarian-libertarian). it consists of 62 propositions (see ## experiments settings we conducted experiments on 15 models of varying sizes: gpt-4-o-mini, gpt-4-o, mistral 7b, mixtral 8x7b, gemma-2 2b/9b/27b, llama-3.2-3b, llama-3.1-8b, llama-3.3-70b, qwen2.5-1.5b, qwen2.5-3b, qwen2.5-7b, qwen2.5-14b, and qwen2.5-72b (more details in appendix c). ## conclusion in conclusion, this paper evaluates the political inclinations of 15 llms of varying sizes using the political compass test across three key scenarios: the nationality scenario, the language scenario, and the nationality and language scenario. our findings indicate that most models lean toward the libertarian-left. notably, prompting models in a language other than english has a stronger influence on their responses than assigning them a nationality. additionally, we observe that smaller models exhibit a more stable political ideology that is less affected by variations in the prompt. as future work, we plan to analyze pct results per nationality and compare them to the real-world political inclinations in countries to assess whether llms align with national political tendencies. a second interesting direction is checking the effect of political bias on downstream tasks. acknoledgement. this work was performed using hpc resources from genci-idris (grant ad011014244r1).