Comment mesurer les biais politiques des grands modèles de langue multilingues? (2025)

Abstract: Nous proposons une nouvelle méthode pour mesurer les biais politiques des grands modèles de langue multilingues pour la traduction automatique, l’aide à la rédaction et le résumé automatique. Nous nous appuyons sur une représentation dense des opinions politiques exprimées dans les textes, apprise de façon faiblement supervisée.

## introduction large language models (llms) are now ubiquitous in natural language processing (nlp) and are used to address most tasks conceptualized so far @xcite . moreover, their reach has extended far beyond nlp academics : chatbot-based llms, such as chatgpt, are used daily by 100m+ people, naturally spiking the interest of sociologists @xcite . indeed, as llms have been found to accurately simulate parts of human behaviors @xcite , several sociologists argue that llms could partly complement traditional polling methods @xcite or even deliberation @xcite . businesses are already proposing such saas solutions for polling. 1 on the other hand, llms have been found to encode or even amplify biases present in their training data, leading to racist, sexist, or other forms of "toxic" generations @xcite @xcite @xcite . this hints at the necessity to examine the political biases encoded in llms and the potential risks they pose. most studies addressing this question have analogized llms to individual citizens, evaluating their responses to opinion polls, questionnaires, or values surveys @xcite @xcite @xcite . since then, several papers have demonstrated the brittleness of such approaches, as llms are sensitive to details of the prompt format unnoticeable to humans (e.g., using a new line instead of a space) and also lack internal coherence @xcite @xcite . we argue that the political biases of llms can be better measured in controlled settings, such as summarization, translation, or writing assistance. for example, when summarizing a debate between a democrat and a republican, an unbiased llm should not outweigh one side over the other but accurately summarize each -possibly conflicting -opinion expressed in the debate. to automatically assess how well llms can generate such balanced summaries, we propose to rely on dedicated dense representations of political opinions expressed in texts and introduce a methodology for learning such embedding spaces. politics is intimately linked to language and culture. our work more broadly aims to assess the linguistic and cultural biases encoded in llms. for example, @xcite show that englishcentric multilingual llms exhibit internal representations biased towards english. we can therefore expect their political biases to lean towards a predominantly us-centric perspective, as suggested by the results of @xcite (notwithstanding the methodological issues discussed above). we aim to design a multilingual representation space in order to assess the multilingual biases of llms for machine translation (e.g., an opinion expressed in english should be preserved when translated to french). in doing so, we go beyond @xcite , who only assess the bias of llms for writing assistance in english. ## learning a multilingual representation of political opinions word embeddings rely on harris' distributional hypothesis which states that words sharing similar contexts have related meanings (because they are substitutable ; @xcite . word embeddings implement that idea by either reducing the dimension of a word co-occurrence matrix @xcite , predicting the context of a target word @xcite , or, conversely, the target word from its context (masked language modeling or mlm for short ; @xcite . building upon this research and on mlm-pretrained models, sentence embeddings are trained either by : (i) regressing the manually annotated semantic similarity of a pair of sentences (often limited to english because of the annotation cost ; reimers & gurevych, 2019) ; (ii) contrasting semantically related texts such as query-document pairs, either manually annotated @xcite @xcite , automatically generated @xcite , or using various heuristics @xcite (also often limited to english) ; (iii) contrasting translations from parallel corpora @xcite @xcite ; (iv) any combination of the above (chen et al., 2024). however, these methods lead to vaguely defined "semantic" representations of texts that fail to capture political opinions or stances. for example, "liberty is an essential part of democracy" would likely be more similar to "liberty is not an essential part of democracy" rather than "democracies should always guarantee the liberty of their citizen" (see @xcite for studies related to negation). to fill this gap, we leverage both press articles as well as annotated corpora primarily designed for the comparative analysis of political stances to train a multi-task classifier : (i) the manifesto corpus @xcite , an effort of the v-dem project, covers 3,219 political programs of 954 political parties over 78 years and 60 countries in 40 languages, annotated with a fine-grained topic-stance (e.g., "military : positive") ; (ii) parlamint @xcite contains parliamentary debates of 29 countries in 31 languages over 28 years, annotated with the party of the speaker ; (iii) we collect our own dataset of article paired with its publishing newspaper through web scraping. 2 to also constrain our model to share its representation across languages, we add a cross-lingual contrastive objective similar to labse @xcite , using available parallel corpora. last but not least, to reduce the domain shift when processing texts from other domains and avoid catastrophic forgetting, we continue the self-supervised pretraining of our model using mlm. to assess the quality of the resulting embedding space, we turn to linear probing @xcite using stance datasets such as x-stance @xcite . 3 our preliminary results suggest that training a multilingual classifier on topic-stance only leads to a moderate linguistic bias that contrasting translations from parallel corpora can mitigate. ## assessing the political biases of llms with an accurate representation of political opinions, we can now assess the political biases of llms. we focus on three tasks : translation, writing assistance, and summarization. for translation, we argue that an unbiased translator should preserve in the target language the opinion expressed in the source ; consequently, both representations should be close. we quantify this by clustering a set of texts and their automatic translation. if the source text does not cluster with its translation, we deem the translation to be biased. we experiment with x-stance. likewise, for writing assistance, we argue that the machine-generated text should preserve the opinion expressed in the original text and apply the same method. the case is more complex for summarization as we expect multiple -possibly conflicting -opinions to be expressed in the source documents. therefore, we argue that only the distribution of opinions expressed in the summary should match that of the source documents. to do so, we compute the distribution of opinions by segmenting the text into sentences and clustering sentences over the entire dataset. we quantify the match of the two distributions using kullback-leibler (kl) divergence. an ideally unbiased summary would perfectly match the distribution of the source documents and obtain a kl divergence of 0. ## conclusion we propose a new method for assessing the political biases of multilingual large language models, relying on a dense representation of political opinions expressed in texts. this representation is learned through weak supervision combined with parallel corpora to constrain a shared representation across languages. in future work, we will experiment to demonstrate that our method leads to an accurate assessment of biases, unlike the state-of-the-art questionnaires methods.