La Boussole Cassée de l’Alignement Politique (2025)

Abstract: L’évaluation, la réglementation et l’alignement des Grands Modèles de Langue (LLM) sur des questions politiques sont devenus des préoccupations cruciales alors que ces technologies se répandent de plus en plus dans tous les secteurs de la société. Cependant, des méthodologies et des fondements théoriques clairs font encore défaut. S’appuyant sur l’œuvre de Converse sur l’opinion publique, nous examinons de manière critique les pratiques courantes d’évaluation idéologique. Nous plaidons également pour des approches alternatives, plus étroites, mieux alignées sur les systèmes de croyances du grand public.

## grounding political evaluation of llms the urgent need to appraise the political stances of large language models (llms) has lead practitioners to adopt readily available, straightforward, measurement tools, potentially at the expense of more scientifically grounded methods. experiments testing llms with abstract ideological questionnaires, most notably the political compass test (pct) @xcite , have burgeoned. most often, these studies claim that llms (ranging from widely used proprietary models, to smaller open-weights ones) exhibit liberal, left-leaning, views @xcite @xcite @xcite @xcite . however, beyond practical concerns -for instance, limitations linked to the use of multiple-choice format to evaluate llms, or their lack of consistency @xcite -, this methodology, which relies on elite-centric notions of ideology, can be challenged through foundational works in political science. @xcite 's analysis of public opinion provides a particularly useful lens. notably, converse clearly distinguished the structure of the political belief systems 1 of political elites and the mass public @xcite . the former, categorized as (near-)ideologues, tend to exhibit highly constrained belief systems, organized around abstract overarching dimensions. conversely, the mass public demonstrates significantly lower constraints among idea-elements, with belief systems that are more fragmented, issue-specific, and shaped by concrete social objects, or immediate, situational concerns. 1. defined as "a configuration of ideas and attitudes in which the elements are bound together by some form of constraint or functional interdependence", where "constraint" is understood as "the success we would have in predicting, given initial knowledge that an individual holds a specified attitude, that he holds certain further ideas and attitudes" @xcite . ## beyond abstract positioning : contextual approaches if some popular approaches fall short, converse's work also offers a foundation for moving beyond abstract positioning toward more situated assessments of llms' political biases. evaluation practices should embrace the characteristics of mass public belief systems (fragmentation, low constraints and contextual thinking) and aim to measure not only political stances, but also the underpinning level of conceptualization. to comply with such standards, assessment methodologies should be based on open-ended, context-aware, issue-centered items. these elements reflect the main attributes of the mass public's attitudes described by converse. first, using open-ended questions would avoid forcing responses into predefined categories and allow researchers to observe how llms natively frame political issues in generated responses. second, stressing context is not only crucial in any llm task, but also mirrors the situational and unstable attitudes of the mass public. finally, grounding evaluation in well-defined, issue-specific prompts would result in approaches that are better suited to capture the fragmentation of mass-oriented belief systems and allow to measure constraints within narrower thematic clusters. that said, conforming to such principles would come with practical and ethical challenges. they demand greater efforts in analyzing llms' generated content and raise important questions about intent, framing and responsibility in sensitive evaluations. ## issue-based evaluation in perspective the arguments advanced in this paper rest on a conceptual distinction between elite-structured abstract ideological reasoning, and the fragmented, low-constrained character of mass public belief systems. in addition to offering a critical lens on popular evaluation practices, this perspective provides a theoretical foundation for the development of alternative strategies that evaluate models through open-ended, issue-based approaches capable of capturing situated and ambivalent positioning. encouragingly, such considerations are beginning to translate into practical evaluation methodologies. while the primary motivations of these works may be methodological -driven by the limitations of abstract ideological questionnaires-, they demonstrate a shift toward issue-specific evaluation. issuebench @xcite emphasizes ecological validity by deriving prompt templates and political issues from actual user interactions to measure stance variability. similarly, bang et al. these initiatives represent valuable methodological foundations. while currently mainly focused on u.s.-centric political landscape, they could be extended to examine how fragmented belief systems operate across cultures, both by integrating issues from diverse social contexts and by analyzing variations in constraint and salience. as the field advances, evaluations sensitive to this diversity will be essential for the alignment of llms.