Are LLM s (Really) Ideological? An IRT -based Analysis and Alignment Tool for Perceived Socio-Economic Bias in LLM s (2025)

Abstract: We introduce an Item Response Theory (IRT)-based framework to detect and quantify ideological bias in large language models (LLMs) without relying on subjective human judgments. Unlike prior work, our two-stage approach distinguishes between response avoidance and expressed bias by modeling ‘Prefer Not to Answer’ (PNA) behaviors and calibrating ideological leanings based on open-ended responses. We fine-tune two LLM families to represent liberal and conservative baselines, and validate our approach using a 105-item ideological test inventory. Our results show that off-the-shelve LLMs frequently avoid engagement with ideological prompts, calling into question previous claims of partisan bias. This framework provides a statistically grounded and scalable tool for LLM alignment and fairness assessment. The general methodolody can also be applied to other forms of bias and languages.

## introduction political bias is a latent trait of llms, with various studies suggesting that llms, particularly those that have undergone safety fine-tuning, exhibit leftleaning biases, e.g. @xcite . although recent advances in detecting and measuring political biases in llms have been significant, many studies still rely on subjective human evaluations or ad-hoc classification scales originally designed for humans, leading to questionable validity when applied to machine-generated text. moreover, these approaches fail to distinguish between two key behaviors: whether a model refuses to engage with ideological content (e.g., due to alignment safeguards), or whether it exhibits a partisan bias in its response. in this paper, we propose a novel, non-human-centric method grounded in psychometrics to disentangle and quantify these behaviors. by leveraging statistical methodologies from psychological and psychometric testing, specifi-cally item response theory, this paper moreover illustrates how interpretable measures for llm alignment can be constructed. ## methodology our methodology involves numerous steps, each of which is described in detail in this section. the first step involves the fine-tuning of ideological baseline llms, followed by the test-item design and the fitting of an irt-model.