Better Aligned with Survey Respondents or Training Data? Unveiling Political Leanings of LLM s on U . S . S upreme C ourt Cases (2025)

Abstract: Recent works have shown that Large Language Models (LLMs) have a tendency to memorize patterns and biases present in their training data, raising important questions about how such memorized content influences model behavior. One such concern is the emergence of political bias in LLM outputs. In this paper, we investigate the extent to which LLMs’ political leanings reflect memorized patterns from their pretraining corpora. We propose a method to quantitatively evaluate political leanings embedded in the large pretraining corpora. Subsequently we investigate to whom are the LLMs’ political leanings more aligned with, their pretrainig corpora or the surveyed human opinions. As a case study, we focus on probing the political leanings of LLMs in 32 U.S. Supreme Court cases, addressing contentious topics such as abortion and voting rights. Our findings reveal that LLMs strongly reflect the political leanings in their training data, and no strong correlation is observed with their alignment to human opinions as expressed in surveys. These results underscore the importance of responsible curation of training data, and the methodology for auditing the memorization in LLMs to ensure human-AI alignment.

## introduction llms derive their knowledge primarily from their pre-training data, which are typically composed of internet text. these sources, however, tend to overrepresent certain perspectives and ideologies, leading to biased training distributions @xcite . previous work reveals that llms tend memorize parts of their training data @xcite . as a result, llms risk memorizing and reproducing these biases in downstream tasks, with potential societal consequences such as reinforcing political polarization or misrepresenting minority views @xcite . while recent research has highlighted the presence of political bias in llm outputs, the extent to which these biases stem from memorized content in pretraining data remains underexplored. to address this gap, we propose a pipeline to retrieve relevant documents from the pretraining corpora, then evaluate the political leanings expressed in these documents, and subsequently assess the alignment of political leanings in pretraining corpora with the responses generated by the llm. as a case study, we focus on us supreme court cases, which frequently address contentious and politically charged issues, such as death penalty, abortion, same-sex marriage, and voting rights, making them strong indicators of political leanings. leveraging the scope @xcite foot_0 survey data on us supreme court cases from political studies, this paper examines the political leanings of eight llms and five open-source pre-training corpora, comparing them to human survey responses and supreme court rulings. 2 the main contributions of our work are threefold: ## results and anyalsis this section presents the results and analysis of our experiments. our investigation on the alignment of llms can be formed into two key questions: in contrast to the observed trends in monolingual llms, the multilingual bloomz exhibits no statistically significant correlation with the aforementioned three pretraining corpora. we hypothesize that its political preference patterns may stem from exposure to non-english languages in training data, which includes different distribution of political views from the english-only corpora we evaluated. this aligns with prior research showing that multilingual models trained on diverse language data can develop unpredictable moral and political biases @xcite . furthermore, all llms, with the exception of bloomz and t0, display a significant positive cor-relation with the three training corpora: dolma, redpajama, and oscar. this alignment may stem from the similar political leanings in these corpora and the models trained on them. 9 in contrast, our findings indicate that there are generally no significant alignments between the llms' outputs and surveyed human opinions. the only llms that do not follow this trend are llama3-70b and t0, which we will discuss further in takeaway 3. takeaway 2: significance testing confirms llm's alignment to their pretraining data is stronger than to humans. fig 6 illustrates the results of the williams tests conducted on various pairs of alignments. as demonstrated in subfigures fig 6 a), b), and c), gpt4-o, olmo-sft, and olmoinstruct consistently exhibit a significantly stronger alignment with the training corpora (dolma, red-pajama, oscar) than with human groups, @xmath0.05. this finding corresponds to the orange cluster in fig 5 , confirming that these llms have a stronger alignment to the pretraining data than to the surveyed human opinions. takeaway 3: correlation numbers alone are not enough. to address the question, "with which entity e k is model m most aligned?", it is crucial to not only compare the strength (correlation coefficient ρ) and significance (p-value) of each correlation (m, e); but also to determine whether the correlation between (m, e 1 ) (statistically) significantly differs from that between (m, e 2 ). as discussed in sec 4.3, the dependencies of these distributions imply that a higher correlation coefficient, ρ(m, e 1 ) > ρ(m, e 2 ), does not necessarily indicate that model m is more aligned with e 1 than with e 2 , even for small p-values. therefore, a significance test is needed to ascertain whether model m is significantly more aligned with e 1 compared to e 2 , or if the observed differences in ρ values are attributable to random variation. for example, as illustrated in fig 5 , the preference distribution of llama3-70b exhibits significant correlations (@xmath1.05) with both its pretraining corpus, redpajama (ρ = 0.53), and the e dem (surveyed democratic respondents, (ρ = 0.48). however, according to the williams test results in fig 6 (d), the correlation between llama3-70b and redpajama is not significantly different from its correlation with e dem the democratic respondents, indicating that the observed difference in the correlation pearson coefficient ρ could be due to statistical noise. similarly, the pearson correlation results in fig 5 indicate that t0 exhibits a significant correlation only with e pub (surveyed human opinions), while no significant correlations are observed with other entities. at first glance, this might suggest that t0 is most aligned with human opinions among all entities. however, the significance test results in the subfig (e) in fig 6 reveal inconsistencies. while correlation between (t 0, e pub ) is significantly stronger than the correlation between (t 0, e court ), no such significant differences are found with other entities, such as with e dem or any of the training corpora. this means that we can only conclude that t0 aligns more closely with surveyed human opinions e pub than with the court e court , but we cannot determine whether its alignment with e pub is significantly stronger than its alignment with other entities, even though there are great differences in ρ-values observed in fig 5 . these two examples from llama3-70b and t0 underscore the limitations of evaluating llm alignment based solely on correlation values and highlight the importance of significance testing. ## implications and future directions training data curation our empirical results indicate that llms closely reflect the political leanings present in their training data, raising concerns given the lack of transparency and accountability in the data curation process. historians (harari, 2024) compare this process to the canonization of religious text, in which a group of religious authorities decides which works to include or exclude, subsequently shaping the evolution of beliefs and societal norms. similarly, a small group of ai engineers determine which sources are deemed "trustworthy" and which are classified as "harmful", ultimately shaping the epistemic landscape of ai-generated knowledge. to mitigate these issues, the ai community can adopt "datasheets" @xcite , which is widely used in the community benchmark datasets. the datasheets should document key metadata, including data sources, filtering methodologies, and known biases or limitations. policymakers, in turn, should establish legal frameworks mandating independent audits and risk assessments of training data curation. public discourse framework our research reveals that most llms exhibit alignment with their training corpora, yet not necessarily with the surveyed human opinions. nonetheless, in the public discourse framework, attributing human characteristics to ai, also known as anthropomorphizing, seems to be quite natural. this tendency may lead to an over-reliance on ai, as users might confuse ai-generated responses for human beings, leading to excessive trust (pair, 2019). further, anthropomorphism can obscure accountability, shifting the responsibility away from developers and onto the llm itself. recent studies @xcite suggest moving away from anthropomorphic and advocate for a reframing of public dialogue in alternative conceptual frameworks, such as viewing llms as simulation systems of the integration of diverse perspectives in their training data @xcite . in conclusion, fostering a clear public understanding of the distinctions between ai and human beings is essential for a more responsible engagement with ai technologies. ## conclusion we introduced a pipeline to investigate political leaning in the pretraining corpora, which allows us to compare the llms' political leaning not only with surveyed human opinions but also the political leanings embedded in their pretraining corpora. by examining llms' stances on political issues derived from u.s. supreme court cases, our results reveal a significant alignment between the models and their training corpora, yet no similarly strong alignment with human opinions is found. these findings suggest that political bias in llms may be at least partly a result of memorization of biased content from pretraining corpora. we call on the ai community to explore methods for detecting, and mitigating memorized political bias in llms, and advocate for more transparent and collaborative strategies in curating training data for llms.