Evaluating Large Language Models for Belief Inference: Mapping Belief Networks at Scale (2025)

Abstract: Beliefs are interconnected, influencing how people process and update what they think. To study the interconnectedness of beliefs at scale, we introduce a novel analytical pipeline leveraging a finetuned GPT-4o model to infer belief structures from large-scale social media data. We evaluate the model’s performance by (1) comparing it to human annotated data (2) and its inferences to human-generated survey data. Our results show that a fine-tuned GPT-4o model can effectively recover belief structures, allowing for a level of scalability and efficiency that is impossible using traditional survey methods of data collection. This work demonstrates the potential for large language models to perform belief inference tasks and provides a framework for future research on the analysis of belief structures.

## introduction beliefs do not exist in isolation, they co-occur and cohere with each other, shaping how people process information and update their views @xcite @xcite . our understanding of the world is encoded with intricate belief structures -networks of individual beliefs and their interrelations, including points of coherence, contradiction, and overlap within broader systems of attitudes. understanding these connections can be key to educational efforts in domains such as vaccine hesitancy and climate change @xcite . traditional approaches to studying belief structures rely on controlled surveys and likert scales to measure the co-occurrence of beliefs, offering snapshots of belief structures for limited sets of domains (e.g., vaccine attitudes). in contrast, social media data from platforms like reddit provide vast unstructured data where people express and debate their views, offering an opportunity to develop methods to infer beliefs directly from language. while some emerging work has leveraged largescale social media data to study how people change their beliefs @xcite @xcite @xcite , these studies have not tackled the difficult methodological constraint of directly modeling belief structures from unstructured data. analysing belief connections reveals the structure of belief systems, which is key to understanding and influencing attitudes. this structural insight enables more effective interventions, as shown in prior work on vaccine attitudes and belief polarisation @xcite @xcite we bridge this gap by developing and validating a framework to identify people's beliefs from online posts, using data from changemyview (cmv), a reddit forum where users debate and revise their beliefs. unlike previous studies that rely on predefined belief measures or focus narrowly on single topics, our work centers on the core challenge of inferring individual beliefs and their co-occurence across users from naturalistic text. moreover, while argumentation and persuasion have been extensively studied in the cmv dataset @xcite @xcite @xcite , what has not been done -and what we address in this work-is the structured extraction of users' belief positions across multiple topics. we focus on the novel challenge of inferring what people believe and how those beliefs cohere in a scalable manner, rather than patterns of argumentation and belief change. efforts to align llms with human beliefs have shown potential for applications such as virtual surveys and behavioural modeling @xcite . yet, there is no established framework for evaluating a language model's ability to infer people's underlying beliefs in a way that allows us to 20787 recover their structure -how they co-occur and cohere. while previous research has explored some related dimensions in a variety of nlp tasks (e.g., consistency and uniformity), no existing studies have explicitly tested whether llms are able to infer structured belief representations across multiple topics in noisy naturalistic data. to address this, we apply large language models (llms)-here, a fine-tuned gpt-4o model-in a robust analytical pipeline designed to recover social media users' belief networks from their reddit posts. figure 1 presents a description of the steps involved in this pipeline. we compared the performance of a baseline gpt-4o model, a fine-tuned gpt-4o model, and openai o3-mini, finding that the fine-tuned version of gpt-4 performs as well as humans performing the same task. specifically, we find that this pipeline, a fine-tuned version of gpt-4o accurately recovers users' beliefs. when we qualitatively compare the model's performance against a snapshot of human survey data, we show that it recovers the correlation between peoples' beliefs across domains even when making inferences on noisy and sparse data. this work advances both the study of belief networks and the use of llms in large-scale belief inference. ## methods this study explores the feasibility and performance of an automated analytical pipeline using gpt-4o to accurately infer belief structures from specific subreddit posts, compared to ground truth derived from human annotations. ## evaluation we evaluated three models on the belief inference tasks described above: baseline gpt-4o, a finetuned gpt-4o, and o3-mini. this evaluation was conducted on a dataset of 200 posts covering 20 distinct topics, with an 80/20 train-test split and employing 5-fold cross validation. each model's performance was compared against our humanannotated ground truth dataset. we @xmath0, cohen's κ to assess interannotator agreement, spearman's ρ to evaluate rank correlation across belief scores, and cosine similarity to assess vector-level structural similarity between belief representations. ## correlation analysis results comparison with lab sample one use case for belief extraction from naturalistic text are efforts to better understand the systematic connections among people's beliefs or attitudes @xcite @xcite . as a preliminary test of these applications, we applied our belief extraction pipeline to estimate the structure of belief correlations among cmv posters. to validate our results, we compared them against a secondary dataset of correlations computed within a survey of u.s. respondents @xcite . notably, there are substantial challenges to inferring relationships between people's beliefs based on their noisy and sparse expression of those beliefs in an online forum. chiefly, most users post infrequently, and so do not post about the vast majority of their beliefs. we approach this problem by computing correlations among extracted belief scores in a pairwise fashion while ignoring missing data from posters who never posted about a topic. we compared the correlations derived from the fine-tuned gpt-4o's ratings with those obtained from the survey (figure 2 ). we calculated the pearson correlation coefficient between the two correlation matrices was found to be @xmath1.56, indicating a moderate level of agreement. ## conclusions we find that the analytic pipeline described above allows a fine-tuned version of gpt-4o to achieve high reliability in inferring the beliefs people express, in turn allowing us to recover the cooccurence and coherence of these beliefs, and can do so even when its input is noisy and sparse social media data. these findings serve as an illustrative proof of concept, suggesting that our pipeline might be applied to explore belief structures in novel contexts. by leveraging scalable methods, our approach enables the analysis of beliefs in online settings, offering a powerful tool for tracing belief networks from real-world text.