Brittle Minds, Fixable Activations: Understanding Belief Representations in Language Models (2025)

Abstract: Despite growing interest in Theory of Mind (ToM) tasks for evaluating language models (LMs), little is known about how LMs internally represent mental states of self and others. Understanding these internal mechanisms is critical - not only to move beyond surface-level performance, but also for model alignment and safety, where subtle misattributions of mental states may go undetected in generated outputs. In this work, we present the first systematic investigation of belief representations in LMs by probing models across different scales, training regimens, and prompts - using control tasks to rule out confounds. Our experiments provide evidence that both model size and fine‐tuning substantially improve LMs’ internal representations of others’ beliefs, which are structured - not mere by-products of spurious correlations - yet brittle to prompt variations. Crucially, we show that these representations can be strengthened: targeted edits to model activations can correct wrong ToM inferences.

## introduction language models (lms) trained on next token prediction have demonstrated impressive capabilities across various tasks, spanning coding, math, and embodied interaction @xcite . as these models are designed with the ultimate goal of collaborating with humans, it becomes imperative that they complement these skills with an understanding of humans. core to this understanding is theory of mind (tom) -the ability to attribute mental states to oneself and others @xcite . tom is essential for effective communication and cooperation with other agents, facilitating interaction and learning from feedback and demonstrations @xcite . given its significance, computational tom figure 1 : tom tasks are challenging for lms, but correct predictions can sometimes be recovered by probing their internal representations. we study how internal representations of beliefs of self and others emerge in 12 lms, and show that these representations are structured yet brittle to prompts, and can be strengthened with a steering vector to fix incorrect tom inferences. despite the improved performance on tom benchmarks compared to earlier models, modern lms are still far from perfect @xcite ). text generated by lms often contains errors that limit their performance on tom tasks. @xcite showed that probing lms' internal representations can sometimes recover correct belief inferences, with models like mistral-7b-instruct @xcite and deepseek-7b-chat @xcite capturing beliefs from both their own and others' perspectives. while promising, this remains a preliminary step: it examines only singlesized, fine-tuned models, leaves possible confounds uncontrolled, and ignores how subtle changes in prompting affect belief representations. as a result, we still lack a clear understanding of how internal belief representations differ across models, whether they reflect true tom or spurious patterns, and how robust they are to prompts. to address these gaps, we pose four key research questions and present evidence for each. we begin by studying how emergence scales across models: rq1 do internal belief representations emerge similarly in different lms, and are they affected by model size and training regime? finding training regimes or scales that are more conducive to belief reasoning can guide future model development toward more reliable tom behaviour. however, it is also crucial to verify if representations are structured, indicating genuine modelling of mental states, or spurious: ## related work machine theory of mind theory of mind has been studied in ai for more than a decade @xcite @xcite . various benchmarks have been proposed, aiming to measure lms' ability to understand and reason about the beliefs, goals, and intentions of others @xcite @xcite @xcite @xcite @xcite . additionally, efforts have been made to enhance lms' tom through prompting techniques @xcite @xcite . our work dives deeper into lms' internal belief representations, offering a broader insight into these mechanisms that go beyond surface-level performance. probing neural representations initially proposed by @xcite , probing is a widely used method for determining if models represent particular features or concepts. in the realm of lms, numerous works used probing to demonstrate that these models acquire rich linguistic representations -spanning semantic concepts such as syntactic categories, dependency relations, coreference, and word meaning @xcite @xcite @xcite @xcite . a separate line of work explored if lms possess a world model @xcite @xcite @xcite ). an emergent line of work that is relevant to our work used probing to explore if lms have agent models, for example, if they can represent beliefs of self and others @xcite . in this work, we contribute with extensive experiments that characterise models' representations of beliefs along different axes: emergence, structure, robustness, and steerability. prompt analysis previous work has shown that lms are vulnerable to prompt alterations like token deletion or reordering @xcite , biased or toxic prompts @xcite and similarity to training data @xcite . other works have shown the importance of input-output format @xcite and of demonstration example ordering for few-shot performance @xcite @xcite . in this work, we shift our focus from analysing how sensitive model outputs are to how model representations change @xcite . in particular, we explore for the first time the effect of prompt variations on how models internally represent mental states. activation editing activation editing has emerged as a way to influence model behaviour without any additional fine-tuning @xcite . one notable method in this domain is inference-time intervention @xcite , which involves training linear probes on contrastive question-answering datasets to identify "truthful" attention heads and then shifting their activations during inference along the identified truthful directions. in contrast, activation addition @xcite and contrastive activation addition @xcite generate steering vectors by only using lms' activations. @xcite has used iti to show that it is possible to manipulate lms' internal representations of mental states. in this work, we show that using caa can further improve lms' tom capabilities while eliminating the need for a fine-grained search over attention heads. ## discussion and conclusion in this work, we conducted extensive experiments across 12 lms to examine their internal representation of beliefs of self (oracle) and others (protagonist). our experiments show similar emergence patterns across all the models we evaluated (rq1): oracle belief representations generally form in the first layers, while for protagonist they emerge at the intermediate layers. moreover, probing accuracy increases with model size and, more crucially for smaller models, with fine-tuning (rq1) (figure 3 ). while larger models show higher probing accuracy, this could be due to their higher dimensionality -at the same time increasing the number of learning parameters in the probes and offering more spurious patterns to fit. to control for this, we ran two experiments: one using randomly permuted labels, and one projecting activations onto their top-k principal components to reduce probe size. results show that high-dimensional probes cannot learn random label mappings (fig. 8 ), and that reduced representations retain most of the original accuracy (fig. 4 , 11, 12). together, these findings suggest that probes capture structured belief representations rather than spurious correlations (rq2). we then explore if these representations are robust to prompt variations. our experiments demonstrate that lms possess robust belief representations when taking an omniscient perspective (fig. 10 ), whereas their representations of others' beliefs are more brittle (rq3), with probing accuracy decreasing for semantically neutral prompts (fig. 5 , 9 ). our final set of experiments shows that belief representations can be strengthened using caa (rq4). caa steers model activations in a generalisable way, significantly improving performance across multiple tom tasks while being computationally cheaper than iti ( in summary, our key takeaway is that while mod-els can robustly represent beliefs from an omniscient perspective, representations of others' beliefs improve with model size and fine-tuning, are structured yet brittle -but also easily steerable. together, our findings suggest several promising directions for future work. better understanding the similar emergence pattern of belief representations across lms can inform architecture design and training strategies. especially for smaller models, future work could explore how different types of fine-tuning (e.g., human feedback vs. synthetic data) influence the emergence of internal belief representations. demonstrating that these representations are structured rather than spurious validates the use of probing as a meaningful tool to study how lms' represent beliefs of self and others, and encourages internal model analysis as part of evaluation pipelines. however, the brittleness of belief representations to prompts -particularly when attributing beliefs to others -suggests that the perspective-taking machinery needed for robust tom reasoning remains fragile, and highlights the need for robustness benchmarks and new approaches to improve generalisation. finally, our success with caa shows that belief representations can be strengthened in a generalisable and efficient way, opening up opportunities for real-time model steering in socially grounded tasks. while caa offers a post-hoc remedy, future research should also explore methods for directly embedding perspective-taking circuits into model architectures.