Machine Theory of Mind Needs Machine Validation (2025)

Abstract: In the last couple years, there has been a flood of interest in studying the extent to which language models (LMs) have a theory of mind (ToM) — the ability to ascribe mental states to themselves and others. The results provide an unclear picture of the current state of the art, with some finding near-human performance and others near-zero. To make sense of this landscape, we perform a survey of 16 recent studies aimed at measuring ToM in LMs and find that, while almost all perform checks for human identifiable issues, less than half do so for patterns only a machine might exploit. Among those that do perform such validation, which we call machine validation, none identify LMs to exceed human performance. We conclude that the datasets that show high LM performance on ToM tasks are easier than their peers, likely due to the presence of spurious patterns in the data, and we caution against building ToM benchmarks relying solely on human validation of the data.

## introduction in cognitive science, theory of mind (tom) refers broadly to the capacity to reason about the mental states of oneself and others (e.g., beliefs, intentions, emotions) -especially when they may differ from one's own @xcite . in recent years there has been an explosion of interest in understanding and quantifying the extent to which language models (lms) demonstrate this ability. numerous benchmark datasets have been designed to measure this using narratives @xcite @xcite , human conversation @xcite , and adversarial data generation @xcite . despite, or perhaps due to, the growth of tom evaluation tools in both diversity and number, the extent to which one can say that lms display tom remains unclear. some evaluation metrics find that tom is almost non-existent in modern models @xcite , others determine that there is evidence but they lack some sort of robustness @xcite , while still others find that they already meet or exceed human performance in some respects @xcite . this contradictory set of results leaves the working scientist wondering -do lms have tom? in this position paper we argue that the variety of results seen across these evaluations is, at least in part, due to a lack of what we refer to as "machine validation", an analysis aimed at identifying patterns in data that neural models (but not humans) might exploit. we begin with a brief history of approaches to measuring tom prior to 2020, and a discussion of how the data may mislead lm-based studies ( §2). we discuss the notion of machine validation ( §3), and then perform a meta-analysis of 16 recent papers introducing tom datasets ( §4) and find that those which report strong zero-shot lm performance tend to lack a form of machine validation. we present fine-tuning baselines for a sample of four datasets from our meta-analysis ( §5); we find that simple models achieve perfect or near-perfect performance on the datasets that omitted machine validation, leading us to outline a suggested workflow for creating tom (and other) datasets ( §6). we conclude with some final recommendations for the study of lm tom going forward ( §7). ## theory of mind in language models the term theory of mind was first introduced by psychologists @xcite studying the behavior of chimpanzees. they posit that an agent has a tom "if [they] impute mental states to @xcite self and others". the study of tom was later extended to examine the behavior of children including the, now famous, sally-anne test @xcite which presents subjects with a narrated or acted scene about two or more agents, and a question to see if the subjects understand the story agents' cognitive state. this style of observer-based probing is especially amenable to the study of tom in lms, where question answering is already a well studied capability @xcite @xcite . as a result, a number of datasets inspired by psychological experiments have been adapted for lms over the years. nematzadeh et al. (2018) produce a template-based question answering corpus (tom-babi) generated from stories inspired by the sally-anne test. @xcite note that such formulaic data results in a flawed evaluation, especially when using supervised methods, and produce their own templatic corpus (tomi) which introduces more noise such as distractor sentences and reorderings. despite these improvements, @xcite find tomi to be vulnerable to similar issues. while recent approaches (see §4) differ greatly from their predecessors, concerns regarding models exploiting spurious correlations @xcite in order to display so-called illusory tom have remained. early work on machine tom did not necessarily focus on zero-shot performance @xcite @xcite or even the inclusion of language as input @xcite . as zero-shot performance has gained priority, fewer studies seem to provide fine-tuned baselines for comparison. we argue that one manner of checking for the presence of surface cues is to provide these simple, fine-tuned baselines. as humans are not thought to be exploiting such patterns for tom, very strong performance of simple models (often prone to relying on these patterns) can be an indicator of undesirable correlations in data or a task that is somehow easier than prior work. we keep these observations in mind in our meta-analysis. ## machine validation any step taken to ensure that machine performance on a benchmark is not due to the exploitation of spurious correlations specific to machine systems is a form of what we term machine validation, i.e., validation designed specifically for machine "subjects". we find that, despite many tom datasets being designed for machine subjects, there is an over-reliance on validation techniques more suitable for humans or, in some cases, no discussion of validation at all. we are not the first to call for additional machine validation @xcite 2024; ullman, 2023 ## meta-analysis to obtain the 16 papers selected for analysis, we searched the acl anthology for papers since 2020 matching the keyword "theory of mind" and manually inspected their content. we discarded papers which primarily contributed methods for improving models of tom, rather than evaluation resources. while a number of datasets in related topics may be relevant (e.g., emotion recognition), we restrict our analysis to those specifically designed for tom. a similar process was repeated by searching google scholar using the term "language model theory of mind". we then read the identified papers, paying special attention to the manner in which their data was created, validated, and used in evaluation. we also reviewed several papers in this citation network which did not meet our recency threshold. the final collection is a diverse sample. it includes a number of datasets compiled to test lm evals data evals superhuman metadata study zero-shot few-shot fine-tuning human in 1+ expt. perspective format source size common-tom (soubki et al., 2024) 60.6 -64 80 no observer mc (2) natural 7,374 fantom (kim et al., 2023) 26.6 -53.7 87.5 no observer fr, mc (2) lm 10,317 opentom (xu et al., 2024) 52.8 -72.7 92.2 no observer mc (2/3) lm 13,708 tombench (chen et al., 2024) 74.7 --86.1 no observer mc (4) manual 2,470 social iqa (sap et al., 2022) 42 73 83* 87 no observer mc (3) mturk 1,954 mindcraft (bara et al., 2021) --41.7 56.7 no interactant mc (3, 21) natural 1,200 fauxpas-eai (shapira et al., 2023) 40 --82 no observer mc (2) manual 40 mmtom-qa (jin et al., 2024) 46.7 -76.7 93 no observer mc (2) lm 600 hi-tom (wu et al., 2023) 58.9 ---no (?) observer mc (15) template 600 adv-csfb (shapira et al., 2024) 70 ---no (?) observer mc (3) manual 183 exploretom (sclar et al., 2024) 74 ---no (?) observer fr, mc (2) lm 1,000* epitome (jones et al., 2024) 58.9 --70.6 yes observer fr, mc (2) manual 446 bigtom (gandhi et al., 2024) 84.5 89.7 -86 yes observer mc (2) lm 5,000 strachan et al. (2024) 88.2 --89.2 yes observer mc (2) manual 105 motomqa (street et al., 2024) 88.6 --90.4 yes observer mc (2) manual 70 simpletom (gu et al., 2024) 89.5 97.1* --yes (?) observer mc (2) lm 3,441 ## case study we hypothesize that datasets which report superhuman performance will likely see strong performance in fine-tuning experiments (i.e., fail machine validation). to investigate this we compare the finetuning performance of bert @xcite , gpt-2 @xcite , and flan-t5-base (chung et al., 2022) across two datasets which did not find superhuman performance (fantom and common-tom) and two datasets which did (sim-pletom and bigtom). the models were chosen to be relatively small by modern standards and include an encoder-only model (bert, ~110m params), decoder-only model (gpt-2, ~137m params), and encoder-decoder model (flan-t5, ~248m params). the datasets were selected somewhat arbitrarily from our set of 16 studies to include datasets which we perceived to report poor, moderate, and strong performance, respectively. for fantom we discard the free response questions to maintain comparability. we average over three seeds for common-tom using the author's splits and, for all other datasets, over five folds using cross-validation. we report the average accuracy of all models across all runs. further details, including hyperparameters, can be found in appendix b. ## validating your benchmark the results of our survey and case study suggest what we suspected from the start -human validation is not enough for tom benchmarks. roughly half of the surveyed papers discuss some form of machine validation in the design phase but only six provide such analysis after construction. we outline a workflow for validating tom benchmarks here. when designing tasks, think carefully about the sort of heuristics that a model might use to perform well while avoiding tom reasoning. two ways to make this less likely are to introduce noise, and to construct adversarial examples. some examples of noise include adding distractor sentences @xcite and including (possibly multiple) rephrasings of the same task @xcite . this can be complemented by the addition of adversarial examples, such as entries that should be impossible to predict @xcite , or that vary the scenario to reveal model biases @xcite . if this introduces subsets that can be used to detect when models are relying on surface cues, this analysis should be included and made clear to users. ideally, there are few spurious correlations in the benchmark for models to exploit in the first place, but some work should be done to estimate their prevalence in the completed benchmark. this gives an idea of how likely strong performance is to be a false positive. after construction, always train a simple fine-tuned baseline. this could be a small lm (as we do in §5) or a more classical statistical model. if this baseline performs unexpectedly well, consider searching for lexical overlaps @xcite , employing one of the growing numbers of interpretability techniques @xcite , or returning to options from the design phase. we have found that less than half of the 16 lm tom studies we examined evaluate their dataset for patterns only a machine might exploit (i.e., perform machine validation). among those which do perform such validation, none identify lms to exceed human performance on any aspect of their benchmark, while all studies that find superhuman performance omit such checks. we then performed machine validation by providing a fine-tuning baseline. we found that a small, fine-tuned system could achieve near perfect accuracy on the datasets which did not perform machine validation. this indicates these datasets are, in some sense, easier than their peers, likely due to the presence of spurious patterns in the data. in the following paragraphs we offer some closing thoughts and suggestions. how do you interpret lm performance on tests designed for humans? it is notable that tom was first studied in animals, and the manner of testing underwent significant changes when attention was turned towards humans. it is entirely possible, as others have also noted @xcite @xcite , that our methods will need to change further to study this phenomena in lms. in the case of animals to humans, experimenters had to mind the change in capabilities between these two subjects. when observing the performance of lms on tests originally used for humans, we can't necessarily take away the same conclusions -the capabilities of the subject have changed again. models may exploit patterns present in our evaluations, otherwise undetectable by humans, that do not broadly generalize to what we wish to measure. other evaluation options changing our evaluation approach might avoid this situation altogether. moving away from observer-based tom evaluations towards ones where the agent is situated @xcite , adaptive evaluations @xcite , and simulated environments @xcite all reduce the chances of measuring primary spurious patterns. in other words, we should couple evaluations more closely to the conditions in which tom is actually used. fine-tuning small models is necessary but not sufficient fine-tuning small models situates the difficulty of a dataset. unexpectedly strong performance is likely an indicator of undesirable patterns or relative ease. while this may not directly say what in the data models are exploiting, it will indicate that there is probably an issue. the growing number of interpretability techniques @xcite and even more classical approaches like measuring lexical overlap @xcite can help to track down the culprit. we can borrow from the extensive work on more general qa datasets which has run into similar issues, like shortcutting @xcite . this is a sufficient, not a necessary condition. it doesn't mean the dataset is free of spurious patterns but if it fails, then it likely means trouble. but do lms have tom? this is a tricky question. if the question is simply "can they infer mental states?", as described in @xcite ), the answer is plainly, yes. however, this has never been the problem. the trouble has always been making sense of the inconsistencies in their performance across seemingly similar contexts. evaluation tools should not be aimed at measuring the presence of tom but the robustness of tom @xcite . with few common goalposts to situate the difficulty of so many tom datasets, it's hard to say if models are improving, but it seems clear that performance is not yet robust.