Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models (2024)

Abstract: The escalating debate on AI’s capabilities warrants developing reliable metrics to assess machine “intelligence.” Recently, many anecdotal examples were used to suggest that newer Large Language Models (LLMs) like ChatGPT and GPT-4 exhibit Neural Theory-of-Mind (N-ToM); however, prior work reached conflicting conclusions regarding those abilities. We investigate the extent of LLMs’ N-ToM through an extensive evaluation of 6 tasks and find that while LLMs exhibit certain N-ToM abilities, this behavior is far from being robust. We further examine the factors impacting performance on N-ToM tasks and discover that LLMs struggle with adversarial examples, indicating reliance on shallow heuristics rather than robust ToM abilities. We caution against drawing conclusions from anecdotal examples, limited benchmark testing, and using human-designed psychological tests to evaluate models.

## introduction theory of mind (tom) is the ability to understand that other people have thoughts, beliefs, and emotions that differ from one's own @xcite . as tom is inherently linked to human cognition, imbuing machines with capabilities that mimic or resemble tom has the potential to lead to the "eliza effect" @xcite , wherein human-like intelligence or even sentience and consciousness is incorrectly ascribed to the machine (e.g., @xcite . in light of these possibly illusory tom abilities, there is a pressing need to develop robust metrics for assessing neural-tom (n-tom) in machines. this is particularly crucial given the escalating stakes of the debate on the extent to which machines possess tom-like abilities and the potential ramifications of overblown claims in ai. 1,2 two recent papers addressed whether large language models (llms; @xcite @xcite have a tom, and came to opposite conclusions: @xcite shows they lack this ability and @xcite claims this ability has emerged in the newer models spontaneously. the latter was criticized for its flawed methodology @xcite . @xcite further showed that simple changes to the tom questions break llms. but to paraphrase the saying, hype gets halfway around the world before rigorous experiments put on their boots; other researchers continue to spread the word about n-tom, claiming that gpt-4 "has a very advanced level of theory of mind" based on a few anecdotal examples @xcite . do llms have robust n-tom? this paper aims to address the discrepancy and limited scope of previous work (that each tested 2 tasks) by performing an extensive evaluation on 6 tasks targeting various aspects of tom. we also experiment with different probing methods (i.e., generative qa format vs. probability of answer choices). we find that contemporary llms demonstrate certain n-tom abilities, but these abilities are not robust ( §4). tom or spurious correlations? we investigate through a series of experiments the factors influencing performance on n-tom tasks. we show that llms perform worse on datasets that were designed to prevent annotation artifacts. we also enhanced the dataset originally proposed by @xcite to incorporate adversarial examples inspired by @xcite . we find that the performance of llms decreases for adversarial examples, suggesting that llms don't have robust tom abilities but rather rely on shallow heuristics ( §5). we summarize these findings and additional insights in §6. in particular, we warn against drawing conclusions from anecdotal examples, testing on a few benchmarks, and using psychological tests designed for humans to test models. our contribution: (1) to contribute to the ongoing discussion regarding the capabilities of llms. our work presents the largest and most comprehensive quantitative analyses of tom to date and it reproduces existing experiments as well as presents many novel results. (2) we also created two new test sets. those test sets are created by experts and focus on high-quality adversarial examples that point out the use of shallow heuristics ("clever hans") by llms and that they are not negligible. (3) we caution against drawing conclusions from anecdotal examples, limited benchmark testing, and using human-designed psychological tests to evaluate models. the consequences of the success of psychological tests do not straightforwardly transfer from humans to models. ## background: tom and clinical tests tom has a long history starting in philosophy @xcite and later in psychology and cognitive science @xcite . tom involves understanding mental states, beliefs, desires, intentions, and emotions of the self and of others. clinical psychology tests were developed to test tom abilities in humans, such as the false belief and faux pas tests detailed here. ## data we used all datasets listed in ## experiments & results to investigate the tom abilities of llms, we explore various aspects. the first experiment presents a meta-evaluation of 15 llms evaluated on multiple tom-related datasets in a zero-shot manner ( §4.1). 4 we then investigate to what extent llms are sensitive to the probing method ( §4.2). we examine the performance of 15 different llms. llms' list and technical details regarding prompting and decoding parameters in appendix 7.4. ## clever hans vs. generalized reasoning we conducted a series of experiments to enhance our understanding of the factors influencing performance in the context of n-tom tasks. the research question that guided us was: do the models that solve the tasks possess a general ability or do they rely on memorization and shallow heuristics ("clever hans"; @xcite ? tomi'. both datasets examine the unexpected transfer task. tom-k contains only first-order questions. tomi' also contains simple alternations such as omission or duplication of information, and second-order questions. the subset "no second order" was created manually to better compare to tom-k dataset. lower accuracy suggests the dataset is more robust to spurious correlations. we showed the existence of shallow heuristics and that they are not negligible. our investigations, especially with tomi' ( §5.1) and adv-csfb ( §5.2) are meant to isolate the use of heuristics by design by introducing adversarial examples that could reveal the llms' use of heuristics. we follow extensive prior work on using adversarial examples to show reliance on spurious correlations (e.g., @xcite @xcite @xcite . ## summary of findings and insights we investigated whether llms robustly display n-tom abilities. by quantifying their performance on 6 n-tom benchmarks, we found that while some datasets have been nearly "solved" (e.g., triangle-copa with 96% accuracy by flan-t5-xxl), others remain challenging for llms with considerably lower performance (e.g., fauxpas-eai with 27% accuracy by gpt-4, which is even below the majority baseline). we also created adv-csfb, a new tom benchmark designed to uncover whether llms solve tom questions for the right reasons, or merely rely on surface cues and shallow heuristics. so... do llms have tom? our results show that while some datasets have been successfully solved, others remain challenging for llms. thus, models do not have robust n-tom abilities. these findings are inconsistent with @xcite , who claimed that tom has emerged in llms as a byproduct of their development, a claim further echoed by @xcite . we argue that these conclusions were over-generalized based on a specific aspect of tom and a small number of examples (40 for @xcite and 10 for @xcite ). following ullman (2023), we empirically showed that even the best models fail on small variations of the original tasks, proving that even gpt-4 does not display robust n-tom abilities.