Theory of Mind in Large Language Models: Examining Performance of 11 State-of-the-Art models vs. Children Aged 7-10 on Advanced Tests (2023)

Abstract: To what degree should we ascribe cognitive capacities to Large Language Models (LLMs), such as the ability to reason about intentions and beliefs known as Theory of Mind (ToM)? Here we add to this emerging debate by (i) testing 11 base- and instruction-tuned LLMs on capabilities relevant to ToM beyond the dominant false-belief paradigm, including non-literal language usage and recursive intentionality; (ii) using newly rewritten versions of standardized tests to gauge LLMs’ robustness; (iii) prompting and scoring for open besides closed questions; and (iv) benchmarking LLM performance against that of children aged 7-10 on the same tasks. We find that instruction-tuned LLMs from the GPT family outperform other models, and often also children. Base-LLMs are mostly unable to solve ToM tasks, even with specialized prompting. We suggest that the interlinked evolution and development of language and ToM may help explain what instruction-tuning adds: rewarding cooperative communication that takes into account interlocutor and context. We conclude by arguing for a nuanced perspective on ToM in LLMs.

## introduction machines that can think like us have always triggered our imagination. contemplation of such machines can be traced as far back as antiquity (liveley and thomas, 2020), and peaked with the advent of all kinds of 'automata' in the early days of the industrial revolution @xcite before settling in computer science from the 1950s @xcite . currently people around the world can interact with powerful chatbots driven by large language models (llms), such as openai's chatgpt @xcite , and wonder to what degree such systems are capable of thought. llms are large-scale deep neural networks, trained on massive amounts of text from the web. *equal contribution. they are vastly complex systems: even if all details about their architecture, training data, and optional fine-tuning procedures are known (which is currently not the case for the most competitive models), it is very difficult to oversee their capabilities and predict how they will perform on a variety of tasks. researchers from linguistics @xcite , psychology @xcite @xcite , psychiatry @xcite , epistemology @xcite , logic @xcite , and other fields, have therefore started to study llms as new, 'alien' entities, with their own sort of intelligence, that needs to be probed with experiments, an endeavour recently described as 'machine psychology' @xcite . this not only yields knowledge about what llms are capable of, but also provides a unique opportunity to shed new light on questions surrounding our own intelligence @xcite ). here we focus on attempts to determine to what degree llms demonstrate a capacity for theory of mind (tom), defined as the ability to work with beliefs, intentions, desires, and other mental states, to anticipate and explain behaviour in social settings @xcite . we first address the question how llms perform on standardized, language-based tasks used to assess tom capabilities in humans. we extend existing work in this area, surveyed in section 2, in four ways: by (i) testing 11 models (see we are aware of issues regarding llm training and deployment, for example regarding the biases they inherit @xcite , problems for educators @xcite , and ethical concerns in obtaining human feedback @xcite . ongoing reflection on the use of llms is necessary, but outside the scope of this paper. ## methodology here we describe our tasks and procedures for testing llms and children; all code, materials, and data are on osf: https://shorturl.at/fqr34 . ## discussion summing up the results for the sally-anne tests, while it is less surprising that base-llms and smaller instruct-llms struggle with increasing test complexity and deviations, it is striking that second-order tom immediately perturbs some large instruct-llms (e.g. palm2-chat), and that adding deviations from the original test formulations pushed performance of even the most competitive models down (e.g. gpt-4, gpt-3.5). this initially suggests that performance on tom tasks does not generalize well beyond a few standard contexts in llms, in line with earlier work @xcite @xcite . for the strange stories we saw that base-llms perform generally below child level. most instruct-llms perform close to or above child level, particularly as items become more complex and child performance drops much more dramatically than llm performance. levels of deviation from the original test formulation seem to have made almost no impact for the ss, suggesting that the capacity to deal with non-literal language targeted by the strange stories test does generalize to novel contexts. we conclude that instruct-llms are quite capable at interpreting non-literal language, a skill that in humans involves tom. since the training data of llms includes numerous books and fora, which are typically rich in irony, misunderstanding, jokes, sarcasm, and similar figures of speech, we tentatively suggest that llms are in general wellequipped to handle the sort of scenarios covered in the strange stories. this should in theory include base-llms, but it could be that their knowledge does not surface due to the test format, even after specialized prompting. going one step further, we hypothesize that sally-ann is generally harder for llms given that this test relies less on a very specific sort of advanced language ability, but more on a type of behaviourally-situated reasoning that llms have limited access to during training (see also @xcite . the imposing memory test was the most chal-lenging for both base-and instruct-llms. since our version of it was never published before, it constitutes another robustness test, which only gpt-4 as largest instruct-llm seems to pass well. the gap between base-and instruct-llms is best summarized in figure 4 . here we see that no base-llm achieves child level: all llms approaching or exceeding child performance are larger instruct-llms. our adapted prompts and insertion of correct answers for motivation questions did not make a difference. we suggest that another issue for base-llms, besides the prompt format, was prompt length. this was highest for im, which can explain why they struggled most with this test. prompt length, in relation to the models' varying context window sizes and ability to engage in what hagendorff et al. (2023) call chain-of-thought reasoning, merits further research (see also @xcite . we tested whether there was a difference between model performance on closed versus open questions across all three tasks, but found no signal: the models that struggled with closed questions were also those that performed low on open questions (for more details and additional information on prompting, see appendix a on osf). evidence is emerging that most llm capacities are learned during self-supervised pre-training @xcite , which suggests that base-llms are essentially 'complete' models. yet instruction-tuning, even in small amounts @xcite , adds adherence to the desired interaction format and teaches llms, as it were, to apply their knowledge appropriately. we see a parallel between instruction-tuning and the role for rewarding cooperative communication in human evolution and development. it has been argued extensively that human communication is fundamentally cooperative in that it relies on a basic ability and willingness to engage in mental coordination (e.g @xcite . it is a key characteristic of the socio-cultural niche in which we evolved that, when growing up, we are constantly being rewarded for showing such willingness and cooperating with others to achieve successful communicative interactions @xcite . reversely, if we do not, we are being punished, explicitly or implicitly via increasing social exclusion @xcite . this brings us back to our context: instruction-tuning essentially rewards similar cooperative principles, but punishes the opposite, which may amount to an enhanced capacity for coordinating with an interaction partner's perspective, in humans and llms alike. this is reflected in performance on tom tasks, which are banking on this capacity too. finally, we do not claim that llms that performed well also have tom in the way that humans have it. validity of cognitive tests such as those used in tom research is a general issue (e.g. van duijn, 2016). yet for humans tom tests are validated 'quick probes': decades of research have shown that proficiency on such tests correlates with an array of real-world social and cognitive abilities @xcite . for llms we are in a very early stage of figuring out what is entailed by proficon tom tests: on the one hand it is impressive that some models show a degree of robust performance, without explicit training on tom. on the other hand it remains an open question whether this amounts to any actual capacities in the social-cognitive domain, in which they are clearly very differently grounded (if at all) compared to humans. for future research we believe in the format of testing models that differ in other respects than just size, on a varied array of tasks, with multiple tests per test item, to gain further insight into the aspects that explain variability in performance. for this, more openness about architecture and training procedures of current and future llms is imperative. in addition, we believe to have contributed to the debate by benchmarking llm results on child data, but more of this is needed. we had limited samples and age distributions, and tests were not presented in optimal ways (see section 3.5). we emphasize that our results need to be seen within the time frame of late spring 2023. the fast pace with which llms are currently released and, in some cases, updated, makes them a moving target. there are indications that specific capacities of models from the gpt-family have declined over time, perhaps as a result of such updates (e.g., handling math problems and producing code; @xcite . future studies need to address how such developments impact the capacities assessed in this paper. ## conclusion we have shown that a majority of recent large language models operate below performance of children aged 7-10 on three standardized tests relevant to theory of mind. yet those that are largest in terms of parameters, and most heavily instructiontuned, surpass children, with gpt-4 well above all other models, including more recent competitors like palm2-chat and palm2 (see figure 4 ). we have interpreted these findings by drawing a parallel between instruction-tuning and rewarding cooperative interaction in human evolution. we concede that researching the degree to which llms are capable of anything like thought in the human sense has only just begun, which leaves the field with exciting challenges ahead.