Towards A Holistic Landscape of Situated Theory of Mind in Large Language Models (2023)

Abstract: Large Language Models (LLMs) have generated considerable interest and debate regarding their potential emergence of Theory of Mind (ToM). Several recent inquiries reveal a lack of robust ToM in these models and pose a pressing demand to develop new benchmarks, as current ones primarily focus on different aspects of ToM and are prone to shortcuts and data leakage. In this position paper, we seek to answer two road-blocking questions: (1) How can we taxonomize a holistic landscape of machine ToM? (2) What is a more effective evaluation protocol for machine ToM? Following psychological studies, we taxonomize machine ToM into 7 mental state categories and delineate existing benchmarks to identify under-explored aspects of ToM. We argue for a holistic and situated evaluation of ToM to break ToM into individual components and treat LLMs as an agent who is physically situated in environments and socially situated in interactions with humans. Such situated evaluation provides a more comprehensive assessment of mental states and potentially mitigates the risk of shortcuts and data leakage. We further present a pilot study in a grid world setup as a proof of concept. We hope this position paper can facilitate future research to integrate ToM with LLMs and offer an intuitive means for researchers to better position their work in the landscape of ToM.

## introduction the term theory of mind (tom, sometimes also referred to as mentalization or mindreading) was first introduced by @xcite as agents' ability to impute mental states to themselves and others. many aspects of human cognition and social reasoning rely on tom modeling of others' mental states @xcite @xcite . this is crucial for understanding and predicting others' actions @xcite , planning over others' beliefs and next actions @xcite , and various forms of reasoning and decision-making @xcite . inspired by human tom, ai researchers have made explicit and implicit efforts to develop a machine tom for social intelligence: ai agents that engage in social interactions with humans @xcite and other agents @xcite . likewise, a machine tom can enhance agents' capacity for interactions @xcite , explainable decision-making @xcite , dialogue communication @xcite , and collaborative task planning @xcite . machine tom has received an increasing amount of attention in the research community, especially as the field is reshaped by large language models (llms) such as chatgpt @xcite and @xcite . this highlights an ongoing debate and discussion on whether a machine tom has emerged in llms. while llms have demonstrated some capability of inferring communicative intentions, beliefs, and desires @xcite @xcite , researchers also reported concerns regarding a lack of robust agency in llms for complex social and belief reasoning tasks @xcite and in-context pragmatic communication @xcite . emerged or not emerged, that remains a question (or may not even be the central question to ask). in our view, existing evaluation protocols do not fully resolve this debate. most current benchmarks focus only on a (few) aspect(s) of tom, in the form of written stories, and are prone to data contamination, shortcuts, and spurious correlations @xcite @xcite . prior to embarking on extensive data collection for new tom benchmarks, it is crucial to address two key questions: (1) how can we taxonomize a holistic landscape of machine tom? (2) what is a more effective evaluation protocol for machine tom? to embrace the transformation brought by llms and explore their full potential in understanding and modeling tom, this position paper calls for a holistic investigation that taxonomizes tom using the abilities in theory of mind space (atoms) framework @xcite . after a review of existing benchmarks under this framework, we put forward a situated evaluation of tom, one that treats llms as agents who are physically situated in environments and socially situated in interactions with humans. we hope this paper will offer an intuitive means to identify research priorities and to help gain a deeper understanding of, as well as to effectively utilize, llms in tom modeling for ai agents in the future. ## large language models as theory of mind agents since the advent of pre-trained language models, the research community has questioned whether they possess intrinsic mental states to represent the environment @xcite @xcite and comprehend the mental states of others @xcite through the textual description (observation) of behavioral cues. the relatively recent breakthroughs of llms have created many discussions and debates, primarily concerning the extent to which llms possess various capabilities required for a machine tom. in this section, we first survey recent research presenting evidence and counter-evidence for the emergence of tom in llms. we conclude the discussion with the limitations of current evaluation protocols. ## towards a situated theory of mind 4.1 why a situated tom? there have been concerns that cognitive inquiries are inadequate for gaining insight into understanding tom for llms @xcite . however, we believe that the primary problem lies in using story-based probing as proxies for psychological tests, which situate human subjects in specific physical or social environments and record their responses to various cues. we, therefore, call for a situated evaluation of tom, in which the tested llms are treated like agents who are physically situated in environments and socially situated in interactions with others. ## conclusion in this position paper, we survey and summarize the ongoing debate regarding the presence of a machine tom within llms, and identify the inadequate evaluation protocols as the roadblock. many benchmarks focus only on a few aspects of tom, and are prone to shortcuts. to mediate this issue, we follow the atoms framework to offer a holistic review of existing benchmarks and identify underexplored aspects of tom. we further call for a situated evaluation of tom, one that is physically situated in environments and socially situated in interactions with humans. we hope this work can facilitate future research towards llms as tom agents, and offer an intuitive means for researchers to position their work in the landscape of tom.