Hi- T o M : A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Models (2023)

Abstract: Theory of Mind (ToM) is the ability to reason about one’s own and others’ mental states. ToM plays a critical role in the development of intelligence, language understanding, and cognitive processes. While previous work has primarily focused on first and second-order ToM, we explore higher-order ToM, which involves recursive reasoning on others’ beliefs. %We also incorporate a new deception mechanism in ToM reasoning. We introduce Hi-ToM, a Higher Order Theory of Mind benchmark. Our experimental evaluation using various Large Language Models (LLMs) indicates a decline in performance on higher-order ToM tasks, demonstrating the limitations of current LLMs. We conduct a thorough analysis of different failure cases of LLMs, and share our thoughts on the implications of our findings on the future of NLP.

## introduction theory of mind (tom) refers to the ability to understand and reason about the mental states of others such as intentions and beliefs, and also to distinguish them from one's own @xcite . such an ability has been considered a crucial point in the development of intelligence functions @xcite @xcite , and previous research has demonstrated that tom reasoning is highly related to linguistic and cognitive processes @xcite . tom has thus been widely used as a protocol to evaluate the language understanding and reasoning ability of intelligence agents @xcite , such as young children @xcite . with the recent advance in large language models (llms), research has been undertaken to evaluate the language skills of llms using tom @xcite . most of the previous * contributed equally to this work. † corresponding author of this work. work has been confined to first-order and secondorder tom, where llms are asked to perform inference on others' belief of reality in one or two passes, e.g., the first and second-order questions in figure 2 (see section 2 for a more comprehensive discussion of tom background and the evaluation of tom in llms). higher-order tom, referring to third-order reasoning and beyond, requires recursive reasoning on others' beliefs in multiple passes. figure 1 shows a higher-order tom example from the tv series friends. in figure 1 , a character says "they don't know that we know they know we know" when she and the other character try to recursively identify the situation. such an example underscores that human beings are capable of higher-order tom in daily interactions. in addition, evidence shows that higher-order tom is not only essential to communicate effectively in complicated scenarios, such as multi-party conversations @xcite @xcite , but it also enables better emotional support and empathetic communication @xcite . however, because of a lack of higher-order tom datasets in the nlp community, there is significantly less research on higher-order tom compared to the lower orders. ## the hi-tom dataset to systematically examine how effectively llms reason theory of mind (tom) at different orders, each story is coupled with five questions that require the zeroth to fourth level of tom reasoning, respectively. following @xcite and @xcite , we automatically generate hi-tom stories. additionally, we manually review the generated stories, questions, and answers to ensure that they are consistent with each other, and they are logically correct. ## experimental results the experimental results also reveal the following noteworthy patterns: llms' performance decreases as there are more deception communications involved. the performance drops when deception communications are involved. to further investigate the models' reasoning abilities in handling deceptive agent communications, we plot the resulting accuracy versus the number of deceptive communication sentences ("deception times") per story for gpt-4 in figure 4 . as shown in figure 4 , as deception times increase from 0 to 4, the joint accuracy experiences drops of 32%, 18.1%, 16%, and 11% respectively for the four tom orders. this suggests that the deceptive agent communications challenge the llms in their tom reasoning process. ## implications on the future of nlp we believe our work has important implications on the future directions of nlp with respect to the two-way relation between artificial intelligence and human intelligence. enhance llms' tom ability from the perspective of human intelligence. according to @xcite , human decisions are supported and guided by the cooperation of two capabilities or two systems: system 1 for intuitive, imprecise, fast, and often unconscious decisions ("thinking fast"), and system 2 for more complex situations with logical and rational thinking ("thinking slow"). this theory has inspired works in computer vision and natural language processing communities to explicitly equip models with the two systems @xcite . through our examination of llms' tom ability, we find failure cases that resemble the characteristics of system 1 thinking; for instance, llms may invent causes and intentions ("hallucination"), or substitute an easier question for a difficult one ("insufficient reasoning-depth"). furthermore, the significant performance drop from zeroth to fourth order tom in hi-tom suggests that llms may be more inclined to system 1 thinking rather than system 2 thinking, as higher-order tom requires careful in-depth logical inference. however, there exists a line of research combining the symbolic reasoning process @xcite , which aligns with system 2 thinking, with connectionist paradigm or neural learning @xcite , which captures the intuitive and pattern recognition aspects of system 1 thinking @xcite . this integration holds the promise of enabling ai systems to perform complex tasks that require both logical deduction and statistical generalization. we believe our findings of tom limitations of llms alongside this previous line of research clearly points to a direction where neural and symbolic approaches are combined in order to achieve abilities that are more closely aligned to human intelligence. understanding humans through the lens of llms. tom plays a crucial role in understanding human intelligence, as it is an important aspect of human cognition that enables us to make inferences about others' thoughts, emotions, and behaviors. enabling progress in tom reasoning in llms entails progress in emulating the functioning of human mind, which in turn offers intriguing possibilities for gaining insights into human interactions and the emergence of intelligence. while it is important to recognize that the analogy between human and artificial intelligence has its limitations and is a subject of debate within the nlp community @xcite , recent research has explored the extrinsic understanding of human interactions through multi-agent systems @xcite . this approach allows us to observe how llms can mimic and simulate aspects of human behavior and communication. by studying llms and their intrinsic properties, such as the emergence of intelligence, we can gain valuable insights into the fundamental processes underlying human cognition @xcite . researchers have also developed methods to elicit human-like behavior from llms, providing further opportunities to explore and understand the capabilities and limitations of these models @xcite . while llms offer a close-up view of human-like language processing, it is crucial to approach the topic with caution and recognize the complexities and nuances of human intelligence and behavior. enhance llms' tom abilities for better nlp applications. in daily life, our tom ability plays a vital role in understanding others' intentions, therefore helping us in our communication. in hi-tom, we enable higher-order tom reasoning, which in turn can lead to improvements in llms performance on tasks such as deception detection, emotional support, empathetic communication, and others. additionally, since llms represent foundational models that are used across various nlp tasks and applications, enhancing the abilities of llms opens up exciting possibilities for improving specific nlp tasks that benefit from these models. ## conclusion in this paper, we introduce hi-tom, the first tom benchmark that contains higher-order tom tasks. we demonstrated that llms' performance suffers a significant drop in tom tasks from lower to higher order. by proposing hi-tom, we hope to address the challenges of tom in complicated scenarios and spark further research on enhancing the reasoning ability of llms. furthermore, we present our insights on the future of nlp and discuss potential directions for enhancing llms. our aim is to stimulate research that draws inspiration from human intelligence, strives to understand humans better, and ultimately leads to the development of nlp applications that better cater to the needs of humans.