T o MC hallenges: A Principle-Guided Dataset and Diverse Evaluation Tasks for Exploring Theory of Mind (2023)

Abstract: Theory of Mind (ToM), the capacity to comprehend the mental states of distinct individuals, is essential for numerous practical applications. With the development of large language models (LLMs), there is a heated debate about whether they are able to perform ToM tasks. Previous studies have used different tasks and prompts to test the ToM on LLMs and the results are inconsistent: some studies asserted these models are capable of exhibiting ToM, while others suggest the opposite. In this study, We present ToMChallenges, a dataset for comprehensively evaluating the Theory of Mind based on Sally-Anne and Smarties tests with a diverse set of tasks. In addition, we also propose an auto-grader to streamline the answer evaluation process. We tested three models: davinci, turbo, and gpt-4. Our evaluation results and error analyses show that LLMs have inconsistent behaviors across prompts and tasks. Performing the ToM tasks robustly remains a challenge for the LLMs. In addition, our paper wants to raise awareness in evaluating the ToM in LLMs and we want to invite more discussion on how to design the prompts and tasks for ToM tasks that can better access the LLMs’ ability.

## introduction with the recent advancement of large language models (llms; @xcite @xcite , expectations for artificial intelligence systems to effectively interact with people have significantly increased. this may necessitate the development of human-like capabilities in these systems, such as reasoning not only about their own observations and beliefs but also understanding the mental states of others. this ability, termed as theory of mind (tom), refers to the capacity to attribute mental states-such as beliefs, 1 the data and code are available at https://github.com/xiaomeng-ma/tomchallenges . emotions, and intentions-to oneself and others @xcite . in psychology, it is an essential milestone in the social development of a child. however, the challenges that persist are whether llms have already developed tom capabilities and how to identify the appropriate tool to accurately assess these capabilities. recent studies addressing those issues often draw inconsistent conclusions, some studies asserting that models exhibit tom @xcite @xcite , some suggest the opposite @xcite @xcite @xcite , and others maintain caution and questions @xcite . these varied results could be due to different evaluation methods. first, these studies have tested the models on different tasks, ranging from tasks of perspective-taking reasoning (i.e., does the other person know what i know; e.g., @xcite to intention ascription (i.e., what does a movie character intend to do at the end of an open-ended movie; e.g., @xcite . additionally, the type of prompts varies across studies. for in-stance, @xcite and @xcite used question answering prompts, while @xcite employed sentence completion prompts. this lack of clear principles in approaches poses challenges to the validity of tom assessments for llms. if only specific prompts lead to high-performance results while others do not, it becomes questionable whether the correct responses truly reflect tom or are simply the result of algorithmic shortcuts. similarly, if some tasks are not valid for assessing tom, the results cannot be interpreted in terms of models' tom capability regardless of the conclusions drawn. what is considered a valid tom test? a valid test should be both theoretically grounded and methodologically validated to ensure it measures the intended subject, and the results are not skewed by other factors. from a theoretical standpoint, tom theories in child development @xcite @xcite suggest that valid tests should focus on assessing the respondent's ability to a) represent mental states of one's own and others based on physical events (but not other factors such as emotions and intentions) (mentalizing), and b) differentiate one's own mental state and other's (false-belief understanding). tasks not meeting these criteria might not be considered valid assessments because they either introduce confounding factors such as emotional or social ascription or fail to contrast the respondent's mental state and other's mental state. from a methodological perspective, both psychology and nlp studies demand rigorous evaluation to ensure measurement validity. unlike psychology studies where individual subjects can be randomly assigned to experimental and control conditions to yield reproducible results, llms like gpt-4, being a single 'subject', lack the capacity for reproducibility in the traditional sense. therefore, any claims about an llm possessing humanlike capabilities must be substantiated after validation with a variety of prompts and tasks, provided these tasks align with the theoretical framework of the intended measurement. ## tomchallenges and tasks we aim to build a corpus based on two types of tests: sally-anne test and smarties test, which fit the tom test criteria. below we describe how we construct tomchallenges data, and how we design our evaluation tasks. ## results and analyses in this section, we present the results of our evaluation for all models on sally-anne and smarties tests. as we create 30 variations of the narrative for each test, and each narrative comes with 6 questions (reality, belief, 1sta, 1stb, 2nda, 2ndb), and each question is tested on 6 tasks, an idealized model that is capable to solve theory of mind tasks should be able to achieve high accuracy on all questions across different tasks and in most of the narratives. ## conclusions in this study, we proposed tomchallenges to comprehensively test the tom on llms. the dataset is constructed based on the sally-anne and smarties tests. for each test, we created a template to generate variations of the test. in addition, we incorporated 6 types of questions to examine the model's understanding of reality, belief, 1st order belief, and 2nd order belief. we also included 6 tasks with different prompts for evaluation, considering the impact of prompts on model performance. this evaluation method serves a dual purpose: it not only measures whether the model has tom capacity, but also measures the robustness of the model in performing the tom tasks. in addition, we also create an effective auto-grader that achieved high accuracy in evaluating the more free-formed answers of the tom tasks. using 30 variations of sally-anne and smarties tests, we found that the gpt-based models can not reliably perform the tom tasks, since the accuracy varies across different tasks, questions, and narratives. through error analysis, we found that the failure of the models can not be simply concluded as they lack the ability to reason reality and beliefs. instead, many of the errors reflect the inherent limitations of the llms, which might not be related to the cognitive explanations of the tom. we hope our study could invite more discussions on evaluating the tom in llms. further studies could investigate how and why different prompt types would affect the model's performance, how to improve the robustness of the model's performance, how to interpret the results more appropriately, etc. in addition, one limitation of our study is that we did not conduct human experiments. it is unclear if human adults are able to achieve (almost) perfect performance across different stories, prompts and tasks. future studies could also compare human results and the llms' behaviors side by side for evaluation. we hope that future discussions could not only focus on whether llms exhibit the tom, but also attend to the significance and implications of the tom in llms.