FANT o M : A Benchmark for Stress-testing Machine Theory of Mind in Interactions (2023)

Abstract: Theory of mind (ToM) evaluations currently focus on testing models using passive narratives that inherently lack interactivity. We introduce FANToM, a new benchmark designed to stress-test ToM within information-asymmetric conversational contexts via question answering. Our benchmark draws upon important theoretical requisites from psychology and necessary empirical considerations when evaluating large language models (LLMs). In particular, we formulate multiple types of questions that demand the same underlying reasoning to identify illusory or false sense of ToM capabilities in LLMs. We show that FANToM is challenging for state-of-the-art LLMs, which perform significantly worse than humans even with chain-of-thought reasoning or fine-tuning.

## introduction existing evaluations for language models' theory of mind (tom) -i.e., the ability to understand the mental states (e.g., thoughts, beliefs, and intentions) of others @xcite , is primarily focused on using situation descriptions (i.e., narratives) as the target domain @xcite @xcite . however, tom capabilities play an even more important role in understanding dynamic social interactions, as they form a crucial component of effective communication @xcite . furthermore, as narratives condense situation information into short texts, reporting biases can cause them to include spurious correlations or surface cues @xcite . these can be exploited by large language models (llms) to display illusory tom -i.e., a false sense of robust social reasoning by models. 2 in this work, we introduce fantom, an english benchmark for stress-testing machine tom in interactions -i.e., conversations. as conversations present interactions in their raw form, they are much less susceptible to reporting biases, and are more aligned with real-world scenarios requiring tom reasoning. fantom consists of 10k questions covering 256 multiparty conversations around a certain topic while characters enter and leave the discussion, leading to distinct mental states between characters due to information asymmetry. the goal of fantom is to effectively measure how well models can track the belief of multiple characters in conversations where some information may be inaccessible to some participants. for example, in figure 1 , kailey briefly steps away from the conversation to get a cup of coffee, while the others continue discussing linda's new dog. the information exchanged during kailey's absence remains unknown to kailey, and only the information shared after kailey's return becomes accessible. we convert factual question-answer pairs to obtain multiple challenging questions about characters' beliefs concerning the inaccessible information. our aim is to design questions at different levels that evaluate a model's capability for a coherent understanding of others' mental states. in doing so, we are particularly interested in identifying instances of illusory tom, which we define as situations where a model may answer some questions correctly but fails to answer others that require the same type of tom reasoning. the analysis of evaluation results on fantom reveals several interesting findings ( §4): (1) first, existing neural models score significantly lower than humans on individual questions and on the full set of questions by more than 70% on average. (2) while chain-of-thought reasoning (cot) does improve performance in most models, it does not substantially bridge the gap with human performance. (3) although our benchmark is not meant for training, we observe that fine-tuning can help models achieve scores higher than human performance on individual question types. however, when it comes to metrics that require coherent responses across multiple question types, the finetuned model still significantly underperforms compared to humans. (4) additionally, we find that models exhibit different error types depending on the format of questions, despite all questions requiring the same underlying reasoning. (5) moreover, our results indicate that cot has a selective impact on performance, showing improvement only in specific scenarios. to the best of our knowledge, fantom is the first benchmark to introduce conversation-based tom evaluation for language-based models. our benchmark design and experiment results yield important insights into the debate around tom @xcite and the development of artificial general intelligence @xcite in llms. we release our benchmark to spark further discussions on evaluating the tom capabilities of llms. ## design considerations for fantom we go over the important design choices that we made when constructing fantom. our goal is to incorporate (1) social interactions that necessitate natural theory of mind (tom) reasoning ( §2.1), (2) essential theoretical prerequisites for validating tom from psychology ( §2.2), and (3) empirical findings that must be taken into account when evaluating large language models ( §2.3). ## fantom overview following the success of previous works @xcite , we automatically construct full conversations using the large language model (llm) instructgpt davinci-003 @xcite . we also generate theory of mind (tom) question-answer pairs related to the conversation participants' beliefs using a specially designed pipeline. in preliminary explorations, we find off-the-shelf llms struggle with directly generating tom question-answer pairs for a given conversation. our pipeline consists of three steps: (1) generate conversations with information asymmetry ( §3.1), ( 2 ) generate fact question-answer (qa) pairs ( §3.2), and (3) construct tom (e.g., belief) qa pairs from the fact qa pairs ( §3.3). we use different evaluation methods for each question types ( §3.4), and validate the final dataset ( §3.5). ## experiments baseline models we test a total of thirteen recent instruction-tuned neural language models: gpt-4 (gpt-4-0613 and gpt-4-0314; openai, 2023), chat-gpt (gpt-3.5-turbo-0613; openai, 2022), instruct-gpt (davinci-003 and curie-001; @xcite , flan-t5-xl and flan-t5-xxl @xcite , flan-ul2 @xcite , falcon instruct (7b and 40b; @xcite , mistral instruct 7b @xcite , zephyr 7b (huggingface, 2023), and llama-2 chat 70b @xcite . descriptions for each model are in appendix b. although our benchmark is not meant for training, we also fine-tune flan-t5-xl @xcite by randomly splitting fantom according to the conversation's main topics. we then test the model on unseen conversation topics. more details can be found in appendix b. ## conclusion & discussion we introduced fantom, a new benchmark for stress-testing theory of mind (tom) capabilities of neural language models in conversations via question answering. our benchmark is built upon essential theoretical requisites and empirical considerations required for validating tom in large language models (llms). the conversations in our benchmark involve information asymmetry, with characters joining and leaving the discussion while it continues, to simulate distinct mental states. to identify illusory tom, we crafted multiple types of challenging belief questions regarding the conversation participants' mental states by converting factual questions. our evaluation results show that coherent tom reasoning is challenging for current llms, performing significantly worse than humans even when using chain-of-thought reasoning or fine-tuning. although there has been recent debates around whether current llms possess tom capabilities or not (whang, 2023), our results indicate that this capacity has not yet emerged in any manner. previous instances of success on well-known psychology tom tests may be attributed to exposure during the pretraining phase @xcite . our work highlights the need for novel interaction-oriented benchmarks that introduce scenarios not encountered during training, and also aligning more closely with real-world use cases as llms are increasingly being deployed in interactive settings. our results also shed light on a broader issue in neural models -the lack of internal consistency @xcite . we find they often fail to provide consistent answers to questions requiring the same underlying tom reasoning. to address this concern, future works can explore various directions, such as grounding reasoning in pragmatics @xcite , visual information @xcite , or belief graphs @xcite . another issue that our work touches upon is the reporting biases inherent in language models. we observed that models often exhibit biases in their responses, showing a tendency to overly rely on the information they are conditioned on, such as preferring answers that have high overlap with the context @xcite . however, to achieve successful tom reasoning, it is crucial to distinguish between accessible and inaccessible information for a particular agent, rather than blindly using all information available to the model. one potential approach to mitigate this is to combine pretraining with interactive learning @xcite . in the spirit of encouraging future research in this direction, we make our benchmark publicly available at https://hyunw.kim/fantom .