Minding Language Models’ (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Tracker (2023)

Abstract: Theory of Mind (ToM)—the ability to reason about the mental states of other people—is a key element of our social intelligence. Yet, despite their ever more impressive performance, large-scale neural language models still lack basic theory of mind capabilities out-of-the-box. We posit that simply scaling up models will not imbue them with theory of mind due to the inherently symbolic and implicit nature of the phenomenon, and instead investigate an alternative: can we design a decoding-time algorithm that enhances theory of mind of off-the-shelf neural language models without explicit supervision? We present SymbolicToM, a plug-and-play approach to reason about the belief states of multiple characters in reading comprehension tasks via explicit symbolic representation. More concretely, our approach tracks each entity’s beliefs, their estimation of other entities’ beliefs, and higher-order levels of reasoning, all through graphical representations, allowing for more precise and interpretable reasoning than previous approaches. Empirical results on the well-known ToMi benchmark (Le et al., 2019) demonstrate that SymbolicToM dramatically enhances off-the-shelf neural networks’ theory of mind in a zero-shot setting while showing robust out-of-distribution performance compared to supervised baselines. Our work also reveals spurious patterns in existing theory of mind benchmarks, emphasizing the importance of out-of-distribution evaluation and methods that do not overfit a particular dataset.

## introduction reasoning about other people's intentions, desires, thoughts, and beliefs is a cornerstone of human social intelligence. children naturally develop an understanding of every individual's unique mental state and how it might impact their actions @xcite . known as theory of mind (tom) @xcite , this ability is crucial for efficient and effective communication. alice and bob are in a room with a basket and a box. alice puts some celery in the basket and leaves the room. bob then moves the celery into the box. where will bob search for the celery? (*) where does bob think that alice will look for the celery when she returns? (**) cognitive and literary studies have extensively argued theory of mind's key role in understanding stories, in order to explain and predict each character's actions @xcite @xcite . as exemplified in figure 1 , readers need to model bob's mental state (called first-order tom), as well as bob's estimation of alice's mental state (second-order tom) to answer questions. despite recent progress in language understanding abilities, large language models have been shown to lack theory of mind skills @xcite . existing efforts to enable them have primarily relied on supervised methods (e.g., @xcite @xcite . however, current reading comprehension datasets for theory of mind reasoning are simplistic and lack diversity, leading to brittle downstream models which, as we show, fail in the presence of even slight out-of-distribution perturbations. we introduce symbolictom, an inferencetime method that improves large language models' theory of mind capabilities by augmenting them with an explicit symbolic graphical representation of each character's beliefs. unlike prior efforts, our approach does not require training and instead divides the problem into simpler subtasks, leveraging off-the-shelf models to solve them, and carefully consolidating their results. this makes symbolic-tom significantly more robust than existing models trained specifically for theory of mind behavior. while beliefs about the world state differ among people, most existing work on encoding belief states do not model this behavior relying on singular graphs @xcite . symbolictom, instead, utilizes a set of graphs, each representing what the character p 1 thinks that p 2 believes that [...] p m assumes to be the current state of the world, where m is the maximum reasoning depth as determined by the user. this explicit, recursive mental state representation enables the model to answer questions from the perspective of each character. symbolictom's process of selecting and querying a particular character's graph grounds it in cognitive science research arguing theory of mind as an essential mechanism of selective attention @xcite . our approach also instills desirable inductive biases, such as object permanence-for example, object locations (represented by edges in the graphs) are assumed to be constant until the method can infer a change. although existing nlp datasets only test up to second-order reasoning (i.e., @xmath0, symbolictom is designed to work at any depth. symbolictom dramatically improves the performance of large language models in theory of mind reading comprehension tasks. for example, gpt-3-davinci's @xcite accuracy on the tomi benchmark @xcite increases by 38 absolute points using symbolictom (yielding 92% accuracy averaging across question types). furthermore, we extend the tomi test sets with diverse story structures and sentence paraphrases and demonstrate that our approach is significantly more robust than supervised approaches. ## motivation and background although large-scale language models have recently shown improvements in some classic theory of mind examples, they are still far from reliably showing theory of mind capabilities @xcite @xcite . while the training data for these models includes human-written stories which require theory of mind reasoning, this information is largely implicit and hence difficult for models to learn. chatgpt and gpt3-davinci's incorrect answers to figure 1 's question #2 are shown below. 1 chatgpt (gpt-3.5-turbo): based on the information provided, bob would likely think that alice will look for the celery in the box when she returns. since bob moved the celery from the basket to the box, he would assume that alice would expect to find it in its new location. gpt3 (text-davinci-003): bob will likely think that alice will look for the celery in the box, since that is where he moved it. natural stories which make theory of mind explicit are scarce, necessitating automatically generated, template-based datasets like tom-babi @xcite and tomi @xcite . however, templated narratives cover limited types of interactions, and include only simplistic discourse and sentence structures. on the other hand, relying on human-generated data, e.g., in situated dialogue @xcite , leads to barriers in dataset size due to high annotation costs. moreover, another source of data-text-based games with multiple characters-also faces limitations; in particular, modeling mental states is required mainly to infer intents @xcite and to maintain a consistent style of each character @xcite . rather, in this work, we aim to study and evaluate differences in knowledge and beliefs among multiple characters, traditional cognitive aspects of theory of mind. to the best of our knowledge, the only available datasets for measuring theory of mind in reading comprehension tasks are tom-babi and tomi. because of their templated nature, supervised training on them is prone to overfitting to spurious artifacts in the data. while tomi was developed to counter this behavior in tom-babi by introducing noise in the form of flexible sentence ordering and distractor sentences and characters, we show it still faces the same pitfalls. ## fundamental issues in existing tom datasets construction of tomi as introduced in §2, the sole large-scale theory of mind dataset for reading comprehension tasks is tomi @xcite . barring its added distractor characters and sentences, tomi strictly mimics the sally-anne test, a widely adopted evaluation for assessing children's social cognitive ability to reason about others' mental states @xcite . stories are structured are as follows: characters a and b are in a room, and a moves an object from an opaque container to another; b may or may not leave the room before a moves the object. b will know the object's new location if and only if they were in the room at the time it was moved. four types of tom questions are posed: first-order or second-order, probing a character about either a true or a false belief (i.e, belief that matches reality or not). tomi also includes questions probing about reality (or zeroth-order tom, @xcite and memory. tomi has six types of sentences (i.e. six primitives) with set phrasing. these include someone (a) entering or (b) exiting a room; the location of (c) an object or (d) a person; (e) someone moving an object; and (f) someone's opinion about an object (distractors). primitives are combined into stories with a finite list of possible orderings. despite the limited types of primitives, correctly answering questions requires high-order levels of reasoning. templated stories are filled with randomly sampled objects, locations, containers, and rooms from a set list. tomi implicitly assumes that questions about the story do not depend on these decisions, only on the underlying story template. yet, in a small-scale human study, we find physical com-1. oliver entered the front yard. 2. ethan entered the front yard. 3. liam entered the kitchen. 4. objecta is in the basket. 5. ethan exited the front yard. 6. ethan entered the kitchen. 7. oliver moved objecta to the containerx. 8. where does ethan think objecta is? tomi gold label: basket to solve these unintentional ambiguities and additional 3-hop reasoning requirements, and instead solely measure theory of mind reasoning skills, we automatically add a sentence that disambiguates the location of each container immediately after each primitive (c) or (e) (e.g., adding "the basket is in the front yard" as line 5 in ## experiments we experiment with several base lms, and evaluate each of them both out-of-the-box via zeroshot prompting, and by applying symbolictom to tomi stories to produce answers. we evaluate macaw-3b (tafjord and clark, 2021), gpt3-{curie,davinci} @xcite , flan-t5-{xl,xxl} (chung et al., 2022), llama-{7b, 13b} @xcite , gpt3.5 (openai, 2022), and gpt4 (openai, 2023). we use wanli @xcite for identifying nli contradictions, and the allennlp library @xcite for openie. we additionally refine each subject and object in extracted triples to remove any stopwords that may be accidentally included by openie. we first evaluate symbolictom's performance as a plug-and-play method for different base lms on tomi ( §5.1). we then test whether performance gains are robust to tomi story structure modifications ( §5.2). finally, we explore symbol-ictom's robustness to linguistic diversity ( §5.3). supervised models for comparison, we train two supervised models: textual time travel (ttt) @xcite , and a fine-tuned gpt3-curie. ttt is a modification of entnet @xcite designed for theory of mind tasks; gpt3-curie is finetuned on 6000 tomi examples for one epoch. gpt3-curie achieves near-perfect performance when finetuned on tomi (98.5% accuracy when averaging all questions; ## conclusions theory of mind is an essential social intelligence ability. developing agents with theory of mind is requisite for a wide range of applications, including reading comprehension, tutoring, dialogue, personalization, and negotiation. for example, in reading comprehension settings (and broadly for natural language understanding), having a multi-level understanding of texts is crucial for providing meaningful and contextualized answers: stories often rely on theory of mind reasoning to create conflict (e.g., in murder mysteries, drama, and romances, as in the final acts of romeo and juliet). we present symbolictom, a plug-and-play method to enable theory of mind reasoning in language models via explicit symbolic representations in the form of nested belief states. symbolic-tom requires no training or fine-tuning, a key aspect for a domain with scarce supervised data and limited success in learning from massive unlabeled text alone. with experiments on reading comprehension tasks, our approach demonstrates dramatic improvement in the accuracy of base language models, especially for false-belief scenarios. we also show that, in contrast to supervised methods, symbolictom is highly robust to story perturbations and out-of-domain inputs where supervised methods suffer significant degradations (as in, e.g., @xcite . 5 our results show the promise of augmenting neural language models with symbolic knowledge for improving their social reasoning skills. we leave to future work to investigate similar approaches for other types of social intelligence; as well as develop new datasets that cover a more diverse set of interactions.