Textual Time Travel: A Temporally Informed Approach to Theory of Mind (2021)

Abstract: Natural language processing systems such as dialogue agents should be able to reason about other people’s beliefs, intentions and desires. This capability, called theory of mind (ToM), is crucial, as it allows a model to predict and interpret the needs of users based on their mental states. A recent line of research evaluates the ToM capability of existing memory-augmented neural models through question-answering. These models perform poorly on false belief tasks where beliefs differ from reality, especially when the dataset contains distracting sentences. In this paper, we propose a new temporally informed approach for improving the ToM capability of memory-augmented neural models. Our model incorporates priors about the entities’ minds and tracks their mental states as they evolve over time through an extended passage. It then responds to queries through textual time travel–i.e., by accessing the stored memory of an earlier time step. We evaluate our model on ToM datasets and find that this approach improves performance, particularly by correcting the predicted mental states to match the false belief.

## introduction humans have evolved social intelligence to reinforce cooperation in society @xcite . in human interactions, understanding another person's mental states -for example, their purpose, intention, knowledge, belief, thoughts, doubts, likes and needs -is a critical step in correctly interpreting or predicting their behavior @xcite . this ability to attribute mental states to oneself and to others, called theory of mind (tom), becomes increasingly necessary for natural language processing (nlp) systems as they integrate into modern society. acquiring tom capabilities permits more accurate responses in several situations. for example, it allows for disambiguating a difficult query by correctly deducing the true needs of the user based on their mental state, thereby providing the missing piece in solving inference and reasoning tasks. figure 1 shows a scenario with two people interacting with an intelligent virtual assistant. in this example, the assistant (bot) uses theory of mind to reason about ella's belief about the location of the umbrella, which differs from reality. clearly, demonstrating social intelligence is a major barrier that needs to be crossed in our march towards the applied endgoal of creating nlp systems that blend seamlessly into the human world @xcite . a recent line of work evaluates the theory-ofmind capability of memory-augmented neural models via a question answering task @xcite @xcite . this task consists of stories with multiple entities that interact with each other in a synthetic environment, followed by a question about the entities' beliefs. memory augmented neural models like entnet @xcite , that are successful at solving reasoning tasks such as babi @xcite by tracking world states, perform poorly at false belief tasks where the mental states of the entities do not match the world states. moreover, these models are sensitive to distracting sentences, which decrease their accuracy. in this paper we propose a new model, that we call textual time travel, to correctly track the mental states of the entities when they have a false belief. our key insight is to incorporate priors about the entities' minds in our model and add a temporal dimension to the neural model's memory. this allows us to track the changes in the mental states with time as the story progresses. 1 we also aim to have an interpretable model of the world and the mental states of the entities. our temporallyinformed neural model allows us to visualize how it tracks changes in these states. we find that our model does indeed track the mental states of the entities, and with additional supervision it can provide correct responses when the entities have false beliefs, thus improving performance. ## related work theory of mind has been extensively researched by the psychology community over the last few decades @xcite . generally, false belief tasks are used to test the tom capabilities of children and animals @xcite @xcite @xcite . in developmental psychology, the famous sally-anne test @xcite is widely adopted to assess a child's ability to attribute false beliefs to others. ## theory of mind datasets the tom and tomi datasets contain multiple stories with a structure similar to the sally-anne and ice-cream van tests, but with a variety of entities and objects. the tom dataset @xcite follows a strict template to generate the stories and has a simple random sentence as a distractor. the tomi dataset @xcite aims to addresses the dataset biases of tom by breaking the strict event sequence. it adds actions involving unrelated entities and randomizes the order of events in the story. each story has two main entities, an object of interest, a main location where the events take place, and two containers for the object. the story begins with the introduction of the two entities, the object and their locations. tomi consists of 1000 stories of 3 story types: each story is followed by a question. the model must provide an answer based on the mental states of the entities. in this task, the mental states refer to the beliefs of an entity about the locations of entities and objects in the artificial environment. each story is included in the dataset six times, once for each question that is associated with it, distributed by type as follows: ## experiments as described above, to train the learned textual time travel model we add "what is the location @xmath0. we then randomly sample 1000 of these stories for the training dataset. we initialize the prelu slopes to 1, and initialize all the other weights with values drawn from a gaussian distribution with mean zero and standard deviation 0.1. we set the key w j to the word embeddings of all the named entities in the dataset, and the memory cell contents h (0) j are initialised with w j . we remark that initializing w j with glove @xcite pre-trained word embeddings did not improve performance. we use the adam optimizer (kingma and ba, 2015) with a batchsize of 32, and clip the gradients at 40. we start with an initial learning rate of 0.01 and we halve the learning rate after every 25 epochs. we train the models on the corrected tomi dataset for 200 epochs. ## results what is the location of abigail? unknown 2. carter entered the porch. what is the location of abigail? unknown 3. abigail entered the porch. what is the location of abigail? mainloc(porch) 4. the potato is in the green-suitcase. what is the location of abigail? mainloc(porch) 5. abigail exited the porch. what is the location of abigail? unknown 6. abigail entered the hall. what is the location of abigail? altloc(hall) 7. carter moved the potato to the green-envelope. what is the location of abigail? altloc(hall) 8. oliver entered the hall. what is the location of abigail? altloc(hall) 9. where will abigail look for the potato? ## discussion in this section, we describe an ablation study and analyze the model by considering its internal memory gate activations, and by examining its errors. ## conclusion tom is an important capability that nlp systems need to acquire in order to have human-like reasoning abilities. understanding and predicting the mental states of others will help in comprehending their intentions and needs, and thereby generate better responses in interactive systems like dialogue agents. in this paper, we attempt to improve the tom abilities of memory-augmented neural models by building priors about the entities' minds and performing textual time travel (i.e., retrieving the metal states of entities from earlier timesteps). we find that our heuristic and learned textual time travel approaches improve performance, particularly on false belief tasks. starting from synthetic datasets like tomi is necessary because they allow the development and testing of new techniques in controlled environments. these datasets act as prerequisites for new models to pass, and models that fail on these are unlikely to scale to real world data. in a naturalistic setting like qa and dialogue, it is much harder to find instances involving false beliefs automatically. more importantly, identifying and controlling for confounding variables in naturalistic data could be more difficult. for example, issues such as recency or lexical overlap might result in clever hans phenomena as shown for nli @xcite . demonstrating that our approach works for tomi is a first step towards building models with complete tom capabilities. theory of mind is a complex problem and we believe that we can make progress by gradually increasing the complexities of the tom tasks in a controlled setting. this work is a part of a bottom-up process for solving tom. to this end, our approach adds the missing piece of incorporating mental-state tracking along the time axis. with this prerequisite met and barrier crossed, we can move towards tackling other challenges in tom in the future.