Paper: MMToM-QA: Multimodal Theory of Mind Question Answering (ICLR-2024)

Reviewer comment (Experiments):
Related to the aforementioned comment, the textual parsing and information fusion rely heavily on pretrained large language models (e.g., GPT-4) and a collection of carefully-tuned prompts (for household scenarios), it is relatively unclear whether or not such a paradigm is able to accommodate the diversity of more complicated environments.

Author response:
## It is relatively unclear whether or not such a paradigm is able to accommodate the diversity of more complicated environments.

We want to clarify that our prompts are quite generic and we have already demonstrated the generalizability of the current textual parsing. In particular, we have not turned the prompt, but rather just designed a simple template that prompts GPT-4 to parse any given text to state predicates. The examples provided in the prompt only include about 5% of objects, actions, and state predicates. In the test set, there are many unseen objects, actions, and state predicates. However, we didn’t observe any failure in textual parsing. This demonstrates the generalization ability of the parsing method. We discussed how we may further apply the method to even more diverse environments in the revised Section 6.

Reviewer comment (Evaluation):
It appears that there is a significant variance in the performance of different types of problems, any idea why (e.g., GPT-4 fails drastically in setting 2.3)?

Author response:
## Why is there a significant variance in the performance of different types of problems?
1. GPT-4 excels at the basic belief inference question type (1.1), in which the person’s belief is the same as the true world state (true belief). However, it tends to confuse a person’s belief with the true world state and thus predicts the person’s belief to be the same as the true world state in 1.2 even though the person has a false belief (the belief is different from the true world state) in 1.2. It performs relatively well in 1.3 because the person always has a true belief in those questions. Its accuracy in 1.3 is not as high as 1.1, because 1.3 requires GPT-4 to keep track of the person’s belief over a long period and not just based on the latest action. This shows that GPT-4 has a limit in tracking a person’s belief over a long sequence of actions.
2. GPT-4 in general performs at the chance level for all goal types except 2.3 (which is lower than chance). In 2.3, the person updates the belief after checking inside a container and decides. Since they cannot find the goal object, they then close the container without picking any object from the container. GPT-4 consistently predicts that the person’s goal is to get one of the objects inside in 2.3 instead of predicting that the person is more likely to pursue objects that are not inside the container. We hypothesize that this is because GPT-4 fails to update the person’s belief based on the actions of opening and closing the container, and mistakenly thinks that the person opens the container because the goal has to be one of the objects inside the container. 

We have included these discussions in the revised Section 5.2 and Section 6.

Reviewer comment (Experiments):
MMToM-QA is flawed. VQA has been studied for years and the issues in crowdsourcing and annotations are clearly mentioned in many prior works. However, the paper lacks quantitative studies regarding these issues. For example, whether the questions designed can be attacked by counting the repeating semantic meaningful words, whether the text question already implied the answers. From Table 1, the high performance in text-only models may mean these flaws in dataset design.

Author response:
## VQA has been studied for years and the issues in crowdsourcing and annotations are clearly mentioned in many prior works. 

We want to respectfully point out that our benchmark is evaluating a fundamentally different problem than VQA. We also did not use any crowdsourcing and annotations when constructing the dataset. We further clarified this in the revised related work section. We also summarize three key differences between MMToM-QA and VQA as follows.
1. Our benchmark evaluates whether a model can **infer a person’s hidden mental state (beliefs and goals)** from observable information about the world state and actions (as explained in the introduction). This is different from questions tested in VQA. Questions regarding human activities in VQA do not evaluate Theory of Mind.
2. Prior works (e.g., Netanyahu et al., 2021) have also shown that models that perform well in human activity recognition do not perform well in Theory of Mind reasoning. This further provides empirical evidence about the difference between ToM and human activity recognition. 
3. Finally, our benchmark emphasizes the ability to extract useful information from **multimodal inputs**, which is not evaluated in VQA. In sum, our multimodal Theory of Mind benchmark studies a new problem that is fundamentally different from VQA benchmarks, both in terms of the evaluated model capacities, available modalities, and the construction of the benchmark. We further discuss this critical difference between our multimodal Theory of Mind in the revised related work section.

## From Table 1, the high performance in text-only models may mean these flaws in dataset design.

As shown in the examples in Figure 2, each question in our benchmark features a unique environment and human actions; scenarios between different types of questions are also drastically different. There are no simple patterns a model can use to guess the answers and achieve high accuracy in all types of questions. In our benchmark, some models perform better in the text-only condition compared to the video-only condition, because there is more state information provided in the text and the textual parsing is also easier than constructing scene graphs from videos. For instance, from the video, we cannot see what’s inside a container when it is closed or objects occluded by the person. However, we can know this crucial information from the text.  Additionally, in our human experiments, human participants also performed better in the text-only condition compared to the video-only condition, which further demonstrates that the results are not caused by adversarial attacks or simple word patterns, but by the nature of the ToM tasks and how they rely on different kinds of modalities. Findings on how humans and models perform in different conditions and question types shed light on what kinds of information are necessary when reasoning about different aspects of the mental state, which we discussed in the revised Section 6.

Reviewer comment (Experiments):
The conclusion that "large language models and large multimodal models still lack robust ToM capacity" is not convincing, as the paper only tests BIP-ALM on the proposed MMToM-QA dataset. There are many other datasets that evaluate models' understanding of people's activities, such as VQA [1] and VCR [2]. The paper should compare BIP-ALM to the state-of-the-art methods on these benchmarks and also cite these benchmarks in related works.

Author response:
## The conclusion that "large language models and large multimodal models still lack robust ToM capacity" is not convincing… There are many other datasets that evaluate models' understanding of people's activities.

In the revised Section 5 and Section 6, we discussed the failures exhibited by large language models and large multimodal models, explaining why they lack robust ToM capacity. Since we are evaluating Theory of Mind reasoning, which is entirely missing in VQA and VCR, we believe that VQA and VCR are not suitable for our goal. Our benchmark is complementary to existing benchmarks by focusing on a model’s ability to infer a person’s mental states (goals and beliefs) from observable actions. We have cited and discussed these benchmarks in the related work.

Reviewer comment (Evaluation):
The paper should do a better job of justifying the proposed task itself. The paper argues that Theory of Mind is important for developing machines with human-level social intelligence. However, it is unclear why having machines with “human-level social intelligence” is necessary. First, can “social intelligence” be measured or evaluated? It is still unclear how to define intelligence alone and how to measure it within the machine-learning context. Second, what types of applications will benefit from this skill?

Author response:
We further clarified the importance of social intelligence in the revised introduction. We also explained that we are focusing on one specific aspect of social intelligence, Theory of Mind (ToM), defined what Theory of Mind is, and discussed how it has been quantitatively evaluated in prior works. We also summarize it here:

1. Social intelligence is one of the fundamental aspects of human intelligence as cognitive science studies have pointed out (e.g., Wimmer & Perner, 1983; Gergely et al., 1995; Gergely & Csibra, 2003; Spelke & Kinzler, 2007). It serves as the foundation of human social interactions (Warneken & Tomasello, 2006) and social learning (Csibra & Gergely, 2006). 
2. Theory of Mind is a key aspect of social intelligence, which has been demonstrated to be important for building socially intelligent systems, such as assistive robots (e.g., Dautenhahn et al. 2007, Hadfield-Menell et al., 2016; Puig et al., 2023), AI teachers (e.g., Wang et al., 2021), autonomous vehicles (e.g., Chandra et al., 2020), and cooperative embodied agents (e.g., Bara et al., 2021; Sclar et al., 2022a). These are well-studied in ML/AI, robotics, and HCI literature. **Instead of focusing on the broad concept of social intelligence, our work focuses on Theory of Mind.** 
3. There have been many studies on design experiments to quantitatively evaluate humans’ Theory of Mind in cognitive science (e.g., Gergely & Csibra, 2003; Baker et al., 2009; Ullman et al., 2009; Baker at al., 2017; Shu et al., 2020) and on developing benchmarks to **quantitatively** evaluate machine Theory of Mind in ML/AL (Rabinowitz et al., 2018; Kosinski, 2023; Sap et al., 2019; 2022; Ullman, 2023; Shapira et al., 2023; Shu et al., 2021; Moghaddam & Honey, 2023; Nematzadeh et al., 2018; Gandhi et al., 2021). Specifically, the experiments and benchmarks in the prior works evaluate the accuracy of goal inference and belief inference based on observed behaviors and use these quantitative evaluations as an indicator of whether a model has ToM capacity. Our benchmark builds on these prior works with significant extensions including more complex scenarios and multimodal inputs.

Reviewer comment (Evaluation):
Without a clear goal and definition, it is hard to validate whether the proposed benchmark aligns with the task described in the introduction of the paper. Can we draw conclusions from the results in the proposed benchmark about a model’s social intelligence? It is hard to say in the current setting.

Author response:
In our introduction, we have stated the goal of the benchmark, which is to evaluate whether a model can infer the mental state of a person, including goals and beliefs, from multimodal inputs of that person’s behavior (i.e., multimodal Theory of Mind). We have defined what Theoy of Mind is (i.e., the ability to infer a person’s mental state (belief and goal) based on the observed behaviors) and discussed how prior works have been evaluating Theory of Mind and building models to engineer machine Theory of Mind. Our benchmark quantitatively evaluates a model’s belief and goal inference accuracy of a model in different scenarios. It also evaluates the effect of different modalities on human ToM and machine ToM. It is well grounded in the prior works on Theory of Mind in both cognitive science and ML/AI and significantly extends the prior works.  As discussed in Section 5 and Section 6, the successes and failures of a model in different types of questions reveal their capacities as well as limitations in Theory of Mind.

Reviewer comment (Experiments):
The text is unclear about using synthetic videos in the test set. As it is written in Section 3.1, it looks like the 134 test videos are real image videos, whereas the 1000 training videos are synthetically generated. However, in Section 3.3, it looks like the test videos are also synthetic. If test videos are not synthetic, how are they obtained?

Author response:
All videos in both the test set and training set are synthetic and are procedurally generated in the same way. We made this clearer in the revised Section 3.3.

Reviewer comment (Experiments):
What are the statistics of the benchmark per question type?

Author response:
We added the number of questions for each question type in the revised Section 3.1.

Reviewer comment (Presentation):
Please include an overall column in Table 1 or the actual numbers in Figure 4.

Author response:
Thanks for the suggestion. We added the overall column in the revised Table 1.

Reviewer comment (Experiments):
More baselines like few-shot GPT4/3.5, evaluating LLM/Multimodal LLM with chain-of-thought types of reasoning process, open-sourced models with different sizes should be further studied to provide a better understanding of the performance of existing methods on this task.

Author response:
We did not evaluate few-shot GPT 4/3.5 because we followed the standard evaluation in the prior ToM QA evaluation (e.g., Shapira et al., 2023), which adopted zero-shot evaluation. We provided more explanation in the general response and clarified this in the revised Section 3.4. Per your suggestion, in the revision, we evaluated few-shot GPT 4/3.5, LLM/LMM with chain-of-thought, and open-sourced models with different sizes. We have discussed the results in Section 5.3. Specifically,

1. We found no meaningful improvement for almost all baselines after using different few-shot or chain-of-thought prompting. They still perform no better than chance. The only exception is GPT-4 in the text-only condition, which has an improvement in simpler types (e.g., Type 1.3) with few-shot prompting but still performs poorly on harder types, e.g., false belief (Type 1.2). 
2. We also found that all baselines performed no better than chance, regardless of the model sizes.

Reviewer comment (Evaluation):
Error analysis of existing models like GPT-4 or VideoLLAMA should be provided since it is important for the audience to understand how the proposed method solves the flaws in existing methods exactly. With the recent release of GPT4V, it will be great if some insights could be drawn from this case study as well.

Author response:
Thanks for the suggestion. We have revised Section 5 and Section 6 to include more error analyses. We also summarize them here:

1. GPT-4 excels at the basic belief inference question type (1.1), in which the person’s belief is the same as the true world state (true belief). However, it tends to confuse a person’s belief with the true world state and thus predict the person’s belief to be the same as the true world state in 1.2 even though the person always has a false belief (the belief is different from the true world state) in 1.2. It performs relatively well in 1.3 because the person always has a true belief in those questions. Its accuracy in 1.3 is not as high as 1.1, because 1.3 requires the GPT-4 to keep track of the agent’s person over a long period and not just based on its latest action, which shows that GPT-4 has a limit in tracking a person’s belief over a long sequence of actions.
2. GPT-4 in general performs at the chance level for all goal types except 2.3. In 2.3, the person updates the belief after checking inside a container and decides. Since they cannot find the goal object, they then close the container without picking any object from the container. GPT-4 consistently predicts that the person’s goal is to get one of the objects inside in 2.3 instead of predicting that the person is more likely to pursue objects that are not inside the container. We hypothesize that this is because GPT-4 fails to update the person’s belief based on the opens and closes container, and mistakenly thinks that the person opens the container because the goal has to be one of the objects inside the container. 
3. Large multimodal models, including Video-LLaMA and InstructBLIP fail to perform significantly better than chance for all types of questions, unlike GPT-4 which could excel in simple types. Additionally, we observe that captions generated by Video-LLaMA did provide useful information extracted from the visual inputs as the performance of captions + GPT-4 is better than GPT-4 alone as well as using Video-LLaMA to directly answer the questions. This demonstrates that while Video-LLaMA could provide useful captions describing

Reviewer comment (Experiments):
Currently the evaluation setting among different models is also not very clear. For example, models like VideoLLAMA actually takes video frames as input but other LLM models seem to only take parsed information. It is important to clearly annotate the exact format of each modality for all the model variants to make it clear to understand the possible difference from the input side.

Author response:
Only multimodal models (VideoLLaMA and InstructBLIP) can take video frames as inputs. Language models (GPT-4/3.5, GPT-J, LLaMA) can only take in parsed information. We clarified this in Section 5.1.

Reviewer comment (Experiments):
Why is the proposed method not applied to VideoLLAMA and Instruct BLip?

Author response:
Our method is Bayesian Inverse Planning Accelerated by **Language Models**. We use language models to evaluate action likelihoods based on parsed symbolic information. This allows us to fuse information from both visual inputs and language inputs into a unified representation. It also allows us to finetune language models only on symbolic representations and apply the finetuned language models to any type of modalities. In contrast, VideoLLaMA and InstructBLIP are not language models but multimodal models that directly take in both video frames and language inputs. They are not what our method relies on.

Reviewer comment (Experiments):
Is the synthetic video data really capturing important goal and belief in real world? What is the domain gap? There should be at least case studies on some real procedural videos, investigating the possible domain shift to validate the usage of synthetic video and revealing the possible limitation of this benchmark more explicitly.

Author response:
Thanks for the suggestion. 
1) We used the method proposed in the prior work (Puig et al., 2020) to synthesize people’s mental states and actions since the prior work has demonstrated that there is no significant difference between synthetic behaviors and real human behaviors. Cognitive science studies (e.g., Baker et al., 2017) have also verified that human mental update and action planning can be captured by POMDP, which is how we define the simulated agent in our work. 
2) To evaluate whether a model can understand real human mental states based on real human actions, we have additionally collected real human actions by using the human interface in the VirutalHome-Social platform. We then rendered videos based on collected **real human actions**. In particular, to create a human-generated video, we assigned a goal to a participant and asked the participant to control the avatar based on the egocentric observation to reach the goal. The resulting human actions are then used to render the video. We recruited 3 participants who had no prior exposure to the system. In total, we created 40 human-generated videos and created 120 questions as the human test set. The mental state updates in these human-generated videos are the real human mental state updates (human participants had to update their own beliefs to control the avatar to reach the assigned goal). Moreover, we used 2 new apartments that are unseen in the original test set and the training set in these human videos, so that we can further evaluate the generalizability of our method in unseen environments. Results are reported in Section 5.3 and Table 7 in Appendix B.1. It shows that our method performs at the same level in the human test set compared to the main test set. It demonstrates that our method can successfully generalize to real human behavior in unseen environments. It also suggests that synthetic behavior captures how humans update mental states and take actions.
We leave the investigation of real human activities in broader goals for future work. Compared to prior ToM benchmarks which are all generated in toy domains, our benchmark has already taken a significant step toward ToM in realistic and complex scenarios.