Paper: How FaR Are Large Language Models From Agents with Theory-of-Mind? (ICLR-2024)

Reviewer comment (Novelty):
The authors mention that the main contribution of this work is evaluating the ability of LLMs to take actions based on ToM inferences, but do not mention a paper that does the same: "Understanding Social Reasoning in Language Models with Language Models", Gandhi et al. 2023. This work also tests the ability of LLMs to take appropriate actions in Sally-and-Anne-type scenarios, evaluate humans on it as well, and find similar results (models struggle more with the actions than with the inferences).

Author response:
__[Main concern with this paper is contribution in light of Gandhi et al. 2023. doing the same]__

Gandhi et al. 2023. proposes BigToM, which is an excellent benchmark for measuring ToM in LMs. While there is a surface-level similarity in the task setups (both contain “action” probing), there is an important distinction between action probing in BigToM and T4D: probing beliefs of others’ actions such as “*What will A do?*” (BigToM) vs. probing model’s own action decisions based on beliefs of others such as “*Who should you provide help with?*” (T4D).  Specifically, in T4D, models have to treat themselves as situated agents and decide their **own** actions, based on potential reasoning about others’ mental states as well as predictions of others' actions. We argue that these are complementary directions, and both are important to understand LLMs’ capabilities related to ToM. In summary, T4D builds on previous benchmarks by focusing on models’ capabilities to determine actions for themselves based on mental state reasoning. We have added this in the extended related work section in the updated draft in Appendix A.

(Gentle note that according to the last question in FAQ from ICLR Reviewer Guide (https://iclr.cc/Conferences/2024/ReviewerGuide), “if a paper was published (i.e., at a peer-reviewed venue) on or after May 28, 2023, authors are not required to compare their own work to that paper.” and “they may be excused for not knowing about papers not published in peer-reviewed conference proceedings or journals, which includes papers exclusively available on arXiv.” and Gandhi et al. 2023. Was released on arXiv on June 21, 2023).

Reviewer comment (Experiments):
The authors apply no control conditions, like true belief situations, to further investigate whether models are actually doing ToM-reasoning (which is common in research investigating ToM in LLMs).

Author response:
__[Controlled experiment with true beliefs]__

Thanks for the great point! We did controlled experiments with true beliefs (i.e., the item stays in the same location) paired with all our experiments with false beliefs. We have added these results in the appendix F. Notably, since we are probing the model agent's action choices given observations, when all characters hold true beliefs, it is unclear to whom the model agent should help provide information. So, instead of assigning gold labels for the controlled set, we calculate the delta value between the false belief set and the true belief set. The higher delta indicates that the model is more inclined to help when there is a false belief, and the lower indicates that the model might not be using ToM to decide actions space  (since it is similarly likely to choose the character with or without false beliefs). We found that the delta is very low (<5%) for all LLMs when tested 0-shot, and applying FaR helps increase the delta values to around 30%. This indicates that models do not tend to leverage mental state thinking in T4D, but FaR can help them locate relevant ToM inferences to determine the right action options.

Reviewer comment (Experiments):
Not necessarily a weakness but I'd like to see some discussion; could there be spurious correlations in the examples? E.g. in Figure 2 Owen is mentioned more often than the others and is also the answer.

Author response:
__[Could there be spurious correlations in the examples?]__

ToMi was extended from an earlier work [1] that uses QA to probe ToM using sally-anne tests by removing biases from the character name mentions (discussed in Section 3 of the ToMi paper [2]). Furthermore, we included an observation-answer-only baseline without the questions and found close-to-random performance, confirming that there is no strong spurious correlation between the observation and the answers.

[1] Aida Nematzadeh, Kaylee Burns, Erin Grant, Alison Gopnik, and Tom Griffiths. Evaluating theory of mind in question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2392–2400, 2018.

[2] Le, Matthew, Y-Lan Boureau, and Maximilian Nickel. "Revisiting the evaluation of theory of mind through question answering." Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

Reviewer comment (Novelty):
To summarise; my main concern with this paper is contribution in light of another work doing the same.

Author response:
__[Main concern with this paper is contribution in light of Gandhi et al. 2023. doing the same]__

Gandhi et al. 2023. proposes BigToM, which is an excellent benchmark for measuring ToM in LMs. While there is a surface-level similarity in the task setups (both contain “action” probing), there is an important distinction between action probing in BigToM and T4D: probing beliefs of others’ actions such as “*What will A do?*” (BigToM) vs. probing model’s own action decisions based on beliefs of others such as “*Who should you provide help with?*” (T4D).  Specifically, in T4D, models have to treat themselves as situated agents and decide their **own** actions, based on potential reasoning about others’ mental states as well as predictions of others' actions. We argue that these are complementary directions, and both are important to understand LLMs’ capabilities related to ToM. In summary, T4D builds on previous benchmarks by focusing on models’ capabilities to determine actions for themselves based on mental state reasoning. We have added this in the extended related work section in the updated draft in Appendix A.

(Gentle note that according to the last question in FAQ from ICLR Reviewer Guide (https://iclr.cc/Conferences/2024/ReviewerGuide), “if a paper was published (i.e., at a peer-reviewed venue) on or after May 28, 2023, authors are not required to compare their own work to that paper.” and “they may be excused for not knowing about papers not published in peer-reviewed conference proceedings or journals, which includes papers exclusively available on arXiv.” and Gandhi et al. 2023. Was released on arXiv on June 21, 2023).

Reviewer comment (Theory):
The formal reasoning steps and inferences required for T4D are not clearly specified. For a model to succeed at T4D, it likely needs to make inferences about agents' utilities, beliefs, perceptual access and how helping might alter beliefs to increase utilities. However, the paper does not delineate the full reasoning process of inferring and comparing utilities across agents that would lead to selecting the correct action. Clearer formulation of the multi-step inference process needed could better highlight the reasoning challenges in T4D.

Author response:
Thanks for the suggestion. Indeed, for models to succeed at T4D, they need to reason about utilities, ToM inferences, observations, etc. In Section 4.2 (especially Table 2), we break down the multi-step inference process from collected human-written rationales and defined three reasoning challenges: **question decomposition** that captures reasoning about utilities (who benefits from information the most is who needs it most to fulfill a goal), **ToM inferences** that capture reasoning about mental states (A has a false belief of a location of an item), and **common sense assumptions** that capture reasoning about observations (percepts). Moreover, we provide experimental results of the model's performance change when we give hints regarding each challenge. We will articulate this part more clearly.

Reviewer comment (Experiments):
Lack of probing methods: the authors only try a single method of probing LLMs, which is asking a direct question. The work would improve if alternate probing methods were also tried, for example asking the model to complete a sentence; i.e., actually do instead of what it thinks it would do.

Author response:
Good suggestion! We tried generative LM probing in the early stage of designing T4D, i.e., asking models to complete a sentence and observing a similar trend. However, we decided on the multiple-choice format because providing options makes the task more well-defined because evaluating free-form generation without probability logit access is error-prone ([1]). Further, adding multiple choices also circumscribes the scope and the goals of the task -- open-ended generation leaves room for fantastical outputs, whereas we are interested in specifically probing the model's capabilities for T4D grounded in the situation.

Reviewer comment (Novelty):
A justification of how this benchmark builds over other Theory of Mind benchmarks [ones in the paper and 1, 2, 3] is needed.

Author response:
Newer benchmarks such as BigToM are very good studies measuring ToM in LMs. While there is a surface-level similarity in the task setups (both contain “action” probing), there is an important distinction between action probing in BigToM and T4D: probing beliefs of others’ actions such as “*What will A do?*” (BigToM) vs. probing model’s own action decisions based on beliefs of others such as “*Who should you provide help with?*” (T4D).  Specifically, in T4D, models have to treat themselves as situated agents and decide their **own** actions, based on potential reasoning about others’ mental states and predictions of others' actions. We argue that these are complementary directions, and both are important to understand LLMs’ capabilities related to ToM. In summary, T4D builds on previous benchmarks by focusing on models’ capabilities to determine actions for themselves based on mental state reasoning.

(Gentle note that according to the last question in FAQ from ICLR Reviewer Guide (https://iclr.cc/Conferences/2024/ReviewerGuide), “*if a paper was published (i.e., at a peer-reviewed venue) on or after May 28, 2023, authors are not required to compare their own work to that paper.*” and “*they may be excused for not knowing about papers not published in peer-reviewed conference proceedings or journals, which includes papers exclusively available on arXiv.*” and Gandhi et al. 2023. was released on arXiv on June 21, 2023, some other papers in the suggested list are also only on arXiv or released later than May 28, 2023).

Reviewer comment (Experiments):
Lack of evaluations: The probing method that they use is restricted to be questions about helping others by providing information.

Author response:
In 6.4, we also include results from T4D-Faux Pas tests, where we convert the original tests from [5] that ask questions of “*Who committed a faux pas?*” to probing action decisions about providing emotional support “*Who should you provide emotional support?*”. In future work, we plan to extend to more diverse domains, including scenarios from BigToM and others.

Reviewer comment (Experiments):
Only test with GPT-4 and Palm-2, both closed source models.

Author response:
We also probed Llama 2-70B on T4D and have included results in the Appendix F in the updated paper. We find a similar trend of models struggling on T4D (22%) significantly more than ToMi (63%).

Reviewer comment (Experiments):
Not a fair comparison across methods. The prompt for FaR is much longer and is much more task-specific. It is relatively unsurprising that such scaffolding helps the model.

Author response:
RE length: Prior work has shown that length can both hurt and help LLMs, ([3], [4]) and it is unclear if more complex prompts are always better. Further, we experiment with baselines such as ToT, which have similar length prompts.

RE task-specific: We show that FaR can generalize to other domains different from the original ToMi set in Section 6.4. Furthermore, through analysis in Section 4.2, we find that the key bottleneck of LLMs finding T4D hard is that without hand-holding inference questions such as “what are A’s mental states about X?”, models could not locate the relevant inference to decide an action. Thus, we designed FaR specifically to not leak the mental states and instead of proposing a generalizable reasoning structure about predicting and then reflecting (discussed in Section 5, first 2 paragraphs).

Reviewer comment (Experiments):
No justification as to why this method is better and not another alternative. For example asking the ToMi question first, “Where does the <agent> believe object is?” and then asking the model about the helping questions are simpler scaffolds that the authors fail to test.  
- A 1-shot baseline is needed  
- Methods need to be compared based on number of tokens in the prompt.

Author response:
A key reasoning challenge in T4D is that models must *self-discover* that they should leverage mental state reasoning to decide actions. If asked in a 2-step way, the task will leak about "what to infer about" to LLMs. In contrast, FaR did not leak that models should reason about the mental states and provide a reasoning structure.

Reviewer comment (Experiments):
As a method being contributed, there needs to be a broader set of evaluations across domains.

Author response:
We include story structure changes and T4D-Faux Pas generalization tests in 6.4, both of which show consistent improvements from FaR. In our pilot study, we also tested another widely-used ToM test: the smarties test, where the label of a container does not match the content, and found a similar trend in experimental results.​​

Reviewer comment (Experiments):
This work illustrates the differences among FaR and GPT-4 in real-world scenarios. I am also curious about what it will look like with different prompting methods, such as Chain-of-Thought and Tree-of-Though. This work does not introduce existing prompting methods in detail, which will confuse the audience to some degree.

Author response:
__[Curious about what it will look like with different prompting methods, such as Chain-of-Thought and Tree-of-Though]__

We did include comparisons with chain-of-thought and tree-of-thought prompting methods as shown in Figure 5 and explained in detail in Section 6.1. We also included the full prompts in Appendix D.

Reviewer comment (Theory):
FaR breaks the T4D process into three steps, question decomposition, theory-of-mind inference, and common sense assumption. However, the motivations for the breakdown have not been introduced in this work. I would like to suggest that this work explains the motivations clearly.

Author response:
__[FaR breaks the T4D process into three steps. However, the motivations for the breakdown have not been introduced in this work.]__

The three steps are our attempts to analyze what makes T4D challenging. We arrived at the three reasoning challenges from collecting human-written rationales as explained in detail in Section 4.2 and Appendix C. Additionally, through experiments, we found that providing hints regarding each challenge results in very different LLM performance, which helped us speculate that the key reason why T4D is hard for LLMs is that models struggle to locate relevant inference evidence to determine their actions. FaR actually did not incorporate the 3 steps as it would leak the reasoning process. Instead, FaR proposes a generalizable reasoning structure with two steps: Foresee and Reflect that helps models to arrive at relevant inferences.

Reviewer comment (Theory):
What is the motivation for the three steps in the T4D process?

Author response:
__[FaR breaks the T4D process into three steps. However, the motivations for the breakdown have not been introduced in this work.]__

The three steps are our attempts to analyze what makes T4D challenging. We arrived at the three reasoning challenges from collecting human-written rationales as explained in detail in Section 4.2 and Appendix C. Additionally, through experiments, we found that providing hints regarding each challenge results in very different LLM performance, which helped us speculate that the key reason why T4D is hard for LLMs is that models struggle to locate relevant inference evidence to determine their actions. FaR actually did not incorporate the 3 steps as it would leak the reasoning process. Instead, FaR proposes a generalizable reasoning structure with two steps: Foresee and Reflect that helps models to arrive at relevant inferences.