Paper: Planning with Theory of Mind for Few-Shot Adaptation in Sequential Social Dilemmas (ICLR-2024)

Reviewer comment (Experiments):
The primary methodological limitation is that it is unclear how much of PToM’s performance gains (faster convergence in self-play as in Figure 3, and better few-shot adaptation in Table 1) are due to model-based planning via MCTS versus goal inference over other agents. Any model-based RL algorithm might similarly converge faster compared to model-free MARL baselines. Therefore, it is important to include a model-based RL baseline in both Figure 3 and Table 1 to better assess the specific contribution of the ToM component. Possibly, the direct-OM ablation in the Supplementary Results could serve this purpose, or alternatively, one could test neurally guided MCTS assuming opponents act randomly.

Author response:
We have tried a method of planning through self-play like AlphaZero. However, its computational complexity proved prohibitively high. The number of nodes required for expansion in MCTS increases exponentially with the increasing number of agents. It poses a significant chanllenge. Given our resource constraints, ensuring efficient training speed became impractical. Thus, we forwent AlphaZero as a direct baseline. A similar rationale precluded the adoption of MuZero.

An alternative approach to implement self-play involves substituting one's own policy network for the opponent modeling network in direct-OM. In the symmetric games, which are studied by this paper, this approach is similar to direct-OM. However, it may not be feasible to apply this approach in the environments characterized by asymmetric player roles and rewards, Like battle of the sexes, another classic paradigm of social dilemmas.

We use the direct-OM ablation as a model-based baseline. Direct-OM performs MCTS based on opponent modeling instead of self-play.From the results of direct-OM, we believe fast convergence mainly comes from model-based planning. As the baselines in our comparison are predominantly model-free methods, we acknowledge the unfairness of directly comparing the number of convergence samples. Consequently, we have opted to remove the corresponding statements and figures from the paper. Nevertheless, we believe it is crucial to emphasize the merit of applying model-based methods to improve sample efficiency in the multi-agent context.

Reviewer comment (Reproducibility):
The paper’s clarity is generally good, but important details are missing regarding how the opponent module is trained and how its replay buffer is collected. Specifically, Eq. (3) implies the need for a replay buffer with state, action, and goal history for each goal \( g \). It is unclear how the goal history is collected without ground-truth goals in the environment simulator—whether it is imputed or generated via sampling at the start of each MCTS round. This remains unclear even after checking the PToM pseudocode.

Author response:
We would like to clarify the process of training with goal. During each episode, PToM will not obtain the opponent's ground-truth goal. After an episode ends, if PToM observes the opponent's complete trajectory, PToM can parse out the goal actually accomplished by the opponent in this episode and put it into the replay buffer; if the opponent's complete trajectory is not observed (as may happen in SSH), the opponent's goal cannot be obtained, and this trajectory will not participate in training. Since the goals in the environment have relatively clear meanings (and correspond to cooperation and defection in social dilemmas), such parsing does not consume many additional resources. 

We would also like to note that no matter in the training or evaluation phase, opponents are not required to have explicit goals. All of the learning baselines, LOLA, SI, A3C, and PS-A3C,  do not explicitly model goals. But PToM assumes that the opponent has a goal, or more broadly, mental state, and infers this assumed mental state through opponent's behavior. This is a way to use ToM in agent modeling.

Reviewer comment (Theory):
The inter-ToM belief update in Eq. (2) seems ad hoc. While taking a weighted average of current and past goal beliefs is intuitively reasonable, it could be better justified as a Bayesian update across episodes, under a model where opponents might resample their goal from the uniform prior each timestep. This justification could remove the requirement of knowing the true goal at episode end, since one could perform an inter-episode Bayesian update without true goal knowledge.

Author response:
We are very grateful for this valuable suggestion. Based on Bayesian theorem, Eq. (2) can be modified as 

$$ b_{ij}^{K,0}(g_j)  = \frac{1}{Z_2}[\alpha b_{ij}^{K-1,0}(g_j) + (1-\alpha) p(g_j| \tau)]  = \frac{1}{Z_2}[\alpha b_{ij}^{K-1,0}(g_j) + (1-\alpha) \frac{p(\tau|g_j)p(g_j)}{p(\tau)}] \\\\
= \frac{1}{Z_2}[\alpha b_{ij}^{K-1,0}(g_j) + (1-\alpha) \frac{p(s_0)p(g_j)\prod_{t=0}^{T_{max} - 1} \pi_{\theta_j}(a_t|s_t, g_j)p(s_{t+1}|s_t, a_t) }{p(s_0)\sum_k p(g_k)\prod_{t=0}^{T_{max} - 1} \pi_{\theta_k}(a_t|s_t, g_k)p(s_{t+1}|s_t, a_t)}] \\\\ 
= \frac{1}{Z_2}[\alpha b_{ij}^{K-1,0}(g_j) + (1-\alpha)b_{ij}^{K-1,T_{max}}(g_j)], $$

where $\tau = (s^{K-1, 0}, a^{K-1, 0}, s^{K-1, 1}, a^{K-1, 1}, ..., s^{K-1, T_{max} - 1}, a^{K-1, T_{max} - 1}, s^{K-1, T_{max}})$ is a trajectory in episode $K-1$, $\pi_{\theta_j}$ is the goal-conditioned policy of the goal $g_j$ parameterized by $\theta_j$. Base on the modified equation, the update of inter-ToM does not require knowing other agents' goals at the end of episodes. Instead, the goal inferred by intra-ToM at the last timestep in episode $k-1$ is used to update inter-ToM.

 During the few-shot adaptation phase, where $b_{ij}^{K-1,T_{max}}(g_j)$ is accurate, above modification does not compromise performance. (We have made a visualization for belief update in SSH in our revised manuscript, which can support this statement.) However, considering that goals are also used in goal-conditioned policy training, the precise impact of these changes in the training stage remains uncertain. 

We think that the weighted average of current and past goal beliefs, shown in Eq.(2), is an effective way to model opponnets' goals. Directly using $b_{ij}^{K-1,T_{max}}(g_j)$ as the likelihood to perform Bayesian update may be a bit aggressive, which may lead to poor modeling of opponents adopting mixed strategies. An alternative avenue for exploration is to directly model the probability distribution of the goal based on the initial state of the current episode, i.e., $Pr(g_j | s^{K, 0})$.

For inter-ToM, we consider how history discloses its choice of goals. An effective method is Monte Carlo estimation $\frac{1}{K} \sum_{l = 0}^{K-1} \mathbb{1}(g_j^l = g)$. However, when opponent behavior changes over time, old records will cause estimation bias, so we adopt time-discounted estimation$\frac{1 - \alpha}{1 - \alpha^K}\sum_{l = 0}^{K-1} \alpha^{K-1-l}\mathbb{1}(g_j^l = g)$ where Equation 2 implies.

Reviewer comment (Evaluation):
The method compares against other baselines (including model-free approaches), however, it is not clear how fair this comparison is. One has to take into account multiple factors: planning budget, number of repeated planning runs, time to train each of the networks, number of parameters, number of network inferences, wall time, etc.

Author response:
Thank you very much for your advice! We agree with you that Figure 3 does not show a reasonable comparison of sample efficiency between our approach and the model-free baselines. We have removed Figure 3 and corresponding content from the manuscript. In addition, to compare the performance between our approach and other planning algorithms, we have used the direct-OM ablation, which can be considered as the implementation of vanilla MCTS,  as a planning baseline. Please find corresponding results in Table 1 and 2 in the revised manuscript.

Reviewer comment (Writing):
The paragraph below Table 1 is unclear.

Author response:
Thanks for your comment! We think this ambiguity comes from the introduce of LI-Ref and improper metrics of algorithms' performance. In the manuscript, algorithms' performance is measured by the average reward over a specific period of timestep, which is not very informative. Although we introduced LI-Ref (a well-established RL agent performing adaptation to other agents through extensive interactions) as a reference to compare and evaluate algorithms' few-shot adaptation ability, it is placed in Appendix F separately, not well-integrated into Table 1. Inspired by your comments, we have adjusted the metric to be the normalized score. Corresponding results can be found in Table 1 (self-play performance) and Table 2 (few-shot adaptation performance) in the revised manuscript. With regard to the paragraph below Table 1, it is removed from the revised manuscript.  The detailed description of LI-Ref is given in Appendix F.1.

Reviewer comment (Evaluation):
In Section 5.2 SPD, it is written that "PToM tends to engage in exploratory cooperative actions to understand the opponent’s characteristics". What is the reasoning behind this statement?

Author response:
When observing agent behavior, we see PToM has shown exploratory cooperative actions. The figures of trajectories can be found https://anonymous.4open.science/r/PToM4SSD-F068 . (Currently, the figures are not exquisite enough. We are sorry for that. We will polish these figures and add them into the revised manuscript later. We hope these rough figures can accurately express PToM's trajectory in adaptation to unseen opponents.) In each figure, the green line with arrows represents the trajectory of PToM at the beginning of an episode, with arrows pointing in the direction of movement.  The light blue and light red regions represent the river where bags of waste may appear and the forest where apples may grow, respectively. In Figure 1 PToM is facing three cooperators, while Figure 2 shows the situation of facing three exploiters. Figures show that PToM may move towards the rubbish area, indicating attempts of cooperative behavior. These attempts will be very important when interacting with conditional cooperators who will cooperate if there are already some cooperators in the environment. This type of trajectory is not found in LOLA, SI and A3C. When facing three cooperators, the average number of steps for LOLA, SI, and A3C in the rubbish area are 1.3, 0.5, and 3.1 respectively, while the average number of steps not in the apple area are 6.2, 4.8, and 11.0. The corresponding quantities for PToM are 14.1 and 22.8. This also reflects a cooperative tendency of PToM. When facing three exploiters, each algorithm spends a large amount of time away from the apple area, and only PToM shows an obvious trajectory to the rubbish area as mentioned above, while the movements of other algorithms look random.

Reviewer comment (Presentation):
The paper would benefit from moving the pseudo code from Appendix A to the main paper.

Author response:
Thanks very much for your comments! Due to space limitation, we have to put the mentioned contents in the Appendix. In the revised manuscript, we further clarify the references to the Appendix in the main text. We hope this works.

Reviewer comment (Presentation):
The parameters used for each environment should be placed in the main body of the paper (in particular, the number of seeds). Consequently, the Authors should consider moving Table 3 from Appendix D.3 to the main paper.

Author response:
Thanks very much for your comments! Due to space limitation, we have to put the mentioned contents in the Appendix. In the revised manuscript, we further clarify the references to the Appendix in the main text. We hope this works.

Reviewer comment (Presentation):
The content of Appendix F (emergence of social intelligence) is interesting enough that the Authors should consider moving it to the main body.

Author response:
Thanks very much for your comments! Due to space limitation, we have to put the mentioned contents in the Appendix. In the revised manuscript, we further clarify the references to the Appendix in the main text. We hope this works.

Reviewer comment (Experiments):
The main weakness of this paper is that the experimental results are quite weak. Table 1 does not show a very large improvement from PToM over the baselines except in some specific cases (e.g. adaptation to exploiters in SS). PToM does not achieve cooperation in SPD. There are no videos or behavioral analyses provided to corroborate qualitative claims about the better behaviour of PToM over the baselines. Can the authors provide such results? For instance, what is the evidence that PToM is significantly better at hunting stags in SSH?

Author response:
Thanks for your helpful comment! We believe the seemingly week results are caused by our improper metric. In the manuscript, algorithms' performance is measured by the average reward over a specific period of timestep, which is not very informative. Although we introduced LI-Ref (a well-established RL agent performing adaptation to other agents through extensive interactions) as a reference to compare and evaluate algorithms' few-shot adaptation ability, it is placed in Appendix E separately, not well-integrated into Table 1. Inspired by your comments, we have adjusted the metric to be the normalized score. Corresponding results can be found in Table 1 (self-play performance) and Table 2 (few-shot adaptation performance) in the revised manuscript. The results show that the reward achieved by PToM and LOLA is already close to the maximum possible reward, thus the advantage of PToM is not significant compared to LOLA. (This is also what the sentence "we find that the leading space of PToM is not significant" tries to express. This sentence has been deleted in the revised manuscript.) In SS and SPD, PToM outperforms other algorithms in 6 out of 7 and 4 out of 7 adaptation scenarios, respectively. After taking variance into consideration, other algorithms achieve best results in no more than 3 adaptation scenarios for each environment. These consistent advantages across different environments demonstrate the effectiveness and robustness of PToM.

Regarding the failure of achieving cooperation in SPD, PToM is designed to be self-interested (i.e. rational), aiming to maximize own reward, regardless of others' benefits. Due to the inherent complexity of SPD, self-interested agents, PToM as well as  LOLA, SI and A3C, tend to select the short-term optimal strategy and fall into the dilemma of total defection. In SPD, only the agents who are long-sighted and take into account others' reward can escape from the dilemma. It is our future work that realizing such agent through developing decentralized-training-decentralized-execution algorithms. It is a great challenge for DTDE algorithms,  and we consider it to be an important future work.  Furthermore, what needs to be emphasized is that our main goal in evaluating PToM and the baselines is not to achieve comprehensive cooperation, but rather to assess their ability to adapt quickly to previously unseen opponents and make appropriate response decisions. In particular, when facing self-interested opponents in SPD, we expect that the focal agent does not cooperate and avoids being exploited by them.

Thanks for your helpful comments on the explanation of experimental results. In the revised manuscript, we have added the visualization of belief updates and corresponding analysis (see details in Figure 3 in Appendix F.2).

We also visualize PToM's behavior trajectories in adaptation to unseen opponents. The figures of trajectories can be found https://anonymous.4open.science/r/PToM4SSD-F068. (Currently, the figures are not exquisite enough. We are sorry for that. We will polish these figures and add them into the revised manuscript later. We hope these rough figures can accurately express PToM's trajectory in adaptation to unseen opponents.). In each figure, the green line with arrows represents the trajectory of PToM at the beginning of an episode, with arrows pointing in the direction of movement.  The light blue and light red regions represent the river where bags of waste may appear and the forest where apples may grow, respectively. In Figure 1 PToM is facing three cooperators, while Figure 2 shows the situation of facing three exploiters. Figures show that PToM may move towards the rubbish area, indicating attempts of cooperative behavior. These attempts will be very important when interacting with conditional cooperators who will cooperate if there are already some cooperators in the environment. This type of trajectory is not found in LOLA, SI and A3C. When facing three cooperators, the average number of steps for LOLA, SI, and A3C in the rubbish area are 1.3, 0.5, and 3.1 respectively, while the average number of steps not in the apple area are 6.2, 4.8, and 11.0. The corresponding quantities for PToM are 14.1 and 22.8. This also reflects a cooperative tendency of PToM. When facing three exploiters, each algorithm spends a large amount of time away from the apple area, and only PToM shows an obvious trajectory to the garbage area as mentioned above, while the movements of other algorithms look random.

Reviewer comment (Evaluation):
Some of the experimental claims are poorly explained and some of them are unsubstantiated. For example, on page 8, what is meant by "we find that the leading space of PToM is not significant"? Also on page 8, the "further intuition" on why PToM is effective at adaptation is not substantiated by analysis of the belief model for the agent during intra- and inter-ToM. Can the authors provide this data?

Author response:
Thanks for your helpful comment! We believe the seemingly week results are caused by our improper metric. In the manuscript, algorithms' performance is measured by the average reward over a specific period of timestep, which is not very informative. Although we introduced LI-Ref (a well-established RL agent performing adaptation to other agents through extensive interactions) as a reference to compare and evaluate algorithms' few-shot adaptation ability, it is placed in Appendix E separately, not well-integrated into Table 1. Inspired by your comments, we have adjusted the metric to be the normalized score. Corresponding results can be found in Table 1 (self-play performance) and Table 2 (few-shot adaptation performance) in the revised manuscript. The results show that the reward achieved by PToM and LOLA is already close to the maximum possible reward, thus the advantage of PToM is not significant compared to LOLA. (This is also what the sentence "we find that the leading space of PToM is not significant" tries to express. This sentence has been deleted in the revised manuscript.) In SS and SPD, PToM outperforms other algorithms in 6 out of 7 and 4 out of 7 adaptation scenarios, respectively. After taking variance into consideration, other algorithms achieve best results in no more than 3 adaptation scenarios for each environment. These consistent advantages across different environments demonstrate the effectiveness and robustness of PToM.

Regarding the failure of achieving cooperation in SPD, PToM is designed to be self-interested (i.e. rational), aiming to maximize own reward, regardless of others' benefits. Due to the inherent complexity of SPD, self-interested agents, PToM as well as  LOLA, SI and A3C, tend to select the short-term optimal strategy and fall into the dilemma of total defection. In SPD, only the agents who are long-sighted and take into account others' reward can escape from the dilemma. It is our future work that realizing such agent through developing decentralized-training-decentralized-execution algorithms. It is a great challenge for DTDE algorithms,  and we consider it to be an important future work.  Furthermore, what needs to be emphasized is that our main goal in evaluating PToM and the baselines is not to achieve comprehensive cooperation, but rather to assess their ability to adapt quickly to previously unseen opponents and make appropriate response decisions. In particular, when facing self-interested opponents in SPD, we expect that the focal agent does not cooperate and avoids being exploited by them.

Thanks for your helpful comments on the explanation of experimental results. In the revised manuscript, we have added the visualization of belief updates and corresponding analysis (see details in Figure 3 in Appendix F.2).

We also visualize PToM's behavior trajectories in adaptation to unseen opponents. The figures of trajectories can be found https://anonymous.4open.science/r/PToM4SSD-F068. (Currently, the figures are not exquisite enough. We are sorry for that. We will polish these figures and add them into the revised manuscript later. We hope these rough figures can accurately express PToM's trajectory in adaptation to unseen opponents.). In each figure, the green line with arrows represents the trajectory of PToM at the beginning of an episode, with arrows pointing in the direction of movement.  The light blue and light red regions represent the river where bags of waste may appear and the forest where apples may grow, respectively. In Figure 1 PToM is facing three cooperators, while Figure 2 shows the situation of facing three exploiters. Figures show that PToM may move towards the rubbish area, indicating attempts of cooperative behavior. These attempts will be very important when interacting with conditional cooperators who will cooperate if there are already some cooperators in the environment. This type of trajectory is not found in LOLA, SI and A3C. When facing three cooperators, the average number of steps for LOLA, SI, and A3C in the rubbish area are 1.3, 0.5, and 3.1 respectively, while the average number of steps not in the apple area are 6.2, 4.8, and 11.0. The corresponding quantities for PToM are 14.1 and 22.8. This also reflects a cooperative tendency of PToM. When facing three exploiters, each algorithm spends a large amount of time away from the apple area, and only PToM shows an obvious trajectory to the garbage area as mentioned above, while the movements of other algorithms look random.

Reviewer comment (Novelty):
There is some important missing related work which should be cited on theory of mind in the context of RL: https://arxiv.org/pdf/2102.02274.pdf, https://arxiv.org/pdf/1901.09207.pdf. Ideally these methods would be provided as baselines, or the authors would explain why their method was clearly an improvement from a theoretical standpoint. Can the authors comment on this?

Author response:
Thank you for providing the related work! We have cited these papers in our revised manuscript. The paper https://arxiv.org/pdf/2102.02274.pdf  provides a recursive deep generative model to learn belief states of other agents at different orders by sampling latent variables. The framework PR2 proposed by  https://arxiv.org/pdf/1901.09207.pdf explicitly takes into account the influence of my action on opponent policy when inferring opponent policy. Due to the time constraint, we cannot give experimental results of PR2-AC currently. Similarly, when modeling other agents, the existing baseline LOLA considers the impact of my policy on the policy gradients of other agents, and SI considers the influence of my action on other agents' policies. What's more, LOLA and SI have been tested in social dilemmas and performed well. Thus, considering our focus on sequential social dilemmas, we think LOLA and SI are representatives of MARL algorithms equipped with the ability of recursively modeling others. 

We have tried to give a theoretical analysis of our approach in SSDs. However, as the complexity and non-stability of the environments, we failed to give theoretical guarantees. Instead, we give a concise theoretical analysis of our approach in two-player matrix-form games (please see details in Appendix B). This analysis demonstrates the convergence of PToM to the optimal response strategy. Meanwhile, empirical results show the consistent improvement of PToM, although the advantage of PToM is not very significant in some cases. Ablation study indicates that hierarchical modeling of other agents and ToM based inference on others' goals are crucial to the adaptation to unseen opponents.

Reviewer comment (Novelty):
The authors could also cite more recent work in the LOLA line: e.g. https://arxiv.org/abs/2205.01447.

Author response:
Thank you for providing the related work! We have added this paper in the revised manuscript.

Reviewer comment (Theory):
Above equation (3) the authors seem to assume that the focal agent has access to the goals of its opponents during training. Is this correct? Yet earlier in Section 3, the authors claim that agent j's true goal is inaccessible to agent i. Can they clarify this apparent contradiction (perhaps the goals are known during training but not during execution, in the usual centralized training, decentralized execution paradigm)?

Author response:
We would like to clarify the process of training with goal. During each episode, PToM will not obtain the opponent's ground-truth goal. After an episode ends, if PToM observes the opponent's complete trajectory, PToM can parse out the goal actually accomplished by the opponent in this episode and put it into the replay buffer; if the opponent's complete trajectory is not observed (as may happen in SSH), the opponent's goal cannot be obtained, and this trajectory will not participate in training. Since the goals in the environment have relatively clear meanings (corresponding to cooperation and defection in social dilemmas), such parsing does not consume many additional resources. 

We would also like to note that no matter in the training or evaluation phase, opponents are not required to have explicit goals. All of the learning baselines, LOLA, SI, A3C, and PS-A3C,  do not explicitly model goals. But PToM assumes that the opponent has a goal, or more broadly, mental state, and infers this assumed mental state through the opponent's behavior. This is a way to use ToM in agent modeling. Therefore, we do not consider the training to be centralized, since we do not get the ground-truth goal information or opponents' models through a centralized module, and even the opponent algorithm does not have a goal variable as ground-truth.

Reviewer comment (Evaluation):
In Figure 3, does the x-axis for PToM take into account the experience steps used in MCTS? This is unclear, and if not, these experience steps should also be accounted for, to make this a fair comparison between algorithms.

Author response:
Thank you very much for your advice! We agree with you that Figure 3 does not show a reasonable comparison of sample efficiency between our approach and the model-free baselines. We have removed Figure 3 and corresponding content from the manuscript. In addition, to compare the performance between our approach and other planning algorithms, we have added direct-OM as a baseline. Direct-OM can be considered as the implementation of vanilla MCTS.  Please find and the corresponding results in Table 1 and Table 2 in the revised manuscript.