Paper: Do not Start with Trembling Hands: Improving Multi-agent Reinforcement Learning with Stable Prefix Policy (ICLR-2024)

Reviewer comment (Reproducibility):
In the abstract it is claimed that [you] "find that $\epsilon$-greedy policies can be deemed..."; it is unclear how and why this was not already known. In its second part it is unclear. How do you compute such policies? What do you mean by "plan an existing optimal policy"? The description of what was done is unclear to me, and how this was done is absent.

Author response:
> In the abstract it is claimed that [you] " find that $\epsilon$- greedy policies can be deemed...", it is unclear how and why this was not already known. In its second part unclear. How do you compute such policies?  What do you mean by "plan an existing optimal policy"?

Based on the $\epsilon$-greedy algorithm, agents choose the best action with the probability of 1-$\epsilon$ and a random action otherwise. In game theory, players who make decisions with trembling hands will choose actions greedily with a probability of $p$ and other (random) actions with $1-p$. In such a way, the $\epsilon$-greedy method and the trembling hands' decision-making are correlated. Meanwhile [1] also states the relationship between $\epsilon$-greedy and the trembling hands process.

In this paper, the stable prefix policy can be considered as a trajectory template. We establish a trajectory tree based on multiple historical interactive trajectories. In each episode, agents are in the initial state at the first time step. The trajectory tree will generate a potential optimal route of states from that initial state by greedily selecting the subsequent states with the largest value, which is the process of 'plan an existing optimal policy'.

Reviewer comment (Writing):
The related works section addresses the background rather than the related works, and the background is insufficient in the exposition to provide tools to understand what will be done later. Trembling Hands Nash Equilibria are never defined, for example. This leads to the fact that in the proposed method, it was unclear to me what portions of the whole regime are proper contributions of the work and what are not.

Author response:
> The related works section addresses the background rather than the related works, and the background is insufficient in the exposition to provide tools to understand what will be done later.

In this paper, we are focusing on the MARL exploration problem rather than the solution to the THPNE problem. Thus, we introduce MARL and some NE definitions in the Related Work section and introduce MARL basic notations and settings in the Background section. These contents might be sufficient for introducing the subsequent MARL methods.

Reviewer comment (Theory):
The Theoretical Analysis is absent, meaning that in the way it is done it is mostly unclear what it should suggest.

Author response:
> The Theoretical Analysis is absent, meaning that in the way it is done is mostly unclear what it should suggest.

We provide some theoretical analysis of the convergence guarantee and computational complexity of our algorithm in section 4.4. The THPNE problem is used for introducing the exploration dilemma, so we did not provide the theoretical analysis of the THPNE, which, we think, is not quite relevant to MARL problems in this paper.

Reviewer comment (Writing):
Some English phrasing is wrong and some typos are present (for example there should be an $\epsilon$ at the 9th line of the first page I believe).

Author response:
> some English phrasing is wrong and some typos are present (for example there should be an $\epsilon$ at the 9th line of the first page I believe)

We will double-check the expression methods and typos in this paper. However, at the 9th line of the first page, we want to express that agents choose the best actions with the largest values. Thus, agents choose actions greedily and there should not be an epsilon here.

Reviewer comment (Theory):
The proposed method is based on the premise that agents should be capable of finding a policy toward success from historical interactions. In other words, SPP relies heavily on the performance of the underlying algorithm.

Author response:
Our SPP approach divides the decision-making process into two stages: pre-decision based on planning and post-decision with action noise. Where the pre-decision is given through UCT planning, it is not necessarily required that the historical interactions contain a successful policy, which means that even if there are no successful policies in the historical trajectory, UCT planning is still able to select the regions with the highest value or exploration potential at the moment based on the existing interactions. In contrast to relying heavily on the base policy, our SPP approach takes advantage of this favorable property of UCT planning and instead improves the performance of the base policy. Our experimental results on SMAC benchmarks can fully validate this.

Reviewer comment (Experiments):
The trembling hands is a concept in multi-agent games, but this paper only provides solutions in cooperative scenarios (Dec-POMDP problems). At the same time, SPP is only applied to value decomposition methods.

Author response:
Thank you very much for your comments. For the perspective of better demonstrating the performance improvement of our SPP method compared to the original MARL method, we choose to test our method on the most popular benchmarks with MARL algorithms in the current MARL community, which mostly require computing the joint utility function to reach a better equilibrium of the game, and thus focus on solving the problem of credit allocation for cooperative MARL. Technically, our SPP method works for those scenarios with suboptimal equilibria due to trembling hands. We will actively look for MARL benchmarks other than cooperative scenarios for testing and welcome your further suggestions.

Reviewer comment (Experiments):
The proposed method was only evaluated on SMAC (the description of the experimental results in MPE is skimpy and unconvincing). SMAC is a popular multi-agent experimental platform but has been pointed out to have many shortcomings [1]. More and more researchers in the MARL community advocate conducting experiments in multiple different domains to evaluate the proposed algorithm comprehensively [2].

Author response:
We also fully agree with this comment that the SMACv1 environment lacks randomness in initializing the task settings. SMACv2 in contrast provides the configuration settings including the number of agents and enemies, the initialized positions, and the probability of generating an agent type. However, as far as we are concerned, this configuration provides too much randomness (I'd like to call it unstable). We believe SMACv2 is more suitable to test the migration capability and few-shot scenario adaptation ability of an algorithm. Despite that, we tested our approach on three scenarios of the SMACv2 environment. The items in the table are the average winning rate among 5 seeds and their variance.

| 5_vs_5       | 0.5M           | 1M             | 1.5M           | 2M             |
| ------------ | -------------- | -------------- | -------------- | -------------- |
| MAPPO        | 8,67$\pm$4.82  | 21.39$\pm$7.91 | 28.77$\pm$6.56 | 35.94$\pm$9.24 |
| QMIX         | 20.15$\pm$4.59 | 32.68$\pm$5.80 | 41.48$\pm$5.74 | 50.21$\pm$3.12 |
| Our Approach | 19.07$\pm$5.62 | 38.97$\pm$9.32 | 45.11$\pm$8.34 | 52.24$\pm$6.73 |

| 10_vs_10     | 0.5M           | 1M             | 1.5M            | 2M              |
| ------------ | -------------- | -------------- | --------------- | --------------- |
| MAPPO        | 3.54$\pm$2.50  | 11.84$\pm$1.85 | 16.98$\pm$2.18  | 30.85$\pm$7.77  |
| QMIX         | 11.09$\pm$2.24 | 20.56$\pm$4.48 | 31.54$\pm$6.75  | 42.29$\pm$12.41 |
| Our Approach | 6.70$\pm$2.87  | 24.48$\pm$8.78 | 37.75$\pm$10.35 | 43.45$\pm$9.80  |

| 10_vs_11     | 0.5M          | 1M            | 1.5M          | 2M             |
| ------------ | ------------- | ------------- | ------------- | -------------- |
| MAPPO        | 0.76$\pm$0    | 0.83$\pm$0.91 | 0.74$\pm$0.47 | 1.84$\pm$1.29  |
| QMIX         | 0.57$\pm$0.44 | 4.29$\pm$2.61 | 11.6$\pm$3.62 | 14.95$\pm$2.05 |
| Our Approach | 0.34$\pm$0.75 | 4.51$\pm$1.92 | 10.3$\pm$4.54 | 12.67$\pm$5.26 |

According to the results, our proposed method can follow the performance of the baseline algorithm in the so-called unstable environments in 2M time steps.

Reviewer comment (Theory):
Is there any theoretical basis to prove that $Q^t_{assem}$ is more accurate than the original $Q_{tot}$?

Author response:
In this paper, similar states belong to a prototype cluster with a value, so similar states share the same value. The $Q_{assem}^t$ is the average value of $Q_{tot}$. In such a way, the expectation of $Q_{assem}^t$ is the same as the expectation of original $Q_{tot}$. The $Q_{assem}^t$ provides lower variance.

Reviewer comment (Theory):
I think that in some scenarios, the SPP variant may be more likely to fall into a local optimal solution. Suppose that in such a scenario, agents can easily access the state corresponding to the suboptimal solution, while the state corresponding to the global optimal solution is in the opposite direction and relatively difficult to access (for example, further away from the initial position of the agents). The SPP variant may directly give up early exploration and find it difficult to converge to the global optimal solution. Of course, the above issue can be alleviated by adjusting $c_{ucb}$, but this requires sufficient prior knowledge.

Author response:
Thank you very much for your comments, we would love to discuss this scenario with you. You say that states oriented to suboptimal solutions are very easy to reach while states oriented to global optimal solutions are very difficult to collect and that the vast majority of data collected by algorithms focusing on exploration in this scenario are suboptimal, and we believe that obtaining an accurate value estimation of the global optimal state that is distinguishable from the suboptimal data is also a very difficult task to solve in this case. Our approach includes an annealing-like process to target such scenarios, in addition to the possibility of tuning the parameters of the UCT to encourage exploration. During annealing the tree in $\text{MCT}^2$ is destroyed and reconstructed, at which point the MARL algorithm resumes single-step exploration.

Reviewer comment (Experiments):
The experiment results do not conclusively demonstrate the effectiveness of the proposed method. In Figure 8, the performance of the proposed policy closely mirrors that of the original QMIX implementation. I would suggest the authors to test on more challenging MARL benchmarks, though those benchmarks often require more exploration, which may pose challenges for the proposed method.

Author response:
> In Figure 8, the performance of the proposed policy closely mirrors that of the original QMIX implementation. I would suggest the authors test on more challenging MARL benchmarks.

In this paper, we provide our main experimental results in Figure 5 which contains the challenging subtasks of the SMAC environment. Among the subtasks, the 5m\_vs\_6m, 3s\_vs\_5z, and 2c\_vs\_64zg tasks are hard tasks. Not all the baseline algorithms can achieve good results. Additionally, the subtasks including MMM2, 3s5z\_vs\_3s6z, and 6h\_vs\_8z are super-hard tasks. Experimental results on these tasks within 2M time steps can show the better performance of our approach. In contrast, Figure 8 shows the other tasks in the SMAC environment and most of them are easy tasks in which the original QMIX as well as other baseline algorithms can achieve acceptable results. This might be the reason why our proposed policy closely mirrors that of the original QMIX.

Reviewer comment (Experiments):
In Section 6, the authors claim that the proposed method can be applied to the critic training in Actor-Critic MARL algorithms. Can you briefly describe how to implement the proposed method in, say, MAPPO? And what is the performance improvement when applying to MAPPO?

Author response:
> In Section 6, the authors claim that the proposed method can be applied to the critic training in Actor-Critic MARL algorithms. Can you briefly describe how to implement the proposed method in, say, MAPPO? And what is the performance improvement when applying to MAPPO?

In the Actor-Critic MARL algorithms, actor networks are responsible for training the action distributions conditioned on observations and the critic networks train the values of states. Applying our SPP method to the Actor-Critic method includes (i)assembling the values with the same cluster nodes and (ii)removing the actors' explorations by taking the maximum of the distributions and using the trajectory templates generated by the planning process to determine whether to interrupt the process and resume single-step explorations like non-SPP-augmented methods. Our approach contributes to the performance improvement of the Actor-Critic method in two ways: on the one hand, assembles the values with the same cluster nodes so that the values for training are more accurate; on the other hand, it determines whether the elimination of the exploration of the first half of the strategy can lead to more fine-grained improvements in the second half of the strategy by planning, which has already balanced exploration and exploitation by UCT.

To focus on the efficiency aspects, we chose three subtasks to show the performance of the MAPPO and our SPP-augmented MAPPO within 2M time steps. The experimental results indicate that our method can improve the performance of Actor-Critic MARL methods. The items in the table are the average winning rates (%) among 5 seeds and their variances.

| MMM2      | 0.5M          | 1M              | 1.5M            | 2M              |
| --------- | ------------- | --------------- | --------------- | --------------- |
| MAPPO     | 4.46$\pm$9.97 | 17.27$\pm$30.11 | 34.27$\pm$35.44 | 42.65$\pm$37.72 |
| SPP+MAPPO | 9.34$\pm$5.54 | 46.81$\pm$19.63 | 51.03$\pm$14.95 | 62.42$\pm$14.98 |

| 5m_vs_6m  | 0.5M           | 1M              | 1.5M            | 2M              |
| --------- | -------------- | --------------- | --------------- | --------------- |
| MAPPO     | 6.57$\pm$1.47  | 17.83$\pm$16.47 | 45.38$\pm$15.48 | 49.58$\pm$23.32 |
| SPP+MAPPO | 36.58$\pm$6.54 | 53.32$\pm$11.27 | 57.57$\pm$11.19 | 60.65$\pm$10.56 |

| 3s5z_vs_3s6z | 0.5M          | 1M             | 1.5M            | 2M              |
| ------------ | ------------- | -------------- | --------------- | --------------- |
| MAPPO        | 8.00$\pm$1.79 | 1.96$\pm$3.92  | 9.17$\pm$17.84  | 13.09$\pm$21.71 |
| SPP+MAPPO    | 5.63$\pm$0.97 | 7.29$\pm$12.63 | 15.02$\pm$26.02 | 20.80$\pm$35.85 |

Reviewer comment (Writing):
In the matrix game presented in Section 1, should the $epsilon$ for player 1 be 0.1?

Author response:
> In the matrix game presented in Section 1, should the epsilon for player 1 be 0.1? 

Your concerns are much appreciated. We double-check the corresponding section. With $\epsilon$ set to 0.2, the player will have a 0.8 probability of going into the greedy mode and a 0.2 probability of going into a mode where the action is chosen randomly; however, since there is also a 0.1 probability that the original greedy action will be chosen in random mode, the value of $\epsilon$ should be taken as 0.2, at which point the greedy action will take the value of 0.1

Reviewer comment (Experiments):
Can the proposed method be applied to scenarios with continuous action spaces?

Author response:
> Can the proposed method be applied to scenarios with continuous action spaces? 

The original intent of our paper was to address the problem of trembling hands that exists in value-based MARL methods. However, our SPP method contains major components: the UCT planning[1], Q-value assembling and cancelation of action noise are technically applicable to continuous action spaces. We have found that our approach is surprisingly able to improve the performance of Actor-critic based, so we are reasonably confident that our SPP approach is also capable of applying to continuous action spaces. We will actively search for suitable benchmarks for testing and welcome your further suggestions.

In the remaining time we would like to address the remaining concerns that you have that prevent you from accepting the paper. We would appreciate it if you could engage with us on the remaining concerns, as we want to address them.



[1] Hubert, Thomas, et al. "Learning and planning in complex action spaces." International Conference on Machine Learning.