Paper: Learning to Explore with In-Context Policy for Fast Peer Adaptation (ICLR-2024)

Reviewer comment (Theory):
The proposed framework depends on the availability of the finite set $\Psi$ of peer types, therefore either a diverse set of peers should be available at training time or the agent might fail to generalize to previously unseen types of peers during evaluation. This is a major limitation of this approach.

Author response:
A1: Generating a diverse pool of agents for zero-shot multi-agent coordination is a widely studied topic[1][2]. It is true that our method can also benefit from a diverse set of peers at training. However, it should be emphasized that PAIC focuses on how to learn a policy to balance exploration and exploitation for adaptation given a training peer pool, which is orthogonal to the diverse pool generation problem. The training pools in our experiments are shared across all methods, and PAIC still outperforms all the baselines and ablations, showing the effectiveness of PAIC. Note that the previous works (GSCU, TrajDi, …) also train the models in a finite set of model pools. Moreover, the types of the test peer policy in the newly added watchtower predator-and-prey are in fact **unseen**. Please refer to Appendix E for more details.

Reviewer comment (Experiments):
It would help to add an algorithm box for the evaluation phase - particularly, I am confused whether or not the policy or peer type prediction network is fine-tuned during the adaptation to test agents. What are the differences between training and test settings? This would also help me better understand the results in Fig 4.

Author response:
A2: During evaluation, all networks are frozen without any fine-tuning. In fact, peer identification is only an auxiliary task for boosting the training of the context encoder. During testing/evaluation, the peer identification networks are not used at all. We only use the context encoder to obtain the latent representation of the peer. There are 2 differences between training and test procedures.  One is that the testing and training peer pool (policies) are different. The other difference is that all networks are fixed and the peer identification head is dropped during training.

Reviewer comment (Experiments):
In Sec 4.4, please clearly specify the definitions of each baseline - Fig 5 labels "PAIC-reward" and "PAIC-reward-aux" but they are not mentioned in the text.

Author response:
A4: Thanks for pointing out this oversight! PAIC-reward means that we reserve the auxiliary task to train the context encoder but remove the auxiliary exploration reward. PAIC-reward-aux means we both remove the auxiliary task and the exploration reward. Figure 5 shows that the auxiliary task can slightly boost the context encoder and the exploration reward significantly contributes to the performance. We have added more explanations in Sec 4.4 to clarify the terms.

Reviewer comment (Evaluation):
In Table 3, for Train 1, why is there a drop in performance for larger pool size N (i.e. N=18 to N=36)?

Author response:
A6: Good point! This is caused by the random seed. We rerun the experiments on 3 seeds and report the average results in the Table. 3, showing the drop disappeared in N=36. We have updated the results in the revision.

Reviewer comment (Presentation):
Fig 6 - Is the t-SNE plot for Overcooked? Please make it explicit.

Author response:
A7: Thanks for pointing out this oversight! This t-SNE plot is for Overcooked. We have modified the caption of Fig 6 to clarify it.

Reviewer comment (Novelty):
The reviewer failed to discern the rationale behind the authors' proposal of the in-context policy as a novel concept. In-context policies fundamentally rely on previous trajectories, a characteristic shared by nearly all multi-agent policies. What distinguishes this approach is the utilization of trajectories from various instances. However, if an RNN or a Transformer is used to represent agents' policies, it might also remember information from previous episodes.

Author response:
We agree that in-context policy is a kind of method that utilizes history trajectories, but the concept of in-context policy is not the core contribution of our work. As you mentioned, both PAIC and the existing RNN/Transformer-based methods can encode context information for downstream tasks. However, PAIC has a significant advantage over the previous methods in terms of learning to explore the strategy of unknown peers.  PAIC introduces a peer identification auxiliary task that serves two purposes: first, it facilitates the training of the context encoder by providing a supervised signal; second, it generates intrinsic rewards that mitigate the sparse reward problem and encourage exploration. The intrinsic rewards are based on the information gained from the states, which means PAIC can actively seek informative states to model the peers better. Therefore, PAIC can adapt to unknown peers faster and more accurately.

Reviewer comment (Experiments):
The justification for employing a Multi-Layer Perceptron (MLP) to encode trajectories merits a more thorough validation. As the authors suggest, utilizing MLP treats the context as a collection of state-action pairs, which disregards the intra- and inter-episode temporal order. This oversight could certainly have a negative impact on context encoding performance. Furthermore, the reviewer does not concur that alternative network architectures are incapable of capturing long-term dependencies and mitigating overfitting. Transformers, for instance, have been widely adopted in the encoding of multi-agent trajectories.

Author response:
We tried to replace MLP with Transformer for the context encoder. To visualize the comparison between MLP and Transformer-based context encoder, we add more curves and data in Appendix G. It is shown that for the Transformer-based context encoding, the peer identification loss quickly drops before a rise, while the loss of MLP version steadily decreases. This is because the Transformer-based context encoder quickly overfits the context distribution in the early stage of training. As the learning proceeds, the agent policy changes and leads to a different observation (context) distribution, causing the identification loss to rise. As for the MLP-based encoder, it does not overfit the context in the early stage and progressively improves throughout training. The success rate curve in Figure 11(a) also supports our claim. Please refer to Appendix G.2 for more details.

Reviewer comment (Experiments):
The context only includes ego information? How can a peer be identified without its information?

Author response:
Yes, the context does only include ego information since we consider the partially observable setting. For example in the kuhn poker game, the hand is not revealed until the showdown stage. The peer information is partially revealed in the ego observation, so the agent should actively explore to gain enough information to identify the strategy of the peer.  This is also the reason why we introduce the auxiliary reward to encourage active exploration behavior, making PAIC different from other recurrent models.

Reviewer comment (Experiments):
What if a transformer is used for context encoding? Would it be better than an MLP?

Author response:
We tried to replace MLP with Transformer for the context encoder. To visualize the comparison between MLP and Transformer-based context encoder, we add more curves and data in Appendix G. It is shown that for the Transformer-based context encoding, the peer identification loss quickly drops before a rise, while the loss of MLP version steadily decreases. This is because the Transformer-based context encoder quickly overfits the context distribution in the early stage of training. As the learning proceeds, the agent policy changes and leads to a different observation (context) distribution, causing the identification loss to rise. As for the MLP-based encoder, it does not overfit the context in the early stage and progressively improves throughout training. The success rate curve in Figure 11(a) also supports our claim. Please refer to Appendix G.2 for more details.

Reviewer comment (Presentation):
In Figure 5, what is the difference between PAIC-reward and PAIC-reward-aux?

Author response:
Thanks for pointing out this oversight! PAIC-reward means that we reserve the auxiliary task to train the context encoder but remove the auxiliary exploration reward. PAIC-reward-aux means we both remove the auxiliary task and the exploration reward. Figure 5 shows that the auxiliary task can slightly boost the context encoder and the exploration reward significantly contributes to the performance. We have added more explanations in Sec 4.4 to clarify the terms.

Reviewer comment (Experiments):
More ablations on other scenarios are expected.

Author response:
We further conduct an ablation study in the new watchtower predator-and-prey environments, showing consistent results as the previous environments. Please refer to the Appendix. E for more details. The results in the following Table demonstrate the effectiveness of the PAIC in the new environment, compared with baseline methods.
|   PAIC   | Generalist | LIAM | LIAMX | LILI | PAIC-reward | PAIC-reward-aux |
| :--: | :--: | :--: | :--: | :--: | :--: | :--: |
| **-3.934 $\pm$ 0.038** | -4.728 $\pm$ 0.031 | -4.623 $\pm$ 0.018 | -4.337 $\pm$ 0.079 | -4.714 $\pm$ 0.094 | -4.160 $\pm$ 0.065 | -4.544 $\pm$ 0.033 |

Reviewer comment (Experiments):
The paper only considers purely cooperative and competitive environments. It would be interesting to see whether PAIC can handle more complex mixed-motive environments.

Author response:
PAIC can perform well in the mixed-motivate multi-agent environment. Here, we introduce a **watchtower predator-and-prey environment** for further evaluation. In this environment, the prey has different moving patterns across training and testing. The predator only can observe the states in a small range of circles. There are four watchtowers in the corner. When the predator is in the watchtower, it can get the global state. In that case, the predator has to balance chasing the closest prey (own interests) and going to the towner to make the other prey visible to the team (collective welfare). More details about the environment are available in the Appendix. E. The results in the following Table demonstrate the effectiveness of the PAIC in such environments, compared with baseline methods.

Reviewer comment (Experiments):
How does PAIC perform in more complex mixed-motive environments, where agents have to balance their own interests and the collective welfare?

Author response:
PAIC can perform well in the mixed-motivate multi-agent environment. Here, we introduce a **watchtower predator-and-prey environment** for further evaluation. In this environment, the prey has different moving patterns across training and testing. The predator only can observe the states in a small range of circles. There are four watchtowers in the corner. When the predator is in the watchtower, it can get the global state. In that case, the predator has to balance chasing the closest prey (own interests) and going to the towner to make the other prey visible to the team (collective welfare). More details about the environment are available in the Appendix. E. The results in the following Table demonstrate the effectiveness of the PAIC in such environments, compared with baseline methods.

Reviewer comment (Evaluation):
What are the implications of PAIC for human-AI interaction, and how can it be evaluated in real-world settings involving human peers?

Author response:
Good question! In fact, one of the potentials of PAIC is to enhance human-AI interaction. In the future, when AI agents are spread across the world, they may encounter unfamiliar human users every day. The AI agents need to quickly identify the behavior patterns and the internal values of human users so as to serve them, where PAIC can help. As for evaluating PAIC in real-world settings involving human peers, we can consider the scenario where human players need an AI agent partner to cooperate or compete with. We can first collect some human trajectories to train human proxy agents for the training peer pool. Then we can pair the trained PAIC agent with human players in the game, for example Overcooked[1], to see how the PAIC agent adapts to human players.

Reference: [1] Strouse, D.J., McKee, K., Botvinick, M., Hughes, E. and Everett, R., 2021. Collaborating with humans without human data. *Advances in Neural Information Processing Systems*, *34*, pp.14502-14515.