From: NeurIPS.cc/2022/Conference/Program_Chairs
title: Paper Decision
decision: Accept

From: NeurIPS.cc/2022/Conference/Paper2900/Area_Chair_Qw2D
metareview: This paper proposes a test-time algorithmic modification to address a multi-agent coordination problem where agents choose incompatible strategies due to symmetries in the environment, showing that it is applicable to ZSC tasks like Hanabi. The proposed symmetrizer is applied to LSTM recurrent policies.

Reviews were mixed for this paper, and I really valued the in-depth discussion amongst reviewers and between reviewers and authors. I am recommending acceptance primarily based on the interesting discussion and new experimental results surfaced between reviewer Y7M6 and the authors:
1. The feedback from reviewer Y7M6 and authors increased my confidence in the statistical correctness of scores wrt baseline, and it is clear that the authors ensured that the baselines were computed fairly. 
2. The rebuttal-revised version of the paper shows that the symmetrizer improves upon any self-play algorithm, such as OBL level 4 and 5. Reviewer Y7M6 points out "I think it would be better to claim that this work can improve state-of-the-art competitive models to become better instead of claiming this work achieves the new state-of-the-art ZSC performance in Hanabi. ". I believe the authors have shown that with their experiments on OBL L5.

The primary critique is that many of the scores were changed late in the review process, which does call into question whether the empirical rigor should be re-examined via another round of review. The authors have gone to great lengths to improve the rigour in the rebuttal. Beyond the specifics of empirical evaluation, I think the idea presented in the paper is interesting and worth sharing with the community: this paper presents an "explicit symmetrizer", whereas to my knowledge, most of MARL focuses on "implicit symmetrizers" like OBL, well-tuned MAPPO, data augmentation, neural fictitious play-like schemes. Furthermore, the technique appears to be quite complementary and can be combined with existing MARL algorithms. 

Some asks: 
- this paper uses a combination of training and inference time symmetrizers, so please make it clear in the final version of the paper how each of these contribute to the stated performance gains. 
- it would be helpful to mention some of the statistical comments around "In our experience, a pool of 12 policies achieving 2.52 is particularly unlucky and is the result of several seeds that happen to completely misalign with all other policies." in an appendix, even if it's sort of anecdotal. 


recommendation: Accept
confidence: Less certain
award: No

From: NeurIPS.cc/2022/Conference/Paper2900/Authors
title: Reply to ef8A
comment: Thank you for your comment and consideration.

The paragraph in Section 3 promising an empirical test was regarding testing whether the averaging scheme helped in improving ZSC (which we did find as EQC improves ZSC), not to comparing it to using $e \in G$. We agree that a formal ablation comparing the two approaches would be interesting and valuable; in our own testing, we did find the averaging scheme tended to work better than the choice of $e \in G$. In light of the discussion period ending in a few hours, we will add this ablation study for the camera-ready deadline.

Thank you again for your helpful criticisms.

From: NeurIPS.cc/2022/Conference/Paper2900/Reviewer_ef8A
title: Response to Authors
comment: Thank you for your response.

I appreciate all the changes that you have made and I find the paper has improved. 

Nevertheless, reading other reviews, I am growing particularly sensible to Reviewer VbG9's point about the extension of equivariance to LSTMs:

In your reply to their review, you mention addition of details in Section 3, which I find commendable, but then the said paragraph points to Section 4 for an empirical test of the choice that was made between the two regularisation approaches: 

- (i) averaging scheme, vs 
- (ii) using the identity element $e\in G$ alone, 

I am realising that I initially understood that sentence wrongly ; it does not mean that you would propose an ablation study in Section 4 to compare the two regularisation approaches, but rather that you validate your choice of the averaging scheme (i) by showing that performance in that context is increased compared to baselines...

Yet, I would very much appreciate the paper to contain said ablation study between the two (or more) reuglarisation approaches for the hidden and cell states of the LSTMs in the architectuer, as I think it would be a fairly valuable contribution to make, and it is important to highlight it as a point where some questions remain open.

Still, I will raise my rating of the paper in accordance to my initial review from 7 to 8, as I find the theoretical guarantees to be impactful and, following revisions, also sufficiently numerically evidenced.

From: NeurIPS.cc/2022/Conference/Paper2900/Authors
title: Reply to Y7M6
comment: We appreciate the comments and thank you again for your helpful criticisms.

1. We maintain that the main claims (and the writing) have not materially changed (and see 2. below), which are that 1) equivariant networks exactly solve symmetry breaking; 2) EQC can be used as a policy-improvement operator. The background papers used for comparison have largely not changed (with the exception of additionally now considering the OBL policy population to further demonstrate the generality of EQC), we just now use the original pre-trained models from the relevant papers rather than training our own seeds of them.

2. We rephrase the claim as you recommend. We note that we show in the paper that the difference between OBL + EQC and OBL is statistically significant using permutation testing.

3. We apologize for the confusion. In Section 4.1, we show that despite using far fewer symmetries, we can still do better in ZSC than OP $\textit{due to symmetrization}$. We now additionally clarify that the G-OP policies are symmetrized at inference time in the caption of Table 1, but note this was stated in Sections 3 and 4.1. The comparison between OP and the symmetrized policies is fair because both policies are given access to symmetries (and OP given access even to far more symmetries), and we compare how effectively each policy type leverages this symmetry access: specifically, we compare if additionally constraining wrt symmetries at the model-level (equivariance) leads to improvement in coordination beyond constraining wrt the same symmetries purely at the data-level (data augmentation). Such comparisons between equivariance and data augmentation in given domains are common in literature and the subject of active research (eg [1]); in essence, section 4.1 quantifies the improvement that is gained from constraining for symmetry at the model-level versus the data-level for the multi-agent coordination domain, where we found that equivariant policies are far more “symmetry efficient”.

Please let us know if we need to address anything else.

[1] Gerken, Jan, et al. "Equivariance versus Augmentation for Spherical Images." International Conference on Machine Learning. PMLR, 2022.

From: NeurIPS.cc/2022/Conference/Paper2900/Reviewer_Y7M6
title: Reply to Authors
comment: Thanks for your reply. Overall I think the paper is improving. Response below.

1. While the overall thesis of the paper may still be similar to the initially submitted version, the claims of the paper and the ability to compare to other significant background papers that were used for comparison is fundamental, and these features are new to the revision. Even without considering any other edits, I believe this warrants the time to do another full review.

2. Thank you for using OBL Level 5. I think it would be better to claim that this work can improve state-of-the-art competitive models to become better instead of claiming this work achieves the new state-of-the-art ZSC performance in Hanabi. The latter claim leans on this work's highest score which, according to this work, is only 0.12 above the baseline score (achieved by symmetrizing that same baseline) and the error bounds almost overlap. On top of that, this claim also relies on [A] having made a mistake.

3. This reply confused me on G-OP training:

   - In the initial rebuttal, this was said in response to asking the differences between using G-OP training and using OP: "We train the G-OP policies on small subgroups of all symmetries, ie small subsets of the symmetries OP trains on (the cyclic group of order 5 and the dihedral group of order 10). We show in Sec 4.1 that despite training on far fewer symmetries we can still do better in ZSC than OP." 

   - However, in response to the follow-up question on why training on small sugroups performs better in ZSC than OP, this was said "The G-OP trained agents are symmetrized at inference time with our proposed mapping to be made equivariant, as stated in Sections 3 and 4.1." 

   - I'm now confused what the claim is here. Is it that training on fewer symmetries can do better in ZSC than OP or is it that symmetrizing at inference time is required to do better than OP? If it is the former, then what is the reasoning behind on why training on subgroups does better than OP as in Table 1? If it's the latter, then Table 1 should mention there is a symmetrizer used, and that does not seem like a fair comparison between these scores (if $C_{5}$-Equivariant and $D_{10}$-Equivariant get to use a symmetrizer at inference and OP does not). Also, if it's the case that training on fewer subgroups of symmetries by itself does not help over OP, then is the only technique presented in this work that gets an empirical benefit the inference-time symmetrizer?

From: NeurIPS.cc/2022/Conference/Paper2900/Authors
title: Reply to Y7M6
comment: Thanks for your comments.

1. While we have updated the paper and scores to reflect the reviewers' comments and requests, the changes are not fundamental and the thesis of the paper is very much the same: Section 3 has merely been split into subsections for ease of reading, without any profound revision of content, and while the scores in Section 4 are updated to reflect comparison with pre-trained models in the literature, the body of writing and conclusions drawn from the scores are the same.

2. We now report the scores with OBL Level 5 instead of OBL Level 4, and indeed still find EQC improves on OBL Level 5. We note that OBL can be iterated arbitrarily, and that the OBL paper itself only lists scores up to Level 4. We also note that the scores we found in cross-play are markedly different than what [1] reports: we found that OBL Level 5 attains cross-play scores of 23.77 ± 0.06 (slightly better than OBL Level 4), while [1] reports 24.2 ± 0.01. It is possible [1] confused cross-play scores with self-play scores.

3. The G-OP trained agents are symmetrized at inference time with our proposed mapping to be made equivariant, as stated in Sections 3 and 4.1. Thus despite training on less symmetries, symmetrizing wrt these few symmetries still outperforms a policy trained on all Hanabi symmetries. We also see in Section 4.2 that symmetrizing OP policies wrt a small subgroup of symmetries significantly improves the OP policies.

Please let us know of any other queries.

[1] Keane Lucas and Ross E. Allen. 2022. Any-Play: An Intrinsic Augmentation for Zero-Shot Coordination. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems (AAMAS '22). International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 853–861.

From: NeurIPS.cc/2022/Conference/Paper2900/Reviewer_Y7M6
title: Response to Authors
comment: Thank you for your response, as it did help me understand parts of your paper more. This paper has been significantly improved, especially by using baseline models that allow better comparison to related work. Some comments and a question:
- It looks like many sections in the paper have been edited, there is a new claim of SOTA score, and all scores and percentages in the results section previously reported are significantly different than what was initially submitted, which may warrant a re-submission of this paper. 
- In my understanding, this revision includes a new claim of a new state-of-the-art score (of 23.93) on ZSC for the Hanabi benchmark environment. However, this work achieved this by building upon OBL Level 4 models, whereas OBL Level 5 models (also freely available from the same source) achieve a ZSC cross-play score of 24.2 as reported in [A]. It's possible that this work could build upon the Level 5 model to get a higher score, but the new state-of-the-art claim the current revision makes seems inaccurate.
- G-OP training: In my understanding, one of the claims of this paper is that EQC outperforms OP (another symmetry-aware baseline). However, the only difference in training between the baseline OP training and this work's G-OP training is that this work limits training to subgroups of symmetries instead of all of them like Other-Play. What is the reasoning on why this would do empirically better or is superior to training with Other-Play?

[A] Keane Lucas and Ross E. Allen. 2022. Any-Play: An Intrinsic Augmentation for Zero-Shot Coordination. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems (AAMAS '22). International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 853–861.

From: NeurIPS.cc/2022/Conference/Paper2900/Authors
title: Response to Reviewer ef8A
comment: We thank the reviewer for their review.

Q0. Statistical tests

Thank you for your recommendation to include statistical tests to support the results.
We conducted Monte Carlo permutation tests to evaluate the improvement incurred by equivariance and have found that the improvement is indeed significant. We have included these findings in the paper. We opted for permutation tests rather than the two sample K-S test, as scores in Hanabi are discrete and not continuous. Additionally, we have redone our experiments to be more directly comparable with baselines in the literature. We have increased our computing resources to match those of other works and consequently achieved more competitive scores in Section 4. We additionally used pre-trained models supplied by other papers as baselines. In particular, we achieved a new state of the art on the Hanabi benchmark with OBL + EQC.

Q2: The symmetrizer does not require all the symmetries of \Phi to be known. The group G used in the symmetrizer must be fully known, but one can choose G to be a subgroup of \Phi. We do not intend to claim that choosing a subgroup G of \Phi is theoretically sufficient to prevent all symmetry breaking, but it is sufficient to prevent symmetry breaking wrt symmetries in G. We have rephrased this passage.

We do not consider S_5-equivariant agents in this work, as S_5 is already a group of fairly large order. The OP agents we compare against were trained with permutations from S_5, while the equivariant agents only with permutations from C_5 and D_10 respectively. Despite being trained on far fewer permutations, the equivariant agents perform better. We’ve made line 257 more clear as to what it’s referring to.

Q1: Any group can be viewed as a group of permutations, or as a group of permutation matrices. This is known as the permutation representation of a group. Also, the terminology “permutation matrix” alludes to the computational implementation of group actions as matrix products, and saying “matrix” instead of “function” simply emphasises that the function is a linear map of the input. We now state this in the paper.

Limitations

We have added to the Conclusion that our work focuses on the fundamental problem of correcting symmetry breaking, which is an important component but not the only component of coordination.

Remaining comments

We have changed “class of symmetries” to “set of symmetries”.

Following your suggestion, we expanded the proof of Proposition 3 in Section 3 to ease understanding.

We agree that “first approach” is somewhat ambiguous and have renamed “first approach” to “naive approach”.

We kindly thank the reviewer for their comments. The inclusion of the statistical tests as well as the refinement of the language has made the work a lot more convincing.

From: NeurIPS.cc/2022/Conference/Paper2900/Authors
title: Response to Reviewer Y7M6
comment: Thanks for your review.

Baseline models

Thank you for pointing out a concern with not using pre-trained baseline models. We trained our own baseline models to ensure the models were directly comparable to the EQC models. In [1], the authors used more compute than in our experiments: they use 2 GPUs rather than 1 for simulation, which, as the authors say, “has a profound effect on reducing the wall clock time required to achieve competitive performance”. In our work, we used a single GPU for simulation. Thus, the EQC models trained with our resources were not directly comparable with those of [1]. In addition, it is worth noting there is a lot of variance when evaluating cross-play scores of different seeds, and 9.43 is an outcome within reason for SAD. In our experience, a pool of 12 policies achieving 2.52 is particularly unlucky and is the result of several seeds that happen to completely misalign with all other policies.

In light of your comments, we now use an additional GPU for simulation for our equiv agents. This ensures our EQC models can be properly compared to [1]. In Sec 4, we revised our experiments to use the pretrained baselines instead of our own trained versions. We find our equiv agents outperform the OP agents from [1] (Sec 4.1), and both the SAD and OP baselines from [1] benefit from symmetrization at inference time (Sec 4.2). We also add statistical tests to strengthen our results.

Comparison with OBL

We agree OBL[2] doesnt require explicit formulation of symmetries. But, OBL requires simulator/env access to run rollouts of simulated play, which is arguably an even stronger assumption. EQC can be used under the assumption of only partial symmetry access, where an agent can become equiv wrt a small subgroup of all symmetries (see Sec 4.1 and 4.2 with the cyclic and dihedral groups).
Also, iterating multiple levels of OBL is very expensive.
Ultimately, OBL and EQC address related but different issues and are thus complementary to each other: 
OBL avoids the emergence of arbitrary communication protocols independently of the presence of environmental symmetries. However, it was specifically developed for turn-based partially observable settings. 
In contrast our work applies to any setting where symmetries are present, even if it is fully observable and players move at the same time. 
Practically speaking, EQC is an improved version of OP and can eg be used to solve the lever game in Sec 2.5, or utilise cheap-talk channels in iterated interactions; OBL fails in both of these settings.

Finally, EQC is also a policy-improvement operator (Sec 4.2), and in light of your comments we have added results to the paper demonstrating that applying EQC to OBL improves OBL, thus achieving new SOTA results in ZSC on the Hanabi benchmark. We’ve clarified the comparison with OBL in the paper.

LSTM equivariance

In our architecture, equivariance is ensured by averaging over all permutations of the observations and agent actions. We don’t conduct group actions on the LSTM hidden and cell states, as these need not be altered to ensure equiv in the observation-action spaces. The role of our modification to the LSTM layer (see Sec 3) is to keep performance in ZSC high while the network is transformed to an equiv one. All propositions in the paper account for LSTM nets. The proofs can be trivially adapted as follows: adding the hidden and cell states, h_t and c_t, as inputs along with the observations, x, and applying the group actions only to x. We opted not to add h_t and c_t in these proofs for ease of notation. We’ve made all this more explicit in Sec 3

Lever game

To find a payoff maximising policy during selfplay training, SP converges to a policy that picks one of the 1.0 levers, since during joint training agents can consistently break the symmetry between levers. At test time, this fails with novel partners who may have converged on different 1.0 levers. If we constrain ourselves to equiv policies while maximising payoff, we will get the policy that picks the 0.9 lever, because an equiv policy that picks 1.0 levers must choose uniformly at random to preserve equivariance, resulting in an expected return of 0.11, compared with 0.9 for choosing the 0.9 lever. We have clarified this in Sec 2.5

Symmetries during training

We train the G-OP policies on small subgroups of all symmetries, ie small subsets of the symmetries OP trains on (the cyclic group of order 5 and the dihedral group of order 10). We show in Sec 4.1 that despite training on far fewer symmetries we can still do better in ZSC than OP. Also, in Sec 4.2 we show we can symmetrize any policy, including OP, at inference time with EQC and still improve ZSC, which is significant

We thank the reviewer for their help in making the paper significantly stronger and more clear. Please let us know of any further comments/questions that would need to be addressed before you feel comfortable raising the score

[1] OP
[2] OBL

From: NeurIPS.cc/2022/Conference/Paper2900/Authors
title: Response to Reviewer VbG9
comment: We thank the reviewer for their comments.

Our main contributions are 1) showing that the theory of equivariant networks exactly solves symmetry breaking, which is a fundamental problem in multi-agent coordination, and 2) demonstrating that EQC, our method, can be used as a policy-improvement operator for any generic policy, which we validate on a highly challenging multi-agent benchmark (Hanabi). We agree that the architectural choices we propose for adapting LSTMs to an equivariant structure capable of multi-agent coordination are of note as well. We have further clarified the contributions.

Regarding your listed weakness that Section 3 is hard to follow, we thank you for your suggestion and have split the sections’ topics into subsections for ease of reading (3.1 Architecture, 3.2 A Theory for Symmetry and Coordination, 3.3 Algorithmic Approach), as you recommended.

Regarding your questions:

(1) + (2) LSTM Architecture modifications

The LSTM’s hidden and cell states need not be altered in order to ensure equivariance: this is because in our architecture we average over all symmetry permutations of the observations and agent actions, which ensures equivariance. If at every timestep we input random tensors for the hidden and cell states, our proposed architecture dictates that the network would still be equivariant in the observation-action spaces. The role of our architectural modification to the LSTM layer is to keep performance in zero-shot coordination high while the network is transformed to an equivariant one. All propositions in the paper indeed account for LSTM nets, where the proofs can be trivially adapted as follows: adding the hidden and cell states, h_t and c_t, as inputs along with the observations, x, and applying the group actions only to x. We opted not to add h_t and c_t in these proofs for ease of notation. We’ve made all this more clear in the updated version of the paper.
	
(3) Regarding asymptotic performance

In [1], the authors find that equivariance outperforms data augmentation in supervised learning even after increasing model size for data augmentation. 

​​In RL, increasing the size of the model can hurt training: for example, [2] turns a set of supervised learning image tasks into RL tasks, then compares the performance of a simple CNN with ResNet-18 (successful architecture in supervised learning tasks). In their experiments, the simple CNN outperforms the ResNet-18. This is evidence that training deep models with RL is difficult.

Equivariance outperforms data augmentation in supervised learning, even after increasing model size for data augmentation; and training deeper/bigger models with RL is difficult (as stated above). We would therefore not expect data augmentation (OP) to outperform equivariance (EQC) in our setting as we increase the model size.

(4) The only modification to the LSTM is the one mentioned in line 147. We found good performance in coordination with only this modification.

(5) We thank the reviewer for pointing out opportunities for improving the clarity of our paper. The batchsize we use is 128 and each sample in the batch is permuted with a randomly selected group element. We have clarified this in the paper.

(6) We assume this means the hyperparameters used for training, which are mostly set via empirical finetuning, as well as by using the hyperparameters used in prior works such as [3, 4] as references. You may refer to the appendix for the complete list of hyperparameter choices. Please clarify the meaning of “network parameter”.

Regarding prior knowledge of symmetries: our work does not require access to all symmetries. An agent can be equivariant with regards to a small subgroup of the group of all symmetries while achieving improvement in coordination (see Sections 4.1 and 4.2). Assumptions of some form are made in nearly all works on zero-shot coordination. For example [6] makes the strong assumption of access to the simulator (environment). Deriving symmetries is an interesting direction for future work. Works such as [5] exist as possible means to derive symmetries and relax the a priori access assumption. 


Finally, we would like to inform the reviewer that we have incorporated new results in our paper that demonstrate EQC improves on the state of the art for zero-shot coordination on the Hanabi benchmark.

We thank the reviewer for helping significantly improve the paper.

[1] Equivariance vs Augmentation for Spherical Images, Gerken et al 2022
[2] Natural environment benchmarks for RL, Zhang et al 2018
[3] Other-Play, Hu et al 2020
[4] Simplified action decoder, Hu et al 2020
[5] Quasi-equivalence discovery for ZS-emergent communication, Bullard et al 2021
[6] Off-belief learning, Hu et al 2021

From: NeurIPS.cc/2022/Conference/Paper2900/Reviewer_VbG9
rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary: The authors propose to use equivariant networks to handle the symmetry-breaking problem in multi-agent coordination settings. To generate an equivariant history-dependent policy, the authors present the equivariant LSTM structure and validate its performance in Hanabi games. 
strengths_and_weaknesses: Strengths:

 - I found the work interesting and believe the main technical contributions are (1) extending an equivariant MLP to an equivariant LSTM structure, (2) leveraging the permutation group while prior work mainly leverages the SO(2), (3) extending the equivariant net to game applications.

Weaknesses:

Originality:

 - Although the extension to the equivariant LSTM structure is interesting, I found the paper does not have an analysis tailored to LSTM (see the first two questions in the clarification part). Thus the contribution of this paper is not significant to me, but I look forward to the answers to the clarification questions below.

Clarification:

 - I found the Section 3 Method hard to follow. It could be improved by listing several subsections instead and highlighting the main contribution as extending the equivariant neural net to the LSTM setting.

questions: About contribution and soundness: 
- In section 3 line 147, the authors proposed a modification for LSTM layers, which is interesting. However, how is the modification here related to propositions 3 and 4 in the following line 155 and line 160? I assume that propositions 3 and 4 are not tailored for LSTM nets. 
 - A follow-up question is how the modifications in line 147 affect the equivariance of the proposed LSTM network?
 - What is the effect of the size of the network on the asymptotic performance? I assume that with a larger network and adequate data, the OP may have better performance. The equivariant net for sure helps improve the data efficiency but not necessarily the asymptotic performance. 

About writing and clarification:
 - The network modification is mysterious. Does the modification to LSTM only include the one mentioned in line 147?
 - In option 1 in line 177, the authors mention a mini-batch, is the batch size equal to 1? What is the effect of the group element batch size? 
 - How is the network parameter selected in this paper?

Minor issues:
 - Figure 1 is not referred to in the main content.


limitations: The authors addressed one limitation which is the prior knowledge of the group symmetry, which I think is crucial. 
ethics_flag: No
ethics_review_area: I don’t know
soundness: 2 fair
presentation: 2 fair
contribution: 2 fair

From: NeurIPS.cc/2022/Conference/Paper2900/Reviewer_Y7M6
rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary: This work approaches the zero-shot-coordination (ZSC) problem by explicitly restricting the learnable (or act-able) policies to only those that do not rely on known asymmetric convention in the environment. They do this by using a 'symmetrizer' that can be used to make any regular trained policy equivariant post-training or be used during training to train a network to act more equivariant (although a symmetrizer can still be used with this network on inference as well). Specifically, this work seeks to outperform prior symmetry-aware ZSC algorithms on the benchmark environment Hanabi.
strengths_and_weaknesses: Strengths
- This paper extends equivariance to recurrent neural networks
- This work is well written, especially considering the complexity of the material discussed
- This work does seem like a further mathemtical advance on eliminating symmetries similar to how Other-Play approaches ZSC

Weaknesses
- The original models for the basline are freely available, but this work chooses to train their own models. This adds another difference that makes it harder to compare to the baseline and calls into question the accuracy of the results.
- This paper seems to present an incrementally better solution towards dealing with explicitly defined symmetries in an environment, but more recent work (such as OBL [21]) does not require this explicit definition.
- I am uncertain of any empirical advantages it provides with the current state-of-the-art in ZSC. For example, Off-Belief-Learning [21], like this work, is also shown to eventually converge to the same policy regardless of the initialization, but achieves better results (20.85-23.76 cross-play) than this work without needing to explicitly define symmetries [21]
- This work chooses the same set of symmetries as Other-Play, and seems to train the neural networks the same or very similar as Other-Play to be invariant to these symmetries.
- In my understanding, this work trains its own versions of the baseline models, which seem to get very different scores than the original baseline papers report. However, these original papers publish already trained models that could be used (more details in questions)
questions: - This paper reports results for the baseline in Section 4.1 that are significantly different than the reported results in the baseline papers [19] and [20]. For example, the cross-play scores of Other-Play [20] in this work are 11.12, in the original OP paper they are 15.32 or 22.07 (with or without the auxilliary task, respectively) [20] and in this work SAD cross-play is reported as 9.43 in Table 2 but is reported as 2.52 in the original paper[20] (Table 1). What is the reason for these differences?

- How do you construct group actions suitable for cell state and hidden state i the LSTM as discussed in Proposition 2 in Section 3? In my understanding, the different dimensions in these states do not normally correspond to known observations or actions.

- If possible, I would appreciate more of a walkthrough in why the $\pi^{\text{equiv}}$ policy in section 2.5 will always choose the 0.9 lever.

- On page 5, what are the differeneces between the strategy of using "G-OP" and using Other-Play? Is it that Other-Play trains with more permutations?
limitations: Yes

Post-rebuttal change:
Increased score from 3 to 4 to reflect improvement of paper in rebuttal. However, as discussed elsewhere, I still have concerns on the magnitude of the changes in the revision and remaining uncertainty on experiment methodology and comparisons.
ethics_flag: No
soundness: 2 fair
presentation: 3 good
contribution: 2 fair

From: NeurIPS.cc/2022/Conference/Paper2900/Reviewer_ef8A
rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary: In cooperative MARL, following training with a given set of partners, it is important that the train agents be able to zero-shot coordinate with novel partners. 
A common failure mode, entitled symmetry breaking, happens when the trained agents learned on out of many equivalent (thanks to the game's symmetries) but mutually incompatible policies, yielding high performance with training partners but very low performance with novel partners.

To solve this failure mode, this paper proposes to build equivariant network architectures (assuming agents are modelled with deep neural networks, it means that "symmetric changes to their observation cause a corresponding change to their output"), in order for agents in Dec-POMDPs to learn policies that do not break symmetries.

Previous papers, like Other-Play which proposes an optimization algorithm that trains "agents to maximize expected return when randomly matched with symmetry-equivalent policies of their training time partners" via data augmentation, do not have equivariance guarantees.

Instead, this paper proposes Equivariant Coordinator (EQC) which has the following features:
- it comes with theoretical equivariance guarantees - the paper provides mathematical and numerical evidences; 
- it encodes the constraint of equivariance over the game's symmetries "not in the data, but in the network itself".
- it can be applied in two ways, either during training time (via a tractable approximation) or can be used as a policy-improvement operator at test-time (without approximation).

Comparing EQC with state-of-the-art and baseline approaches in zero-shot coordination (like OP, SAD, VDN) shows better performance, independantly of the way it is applied.
strengths_and_weaknesses: # Originality :
To my knowledge, the paper is the first to propose to build equivariant network architectures to encode environment's symmetries, in the context of zero-shot coordination.

# Quality : 
The quality of the paper is high, the maths seems very sound to me, and the experiments are compelling and well-rounded, as far as I can see.

Reproducibility seems strong to me.

Nevertheless, there is one quality/soundness issue around the strength of the numerical evidence:
--> line 287 :  Given the size of the standard deviations, I would appreciate that the authors perform a more thorough investigation using statistical tests to have a quantitative measure of the extent with which the hypothesis is actually supported by the results presented. For instance, using two-sample Kolmogorov-Smirnov tests between each condition's distribution, with the alternative argument set as 'less' or 'greater' [1].

[1] : SciPy's two-sample Kolmogorove-Smirnov test : https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ks_2samp.html


# Clarity :
I have found the paper to be mainly well-written and easy to follow.
That being said, I would invite the authors to address the following, mainly to increase accessibility and completeness of the paper:

--> line 102 & 140 : Clarity : I feel I need more details on the nature of $\Psi$, why defining it as a **class** of symmetries rather than a set of symmetries, for instance? It may be obvious but I am missing the point, I am sorry.

Could this choice be influenced by the fact that $L_g$ and $K_g$ /$K_g^{-1}$are denoted as"permutation matrices" rather than functions? (line 140)
Their usage in Eq.2 would invite for a functional definition, if I understand correctly. as they operate on $x,\psi(x)\in (\mathbb{R}^d)^2$ ...

--> line 157: Clarity : I would invite the authors to provide an easier reading experience by detailing that proof:

$$
\begin{align} 
\pi ^ 1(a | \tau ) &= \pi^1( \phi\circ\phi^{-1}(a) | \tau ) & \text{from } id=\phi\circ\phi^{-1} \\
&= \pi^1(K_{\phi}\circ\phi^{-1}(a) |\tau) & \text{by identification}\\
&= \pi^1( \phi^{-1}(a) | L_{\phi}(\tau) ) & \text{from Eq.2}\\
&= \pi^1(\phi^{-1}(a) | \phi(\tau) ) & \text{by identification}\\
&= \pi^2(a|\tau) & \text{from }\pi^1=\phi(\pi^2)
\end{align}
$$

--> line 200 : Clarity : the denomination "first approach" is ambiguisly used, maybe? Isn't G-OP the first approach, while the second approach is to use the *symmetrizer* $S$ on SP-trained $\psi$ and deploy $S(\psi)$?

--> line 220-221 : Clarity/Soundness : I am not sure I can agree with the last sentence: I do not think it is a fair description to say that the present method does not require G to be fully known, as the symmetrizer relies on it to be fully known, no?

I agree that, in practice, the paper shows numerical results that a subset of all the environment symmetries ***may*** be sufficient, but the theoretical evidence requires full knowledge of them, if I understand correctly.
Therefore I would invite the authors to rephrase here.

--> Table 1 : missing result in Table 1, for S5-equivariant agents, maybe? Line 257 seems to refer to it but those results are not in the table.


# Significance :
I think the paper is rightfully placed in the literature and very significant to the field of machine learning as zero-shot/human coordination is a very important problem for the field going forward. 
While the theoretical evidence are, in my viewpoint, very significant, I must confess that I am left disappointed with the numerical results not providing, at first glance, a very significant improvement to the state-of-the-art.
Thus, I am hoping that the statistical analysis that I am inviting the authors to perform will provide a compelling statistical significance argument to the numerical results, hopefully.
If not, then at least readers will know that there is still legitimately some work to do here.


questions: Q0 : Maybe the most important point for me, could you provide statistical analysis of the significance of the numerical results, possibly using two-sample KS tests as detailed above, please?

Addressing this would allow me to raise up my appreciation of the paper.

Q2 : Could you rephrase around line 220 in order to clearly differentiates the theoretical needs and guarantees from the practical ones, please?

Q1 : Could you clarify the denotation of "permutation matrices" as opposed to functions, maybe, as detailed above, please?
limitations: I believe the authors have mainly adequately addressed the limitations and potential negative societal impact.
Nevertheless, I would welcome the paper from investigating and/or discussing further the possible limitations that are the root cause of the numerical results falling short of solving the benchmark, maybe?


## Post-Rebuttal Changes:

I am raising my rating of the paper from 7 to 8.
Indeed, following the revisions made by the authors, I find the theoretical guarantees to be now both impactful and also sufficiently numerically evidenced.

ethics_flag: No
soundness: 3 good
presentation: 3 good
contribution: 4 excellent