Paper: Off-the-Grid MARL: Datasets with Baselines for Offline Multi-Agent Reinforcement Learning (ICLR-2024)

Reviewer comment (Evaluation):
The categorization of datasets heavily based on the quality of experience may inadvertently introduce biases. A more well-rounded evaluation could be achieved by integrating additional qualitative and quantitative metrics.

Author response:
* Determining the boundaries for the different datasets (Good, Medium, Poor) required an in depth qualitative analysis of the various environments and how return related to agent skills. Unfortunately we opted not to include the details in the final writeup because we thought it would not be important to practitioners using our datasets. However, we see now that in fact the details may be valuable and have therefore added them in the appendix and we kindly invite the reviewer to read this analysis.

Reviewer comment (Evaluation):
The results section provides an overview of algorithmic performance but lacks analytical depth. Explaining the reasons behind the observed performances, such as the underperformance of vanilla QMIX, could offer more substantial insights.

Author response:
* As with related offline RL dataset publications RL Unplugged [1] and D4RL [2], providing detailed theoretical or empirical evidence for a discussion on why one baseline outperforms another is not the focus of this work. The purpose of including baselines was so that future works can easily compare their algorithms to baselines on our datasets. We agree however that a deeper analytical analysis is a valuable direction for future work.

Reviewer comment (Novelty):
There is a similar work, "Off-the-Grid MARL: Datasets and Baselines for Offline Multi-Agent Reinforcement Learning," that has been previously published in AAMAS. Does this submission introduce novel datasets or environments that extend beyond those covered in the AAMAS paper? Are there any innovative algorithmic approaches, evaluation metrics, or experimental setups that were not addressed in the prior publication? Further, how does the current paper tackle the challenges and limitations identified in the AAMAS publication?

Author response:
* The paper at AAMAS was an extended abstract and therefore non-archival and has not been published in any proceedings. Our work improves upon the extended abstract in several significant ways. Firstly, several new environments were supported, namely SMACv2, KAZ, MPE, Voltage Control and CityLearn. Secondly, we added a dataset generated from human players. And finally, we added a competitive offline dataset.

Reviewer comment (Novelty):
While this paper undeniably provides significant aid to the research community in terms of establishing a baseline database and engineering groundwork for MARL, its depth seems somewhat superficial. The ideas, though functional, are straightforward by testing different algorithms in different environments and producing new datasets (by using the old method).

Author response:
* We feel the significance of dataset papers to the field can not be overstated. RL Unplugged [1] and D4RL [2] both helped drive progress in the field of single-agent offline RL and enabled many breakthroughs in recent years [3,4]. Moreover, we echo reviewer KxKX’s remark that *“Benchmarking is essential in machine-learning communities as well as multi-agent learning communities.”* Without proper benchmarking contributions it's impossible to get a clear overview of the current state of the field. We believe it is for these reasons that ICLR welcomes “Dataset and Benchmark” contributions (see the list of subject areas at https://iclr.cc/Conferences/2024/CallForPapers).

Reviewer comment (Novelty):
This work is engineering important but has little contribution to the theoretical underpinnings or conceptual advancements in the field.

Author response:
* We feel the significance of dataset papers to the field can not be overstated. RL Unplugged [1] and D4RL [2] both helped drive progress in the field of single-agent offline RL and enabled many breakthroughs in recent years [3,4]. Moreover, we echo reviewer KxKX’s remark that *“Benchmarking is essential in machine-learning communities as well as multi-agent learning communities.”* Without proper benchmarking contributions it's impossible to get a clear overview of the current state of the field. We believe it is for these reasons that ICLR welcomes “Dataset and Benchmark” contributions (see the list of subject areas at https://iclr.cc/Conferences/2024/CallForPapers).

Reviewer comment (Reproducibility):
The major concern of the paper is the correctness of the implementation of the baselines. OMAR definitely outperforms CQL in many tasks, as reported in "Beyond Conservatism: Diffusion Policies in Offline Multi-agent Reinforcement Learning https://arxiv.org/abs/2307.01472". However, this is not true in Table D.5. As a dataset and benchmark paper, I think it's crucial to ensure that the results are replicable and the claims made for previous baselines are correct.

Author response:
We thank the reviewer for their feedback.

*“OMAR definitely outperforms CQL in many tasks, as reported in "Beyond 
Conservatism …”*

We are confident that our implementation is correct as it closely resembles the results reported by Barde et al. (2023) “A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problem” (https://arxiv.org/pdf/2305.17198.pdf). Here the authors evaluate OMAR on two new continuous action settings, namely Reacher (Table 2) and Ant (Table 3). In their results the authors show that the performance of ICQL and OMAR are very similar, with OMAR’s mean performance only marginally superior to ICQL in 7 out of 11 settings and their uncertainty estimates overlapping in all but one scenario. Furthermore, both OMAR and ICQL are outperformed by ITD3+BC and BC in all but one setting, and usually by some margin. These results closely resemble our reported finding, namely that the performance of ICQL and OMAR are very similar but significantly worse than ITD3+BC and BC on all tested continuous action settings.

The original OMAR work and the work from the reviewer’s cited paper share authors and therefore do not provide an independent verification of OMAR’s performance. To the best of our knowledge, the paper we cite above, represents the only independent study that uses OMAR, and this work corroborates our findings in terms of the performance of OMAR compared to other algorithms such as ICQL, ITD3+BC and BC.

In addition, we will add the following remark to the appendix to clarify why there might be a discrepancy in performance. *“We could not make OMAR or ICQL perform well on our tasks. We are unsure if this is because these algorithms perform poorly on our specific tasks, or if our implementations are missing an important detail. But since our results closely resemble the results reported by Barde et al. (2023), an independent work to the original OMAR paper, we decided to include them.”*

We trust we have provided enough evidence to validate our reported results using   OMAR and request the reviewer to kindly consider improving their score or provide us with their reason for why this is not possible.

Reviewer comment (Novelty):
Although this paper includes a novelty about MARL extension from single-agent RL datasets and baselines, other points seem to be ordinary. More challenging benchmarks and more real-world scenarios might provide more significance.

Author response:
* We agree that adding additional environments such as Google Football would be valuable. However, our priority has been to initially support all of the most popular MARL benchmarks including SMAC v1 & v2, MPE, MAMuJoCo and PettingZoo. Having said that, the data recorder provided in OG-MARL is flexible enough for researchers to use on a wide range of currently unsupported environments with minimal effort. Please see the tutorial we provided in the OG-MARL code for how to record data in a new environment.

Reviewer comment (Novelty):
P6: The authors said that “We chose these environments because they have visual (pixel-based) observations of varying sizes; an important dimension along which prior works have failed to evaluate their algorithms”. What are the prior works specifically and why did they fail the evaluation?

Author response:
* Due to space constraints we could only include a subset of all the baseline experiments we ran in the main text. All the baselines on the other environments were included in the Appendix. We chose to include the experiments on environments with pixel-based observations in the main text because prior offline MARL [1,2,3] works had not used such environments and in [4] the authors emphasize that benchmarks on pixel-based observations have been lacking in the broader offline RL literature.

Reviewer comment (Experiments):
For human data (KAZ), the detailed description will be described because humans have diversity and usually the property of the human participants (e.g., age and the game experience) should be reported. If possible, comparison with the data from RL algorithms will estimate the property of human data.

Author response:
With regard to the data acquisition, we will add the following to the main text. “We collected data from 10 people aged between 20 and 30 years. Half the participants were female and the other half were male. The players were unfamiliar with the game when they played it for the first time and had to learn through trial and error. Each pair of players was allowed to play the game for 10 episodes.”

Reviewer comment (Experiments):
The paper mentioned about KAZ that “The players where given no instruction on how to play the game and had to learn through trial and error.” but does it mean the data may include not only “learned” data but also “learning” data? The data acquisition process can be clarified.

Author response:
* Yes. To make sure the human-generated dataset included sufficiently diverse data we opted to include “learning” data.

Reviewer comment (Experiments):
More challenging MARL benchmarks such as team sports (e.g., [1] [2]) or more real-world robotics data might provide more significance.  
[1] Kurach et al. Google Research Football: A Novel Reinforcement Learning Environment, AAAI, 2020  
[2] Liu et al. From motor control to team play in simulated humanoid football, Science Robotics, 2022

Author response:
* We agree that adding additional environments such as Google Football would be valuable. However, our priority has been to initially support all of the most popular MARL benchmarks including SMAC v1 & v2, MPE, MAMuJoCo and PettingZoo. Having said that, the data recorder provided in OG-MARL is flexible enough for researchers to use on a wide range of currently unsupported environments with minimal effort. Please see the tutorial we provided in the OG-MARL code for how to record data in a new environment.

Reviewer comment (Novelty):
Please clarify the difference between this work and another recent work [1].

Author response:
* The AAMAS version of Off-the-Grid MARL was an extended abstract and therefore non-archival and has not been published in any proceedings. Our work includes several new environments (SMACv2, CityLearn, KAZ, MPE and Voltage Control). Additionally, we added a dataset generated from human players and a competitive MARL dataset.