Paper: Sample-Efficient Quality-Diversity by Cooperative Coevolution (ICLR-2024)

Reviewer comment (Reproducibility):
I have strong doubts about the larger benchmark where CCQD is compared on the QDax suite to QDRL competitors. While I appreciate that the authors considered a large number of environments and repeated each experiment five times, I have serious doubts about the shown results as they don't match the ones reported in recent works such as PBT-ME (ICLR last year) produced on the exact same benchmark and for which the code is available.

Author response:
Thanks for your suggestion. As this work is more concerned with the sample efficiency, we originally trained over 5e7 timesteps. According to your suggestion, we have revised to run all the methods longer to show their asymptotic performance. As shown in Table 1,  Figure 2, and Table 8, our main conclusion still holds. The average ranks of the methods do not change a lot, where the ranks of QD-PG and PBT-ME get better slightly, and those of others get worse slightly. CCQD still has the best average rank (i.e., 1.12 and 1.88, respectively) on QD-Score AUC and Max Fitness.

Thanks to your suggestion, we have checked the reason for the roughly 4-times difference (e.g., 1500 vs. 6000) in Max Fitness that you mentioned. We find that the difference is not due to the lack of convergence, but to the different settings of the environments. We used episode length 250 in our initial experiments, which is consistent with previous works [1-2] and the default setting of QDax [3], while the maximum episode length is 1000 in [4]. The fitness of the environments is mainly determined by the total distance the robot moves, and a longer episode length allows the robot to move farther. Thus, the 4-times maximum episode length of the environments results in approximately 4-times Max Fitness. The similar results with our paper in these environments can be found in previous works [1-2]. 

To further illustrate this, we additionally re-run all the methods in the environments with episode length 1000, which is consistent with the setting in [4]. As shown in Table 7, Table 9 and Figure 11, the Max Fitness of the methods in these environments now is about 4 times as high as our original environments, and is consistent with the results in the previous work that use 1000 as the episode length, such as [4]. In this setting, CCQD still has the best average rank (i.e., 1.75 and 1.38, respectively) on QD-Score AUC and Max Fitness. 

From these updated and newly added empirical results, we can also find that the performance rank of PBT-ME with previous methods is mostly consistent with that reported in the original paper of PBT-ME [4]. For example, in Figure 11, the QD-Scores of PBT-ME are better than those of PGA-ME and QD-PG in Ant Uni and HalfCheetah Uni, and worse than those in Walker2D Uni; in Tables 8-9 (which give the comparison of the Max Fitness of different methods and are added to the revised paper due to your suggestion) in Appendix C, PBT-ME achieves the highest average rank of Max Fitness except CCQD in both environmental settings.

We put the added results of using the episode length 1000 in the appendix. But if you think that it would be beneficial to present the results of using the episode length 1000 rather than 250 in the main paper, please feel free to let us know.

Reviewer comment (Experiments):
The authors claim "PBT-ME is capable of adjusting the algorithm’s hyperparameters while optimizing, resulting in the need of a significant amount of computational resources" while one of the main points and contributions of the paper is to show that tuning the hyperparameters can be done with the same budget, and even in some cases improve the efficiency. The reported conclusions and performance ranking to other methods are not in line with the ones published while both the code and benchmark are open. I think this might come from the fact that all the runs were stopped too early and I would expect the authors to find back the original paper claim if they wait long enough to see the beginning of training convergence.

Author response:
The implementation of PBT-ME is based on its official implementation in QDax [5]. In our initial experiments, we kept the common hyperparameters consistent (e.g., population size and variation operators) to ensure a fair comparison. Thanks to your comment, we have revised to utilize the setting from the original paper to re-run PBT-ME. Here is the PBT-ME's performance comparison between our setting and the original setting:

|                 |    QD-Score |  QD-Score |    Coverage | Coverage | Max Fitness | Max Fitness |
|-----------------|------------:|----------:|------------:|---------:|------------:|------------:|
|                 | Our Setting |  Original | Our Setting | Original | Our Setting |    Original |
| Hopper Uni      |    609244.1 |  609217.0 |        75.8 |     77.3 |       770.6 |       702.1 |
| Walker2D Uni    |    633808.3 |  654797.5 |        69.6 |     72.7 |       725.1 |       692.3 |
| HalfCheetah Uni |   2974148.7 | 3020206.8 |       100.0 |    100.0 |      1928.4 |      1767.9 |
| Ant Uni         |    854323.4 |  952361.1 |        67.4 |     73.3 |      1808.1 |      1752.0 |
| Point Maze      |    234876.8 |  283454.1 |        73.3 |     81.5 |      -131.0 |      -105.0 |
| Ant Maze        |   1090822.0 | 1111988.9 |        32.4 |     34.5 |     -3498.6 |     -3932.8 |

We find that the results of PBT-ME under the two settings are quite similar. Under our setting, the Max Fitness is slightly higher, while under the original setting, the QD-Score is slightly higher. Note that we have not re-run PBT-ME in the two humanoid-based environments due to the limitation of device; we will do it and update the results in the final version.

Reviewer comment (Experiments):
Despite these limitations, in most cases the sample efficiency gain that is shown remains limited and I would expect more from an efficient representation/decision decoupling. In that context it is also hard to say if the small obtained gains compared to PGA-ME for instance are due to the co-evolution strategy being proposed or to some other minor changes like the PeVFA update of the critic or the fact that the population critic has been replaced by one critic per agent.

Author response:
Thanks for your comments. We believe the performance gain by coevolution is significant. The reasons are as follows.

1. The main difference between CCQD and DQN-ME in the context of Atari-Pong is the utilization of coevolution, where CCQD does not use other techniques such as PeVFA. We can find from Figure 4(c) that the performance of CCQD is significantly better than DQN-ME in Atari-Pong, which thus validates the effectiveness of the coevolution strategy. We have made revisions to the paper to enhance the clarity of this point. Thank you for your feedback. 

2. We conducted ablation studies on the number of critics and using of PeVFA, as shown in Appendix C.6 and C.7, respectively. The results show that using only one critic or not employing PeVFA does not decrease the performance significantly. Moreover, the performance remains significantly superior to the PGA-ME baseline, highlighting the effectiveness of coevolution.

Reviewer comment (Experiments):
The main experiments of the paper show strong weaknesses which make it unconvincing. It would be important for the authors to mitigate these points by running all methods longer, until observing the beginning of asymptotic convergence, and also by confirming findings of previous works. In case the authors find different results from previous works, a strong discussion about what could trigger these changes is expected. In the current state the work is not ready for publication, but if the authors were able to strengthen their experimental setup, this work could be acceptable for ICLR.

Author response:
Thanks for your suggestion. As this work is more concerned with the sample efficiency, we originally trained over 5e7 timesteps. According to your suggestion, we have revised to run all the methods longer to show their asymptotic performance. As shown in Table 1,  Figure 2, and Table 8, our main conclusion still holds. The average ranks of the methods do not change a lot, where the ranks of QD-PG and PBT-ME get better slightly, and those of others get worse slightly. CCQD still has the best average rank (i.e., 1.12 and 1.88, respectively) on QD-Score AUC and Max Fitness.

Thanks to your suggestion, we have checked the reason for the roughly 4-times difference (e.g., 1500 vs. 6000) in Max Fitness that you mentioned. We find that the difference is not due to the lack of convergence, but to the different settings of the environments. We used episode length 250 in our initial experiments, which is consistent with previous works [1-2] and the default setting of QDax [3], while the maximum episode length is 1000 in [4]. The fitness of the environments is mainly determined by the total distance the robot moves, and a longer episode length allows the robot to move farther. Thus, the 4-times maximum episode length of the environments results in approximately 4-times Max Fitness. The similar results with our paper in these environments can be found in previous works [1-2]. 

To further illustrate this, we additionally re-run all the methods in the environments with episode length 1000, which is consistent with the setting in [4]. As shown in Table 7, Table 9 and Figure 11, the Max Fitness of the methods in these environments now is about 4 times as high as our original environments, and is consistent with the results in the previous work that use 1000 as the episode length, such as [4]. In this setting, CCQD still has the best average rank (i.e., 1.75 and 1.38, respectively) on QD-Score AUC and Max Fitness. 

From these updated and newly added empirical results, we can also find that the performance rank of PBT-ME with previous methods is mostly consistent with that reported in the original paper of PBT-ME [4]. For example, in Figure 11, the QD-Scores of PBT-ME are better than those of PGA-ME and QD-PG in Ant Uni and HalfCheetah Uni, and worse than those in Walker2D Uni; in Tables 8-9 (which give the comparison of the Max Fitness of different methods and are added to the revised paper due to your suggestion) in Appendix C, PBT-ME achieves the highest average rank of Max Fitness except CCQD in both environmental settings.

We put the added results of using the episode length 1000 in the appendix. But if you think that it would be beneficial to present the results of using the episode length 1000 rather than 250 in the main paper, please feel free to let us know.