Paper: Negatively Correlated Ensemble Reinforcement Learning for Online Diverse Game Level Generation (ICLR-2024)

Reviewer comment (Presentation):
Figure 1 has many math symbols and no caption on high-level ideas of each component; in addition, the meaning of $i\leftarrow 2$ in the figure is unclear.

Author response:
**1.Figure 1 has many math symbols and no caption on high-level ideas of each component; in addition, the meaning of $i \gets 2$ in the figure is unclear.**

We have updated Figure 1 and its caption to explain the components and math symbols.

Reviewer comment (Writing):
There is no clear definition on what a state is in the paper; the readers can only speculate that it is a latent vector from the end of Section 2.1 ("... a randomly sampled latent vector is used as the initial state").

Author response:
**2.There is no clear definition on what a state is in the paper; the readers can only speculate that it is a latent vector from the end of Section 2.1 ("... a randomly sampled latent vector is used as the initial state").**

We are sorry for such unclarity in the paper.  We have revised the description of the domain (state space, action space, reward criterion and diversity criterion) to make it clearer. In the revised paper, Section 3 “Problem Formulation” describes state and action. A state is a concatenated vector of a fixed number of latent vectors from recently generated segments. If there are not enough segments ($< n$) to construct a complete state, zeros will be padded in the vacant entries.

Reviewer comment (Theory):
There is no description on how Wasserstein distance is calculated. It is true that 2-Wasserstein distance between Gaussian distributions can be easily calculated, but it would be better if the Gaussian property can be emphasized at the beginning of Section 3.2, a formula can be given to make the paper self-contained, and "2-Wasserstein" instead of "Wasserstein" is specified.

Author response:
**3.There is no description on how Wasserstein distance is calculated. It is true that 2-Wasserstein distance between Gaussian distributions can be easily calculated, but it would be better if the Gaussian property can be emphasized at the beginning of Section 3.2, a formula can be given to make the paper self-contained, and "2-Wasserstein" instead of "Wasserstein" is specified.**

We have updated Section 3.2 as suggested by the reviewer. In the revision, Appendix C.2 is added to provide the formulation of 2-Wasserstein distance and how it can be calculated between Gaussian distributions.

Reviewer comment (Writing):
typos: in conclusion, bettwe -> better.

Author response:
**4.typos: in conclusion, bettwe -> better.**

We have fixed it and checked through the paper for typos.

Reviewer comment (Evaluation):
In Table 1, the trade-off of NCERL between reward and diversity in some of the environments does not follow the trend; for example, in Mario Puzzle, the diversity with $\lambda$ seems to have two peaks (0.2 and 0.5), and the reward at $\lambda=0.1$ is the worst despite of low diversity. Could the author explain this?

Author response:
**Q1: In Table 1, the trade-off of NCERL between reward and diversity in some of the environments does not follow the trend; for example, in Mario Puzzle, the diversity with λ seems to have two peaks (0.2 and 0.5), and the reward at λ=0.1 is the worst despite of low diversity. Could the author explain this?**

We consider this may be caused by some local optima during the training. Therefore, the training results of NCERL can be a little bit unstable. The black-box decoder further makes the training results hard to predict. The $\lambda$ factor does not affect the trade-off between reward and diversity but also affects the exploration efficiency, making the effect of $\lambda$ on the performance more non-monotonous. The relative poor performance of the generator trained with $\lambda = 0.1$ can be attributed to an abnormal trial which only yielded $30.23$ of average reward and $1723$ of diversity. For future work, we consider integrating our method with multi-objective reinforcement learning [1] to train a set of non-dominated policies with mutable regularisation coefficients. This could potentially bypass the problem of unstable and non-monotonous effects of the regularisation weight. To facilitate a more in-depth analysis of the results, we present the reward and diversity of each independent trial. Full results are presented and discussed in Appendix E.1.

Reviewer comment (Experiments):
Currently, there is only comparison between NCERL, ensemble RL methods and non-ensemble RL methods, and the encoding is over GAN. What is the performance of non-RL solutions, such as scripted (possibly with learnable parameters) solution, or supervised/self-supervised learning over more recent generative models such as diffusion models or VAEs?

Author response:
**Q2: Currently, there is only a comparison between NCERL, ensemble RL methods and non-ensemble RL methods, and the encoding is over GAN. What is the performance of non-RL solutions, such as scripted (possibly with learnable parameters) solutions, or supervised/self-supervised learning over more recent generative models such as diffusion models or VAEs?**

Due to the limited time, we are not able to compare our approach with all the non-RL approaches listed by the reviewer, but we made an effort to conduct a meaningful comparison. We use the code of [2] to train diffusion models (DDPM) for five independent trials. Specifically, we utilized the code from [2] to train DDPM for five independent trials. In this process, we randomly sampled 25 noises to generate 25 level segments, then concatenated them with an initial segment used in the RL generator testing. This procedure was applied to each of the 500 initial segments used for testing RL generators, resulting in a test set of 500 levels. We evaluated these test sets in terms of both reward and diversity. 

|Task|Criterion|DDPM|$\lambda$=$0.0$|$\lambda$=$0.1$|$\lambda$=$0.2$|$\lambda$=$0.3$|$\lambda$=$0.4$|$\lambda$=$0.5$|
|-------------|-----------|-----------------|-----------------|-----------------|-----------------|-----------------|-----------------|-----------------|
|MarioPuzzle|Reward|*-29.45*|**55.24**|51.42|53.78|53.22|54.59|53.26|
|MarioPuzzle|Diversity|1630|*1342*|1570|1940|1688|1698|**1967**|
|MultiFacet|Reward|*-119.4*|**46.39**|46.16|45.35|37.87|40.86|35.77|
|MultiFacet|Diversity|**1630**|*401.6*|492.3|620.2|1024|889.6|1142|

According to the results, DDPM exhibited good diversity scores but performed poorly in terms of reward. This is attributed to the fact that the training of DDPM does not consider reward functions that evaluate the quality of generated levels. To our knowledge, DDPM can not optimise certain objectives to cater to customised objectives, while the scripted method relies on domain knowledge and needs significant development costs. The generated samples of DDPM are available in our anonymous code repository (https://anonymous.4open.science/r/NCERL-Diverse-PCG-4F25/, the generation_results folder). 

[1] Hayes, Conor F., et al. "A practical guide to multi-objective reinforcement learning and planning." Autonomous Agents and Multi-Agent Systems 36.1 (2022): 26.

[2] Lee, Hyeon Joon, and Edgar Simo-Serra. "Using Unconditional Diffusion Models in Level Generation for Super Mario Bros." 2023 18th International Conference on Machine Vision and Applications. IEEE, 2023.

Reviewer comment (Theory):
the proposed approach assumes that the reward function is known and fixed. However, in practice, the reward function may be unknown or may change over time. Therefore, the proposed approach may not be applicable in such scenarios.

Author response:
Regarding the comment "**the proposed approach assumes that the reward function is known and fixed. However, in practice, the reward function may be unknown or may change over time. Therefore, the proposed approach may not be applicable in such scenarios.**": Although the reward function used in this paper is known and fixed, such a problem setting has extensive application scenarios in the domain of game content generation [1,2,3,6]. However, the reviewer has raised an interesting future research direction. Our proposed method can be made compatible with other approaches to tackle the challenge of unknown or unfixed rewards. For the issue of unknown rewards, inverse reinforcement learning can be used to build some reward functions from human demonstrations [9]. In the context of game content generation, it is possible to collect human-authored levels as demonstrations and learn a reward model from them with inverse reinforcement learning. Our method can be combined with inverse reinforcement learning to address tasks with unknown rewards. When facing the challenge of unfixed rewards, integrating reward estimation and replay sampling techniques [10] into our framework can be a viable solution. By combining our approach with these techniques, one can handle game content generation tasks with unknown or unfixed rewards.

Reviewer comment (Experiments):
the paper only considers a single game genre (platformer) and a single game engine (Super Mario Bros.). The proposed approach may not be directly applicable to other game genres or engines.

Author response:
Regarding the comment "**the paper only considers a single game genre (platformer) and a single game engine (Super Mario Bros.). The proposed approach may not be directly applicable to other game genres or engines.**": This paper employs Super Mario Bros. as the benchmark for our approach, as it is commonly used and representative within the Procedural Content Generation (PCG) community, and it is open-source. For more details on the potential applications of our approach to other games, please refer to our response to Q1.

Reviewer comment (Experiments):
can the proposed approach be directly applicable to 3D game levels or other types of game content? or what are the difficulties in this extension, such as complex reward design or high computational burden?

Author response:
**Q1: can the proposed approach be directly applicable to 3D game levels or other types of game content? or what are the difficulties in this extension, such as complex reward design or high computational burden?**

Our method can be directly applicable to 3D game levels like Minecraft generation [1] and other types of game content like music [2] and narrative generation [3] since RL has been applied to generate that content. Those applications could benefit from our proposed method since we only change the RL algorithm while maintaining the problem setting unchanged.

When applying our approach to other scenarios, the design of the reward can be challenging, as indicated by the reviewer. However, the procedural content generation (PCG) community has proposed a range of evaluation metrics [4], which can serve as the reward function in our framework. Taking Minecraft as an example again, Jiang et al. [1] have proposed some reward functions to train controllable RL-based game level generators, while there is also a set of evaluation metrics verified based on human evaluation scores [5], those metrics could be used as reward functions. For the issue of high computational burden, we have devised and implemented an asynchronous framework to speed up the training. On the other hand, the use of an action decoder, proposed in [6], contributes to faster generation speeds. There are some generative models that can generate high-quality 3D game levels, such as world-GAN [7] for Minecraft world generation and VAE-GAN for 3D indoor scene generation [8]. By employing these models as decoders, our method can be directly applied to the corresponding application scenarios with rapid generation speeds.

[1] Jiang, Zehua, et al. "Learning Controllable 3D Level Generators." Proceedings of the 17th International Conference on the Foundations of Digital Games. 2022.

[2] Jaques, Natasha, et al. "Generating music by fine-tuning recurrent neural networks with reinforcement learning." (2016).

[3] Huang, Qiuyuan, et al. "Hierarchically structured reinforcement learning for topically coherent visual story generation." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. No. 01. 2019.

[4] Shaker, Noor, Julian Togelius, and Mark J. Nelson. "Procedural content generation in games." (2016): 978-3.

[5] Hervé, Jean-Baptiste, and Christoph Salge. "Comparing PCG metrics with Human Evaluation in Minecraft Settlement Generation." Proceedings of the 16th International Conference on the Foundations of Digital Games. 2021.

[6] Shu, Tianye, Jialin Liu, and Georgios N. Yannakakis. "Experience-driven PCG via reinforcement learning: A Super Mario Bros study." 2021 IEEE Conference on Games (CoG). IEEE, 2021.

[7] Awiszus, Maren, Frederik Schubert, and Bodo Rosenhahn. "World-gan: a generative model for minecraft worlds." 2021 IEEE Conference on Games (CoG). IEEE, 2021.

[8] Li, Shuai, and Hongjun Li. "Deep Generative Modeling Based on VAE-GAN for 3D Indoor Scene Synthesis." International Journal of Computer Games Technology 2023 (2023).

Reviewer comment (Novelty):
Because the experiments are largely domain-specific and improve on diversity rather than pure performance, the significance of this work is limited.

Author response:
Besides the above point-to-point responses to the reviewer's questions, we would like to clarify the significance of this work regarding the reviewer's comments.

Regarding the comment "**Because the experiments are largely domain-specific and improve on diversity rather than pure performance, the significance of this work is limited.**": Although our work is domain-specific, online level generation is a broad and crucial area with extensive industrial applications. Procedural content generation via reinforcement learning is actively researched and applied in various gaming domains, including 2D [1] and 3D games [2], as well as virtual reality games [3]. On the other hand, many commercial games feature online level generation, for example, Minecraft, No Man’s Sky and a wide range of roguelike games like Spelunky and Hades. While traditional methods typically rely on human-crafted rules, machine learning-based methods can largely reduce the development cost [4-7]. Moreover, though we do not improve on reward, diversity itself is a goal of interest in not only game content generation, but also other RL directions like multi-agent RL [8] and quality-diversity RL [9]. 

[1] Khalifa, Ahmed, et al. "PCGRL: Procedural content generation via reinforcement learning." Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment. Vol. 16. No. 1. 2020.

[2] Jiang, Zehua, et al. "Learning Controllable 3D Level Generators." Proceedings of the 17th International Conference on the Foundations of Digital Games. 2022.

[3] Mahmoudi-Nejad, Athar, Matthew Guzdial, and Pierre Boulanger. "Arachnophobia exposure therapy using experience-driven procedural content generation via reinforcement learning (EDPCGRL)." Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment. Vol. 17. No. 1. 2021.

[4] Yannakakis, Georgios N., and Julian Togelius. Artificial intelligence and games. Vol. 2. New York: Springer, 2018.

[5] Shaker, Noor, Julian Togelius, and Mark J. Nelson. "Procedural content generation in games." (2016): 978-3.

[6] Liu, Jialin, et al. "Deep learning for procedural content generation." Neural Computing and Applications 33.1 (2021): 19-37.

[7] Guzdial, Matthew, Sam Snodgrass, and Adam J. Summerville. Procedural Content Generation Via Machine Learning: An Overview. Springer, 2022.

[8] Cui, B., Lupu, A., Sokota, S., Hu, H., Wu, D. J., & Foerster, J. N. (2022, September). Adversarial Diversity in Hanabi. In The Eleventh International Conference on Learning Representations.

[9] Wu, Shuang, et al. "Quality-Similar Diversity via Population Based Reinforcement Learning." The Eleventh International Conference on Learning Representations. 2022.

Reviewer comment (Novelty):
While there are novel derivations and a clean interpretation of regularizing the policy gradient, the idea of representing an agent as sub-policies has been explored in fields such as skill discovery and hierarchical reinforcement learning, which were not referenced in this work.

Author response:
Regarding the comment "**While there are novel derivations and a clean interpretation of regularizing the policy gradient, the idea of representing an agent as sub-policies has been explored in fields such as skill discovery and hierarchical reinforcement learning, which were not referenced in this work.**": Indeed, the idea of representing an agent as sub-policies has been explored in skill discovery and hierarchical reinforcement learning. We have added the relevant reference [10,11]  in the "Population-based RL" part of Section 2 of our revised paper.

[10] Pateria, Shubham, et al. "Hierarchical reinforcement learning: A comprehensive survey." ACM Computing Surveys (CSUR) 54.5 (2021): 1-35.

[11] Konidaris, George, and Andrew Barto. "Skill discovery in continuous reinforcement learning domains using skill chaining." Advances in neural information processing systems 22 (2009).

Reviewer comment (Evaluation):
The description of the domain is unclear to me as a reader, e.g. what is the action space of an agent generating Mario levels? What are the criteria used to evaluate reward and diversity? I would have liked to see examples of the generated levels.

Author response:
**Q4: The description of the domain is unclear to me as a reader, e.g. what is the action space of an agent generating Mario levels? What are the criteria used to evaluate reward and diversity?**

We have revised the description of the domain (state space, action space, reward criterion and diversity criterion) to make it clearer. In the revised paper, Section 3 “Problem Formulation” describes state and action, Section 6.1 "Online Level Generation Tasks" briefly describes the reward and Section 6.2 "Performance Criteria" introduces the reward and diversity criteria. Due to the page limit, more details including the formulations of reward function and performance criteria are presented in Appendix D.4 and D.1. The state space, action space, reward criterion, and diversity criterion are described as follows.

**State space**:  A $dn$ dimensional continuous space, where $d$ is the dimensionality of the latent vector of the action decoder and $n$ is the number of recently generated segments being considered in the reward function. A state is a concatenated vector of a fixed number of latent vectors of recently generated segments. If there are not enough segments have been generated (< n) to construct a state, zeros will be padded in the vacant entries.

**Action space**: A $d$-dimensional continuous space. An action is a latent vector which can be decoded into a level segment by the decoder. The decoder is a trained GAN in this work.

**Reward criterion**: reward criterion for a generator is calculated as $R = \sum_{t=1}^h R_t$ for each level, i.e., each MDP trajectory, then averaged over all levels generated for testing performance. The reward functions are adopted from previous level generation papers. We described them in Section 6.1 "Online Level Generation Tasks" formulated and described them in Appendix D.1. The description has been revised to make it clearer.

**Diversity criterion**: Diversity score of a generator is calculated as $D = \frac{2}{n(n-1)} \sum_{i=1}^n \sum_{j \neq i} \Delta(x_i, x_j)$, where $\Delta(\cdot, \cdot)$ indicates the Hamming distance, i.e., how many different tiles are there between the two levels to be compared; and $x_i$, $x_j$ indicate the $i^{\text{th}}$ one and the $j^{\text{th}}$ one in the $n$ levels generated for testing performance.

Reviewer comment (Novelty):
The section on asynchronous evaluation seems orthogonal to the main contribution of the work. Asynchronous RL has been explored in the actor-critic setting (e.g. A3C), which this work uses as it builds off SAC. Is there a specific connection between the asynchronous implementation and the novel contribution here?

Author response:
**Q1: The section on asynchronous evaluation seems orthogonal to the main contribution of the work. Asynchronous RL has been explored in the actor-critic setting (e.g. A3C), which this work uses as it builds off SAC. Is there a specific connection between the asynchronous implementation and the novel contribution here?**

Indeed, the asynchronous evaluation is orthogonal to the main contribution of this work. Therefore, we removed it from the contribution statements in Section 1 of the revised paper.

Reviewer comment (Experiments):
What does the behavior of the weighting-selector policy look like? It would provide more clarity into the method to showcase how often this selector policy utilizes specific sub-policies, or if certain sub-policies go unused.

Author response:
**Q2: What does the behaviour of the weighting-selector policy look like? It would provide more clarity into the method to showcase how often this selector policy utilizes specific sub-policies, or if certain sub-policies go unuse**

If the $\lambda$ used in training is large, the probabilities of selecting sub-policies change over time step, and all the sub-policies are used. While if the $\lambda$ is small, it is possible that some sub-policies go unused. To showcase the selection probabilities, we pick two NCERL generators trained with $m=5, \lambda= 0.5$ and $m = 5, \lambda=0.1$ and record their selection probabilities during stochastically generating two levels from the same initial state, respectively.

Regarding the generator trained with $\lambda = 0.5$, all the sub-policies are activated multiple times within the two trials. Regarding the generator trained with $\lambda = 0.1$, some of the selection probability is near zero and sub-policies 2 and 3 are never used within the two trials. As the regularisation coefficient $\lambda$ increases, the probabilities of selecting sub-policies become more uniform. The tables are included in Appendix E.2  of our revised paper with the images of generated levels in those two trials. We also present them as follows for reference.

Reviewer comment (Presentation):
It may help to label the other comparison methods in Figure 4.

Author response:
**Q3: It may help to label the other comparison methods in Figure 4.**

Thank you for your constructive suggestion. We have added the legends to label the algorithms and $\lambda$ values in Figure 4.