Paper: A Competition Winning Deep Reinforcement Learning Agent in microRTS (ICLR-2024)

Reviewer comment (Novelty):
A method that can well solve the performance under multiple different maps is not proposed.

Author response:
**Generalization to different maps:** In Table 4, we show that BC-PPO-Agent performs as  
well as AnonymizedAI on the 3 maps AnonymizedAI needed fine-tuned models while only  
using map-size specific models. We think this is a promising direction to  
generalization, and we added a paragraph at the end of the Behavior Cloning Results  
Section stating this.

Reviewer comment (Experiments):
How to ensure the high performance of the model in the new hidden map or an untrained map?

Author response:
**Generalization to different maps:** In Table 4, we show that BC-PPO-Agent performs as  
well as AnonymizedAI on the 3 maps AnonymizedAI needed fine-tuned models while only  
using map-size specific models. We think this is a promising direction to  
generalization, and we added a paragraph at the end of the Behavior Cloning Results  
Section stating this.  

**Generalization to larger maps:** We are working on generalizing to larger maps. We  
extended the Training on Larger Maps Discussion Section to include our current ideas.

Reviewer comment (Reproducibility):
The masking significantly reduces the action space per turn and makes training more efficient, and how to determine and obtain the action needs to be masked?

Author response:
**What to action mask?:** We used MicroRTS-Py as the starting point and masked out a  
couple more cases where the action would cause nothing to happen. Implementing action  
masking is a major contribution of human knowledge to the learner but has been  
well-established as critical for learning in microRTS.

Reviewer comment (Experiments):
Although an impressive feat of engineering, the paper provides little insight about which parts of the design were particularly important, how exploitable the method was, how to improve performance on larger maps, the required deeper strategy, and other important research questions surrounding designing a good RTS agent.

Author response:
We agree that the paper could be improved by
a thorough analysis of design decisions. Unfortunately, we were time-constrained by the
competition deadline and were experimenting with many different ideas going up to the
deadline. We ran training runs with changes and kept the changes that made 
improvements or we believed would be useful down the line.

One example of this is the scaling the policy loss by the number of units that could
take an action, which was critical for our behavior cloning. We have now added our reasoning into the paper on
why this is important; however, we believe this could be a candidate for an ablation
study to determine if any learning rate works without this scaling.

Other candidates for ablation studies include:
- transfer learning smaller map agents to larger maps
- varying reward weights, value loss coefficients, and entropy coefficients over time
- cost-based rewards

Reviewer comment (Experiments):
The submission could be significantly improved by focusing more on understanding why and how the system itself works, for example by including ablations of the different design decisions or attempting to train a deep RL agent on the largest map.

Author response:
We agree that the paper could be improved by
a thorough analysis of design decisions. Unfortunately, we were time-constrained by the
competition deadline and were experimenting with many different ideas going up to the
deadline. We ran training runs with changes and kept the changes that made 
improvements or we believed would be useful down the line.

One example of this is the scaling the policy loss by the number of units that could
take an action, which was critical for our behavior cloning. We have now added our reasoning into the paper on
why this is important; however, we believe this could be a candidate for an ablation
study to determine if any learning rate works without this scaling.

Other candidates for ablation studies include:
- transfer learning smaller map agents to larger maps
- varying reward weights, value loss coefficients, and entropy coefficients over time
- cost-based rewards

Reviewer comment (Novelty):
Instead of offering such analysis, the paper is mostly a grab-bag of previously-known techniques that combine to produce a very good agent.

Author response:
We agree that the paper could be improved by
a thorough analysis of design decisions. Unfortunately, we were time-constrained by the
competition deadline and were experimenting with many different ideas going up to the
deadline. We ran training runs with changes and kept the changes that made 
improvements or we believed would be useful down the line.

One example of this is the scaling the policy loss by the number of units that could
take an action, which was critical for our behavior cloning. We have now added our reasoning into the paper on
why this is important; however, we believe this could be a candidate for an ablation
study to determine if any learning rate works without this scaling.

Other candidates for ablation studies include:
- transfer learning smaller map agents to larger maps
- varying reward weights, value loss coefficients, and entropy coefficients over time
- cost-based rewards

Reviewer comment (Experiments):
You mention that scaling the behaviour cloning loss by the number of units that could take an action was critical to get it to train. Have you investigated why this is?

Author response:
One example of this is the scaling the policy loss by the number of units that could
take an action, which was critical for our behavior cloning. We have now added our reasoning into the paper on
why this is important; however, we believe this could be a candidate for an ablation
study to determine if any learning rate works without this scaling.

Reviewer comment (Experiments):
This paper covers a considerable amount of text on technical details, such as policy networks and speeding up inference. However, it is hard to gain insights for training DRL agents on other complex scenarios, such as Mahjong and Stratego. [See the questions below]

Author response:
**Applicability to other strategy games:** We focused on
microRTS because of its unique challenges and the existence of a benchmark without a
strong DRL agent. We have used the same techniques to train
agents for the Lux Season 2 competitions, which is a similar turn-based game requiring
the coordination of many units individually. We have not tried other games, but we
believe that many of these techniques would be applicable to other games. For example,
creating a curriculum of simpler scenarios and gradually increasing difficulty or using
a demonstration dataset to create a starting point for training are generally
applicable.

Reviewer comment (Novelty):
Discussions on some related works are missing, such as SCC [1], it achieves top human performance defeating GrandMaster players in test matches and top professional players in a live StarCraft II event with order of magnitude less computation.

Author response:
**SCC and related works discussion:** Thanks for pointing us to the SCC paper. It's
definitely impressive to accomplish similar results to AlphaStar with a smaller model
and order of magnitude less compute. We knew of TStarBot-X as another successful
StarCraft II agent trained with orders of magnitude less computation scale. We didn't go
into these papers because these still required clusters of GPUs to train (SCC doesn't
give specifics but order of magnitude less is tens of TPUs still). We thought microRTS and the
Kaggle Lux competitions were more relevant given similar scale.

Reviewer comment (Evaluation):
In Sec. 2.1, why self-play failed in UAS and GridNet?

Author response:
MicroRTS-Py had a bug where the second player would see resources (which should be
  unowned) as owned by the first player. This was fixed for our agent. We moved that
  mention up into the MicroRTS-Py section to make it more clear.

Reviewer comment (Experiments):
Why did not you use supervised learning?

Author response:
Behavior cloning treats a reinforcement learning problem as a supervised learning
   problem by making the ground truth the actions taken by the agent being cloned.

Reviewer comment (Writing):
What is Squnet?

Author response:
squnet is the name of our fewer parameter model. It is short for "Squeeze U-Net"
    because it uses squeeze excitation like DoubleCone while being shaped like a U-Net. It is not
    a U-Net.

Reviewer comment (Experiments):
What is the key takeaway for readers who want to use this proposed pipeline to train DRL agents on other complex scenarios, such as Stratego, Mahjong, PUBG and Honour of Kings?

Author response:
**Applicability to other strategy games:** We focused on
microRTS because of its unique challenges and the existence of a benchmark without a
strong DRL agent. We have used the same techniques to train
agents for the Lux Season 2 competitions, which is a similar turn-based game requiring
the coordination of many units individually. We have not tried other games, but we
believe that many of these techniques would be applicable to other games. For example,
creating a curriculum of simpler scenarios and gradually increasing difficulty or using
a demonstration dataset to create a starting point for training are generally
applicable.

Reviewer comment (Experiments):
It would be interesting to know whether the authors plan to explore the inclusion of more advanced self-play strategies, and how they believe these could potentially impact AnonymizedAI's performance.

Author response:
**Advanced self-play techniques:** In this paper we trained our agent against a combination of scripted opponents, the current agent, and past versions of the agent. Past versions of the agent were taken from periodic checkpoints during training, and the selected version was chosen randomly. As suggested, using fictitious self-play similar to TiZero or AlphaZero could be a promising direction for future work, especially for larger maps where multiple strategies are likely viable. A paragraph has been added to the Discussions sections.

Reviewer comment (Experiments):
The authors have mentioned the possibility of using the imitation learning approach for bootstrapping models with novel behaviors. Could they provide more insights into potential novel behaviors they are considering and how these could improve the agent's performance?

Author response:
**Novel behaviors from behavior cloning:** We see using the term "novel" was confusing. We replaced most instances of "novel" with "demonstrated", "competitive", and "effective".

Reviewer comment (Reproducibility):
I did not find a description of the hardware information for training the agents in the paper, such as the CPU model and number of cores, GPU model and number of cores, the size of training memory usage, etc. I wonder if the author could provide this information.

Author response:
**Training hardware details:** Sorry, this was an oversight. This has been added to Training Durations Supplementary Section.