Paper: CivRealm: A Learning and Reasoning Odyssey in Civilization for Decision-Making Agents (ICLR-2024)

Reviewer comment (Experiments):
Some details are not clear enough. As this is a multi-agent environment, what are the baseline RL and LLM agents play against? I didn't seem to find such details in the description.

Author response:
Thank you very much for bringing this to our attention. The opponent players that our baseline methods play against are built-in AIs implemented by the original Freeciv team. We have added this detail in Appendix A.1 of the revision.

Reviewer comment (Experiments):
The proposed baselines are not working in the (full game) environment. Maybe it is better to also set different difficulty levels (e.g. by different task horizons) for the full game so that later research can more easily be evaluated on the benchmark.

Author response:
Thank you for the suggestion! We will be adding APIs to allow users to configure the length of a full game. This will enable later research projects to evaluate themselves on full games with varying time horizons.

The current full game is also highly customizable in several ways:

- The number of opponents and the map size can be adjusted.
- The level of resources on the map can be customized.
- It offers different levels of difficulty in terms of the built-in AI opponents.

Reviewer comment (Presentation):
The text in Figure 4 is too small to view.

Author response:
Thank you for the reminder. We have updated Figure 4 in the revised paper.

Reviewer comment (Evaluation):
Why the success rate of the RL agent in some mini-games (e.g., SettlerBuildCity) is higher for the hard setting than for the easier settings?

Author response:
We have also observed this interesting phenomenon. This counterintuitive situation arises from a *shortcut strategy* discovered by the RL agent. Specifically, as outlined in Appendix A.2 of the paper and detailed at <https://civrealm.github.io/civrealm/advanced_materials/minigame.html>, the winning condition for the hard mode of the SettlerBuildCity task is determined by averaging the potential city positions on the minigame map. In the harder mode, the land terrain is considerably less favorable, with an abundance of deserts, tundra, and swamps, resulting in lower tile production values for food, production, and trade. Consequently, the winning score is lower due to the averaging across these poorer terrains, resulting in a ranking of minimal winning scores as follows: easy > normal > hard.

Originally, the goal of the hard mode was to locate and settle in a resource-rich land area (e.g., forests or rivers) amidst these challenging terrains. However, sea and ocean tiles always offer more resources compared to the widely distributed poor land terrains. We have observed that tensor-based agents tend to exploit this unique scenario and learn a shortcut strategy of exclusively building coastal cities. This approach easily results in above-average production values for inland terrain, meeting the winning criteria for settlerBuildCity[hard].

Reviewer comment (Reproducibility):
It would be great if the authors could provide videos of the experiments or visualization ways in the future code repo.

Author response:
We included two videos in the supplementary materials: "Mastaba" and "BaseLang," each showcasing a complete game. In terms of visualization, our codebase offers users two functionalities: 1) the ability to visually observe the agent playing the game and 2) the option to automatically record screenshots of the game. We also customized the [FCIV-NET](https://github.com/fciv-net/fciv-net) version of the game for users, which provides a 3D visualization (shown in Fig 1).

Reviewer comment (Experiments):
I'm not sure how robust the client-server architecture used in the framework is. There are no comments on the robustness of this.

Author response:
That is an excellent question! During the development of CivRealm, we adhered to the test-driven development practice. We created unit tests, integration tests, and end-to-end tests to verify the reliability of various game operations and the training environment. Additionally, we conducted testing on multiple platforms, including Linux, Mac OS, and Windows, to ensure smooth gameplay for our agents to the best of our efforts. Most of the test code is available in the [civrealm/tests](https://github.com/civrealm/civrealm/tree/main/tests) folder within the released `civrealm` repository for your reference.

Reviewer comment (Reproducibility):
Is there support for designing custom games? I see Lua script being mentioned to specify the reward structure, new goals, etc. Is there also a support for new maps?

Author response:
Yes, we have uploaded a `civrealm-sav` repository in our released code. This repository enables users to customize new games and generate new maps. For a detailed tutorial, please refer to the 'Create new Mini-Game' content at: <https://civrealm.github.io/civrealm/advanced_materials/minigame.html>.