Paper: Large Language Models as Rational Players in Competitive Economics Games (ICLR-2024)

Reviewer comment (Reproducibility):
As a paper that selects "dataset and benchmark" as the primary area, the experiments are not thorough enough to support the main claims and reproducibility is questionable. Most importantly, the prompts which are central for reproducibility are not presented.

Author response:
We believe the updated version fixed all these concerns from the reviewer.

Reviewer comment (Experiments):
Only multiple versions of GPT are considered, neglecting other commonly used LLMs (e.g., Claude 2 or LLama 2) which have been shown to have quite different performances in games compared to GPT, and only the results of average values are reported (e.g., Figure 2 and Figure 4).

Author response:
We believe the updated version fixed all these concerns from the reviewer.

Reviewer comment (Experiments):
The paper claims that they have varied the prompts to show the robustness of their results, but these experiments are not presented.

Author response:
We believe the updated version fixed all these concerns from the reviewer.

Reviewer comment (Reproducibility):
The paper acknowledges the importance of prompt sensitivity but fails to provide a detailed account of prompt structures, limiting the reproducibility of the experiments. It does not investigate different methods of prompting or incorporating historical data into prompts.

Author response:
Thanks for brining this up, we provided all the prompts we used in the updated Appendix B.

Reviewer comment (Experiments):
There is no discussion on whether the stability of outputs in the homogeneous model setting correlates with a consistent strategy distribution between all agents, nor is there an exploration of how the model's temperature setting influences strategy uniformity.

Author response:
Thanks for pointing this out. In our updated version, we use the self-compete set up for sanity check to verify consistency in strategies.

Reviewer comment (Experiments):
The experiments lack scenarios where LLMs interact with actual strategic agents, which would test the models in more realistic strategic environments.

Author response:
Thanks for the suggestions. In our updated version, we established several agents with hard-coded rational or irrational strategies, and showed the results of LLMs competing with these different baselines. Other pre-defined strategies were not included at the moment. Since the paper mainly focused on exploring competition between different LLM-based agents, actual strategic agents were not considered currently, but could serve as interesting extension.

Reviewer comment (Experiments):
How does the variation in prompt structure affect the strategic decision-making of LLMs, and what would be a good prompt structure to accurately assess their performance?

Author response:
Thanks for this good question! We agree that the prompt structure may affect the strategic decision-making of LLM, thus we decided to introduce the least amount of prompt engineering into this work. In the meantime, we have also set up the APIs for future users of EconArena to customise their own prompts for all games. A detailed discussion about the optimal prompt structure is not the focus of this work, as our aim is to provide the community a public economics environment for evaluating LLMs.

Reviewer comment (Experiments):
What role did the temperature parameter play in the stability of output in games where all models were the same? The paper argues about gpt-4 facing the inconsistency other models. For a given temperature parameter there would also be self-inconsistency. Also, as LLMs are rolled out into ever more realistic scenarios, the assumption of competing with the same agent is not realistic.

Author response:
Thanks for pointing this out. In our updated version, we use the self-compete set up for sanity check to verify consistency in strategies.

Reviewer comment (Evaluation):
How were the answers considered? Was it the next token in the generation process, or a number found after allowing the LLM to complete the sentence until reaching the <eos> token?

Author response:
Thanks for the question. We didn't post-process the answers returned from LLMs, as we specified the format that LLMs should follow in the prompts.

Reviewer comment (Evaluation):
What are the effective ways to prompt LLMs to account for historical information, and how can this be standardized to assess performance sensitivity to prompt phrasing? Since there is sensitivity in the prompt and as the history becomes larger, how is the information of the history compressed into the prompt?

Author response:
Thanks for bringing up this point. As we stated in our updated paper, the prioritised objective of this work is to provide the community a dynamic economics environments for evaluating LLMs. In the meantime, we tried to introduce the least amount of prompt engineering. Explore the optimal prompt structure for game history is beyond the scope of this work.