Paper: Suspicion-Agent: Playing Imperfect Information Games with Theory of Mind Aware GPT-4 (ICLR-2024)

Reviewer comment (Evaluation):
The article's most significant weakness is the unreliability of the experimental results and its tendency to excessively exaggerate performance, as outlined in the questions section.

Author response:
Thank you for your time and effort in sharing critical feedback regarding our work. We provide the following response to address your concerns about results limitations, and thus respectfully hope you can consider the response to the final decision.

1. the evaluation of the method is conceptually flawed.

A: We acknowledge the concerns you raised regarding the experiments in our paper, and the omission of Nash equilibrium, a pivotal concept in evaluating algorithms in game settings. To address this, we added the relevant discussion in our paper, added the concept to the main paper and conducted additional experiments, specifically focusing on the CFR+ algorithm. Given the codebase of RLCard (https://github.com/datamllab/rlcard), we further train the CFR+ algorithm for 2000000 iterations and play our suspicion agent with CFR+, getting the following results:
|                      |     CFR+ |  Ours  (GPT-3.5) |  Ours (GPT-4) |        
|:--------------------:|:--------------------------------------------------------:|:------------------------------------------------------:|:------------------:|
|    CFR+      |  - |                     +126                   |                      +22                      |   
|      Ours (GPT-3.5)  | -126 |                             -                            |              
|     Ours (GPT-4)     |   -22  |                             -                            |                            -                           | 

which indicates the current LLM (without training examples and fine-tuning) cannot beat algorithms that reach Nash equilibrium when the iterations are large enough. Therefore, we have moderated our claims in the paper. Our revised findings demonstrate that our LLM may not surpass Nash Equilibrium strategies. **As a preliminary experimental study, Our focus is not on achieving Nash Equilibrium directly with the LLM but on exploring the design of an effective prompting system that can unlock the potential of LLMs in these complex game environments.** Our results highlight that  Suspicion-Agent can adaptively alter its behavior patterns when facing different opponents. This ability to effectively compete against simpler learning-based methods, which utilize reinforcement learning, is a noteworthy contribution to the field. Our research provides a foundation for further exploration into how LLMs can be utilized and improved for strategic decision-making in imperfect information games.

**In addition to the new results, as the first work to design the prompting system to enable LLM for imperfect information games, we still believe our proposed methods and the preliminary public experimental study and data are valuable for the community.**

1. We experimentally demonstrate that even with pre-trained data, **LLM itself  cannot play various imperfect information games well as Table 3 shows** ((-72 no ToM vs +24 second-order ToM)). Then we introduce the **first prompting system** to enable large language models like GPT-4 to play various imperfect information games using only the game rules and observations without any extra training, this may  inspire more LLM-based subsequent works.
2. Leveraging the theory-of-mind capabilities of large language models, our work is **the first to demonstrate that a simple prompting system can enable GPT-4 to outperform some learning-based algorithms like DMC and NFSP, even without specialized training or examples. However, the further experiments prove that it still cannot beat algorithms that can achieve** Nash-Equilibrium such as CFR+. In addition, we also qualitatively evaluate the potential of LLM into the different imperfect information games,  which may inspire more subsequent works.
3. We comprehensively identify and discuss the current limitations of employing large language models in imperfect information games, contributing valuable insights for the further AI-Agent research.
4. We are preparing to make all our code and interactive data publicly available. We hope that this will advance the community's understanding of the capabilities of large language models, particularly GPT-4. We also hope this will catalyze the development of more efficient models in the field of imperfect information games.

Reviewer comment (Evaluation):
The experimental results in the paper appear questionable and could potentially mislead readers about the capabilities of the LLM in handling imperfect information games.

Author response:
Thank you for your time and effort in sharing critical feedback regarding our work. We provide the following response to address your concerns about results limitations, and thus respectfully hope you can consider the response to the final decision.

1. the evaluation of the method is conceptually flawed.

A: We acknowledge the concerns you raised regarding the experiments in our paper, and the omission of Nash equilibrium, a pivotal concept in evaluating algorithms in game settings. To address this, we added the relevant discussion in our paper, added the concept to the main paper and conducted additional experiments, specifically focusing on the CFR+ algorithm. Given the codebase of RLCard (https://github.com/datamllab/rlcard), we further train the CFR+ algorithm for 2000000 iterations and play our suspicion agent with CFR+, getting the following results:
|                      |     CFR+ |  Ours  (GPT-3.5) |  Ours (GPT-4) |        
|:--------------------:|:--------------------------------------------------------:|:------------------------------------------------------:|:------------------:|
|    CFR+      |  - |                     +126                   |                      +22                      |   
|      Ours (GPT-3.5)  | -126 |                             -                            |              
|     Ours (GPT-4)     |   -22  |                             -                            |                            -                           | 

which indicates the current LLM (without training examples and fine-tuning) cannot beat algorithms that reach Nash equilibrium when the iterations are large enough. Therefore, we have moderated our claims in the paper. Our revised findings demonstrate that our LLM may not surpass Nash Equilibrium strategies. **As a preliminary experimental study, Our focus is not on achieving Nash Equilibrium directly with the LLM but on exploring the design of an effective prompting system that can unlock the potential of LLMs in these complex game environments.** Our results highlight that  Suspicion-Agent can adaptively alter its behavior patterns when facing different opponents. This ability to effectively compete against simpler learning-based methods, which utilize reinforcement learning, is a noteworthy contribution to the field. Our research provides a foundation for further exploration into how LLMs can be utilized and improved for strategic decision-making in imperfect information games.

**In addition to the new results, as the first work to design the prompting system to enable LLM for imperfect information games, we still believe our proposed methods and the preliminary public experimental study and data are valuable for the community.**

1. We experimentally demonstrate that even with pre-trained data, **LLM itself  cannot play various imperfect information games well as Table 3 shows** ((-72 no ToM vs +24 second-order ToM)). Then we introduce the **first prompting system** to enable large language models like GPT-4 to play various imperfect information games using only the game rules and observations without any extra training, this may  inspire more LLM-based subsequent works.
2. Leveraging the theory-of-mind capabilities of large language models, our work is **the first to demonstrate that a simple prompting system can enable GPT-4 to outperform some learning-based algorithms like DMC and NFSP, even without specialized training or examples. However, the further experiments prove that it still cannot beat algorithms that can achieve** Nash-Equilibrium such as CFR+. In addition, we also qualitatively evaluate the potential of LLM into the different imperfect information games,  which may inspire more subsequent works.
3. We comprehensively identify and discuss the current limitations of employing large language models in imperfect information games, contributing valuable insights for the further AI-Agent research.
4. We are preparing to make all our code and interactive data publicly available. We hope that this will advance the community's understanding of the capabilities of large language models, particularly GPT-4. We also hope this will catalyze the development of more efficient models in the field of imperfect information games.

Reviewer comment (Evaluation):
In the experiments on two-player zero-sum imperfect information games (specifically Leduc Hold’em), with CFR as one comparative method, the author claimed that Suspicion-Agent significantly outperformed CFR—a claim that appears to be entirely unreliable and incorrect.

Author response:
Thank you for your time and effort in sharing critical feedback regarding our work. We provide the following response to address your concerns about results limitations, and thus respectfully hope you can consider the response to the final decision.

1. the evaluation of the method is conceptually flawed.

A: We acknowledge the concerns you raised regarding the experiments in our paper, and the omission of Nash equilibrium, a pivotal concept in evaluating algorithms in game settings. To address this, we added the relevant discussion in our paper, added the concept to the main paper and conducted additional experiments, specifically focusing on the CFR+ algorithm. Given the codebase of RLCard (https://github.com/datamllab/rlcard), we further train the CFR+ algorithm for 2000000 iterations and play our suspicion agent with CFR+, getting the following results:
|                      |     CFR+ |  Ours  (GPT-3.5) |  Ours (GPT-4) |        
|:--------------------:|:--------------------------------------------------------:|:------------------------------------------------------:|:------------------:|
|    CFR+      |  - |                     +126                   |                      +22                      |   
|      Ours (GPT-3.5)  | -126 |                             -                            |              
|     Ours (GPT-4)     |   -22  |                             -                            |                            -                           | 

which indicates the current LLM (without training examples and fine-tuning) cannot beat algorithms that reach Nash equilibrium when the iterations are large enough. Therefore, we have moderated our claims in the paper. Our revised findings demonstrate that our LLM may not surpass Nash Equilibrium strategies. **As a preliminary experimental study, Our focus is not on achieving Nash Equilibrium directly with the LLM but on exploring the design of an effective prompting system that can unlock the potential of LLMs in these complex game environments.** Our results highlight that  Suspicion-Agent can adaptively alter its behavior patterns when facing different opponents. This ability to effectively compete against simpler learning-based methods, which utilize reinforcement learning, is a noteworthy contribution to the field. Our research provides a foundation for further exploration into how LLMs can be utilized and improved for strategic decision-making in imperfect information games.

**In addition to the new results, as the first work to design the prompting system to enable LLM for imperfect information games, we still believe our proposed methods and the preliminary public experimental study and data are valuable for the community.**

1. We experimentally demonstrate that even with pre-trained data, **LLM itself  cannot play various imperfect information games well as Table 3 shows** ((-72 no ToM vs +24 second-order ToM)). Then we introduce the **first prompting system** to enable large language models like GPT-4 to play various imperfect information games using only the game rules and observations without any extra training, this may  inspire more LLM-based subsequent works.
2. Leveraging the theory-of-mind capabilities of large language models, our work is **the first to demonstrate that a simple prompting system can enable GPT-4 to outperform some learning-based algorithms like DMC and NFSP, even without specialized training or examples. However, the further experiments prove that it still cannot beat algorithms that can achieve** Nash-Equilibrium such as CFR+. In addition, we also qualitatively evaluate the potential of LLM into the different imperfect information games,  which may inspire more subsequent works.
3. We comprehensively identify and discuss the current limitations of employing large language models in imperfect information games, contributing valuable insights for the further AI-Agent research.
4. We are preparing to make all our code and interactive data publicly available. We hope that this will advance the community's understanding of the capabilities of large language models, particularly GPT-4. We also hope this will catalyze the development of more efficient models in the field of imperfect information games.

Reviewer comment (Theory):
In two-player zero-sum games, following a Nash equilibrium strategy ensures a non-negative expected payoff against any strategy. CFR is widely used for calculating Nash equilibrium in such games, and given that Leduc Hold’em is a small game for which CFR can easily determine Nash equilibrium, no algorithm should theoretically be able to outperform CFR in expectation. Therefore, the claim that Suspicion-Agent can beat CFR on Leduc Hold’em is theoretically incorrect.

Author response:
Thank you for your time and effort in sharing critical feedback regarding our work. We provide the following response to address your concerns about results limitations, and thus respectfully hope you can consider the response to the final decision.

1. the evaluation of the method is conceptually flawed.

A: We acknowledge the concerns you raised regarding the experiments in our paper, and the omission of Nash equilibrium, a pivotal concept in evaluating algorithms in game settings. To address this, we added the relevant discussion in our paper, added the concept to the main paper and conducted additional experiments, specifically focusing on the CFR+ algorithm. Given the codebase of RLCard (https://github.com/datamllab/rlcard), we further train the CFR+ algorithm for 2000000 iterations and play our suspicion agent with CFR+, getting the following results:
|                      |     CFR+ |  Ours  (GPT-3.5) |  Ours (GPT-4) |        
|:--------------------:|:--------------------------------------------------------:|:------------------------------------------------------:|:------------------:|
|    CFR+      |  - |                     +126                   |                      +22                      |   
|      Ours (GPT-3.5)  | -126 |                             -                            |              
|     Ours (GPT-4)     |   -22  |                             -                            |                            -                           | 

which indicates the current LLM (without training examples and fine-tuning) cannot beat algorithms that reach Nash equilibrium when the iterations are large enough. Therefore, we have moderated our claims in the paper. Our revised findings demonstrate that our LLM may not surpass Nash Equilibrium strategies. **As a preliminary experimental study, Our focus is not on achieving Nash Equilibrium directly with the LLM but on exploring the design of an effective prompting system that can unlock the potential of LLMs in these complex game environments.** Our results highlight that  Suspicion-Agent can adaptively alter its behavior patterns when facing different opponents. This ability to effectively compete against simpler learning-based methods, which utilize reinforcement learning, is a noteworthy contribution to the field. Our research provides a foundation for further exploration into how LLMs can be utilized and improved for strategic decision-making in imperfect information games.

**In addition to the new results, as the first work to design the prompting system to enable LLM for imperfect information games, we still believe our proposed methods and the preliminary public experimental study and data are valuable for the community.**

1. We experimentally demonstrate that even with pre-trained data, **LLM itself  cannot play various imperfect information games well as Table 3 shows** ((-72 no ToM vs +24 second-order ToM)). Then we introduce the **first prompting system** to enable large language models like GPT-4 to play various imperfect information games using only the game rules and observations without any extra training, this may  inspire more LLM-based subsequent works.
2. Leveraging the theory-of-mind capabilities of large language models, our work is **the first to demonstrate that a simple prompting system can enable GPT-4 to outperform some learning-based algorithms like DMC and NFSP, even without specialized training or examples. However, the further experiments prove that it still cannot beat algorithms that can achieve** Nash-Equilibrium such as CFR+. In addition, we also qualitatively evaluate the potential of LLM into the different imperfect information games,  which may inspire more subsequent works.
3. We comprehensively identify and discuss the current limitations of employing large language models in imperfect information games, contributing valuable insights for the further AI-Agent research.
4. We are preparing to make all our code and interactive data publicly available. We hope that this will advance the community's understanding of the capabilities of large language models, particularly GPT-4. We also hope this will catalyze the development of more efficient models in the field of imperfect information games.

Reviewer comment (Experiments):
One possible explanation for the experimental results is that the sample size of 100 games is too small for poker games, which have substantial randomness. Such a small sample can lead to highly variable, unrepresentative results, and more games would be needed for accurate assessment (although this may be cost-prohibitive for Suspicion-Agent).

Author response:
Thank you very much for time and effort to providing us with more references and we will discuss with them in the final version to improve our paper. 

First, we agree with you that randomness plays a significant role for imperfect information game evaluation. To improve the randomness of our experiments, we already designed two types of experiments to compare Suspicion Agent with baselines:

1. Reducing the randomness of Random Seeds: 
    
    Our agent plays against different baselines for 100 games utilizing varying random seeds for each game. This tactic is intended to dampen the stochastic variability introduced by the random seed settings. 
    
2. **Reducing the randomness of Card Strength/Position: (which is also used in the recommended AIVAT [1])**
    
    Given the limited games, **the distributions of strong hand and weak hands are not equal for each player in limited game rounds.** To ensure a fair comparison, it was crucial that both the Suspicion-Agent and the opponent model experienced the same card strength across an equal number of games. **To achieve this balance, we employed a strategic approach: initially, the Suspicion-Agent was placed in position 0 for the first set of 50 games. Subsequently, we reproduce these 50 games with same cards as the first one but switched the positions, placing the Suspicion-Agent in position 1 and the baseline model in position 0. This method allowed us to maintain an equal card strength for both the Suspicion-Agent and the baseline model over a total of 100 games, thereby enabling a more accurate evaluation of each model's performance.**
    

Given these two designs, we believe that we can alleviate the effect of randomness in imperfect information games. Given the results in Table 1 and Table2 in the main paper: Suspicion Agent achieves superior performance over learning-based baselines except CFR+ in all 100 random games, 50 games in position 0 and 50 games in position 1. They are the clear evidence to show the advantages of our suspicion agent.

Lastly, we would like to highlight that our study is not only the first to enable LLM for imperfect information games but also the first to consider the interaction between two agents by introducing the theory of mind using LLM. We believe our work sets a valuable baseline and will serve as an inspiration for future research in this area. We respectfully hope that you reconsider your decision in light of these contributions and the innovative aspects of our study.

[1] Burch, Neil, et al. "Aivat: A new variance reduction technique for agent evaluation in imperfect information games." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 32. No. 1. 2018.

Reviewer comment (Experiments):
Another possible explanation is that the CFR algorithm used in the experiments may not have converged, leading to strategies that deviate significantly from Nash equilibrium. This would undermine the validity of the performance comparison.

Author response:
Thank you for your time and effort in sharing critical feedback regarding our work. We provide the following response to address your concerns about results limitations, and thus respectfully hope you can consider the response to the final decision.

1. the evaluation of the method is conceptually flawed.

A: We acknowledge the concerns you raised regarding the experiments in our paper, and the omission of Nash equilibrium, a pivotal concept in evaluating algorithms in game settings. To address this, we added the relevant discussion in our paper, added the concept to the main paper and conducted additional experiments, specifically focusing on the CFR+ algorithm. Given the codebase of RLCard (https://github.com/datamllab/rlcard), we further train the CFR+ algorithm for 2000000 iterations and play our suspicion agent with CFR+, getting the following results:
|                      |     CFR+ |  Ours  (GPT-3.5) |  Ours (GPT-4) |        
|:--------------------:|:--------------------------------------------------------:|:------------------------------------------------------:|:------------------:|
|    CFR+      |  - |                     +126                   |                      +22                      |   
|      Ours (GPT-3.5)  | -126 |                             -                            |              
|     Ours (GPT-4)     |   -22  |                             -                            |                            -                           | 

which indicates the current LLM (without training examples and fine-tuning) cannot beat algorithms that reach Nash equilibrium when the iterations are large enough. Therefore, we have moderated our claims in the paper. Our revised findings demonstrate that our LLM may not surpass Nash Equilibrium strategies. **As a preliminary experimental study, Our focus is not on achieving Nash Equilibrium directly with the LLM but on exploring the design of an effective prompting system that can unlock the potential of LLMs in these complex game environments.** Our results highlight that  Suspicion-Agent can adaptively alter its behavior patterns when facing different opponents. This ability to effectively compete against simpler learning-based methods, which utilize reinforcement learning, is a noteworthy contribution to the field. Our research provides a foundation for further exploration into how LLMs can be utilized and improved for strategic decision-making in imperfect information games.

**In addition to the new results, as the first work to design the prompting system to enable LLM for imperfect information games, we still believe our proposed methods and the preliminary public experimental study and data are valuable for the community.**

1. We experimentally demonstrate that even with pre-trained data, **LLM itself  cannot play various imperfect information games well as Table 3 shows** ((-72 no ToM vs +24 second-order ToM)). Then we introduce the **first prompting system** to enable large language models like GPT-4 to play various imperfect information games using only the game rules and observations without any extra training, this may  inspire more LLM-based subsequent works.
2. Leveraging the theory-of-mind capabilities of large language models, our work is **the first to demonstrate that a simple prompting system can enable GPT-4 to outperform some learning-based algorithms like DMC and NFSP, even without specialized training or examples. However, the further experiments prove that it still cannot beat algorithms that can achieve** Nash-Equilibrium such as CFR+. In addition, we also qualitatively evaluate the potential of LLM into the different imperfect information games,  which may inspire more subsequent works.
3. We comprehensively identify and discuss the current limitations of employing large language models in imperfect information games, contributing valuable insights for the further AI-Agent research.
4. We are preparing to make all our code and interactive data publicly available. We hope that this will advance the community's understanding of the capabilities of large language models, particularly GPT-4. We also hope this will catalyze the development of more efficient models in the field of imperfect information games.

Reviewer comment (Evaluation):
In either of the above scenarios, there are significant concerns regarding the reliability of the current experimental results.

Author response:
Thank you for your time and effort in sharing critical feedback regarding our work. We provide the following response to address your concerns about results limitations, and thus respectfully hope you can consider the response to the final decision.

1. the evaluation of the method is conceptually flawed.

A: We acknowledge the concerns you raised regarding the experiments in our paper, and the omission of Nash equilibrium, a pivotal concept in evaluating algorithms in game settings. To address this, we added the relevant discussion in our paper, added the concept to the main paper and conducted additional experiments, specifically focusing on the CFR+ algorithm. Given the codebase of RLCard (https://github.com/datamllab/rlcard), we further train the CFR+ algorithm for 2000000 iterations and play our suspicion agent with CFR+, getting the following results:
|                      |     CFR+ |  Ours  (GPT-3.5) |  Ours (GPT-4) |        
|:--------------------:|:--------------------------------------------------------:|:------------------------------------------------------:|:------------------:|
|    CFR+      |  - |                     +126                   |                      +22                      |   
|      Ours (GPT-3.5)  | -126 |                             -                            |              
|     Ours (GPT-4)     |   -22  |                             -                            |                            -                           | 

which indicates the current LLM (without training examples and fine-tuning) cannot beat algorithms that reach Nash equilibrium when the iterations are large enough. Therefore, we have moderated our claims in the paper. Our revised findings demonstrate that our LLM may not surpass Nash Equilibrium strategies. **As a preliminary experimental study, Our focus is not on achieving Nash Equilibrium directly with the LLM but on exploring the design of an effective prompting system that can unlock the potential of LLMs in these complex game environments.** Our results highlight that  Suspicion-Agent can adaptively alter its behavior patterns when facing different opponents. This ability to effectively compete against simpler learning-based methods, which utilize reinforcement learning, is a noteworthy contribution to the field. Our research provides a foundation for further exploration into how LLMs can be utilized and improved for strategic decision-making in imperfect information games.

**In addition to the new results, as the first work to design the prompting system to enable LLM for imperfect information games, we still believe our proposed methods and the preliminary public experimental study and data are valuable for the community.**

1. We experimentally demonstrate that even with pre-trained data, **LLM itself  cannot play various imperfect information games well as Table 3 shows** ((-72 no ToM vs +24 second-order ToM)). Then we introduce the **first prompting system** to enable large language models like GPT-4 to play various imperfect information games using only the game rules and observations without any extra training, this may  inspire more LLM-based subsequent works.
2. Leveraging the theory-of-mind capabilities of large language models, our work is **the first to demonstrate that a simple prompting system can enable GPT-4 to outperform some learning-based algorithms like DMC and NFSP, even without specialized training or examples. However, the further experiments prove that it still cannot beat algorithms that can achieve** Nash-Equilibrium such as CFR+. In addition, we also qualitatively evaluate the potential of LLM into the different imperfect information games,  which may inspire more subsequent works.
3. We comprehensively identify and discuss the current limitations of employing large language models in imperfect information games, contributing valuable insights for the further AI-Agent research.
4. We are preparing to make all our code and interactive data publicly available. We hope that this will advance the community's understanding of the capabilities of large language models, particularly GPT-4. We also hope this will catalyze the development of more efficient models in the field of imperfect information games.

Reviewer comment (Evaluation):
The claim that Suspicion-Agent can outperform traditional algorithms like CFR without specialized training or examples is incorrect, and the paper requires substantial revision before it could be suitable for publication.

Author response:
Thank you for your time and effort in sharing critical feedback regarding our work. We provide the following response to address your concerns about results limitations, and thus respectfully hope you can consider the response to the final decision.

1. the evaluation of the method is conceptually flawed.

A: We acknowledge the concerns you raised regarding the experiments in our paper, and the omission of Nash equilibrium, a pivotal concept in evaluating algorithms in game settings. To address this, we added the relevant discussion in our paper, added the concept to the main paper and conducted additional experiments, specifically focusing on the CFR+ algorithm. Given the codebase of RLCard (https://github.com/datamllab/rlcard), we further train the CFR+ algorithm for 2000000 iterations and play our suspicion agent with CFR+, getting the following results:
|                      |     CFR+ |  Ours  (GPT-3.5) |  Ours (GPT-4) |        
|:--------------------:|:--------------------------------------------------------:|:------------------------------------------------------:|:------------------:|
|    CFR+      |  - |                     +126                   |                      +22                      |   
|      Ours (GPT-3.5)  | -126 |                             -                            |              
|     Ours (GPT-4)     |   -22  |                             -                            |                            -                           | 

which indicates the current LLM (without training examples and fine-tuning) cannot beat algorithms that reach Nash equilibrium when the iterations are large enough. Therefore, we have moderated our claims in the paper. Our revised findings demonstrate that our LLM may not surpass Nash Equilibrium strategies. **As a preliminary experimental study, Our focus is not on achieving Nash Equilibrium directly with the LLM but on exploring the design of an effective prompting system that can unlock the potential of LLMs in these complex game environments.** Our results highlight that  Suspicion-Agent can adaptively alter its behavior patterns when facing different opponents. This ability to effectively compete against simpler learning-based methods, which utilize reinforcement learning, is a noteworthy contribution to the field. Our research provides a foundation for further exploration into how LLMs can be utilized and improved for strategic decision-making in imperfect information games.

**In addition to the new results, as the first work to design the prompting system to enable LLM for imperfect information games, we still believe our proposed methods and the preliminary public experimental study and data are valuable for the community.**

1. We experimentally demonstrate that even with pre-trained data, **LLM itself  cannot play various imperfect information games well as Table 3 shows** ((-72 no ToM vs +24 second-order ToM)). Then we introduce the **first prompting system** to enable large language models like GPT-4 to play various imperfect information games using only the game rules and observations without any extra training, this may  inspire more LLM-based subsequent works.
2. Leveraging the theory-of-mind capabilities of large language models, our work is **the first to demonstrate that a simple prompting system can enable GPT-4 to outperform some learning-based algorithms like DMC and NFSP, even without specialized training or examples. However, the further experiments prove that it still cannot beat algorithms that can achieve** Nash-Equilibrium such as CFR+. In addition, we also qualitatively evaluate the potential of LLM into the different imperfect information games,  which may inspire more subsequent works.
3. We comprehensively identify and discuss the current limitations of employing large language models in imperfect information games, contributing valuable insights for the further AI-Agent research.
4. We are preparing to make all our code and interactive data publicly available. We hope that this will advance the community's understanding of the capabilities of large language models, particularly GPT-4. We also hope this will catalyze the development of more efficient models in the field of imperfect information games.

Reviewer comment (Experiments):
The game of Texas and Ludec is still fairly simple: which means the rounds per game is not large and the context window may be able to handle several full games without compressing or discarding any history. (I am not very sure about each module's context structure in the construction among rounds / games, frankly.) And the situation does not change much in large scale.

Author response:
A:  Good question. To address this, we have incorporated a summarization module in our system. This module is designed to concisely summarize the progress of each game, thereby efficiently managing the use of context space. This approach not only optimizes the use of the context window but also highlights the significance of the long context window capability in Language Learning Models (LLMs) for AI agents.

Reviewer comment (Experiments):
Most of the templates are given, so the GPT-4 is thinking in the given logic. Only those games with very clear given logic can be played by second-order ToM version of Suspicion-Agent.

Author response:
A:  Good point. Even without predefined templates, GPT-4 has the capability to generate reasonable analyses, showcasing its second-order Theory of Mind (ToM) abilities. However, one of the inherent challenges with LLMs like GPT-4 is their tendency towards producing hallucinations—outputs that might seem plausible but are factually inaccurate or inconsistent. To mitigate this issue, we introduced templates into our system. These templates serve as a guiding framework, helping to steer GPT-4's responses towards more accurate and relevant analyses by providing a structured approach to processing and responding to information. The effectiveness of this method is evident in the results, where the use of templates has significantly alleviated the problem of hallucinations, leading to more reliable and coherent outputs from the model.

Reviewer comment (Experiments):
Is there a study on the intermediate responses from GPT-3.5 and GPT-4, about **correctness in calculations**, on the distribution of opponent's hand, the probability of winning rate, expected rewards, etc.? I think the agent did not specifically develop a numerical model for calculating them and insert them into prompts.

Author response:
A:  In response to your question, we have conducted a qualitative evaluation focusing on the correctness of calculations in the intermediate responses generated by GPT-3.5 and GPT-4. Our findings indicate that GPT-3.5 shows limited proficiency in reasoning and calculation. On the other hand, GPT-4 demonstrates a notable improvement in this area, achieving approximately 80% accuracy in calculations relevant to playing games. This improved performance in GPT-4 can be attributed to its more advanced algorithms and larger training dataset, which enhance its ability to process and analyze complex game situations that involve probabilistic and numerical reasoning.

Reviewer comment (Experiments):
If the above calculation values of GPT-3.5 is usually incorrect, then would checking and correcting them be effective in leveraging its behavior?

Author response:
A: Good Point. While addressing the calculation inaccuracies of GPT-3.5 could potentially improve its performance, it's important to consider that the model's limitations extend beyond just calculation errors. GPT-3.5 also exhibits challenges with reasoning abilities, particularly in comprehending and applying complex game rules. This aspect significantly impacts its capability to effectively play imperfect information games

Reviewer comment (Theory):
Is there a necessity in constructing or thinking with 3rd-order (or even higher) ToM, while playing the game?

Author response:
A: This is an intriguing question. The necessity of constructing or utilizing third-order (or higher) Theory of Mind (ToM) in game-playing scenarios has been a topic of discussion in previous research [1,2]. These studies suggest that ToM beyond the second-order tends to become less effective. This reduced effectiveness is primarily attributed to the increased complexity involved in calculating third-order (or higher) ToM. This increase in complexity can lead to diminishing returns in terms of strategic advantage, especially in practical gaming scenarios. Players or AI agents attempting to anticipate an opponent's beliefs about another player's beliefs (which is the essence of third-order ToM) often face considerable challenges in accurately processing and utilizing this information effectively. Thus, while higher-order ToM may provide theoretical insights into more complex strategic thinking, its practical application in game-playing contexts is limited by these computational and cognitive constraints.

Reviewer comment (Experiments):
Has anyone (human player) played with Suspicion Agent? How does it behave? Is there any bluffing tricks to win it?

Author response:
A: Following your suggestions, we have conducted experiments involving human players interacting with the Suspicion Agent. During these interactions, the Suspicion Agent typically exhibits an aggressive play style initially. This observation aligns with findings from previous research [3]. During the course of the game, we noticed that once human players are bluffed by the Suspicion Agent, they tend to adjust their strategy. Normally, human players become more cautious and start to second-guess the Agent's moves. This change in human behavior is a reaction to the aggressive tactics of the Suspicion Agent. The key for human players to succeed against it seems to involve a mix of caution and strategic risk-taking. Recognizing the bluffing patterns of the Agent and responding with well-timed aggressive moves of their own can be effective. In our experiments, because most human players are not the top player of poker, Suspicion-Agent can normally get positive rewards.

Reviewer comment (Writing):
In Algorithm 1. Typos: "My Pettern" (My Pattern, I guess), and "Reflection" (Reflexion).

Author response:
Thank you for your suggestion, we already fixed the typos following your suggestion in the updated version.

Reviewer comment (Experiments):
The game on Leduc Hold’em is too specific. It is possible that the winning strategy of playing Leduc Hold’em was discussed online and then used to train LLMs. Then, the assumption that LLMs are trained without specialized training or examples for playing this game is not held anymore.

Author response:
A: **We agree that the pre-trained knowledge is helpful for LLM. However, the pre-training itself cannot enable LLM to perform imperfect information well. As Table 3 shows, the simple prompting without ToM prompts is not able to achieve promising results, (-72 no ToM vs +24 second-order ToM). This is a strong evidence to show the effectiveness of our prompting system and the potential of LLM in imperfect information games.**

Reviewer comment (Experiments):
CFR used in experiments is too old, and new versions should be considered, e.g., CFR+, Discounted CFR, predictive CFR.  

Tammelin, O., 2014. Solving large imperfect information games using CFR+. arXiv preprint arXiv:1407.5042.  

Brown, N. and Sandholm, T., 2019, July. Solving imperfect-information games via discounted regret minimization. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 33, No. 01, pp. 1829-1836).  

Farina, G., Kroer, C. and Sandholm, T., 2021, May. Faster game solving via predictive blackwell approachability: Connecting regret matching and mirror descent. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 35, No. 6, pp. 5363-5371).

Author response:
A: We acknowledge the concerns you raised regarding the experiments in our paper, and the omission of Nash equilibrium, a pivotal concept in evaluating algorithms in game settings. To address this, we added the relevant discussion in our paper, added the concept to the main paper and conducted additional experiments, specifically focusing on the CFR+ algorithm. Given the codebase of RLCard (https://github.com/datamllab/rlcard), we further train the CFR+ algorithm for 2000000 iterations and play our suspicion agent with CFR+, getting the following results:
|                      |     CFR+ |  Ours  (GPT-3.5) |  Ours (GPT-4) |        
|:--------------------:|:--------------------------------------------------------:|:------------------------------------------------------:|:------------------:|
|    CFR+      |  - |                     +126                   |                      +22                      |   
|      Ours (GPT-3.5)  | -126 |                             -                            |              
|     Ours (GPT-4)     |   -22  |                             -                            |                            -                           | 

which indicates the current LLM (without training examples and fine-tuning) cannot beat algorithms that reach Nash equilibrium when the iterations are large enough. Therefore, we have moderated our claims in the paper. Our revised findings demonstrate that our LLM may not surpass Nash Equilibrium strategies. **As a preliminary experimental study, Our focus is not on achieving Nash Equilibrium directly with the LLM but on exploring the design of an effective prompting system that can unlock the potential of LLMs in these complex game environments.** Our results highlight that  Suspicion-Agent can adaptively alter its behavior patterns when facing different opponents. This ability to effectively compete against simpler learning-based methods, which utilize reinforcement learning, is a noteworthy contribution to the field. Our research provides a foundation for further exploration into how LLMs can be utilized and improved for strategic decision-making in imperfect information games.

Reviewer comment (Experiments):
The CFR strategy used in experiments may not be trained well to be a Nash equilibrium strategy. In two-player zero-sum games, a Nash equilibrium strategy is not exploitable. However, in the proposed experiments, CFR strategies are exploited: “CFR may occasionally raise even with a mid-level hand to exert pressure on Suspicion-Agent. In these instances, Suspicion-Agent’s tendency to fold can lead to losses……………..when CFR chooses to check—often indicating a weak hand—or when DMC checks—suggesting its hand doesn’t align with the public cards—Suspicion-Agent will raise as a bluff to induce folds from the opponents.” Another possible reason is that the CFR (mixed) strategy is not presented well in experiments.

Author response:
A: We acknowledge the concerns you raised regarding the experiments in our paper, and the omission of Nash equilibrium, a pivotal concept in evaluating algorithms in game settings. To address this, we added the relevant discussion in our paper, added the concept to the main paper and conducted additional experiments, specifically focusing on the CFR+ algorithm. Given the codebase of RLCard (https://github.com/datamllab/rlcard), we further train the CFR+ algorithm for 2000000 iterations and play our suspicion agent with CFR+, getting the following results:
|                      |     CFR+ |  Ours  (GPT-3.5) |  Ours (GPT-4) |        
|:--------------------:|:--------------------------------------------------------:|:------------------------------------------------------:|:------------------:|
|    CFR+      |  - |                     +126                   |                      +22                      |   
|      Ours (GPT-3.5)  | -126 |                             -                            |              
|     Ours (GPT-4)     |   -22  |                             -                            |                            -                           | 

which indicates the current LLM (without training examples and fine-tuning) cannot beat algorithms that reach Nash equilibrium when the iterations are large enough. Therefore, we have moderated our claims in the paper. Our revised findings demonstrate that our LLM may not surpass Nash Equilibrium strategies. **As a preliminary experimental study, Our focus is not on achieving Nash Equilibrium directly with the LLM but on exploring the design of an effective prompting system that can unlock the potential of LLMs in these complex game environments.** Our results highlight that  Suspicion-Agent can adaptively alter its behavior patterns when facing different opponents. This ability to effectively compete against simpler learning-based methods, which utilize reinforcement learning, is a noteworthy contribution to the field. Our research provides a foundation for further exploration into how LLMs can be utilized and improved for strategic decision-making in imperfect information games.

Reviewer comment (Experiments):
The comparison results among different algorithms for Game Coup and Texas Hold’em Limit should be presented.

Author response:
A: Limited by the budgets, we just showcased it in the original paper. Following your suggestions, we evaluate Coup with 5 people, each people can play it in 5 games. The average of winning rate of our proposed method against human being is 72%, which also proved the effectiveness of the proposed method.

Reviewer comment (Evaluation):
Nash equilibrium is never mentioned in the paper, which is the correct way of assessing the performance of algorithms in games.

Author response:
We acknowledge the concerns you raised regarding the experiments in our paper, and the omission of Nash equilibrium, a pivotal concept in evaluating algorithms in game settings. To address this, we added the relevant discussion in our paper, added the concept to the main paper and conducted additional experiments, specifically focusing on the CFR+ algorithm. Given the codebase of RLCard (https://github.com/datamllab/rlcard), we further train the CFR+ algorithm for 2000000 iterations and play our suspicion agent with CFR+, getting the following results:
|                      |     CFR+ |  Ours  (GPT-3.5) |  Ours (GPT-4) |        
|:--------------------:|:--------------------------------------------------------:|:------------------------------------------------------:|:------------------:|
|    CFR+      |  - |                     +126                   |                      +22                      |   
|      Ours (GPT-3.5)  | -126 |                             -                            |              
|     Ours (GPT-4)     |   -22  |                             -                            |                            -                           | 

which indicates the current LLM (without training examples and fine-tuning) cannot beat algorithms that reach Nash equilibrium when the iterations are large enough. Therefore, we have moderated our claims in the paper. Our revised findings demonstrate that our LLM may not surpass Nash Equilibrium strategies. **As a preliminary experimental study, Our focus is not on achieving Nash Equilibrium directly with the LLM but on exploring the design of an effective prompting system that can unlock the potential of LLMs in these complex game environments.** Our results highlight that  Suspicion-Agent can adaptively alter its behavior patterns when facing different opponents. This ability to effectively compete against simpler learning-based methods, which utilize reinforcement learning, is a noteworthy contribution to the field. Our research provides a foundation for further exploration into how LLMs can be utilized and improved for strategic decision-making in imperfect information games.

Reviewer comment (Experiments):
The opponents considered are not optimal choices; new algorithms such as CFR+ or its optimistic variants (e.g. [1,2]) are now the state of the art in solving such games.

Author response:
A: Thanks for your suggestions, we add the discussion about these works to the updated version.

Reviewer comment (Experiments):
DQN, DMC and NFSP are deep learning approaches that are not well motivated in such small games.

Author response:
Leveraging the theory-of-mind capabilities of large language models, our work is **the first to demonstrate that a simple prompting system can enable GPT-4 to outperform some learning-based algorithms like DMC and NFSP, even without specialized training or examples. However, the further experiments prove that it still cannot beat algorithms that can achieve** Nash-Equilibrium such as CFR+. In addition, we also qualitatively evaluate the potential of LLM into the different imperfect information games,  which may inspire more subsequent works.

Reviewer comment (Evaluation):
In such games, any good algorithm reaches ~0 exploitability (meaning that it finds optimal strategies, in terms of NE) and, by definition, no strategy can win against such strategies when considering enough games; thus, either the CFR implementation is wrong (or far from convergence) or the evaluation methodology is flawed in other ways.

Author response:
We acknowledge the concerns you raised regarding the experiments in our paper, and the omission of Nash equilibrium, a pivotal concept in evaluating algorithms in game settings. To address this, we added the relevant discussion in our paper, added the concept to the main paper and conducted additional experiments, specifically focusing on the CFR+ algorithm. Given the codebase of RLCard (https://github.com/datamllab/rlcard), we further train the CFR+ algorithm for 2000000 iterations and play our suspicion agent with CFR+, getting the following results:
|                      |     CFR+ |  Ours  (GPT-3.5) |  Ours (GPT-4) |        
|:--------------------:|:--------------------------------------------------------:|:------------------------------------------------------:|:------------------:|
|    CFR+      |  - |                     +126                   |                      +22                      |   
|      Ours (GPT-3.5)  | -126 |                             -                            |              
|     Ours (GPT-4)     |   -22  |                             -                            |                            -                           | 

which indicates the current LLM (without training examples and fine-tuning) cannot beat algorithms that reach Nash equilibrium when the iterations are large enough. Therefore, we have moderated our claims in the paper. Our revised findings demonstrate that our LLM may not surpass Nash Equilibrium strategies. **As a preliminary experimental study, Our focus is not on achieving Nash Equilibrium directly with the LLM but on exploring the design of an effective prompting system that can unlock the potential of LLMs in these complex game environments.** Our results highlight that  Suspicion-Agent can adaptively alter its behavior patterns when facing different opponents. This ability to effectively compete against simpler learning-based methods, which utilize reinforcement learning, is a noteworthy contribution to the field. Our research provides a foundation for further exploration into how LLMs can be utilized and improved for strategic decision-making in imperfect information games.

Reviewer comment (Theory):
Do you consider only two players games?

Author response:
Yes, as a preliminary proof-concept study, we only consider two players games to make the problem simplified, and we would like extend it into multi-agent (more than 2) in the subsequent works.

Reviewer comment (Evaluation):
Why are the evaluations made in terms of win/loss instead of exploitability as common in GT?

Author response:
We acknowledge the concerns you raised regarding the experiments in our paper, and the omission of Nash equilibrium, a pivotal concept in evaluating algorithms in game settings. To address this, we added the relevant discussion in our paper, added the concept to the main paper and conducted additional experiments, specifically focusing on the CFR+ algorithm. Given the codebase of RLCard (https://github.com/datamllab/rlcard), we further train the CFR+ algorithm for 2000000 iterations and play our suspicion agent with CFR+, getting the following results:
|                      |     CFR+ |  Ours  (GPT-3.5) |  Ours (GPT-4) |        
|:--------------------:|:--------------------------------------------------------:|:------------------------------------------------------:|:------------------:|
|    CFR+      |  - |                     +126                   |                      +22                      |   
|      Ours (GPT-3.5)  | -126 |                             -                            |              
|     Ours (GPT-4)     |   -22  |                             -                            |                            -                           | 

which indicates the current LLM (without training examples and fine-tuning) cannot beat algorithms that reach Nash equilibrium when the iterations are large enough. Therefore, we have moderated our claims in the paper. Our revised findings demonstrate that our LLM may not surpass Nash Equilibrium strategies. **As a preliminary experimental study, Our focus is not on achieving Nash Equilibrium directly with the LLM but on exploring the design of an effective prompting system that can unlock the potential of LLMs in these complex game environments.** Our results highlight that  Suspicion-Agent can adaptively alter its behavior patterns when facing different opponents. This ability to effectively compete against simpler learning-based methods, which utilize reinforcement learning, is a noteworthy contribution to the field. Our research provides a foundation for further exploration into how LLMs can be utilized and improved for strategic decision-making in imperfect information games.