Paper: Optimal Action Abstraction for Imperfect Information Extensive-Form Games (ICLR-2024)

Reviewer comment (Novelty):
The paper focuses on a benchmark—namely heads-up no-limit Poker—that has been essentially already solved, as prior work has already developed techniques to find superhuman strategies in that game. It would be more meaningful if the authors used their new algorithm to make progress on a benchmark that has been elusive using prior techniques.

Author response:
As we mentioned in Footnote 2, HUNL is an important version of poker for investigating mixed strategy two-player zero-sum IIEFGs due to its complex nature and extremely large decision space. Although HUNL's AI has achieved superhuman performance, the problem of HUNL itself has still not been fully solved. We believe that all current HUNL AIs are still not achieving a Nash equilibrium strategy due to the extremely large decision space (more than $10^{160}$). Thus, we propose a general approach that can be used on IIEFGs and have made considerable progress on HUNL benchmark. On the other hand, we also agree with the reviewer that extending our method to more benchmarks is interesting and plan to study that in our future research.

Reviewer comment (Experiments):
The comparison is not made with state of the art models (Libratus and Deepstack). Even if these models are not publicly available, it is a significant weakness if the new approach does not attain state of the art performance.

Author response:
We would like to first highlight that  the core contribution of our work is to propose a new method for general IIEFG to select good action abstractions with RL. Our novel approach has the potential to apply to problems that fall into the IIEFG setting. Here is for the comparison between our method and Libratus. BabyTartanian8 [1] is the champion of 2016 annual computer poker competition. Here are some head-up results against BabyTartanian8, ReBeL wins 9 mbb/hand, Slumbot loses 12 mbb/hand, Libratus wins 63 mbb/hand. Relative to BabyTartanian8, Libratus achieved about 54 mbb/hand win-rate more than ReBeL. For comparison, RL-CFR achieved a win rate of 64 mbb/hand in heads-up evaluation against the ReBeL's replication. This suggests that RL-CFR achieves a lift comparable to how Libratus compared to ReBeL.

Reviewer comment (Experiments):
It is unclear how Libratus or Deepstack performs against either Rebel or Slumbot. Knowing this would provide an indication of how the new approach performs against Libratus or Deepstack.

Author response:
We would like to first highlight that  the core contribution of our work is to propose a new method for general IIEFG to select good action abstractions with RL. Our novel approach has the potential to apply to problems that fall into the IIEFG setting. Here is for the comparison between our method and Libratus. BabyTartanian8 [1] is the champion of 2016 annual computer poker competition. Here are some head-up results against BabyTartanian8, ReBeL wins 9 mbb/hand, Slumbot loses 12 mbb/hand, Libratus wins 63 mbb/hand. Relative to BabyTartanian8, Libratus achieved about 54 mbb/hand win-rate more than ReBeL. For comparison, RL-CFR achieved a win rate of 64 mbb/hand in heads-up evaluation against the ReBeL's replication. This suggests that RL-CFR achieves a lift comparable to how Libratus compared to ReBeL.

Reviewer comment (Experiments):
Rebel was not tailored to Poker; its goal was to perform well across many different benchmarks. Therefore, a comparison with an agent designed specifically for Poker might not be appropriate.

Author response:
We would like to highlight that RL-CFR, like ReBeL, is not tailor-made for poker. Instead, it can be applied to choosing  appropriate action abstractions for general IIEFGs. As you mentioned, Slumbot is not a particularly strong agent compared to either Libratus or Deepstack. However, Slumbot is the only HUNL AI we know that offers online competition testing. Our comparison with Slumbot is mainly used to verify the reliability of our method. As shown in the experiments, RL-CFR achieves considerable win-rates against default action abstraction method ReBeL in both the heads-up evaluation and the indirect evaluation with the Slumbot.

Reviewer comment (Experiments):
Slumbot is not a particularly strong agent compared to either Libratus or Deepstack.

Author response:
We would like to highlight that RL-CFR, like ReBeL, is not tailor-made for poker. Instead, it can be applied to choosing  appropriate action abstractions for general IIEFGs. As you mentioned, Slumbot is not a particularly strong agent compared to either Libratus or Deepstack. However, Slumbot is the only HUNL AI we know that offers online competition testing. Our comparison with Slumbot is mainly used to verify the reliability of our method. As shown in the experiments, RL-CFR achieves considerable win-rates against default action abstraction method ReBeL in both the heads-up evaluation and the indirect evaluation with the Slumbot.

Reviewer comment (Writing):
The way citations are used is not syntactically sound. For example, "(Meng et al.) proposes" should instead be "Meng et al. (2023) propose".

Author response:
Thank you very much for your help. We have fixed these typos.

Reviewer comment (Writing):
On page 8, "our conduct" is a typo.

Author response:
Thank you very much for your help. We have fixed these typos.

Reviewer comment (Presentation):
There is unnecessary space before Footnote 6.

Author response:
Thank you very much for your help. We have fixed these typos.

Reviewer comment (Novelty):
Failure to adequately summarize prior work in action abstraction. Some of the most important concepts (e.g., imperfect recall, and its effect on theoretical guarantees) are not discussed at all.

Author response:
* Q1. Failure to adequately summarize prior work in action abstraction. Some of the most important concepts (e.g. imperfect recall, and its effect on theoretical guarantees, is not discussed at all).

* A1. Thank you for your comments, we have included these prior works to the background in revision.

Reviewer comment (Theory):
The reader should be informed that abstraction can affect solution quality in surprising ways. While the authors mention that using a smaller $\mathcal{AA}_{base}$ as in Moravcík et al., 2017, leads to a decrease in win rate, it is also known that using a larger abstraction does not necessarily lead to improved solution quality. Waugh et al. (2009) discuss these abstraction pathologies in-depth.

Author response:
* First of all, we note that our claim does not controdict with that in Waugh et al. (2009). Instead, the strategy given by RL-CFR confirms the view of Waugh et al. "larger abstractions do not necessarily lead to solutions that are closer to optimal in the real game". This is shown in the examples in Appendix G, where we can obtain a better strategy by choosing a smaller action abstraction.

Reviewer comment (Theory):
The title of the paper including “Optimal” makes it seem like the paper will include theoretical guarantees on optimality of the learned abstraction, but it does not.

Author response:
* Second, as reviewers mentioned, we does not address any theory or optimality in the traditional sense. The main contribution of our work is to design a novel MDP formulation for IIEFGs, and propose an algorithmic framework based on RL and CFR to achieve good performance.

Reviewer comment (Experiments):
The claim “Compared to other methods for choosing action abstractions (Hawkin et al., 2011; 2012; Zarick et al., 2020), RL-CFR has a wider range of applicability and faster convergence” is not supported by evidence. The tested application in experiments is only poker, and no other abstraction algorithm is used.

Author response:
* Q2.  "Compared to other methods for choosing action abstractions (Hawkin et al., 2011; 2012; Zarick et al., 2020), RL-CFR has a wider range of applicability and faster convergence." I don't see evidence supporting either of these claims. The tested application in experiments is only poker, and no other abstraction algorithm is used.

* A2. Faster convergence here means that compared to these previous methods of choosing dynamic action abstractions, RL-CFR converges faster  because it selects an action abstraction that is fixed during CFR solving. In page 102 of [1], it has been demonstrated "the algorithm converges more slowly than running CFR with fixed bet sizes". By the way, we have compared RL-CFR with some other abstraction algorithms listed in Table 2. A wider range of applicability means that RL-CFR constructs an MDP that solve a kind of IIEFGs, including but not limited to poker.

Reviewer comment (Experiments):
The main question is regarding the cost of running RL-CFR. Is it possibly more efficient to run ReBel with a larger abstracted action set?

Author response:
* Q5. The main question I have is regarding the cost of running RL-CFR. Is it possibly more efficient to run ReBel with a larger abstracted action set?

* A5. As we mentioned in previous questions, the size of RL-CFR's action abstraction is no more than the size of default action abstraction in our experiments. A larger action abstraction would theoretically have a better strategy, but this would result in a larger game tree and more solving time, and the default action abstraction size we chose is relatively reasonable.

Reviewer comment (Theory):
Public belief states and their importance could be explained better. The reviewer is unsure what is meant by “Public belief state (PBS) is an assumption(?) that treats players’ strategies as common knowledge for reducing the state of large IIEFGs significantly.” It is unclear how a PBS reduces the size of the state space, and as far as the reviewer understands, all variants of CFR treat strategies as common knowledge.

Author response:
* Q6. Public belief states and their importance could be explained better. I'm not sure what is meant by the following: "Public belief state (PBS) is an assumption(?) that treats players’ strategies as common knowledge for reducing the state of large IIEFGs significantly". I don't understand how a PBS reduces the size of the state space, and, as far as I understand, all variants of CFR treat strategies as common knowledge. 

* A6. In IIEFG, each state needs to include both the private information of the two players and the public information of the game. For the private information, because there are many different possibilities, a single enumeration can lead to the number of states becoming very large. In PBS, each state contains a group of states from the original IIEFG that are indistinguishable for at least one player.

Reviewer comment (Writing):
Phrasing elsewhere seems to imply that we have as a presupposed goal the desire to implement Deep RL somehow for IIEFGs. But why do we have this goal in the first place? For example, in Section 5 it states: "It is important to note that applying the DRL approach to IIEFGs is highly nontrivial. The key challenge comes from the fact that one has to decide the action probability distributions for all information sets..." — the point of this sentence is unclear, and the "key challenge" is not clearly explained.

Author response:
* A5. In the previous problem we mentioned several properties of IIEFGs that make RL a very suitable solution. Thus, we propose a RL framework for solving IIEFGs. However, IIEFG is very different from those common games. While common games only need to choose an optimal action for the current state, IIEFG needs to compute a mixed strategy for all feasible actions for all information sets. It is difficult to get such a mixed strategy directly by RL, so we propose the method of selecting the action abstraction by RL and computing the strategy by CFR. We have modified this sentence in the revision.

Reviewer comment (Theory):
Section 4 (State) should clarify that the PBS to PS reduction is lossy. While defining state as the PBS satisfies the Markov assumption, it is less clear if PS as state does. If convergence or optimality were touched upon in this paper (and they should be), then this choice of PS rather than PBS may introduce a problem.

Author response:
* A6. The compression process from PBS to PS results in a loss of information. However, we believe that such compression is necessary and has no impact on RL performance. Since the PS contains the previous actions of both players, if both players perform an equilibrium strategy, we can infer the PBS based on these actions. PBS is necessary for CFR, but not for the action network used to select an action abstraction. Because when using PS, we are only choosing the action abstraction without CFR solving. Thus,  we believe  it is reasonable to use PS as the state in the MDP. Later in the rebuttal we will also mention the problems that can result from choosing PBS as the state in the MDP.

Reviewer comment (Experiments):
The paper would benefit greatly from qualitative results showing examples of action abstractions chosen by the RL agent.

Author response:
* A7. We will revised it in appendix in final revision.

Reviewer comment (Theory):
Section 3 (Public Belief State): "In general, a PBS is described by the joint probability distribution of the possible infostates of the players", but "PBS β" seems to be defined as the marginal probability distributions for each player. In poker, you can go from the marginal distributions to the joint distribution, but this is not always the case. If so, this should be clarified.

Author response:
* A8. Thank you very much for your help. We have fixed this sentence in revision.

Reviewer comment (Reproducibility):
Section 4 (MDP definition): The state, actions, and rewards are defined for this MDP, but the state transitions are not defined anywhere. While one could infer from Algorithm 1, it would be much clearer if these were defined explicitly.

Author response:
* A9. We have made this clear in the last paragraph of Section 4 of the revision.

Reviewer comment (Writing):
Introduction typo: "may depends on" should be "may depend on".

Author response:
* A10. Thank you very much for your help. We have fixed these typos.