From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: The paper makes an interesting contribution to the problem of finding equilibria in partial information games through the addition of public information. However, it does not very clearly position itself to the state of the art and it does not compare with alternative techniques experimentally. 
justification_for_why_not_higher_score: Although the paper is mainly theoretical, an experimental comparison with alternative techniques would normally be necessary. One reviewer thinks a major re-write would be necessary, and I also agree. 
justification_for_why_not_lower_score: N/A

From: ICLR.cc/2023/Conference/Paper3028/Authors
title: Comment
comment: We added our thoughts below.

> The example in Section 3 is not a standard partial-information one. Indeed, I would call the example a full-information (the reward and actions are known) simultaneous-move game.

We agree that the examples in Section 3 are equivalent (in an immediate sense) to simultaneous move games. We refer to them as imperfect information because they can be thought of as extensive-form games (as pictured in the figures) in which the second acting player cannot distinguish between histories in which the first acting player selected different actions. We chose these games because they are very simple, making it easy to illustrate our points. However, the points raised in Section 3 also apply for partially observable settings that are not immediately equivalent to simultaneous move games, such as the one shown in Section B.1.

> As far as I can tell, PuB-AMG is just the corresponding setting for non-common payoff games.

We agree with your characterization.

---
Thanks very much for your engagement! We really appreciate it.

From: ICLR.cc/2023/Conference/Paper3028/Area_Chair_SU8S
title: Further on information
comment: Let me add my own notes to this. 
- The example in Section 3 is not a standard partial-information one. Indeed, I would call the example a full-information (the reward and actions are known) simultaneous-move game. The standard solution method would be for player 2 to compute a Nash equilibrium. Pub-MDP is essentially having player 2 know 1's type (policy in this case).   In that sense, it is quite similar to a Bayesian game with a deterministic prior.
- As far as I can tell, PuB-AMG is just the corresponding setting for non-common payoff games.
- Nevertheless, I think that the contributions are valid, even if the relation to other settings is not made perfectly crystal clear. Perhaps the reviewers can comment on this?

From: ICLR.cc/2023/Conference/Paper3028/Authors
title: Maybe?
comment: From our perspective, the fact that the submission gives a meaningful reduction to a perfect information game is central to the submission's value.

While we cannot speak for reviewer D4EG, they did list the submission's usage of "perfect information" as one of four weaknesses in their original review. We feel that the other three listed weaknesses have already been addressed or are easily addressable:
1. We are happy to add a citation to [4] to camera ready.
2. We are happy to add our response regarding the limitations of computing KL regularized equilibria to the camera ready.
3. We have already added the context that the approximate value functions in [5] can be made continuous and also added a proof to back up our own claims of value continuity.

From: ICLR.cc/2023/Conference/Paper3028/Area_Chair_SU8S
title: Importance of perfect information
comment: It is my understanding that whether or not the transformed game can be called 'perfect information' or not is only a minor issue.

From: ICLR.cc/2023/Conference/Paper3028/Authors
title: Checking In
comment: Thanks again for your continued engagement!

We wanted to ping in hopes that the reviewer might respond to our post above. In particular, the reviewer offered to retract their criticism regarding whether or not the PuB-AMG is perfect information if we were able to provide an external publication that states that the PuB-AMG is perfect information. We would love to hear whether the reviewer agrees that we have done so and whether, in light of this fact, the reviewer might consider raising their score.

From: ICLR.cc/2023/Conference/Paper3028/Authors
title: Thanks for your engagement
comment: Thanks for this clarification! We would like to emphasize to the reviewer that none of the submission's academic predecessors have been published in game theory journals -- they were published in AI, ML, and control conferences. Thus, we respectfully disagree that Games and Economic Behavior would necessarily be a better place to find expert referees than ICLR. Indeed, both reviewer igpY and reviewer D4EG appear to be experts on the subject matter of the submission.

From: ICLR.cc/2023/Conference/Paper3028/Reviewer_X9vx
title: concerns remain
comment: Thank you for your reply to my concerns.

One theme of them is that I have not understood the paper.  While one reason for this will be that I do not work in this exact field, I think another is that the paper is moving too quickly through relatively unfamiliar terrain.

I had recommended considering resubmission to a 'mainstream' game theory journal, both to allow greater length to properly explain concepts, and to bring to bear expert referees who can refine the paper, working through its proofs, etc.

By this, I did not mean 'traditional' game theory.  I had in mind something like  _Games and Economic Behavior_ which - as I read it - has evolved to be more 'discipline agnostic' - particularly incorporating the CS community in recent years.

From: ICLR.cc/2023/Conference/Paper3028/Authors
title: Thanks for your response!
comment: Thanks for your engagement -- we really appreciate your response!

> Indeed, it seems there was a misunderstanding. I understood that in PuB-AMGs the players receive the current PBS, and they do not receive the individual private observations (as defined in Section 2.1). What I didn't notice is that in fact, there is no notion of observations in PuB-AMGs, there is only notion of PBSs, which both players receive, hence why I think you used the term "fully-observable". I still think however using the term "fully observable" for a distribution over histories is not a good choice of language and is unnecessarily confusing. But maybe I'm wrong and this is a valid use of such language -- if you can provide a reference for a published work that uses such term similarly I would concede and take this criticism back. However, I am not convinced of calling PuB-AMGs "perfect information", as I understand this term to be all players know the current history with probability 1.

PuB-AMGs are perfect information because they are alternating Markov games and alternating Markov games are definitionally perfect information. 

In PuB-AMGs, all players do know the history; the catch here is that the PuB-AMG and the original game are **different games and have different histories**. The history of the PuB-AMG is the sequence of PBSs and decision rules, which is known to both players.

> if you can provide a reference for a published work that uses such term similarly I would concede and take this criticism back.

**We are happy to provide such a reference.** The following sentence is quoted from Section 4 of the ReBeL paper: "In this section we describe a mechanism for converting any imperfect-information game into a continuous state (and action) space perfect-information game where the state description contains the probabilistic belief distribution of all agents."

> Also I think there is a typo:

Thanks for this catch! We have fixed it in our current draft.

> Are you planning on doing these? Or do you think they are irrelevant? (if yes, why?)

We do not think they are irrelevant, but we think that they would be very difficult to perform in an apples-to-apples manner because of the fact that both setups involve a significant number of non-overlapping design choices. While we are interested in using the contribution of the submission to build a ReBeL-like agent, we believe that, to preserve the clarity of the submission, it would be better to keep such an engineering effort as separate work.

> Can you please add your response comments to text on the limitations?

The period for submission updates is over, but we would be happy to add these to the camera ready version.

> especially since you added new claims about continuity of value functions

Respectfully, we did not add any new claims. The claim that the value function is continuous existed in the original submission, as the reviewer themselves pointed out. While we did add a proof for this claim (in response to the reviewer's comments about a lack of proof), there is no technical innovation -- very similar results already exist in literature.

> I also support suggestion of X9vx for submitting to a dedicated game theory journal.

Respectfully, we do not agree with this suggestion. As can clearly be observed from our bibliography, none of the submission's academic predecessors have been published in game theory journals -- they were published in AI, ML, and control conferences; *our target audience is not game theorists.* Rather, we are targeting researchers interested in papers like ReBeL, Player of Games, and those which apply HSVI to PuB-AMGs (none of which are published in game theory conferences).

Thanks again for your reply and detailed comments!

From: ICLR.cc/2023/Conference/Paper3028/Reviewer_D4EG
title: Response
comment: 
1. The PuB-AMG is perfect information.

> There may be a misunderstanding here. PuB-AMGs are definitionally fully observable. In the PuB-AMG, both players only observe the public belief state; neither player observes any private observations of the original game. Could the reviewer elaborate on what they mean by "they do not receive opponent's observations"?

Indeed, it seems there was a misunderstanding. I understood that in PuB-AMGs the players receive the current PBS, and they do not receive the individual private observations (as defined in Section 2.1).
What I didn't notice is that in fact, there is no notion of observations in PuB-AMGs, there is only notion of PBSs, which both players receive, hence why I think you used the term "fully-observable".
I still think however using the term "fully observable" for a __distribution over histories__ is not a good choice of language and is unnecessarily confusing.
But maybe I'm wrong and this is a valid use of such language -- if you can provide a reference for a published work that uses such term similarly I would concede and take this criticism back.
However, I am not convinced of calling PuB-AMGs "perfect information", as I understand this term to be all players know the current history with probability 1.

2. The PuB-AMG is not a new concept and we do not claim it as such in this paper.

Yes I agree.

3. Resolving the non-correspondence problem may open the door to a new class of HSVI-based algorithms for tabularly solving imperfect information games.

Yes I agree.

---

Thank you for all the additions and clarifications you made! I think they are valuable and will make the paper stronger.

Other replies:

> The idea of using PBSs to construct perfect information games predates [3]'s usage of value-solving subgames.

Indeed, my reference was not the earliest. I think there is even earlier refence for the construction than you mentioned (From the top of my head I think at least CFR-D paper, or perhaps earlier).


> We would also be happy to add a citation to [4]. Would the reviewer kindly point us to the part of the paper discussing the non-correspondence issue? We read [4] but are having trouble relating the setting in [4] to the non-correspondence issue discussed in the submission.

The non-correspondence problem is the same as claiming the online algorithm for the red player (i.e best response to the current PBS) is not strongly globally consistent with an equilibrium.

Also I think there is a typo:

"Because the red player is conditioning on the blue player’s decision rule, it achieves the the optimal value by playing any best response to the **red** player. In the perturbed rock-paper-scissors game, all policies are best responses to the Nash equilibrium. Thus, there is nothing constraining the red player to the Nash equilibrium policy of the original game."

**red** should be **blue** player.

> We have added such a proof to appendix Section D.

Great! I am very interested in reading the appendix.

---

Follow-up on my initial comments:

> The paper could be made stronger by including experimental comparisons with gadget games to show (if) convergence to equilibrium shows faster in practice.

Are you planning on doing these? Or do you think they are irrelevant? (if yes, why?)

> There is a missing text on limitations of the method

Can you please add your response comments to text on the limitations?

--- 

Summary:

Thank you authors for the paper and fruitfull discussion. 
As I claimed, addressing all the criticism would require a major rewrite, and I think it is best to be reviewed again -- especially since you added new claims about continuity of value functions.
I also support suggestion of X9vx for submitting to a dedicated game theory journal.

From: ICLR.cc/2023/Conference/Paper3028/Authors
title: Follow Up
comment: Hello reviewer D4EG, we would be grateful if you can confirm whether our response has addressed your concerns, and let us know if any issues remain. To recap, we hope we were able to convince the reviewer of the points listed above, which we have listed again here for convenience:
1. The PuB-AMG is perfect information.
2. The PuB-AMG is not a new concept and we do not claim it as such in this paper.
3. Resolving the non-correspondence problem may open the door to a new class of HSVI-based algorithms for tabularly solving imperfect information games.

From: ICLR.cc/2023/Conference/Paper3028/Authors
title: Follow Up
comment: Hello reviewer X9vx, we would be grateful if you can confirm whether our response has addressed your concerns, and let us know if any issues remain.

From: ICLR.cc/2023/Conference/Paper3028/Authors
title: Follow Up
comment: Hello reviewer igpY, we would be grateful if you can confirm whether our response has addressed your concerns, and let us know if any issues remain. To recap, we:
- Clarified our contribution's complementary nature to CFR.
- Clarified that our contribution removes imperfect information in a stronger sense than Brown et al. (2020).
- Improved the clarity of the submission as suggested by the reviewer.

From: ICLR.cc/2023/Conference/Paper3028/Authors
title: Author Response (2/2)
comment: > A lot of the practical speedups of CFR-based methods comes from pruning unreachable subtrees. When using regularized equilibria, all actions are in the support, and no subtrees can be pruned.

This is only true when doing non-vector and non-MCCFR based approaches. These approaches are not used in large scale settings.

> In introduction, authors write existing approaches are hampered by discontinous functions (i.e. values for AOH not being convex in terms of their associated beliefs). However, [5] proved that approximative value functions, unlike optimal value functions, can be made continuous.

Thanks for pointing this out. We added a note about the result in [5]. 

>  Additionally, authors do not prove whether their approach yields continous functions or not.

Good point -- **we have added such a proof to appendix Section D.**

> Definition 6.1. \rho in is not defined.

We have added comments clarifying that \rho is a reference policy.

> Definition 5.1. R is a real-valued funtion of what? What is pi(H_i^t) ?

$\mathfrak{R}$ is a real-valued function of the history, the action, and the distribution from which the action is sampled; $\pi(H_{\iota}^t)$ is the policy of the acting player at information state $H_{\iota}^t$ -- i.e., the distribution from which the action is sampled.

> Experiments: Regularized exploitability is undefined.

We added a parenthetical clarifying this.

> Is annealed regularization the same algorithm as in [1], i.e. double-iteration algorithm where one approximates reg. equilibrium, and then iterates over the regularization step size (to speed-up convergence)?

It is a bit different from the approach in [1] in that the scheduling is not double iteration. Instead, the amount of regularization decreases by a small amount at every time step.

> "We observe that all objectives with positive regularization converge to zero regularized exploitability, in accordance with our theory." -- Can you refer the theorem?

Thanks for pointing this out -- we agree this was a bit unclear. We changed the sentence to: "We observe that, in accordance Theorem 1, the approaches with non-zero regularization that converge to zero regularized exploitability in the PuB-AMG also converge to zero regularized exploitability in the original game. In contrast, the unregularized approach does not converge, despite converging in the PuB-AMG."

> Figure 4,5: Why did you pick 1/sqrt(t) and 1/t, and not just one consistenly?

We found that $1/\sqrt{t}$ worked better in extensive-form games and $1/t$ worked better in normal-form games.

> I would be interested in comparison of minimaxEnt/minimaxKL in experiments.

We ran some preliminary experiments on this and did not observe a qualitative difference between the two in perturbed RPS or Kuhn Poker.

> Solving non-correspondence problem using value-solving subgames is a straightforward application of [1], and generalization of UG is an application of brouwer FP theorem.

We respectfully somewhat disagree with these characterizations. Prior to our submission, the idea that the non-correspondence problem could be resolved using uniqueness guaranteeing objectives was unrealized. Furthermore, the generalization requires more than Brouwer's; Brouwer's only shows the existence of an equilibrium, not that the objective is uniqueness guaranteeing.

---

To summarize, we hope that we were able to convince the reviewer of the following points:
1. The PuB-AMG is perfect information.
2. The PuB-AMG is not a new concept and we do not claim it as such in this paper.
3. Resolving the non-correspondence problem may open the door to a new class of HSVI-based algorithms for tabularly solving imperfect information games.

We appreciate the reviewer's expert commentary regarding the relationship between various aspects of our submission and imperfect information game literature and are optimistic for further fruitful discussion.

From: ICLR.cc/2023/Conference/Paper3028/Authors
title: Author Response (1/2)
comment: Thanks very much for your thoughtful review!

> Futher, the diagram in Figure 1 suggests that PuB-AMGs are fully observable. This is false. The players do not receive full observations about what is going on the game -- they do not receive opponent's observations.

There may be a misunderstanding here. PuB-AMGs are definitionally fully observable. In the PuB-AMG, both players only observe the public belief state; neither player observes any private observations of the original game. **Could the reviewer elaborate on what they mean by "they do not receive opponent's observations"?**

> A similar "perfect-information" game transformation would be to let a player pick a strategy for the whole game, announce it publicly, and let opponent do the same, and evaluate the expected utility of the game. Pub-AMGs trick is the strategies are additionally factored over public states.

We agree! Though, that said, this "trick" is important.

> One could get a superficial impression that with the transformation, you can take any off-the-shelf algorithm for perfect-information games, such as AlphaZero, to find an equilibrium in imperfect-information games, which is not true.

We agree that AlphaZero would be a bad choice, as was also pointed out by the ReBeL paper. However, there is a community of researchers that has been working on using HSVI, which is a strong algorithm for solving small Dec-POMDPs via the PuB-AMG, to solve PuB-AMGs (e.g., Horak & Bosansky, 2019; Buffet et al., 2020; Delage et al., 2021). Combining their work with our resolution of the non-correspondence problem could yield a new approach to solving imperfect information two-player zero-sum games using a "perfect information" algorithm (i.e., HSVI).

**We added a discussion section to make the likely use cases of our contribution more clear.**

> I view it as "rebranding" of value-solving games as a new concept, i.e. the PuB-AMGs.

We are a bit confused regarding the reviewers comments on value-solving subgames. The reviewer cites [3] for value-solving subgames. This term appears in a single footnote in [3] and was first used in the Sept 2020 version. By that point, the July 2020 version ReBeL (which [3] cites for introducing the term public belief state) had already discussed PuB-AMGs as perfect information games (although it did not explicitly name them). To summarize:
- The idea of using PBSs to construct perfect information games predates [3]'s usage of value-solving subgames.
- The term value-solving subgames was merely mentioned as a footnote.

Additionally, very similar constructions to the PuB-AMG have already been formalized in existing works (Buffet et al., 2020; Delage et al., 2021). Thus, **we do not feel that PuB-AMGs are conceptually new. We have added some text to the PuB-AMG section to emphasize this point.**

> The non-correspondence problem has been studied before and is not cited, i.e. [4].

Thanks for this pointer! Indeed, we made no claim of having discovered the non-correspondence problem. We believe that it was understood among imperfect information games researchers going back at least throughout the 2010s. **We have added a references Ganzfried & Sandholm (2015), Brown (2020) to make clear that we are not claiming to have discovered the issue.** We would also be happy to add a citation to [4]. Would the reviewer kindly point us to the part of the paper discussing the non-correspondence issue? We read [4] but are having trouble relating the setting in [4] to the non-correspondence issue discussed in the submission.

> Computing KL-regularized equilibrium for arbitrary small $\epsilon > 0$ is slow.

In large games, we think that it would not be necessary to use arbitrarily small $\epsilon$. Indeed, in single-agent settings, it is common for deep RL algorithms to use constant positive regularization to improve performance on the unregularized objective.

> Further, computing regularized equilibrium in sequential games needs recursive computation of regularized best responses, yielding asymptotic complexity of $O(T^D)$, where $D$ is depth of the game and $T$ is number of regret minimizer iterations. The algorithm has been applied to Kuhn Poker (with some tricks to make if faster, as explained in appendix), and is impractical for even slighly bigger game, like Leduc Poker.

Indeed, the naive approach in our experiments is not particularly good way of going about solving the PuB-AMG. However, as discussed above, HSVI may be able to handle such settings much more efficiently. We would also note here that the "tabular" version of ReBeL requires exactly the same recursive best response calls.

From: ICLR.cc/2023/Conference/Paper3028/Authors
title: Author Response
comment: Thanks very much for your thoughtful review!

### Comments on Summary

We have some clarifications regarding the summary.

> In games in which players' incentives are aligned, it may be enough for one player to announce its strategy (Nayyer et al.).

This is not quite right -- it is enough for *all players* to announce their strategies.

> The current manuscript argues that imperfect information can also be abstracted away from when one player has a unique strategy, allowing the other to determine which node has been reached.

This is not quite right -- we argue that imperfect information can be abstracted away when the objective guarantees a unique equilibrium in all subgames.

### Comments on Main Review

> to me, basic elements of the paper were unclear, including what type of equilibrium we are interested in (Nash, subgame perfect, Perfect Bayesian...)

We are interested in equilibria satisfying the minimax condition in Definition 5.1. These equilibria are Nash equilibrium of a regularized game; in the original game, they are not necessarily Nash equilibria.

> lists of specialist notation are provided at a number of points without sufficient explanation for a non-specialist reader to understand. I would like to see the reasoning related more tightly to more 'mainstream' game theory. (How, for example, is this related to work developing from Mailath and Samuelson's imperfect observability?)

We feel that the submission is of greatest interest to two communities of researchers: 1) Researchers interested subgame decomposition in two-player zero-sum games; 2) Researchers interested transferring the "common-information approach to decentralized stochastic control / Dec-POMDPs" to two-player zero-sum games (e.g., Wiggers et al., 2016; Horak & Bosansky, 2019; Buffet et al., 2020; Delage et al., 2021). We feel that neither literature regarding (1) nor the literature regarding (2) are particularly tightly tied to 'maintstream' game theory; thus, we are unsure that a tighter connection to mainstream game theory would be the right way to communicate the contribution.

> it is hard for me to assess this given my concerns about clarity, above. I think that the paper's point is simple: uniqueness removes uncertainty.

This is not quite right -- we would characterize it as: uniqueness removes non-correspondence.

> the paper itself suggests (e.g. around Corollary 1) that some of its main results can be found elsewhere

We would not characterize it this way. Corollary 1 says that our results can be combined with other results to yield more general applicability; all of our main results are included in the paper.

> If the manuscript is not accepted here, I think it could be a fruitful exercise to rewrite it for submission to a dedicated game theory journal. 

As articulated above, our target audience is not dedicated game theorists.

From: ICLR.cc/2023/Conference/Paper3028/Authors
title: Author Response (2/2)
comment: > how does your proposed procedure computationally compare with CFR?

We feel the main use case of our reduction is for an algorithm like ReBeL, which uses CFR as a sub-procedure. *Similarly, one could use CFR as a subprocedure to solve regularized PuB-AMGs with function approximation. Thus, we feel the efficacy of CFR is complementary to our contributions, rather than a point of comparison.*

> Symbol definitions

We use capital letters to denote random variables and lowercase letters to denote particular instantiations. We have added a sentence about this to the notation section.

> Typos: "is has" 

Fixed.

> Especially Sec. 4 of Brown et al. (2020) already introduces the PBS as a way to remove private information in a two-player zero-sum game. 

We would note here that, unlike our work, Brown et al. (2020)'s result does not yield an equilibrium preserving reduction to perfect information games. Indeed, care must be taken at test time using randomization, no-regret learning, and avoiding re-planning to make sure that the resulting policy approximates an equilibrium. In contrast, our result could facilitate the computation of a closed-form regularized equilibrium.

From: ICLR.cc/2023/Conference/Paper3028/Authors
title: Author Response (1/2)
comment: Thanks very much for your thoughtful review!

> the paper provides an overview of the main sources of complexity when computing a Nash Equilibrium in two-player zero-sum game with imperfect information

We would just make clear here that these are only problems for backward induction dynamic programming approaches (e.g., Brown et al., 2020, Buffet et al., 2020; Delage et al., 2021, Schmid et al. 2021). Of course, there are other ways of computing Nash equilibria in two-player zero-sum games.

> More importantly, I think that the lack of an algorithmic perspective on the results makes very unclear any evaluation of impact of the proposed framework. I would have liked to see a justified algorithm-like procedure to wrap-up the contribution of the paper, and a transparent evaluation of the time/complexity needed to solve game instances , maybe with a comparison to the CFR algorithm that is currently state-of-the-art.

We intentionally chose not to commit to an algorithmic perspective for a few of reasons:
- First, we see the result in the submission as somewhat analogous to that of Nayyar (2013)'s work. Their work, like ours, shows the reduction from common-payoff problems to single-agent problems but does not commit to a particular approach to solving the resulting problem. Since then, a wide variety of approaches have found use cases on top of Nayyar's reduction.
- Second, we feel that the concepts required in the submission are already difficult to explain clearly in the allotted space. Introducing a ReBeL-like deep RL methodology on top of the existing concepts would add significant additional complication and require sacrificing some of the existing clarity to make sufficient space.

Also, we note that CFR is complementary to our contribution -- not a baseline. Indeed, CFR could be used as a subprocedure of an expert iteration algorithm for approximately solving PuB-AMGs (in a similar fashion as was done for ReBeL).

> In particular, the Pub-MDP definition is very formal and it is complex to get the overall picture due to the amount of symbols used (especially for the definitions of $\mathcal{\tilde{T}}$ and $\tilde{s}$)

We agree -- this was also bothering us. We have uploaded a revised version in which we feel the definitions are a bit easier to comprehend.

> The other point is Theorem 3. I'd suggest a preliminary explanation of the $q$ distribution (I expect it to be q-values) and the policy greedification function $g$ and a expansion of the sketch of proof by including more about the relation between interior image and uniqueness.

Indeed, it is $q$ values. 

Regarding the relationship between interior image and uniqueness: Assuming an interior image is convenient for guaranteeing uniqueness because, if the image is not interior, it means that there are actions played with probability zero. If actions are played with probability zero, it means that there are subtrees of the game that are never reached. Within such subtrees, the policy can be changed arbitrarily without affecting the expected return, leading to non-uniqueness. It is possible that our results could be extended to a weakner notion, such as "uniqueness up to reachability" -- meaning that the equilibrium policies are unique for any action observation history reached with positive probability. We chose not to pursue this direction in the submission, as we found it likely to make the analysis and discussion messier.

We have added a sentence regarding the $q$ and $g$ into the theorem statement. Expanding the proof sketch is more difficult because we end up over the page limit, so we have left it as is for now.

> regarding the proof of existence of equilibrium in Theorem 3: why isn't it already guaranteed by the Nash theorem?

It isn't already guaranteed by the standard result because it is not clear that it is possible to write a strategically equivalent normal-form game.

> regarding the Experiments in Section 7: why was a MMD vs Regularized-BR setting used?

We will discuss the question as if it is about the normal form game experiments for simplicity. In these experiments, the second moving player does use a regularized best response. However, the first moving player cannot easily compute a closed form regularized best response to the second moving player because the second moving player's policy is a function of the first moving player's policy. Thus, we instead have the first moving player perform a kind of iterative ascent on the surface induced by the second moving player's best response function. Other choices (such as CFR) would also work as a replacement to MMD here.

From: ICLR.cc/2023/Conference/Paper3028/Reviewer_igpY
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: The main contributions of the paper are twofold:

* on one hand, the paper provides an overview of the main sources of complexity when computing a Nash Equilibrium in two-player zero-sum game with imperfect information. Those are named *backward dependence problem* and *non-correspondence problem*.
* on the other hand, the paper propose a novel resolution scheme that addresses these two problems by combining the Pub-MDP construction developed by Nayyar et al. (2013) with the minmaxKL framework developed by Perolat et al (2021). These two techniques respectively address the backward dependence problem by transforming the game to remove the private information and the non-correspondence problem by guaranteeing a unique Regularized Nash Equilibrium in any subgame.

This is a theoretical paper that introduces and formally defines each concept and gradually constructs the final proposed solving procedure. Small experiments are provided to add empirical evidence of convergence of the proposed method.
strength_and_weaknesses: The strengths of the paper are a well-explained setting, in which the crucial problems are clearly highlighted, a strong formalism adopted, a well explained relation with previous works, and a novel result which connects in an actionable framework imperfect information and perfect information settings.

On the other hand, I found crucial steps of the paper to be difficult to quickly comprehend due to the heavy formalism adopted and a lack of insightful comments (more details on this in the next Clarity section). More importantly, I think that the lack of an algorithmic perspective on the results makes very unclear any evaluation of impact of the proposed framework. I would have liked to see a justified algorithm-like procedure to wrap-up the contribution of the paper, and a transparent evaluation of the time/complexity needed to solve game instances , maybe with a comparison to the CFR algorithm that is currently state-of-the-art.
clarity,_quality,_novelty_and_reproducibility: **Clarity:** the setting of the paper and its main problems are well described and clear to follow. The formalism used is precise and clearly defined. However, two main points of the paper need extra accompanying explanations to improve their clarity. In particular, the Pub-MDP definition is very formal and it is complex to get the overall picture due to the amount of symbols used (especially for the definitions of $\tilde{\mathcal T}$ and $\tilde{s}$). I'd suggest some `\underbrace{}` or an accompanying comment as done in the definition of $\tilde{\mathbb{A}}$ for the Pub-AMG. The other point is Theorem 3. I'd suggest a preliminary explanation of the $q$ distribution (I expect it to be q-values) and the *policy greedification function* $g$ and a expansion of the sketch of proof by including more about the relation between interior image and uniqueness.

I have some questions: 

* regarding the proof of existence of equilibrium in Theorem 3: why isn't it already guaranteed by the Nash theorem?
* regarding the Experiments in Section 7: why was a MMD vs Regularized-BR setting used?
* how does your proposed procedure computationally compare with CFR?

Moreover, some symbols used are unclear and/or undefined:

* $h^{t+1}$ instead of $H^{t+1}$ in the definition of $\tilde{s}$ of the Pub-MDP
* $\pi(H^t_\iota)$ in Definition 5.1
* $q$ in Theorem 3

Typos: "is has" in the sketch of proof of Theorem 3

**Quality**: the paper addresses a complex topic in a rigorous way; even though I did not check any proog in the Appendix apart from the one of Theorem 3, the provided sketches of proofs are sound and can be easily followed by an expert of the related literature

**Novelty**: The Pub-MDP results are well known in the literature thanks to many important papers. Especially Section 4 of Brown et al. (2020) already introduces the PBS as a way to remove private information in a two-player zero-sum game. The real novel result of the paper therefore resides in the use of the minimaxKL framework by Perolat et al. (2021) and it uniqueness of equilibrium guarantee to develop a game-solving procedure.

**Reproducibility**: the theoretical results are well documented and can be easily reproduced. The lack of both a formal procedure and/or open source code may make more difficult the experimental reproducibility of the paper, even though it is not a major issue.
summary_of_the_review: Overall, I find the result offered by the paper and the justifications provided really interesting for the game-solving community. On the other hand, the weaknesses highlighted previously are important issues that should be properly addressed to allows a smoother and comprehensive exposition.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: Not applicable
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3028/Reviewer_X9vx
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: In games of imperfect information, players' payoffs are known, but actions may not be.  An example would be a two player simultaneous move game, whose game tree depicts one player as moving second, but unable to distinguish between the nodes reachable from each of the other player's actions.

As imperfect information complicates reasoning, some method for handling it must be developed when deriving equilibria.

In games in which players' incentives are aligned, it may be enough for one player to announce its strategy (Nayyer et al.).

The current manuscript argues that imperfect information can also be abstracted away from when one player has a unique strategy, allowing the other to determine which node has been reached.
strength_and_weaknesses: **Strengths**

**Weaknesses**

See below.  On the latter, I am primarily concerned that the manuscirpt's density makes it difficult to read and understand.
clarity,_quality,_novelty_and_reproducibility: **Clarity**
- to me, basic elements of the paper were unclear, including what type of equilibrium we are interested in (Nash, subgame perfect, Perfect Bayesian...)
- lists of specialist notation are provided at a number of points without sufficient explanation for a non-specialist reader to understand
- I would like to see the reasoning related more tightly to more 'mainstream' game theory.  (How, for example, is this related to work developing from Mailath and Samuelson's imperfect observability?)

**Quality**
- it is hard for me to assess this given my concerns about clarity, above.  I _think_ that the paper's point is simple: uniqueness removes uncertainty.

**Novelty**
- I have not seen arguments quite like this before
- the paper itself suggests (e.g. around Corollary 1) that some of its main results can be found elsewhere

**Reproducibility**
- not attempted
summary_of_the_review: Overall, I felt that the paper was trying to pack too much specialist material in, without motivating it enough for a more general reader to understand.  This also meant, unfortunately, that the paper was only able to motivate its connection to machine learning in passing.

If the manuscript is not accepted here, I think it could be a fruitful exercise to rewrite it for submission to a dedicated game theory journal.  This would allow it more 'space to breathe', without the tight page limit, and require it to connect more tightly to the existing literature, strengthening its foundations.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3028/Reviewer_D4EG
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: In two-player zero-sum games, when we modify utility objective of minimax optimization with an appropriate regularization term, it is always possible to find a unique solution that has error at most $\epsilon > 0$ in the original optimization problem. This is an important previous result from [1]. Let's call any such objective modification "uniqueness-guaranteeing" (UG).

This paper shows that solution of UG optimization tackles the "non-correspondence problem" between the subgame strategies and the trunk strategies (part of the game played above the subgame), simply from the virtue of UG solution being unique (so there is no danger of using two incompatible equilibria). 
In contrast, in the existing literature, the correspondence has been solved for arbitrary equilibria using gadget games [2].
This work shows that one can use value-solving subgames [3] when using UG, without the need to use the gadget game.

Value-solving subgames are rooted in public belief states, where players publicly announce what strategies they played so far.
Authors thus identify that optimizing minimax UG objective in imperfect-information game is equivalent to optimizing minimax UG objective in what authors call "perfect-information" PuB-AMGs.

[1] From poincaré recurrence to convergence in imperfect information games: Finding equilibrium via regularization.

[2] Solving imperfect information games using decomposition.

[3] Rethinking formal models of partially observable multiagent decision making.

[4] Sound algorithms in imperfect information games

[5] Player of games.

strength_and_weaknesses: Strengths:
+ Provably correct solution of the non-correspondence problem without using gadget games. This is important, because empirically strategies in gadget games converge very slowly, often at rate close to their upper bound, and value-solving subgames are empirically fast. 
+ Authors generalize result from [1], i.e. what regularizations are UG.

Weaknesses:
- I find introduction of PuB-AMGs and calling them "perfect-information" games misleading at best, and disingenuous at worst. 
  A similar "perfect-information" game transformation would be to let a player pick a strategy for the whole game, announce it publicly, and let opponent do the same, and evaluate the expected utility of the game. Pub-AMGs trick is the strategies are additionally factored over public states.
  One could get a superficial impression that with the transformation, you can take any off-the-shelf algorithm for perfect-information games, such as AlphaZero, to find an equilibrium in imperfect-information games, which is not true. In this sense the reduction is not particularly useful. I view it as "rebranding" of value-solving games as a new concept, i.e. the PuB-AMGs.
  Futher, the diagram in Figure 1 suggests that PuB-AMGs are fully observable. This is false. The players do not receive full observations about what is going on the game -- they do not receive opponent's observations.
- The non-correspondence problem has been studied before and is not cited, i.e. [4].
- There is a missing text on limitations of the method:
  - Computing KL-regularized equilibrium for arbitrary small $\epsilon > 0$ is slow. Further, computing regularized equilibrium in sequential games needs recursive computation of regularized best responses, yielding asymptotic complexity of $O(T^D)$, where D is depth of the game and T is number of regret minimizer iterations. The algorithm has been applied to Kuhn Poker (with some tricks to make if faster, as explained in appendix), and is impractical for even slighly bigger game, like Leduc Poker.
  - A lot of the practical speedups of CFR-based methods comes from pruning unreachable subtrees. When using regularized equilibria, all actions are in the support, and no subtrees can be pruned.
- In introduction, authors write existing approaches are hampered by discontinous functions (i.e. values for AOH not being convex in terms of their associated beliefs). However, [5] proved that approximative value functions, unlike optimal value functions, can be made continuous. Additionally, authors do not prove whether their approach yields continous functions or not.

Other comments:
- Definition 6.1. \rho in  is not defined.
- Definition 5.1. R is a real-valued funtion of what? What is pi(H_i^t) ?
- Experiments: Regularized exploitability is undefined.
- Is annealed regularization the same algorithm as in [1], i.e. double-iteration algorithm where one approximates reg. equilibrium, and then iterates over the regularization step size (to speed-up convergence)?
- "We observe that all objectives with positive regularization converge to zero regularized exploitability, in accordance with our theory." -- Can you refer the theorem?
- Figure 4,5: Why did you pick 1/sqrt(t) and 1/t, and not just one consistenly?
- I would be interested in comparison of minimaxEnt/minimaxKL in experiments.

clarity,_quality,_novelty_and_reproducibility: The paper was well written, however I think it would be made clearer if it was written using the existing concept of value-solving subgames, instead of introducing new game formalism.
Some symbols and terms were not defined. Sometimes I could guess their meaning from related literature, but a clarifying revision is required.
Solving non-correspondence problem using value-solving subgames is a straightforward application of [1], and generalization of UG is an application of brouwer FP theorem.


summary_of_the_review: Overall I like the direction of the paper. It has two substantial contributions, solving non-correspondence problem using value-solving subgames, and generalization of UG.
The paper could be made stronger by including experimental comparisons with gadget games to show (if) convergence to equilibrium shows faster in practice.
However, addressing all the criticism would require a major rewrite. I recommend authors to resubmit their revised work.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough