From: NeurIPS.cc/2022/Conference/Program_Chairs
title: Paper Decision
decision: Accept

From: NeurIPS.cc/2022/Conference/Paper4988/Area_Chair_m3GR
metareview: All reviews are strongly positive for this paper. The reviewers appreciate the
many interesting contributions made by this paper. For my own part, I concur
with the reviewers: this is a very nice paper with several interesting results that deepen our understanding of several aspects of EFGs.
recommendation: Accept
confidence: Certain
award: Yes

From: NeurIPS.cc/2022/Conference/Paper4988/Reviewer_ZYdg
title: I agree!
comment: I agree! Thanks for your response.

From: NeurIPS.cc/2022/Conference/Paper4988/Authors
title: Response to Reviewer ZYdg
comment: Thank you for your valuable reviews and positive feedback to our paper! We respond to the questions as follows.

---“Model-free learning within bandit feedback… the decision space of each player (treeplex) needs to be discovered as part of the learning process… for instance [1] and [2]... Do you think that your approach could be used to address that setting as well?”  
Currently, our result for the bandit-feedback case (Algorithm 5, Theorem 7) does require knowing the structure of each player’s decision space (treeplex) beforehand, as the balanced exploration policies (Definition E.1) used in the Algorithm depends on that structure. Therefore our algorithm does not yet address the unknown tree structure setting like [1,2]. We believe that extending to that setting, e.g. by “learning” an exploration policy on-the-fly that is as good as the balanced one for our purpose, would be an interesting open question.

From: NeurIPS.cc/2022/Conference/Paper4988/Authors
title: Response to Reviewer 4HYo
comment: Thank you for your valuable reviews and positive feedback to our paper! 

Re. the typo in Eq. (1): Right, the equation was missing a summation; we have added that in our revision.

From: NeurIPS.cc/2022/Conference/Paper4988/Authors
title: Response to Reviewer mcQf
comment: Thank you for your valuable reviews and positive feedback to our paper!

From: NeurIPS.cc/2022/Conference/Paper4988/Reviewer_mcQf
rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary: This work addresses the task of developing an efficient algorithm for $\Phi$-regret minimization in extensive-form games (EFGs). As a first step toward this goal, the authors consider the problem of regret minimization in the normal-form representation of the EFG that is known to have an exponential dimension with respect to the ATG size.  In such a scenario, they provide a computationally efficient version of the algorithm $\Phi$-Hedge that, under a suitable definition of the set $\Phi$ can be used to approximate equilibria like for instance the EFCE.
In the second part of the paper, the authors show that their version of $\Phi$-Hedge is equivalent to FTRL and OMD with suitably defined dilated entropies and divergences. Based on this equivalence, they recover near-optimal regret bounds for both the full feedback case and the bandit feedback one (in the latter case regret bounds are in high probability).
strengths_and_weaknesses: Strengths: 
The paper provides a solid contribution to the field of regret minimization in games. The writing is clear, overall resulting in a very good quality of presentation. As far as I'm concerned the theoretical claims are sound, novel, and non-trivial. The review of the related works seems accurate. For such reasons, I believe that this is a work that is ready to be published at a venue like NeurIPS.

Weaknesses: 
Since the claims of the paper are those of an efficient algorithm, an experimental evaluation comparing their solution with previous state-of-the-art for computing EFCE (e.g., [1], [2]) could have been proposed. However, I understand that the contribution that this paper intends to propose is mainly theoretical, hence I would classify such weakness as a minor. 

[1] Farina et al. "Simple Uncoupled No-Regret Learning Dynamics for Extensive-Form Correlated Equilibrium". 
[2] Celli et al. "No-Regret Learning Dynamics for Extensive-Form Correlated Equilibrium".
questions: I do not have any question for the authors. 
limitations: This work is theoretical, hence it has no potential negative societal impact.
ethics_flag: No
soundness: 3 good
presentation: 4 excellent
contribution: 3 good

From: NeurIPS.cc/2022/Conference/Paper4988/Reviewer_4HYo
rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary: The paper studies the problem of computing an EFCE in extensive-form games. First, it presents an efficient implementation of $\Phi$-Hedge which minimizes a suitable notion of regret (i.e., trigger regret). Then, the authors show that such algorithm is equivalent to an OMD framework with dilated regularizers (EFCE-OMD). The authors provide regret upper bounds for EFCE-OMD in the full feedback and bandit feedback setting. Such bounds can be further improved via a modified version of the algorithm (Balanced EFCE-OMD) to match the information-theoretic lower bound. 

strengths_and_weaknesses: The paper is well-written and technically solid. The contributions span several interesting directions related to the problem of computing correlated equilibria in extensive-form games. 

A small typo: eq. (1) is probably missing a summation.
questions: None.
limitations: N/A
ethics_flag: No
soundness: 4 excellent
presentation: 4 excellent
contribution: 4 excellent

From: NeurIPS.cc/2022/Conference/Paper4988/Reviewer_ZYdg
rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary: The paper shows gives uncoupled no-regret dynamics for EFCE in extensive-form games with improved sample complexity both under full-information and (adversarial) bandit feedback. The regret bound in the full-information case matches concurrent work by Song et al.; the bandit case yields state-of-the-art results in that it applies even to adversarial opponents.

The proposed algorithm, from a conceptual standpoint, takes the extensive-form game and converts it into a normal-form game. Then, it applies the Phi-hedge algorithm for the choice of trigger deviation functions. The main hurdle is sidestepping the exponential blowup due to the normal-form conversion. In that sense, the paper provides a positive result that complements well the recent result by Farina et al. on kernelizability of OMWU (KOMWU) learning dynamics.

As an additional result, the authors point out that both KOMWU and their algorithm can be implemented in extensive-form games as specific instances of OMD with dilated entropy regularizers.
strengths_and_weaknesses: I think the paper is interesting and well organized. The results are important and fit well within the current efforts on the topic and the existent literature. I also found the remark on KOMWU being equivalent on OMD+dilated entropy regularization very interesting. Overall, I have a very positive opinion of the paper.

Originality: The paper presents original research and is technically strong. None of the results can be classified as groundbreaking, but at least one of the result was unexpected to me (equivalence between KOMWU and OMD+dilated regularizer), and the other results represent new state-of-the-art bounds.

Significance: the results are natural and well-motivated. I am confident that the paper will be well-received within the learning-in-games community.

Quality and clarity: the paper represents high-quality work. Both the body and the supplementary material are curated and easy to follow, at least for an expert in the field.
questions: In the past, certain results have focused on model-free learning within the bandit feedback. Effectively, what that means is that the structure of the decision space of each player (treeplex) needs to be discovered as part of the learning process. See for instance [1] and [2]. Because the structure of the decision problem is not known upfront, it is not immediately clear that one can instantiate a balanced exploration strategy. Do you think that your approach could be used to address that setting as well?


[1] Farina and Sandholm. Model-Free Online Learning in Unknown Sequential Decision Making Problems and Games.

[2] Kozuno, Menard, Munos, and Valko. Model-Free Learning for Two-Player Zero-Sum Partially Observable Markov Games with Perfect Recall.
limitations: No unaddressed concerns.
ethics_flag: No
soundness: 4 excellent
presentation: 4 excellent
contribution: 4 excellent