Paper: Toward Optimal Policy Population Growth in Two-Player Zero-Sum Games (ICLR-2024)

Reviewer comment (Theory):
The premise of the paper, PSRO/DO can result in increases in exploitability between iterations, is not sufficiently justified or explained. Wouldn't a more straightforward approach to just be take the best-found-so-far solution across PSRO iterations? With this slight modification on the return statement, you obtain the desired anytime property.

Author response:
**Reviewer's Comment:**  
The reviewer suggests outputting the best-found-so-far solution across PSRO iterations to obtain the desired anytime property.

**Response:**  
This is a reasonable suggestion. We had thought about this before submitting the paper but had decided not to include it because it does not perform as well. We have included this in Appendix F.4 in the final version of the paper. It performs better than PSRO but slightly worse than APSRO. A similar result is also evident in Figure 4.a where PSRO performance is near monotonic. Furthermore, we want to point out that this best-so-far technique could also be applied to APSRO and SP-PSRO.

Reviewer comment (Theory):
Another premise of the paper is that "PSRO and APSRO add pure-strategy (i.e. deterministic) best responses in each iteration." PSRO does not have this requirement, and in fact, begins by adding a uniformly stochastic policy to the strategy set. As it is relatively common for implementations of PSRO to include only deterministic policies, I am fine with this being used as a premise, but I think this needs to be directly discussed in the text. This fact also has the side-effect of suggesting an alternative simpler approach to APSRO/SP-PSRO in differing methods of computing stochastic policies.

Author response:
**Reviewer's Comment:**  
The paper originally did not clarify that common implementations of PSRO add pure strategies in each iteration.

**Response:**  
We've revised the text (highlighted in blue) to specify that *common* implementations of PSRO add pure strategies every iteration.

Reviewer comment (Novelty):
ADO is marginally novel compared to Range of Skill (RoS) and DO, and I think its limitations are not discussed well enough. To me, the main intuition behind DO/PSRO is when you cannot consider analytical solutions to game reasoning, and instead need to consider a restricted/empirical version of the game. ADO considers a partially restricted version of the game (only expanding a single player's policy-space), and as a result is intractable and bears common complaints to direct game reasoning. The authors should discuss this and the complexity trade-offs more directly.

Author response:
**Reviewer's Comment:**  
Concerns were raised about the novelty of ADO and its comparison with Range of Skill (RoS) and DO.

**Response:**  
We are upfront in our paper about the similarities of ADO with RoS and DO. We do not view ADO as our primary contribution but rather as a theoretical foundation on which we build the other scalable algorithms. We added to the text: “ADO primarily serves as a foundational element for the development of our subsequent algorithm, APSRO. This foundational role is crucial as it lays the groundwork for APSRO's convergence guarantees.”

Reviewer comment (Theory):
The authors should address the focus on NE, when PSRO is flexible to any chosen solution concept. They should also state for all of their algorithms if they are constraining this aspect of PSRO.

Author response:
**Reviewer's Comment:**  
The paper primarily focuses on NE, while PSRO is adaptable to various solution concepts.

**Response:**  
We've added a line clarifying that while our current focus is on two-player zero-sum games and NE. These techniques (at least APSRO) directly carry over to the TMECor solution concept for team games [1].

We are not sure whether that is the type of refinement that the reviewer is asking about. Or perhaps the reviewer is suggesting some normal-form refinement such as CCE. Or, perhaps the reviewers is suggesting some extensive-form refinement such as trembling hand. If the reviewer clarifies what sorts of refinements the reviewer is alluding to, we can try to provide further answers.

Reviewer comment (Experiments):
Very limited analysis of the design of APSRO/SP-PSRO. Consider including ablation-style experiments.

Author response:
**Reviewer's Comment:**  
Consider including ablation-style experiments.

**Response:**  
In the appendix, we've included additional experiments with SP-PSRO not-anytime and SP-PSRO last-iterate. We believe this covers the additions present in our algorithms. Are there any other particular ablations the reviewer would like us to include?

Reviewer comment (Writing):
Restricted games and empirical games are used interchangeably when they are not interchangeable objects.

Author response:
**Reviewer's Comment:**  
Last two weakness bullets: need for clearer language and notation, particularly for an ML audience.

**Response:**  
We have revised the language for clarity, aligning it more closely with the ML audience's familiarity. Notations have been adjusted for consistency and comprehensibility. In particular, we have changed any mention of “empirical game” to “restricted game”. Changes are in blue.

Reviewer comment (Writing):
This is more minor, but I think is very important: the language and notation are more posed to a game theory audience than to the ML audience of ICLR. Following the notation and definitions in, say, the PSRO paper would make this paper significantly more approachable by this community. I personally find trouble whenever the RL definitions/notations are conflated with the notations used within game-theoretic reasoning. For example, when you see π, is it a single policy, or a distribution over policies (i.e., solution)?

Author response:
**Reviewer's Comment:**  
Last two weakness bullets: need for clearer language and notation, particularly for an ML audience.

**Response:**  
We have revised the language for clarity, aligning it more closely with the ML audience's familiarity. Notations have been adjusted for consistency and comprehensibility. In particular, we have changed any mention of “empirical game” to “restricted game”. Changes are in blue.

Reviewer comment (Writing):
I would suggest the authors consider a different name than Self-Play PSRO. This makes me think of PSRO with the MSS set to Self-Play, and doesn't really hint at all as to the nature of the underlying algorithm.

Author response:
**Reviewer's Comment:**  
The name 'Self-Play PSRO' might be confusing.

**Response:**  
We acknowledge this potential confusion. The name was chosen to reflect the mixed strategy found via training against the opponent's best response. We are open to renaming it based on suggestions you may have.

Reviewer comment (Experiments):
Why bother forcing SP-PSRO to train off-policy? The focus of this work is on anytime stopping not on computational efficiency. Have the authors tried using on-policy algorithms? This could result in further benefits.

Author response:
**Reviewer's Comment:**  
Query about the necessity of off-policy training in SP-PSRO.

**Response:**  
Sample efficiency is a significant concern for us. SP-PSRO is introduced with efficiency as the main motivation, which justifies the use of an off-policy algorithm to train the new strategy. Using an on-policy algorithm would add significant experience overhead in training the new strategy in combination with the standard APSRO/PSRO best response. 

It is also possible that we are not understanding correctly what the reviewer is asking. In that case, please clarify the question/comment and we will elaborate further. For example, is the reviewer talking about on-policy for the best responder or for the new strategy?

Reviewer comment (Experiments):
"Why the episode is terminated, the time average of v is added to the population." Why is episode used here? Do you only train your policy for a singular episode?

Author response:
**Reviewer’s Comment**
"Why the episode is terminated, the time average of v is added to the population." Why is episode used here? Do you only train your policy for a singular episode?

**Response**
Thank you for noticing this. We have changed the wording to “When the *iteration* is finished, the time-average $\bar{\nu_i}$ of $\nu_i$ is added to the population.”

Reviewer comment (Evaluation):
Limitations should discuss that while SP-PSRO uses same budget as APSRO, they're both way more expensive than PSRO.

Author response:
**Reviewer's Comment:**  
Limitations should discuss that while SP-PSRO uses same budget as APSRO, they're both way more expensive than PSRO.

**Response:**  
This is incorrect. APSRO uses the same budget as PSRO in terms of the number of episodes / environment interactions. Wall clock time is slightly slower. We have added the following to clear this up: “Importantly, compared to PSRO, APSRO uses the same amount of episodes and environment interactions. The only difference between the two is that APSRO changes the restricted distribution dynamically via a no-regret procedure.”

Reviewer comment (Theory):
Limitations need to also discuss that this analysis is on NE; PSRO is more general.

Author response:
**Reviewer’s Comment**
Limitations need to also discuss that this analysis is on NE, PSRO is more general

**Response**
We have added to the limitations that our method only focuses on Nash equilibrium. See also our response to number 4.

Reviewer comment (Experiments):
You've had to add lambda to apply your algorithms to these games. What is the impact of lambda on the performances you've demonstrated?

Author response:
**Reviewer’s Comment**
What is the impact of lambda on the performances you've demonstrated?

**Response**
We simulate best response training dynamics in these normal form experiments to demonstrate the relative performance of our algorithms on games that are quick and easy to run. So the lambda term should not be viewed as a hyperparameter that needs to be tuned, but rather a fundamental part of the game and environment experimental setup. Since you have asked for ablations on lambda, we have included them in the appendix. These results show that changing the lambda parameter does not change the performance that much.

Reviewer comment (Evaluation):
"As shown in Figure 4, unlike PSRO, APSRO tends not to increase exploitability." I don't think eyeballing Fig. 4 or 5 really provides enough evidence to support this claim. PSRO only looks guilty of a large exploitability increase in Leduc Poker, and in all other cases both algorithms are erratic.

Author response:
**Reviewer’s Comment**
As shown in Figure 4, unlike PSRO, APSRO tends not to increase exploitability

**Response**
We have removed this line.

Reviewer comment (Experiments):
"Note that APSRO and SP-PSRO only reach an ϵ-NE because they use a finite number of regret minimization updates to determine the restricted distribution, while PSRO is able to exactly compute a NE." I would have preferred the authors to have included an analysis of this, such as: A/SP-PSRO with exact equilibrium computation, and PSRO with regret minimization for computing equilibrium. This would help better understand where the benefits are coming from.

Author response:
**Reviewer’s Comment**
I would have preferred the authors to have included an analysis of A/SP-PSRO with exact equilibrium computation and PSRO with regret minimization for computing equilibrium.

**Response**

*A/SP-PSRO with exact equilibrium computation.*
The exact version of APSRO is ADO (Algorithm 1). We present experimental results for ADO in Appendix C. SP-PSRO has no exact counterpart because it relies on computing a time average of the new strategy over the course of a finite, boundedly accurate regret-minimization procedure.

*PSRO with regret minimization for computing equilibrium.*
The NE for the restricted game solved by DO and PSRO is easily tractable in most realistic scenarios, and we use exact methods rather than regret-minimization to solve DO’s/PSRO’s restricted game. PSRO with regret minimization for computing equilibrium to this restricted game will provide an equally or slightly less accurate restricted solution than an exact computation, depending on how long the regret minimization is run.

Reviewer comment (Theory):
"We hypothesize that this is due to the APSRO iterations not being long enough for the no-regret process to converge." Thank you for including this point. I was wondering if you had any insights into how we could know when we should apply APSRO based on this?

Author response:
**Reviewer's Comment:**  
Concerns about the applicability of APSRO in different scenarios.

**Response:**  
We acknowledge the limitation in predicting when APSRO might outperform PSRO. This mirrors the inherent uncertainties in PSRO performance. We have, however, observed consistent improvements in most games tested.

Reviewer comment (Writing):
While the text is globally easy to read I think that more pointers to the appendix, in particular for the experiments, could improve the clarity of the presented work. Some additional explanations about the computation of $\pi^r$ in the different algorithm is missing at least in the main text.

Author response:
We acknowledge your suggestion for clearer guidance towards the appendix, especially concerning our experimental framework. In our revised manuscript, we have included references to the appendix to clear up confusion. Revised text is in blue.

Reviewer comment (Evaluation):
The improvement in terms of monotonically of the exploitability curve that motivates the introduction of the different new algorithms is not completely clear in the experiments, especially for the APSRO algorithm vs PSRO algorithm.

Author response:
APSRO indeed does not show a completely monotone decrease in exploitability due to the imprecision in the best response and no-regret procedure. However, our experiments show that in most settings APSRO does not increase exploitability as much as PSRO. Furthermore, APSRO is a core component of SP-PSRO, which exhibits drastic increases in performance.

Reviewer comment (Experiments):
The SP-PSRO involves many different components: Q-learning for BR, regret minimization for solving the restricted games, potential reservoir sampling and additional supervised learning to train the mixed policy which results in a rather involved algorithm. Maybe it could be interesting to identify the key components of the SP-PSRO.

Author response:
Your observation about the complexity of the SP-PSRO algorithm is well-taken. Appendices F.1 and F.2 demonstrate that each component of SP-PSRO is in fact necessary.

Reviewer comment (Theory):
It could be useful to describe in details what is the restricted games and how the Nash equilibrium of these games is approximated by regret minimization. And maybe you could also provide more details for the proof of Theorem 3.

Author response:
In response to your suggestion, we have updated the text with clearer references to the appendix, which contains a detailed description of the restricted games and the methods used for approximating Nash equilibria, ensuring a clearer understanding of these critical aspects. We have also included a proof of Theorem 3.

Reviewer comment (Experiments):
What is the average size of the population in the different experiments? Because one possible drawback of the presented methods is that we have to store all the polices of the populations (and possibly for the average in SP-PSRO). Could you comment on this point?

Author response:
We have updated the appendix to include data on the population sizes and disk space usage for deep RL experiments. The largest population we generated had 148 policies when running SP-PSRO on 4-Repeated RPS, taking 22MB of storage.

Reviewer comment (Writing):
P3, beginning of Section 3: It is a bit hard to follow this part without knowing in advance what is DO and PSRO.

Author response:
We have included a reference to DO in the appendix to clear up the beginning of section 3

Reviewer comment (Writing):
P3, (1): It is not completely clear with the current notations if the policy $\pi_i \in \Pi_i$ could be a mixture of the policies in the population or should be exactly one of them. The introduction suggested that the first statement is the correct one? It could be also interesting to remark that a mixture of policy is also a policy somewhere.

Author response:
\pi_i \in \Pi_i is indeed a single policy. We have updated the paper to make this clearer.

Reviewer comment (Reproducibility):
P5, beginning of Section 4: Can you describe precisely how you do regret minimization among the population of policies in the restricted game? In particular how do you deal with an increasing number of policies?

Author response:
We have included a reference to the exact regret minimization procedure in the appendix at the beginning of section 4

Reviewer comment (Experiments):
P8, Section 6.1: It would be interesting to compare also with ADO to see what the new Double oracle brings over PSRO and how well APSRO approximate ADO. Which hyper-parameters did you use and which regret minimizer did you use in APSRO and SP-PSRO? (pointers to the appendix). Why not use the same horizon for all the games? How many seeds did you use in the experiments?

Author response:
We did not include ADO in our normal form games because the performance was almost exactly the same as APSRO, so we didn’t want to make it confusing to read. We have included references to the appendix for training details. We didn’t use the same horizon because some games converged faster. Would you like us to include these results for the camera-ready version?

Reviewer comment (Evaluation):
P8, Section 6.2: Same remark — it would be clearer to add reference to the appendix about the details of the experiments. How do you explain intuitively the gap between SP-PSRO and the other algorithms? It seems that SP-PSRO converges quickly but then gets stuck to a sub-optimal solution; e.g. in Leduc Poker it converges to an exploitability of ~0.4 which is not that good for this game. It is not very clear in Fig 5 that the exploitability of APSRO decreases more monotonically than the one of PSRO, contrary to what is claimed in the previous sections (in particular for tiny battleship). Could you comment on this point?

Author response:
We have included a reference to the Appendix for training details. Our intuition, as also described in the paper, is that one should include mixed strategies in the population. Ideally, these mixed strategies will be very close to a NE, so that is why we use a modified fictitious play procedure to produce the mixed strategies. You are right that SP-PSRO seems to have a big boost in performance at the beginning, but we don’t see other methods like PSRO or APSRO overtaking it. This makes sense because we can only improve our population by adding a policy to it, as long as we then output the least-exploitable distribution. As to the non-monotonicity of APSRO in Tiny Battleship, we acknowledge the limitation in predicting when APSRO might outperform PSRO. We hypothesize that APSRO's no-regret procedure was unable to find a satisfactory solution given the small budget allocated to each DO iteration. Hyperparameter tuning may yield better results.  We have, however, observed consistent improvements in most games tested.

Reviewer comment (Experiments):
P8, Section 6.2: Same question as above. And would it be possible to use larger horizon in Fig 6 and Fig 5 to see if the exploitabilities of SP-PSRO and APSRO cross or not? How do you compute the average strategy in the experiments?

Author response:
We do not extend the length of experiments because we do not witness the exploitability lines crossing in normal form experiments, and each run already takes multiple days to complete. We included a reference to the appendix which contains the details of how we compute average strategies.