Paper: Co-Learning Empirical Games & World Models (ICLR-2024)

Reviewer comment (Novelty):
There exists an extensive array of literature related to diversity within the PSRO framework which has not been discussed by the paper. For example [1][2][3][4][5]. In these works multiple diversity measures are recruited to measure the diversity over a PSRO population, and I would be interested if the authors could comment on specifically why they have chosen their variant of diversity measure over the others in the literature.

Author response:
Thank you for your kind words on the quality of the presentation and execution of this work. We address your questions about strategic diversity, paraphrased here, below.

> “Why have the authors chosen their measurement of strategic diversity over related works?”

Our reference to “strategic diversity” is from the perspective of learning a world model. As this is a well-defined supervised learning problem, for our purposes strategic diversity amounts to wanting the training dataset to reasonably cover the test dataset. We do not mean to suggest that we are introducing a new method of measuring diversity. Instead, we’re merely framing and motivating it from the perspective of world model learning in a multiagent system. Hence, we do not claim this as a contribution nor include an extensive literature review on this topic.

The reviewer offers several citations related to how considering strategic diversity may aid in strategy exploration (the process of selecting which new policy to include within an empirical game). Often these metrics of diversity ask how different is a candidate policy from a set of fixed policies. These metrics can be over the policies themselves (behavioral diversity), or how the policy strategically relates to the set of policies (response diversity). The cited works sometimes adopt ostensibly the same definition of diversity (at least in part) when they refer to behavioral diversity. The quality of these metrics is evaluated by how well they aid in improving the quality of an empirical game. 

The use of strategic diversity in the cited work is thus orthogonal to the consideration of world-model learning addressed here. This raises an interesting question: what is the impact of Dyna-PSRO when augmented to include diversity in its strategy exploration subroutine? We did not consider that here as  Dyna-PSRO is already a rather complex algorithm, and varying strategy exploration approaches would have complicated evaluation of our primary hypotheses.

Speculating, one would expect to see that the Dyna-PSRO+diversity methods variant would continue to be experientially cheaper than PSRO+diversity due to its inherent transfer learning. However, we expect that the difference between Dyna-PSRO+diversity and PSRO+diversity to be smaller than Dyna-PSRO and PSRO. This is because +diversity methods are using their experiential budget more effectively on game solving. This raises the related question as to how these diversity metrics aid in learning world models, and how well the world models trained through Dyna-PSR+diversity generalize when compared to Dyna-PSRO. These questions should be investigated in future work, but notably don’t undermine any results within this manuscript.

Reviewer comment (Evaluation):
The empirical results of the paper could have benefited from a clearer exposition, and some of these results are not as strong as claimed.

Author response:
The results may not be as strong as we would have preferred, but we argue they are sufficiently positive to validate this as a promising direction of work. We found several trends:
Including the random policy negatively impacted observation accuracy.
Including the random policy positively impacted reward recall.
The inclusion of more policies correspondingly reduces the observation loss (cross-entropy).
3.64±0.24, 5.33±0.89, 4.25±0.75, 1.99±0.23, 3.76±0.67, 1.83±0.24, 1.45±0.20 (following the order in the paper). 
We will include this additional result in the appendix.
More importantly, this experiment offered insights into world model training, which we could deploy within Dyna-PSRO.

Reviewer comment (Evaluation):
In Figure 2, while the setup is reasonable for assessing Claim 1, the results are not as convincing as desired — there is no clear trend between diversity of strategy-space samples and accuracy, nor clear trends regarding accuracy restricted to the strategies used for training.

Author response:
The results may not be as strong as we would have preferred, but we argue they are sufficiently positive to validate this as a promising direction of work. We found several trends:
Including the random policy negatively impacted observation accuracy.
Including the random policy positively impacted reward recall.
The inclusion of more policies correspondingly reduces the observation loss (cross-entropy).
3.64±0.24, 5.33±0.89, 4.25±0.75, 1.99±0.23, 3.76±0.67, 1.83±0.24, 1.45±0.20 (following the order in the paper). 
We will include this additional result in the appendix.
More importantly, this experiment offered insights into world model training, which we could deploy within Dyna-PSRO.

Reviewer comment (Experiments):
In Figure 2, the provided interpretation lacks coherency; if class imbalance is causing problems, perhaps a simpler game and setting should be used to better demonstrate the desired claim.

Author response:
We agree that had we chosen to study simpler games or force artificial diversity into our set of policies (see reviewer c2GE) that we could construct overwhelmingly positive results. We chose not to do this because we suspect that its results would not generalize to the practical cases of interest.

Reviewer comment (Presentation):
In Figure 4, the distinction between Decision-Time (DT) planning as used in this paper and Markov-Chain Monte-Carlo (MCMC) to obtain better value function estimates is unclear; it is questioned whether the RL model is already taking the maximum-estimated-value action.

Author response:
Decision-time planning is a classical reinforcement learning technique where planning is focused on a specific state to inform its action selection. Within this specific work, we take a very vanilla approach and do a width-and-depth-limited lookahead search. Given some candidate state s, we sample k actions, and use predict successor states/rewards for each of these k actions. Note that we’re not sampling stochastically from the transition dynamics in this work as we’re assuming a deterministic world model. We repeat this process for a fixed depth. The estimate of the action values from the original candidate state is then the maximum reward + bootstrap (using the value function to estimate the continued return from the final state in the search process) of each subtree of the search process. 

There are a number of ways that this differs from MCMC. The first is, that we are assuming deterministic transitions so have no need to sample over transition spaces. The second is that we do sample the full return in our planning procedure as one would with a Monte Carlo method, and instead leverage our value function to bootstrap this estimate. 

It is possible to use Monte Carlo methods for Decision-Time planning in Dyna-PSRO for correspondingly compatible classes of games. The benefit of using the world model is its predicted partial returns should offer a more accurate estimate of the value function. This does assume inaccuracies in the value function and corresponding accuracies in the world model.

Reviewer comment (Evaluation):
In Figure 4, the results appear to suggest that Decision-Time planning alone markedly degrades performance, yet this is ignored in the text. There is no clear justification for the claim that "world models offer the potential to improve response calculation through DT planning" based on this figure.

Author response:
The improvements of DT planning can be seen by it computing a policy nearly twice as strong as policy without planning (Figure 4). The caveat of this result, as discussed in the paper, is that this background planning should be used in conjunction with decision-time planning. As we mention in the preceding sentence “[T]he main outcome of these experiments is the observation that _multi-faceted_ planning is unlikely to harm a response calculation, and has a potentially large benefit when applied effectively.”

Reviewer comment (Presentation):
As a minor point regarding Figure 4, the x-axis and alignment of the loss curves on the right panel should be adjusted to agree with the analogous presentation in Figure 3, as is done in the left panel.

Author response:
We thank the reviewer for continuing to offer help that will improve our manuscript. We're glad that we were able to clear up some of your concerns and questions. The crux of the mentioned concerns lies in the relationship between presenting results and tying them back into the overall narrative of the work (esp., hypotheses). We agree this is an area for improvement and will continue to work towards this goal for the final version of the manuscript. The analogy in Figure 4 correctly captures one such point of improvement. The later interpretation of the result is stronger and more clear. We thank the reviewer for this helpful suggestion and will make the corresponding change. We are open to and welcome any further questions or suggestions you may have.

Reviewer comment (Experiments):
While an improvement over PRSO is shown, Dyna-PRSO should be compared against additional baselines in an ablation study — for example, models that incorporate only the world model or only the "Dyna" RL component to solve the empirical games component — to more rigorously establish the claims of the paper.

Author response:
World models do not constitute game-solving algorithms and would amount to self-play/independent-RL which is well-known in the literature to exhibit poor generalization performance. Moreover, these potential baselines are not optimizing towards a solution concept, and therefore, are addressing a different problem then PSRO/Dyna-PSRO and would be unfit to compare against. 

We’re not sure what the reviewer here suggests by only “incorporating the ‘Dyna’ RL component to solve the empirical games component” Dyna is definitionally the integration of planning, learning, and acting. The closest baseline is removing planning, which is the PSRO baseline we have included. 

The experiments in the preceding sections are meant to serve as a variety of candidate baselines for Dyna-PSRO with varying degrees of planning. Running each of the planning ablations in the full game-solving context would be extremely computationally demanding. Therefore, we settled for studying them in limited settings and deploying the must successful configuration for game solving. Therefore, we believe we have offered a diversity of baselines to establish the effective components of Dyna-PSRO.

Reviewer comment (Presentation):
There is a need for clarification on the difference between Decision-Time (DT) planning and Markov-Chain Monte-Carlo (MCMC) as posed in questions regarding Figure 4.

Author response:
Decision-time planning is a classical reinforcement learning technique where planning is focused on a specific state to inform its action selection. Within this specific work, we take a very vanilla approach and do a width-and-depth-limited lookahead search. Given some candidate state s, we sample k actions, and use predict successor states/rewards for each of these k actions. Note that we’re not sampling stochastically from the transition dynamics in this work as we’re assuming a deterministic world model. We repeat this process for a fixed depth. The estimate of the action values from the original candidate state is then the maximum reward + bootstrap (using the value function to estimate the continued return from the final state in the search process) of each subtree of the search process. 

There are a number of ways that this differs from MCMC. The first is, that we are assuming deterministic transitions so have no need to sample over transition spaces. The second is that we do sample the full return in our planning procedure as one would with a Monte Carlo method, and instead leverage our value function to bootstrap this estimate. 

It is possible to use Monte Carlo methods for Decision-Time planning in Dyna-PSRO for correspondingly compatible classes of games. The benefit of using the world model is its predicted partial returns should offer a more accurate estimate of the value function. This does assume inaccuracies in the value function and corresponding accuracies in the world model.

Reviewer comment (Evaluation):
There is a need for clarification on how Figure 4 supports the claim that DT improves response calculation, as posed in the questions regarding Figure 4.

Author response:
The improvements of DT planning can be seen by it computing a policy nearly twice as strong as policy without planning (Figure 4). The caveat of this result, as discussed in the paper, is that this background planning should be used in conjunction with decision-time planning. As we mention in the preceding sentence “[T]he main outcome of these experiments is the observation that _multi-faceted_ planning is unlikely to harm a response calculation, and has a potentially large benefit when applied effectively.”

Reviewer comment (Theory):
This is probably the theoretician in me, but it would be nice to see some theoretical guarantees measuring how much of an improvement Dyna-PSRO is over PSRO. I understand that this might be difficult to obtain given that it might be game-dependent.

Author response:
We agree that theoretical results would be advantageous. However, due to the complexity of studying games in general, we were unable to construct any bounds that improved upon PSRO. This is primarily due to the convergence of learning-based game-solving algorithms hinging on using epsilon best-response subroutines. In idealized settings, one could construct games where model-based RL (Dyna-PSRO) is guaranteed to produce a smaller epsilon BR than model-free (PSRO). However, it’s not understood how these small epsilon errors contribute to the overall algorithm's long-term progress toward game-solving. This is because small epsilon noise in the best response can help PSRO explore new policies in a way that could further improve the empirical game. We agree that this is a super interesting research direction, and any progress toward it would be a great advancement.

Reviewer comment (Experiments):
I see you selected two versions of Harvest and Running with Scissors for testing. Is there any reason behind these choices?

Author response:
We selected these games for several reasons. The four reasons behind choosing Harvest were: (1) we had access to both simple (categorical) and more complex (RGB images) variants of the game allowing for in-depth analysis, (2) its popularity in the field of multiagent reinforcement learning, (3) we had experiences running experiments on the game, and (4) it is general-sum meaning it represents an important class of games. As for Running With Scissors, we both similarly had experience training agents in this game as well as it being a strictly competitive (zero-sum) non-transitive game. Thereby, representing a different and important class of games than Harvest.

Reviewer comment (Evaluation):
This is an honest question: Were you surprised by the results? By combining the two (empirical game and world model) you would expect that the training should be no worse, no? However, the paper measures, at least empirically, how much does combining the two improves the SumRegret and the return?

Author response:
We were not generally surprised that Dyna-PSRO outperformed PSRO, because including transfer learning should at least preserve performance. It is possible for transfer learning to hurt performance, but because we had studied the co-benefit problems independently and extensively before deploying them in Dyna-PSRO, we had confidence that it was unlikely to hurt performance in the worst case. We were, however, surprised by how large of an improvement that was possible without employing more advanced planning algorithms.

Reviewer comment (Experiments):
Is there anything that can be said about how many less experiences are needed in the Dyna-PSRO vs. the standard PSRO?

Author response:
This is a really good question and one we’re still thinking about. As we mentioned in Q1, the community currently doesn’t fully understand the relationship between the best-response quality and how many rounds of empirical game expansion are required. As a result, only last-iterate convergence guarantees could be readily made and they would be straightforward adoptions of guarantees from the model-based reinforcement learning literature (requiring strong assumptions about the nature of the game). Speculating, we would expect games with simpler dynamics and a larger strategic space to benefit more from Dyna-PSRO. However, games with exceptionally complex dynamics may see little benefit as learning an effective world model is difficult.

Reviewer comment (Writing):
I'm assuming NE in the first paragraph of Section 2 refers to Nash equilibria, but it should be spelled out once for the sake of completeness.

Author response:
Thank you for the editorial suggestions, we agree that fully defining these terms will help with the completeness and clarity of the work. We will incorporate these changes in the final version of the paper.

Reviewer comment (Writing):
Section 3 could be spelled out a bit better in terms of definitions. There are some variables undefined like $\Delta(\mathcal{R})$ and $\mathcal{A})$ which should be spelled out for the sake of completeness, though I know they are of common use.

Author response:
Thank you for the editorial suggestions, we agree that fully defining these terms will help with the completeness and clarity of the work. We will incorporate these changes in the final version of the paper.

Reviewer comment (Reproducibility):
The minor weakness is the relative lack of clarity on the approach in the paper itself. The appendix implementation details help but there's still some things that are unclear. As an example of what I mean, the authors state: "This schedule starts out training as a variation of teacher forcing, and slowly transitions to fully auto-regressive." It's clear from this statement what is happening at a high level, but not the exact setup for how the models are trained. This kind of imprecision is unfortunately common in the current draft concerning the authors' approach.

Author response:
Thank you for your kind words on the originality, quality, and interest of our work. We address your comments and questions below (paraphrased).

> “Lack of clarity in the approach. For example, action-conditioned scheduled sampling.” / Question 1

We apologize for the lack of clarity and would be more than happy to elaborate on any specific details that any reviewer found unclear. We agree that this is an area for improvement in the manuscript as the algorithms being used are complex and challenging to convey in limited pages. Despite this, we hope that our effort to include pseudocode, hyperparameter tables, algorithmic descriptions, graphical descriptions, and a promise to release code help alleviate these concerns in the long-term. We will continue to work on improving the clarity of the text as suggested.

In regards to action-conditioned schedule sampling, we believe that clarity may have been compromised due to separating details across sections C.2.1 and C.2.2 and to a missing high-level description of the World Model training process. Thank you for pointing out this missing detail. The world model is trained as a sequence-to-sequence supervised learning problem. This motivated our analogies to “teacher forcing” and “auto-regressive” training regimes. To expand, as requested in Question 1, we train our world models in the strategic diversity section on a fixed dataset. For each gradient step, we sample a mini-batch of sub-trajectories of length 20. Part (5 timesteps) of the sub-trajectories are used to burn-in/initialize the recurrent state of the world model. Notably, this means that we’re not performing backpropagation through these timesteps. For the remaining timesteps, we perform action-conditioned scheduled sampling (C.2.1, Algorithm 1) to train our model. Essentially, for the remaining timesteps we take the previous world model's predictions and randomly feed them as input to the model (auto-regressively) as opposed to using the true next timestep from the dataset. We define an epsilon-schedule (C.2.2) that describes how this random sampling occurs throughout training. At the beginning of training, we are performing teacher forcing: giving the world model ground truth inputs as it predicts the full sub-trajectory. After many updates, we start randomly not using ground-trough timesteps within a trajectory, but instead feed in the models previous output as input. By the end of training we are only giving the world model the burn-in timesteps and all inputs are auto-regressive. This allows the world model to learn to make accurate predictions given any artifacts induced by the world model. We will update the text of the paper to make this all clearer.

Reviewer comment (Evaluation):
The major weakness of the paper is that the results do not fully seem to support the authors' stated claims. Section 3.1 seems to actually contradict the claims that empirical game learning can benefit world model learning, since the approach is outperformed in terms of reward prediction by a random sampling approach.

Author response:
Thank you for your kind words on the originality, quality, and interest of our work. We address your comments and questions below (paraphrased).

> “Results do not fully seem to support the authors’ stated claims.” / Question 2

It is not accurate to say that “random sampling approach[es]” would outperform any of the world models we trained. The class imbalance in rewards, favoring unrewarding states, means that random sampling would actually perform quite poorly in terms of reward-prediction accuracy. It is true, however, that predicting only the most frequent class, 0 reward, will register as having a very strong reward-prediction accuracy. This doesn’t indicate a problem with our trained world models, but with using accuracy to evaluate the quality of world models. These metrics don’t measure their effectiveness in the downstream task of interest: planning. We briefly mention this in Section 3.1’s discussion, but the nuance is omitted due to paper length requirements. As a stopgap to measuring planning performance in isolation, we considered further metrics such as recall and saw small positive benefits. Furthermore, in Section 3.2 and the appendix we perform planning experiments with both the {1,2,3} world model and the {1} world model. When compared (background: Fig3 v Fig16, decision-time: Fig4 v Fig19), we found that the more strategically diverse world model led to much more successful policy training. Moreover, if you examine the world models observation cross-entropy loss across the same diversity order in the paper you observe: 3.64±0.24, 5.33±0.89, 4.25±0.75, 1.99±0.23, 3.76±0.67, 1.83±0.24, 1.45±0.20. This is another instance of seeing a positive correlation between strategic diversity in world model performance. These arguments do not directly appear in the body of the paper and we will revise the manuscript to make these results clear. In total, we believe that all of this evidence adequately supports the strategic diversity claim in the paper and that empirical games benefit world models.  We would also like to stress that the strategic diversity result is meant only as a point of novelty (as mentioned in the introduction), and is not meant to be a major claim or contribution of the paper.

Reviewer comment (Experiments):
The games employed throughout the paper are rather simple in comparison to the common games used to evaluate world models (e.g., Pong, Doom, Pacman, Cheetah Run, etc.). While the focus on multiagent settings is understood, it might have been helpful to construct a setting more like a traditional world model environment, which would have also allowed for a comparison against a World Model baseline.

Author response:
> “Games are simpler than standard games used to evaluate world models.” / Question 3

We respectfully disagree that our selected games are appreciably simpler than the standard world-model games (save, Minecraft, but there is limited success in this domain). A key difficulty in our games not present in the recommended games is that they are partially observable. This adds a huge amount of complexity to training an effective world model as it must infer from a small viewbox around the agent the state of the much larger world around it. Moreover, multiagent systems cause the dynamics learned by the world model to present as nonstationary as other agents are taking unseen actions. This furthers the difficulty of the learning problem. 


> Question 4: Why not include a world model baseline?

Standard model-based reinforcement learning (MBRL) algorithms are not optimized for game solutions. Instead, they are typically trained in a way that would generate one set of performant agents that would be unable to generalize to different coplayers (a well-known issue in multiagent learning algorithms). PSRO modifies RL to optimize for game solutions through its empirical game. Therefore, a baseline MBRL algorithm without an empirical game would be an unfair comparison as it is optimized for a different objective. It would be expected to perform quite poorly, and how to directly and fairly compare the two algorithms (PSRO vs MBRL) is unclear. 

To speculate, we would expect MBRL to perform roughly as well as PSRO at ~3 epochs. This is because, at this point in PSRO, we have essentially trained a BR(Random) and BR(BR(Random)). Why we speculate this is comparative to MBRL is because of symmetry breaking. At some point when the two MBRL policies are training, one agent will learn an effective policy (roughly BR(Random)) and the other can only learn to do the best it can in this setting (roughly BR(BR(Random)). Anecdotally, this overfitting to one specific interaction protocol has been repeatedly observed in many multiagent works. We would speculate overcoming this point in training would require considering the strategic nature of the game explicitly, as PSRO most naturally does.

Reviewer comment (Evaluation):
As it stands, the takeaway from the results is that world models benefit empirical games but not necessarily the other way around. This is still a contribution, but a more limited one with less general significance, and it does not match the authors' stated claims and contributions.

Author response:
Thank you for continuing to engage with our work. We're glad our response to Q1 addresses your immediate questions. Below we address the unresolved comments and questions.

## Q2

**Supporting Claims.** The reviewer is correct to suggest that planning performance does not necessarily imply that it occurred because of "a broader consideration of possible game dynamics," by the world models. Instead, we claim that the combined 5 different results together support this claim:
- Including a random policy hurts observation-prediction accuracy, because the dataset has a reduced set of game dynamics (Figure 15).
- Including a random policy helps reward-prediction accuracy, because it helps the class imbalance in rewarding data (Figure 15).
- A world model trained on more strategically diverse data (that _should_ support a broader set of game dynamics) is more performant at _background_ planning ({1}: Figure 17; {1, 2, 3}: Figure 3).
- A world model trained on more strategically diverse data (that _should_ support a broader set of game dynamics) is more performant at _decision-time_ planning ({1}: Figure 18; {1, 2, 3}: Figure 4).
- Including additional strategic policies improves the observation loss (cross-entropy) of the world model. 

| {1} | {2} | {3} | {1, 2} | {2, 3} | {1, 3} | {1, 2, 3} |
|---|---|---|---|---|---|---|
|3.64±0.24| 5.33±0.89| 4.25±0.75| 1.99±0.23| 3.76±0.67| 1.83±0.24| 1.45±0.20|  

We agree that these results are not overwhelmingly positive, but we argue that they sufficiently support the claim. Our concluding statement summarizing the totality of our results on strategic diversity reflect this strength of results: "Overall, these results provide **some evidence supporting the claim** that strategic diversity enhances the training of world models." Moreover, we have not listed this claim as a major contribution of this work. We did not list this claim as a contribution as we felt it was generally known in the community: assuming a perfect learning algorithm training on a dataset (covered transitions) that better covers the test dataset (possible transitions) supports a more general [world] model. Therefore, we did not construct a trivial example where the results were overwhelming positive, and instead chose a practical investigation.


**Background Planning Results.** There are two main things to consider when evaluating the performance of a transfer learning algorithm: (1) the time to convergence, and (2) the final performance. The former indicates how much sooner one could _stop_ the learning algorithm, and gain benefit (measured by number of updates) from the early stopping. This is a softer metric, because determining the exact point of convergence is often unclear. Therefore, we have only light qualitative results on this front: planner {1,2,3} starts learning earlier. The later is a more clear metric, and we would argue that the results are not "fairly comparable" as there is strict improvements in terms of average return and reduction in variance:

|Learner | Final Return |
|---| ----|
|Baseline|23.00 ± 4.01|
|Planner {1}| 26.05 ± 1.30|
|Planner {1, 2, 3}| 31.17 ± 0.25|

Reviewer comment (Evaluation):
I find it hard to understand why PSRO is a reasonable algorithm for the Harvest game, and the authors do not provide a strong argument or qualitative evidence here. At a high-level, the Harvest game reduces to a social dilemma, and therefore the Nash solution is not the desirable one (for it means that everyone will defect and the resources will be exhausted, leading to low individual and collective return). I suspect that PSRO and Dyna-PSRO are finding a variety of different solutions that succeed in defection. Indeed, low regret against exploitative opponents will exactly correspond to defection. Instead, what one should be looking for in this context are policies which incentivize others to cooperate (along the lines of opponent modelling, for instance). Now, world modelling does, in principle, help here (see this recent paper: https://arxiv.org/pdf/2305.11358.pdf). But without clear measurements of the individual return, collective return and qualitative analysis of the policies, the reader cannot judge whether the world model yields better or worse outcomes in Harvest.

Author response:
> “Do PSRO or Dyna-PSRO find cooperative solutions in Harvest? If not, why use them?”

Our perspective is that PSRO is a method for finding solutions (e.g., Nash equilibria) for games, and we evaluate our methods based on their computational performance in finding a game solution. In some settings one might prefer to find some solutions (e.g., “cooperative” or high welfare) rather than others; how to address that in PSRO is a topic of much current research but not what we address in this paper. We would argue that being able to find any solution of a complex game is often quite interesting and valuable.

Actually, any equilibrium solution for Harvest will balance cooperative and competitive elements, and we do (anecdotally) observe this in the solutions found by our algorithms. This was qualitatively observed as the agents established a pattern of harvesting the orchard in a circular pattern. Moreover, we found it rare that a policy would take the competitive “tag” action due to it being difficult to effectively learn. We agree that by mentioning in section 3.1 that the “PSRO policies are highly competitive, tending to over-harvest,”  may be misleading. This discussion comment was meant only to provide some intuitive insight about this very small subsample of policy behavior, and not commentary about solutions found by the full game-solving algorithm (PSRO or Dyna-PSRO). 

An interesting question to explore in the future is how alternative solution concepts or MSSs affect the advantages and disadvantages of PSRO and Dyna-PSRO.

Reviewer comment (Evaluation):
Do the PSRO or Dyna-PSRO agents find cooperative solutions in Harvest? If not (as the case seems to be from the discussion on page 5), what is the argument for using PSRO / Dyna-PSRO in this environment?

Author response:
> “Do PSRO or Dyna-PSRO find cooperative solutions in Harvest? If not, why use them?”

Our perspective is that PSRO is a method for finding solutions (e.g., Nash equilibria) for games, and we evaluate our methods based on their computational performance in finding a game solution. In some settings one might prefer to find some solutions (e.g., “cooperative” or high welfare) rather than others; how to address that in PSRO is a topic of much current research but not what we address in this paper. We would argue that being able to find any solution of a complex game is often quite interesting and valuable.

Actually, any equilibrium solution for Harvest will balance cooperative and competitive elements, and we do (anecdotally) observe this in the solutions found by our algorithms. This was qualitatively observed as the agents established a pattern of harvesting the orchard in a circular pattern. Moreover, we found it rare that a policy would take the competitive “tag” action due to it being difficult to effectively learn. We agree that by mentioning in section 3.1 that the “PSRO policies are highly competitive, tending to over-harvest,”  may be misleading. This discussion comment was meant only to provide some intuitive insight about this very small subsample of policy behavior, and not commentary about solutions found by the full game-solving algorithm (PSRO or Dyna-PSRO). 

An interesting question to explore in the future is how alternative solution concepts or MSSs affect the advantages and disadvantages of PSRO and Dyna-PSRO.

Reviewer comment (Presentation):
The order in which the paper is presented is confusing. The main algorithm, Dyna-PSRO, is not introduced until very late in the paper. Since this is the main result, the authors would do better to introduce this first, and then present the additional sections as ablations or analyses.

Author response:
> “Organization of the paper is confusing.”

We chose to organize the paper in this way because we view section 3, which contains the referred “ablations or analyses”, as a methods section that is essential to understanding any presented results with respect to Dyna-PSRO. The experimental details found within section 3 are directly used by section 4, and any results on Dyna-PSRO would be impossible to interpret without those details.

Reviewer comment (Experiments):
The results do not compare to existing strong baselines. For instance there are many existing papers that produce agents with good performance on Harvest and Running with Scissors (e.g. https://arxiv.org/pdf/1906.01470.pdf, https://arxiv.org/pdf/2102.02274.pdf, https://arxiv.org/abs/1803.08884). To what extent does this new method outperform the baselines?

Author response:
> “Does not compare to existing baselines.”

The reviewer is correct to note that there exist many works that have studied these games as they appear in MeltingPot. What separates this line of work from them is their goal: the cited work focuses on primarily either studying the game itself or on simultaneously learning policies. Their primary interest isn’t in solving the game for any specific solution (e.g., Nash). To this end, these works likely have extremely strong policies in specific strategic settings, that may perform poorly with other coplayers (i.e., against policies that they were not trained with). This is not because either method is better or worse, but because the goals of these works are different. The PSRO line of work is interested in developing general game-solving algorithms that are notably flexible on solution concepts. Therefore, we think making direction comparisons is unclear for such disparate algorithms, and unfair as they are solving meaningfully different problems.

Reviewer comment (Presentation):
The results are very hard to interpret. In Figure 5, why do the Dyna-PSRO curves terminate before the PSRO curves? In Figures 3 and 4, what is the difference between the left-hand plot and the right-hand plot? The Figure captions in Section 3.2 require significant clarification, because it is extremely hard for the reader to assess the results in this section at present.

Author response:
> In Figure 5, why do the Dyna-PSRO curves terminate before the PSRO curves?

This is because Dyna-PSRO has a longer walltime than PSRO, so we were more limited in how long we were able to run the algorithm. Note that in “Harvest: Categorical” Dyna-PSRO has converged on the x-axis, and that in the other two games, Dyna-PSRO outperforms the other two games when PSRO is allowed to run for many more iterations. 

> In Figures 3 and 4, what is the difference between the left-hand plot and the right-hand plot?

The difference between the two plots is in the x-axis. The left-hand plot measures each learning algorithm's performance against  “Real Experiences” used during training; whereas, the right-hand plot measures against both “Real and Planned Experiences.” These terms are introduced in Section 3.2.1 (paragraph one, bolded terms), where real experiences are data from interaction with the real game, and planned experiences are data generated from interaction with the world model. The left-hand plot shows the comparison of learners against the objective of interest since we’re assuming generating real experiences is the bottleneck in game solving. The right-hand plot offers additional insight into the respective algorithms by further comparing them by the amount of additional planned experiences they use. This offers insights into the training dynamics of these learners, and how they tend to take more wall time (as they need many more gradient steps) to run than the baseline methods. We will update the figure captions in this section to make this more clear.

Reviewer comment (Novelty):
There are some missing citations e.g. to the Dreamer line of work (https://arxiv.org/pdf/2301.04104v1.pdf and citations therein), the MuZero line of work (https://arxiv.org/abs/2111.00210 and citations therein), and the aforementioned paper on world modelling in the Harvest game (https://arxiv.org/pdf/2305.11358.pdf).

Author response:
> “Missing citations”

We will add the suggested citations to the final version of the manuscript, thank you for the suggestions.