Paper: Best Response Shaping (ICLR-2024)

Reviewer comment (Theory):
The fundamental issue of this work is that the paper directly assumes **symmetric NEs** throughout the entire paper. If we know **in advance** that the desired NE **must** be symmetric, then all the technical derivations in this paper are correct. However, although IPD falls into this category, this remains an extremely strong assumption and you cannot really leverage this assumption during algorithmic design, which, to some extent, is a cheating strategy.

Author response:
We would like to remind the reviewer that we are not making the assumption that the solutions to our games are symmetric Nash Equilibria: the detective and agent do not share parameters with each other therefore the claim that we assume symmetric NE’s is **false** and not incorporated into the design of the algorithm. We recognize that the main variation of the algorithm incorporates a self-play loss that, if this were to be the only loss, would make it impossible to learn non-symmetric NE’s in 1-step Matrix form games. But we would like to remind the reviewer that our scope is iterated social dilemmas.

Reviewer comment (Theory):
The claims in Section 4.3.2 are problematic. The statement "_this is equivalent to training an agent with self-play with reward sharing_" is **wrong**. Equation (7) is only equivalent to **reward sharing _with parameter sharing_**. "Self-play with reward sharing" should refer to learning two different policies with different parameters while sharing the game reward across agents. Self-play with reward sharing could learn an NE in the XOR game, while Equation (7) cannot due to parameter sharing.

Author response:
Regarding the **problematic claims**:  

The reviewer states *“This claim is wrong. Equation (7) is only equivalent to reward sharing with parameter sharing.”* We would like to remind the reviewer that, by definition, self-play amounts to parameter sharing, therefore the original claim is **not wrong**. The reviewer also writes “Self-play with reward sharing'' should refer to the case where "you learn two different policies with different parameters while the game reward is shared across agents". This by definition cannot be self-play.

Reviewer comment (Theory):
The statement "_In zero-sum games, this update will have no effect as the gradient would be zero_" is correct but meaningless. In a symmetric and zero-sum game, the expected reward is always 0 when the players use identical policies, which means you have zero gradient everywhere.

Author response:
The reviewer writes *“a game is both symmetric and zero-sum if and only if it is a zero game”*, this claim is **wrong**. We are only aware of the definition of “zero-game” in the context of combinatorial games (from wikipedia): “the zero game is the game where neither player has any legal options”. A fundamental assumption of combinatorial games is a turn-taking nature. Therefore any non turn-taking symmetric zero sum game (e.g. rock paper scissors) is not a zero game.

Reviewer comment (Theory):
The concept of _symmetric game_ only applies to two-player games. For general $N$‑player games, assuming the game is symmetric essentially means assuming the game is cooperative.

Author response:
The reviewer states “the concept of symmetric game only applies to two-player games.”, this statement is **wrong**. N-player symmetric games exist and are clearly defined. Tragedy of the Commons is one of them.

Reviewer comment (Presentation):
Section 2.1 presents an $N$‑player game formulation. However, the entire paper adopts a two-player setting.

Author response:
Regarding **minor issues**:  

1. The n-player formulation is standard in the literature and provides a general framework in which the two-player formulation is an instance.

Reviewer comment (Writing):
There are notation issues that cause confusion about which agent an action refers to. For instance, in Equation 6, $b$ denotes the action from agent 2, but this notation for $b$ was never introduced before.

Author response:
Regarding **minor issues**:  

3. We are aware of some notational inconsistencies and we thank the reviewer for pointing them out, we will definitely correct these.

Reviewer comment (Evaluation):
My major concern is that the explanation why BRS outperforms POLA is not very clear. The authors mention that "When the opponent is specifically trained to maximize its own return against a fixed policy trained by POLA, the first exploits the former." However, it is not clear whether this issue would be avoided when POLA is replaced by the proposed BRS. Additionally, the paper claims that "POLA can't differentiate through all opponent optimization steps," but more experiments are needed to support this claim. Besides, the authors also state that "the interactions between the agent and the detective mirror the foundational Stackelberg setup," but it is still not clear whether convergence to Stackelberg equilibrium helps to improve social welfare. More detailed analysis is needed to clarify these points.

Author response:
What we intended to demonstrate is that BRS produces a final, fixed policy that is not exploitable by a learning policy trained against it. We believe that the performance against MCTS demonstrates that while POLA is exploited, BRS is not. Given that both the MCTS and the learning opponent approximate the best response (but the MCTS is stronger) then we did not feel the necessity of showing the performance against a learning opponent. We do have experimental results against a learning opponent for BRS and POLA and will incorporate them in the paper. We also share the reviewer’s concern that a Stackelberg equilibrium might be undesirable for social welfare and suspect this is the fundamental reason for the need of the prosocial self-play loss. However, it would be helpful for us if the reviewer clarifies what kind of analysis would help solidify these points.

Reviewer comment (Experiments):
The training curves for baselines and proposed methods are missing. Also it would be helpful to test the proposed method in more complicated social dilemma environments, such as those presented in [2].

Author response:
We will include the training curves for these algorithms, but considering more complicated social dilemmas puts BRS to a higher standard than its predecessors, namely POLA.

Reviewer comment (Writing):
Some typos need to be fixed, including Alg 1., $\theta_1' \leftarrow \theta_1+z$ should be $\theta_1 \leftarrow \theta_1+z$.

Author response:
We thank the reviewer for pointing out these typos and we will fix them.

Reviewer comment (Theory):
In Corollary D.3, the paper uses $r^1\left(s_t, a_t, b_t\right)+r^2\left(s_t, b_t, a_t\right)=0$ which implies $R^1(\tau)=-R^2(\tau)$. However, in proposition D.2, the authors use Lemma D.1 which assumes $r^1\left(s_t, a_t, b_t\right)=r^2\left(o_t, b_t, a_t\right)$ so as to imply  ${\mathbb{E}}\left[R^1(\tau)\right]={\mathbb{E}}\left[R^2(\tau)\right]$. Since Corollary D.3 also involves proposition D.2, this creates a potential inconsistency, and the proof of Corollary D.3 may not be correct. The authors are requested to provide clarification on this matter.

Author response:
We thank the reviewer for taking the time to read through the proof and will do our best effort to clarify it. These two statements refer to different properties of the game. In particular $r^1(s_t, a_t, b_t) = -r^2(s_t, b_t, a_t)$ refers to the zero sum property and $r^1(s_t, a_t, b_t) = r^2(o_t, b_t, a_t)$ refers to the symmetry property. The key observation is that $s_t \neq o_t$, therefore $r^1(s_t, a_t, b_t) = r^2(o_t, b_t, a_t)$ does not imply $\mathbb{E}\left[R^1(\tau)\right] =\mathbb{E}\left[R^2(\tau)\right]$. The correct implication would look something of the form  $\mathbb{E}\left[R^1(\tau)\right] =\mathbb{E}\left[R^2(\tau_o)\right]$ where for each $\tau$ there exists a $\tau_o$ with the same probability (in the self-play case). As an example consider a state $s_t$ in the Coin Game where the blue player takes the blue coin and the red player does not. The $o_t$ state refers to an analogous state in which the red player takes the red coin and the blue player does not. By symmetry of the game it should be the case that $r^1(s_t, a_t, b_t) = r^2(o_t, b_t, a_t)$. And, if the game rewards are set appropriately, it can simultaneously hold that $r^1(s_t, a_t, b_t) = -r^2(s_t, b_t, a_t)$ (that would correspond to the red player being punished with a negative reward of what the blue player got for taking the blue coin in state $s_t$).

Reviewer comment (Novelty):
The training paradigm described here is closely related to [PSRO](https://arxiv.org/pdf/1711.00832.pdf), which is not mentioned in the paper.

Author response:
We will indeed include PSRO in our literature review as it highlights the contribution of BRS over previous existing approaches. PSRO extends an existing set of policies by the best response policy to a meta-strategy of previous policies. PSRO does not differentiate through this best response generation mechanism. This is in contrast to the essence of BRS. Let’s imagine PSRO on IPD. First, it starts with a random policy. Next, always defect as the best response is added. Next, the best response to the existing set is added which is again always defect. Therefore, PSRO cannot discover tit for tat(TFT) on IPD. However, BRS is different. BRS trains an agent against a best response opponent by being aware that the opponent will be the best response. That pushes BRS to learn TFT.

Reviewer comment (Writing):
The writing does not make it clear that the detective and agent take turns during optimization. The title of Algorithm 1 (that it says "a single iteration") is the *only* way I knew that this is how the training is done. Otherwise, it seems as though the detective is first trained on a pre-selected buffer B, which *significantly changes the reader's understanding of the method*.

Author response:
We thank the reviewer for their deep reading of the paper. We will rewrite the description of the algorithm to make it clear that the agent and the detective are being trained simultaneously.

Reviewer comment (Writing):
The whole description of "Simulation-Based Question Answering" makes the setup far more confusing than it needs to be. Here is my understanding: The conditioning vector consists of the Q-Values of each action of a random opponent policy in that state. The Q-Values are estimated by sampling trajectories. It is not described in the paper like this, and is instead treated as a complicated "Question-Answering" scheme.

Author response:
The QA description shows that the detective can use any conditioning mechanism that requires querying the agent in different states. The choice for the coin game was an instance of the QA.

Reviewer comment (Writing):
It's unclear why the authors have a paragraph about Diplomacy in the related work and a paragraph about self-play from human data. It's fine to include it, but the authors should explain, *in the paper*, why they include it.

Author response:
Diplomacy is mentioned as a large-scale game in which without forming local alliances it is impossible to win the game. Because naive RL agents don’t develop the necessary reciprocation based cooperation the successful approaches train on human data. Therefore, one long term goal is to train agents that do well on Diplomacy without any human data to teach them the tit for tat like behavior.

Reviewer comment (Writing):
The authors do not write out the bi-level optimization in Equation 12, making it seem like a single-level optimization. Writing it out would make this significantly more clear that it's a bi-level process.

Author response:
We don't have an Equation 12; perhaps you're referring to Equation 2? Indeed, we will make it clear that theta2_star represents the entire optimization process, encompassing differentiation throughout. We appreciate the reviewer's clarification on this.

Reviewer comment (Experiments):
The authors state that "in this paper, [they] advocate that a reasonable point of comparison is the agent’s outcome when facing a best response opponent, which we approximate by Monte Carlo Tree Search (MCTS)." I cannot find where in the paper the authors advocate for this. This is a significant claim, since arguably POLA and LOLA are not interested in this setting as much and seem to be far more interested in learning *online* against opponents that are also *learning online*. This is rather important, as it makes head-to-head comparisons with POLA a bit more questionable, seeing as it was designed for a different setting.

Author response:
POLA is the only method that gives us a strong baseline for the coin game. We agree with the reviewer that while we care about generating a static policy, POLA cares more about learning online. However, the POLA authors themselves evaluate the final POLA against always defect and always cooperate opponents indicating interest in studying the properties of static final policy. Also, producing a final static policy that does well against learning opponents is valuable.

Reviewer comment (Evaluation):
BRS is *more exploitable than POLA* in Figure 3 against AD, seemingly contradicting the purpose of the paper. BRS-NOSP performs better, but at the cost of social welfare. However, I understand that you can't get the best of everything in general-sum games (See Q4 below).

Author response:
**First**, the best response to TFT on IPD is always cooperate. This is important as learning opponents find the best response to the deployed agent eventually. Our MCTS experiments show that while the best response to BRS is almost always cooperation, this is not the case for POLA. 

**Second**, TFT agents always cooperate with themselves. This is not the case for POLA while it is for BRS agents. 
**Third**, As TFT starts with cooperation at the first move it gets a lower return in head to head game with always defect. But, that is a desirable property of TFT for social settings, assuming cooperation and also keeping the door open for forgiveness. BRS agents are exploited slightly more by always defect than POLA agents. But, first, still compared to always cooperate they are exploited much less. Second, the always defect is not being a rational opponent against BRS here as it could have a much higher return if it had cooperated. Third, this tendency of BRS to break the defect loops while making it a bit more exploited against always defect gives it its ability to cooperate with itself.

Reviewer comment (Presentation):
The authors collected head-to-head results of POLA and BRS (in the Appendix Figure 7), but have elected not to show them in the main text. It seems as though POLA outperforms BRS in the head-to-head, further indicating that BRS is exploitable.

Author response:
Regarding the head to head results, we don’t think solving social dilemmas is about comparing head to head results between two algorithms and indicating the algorithm with the higher return as the winner. TFT gets exploited by always defect but still tit for tat is a much better strategy. We can’t have complete non-exploitability and complete cooperation at the same time but we argue BRS makes a better trade off.

Reviewer comment (Evaluation):
The authors write that the Good Shepherd is not scalable; however, their approach also seems similarly difficult to scale. (See Q3: below). What is the sample efficiency in terms of simulated environment steps?

Author response:
Regarding scalability, BRS is more scalable than Good Shepherd which differentiates through a gigantic computational graph of the opponent learning from scratch to convergence. But, we agree with the reviewer, BRS needs to do the required simulations and this limits its scalability. More advanced question answering mechanisms can improve the sample efficiency.

Reviewer comment (Novelty):
Some important references have been missed such as [1] and [2]. For example [1] introduces a method for incorporating diversity into best-response dynamics leading to a diverse fictitious play/diverse best response dynamics. Since fictitious play is based on a type of best-response method, a discussion on how the authors’ paper compares to [1] is needed to be able to properly evaluate the authors’ current contribution.

Author response:
We appreciate the reviewer pointing these references out, we will include them in the literature review.  We would like to clarify that [1] makes two major assumptions that significantly simplify the problem. First, [1] is restricted to normal form games so the best response that is being computed corresponds to a mixed strategy. In contrast, BRS computes an approximation to the best response in policy space, which is significantly harder (and unclear how to do using [1]). Second, because [1] deals with matrix form games, it is relatively straightforward to define a sampling method for the policies, in their case a Determinantal Point Process, due to the linear nature of the underlying strategies. In contrast BRS adds diversity to fictitious play by keeping a replay buffer of encountered agents and adding noise to the policy parameters. So it remains unclear to us how a) [1] is able to scale beyond normal form games into games that require a neural network policy parameterization and b) that [1] provides a solution to social dilemmas.

Reviewer comment (Experiments):
In the conclusion, the authors state that one of the aims is improving the scalability and non-exploitability of agents. However, it is unclear about the practicalities of such a method in larger scale games as well as how it could ever be adapted beyond two-player games.

Author response:
Regarding the conclusion, the reviewer correctly points out that BRS is not a method that works for n-player games as it is stated in the limitations of the paper. However, because BRS allows for the use of neural network policies it can scale to more complex games that use higher dimensional representations of the environment.

Reviewer comment (Experiments):
Q1: How does the method compare to other approaches that introduce diversity into opponent modelling such as [1] and [2]?

Author response:
In BRS since the policies are non-linear, diversity is found using a replay buffer of policies and adding gaussian noise to the parameters of these policies. Which contrasts with the Determinantal Point Processes used in [1] and [2]. However, diversity seems to be not necessary for the Coin Game experiments as shown in the ablation section.

Reviewer comment (Theory):
Q2: Can the authors comment on whether or not (and how) this could be applied in games where the opponent’s rewards are unobserved?

Author response:
BRS cannot be applied to games where the reward for the other player remains unobserved unless that reward can be inferred from the behavior of the other player. Nonetheless, similar assumptions are done in the literature and used only during training; at deployment time the assumption is not necessary.

Reviewer comment (Evaluation):
Q3: Although the cooperation regularization method yields a useful tool that does not produce cooperative effects in zero-sum games, its effect in games in which in some game states the players ought to behave adversarially and in other game states cooperatively (for example the weakest link) is unclear to me. Can the authors shed any light on this?

Author response:
We do not know the answer to this question but we believe it is a very important one that should be explored in future work. Weakest link resembles Pure Diplomacy in the necessity for cooperation during the initial stages of the game and the need for competitive policies afterwards, and Pure Diplomacy has been one of the setups that we are interested in pursuing. We also believe BRS may work in a two player game with this structure, because the first gradient component (against the detective) induces adversarial behavior whereas the second one (self-play) induces cooperative behavior.