Paper: LOQA: Learning with Opponent Q-Learning Awareness (ICLR-2024)

Reviewer comment (Experiments):
The evaluation is limited in diversity, comparing LOQA to a single baseline algorithm (POLA) on a single environment (coin game). The selection of POLA vs alternative LOLA extensions is justified, but I am unsure how M-FOS is neglected when it has also been demonstrated to be effective at the coin game? Furthermore, the choice to only evaluate against POLA on the coin game is limiting, when full-history IPD and chicken would be equally interesting and strengthen the results.

Author response:
We agree with the reviewer in all of these points. In particular, we will incorporate M-FOS as a baseline for all of our experiments and add full history IPD and Chicken. We did not incorporate M-FOS initially because according to Figure 7 of the M-FOS paper the solution achieved is marginally better than a naive (PPO) agent.

Reviewer comment (Experiments):
Whilst the training dynamics of POLA are consistent across seeds, LOQA seems to have significant variance (Figure 4). The number of LOQA seeds should be increased beyond 3 to handle this variance.

Author response:
We will also run more LOQA seeds since we believe it is reasonable and doable before the rebuttal deadline.

Reviewer comment (Presentation):
Nitpick: The scaling of LOQA in figure 3 would be clearer with a log-scale y-axis.

Author response:
We will try a variation of Figure 3 using a log scale and keep it if it still clearly showcases the results.

Reviewer comment (Novelty):
Related work lacks discussion of MARL approaches to learning prosocial equilibria other than reciprocity-based or opponent-shaping-based, such as reward redistribution https://ala2020.vub.ac.be/papers/ALA2020_paper_45.pdf https://arxiv.org/abs/2004.13332, mediation https://arxiv.org/pdf/2306.08419.pdf, contracts https://arxiv.org/pdf/2208.10469.pdf, and similarity-based equilibria https://arxiv.org/pdf/2211.14468.pdf.

Author response:
We thank the reviewer for the suggested additions to the relevant literature, in particular we believe that similarity-based equilibria should be discussed. However, we want to emphasize that we are interested in the setup of solving social dilemmas without third parties (reward redistribution,  mediation), contracts, or any other changes to the underlying game.

Reviewer comment (Novelty):
The contribution is limited to modifying an existing algorithm and improving its speed, but not performance. The new algorithm is also quite complex. I do not think the contribution is sufficient for ICLR. A workshop would be a better fit.

Author response:
We respectfully disagree with the reviewer. LOQA does not belong to the LOLA family and as such is not a modification of an existing algorithm. We feel that LOQA is being held to a higher standard than existing methods in the literature. How is a novel method that improves the state of the art for neural network solutions to social dilemmas (both in terms of optimality and speed) only suited for a workshop, especially considering that previous publications (POLA, M-FOS) have been published to top machine learning venues? Also, we kindly request the reviewer to clarify the definition of 'complexity' in this context as we believe LOQA is relatively simple. In contrast to previous work, LOQA does not have inner outer loops, differentiation through optimization steps, and meta-games. It would be helpful if they could offer arguments to support the views that a) LOQA is more complex compared to existing methods, and b) why this kind of complexity might be considered undesirable.

Reviewer comment (Theory):
I am not sure why in 5.1 the Q-value $\hat{Q}^2$ is approximated as an empirical return. Since we need access to the real $Q^2$ or its approximation through opponent modeling regardless, can we not use a 1-step or n-step temporal difference instead (which would give a better bias-variance)?

Author response:
We experimented with bootstrapped versions of the $\hat{Q}^2$ estimate for many different lengths initially with worse results than using the full trajectory. We believe that, since the only differentiable parts of the estimates are the rewards, making the full trajectory differentiable works better for longer games.

Reviewer comment (Experiments):
In general, the experimental part has room for improvement. When this line of research on LOLA has a few prior works, a comparison with a decent amount of previous works is necessary so that we know the proposed method is better. The good performance of a particular method under a particular environment is not the reason to abandon other methods, especially when POLA [1] did not compare with M-FOS [2].

Author response:
First, we want to restate that there are not many existing methods that work in the Coin Game. That being said, we are only aware of POLA and M-FOS; LOLA has been shown to not work by the POLA authors. We will add an M-FOS comparison as we believe it is a valid and reasonable concern. We did not do so initially because we had reason to believe that the M-FOS results were marginally better than those of naive agents. We refer the reviewer to Figure 7. in the M-FOS paper, where the odds of taking their own coin reach 0.54 which demonstrates weak cooperation compared to LOQA and POLA.

Reviewer comment (Novelty):
I think the novelty in the proposed method is limited based on the papers cited by it. The key idea is that each agent model the opponents policy using the rewards obtained by the opponents.

Author response:
We believe that the reviewer may have missed the key idea of the paper, namely that the Q value of the opponent can be differentiated w.r.t. the agent’s policy parameters via reinforce. In such a way, it is a fundamentally different algorithm from those that we cite in the paper. This idea enables major computational savings. In contrast to previous works which compute explicit gradients w.r.t to optimization steps or model the problem as a meta-game (both of which are computationally expensive), LOQA uses reinforce estimators that are fast and easy to compute.

Reviewer comment (Writing):
Can the authors concretely define "partially competitive settings"?

Author response:
We agree with the reviewer that "partially competitive settings" is not clearly defined. We will replace this term with “social dilemmas”.

Reviewer comment (Theory):
What is the information structure assumed for each agent? If each agent can see the entire world state and also observe the actions of the other agent, then the policy for each agent should also depend on the history of the actions of the agents in addition to the current state for no loss of optimality.

Author response:
Our policies in the Coin Game are GRU policies that aggregate the entire observation history (without actions) up to the current time step through the hidden state propagation. Therefore, the agents condition on the entire observation history.

Reviewer comment (Theory):
How can the opponent's true Q function be replaces with an estimate by the agent? Are we assuming that opponent observations and rewards are common information in the game? This is often not the case in real-worls multi-agent settings.

Author response:
Yes, we assume that the rewards of both agents can be observed by both agents. This might not be the case in many settings, but we believe it is a reasonable assumption as previous works also make it.

Reviewer comment (Theory):
It is not clear if LOQA will extend to all general-sum games. Also, the sub-optimality for each agent due to the modified objective function needs to be quantified and bounded. Are the authors proposing LOQA only for sequential social dilemmas?

Author response:
The reviewer’s concern that LOQA may not generalize to all general sum games is valid, but we intended to demonstrate its usefulness only in social dilemmas.

Reviewer comment (Theory):
Can the authors establish or reason about the solution concept achieved using LOQA? Will it be a socially optimal solution. In non-symmetric games what will be the extent of sub-optimality for each agent?

Author response:
LOQA achieves a Nash equilibrium (tit-for-tat) on one-step history IPD as demonstrated by the experimental results. Beyond that it is difficult to formally characterize the solutions found by LOQA and it is not the scope of the paper.

Reviewer comment (Theory):
By modifying the objective function (returns) optimized by each agent, LOQA changes the underlying game. Can the authors show that a Nash equilibrium (or any refinement of it) of this modified game is a socially optimal solution of the original game?

Author response:
LOQA does not modify the returns optimized by each agent, so this statement is not true.

Reviewer comment (Theory):
Also, will decentralized REINFORCE algorithm lead to attaining a Nash equilibrium in the above-defined modified game?

Author response:
As stated in (6), there is no modification to the underlying game, thus we will not address this question.

Reviewer comment (Theory):
How does this analysis extend to the case when all agents in a game are using LOQA?

Author response:
Then again refer to (6).