Paper: Zero-Sum Positional Differential Games as a Framework for Robust Reinforcement Learning: Deep Q-Learning Approach (ICLR-2024)

Reviewer comment (Theory):
One point of confusion for me is in Section 3 where the authors state that "to solve differential games by RL algorithms, it is necessary to discretize them in time，we describe such a discretization...". If employing RL to address differential games still necessitates discretization, then why consider robust RL problems within the framework of differential games? Wouldn't it be more straightforward to tackle the issue directly within the framework of discrete robust MDPs? What additional advantages does considering robustness in differential games provide? Furthermore, how does the efficacy of using discretization to address differential games compare with directly solving a discrete robust MDP?

Author response:
We are not sure that there is a well-established concept of robust MDP. Basically, Markov games with pure or mixed policies are taken as such. We tried to describe the advantage of differential games compared with both of these options in the answer to the previous question. If you mean Markov games in pure policies described in Section 3 by robust MDP, then, as we noted above and in the paper (before Theorem 1), their main drawback is the non-existence of value (Nash equilibrium) in general case, which conceptually prevents the use of a shared q-function. However, as we show in Theorem 1, the fact that we consider not arbitrary games but the games which are a time-discretization of differential games satisfying Isaacs's condition allows us to theoretically justify that with a sufficiently small time-discretization step, the shared q-function can be used. If you meant something else by robust MDP, please feel free to clarify, and we will also be glad to discuss this.

If we understand your question correctly, then the answer is as follows. In fact, in our experiments, almost all the algorithms we consider aim to solve discrete-time RRL or MARL problems. 2xDDQN represents a naive decentralized learning approach often used in MARL in discrete time. RARL represents the decentralized approach that was used by Pinto et al. (2017) to solve RRL in discrete time. MADDPG is a basic centralized algorithm for solving general MARLs in discrete time. MADQN is its development for the case of a discrete action space. NashDQN is an algorithm aimed at solving Markov games in discrete time. Only CounterDQN and our IDQN and DIDQN are algorithms for which it is theoretically essential that an environment is a discretization of some differential game (in continuous time).

Reviewer comment (Theory):
In section 2, the authors could provide more insights into what is the difference between the standard formulation and differential game and why it is important to have a deterministic payoff. How the fact of the PDG is important to this paper and when the Isaacs's condition is met or not met.

Author response:
You are correct that this is a crucial issue, and we paid attention to it in the third paragraph of the Introduction section dedicated to Markov games. Following your recommendation, we have also added the following comments regarding this at the beginning of section 2: 

*Recent studies consider RRL problems within the framework of zero-sum Markov games in pure or mixed policies. In the case of pure policies, it is known (e.g., paper-rock-scissors) that Markov games may not have a value (Nash equilibrium), which conceptually prevents the development of centralized learning algorithms based on shared q-functions. In the case of mixed policies, such formalization may also be inappropriate if, according to the problem statement (for example, in the case of developing expensive or safe control systems), it is required to seek robust policies guaranteeing a deterministic payoff value. In this section, we describe the framework of the positional differential games, which allows us, on the one hand, to consider the pure agents' policies and deterministic values of payoffs and, on the other hand, to obtain the fact of the existence of a value in a reasonably general case.*

Thank you for this question. We discussed the importance of the fact that the Markov game from Section 3 is a time-discretization of some differential game before Theorem 1. This is what helps us to prove it. We also tried to highlight the importance of the fulfilment and non-fulfilment of Isaacs's condition in the Limitations section. Following your recommendations, we also added the following short comments at the end of section 2:

*We also note that in order to obtain further results, it is essential not only the existence of a value but also the fulfilment of Isaacs's condition as such.*

Reviewer comment (Evaluation):
In the experiment section, the metric of 'stability' appears without proper definition and explanation. For example, is 'stability' equivalent to 'deterministic payoff'? If so, why not just stick to the latter?

Author response:
Thank you for this comment. We mean stability with respect to running. That is, we say the algorithm is stable if it shows a similar result in all 5 runnings. In Figure 3, this means a narrow pale bar. The algorithm is unstable if, in some of the runnings, the algorithm performs worse than in the others. This corresponds to a wide pale bar. For example, MADDPG on Swimmer is unstable because, on at least one of the runnings, it showed a wide bar. We have added additional clarification to the article, chenging 

*Thus, looking at such a visualization, we can make conclusions about the stability*

to

*Thus, looking at such a visualization, we can make conclusions about the stability (with respect to running)*

and

*which reflects, on the one hand, the potential ability of MADDPG to find policies close to optimal, but, on the other hand, its instability.*

to

*which reflects, on the one hand, the potential ability of MADDPG to find policies close to optimal, but, on the other hand, its instability with respect to running.*

Reviewer comment (Evaluation):
In the experiment section again, some analysis seems to be out of place and is unclear how it is related to the topic. For example, "it is more efficient for agents..." What is efficiency exactly, and how is it compared across CounterDQN and MADQN?

Author response:
Let us clarify how the experimental analysis relates to the topic of the paper. The main goal of the paper is to train policies that are robust with respect to the opponent's actions. In our experiments, we measure precisely this robustness. To do this, we consider a comprehensive evaluation scheme (the details of which can be found in the Experiments section) and offer a corresponding visualization.

Thank you for this question, which prompted us to insert additional explanations into the paper. By efficiency, we mean the exploitability of trained agents, that is, the difference between the obtained approximations of their guaranteed results $V_u^{\pi_u}$ and $V_v^{\pi_v}$ describing the distance to a value (Nash equilibrium). In Figure 3, this exploitability is expressed in bar width. The smaller the bar width, the better the algorithm efficiency. We have supplemented the paper with a brief explanation of this at the end of the Evaluation scheme paragraph:

*The width of the bars illustrates the exploitability of both agents, that is, the difference between the obtained approximations of $V_u^{\pi_u}$ and $V_v^{\pi_v}$. If they are close to the optimal guaranteed results $V_u$  and $V_v$, then the width should be close to zero (if a value exists ($V_u = V_v$)). Thus, looking at such a visualization, we can make conclusions about the stability (with respect to running) and efficiency (with respect to exploitability) of the algorithms.*

Reviewer comment (Theory):
The reasoning and motivation for using positional differential games instead of Markov games as the framework are unclear. Markov games can be deterministic.

Author response:
You are right that Markov games can be deterministic. We discussed why their consideration would not lead to a shared q-function in Section 3 before Theorem 1. Namely, arbitrary deterministic Markov games may have no value (or Nash equilibrium) (for example, in rock-paper-scissors). The feature of the class of deterministic Markov games that we are considering is that they are a time-discretization of differential games for which the Isaacs condition is satisfied. This allows us to obtain the result of Theorem 1 and derive effective algorithms. We also further covered this issue in the first paragraph of section 2.

Reviewer comment (Theory):
The discretization of action space makes it unclear under what conditions the main theorem still holds.

Author response:
Thank you for your comment. We have added the following explanation in Corollary 1 (formerly Remark): 

*Let Isaacs's condition (6) holds. Let the value function $V(\tau,w)$ be continuously differentiable at every $(\tau,w) \in [0,T] \times \mathbb R^n$.*

Reviewer comment (Theory):
The effects of time discretization are unclear. How does it affect the estimation error or the convergence?

Author response:
From the results of Theorem 1 (a), it can be shown that the smaller the time-discretization step $d(\Delta)$, the smaller the error $\varepsilon$. This can be proved by analogy with how the definition of the limit by sequences is obtained from $(\varepsilon,\delta)$-definition of the limit. A specific estimate, in our case, can be obtained from the proof of the theorem, but it is unlikely to be of practical interest since it will typically depend on the continuity property of the right-hand side of dynamical system (1) and quality index (2).

Reviewer comment (Writing):
In Equ. 3, should it be $t_{m+1}$ instead of $\tau_{m+1}$?

Author response:
Thank you very much for this note. This is indeed a typo, which we have corrected in the updated version of the paper.

Reviewer comment (Writing):
In Section 5, when the environments is described, what is the difference between $R$ and $\mathbb{R}$?

Author response:
Thank you very much for your note. It's a typo. There should be $\mathbb{R}^n$ everywhere. We corrected this in the updated version of the paper.

Reviewer comment (Reproducibility):
What are the specifications of the system used to run the experiments?

Author response:
In our experiments, we use both well-known examples of differential games and examples based on the MuJoCo simulator. A brief description of them is given in the Environments paragraph of the Experiments section. A detailed mathematical description is given in Appendix E. If you have additional questions regarding it, we will be glad to answer them.

Reviewer comment (Presentation):
I think you should name your main theorem. At the moment, you referred to it as "Theorem" and I think it would help if you name it. Either Theorem 1, Main Theorem, or some other name. Similarly, I would enumerate the remark on page 5 and maybe call it a corollary.

Author response:
Thank you for this comment. Indeed, it is better to call the theorem as "Theorem 1", and the remark as "Corollary 1".