Paper: Byzantine Robust Cooperative Multi-Agent Reinforcement Learning as a Bayesian Game (ICLR-2024)

Reviewer comment (Presentation):
My main concern with the paper is the lack of self-containment: The paper excessively references the appendix, which hurts readability (it is not convenient to peek into the appendix for almost every paragraph of the main paper).

Author response:
We appreciate this valuable feedback. To address this, we have:

* Removed unnecessary discussions, reducing references to the appendix from 17 times to 8 times and decreasing the appendix length from 21 pages to 12 pages. We have meticulously revised the manuscript to ensure it remains focused on the core message. For completeness, we will host a GitHub page for additional experiment results upon acceptance.

* Included a concise proof sketch for each theorem to enhance self-containment.

* Reduced the use of mathematical notations and added more textual explanations.

Please refer to our global response and the revised manuscript for more details.

Reviewer comment (Presentation):
The paper needs a clear prioritization of what is really important and what can be safely left in the appendix. For example, important proofs can be sketched informally with little space, but delegating them to the appendix completely is not acceptable.

Author response:
We appreciate this valuable feedback. To address this, we have:

* Removed unnecessary discussions, reducing references to the appendix from 17 times to 8 times and decreasing the appendix length from 21 pages to 12 pages. We have meticulously revised the manuscript to ensure it remains focused on the core message. For completeness, we will host a GitHub page for additional experiment results upon acceptance.

* Included a concise proof sketch for each theorem to enhance self-containment.

* Reduced the use of mathematical notations and added more textual explanations.

Please refer to our global response and the revised manuscript for more details.

Reviewer comment (Presentation):
Before checking if I can raise my score, I need a revised version of the paper that is more self-contained, e.g., where the number of appendix references is significantly reduced and proofs are at least sketched such that the reader is not forced to switch documents all the time.

Author response:
We appreciate this valuable feedback. To address this, we have:

* Removed unnecessary discussions, reducing references to the appendix from 17 times to 8 times and decreasing the appendix length from 21 pages to 12 pages. We have meticulously revised the manuscript to ensure it remains focused on the core message. For completeness, we will host a GitHub page for additional experiment results upon acceptance.

* Included a concise proof sketch for each theorem to enhance self-containment.

* Reduced the use of mathematical notations and added more textual explanations.

Please refer to our global response and the revised manuscript for more details.

Reviewer comment (Theory):
In Dec-POMDPs (Section 2.1), policies select their actions by the history of past observations and actions. Conditioning only on observations is insufficient since they are not Markovian. The definition in Section 2.2 is correct, though.

Author response:
Thank you for pointing out the concern. Please allow us to clarify that in our original draft, for both defender and adversary, their policies are conditioned on histories $H$, not observations $o$. For example:

* Section. 2.1, paragraph 2, "...each agent $i$ selects its action $a_t^i \in \mathcal A^i$ using its policy $\pi^i(\cdot|H_i^t)$"; "the goal of all agents is to learn a joint policy ... that maximize the long-term return "$J(\mathbf{\pi}) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^t r_t|s_0, \mathbf{a}_t \sim \pi(\cdot|H_t)\right]$".
* Section. 2.2, paragraph 2, "replaced by action $\hat{a}^i_t$ sampled from an adversary with policy $\hat{\pi}^i(\cdot|H_t^i, \theta)$"; "the value function can be defined as $V_\theta(s) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^t r_t|s_0=s, \mathbf{a}_t \sim \pi(\cdot|H_t), \hat{\mathbf{a}}_t \sim \hat{\pi}(\cdot|H_t, \theta)\right]$".

A thorough review of our paper confirms that the policy of both defenders and adversaries rely on histories, not observations.

Reviewer comment (Writing):
Line 119: “was replaced” - “**is** replaced”

Author response:
Thank you for your suggestions. We have meticulously revised the manuscript to correct grammatical errors and enhanced the clarity of Figure 2 in our revised submission.

Reviewer comment (Writing):
Line 277: “four type of threats” - “four type**s** of threats”

Author response:
Thank you for your suggestions. We have meticulously revised the manuscript to correct grammatical errors and enhanced the clarity of Figure 2 in our revised submission.

Reviewer comment (Presentation):
The text size in Figure 2 is too small for printed versions.

Author response:
Thank you for your suggestions. We have meticulously revised the manuscript to correct grammatical errors and enhanced the clarity of Figure 2 in our revised submission.

Reviewer comment (Theory):
Why do adversaries condition on observations despite them not being Markovian (in contrast to the non-adversarial policies)? Is this intentional or a mistake?

Author response:
Thank you for pointing out the concern. Please allow us to clarify that in our original draft, for both defender and adversary, their policies are conditioned on histories $H$, not observations $o$. For example:

* Section. 2.1, paragraph 2, "...each agent $i$ selects its action $a_t^i \in \mathcal A^i$ using its policy $\pi^i(\cdot|H_i^t)$"; "the goal of all agents is to learn a joint policy ... that maximize the long-term return "$J(\mathbf{\pi}) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^t r_t|s_0, \mathbf{a}_t \sim \pi(\cdot|H_t)\right]$".
* Section. 2.2, paragraph 2, "replaced by action $\hat{a}^i_t$ sampled from an adversary with policy $\hat{\pi}^i(\cdot|H_t^i, \theta)$"; "the value function can be defined as $V_\theta(s) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^t r_t|s_0=s, \mathbf{a}_t \sim \pi(\cdot|H_t), \hat{\mathbf{a}}_t \sim \hat{\pi}(\cdot|H_t, \theta)\right]$".

A thorough review of our paper confirms that the policy of both defenders and adversaries rely on histories, not observations.

Reviewer comment (Evaluation):
Do you have any intuition on why M3DDPG performs so poorly in the presence of an adversary compared to the standard MAPPO without any robustness mechanism in Figure 4?

Author response:
There are two key reasons for M3DDPG’s underperformance in adversarial contexts, as compared to standard MAPPO without robustness mechanisms.

First, as discussed in Proposition 2.2, the existence of a pure-strategy equilibrium (RMPBE) isn't guaranteed. M3DDPG, which derives from MADDPG, inherently generates deterministic policies. Such policies may not be optimal in robust cooperative MARL contexts where a pure-strategy equilibrium is not assured. Despite this, M3DDPG was included as a baseline due to its recognition in the field.

Second, as a part of the max entropy RL family, MAPPO employs stochastic policies that encourage diverse solution strategies, thereby enhancing resilience to perturbations. This advantage of max entropy policies is well-documented in several studies [1, 2, 3, 4].

[1] Ziebart, Brian D. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Ph.D. Thesis, Carnegie Mellon University, 2010.

[2] Haarnoja, Tuomas, et al. "Reinforcement learning with deep energy-based policies." ICML 2017.

[3] Haarnoja, Tuomas, et al. "Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor." ICML 2018.

[4] Eysenbach, Benjamin, and Sergey Levine. "Maximum entropy rl (provably) solves some robust rl problems." ICLR 2022.

Reviewer comment (Theory):
The fact that convergence cannot be guaranteed despite the assumptions made is a weakness of the paper. The method proposed is complex, and the assumptions are strong, yet there is no guarantee of convergence.

Author response:
Thank you for pointing out the concern. Firstly, please allow us to clarify that we have established convergence in our original draft. As stated on page 6, Section 3.2, "Using common assumptions in stochastic approximation proposed by Borkar (Borkar, 1997; Borkar & Meyn, 2000; Borkar, 2009), we show that Theorem 3.1 converges almost surely to ex interim robust Markov perfect Bayesian equilibrium, with detailed assumptions and proof deferred to Appendix. A.8". This proof leverages stochastic approximation theory and is underscored in both our abstract and introduction.

Secondly, the term "almost sure" convergence implies that the sequence of defender and attacker policies converge with probability 1 as the number of iterations goes to infinity.

Thirdly, for greater clarity, we have added a bolded subtitle "Convergence" in Section 3.2 and restructured the relevant paragraph.

Fourthly, while our approach and methodology indeed rely on specific assumptions and mathematical techniques, the implementation of our algorithm is practical and straightforward. It involves conditioning the policy and value function on the belief of the current type, combined with a two-timescale update strategy (i.e., faster policy updates for the adversary and slower for the defender).

Reviewer comment (Presentation):
The proofs are all in the Appendix, which makes the paper quite long, too long for a conference paper, and that makes it appear incomplete. Even the analysis of two of the environments used for the experiments is in the Appendix. Expecting the reviewers to read a paper of 36 pages is too much for a conference. Without the Appendices the paper is incomplete.

Author response:
We appreciate this valuable feedback. To address this, we have:

* Removed unnecessary discussions, reducing references to the appendix from 17 times to 8 times and decreasing the appendix length from 21 pages to 12 pages. We have meticulously revised the manuscript to ensure it remains focused on the core message. For completeness, we will host a GitHub page for additional experiment results upon acceptance.

* Included a concise proof sketch for each theorem to enhance self-containment.

* Reduced the use of mathematical notations and added more textual explanations.

Please refer to our global response and the revised manuscript for more details.

Reviewer comment (Writing):
The paper is hard to read, it assumes significant knowledge of the field and is full of acronyms and citations that break the flow of the text.

Author response:
Thank you for this suggestion. In our revised version, we have carefully go through the whole paper and reduced the use of mathematical notations and added more textual explanations. However, it's important to note that certain expressions equations cannot be overly simplified for correctness.

Reviewer comment (Presentation):
I still object to the length of the paper even after the discussion and changes made to it.

Author response:
We appreciate this valuable feedback. To address this, we have:

* Removed unnecessary discussions, reducing references to the appendix from 17 times to 8 times and decreasing the appendix length from 21 pages to 12 pages. We have meticulously revised the manuscript to ensure it remains focused on the core message. For completeness, we will host a GitHub page for additional experiment results upon acceptance.

* Included a concise proof sketch for each theorem to enhance self-containment.

* Reduced the use of mathematical notations and added more textual explanations.

Please refer to our global response and the revised manuscript for more details.

Reviewer comment (Presentation):
The document frequently refers to its appendix, suggesting that it does not stand alone as effectively as it could.

Author response:
We appreciate this valuable feedback. To address this, we have:

* Removed unnecessary discussions, reducing references to the appendix from 17 times to 8 times and decreasing the appendix length from 21 pages to 12 pages. We have meticulously revised the manuscript to ensure it remains focused on the core message. For completeness, we will host a GitHub page for additional experiment results upon acceptance.

* Included a concise proof sketch for each theorem to enhance self-containment.

* Reduced the use of mathematical notations and added more textual explanations.

Please refer to our global response and the revised manuscript for more details.

Reviewer comment (Writing):
To enhance the feeling of a complete and self-contained work, the authors should further take simplification, such as using simpler terms within the equations for greater clarity.

Author response:
We appreciate this valuable feedback. To address this, we have:

* Removed unnecessary discussions, reducing references to the appendix from 17 times to 8 times and decreasing the appendix length from 21 pages to 12 pages. We have meticulously revised the manuscript to ensure it remains focused on the core message. For completeness, we will host a GitHub page for additional experiment results upon acceptance.

* Included a concise proof sketch for each theorem to enhance self-containment.

* Reduced the use of mathematical notations and added more textual explanations.

Please refer to our global response and the revised manuscript for more details.

Reviewer comment (Evaluation):
How can the proposed method benefit real-world applications? Any concrete examples?

Author response:
Let's illustrate this by examples. In robot swarm control [1, 2], individual robots may behave unpredictably due to malfunctions or external interference [3, 4]. Our approach studies the lower bound for these scenarios where a swarm faces a worst-case, non-oblivious adversary, as discussed in our introduction.

Apart from robot swarms, in traffic light control [5], individual lights might malfunction or be compromised, leading to a non-cooperative system. Similarly, in power grid control [6], nodes might fail or transmit erroneous signals. Our cooperative MARL approach is designed to adaptively respond to these challenges, ensuring efficient operation of these systems during deployment. We plan to apply our algorithm in these domains as future work, as discussed in our conclusion section.

[1] Hüttenrauch, Maximilian, Sosic Adrian, and Gerhard Neumann. "Deep reinforcement learning for swarm systems." JMLR 2019.

[2] Batra, Sumeet, et al. "Decentralized control of quadrotor swarms with end-to-end deep reinforcement learning." CoRL 2022.

[3] Giray, Sait Murat. "Anatomy of unmanned aerial vehicle hijacking with signal spoofing." RAST 2013.

[4] Ly, Bora, and Romny Ly. "Cybersecurity in unmanned aerial vehicles (UAVs)." Journal of Cyber Security Technology 2021.

[5] Chu, Tianshu, et al. "Multi-agent deep reinforcement learning for large-scale traffic signal control." IEEE TITS 2019.

[6] Xi, Lei, et al. "Smart generation control based on multi-agent reinforcement learning with the idea of the time tunnel." Energy 2018.