Original: E VALUATING R OBUSTNESS OF C OOPERATIVE MARL
Revised: E VALUATING R OBUSTNESS OF C OOPERATIVE MARL

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In recent years, a proliferation of methods were developed for multi-agent reinforcement learning (MARL).
Revised: In recent years, a proliferation of methods were developed for multi-agent reinforcement learning (MARL).

Original: In this paper, we focus on evaluating the robustness of MARL agents in continuous control tasks.
Revised: In this paper, we focus on evaluating the robustness of MARL agents in continuous control tasks.

Original: In particular, we propose the ﬁrst model-based approach to perform adversarial attacks for continuous MARL.
Revised: In particular, we propose the ﬁrst model-based approach to perform adversarial attacks for continuous MARL.

Original: The attack aims at adversarially perturbing the states of agent(s) to misguide them to take damaging actions that lower team rewards.
Revised: The attack aims at adversarially perturbing the states of agent(s) to misguide them to take damaging actions that lower team rewards.

Original: A deep neural network is trained to represent the dynamics of the environment.
Revised: A deep neural network is trained to represent the dynamics of the environment.

Original: We then solve an optimization problem with the learned dynamic model to yield small perturbations.
Revised: We then solve an optimization problem with the learned dynamics model to yield small perturbations.

Original: In addition, we discuss several strategies to optimally select adversary agents for the attack.
Revised: In addition, we discuss several strategies to optimally select adversary agents for the attack.

Original: Numerical experiments on multi-agent Mujoco tasks verify the effectiveness of our proposed approach.
Revised: Numerical experiments on multi-agent Mujoco tasks verify the effectiveness of our proposed approach.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep neural networks are known to be vulnerable to adversarial examples, where a small and often imperceptible adversarial perturbation can easily fool the state-of-the-art deep neural network classiﬁers (Szegedy et al., 2013; Nguyen et al., 2015; Goodfellow et al., 2014; Papernot et al., 2016).
Revised: Deep neural networks are known to be vulnerable to adversarial examples, where a small and often imperceptible adversarial perturbation can easily fool the state-of-the-art deep neural network classiﬁers (Szegedy et al., 2013; Nguyen et al., 2015; Goodfellow et al., 2014; Papernot et al., 2016).

Original: Since then, a wide variety of deep learning tasks have been shown to also be vulnerable to adversarial attacks, ranging from various computer vision tasks to natural language processing tasks (Jia &
Revised: Since then, a wide variety of deep learning tasks have been shown to also be vulnerable to adversarial attacks, ranging from various computer vision tasks to natural language processing tasks (Jia &

Original: Liang, 2017; Zhang et al., 2020b; Jin et al., 2020; Alzantot et al., 2018).
Revised: Liang, 2017; Zhang et al., 2020b; Jin et al., 2020; Alzantot et al., 2018).

Original: Perhaps unsurprisingly, deep reinforcement learning (DRL) agents are also vulnerable to adversarial attacks, as ﬁrst shown in Huang et al.
Revised: Perhaps unsurprisingly, deep reinforcement learning (DRL) agents are also vulnerable to adversarial attacks, as ﬁrst shown in Huang et al.

Original: (2017) for atari games DRL agents.
Revised: (2017) for atari games DRL agents.

Original: In Huang et al.
Revised: In Huang et al.

Original: , the authors study the effectiveness of adversarial examples on a policy network trained on Atari games under the situation where the attacker has access to the neural network of  target policy.
Revised: , the authors study the effectiveness of adversarial examples on a policy network trained on Atari games under the situation where the attacker has access to the neural network of the target policy.

Original: In Lin et al.
Revised: In Lin et al.

Original: (2017), the authors further investigate a strategically-time attack by attacking trained agents on
Revised: (2017), the authors further investigate a strategically-time attack by attacking trained agents on

Original: Atari games at a subset of the time step.
Revised: Atari games at a subset of the time step.

Original: Meanwhile, Kos & Song (2017) use the fast gradient sign method (FGSM) to generate adversarial perturbation on the A3C algorithm (Mnih et al., 2016) and explore training with random noise and FGSM perturbation to improve resilience against adversarial examples.
Revised: Meanwhile, Kos & Song (2017) use the fast gradient sign method (FGSM) to generate adversarial perturbation on the A3C algorithm (Mnih et al., 2016) and explore training with random noise and FGSM perturbation to improve resilience against adversarial examples.

Original: While the above research endeavors focused on actions that take discrete values, another line of research tackle a more challenging problem on DRL with continuous action space (Weng et al., 2019; Gleave et al., 2019).
Revised: While the above research endeavors focused on actions that take discrete values, another line of research tackles a more challenging problem on DRL with continuous action spaces (Weng et al., 2019; Gleave et al., 2019).

Original: Speciﬁcally, Weng et al.
Revised: Speciﬁcally, Weng et al.

Original: (2019) consider a two-step algorithm which determines adversarial perturbation to be closer to a target state using a learnt dynamic model, and Gleave et al.
Revised: (2019) consider a two-step algorithm which determines adversarial perturbation to be closer to a target state using a learnt dynamics model, and Gleave et al.

Original: (2019) propose a physically realistic thread model and demonstrates the existence of adversarial policies in zero-sum simulated robotics games.
Revised: (2019) propose a physically realistic thread model and demonstrates the existence of adversarial policies in zero-sum simulated robotics games.

Original: While most of the existing DRL attack algorithms focus on the single DRL agent setting 1 , in this work we propose to study the vulnerability of multi-agent DRL, which has been widely applied in many safety-critical real-world applications including swarm robotics (Dudek et al., 1993), electricity distribution, and trafﬁc control (OroojlooyJadid & Hajinezhad, 2019).
Revised: While most of the existing DRL attack algorithms focus on the single DRL agent setting 1 , in this work we propose to study the vulnerability of multi-agent DRL, which has been widely applied in many safety-critical real-world applications including swarm robotics (Dudek et al., 1993), electricity distribution, and trafﬁc control (OroojlooyJadid & Hajinezhad, 2019).

Original: In particular, we focus on the collaborative multi-agent reinforcement learning (c-MARL) setting, where a group of agents is trained to generate joint actions to maximize certain (team) reward function.
Revised: In particular, we focus on the collaborative multi-agent reinforcement learning (c-MARL) setting, where a group of agents is trained to generate joint actions to maximize certain (team) reward function.

Original: We note that c-MARL is a more challenging yet interesting setting than the single DRL agent setting, as the interactions between agents commands consideration of additional layers of complications.
Revised: We note that c-MARL is a more challenging yet interesting setting than the single DRL agent setting, as the interactions between agents commands consideration of additional layers of complications.

Original: Contribution.
Revised: Contribution.

Original: Our contribution can be summarized as follows:
Revised: Our contribution can be summarized as follows:

Original: • We propose a new adversarial attack framework on MARL with continuous action space, where we name it cMBA
Revised: • We propose a new adversarial attack framework on MARL with continuous action space, where we name it cMBA

Original: (model-based attack on c-MARL).
Revised: (model-based attack on c-MARL).

Original: The attack comprises two steps: learning a representation of the environment dynamics and then determining proper observation perturbations based on the learned dynamic model • We formulate the process of selecting an victim agent as a mixed-integer programming problem and propose an approximate formulation that can be efﬁciently solved by a ﬁrstorder method.
Revised: The attack comprises two steps: learning a representation of the environment dynamics and then determining proper observation perturbations based on the learned dynamics model. • We formulate the process of selecting an victim agent as a mixed-integer programming problem and propose an approximate formulation that can be efﬁciently solved by a ﬁrstorder method.

Original: • We evaluate our attack framework with two baselines and four multi-agent MuJoCo tasks (Peng et al., 2020), with number of agents ranging from 2 to 6, to verify the effectiveness of our approach.
Revised: • We evaluate our attack framework with two baselines and four multi-agent MuJoCo tasks (Peng et al., 2020), with number of agents ranging from 2 to 6, to verify the effectiveness of our approach.

Original: Our model-based attack consistently outperforms the two baselines in all tested environments.
Revised: Our model-based attack consistently outperforms the two baselines in all tested environments.

Original: Paper outline.
Revised: Paper outline.

Original: Section 2 discusses related works in adversarial attacks for MARL along with the problem settings.
Revised: Section 2 discusses related works in adversarial attacks for MARL along with the problem settings.

Original: We describe the main attack framework cMBA in Section 3, including the training of the dynamic model and solving the sub-optimization problem.
Revised: We describe the main attack framework cMBA in Section 3, including the training of the dynamics model and solving the sub-optimization problem.

Original: In Section 3.3, we introduce a mixed-integer program to ﬁnd the optimal set of victim agents and propose an approximate formulation that can be efﬁciently solved using a ﬁrst-order optimization method.
Revised: In Section 3.3, we introduce a mixed-integer program to ﬁnd the optimal set of victim agents and propose an approximate formulation that can be efﬁciently solved using a ﬁrst-order optimization method.

Original: Section 4 presents the evaluation of our approach on several multi-agent MuJoCo environments.
Revised: Section 4 presents the evaluation of our approach on several multi-agent MuJoCo environments.

Original: Therein, we also study the performance of different variants of our model-based attack as discussed in Section 3.3.
Revised: Therein, we also study the performance of different variants of our model-based attack as discussed in Section 3.3.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Adversarial attacks for DRL agents.
Revised: Adversarial attacks for DRL agents.

Original: In the c-MARL setting, although there exists a signiﬁcant literature focusing on the adversarial training of MARL agents (Phan et al., 2020; 2021; Zhang et al., 2020a), yet a very few of them addresses the scenario of adversarial attacks in c-MARL.
Revised: In the c-MARL setting, although there exists a signiﬁcant literature focusing on the adversarial training of MARL agents (Phan et al., 2020; 2021; Zhang et al., 2020a), yet a very few of them addresses the scenario of adversarial attacks in c-MARL.

Original: Lin et al.
Revised: Lin et al.

Original: (2020) propose a two-step attack on a single agent in a c-MARL environment, where they extend existing methods to generate adversarial examples guided by a learned adversarial policy.
Revised: (2020) propose a two-step attack on a single agent in a c-MARL environment, where they extend existing methods to generate adversarial examples guided by a learned adversarial policy.

Original: The attack in Lin et al.
Revised: The attack in Lin et al.

Original: (2020) is evaluated under the StarCraft Multi-Agent Challenge (SMAC) environment (Samvelyan et al., 2019) where the action spaces are discrete.
Revised: (2020) is evaluated under the StarCraft Multi-Agent Challenge (SMAC) environment (Samvelyan et al., 2019) where the action spaces are discrete.

Original: To the best of our knowledge, there has not been work considering adversarial attacks on the c-MARL setting with continuous action spaces.
Revised: To the best of our knowledge, there has not been work considering adversarial attacks on the c-MARL setting with continuous action spaces.

Original: The closest approach to ours is the enchanting attack in Lin et al.
Revised: The closest approach to ours is the enchanting attack in Lin et al.

Original: (2017) and the modelbased attack in Weng et al.
Revised: (2017) and the modelbased attack in Weng et al.

Original: (2019); however, these two works focused on single DRL agent setting.
Revised: (2019); however, these two works focused on single DRL agent setting.

Original: The enchanting attack in Lin et al.
Revised: The enchanting attack in Lin et al.

Original: (2017) uses a video prediction model to predict the future state and ﬁnds perturbation that minimizes a distance function between the predicted future state and a target state.
Revised: (2017) uses a video prediction model to predict the future state and ﬁnds perturbation that minimizes a distance function between the predicted future state and a target state.

Original: Similarly, Weng et al.
Revised: Similarly, Weng et al.

Original: (2019) consider two formulations for observation and action perturbation that incorporates a learned dynamic model for future state prediction.
Revised: (2019) consider two formulations for observation and action perturbation that incorporates a learned dynamic model for future state prediction.

Original: In addition to the model-based approach, in this paper, we consider the selection of victim agents, which is unique to the multi-agent DRL setting.
Revised: In addition to the model-based approach, in this paper, we consider the selection of victim agents, which is unique to the multi-agent DRL setting.

Original: Training c-MARL agents.
Revised: Training c-MARL agents.

Original: One simple approach to train c-MARL agents is to let each agent learn their own policy independently as in Independent Q-Learning (IQL)
Revised: One simple approach to train c-MARL agents is to let each agent learn their own policy independently as in Independent Q-Learning (IQL)

Original: (Tan, 1993; Tampuu et al., 2017).
Revised: (Tan, 1993; Tampuu et al., 2017).

Original: However, this training strategy does not capture the interaction between agents.
Revised: However, this training strategy does not capture the interaction between agents.

Original: Alternatively, one can follow a common paradigm called centralized training with decentralized execution (CTDE) (Oliehoek et al., 2008; Oliehoek & Amato, 2016; Kraemer & Banerjee, 2016).
Revised: Alternatively, one can follow a common paradigm called centralized training with decentralized execution (CTDE) (Oliehoek et al., 2008; Oliehoek & Amato, 2016; Kraemer & Banerjee, 2016).

Original: Gupta et al.
Revised: Gupta et al.

Original: (2017) extend DQN (Mnih et al., 2015), TRPO (Schulman et al., 2015), and DDPG (Lillicrap et al., 2015) to c-MARL, where a parameter sharing neural network policy is trained in a centralized fashion but still allows decentralized behavior across agents.
Revised: (2017) extend DQN (Mnih et al., 2015), TRPO (Schulman et al., 2015), and DDPG (Lillicrap et al., 2015) to c-MARL, where a parameter sharing neural network policy is trained in a centralized fashion but still allows decentralized behavior across agents.

Original: Similar to this, COMA (Foerster et al., 2018) is an actor-critic method that uses a centralized critic to estimate counterfactual advantage for decentralized policies.
Revised: Similar to this, COMA (Foerster et al., 2018) is an actor-critic method that uses a centralized critic to estimate counterfactual advantage for decentralized policies.

Original: While COMA uses a single critic for all agents, MADDPG (Lowe et al., 2017) learns a centralized critic for each agent and evaluates on continuous control.
Revised: While COMA uses a single critic for all agents, MADDPG (Lowe et al., 2017) learns a centralized critic for each agent and evaluates on continuous control.

Original: Instead of having fully centralized critic, DOP (Wang et al., 2020b) and FACMAC (Peng et al., 2020) apply the value decomposition idea (Sunehag et al., 2017; Rashid et al., 2018; 2020; Son et al., 2019; Wang et al., 2020a) into multi-agent actor-critic by having a centralized but factorized critic.
Revised: Instead of having fully centralized critic, DOP (Wang et al., 2020b) and FACMAC (Peng et al., 2020) apply the value decomposition idea (Sunehag et al., 2017; Rashid et al., 2018; 2020; Son et al., 2019; Wang et al., 2020a) into multi-agent actor-critic by having a centralized but factorized critic.

Original:  C MBA: M ODEL - BASED ATTACK FOR C -MARL 3. P ROBLEM SETTING
Revised:  C MBA: M ODEL - BASED ATTACK FOR C -MARL 3. P ROBLEM SETTING

Original: We consider multi-agent tasks with continuous action spaces modeled as a Decentralized Partially
Revised: We consider multi-agent tasks with continuous action spaces modeled as a Decentralized Partially

Original: Observable Markov Decision Process (Dec-POMDP) (Oliehoek & Amato, 2016).
Revised: Observable Markov Decision Process (Dec-POMDP) (Oliehoek & Amato, 2016).

Original: A Dec-POMDP has a ﬁnite set of agents N ≡ { 1 , · · · , n } associated with a set of states S describing global states, a set of continuous actions
Revised: A Dec-POMDP has a ﬁnite set of agents N ≡ { 1 , · · · , n } associated with a set of states S describing global states, a set of continuous actions

Original: A i , and a set of individual partial observation O i for each agent i ∈ N .
Revised: A i , and a set of individual partial observation O i for each agent i ∈ N .

Original: Given the current observation o it ∈
Revised: Given the current observation o it ∈

Original: O i , the action a it ∈
Revised: O i , the action a it ∈

Original: A i is selected by a parametrized policy π θ i : O i
Revised: A i is selected by a parametrized policy π θ i : O i

Original: → A i .
Revised: → A i .

Original: The next state agent i moves to is determined by the state transition function
Revised: The next state agent i moves to is determined by the state transition function

Original: P i : S × A i → S .
Revised: P i : S × A i → S .

Original: After that, agent i receives a reward r it calculated from a reward function
Revised: After that, agent i receives a reward r it calculated from a reward function

Original: R i : S × A i
Revised: R i : S × A i

Original: → R and observes a local observation o it +1 ∈
Revised: → R and observes a local observation o it +1 ∈

Original: O i correlated with the global state s t +1 ∈ S .
Revised: O i correlated with the global state s t +1 ∈ S .

Original: In addition, Dec-POMDP is associated with an initial state distribution P 0 and a discount factor γ .
Revised: In addition, Dec-POMDP is associated with an initial state distribution P 0 and a discount factor γ .

Original: In this setting, each agent tries to ﬁnd the policy that maximizes its own total expected return
Revised: In this setting, each agent tries to ﬁnd the policy that maximizes its own total expected return

Original: R i =
Revised: R i =

Original: (cid:80) Tt =

Revised: (cid:80) Tt =

Original: γ t r it where T is the sampling horizon.
Revised: γ t r it where T is the sampling horizon.

Original: 3. P ROBLEM FORMULATION
Revised: 3. P ROBLEM FORMULATION

Original: Our goal is to generate adversarial perturbations imposed to the victim agents’ input (state) in order to deteriorate the total team reward.
Revised: Our goal is to generate adversarial perturbations imposed to the victim agents’ input (state) in order to deteriorate the total team reward.

Original: The added noise encourages the victim agents’ observation to be close to a desired adversarial state corresponding to low reward.
Revised: The added noise encourages the victim agents’ observation to be close to a desired adversarial state corresponding to low reward.

Original: To avoid sampling from the environment, we use a pre-trained model that learns the dynamics of the environment to predict the next state from the perturbed observation and current action, then ﬁnd the suitable noise that minimizes the distance between the predicted next state and a predeﬁned target state.
Revised: To avoid sampling from the environment, we use a pre-trained model that learns the dynamics of the environment to predict the next state from the perturbed observation and current action, then ﬁnd the suitable noise that minimizes the distance between the predicted next state and a predeﬁned target state.

Original: This process can be formulated as an optimization problem as follows.
Revised: This process can be formulated as an optimization problem as follows.

Original: Formally, we consider a multi-agent setting with |N| = n agents, each agent i ∈ N observes state s it locally and takes action a it .
Revised: Formally, we consider a multi-agent setting with |N| = n agents, each agent i ∈ N observes state s it locally and takes action a it .

Original: Let s t =
Revised: Let s t =

Original: ( s 1 t , · · · , s nt ) ∈ S be the global state at time step t which is concatenated from local states s it for each agent i = 1 , · · · , n .
Revised: ( s 1 t , · · · , s nt ) ∈ S be the global state at time step t which is concatenated from local states s it for each agent i = 1 , · · · , n .

Original: We also denote the joint action a t =
Revised: We also denote the joint action a t =

Original: ( a 1 t , · · · , a nt ) concatenated from each agent’s action a it .
Revised: ( a 1 t , · · · , a nt ) concatenated from each agent’s action a it .

Original: Let L t ∈ N be the set of victim agents at time step t , i.e. the set of agents that can be attacked.
Revised: Let L t ∈ N be the set of victim agents at time step t , i.e. the set of agents that can be attacked.

Original: Let f d : S ×
Revised: Let f d : S ×

Original: A → S be a parametrized function that approximates the dynamics of the environment, where A is the set of concatenated states, one from each A i .
Revised: A → S be a parametrized function that approximates the dynamics of the environment, where A is the set of concatenated actions, one from each A i .

Original: We assume that a pre-trained MARL policy π ( · ) is given.
Revised: We assume that a pre-trained MARL policy π ( · ) is given.

Original: Let s target be the target state which can lead to poor performance to the agent.
Revised: Let s target be the target state which can lead to poor performance to the agent.

Original: We denote (cid:15) as an upper bound on budget constraint w.r.t some p -norm (cid:107)·(cid:107) p .
Revised: We denote (cid:15) as an upper bound on budget constraint w.r.t some p -norm (cid:107)·(cid:107) p .

Original: The state perturbation ∆ s = (∆ s 1 , · · · , ∆ s n ) (we suppress the dependence on t of ∆ s to avoid overloading the notation) to s t is the solution to the following problem
Revised: The state perturbation ∆ s = (∆ s 1 , · · · , ∆ s n ) (we suppress the dependence on t of ∆ s to avoid overloading the notation) to s t is the solution to the following problem

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where 0 is a zero vector, and the observation space is constrained within the (vectorized) intervals between (cid:96) S and u S .
Revised: where 0 is a zero vector, and the observation space is constrained within the (vectorized) intervals between (cid:96) S and u S .

Original: Let us ﬁrst provide some insights for the formulation (1).
Revised: Let us ﬁrst provide some insights for the formulation (1).

Original: For each agent, using the trained policy π , we can compute the corresponding action a it given its (possibly perturbed) local state s it or s it +∆ s i .
Revised: For each agent, using the trained policy π , we can compute the corresponding action a it given its (possibly perturbed) local state s it or s it +∆ s i .

Original: From the concatenated state-action pair ( s t , a t ) , we can predict the next state
Revised: From the concatenated state-action pair ( s t , a t ) , we can predict the next state

Original: ˆ s t +1 via the learned dynamic model f d .
Revised: ˆ s t +1 via the learned dynamics model f d .

Original: Then by minimizing the distance between the predicted next state and the target state under the budget constraint, we are forcing the agent to move closer to a damaging state in the next transition.
Revised: Then by minimizing the distance between the predicted next state and the target state under the budget constraint, we are forcing the agent to move closer to a damaging state in the next transition.

Original: We also note that the last equality constraint indicates that we only allow perturbing agents within the victim set L t .
Revised: We also note that the last equality constraint indicates that we only allow perturbing agents within the victim set L t .

Original: Problem (1) can be efﬁciently solved by proximal-gradient-based methods.
Revised: Problem (1) can be efﬁciently solved by proximal-gradient-based methods.

Original: Firstly, by substituting the deﬁnition of a t into ˆ s t +1 , (1) is equivalent to
Revised: Firstly, by substituting the deﬁnition of a t into ˆ s t +1 , (1) is equivalent to

Original: [Equation]
Revised: [Equation]

Original: where C pt := { x = ( x 1 , · · · , x n ) : (cid:107) x (cid:107) p ≤ (cid:15), (cid:96) S − s t ≤ x ≤ u S − s t and x i = 0 for i
Revised: where C pt := { x = ( x 1 , · · · , x n ) : (cid:107) x (cid:107) p ≤ (cid:15), (cid:96) S − s t ≤ x ≤ u S − s t and x i = 0 for i

Original: / ∈
Revised: / ∈

Original: L t } .
Revised: L t } .

Original: If we choose the distance function as d ( a, b ) = (cid:107) a − b (cid:107) 2 , we can use the projected gradient descent (PGD) algorithm (Nesterov, 2003) to solve (2).
Revised: If we choose the distance function as d ( a, b ) = (cid:107) a − b (cid:107) 2 , we can use the projected gradient descent (PGD) algorithm (Nesterov, 2003) to solve (2).

Original: The PGD iteration to update y k at iteration k can be described as
Revised: The PGD iteration to update y k at iteration k can be described as

Original: [Equation]
Revised: [Equation]

Original: where proj C pt ( · ) is the projection to the convex set C pt and η is the learning rate.
Revised: where proj C pt ( · ) is the projection to the convex set C pt and η is the learning rate.

Original: The projection is simple to calculate since C pt is the intersection of a unit ball in the p -norm and a box.
Revised: The projection is simple to calculate since C pt is the intersection of a unit ball in the p -norm and a box.

Original: The whole attack process can be summarized in Algorithm 1.
Revised: The whole attack process can be summarized in Algorithm 1.

Original: Algorithm
Revised: Algorithm

Original: 1: Initialization:
Revised: 1: Initialization:

Original: Given s t , s target , π , f d , L t ; initialize ∆ s 0 ; choose learning rate η >2: For k := 0 , · · · , K − 1 do 3:
Revised: Given s t , s target , π , f d , L t ; initialize ∆ s 0 ; choose learning rate η >2: For k := 0 , · · · , K − 1 do 3:

Original: Compute a t = ( a 1 t , · · · , a nt ) as a i t = (cid:26) π ( s i t + ∆ s i ) if i ∈ L t π ( s it ) otherwise .
Revised: Compute a t = ( a 1 t , · · · , a nt ) as a i t = (cid:26) π ( s i t + ∆ s i ) if i ∈ L t π ( s it ) otherwise .

Original: Compute ˆ s t +1 =
Revised: Compute ˆ s t +1 =

Original: f d ( s t , a t ) 5:
Revised: f d ( s t , a t ) 5:

Original: Update ∆ s as ∆ s k +1 =
Revised: Update ∆ s as ∆ s k +1 =

Original: proj C t [∆ s k − η ∇ ∆ s d (ˆ s t +1 , s target )]
Revised: proj C t [∆ s k − η ∇ ∆ s d (ˆ s t +1 , s target )]

Original: 6: End For
Revised: 6: End For

Original: Learning dynamic model.
Revised: Learning dynamics model.

Original: The most important factor that affects the solution of (2) is the quality of the learned dynamic model f d .
Revised: The most important factor that affects the solution of (2) is the quality of the learned dynamics model f d .

Original: If the dynamics is known, we can solve (2) exactly so that the solution is indeed the optimal perturbation.
Revised: If the dynamics is known, we can solve (2) exactly so that the solution is indeed the optimal perturbation.

Original: However, it is hardly the case in practice because we often approximate the dynamics using function approximators such as neural networks.
Revised: However, it is hardly the case in practice because we often approximate the dynamics using function approximators such as neural networks.

Original: The parameter w for f d is the solution of the following optimization problem
Revised: The parameter w for f d is the solution of the following optimization problem

Original: [Equation]
Revised: [Equation]

Original: where D is a collection of state transition { ( s icur , s inext ) } |D| i =1 where s next is the actual state that the environment transitions to after taking action a t determined by a given policy.
Revised: where D is a collection of state transition { ( s icur , s inext ) } |D| i =1 where s next is the actual state that the environment transitions to after taking action a t determined by a given policy.

Original: In particular, we separately collect transitions using the pre-trained policy π and a random policy to obtain D train and D random .
Revised: In particular, we separately collect transitions using the pre-trained policy π and a random policy to obtain D train and D random .

Original: Then the dataset D is built as D = D train ∪ D random .
Revised: Then the dataset D is built as D = D train ∪ D random .

Original: Problem (4) is a standard supervised learning problem and can be solved using gradient-based learning.
Revised: Problem (4) is a standard supervised learning problem and can be solved using gradient-based learning.

Original: We describe the full process of training the dynamic model in Algorithm 2 where the GradientBasedUpdate step is any gradient-descent-type update.
Revised: We describe the full process of training the dynamics model in Algorithm 2 where the GradientBasedUpdate step is any gradient-descent-type update.

Original: π tr
Revised: pre

Original:  a random policy π rd  initialize
Revised: trained policy π tr and a random policy π rd ; initialize

Original:  train  D random  collecting a set of transitions D train and D random using policy π tr and π rd , respectively
Revised: D := D train ∪ D random by collecting a set of transitions D train and D random using policy π tr and π rd , respectively.

Original: 3. V ICTIM AGENT SELECTION
Revised: 3. V ICTIM AGENT SELECTION

Original: In this subsection, we discuss different strategies to select victim agents when performing adversarial attacks in the MARL setting.
Revised: In this subsection, we discuss different strategies to select victim agents when performing adversarial attacks in the MARL setting.

Original: Suppose that we would like to perform an attack on any agent but the budget on the victim agents is n a ≤ n .
Revised: Suppose that we would like to perform an attack on any agent but the budget on the victim agents is n a ≤ n .

Original: In other words, an attacker can only choose a subset of agents to attack during each time step, and the goal of an attacker is to choose the most vulnerable agents so that the total reward is utmost affected.
Revised: In other words, an attacker can only choose a subset of agents to attack during each time step, and the goal of an attacker is to choose the most vulnerable agents so that the total reward is utmost affected.

Original: We note that this scenario is unique in the setting of multi-agent DRL setting, as in the single DRL agent setting we can only attack the same agent all the time.
Revised: We note that this scenario is unique in the setting of multi-agent DRL setting, as in the single DRL agent setting we can only attack the same agent all the time.

Original: To start with, we ﬁrst formulate a mixed-integer program to perform the attack on a set of optimally victim agents as below:
Revised: To start with, we ﬁrst formulate a mixed-integer program to perform the attack on a set of optimally victim agents as below:

Original: [Equation]
Revised: [Equation]

Original: where we introduce a new binary variable w to select the appropriate agents’ input to perturb and n a is the total number of victim agents we can attack.
Revised: where we introduce a new binary variable w to select the appropriate agents’ input to perturb and n a is the total number of victim agents we can attack.

Original: Due to the existence of the new binary variable, problem (5) is much harder to solve than before.
Revised: Due to the existence of the new binary variable, problem (5) is much harder to solve than before.

Original: We instead solve a proxy of (5) as follows
Revised: We instead solve a proxy of (5) as follows

Original: [Equation]
Revised: [Equation]

Original: where W ( · , θ ) is a function parametrized by θ that takes current state as input and outputs the weight to distribute the noise to each agent.
Revised: where W ( · , θ ) is a function parametrized by θ that takes current state as input and outputs the weight to distribute the noise to each agent.

Original: Suppose we represent W ( · , θ ) by a neural network, we can rewrite the formulation (6) as
Revised: Suppose we represent W ( · , θ ) by a neural network, we can rewrite the formulation (6) as

Original: [Equation]
Revised: [Equation]

Original: because the last constraint in (6) can be enforced by using a softmax activation in the neural network
Revised: because the last constraint in (6) can be enforced by using a softmax activation in the neural network

Original: W ( · , θ ) .
Revised: W ( · , θ ) .

Original: As a result, (7) can be efﬁciently solved by using PGD.
Revised: As a result, (7) can be efﬁciently solved by using PGD.

Original: We present the pseudo-code of the attack in Algorithm 3.
Revised: We present the pseudo-code of the attack in Algorithm 3.

Original: After K steps of PGD update, we deﬁne the index i n − j as the j -th largest value within W ( s t , θ K ) ∈ R n , i.e. we have W i ( n ) ( s t , θ K ) ≥ W i ( n − 1) ( s t , θ K ) ≥ · · · ≥
Revised: After K steps of PGD update, we deﬁne the index i n − j as the j -th largest value within W ( s t , θ K ) ∈ R n , i.e. we have W i ( n ) ( s t , θ K ) ≥ W i ( n − 1) ( s t , θ K ) ≥ · · · ≥

Original: W i (1) ( s t , θ K ) .
Revised: W i (1) ( s t , θ K ) .

Original: Let I j be the index set of top- j largest outputs of the W ( s t , θ K ) network.
Revised: Let I j be the index set of top- j largest outputs of the W ( s t , θ K ) network.

Original: The ﬁnal perturbation returned by our victim agent selection strategy will be (cid:99) ∆ s =
Revised: The ﬁnal perturbation returned by our victim agent selection strategy will be (cid:99) ∆ s =

Original: (( (cid:99) ∆ s ) 1 , · · · , ( (cid:99) ∆ s ) n ) where ( (cid:99) ∆ s )
Revised: (( (cid:99) ∆ s ) 1 , · · · , ( (cid:99) ∆ s ) n ) where ( (cid:99) ∆ s )

Original: i = 

Revised: i =

Original: if i / ∈
Revised: if i / ∈

Original: I n a and ( (cid:99) ∆ s ) i = (∆ s K ) i if i ∈ I n a .
Revised: I n a and ( (cid:99) ∆ s ) i = (∆ s K ) i if i ∈ I n a .

Original: Algorithm
Revised: Algorithm

Original: 1: Initialization:
Revised: 1: Initialization:

Original: Given s t , s target , π , f d , n a ; initialize ∆ s 0 ; choose learning rate η > 0 λ > 0 .
Revised: Given s t , s target , π , f d , n a ; initialize ∆ s 0 ; choose learning rate η > 0 λ > 0 .

Original: 2: For k := 0 , · · · , K − 1 do 3:
Revised: 2: For k := 0 , · · · , K − 1 do 3:

Original: Compute a t =
Revised: Compute a t =

Original: π ( s t + (cid:104) ∆ s, W ( s t , θ ) (cid:105) ) 4:
Revised: π ( s t + (cid:104) ∆ s, W ( s t , θ ) (cid:105) ) 4:

Original: Compute ˆ s t +1 =
Revised: Compute ˆ s t +1 =

Original: f d ( s t , a t ) 5:
Revised: f d ( s t , a t ) 5:

Original: Update ∆ s : ∆ s k +1 =
Revised: Update ∆ s : ∆ s k +1 =

Original: proj C pt [∆ s k − η ∇ ∆ s d (ˆ s t +1 , s target )] 6:
Revised: proj C pt [∆ s k − η ∇ ∆ s d (ˆ s t +1 , s target )] 6:

Original: Update θ : θ k +

Revised: Update θ : θ k +

Original: =
Revised: =

Original: θ k − λ ∇ θ d (ˆ s t +1 , s target ) 7: End For
Revised: θ k − λ ∇ θ d (ˆ s t +1 , s target ) 7: End For

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: Remark 3.1 For this attack, we assume each agent i has access to the other agent’s observation to form the concatenated state s t .
Revised: Remark 3.1 For this attack, we assume each agent i has access to the other agent’s observation to form the concatenated state s t .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We perform the attack on various multi-agent MuJoCo (MA-MuJoCo) environments including
Revised: We perform the attack on various multi-agent MuJoCo (MA-MuJoCo) environments including

Original: Walker2d (2x3), HalfCheetah (2x3), HalfCheetah (6x1), and Ant (4x2).
Revised: Walker2d (2x3), HalfCheetah (2x3), HalfCheetah (6x1), and Ant (4x2).

Original: The pair environment name (config) indicates the name of MuJoCo environment along with the agent partition, where a conﬁguration of 2x3 means there are in total 2 agents and each agent has 3 actions.
Revised: The pair environment name (config) indicates the name of MuJoCo environment along with the agent partition, where a conﬁguration of 2x3 means there are in total 2 agents and each agent has 3 actions.

Original: Note that we cannot directly apply methods such as the fast gradient sign method (FGSM) or JSMA (an attack using saliency map) as in Lin et al.
Revised: Note that we cannot directly apply methods such as the fast gradient sign method (FGSM) or JSMA (an attack using saliency map) as in Lin et al.

Original: (2020), since those methods require a “target” action which is not available in continuous action space.
Revised: (2020), since those methods require a “target” action which is not available in continuous action space.

Original: Therefore, we consider two baselines: Uniform and Gaussian baselines where the perturbation follows either Uniform distribution U ( − (cid:15), (cid:15) ) or
Revised: Therefore, we consider two baselines: Uniform and Gaussian baselines where the perturbation follows either Uniform distribution U ( − (cid:15), (cid:15) ) or

Original: Normal distribution N (0 , (cid:15) ) .
Revised: Normal distribution N (0 , (cid:15) ) .

Original: Variants of model-based attack.
Revised: Variants of model-based attack.

Original: We consider the following variants of our model-based attack:
Revised: We consider the following variants of our model-based attack:

Original: • Model-based attack on ﬁxed agents: perform Algorithm 1 using a ﬁxed set of victim agents L .
Revised: • Model-based attack on ﬁxed agents: perform Algorithm 1 using a ﬁxed set of victim agents L .

Original: • Best model-based attack on ﬁxed agents: among the model-based attack on ﬁxed agents, pick the subset that achieves the best performance.
Revised: • Best model-based attack on ﬁxed agents: among the model-based attack on ﬁxed agents, pick the subset that achieves the best performance.

Original: • Model-based attack on random agents: perform Algorithm 1 with L t , which is sampled uniformly from N such that |L t | = n a ( n a is the total victim agents).
Revised: • Model-based attack on random agents: perform Algorithm 1 with L t , which is sampled uniformly from N such that |L t | = n a ( n a is the total victim agents).

Original: • Model-based attack with learned victim selection: perform Algorithm 3 to select vulnerable agents and perform the attack on them.
Revised: • Model-based attack with learned victim selection: perform Algorithm 3 to select vulnerable agents and perform the attack on them.

Original: • Greedy victim selection: perform a sweep over subsets of agents with size n a .
Revised: • Greedy victim selection: perform a sweep over subsets of agents with size n a .

Original: For each subset, perform Algorithm 1 to obtain proper perturbation and retrieve the corresponding objective value (distance between predicted state and target state).
Revised: For each subset, perform Algorithm 1 to obtain proper perturbation and retrieve the corresponding objective value (distance between predicted state and target state).

Original: Select the subset corresponding to the lowest objective value as the victim agents.
Revised: Select the subset corresponding to the lowest objective value as the victim agents.

Original: Experiment setup.
Revised: Experiment setup.

Original: We ﬁrst use MADDPG (Lowe et al., 2017) to train MARL agents for the four MA-MuJoCo environments listed above.
Revised: We ﬁrst use MADDPG (Lowe et al., 2017) to train MARL agents for the four MA-MuJoCo environments listed above.

Original: Using the trained agents, we collect datasets containing one million transitions to train the dynamic model for each environment.
Revised: Using the trained agents, we collect datasets containing one million transitions to train the dynamics model for each environment.

Original: The dynamic model is a fully connected neural network with three hidden layers of 1000 neurons.
Revised: The dynamics model is a fully connected neural network with three hidden layers of 1000 neurons.

Original: We also use a fully-connected neural network for W ( s t , θ ) in (6) with two hidden layers of 200 neurons.
Revised: We also use a fully-connected neural network for W ( s t , θ ) in (6) with two hidden layers of 200 neurons.

Original: We use AdamW (Loshchilov & Hutter, 2017) as the optimizer and select the best learning rate from { 1 , 5 } × { 10 − 5 , 10 − 4 , 10 − 3 } (the best learning rate is the one achieving lowest prediction error on a test set of 80 , 000 samples).
Revised: We use AdamW (Loshchilov & Hutter, 2017) as the optimizer and select the best learning rate from { 1 , 5 } × { 10 − 5 , 10 − 4 , 10 − 3 } (the best learning rate is the one achieving lowest prediction error on a test set of 80 , 000 samples).

Original: For our model-based attack, we run PGD for K = 30 steps to solve (2) and (5).
Revised: For our model-based attack, we run PGD for K = 30 steps to solve (2) and (5).

Original: We perform each attack over 16 episodes then average the rewards.
Revised: We perform each attack over 16 episodes then average the rewards.

Original: We also illustrate the standard deviation of rewards using the shaded area in the plots.
Revised: We also illustrate the standard deviation of rewards using the shaded area in the plots.

Original: Model-free baselines vs model-based attack.
Revised: Model-free baselines vs model-based attack.

Original: In this experiment, we run the two baseline attacks along with our model-based attack on the four MA-MuJoCo environments, in which there is only one victim agent ( n a = 1 ).
Revised: In this experiment, we run the two baseline attacks along with our model-based attack on the four MA-MuJoCo environments, in which there is only one victim agent ( n a = 1 ).

Original: Figure 1 illustrates the performance when we perform these attacks on speciﬁc agents under multiple budget levels using (cid:96) ∞ -norm.
Revised: Figure 1 illustrates the performance when we perform these attacks on speciﬁc agents under multiple budget levels using (cid:96) ∞ -norm.

Original: For a ﬁxed agent, our model-based attack consistently outperforms the two baselines.
Revised: For a ﬁxed agent, our model-based attack consistently outperforms the two baselines.

Original: In particular, our model-based attack yields much lower rewards under low budget constraints (small (cid:15) )
Revised: In particular, our model-based attack yields much lower rewards under low budget constraints (small (cid:15) )

Original: compared to the two baselines.
Revised: compared to the two baselines.

Original: To better visualize the performance difference, 2 illustrates the environment with and without attacks captured at different time-steps.
Revised: To better visualize the performance difference, 2 illustrates the environment with and without attacks captured at different time-steps.

Original: From Figure 2, our model-based attack is able to make the MuJoCo agent ﬂip, which terminates the episode at the 409-th timestep.
Revised: From Figure 2, our model-based attack is able to make the MuJoCo agent ﬂip, which terminates the episode at the 409-th timestep.

Original: The episode length and total rewards for each variant are: No attack (1000 , 2644 .
Revised: The episode length and total rewards for each variant are: No attack (1000 , 2644 .

Original: 60) , Uniform (1000 , 1758 . 97) , Gaussian (891 , 1399 . 51) ,
Revised: 60) , Uniform (1000 , 1758 . 97) , Gaussian (891 , 1399 . 51) ,

Original: Ours (409 , 287 . 06) .
Revised: Ours (409 , 287 . 06) .

Original: We also investigate how the state values change during these attacks.
Revised: We also investigate how the state values change during these attacks.

Original: Figure 3 presents different recordings of state values under adversarial attacks compared to no attack.
Revised: Figure 3 presents different recordings of state values under adversarial attacks compared to no attack.

Original: Consider state index 8 , which represents the horizontal velocity of the agent.
Revised: Consider state index 8 , which represents the horizontal velocity of the agent.

Original: For the HalfCheetah environment, as the goal is to make the agent move forward as fast as possible, we expect the reward to be proportional to this state value.
Revised: For the HalfCheetah environment, as the goal is to make the agent move forward as fast as possible, we expect the reward to be proportional to this state value.

Original: From Figure 3, all three attacks have fairly sparse red fractions across timesteps, which result in a much lower reward compared to the no-attack setting.
Revised: From Figure 3, all three attacks have fairly sparse red fractions across timesteps, which result in a much lower reward compared to the no-attack setting.

Original: Among the three attacks, our model-based ones appear to have the most sparse red fractions leading to the lowest rewards.
Revised: Among the three attacks, our model-based ones appear to have the most sparse red fractions leading to the lowest rewards.

Original: In addition, the model-based attack appears to show its advantage in environments with more agents.
Revised: In addition, the model-based attack appears to show its advantage in environments with more agents.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Step Step
Revised: Step Step

Original: Step Step
Revised: Step Step

Original: Step Step
Revised: Step Step

Original: [Figure]
Revised: [Figure]

Original: f o r m
Revised: f o r m

Original: G a u ss i a n
Revised: G a u ss i a n

Original: O u
Revised: O u

Original: [Figure]
Revised: [Figure]

Original: s
Revised: s

Original: In particular
Revised: In particular

Original: [Figure]
Revised: [Figure]

Original: our approach results in lower rewards under a smaller budget as seen in HalfCheetah (6x1) and Ant (4x2) environments
Revised: our approach results in lower rewards under a smaller budget as seen in HalfCheetah (6x1) and Ant (4x2) environments

Original: [Figure]
Revised: [Figure]

Original: Effectiveness of learned victim selection.
Revised: Effectiveness of learned victim selection.

Original: We further demonstrate the performance of our modelbased attack with and without learned victim selection on one victim agent.
Revised: We further demonstrate the performance of our modelbased attack with and without learned victim selection on one victim agent.

Original: Figure 4 illustrates the performance of four model-based variants: the best performing model-based attack on the ﬁxed agent (Best Fixed Agent), model-based attack on the random agent (Random Agent), model-based with learned victim selection as in Algorithm 3 (Learned victim Selection), and model-based with greedy victim selection (Greedy).
Revised: Figure 4 illustrates the performance of four model-based variants: the best performing model-based attack on the ﬁxed agent (Best Fixed Agent), model-based attack on the random agent (Random Agent), model-based with learned victim selection as in Algorithm 3 (Learned victim Selection), and model-based with greedy victim selection (Greedy).

Original: Surprisingly, even though the greedy approach employs a bruteforce strategy to select the agent with the closest distance, its performance reveals that selecting victim agents greedily might not be a good strategy.
Revised: Surprisingly, even though the greedy approach employs a bruteforce strategy to select the agent with the closest distance, its performance reveals that selecting victim agents greedily might not be a good strategy.

Original: The learned victim selection seems to be better than the random or greedy strategy and is comparable with the random one in HalfCheetah environment.
Revised: The learned victim selection seems to be better than the random or greedy strategy and is comparable with the random one in HalfCheetah environment.

Original: The best-ﬁxed agent variant has good performance under low budget constraints in
Revised: The best-ﬁxed agent variant has good performance under low budget constraints in

Original: HalfCheetah (6x1) and Ant (4x2) environments.
Revised: HalfCheetah (6x1) and Ant (4x2) environments.

Original: We want to emphasize that although the learned victim selection strategy in Section 3.3 is sub-optimal, it works well in these four environments.
Revised: We want to emphasize that although the learned victim selection strategy in Section 3.3 is sub-optimal, it works well in these four environments.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper, we propose a new attack algorithm named cMBA for evaluating the robustness of c-MARL environment with continuous action space.
Revised: In this paper, we propose a new attack algorithm named cMBA for evaluating the robustness of c-MARL environment with continuous action space.

Original: Our cMBA algorithm is the ﬁrst to consider the c-MARL and the adversarial perturbation is computed by solving an optimization problem with learned dynamics models.
Revised: Our cMBA algorithm is the ﬁrst to consider the c-MARL and the adversarial perturbation is computed by solving an optimization problem with learned dynamics models.

Original: The cMBA approach outperforms the other baselines attack by a large margin under 4 multi-agent MuJoCo environments especially ones with larger number of agents.
Revised: The cMBA approach outperforms the other baselines attack by a large margin under 4 multi-agent MuJoCo environments especially ones with larger number of agents.

Original: Unique to multi-agent setting, we also study different strategies to select victim agents.
Revised: Unique to multi-agent setting, we also study different strategies to select victim agents.

Original: Extensive experiment results on standard c-MARL benchmarks show that our proposed model-based attack and victim-selection strategy can successfully degrade the performance of well-trained c-MARL agents while outperforming other baselines by a large margin.
Revised: Extensive experiment results on standard c-MARL benchmarks show that our proposed model-based attack and victim-selection strategy can successfully degrade the performance of well-trained c-MARL agents while outperforming other baselines by a large margin.

Original: A future direction is to consider a timed attack strategy where the perturbation is added at certain timesteps.
Revised: A future direction is to consider a timed attack strategy where the perturbation is added at certain timesteps.

Original: We can also consider longer planning step, i.e. trying to reach a target state within the next T steps instead of 1.
Revised: We can also consider longer planning step, i.e. trying to reach a target state within the next T steps instead of 1.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: This paper does not contain ethics concerns.
Revised: This paper does not contain ethics concerns.

Original: A M ODE D ETAILS ON E XPERIMENT S ETUP IN S ECTION
Revised: A M ODE D ETAILS ON E XPERIMENT S ETUP IN S ECTION

Original: Specifying target state for each environment.
Revised: Specifying target state for each environment.

Original: To perform our model based attack, we need to specify a target state that potentially worsens the total reward.
Revised: To perform our model based attack, we need to specify a target state that potentially worsens the total reward.

Original: In multi-agent MuJoCo environments, each agent has access to its own observation of the agent consisting the position-related and velocityrelated information.
Revised: In multi-agent MuJoCo environments, each agent has access to its own observation of the agent consisting the position-related and velocityrelated information.

Original: The position-related information includes part of x, y, z coordinates and the quarternion that represents the orientation of the agent.
Revised: The position-related information includes part of x, y, z coordinates and the quarternion that represents the orientation of the agent.

Original: The velocity-related information contains global linear velocities and angular velocities for each joint in a MuJoCo agent.
Revised: The velocity-related information contains global linear velocities and angular velocities for each joint in a MuJoCo agent.

Original: We refer the reader to Todorov et al.
Revised: We refer the reader to Todorov et al.

Original: (2012) for more information about each MuJoCo environment.
Revised: (2012) for more information about each MuJoCo environment.

Original: Now we describe the design of this target state for each environment as follows:
Revised: Now we describe the design of this target state for each environment as follows:

Original: • Walker (2x3) environment: Since the episode ends whenever the agent falls,
Revised: • Walker (2x3) environment: Since the episode ends whenever the agent falls,

Original: i
Revised: i

Original: e. the z coordinate falls below certain threshold.
Revised: e. the z coordinate falls below certain threshold.

Original: In this environment, the target state has a value of 0 for the index that corresponds to the z coordinate of the MuJoCo agent (index 0).
Revised: In this environment, the target state has a value of 0 for the index that corresponds to the z coordinate of the MuJoCo agent (index 0).

Original: • HalfCheetah (2x3) and HalfCheetah (6x1) environments: As the goal is to make agent moves as fast as possible, we set the index corresponding to the linear velocities to 0 (index 8).
Revised: • HalfCheetah (2x3) and HalfCheetah (6x1) environments: As the goal is to make agent moves as fast as possible, we set the index corresponding to the linear velocities to 0 (index 8).

Original: • Ant (4x2) environment:
Revised: • Ant (4x2) environment:

Original: As the agent can move freely in a 2D-plan, we set the index corresponding to the x, y linear velocities to 0 (indices 13 and 14).
Revised: As the agent can move freely in a 2D-plan, we set the index corresponding to the x, y linear velocities to 0 (indices 13 and 14).

Original: B A DDITIONAL E XPERIMENTS
Revised: B A DDITIONAL E XPERIMENTS

Original: In this section, we present experimental results in addition to ones presented in Section 4.
Revised: In this section, we present experimental results in addition to ones presented in Section 4.

Original: Figure 5 illustrates the environment with and without attacks captured at different timesteps.
Revised: Figure 5 illustrates the environment with and without attacks captured at different timesteps.

Original: From
Revised: From

Original: Figure 5, our model based attack is able to make the MuJoCo agent fall down which terminates the episode at the 65-th timestep.
Revised: Figure 5, our model based attack is able to make the MuJoCo agent fall down which terminates the episode at the 65-th timestep.

Original: The episode length and total rewards for each variant are: No attack (478 , 1736 . 44) , Uniform (382 , 1037 .
Revised: The episode length and total rewards for each variant are: No attack (478 , 1736 . 44) , Uniform (382 , 1037 .

Original: 73) , Gaussian (90 , 32 . 24) , Ours (63 , − 34 . 26) .
Revised: 73) , Gaussian (90 , 32 . 24) , Ours (63 , − 34 . 26) .

Original: Figureillustrates how the state values change during an episode.
Revised: Figureillustrates how the state values change during an episode.

Original: These state values correspond to the agent shown in Figure 5.
Revised: These state values correspond to the agent shown in Figure 5.

Original: If we look at how the noise values change as in Figure 7, the noise generated by our approach appears to maximize the permissible noise budget.
Revised: If we look at how the noise values change as in Figure 7, the noise generated by our approach appears to maximize the permissible noise budget.

Original: In addition to the (cid:96) ∞ -norm budget constraint, we also evaluate adversarial attacks using the (cid:96) 1norm constraint.
Revised: In addition to the (cid:96) ∞ -norm budget constraint, we also evaluate adversarial attacks using the (cid:96) 1norm constraint.

Original: Note that using (cid:96) 1 -norm for budget constraint is more challenging as the attack needs to distribute the noise across all states while in the (cid:96) ∞ -norm the computation of perturbation for individual state is independent.
Revised: Note that using (cid:96) 1 -norm for budget constraint is more challenging as the attack needs to distribute the noise across all states while in the (cid:96) ∞ -norm the computation of perturbation for individual state is independent.

Original: Figure 8 illustrates an episode of a Walker agent with and without attack.
Revised: Figure 8 illustrates an episode of a Walker agent with and without attack.

Original: Our cMBA approach is able to make the agent fall at the 255 -th timestep with the episode length and rewards for each setting as: No attack (661 , 2513 . 57) , Uniform (403 , 1144 .
Revised: Our cMBA approach is able to make the agent fall at the 255 -th timestep with the episode length and rewards for each setting as: No attack (661 , 2513 . 57) , Uniform (403 , 1144 .

Original: Gaussian (455 , 1637 . 16) , Ours (253 , 759 . 85) .
Revised: Gaussian (455 , 1637 . 16) , Ours (253 , 759 . 85) .

Original: From the noise values in Figure 7, our cMBA method appears to put noise on a few selective states at each timestep, similar to the Gausian noise setting. 
Revised: From the noise values in Figure 7, our cMBA method appears to put noise on a few selective states at each timestep, similar to the Gausian noise setting. Additional experiments using the approach in Lin et al.

Revised: (2020) for continuous action spaces:

Revised: In this experiment, we follow the approach in Lin et al.

Revised: (2020) in the Ant (4x2) environment where we train an adversarial policy for one agent trying to minimize the total team reward while the remaining agents use the trained MARL policy.

Revised: The adversarial policy is trained for 1 million timesteps.

Original: [Figure]
Revised: The results using this approach compared with our approach and the two baselines are presented in Figure 11. From Figure 11, our cMBA approach outperforms Lin et al. (2020)’s approach as it is able to achieve lower team rewards when attacking the same agent. We also note that Lin et al.

Revised: (2020)’s approach is better than the two baselines using Uniform and Gaussian noise.

Revised: Effect of using (cid:107)·(cid:107) 1 budget constraint: In this experiment, we replace the (cid:107)·(cid:107) ∞ by the (cid:107)·(cid:107) 1 for budget constraint.

Revised: Using (cid:107)·(cid:107) 1 constraint will produce a signiﬁcant different adversarial noise pattern

Revised: [Figure]

Original: Step
Revised: Step

Original: [Figure]
Revised: [Figure]

Original: Step
Revised: Step

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: compared to (cid:107)·(cid:107) ∞ as producing adversarial noise when using (cid:107)·(cid:107) ∞ is independent for each state while it is not the case for (cid:107)·(cid:107) 1 .

Revised: We run different attacks on two environments, HalfCheetah (6x1) and Ant (4x2), and the results are presented in Figure 12 and Figure 13.

Revised: From these ﬁgures, cMBA outperforms other baselines.

Revised: In particular, it is able to achieve much lower rewards under smaller budget constraint which shows the advantage of our approach

Original: [Figure] 
Revised: [Figure] [Figure]

Revised: Adversarial attacks using dynamics model with various accuracy: In this test, we compare the performance of our attack when using trained dynamics model with different mean-squared error (MSE).

Original: [Figure]
Revised: We use the Ant (4x2) environment and consider the following 3 dynamics models: • The ﬁrst model is trained using 1 million samples for 100 epochs and select the model with the best performance. We denote this model ”1M - Best epoch”. • The second model is trained using 1 million samples for only 1 epoch.

Original: [Figure]
Revised: We denote this model ”1M - 1st epoch”. • We further reduce the number of samples used for training the dynamic model to only 200,000 and train for only 1 epoch. We denote this model ”200K - 1st epoch”. These models are evaluated on a predeﬁned test set consisting of 100,000 samples.

Revised: The test MSE of these models are 0.33, 0.69, 0.79, respectively, with the initial test MSE of 1.241.

Original: [Figure]
Revised: [Figure]

Revised: Figure 14 depicts the attacks using these three models on the same agent in Ant (4x2) environment using (cid:107)·(cid:107) ∞ budget constraint.

Revised: Interestingly, the dynamics model trained with only 200,000 samples for 1 epoch can achieve comparable performance with the other two models using much more samples.

Revised: Comparison between attacking all agents vs attacking 1 agent: In this experiment, we compare performance of different approaches when attacking one victim agent or simply perturbing all agents under the same budget constraint.

Revised: We compare different approaches in the Ant (4x2) environment.

Revised: In particular, we use the two baselines and our cMBA approach to attack each agent out of 4 agent, denoted as (agent i ) for i = 0 , 1 , 2 , 3 .

Original: [Figure]
Revised: We also use these approach in which we simultaneously perturb the inputs to all agents, denoted as (4 agents). In addition, we also illustrate the performance of our victim selection scheme when we only want to attack either 1 or two agents at the same time. We report the mean and standard deviation of rewards using these approach with several values of budget level ε in Table 1 where we use (cid:107) · (cid:107) ∞ for the budget constraint. From Table 1, the performance is getting better as the attacker is stronger, i.e. more agents are attacked at the same time.

Original: [Figure]
Revised: This makes sense as for (cid:107) · (cid:107) ∞ , the adversarial noise for each agent can be computed independently. Now, we consider the (cid:107) · (cid:107) 1 budget constraint. This is a more interesting setting as the total noise budget is ﬁxed the adversarial noise of one agent will affect one from the other. The results are shown using several values of budget level ε in Table 2.

Revised: We can observe that in this case attacking more agents is not always effective.

Revised: [Table]

Revised: [Table]

Original: [Figure]
Revised: [Table] Victim [Table] [Table]

Revised: Learned Victim Selection

Original: [Figure]
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]