Original: T OWARD EVALUATING ROBUSTNESS OF DEEP REINFORCEMENT LEARNING WITH CONTINUOUS CONTROL
Revised: T OWARD EVALUATING ROBUSTNESS OF DEEP REINFORCEMENT LEARNING WITH CONTINUOUS CONTROL

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep reinforcement learning has achieved great success in many previously difﬁcult reinforcement learning tasks, yet recent studies show that deep RL agents are also unavoidably susceptible to adversarial perturbations, similar to deep neural networks in classiﬁcation tasks.
Revised: Deep reinforcement learning has achieved great success in many previously difﬁcult reinforcement learning tasks, yet recent studies show that deep RL agents are also unavoidably susceptible to adversarial perturbations, similar to deep neural networks in classiﬁcation tasks.

Original: Prior works mostly focus on model-free adversarial attacks and agents with discrete actions.
Revised: Prior works mostly focus on model-free adversarial attacks and agents with discrete actions.

Original: In this work, we study the problem of continuous control agents in deep RL with adversarial attacks and propose the ﬁrst two-step algorithm based on learned model dynamics.
Revised: In this work, we study the problem of continuous control agents in deep RL with adversarial attacks and propose the ﬁrst two-step algorithm based on learned model dynamics.

Original: Extensive experiments on various MuJoCo domains (Cartpole, Fish, Walker, Humanoid) demonstrate that our proposed framework is much more effective and efﬁcient than model-free attacks baselines in degrading agent performance as well as driving agents to unsafe states
Revised: Extensive experiments on various MuJoCo domains (Cartpole, Fish, Walker, Humanoid) demonstrate that our proposed framework is much more effective and efﬁcient than model-free attacks baselines in degrading agent performance as well as driving agents to unsafe states.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep reinforcement learning (RL) has revolutionized the ﬁelds of AI and machine learning over the last decade.
Revised: Deep reinforcement learning (RL) has revolutionized the ﬁelds of AI and machine learning over the last decade.

Original: The introduction of deep learning has achieved unprecedented success in solving many problems that were intractable in the ﬁeld of RL, such as playing Atari games from pixels and performing robotic control tasks (Mnih et al., 2015;
Revised: The introduction of deep learning has achieved unprecedented success in solving many problems that were intractable in the ﬁeld of RL, such as playing Atari games from pixels and performing robotic control tasks (Mnih et al., 2015;

Original: Lillicrap et al., 2015; Tassa et al., 2018).
Revised: Lillicrap et al., 2015; Tassa et al., 2018).

Original: Unfortunately, similar to the case of deep neural network classiﬁers with adversarial examples, recent studies show that deep RL agents are also vulnerable to adversarial attacks.
Revised: Unfortunately, similar to the case of deep neural network classiﬁers with adversarial examples, recent studies show that deep RL agents are also vulnerable to adversarial attacks.

Original: A commonly-used threat model allows the adversary to manipulate the agent’s observations at every time step, where the goal of the adversary is to decrease the agent’s total accumulated reward.
Revised: A commonly-used threat model allows the adversary to manipulate the agent’s observations at every time step, where the goal of the adversary is to decrease the agent’s total accumulated reward.

Original: As a pioneering work in this ﬁeld, (Huang et al., 2017) show that by leveraging the FGSM attack on each time frame, an agent’s average reward can be signiﬁcantly decreased with small input adversarial perturbations in ﬁve Atari games.
Revised: As a pioneering work in this ﬁeld, (Huang et al., 2017) show that by leveraging the FGSM attack on each time frame, an agent’s average reward can be signiﬁcantly decreased with small input adversarial perturbations in ﬁve Atari games.

Original: (Lin et al., 2017) further improve the efﬁciency of the attack in (Huang et al., 2017) by leveraging heuristics of detecting a good time to attack and luring agents to bad states with sample-based Monte-Carlo planning on a trained generative video prediction model.
Revised: (Lin et al., 2017) further improve the efﬁciency of the attack in (Huang et al., 2017) by leveraging heuristics of detecting a good time to attack and luring agents to bad states with sample-based Monte-Carlo planning on a trained generative video prediction model.

Original: Since the agents have discrete actions in Atari games (Huang et al., 2017; Lin et al., 2017), the problem of attacking Atari agents often reduces to the problem of ﬁnding adversarial examples on image classiﬁers, also pointed out in (Huang et al., 2017), where the adversaries intend to craft the input perturbations that would drive agent’s new action to deviate from its nominal action.
Revised: Since the agents have discrete actions in Atari games (Huang et al., 2017; Lin et al., 2017), the problem of attacking Atari agents often reduces to the problem of ﬁnding adversarial examples on image classiﬁers, also pointed out in (Huang et al., 2017), where the adversaries intend to craft the input perturbations that would drive agent’s new action to deviate from its nominal action.

Original: However, for agents with continuous actions, the above strategies can not be directly applied.
Revised: However, for agents with continuous actions, the above strategies can not be directly applied.

Original: Recently, (Uesato et al., 2018) studied the problem of adversarial testing for continuous control domains in a similar but slightly different setting.
Revised: Recently, (Uesato et al., 2018) studied the problem of adversarial testing for continuous control domains in a similar but slightly different setting.

Original: Their goal was to efﬁciently and effectively ﬁnd catastrophic failure given a trained agent and to predict its failure probability.
Revised: Their goal was to efﬁciently and effectively ﬁnd catastrophic failure given a trained agent and to predict its failure probability.

Original: The key to success in (Uesato et al., 2018) is the availability of agent training history.
Revised: The key to success in (Uesato et al., 2018) is the availability of agent training history.

Original: However, such information may not always be accessible to the users, analysts, and adversaries.
Revised: However, such information may not always be accessible to the users, analysts, and adversaries.

Original: Therefore, in this paper we study the robustness of deep RL agents in a more challenging setting where the agent has continuous actions and its training history is not available.
Revised: Therefore, in this paper we study the robustness of deep RL agents in a more challenging setting where the agent has continuous actions and its training history is not available.

Original: We consider the threat models where the adversary is allowed to manipulate an agent’s observations or actions with small perturbations, and we propose a two-step algorithmic framework to ﬁnd efﬁcient adversarial attacks based on learned dynamics models.
Revised: We consider the threat models where the adversary is allowed to manipulate an agent’s observations or actions with small perturbations, and we propose a two-step algorithmic framework to ﬁnd efﬁcient adversarial attacks based on learned dynamics models.

Original: Experimental results show that our proposed modelbased attack can successfully degrade agent performance and is also more effective and efﬁcient than model-free attacks baselines.
Revised: Experimental results show that our proposed modelbased attack can successfully degrade agent performance and is also more effective and efﬁcient than model-free attacks baselines.

Original: The contributions of this paper are the following:
Revised: The contributions of this paper are the following:

Original: [Figure] • To the best of our knowledge, we propose the ﬁrst model-based attack on deep RL agents with continuous actions.
Revised: • To the best of our knowledge, we propose the ﬁrst model-based attack on deep RL agents with continuous actions.

Original: Our proposed attack algorithm is a general two-step algorithm and can be directly applied to the two commonly-used threat models (observation manipulation and action manipulation).
Revised: Our proposed attack algorithm is a general two-step algorithm and can be directly applied to the two commonly-used threat models (observation manipulation and action manipulation).

Original: • We study the efﬁciency and effectiveness of our proposed model-based attack with modelfree attack baselines based on random searches and heuristics ( rand-U , rand-B , ﬂip , see Section 4).
Revised: • We study the efﬁciency and effectiveness of our proposed model-based attack with modelfree attack baselines based on random searches and heuristics ( rand-U , rand-B , ﬂip , see Section 4).

Original: We show that our model-based attack can degrade agent performance more signiﬁcantly and efﬁciently than model-free attacks, which remain ineffective in numerous MuJoCo domains ranging from Cartpole, Fish, Walker, and Humanoid.
Revised: We show that our model-based attack can degrade agent performance more signiﬁcantly and efﬁciently than model-free attacks, which remain ineffective in numerous MuJoCo domains ranging from Cartpole, Fish, Walker, and Humanoid.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Adversarial attacks in reinforcement learning.
Revised: Adversarial attacks in reinforcement learning.

Original: Compared to the rich literature of adversarial examples in image classiﬁcations (Szegedy et al., 2013) and other applications (including natural language processing (Jia & Liang, 2017), speech (Carlini & Wagner, 2018), etc), there is relatively little prior work studying adversarial examples in deep RL.
Revised: Compared to the rich literature of adversarial examples in image classiﬁcations (Szegedy et al., 2013) and other applications (including natural language processing (Jia & Liang, 2017), speech (Carlini & Wagner, 2018), etc), there is relatively little prior work studying adversarial examples in deep RL.

Original: One of the ﬁrst several works in this ﬁeld are (Huang et al., 2017) and (Lin et al., 2017), where both works focus on deep RL agent in Atari games with pixels-based inputs and discrete actions.
Revised: One of the ﬁrst several works in this ﬁeld are (Huang et al., 2017) and (Lin et al., 2017), where both works focus on deep RL agent in Atari games with pixels-based inputs and discrete actions.

Original: In addition, both works assume the agent to be attacked has accurate policy and the problem of ﬁnding adversarial perturbation of visual input reduces to the same problem of ﬁnding adversarial examples on image classiﬁers.
Revised: In addition, both works assume the agent to be attacked has accurate policy and the problem of ﬁnding adversarial perturbation of visual input reduces to the same problem of ﬁnding adversarial examples on image classiﬁers.

Original: Hence, (Huang et al., 2017) applied FGSM (Goodfellow et al., 2015) to ﬁnd adversarial perturbations and (Lin et al., 2017) further improved the efﬁciency of the attack by heuristics of observing a good timing to attack – when there is a large gap in agents action preference between most-likely and leastlikely action.
Revised: Hence, (Huang et al., 2017) applied FGSM (Goodfellow et al., 2015) to ﬁnd adversarial perturbations and (Lin et al., 2017) further improved the efﬁciency of the attack by heuristics of observing a good timing to attack – when there is a large gap in agents action preference between most-likely and leastlikely action.

Original: In a similar direction, (Uesato et al., 2018) study the problem of adversarial testing by leveraging rejection sampling and the agent training histories.
Revised: In a similar direction, (Uesato et al., 2018) study the problem of adversarial testing by leveraging rejection sampling and the agent training histories.

Original: With the availability of training histories, (Uesato et al., 2018) successfully uncover bad initial states with much fewer samples compared to conventional Monte-Carlo sampling techniques.
Revised: With the availability of training histories, (Uesato et al., 2018) successfully uncover bad initial states with much fewer samples compared to conventional Monte-Carlo sampling techniques.

Original: Recent work by (Gleave et al., 2019) consider an alternative setting where the agent is attacked by another agent (known as adversarial policy), which is different from the two threat models considered in this paper.
Revised: Recent work by (Gleave et al., 2019) consider an alternative setting where the agent is attacked by another agent (known as adversarial policy), which is different from the two threat models considered in this paper.

Original: Finally, besides adversarial attacks in deep RL, a recent work (Wang et al., 2019) study veriﬁcation of deep RL agent under attacks, which is beyond the scope of this paper.
Revised: Finally, besides adversarial attacks in deep RL, a recent work (Wang et al., 2019) study veriﬁcation of deep RL agent under attacks, which is beyond the scope of this paper.

Original: Learning dynamics models.
Revised: Learning dynamics models.

Original: Model-based RL methods ﬁrst acquire a predictive model of the environment dynamics, and then use that model to make decisions (Atkeson & Santamaria, 1997).
Revised: Model-based RL methods ﬁrst acquire a predictive model of the environment dynamics, and then use that model to make decisions (Atkeson & Santamaria, 1997).

Original: These model-based methods tend to be more sample efﬁcient than their model-free counterparts, and the learned dynamics models can be useful across different tasks.
Revised: These model-based methods tend to be more sample efﬁcient than their model-free counterparts, and the learned dynamics models can be useful across different tasks.

Original: Various works have focused on the most effective ways to learn and utilize dynamics models for planning in RL (Kurutach et al., 2018;
Revised: Various works have focused on the most effective ways to learn and utilize dynamics models for planning in RL (Kurutach et al., 2018;

Original: Chua et al., 2018; Chiappa et al., 2017; Fu et al., 2016).
Revised: Chua et al., 2018; Chiappa et al., 2017; Fu et al., 2016).

Original:  P ROPOSED FRAMEWORK
Revised:  P ROPOSED FRAMEWORK

Original: In this section, we ﬁrst describe the problem setup and the two threat models considered in this paper.
Revised: In this section, we ﬁrst describe the problem setup and the two threat models considered in this paper.

Original: Next, we present an algorithmic framework to rigorously design adversarial attacks on deep
Revised: Next, we present an algorithmic framework to rigorously design adversarial attacks on deep

Original: RL agents with continuous actions.
Revised: RL agents with continuous actions.

Original: 3. P ROBLEM SETUP AND FORMULATION
Revised: 3. P ROBLEM SETUP AND FORMULATION

Original: Let s i ∈ R N and a i ∈ R M be the observation vector and action vector at time step i , and let π : R N → R M be the deterministic policy (agent).
Revised: Let s i ∈ R N and a i ∈ R M be the observation vector and action vector at time step i , and let π : R N → R M be the deterministic policy (agent).

Original: Let f : R N × R M
Revised: Let f : R N × R M

Original: → R N be the dynamics model of the system (environment) which takes current state-action pair ( s i , a i ) as inputs and outputs the next state
Revised: → R N be the dynamics model of the system (environment) which takes current state-action pair ( s i , a i ) as inputs and outputs [Figure]

Original:  s i +1 .
Revised: the next state s i +1 .

Original: We are now in the role of an adversary, and as an adversary, our goal is to drive the agent to the (un-safe) target states s target within the (cid:15) budget constraints.
Revised: We are now in the role of an adversary, and as an adversary, our goal is to drive the agent to the (un-safe) target states s target within the (cid:15) budget constraints.

Original: We can formulate this goal into two optimization problems, as we will illustrate shortly below.
Revised: We can formulate this goal into two optimization problems, as we will illustrate shortly below.

Original: Within this formalism, we can consider two threat models:
Revised: Within this formalism, we can consider two threat models:

Original: Threat model (i): Observation manipulation.
Revised: Threat model (i): Observation manipulation.

Original: For the threat model of observation manipulation, an adversary is allowed to manipulate the observation s i that the agent perceived within an (cid:15) budget:
Revised: For the threat model of observation manipulation, an adversary is allowed to manipulate the observation s i that the agent perceived within an (cid:15) budget:

Original: [Equation]
Revised: [Equation]

Original: where ∆ s i ∈ R N is the crafted perturbation and U s ∈ R N , L s ∈ R N are the observation limits.
Revised: where ∆ s i ∈ R N is the crafted perturbation and U s ∈ R N , L s ∈ R N are the observation limits.

Original: Threat model (ii): Action manipulation.
Revised: Threat model (ii): Action manipulation.

Original: For the threat model of action manipulation, an adversary can craft ∆ a i ∈ R M such that
Revised: For the threat model of action manipulation, an adversary can craft ∆ a i ∈ R M such that

Original: [Equation]
Revised: [Equation]

Original: where U a ∈ R M , L a ∈ R M are the limits of agent’s actions.
Revised: where U a ∈ R M , L a ∈ R M are the limits of agent’s actions.

Original: Our formulations.
Revised: Our formulations.

Original: Given an initial state s 0 and a pre-trained policy π , our (adversary) objective is to minimize the total distance of each state s i to the pre-deﬁned target state s target up to the unrolled (planning) steps T .
Revised: Given an initial state s 0 and a pre-trained policy π , our (adversary) objective is to minimize the total distance of each state s i to the pre-deﬁned target state s target up to the unrolled (planning) steps T .

Original: This can be written as the following optimization problems in Equations 3 and 4 for the Threat model (i) and (ii) respectively:
Revised: This can be written as the following optimization problems in Equations 3 and 4 for the Threat model (i) and (ii) respectively:

Original: [Equation]
Revised: [Equation]

Original: A common choice of d ( x, y ) is the squared (cid:96) 2 distance (cid:107) x − y (cid:107) 22 and f is the learned dynamics model of the system, and T is the unrolled (planning) length using the dynamics models.
Revised: A common choice of d ( x, y ) is the squared (cid:96) 2 distance (cid:107) x − y (cid:107) 22 and f is the learned dynamics model of the system, and T is the unrolled (planning) length using the dynamics models.

Original: 3. O UR ALGORITHM
Revised: 3. O UR ALGORITHM

Original: In this section, we propose a two-step algorithm to solve Equations 3 and 4.
Revised: In this section, we propose a two-step algorithm to solve Equations 3 and 4.

Original: The core of our proposal consists of two important steps: learn a dynamics model f of the environment and deploy optimization technique to solve Equations 3 and 4.
Revised: The core of our proposal consists of two important steps: learn a dynamics model f of the environment and deploy optimization technique to solve Equations 3 and 4.

Original: We ﬁrst discuss the details of each factor, and then present the full algorithm by the end of this section.
Revised: We ﬁrst discuss the details of each factor, and then present the full algorithm by the end of this section.

Original: Step 1: learn a good dynamics model f .
Revised: Step 1: learn a good dynamics model f .

Original: Ideally, if f is the exact (perfect) dynamics model of the environment and assuming we have an optimization oracle to solve Equations 3 and 4, then the solutions are indeed the optimal adversarial perturbations that give the minimal total loss with (cid:15) -budget constraints.
Revised: Ideally, if f is the exact (perfect) dynamics model of the environment and assuming we have an optimization oracle to solve Equations 3 and 4, then the solutions are indeed the optimal adversarial perturbations that give the minimal total loss with (cid:15) -budget constraints.

Original: Thus, learning a good dynamics model can conceptually help on developing a strong attack.
Revised: Thus, learning a good dynamics model can conceptually help on developing a strong attack.

Original: Depending on the environment, different forms of f can be applied.
Revised: Depending on the environment, different forms of f can be applied.

Original: For example, if the environment of concerned is close to a linear system, then we could let f ( s, a ) =
Revised: For example, if the environment of concerned is close to a linear system, then we could let f ( s, a ) =

Original: As + Bu , where A and B are unknown matrices to be learned from the sample trajectories ( s i , a i , s i +1 ) pairs.
Revised: As + Ba , where A and B are unknown matrices to be learned from the sample trajectories ( s i , a i , s i +1 ) pairs.

Original: For a more complex environment, we could decide if we still want to use a simple linear model (the next state prediction may be far deviate from the true next state and thus the learned dynamical model is less useful) or instead switch to a non-linear model, e.g. neural networks, which usually has better prediction power but may require more training samples.
Revised: For a more complex environment, we could decide if we still want to use a simple linear model (the next state prediction may be far deviate from the true next state and thus the learned dynamical model is less useful) or instead switch to a non-linear model, e.g. neural networks, which usually has better prediction power but may require more training samples.

Original: For either case, the model parameters A, B or neural network parameters can be learned via standard supervised learning with the sample trajectories pairs ( s i , a i , s i +1 ) .
Revised: For either case, the model parameters A, B or neural network parameters can be learned via standard supervised learning with the sample trajectories pairs ( s i , a i , s i +1 ) .

Original: Step 2: solve Equations 3 and 4.
Revised: Step 2: solve Equations 3 and 4.

Original: Once we learned a dynamical model f , the next immediate task is to solve Equation 3 and 4 to compute the adversarial perturbations of observations/actions.
Revised: Once we learned a dynamical model f , the next immediate task is to solve Equation 3 and 4 to compute the adversarial perturbations of observations/actions.

Original: When the planning (unrolled) length T > 1 , Equation 3 usually can not be directly solved by off-theshelf convex optimization toolbox since the deel RL policy π is usually a non-linear and non-convex neural network.
Revised: When the planning (unrolled) length T > 1 , Equation 3 usually can not be directly solved by off-theshelf convex optimization toolbox since the deel RL policy π is usually a non-linear and non-convex neural network.

Original: Fortunately, we can incorporate the two equality constraints of Equation 3 into the objective and with the remaining (cid:15) -budget constraint (Equation 1), Equation 3 can be solved via projected gradient descent (PGD) 1 .
Revised: Fortunately, we can incorporate the two equality constraints of Equation 3 into the objective and with the remaining (cid:15) -budget constraint (Equation 1), Equation 3 can be solved via projected gradient descent (PGD) 1 .

Original: Similarly, Equation 4 can be solved via PGD to get ∆ a i .
Revised: Similarly, Equation 4 can be solved via PGD to get ∆ a i .

Original: We note that, similar to the n -step model predictive control, our algorithm could use a much larger planning (unrolled) length T when solving Equations 3 and 4 and then only apply the ﬁrst n ( ≤ T ) adversarial perturbations on the agent over n time steps.
Revised: We note that, similar to the n -step model predictive control, our algorithm could use a much larger planning (unrolled) length T when solving Equations 3 and 4 and then only apply the ﬁrst n ( ≤ T ) adversarial perturbations on the agent over n time steps.

Original: Besides, with the PGD framework, f is not limited to feed-forward neural networks.
Revised: Besides, with the PGD framework, f is not limited to feed-forward neural networks.

Original: Our proposed attack is summarized in Algorithm 2 for
Revised: Our proposed attack is summarized in Algorithm 2 for

Original: Step 1 , and Algorithm 3 for Step 2 .
Revised: Step 1 , and Algorithm 3 for Step 2 .

Original: S

Original: 10: end

Original: : Return

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we conduct experiments on standard reinforcement learning environment for continuous control (Tassa et al., 2018).
Revised: In this section, we conduct experiments on standard reinforcement learning environment for continuous control (Tassa et al., 2018).

Original: We demonstrate results on 4 different environments in MuJoCo Tassa et al.
Revised: We demonstrate results on 4 different environments in MuJoCo Tassa et al.

Original: and corresponding tasks: Cartpole-balance/swingup, Fish-upright, Walkerstand/walk and Humanoid-stand/walk.
Revised: and corresponding tasks: Cartpole-balance/swingup, Fish-upright, Walkerstand/walk and Humanoid-stand/walk.

Original: For the deep RL agent, we train a state-of-the-art D4PG[Table] [Table] agent (Barth-Maron et al., 2018) with default Gaussian noise N ( 0 , 0 .
Revised: For the deep RL agent, we train a state-of-the-art D4PG agent (Barth-Maron et al., 2018) with default Gaussian noise N ( 0 , 0 .

Original: 3I ) on the action.
Revised: 3I ) on the action.

Original: The organization is as follows: we ﬁrst evaluate the effectiveness of our proposed model-based attack and three model-free baselines in terms of both loss and reward.
Revised: The organization is as follows: we ﬁrst evaluate the effectiveness of our proposed model-based attack and Algorithm 1: Input: pre-trained policy π , learned dynamical model f

Revised: ( s, a ; W ) , threat model, maximum perturbation magnitude (cid:15) , unroll length T , apply perturbation length n ( ≤ T ) 2:

Revised: Output: a sequence of perturbation δ 1 , . . .

Original: Next, we demonstrate the efﬁciency of our proposed attack in terms of sample complexity.
Revised: , δ n 3: if threat model is observation manipulation (Eq. 1) then 4: Solve Eq. 3 with parameters ( π, f, (cid:15), T ) via PGD to get δ 1 , . . , δ T 5: else if threat model is action manipulation (Eq. 2) then 6:

Revised: Solve Eq.

Revised: 4 with parameters ( π, f, (cid:15), T ) via PGD to get δ 1 , . . .

Revised: , δ T 7: end if 8: Return δ 1 , . .

Revised: , δ n three model-free baselines in terms of both loss and reward.

Revised: Next, we demonstrate the efﬁciency of our proposed attack in terms of sample complexity.

Original: Evaluations.
Revised: Evaluations.

Original: We conduct experiments for 10 different runs, where the environment is reset to different initial states in different runs.
Revised: We conduct experiments for 10 different runs, where the environment is reset to different initial states in different runs.

Original: For each run, we attack the agent for one episode withtime steps (the default time intervals is usually 10 ms) and we compute the total loss and total return reward.
Revised: For each run, we attack the agent for one episode withtime steps (the default time intervals is usually 10 ms) and we compute the total loss and total return reward.

Original: The total loss calculates the total distance of current state to the unsafe states and the total return reward measures the true accumulative reward from the environment based on agent’s action.
Revised: The total loss calculates the total distance of current state to the unsafe states and the total return reward measures the true accumulative reward from the environment based on agent’s action.

Original: Hence, the attack algorithm is stronger if the total return reward and the total loss are smaller.
Revised: Hence, the attack algorithm is stronger if the total return reward and the total loss are smaller.

Original: Baselines.
Revised: Baselines.

Original: We compare our algorithm with the following model-free attack baselines with random searches and heuristics:
Revised: We compare our algorithm with the following model-free attack baselines with random searches and heuristics:

Original: • rand-U : generate m randomly perturbed trajectories from Uniform distribution with interval [ − (cid:15), (cid:15) ] and return the trajectory with the smallest loss (or reward), • rand-B : generate m randomly perturbed trajectories from Bernoulli distribution with probability 1 / 2 and interval [ − (cid:15), (cid:15) ] , and return the trajectory with the smallest loss (or reward), • ﬂip : generate perturbations by ﬂipping agent’s observations/actions within the (cid:15) budget in (cid:96) ∞ norm.
Revised: • rand-U : generate m randomly perturbed trajectories from Uniform distribution with interval [ − (cid:15), (cid:15) ] and return the trajectory with the smallest loss (or reward), • rand-B : generate m randomly perturbed trajectories from Bernoulli distribution with probability 1 / 2 and interval [ − (cid:15), (cid:15) ] , and return the trajectory with the smallest loss (or reward), • ﬂip : generate perturbations by ﬂipping agent’s observations/actions within the (cid:15) budget in (cid:96) ∞ norm.

Original: For rand-U and rand-B , they are similar to Monte-Carlo sampling methods, where we generate m sample trajectories from random noises and report the loss/reward of the best trajectory (with minimum loss or reward among all the trajectories).
Revised: For rand-U and rand-B , they are similar to Monte-Carlo sampling methods, where we generate m sample trajectories from random noises and report the loss/reward of the best trajectory (with minimum loss or reward among all the trajectories).

Original: We set m = 1000 throughout the experiments.
Revised: We set m = 1000 throughout the experiments.

Original: Our algorithm.
Revised: Our algorithm.

Original: A 4-layer feed-forward neural network with 1000 hidden neurons per layer is trained as the dynamics model f respectively for the domains of Cartpole, Fish, Walker and Humanoid.
Revised: A 4-layer feed-forward neural network with 1000 hidden neurons per layer is trained as the dynamics model f respectively for the domains of Cartpole, Fish, Walker and Humanoid.

Original: We use standard (cid:96) 2 loss (without regularization) to learn a dynamics model f .
Revised: We use standard (cid:96) 2 loss (without regularization) to learn a dynamics model f .

Original: Instead of using recurrent neural network to represent f , we found that the 1-step prediction for dynamics with the 4-layer feed-forward network is already good for the MuJoCo domains we are studying.
Revised: Instead of using recurrent neural network to represent f , we found that the 1-step prediction for dynamics with the 4-layer feed-forward network is already good for the MuJoCo domains we are studying.

Original: Specifically, for the Cartpole and Fish, we found that 1000 episodes ( 1 e 6 training points) are sufﬁcient to train a good dynamics model (the mean square error for both training and test losses are at the order of 10 − 5 for Cartpole and 10 − 2 for Fish), while for the more complicated domain like Walker and Humanoid, more training points ( 5 e 6 ) are required to achieve a low test MSE error (at the order of 10 − 1 and 10 0 for Walker and Humanoid respectively).
Revised: Specifically, for the Cartpole and Fish, we found that 1000 episodes ( 1 e 6 training points) are sufﬁcient to train a good dynamics model (the mean square error for both training and test losses are at the order of 10 − 5 for Cartpole and 10 − 2 for Fish), while for the more complicated domain like Walker and Humanoid, more training points ( 5 e 6 ) are required to achieve a low test MSE error (at the order of 10 − 1 and 10 0 for Walker and Humanoid respectively).

Original: Consequently, we use larger planning (unrolled) length for Cartpole and Fish (e.g. T = 10 , 20 ), while a smaller T (e.g. 3 or 5 ) is used for Walker and Humanoid.
Revised: Consequently, we use larger planning (unrolled) length for Cartpole and Fish (e.g. T = 10 , 20 ), while a smaller T (e.g. 3 or 5 ) is used for Walker and Humanoid.

Original: Meanwhile, we focus on applying projected gradient descent (PGD) to solve Equation 3 and 4.
Revised: Meanwhile, we focus on applying projected gradient descent (PGD) to solve Equation 3 and 4.

Original: We use Adam as the optimizer with optimization steps equal to 30 and we report the best result for each run from a combination of 6 learning rates, 2 unroll length { T 1 , T 2 } and n steps of applying PGD solution with n ≤ T i .
Revised: We use Adam as the optimizer with optimization steps equal to 30 and we report the best result for each run from a combination of 6 learning rates, 2 unroll length { T 1 , T 2 } and n steps of applying PGD solution with n ≤ T i .

Original: 4. R ESULTS
Revised: 4. R ESULTS

Original: For observation manipulation, we report the results on Walker, Humanoid and Cartpole domains with tasks (stand, walk, balance, swingup) respectively.
Revised: For observation manipulation, we report the results on Walker, Humanoid and Cartpole domains with tasks (stand, walk, balance, swingup) respectively.

Original: The unsafe states s target for Walker and
Revised: The unsafe states s target for Walker and

Original: Humanoid are set to be zero head height, targeting the situation of falling down.
Revised: Humanoid are set to be zero head height, targeting the situation of falling down.

Revised: For Cartpole, the

Revised: [Table]

Revised: [Table]

Original: For Cartpole, the  unsafe states are set to have 180 ◦ pole angle, corresponding to the cartpole not swinging up and nor balanced.
Revised: [Table] [Table] Only our proposed attack can constantly make the Walker fall down (since we are minimizing its head height to be zero). unsafe states are set to have 180 ◦ pole angle, corresponding to the cartpole not swinging up and nor balanced.

Original: For the Fish domain, the unsafe states for the upright task target the pose of swimming ﬁsh to be not upright, e.g. zero projection on the z -axis.
Revised: For the Fish domain, the unsafe states for the upright task target the pose of swimming ﬁsh to be not upright, e.g. zero projection on the z -axis.

Original: The full results of both two threat models on observation manipulation and action manipulation are shown in Table 1a, b and c, d respectively.
Revised: The full results of both two threat models on observation manipulation and action manipulation are shown in Table 1a, b and c, d respectively.

Original: Since the loss is deﬁned as the distance to the target (unsafe) state, the lower the loss, the stronger the attack.
Revised: Since the loss is deﬁned as the distance to the target (unsafe) state, the lower the loss, the stronger the attack.

Original: It is clear that our proposed attack achieves much lower loss in Table 1a & c than the other three model-free baselines, and the averaged ratio is also listed in 1b & d. Notably, over the 10 runs, our proposed attack always outperforms baselines for the threat model of observation perturbation and the Cartpole domain for the threat model of action perturbation, while still superior to the baselines despite losing two times to the ﬂip baseline on the Fish domain.
Revised: It is clear that our proposed attack achieves much lower loss in Table 1a & c than the other three model-free baselines, and the averaged ratio is also listed in 1b & d. Notably, over the 10 runs, our proposed attack always outperforms baselines for the threat model of observation perturbation and the Cartpole domain for the threat model of action perturbation, while still superior to the baselines despite losing two times to the ﬂip baseline on the Fish domain.

Original: Only our proposed attack can constantly make the Walker fall down (since we are minimizing its head height to be zero).

Original: To have a better sense on the numbers, we give some quick examples below.
Revised: To have a better sense on the numbers, we give some quick examples below.

Original: For instance, as shown in Table 1a and b, we show that the average total loss of walker head height is almost unaffected for the three baselines – if the walker successfully stand or walk, its head height usually has to be greater than 1 . 2 at every time step, which is 1440 for one episode – while our attack can successfully lower the walker head height by achieving an average of total loss of 258(468) , which is roughly 0 .
Revised: For instance, as shown in Table 1a and b, we show that the average total loss of walker head height is almost unaffected for the three baselines – if the walker successfully stand or walk, its head height usually has to be greater than 1 . 2 at every time step, which is 1440 for one episode – while our attack can successfully lower the walker head height by achieving an average of total loss of 258(468) , which is roughly 0 .

Original: 68) per time step for the stand (walk) task.
Revised: 68) per time step for the stand (walk) task.

Original: Similarly, for the humanoid results, a successful humanoid usually has head height greater than 1 . 4 , equivalently a total loss of 1960 for one episode, and Table 1a shows that the d4pg agent is robust to the perturbations generated from the three modelfree baselines while being vulnerable to our proposed attack.
Revised: Similarly, for the humanoid results, a successful humanoid usually has head height greater than 1 . 4 , equivalently a total loss of 1960 for one episode, and Table 1a shows that the d4pg agent is robust to the perturbations generated from the three modelfree baselines while being vulnerable to our proposed attack.

Original: Indeed, as shown in Figure 2, the walker and humanoid falls down quickly (head height is close to zero) under our specially-designed attack while remaining unaffected for all the other baselines.
Revised: Indeed, as shown in Figure 2, the walker and humanoid falls down quickly (head height is close to zero) under our specially-designed attack while remaining unaffected for all the other baselines.

Original: 4. D ISCUSSION
Revised: 4. D ISCUSSION

Original: Evaluating on the total reward.
Revised: Evaluating on the total reward.

Original: Often times, the reward function is a complicated function and its exact deﬁnition is often unavailable.
Revised: Often times, the reward function is a complicated function and its exact deﬁnition is often unavailable.

Original: Learning the reward function is also an active research ﬁeld, which is not in the coverage of this paper.
Revised: Learning the reward function is also an active research ﬁeld, which is not in the coverage of this paper.

Original: Nevertheless, as long as we have some knowledge of unsafe states (which is often the case in practice), then we can deﬁne unsafe states that are related to low reward and thus performing attacks based on unsafe states (i.e. minimizing the total loss of distance to unsafe states) would naturally translate to decreasing the total reward of agent.
Revised: Nevertheless, as long as we have some knowledge of unsafe states (which is often the case in practice), then we can deﬁne unsafe states that are related to low reward and thus performing attacks based on unsafe states (i.e. minimizing the total loss of distance to unsafe states) would naturally translate to decreasing the total reward of agent.

Original: As demonstrated in Table 2, the results have the same trend of the total loss result in Table 1, where our proposed attack signiﬁcantly outperforms all the other three baselines.
Revised: As demonstrated in Table 2, the results have the same trend of the total loss result in Table 1, where our proposed attack signiﬁcantly outperforms all the other three baselines.

Original: In particular, our method can lower the average total reward up to 4 .
Revised: In particular, our method can lower the average total reward up to 4 .

Original: 96 × compared to the baselines result, while the baseline results are close to the perfect total reward of 1000 . 
Revised: 96 × compared to the baselines result, while the baseline results are close to the perfect total reward of 1000 . [Table] [Table] [Table]

Revised: [Table]

Original: Evaluating on the efﬁciency of attack.
Revised: Evaluating on the efﬁciency of attack.

Original: We also study the efﬁciency of the attack in terms of sample complexity, i.e. how many episodes do we need to perform an effective attack?
Revised: We also study the efﬁciency of the attack in terms of sample complexity, i.e. how many episodes do we need to perform an effective attack?

Original: Here we adopt the convention in control suite (Tassa et al., 2018) where one episode corresponds totime steps (samples), and we learn the neural network dynamical model f with different number of episodes.
Revised: Here we adopt the convention in control suite (Tassa et al., 2018) where one episode corresponds totime steps (samples), and we learn the neural network dynamical model f with different number of episodes.

Original: Figure 3 plots the total head height loss of the walker (task stand) for the three baselines and our method with dynamical model f trained with three different number of samples: { 5 e 5 , 1 e 6 , 5 e 6 } , or equivalently { 500 , 1000 , 5000 } episodes.
Revised: Figure 3 plots the total head height loss of the walker (task stand) for the three baselines and our method with dynamical model f trained with three different number of samples: { 5 e 5 , 1 e 6 , 5 e 6 } , or equivalently { 500 , 1000 , 5000 } episodes.

Original: We note that the sweep of hyper parameters is the same for all the three models, and the only difference is the number of training samples.
Revised: We note that the sweep of hyper parameters is the same for all the three models, and the only difference is the number of training samples.

Original: The results show that for the baselines rand-U and ﬂip , the total losses are roughly at the order of 1400 - 1500 , while [Table] [Table] [Table]
Revised: The results show that for the baselines rand-U and ﬂip , the total losses are roughly at the order of 1400 - 1500 , while a stronger baseline rand-B still has total losses of 900 - 1200 .

Original: [Table]

Original: a stronger baseline rand-B still has total losses of 900 - 1200 .

Original: However, if we solve Equation 3 with f trained by 5 e 5 or 1 e 6 samples [Table] the total losses can be decreased to the order of 400 - 700 and are already winning over the three baselines by a signiﬁcant margin.
Revised: However, if we solve Equation 3 with f trained by 5 e 5 or 1 e 6 samples, the total losses can be decreased to the order of 400 - 700 and are already winning over the three baselines by a signiﬁcant margin.

Original: Same as our expectation, if we use more samples (e.g. 5 e 6 , which is 5 - 10 times more), to learn a more accurate dynamics model, then it is beneﬁcial to our attack method – the total losses can be further decreased by more than 2 × and are at the order of 50 - 250 over 10 different runs.
Revised: Same as our expectation, if we use more samples (e.g. 5 e 6 , which is 5 - 10 times more), to learn a more accurate dynamics model, then it is beneﬁcial to our attack method – the total losses can be further decreased by more than 2 × and are at the order of 50 - 250 over 10 different runs.

Original: Here we also give a comparison between our model-based attack to existing works (Uesato et al., 2018; Gleave et al., 2019) on the sample complexity.
Revised: Here we also give a comparison between our model-based attack to existing works (Uesato et al., 2018; Gleave et al., 2019) on the sample complexity.

Original: In (Uesato et al., 2018), 3 e 5 episodes of training data is used to learn the adversarial value function, which is roughly 1000 × more data than even our strongest adversary (with 5 e 3 episodes).
Revised: In (Uesato et al., 2018), 3 e 5 episodes of training data is used to learn the adversarial value function, which is roughly 1000 × more data than even our strongest adversary (with 5 e 3 episodes).

Original: Similarly, (Gleave et al., 2019) use roughly 2 eepisodes to train an adversary via deep RL, which is roughly 4 × more data than ours 2 .
Revised: Similarly, (Gleave et al., 2019) use roughly 2 eepisodes to train an adversary via deep RL, which is roughly 4 × more data than ours 2 .

Original:  C ONCLUSIONS AND FUTURE WORKS
Revised:  C ONCLUSIONS AND FUTURE WORKS

Original: In this paper, we study the problem of adversarial attacks in deep RL with continuous control for two commonly-used threat models (observation manipulation and action manipulation).
Revised: In this paper, we study the problem of adversarial attacks in deep RL with continuous control for two commonly-used threat models (observation manipulation and action manipulation).

Original: Based on the threat models, we proposed the ﬁrst model-based attack algorithm and showed that our formulation can be easily solved by off-the-shelf gradient-based solvers.
Revised: Based on the threat models, we proposed the ﬁrst model-based attack algorithm and showed that our formulation can be easily solved by off-the-shelf gradient-based solvers.

Original: Through extensive experiments on 4 MuJoCo domains (Cartpole, Fish, Walker, Humanoid), we show that our proposed algorithm outperforms all the model-free based attack baselines by a large margin.
Revised: Through extensive experiments on 4 MuJoCo domains (Cartpole, Fish, Walker, Humanoid), we show that our proposed algorithm outperforms all the model-free based attack baselines by a large margin.

Original: There are several interesting future directions can be investigated based on this work and  is detailed in Appendix.
Revised: There are several interesting future directions can be investigated based on this work, including learning reward functions to facilitate a more effective attack, extending our current approach to develop effective black-box attacks, and incorporating our proposed attack algorithm to adversarial training of the deep RL agents. In particular, we think there are three important challenges that need to be addressed to study adversarial training of RL agents along with our proposed attacks: The adversary and model need to be jointly updated. How do we balance these two updates, and make sure the adversary is well-trained at each point in training?

Revised: How do we avoid cycles in the training process due to the agent overﬁtting to the current adversary?

Revised: How do we ensure the adversary doesn’t overly prevent exploration / balance unperturbed vs. robust performance?

Original: M ORE ILLUSTRATION ON F IGURE
Revised: A PPENDIX

Revised: A. M ORE ILLUSTRATION ON F IGURE

Original: The meaning of Fig 3 is to show how the accuracy of the learned models affects our proposed technique:
Revised: The meaning of Fig 3 is to show how the accuracy of the learned models affects our proposed technique:

Original: we ﬁrst learned 3 models with 3 different number of samples: 5e5, 1e6, 5e6 and we found that with more training samples (e.g. 5e6, equivalently 5000 episodes), we are able to learn a more accurate model than the one with 5e5 training samples; 2.
Revised: we ﬁrst learned 3 models with 3 different number of samples: 5e5, 1e6, 5e6 and we found that with more training samples (e.g. 5e6, equivalently 5000 episodes), we are able to learn a more accurate model than the one with 5e5 training samples; 2.

Original: we plot the attack results of total loss for our technique with 3 learned models (denoted as PGD, num train) as well as the baselines (randU, randB, Flip) on 10 different runs (initializations).
Revised: we plot the attack results of total loss for our technique with 3 learned models (denoted as PGD, num train) as well as the baselines (randU, randB, Flip) on 10 different runs (initializations).

Original: We show with the more accurate learned model (5e6 training samples), we are able to achieve a stronger attack (the total losses are at the order of 50-200 over 10 different runs) than the less accurate learned model (e.g. 5e5 training samples).
Revised: We show with the more accurate learned model (5e6 training samples), we are able to achieve a stronger attack (the total losses are at the order of 50-200 over 10 different runs) than the less accurate learned model (e.g. 5e5 training samples).

Original: However, even with a less accurate learned model, the total losses are on the order of 400-700, which already outperforms the best baselines by a margin of 1.3-2 times.
Revised: However, even with a less accurate learned model, the total losses are on the order of 400-700, which already outperforms the best baselines by a margin of 1.3-2 times.

Original: This result in Fig 3 also suggest that a very accurate model isn’t necessarily needed in our proposed method to achieve effective attack.
Revised: This result in Fig 3 also suggest that a very accurate model isn’t necessarily needed in our proposed method to achieve effective attack.

Original: Of course, if the learned model is more accurate, then we are able to degrade agent’s performance even more.
Revised: Of course, if the learned model is more accurate, then we are able to degrade agent’s performance even more.

Original: [Figure]
Revised: [Figure]

Original: M ORE DETAILS ON B ASELINE IMPLEMENTATIONS
Revised: M ORE DETAILS ON B ASELINE IMPLEMENTATIONS

Original: For the baselines (rand-U and rand-B), the adversary generates 1000 trajectories with random noise directly and we report the best loss/reward at the end of each episode.
Revised: For the baselines (rand-U and rand-B), the adversary generates 1000 trajectories with random noise directly and we report the best loss/reward at the end of each episode.

Original: The detailed steps are listed below:Step 1: The
Revised: The detailed steps are listed below:

Original:  perturbations are generated from a uniform distribution or a bernoulli distribution within the range eps, [-eps
Revised: Step 1: The perturbations are generated from a uniform distribution or a bernoulli distribution within the range eps] for each trajectory, and we record the total reward and total loss for each trajectory from the true environment (the MuJoCo simulator)

Original: for each trajectory, and

Original: we

Original: record the total reward and total loss for each trajectory from the true environment (the MuJoCo simulator Step 2: Take the best (lowest) total reward/loss among 1000 trajectories and report in Table 1 and2.
Revised: Step 2: Take the best (lowest) total reward/loss among 1000 trajectories and report in Table 1 and

Original: We note that here we assume the baseline adversary has an ”unfair advantage” since they have access to the true reward (and then take the best attack result among 1000 trials), whereas our techniques do not have access to this information.
Revised: We note that here we assume the baseline adversary has an ”unfair advantage” since they have access to the true reward (and then take the best attack result among 1000 trials), whereas our techniques do not have access to this information.

Original: Without this advantage, the baseline adversaries (rand-B, rand-U) may be weaker if they use their learned model to ﬁnd the best attack sequence.
Revised: Without this advantage, the baseline adversaries (rand-B, rand-U) may be weaker if they use their learned model to ﬁnd the best attack sequence.

Original: In any case,
Revised: In any case,

Original: Table 1 and 2 demonstrate that our proposed attack can successfully uncover vulnerabilities of deep
Revised: Table 1 and 2 demonstrate that our proposed attack can successfully uncover vulnerabilities of deep

Original: RL agents while the baselines cannot.
Revised: RL agents while the baselines cannot.

Original: For the baseline ’ﬂip’, we add the perturbation (with the opposite sign and magnitude (cid:15) ) on the original state/action and project the perturbed state/action are within its limits.
Revised: For the baseline ’ﬂip’, we add the perturbation (with the opposite sign and magnitude (cid:15) ) on the original state/action and project the perturbed state/action are within its limits.

Original: A. S CORE OF LEARNED POLICY WITHOUT ATTACKS
Revised: A. S CORE OF LEARNED POLICY WITHOUT ATTACKS

Original: We use default total timesteps = 1000, and the maximum total reward is 1000.
Revised: We use default total timesteps = 1000, and the maximum total reward is 1000.

Original: We report the total reward of the d4pg agents used in this paper below.
Revised: We report the total reward of the d4pg agents used in this paper below.

Original: The agents are well-trained and have total reward close to 1000. [Table]
Revised: The agents are well-trained and have total reward close to 1000, which outperforms agents trained by other learning algorithms on the same tasks (e.g. DDPG, A3C in Sec 6 (Tassa et al., 2018);

Original: There are several interesting future directions can be investigated based on this work, including learning reward functions to facilitate a more effective attack, extending our current approach to develop effective black-box attacks, and incorporating our proposed attack algorithm to adversarial training of the deep RL agents. In particular, we think there are three important challenges that need to be addressed to study adversarial training of RL agents along with our proposed attacks: The adversary and model need to be jointly updated.
Revised: PPO in Sec 5 (Abdolmaleki et al., 2018)), and thus the agents in this paper can be regarded as state-of-the-art RL agents for these continuous control domain tasks.

Original: How do we balance these two updates, and make sure the adversary is well-trained at each point in training?
Revised: The attack results in Table 1 and 2 in our manuscript are hence suggested to be representative.

Original: How do we avoid cycles in the training process due to the agent overﬁtting to the current adversary? How do we ensure the adversary doesn’t overly prevent exploration / balance unperturbed vs. robust performance?
Revised: [Table]