Original: A study of natural robustness of deep reinforcement learning algorithms towards adversarial perturbationss AffiliationAddress
Revised: A study of natural robustness of deep reinforcement learning algorithms towards adversarial perturbations

Original: Abstract
Revised: Abstract

Original: Deep reinforcement learning (DRL) has been shown to have numerous potentialapplications in the real world.
Revised: Deep reinforcement learning (DRL) has been shown to have numerous potential applications in the real world.

Original: However, DRL algorithms are still extremelysensitive to noise and adversarial perturbations, hence inhibiting the deployment of
Revised: However, DRL algorithms are still extremely sensitive to noise and adversarial perturbations, hence inhibiting the deployment of

Original: RL in many real-life applications.
Revised: RL in many real-life applications.

Original: Analyzing the robustness of DRL algorithms toadversarial attacks is an important prerequisite to enabling the widespread adoptionof DRL algorithms.
Revised: Analyzing the robustness of DRL algorithms to adversarial attacks is an important prerequisite to enabling the widespread adoption of DRL algorithms.

Original: Common perturbations on DRL frameworks during test timeinclude perturbations to the observation and the action channel.
Revised: Common perturbations on DRL frameworks during test time include perturbations to the observation and the action channel.

Original: Compared withobservation channel attacks, action channel attacks are less studied; hence, fewcomparisons exist that compare the effectiveness of these attacks in DRL literature.
Revised: Compared with observation channel attacks, action channel attacks are less studied; hence, few comparisons exist that compare the effectiveness of these attacks in DRL literature.

Original: In this work, we examined the effectiveness of these two paradigms of attacks oncommon DRL algorithms and studied the natural robustness of DRL algorithmstowards various adversarial attacks in hopes of gaining insights into the individualresponse of each type of algorithm under different attack conditions.
Revised: In this work, we examined the effectiveness of these two paradigms of attacks on common DRL algorithms and studied the natural robustness of DRL algorithms towards various adversarial attacks in hopes of gaining insights into the individual response of each type of algorithm under different attack conditions.

Original:  Introduction
Revised:  Introduction

Original: Deep reinforcement learning (DRL) has seen substantial successes in multiple domains of applicationssuch as design [1], scheduling [2] and robotic control applications in industrial automation [3].
Revised: Deep reinforcement learning (DRL) has seen substantial successes in multiple domains of applications such as design [1], scheduling [2] and robotic control applications in industrial automation [3].

Original: Contrary to supervised learning, RL algorithms train an agent learns to perform a given task in anenvironment by making sequential actions and observing the resulting rewards to learn an optimalpolicy.
Revised: Contrary to supervised learning RL algorithms train an agent learns to perform a given task in an environment by making sequential actions and observing the resulting rewards to learn an optimal policy.

Original: In recent years, advancements in neural networks have led to the popularity of DRL, wherea deep neural network represents the RL policy.
Revised: In recent years, advancements in neural networks have led to the popularity of DRL where a deep neural network represents the RL policy.

Original: Although neural networks are powerful functionapproximators, they are also extremely easy to fool into making erroneous predictions by applyingperturbation on the model’s inputs [4].
Revised: Although neural networks are powerful function approximators, they are also extremely easy to fool into making erroneous predictions by applying perturbation on the model’s inputs [4].

Original: This observation led to numerous studies on the robustness ofdeep learning algorithms.
Revised: This observation led to numerous studies on the robustness of deep learning algorithms.

Original: A study by [5] proved that similar adversarial attacks could also be extendedto manipulate RL agents where the RL agent is vulnerable to subtle adversarial attacks that are notperceivable to humans but could cause a significant change in RL policy’s actions.
Revised: A study by [5] proved that similar adversarial attacks could also be extended to manipulate RL agents where the RL agent is vulnerable to subtle adversarial attacks that are not perceivable to humans but could cause a significant change in RL policy’s actions.

Original: Subsequently, thishas led to the development of several other successful adversarial attacks [6, 7, 8, 9, 10].
Revised: Subsequently, this has led to the development of several other successful adversarial attacks [6, 7, 8, 9, 10].

Original: While numerous works have developed DRL algorithms that are robust towards different perturbations [11, 12, 13, 14], to the best of our knowledge, a study that compares the response of popularbenchmark DRL algorithms towards common adversarial perturbations is still lacking in the literatureand this work aims to fill in such a gap.
Revised: While numerous works have developed DRL algorithms that are robust towards different perturbations [11, 12, 13, 14], to the best of our knowledge, a study that compares the response of popular benchmark DRL algorithms towards common adversarial perturbations is still lacking in the literature and this work aims to fill in such a gap.

Original: Specifically, in this work, we analyze the performance ofmultiple DRL algorithms commonly used in literature when subjected to observation and actionperturbations.
Revised: Specifically, in this work, we analyze the performance of multiple DRL algorithms commonly used in literature when subjected to observation and action perturbations.

Original: As a first step, we restrict our experiments to continuous state-action environments,which provide a more realistic proxy to industrial robotic applications, where adversarial attacks areof greater concern.
Revised: As a first step, we restrict our experiments to continuous state-action environments, which provide a more realistic proxy to industrial robotic applications, where adversarial attacks are of greater concern.

Original: Our experiments aim to answer the following questions: (1) Are the existing
Revised: Our experiments aim to answer the following questions: (1) Are the existing

Revised: Deep Reinforcement Learning Workshop at the 36th Conference on Neural Information Processing Systems (NeurIPS 2022).

Original: DRL algorithms especially sensitive to one class of adversarial perturbations over the other
Revised: DRL algorithms especially sensitive to one class of adversarial perturbations over the other?

Original: (e.g., observation vs. action space perturbations), (2) Is there a specific DRL algorithm that is naturallymore robust than all other algorithms under adversarial perturbations and (3) Is there a limitationof the magnitude of the perturbation on the degradation of the DRL performance, i.e., is there anempirically observable threshold in which perturbations below or above this threshold will not affectthe behavior of the DRL policy?
Revised: (e.g., observation vs. action space perturbations), (2) Is there a specific DRL algorithm that is naturally more robust than all other algorithms under adversarial perturbations and (3) Is there a limitation of the magnitude of the perturbation on the degradation of the DRL performance, i.e., is there an empirically observable threshold in which perturbations below or above this threshold will not affect the behavior of the DRL policy?

Original: Overall, our experiments serve to provide insight into existing DRLalgorithms’ potential natural robustness and act as a stepping stone to developing more robust DRLalgorithms.
Revised: Overall, our experiments serve to provide insight into existing DRL algorithms’ potential natural robustness and act as a stepping stone to developing more robust DRL algorithms.

Original:  Related Works
Revised:  Related Works

Original: Adversarial attacks on deep neural networks were first popularized by Szedegy et al.
Revised: Adversarial attacks on deep neural networks were first popularized by Szedegy et al.

Original: [15], whoperformed adversarial attacks on image classification algorithms by adding perturbations to the inputimages.
Revised: [15], who performed adversarial attacks on image classification algorithms by adding perturbations to the input images.

Original: These attacks aim to trick the model into misclassifying images.
Revised: These attacks aim to trick the model into misclassifying images.

Original: Generally, adversarial attackscan be divided into white-, gray- and black-box attacks, depending on the amount of knowledgethe adversary has about the machine learning (ML) model it is targeting [16].
Revised: Generally, adversarial attacks can be divided into white-, gray- and black-box attacks, depending on the amount of knowledge the adversary has about the machine learning (ML) model it is targeting [16].

Original: In white-box attacks,the adversary has complete knowledge about the target ML model, such as the learned weights,training parameters, and training and testing data.
Revised: In white-box attacks, the adversary has complete knowledge about the target ML model, such as the learned weights, training parameters, and training and testing data.

Original: With that information, these attacks can be usedto analyze worst-case scenarios [17].
Revised: With that information, these attacks can be used to analyze worst-case scenarios [17].

Original: In black-box attacks, the adversary has no knowledge of themodel or any of its parameters.
Revised: In black-box attacks, the adversary has no knowledge of the model or any of its parameters.

Original: Hence, black-box attacks are often established based on the modelinputs, confidence scores, or perturbing the feedback of the ML model [18].
Revised: Hence, black-box attacks are often established based on the model inputs, confidence scores, or perturbing the feedback of the ML model [18].

Original: Meanwhile, gray-boxattacks fall in between the spectrum of white- and black-box attacks, with the adversary having partialknowledge of the ML model being targeted.
Revised: Meanwhile, gray-box attacks fall in between the spectrum of white- and black-box attacks, with the adversary having partial knowledge of the ML model being targeted.

Original: In the context of adversarial attacks on DRL, in [6],the authors showed that Deep Q-Networks (DQN) are vulnerable to adversarial state perturbations.
Revised: In the context of adversarial attacks on DRL, in [6], the authors showed that Deep Q-Networks (DQN) are vulnerable to adversarial state perturbations.

Original: The adversarial perturbations were generated using the Fast Gradient Sign Method (FGSM) and
Revised: The adversarial perturbations were generated using the Fast Gradient Sign Method (FGSM) and

Original: Jacobian-based Saliency Map Attack (JSMA) [19].
Revised: Jacobian-based Saliency Map Attack (JSMA) [19].

Original: Additionally, they also implemented a black-boxand showed a success rate of 70%.
Revised: Additionally, they also implemented a black-box and showed a success rate of 70%.

Original: In [5], the authors employed similar attacks as in [6] butimplemented the attacks on other DRL algorithms such as DQN, Trust Region Policy Optimization(TRPO), and Asynchronous Advantage Actor-Critic (A3C) methods in both white and black-boxsettings.
Revised: In [5], the authors employed similar attacks as in [6] but implemented the attacks on other DRL algorithms such as DQN, Trust Region Policy Optimization (TRPO), and Asynchronous Advantage Actor-Critic (A3C) methods in both white and black-box settings.

Original: Their result demonstrates that DQN is more susceptible to adversarial attacks than others.
Revised: Their result demonstrates that DQN is more susceptible to adversarial attacks than others.

Original: Recent works have also demonstrated that observation-based adversarial perturbations can manifestthemselves in a multi-agent RL system where a corrupted agent can behave adversarially to fool theother RL agent [20].
Revised: Recent works have also demonstrated that observation-based adversarial perturbations can manifest themselves in a multi-agent RL system where a corrupted agent can behave adversarially to fool the other RL agent [20].

Revised: For clarification, we only consider a single-player environment and optimization base white box attack in this paper.

Revised: The multi-agent RL system [20] and the learning base attack [21] are not in the scope of our attacks.

Revised: That’s why we are not considering these papers.

Original: Besides perturbations in the observation space, the attacks can also occur in theaction space in the form of perturbations on the actuators.
Revised: Besides perturbations in the observation space, the attacks can also occur in the action space in the form of perturbations on the actuators.

Original: For example, Lee et al.
Revised: For example, Lee et al.

Original: [10] proposedcoupled spatial-temporal action space attacks that can reveal the potential vulnerabilities of the DRLmodel.
Revised: [10] proposed coupled spatial-temporal action space attacks that can reveal the potential vulnerabilities of the DRL model.

Original: Additionally, action space perturbations can also manifest in the form of environmental noiseor changes in environmental factors [11].
Revised: Additionally, action space perturbations can also manifest in the form of environmental noise or changes in environmental factors [11].

Original: For brevity, we refer interested readers to the more detailedand complete taxonomy of adversarial attacks presented in [21].
Revised: For brevity, we refer interested readers to the more detailed and complete taxonomy of adversarial attacks presented in [22].

Revised: On the other hand, environmental attacks aim to change the physical properties of the environment.

Revised: In our case, we are implementing the Mujoco environments, and the perturbations aren’t occurring in the physical DRL system.

Revised: Therefore, the feedback is accurately obtained from the environment.

Revised: That’s why the adversarial attack presented in [23] isn’t considered in this paper.

Original:  Methodology
Revised:  Methodology

Original: In this section, we provide a brief description of the experiments we conducted to compare theperformance and response of common DRL algorithms to adversarial attacks.
Revised: In this section, we provide a brief description of the experiments we conducted to compare the performance and response of common DRL algorithms to adversarial attacks.

Original: 3.1 Selection of environments and algorithms
Revised: 3.1 Selection of environments and algorithms

Original: We conducted our experiments on five different continuous control environments based on OpenAI
Revised: We conducted our experiments on five different continuous control environments based on OpenAI

Original: Gym [22] MuJoCo environments.
Revised: Gym [24] MuJoCo environments.

Original: The five selected environments are: i) Ant, ii)
Revised: The five selected environments are: i) Ant, ii)

Original: HalfCheetah, iii)
Revised: HalfCheetah, iii)

Original: Swimmer, iv) Walker, and v) Hopper.
Revised: Swimmer, iv) Walker, and v) Hopper.

Original: To facilitate a more accurate comparison, all experiments wererun with six random seeds, and for each seed, we ran 100 episodes and reported the average scoreacross all episodes and seeds.
Revised: To facilitate a more accurate comparison, all experiments were run with six random seeds, and for each seed, we ran 100 episodes and reported the average score across all episodes and seeds.

Original: In terms of selecting the DRL algorithms to compare, we selectedfive popular algorithms that are commonly used as benchmarks in continuous control tasks: 1)
Revised: In terms of selecting the DRL algorithms to compare, we selected five popular algorithms that are commonly used as benchmarks in continuous control tasks: 1)

Original: Proximal Policy Optimization (PPO) [23], 2) Deep Deterministic Policy Gradient (DDPG) [24], 3)
Revised: Proximal Policy Optimization (PPO) [25], 2) Deep Deterministic Policy Gradient (DDPG) [26], 3)

Original: Trust Region Policy Optimization (TRPO
Revised: Trust Region Policy Optimization (TRPO) [27], 4)

Revised: Twin Delayed DDPG (TD3) [28], and 5) Soft

Revised: Actor-Critic (SAC) [29].

Original: 3.2 Black-box attacks
Revised: 3.2 Black-box attacks

Original: Next, we describe the suite of black-box attacks that we implemented as part of our experimentsto compare the performance of the DRL agents.
Revised: Next, we describe the suite of black-box attacks that we implemented as part of our experiments to compare the performance of the DRL agents.

Original: As an initial step, we limit the scope of black-boxattacks in this paper to simple additive perturbations.
Revised: As an initial step, we limit the scope of black-box attacks in this paper to simple additive perturbations.

Original: However, we highlight that black-box attackstrategies extend beyond naive additive perturbations and will be a key focus of future studies.
Revised: However, we highlight that black-box attack strategies extend beyond naive additive perturbations and will be a key focus of future studies.

Original: To fully investigate the behavior of the policies in a comprehensive manner, we develop multiplestrategies for black-box attacks.
Revised: To fully investigate the behavior of the policies in a comprehensive manner, we develop multiple strategies for black-box attacks.

Original: These strategies were generated by identifying the three stageswhere the perturbations can be performed.
Revised: These strategies were generated by identifying the three stages where the perturbations can be performed.

Original: The first stage consists of the channel of perturbation,where the perturbation can either be added to the observations of the agent or the actions of theagent.
Revised: The first stage consists of the channel of perturbation, where the perturbation can either be added to the observations of the agent or the actions of the agent.

Original: The second stage involves the magnitude of perturbation, where the magnitude of perturbationis either random, bounded by the action space, or bounded by the magnitude of the actual actiontaken by the agent.
Revised: The second stage involves the magnitude of perturbation, where the magnitude of perturbation is either random, bounded by the action space, or bounded by the magnitude of the actual action taken by the agent.

Original: The third stage involves the direction in which the perturbation is appliedto.
Revised: The third stage involves the direction in which the perturbation is applied to.

Original: Since the actions and observations in these environments are multidimensional vectors, theperturbations can be added in four different ways: 1) consistently adding noise following the signsof individual observations/actions, 2) consistently adding noise that is opposite the sign of theindividual observations/actions, 3) randomly selecting a direction for each time step and adding theperturbation according to the direction and 4) randomly selecting a direction for each time step andeach observation or action and adding the perturbation according to the direction.
Revised: Since the actions and observations in these environments are multidimensional vectors, the perturbations can be added in four different ways: 1) consistently adding noise following the signs of individual observations/actions, 2) consistently adding noise that is opposite the sign of the individual observations/actions, 3) randomly selecting a direction for each time step and adding the perturbation according to the direction and 4) randomly selecting a direction for each time step and each observation or action and adding the perturbation according to the direction.

Original: The suite of allpossible black-box attacks can be summarized according to Fig.
Revised: The suite of all possible black-box attacks can be summarized according to Fig.

Original: 1, where selecting a choice at eachstage will result in a valid strategy.
Revised: 1, where selecting a choice at each stage will result in a valid strategy.

Revised: two channels, with the constraint on the attack following one of the three magnitudes and the specific instantiation following one of the four directions.

Revised: The names of each attack are denoted by the abbreviation from each stage, e,g: attacks on the observation with constraint from percent of observation with the same direction denotes O_PO_S.

Original: 3.3 White-box attacks
Revised: 3.3 White-box attacks

Original: Next, we describe the three white-box attack strategies that we selected to test on the commonbenchmark RL algorithms.
Revised: Next, we describe the three white-box attack strategies that we selected to test on the common benchmark RL algorithms.

Original: All three attacks leverage the gradient information to craft the attacks andthese three attack strategies were selected to study both perturbations on the observation and actionchannels.
Revised: All three attacks leverage the gradient information to craft the attacks and these three attack strategies were selected to study both perturbations on the observation and action channels.

Original: Specifically, the white-box attacks we implemented are the Fast Gradient Sign Method(FGSM) [4], Projected Gradient Descent (PGD) algorithm [28] and the Myopic Action Space (MAS)attack algorithm [10].
Revised: Specifically, the white-box attacks we implemented are the Fast Gradient Sign Method (FGSM) [4], Projected Gradient Descent (PGD) algorithm [30] and the Myopic Action Space (MAS) attack algorithm [10].

Original: Fast Gradient Sign Method : FGSM generates a perturbation by taking the sign of the loss’s gradientwith respect to observation and adding that perturbation to the observation.
Revised: Fast Gradient Sign Method : FGSM generates a perturbation by taking the sign of the loss’s gradient with respect to observation and adding that perturbation to the observation.

Original: By adding the perturbation,
Revised: By adding the perturbation,

Original: FGSM seeks to find a perturbation that increases the loss function, hence fooling the agent into takingpoor action.
Revised: FGSM seeks to find a perturbation that increases the loss function, hence fooling the agent into taking poor action.

Original: Formally, the FGSM method can be defined as follows:
Revised: Formally, the FGSM method can be defined as follows:

Original: [Equation]
Revised: [Equation]

Original: where s and ˆ s denote the original and perturbed observation respectively, ϵ denotes a budget thatscales the perturbation to keep it undetectable, and L denotes the policy’s loss function.
Revised: where s and ˆ s denote the original and perturbed observation respectively, ϵ denotes a budget that scales the perturbation to keep it undetectable, and L denotes the policy’s loss function.

Original: To instantiatethese attacks in practice, we used the actor network’s loss as the loss function to obtain the gradientsto compute the perturbation for each of the RL benchmark algorithms.
Revised: To instantiate these attacks in practice, we used the actor network’s loss as the loss function to obtain the gradients to compute the perturbation for each of the RL benchmark algorithms.

Original: Projected Gradient Descent : PGD is an iterative attack method that works similarly to FGSM inprinciple.
Revised: Projected Gradient Descent : PGD is an iterative attack method that works similarly to FGSM in principle.

Original: While the FGSM attacks only take a single gradient step, the PGD performs multiplegradient steps to maximize the loss function and finally projects the perturbation back into the budgetof ϵ .
Revised: While the FGSM attacks only take a single gradient step, the PGD performs multiple gradient steps to maximize the loss function and finally projects the perturbation back into the budget of ϵ .

Original: In our implementation, we set the number of iterations of PGD to be 25 after empiricallyobserving that the degradation in performance of the RL policy displays no significant changes after25 iterations.
Revised: In our implementation, we set the number of iterations of PGD to be 25 after empirically observing that the degradation in performance of the RL policy displays no significant changes after 25 iterations.

Original: Myopic Action Space attack algorithm : MAS is an attack algorithm that generates perturbationthat attacks the action channel rather than the observation channel.
Revised: Myopic Action Space attack algorithm : MAS is an attack algorithm that generates perturbation that attacks the action channel rather than the observation channel.

Original: It follows the same principle of
Revised: It follows the same principle of

Original: FGSM and PGD of generating perturbations but takes gradients of the reward function with respectto the action instead of the observation.
Revised: FGSM and PGD of generating perturbations but takes gradients of the reward function with respect to the action instead of the observation.

Original: Since the gradients of the reward function with respect to theaction might not be accessible, the gradients of the action probabilities or value function are taken asa proxy of the reward function to generate the perturbation, which is subsequently added to the RLagent’s actions.
Revised: Since the gradients of the reward function with respect to the action might not be accessible, the gradients of the action probabilities or value function are taken as a proxy of the reward function to generate the perturbation, which is subsequently added to the RL agent’s actions.

Original:  Results and discussions
Revised:  Results and discussions

Original: In this section, we present the results of our experiments comparing the performance of the RLalgorithms when subjected to the different adversarial attacks as discussed in the previous section.
Revised: In this section, we present the results of our experiments comparing the performance of the RL algorithms when subjected to the different adversarial attacks as discussed in the previous section.

Original: To fully understand the efficacy of each attack, we first trained the five RL policies using PFRL’simplementation [29] on the five MuJoCo environments and ensure that the final rewards are similarto the reported scores.
Revised: To fully understand the efficacy of each attack, we first trained the five RL policies using PFRL’s implementation [31] on the five MuJoCo environments and ensure that the final rewards are similar to the reported scores.

Original: As such, the subsequent results are all based on mounting the attacks on thetrained RL policies during test time.
Revised: As such, the subsequent results are all based on mounting the attacks on the trained RL policies during test time.

Original: All experiments were performed on an internal cluster usingthree GeForce GTX TITAN X GPUs for training the RL agents and Intel(R) 
Revised: All experiments were performed on an internal cluster using three GeForce GTX TITAN X GPUs for training the RL agents and Intel(R) Xeon(R)

Original: Xeon(R) CPU E5-1650v3 CPUs for testing and mounting the attacks.
Revised: CPU E5v3 CPUs for testing and mounting the attacks.

Original: 4.1 Comparison of different black-box attack strategies
Revised: 4.1 Comparison of different black-box attack strategies

Original: We begin by visualizing and comparing the effects of different attack strategies on the five RLalgorithms.
Revised: We begin by visualizing and comparing the effects of different attack strategies on the five RL algorithms.

Original: To measure the effectiveness of each attack, we measure the percentage change inrewards, denoted as %∆ R and defined as the change in rewards due to an attack as a fraction of theoriginal rewards achieved by the trained policy.
Revised: To measure the effectiveness of each attack, we measure the percentage change in rewards, denoted as %∆ R and defined as the change in rewards due to an attack as a fraction of the original rewards achieved by the trained policy.

Original: To compare the attacks, we plot the %∆ R for each
Revised: To compare the attacks, we plot the %∆ R for each

Original: RL algorithm as a stacked bar plot to measure the overall effectiveness of each attack strategy on allthe algorithms.
Revised: RL algorithm as a stacked bar plot to measure the overall effectiveness of each attack strategy on all the algorithms.

Original: As an illustrative example, we show the comparison for the HalfCheetah environmentin Fig.
Revised: As an illustrative example, we show the comparison for the HalfCheetah environment in Fig.

Original: An important parameter when mounting these attacks is the constraint on the magnitudeof the perturbations or the attack budget, ϵ .
Revised: An important parameter when mounting these attacks is the constraint on the magnitude of the perturbations or the attack budget, ϵ .

Original: To obtain a comprehensive comparison, we mounted allthe attacks at four budget levels: 25%, 50%, 100%, and 200%.
Revised: To obtain a comprehensive comparison, we mounted all the attacks at four budget levels: 25%, 50%, 100%, and 200%.

Original: As shown in Fig 2, the first observation that can be made is that all the different attack strategiesresulted in a negative %∆ R across all RL algorithms in the HalfCheetah environments across all ϵvalues.
Revised: As shown in Fig 2, the first observation that can be made is that all the different attack strategies resulted in a negative %∆ R across all RL algorithms in the HalfCheetah environments across all ϵ values.

Original: We did, however, observe certain environments seen in appendix Fig. 8, 10, 9, 11 wheresome attack strategies resulted in a positive %∆ R ; nonetheless, the overall trend remains negative.
Revised: We did, however, observe certain environments seen in appendix Fig. 8, 10, 9, 11 where some attack strategies resulted in a positive %∆ R ; nonetheless, the overall trend remains negative.

Original: The second observation we made is that even the most ineffective attack strategies saturate when thebudget is above 100%, with the most drastic changes in %∆ R occurring below the budget of 100%.
Revised: The second observation we made is that even the most ineffective attack strategies saturate when the budget is above 100%, with the most drastic changes in %∆ R occurring below the budget of 100%.

Original: As such, in our following experiments below, we focused only on budget levels below 100% but at afiner resolution.
Revised: As such, in our following experiments below, we focused only on budget levels below 100% but at a finer resolution.

Original: Comparing the attack strategies on the observation channel versus the action channels, we observedthat overall, attacks on the observation channel are as effective and sometimes more effective (in
Revised: Comparing the attack strategies on the observation channel versus the action channels, we observed that overall, attacks on the observation channel are as effective and sometimes more effective (in

Original: Hopper, Swimmer, and Walker) than action channel attacks up to a certain budget value, specificallyfor ϵ = 25% and 50%.
Revised: Hopper, Swimmer, and Walker) than action channel attacks up to a certain budget value, specifically for ϵ = 25% and 50%.

Original: Beyond ϵ = 50%, the action channel attacks are more effective while theobservation channel generally saturates (elaborated in further detail in the following sections).
Revised: Beyond ϵ = 50%, the action channel attacks are more effective while the observation channel generally saturates (elaborated in further detail in the following sections).

Original: Interms of the different strategies of attacks, overall, we observed that strategies that add a perturbationthat has an opposite sign (flip direction) to the original action/observation value are the most effective,while strategies that add a perturbation that has the same sign (same direction) are the least effective.
Revised: In terms of the different strategies of attacks, overall, we observed that strategies that add a perturbation that has an opposite sign (flip direction) to the original action/observation value are the most effective, while strategies that add a perturbation that has the same sign (same direction) are the least effective.

Original: Furthermore, the attack strategy that perturbs the individual elements of the observation channel/action channel is also slightly more effective than randomly perturbing the entire vector in a random direction.
Revised: Furthermore, the attack strategy that perturbs the individual elements of the observation channel/action channel is also slightly more effective than randomly perturbing the entire vector in a random direction.

Original: We highlight that these trends are observed consistently across all five benchmark algorithms and allenvironments, except for the Ant environment.
Revised: We highlight that these trends are observed consistently across all five benchmark algorithms and all environments, except for the Ant environment.

Original: Based on our observations for the Ant environment (as shown in Appendix Fig 8), the responseof the benchmark algorithm towards action channel attacks deviates slightly from the rest of theenvironments.
Revised: Based on our observations for the Ant environment (as shown in Appendix Fig 8), the response of the benchmark algorithm towards action channel attacks deviates slightly from the rest of the environments.

Original: Specifically, the attacks that add perturbation in an opposing direction in the actionspace resulted in a positive %∆ R for most RL algorithms, while perturbations of the same directionended up being one of the most effective strategies.
Revised: Specifically, the attacks that add perturbation in an opposing direction in the action space resulted in a positive %∆ R for most RL algorithms, while perturbations of the same direction ended up being one of the most effective strategies.

Original: We hypothesize that this is possibly due to themore complex 3-dimensional non-linear interaction of the action space of the Ant robot as comparedto the rest of the environments, which are restricted to the 2-dimensional plane.
Revised: We hypothesize that this is possibly due to the more complex 3-dimensional non-linear interaction of the action space of the Ant robot as compared to the rest of the environments, which are restricted to the 2-dimensional plane.

Original: This also alludes tothe fact that the benchmark RL algorithm has not converged to the optimal policies in practice andthe perturbations ended up being less effective.
Revised: This also alludes to the fact that the benchmark RL algorithm has not converged to the optimal policies in practice and the perturbations ended up being less effective.

Original: 4.2 Comparison of the robustness of different policies
Revised: 4.2 Comparison of the robustness of different policies

Original: Next, we present the comparisons between the overall robustness of different RL policies across allenvironments and black-box attacks.
Revised: Next, we present the comparisons between the overall robustness of different RL policies across all environments and black-box attacks.

Original: The sum of all the %∆ R across all attacks and all environments for each policy is illustrated in Fig 3.
Revised: The sum of all the %∆ R across all attacks and all environments for each policy is illustrated in Fig 3.

Original: We hope that such a plot would reveal the natural robustness ofeach type of policy, i.e., how insensitive an RL algorithm is to perturbations if it wasn’t specificallytrained to be robust in the first place?
Revised: We hope that such a plot would reveal the natural robustness of each type of policy, i.e., how insensitive an RL algorithm is to perturbations if it wasn’t specifically trained to be robust in the first place?

Original: In summary, we observed that both TD3 and SAC exhibitthe most robustness across all environments and for all values of budget on the magnitude ofperturbation, ϵ .
Revised: In summary, we observed that both TD3 and SAC exhibit the most robustness across all environments and for all values of budget on the magnitude of perturbation, ϵ .

Original: Additionally, TD3 and SAC were also the least affected when increasing ϵ , while theother three algorithms %∆ R increased significantly, with the Ant and HalfCheetah environmentscontributing to most of the change.
Revised: Additionally, TD3 and SAC were also the least affected when increasing ϵ , while the other three algorithms %∆ R increased significantly, with the Ant and HalfCheetah environments contributing to most of the change.

Original: On the other hand, we note that DDPG was overall the policythat is most sensitive to perturbations, especially in the Ant environment.
Revised: On the other hand, we note that DDPG was overall the policy that is most sensitive to perturbations, especially in the Ant environment.

Original: Removing the outliereffect of the Ant environment, TRPO ranks as the policy most vulnerable to perturbations.
Revised: Removing the outlier effect of the Ant environment, TRPO ranks as the policy most vulnerable to perturbations.

Original: Whileit is not clear why DDPG or TRPO are so sensitive, we hypothesize that the reason both SAC and
Revised: While it is not clear why DDPG or TRPO are so sensitive, we hypothesize that the reason both SAC and

Original: TD3 are more robust is because of their shared implementation of having two Q-values to reduceoverestimation.
Revised: TD3 are more robust is because of their shared implementation of having two Q-values to reduce overestimation.

Original: More importantly, SAC also includes an entropy bonus term in the objective function,while TD3 implements a target policy smoothing that includes noise to the action, both of which canbe considered an indirect way of incorporating adversarial training in the learning process.
Revised: More importantly, SAC also includes an entropy bonus term in the objective function, while TD3 implements a target policy smoothing that includes noise to the action, both of which can

Original: However,this hypothesis remains to be further verified.

Original: 4.3 The effect of budget ϵ on %∆ R
Revised: 4.3 The effect of budget ϵ on %∆ R

Original: While it is clear that the value of ϵ , the budget on the magnitude of perturbation, affects the effectiveness of an attack in a positively correlated manner, we study the relationship between these twovariables in more detail in this section.
Revised: While it is clear that the value of ϵ , the budget on the magnitude of perturbation, affects the effectiveness of an attack in a positively correlated manner we study the relationship between these two variables in more detail in this section.

Original: We repeated the experiments shown in Fig. 2 by varying thevalues of ϵ at a finer resolution of 5%.
Revised: We repeated the experiments shown in Fig. 2 by varying the values of ϵ at a finer resolution of 5%.

Original: The solid lines in Fig. 4(a) and (b) represents the average of%∆ R across all environments for each RL algorithm for action channel attacks and observation channel attacks respectively.
Revised: The solid lines in Fig. 4(a) and (b) represents the average of %∆ R across all environments for each RL algorithm for action channel attacks and observation channel attacks respectively.

Original: From this plot, we can make several more interesting observations.
Revised: From this plot, we can make several more interesting observations.

Original: Firstly,we see that perturbations in the observation channel are effective but have diminishing effectivenessas seen by the saturating trends of the %∆ R in Fig. 4(b).
Revised: Firstly, we see that perturbations in the observation channel are effective but have diminishing effectiveness as seen by the saturating trends of the %∆ R in Fig. 4(b).

Original: In contrast, perturbations in the action spacedo not display this characteristic as we see that %∆ R still decreases at a linear rate as the attackbudget increases up until 200%.
Revised: In contrast, perturbations in the action space do not display this characteristic as we see that %∆ R still decreases at a linear rate as the attack budget increases up until 200%.

Original: However, one caveat is that a major contributor of the continueddecrease of rewards was due to the attacks mounted on the Ant environment, as discussed in theprevious section.
Revised: However, one caveat is that a major contributor of the continued decrease of rewards was due to the attacks mounted on the Ant environment, as discussed in the previous section.

Original: Removing the Ant environment from the trends (as shown in the dotted lines of
Revised: Removing the Ant environment from the trends (as shown in the dotted lines of

Original: Fig. 4(a) and (b))revealed that the %∆ R decreases less drastically for action channel attacks, but isstill more significant than observation channel attacks.
Revised: Fig. 4(a) and (b))revealed that the %∆ R decreases less drastically for action channel attacks but is still more significant than observation channel attacks.

Original: This further validates our hypothesis that RLagents that operate in environments with a higher degree of freedom are likely to be more sensitive toperturbations and display catastrophic failures.
Revised: This further validates our hypothesis that RL agents that operate in environments with a higher degree of freedom are likely to be more sensitive to perturbations and display catastrophic failures.

Original: Another observation that can be made is that in the regime of the ϵ < 50%, attacks on the agent’sobservation channel cause a much more significant drop in performance than attacks on the agent’saction channel.
Revised: Another observation that can be made is that in the regime of the ϵ < 50%, attacks on the agent’s observation channel cause a much more significant drop in performance than attacks on the agent’s action channel.

Original: This observation is further validated when we visualize the %∆ R for every 5%increment of ϵ in Fig.
Revised: This observation is further validated when we visualize the %∆ R for every 5% increment of ϵ in Fig.

Original: Once again, we only present the experiments for HalfCheetah for brevity,with the visualization for the rest of the environments shown in the appendix Fig.
Revised: Once again, we only present the experiments for HalfCheetah for brevity, with the visualization for the rest of the environments shown in the appendix Fig.

Original: From the figure, we can observe that the largest drop in rewards for observation channel attacks(bars with diagonal patterns) occurs when ϵ is between 0 to 10%.
Revised: From the figure, we can observe that the largest drop in rewards for observation channel attacks (bars with diagonal patterns) occurs when ϵ is between 0 to 10%.

Original: Meanwhile, we observe the exactopposite trend in action channel attacks where the initial effect when ϵ is between 0 to 5% was smallbut the %∆ R increases as we increase ϵ .
Revised: Meanwhile, we observe the exact opposite trend in action channel attacks where the initial effect when ϵ is between 0 to 5% was small but the %∆ R increases as we increase ϵ .

Original: 4.4 Comparison of different white-box attacks
Revised: 4.4 Comparison of different white-box attacks

Original: Next, we compare the effects of the three white-box attack strategies we had selected on the performance of the benchmark RL policies.
Revised: Next, we compare the effects of the three white-box attack strategies we had selected on the performance of the benchmark RL policies.

Original: While this is by no means a comprehensive experimentof white-box attack strategies, we hope that the results in these sections will provide some initialinsights.
Revised: While this is by no means a comprehensive experiment of white-box attack strategies, we hope that the results in these sections will provide some initial insights.

Original: Fig.
Revised: Fig.

Original: 6 illustrates the average %∆ R across all environments for each attack strategy.
Revised: 6 illustrates the average %∆ R across all environments for each attack strategy.

Original: Similarto the black-box attacks we implemented, we observed that all white-box attacks resulted in a generalnegative trend.
Revised: Similar to the black-box attacks we implemented, we observed that all white-box attacks resulted in a general negative trend.

Original: Compared to black-box attacks, we observed that the decrease in %∆ R is muchsteeper than the trends observed in Fig. 4.
Revised: Compared to black-box attacks, we observed that the decrease in %∆ R is much steeper than the trends observed in Fig. 4.

Original: However, it is worth highlighting that the context and rangeof the ϵ values used in white-box experiments are different.
Revised: However, it is worth highlighting that the context and range of the ϵ values used in white-box experiments are different.

Original: While the values of ϵ in the black-boxexperiments were expressed as a percentage of the action/observation space or the actual values ofthe action/observations themselves, the values of ϵ used in the white-box experiments were basedon the values reported in the literature.
Revised: While the values of ϵ in the black-box experiments were expressed as a percentage of the action/observation space or the actual values of the action/observations themselves, the values of ϵ used in the white-box experiments were based on the values reported in the literature.

Original: Furthermore, the black-box attack strategies we proposedfollowed the strategy of adding noise, while the white-box strategy we implemented all incorporatedsome form of optimization.
Revised: Furthermore, the black-box attack strategies we proposed followed the strategy of adding noise, while the white-box strategy we implemented all incorporated some form of optimization.

Original: As such, no direct comparison between white and black-box trends canbe made.
Revised: As such, no direct comparison between white and black-box trends can be made.

Original: Nevertheless, an interesting observation we made is that while increasing the value of ϵresulted in a monotonic decrease in %∆ R for black-box attacks, the %∆ R for white-box attacksexhibited some form of fluctuations, although we still observe a general decreasing trend.
Revised: Nevertheless, an interesting observation we made is that while increasing the value of ϵ resulted in a monotonic decrease in %∆ R for black-box attacks, the %∆ R for white-box attacks exhibited some form of fluctuations, although we still observe a general decreasing trend.

Original: Comparing the attacks on the observation channel (FGSM and PGD) versus attacks on the actionchannel (MAS), we observe that, in general, both types of attacks perform similarly asymptoticallyas we increase the value of ϵ .
Revised: Comparing the attacks on the observation channel (FGSM and PGD) versus attacks on the action channel (MAS), we observe that, in general, both types of attacks perform similarly asymptotically as we increase the value of ϵ .

Original: However, at lower values of ϵ , action channel attacks have a highervariance in terms of the %∆ R across different algorithms.
Revised: However, at lower values of ϵ , action channel attacks have a higher variance in terms of the %∆ R across different algorithms.

Original: Specifically, we observe that the smallervalues of ϵ increased the %∆ R by almost 100% for TD3.
Revised: Specifically, we observe that the smaller values of ϵ increased the %∆ R by almost 100% for TD3.

Original: However, removing the results of the Antenvironment from the trend (dotted lines) showed that the trend for TD3 reverts to a trend that followsthe rest of the environment.
Revised: However, removing the results of the Ant environment from the trend (dotted lines) showed that the trend for TD3 reverts to a trend that follows the rest of the environment.

Original: When we compare the performance across different RL policies, we observe that most of thealgorithms had similar robustness, with DDPG and TD3 being the most sensitive to perturbations anddisplaying the largest %∆ R when subjected to observation channel attacks.
Revised: When we compare the performance across different RL policies, we observe that most of the algorithms had similar robustness, with DDPG and TD3 being the most sensitive to perturbations and displaying the largest %∆ R when subjected to observation channel attacks.

Original: Once again, these trendsbecame less extreme once removing the effect of the Ant environment.
Revised: Once again, these trends became less extreme once removing the effect of the Ant environment.

Original: In terms of action channelattack (MAS), one interesting observation is that most RL algorithms performed similarly except for
Revised: In terms of action channel attack (MAS), one interesting observation is that most RL algorithms performed similarly except for

Original: SAC, which displayed a large drop in performance even with a small value of ϵ .
Revised: SAC, which displayed a large drop in performance even with a small value of ϵ .

Original: This is a surprisingobservation as SAC was one of the most robust policies in the black-box attack experiments, andeven removing the effect of the Ant environment did not change the trends significantly.
Revised: This is a surprising observation as SAC was one of the most robust policies in the black-box attack experiments, and even removing the effect of the Ant environment did not change the trends significantly.

Original: As such, itwould be interesting for future studies to investigate why SAC is robust towards observation channelperturbation but becomes particularly sensitive to action space perturbation, specifically the white-box
Revised: As such, it would be interesting for future studies to investigate why SAC is robust towards observation channel perturbation but becomes particularly sensitive to action space perturbation, specifically the white-box

Original: MAS attack.
Revised: MAS attack.

Original: [Figure]
Revised: [Figure]

Original: 4.5 Summary and discussions
Revised: 4.5 Summary and discussions

Original: To summarize our findings, we compile our observations into Fig.
Revised: To summarize our findings, we compile our observations into Fig.

Original: 7 and rank the benchmark RLalgorithms according to three criteria: robustness, range of robustness, and sensitivity.
Revised: and rank the benchmark RL algorithms according to three criteria: robustness, range of robustness, and sensitivity.

Original: Furthermore,we classified the algorithm’s characteristics according to black-box attacks Fig.
Revised: Furthermore, we classified the algorithm’s characteristics according to black-box attacks Fig.

Original: 7a and white-boxattacks Fig.
Revised: 7a and white-box attacks Fig.

Original: 7b.
Revised: 7b.

Original: The horizontal axis represents an algorithm’s robustness, where we define robustness as the average %∆ R across all attacks and all environments.
Revised: The horizontal axis represents an algorithm’s robustness, where we define robustness as the average %∆ R across all attacks and all environments.

Original: The vertical axis represents analgorithm’s sensitivity.
Revised: The vertical axis represents an algorithm’s sensitivity.

Original: We define sensitivity by taking the average difference for all %∆ R acrossall possible pairs of strategies and computing the absolute value of it.
Revised: We define sensitivity by taking the average difference for all %∆ R across [Figure] all possible pairs of strategies and computing the absolute value of it.

Original: Intuitively, the sensitivitygives us a sense of how much we can expect the performance of an RL policy will change whensubjected to different attacks.
Revised: Intuitively, the sensitivity gives us a sense of how much we can expect the performance of an RL policy will change when subjected to different attacks.

Original: Finally, the size of the circles in Fig.
Revised: Finally, the size of the circles in Fig.

Original: 7 represents the range of therobustness of an algorithm by taking the difference between the maximum and minimum %∆ Runder the white-box and black-box attack scenario, respectively.
Revised: 7 represents the range of the robustness of an algorithm by taking the difference between the maximum and minimum %∆ R under the white-box and black-box attack scenario, respectively.

Original: Generally speaking, we observe that
Revised: Generally speaking, we observe that

Original: [Figure] TD3 exhibits the best robustness across both white-box and black-box attacks, while SAC performswell under black-box attacks but performs extremely poorly on white-box attacks.
Revised: TD3 exhibits the best robustness across both white-box and black-box attacks, while SAC performs well under black-box attacks but performs extremely poorly on white-box attacks.

Original: We also note that
Revised: We also note that

Original: PPO and TRPO are robust to a certain extent with medium sensitivities, but DDPG ranks the lowestin terms of having low robustness and high sensitivity.
Revised: PPO and TRPO are robust to a certain extent with medium sensitivities, but DDPG ranks the lowest in terms of having low robustness and high sensitivity.

Original: Finally, we also highlight that black-boxattacks have a larger range of effects on the RL policies in general (larger circles) when compared towhite-box attacks, which have more consistent effects (smaller circles).
Revised: Finally, we also highlight that black-box attacks have a larger range of effects on the RL policies in general (larger circles) when compared to white-box attacks, which have more consistent effects (smaller circles).

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we compared commonly used benchmark RL algorithms’ robustness towards varioustypes of perturbation during test time.
Revised: In this work, we compared commonly used benchmark RL algorithms’ robustness towards various types of perturbation during test time.

Original: We designed a suite of simple black-box attack strategies toperturb the RL agent’s observation and action channels, and we also implemented three commonlyused white-box optimization-based attacks that perturbed the agent’s observation and action channels.
Revised: We designed a suite of simple black-box attack strategies to perturb the RL agent’s observation and action channels, and we also implemented three commonly used white-box optimization-based attacks that perturbed the agent’s observation and action channels.

Original: From our experiments, we made the following conclusions: Firstly, from the black-box attackstrategies we tested, a recurring theme is that observation channel attacks are more effective thanaction channel attacks, but only until a certain threshold on the magnitude of the perturbation.
Revised: From our experiments, we made the following conclusions: Firstly, from the black-box attack strategies we tested, a recurring theme is that observation channel attacks are more effective than action channel attacks, but only until a certain threshold on the magnitude of the perturbation.

Original: Beyond this threshold, the effects of observation attacks saturate while action channel attacks maycontinue to have some effect.
Revised: Beyond this threshold, the effects of observation attacks saturate while action channel attacks may continue to have some effect.

Original: We also find that the Ant environment generally amplifies the effect ofattacks.
Revised: We also find that the Ant environment generally amplifies the effect of attacks.

Original: In terms of the robustness of different policies under black-box attacks, SAC and TD3 weregenerally robust, while DDPG and TRPO were the most sensitive to perturbations.
Revised: In terms of the robustness of different policies under black-box attacks, SAC and TD3 were generally robust, while DDPG and TRPO were the most sensitive to perturbations.

Original: When subjected tooptimization-based white-box attacks in the observation channel, most policies performed similarly,with DDPG and TD3 being the most sensitive, while SAC was found to be extremely sensitive toaction channel attacks.
Revised: When subjected to optimization-based white-box attacks in the observation channel, most policies performed similarly, with DDPG and TD3 being the most sensitive, while SAC was found to be extremely sensitive to action channel attacks.

Original: We find it intriguing that two of the most robust policies under black-boxattacks ended up being the most sensitive to attacks under white-box attacks, and future work willseek to further understand this phenomenon.
Revised: We find it intriguing that two of the most robust policies under black-box attacks ended up being the most sensitive to attacks under white-box attacks, and future work will seek to further understand this phenomenon.

Original: Furthermore, we will extend this study to include a morecomprehensive comparison of existing optimization-based black-box attacks and white-box attacks.Checklist
Revised: Furthermore, we will extend this study to include a more comprehensive comparison of existing optimization-based black-box attacks and white-box attacks.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? [Yes]
Revised: A Implementation

Original: (b) Did you describe the limitations of your work? [Yes] (c) Did you discuss any potential negative societal impacts of your work? [Yes]
Revised: Codes to reproduce the results can be found at https://github.com/super864/

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] If you are including theoretical results...
Revised: Natural-Robustness-RL

Original: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results? [N/A]3. If you ran experiments...
Revised: B Broader Impact

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes](b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes](c)

Original: Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes](d) Did you include the total amount of computing and  the type of resources used (e.g.,type of GPUs, internal cluster, or cloud provider)? [Yes] If you are using existing assets (e
Revised: This study investigates and highlights the potential vulnerabilities of commonly used benchmark RL algorithms to a suite of different white-box and black-box attacks.

Original: g., code, data, models) or curating/releasing new assets... your work uses existing assets, did you cite the creators? [N A
Revised: As such, there is a potential for the results of these study to be used with malicious intent to mount attacks on existing RL algorithms that has been deployed in production.

Original: Did you mention the license of the assets?

Original: [N/A] Did you include any new assets either in the supplemental material or as a URL? [N/A] Did you discuss whether and how consent was obtained from people whose data you’re
Revised: Nonetheless, we believe that the results of this study may also be used as a guideline to select a more robust RL policy or as a stepping stone to developing a more robust RL algorithm.

Original: using/curating?

Original: [N/A] Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content? [N/A] If you used crowdsourcing or conducted research with human subjects...
Revised: Hence, we truly believe that the benefits of the results of this study will outweigh the potential negative societal impact.

Original: Did you include the full text of instructions given to participants and screenshots, if

Original: applicable? [N/A] you describe any potential participant risks, with links to Institutional Review approvals, if applicable?
Revised: C Additional results on other environments

Original: [N/A]

Original: you the estimated hourly wage paid to participants and the total amount spent on participant compensation [N
Revised: This section presents the comparison plots for the other environments that were not shown in the main manuscript.

Original: A