Original: B OUNDED M YOPIC A DVERSARIES FOR D EEP R EINFORCEMENT L EARNING A GENTS
Revised: B OUNDED M YOPIC A DVERSARIES FOR D EEP R EINFORCEMENT L EARNING A GENTS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Adversarial attacks against deep neural networks have been widely studied.
Revised: Adversarial attacks against deep neural networks have been widely studied.

Original: Adversarial examples for deep reinforcement learning (DeepRL) have signiﬁcant security implications, due to the deployment of these algorithms in many application domains.
Revised: Adversarial examples for deep reinforcement learning (DeepRL) have signiﬁcant security implications, due to the deployment of these algorithms in many application domains.

Original: In this work we formalize an optimal myopic adversary for deep reinforcement learning agents.
Revised: In this work we formalize an optimal myopic adversary for deep reinforcement learning agents.

Original: Our adversary attempts to ﬁnd a bounded perturbation of the state which minimizes the value of the action taken by the agent.
Revised: Our adversary attempts to ﬁnd a bounded perturbation of the state which minimizes the value of the action taken by the agent.

Original: We show with experiments in various games in the Atari environment that our attack formulation achieves signiﬁcantly larger impact as compared to the current state-of-the-art.
Revised: We show with experiments in various games in the Atari environment that our attack formulation achieves signiﬁcantly larger impact as compared to the current state-of-the-art.

Original: Furthermore, this enables us to lower the bounds by several orders of magnitude on the perturbation needed to efﬁciently achieve signiﬁcant impacts on DeepRL agents
Revised: Furthermore, this enables us to lower the bounds by several orders of magnitude on the perturbation needed to efﬁciently achieve signiﬁcant impacts on DeepRL agents

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep Neural Networks (DNN) have become a powerful tool and currently DNNs are widely used in speech recognition (Hannun et al., 2014), computer vision (Krizhevsky et al., 2012), natural language processing (Sutskever et al., 2014), and self learning systems as deep reinforcement learning agents (Mnih et al.
Revised: Deep Neural Networks (DNN) have become a powerful tool and currently DNNs are widely used in speech recognition (Hannun et al., 2014), computer vision (Krizhevsky et al., 2012), natural language processing (Sutskever et al., 2014), and self learning systems as deep reinforcement learning agents (Mnih et al.

Original: (2015), Mnih et al.
Revised: (2015), Mnih et al.

Original: (2016), Schulman et al.
Revised: (2016), Schulman et al.

Original: (2015), Lillicrap et al.
Revised: (2015), Lillicrap et al.

Original: Along with the overwhelming success of DNNs in various domains there has also been a line of research investigating their weaknesses.
Revised: Along with the overwhelming success of DNNs in various domains there has also been a line of research investigating their weaknesses.

Original: Szegedy et al.
Revised: Szegedy et al.

Original: (2014) observed that adding imperceptible perturbations to images can lead a DNN to misclassify the input image.
Revised: (2014) observed that adding imperceptible perturbations to images can lead a DNN to misclassify the input image.

Original: The authors argue that the existence of these so called adversarial examples is a form of overﬁtting.
Revised: The authors argue that the existence of these so called adversarial examples is a form of overﬁtting.

Original: In particular, they hypothesize that a very complicated neural network behaves well on the training set, but nonetheless, performs poorly on the testing set enabling exploitation by the attacker.
Revised: In particular, they hypothesize that a very complicated neural network behaves well on the training set, but nonetheless, performs poorly on the testing set enabling exploitation by the attacker.

Original: However, they discovered different DNN models were misclassifying the same adversarial examples and assigning them the same class instead of making random mistakes.
Revised: However, they discovered different DNN models were misclassifying the same adversarial examples and assigning them the same class instead of making random mistakes.

Original: This led Goodfellow et al.
Revised: This led Goodfellow et al.

Original: (2015) to propose that
Revised: (2015) to propose that

Original: the
Revised: the

Original: DNN models were actually learning approximately linear functions resulting in underﬁtting the data.
Revised: DNN models were actually learning approximately linear functions resulting in underﬁtting the data.

Original: Recent work by Mnih et al.
Revised: Recent work by Mnih et al.

Original: (2015) introduced the use of DNNs as function approximators in reinforcement learning, improving the state of the art in this area.
Revised: (2015) introduced the use of DNNs as function approximators in reinforcement learning, improving the state of the art in this area.

Original: Because these deep reinforcement learning agents utilize DNNs, they are also susceptible to this type of adversarial examples.
Revised: Because these deep reinforcement learning agents utilize DNNs, they are also susceptible to this type of adversarial examples.

Original: Currently, deep reinforcement learning has been applied to many areas such as network system control (Jay et al.
Revised: Currently, deep reinforcement learning has been applied to many areas such as network system control (Jay et al.

Original: Chu et al.
Revised: Chu et al.

Original: (2020), Chinchali et al.
Revised: (2020), Chinchali et al.

Original: ﬁnancial trading Noonan (2017), blockchain protocol security Hou et al.
Revised: ﬁnancial trading Noonan (2017), blockchain protocol security Hou et al.

Original: , grid operation and security (Duan et al.
Revised: , grid operation and security (Duan et al.

Original: (2019), Huang et al.
Revised: (2019), Huang et al.

Original: , cloud computing Chen et al.
Revised: , cloud computing Chen et al.

Original: (2018), robotics (Gu et al.
Revised: (2018), robotics (Gu et al.

Original: (2017), Kalashnikov et al.
Revised: (2017), Kalashnikov et al.

Original: autonomous driving Dosovitsky et al.
Revised: autonomous driving Dosovitsky et al.

Original: (2017), and medical treatment and diagnosis (Tseng et al.
Revised: (2017), and medical treatment and diagnosis (Tseng et al.

Original: (2017), Popova et al.
Revised: (2017), Popova et al.

Original: (2018), Thananjeyan et al.
Revised: (2018), Thananjeyan et al.

Original: (2017), Daochang & Jiang (2018), Ghesu et al.
Revised: (2017), Daochang & Jiang (2018), Ghesu et al.

Original: A more particular scenario where adversarial perturbations might be of signiﬁcant interest is a ﬁnancial trading market where the DeepRL agent is trained on observations consisting of the order book.
Revised: A more particular scenario where adversarial perturbations might be of signiﬁcant interest is a ﬁnancial trading market where the DeepRL agent is trained on observations consisting of the order book.

Original: In such a setting it is possible to compromise the whole trading system with an extremely small subset of adversaries.
Revised: In such a setting it is possible to compromise the whole trading system with an extremely small subset of adversaries.

Original: In particular, the (cid:96) 1 -norm bounded perturbations dicussed in our paper have sparse solutions, and thus can be used as a basis for an attack in such a scenario.
Revised: In particular, the (cid:96) 1 -norm bounded perturbations dicussed in our paper have sparse solutions, and thus can be used as a basis for an attack in such a scenario.

Original: Moreover, the magnitude of the (cid:96) 1 -norm bounded perturbations produced by our attack is orders of magnitude smaller than previous approaches, and thus our proposed perturbations result in a stealth attack more likely to evade automatic anomaly detection schemes.
Revised: Moreover, the magnitude of the (cid:96) 1 -norm bounded perturbations produced by our attack is orders of magnitude smaller than previous approaches, and thus our proposed perturbations result in a stealth attack more likely to evade automatic anomaly detection schemes.

Original: Considering the wide spectrum of deep reinforcement learning algorithm deployment it is crucial to investigate the resilience of these algorithms before they are used in real world application domains.
Revised: Considering the wide spectrum of deep reinforcement learning algorithm deployment it is crucial to investigate the resilience of these algorithms before they are used in real world application domains.

Original: Moreover, adversarial formulations are a ﬁrst step to understand these algorithms and build generalizable, reliable and robust deep reinforcement learning agents.
Revised: Moreover, adversarial formulations are a ﬁrst step to understand these algorithms and build generalizable, reliable and robust deep reinforcement learning agents.

Original: Therefore, in this paper we study adversarial attack formulations for deep reinforcement learning agents and make the following contributions
Revised: Therefore, in this paper we study adversarial attack formulations for deep reinforcement learning agents and make the following contributions

Original: : • We deﬁne the optimal myopic adversary, whose aim is to minimize the value of the action taken by the agent in each state, and formulate the optimization problem that this adversary seeks to solve.
Revised: : • We deﬁne the optimal myopic adversary, whose aim is to minimize the value of the action taken by the agent in each state, and formulate the optimization problem that this adversary seeks to solve.

Original: • We introduce a differentiable approximation for the optimal myopic adversarial formulation.
Revised: • We introduce a differentiable approximation for the optimal myopic adversarial formulation.

Original: • We compare the impact results of our attack formulation to previous formulations in different games in the Atari environment.
Revised: • We compare the impact results of our attack formulation to previous formulations in different games in the Atari environment.

Original: • We show that the new formulation ﬁnds a better direction for the adversarial perturbation and increases the attack impact for bounded perturbations.
Revised: • We show that the new formulation ﬁnds a better direction for the adversarial perturbation and increases the attack impact for bounded perturbations.

Original: (Conversely, our formulation decreases the magnitude of the pertubation required to efﬁciently achieve a signiﬁcant impact.)
Revised: (Conversely, our formulation decreases the magnitude of the pertubation required to efﬁciently achieve a signiﬁcant impact.)

Original:  R ELATED W ORK AND B ACKGROUND 2. A DVERSARIAL R EINFORCEMENT L EARNING
Revised:  R ELATED W ORK AND B ACKGROUND 2. A DVERSARIAL R EINFORCEMENT L EARNING

Original: Adversarial reinforcement learning is an active line of research directed towards discovering the weaknesses of deep reinforcement learning algorithms.
Revised: Adversarial reinforcement learning is an active line of research directed towards discovering the weaknesses of deep reinforcement learning algorithms.

Original: Gleave et al.
Revised: Gleave et al.

Original: (2020) model the interaction between the agent and the adversary as a two player Markov game and solve the reinforcement learning problem for the adversary via Proximal Policy Optimization introduced by Schulman et al.
Revised: (2020) model the interaction between the agent and the adversary as a two player Markov game and solve the reinforcement learning problem for the adversary via Proximal Policy Optimization introduced by Schulman et al.

Original: They ﬁx the victim agent’s policy and only allow the adversary to take natural actions to disrupt the agent instead of using (cid:96) p -norm bound pixel perturbations.
Revised: They ﬁx the victim agent’s policy and only allow the adversary to take natural actions to disrupt the agent instead of using (cid:96) p -norm bound pixel perturbations.

Original: Pinto et al.
Revised: Pinto et al.

Original: (2017) model the adversary and the victim as a two player zero-sum discounted Markov game and train the victim in the presence of the adversary to make the victim more robust.
Revised: (2017) model the adversary and the victim as a two player zero-sum discounted Markov game and train the victim in the presence of the adversary to make the victim more robust.

Original: Mandlekar et al.
Revised: Mandlekar et al.

Original: (2017) use a gradient based perturbation to make the agent more robust as compared to random perturbations.
Revised: (2017) use a gradient based perturbation to make the agent more robust as compared to random perturbations.

Original: Huang et al.
Revised: Huang et al.

Original: (2017) and Kos & Song (2017) use the fast gradient sign method (FGSM) to show deep reinforcement learning agents are vulnarable to adversarial perturbations.
Revised: (2017) and Kos & Song (2017) use the fast gradient sign method (FGSM) to show deep reinforcement learning agents are vulnarable to adversarial perturbations.

Original: Pattanaik et al.
Revised: Pattanaik et al.

Original: (2018) use a gradient based formulation to increase the robustness of deep reinforcement learning agents.
Revised: (2018) use a gradient based formulation to increase the robustness of deep reinforcement learning agents.

Original: 2. A DVERSARIAL A TTACK M ETHODS
Revised: 2. A DVERSARIAL A TTACK M ETHODS

Original: Goodfellow et al. (2015) introduced the fast gradient method (FGM)
Revised: Goodfellow et al. (2015) introduced the fast gradient method (FGM)

Original: [Equation]
Revised: [Equation]

Original: for crafting adversarial examples for image classiﬁcation by taking the gradient of the cost function
Revised: for crafting adversarial examples for image classiﬁcation by taking the gradient of the cost function

Original: J ( x, y ) used to train the neural network in the direction of the input.
Revised: J ( x, y ) used to train the neural network in the direction of the input.

Original: Here x is the input and y is the output label for image classiﬁcation.
Revised: Here x is the input and y is the output label for image classiﬁcation.

Original: As mentioned in the previous section FGM was ﬁrst adapted to the deep reinforcement learning setting by Huang et al.
Revised: As mentioned in the previous section FGM was ﬁrst adapted to the deep reinforcement learning setting by Huang et al.

Original: Subsequently, Pattanaik et al.
Revised: Subsequently, Pattanaik et al.

Original: (2018) introduced a variant of FGM, in which a few random samples are taken in the gradient direction, and the best is chosen.
Revised: (2018) introduced a variant of FGM, in which a few random samples are taken in the gradient direction, and the best is chosen.

Original: However, the main difference between the approach of Huang et al.
Revised: However, the main difference between the approach of Huang et al.

Original: (2017) and Pattanaik et al.
Revised: (2017) and Pattanaik et al.

Original: (2018) is in the choice of the cost function J used to determine the gradient direction.
Revised: (2018) is in the choice of the cost function J used to determine the gradient direction.

Original: In the next section we will outline the different cost functions used in these two different formulations.
Revised: In the next section we will outline the different cost functions used in these two different formulations.

Original: 2. A DVERSARIAL A TTACK F ORMULATIONS
Revised: 2. A DVERSARIAL A TTACK F ORMULATIONS

Original: In a bounded attack formulation for deep reinforcement learning, the aim is to try to ﬁnd a perturbed state s adv in a ball
Revised: In a bounded attack formulation for deep reinforcement learning, the aim is to try to ﬁnd a perturbed state s adv in a ball

Original: [Equation]
Revised: [Equation]

Original: that minimizes the expected cumulative reward of the agent.
Revised: that minimizes the expected cumulative reward of the agent.

Original: It is important to note that the agent will always try to take the best action depending only on its perception of the state, and independent from the unperturbed state.
Revised: It is important to note that the agent will always try to take the best action depending only on its perception of the state, and independent from the unperturbed state.

Original: Therefore, in the perturbed state the agent will still choose the action, a ∗ ( s adv ) =
Revised: Therefore, in the perturbed state the agent will still choose the action, a ∗ ( s adv ) =

Original: arg max a Q ( s adv , a ) , which maximizes the state action value function in state s adv .
Revised: arg max a Q ( s adv , a ) , which maximizes the state action value function in state s adv .

Original: It has been an active line of research to ﬁnd the right direction for the adversarial perturbation in the deep reinforcement learning domain.
Revised: It has been an active line of research to ﬁnd the right direction for the adversarial perturbation in the deep reinforcement learning domain.

Original: The ﬁrst attack of this form was formulated by Huang et al.
Revised: The ﬁrst attack of this form was formulated by Huang et al.

Original: (2017) and concurrently by Kos & Song (2017) by trying to minimize the probability of the best possible action in the given state,
Revised: (2017) and concurrently by Kos & Song (2017) by trying to minimize the probability of the best possible action in the given state,

Original: [Equation]
Revised: [Equation]

Original: Note that π ( s, a ) is the softmax policy of the agent given by
Revised: Note that π ( s, a ) is the softmax policy of the agent given by

Original: [Equation]
Revised: [Equation]

Original: where T is called the temperature constant.
Revised: where T is called the temperature constant.

Original: When the temperature constant is not relevant to the discussion we will drop the subscript and use the notation π
Revised: When the temperature constant is not relevant to the discussion we will drop the subscript and use the notation π

Original: ( s, a ) .
Revised: ( s, a ) .

Original: It is important to note that π ( s, a ) is not the actual policy used by the agent.
Revised: It is important to note that π ( s, a ) is not the actual policy used by the agent.

Original: Indeed the DRL agent deterministically chooses the action a ∗ maximizing Q ( s, a ) (or equivalently π ( s, a ) ).
Revised: Indeed the DRL agent deterministically chooses the action a ∗ maximizing Q ( s, a ) (or equivalently π ( s, a ) ).

Original: The softmax operation is only introduced in order to calculate the adversarial perturbation direction.
Revised: The softmax operation is only introduced in order to calculate the adversarial perturbation direction.

Original: Lin et al.
Revised: Lin et al.

Original: (2017) formulated another attack with unbounded perturbation based on the Carlini & Wagner (2017) attack where the goal is to minimize the perturbation subject to choosing any action other than the best.
Revised: (2017) formulated another attack with unbounded perturbation based on the Carlini & Wagner (2017) attack where the goal is to minimize the perturbation subject to choosing any action other than the best.

Original: [Equation]
Revised: [Equation]

Original: subject to
Revised: subject to

Original: [Equation]
Revised: [Equation]

Original: Pattanaik et al.
Revised: Pattanaik et al.

Original: (2018) formulated yet another attack which aims to maximize the probability of the worst possible action in the given state,
Revised: (2018) formulated yet another attack which aims to maximize the probability of the worst possible action in the given state,

Original: [Equation]
Revised: [Equation]

Original: and further showed that their attack formulation (5) is more effective than (3).
Revised: and further showed that their attack formulation (5) is more effective than (3).

Original: Pattanaik et al.
Revised: Pattanaik et al.

Original: (2018) also introduce the notion of targeted attacks to the reinforcement learning domain.
Revised: (2018) also introduce the notion of targeted attacks to the reinforcement learning domain.

Original: In their paper they take the cross entropy loss between the optimal policy in the given state and their adversarial probability distribution and try to increase the probability of a w .
Revised: In their paper they take the cross entropy loss between the optimal policy in the given state and their adversarial probability distribution and try to increase the probability of a w .

Original: However, just trying to increase the probability of a w in the softmax policy i.e. π ( s adv , a w ) is not sufﬁcient to target a w in the actual policy followed by the agent.
Revised: However, just trying to increase the probability of a w in the softmax policy i.e. π ( s adv , a w ) is not sufﬁcient to target a w in the actual policy followed by the agent.

Original: In fact the agent can end up in a state where,
Revised: In fact the agent can end up in a state where,

Original: [Equation]
Revised: [Equation]

Original: Although π ( s adv , a w ) has increased, the action a w will not be taken.
Revised: Although π ( s adv , a w ) has increased, the action a w will not be taken.

Original: However, it might be still possible to ﬁnd a perturbed state s (cid:48) adv for which,
Revised: However, it might be still possible to ﬁnd a perturbed state s (cid:48) adv for which,

Original: [Equation]
Revised: [Equation]

Original: Therefore, maximizing the probability of taking the worst possible action in the given state is not actually the right formulation to ﬁnd the correct direction for adversarial perturbation.
Revised: Therefore, maximizing the probability of taking the worst possible action in the given state is not actually the right formulation to ﬁnd the correct direction for adversarial perturbation.

Original:  O PTIMAL M YOPIC A DVERSARIAL F ORMULATION
Revised:  O PTIMAL M YOPIC A DVERSARIAL F ORMULATION

Original: To address the problem described in Section 2.3 we deﬁne an optimal myopic adversary to be an adversary which aims to minimize the value of the action taken by the agent myopically for each state.
Revised: To address the problem described in Section 2.3 we deﬁne an optimal myopic adversary to be an adversary which aims to minimize the value of the action taken by the agent myopically for each state.

Original: The value of the action chosen by the agent in the unperturbed state is Q ( s, a ∗ ( s )) , and the value (measured in the unperturbed state) of the action chosen by the agent under the inﬂuence of the adversarial observation is Q ( s, a ∗ ( s adv )) .
Revised: The value of the action chosen by the agent in the unperturbed state is Q ( s, a ∗ ( s )) , and the value (measured in the unperturbed state) of the action chosen by the agent under the inﬂuence of the adversarial observation is Q ( s, a ∗ ( s adv )) .

Original: The difference between these is the actual impact of the attack.
Revised: The difference between these is the actual impact of the attack.

Original: Therefore, in each state s the optimal myopic adversary must solve the following optimization problem
Revised: Therefore, in each state s the optimal myopic adversary must solve the following optimization problem

Original: [Equation]
Revised: [Equation]

Original: By (4), we may rewrite (8) in terms of the softmax policies π ( s, a ) ,
Revised: By (4), we may rewrite (8) in terms of the softmax policies π ( s, a ) ,

Original: [Equation]
Revised: [Equation]

Original: Since π ( s, a ∗ ( s )) does not depend on s adv , (9) is equivalent to solving
Revised: Since π ( s, a ∗ ( s )) does not depend on s adv , (9) is equivalent to solving

Original: [Equation]
Revised: [Equation]

Original: 3. E FFICIENT A PPROXIMATION OF O PTIMAL M YOPIC A DVERSARIAL F ORMULATION
Revised: 3. E FFICIENT A PPROXIMATION OF O PTIMAL M YOPIC A DVERSARIAL F ORMULATION

Original: Having an arg max operator in the cost function is unpleasant, since it is non-differentiable.
Revised: Having an arg max operator in the cost function is unpleasant, since it is non-differentiable.

Original: Instead we can approximate arg max by decreasing the temperature T for π T ( s adv , a ) ,
Revised: Instead we can approximate arg max by decreasing the temperature T for π T ( s adv , a ) ,

Original: [Equation]
Revised: [Equation]

Original: The intuition for (11) is that, by (3), as we decrease the temperature the value of e ( Q ( s,a ∗ ) / T adv ) for the action a ∗ which maximizes Q ( s, a ) will dominate the other actions.
Revised: The intuition for (11) is that, by (3), as we decrease the temperature the value of e ( Q ( s,a ∗ ) / T adv ) for the action a ∗ which maximizes Q ( s, a ) will dominate the other actions.

Original: Thus π ( s, arg max a { π ( s adv , a ) } ) can be approximated by,
Revised: Thus π ( s, arg max a { π ( s adv , a ) } ) can be approximated by,

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: our original optimization problem can be expressed as,
Revised: our original optimization problem can be expressed as,

Original: [Equation]
Revised: [Equation]

Original: In practice we will not be decreasing T adv to 0 as this would be equal to applying the non-differentiable argmax operation.
Revised: In practice we will not be decreasing T adv to 0 as this would be equal to applying the non-differentiable argmax operation.

Original: Instead we will replace the arg max operation with the approximation π T adv ( s, a ) for a small value of T adv .
Revised: Instead we will replace the arg max operation with the approximation π T adv ( s, a ) for a small value of T adv .

Original: In general it is not guarenteed that the minimum of the cost function in (14) with this approximation will be close to the minimum of the original cost function with the arg max operation.
Revised: In general it is not guarenteed that the minimum of the cost function in (14) with this approximation will be close to the minimum of the original cost function with the arg max operation.

Original: To see why, note that this approximation is equivalent to ﬁrst switching the limit and minimum in (14) to obtain,
Revised: To see why, note that this approximation is equivalent to ﬁrst switching the limit and minimum in (14) to obtain,

Original: [Equation]
Revised: [Equation]

Original: and second replacing the limit with a small value of T adv .
Revised: and second replacing the limit with a small value of T adv .

Original: There are two possible issues with this approach.
Revised: There are two possible issues with this approach.

Original: First of all, due to the non-convexity of the minimand, exchanging the limit and minimum may not yield an equality between (14) and (15).
Revised: First of all, due to the non-convexity of the minimand, exchanging the limit and minimum may not yield an equality between (14) and (15).

Original: Secondly, even if this exchange is legitimate, it is not clear how to choose a sufﬁciently small value for T adv in order to obtain a good approximation to (15).
Revised: Secondly, even if this exchange is legitimate, it is not clear how to choose a sufﬁciently small value for T adv in order to obtain a good approximation to (15).

Original: However, we show in our experiments that using this approximation in the cost function gives state-of-the-art results.
Revised: However, we show in our experiments that using this approximation in the cost function gives state-of-the-art results.

Original: There is another caveat which applies to all myopic adversaries including those in prior work.
Revised: There is another caveat which applies to all myopic adversaries including those in prior work.

Original: The
Revised: The

Original: Q-value is a good estimate of the discounted future rewards of the agent, only assuming that the agent continues to take the action maximizing the Q-value in future states.
Revised: Q-value is a good estimate of the discounted future rewards of the agent, only assuming that the agent continues to take the action maximizing the Q-value in future states.

Original: Since myopic attacks are applied in each state, this may make minimizing the Q-value a non-optimal attack strategy when the future is taken into account.
Revised: Since myopic attacks are applied in each state, this may make minimizing the Q-value a non-optimal attack strategy when the future is taken into account.

Original: There is also always the risk that the agent’s Q-value is miscalibrated.
Revised: There is also always the risk that the agent’s Q-value is miscalibrated.

Original: However, our experiments show that our myopic formulation performs well despite these potential limitations.
Revised: However, our experiments show that our myopic formulation performs well despite these potential limitations.

Original: 3. E XPERIMENTAL S ETUP
Revised: 3. E XPERIMENTAL S ETUP

Original: In our experiments we averaged over 10 episodes for each Atari game (Bellemare et al., 2013) in
Revised: In our experiments we averaged over 10 episodes for each Atari game (Bellemare et al., 2013) in

Original: Figure 1 from the Open AI gym environment (Brockman et al., 2016).
Revised: Figure 1 from the Open AI gym environment (Brockman et al., 2016).

Original: Agents are trained with
Revised: Agents are trained with

Original: Double DQN (Wang et al., 2016).
Revised: Double DQN (Wang et al., 2016).

Original: We compared the attack impact of our Myopic formulation with the previous formulations of Huang et al.
Revised: We compared the attack impact of our Myopic formulation with the previous formulations of Huang et al.

Original: (2017) and Pattanaik et al.
Revised: (2017) and Pattanaik et al.

Original: The attack impact has been normalized by comparing an unattacked agent, which chooses the action corresponding to the maximum state-action value in each state, with an agent that chooses the action corresponding to the minimum state-action value in each state.
Revised: The attack impact has been normalized by comparing an unattacked agent, which chooses the action corresponding to the maximum state-action value in each state, with an agent that chooses the action corresponding to the minimum state-action value in each state.

Original: Formally, let R max be the average return for the agent who always chooses the best action in a given state, let R min be the average return for the agent who always chooses the worst possible action in a given state, and let R a be the average return of the agent under attack.
Revised: Formally, let R max be the average return for the agent who always chooses the best action in a given state, let R min be the average return for the agent who always chooses the worst possible action in a given state, and let R a be the average return of the agent under attack.

Original: We deﬁne the impact I, as
Revised: We deﬁne the impact I, as

Original: [Equation]
Revised: [Equation]

Original: This normalization was chosen because we observed that in Atari environments agents can still collect stochastic rewards even when choosing the worst possible action in each state until the game ends.
Revised: This normalization was chosen because we observed that in Atari environments agents can still collect stochastic rewards even when choosing the worst possible action in each state until the game ends.

Original: See more details of the setup in Appendix A.2.
Revised: See more details of the setup in Appendix A.2.

Original: 3. A DVERSARIAL T EMPERATURE
Revised: 3. A DVERSARIAL T EMPERATURE

Original: Based on the discussion in Section 3 we expect that as T adv decreases the function in (14) becomes a better approximation of the arg max function.
Revised: Based on the discussion in Section 3 we expect that as T adv decreases the function in (14) becomes a better approximation of the arg max function.

Original: Thus this results in a higher attack impact as T adv decreases.
Revised: Thus this results in a higher attack impact as T adv decreases.

Original: However, after a certain threshold the function in (14) becomes too close to the arg max
Revised: However, after a certain threshold the function in (14) becomes too close to the arg max

Original: [Table]
Revised: [Table]

Original: 0.482 ± 0.07 Seaquest 0.386 ± 0.
Revised: 0.482 ± 0.07 Seaquest 0.386 ± 0.

Original: 0.468 ± 0.
Revised: 0.468 ± 0.

Original: 0.576 ± 0.17 Pong 0.000 ± 0.
Revised: 0.576 ± 0.17 Pong 0.000 ± 0.

Original: 0.043 ± 0.
Revised: 0.043 ± 0.

Original: 0.916 ± 0.09 UpNDown -0.007 ± 0.
Revised: 0.916 ± 0.09 UpNDown -0.007 ± 0.

Original: 0.436 ± 0.
Revised: 0.436 ± 0.

Original: 0.892 ± 0.
Revised: 0.892 ± 0.

Original: function which is non-differentiable.
Revised: function which is non-differentiable.

Original: Therefore, in practice, beyond this threshold the quality of the solutions given by the gradient based optimization will decrease, and so the attack impact will be lower.
Revised: Therefore, in practice, beyond this threshold the quality of the solutions given by the gradient based optimization will decrease, and so the attack impact will be lower.

Original: Indeed this can be observed in Figure 2.
Revised: Indeed this can be observed in Figure 2.

Original: In our experiments we chose T adv to maximize the impact by grid search.
Revised: In our experiments we chose T adv to maximize the impact by grid search.

Original: 3. I MPACT C OMPARISON FOR (cid:96) p - NORM B OUNDED P ERTURBATIONS
Revised: 3. I MPACT C OMPARISON FOR (cid:96) p - NORM B OUNDED P ERTURBATIONS

Original: Table 1 shows the mean and standard deviation of the impact values under (cid:96) 1 and (cid:96) 2 -norm bounds.
Revised: Table 1 shows the mean and standard deviation of the impact values under (cid:96) 1 and (cid:96) 2 -norm bounds.

Original: The tables show that the proposed attack results in higher mean impact for all games and under all norms, and it results in a lower standard deviation almost always.
Revised: The tables show that the proposed attack results in higher mean impact for all games and under all norms, and it results in a lower standard deviation almost always.

Original: In particular, this is an indication that our myopic attack formulation achieves higher impact more consistently than the previous formulations.
Revised: In particular, this is an indication that our myopic attack formulation achieves higher impact more consistently than the previous formulations.

Original: More results on (cid:96) ∞ -norm bound can be found in Section A.1 of the Appendix.
Revised: More results on (cid:96) ∞ -norm bound can be found in Section A.1 of the Appendix.

Original: Figure 3 shows the attack impact as a function of the perturbation bound for each formulation in three games.
Revised: Figure 3 shows the attack impact as a function of the perturbation bound for each formulation in three games.

Original: As (cid:15) decreases, our myopic formulation exhibits higher impact relative to the other formulations.
Revised: As (cid:15) decreases, our myopic formulation exhibits higher impact relative to the other formulations.

Original: Recall that Goodfellow et al.
Revised: Recall that Goodfellow et al.

Original: (2015) argue that small adversarial perturbations shift the input across approximately linear decision boundaries learned by neural networks.
Revised: (2015) argue that small adversarial perturbations shift the input across approximately linear decision boundaries learned by neural networks.

Original: Therefore, having a higher impact with smaller norm bound is evidence that our myopic formulation ﬁnds a better direction (i.e. one that points more directly at such a decision boundary) to search for the perturbation.
Revised: Therefore, having a higher impact with smaller norm bound is evidence that our myopic formulation ﬁnds a better direction (i.e. one that points more directly at such a decision boundary) to search for the perturbation.

Original: [Figure]
Revised: [Figure]

Original: 3. D ISTRIBUTION ON A CTIONS T AKEN
Revised: 3. D ISTRIBUTION ON A CTIONS T AKEN

Original: In order to understand the superior performance of the proposed adversarial formulation it is worthwhile to analyze the distribution of the actions taken by the agents under attack.
Revised: In order to understand the superior performance of the proposed adversarial formulation it is worthwhile to analyze the distribution of the actions taken by the agents under attack.

Original: For each formulation, we recorded the empirical probability p ( s, a k ) that the attacked agent chooses the k th ranked action a k .
Revised: For each formulation, we recorded the empirical probability p ( s, a k ) that the attacked agent chooses the k th ranked action a k .

Original: The ranking of the actions is according to their value in the unperturbed state.
Revised: The ranking of the actions is according to their value in the unperturbed state.

Original: Without perturbation agents would always take their ﬁrst ranked action, any deviation from this is a consequence of the adversarial perturbations.
Revised: Without perturbation agents would always take their ﬁrst ranked action, any deviation from this is a consequence of the adversarial perturbations.

Original: We average these values over 10 episodes to obtain the average empirical probability that the k th action is taken E e ∼ ρ
Revised: We average these values over 10 episodes to obtain the average empirical probability that the k th action is taken E e ∼ ρ

Original: [ p ( s, a k )].
Revised: [ p ( s, a k )].

Original: Here ρ is the distribution on the episodes e of the game induced by the stochastic nature of the Atari games.
Revised: Here ρ is the distribution on the episodes e of the game induced by the stochastic nature of the Atari games.

Original: We plot the results in Figure 4.
Revised: We plot the results in Figure 4.

Original: The legends in Figure 4 show the average empirical probabilities of taking the best action a ∗ .
Revised: The legends in Figure 4 show the average empirical probabilities of taking the best action a ∗ .

Original: It can be seen that in general E e ∼ ρ [ p ( s, a ∗ )] is lower and E e ∼ ρ [ p ( s, a w )] is higher for our Myopic formulation when compared to other attack formulations.
Revised: It can be seen that in general E e ∼ ρ [ p ( s, a ∗ )] is lower and E e ∼ ρ [ p ( s, a w )] is higher for our Myopic formulation when compared to other attack formulations.

Original: We realized that the formulations which have higher impact at the end of the game might still be choosing the best action more often than the ones which have lower attack impact as shown in the legend of Figure 4 for the Pong game.
Revised: We realized that the formulations which have higher impact at the end of the game might still be choosing the best action more often than the ones which have lower attack impact as shown in the legend of Figure 4 for the Pong game.

Original: In this case both Huang et al.
Revised: In this case both Huang et al.

Original: (2017) and Pattanaik et al.
Revised: (2017) and Pattanaik et al.

Original: (2018) cause the agent to choose the best action a ∗ less frequently than our formulation, but still end up with lower impact.
Revised: (2018) cause the agent to choose the best action a ∗ less frequently than our formulation, but still end up with lower impact.

Original: If we look at Figure 4 it can be seen that E e ∼ ρ [ p ( s, a 2 nd )] is much higher for both Huang et al.
Revised: If we look at Figure 4 it can be seen that E e ∼ ρ [ p ( s, a 2 nd )] is much higher for both Huang et al.

Original: (2017) and Pattanaik et al.
Revised: (2017) and Pattanaik et al.

Original: compared to our Myopic formulation.
Revised: compared to our Myopic formulation.

Original: However, the key to achieving a greater impact is to cause the agent to choose the lowest ranked actions more frequently.
Revised: However, the key to achieving a greater impact is to cause the agent to choose the lowest ranked actions more frequently.

Original: As can be seen from Figure 4, our Myopic formulation does this more successfully.
Revised: As can be seen from Figure 4, our Myopic formulation does this more successfully.

Original: More detailed results can be found in Appendix A.5.
Revised: More detailed results can be found in Appendix A.5.

Original: 3. S TATE -A CTION V ALUES OVER T IME
Revised: 3. S TATE -A CTION V ALUES OVER T IME

Original: In this section we investigate state-action values of the agents over time without an attack, under attack with the Pattanaik et al.
Revised: In this section we investigate state-action values of the agents over time without an attack, under attack with the Pattanaik et al.

Original: (2018) formulation, and under attack with our myopic formulation.
Revised: (2018) formulation, and under attack with our myopic formulation.

Original: In Table 2 it is interesting to observe that under attack the mean of the state-action values over the episodes might be higher despite the average return being lower.
Revised: In Table 2 it is interesting to observe that under attack the mean of the state-action values over the episodes might be higher despite the average return being lower.

Original: One might think that an attack with greater impact might visit lower valued states on average.
Revised: One might think that an attack with greater impact might visit lower valued states on average.

Original: However, in our experiments we have found that the gap between E e ∼ ρ [ Q ( s, a ∗ ( s ))] and E e ∼ ρ [ Q ( s, a ∗ ( s adv )] relative to the gap
Revised: However, in our experiments we have found that the gap between E e ∼ ρ [ Q ( s, a ∗ ( s ))] and E e ∼ ρ [ Q ( s, a ∗ ( s adv )] relative to the gap

Original: [Table]
Revised: [Table]

Original: between E e ∼ ρ [ Q ( s, a ∗ ( s ))]
Revised: between E e ∼ ρ [ Q ( s, a ∗ ( s ))]

Original: and E e ∼ ρ [ Q ( s, a w )] is a much more signiﬁcant factor in determining the magnitude of the impact.
Revised: and E e ∼ ρ [ Q ( s, a w )] is a much more signiﬁcant factor in determining the magnitude of the impact.

Original: Therefore, we deﬁne the quantity
Revised: Therefore, we deﬁne the quantity

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Observe in Table 2 the magnitudes of E e ∼ ρ [ Q ( s, a ∗ ( s ))] , E e ∼ ρ [ Q ( s, a w )] , and E e ∼ ρ [ Q ( s, a ∗ ( s adv )] are not strictly correlated to the impacts for each formulation.
Revised: Observe in Table 2 the magnitudes of E e ∼ ρ [ Q ( s, a ∗ ( s ))] , E e ∼ ρ [ Q ( s, a w )] , and E e ∼ ρ [ Q ( s, a ∗ ( s adv )] are not strictly correlated to the impacts for each formulation.

Original: However, Q loss is always higher for our Myopic attack compared to the previous formulations.
Revised: However, Q loss is always higher for our Myopic attack compared to the previous formulations.

Original: Recall that in Equation (3), our original optimization problem was designed to maximize the gap between Q ( s, a ∗ ( s )) and Q ( s, a ∗ ( s adv )) in each state.
Revised: Recall that in Equation (3), our original optimization problem was designed to maximize the gap between Q ( s, a ∗ ( s )) and Q ( s, a ∗ ( s adv )) in each state.

Original: The results in Table 2 are evidence both that we achieve this original goal and that solving our initial optimization problem for each state leads to lower average return at the end of the game.
Revised: The results in Table 2 are evidence both that we achieve this original goal and that solving our initial optimization problem for each state leads to lower average return at the end of the game.

Original: More results on this matter can be found in Appendix A.4.
Revised: More results on this matter can be found in Appendix A.4.

Original: The mean Q-value per episode under attack increases in some games.
Revised: The mean Q-value per episode under attack increases in some games.

Original: We believe that this may be due to the fact that the Q-value in a given state is only an accurate representation of the expected rewards of an unattacked agent.
Revised: We believe that this may be due to the fact that the Q-value in a given state is only an accurate representation of the expected rewards of an unattacked agent.

Original: When the agent is under attack, there might be states with high
Revised: When the agent is under attack, there might be states with high

Original: Q-value which are dangerous for the attacked agent (e.g. states where the attack causes the agent to immediately lose the game).
Revised: Q-value which are dangerous for the attacked agent (e.g. states where the attack causes the agent to immediately lose the game).

Original: More results can be found in Appendix A.3.
Revised: More results can be found in Appendix A.3.

Original:  F UTURE E XTENSIONS TO C ONTINUOUS A CTION S ETS
Revised:  F UTURE E XTENSIONS TO C ONTINUOUS A CTION S ETS

Original: We note in this section that, at least from a mathematical point of view, it is possible to extend our formulation to continuous control tasks.
Revised: We note in this section that, at least from a mathematical point of view, it is possible to extend our formulation to continuous control tasks.

Original: Observe that the adversarial objective in (8) applies to continuous control problems too.
Revised: Observe that the adversarial objective in (8) applies to continuous control problems too.

Original: The difﬁculty we are describing in (10) is not an issue in continuous control tasks.
Revised: The difﬁculty we are describing in (10) is not an issue in continuous control tasks.

Original: For instance, in Deep Deterministic Policy Gradient (DDPG) or Proximal Policy
Revised: For instance, in Deep Deterministic Policy Gradient (DDPG) or Proximal Policy

Original: Optimization (PPO) the adversarial policy is already approximated by the actor network µ θ ( s ) .
Revised: Optimization (PPO) the adversarial policy is already approximated by the actor network µ θ ( s ) .

Original: That is: a ∗ ( s adv ) =
Revised: That is: a ∗ ( s adv ) =

Original: µ θ ( s adv ) .
Revised: µ θ ( s adv ) .

Original: Unlike in the case of discrete action sets, this means that the solution to the problem in (8) can be approximated through gradient descent following,
Revised: Unlike in the case of discrete action sets, this means that the solution to the problem in (8) can be approximated through gradient descent following,

Original: [Equation]
Revised: [Equation]

Original: where the gradient is taken with respect to s adv .
Revised: where the gradient is taken with respect to s adv .

Original: In a wayactually it is  easier to construct the adversarial examples in continuous control tasks because the learning algorithm already produces a differentiable approximation µ θ ( s ) to the argmax operation used in action selection.
Revised: In a way it is actually easier to construct the adversarial examples in continuous control tasks because the learning algorithm already produces a differentiable approximation µ θ ( s ) to the argmax operation used in action selection.

Original: In our paper we focused on the derivation for the case of a discrete action set because the optimization problem in (10) is harder to solve.
Revised: In our paper we focused on the derivation for the case of a discrete action set because the optimization problem in (10) is harder to solve.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper we studied formulations of adversarial attacks on deep reinforcement learning agents and deﬁned the optimal myopic adversary which incorporates the action taken in the adversarial state into the cost function.
Revised: In this paper we studied formulations of adversarial attacks on deep reinforcement learning agents and deﬁned the optimal myopic adversary which incorporates the action taken in the adversarial state into the cost function.

Original: By introducing a differentiable approximation to the value of the action taken by the agent under the inﬂuence of the adversary we ﬁnd a direction for adversarial perturbation which more effectively decreases the Q -value of the agent in the given state.
Revised: By introducing a differentiable approximation to the value of the action taken by the agent under the inﬂuence of the adversary we ﬁnd a direction for adversarial perturbation which more effectively decreases the Q -value of the agent in the given state.

Original: In our experiments we demonstrated the efﬁcacy of our formulation as compared to the previous formulations in the Atari environment for various games.
Revised: In our experiments we demonstrated the efﬁcacy of our formulation as compared to the previous formulations in the Atari environment for various games.

Original: Adversarial formulations are inital steps towards building resilient and reliable DeepRL agents, and we believe our adversarial formulation can help to set a new baseline towards the robustiﬁcation of DeepRL algorithms
Revised: Adversarial formulations are inital steps towards building resilient and reliable DeepRL agents, and we believe our adversarial formulation can help to set a new baseline towards the robustiﬁcation of DeepRL algorithms

Original: A A PPENDIX A. (cid:96) ∞ - NORM B OUND
Revised: A A PPENDIX A. (cid:96) ∞ - NORM B OUND

Original: [Table]
Revised: [Table]

Original: One observation from these results is that the myopic formulation performs worse in the (cid:96) ∞ -norm bound than it does in the (cid:96) 1 -norm bound and the (cid:96) 2 -norm bound when (cid:15) =
Revised: One observation from these results is that the myopic formulation performs worse in the (cid:96) ∞ -norm bound than it does in the (cid:96) 1 -norm bound and the (cid:96) 2 -norm bound when (cid:15) =

Original: 10 − 8 .
Revised: 10 − 8 .

Original: A priori one might expect the performance of the myopic formulation to best in the (cid:96) ∞ -norm bound because the unit (cid:96) ∞ ball contains the unit balls of the (cid:96) 1 and (cid:96) 2 norms.
Revised: A priori one might expect the performance of the myopic formulation to best in the (cid:96) ∞ -norm bound because the unit (cid:96) ∞ ball contains the unit balls of the (cid:96) 1 and (cid:96) 2 norms.

Original: To investigate this further we examined the behaviour of (cid:96) 2 -norm bound and (cid:96) ∞ -norm bound while varying (cid:15) in Figure (5).
Revised: To investigate this further we examined the behaviour of (cid:96) 2 -norm bound and (cid:96) ∞ -norm bound while varying (cid:15) in Figure (5).

Original: Observe that at very small scales the (cid:96) 2 -norm bounded perturbation has larger impact, but at larger scales the (cid:96) ∞ -norm bounded perturbation generally performs better than the (cid:96) 2 -norm bounded perturbation.
Revised: Observe that at very small scales the (cid:96) 2 -norm bounded perturbation has larger impact, but at larger scales the (cid:96) ∞ -norm bounded perturbation generally performs better than the (cid:96) 2 -norm bounded perturbation.

Original: We link this phenomenon to the decision boundary behaviour in different scales.
Revised: We link this phenomenon to the decision boundary behaviour in different scales.

Original: Recall that in the (cid:96) 2 -norm bound we use a perturbation in the gradient direction of length (cid:15) .
Revised: Recall that in the (cid:96) 2 -norm bound we use a perturbation in the gradient direction of length (cid:15) .

Original: However, in the (cid:96) ∞ -norm bound we use a perturbation given by (cid:15) times the sign of the gradient.
Revised: However, in the (cid:96) ∞ -norm bound we use a perturbation given by (cid:15) times the sign of the gradient.

Original: This corresponds to the (cid:96) ∞ -norm bounded perturbation which causes the maximum possible change for a linear function.
Revised: This corresponds to the (cid:96) ∞ -norm bounded perturbation which causes the maximum possible change for a linear function.

Original: Compared to previous formulations decreasing the temperature in the softmax makes our objective function more nonlinear.
Revised: Compared to previous formulations decreasing the temperature in the softmax makes our objective function more nonlinear.

Original: Thus, at smaller scales it is crucial for the perturbation to be in the exact direction of the gradient, since the decision boundary behaves nonlinearly.
Revised: Thus, at smaller scales it is crucial for the perturbation to be in the exact direction of the gradient, since the decision boundary behaves nonlinearly.

Original: However, at larger scales the decision boundary is approximately linear, which gives a better result in the (cid:96) ∞ -norm bound.
Revised: However, at larger scales the decision boundary is approximately linear, which gives a better result in the (cid:96) ∞ -norm bound.

Original: A. R ANDOMIZED I TERATIVE S EARCH AND I MPACTS
Revised: A. R ANDOMIZED I TERATIVE S EARCH AND I MPACTS

Original: It can be seen from Figure 6 that the impact of the Pattanaik et al.
Revised: It can be seen from Figure 6 that the impact of the Pattanaik et al.

Original: (2018) formulation increases as n increases, while our myopic formulation has a greater impact even when n is equal to 1 .
Revised: (2018) formulation increases as n increases, while our myopic formulation has a greater impact even when n is equal to 1 .

Original: In particular, this is again an indication that our cost function provides a better direction in which to search for adversarial perturbations as compared to previous formulations.
Revised: In particular, this is again an indication that our cost function provides a better direction in which to search for adversarial perturbations as compared to previous formulations.

Original: Our attack, as well as those of Huang et al.
Revised: Our attack, as well as those of Huang et al.

Original: (2017) and Pattanaik et al.
Revised: (2017) and Pattanaik et al.

Original: (2018), are all based on computing the gradient of the respective cost functions and choosing a point along the gradient direction to be the adversarial perturbation.
Revised: (2018), are all based on computing the gradient of the respective cost functions and choosing a point along the gradient direction to be the adversarial perturbation.

Original: Pattanaik et al.
Revised: Pattanaik et al.

Original: (2018) used a randomized search, where n random points are sampled in the gradient direction and the one with lowest Q -value is chosen as the perturbation.
Revised: (2018) used a randomized search, where n random points are sampled in the gradient direction and the one with lowest Q -value is chosen as the perturbation.

Original: We follow Pattanaik et al.
Revised: We follow Pattanaik et al.

Original: (2018) in using randomized search, and in all previous results mentioned above we have set n = 5 .
Revised: (2018) in using randomized search, and in all previous results mentioned above we have set n = 5 .

Original: In Table 4 we set n = 1 and compare the impact of the three different formulations.
Revised: In Table 4 we set n = 1 and compare the impact of the three different formulations.

Original: Even in this more restrictive setting the performance of our formulation remains high, while the performance of Pattanaik et al.
Revised: Even in this more restrictive setting the performance of our formulation remains high, while the performance of Pattanaik et al.

Original: (2018) degrades signiﬁcantly.
Revised: (2018) degrades signiﬁcantly.

Original: In particular, this demonstrates that the gradient of our cost function is a better direction to ﬁnd an adversarial perturbation.
Revised: In particular, this demonstrates that the gradient of our cost function is a better direction to ﬁnd an adversarial perturbation.

Original: [Table]
Revised: [Table]

Original: A. G AMES AND A GENT B EHAVIOUR
Revised: A. G AMES AND A GENT B EHAVIOUR

Original: In this section we share our observations on the behaviour of the trained agent under attack.
Revised: In this section we share our observations on the behaviour of the trained agent under attack.

Original: In Figure 9 the agent performs well until it suddenly decides to stand still and wait for the enemy to arrive.
Revised: In Figure 9 the agent performs well until it suddenly decides to stand still and wait for the enemy to arrive.

Original: Similarly, in Figure 7 the trained agent performs quite well again until it decides to jump in front of the truck.
Revised: Similarly, in Figure 7 the trained agent performs quite well again until it decides to jump in front of the truck.

Original: Finally, in Figure 8 the trained agent forgets to recharge its oxygen even though it is earning many points from shooting the ﬁshes and saving the divers.
Revised: Finally, in Figure 8 the trained agent forgets to recharge its oxygen even though it is earning many points from shooting the ﬁshes and saving the divers.

Original: A. Q- VALUES OVER T IME
Revised: A. Q- VALUES OVER T IME

Original: In this section we plot state-action values of the agents over time without an attack, under attack with the Pattanaik et al.
Revised: In this section we plot state-action values of the agents over time without an attack, under attack with the Pattanaik et al.

Original: (2018) formulation, and under attack with our myopic formulation.
Revised: (2018) formulation, and under attack with our myopic formulation.

Original: In Figure 12 it is interesting to observe that under attack the mean of the state-action values over the episodes might be higher despite the average return being lower. .
Revised: In Figure 12 it is interesting to observe that under attack the mean of the state-action values over the episodes might be higher despite the average return being lower. .

Original: A. R ESULTS ON AVERAGE EMPIRICAL PROBABILITIES OF ACTIONS
Revised: A. R ESULTS ON AVERAGE EMPIRICAL PROBABILITIES OF ACTIONS

Original: In this section we provide a detailed table on the average empirical probabilities of the best action a ∗ and the worst action a w .
Revised: In this section we provide a detailed table on the average empirical probabilities of the best action a ∗ and the worst action a w .

Original: It can be seen that in general E e ∼ ρ [ p ( s, a ∗ )] is lower and E e ∼ ρ [ p ( s, a w )] is higher for our Myopic formulation when compared to other attack formulations
Revised: It can be seen that in general E e ∼ ρ [ p ( s, a ∗ )] is lower and E e ∼ ρ [ p ( s, a w )] is higher for our Myopic formulation when compared to other attack formulations

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A. S ENSITIVITY A NALYSIS
Revised: A. S ENSITIVITY A NALYSIS

Original: In this section we attempt to gain an insight into which pixels in the visited states are most sensitive to perturbation.
Revised: In this section we attempt to gain an insight into which pixels in the visited states are most sensitive to perturbation.

Original: For each pixel i, j we measure the drop in the state action values when perturbing that pixel by a small amount.
Revised: For each pixel i, j we measure the drop in the state action values when perturbing that pixel by a small amount.

Original: Formally, let a sen = arg max a Q ( s sensitivity , a ) , where s sensitivity is equal to s except that a small perturbation γ has been added to the i, j -th pixel of s .
Revised: Formally, let a sen = arg max a Q ( s sensitivity , a ) , where s sensitivity is equal to s except that a small perturbation γ has been added to the i, j -th pixel of s .

Original: Then we measure
Revised: Then we measure

Original: [Equation]
Revised: [Equation]

Original: We plot the values from 19 in Figure 15.
Revised: We plot the values from 19 in Figure 15.

Original: Note that any non-zero value (i.e. any lighter colored pixel in the plot) indicates that a γ perturbation to the corresponding pixel will cause the agent to take an action different from the optimal one.
Revised: Note that any non-zero value (i.e. any lighter colored pixel in the plot) indicates that a γ perturbation to the corresponding pixel will cause the agent to take an action different from the optimal one.

Original: Lighter colored pixels correspond to larger drops in Q-value.
Revised: Lighter colored pixels correspond to larger drops in Q-value.

Original: A. P ERTURBATION H EATMAPS
Revised: A. P ERTURBATION H EATMAPS

Original: In thise section we demonstrate the heatmaps of the raw pixel perturbations and the preprocessed frames from which the perturbations are computed.
Revised: In thise section we demonstrate the heatmaps of the raw pixel perturbations and the preprocessed frames from which the perturbations are computed.