Original: T HE A DVERSARIAL R EGULATION OF THE T EMPORAL D IFFERENCE L OSS C OSTS M ORE T HAN E XPECTED
Revised: T HE A DVERSARIAL R EGULATION OF THE T EMPORAL D IFFERENCE L OSS C OSTS M ORE T HAN E XPECTED

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep reinforcement learning research has enabled reaching signiﬁcant performance levels for sequential decision making in MDPs with highly complex observations and state dynamics with the aid of deep neural networks.
Revised: Deep reinforcement learning research has enabled reaching signiﬁcant performance levels for sequential decision making in MDPs with highly complex observations and state dynamics with the aid of deep neural networks.

Original: However, this aid came with a cost that is inherent to deep neural networks which have increased sensitivities towards indistinguishable peculiarly crafted non-robust directions.
Revised: However, this aid came with a cost that is inherent to deep neural networks which have increased sensitivities towards indistinguishable peculiarly crafted non-robust directions.

Original: To alleviate these sensitivities several studies suggested techniques to cope with this problem via explicitly regulating the temporal difference loss for the worst-case sensitivity.
Revised: To alleviate these sensitivities several studies suggested techniques to cope with this problem via explicitly regulating the temporal difference loss for the worst-case sensitivity.

Original: In our study we show that these worst-case regularization techniques come with a cost that intriguingly causes inconsistencies and overestimations in the stateaction value functions.
Revised: In our study we show that these worst-case regularization techniques come with a cost that intriguingly causes inconsistencies and overestimations in the stateaction value functions.

Original: Furthermore, our results essentially demonstrate that vanilla trained deep reinforcement learning policies have more accurate and consistent estimates for the state-action values.
Revised: Furthermore, our results essentially demonstrate that vanilla trained deep reinforcement learning policies have more accurate and consistent estimates for the state-action values.

Original: We believe our results reveal foundational intrinsic properties of the adversarial training techniques and demonstrate the need to rethink the approach to robustness in deep reinforcement learning.
Revised: We believe our results reveal foundational intrinsic properties of the adversarial training techniques and demonstrate the need to rethink the approach to robustness in deep reinforcement learning.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Advancements in deep neural networks have recently proliferated leading to expansion in the domains where deep neural networks are utilized including image classiﬁcation (Krizhevsky et al., 2012), natural language processing (Sutskever et al., 2014), speech recognition (Hannun et al., 2014) and self learning systems via exploration.
Revised: Advancements in deep neural networks have recently proliferated leading to expansion in the domains where deep neural networks are utilized including image classiﬁcation (Krizhevsky et al., 2012), natural language processing (Sutskever et al., 2014), speech recognition (Hannun et al., 2014) and self learning systems via exploration.

Original: In particular, deep reinforcement learning has become an emerging ﬁeld with the introduction of deep neural networks as function approximators (Mnih et al., 2015).
Revised: In particular, deep reinforcement learning has become an emerging ﬁeld with the introduction of deep neural networks as function approximators (Mnih et al., 2015).

Original: Hence, deep neural policies have been deployed in many different domains from pharmaceuticals to self driving cars (Daochang & Jiang, 2018; Huan-Hsin et al., 2017; Noonan, 2017).
Revised: Hence, deep neural policies have been deployed in many different domains from pharmaceuticals to self driving cars (Daochang & Jiang, 2018; Huan-Hsin et al., 2017; Noonan, 2017).

Original: As the advancements in deep neural networks continued a line of research focused on their vulnerabilities towards a certain type of speciﬁcally crafted perturbations computed via the cost function used to train the neural network (Szegedy et al., 2014;
Revised: As the advancements in deep neural networks continued a line of research focused on their vulnerabilities towards a certain type of speciﬁcally crafted perturbations computed via the cost function used to train the neural network (Szegedy et al., 2014;

Original: Goodfellow et al., 2015; Madry et al., 2018;
Revised: Goodfellow et al., 2015; Madry et al., 2018;

Original: Kurakin et al., 2016; Dong et al., 2018).
Revised: Kurakin et al., 2016; Dong et al., 2018).

Original: While some research focused on producing optimal (cid:96) p -norm bounded perturbations to cause the most possible damage to the deep neural network models, an extensive amount of work focused on making the networks robust to such perturbations (Madry et al., 2018;
Revised: While some research focused on producing optimal (cid:96) p -norm bounded perturbations to cause the most possible damage to the deep neural network models, an extensive amount of work focused on making the networks robust to such perturbations (Madry et al., 2018;

Original: Carmon et al., 2019; Raghunathan et al., 2020).
Revised: Carmon et al., 2019; Raghunathan et al., 2020).

Original: The vulnerability to such particularly optimized adversarial directions was inherited by deep neural policies as well (Huang et al., 2017; Kos & Song, 2017; Korkmaz, 2022).
Revised: The vulnerability to such particularly optimized adversarial directions was inherited by deep neural policies as well (Huang et al., 2017; Kos & Song, 2017; Korkmaz, 2022).

Original: Thus, robustness to such perturbations in deep reinforcement learning became a concern for the machine learning community, and several studies proposed various methods to increase robustness (Pinto et al., 2017; Gleave et al., 2020).
Revised: Thus, robustness to such perturbations in deep reinforcement learning became a concern for the machine learning community, and several studies proposed various methods to increase robustness (Pinto et al., 2017; Gleave et al., 2020).

Original: Thus, in this paper we focus on adversarially trained deep neural policies and the state-action value function learned by these training methods in the presence of an adversary.
Revised: Thus, in this paper we focus on adversarially trained deep neural policies and the state-action value function learned by these training methods in the presence of an adversary.

Original: In more detail, in this paper we aim to seek answers for the following questions: (i) How accurate is the state-action value function on estimating the values for state-action pairs in MDPs with high dimensional state representations?
Revised: In more detail, in this paper we aim to seek answers for the following questions: (i) How accurate is the state-action value function on estimating the values for state-action pairs in MDPs with high dimensional state representations?

Original: , (ii) Does adversarial training affect the estimates of the state-action value function?
Revised: , (ii) Does adversarial training affect the estimates of the state-action value function?

Original: , (iii) What are the effects of training with worst-case distributional shift on the state-action value function representation for the optimal actions?
Revised: , (iii) What are the effects of training with worst-case distributional shift on the state-action value function representation for the optimal actions?

Original: and (iv) Are there any fundamental trade-offs intrinsic to explicit worst-case regularization in deep neural policy training?
Revised: and (iv) Are there any fundamental trade-offs intrinsic to explicit worst-case regularization in deep neural policy training?

Original: To be able to answer these questions we focus on adversarial training and robustness in deep neural policies and make the following contributions:
Revised: To be able to answer these questions we focus on adversarial training and robustness in deep neural policies and make the following contributions:

Original: • We conduct an investigation on the state-action values learnt by the state-of-the-art adversarially trained deep neural policies and vanilla trained deep neural policies.
Revised: • We conduct an investigation on the state-action values learnt by the state-of-the-art adversarially trained deep neural policies and vanilla trained deep neural policies.

Original: • We provide theoretically motivated justiﬁcation for how adversarial training might change the state-action value function.
Revised: • We provide theoretically motivated justiﬁcation for how adversarial training might change the state-action value function.

Original: • We perform several experiments in Atari games with large state spaces from the Arcade Learning Environment (ALE).
Revised: • We perform several experiments in Atari games with large state spaces from the Arcade Learning Environment (ALE).

Original: With our systematic analysis we show that vanilla trained deep neural policies have a more accurate representation of the sub-optimal actions compared to the state-of-the-art adversarially trained deep neural policies.
Revised: With our systematic analysis we show that vanilla trained deep neural policies have a more accurate representation of the sub-optimal actions compared to the state-of-the-art adversarially trained deep neural policies.

Original: Furthermore, we show the inconsistencies in the action ranking in the state-of-the-art adversarially trained deep neural policies.
Revised: Furthermore, we show the inconsistencies in the action ranking in the state-of-the-art adversarially trained deep neural policies.

Original: Thus, these results demonstrate the loss of information in state-action value function as a novel fundamental trade-off intrinsic to adversarial training.
Revised: Thus, these results demonstrate the loss of information in state-action value function as a novel fundamental trade-off intrinsic to adversarial training.

Original: • More importantly, we demonstrate that state-of-the-art adversarially trained deep neural policies learn overestimated state-action value functions.
Revised: • More importantly, we demonstrate that state-of-the-art adversarially trained deep neural policies learn overestimated state-action value functions.

Original: • Finally, we explain how our results call into question the hypothesis initially proposed by Bellemare et al.
Revised: • Finally, we explain how our results call into question the hypothesis initially proposed by Bellemare et al.

Original: (2016) relating the action gap and overestimation.
Revised: (2016) relating the action gap and overestimation.

Original:  B ACKGROUND AND P RELIMINARIES2. P RELIMINARIES
Revised:  B ACKGROUND AND P RELIMINARIES

Revised: Preliminaries:

Original: In deep reinforcement learning the goal is to learn a policy for taking actions in aMarkov Decision
Revised: In deep reinforcement learning the goal is to learn a policy for taking actions in a

Original:  Process (MDP) that maximize discounted expected cumulative reward.
Revised: Markov Decision Process (MDP) that maximize discounted expected cumulative reward.

Original: An MDP is represented by a tuple M =
Revised: An MDP is represented by a tuple M =

Original: ( S, A, P, r, ρ 0 , γ ) where S is a set of continuous states, A is a discrete set of actions, P is a transition probability distribution on S ×
Revised: ( S, A, P, r, ρ 0 , γ ) where S is a set of continuous states, A is a discrete set of actions, P is a transition probability distribution on S ×

Original: A × S , r : S ×
Revised: A × S , r : S ×

Original: A → R is a reward function, ρ 0 is the initial state distribution, and γ is the discount factor.
Revised: A → R is a reward function, ρ 0 is the initial state distribution, and γ is the discount factor.

Original: The goal in reinforcement learning is to learn a policy π : S → P ( A ) which maps states to probability distributions on actions in order to maximize the expected cumulative reward R = E (cid:80) T − 1 t =

Revised: The goal in reinforcement learning is to learn a policy π : S → P ( A ) which maps states to probability distributions on actions in order to maximize the expected cumulative reward R = E (cid:80) T − 1 t =

Original: γ t r ( s t , a t ) where a t ∼ π ( s t ) .
Revised: γ t r ( s t , a t ) where a t ∼ π ( s t ) .

Original: In Q -learning Watkins (1989) the goal is to learn the optimal state-action value function
Revised: In Q -learning Watkins (1989) the goal is to learn the optimal state-action value function

Original: Q ∗ ( s, a ) =
Revised: Q ∗ ( s, a ) =

Original: R ( s, a ) + (cid:80) s (cid:48)
Revised: R ( s, a ) + (cid:80) s (cid:48)

Original: ∈ S P ( s (cid:48) | s, a ) max a
Revised: ∈ S P ( s (cid:48) | s, a ) max a

Original: (cid:48) ∈
Revised: (cid:48) ∈

Original: A Q ∗ ( s (cid:48) , a (cid:48) ) .
Revised: A Q ∗ ( s (cid:48) , a (cid:48) ) .

Original: Thus, the optimal policy is determined by choosing the action a ∗ ( s ) =
Revised: Thus, the optimal policy is determined by choosing the action a ∗ ( s ) =

Original: arg max a Q ( s, a ) in state s .
Revised: arg max a Q ( s, a ) in state s .

Original:  Szegedy et al.
Revised: Adversarial Crafting and Training: Szegedy et al.

Original: (2014) observed that imperceptible perturbations could change the decision of a deep neural network and proposed a box constrained optimization method to produce such perturbations.
Revised: (2014) observed that imperceptible perturbations could change the decision of a deep neural network and proposed a box constrained optimization method to produce such perturbations.

Original: Goodfellow et al. (2015) suggested a faster method to produce such perturbations based on the linearization of the cost function used in training the network.
Revised: Goodfellow et al. (2015) suggested a faster method to produce such perturbations based on the linearization of the cost function used in training the network.

Original: Kurakin et al.
Revised: Kurakin et al.

Original: (2016) proposed the iterative version of the fast gradient sign method proposed by Goodfellow et al.
Revised: (2016) proposed the iterative version of the fast gradient sign method proposed by Goodfellow et al.

Original: (2015) inside an (cid:15) -ball.
Revised: (2015) inside an (cid:15) -ball.

Original: [Equation]
Revised: [Equation]

Original: in which J ( x, y ) represents the cost function used to train the deep neural network, x represents the input, and y represents the output labels.
Revised: in which J ( x, y ) represents the cost function used to train the deep neural network, x represents the input, and y represents the output labels.

Original: While several other methods have been proposed (e.g.
Revised: While several other methods have been proposed (e.g.

Original: Korkmaz (2020)) using a momentum-based extension of the iterative fast gradient sign method,
Revised: Korkmaz (2020)) using a momentum-based extension of the iterative fast gradient sign method,

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: (3) adversarial training has mostly been conducted with perturbations computed by projected gradient descent (PGD) proposed by Madry et al.
Revised: adversarial training has mostly been conducted with perturbations computed by projected gradient descent (PGD) proposed by Madry et al.

Original: (2018) (i.e. Equation 1).
Revised: (2018) (i.e. Equation 1).

Original:  The initial investigation on resilience of deep neural policies was conducted by Kos & Song (2017) and Huang et al.
Revised: Adversaries and Training in Deep Neural Policies: The initial investigation on resilience of deep neural policies was conducted by Kos & Song (2017) and Huang et al.

Original: (2017) concurrently based on the utilization of the fast gradient sign method proposed by Goodfellow et al.
Revised: (2017) concurrently based on the utilization of the fast gradient sign method proposed by Goodfellow et al.

Original: Korkmaz (2022) showed that deep reinforcement learning policies learn shared adversarial features across MDPs.
Revised: Korkmaz (2022) showed that deep reinforcement learning policies learn shared adversarial features across MDPs.

Original: While several studies focused on improving optimization techniques to compute optimal perturbations, a line of research focused on making deep neural policies resilient to these perturbations.
Revised: While several studies focused on improving optimization techniques to compute optimal perturbations, a line of research focused on making deep neural policies resilient to these perturbations.

Original: Mandlekar et al.
Revised: Mandlekar et al.

Original: (2017) proposed including these perturbations in training time to increase resilience for robotic setups.
Revised: (2017) proposed including these perturbations in training time to increase resilience for robotic setups.

Original: Pinto et al.
Revised: Pinto et al.

Original: (2017) proposed to model the dynamics between the adversary and the deep neural policy as a zero-sum game where the goal of the adversary is to minimize expected cumulative rewards of the deep neural policy.
Revised: (2017) proposed to model the dynamics between the adversary and the deep neural policy as a zero-sum game where the goal of the adversary is to minimize expected cumulative rewards of the deep neural policy.

Original: Gleave et al.
Revised: Gleave et al.

Original: (2020) approached this problem with an adversary model which is restricted to take natural actions in the MDP instead of modifying the observations with (cid:96) p -norm bounded perturbations.
Revised: (2020) approached this problem with an adversary model which is restricted to take natural actions in the MDP instead of modifying the observations with (cid:96) p -norm bounded perturbations.

Original: The authors model this dynamic as a zero-sum Markov game and solve it via self play.
Revised: The authors model this dynamic as a zero-sum Markov game and solve it via self play.

Original: Recently, Huan et al.
Revised: Recently, Huan et al.

Original: (2020) proposed to model this interaction between the adversary and the deep neural policy as a state-adversarial MDP, and claimed that their proposed algorithm State Adversarial Double Deep Q-Network (SA-DDQN) learns theoretically certiﬁed robust policies against natural noise and perturbations.
Revised: (2020) proposed to model this interaction between the adversary and the deep neural policy as a state-adversarial MDP, and claimed that their proposed algorithm State Adversarial Double Deep Q-Network (SA-DDQN) learns theoretically certiﬁed robust policies against natural noise and perturbations.

Original: More recently, several empirical concerns have been raised on the robustness of theoretically certiﬁed adversarially trained deep neural policies Ezgi (2021).
Revised: More recently, several empirical concerns have been raised on the robustness of theoretically certiﬁed adversarially trained deep neural policies Ezgi (2021).

Original: In our work, we systematically investigate and theoretically motivate the problems caused by adversarial training on the state-action value function learned by deep neural policies.
Revised: In our work, we systematically investigate and theoretically motivate the problems caused by adversarial training on the state-action value function learned by deep neural policies.

Original:  A DVERSARIAL T RAINING AND THE S TATE -A CTION V ALUE F UNCTION
Revised:  A DVERSARIAL T RAINING AND THE S TATE -A CTION V ALUE F UNCTION

Original: In this paper we aim to answer the following questions:
Revised: In this paper we aim to answer the following questions:

Original: • How does training with explicit worst-case regularization affect the estimates of the optimal state-action values in MDPs with high dimensional state representations?
Revised: • How does training with explicit worst-case regularization affect the estimates of the optimal state-action values in MDPs with high dimensional state representations?

Original: • What is the accuracy of the state-action value function representation for the non-optimal actions in deep neural policies?
Revised: • What is the accuracy of the state-action value function representation for the non-optimal actions in deep neural policies?

Original: • Does state-of-the-art adversarial training affect the state-action value estimates?
Revised: • Does state-of-the-art adversarial training affect the state-action value estimates?

Original: • Are there any intrinsic trade-offs tied to adversarial deep neural policy training
Revised: • Are there any intrinsic trade-offs tied to adversarial deep neural policy training?

Original: While the goal in Q -learning is to learn the state-action value function Q ( s, a ) that maximizes expected discounted cumulative rewards, in deep Q -learning an additional concern arises from susceptibility towards adversarial perturbations due to the nonlinear function approximator used in learning the Q -function.
Revised: While the goal in Q -learning is to learn the state-action value function Q ( s, a ) that maximizes expected discounted cumulative rewards, in deep Q -learning an additional concern arises from susceptibility towards adversarial perturbations due to the nonlinear function approximator used in learning the Q -function.

Original: Ideally, one might hope that adversarial training would reduce the vulnerability of the Q -function to adversarial perturbations while preserving the Q -values of the non-perturbed states as much as possible.
Revised: Ideally, one might hope that adversarial training would reduce the vulnerability of the Q -function to adversarial perturbations while preserving the Q -values of the non-perturbed states as much as possible.

Original: The theoretically motivated adversarial training techniques achieve certiﬁed defense against adversarial perturbations inside the (cid:15) -ball D (cid:15) ( s ) =
Revised: The theoretically motivated adversarial training techniques achieve certiﬁed defense against adversarial perturbations inside the (cid:15) -ball D (cid:15) ( s ) =

Original: { ¯ s : (cid:107) s − ¯ s (cid:107) ∞ ≤ (cid:15) } .
Revised: { ¯ s : (cid:107) s − ¯ s (cid:107) ∞ ≤ (cid:15) } .

Original: However, we show that this approach induces signiﬁcant changes in the Q -function so that
Revised: However, we show that this approach induces signiﬁcant changes in the Q -function so that

Original: the
Revised: the

Original: Q -function loses its accuracy for the non-perturbed states.
Revised: Q -function loses its accuracy for the non-perturbed states.

Original: In particular, adversarial training causes deep neural policies to learn overestimated state-action values, and the Q -values for non-optimal actions are reduced in accuracy to the point where their relative ranking changes.
Revised: In particular, adversarial training causes deep neural policies to learn overestimated state-action values, and the Q -values for non-optimal actions are reduced in accuracy to the point where their relative ranking changes.

Original: In the remainder of this section we give theoretical motivation for these empirical results.
Revised: In the remainder of this section we give theoretical motivation for these empirical results.

Original: In particular we give a simple example using linear function approximation which can potentially lead to overestimation for the Q -values of the optimal actions, and reordering of the ranking of non-optimal actions.
Revised: In particular we demonstrate that in the setting of linear function approximation, adversarial training can potentially lead to overestimation for the Q -values of the optimal actions, and reordering of the ranking of nonoptimal actions.

Original: The basic approach of adversarial training techniques is based on adding a regularizer to the standard Q -learning update.
Revised: The basic approach of adversarial training techniques is based on adding a regularizer to the standard Q -learning update.

Original: The regularizer is designed to penalize Q -functions for which a perturbed state ¯ s ∈ D (cid:15) ( s ) can change the identity of the highest Q -value action.
Revised: The regularizer is designed to penalize Q -functions for which a perturbed state ¯ s ∈ D (cid:15) ( s ) can change the identity of the highest Q -value action.

Original: For the baseline adversarial training technique we will theoretically analyze the effects of this regularizer.
Revised: For the baseline adversarial training technique we will theoretically analyze the effects of this regularizer.

Original: Deﬁnition 3.1 (Huan et al.
Revised: Deﬁnition 3.1 (Huan et al.

Original: For a state s let a ∗ ( s ) =
Revised: For a state s let a ∗ ( s ) =

Original: arg max a Q ( s, a ) .
Revised: arg max a Q ( s, a ) .

Original: The regularizer is given by
Revised: The regularizer is given by

Original: [Equation]
Revised: [Equation]

Original: The adversarial training algorithm proceeds by adding R ( θ ) to the standard temporal difference loss
Revised: The adversarial training algorithm proceeds by adding R ( θ ) to the standard temporal difference loss

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: used in DQN and minimizing via stochastic gradient descent.
Revised: used in DQN and minimizing via stochastic gradient descent.

Original: We now proceed with an example where Q θ is a linear function  of state feature vectors.
Revised: We now describe the construction of an MDP M with linear function approximation where the use of the regularizer causes overestimation and reordering of suboptimal actions. There are two states parametrized by feature vectors s 1 , s 2 ∈ R n , and there are three possible actions { a i } 3 i =1 in each state. Taking any of the three actions in state s 1 leads to a transition to state s 2 and vice versa. Let

Original: Suppose we have an MDP where every state corresponds to a feature vector s ∈ R n ,and there are three possible actions { a i } 3 i =1 in each state.
Revised: 1 > λ > 0 be the discount factor, and let δ > η > 0 be small constants with λ > δ > η . The rewards for each action are as follows: r ( s 1 , a 1 ) = 1 − λ , r ( s 1 , a 2 )

Original: Let the Q -function be parametrized by θ
Revised: = η − λ , r ( s 1 , a 3 ) = δ − λ , r ( s 2 , a 1 ) = η − λ , r ( s 2 , a 2 ) = 1 − λ , and r ( s 2 , a 3 ) =

Revised: δ − λ .

Revised: Clearly, the optimal policy is to always take action a 1 in state s 1 , and action a 2 in state s 2 as these are the only actions giving positive reward.

Revised: Thus the optimal state-action values are given by: Q ∗ ( s 1 , a 1 ) =

Revised: Q ∗ ( s 2 , a 2 ) =

Original: {+(cid:80) ∞ t =
+}
Revised: {+(cid:80) ∞ t =
+}

Revised: (1 − λ ) λ t = 1 ,

Revised: Q ∗ ( s 1 , a 2 ) =

Revised: Q ∗ ( s 2 , a 1 ) =

Revised: η − λ + λ (cid:80) ∞ t =0 (1 − λ ) λ t

Revised: = η , and Q ∗ ( s 1 , a 3 ) =

Revised: Q ∗ ( s 2 , a 3 ) =

Original: {+δ − λ + λ (cid:80) ∞ t =
+}
Revised: {+δ − λ + λ (cid:80) ∞ t =
+}

Revised: (1 − λ ) λ t = δ .

Revised: Let the Q -function be linearly parametrized by θ

Original: = ( θ 1 , θ 2 , θ 3 ) so that Q  ( s, a i ) =
Revised: = ( θ 1 , θ 2 , θ 3 ) so that Q θ ( s, a i ) =

Original: (cid:104)
Revised: (cid:104)

Original: θ i , s (cid:105) .
Revised: θ i , s (cid:105) .

Original: Finally, suppose that the optimal linear approximator for the Q -function is given by parameters θ ∗ such that the vectors θ ∗ i are orthonormal, and  the state embeddings fall into one of two types:
Revised: Finally, let θ ∗ i for i ∈ { 1 , 2 , 3 } be three orthonormal vectors, and let the state feature vectors satisfy:

Original: [Equation]
Revised: [Equation]

Original: where δ > η > 0 , and both are small constants e.g. δ =0 .
Revised: Then it follows that the optimal Q -function is parametrized by θ ∗ =

Original:  2 , η =0 .
Revised: ( θ ∗ 1 , θ ∗ 2 , θ ∗ 3 ) i.e. Q θ ∗ ( s, a ) =

Original:  Thus, according to the function Q θ ∗ ( s, a ) , for states of type 1 the best action is a 1 , for states of type 2 the best action is a 2 , and in all states the second-best action is a 3 .
Revised: Q ∗ ( s, a ) for all s and a . Thus, according to the function Q θ ∗ ( s, a ) , for s 1 the best action is a 1 , for s 2 the best action is a 2 , and in all states the second-best action is a 3 .

Original: Next we identify the optimal perturbations used in the computation of the regularizer R ( θ ∗ ) for this setting.
Revised: Next we identify the optimal perturbations used in the computation of the regularizer R ( θ ∗ ) for this setting.

Original: Proposition 3.1.
Revised: Proposition 3.1.

Original: In the setting of Example 3 suppose that (cid:15) < δ − η 2 .
Revised: In the MDP M suppose that (cid:15) < δ − η 2 .

Original: For  a state s of type 1:
Revised: For s = s 1 : s + (cid:15) √ 2 ( θ ∗ 3 − θ ∗ 1 ) = arg max ¯ s ∈ D (cid:15) ( s ) max a (cid:54) = a ∗ ( s ) Q θ ∗ (¯ s, a ) − Q θ ∗ (¯ s, a ∗ ( s )) 2.

Original: [Equation] Fora state s of type 2:
Revised: For s = s 2 : s + (cid:15) √ 2 ( θ ∗ 3 − θ ∗ 2 ) =

Original: [Equation]
Revised: arg max ¯ s ∈ D (cid:15) ( s ) max a (cid:54) = a ∗ ( s ) Q θ ∗ (¯ s, a ) − Q θ ∗ (¯ s, a ∗ ( s ))

Original: Proof.
Revised: Proof.

Original: We will prove item 1, and item 2 will follow from an identical argument with roles of θ ∗and θ ∗ 2 swapped.
Revised: We will prove item 1, and item 2 will follow from an identical argument with roles of θ ∗ 1 and θ ∗ 2 swapped.

Original: Let s be a state of type 1.
Revised: Let s = s 1 .

Original: Any ¯ s ∈ D (cid:15) ( s ) can be written as s + (cid:15)v where v is a unit vector.
Revised: Any ¯ s ∈ D (cid:15) ( s ) can be written as s + (cid:15)v where v is a unit vector.

Original: Thus, (cid:104) θ ∗ 3 , ¯ s (cid:105) =
Revised: Thus, (cid:104) θ ∗ 3 , ¯ s (cid:105) =

Original: (cid:104) θ ∗ 3 , s (cid:105)
Revised: (cid:104) θ ∗ 3 , s (cid:105)

Original: + (cid:15) (cid:104) θ ∗ 3 , v (cid:105) >
Revised: + (cid:15) (cid:104) θ ∗ 3 , v (cid:105) >

Original: (cid:104) θ ∗ 3 , s (cid:105) − (cid:15) =
Revised: (cid:104) θ ∗ 3 , s (cid:105) − (cid:15) =

Original: δ − (cid:15).
Revised: δ − (cid:15).

Original: Similarly we have (cid:104) θ ∗ 2 , ¯ s (cid:105) <
Revised: Similarly we have (cid:104) θ ∗ 2 , ¯ s (cid:105) <

Original: (cid:104) θ ∗ 2 , s (cid:105)
Revised: (cid:104) θ ∗ 2 , s (cid:105)

Original: + (cid:15) =
Revised: + (cid:15) =

Original: η + (cid:15) .
Revised: η + (cid:15) .

Original: Since (cid:15) <
Revised: Since (cid:15) <

Original: δ − η 2 , we conclude that (cid:104) θ ∗ 3 , ¯ s (cid:105) >
Revised: δ − η 2 , we conclude that (cid:104) θ ∗ 3 , ¯ s (cid:105) >

Original: (cid:104) θ ∗ 2 , ¯ s (cid:105) for all ¯ s ∈ D (cid:15) ( s ) .
Revised: (cid:104) θ ∗ 2 , ¯ s (cid:105) for all ¯ s ∈ D (cid:15) ( s ) .

Original: Therefore, in state s the action maximizing max a (cid:54) = a ∗ ( s ) Q θ ∗ (¯ s, a ) − Q θ ∗ (¯ s, a ∗ ( s )) will always be a 3 .
Revised: Therefore, in state s the action maximizing max a (cid:54) = a ∗ ( s ) Q θ ∗ (¯ s, a ) − Q θ ∗ (¯ s, a ∗ ( s )) will always be a 3 .

Original: This implies that
Revised: This implies that

Original: [Equation]
Revised: [Equation]

Original:  This is the maximum in a ball of radius (cid:15) around s of the linear function (cid:104) θ ∗ 3 − θ ∗ 1 , ¯ s (cid:105) .
Revised: [Equation] This is the maximum in a ball of radius (cid:15) around s of the linear function (cid:104) θ ∗ 3 − θ ∗ 1 , ¯ s (cid:105) .

Original: Therefore the maximum is achieved by ¯ s
Revised: Therefore the maximum is achieved by ¯ s

Original: = s + (cid:15) √ 2 ( θ ∗ 3 − θ ∗ 1 ) as desired.
Revised: = s + (cid:15) √ 2 ( θ ∗ 3 − θ ∗ 1 ) as desired.

Original: In words, the optimal direction to perturb a state sof type 1 in order to have a ∗ ( s ) (cid:54) = a ∗ (¯ s ) is toward θ ∗ 3 − θ ∗ 1 .
Revised: In words, the optimal direction to perturb the state s 1 in order to have a ∗ ( s ) (cid:54) = a ∗ (¯ s ) is toward θ ∗ 3 − θ ∗ 1 .

Original: Similarly for a state of type 2 , the optimal perturbation is toward θ ∗ 3 − θ ∗ 2 .
Revised: Similarly for the state s 2 , the optimal perturbation is toward θ ∗ 3 − θ ∗ 2 .

Original: Next we use this fact to show that in order to decrease the regularizer it is sufﬁcient to simply increase the magnitude of θ 1 and θ 2 , and decrease the magnitude of θ 3 .
Revised: Next we use this fact to show that in order to decrease the regularizer it is sufﬁcient to simply increase the magnitude of θand θ 2 , and decrease the magnitude of θ 3 .

Original: Proposition 3.2.
Revised: Proposition 3.2.

Original: In the setting of Example 3 let γ > 0 and suppose that (cid:15) <
Revised: In the MDP M let γ > 0 and suppose that (cid:15) <

Original: (1 − γ ) δ − (1+ γ ) η.
Revised: (1 − γ ) δ − (1+ γ ) η.

Original: Let θ = ( θ 1 , θ 2 , θ 3 ) be given by θ 1 = (1 + γ ) θ ∗ 1 , θ 2 = (1 + γ ) θ ∗ 2 and θ 3 = (1 − γ ) θ ∗ 3 .
Revised: Let θ = ( θ 1 , θ 2 , θ 3 ) be given by θ 1 = (1 + γ ) θ ∗ 1 , θ 2 = (1 + γ ) θ ∗ 2 and θ 3 = (1 − γ ) θ ∗ 3 .

Original: Then R ( θ ) <
Revised: Then R ( θ ) <

Original: R ( θ ∗ ) .
Revised: R ( θ ∗ ) .

Original: Proof.
Revised: Proof.

Original: By an identical argument to that in Proposition 3.1 we have that a 3 is always the action maximizing max a (cid:54) =
Revised: By an identical argument to that in Proposition 3.1 we have that a 3 is always the action maximizing max a (cid:54) =

Original: a ∗ ( s ) Q θ (¯ s, a ) − Q
Revised: a ∗ ( s ) Q θ (¯ s, a ) − Q

Original: θ
Revised: θ

Original: (¯ s, a ∗ ( s ))
Revised: (¯ s, a ∗ ( s ))

Original: whenever (cid:15) < (1 − γ ) δ − (1+ γ ) η.
Revised: whenever (cid:15) < (1 − γ ) δ − [Equation] 1+ [Equation]

Original: This condition is satisﬁed by assumption.
Revised: This condition is satisﬁed by assumption.

Original: Therefore, we conclude that in a state s of type 1, the optimal ¯ s ∈ D (cid:15) ( s ) for the scaled parameters θ is given by ¯ s 
Revised: Therefore, we conclude that for s = s 1 , the optimal ¯ s ∈ D (cid:15) ( s ) for the scaled parameters θ is given by ¯ s = s + [Equation]

Original: [-= s + (cid:15) √ 2(
-]
Revised: [-= s + (cid:15) √ 2(
-][Equation] 1+ [Equation]

Original: + γ 2 ) ( θ 3 − θ 1 ) .
Revised: 2 ) ( θ 3 − θ 1 ) .

Original: Therefore, the contribution to the sum deﬁning R ( θ ) from states sof type 1 is given by
Revised: Therefore, the contribution to the sum deﬁning R ( θ ) from state s 1 is given by

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where the last step uses the fact that s = θ ∗ 1 + δθ ∗ 3 + ηθ ∗ 2 and that the vectors θ ∗ i are orthonormal.
Revised: where the last step uses the fact that s = θ ∗ 1 + δθ ∗ 3 + ηθ ∗ 2 and that the vectors θ ∗ i are orthonormal.

Original: Next using the fact that (cid:112) 1 + γ 2 < 1 + γ for all γ > 0 we conclude
Revised: Next using the fact that (cid:112) 1 + γ 2 < 1 + γ for all γ > 0 we conclude

Original: [Equation]
Revised: [Equation]

Original: The ﬁnal inequality follows from the fact that (cid:15) <
Revised: The ﬁnal inequality follows from the fact that (cid:15) <

Original: δ 2 so (cid:15)γ √ 2 − γδ < 0 .
Revised: δ 2 so (cid:15)γ √ 2 − γδ < 0 .

Original: Switching to type 2 actions an identical proof (with θ 1 replaced by θ 2 ) yields the same value for the contribution of type 2 actions to the sum.
Revised: Switching to type 2 actions an identical proof (with θ 1 replaced by θ 2 ) yields the same value for the contribution of type 2 actions to the sum.

Original: By Proposition 3.1, the contribution of each type of state to the sum deﬁning R ( θ ∗ ) isgiven by
Revised: By Proposition 3.1, the contribution of each type of state to the sum deﬁning R ( θ ∗ ) is

Original: [Equation]
Revised: [Equation]

Original: Clearly the contribution of each state in 9 is strictly less than that in 10.
Revised: Clearly the contribution of each state in 7 is strictly less than that in 8.

Original: Therefore R ( θ ) <
Revised: Therefore R ( θ ) <

Original: R ( θ ∗ ) .
Revised: R ( θ ∗ ) .

Revised: Theorem 3.3.

Revised: If a linear state-action value function approximator Q θ ( s, a ) is used, then the regularizer R ( θ ) can lead to overestimation of the value of the optimal action, and re-ordering of the values of the suboptimal actions.

Revised: Proof.

Revised: Consider the MDP M with linear function approximation constructed above.

Original: Increasing the magnitude of θ ∗ 1 and θ ∗ 2 by a factor of 1 + γ leads to overestimation of the Q -value of the best action instates of both type 1 and type 2 by the same factor.
Revised: Increasing the magnitude of θ ∗ 1 and θ ∗ 2 by a factor of 1 + γ leads to overestimation of the Q -value of the best action in both state s 1 and s 2 by the same factor.

Original: Additionally decreasing the magnitude of θ ∗ 3 can lead to a change in the ranking of the suboptimal actions.
Revised: Additionally decreasing the magnitude of θ ∗ 3 can lead to a change in the ranking of the suboptimal actions.

Original: Indeed if 1+ γ 1 − γ > δη then a 3 will become the third ranked action in bothtypes of states.
Revised: Indeed if 1+ γ 1 − γ > δη then a 3 will become the third ranked action in both states.

Original: Therefore Proposition 3.2 indicates that changing θ to decrease the regularizer R ( θ ) can lead to both overestimation of the ﬁrst ranked action, and re-ordering of the ranking of the suboptimal actions.
Revised: Therefore, Proposition 3.2 proves that changing θ to decrease the regularizer R ( θ ) can lead to both overestimation of the ﬁrst ranked action, and re-ordering of the ranking of the suboptimal actions.

Original: While we showed how this can potentially happen for a simple example with linear function approximation, we will see that this is a general phenomenon which occurs with neural-network approximation of the Q -function in adversarially trained agents.
Revised: While we showed how this can potentially happen in the case of linear function approximation, we will see that this is a general phenomenon which occurs with neural-network approximation of the Q -function in adversarially trained agents.

Original: It is important to note that the issues we identify are a result of the fundamental differences between deep neural policies and classiﬁcation tasks where adversarial training has previously been applied.
Revised: It is important to note that the issues we identify are a result of the fundamental differences between deep neural policies and classiﬁcation tasks where adversarial training has previously been applied.

Original: In particular, the fact that the state-action value function Q ( s, a ) has a meaning (i.e. measuring expected cumulative rewards) with regard to the MDP beyond simply labelling the optimal action correctly is the root cause of the effects that we observe.
Revised: In particular, the fact that the state-action value function Q ( s, a ) has a meaning (i.e. measuring expected cumulative rewards) with regard to the MDP beyond simply labelling the optimal action correctly is the root cause of the effects that we observe.

Original: In other words, simply penalizing the state-action value function for assigning the wrong “label” to an adversarial example can have unintended, potentially detrimental consequences for learning an accurate state-action value function.
Revised: In other words, simply penalizing the state-action value function for assigning the wrong “label” to an adversarial example can have unintended, potentially detrimental consequences for learning an accurate state-action value function.

Original:  M EASURING T HE A CCURACY OF S TATE -A CTION V ALUES
Revised:  M EASURING T HE A CCURACY OF S TATE -A CTION V ALUES

Original: In this section we provide a methodology to measure the accuracy of the state-action value function in representing values for the non-optimal actions.
Revised: In this section we provide a methodology to measure the accuracy of the state-action value function in representing values for the non-optimal actions.

Original: At a high-level, our approach is based on action modiﬁcation and the relative performance drop P as deﬁned below:
Revised: At a high-level, our approach is based on action modiﬁcation and the relative performance drop P as deﬁned below:

Original: Deﬁnition 4.1.
Revised: Deﬁnition 4.1.

Original: The performance drop of an agent when modifying the agent’s actions is given by
Revised: The performance drop of an agent when modifying the agent’s actions is given by

Original: [Equation]
Revised: [Equation]

Original: where Score base represent the baseline run of the game with no action modiﬁcation, Score min represents the minimum score available for a given game, and Score actmod represents the run of the game where the actions of the agent are modiﬁed for a fraction of the state observations.
Revised: where Score base represent the baseline run of the game with no action modiﬁcation, Score min represents the minimum score available for a given game, and Score actmod represents the run of the game where the actions of the agent are modiﬁed for a fraction of the state observations.

Original: We now explain precisely how we propose to measure “accuracy” for non-optimal actions.
Revised: We now explain precisely how we propose to measure “accuracy” for non-optimal actions.

Original: Formally, let a i be the i th best action decided by the deep neural policy in a given state s (i.e. Q ( s, a ) is sorted in decreasing order, and a i is the action corresponding to i th largest Q -value).
Revised: Formally, let a i be the i th best action decided by the deep neural policy in a given state s (i.e. Q ( s, a ) is sorted in decreasing order, and a i is the action corresponding to i th largest Q -value).

Original: For a trained agent, the value of Q ( s, a i ) should represent the expected cumulative rewards obtained by taking action a i in state s , and then taking the highest Q -value action (i.e. a 1 ) in every subsequent state.
Revised: For a trained agent, the value of Q ( s, a i ) should represent the expected cumulative rewards obtained by taking action a i in state s , and then taking the highest Q -value action (i.e. a 1 ) in every subsequent state.

Original: Thus, a natural test to perform would be: pick a random state s , make the agent choose action a i in state s , and in all other states have the agent choose the highest Q -value action.
Revised: Thus, a natural test to perform would be: pick a random state s , make the agent choose action a i in state s , and in all other states have the agent choose the highest Q -value action.

Original: By comparing the relative performance drop P in this test to a clean run where the agent always takes the highest Q -value action, one can measure the decline in rewards caused by taking action a i .
Revised: By comparing the relative performance drop P in this test to a clean run where the agent always takes the highest Q -value action, one can measure the decline in rewards caused by taking action a i .

Original: Further, we can provide a measure of accuracy for the state-action value function by comparing the results of the test for each i ∈ { 1 , 2 . . .
Revised: Further, we can provide a measure of accuracy for the state-action value function by comparing the results of the test for each i ∈ { 1 , 2 . . .

Original: | A |} , and checking that the relative performance drops P i are in the correct order i.e.
Revised: | A |} , and checking that the relative performance drops P i are in the correct order i.e.

Original: [Equation] P | A | .
Revised: 0 = P 1 ≤ P 2 · · · ≤ P | A | .

Original: However, there is an issue with the above proposal.
Revised: However, there is an issue with the above proposal.

Original: It is often the case that there are many states s in which the action taken has very little impact on the ﬁnal rewards.
Revised: It is often the case that there are many states s in which the action taken has very little impact on the ﬁnal rewards.

Original: Instead, there are a relatively smaller number of critical states in which the action taken has a large impact.
Revised: Instead, there are a relatively smaller number of critical states in which the action taken has a large impact.

Original: Thus, picking a single random state s in which to take action a i will have a statistically insigniﬁcant impact on the ﬁnal rewards in the game.
Revised: Thus, picking a single random state s in which to take action a i will have a statistically insigniﬁcant impact on the ﬁnal rewards in the game.

Original: Therefore we modify the test described above by instead sampling a p -fraction of the states in the episode uniformly at random, and making the agent take action a i in each of the sampled states.
Revised: Therefore we modify the test described above by instead sampling a p -fraction of the states in the episode uniformly at random, and making the agent take action a i in each of the sampled states.

Original: We then record the relative performance drop as a function of p , yielding a reward curve P i ( p ) .
Revised: We then record the relative performance drop as a function of p , yielding a reward curve P i ( p ) .

Original: More formally, we deﬁne
Revised: More formally, we deﬁne

Original: Deﬁnition 4.2.
Revised: Deﬁnition 4.2.

Original: Let M be an MDP and Q ( s, a ) be a state-action value function for M .
Revised: Let M be an MDP and Q ( s, a ) be a state-action value function for M .

Original: In each state label the actions a 1 , . . .
Revised: In each state label the actions a 1 , . . .

Original: a | A | in order so that Q ( s, a 1 ) ≥ Q ( s, a 2 ) · · · ≥ Q ( s, a | A | ) .
Revised: a | A | in order so that Q ( s, a 1 ) ≥ Q ( s, a 2 ) · · · ≥ Q ( s, a | A | ) .

Original: We deﬁne the performance curve P i ( p ) to be the expected performance drop of an agent in M which takes action a i in a randomly sampled p -fraction of states, and takes action a 1 in all other states.
Revised: We deﬁne the performance curve P i ( p ) to be the expected performance drop of an agent in M which takes action a i in a randomly sampled p -fraction of states, and takes action a 1 in all other states.

Original: Using these reward curves one can check whether P i ( p ) lies above P j ( p ) whenever i > j .
Revised: Using these reward curves one can check whether P i ( p ) lies above P j ( p ) whenever i > j .

Original: Of course one curve may not always lie strictly above or below another, so we introduce the following deﬁnition to quantitatively capture the relative ordering of performance drop curves.
Revised: Of course one curve may not always lie strictly above or below another, so we introduce the following deﬁnition to quantitatively capture the relative ordering of performance drop curves.

Original: Deﬁnition 4.3.
Revised: Deﬁnition 4.3.

Original: Let F : [0 , 1] → [0 , 1] and G : [0 , 1] → [0 , 1] .
Revised: Let F : [0 , 1] → [0 , 1] and G : [0 , 1] → [0 , 1] .

Original: For any τ > 0 , we say that the F τ -dominates G if (cid:82) 1 0 ( F ( p ) − G ( p )) dp > τ .
Revised: For any τ > 0 , we say that the F τ -dominates G if (cid:82) 1 0 ( F ( p ) − G ( p )) dp > τ .

Original: To compare the accuracy of state-action values for vanilla versus adversarially trained agents, we can thus perform the above test, and check the relative ordering of the curves
Revised: To compare the accuracy of state-action values for vanilla versus adversarially trained agents, we can thus perform the above test, and check the relative ordering of the curves

Original: P i ( p ) using Deﬁnition 4.
Revised: P i ( p ) using Deﬁnition 4.

Original: for each agent type.
Revised: for each agent type.

Original: In addition, we can also directly compare for each i the curve P
Revised: In addition, we can also directly compare for each i the curve P

Original: adv i ( p ) for the adversarially trained agent with the curve P vanilla i ( p ) for the vanilla trained agent.
Revised: adv i ( p ) for the adversarially trained agent with the curve P vanilla i ( p ) for the vanilla trained agent.

Original: This is possible because P i ( p ) measures the performance drop of the agent relative to a clean run, and so always takes values on a normalized scale from 0 to 1 .
Revised: This is possible because P i ( p ) measures the performance drop of the agent relative to a clean run, and so always takes values on a normalized scale from 0 to 1 .

Original: Thus, if we observe for example that P adv 2 ( p ) τ -dominates
Revised: Thus, if we observe for example that P adv 2 ( p ) τ -dominates

Original: P vanilla( p ) for some τ > 0 , we can conclude that the state-action value function of the vanilla trained agent more accurately represents the second-best action than that of the adversarially trained agent.
Revised: P vanilla( p ) for some τ > 0 , we can conclude that the state-action value function of the vanilla trained agent more accurately represents the second-best action than that of the adversarially trained agent.

Original:  E XPERIMENTAL D ETAILS
Revised:  E XPERIMENTAL D ETAILS

Original: The experiments are conducted in high dimensional state representation MDPs.
Revised: The experiments are conducted in high dimensional state representation MDPs.

Original: In particular, our experiments are conducted in the Arcade Learning Environment (ALE) (Bellemare et al., 2013) in the
Revised: In particular, our experiments are conducted in the Arcade Learning Environment (ALE) (Bellemare et al., 2013) in the

Original: OpenAI (Brockman et al., 2016) baseline version.
Revised: OpenAI (Brockman et al., 2016) baseline version.

Original: The vanilla trained deep neural policy is trained via
Revised: The vanilla trained deep neural policy is trained via

Original: Double Deep Q-Network (DDQN) (Wang et al., 2016) initially proposed by Hasselt et al. (2016) with prioritized experience replay proposed by (Schaul et al., 2016), and the state-of-the-art adversarially trained deep neural policy is trained via State-Adversarial Double Deep Q-Network (SA-DDQN)
Revised: Double Deep Q-Network (DDQN) (Wang et al., 2016) initially proposed by Hasselt et al. (2016) with prioritized experience replay proposed by (Schaul et al., 2016), and the state-of-the-art adversarially trained deep neural policy is trained via State-Adversarial Double Deep Q-Network (SA-DDQN)

Original: (Section 2) with prioritized experience replay (Schaul et al., 2016).
Revised: (Section 2) with prioritized experience replay (Schaul et al., 2016).

Original: The results are averaged over 10 episodes.
Revised: The results are averaged over 10 episodes.

Original: We explain in detail all the necessary hyperparameters for the implementation in the supplementary material.
Revised: We explain in detail all the necessary hyperparameters for the implementation in the supplementary material.

Original: The standard error of the mean is included for all of the ﬁgures and tables.
Revised: The standard error of the mean is included for all of the ﬁgures and tables.

Original: Note that in the main body of the paper we focus on the baseline adversarial training.
Revised: Note that in the main body of the paper we focus on the baseline adversarial training.

Original: In the supplementary material we also provide analysis on the follow-up more recent studies in adversarial training techniques.
Revised: In the supplementary material we also provide analysis on the follow-up more recent studies in adversarial training techniques.

Original: The results reported for all of the adversarial training techniques remains the same that the adversarially trained policies learn inaccurate, inconsistent and overestimated state-action values.
Revised: The results reported for all of the adversarial training techniques remains the same that the adversarially trained policies learn inaccurate, inconsistent and overestimated state-action values.

Original:  A N A NALYSIS ON THE S TATE -A CTION V ALUE F UNCTION R EPRESENTATION
Revised:  A N A NALYSIS ON THE S TATE -A CTION V ALUE F UNCTION R EPRESENTATION

Original: In this section we demonstrate that the state-action value function of adversarially trained deep neural policies provides inaccurate estimates for the non-optimal actions, and learns overestimated state-action values.
Revised: In this section we demonstrate that the state-action value function of adversarially trained deep neural policies provides inaccurate estimates for the non-optimal actions, and learns overestimated state-action values.

Original: This conﬁrms that the theoretically-motivated problems discussed in Section 3 do indeed occur in practice for deep neural policies.
Revised: This conﬁrms that the theoretically-motivated problems discussed in Section 3 do indeed occur in practice for deep neural policies.

Original: In particular, to evaluate the accuracy on non-optimal actions we use the methodology discussed in Section 4 of measuring the performance 
Revised: In particular, to evaluate the accuracy on non-optimal actions we use the methodology discussed in Section 4 of measuring the performance drop P i ( p ) that occurs when causing the deep neural policy to take the i -th best action in a p fraction

Original: [Table]
Revised: [Table]

Original: drop P i ( p ) that occurs when causing the deep neural policy to take the i -th best action in a p fraction of states.
Revised: of states.

Original: Our aim is to provide an analysis on how accurate the state-action value function is in representing values for both optimal and non-optimal actions for vanilla trained deep neural policies and state-of-the-art adversarially trained deep neural policies.
Revised: Our aim is to provide an analysis on how accurate the state-action value function is in representing values for both optimal and non-optimal actions for vanilla trained deep neural policies and state-of-the-art adversarially trained deep neural policies.

Original: 6. I NACCURACY OF S TATE -A CTION V ALUES FOR N ON - OPTIMAL A CTIONS
Revised: 6. I NACCURACY OF S TATE -A CTION V ALUES FOR N ON - OPTIMAL A CTIONS

Original: In Figure 1 we show the performance drop P 2 ( p ) as a function of the fraction of states p in which the action modiﬁcation is applied for state-of-the-art adversarially trained deep neural policies and vanilla trained deep neural policies.
Revised: In Figure 1 we show the performance drop P 2 ( p ) as a function of the fraction of states p in which the action modiﬁcation is applied for state-of-the-art adversarially trained deep neural policies and vanilla trained deep neural policies.

Original: In particular, the action modiﬁcation is set for the second best action a 2 decided by the state-action value function Q ( s, a ) .
Revised: In particular, the action modiﬁcation is set for the second best action a 2 decided by the state-action value function Q ( s, a ) .

Original: As we increase the fraction of states in which the action modiﬁcation set to a 2 is applied, we observe a performance drop for both of the deep neural policies.
Revised: As we increase the fraction of states in which the action modiﬁcation set to a 2 is applied, we observe a performance drop for both of the deep neural policies.

Original: However, we observe that the vanilla trained deep neural policies experience a lower performance drop with this modiﬁcation.
Revised: However, we observe that the vanilla trained deep neural policies experience a lower performance drop with this modiﬁcation.

Original: Especially in BankHeist we observe that the performance drop does not exceed 0 .
Revised: Especially in BankHeist we observe that the performance drop does not exceed 0 .

Original: 55 even when the action modiﬁcation is applied for a large fraction of the visited states for the vanilla trained deep neural policies.
Revised: 55 even when the action modiﬁcation is applied for a large fraction of the visited states for the vanilla trained deep neural policies.

Original: This gap in the performance drop between the adversarially trained and vanilla trained deep neural policies indicates that the state-action value function learnt by vanilla trained deep neural policies has a better estimate for the non-optimal actions.
Revised: This gap in the performance drop between the adversarially trained and vanilla trained deep neural policies indicates that the state-action value function learnt by vanilla trained deep neural policies has a better estimate for the non-optimal actions.

Original: As we measured the impact of a 2 modiﬁcation on the policy performance, we further test a w = arg min a Q ( s, a ) modiﬁcation (i.e. worst possible action in a given state modiﬁcation) on the deep neural policy.
Revised: As we measured the impact of a 2 modiﬁcation on the policy performance, we further test a w = arg min a Q ( s, a ) modiﬁcation (i.e. worst possible action in a given state modiﬁcation) on the deep neural policy.

Original: Figure 2 shows that the performance drop P w ( p ) is higher in the vanilla trained deep neural policies compared to adversarially trained deep neural policies when the action modiﬁcation is set to a w .
Revised: Figure 2 shows that the performance drop P w ( p ) is higher in the vanilla trained deep neural policies compared to adversarially trained deep neural policies when the action modiﬁcation is set to a w .

Original: This again further demonstrates that the state-action value function learnt by the vanilla trained deep neural policy has a more accurate representation over the non-optimal actions.
Revised: This again further demonstrates that the state-action value function learnt by the vanilla trained deep neural policy has a more accurate representation over the non-optimal actions.

Original: We hypothesize that adversarial training places higher emphasis on ensuring that the highest ranked action (i.e. the action that maximizes the state-action value function in a given state) does not change under small (cid:96) p -norm bounded perturbations, rather than accurately computing the state-action value function as discussed in Section 3.
Revised: We hypothesize that adversarial training places higher emphasis on ensuring that the highest ranked action (i.e. the action that maximizes the state-action value function in a given state) does not change under small (cid:96) p -norm bounded perturbations, rather than accurately computing the state-action value function as discussed in Section 3.

Original: Since historically Q -learning suffered from overestimation of Q -values, a method which places higher emphasis on the highest ranked action risks converging to a state-action value function with overestimated Q -values.
Revised: Since historically Q -learning suffered from overestimation of Q -values, a method which places higher emphasis on the highest ranked action risks converging to a state-action value function with overestimated Q -values.

Original: We further demonstrate this in Section 6.2.
Revised: We further demonstrate this in Section 6.2.

Original: 6. O VERESTIMATION OF Q- VALUES IN A DVERSARIALLY T RAINED D EEP N EURAL P OLICIES
Revised: 6. O VERESTIMATION OF Q- VALUES IN A DVERSARIALLY T RAINED D EEP N EURAL P OLICIES

Original: Overestimation of Q -values was initially discussed by Thrun & Schwartz (1993) as a byproduct of the use of function approximators, and was subsequently explained as being caused by the use of the max operator in Q -learning (van Hasselt, 2010).
Revised: Overestimation of Q -values was initially discussed by Thrun & Schwartz (1993) as a byproduct of the use of function approximators, and was subsequently explained as being caused by the use of the max operator in Q -learning (van Hasselt, 2010).

Original: Furthermore, overestimation bias resulting in learning of sub-optimal policies was demonstrated in practice by Hasselt et al.
Revised: Furthermore, overestimation bias resulting in learning of sub-optimal policies was demonstrated in practice by Hasselt et al.

Original: In this subsection we empirically demonstrate that state-of-the-art adversarial training indeed leads to overestimation in
Revised: In this subsection we empirically demonstrate that state-of-the-art adversarial training indeed leads to overestimation in

Original: Q -values, as hypothesized in Section 3.
Revised: Q -values, as hypothesized in Section 3.

Original: In particular, Figure 4 and Table 2 show the overestimation bias on the state-action values learned by the state-of-the-art adversarially trained deep neural policies.
Revised: In particular, Figure 4 and Table 2 show the overestimation bias on the state-action values learned by the state-of-the-art adversarially trained deep neural policies.

Original: [Table]
Revised: [Table]

Original: Considering that overestimation bias is still an issue and active area of research for vanilla deep neural policy training (Lan et al., 2020; Anschel et al., 2017; Kuznetsov et al., 2020), the additional bias introduced intrinsic to adversarial training must be addressed to be able to learn optimal policies.
Revised: Considering that overestimation bias is still an issue and active area of research for vanilla deep neural policy training (Lan et al., 2020; Anschel et al., 2017; Kuznetsov et al., 2020), the additional bias introduced intrinsic to adversarial training must be addressed to be able to learn optimal policies.

Original: 6. I NCONSISTENCIES IN A CTION R ANKING IN A DVERSARIALLY T RAINED D EEP N EURAL P OLICIES
Revised: 6. I NCONSISTENCIES IN A CTION R ANKING IN A DVERSARIALLY T RAINED D EEP N EURAL P OLICIES

Original: In this subsection we demonstrate the inconsistencies in the non-optimal action ranking in adversarially trained policies.
Revised: In this subsection we demonstrate the inconsistencies in the nonoptimal action ranking in adversarially trained policies.

Original: In particular, in Figure 3 in BankHeist choosing the worst action leads to a smaller performance drop than choosing the second best action i.e. P w ( p ) <
Revised: In particular, in Figure 3 in BankHeist choosing the worst action leads to a smaller performance drop than choosing the second best action i.e. P w ( p ) <

Original: P 2 ( p ) for all p .
Revised: P 2 ( p ) for all p .

Original: Thus, this demonstrates that the state-action value function is not ranking the sub-optimal actions accurately.
Revised: Thus, this demonstrates that the state-action value function is not ranking the sub-optimal actions accurately.

Original: While learning an accurate representation of the state-action values is important for obtaining a policy that aims to maximize expected cumulative rewards, learning the correct order of the actions can also solve this problem.
Revised: While learning an accurate representation of the state-action values is important for obtaining a policy that aims to maximize expected cumulative rewards, learning the correct order of the actions can also solve this problem.

Original: Furthermore, in some cases the deep neural policy indeed must know the correct order of the actions due to the presence of an obstruction that blocks the optimal action either due to the existence of other agents or environmental effects (Rashid et al., 2020; Gleave et al., 2020).
Revised: Furthermore, in some cases the deep neural policy indeed must know the correct order of the actions due to the presence of an obstruction that blocks the optimal action either due to the existence of other agents or environmental effects (Rashid et al., 2020; Gleave et al., 2020).

Original: In particular, in safe reinforcement learning several algorithms have been proposed to learn the ranking of the actions so that the agent can choose the next-best ranked action in safety critical situations (Alshiekh et al., 2018).
Revised: In particular, in safe reinforcement learning several algorithms have been proposed to learn the ranking of the actions so that the agent can choose the next-best ranked action in safety critical situations (Alshiekh et al., 2018).

Original: Some work has also pointed out that in some cases learning the relative rank of the actions (Lin & Zhou, 2020) can be more sample efﬁcient than learning correct estimates of the state-action values.
Revised: Some work has also pointed out that in some cases learning the relative rank of the actions (Lin & Zhou, 2020) can be more sample efﬁcient than learning correct estimates of the state-action values.

Original: While the inconsistency in action ranking for adversarially trained deep neural policies can be seen as a vulnerability problem from a security point of view, most intriguingly these results demonstrate the loss of information in state-action value function as a novel fundamental trade-off intrinsic to adversarial training.
Revised: While the inconsistency in action ranking for adversarially trained deep neural policies can be seen as a vulnerability problem from a security point of view, most intriguingly these results demonstrate the loss of information in state-action value function as a novel fundamental trade-off intrinsic to adversarial training.

Original: 6. A CTION G AP P HENOMENON
Revised: 6. A CTION G AP P HENOMENON

Original: The action gap is deﬁned as the difference between the state-action value of the optimal action and the state-action value of the second ranked action.
Revised: The action gap is deﬁned as the difference between the state-action value of the optimal action and the state-action value of the second ranked action.

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Initially, Farahmand (2011) describes the existence of a large action gap as a desirable property of an MDP, which makes learning an optimal policy easier.
Revised: Initially, Farahmand (2011) describes the existence of a large action gap as a desirable property of an MDP, which makes learning an optimal policy easier.

Original: Subsequently, Bellemare et al.
Revised: Subsequently, Bellemare et al.

Original: (2016) proposed a connection between the action gap and the overestimation of Q -values, and in particular hypothesized that increasing the action gap of the learned value function causes a decrease in overestimation of Q -values.
Revised: (2016) proposed a connection between the action gap and the overestimation of Q -values, and in particular hypothesized that increasing the action gap of the learned value function causes a decrease in overestimation of Q -values.

Original: Following this study, several papers built on the hypothesis that increasing
Revised: Following this study, several papers built on the hypothesis that increasing

Original: [Table]
Revised: [Table]

Original: the action gap causes reduction in bias (Smirnova & Dohmatob, 2020;
Revised: the action gap causes reduction in bias (Smirnova & Dohmatob, 2020;

Original: Fox et al., 2016; Jain et al., 2020; Lu et al., 2019).
Revised: Fox et al., 2016; Jain et al., 2020; Lu et al., 2019).

Original: In Figure 5 we show that adversarial training increases the action gap.
Revised: In Figure 5 we show that adversarial training increases the action gap.

Original: Thus, the fact that adversarially trained deep neural policies overestimate the optimal state-action values (see Section 6.2) refutes the hypothesis that increasing the action gap is the sole cause of a decrease in overestimation bias of state-action values.
Revised: Thus, the fact that adversarially trained deep neural policies overestimate the optimal state-action values (see Section 6.2) refutes the hypothesis that increasing the action gap is the sole cause of a decrease in overestimation bias of state-action values.

Original: We hypothesize that the consistent Bellman operator (Bellemare et al., 2016) may cause a decrease in overestimation for a different reason.
Revised: We hypothesize that the consistent Bellman operator (Bellemare et al., 2016) may cause a decrease in overestimation for a different reason.

Original: In particular, the consistent Bellman operator corresponds to a special case of a certain reparameterization of
Revised: In particular, the consistent Bellman operator corresponds to a special case of a certain reparameterization of

Original: Kullback-Leibler regularization for value iteration (Vieillard et al., 2020).
Revised: Kullback-Leibler regularization for value iteration (Vieillard et al., 2020).

Original: Thus, it may be the case that the decrease in overestimation of Q -values and improvement in performance is due to a type of implicit regularization rather than to an increase of the action gap.
Revised: Thus, it may be the case that the decrease in overestimation of Q -values and improvement in performance is due to a type of implicit regularization rather than to an increase of the action gap.

Original: Hence, our results show that increasing the action gap alone may coincide with an increase in overestimation of Q -values.
Revised: Hence, our results show that increasing the action gap alone may coincide with an increase in overestimation of Q -values.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper we focus on the state-action value function learnt via the state-of-the-art adversarially trained deep neural policies and vanilla trained deep neural policies.
Revised: In this paper we focus on the state-action value function learnt via the state-of-the-art adversarially trained deep neural policies and vanilla trained deep neural policies.

Original: We provide theoretical analysis on the fundamental effects caused by adversarial training on the state-action value function.
Revised: We provide theoretical analysis on the fundamental effects caused by adversarial training on the state-action value function.

Original: Furthermore, we conduct manifold experiments in the Arcade Learning Environment and with our systematic analysis we demonstrate that vanilla trained deep neural policies have more accurate and consistent estimates for the state-action values than the state-of-the-art adversarially trained deep neural policies.
Revised: Furthermore, we conduct manifold experiments in the Arcade Learning Environment and with our systematic analysis we demonstrate that vanilla trained deep neural policies have more accurate and consistent estimates for the state-action values than the state-of-the-art adversarially trained deep neural policies.

Original: More intriguingly, we show that adversarially trained deep neural policies in certain MDPs completely loses all the information in the state-action value function that contains the relative ranking of the actions.
Revised: More intriguingly, we show that adversarially trained deep neural policies in certain MDPs completely loses all the information in the state-action value function that contains the relative ranking of the actions.

Original: More importantly, we show that state-of-the-art adversarially trained deep neural policies learn overestimated state-action values.
Revised: More importantly, we show that state-of-the-art adversarially trained deep neural policies learn overestimated state-action values.

Original: We believe our investigation lays out intrinsic properties of adversarial training while systematically revealing the underlying vulnerabilities, and can be conducive to building robust and optimal deep neural policies.
Revised: We believe our investigation lays out intrinsic properties of adversarial training while systematically revealing the underlying vulnerabilities, and can be conducive to building robust and optimal deep neural policies.