Original: D AYLIGHT : A SSESSING G ENERALIZATION S KILLS OF D EEP R EINFORCEMENT L EARNING A GENTS
Revised: D AYLIGHT : A SSESSING G ENERALIZATION S KILLS OF D EEP R EINFORCEMENT L EARNING A GENTS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep reinforcement learning algorithms have recently achieved signiﬁcant success in learning high-performing policies from purely visual observations.
Revised: Deep reinforcement learning algorithms have recently achieved signiﬁcant success in learning high-performing policies from purely visual observations.

Original: The ability to perform end-to-end learning from raw high dimensional input alone has led to deep reinforcement learning algorithms being deployed in a variety of ﬁelds.
Revised: The ability to perform end-to-end learning from raw high dimensional input alone has led to deep reinforcement learning algorithms being deployed in a variety of ﬁelds.

Original: Thus, understanding and improving the ability of deep reinforcement learning agents to generalize to unseen data distributions is of critical importance.
Revised: Thus, understanding and improving the ability of deep reinforcement learning agents to generalize to unseen data distributions is of critical importance.

Original: Much recent work has focused on assessing the generalization of deep reinforcement learning agents by introducing speciﬁcally crafted adversarial perturbations to their inputs.
Revised: Much recent work has focused on assessing the generalization of deep reinforcement learning agents by introducing speciﬁcally crafted adversarial perturbations to their inputs.

Original: In this paper, we propose another approach that we call daylight : a framework to assess the generalization skills of trained deep reinforcement learning agents.
Revised: In this paper, we propose another approach that we call daylight : a framework to assess the generalization skills of trained deep reinforcement learning agents.

Original: Rather than focusing on worst-case analysis of distribution shift, our approach is based on black-box perturbations that correspond to semantically meaningful changes to the environment or the agent’s visual observation system ranging from brightness to compression artifacts.
Revised: Rather than focusing on worst-case analysis of distribution shift, our approach is based on black-box perturbations that correspond to semantically meaningful changes to the environment or the agent’s visual observation system ranging from brightness to compression artifacts.

Original: We demonstrate that even the smallest changes in the environment cause the performance of the agents to degrade signiﬁcantly in various games from the Atari environment despite having orders of magnitude lower perceptual similarity distance compared to state-of-the-art adversarial attacks.
Revised: We demonstrate that even the smallest changes in the environment cause the performance of the agents to degrade signiﬁcantly in various games from the Atari environment despite having orders of magnitude lower perceptual similarity distance compared to state-of-the-art adversarial attacks.

Original: We show that our framework captures a diverse set of bands in the Fourier spectrum, giving a better overall understanding of the agent’s generalization capabilities.
Revised: We show that our framework captures a diverse set of bands in the Fourier spectrum, giving a better overall understanding of the agent’s generalization capabilities.

Original: We believe our work can be crucial towards building resilient and generalizable deep reinforcement learning agents.
Revised: We believe our work can be crucial towards building resilient and generalizable deep reinforcement learning agents.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Following the initial work of Mnih et al.
Revised: Following the initial work of Mnih et al.

Original: , the use of DNNs as function approximators in reinforcement learning has led to a dramatic increase in the capabilities of RL agents Schulman et al.
Revised: , the use of DNNs as function approximators in reinforcement learning has led to a dramatic increase in the capabilities of RL agents Schulman et al.

Original: (2017); Lillicrap et al.
Revised: (2017); Lillicrap et al.

Original: In particular, these developments allow for the direct learning of strong policies from raw, high-dimensional inputs such as visual observations.
Revised: In particular, these developments allow for the direct learning of strong policies from raw, high-dimensional inputs such as visual observations.

Original: With the successes of these new methods come new challenges regarding the robustness and generalization capabilities of deep
Revised: With the successes of these new methods come new challenges regarding the robustness and generalization capabilities of deep

Original: RL agents.
Revised: RL agents.

Original: One line of research has focused on the high sensitivity of deep neural networks to imperceptible, adversarial perturbations to visual inputs, ﬁrst in the setting of image classiﬁcation Szegedy et al.
Revised: One line of research has focused on the high sensitivity of deep neural networks to imperceptible, adversarial perturbations to visual inputs, ﬁrst in the setting of image classiﬁcation Szegedy et al.

Original: (2014); Goodfellow et al.
Revised: (2014); Goodfellow et al.

Original: (2015) and more recently for deep reinforcement learning Huang et al.
Revised: (2015) and more recently for deep reinforcement learning Huang et al.

Original: ; Kos & Song (2017).
Revised: ; Kos & Song (2017).

Original: Since one of the main reasons for the success and popularity of deep
Revised: Since one of the main reasons for the success and popularity of deep

Original: RL is the ability to learn directly from visual observations alone, this non-robustness to small adversarial perturbations is a serious concern Chokshi (2020); Vlasic & Boudette (2016); Kunkle (2018).
Revised: RL is the ability to learn directly from visual observations alone, this non-robustness to small adversarial perturbations is a serious concern Chokshi (2020); Vlasic & Boudette (2016); Kunkle (2018).

Original: However, existing adversarial formulations for deep reinforcement learning require high computational effort to produce the perturbations, knowledge of the network used to train the agent, knowledge of the environment, real-time access to and manipulation of the agent’s state observations.
Revised: However, existing adversarial formulations for deep reinforcement learning require high computational effort to produce the perturbations, knowledge of the network used to train the agent, knowledge of the environment, real-time access to and manipulation of the agent’s state observations.

Original: In this paper, we propose a more realistic scenario where we do not have access to any of the above, and the adversary essentially consists of realistic changes in the natural environment or in the agent’s observation system.
Revised: In this paper, we propose a more realistic scenario where we do not have access to any of the above, and the adversary essentially consists of realistic changes in the natural environment or in the agent’s observation system.

Original: For instance, if our deep reinforcement learning agent is operating a self-driving car one could plausibly expect changes in daylight levels, shifts in angle due to terrain, fog on the camera lens, or compression artifacts from the camera processor.
Revised: For instance, if our deep reinforcement learning agent is operating a self-driving car one could plausibly expect changes in daylight levels, shifts in angle due to terrain, fog on the camera lens, or compression artifacts from the camera processor.

Original: We believe that our proposed framework is semantically more meaningful than arbitrary (cid:96) p -norm bounded pixel perturbations.
Revised: We believe that our proposed framework is semantically more meaningful than arbitrary (cid:96) p -norm bounded pixel perturbations.

Original: Prior work on image classiﬁcation Dodge & Karam (2016) showed that image quality distortions can reduce the accuracy of DNN classﬁers.
Revised: Prior work on image classiﬁcation Dodge & Karam (2016) showed that image quality distortions can reduce the accuracy of DNN classﬁers.

Original: Moreover, recent work by Ford et al.
Revised: Moreover, recent work by Ford et al.

Original: (2019) showed that while adversarial training for image classiﬁers reduced their vulnerability towards perturbations corresponding to high frequency in the Fourier domain, it actually made the models more vulnerable to low frequency perturbations including fog and contrast changes.
Revised: (2019) showed that while adversarial training for image classiﬁers reduced their vulnerability towards perturbations corresponding to high frequency in the Fourier domain, it actually made the models more vulnerable to low frequency perturbations including fog and contrast changes.

Original: Therefore, it is important to investigate model robustness and generalization throughout different bands in the frequency domain.
Revised: Therefore, it is important to investigate model robustness and generalization throughout different bands in the frequency domain.

Original: We believe that being able to accurately assess the generalization capabilities of deep reinforcement learning agents is an initial step towards building robust and reliable agents.
Revised: We believe that being able to accurately assess the generalization capabilities of deep reinforcement learning agents is an initial step towards building robust and reliable agents.

Original: For these reasons, in this work we investigate the robustness of trained deep reinforcement learning agents and make the following contributions:
Revised: For these reasons, in this work we investigate the robustness of trained deep reinforcement learning agents and make the following contributions:

Original: • We propose a realistic threat model called daylight and a generalization framework for deep reinforcement learning agents that aims to assess the robustness of the agents to basic environmental and observational changes.
Revised: • We propose a realistic threat model called daylight and a generalization framework for deep reinforcement learning agents that aims to assess the robustness of the agents to basic environmental and observational changes.

Original: • We run multiple experiments in the Atari environment in various games to demonstrate the degradation in performance of deep reinforcement learning agents.
Revised: • We run multiple experiments in the Atari environment in various games to demonstrate the degradation in performance of deep reinforcement learning agents.

Original: • We compare our threat model with the state-of-the-art adversarial method based on (cid:96) p -norm changes, and we show that our daylight framework results in competitive, and almost always larger impact, with lower perceptual similarity distance.
Revised: • We compare our threat model with the state-of-the-art adversarial method based on (cid:96) p -norm changes, and we show that our daylight framework results in competitive, and almost always larger impact, with lower perceptual similarity distance.

Original: • We evaluate the daylight framework in the time domain and show that several works based on the timing perspective of adversarial formulations can be revisited within our daylight framework.
Revised: • We evaluate the daylight framework in the time domain and show that several works based on the timing perspective of adversarial formulations can be revisited within our daylight framework.

Original: Finally, we investigate the frequency domain of our framework and state-of-the-art targeted attacks.
Revised: Finally, we investigate the frequency domain of our framework and state-of-the-art targeted attacks.

Original: We show that our framework captures different bands of the frequency spectrum, thus yielding a better estimate of the model robustness.
Revised: We show that our framework captures different bands of the frequency spectrum, thus yielding a better estimate of the model robustness.

Original:  R ELATED W ORK 2. P RELEMINARIES
Revised:  R ELATED W ORK 2. P RELEMINARIES

Original: In this paper we consider Markov Decision Processes (MDPs) given by a tuple ( S , A , P, r, γ, s 0 ) .
Revised: In this paper we consider Markov Decision Processes (MDPs) given by a tuple ( S , A , P, r, γ, s 0 ) .

Original: The reinforcement learning agent interacts with the MDP by observing states s ∈ S , and then taking actions a ∈ A .
Revised: The reinforcement learning agent interacts with the MDP by observing states s ∈ S , and then taking actions a ∈ A .

Original: The probability of transitioning to state s (cid:48) when the agent takes action a in state s is determined by the transition probability P : S ×
Revised: The probability of transitioning to state s (cid:48) when the agent takes action a in state s is determined by the transition probability P : S ×

Original: A × S → R .
Revised: A × S → R .

Original: The reward received by the agent when taking action a in state s is given by the reward function r : S ×
Revised: The reward received by the agent when taking action a in state s is given by the reward function r : S ×

Original: A → R .
Revised: A → R .

Original: The goal of the agent is to learn a policy π θ : S ×
Revised: The goal of the agent is to learn a policy π θ : S ×

Original: A → R which takes an action a in state s that maximizes the cumulative discounted reward (cid:80) T − 1 t =0 γ t r ( s t , a t ) .
Revised: A → R which takes an action a in state s that maximizes the cumulative discounted reward (cid:80) T − 1 t =0 γ t r ( s t , a t ) .

Original: Here s 0 is the initial state of the agent, and γ is the discount factor.
Revised: Here s 0 is the initial state of the agent, and γ is the discount factor.

Original: For deep Q networks (DQN) the optimal policy is determined by learning the state-action value function Q ( s, a ) .
Revised: For deep Q networks (DQN) the optimal policy is determined by learning the state-action value function Q ( s, a ) .

Original: For a state s we use F ( s ) to denote the 2D discrete Fourier transform.
Revised: For a state s we use F ( s ) to denote the 2D discrete Fourier transform.

Original: 2. C RAFTING A DVERSARIAL P ERTURBATIONS
Revised: 2. C RAFTING A DVERSARIAL P ERTURBATIONS

Original: Szegedy et al.
Revised: Szegedy et al.

Original: (2014) proposed to minimize the distance between the original image and adversarially produced image to create adversarial perturbations.
Revised: (2014) proposed to minimize the distance between the original image and adversarially produced image to create adversarial perturbations.

Original: The authors used box-constrained L-BFGS to solve this optimization problem.
Revised: The authors used box-constrained L-BFGS to solve this optimization problem.

Original: Goodfellow et al.
Revised: Goodfellow et al.

Original: (2015) introduced the fast gradient method (FGM)
Revised: (2015) introduced the fast gradient method (FGM)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: for crafting adversarial examples in image classiﬁcation by taking the gradient of the cost function
Revised: for crafting adversarial examples in image classiﬁcation by taking the gradient of the cost function

Original: J ( x, y ) used to train the neural network in the direction of the input, where x is the input, y is the output label, and J ( x, y ) is the cost function for image classiﬁcation.
Revised: J ( x, y ) used to train the neural network in the direction of the input, where x is the input, y is the output label, and J ( x, y ) is the cost function for image classiﬁcation.

Original: Carlini & Wagner (2017) introduced targeted attacks in the image classiﬁcation domain based on distance minimization between the adversarial image and the original image while targeting a particular label.
Revised: Carlini & Wagner (2017) introduced targeted attacks in the image classiﬁcation domain based on distance minimization between the adversarial image and the original image while targeting a particular label.

Original: In the deep reinforcement learning domain the Carlini & Wagner (2017) formulation is min s adv ∈ S (cid:107) s adv − s (cid:107) p subject to a ∗ ( s ) (cid:54)
Revised: In the deep reinforcement learning domain the Carlini & Wagner (2017) formulation is min s adv ∈ S (cid:107) s adv − s (cid:107) p subject to a ∗ ( s ) (cid:54)

Original: = a ∗ ( s adv ) , where s is the unperturbed input, s adv is the adversarially perturbed input, a ∗ ( s ) is the action taken in the unperturbed state, and a ∗ ( s adv ) is the action taken in the adversarial state.
Revised: = a ∗ ( s adv ) , where s is the unperturbed input, s adv is the adversarially perturbed input, a ∗ ( s ) is the action taken in the unperturbed state, and a ∗ ( s adv ) is the action taken in the adversarial state.

Original: This formulation attempts to minimize the distance to the original state, constrained to states leading to sub-optimal actions as determined by the Q -network.
Revised: This formulation attempts to minimize the distance to the original state, constrained to states leading to sub-optimal actions as determined by the Q -network.

Original: In contrast to adversarial attacks, in our proposed threat model we will not need any information on the cost function used to train the network, Q -network of the trained agent or the visited states themselves.
Revised: In contrast to adversarial attacks, in our proposed threat model we will not need any information on the cost function used to train the network, Q -network of the trained agent or the visited states themselves.

Original: 2. A DVERSARIAL A PPROACH IN D EEP R EINFORCEMENT L EARNING
Revised: 2. A DVERSARIAL A PPROACH IN D EEP R EINFORCEMENT L EARNING

Original: The ﬁrst adversarial attacks on deep reinforcement learning introduced by Huang et al.
Revised: The ﬁrst adversarial attacks on deep reinforcement learning introduced by Huang et al.

Original: (2017) and
Revised: (2017) and

Original: Kos & Song (2017) adapted FGSM from image classiﬁcation to the deep RL setting.
Revised: Kos & Song (2017) adapted FGSM from image classiﬁcation to the deep RL setting.

Original: Subsequently,
Revised: Subsequently,

Original: Mandlekar et al.
Revised: Mandlekar et al.

Original: (2017) used FGSM perturbations for adversarial training of deep RL agents.
Revised: (2017) used FGSM perturbations for adversarial training of deep RL agents.

Original: Pinto et al.
Revised: Pinto et al.

Original: ; Gleave et al.
Revised: ; Gleave et al.

Original: (2020) focused on modeling the interaction between adversary and the agent, while Lin et al.
Revised: (2020) focused on modeling the interaction between adversary and the agent, while Lin et al.

Original: (2017); Sun et al.
Revised: (2017); Sun et al.

Original: (2020) focused on strategically timing when (i.e. in which state) to attack an agent using perturbations computed with the Carlini & Wagner (2017) adversarial formulation.
Revised: (2020) focused on strategically timing when (i.e. in which state) to attack an agent using perturbations computed with the Carlini & Wagner (2017) adversarial formulation.

Original: 2. P ERCEPTUAL S IMILARITY
Revised: 2. P ERCEPTUAL S IMILARITY

Original: Zhang et al.
Revised: Zhang et al.

Original: (2018) found that internal activations of networks trained for high-level tasks correspond to human perceptual judgements across different network architectures Iandola et al.
Revised: (2018) found that internal activations of networks trained for high-level tasks correspond to human perceptual judgements across different network architectures Iandola et al.

Original: Krizhevsky et al.
Revised: Krizhevsky et al.

Original: (2012), Simonyan & Zisserman (2015) without calibration.
Revised: (2012), Simonyan & Zisserman (2015) without calibration.

Original: Furthermore, the authors propose a method to measure the perceptual distance between two images with the Learned
Revised: Furthermore, the authors propose a method to measure the perceptual distance between two images with the Learned

Original: Perceptual Image Patch Similarity (LPIPS) metric.
Revised: Perceptual Image Patch Similarity (LPIPS) metric.

Original: We compare the distance between adversarial states s adv and the original states s with the LPIPS metric.
Revised: We compare the distance between adversarial states s adv and the original states s with the LPIPS metric.

Original: We refer to the LPIPS metric as P similarity throughout the paper.
Revised: We refer to the LPIPS metric as P similarity throughout the paper.

Original: P similarity ( s, s adv ) returns the distance between s and s adv based on network activations.
Revised: P similarity ( s, s adv ) returns the distance between s and s adv based on network activations.

Original: Zhang et al.
Revised: Zhang et al.

Original: (2018) show that P similarity results in a reliable approximation of human perception.
Revised: (2018) show that P similarity results in a reliable approximation of human perception.

Original: 2. I MPACT
Revised: 2. I MPACT

Original: We deﬁne the normalized impact of an adversary on the agent as,
Revised: We deﬁne the normalized impact of an adversary on the agent as,

Original: [Equation]
Revised: [Equation]

Original: Score max is the score at the end of the episode achieved by the agent who takes the action that maximizes its Q ( s, a ) function in every state visited, and Score min is the score at the end of the episode achieved by the agent who takes the action that minimizes its Q ( s, a ) function in every state visited.
Revised: Score max is the score at the end of the episode achieved by the agent who takes the action that maximizes its Q ( s, a ) function in every state visited, and Score min is the score at the end of the episode achieved by the agent who takes the action that minimizes its Q ( s, a ) function in every state visited.

Original: Score adv is the score at the end of the episode achieved by the agent who takes the action that maximizes Q ( s adv , a ) under the inﬂuence of the adversary in every state visited.
Revised: Score adv is the score at the end of the episode achieved by the agent who takes the action that maximizes Q ( s adv , a ) under the inﬂuence of the adversary in every state visited.

Original: We report the results in a normalized scale, because we observed the agent can collect stochastic rewards even though it chooses the action that minimizes its Q ( s, a ) function in every state visited.
Revised: We report the results in a normalized scale, because we observed the agent can collect stochastic rewards even though it chooses the action that minimizes its Q ( s, a ) function in every state visited.

Original:  D AYLIGHT : A G ENERALIZATION T ESTING F RAMEWORK
Revised:  D AYLIGHT : A G ENERALIZATION T ESTING F RAMEWORK

Original: In our generalization framework 1 we propose a baseline to test the robustness of trained deep reinforcement learning agents with respect to several realistic failures of the agent’s observations.
Revised: In our generalization framework 1 we propose a baseline to test the robustness of trained deep reinforcement learning agents with respect to several realistic failures of the agent’s observations.

Original: This is in contrast to prior work that focused on the presence of a strong adversary with prior access to training details of the agent’s neural network, real time access to agent’s perception system, and highly computationally demanding adversarial formulations used to compute simultaneous perturbations.
Revised: This is in contrast to prior work that focused on the presence of a strong adversary with prior access to training details of the agent’s neural network, real time access to agent’s perception system, and highly computationally demanding adversarial formulations used to compute simultaneous perturbations.

Original: In our model we consider the environment itself as an adversary and we examine several environmental changes such as: changes in the brightness of the environment, blurring of the observation, rotation of the observation, perspective transformation, shifting, and compression artifacts.
Revised: In our model we consider the environment itself as an adversary and we examine several environmental changes such as: changes in the brightness of the environment, blurring of the observation, rotation of the observation, perspective transformation, shifting, and compression artifacts.

Original: These changes from our model can be easily linked to naturally occurring changes in the environment.
Revised: These changes from our model can be easily linked to naturally occurring changes in the environment.

Original: For instance, for a self driving car a brightness change can be linked to the time of day, or the appearance of reﬂective objects or shadows.
Revised: For instance, for a self driving car a brightness change can be linked to the time of day, or the appearance of reﬂective objects or shadows.

Original: Rotation, perspective transformation, and shifting can be linked to driving on a road with varied terrain.
Revised: Rotation, perspective transformation, and shifting can be linked to driving on a road with varied terrain.

Original: Blurring can be linked to a rainy day, foggy weather or a fogged up camera lens utilized by the agent.
Revised: Blurring can be linked to a rainy day, foggy weather or a fogged up camera lens utilized by the agent.

Original: In the remainder of this section we compare the impact values and perceptual similarities of the daylight framework with the state-of-the-art targeted adversarial attack proposed by Carlini & Wagner (2017).
Revised: In the remainder of this section we compare the impact values and perceptual similarities of the daylight framework with the state-of-the-art targeted adversarial attack proposed by Carlini & Wagner (2017).

Original: 3. B RIGHTNESS AND C ONTRAST
Revised: 3. B RIGHTNESS AND C ONTRAST

Original: The ﬁrst component of our framework focusing on testing the trained agents in different brightness and contrast levels using a linear brightness and contrast transformation,
Revised: The ﬁrst component of our framework focusing on testing the trained agents in different brightness and contrast levels using a linear brightness and contrast transformation,

Original: [Equation]
Revised: [Equation]

Original: where s ( i, j ) is the ij th pixel of state s , and α and β are the linear brightness parameters.
Revised: where s ( i, j ) is the ij th pixel of state s , and α and β are the linear brightness parameters.

Original: In Tablewe show the impacts and perceptual similarity distances with corresponding α, β values.
Revised: In Tablewe show the impacts and perceptual similarity distances with corresponding α, β values.

Original: In all of the games except BankHeist brightness and contrast change results in higher impact.
Revised: In all of the games except BankHeist brightness and contrast change results in higher impact.

Original: The perceptual similarity distance of brightness and contrast is lower in every game.
Revised: The perceptual similarity distance of brightness and contrast is lower in every game.

Original: In Figure 3 we show the corresponding states s and s adv for all of the daylight framework.
Revised: In Figure 3 we show the corresponding states s and s adv for all of the daylight framework.

Original: [Table]
Revised: [Table]

Original: 3. B LURRING
Revised: 3. B LURRING

Original: The second component in our daylight framework is blurring.
Revised: The second component in our daylight framework is blurring.

Original: Median bluring is a nonlinear noise removal technique that replaces the original pixel value with the median pixel value of its neighbouring pixels.
Revised: Median bluring is a nonlinear noise removal technique that replaces the original pixel value with the median pixel value of its neighbouring pixels.

Original: A kernel size k means that the median is computed over a k × k neighborhood of the original pixel.
Revised: A kernel size k means that the median is computed over a k × k neighborhood of the original pixel.

Original: In Table 2 we show the impact values and perceptual similarity distance with corresponding kernel size.
Revised: In Table 2 we show the impact values and perceptual similarity distance with corresponding kernel size.

Original: Only in BankHeist do we observe that the impacts and perceptual similarity distance are comparable.
Revised: Only in BankHeist do we observe that the impacts and perceptual similarity distance are comparable.

Original: For the rest of the games impact is higher and perceptual similarity distance is lower for blurring
Revised: For the rest of the games impact is higher and perceptual similarity distance is lower for blurring

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 3. R OTATION
Revised: 3. R OTATION

Original: The next component in our daylight framework is rotation.
Revised: The next component in our daylight framework is rotation.

Original: In Table 3 we show impact values and perceptual similarity distance with corresponding rotation angle.
Revised: In Table 3 we show impact values and perceptual similarity distance with corresponding rotation angle.

Original: In all of the games except Pong rotation results in higher impact and orders of magnitude lower perceptual similarity distance.
Revised: In all of the games except Pong rotation results in higher impact and orders of magnitude lower perceptual similarity distance.

Original: In
Revised: In

Original: Pong the impact is comparable and the perceptual similarity distance is lower by a factor of 2 .
Revised: Pong the impact is comparable and the perceptual similarity distance is lower by a factor of 2 .

Original: [Table]
Revised: [Table]

Original: 3. S HIFTING
Revised: 3. S HIFTING

Original: The next component in our daylight framework is shifting.
Revised: The next component in our daylight framework is shifting.

Original: Shifting an image moves the elements of the image matrix along any dimension by any number of elements.
Revised: Shifting an image moves the elements of the image matrix along any dimension by any number of elements.

Original: In this subsection we will shift the inputs in the x or y direction with as few pixels shifted as possible.
Revised: In this subsection we will shift the inputs in the x or y direction with as few pixels shifted as possible.

Original: We use [ t i , t j ] to denote the distance shifted, where t i is in the direction of x and t j is in the direction of y .
Revised: We use [ t i , t j ] to denote the distance shifted, where t i is in the direction of x and t j is in the direction of y .

Original: In Table 4 we show the impact values and perceptual similarity distances for both Carlini & Wagner (2017) and shifting.
Revised: In Table 4 we show the impact values and perceptual similarity distances for both Carlini & Wagner (2017) and shifting.

Original: For all of the games shifting yields higher impact and lower perceptual similarity distance
Revised: For all of the games shifting yields higher impact and lower perceptual similarity distance

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 3. C OMPRESSION A RTIFACTS
Revised: 3. C OMPRESSION A RTIFACTS

Original: In this section we look at jpeg compression artifacts caused by the discrete cosine transform (DCT) resulting in the loss of high frequency components (ringing and blocking).
Revised: In this section we look at jpeg compression artifacts caused by the discrete cosine transform (DCT) resulting in the loss of high frequency components (ringing and blocking).

Original: In Table 5 we show the impact values and perceptual similarities of Carlini & Wagner (2017) and compression artifacts (CA)
Revised: In Table 5 we show the impact values and perceptual similarities of Carlini & Wagner (2017) and compression artifacts (CA)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Only in Pong and Riverraid do we observe a lower impact than Carlini & Wagner (2017) while the perceptual similarity distance was orders of magnitude smaller for compression artifacts.
Revised: Only in Pong and Riverraid do we observe a lower impact than Carlini & Wagner (2017) while the perceptual similarity distance was orders of magnitude smaller for compression artifacts.

Original: In
Revised: In

Original: BankHeist the impact is comparable, and in the rest of the games compression artifacts result in higher impact and lower perceptual similarity distance compared to Carlini & Wagner (2017).
Revised: BankHeist the impact is comparable, and in the rest of the games compression artifacts result in higher impact and lower perceptual similarity distance compared to Carlini & Wagner (2017).

Original: 3. P ERSPECTIVE T RANSFORMATION
Revised: 3. P ERSPECTIVE T RANSFORMATION

Original: The ﬁnal component of our daylight framework is perspective transformation.
Revised: The ﬁnal component of our daylight framework is perspective transformation.

Original: Given four points in the plane deﬁning a convex quadrangle, there is a unique perspective transformation mapping the corners of the square to these four points (see Equation 5 and Equation 6).
Revised: Given four points in the plane deﬁning a convex quadrangle, there is a unique perspective transformation mapping the corners of the square to these four points (see Equation 5 and Equation 6).

Original: We deﬁne the norm of a perspective transformation as the maximum distance that one of the corners of the square moves under this mapping.
Revised: We deﬁne the norm of a perspective transformation as the maximum distance that one of the corners of the square moves under this mapping.

Original: In Table 6 we show impact values and perceptual similarity distance with respect to perspective norms.
Revised: In Table 6 we show impact values and perceptual similarity distance with respect to perspective norms.

Original: For all the games we observe perspective transformation yields higher impact and lower perceptual similarity distance.
Revised: For all the games we observe perspective transformation yields higher impact and lower perceptual similarity distance.

Original: [Table]
Revised: [Table]

Original:  F OURIER D OMAIN
Revised:  F OURIER D OMAIN

Original: Ford et al.
Revised: Ford et al.

Original: (2019) showed DNN models are robust to certain bands of perturbations in the frequency domain.
Revised: (2019) showed DNN models are robust to certain bands of perturbations in the frequency domain.

Original: Furthermore, they showed that adversarial training shifts the vulnerability from high frequency noise towards low frequency noise.
Revised: Furthermore, they showed that adversarial training shifts the vulnerability from high frequency noise towards low frequency noise.

Original: Moreover, Yin et al.
Revised: Moreover, Yin et al.

Original: (2019) claim that a framework that aims to measure robustness and generalization needs to ﬁrmly encapsulate different directions of the spectrum in the frequency domain.
Revised: (2019) claim that a framework that aims to measure robustness and generalization needs to ﬁrmly encapsulate different directions of the spectrum in the frequency domain.

Original: In this section we show that the daylight framework indeed captures a broader set of directions in the frequency domain
Revised: In this section we show that the daylight framework indeed captures a broader set of directions in the frequency domain

Original: [Table]
Revised: [Table]

Original: In Figure 2 we show the Fourier spectrum of the original state s and the perturbed states s adv from the daylight framework and Carlini & Wagner (2017).
Revised: In Figure 2 we show the Fourier spectrum of the original state s and the perturbed states s adv from the daylight framework and Carlini & Wagner (2017).

Original: In Figure 1 we show the power spectral density of the original state compared to several perturbations from our proposed daylight framework and
Revised: In Figure 1 we show the power spectral density of the original state compared to several perturbations from our proposed daylight framework and

Original: Carlini & Wagner (2017).
Revised: Carlini & Wagner (2017).

Original: In Figure 1 we observe that while Carlini & Wagner (2017) increases the higher frequencies, compression artifacts decrease the magnitude of the higher frequency band.
Revised: In Figure 1 we observe that while Carlini & Wagner (2017) increases the higher frequencies, compression artifacts decrease the magnitude of the higher frequency band.

Original: On the other hand, brightness and contrast decreases the magnitude of the low frequency band, and shifting increases the midband.
Revised: On the other hand, brightness and contrast decreases the magnitude of the low frequency band, and shifting increases the midband.

Original: Blurring decreases the midband and high frequencies together, and perspective transformation decreases the low frequecies and high frequencies while increasing the midband.
Revised: Blurring decreases the midband and high frequencies together, and perspective transformation decreases the low frequecies and high frequencies while increasing the midband.

Original: We believe capturing the susceptibilities towards perturbations in different bands of the frequency domain is a key step towards building robust agents.
Revised: We believe capturing the susceptibilities towards perturbations in different bands of the frequency domain is a key step towards building robust agents.

Original:  T IMING P ERSPECTIVE
Revised:  T IMING P ERSPECTIVE

Original: In the previous sections we tested our agents generalization capabilities in modiﬁed environments with our initial thread model.
Revised: In the previous sections we tested our agents generalization capabilities in modiﬁed environments with our initial thread model.

Original: This modiﬁcation to the environment was applied to every state that the agent visited for both Carlini & Wagner (2017) and our daylight framework.
Revised: This modiﬁcation to the environment was applied to every state that the agent visited for both Carlini & Wagner (2017) and our daylight framework.

Original: In this section we will examine the effect of both adversarial models when the perturbations are applied to only a small fraction of states.
Revised: In this section we will examine the effect of both adversarial models when the perturbations are applied to only a small fraction of states.

Original: For this purpose we introduce the adversarial states s adv in randomly sampled states where the observation s adv is observed by the agent with probability p , and the original states s is observed observed by the agent with probability 1 − p .
Revised: For this purpose we introduce the adversarial states s adv in randomly sampled states where the observation s adv is observed by the agent with probability p , and the original states s is observed observed by the agent with probability 1 − p .

Original: We use n s adv to denote the number of states where the agent observed s adv instead of the original state s , and we use n s to denote the total number of states visited by the agent in the given episode.
Revised: We use n s adv to denote the number of states where the agent observed s adv instead of the original state s , and we use n s to denote the total number of states visited by the agent in the given episode.

Original: We use e adv to denote the fraction n s adv /n s of adversarial perturbations per episode.
Revised: We use e adv to denote the fraction n s adv /n s of adversarial perturbations per episode.

Original: [Table]
Revised: [Table]

Original: In Table 7 we show the attack impacts of Carlini & Wagner (2017) and daylight framework with corresponding adversarial observation probability p averaged over 10 random episodes.
Revised: In Table 7 we show the attack impacts of Carlini & Wagner (2017) and daylight framework with corresponding adversarial observation probability p averaged over 10 random episodes.

Original: See Appendix
Revised: See Appendix

Original: A.4 for more details.
Revised: A.4 for more details.

Original: Even for low p values our proposed daylight framework obtains higher impact.
Revised: Even for low p values our proposed daylight framework obtains higher impact.

Original: Thus, to capture a broader view on the robustness of the agent, the prior work on the timing perspective by Sun et al.
Revised: Thus, to capture a broader view on the robustness of the agent, the prior work on the timing perspective by Sun et al.

Original: Lin et al.
Revised: Lin et al.

Original: based on worst-case distributional shift, can be revisited with our daylight framework.
Revised: based on worst-case distributional shift, can be revisited with our daylight framework.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In our experiments we trained our agents with DDQN Wang et al.
Revised: In our experiments we trained our agents with DDQN Wang et al.

Original: (2016) in the OpenAI
Revised: (2016) in the OpenAI

Original: Gym
Revised: Gym

Original: Brockman et al.
Revised: Brockman et al.

Original: Atari environment Bellemare et al.
Revised: Atari environment Bellemare et al.

Original: We test trained agents from several Atari environments averaged over 10 episodes.
Revised: We test trained agents from several Atari environments averaged over 10 episodes.

Original: In Figure 3 we show the original states and the environmental modiﬁcations.
Revised: In Figure 3 we show the original states and the environmental modiﬁcations.

Original: Interestingly, we found that a majority of the games have high robustness against rotation.
Revised: Interestingly, we found that a majority of the games have high robustness against rotation.

Original: On the other hand, shifting and perspective transformation can reach a higher impact level than the state-of-the-art targeted attack while not being recognizable by human perception.
Revised: On the other hand, shifting and perspective transformation can reach a higher impact level than the state-of-the-art targeted attack while not being recognizable by human perception.

Original: We observed that in some games, such as Pong and Riverraid, brightness and contrast requires radical changes to cause the agent to fail, while for others the change required is imperceptible.
Revised: We observed that in some games, such as Pong and Riverraid, brightness and contrast requires radical changes to cause the agent to fail, while for others the change required is imperceptible.

Original: Another thing we observed is that for games like Pong, which is relatively trivial compared to other games in the Atari environment, the threshold values for the environmental modiﬁcation are higher.
Revised: Another thing we observed is that for games like Pong, which is relatively trivial compared to other games in the Atari environment, the threshold values for the environmental modiﬁcation are higher.

Original: When the complexity in the game increases the environment modiﬁcation thresholds decrease drastically.
Revised: When the complexity in the game increases the environment modiﬁcation thresholds decrease drastically.

Original: We think that this issue could become more important as deep reinforcement learning agents are deployed in more complex and realistic scenarios.
Revised: We think that this issue could become more important as deep reinforcement learning agents are deployed in more complex and realistic scenarios.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper we studied a realistic threat model based on basic environmental changes and proposed a framework called daylight to asses the generalization capabilities of deep reinforcement learning agents.
Revised: In this paper we studied a realistic threat model based on basic environmental changes and proposed a framework called daylight to asses the generalization capabilities of deep reinforcement learning agents.

Original: We compared our daylight framework with the state-of-the-art adversarial attacks in the Atari environment.
Revised: We compared our daylight framework with the state-of-the-art adversarial attacks in the Atari environment.

Original: We demonstrated that our framework achieves higher impact on agent performance with lower perceptual similarity distance without having access to agents training details, real time access to agents memory and perception system, and computationally demanding adversarial formulations to compute simultaneous perturbations.
Revised: We demonstrated that our framework achieves higher impact on agent performance with lower perceptual similarity distance without having access to agents training details, real time access to agents memory and perception system, and computationally demanding adversarial formulations to compute simultaneous perturbations.

Original: We investigated perturbations in the time domain and showed that the studies based on imperceptible perturbations can be revisited within the daylight framework.
Revised: We investigated perturbations in the time domain and showed that the studies based on imperceptible perturbations can be revisited within the daylight framework.

Original: Finally, we show that each component of our framework contains distinct bands in the frequency domain, resulting in a better estimate of the generalization capabilities of trained agents.
Revised: Finally, we show that each component of our framework contains distinct bands in the frequency domain, resulting in a better estimate of the generalization capabilities of trained agents.

Original: We believe our framework can be instrumental towards generalization and robustiﬁcation of deep reinforcement learning algorithms.
Revised: We believe our framework can be instrumental towards generalization and robustiﬁcation of deep reinforcement learning algorithms.

Original: A A PPENDIX A. R AW S CORE R ESULTS
Revised: A A PPENDIX A. R AW S CORE R ESULTS

Original: In this section we provide the raw scores of the agents under the observation modiﬁcations from both the state-of-the-art adversarial attack and the Daylight framework with components: Brightness &
Revised: In this section we provide the raw scores of the agents under the observation modiﬁcations from both the state-of-the-art adversarial attack and the Daylight framework with components: Brightness &

Original: Contrast, Blurring, Rotation, Shifting, Compression Artifacts, and Perspective Transform .
Revised: Contrast, Blurring, Rotation, Shifting, Compression Artifacts, and Perspective Transform .

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: W) and rotation with corresponding perceptual similarity P similarity , and the rotation angle (RD denotes rotation degree)
Revised: W) and rotation with corresponding perceptual similarity P similarity , and the rotation angle (RD denotes rotation degree)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A. I MPACTS WITH F IXED S CORES
Revised: A. I MPACTS WITH F IXED S CORES

Original: For the scope of the paper we used the Impact deﬁnition in Equation 2 when we compare our proposed
Revised: For the scope of the paper we used the Impact deﬁnition in Equation 2 when we compare our proposed

Original: Daylight framework to the state-of-the-art targeted attacks.
Revised: Daylight framework to the state-of-the-art targeted attacks.

Original: For more generalized comparison between different algorithms and different games one can use Score clean as the score of the agent without any modiﬁcation to agent’s observations system at the end of the episode and Score ﬁxedmin as the ﬁxed minimum score for a given game.
Revised: For more generalized comparison between different algorithms and different games one can use Score clean as the score of the agent without any modiﬁcation to agent’s observations system at the end of the episode and Score ﬁxedmin as the ﬁxed minimum score for a given game.

Original: Thus, we deﬁne the generalized impact as,
Revised: Thus, we deﬁne the generalized impact as,

Original: [Equation]
Revised: [Equation]

Original: From Table 14 through Table 19 we
Revised: From Table 14 through Table 19 we

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  P OLICY G RADIENT M ETHODS UNDER D AYLIGHT F RAMEWORK
Revised: A. P OLICY G RADIENT M ETHODS UNDER D AYLIGHT F RAMEWORK

Original: In this section we investigatethe policy gradient methods.
Revised: In this section we investigate policy gradient methods.

Original: In particular, Table 20 shows the raw scores of the agent trained with A3C under  Daylight framework 
Revised: In particular, Table 20 shows the raw scores, and generalized impacts I general of the agent trained with A3C under the Daylight framework with following observation modiﬁcations: brightness & constrast, blurring, rotation, shifting, compression artifacts and perspective transform. In Table 20the exact same hyperparameters have been used as stated in Table 1 through Table 6 for the Daylight framework. Note that Daylight hyperparameters refers for brightnes and contrast to [ α, β ] , for blurring to the kernel size, for rotation to rotation degree, for shifting to [ t i , t j ]

Original: [Table]
Revised: [Table]

Revised: and for perspective transformation to perspective norm.

Revised: Shifting and compression artifacts have nearly maximal impact on the performance of the agent trained with A3C, while the other perturbations all have impact at least 0.9.

Revised: Note that we did not change the Daylight hyperparameters for a direct comparison between the A3C agent and the DDQN agent.

Revised: Therefore, although impact is slightly lower for brightness & contrast for A3C than for DDQN, it is possible that choosing different values of α and β while minimizing the perceptual similarity P similarity can still result in a higher impact for an agent trained with A3C.

Original: [Table]
Revised: [Table]

Original: Pong - -20.

Original: A. T IMING P ERSPECTIVE
Revised: A. T IMING P ERSPECTIVE

Original: Note that the fraction e adv can differ slightly from p due to random ﬂuctuations, therefore we also report these values in Table 7.
Revised: Note that the fraction e adv can differ slightly from p due to random ﬂuctuations, therefore we also report these values in Table 7.

Original: Note that e adv varies between games.
Revised: Note that e adv varies between games.

Original: This was done because each game has a different minimum threshold for e adv to achieve stable impact across episodes
Revised: This was done because each game has a different minimum threshold for e adv to achieve stable impact across episodes.

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: A. F ORMULAS FOR P ERSPECTIVE T RANSFORMATION
Revised: A. F ORMULAS FOR P ERSPECTIVE T RANSFORMATION

Original: [Equation]
Revised: [Equation]

Revised: [Equation]