Original: I LLUSORY ADVERSARIAL ATTACKS ON SEQUENTIAL DECISION - MAKERS AND COUNTERMEASURES
Revised: I LLUSORY ADVERSARIAL ATTACKS ON SEQUENTIAL DECISION - MAKERS AND COUNTERMEASURES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Autonomous decision-making agents deployed in the real world need to be robust against possible adversarial attacks on sensory inputs.
Revised: Autonomous decision-making agents deployed in the real world need to be robust against possible adversarial attacks on sensory inputs.

Original: Existing work on adversarial attacks focuses on the notion of perceptual invariance popular in computer vision.
Revised: Existing work on adversarial attacks focuses on the notion of perceptual invariance popular in computer vision.

Original: We observe that such attacks can often be detected by victim agents, since they result in action-observation sequences that are not consistent with the dynamics of the environment.
Revised: We observe that such attacks can often be detected by victim agents, since they result in action-observation sequences that are not consistent with the dynamics of the environment.

Original: Furthermore, real-world agents, such as physical robots, commonly operate under human supervisors who are not susceptible to such attacks.
Revised: Furthermore, real-world agents, such as physical robots, commonly operate under human supervisors who are not susceptible to such attacks.

Original: We propose to instead focus on attacks that are statistically undetectable .
Revised: We propose to instead focus on attacks that are statistically undetectable .

Original: Specifically, we propose illusory attacks, a novel class of adversarial attack that is consistent with the environment dynamics.
Revised: Specifically, we propose illusory attacks, a novel class of adversarial attack that is consistent with the environment dynamics.

Original: We introduce a novel algorithm that can learn illusory attacks end-to-end.
Revised: We introduce a novel algorithm that can learn illusory attacks end-to-end.

Original: We empirically verify that our algorithm generates attacks that, in contrast to current methods, are undetectable to both AI agents with an environment dynamics model, as well as to humans.
Revised: We empirically verify that our algorithm generates attacks that, in contrast to current methods, are undetectable to both AI agents with an environment dynamics model, as well as to humans.

Original: Furthermore, we show that existing robustification approaches are relatively ineffective against illusory attacks.
Revised: Furthermore, we show that existing robustification approaches are relatively ineffective against illusory attacks.

Original: Our findings highlight the need to ensure that real-world AI, and human-AI, systems are designed to make it difficult to corrupt sensory observations in ways that are consistent with the environment dynamics
Revised: Our findings highlight the need to ensure that real-world AI, and human-AI, systems are designed to make it difficult to corrupt sensory observations in ways that are consistent with the environment dynamics

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep reinforcement learning algorithms (Mnih et al., 2015; Schulman et al., 2017; Haarnoja et al., 2018; Salimans et al., 2017, DQN, PPO, SAC, ES) have found applications across a number of sequential decision-making problems, ranging from simulated and real-world robotics (Todorov et al., 2012; Andrychowicz et al., 2020) to arcade games (Mnih et al., 2015).
Revised: Deep reinforcement learning algorithms (Mnih et al., 2015; Schulman et al., 2017; Haarnoja et al., 2018; Salimans et al., 2017, DQN, PPO, SAC, ES) have found applications across a number of sequential decision-making problems, ranging from simulated and real-world robotics (Todorov et al., 2012; Andrychowicz et al., 2020) to arcade games (Mnih et al., 2015).

Original: It has recently been found, however, that deep neural network control policies conditioning on high-dimensional sensory input are prone to adversarial attacks on the input observations, which poses threats to security and safety-critical applications (Kos & Song, 2017; Huang et al., 2017) and thus motivates research into robust learning algorithms (Zhang et al., 2020).
Revised: It has recently been found, however, that deep neural network control policies conditioning on high-dimensional sensory input are prone to adversarial attacks on the input observations, which poses threats to security and safety-critical applications (Kos & Song, 2017; Huang et al., 2017) and thus motivates research into robust learning algorithms (Zhang et al., 2020).

Original: Existing frameworks of attacks on sequential decision-makers are largely inspired by pioneering work on perceptually invariant attacks in supervised computer vision settings (Ilahi et al., 2021).
Revised: Existing frameworks of attacks on sequential decision-makers are largely inspired by pioneering work on perceptually invariant attacks in supervised computer vision settings (Ilahi et al., 2021).

Original: Unlike supervised settings, however, sequential decision-making settings involve temporally-extended environment interactions which give rise to temporally-correlated sequences of observations.
Revised: Unlike supervised settings, however, sequential decision-making settings involve temporally-extended environment interactions which give rise to temporally-correlated sequences of observations.

Original: In this paper, we argue that their failure to take temporal consistency considerations into account renders existing observation-space adversarial attacks ineffective in many settings of practical interest.
Revised: In this paper, we argue that their failure to take temporal consistency considerations into account renders existing observation-space adversarial attacks ineffective in many settings of practical interest.

Original: AI agents often have access to an approximate or exact world model (Sutton, 2022; Ha & Schmidhuber, 2018).
Revised: AI agents often have access to an approximate or exact world model (Sutton, 2022; Ha & Schmidhuber, 2018).

Original: In addition, humans have the ability to perform “intuitive physics” (Hamrick et al., 2016), using robust but qualitative internal models of the world (Battaglia et al., 2013).
Revised: In addition, humans have the ability to perform “intuitive physics” (Hamrick et al., 2016), using robust but qualitative internal models of the world (Battaglia et al., 2013).

Original: This makes it possible to use one’s understanding of the world to detect a large range of existing adversarial attacks, by spotting inconsistencies in observation sequences.
Revised: This makes it possible to use one’s understanding of the world to detect a large range of existing adversarial attacks, by spotting inconsistencies in observation sequences.

Original: Existing observation-space adversarial attacks (Ilahi et al., 2021; Chen et al., 2019; Qiaoben et al., 2021; Sun et al., 2020) ignore these facts.
Revised: Existing observation-space adversarial attacks (Ilahi et al., 2021; Chen et al., 2019; Qiaoben et al., 2021; Sun et al., 2020) ignore these facts.

Original: As a result, these attacks produce observation trajectories that are inconsistent with the dynamics of the unattacked environment.
Revised: As a result, these attacks produce observation trajectories that are inconsistent with the dynamics of the unattacked environment.

Original: The consequences of this are twofold.
Revised: The consequences of this are twofold.

Original: First, state-of-the-art adversarial attacks can be trivially detected by victim agents with access to even low-accuracy world models, i.e., models of the environment dynamics.
Revised: First, state-of-the-art adversarial attacks can be trivially detected by victim agents with access to even low-accuracy world models, i.e., models of the environment dynamics.

Original: Second, in cases where AI agents are supervised by humans, humans may also can
Revised: Second, in cases where AI agents are supervised by humans, humans may also can

Original: [Figure]
Revised: [Figure]

Original: replaces up
Revised: replaces up

Original: However be able to detect these adversarial attacks.
Revised: However be able to detect these adversarial attacks.

Original: In other security contexts such as computer networks, real-world adversarial attacks typically attempt to evade detection in an effort to avoid triggering security escalations, and this must be taken into account by the defender.
Revised: In other security contexts such as computer networks, real-world adversarial attacks typically attempt to evade detection in an effort to avoid triggering security escalations, and this must be taken into account by the defender.

Original: While this insight has been exploited in the cybersecurity community (Provos, 2001; Claffy & Dainotti, 2012; Cazorla et al., 2018), undetectable adversarial attacks on sequential decision-makers and their defences have not yet been systematically explored in the AI community.
Revised: While this insight has been exploited in the cybersecurity community (Provos, 2001; Claffy & Dainotti, 2012; Cazorla et al., 2018), undetectable adversarial attacks on sequential decision-makers and their defences have not yet been systematically explored in the AI community.

Original: In this paper, we introduce illusory attacks , a novel class of adversarial attacks on sequential decisionmakers that result in observation space attacks that are consistent with environment dynamics.
Revised: In this paper, we introduce illusory attacks , a novel class of adversarial attacks on sequential decisionmakers that result in observation space attacks that are consistent with environment dynamics.

Original: We show that illusory attacks can succeed where existing attacks do not, and, in particular, can successfully fool humans.
Revised: We show that illusory attacks can succeed where existing attacks do not, and, in particular, can successfully fool humans.

Original: Illusory attacks therefore pose a specific safety and security threat to human–AI interaction and human-in-the-loop settings (Schirner et al., 2013; Zerilli et al., 2019) as they may can be undetectable even to attentive human supervisors.
Revised: Illusory attacks therefore pose a specific safety and security threat to human–AI interaction and human-in-the-loop settings (Schirner et al., 2013; Zerilli et al., 2019) as they may can be undetectable even to attentive human supervisors.

Original: Illusory attacks seek to remain undetected and must hence attack the victim by replacing its perceived reality by an internally coherent alternative one.
Revised: Illusory attacks seek to remain undetected and must hence attack the victim by replacing its perceived reality by an internally coherent alternative one.

Original: We show that perfect (statistically undetectable) illusory attacks exist in a variety of environments.
Revised: We show that perfect (statistically undetectable) illusory attacks exist in a variety of environments.

Original: We then present the W -illusory attack framework (see Figure 1 for illustration), which introduces a world-model consistency optimisation objective that encourages the resultant victim’s action-observation histories to be consistent with the environment dynamics.
Revised: We then present the W -illusory attack framework (see Figure 1 for illustration), which introduces a world-model consistency optimisation objective that encourages the resultant victim’s action-observation histories to be consistent with the environment dynamics.

Original: We show that W -illusory attacks can be efficiently learnt, and are undetectable by humans, unlike MNP (Kumar et al., 2021) or SA-MDP attacks (Zhang et al., 2021).
Revised: We show that W -illusory attacks can be efficiently learnt, and are undetectable by humans, unlike MNP (Kumar et al., 2021) or SA-MDP attacks (Zhang et al., 2021).

Original: We empirically demonstrate that existing victim robustification methods are largely ineffective against illusory attacks.
Revised: We empirically demonstrate that existing victim robustification methods are largely ineffective against illusory attacks.

Original: This leads us to suggest that the existence of reality feedback , i.e., observation channels that are hardened against adversarial interference, may can play a decisive role in certifying the safety of real-world AI, and human–AI, systems.
Revised: This leads us to suggest that the existence of reality feedback , i.e., observation channels that are hardened against adversarial interference, may can play a decisive role in certifying the safety of real-world AI, and human–AI, systems.

Original: Our work makes the following contributions:
Revised: Our work makes the following contributions:

Original: • We formalise perfect illusory attacks , a novel framework for undetectable adversarial attacks (see Section 4.2).
Revised: • We formalise perfect illusory attacks , a novel framework for undetectable adversarial attacks (see Section 4.2).

Original: We give examples in common benchmark environments in Section 5.3.
Revised: We give examples in common benchmark environments in Section 5.3.

Original: • We introduce W -illusory attacks, a computationally feasible learning algorithm for adversarial attacks that generate victim action-observation sequences that are consistent with the unperturbed environment dynamics (see Section 4.5).
Revised: • We introduce W -illusory attacks, a computationally feasible learning algorithm for adversarial attacks that generate victim action-observation sequences that are consistent with the unperturbed environment dynamics (see Section 4.5).

Original: • We show that, compared to state-of-the-art adversarial attacks, W -illusory attacks are significantly less likely to be detected by AI agents (Section 5.2), as well as by humans (Section 5.4).
Revised: • We show that, compared to state-of-the-art adversarial attacks, W -illusory attacks are significantly less likely to be detected by AI agents (Section 5.2), as well as by humans (Section 5.4).

Original: • We demonstrate that victim robustification against W -illusory attacks is challenging unless the environment admits reality feedback (see Section 5.5).
Revised: • We demonstrate that victim robustification against W -illusory attacks is challenging unless the environment admits reality feedback (see Section 5.5).

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Adversarial attacks literature originates in non-sequential decision-making applications such as image classification (Szegedy et al., 2013), where the goal is to find perturbations δ for a given classifier f such that f yields different predictions for x and x + δ , despite the difference between x and x + δ being imperceptible to humans.
Revised: Adversarial attacks literature originates in non-sequential decision-making applications such as image classification (Szegedy et al., 2013), where the goal is to find perturbations δ for a given classifier f such that f yields different predictions for x and x + δ , despite the difference between x and x + δ being imperceptible to humans.

Original: To enforce the imperceptibility requirement, such works use simple minimum-norm perturbations constraints as a proxy (Goodfellow et al., 2014).
Revised: To enforce the imperceptibility requirement, such works use simple minimum-norm perturbations constraints as a proxy (Goodfellow et al., 2014).

Original: This line of work has been extended to adversarial attacks on sequential decision-making agents (Oikarinen et al., 2021b; Kumar et al., 2021;
Revised: This line of work has been extended to adversarial attacks on sequential decision-making agents (Oikarinen et al., 2021b; Kumar et al., 2021;

Original: Cohen et al., 2019; Everett et al., 2021;
Revised: Cohen et al., 2019; Everett et al., 2021;

Original: MoosaviDezfooli et al., 2017; Chaubey et al., 2020; Wu et al., 2021; Ilahi et al., 2021; Chen et al., 2019;
Revised: MoosaviDezfooli et al., 2017; Chaubey et al., 2020; Wu et al., 2021; Ilahi et al., 2021; Chen et al., 2019;

Original: Qiaoben et al., 2021; Sun et al., 2020; Zhang et al., 2020), largely building upon the minimum-norm perturbation (MNP) framework.
Revised: Qiaoben et al., 2021; Sun et al., 2020; Zhang et al., 2020), largely building upon the minimum-norm perturbation (MNP) framework.

Original: In the MNP framework, the adversary can modify the victim’s observations up to a fixed step- or episode-wise perturbation budget.
Revised: In the MNP framework, the adversary can modify the victim’s observations up to a fixed step- or episode-wise perturbation budget.

Original: While many adversarial attack frameworks require white-box access to the victim’s policy, Zhang et al.
Revised: While many adversarial attack frameworks require white-box access to the victim’s policy, Zhang et al.

Original: and Sun et al.
Revised: and Sun et al.

Original: (2021) use reinforcement learning to learn adversarial policies, thus require only black-box access to the victim’s policy.
Revised: (2021) use reinforcement learning to learn adversarial policies, thus require only black-box access to the victim’s policy.

Original: Assuming a different setting, Hussenot et al.
Revised: Assuming a different setting, Hussenot et al.

Original: (2019) introduce a class of adversaries for which a unique mask is precomputed and added to the agents’ observation at every time step.
Revised: (2019) introduce a class of adversaries for which a unique mask is precomputed and added to the agents’ observation at every time step.

Original: Our framework differs from these previous works both in that it takes into account the temporal correlation of sequences of observations, and in that it focuses on learning adversarial attacks that are undetectable.
Revised: Our framework differs from these previous works both in that it takes into account the temporal correlation of sequences of observations, and in that it focuses on learning adversarial attacks that are undetectable.

Original: Work towards robust agents in sequential decision environments uses randomized smoothing (Kumar et al., 2021; Wu et al., 2021), test-time hardening by computing confidence bounds (Everett et al., 2021), training with adversarial loss functions (Oikarinen et al., 2021a), and co-training with adversarial agents (Zhang et al., 2021).
Revised: Work towards robust agents in sequential decision environments uses randomized smoothing (Kumar et al., 2021; Wu et al., 2021), test-time hardening by computing confidence bounds (Everett et al., 2021), training with adversarial loss functions (Oikarinen et al., 2021a), and co-training with adversarial agents (Zhang et al., 2021).

Original: We compare against, and build upon, this line of work.
Revised: We compare against, and build upon, this line of work.

Original: Another extensive body of work focuses on detecting adversarial attacks .
Revised: Another extensive body of work focuses on detecting adversarial attacks .

Original: Lin et al.
Revised: Lin et al.

Original: (2017) develop an action-conditioned frame module that allows agents to detect adversarial attacks by comparing both the module’s action distribution with the realised action distribution.
Revised: (2017) develop an action-conditioned frame module that allows agents to detect adversarial attacks by comparing both the module’s action distribution with the realised action distribution.

Original: Tekgul et al.
Revised: Tekgul et al.

Original: (2021) detect adversaries by evaluating the feasibility of past action sequences.
Revised: (2021) detect adversaries by evaluating the feasibility of past action sequences.

Original: Compared to these works, our proposed methods hinge on detectability based on stronger notions of statistical indistinguishability.
Revised: Compared to these works, our proposed methods hinge on detectability based on stronger notions of statistical indistinguishability.

Original: Further, it focuses on learning undetectable attacks, rather than detecting adversarial attacks.
Revised: Further, it focuses on learning undetectable attacks, rather than detecting adversarial attacks.

Original: There exists only a limited body of literature on undetectable (or “stealthy” ) adversarial attacks (Li et al., 2019; Sun et al., 2020) on sequential-decision-making agents, none of which achieves statistical indistinguishability, nor takes into account temporally-extended observation dependencies; Huang &
Revised: There exists only a limited body of literature on undetectable (or “stealthy” ) adversarial attacks (Li et al., 2019; Sun et al., 2020) on sequential-decision-making agents, none of which achieves statistical indistinguishability, nor takes into account temporally-extended observation dependencies; Huang &

Original: Zhu (2019) consider stealthy attacks on reward signals used for RL.
Revised: Zhu (2019) consider stealthy attacks on reward signals used for RL.

Original: In cybersecurity, similar ideas have been explored for sensor attacks on linear control systems (Mo & Sinopoli, 2010; Pasqualetti et al., 2015).
Revised: In cybersecurity, similar ideas have been explored for sensor attacks on linear control systems (Mo & Sinopoli, 2010; Pasqualetti et al., 2015).

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: MDP and POMDP.
Revised: MDP and POMDP.

Original: A Markov decision process (Bellman, 1958, MDP) is a tuple ⟨S , A , p, r, γ ⟩ where S is a discrete or continuous state space,
Revised: A Markov decision process (Bellman, 1958, MDP) is a tuple ⟨S , A , p, r, γ ⟩ where S is a discrete or continuous state space,

Original: A is a discrete or continuous action space, p : S ×A×
Revised: A is a discrete or continuous action space, p : S ×A×

Original: S → [0 , 1] is the state transition probability, r : S ×A× R
Revised: S → [0 , 1] is the state transition probability, r : S ×A× R

Original: →
Revised: →

Original: [0 , 1] is a probabilistic reward function, and γ ∈ [0 , 1] is a scalar discount factor.
Revised: [0 , 1] is a probabilistic reward function, and γ ∈ [0 , 1] is a scalar discount factor.

Original: A Markov decision process proceeds by sampling an initial state s 0 ∼ p ( ∅ ) ∈ S , upon which the agent may take takes an action a 0 ∼ π ( s 0 )
Revised: A Markov decision process proceeds by sampling an initial state s 0 ∼ p ( ∅ ) ∈ S , upon which the agent may take takes an action a 0 ∼ π ( s 0 )

Original: ∈
Revised: ∈

Original: A triggering a state transition s 1 ∼ p ( s 0 , a 1 ) and returning a reward r 0 ∼ r ( s 1 , a 0 , s 0 ) .
Revised: A triggering a state transition s 1 ∼ p ( s 0 , a 1 ) and returning a reward r 0 ∼ r ( s 1 , a 0 , s 0 ) .

Original: Here, π : S × A → [0 , 1] is a stochastic agent policy.
Revised: Here, π : S × A → [0 , 1] is a stochastic agent policy.

Original: A partially-observable MDP ( ˚Astr¨om, 1965, POMDP) is a latent MDP ⟨S , A , p, r, γ ⟩ , together with a stochastic observation function Z : S × O → [0 , 1] .
Revised: A partially-observable MDP ( ˚Astr¨om, 1965, POMDP) is a latent MDP ⟨S , A , p, r, γ ⟩ , together with a stochastic observation function Z : S × O → [0 , 1] .

Original: In a POMDP, the agent cannot observe states s t directly, but has to act based on observations
Revised: In a POMDP, the agent cannot observe states s t directly, but has to act based on observations

Original: z t ∼ Z
Revised: z t ∼ Z

Original: ( s t ) .
Revised: ( s t ) .

Original: We further define a state-observation action-observation trajectory of length n as τ = ( z 0 , a 0 , . . .
Revised: We further define a state-observation action-observation trajectory of length n as τ = ( z 0 , a 0 , . . .

Original: , z n ) τ =
Revised: , z n ) τ =

Original: ( z 0 , a 0 , r 0 , . . .
Revised: ( z 0 , a 0 , r 0 , . . .

Original: , z n ) , and the distribution over trajectories generated by a policy π acting in a (PO)MDP E as P E ,π ( τ ) .
Revised: , z n ) , and the distribution over trajectories generated by a policy π acting in a (PO)MDP E as P E ,π ( τ ) .

Original: We assume that agents do not have access to the reward signal at testtime.
Revised: We assume that agents do not have access to the reward signal at testtime.

Original: We denote that two distributions P E ,π ( τ ) , P ′E ,π ( τ ) are identical up to reward by writing
Revised: We denote that two distributions P E ,π ( τ ) , P ′E ,π ( τ ) are identical up to reward by writing

Original: P E ,π ( τ ) =
Revised: P E ,π ( τ ) =

Original: P ′ E ,π ( τ ) P E ,π ( τ ) ¬ r = P ′ E ,π ( τ ) .
Revised: P ′ E ,π ( τ ) P E ,π ( τ ) ¬ r = P ′ E ,π ( τ ) .

Original: Observation-space adversarial attacks on sequential decision making agents.
Revised: Observation-space adversarial attacks on sequential decision making agents.

Original: Observationspace adversarial attacks (Kumar et al., 2021) consider the scenario where an adversary can manipulate the observation given to a victim agent at test-time, thus introducing partial observability to the victim’s test-time environment.
Revised: Observationspace adversarial attacks (Kumar et al., 2021) consider the scenario where an adversary can manipulate the observation given to a victim agent at test-time, thus introducing partial observability to the victim’s test-time environment.

Original: The victim’s observation at time step t is then given as ˜ z t =
Revised: The victim’s observation at time step t is then given as ˜ z t =

Original: z t + ϵ t instead of z t , where ϵ t ∈ R is the perturbation introduced by the adversary.
Revised: z t + ϵ t instead of z t , where ϵ t ∈ R is the perturbation introduced by the adversary.

Original: When modeling the adversarial perturbations as generated by a state-conditioned adversarial agent with policy π adv , we can write ˜ z t = π adv ( s t ) .
Revised: When modeling the adversarial perturbations as generated by a state-conditioned adversarial agent with policy π adv , we can write ˜ z t = π adv ( s t ) .

Original: In many scenarios, the size of the perturbation is bounded by a budget
Revised: In many scenarios, the size of the perturbation is bounded by a budget

Original: B > 0 ; for simplicity, we consider zero-sum adversarial attacks, where the adversary minimizes the expected return of the victim.
Revised: B > 0 ; for simplicity, we consider zero-sum adversarial attacks, where the adversary minimizes the expected return of the victim.

Original: This yields the following definition of an optimal state-conditioned observation-space adversary :
Revised: This yields the following definition of an optimal state-conditioned observation-space adversary :

Original: [Equation]
Revised: [Equation]

Original: where a t ∼ π v (cid:0) · | ˜ z t (cid:1) ,
Revised: where a t ∼ π v (cid:0) · | ˜ z t (cid:1) ,

Original: ˜ z t ∼ π adv ( s t
Revised: ˜ z t ∼ π adv ( s t

Original: [Equation]
Revised: [Equation]

Original: , s
Revised: , s

Original: [Equation]
Revised: [Equation]

Original: +1 ∼ p (
Revised: +1 ∼ p (

Original: [Equation]
Revised: [Equation]

Original: This concept can be formalized as a state-adversary MDP SA-MDP (Zhang et al., 2020), which is defined as a tuple of the victim MDP ⟨S , A , p, r, γ ⟩ together with an adversary ν : S → P ( S ) and a mapping B : S → S , limiting supp ν ( ·| s ) ∈ B ( s ) .
Revised: This concept can be formalized as a state-adversary MDP SA-MDP (Zhang et al., 2020), which is defined as a tuple of the victim MDP ⟨S , A , p, r, γ ⟩ together with an adversary ν : S → P ( S ) and a mapping B : S → S , limiting supp ν ( ·| s ) ∈ B ( s ) .

Original: Robustifying against adversarial attacks with co-training.
Revised: Robustifying against adversarial attacks with co-training.

Original: The optimal policy π ∗ v in an SA-MDP thereby defines the optimal robust policy of a victim under the presence of an optimal adversary ν ∗ .
Revised: The optimal policy π ∗ v in an SA-MDP thereby defines the optimal robust policy of a victim under the presence of an optimal adversary ν ∗ .

Original: To find such optimally robust victim policies, Zhang et al.
Revised: To find such optimally robust victim policies, Zhang et al.

Original: (2021, ATLA) introduce a co-training approach to SA-MDPs, where victim and adversary are trained in turn while keeping the other’s policy fixed.
Revised: (2021, ATLA) introduce a co-training approach to SA-MDPs, where victim and adversary are trained in turn while keeping the other’s policy fixed.

Original:  M ETHODS
Revised:  M ETHODS

Original: In this paper, we are concerned with finding adversarial attack policies π a that are undetectable .
Revised: In this paper, we are concerned with finding adversarial attack policies π a that are undetectable .

Original: Our setting differs from traditional observation-space attacks (see Section 3) in three ways.
Revised: Our setting differs from traditional observation-space attacks (see Section 3) in three ways.

Original: First, we assume that the victim has access to a world model m v .
Revised: First, we assume that the victim has access to a world model m v .

Original: Second, we assume that perturbation budgets may be unconstrained we consider unconstrained perturbation budgets.
Revised: Second, we assume that perturbation budgets may be unconstrained we consider unconstrained perturbation budgets.

Original: Third, we assume that the victim has a costly contingency option which it can safely execute once it has detected the presence of a test-time adversary.
Revised: Third, we assume that the victim has a costly contingency option which it can safely execute once it has detected the presence of a test-time adversary.

Original: While seemingly unusual, we argue that the above three assumptions are realistic in many settings:
Revised: While seemingly unusual, we argue that the above three assumptions are realistic in many settings:

Original: Often victims can learn accurate world models from unperturbed train-time samples, or rely on the experience of a human supervisor for anomaly detection.
Revised: Often victims can learn accurate world models from unperturbed train-time samples, or rely on the experience of a human supervisor for anomaly detection.

Original: In addition, in some settings the constraint on small perturbations seems arbitrary, e.g. when an attacker is replacing the entire video feed of a surveillance camera.
Revised: In addition, in some settings the constraint on small perturbations seems arbitrary, e.g. when an attacker is replacing the entire video feed of a surveillance camera.

Original: Lastly, contingency options for AI agents are a common feature of real-world systems, such as e.g. self-driving cars that can hand-over to a human controller, or security level escalation procedures in cyber-physical systems (Z¨uren, 2021).
Revised: Lastly, contingency options for AI agents are a common feature of real-world systems, such as e.g. self-driving cars that can hand-over to a human controller, or security level escalation procedures in cyber-physical systems (Z¨uren, 2021).

Original: 4. D ETECTABILITY OF ADVERSARIAL ATTACKS
Revised: 4. D ETECTABILITY OF ADVERSARIAL ATTACKS

Original: An adversarial attack can be detected if the attacked MDP E ′ , which includes the adversary with π adv , can be statistically distinguished from the train-time MDP E .
Revised: An adversarial attack can be detected if the attacked MDP E ′ , which includes the adversary with π adv , can be statistically distinguished from the train-time MDP E .

Original: Definition 4.1 (Statistical indistinguishability) .
Revised: Definition 4.1 (Statistical indistinguishability) .

Original: Let E :=
Revised: Let E :=

Original: ⟨S , A , p, r, γ ⟩ be a MDP with horizon
Revised: ⟨S , A , p, r, γ ⟩ be a MDP with horizon

Original: T. Let P be a partially-observable discrete-time stochastic control process, with observation space
Revised: T. Let P be a partially-observable discrete-time stochastic control process, with observation space

Original: Z := S , action space A , and horizon T. Let π : S ×
Revised: Z := S , action space A , and horizon T. Let π : S ×

Original: A × ( S × A ) T − 1 → [0 , 1] be a sampling policy that may possibly conditions on the whole action-observation history.
Revised: A × ( S × A ) T − 1 → [0 , 1] be a sampling policy that may possibly conditions on the whole action-observation history.

Original: Then P is statistically indistinguishable from E under π if and only if P π,
Revised: Then P is statistically indistinguishable from E under π if and only if P π,

Original: E
Revised: E

Original: = P π, E ′ P E ,π ¬ r = P P ,π .
Revised: = P π, E ′ P E ,π ¬ r = P P ,π .

Original: 4. U NDETECTABLE ADVERSARIAL ATTACKS
Revised: 4. U NDETECTABLE ADVERSARIAL ATTACKS

Original: We define perfect illusory attacks, i.e. adversarial attacks that are undetectable, as those where E and E ′ are statistically indistinguishable.
Revised: We define perfect illusory attacks, i.e. adversarial attacks that are undetectable, as those where E and E ′ are statistically indistinguishable.

Original: Importantly, since in our setting reward signals cannot be observed at test-time, indistinguishability need only hold for observation transitions.
Revised: Importantly, since in our setting reward signals cannot be observed at test-time, indistinguishability need only hold for observation transitions.

Original: For simplicity, we only consider zero-sum adversarial attacks, i.e. those that try to minimise the expected return of the victim.
Revised: For simplicity, we only consider zero-sum adversarial attacks, i.e. those that try to minimise the expected return of the victim.

Original: Definition 4.2 (Perfect illusory attack) .
Revised: Definition 4.2 (Perfect illusory attack) .

Original: A perfect illusory attack on an environment E for a given victim policy π v is an adversarial policy π a , such that E and E ′ are statistically indistinguishable under π v , where E ′ is the environment included by π a attacking E .
Revised: A perfect illusory attack on an environment E for a given victim policy π v is an adversarial policy π a , such that E and E ′ are statistically indistinguishable under π v , where E ′ is the environment included by π a attacking E .

Original: Note that the definition of perfect illusory attacks makes no reference to the adversary’s performance (victim reward).
Revised: Note that the definition of perfect illusory attacks makes no reference to the adversary’s performance (victim reward).

Original: We prove that non-trivial (i.e. those actually changing the observation) perfect illusory attacks do exist in some, but not all, victim policy/environment pairs (see Appendix 7.1).
Revised: We prove that non-trivial (i.e. those actually changing the observation) perfect illusory attacks do exist in some, but not all, victim policy/environment pairs (see Appendix 7.1).

Original: We provide further examples of perfect illusory attacks in Section 5.
Revised: We provide further examples of perfect illusory attacks in Section 5.

Original: Definition 4.3 (Optimal illusory attack) .
Revised: Definition 4.3 (Optimal illusory attack) .

Original: An illusory attack with π adv on ( E , π v ) is optimal if and only if it is the perfect illusory attack with the highest expected adversarial return, i.e.
Revised: An illusory attack with π adv on ( E , π v ) is optimal if and only if it is the perfect illusory attack with the highest expected adversarial return, i.e.

Original: [Equation]
Revised: [Equation]

Original: , where r t is the victim reward.
Revised: , where r t is the victim reward.

Original: As the constraints in Equation 2 are difficult to optimise exactly, we instead consider a relaxed weighted objective
Revised: As the constraints in Equation 2 are difficult to optimise exactly, we instead consider a relaxed weighted objective

Original: [Equation]
Revised: [Equation]

Original: where λ is a hyper-parameter that determines the weighing of the two objectives, and D is some distance measure between distributions, such as the KL-divergence.
Revised: where λ is a hyper-parameter that determines the weighing of the two objectives, and D is some distance measure between distributions, such as the KL-divergence.

Original: Adversaries that trade off between remaining undetected and minimizing the victim reward.
Revised: Adversaries that trade off between remaining undetected and minimizing the victim reward.

Original: In most environments, an adversary will need to trade-off its expected adversarial return with the desire to avoid detection.
Revised: In most environments, an adversary will need to trade-off its expected adversarial return with the desire to avoid detection.

Original: In practice, illusory attacks may not have to be perfect in order to be effective even non-perfect illusory attacks can be effective, for example when victims only have access to only n test-time samples from E ′ .
Revised: In practice, illusory attacks may not have to be perfect in order to be effective even non-perfect illusory attacks can be effective, for example when victims only have access to only n test-time samples from E ′ .

Original: In fact, n can usually be actively upper-bounded by the adversary’s preferences to engage or not.
Revised: In fact, n can usually be actively upper-bounded by the adversary’s preferences to engage or not.

Original: In addition, the victim’s world model may not be exact can be imperfect, particularly if learnt, or if relying on human experience.
Revised: In addition, the victim’s world model may not be exact can be imperfect, particularly if learnt, or if relying on human experience.

Original: We define illusory attacks under such relaxations in Appendix 7.2.
Revised: We define illusory attacks under such relaxations in Appendix 7.2.

Original: 4. W ORLD MODEL - CONSISTENT ADVERSARIAL ATTACKS .
Revised: 4. W ORLD MODEL - CONSISTENT ADVERSARIAL ATTACKS .

Original: Analytically optimising the distance D ( P E ,π v , P E ′ ,π v ) is usually intractable in all but the smallest environments.
Revised: Analytically optimising the distance D ( P E ,π v , P E ′ ,π v ) is usually intractable in all but the smallest environments.

Original: To arrive at a tractable optimisation objective, we instead consider a two-step test that allows to determine whether ˆ P E ,π v ¬ r = ˆ P E ′ ,π v .
Revised: To arrive at a tractable optimisation objective, we instead consider a two-step test that allows to determine whether ˆ P E ,π v ¬ r = ˆ P E ′ ,π v .

Original: Theorem 4.4 (Testing MDP equivalence) .
Revised: Theorem 4.4 (Testing MDP equivalence) .

Original: To determine whether a discrete-time stochastic control process M is equivalent to a given Markov process E , it is sufficient to test both whether M has the
Revised: To determine whether a discrete-time stochastic control process M is equivalent to a given Markov process E , it is sufficient to test both whether M has the

Original: Markov property, and whether M ’s transition probabilities match those of E (Shi et al., 2020).
Revised: Markov property, and whether M ’s transition probabilities match those of E (Shi et al., 2020).

Original: We further assume from now that the victim has access to an exact, or approximate, world model m v ( z ′ | z, a ) of E , which estimates the probability of the next observations given the current observation and action taken in E .
Revised: We further assume from now that the victim has access to an exact, or approximate, world model m v ( z ′ | z, a ) of E , which estimates the probability of the next observations given the current observation and action taken in E .

Original: We also assume the victim has an estimate ˆ P ( z ′ | z, a ) of the same quantity but in E ′ .
Revised: We also assume the victim has an estimate ˆ P ( z ′ | z, a ) of the same quantity but in E ′ .

Original: We now define W -illusory attacks, for which the victim’s world model estimate matches the estimate of the transition probabilities.
Revised: We now define W -illusory attacks, for which the victim’s world model estimate matches the estimate of the transition probabilities.

Original: Definition 4.5 ( W -illusory attacks) .
Revised: Definition 4.5 ( W -illusory attacks) .

Original: A W -illusory attack on E is an adversarial attack π a that is consistent with the victim’s model of the observation-transition probabilities m v .
Revised: A W -illusory attack on E is an adversarial attack π a that is consistent with the victim’s model of the observation-transition probabilities m v .

Original: Formally,
Revised: Formally,

Original: [Equation]
Revised: [Equation]

Original: We note that world-model consistency alone does not imply the Markov property, as ˆ P does not account for long term correlations.
Revised: We note that world-model consistency alone does not imply the Markov property, as ˆ P does not account for long term correlations.

Original: So importantly, W -illusory attacks can in general change the distribution of trajectories.
Revised: So importantly, W -illusory attacks can in general change the distribution of trajectories.

Original: 4. L EARNING WORLD MODEL CONSISTENT ADVERSARIAL ATTACKS .
Revised: 4. L EARNING WORLD MODEL CONSISTENT ADVERSARIAL ATTACKS .

Original: We now define the Illusory-MDP ( I -MDP) used to learn illusory adversarial attacks.
Revised: We now define the Illusory-MDP ( I -MDP) used to learn illusory adversarial attacks.

Original: This is the MDP the adversary trains on.
Revised: This is the MDP the adversary trains on.

Original: Definition 4.6 (Illusory MDP ( I -MDP)) .
Revised: Definition 4.6 (Illusory MDP ( I -MDP)) .

Original: Given a MDP E := ⟨S , A , p, r, γ ⟩ with episode horizon T and a victim policy π , an Illusory MDP ( I -MDP) E I :
Revised: Given a MDP E := ⟨S , A , p, r, γ ⟩ with episode horizon T and a victim policy π , an Illusory MDP ( I -MDP) E I :

Original: = I ( E , π ) consists of a tuple (cid:68) ˜ S , ˜ A , ˜ p, ˜ r, γ (cid:69) where ˜ A := S , ˜ S :
Revised: = I ( E , π ) consists of a tuple (cid:68) ˜ S , ˜ A , ˜ p, ˜ r, γ (cid:69) where ˜ A := S , ˜ S :

Original: = S × (cid:16) ˜
Revised: = S × (cid:16) ˜

Original: A × A (cid:17) T − 1 , ˜ p (˜ s t +1 | ˜ a t , ˜ s t ) :
Revised: A × A (cid:17) T − 1 , ˜ p (˜ s t +1 | ˜ a t , ˜ s t ) :

Original: =
Revised: =

Original: π ( a t | ˜ a ≤ t , a <t ) p ( s t +1 | s t ,
Revised: π ( a t | ˜ a ≤ t , a <t ) p ( s t +1 | s t ,

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: ∀ t, (5) and ˜ r
Revised: ∀ t, (5) and ˜ r

Original: (˜ r t +1 | ˜ s t +1 , ˜ a t , ˜ s t ) :=
Revised: (˜ r t +1 | ˜ s t +1 , ˜ a t , ˜ s t ) :=

Original: [Equation]
Revised: [Equation]

Original: ( a t | ˜ a ≤ t , a <t )
Revised: ( a t | ˜ a ≤ t , a <t )

Original: [Equation]
Revised: [Equation]

Original: ( r t +1 | s t +
Revised: ( r t +1 | s t +

Original: [Equation]
Revised: [Equation]

Original: a t
Revised: a t

Original: [Equation]
Revised: [Equation]

Original: s t )
Revised: s t )

Original: [Equation]
Revised: [Equation]

Original: ∀
Revised: ∀

Original: [Equation]
Revised: [Equation]

Original: As we assume zero-sum adversarial settings, r t +1 = − ˜ r t +1 .
Revised: As we assume zero-sum adversarial settings, r t +1 = − ˜ r t +1 .

Original: Note that the victim’s policy conditions on its action-observation history, and not just on its last observation, as it contains a sequence detector.
Revised: Note that the victim’s policy conditions on its action-observation history, and not just on its last observation, as it contains a sequence detector.

Original: As a result, the I -MDP differs from the SA-MDP (Zhang et al., 2021) in that the state ˜ s t := { s t , ˜ a <t , a <t } , on which the adversarial policy π adv is conditioned, is given by the state of the
Revised: As a result, the I -MDP differs from the SA-MDP (Zhang et al., 2021) in that the state ˜ s t := { s t , ˜ a <t , a <t } , on which the adversarial policy π adv is conditioned, is given by the state of the

Original: MDP E , as well as the victim’s action-observation history.
Revised: MDP E , as well as the victim’s action-observation history.

Original: Definition 4.7 (Learning W -illusory attacks) .
Revised: Definition 4.7 (Learning W -illusory attacks) .

Original: Given a fixed victim policy π v , a W -illusory attack π a on E can be learnt by optimising
Revised: Given a fixed victim policy π v , a W -illusory attack π a on E can be learnt by optimising

Original: [Equation]
Revised: [Equation]

Original: Here, we choose the L 2 -distance, i.e. ∥ m a ( a t − 1 , π v ( a t − 1 )) − a t ∥ 22 for deterministic environments, where m a is the adversary’s world model.
Revised: Here, we choose the L 2 -distance, i.e. ∥ m a ( a t − 1 , π v ( a t − 1 )) − a t ∥ 22 for deterministic environments, where m a is the adversary’s world model.

Original: For stochastic environments, a possible choice would be D KL , however, alternatively, one could simply employ the maximum-likelihood objective log m a ( a t | a t − 1 , π v ( a t − 1 ) .
Revised: For stochastic environments, a possible choice would be D KL , however, alternatively, one could simply employ the maximum-likelihood objective log m a ( a t | a t − 1 , π v ( a t − 1 ) .

Original: Non-Markovian temporal correlations in the adversarial policy can be suppressed by restricting the horizon of the adversarial policy’s observation space ˜ S , which could be detected by the victim (Shi et al., 2020).
Revised: Non-Markovian temporal correlations in the adversarial policy can be suppressed by restricting the horizon of the adversarial policy’s observation space ˜ S , which could be detected by the victim (Shi et al., 2020).

Original: 4. R EALITY FEEDBACK
Revised: 4. R EALITY FEEDBACK

Original: So far we have assumed that the adversary can corrupt all parts of the victim’s observation.
Revised: So far we have assumed that the adversary can corrupt all parts of the victim’s observation.

Original: However, in practice, the AI agent may be able to could receive a limited amount of unperturbed environment (“reality”) feedback through robust channels which it may, in principle, can use to act optimally in the presence of an adversary.
Revised: However, in practice, the AI agent may be able to could receive a limited amount of unperturbed environment (“reality”) feedback through robust channels which it may, in principle, can use to act optimally in the presence of an adversary.

Original: Definition 4.8 (Reality feedback) .
Revised: Definition 4.8 (Reality feedback) .

Original: We define reality feedback ζ as a part of the victim’s observation in E ′ that cannot be corrupted by the adversary, i.e. we assume that Z :
Revised: We define reality feedback ζ as a part of the victim’s observation in E ′ that cannot be corrupted by the adversary, i.e. we assume that Z :

Original: = Z 0 × Z ζ , where the adversary has access to z 0 ∈ Z 0 but not z ζ ∈ Z ζ .
Revised: = Z 0 × Z ζ , where the adversary has access to z 0 ∈ Z 0 but not z ζ ∈ Z ζ .

Original:  E XPERIMENTAL EVALUATION
Revised:  E XPERIMENTAL EVALUATION

Original: In Section 5.2, we first provide empirical evidence that state-of-the-art attacks on sequential decisionmakers can be detected by AI victims with a learnt world model.
Revised: In Section 5.2, we first provide empirical evidence that state-of-the-art attacks on sequential decisionmakers can be detected by AI victims with a learnt world model.

Original: In Section 5.3 we then provide examples of both perfect illusory attacks (see Definition 4.2), as well as learned W -illusory attacks (see Definition 4.5).
Revised: In Section 5.3 we then provide examples of both perfect illusory attacks (see Definition 4.2), as well as learned W -illusory attacks (see Definition 4.5).

Original: In Section 5.4, we give evidence that humans can efficiently detect state-of-theart adversarial attacks, but may can be unable to detect W -illusory attacks.
Revised: In Section 5.4, we give evidence that humans can efficiently detect state-of-theart adversarial attacks, but may can be unable to detect W -illusory attacks.

Original: Lastly, we investigate the effectiveness of state-of-the-art adversarial robustification algorithms against W -illusory attacks (Section 5.5).
Revised: Lastly, we investigate the effectiveness of state-of-the-art adversarial robustification algorithms against W -illusory attacks (Section 5.5).

Original: We find that effective defences against illusory attacks may can require the victim to have access to unperturbed feedback channels (Section 5.5).
Revised: We find that effective defences against illusory attacks may can require the victim to have access to unperturbed feedback channels (Section 5.5).

Original: 5. E XPERIMENTAL SETUP
Revised: 5. E XPERIMENTAL SETUP

Original: We compare illusory attacks against two adversarial attack benchmarks , (Kumar et al., 2021, MNP) and (Zhang et al., 2021, SA-MDP) (see Section 3).
Revised: We compare illusory attacks against two adversarial attack benchmarks , (Kumar et al., 2021, MNP) and (Zhang et al., 2021, SA-MDP) (see Section 3).

Original: Together, we consider these attacks to be representative of existing state-of-the-art approaches.
Revised: Together, we consider these attacks to be representative of existing state-of-the-art approaches.

Original: Both MNP and SA-MDP attacks require choosing perturbation budgets, which we here fix at β ∈ { 0 . 05 , 0 . 2 } relative to normalised observation vectors.
Revised: Both MNP and SA-MDP attacks require choosing perturbation budgets, which we here fix at β ∈ { 0 . 05 , 0 . 2 } relative to normalised observation vectors.

Original: To allow for fair comparisons, where necessary we also impose these budgets on our generated illusory attacks.
Revised: To allow for fair comparisons, where necessary we also impose these budgets on our generated illusory attacks.

Original: We consider two choices of victim policy : naive policies learnt through train-time return maximisation in the unattacked environment, and ATLA policies learnt through co-training (Zhang et al., 2021, ATLA).
Revised: We consider two choices of victim policy : naive policies learnt through train-time return maximisation in the unattacked environment, and ATLA policies learnt through co-training (Zhang et al., 2021, ATLA).

Original: We evaluate all methods on three simulated environments .
Revised: We evaluate all methods on three simulated environments .

Original: Gridworld (see Figure 1) involves navigating to a “good” grid cell while avoiding a “bad” grid cell.
Revised: Gridworld (see Figure 1) involves navigating to a “good” grid cell while avoiding a “bad” grid cell.

Original: The OpenAI gym (Brockman et al., 2016) environments CartPole and Pendulum (see Figures 2 and 3) feature continuous control tasks that require balancing a pole with discrete actions, and swinging up a pendulum with continuous torques.
Revised: The OpenAI gym (Brockman et al., 2016) environments CartPole and Pendulum (see Figures 2 and 3) feature continuous control tasks that require balancing a pole with discrete actions, and swinging up a pendulum with continuous torques.

Original: In accordance with Section 4.4, we assume that the adversary has access to an exact model of the environment dynamics m a .
Revised: In accordance with Section 4.4, we assume that the adversary has access to an exact model of the environment dynamics m a .

Original: To ensure reproducibility , all results are averaged over five runs with independent random seeds, and we report full results with standard deviations in Table 7.5.1 in the Appendix.
Revised: To ensure reproducibility , all results are averaged over five runs with independent random seeds, and we report full results with standard deviations in Table 7.5.1 in the Appendix.

Original: We release our code at anonymized .
Revised: We release our code at anonymized .

Original: 5. D ETECTING STATE - OF - THE - ART ADVERSARIAL ATTACKS WITH LEARNED WORLD MODELS
Revised: 5. D ETECTING STATE - OF - THE - ART ADVERSARIAL ATTACKS WITH LEARNED WORLD MODELS

Original: We demonstrate that existing state-of-the-art adversarial attacks can be detected using learned world models.
Revised: We demonstrate that existing state-of-the-art adversarial attacks can be detected using learned world models.

Original: To this end, we use train-time trajectory rollouts to train a small neural network-based world model ˆ m v for each environment (see Appendix 7.3).
Revised: To this end, we use train-time trajectory rollouts to train a small neural network-based world model ˆ m v for each environment (see Appendix 7.3).

Original: We then evaluate each ˆ m v on episodes from a holdout set of test-time trajectories from both the respective attacked and unattacked environment and under both naive and ATLA victim policies (see Section 5.1).
Revised: We then evaluate each ˆ m v on episodes from a holdout set of test-time trajectories from both the respective attacked and unattacked environment and under both naive and ATLA victim policies (see Section 5.1).

Original: To account for the neural network models’ finite precision, we regard transitions to be consistent with m v if they lie within the 5 σ confidence bound, which we computed on a held-out set of train-time transitions.
Revised: To account for the neural network models’ finite precision, we regard transitions to be consistent with m v if they lie within the 5 σ confidence bound, which we computed on a held-out set of train-time transitions.

Original: We find that both MNP and SA-MDP attacks frequently generate world-model inconsistent transitions, which allows for > 93% of attacked episodes to be detected with high accuracy, given the falsepositive rate of ˆ m v on unattacked environments of < 1% .
Revised: We find that both MNP and SA-MDP attacks frequently generate world-model inconsistent transitions, which allows for > 93% of attacked episodes to be detected with high accuracy, given the falsepositive rate of ˆ m v on unattacked environments of < 1% .

Original: Figure 1 gives a graphical illustration of attacks on Gridworld.
Revised: Figure 1 gives a graphical illustration of attacks on Gridworld.

Original: We provide videos of all attacks in the supplementary material (see Appendix 7.5.3).
Revised: We provide videos of all attacks in the supplementary material (see Appendix 7.5.3).

Original: [Table]
Revised: [Table]

Original: 5. L EARNING UNDETECTABLE ADVERSARIAL ATTACKS
Revised: 5. L EARNING UNDETECTABLE ADVERSARIAL ATTACKS

Original: We first show concrete examples of non-trivial perfect illusory attacks as defined in Section 4.2.
Revised: We first show concrete examples of non-trivial perfect illusory attacks as defined in Section 4.2.

Original: We then empirically investigate W -illusory attacks according to Definition 4.5.
Revised: We then empirically investigate W -illusory attacks according to Definition 4.5.

Original: Perfect illusory attacks.
Revised: Perfect illusory attacks.

Original: We find that Pendulum and CartPole admit perfect, i [Table] e. entirely undetectable, illusory attacks (see Definition 4.2) that significantly impact the victim’s performance independently of its policy.
Revised: We find that Pendulum and CartPole admit perfect, i.e. entirely undetectable, illusory attacks (see Definition 4.2) that significantly impact the victim’s performance independently of its policy.

Original: For Cartpole , such adversarial policies can be constructed by starting off with a coordinate transform z 0 = − s 0 , and then using the adversary’s world model to generate all subsequent observations (see Figure 2).
Revised: For Cartpole , such adversarial policies can be constructed by starting off with a coordinate transform z 0 = − s 0 , and then using the adversary’s world model to generate all subsequent observations (see Figure 2).

Original: Constructing the adversary’s policy proceeds equivalently for
Revised: Constructing the adversary’s policy proceeds equivalently for

Original: Pendulum, as both have symmetric initial state distributions.
Revised: Pendulum, as both have symmetric initial state distributions.

Original: The existence of strong attacks for two popular benchmark environments may astonish, as effective perfect illusory attacks do not always exist (see Appendix 7.1).
Revised: The existence of strong attacks for two popular benchmark environments may astonish, as effective perfect illusory attacks do not always exist (see Appendix 7.1).

Original: This may suggest that robotic control environments of practical interest could be particularly susceptible to undetectable adversarial attacks.
Revised: This may suggest that robotic control environments of practical interest could be particularly susceptible to undetectable adversarial attacks.

Original: W -illusory attacks.
Revised: W -illusory attacks.

Original: We now demonstrate how illusory attacks can be learnt rather than analytically constructed.
Revised: We now demonstrate how illusory attacks can be learnt rather than analytically constructed.

Original: We employ Algorithm 1 (see Appendix 7.5), which implements W -illusory adversarial attacks.
Revised: We employ Algorithm 1 (see Appendix 7.5), which implements W -illusory adversarial attacks.

Original: W -illusory adversarial attacks trade off between the objectives of detectability and adversarial performance.
Revised: W -illusory adversarial attacks trade off between the objectives of detectability and adversarial performance.

Original: In all of our environments, we choose our detectability objective to be proportional to the L ∞ -norm of the distance between the correct next observation, i.e. the next observation expected by the victim, and the adversarial observation generated by the adversary (cf.
Revised: In all of our environments, we choose our detectability objective to be proportional to the L ∞ -norm of the distance between the correct next observation, i.e. the next observation expected by the victim, and the adversarial observation generated by the adversary (cf.

Original: Definition 4.7).
Revised: Definition 4.7).

Original: We find the weighting parameter λ
Revised: We find the weighting parameter λ

Original: = 10 using a grid search (ablation in Appendix 7.5.2).
Revised: = 10 using a grid search (ablation in Appendix 7.5.2).

Original: Note that while Algorithm 1 could, in principle, discover the exact perfect illusory attacks presented above for both Cartpole and Pendulum , we do not find this to be the case in practice.
Revised: Note that while Algorithm 1 could, in principle, discover the exact perfect illusory attacks presented above for both Cartpole and Pendulum , we do not find this to be the case in practice.

Original: We suggest that this is due the convergence to sub-optimal local equilibria. 
Revised: We suggest that this is due the convergence to sub-optimal local equilibria. [Table]

Original: Detection.
Revised: Detection.

Original: Wenow compare and contrast W -illusory attacks, MNP attacks and SA-MDP attacks for perturbation budgets β
Revised: We compare and contrast W -illusory attacks, MNP attacks and SA-MDP attacks for perturbation budgets β

Original: = 0 .
Revised: = 0 .

Original: We observe that illusory attacks are much less likely to be detected by the learned world model ˆ m v (see section 5.2) than MNP and SA-MDP attacks (see
Revised: We observe that illusory attacks are much less likely to be detected by the learned world model ˆ m v (see section 5.2) than MNP and SA-MDP attacks (see

Original: Table 1).
Revised: Table 1).

Original: This finding is independent of the perturbation budget β .
Revised: This finding is independent of the perturbation budget β .

Original: ATLA victims are based on a recurrent neural network policy, enabling them, in principle, to learn an implicit model of the environment dynamics during train-time.
Revised: ATLA victims are based on a recurrent neural network policy, enabling them, in principle, to learn an implicit model of the environment dynamics during train-time.

Original: Co-training with such world model-aware victims, could, in theory, result in world-model consistent adversarial attacks.
Revised: Co-training with such world model-aware victims, could, in theory, result in world-model consistent adversarial attacks.

Original: However, we empirically find this not to be the case: As shown in Table 1, attacks on ATLA agents are detected just as likely as those on naive agents.
Revised: However, we empirically find this not to be the case: As shown in Table 1, attacks on ATLA agents are detected just as likely as those on naive agents.

Original: This underlines the need for explicit world model consistency constraints during adversarial training (see Definition 4.7).
Revised: This underlines the need for explicit world model consistency constraints during adversarial training (see Definition 4.7).

Original: We provide videos of all types of attacks for different seeds in the supplementary material (see Appendix 7.5.3).
Revised: We provide videos of all types of attacks for different seeds in the supplementary material (see Appendix 7.5.3).

Original: [Table]

Original: Victim performance under different attacks.
Revised: Victim performance under different attacks.

Original: We find that the average reward achieved by the victim (Table 1, 4 th column) is generally lower for β = 0 .
Revised: We find that the average reward achieved by the victim (Table 1, 4 th column) is generally lower for β = 0 .

Original: 2 , and also generally lower for SA-MDP attacks than for W -illusory attacks.
Revised: 2 , and also generally lower for SA-MDP attacks than for W -illusory attacks.

Original: This is to be expected, as W -illusory attacks trade off between minimizing the reward of the victim and generating consistent observation sequences, while an
Revised: This is to be expected, as W -illusory attacks trade off between minimizing the reward of the victim and generating consistent observation sequences, while an

Original: SA-MDP attack solely minimizes the victim reward.
Revised: SA-MDP attack solely minimizes the victim reward.

Original: Given that illusory attacks are significantly more constrained than SA-MDP attacks, we consider it surprising that undetectability can be traded in for only minor decreases in adversarial performance.
Revised: Given that illusory attacks are significantly more constrained than SA-MDP attacks, we consider it surprising that undetectability can be traded in for only minor decreases in adversarial performance.

Original: 5. F OOLING HUMANS WITH ILLUSORY ATTACKS
Revised: 5. F OOLING HUMANS WITH ILLUSORY ATTACKS

Original: We perform a controlled experiment with n = 10 human participants in order to collect evidence for our claim that while humans can efficiently detect state-of-the-art adversarial attacks, they cannot detect W -illusory attacks.
Revised: We perform a controlled experiment with n = 10 human participants in order to collect evidence for our claim that while humans can efficiently detect state-of-the-art adversarial attacks, they cannot detect W -illusory attacks.

Original: Participants were first shown example videos of both CartPole and
Revised: Participants were first shown example videos of both CartPole and

Original: Pendulum trajectories, exposing them to environment-specific dynamics.
Revised: Pendulum trajectories, exposing them to environment-specific dynamics.

Original: Participants were then asked to classify a random mixture of videos of test-time trajectories from both unattacked and attacked
Revised: Participants were then asked to classify a random mixture of videos of test-time trajectories from both unattacked and attacked

Original: CartPole and Pendulum environments; further details in Appendix 7.6.
Revised: CartPole and Pendulum environments; further details in Appendix 7.6.

Original: We found that participants were able to visually classify MNP and SA-MDP attacks from videos with high accuracy, similarly for unattacked trajectories.
Revised: We found that participants were able to visually classify MNP and SA-MDP attacks from videos with high accuracy, similarly for unattacked trajectories.

Original: However, humans struggled to correctly classify W -illusory attacks more accurately than non-attacks (see Table 2) as attacked.
Revised: However, humans struggled to correctly classify W -illusory attacks more accurately than non-attacks (see Table 2) as attacked.

Original: We establish the statistical significance of these results using a z -test statistic (see Section 7.6), which suggests that humans may can be unable to detect such illusory attacks.
Revised: We establish the statistical significance of these results using a z -test statistic (see Section 7.6), which suggests that humans may can be unable to detect such illusory attacks.

Original: 5. R OBUSTIFYING AGAINST ILLUSORY ATTACKS
Revised: 5. R OBUSTIFYING AGAINST ILLUSORY ATTACKS

Original: Having established the effectiveness of illusory attacks in Sections 5.3 and 5.4, we now investigate how victims casn be robustified.
Revised: Having established the effectiveness of illusory attacks in Sections 5.3 and 5.4, we now investigate how victims casn be robustified.

Original: We first consider two standard defences against adversarial attacks, randomized smoothing (Kumar et al., 2021) and adversarial pretraining (ATLA, (Zhang et al., 2021)).
Revised: We first consider two standard defences against adversarial attacks, randomized smoothing (Kumar et al., 2021) and adversarial pretraining (ATLA, (Zhang et al., 2021)).

Original: We also add an ablation of ATLA that co-trains the victim with a W -illusory adversaries.
Revised: We also add an ablation of ATLA that co-trains the victim with a W -illusory adversaries.

Original: As summarised in Table 1, we find that defences are generally more effective at restoring victim performance at β = 0 .
Revised: As summarised in Table 1, we find that defences are generally more effective at restoring victim performance at β = 0 .

Original: For β = 0 .
Revised: For β = 0 .

Original: 2 , all defences result in only minor improvements.
Revised: 2 , all defences result in only minor improvements.

Original: With only minor variations, these trends hold across all three robustification methods we deployed.
Revised: With only minor variations, these trends hold across all three robustification methods we deployed.

Original: Note that the victim performance achieved by applying randomized smoothing for β = 0 .
Revised: Note that the victim performance achieved by applying randomized smoothing for β = 0 .

Original: 2 in CartPole is equivalent to that of a random policy.
Revised: 2 in CartPole is equivalent to that of a random policy.

Original: Overall, our findings suggest that none of the robustification methods studied are particularly effective against illusory attacks.
Revised: Overall, our findings suggest that none of the robustification methods studied are particularly effective against illusory attacks.

Original: Exploiting reality feedback.
Revised: Exploiting reality feedback.

Original: We conclude our empirical investigations by exploring the importance of reality feedback in victim robustification (see Section 4.5).
Revised: We conclude our empirical investigations by exploring the importance of reality feedback in victim robustification (see Section 4.5).

Original: We establish two reality feedback scenarios for CartPole: one where the cart observation is unperturbed and one where the observation of the pole is unperturbed.
Revised: We establish two reality feedback scenarios for CartPole: one where the cart observation is unperturbed and one where the observation of the pole is unperturbed.

Original: We find that robustifying the victim agent through pre-training may be essential to enable enables victim agents to learn how to effectively use reality feedback.
Revised: We find that robustifying the victim agent through pre-training may be essential to enable enables victim agents to learn how to effectively use reality feedback.

Original: Our results further suggest that having access to informative reality feedback channels can allow for significant robustification.
Revised: Our results further suggest that having access to informative reality feedback channels can allow for significant robustification.

Original: See Appendix 7.7 for further details.
Revised: See Appendix 7.7 for further details.

Original:  C ONCLUSION AND FUTURE WORK
Revised:  C ONCLUSION AND FUTURE WORK

Original: In this paper, we introduce perfect illusory attacks , a novel class of adversarial attacks on sequential decision-makers that is undetectable.
Revised: In this paper, we introduce perfect illusory attacks , a novel class of adversarial attacks on sequential decision-makers that is undetectable.

Original: Importantly, we show that W -illusory attacks, unlike state-ofthe-art adversarial attacks, can fool both AI victim agents with access to a model of the environment dynamics, as well as human supervisors.
Revised: Importantly, we show that W -illusory attacks, unlike state-ofthe-art adversarial attacks, can fool both AI victim agents with access to a model of the environment dynamics, as well as human supervisors.

Original: Our results imply that robustification against illusory attacks is mostly ineffective unless the victim has access to reality feedback, making the provisioning of such hardened feedback channels imperative to designers of safe real-world autonomous or human-in-the-loop decision-making systems.
Revised: Our results imply that robustification against illusory attacks is mostly ineffective unless the victim has access to reality feedback, making the provisioning of such hardened feedback channels imperative to designers of safe real-world autonomous or human-in-the-loop decision-making systems.

Original: Future work may investigate illusory attacks in large-scale and partially observable victim environments.
Revised: Future work may investigate illusory attacks in large-scale and partially observable victim environments.

Original: Similarly, evaluations on real-world settings would be beneficial, e.g. using sim-to-real methodology.
Revised: Similarly, evaluations on real-world settings would be beneficial, e.g. using sim-to-real methodology.

Original: Our work further does not investigate the effect of constraining the number of test-time interactions between adversary and victim, or the effect of world model imperfections.
Revised: Our work further does not investigate the effect of constraining the number of test-time interactions between adversary and victim, or the effect of world model imperfections.

Original: Future work must also develop novel approaches and algorithms for victim robustification against illusory attacks.
Revised: Future work must also develop novel approaches and algorithms for victim robustification against illusory attacks.

Original: In particular, such methods should be able to maximise the utility of existing reality feedback channels through test-time information gathering behaviour.
Revised: In particular, such methods should be able to maximise the utility of existing reality feedback channels through test-time information gathering behaviour.

Original: Lastly, we believe that a conceptual framework unifying both budgeted and illusory attacks could be of value to the community.
Revised: Lastly, we believe that a conceptual framework unifying both budgeted and illusory attacks could be of value to the community.

Original:  A PPENDIX 7. P
Revised:  A PPENDIX 7. P

Original: [Figure]
Revised: [Figure]

Original: Proof that perfect illusory attacks exist.
Revised: Proof that perfect illusory attacks exist.

Original: We consider an example MDP (see Figure 4a) where a victim starts in node 1 or 2 each with probability 12 and can go up , down , or right in both states 1 and 2 .
Revised: We consider an example MDP (see Figure 4a) where a victim starts in node 1 or 2 each with probability 12 and can go up , down , or right in both states 1 and 2 .

Original: The episode terminates immediately with a return of 0 should the victim reach state 4 .
Revised: The episode terminates immediately with a return of 0 should the victim reach state 4 .

Original: Otherwise, the agent receives a reward of +1 if it reaches state 3 within a maximum of 2 steps.
Revised: Otherwise, the agent receives a reward of +1 if it reaches state 3 within a maximum of 2 steps.

Original: The optimal victim policy is therefore to take paths 1 → 3 if starting in state 1 , and take one of the two possible paths 2 → 1 → 3 otherwise.
Revised: The optimal victim policy is therefore to take paths 1 → 3 if starting in state 1 , and take one of the two possible paths 2 → 1 → 3 otherwise.

Original: The victim observes the labelled state graph, as well as its current state label.
Revised: The victim observes the labelled state graph, as well as its current state label.

Original: Clearly, choosing π adv (1) = 2 and π adv (2) = 1 constitutes a perfect illusory attack in this environment.
Revised: Clearly, choosing π adv (1) = 2 and π adv (2) = 1 constitutes a perfect illusory attack in this environment.

Original: Proof that perfect illusory attacks don’t always exist.
Revised: Proof that perfect illusory attacks don’t always exist.

Original: To show that some environments do not admit perfect illusory attacks, consider the modified environment in Figure 4b.
Revised: To show that some environments do not admit perfect illusory attacks, consider the modified environment in Figure 4b.

Original: Here, clearly a timestepconditioned victim policy that takes the action sequence ⟨ up , right ⟩ independently of observations cannot be perfectly attacked.
Revised: Here, clearly a timestepconditioned victim policy that takes the action sequence ⟨ up , right ⟩ independently of observations cannot be perfectly attacked.

Original: 7. F INITE - SAMPLE ILLUSORY ATTACKS
Revised: 7. F INITE - SAMPLE ILLUSORY ATTACKS

Original: Definition 7.1 (Finite-Sample Illusory Attacks) .
Revised: Definition 7.1 (Finite-Sample Illusory Attacks) .

Original: A finite-sample illusory attack is an adversarial attack that is not detectable by the victim within a given test-time interaction budget n .
Revised: A finite-sample illusory attack is an adversarial attack that is not detectable by the victim within a given test-time interaction budget n .

Original: Formally, an n -sample illusory attack on an environment E with victim policy π is an adversarial policy π adv for which ∄ D, ∄ δ > 0 such that simultaneously
Revised: Formally, an n -sample illusory attack on an environment E with victim policy π is an adversarial policy π adv for which ∄ D, ∄ δ > 0 such that simultaneously

Original: D (cid:16) P (ˆ τ n |E )
Revised: D (cid:16) P (ˆ τ n |E )

Original: ||
Revised: ||

Original: ˆ P (ˆ τ n ) (cid:17) <
Revised: ˆ P (ˆ τ n ) (cid:17) <

Original: δ, ∀ ˆ τ n ∼ P ⟨E , π v ⟩ and D (cid:16) P (ˆ τ n |E )
Revised: δ, ∀ ˆ τ n ∼ P ⟨E , π v ⟩ and D (cid:16) P (ˆ τ n |E )

Original: ||
Revised: ||

Original: ˆ P (ˆ τ n ) (cid:17)
Revised: ˆ P (ˆ τ n ) (cid:17)

Original: [Equation]
Revised: [Equation]

Original: Note that this definition is symmetric between adversary and victim.
Revised: Note that this definition is symmetric between adversary and victim.

Original: In practice, the victim will choose a D, δ such that its average test-time performance is maximised.
Revised: In practice, the victim will choose a D, δ such that its average test-time performance is maximised.

Original: As the victim is, however, not able to exactly infer E E ′ ,π v [ R ] as it does not have access to the adversary’s policy, the adversary in turn cannot exactly deduce how exactly the victim chooses D, δ .
Revised: As the victim is, however, not able to exactly infer E E ′ ,π v [ R ] as it does not have access to the adversary’s policy, the adversary in turn cannot exactly deduce how exactly the victim chooses D, δ .

Original: Therefore, in practice, the adversary might wish to choose an isotropic D , such as D KL , and construct a policy π a dv that minimises this metric as far as possible given a desired performance-detectability trade-off.
Revised: Therefore, in practice, the adversary might wish to choose an isotropic D , such as D KL , and construct a policy π a dv that minimises this metric as far as possible given a desired performance-detectability trade-off.

Original: 7. W ORLDMODEL TO DETECT ADVERSARIAL ATTACKS
Revised: 7. W ORLDMODEL TO DETECT ADVERSARIAL ATTACKS

Original: Setup.
Revised: Setup.

Original: We assume that the victim is trained in the unperturbed MDP M unperturbed for k episodes of length n .
Revised: We assume that the victim is trained in the unperturbed MDP M unperturbed for k episodes of length n .

Original: During training, the agent records the observed environment transition tuples t i =
Revised: During training, the agent records the observed environment transition tuples t i =

Original: ( s i , a i , s i +1 ) and stores these in a set D unp = { t i } i = k ∗ ( n − 1) i
Revised: ( s i , a i , s i +1 ) and stores these in a set D unp = { t i } i = k ∗ ( n − 1) i

Original: = .
Revised: = .

Original: At test time, the victim executes the learned policy and is attacked by an adversary agent, which replaces the true observation z t by the adversarial observation ˆ z t .
Revised: At test time, the victim executes the learned policy and is attacked by an adversary agent, which replaces the true observation z t by the adversarial observation ˆ z t .

Original: Learning a dynamics model.
Revised: Learning a dynamics model.

Original: We use the the set of unperturbed transitions D unp to learn an approximate dynamics model ˆ T ( s t | s t − 1 , a t − 1 ) .
Revised: We use the the set of unperturbed transitions D unp to learn an approximate dynamics model ˆ T ( s t | s t − 1 , a t − 1 ) .

Original: We implement ˆ T as a Multi-Layer-Perceptron with two hidden layers of size 10, and train the model with an l 2 loss for 10 epochs, using a learning rate of 0 .
Revised: We implement ˆ T as a Multi-Layer-Perceptron with two hidden layers of size 10, and train the model with an l 2 loss for 10 epochs, using a learning rate of 0 .

Original: 001 and the ADAM (Kingma & Ba, 2014) optimizer.
Revised: 001 and the ADAM (Kingma & Ba, 2014) optimizer.

Original: We then evaluate the model on a held-out set of the training observations and compute the 5 σ confidence bound on the l 0 distance, which we refer to as c .
Revised: We then evaluate the model on a held-out set of the training observations and compute the 5 σ confidence bound on the l 0 distance, which we refer to as c .

Original: We use then use c as a threshold to detect adversarial attacks.
Revised: We use then use c as a threshold to detect adversarial attacks.

Original: The high confidence ensures that the false positive rate remains low.
Revised: The high confidence ensures that the false positive rate remains low.

Original: Note that an adversary with access to ˆ m v could attempt to learn universal perturbations (MoosaviDezfooli et al., 2017) that simultaneously attack both π v and ˆ m v .
Revised: Note that an adversary with access to ˆ m v could attempt to learn universal perturbations (MoosaviDezfooli et al., 2017) that simultaneously attack both π v and ˆ m v .

Original: However, adversaries without access to ˆ m v may be unable to exploit its imperfections and instead need to assume it is perfect and hence justifying our comparison to solely policy-based MNP/SA-MDP attacks.
Revised: However, adversaries without access to ˆ m v may be unable to exploit its imperfections and instead need to assume it is perfect and hence justifying our comparison to solely policy-based MNP/SA-MDP attacks.

Original: Using a dynamics model to detect adversarial attacks.
Revised: Using a dynamics model to detect adversarial attacks.

Original: We then evaluate the accuracy in detecting adversarial attacks for all scenarios listed in Table 1.
Revised: We then evaluate the accuracy in detecting adversarial attacks for all scenarios listed in Table 1.

Original: More specifically, at every time step, the victim uses the learned world model ˆ m v to predict the next observation given the previous observation and action taken.
Revised: More specifically, at every time step, the victim uses the learned world model ˆ m v to predict the next observation given the previous observation and action taken.

Original: If the l 0 distance between the predicted observation and the actual (adversarial) observation is larger than the threshold c , the victim classifies the episode as adversarially attacked.
Revised: If the l 0 distance between the predicted observation and the actual (adversarial) observation is larger than the threshold c , the victim classifies the episode as adversarially attacked.

Original: Results for this detection mechanism are given in Table 1.
Revised: Results for this detection mechanism are given in Table 1.

Original: In the six-cell gridworld environment depicted in Figure 1, we similarly evaluated detection accuracy, but only considered naive victim agents.
Revised: In the six-cell gridworld environment depicted in Figure 1, we similarly evaluated detection accuracy, but only considered naive victim agents.

Original: We found that 100% of adversarial attacks could be detected in this environment, while 0% of illusory attacks were detected.
Revised: We found that 100% of adversarial attacks could be detected in this environment, while 0% of illusory attacks were detected.

Original: We show the resulting sequences of observations and true states in Figure 1.
Revised: We show the resulting sequences of observations and true states in Figure 1.

Original: 7. P ERFECT ILLUSORY ATTACKS
Revised: 7. P ERFECT ILLUSORY ATTACKS

Original: We implemented perfect illusory attacks in Pendulum and CartPole by setting the first observation zto the negative of the true first state, i.e. z 0 = − s 0 .
Revised: We implemented perfect illusory attacks in Pendulum and CartPole by setting the first observation zto the negative of the true first state, i.e. z 0 = − s 0 .

Original: We compute all subsequent observations z 1 , ..., z T with the worldmodel m a .
Revised: We compute all subsequent observations z 1 , ..., z T with the worldmodel m a .

Original: As the distribution over initial states is symmetric in both environments, this approach of generating illusory attacks satisfies the conditions of a perfect illusory attack (see
Revised: As the distribution over initial states is symmetric in both environments, this approach of generating illusory attacks satisfies the conditions of a perfect illusory attack (see

Original: Definition 4.2).
Revised: Definition 4.2).

Original: We provide videos of the generated perfect illusory attacks in the supplementary material in the respective folder.
Revised: We provide videos of the generated perfect illusory attacks in the supplementary material in the respective folder.

Original: We found that the average reward achieved by the victim under a perfect illusory attack in CartPole was 41 .
Revised: We found that the average reward achieved by the victim under a perfect illusory attack in CartPole was 41 .

Original: 2 ± 6 . 8 , while it was − 628 .
Revised: 2 ± 6 . 8 , while it was − 628 .

Original: 32 ± 60 . 1 in Pendulum.
Revised: 32 ± 60 . 1 in Pendulum.

Original: Figure 2 illustrates a perfect illusory attack in CartPole.
Revised: Figure 2 illustrates a perfect illusory attack in CartPole.

Original: 7. L EARNING W- ILLUSORY ATTACKS WITH REINFORCEMENT LEARNING
Revised: 7. L EARNING W- ILLUSORY ATTACKS WITH REINFORCEMENT LEARNING

Original: We next describe the algorithm used to learn W -illusory adversarial attacks and the training procedures used to compute the results in Table 1.
Revised: We next describe the algorithm used to learn W -illusory adversarial attacks and the training procedures used to compute the results in Table 1.

Original: Setup.
Revised: Setup.

Original: We use the CartPole and Pendulum gym (Brockman et al., 2016) environments.
Revised: We use the CartPole and Pendulum gym (Brockman et al., 2016) environments.

Original: The world model is implemented using the given physics computed for a new environment step.
Revised: The world model is implemented using the given physics computed for a new environment step.

Original: We normalise observations in both environments by the maximum absolute observation.
Revised: We normalise observations in both environments by the maximum absolute observation.

Original: We train the victim with
Revised: We train the victim with

Original: PPO (Schulman et al., 2017) and use the implementation of PPO given in Raffin et al. (2021), while not making any changes to the given hyperparameters.
Revised: PPO (Schulman et al., 2017) and use the implementation of PPO given in Raffin et al. (2021), while not making any changes to the given hyperparameters.

Original: In both environments we train the naive victim for 1 million environment steps.
Revised: In both environments we train the naive victim for 1 million environment steps.

Original: We implement the ATLA (Zhang et al., 2021) victim by co-training it with an adversary agent, and follow the original implementation of the authors 2 .
Revised: We implement the ATLA (Zhang et al., 2021) victim by co-training it with an adversary agent, and follow the original implementation of the authors 2 .

Original: We implement the ablation of ATLA (Zhang et al., 2021) that trains the victim with an illusory adversary by merely replacing the ATLA adversary with an illusory adversary, which was implemented as stated in algorithm 7.5.
Revised: We implement the ablation of ATLA (Zhang et al., 2021) that trains the victim with an illusory adversary by merely replacing the ATLA adversary with an illusory adversary, which was implemented as stated in algorithm 7.5.

Original: For co-training, we alternate between training the victim and the adversary agent every 400 environment steps.
Revised: For co-training, we alternate between training the victim and the adversary agent every 400 environment steps.

Original: This parameter has likewise been found in a small evaluation study, and was chosen as it yields non-oscillating behaviour.
Revised: This parameter has likewise been found in a small evaluation study, and was chosen as it yields non-oscillating behaviour.

Original: We further investigated different ratios between training steps of the adversary and training steps of the victim, but found that a ratio of one, i.e. equal training of both, yields the most stable and results in co-training.
Revised: We further investigated different ratios between training steps of the adversary and training steps of the victim, but found that a ratio of one, i.e. equal training of both, yields the most stable and results in co-training.

Original: We implement the illusory adversary agent with SAC (Haarnoja et al., 2018), where we likewise use the implementation given in Raffin et al.
Revised: We implement the illusory adversary agent with SAC (Haarnoja et al., 2018), where we likewise use the implementation given in Raffin et al.

Original: We initially ran a small study and investigated four different algorithms as possible implementations for the adversary agent, where we found that SAC yields best performance and training stability.
Revised: We initially ran a small study and investigated four different algorithms as possible implementations for the adversary agent, where we found that SAC yields best performance and training stability.

Original: We train all adversarial attacks for three million environment steps.
Revised: We train all adversarial attacks for three million environment steps.

Original: We implemented randomized smoothing as a standard defence against adversarial attacks on RL agents, as introduced in Kumar et al.
Revised: We implemented randomized smoothing as a standard defence against adversarial attacks on RL agents, as introduced in Kumar et al.

Original: We use the author’s original implementation 3 .
Revised: We use the author’s original implementation 3 .

Original: In contrast to the authors, we compute the budget with respect to the normalised observations, but otherwise follow the per-step budget definition in Kumar et al.
Revised: In contrast to the authors, we compute the budget with respect to the normalised observations, but otherwise follow the per-step budget definition in Kumar et al.

Original: Input: environment env , adversary worldmodel m a , illusory reward weighing parameter λ , victim policy π v , iterations N
Revised: Input: environment env , adversary worldmodel m a , illusory reward weighing parameter λ , victim policy π v , iterations N

Original: Initialize π ψadv , parametrised be a neural network with parameters ψ .
Revised: Initialize π ψadv , parametrised be a neural network with parameters ψ .

Original: while iteration <
Revised: while iteration <

Original: N do s 

Revised: N do s

Original: = env (
Revised: = env (

Original: ) z 

Revised: ) z

Original: = π ψadv ( s 0 )
Revised: = π ψadv ( s 0 )

Original: ▷ Initial perturbation only depends on initial system state a 0 =
Revised: ▷ Initial perturbation only depends on initial system state a 0 =

Original: π v ( clip ( z 0 ))
Revised: π v ( clip ( z 0 ))

Original: ▷ Adversarial observation is clipped to permissible range s 1 , r v 1 , done = env ( a 0 )
Revised: ▷ Adversarial observation is clipped to permissible range s 1 , r v 1 , done = env ( a 0 )

Original: r adv= − r v▷
Revised: r adv= − r v▷

Original: No illusory reward at first step while done=
Revised: No illusory reward at first step while done=

Original: False do z t =
Revised: False do z t =

Original: π ψadv ( s t , z t − 1 , a t − 1 )
Revised: π ψadv ( s t , z t − 1 , a t − 1 )

Original: ▷ Adversary conditions on past information a t = π v ( clip ( z t )) s t +1 , r vt +1 , done =
Revised: ▷ Adversary conditions on past information a t = π v ( clip ( z t )) s t +1 , r vt +1 , done =

Original: env ( a t ) r advt +1 = − r vt +1 − λ · ∥ z t ; m a ( z t − 1 , a t − 1 ) ∥ ∞ ▷ Illusory reward added end while update π ψadv ▷ Adversary policy updated using past experience end while 7.5.
Revised: env ( a t ) r advt +1 = − r vt +1 − λ · ∥ z t ; m a ( z t − 1 , a t − 1 ) ∥ ∞ ▷ Illusory reward added end while update π ψadv ▷ Adversary policy updated using past experience end while 7.5.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Table 7.5.1 shows all results for all scenarios evaluated; we report mean and standard deviation.
Revised: Table 7.5.1 shows all results for all scenarios evaluated; we report mean and standard deviation.

Original: Note that each scenario was evaluated for five random seeds; Table 1 in the main paper is derived from
Revised: Note that each scenario was evaluated for five random seeds; Table 1 in the main paper is derived from

Original: Table 7.5.1.
Revised: Table 7.5.1.

Original: 7.5. A BLATION : I NFLUENCE OF LAMBDA ON RESULTS
Revised: 7.5. A BLATION : I NFLUENCE OF LAMBDA ON RESULTS

Original: We investigated the influence of the λ parameter in a small study.
Revised: We investigated the influence of the λ parameter in a small study.

Original: Generally, the λ parameter determines the trade off of the illusory adversary between generating consistent observation sequences, and minimizing the victim’s reward.
Revised: Generally, the λ parameter determines the trade off of the illusory adversary between generating consistent observation sequences, and minimizing the victim’s reward.

Original: We considered λ ∈ { 1 , 10 , 100 } , with β = 0 . 2 .
Revised: We considered λ ∈ { 1 , 10 , 100 } , with β = 0 . 2 .

Original: Table 4 shows that smaller weighing parameters
Revised: Table 4 shows that smaller weighing parameters

Original: λ generally lead to lower victim rewards, which is in line with the fact that for λ = 0 , the objective of the adversary is solely to minimise the victim reward.
Revised: λ generally lead to lower victim rewards, which is in line with the fact that for λ = 0 , the objective of the adversary is solely to minimise the victim reward.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: V IDEOS OF ALL ADVERSARIAL ATTACKS
Revised: V IDEOS OF ALL ADVERSARIAL ATTACKS

Original: We provide videos for different seeds for all adversarial attacks in the supplementary material.
Revised: We provide videos for different seeds for all adversarial attacks in the supplementary material.

Original: The folders are named respectively.
Revised: The folders are named respectively.

Original: Note that the left side shows the environment as observed by the victim agent, while the right side shows the true environment state.
Revised: Note that the left side shows the environment as observed by the victim agent, while the right side shows the true environment state.

Original: Please further note that the episode in CartPole terminates once the angle has crossed a specific threshold
Revised: Please further note that the episode in CartPole terminates once the angle has crossed a specific threshold

Original: [Table]
Revised: [Table]

Original: which is why the episodes might appear to be shorter than otherwise expected.
Revised: which is why the episodes might appear to be shorter than otherwise expected.

Original: All videos were generated for a budget β = 0 .
Revised: All videos were generated for a budget β = 0 .

Original: 7. H UMAN STUDY
Revised: 7. H UMAN STUDY

Original: We performed a controlled experiment with n = 10 human participants.
Revised: We performed a controlled experiment with n = 10 human participants.

Original: All participants were graduate-level university students, while none had prior knowledge about the objective of the study.
Revised: All participants were graduate-level university students, while none had prior knowledge about the objective of the study.

Original: Participants participated voluntarily; we estimate the time needed per participant was aroundminutes.
Revised: Participants participated voluntarily; we estimate the time needed per participant was aroundminutes.

Original: Participants were handed a slide show which contained all relevant information.
Revised: Participants were handed a slide show which contained all relevant information.

Original: This slide show is included in the supplementary material in the respective folder.
Revised: This slide show is included in the supplementary material in the respective folder.

Original: We further add the sheet with ground truth labels for all video sequences.
Revised: We further add the sheet with ground truth labels for all video sequences.

Original: After consenting to participate, participants were provided with the slide show and an online sheet to indicate their answers.
Revised: After consenting to participate, participants were provided with the slide show and an online sheet to indicate their answers.

Original: The study was self-paced and done by the participants without further supervision.
Revised: The study was self-paced and done by the participants without further supervision.

Original: We further advised participants not to do the study if they would not feel comfortable at any point in time.
Revised: We further advised participants not to do the study if they would not feel comfortable at any point in time.

Original: The first 18 videos in the slide show are 100-frame long sequences of the Pendulum environment, i.e. six each for either no adversarial attack, a W -illusory attack or an SA-MDP attack (Zhang et al., 2021).
Revised: The first 18 videos in the slide show are 100-frame long sequences of the Pendulum environment, i.e. six each for either no adversarial attack, a W -illusory attack or an SA-MDP attack (Zhang et al., 2021).

Original: The subsequent 24 10-frame long sequences of the CartPole environment include the same attacks as for Pendulum, plus an additional six sequences under an MNP attack (Kumar et al., 2021).
Revised: The subsequent 24 10-frame long sequences of the CartPole environment include the same attacks as for Pendulum, plus an additional six sequences under an MNP attack (Kumar et al., 2021).

Original: Note that MNP attacks cannot be applied to the Pendulum environment, as it has a continuous action space.
Revised: Note that MNP attacks cannot be applied to the Pendulum environment, as it has a continuous action space.

Original: For all attacks, the budget β was 0.2.
Revised: For all attacks, the budget β was 0.2.

Original: Within both groups, the order of the videos shown was randomised.
Revised: Within both groups, the order of the videos shown was randomised.

Original: Subjects had to choose for each video between correct and false , where false indicates that subjects find that the system observed in the respective video is not the same as the one observed in the introductory video.
Revised: Subjects had to choose for each video between correct and false , where false indicates that subjects find that the system observed in the respective video is not the same as the one observed in the introductory video.

Original: Table 5 includes all empirically estimated probabilities of classifying a segment from a given class as false , split up also by the different environments CartPole and Pendulum.
Revised: Table 5 includes all empirically estimated probabilities of classifying a segment from a given class as false , split up also by the different environments CartPole and Pendulum.

Original: Result analysis.
Revised: Result analysis.

Original: We find that subjects reported an unexpectedly high proportion (one third) of unattacked sequences as false .
Revised: We find that subjects reported an unexpectedly high proportion (one third) of unattacked sequences as false .

Original: Subjects classified sequences which were under an MNP or SAMDP (Zhang et al., 2020) attack as not from the same system with a probability higher than 80%, while categorizing illusory attacks as false with a probability of only 32.4%.
Revised: Subjects classified sequences which were under an MNP or SAMDP (Zhang et al., 2020) attack as not from the same system with a probability higher than 80%, while categorizing illusory attacks as false with a probability of only 32.4%.

Original: We computed the double-sided z-statistics and were able to reject both the hypothesis that p ( false | SA-MDP ) =
Revised: We computed the double-sided z-statistics and were able to reject both the hypothesis that p ( false | SA-MDP ) =

Original: p ( false | none ) and the hypothesis that p ( false | MNP ) =
Revised: p ( false | none ) and the hypothesis that p ( false | MNP ) =

Original: p ( false | none ) for α = 0 .
Revised: p ( false | none ) for α = 0 .

Original: 05 , while the hypothesis that p ( false | W -illusory ) =
Revised: 05 , while the hypothesis that p ( false | W -illusory ) =

Original: p ( false | none ) cannot be rejected.
Revised: p ( false | none ) cannot be rejected.

Original: We conclude that subjects were able
Revised: We conclude that subjects were able

Original: to
Revised: to

Original: [Table]
Revised: [Table]

Original: distinguish SA-MDP and MNP attacks from unattacked sequences, while being unable to distinguish illusory attacks from unattacked sequences.
Revised: distinguish SA-MDP and MNP attacks from unattacked sequences, while being unable to distinguish illusory attacks from unattacked sequences.

Original: 7. R EALITY FEEDBACK
Revised: 7. R EALITY FEEDBACK

Original: Setup.
Revised: Setup.

Original: We evaluate the importance of realism feedback in the CartPole environment by investigating two possible scenarios.
Revised: We evaluate the importance of realism feedback in the CartPole environment by investigating two possible scenarios.

Original: Note that the observation in CartPole is given as a four-dimensional vector of the pole angle and angular velocity, as well as cart position and velocity.
Revised: Note that the observation in CartPole is given as a four-dimensional vector of the pole angle and angular velocity, as well as cart position and velocity.

Original: In the first test scenario, the victim correctly observes the pole, while the adversary can perturb the observation of the cart; the second scenario is vice versa.
Revised: In the first test scenario, the victim correctly observes the pole, while the adversary can perturb the observation of the cart; the second scenario is vice versa.

Original: We investigate two test cases for each scenario:
Revised: We investigate two test cases for each scenario:

Original: First, attacking a naive victim, and second, attacking an agent pretrained with co-training.
Revised: First, attacking a naive victim, and second, attacking an agent pretrained with co-training.

Original: Results and discussion.
Revised: Results and discussion.

Original: Table 6 shows that the reward achieved by the victim is generally higher when pretrained with co-training.
Revised: Table 6 shows that the reward achieved by the victim is generally higher when pretrained with co-training.

Original: We hypothesize that this pretraining enables the agent to learn how to utilize the reality feedback effectively.
Revised: We hypothesize that this pretraining enables the agent to learn how to utilize the reality feedback effectively.

Original: The achieved victim performance when reality feedback contains information about the pole is more than 10 times larger than when containing information on the cart instead.
Revised: The achieved victim performance when reality feedback contains information about the pole is more than 10 times larger than when containing information on the cart instead.

Original: This seems intuitive, as the observation of the pole appears much more useful for the task of stabilizing the pole, and underlines the importance of equipping agents with strong reality feedback channels.
Revised: This seems intuitive, as the observation of the pole appears much more useful for the task of stabilizing the pole, and underlines the importance of equipping agents with strong reality feedback channels.