Original: Provable Defense against Backdoor Policies in Reinforcement Learning
Revised: Provable Defense against Backdoor Policies in Reinforcement Learning

Original: Abstract
Revised: Abstract

Original: We propose a provable defense mechanism against backdoor policies in reinforcement learning under subspace trigger assumption.
Revised: We propose a provable defense mechanism against backdoor policies in reinforcement learning under subspace trigger assumption.

Original: A backdoor policy is a security threat where an adversary publishes a seemingly well-behaved policy which in fact allows hidden triggers.
Revised: A backdoor policy is a security threat where an adversary publishes a seemingly well-behaved policy which in fact allows hidden triggers.

Original: During deployment, the adversary can modify observed states in a particular way to trigger unexpected actions and harm the agent.
Revised: During deployment, the adversary can modify observed states in a particular way to trigger unexpected actions and harm the agent.

Original: We assume the agent does not have the resources to re-train a good policy.
Revised: We assume the agent does not have the resources to re-train a good policy.

Original: Instead, our defense mechanism sanitizes the backdoor policy by projecting observed states to a ‘safe subspace’, estimated from a small number of interactions with a clean (non-triggered) environment.
Revised: Instead, our defense mechanism sanitizes the backdoor policy by projecting observed states to a ‘safe subspace’, estimated from a small number of interactions with a clean (non-triggered) environment.

Original: Our sanitized policy achieves ϵ approximate optimality in the presence of triggers, provided the number of clean interactions isO (cid:16) D
Revised: Our sanitized policy achieves ✏ approximate optimality in the presence of triggers, provided the number of clean interactions is

Original: [Equation]
Revised: [Equation]

Original: γ [Equation] (cid:17) where γ is the discounting factor and D is the dimension of state space.
Revised: ⌘ where  is the discounting factor and D is the dimension of state space.

Original: Empirically, we show that our sanitization defense performs well on two
Revised: Empirically, we show that our sanitization defense performs well on two

Original: Atari game environments.
Revised: Atari game environments.

Original:  Motivation
Revised:  Motivation

Original: The success of reinforcement learning (RL) brings up a new security issue: Often the task is so complex that it takes considerable amount of resources to train a good policy.
Revised: The success of reinforcement learning (RL) brings up a new security issue: Often the task is so complex that it takes considerable amount of resources to train a good policy.

Original: Such resources are increasingly restricted to large corporations or nation states.
Revised: Such resources are increasingly restricted to large corporations or nation states.

Original: Consequently, we imagine in the future many users of RL have to be content with obtaining pretrained policies, without the possibility to (re)train the policy themselves.
Revised: Consequently, we imagine in the future many users of RL have to be content with obtaining pretrained policies, without the possibility to (re)train the policy themselves.

Original: Downloading a pretrained RL policy from an untrusted party opens up a new attack surface known as backdoor policy attacks.
Revised: Downloading a pretrained RL policy from an untrusted party opens up a new attack surface known as backdoor policy attacks

Original: In a backdoor policy attack an adversary prepares a pair ( π † , f ) where π † is a backdoor policy and f is a trigger function.
Revised: In a backdoor policy attack, an adversary prepares a pair ( ⇡ † , f ) where ⇡ † is a backdoor policy and f is a trigger function.

Original: The adversary publishes the backdoor policy π † to be downloaded and used by interested users.
Revised: The adversary publishes the backdoor policy ⇡ † to be downloaded and used by interested users.

Original: π † behaves like an optimal policy under normal deployment, until it is triggered.
Revised: ⇡ † behaves like an optimal policy under normal deployment, until it is triggered.

Original: A naive user, interested in performing well in the underlying RL problem, downloads the backdoor policy π † unaware of the fact that adversary can activate the trigger to make the policy perform poorly in the adversarial environment.
Revised: A naive user, interested in performing well in the underlying RL problem, downloads the backdoor policy ⇡ † unaware of the fact that adversary can activate the trigger to make the policy perform poorly in the adversarial environment.

Original: For example, π † may be the driving policy for an autonomous car.
Revised: For example, ⇡ † may be the driving policy for an autonomous car.

Original: The car will drive normally under π † until the attacker puts up a special sticker somewhere within the car’s camera view.
Revised: The car will drive normally under ⇡ † until the attacker puts up a special sticker somewhere within the car’s camera view.

Original: The sticker acts as a trigger; when the trigger is present, the policy π † produces undesirable actions such as crashing the car.
Revised: The sticker acts as a trigger; when the trigger is present, the policy ⇡ † produces undesirable actions such as crashing the car.

Original: Backdoor attacks are well-studied in supervised learning, where instead of a policy π † it is a prediction model that contains built-in backdoors that can be triggered to make wrong predictions [4][11][13].
Revised: Backdoor attacks are well-studied in supervised learning, where instead of a policy ⇡ † it is a prediction model that contains built-in backdoors that can be triggered to make wrong predictions [4][11][13].

Original: Backdoor policy attack in
Revised: Backdoor policy attack in

Original: RL brings additional attack opportunities in that the adversary can plan the attack sequentially: the immediate reward of a triggered round can even be good, as long as its state transition leads to much worst long-term rewards.
Revised: RL brings additional attack opportunities in that the adversary can plan the attack sequentially: the immediate reward of a triggered round can even be good, as long as its state transition leads to much worst long-term rewards.

Original: Initial empirical work has shown backdoor policy attack is a valid security concern [6][7][19], but there has been neither formal attack specification nor provable defense.
Revised: Initial empirical work has shown backdoor policy attack is a valid security concern [6][7][19], but there has been neither formal attack speciﬁcation nor provable defense.

Original: This paper makes the following contributions: 1.
Revised: This paper makes the following contributions: 1.

Original: We formally define backdoor policies and associated trigger function in RL called “subspace trigger”.
Revised: We formally deﬁne backdoor policies and associated trigger function in RL called “subspace trigger”.

Original: We present a defense algorithm with performance guarantees against all subspace trigger based adversaries.
Revised: We present a defense algorithm with performance guarantees against all subspace trigger based adversaries.

Original: We empirically verify the performance of our sanitization algorithm on two Atari game environments.
Revised: We empirically verify the performance of our sanitization algorithm on two Atari game environments.

Original: Our defense algorithm does not require retraining of the policy – which is impractical for the user resources we envision.
Revised: Our defense algorithm does not require retraining of the policy – which is impractical for the user resources we envision.

Original: Instead, it is a wrapper method around the backdoor policy, rendering it insensitive to triggers and hence harmless.
Revised: Instead, it is a wrapper method around the backdoor policy, rendering it insensitive to triggers and hence harmless.

Original: For this reason we call our defense “sanitization”.
Revised: For this reason we call our defense “sanitization”.

Original: The key idea is to project potentially triggered states to a safe subspace.
Revised: The key idea is to project potentially triggered states to a safe subspace.

Original: In this safe subspace, the backdoor policy behaves like an optimal policy.
Revised: In this safe subspace, the backdoor policy behaves like an optimal policy.

Original: We estimate the safe subspace with relatively cheap Singular
Revised: We estimate the safe subspace with relatively cheap Singular

Original: Value Decomposition, under the assumption that the user can deploy the backdoor policy in an assured no-trigger environment for some episodes.
Revised: Value Decomposition, under the assumption that the user can deploy the backdoor policy in an assured no-trigger environment for some episodes.

Original:  Related Work
Revised:  Related Work

Original: The vast majority of backdoor attack literature targets supervised learning.
Revised: The vast majority of backdoor attack literature targets supervised learning.

Original: Some work requires original training data and expensive model retraining [16] [11] [8] [12] [3] [10] [1] which we avoid.
Revised: Some work requires original training data and expensive model retraining [16] [11] [8] [12] [3] [10] [1] which we avoid.

Original: Some require surgical modification to the whitebox backdoor model [20] [4] [9] while our method is a wrapper.
Revised: Some require surgical modiﬁcation to the whitebox backdoor model [20] [4] [9] while our method is a wrapper.

Original: We also do not attempt to reverse engineer the trigger as done in [18].
Revised: We also do not attempt to reverse engineer the trigger as done in [18].

Original: Our method is in spirit closely related to [12], which uses autoencoder to find the equivalent to our “safe subspace”.
Revised: Our method is in spirit closely related to [12], which uses autoencoder to ﬁnd the equivalent to our “safe subspace”.

Original: The connection is not incidental: our SVD is a special case of autoencoder.
Revised: The connection is not incidental: our SVD is a special case of autoencoder.

Original: Nonetheless, we study backdoor policies in RL, where the key difference is that damage is inflicted via long term return instead of immediate reward while their work is in supervised learning setting and is empirical.
Revised: Nonetheless, we study backdoor policies in RL, where the key difference is that damage is inﬂicted via long term return instead of immediate reward while their work is in supervised learning setting and is empirical.

Original: The study on backdoor policies in RL is only recently emerging [7][6][19].
Revised: The study on backdoor policies in RL is only recently emerging [7][6][19].

Original: Most of these work are again empirical, while we provide formal guarantees.
Revised: Most of these work are again empirical, while we provide formal guarantees.

Original: Some of the recent works have also studied backdoor attacks in multi-agent RL setting - [19] proposed a backdoor attack where the backdoor behavior is triggered through a specific sequence of actions by the opponent which is clearly different from our setting where the attacker injects trigger directly in the state space.
Revised: Some of the recent works have also studied backdoor attacks in multi-agent RL setting - [19] proposed a backdoor attack where the backdoor behavior is triggered through a speciﬁc sequence of actions by the opponent which is clearly different from our setting where the attacker injects trigger directly in the state space.

Original: A follow-up work [5] proposed an empirical strategy to detect backdoor action sequence triggers in this setting which is also different from ours.
Revised: A follow-up work [5] proposed an empirical strategy to detect backdoor action sequence triggers in this setting which is also different from ours.

Original:  A Formal Definition of Backdoor Policies in RL
Revised:  A Formal Deﬁnition of Backdoor Policies in RL

Original: Let M =
Revised: Let M =

Original: ( S , A , P, R, µ, γ ) denote the environment MDP with a continuous state space S
Revised: ( S , A , P, R, µ,  ) denote the environment MDP with a continuous state space S

Original: = R D , discrete action space A , a transition function P : S ×
Revised: = R D , discrete action space A , a transition function P : S ⇥ A !

Original: A → ∆( S ) , a reward function R : S ×
Revised:  ( S ) , a reward function R : S ⇥ A !

Original: A → [0 , 1] , an initial state distribution µ and a discounting factor γ ∈ [0 , 1) .
Revised: [0 , 1] , an initial state distribution µ and a discounting factor  2 [0 , 1) .

Original: The objective of the agent is to find a policy that achieves the maximum value under the given MDP M ,
Revised: The objective of the agent is to ﬁnd a policy that achieves the maximum value under the given MDP M ,

Original: [Equation]
Revised: [Equation]

Original: (1) where V π M = E [ (cid:80) ∞ t =

Revised: [Equation] ) where V ⇡ M = E [ P 1 t =

Original: γ t R ( s t , π ( s t ))] is the expected discounted cumulative rewards of following policy π under MDP M .
Revised:  t R ( s t , ⇡ ( s t ))] is the expected discounted cumulative rewards of following policy ⇡ under MDP M .

Original: The attacker first chooses an optimal policy π ∗ with a discounted state occupancy under M given as d π ∗ M ( s ) =
Revised: The attacker ﬁrst chooses an optimal policy ⇡ ⇤ with a discounted state occupancy under M given as d ⇡ ⇤ M ( s ) =

Original: (1 − γ )(cid:80) ∞
Revised: (1   )

Original:  t =0 γ{+=
 +} t P ( s t = s | s 0 ∼ µ, π ∗ ) .
Revised: P 1 t {+=
 +} t P ( s t = s | s 0 ⇠ µ, ⇡ ⇤ ) .

Original: Let T ⊂ S denote the support of d π ∗ M , i.e. T is the smallest closed subset of R D s.t.
Revised: Let T ⇢ S denote the support of d ⇡ ⇤ M , i.e. T is the smallest closed subset of R D s.t.

Original: P ( T )
Revised: P ( T )

Original: = 1 .
Revised: = 1 .

Original: For simplicity, we assume that E s ∼ d π ∗M [ s ] = 0 .
Revised: For simplicity, we assume that E s ⇠ d ⇡ ⇤ M [ s ] = 0 .

Original: This assumption keeps the analysis clean but our algorithm and its results can be directly extended to the non-zero mean case.
Revised: This assumption keeps the analysis clean but our algorithm and its results can be directly extended to the non-zero mean case.

Original: Now, consider the eigen decomposition of the state covariance matrix Σ under d π ∗ M , Σ
Revised: Now, consider the eigen decomposition of the state covariance matrix ⌃ under d ⇡ ⇤ M , ⌃ =

Original: = E s ∼ d π ∗M [ ss ⊤ ] =
Revised: E s ⇠ d ⇡ ⇤ M [ ss > ] =

Original: D(cid:88) i
Revised: D

Original:  =λ
Revised: X i =

Original:  i u i u ⊤ i where λ 1 ≥ · · · λ d ≥ λ d +1 ≥ · · · ≥ λ D .
Revised:  i u i u > i where  1  · · ·  d   d +1  · · ·   D .

Original: [Figure]
Revised: [Figure]

Original: We denote by E = span ( { u i }| di =1 ) the top d eigen subspace of Σ and E ⊥ = span ( { u i }| Di = d +1 ) is its complement.
Revised: We denote by E = span ( { u i }| di =1 ) the top d eigen subspace of ⌃ and E ? = span ( { u i }| Di = d +1 ) is its complement.

Original: The projection operator into E, E ⊥ is given by Proj E 
Revised: The projection operator into E, E ? is given by Proj E =

Original: = (cid:80) di

Original:  =1 u i u ⊤ i , Proj ⊥ E 
Revised: P di =1 u i u > i , Proj ? E =

Original: = (cid:80)
Revised: P

Original: Di = d +1 u i u ⊤ i respectively, see Figure 1.
Revised: Di = d +1 u i u > i respectively, see Figure 1.

Original: We call E the safe subspace of the state space (to be explained in Assumption 3).
Revised: We call E the safe subspace of the state space (to be explained in Assumption 3).

Original: Assumption 1 The occupancy distribution d π ∗ M has a bounded support along the smallest D − d eigen-subspace 
Revised: Assumption 1 The occupancy distribution d ⇡ ⇤ M has a bounded support along the smallest D  d eigen-subspace E ?

Original: E ⊥ i.e. ∃ C 0 ∈ R s.t.
Revised: i.e. 9 C 0 2 R s.t.

Original: under d π ∗ M , P ( { s ∈ S : ∥ Proj E ⊥ ( s ) ∥ 2 ≤ C 0 } )
Revised: under d ⇡ ⇤ M , P ( { s 2 S : k Proj E ? ( s ) k 2  C 0 } )

Original: = 1 .
Revised: = 1 .

Original: Further, we assume that d π ∗ M is K -subGaussian.
Revised: Further, we assume that d ⇡ ⇤ M is K -subGaussian.

Original: In many real world examples, states are often bounded along standard basis directions so this assumption is reasonable.
Revised: In many real world examples, states are often bounded along standard basis directions so this assumption is reasonable.

Original: It gives the attacker the flexibility to insert into π † arbitrary backdoor behaviors on states outside the bounded support T while preserving the performance of optimal policy inside T .
Revised: It gives the attacker the ﬂexibility to insert into ⇡ † arbitrary backdoor behaviors on states outside the bounded support T while preserving the performance of optimal policy inside T .

Original: Backdoor Policy π †
Revised: Backdoor Policy ⇡ †

Original: The attacker then constructs a backdoor policy π † , which behaves the same as π ∗ everywhere in the support T of d π ∗ M , but outside this support π † can differ arbitrarily.
Revised: The attacker then constructs a backdoor policy ⇡ † , which behaves the same as ⇡ ⇤ everywhere in the support T of d ⇡ ⇤ M , but outside this support ⇡ † can differ arbitrarily.

Original: However, we assume that overall π † is L -Lipschitz.
Revised: However, we assume that overall ⇡ † is L -Lipschitz.

Original: Assumption 2 The backdoor policy π † is L -Lipschitz smooth i.e. 
Revised: Assumption 2 The backdoor policy ⇡ † is L -Lipschitz smooth i.e. 9 L 2 R s.t. 8 s [Equation] s

Original: [Equation]
Revised: [Equation]

Revised: S , k ⇡ † ( s )  ⇡

Revised: [Equation]

Original:  In practice, people often use gradient descent trained neural network policies and it is reasonable to assume them to be Lipschitz smooth [2] [14].
Revised: ( s [Equation] k 1  L k s  s 0 k 2 . In practice, people often use gradient descent trained neural network policies and it is reasonable to assume them to be Lipschitz smooth [2] [14].

Original: By letting π † = π ∗ on T , the support of d π ∗ M , the attacker remains stealthy in a clean (non-triggered) environment and can coax the agent into believing that the backdoor policy π † is a good one.
Revised: By letting ⇡ † = ⇡ ⇤ on T , the support of d ⇡ ⇤ M , the attacker remains stealthy in a clean (non-triggered) environment and can coax the agent into believing that the backdoor policy ⇡ † is a good one.

Original: Subspace Trigger function f
Revised: Subspace Trigger function f

Original: In tandem with the backdoor policy π † , the attacker also prepares a trigger function f : ( S ) ∗ → E ⊥ to be used during deployment, where ( S ) ∗ is the set of all possible state sequence in the history.
Revised: In tandem with the backdoor policy ⇡ † , the attacker also prepares a trigger function f : ( S ) ⇤ ! E ? to be used during deployment, where ( S ) ⇤ is the set of all possible state sequence in the history.

Original: Importantly, such a trigger function restricts the adversary to only inject triggers in E ⊥ subspace and hence we call it “subspace trigger” function.
Revised: Importantly, such a trigger function restricts the adversary to only inject triggers in E ? subspace and hence we call it “subspace trigger” function.

Original: Also, note that f is adaptive in that it can depend on the entire history.
Revised: Also, note that f is adaptive in that it can depend on the entire history.

Original: For simplicity, we consider f that only depends on history of states but it can be trivially generalized to include history of actions and rewards as well.
Revised: For simplicity, we consider f that only depends on history of states but it can be trivially generalized to include history of actions and rewards as well.

Original: At any time step t during deployment where the agent uses π † , the adversary can inject a trigger f ( s 0: t ) (“the sticker”) into the current state s t so that the agent perceives a triggered state s † t = s t + f ( s 0: t ) .
Revised: At any time step t during deployment where the agent uses ⇡ † , the adversary can inject a trigger f ( s 0: t ) (“the sticker”) into the current state s t so that the agent perceives a triggered state s † t = s t + f ( s 0: t ) .

Original: Consequently, the agent will take a potentially harmful action a † t ∼ π † ( s † t ) at state s t .
Revised: Consequently, the agent will take a potentially harmful action a † t ⇠ ⇡ † ( s † t ) at state s t .

Original: We emphasize that the trigger only applies to the agent’s perceived state, i.e. the underlying MDP state is still at s t .
Revised: We emphasize that the trigger only applies to the agent’s perceived state, i.e. the underlying MDP state is still at s t .

Original: As interaction protocol 1 specifies, transition and reward happen according to ( s t , a † t ) , not ( s † t , a † t ) .
Revised: As interaction protocol 1 speciﬁes, transition and reward happen according to ( s t , a † t ) , not ( s † t , a † t ) .

Original: By carefully designing f and coupling the behavior of backdoor policy on the triggered states, the adversary can either drastically decrease the value the agent receives, or force the agent to perform a nefarious action sequence.
Revised: By carefully designing f and coupling the behavior of backdoor policy on the triggered states, the adversary can either drastically decrease the value the agent receives, or force the agent to perform a nefarious action sequence.

Original: Given an agent using a backdoor policy π † , and an adversary with a trigger function f , the interaction between them in an environment (possibly backdoored) takes place as follows:
Revised: Given an agent using a backdoor policy ⇡ † , and an adversary with a trigger function f , the interaction between them in an environment (possibly backdoored) takes place as follows:

Original: Procedure 1 Interaction Protocol in a Backdoor Environment
Revised: Procedure 1 Interaction Protocol in a Backdoor Environment

Original: Entities: MDP environment M , RL agent accepted backdoor policy π from the adversary, adversary has trigger function 
Revised: Entities: MDP environment M , RL agent accepted backdoor policy ⇡ † from the adversary, adversary has trigger function f . 1: the environment draws s 0 ⇠ µ . 2: for t = 0 , 1 , · · · do 3: the adversary injects trigger f

Original: f 1: the environment draws s 0 ∼ µ . 2: for t = 0 , 1 , · · · do 3: the adversary injects trigger f ( s 0: t ) to state s t (adversary can choose to set f ( · ) = 

Revised: ( s 0: t ) to state s t (adversary can choose to set f ( · ) =

Original: meaning no trigger).
Revised: meaning no trigger).

Original: 4: the agent observes s † t =
Revised: 4: the agent observes s † t =

Original: s t + f ( s 0: t ) and takes the action a † t ∼ π † ( s † t ) .
Revised: s t + f ( s 0: t ) and takes the action a † t ⇠ ⇡ † ( s † t ) .

Original: 5: the environment evolves s t +1 ∼ P ( ·| s t , a † t ) , r t ∼ R ( s t , a † t )
Revised: 5: the environment evolves s t +1 ⇠ P ( ·| s t , a † t ) , r t ⇠ R ( s t , a † t )

Original: Assumption 3 The trigger function f : ( S ) ∗ → E ⊥ is adaptive and the adversary can only inject a trigger in the E ⊥ subspace of the state space S .
Revised: Assumption 3 The trigger function f : ( S ) ⇤ ! E ? is adaptive and the adversary can only inject a trigger in the E ? subspace of the state space S .

Original: Further, we assume that the perceived triggered states are B -bounded in expectation as below,
Revised: Further, we assume that the perceived triggered states are B -bounded in expectation as below,

Original: [Equation]
Revised: [Equation]

Original: ∥ [Equation] ] ≤ B where d π ◦ f, 0: t M denotes the distribution of partial state trajectory up to time step t under policy π and trigger function f .
Revised: where d ⇡  f, 0: t M denotes the distribution of partial state trajectory up to time step t under policy ⇡ and trigger function f .

Original: This assumption requires the adversary to keep the perceived triggered states to be B -bounded in expectation at every time step.
Revised: This assumption requires the adversary to keep the perceived triggered states to be B -bounded in expectation at every time step.

Original: Note that the adversary can still sometime inject large triggers as long as it keeps the average triggered states bounded.
Revised: Note that the adversary can still sometime inject large triggers as long as it keeps the average triggered states bounded.

Original: [Figure]
Revised: [Figure]

Original: As an example, we present a toy MDP in Figure 2 where states are in R 2 .
Revised: As an example, we present a toy MDP in Figure 2 where states are in R 2 .

Original: The initial state is always state F .
Revised: The initial state is always state F .

Original: There are two actions: left and right, and non-zero rewards are mentioned below an action.
Revised: There are two actions: left and right, and non-zero rewards are mentioned below an action.

Original: Transition is deterministic.
Revised: Transition is deterministic.

Original: Clearly, an optimal policy π ∗ is to take the left action at F and E which yields a zero immediate reward but γ/ (1 − γ ) long term return.
Revised: Clearly, an optimal policy ⇡ ⇤ is to take the left action at F and E which yields a zero immediate reward but  / (1   ) long term return.

Original: The x-axis is the safe subspace
Revised: The x-axis is the safe subspace

Original: E .
Revised: E .

Original: Note that the states A to C are non-reachable in the clean environment which gives the attacker leverage to hide the backdoor actions.
Revised: Note that the states A to C are non-reachable in the clean environment which gives the attacker leverage to hide the backdoor actions.

Original: The attacker’s backdoor policy π † is denoted by the orange arrows in the above diagram.
Revised: The attacker’s backdoor policy ⇡ † is denoted by the orange arrows in the above diagram.

Original: In particular, B is far enough from F so that the deterministic choice π † ( B ) = “right” is allowed by the Lipschitz condition.
Revised: In particular, B is far enough from F so that the deterministic choice ⇡ † ( B ) = “right” is allowed by the Lipschitz condition.

Original: In the backdoor environment, when the agent starts at state s 0 =
Revised: In the backdoor environment, when the agent starts at state s 0 =

Original: F , the adversary injects a trigger f ( F ) ∈ R 2 in upward direction along E ⊥ so that the agent perceives its state as s † [-
-] = F + f ( F ) =
Revised: F , the adversary injects a trigger f ( F ) 2 R 2 in upward direction along E ? so that the agent perceives its state as s † [-
-]0 = F + f ( F ) =

Original: B instead.
Revised: B instead.

Original: The agent will then choose action a † 0 = “right” as directed by its policy π † ( B ) .
Revised: The agent will then choose action a † 0 = “right” as directed by its policy ⇡ † ( B ) .

Original: However, the underlying MDP has its true state at F , so the agent moves to G .
Revised: However, the underlying MDP has its true state at F , so the agent moves to G .

Original: The agent may be happy that it received an immediate reward of 1 compared to 0 reward on left.
Revised: The agent may be happy that it received an immediate reward of 1 compared to 0 reward on left.

Original: However, the readers can see the agent is now doomed in the long term without further attacks.
Revised: However, the readers can see the agent is now doomed in the long term without further attacks.

Original: For γ close to 1, the optimal value γ 1 − γ in the clean environment can be arbitrarily large while the value under the above attack is 1.
Revised: For  close to 1, the optimal value  1   in the clean environment can be arbitrarily large while the value under the above attack is 1.

Original: This example contrasts sharply with backdoor attacks on supervised learning which focuses on instantaneous gratification.
Revised: This example contrasts sharply with backdoor attacks on supervised learning which focuses on instantaneous gratiﬁcation.

Original: Given a policy π and a trigger function f , the value of the policy π in the triggered environment is given as,
Revised: Given a policy ⇡ and a trigger function f , the value of the policy ⇡ in the triggered environment is given as,

Original: [Equation]
Revised: [Equation]

Original: Here, we note that the trigger function f affects the value of the agent only through action selection.
Revised: Here, we note that the trigger function f affects the value of the agent only through action selection.

Original: Also, since f is adaptive (i.e. it depends on current state and history), the composition of a Markovian policy π with the trigger function f leads to a non-Markovian policy π ◦ f .
Revised: Also, since f is adaptive (i.e. it depends on current state and history), the composition of a Markovian policy ⇡ with the trigger function f leads to a non-Markovian policy ⇡  f .

Original: Goal of the defender The defender is provided with a backdoor policy π † , and has a sample of interactions between π † and the clean environment M .
Revised: The defender is provided with a backdoor policy ⇡ † , and has a sample of interactions between ⇡ † and the clean environment M .

Original: This is realistic in many applications: for example, even if a user does not trust the driving policy π † she downloads, she can test drive the car for a few days in an enclosed driving facility.
Revised: This is realistic in many applications: for example, even if a user does not trust the driving policy ⇡ † she downloads, she can test drive the car for a few days in an enclosed driving facility.

Original: The goal of the defender is to sanitize the backdoor policy π † such that the sanitized policy performs near-optimally even in the presence of trigger function f .
Revised: The goal of the defender is to sanitize the backdoor policy ⇡ † such that the sanitized policy performs near-optimally even in the presence of trigger function f .

Original:  Sanitization Algorithm and Guarantees
Revised:  Sanitization Algorithm and Guarantees

Original: We propose Algorithm 2 to sanitize and render backdoor policy harmless.
Revised: We propose Algorithm 2 to sanitize and render backdoor policy harmless.

Original: Our sanitization algorithm works in an unsupervised manner by first recovering an estimate of the safe subspace using the clean samples from d π † M and projecting every states onto this empirical clean subspace to sanitize the state before feeding into the backdoor policy π † (recall the agent is stuck with π † since she does not have the resources to retrain).
Revised: Our sanitization algorithm works in an unsupervised manner by ﬁrst recovering an estimate of the safe subspace using the clean samples from d ⇡ † M and projecting every states onto this empirical clean subspace to sanitize the state before feeding into the backdoor policy ⇡ † (recall the agent is stuck with ⇡ † since she does not have the resources to retrain).

Original: Algorithm
Revised: Algorithm

Original: Entities: MDP environment M , RL agent with policy π † , adversary with trigger function f .
Revised: Entities: MDP environment M , RL agent with policy ⇡ † , adversary with trigger function f .

Original: Inputs: sample access to clean environment M , number of clean samples n .
Revised: Inputs: sample access to clean environment M , number of clean samples n .

Original: 1: Sanitization phase : 2: run π † for n clean episodes, collect { s j }| nj =1 i.i.d ∼ d π † M .
Revised: 1: Sanitization phase : 2: run ⇡ † for n clean episodes, collect { s j }| nj =1 i.i.d ⇠ d ⇡ † M .

Original: 3: calculate the empirical covariance Σ n = 1 n[-(cid:80) nj =
-]
Revised: 3: calculate the empirical covariance ⌃ n = 1 n[-(cid:80) nj =
-]

Original: {+P nj =
+} s j s ⊤ j and its eigen decomposition, Σ n =[-(cid:80) Di =
-]
Revised: {+P nj =
+} s j s > j and its eigen decomposition, ⌃ n =[-(cid:80) Di =
-]

Original: ˆ λ
Revised: P

Original:  i ˆ u i ˆ u ⊤ i where ˆ λ 1 ≥ · · · ˆ λ d ≥ ˆ λ d +1 ≥ · · · ≥ ˆ λ D .
Revised: Di =1 ˆ  i ˆ u i ˆ u > i where ˆ  1  · · · ˆ  d  ˆ  d +1  · · ·  ˆ  D .

Original: 4: construct the empirical projection operator Proj E n =(cid:80) di
Revised: 4: construct the empirical projection operator Proj E n =

Original: [-=
-] ˆ u
Revised: [-=
-]P di =1 ˆ u

Original: i ˆ u ⊤ i .
Revised: i ˆ u > i .

Original: 5: define the sanitized policy π † E n : S → ∆( A )
Revised: 5: deﬁne the sanitized policy ⇡ † E n : S !

Original:  s.t. ∀ s ∈ S , π † E n ( s ) =
Revised:  ( A ) s.t. 8 s 2 S , ⇡ † E n ( s ) =

Original: π † ◦ Proj E n ( s ) .
Revised: ⇡ †  Proj E n ( s ) .

Original: 6: Deployment phase : 7: at every time step t , the agents takes the action π † E n ( s † t ) .
Revised: 6: Deployment phase : 7: at every time step t , the agents takes the action ⇡ † E n ( s † t ) .

Original: Algorithm 2 has a strong guarantee.
Revised: Algorithm 2 has a strong guarantee.

Original: We recall that the backdoor policy performs optimally in the clean environment i.e. V π † M = V ∗M .
Revised: We recall that the backdoor policy performs optimally in the clean environment i.e. V ⇡ † M = V ⇤ M .

Original: So, we would like the performance of sanitized policy π † E n in the triggered environment ( M , f ) to be as close to the performance of the optimal policy in the clean environment.
Revised: So, we would like the performance of sanitized policy ⇡ † E n in the triggered environment ( M , f ) to be as close to the performance of the optimal policy in the clean environment.

Original: Thus, we are interested in upper bounding the performance difference V ∗M - V π † En ◦ f M .
Revised: Thus, we are interested in upper bounding the performance difference V ⇤ M - V ⇡ † En  f M .

Original: [Figure]
Revised: [Figure]

Original: Note that the square trigger (top

Original: ) in the triggered state in figure (c) is filtered out after projection onto the empirical safe subspace as in figure (d).

Original: Theorem 4 Let π † be the backdoor policy, f be the triggered function and let d π † M denote the discounted state occupancy distribution of π † under clean environment MDP M .
Revised: Theorem 4 Let ⇡ † be the backdoor policy, f be the triggered function and let d ⇡ † M denote the discounted state occupancy distribution of ⇡ † under clean environment MDP M .

Original: Further, let δ ∗ =
Revised: Further, let  ⇤ =

Original: λ d − λ d +1 > 0 be the eigen gap between the safe subspace E and E ⊥ as defined in (2) .
Revised:  d   d +1 > 0 be the eigen gap between the safe subspace E and E ? as deﬁned in (2) .

Original: Under assumptions 1, 2, 3 stated above, for a defender that uses the sanitized policy π † E n as defined in
Revised: Under assumptions 1, 2, 3 stated above, for a defender that uses the sanitized policy ⇡ † E n as deﬁned in

Original: Algorithm 2 against an attacker with trigger function f , we have that,
Revised: Algorithm 2 against an attacker with trigger function f , we have that,

Original: [Equation]
Revised: [Equation]

Original: with probability [Equation] [Equation] estimation error (4) where L is the Lipschitz constant of the policy π † in assumption 2, B is the upper bound on the expected observed triggered states in assumption 3, K is the sub-Gaussian parameter of the stateoccupancy distribution d π † M , γ is the discount factor and C is some fixed constant.
Revised: where L is the Lipschitz constant of the policy ⇡ † in assumption 2, B is the upper bound on the expected observed triggered states in assumption 3, K is the sub-Gaussian parameter of the stateoccupancy distribution d ⇡ † M ,  is the discount factor and C is some ﬁxed constant.

Original: We make the following remarks:
Revised: We make the following remarks:

Original: • Our safety guarantee is an ( ϵ, δ ) -PAC style guarantee with both approximation and estimation error terms.
Revised: • Our safety guarantee is an ( ✏ ,  ) -PAC style guarantee with both approximation and estimation error terms.

Original: The first term is a fixed approximation error ϵ app that the defense algorithm has to suffer even in the presence of infinite clean samples.
Revised: The ﬁrst term is a ﬁxed approximation error ✏ app that the defense algorithm has to suffer even in the presence of inﬁnite clean samples.

Original: However, if the eigen energy of E ⊥ subspace (cid:80) Di = d +1 λ i = [-
 i e.-] if the adversary injects trigger only in the null eigen spaces of Σ , the defender can avoid the approximation error.
Revised: However, if the eigen energy of E ? subspace P Di = d +1  i = [-
 i e.-]0 , i.e. if the adversary injects trigger only in the null eigen spaces of ⌃ , the defender can avoid the approximation error.

Original: • The second term ϵ is the estimation error which scales as O (1 / √ n ) and can be made as small as required with more clean samples.
Revised: • The second term ✏ is the estimation error which scales as O (1 / p n ) and can be made as small as required with more clean samples.

Original: • The overall sample complexity grows polynomially in the effective horizon 1 / (1 − γ ) , the inverse eigen-gap 1 δ ∗ , the inverse estimation error 1 ϵ and the dimension of state space D .
Revised: • The overall sample complexity grows polynomially in the effective horizon 1 / (1   ) , the inverse eigen-gap 1  ⇤ , the inverse estimation error 1 ✏ and the dimension of state space D .

Original: We note that the lower the eigen separation between the safe subspace and its complement, the more difficult will it be for the defender to recover the safe subspace using samples; hence more difficult the defense.
Revised: We note that the lower the eigen separation between the safe subspace and its complement, the more difﬁcult will it be for the defender to recover the safe subspace using samples; hence more difﬁcult the defense.

Original: Proof: The key step in the proof is the following value decomposition:
Revised: Proof: The key step in the proof is the following value decomposition:

Original: [Equation]
Revised: [Equation]

Original: Term (1) leads to an approximation error in the clean environment which cannot be avoided even in presence of infinite samples.
Revised: Term (1) leads to an approximation error in the clean environment which cannot be avoided even in presence of inﬁnite samples.

Original: Term (2) is the performance difference of the true projected policy π † E = π † ◦ Proj E when acting on clean vs triggered environment.
Revised: Term (2) is the performance difference of the true projected policy ⇡ † E = ⇡ †  Proj E when acting on clean vs triggered environment.

Original: Term (3) is the estimation error that arises due to the fact that the defender only gets sample access to the clean environment.
Revised: Term (3) is the estimation error that arises due to the fact that the defender only gets sample access to the clean environment.

Original: Since the triggers always lie in E ⊥ (by assumption 3), projection of s † t onto E engulfs the trigger part of it effectively reducing the second error term to zero.
Revised: Since the triggers always lie in E ? (by assumption 3), projection of s † t onto E engulfs the trigger part of it effectively reducing the second error term to zero.

Original: To see this, simply observe that π † E ( s t + f
Revised: To see this, simply observe that ⇡ † E ( s t + f

Original: ( s 0: t ))
Revised: ( s 0: t ))

Original: = π † ( Proj E ( s t + f ( s 0: t )))
Revised: = ⇡ † ( Proj E ( s t + f ( s 0: t )))

Original: = π † ( Proj E ( s t ))=
Revised: = ⇡ † ( Proj E ( s t ))

Original: π † E ( s t ) using linearity of projection operator and orthogonality of triggers f ( · ) to E subspace, which implies,
Revised: = ⇡ † E ( s t ) using linearity of projection operator and orthogonality of triggers f ( · ) to E subspace, which implies,

Original: [Equation]
Revised: [Equation]

Original: We now bound the approximation error in the clean environment, the term (1) in (6):
Revised: We now bound the approximation error in the clean environment, the term (1) in (6):

Original: [Equation]
Revised: [Equation]

Original:  where (1) uses performance difference lemma (Lemma 8)
Revised: [Equation] where (1) uses performance difference lemma (Lemma 8), (2) is Holders inequality, (3) uses LLipschitzness of ⇡ † and Jensen’s inequality

Original: [Equation]
Revised: [Equation]

Original: (2) is Holders inequality, (3) uses LLipschitzness of π † and Jensen’s inequality, (4) uses ∥ Q π † E ∥ ∞ ≤ 1 / (1 − γ ) and (5) follows from the following :
Revised: (4) uses k Q ⇡ † E k 1  1 / (1   ) and (5) follows from the following :

Original: [Equation]
Revised: [Equation]

Original: Next
Revised: Next

Original: [Equation]
Revised: [Equation]

Original: we bound the estimation error in the triggered environment, the third term in (6).
Revised: we bound the estimation error in the triggered environment [Equation] the third term in (6).

Original: Let π = π † ◦ Proj E and π ′ = π † ◦ Proj E n , then the difference in value of these policies in a triggered environment is given as
Revised: Let ⇡ = ⇡ †  Proj E and ⇡ 0 = ⇡ †  Proj E n , then the difference in value of these policies in a triggered environment is given as

Original: [Equation]
Revised: [Equation]

Original: where (1) follows from performance difference lemma for non-Markovian policies (Lemma 8), (2) follows from Holder’s inequality, (3) follows from ∥ Q ∥ ∞ ≤ 1 / (1 − γ ) , (4) follows from
Revised: where (1) follows from performance difference lemma for non-Markovian policies (Lemma 8), (2) follows from Holder’s inequality, (3) follows from k Q k 1  1 / (1   ) , (4) follows from

Original: L -Lipschitzness of π † , (5) follows from L 2 matrix vector norm inequality and (6) uses the Bboundedness of the trigger functions f as defined in 3.
Revised: L -Lipschitzness of ⇡ † , (5) follows from L 2 matrix vector norm inequality and (6) uses the Bboundedness of the trigger functions f as deﬁned in 3.

Original: Now, if n ≥ CdB 2 L 2 K 4 ∥ Σ
Revised: Now, if n  CdB 2 L 2 K 4 k ⌃ k 2

Original: ∥
Revised: [Equation]

Original: δ ∗ 2 (1 − γ ) 4 · ϵ 2 (cid:0) D 
Revised: D [Equation] log [Equation]

Original: + log 2 δ (cid:1) , using Lemma 6 with (11), we can upper bound the estimation error by ϵ , i.e. 
Revised:  , using Lemma 6 with (11), we can upper bound the estimation error by ✏ , i.e. [Equation]

Original: [Equation]
Revised: [Equation]

Original: Finishing the proof of Theorem 4 : We plug back (7), (8), (12) in (6) to conclude the proof.
Revised: Finishing the proof of Theorem 4 : We plug back (7), (8), (12) in (6) to conclude the proof.

Original: □
Revised: ⇤

Original: Computational Complexity
Revised: Computational Complexity

Original: The computation complexity of our subspace sanitization algorithm is
Revised: The computation complexity of our subspace sanitization algorithm is

Original: O ( D 3 + T · D 2 ) where D is the dimension of the state space and T is the total number of time step the agent runs during deployment.
Revised: O ( D 3 + T · D 2 ) where D is the dimension of the state space and T is the total number of time step the agent runs during deployment.

Original: The algorithm incurs a one time cost of O ( D 3 ) for computing the
Revised: The algorithm incurs a one time cost of O ( D 3 ) for computing the

Original: SVD of the empirical covariance matrix and the projection operation at each time step of deployment requires O
Revised: SVD of the empirical covariance matrix and the projection operation at each time step of deployment requires O

Original: ( D 2 ) compute time.
Revised: ( D 2 ) compute time.

Original:  Experiments
Revised:  Experiments

Original: In this section, we present some experimental results that validate our sanitization algorithm against backdoor attacks in Atari game environments.
Revised: In this section, we present some experimental results that validate our sanitization algorithm against backdoor attacks in Atari game environments.

Original: We performed the sanitization experiments on backdoor policies in two Atari games: (i) Boxing-Ram game with a vector RAM state space, (ii) Breakout game with an image state space.
Revised: We performed the sanitization experiments on backdoor policies in two Atari games: (i) Boxing-Ram game with a vector RAM state space, (ii) Breakout game with an image state space.

Original: In each case, we acted as the attacker and pre-trained a backdoor policy using a reward poisoning schema as described in [7].
Revised: In each case, we acted as the attacker and pre-trained a backdoor policy using a reward poisoning schema as described in [7].

Original: The backdoor policy was trained to a level such that the agent performs well in clean environment but fares poorly in the triggered environment.
Revised: The backdoor policy was trained to a level such that the agent performs well in clean environment but fares poorly in the triggered environment.

Original: In each case, we used our sanitization algorithm to sanitize the backdoor policy and reported their results.
Revised: In each case, we used our sanitization algorithm to sanitize the backdoor policy and reported their results.

Original: 5.1 Boxing-Ram game
Revised: 5.1 Boxing-Ram game

Original: Environment and attack specification The state consists of a 32 dimensional byte RAM vector: a concatenation of four 8-byte vectors representing information stored in the simulator’s RAM from the latest four time steps.
Revised: Environment and attack speciﬁcation The state consists of a 32 dimensional byte RAM vector: a concatenation of four 8-byte vectors representing information stored in the simulator’s RAM from the latest four time steps.

Original: The attacker exploits an unused byte (which is always zero in the clean environment) to inject triggers directly into the vector RAM state space.
Revised: The attacker exploits an unused byte (which is always zero in the clean environment) to inject triggers directly into the vector RAM state space.

Original: Concretely, f = 255 · ewhere e 28 is 28 th canonical basis vector in R 32 .
Revised: Concretely, f = 255 · ewhere e 28 is 28 th canonical basis vector in R 32 .

Original: The action space consists of 18 different actions: moving left, right, up, down, punch, and combinations thereof.
Revised: The action space consists of 18 different actions: moving left, right, up, down, punch, and combinations thereof.

Original: Backdoor policy The backdoor policy consists of a 4 layered fully connected neural network with second layer shared to the critic network.
Revised: Backdoor policy The backdoor policy consists of a 4 layered fully connected neural network with second layer shared to the critic network.

Original: The joint actor critic network is trained using PPO algorithm [15] implemented in Pytorch.
Revised: The joint actor critic network is trained using PPO algorithm [15] implemented in Pytorch.

Original: The backdoor policy has been trained using an environment poisoning scheme following [6] to train the agent to seek low value in the triggered environment.
Revised: The backdoor policy has been trained using an environment poisoning scheme following [6] to train the agent to seek low value in the triggered environment.

Original: The policy has been trained to a level such that it receives high returns in the clean environment.
Revised: The policy has been trained to a level such that it receives high returns in the clean environment.

Original: However, when the attacker activates the trigger, the agent will move close to the opponent to get beaten down without fighting back, thus minimizing its return.
Revised: However, when the attacker activates the trigger, the agent will move close to the opponent to get beaten down without ﬁghting back, thus minimizing its return.

Original: 5.2 Breakout game
Revised: 5.2 Breakout game

Original: Environment and the attack specification We consider a version of breakout game where the states consist of a stack of four down-scaled board images (4 × 84 × 84) from the last 4 time steps and the action space consists of three actions: move left, move right, and do not move.
Revised: Environment and the attack speciﬁcation We consider a version of breakout game where the states consist of a stack of four down-scaled board images (4 ⇥ 84 ⇥ 84) from the last 4 time steps and the action space consists of three actions: move left, move right, and do not move.

Original: The attack directly takes place in the image state space where the attacker injects a 6 × 6 pixel square trigger on the top left part of the arena, see Figure 3.
Revised: The attack directly takes place in the image state space where the attacker injects a 6 ⇥ 6 pixel square trigger on the top left part of the arena, see Figure 3.

Original: Backdoor policy The backdoor policy is a neural network with two convolutional layers followed by two dense layers with ReLU activation units.
Revised: Backdoor policy The backdoor policy is a neural network with two convolutional layers followed by two dense layers with ReLU activation units.

Original: The policy has been trained using environment poisoning attack scheme following [7] with a backdoor objective to force the agent to take ‘do not move’ action in the triggered states.
Revised: The policy has been trained using environment poisoning attack scheme following [7] with a backdoor objective to force the agent to take ‘do not move’ action in the triggered states.

Original: Empirically, the policy has been trained to a level such that
Revised: Empirically, the policy has been trained to a level such that

Original: [Figure]
Revised: [Figure]

Original: i [Figure] it consistently performs very well in the clean environment receiving high returns.
Revised: it consistently performs very well in the clean environment receiving high returns.

Original: However, in presence of trigger, it falters into not moving at all thus losing lives very quickly.
Revised: However, in presence of trigger, it falters into not moving at all thus losing lives very quickly.

Original: 5.3 Empirical results of our sanitization algorithm
Revised: 5.3 Empirical results of our sanitization algorithm

Original: For a fixed sanitization sample count n , the defender collects n clean episodes from the environment and chooses an independent sample from each episode to get n samples d π † M .
Revised: For a ﬁxed sanitization sample count n , the defender collects n clean episodes from the environment and chooses an independent sample from each episode to get n samples d ⇡ † M .

Original: Empirically, we observe that choosing roughly 2 n correlated samples also works for sanitization in these examples.
Revised: Empirically, we observe that choosing roughly 2 n correlated samples also works for sanitization in these examples.

Original: It then constructs a sanitized policy (as defined in Algorithm 2) using top d eigen bases of the empirical covariance matrix.
Revised: It then constructs a sanitized policy (as deﬁned in Algorithm 2) using top d eigen bases of the empirical covariance matrix.

Original: The safe space dimension d is decided using the eigen gap in the empirical covariance matrix (all eigenvectors with eigenvalues ≥ 10 − 10 are chosen as ‘safe subspace’, see section 5.4 for further discussion).
Revised: The safe space dimension d is decided using the eigen gap in the empirical covariance matrix (all eigenvectors with eigenvalues  10  10 are chosen as ‘safe subspace’, see section 5.4 for further discussion).

Original: In Figure 4, the blue and brown lines show the performance of backdoor policy in clean and triggered environment, respectively.
Revised: In Figure 4, the blue and brown lines show the performance of backdoor policy in clean and triggered environment, respectively.

Original: The y -axis shows the mean and standard deviation of empirical agent value obtained from 4 independent trials.
Revised: The y -axis shows the mean and standard deviation of empirical agent value obtained from 4 independent trials.

Original: In each trial, we estimated the empirical value from averaging the returns obtained from 5 independent episodes sampled using the respective (policy, trigger) pair.
Revised: In each trial, we estimated the empirical value from averaging the returns obtained from 5 independent episodes sampled using the respective (policy, trigger) pair.

Original: We observe that the backdoor policy performed well in the clean environment (blue).
Revised: We observe that the backdoor policy performed well in the clean environment (blue).

Original: However, its performance dropped significantly in the presence of triggers (brown).
Revised: However, its performance dropped signiﬁcantly in the presence of triggers (brown).

Original: Next, the green line shows the performance of the sanitized policy constructed using n clean samples (on x -axis) from d π † M .
Revised: Next, the green line shows the performance of the sanitized policy constructed using n clean samples (on x -axis) from d ⇡ † M .

Original: We clearly observe that in both the examples, the performance of sanitized policy increases with n and after a sufficient number of clean samples the sanitized policy recovers backs the original clean performance (of backdoor policy in the clean environment, blue line) even when acting in the triggered environment.
Revised: We clearly observe that in both the examples, the performance of sanitized policy increases with n and after a sufﬁcient number of clean samples the sanitized policy recovers backs the original clean performance (of backdoor policy in the clean environment, blue line) even when acting in the triggered environment.

Original: This empirically verifies the success of our sanitization algorithm against subspace backdoor attacks.
Revised: This empirically veriﬁes the success of our sanitization algorithm against subspace backdoor attacks

Original:  5.4 Dependence of our algorithm on the dimension d of safe subspace
Revised: . 5.4 Dependence of our algorithm on the dimension d of safe subspace

Original: Our theory assumes that the defender knows the dimension of the safe space d which might not always hold true in practice.
Revised: Our theory assumes that the defender knows the dimension of the safe space d which might not always hold true in practice.

Original: Though, the defender still has an access to eigen spectrum of the clean empirical covariance matrix Σ n which it can use to estimate d and the corresponding ‘safe subspace’.
Revised: Though, the defender still has an access to eigen spectrum of the clean empirical covariance matrix ⌃ n which it can use to estimate d and the corresponding ‘safe subspace’.

Original: A good estimate of d is important, because both underestimation and overestimation can lead to loss in performance.
Revised: A good estimate of d is important, because both underestimation and overestimation can lead to loss in performance.

Original: Specifically, with underestimation the defender might lose important dimensions in the safe subspace, and with overestimation the defender may include spurious dimensions from E ⊥ hampering effective sanitization in both scenarios.
Revised: Speciﬁcally, with underestimation the defender might lose important dimensions in the safe subspace, and with overestimation the defender may include spurious dimensions from E ? hampering effective sanitization in both scenarios.

Original: This phenomenon is depicted by the empirical value curve (green) in Figure 5, where we see that the performance of the sanitized policy decreases both with underestimation and overestimation of d .
Revised: This phenomenon is depicted by the empirical value curve (green) in Figure 5, where we see that the performance of the sanitized policy decreases both with underestimation and overestimation of d .

Original: In practice, one can choose d based on the spectral gap of the empirical covariance matrix Σ n .
Revised: In practice, one can choose d based on the spectral gap of the empirical covariance matrix ⌃ n .

Original: From the singular value spectrum curve (blue) in
Revised: From the singular value spectrum curve (blue) in

Original: [Figure]
Revised: [Figure]

Original: Figure 5, we observe that the correct threshold corresponds to dimension just before the first major dip in singular values; which occurs after dimension 24 in Boxing-Ram game and near dimension 20000 in Breakout game.
Revised: Figure 5, we observe that the correct threshold corresponds to dimension just before the ﬁrst major dip in singular values; which occurs after dimension 24 in Boxing-Ram game and near dimension 20000 in Breakout game.

Original: We used these thresholds to select d in our experiments.
Revised: We used these thresholds to select d in our experiments.

Original:  Conclusion
Revised:  Conclusion

Original: We formally specified backdoor policy with subspace trigger attacks in RL.
Revised: We formally speciﬁed backdoor policy with subspace trigger attacks in RL.

Original: We then proposed a sanitization algorithm that allows a user to safely use a backdoor policy under subspace trigger attackers.
Revised: We then proposed a sanitization algorithm that allows a user to safely use a backdoor policy under subspace trigger attackers.

Original: Our defense has the advantage of being a wrapper method and does not require expensive policy retraining.
Revised: Our defense has the advantage of being a wrapper method and does not require expensive policy retraining.

Original: Our sanitization work makes RL safer and contributes to trustworthy AI.
Revised: Our sanitization work makes RL safer and contributes to trustworthy AI.

Original: Future work will address the limitations of our defense, namely the need for clean environment interactions and the assumption that triggers reside in a subspace E ⊥ .
Revised: Future work will address the limitations of our defense, namely the need for clean environment interactions and the assumption that triggers reside in a subspace E ? .

Original: Acknowledgment This work was supported in part by NSF grants 1545481, 1704117, 1836978, 2023239, 2041428, 2202457, ARO MURI W911NF2110317, and AF CoE FA9550-18-1-0166.Appendix
Revised: Acknowledgment This work was supported in part by NSF grants 1545481, 1704117, 1836978, 2023239, 2041428, 2202457, ARO MURI W911NF and AF CoE FA9550-18-1-0166.

Original: Lemma 5 (Davis Kahn sin Θ bound [21]) Let Σ , ˆΣ ∈ R D

Original: × D be symmetric, with eigen values λ 1 ≥ · · · λ D and ˆ λ 1 ≥ · · · ˆ λ D , where Σ v i

Original: =

Original: λ i v i , ∀ i ∈

Original: [ D ] and ˆΣˆ v i =

Original: ˆ λ

Original: i ˆ

Original: v i , ∀

Original: i ∈

Original: [ D ] .

Original: Further, define E =

Original: [ v 1 , · · · , v d ] ∈ R D × d , ˆ E =

Original: [ˆ v 1 , · · · , ˆ v d ] and assume that δ ∗ =

Original: λ d − λ d +1 > 0 .

Original: Then, we have that,

Original: [Equation]

Original: Lemma 6 (Concentration of sub-Gaussian covariance) Let D be a zero-mean sub-Gaussian distribution in R D s.t. for X ∼ D , we have that, ∥⟨ X, x ⟩∥ ψ 2 ≤ K ∥⟨ X, x ⟩∥ 2 , ∀ x ∈ R D .

Original: Denote the covariance matrix Σ

Original: =

Original: E X

Original: ∼D (cid:2) XX T (cid:3) and Σ n = 1 n (cid:80) ni =1 X i X ⊤ i , where X i i.i.d ∼ D .

Original: Then ∀ δ > 0 , we have that,

Original: [Equation]

Original: with probability ≥ 1 − δ, for some fixed small constant C .

Original: Proof: Refer to Theorem 4.7.1 and Exercise 4.7.3 in Vershynin [17].

Original: □

Original: Note that in large data regime, i.e. n >> D , only the first term dominates while in the small data and large dimension regime the second terms dominates.

Original: In large data regime, for any given error parameter ϵ > 0 , one can sample n large enough and so the first term dominates the second one.

Original: Lemma 7 (Concentration of Projection matrix) Let X ∈ R D be a zero mean sub-gaussian random vector with bounded sub-gaussian norm i.e.

Original: [Equation]

Original: Given n i.i.d sample { X i }| ni =1 i.i.d ∼ P X , let Σ n = 1 n (cid:80) ni

Original: =1 X

Original: i X

Original: ⊤ i be the empirical estimate of the covariance matrix Σ

Original: = E [ XX ⊤ ] .

Original: Further, let the eigen decomposition of Σ , Σ n be given as,

Original: [Equation]

Original: and Proj E = (cid:80) di =1 u i u ⊤ i and Proj E n = (cid:80) di

Original: [-=
-]
Revised: [-=
-]

Original: ˆ u

Original: i ˆ u ⊤ i denote the projection operator onto the top d true and empirical eigen subspaces E = span ( { u i }| di =1 ) and E n = span ( { ˆ u i }| di =1 ) respectively.

Original: Then, if n ≥ CdK 4 ∥ Σ ∥

Original: [Equation]

Original: 2 · ϵ

Original: [Equation]

Original: log 2 δ (cid:1) , we have,

Original: [Equation]

Original: where δ ∗ is the eigen gap between top d and the rest eigen subspace of true covariance matrix Σ .

Original: Proof: Let E ⊤ E n = U cos Θ( E, E n ) V T denote the singular value decomposition of E ⊤ E n and cos Θ( E, E n ) =

Original: diag (cos θ 1 , · · · , cos θ d ) be the diagonal matrix with the cosine of principal angles { θ 1 , · · · , θ d } between E and E n subspaces as its diagonal entries.

Original: With this definition, it is easy to show that,

Original: [Equation]

Original: Next, using sin Θ variant of Davis Kahn theorem 5, we have,

Original: [Equation]

Original: Note that we have used the fact that L 2 norm is upper bounded by Frobenious norm.

Original: Next, using concentration bound for covariance matrix of sub-gaussian random vectors 6(in large sample regime), we have, w.p. ≥ 1 − δ,

Original: [Equation]

Original: Using it along with (16), gives us, w.p. ≥ 1 − δ,

Original: [Equation]

Original: Finally, choosing n ≥

Original: [Equation]

Original: , establishes the lemma.

Original: □

Original: Lemma 8 (Performance difference lemma for a general policy) Let π, π ′ be two time and history dependent non-Markovian policies acting in environment MDP M .

Original: Then, the performance difference between the value function of these policy in environment M is given by :

Original: [Equation]

Original: Further, if π, π ′ are Markovian, we have,

Original: [Equation]

Original: Proof: We provide a complete proof of the performance difference lemma for self sufficiency.

Original: [Equation]

Original: Further

Original: [Equation]

Original: if π

Original: [Equation]

Original: π ′ are Markovian

Original: [Equation]

Original: it simplifies to

Original: [Equation]