Original: COPA: C ERTIFYING R OBUST P OLICIES FOR O FFLINE R EINFORCEMENT L EARNING AGAINST P OISONING A TTACKS
Revised: COPA: C ERTIFYING R OBUST P OLICIES FOR O FFLINE R EINFORCEMENT L EARNING AGAINST P OISONING A TTACKS

Original: A BSTRACT
Revised: A BSTRACT

Original: As reinforcement learning (RL) has achieved near human-level performance in a variety of tasks, its robustness has raised great attention.
Revised: As reinforcement learning (RL) has achieved near human-level performance in a variety of tasks, its robustness has raised great attention.

Original: While a vast body of research has explored test-time (evasion) attacks in RL and corresponding defenses, its robustness against training-time (poisoning) attacks remains largely unanswered.
Revised: While a vast body of research has explored test-time (evasion) attacks in RL and corresponding defenses, its robustness against training-time (poisoning) attacks remains largely unanswered.

Original: In this work, we focus on certifying the robustness of ofﬂine RL in the presence of poisoning attacks, where a subset of training trajectories could be arbitrarily manipulated.
Revised: In this work, we focus on certifying the robustness of ofﬂine RL in the presence of poisoning attacks, where a subset of training trajectories could be arbitrarily manipulated.

Original: We propose the ﬁrst certiﬁcation framework, COPA, to certify the number of poisoning trajectories that can be tolerated regarding different certiﬁcation criteria.
Revised: We propose the ﬁrst certiﬁcation framework, COPA, to certify the number of poisoning trajectories that can be tolerated regarding different certiﬁcation criteria.

Original: Given the complex structure of RL, we propose two certiﬁcation criteria: per-state action stability and cumulative reward bound .
Revised: Given the complex structure of RL, we propose two certiﬁcation criteria: per-state action stability and cumulative reward bound .

Original: To further improve the certiﬁcation, we propose new partition and aggregation protocols to train robust policies.
Revised: To further improve the certiﬁcation, we propose new partition and aggregation protocols to train robust policies.

Original: We further prove that some of the proposed certiﬁcation methods are theoretically tight and some are NP-Complete problems.
Revised: We further prove that some of the proposed certiﬁcation methods are theoretically tight and some are NP-Complete problems.

Original: We leverage COPA to certify three RL environments trained with different algorithms and conclude: (1) The proposed robust aggregation protocols such as temporal aggregation can signiﬁcantly improve the certiﬁcations; (2) Our certiﬁcations for both per-
Revised: We leverage COPA to certify three RL environments trained with different algorithms and conclude: (1) The proposed robust aggregation protocols such as temporal aggregation can signiﬁcantly improve the certiﬁcations; (2) Our certiﬁcations for both per-

Original: state action stability and cumulative reward bound are efﬁcient and tight; (3) The certiﬁcation for different training algorithms and environments are different, implying their intrinsic robustness properties.
Revised: state action stability and cumulative reward bound are efﬁcient and tight; (3) The certiﬁcation for different training algorithms and environments are different, implying their intrinsic robustness properties.

Original: All experimental results are available at https://copa-leaderboard.github.io .
Revised: All experimental results are available at https://copa-leaderboard.github.io .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Reinforcement learning (RL) has been widely applied to a range of applications, including robotics (Kober et al., 2013;
Revised: Reinforcement learning (RL) has been widely applied to a range of applications, including robotics (Kober et al., 2013;

Original: Deisenroth et al., 2013; Polydoros & Nalpantidis, 2017) and autonomous vehicles (Shalev-Shwartz et al., 2016; Sallab et al., 2017).
Revised: Deisenroth et al., 2013; Polydoros & Nalpantidis, 2017) and autonomous vehicles (Shalev-Shwartz et al., 2016; Sallab et al., 2017).

Original: In particular, ofﬂine RL (Levine et al., 2020) is proposed to leverage previously observed data to train the policy without requiring expensive interaction with environments or online data collection, and enables the reuse of training data (Agarwal et al., 2020).
Revised: In particular, ofﬂine RL (Levine et al., 2020) is proposed to leverage previously observed data to train the policy without requiring expensive interaction with environments or online data collection, and enables the reuse of training data (Agarwal et al., 2020).

Original: However, ofﬂine RL also raises a great safety concern on poisoning attacks (Kiourti et al., 2020; Wang et al., 2020; 2021).
Revised: However, ofﬂine RL also raises a great safety concern on poisoning attacks (Kiourti et al., 2020; Wang et al., 2020; 2021).

Original: A recent survey of industry reports that data poisoning is signiﬁcantly more concerning than other threats (Kumar et al., 2020).
Revised: A recent survey of industry reports that data poisoning is signiﬁcantly more concerning than other threats (Kumar et al., 2020).

Original: For ofﬂine RL, the situation is even worse, and a recent theoretical study shows that the robust ofﬂine RL against poisoning attacks is a strictly harder problem than online RL (Zhang et al., 2021).
Revised: For ofﬂine RL, the situation is even worse, and a recent theoretical study shows that the robust ofﬂine RL against poisoning attacks is a strictly harder problem than online RL (Zhang et al., 2021).

Original: Although there are several empirical and certiﬁed defenses for classiﬁcation tasks against poisoning attacks (Peri et al., 2020; Levine & Feizi, 2021; Weber et al., 2020), it is challenging and shown ineffective to directly apply them to RL given its complex structure (Kiourti et al., 2020).
Revised: Although there are several empirical and certiﬁed defenses for classiﬁcation tasks against poisoning attacks (Peri et al., 2020; Levine & Feizi, 2021; Weber et al., 2020), it is challenging and shown ineffective to directly apply them to RL given its complex structure (Kiourti et al., 2020).

Original: Thus, robust ofﬂine RL against poisoning attacks remains largely unexplored with no mention of robustness certiﬁcation.
Revised: Thus, robust ofﬂine RL against poisoning attacks remains largely unexplored with no mention of robustness certiﬁcation.

Original: In addition, though some theoretical analyses provide general robustness bounds for
Revised: In addition, though some theoretical analyses provide general robustness bounds for

Original: RL, they either assume a bounded distance between the learned and Bellman optimal policies or are limited to linear MDPs (Zhang et al., 2021).
Revised: RL, they either assume a bounded distance between the learned and Bellman optimal policies or are limited to linear MDPs (Zhang et al., 2021).

Original: To the best of our knowledge, there is no robust RL method that is able to provide practically computable certiﬁed robustness against poisoning attacks.
Revised: To the best of our knowledge, there is no robust RL method that is able to provide practically computable certiﬁed robustness against poisoning attacks.

Original: In this paper, we tackle this problem by proposing the ﬁrst framework of Certifying robust policies for general ofﬂine RL against poisoning attacks ( COPA ).
Revised: In this paper, we tackle this problem by proposing the ﬁrst framework of Certifying robust policies for general ofﬂine RL against poisoning attacks ( COPA ).

Original: Certiﬁcation Criteria.
Revised: Certiﬁcation Criteria.

Original: One critical challenge in certifying robustness for ofﬂine RL is the certiﬁcation criteria, since the prediction consistency is no longer the only goal as in classiﬁcation.
Revised: One critical challenge in certifying robustness for ofﬂine RL is the certiﬁcation criteria, since the prediction consistency is no longer the only goal as in classiﬁcation.

Original: We propose two criteria based on the properties of RL: per-state action stability and cumulative reward bound.
Revised: We propose two criteria based on the properties of RL: per-state action stability and cumulative reward bound.

Original: The former guarantees that at a speciﬁc time, the policy learned with COPA will predict the same action before and after attacks under certain conditions.
Revised: The former guarantees that at a speciﬁc time, the policy learned with COPA will predict the same action before and after attacks under certain conditions.

Original: This is important for guaranteeing the safety of the policy at critical states, e.g. , braking when seeing pedestrians.
Revised: This is important for guaranteeing the safety of the policy at critical states, e.g. , braking when seeing pedestrians.

Original: For cumulative reward bound, a lower bound of the cumulative reward for the policy learned with COPA is guaranteed under certain poisoning conditions.
Revised: For cumulative reward bound, a lower bound of the cumulative reward for the policy learned with COPA is guaranteed under certain poisoning conditions.

Original: This directly guarantees the worst-case overall performance.
Revised: This directly guarantees the worst-case overall performance.

Original: COPA Framework.
Revised: COPA Framework.

Original: COPA is composed of two components: policy partition and aggregation protocol and robustness certiﬁcation method.
Revised: COPA is composed of two components: policy partition and aggregation protocol and robustness certiﬁcation method.

Original: We propose three policy partition aggregation protocols: PARL (Per-State Partition Aggregation), TPARL (Temporal Partition Aggregation), and
Revised: We propose three policy partition aggregation protocols: PARL (Per-State Partition Aggregation), TPARL (Temporal Partition Aggregation), and

Original: DPARL (Dynamic Temporal Partition Aggregation), and propose certiﬁcation methods for each of them corresponding to both proposed certiﬁcation criteria.
Revised: DPARL (Dynamic Temporal Partition Aggregation), and propose certiﬁcation methods for each of them corresponding to both proposed certiﬁcation criteria.

Original: In addition, for per-state action stability , we prove that our certiﬁcations for PARL and TPARL are theoretically tight.
Revised: In addition, for per-state action stability , we prove that our certiﬁcations for PARL and TPARL are theoretically tight.

Original: For cumulative reward bound , we propose an adaptive search algorithm, where we compute the possible action set for each state under certain poisoning conditions.
Revised: For cumulative reward bound , we propose an adaptive search algorithm, where we compute the possible action set for each state under certain poisoning conditions.

Original: Concretely, we propose a novel method to compute the precise action set for PARL and efﬁcient algorithms to compute a superset of the possible action set which leads to sound certiﬁcation for TPARL and DPARL.
Revised: Concretely, we propose a novel method to compute the precise action set for PARL and efﬁcient algorithms to compute a superset of the possible action set which leads to sound certiﬁcation for TPARL and DPARL.

Original: We further prove that for PARL our certiﬁcation is theoretically tight, for TPARL the theoretically tight certiﬁcation is NP -complete, and for DPARL it is open whether theoretically tight certiﬁcation exists.
Revised: We further prove that for PARL our certiﬁcation is theoretically tight, for TPARL the theoretically tight certiﬁcation is NP -complete, and for DPARL it is open whether theoretically tight certiﬁcation exists.

Original: Technical Contributions.
Revised: Technical Contributions.

Original: We take the ﬁrst step towards certifying the robustness of ofﬂine
Revised: We take the ﬁrst step towards certifying the robustness of ofﬂine

Original: RL against poisoning attacks, and we make contributions on both theoretical and practical fronts.
Revised: RL against poisoning attacks, and we make contributions on both theoretical and practical fronts.

Original: • We abstract and formulate the robustness certiﬁcation for ofﬂine RL against poisoning attacks, and we propose two certiﬁcation criteria: per-state action stability and cumulative reward bound.
Revised: • We abstract and formulate the robustness certiﬁcation for ofﬂine RL against poisoning attacks, and we propose two certiﬁcation criteria: per-state action stability and cumulative reward bound.

Original: • We propose the ﬁrst framework COPA for certifying robustness of ofﬂine RL against poisoning attacks.
Revised: • We propose the ﬁrst framework COPA for certifying robustness of ofﬂine RL against poisoning attacks.

Original: COPA includes novel policy aggregation protocols and certiﬁcation methods.
Revised: COPA includes novel policy aggregation protocols and certiﬁcation methods.

Original: • We prove the tightness of the proposed certiﬁcation methods for the aggregation protocol PARL.
Revised: • We prove the tightness of the proposed certiﬁcation methods for the aggregation protocol PARL.

Original: We also prove the computational hardness of the certiﬁcation for TPARL.
Revised: We also prove the computational hardness of the certiﬁcation for TPARL.

Original: • We conduct thorough experimental evaluation for COPA on different RL environments with three ofﬂine RL algorithms, demonstrating the effectiveness of COPA, together with several interesting ﬁndings.
Revised: • We conduct thorough experimental evaluation for COPA on different RL environments with three ofﬂine RL algorithms, demonstrating the effectiveness of COPA, together with several interesting ﬁndings.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Poisoning attacks (Nelson et al., 2008; Diakonikolas et al., 2016) are critical threats in machine learning, which are claimed to be more concerning than other threats (Kumar et al., 2020).
Revised: Poisoning attacks (Nelson et al., 2008; Diakonikolas et al., 2016) are critical threats in machine learning, which are claimed to be more concerning than other threats (Kumar et al., 2020).

Original: Poisoning attacks widely exist in classiﬁcation (Schwarzschild et al., 2021), and both empirical defenses (Liu et al., 2018; Chacon et al., 2019; Peri et al., 2020;
Revised: Poisoning attacks widely exist in classiﬁcation (Schwarzschild et al., 2021), and both empirical defenses (Liu et al., 2018; Chacon et al., 2019; Peri et al., 2020;

Original: Steinhardt et al., 2017) and certiﬁed defenses (Weber et al., 2020; Jia et al., 2020; Levine & Feizi, 2021) have been proposed.
Revised: Steinhardt et al., 2017) and certiﬁed defenses (Weber et al., 2020; Jia et al., 2020; Levine & Feizi, 2021) have been proposed.

Original: After Kiourti et al.
Revised: After Kiourti et al.

Original: (2020) show the existence of effective backdoor poisoning attacks in RL, a recent work theoretically and empirically validates the existence of reward poisoning in online
Revised: (2020) show the existence of effective backdoor poisoning attacks in RL, a recent work theoretically and empirically validates the existence of reward poisoning in online

Original: RL (Zhang et al., 2020b).
Revised: RL (Zhang et al., 2020b).

Original: Furthermore, Zhang et al.
Revised: Furthermore, Zhang et al.

Original: (2021) theoretically prove that the ofﬂine RL is more difﬁcult to be robustiﬁed against poisoning than online RL considering linear MDP.
Revised: (2021) theoretically prove that the ofﬂine RL is more difﬁcult to be robustiﬁed against poisoning than online RL considering linear MDP.

Original: From the defense side, Zhang et al.
Revised: From the defense side, Zhang et al.

Original: (2021) propose robust variants of the Least-Square Value Iteration algorithm that provides probabilistic robustness guarantees under linear MDP assumption.
Revised: (2021) propose robust variants of the Least-Square Value Iteration algorithm that provides probabilistic robustness guarantees under linear MDP assumption.

Original: In addition, Robust RL against reward poisoning is studied in Banihashem et al.
Revised: In addition, Robust RL against reward poisoning is studied in Banihashem et al.

Original: , but robust
Revised: , but robust

Original: RL against general poisoning is less explored.
Revised: RL against general poisoning is less explored.

Original: In this background, we aim to provide the certiﬁed robustness for general ofﬂine RL algorithms against poisoning attacks, which is the ﬁrst work that achieves the goal.
Revised: In this background, we aim to provide the certiﬁed robustness for general ofﬂine RL algorithms against poisoning attacks, which is the ﬁrst work that achieves the goal.

Original: We discuss broader related work in Appendix I.
Revised: We discuss broader related work in Appendix I.

Original:  C ERTIFICATION C RITERIA OF COPA
Revised:  C ERTIFICATION C RITERIA OF COPA

Original: In this section, we propose two robustness certiﬁcation criteria for ofﬂine RL against general poisoning attacks: per-state action stability and cumulative reward bound.
Revised: In this section, we propose two robustness certiﬁcation criteria for ofﬂine RL against general poisoning attacks: per-state action stability and cumulative reward bound.

Original: Ofﬂine RL.
Revised: Ofﬂine RL.

Original: We model the RL environment by an episodic ﬁnite-horizon Markov decision process (MDP) E = ( S , A , R, P, H, d 0 ) , where S is the set of states, A is the set of discrete actions,
Revised: We model the RL environment by an episodic ﬁnite-horizon Markov decision process (MDP) E = ( S , A , R, P, H, d 0 ) , where S is the set of states, A is the set of discrete actions,

Original: R : S ×
Revised: R : S ×

Original: A → R is the reward function, P : S ×
Revised: A → R is the reward function, P : S ×

Original: A → P ( S ) is the stochastic transition function with P ( · ) deﬁning the set of probability measures, H is the time horizon, and d 0 ∈ P ( S ) is the distribution of the initial state.
Revised: A → P ( S ) is the stochastic transition function with P ( · ) deﬁning the set of probability measures, H is the time horizon, and d 0 ∈ P ( S ) is the distribution of the initial state.

Original: At time step t , the RL agent is at state s t ∈ S .
Revised: At time step t , the RL agent is at state s t ∈ S .

Original: After choosing action a t ∈ A , the agent transitions to the next state s t +1 ∼ P ( s t , a t ) and receives reward r t =
Revised: After choosing action a t ∈ A , the agent transitions to the next state s t +1 ∼ P ( s t , a t ) and receives reward r t =

Original: R ( s t , a t ) .
Revised: R ( s t , a t ) .

Original: After H time steps, the cumulative reward J = (cid:80)
Revised: After H time steps, the cumulative reward J = (cid:80)

Original: H − 1 t =0 r t .
Revised: H − 1 t =0 r t .

Original: We denote a consecutive sequence of all states between time step l and r as s l : r :=
Revised: We denote a consecutive sequence of all states between time step l and r as s l : r :=

Original: [ s l , s l +1 , . . .
Revised: [ s l , s l +1 , . . .

Original: , s r ] .
Revised: , s r ] .

Original: Here we focus on ofﬂine RL, for which the threat of poisoning attacks is practical and more challenging to deal with (Zhang et al., 2021).
Revised: Here we focus on ofﬂine RL, for which the threat of poisoning attacks is practical and more challenging to deal with (Zhang et al., 2021).

Original: Concretely, in ofﬂine RL, a training dataset D = { τ i } Ni = consists of logged trajectories, where each trajectory τ = { ( s j , r j , a j , s (cid:48) j ) } lj
Revised: Concretely, in ofﬂine RL, a training dataset D = { τ i } Ni = consists of logged trajectories, where each trajectory τ = { ( s j , r j , a j , s (cid:48) j ) } lj

Original: =1 ∈ ( S ×
Revised: =1 ∈ ( S ×

Original: A × R × S ) l consists of multiple tuples denoting the transitions ( i.e. , starting from state s j , taking the action a j , receiving reward r j , and transitioning to the next state s (cid:48) j ).
Revised: A × R × S ) l consists of multiple tuples denoting the transitions ( i.e. , starting from state s j , taking the action a j , receiving reward r j , and transitioning to the next state s (cid:48) j ).

Original: Poisoning Attacks.
Revised: Poisoning Attacks.

Original: Training dataset D can be poisoned in the following manner.
Revised: Training dataset D can be poisoned in the following manner.

Original: For each trajectory τ ∈ D , the adversary is allowed to replace it with an arbitrary trajectory (cid:101) τ , generating a manipulated dataset (cid:101) D .
Revised: For each trajectory τ ∈ D , the adversary is allowed to replace it with an arbitrary trajectory (cid:101) τ , generating a manipulated dataset (cid:101) D .

Original: We denote D (cid:9) (cid:101) D
Revised: We denote D (cid:9) (cid:101) D

Original: =
Revised: =

Original: ( D \ (cid:101) D ) (cid:83) ( (cid:101) D \ D ) as the symmetric difference between two datasets D and (cid:101) D .
Revised: ( D \ (cid:101) D ) (cid:83) ( (cid:101) D \ D ) as the symmetric difference between two datasets D and (cid:101) D .

Original: For instance, adding or removing one trajectory causes a symmetric difference of magnitude 1 , while replacing one trajectory with a new one leads to a symmetric difference of magnitude 2 .
Revised: For instance, adding or removing one trajectory causes a symmetric difference of magnitude 1 , while replacing one trajectory with a new one leads to a symmetric difference of magnitude 2 .

Original: We refer to the size of the symmetric difference as the poisoning size .
Revised: We refer to the size of the symmetric difference as the poisoning size .

Original: Certiﬁcation Goal.
Revised: Certiﬁcation Goal.

Original: To provide the robustness certiﬁcation against poisoning attacks introduced above, we aim to certify the test-time performance of the trained policy in a clean environment.
Revised: To provide the robustness certiﬁcation against poisoning attacks introduced above, we aim to certify the test-time performance of the trained policy in a clean environment.

Original: Speciﬁcally, in the training phase , the RL training algorithm and our aggregation protocol can be jointly modeled by M : D → ( S (cid:63) → A ) which provides an aggregated policy, where S (cid:63) denotes the set of all consecutive state sequences.
Revised: Speciﬁcally, in the training phase , the RL training algorithm and our aggregation protocol can be jointly modeled by M : D → ( S (cid:63) → A ) which provides an aggregated policy, where S (cid:63) denotes the set of all consecutive state sequences.

Original: Our goal is to provide robustness certiﬁcation for the poisoned aggregated policy ˜ π
Revised: Our goal is to provide robustness certiﬁcation for the poisoned aggregated policy ˜ π

Original: = M ( (cid:101) D ) , given bounded poisoning size ( i.e. , | D (cid:9) (cid:101) D | ≤ K ).
Revised: = M ( (cid:101) D ) , given bounded poisoning size ( i.e. , | D (cid:9) (cid:101) D | ≤ K ).

Original: Robustness Certiﬁcation Criteria: Per-State Action Stability
Revised: Robustness Certiﬁcation Criteria: Per-State Action Stability

Original: We ﬁrst aim to certify the robustness of the poisoned policy in terms of the stability of per-state action during test time.
Revised: We ﬁrst aim to certify the robustness of the poisoned policy in terms of the stability of per-state action during test time.

Original: Deﬁnition 1 (Robustness Certiﬁcation for Per-State Action Stability) .
Revised: Deﬁnition 1 (Robustness Certiﬁcation for Per-State Action Stability) .

Original: Given a clean dataset D , we deﬁne the robustness certiﬁcation for per-state action stability as that for any (cid:101)
Revised: Given a clean dataset D , we deﬁne the robustness certiﬁcation for per-state action stability as that for any (cid:101)

Original: D satisfying | D (cid:9) (cid:101) D | ≤
Revised: D satisfying | D (cid:9) (cid:101) D | ≤

Original: K , the action predictions of the poisoned and clean policies for the state (or state sequence) s are the same, i.e. , ˜ π
Revised: K , the action predictions of the poisoned and clean policies for the state (or state sequence) s are the same, i.e. , ˜ π

Original: = M ( (cid:101) D ) , π = M ( D ) , ˜ π
Revised: = M ( (cid:101) D ) , π = M ( D ) , ˜ π

Original: ( s ) =
Revised: ( s ) =

Original: π ( s ) , under the the tolerable poisoning threshold K .
Revised: π ( s ) , under the the tolerable poisoning threshold K .

Original: In an episode, we denote the tolerable poisoning threshold for the state at step t by K t .
Revised: In an episode, we denote the tolerable poisoning threshold for the state at step t by K t .

Original: The deﬁnition encodes the requirement that, for a particular state, any poisoned policy will always give the same action prediction as the clean one, as long as the poisoning size is within K ( K computed in Section 4).
Revised: The deﬁnition encodes the requirement that, for a particular state, any poisoned policy will always give the same action prediction as the clean one, as long as the poisoning size is within K ( K computed in Section 4).

Original: In this deﬁnition, s could be either a state or a state sequence, since our aggregated policy (deﬁned in Section 3) may aggregate multiple recent states to make an action choice.
Revised: In this deﬁnition, s could be either a state or a state sequence, since our aggregated policy (deﬁned in Section 3) may aggregate multiple recent states to make an action choice.

Original: Robustness Certiﬁcation Criteria: Lower Bound of Cumulative Reward.
Revised: Robustness Certiﬁcation Criteria: Lower Bound of Cumulative Reward.

Original: We also aim to certify poisoned policy’s overall performance in addition to the prediction at a particular state.
Revised: We also aim to certify poisoned policy’s overall performance in addition to the prediction at a particular state.

Original: Here we measure the overall performance by the cumulative reward J ( π ) (formally deﬁned in Appendix A).
Revised: Here we measure the overall performance by the cumulative reward J ( π ) (formally deﬁned in Appendix A).

Original: Now we are ready to deﬁne the robustness certiﬁcation for cumulative reward bound.
Revised: Now we are ready to deﬁne the robustness certiﬁcation for cumulative reward bound.

Original: Deﬁnition 2 (Robustness Certiﬁcation for Cumulative Reward Bound) .
Revised: Deﬁnition 2 (Robustness Certiﬁcation for Cumulative Reward Bound) .

Original: Robustness certiﬁcation for cumulative reward bound is the lower bound of cumulative reward J K such that J K ≤ J (˜ π ) for any ˜
Revised: Robustness certiﬁcation for cumulative reward bound is the lower bound of cumulative reward J K such that J K ≤ J (˜ π ) for any ˜

Original: π
Revised: π

Original: = M ( (cid:101) D )
Revised: = M ( (cid:101) D )

Original: where | D (cid:9) (cid:101) D | ≤ K , i.e. , ˜ π is trained on poisoned dataset (cid:101) D within poisoning size K .
Revised: where | D (cid:9) (cid:101) D | ≤ K , i.e. , ˜ π is trained on poisoned dataset (cid:101) D within poisoning size K .

Original:  C ERTIFICATION P ROCESS OF COPA
Revised:  C ERTIFICATION P ROCESS OF COPA

Original: In this section, we introduce our framework COPA, which is composed of training protocols , aggregation protocols , and certiﬁcation methods .
Revised: In this section, we introduce our framework COPA, which is composed of training protocols , aggregation protocols , and certiﬁcation methods .

Original: The training protocol combined with an ofﬂine RL training algorithm provides subpolicies.
Revised: The training protocol combined with an ofﬂine RL training algorithm provides subpolicies.

Original: The aggregation protocol aggregates the subpolicies as an aggregated policy.
Revised: The aggregation protocol aggregates the subpolicies as an aggregated policy.

Original: The certiﬁcation method certiﬁes the robustness of the aggregated policy against poisoning attacks corresponding to different certiﬁcation criteria provided in Section 3.
Revised: The certiﬁcation method certiﬁes the robustness of the aggregated policy against poisoning attacks corresponding to different certiﬁcation criteria provided in Section 3.

Original: [Table]
Revised: [Table]

Original: Overview of Theoretical Results.
Revised: Overview of Theoretical Results.

Original: In Table 1, we present an overview of our theoretical results:
Revised: In Table 1, we present an overview of our theoretical results:

Original: For each proposed aggregation protocol and certiﬁcation criteria, we provide the corresponding certiﬁcation method and core theorems, and we also provide the tightness analysis for each certiﬁcation.
Revised: For each proposed aggregation protocol and certiﬁcation criteria, we provide the corresponding certiﬁcation method and core theorems, and we also provide the tightness analysis for each certiﬁcation.

Original: 4. P ARTITION -B ASED T RAINING P ROTOCOL
Revised: 4. P ARTITION -B ASED T RAINING P ROTOCOL

Original: COPA’s training protocol contains two stages: partitioning and training.
Revised: COPA’s training protocol contains two stages: partitioning and training.

Original: We denote D as the entire ofﬂine RL training dataset.
Revised: We denote D as the entire ofﬂine RL training dataset.

Original: We abstract an ofﬂine RL training algorithm
Revised: We abstract an ofﬂine RL training algorithm

Original: ( e.g. , DQN) by M 0 : 2 D → Π , where 2 D is the power set of D , and Π = { π : S → A} is the set of trained policies.
Revised: ( e.g. , DQN) by M 0 : 2 D → Π , where 2 D is the power set of D , and Π = { π : S → A} is the set of trained policies.

Original: Each trained policy in Π is a function mapping a given state to the predicted action.
Revised: Each trained policy in Π is a function mapping a given state to the predicted action.

Original: Partitioning Stage.
Revised: Partitioning Stage.

Original: In this stage, we separate the training dataset D into u partitions { D i } u − 1 i = that satisfy (cid:83) u − 1 i =

Revised: In this stage, we separate the training dataset D into u partitions { D i } u − 1 i = that satisfy (cid:83) u − 1 i =

Original: D i =
Revised: D i =

Original: D and ∀ i (cid:54)
Revised: D and ∀ i (cid:54)

Original: =
Revised: =

Original: j, D i ∩ D j = ∅ .
Revised: j, D i ∩ D j = ∅ .

Original: Concretely, when performing partitioning, for each trajectory τ ∈ D , we deterministically assign it to one unique partition.
Revised: Concretely, when performing partitioning, for each trajectory τ ∈ D , we deterministically assign it to one unique partition.

Original: The assignment is only dependent on the trajectory τ itself, and not impacted by any modiﬁcation to other parts of the training set.
Revised: The assignment is only dependent on the trajectory τ itself, and not impacted by any modiﬁcation to other parts of the training set.

Original: One design choice of such a deterministic assignment is using a deterministic hash function h to compute the assignment, i.e. , D i = { τ ∈ D | h ( τ ) ≡ i (mod u ) } , ∀ i ∈ [ u ] .
Revised: One design choice of such a deterministic assignment is using a deterministic hash function h to compute the assignment, i.e. , D i = { τ ∈ D | h ( τ ) ≡ i (mod u ) } , ∀ i ∈ [ u ] .

Original: Training Stage.
Revised: Training Stage.

Original: In this stage, for each training data partition D i , we independently apply an
Revised: In this stage, for each training data partition D i , we independently apply an

Original: RL algorithm M 0 to train a policy π i = M 

Revised: RL algorithm M 0 to train a policy π i = M

Original: ( D i ) .
Revised: ( D i ) .

Original: Hereinafter, we call these trained polices as subpolicies to distinguish from the aggregated policies.
Revised: Hereinafter, we call these trained polices as subpolicies to distinguish from the aggregated policies.

Original: Concretely, let [ u ] := { 0 , 1 , . .
Revised: Concretely, let [ u ] := { 0 , 1 , . .

Original: , u − 1 } .
Revised: , u − 1 } .

Original: For these subpolicies, the policy indicator 1 i,a : S → { 0 , 1 } is deﬁned by 1 i,a ( s ) :
Revised: For these subpolicies, the policy indicator 1 i,a : S → { 0 , 1 } is deﬁned by 1 i,a ( s ) :

Original: = 

Revised: =

Original: [ π i ( s ) = a ] , indicating whether subpolicy π
Revised: [ π i ( s ) = a ] , indicating whether subpolicy π

Original: i chooses action a at state s .
Revised: i chooses action a at state s .

Original: The aggregated action count n a : S → N ≥ 0 is the number of votes across all the subpolicies for action a given state s : n a ( s ) :
Revised: The aggregated action count n a : S → N ≥ 0 is the number of votes across all the subpolicies for action a given state s : n a ( s ) :

Original: = |{ i | π i ( s )
Revised: = |{ i | π i ( s )

Original: = a, i ∈ [ u ] }| = (cid:80)
Revised: = a, i ∈ [ u ] }| = (cid:80)

Original: u − 1 i =

Revised: u − 1 i =

Original: 1 i,a ( s ) .
Revised: 1 i,a ( s ) .

Original: Speciﬁcally, we denote n a ( s l : r ) for (cid:80) rj
Revised: Speciﬁcally, we denote n a ( s l : r ) for (cid:80) rj

Original: = l n a ( s j ) , i.e. , the sum of votes for states between time step l and r .
Revised: = l n a ( s j ) , i.e. , the sum of votes for states between time step l and r .

Original: A detailed algorithm of the training protocol is in Appendix E.1.
Revised: A detailed algorithm of the training protocol is in Appendix E.1.

Original: Now we are ready to introduce the proposed aggregation protocols in COPA (PARL, TPARL,
Revised: Now we are ready to introduce the proposed aggregation protocols in COPA (PARL, TPARL,

Original: DPARL) that generate aggregated policies based on subpolicies, and corresponding certiﬁcation.
Revised: DPARL) that generate aggregated policies based on subpolicies, and corresponding certiﬁcation.

Original: 4. A GGREGATION P ROTOCOLS : PARL, TPARL, DPARL
Revised: 4. A GGREGATION P ROTOCOLS : PARL, TPARL, DPARL

Original: With u learned subpolicies { π i } u − 1 i =0 , we propose three different aggregation protocols in COPA to form three types of aggregated policies for each certiﬁcation criteria: PARL, TPARL, and DPARL.
Revised: With u learned subpolicies { π i } u − 1 i =0 , we propose three different aggregation protocols in COPA to form three types of aggregated policies for each certiﬁcation criteria: PARL, TPARL, and DPARL.

Original: Per-State Partition Aggregation (PARL).
Revised: Per-State Partition Aggregation (PARL).

Original: Inspired by aggregation in classiﬁcation (Levine &
Revised: Inspired by aggregation in classiﬁcation (Levine &

Original: Feizi, 2021), PARL aggregates subpolicies by choosing actions with the highest votes.
Revised: Feizi, 2021), PARL aggregates subpolicies by choosing actions with the highest votes.

Original: We denote the PARL aggregated policy by π P : S → A .
Revised: We denote the PARL aggregated policy by π P : S → A .

Original: When there are multiple highest voting actions, we break ties deterministically by returning the “smaller” ( < ) action, which can be deﬁned by numerical order, lexicographical order, etc.
Revised: When there are multiple highest voting actions, we break ties deterministically by returning the “smaller” ( < ) action, which can be deﬁned by numerical order, lexicographical order, etc.

Original: Throughout the paper, we assume arg max over A always uses < operator to break ties.
Revised: Throughout the paper, we assume arg max over A always uses < operator to break ties.

Original: The formal deﬁnition of the protocol is in Appendix A.
Revised: The formal deﬁnition of the protocol is in Appendix A.

Original: The intuition behind PARL is that the poisoning attack within size K can change at most K subpolicies.
Revised: The intuition behind PARL is that the poisoning attack within size K can change at most K subpolicies.

Original: Therefore, as long as the margin between the votes for top and runner-up actions is larger than 2 K for the given state, after poisoning, we can guarantee that the aggregated PARL policy will not change its action choice.
Revised: Therefore, as long as the margin between the votes for top and runner-up actions is larger than 2 K for the given state, after poisoning, we can guarantee that the aggregated PARL policy will not change its action choice.

Original: We will formally state the robustness guarantee in Section 4.3.
Revised: We will formally state the robustness guarantee in Section 4.3.

Original: Temporal Partition Aggregation (TPARL).
Revised: Temporal Partition Aggregation (TPARL).

Original: In the sequential decision making process of RL, it is likely that certain important states are much more vulnerable to poisoning attacks, which we refer to as bottleneck states .
Revised: In the sequential decision making process of RL, it is likely that certain important states are much more vulnerable to poisoning attacks, which we refer to as bottleneck states .

Original: Therefore, the attacker may just change the action predictions for these bottleneck states to deteriorate the overall performance, say, the cumulative reward.
Revised: Therefore, the attacker may just change the action predictions for these bottleneck states to deteriorate the overall performance, say, the cumulative reward.

Original: For example, in Pong game, we may lose the round when choosing an immediate bad action when the ball is closely approaching the paddle.
Revised: For example, in Pong game, we may lose the round when choosing an immediate bad action when the ball is closely approaching the paddle.

Original: Thus, to improve the overall certiﬁed robustness, we need to focus on improving the tolerable poisoning threshold for these bottleneck states.
Revised: Thus, to improve the overall certiﬁed robustness, we need to focus on improving the tolerable poisoning threshold for these bottleneck states.

Original: Given such intuition and goal, we propose Temporal Partition Aggregation (TPARL) and the aggregated policy is denoted as π T , which is formally deﬁned in Deﬁnition 8 in Appendix A.
Revised: Given such intuition and goal, we propose Temporal Partition Aggregation (TPARL) and the aggregated policy is denoted as π T , which is formally deﬁned in Deﬁnition 8 in Appendix A.

Original: TPARL is based on two insights: (1) Bottleneck states have lower tolerable poisoning threshold, which is because the vote margin between the top and runner-up actions is smaller at such state; (2) Some RL tasks satisfy temporal continuity (Legenstein et al., 2010; Veerapaneni et al., 2020), indicating that good action choices are usually similar across states of adjacent time steps, i.e. , adjacent states.
Revised: TPARL is based on two insights: (1) Bottleneck states have lower tolerable poisoning threshold, which is because the vote margin between the top and runner-up actions is smaller at such state; (2) Some RL tasks satisfy temporal continuity (Legenstein et al., 2010; Veerapaneni et al., 2020), indicating that good action choices are usually similar across states of adjacent time steps, i.e. , adjacent states.

Original: Hence, we leverage the subpolicies’ votes from adjacent states to enlarge the vote margin, and thus increase the tolerable poisoning threshold.
Revised: Hence, we leverage the subpolicies’ votes from adjacent states to enlarge the vote margin, and thus increase the tolerable poisoning threshold.

Original: To this end, in TPARL, we predetermine a window size W , and choose the action with the highest votes across recent W states.
Revised: To this end, in TPARL, we predetermine a window size W , and choose the action with the highest votes across recent W states.

Original: Dynamic Temporal Partition Aggregation (DPARL).
Revised: Dynamic Temporal Partition Aggregation (DPARL).

Original: The TPARL uses a ﬁxed window size W across all states.
Revised: The TPARL uses a ﬁxed window size W across all states.

Original: Since the speciﬁcation of the window size W requires certain prior knowledge, plus that the same ﬁxed window size W may not be suitable for all states, it is preferable to perform dynamic temporal aggregation by using a ﬂexible window size.
Revised: Since the speciﬁcation of the window size W requires certain prior knowledge, plus that the same ﬁxed window size W may not be suitable for all states, it is preferable to perform dynamic temporal aggregation by using a ﬂexible window size.

Original: Therefore, we propose Dynamic
Revised: Therefore, we propose Dynamic

Original: Temporal Partition Aggregation (DPARL), which dynamically selects the window size W towards maximizing the tolerable poisoning threshold per step .
Revised: Temporal Partition Aggregation (DPARL), which dynamically selects the window size W towards maximizing the tolerable poisoning threshold per step .

Original: Intuitively, DPARL selects the window size
Revised: Intuitively, DPARL selects the window size

Original: W such that the average vote margin over selected states is maximized.
Revised: W such that the average vote margin over selected states is maximized.

Original: To guarantee that only recent states are chosen, we further constrain the maximum window size ( W max ).
Revised: To guarantee that only recent states are chosen, we further constrain the maximum window size ( W max ).

Original: Deﬁnition 3 (Dynamic Temporal Partition Aggregation) .
Revised: Deﬁnition 3 (Dynamic Temporal Partition Aggregation) .

Original: Given subpolicies { π i } u − 1 i =0 and maximum window size W max , at time step t , the Dynamic Temporal Partition Aggregation (DPARL) deﬁnes an aggregated policy π D : S min { t +1 ,W max } →
Revised: Given subpolicies { π i } u − 1 i =0 and maximum window size W max , at time step t , the Dynamic Temporal Partition Aggregation (DPARL) deﬁnes an aggregated policy π D : S min { t +1 ,W max } →

Original: A such that
Revised: A such that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: t ) , where W
Revised: t ) , where W

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: In the above equation, n a is deﬁned in Section 3 and ∆ Wt is given by
Revised: In the above equation, n a is deﬁned in Section 3 and ∆ Wt is given by

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: In the above deﬁnition, ∆ Wt encodes the average vote margin between top action a 1 and runner-up action a 2 if choosing window size W .
Revised: In the above deﬁnition, ∆ Wt encodes the average vote margin between top action a 1 and runner-up action a 2 if choosing window size W .

Original: Thus, W (cid:48) locates the window size with maximum average vote margin, and its corresponding action is selected.
Revised: Thus, W (cid:48) locates the window size with maximum average vote margin, and its corresponding action is selected.

Original: Again, we use the mechanism described in
Revised: Again, we use the mechanism described in

Original: PARL to break ties.
Revised: PARL to break ties.

Original: Robustness certiﬁcation methods for DPARL are in Sections 4.3 and 4.4.
Revised: Robustness certiﬁcation methods for DPARL are in Sections 4.3 and 4.4.

Original: In Appendix B, we present a concrete example to demonstrate how different aggregation protocols induce different tolerable poisoning thresholds, and illustrate bottleneck and non-bottleneck states.
Revised: In Appendix B, we present a concrete example to demonstrate how different aggregation protocols induce different tolerable poisoning thresholds, and illustrate bottleneck and non-bottleneck states.

Original: 4. C ERTIFICATION OF P ER -S TATE A CTION S TABILITY
Revised: 4. C ERTIFICATION OF P ER -S TATE A CTION S TABILITY

Original: In this section, we present our robustness certiﬁcation theorems and methods for per-state action .
Revised: In this section, we present our robustness certiﬁcation theorems and methods for per-state action .

Original: For each of the aggregation protocols (PARL, TPARL, and DPARL), at each time step t , we will compute a valid tolerable poisoning threshold K as deﬁned in Deﬁnition 1, such that the chosen action at step t does not change as long as the poisoning size K ≤ K .
Revised: For each of the aggregation protocols (PARL, TPARL, and DPARL), at each time step t , we will compute a valid tolerable poisoning threshold K as deﬁned in Deﬁnition 1, such that the chosen action at step t does not change as long as the poisoning size K ≤ K .

Original: Certiﬁcation for PARL.
Revised: Certiﬁcation for PARL.

Original: Due to the space limit, we defer the robustness certiﬁcation method for
Revised: Due to the space limit, we defer the robustness certiﬁcation method for

Original: PARL to Appendix C.
Revised: PARL to Appendix C.

Original: The certiﬁcation method is based on Theorem 8.
Revised: The certiﬁcation method is based on Theorem 8.

Original: We further show the theoretical tightness of the certiﬁcation in Proposition 9.
Revised: We further show the theoretical tightness of the certiﬁcation in Proposition 9.

Original: All the theorem statements are in Appendix C.
Revised: All the theorem statements are in Appendix C.

Original: Certiﬁcation for TPARL.
Revised: Certiﬁcation for TPARL.

Original: We certify the robustness of TPARL following Theorem 1.
Revised: We certify the robustness of TPARL following Theorem 1.

Original: Theorem 1.
Revised: Theorem 1.

Original: Let D be the clean training dataset; let π i
Revised: Let D be the clean training dataset; let π i

Original: =
Revised: =

Original: M 

Revised: M

Original: ( D i ) , 0 ≤ i ≤ u − 1 be the learned subpolicies according to Section 4.1 from which we deﬁne n a (Section 3); and let π T be the Temporal
Revised: ( D i ) , 0 ≤ i ≤ u − 1 be the learned subpolicies according to Section 4.1 from which we deﬁne n a (Section 3); and let π T be the Temporal

Original: Partition Aggregation policy: π T =
Revised: Partition Aggregation policy: π T =

Original: M ( D ) where M abstracts the whole training-aggregation process.
Revised: M ( D ) where M abstracts the whole training-aggregation process.

Original: (cid:101) D is a poisoned dataset and (cid:102)
Revised: (cid:101) D is a poisoned dataset and (cid:102)

Original: π T is the poisoned policy: (cid:102) π T =
Revised: π T is the poisoned policy: (cid:102) π T =

Original: M ( (cid:101) D ) .
Revised: M ( (cid:101) D ) .

Original: For a given state s t encountered at time step t during test time, let a :=
Revised: For a given state s t encountered at time step t during test time, let a :=

Original: π T ( s max { t − W +1 , 0 } : t ) , then at time step t the tolerable poisoning threshold (see Deﬁnition 1)
Revised: π T ( s max { t − W +1 , 0 } : t ) , then at time step t the tolerable poisoning threshold (see Deﬁnition 1)

Original: [Equation]
Revised: [Equation]

Original: where { h ( i ) a,a (cid:48) } ui =1 is a nonincreasing permutation of   min { W − 1 ,t } (cid:88)
Revised: where { h ( i ) a,a (cid:48) } ui =1 is a nonincreasing permutation of   min { W − 1 ,t } (cid:88)

Original: j = 1 i,a ( s t − j ) + min { W, t + 1 } − min { W − 1 ,t } (cid:88)
Revised: j = 1 i,a ( s t − j ) + min { W, t + 1 } − min { W − 1 ,t } (cid:88)

Original: j = 1 i,a (cid:48) ( s t − j )   u −i =
Revised: j = 1 i,a (cid:48) ( s t − j )   u −i =

Original: =: { h i,a,a (cid:48) } u − 1 i =0 , (4) and δ a,a (cid:48) :
Revised: =: { h i,a,a (cid:48) } u − 1 i =0 , (4) and δ a,a (cid:48) :

Original: = n a ( s max { t − W +1 , 0 } : t ) − ( n a (cid:48) ( s max { t − W +1 , 0 } : t ) + 1 [ a (cid:48) < a ]) .
Revised: = n a ( s max { t − W +1 , 0 } : t ) − ( n a (cid:48) ( s max { t − W +1 , 0 } : t ) + 1 [ a (cid:48) < a ]) .

Original: Here, 1 i,a ( s ) = 1 [ π i ( s ) =
Revised: Here, 1 i,a ( s ) = 1 [ π i ( s ) =

Original: a ] (Section 3), and W is the window size.
Revised: a ] (Section 3), and W is the window size.

Original: Remark.
Revised: Remark.

Original: We defer the detailed proof to Appendix F.2.
Revised: We defer the detailed proof to Appendix F.2.

Original: The theorem provides a per-state action certiﬁcation for TPARL.
Revised: The theorem provides a per-state action certiﬁcation for TPARL.

Original: The detailed algorithm is in Algorithm 3 (Appendix E.2).
Revised: The detailed algorithm is in Algorithm 3 (Appendix E.2).

Original: The certiﬁcation time complexity per state is O ( |A| u ( W +log u )) and can be further optimized to O ( |A| u log u ) with proper preﬁx sum caching across time steps.
Revised: The certiﬁcation time complexity per state is O ( |A| u ( W +log u )) and can be further optimized to O ( |A| u log u ) with proper preﬁx sum caching across time steps.

Original: We prove the certiﬁcation for TPARL is theoretically tight in Proposition 2 (proof in Appendix F.3).
Revised: We prove the certiﬁcation for TPARL is theoretically tight in Proposition 2 (proof in Appendix F.3).

Original: We also prove that directly extending Theorem 8 for
Revised: We also prove that directly extending Theorem 8 for

Original: TPARL (Corollary 10) is loose in Appendix F.4.
Revised: TPARL (Corollary 10) is loose in Appendix F.4.

Original: Proposition 2.
Revised: Proposition 2.

Original: Under the same condition as Theorem 1, for any time step t , there exists an
Revised: Under the same condition as Theorem 1, for any time step t , there exists an

Original: RL learning algorithm M 0 , and a poisoned dataset (cid:101) D , such that | D (cid:9) (cid:101) D | =
Revised: RL learning algorithm M 0 , and a poisoned dataset (cid:101) D , such that | D (cid:9) (cid:101) D | =

Original: K t + 1 , and (cid:102) π T ( s
Revised: K t + 1 , and (cid:102) π T ( s

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) (cid:54) =
Revised: ) (cid:54) =

Original: π
Revised: π

Original: [Equation]
Revised: [Equation]

Original: ( s
Revised: ( s

Original: [Equation]
Revised: [Equation]

Original: { t − W +1 , 0 } : t ) .
Revised: { t − W +1 , 0 } : t ) .

Original: Certiﬁcation for DPARL.
Revised: Certiﬁcation for DPARL.

Original: Theorem 3 provides certiﬁcation for DPARL.
Revised: Theorem 3 provides certiﬁcation for DPARL.

Original: Theorem 3.
Revised: Theorem 3.

Original: Let D be the clean training dataset; let π i
Revised: Let D be the clean training dataset; let π i

Original: =
Revised: =

Original: M 

Revised: M

Original: ( D i ) , 0 ≤ i ≤ u − 1 be the learned subpolicies according to Section 4.1 from which we deﬁne n a (see Section 3); and let π D be the
Revised: ( D i ) , 0 ≤ i ≤ u − 1 be the learned subpolicies according to Section 4.1 from which we deﬁne n a (see Section 3); and let π D be the

Original: Dynamic Temporal Partition Aggregation: π D = M
Revised: Dynamic Temporal Partition Aggregation: π D = M

Original: ( D ) where M abstracts the whole trainingaggregation process.
Revised: ( D ) where M abstracts the whole trainingaggregation process.

Original: (cid:101) D is a poisoned dataset and (cid:102)
Revised: (cid:101) D is a poisoned dataset and (cid:102)

Original: π D is the poisoned policy: (cid:102) π D = M
Revised: π D is the poisoned policy: (cid:102) π D = M

Original: ( (cid:101) D ) .
Revised: ( (cid:101) D ) .

Original: For a given state s t encountered at time step t during test time, let a :=
Revised: For a given state s t encountered at time step t during test time, let a :=

Original: π D ( s max { t − W max +1 , 0 } : t ) and W (cid:48) be the chosen time window (according to Equation (1) ), then tolerable poisoning threshold
Revised: π D ( s max { t − W max +1 , 0 } : t ) and W (cid:48) be the chosen time window (according to Equation (1) ), then tolerable poisoning threshold

Original: [Equation]
Revised: [Equation]

Original: where K t is deﬁned by Equation (3) with W as W (cid:48) and L W ∗ ,W (cid:48) a,a (cid:48)(cid:48) deﬁned by the below Deﬁnition 4.
Revised: where K t is deﬁned by Equation (3) with W as W (cid:48) and L W ∗ ,W (cid:48) a,a (cid:48)(cid:48) deﬁned by the below Deﬁnition 4.

Original: Deﬁnition 4 ( L in Theorem 3) .
Revised: Deﬁnition 4 ( L in Theorem 3) .

Original: Under the same condition as Theorem 3, for given W ∗ , W (cid:48) , a, a (cid:48) , a (cid:48)(cid:48) , we let a # := arg max a 0 (cid:54) = a (cid:48) ,a 0 ∈A n a 0 ( s t − W ∗
Revised: Under the same condition as Theorem 3, for given W ∗ , W (cid:48) , a, a (cid:48) , a (cid:48)(cid:48) , we let a # := arg max a 0 (cid:54) = a (cid:48) ,a 0 ∈A n a 0 ( s t − W ∗

Original: +1: t ) , then
Revised: +1: t ) , then

Original: L W ∗ ,
Revised: L W ∗ ,

Original: [Equation]
Revised: [Equation]

Original: where n wa is a shorthand of n a ( s t − w +1: t ) and { g ( i ) } ui =1 is a nonincreasing permutation of { g i } u − 1 i =0 .
Revised: where n wa is a shorthand of n a ( s t − w +1: t ) and { g ( i ) } ui =1 is a nonincreasing permutation of { g i } u − 1 i =0 .

Original: Each g i is deﬁned by
Revised: Each g i is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: , where
Revised: , where

Original: [Equation]
Revised: [Equation]

Original: Proof sketch.
Revised: Proof sketch.

Original: A successful poisoning attack should change the chosen action from a to an another a (cid:48) .
Revised: A successful poisoning attack should change the chosen action from a to an another a (cid:48) .

Original: If after attack, the chosen window size is still W (cid:48) , the poisoning size should be at least larger than
Revised: If after attack, the chosen window size is still W (cid:48) , the poisoning size should be at least larger than

Original: K t according to Theorem 1.
Revised: K t according to Theorem 1.

Original: If the chosen window size is not W (cid:48) , we ﬁnd out that the poisoning size is at least min a (cid:48)(cid:48) (cid:54) =
Revised: If the chosen window size is not W (cid:48) , we ﬁnd out that the poisoning size is at least min a (cid:48)(cid:48) (cid:54) =

Original: a L W ∗ ,W (cid:48)
Revised: a L W ∗ ,W (cid:48)

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: ,a
Revised: ,a

Original: [Equation]
Revised: [Equation]

Original: +1 from a greedy-based analysis.
Revised: +1 from a greedy-based analysis.

Original: Formal proof is in Appendix F.5.
Revised: Formal proof is in Appendix F.5.

Original: Remark.
Revised: Remark.

Original: The theorem provides a valid per-state action certiﬁcation for DPARL policy.
Revised: The theorem provides a valid per-state action certiﬁcation for DPARL policy.

Original: The detailed algorithm is in Algorithm 4 (Appendix E.2).
Revised: The detailed algorithm is in Algorithm 4 (Appendix E.2).

Original: The certiﬁcation time complexity per state is
Revised: The certiﬁcation time complexity per state is

Original: O ( W 2max |A| 2 u + W max |A| 2 u log u ) , which in practice adds similar overhead compared with TPARL certiﬁcation (see Appendix H.3).
Revised: O ( W 2max |A| 2 u + W max |A| 2 u log u ) , which in practice adds similar overhead compared with TPARL certiﬁcation (see Appendix H.3).

Original: Unlike certiﬁcation for PARL and TPARL, the certiﬁcation given by Theorem 3 is not theoretically tight.
Revised: Unlike certiﬁcation for PARL and TPARL, the certiﬁcation given by Theorem 3 is not theoretically tight.

Original: An interesting future work would be providing a tighter per-state action certiﬁcation for DPARL.
Revised: An interesting future work would be providing a tighter per-state action certiﬁcation for DPARL.

Original: 4. C ERTIFICATION OF C UMULATIVE R EWARD B OUND
Revised: 4. C ERTIFICATION OF C UMULATIVE R EWARD B OUND

Original: In this section, we present our robustness certiﬁcation for cumulative reward bound .
Revised: In this section, we present our robustness certiﬁcation for cumulative reward bound .

Original: We assume the deterministic RL environment throughout the cumulative reward certiﬁcation for convenience, i.e. , the transition function is P : S×A → S and the initial state is a ﬁxed s 0 ∈ S .
Revised: We assume the deterministic RL environment throughout the cumulative reward certiﬁcation for convenience, i.e. , the transition function is P : S×A → S and the initial state is a ﬁxed s 0 ∈ S .

Original: The certiﬁcation goal, as listed in Deﬁnition 6, is to obtain a lower bound of cumulative reward under poisoning attacks, given bounded poisoning size K .
Revised: The certiﬁcation goal, as listed in Deﬁnition 6, is to obtain a lower bound of cumulative reward under poisoning attacks, given bounded poisoning size K .

Original: The cumulative reward certiﬁcation is based on a novel adaptive search algorithm COPA-S EARCH inspired from Wu et al.
Revised: The cumulative reward certiﬁcation is based on a novel adaptive search algorithm COPA-S EARCH inspired from Wu et al.

Original: ; we tailor the algorithm to certify against poisoning attacks.
Revised: ; we tailor the algorithm to certify against poisoning attacks.

Original: We defer detailed discussions and complexity analysis to Appendix E.3.
Revised: We defer detailed discussions and complexity analysis to Appendix E.3.

Original: COPA-S EARCH Algorithm Description.
Revised: COPA-S EARCH Algorithm Description.

Original: The pseudocode is in Algorithm 5 (Appendix E.3).
Revised: The pseudocode is in Algorithm 5 (Appendix E.3).

Original: The method starts from the base case: when the poisoning threshold K cur = 0 , the lower bound of cumulative reward J K cur is exactly the reward without poisoning.
Revised: The method starts from the base case: when the poisoning threshold K cur = 0 , the lower bound of cumulative reward J K cur is exactly the reward without poisoning.

Original: The method then gradually increases the poisoning threshold K cur , by ﬁnding the immediate larger K (cid:48) >
Revised: The method then gradually increases the poisoning threshold K cur , by ﬁnding the immediate larger K (cid:48) >

Original: K cur that can expand the possible action set along the trajectory.
Revised: K cur that can expand the possible action set along the trajectory.

Original: With the increase of K cur ← K (cid:48) , the attack may cause the poisoned policy ˜ π to take different actions at some states, thus resulting in new trajectories.
Revised: With the increase of K cur ← K (cid:48) , the attack may cause the poisoned policy ˜ π to take different actions at some states, thus resulting in new trajectories.

Original: We need to ﬁgure out a set of all possible actions to exhaustively traverse all possible trajectories.
Revised: We need to ﬁgure out a set of all possible actions to exhaustively traverse all possible trajectories.

Original: We will introduce theorems to compute this set of possible actions.
Revised: We will introduce theorems to compute this set of possible actions.

Original: With this set, the method effectively explores these new trajectories by formulating them as expanded branches of a trajectory tree.
Revised: With this set, the method effectively explores these new trajectories by formulating them as expanded branches of a trajectory tree.

Original: Once all new trajectories are explored, the method examines all leaf nodes of the tree and ﬁgures out the minimum reward among them, which is the new lower bound of cumulative reward J K (cid:48) under new poisoning size K (cid:48) .
Revised: Once all new trajectories are explored, the method examines all leaf nodes of the tree and ﬁgures out the minimum reward among them, which is the new lower bound of cumulative reward J K (cid:48) under new poisoning size K (cid:48) .

Original: We then repeat this process of increasing poisoning size from K (cid:48) and expanding with new trajectories until we reach a predeﬁned threshold for poisoning size K .
Revised: We then repeat this process of increasing poisoning size from K (cid:48) and expanding with new trajectories until we reach a predeﬁned threshold for poisoning size K .

Original: Deﬁnition 5 (Possible Action Set) .
Revised: Deﬁnition 5 (Possible Action Set) .

Original: Given previous states s 0: t , the subpolicies { π i } u − 1 i =0 , the aggregation protocol (PARL, TPARL, or DPARL), and the poisoning size K , the possible action set A at step t is a subset of action space:
Revised: Given previous states s 0: t , the subpolicies { π i } u − 1 i =0 , the aggregation protocol (PARL, TPARL, or DPARL), and the poisoning size K , the possible action set A at step t is a subset of action space:

Original: A ⊆ A , such that for any poisoned policy (cid:101) π , as long as the poisoning size is within K , the chosen action at step t will always be in A , i.e. , a t = (cid:101) π
Revised: A ⊆ A , such that for any poisoned policy (cid:101) π , as long as the poisoning size is within K , the chosen action at step t will always be in A , i.e. , a t = (cid:101) π

Original: ( s 0: t )
Revised: ( s 0: t )

Original: ∈
Revised: ∈

Original: A .
Revised: A .

Original: Possible Action Set for PARL.
Revised: Possible Action Set for PARL.

Original: The following theorem gives the possible action set for PARL.
Revised: The following theorem gives the possible action set for PARL.

Original: Theorem 4 (Tight PARL Action Set) .
Revised: Theorem 4 (Tight PARL Action Set) .

Original: Under the condition of Deﬁnition 5, suppose the aggregation protocol is PARL as deﬁned in Deﬁnition 7,
Revised: Under the condition of Deﬁnition 5, suppose the aggregation protocol is PARL as deﬁned in Deﬁnition 7,

Original: then the possible action set at step t
Revised: then the possible action set at step t

Original: [Equation]
Revised: [Equation]

Original: We defer the proof to Appendix F.6.1.
Revised: We defer the proof to Appendix F.6.1.

Original: Furthermore, in Appendix F.6.2 we show that: 1)
Revised: Furthermore, in Appendix F.6.2 we show that: 1)

Original: The theorem gives theoretically tight possible action set; 2)
Revised: The theorem gives theoretically tight possible action set; 2)

Original: In contrast, directly extending PARL’s per-state certiﬁcation gives loose certiﬁcation.
Revised: In contrast, directly extending PARL’s per-state certiﬁcation gives loose certiﬁcation.

Original: Possible Action Set for TPARL.
Revised: Possible Action Set for TPARL.

Original: The following theorem gives the possible action set for TPARL.
Revised: The following theorem gives the possible action set for TPARL.

Original: Theorem 5.
Revised: Theorem 5.

Original: Under the condition of Deﬁnition 5, suppose the aggregation protocol is TPARL as deﬁned in Deﬁnition 8, then the possible action set at step t
Revised: Under the condition of Deﬁnition 5, suppose the aggregation protocol is TPARL as deﬁned in Deﬁnition 8, then the possible action set at step t

Original: [Equation]
Revised: [Equation]

Original: where h ( i ) a (cid:48) ,a and δ a (cid:48) ,a follow the deﬁnition in Theorem 1.
Revised: where h ( i ) a (cid:48) ,a and δ a (cid:48) ,a follow the deﬁnition in Theorem 1.

Original: We defer the proof to Appendix F.7.
Revised: We defer the proof to Appendix F.7.

Original: The possible action set here is no longer theoretically tight.
Revised: The possible action set here is no longer theoretically tight.

Original: Indeed, the problem of computing a possible action set with minimum cardinality for TPARL is
Revised: Indeed, the problem of computing a possible action set with minimum cardinality for TPARL is

Original: NP -complete as we shown in the following theorem (proved in Appendix F.8), where we reduce computing theoretically tight possible action set to the set cover problem (Karp, 1972).
Revised: NP -complete as we shown in the following theorem (proved in Appendix F.8), where we reduce computing theoretically tight possible action set to the set cover problem (Karp, 1972).

Original: This result can be viewed as the hardness of targeted attack.
Revised: This result can be viewed as the hardness of targeted attack.

Original: In other words, the optimal untargeted attack on TPARL can be found in polynomial time, while the optimal targeted attack on TPARL is NPcomplete, which indicates the robustness property of proposed TPARL.
Revised: In other words, the optimal untargeted attack on TPARL can be found in polynomial time, while the optimal targeted attack on TPARL is NPcomplete, which indicates the robustness property of proposed TPARL.

Original: Theorem 6.
Revised: Theorem 6.

Original: Under the condition of Deﬁnition 5, suppose we use TPARL (Deﬁnition 8) as the aggregation protocol, then computing a possible action set
Revised: Under the condition of Deﬁnition 5, suppose we use TPARL (Deﬁnition 8) as the aggregation protocol, then computing a possible action set

Original: A ( K ) such that any possible action set S satisﬁes | A ( K ) | ≤ | S | is NP -complete.
Revised: A ( K ) such that any possible action set S satisﬁes | A ( K ) | ≤ | S | is NP -complete.

Original: Possible Action Set for DPARL.
Revised: Possible Action Set for DPARL.

Original: The following theorem gives the possible action set for DPARL.
Revised: The following theorem gives the possible action set for DPARL.

Original: Theorem 7.
Revised: Theorem 7.

Original: Under the condition of Deﬁnition 5, suppose the aggregation protocol is TPARL as deﬁned in Deﬁnition 3, then the possible action set at step t
Revised: Under the condition of Deﬁnition 5, suppose the aggregation protocol is TPARL as deﬁned in Deﬁnition 3, then the possible action set at step t

Original: A ( K ) =
Revised: A ( K ) =

Original: { a t } ∪
Revised: { a t } ∪

Original: [Equation]
Revised: [Equation]

Original: where a t = π D ( s max { t − W max +1 , 0 } : t ) is the clean policy’s chosen action, W (cid:48) is deﬁned by Equation (1) , L W ∗ ,W (cid:48) a (cid:48) ,a (cid:48)(cid:48) is deﬁned by Deﬁnition 4 with a being replaced by a t , and h ( i ) a (cid:48) ,a , δ a (cid:48) ,a is deﬁned in
Revised: where a t = π D ( s max { t − W max +1 , 0 } : t ) is the clean policy’s chosen action, W (cid:48) is deﬁned by Equation (1) , L W ∗ ,W (cid:48) a (cid:48) ,a (cid:48)(cid:48) is deﬁned by Deﬁnition 4 with a being replaced by a t , and h ( i ) a (cid:48) ,a , δ a (cid:48) ,a is deﬁned in

Original: Theorem 1 with W replaced by W (cid:48) .
Revised: Theorem 1 with W replaced by W (cid:48) .

Original: We prove the theorem in Appendix F.8.
Revised: We prove the theorem in Appendix F.8.

Original: As summarized in Table 1, we further prove the tightness or hardness of certiﬁcation for PARL and TPARL, while for DPARL it is an interesting open problem on whether theoretical tight certiﬁcation is possible in polynomial time.
Revised: As summarized in Table 1, we further prove the tightness or hardness of certiﬁcation for PARL and TPARL, while for DPARL it is an interesting open problem on whether theoretical tight certiﬁcation is possible in polynomial time.

Original: [Figure]
Revised: [Figure]

Original: denoting the standard deviation.
Revised: denoting the standard deviation.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we present the evaluation for our COPA framework, speciﬁcally, the aggregation protocols (Section 4.2) and the certiﬁcation methods under different certiﬁcation criteria (Sections 4.3 and 4.4).
Revised: In this section, we present the evaluation for our COPA framework, speciﬁcally, the aggregation protocols (Section 4.2) and the certiﬁcation methods under different certiﬁcation criteria (Sections 4.3 and 4.4).

Original: We defer the description of the ofﬂine RL algorithms (DQN (Mnih et al., 2013),
Revised: We defer the description of the ofﬂine RL algorithms (DQN (Mnih et al., 2013),

Original: QR-DQN (Dabney et al., 2018), and C51 (Bellemare et al., 2017)) used for training the subpolicies to Appendix G.1, and the concrete experimental procedures to Appendix G.2.
Revised: QR-DQN (Dabney et al., 2018), and C51 (Bellemare et al., 2017)) used for training the subpolicies to Appendix G.1, and the concrete experimental procedures to Appendix G.2.

Original: As a summary, we obtain similar conclusions from per-state action certiﬁcation and reward certiﬁcation: 1) QR-DQN and C51 are oftentimes more certiﬁably robust than DQN; 2) temporal aggregation (TPARL and DPARL) achieves higher certiﬁcation for environments satisfying temporal continuity, e.g. ,
Revised: As a summary, we obtain similar conclusions from per-state action certiﬁcation and reward certiﬁcation: 1) QR-DQN and C51 are oftentimes more certiﬁably robust than DQN; 2) temporal aggregation (TPARL and DPARL) achieves higher certiﬁcation for environments satisfying temporal continuity, e.g. ,

Original: Freeway; 3) larger partition number improves the certiﬁed robustness; 4) Freeway is the most stable and robust environment among the three.
Revised: Freeway; 3) larger partition number improves the certiﬁed robustness; 4) Freeway is the most stable and robust environment among the three.

Original: More interesting discussions are deferred to Appendix H.
Revised: More interesting discussions are deferred to Appendix H.

Original: 5. E VALUATION OF R OBUSTNESS C ERTIFICATION FOR P ER -S TATE A CTION S TABILITY
Revised: 5. E VALUATION OF R OBUSTNESS C ERTIFICATION FOR P ER -S TATE A CTION S TABILITY

Original: We provide the robustness certiﬁcation for per-state action stability based on Section 4.3.
Revised: We provide the robustness certiﬁcation for per-state action stability based on Section 4.3.

Original: Experimental Setup and Metrics.
Revised: Experimental Setup and Metrics.

Original: We evaluate the aggregated policies π P , π T , and π D following
Revised: We evaluate the aggregated policies π P , π T , and π D following

Original: Section 4.2.
Revised: Section 4.2.

Original: Basically, in each run, we run one trajectory (of maximum length H ) using the derived policy, and compute K t at each time step t .
Revised: Basically, in each run, we run one trajectory (of maximum length H ) using the derived policy, and compute K t at each time step t .

Original: Given { K t } H − 1 t =0 , we obtain a cumulative histogram— for each threshold K , we count the time steps that achieve a threshold no smaller than it and then normalize,
Revised: Given { K t } H − 1 t =0 , we obtain a cumulative histogram— for each threshold K , we count the time steps that achieve a threshold no smaller than it and then normalize,

Original: i.e. , (cid:80) H − 1 t =0 

Revised: i.e. , (cid:80) H − 1 t =0

Original: [ K t ≥ K ] / H .
Revised: [ K t ≥ K ] / H .

Original: We call this quantity stability ratio since it reﬂects the per-state action stability w.r.t.
Revised: We call this quantity stability ratio since it reﬂects the per-state action stability w.r.t.

Original: given poisoning thresholds.
Revised: given poisoning thresholds.

Original: We also compute an average tolerable poisoning thresholds for a trajectory, deﬁned as (cid:80) H − 1 t =

Revised: We also compute an average tolerable poisoning thresholds for a trajectory, deﬁned as (cid:80) H − 1 t =

Original: K t / H .
Revised: K t / H .

Original: More details are deferred to Appendix G.2.
Revised: More details are deferred to Appendix G.2.

Original: Evaluation Results.
Revised: Evaluation Results.

Original: We present the comparison of per-state action certiﬁcation for different RL methods and certiﬁcation methods in Figure 1.
Revised: We present the comparison of per-state action certiﬁcation for different RL methods and certiﬁcation methods in Figure 1.

Original: We plot partial poisoning thresholds on the x -axes here, and omit full results in Appendix H.6, where we also report the average tolerable poisoning thresholds.
Revised: We plot partial poisoning thresholds on the x -axes here, and omit full results in Appendix H.6, where we also report the average tolerable poisoning thresholds.

Original: We additionally report benign empirical reward and the comparisons with standard training in Appendices H.1 and H.2, as well as more analytical statistics in Appendices H.3 and H.4.
Revised: We additionally report benign empirical reward and the comparisons with standard training in Appendices H.1 and H.2, as well as more analytical statistics in Appendices H.3 and H.4.

Original: The cumulative histograms in Figure 1 can be compared in different levels.
Revised: The cumulative histograms in Figure 1 can be compared in different levels.

Original: Basically, we compare the stability ratio at each tolerable poisoning thresholds K — higher ratio at larger poisoning size indicates stronger certiﬁed robustness .
Revised: Basically, we compare the stability ratio at each tolerable poisoning thresholds K — higher ratio at larger poisoning size indicates stronger certiﬁed robustness .

Original: On the RL algorithm level, QR-DQN and C51 consistently outperform the baseline DQN, and C51 has a substantial advantage particularly in Highway.
Revised: On the RL algorithm level, QR-DQN and C51 consistently outperform the baseline DQN, and C51 has a substantial advantage particularly in Highway.

Original: On the aggregation protocol level, we observe different behaviors in different environments.
Revised: On the aggregation protocol level, we observe different behaviors in different environments.

Original: On Freeway, methods with temporal aggregation (TPARL and DPARL) achieve higher robustness, and DPARL achieves the highest certiﬁed robustness in most cases; while on Breakout and Highway, the singlestep aggregation PARL is oftentimes better.
Revised: On Freeway, methods with temporal aggregation (TPARL and DPARL) achieve higher robustness, and DPARL achieves the highest certiﬁed robustness in most cases; while on Breakout and Highway, the singlestep aggregation PARL is oftentimes better.

Original: This difference is due to the different properties of environments.
Revised: This difference is due to the different properties of environments.

Original: Our temporal aggregation is developed based on the assumption of consistent action
Revised: Our temporal aggregation is developed based on the assumption of consistent action

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: selection in adjacent time steps.
Revised: selection in adjacent time steps.

Original: This assumption is true in Freeway while violated in Breakout and
Revised: This assumption is true in Freeway while violated in Breakout and

Original: Highway.
Revised: Highway.

Original: A more detailed explanation of environment properties is omitted to Appendix H.5.
Revised: A more detailed explanation of environment properties is omitted to Appendix H.5.

Original: On the partition number level, a larger partition number generally allows larger tolerable poisoning thresholds as shown in Appendix H.6.
Revised: On the partition number level, a larger partition number generally allows larger tolerable poisoning thresholds as shown in Appendix H.6.

Original: Finally, on the RL environment level, Freeway achieves much higher certiﬁed robustness for per-state action stability than Highway, followed by Breakout, implying that Freeway is an environment that accommodates more stable and robust policies [Figure]
Revised: Finally, on the RL environment level, Freeway achieves much higher certiﬁed robustness for per-state action stability than Highway, followed by Breakout, implying that Freeway is an environment that accommodates more stable and robust policies.

Original: 5. E VALUATION OF R OBUSTNESS C ERTIFICATION FOR C UMULATIVE R EWARD B OUND
Revised: 5. E VALUATION OF R OBUSTNESS C ERTIFICATION FOR C UMULATIVE R EWARD B OUND

Original: We provide the robustness certiﬁcation for cumulative reward bound according to Section 4.4.
Revised: We provide the robustness certiﬁcation for cumulative reward bound according to Section 4.4.

Original:  and Metrics.
Revised: Experimental Setup and Metrics.

Original: We evaluate the aggregated policies π P , π T , and π D following Theorem 4, Theorem 5 and Theorem 7.
Revised: We evaluate the aggregated policies π P , π T , and π D following Theorem 4, Theorem 5 and Theorem 7.

Original: We compute the lower bounds of the cumulative reward
Revised: We compute the lower bounds of the cumulative reward

Original: J K w.r.t.
Revised: J K w.r.t.

Original: the poisoning size K using the COPA-S EARCH algorithm introduced in Section 4.4.
Revised: the poisoning size K using the COPA-S EARCH algorithm introduced in Section 4.4.

Original: We provide details of the evaluated trajectory length along with the rationales in Appendix G.2.
Revised: We provide details of the evaluated trajectory length along with the rationales in Appendix G.2.

Original: Evaluation Results.
Revised: Evaluation Results.

Original: We present the comparison of reward certiﬁcation for different RL algorithms and certiﬁcation methods in Figure 2.
Revised: We present the comparison of reward certiﬁcation for different RL algorithms and certiﬁcation methods in Figure 2.

Original: Essentially, at each poisoning size K , we compare the lower bound of cumulative reward achieved by different RL algorithms and certiﬁcation methods— higher value of the lower bound implies stronger certiﬁed robustness .
Revised: Essentially, at each poisoning size K , we compare the lower bound of cumulative reward achieved by different RL algorithms and certiﬁcation methods— higher value of the lower bound implies stronger certiﬁed robustness .

Original: On the RL algorithm level, QRDQN and C51 almost invariably outperform the baseline DQN algorithm.
Revised: On the RL algorithm level, QRDQN and C51 almost invariably outperform the baseline DQN algorithm.

Original: On the aggregation protocol level, methods with temporal aggregation consistently surpass the single-step aggregation
Revised: On the aggregation protocol level, methods with temporal aggregation consistently surpass the single-step aggregation

Original: PARL on Freeway but not the other two, as analyzed in Section 5.1.
Revised: PARL on Freeway but not the other two, as analyzed in Section 5.1.

Original: In addition, we note that DPARL is sometimes not as robust as TPARL.
Revised: In addition, we note that DPARL is sometimes not as robust as TPARL.

Original: We hypothesize two reasons: 1) the dynamic mechanism is more susceptible to the attack, e.g. , the selected optimal window size is prone to be manipulated; 2) the lower bound is looser for DPARL given the difﬁculty of computing the possible action set in DPARL (discussed in Theorem 7).
Revised: We hypothesize two reasons: 1) the dynamic mechanism is more susceptible to the attack, e.g. , the selected optimal window size is prone to be manipulated; 2) the lower bound is looser for DPARL given the difﬁculty of computing the possible action set in DPARL (discussed in Theorem 7).

Original: On the partition number level, a larger partition number ( u = 50 ) demonstrates higher robustness.
Revised: On the partition number level, a larger partition number ( u = 50 ) demonstrates higher robustness.

Revised: On the horizon length level, the robustness ranking of different policies is similar under different horizon lengths with slight differences, corresponding to the property of ﬁnite-horizon RL.

Original: On the RL environment level, Freeway can tolerate a larger poisoning size than Breakout and Highway.
Revised: On the RL environment level, Freeway can tolerate a larger poisoning size than Breakout and Highway.

Original: More results and discussions are in Appendix H.7.
Revised: More results and discussions are in Appendix H.7.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper, we proposed COPA , the ﬁrst framework for certifying robust policies for ofﬂine RL against poisoning attacks.
Revised: In this paper, we proposed COPA , the ﬁrst framework for certifying robust policies for ofﬂine RL against poisoning attacks.

Original: COPA includes three policy aggregation protocols.
Revised: COPA includes three policy aggregation protocols.

Original: For each aggregation protocol, COPA provides a sound certiﬁcation for both per-state action stability and cumulative reward bound.
Revised: For each aggregation protocol, COPA provides a sound certiﬁcation for both per-state action stability and cumulative reward bound.

Original: Experimental evaluations on different environments and different ofﬂine RL training algorithms show the effectiveness of our robustness certiﬁcation in a wide range of scenarios. 
Revised: Experimental evaluations on different environments and different ofﬂine RL training algorithms show the effectiveness of our robustness certiﬁcation in a wide range of scenarios. A CKNOWLEDGMENTS

Revised: This work is partially supported by the NSF grant No.1910100, NSF CNS 20-46726 CAR, Alfred P.

Revised: Sloan Fellowship, the U.S. Department of Energy by the Lawrence Livermore National Laboratory under Contract

Original:  Ethics Statement.
Revised: No. DE-AC52-07NA27344 and LLNL LDRD Program Project No. 20-ER-014, and Amazon Research Award. Ethics Statement.

Original: In this paper, we prove the ﬁrst framework for certifying robust policies for ofﬂine RL against poisoning attacks.
Revised: In this paper, we prove the ﬁrst framework for certifying robust policies for ofﬂine RL against poisoning attacks.

Original: On the one hand, such framework provides rigorous guarantees in practice RL tasks and thus signiﬁcantly alleviates the security vulnerabilities of ofﬂine RL algorithms against training-phase poisoning attacks.
Revised: On the one hand, such framework provides rigorous guarantees in practice RL tasks and thus signiﬁcantly alleviates the security vulnerabilities of ofﬂine RL algorithms against training-phase poisoning attacks.

Original: The evaluation of different RL algorithms also provides a better understanding about the different degrees of security vulnerabilities across different RL algorithms.
Revised: The evaluation of different RL algorithms also provides a better understanding about the different degrees of security vulnerabilities across different RL algorithms.

Original: On the other hand, the robustness guarantee provided by our framework only holds under speciﬁc conditions of the attack.
Revised: On the other hand, the robustness guarantee provided by our framework only holds under speciﬁc conditions of the attack.

Original: Speciﬁcally, we require the attack to change only a bounded number of training trajectories.
Revised: Speciﬁcally, we require the attack to change only a bounded number of training trajectories.

Original: Therefore, users should be aware of such limitations of our framework, and should not blindly trust the robustness guarantee when the attack can change a large number of training instances or modify the training algorithm itself.
Revised: Therefore, users should be aware of such limitations of our framework, and should not blindly trust the robustness guarantee when the attack can change a large number of training instances or modify the training algorithm itself.

Original: As a result, we encourage researchers to understand the potential risks, and evaluate whether our constraints on the attack align with their usage scenarios when applying our COPA to real-world applications.
Revised: As a result, we encourage researchers to understand the potential risks, and evaluate whether our constraints on the attack align with their usage scenarios when applying our COPA to real-world applications.

Original: We do not expect any ethics issues raised by our work.
Revised: We do not expect any ethics issues raised by our work.

Original: Reproducibility Statement.
Revised: Reproducibility Statement.

Original: All theorem statements are substantiated with rigorous proofs in Appendix F. In Appendix E, we list the pseudocode for all key algorithms.
Revised: All theorem statements are substantiated with rigorous proofs in Appendix F. In Appendix E, we list the pseudocode for all key algorithms.

Original: Our experimental evaluation is conducted with publicly available OpenAI Gym toolkit (Brockman et al., 2016).
Revised: Our experimental evaluation is conducted with publicly available OpenAI Gym toolkit (Brockman et al., 2016).

Original: We introduce all experimental details in both Section 5 and Appendix G. Speciﬁcally, we build the code upon the open-source code base of Agarwal et al.
Revised: We introduce all experimental details in both Section 5 and Appendix G. Speciﬁcally, we build the code upon the open-source code base of Agarwal et al.

Original: and we upload the source code at https://github.com/AI-secure/COPA for reproducibility purpose.
Revised: and we upload the source code at https://github.com/AI-secure/COPA for reproducibility purpose.

Original: A O MITTED D EFINITIONS A. C UMULATIVE R EWARD
Revised: A O MITTED D EFINITIONS A. C UMULATIVE R EWARD

Original: Deﬁnition 6 (Cumulative Reward) .
Revised: Deﬁnition 6 (Cumulative Reward) .

Original: Let P : S ×
Revised: Let P : S ×

Original: A → P ( S ) be the transition function with P ( · ) deﬁning the set of probability measures.
Revised: A → P ( S ) be the transition function with P ( · ) deﬁning the set of probability measures.

Original: Let R, d 0 , H be the reward function, initial state distribu- tion, and time horizon.
Revised: Let R, d 0 , H be the reward function, initial state distribu- tion, and time horizon.

Original: We denote J ( π ) as the cumulative reward of the given policy π :
Revised: We denote J ( π ) as the cumulative reward of the given policy π :

Original: J ( π ) :
Revised: J ( π ) :

Original: =
Revised: =

Original: E
Revised: E

Original: H − 1 (cid:88) t =
Revised: H − 1 (cid:88) t =

Original: R ( s t , a t ) , where a t = π ( s t ) , s t +1 ∼ P ( s t , a t ) , s 0 ∼ d 0 , (11)
Revised: R ( s t , a t ) , where a t = π ( s t ) , s t +1 ∼ P ( s t , a t ) , s 0 ∼ d 0 , (11)

Original: It is natural to adapt this deﬁnition when there is a discount factor.
Revised: It is natural to adapt this deﬁnition when there is a discount factor.

Original: If policy π considers recent n t states instead of only s t to make action predictions, we can change a t =
Revised: If policy π considers recent n t states instead of only s t to make action predictions, we can change a t =

Original: π ( s t ) to a t =
Revised: π ( s t ) to a t =

Original: π ( s t − n t +1: t ) in Equation (11).
Revised: π ( s t − n t +1: t ) in Equation (11).

Original: A. PARL (P ER -S TATE P ARTITION A GGREGATION ) P ROTOCOL
Revised: A. PARL (P ER -S TATE P ARTITION A GGREGATION ) P ROTOCOL

Original: Deﬁnition 7 (Per-State Partition Aggregation) .
Revised: Deﬁnition 7 (Per-State Partition Aggregation) .

Original: Given subpolicies { π i } u − 1 i =0 , the Per-State Partition Aggregation (PARL) protocol deﬁnes an aggregated policy π P : S
Revised: Given subpolicies { π i } u − 1 i =0 , the Per-State Partition Aggregation (PARL) protocol deﬁnes an aggregated policy π P : S

Original: → A such that
Revised: → A such that

Original: [Equation]
Revised: [Equation]

Original: where n a ( s ) is deﬁned in Section 3.
Revised: where n a ( s ) is deﬁned in Section 3.

Original: As mentioned in Section 4, when the arg max in Equation (12) returns multiple elements, we break ties deterministically by returning the “smaller” ( < ) action, which can be deﬁned by numerical order, lexicographical order, etc.
Revised: As mentioned in Section 4, when the arg max in Equation (12) returns multiple elements, we break ties deterministically by returning the “smaller” ( < ) action, which can be deﬁned by numerical order, lexicographical order, etc.

Original: A. TPARL (T EMPORAL P ARTITION A GGREGATION ) P ROTOCOL
Revised: A. TPARL (T EMPORAL P ARTITION A GGREGATION ) P ROTOCOL

Original: Deﬁnition 8 (Temporal Partition Aggregation) .
Revised: Deﬁnition 8 (Temporal Partition Aggregation) .

Original: Given subpolicies { π i } u − 1 i =0 and window size W , at time step t , the Temporal Partition Aggregation (TPARL) deﬁnes an aggregated policy π T :
Revised: Given subpolicies { π i } u − 1 i =0 and window size W , at time step t , the Temporal Partition Aggregation (TPARL) deﬁnes an aggregated policy π T :

Original: S min { t +1 ,W } →
Revised: S min { t +1 ,W } →

Original: A such that
Revised: A such that

Original: [Equation]
Revised: [Equation]

Original: where s l : r and n a are deﬁned in Section 3.
Revised: where s l : r and n a are deﬁned in Section 3.

Original: In the above deﬁnition, at time step t , the input of policy π T contains all states between time step max { t − W + 1 , 0 } and t within window size W ; and the output is the policy prediction with the highest votes across these states.
Revised: In the above deﬁnition, at time step t , the input of policy π T contains all states between time step max { t − W + 1 , 0 } and t within window size W ; and the output is the policy prediction with the highest votes across these states.

Original: Recall that in Section 3, we deﬁne n a ( s l : r )
Revised: Recall that in Section 3, we deﬁne n a ( s l : r )

Original: = (cid:80) rj
Revised: = (cid:80) rj

Original: = l n a ( s j ) .
Revised: = l n a ( s j ) .

Original: We break ties in the same way as in PARL.
Revised: We break ties in the same way as in PARL.

Original: B I LLUSTRATION OF COPA A GGREGATION P ROTOCOLS
Revised: B I LLUSTRATION OF COPA A GGREGATION P ROTOCOLS

Original: We present a concrete example to demonstrate how different aggregation protocols induce different tolerable poisoning thresholds, illustrate bottleneck and non-bottleneck states, and provide additional discussions.
Revised: We present a concrete example to demonstrate how different aggregation protocols induce different tolerable poisoning thresholds, illustrate bottleneck and non-bottleneck states, and provide additional discussions.

Original: Suppose the action space contains two actions
Revised: Suppose the action space contains two actions

Original: A = { a 1 , a 2 } , and there are six subpolicies { π i } u − 1 i = where u = 6 .
Revised: A = { a 1 , a 2 } , and there are six subpolicies { π i } u − 1 i = where u = 6 .

Original: We are at time step t = 7 now.
Revised: We are at time step t = 7 now.

Original: When there is no poisoning, the subpolicies’ action predictions for state s 0 to s 7 are shown by Table 2.
Revised: When there is no poisoning, the subpolicies’ action predictions for state s 0 to s 7 are shown by Table 2.

Original: After aggregation, all aggregated policies will choose action a 1 at t = 7 .
Revised: After aggregation, all aggregated policies will choose action a 1 at t = 7 .

Original: PARL (Deﬁnition 7).
Revised: PARL (Deﬁnition 7).

Original: The PARL aggregation protocol only uses the current state s 7 to aggregate the votes.
Revised: The PARL aggregation protocol only uses the current state s 7 to aggregate the votes.

Original: On s 7 , three subpolicies choose action a 1 and three others choose action a 2 .
Revised: On s 7 , three subpolicies choose action a 1 and three others choose action a 2 .

Original: By breaking the tie with a 1 < a 2 , the PARL policy π P ( s 7 )
Revised: By breaking the tie with a 1 < a 2 , the PARL policy π P ( s 7 )

Original: = a 1 .
Revised: = a 1 .

Original: The tolerable poisoning threshold K t = 0 , because one action ﬂipping from a 1 to a 2 by poisoning only one subpolicy can change the aggregated policy to a 2 .
Revised: The tolerable poisoning threshold K t = 0 , because one action ﬂipping from a 1 to a 2 by poisoning only one subpolicy can change the aggregated policy to a 2 .

Original: TPARL (Deﬁnition 8).
Revised: TPARL (Deﬁnition 8).

Original: The TPARL aggregation protocol uses window size W = 7 .
Revised: The TPARL aggregation protocol uses window size W = 7 .

Original: This implies that, we aggregate all states s 1:7 to count the votes and decide the action.
Revised: This implies that, we aggregate all states s 1:7 to count the votes and decide the action.

Original: Since n a 1 ( s 1:7 ) = 36 and
Revised: Since n a 1 ( s 1:7 ) = 36 and

Original: Action Prediction for
Revised: Action Prediction for

Original: [Table]
Revised: [Table]

Original: PARL ( π P , Deﬁnition 7)
Revised: PARL ( π P , Deﬁnition 7)

Original: TPARL with W = 7 ( π T , Deﬁnition 8)
Revised: TPARL with W = 7 ( π T , Deﬁnition 8)

Original: DPARL with W max = 8 ( π D , Deﬁnition 3)  n a 2 ( s 1:7 ) = 6 , after poisoning two subpolicies, we still (cid:101) n a 1 ( s 1:7 ) ≥ (cid:101) n a 2 ( s 1:7 ) .
Revised: DPARL with W max = 8 ( π D , Deﬁnition 3)  n a 2 ( s 1:7 ) = 6 , after poisoning two subpolicies, we still (cid:101) n a 1 ( s 1:7 ) ≥ (cid:101) n a 2 ( s 1:7 ) .

Original: Thus, we achieve an tolerable poisoning threshold K t = 2 .
Revised: Thus, we achieve an tolerable poisoning threshold K t = 2 .

Original: Compared with PARL, the temporal aggregation in TPARL increases the vote margin between aand a 2 and thus improve the tolerable poisoning threshold at current state.
Revised: Compared with PARL, the temporal aggregation in TPARL increases the vote margin between aand a 2 and thus improve the tolerable poisoning threshold at current state.

Original: DPARL (Deﬁnition 3).
Revised: DPARL (Deﬁnition 3).

Original: The DPARL aggregation protocol chooses the window size W (cid:48)
Revised: The DPARL aggregation protocol chooses the window size W (cid:48)

Original: = 8 since this window size gives the largest average vote margin ∆ W (cid:48) t (deﬁned in Equation (1)).
Revised: = 8 since this window size gives the largest average vote margin ∆ W (cid:48) t (deﬁned in Equation (1)).

Original: We can easily ﬁnd out that with poisoning size K = 1 , we will still choose (cid:102) W (cid:48)
Revised: We can easily ﬁnd out that with poisoning size K = 1 , we will still choose (cid:102) W (cid:48)

Original: = 

Revised: =

Original: as the window size and the resulting votes for a 1 can be kept higher than votes for a 2 .
Revised: as the window size and the resulting votes for a 1 can be kept higher than votes for a 2 .

Original: However, when it comes to K
Revised: However, when it comes to K

Original: = 2 , if we totally ﬂip subpolicies π 0 and π 1 to let them always predict action a 2 , then the DPARL aggregated policy will turn to choose W (cid:48)
Revised: = 2 , if we totally ﬂip subpolicies π 0 and π 1 to let them always predict action a 2 , then the DPARL aggregated policy will turn to choose W (cid:48)

Original: = 1 as the window size and then choose action a 2 as the action output.
Revised: = 1 as the window size and then choose action a 2 as the action output.

Original: Thus, we achieve a tolerable poisoning threshold K t = 1 this time.
Revised: Thus, we achieve a tolerable poisoning threshold K t = 1 this time.

Original: Illustrations of Bottleneck and Non-Bottleneck States.
Revised: Illustrations of Bottleneck and Non-Bottleneck States.

Original: According to our illustration in Section 4.2, bottleneck states are those where subpolicies vote for diverse actions but the best action is the same as previous states, and non-bottleneck states are those where subpolicies mostly vote for the same action.
Revised: According to our illustration in Section 4.2, bottleneck states are those where subpolicies vote for diverse actions but the best action is the same as previous states, and non-bottleneck states are those where subpolicies mostly vote for the same action.

Original: As we can see, s 0 to s 6 are non-bottleneck states since the subpolicies almost unanimously vote for one action.
Revised: As we can see, s 0 to s 6 are non-bottleneck states since the subpolicies almost unanimously vote for one action.

Original: In contrast, s 7 is a bottleneck state.
Revised: In contrast, s 7 is a bottleneck state.

Original: From the perspectives of different aggregation protocols , we observe that for the bottleneck state s 7 , both TPARL and DPARL exploit temporal aggregation to boost the certiﬁed robust poisoning threshold (from 0 to 2 and from 0 to 1 respectively), thus demonstrating the effectiveness of TPARL and DPARL on improving certiﬁed robustness.
Revised: From the perspectives of different aggregation protocols , we observe that for the bottleneck state s 7 , both TPARL and DPARL exploit temporal aggregation to boost the certiﬁed robust poisoning threshold (from 0 to 2 and from 0 to 1 respectively), thus demonstrating the effectiveness of TPARL and DPARL on improving certiﬁed robustness.

Original: On the other hand, for non-bottleneck states s 0 to s 6 , we can easily see that different aggregation protocols do not differ much in terms of provided certiﬁed robustness levels.
Revised: On the other hand, for non-bottleneck states s 0 to s 6 , we can easily see that different aggregation protocols do not differ much in terms of provided certiﬁed robustness levels.

Original: Explanations for Bottleneck and Non-Bottleneck States.
Revised: Explanations for Bottleneck and Non-Bottleneck States.

Original: We provide additional discussions regarding why bottleneck states are vulnerable and why our temporal aggregation strategy can effectively alleviate the problem .
Revised: We provide additional discussions regarding why bottleneck states are vulnerable and why our temporal aggregation strategy can effectively alleviate the problem .

Original: We ﬁrst explain the existence of the bottleneck states.
Revised: We ﬁrst explain the existence of the bottleneck states.

Original: Given the property of the bottleneck states that there is high disagreement among different subpolicies on such states, they may lie close to the decision boundary.
Revised: Given the property of the bottleneck states that there is high disagreement among different subpolicies on such states, they may lie close to the decision boundary.

Original: This may be a result of poisoning, or simply due to the intrinsic difﬁculty of the state.
Revised: This may be a result of poisoning, or simply due to the intrinsic difﬁculty of the state.

Original: On the other hand, such states naturally exist in the rollouts (like natural adversarial examples (Hendrycks et al., 2021)), and may induce high instability of the rollouts during testing.
Revised: On the other hand, such states naturally exist in the rollouts (like natural adversarial examples (Hendrycks et al., 2021)), and may induce high instability of the rollouts during testing.

Original: We next discuss additional intuitions for our temporal aggregation.
Revised: We next discuss additional intuitions for our temporal aggregation.

Original: Essentially, temporal aggregation effectively leverages the adjacent non-bottleneck states to rectify the decisions at the bottleneck states, based on the assumption of temporal continuity which has been revealed and utilized in several previous works (Legenstein et al., 2010; Veerapaneni et al., 2020;
Revised: Essentially, temporal aggregation effectively leverages the adjacent non-bottleneck states to rectify the decisions at the bottleneck states, based on the assumption of temporal continuity which has been revealed and utilized in several previous works (Legenstein et al., 2010; Veerapaneni et al., 2020;

Original: Ye et al., 2021).
Revised: Ye et al., 2021).

Original: C C ERTIFICATION OF P ER -S TATE A CTION S TABILITY FOR PARL
Revised: C C ERTIFICATION OF P ER -S TATE A CTION S TABILITY FOR PARL

Original: We certify the robustness of PARL following Theorem 8.
Revised: We certify the robustness of PARL following Theorem 8.

Original: Theorem 8.
Revised: Theorem 8.

Original: Let D be the clean training dataset; let π i
Revised: Let D be the clean training dataset; let π i

Original: =
Revised: =

Original: M 

Revised: M

Original: ( D i ) , 0 ≤ i ≤ u − 1 be the learned subpolicies according to Section 4.1 from which we deﬁne n a (see Section 3); and let π P be the PerState Partition Aggregation policy: π P = M ( D ) where M abstracts the whole training-aggregation process.
Revised: ( D i ) , 0 ≤ i ≤ u − 1 be the learned subpolicies according to Section 4.1 from which we deﬁne n a (see Section 3); and let π P be the PerState Partition Aggregation policy: π P = M ( D ) where M abstracts the whole training-aggregation process.

Original: (cid:101) D is a poisoned dataset and (cid:102)
Revised: (cid:101) D is a poisoned dataset and (cid:102)

Original: π P is the poisoned policy: (cid:102) π P =
Revised: π P is the poisoned policy: (cid:102) π P =

Original: M
Revised: M

Original: ( (cid:101) D ) .
Revised: ( (cid:101) D ) .

Original: For a given state s t encountered at time step t during test time, let a :=
Revised: For a given state s t encountered at time step t during test time, let a :=

Original: π P ( s t ) , then
Revised: π P ( s t ) , then

Original: [Equation]
Revised: [Equation]

Original: Remark.
Revised: Remark.

Original: The theorem provides a valid per-state action certiﬁcation for PARL, since according to
Revised: The theorem provides a valid per-state action certiﬁcation for PARL, since according to

Original: Deﬁnition 1, as long as the poisoning size is smaller or equal to K t , i.e. , | D (cid:9) (cid:101) D | ≤ K t , the action of the poisoned policy is the same as the clean policy: (cid:102) π P ( s t ) =
Revised: Deﬁnition 1, as long as the poisoning size is smaller or equal to K t , i.e. , | D (cid:9) (cid:101) D | ≤ K t , the action of the poisoned policy is the same as the clean policy: (cid:102) π P ( s t ) =

Original: π P ( s t ) = a .
Revised: π P ( s t ) = a .

Original: To compute K t , according to Equation (14), we rely on the aggregated action count n a for state s t from subpolicies.
Revised: To compute K t , according to Equation (14), we rely on the aggregated action count n a for state s t from subpolicies.

Original: The a (cid:48) < a is the “smaller-than” operator introduced following Deﬁnition 7.
Revised: The a (cid:48) < a is the “smaller-than” operator introduced following Deﬁnition 7.

Original: Proof of Theorem 8.
Revised: Proof of Theorem 8.

Original: Suppose the poisoning size is within K , then after poisoning, the aggregated action count (cid:101) n a ( s t ) ∈ [ n a ( s t ) − K, n a ( s t )
Revised: Suppose the poisoning size is within K , then after poisoning, the aggregated action count (cid:101) n a ( s t ) ∈ [ n a ( s t ) − K, n a ( s t )

Original: + K ] where n a ( s t ) is such count before poisoning.
Revised: + K ] where n a ( s t ) is such count before poisoning.

Original: Thus, to ensure the chosen action does not change, a necessary condition is that (cid:101) n a ( s t ) ≥ (cid:101) n a (cid:48) ( s t )+ 1 [ a (cid:48) <
Revised: Thus, to ensure the chosen action does not change, a necessary condition is that (cid:101) n a ( s t ) ≥ (cid:101) n a (cid:48) ( s t )+ 1 [ a (cid:48) <

Original: a ] for any other a (cid:48) < a .
Revised: a ] for any other a (cid:48) < a .

Original: Solving K yields Equation (14).
Revised: Solving K yields Equation (14).

Original: We use the theorem to compute the per-state action certiﬁcation for each time step t along the trajectory, and the detailed algorithm is in Algorithm 2 (Appendix E).
Revised: We use the theorem to compute the per-state action certiﬁcation for each time step t along the trajectory, and the detailed algorithm is in Algorithm 2 (Appendix E).

Original: Furthermore, we prove that this certiﬁcation is theoretically tight as the following proposition shows.
Revised: Furthermore, we prove that this certiﬁcation is theoretically tight as the following proposition shows.

Original: The proof is in Appendix F.1.
Revised: The proof is in Appendix F.1.

Original: Proposition 9.
Revised: Proposition 9.

Original: Under the same condition as Theorem 8, for any time step t , there exists an RL learning algorithm M 0 , and a poisoned dataset (cid:101) D , such that | D (cid:9) (cid:101) D | =
Revised: Under the same condition as Theorem 8, for any time step t , there exists an RL learning algorithm M 0 , and a poisoned dataset (cid:101) D , such that | D (cid:9) (cid:101) D | =

Original: K t + 1 , and (cid:102) π P ( s t ) (cid:54) =
Revised: K t + 1 , and (cid:102) π P ( s t ) (cid:54) =

Original: π P ( s t ) .
Revised: π P ( s t ) .

Original: D A T ABLE OF P OSSIBLE A CTION S ET
Revised: D A T ABLE OF P OSSIBLE A CTION S ET

Original: [Table]
Revised: [Table]

Original: For all three aggregation protocols, we summarize the expressions used for computing the possible action set
Revised: For all three aggregation protocols, we summarize the expressions used for computing the possible action set

Original: A ( K ) given tolerable poisoning threshold K in Table 3.
Revised: A ( K ) given tolerable poisoning threshold K in Table 3.

Original: E A LGORITHM P SEUDOCODE AND D ISCUSSION E. COPA T RAINING P ROTOCOL
Revised: E A LGORITHM P SEUDOCODE AND D ISCUSSION E. COPA T RAINING P ROTOCOL

Original: Algorithm 1: COPA training protocol.
Revised: Algorithm 1: COPA training protocol.

Original: Input: training dataset
Revised: Input: training dataset

Original: , number partitions h
Revised: , number partitions h

Original: Output: COPA subpolicies { π i } k − 1 i =
Revised: Output: COPA subpolicies { π i } k − 1 i =

Original: (cid:46)
Revised: (cid:46)

Original: E. COPA
Revised: E. COPA

Original: TATE A
Revised: TATE A

Original: ERTIFICATION
Revised: ERTIFICATION

Original: Per-State Partition Aggregation (PARL).
Revised: Per-State Partition Aggregation (PARL).

Original: Algorithm
Revised: Algorithm

Original: Input: environment E =
Revised: Input: environment E =

Original: ( S , A , R, P, H, d 0 ) , subpolicies { π i } k − 1 i =
Revised: ( S , A , R, P, H, d 0 ) , subpolicies { π i } k − 1 i =

Original: Output: COPA robust size at each time step { K t } H − 1 t = 1 s 0 ∼ d(cid:46) sample initial state 2 for t from 0 to H − 1 do for each a ∈ A do
Revised: Output: COPA robust size at each time step { K t } H − 1 t = 1 s 0 ∼ d(cid:46) sample initial state 2 for t from 0 to H − 1 do for each a ∈ A do

Original: Compute n a ( s t ) from subpolicies’ { π i } k − 1 i =0 decisions (cid:46) n a is defined in Section
Revised: Compute n a ( s t ) from subpolicies’ { π i } k − 1 i =0 decisions (cid:46) n a is defined in Section

Original: Determine the chosen action a t ← π P ( s t ) according to PARL (Deﬁnition 7)
Revised: Determine the chosen action a t ← π P ( s t ) according to PARL (Deﬁnition 7)

Original: Compute K t according to Equation (14) in Theorem s t +1 ∼ P ( s t , a t ) 8 return { K t } H − 1 t =
Revised: Compute K t according to Equation (14) in Theorem s t +1 ∼ P ( s t , a t ) 8 return { K t } H − 1 t =

Original: Temporal Partition Aggregation (TPARL).
Revised: Temporal Partition Aggregation (TPARL).

Original: Algorithm 3:
Revised: Algorithm 3:

Original: Input: environment E =
Revised: Input: environment E =

Original: ( S , A , R, P, H, d 0 ) , subpolicies { π i } k − 1 i =0 , window size W
Revised: ( S , A , R, P, H, d 0 ) , subpolicies { π i } k − 1 i =0 , window size W

Original: Output: COPA robust size at each time step { K t } H − 1 t = 1 s 0 ∼ d(cid:46) sample initial state 2 for t from 0 to H − 1 do for each a ∈ A do
Revised: Output: COPA robust size at each time step { K t } H − 1 t = 1 s 0 ∼ d(cid:46) sample initial state 2 for t from 0 to H − 1 do for each a ∈ A do

Original: Compute n a ( s t ) from subpolicies’ { π i } k − 1 i =0 decisions (cid:46) n a is defined in Section
Revised: Compute n a ( s t ) from subpolicies’ { π i } k − 1 i =0 decisions (cid:46) n a is defined in Section

Original: Determine the chosen action a t ← π T ( s t ) according to TPARL (Deﬁnition 8) p min ← ∞ for each a (cid:48) ∈ A , a (cid:48) (cid:54)
Revised: Determine the chosen action a t ← π T ( s t ) according to TPARL (Deﬁnition 8) p min ← ∞ for each a (cid:48) ∈ A , a (cid:48) (cid:54)

Original: = a t do
Revised: = a t do

Original: Dynamic Temporal Partition
Revised: Dynamic Temporal Partition

Original: 7) { g ( i ) } ki =1 ← sorted ( { g i } k − 1 i =0 , reverse = True )
Revised: 7) { g ( i ) } ki =1 ← sorted ( { g i } k − 1 i =0 , reverse = True )

Original: Compute n W ∗ a (cid:48) , n W ∗ a # , n W (cid:48) a t , n W (cid:48) a (cid:48)(cid:48) tmp ← W (cid:48) ( n W ∗ a (cid:48) − n W ∗ a # ) − W ∗ ( n W (cid:48) a t − n W (cid:48) a (cid:48)(cid:48) ) − 1 [ a (cid:48) > a t ] sum ← 0 , p ← for j from 1 to k do if sum + tmp ≥ 0 then p ← j − break p ← j, sum ← sum + g ( j )
Revised: Compute n W ∗ a (cid:48) , n W ∗ a # , n W (cid:48) a t , n W (cid:48) a (cid:48)(cid:48) tmp ← W (cid:48) ( n W ∗ a (cid:48) − n W ∗ a # ) − W ∗ ( n W (cid:48) a t − n W (cid:48) a (cid:48)(cid:48) ) − 1 [ a (cid:48) > a t ] sum ← 0 , p ← for j from 1 to k do if sum + tmp ≥ 0 then p ← j − break p ← j, sum ← sum + g ( j )

Original: L W ∗ ,W (cid:48) a (cid:48) ,a (cid:48)(cid:48) ← p, K
Revised: L W ∗ ,W (cid:48) a (cid:48) ,a (cid:48)(cid:48) ← p, K

Original: Dt ← min { K Dt , L W ∗ ,W (cid:48) a (cid:48) ,a (cid:48)(cid:48) } s t +1 ∼ P ( s t , a t ) 30 return { K Dt } H − 1 t =
Revised: Dt ← min { K Dt , L W ∗ ,W (cid:48) a (cid:48) ,a (cid:48)(cid:48) } s t +1 ∼ P ( s t , a t ) 30 return { K Dt } H − 1 t =

Original: E. COPA C UMULATIVE R EWARD C ERTIFICATION
Revised: E. COPA C UMULATIVE R EWARD C ERTIFICATION

Original: COPA-S EARCH alternately executes the procedure of trajectory exploration and expansion and poisoning threshold growth .
Revised: COPA-S EARCH alternately executes the procedure of trajectory exploration and expansion and poisoning threshold growth .

Original: In trajectory exploration and expansion, COPA-S EARCH organizes all possible trajectories in the form of a search tree and progressively grows it.
Revised: In trajectory exploration and expansion, COPA-S EARCH organizes all possible trajectories in the form of a search tree and progressively grows it.

Original: For each node (representing a state), we leverage Theorems 4, 5 and 7 to compute the Possible Action Set.
Revised: For each node (representing a state), we leverage Theorems 4, 5 and 7 to compute the Possible Action Set.

Original: We then expand the tree branches corresponding to the actions in the derived set.
Revised: We then expand the tree branches corresponding to the actions in the derived set.

Original: In poisoning threshold growth, when all trajectories for the current poisoning threshold are explored, we increase K (cid:48) to seek certiﬁcation under larger poisoning sizes, via maintaining a priority queue of all poisoning sizes during the expansion of the tree.
Revised: In poisoning threshold growth, when all trajectories for the current poisoning threshold are explored, we increase K (cid:48) to seek certiﬁcation under larger poisoning sizes, via maintaining a priority queue of all poisoning sizes during the expansion of the tree.

Original: The iterative procedures end when the priority queue becomes empty.
Revised: The iterative procedures end when the priority queue becomes empty.

Original: The highlighted line in the following algorithm needs to inject different algorithms based on the aggregation protocol: for PARL ( π P ), use Theorem 4; for TPARL ( π T ), use Theorem 5; for
Revised: The highlighted line in the following algorithm needs to inject different algorithms based on the aggregation protocol: for PARL ( π P ), use Theorem 4; for TPARL ( π T ), use Theorem 5; for

Original: DPARL ( π D ), use Theorem 7.
Revised: DPARL ( π D ), use Theorem 7.

Original: E XPAND t
Revised: E XPAND t

Original: Section 4.
Revised: Section 4.

Original: return do do result
Revised: return do do result

Original: M
Revised: M

Original: Time Complexity.
Revised: Time Complexity.

Original: The complexity of COPA-S EARCH is O
Revised: The complexity of COPA-S EARCH is O

Original: ( H |S explored | (log |S explored | + |A| T )) , where |S explored | is the number of explored states throughout the search procedure, which is no larger than cardinality of state set S , H is the horizon length, |A| is the cardinality of action set, and T is the time complexity of per-state action certiﬁcation.
Revised: ( H |S explored | (log |S explored | + |A| T )) , where |S explored | is the number of explored states throughout the search procedure, which is no larger than cardinality of state set S , H is the horizon length, |A| is the cardinality of action set, and T is the time complexity of per-state action certiﬁcation.

Original: The main bottleneck of the algorithm is the large number of possible states, which is in the worse case exponential to state dimension.
Revised: The main bottleneck of the algorithm is the large number of possible states, which is in the worse case exponential to state dimension.

Original: However, to provide a deterministic worst-case certiﬁcation agnostic to environment properties, exploring all possible states may be inevitable.
Revised: However, to provide a deterministic worst-case certiﬁcation agnostic to environment properties, exploring all possible states may be inevitable.

Original: Relation with Wu et al.
Revised: Relation with Wu et al.

Original: The COPA-S EARCH algorithm is inspired from Wu et al.
Revised: The COPA-S EARCH algorithm is inspired from Wu et al.

Original: (2022, Algorithm 3), which also leverages tree search to explore all possible states and thus derive the robustness guarantee.
Revised: (2022, Algorithm 3), which also leverages tree search to explore all possible states and thus derive the robustness guarantee.

Original: However, the major distinction is that their algorithm tries to derive a probabilistic guarantee of RL robustness against state perturbations, while COPA-S EARCH derives deterministic guarantee of RL robustness against poisoning attacks.
Revised: However, the major distinction is that their algorithm tries to derive a probabilistic guarantee of RL robustness against state perturbations, while COPA-S EARCH derives deterministic guarantee of RL robustness against poisoning attacks.

Original: We think this general tree search methodology can be further extended to provide certiﬁcation beyond evasion attack (Wu et al., 2022) and poisoning attack (our work) and we leave it as future work.
Revised: We think this general tree search methodology can be further extended to provide certiﬁcation beyond evasion attack (Wu et al., 2022) and poisoning attack (our work) and we leave it as future work.

Original: Extension to stochastic MDPs.
Revised: Extension to stochastic MDPs.

Original: In the current version of COPA-S EARCH , the exhaustive search is enabled by the deterministic MDP assumption.
Revised: In the current version of COPA-S EARCH , the exhaustive search is enabled by the deterministic MDP assumption.

Original: However, we foresee the potential of conveniently extending COPA-S EARCH to stochastic MDPs and will discuss one concrete method below.
Revised: However, we foresee the potential of conveniently extending COPA-S EARCH to stochastic MDPs and will discuss one concrete method below.

Original: In contrast to interacting with the environment for one time to obtain the deterministic next state transition in the deterministic MDP case, in the case of stochastic MDPs, we can leverage sampling ( i.e. , by repeatedly taking the same action at the same state) to obtain the set of high probability next state transitions with high conﬁdence.
Revised: In contrast to interacting with the environment for one time to obtain the deterministic next state transition in the deterministic MDP case, in the case of stochastic MDPs, we can leverage sampling ( i.e. , by repeatedly taking the same action at the same state) to obtain the set of high probability next state transitions with high conﬁdence.

Original: In this way, our COPA-S EARCH will be able to yield a probabilistic bound , in comparison with the deterministic bound achieved in this paper enabled by the deterministic MDP assumption.
Revised: In this way, our COPA-S EARCH will be able to yield a probabilistic bound , in comparison with the deterministic bound achieved in this paper enabled by the deterministic MDP assumption.

Original: F P ROOFS F. T IGHTNESS OF P ER -S TATE A CTION C ERTIFICATION IN PARL
Revised: F P ROOFS F. T IGHTNESS OF P ER -S TATE A CTION C ERTIFICATION IN PARL

Original: Proof of Proposition 9.
Revised: Proof of Proposition 9.

Original: We prove by construction.
Revised: We prove by construction.

Original: Given the state s t , we ﬁrst locate the subpolicies whose chosen action is a , and denote the set of them as
Revised: Given the state s t , we ﬁrst locate the subpolicies whose chosen action is a , and denote the set of them as

Original: [Equation]
Revised: [Equation]

Original: We also denote a (cid:48)
Revised: We also denote a (cid:48)

Original: =
Revised: =

Original: arg max a (cid:48)
Revised: arg max a (cid:48)

Original: (cid:54) =
Revised: (cid:54) =

Original: a n a (cid:48) ( s t )+ 1 [ a (cid:48) < a ] .
Revised: a n a (cid:48) ( s t )+ 1 [ a (cid:48) < a ] .

Original: According to K t ’s deﬁnition (Equation (14)),
Revised: According to K t ’s deﬁnition (Equation (14)),

Original: [Equation]
Revised: [Equation]

Original: We now pick an arbitrary subset B (cid:48) ⊆
Revised: We now pick an arbitrary subset B (cid:48) ⊆

Original: B such that | B (cid:48)
Revised: B such that | B (cid:48)

Original: | =
Revised: | =

Original: K t + 1 .
Revised: K t + 1 .

Original: For each i ∈ B (cid:48) , we locate its corresponding parititioned dataset D i for training subpolicy
Revised: For each i ∈ B (cid:48) , we locate its corresponding parititioned dataset D i for training subpolicy

Original: π i .
Revised: π i .

Original: We insert one point p i to D i , such that our chosen learning algorithm M 0 can train a subpolicy ˜
Revised: We insert one point p i to D i , such that our chosen learning algorithm M 0 can train a subpolicy ˜

Original: π
Revised: π

Original: i = M 

Revised: i = M

Original: ( D i ∪ { p i } ) that satisﬁes ˜ π
Revised: ( D i ∪ { p i } ) that satisﬁes ˜ π

Original: i ( s t ) = a (cid:48) .
Revised: i ( s t ) = a (cid:48) .

Original: For example, the point could be p i
Revised: For example, the point could be p i

Original: = { ( s t , a (cid:48) , s (cid:48) , ∞ ) } and M 0 learns the action with maximum reward for the memorized nearest state, where s (cid:48) can be adjusted to make sure the point is hashed to parition i .
Revised: = { ( s t , a (cid:48) , s (cid:48) , ∞ ) } and M 0 learns the action with maximum reward for the memorized nearest state, where s (cid:48) can be adjusted to make sure the point is hashed to parition i .

Original: Then, we construct the poisoned dataset
Revised: Then, we construct the poisoned dataset

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have (cid:101) D
Revised: Therefore, we have (cid:101) D

Original: [Equation]
Revised: [Equation]

Original: D = ∪ i ∈ B
Revised: D = ∪ i ∈ B

Original: [Equation]
Revised: [Equation]

Original: p i = | B (cid:48) | =
Revised: p i = | B (cid:48) | =

Original: K t + 1 .
Revised: K t + 1 .

Original: On this poisoned dataset, we train subpolicies { ˜ π i } u − 1 i =0 and get the aggregated policy (cid:102) π P .
Revised: On this poisoned dataset, we train subpolicies { ˜ π i } u − 1 i =0 and get the aggregated policy (cid:102) π P .

Original: To study (cid:102) π P ( s t ) , we compute the aggregated action count (cid:101) n a on these poisoned subpolicies.
Revised: To study (cid:102) π P ( s t ) , we compute the aggregated action count (cid:101) n a on these poisoned subpolicies.

Original: We found that
Revised: We found that

Original: (cid:101) n a ( s t )
Revised: (cid:101) n a ( s t )

Original: = n a ( s t ) − | B (cid:48)
Revised: = n a ( s t ) − | B (cid:48)

Original: | and (cid:101) n a (cid:48) ( s t )
Revised: | and (cid:101) n a (cid:48) ( s t )

Original: = n a (cid:48) ( s t ) + | B (cid:48) | .
Revised: = n a (cid:48) ( s t ) + | B (cid:48) | .

Original: Therefore, (cid:101) n a ( s t ) − (cid:101) n a (cid:48) ( s t ) =
Revised: Therefore, (cid:101) n a ( s t ) − (cid:101) n a (cid:48) ( s t ) =

Original: n a ( s t ) − n a (cid:48)
Revised: n a ( s t ) − n a (cid:48)

Original: [Equation]
Revised: [Equation]

Original: s t ) −
Revised: s t ) −

Original: [Equation]
Revised: [Equation]

Original: t +
Revised: t +

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: n a (cid:48)
Revised: n a (cid:48)

Original: [Equation]
Revised: [Equation]

Original: Therefore, if a (cid:48) < a , (cid:101) n a ( s t ) ≤ (cid:101) n a (cid:48) ( s t ) , and a (cid:48) has higher priority to be chosen than a ; if a (cid:48) > a , (cid:101) n a ( s t ) ≤ (cid:101) n a (cid:48) ( s t ) − 1 , and a (cid:48) still has higher priority to be chosen than a .
Revised: Therefore, if a (cid:48) < a , (cid:101) n a ( s t ) ≤ (cid:101) n a (cid:48) ( s t ) , and a (cid:48) has higher priority to be chosen than a ; if a (cid:48) > a , (cid:101) n a ( s t ) ≤ (cid:101) n a (cid:48) ( s t ) − 1 , and a (cid:48) still has higher priority to be chosen than a .

Original: Hence, (cid:102) π P ( s t ) (cid:54) =
Revised: Hence, (cid:102) π P ( s t ) (cid:54) =

Original: a = π P ( s t ) .
Revised: a = π P ( s t ) .

Original: To this point, we conclude the proof with a feasible construction.
Revised: To this point, we conclude the proof with a feasible construction.

Original: F. P ER -S TATE A CTION C ERTIFICATION IN TPARL
Revised: F. P ER -S TATE A CTION C ERTIFICATION IN TPARL

Original: Proof of Theorem 1.
Revised: Proof of Theorem 1.

Original: For ease of notation, we let w = min
Revised: For ease of notation, we let w = min

Original: { W, t +1 }
Revised: { W, t +1 }

Original: so w is the actual window size used at step t .
Revised: so w is the actual window size used at step t .

Original: We let t 

Revised: We let t

Original: = t − w + 1 , i.e. , t 0 is the actual start time step for TPARL aggregation at step t .
Revised: = t − w + 1 , i.e. , t 0 is the actual start time step for TPARL aggregation at step t .

Original: Now we can write the chosen action at step t without poisoning as a = π T ( s t 0 : t ) .
Revised: Now we can write the chosen action at step t without poisoning as a = π T ( s t 0 : t ) .

Original: We prove the theorem by contradiction: We assume that there is a poisoning attack whose poisoning size K ≤ K t where K t is deﬁned by Equation (3) in the theorem, and after poisoning, the chosen action is a T (cid:54)
Revised: We prove the theorem by contradiction: We assume that there is a poisoning attack whose poisoning size K ≤ K t where K t is deﬁned by Equation (3) in the theorem, and after poisoning, the chosen action is a T (cid:54)

Original: =
Revised: =

Original: a .
Revised: a .

Original: We denote { ˜ π i } u − 1 i =0 to the poisoned subpolicies and (cid:102)
Revised: We denote { ˜ π i } u − 1 i =0 to the poisoned subpolicies and (cid:102)

Original: π T to the poisoned TPARL aggregated policy.
Revised: π T to the poisoned TPARL aggregated policy.

Original: From the deﬁnition of K t , we have
Revised: From the deﬁnition of K t , we have

Original: [Equation]
Revised: [Equation]

Original: since K ≤ K t , and each h ( i ) a,a T is an element of h i (cid:48) ,a,a T for some i (cid:48) where
Revised: since K ≤ K t , and each h ( i ) a,a T is an element of h i (cid:48) ,a,a T for some i (cid:48) where

Original: [Equation]
Revised: [Equation]

Original: by Equation (4) in the theorem.
Revised: by Equation (4) in the theorem.

Original: Since the poisoning attack within threshold K can at most affect K subpolicies, we let B be the set of affected policies and assume | B | =
Revised: Since the poisoning attack within threshold K can at most affect K subpolicies, we let B be the set of affected policies and assume | B | =

Original: K without loss of generality.
Revised: K without loss of generality.

Original: Formally, B = { i ∈
Revised: Formally, B = { i ∈

Original: [ u ] | ∃ t (cid:48) ∈
Revised: [ u ] | ∃ t (cid:48) ∈

Original: [ t 0 , t ] , ˜ π
Revised: [ t 0 , t ] , ˜ π

Original: i ( s t (cid:48) )
Revised: i ( s t (cid:48) )

Original: (cid:54) =
Revised: (cid:54) =

Original: π i ( s t (cid:48) ) } .
Revised: π i ( s t (cid:48) ) } .

Original: Therefore, according to the monotonicity of h ( i ) a,a T , from Equation (15),
Revised: Therefore, according to the monotonicity of h ( i ) a,a T , from Equation (15),

Original: [Equation]
Revised: [Equation]

Original: According to the assumption of successfully poisoning attack, after attack, the sum of aggregated vote for a T plus 1 [ a T < a ] should be larger then that of a .
Revised: According to the assumption of successfully poisoning attack, after attack, the sum of aggregated vote for a T plus 1 [ a T < a ] should be larger then that of a .

Original: Formally, after the poisoning,
Revised: Formally, after the poisoning,

Original: [Equation]
Revised: [Equation]

Original: or equivalently
Revised: or equivalently

Original: [Equation]
Revised: [Equation]

Original: < 0 .
Revised: < 0 .

Original: From the statement of Theorem 1,
Revised: From the statement of Theorem 1,

Original: [Equation]
Revised: [Equation]

Original: Take the difference of the above two equations, we know that the attack satisﬁes
Revised: Take the difference of the above two equations, we know that the attack satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Since the attack only changes the subpolicies in B , we have
Revised: Since the attack only changes the subpolicies in B , we have

Original: [Equation]
Revised: [Equation]

Original:  
Revised:  

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: − w
Revised: − w

Original: [Equation]
Revised: [Equation]

Original: 1 [˜ π
Revised: 1 [˜ π

Original: [Equation]
Revised: [Equation]

Original: ( s
Revised: ( s

Original: [Equation]
Revised: [Equation]

Original: − j )
Revised: − j )

Original: = a T ] 
Revised: = a T ] 

Original: [Equation]
Revised: [Equation]

Original: Inject them into Equation (17) yields (cid:88) i ∈ B   w − 1 (cid:88)
Revised: Inject them into Equation (17) yields (cid:88) i ∈ B   w − 1 (cid:88)

Original: j = 1 i,a ( s t − j ) + w − w − 1 (cid:88)
Revised: j = 1 i,a ( s t − j ) + w − w − 1 (cid:88)

Original: j = 1 i,a T ( s t − j ) 
Revised: j = 1 i,a T ( s t − j ) 

Original:  (cid:124) (cid:123)(cid:122) (cid:125) h
Revised:  (cid:124) (cid:123)(cid:122) (cid:125) h

Original: i,a,aT > δ a,a T which contradicts Equation (16) and thus concludes the proof.
Revised: i,a,aT > δ a,a T which contradicts Equation (16) and thus concludes the proof.

Original: F. T IGHTNESS OF P ER -S TATE A CTION C ERTIFICATION IN TPARL
Revised: F. T IGHTNESS OF P ER -S TATE A CTION C ERTIFICATION IN TPARL

Original: Proof of Proposition 2.
Revised: Proof of Proposition 2.

Original: For ease of notation, we let w = min { W, t + 1 } so w is the actual window size used at step t .
Revised: For ease of notation, we let w = min { W, t + 1 } so w is the actual window size used at step t .

Original: We let t 

Revised: We let t

Original: = t − w + 1 , i.e. , t 0 is the actual start time step for TPARL aggregation at step t .
Revised: = t − w + 1 , i.e. , t 0 is the actual start time step for TPARL aggregation at step t .

Original: Now we can write the chosen action at step t without poisoning as a = π T ( s t 0 : t ) .
Revised: Now we can write the chosen action at step t without poisoning as a = π T ( s t 0 : t ) .

Original: We prove by construction, i.e. , we construct an poisoning attack with poisoning size K t + 1 that deviates the prediction of the poisoned policy.
Revised: We prove by construction, i.e. , we construct an poisoning attack with poisoning size K t + 1 that deviates the prediction of the poisoned policy.

Original: Speciﬁcally, we aim to craft a poisoned dataset (cid:101) D with poisoning size | D (cid:9) (cid:101) D | =
Revised: Speciﬁcally, we aim to craft a poisoned dataset (cid:101) D with poisoning size | D (cid:9) (cid:101) D | =

Original: K t +1 , such that for certain learning algorithm M 0 , after partitioning and learning on the poisoned dataset, the poisoned subpolicies ˜
Revised: K t +1 , such that for certain learning algorithm M 0 , after partitioning and learning on the poisoned dataset, the poisoned subpolicies ˜

Original: π
Revised: π

Original: i = M 0 ( (cid:101) D i ) can be aggregated to produce different action prediction: (cid:102) π T ( s t 0 : t ) (cid:54) = a .
Revised: i = M 0 ( (cid:101) D i ) can be aggregated to produce different action prediction: (cid:102) π T ( s t 0 : t ) (cid:54) = a .

Original: Before construction, we ﬁrst show that K t given by Equation (3) satisﬁes K t < u .
Revised: Before construction, we ﬁrst show that K t given by Equation (3) satisﬁes K t < u .

Original: If K t = u , it means that for arbitrary a (cid:48) (cid:54) = a
Revised: If K t = u , it means that for arbitrary a (cid:48) (cid:54) = a

Original: , u (cid:88) i =
Revised: , u (cid:88) i =

Original: h (
Revised: h (

Original: [Equation]
Revised: [Equation]

Original: = u −
Revised: = u −

Original: [Equation]
Revised: [Equation]

Original: (cid:88) i
Revised: (cid:88) i

Original: = h
Revised: = h

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: ,a (cid:48) =
Revised: ,a (cid:48) =

Original: u − 1 (cid:88) i =   w − 1 (cid:88)
Revised: u − 1 (cid:88) i =   w − 1 (cid:88)

Original: j = 1 i,a ( s t − j ) + w − w − 1 (cid:88)
Revised: j = 1 i,a ( s t − j ) + w − w − 1 (cid:88)

Original: j = 1 i,a (cid:48) ( s t − j ) 
Revised: j = 1 i,a (cid:48) ( s t − j ) 

Original:  = n a ( s t 0 : t ) + uw − n a (cid:48) ( s t 0 : t ) ≤ δ a,a (cid:48)
Revised:  = n a ( s t 0 : t ) + uw − n a (cid:48) ( s t 0 : t ) ≤ δ a,a (cid:48)

Original: = n a ( s t 0 : t ) − n a (cid:48) ( s t 0 : t ) − 1 [ a (cid:48) < a ] , which implies uw ≤ 0 contracting uw > 0 .
Revised: = n a ( s t 0 : t ) − n a (cid:48) ( s t 0 : t ) − 1 [ a (cid:48) < a ] , which implies uw ≤ 0 contracting uw > 0 .

Original: Now, we know K t < u , so K t + 1 , our poisoning size, is smaller or equal to u .
Revised: Now, we know K t < u , so K t + 1 , our poisoning size, is smaller or equal to u .

Original: We start our construction by choosing an action a T :
Revised: We start our construction by choosing an action a T :

Original: [Equation]
Revised: [Equation]

Original: According to the deﬁnition of K t in Equation (3), for a T we have
Revised: According to the deﬁnition of K t in Equation (3), for a T we have

Original: [Equation]
Revised: [Equation]

Original: We locate the subpolicies to poison as
Revised: We locate the subpolicies to poison as

Original: B = { i ∈
Revised: B = { i ∈

Original: [ u ] | h ia,a T is h ( j ) a,a T in the nonincreasing permutation in Equation (4) , j ≤ K t + 1 } (19)
Revised: [ u ] | h ia,a T is h ( j ) a,a T in the nonincreasing permutation in Equation (4) , j ≤ K t + 1 } (19)

Original: Therefore, | B | =
Revised: Therefore, | B | =

Original: K t + 1 and (cid:88) i ∈
Revised: K t + 1 and (cid:88) i ∈

Original: B h
Revised: B h

Original: i,a,
Revised: i,a,

Original: [Equation]
Revised: [Equation]

Original: a,
Revised: a,

Original: [Equation]
Revised: [Equation]

Original: T (20) by Equation (18).
Revised: T (20) by Equation (18).

Original: For each of these subpolicies i ∈ B , we locate its corresponding partitioned dataset D i for training subpolicy
Revised: For each of these subpolicies i ∈ B , we locate its corresponding partitioned dataset D i for training subpolicy

Original: π i , and insert one trajectory p i to D i , such that our chosen learning algorithm M 0 can train a subpolicy ˜
Revised: π i , and insert one trajectory p i to D i , such that our chosen learning algorithm M 0 can train a subpolicy ˜

Original: π
Revised: π

Original: i = M
Revised: i = M

Original: ( D i ∪ { p i } ) satisfying ˜
Revised: ( D i ∪ { p i } ) satisfying ˜

Original: π i ( s t (cid:48) )
Revised: π i ( s t (cid:48) )

Original: = a T for any t (cid:48) ∈ [ t 0 , t ] .
Revised: = a T for any t (cid:48) ∈ [ t 0 , t ] .

Original: For example, the trajectory could be p i
Revised: For example, the trajectory could be p i

Original: = { ( s t (cid:48) , a T , s (cid:48) , ∞ ) } tt (cid:48)
Revised: = { ( s t (cid:48) , a T , s (cid:48) , ∞ ) } tt (cid:48)

Original: =
Revised: =

Original: t 0 and M 0 learns the action with maximum reward for the memorized nearest state, where s (cid:48) can be adjusted to make sure the trajectory is hashed to partition i .
Revised: t 0 and M 0 learns the action with maximum reward for the memorized nearest state, where s (cid:48) can be adjusted to make sure the trajectory is hashed to partition i .

Original: Then, we construct the poisoned dataset
Revised: Then, we construct the poisoned dataset

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have (cid:101) D (cid:9) D
Revised: Therefore, we have (cid:101) D (cid:9) D

Original: = ∪ i ∈ B (cid:48)
Revised: = ∪ i ∈ B (cid:48)

Original: p i = | B (cid:48)
Revised: p i = | B (cid:48)

Original: | =
Revised: | =

Original: K t + 1 .
Revised: K t + 1 .

Original: Now, we compare the aggregated votes for a and a T before and after poisoning:
Revised: Now, we compare the aggregated votes for a and a T before and after poisoning:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Now we compare the margin between aggregated votes for a and a T after poisoning:
Revised: Now we compare the margin between aggregated votes for a and a T after poisoning:

Original: [Equation]
Revised: [Equation]

Original: As a result
Revised: As a result

Original: [Equation]
Revised: [Equation]

Original: after poisoning
Revised: after poisoning

Original: [Equation]
Revised: [Equation]

Original: a T has higher priority to be chosen than a , i.e. , (cid:102) π T ( s t 0 : t ) (cid:54) = a = π T ( s t 0 : t ) .
Revised: a T has higher priority to be chosen than a , i.e. , (cid:102) π T ( s t 0 : t ) (cid:54) = a = π T ( s t 0 : t ) .

Original: To this point, we conclude the proof with a feasible attack construction.
Revised: To this point, we conclude the proof with a feasible attack construction.

Original: F. L OOSE P ER -S TATE A CTION C ERTIFICATION IN TPARL AND C OMPARISION WITH T IGHT O NE
Revised: F. L OOSE P ER -S TATE A CTION C ERTIFICATION IN TPARL AND C OMPARISION WITH T IGHT O NE

Original: The following corollary of Theorem 8 states a loose per-state action certiﬁcation.
Revised: The following corollary of Theorem 8 states a loose per-state action certiﬁcation.

Original: Corollary 10.
Revised: Corollary 10.

Original: Under the same condition as Theorem 1,
Revised: Under the same condition as Theorem 1,

Original: [Equation]
Revised: [Equation]

Original: is a tolerable poisoning threshold at time step t in Deﬁnition 1, where W is the window size.
Revised: is a tolerable poisoning threshold at time step t in Deﬁnition 1, where W is the window size.

Original: Proof of Corollary 10.
Revised: Proof of Corollary 10.

Original: We let w = min { t +1 , W } to be the actual aggregation window size at time step t .
Revised: We let w = min { t +1 , W } to be the actual aggregation window size at time step t .

Original: After poisoning with size K , at most K subpolicies are affected and each affected policy can only make ± w changes to the aggregated action count.
Revised: After poisoning with size K , at most K subpolicies are affected and each affected policy can only make ± w changes to the aggregated action count.

Original: This implies that, after poisoning, for any action a (cid:48) ∈
Revised: This implies that, after poisoning, for any action a (cid:48) ∈

Original: A , the aggregated vote count (cid:101) n a (cid:48) ( s t − w +1: t ) ∈ [ n
Revised: A , the aggregated vote count (cid:101) n a (cid:48) ( s t − w +1: t ) ∈ [ n

Original: [Equation]
Revised: [Equation]

Original: (cid:48) ( s
Revised: (cid:48) ( s

Original: [Equation]
Revised: [Equation]

Original: w
Revised: w

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) − uw, n a (cid:48) ( s t − w +1:
Revised: ) − uw, n a (cid:48) ( s t − w +1:

Original: [Equation]
Revised: [Equation]

Original: ) + uw ] .
Revised: ) + uw ] .

Original: Thus, when K ≤ K t where K t is deﬁned in Theorem 1, for any a (cid:48) (cid:54) = a , we have ˜ n a ( s t − w +
Revised: Thus, when K ≤ K t where K t is deﬁned in Theorem 1, for any a (cid:48) (cid:54) = a , we have ˜ n a ( s t − w +

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: − ˜ n
Revised: − ˜ n

Original: [Equation]
Revised: [Equation]

Original: ( s
Revised: ( s

Original: [Equation]
Revised: [Equation]

Original: w
Revised: w

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: t ) − 1 [ a
Revised: t ) − 1 [ a

Original: [Equation]
Revised: [Equation]

Original: < a ]
Revised: < a ]

Original: = n a ( s t
Revised: = n a ( s t

Original: [Equation]
Revised: [Equation]

Original: w
Revised: w

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: ( s
Revised: ( s

Original: [Equation]
Revised: [Equation]

Original: a (cid:48)
Revised: a (cid:48)

Original: < a ] − 2 Kw
Revised: < a ] − 2 Kw

Original: [Equation]
Revised: [Equation]

Original: From the deﬁnition of TPARL protocol, the poisoned policy still chooses action a , which implies
Revised: From the deﬁnition of TPARL protocol, the poisoned policy still chooses action a , which implies

Original: K t is a tolerable poisoning threshold.
Revised: K t is a tolerable poisoning threshold.

Original: In the main text, we mention that the certiﬁcation from Corollary 10 is looser than that from Theorem 1.
Revised: In the main text, we mention that the certiﬁcation from Corollary 10 is looser than that from Theorem 1.

Original: This assertion is based on the following two facts:
Revised: This assertion is based on the following two facts:

Original: According to Proposition 2, the certiﬁcation given by Theorem 1 is theoretically tight, which means that any other certiﬁcation can only be as tight as Theorem 1 or looser than it.
Revised: According to Proposition 2, the certiﬁcation given by Theorem 1 is theoretically tight, which means that any other certiﬁcation can only be as tight as Theorem 1 or looser than it.

Original: There exists examples where the computed K t from Theorem 1 is larger than that from Corollary 10.
Revised: There exists examples where the computed K t from Theorem 1 is larger than that from Corollary 10.

Original: For instance, suppose W = 5 , action set A
Revised: For instance, suppose W = 5 , action set A

Original: = { a 1 , a 2 } , and
Revised: = { a 1 , a 2 } , and

Original: there
Revised: there

Original: are three subpolicies
Revised: are three subpolicies

Original: At time step t
Revised: At time step t

Original: = 4 , π
Revised: = 4 , π

Original: i
Revised: i

Original: for
Revised: for

Original: s to s 4 are [ a 1 , a
Revised: s to s 4 are [ a 1 , a

Original: , a
Revised: , a

Original: , a , a
Revised: , a , a

Original: ] for
Revised: ] for

Original: all subpolicies
Revised: all subpolicies

Original: i.e. , i
Revised: i.e. , i

Original: ∈
Revised: ∈

Original: Thus, the benign policy π T ( s 0:4 ) = a 1 .
Revised: Thus, the benign policy π T ( s 0:4 ) = a 1 .

Original: By computation, the K t from Theorem 1 is 1 ; while the K t from
Revised: By computation, the K t from Theorem 1 is 1 ; while the K t from

Original: Corollary 10 is 0 .
Revised: Corollary 10 is 0 .

Original: Indeed, Corollary 10 can be viewed as using 2 w to upper bound h
Revised: Indeed, Corollary 10 can be viewed as using 2 w to upper bound h

Original: i,a,a (cid:48)
Revised: i,a,a (cid:48)

Original: =
Revised: =

Original: w + (cid:80) w − 1 j =0 1 i,a ( s t − j ) − (cid:80) w − 1 j =0 1 i,a (cid:48) ( s t − j ) .
Revised: w + (cid:80) w − 1 j =0 1 i,a ( s t − j ) − (cid:80) w − 1 j =0 1 i,a (cid:48) ( s t − j ) .

Original: Intuitively, Corollary 10 assumes every subpolicy can provide 2 w vote margin shrinkage, and Theorem 1 uses h i,a,a (cid:48) to capture the precise worse-case margin shrinkage and thus provides a tighter certiﬁcation.
Revised: Intuitively, Corollary 10 assumes every subpolicy can provide 2 w vote margin shrinkage, and Theorem 1 uses h i,a,a (cid:48) to capture the precise worse-case margin shrinkage and thus provides a tighter certiﬁcation.

Original: F. P ER -S TATE A CTION C ERTIFICATION IN DPARL
Revised: F. P ER -S TATE A CTION C ERTIFICATION IN DPARL

Original: Proof of Theorem 3.
Revised: Proof of Theorem 3.

Original: Without loss of generality, we assume W max ≤ t + 

Revised: Without loss of generality, we assume W max ≤ t +

Original: and otherwise we let
Revised: and otherwise we let

Original: W max ← min { W max , t + 1 } .
Revised: W max ← min { W max , t + 1 } .

Original: We let t 0 =
Revised: We let t 0 =

Original: max { t − W max + 1 , 0 } be the start time step of the maximum possible window.
Revised: max { t − W max + 1 , 0 } be the start time step of the maximum possible window.

Original: To prove the theorem, our general methodology is to enumerate all possible cases of a successful attack, and derive the tolerable poisoning threshold for each case respectively.
Revised: To prove the theorem, our general methodology is to enumerate all possible cases of a successful attack, and derive the tolerable poisoning threshold for each case respectively.

Original: Taking a minimum over these tolerable poisoning thresholds gives the required result.
Revised: Taking a minimum over these tolerable poisoning thresholds gives the required result.

Original: Speciﬁcally, we denote P to the predicate of robustness under poisoning attack: P =
Revised: Speciﬁcally, we denote P to the predicate of robustness under poisoning attack: P =

Original: [ (cid:102) π D ( s t 0 : t ) = a ] , and denote K to the poisoning attack size.
Revised: [ (cid:102) π D ( s t 0 : t ) = a ] , and denote K to the poisoning attack size.

Original: Therefore, we can decompose P as such:
Revised: Therefore, we can decompose P as such:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Recall that W (cid:48) is the chosen window size by the protocol DPARL with unattacked subpolicies π D (Equation (1)).
Revised: Recall that W (cid:48) is the chosen window size by the protocol DPARL with unattacked subpolicies π D (Equation (1)).

Original: In Equation (22), the predicate P ( W (cid:48) ) means that after poisoning attack, whether the prediction under window size W (cid:48) is still a ; the predicate Q ( W ∗ , a (cid:48) ) means that after poisoning attack, whether the chosen action is a (cid:48) at window size W ∗ and average vote margin is larger (or equal if a (cid:48) <
Revised: In Equation (22), the predicate P ( W (cid:48) ) means that after poisoning attack, whether the prediction under window size W (cid:48) is still a ; the predicate Q ( W ∗ , a (cid:48) ) means that after poisoning attack, whether the chosen action is a (cid:48) at window size W ∗ and average vote margin is larger (or equal if a (cid:48) <

Original: a ) at window size W ∗ compared to W (cid:48) .
Revised: a ) at window size W ∗ compared to W (cid:48) .

Original: Formally, let (cid:101) n a be the aggregated action count after poisoning and (cid:101)
Revised: Formally, let (cid:101) n a be the aggregated action count after poisoning and (cid:101)

Original: ∆
Revised: ∆

Original: Wt be the average vote margin after poisoning at window W (see Equation (2)),
Revised: Wt be the average vote margin after poisoning at window W (see Equation (2)),

Original: [Equation]
Revised: [Equation]

Original: According to Theorem 1,
Revised: According to Theorem 1,

Original: [Equation]
Revised: [Equation]

Original: where K t is deﬁned by Equation (3) with W replaced by W (cid:48) .
Revised: where K t is deﬁned by Equation (3) with W replaced by W (cid:48) .

Original: The following Lemma 11 shows a sufﬁcient condition for Q ( W ∗ , a (cid:48) ) .
Revised: The following Lemma 11 shows a sufﬁcient condition for Q ( W ∗ , a (cid:48) ) .

Original: We then aggregate these conditions together with minimum to obtain a sufﬁcient condition for P :
Revised: We then aggregate these conditions together with minimum to obtain a sufﬁcient condition for P :

Original: [Equation]
Revised: [Equation]

Original: and thus conclude the proof
Revised: and thus conclude the proof

Original: [Equation]
Revised: [Equation]

Original: Lemma 11.
Revised: Lemma 11.

Original: Let Q ( W ∗ , a (cid:48) ) , K , W (cid:48) be the same as deﬁned in proof of Theorem 3, then
Revised: Let Q ( W ∗ , a (cid:48) ) , K , W (cid:48) be the same as deﬁned in proof of Theorem 3, then

Original: [Equation]
Revised: [Equation]

Original: where L W ∗ ,W (cid:48) a (cid:48) ,a (cid:48)(cid:48) is deﬁned in Deﬁnition 4.
Revised: where L W ∗ ,W (cid:48) a (cid:48) ,a (cid:48)(cid:48) is deﬁned in Deﬁnition 4.

Original: Proof.
Revised: Proof.

Original: We prove the equivalent form:
Revised: We prove the equivalent form:

Original: [Equation]
Revised: [Equation]

Original: Suppose a poisoning attack can successfully achieve Q ( W ∗ , a (cid:48) )
Revised: Suppose a poisoning attack can successfully achieve Q ( W ∗ , a (cid:48) )

Original: [Equation]
Revised: [Equation]

Original: we now induce the requirement on its poisoning size K .
Revised: we now induce the requirement on its poisoning size K .

Original: First, we notice that
Revised: First, we notice that

Original: [Equation]
Revised: [Equation]

Original: since
Revised: since

Original: [Equation]
Revised: [Equation]

Original: is an integer by deﬁnition.
Revised: is an integer by deﬁnition.

Original: According to the deﬁnition and Q ( W ∗ , a (cid:48) )
Revised: According to the deﬁnition and Q ( W ∗ , a (cid:48) )

Original: ’s assumption that arg max a ∈A (cid:101) n a ( s t − W ∗ +1: t ) = a (cid:48)
Revised: ’s assumption that arg max a ∈A (cid:101) n a ( s t − W ∗ +1: t ) = a (cid:48)

Original: [Equation]
Revised: [Equation]

Original: , where “ ≤ ” comes from the fact that the margin in (cid:101) ∆ W ∗ t should be with respect to the runner-up class after poisoning, and computing with respect to any other class provides an upper bound.
Revised: , where “ ≤ ” comes from the fact that the margin in (cid:101) ∆ W ∗ t should be with respect to the runner-up class after poisoning, and computing with respect to any other class provides an upper bound.

Original: Here we choose a # = arg max a 0 (cid:54) = a (cid:48) ,a 0 ∈A n a 0 ( s t − W ∗
Revised: Here we choose a # = arg max a 0 (cid:54) = a (cid:48) ,a 0 ∈A n a 0 ( s t − W ∗

Original: +1: t ) (see Deﬁnition 4), the runner-up class before poisoning, to empirically shrink the gap between the bound and actual margin.
Revised: +1: t ) (see Deﬁnition 4), the runner-up class before poisoning, to empirically shrink the gap between the bound and actual margin.

Original: On the other hand,
Revised: On the other hand,

Original: [Equation]
Revised: [Equation]

Original: where “ ≥ ” comes from the fact that the margin in (cid:101) ∆ W (cid:48) t should use the top class after poisoning, and computing with any other class provides a lower bound.
Revised: where “ ≥ ” comes from the fact that the margin in (cid:101) ∆ W (cid:48) t should use the top class after poisoning, and computing with any other class provides a lower bound.

Original: Thus, from Equation (26) and the above two relaxations, we get
Revised: Thus, from Equation (26) and the above two relaxations, we get

Original: [Equation]
Revised: [Equation]

Original: ⇒∃
Revised: ⇒∃

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: For each a (cid:48)(cid:48) (cid:54) = a , now we use the last equation as the condition, and show that K > L W ∗ ,W (cid:48) a (cid:48) ,a (cid:48)(cid:48) is a necessary condition.
Revised: For each a (cid:48)(cid:48) (cid:54) = a , now we use the last equation as the condition, and show that K > L W ∗ ,W (cid:48) a (cid:48) ,a (cid:48)(cid:48) is a necessary condition.

Original: This proposition is equivalent to
Revised: This proposition is equivalent to

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: > a ] .
Revised: > a ] .

Original: Suppose a poisoning attack within poisoning size K changes the subpolicies in set B ⊆
Revised: Suppose a poisoning attack within poisoning size K changes the subpolicies in set B ⊆

Original: [ u ] .
Revised: [ u ] .

Original: Note that | B | ≤ K .
Revised: Note that | B | ≤ K .

Original: We inspect the objective in Equation (27):
Revised: We inspect the objective in Equation (27):

Original: [Equation]
Revised: [Equation]

Original: 1 [ π i ( s t − w +1 ) = a (cid:48) ] − W (cid:48)
Revised: 1 [ π i ( s t − w +1 ) = a (cid:48) ] − W (cid:48)

Original: W ∗ − 1 (cid:88) w = 1 [ π i ( s t − w +1 )
Revised: W ∗ − 1 (cid:88) w = 1 [ π i ( s t − w +1 )

Original: = a # ] − W ∗
Revised: = a # ] − W ∗

Original: W (cid:48) − 1 (cid:88) w = 1 [ π i ( s t − w +1 )
Revised: W (cid:48) − 1 (cid:88) w = 1 [ π i ( s t − w +1 )

Original: = a ] + W ∗
Revised: = a ] + W ∗

Original: W (cid:48) − 1 (cid:88) w = 1 [ π i ( s t − w +1 )
Revised: W (cid:48) − 1 (cid:88) w = 1 [ π i ( s t − w +1 )

Original: = a (cid:48)(cid:48) ] 
Revised: = a (cid:48)(cid:48) ] 

Original:  =
Revised:  =

Original: W (cid:48)
Revised: W (cid:48)

Original: ( n a (cid:48) ( s t − W ∗ +1: t ) − n a # ( s t − W ∗
Revised: ( n a (cid:48) ( s t − W ∗ +1: t ) − n a # ( s t − W ∗

Original: +1: t ))
Revised: +1: t ))

Original: − W ∗ ( n a ( s t − W (cid:48) +1: t ) − n a (cid:48)(cid:48) ( s t − W (cid:48)
Revised: − W ∗ ( n a ( s t − W (cid:48) +1: t ) − n a (cid:48)(cid:48) ( s t − W (cid:48)

Original: +1: t ))
Revised: +1: t ))

Original: − 1 [ a (cid:48)
Revised: − 1 [ a (cid:48)

Original: > a ] + (cid:88) i
Revised: > a ] + (cid:88) i

Original: ∈
Revised: ∈

Original: B max { W ∗ ,W (cid:48) } (cid:88)
Revised: B max { W ∗ ,W (cid:48) } (cid:88)

Original: w = σ w (˜ π
Revised: w = σ w (˜ π

Original: i ( s t − w )) − σ w ( π i ( s t − w )) ≤ W (cid:48)
Revised: i ( s t − w )) − σ w ( π i ( s t − w )) ≤ W (cid:48)

Original: ( n a (cid:48) ( s t − W ∗ +1: t ) − n a # ( s t − W ∗
Revised: ( n a (cid:48) ( s t − W ∗ +1: t ) − n a # ( s t − W ∗

Original: +1: t ))
Revised: +1: t ))

Original: − W ∗ ( n a ( s t − W (cid:48) +1: t ) − n a (cid:48)(cid:48) ( s t − W (cid:48)
Revised: − W ∗ ( n a ( s t − W (cid:48) +1: t ) − n a (cid:48)(cid:48) ( s t − W (cid:48)

Original: +1: t ))
Revised: +1: t ))

Original: − 1 [ a (cid:48)
Revised: − 1 [ a (cid:48)

Original: > a ] + (cid:88) i
Revised: > a ] + (cid:88) i

Original: ∈
Revised: ∈

Original: B max { W ∗ ,W (cid:48) } (cid:88) w = max a 0 ∈A σ w ( a 0 ) − σ w ( π i ( s t − w ))
Revised: B max { W ∗ ,W (cid:48) } (cid:88) w = max a 0 ∈A σ w ( a 0 ) − σ w ( π i ( s t − w ))

Original: (cid:124) (cid:123)(cid:122) (cid:125)
Revised: (cid:124) (cid:123)(cid:122) (cid:125)

Original: g i ( a ) ≤ W (cid:48)
Revised: g i ( a ) ≤ W (cid:48)

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: i
Revised: i

Original: [Table]
Revised: [Table]

Original: ( c ) < 0 .
Revised: ( c ) < 0 .

Original: Thus, Equation (27) is proved.
Revised: Thus, Equation (27) is proved.

Original: Therefore, K > L W ∗ ,W (cid:48) a (cid:48) ,a (cid:48)(cid:48) is a necessary condition for Q ( W ∗ , a (cid:48) ) , i.e. , Equation (25).
Revised: Therefore, K > L W ∗ ,W (cid:48) a (cid:48) ,a (cid:48)(cid:48) is a necessary condition for Q ( W ∗ , a (cid:48) ) , i.e. , Equation (25).

Original: In the above deriviation, the deﬁnitions of g i , g i , and σ w are from Equation (7).
Revised: In the above deriviation, the deﬁnitions of g i , g i , and σ w are from Equation (7).

Original: ( a ) comes from the facts that { g ( i ) } ui =1 is a nondecreasing permutation of { g i } u − 1 i =0 , g i ≥ 0 , and | B | ≤ K .
Revised: ( a ) comes from the facts that { g ( i ) } ui =1 is a nondecreasing permutation of { g i } u − 1 i =0 , g i ≥ 0 , and | B | ≤ K .

Original: ( b ) comes from the assumption K ≤ L W ∗ ,W (cid:48) a,a (cid:48)(cid:48) and also g ( i ) ≥ 0 .
Revised: ( b ) comes from the assumption K ≤ L W ∗ ,W (cid:48) a,a (cid:48)(cid:48) and also g ( i ) ≥ 0 .

Original: ( c ) comes from the deﬁnition in
Revised: ( c ) comes from the deﬁnition in

Original: Equation (6).
Revised: Equation (6).

Original: F.
Revised: F.

Original: P OSSIBLE A CTION S ET IN PARL AND C OMPARISON F.6. C ERTIFICATION
Revised: P OSSIBLE A CTION S ET IN PARL AND C OMPARISON F.6. C ERTIFICATION

Original: Proof of Theorem 4.
Revised: Proof of Theorem 4.

Original: According to the deﬁnition of possible action set, we only need to prove the contrary: for any a ∈ A \ A T ( K ) , within poisoning size K , the poisoned policy cannot choose a : (cid:102) π P ( s t ) (cid:54) = a .
Revised: According to the deﬁnition of possible action set, we only need to prove the contrary: for any a ∈ A \ A T ( K ) , within poisoning size K , the poisoned policy cannot choose a : (cid:102) π P ( s t ) (cid:54) = a .

Original: According to Equation (8), any a ∈ A \ A T ( K ) satisﬁes (cid:88)
Revised: According to Equation (8), any a ∈ A \ A T ( K ) satisﬁes (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Given poisoning size K , since each poisoning size can affect only one subpolicy, we know
Revised: Given poisoning size K , since each poisoning size can affect only one subpolicy, we know

Original: [Equation]
Revised: [Equation]

Original: where (cid:101) n a denotes to the poisoned aggregated action count.
Revised: where (cid:101) n a denotes to the poisoned aggregated action count.

Original: We suppose the attack could be successful, then for a (cid:48) <
Revised: We suppose the attack could be successful, then for a (cid:48) <

Original: a , (cid:101) n a (cid:48)
Revised: a , (cid:101) n a (cid:48)

Original: ( s t ) ≤ (cid:101) n a ( s t ) − 1 , and thus n a (cid:48)
Revised: ( s t ) ≤ (cid:101) n a ( s t ) − 1 , and thus n a (cid:48)

Original: ( s t ) − (cid:101) n a (cid:48)
Revised: ( s t ) − (cid:101) n a (cid:48)

Original: ( s t ) ≥ n a (cid:48)
Revised: ( s t ) ≥ n a (cid:48)

Original: ( s t ) − n a ( s t ) − K + 1 .
Revised: ( s t ) − n a ( s t ) − K + 1 .

Original: Similarly, for a (cid:48) > a , (cid:101) a (cid:48) ( s t ) ≤ (cid:101) n a ( s t ) , and thus n a (cid:48)
Revised: Similarly, for a (cid:48) > a , (cid:101) a (cid:48) ( s t ) ≤ (cid:101) n a ( s t ) , and thus n a (cid:48)

Original: ( s t ) − (cid:101) n a (cid:48)
Revised: ( s t ) − (cid:101) n a (cid:48)

Original: ( s t ) ≥ n a (cid:48)
Revised: ( s t ) ≥ n a (cid:48)

Original: ( s t ) − n a ( s t ) − K .
Revised: ( s t ) − n a ( s t ) − K .

Original: Also, for any a (cid:48) (cid:54) =
Revised: Also, for any a (cid:48) (cid:54) =

Original: a after poisoning n a (cid:48) ( s t ) − (cid:101) n a (cid:48)
Revised: a after poisoning n a (cid:48) ( s t ) − (cid:101) n a (cid:48)

Original: ( s t ) ≥ 0 ; otherwise deviating the difference subpolicies’ decisions’ from a (cid:48) to a is strictly no-worse.
Revised: ( s t ) ≥ 0 ; otherwise deviating the difference subpolicies’ decisions’ from a (cid:48) to a is strictly no-worse.

Original: Given these facts, the amount of votes that need to be reduced is the LHS of Equation (28) which is larger than K .
Revised: Given these facts, the amount of votes that need to be reduced is the LHS of Equation (28) which is larger than K .

Original: However, we only have K poisoning size, i.e. , K votes that can be reduced.
Revised: However, we only have K poisoning size, i.e. , K votes that can be reduced.

Original: As a result, our assumption that the attack could be successful is falsiﬁed and the poisoned policy cannot choose a .
Revised: As a result, our assumption that the attack could be successful is falsiﬁed and the poisoned policy cannot choose a .

Original: Corollary 12 (Loose PARL Action Set) .
Revised: Corollary 12 (Loose PARL Action Set) .

Original: Under the condition of Deﬁnition 5, suppose the aggregation protocol is PARL as deﬁned in Deﬁnition 7,
Revised: Under the condition of Deﬁnition 5, suppose the aggregation protocol is PARL as deﬁned in Deﬁnition 7,

Original: then the possible action set at step t
Revised: then the possible action set at step t

Original: [Equation]
Revised: [Equation]

Original: Proof of Corollary 12.
Revised: Proof of Corollary 12.

Original: Again, we prove the contrary, for any a ∈ A\ A L ( K ) , within poisoning size
Revised: Again, we prove the contrary, for any a ∈ A\ A L ( K ) , within poisoning size

Original: K , the poisoned policy cannot choose a : (cid:102) π P ( s t ) (cid:54) = a .
Revised: K , the poisoned policy cannot choose a : (cid:102) π P ( s t ) (cid:54) = a .

Original: According to Equation (29), let a m = arg max a (cid:48) ∈A n a
Revised: According to Equation (29), let a m = arg max a (cid:48) ∈A n a

Original: [Equation]
Revised: [Equation]

Original: ( s t ) , then any a ∈ A \ A L ( K ) satisﬁes
Revised: ( s t ) , then any a ∈ A \ A L ( K ) satisﬁes

Original: [Equation]
Revised: [Equation]

Original: After poisoning, we thus have n
Revised: After poisoning, we thus have n

Original: [Equation]
Revised: [Equation]

Original: n a
Revised: n a

Original: [Equation]
Revised: [Equation]

Original: s t
Revised: s t

Original: [Equation]
Revised: [Equation]

Original: > − 1 [ a > a m ] = ⇒ n
Revised: > − 1 [ a > a m ] = ⇒ n

Original: [Equation]
Revised: [Equation]

Original: m ( s t
Revised: m ( s t

Original: [Equation]
Revised: [Equation]

Original: − n a
Revised: − n a

Original: [Equation]
Revised: [Equation]

Original: s t
Revised: s t

Original: [Equation]
Revised: [Equation]

Original: ≥ 1 − 1 [ a > a m ]
Revised: ≥ 1 − 1 [ a > a m ]

Original: From the deﬁnition, a / ∈
Revised: From the deﬁnition, a / ∈

Original: A L ( K ) so a (cid:54) = a m .
Revised: A L ( K ) so a (cid:54) = a m .

Original: If a m < a , n a m ( s t ) ≥ n a ( s t ) ; if a m > a , n a m ( s t ) > n a ( s t ) .
Revised: If a m < a , n a m ( s t ) ≥ n a ( s t ) ; if a m > a , n a m ( s t ) > n a ( s t ) .

Original: In both cases, a m has higher priority to be chosen than a , and thus the poisoned policy cannot choose a .
Revised: In both cases, a m has higher priority to be chosen than a , and thus the poisoned policy cannot choose a .

Original: F.6. C OMPARISON
Revised: F.6. C OMPARISON

Original: Theorem 13.
Revised: Theorem 13.

Original: Under the condition of Deﬁnition 5, suppose the aggregation protocol is PARL as deﬁned in Deﬁnition 7, A T ( K ) , A L ( K ) are deﬁned according to Equations (8) and (29) accordingly, then
Revised: Under the condition of Deﬁnition 5, suppose the aggregation protocol is PARL as deﬁned in Deﬁnition 7, A T ( K ) , A L ( K ) are deﬁned according to Equations (8) and (29) accordingly, then

Original: A T ( K ) ⊆
Revised: A T ( K ) ⊆

Original: A L ( K ) ; and there are subpolicies { π i } u −
Revised: A L ( K ) ; and there are subpolicies { π i } u −

Original: Given subpolicies { π i } u − 1 i =0 and state s t , for any a ∈ A T ( K ) , there exists a poisoned training set (cid:101) D whose poisoning size | D (cid:9) (cid:101) D | ≤ K and some RL training mechanism, such that (cid:102) π P ( s t ) =
Revised: Given subpolicies { π i } u − 1 i =0 and state s t , for any a ∈ A T ( K ) , there exists a poisoned training set (cid:101) D whose poisoning size | D (cid:9) (cid:101) D | ≤ K and some RL training mechanism, such that (cid:102) π P ( s t ) =

Original: a where (cid:102) π P is the poisoned PARL policy trained on (cid:101) D .
Revised: a where (cid:102) π P is the poisoned PARL policy trained on (cid:101) D .

Original: Proof of Theorem 13.
Revised: Proof of Theorem 13.

Original: We prove the two arguments separately.
Revised: We prove the two arguments separately.

Original: We ﬁrst prove A T ( K ) ⊆
Revised: We ﬁrst prove A T ( K ) ⊆

Original: A L ( K ) .
Revised: A L ( K ) .

Original: For any a ∈ A T
Revised: For any a ∈ A T

Original: K
Revised: K

Original: let a m
Revised: let a m

Original: = arg max a
Revised: = arg max a

Original: [Equation]
Revised: [Equation]

Original: n a
Revised: n a

Original: [Equation]
Revised: [Equation]

Original: s t ) , then
Revised: s t ) , then

Original: (cid:88)
Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: where the last proposition is exactly the set selector of A L ( K ) so a ∈
Revised: where the last proposition is exactly the set selector of A L ( K ) so a ∈

Original: A L ( K ) .
Revised: A L ( K ) .

Original: We then prove that A T ( K ) (cid:40)
Revised: We then prove that A T ( K ) (cid:40)

Original: A L ( K ) can happen by construction.
Revised: A L ( K ) can happen by construction.

Original: Suppose that there are three actions in the action space:
Revised: Suppose that there are three actions in the action space:

Original: A = { a 1 , a 2 , a 3 } .
Revised: A = { a 1 , a 2 , a 3 } .

Original: We construct subpolicies for current state s t such that the aggregated action counts are
Revised: We construct subpolicies for current state s t such that the aggregated action counts are

Original: [Equation]
Revised: [Equation]

Original: Given poisoning size K = 5 , we ﬁnd that
Revised: Given poisoning size K = 5 , we ﬁnd that

Original: [Equation]
Revised: [Equation]

Original: ]) +
Revised: ]) +

Original: [Equation]
Revised: [Equation]

Original: < a 3 ]) =
Revised: < a 3 ]) =

Original: 9 > K =
Revised: 9 > K =

Original: [Equation]
Revised: [Equation]

Original: = ⇒
Revised: = ⇒

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: / ∈ a T ( K ) .
Revised: / ∈ a T ( K ) .

Original: Therefore, A T ( K ) (cid:40)
Revised: Therefore, A T ( K ) (cid:40)

Original: A L ( K ) for these subpolicies and state s t .
Revised: A L ( K ) for these subpolicies and state s t .

Original: We prove by construction.
Revised: We prove by construction.

Original: For any a ∈ A T ( K ) , we construct the set of subpolicies to poison
Revised: For any a ∈ A T ( K ) , we construct the set of subpolicies to poison

Original: B a ⊆
Revised: B a ⊆

Original: [ u ] such that | B a | ≤ K , then describe the corresponding poisoned dataset (cid:101) D a , and ﬁnally prove that the poisoned policy (cid:102) π P a ( s t )
Revised: [ u ] such that | B a | ≤ K , then describe the corresponding poisoned dataset (cid:101) D a , and ﬁnally prove that the poisoned policy (cid:102) π P a ( s t )

Original: = a .
Revised: = a .

Original: For a ∈ A T ( K ) , by deﬁnition (Equation (8)), we know (cid:88)
Revised: For a ∈ A T ( K ) , by deﬁnition (Equation (8)), we know (cid:88)

Original: [Equation]
Revised: [Equation]

Original: We now deﬁne a set of actions C a ⊆
Revised: We now deﬁne a set of actions C a ⊆

Original: A such that
Revised: A such that

Original: [Equation]
Revised: [Equation]

Original: According to this deﬁnition, a / ∈ C a since t a,a ≤ 0 .
Revised: According to this deﬁnition, a / ∈ C a since t a,a ≤ 0 .

Original: Fact F.1.
Revised: Fact F.1.

Original: For a ∈ A T ( K ) and any a (cid:48) ∈ A , n a ( s t )
Revised: For a ∈ A T ( K ) and any a (cid:48) ∈ A , n a ( s t )

Original: + K − 1 [ a (cid:48) <
Revised: + K − 1 [ a (cid:48) <

Original: a ] ≥ 0 .
Revised: a ] ≥ 0 .

Original: Proof of Fact F.1.
Revised: Proof of Fact F.1.

Original: Suppose n a ( s t ) + K − 1 [ a (cid:48) < a ] ≤ 0 , then n a ( s t )
Revised: Suppose n a ( s t ) + K − 1 [ a (cid:48) < a ] ≤ 0 , then n a ( s t )

Original: =
Revised: =

Original: K = 0 and a (cid:48) < a .
Revised: K = 0 and a (cid:48) < a .

Original: Then (cid:88) a (cid:48) ∈A max { n a (cid:48) ( s t ) − n a ( s t ) − K + 1 [ a (cid:48)
Revised: Then (cid:88) a (cid:48) ∈A max { n a (cid:48) ( s t ) − n a ( s t ) − K + 1 [ a (cid:48)

Original: < a ] , 0 } ≥ (cid:88) a (cid:48)
Revised: < a ] , 0 } ≥ (cid:88) a (cid:48)

Original: [Equation]
Revised: [Equation]

Original: which contradicts the requirement that the LHS of the above inequality should be ≤ K = 0 .
Revised: which contradicts the requirement that the LHS of the above inequality should be ≤ K = 0 .

Original: Give Fact F.1, for any a (cid:48) ∈ C a , n a (cid:48) ( s t ) ≥ t a,a (cid:48) .
Revised: Give Fact F.1, for any a (cid:48) ∈ C a , n a (cid:48) ( s t ) ≥ t a,a (cid:48) .

Original: Notice that n a (cid:48) ( s t ) is the number of subpolicies that vote for action a (cid:48) at state s t .
Revised: Notice that n a (cid:48) ( s t ) is the number of subpolicies that vote for action a (cid:48) at state s t .

Original: Therefore, we can pick an arbitrary subset of those subpolicies whose cardinality is t a,a (cid:48) .
Revised: Therefore, we can pick an arbitrary subset of those subpolicies whose cardinality is t a,a (cid:48) .

Original: We denote B aa (cid:48) to such subset: B aa
Revised: We denote B aa (cid:48) to such subset: B aa

Original: [Equation]
Revised: [Equation]

Original: Now deﬁne B aα : B aα =
Revised: Now deﬁne B aα : B aα =

Original: (cid:83)
Revised: (cid:83)

Original: a (cid:48) ∈ C a B aa (cid:48) .
Revised: a (cid:48) ∈ C a B aa (cid:48) .

Original: We construct B aβ to be an arbitrary subset of those subpolicies whose prediction is not a and who are not in B aα , and limit the B aβ ’s cardinality:
Revised: We construct B aβ to be an arbitrary subset of those subpolicies whose prediction is not a and who are not in B aα , and limit the B aβ ’s cardinality:

Original: [Equation]
Revised: [Equation]

Original: Such B aβ can be selected, because:
Revised: Such B aβ can be selected, because:

Original: • From deﬁnition, B aα ⊆ { i ∈ [ u ] |
Revised: • From deﬁnition, B aα ⊆ { i ∈ [ u ] |

Original: π i ( s t ) (cid:54) = a } , where the cardinality of { i ∈ [ u ] |
Revised: π i ( s t ) (cid:54) = a } , where the cardinality of { i ∈ [ u ] |

Original: π i ( s t ) (cid:54) = a } is u − n a ( s t ) .
Revised: π i ( s t ) (cid:54) = a } is u − n a ( s t ) .

Original: So 0 ≤ u − n a ( s t ) − | B aα | .
Revised: So 0 ≤ u − n a ( s t ) − | B aα | .

Original: • Since | B aα | ≤ (cid:88)
Revised: • Since | B aα | ≤ (cid:88)

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: K − | B aα | ≥ 0 , and thus | B aβ | ≥ 0 .
Revised: K − | B aα | ≥ 0 , and thus | B aβ | ≥ 0 .

Original: Here ( ∗ ) is due to Equation (30).
Revised: Here ( ∗ ) is due to Equation (30).

Original: • The superset { i ∈ [ u ] |
Revised: • The superset { i ∈ [ u ] |

Original: π i ( s t ) (cid:54) = a } \ B aα has cardinality u − n a ( s t ) − | B aα | and | B aβ | ≤ u − n a ( s t ) − | B aα | .
Revised: π i ( s t ) (cid:54) = a } \ B aα has cardinality u − n a ( s t ) − | B aα | and | B aβ | ≤ u − n a ( s t ) − | B aα | .

Original: To this point, we can deﬁne the set of subpolicies to poison:
Revised: To this point, we can deﬁne the set of subpolicies to poison:

Original: [Equation]
Revised: [Equation]

Original: and | B a
Revised: and | B a

Original: [Equation]
Revised: [Equation]

Original: = | B aα
Revised: = | B aα

Original: [Equation]
Revised: [Equation]

Original: + | B aβ | ≤ K .
Revised: + | B aβ | ≤ K .

Original: In a similar fashion as the attack construction in proof of Proposition 9, for each i ∈ B a , we locate its corresponding partitioned dataset D i for training subpolicy
Revised: In a similar fashion as the attack construction in proof of Proposition 9, for each i ∈ B a , we locate its corresponding partitioned dataset D i for training subpolicy

Original: π ai .
Revised: π ai .

Original: We inset one trajectory p ai to D i such that our chosen learning algorithm M 0 can train a subpolicy ˜ π ai = M 

Revised: We inset one trajectory p ai to D i such that our chosen learning algorithm M 0 can train a subpolicy ˜ π ai = M

Original: ( D i ∪{ p ai } ) such that ˜ π ai ( s t ) = a .
Revised: ( D i ∪{ p ai } ) such that ˜ π ai ( s t ) = a .

Original: For example, the trajectory could be p ai = { ( s t , a, s (cid:48) , ∞ ) } where s (cid:48) is an arbitrary state that guarantees p ai is hashed to partition i ; and M 0 learns the action with maximum reward for the memorized nearest state.
Revised: For example, the trajectory could be p ai = { ( s t , a, s (cid:48) , ∞ ) } where s (cid:48) is an arbitrary state that guarantees p ai is hashed to partition i ; and M 0 learns the action with maximum reward for the memorized nearest state.

Original: Then, the poisoned dataset
Revised: Then, the poisoned dataset

Original: [Equation]
Revised: [Equation]

Original: Thus, (cid:101) D a is | D (cid:9) (cid:101) D a | = | B a | ≤ K , i.e. , the constructed attack’s poisoning size is within K .
Revised: Thus, (cid:101) D a is | D (cid:9) (cid:101) D a | = | B a | ≤ K , i.e. , the constructed attack’s poisoning size is within K .

Original: We now analyze the action prediction of the poisoned policy (cid:102) π P .
Revised: We now analyze the action prediction of the poisoned policy (cid:102) π P .

Original: For action a , after poisoning,
Revised: For action a , after poisoning,

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: ( s t )
Revised: ( s t )

Original: +min { K, u − n a ( s t ) }
Revised: +min { K, u − n a ( s t ) }

Original: = min { n a ( s t )+ K, u } .
Revised: = min { n a ( s t )+ K, u } .

Original: If (cid:101) n a ( s t )
Revised: If (cid:101) n a ( s t )

Original: =
Revised: =

Original: u , then all subpolicies vote for a , apparently (cid:102) π P a ( s t )
Revised: u , then all subpolicies vote for a , apparently (cid:102) π P a ( s t )

Original: =
Revised: =

Original: a ; otherwise, (cid:101) n a ( s t )
Revised: a ; otherwise, (cid:101) n a ( s t )

Original: = n a ( s t )
Revised: = n a ( s t )

Original: + K .
Revised: + K .

Original: In this case, for any action a (cid:48) ∈ C a , since we at least choose subpolicies in B aa (cid:48) and change their action prediction to a , the aggregated action count after poisoning is (cid:101) n a (cid:48) ( s t ) ≤ n a (cid:48) ( s t )
Revised: In this case, for any action a (cid:48) ∈ C a , since we at least choose subpolicies in B aa (cid:48) and change their action prediction to a , the aggregated action count after poisoning is (cid:101) n a (cid:48) ( s t ) ≤ n a (cid:48) ( s t )

Original: −| B aa (cid:48) | = n a (cid:48) ( s t ) − t a,a (cid:48)
Revised: −| B aa (cid:48) | = n a (cid:48) ( s t ) − t a,a (cid:48)

Original: =
Revised: =

Original: n a (cid:48) ( s t ) − n a (cid:48)
Revised: n a (cid:48) ( s t ) − n a (cid:48)

Original: ( s t )+ n a ( s t )
Revised: ( s t )+ n a ( s t )

Original: + K − 1 [ a (cid:48) <
Revised: + K − 1 [ a (cid:48) <

Original: a ] = n a ( s t )
Revised: a ] = n a ( s t )

Original: + K − 1 [ a (cid:48) <
Revised: + K − 1 [ a (cid:48) <

Original: a ] = ˜ n a ( s t ) − 1 [ a (cid:48) < a ] .
Revised: a ] = ˜ n a ( s t ) − 1 [ a (cid:48) < a ] .

Original: Thus, a (cid:48) has lower priority to be chosen than a .
Revised: Thus, a (cid:48) has lower priority to be chosen than a .

Original: For any action a (cid:48) / ∈ C a and a (cid:48) (cid:54) = a , by the deﬁnition of C a , t a,a (cid:48)
Revised: For any action a (cid:48) / ∈ C a and a (cid:48) (cid:54) = a , by the deﬁnition of C a , t a,a (cid:48)

Original: = n a (cid:48) ( s t ) − n a ( s t ) − K + 1 [ a (cid:48) < a ] ≤ 0 .
Revised: = n a (cid:48) ( s t ) − n a ( s t ) − K + 1 [ a (cid:48) < a ] ≤ 0 .

Original: After poisoning, the vote of a (cid:48) does not increase, i.e. , (cid:101) n a (cid:48) ( s t ) ≤ n a (cid:48) ( s t ) ≤ n a ( s t ) + K − 1 [ a (cid:48) < a ] = (cid:101) n a ( s t ) − 1 [ a (cid:48) < a ] .
Revised: After poisoning, the vote of a (cid:48) does not increase, i.e. , (cid:101) n a (cid:48) ( s t ) ≤ n a (cid:48) ( s t ) ≤ n a ( s t ) + K − 1 [ a (cid:48) < a ] = (cid:101) n a ( s t ) − 1 [ a (cid:48) < a ] .

Original: Thus, a (cid:48) also has lower priority to be chosen than a .
Revised: Thus, a (cid:48) also has lower priority to be chosen than a .

Original: In conclusion, we have (cid:102)
Revised: In conclusion, we have (cid:102)

Original: π P a ( s t )
Revised: π P a ( s t )

Original: = a .
Revised: = a .

Original: To this point, for any a ∈ A T ( K ) , we successfully construct the corresponding poisoned dataset (cid:101) D a within poisoning size K such that (cid:102)
Revised: To this point, for any a ∈ A T ( K ) , we successfully construct the corresponding poisoned dataset (cid:101) D a within poisoning size K such that (cid:102)

Original: π P a ( s t ) = a , thus concludes the proof.
Revised: π P a ( s t ) = a , thus concludes the proof.

Original: F. P OSSIBLE A CTION S ET IN TPARL
Revised: F. P OSSIBLE A CTION S ET IN TPARL

Original: Proof of Theorem 5.
Revised: Proof of Theorem 5.

Original: For ease of notation, we let w = min { W, t + 1 } , so w is the actual window size used at step t .
Revised: For ease of notation, we let w = min { W, t + 1 } , so w is the actual window size used at step t .

Original: We let t 

Revised: We let t

Original: = t − w + 1 , i.e. , t 0 is the actual start time step for TPARL aggregation at our current step t .
Revised: = t − w + 1 , i.e. , t 0 is the actual start time step for TPARL aggregation at our current step t .

Original: Now we can write chosen action at step t without poisoning as π T ( s t 0 : t ) .
Revised: Now we can write chosen action at step t without poisoning as π T ( s t 0 : t ) .

Original: We only need to prove the contrary: for any a ∈ A \ A ( K ) , within poisoning size K , the poisoned policy cannot choose a : (cid:102) π T ( s t 0 : t ) (cid:54) = a .
Revised: We only need to prove the contrary: for any a ∈ A \ A ( K ) , within poisoning size K , the poisoned policy cannot choose a : (cid:102) π T ( s t 0 : t ) (cid:54) = a .

Original: According to Equation (9), for such a , there exists a (cid:48) (cid:54) =
Revised: According to Equation (9), for such a , there exists a (cid:48) (cid:54) =

Original: a such that K
Revised: a such that K

Original: [Equation]
Revised: [Equation]

Original: Suppose there exists such poisoning attack that lets (cid:102) π T ( s t 0 : t )
Revised: Suppose there exists such poisoning attack that lets (cid:102) π T ( s t 0 : t )

Original: =
Revised: =

Original: a .
Revised: a .

Original: This implies that for a (cid:48) , after poisoning, we have
Revised: This implies that for a (cid:48) , after poisoning, we have

Original: [Equation]
Revised: [Equation]

Original: where (cid:101) n a is the aggregated action count after poisoning.
Revised: where (cid:101) n a is the aggregated action count after poisoning.

Original: Since the poisoning size is within K , it can affect at most K subpolicies.
Revised: Since the poisoning size is within K , it can affect at most K subpolicies.

Original: We let B ⊆
Revised: We let B ⊆

Original: [ u ] , | B | ≤ K to represent the affected subpolicy set
Revised: [ u ] , | B | ≤ K to represent the affected subpolicy set

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: j = 

Revised: j =

Original: i,a (cid:48) ( s t − j ) + w − w − 1 (cid:88)
Revised: i,a (cid:48) ( s t − j ) + w − w − 1 (cid:88)

Original: j = 1 i,a ( s t − j ) 
Revised: j = 1 i,a ( s t − j ) 

Original:  =
Revised:  =

Original: δ a (cid:48) ,a − (cid:88) i ∈
Revised: δ a (cid:48) ,a − (cid:88) i ∈

Original: B h i,a (cid:48) ,a ( a ) ≥ δ a (cid:48) ,a − | B | (cid:88) i
Revised: B h i,a (cid:48) ,a ( a ) ≥ δ a (cid:48) ,a − | B | (cid:88) i

Original: = h ( i ) a (cid:48) ,a ( b ) ≥
Revised: = h ( i ) a (cid:48) ,a ( b ) ≥

Original: δ a (cid:48) ,a −
Revised: δ a (cid:48) ,a −

Original: K (cid:88) i
Revised: K (cid:88) i

Original: = h ( i ) a (cid:48) ,a ( c ) ≥ 0 .
Revised: = h ( i ) a (cid:48) ,a ( c ) ≥ 0 .

Original: This contradicts with Equation (32), and thus the assumption is falsiﬁed, i.e. , there is no such poisoning attack that let (cid:102) π T ( s t 0 : t ) =
Revised: This contradicts with Equation (32), and thus the assumption is falsiﬁed, i.e. , there is no such poisoning attack that let (cid:102) π T ( s t 0 : t ) =

Original: a .
Revised: a .

Original: In the above equations, ( a ) is due to the fact that h ( i ) a (cid:48) ,a is a nonincreasing permutation of { h i,a (cid:48) ,a } u − 1 i =0 .
Revised: In the above equations, ( a ) is due to the fact that h ( i ) a (cid:48) ,a is a nonincreasing permutation of { h i,a (cid:48) ,a } u − 1 i =0 .

Original: ( b ) is due to the facts that | B | ≤ K and h ( i ) a (cid:48) ,a ≥ 0 .
Revised: ( b ) is due to the facts that | B | ≤ K and h ( i ) a (cid:48) ,a ≥ 0 .

Original: ( c ) comes from Equation (31).
Revised: ( c ) comes from Equation (31).

Original: F. H ARDNESS FOR C OMPUTING T IGHT P OSSIBLE A CTION S ET IN TPARL
Revised: F. H ARDNESS FOR C OMPUTING T IGHT P OSSIBLE A CTION S ET IN TPARL

Original: Proof of Theorem 6.
Revised: Proof of Theorem 6.

Original: By Deﬁnition 5, the possible action set with minimum cardinality (called minimal possible action set hereinafter) is unique.
Revised: By Deﬁnition 5, the possible action set with minimum cardinality (called minimal possible action set hereinafter) is unique.

Original: Otherwise, suppose A and B are both minimal possible action set, but A (cid:54)
Revised: Otherwise, suppose A and B are both minimal possible action set, but A (cid:54)

Original: =
Revised: =

Original: B , then A ∩ B is a smaller and valid possible action set.
Revised: B , then A ∩ B is a smaller and valid possible action set.

Original: Therefore, the oracle that returns the minimal possible action set, denoted by MINSET , can tell whether any action a ∈ MINSET and thus whether any action a can be chosen by some poisoned policy (cid:101) π whose poisoning size is within K .
Revised: Therefore, the oracle that returns the minimal possible action set, denoted by MINSET , can tell whether any action a ∈ MINSET and thus whether any action a can be chosen by some poisoned policy (cid:101) π whose poisoning size is within K .

Original: In other words, the problem of determining whether an action a can be chosen by some poisoned policy ˜ π whose poisoning size is within K , denoted by ATTKACT , is polynomially equivalent to MINSET : MINSET ≡ P ATTKACT .
Revised: In other words, the problem of determining whether an action a can be chosen by some poisoned policy ˜ π whose poisoning size is within K , denoted by ATTKACT , is polynomially equivalent to MINSET : MINSET ≡ P ATTKACT .

Original: Now, we show a polynomial reduction from the set cover problem to ATTKACT , which implies that our MINSET problem is an
Revised: Now, we show a polynomial reduction from the set cover problem to ATTKACT , which implies that our MINSET problem is an

Original: NP -complete problem.
Revised: NP -complete problem.

Original: The decision version of the set cover problem (Karp, 1972), denoted by SETCOVER , is a wellknown NP -complete problem and is deﬁned as follows.
Revised: The decision version of the set cover problem (Karp, 1972), denoted by SETCOVER , is a wellknown NP -complete problem and is deﬁned as follows.

Original: The inputs are
Revised: The inputs are

Original: a universal set of n elements: U = { u 1 , u 2 , · · · , u n } ; 2.
Revised: a universal set of n elements: U = { u 1 , u 2 , · · · , u n } ; 2.

Original: a set of subsets of U : V = { V 1 , · · · , V m } , V i ⊆
Revised: a set of subsets of U : V = { V 1 , · · · , V m } , V i ⊆

Original: U , 1 ≤ i ≤ m, (cid:83) mi
Revised: U , 1 ≤ i ≤ m, (cid:83) mi

Original: =1 V i
Revised: =1 V i

Original: =
Revised: =

Original: U ; 3.
Revised: U ; 3.

Original: a positive number K ∈ R + .
Revised: a positive number K ∈ R + .

Original: The output is a boolean variable b , indicating that whether there exists a subset W ⊆ V , |W| ≤ K , such that ∀ u i ∈ U , ∃ V ∈ W , u i ∈ V .
Revised: The output is a boolean variable b , indicating that whether there exists a subset W ⊆ V , |W| ≤ K , such that ∀ u i ∈ U , ∃ V ∈ W , u i ∈ V .

Original: Given an oracle to ATTKACT , we need to show SETCOVER can be solved in polynomial time, i.e. , SETCOVER ≤ P ATTKACT .
Revised: Given an oracle to ATTKACT , we need to show SETCOVER can be solved in polynomial time, i.e. , SETCOVER ≤ P ATTKACT .

Original: • If K ≥ n : We scan all sets V i ∈ V .
Revised: • If K ≥ n : We scan all sets V i ∈ V .

Original: To being with, we have a record set S ← ∅ , and an answer set W ← ∅ .
Revised: To being with, we have a record set S ← ∅ , and an answer set W ← ∅ .

Original: Whenever we encounter a set that contains a new element u i / ∈ S , we put W ← W ∪ { V j } , and record this element S ← S ∪ { u i } .
Revised: Whenever we encounter a set that contains a new element u i / ∈ S , we put W ← W ∪ { V j } , and record this element S ← S ∪ { u i } .

Original: After one scan pass, W covers all elements of U (since (cid:83) mj
Revised: After one scan pass, W covers all elements of U (since (cid:83) mj

Original: =1 V j = U ),
Revised: =1 V j = U ),

Original: and
Revised: and

Original: |W| ≤ n ≤ K .
Revised: |W| ≤ n ≤ K .

Original: Therefore, W is a valid set cover.
Revised: Therefore, W is a valid set cover.

Original: Since we can always ﬁnd such W , we can directly answer true .
Revised: Since we can always ﬁnd such W , we can directly answer true .

Original: • If K ≥ m : We can directly return V as a valid set cover, since (cid:83) mj
Revised: • If K ≥ m : We can directly return V as a valid set cover, since (cid:83) mj

Original: =1 V j = U and |V| ≤ m ≤ K .
Revised: =1 V j = U and |V| ≤ m ≤ K .

Original: Thus, we can answer true .
Revised: Thus, we can answer true .

Original: • If K < min { n, m } : This is the general case which we need to handle.
Revised: • If K < min { n, m } : This is the general case which we need to handle.

Original: Now we construct the ATTKACT ( K ) problem so that we can trigger its oracle to solve SETCOVER .
Revised: Now we construct the ATTKACT ( K ) problem so that we can trigger its oracle to solve SETCOVER .

Original: The poisoning size is K . 2.
Revised: The poisoning size is K . 2.

Original: The action space A
Revised: The action space A

Original: =
Revised: =

Original: U ∪ { b } ∪ Γ where Γ :=
Revised: U ∪ { b } ∪ Γ where Γ :=

Original: , # m 2 n } .
Revised: , # m 2 n } .

Original: ( |A| = n + 1 + m 2 n .)
Revised: ( |A| = n + 1 + m 2 n .)

Original: The sorting of actions is u 1 < u 2 < · · · < u n < b < # 1 < · · · < # m 2 n . 3.
Revised: The sorting of actions is u 1 < u 2 < · · · < u n < b < # 1 < · · · < # m 2 n . 3.

Original: The subpolicies are { π aj } mj =1 ∪ { π bi , π ci,j | 1 ≤ i ≤ n, 1 ≤ j ≤ K − 1 } , where π aj corresponds to V j ∈ V , and π bi , π ci,j correspond to u i ∈ U .
Revised: The subpolicies are { π aj } mj =1 ∪ { π bi , π ci,j | 1 ≤ i ≤ n, 1 ≤ j ≤ K − 1 } , where π aj corresponds to V j ∈ V , and π bi , π ci,j correspond to u i ∈ U .

Original: (Number of subpolicies u
Revised: (Number of subpolicies u

Original: = m + Kn ≤ m + n 2 .)
Revised: = m + Kn ≤ m + n 2 .)

Original: The current time step is t = nm , and the window size W = nm .
Revised: The current time step is t = nm , and the window size W = nm .

Original: The input action is b , i.e. , asking whether b can be chosen by some poisoned TPARL policy, i.e. , (cid:102) π T ( s 1: t )
Revised: The input action is b , i.e. , asking whether b can be chosen by some poisoned TPARL policy, i.e. , (cid:102) π T ( s 1: t )

Original: =
Revised: =

Original: b , if the poisoning size is within K .
Revised: b , if the poisoning size is within K .

Original: Now we construct the states at each step t ( 1 ≤ t ≤ nm ) so that the subpolicies’ action predictions at these steps are as follows.
Revised: Now we construct the states at each step t ( 1 ≤ t ≤ nm ) so that the subpolicies’ action predictions at these steps are as follows.

Original: Count the appearing time of each u i in V , and denote it by c i : c i = (cid:80) mj
Revised: Count the appearing time of each u i in V , and denote it by c i : c i = (cid:80) mj

Original: =1 1 [ u i ∈
Revised: =1 1 [ u i ∈

Original: V j ] .
Revised: V j ] .

Original: For each u i , select a V j 0 that contains u i , and in the corresponding π aj 0 , assign m
Revised: For each u i , select a V j 0 that contains u i , and in the corresponding π aj 0 , assign m

Original: − c
Revised: − c

Original: i + 1 steps to predict u i ; for all other V j that contains u i , in the corresponding π aj , assign one step to predict u i .
Revised: i + 1 steps to predict u i ; for all other V j that contains u i , in the corresponding π aj , assign one step to predict u i .

Original: After
Revised: After

Original: this process, each π aj at least has one time step whose action prediction is u i for each u i ∈
Revised: this process, each π aj at least has one time step whose action prediction is u i for each u i ∈

Original: V
Revised: V

Original: j
Revised: j

Original: Among all { π aj } mj =
Revised: Among all { π aj } mj =

Original: and
Revised: and

Original: all time steps 1 ≤ t
Revised: all time steps 1 ≤ t

Original: ≤
Revised: ≤

Original: nm
Revised: nm

Original: , mn step-action cells are ﬁlled, and the remaining ( m 2 n − mn ) cells are ﬁlled by # l ∈ Γ sequentially.
Revised: , mn step-action cells are ﬁlled, and the remaining ( m 2 n − mn ) cells are ﬁlled by # l ∈ Γ sequentially.

Original: For each π bi , arbitrarily select W − m time steps to assign action prediction as u i ; and ﬁll in other m time steps by remaining # l ∈ Γ sequentially.
Revised: For each π bi , arbitrarily select W − m time steps to assign action prediction as u i ; and ﬁll in other m time steps by remaining # l ∈ Γ sequentially.

Original: For each π ci,j , for all time steps, let the action prediction be u i .
Revised: For each π ci,j , for all time steps, let the action prediction be u i .

Original: As we can observe, the number of actions, the number of subpolicies, and the window size are all bounded by a polynomial of n and m .
Revised: As we can observe, the number of actions, the number of subpolicies, and the window size are all bounded by a polynomial of n and m .

Original: Therefore, such construction can be done in polynomial time.
Revised: Therefore, such construction can be done in polynomial time.

Original: We then show SETCOVER
Revised: We then show SETCOVER

Original: [Equation]
Revised: [Equation]

Original: Suppose the covering set is W ⊆ V , we denote K (cid:48) to |W| , and construct the ATTKACT problem with poisoning size K (cid:48) as described above.
Revised: Suppose the covering set is W ⊆ V , we denote K (cid:48) to |W| , and construct the ATTKACT problem with poisoning size K (cid:48) as described above.

Original: We can construct a poisoning strategy to let (cid:102) π T ( s 1: nm )
Revised: We can construct a poisoning strategy to let (cid:102) π T ( s 1: nm )

Original: =
Revised: =

Original: b , The poisoning strategy is to ﬁnd out π a j for each V j ∈ W , and to let them predict action b throughout all time steps: (cid:101) π aj ( s (cid:48) t )
Revised: b , The poisoning strategy is to ﬁnd out π a j for each V j ∈ W , and to let them predict action b throughout all time steps: (cid:101) π aj ( s (cid:48) t )

Original: = b, 1 ≤ t (cid:48) ≤ nm .
Revised: = b, 1 ≤ t (cid:48) ≤ nm .

Original: After poisoning, the aggregated action count (cid:101) n b ( s 1: nm ) = |W| × nm =
Revised: After poisoning, the aggregated action count (cid:101) n b ( s 1: nm ) = |W| × nm =

Original: K (cid:48) nm .
Revised: K (cid:48) nm .

Original: Since W covers every u i ∈ U , for each u i ∈ U there exists a set V j (cid:51) u i , whose corresponding (cid:101) π aj is poisoned to predict b .
Revised: Since W covers every u i ∈ U , for each u i ∈ U there exists a set V j (cid:51) u i , whose corresponding (cid:101) π aj is poisoned to predict b .

Original: Thus, (cid:101) n u i ( s 1: nm ) <
Revised: Thus, (cid:101) n u i ( s 1: nm ) <

Original: n u i ( s 1: nm ) = m
Revised: n u i ( s 1: nm ) = m

Original: + ( W − m )
Revised: + ( W − m )

Original: + W × ( K (cid:48) − 1) =
Revised: + W × ( K (cid:48) − 1) =

Original: WK (cid:48)
Revised: WK (cid:48)

Original: =
Revised: =

Original: K (cid:48) nm .
Revised: K (cid:48) nm .

Original: For any # l ∈ Γ , (cid:101) n # l ( s 1: nm ) ≤ n # l ( s 1: nm )
Revised: For any # l ∈ Γ , (cid:101) n # l ( s 1: nm ) ≤ n # l ( s 1: nm )

Original: = 1 .
Revised: = 1 .

Original: In summary, (cid:101) n u
Revised: In summary, (cid:101) n u

Original: i ( s 1: nm ) < K (cid:48) nm, (cid:101) n b ( s 1: nm ) =
Revised: i ( s 1: nm ) < K (cid:48) nm, (cid:101) n b ( s 1: nm ) =

Original: K (cid:48) nm, (cid:101) n # l ( s 1: nm ) = 1 .
Revised: K (cid:48) nm, (cid:101) n # l ( s 1: nm ) = 1 .

Original: Thus, after TPARL aggregation, the poisoned policy (cid:102) π T ( s 1: nm ) = b , and therefore b ∈
Revised: Thus, after TPARL aggregation, the poisoned policy (cid:102) π T ( s 1: nm ) = b , and therefore b ∈

Original: ATTKACT ( K (cid:48) ) .
Revised: ATTKACT ( K (cid:48) ) .

Original: – ⇐ = :
Revised: – ⇐ = :

Original: Suppose it is K (cid:48) that let b ∈ ATTKACT ( K (cid:48) ) , which implies that there exists such a poisoning attack within size K (cid:48) that misleads the poisoned policy to b : (cid:102) π T ( s 1: nm )
Revised: Suppose it is K (cid:48) that let b ∈ ATTKACT ( K (cid:48) ) , which implies that there exists such a poisoning attack within size K (cid:48) that misleads the poisoned policy to b : (cid:102) π T ( s 1: nm )

Original: =
Revised: =

Original: b .
Revised: b .

Original: Since the poisoning size is K (cid:48) , after poisoning the aggregated action count
Revised: Since the poisoning size is K (cid:48) , after poisoning the aggregated action count

Original: [Equation]
Revised: [Equation]

Original: For each u i ∈ U , since n u i ( s 1: nm ) =
Revised: For each u i ∈ U , since n u i ( s 1: nm ) =

Original: m
Revised: m

Original: + ( W − m )
Revised: + ( W − m )

Original: + W × ( K (cid:48) − 1) =
Revised: + W × ( K (cid:48) − 1) =

Original: K (cid:48) nm , we always have
Revised: K (cid:48) nm , we always have

Original: [Equation]
Revised: [Equation]

Original: where ( ∗ ) is due to the condition of successful attack.
Revised: where ( ∗ ) is due to the condition of successful attack.

Original: We denote the set of poisoned subpolicies by Π ( | Π | ≤ K (cid:48) ).
Revised: We denote the set of poisoned subpolicies by Π ( | Π | ≤ K (cid:48) ).

Original: Therefore, Equation (33) implies that for each u i ∈
Revised: Therefore, Equation (33) implies that for each u i ∈

Original: U , there exists at least one subpolicy
Revised: U , there exists at least one subpolicy

Original: [Equation]
Revised: [Equation]

Original: that is poisoned by the attack, otherwise the aggregated vote (cid:101) n u i cannot change.
Revised: that is poisoned by the attack, otherwise the aggregated vote (cid:101) n u i cannot change.

Original: We partition Γ by Γ a and Γ bc , where
Revised: We partition Γ by Γ a and Γ bc , where

Original: [Equation]
Revised: [Equation]

Original: We construct additional poisoning set Γ a + following this process:
Revised: We construct additional poisoning set Γ a + following this process:

Original: In the beginning, Γ a + ← ∅ .
Revised: In the beginning, Γ a + ← ∅ .

Original: For each u i ∈ U , if Γ a ∩ { π aj | u i ∈
Revised: For each u i ∈ U , if Γ a ∩ { π aj | u i ∈

Original: V j } is not empty, skip.
Revised: V j } is not empty, skip.

Original: Otherwise, according to
Revised: Otherwise, according to

Original: Equation (34), Γ b ∩ { π bi , π ci,j | 1 ≤ j ≤ K − 1 } is not empty.
Revised: Equation (34), Γ b ∩ { π bi , π ci,j | 1 ≤ j ≤ K − 1 } is not empty.

Original: In this case, we ﬁnd an arbitrary covering set of u i , namely V j 0 (cid:51) u
Revised: In this case, we ﬁnd an arbitrary covering set of u i , namely V j 0 (cid:51) u

Original: i , and put π aj 0 into Γ a + .
Revised: i , and put π aj 0 into Γ a + .

Original: When the process terminates, we ﬁnd that for each u i ∈ U ,
Revised: When the process terminates, we ﬁnd that for each u i ∈ U ,

Original: [Equation]
Revised: [Equation]

Original: Following the mapping { π aj } mj =1 ←→ { V j | 1 ≤ j ≤ m } =
Revised: Following the mapping { π aj } mj =1 ←→ { V j | 1 ≤ j ≤ m } =

Original: V , the subset (Γ a ∪ Γ a + ) ⊆
Revised: V , the subset (Γ a ∪ Γ a + ) ⊆

Original: { π aj } mj =1 can be mapped to ( W a
Revised: { π aj } mj =1 can be mapped to ( W a

Original: ∪W
Revised: ∪W

Original: a + ) ⊆ V .
Revised: a + ) ⊆ V .

Original: From Equation (35), ( W a ∪W a + ) is a valid set cover for U .
Revised: From Equation (35), ( W a ∪W a + ) is a valid set cover for U .

Original: We now study the cardinality of this set cover.
Revised: We now study the cardinality of this set cover.

Original: From the process, we know that every V j 0 ∈ W a
Revised: From the process, we know that every V j 0 ∈ W a

Original: + corresponds to a different set in Γ b .
Revised: + corresponds to a different set in Γ b .

Original: Thus, |W a + | ≤ | Γ b | , which implies that
Revised: Thus, |W a + | ≤ | Γ b | , which implies that

Original: [Equation]
Revised: [Equation]

Original: To this point, we successfully construct a set cover within cardinally K (cid:48) ≤ K that covers U , so SETCOVER = true .
Revised: To this point, we successfully construct a set cover within cardinally K (cid:48) ≤ K that covers U , so SETCOVER = true .

Original: Therefore, we can check whether SETCOVER = true by iterating K (cid:48) from 1 to K ( < min { n, m } iterations), constructing the ATTKACT ( K (cid:48) ) problem, and querying the oracle.
Revised: Therefore, we can check whether SETCOVER = true by iterating K (cid:48) from 1 to K ( < min { n, m } iterations), constructing the ATTKACT ( K (cid:48) ) problem, and querying the oracle.

Original: The whole process can be done in polynomial time assumping O (1) computation time of the
Revised: The whole process can be done in polynomial time assumping O (1) computation time of the

Original: ATTKACT ( K (cid:48) ) oracle.
Revised: ATTKACT ( K (cid:48) ) oracle.

Original: To this point, we have shown SETCOVER ≤ P ATTKACT .
Revised: To this point, we have shown SETCOVER ≤ P ATTKACT .

Original: On the other hand, an undeterminisitic
Revised: On the other hand, an undeterminisitic

Original: Turing machine can try different poisoning strategies by branching on whether to poison current subpolicy and what actions to be assigned to each poisoned subpolicy.
Revised: Turing machine can try different poisoning strategies by branching on whether to poison current subpolicy and what actions to be assigned to each poisoned subpolicy.

Original: The decision of whether the poisoning is successful can be done in polynomial time and b ∈ ATTKACT ( K ) corresponds to the existence of successfully attacked branches.
Revised: The decision of whether the poisoning is successful can be done in polynomial time and b ∈ ATTKACT ( K ) corresponds to the existence of successfully attacked branches.

Original: Thus, ATTKACT ∈ NP .
Revised: Thus, ATTKACT ∈ NP .

Original: Given that SETCOVER is an NP -complete problem, so does ATTKACT and MINSET (since MINSET ≡ P ATTKACT ).
Revised: Given that SETCOVER is an NP -complete problem, so does ATTKACT and MINSET (since MINSET ≡ P ATTKACT ).

Original: F. P OSSIBLE A CTION S ET IN DPARL
Revised: F. P OSSIBLE A CTION S ET IN DPARL

Original: Proof of Theorem 7.
Revised: Proof of Theorem 7.

Original: For ease of notation, we assume that W max ≤ t + 1 , and otherwise we let
Revised: For ease of notation, we assume that W max ≤ t + 1 , and otherwise we let

Original: W max ← t + 1 .
Revised: W max ← t + 1 .

Original: We let t 0 =
Revised: We let t 0 =

Original: max { t − W max + 1 , 0 } be the start time step of the maximum possible window.
Revised: max { t − W max + 1 , 0 } be the start time step of the maximum possible window.

Original: We only need to prove the contrary: for any a ∈ A \ A ( K ) , within poisoning size K , the poisoned policy cannot choose a : (cid:102) π D
Revised: We only need to prove the contrary: for any a ∈ A \ A ( K ) , within poisoning size K , the poisoned policy cannot choose a : (cid:102) π D

Original: ( s t 0 : t ) (cid:54) = a .
Revised: ( s t 0 : t ) (cid:54) = a .

Original: We prove by contradiction: we assume that there exists such a poisoning attack within poisoning size K that lets (cid:102) π D ( s t 0 : t )
Revised: We prove by contradiction: we assume that there exists such a poisoning attack within poisoning size K that lets (cid:102) π D ( s t 0 : t )

Original: = a .
Revised: = a .

Original: From the expression of
Revised: From the expression of

Original: A ( K ) (Equation (10)), a (cid:54) =
Revised: A ( K ) (Equation (10)), a (cid:54) =

Original: a t = π D ( s t 0 : t ) .
Revised: a t = π D ( s t 0 : t ) .

Original: Suppose the selected time window before the attack is
Revised: Suppose the selected time window before the attack is

Original: W (cid:48) (selected according to Equation (1) based on ∆ Wt and n a ), and the selected time window after the attack is (cid:102) W (cid:48) (selected according to Equation (1) based on (cid:101) ∆
Revised: W (cid:48) (selected according to Equation (1) based on ∆ Wt and n a ), and the selected time window after the attack is (cid:102) W (cid:48) (selected according to Equation (1) based on (cid:101) ∆

Original: Wt and (cid:101) n a ).
Revised: Wt and (cid:101) n a ).

Original: • If (cid:102) W (cid:48)
Revised: • If (cid:102) W (cid:48)

Original: =
Revised: =

Original: W (cid:48) :
Revised: W (cid:48) :

Original: Suppose we use the TPARL aggregation policy with window size W = W (cid:48) instead of current
Revised: Suppose we use the TPARL aggregation policy with window size W = W (cid:48) instead of current

Original: DPARL aggregation policy, then we will have π T ( s t − W (cid:48)
Revised: DPARL aggregation policy, then we will have π T ( s t − W (cid:48)

Original: +1: t ) =
Revised: +1: t ) =

Original: a t and (cid:102) π T ( s t − W (cid:48) +1: t ) = a .
Revised: a t and (cid:102) π T ( s t − W (cid:48) +1: t ) = a .

Original: Thus, according to the deﬁnition of possible action set (Deﬁnition 5), a ∈ A ( K ) where A ( K ) is deﬁned by Equation (9) in Theorem 5.
Revised: Thus, according to the deﬁnition of possible action set (Deﬁnition 5), a ∈ A ( K ) where A ( K ) is deﬁned by Equation (9) in Theorem 5.