Original: This implies that a ∈ A ( K ) where A ( K ) is deﬁned by
Revised: This implies that a ∈ A ( K ) where A ( K ) is deﬁned by

Original: Equation (10), which contradicts the assumption that a ∈ A \ A ( K ) .
Revised: Equation (10), which contradicts the assumption that a ∈ A \ A ( K ) .

Original: • If (cid:102) W (cid:48) (cid:54) =
Revised: • If (cid:102) W (cid:48) (cid:54) =

Original: W (cid:48) :
Revised: W (cid:48) :

Original: According to the deﬁnition in Equation (10),
Revised: According to the deﬁnition in Equation (10),

Original: [Equation]
Revised: [Equation]

Original: We deﬁne a # t = arg max a 0 (cid:54) =
Revised: We deﬁne a # t = arg max a 0 (cid:54) =

Original: a t (cid:101) n a 0 ( s t −
Revised: a t (cid:101) n a 0 ( s t −

Original: [Equation]
Revised: [Equation]

Original: 1: t ) .
Revised: 1: t ) .

Original: Then, the above equation implies that
Revised: Then, the above equation implies that

Original: [Equation]
Revised: [Equation]

Original: and thus
Revised: and thus

Original: [Equation]
Revised: [Equation]

Original: Since g ( i ) ≥ 0 by deﬁnition (Equation (7)), K
Revised: Since g ( i ) ≥ 0 by deﬁnition (Equation (7)), K

Original: [Equation]
Revised: [Equation]

Original: 36) where a # = arg max a 0 (cid:54)
Revised: 36) where a # = arg max a 0 (cid:54)

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: a 0 ∈A n a 0 ( s t − (cid:102) W (cid:48) +1: t ) and n wa is a shorthand of n a ( s t − w +1: t ) .
Revised: a 0 ∈A n a 0 ( s t − (cid:102) W (cid:48) +1: t ) and n wa is a shorthand of n a ( s t − w +1: t ) .

Original: Following the derivation from Equation (27) to ( a ) , we have K (cid:88) i =
Revised: Following the derivation from Equation (27) to ( a ) , we have K (cid:88) i =

Original: [Equation]
Revised: [Equation]

Original: Combined with Equation (36)
Revised: Combined with Equation (36)

Original: [Equation]
Revised: [Equation]

Original: On the other hand
Revised: On the other hand

Original: [Equation]
Revised: [Equation]

Original: the successful attack assumption,
Revised: the successful attack assumption,

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: e. , (cid:102) π D ( s t 0 : t ) = a and π D ( s t 0 : t ) =
Revised: e. , (cid:102) π D ( s t 0 : t ) = a and π D ( s t 0 : t ) =

Original: a t , implies that
Revised: a t , implies that

Original: [Equation]
Revised: [Equation]

Original: where “ > ” is “ ≥ ” if a < a t .
Revised: where “ > ” is “ ≥ ” if a < a t .

Original: In the above equation,
Revised: In the above equation,

Original: [Equation]
Revised: [Equation]

Original: Intuitively, after poisoning, a (2) is the runner-up action at window (cid:102) W (cid:48) , a W (cid:48) is the action at window
Revised: Intuitively, after poisoning, a (2) is the runner-up action at window (cid:102) W (cid:48) , a W (cid:48) is the action at window

Original: W (cid:48) , and a (2) W (cid:48) is the runner-up action at window W (cid:48) .
Revised: W (cid:48) , and a (2) W (cid:48) is the runner-up action at window W (cid:48) .

Original: We rewrite Equation (38) to
Revised: We rewrite Equation (38) to

Original: [Equation]
Revised: [Equation]

Original: We have the following two observations:
Revised: We have the following two observations:

Original: 1. (cid:101) n (cid:102) W (cid:48) a − (cid:101) n (cid:102) W (cid:48) a # ≥ (cid:101) n (cid:102) W (cid:48) a − (cid:101) n (cid:102) W (cid:48) a (2) , since a is the top action and a (2) is the runner-up action and their margin should be the smallest.
Revised: 1. (cid:101) n (cid:102) W (cid:48) a − (cid:101) n (cid:102) W (cid:48) a # ≥ (cid:101) n (cid:102) W (cid:48) a − (cid:101) n (cid:102) W (cid:48) a (2) , since a is the top action and a (2) is the runner-up action and their margin should be the smallest.

Original: (cid:101) n W (cid:48) a W (cid:48) − (cid:101) n W (cid:48) a (2) W (cid:48) ≥ (cid:101) n W (cid:48) a t − (cid:101) n W (cid:48) a # t , because 1) if a t = a W (cid:48) , LHS equals to RHS; 2) if a t (cid:54) =
Revised: (cid:101) n W (cid:48) a W (cid:48) − (cid:101) n W (cid:48) a (2) W (cid:48) ≥ (cid:101) n W (cid:48) a t − (cid:101) n W (cid:48) a # t , because 1) if a t = a W (cid:48) , LHS equals to RHS; 2) if a t (cid:54) =

Original: a W (cid:48)
Revised: a W (cid:48)

Original: [Equation]
Revised: [Equation]

Original: LHS ≥ 0 and RHS ≤ 0 .
Revised: LHS ≥ 0 and RHS ≤ 0 .

Original: Plugging these two observations to two sides of Equation (39), we get
Revised: Plugging these two observations to two sides of Equation (39), we get

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: This contradicts with Equation (37).
Revised: This contradicts with Equation (37).

Original: Since in both cases, we ﬁnd contradictions.
Revised: Since in both cases, we ﬁnd contradictions.

Original: Now we can conclude that for any a ∈ A\ A ( K ) , within poisoning size K , the poisoned policy cannot choose a : (cid:102) π D
Revised: Now we can conclude that for any a ∈ A\ A ( K ) , within poisoning size K , the poisoned policy cannot choose a : (cid:102) π D

Original: ( s t 0 : t ) (cid:54) = a .
Revised: ( s t 0 : t ) (cid:54) = a .

Original: G A DDITIONAL E XPERIMENTAL D ETAILS G. D ETAILS OF THE O FFLINE RL ALGORITHMS AND I MPLEMENTATIONS
Revised: G A DDITIONAL E XPERIMENTAL D ETAILS G. D ETAILS OF THE O FFLINE RL ALGORITHMS AND I MPLEMENTATIONS

Original: We experimented with three ofﬂine RL algorithms: DQN (Mnih et al., 2013), QR-DQN (Dabney et al., 2018), and C51 (Bellemare et al., 2017).
Revised: We experimented with three ofﬂine RL algorithms: DQN (Mnih et al., 2013), QR-DQN (Dabney et al., 2018), and C51 (Bellemare et al., 2017).

Original: The ﬁrst one is the standard baseline, while the latter two are distributional RL algorithms which show SOTA results in ofﬂine RL tasks.
Revised: The ﬁrst one is the standard baseline, while the latter two are distributional RL algorithms which show SOTA results in ofﬂine RL tasks.

Original: We ﬁrst brieﬂy introduce the algorithm ideas, followed by the implementation details.
Revised: We ﬁrst brieﬂy introduce the algorithm ideas, followed by the implementation details.

Original: Algorithm Ideas.
Revised: Algorithm Ideas.

Original: The core of Q-learning (Watkins & Dayan, 1992) is the Bellman optimality equation (Bellman, 1966)
Revised: The core of Q-learning (Watkins & Dayan, 1992) is the Bellman optimality equation (Bellman, 1966)

Original: [Equation]
Revised: [Equation]

Original: where a parameterized Q θ is adopted to approximate the optimal Q (cid:63) and iteratively improved.
Revised: where a parameterized Q θ is adopted to approximate the optimal Q (cid:63) and iteratively improved.

Original: In
Revised: In

Original: DQN (Mnih et al., 2013) speciﬁcally, the parameterization is achieved by using a convolutional neural network (LeCun et al., 1998).
Revised: DQN (Mnih et al., 2013) speciﬁcally, the parameterization is achieved by using a convolutional neural network (LeCun et al., 1998).

Original: In contrast to estimating the mean action value Q π ( s, a ) in
Revised: In contrast to estimating the mean action value Q π ( s, a ) in

Original: DQN, distributional RL algorithms estimate a density over the values of the state-action pairs.
Revised: DQN, distributional RL algorithms estimate a density over the values of the state-action pairs.

Original: The distributional Bellman optimality can be expressed as follows:
Revised: The distributional Bellman optimality can be expressed as follows:

Original: [Equation]
Revised: [Equation]

Original: )) where r ∼ R ( s, a )
Revised: )) where r ∼ R ( s, a )

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: ∼ P
Revised: ∼ P

Original: [Equation]
Revised: [Equation]

Original: · | s
Revised: · | s

Original: [Equation]
Revised: [Equation]

Original: a ) .
Revised: a ) .

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Concretely, QR-DQN (Dabney et al., 2018) approximates the density D (cid:63) with a uniform mixture of K Dirac delta functions, while C51 (Bellemare et al., 2017) approximates the density using a categorical distribution over a set of anchor points.
Revised: Concretely, QR-DQN (Dabney et al., 2018) approximates the density D (cid:63) with a uniform mixture of K Dirac delta functions, while C51 (Bellemare et al., 2017) approximates the density using a categorical distribution over a set of anchor points.

Original: Implementation Details.
Revised: Implementation Details.

Original: For training the subpolicies using ofﬂine RL training algorithms, we use the code base of Agarwal et al.
Revised: For training the subpolicies using ofﬂine RL training algorithms, we use the code base of Agarwal et al.

Original: The conﬁguration ﬁles containing the detailed hyperparameters can be found at their public repository https://github.com/google-research/ batch_rl/tree/master/batch_rl/fixed_replay/configs .
Revised: The conﬁguration ﬁles containing the detailed hyperparameters can be found at their public repository https://github.com/google-research/ batch_rl/tree/master/batch_rl/fixed_replay/configs .

Original: For the three methods
Revised: For the three methods

Original: DQN, QR-DQN, and C51, the names of the conﬁguration ﬁles are dqn.gin , quantile.gin , and c51.gin , respectively.
Revised: DQN, QR-DQN, and C51, the names of the conﬁguration ﬁles are dqn.gin , quantile.gin , and c51.gin , respectively.

Original: On each partition, we train the subpolicy for 50 epochs.
Revised: On each partition, we train the subpolicy for 50 epochs.

Original: G. C ONCRETE E XPERIMENTAL P ROCEDURES
Revised: G. C ONCRETE E XPERIMENTAL P ROCEDURES

Original: We conduct experiments on two Atari 2600 games, Freeway and Breakout from OpenAI
Revised: We conduct experiments on two Atari 2600 games, Freeway and Breakout from OpenAI

Original: Gym (Brockman et al., 2016), and one autonomous driving environment
Revised: Gym (Brockman et al., 2016), and one autonomous driving environment

Original: Highway (Leurent, 2018), following Section 4.
Revised: Highway (Leurent, 2018), following Section 4.

Original: Concretely, in the training stage, we ﬁrst partition the training dataset into u partitions ( u = 30 , 50 , or 100 ) by using the hash function h ( τ ) pre-deﬁned in each environment.
Revised: Concretely, in the training stage, we ﬁrst partition the training dataset into u partitions ( u = 30 , 50 , or 100 ) by using the hash function h ( τ ) pre-deﬁned in each environment.

Original: Let s i be the i -th value in the state representation and d be the dimensionality of the state.
Revised: Let s i be the i -th value in the state representation and d be the dimensionality of the state.

Original: In Atari games where the state is the game frame, h ( τ ) is deﬁned as the sum of all pixel values of all frames in the trajectory τ , i.e. , h ( τ ) =
Revised: In Atari games where the state is the game frame, h ( τ ) is deﬁned as the sum of all pixel values of all frames in the trajectory τ , i.e. , h ( τ ) =

Original: (cid:80) s
Revised: (cid:80) s

Original: ∈ τ (cid:80) i ∈ [ d ] s i .
Revised: ∈ τ (cid:80) i ∈ [ d ] s i .

Original: In Highway where the state is a ﬂoating point number scalar containing the positions and velocities of all vehicles, we deﬁne h ( τ ) =
Revised: In Highway where the state is a ﬂoating point number scalar containing the positions and velocities of all vehicles, we deﬁne h ( τ ) =

Original: (cid:80) s
Revised: (cid:80) s

Original: ∈ τ (cid:80) i ∈ [ d ] f ( s i ) where f : R → Z is a deterministic function that maps the given ﬂoat value to integer space.
Revised: ∈ τ (cid:80) i ∈ [ d ] f ( s i ) where f : R → Z is a deterministic function that maps the given ﬂoat value to integer space.

Original: Concretely, we take f ( x ) as the the sum of the higher 16 bits and the lower 16 bits of its 32 -bit representation under IEEE 754 standard (group of the Microprocessor Standards Subcommittee & Institute, 1985).
Revised: Concretely, we take f ( x ) as the the sum of the higher 16 bits and the lower 16 bits of its 32 -bit representation under IEEE 754 standard (group of the Microprocessor Standards Subcommittee & Institute, 1985).

Original: We then train the subpolicies on the partitions with ofﬂine RL training algorithm M 0 ∈ { DQN (Mnih et al., 2013), QR-DQN (Dabney et al., 2018), C51 (Bellemare et al., 2017) } .
Revised: We then train the subpolicies on the partitions with ofﬂine RL training algorithm M 0 ∈ { DQN (Mnih et al., 2013), QR-DQN (Dabney et al., 2018), C51 (Bellemare et al., 2017) } .

Original: The detailed description of the algorithms can be found in Appendix G.1.
Revised: The detailed description of the algorithms can be found in Appendix G.1.

Original: In the aggregation stage, we apply the three proposed aggregation protocols (PARL (Theorem 8),
Revised: In the aggregation stage, we apply the three proposed aggregation protocols (PARL (Theorem 8),

Original: TPARL (Theorem 1), and DPARL (Theorem 3)) on the trained u subpolicies and derive the aggregated policies π P , π T , and π D accordingly.
Revised: TPARL (Theorem 1), and DPARL (Theorem 3)) on the trained u subpolicies and derive the aggregated policies π P , π T , and π D accordingly.

Original: Finally, in the certiﬁcation stage, for each aggregated policy, we provide the per-state action and cumulative reward certiﬁcation following our theorems.
Revised: Finally, in the certiﬁcation stage, for each aggregated policy, we provide the per-state action and cumulative reward certiﬁcation following our theorems.

Original: When certifying the per-state action , we constrain the maximum trajectory length H = 1000 for Atari games and H = 30 for Highway (which is the full length in Highway) and report results averaged over 20 runs; for the cumulative reward certiﬁcation, we adopt the trajectory length H = 400 for evaluating Freeway, H = 75 for
Revised: When certifying the per-state action , we constrain the maximum trajectory length H = 1000 for Atari games and H = 30 for Highway (which is the full length in Highway) and report results averaged over 20 runs; for the cumulative reward certiﬁcation, we adopt the trajectory length H = 400 for evaluating Freeway, H = 75 for

Original: Breakout, and H = 30 for Highway.
Revised: Breakout, and H = 30 for Highway.

Original: Conﬁguration of Trajectory Length H .
Revised: Conﬁguration of Trajectory Length H .

Original: For Atari games , we do not evaluate the full episode length (up to tens of thousands steps), since our goal is to compare the relative certiﬁed robustness of different RL algorithms, and the evaluation on relatively short trajectory is sufﬁcient under affordable computation cost.
Revised: For Atari games , we do not evaluate the full episode length (up to tens of thousands steps), since our goal is to compare the relative certiﬁed robustness of different RL algorithms, and the evaluation on relatively short trajectory is sufﬁcient under affordable computation cost.

Original: Moreover, different episodes in Atari games are oftentimes of different lengths; thus it is necessary that we restrict the episode length to enable a fair comparison.
Revised: Moreover, different episodes in Atari games are oftentimes of different lengths; thus it is necessary that we restrict the episode length to enable a fair comparison.

Original: For
Revised: For

Original: Highway , we evaluate on the full episode ( H = 30 ) where we can efﬁciently achieve effective comparisons.
Revised: Highway , we evaluate on the full episode ( H = 30 ) where we can efﬁciently achieve effective comparisons.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: H A DDITIONAL E VALUATION R ESULTS AND D ISCUSSIONS H. B ENIGN E MPIRICAL P ERFORMANCE
Revised: H A DDITIONAL E VALUATION R ESULTS AND D ISCUSSIONS H. B ENIGN E MPIRICAL P ERFORMANCE

Original: We present the benign empirical cumulative rewards of the three aggregation protocols (PARL,
Revised: We present the benign empirical cumulative rewards of the three aggregation protocols (PARL,

Original: TPARL, and DPARL) applied on subpolicies trained using three ofﬂine RL algorithms (DQN, QRDQN, and C51) in Table 4.
Revised: TPARL, and DPARL) applied on subpolicies trained using three ofﬂine RL algorithms (DQN, QRDQN, and C51) in Table 4.

Original: The cumulative reward is obtained by running a trajectory of maximum length H and accumulating the reward achieved at each time step.
Revised: The cumulative reward is obtained by running a trajectory of maximum length H and accumulating the reward achieved at each time step.

Original: We discuss the conclusions on multiple levels.
Revised: We discuss the conclusions on multiple levels.

Original: We choose H = 1000 for Atari games and H =for Highway environment and report results averaged over 20 runs.
Revised: We choose H = 1000 for Atari games and H =for Highway environment and report results averaged over 20 runs.

Original: On the RL algorithm level, we see that QR-DQN achieves the highest score in most cases.
Revised: On the RL algorithm level, we see that QR-DQN achieves the highest score in most cases.

Original: On the aggregation protocol level, temporal aggregation enhances the performance on Freeway while dampens the performance on
Revised: On the aggregation protocol level, temporal aggregation enhances the performance on Freeway while dampens the performance on

Original: Breakout and Highway.
Revised: Breakout and Highway.

Original: We particularly note that the results obtained by using temporal aggregation (TPARL and DPARL) gives signiﬁcantly smaller variance compared to the single-step aggregation
Revised: We particularly note that the results obtained by using temporal aggregation (TPARL and DPARL) gives signiﬁcantly smaller variance compared to the single-step aggregation

Original: PARL, especially in Breakout.
Revised: PARL, especially in Breakout.

Original: This implies that temporal aggregation can help stabilize the policy.
Revised: This implies that temporal aggregation can help stabilize the policy.

Original: However, the conclusion does not hold in Highway, which is an interesting phenomenon that is worth investigating.
Revised: However, the conclusion does not hold in Highway, which is an interesting phenomenon that is worth investigating.

Original: On the partition number level, a larger partition number can give slightly better results.
Revised: On the partition number level, a larger partition number can give slightly better results.

Original: On the environment level, Freeway is simpler and more stable than Breakout and
Revised: On the environment level, Freeway is simpler and more stable than Breakout and

Original: Highway.
Revised: Highway.

Original: H. C OMPARISON OF COPA WITH S TANDARD T RAINING
Revised: H. C OMPARISON OF COPA WITH S TANDARD T RAINING

Original: We provide additional experimental results on the comparison between the empirical performance of our COPA and the standard training in terms of the convergence speed (Figure 3) and policy quality (Table 5).
Revised: We provide additional experimental results on the comparison between the empirical performance of our COPA and the standard training in terms of the convergence speed (Figure 3) and policy quality (Table 5).

Original: In Figure 3, we aim to show the comparison of the convergence speed.
Revised: In Figure 3, we aim to show the comparison of the convergence speed.

Original: For our proposed training , we plot all training curves of the sampled 5 subpolicies in blue, where each subpolicy is trained on one partition of the dataset.
Revised: For our proposed training , we plot all training curves of the sampled 5 subpolicies in blue, where each subpolicy is trained on one partition of the dataset.

Original: (We do not plot all 50 curves for visual clarity.)
Revised: (We do not plot all 50 curves for visual clarity.)

Original: For standard training , we plot the training curve of the standard policy in red, where the single policy is trained on the entire dataset.
Revised: For standard training , we plot the training curve of the standard policy in red, where the single policy is trained on the entire dataset.

Original: We see that on Freeway , most subpolicies converge slower than the standard policy, but will reach similar convergence value as the standard training one; while there also exist a few subpolicies that fail to be trained.
Revised: We see that on Freeway , most subpolicies converge slower than the standard policy, but will reach similar convergence value as the standard training one; while there also exist a few subpolicies that fail to be trained.

Original: On Breakout , within 50 epochs, we observe substantial ﬂuctuation for the training curves of all the policies, as well as large variance for the achieved reward at the last
Revised: On Breakout , within 50 epochs, we observe substantial ﬂuctuation for the training curves of all the policies, as well as large variance for the achieved reward at the last

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: epoch.
Revised: epoch.

Original: Thus, on these two Atari games , we cannot draw conclusions w.r.t.
Revised: Thus, on these two Atari games , we cannot draw conclusions w.r.t.

Original: the convergence.
Revised: the convergence.

Original: Given more computational resources to run more epochs, we would be able to draw more informative conclusions.
Revised: Given more computational resources to run more epochs, we would be able to draw more informative conclusions.

Original: In contrast, on Highway , all subpolicies display similarly good convergence properties, showing comparable convergence speed and value with the standard training on the entire dataset.
Revised: In contrast, on Highway , all subpolicies display similarly good convergence properties, showing comparable convergence speed and value with the standard training on the entire dataset.

Original: In Table
Revised: In Table

Original: [Table]
Revised: [Table]

Original: we aim to compare the policy quality of the aggregated policy derived in our COPA framework ( i
Revised: we aim to compare the policy quality of the aggregated policy derived in our COPA framework ( i

Original: [Table]
Revised: [Table]

Original: e. , PARL, TPARL
Revised: e. , PARL, TPARL

Original: [Table]
Revised: [Table]

Original: and DPARL) with the policy obtained from standard training.
Revised: and DPARL) with the policy obtained from standard training.

Original: We see that on Freeway , our three protocols achieve comparable results with the policy obtained by standard training on the entire dataset; on Breakout , although the quality of our obtained policies is lower, our policies are much more stable with signiﬁcantly lower variance than the standard training; on Highway , only PARL obtains comparable results to the standard training policy, indicating the lack of temporal continuity in this environment
Revised: We see that on Freeway , our three protocols achieve comparable results with the policy obtained by standard training on the entire dataset; on Breakout , although the quality of our obtained policies is lower, our policies are much more stable with signiﬁcantly lower variance than the standard training; on Highway , only PARL obtains comparable results to the standard training policy, indicating the lack of temporal continuity in this environment

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: H. M ORE A NALYTICAL S TATISTICS FOR DPARL
Revised: H. M ORE A NALYTICAL S TATISTICS FOR DPARL

Original: Selected Window Size.
Revised: Selected Window Size.

Original: We provide the mean and variance for the selected window sizes in
Revised: We provide the mean and variance for the selected window sizes in

Original: DPARL in Table 6.
Revised: DPARL in Table 6.

Original: In our experiments, the maximum window size W max = 5 , while the average selected window sizes are all around half of the maximum value.
Revised: In our experiments, the maximum window size W max = 5 , while the average selected window sizes are all around half of the maximum value.

Original: Thus, the average certiﬁcation time is expected to be much smaller compared with the worst-case time complexity.
Revised: Thus, the average certiﬁcation time is expected to be much smaller compared with the worst-case time complexity.

Original: Running Time.
Revised: Running Time.

Original: We provide the running time in Table 7.
Revised: We provide the running time in Table 7.

Original: Speciﬁcally, we compare the running time of DPARL ( W max = 5 ) applied on u = 30 or 50 subpolicies trained using ofﬂine RL algorithm
Revised: Speciﬁcally, we compare the running time of DPARL ( W max = 5 ) applied on u = 30 or 50 subpolicies trained using ofﬂine RL algorithm

Original: DQN with the normal testing which tests the runtime of a single trained DQN policy without using our framework.
Revised: DQN with the normal testing which tests the runtime of a single trained DQN policy without using our framework.

Original: As we have shown in the remark of Theorem 3 in Section 4.3, the time complexity of
Revised: As we have shown in the remark of Theorem 3 in Section 4.3, the time complexity of

Original: DPARL is O (cid:0) W 2max |A| 2 u + W max |A| 2 u log u (cid:1) .
Revised: DPARL is O (cid:0) W 2max |A| 2 u + W max |A| 2 u log u (cid:1) .

Original: In Table 7, we also see that the runtime of DPARL scales roughly quasilinearly with the number of subpolicies u , and quadratically with the action set size |A| , which is 3 for Freeway and 4 for Breakout.
Revised: In Table 7, we also see that the runtime of DPARL scales roughly quasilinearly with the number of subpolicies u , and quadratically with the action set size |A| , which is 3 for Freeway and 4 for Breakout.

Original: We omit the runtime for Highway, since the horizon length, the environment type, and the neural network size are all different for Highway and
Revised: We omit the runtime for Highway, since the horizon length, the environment type, and the neural network size are all different for Highway and

Original: Atari games.
Revised: Atari games.

Original: H. M AXIMUM T OLERABLE P OISONING T HRESHOLD VS . T OTAL T RAJECTORY N UMBER
Revised: H. M AXIMUM T OLERABLE P OISONING T HRESHOLD VS . T OTAL T RAJECTORY N UMBER

Original: We provide the total number of trajectories in the ofﬂine dataset used for training the environments
Revised: We provide the total number of trajectories in the ofﬂine dataset used for training the environments

Original: Freeway, Breakout and Highway below, as well as the corresponding ratio of the maximum tolerable poisoning threshold w.r.t.
Revised: Freeway, Breakout and Highway below, as well as the corresponding ratio of the maximum tolerable poisoning threshold w.r.t.

Original: the total number of trajectories.
Revised: the total number of trajectories.

Original: For Freeway , the total number of trajectories in the entire ofﬂine dataset is 121,892, and the ratio ( max K t / # total trajectories) is 0.0002; for Breakout , the number is 209,049, and the ratio is 0.0001.
Revised: For Freeway , the total number of trajectories in the entire ofﬂine dataset is 121,892, and the ratio ( max K t / # total trajectories) is 0.0002; for Breakout , the number is 209,049, and the ratio is 0.0001.

Original: We emphasize that, as shown in previous work on probable defense against poisoning in supervised learning (Levine & Feizi, 2020), the number of instances they can certify is also not high especially for challenging tasks ( e.g. , certifying 20 instances on the dataset GTSRB in Levine & Feizi (2020), attaining the ratio 0.0005), which may imply the intrinsic difﬁculty of certifying against poisoning attacks.
Revised: We emphasize that, as shown in previous work on probable defense against poisoning in supervised learning (Levine & Feizi, 2020), the number of instances they can certify is also not high especially for challenging tasks ( e.g. , certifying 20 instances on the dataset GTSRB in Levine & Feizi (2020), attaining the ratio 0.0005), which may imply the intrinsic difﬁculty of certifying against poisoning attacks.

Original: Given such difﬁculty, we already achieve reasonable certiﬁcation as the ﬁrst work on certiﬁed robust RL against poisoning, and we hope future works can further improve upon our results.
Revised: Given such difﬁculty, we already achieve reasonable certiﬁcation as the ﬁrst work on certiﬁed robust RL against poisoning, and we hope future works can further improve upon our results.

Original: H. C OMPARISON OF F REEWAY , B REAKOUT , AND H IGHWAY
Revised: H. C OMPARISON OF F REEWAY , B REAKOUT , AND H IGHWAY

Original: Freeway and Breakout are two typical types of games with distinctive game properties.
Revised: Freeway and Breakout are two typical types of games with distinctive game properties.

Original: Freeway is a simple game where the agent aims to cross the road and avoid the trafﬁc.
Revised: Freeway is a simple game where the agent aims to cross the road and avoid the trafﬁc.

Original: In most of the time steps, the agent would take the “forward” action; only when there is a need to avoid the trafﬁc will the agent stop and wait.
Revised: In most of the time steps, the agent would take the “forward” action; only when there is a need to avoid the trafﬁc will the agent stop and wait.

Original: In comparison, the game Breakout involves more complicated interactions between the paddle and the environment.
Revised: In comparison, the game Breakout involves more complicated interactions between the paddle and the environment.

Original: The paddle position and velocity both play important roles in order to catch and bounce the ball at an appropriate angle, which requires a frequent switch of the paddle moving direction.
Revised: The paddle position and velocity both play important roles in order to catch and bounce the ball at an appropriate angle, which requires a frequent switch of the paddle moving direction.

Original: Similar to Breakout, as an autonomous driving environment, Highway requires the agent to accurately analyze the rapidly changing environment around it and react quickly, leading to frequent changes in vehicle’s actions.
Revised: Similar to Breakout, as an autonomous driving environment, Highway requires the agent to accurately analyze the rapidly changing environment around it and react quickly, leading to frequent changes in vehicle’s actions.

Original: Given the properties of the environments, we note that in Freeway, adjacent time steps may share consistent action selections, which is not necessarily true in the Breakout and Highway.
Revised: Given the properties of the environments, we note that in Freeway, adjacent time steps may share consistent action selections, which is not necessarily true in the Breakout and Highway.

Original: Thus, it would be expected to be beneﬁcial to consider the past history for making the current decision in
Revised: Thus, it would be expected to be beneﬁcial to consider the past history for making the current decision in

Original: Freeway, while the past history may interfere with the action selection in Breakout and Highway.
Revised: Freeway, while the past history may interfere with the action selection in Breakout and Highway.

Original: [Table]
Revised: [Table]

Original: Comparisons of Bottleneck states in Atari Games.
Revised: Comparisons of Bottleneck states in Atari Games.

Original: In Breakout, the game goal is to control the paddle so that the ball is bounced to hit the brick.
Revised: In Breakout, the game goal is to control the paddle so that the ball is bounced to hit the brick.

Original: There are clearly very different stages in the Breakout game, e.g. , when the ball is ﬂying towards the paddle and when it is bounced back.
Revised: There are clearly very different stages in the Breakout game, e.g. , when the ball is ﬂying towards the paddle and when it is bounced back.

Original: An example of the bottleneck state is when the ball is approaching the paddle.
Revised: An example of the bottleneck state is when the ball is approaching the paddle.

Original: The poisoning behavior may lead to the disastrous effect that the policy learns to control the paddle to slide in the opposite direction of the ball at such bottleneck states, while other states may not be as vulnerable.
Revised: The poisoning behavior may lead to the disastrous effect that the policy learns to control the paddle to slide in the opposite direction of the ball at such bottleneck states, while other states may not be as vulnerable.

Original: In comparison, in Freeway, the game goal is to control the agent to cross the road while avoiding the trafﬁc.
Revised: In comparison, in Freeway, the game goal is to control the agent to cross the road while avoiding the trafﬁc.

Original: The agent is faced with similar trafﬁc conditions everywhere on the road, and can make stable choices regardless of the complex situation.
Revised: The agent is faced with similar trafﬁc conditions everywhere on the road, and can make stable choices regardless of the complex situation.

Original: Thus there are very few bottleneck states .
Revised: Thus there are very few bottleneck states .

Original: A Study on the Frequency of Action Change in Highway.
Revised: A Study on the Frequency of Action Change in Highway.

Original: In Table 8, we present the action change ratio (# action changes / trajectory length) in Highway environment.
Revised: In Table 8, we present the action change ratio (# action changes / trajectory length) in Highway environment.

Original: Comparing among the aggregation protocols, we ﬁrst note that the single-step aggregation policy π P frequently alters actions as the reaction to the rapidly changing driving environment, while the temporal aggregation protocol π T changes actions less frequently.
Revised: Comparing among the aggregation protocols, we ﬁrst note that the single-step aggregation policy π P frequently alters actions as the reaction to the rapidly changing driving environment, while the temporal aggregation protocol π T changes actions less frequently.

Original: This is because in TPARL, the agent is explicitly forced to take stable actions based on a ﬁxed window size.
Revised: This is because in TPARL, the agent is explicitly forced to take stable actions based on a ﬁxed window size.

Original: Second, comparing the action change ratio with the benign empirical performance in Table 4, we observe that π P and π T achieves the highest and lowest empirical performance respectively, which indicates the correlation between action change ratio with the achieved empirical reward.
Revised: Second, comparing the action change ratio with the benign empirical performance in Table 4, we observe that π P and π T achieves the highest and lowest empirical performance respectively, which indicates the correlation between action change ratio with the achieved empirical reward.

Original: Speciﬁcally, in TPARL, the action change is limited as a result of the enforced window size, leading to the limited benign performance.
Revised: Speciﬁcally, in TPARL, the action change is limited as a result of the enforced window size, leading to the limited benign performance.

Original: H. F ULL R ESULTS OF R OBUSTNESS C ERTIFICATION FOR P ER -S TATE A CTION S TABILITY
Revised: H. F ULL R ESULTS OF R OBUSTNESS C ERTIFICATION FOR P ER -S TATE A CTION S TABILITY

Original: As a complete set of results for per-state action certiﬁcation apart from Figure 1 in Section 5.1, we present the entire cumulative histogram of tolerable poisoning thresholds in Figure 4 and Figure 5, together with the average tolerable poisoning thresholds in Table 9.
Revised: As a complete set of results for per-state action certiﬁcation apart from Figure 1 in Section 5.1, we present the entire cumulative histogram of tolerable poisoning thresholds in Figure 4 and Figure 5, together with the average tolerable poisoning thresholds in Table 9.

Original: The deﬁnitions of the two metrcis can be found in Section 5.1.
Revised: The deﬁnitions of the two metrcis can be found in Section 5.1.

Original: Basically, the conclusions in terms of the comparisons on the RL algorithm level, the aggregation protocol level, the partition number level, and the game level are all similar to that derived in Section 5.1.
Revised: Basically, the conclusions in terms of the comparisons on the RL algorithm level, the aggregation protocol level, the partition number level, and the game level are all similar to that derived in Section 5.1.

Original: H. F ULL R ESULTS OF R OBUSTNESS C ERTIFICATION FOR C UMULATIVE R EWARD B OUND
Revised: H. F ULL R ESULTS OF R OBUSTNESS C ERTIFICATION FOR C UMULATIVE R EWARD B OUND

Original: In addition to the evaluation results provided in Figure 2 in section 5.2, we provide a more comprehensive set of evaluation results with more settings of trajectory length H here in Figure 6.
Revised: In addition to the evaluation results provided in Figure 2 in section 5.2, we provide a more comprehensive set of evaluation results with more settings of trajectory length H here in Figure 6.

Original: We draw similar conclusions as discussed in Section 5.2.
Revised: We draw similar conclusions as discussed in Section 5.2.

Original: Speciﬁcally, in Highway, C51 achieves higher certiﬁed lower bounds than other two when the poisoning size is large, which can be explained by the larger portion of states that can tolerate large poisoning sizes as shown in Figure 5.
Revised: Speciﬁcally, in Highway, C51 achieves higher certiﬁed lower bounds than other two when the poisoning size is large, which can be explained by the larger portion of states that can tolerate large poisoning sizes as shown in Figure 5.

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: We additionally point out that the value J K achieved at poisoning size K = 0 ( e.g. , J K =for Freeway, 2 for Breakout, and 28 .
Revised: We additionally point out that the value J K achieved at poisoning size K = 0 ( e.g. , J K =for Freeway, 2 for Breakout, and 28 .

Original: 31 for Highway under π P over u = 50 DQN subpolicies) corresponds to the case where there is no poisoning at all on the training set.
Revised: 31 for Highway under π P over u = 50 DQN subpolicies) corresponds to the case where there is no poisoning at all on the training set.

Original: Our successive bounds under larger K are non-vacuous compared to this value.
Revised: Our successive bounds under larger K are non-vacuous compared to this value.

Original: I A B ROADER D ISCUSSION ON R ELATED W ORK I. P OISONING A TTACKS IN RL
Revised: I A B ROADER D ISCUSSION ON R ELATED W ORK I. P OISONING A TTACKS IN RL

Original: Below, we provide a brief discussion on the related works on policy poisoning and reward poisoning (Ma et al., 2019; Sun et al., 2021; Huang & Zhu, 2019).
Revised: Below, we provide a brief discussion on the related works on policy poisoning and reward poisoning (Ma et al., 2019; Sun et al., 2021; Huang & Zhu, 2019).

Original: Ma et al.
Revised: Ma et al.

Original: and Huang & Zhu (2019) study reward poisoning where the attacker can modify the rewards in an ofﬂine dataset or the reward signals during the online interaction.
Revised: and Huang & Zhu (2019) study reward poisoning where the attacker can modify the rewards in an ofﬂine dataset or the reward signals during the online interaction.

Original: The attacker’s goal is to force learning a particular target policy, or minimize the agent’s reward in the original task.
Revised: The attacker’s goal is to force learning a particular target policy, or minimize the agent’s reward in the original task.

Original: Under this framework, Ma et al.
Revised: Under this framework, Ma et al.

Original: (2019) considers two victim learners with speciﬁc assumptions on the learner structure and develops attacks for them, while Huang & Zhu (2019) provides a theoretical analysis on the conditions for successful attacks against a Q-learning agent.
Revised: (2019) considers two victim learners with speciﬁc assumptions on the learner structure and develops attacks for them, while Huang & Zhu (2019) provides a theoretical analysis on the conditions for successful attacks against a Q-learning agent.

Original: In comparison to only reward poisoning in Ma et al.
Revised: In comparison to only reward poisoning in Ma et al.

Original: and
Revised: and

Original: Huang et al.
Revised: Huang et al.

Original: (2017), Sun et al.
Revised: (2017), Sun et al.

Original: (2021) focuses on general policy poisoning attacks for policy gradient learners in the online RL setting by using the vulnerability-awareness metric to decide when to poison, and the adversarial critic to guide the poisoning.
Revised: (2021) focuses on general policy poisoning attacks for policy gradient learners in the online RL setting by using the vulnerability-awareness metric to decide when to poison, and the adversarial critic to guide the poisoning.

Original: Sun et al.
Revised: Sun et al.

Original: (2021) achieve successful poisoning against policy-based agents in various complex environments in online RL, but it remains an interesting open problem as to how to poison the ofﬂine RL dataset which is agnostic to the learning algorithm.
Revised: (2021) achieve successful poisoning against policy-based agents in various complex environments in online RL, but it remains an interesting open problem as to how to poison the ofﬂine RL dataset which is agnostic to the learning algorithm.

Original: I. E MPIRICALLY R OBUST RL
Revised: I. E MPIRICALLY R OBUST RL

Original: Robust RL against Evasion Attacks.
Revised: Robust RL against Evasion Attacks.

Original: We brieﬂy review several categories of RL methods that demonstrate empirical robustness against evasion attacks.
Revised: We brieﬂy review several categories of RL methods that demonstrate empirical robustness against evasion attacks.

Original: Randomization methods (Tobin et al., 2017; Akkaya et al., 2019) were ﬁrst proposed to encourage exploration.
Revised: Randomization methods (Tobin et al., 2017; Akkaya et al., 2019) were ﬁrst proposed to encourage exploration.

Original: This type of method was later systematically studied for its potential to improve model robustness.
Revised: This type of method was later systematically studied for its potential to improve model robustness.

Original: NoisyNet (Fortunato et al., 2017) adds parametric noise to the network’s weight during training, providing better resilience to both training-time and test-time attacks (Behzadan & Munir, 2017; 2018), also reducing the transferability of adversarial examples, and enabling quicker recovery with fewer number of transitions during phase transition.
Revised: NoisyNet (Fortunato et al., 2017) adds parametric noise to the network’s weight during training, providing better resilience to both training-time and test-time attacks (Behzadan & Munir, 2017; 2018), also reducing the transferability of adversarial examples, and enabling quicker recovery with fewer number of transitions during phase transition.

Original: Under the adversarial training framework, Kos & Song (2017) and Behzadan & Munir (2017) show that re-training with random noise and FGSM perturbations increases the resilience against adversarial examples.
Revised: Under the adversarial training framework, Kos & Song (2017) and Behzadan & Munir (2017) show that re-training with random noise and FGSM perturbations increases the resilience against adversarial examples.

Original: Pattanaik et al.
Revised: Pattanaik et al.

Original: (2018) leverage attacks using an engineered loss function speciﬁcally designed for RL to signiﬁcant increase the robustness to parameter variations.
Revised: (2018) leverage attacks using an engineered loss function speciﬁcally designed for RL to signiﬁcant increase the robustness to parameter variations.

Original: RSDQN (Fischer et al., 2019) is an imitation learning based approach that trains a robust student-DQN in parallel with a standard DQN in order to incorporate the constrains such as SOTA adversarial defenses (Madry et al., 2017; Mirman et al., 2018).
Revised: RSDQN (Fischer et al., 2019) is an imitation learning based approach that trains a robust student-DQN in parallel with a standard DQN in order to incorporate the constrains such as SOTA adversarial defenses (Madry et al., 2017; Mirman et al., 2018).

Original: SA-DQN (Zhang et al., 2020a) is a regularization based method that adds regularizers to the training loss function to encourage the top- 1 action to stay unchanged under perturbation.
Revised: SA-DQN (Zhang et al., 2020a) is a regularization based method that adds regularizers to the training loss function to encourage the top- 1 action to stay unchanged under perturbation.

Original: Built on top of the neural network veriﬁcation algorithms (Gowal et al., 2018;
Revised: Built on top of the neural network veriﬁcation algorithms (Gowal et al., 2018;

Original: Weng et al., 2018),
Revised: Weng et al., 2018),

Original: Radial-RL (Oikarinen et al., 2020) proposes to minimize an adversarial loss function that incorporates the upper bound of the perturbed loss, computed using certiﬁed bounds from veriﬁcation algorithms.
Revised: Radial-RL (Oikarinen et al., 2020) proposes to minimize an adversarial loss function that incorporates the upper bound of the perturbed loss, computed using certiﬁed bounds from veriﬁcation algorithms.

Original: CARRL (Everett et al., 2021) aims to compute the lower bounds of action-values under potential perturbation and select actions according to the worst-case bound, but it relies on linear bounds (Weng et al., 2018) and is only suitable for low-dimensional environments.
Revised: CARRL (Everett et al., 2021) aims to compute the lower bounds of action-values under potential perturbation and select actions according to the worst-case bound, but it relies on linear bounds (Weng et al., 2018) and is only suitable for low-dimensional environments.

Original: These robust RL methods only provide empirical robustness against perturbed state inputs during test time, but cannot provide theoretical guarantees for the performance of the trained models under any bounded perturbations.
Revised: These robust RL methods only provide empirical robustness against perturbed state inputs during test time, but cannot provide theoretical guarantees for the performance of the trained models under any bounded perturbations.

Original: Robust RL against Poisoning Attacks.
Revised: Robust RL against Poisoning Attacks.

Original: Banihashem et al.
Revised: Banihashem et al.

Original: (2021) considers the threat model of reward poisoning attacks proposed by Ma et al.
Revised: (2021) considers the threat model of reward poisoning attacks proposed by Ma et al.

Original: Rakhsha et al.
Revised: Rakhsha et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: (2020b), where the attacker aims to force learning a target policy while minimizing the cost of reward manipulation.
Revised: (2020b), where the attacker aims to force learning a target policy while minimizing the cost of reward manipulation.

Original: As shown in Ma et al.
Revised: As shown in Ma et al.

Original: Rakhsha et al.
Revised: Rakhsha et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: (2020b)
Revised: (2020b)

Original: , this optimization problem is feasible and always has a unique optimal solution.
Revised: , this optimization problem is feasible and always has a unique optimal solution.

Original: Leveraging this property, Banihashem et al.
Revised: Leveraging this property, Banihashem et al.

Original: (2021) formulates the defense task as another optimization problem which aims to optimize the agent’s worst case performance among the set of plausible candidates of the true reward function.
Revised: (2021) formulates the defense task as another optimization problem which aims to optimize the agent’s worst case performance among the set of plausible candidates of the true reward function.

Original: They speciﬁcally consider two settings regarding the agent’s knowledge of the attack parameter, and provide lower bounds on the performance of the defense policy.
Revised: They speciﬁcally consider two settings regarding the agent’s knowledge of the attack parameter, and provide lower bounds on the performance of the defense policy.

Original: In contrast to Banihashem et al.
Revised: In contrast to Banihashem et al.

Original: (2021) which focuses on a speciﬁc type of attack, our paper considers the threat model of general poisoning attacks, where the attacker has the power to manipulate the training trajectories arbitrarily.
Revised: (2021) which focuses on a speciﬁc type of attack, our paper considers the threat model of general poisoning attacks, where the attacker has the power to manipulate the training trajectories arbitrarily.

Original: Given limited knowledge regarding the attack, our proposed COPA framework is general and applicable to any potential attack.
Revised: Given limited knowledge regarding the attack, our proposed COPA framework is general and applicable to any potential attack.

Original: Our robustness is derived from the aggregation over both sub-policy level and temporal level.
Revised: Our robustness is derived from the aggregation over both sub-policy level and temporal level.

Original: One similarity between Banihashem et al.
Revised: One similarity between Banihashem et al.

Original: and our work is that we both aim to provide lower bounds of the cumulative reward for our proposed method as the provable guarantee / certiﬁcation criteria.
Revised: and our work is that we both aim to provide lower bounds of the cumulative reward for our proposed method as the provable guarantee / certiﬁcation criteria.

Original: I. R OBUSTNESS C ERTIFICATION FOR RL
Revised: I. R OBUSTNESS C ERTIFICATION FOR RL

Original: Wu et al.
Revised: Wu et al.

Original: (2022) provide the ﬁrst robustness certiﬁcation for RL against test-time evasion attacks following the line of work on randomized smoothing (Cohen et al., 2019;
Revised: (2022) provide the ﬁrst robustness certiﬁcation for RL against test-time evasion attacks following the line of work on randomized smoothing (Cohen et al., 2019;

Original: Salman et al., 2019).
Revised: Salman et al., 2019).

Original: Concretely, they apply per-state smoothing to achieve the certiﬁcation for per-state action stability, as well as trajectory smoothing to obtain the certiﬁcation for cumulative reward.
Revised: Concretely, they apply per-state smoothing to achieve the certiﬁcation for per-state action stability, as well as trajectory smoothing to obtain the certiﬁcation for cumulative reward.

Original: Notably, Wu et al.
Revised: Notably, Wu et al.

Original: (2022) propose an adaptive tree search algorithm to explore all possible trajectories and thus derive the robustness guarantee for cumulative reward.
Revised: (2022) propose an adaptive tree search algorithm to explore all possible trajectories and thus derive the robustness guarantee for cumulative reward.

Original: The relationship between our COPA-S EARCH and their Algorithm 3 is discussed in detail in Appendix E.3.
Revised: The relationship between our COPA-S EARCH and their Algorithm 3 is discussed in detail in Appendix E.3.

Original: In comparison, robustness in their work is derived from smoothing , while our robustness comes from aggregation .
Revised: In comparison, robustness in their work is derived from smoothing , while our robustness comes from aggregation .

Original: We propose aggregation protocols and certiﬁcation methods that leverage the temporal information by dynamically aggregating over the past time steps, which is not covered in Wu et al.
Revised: We propose aggregation protocols and certiﬁcation methods that leverage the temporal information by dynamically aggregating over the past time steps, which is not covered in Wu et al.