Original: RORL: Robust Offline Reinforcement Learning via Conservative Smoothing
Revised: RORL: Robust Offline Reinforcement Learning via Conservative Smoothing

Original: Abstract
Revised: Abstract

Original: Offline reinforcement learning (RL) provides a promising direction to exploit massive amount of offline data for complex decision-making tasks.
Revised: Offline reinforcement learning (RL) provides a promising direction to exploit massive amount of offline data for complex decision-making tasks.

Original: Due to the distribution shift issue, current offline RL algorithms are generally designed to be conservative in value estimation and action selection.
Revised: Due to the distribution shift issue, current offline RL algorithms are generally designed to be conservative in value estimation and action selection.

Original: However, such conservatism can impair the robustness of learned policies when encountering observation deviation under realistic conditions, such as sensor errors and adversarial attacks.
Revised: However, such conservatism can impair the robustness of learned policies when encountering observation deviation under realistic conditions, such as sensor errors and adversarial attacks.

Original: To trade off robustness and conservatism, we propose Robust Offline Reinforcement
Revised: To trade off robustness and conservatism, we propose Robust Offline Reinforcement

Original: Learning (RORL) with a novel conservative smoothing technique.
Revised: Learning (RORL) with a novel conservative smoothing technique.

Original: In RORL, we explicitly introduce regularization on the policy and the value function for states near the dataset, as well as additional conservative value estimation on these states.
Revised: In RORL, we explicitly introduce regularization on the policy and the value function for states near the dataset, as well as additional conservative value estimation on these states.

Original: Theoretically, we show RORL enjoys a tighter suboptimality bound than recent theoretical results in linear MDPs.
Revised: Theoretically, we show RORL enjoys a tighter suboptimality bound than recent theoretical results in linear MDPs.

Original: We demonstrate that RORL can achieve stateof-the-art performance on the general offline RL benchmark and is considerably robust to adversarial observation perturbations.
Revised: We demonstrate that RORL can achieve stateof-the-art performance on the general offline RL benchmark and is considerably robust to adversarial observation perturbations.

Original:  Introduction
Revised:  Introduction

Original: Over the past few years, deep reinforcement learning (RL) has been a vital tool for various decisionmaking tasks [35, 48, 46, 11] in a trial-and-error manner.
Revised: Over the past few years, deep reinforcement learning (RL) has been a vital tool for various decisionmaking tasks [36, 49, 47, 11] in a trial-and-error manner.

Original: A major limitation of current deep RL algorithms is that they require intense online interactions with the environment [29, 65].
Revised: A major limitation of current deep RL algorithms is that they require intense online interactions with the environment [30, 67].

Original: These data collecting processes can be costly and even prohibitive in many real-world scenarios such as robotics and health care [29, 51].
Revised: These data collecting processes can be costly and even prohibitive in many real-world scenarios such as robotics and health care [30, 53].

Original: Offline RL [14, 27] is gaining more attention recently since it offers probabilities to learn reinforced decision-making strategies from fully offline datasets.
Revised: Offline RL [14, 28] is gaining more attention recently since it offers probabilities to learn reinforced decision-making strategies from fully offline datasets.

Original: The main challenge of offline RL is the distribution shift between the offline dataset and the learned policy, which would lead to severe overestimation for the out-of-distribution (OOD) actions [14, 27].
Revised: The main challenge of offline RL is the distribution shift between the offline dataset and the learned policy, which would lead to severe overestimation for the out-of-distribution (OOD) actions [14, 28].

Original: To overcome such an issue, a series of model-free offline RL works [57, 14, 67, 28, 31, 2, 64, 6] propose to celebrate conservatism, such as constraining the learned policy close to the supported distribution or penalizing the Q -values of OOD actions.
Revised: To overcome such an issue, a series of model-free offline RL works [59, 14, 69, 29, 32, 2, 66, 6] propose to celebrate conservatism, such as constraining the learned policy close to the supported distribution or penalizing the Q -values of OOD actions.

Original: Besides, another stream of works builds upon model-based algorithms [70, 69, 56], which leverages the ensemble dynamics models to enforce pessimism through uncertainty penalizing or data generation.
Revised: Besides, another stream of works builds upon model-based algorithms [72, 71, 58], which leverages the ensemble dynamics models to enforce pessimism through uncertainty penalizing or data generation.

Original: However, conservatism is not the only concern when applying offline RL to the real world.
Revised: However, conservatism is not the only concern when applying offline RL to the real world.

Original: Due to the sensor errors and model mismatch, the robustness of offline RL is also crucial under the realistic engineering conditions, which has not been well studied yet.
Revised: Due to the sensor errors and model mismatch, the robustness of offline RL is also crucial under the realistic engineering conditions, which has not been well studied yet.

Original: In online RL, a series of works has been studied to learn the optimal policy under worst-case perturbations of the observation [71, 40, 19] or environmental dynamics [55, 42, 43,  4].
Revised: In online RL, a series of works has been studied to learn the optimal policy under worst-case perturbations of the observation [73, 41, 20] or environmental dynamics [57, 43, 44, 4].

Original: Yet, it is non-trivial to apply online robust RL techniques into the offline problems.
Revised: Yet, it is non-trivial to apply online robust RL techniques into the offline problems.

Original: The main challenge is that the perturbation of states may bring OOD observation and extra overestimation for the value function.
Revised: The main challenge is that the perturbation of states may bring OOD observation and extra overestimation for the value function.

Original: New techniques are needed to tackle the conservatism and robustness simultaneously in the offline RL.
Revised: New techniques are needed to tackle the conservatism and robustness simultaneously in the offline RL.

Original: This paper studies robust offline RL against adversarial observation perturbations, where the agent needs to learn the policy conservatively while handling the potential OOD observation with perturbation.
Revised: This paper studies robust offline RL against adversarial observation perturbations, where the agent needs to learn the policy conservatively while handling the potential OOD observation with perturbation.

Original: We first demonstrate that current value-based offline RL algorithms lack the necessary smoothness for the policy, which is visualized in Figure 1.
Revised: We first demonstrate that current value-based offline RL algorithms lack the necessary smoothness for the policy, which is visualized in Figure 1.

Original: As an illustration, we show that a famous baseline method CQL [28] learns a non-smooth value function, leading to significant performance degradation for even a tiny scale perturbation on observation (see Section 3 for details).
Revised: As an illustration, we show that a famous baseline method CQL [29] learns a non-smooth value function, leading to significant performance degradation for even a tiny scale perturbation on observation (see Section 3 for details).

Original: In addition, simply adopting the smoothing technique for existing methods may result in extra overestimation at the boundary of supported distribution and lead the agent toward unsafe areas.
Revised: In addition, simply adopting the smoothing technique for existing methods may result in extra overestimation at the boundary of supported distribution and lead the agent toward unsafe areas.

Original: To this end, we propose Robust Offline Reinforcement Learning (RORL) with a novel conservative smoothing technique, which explicitly handles the overestimation of OOD state-action pairs.
Revised: To this end, we propose Robust Offline Reinforcement Learning (RORL) with a novel conservative smoothing technique, which explicitly handles the overestimation of OOD state-action pairs.

Original: Specifically, we explicitly introduce smooth regularization on both the value functions and policies for states near the dataset support and conservatively estimate the values of these OOD states based on pessimistic bootstrapping.
Revised: Specifically, we explicitly introduce smooth regularization on both the value functions and policies for states near the dataset support and conservatively estimate the values of these OOD states based on pessimistic bootstrapping.

Original: Furthermore, we theoretically prove that RORL yields a valid uncertainty quantifier in linear MDPs and enjoys a tighter suboptimality bound than previous work [6].
Revised: Furthermore, we theoretically prove that RORL yields a valid uncertainty quantifier in linear MDPs and enjoys a tighter suboptimality bound than previous work [6].

Original: In our experiments 3 , we demonstrate that RORL can achieve state-of-the-art (SOTA) performance in the D4RL benchmark [12] with fewer ensemble Q networks than the current SOTA approach [2].
Revised: In our experiments 3 , we demonstrate that RORL can achieve state-of-the-art (SOTA) performance in the D4RL benchmark [12] with fewer ensemble Q networks than the current SOTA approach [2].

Original: The results of the benchmark experiments imply that robust training can lead to performance improvement in non-perturbed environments.
Revised: The results of the benchmark experiments imply that robust training can lead to performance improvement in non-perturbed environments.

Original: Meanwhile, compared with current ensemble-based baselines, RORL is considerably more robust to adversarial perturbations on observations.
Revised: Meanwhile, compared with current ensemble-based baselines, RORL is considerably more robust to adversarial perturbations on observations.

Original: We conduct the adversarial experiments under different attack types, showing consistently superior performance on several continuous control tasks.
Revised: We conduct the adversarial experiments under different attack types, showing consistently superior performance on several continuous control tasks.

Original:  Preliminaries
Revised:  Preliminaries

Original: Offline RL Considering an episodic MDP M =
Revised: Offline RL Considering an episodic MDP M =

Original: ( S , A , T, r, γ, P ) , where S is the state space, A is the action space, T is the length of an episode, r is the reward function, P is the dynamics, and γ is the discount factor.
Revised: ( S , A , T, r, γ, P ) , where S is the state space, A is the action space, T is the length of an episode, r is the reward function, P is the dynamics, and γ is the discount factor.

Original: In offline RL, the objective of the agent is to find an optimal policy by sampling experiences from a fixed dataset D
Revised: In offline RL, the objective of the agent is to find an optimal policy by sampling experiences from a fixed dataset D

Original: = { ( s it , a it , r it , s it +1 ) } .
Revised: = { ( s it , a it , r it , s it +1 ) } .

Original: Nevertheless, directly applying off-policy algorithms in offline RL suffers from the distribution shift problem.
Revised: Nevertheless, directly applying off-policy algorithms in offline RL suffers from the distribution shift problem.

Original: In Q -learning, the value function evaluated on the greedy action a ′ in Bellman operator T Q = r + γ E s ′
Revised: In Q -learning, the value function evaluated on the greedy action a ′ in Bellman operator T Q = r + γ E s ′

Original: [max a ′ ( s ′ , a ′ )] tends to have extrapolation error since ( s ′ , a ′ ) has barely occurred in D .
Revised: [max a ′ ( s ′ , a ′ )] tends to have extrapolation error since ( s ′ , a ′ ) has barely occurred in D .

Original: Pessimistic Bootstrapping for Offline RL (PBRL)
Revised: Pessimistic Bootstrapping for Offline RL (PBRL)

Original: [6] is an uncertainty-based method that uses bootstrapped Q -functions for uncertainty quantification [50] and OOD sampling for regularization.
Revised: [6] is an uncertainty-based method that uses bootstrapped Q -functions for uncertainty quantification [52] and OOD sampling for regularization.

Original: [Figure]
Revised: [Figure]

Original: Specifically, PBRL maintains K bootstrapped Q functions to quantify the epistemic uncertainty [5] and performs pessimistic update to penalize Q functions with large uncertainties.
Revised: Specifically, PBRL maintains K bootstrapped Q functions to quantify the epistemic uncertainty [5] and performs pessimistic update to penalize Q functions with large uncertainties.

Original: The uncertainty is defined as the standard deviation among bootstrapped Q -functions.
Revised: The uncertainty is defined as the standard deviation among bootstrapped Q -functions.

Original: For each bootstrapped Q -function, the Bellman target is defined as (cid:98) T Q ( s, a ) =
Revised: For each bootstrapped Q -function, the Bellman target is defined as (cid:98) T Q ( s, a ) =

Original: r ( s, a )+ γ (cid:98) E s [Equation] ∼ P ( ·| s,a
Revised: r ( s, a )+ γ (cid:98) E s ′ ∼ P ( ·| s,a

Original: [Equation]
Revised: [Equation]

Original: ,a ′ ∼
Revised: ,a ′ ∼

Original: [Equation]
Revised: [Equation]

Original: | [Equation] [Equation] (cid:2) Q ( s ′ , a ′ )
Revised: | s ′ ) (cid:2) Q ( s ′ , a ′ )

Original: − λu ( s ′ , a ′ ) (cid:3) .
Revised: − λu ( s ′ , a ′ ) (cid:3) .

Original: Under linear MDP assumptions, this uncertainty is equivalent to the LCB penalty and is provably efficient [23].
Revised: Under linear MDP assumptions, this uncertainty is equivalent to the LCB penalty and is provably efficient [24].

Original: Furthermore, PBRL incorporates OOD sampling by sampling OOD actions to form ( s, a ood ) pairs, where a ood follows the learned policy.
Revised: Furthermore, PBRL incorporates OOD sampling by sampling OOD actions to form ( s, a ood ) pairs, where a ood follows the learned policy.

Original: The detached learning target for ( s, a ood ) is (cid:98) T ood Q ( s, a ood ) :
Revised: The detached learning target for ( s, a ood ) is (cid:98) T ood Q ( s, a ood ) :

Original: = Q ( s, a ood ) − λu ( s, a ood ) , which introduces uncertainty penalization to enforce pessimistic Q -functions for OOD actions.
Revised: = Q ( s, a ood ) − λu ( s, a ood ) , which introduces uncertainty penalization to enforce pessimistic Q -functions for OOD actions.

Original: Smooth Regularized RL Robust RL aims to learn a robust policy against the adversarial perturbed environment in online RL.
Revised: Smooth Regularized RL Robust RL aims to learn a robust policy against the adversarial perturbed environment in online RL.

Original: SR 2 L [47] enforces smoothness in both the policy and Q -functions.
Revised: SR 2 L [48] enforces smoothness in both the policy and Q -functions.

Original: Specifically, SR 2 L encourages the outputs of the policy and value function to not change much when injecting small perturbations to the states.
Revised: Specifically, SR 2 L encourages the outputs of the policy and value function to not change much when injecting small perturbations to the states.

Original: For state s , SR 2 L constructs a perturbation set B d ( s, ϵ )
Revised: For state s , SR 2 L constructs a perturbation set B d ( s, ϵ )

Original: = { ˆ s : d ( s, ˆ s ) ≤ ϵ } with a metric d ( , ) , which is chosen to be the ℓ p distance, and introduces a smoothness regularizer for policy as R πs =
Revised: = { ˆ s : d ( s, ˆ s ) ≤ ϵ } with a metric d ( , ) , which is chosen to be the ℓ p distance, and introduces a smoothness regularizer for policy as R πs =

Original: E s ∼ ρ π max ˆ s ∈ B d ( s,ϵ ) D
Revised: E s ∼ ρ π max ˆ s ∈ B d ( s,ϵ ) D

Original: ( π ( ·| s )
Revised: ( π ( ·| s )

Original: ∥ π ( ·| ˆ s )) , where D ( ·∥· ) is a distance metric and the max operator gives an adversarial manner to choose ˆ s .
Revised: ∥ π ( ·| ˆ s )) , where D ( ·∥· ) is a distance metric and the max operator gives an adversarial manner to choose ˆ s .

Original: Similarly, the smoothness regularizer for the value function is defined as R Vs =
Revised: Similarly, the smoothness regularizer for the value function is defined as R Vs =

Original: E s ∼ ρ π ,a ∼ π max ˆ s ∈ B d ( s,ϵ ) ( Q ( s, a ) −
Revised: E s ∼ ρ π ,a ∼ π max ˆ s ∈ B d ( s,ϵ ) ( Q ( s, a ) −

Original: Q (ˆ s, a )) 2 .
Revised: Q (ˆ s, a )) 2 .

Original: SR 2 L is shown to improve robustness against both random and adversarial perturbations.
Revised: SR 2 L is shown to improve robustness against both random and adversarial perturbations.

Original:  Robustness of Offline RL: A Motivating Example
Revised:  Robustness of Offline RL: A Motivating Example

Original: We give a motivating example to illustrate the robustness of the popular CQL [28] policies.
Revised: We give a motivating example to illustrate the robustness of the popular CQL [29] policies.

Original: We introduce an adversarial attack on state s to obtain ˆ s = arg max ˆ
Revised: We introduce an adversarial attack on state s to obtain ˆ s = arg max ˆ

Original: s ∈ B d ( s,ϵ ) D J ( π θ ( ·| s )
Revised: s ∈ B d ( s,ϵ ) D J ( π θ ( ·| s )

Original: ∥
Revised: ∥

Original: π θ ( ·| ˆ s )) , where B d ( s, ϵ ) =
Revised: π θ ( ·| ˆ s )) , where B d ( s, ϵ ) =

Original: { ˆ s : d ( s, ˆ s ) ≤ ϵ } is the perturbation set and the metric d ( , ) is chosen to be the ℓ ∞ norm.
Revised: { ˆ s : d ( s, ˆ s ) ≤ ϵ } is the perturbation set and the metric d ( , ) is chosen to be the ℓ ∞ norm.

Original: The Jeffrey’s divergence D J for two distributions P , Q is defined by: D J ( P ∥ Q ) =
Revised: The Jeffrey’s divergence D J for two distributions P , Q is defined by: D J ( P ∥ Q ) =

Original: 12 [ D KL ( P ∥ Q ) + D KL ( Q ∥ P )] .
Revised: 12 [ D KL ( P ∥ Q ) + D KL ( Q ∥ P )] .

Original: To obtain ˆ s , we take gradient assent with respect to the loss function
Revised: To obtain ˆ s , we take gradient assent with respect to the loss function

Original: D J ( π θ ( ·| s ) ∥
Revised: D J ( π θ ( ·| s ) ∥

Original: π θ ( ·| ˆ s )) and restrict the outputs to the B d ( s, ϵ ) set, where π θ is a learned CQL policy.
Revised: π θ ( ·| ˆ s )) and restrict the outputs to the B d ( s, ϵ ) set, where π θ is a learned CQL policy.

Original: We remark that the the perturbation is applied on normalized observations following prior work [71].
Revised: We remark that the the perturbation is applied on normalized observations following prior work [73].

Original: In the walker-medium-v2 task from D4RL [12], we use various ϵ for adversarial attack to evaluate the robustness of CQL policies.
Revised: In the walker-medium-v2 task from D4RL [12], we use various ϵ for adversarial attack to evaluate the robustness of CQL policies.

Original: Specifically, we use ϵ ∈ { 0 , 0 . 05 , 0 . 1 , 0 . 14 } to control the strengths of the attack, where we have ˆ s = s if ϵ = 0 .
Revised: Specifically, we use ϵ ∈ { 0 , 0 . 05 , 0 . 1 , 0 . 14 } to control the strengths of the attack, where we have ˆ s = s if ϵ = 0 .

Original: Given a specific ϵ , we sample N state-action pairs { ( s i , a i ) } from the offline dataset, and then perform adversarial attack to obtain { (ˆ s i , a i ) } and the corresponding Q -values { Q i (ˆ s i , a i ) } , where the Q -function is the trained critic of CQL.
Revised: Given a specific ϵ , we sample N state-action pairs { ( s i , a i ) } from the offline dataset, and then perform adversarial attack to obtain { (ˆ s i , a i ) } and the corresponding Q -values { Q i (ˆ s i , a i ) } , where the Q -function is the trained critic of CQL.

Original: Figure 2(a) shows the relationship between ˆ s i and the corresponding Q i with different ϵ .
Revised: Figure 2(a) shows the relationship between ˆ s i and the corresponding Q i with different ϵ .

Original: To visualize ˆ
Revised: To visualize ˆ

Original: s i , we perform PCA dimensional reduction [53] and choose one of the reduced dimensions to represent ˆ s i .
Revised: s i , we perform PCA dimensional reduction [55] and choose one of the reduced dimensions to represent ˆ s i .

Original: More details can be found in Appendix B.3.
Revised: More details can be found in Appendix B.3.

Original: With the increase of ϵ in the adversarial attack, the Q -curve has greater deviation compared to the curve with ϵ = 0 .
Revised: With the increase of ϵ in the adversarial attack, the Q -curve has greater deviation compared to the curve with ϵ = 0 .

Original: The result signifies that the Q -function of CQL is not smooth in the state space, which makes the adversarial noises easily affect the Q values.
Revised: The result signifies that the Q -function of CQL is not smooth in the state space, which makes the adversarial noises easily affect the Q values.

Original: As a comparison, we apply the proposed conservative smoothing loss in CQL
Revised: As a comparison, we apply the proposed conservative smoothing loss in CQL

Original: OfflineDataset ̂𝑠,𝑎 Perturbation
Revised: OfflineDataset ̂𝑠,𝑎 Perturbation

Original: OOD sampling ̂𝑠, $𝑎
Revised: OOD sampling ̂𝑠, $𝑎

Original: Uncertainty Penalty
Revised: Uncertainty Penalty

Original: Conservative Smooth
Revised: Conservative Smooth

Original: Ensemble Soft Q -Learning Bellman
Revised: Ensemble Soft Q -Learning Bellman

Original: [Figure]
Revised: [Figure]

Original: Initialize policy π θ and Q
Revised: Initialize policy π θ and Q

Original: [Table]
Revised: [Table]

Original: functions { Q ϕ 1 , . .
Revised: functions { Q ϕ 1 , . .

Original: . , Q ϕ K } .
Revised: . , Q ϕ K } .

Original: while not converged do
Revised: while not converged do

Original: Sample mini-batch transitions ( s, a, r, s ′ ) from D .
Revised: Sample mini-batch transitions ( s, a, r, s ′ ) from D .

Original: Sample ˆ s from B d ( s, ϵ ) to obtain (ˆ s, a ) pairs.
Revised: Sample ˆ s from B d ( s, ϵ ) to obtain (ˆ s, a ) pairs.

Original: Calculate the Q smooth loss L smooth .
Revised: Calculate the Q smooth loss L smooth .

Original: Sample OOD actions ˆ a ∼ π θ (ˆ s ) .
Revised: Sample OOD actions ˆ a ∼ π θ (ˆ s ) .

Original: Calculate uncertainty u
Revised: Calculate uncertainty u

Original: (ˆ s, ˆ a ) and the OOD loss L ood .
Revised: (ˆ s, ˆ a ) and the OOD loss L ood .

Original: Train each Q function Q ϕ i with Eq.
Revised: Train each Q function Q ϕ i with Eq.

Original: Train the policy π θ with Eq.
Revised: Train the policy π θ with Eq.

Original: training (i.e., CQL-smooth ) and use the same evaluation method to obtain ˆ s i and Q i .
Revised: training (i.e., CQL-smooth ) and use the same evaluation method to obtain ˆ s i and Q i .

Original: According to the result in Figure 2(b), the value function becomes smoother.
Revised: According to the result in Figure 2(b), the value function becomes smoother.

Original: In addition, we show how the adversarial attack affects the final performance of offline RL policies.
Revised: In addition, we show how the adversarial attack affects the final performance of offline RL policies.

Original: We use ϵ ∈ [0 , 0 . 15] to evaluate both the original CQL policies (i.e., CQL ) and CQL with conservative smoothing loss (i.e., CQL-smooth ) in adversarial attack.
Revised: We use ϵ ∈ [0 , 0 . 15] to evaluate both the original CQL policies (i.e., CQL ) and CQL with conservative smoothing loss (i.e., CQL-smooth ) in adversarial attack.

Original: Figure 2(c) shows the performance with different settings of ϵ .
Revised: Figure 2(c) shows the performance with different settings of ϵ .

Original: We find that our smooth constraints significantly improve the robustness of
Revised: We find that our smooth constraints significantly improve the robustness of

Original: CQL, especially for large adversarial noises.
Revised: CQL, especially for large adversarial noises.

Original:  Robust Offline RL via Conservative Smoothing
Revised:  Robust Offline RL via Conservative Smoothing

Original: In RORL, we develop smooth regularization on both the policy and the value function for states near the dataset.
Revised: In RORL, we develop smooth regularization on both the policy and the value function for states near the dataset.

Original: The smooth constraints make the policy and the Q -functions robust to observation perturbations.
Revised: The smooth constraints make the policy and the Q -functions robust to observation perturbations.

Original: Nevertheless, the smoothness may also lead to value overestimation in areas outside the supported dataset.
Revised: Nevertheless, the smoothness may also lead to value overestimation in areas outside the supported dataset.

Original: To address this problem, we adopt bootstrapped Q -functions [38, 6] for uncertainty quantification and sample perturbed states and OOD actions for penalization.
Revised: To address this problem, we adopt bootstrapped Q -functions [39, 6] for uncertainty quantification and sample perturbed states and OOD actions for penalization.

Original: RORL obtains conservative and smooth value estimation on OOD states, which can improve the generalization ability of offline RL algorithms.
Revised: RORL obtains conservative and smooth value estimation on OOD states, which can improve the generalization ability of offline RL algorithms.

Original: The overall architecture of RORL is given in Figure 3.
Revised: The overall architecture of RORL is given in Figure 3.

Original: Robust Q -function We sample three sets of state-action pairs and apply different loss functions to obtain a conservative and smooth policy.
Revised: Robust Q -function We sample three sets of state-action pairs and apply different loss functions to obtain a conservative and smooth policy.

Original: Specifically, for a ( s, a ) pair sampled from D , we construct a perturbation set B d ( s, ϵ ) to obtain (ˆ s, a ) pairs, where ˆ s ∈ B d ( s, ϵ ) and ϵ is the perturbation scale.
Revised: Specifically, for a ( s, a ) pair sampled from D , we construct a perturbation set B d ( s, ϵ ) to obtain (ˆ s, a ) pairs, where ˆ s ∈ B d ( s, ϵ ) and ϵ is the perturbation scale.

Original: The perturbation set B d ( s, ϵ ) =
Revised: The perturbation set B d ( s, ϵ ) =

Original: { ˆ s : d ( s, ˆ s ) ≤ ϵ } for state s is an ϵ -radius ball measured in metric d ( , ) , which is the ℓ ∞ norm in our paper.
Revised: { ˆ s : d ( s, ˆ s ) ≤ ϵ } for state s is an ϵ -radius ball measured in metric d ( , ) , which is the ℓ ∞ norm in our paper.

Original: Then we perform OOD sampling by using the current policy π θ to obtain (ˆ s, ˆ a ) pairs, where ˆ a ∼ π θ (ˆ s ) .
Revised: Then we perform OOD sampling by using the current policy π θ to obtain (ˆ s, ˆ a ) pairs, where ˆ a ∼ π θ (ˆ s ) .

Original: RORL contains K ensemble Q -functions.
Revised: RORL contains K ensemble Q -functions.

Original: We denote the parameters of the i -th Q -function and the target Q -function as ϕ i and ϕ ′
Revised: We denote the parameters of the i -th Q -function and the target Q -function as ϕ i and ϕ ′

Original: i , respectively.
Revised: i , respectively.

Original: In the following, we give different learning targets for ( s, a ) , (ˆ s, a ) , and (ˆ s, ˆ a ) pairs.
Revised: In the following, we give different learning targets for ( s, a ) , (ˆ s, a ) , and (ˆ s, ˆ a ) pairs.

Original: First, for a ( s, a ) pair sampled from D , we apply extended soft Q -learning to obtain the target as
Revised: First, for a ( s, a ) pair sampled from D , we apply extended soft Q -learning to obtain the target as

Original: [Equation]
Revised: [Equation]

Original: where the next- Q function takes minimum value among the target Q -functions and log π θ ( a ′ | s ′ ) is the entropy regularization.
Revised: where the next- Q function takes minimum value among the target Q -functions and log π θ ( a ′ | s ′ ) is the entropy regularization.

Original: Note that Eq.
Revised: Note that Eq.

Original: (1) is the same learning target of SAC- N in [2].
Revised: (1) is the same learning target of SAC- N in [2].

Original: Then, for a (ˆ s, a ) pair with a perturbed state, we enforce smoothness in each Q -function by minimizing the Q -value difference between Q ( s, a ) and Q (ˆ s, a ) .
Revised: Then, for a (ˆ s, a ) pair with a perturbed state, we enforce smoothness in each Q -function by minimizing the Q -value difference between Q ( s, a ) and Q (ˆ s, a ) .

Original: In particular, we choose an adversarial ˆ s ∈ B d ( s, ϵ ) that maximizes a inner objective L ( Q (ˆ s, a ) , Q ( s, a )) , and then train each Q -function to minimize a loss function L smooth with the adversarial ˆ s .
Revised: In particular, we choose an adversarial ˆ s ∈ B d ( s, ϵ ) that maximizes a inner objective L ( Q (ˆ s, a ) , Q ( s, a )) , and then train each Q -function to minimize a loss function L smooth with the adversarial ˆ s .

Original: Intuitively, we want the Q -function to be smooth under the most difficult (i.e., adversarial) perturbation in B d ( s, ϵ ) .
Revised: Intuitively, we want the Q -function to be smooth under the most difficult (i.e., adversarial) perturbation in B d ( s, ϵ ) .

Original: The smooth loss function for Q ϕ i is as follows:
Revised: The smooth loss function for Q ϕ i is as follows:

Original: [Equation]
Revised: [Equation]

Original: We denote δ ( s, ˆ s, a ) =
Revised: We denote δ ( s, ˆ s, a ) =

Original: Q ϕ i (ˆ s, a ) − Q ϕ
Revised: Q ϕ i (ˆ s, a ) − Q ϕ

Original: i ( s, a ) and remark that if δ ( s, ˆ s, a ) > 0 , the perturbed state may induce an overestimated Q -value that we need to smooth.
Revised: i ( s, a ) and remark that if δ ( s, ˆ s, a ) > 0 , the perturbed state may induce an overestimated Q -value that we need to smooth.

Original: In contrast, if δ ( s, ˆ s, a ) < 0 , the perturbed Q -function is underestimated, which does not cause a serious problem in offline RL.
Revised: In contrast, if δ ( s, ˆ s, a ) < 0 , the perturbed Q -function is underestimated, which does not cause a serious problem in offline RL.

Original: As a result, we use different weights for δ ( s, ˆ s, a ) + and δ ( s, ˆ s, a ) − , where x + = max( x, 0) and x − = min( x, 0) .
Revised: As a result, we use different weights for δ ( s, ˆ s, a ) + and δ ( s, ˆ s, a ) − , where x + = max( x, 0) and x − = min( x, 0) .

Original: The definition of L ( · , · ) is give as follows:
Revised: The definition of L ( · , · ) is give as follows:

Original: [Equation]
Revised: [Equation]

Original: where we can choose τ ≤ 0 .
Revised: where we can choose τ ≤ 0 .

Original: In L smooth , we does not introduce OOD action ˆ a for smoothing since the actions are desired to be close to the behavior actions for areas near the offline dataset.
Revised: In L smooth , we does not introduce OOD action ˆ a for smoothing since the actions are desired to be close to the behavior actions for areas near the offline dataset.

Original: Finally, to prevent overestimation of OOD states and actions, we use bootstrapped uncertainty u
Revised: Finally, to prevent overestimation of OOD states and actions, we use bootstrapped uncertainty u

Original: (ˆ s, ˆ a ) as the penalty for Q (ˆ s, ˆ a ) , where ˆ a ∼ π θ (ˆ s ) is an OOD action sampled from the current policy π θ .
Revised: (ˆ s, ˆ a ) as the penalty for Q (ˆ s, ˆ a ) , where ˆ a ∼ π θ (ˆ s ) is an OOD action sampled from the current policy π θ .

Original: We remark that a similar OOD sampling is also used in PBRL [6].
Revised: We remark that a similar OOD sampling is also used in PBRL [6].

Original: The difference is that PBRL only penalizes the OOD actions for in-distribution states, while RORL penalizes both the OOD states and
Revised: The difference is that PBRL only penalizes the OOD actions for in-distribution states, while RORL penalizes both the OOD states and

Original: OOD actions to provide conservatism for unfamiliar areas.
Revised: OOD actions to provide conservatism for unfamiliar areas.

Original: We follow PBRL and use a loss function as:
Revised: We follow PBRL and use a loss function as:

Original: [Equation]
Revised: [Equation]

Original: where the pseudo-target for the OOD datapoints is computed as: (cid:98) T ood
Revised: where the pseudo-target for the OOD datapoints is computed as: (cid:98) T ood

Original: Q ϕ
Revised: Q ϕ

Original: i (ˆ s, ˆ a ) :=
Revised: i (ˆ s, ˆ a ) :=

Original: Q ϕ i (ˆ s, ˆ a ) − u (ˆ s, ˆ a ) , which is detached from gradients similar to the conventional TD target.
Revised: Q ϕ i (ˆ s, ˆ a ) − u (ˆ s, ˆ a ) , which is detached from gradients similar to the conventional TD target.

Original: The bootstrapped uncertainty u (ˆ s, ˆ a ) is defined as the standard deviation among the Q -ensemble:
Revised: The bootstrapped uncertainty u (ˆ s, ˆ a ) is defined as the standard deviation among the Q -ensemble:

Original: [Equation]
Revised: [Equation]

Original: The ensemble technique [38] forms an estimation of the Q -posterior, which yields diverse predictions and large penalty u
Revised: The ensemble technique [39] forms an estimation of the Q -posterior, which yields diverse predictions and large penalty u

Original: (ˆ s, ˆ a ) on areas with scarce data.
Revised: (ˆ s, ˆ a ) on areas with scarce data.

Original: Combining the loss functions above, RORL has the following loss function for each Q ϕ
Revised: Combining the loss functions above, RORL has the following loss function for each Q ϕ

Original: i :
Revised: i :

Original: [Equation]
Revised: [Equation]

Original: Robust Policy We learn a robust policy by using a smooth constraint to make the policy change less under perturbations.
Revised: Robust Policy We learn a robust policy by using a smooth constraint to make the policy change less under perturbations.

Original: Similarly, we choose an adversarial state ˆ s ∈ B d ( s, ϵ ) that maximizes
Revised: Similarly, we choose an adversarial state ˆ s ∈ B d ( s, ϵ ) that maximizes

Original: D J (cid:0) π θ ( ·| s ) ∥
Revised: D J (cid:0) π θ ( ·| s ) ∥

Original: π θ ( ·| ˆ s ) (cid:1) , and then minimize the policy difference between π θ ( ·| s ) and π θ ( ·| ˆ s ) .
Revised: π θ ( ·| ˆ s ) (cid:1) , and then minimize the policy difference between π θ ( ·| s ) and π θ ( ·| ˆ s ) .

Original: To conclude, we minimize the following loss function for π θ :
Revised: To conclude, we minimize the following loss function for π θ :

Original: [Equation]
Revised: [Equation]

Original: where the first term aims to maximize the minimum of the ensemble Q -functions to obtain a conservative policy, and the second term is the entropy regularization.
Revised: where the first term aims to maximize the minimum of the ensemble Q -functions to obtain a conservative policy, and the second term is the entropy regularization.

Original:  Theoretical Analysis
Revised:  Theoretical Analysis

Original: We analyze a simplified learning objective of RORL in linear MDPs [22, 23], where the feature map of the state-action pair takes the form of ϕ : S ×
Revised: We analyze a simplified learning objective of RORL in linear MDPs [23, 24], where the feature map of the state-action pair takes the form of ϕ : S ×

Original: A → R d , and both the transition function and the reward function are assumed to be linear in ϕ .
Revised: A → R d , and both the transition function and the reward function are assumed to be linear in ϕ .

Original: The parameter (cid:101) w t of RORL can be solved in closed form following the least squares value iteration (LSVI), which minimizes the following loss function.
Revised: The parameter (cid:101) w t of RORL can be solved in closed form following the least squares value iteration (LSVI), which minimizes the following loss function.

Original: [Equation]
Revised: [Equation]

Original: where we have Q w ( s it , a it )
Revised: where we have Q w ( s it , a it )

Original: = ϕ
Revised: = ϕ

Original: ( s it , a it ) ⊤ w since the Q -function is also linear in ϕ .
Revised: ( s it , a it ) ⊤ w since the Q -function is also linear in ϕ .

Original: The first term in Eq.
Revised: The first term in Eq.

Original: (7) is the ordinary TD-error, where we consider the setting of γ = 1 and the Q -target is y it =
Revised: (7) is the ordinary TD-error, where we consider the setting of γ = 1 and the Q -target is y it =

Original: r ( s it , a it )
Revised: r ( s it , a it )

Original: + V t +

Revised: + V t +

Original: ( s it +1 ) .
Revised: ( s it +1 ) .

Original: The second term is the proposed conservative smoothing loss.
Revised: The second term is the proposed conservative smoothing loss.

Original: Specifically, ˆ s it ∼ D
Revised: Specifically, ˆ s it ∼ D

Original: ood
Revised: ood

Original: ( s it ) are sampled from a l ∞ ball of center s it and norm ϵ > 0 , which can also be formulated as ˆ s it
Revised: ( s it ) are sampled from a l ∞ ball of center s it and norm ϵ > 0 , which can also be formulated as ˆ s it

Original: ∼ B d
Revised: ∼ B d

Original: ( s it , ϵ ) .
Revised: ( s it , ϵ ) .

Original: The third term is the additional OOD-sampling loss, which enforces conservatism for OOD states and OOD actions.
Revised: The third term is the additional OOD-sampling loss, which enforces conservatism for OOD states and OOD actions.

Original: In contrast to PBRL [6], we use perturbed states sampled from D ood = m (cid:83) i
Revised: In contrast to PBRL [6], we use perturbed states sampled from D ood = m (cid:83) i

Original: =1 D ood ( s it ) rather than states from dataset.
Revised: =1 D ood ( s it ) rather than states from dataset.

Original: The OOD action ˆ a is sampled from policy π .
Revised: The OOD action ˆ a is sampled from policy π .

Original: The explicit solution of Eq.
Revised: The explicit solution of Eq.

Original: (7) takes the following form:
Revised: (7) takes the following form:

Original: [Equation]
Revised: [Equation]

Original: where the covariance matrix (cid:101) Λ t is defined as
Revised: where the covariance matrix (cid:101) Λ t is defined as

Original: [Equation]
Revised: [Equation]

Original: We denote the first term and the second term as (cid:101) Λ in and (cid:101) Λ ood t , which represent the covariance matrices induced by the offline samples and OOD samples, respectively.
Revised: We denote the first term and the second term as (cid:101) Λ in and (cid:101) Λ ood t , which represent the covariance matrices induced by the offline samples and OOD samples, respectively.

Original: Nevertheless, in linear MDPs, it is difficult to ensure the covariance (cid:101) Λ in + (cid:101) Λ ood t ⪰ λ · I , since it requires that the embeddings of the samples are isotropic to make the eigenvalues of the corresponding covariance matrix lower bounded.
Revised: Nevertheless, in linear MDPs, it is difficult to ensure the covariance (cid:101) Λ in + (cid:101) Λ ood t ⪰ λ · I , since it requires that the embeddings of the samples are isotropic to make the eigenvalues of the corresponding covariance matrix lower bounded.

Original: This condition holds if we can sample embeddings uniformly from the whole embedding space.
Revised: This condition holds if we can sample embeddings uniformly from the whole embedding space.

Original: However, since the offline dataset has limited coverage in the state-action space and the OOD samples come from limited l ∞ -balls around the offline data, (cid:101) Λ in + (cid:101)
Revised: However, since the offline dataset has limited coverage in the state-action space and the OOD samples come from limited l ∞ -balls around the offline data, (cid:101) Λ in + (cid:101)

Original: Λ ood t cannot be guaranteed to be positive definite.
Revised: Λ ood t cannot be guaranteed to be positive definite.

Original: PBRL [6] uses the assumption of (cid:101)
Revised: PBRL [6] uses the assumption of (cid:101)

Original: Λ ood t ⪰ λ · I , while it is unachievable empirically.
Revised: Λ ood t ⪰ λ · I , while it is unachievable empirically.

Original: In RORL, we solve this problem by introducing an additional conservative smoothing loss, which induces a covariance matrix as (cid:101) Λ ood_diff t
Revised: In RORL, we solve this problem by introducing an additional conservative smoothing loss, which induces a covariance matrix as (cid:101) Λ ood_diff t

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: the third term in Eq.
Revised: the third term in Eq.

Original: The following theorem gives the guarantees of (cid:101) Λ ood_diff t ⪰ λ · I .
Revised: The following theorem gives the guarantees of (cid:101) Λ ood_diff t ⪰ λ · I .

Original: Theorem 1.
Revised: Theorem 1.

Original: Assume ∃ i ∈
Revised: Assume ∃ i ∈

Original: [1 , m ] the vector group of all ˆ s it ∼ D ood
Revised: [1 , m ] the vector group of all ˆ s it ∼ D ood

Original: ( s it ) : { ϕ (ˆ s it , a it ) − ϕ
Revised: ( s it ) : { ϕ (ˆ s it , a it ) − ϕ

Original: ( s it , a it ) } be full rank, then the covariance matrix (cid:101) Λ ood _ diff t is positive-definite: (cid:101) Λ ood _ diff t ⪰ λ
Revised: ( s it , a it ) } be full rank, then the covariance matrix (cid:101) Λ ood _ diff t is positive-definite: (cid:101) Λ ood _ diff t ⪰ λ

Original: I where λ > 0 .
Revised: I where λ > 0 .

Original: Recall the covariance matrix of PBRL is (cid:101) Λ PBRL t
Revised: Recall the covariance matrix of PBRL is (cid:101) Λ PBRL t

Original: = (cid:101) Λ in t + (cid:101) Λ ood t , and RORL has a covariance matrix as (cid:101) Λ t = (cid:101) Λ PBRL t
Revised: = (cid:101) Λ in t + (cid:101) Λ ood t , and RORL has a covariance matrix as (cid:101) Λ t = (cid:101) Λ PBRL t

Original: + (cid:101) Λ ood_diff t , we have the following corollary based on Theorem 1.
Revised: + (cid:101) Λ ood_diff t , we have the following corollary based on Theorem 1.

Original: Corollary 1.
Revised: Corollary 1.

Original: Under the linear MDP assumptions and conditions in Theorem 1, we have (cid:101) Λ t ⪰ (cid:101) Λ PBRL t .
Revised: Under the linear MDP assumptions and conditions in Theorem 1, we have (cid:101) Λ t ⪰ (cid:101) Λ PBRL t .

Original: Further, the covariance matrix (cid:101) Λ t of RORL is positive-definite: (cid:101) Λ t ⪰ λ · I , where λ > 0 .
Revised: Further, the covariance matrix (cid:101) Λ t of RORL is positive-definite: (cid:101) Λ t ⪰ λ · I , where λ > 0 .

Original: Recent theoretical analysis shows that an appropriate uncertainty quantification is essential to provable efficiency in offline RL [23, 63, 6].
Revised: Recent theoretical analysis shows that an appropriate uncertainty quantification is essential to provable efficiency in offline RL [24, 65, 6].

Original: Pessimistic Value Iteration [23] defines a general ξ -uncertainty quantifier as the penalty and achieves provable efficient pessimism in offline RL.
Revised: Pessimistic Value Iteration [24] defines a general ξ -uncertainty quantifier as the penalty and achieves provable efficient pessimism in offline RL.

Original: In linear MDPs,
Revised: In linear MDPs,

Original: Lower Confidence Bound (LCB)-penalty [1, 22] is known to be a ξ -uncertainty quantifier for appropriately selected β t as Γ lcb ( s t , a t )
Revised: Lower Confidence Bound (LCB)-penalty [1, 23] is known to be a ξ -uncertainty quantifier for appropriately selected β t as Γ lcb ( s t , a t )

Original: =
Revised: =

Original: β t · (cid:2) ϕ
Revised: β t · (cid:2) ϕ

Original: ( s t , a t ) ⊤ Λ − 1 t ϕ ( s t , a t ) (cid:3) 1 / 2 .
Revised: ( s t , a t ) ⊤ Λ − 1 t ϕ ( s t , a t ) (cid:3) 1 / 2 .

Original: Following the analysis of PBRL [6], since the bootstrapped uncertainty is an estimation of the LCB-penalty and the OOD sampling provides a covariance matrix (cid:101) Λ t ⪰ λ · I given in Corollary 1, the proposed RORL also forms a valid ξ -uncertainty quantifier.
Revised: Following the analysis of PBRL [6], since the bootstrapped uncertainty is an estimation of the LCB-penalty and the OOD sampling provides a covariance matrix (cid:101) Λ t ⪰ λ · I given in Corollary 1, the proposed RORL also forms a valid ξ -uncertainty quantifier.

Original: This allows us to further characterize the optimality gap based on the pessimistic value iteration [23, 6].
Revised: This allows us to further characterize the optimality gap based on the pessimistic value iteration [24, 6].

Original: We have the following suboptimality gap under linear MDP assumptions.
Revised: We have the following suboptimality gap under linear MDP assumptions.

Original: Corollary 2.
Revised: Corollary 2.

Original: [Equation]
Revised: [Equation]

Original: Detailed proof can be found in Appendix A. Corollary 2 indicates that RORL enjoys a tighter suboptimality bound than PBRL [6].
Revised: Detailed proof can be found in Appendix A. Corollary 2 indicates that RORL enjoys a tighter suboptimality bound than PBRL [6].

Original:  Experiments
Revised:  Experiments

Original: We evaluate our method on the D4RL benchmark [12] with various continuous-control tasks and datasets.
Revised: We evaluate our method on the D4RL benchmark [12] with various continuous-control tasks and datasets.

Original: We compare RORL with several offline RL algorithms, including (i) BC that performs
Revised: We compare RORL with several offline RL algorithms, including (i) BC that performs

Original: [Table]
Revised: [Table]

Original: behavior cloning, (ii) CQL [28] that learns conservative value function for OOD actions, (iii) EDAC [2] that learns a diversified Q -ensemble to enforce conservatism, and (iv) PBRL [6] that performs uncertainty penalization and OOD sampling.
Revised: behavior cloning, (ii) CQL [29] that learns conservative value function for OOD actions, (iii) EDAC [2] that learns a diversified Q -ensemble to enforce conservatism, and (iv) PBRL [6] that performs uncertainty penalization and OOD sampling.

Original: We also include a basic SAC-10 algorithm as a baseline [2], which is an extension of SAC with 10 Q -functions.
Revised: We also include a basic SAC-10 algorithm as a baseline [2], which is an extension of SAC with 10 Q -functions.

Original: Among these methods, EDAC [2] and
Revised: Among these methods, EDAC [2] and

Original: PBRL [6] are related to RORL since all these methods apply Q -ensemble for conservatism.
Revised: PBRL [6] are related to RORL since all these methods apply Q -ensemble for conservatism.

Original: EDAC needs much more Q -networks (i.e., 10 ∼ 50) for hopper tasks than PBRL and RORL that only use 10 Q -networks.
Revised: EDAC needs much more Q -networks (i.e., 10 ∼ 50) for hopper tasks than PBRL and RORL that only use 10 Q -networks.

Original: For fair comparison, we also report the reproduced results of EDAC-10.
Revised: For fair comparison, we also report the reproduced results of EDAC-10.

Original: To assign uniform adversarial attack budget on each dimension of observations, we normalize the observations for SAC-10, EDAC and RORL.
Revised: To assign uniform adversarial attack budget on each dimension of observations, we normalize the observations for SAC-10, EDAC and RORL.

Original: Besides, we use different perturbation scales for the policy smoothing loss, the Q smoothing loss and the OOD loss, namely ϵ P , ϵ Q and ϵ ood .
Revised: Besides, we use different perturbation scales for the policy smoothing loss, the Q smoothing loss and the OOD loss, namely ϵ P , ϵ Q and ϵ ood .

Original: More hyper-parameters and implementation details are provided in Appendix B.
Revised: More hyper-parameters and implementation details are provided in Appendix B.

Original: 6.1 Benchmark Results
Revised: 6.1 Benchmark Results

Original: We evaluate each method on Gym domain that includes three environments (HalfCheetah, Hopper, and
Revised: We evaluate each method on Gym domain that includes three environments (HalfCheetah, Hopper, and

Original: Walker2d) with five types of datasets (random, medium, medium-replay, medium-expert, and expert) for each environment.
Revised: Walker2d) with five types of datasets (random, medium, medium-replay, medium-expert, and expert) for each environment.

Original: The medium-replay dataset contains experiences collected in training a mediumlevel policy.
Revised: The medium-replay dataset contains experiences collected in training a mediumlevel policy.

Original: The random/medium/expert dataset is generated by a single random/medium/expert policy.
Revised: The random/medium/expert dataset is generated by a single random/medium/expert policy.

Original: The medium-expert dataset is a mixture of medium and expert datasets.
Revised: The medium-expert dataset is a mixture of medium and expert datasets.

Original: For benchmark experiments, we set small perturbation scales ϵ P , ϵ Q , and ϵ ood within { 0 .
Revised: For benchmark experiments, we set small perturbation scales ϵ P , ϵ Q , and ϵ ood within { 0 .

Original: 01 } when training RORL and do not include observation perturbation in the testing time.
Revised: 01 } when training RORL and do not include observation perturbation in the testing time.

Original: Table 1 reports the performance of the average normalized score with standard deviation.
Revised: Table 1 reports the performance of the average normalized score with standard deviation.

Original: (i) SACis unstable on several walker2d and hopper tasks since the ensemble number is relatively small to provide reliable uncertainties for SAC- N [2].
Revised: (i) SACis unstable on several walker2d and hopper tasks since the ensemble number is relatively small to provide reliable uncertainties for SAC- N [2].

Original: (ii) EDAC solves this problem by gradient diversity constraints while still requiring 10 ∼ 50 Q -networks to obtain reasonable performance.
Revised: (ii) EDAC solves this problem by gradient diversity constraints while still requiring 10 ∼ 50 Q -networks to obtain reasonable performance.

Original: In contrast,
Revised: In contrast,

Original: RORL only uses 10 ensemble Q -networks to achieve better or comparable performance with EDAC.
Revised: RORL only uses 10 ensemble Q -networks to achieve better or comparable performance with EDAC.

Original: Additionally, we also show that RORL outperforms EDAC-10 by a large margin.
Revised: Additionally, we also show that RORL outperforms EDAC-10 by a large margin.

Original: (iii) PBRL chooses an alternative OOD-sampling technique to reduce the ensemble numbers.
Revised: (iii) PBRL chooses an alternative OOD-sampling technique to reduce the ensemble numbers.

Original: According to the result,
Revised: According to the result,

Original: RORL significantly outperforms PBRL with the same ensemble number.
Revised: RORL significantly outperforms PBRL with the same ensemble number.

Original: The reason is RORL additionally uses conservative smoothing loss for perturbed states and penalizes values of these states based on uncertainty estimation, which may improve the generalization ability of the learned policy on continuous state space.
Revised: The reason is RORL additionally uses conservative smoothing loss for perturbed states and penalizes values of these states based on uncertainty estimation, which may improve the generalization ability of the learned policy on continuous state space.

Original: We remark that RORL significantly improves over the current
Revised: We remark that RORL significantly improves over the current

Original: SOTA results on walker2d and hopper tasks, probably because these two tasks require a more precise balance of conservatism and robustness for better performance.
Revised: SOTA results on walker2d and hopper tasks, probably because these two tasks require a more precise balance of conservatism and robustness for better performance.

Original: 6.2 Adversarial Attack
Revised: 6.2 Adversarial Attack

Original: We adopt three attack methods, namely random , action diff , and min Q following prior works [71, 42].
Revised: We adopt three attack methods, namely random , action diff , and min Q following prior works [73, 43].

Original: Given perturbation scale ϵ , the later two methods perform adversarial perturbation on observations
Revised: Given perturbation scale ϵ , the later two methods perform adversarial perturbation on observations

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: and are given access to the agent’s policy and value functions.
Revised: and are given access to the agent’s policy and value functions.

Original: Details about the three attack methods are as follows.
Revised: Details about the three attack methods are as follows.

Original: • random uniformly samples perturbed states in an l ∞ ball of norm
Revised: • random uniformly samples perturbed states in an l ∞ ball of norm

Original: ϵ . • action diff is an effective attack based on the agent’s policy and is proved to be an upper bound on the performance difference between perturbed and unperturbed environments [71].
Revised: ϵ . • action diff is an effective attack based on the agent’s policy and is proved to be an upper bound on the performance difference between perturbed and unperturbed environments [73].

Original: It directly finds perturbed states in an l ∞ ball of norm ϵ to satisfy:
Revised: It directly finds perturbed states in an l ∞ ball of norm ϵ to satisfy:

Original: [Equation]
Revised: [Equation]

Original: • min Q requires both the agent’s policy and value function to perform a relatively stronger attack.
Revised: • min Q requires both the agent’s policy and value function to perform a relatively stronger attack.

Original: The attacker finds a perturbed state to minimize the expected return of taking an action from that state: min ˆ s ∈ B d
Revised: The attacker finds a perturbed state to minimize the expected return of taking an action from that state: min ˆ s ∈ B d

Original: [Equation]
Revised: [Equation]

Original: s,ϵ ) Q
Revised: s,ϵ ) Q

Original: [Equation]
Revised: [Equation]

Original: s, π
Revised: s, π

Original: [Equation]
Revised: [Equation]

Original: (ˆ s )) .
Revised: (ˆ s )) .

Original: For ensemble-based algorithms, Q is set as the mean of ensemble Q functions.
Revised: For ensemble-based algorithms, Q is set as the mean of ensemble Q functions.

Original: In our experiments, the two objectives of action diff and min Q are optimized via two ways.
Revised: In our experiments, the two objectives of action diff and min Q are optimized via two ways.

Original: Specifically, we optimize the objectives through:
Revised: Specifically, we optimize the objectives through:

Original: (1) selecting the best perturbed state from uniformly sampled 50 states, which has the advantage of simplicity and little computation cost.
Revised: (1) selecting the best perturbed state from uniformly sampled 50 states, which has the advantage of simplicity and little computation cost.

Original: For attacks with this type of optimization, we use their original names without specifying.
Revised: For attacks with this type of optimization, we use their original names without specifying.

Original: (2) uniformly sampling 20 initial states and performing gradient decent for 10 steps with a step size of 110 ϵ from each initial state to find the best perturbed state.
Revised: (2) uniformly sampling 20 initial states and performing gradient decent for 10 steps with a step size of 110 ϵ from each initial state to find the best perturbed state.

Original: Note that we need to clip
Revised: Note that we need to clip

Original: the perturbed states within the l ∞ ball at the end of each optimization step.
Revised: the perturbed states within the l ∞ ball at the end of each optimization step.

Original: Among the attacks using this optimization, we specifically remark "mixed-order" in their names.
Revised: Among the attacks using this optimization, we specifically remark "mixed-order" in their names.

Original: We compare RORL with ensemble-based baselines EDAC and SAC-10 on halfcheetah-medium-v2, walker2d-medium-v2, and hopper-medium-v2 datasets.
Revised: We compare RORL with ensemble-based baselines EDAC and SAC-10 on halfcheetah-medium-v2, walker2d-medium-v2, and hopper-medium-v2 datasets.

Original: To handle large adversarial noise, we set the perturbation scales ϵ P , ϵ Q and ϵ ood within { 0 . 01 , 0 . 03 , 0 . 05 , 0 . 07 } in RORL’s training phase.
Revised: To handle large adversarial noise, we set the perturbation scales ϵ P , ϵ Q and ϵ ood within { 0 . 01 , 0 . 03 , 0 . 05 , 0 . 07 } in RORL’s training phase.

Original: More detailed description can be found in Appendix B.
Revised: More detailed description can be found in Appendix B.

Original: The results are shown in Figure 4.
Revised: The results are shown in Figure 4.

Original: In the results, RORL exhibits improved robustness than other baselines under five types of adversarial attacks.
Revised: In the results, RORL exhibits improved robustness than other baselines under five types of adversarial attacks.

Original: On the other hand, we find that random attack is not effective for ensemble-based offline
Revised: On the other hand, we find that random attack is not effective for ensemble-based offline

Original: RL algorithms, and the “mixed order” attack brings more significant performance drop than vanilla zero-order optimization.
Revised: RL algorithms, and the “mixed order” attack brings more significant performance drop than vanilla zero-order optimization.

Original: [Figure]
Revised: [Figure]

Original: 6.3 Ablations
Revised: 6.3 Ablations

Original: We conduct ablation studies on the walker2d-medium-v2 dataset to evaluate the importance of three terms, i.e., the policy smoothing loss, the Q smoothing term and the OOD loss.
Revised: We conduct ablation studies on the walker2d-medium-v2 dataset to evaluate the importance of three terms, i.e., the policy smoothing loss, the Q smoothing term and the OOD loss.

Original: From the results in
Revised: From the results in

Original: Figure 5, we can conclude that each loss contributes to the performance of RORL under adversarial observation attacks.
Revised: Figure 5, we can conclude that each loss contributes to the performance of RORL under adversarial observation attacks.

Original: The OOD loss is the most essential term, without which the performance is worse than RORL at almost all perturbation scales and all types of attacks.
Revised: The OOD loss is the most essential term, without which the performance is worse than RORL at almost all perturbation scales and all types of attacks.

Original: The policy smoothing loss is also important, especially for perturbation scales larger than 0.2.
Revised: The policy smoothing loss is also important, especially for perturbation scales larger than 0.2.

Original: In addition, Q smooth loss has the minimal impact, which is reasonable since the basic algorithm SAC-10 is based on 10 ensemble Q networks.
Revised: In addition, Q smooth loss has the minimal impact, which is reasonable since the basic algorithm SAC-10 is based on 10 ensemble Q networks.

Original: More ablations on the number of Q networks, the effect of ϵ ood and τ , and a comparison with more baselines can be found in Appendix C.
Revised: More ablations on the number of Q networks, the effect of ϵ ood and τ , and a comparison with more baselines can be found in Appendix C.

Original: 6.4 Computational Cost Comparison
Revised: 6.4 Computational Cost Comparison

Original: [Table]
Revised: [Table]

Original: We compare the computational cost of RORL with prior works on a single machine with one GPU (Tesla V 32G).
Revised: We compare the computational cost of RORL with prior works on a single machine with one GPU (Tesla V 32G).

Original: For each method, we measure the average epoch time (i.e., 1 × 10 3 training steps) and the GPU memory usage on the hopper-medium-v2 task.
Revised: For each method, we measure the average epoch time (i.e., 1 × 10 3 training steps) and the GPU memory usage on the hopper-medium-v2 task.

Original: More discussions are provided in Appendix C.1.
Revised: More discussions are provided in Appendix C.1.

Original: As shown in Table 2, RORL runs slightly faster than CQL and much faster than PBRL.
Revised: As shown in Table 2, RORL runs slightly faster than CQL and much faster than PBRL.

Original: PBRL is so slow because it uses 10 Q networks and needs OOD action sampling.
Revised: PBRL is so slow because it uses 10 Q networks and needs OOD action sampling.

Original: In
Revised: In

Original: RORL, we also include the OOD state-action sampling and the robust training procedure, but we implemented these procedures efficiently based on the parallelization of Q networks.
Revised: RORL, we also include the OOD state-action sampling and the robust training procedure, but we implemented these procedures efficiently based on the parallelization of Q networks.

Original: Even so, RORL is still slower than SAC-10 and EDAC.
Revised: Even so, RORL is still slower than SAC-10 and EDAC.

Original: As demonstrated in our experiments, RORL enjoys significantly better robustness than EDAC and SACunder adversarial perturbations.
Revised: As demonstrated in our experiments, RORL enjoys significantly better robustness than EDAC and SACunder adversarial perturbations.

Original: Regarding the GPU memory consumption, RORL uses comparable memory to PBRL and EDAC, with only 16 .
Revised: Regarding the GPU memory consumption, RORL uses comparable memory to PBRL and EDAC, with only 16 .

Original: 7% more memory usage.
Revised: 7% more memory usage.

Original:  Related Works
Revised:  Related Works

Original: Offline RL Research related to offline RL has experienced explosive growth in recent years.
Revised: Offline RL Research related to offline RL has experienced explosive growth in recent years.

Original: In model-free domain, offline RL methods focus on correcting the extrapolation error [14] in the off- policy algorithms.
Revised: In model-free domain, offline RL methods focus on correcting the extrapolation error [14] in the off- policy algorithms.

Original: The natural idea is to regularize the learned policy near the dataset distribution [57, 61, 36, 59, 67, 13, 64].
Revised: The natural idea is to regularize the learned policy near the dataset distribution [59, 63, 37, 61, 69, 13, 66].

Original: For example, MARVIL reweights the policy with exponential advantage, which implicitly guarantees the policy within the KL-divergence neighborhood of the behavior policy.
Revised: For example, MARVIL reweights the policy with exponential advantage, which implicitly guarantees the policy within the KL-divergence neighborhood of the behavior policy.

Original: Another stream of model-free methods prevents the selection of OOD actions by penalizing their
Revised: Another stream of model-free methods prevents the selection of OOD actions by penalizing their

Original: Q -value [27, 28, 2, 10] or V -learning [32, 26].
Revised: Q -value [28, 29, 2, 10] or V -learning [33, 27].

Original: With the ensemble Q networks and the additional loss term to diversify their gradients, EDAC [2] achieves SOTA performance in the D4RL benchmark.
Revised: With the ensemble Q networks and the additional loss term to diversify their gradients, EDAC [2] achieves SOTA performance in the D4RL benchmark.

Original: Instead of diversifying gradients, PBRL [6] proposes an explicit value underestimation of OOD actions according to the uncertainty, which requires fewer ensemble networks.
Revised: Instead of diversifying gradients, PBRL [6] proposes an explicit value underestimation of OOD actions according to the uncertainty, which requires fewer ensemble networks.

Original: Inspired by EDAC and PBRL, we build our work upon ensemble networks, focusing more on the smoothness over the state space.
Revised: Inspired by EDAC and PBRL, we build our work upon ensemble networks, focusing more on the smoothness over the state space.

Original: Besides the surprising empirical results, theoretical analysis of offline reinforcement learning algorithms is of increasing interest [9, 23, 44, 63, 68].
Revised: Besides the surprising empirical results, theoretical analysis of offline reinforcement learning algorithms is of increasing interest [9, 24, 45, 65, 70].

Original: Though the assumptions for the dataset vary in the different papers, they all suggest that pessimism and conservatism are necessary for offline RL.
Revised: Though the assumptions for the dataset vary in the different papers, they all suggest that pessimism and conservatism are necessary for offline RL.

Original: Our theoretical results can be viewed as robust extensions to previous theoretical results [23, 6].
Revised: Our theoretical results can be viewed as robust extensions to previous theoretical results [24, 6].

Original: Robust RL The research line of robust RL can be traced back to H ∞ -control theory [62, 7], where policies are optimized to be well-performed in the worst possible deterministic environment.
Revised: Robust RL The research line of robust RL can be traced back to H ∞ -control theory [64, 7], where policies are optimized to be well-performed in the worst possible deterministic environment.

Original: Depending on the definition, there are different streams of research on robust RL.
Revised: Depending on the definition, there are different streams of research on robust RL.

Original: As the extension of robust control to MDPs, Robust MDPs (RMDPs) [37, 20, 45, 18] are proposed to formulate the perturbation of transition probabilities for MDPs.
Revised: As the extension of robust control to MDPs, Robust MDPs (RMDPs) [38, 21, 46, 19] are proposed to formulate the perturbation of transition probabilities for MDPs.

Original: Though some recent analyses with theoretical guarantees come out under specific assumptions for RMDPs [73, 66, 30], there is currently no practical algorithm to solve RMDPs in a large-scale problem, expect some linear approximation attempt [52].
Revised: Though some recent analyses with theoretical guarantees come out under specific assumptions for RMDPs [75, 68, 31], there is currently no practical algorithm to solve RMDPs in a large-scale problem, expect some linear approximation attempt [54].

Original: In online RL, domain randomization [54, 34] assumes the model uncertainty can be predefined in data collection by changing the setup of a simulator.
Revised: In online RL, domain randomization [56, 35] assumes the model uncertainty can be predefined in data collection by changing the setup of a simulator.

Original: However, it is not practical for offline RL.
Revised: However, it is not practical for offline RL.

Original: Robust Adversarial Reinforcement Learning (RARL) [42] and Noisy Robust Markov
Revised: Robust Adversarial Reinforcement Learning (RARL) [43] and Noisy Robust Markov

Original: Decision Process (NR-MDP)
Revised: Decision Process (NR-MDP)

Original: [24] study the robust RL with the perturbed actions, showing that the policy robustness to adversarial or noisy actions can also induce robustness for model parameter changes.
Revised: [25] study the robust RL with the perturbed actions, showing that the policy robustness to adversarial or noisy actions can also induce robustness for model parameter changes.

Original: The most related work to ours is SR 2 L [47], which shows policy smoothing can lead to significant performance improvement in the online setting.
Revised: The most related work to ours is SR 2 L [48], which shows policy smoothing can lead to significant performance improvement in the online setting.

Original: In contrast, we focus on the offline setting and tackle the potential overestimation of perturbed states.
Revised: In contrast, we focus on the offline setting and tackle the potential overestimation of perturbed states.

Original: Another related work is S4RL [49], where the authors study different data augmentation methods to smooth observations in offline RL.
Revised: Another related work is S4RL [50], where the authors study different data augmentation methods to smooth observations in offline RL.

Original: Their result supports the necessity of state smoothing.
Revised: Their result supports the necessity of state smoothing.

Original: More related works are discussed in Appendix D.
Revised: More related works are discussed in Appendix E.

Original:  Conclusion
Revised:  Conclusion

Original: We propose Robust Offline Reinforcement Learning (RORL) to trade-off conservatism and robustness for offline RL.
Revised: We propose Robust Offline Reinforcement Learning (RORL) to trade-off conservatism and robustness for offline RL.

Original: To achieve that, we introduce the conservative smoothing technique for the perturbed states while actively underestimating their values based on pessimistic bootstrapping to keep conservative.
Revised: To achieve that, we introduce the conservative smoothing technique for the perturbed states while actively underestimating their values based on pessimistic bootstrapping to keep conservative.

Original: We show that RORL can achieve comparable or even better performance with fewer ensemble
Revised: We show that RORL can achieve comparable or even better performance with fewer ensemble

Original: Q networks than previous methods in the offline RL benchmark.
Revised: Q networks than previous methods in the offline RL benchmark.

Original: In addition, we demonstrate that
Revised: In addition, we demonstrate that

Original: RORL is considerably robust to adversarial perturbations across different types of attacks.
Revised: RORL is considerably robust to adversarial perturbations across different types of attacks.

Original: We hope our work can promote the application of offline RL under real-world engineering conditions.
Revised: We hope our work can promote the application of offline RL under real-world engineering conditions.

Original: The main limitation of our method is that the adversarial state sampling slows down the computing process, which may be improved in future work.
Revised: The main limitation of our method is that the adversarial state sampling slows down the computing process, which may be improved in future work.

Original: Also, an interesting direction is to smooth or penalize the policy and Q functions in latent spaces rather than the normalized observation space.
Revised: Also, an interesting direction is to smooth or penalize the policy and Q functions in latent spaces rather than the normalized observation space.

Original: Acknowledgements
Revised: Acknowledgements

Original: This work was in part supported by Tencent Robotics X and Shanghai AI Laboratory, and in part by
Revised: This work was in part supported by Tencent Robotics X and Shanghai AI Laboratory, and in part by

Original: Science and Technology Innovation 2030 – “New Generation Artificial Intelligence” Major Project (No. 2018AAA0100904) and National Natural Science Foundation of China (62176135).
Revised: Science and Technology Innovation 2030 – “New Generation Artificial Intelligence” Major Project (No. 2018AAA0100904) and National Natural Science Foundation of China (62176135).

Original: The authors would like to thank the anonymous reviewers.
Revised: The authors would like to thank the anonymous reviewers.

Original: Rui Yang thanks Yi Wang and Haoyi Song for valuable discussion.
Revised: Rui Yang thanks Yi Wang and Haoyi Song for valuable discussion.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information on how to answer these questions.
Revised: Please read the checklist guidelines carefully for information on how to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] See Section ?? .
Revised: [Yes] See Section ?? .

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [No] The code and the data are proprietary.
Revised: [No] The code and the data are proprietary.

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] (c) Did you discuss any potential negative societal impacts of your work?
Revised: [Yes] (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] (b) Did you include complete proofs of all theoretical results?
Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: [Yes] See Appendix A. 3.
Revised: [Yes] See Appendix A. 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] See Sec 1 and Appendix B. (b) Did you specify all the training details (e.g., data splits, hyper-parameters, how they were chosen)?
Revised: [Yes] See Sec 1 and Appendix B. (b) Did you specify all the training details (e.g., data splits, hyper-parameters, how they were chosen)?

Original: [Yes] See Appendix B. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [Yes] See Appendix B. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] See Sec 6.
Revised: [Yes] See Sec 6.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Appendix B. 4.
Revised: [Yes] See Appendix B. 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] We cited D4RL [12] and EDAC[2] for their datasets and code.
Revised: [Yes] We cited D4RL [12] and EDAC[2] for their datasets and code.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] We included our code in the anonymized link.
Revised: [Yes] We included our code in the anonymized link.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [Yes] Opensource code and dataset.
Revised: [Yes] Opensource code and dataset.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 5.
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: A Theoretical Analysis
Revised: A Theoretical Analysis

Original: In this section, we provide detailed theoretical analysis and proofs in linear MDPs [22].
Revised: In this section, we provide detailed theoretical analysis and proofs in linear MDPs [23].

Original: A.1 LSVI Solution
Revised: A.1 LSVI Solution

Original: In linear MDPs, we assume that the transition dynamics and reward function take the form of
Revised: In linear MDPs, we assume that the transition dynamics and reward function take the form of

Original: P t ( s t +1 | s t , a t ) = ⟨ ψ ( s t +1 ) , ϕ ( s t , a t ) ⟩ ,
Revised: P t ( s t +1 | s t , a t ) = ⟨ ψ ( s t +1 ) , ϕ ( s t , a t ) ⟩ ,

Original: [Equation]
Revised: [Equation]

Original: ∀
Revised: ∀

Original: [Equation]
Revised: [Equation]

Original: ×
Revised: ×

Original: [Equation]
Revised: [Equation]

Original: × S
Revised: × S

Original: [Equation]
Revised: [Equation]

Original: (10) where the feature embedding ϕ : S × A (cid:55)→ R d is known.
Revised: (10) where the feature embedding ϕ : S × A (cid:55)→ R d is known.

Original: We further assume that the reward function r : S × A (cid:55)→ [0 , 1] is bounded and the feature is bounded by ∥ ϕ ∥ 2 ≤ 1 .
Revised: We further assume that the reward function r : S × A (cid:55)→ [0 , 1] is bounded and the feature is bounded by ∥ ϕ ∥ 2 ≤ 1 .

Original: Given the offline dataset D , the parameter w t can be solved in the closed-form by following the LSVI algorithm, which minimizes the following loss function,
Revised: Given the offline dataset D , the parameter w t can be solved in the closed-form by following the LSVI algorithm, which minimizes the following loss function,

Original: [Equation]
Revised: [Equation]

Original: where V t +1 is the estimated value function in the ( t + 1) -th step, and y i t
Revised: where V t +1 is the estimated value function in the ( t + 1) -th step, and y i t

Original: = r
Revised: = r

Original: ( s i t , a i t )
Revised: ( s i t , a i t )

Original: + V t +

Revised: + V t +

Original: ( s i t +1 ) is the target of LSVI.
Revised: ( s i t +1 ) is the target of LSVI.

Original: The explicit solution to (11) takes the form of
Revised: The explicit solution to (11) takes the form of

Original: [Equation]
Revised: [Equation]

Original: where Λ
Revised: where Λ

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: A.2 RORL Solution
Revised: A.2 RORL Solution

Original: In RORL, since we introduce the conservative smoothing loss and the OOD loss to learn the Q value function, the parameter (cid:101) w t of RORL can be solved as follows:
Revised: In RORL, since we introduce the conservative smoothing loss and the OOD loss to learn the Q value function, the parameter (cid:101) w t of RORL can be solved as follows:

Original: [Equation]
Revised: [Equation]

Original: which is a simplified learning objective for linear MDPs.
Revised: which is a simplified learning objective for linear MDPs.

Original: The first term is the ordinary TD-error, the second term is the Q value smoothing loss
Revised: The first term is the ordinary TD-error, the second term is the Q value smoothing loss

Original: [Equation]
Revised: [Equation]

Original: and the third term is the additional OOD loss.
Revised: and the third term is the additional OOD loss.

Original: The explicit solution of Eq.
Revised: The explicit solution of Eq.

Original: (13) takes the following form by following LSVI:
Revised: (13) takes the following form by following LSVI:

Original: [Equation]
Revised: [Equation]

Original: where the covariance matrix (cid:101) Λ t is defined as
Revised: where the covariance matrix (cid:101) Λ t is defined as

Original: [Equation]
Revised: [Equation]

Original: We denote the first term of Eq.
Revised: We denote the first term of Eq.

Original: (15) as (cid:101) Λ in t , the second term as (cid:101) Λ ood t , and the third term as (cid:101) Λ ood_diff t .
Revised: (15) as (cid:101) Λ in t , the second term as (cid:101) Λ ood t , and the third term as (cid:101) Λ ood_diff t .

Original: A.3 ξ -Uncertainty Quantifier
Revised: A.3 ξ -Uncertainty Quantifier

Original: Theorem (Theorem 1 restate) .
Revised: Theorem (Theorem 1 restate) .

Original: Assume ∃ i ∈ [1 , m ] the vector group of all ˆ s it ∼ D ood
Revised: Assume ∃ i ∈ [1 , m ] the vector group of all ˆ s it ∼ D ood

Original: ( s it ) : { ϕ (ˆ s it , a it ) − ϕ
Revised: ( s it ) : { ϕ (ˆ s it , a it ) − ϕ

Original: ( s it , a it ) } is full rank, then the covariance matrix (cid:101) Λ ood _ diff t is positive-definite: (cid:101) Λ ood _ diff t ⪰ λ
Revised: ( s it , a it ) } is full rank, then the covariance matrix (cid:101) Λ ood _ diff t is positive-definite: (cid:101) Λ ood _ diff t ⪰ λ

Original: · I where λ > 0 .
Revised: · I where λ > 0 .

Original: Proof.
Revised: Proof.

Original: For the (cid:101) Λ ood _ diff t matrix (i.e., the third part in Eq.
Revised: For the (cid:101) Λ ood _ diff t matrix (i.e., the third part in Eq.

Original: (15)), we denote the covariance matrix for a specific i as Φ it .
Revised: (15)), we denote the covariance matrix for a specific i as Φ it .

Original: Then we have (cid:101) Λ ood _ diff t
Revised: Then we have (cid:101) Λ ood _ diff t

Original: =
Revised: =

Original: (cid:80) mi
Revised: (cid:80) mi

Original: =

Revised: =

Original: Φ it .
Revised: Φ it .

Original: In the following, we discuss the condition of positive-definiteness of Φ it .
Revised: In the following, we discuss the condition of positive-definiteness of Φ it .

Original: For the simplicity of notation, we omit the superscript and subscript of s it and a it for given i and t .
Revised: For the simplicity of notation, we omit the superscript and subscript of s it and a it for given i and t .

Original: Specifically, we define
Revised: Specifically, we define

Original: [Equation]
Revised: [Equation]

Original: where j ∈ { 1 , . . .
Revised: where j ∈ { 1 , . . .

Original: , N } indicates we sample
Revised: , N } indicates we sample

Original: | B d ( s it , ϵ ) | = N perturbed states for each s .
Revised: | B d ( s it , ϵ ) | = N perturbed states for each s .

Original: For a nonzero vector y ∈ R d , we have
Revised: For a nonzero vector y ∈ R d , we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows from the observation that (cid:0) ϕ
Revised: where the last inequality follows from the observation that (cid:0) ϕ

Original: (ˆ s j , a ) −
Revised: (ˆ s j , a ) −

Original: [Equation]
Revised: [Equation]

Original: a ) (cid:1) ⊤ y is a scalar.
Revised: a ) (cid:1) ⊤ y is a scalar.

Original: Then Φ it is always positive semi-definite .
Revised: Then Φ it is always positive semi-definite .

Original: In the following, we denote z j = ϕ
Revised: In the following, we denote z j = ϕ

Original: (ˆ s j , a ) − ϕ
Revised: (ˆ s j , a ) − ϕ

Original: ( s, a ) .
Revised: ( s, a ) .

Original: Then we need to prove that the condition to make Φ it positive definite is rank[ z 1 , . . .
Revised: Then we need to prove that the condition to make Φ it positive definite is rank[ z 1 , . . .

Original: , z N ] =
Revised: , z N ] =

Original: d , where d is the feature dimension.
Revised: d , where d is the feature dimension.

Original: Our proof follows contradiction.
Revised: Our proof follows contradiction.

Original: In Eq.
Revised: In Eq.

Original: (16), when y ⊤ Φ it y = 0 with a nonzero vector y , we have z ⊤ j y = 0 for all j = 1 , . . .
Revised: (16), when y ⊤ Φ it y = 0 with a nonzero vector y , we have z ⊤ j y = 0 for all j = 1 , . . .

Original: , N .
Revised: , N .

Original: Suppose the set { z 1 , . .
Revised: Suppose the set { z 1 , . .

Original: , z N } spans R d , then there exist real numbers { α 1 , . .
Revised: , z N } spans R d , then there exist real numbers { α 1 , . .

Original: . , α N } such that y =
Revised: . , α N } such that y =

Original: α 1 z 1 + · · · + α N z N .
Revised: α 1 z 1 + · · · + α N z N .

Original: But we have y ⊤ y
Revised: But we have y ⊤ y

Original: = α 1 z ⊤ 1 y + · · · + α N z ⊤ N y =
Revised: = α 1 z ⊤ 1 y + · · · + α N z ⊤ N y =

Original: α 1 × 0+ . . .
Revised: α 1 × 0+ . . .

Original: + α N × 0 = 0 , yielding that y = 0 , which forms a contradiction.
Revised: + α N × 0 = 0 , yielding that y = 0 , which forms a contradiction.

Original: Hence, if the set { z 1 , . . .
Revised: Hence, if the set { z 1 , . . .

Original: , z N } spans R d , which is equivalent to rank[ z 1 , . . .
Revised: , z N } spans R d , which is equivalent to rank[ z 1 , . . .

Original: , z N ] =
Revised: , z N ] =

Original: d , then Φ it is positive definite .
Revised: d , then Φ it is positive definite .

Original: Under the given conditions, we know that ∃ k ∈ [1 , m ] , for any nonzero vector y ∈ R d , y ⊤ Φ kt y > 0 .
Revised: Under the given conditions, we know that ∃ k ∈ [1 , m ] , for any nonzero vector y ∈ R d , y ⊤ Φ kt y > 0 .

Original: We have y ⊤ (cid:101) Λ ood _ diff t
Revised: We have y ⊤ (cid:101) Λ ood _ diff t

Original: y
Revised: y

Original: =
Revised: =

Original: (cid:80) mi =

Revised: (cid:80) mi =

Original: y ⊤ Φ it y ≥
Revised: y ⊤ Φ it y ≥

Original: y ⊤ Φ kt y > 0 .
Revised: y ⊤ Φ kt y > 0 .

Original: Therefore, (cid:101) Λ ood _ diff t is positive definite, which concludes our proof.
Revised: Therefore, (cid:101) Λ ood _ diff t is positive definite, which concludes our proof.

Original: Remark.
Revised: Remark.

Original: As a special case, when (i) the size of B d ( s it , ϵ ) is sufficient, (ii) the dimension of states is the same as the feature ϕ ( s, a ) and ϕ ( s, a )
Revised: As a special case, when (i) the size of B d ( s it , ϵ ) is sufficient, (ii) the dimension of states is the same as the feature ϕ ( s, a ) and ϕ ( s, a )

Original: =
Revised: =

Original: s and (iii) each dimension of the state perturbation
Revised: s and (iii) each dimension of the state perturbation

Original: ˆ s it − s it is independent, the matrix (cid:101) Λ ood_diff t satisfies: (cid:101) Λ ood_diff t =
Revised: ˆ s it − s it is independent, the matrix (cid:101) Λ ood_diff t satisfies: (cid:101) Λ ood_diff t =

Original: [Equation]
Revised: [Equation]

Original: When we use neural networks as the feature extractor, the assumption in the above Theorem needs (i) the size of samples B d ( s it , ϵ ) is sufficient, and (ii) the neural network maintains useful variability for state-action features.
Revised: When we use neural networks as the feature extractor, the assumption in the above Theorem needs (i) the size of samples B d ( s it , ϵ ) is sufficient, and (ii) the neural network maintains useful variability for state-action features.

Original: To obtain the second constraint, we require that the Jacobian matrix of ϕ ( s, a ) has full rank.
Revised: To obtain the second constraint, we require that the Jacobian matrix of ϕ ( s, a ) has full rank.

Original: Nevertheless, when we use a network as the feature embedding, such a condition can generally be met since the neural network has high randomness and nonlinearity, which results in the feature embedding with sufficient variability.
Revised: Nevertheless, when we use a network as the feature embedding, such a condition can generally be met since the neural network has high randomness and nonlinearity, which results in the feature embedding with sufficient variability.

Original: Generally, we only need to enforce a bi-Lipschitz continuity for the feature embedding.
Revised: Generally, we only need to enforce a bi-Lipschitz continuity for the feature embedding.

Original: We denote x 1 = ( s 1 , a ) and x 2 = ( s 2 , a ) as two different inputs.
Revised: We denote x 1 = ( s 1 , a ) and x 2 = ( s 2 , a ) as two different inputs.

Original: x k 1 is the k -th dimension of x 1 .
Revised: x k 1 is the k -th dimension of x 1 .

Original: The bi-Lipschitz constraint can be formed as
Revised: The bi-Lipschitz constraint can be formed as

Original: [Equation]
Revised: [Equation]

Original: where C 1 < C 2 are two positive constants.
Revised: where C 1 < C 2 are two positive constants.

Original: The lower-bound C 1 ensures the features space has enough variability for perturbed states, and the upper-bound can be obtained by Spectral regularization
Revised: The lower-bound C 1 ensures the features space has enough variability for perturbed states, and the upper-bound can be obtained by Spectral regularization

Original: [16] that makes the network easy to coverage.
Revised: [17] that makes the network easy to coverage.

Original: An approach to obtain bi-Lipschitz continuity is to regularize the norm of the gradients by using the gradient penalty as
Revised: An approach to obtain bi-Lipschitz continuity is to regularize the norm of the gradients by using the gradient penalty as

Original: [Equation]
Revised: [Equation]

Original: In experiments, we do not use explicit constraints (e.g., Spectral regularization) for the upper bound since the state has relatively low dimensions, and we find a small fully connected network does not resulting in a large C 2 empirically.
Revised: In experiments, we do not use explicit constraints (e.g., Spectral regularization) for the upper bound since the state has relatively low dimensions, and we find a small fully connected network does not resulting in a large C 2 empirically.

Original: Recall the covariance matrix of PBRL is (cid:101) Λ PBRL t[Equation] ood
Revised: Recall the covariance matrix of PBRL is (cid:101) Λ PBRL t

Original: [Equation]
Revised: [Equation]

Original: , and RORL has a covariance matrix as
Revised: , and RORL has a covariance matrix as

Original: [Equation]
Revised: [Equation]

Original: , we have the following corollary based on Theorem 1.
Revised: , we have the following corollary based on Theorem 1.

Original: Corollary (Corollary 1 restate) .
Revised: Corollary (Corollary 1 restate) .

Original: Under the linear MDP assumptions and conditions in Theorem 1, we have (cid:101) Λ t ⪰ (cid:101) Λ PBRL t .
Revised: Under the linear MDP assumptions and conditions in Theorem 1, we have (cid:101) Λ t ⪰ (cid:101) Λ PBRL t .

Original: Further, the covariance matrix (cid:101) Λ t of RORL is positive-definite: (cid:101) Λ t ⪰ λ · I , where λ > 0 .
Revised: Further, the covariance matrix (cid:101) Λ t of RORL is positive-definite: (cid:101) Λ t ⪰ λ · I , where λ > 0 .

Original: Recent theoretical analysis shows that an appropriate uncertainty quantification is essential for provable efficiency in offline RL [23, 63, 6].
Revised: Recent theoretical analysis shows that an appropriate uncertainty quantification is essential for provable efficiency in offline RL [24, 65, 6].

Original: Pessimistic Value Iteration [23] defines a general ξ -uncertainty quantifier as the penalty and achieves provable efficient pessimism in offline RL.
Revised: Pessimistic Value Iteration [24] defines a general ξ -uncertainty quantifier as the penalty and achieves provable efficient pessimism in offline RL.

Original: We give the definition of a ξ -uncertainty quantifier as follows.
Revised: We give the definition of a ξ -uncertainty quantifier as follows.

Original: Definition 1 ( ξ -Uncertainty Quantifier [23]) .
Revised: Definition 1 ( ξ -Uncertainty Quantifier [24]) .

Original: The set of penalization { Γ t } t ∈ [ T ] forms a ξ -Uncertainty
Revised: The set of penalization { Γ t } t ∈ [ T ] forms a ξ -Uncertainty

Original: Quantifier if it holds with probability at least 1 − ξ that |
Revised: Quantifier if it holds with probability at least 1 − ξ that |

Original: [Equation]
Revised: [Equation]

Original: for all ( s, a ) ∈ S × A , where T is the Bellman operator and (cid:98) T is the empirical Bellman operator that estimates T based on the data.
Revised: for all ( s, a ) ∈ S × A , where T is the Bellman operator and (cid:98) T is the empirical Bellman operator that estimates T based on the data.

Original: In linear MDPs, Lower Confidence Bound (LCB)-penalty [1, 22] is known to be a ξ -uncertainty quantifier for appropriately selected β t as Γ lcb ( s t , a t ) =
Revised: In linear MDPs, Lower Confidence Bound (LCB)-penalty [1, 23] is known to be a ξ -uncertainty quantifier for appropriately selected β t as Γ lcb ( s t , a t ) =

Original: β t · (cid:2) ϕ
Revised: β t · (cid:2) ϕ

Original: ( s t , a t ) ⊤ Λ − 1 t ϕ ( s t , a t ) (cid:3) 1 / 2 .
Revised: ( s t , a t ) ⊤ Λ − 1 t ϕ ( s t , a t ) (cid:3) 1 / 2 .

Original: Following the analysis of PBRL [6], since the bootstrapped uncertainty is an estimation of the LCB-penalty, the proposed RORL also form a valid ξ -uncertainty quantifier with the covariance matrix (cid:101) Λ t ⪰ λ · I given in Corollary 1.
Revised: Following the analysis of PBRL [6], since the bootstrapped uncertainty is an estimation of the LCB-penalty, the proposed RORL also form a valid ξ -uncertainty quantifier with the covariance matrix (cid:101) Λ t ⪰ λ · I given in Corollary 1.

Original: Theorem 2.
Revised: Theorem 2.

Original: For all the OOD datapoint (ˆ s, ˆ a, ˆ y ) ∈
Revised: For all the OOD datapoint (ˆ s, ˆ a, ˆ y ) ∈

Original: D ood , if we set ˆ y = T V t +1 ( s ood , a ood ) , it then holds for β t
Revised: D ood , if we set ˆ y = T V t +1 ( s ood , a ood ) , it then holds for β t

Original: = O (cid:0) T · √ d · log ( T/ξ ) (cid:1)
Revised: = O (cid:0) T · √ d · log ( T/ξ ) (cid:1)

Original: that
Revised: that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: forms a valid ξ -uncertainty quantifier, where (cid:101) Λ t is the covariance matrix of RORL.
Revised: forms a valid ξ -uncertainty quantifier, where (cid:101) Λ t is the covariance matrix of RORL.

Original: Proof.
Revised: Proof.

Original: The proof follows that of the analysis of PBRL [6] in linear MDPs [23].
Revised: The proof follows that of the analysis of PBRL [6] in linear MDPs [24].

Original: We define the empirical Bellman operator of RORL as (cid:101) T , then
Revised: We define the empirical Bellman operator of RORL as (cid:101) T , then

Original: [Equation]
Revised: [Equation]

Original: where (cid:101) w t follows the solution in Eq.
Revised: where (cid:101) w t follows the solution in Eq.

Original: Then it suffices to upper bound the following difference between the empirical Bellman operator and Bellman operator
Revised: Then it suffices to upper bound the following difference between the empirical Bellman operator and Bellman operator

Original: [Equation]
Revised: [Equation]

Original: Here we define w t as follows
Revised: Here we define w t as follows

Original: [Equation]
Revised: [Equation]

Original: where θ and ψ are defined in Eq.
Revised: where θ and ψ are defined in Eq.

Original: It then holds that
Revised: It then holds that

Original: [Equation]
Revised: [Equation]

Original: where we plug the solution of (cid:101) w t in Eq.
Revised: where we plug the solution of (cid:101) w t in Eq.

Original: Meanwhile, by the definitions of (cid:101) Λ t and w t in Eq.
Revised: Meanwhile, by the definitions of (cid:101) Λ t and w t in Eq.

Original: (15) and Eq.
Revised: (15) and Eq.

Original: (19), respectively, we have
Revised: (19), respectively, we have

Original: [Equation]
Revised: [Equation]

Original: Plugging Eq.
Revised: Plugging Eq.

Original: (21) into Eq.
Revised: (21) into Eq.

Original: (20) yields
Revised: (20) yields

Original: [Equation]
Revised: [Equation]

Original: where we define
Revised: where we define

Original: [Equation]
Revised: [Equation]

Original: Following the standard analysis based on the concentration of self-normalized process [1, 3, 58, 22, 23] and the fact that Λ ood ⪰ λ · I , it holds that
Revised: Following the standard analysis based on the concentration of self-normalized process [1, 3, 60, 23, 24] and the fact that Λ ood ⪰ λ · I , it holds that

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − ξ , where β t =
Revised: with probability at least 1 − ξ , where β t =

Original: O (cid:0) T · √ d · log ( T/ξ ) (cid:1) .
Revised: O (cid:0) T · √ d · log ( T/ξ ) (cid:1) .

Original: Meanwhile, by setting y =
Revised: Meanwhile, by setting y =

Original: T V t +1 ( s ood , a ood ) , it holds that (ii) = 0 .
Revised: T V t +1 ( s ood , a ood ) , it holds that (ii) = 0 .

Original: For (iii), we have (cid:16) ϕ
Revised: For (iii), we have (cid:16) ϕ

Original: (ˆ s it , a it ) ϕ
Revised: (ˆ s it , a it ) ϕ

Original: [Equation]
Revised: [Equation]

Original: ˆ s it
Revised: ˆ s it

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: = ϕ
Revised: = ϕ

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: Since we enforce smoothness for the value function, we have T V t +

Revised: Since we enforce smoothness for the value function, we have T V t +

Original: (ˆ s it , a it )
Revised: (ˆ s it , a it )

Original: ≈ T V t +1 ( s it , a it ) .
Revised: ≈ T V t +1 ( s it , a it ) .

Original: Thus (iii) ≈ 0 .
Revised: Thus (iii) ≈ 0 .

Original: To conclude, we obtain from Eq.
Revised: To conclude, we obtain from Eq.

Original: (22) that
Revised: (22) that

Original: [Equation]
Revised: [Equation]

Original: for all ( s, a ) ∈ S × A with probability at least 1 − ξ .
Revised: for all ( s, a ) ∈ S × A with probability at least 1 − ξ .

Original: A.4 Suboptimality Gap
Revised: A.4 Suboptimality Gap

Original: Theorem 2 allows us to further characterize the optimality gap based on the pessimistic value iteration [23].
Revised: Theorem 2 allows us to further characterize the optimality gap based on the pessimistic value iteration [24].

Original: First, we give the following lemma.
Revised: First, we give the following lemma.

Original: Lemma 1.
Revised: Lemma 1.

Original: Given two positive definite matrix A and B, it holds that:
Revised: Given two positive definite matrix A and B, it holds that:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Proof.
Revised: Proof.

Original: Leveraging the properties of generalized Rayleigh quotient, we have
Revised: Leveraging the properties of generalized Rayleigh quotient, we have

Original: [Equation]
Revised: [Equation]

Original: Since B and A − 1 are both positive definite, the eigenvalues of BA − 1 are all positive: λ min (cid:0) BA − 1 (cid:1) > 0 .
Revised: Since B and A − 1 are both positive definite, the eigenvalues of BA − 1 are all positive: λ min (cid:0) BA − 1 (cid:1) > 0 .

Original: This ends the proof.
Revised: This ends the proof.

Original: Then, according to the definition of LCB-penalty in Eq.
Revised: Then, according to the definition of LCB-penalty in Eq.

Original: (18), since (cid:101) Λ t = (cid:101) Λ PBRL t
Revised: (18), since (cid:101) Λ t = (cid:101) Λ PBRL t

Original: + (cid:101) Λ ood_diff t with (cid:101) Λ ood_diff t ⪰ λ
Revised: + (cid:101) Λ ood_diff t with (cid:101) Λ ood_diff t ⪰ λ

Original: I .
Revised: I .

Original: we have the relationship of the LCB-penalty between RORL and PBRL as follows.
Revised: we have the relationship of the LCB-penalty between RORL and PBRL as follows.

Original: Corollary 3.
Revised: Corollary 3.

Original: Suppose Λ PBRL t is positive definite.
Revised: Suppose Λ PBRL t is positive definite.

Original: The RORL-induced LCB-penalty term is less than the
Revised: The RORL-induced LCB-penalty term is less than the

Original: PBRL-induced LCB-penalty, as Γ lcb t ( s t , a t ) =
Revised: PBRL-induced LCB-penalty, as Γ lcb t ( s t , a t ) =

Original: β
Revised: β

Original: [Equation]
Revised: [Equation]

Original: (cid:3)
Revised: (cid:3)

Original: [Equation]
Revised: [Equation]

Original: <
Revised: <

Original: [Equation]
Revised: [Equation]

Original: lcb _ PBRL
Revised: lcb _ PBRL

Original: [Equation]
Revised: [Equation]

Original: ( s
Revised: ( s

Original: [Equation]
Revised: [Equation]

Original: , a
Revised: , a

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Since (cid:101) Λ t = (cid:101) Λ PBRL t
Revised: Since (cid:101) Λ t = (cid:101) Λ PBRL t

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: t and (cid:101) Λ ood_diff t ⪰ λI , we have ϕ
Revised: t and (cid:101) Λ ood_diff t ⪰ λI , we have ϕ

Original: [Equation]
Revised: [Equation]

Original: where the inequality directly follows Lemma 1.
Revised: where the inequality directly follows Lemma 1.

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: Theorem 2 and Corollary 3 allow us to further characterize the optimality gap of the pessimistic value iteration.
Revised: Theorem 2 and Corollary 3 allow us to further characterize the optimality gap of the pessimistic value iteration.

Original: In particular, we have the following suboptimality gap under linear MDP assumptions.
Revised: In particular, we have the following suboptimality gap under linear MDP assumptions.

Original: Corollary (Corollary 2 restate) .
Revised: Corollary (Corollary 2 restate) .

Original: Under the same conditions as Theorem 2, it holds that
Revised: Under the same conditions as Theorem 2, it holds that

Original: SubOpt
Revised: SubOpt

Original: [Equation]
Revised: [Equation]

Original: We refer to Jin et al [23] for a detailed proof of the first inequality.
Revised: We refer to Jin et al [24] for a detailed proof of the first inequality.

Original: The second inequality is directly induced by Γ lcb t ( s t , a t ) <
Revised: The second inequality is directly induced by Γ lcb t ( s t , a t ) <

Original: Γ lcb _ PBRL t
Revised: Γ lcb _ PBRL t

Original: ( s t , a t ) in Corollary 3.
Revised: ( s t , a t ) in Corollary 3.

Original: The optimality gap is informationtheoretically optimal under the linear MDP setup with finite horizon [23].
Revised: The optimality gap is informationtheoretically optimal under the linear MDP setup with finite horizon [24].

Original: Therefore, RORL enjoys a tighter suboptimality bound than PBRL [6] in linear MDPs.
Revised: Therefore, RORL enjoys a tighter suboptimality bound than PBRL [6] in linear MDPs.

Original: B Implementation Details and Experimental Settings
Revised: B Implementation Details and Experimental Settings

Original: In this section, we provide detailed implementation and experimental settings.
Revised: In this section, we provide detailed implementation and experimental settings.

Original: B.1 Implementation Details
Revised: B.1 Implementation Details

Original: SAC-10 Our SAC-10 implementation is based on [2], which is open-source.
Revised: SAC-10 Our SAC-10 implementation is based on [2], which is open-source.

Original: We keep the default parameters as EDAC [2] except for the ensemble size set to 10 in our paper.
Revised: We keep the default parameters as EDAC [2] except for the ensemble size set to 10 in our paper.

Original: In addition, we normalize each dimension of observations to a standard normal distribution for consistency with RORL.
Revised: In addition, we normalize each dimension of observations to a standard normal distribution for consistency with RORL.

Original: The hyper-parameters are listed in Table 3.
Revised: The hyper-parameters are listed in Table 3.

Original: [Table]
Revised: [Table]

Original: EDAC Our EDAC implementation is based on the open-source code of the original paper [2].
Revised: EDAC Our EDAC implementation is based on the open-source code of the original paper [2].

Original: In the benchmark results, we directly report results from the paper which are the previous SOTA performance on the D4RL Mujoco benchmark.
Revised: In the benchmark results, we directly report results from the paper which are the previous SOTA performance on the D4RL Mujoco benchmark.

Original: As for other experiments, we also normalize the observations and use 10 ensemble Q networks for consistency with RORL, and set the gradient diversity term η = 1 by default.
Revised: As for other experiments, we also normalize the observations and use 10 ensemble Q networks for consistency with RORL, and set the gradient diversity term η = 1 by default.

Original: RORL We implement RORL based on SAC-10 and keep the hyper-parameters the same.
Revised: RORL We implement RORL based on SAC-10 and keep the hyper-parameters the same.

Original: The differences are the introduced policy and Q network smoothing techniques and the additional value underestimation on OOD state-action pairs.
Revised: The differences are the introduced policy and Q network smoothing techniques and the additional value underestimation on OOD state-action pairs.

Original: In Eq.
Revised: In Eq.

Original: (5), the coefficient β Q for the Q network smoothing loss L smooth is set to 0 .
Revised: (5), the coefficient β Q for the Q network smoothing loss L smooth is set to 0 .

Original: 0001 for all tasks, and the coefficient β ood for the OOD loss L ood is tuned within { 0 . 0 , 0 . 1 , 0 . 5 } .
Revised: 0001 for all tasks, and the coefficient β ood for the OOD loss L ood is tuned within { 0 . 0 , 0 . 1 , 0 . 5 } .

Original: Besides, the coefficient β P of the policy smoothing loss in Eq.
Revised: Besides, the coefficient β P of the policy smoothing loss in Eq.

Original: (6) is searched in { 0 . 1 , 1 . 0 } .
Revised: (6) is searched in { 0 . 1 , 1 . 0 } .

Original: When training the policy and value functions in RORL, we randomly sample n perturbed observations from a l ∞ ball of norm ϵ and select the one that maximizes D J ( π θ ( ·| s ) ∥
Revised: When training the policy and value functions in RORL, we randomly sample n perturbed observations from a l ∞ ball of norm ϵ and select the one that maximizes D J ( π θ ( ·| s ) ∥

Original: π θ ( ·| ˆ s )) or
Revised: π θ ( ·| ˆ s )) or

Original: L smooth , respectively.
Revised: L smooth , respectively.

Original: We denote the perturbation scales for the Q value functions, the policy, and the OOD loss as ϵ Q , ϵ P and ϵ ood .
Revised: We denote the perturbation scales for the Q value functions, the policy, and the OOD loss as ϵ Q , ϵ P and ϵ ood .

Original: The number of sampled perturbed observations n is tuned within { 10 , 20 } .
Revised: The number of sampled perturbed observations n is tuned within { 10 , 20 } .

Original: The OOD loss underestimates the values for n perturbed states
Revised: The OOD loss underestimates the values for n perturbed states

Original: ˆ s ∼ B d ( s, ϵ ) with actions sampled from the current policy ˆ a ∼ π θ (ˆ s ) .
Revised: ˆ s ∼ B d ( s, ϵ ) with actions sampled from the current policy ˆ a ∼ π θ (ˆ s ) .

Original: For each ˆ s , we sample a single ˆ a for the OOD loss.
Revised: For each ˆ s , we sample a single ˆ a for the OOD loss.

Original: Regarding the Q smoothing loss in Eq.
Revised: Regarding the Q smoothing loss in Eq.

Original: (3), the parameter τ is set to 0 .
Revised: (3), the parameter τ is set to 0 .

Original: 2 in all tasks for conservative value estimation.
Revised: 2 in all tasks for conservative value estimation.

Original: All the hyper-parameters used in RORL for the benchmark experiments and adversarial experiments are listed in Table 4 and Table 5 respectively.
Revised: All the hyper-parameters used in RORL for the benchmark experiments and adversarial experiments are listed in Table 4 and Table 5 respectively.

Original: Note that for halfcheetah tasks, 10 ensemble Q networks already enforce sufficient pessimism for OOD state-action pairs, thus we do not need additional OOD loss for these tasks.
Revised: Note that for halfcheetah tasks, 10 ensemble Q networks already enforce sufficient pessimism for OOD state-action pairs, thus we do not need additional OOD loss for these tasks.

Original: As for the OOD loss L ood in Eq.
Revised: As for the OOD loss L ood in Eq.

Original: (4), we remark that the pseudo-target (cid:98) T ood
Revised: (4), we remark that the pseudo-target (cid:98) T ood

Original: Q ϕ
Revised: Q ϕ

Original: i (ˆ s, ˆ a ) for the OOD state-action pairs (ˆ s, ˆ a ) can be implemented in two ways: (cid:98) T ood Q ϕ
Revised: i (ˆ s, ˆ a ) for the OOD state-action pairs (ˆ s, ˆ a ) can be implemented in two ways: (cid:98) T ood Q ϕ

Original: i (ˆ s, ˆ a ) :=
Revised: i (ˆ s, ˆ a ) :=

Original: Q ϕ i (ˆ s, ˆ a ) − λu (ˆ s, ˆ a ) and (cid:98) T ood Q ϕ
Revised: Q ϕ i (ˆ s, ˆ a ) − λu (ˆ s, ˆ a ) and (cid:98) T ood Q ϕ

Original: i (ˆ s, ˆ a ) :
Revised: i (ˆ s, ˆ a ) :

Original: = min i =1 ,...,K Q ϕ
Revised: = min i =1 ,...,K Q ϕ

Original: i (ˆ s, ˆ a ) .
Revised: i (ˆ s, ˆ a ) .

Original: We refer to the two targets as the “minus target” and the “min target”, and compare them in Appendix C.14.
Revised: We refer to the two targets as the “minus target” and the “min target”, and compare them in Appendix C.14.

Original: Intuitively, the “minus target” introduces an additional parameter λ
Revised: Intuitively, the “minus target” introduces an additional parameter λ

Original: but is more flexible to tune for different environments and different types of data.
Revised: but is more flexible to tune for different environments and different types of data.

Original: In contrast, the “min target” requires tuning the number of ensemble Q networks and cannot enforce appropriate conservatism for all tasks given only 10 ensemble Q networks.
Revised: In contrast, the “min target” requires tuning the number of ensemble Q networks and cannot enforce appropriate conservatism for all tasks given only 10 ensemble Q networks.

Original: Following PBRL [6]
Revised: Following PBRL [6]

Original: [Table]
Revised: [Table]

Original: we also decay the OOD regularization coefficient λ with decay pace d for each training step to stabilize L ood , because we need strong OOD regularization at the beginning of training and need to avoid too large OOD loss that leads the value function to be fully negative.
Revised: we also decay the OOD regularization coefficient λ with decay pace d for each training step to stabilize L ood , because we need strong OOD regularization at the beginning of training and need to avoid too large OOD loss that leads the value function to be fully negative.

Original: λ
Revised: λ

Original: and d are also listed in the two tables
Revised: and d are also listed in the two tables

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: B.2 Experimental Settings
Revised: B.2 Experimental Settings

Original: For all experiments, we train algorithms for 3000 epochs (1000 training steps per epoch, i.e., 3 million steps in total) following EDAC [2].
Revised: For all experiments, we train algorithms for 3000 epochs (1000 training steps per epoch, i.e., 3 million steps in total) following EDAC [2].

Original: We use small perturbation scales to train the Q networks and the policy network for the benchmark experiments and relatively large scales for the adversarial attack experiments as listed in Table 4 and Table 5.
Revised: We use small perturbation scales to train the Q networks and the policy network for the benchmark experiments and relatively large scales for the adversarial attack experiments as listed in Table 4 and Table 5.

Original: In the benchmark results, we evaluate algorithms for 1000 steps in clean environments (without adversarial attack) at the end of each epoch.
Revised: In the benchmark results, we evaluate algorithms for 1000 steps in clean environments (without adversarial attack) at the end of each epoch.

Original: The reported results are normalized to d4rl scores that measure how the performance compared with the expert score and the random score: thenormalized score = 100 ×
Revised: The reported results are normalized to d4rl scores that measure how the performance compared with the expert score and the random score: thenormalized score = 100 ×

Original: [Equation]
Revised: [Equation]

Original: score
Revised: score

Original: [Equation]
Revised: [Equation]

Original: random score .
Revised: random score .

Original: Besides, the benchmark results are averaged over 4 random seeds.
Revised: Besides, the benchmark results are averaged over 4 random seeds.

Original: Regarding the adversarial attack experiments, we evaluate algorithms in perturbed environments that performing “random”, “action diff”, and “min Q” attack with zerothorder and mixed-order optimizations as discussed in Section 6.2.
Revised: Regarding the adversarial attack experiments, we evaluate algorithms in perturbed environments that performing “random”, “action diff”, and “min Q” attack with zerothorder and mixed-order optimizations as discussed in Section 6.2.

Original: Similar to prior work [71], agents receive observations with malicious noise and the environments do not change their internal transition dynamics.
Revised: Similar to prior work [73], agents receive observations with malicious noise and the environments do not change their internal transition dynamics.

Original: We evaluate each algorithm for 10 trajectories (1000 steps per trajectory) and average their returns over 4 random seeds.
Revised: We evaluate each algorithm for 10 trajectories (1000 steps per trajectory) and average their returns over 4 random seeds.

Original: B.3 Visualization Settings of CQL
Revised: B.3 Visualization Settings of CQL

Original: For visualizing the relationship between the Q -function and the state space (i.e., Figure 2 and Figure 6), we sample 2560 adversarial transitions from the offline dataset for each attack ϵ and calculate the corresponding Q -function.
Revised: For visualizing the relationship between the Q -function and the state space (i.e., Figure 2 and Figure 6), we sample 2560 adversarial transitions from the offline dataset for each attack ϵ and calculate the corresponding Q -function.

Original: Since the state has relatively high dimensions (i.e., 11 or 17), we perform
Revised: Since the state has relatively high dimensions (i.e., 11 or 17), we perform

Original: PCA dimensional reduction to reduce the state to 4 dimensions.
Revised: PCA dimensional reduction to reduce the state to 4 dimensions.

Original: We find the Q -function generally has a strong correlation to one or two dimensions of the state after dimensional reduction.
Revised: We find the Q -function generally has a strong correlation to one or two dimensions of the state after dimensional reduction.

Original: For other dimensions, the relationship between the Q -value and the PCA-reduced state often has one or two peaks, which has less variety in the curve.
Revised: For other dimensions, the relationship between the Q -value and the PCA-reduced state often has one or two peaks, which has less variety in the curve.

Original: [Figure]
Revised: [Figure]

Original: C Additional Experimental Results
Revised: C Additional Experimental Results

Original: In this section
Revised: In this section

Original: [Figure]
Revised: [Figure]

Original: we present additional ablation studies and adversarial experiments.
Revised: we present additional ablation studies and adversarial experiments.

Original: C.1 Computational Cost Comparison
Revised: C.1 Computational Cost Comparison

Original: In this subsection
Revised: In this subsection

Original: [Figure]
Revised: [Figure]

Original: we compare the computational cost of RORL with prior works on a single machine with one GPU (Tesla V100 32G) and one CPU (Intel Xeon Platinum 8255C @ 2.50GHz).
Revised: we compare the computational cost of RORL with prior works on a single machine with one GPU (Tesla V100 32G) and one CPU (Intel Xeon Platinum 8255C @ 2.50GHz).

Original: For each method, we measure the average epoch time (i
Revised: For each method, we measure the average epoch time (i

Original: [Figure]
Revised: [Figure]

Original: e., 1 × 10 3 training steps) and the GPU memory usage on the hopper-medium-v2 task.
Revised: e., 1 × 10 3 training steps) and the GPU memory usage on the hopper-medium-v2 task.

Original: For CQL, PBRL, SAC- N , and EDAC, we evaluate the computational cost based on their official code.
Revised: For CQL, PBRL, SAC- N , and EDAC, we evaluate the computational cost based on their official code.

Original: As shown in Table
Revised: As shown in Table

Original: [Figure]
Revised: [Figure]

Original: RORL runs slightly faster than CQL, mainly because CQL needs the OOD action sampling and the logsumexp approximation.
Revised: RORL runs slightly faster than CQL, mainly because CQL needs the OOD action sampling and the logsumexp approximation.

Original: For ensemble-based baselines, RORL runs
Revised: For ensemble-based baselines, RORL runs

Original: [Figure]
Revised: [Figure]

Original: much faster than PBRL, requiring only 28.7 % of PBRL’s epoch time.
Revised: much faster than PBRL, requiring only 28.7 % of PBRL’s epoch time.

Original: PBRL is so slow because it uses 10 ensemble Q networks for uncertainty measure and needs OOD action sampling for value underestimation.
Revised: PBRL is so slow because it uses 10 ensemble Q networks for uncertainty measure and needs OOD action sampling for value underestimation.

Original: In RORL, we also include the OOD state-action sampling and additional adversarial training procedures, but we implement these procedures efficiently based on GPU operation and parallelism.
Revised: In RORL, we also include the OOD state-action sampling and additional adversarial training procedures, but we implement these procedures efficiently based on GPU operation and parallelism.

Original: Even so, RORL is still slower than SAC-10 and EDAC.
Revised: Even so, RORL is still slower than SAC-10 and EDAC.

Original: But as demonstrated in our experiments, RORL enjoys significantly better robustness than EDAC and SAC-10 under different types of perturbations.
Revised: But as demonstrated in our experiments, RORL enjoys significantly better robustness than EDAC and SAC-10 under different types of perturbations.

Original: As for the GPU memory consumption, RORL uses comparable memory to
Revised: As for the GPU memory consumption, RORL uses comparable memory to

Original: PBRL and EDAC, with only 16 .
Revised: PBRL and EDAC, with only 16 .

Original: 7% more memory usage.
Revised: 7% more memory usage.

Original: Furthermore, we analyze the computational cost of RORL’s components ( Q smoothing, policy smoothing, and the OOD loss).
Revised: Furthermore, we analyze the computational cost of RORL’s components ( Q smoothing, policy smoothing, and the OOD loss).

Original: Specifically, we measure the average epoch time of SAC- 10 +Policy
Revised: Specifically, we measure the average epoch time of SAC- 10 +Policy

Original: Smooth , SAC- 10 +Q Smooth , SAC- 10 +OOD Loss in Figure 7(a), and calculate the corresponding memory usage of each component in Figure 7(b).
Revised: Smooth , SAC- 10 +Q Smooth , SAC- 10 +OOD Loss in Figure 7(a), and calculate the corresponding memory usage of each component in Figure 7(b).

Original: For the training time, SAC- 10 +Q Smooth runs the slowest and SAC- 10 +Policy Smooth runs slightly slower than SAC- 10 +OOD Loss .
Revised: For the training time, SAC- 10 +Q Smooth runs the slowest and SAC- 10 +Policy Smooth runs slightly slower than SAC- 10 +OOD Loss .

Original: This is mainly because sampling the worst-case perturbation occupies the most time.
Revised: This is mainly because sampling the worst-case perturbation occupies the most time.

Original: In addition, since we use an ensemble of 10 Q networks, the memory usage of the Q smoothing loss and the OOD loss (both need to pass n perturbed states to 10 Q networks) is larger than the policy smoothing loss
Revised: In addition, since we use an ensemble of 10 Q networks, the memory usage of the Q smoothing loss and the OOD loss (both need to pass n perturbed states to 10 Q networks) is larger than the policy smoothing loss

Original: [Figure]
Revised: [Figure]

Original: C.2 Ablations on Benchmark Results
Revised: C.2 Ablations on Benchmark Results

Original: In the benchmark experiments
Revised: In the benchmark experiments

Original: [Figure]
Revised: [Figure]

Original: RORL outperforms other baselines, especially in walker2d tasks.
Revised: RORL outperforms other baselines, especially in walker2d tasks.

Original: We conduct ablation studies on this task to verify the effectiveness of RORL’s components.
Revised: We conduct ablation studies on this task to verify the effectiveness of RORL’s components.

Original: In Figure
Revised: In Figure

Original: (a), we can find that each introduced loss (i.e., the OOD loss, the policy smoothing loss and the Q smoothing loss) influences the performance on the walker2d-medium-v2 task.
Revised: (a), we can find that each introduced loss (i.e., the OOD loss, the policy smoothing loss and the Q smoothing loss) influences the performance on the walker2d-medium-v2 task.

Original: Specifically, the OOD loss affects the most, without which the performance would drop close to SAC-N’s performance.
Revised: Specifically, the OOD loss affects the most, without which the performance would drop close to SAC-N’s performance.

Original: In addition, the Q smoothing loss is helpful for stabilizing the training and final performance in clean environments.
Revised: In addition, the Q smoothing loss is helpful for stabilizing the training and final performance in clean environments.

Original: In Figure 8 (b), we evaluate the performance of RORL with varying τ .
Revised: In Figure 8 (b), we evaluate the performance of RORL with varying τ .

Original: The results suggest that τ is an important factor that balances the learning of in-distribution and out-of-distribution Q values.
Revised: The results suggest that τ is an important factor that balances the learning of in-distribution and out-of-distribution Q values.

Original: In Eq.
Revised: In Eq.

Original: (3), we want to assign larger weights ( 1 − τ ) on the δ ( s, ˆ s, a ) 2+ and smaller weights ( τ ) on the δ ( s, ˆ s, a ) 2 − to underestimate the values of OOD states, where δ ( s, ˆ s, a ) =
Revised: (3), we want to assign larger weights ( 1 − τ ) on the δ ( s, ˆ s, a ) 2+ and smaller weights ( τ ) on the δ ( s, ˆ s, a ) 2 − to underestimate the values of OOD states, where δ ( s, ˆ s, a ) =

Original: Q ϕ i (ˆ s, a ) − Q ϕ
Revised: Q ϕ i (ˆ s, a ) − Q ϕ

Original: i ( s, a ) .
Revised: i ( s, a ) .

Original: On the contrary, a too small τ can also lead to overestimation of in-distribution state-action pairs.
Revised: On the contrary, a too small τ can also lead to overestimation of in-distribution state-action pairs.

Original: In
Revised: In

Original: Figure 8 (b), τ = 0 leads to poor performance while larger τ = 0 .
Revised: Figure 8 (b), τ = 0 leads to poor performance while larger τ = 0 .

Original: also result in performance worse than RORL without Q smoothing.
Revised: also result in performance worse than RORL without Q smoothing.

Original: Empirically, we find τ = 0 .
Revised: Empirically, we find τ = 0 .

Original: 2 works well across different tasks and set τ = 0 .
Revised: 2 works well across different tasks and set τ = 0 .

Original: 2 by default for all experiments.
Revised: 2 by default for all experiments.

Original: In the above analysis, we know that the OOD loss is a key component in RORL.
Revised: In the above analysis, we know that the OOD loss is a key component in RORL.

Original: We further study the impact of the OOD loss and ϵ ood on the performance and the value estimation.
Revised: We further study the impact of the OOD loss and ϵ ood on the performance and the value estimation.

Original: As shown in Figure(a), when ϵ ood = 0 , the performance of RORL drops significantly, which illustrates the effectiveness of underestimating values of OOD states since the smoothness of RORL may overestimate these values.
Revised: As shown in Figure(a), when ϵ ood = 0 , the performance of RORL drops significantly, which illustrates the effectiveness of underestimating values of OOD states since the smoothness of RORL may overestimate these values.

Original: From Figure 9 (b), we can verify that the OOD loss with ϵ ood > 0 contributes to the value underestimation.
Revised: From Figure 9 (b), we can verify that the OOD loss with ϵ ood > 0 contributes to the value underestimation.

Original: [Figure]
Revised: [Figure]

Original: C.3 Robustness Measures
Revised: C.3 Robustness Measures

Original: In prior works [47, 71], the authors only demonstrate the robustness of algorithms via comparing the return curves with different attack scales.
Revised: In prior works [48, 73], the authors only demonstrate the robustness of algorithms via comparing the return curves with different attack scales.

Original: To better measure the robustness of RL algorithms, we consider the robust score as the areas under the perturbation curve in Figure 4.
Revised: To better measure the robustness of RL algorithms, we consider the robust score as the areas under the perturbation curve in Figure 4.

Original: Since the returns in the figure have been normalized as introduced in Appendix B.2, we can simply calculate the robust score for each attack strategy as:
Revised: Since the returns in the figure have been normalized as introduced in Appendix B.2, we can simply calculate the robust score for each attack strategy as:

Original: [Equation]
Revised: [Equation]

Original: where Rs is the list of returns under N monotonically increasing attack scales.
Revised: where Rs is the list of returns under N monotonically increasing attack scales.

Original: The introduced robust score treats different attack scales equally.
Revised: The introduced robust score treats different attack scales equally.

Original: However, in many real scenarios, we would pay more attention to larger-scale disturbances.
Revised: However, in many real scenarios, we would pay more attention to larger-scale disturbances.

Original: To this end, we also define a weighted robust score as:
Revised: To this end, we also define a weighted robust score as:

Original: [Equation]
Revised: [Equation]

Original: where the weights are assigned according to the scale order.
Revised: where the weights are assigned according to the scale order.

Original: In Table 6 and Table 7, RORL consistently outperforms EDAC and SAC- 10 on the two robustness metrics.
Revised: In Table 6 and Table 7, RORL consistently outperforms EDAC and SAC- 10 on the two robustness metrics.

Original: For walker2d and hopper tasks,
Revised: For walker2d and hopper tasks,

Original: [Table] [Table]
Revised: [Table] [Table]

Original: RORL surpasses EDAC by more than 10 points on both the robust score and the weighted robust score .
Revised: RORL surpasses EDAC by more than 10 points on both the robust score and the weighted robust score .

Original: C.4 Ablations of Components in the Adversarial Experiments
Revised: C.4 Ablations of Components in the Adversarial Experiments

Original: In Section 6.3, we conducted ablations of RORL’s major components in the adversarial settings.
Revised: In Section 6.3, we conducted ablations of RORL’s major components in the adversarial settings.

Original: In this subsection, we provide robust scores of the ablation results over 4 random seeds in Table 8.
Revised: In this subsection, we provide robust scores of the ablation results over 4 random seeds in Table 8.

Original: Besides, results of ϵ ood = 0 are also included to demonstrate the effectiveness of penalizing values of OOD states.
Revised: Besides, results of ϵ ood = 0 are also included to demonstrate the effectiveness of penalizing values of OOD states.

Original: From Table 8, we can conclude that the OOD loss is the most essential component of RORL, and only penalizing in-distribution states is insufficient for adversarial perturbations.
Revised: From Table 8, we can conclude that the OOD loss is the most essential component of RORL, and only penalizing in-distribution states is insufficient for adversarial perturbations.

Original: To summarize, the order of the importance of each component is: OOD loss > ϵ ood > policy smoothing loss > Q smoothing loss.
Revised: To summarize, the order of the importance of each component is: OOD loss > ϵ ood > policy smoothing loss > Q smoothing loss.

Original: The conclusion may be different for different tasks, for example we found that the halfcheetah task does not even need the OOD loss because the SAC-10 framework already provides it with sufficient pessimism.
Revised: The conclusion may be different for different tasks, for example we found that the halfcheetah task does not even need the OOD loss because the SAC-10 framework already provides it with sufficient pessimism.

Original: C.5 Ablations on the Number of Ensemble Q Networks
Revised: C.5 Ablations on the Number of Ensemble Q Networks

Original: We conduct the adversarial attack experiments with different number of bootstrapped Q networks in
Revised: We conduct the adversarial attack experiments with different number of bootstrapped Q networks in

Original: RORL.
Revised: RORL.

Original: As shown in Figure 10, the robustness of RORL improves as the ensemble size K increases.
Revised: As shown in Figure 10, the robustness of RORL improves as the ensemble size K increases.

Original: For K = 6 , 8 , 10 , RORL has similar initial performance but K = 10 considerably outperforms others as the attack scale increases.
Revised: For K = 6 , 8 , 10 , RORL has similar initial performance but K = 10 considerably outperforms others as the attack scale increases.

Original: Therefore, we set K = 10 by default in our paper.
Revised: Therefore, we set K = 10 by default in our paper.

Original: C.6 Ablations of τ for the Adversarial Experiments
Revised: C.6 Ablations of τ for the Adversarial Experiments

Original: In this subsection, we study the performance under attacks with varying τ ∈ { 0 . 0 , 0 . 2 , 0 . 5 , 1 . 0 } .
Revised: In this subsection, we study the performance under attacks with varying τ ∈ { 0 . 0 , 0 . 2 , 0 . 5 , 1 . 0 } .

Original: From the results in Figure 11, we find τ = 0 .
Revised: From the results in Figure 11, we find τ = 0 .

Original: 2 slightly outperforms the others on 4 out of the random
Revised: 2 slightly outperforms the others on 4 out of the random

Original: [Figure]
Revised: [Figure]

Original: diff
Revised: diff

Original: [Figure]
Revised: [Figure]

Original: diff mixed order min Q min Q mixed order
Revised: diff mixed order min Q min Q mixed order

Original: Average Score
Revised: Average Score

Original: RORL 94.
Revised: RORL 94.

Original: 71.2 no OOD 68.
Revised: 71.2 no OOD 68.

Original: 45.2 no P smooth 92.
Revised: 45.2 no P smooth 92.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: no Q smooth 92.
Revised: no Q smooth 92.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: attack types.
Revised: attack types.

Original: The results are also consistent with the ablation studies of the benchmark experiments in Appendix C.2.
Revised: The results are also consistent with the ablation studies of the benchmark experiments in Appendix C.2.

Original: Accordingly, we set τ = 0 .
Revised: Accordingly, we set τ = 0 .

Original: 2 by default for all experiments in our paper.
Revised: 2 by default for all experiments in our paper.

Original: [Figure]
Revised: [Figure]

Original: C.7 Ablations on the Number of Sampled Perturbed Observations
Revised: C.7 Ablations on the Number of Sampled Perturbed Observations

Original: We ablate the number of sampled perturbed observations in Figure 12.
Revised: We ablate the number of sampled perturbed observations in Figure 12.

Original: From the figure, we can conclude that the robustness of RORL improves as the number of samples n increases.
Revised: From the figure, we can conclude that the robustness of RORL improves as the number of samples n increases.

Original: At the same time, the computational cost also increases as n increases.
Revised: At the same time, the computational cost also increases as n increases.

Original: Therefore, we can choose n according to the computational budget.
Revised: Therefore, we can choose n according to the computational budget.

Original: Interestingly, RORL with n = 1 already outperforms SAC-10 by a large margin, which could be an appropriate option when computing resources are limited.
Revised: Interestingly, RORL with n = 1 already outperforms SAC-10 by a large margin, which could be an appropriate option when computing resources are limited.

Original: C.8 Adversarial Attack with Different Q Functions
Revised: C.8 Adversarial Attack with Different Q Functions

Original: In our experiments, it is assumed that the ’min Q ’ and the ’min Q mixed order’ attackers have access to the corresponding Q value functions of the attacked agent.
Revised: In our experiments, it is assumed that the ’min Q ’ and the ’min Q mixed order’ attackers have access to the corresponding Q value functions of the attacked agent.

Original: Generally, the assumption is strong for many real-world scenarios.
Revised: Generally, the assumption is strong for many real-world scenarios.

Original: In addition, the comparison does not take into account the impact of attacking with different Q functions.
Revised: In addition, the comparison does not take into account the impact of attacking with different Q functions.

Original: Intuitively, conservative and smoothed Q functions make it easier for attackers to find the most impactful perturbation to degrade the performance.
Revised: Intuitively, conservative and smoothed Q functions make it easier for attackers to find the most impactful perturbation to degrade the performance.

Original: To investigate the impact of different Q functions, we swap the attacker’s Q -function, i.e. using RORL’s
Revised: To investigate the impact of different Q functions, we swap the attacker’s Q -function, i.e. using RORL’s

Original: Q -functions to attack EDAC and using EDAC’s Q -functions to attack RORL .
Revised: Q -functions to attack EDAC and using EDAC’s Q -functions to attack RORL .

Original: In Figure 13, we can conclude:
Revised: In Figure 13, we can conclude:

Original: (1) RORL outperforms EDAC with a wider margin when using the same Q functions.
Revised: (1) RORL outperforms EDAC with a wider margin when using the same Q functions.

Original: Surprisingly, the difference of normalized scores increases from 37.3 to 51.2 for walker2d-mediumv2 task with the largest ’min Q’ attack.
Revised: Surprisingly, the difference of normalized scores increases from 37.3 to 51.2 for walker2d-mediumv2 task with the largest ’min Q’ attack.

Original: [Figure]
Revised: [Figure]

Original: (2) The value function of EDAC may still not be smooth and can mislead the attackers.
Revised: (2) The value function of EDAC may still not be smooth and can mislead the attackers.

Original: In contrast, RORL successfully learns smooth value functions, which may facilitate further research on stronger attack strategies for robust offline RL
Revised: In contrast, RORL successfully learns smooth value functions, which may facilitate further research on stronger attack strategies for robust offline RL

Original: [Figure]
Revised: [Figure]

Original: C.9 Comparison with EDAC+Smoothing
Revised: C.9 Comparison with EDAC+Smoothing

Original: We also compare EDAC with both policy smoothing and Q smoothing, which leverages the gradient penalty rather than our OOD loss to enforce pessimism on OOD state-action pairs.
Revised: We also compare EDAC with both policy smoothing and Q smoothing, which leverages the gradient penalty rather than our OOD loss to enforce pessimism on OOD state-action pairs.

Original: The hyperparameters are kept the same with EDAC and RORL, except τ = 0 . 5 in EDAC+Smoothing.
Revised: The hyperparameters are kept the same with EDAC and RORL, except τ = 0 . 5 in EDAC+Smoothing.

Original: As shown in Figure 14, the smoothing technique slightly improves the robustness of EDAC under largescale (0.2 ∼ 0.3) adversarial perturbations, but it significantly decreases the overall performance under
Revised: As shown in Figure 14, the smoothing technique slightly improves the robustness of EDAC under largescale (0.2 ∼ 0.3) adversarial perturbations, but it significantly decreases the overall performance under

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: attack.
Revised: attack.

Original: The results imply that directly using smoothing techniques without explicit OOD penalization can even worsen the robust scores of previous SOTA offline RL algorithm.
Revised: The results imply that directly using smoothing techniques without explicit OOD penalization can even worsen the robust scores of previous SOTA offline RL algorithm.

Original: C.10 Comparison with PBRL + S4RL
Revised: C.10 Comparison with PBRL + S4RL

Original: We also include comparison with PBRL and PBRL+S4RL to verify if RORL is more robust than data augmentation for offline RL [49].
Revised: We also include comparison with PBRL and PBRL+S4RL to verify if RORL is more robust than data augmentation for offline RL [50].

Original: The main differences between RORL and S4RL are three folds:
Revised: The main differences between RORL and S4RL are three folds:

Original: (1) S4RL only implicitly smooths the value functions while RORL explicitly smooths them, which is more efficient and enjoys theoretical guarantees.
Revised: (1) S4RL only implicitly smooths the value functions while RORL explicitly smooths them, which is more efficient and enjoys theoretical guarantees.

Original: (2) S4RL does not consider the impact of overestimation on OOD states brought by the data augmentation, which can be harmful for offline RL.
Revised: (2) S4RL does not consider the impact of overestimation on OOD states brought by the data augmentation, which can be harmful for offline RL.

Original: In contrast, RORL further underestimates values for OOD states, which essentially alleviates the potential overestimation.
Revised: In contrast, RORL further underestimates values for OOD states, which essentially alleviates the potential overestimation.

Original: In addition, S4RL selects adversarially perturbed states according to the gradient of Q ( s, π ( s )) , aiming to choose the direction where the Q -value deviates the most.
Revised: In addition, S4RL selects adversarially perturbed states according to the gradient of Q ( s, π ( s )) , aiming to choose the direction where the Q -value deviates the most.

Original: Different from S4RL, RORL samples perturbed states to maximize a conservative smoothing lossL (cid:0) Q
Revised: Different from S4RL, RORL samples perturbed states to maximize a conservative smoothing loss

Original: ϕ
Revised: L

Original: i
Revised: [Equation]

Original:  (ˆ s, a ) , Q ϕ i ( s, a ) (cid:1) and a policy smoothing loss
Revised: Q ϕ i (ˆ s, a ) , Q ϕ i ( s, a ) (cid:1) and a policy smoothing loss

Original: [Equation]
Revised: [Equation]

Original: defined in Section 4.
Revised: defined in Section 4.

Original: The empirical results on halfcheetah-medium-v2 and walker2d-medium-v2 are shown in Figure 15.
Revised: The empirical results on halfcheetah-medium-v2 and walker2d-medium-v2 are shown in Figure 15.

Original: We can observe that S4RL only slightly improves the robustness of PBRL on the walker2d-medium-
Revised: We can observe that S4RL only slightly improves the robustness of PBRL on the walker2d-medium-

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: v2 task and has little impact
Revised: v2 task and has little impact

Original: C.11 Combining Smoothing with IQL
Revised: C.11 Combining Smoothing with IQL

Original: We combine the policy smoothing and Q function smoothing techniques in RORL with IQL [26], a
Revised: We combine the policy smoothing and Q function smoothing techniques in RORL with IQL [27], a

Original: SOTA offline RL algorithm without ensemble Q networks.
Revised: SOTA offline RL algorithm without ensemble Q networks.

Original: We use the default hyper-parameters of
Revised: We use the default hyper-parameters of

Original: IQL and set the hyper-parameters for smoothing the same as in Table 5.
Revised: IQL and set the hyper-parameters for smoothing the same as in Table 5.

Original: The training and evaluation settings keep the same as the adversarial experiments in our paper.
Revised: The training and evaluation settings keep the same as the adversarial experiments in our paper.

Original: As shown in Figure 16, we can observe that IQL with the smoothing technique (short for ’IQL smooth’) slightly improves the robustness on the walker2d-medium-v2 and hopper-medium-v2 tasks, but it has little effect on the halfcheetah-medium-v2 task.
Revised: As shown in Figure 16, we can observe that IQL with the smoothing technique (short for ’IQL smooth’) slightly improves the robustness on the walker2d-medium-v2 and hopper-medium-v2 tasks, but it has little effect on the halfcheetah-medium-v2 task.

Original: This suggests that simply adopting the smoothing technique does not consistently improve the performance in the offline setting.
Revised: This suggests that simply adopting the smoothing technique does not consistently improve the performance in the offline setting.

Original: In contrast, RORL introduces additional
Revised: In contrast, RORL introduces additional

Original: OOD underestimation based on uncertainty measure, which helps to obtain conservatively smoothed policy and value functions.
Revised: OOD underestimation based on uncertainty measure, which helps to obtain conservatively smoothed policy and value functions.

Original: C.12 Comparing the ’max’ and the ’mean’ Operators in Smoothing
Revised: C.12 Comparing the ’max’ and the ’mean’ Operators in Smoothing

Original: In our implementation, we first sample n perturbed states and select the one that maximizes the smoothing losses in Eq.
Revised: In our implementation, we first sample n perturbed states and select the one that maximizes the smoothing losses in Eq.

Original: (2) and Eq.
Revised: (2) and Eq.

Original: It is interesting to see if the ’max’ operator is useful, as we can also use the ’mean’ operator as an alternative, i.e., L mean smooth ( s, a ; ϕ i ) =
Revised: It is interesting to see if the ’max’ operator is useful, as we can also use the ’mean’ operator as an alternative, i.e., L mean smooth ( s, a ; ϕ i ) =

Original: E ˆ s ∈ B
Revised: E ˆ s ∈ B

Original: d ( s,ϵ ) L
Revised: d ( s,ϵ ) L

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: The results are demonstrated in Figure 17.
Revised: The results are demonstrated in Figure 17.

Original: We can find that RORL with the ’max’ operator obtains a more conservative policy under small-scale perturbations and achieves higher robustness under largescale perturbations.
Revised: We can find that RORL with the ’max’ operator obtains a more conservative policy under small-scale perturbations and achieves higher robustness under largescale perturbations.

Original: Since the ’max’ operator has the same complexity as the ’mean’ operator, we use the ’max’ operator by default, which is also a zeroth-order approximation to an inner optimization problem.
Revised: Since the ’max’ operator has the same complexity as the ’mean’ operator, we use the ’max’ operator by default, which is also a zeroth-order approximation to an inner optimization problem.

Original: C.13 Comparing Different Optimization for Perturbation Generation during Training
Revised: C.13 Comparing Different Optimization for Perturbation Generation during Training

Original: In the training period, we use zeroth-order optimization to approximately optimize the Q smoothing loss in Eq.
Revised: In the training period, we use zeroth-order optimization to approximately optimize the Q smoothing loss in Eq.

Original: (2) and the policy smoothing loss: max
Revised: (2) and the policy smoothing loss: max

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: ·| s
Revised: ·| s

Original: [Equation]
Revised: [Equation]

Original: ∥ π
Revised: ∥ π

Original: [Equation]
Revised: [Equation]

Original: ( ·| ˆ s )
Revised: ( ·| ˆ s )

Original: [Equation]
Revised: [Equation]

Original: In this way, we can accelerate training the robust policy and obtain similar performance.
Revised: In this way, we can accelerate training the robust policy and obtain similar performance.

Original: Besides, zeroth-order optimization is commonly applied in black-box attack where we can only access the input and output of neural networks without explicit gradient information.
Revised: Besides, zeroth-order optimization is commonly applied in black-box attack where we can only access the input and output of neural networks without explicit gradient information.

Original: Black-box attack for reinforcement learning might be a promising direction in the future.
Revised: Black-box attack for reinforcement learning might be a promising direction in the future.

Original: We also implemented a first-order version of RORL, which requires an average epoch time of 72.7s on a V100 GPU (while the average epoch time of the zeroth-order method is 29.6s).
Revised: We also implemented a first-order version of RORL, which requires an average epoch time of 72.7s on a V100 GPU (while the average epoch time of the zeroth-order method is 29.6s).

Original: Since the perturbation generation for each training step is independent, we use the first-order optimization for a probability of 0.5 to alleviate the computational cost.
Revised: Since the perturbation generation for each training step is independent, we use the first-order optimization for a probability of 0.5 to alleviate the computational cost.

Original: In Figure 18, we compare the trained policies with zeroth-order and first-order optimization.
Revised: In Figure 18, we compare the trained policies with zeroth-order and first-order optimization.

Original: We can conclude that the two types of optimization for perturbation generation have very similar performance.
Revised: We can conclude that the two types of optimization for perturbation generation have very similar performance.

Original: On halfcheetah-medium task, the first-order version performs slightly better than the zeroth-order version, while the zeroth-order version works
Revised: On halfcheetah-medium task, the first-order version performs slightly better than the zeroth-order version, while the zeroth-order version works

Original: [Figure]
Revised: [Figure]

Original: slightly better on the walker2d-medium task.
Revised: slightly better on the walker2d-medium task.

Original: We think this might be because we train the policy and value networks for 3 × 10 6 training steps, which may narrow the gap of the two optimization methods.
Revised: We think this might be because we train the policy and value networks for 3 × 10 6 training steps, which may narrow the gap of the two optimization methods.

Original: On the contrary, the mixed-order attackers (’action diff mixed order’ and ’min Q mixed order’) work better than zeroth-order attackers (’action diff’ and ’min Q ’) in the evaluation period, as demonstrated in Figure 4.
Revised: On the contrary, the mixed-order attackers (’action diff mixed order’ and ’min Q mixed order’) work better than zeroth-order attackers (’action diff’ and ’min Q ’) in the evaluation period, as demonstrated in Figure 4.

Original: C.14 Comparison of the “minus target” and the “min target”
Revised: C.14 Comparison of the “minus target” and the “min target”

Original: In the OOD loss L ood (Eq.
Revised: In the OOD loss L ood (Eq.

Original: (4)), the pseudo-target (cid:98) T ood Q ϕ
Revised: (4)), the pseudo-target (cid:98) T ood Q ϕ

Original: i (ˆ s, ˆ a ) for the OOD state-action pairs (ˆ s, ˆ a ) can be implemented in two ways to underestimate the values of (ˆ s, ˆ a ) : (cid:98) T ood Q ϕ
Revised: i (ˆ s, ˆ a ) for the OOD state-action pairs (ˆ s, ˆ a ) can be implemented in two ways to underestimate the values of (ˆ s, ˆ a ) : (cid:98) T ood Q ϕ

Original: i (ˆ s, ˆ a ) :=
Revised: i (ˆ s, ˆ a ) :=

Original: Q ϕ i (ˆ s, ˆ a ) − λu (ˆ s, ˆ a ) or (cid:98) T ood Q ϕ
Revised: Q ϕ i (ˆ s, ˆ a ) − λu (ˆ s, ˆ a ) or (cid:98) T ood Q ϕ

Original: i (ˆ s, ˆ a ) :
Revised: i (ˆ s, ˆ a ) :

Original: = min i =1 ,...,K Q ϕ
Revised: = min i =1 ,...,K Q ϕ

Original: i (ˆ s, ˆ a ) ( K = 10 ).
Revised: i (ˆ s, ˆ a ) ( K = 10 ).

Original: The two targets are referred to as the “minus target” and the “min target” respectively.
Revised: The two targets are referred to as the “minus target” and the “min target” respectively.

Original: In Figure 19, we compare the two targets’ D4RL scores in clean environments, and the hyper-parameters are the same as Table 4.
Revised: In Figure 19, we compare the two targets’ D4RL scores in clean environments, and the hyper-parameters are the same as Table 4.

Original: Although the “min target” has less hyper-parameters and achieves comparable performance on the hopper-medium-expert-v2 task, it is unstable and not flexible across different tasks, e.g., significantly overestimating values for the walker2d-random-v2 task and underestimating values for hopper-medium-v2 and walker2d-medium-v2 tasks.
Revised: Although the “min target” has less hyper-parameters and achieves comparable performance on the hopper-medium-expert-v2 task, it is unstable and not flexible across different tasks, e.g., significantly overestimating values for the walker2d-random-v2 task and underestimating values for hopper-medium-v2 and walker2d-medium-v2 tasks.

Original: Therefore, we choose the “minus target” by default in our paper.
Revised: Therefore, we choose the “minus target” by default in our paper.

Original: [Table]
Revised: [Table]

Original: C.15 Experiments in Adroit Domains
Revised: C.15 Experiments in Adroit Domains

Original: We also evaluate RORL in the challenging Adroit domains which control a 24-DoF robotic hand to manipulate a pen, a hammer, a door and a ball.
Revised: We also evaluate RORL in the challenging Adroit domains which control a 24-DoF robotic hand to manipulate a pen, a hammer, a door and a ball.

Original: These domains contain three types of data, namely ‘Expert’, ‘Cloned’, and ‘Human’, for each task.
Revised: These domains contain three types of data, namely ‘Expert’, ‘Cloned’, and ‘Human’, for each task.

Original: The hyper-parameters are listed in Table 9.
Revised: The hyper-parameters are listed in Table 9.

Original: We set β Q = 0 .
Revised: We set β Q = 0 .

Original: 0001 , β ood = 0 .
Revised: 0001 , β ood = 0 .

Original: 5 , τ = 0 .
Revised: 5 , τ = 0 .

Original: 2 , n = 20 , and search β P within { 0 . 01 , 0 . 1 , 1 . 0 } , ϵ Q / ϵ P / ϵ ood within { 0 . 001 , 0 . 005 , 0 . 01 } .
Revised: 2 , n = 20 , and search β P within { 0 . 01 , 0 . 1 , 1 . 0 } , ϵ Q / ϵ P / ϵ ood within { 0 . 001 , 0 . 005 , 0 . 01 } .

Original: For Door/Relocate-human/cloned datasets, the policy learning rate is set to 1 e − 4 .
Revised: For Door/Relocate-human/cloned datasets, the policy learning rate is set to 1 e − 4 .

Original: The other hyper-parameters are the same as in Table 3.
Revised: The other hyper-parameters are the same as in Table 3.

Original: On four expert datasets, we train
Revised: On four expert datasets, we train

Original: RORL for 1000 epochs (1000 gradient steps per epoch).
Revised: RORL for 1000 epochs (1000 gradient steps per epoch).

Original: As for other datasets, we train RORL for 300 epochs because the ‘Cloned’ and ‘Human’ datasets are much smaller.
Revised: As for other datasets, we train RORL for 300 epochs because the ‘Cloned’ and ‘Human’ datasets are much smaller.

Original: In Table [Table] we compare the performance of RORL with other baselines, such as EDAC, PBRL,
Revised: In Table 10, we compare the performance of RORL with other baselines, such as EDAC, PBRL,

Original: TD3+BC, CQL, UWAC, BEAR, and BC.
Revised: TD3+BC, CQL, UWAC, BEAR, and BC.

Original: We can observe that RORL achieves the top two highest score in 6 out of 12 tasks, which further verifies the effectiveness of RORL.
Revised: We can observe that RORL achieves the top two highest score in 6 out of 12 tasks, which further verifies the effectiveness of RORL.

Original: [Table]
Revised: [Table]

Original: C.16 AntMaze Tasks
Revised: C.16 AntMaze Tasks

Revised: The AntMaze domain is a challenging navigation domain with an 8-DoF Ant quadraped robot and three types of datasets, namely “umaze”, “medium”, and “large”.

Revised: In this domain, the agent receives a sparse reward of 0/1, where reward 1 is given only when the ant reaches the desired goal.

Revised: The challenges for AntMaze domain are sparse rewards, undirected and multitask data, which might be beyond the scope of our study.

Revised: To the best of our knowledge, very few ensemble-based offline

Original: To verify whether smoothing benefits policy learning in the challenging AntMaze setting, we apply our policy and value function smoothing techniques on top of IQL (short for ‘IQL+smoothing’).
Revised: RL algorithms can work in this domain, probably because estimating uncertainty in sparse reward setting is difficult. A recent work [15] conducted in-depth research on this problem and found that the independent target is crucial for the uncertainty estimation in ensemble-based offline RL. We adopt the techniques used in [15] for RORL and reported the results in Table 12. We only use the

Revised: OOD loss and policy smoothing loss for RORL, and replace the shared min target in Eq.

Revised: (1) with independent target to train Q functions.

Revised: [Equation]

Revised: In Eq (6), we train the policy with the LCB objective (i.e., mean j =1 ,...,K Q ϕ j

Revised: ( s

Revised: [Equation]

Revised: ) − c · std j =1 ,...,K Q ϕ j ( s, a ) , where c

Revised: = 4 is used in our experiments) or the ‘min Q’ target (i.e., min j =1 ,...,K Q ϕ j ( s, a ) ) to enforce pessimism.

Revised: Following [15], we train RORL for 200 epochs and evaluate the final performance for 100 episodes.

Revised: Instead of changing 0/1 reward to -2/2, we adopt reward shifting [51] to change 0/1 reward to 0.001/10.

Revised: We also find that adding the BC loss to the policy loss is helpful for antmaze-umaze tasks.

Revised: Other hyper-parameters such as the coefficient β BC of BC loss are listed in Table 11.

Revised: We also apply our policy and value function smoothing techniques on top of IQL (short for ‘IQL+smoothing’).

Original: For the hyper-parameters, we use ϵ Q = 0 .
Revised: For the hyper-parameters, we use ϵ Q = 0 .

Original: 01 , ϵ P = 0 .
Revised: 01 , ϵ P = 0 .

Original: 03 , τ = 0 .
Revised: 03 , τ = 0 .

Original: 2 for all six types of datasets, and search β Q ∈ { 0 . 1 , 0 . 01 } , β P ∈ { 0 . 1 , 0 . 5 } , n = 20 ,.
Revised: 2 for all six types of datasets, and search β Q ∈ { 0 . 1 , 0 . 01 } , β P ∈ { 0 . 1 , 0 . 5 } , n = 20 .

Original: Other hyper-parameters keep the default hyper-parameters of IQL [26].
Revised: Other hyper-parameters keep the default hyper-parameters of IQL [27].

Original: In Table 11, we compare  ‘IQL+smoothing’ with both model-free (AWAC [36], TD3+BC [13], CQL [28], and IQL [26]) and model-based (ROMI [56]) baselines.
Revised: In Table 12, we compare RORL and ‘IQL+smoothing’ with both model-free (AWAC [37], TD3+BC [13], CQL [29], and IQL [27]) and model-based (ROMI [58]) baselines.

Revised: RORL achieves the highest average score on the 6 tasks.

Original: On 4 out of 6 tasks, ’IQL+smoothing’ improves the performance of IQL.
Revised: Besides, on 4 out of 6 tasks, ’IQL+smoothing’ improves the performance of IQL.

Original: Intuitively, for sparse reward tasks, smoothing the value functions of nearby states could help with the value propagation, and smoothing the policy can enhance the robustness of learned policies. 
Revised: Intuitively, for sparse reward tasks, smoothing the value functions of nearby states could help with the value propagation, and smoothing the policy can enhance the robustness of learned policies. [Table]

Original: [Table]
Revised: [Table]

Original: C.17 Robustness of the Benchmark Results
Revised: C.17 Robustness of the Benchmark Results

Original: In Figure

Original: [Table]

Original:  we evaluate the robustness of the benchmark results, i [Table] e., how robust each algorithm is to maintain the performance listed in Table 1.
Revised: In Figure 20, we evaluate the robustness of the benchmark results, i.e., how robust each algorithm is to maintain the performance listed in Table 1.

Original: We compare RORL with EDAC, SAC-10 on six tasks.
Revised: We compare RORL with EDAC, SAC-10 on six tasks.

Original: EDAC is reproduced with 10 ensemble Q networks as RORL and SAC-10, and uses η = 1 for all six tasks.
Revised: EDAC is reproduced with 10 ensemble Q networks as RORL and SAC-10, and uses η = 1 for all six tasks.

Original: Note that in the benchmark experiments, RORL is only trained with small smoothing scales within { 0 . 001 , 0 . 005 , 0 . 01 } .
Revised: Note that in the benchmark experiments, RORL is only trained with small smoothing scales within { 0 . 001 , 0 . 005 , 0 . 01 } .

Original: The evaluation perturbation scales are within range [0 . 00 , 0 . 05] and the results are averaged over 4 random seeds.
Revised: The evaluation perturbation scales are within range [0 . 00 , 0 . 05] and the results are averaged over 4 random seeds.

Original: From the results, we can conclude that RORL can successfully keep the highest performance within a certain perturbation scale and the performance of
Revised: From the results, we can conclude that RORL can successfully keep the highest performance within a certain perturbation scale and the performance of

Original: EDAC and SAC-10 decreases faster than RORL for most tasks and attack methods.
Revised: EDAC and SAC-10 decreases faster than RORL for most tasks and attack methods.

Original: The results imply that RORL has better practicability in real-world scenarios.
Revised: The results imply that RORL has better practicability in real-world scenarios.

Revised: D Tips for Customizing RORL

Revised: According to our ablation study result in Appendix C, we summarize some tips for adapting RORL for customized use below.

Revised: • Hyper-parameter Tuning: Since RORL is proposed to solve a challenging problem, it has many hyper-parameters.

Revised: Our first suggestion is to use our hyper-parameter search range in Appendix B.1.

Original: D More Related Works
Revised: You can tune them according to the importance of each component, where the general order is : OOD loss > policy smoothing loss > Q smoothing loss. • Computation Cost: If you want less GPU memory usage and less training time, you can (1) set β Q = 0 and ϵ Q = 0 because the Q smoothing loss contributes the least but consumes a large computational cost, and (2) use a small number n of sampled perturbed states to reduce the GPU memory usage. E More Related Works

Original: Model-Based Offline RL In offline RL, model-based methods use an empirical model learned from the offline dataset to enhance the generalization ability.
Revised: Model-Based Offline RL In offline RL, model-based methods use an empirical model learned from the offline dataset to enhance the generalization ability.

Original: The model can be used as the virtual environment for data collection [70, 25], or to augment the dataset for an existing model-free algorithm [69, 56].
Revised: The model can be used as the virtual environment for data collection [72, 26], or to augment the dataset for an existing model-free algorithm [71, 58].

Original: The main challenges of model-based algorithms are how to learn the accurate empirical model and how to construct the uncertainty measure.
Revised: The main challenges of model-based algorithms are how to learn the accurate empirical model and how to construct the uncertainty measure.

Original: A recent work [21] demonstrates that the transformer model can generate realistic trajectories, which is beneficial for policy learning.
Revised: A recent work [22] demonstrates that the transformer model can generate realistic trajectories, which is beneficial for policy learning.

Original: In contrast, we focus on the model-free methods in this paper and leave the robustness of model-based methods in future work.
Revised: In contrast, we focus on the model-free methods in this paper and leave the robustness of model-based methods in future work.

Original: Adversarial Attack Inspired by adversarial examples in deep learning [17, 39], adversarial attack and policy poisoning [8, 19, 41] are studied to avoid adversarial manipulations on the network policies.
Revised: Adversarial Attack Inspired by adversarial examples in deep learning [18, 40], adversarial attack and policy poisoning [8, 20, 42] are studied to avoid adversarial manipulations on the network policies.

Original: Gleave et al.
Revised: Gleave et al.

Original: [15] study adversarial policy in the behavior level [15].
Revised: [16] study adversarial policy in the behavior level [16].

Original: Data corruption [72, 33, 60] considers the case where an attacker can arbitrarily modify the dataset under a specific budget before training.
Revised: Data corruption [74, 34, 62] considers the case where an attacker can arbitrarily modify the dataset under a specific budget before training.

Original: While adversarial attack in RL is highly related to robust RL, they focus more on adversarial attacks compared to our robustness setting.
Revised: While adversarial attack in RL is highly related to robust RL, they focus more on adversarial attacks compared to our robustness setting.

Original: More effective attack strategies for offline RL can facilitate learning more robust policies.
Revised: More effective attack strategies for offline RL can facilitate learning more robust policies.

Original: [Figure]
Revised: [Figure]