Original: RORL: Robust Ofﬂine Reinforcement Learning via Conservative Smoothing
Revised: RORL: Robust Offline Reinforcement Learning via Conservative Smoothing

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Ofﬂine reinforcement learning (RL) provides a promising direction to exploit themassive amount of ofﬂine data for complex decision-making tasks.
Revised: Offline reinforcement learning (RL) provides a promising direction to exploit massive amount of offline data for complex decision-making tasks.

Original: Due to thedistribution shift issue, current ofﬂine RL algorithms are generally designed to beconservative for value estimation and action selection.
Revised: Due to the distribution shift issue, current offline RL algorithms are generally designed to be conservative in value estimation and action selection.

Original: However, such conservatismimpairs the robustness of learned policies, leading to a signiﬁcant change even fora small perturbation on observations.
Revised: However, such conservatism can impair the robustness of learned policies when encountering observation deviation under realistic conditions, such as sensor errors and adversarial attacks.

Original: To trade off robustness and conservatism, wepropose Robust Ofﬂine Reinforcement Learning (RORL) with a novel conservativesmoothing technique.
Revised: To trade off robustness and conservatism, we propose Robust Offline Reinforcement Learning (RORL) with a novel conservative smoothing technique.

Original: In RORL, we explicitly introduce regularization on the policyand the value function for states near the dataset and additional conservative valueestimation on theseOOD states.
Revised: In RORL, we explicitly introduce regularization on the policy and the value function for states near the dataset, as well as additional conservative value estimation on these states.

Original: Theoretically, we show RORL enjoys a tightersuboptimality bound than recent theoretical results in linear MDPs.
Revised: Theoretically, we show RORL enjoys a tighter suboptimality bound than recent theoretical results in linear MDPs.

Original: We demonstratethat RORL can achieve the state-of-the-art performance on the general ofﬂine RLbenchmark and is considerably robust to adversarial observation perturbation.
Revised: We demonstrate that RORL can achieve stateof-the-art performance on the general offline RL benchmark and is considerably robust to adversarial observation perturbations.

Original:  Introduction
Revised:  Introduction

Original: Over the past few years, deep reinforcement learning (RL) has been a vital tool for various decisionmaking tasks [30, 44, 42, 8] in a trial-and-error manner.
Revised: Over the past few years, deep reinforcement learning (RL) has been a vital tool for various decisionmaking tasks [35, 48, 46, 11] in a trial-and-error manner.

Original: A major limitation of current deep RLalgorithms is that they require intense online interactions with the environment [26].
Revised: A major limitation of current deep RL algorithms is that they require intense online interactions with the environment [29, 65].

Original: These datacollecting processes can be costly and even prohibitive in many real-world scenarios such as roboticsand health care.
Revised: These data collecting processes can be costly and even prohibitive in many real-world scenarios such as robotics and health care [29, 51].

Original: Ofﬂine RL [12, 24] is gaining more attention recently since it offers probabilities tolearn reinforced decision-making strategies from fully ofﬂine datasets.
Revised: Offline RL [14, 27] is gaining more attention recently since it offers probabilities to learn reinforced decision-making strategies from fully offline datasets.

Original: The main challenge of ofﬂine RL is the distribution shift between the ofﬂine dataset and the learnedpolicy, which would lead to signiﬁcant overestimation for the out-of-distribution (OOD) actions[12, 24].
Revised: The main challenge of offline RL is the distribution shift between the offline dataset and the learned policy, which would lead to severe overestimation for the out-of-distribution (OOD) actions [14, 27].

Original: To overcome such an issue, a series of ofﬂine RL works [51, 12, 58, 25, 2, 3] propose tocelebrate conservatism, such as constraining the learned policy close to  supported distribution orpenalizing the Q -values of OOD actions.
Revised: To overcome such an issue, a series of model-free offline RL works [57, 14, 67, 28, 31, 2, 64, 6] propose to celebrate conservatism, such as constraining the learned policy close to the supported distribution or penalizing the Q -values of OOD actions.

Original: In addition, another stream of works builds upon modelbased ofﬂine RL algorithms [61, 60, 50], which leverages the ensemble dynamics models to enforcepessimism via uncertainty penalizing or data generation.
Revised: Besides, another stream of works builds upon model-based algorithms [70, 69, 56], which leverages the ensemble dynamics models to enforce pessimism through uncertainty penalizing or data generation.

Original: However, conservatism is not the only concern when applying ofﬂine RL to the real world.
Revised: However, conservatism is not the only concern when applying offline RL to the real world.

Original: Dueto the sensor errors and model mismatch, the robustness of ofﬂine RL is  crucial under the realisticengineering conditions, which has not been well studied yet.
Revised: Due to the sensor errors and model mismatch, the robustness of offline RL is also crucial under the realistic engineering conditions, which has not been well studied yet.

Original: In online RL, a series of works havebeen studied to learn the optimal policy under worst-case perturbations of the observation [62, 35, 17]or environmental dynamics [49, 37, 38].
Revised: In online RL, a series of works has been studied to learn the optimal policy under worst-case perturbations of the observation [71, 40, 19] or environmental dynamics [55, 42, 43, 4].

Original: Yet, it is non-trivial to apply online robust RL techniquesinto the ofﬂine problems.
Revised: Yet, it is non-trivial to apply online robust RL techniques into the offline problems.

Original: The main challenge is that the perturbation ofeither statesor actions may bring OOD data and extra overestimation for the value function.
Revised: The main challenge is that the perturbation of states may bring OOD observation and extra overestimation for the value function.

Original: New techniques are needed to tackle the conservatism and robustness simultaneously in the ofﬂine RL.
Revised: New techniques are needed to tackle the conservatism and robustness simultaneously in the offline RL.

Original: This paper studies robust ofﬂine RL against adversarial observation perturbation, where the agentneeds to learn the policy conservatively while handling the potential OOD observation with perturbation.
Revised: This paper studies robust offline RL against adversarial observation perturbations, where the agent needs to learn the policy conservatively while handling the potential OOD observation with perturbation.

Original: We ﬁrst demonstrate that current value-based ofﬂine RL algorithms lack the necessarysmoothness for the policy, which is visualized in Figure 1.
Revised: We first demonstrate that current value-based offline RL algorithms lack the necessary smoothness for the policy, which is visualized in Figure 1.

Original: As an illustration, we show that a famousbaseline method CQL [25] learns a non-smooth value function, leading to signiﬁcant performancedegradation for even a tiny scale perturbation on observation (see Section 3 for details).
Revised: As an illustration, we show that a famous baseline method CQL [28] learns a non-smooth value function, leading to significant performance degradation for even a tiny scale perturbation on observation (see Section 3 for details).

Original: Notice thatsimply adopting the smoothing technique for existing methods may result in extra overestimation atthe boundary of supported distribution and lead the agent towardthe unsafe areas.
Revised: In addition, simply adopting the smoothing technique for existing methods may result in extra overestimation at the boundary of supported distribution and lead the agent toward unsafe areas.

Original: To this end, we propose Robust Ofﬂine Reinforcement Learning (RORL) with a novel conservativesmoothing technique, which explicitly handles the overestimation of OOD state-action pairs.
Revised: To this end, we propose Robust Offline Reinforcement Learning (RORL) with a novel conservative smoothing technique, which explicitly handles the overestimation of OOD state-action pairs.

Original: Speciﬁcally, we explicitly introduce smooth regularization on both the values and policies for states nearthe dataset support and conservatively estimate the values of these OOD states based on pessimisticbootstrapping.
Revised: Specifically, we explicitly introduce smooth regularization on both the value functions and policies for states near the dataset support and conservatively estimate the values of these OOD states based on pessimistic bootstrapping.

Original: In addition, we theoretically prove that RORL yields a valid uncertainty quantiﬁer inlinear MDPs and enjoys a tighter suboptimality bound than previous work [3].
Revised: Furthermore, we theoretically prove that RORL yields a valid uncertainty quantifier in linear MDPs and enjoys a tighter suboptimality bound than previous work [6].

Original: In our experiments 1 , we demonstrate that RORL can achieve state-of-the-art (SOTA) performance in  D4RL benchmark [9] for ofﬂine RL given a suitable adversarial perturbation scale during training.
Revised: In our experiments 3 , we demonstrate that RORL can achieve state-of-the-art (SOTA) performance in the D4RL benchmark [12] with fewer ensemble Q networks than the current SOTA approach [2].

Original: Thesurprising performance in the benchmark shows thatthe robust training improves the generalizationability and leads to performance improvementeven in non-perturbed tasks.
Revised: The results of the benchmark experiments imply that robust training can lead to performance improvement in non-perturbed environments.

Original: Meanwhile, comparedwith the current SOTA baseline [2], RORL is considerably more robust to adversarial perturbationson observations.
Revised: Meanwhile, compared with current ensemble-based baselines, RORL is considerably more robust to adversarial perturbations on observations.

Original: We conduct the ablation study under different types of adversarial attacks on theobservations, showingthe consistently superior performance over several  tasks.
Revised: We conduct the adversarial experiments under different attack types, showing consistently superior performance on several continuous control tasks.

Original:  Preliminaries
Revised:  Preliminaries

Original: Ofﬂine RLand PBRL Considering an episodic MDP M =
Revised: Offline RL Considering an episodic MDP M =

Original: ( S , A , T, r,  , P ) , where S is the state58 space, A is the action space, T is the length of the episode, r is the reward function, P is the dynamics,and  is the discount factor.
Revised: ( S , A , T, r, γ, P ) , where S is the state space, A is the action space, T is the length of an episode, r is the reward function, P is the dynamics, and γ is the discount factor.

Original: In ofﬂine RL, the objective of the agent is to ﬁnd an optimal policy bysampling experience from a ﬁxed dataset D = { ( s it , a it , r it , s it +1 ) } .
Revised: In offline RL, the objective of the agent is to find an optimal policy by sampling experiences from a fixed dataset D = { ( s it , a it , r it , s it +1 ) } .

Original: Nevertheless, directly applyingoff-policy algorithms in ofﬂine RL has distribution shift problem.
Revised: Nevertheless, directly applying off-policy algorithms in offline RL suffers from the distribution shift problem.

Original: In Q -learning, the value functionevaluated on the greedy action a 0 in Bellman operator T Q = r +  E s 0 [max a 0 ( s 0 , a 0 )] tends to haveextrapolation error since ( s 0 , a 0 ) has barely occurred in D .
Revised: In Q -learning, the value function evaluated on the greedy action a ′ in Bellman operator T Q = r + γ E s ′ [max a ′ ( s ′ , a ′ )] tends to have extrapolation error since ( s ′ , a ′ ) has barely occurred in D .

Original: There are mainly three kinds of methodsto track the distribution shift problem in ofﬂine RL, including policy constraints, conservative valuefunction, and uncertainty-based methods.

Original: Pessimistic Bootstrapping for Ofﬂine RL (PBRL) [3] is an uncertainty-based method that uses bootstrapped Q -functions for uncertainty quantiﬁcation and OOD sampling for regularization.
Revised: Pessimistic Bootstrapping for Offline RL (PBRL) [6] is an uncertainty-based method that uses bootstrapped Q -functions for uncertainty quantification [50] and OOD sampling for regularization.

Original: Speciﬁcally,
Revised: [Figure]

Original:  PBRL maintains K bootstrapped Q functions to quantify the epistemic uncertainty  and performspessimistic update to penalize Q functions with large uncertainties.
Revised: Specifically, PBRL maintains K bootstrapped Q functions to quantify the epistemic uncertainty [5] and performs pessimistic update to penalize Q functions with large uncertainties.

Original: The uncertainty is deﬁned asthe standard deviation among bootstrapped Q -functions.
Revised: The uncertainty is defined as the standard deviation among bootstrapped Q -functions.

Original: For each bootstrapped Q -function, the 
Revised: For each bootstrapped Q -function, the Bellman target is defined as (cid:98) T Q ( s, a ) = r ( s, a )+ γ (cid:98) E s [Equation] ∼ P ( ·| s,a

Original: [Figure]
Revised: [Equation]

Original: Bellman target is deﬁned as
Revised: ,a ′ ∼ [Equation] |

Original: [Equation] 
Revised: [Equation] [Equation]

Original:  Under linear MDP assumptions, this uncertainty is equivalent to the LCB penalty and is provablyefﬁcient [21].
Revised: (cid:2) Q ( s ′ , a ′ ) − λu ( s ′ , a ′ ) (cid:3) . Under linear MDP assumptions, this uncertainty is equivalent to the LCB penalty and is provably efficient [23].

Original: Furthermore, PBRL incorporates OOD samplingin training by sampling OOD actionsto form ( s, a ood ) pairs, where a ood follows the learned policy.
Revised: Furthermore, PBRL incorporates OOD sampling by sampling OOD actions to form ( s, a ood ) pairs, where a ood follows the learned policy.

Original: The  learning target for ( s, a ood ) isb T ood Q ( s, a ood ) :
Revised: The detached learning target for ( s, a ood ) is (cid:98) T ood Q ( s, a ood ) :

Original: = Q ( s, a ood )   u ( s, a ood ) , which introduces uncertainty penalization to enforcepessimistic Q -functions for OOD actions.
Revised: = Q ( s, a ood ) − λu ( s, a ood ) , which introduces uncertainty penalization to enforce pessimistic Q -functions for OOD actions.

Original: Smooth Regularized RL Robust RL aims to learn a robust policy against the adversarial perturbed environment in online RL.
Revised: Smooth Regularized RL Robust RL aims to learn a robust policy against the adversarial perturbed environment in online RL.

Original: SR 2 L [43] enforces smoothness in both the policy and Qfunctions.
Revised: SR 2 L [47] enforces smoothness in both the policy and Q -functions.

Original: Speciﬁcally, SR 2 L encourages the outputs of the policy and value function to notchange much when injecting small perturbation to the state.
Revised: Specifically, SR 2 L encourages the outputs of the policy and value function to not change much when injecting small perturbations to the states.

Original: For state s , SR 2 L constructs a perturbation set B d ( s, ✏ )=
Revised: For state s , SR 2 L constructs a perturbation set B d ( s, ϵ )

Original:  { ˆ s : d ( s, ˆ s )  ✏ }  and introduces a smoothness regularizer for policy as 
Revised: = { ˆ s : d ( s, ˆ s ) ≤ ϵ } with a metric d ( , ) , which is chosen to be the ℓ p distance, and introduces a smoothness regularizer for policy as R πs = E s ∼ ρ π max ˆ s ∈ B d ( s,ϵ ) D

Original: [Equation]
Revised: ( π ( ·| s )

Original: ✏ ) D ( ⇡ ( ·| s ) k ⇡ ( ·| ˆ s )) , where D ( · k · ) is a distance metric and the max oper- 83 ator gives an adversarial manner to choose ˆ s .
Revised: ∥ π ( ·| ˆ s )) , where D ( ·∥· ) is a distance metric and the max operator gives an adversarial manner to choose ˆ s .

Original: Similarly, the smoothness regularizer for the valuefunction is deﬁned as R Vs =
Revised: Similarly, the smoothness regularizer for the value function is defined as R Vs =

Original: E s ⇠ ⇢ ⇡ ,a ⇠ ⇡ max ˆ s 2 B d ( s, ✏ ) ( Q ( s, a )  Q (ˆ s, a )) 2 .
Revised: E s ∼ ρ π ,a ∼ π max ˆ s ∈ B d ( s,ϵ ) ( Q ( s, a ) − Q (ˆ s, a )) 2 .

Original: The induced smoothness 85 is shown to improve robustness against both random and adversarial perturbations.
Revised: SR 2 L is shown to improve robustness against both random and adversarial perturbations.

Original:  Robustness of Ofﬂine RL: A Motivating Example
Revised:  Robustness of Offline RL: A Motivating Example

Original: We give a motivating example to illustrate the robustness of the popular CQL [25] policies.
Revised: We give a motivating example to illustrate the robustness of the popular CQL [28] policies.

Original: Weintroduce an adversarial attack on state s to obtain ˆ s = arg max 
Revised: We introduce an adversarial attack on state s to obtain ˆ s = arg max ˆ

Original: ˆ s 2 B d ( s, ✏ ) D J ( ⇡ ✓ ( ·| s ) k ⇡ ✓ ( ·| ˆ s )) , where 
Revised: s ∈ B d ( s,ϵ ) D J ( π θ ( ·| s ) ∥ π θ ( ·| ˆ s )) , where B d ( s, ϵ ) =

Original: B d  is the perturbation set to ensure d ( s, ˆ s )  ✏ and the  Jeffrey’s divergence D J for two distributions P , Q is deﬁned by: D J ( P k Q ) =12 [ D KL ( P k Q )
Revised: { ˆ s : d ( s, ˆ s ) ≤ ϵ } is the perturbation set and the metric d ( , ) is chosen to be the ℓ ∞ norm. The Jeffrey’s divergence D J for two distributions P , Q is defined by: D J ( P ∥ Q ) =

Original:  + D KL ( Q k P )] .
Revised: 12 [ D KL ( P ∥ Q ) + D KL ( Q ∥ P )] .

Original: To obtain ˆ s , we take gradientassent with respect to the loss function D J ( ⇡ ✓ ( ·| s ) k ⇡ ✓ ( ·| ˆ s )) and restrict the outputs to the B d ( s, ✏ )92 set, where ⇡ ✓ is a learned CQL policy and ✏ controls the strength of attack.
Revised: To obtain ˆ s , we take gradient assent with respect to the loss function D J ( π θ ( ·| s ) ∥ π θ ( ·| ˆ s )) and restrict the outputs to the B d ( s, ϵ ) set, where π θ is a learned CQL policy.

Original:  In the walker-medium-v2 task from D4RL [9], we use various setups of ✏ for adversarial attack toevaluate the robustness of CQL policies.
Revised: We remark that the the perturbation is applied on normalized observations following prior work [71]. In the walker-medium-v2 task from D4RL [12], we use various ϵ for adversarial attack to evaluate the robustness of CQL policies.

Original: Speciﬁcally, we use ✏ 2 { 0 , 0 . 05 , 0 . 1 , 0 . 14 } to control thestrengths of the attack, where we have ˆ s = s if ✏ = 0 .
Revised: Specifically, we use ϵ ∈ { 0 , 0 . 05 , 0 . 1 , 0 . 14 } to control the strengths of the attack, where we have ˆ s = s if ϵ = 0 .

Original: Given a speciﬁc ✏ , we sample N state-actionpairs { ( s i , a i ) } from the ofﬂine dataset, and then perform adversarial attack to obtain { (ˆ s i , a i ) } and97 also the corresponding Q -values { Q i (ˆ s i , a i ) } , where the Q -function is the trained CQL critic.
Revised: Given a specific ϵ , we sample N state-action pairs { ( s i , a i ) } from the offline dataset, and then perform adversarial attack to obtain { (ˆ s i , a i ) } and the corresponding Q -values { Q i (ˆ s i , a i ) } , where the Q -function is the trained critic of CQL.

Original: Figure 2(a) shows the relationship between ˆ s i and the corresponding Q i with different ✏ .
Revised: Figure 2(a) shows the relationship between ˆ s i and the corresponding Q i with different ϵ .

Original: To visualizeˆ s i , we perform PCA dimensional reduction [47] and choose one of the reduced dimensions torepresent ˆ s i .
Revised: To visualize ˆ s i , we perform PCA dimensional reduction [53] and choose one of the reduced dimensions to represent ˆ s i .

Original:  With the increase of ✏ in the adversarial attack, the Q -curve has greater deviationcompared to the curve with ✏ = 0 .
Revised: More details can be found in Appendix B.3. With the increase of ϵ in the adversarial attack, the Q -curve has greater deviation compared to the curve with ϵ = 0 .

Original: The result signiﬁes that the Q -function of CQL is not smooth instate space, which makes the adversarial noises easily affect the Q -function.
Revised: The result signifies that the Q -function of CQL is not smooth in the state space, which makes the adversarial noises easily affect the Q values.

Original: As a comparison, weapply the proposed conservative smoothing loss in CQL training (i.e., CQL-smooth ) and use the sameevaluation method to obtain ˆ s i and Q i .
Revised: As a comparison, we apply the proposed conservative smoothing loss in CQL OfflineDataset ̂𝑠,𝑎 Perturbation OOD sampling ̂𝑠, $𝑎

Original: According to the result in Figure 2(b), the value functionbecomes smoother. In addition, we show how the adversarial attack affects the ﬁnal performance of ofﬂine RL policies.
Revised: Uncertainty Penalty

Original: We use ✏ 2 [0 , 0 . 15] to evaluate both the original CQL policies (i.e., CQL ) and CQL with conservative OfflineDataset
Revised: Conservative Smooth Ensemble Soft Q -Learning Bellman

Original: [Figure]
Revised: [Figure]

Original: while not converged do Sample mini-batch transitions ( s, a, r, s 0 ) from D . Perform extended soft Q -learning to train multiple Q -functions { Q  1 , . .
Revised: Initialize policy π θ and Q [Table] functions { Q ϕ 1 , . .

Original: . , Q  K } .
Revised: . , Q ϕ K } .

Original:  Sample ˆ s from B d ( s, ✏ ) to obtain (ˆ s, a ) pairs.
Revised: while not converged do Sample mini-batch transitions ( s, a, r, s ′ ) from D . Sample ˆ s from B d ( s, ϵ ) to obtain (ˆ s, a ) pairs.

Original: Perform adversarial smoothing for the Q -network.
Revised: Calculate the Q smooth loss L smooth .

Original:  Calculate uncertainty to penalize (ˆ s, ˆ a ) for conservatism.
Revised: Sample OOD actions ˆ a ∼ π θ (ˆ s ) . Calculate uncertainty u (ˆ s, ˆ a ) and the OOD loss L ood .

Original: Combiningterms to train the critic with Eq. Train the policy  withsmooth constraint as Eq. 
Revised: Train each Q function Q ϕ i with Eq. Train the policy π θ with Eq. training (i.e., CQL-smooth ) and use the same evaluation method to obtain ˆ s i and Q i .

Revised: According to the result in Figure 2(b), the value function becomes smoother.

Revised: In addition, we show how the adversarial attack affects the final performance of offline RL policies.

Original: smoothing calculated smoothing loss (i.e., CQL-smooth ) in adversarial attack.
Revised: We use ϵ ∈ [0 , 0 . 15] to evaluate both the original CQL policies (i.e., CQL ) and CQL with conservative smoothing loss (i.e., CQL-smooth ) in adversarial attack.

Original: Figure 2(c) shows the performance withdifferent settings of ✏ .
Revised: Figure 2(c) shows the performance with different settings of ϵ .

Original: We ﬁnd that our smooth constraints signiﬁcantly improve the robustness of
Revised: We find that our smooth constraints significantly improve the robustness of

Original: CQL, especially for large adversarial noises.
Revised: CQL, especially for large adversarial noises.

Original:  Robust Ofﬂine RL via Conservative Smoothing
Revised:  Robust Offline RL via Conservative Smoothing

Original: In RORL, we develop smooth regularization on both the policy and the value function for statesnear the dataset.
Revised: In RORL, we develop smooth regularization on both the policy and the value function for states near the dataset.

Original: The smooth constraints make the policy and  Q -function robust to perturbation inthe dataset (for ofﬂine training) and interactive samples (for online evaluation).
Revised: The smooth constraints make the policy and the Q -functions robust to observation perturbations.

Original: Nevertheless, thesmoothness may  lead to value overestimation of the edge of the supported dataset.
Revised: Nevertheless, the smoothness may also lead to value overestimation in areas outside the supported dataset.

Original: To address thisproblem, we adopt bootstrapped Q-functions [33] for uncertainty quantiﬁcation and sample OODactions for penalization.
Revised: To address this problem, we adopt bootstrapped Q -functions [38, 6] for uncertainty quantification and sample perturbed states and OOD actions for penalization.

Original: RORL obtains conservative and smooth value estimation on both OODstates and actions, which improves the generalization ability of ofﬂine RL algorithms.
Revised: RORL obtains conservative and smooth value estimation on OOD states, which can improve the generalization ability of offline RL algorithms.

Original: The overallarchitecture of RORL is given in Figure 3.
Revised: The overall architecture of RORL is given in Figure 3.

Original: Robust Q -function We sample three sets of state-action pairs and apply different loss functionsto obtain a conservative  smooth policy.
Revised: Robust Q -function We sample three sets of state-action pairs and apply different loss functions to obtain a conservative and smooth policy.

Original: Speciﬁcally, for a ( s, a ) pair sampled from D , we constructa perturbation set B d ( s, ✏ ) to obtain (ˆ s, a ) pairs, where ˆ s 2 B d ( s, ✏ ) and ✏ .
Revised: Specifically, for a ( s, a ) pair sampled from D , we construct a perturbation set B d ( s, ϵ ) to obtain (ˆ s, a ) pairs, where ˆ s ∈ B d ( s, ϵ ) and ϵ is the perturbation scale. The perturbation set B d ( s, ϵ ) =

Original:  Then we perform OODsampling by using the current policy ⇡ ✓ to obtain (ˆ s, ˆ a ) pairs, where ˆ a ⇠ ⇡ ✓ (ˆ s ) .
Revised: { ˆ s : d ( s, ˆ s ) ≤ ϵ } for state s is an ϵ -radius ball measured in metric d ( , ) , which is the ℓ ∞ norm in our paper. Then we perform OOD sampling by using the current policy π θ to obtain (ˆ s, ˆ a ) pairs, where ˆ a ∼ π θ (ˆ s ) .

Original: RORL contains K 
Revised: RORL contains K ensemble Q -functions.

Original: Q -functions trained by Soft Q -learning as critics.

Original: We denote the parameters of  i -th Q -function andthe target Q -function as  i and  0 i , respectively.
Revised: We denote the parameters of the i -th Q -function and the target Q -function as ϕ i and ϕ ′ i , respectively.

Original: In the following, we give different learning targetsfor ( s, a ) , (ˆ s, a ) , and (ˆ s, ˆ a ) pairs, respectively.
Revised: In the following, we give different learning targets for ( s, a ) , (ˆ s, a ) , and (ˆ s, ˆ a ) pairs.

Original: First, for a ( s, a ) pair sampled from D , we apply extended soft Q -learning to obtain the target as
Revised: First, for a ( s, a ) pair sampled from D , we apply extended soft Q -learning to obtain the target as

Original: [Equation]
Revised: [Equation]

Original: where the next- Q function takes minimum value among the target Q -functions.
Revised: where the next- Q function takes minimum value among the target Q -functions and log π θ ( a ′ | s ′ ) is the entropy regularization. Note that Eq.

Original:  Then, for a (ˆ s, a ) pair with a perturbed state, we enforce smoothness in  Q -function by minimizing the Q -value difference between Q ( s, a ) and Q (ˆ s, a ) .
Revised: (1) is the same learning target of SAC- N in [2]. Then, for a (ˆ s, a ) pair with a perturbed state, we enforce smoothness in each Q -function by minimizing the Q -value difference between Q ( s, a ) and Q (ˆ s, a ) .

Original: In particular, we choose an adversarial ˆ s 2 B d ( s, ✏ )that maximizes  L ( Q (ˆ s, a ) , Q ( s, a )) , and then  minimize this loss function  with the adversarial ˆ s ,where L is a distance measure.
Revised: In particular, we choose an adversarial ˆ s ∈ B d ( s, ϵ ) that maximizes a inner objective L ( Q (ˆ s, a ) , Q ( s, a )) , and then train each Q -function to minimize a loss function L smooth with the adversarial ˆ s .

Original: Intuitively, we wantthat the Q -function is smooth with the mostdifﬁcult (i.e., adversarial) attack in B d ( s, ✏ ) , where ✏ controls the perturbation scale.
Revised: Intuitively, we want the Q -function to be smooth under the most difficult (i.e., adversarial) perturbation in B d ( s, ϵ ) .

Original: To solve such amin-max problem, we maximize L ( Q (ˆ s, a ) , Q ( s, a )) to obtain ˆ s , and then minimize the loss functionwith the selected ˆ s .
Revised: The smooth loss function for Q ϕ i is as follows:

Original: [Equation]
Revised: [Equation]

Original: where L ( · , · ) is the smooth loss function, and we denote  ( s, ˆ s, a ) =
Revised: We denote δ ( s, ˆ s, a ) =

Original: Q  i (ˆ s, a )  Q  i ( s, a ) .
Revised: Q ϕ i (ˆ s, a ) − Q ϕ

Original: Weremark that if  ( s, ˆ s, a ) > 0 , the perturbed state may induce an overestimated Q -value that we needto smooth.
Revised: i ( s, a ) and remark that if δ ( s, ˆ s, a ) > 0 , the perturbed state may induce an overestimated Q -value that we need to smooth.

Original: In contrast, if  ( s, ˆ s, a ) < 0 , the perturbed Q -function is underestimated, which doesnot cause a serious problem in ofﬂine RL.
Revised: In contrast, if δ ( s, ˆ s, a ) < 0 , the perturbed Q -function is underestimated, which does not cause a serious problem in offline RL.

Original: As a result, we use different weights for  ( s, ˆ s, a ) + and ( s, ˆ s, a )  .
Revised: As a result, we use different weights for δ ( s, ˆ s, a ) + and δ ( s, ˆ s, a ) − , where x + = max( x, 0) and x − = min( x, 0) .

Original: The deﬁnition of L ( · , · ) is give as follows,
Revised: The definition of L ( · , · ) is give as follows:

Original: [Equation]
Revised: [Equation]

Original: where we  choose ⌧  0 .
Revised: where we can choose τ ≤ 0 .

Original: 5 , x + = max( x, 0) and x  = min( x, 0) . In this term, we does not introduce OOD action ˆ a for smoothing since the actions are desired to be close to the behavior actionsin thedataset for areas near the ofﬂine data.
Revised: In L smooth , we does not introduce OOD action ˆ a for smoothing since the actions are desired to be close to the behavior actions for areas near the offline dataset.

Original: Finally, to prevent overestimation of OOD states and actions, we use bootstrapped uncertainty u
Revised: Finally, to prevent overestimation of OOD states and actions, we use bootstrapped uncertainty u

Original: (ˆ s, ˆ a )as a penalty for Q (ˆ s, ˆ a ) , where ˆ a ⇠ ⇡ (ˆ s ) is an OOD action sampled from the current policy ⇡ .
Revised: (ˆ s, ˆ a ) as the penalty for Q (ˆ s, ˆ a ) , where ˆ a ∼ π θ (ˆ s ) is an OOD action sampled from the current policy π θ .

Original: We remark that a similar OOD sampling is also used in PBRL [3].
Revised: We remark that a similar OOD sampling is also used in PBRL [6].

Original: The difference is that PBRLonly penalizes the OOD actions, while RORL penalizes both the OOD states andactions to provideconservatism for unfamiliar areas.
Revised: The difference is that PBRL only penalizes the OOD actions for in-distribution states, while RORL penalizes both the OOD states and

Original: The ensemble technique in RL comes from Bootstrapped DQN[33]. The ensemble forms an estimation of the Q -posterior, which yields diverse predictions on areaswith scarce data. u (ˆ s, ˆ a ) quantiﬁes the deviation of a datapoint from the ofﬂine dataset [2, 3].
Revised: OOD actions to provide conservatism for unfamiliar areas.

Original: Wefollow PBRL and use a loss function as:
Revised: We follow PBRL and use a loss function as:

Original: [Equation]
Revised: [Equation]

Original: where the pseudo-target for the OOD datapoints is computed as: 
Revised: where the pseudo-target for the OOD datapoints is computed as: (cid:98) T ood

Original: [Equation]
Revised: Q ϕ

Original: u (ˆ s, ˆ a ) , where u (ˆ s, ˆ a ) := [Equation]
Revised: i (ˆ s, ˆ a ) := Q ϕ i (ˆ s, ˆ a ) − u (ˆ s, ˆ a ) , which is detached from gradients similar to the conventional TD target.

Original: 
Revised: The bootstrapped uncertainty u (ˆ s, ˆ a ) is defined as the standard deviation among the Q -ensemble:

Original: [Equation]
Revised: [Equation]

Original: , The bootstrapped uncertaintyu (ˆ s, ˆ a ) is deﬁned as the standard deviation among the Q -ensemble.
Revised: The ensemble technique [38] forms an estimation of the Q -posterior, which yields diverse predictions and large penalty u (ˆ s, ˆ a ) on areas with scarce data.

Original: Combining the loss functions above, RORL has the following loss function for each Q  i :
Revised: Combining the loss functions above, RORL has the following loss function for each Q ϕ i :

Original: [Equation]
Revised: [Equation]

Original: Robust Policy We learn a robust policy by using a smooth constraint to make the policy changeless with perturbations.
Revised: Robust Policy We learn a robust policy by using a smooth constraint to make the policy change less under perturbations.

Original: Similarly, we choose an adversarial state ˆ s 2 B d ( s, ✏ ) that maximizes
Revised: Similarly, we choose an adversarial state ˆ s ∈ B d ( s, ϵ ) that maximizes

Original: D J  ⇡ ✓ ( ·| s ) k ⇡ ✓ ( ·| ˆ s )  , and then minimize the policy difference between ⇡ ✓ ( ·| s ) and ⇡ ✓ ( ·| ˆ s ) .
Revised: D J (cid:0) π θ ( ·| s ) ∥ π θ ( ·| ˆ s ) (cid:1) , and then minimize the policy difference between π θ ( ·| s ) and π θ ( ·| ˆ s ) .

Original: To159 conclude, we minimize the following loss function for the policy,
Revised: To conclude, we minimize the following loss function for π θ :

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst term aims to maximize the minimum of the ensemble Q -functions to obtain aconservative policy, and the third term is regularization of entropy.
Revised: where the first term aims to maximize the minimum of the ensemble Q -functions to obtain a conservative policy, and the second term is the entropy regularization.  Theoretical Analysis We analyze a simplified learning objective of RORL in linear MDPs [22, 23], where the feature map of the state-action pair takes the form of ϕ : S ×

Revised: A → R d , and both the transition function and the reward function are assumed to be linear in ϕ .

Original: Theoretical Analysis
Revised: The parameter (cid:101) w t of RORL can be solved in closed form following the least squares value iteration (LSVI), which minimizes the following loss function. [Equation] where we have Q w ( s it , a it ) = ϕ

Revised: ( s it , a it ) ⊤ w since the Q -function is also linear in ϕ .

Revised: The first term in Eq.

Revised: (7) is the ordinary TD-error, where we consider the setting of γ = 1 and the Q -target is y it =

Revised: r ( s it , a it )

Original: {++ V t +
+}
Revised: {++ V t +
+}

Revised: ( s it +1 ) .

Revised: The second term is the proposed conservative smoothing loss.

Revised: Specifically, ˆ s it ∼ D

Revised: ood

Revised: ( s it ) are sampled from a l ∞ ball of center s it and norm ϵ > 0 , which can also be formulated as ˆ s it

Revised: ∼ B d

Revised: ( s it , ϵ ) .

Revised: The third term is the additional OOD-sampling loss, which enforces conservatism for OOD states and OOD actions.

Revised: In contrast to PBRL [6], we use perturbed states sampled from D ood = m (cid:83) i

Revised: =1 D ood ( s it ) rather than states from dataset.

Revised: The OOD action ˆ a is sampled from policy π .

Revised: The explicit solution of Eq.

Revised: (7) takes the following form:

Revised: [Equation]

Revised: where the covariance matrix (cid:101) Λ t is defined as

Revised: [Equation]

Revised: We denote the first term and the second term as (cid:101) Λ in and (cid:101) Λ ood t , which represent the covariance matrices induced by the offline samples and OOD samples, respectively.

Revised: Nevertheless, in linear MDPs, it is difficult to ensure the covariance (cid:101) Λ in + (cid:101) Λ ood t ⪰ λ · I , since it requires that the embeddings of the samples are isotropic to make the eigenvalues of the corresponding covariance matrix lower bounded.

Revised: This condition holds if we can sample embeddings uniformly from the whole embedding space.

Revised: However, since the offline dataset has limited coverage in the state-action space and the OOD samples come from limited l ∞ -balls around the offline data, (cid:101) Λ in + (cid:101)

Revised: Λ ood t cannot be guaranteed to be positive definite.

Revised: PBRL [6] uses the assumption of (cid:101)

Revised: Λ ood t ⪰ λ · I , while it is unachievable empirically.

Revised: In RORL, we solve this problem by introducing an additional conservative smoothing loss, which induces a covariance matrix as (cid:101) Λ ood_diff t

Revised: [Equation]

Revised: [Equation]

Revised: the third term in Eq.

Revised: The following theorem gives the guarantees of (cid:101) Λ ood_diff t ⪰ λ · I .

Revised: Theorem 1.

Revised: Assume ∃ i ∈

Revised: [1 , m ] the vector group of all ˆ s it ∼ D ood

Revised: ( s it ) : { ϕ (ˆ s it , a it ) − ϕ

Revised: ( s it , a it ) } be full rank, then the covariance matrix (cid:101) Λ ood _ diff t is positive-definite: (cid:101) Λ ood _ diff t ⪰ λ

Revised: I where λ > 0 .

Revised: Recall the covariance matrix of PBRL is (cid:101) Λ PBRL t

Revised: = (cid:101) Λ in t + (cid:101) Λ ood t , and RORL has a covariance matrix as (cid:101) Λ t = (cid:101) Λ PBRL t

Revised: + (cid:101) Λ ood_diff t , we have the following corollary based on Theorem 1.

Revised: Corollary 1.

Revised: Under the linear MDP assumptions and conditions in Theorem 1, we have (cid:101) Λ t ⪰ (cid:101) Λ PBRL t .

Revised: Further, the covariance matrix (cid:101) Λ t of RORL is positive-definite: (cid:101) Λ t ⪰ λ · I , where λ > 0 .

Revised: Recent theoretical analysis shows that an appropriate uncertainty quantification is essential to provable efficiency in offline RL [23, 63, 6].

Revised: Pessimistic Value Iteration [23] defines a general ξ -uncertainty quantifier as the penalty and achieves provable efficient pessimism in offline RL.

Revised: In linear MDPs,

Revised: Lower Confidence Bound (LCB)-penalty [1, 22] is known to be a ξ -uncertainty quantifier for appropriately selected β t as Γ lcb ( s t , a t )

Revised: =

Revised: β t · (cid:2) ϕ

Revised: ( s t , a t ) ⊤ Λ − 1 t ϕ ( s t , a t ) (cid:3) 1 / 2 .

Revised: Following the analysis of PBRL [6], since the bootstrapped uncertainty is an estimation of the LCB-penalty and the OOD sampling provides a covariance matrix (cid:101) Λ t ⪰ λ · I given in Corollary 1, the proposed RORL also forms a valid ξ -uncertainty quantifier.

Revised: This allows us to further characterize the optimality gap based on the pessimistic value iteration [23, 6].

Revised: We have the following suboptimality gap under linear MDP assumptions.

Revised: Corollary 2.

Revised: [Equation]

Revised: Detailed proof can be found in Appendix A. Corollary 2 indicates that RORL enjoys a tighter suboptimality bound than PBRL [6].

Revised:  Experiments

Revised: We evaluate our method on the D4RL benchmark [12] with various continuous-control tasks and datasets.

Revised: We compare RORL with several offline RL algorithms, including (i) BC that performs

Revised: [Table]

Revised: behavior cloning, (ii) CQL [28] that learns conservative value function for OOD actions, (iii) EDAC [2] that learns a diversified Q -ensemble to enforce conservatism, and (iv) PBRL [6] that performs uncertainty penalization and OOD sampling.

Revised: We also include a basic SAC-10 algorithm as a baseline [2], which is an extension of SAC with 10 Q -functions.

Revised: Among these methods, EDAC [2] and

Revised: PBRL [6] are related to RORL since all these methods apply Q -ensemble for conservatism.

Revised: EDAC needs much more Q -networks (i.e., 10 ∼ 50) for hopper tasks than PBRL and RORL that only use 10 Q -networks.

Revised: For fair comparison, we also report the reproduced results of EDAC-10.

Revised: To assign uniform adversarial attack budget on each dimension of observations, we normalize the observations for SAC-10, EDAC and RORL.

Revised: Besides, we use different perturbation scales for the policy smoothing loss, the Q smoothing loss and the OOD loss, namely ϵ P , ϵ Q and ϵ ood .

Revised: More hyper-parameters and implementation details are provided in Appendix B.

Revised: 6.1 Benchmark Results

Revised: We evaluate each method on Gym domain that includes three environments (HalfCheetah, Hopper, and

Revised: Walker2d) with five types of datasets (random, medium, medium-replay, medium-expert, and expert) for each environment.

Revised: The medium-replay dataset contains experiences collected in training a mediumlevel policy.

Revised: The random/medium/expert dataset is generated by a single random/medium/expert policy.

Revised: The medium-expert dataset is a mixture of medium and expert datasets.

Revised: For benchmark experiments, we set small perturbation scales ϵ P , ϵ Q , and ϵ ood within { 0 .

Revised: 01 } when training RORL and do not include observation perturbation in the testing time.

Revised: Table 1 reports the performance of the average normalized score with standard deviation.

Revised: (i) SACis unstable on several walker2d and hopper tasks since the ensemble number is relatively small to provide reliable uncertainties for SAC- N [2].

Revised: (ii) EDAC solves this problem by gradient diversity constraints while still requiring 10 ∼ 50 Q -networks to obtain reasonable performance.

Revised: In contrast,

Revised: RORL only uses 10 ensemble Q -networks to achieve better or comparable performance with EDAC.

Revised: Additionally, we also show that RORL outperforms EDAC-10 by a large margin.

Revised: (iii) PBRL chooses an alternative OOD-sampling technique to reduce the ensemble numbers.

Revised: According to the result,

Revised: RORL significantly outperforms PBRL with the same ensemble number.

Revised: The reason is RORL additionally uses conservative smoothing loss for perturbed states and penalizes values of these states based on uncertainty estimation, which may improve the generalization ability of the learned policy on continuous state space.

Revised: We remark that RORL significantly improves over the current

Revised: SOTA results on walker2d and hopper tasks, probably because these two tasks require a more precise balance of conservatism and robustness for better performance.

Revised: 6.2 Adversarial Attack

Revised: We adopt three attack methods, namely random , action diff , and min Q following prior works [71, 42].

Revised: Given perturbation scale ϵ , the later two methods perform adversarial perturbation on observations

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: and are given access to the agent’s policy and value functions.

Revised: Details about the three attack methods are as follows.

Revised: • random uniformly samples perturbed states in an l ∞ ball of norm

Revised: ϵ . • action diff is an effective attack based on the agent’s policy and is proved to be an upper bound on the performance difference between perturbed and unperturbed environments [71].

Revised: It directly finds perturbed states in an l ∞ ball of norm ϵ to satisfy:

Revised: [Equation]

Revised: • min Q requires both the agent’s policy and value function to perform a relatively stronger attack.

Revised: The attacker finds a perturbed state to minimize the expected return of taking an action from that state: min ˆ s ∈ B d

Revised: [Equation]

Revised: s,ϵ ) Q

Revised: [Equation]

Revised: s, π

Revised: [Equation]

Revised: (ˆ s )) .

Revised: For ensemble-based algorithms, Q is set as the mean of ensemble Q functions.

Revised: In our experiments, the two objectives of action diff and min Q are optimized via two ways.

Revised: Specifically, we optimize the objectives through:

Revised: (1) selecting the best perturbed state from uniformly sampled 50 states, which has the advantage of simplicity and little computation cost.

Revised: For attacks with this type of optimization, we use their original names without specifying.

Revised: (2) uniformly sampling 20 initial states and performing gradient decent for 10 steps with a step size of 110 ϵ from each initial state to find the best perturbed state.

Revised: Note that we need to clip

Revised: the perturbed states within the l ∞ ball at the end of each optimization step.

Revised: Among the attacks using this optimization, we specifically remark "mixed-order" in their names.

Revised: We compare RORL with ensemble-based baselines EDAC and SAC-10 on halfcheetah-medium-v2, walker2d-medium-v2, and hopper-medium-v2 datasets.

Revised: To handle large adversarial noise, we set the perturbation scales ϵ P , ϵ Q and ϵ ood within { 0 . 01 , 0 . 03 , 0 . 05 , 0 . 07 } in RORL’s training phase.

Revised: More detailed description can be found in Appendix B.

Revised: The results are shown in Figure 4.

Revised: In the results, RORL exhibits improved robustness than other baselines under five types of adversarial attacks.

Revised: On the other hand, we find that random attack is not effective for ensemble-based offline

Revised: RL algorithms, and the “mixed order” attack brings more significant performance drop than vanilla zero-order optimization.

Revised: [Figure]

Revised: 6.3 Ablations

Revised: We conduct ablation studies on the walker2d-medium-v2 dataset to evaluate the importance of three terms, i.e., the policy smoothing loss, the Q smoothing term and the OOD loss.

Revised: From the results in

Revised: Figure 5, we can conclude that each loss contributes to the performance of RORL under adversarial observation attacks.

Revised: The OOD loss is the most essential term, without which the performance is worse than RORL at almost all perturbation scales and all types of attacks.

Revised: The policy smoothing loss is also important, especially for perturbation scales larger than 0.2.

Revised: In addition, Q smooth loss has the minimal impact, which is reasonable since the basic algorithm SAC-10 is based on 10 ensemble Q networks.

Revised: More ablations on the number of Q networks, the effect of ϵ ood and τ , and a comparison with more baselines can be found in Appendix C.

Revised: 6.4 Computational Cost Comparison

Revised: [Table]

Revised: We compare the computational cost of RORL with prior works on a single machine with one GPU (Tesla V 32G).

Revised: For each method, we measure the average epoch time (i.e., 1 × 10 3 training steps) and the GPU memory usage on the hopper-medium-v2 task.

Revised: More discussions are provided in Appendix C.1.

Revised: As shown in Table 2, RORL runs slightly faster than CQL and much faster than PBRL.

Revised: PBRL is so slow because it uses 10 Q networks and needs OOD action sampling.

Revised: In

Revised: RORL, we also include the OOD state-action sampling and the robust training procedure, but we implemented these procedures efficiently based on the parallelization of Q networks.

Revised: Even so, RORL is still slower than SAC-10 and EDAC.

Revised: As demonstrated in our experiments, RORL enjoys significantly better robustness than EDAC and SACunder adversarial perturbations.

Revised: Regarding the GPU memory consumption, RORL uses comparable memory to PBRL and EDAC, with only 16 .

Revised: 7% more memory usage.

Revised:  Related Works

Revised: Offline RL Research related to offline RL has experienced explosive growth in recent years.

Revised: In model-free domain, offline RL methods focus on correcting the extrapolation error [14] in the off- policy algorithms.

Revised: The natural idea is to regularize the learned policy near the dataset distribution [57, 61, 36, 59, 67, 13, 64].

Revised: For example, MARVIL reweights the policy with exponential advantage, which implicitly guarantees the policy within the KL-divergence neighborhood of the behavior policy.

Revised: Another stream of model-free methods prevents the selection of OOD actions by penalizing their

Revised: Q -value [27, 28, 2, 10] or V -learning [32, 26].

Revised: With the ensemble Q networks and the additional loss term to diversify their gradients, EDAC [2] achieves SOTA performance in the D4RL benchmark.

Revised: Instead of diversifying gradients, PBRL [6] proposes an explicit value underestimation of OOD actions according to the uncertainty, which requires fewer ensemble networks.

Revised: Inspired by EDAC and PBRL, we build our work upon ensemble networks, focusing more on the smoothness over the state space.

Revised: Besides the surprising empirical results, theoretical analysis of offline reinforcement learning algorithms is of increasing interest [9, 23, 44, 63, 68].

Revised: Though the assumptions for the dataset vary in the different papers, they all suggest that pessimism and conservatism are necessary for offline RL.

Revised: Our theoretical results can be viewed as robust extensions to previous theoretical results [23, 6].

Revised: Robust RL The research line of robust RL can be traced back to H ∞ -control theory [62, 7], where policies are optimized to be well-performed in the worst possible deterministic environment.

Revised: Depending on the definition, there are different streams of research on robust RL.

Revised: As the extension of robust control to MDPs, Robust MDPs (RMDPs) [37, 20, 45, 18] are proposed to formulate the perturbation of transition probabilities for MDPs.

Revised: Though some recent analyses with theoretical guarantees come out under specific assumptions for RMDPs [73, 66, 30], there is currently no practical algorithm to solve RMDPs in a large-scale problem, expect some linear approximation attempt [52].

Revised: In online RL, domain randomization [54, 34] assumes the model uncertainty can be predefined in data collection by changing the setup of a simulator.

Revised: However, it is not practical for offline RL.

Revised: Robust Adversarial Reinforcement Learning (RARL) [42] and Noisy Robust Markov

Revised: Decision Process (NR-MDP)

Revised: [24] study the robust RL with the perturbed actions, showing that the policy robustness to adversarial or noisy actions can also induce robustness for model parameter changes.

Revised: The most related work to ours is SR 2 L [47], which shows policy smoothing can lead to significant performance improvement in the online setting.

Revised: In contrast, we focus on the offline setting and tackle the potential overestimation of perturbed states.

Revised: Another related work is S4RL [49], where the authors study different data augmentation methods to smooth observations in offline RL.

Revised: Their result supports the necessity of state smoothing.

Revised: More related works are discussed in Appendix D.

Revised:  Conclusion

Revised: We propose Robust Offline Reinforcement Learning (RORL) to trade-off conservatism and robustness for offline RL.

Revised: To achieve that, we introduce the conservative smoothing technique for the perturbed states while actively underestimating their values based on pessimistic bootstrapping to keep conservative.

Revised: We show that RORL can achieve comparable or even better performance with fewer ensemble

Revised: Q networks than previous methods in the offline RL benchmark.

Revised: In addition, we demonstrate that

Revised: RORL is considerably robust to adversarial perturbations across different types of attacks.

Revised: We hope our work can promote the application of offline RL under real-world engineering conditions.

Revised: The main limitation of our method is that the adversarial state sampling slows down the computing process, which may be improved in future work.

Revised: Also, an interesting direction is to smooth or penalize the policy and Q functions in latent spaces rather than the normalized observation space.

Revised: Acknowledgements

Revised: This work was in part supported by Tencent Robotics X and Shanghai AI Laboratory, and in part by

Revised: Science and Technology Innovation 2030 – “New Generation Artificial Intelligence” Major Project (No. 2018AAA0100904) and National Natural Science Foundation of China (62176135).

Revised: The authors would like to thank the anonymous reviewers.

Revised: Rui Yang thanks Yi Wang and Haoyi Song for valuable discussion.

Revised: Checklist

Revised: The checklist follows the references.

Revised: Please read the checklist guidelines carefully for information on how to answer these questions.

Revised: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .

Revised: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.

Revised: For example:

Revised: • Did you include the license to the code and datasets?

Revised: [Yes] See Section ?? .

Revised: • Did you include the license to the code and datasets?

Revised: [No] The code and the data are proprietary.

Revised: • Did you include the license to the code and datasets?

Revised: [N/A]

Revised: Please do not modify the questions and only use the provided macros for your answers.

Revised: Note that the

Revised: Checklist section does not count towards the page limit.

Revised: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Revised: [Yes] (b) Did you describe the limitations of your work?

Revised: [Yes] (c) Did you discuss any potential negative societal impacts of your work?

Revised: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Revised: [Yes] 2.

Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Revised: [Yes] See Appendix A. 3.

Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Revised: [Yes] See Sec 1 and Appendix B. (b) Did you specify all the training details (e.g., data splits, hyper-parameters, how they were chosen)?

Revised: [Yes] See Appendix B. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Revised: [Yes] See Sec 6.

Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Revised: [Yes] See Appendix B. 4.

Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Revised: [Yes] We cited D4RL [12] and EDAC[2] for their datasets and code.

Revised: (b) Did you mention the license of the assets?

Revised: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?

Revised: [Yes] We included our code in the anonymized link.

Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Revised: [Yes] Opensource code and dataset.

Original: We analyze the simpliﬁed learning objective of RORL in linear MDPs [20, 21], where the featuremap of the state-action pair takes the form of  : S ⇥ A !
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] 5. If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: R d , and the transition kernel and thereward function are assumed to be linear in  .
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] A Theoretical Analysis In this section, we provide detailed theoretical analysis and proofs in linear MDPs [22].

Original: The parameter e w t of RORL can be solved in closedform following the least squares value iteration algorithm (LSVI), which minimizes the followingloss function.
Revised: In linear MDPs, we assume that the transition dynamics and reward function take the form of P t ( s t +1 | s t , a t ) = ⟨ ψ ( s t +1 ) , ϕ ( s t , a t ) ⟩ , [Equation]

Revised: ∀

Original: [Equation] 
Revised: [Equation] × [Equation]

Original:  where we have Q w ( s it , a it )) =
Revised: × S [Equation] (10) where the feature embedding ϕ : S × A (cid:55)→ R d is known.

Original:  ( s it , a it ) > w since the Q -function is also linear in  .
Revised: We further assume that the reward function r : S × A (cid:55)→ [0 , 1] is bounded and the feature is bounded by ∥ ϕ ∥ 2 ≤ 1 .

Original: The ﬁrst termin Eq. (7) is the ordinary TD-error, where we consider the setting of  = 1 and the Q -targetis y it =
Revised: Given the offline dataset D , the parameter w t can be solved in the closed-form by following the LSVI algorithm, which minimizes the following loss function, [Equation] where V t +1 is the estimated value function in the ( t + 1) -th step, and y i t

Original:  r ( s it , a it )
Revised: = r ( s i t , a i t )

Original: + V t +

Revised: + V t +

Original: ( s it +1 ) .
Revised: ( s i t +1 ) is the target of LSVI.

Original: The second term is the proposed conservative smoothing loss.
Revised: The explicit solution to (11) takes the form of [Equation]

Original: Speciﬁcally, ˆ s it ⇠ D ood
Revised: where Λ [Equation] = [Equation]

Original: ( s it ) are sampled from a l 1 ball of center s it and norm ✏ > 0 , which can beformulated as ˆ s ⇠ B d ( s it , ✏ ) .
Revised: In RORL, since we introduce the conservative smoothing loss and the OOD loss to learn the Q value function, the parameter (cid:101) w t of RORL can be solved as follows: [Equation]

Original:  The third term is the additional OOD-sampling loss[3], which enforcesconservatism for OOD actions. Differently, we use perturbed states sampled from D ood = m
Revised: which is a simplified learning objective for linear MDPs. The first term is the ordinary TD-error, the second term is the Q value smoothing loss

Original: S i =
Revised: [Equation]

Original: 1 D ood ( s it )rather than states from dataset in PBRL. The OOD action ˆ a is sampled from policy ⇡ .
Revised: and the third term is the additional OOD loss.

Original: The explicitsolution of Eq.
Revised: The explicit solution of Eq.

Original: (7) takes the following form by following LSVI:
Revised: (13) takes the following form by following LSVI:

Original: [Equation]
Revised: [Equation]

Original: where the covariance matrix e ⇤ t is deﬁned as
Revised: where the covariance matrix (cid:101) Λ t is defined as

Original: [Equation]
Revised: [Equation]

Original: We denote the ﬁrst term and the second term as e [Equation] e ⇤ ood 
Revised: We denote the first term of Eq. (15) as (cid:101) Λ in t , the second term as (cid:101) Λ ood t , and the third term as (cid:101) Λ ood_diff t .

Original: t , which represent the covariance matricesinduced by the ofﬂine samples and OOD samples, respectively. Nevertheless, in linear MDPs, it isdifﬁcult to ensure the covariance
Revised: A.3 ξ -Uncertainty Quantifier

Original: [Equation]
Revised: Theorem (Theorem 1 restate) .

Original: ⌫  I , since it requires that the embeddings of the samplesare isotropic to make the eigenvalues of the corresponding covariance matrix lower bounded. Thiscondition holds if we can sample embeddings uniformly from the whole embedding space. However,since the ofﬂine dataset has limited coverage in the state-action space and the OOD samples comefrom a limited l 1 -ball around the ofﬂine data,
Revised: Assume ∃ i ∈ [1 , m ] the vector group of all ˆ s it ∼ D ood

Original: [Equation]
Revised: ( s it ) : { ϕ (ˆ s it , a it ) − ϕ

Original: cannot be guaranteed to be positive deﬁnite. PBRL [3] uses the assumption of e ⇤ ood [Equation]
Revised: ( s it , a it ) } is full rank, then the covariance matrix (cid:101) Λ ood _ diff t is positive-definite: (cid:101) Λ ood _ diff t ⪰ λ

Original: ⌫  I , while it is unachievable empirically. In RORL, we solvethis problem by introducing an additional conservative smoothing loss, which induces a covariancematrix as
Revised: · I where λ > 0 .

Original: [Equation]
Revised: Proof.

Original: ][  (ˆ s it , a it )   ( s it , a it )] > (i.e.,the third term in Eq.
Revised: For the (cid:101) Λ ood _ diff t matrix (i.e., the third part in Eq.

Original: The following theorem gives the guarantees of e ⇤ ood_diff t ⌫  I . Theorem 1 Assume 9 i 2 [1 , m ] the vector group of all ˆ s it ⇠ D ood 
Revised: (15)), we denote the covariance matrix for a specific i as Φ it . Then we have (cid:101) Λ ood _ diff t

Original: ( s it ) : {  (ˆ s it , a it )   ( s it , a it ) } befull rank, then the covariance matrix e ⇤ ood _ di ↵ t is positive-deﬁnite: e ⇤ ood _ di ↵ t ⌫  · I where  > 0 . Recall the covariance matrix of PBRL is e ⇤ PBRL t
Revised: = (cid:80) mi

Original: ={+=
+}
Revised: {+=
+}

Revised: Φ it .

Revised: In the following, we discuss the condition of positive-definiteness of Φ it .

Original: e ⇤ in t + e ⇤ ood
Revised: For the simplicity of notation, we omit the superscript and subscript of s it and a it for given i and t . Specifically, we define [Equation] where j ∈ { 1 , . . .

Revised: , N } indicates we sample

Revised: | B d ( s it , ϵ ) | = N perturbed states for each s .

Revised: For a nonzero vector y ∈ R d , we have

Revised: [Equation]

Revised: [Equation]

Revised: where the last inequality follows from the observation that (cid:0) ϕ

Revised: (ˆ s j , a ) −

Revised: [Equation]

Revised: a ) (cid:1) ⊤ y is a scalar.

Revised: Then Φ it is always positive semi-definite .

Revised: In the following, we denote z j = ϕ

Revised: (ˆ s j , a ) − ϕ

Revised: ( s, a ) .

Revised: Then we need to prove that the condition to make Φ it positive definite is rank[ z 1 , . . .

Revised: , z N ] =

Revised: d , where d is the feature dimension.

Revised: Our proof follows contradiction.

Revised: In Eq.

Revised: (16), when y ⊤ Φ it y = 0 with a nonzero vector y , we have z ⊤ j y = 0 for all j = 1 , . . .

Revised: , N .

Revised: Suppose the set { z 1 , . .

Revised: , z N } spans R d , then there exist real numbers { α 1 , . .

Revised: . , α N } such that y =

Revised: α 1 z 1 + · · · + α N z N .

Revised: But we have y ⊤ y

Revised: = α 1 z ⊤ 1 y + · · · + α N z ⊤ N y =

Revised: α 1 × 0+ . . .

Revised: + α N × 0 = 0 , yielding that y = 0 , which forms a contradiction.

Revised: Hence, if the set { z 1 , . . .

Original: t , and RORL has a covariance matrix ase ⇤ t = e ⇤ PBRL t
Revised: , z N } spans R d , which is equivalent to rank[ z 1 , . . . , z N ] = d , then Φ it is positive definite . Under the given conditions, we know that ∃ k ∈ [1 , m ] , for any nonzero vector y ∈ R d , y ⊤ Φ kt y > 0 .

Revised: We have y ⊤ (cid:101) Λ ood _ diff t

Revised: y

Revised: =

Original: {+(cid:80) mi =
+}
Revised: {+(cid:80) mi =
+}

Revised: y ⊤ Φ it y ≥

Revised: y ⊤ Φ kt y > 0 .

Revised: Therefore, (cid:101) Λ ood _ diff t is positive definite, which concludes our proof.

Revised: Remark.

Revised: As a special case, when (i) the size of B d ( s it , ϵ ) is sufficient, (ii) the dimension of states is the same as the feature ϕ ( s, a ) and ϕ ( s, a )

Revised: =

Revised: s and (iii) each dimension of the state perturbation

Revised: ˆ s it − s it is independent, the matrix (cid:101) Λ ood_diff t satisfies: (cid:101) Λ ood_diff t =

Revised: [Equation]

Revised: When we use neural networks as the feature extractor, the assumption in the above Theorem needs (i) the size of samples B d ( s it , ϵ ) is sufficient, and (ii) the neural network maintains useful variability for state-action features.

Revised: To obtain the second constraint, we require that the Jacobian matrix of ϕ ( s, a ) has full rank.

Revised: Nevertheless, when we use a network as the feature embedding, such a condition can generally be met since the neural network has high randomness and nonlinearity, which results in the feature embedding with sufficient variability.

Revised: Generally, we only need to enforce a bi-Lipschitz continuity for the feature embedding.

Revised: We denote x 1 = ( s 1 , a ) and x 2 = ( s 2 , a ) as two different inputs.

Revised: x k 1 is the k -th dimension of x 1 .

Revised: The bi-Lipschitz constraint can be formed as

Revised: [Equation]

Revised: where C 1 < C 2 are two positive constants.

Revised: The lower-bound C 1 ensures the features space has enough variability for perturbed states, and the upper-bound can be obtained by Spectral regularization

Original: + e ⇤ ood_diff t , we have the following corollary based on Theorem 1.
Revised: [16] that makes the network easy to coverage. An approach to obtain bi-Lipschitz continuity is to regularize the norm of the gradients by using the gradient penalty as [Equation] In experiments, we do not use explicit constraints (e.g., Spectral regularization) for the upper bound since the state has relatively low dimensions, and we find a small fully connected network does not resulting in a large C 2 empirically.

Original: Corollary 1 Under the linear MDP assumptions, we have e ⇤ t ⌫ e ⇤ PBRL t .
Revised: Recall the covariance matrix of PBRL is (cid:101) Λ PBRL t [Equation] ood [Equation]

Original: Further, the covariancematrix e ⇤ t of RORL is positive-deﬁnite: e ⇤ t ⌫  · I , where  > 0 .
Revised: , and RORL has a covariance matrix as [Equation] , we have the following corollary based on Theorem 1. Corollary (Corollary 1 restate) .

Original:  Recent theoretical analysis shows that an appropriate uncertainty quantiﬁcation is essential to provableefﬁciency in ofﬂine RL [21, 56, 3].
Revised: Under the linear MDP assumptions and conditions in Theorem 1, we have (cid:101) Λ t ⪰ (cid:101) Λ PBRL t . Further, the covariance matrix (cid:101) Λ t of RORL is positive-definite: (cid:101) Λ t ⪰ λ · I , where λ > 0 . Recent theoretical analysis shows that an appropriate uncertainty quantification is essential for provable efficiency in offline RL [23, 63, 6].

Original: Pessimistic Value Iteration [21] deﬁnes a general ⇠ -uncertaintyquantiﬁer as the penalty and achieves provable efﬁcient pessimism in ofﬂine RL. 
Revised: Pessimistic Value Iteration [23] defines a general ξ -uncertainty quantifier as the penalty and achieves provable efficient pessimism in offline RL. We give the definition of a ξ -uncertainty quantifier as follows. Definition 1 ( ξ -Uncertainty Quantifier [23]) . The set of penalization { Γ t } t ∈ [ T ] forms a ξ -Uncertainty

Original: In linear MDPs,
Revised: Quantifier if it holds with probability at least 1 − ξ that | [Equation]

Original:  Lower Conﬁdence Bound (LCB)-penalty [1, 20] is known to be a ⇠ -uncertainty quantiﬁer forappropriately selected  t as  lcb ( s t , a t ) =
Revised: for all ( s, a ) ∈ S × A , where T is the Bellman operator and (cid:98) T is the empirical Bellman operator that estimates T based on the data. In linear MDPs, Lower Confidence Bound (LCB)-penalty [1, 22] is known to be a ξ -uncertainty quantifier for appropriately selected β t as Γ lcb ( s t , a t ) =

Original:  t · ⇥  ( s t , a t ) >
Revised: β t · (cid:2) ϕ

Original: ⇤  1 t  ( s t , a t ) ⇤ 1 / 2 .
Revised: ( s t , a t ) ⊤ Λ − 1 t ϕ ( s t , a t ) (cid:3) 1 / 2 .

Original: Following the analysisof PBRL [3], since the bootstrapped uncertainty is an estimationto the LCB-penalty and the OODsampling provides a covariance matrix of e ⇤ PBRL t , the proposed RORL also form a valid ⇠ -uncertaintyquantiﬁer with the covariance matrix e ⇤ t⌫  · I given in Corollary 1.
Revised: Following the analysis of PBRL [6], since the bootstrapped uncertainty is an estimation of the LCB-penalty, the proposed RORL also form a valid ξ -uncertainty quantifier with the covariance matrix (cid:101) Λ t ⪰ λ · I given in Corollary 1. Theorem 2. For all the OOD datapoint (ˆ s, ˆ a, ˆ y ) ∈ D ood , if we set ˆ y = T V t +1 ( s ood , a ood ) , it then holds for β t

Revised: = O (cid:0) T · √ d · log ( T/ξ ) (cid:1)

Original: This allows us to furthercharacterize the optimality gap based on the pessimistic value iteration [21, 3].
Revised: that [Equation] [Equation] forms a valid ξ -uncertainty quantifier, where (cid:101) Λ t is the covariance matrix of RORL.

Revised: Proof.

Revised: The proof follows that of the analysis of PBRL [6] in linear MDPs [23].

Revised: We define the empirical Bellman operator of RORL as (cid:101) T , then

Revised: [Equation]

Revised: where (cid:101) w t follows the solution in Eq.

Revised: Then it suffices to upper bound the following difference between the empirical Bellman operator and Bellman operator

Revised: [Equation]

Revised: Here we define w t as follows

Revised: [Equation]

Revised: where θ and ψ are defined in Eq.

Revised: It then holds that

Revised: [Equation]

Revised: where we plug the solution of (cid:101) w t in Eq.

Revised: Meanwhile, by the definitions of (cid:101) Λ t and w t in Eq.

Revised: (15) and Eq.

Revised: (19), respectively, we have

Revised: [Equation]

Revised: Plugging Eq.

Revised: (21) into Eq.

Revised: (20) yields

Revised: [Equation]

Revised: where we define

Revised: [Equation]

Revised: Following the standard analysis based on the concentration of self-normalized process [1, 3, 58, 22, 23] and the fact that Λ ood ⪰ λ · I , it holds that

Revised: [Equation]

Revised: with probability at least 1 − ξ , where β t =

Revised: O (cid:0) T · √ d · log ( T/ξ ) (cid:1) .

Revised: Meanwhile, by setting y =

Revised: T V t +1 ( s ood , a ood ) , it holds that (ii) = 0 .

Revised: For (iii), we have (cid:16) ϕ

Revised: (ˆ s it , a it ) ϕ

Revised: [Equation]

Revised: ˆ s it

Revised: [Equation]

Revised: a

Revised: [Equation]

Revised: [Equation]

Revised: a

Revised: [Equation]

Revised: a

Revised: [Equation]

Revised: = ϕ

Revised: [Equation]

Revised: a

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Original: {+Since we enforce smoothness for the value function, we have T V t +
+}
Revised: {+Since we enforce smoothness for the value function, we have T V t +
+}

Revised: (ˆ s it , a it )

Revised: ≈ T V t +1 ( s it , a it ) .

Revised: Thus (iii) ≈ 0 .

Revised: To conclude, we obtain from Eq.

Revised: (22) that

Revised: [Equation]

Revised: for all ( s, a ) ∈ S × A with probability at least 1 − ξ .

Revised: A.4 Suboptimality Gap

Revised: Theorem 2 allows us to further characterize the optimality gap based on the pessimistic value iteration [23].

Revised: First, we give the following lemma.

Revised: Lemma 1.

Revised: Given two positive definite matrix A and B, it holds that:

Revised: [Equation]

Revised: [Equation]

Original: We have  the followingsuboptimality gap under linear MDP assumptions.
Revised: Proof. Leveraging the properties of generalized Rayleigh quotient, we have [Equation] Since B and A − 1 are both positive definite, the eigenvalues of BA − 1 are all positive: λ min (cid:0) BA − 1 (cid:1) > 0 .

Revised: This ends the proof.

Revised: Then, according to the definition of LCB-penalty in Eq.

Revised: (18), since (cid:101) Λ t = (cid:101) Λ PBRL t

Revised: + (cid:101) Λ ood_diff t with (cid:101) Λ ood_diff t ⪰ λ

Revised: I .

Revised: we have the relationship of the LCB-penalty between RORL and PBRL as follows.

Revised: Corollary 3.

Revised: Suppose Λ PBRL t is positive definite.

Revised: The RORL-induced LCB-penalty term is less than the

Revised: PBRL-induced LCB-penalty, as Γ lcb t ( s t , a t ) =

Revised: β

Revised: [Equation]

Revised: (cid:3)

Revised: [Equation]

Revised: <

Revised: [Equation]

Revised: lcb _ PBRL

Revised: [Equation]

Revised: ( s

Revised: [Equation]

Revised: , a

Revised: [Equation]

Revised: Proof.

Revised: Since (cid:101) Λ t = (cid:101) Λ PBRL t

Revised: [Equation]

Original: Corollary 2 SubOpt
Revised: [Equation] t and (cid:101) Λ ood_diff t ⪰ λI , we have ϕ [Equation] where the inequality directly follows Lemma 1.

Original:  [Equation]
Revised: Then we have [Equation]

Original: We refer to Appendix A for detailed proof.
Revised: Theorem 2 and Corollary 3 allow us to further characterize the optimality gap of the pessimistic value iteration. In particular, we have the following suboptimality gap under linear MDP assumptions. Corollary (Corollary 2 restate) . Under the same conditions as Theorem 2, it holds that

Original:  The second inequality is directly induced by  lcb t ( s t , a t ) <
Revised: SubOpt [Equation] We refer to Jin et al [23] for a detailed proof of the first inequality. The second inequality is directly induced by Γ lcb t ( s t , a t ) <

Original:  lcb _ PBRL t ( s t , a t ) .
Revised: Γ lcb _ PBRL t ( s t , a t ) in Corollary 3.

Original:  Therefore, RORL enjoys a tighter suboptimality bound than PBRL [3].
Revised: The optimality gap is informationtheoretically optimal under the linear MDP setup with finite horizon [23]. Therefore, RORL enjoys a tighter suboptimality bound than PBRL [6] in linear MDPs.

Original: Experiments
Revised: B Implementation Details and Experimental Settings

Original: We evaluate our method on D4RL benchmark [10] with various continuous-control tasks and datasets. We compare RORL with several ofﬂine RL algorithms, including (i) BC that performs behaviorcloning, (ii) CQL [25] that learns conservative value function for OOD actions, (iii) EDAC [2] thatlearns a diversiﬁed Q -ensemble to enforce conservatism, and (iv) PBRL [3] that performs uncertaintypenalization and OOD sampling. Among these methods, EDAC [2] and PBRL [3] are related to RORL since all these methods apply Q -ensemble for conservatism.
Revised: In this section, we provide detailed implementation and experimental settings.

Original: EDAC needs much more Qnetworks (i.e., 10 ⇠ 50) for hopper tasks than PBRL and RORL that use 10 Q -networks. (v)
Revised: B.1 Implementation Details

Original: Wealso include a basic SAC-10 algorithm as a baseline [2], which is [-an extension of SAC with 
-]
Revised: SAC-10 Our SAC-10 implementation is based on [2], which is [-an extension of SAC with 
-]open-source. We keep the default parameters as EDAC [2] except for the ensemble size set to 10 in our paper.

Original: Q -functions.
Revised: In addition, we normalize each dimension of observations to a standard normal distribution for consistency with RORL. The hyper-parameters are listed in Table 3.

Original: [Table]
Revised: [Table]

Original: We evaluate each method on Gym domain that includes three environments (Half Cheetah, Hopper,and Walker2d) with four non-expert datasets (random, medium, medium-replay, and medium-expert). The medium-replay dataset contains experiences collected in training a medium-level policy. Therandom or medium dataset is generated by a single random or medium policy.
Revised: EDAC Our EDAC implementation is based on the open-source code of the original paper [2].

Original: The mediumexpert dataset is a mixture of medium and expert datasets. We set small perturbation scale ✏ to0.001/0.005/0.01 for halfcheetah/hopper/walker2d tasks respectively when training RORL and do notinclude observation perturbation in the testing time. Table 1 reports the performance of the average normalized score with standard deviation. (i) SAC-10is unstable in several Walker2d and Hopper tasks since the ensemble number is relatively smallto provide reliable uncertainties.
Revised: In the benchmark results, we directly report results from the paper which are the previous SOTA performance on the D4RL Mujoco benchmark.

Original: (ii) EDAC solves this problem by using diversity constraintswhile still requiring 10 ⇠ 50 Q -networks to obtain reasonable performance. In contrast, RORL onlyuses 10 ensemble Q -networks to outperform EDAC with 10 ⇠ 50 networks.
Revised: As for other experiments, we also normalize the observations and use 10 ensemble Q networks for consistency with RORL, and set the gradient diversity term η = 1 by default. RORL We implement RORL based on SAC-10 and keep the hyper-parameters the same. The differences are the introduced policy and Q network smoothing techniques and the additional value underestimation on OOD state-action pairs.

Original: (iii) PBRL chooses analternative OOD-sampling technique to reduce the ensemble numbers. According to the result, RORLsigniﬁcantly outperforms PBRL with the same ensemble number.
Revised: In Eq.

Original: The reason is RORL additionallyuses conservative smoothing loss  for both OOD states andactions, which improves the generalizationability of the policy and Q -functions.
Revised: (5), the coefficient β Q for the Q network smoothing loss L smooth is set to 0 . 0001 for all tasks, and the coefficient β ood for the OOD loss L ood is tuned within { 0 . 0 , 0 . 1 , 0 . 5 } . Besides, the coefficient β P of the policy smoothing loss in Eq.

Original: We remark that RORL signiﬁcantly improves over the current SOTA results on walker2d and hopper tasks, which might be because the two tasks require a moreprecise balancing of conservatism and generalization for better performance. 6.2 Adversarial Attack We adopt three attack methods, namely random , action diff , and min Q following prior works [62, 37].
Revised: (6) is searched in { 0 . 1 , 1 . 0 } .

Original: Given perturbation scale ✏ , the later two methods perform adversarial perturbation on observationsand are given access to theagent’s policy and value functions.
Revised: When training the policy and value functions in RORL, we randomly sample n perturbed observations from a l ∞ ball of norm ϵ and select the one that maximizes D J ( π θ ( ·| s ) ∥ π θ ( ·| ˆ s )) or

Original: • random uniformly samples perturbed states in an l 1 ball of norm ✏ .
Revised: L smooth , respectively.

Original: • action diff is an effective attack based on the agent’s policy andis proved to max- imize an upper bound on the performance difference between perturbed and unper-
Revised: We denote the perturbation scales for the Q value functions, the policy, and the OOD loss as ϵ Q , ϵ P and ϵ ood .

Original: turbed environments [62]. It directly ﬁnds states in an l 1 ball of norm ✏ to minimize:
Revised: The number of sampled perturbed observations n is tuned within { 10 , 20 } . The OOD loss underestimates the values for n perturbed states

Original: min ˆ s 2 B d ( s, ✏ )  D J  ⇡ ✓ ( ·| s ) k ⇡ ✓ ( ·| ˆ  s ) .
Revised: ˆ s ∼ B d ( s, ϵ ) with actions sampled from the current policy ˆ a ∼ π θ (ˆ s ) .

Original: • min Q requires both the agent’s policy and value function to perform a relatively stronger attack.
Revised: For each ˆ s , we sample a single ˆ a for the OOD loss. Regarding the Q smoothing loss in Eq.

Original: The attacker ﬁnds a perturbed state to minimize the expected return of taking an
Revised: (3), the parameter τ is set to 0 . 2 in all tasks for conservative value estimation.

Original: action from that state: min ˆ s 2 B d ( s, ✏ ) Q ( s, ⇡ ✓ (ˆ s )) .
Revised: All the hyper-parameters used in RORL for the benchmark experiments and adversarial experiments are listed in Table 4 and Table 5 respectively. Note that for halfcheetah tasks, 10 ensemble Q networks already enforce sufficient pessimism for OOD state-action pairs, thus we do not need additional OOD loss for these tasks. As for the OOD loss L ood in Eq. (4), we remark that the pseudo-target (cid:98) T ood

Revised: Q ϕ

Revised: i (ˆ s, ˆ a ) for the OOD state-action pairs (ˆ s, ˆ a ) can be implemented in two ways: (cid:98) T ood Q ϕ

Revised: i (ˆ s, ˆ a ) :=

Revised: Q ϕ i (ˆ s, ˆ a ) − λu (ˆ s, ˆ a ) and (cid:98) T ood Q ϕ

Revised: i (ˆ s, ˆ a ) :

Revised: = min i =1 ,...,K Q ϕ

Revised: i (ˆ s, ˆ a ) .

Original: For ensemble-based algorithms, Q is set
Revised: We refer to the two targets as the “minus target” and the “min target”, and compare them in Appendix C.14. Intuitively, the “minus target” introduces an additional parameter λ but is more flexible to tune for different environments and different types of data. In contrast, the “min target” requires tuning the number of ensemble Q networks and cannot enforce appropriate conservatism for all tasks given only 10 ensemble Q networks.

Revised: Following PBRL [6]

Revised: [Table]

Revised: we also decay the OOD regularization coefficient λ with decay pace d for each training step to stabilize L ood , because we need strong OOD regularization at the beginning of training and need to avoid too large OOD loss that leads the value function to be fully negative.

Revised: λ

Original: as the mean of ensemble Q functions.
Revised: and d are also listed in the two tables [Table] [Table] [Table]

Revised: B.2 Experimental Settings

Revised: For all experiments, we train algorithms for 3000 epochs (1000 training steps per epoch, i.e., 3 million steps in total) following EDAC [2].

Revised: We use small perturbation scales to train the Q networks and the policy network for the benchmark experiments and relatively large scales for the adversarial attack experiments as listed in Table 4 and Table 5.

Revised: In the benchmark results, we evaluate algorithms for 1000 steps in clean environments (without adversarial attack) at the end of each epoch.

Revised: The reported results are normalized to d4rl scores that measure how the performance compared with the expert score and the random score: thenormalized score = 100 ×

Revised: [Equation]

Revised: score

Original: In our experiments, the two objectives of action diff and min Q are optimized via two ways.
Revised: [Equation] random score . Besides, the benchmark results are averaged over 4 random seeds. Regarding the adversarial attack experiments, we evaluate algorithms in perturbed environments that performing “random”, “action diff”, and “min Q” attack with zerothorder and mixed-order optimizations as discussed in Section 6.2.

Revised: Similar to prior work [71], agents receive observations with malicious noise and the environments do not change their internal transition dynamics.

Revised: We evaluate each algorithm for 10 trajectories (1000 steps per trajectory) and average their returns over 4 random seeds.

Revised: B.3 Visualization Settings of CQL

Revised: For visualizing the relationship between the Q -function and the state space (i.e., Figure 2 and Figure 6), we sample 2560 adversarial transitions from the offline dataset for each attack ϵ and calculate the corresponding Q -function.

Original: Speciﬁcally, we optimize the objectives through:
Revised: Since the state has relatively high dimensions (i.e., 11 or 17), we perform PCA dimensional reduction to reduce the state to 4 dimensions. We find the Q -function generally has a strong correlation to one or two dimensions of the state after dimensional reduction. For other dimensions, the relationship between the Q -value and the PCA-reduced state often has one or two peaks, which has less variety in the curve.

Revised: [Figure]

Revised: C Additional Experimental Results

Original:  [Figure]
Revised: In this section [Figure]

Revised: we present additional ablation studies and adversarial experiments.

Revised: C.1 Computational Cost Comparison

Original: (1) selecting the best state from uniformly sampled 50 states, which has the advantage ofsimplicity and little computation cost.
Revised: In this subsection [Figure] we compare the computational cost of RORL with prior works on a single machine with one GPU (Tesla V100 32G) and one CPU (Intel Xeon Platinum 8255C @ 2.50GHz). For each method, we measure the average epoch time (i

Original: (2) uniformly sampling 20 initial states and performing gradient decent for 10 steps with a stepsize of 110 ✏ from each initial state to ﬁnd the best perturbed state.
Revised: [Figure] e., 1 × 10 3 training steps) and the GPU memory usage on the hopper-medium-v2 task. For CQL, PBRL, SAC- N , and EDAC, we evaluate the computational cost based on their official code.

Original: Note that we need to clipthe perturbed states within the l 1 ball at the end of each optimization step.
Revised: As shown in Table [Figure] RORL runs slightly faster than CQL, mainly because CQL needs the OOD action sampling and the logsumexp approximation. For ensemble-based baselines, RORL runs

Original: We name this
Revised: [Figure]

Original: optimization method ‘mixed order’ optimization.
Revised: much faster than PBRL, requiring only 28.7 % of PBRL’s epoch time. PBRL is so slow because it uses 10 ensemble Q networks for uncertainty measure and needs OOD action sampling for value underestimation. In RORL, we also include the OOD state-action sampling and additional adversarial training procedures, but we implement these procedures efficiently based on GPU operation and parallelism.

Original: We compare RORL with ensemble-based baselines EDAC and SAC-10 on halfcheetah-medium-v2and walker2d-medium-v2 datasets.
Revised: Even so, RORL is still slower than SAC-10 and EDAC. But as demonstrated in our experiments, RORL enjoys significantly better robustness than EDAC and SAC-10 under different types of perturbations. As for the GPU memory consumption, RORL uses comparable memory to PBRL and EDAC, with only 16 .

Revised: 7% more memory usage.

Original: To handle large adversarial noise, we set the perturbation scale ofthe policy  and Q functions as [0.1, 0.05] for halfcheetah-medium-v2 and [0.1, 0.03] for walker2dmedium-v2 in RORL’s training phase.
Revised: Furthermore, we analyze the computational cost of RORL’s components ( Q smoothing, policy smoothing, and the OOD loss). Specifically, we measure the average epoch time of SAC- 10 +Policy Smooth , SAC- 10 +Q Smooth , SAC- 10 +OOD Loss in Figure 7(a), and calculate the corresponding memory usage of each component in Figure 7(b). For the training time, SAC- 10 +Q Smooth runs the slowest and SAC- 10 +Policy Smooth runs slightly slower than SAC- 10 +OOD Loss .

Original: More detailed description can be found in Appendix B. Theresults are shown in Figure 4.
Revised: This is mainly because sampling the worst-case perturbation occupies the most time.

Original: In  the results, RORL exhibits improved robustness than other baselinesunder adversarial attacks anddecreases the slowest as ✏ increases.
Revised: In addition, since we use an ensemble of 10 Q networks, the memory usage of the Q smoothing loss and the OOD loss (both need to pass n perturbed states to 10 Q networks) is larger than the policy smoothing loss [Figure] C.2 Ablations on Benchmark Results In the benchmark experiments

Original: On the other hand, we ﬁnd thatrandom attack is not effective for ensemble-based ofﬂine RL algorithms, and the ‘mixed order’ attackbrings more signiﬁcant performance drop than vanilla zero-order optimization.
Revised: [Figure] RORL outperforms other baselines, especially in walker2d tasks. We conduct ablation studies on this task to verify the effectiveness of RORL’s components.

Original: 6.3 Ablations
Revised: In Figure

Original: We conducted ablation studies on walker2d-medium-v2 dataset to evaluate the importance of thepolicy smoothing term, the Qfunction smoothing term and thepenalty for OOD states in Figure5.
Revised: (a), we can find that each introduced loss (i.e., the OOD loss, the policy smoothing loss and the Q smoothing loss) influences the performance on the walker2d-medium-v2 task. Specifically, the OOD loss affects the most, without which the performance would drop close to SAC-N’s performance. In addition, the Q smoothing loss is helpful for stabilizing the training and final performance in clean environments. In Figure 8 (b), we evaluate the performance of RORL with varying τ .

Revised: The results suggest that τ is an important factor that balances the learning of in-distribution and out-of-distribution Q values.

Revised: In Eq.

Original: From the results, we can conclude that each loss contributes to  the performance of RORL underadversarial observation attacks.
Revised: (3), we want to assign larger weights ( 1 − τ ) on the δ ( s, ˆ s, a ) 2+ and smaller weights ( τ ) on the δ ( s, ˆ s, a ) 2 − to underestimate the values of OOD states, where δ ( s, ˆ s, a ) = Q ϕ i (ˆ s, a ) − Q ϕ i ( s, a ) . On the contrary, a too small τ can also lead to overestimation of in-distribution state-action pairs.

Original: The policy smoothing loss is essential when the perturbation scale islarge, while the performance of ’no P smooth’ is higher than RORL when the scale is smaller than 0.1.
Revised: In Figure 8 (b), τ = 0 leads to poor performance while larger τ = 0 . also result in performance worse than RORL without Q smoothing. Empirically, we find τ = 0 .

Original: Besides, learning without OOD loss performs worse than RORL at almost all the perturbation scales.
Revised: 2 works well across different tasks and set τ = 0 . 2 by default for all experiments. In the above analysis, we know that the OOD loss is a key component in RORL. We further study the impact of the OOD loss and ϵ ood on the performance and the value estimation.

Revised: As shown in Figure(a), when ϵ ood = 0 , the performance of RORL drops significantly, which illustrates the effectiveness of underestimating values of OOD states since the smoothness of RORL may overestimate these values.

Revised: From Figure 9 (b), we can verify that the OOD loss with ϵ ood > 0 contributes to the value underestimation.

Revised: [Figure]

Revised: C.3 Robustness Measures

Revised: In prior works [47, 71], the authors only demonstrate the robustness of algorithms via comparing the return curves with different attack scales.

Revised: To better measure the robustness of RL algorithms, we consider the robust score as the areas under the perturbation curve in Figure 4.

Revised: Since the returns in the figure have been normalized as introduced in Appendix B.2, we can simply calculate the robust score for each attack strategy as:

Revised: [Equation]

Revised: where Rs is the list of returns under N monotonically increasing attack scales.

Revised: The introduced robust score treats different attack scales equally.

Revised: However, in many real scenarios, we would pay more attention to larger-scale disturbances.

Revised: To this end, we also define a weighted robust score as:

Revised: [Equation]

Revised: where the weights are assigned according to the scale order.

Revised: In Table 6 and Table 7, RORL consistently outperforms EDAC and SAC- 10 on the two robustness metrics.

Revised: For walker2d and hopper tasks,

Revised: [Table]

Revised: [Table]

Revised: RORL surpasses EDAC by more than 10 points on both the robust score and the weighted robust score .

Revised: C.4 Ablations of Components in the Adversarial Experiments

Revised: In Section 6.3, we conducted ablations of RORL’s major components in the adversarial settings.

Revised: In this subsection, we provide robust scores of the ablation results over 4 random seeds in Table 8.

Revised: Besides, results of ϵ ood = 0 are also included to demonstrate the effectiveness of penalizing values of OOD states.

Revised: From Table 8, we can conclude that the OOD loss is the most essential component of RORL, and only penalizing in-distribution states is insufficient for adversarial perturbations.

Revised: To summarize, the order of the importance of each component is: OOD loss > ϵ ood > policy smoothing loss > Q smoothing loss.

Revised: The conclusion may be different for different tasks, for example we found that the halfcheetah task does not even need the OOD loss because the SAC-10 framework already provides it with sufficient pessimism.

Revised: C.5 Ablations on the Number of Ensemble Q Networks

Revised: We conduct the adversarial attack experiments with different number of bootstrapped Q networks in

Revised: RORL.

Revised: As shown in Figure 10, the robustness of RORL improves as the ensemble size K increases.

Revised: For K = 6 , 8 , 10 , RORL has similar initial performance but K = 10 considerably outperforms others as the attack scale increases.

Revised: Therefore, we set K = 10 by default in our paper.

Revised: C.6 Ablations of τ for the Adversarial Experiments

Revised: In this subsection, we study the performance under attacks with varying τ ∈ { 0 . 0 , 0 . 2 , 0 . 5 , 1 . 0 } .

Revised: From the results in Figure 11, we find τ = 0 .

Revised: 2 slightly outperforms the others on 4 out of the random

Revised: [Figure]

Revised: diff

Revised: [Figure]

Revised: diff mixed order min Q min Q mixed order

Revised: Average Score

Revised: RORL 94.

Revised: 71.2 no OOD 68.

Revised: 45.2 no P smooth 92.

Revised: [Figure]

Revised: [Figure]

Revised: no Q smooth 92.

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Original: In addition, Q smooth loss has the minimal impact, which is reasonable since the basic algorithm
Revised: attack types. The results are also consistent with the ablation studies of the benchmark experiments in Appendix C.2. Accordingly, we set τ = 0 . 2 by default for all experiments in our paper.

Revised: [Figure]

Revised: C.7 Ablations on the Number of Sampled Perturbed Observations

Revised: We ablate the number of sampled perturbed observations in Figure 12.

Original:  SAC-10 is based on 10 ensemble Q networks.
Revised: From the figure, we can conclude that the robustness of RORL improves as the number of samples n increases. At the same time, the computational cost also increases as n increases. Therefore, we can choose n according to the computational budget. Interestingly, RORL with n = 1 already outperforms SAC-10 by a large margin, which could be an appropriate option when computing resources are limited.

Original: Our smoothing technique may have a more signiﬁcantimpact on baselines with fewer Q networks, such as CQL in Section 3.
Revised: In our experiments, it is assumed that the ’min Q ’ and the ’min Q mixed order’ attackers have access to the corresponding Q value functions of the attacked agent. Generally, the assumption is strong for many real-world scenarios.

Original: Related Works
Revised: In addition, the comparison does not take into account the impact of attacking with different Q functions.

Original: Ofﬂine RL Research related to ofﬂine RL has experienced explosive growth in recent years. Inmodel-free domain, ofﬂine RL methods focus on correcting the extrapolation error [12] in the offpolicy algorithms.
Revised: Intuitively, conservative and smoothed Q functions make it easier for attackers to find the most impactful perturbation to degrade the performance. To investigate the impact of different Q functions, we swap the attacker’s Q -function, i.e. using RORL’s Q -functions to attack EDAC and using EDAC’s Q -functions to attack RORL .

Original: The natural idea is to regularize the learned policy near the dataset distribution [51,54, 31, 52, 58, 11].
Revised: In Figure 13, we can conclude:

Original: For example, MARVIL reweights the policy with exponential advantage, whichimplicitly guarantees the policy within the KL-divergence neighborhood of  the behavior policy.
Revised: (1) RORL outperforms EDAC with a wider margin when using the same Q functions. Surprisingly, the difference of normalized scores increases from 37.3 to 51.2 for walker2d-mediumv2 task with the largest ’min Q’ attack. [Figure]

Original: Another stream of model-free methods prevents the selection of OOD actions by penalizing their
Revised: (2) The value function of EDAC may still not be smooth and can mislead the attackers. In contrast, RORL successfully learns smooth value functions, which may facilitate further research on stronger attack strategies for robust offline RL

Original: Q-value [24, 25, 2, 7].
Revised: [Figure]

Original: With the ensemble Q networks and  the additional lossterm to diversify theirgradients, EDAC [2] achieves SOTA performance in the D4RL benchmark. Instead of diversifyinggradients, PBRL [3] proposes explicit training of the OOD actions, which achieves comparableperformance with fewer ensemble networks.
Revised: We also compare EDAC with both policy smoothing and Q smoothing, which leverages the gradient penalty rather than our OOD loss to enforce pessimism on OOD state-action pairs. The hyperparameters are kept the same with EDAC and RORL, except τ = 0 . 5 in EDAC+Smoothing.

Original: Inspired by EDAC and PBRL, we build our work uponensemble networks, focusing more on smoothness over the state space.
Revised: As shown in Figure 14, the smoothing technique slightly improves the robustness of EDAC under largescale (0.2 ∼ 0.3) adversarial perturbations, but it significantly decreases the overall performance under

Original: Besides the surprising empirical results, theoretical analysis of ofﬂine reinforcement learning algorithms is of increasing interest [6, 21, 39, 56, 59].
Revised: [Figure]

Original: Though the assumptions for the dataset vary in thedifferent papers, they all suggest that pessimism and conservatism are necessary for ofﬂine RL. Ourpaper can be viewed as a robust extension to the previous work [21] (see Section 5 for details). Robust RL The research line of robust RL can be traced back to H 1 -control theory [55, 4],where policies are optimized to be well-performed in the worst possible deterministic environment. Depending on the deﬁnition, there are different streams of research on robust RL.
Revised: [Figure]

Original: As the extensionof robust control to MDPs, Robust MDPs (RMDPs) [32, 18, 41, 16] are proposed to formulate theperturbation of transition probabilities for MDPs. Though some recent analyses with theoreticalguarantees come out under speciﬁc assumptions for RMDP [64, 57, 27], there is currently no practicalalgorithm to solve RMDP in a large-scale problem, expect some linear approximation attempt [46]. In online RL, domain randomization [48, 29] assumes the model uncertainty can be predeﬁned in datacollection by changing the setup of a simulator.
Revised: [Figure]

Original: However, it is not practical for ofﬂine RL.
Revised: [Figure]

Original: Robust
Revised: attack.

Original: Adversarial Reinforcement Learning (RARL) [37] and Noisy Robust Markov Decision Process(NR-MDP)
Revised: The results imply that directly using smoothing techniques without explicit OOD penalization can even worsen the robust scores of previous SOTA offline RL algorithm. C.10 Comparison with PBRL + S4RL

Original: [22] study the robust RL with the perturbed actions, showing that the policy robustness toadversarial or noisy actions can also induce robustness for model parameter changes. The most related work to ours is SR 2 L [43], which shows policy smoothing can lead to signiﬁcantperformance improvements in the online setting.
Revised: We also include comparison with PBRL and PBRL+S4RL to verify if RORL is more robust than data augmentation for offline RL [49]. The main differences between RORL and S4RL are three folds: (1) S4RL only implicitly smooths the value functions while RORL explicitly smooths them, which is more efficient and enjoys theoretical guarantees.

Original: In contrast, we focus on  the ofﬂine setting andtackle the potential overestimation of perturbed states. Another related work is S4RL [45], where theauthors study different data augmentation methods to smooth observations in ofﬂine RL.
Revised: (2) S4RL does not consider the impact of overestimation on OOD states brought by the data augmentation, which can be harmful for offline RL. In contrast, RORL further underestimates values for OOD states, which essentially alleviates the potential overestimation. In addition, S4RL selects adversarially perturbed states according to the gradient of Q ( s, π ( s )) , aiming to choose the direction where the Q -value deviates the most.

Original: Their resultsupports the necessity of state smoothing.
Revised: Different from S4RL, RORL samples perturbed states to maximize a conservative smoothing loss L (cid:0) Q ϕ i (ˆ s, a ) , Q ϕ i ( s, a ) (cid:1) and a policy smoothing loss

Revised: [Equation]

Revised: defined in Section 4.

Revised: The empirical results on halfcheetah-medium-v2 and walker2d-medium-v2 are shown in Figure 15.

Revised: We can observe that S4RL only slightly improves the robustness of PBRL on the walker2d-medium-

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: v2 task and has little impact

Revised: C.11 Combining Smoothing with IQL

Revised: We combine the policy smoothing and Q function smoothing techniques in RORL with IQL [26], a

Revised: SOTA offline RL algorithm without ensemble Q networks.

Revised: We use the default hyper-parameters of

Revised: IQL and set the hyper-parameters for smoothing the same as in Table 5.

Revised: The training and evaluation settings keep the same as the adversarial experiments in our paper.

Revised: As shown in Figure 16, we can observe that IQL with the smoothing technique (short for ’IQL smooth’) slightly improves the robustness on the walker2d-medium-v2 and hopper-medium-v2 tasks, but it has little effect on the halfcheetah-medium-v2 task.

Revised: This suggests that simply adopting the smoothing technique does not consistently improve the performance in the offline setting.

Revised: In contrast, RORL introduces additional

Revised: OOD underestimation based on uncertainty measure, which helps to obtain conservatively smoothed policy and value functions.

Revised: C.12 Comparing the ’max’ and the ’mean’ Operators in Smoothing

Revised: In our implementation, we first sample n perturbed states and select the one that maximizes the smoothing losses in Eq.

Revised: (2) and Eq.

Revised: It is interesting to see if the ’max’ operator is useful, as we can also use the ’mean’ operator as an alternative, i.e., L mean smooth ( s, a ; ϕ i ) =

Revised: E ˆ s ∈ B

Revised: d ( s,ϵ ) L

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: The results are demonstrated in Figure 17.

Revised: We can find that RORL with the ’max’ operator obtains a more conservative policy under small-scale perturbations and achieves higher robustness under largescale perturbations.

Revised: Since the ’max’ operator has the same complexity as the ’mean’ operator, we use the ’max’ operator by default, which is also a zeroth-order approximation to an inner optimization problem.

Revised: C.13 Comparing Different Optimization for Perturbation Generation during Training

Revised: In the training period, we use zeroth-order optimization to approximately optimize the Q smoothing loss in Eq.

Revised: (2) and the policy smoothing loss: max

Revised: [Equation]

Revised: s

Revised: [Equation]

Revised: s

Revised: [Equation]

Revised: ·| s

Revised: [Equation]

Revised: ∥ π

Revised: [Equation]

Revised: ( ·| ˆ s )

Revised: [Equation]

Revised: In this way, we can accelerate training the robust policy and obtain similar performance.

Revised: Besides, zeroth-order optimization is commonly applied in black-box attack where we can only access the input and output of neural networks without explicit gradient information.

Revised: Black-box attack for reinforcement learning might be a promising direction in the future.

Revised: We also implemented a first-order version of RORL, which requires an average epoch time of 72.7s on a V100 GPU (while the average epoch time of the zeroth-order method is 29.6s).

Revised: Since the perturbation generation for each training step is independent, we use the first-order optimization for a probability of 0.5 to alleviate the computational cost.

Revised: In Figure 18, we compare the trained policies with zeroth-order and first-order optimization.

Revised: We can conclude that the two types of optimization for perturbation generation have very similar performance.

Revised: On halfcheetah-medium task, the first-order version performs slightly better than the zeroth-order version, while the zeroth-order version works

Revised: [Figure]

Revised: slightly better on the walker2d-medium task.

Revised: We think this might be because we train the policy and value networks for 3 × 10 6 training steps, which may narrow the gap of the two optimization methods.

Revised: On the contrary, the mixed-order attackers (’action diff mixed order’ and ’min Q mixed order’) work better than zeroth-order attackers (’action diff’ and ’min Q ’) in the evaluation period, as demonstrated in Figure 4.

Revised: C.14 Comparison of the “minus target” and the “min target”

Revised: In the OOD loss L ood (Eq.

Revised: (4)), the pseudo-target (cid:98) T ood Q ϕ

Revised: i (ˆ s, ˆ a ) for the OOD state-action pairs (ˆ s, ˆ a ) can be implemented in two ways to underestimate the values of (ˆ s, ˆ a ) : (cid:98) T ood Q ϕ

Revised: i (ˆ s, ˆ a ) :=

Revised: Q ϕ i (ˆ s, ˆ a ) − λu (ˆ s, ˆ a ) or (cid:98) T ood Q ϕ

Revised: i (ˆ s, ˆ a ) :

Revised: = min i =1 ,...,K Q ϕ

Revised: i (ˆ s, ˆ a ) ( K = 10 ).

Revised: The two targets are referred to as the “minus target” and the “min target” respectively.

Revised: In Figure 19, we compare the two targets’ D4RL scores in clean environments, and the hyper-parameters are the same as Table 4.

Revised: Although the “min target” has less hyper-parameters and achieves comparable performance on the hopper-medium-expert-v2 task, it is unstable and not flexible across different tasks, e.g., significantly overestimating values for the walker2d-random-v2 task and underestimating values for hopper-medium-v2 and walker2d-medium-v2 tasks.

Revised: Therefore, we choose the “minus target” by default in our paper.

Revised: [Table]

Revised: C.15 Experiments in Adroit Domains

Revised: We also evaluate RORL in the challenging Adroit domains which control a 24-DoF robotic hand to manipulate a pen, a hammer, a door and a ball.

Revised: These domains contain three types of data, namely ‘Expert’, ‘Cloned’, and ‘Human’, for each task.

Revised: The hyper-parameters are listed in Table 9.

Revised: We set β Q = 0 .

Revised: 0001 , β ood = 0 .

Revised: 5 , τ = 0 .

Revised: 2 , n = 20 , and search β P within { 0 . 01 , 0 . 1 , 1 . 0 } , ϵ Q / ϵ P / ϵ ood within { 0 . 001 , 0 . 005 , 0 . 01 } .

Revised: For Door/Relocate-human/cloned datasets, the policy learning rate is set to 1 e − 4 .

Revised: The other hyper-parameters are the same as in Table 3.

Revised: On four expert datasets, we train

Revised: RORL for 1000 epochs (1000 gradient steps per epoch).

Revised: As for other datasets, we train RORL for 300 epochs because the ‘Cloned’ and ‘Human’ datasets are much smaller.

Revised: In Table

Revised: [Table]

Revised: we compare the performance of RORL with other baselines, such as EDAC, PBRL,

Revised: TD3+BC, CQL, UWAC, BEAR, and BC.

Revised: We can observe that RORL achieves the top two highest score in 6 out of 12 tasks, which further verifies the effectiveness of RORL.

Revised: [Table]

Revised: C.16 AntMaze Tasks

Revised: To verify whether smoothing benefits policy learning in the challenging AntMaze setting, we apply our policy and value function smoothing techniques on top of IQL (short for ‘IQL+smoothing’).

Revised: For the hyper-parameters, we use ϵ Q = 0 .

Revised: 01 , ϵ P = 0 .

Revised: 03 , τ = 0 .

Revised: 2 for all six types of datasets, and search β Q ∈ { 0 . 1 , 0 . 01 } , β P ∈ { 0 . 1 , 0 . 5 } , n = 20 ,.

Revised: Other hyper-parameters keep the default hyper-parameters of IQL [26].

Revised: In Table 11, we compare ‘IQL+smoothing’ with both model-free (AWAC [36], TD3+BC [13], CQL [28], and IQL [26]) and model-based (ROMI [56]) baselines.

Revised: On 4 out of 6 tasks, ’IQL+smoothing’ improves the performance of IQL.

Revised: Intuitively, for sparse reward tasks, smoothing the value functions of nearby states could help with the value propagation, and smoothing the policy can enhance the robustness of learned policies.

Revised: [Table]

Revised: C.17 Robustness of the Benchmark Results

Revised: In Figure

Revised: [Table]

Revised: we evaluate the robustness of the benchmark results,

Revised: i

Original: More related works are given in Appendix D.
Revised: [Table] e., how robust each algorithm is to maintain the performance listed in Table 1. We compare RORL with EDAC, SAC-10 on six tasks. EDAC is reproduced with 10 ensemble Q networks as RORL and SAC-10, and uses η = 1 for all six tasks.

Original: We propose Robust Ofﬂine Reinforcement Learning (RORL) to trade-off conservatism and robustnessfor ofﬂine RL.
Revised: Note that in the benchmark experiments, RORL is only trained with small smoothing scales within { 0 . 001 , 0 . 005 , 0 . 01 } .

Original: To achieve that, we introduce the conservative smoothing for the perturbed stateswhile actively underestimating their values based on pessimistic bootstrapping to keep conservative.
Revised: The evaluation perturbation scales are within range [0 . 00 , 0 . 05] and the results are averaged over 4 random seeds.

Original: We show that RORL can achieve comparable performance with fewer ensemble Q networks than theprevious methods in the ofﬂine RL benchmark.
Revised: From the results, we can conclude that RORL can successfully keep the highest performance within a certain perturbation scale and the performance of EDAC and SAC-10 decreases faster than RORL for most tasks and attack methods.

Original: The result supports that our proposed conservativesmoothing techniques beneﬁt general ofﬂine RL by improving the generalization ability. In addition,we demonstrate that RORL is also robust to adversarial perturbations across the different types ofattacks. We hope our work can promote the application of ofﬂine RL under real-world engineeringconditions. The main limitation of our method is that the adversarial state sampling slows down the computingprocess, which may be improved in future work.
Revised: The results imply that RORL has better practicability in real-world scenarios.

Original: Checklist
Revised: D More Related Works

Original: The checklist follows the references. Please read the checklist guidelines carefully for information onhow to answer these questions. For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .

Original: You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description. For example: • Did you include the license to  the code and datasets? [Yes] See Section ?? .•
Revised: Model-Based Offline RL In offline RL, model-based methods use an empirical model learned from the offline dataset to enhance the generalization ability.

Original: Did you include the license to the code and datasets?

Original: [No] The code and the  data areproprietary.• Did you include the license to  the code and datasets? [N/A] Please do not modify the questions and only use the provided macros for your answers.
Revised: The model can be used as the virtual environment for data collection [70, 25], or to augment the dataset for an existing model-free algorithm [69, 56].

Original: Note that the

Original: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope? [Yes] (b) Did you describe the limitations of your work? [Yes](c)
Revised: The main challenges of model-based algorithms are how to learn the accurate empirical model and how to construct the uncertainty measure.

Original: Did you discuss any potential negative societal impacts of your work? [N/A](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem? [Yes]2.
Revised: A recent work [21] demonstrates that the transformer model can generate realistic trajectories, which is beneficial for policy learning.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results? [Yes](b) Did you include complete proofs of all theoretical results? [Yes] See Appendix A.3.
Revised: In contrast, we focus on the model-free methods in this paper and leave the robustness of model-based methods in future work.

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] See Sec 1 andAppendix B.(b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)? [Yes] See Appendix B. (c) Did you report error bars (e.g., with respect to  the random seed after running experiments multiple times)?
Revised: Adversarial Attack Inspired by adversarial examples in deep learning [17, 39], adversarial attack and policy poisoning [8, 19, 41] are studied to avoid adversarial manipulations on the network policies.

Original: [Yes] See Sec 6. (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)? [Yes] See Appendix B. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: Gleave et al.

Original: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] We cited D4RL [10]and EDAC[2] for their datasets and code.

Original: (b) Did you mention the license of the assets?

Original: [Yes]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] We included our code in the anonymized link. (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [15] study adversarial policy in the behavior level [15].

Original: [Yes] Opensource code and dataset. (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content? [N/A]
Revised: Data corruption [72, 33, 60] considers the case where an attacker can arbitrarily modify the dataset under a specific budget before training.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
Revised: While adversarial attack in RL is highly related to robust RL, they focus more on adversarial attacks compared to our robustness setting.

Original: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: More effective attack strategies for offline RL can facilitate learning more robust policies.

Original: spent on participant compensation?

Original: [N/A]
Revised: [Figure]