Original: RORL: Robust Ofﬂine Reinforcement Learning via Conservative Smoothing
Revised: RORL: Robust Ofﬂine Reinforcement Learning via Conservative Smoothing

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Ofﬂine reinforcement learning (RL) provides a promising direction to exploit themassive amount of ofﬂine data for complex decision-making tasks.
Revised: Ofﬂine reinforcement learning (RL) provides a promising direction to exploit themassive amount of ofﬂine data for complex decision-making tasks.

Original: Due to thedistribution shift issue, current ofﬂine RL algorithms are generally designed to beconservative for value estimation and action selection.
Revised: Due to thedistribution shift issue, current ofﬂine RL algorithms are generally designed to beconservative for value estimation and action selection.

Original: However, such conservatismimpairs the robustness of learned policies, leading to a signiﬁcant change even fora small perturbation on observations.
Revised: However, such conservatismimpairs the robustness of learned policies, leading to a signiﬁcant change even fora small perturbation on observations.

Original: To trade off robustness and conservatism, wepropose Robust Ofﬂine Reinforcement Learning (RORL) with a novel conservativesmoothing technique.
Revised: To trade off robustness and conservatism, wepropose Robust Ofﬂine Reinforcement Learning (RORL) with a novel conservativesmoothing technique.

Original: In RORL, we explicitly introduce regularization on the policyand the value function for states near the dataset and additional conservative valueestimation on these OOD states.
Revised: In RORL, we explicitly introduce regularization on the policyand the value function for states near the dataset and additional conservative valueestimation on these OOD states.

Original: Theoretically, we show RORL enjoys a tightersuboptimality bound than recent theoretical results in linear MDPs.
Revised: Theoretically, we show RORL enjoys a tightersuboptimality bound than recent theoretical results in linear MDPs.

Original: We demonstratethat RORL can achieve the state-of-the-art performance on the general ofﬂine RLbenchmark and is considerably robust to adversarial observation perturbation.
Revised: We demonstratethat RORL can achieve the state-of-the-art performance on the general ofﬂine RLbenchmark and is considerably robust to adversarial observation perturbation.

Original:  Introduction
Revised:  Introduction

Original: Over the past few years, deep reinforcement learning (RL) has been a vital tool for various decisionmaking tasks [32, 45, 43, 9] in a trial-and-error manner.
Revised: Over the past few years, deep reinforcement learning (RL) has been a vital tool for various decisionmaking tasks [30, 44, 42, 8] in a trial-and-error manner.

Original: A major limitation of current deep RLalgorithms is that they require intense online interactions with the environment [28].
Revised: A major limitation of current deep RLalgorithms is that they require intense online interactions with the environment [26].

Original: These datacollecting processes can be costly and even prohibitive in many real-world scenarios such as roboticsand health care.
Revised: These datacollecting processes can be costly and even prohibitive in many real-world scenarios such as roboticsand health care.

Original: Ofﬂine RL [13, 26] is gaining more attention recently since it offers probabilities tolearn reinforced decision-making strategies from fully ofﬂine datasets.
Revised: Ofﬂine RL [12, 24] is gaining more attention recently since it offers probabilities tolearn reinforced decision-making strategies from fully ofﬂine datasets.

Original: The main challenge of ofﬂine RL is the distribution shift between the ofﬂine dataset and the learnedpolicy, which would lead to signiﬁcant overestimation for the out-of-distribution (OOD) actions[13, 26].
Revised: The main challenge of ofﬂine RL is the distribution shift between the ofﬂine dataset and the learnedpolicy, which would lead to signiﬁcant overestimation for the out-of-distribution (OOD) actions[12, 24].

Original: To overcome such an issue, a series of ofﬂine RL works [52, 13, 60, 27, 2, 4] propose tocelebrate conservatism, such as constraining the learned policy close to supported distribution orpenalizing the Q -values of OOD actions.
Revised: To overcome such an issue, a series of ofﬂine RL works [51, 12, 58, 25, 2, 3] propose tocelebrate conservatism, such as constraining the learned policy close to supported distribution orpenalizing the Q -values of OOD actions.

Original: In addition, another stream of works builds upon modelbased ofﬂine RL algorithms [63, 62, 51], which leverages the ensemble dynamics models to enforcepessimism via uncertainty penalizing or data generation.
Revised: In addition, another stream of works builds upon modelbased ofﬂine RL algorithms [61, 60, 50], which leverages the ensemble dynamics models to enforcepessimism via uncertainty penalizing or data generation.

Original: However, conservatism is not the only concern when applying ofﬂine RL to the real world.
Revised: However, conservatism is not the only concern when applying ofﬂine RL to the real world.

Original: Dueto the sensor errors and model mismatch, the robustness of ofﬂine RL is crucial under the realisticengineering conditions, which has not been well studied yet.
Revised: Dueto the sensor errors and model mismatch, the robustness of ofﬂine RL is crucial under the realisticengineering conditions, which has not been well studied yet.

Original: In online RL, a series of works havebeen studied to learn the optimal policy under worst-case perturbations of the observation [64, 37, 18]or environmental dynamics [50, 39, 40].
Revised: In online RL, a series of works havebeen studied to learn the optimal policy under worst-case perturbations of the observation [62, 35, 17]or environmental dynamics [49, 37, 38].

Original: Yet, it is non-trivial to apply online robust RL techniquesinto the ofﬂine problems.
Revised: Yet, it is non-trivial to apply online robust RL techniquesinto the ofﬂine problems.

Original: The main challenge is that the perturbation of either states or actions may bring OOD data and extra overestimation for the value function.
Revised: The main challenge is that the perturbation of either states or actions may bring OOD data and extra overestimation for the value function.

Original: New techniques are needed to tackle the conservatism and robustness simultaneously in the ofﬂine RL.
Revised: New techniques are needed to tackle the conservatism and robustness simultaneously in the ofﬂine RL.

Original: This paper studies robust ofﬂine RL against adversarial observation perturbation, where the agentneeds to learn the policy conservatively while handling the potential OOD observation with perturbation.
Revised: This paper studies robust ofﬂine RL against adversarial observation perturbation, where the agentneeds to learn the policy conservatively while handling the potential OOD observation with perturbation.

Original: We ﬁrst demonstrate that current value-based ofﬂine RL algorithms lack the necessarysmoothness for the policy, which is visualized in Figure 1.
Revised: We ﬁrst demonstrate that current value-based ofﬂine RL algorithms lack the necessarysmoothness for the policy, which is visualized in Figure 1.

Original: As an illustration, we show that a famousbaseline method CQL [27] learns a non-smooth value function, leading to signiﬁcant performancedegradation for even a tiny scale perturbation on observation (see Section 3 for details).
Revised: As an illustration, we show that a famousbaseline method CQL [25] learns a non-smooth value function, leading to signiﬁcant performancedegradation for even a tiny scale perturbation on observation (see Section 3 for details).

Original: Notice thatsimply adopting the smoothing technique for existing methods may result in extra overestimation atthe boundary of supported distribution and lead the agent toward the unsafe areas.
Revised: Notice thatsimply adopting the smoothing technique for existing methods may result in extra overestimation atthe boundary of supported distribution and lead the agent toward the unsafe areas.

Original: To this end, we propose Robust Ofﬂine Reinforcement Learning (RORL) with a novel conservativesmoothing technique, which explicitly handles the overestimation of OOD state-action pairs.
Revised: To this end, we propose Robust Ofﬂine Reinforcement Learning (RORL) with a novel conservativesmoothing technique, which explicitly handles the overestimation of OOD state-action pairs.

Original: Speciﬁcally, we explicitly introduce smooth regularization on both the values and policies for states nearthe dataset support and conservatively estimate the values of these OOD states based on pessimisticbootstrapping.
Revised: Speciﬁcally, we explicitly introduce smooth regularization on both the values and policies for states nearthe dataset support and conservatively estimate the values of these OOD states based on pessimisticbootstrapping.

Original: In addition, we theoretically prove that RORL yields a valid uncertainty quantiﬁer inlinear MDPs and enjoys a tighter suboptimality bound than previous work [4].
Revised: In addition, we theoretically prove that RORL yields a valid uncertainty quantiﬁer inlinear MDPs and enjoys a tighter suboptimality bound than previous work [3].

Original: In our experiments, we demonstrate that RORL can achieve state-of-the-art (SOTA) performancein D4RL benchmark [10] for ofﬂine RL given a suitable adversarial perturbation scale duringtraining.
Revised: In our experiments 1 , we demonstrate that RORL can achieve state-of-the-art (SOTA) performance in D4RL benchmark [9] for ofﬂine RL given a suitable adversarial perturbation scale during training.

Original: The surprising performance in the benchmark shows that the robust training improves thegeneralization ability and leads to performance improvement even in non-perturbed tasks.
Revised: Thesurprising performance in the benchmark shows that the robust training improves the generalizationability and leads to performance improvement even in non-perturbed tasks.

Original: Meanwhile,compared with the current SOTA baseline [2], RORL is considerably more robust to adversarialperturbations on observations.
Revised: Meanwhile, comparedwith the current SOTA baseline [2], RORL is considerably more robust to adversarial perturbationson observations.

Original: We conduct the ablation study under different types of adversarialattacks on the observations, showing the consistently superior performance over several tasks.
Revised: We conduct the ablation study under different types of adversarial attacks on theobservations, showing the consistently superior performance over several tasks.

Original:  Preliminaries
Revised:  Preliminaries

Original: Ofﬂine RL and PBRL Considering an episodic MDP M =
Revised: Ofﬂine RL and PBRL Considering an episodic MDP M =

Original: ( S , A , T, r,  , P ) , where S is the state 58 space,
Revised: ( S , A , T, r,  , P ) , where S is the state 58 space,

Original: A is the action space, T is the length of the episode, r is the reward function, P is the dynamics,and  is the discount factor.
Revised: A is the action space, T is the length of the episode, r is the reward function, P is the dynamics,and  is the discount factor.

Original: In ofﬂine RL, the objective of the agent is to ﬁnd an optimal policy bysampling experience from a ﬁxed dataset D = { ( s it , a it , r it , s it +1 ) } .
Revised: In ofﬂine RL, the objective of the agent is to ﬁnd an optimal policy bysampling experience from a ﬁxed dataset D = { ( s it , a it , r it , s it +1 ) } .

Original: Nevertheless, directly applyingoff-policy algorithms in ofﬂine RL has distribution shift problem.
Revised: Nevertheless, directly applyingoff-policy algorithms in ofﬂine RL has distribution shift problem.

Original: In Q -learning, the value functionevaluated on the greedy action a 0 in Bellman operator T Q = r +  E s 0 [max a 0 ( s 0 , a 0 )] tends to haveextrapolation error since ( s 0 , a 0 ) has barely occurred in D .
Revised: In Q -learning, the value functionevaluated on the greedy action a 0 in Bellman operator T Q = r +  E s 0 [max a 0 ( s 0 , a 0 )] tends to haveextrapolation error since ( s 0 , a 0 ) has barely occurred in D .

Original: There are mainly three kinds of methodsto track the distribution shift problem in ofﬂine RL, including policy constraints, conservative valuefunction, and uncertainty-based methods.
Revised: There are mainly three kinds of methodsto track the distribution shift problem in ofﬂine RL, including policy constraints, conservative valuefunction, and uncertainty-based methods.

Original: Pessimistic Bootstrapping for Ofﬂine RL (PBRL) [4] is an uncertainty-based method that uses bootstrapped Q -functions for uncertainty quantiﬁcation and OOD sampling for regularization.
Revised: Pessimistic Bootstrapping for Ofﬂine RL (PBRL) [3] is an uncertainty-based method that uses bootstrapped Q -functions for uncertainty quantiﬁcation and OOD sampling for regularization.

Original: Speciﬁcally,
Revised: Speciﬁcally,

Original: PBRL maintains K bootstrapped Q functions to quantify the epistemic uncertainty and performspessimistic update to penalize Q functions with large uncertainties.
Revised: PBRL maintains K bootstrapped Q functions to quantify the epistemic uncertainty and performspessimistic update to penalize Q functions with large uncertainties.

Original: The uncertainty is deﬁned asthe standard deviation among bootstrapped Q -functions.
Revised: The uncertainty is deﬁned asthe standard deviation among bootstrapped Q -functions.

Original: For each bootstrapped Q -function, the
Revised: For each bootstrapped Q -function, the

Revised: [Figure]

Original: Bellman target is deﬁned as
Revised: Bellman target is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Under linear MDP assumptions, this uncertainty is equivalent to the LCB penalty and is provably
Revised: Under linear MDP assumptions, this uncertainty is equivalent to the LCB penalty and is provablyefﬁcient [21].

Original: Furthermore, PBRL incorporates OOD sampling in training by sampling OOD actions [Figure] to form ( s, a ood ) pairs, where a ood follows the learned policy.
Revised: Furthermore, PBRL incorporates OOD sampling in training by sampling OOD actionsto form ( s, a ood ) pairs, where a ood follows the learned policy.

Original: The learning target for ( s, a ood ) isb
Revised: The learning target for ( s, a ood ) isb

Original: T ood Q ( s, a ood ) :
Revised: T ood Q ( s, a ood ) :

Original: =
Revised: =

Original: Q ( s, a ood )   u ( s, a ood ) , which introduces uncertainty penalization to enforcepessimistic Q -functions for OOD actions.
Revised: Q ( s, a ood )   u ( s, a ood ) , which introduces uncertainty penalization to enforcepessimistic Q -functions for OOD actions.

Original: Smooth Regularized RL Robust RL aims to learn a robust policy against the adversarial perturbedenvironment in online RL.
Revised: Smooth Regularized RL Robust RL aims to learn a robust policy against the adversarial perturbed environment in online RL.

Original: SR 2 L [44] enforces smoothness in both the policy and Q -functions.
Revised: SR 2 L [43] enforces smoothness in both the policy and Qfunctions.

Original: Speciﬁcally, SR 2 L encourages the outputs of the policy and value function to not change much wheninjecting small perturbation to the state.
Revised: Speciﬁcally, SR 2 L encourages the outputs of the policy and value function to notchange much when injecting small perturbation to the state.

Original: For state s , SR 2 L constructs a perturbation set B d ( s, ✏ ) =
Revised: For state s , SR 2 L constructs a perturbation set B d ( s, ✏ ) =

Original: { ˆ s :d ( s, ˆ s )  ✏ }with the metric d ( , ) , which is chosen to be the ` p distance and introduces a smoothnessregularizer for policy asR ⇡ s =
Revised: { ˆ s : d ( s, ˆ s )  ✏ } and introduces a smoothness regularizer for policy as

Original: E s ⇠ ⇢ ⇡ max ˆ s 2 B d ( s, ✏ ) D ( ⇡ ( ·| s )
Revised: [Equation]

Original:  k ⇡ ( ·| ˆ s )) , where D ( · k · ) is a distance83 metric and the max operator gives an adversarial manner to choose ˆ s .
Revised: ✏ ) D ( ⇡ ( ·| s ) k ⇡ ( ·| ˆ s )) , where D ( · k · ) is a distance metric and the max oper- 83 ator gives an adversarial manner to choose ˆ s .

Original: Similarly, the smoothnessregularizer for the value function is deﬁned as R Vs 
Revised: Similarly, the smoothness regularizer for the valuefunction is deﬁned as R Vs =

Original: [Equation]
Revised: E s ⇠ ⇢ ⇡ ,a ⇠ ⇡ max ˆ s 2 B d ( s, ✏ ) ( Q ( s, a )  Q (ˆ s, a )) 2 .

Original: [Equation] SR 2 L is shown to improve robustness against both random and adversarial perturbations.
Revised: The induced smoothness 85 is shown to improve robustness against both random and adversarial perturbations.

Original:  Robustness of Ofﬂine RL: A Motivating Example
Revised:  Robustness of Ofﬂine RL: A Motivating Example

Original: We give a motivating example to illustrate the robustness of the popular CQL [27] policies.
Revised: We give a motivating example to illustrate the robustness of the popular CQL [25] policies.

Original: Weintroduce an adversarial attack on state s to obtain ˆ s = arg max
Revised: Weintroduce an adversarial attack on state s to obtain ˆ s = arg max

Original: ˆ s 2 B d ( s, ✏ ) D J ( ⇡ ✓ ( ·| s ) k ⇡ ✓ ( ·| ˆ s )) ,where B d ( s, ✏ ) =
Revised: ˆ s 2 B d ( s, ✏ ) D J ( ⇡ ✓ ( ·| s ) k ⇡ ✓ ( ·| ˆ s )) , where

Original: { ˆ s : d ( s, ˆ s )  ✏} is the perturbation set and themetric d ( , ) is chosen to bethe ` 1 . The Jeffrey’s divergence D J for two distributions P , Q is deﬁned by: D J ( P k Q ) =
Revised: B d is the perturbation set to ensure d ( s, ˆ s )  ✏ and the Jeffrey’s divergence D J for two distributions P , Q is deﬁned by: D J ( P k Q ) =

Original: 12 [ D KL ( P k Q )
Revised: 12 [ D KL ( P k Q )

Original: + D KL ( Q k P )] .
Revised: + D KL ( Q k P )] .

Original: To obtain ˆ s , we take gradient assent with respect to the loss function D J ( ⇡ ✓ ( ·| s ) k ⇡ ✓ ( ·| ˆ s )) and restrict the outputs to the B d ( s, ✏ )  set, where ⇡ ✓ is a learned CQL policy. We remark that the the perturbation is applied on normalized observations following prior work [64].
Revised: To obtain ˆ s , we take gradientassent with respect to the loss function D J ( ⇡ ✓ ( ·| s ) k ⇡ ✓ ( ·| ˆ s )) and restrict the outputs to the B d ( s, ✏ ) 92 set, where ⇡ ✓ is a learned CQL policy and ✏ controls the strength of attack.

Original: In the walker-medium-v2 task from D4RL [10], we use various setups of ✏ for adversarial attack toevaluate the robustness of CQL policies.
Revised: In the walker-medium-v2 task from D4RL [9], we use various setups of ✏ for adversarial attack toevaluate the robustness of CQL policies.

Original: Speciﬁcally, we use ✏ 2 { 0 , 0 . 05 , 0 . 1 , 0 .
Revised: Speciﬁcally, we use ✏ 2 { 0 , 0 . 05 , 0 . 1 , 0 .

Original: 14 } to control thestrengths of the attack, where we have ˆ s = s if ✏ = 0 .
Revised: 14 } to control thestrengths of the attack, where we have ˆ s = s if ✏ = 0 .

Original: Given a speciﬁc ✏ , we sample N state-actionpairs { ( s i , a i ) } from the ofﬂine dataset, and then perform adversarial attack to obtain { (ˆ s i , a i ) } and 98 also the corresponding Q -values { Q i (ˆ s i , a i ) } , where the Q -function is the trained CQL critic.
Revised: Given a speciﬁc ✏ , we sample N state-actionpairs { ( s i , a i ) } from the ofﬂine dataset, and then perform adversarial attack to obtain { (ˆ s i , a i ) } and 97 also the corresponding Q -values { Q i (ˆ s i , a i ) } , where the Q -function is the trained CQL critic.

Original: Figure 2(a) shows the relationship between ˆ s i and the corresponding Q i with different ✏ .
Revised: Figure 2(a) shows the relationship between ˆ s i and the corresponding Q i with different ✏ .

Original: To visualizeˆ s i , we perform PCA dimensional reduction [48] and choose one of the reduced dimensions torepresent
Revised: To visualizeˆ s i , we perform PCA dimensional reduction [47] and choose one of the reduced dimensions torepresent

Original: ˆ s i .
Revised: ˆ s i .

Original: More details can be found in Appendix B.3.

Original: With the increase of ✏ in the adversarialattack, the Q -curve has greater deviation compared to the curve with ✏ = 0 .
Revised: With the increase of ✏ in the adversarial attack, the Q -curve has greater deviationcompared to the curve with ✏ = 0 .

Original: The result signiﬁesthat the Q -function of CQL is not smooth in state space, which makes the adversarial noises easilyaffect the Q -function.
Revised: The result signiﬁes that the Q -function of CQL is not smooth instate space, which makes the adversarial noises easily affect the Q -function.

Original: As a comparison, we apply the proposed conservative smoothing loss in CQLtraining (i.e., CQL-smooth ) and use the same evaluation method to obtain ˆ s i and Q i .
Revised: As a comparison, weapply the proposed conservative smoothing loss in CQL training (i.e., CQL-smooth ) and use the sameevaluation method to obtain ˆ s i and Q i .

Original: According tothe result in Figure 2(b), the value function becomes smoother.
Revised: According to the result in Figure 2(b), the value functionbecomes smoother.

Original: In addition, we show how the adversarial attack affects the ﬁnal performance of ofﬂine RL policies.
Revised: In addition, we show how the adversarial attack affects the ﬁnal performance of ofﬂine RL policies.

Original: We use ✏ 2 [0 , 0 . 15] to evaluate both the original CQL policies (i.e., CQL ) and CQL with conservativesmoothing loss (i.e., CQL-smooth ) in adversarial attack.
Revised: We use ✏ 2 [0 , 0 . 15] to evaluate both the original CQL policies (i.e., CQL ) and CQL with conservative

Original: Figure 2(c) shows the performance with
Revised: OfflineDataset

Original: [Figure]
Revised: [Figure]

Original: while 
Revised: while not converged do Sample mini-batch transitions ( s, a, r, s 0 ) from D . Perform extended soft Q -learning to train multiple Q -functions { Q 1 , . .

Original: [Table]
Revised: . , Q K } .

Original: [Table] [Table]
Revised: Sample ˆ s from B d ( s, ✏ ) to obtain (ˆ s, a ) pairs. Perform adversarial smoothing for the Q -network.

Original: [Table] ˆ [Table] ) for conservatism.
Revised: Calculate uncertainty to penalize (ˆ s, ˆ a ) for conservatism.

Original: Combining [Table] Train
Revised: Combiningterms to train the critic with Eq.

Original: [Table]
Revised: Train the policy with smooth constraint as Eq. smoothing calculated smoothing loss (i.e., CQL-smooth ) in adversarial attack.

Original: with [Table] as Eq. different settings of ✏ .
Revised: Figure 2(c) shows the performance withdifferent settings of ✏ .

Original: We ﬁnd that our smooth constraints signiﬁcantly improve the robustness of
Revised: We ﬁnd that our smooth constraints signiﬁcantly improve the robustness of

Original: CQL, especially for large adversarial noises.
Revised: CQL, especially for large adversarial noises.

Original:  Robust Ofﬂine RL via Conservative Smoothing
Revised:  Robust Ofﬂine RL via Conservative Smoothing

Original: In RORL, we develop smooth regularization on both the policy and the value function for statesnear the dataset.
Revised: In RORL, we develop smooth regularization on both the policy and the value function for statesnear the dataset.

Original: The smooth constraints make the policy and Q -function robust to perturbation inthe dataset (for ofﬂine training) and interactive samples (for online evaluation).
Revised: The smooth constraints make the policy and Q -function robust to perturbation inthe dataset (for ofﬂine training) and interactive samples (for online evaluation).

Original: Nevertheless, thesmoothness may lead to value overestimation of the edge of the supported dataset.
Revised: Nevertheless, thesmoothness may lead to value overestimation of the edge of the supported dataset.

Original: To address thisproblem, we adopt bootstrapped Q-functions [35] for uncertainty quantiﬁcation and sample OODactions for penalization.
Revised: To address thisproblem, we adopt bootstrapped Q-functions [33] for uncertainty quantiﬁcation and sample OODactions for penalization.

Original: RORL obtains conservative and smooth value estimation on both OODstates and actions, which improves the generalization ability of ofﬂine RL algorithms.
Revised: RORL obtains conservative and smooth value estimation on both OODstates and actions, which improves the generalization ability of ofﬂine RL algorithms.

Original: The overallarchitecture of RORL is given in Figure 3.
Revised: The overallarchitecture of RORL is given in Figure 3.

Original: Robust Q -function We sample three sets of state-action pairs and apply different loss functionsto obtain a conservative smooth policy.
Revised: Robust Q -function We sample three sets of state-action pairs and apply different loss functionsto obtain a conservative smooth policy.

Original: Speciﬁcally, for a ( s, a ) pair sampled from D , we constructa perturbation set B d ( s, ✏ ) to obtain (ˆ s, a ) pairs, where ˆ s 2 B d ( s, ✏ ) and ✏ is the perturbation scale. The perturbation set B d ( s, ✏ ) =
Revised: Speciﬁcally, for a ( s, a ) pair sampled from D , we constructa perturbation set B d ( s, ✏ ) to obtain (ˆ s, a ) pairs, where ˆ s 2 B d ( s, ✏ ) and ✏ .

Original: { ˆ s : d ( s, ˆ s )  ✏ } for state s is an ✏ -radius ball measured in metricd ( , ) , which is the ` 1 in our paper.

Original: Then we perform OOD sampling by using the current policy ⇡ ✓to obtain (ˆ s, ˆ a ) pairs, where ˆ a ⇠ ⇡ ✓ (ˆ s ) .
Revised: Then we perform OODsampling by using the current policy ⇡ ✓ to obtain (ˆ s, ˆ a ) pairs, where ˆ a ⇠ ⇡ ✓ (ˆ s ) .

Original: RORL contains K Q -functions trained by Soft Q -learningas critics.
Revised: RORL contains K Q -functions trained by Soft Q -learning as critics.

Original: We denote the parameters of i -th Q -function and the target Q -function as  i and  0 i ,respectively.
Revised: We denote the parameters of i -th Q -function andthe target Q -function as  i and  0 i , respectively.

Original: In the following, we give different learning targets for ( s, a ) , (ˆ s, a ) , and (ˆ s, ˆ a ) pairs,respectively.
Revised: In the following, we give different learning targetsfor ( s, a ) , (ˆ s, a ) , and (ˆ s, ˆ a ) pairs, respectively.

Original: First, for a ( s, a ) pair sampled from D , we apply extended soft Q -learning to obtain the target as
Revised: First, for a ( s, a ) pair sampled from D , we apply extended soft Q -learning to obtain the target as

Original: [Equation]
Revised: [Equation]

Original: ⇤ , where the next- Qfunction takes minimum value among the target Q -functions.
Revised: where the next- Q function takes minimum value among the target Q -functions.

Original: Then, for a (ˆ s, a ) pair with a perturbed state, we enforce smoothness in Q -function by minimizing the
Revised: Then, for a (ˆ s, a ) pair with a perturbed state, we enforce smoothness in Q -function by minimizing the

Original: Q -value difference between Q ( s, a ) and Q (ˆ s, a ) .
Revised: Q -value difference between Q ( s, a ) and Q (ˆ s, a ) .

Original: In particular, we choose an adversarial ˆ s 2 B d ( s, ✏ )that maximizes L ( Q (ˆ s, a ) , Q ( s, a )) , and then minimize this loss function with the adversarial ˆ s ,where L is a distance measure.
Revised: In particular, we choose an adversarial ˆ s 2 B d ( s, ✏ )that maximizes L ( Q (ˆ s, a ) , Q ( s, a )) , and then minimize this loss function with the adversarial ˆ s ,where L is a distance measure.

Original: Intuitively, we want that the Q -function is smooth with the mostdifﬁcult (i.e., adversarial) attack in B d ( s, ✏ ) , where ✏ controls the perturbation scale.
Revised: Intuitively, we want that the Q -function is smooth with the mostdifﬁcult (i.e., adversarial) attack in B d ( s, ✏ ) , where ✏ controls the perturbation scale.

Original: To solve such amin-max problem, we maximize L ( Q (ˆ s, a ) , Q ( s, a )) to obtain ˆ s , and then minimize the loss functionwith the selected ˆ s .
Revised: To solve such amin-max problem, we maximize L ( Q (ˆ s, a ) , Q ( s, a )) to obtain ˆ s , and then minimize the loss functionwith the selected ˆ s .

Original: [Equation]
Revised: [Equation]

Original: where L ( · , · ) is the smooth loss function, and we denote  ( s, ˆ s, a ) =
Revised: where L ( · , · ) is the smooth loss function, and we denote  ( s, ˆ s, a ) =

Original: Q  i (ˆ s, a )  Q  i ( s, a ) .
Revised: Q  i (ˆ s, a )  Q  i ( s, a ) .

Original: Weremark that if  ( s, ˆ s, a ) > 0 , the perturbed state may induce an overestimated Q -value that we needto smooth.
Revised: Weremark that if  ( s, ˆ s, a ) > 0 , the perturbed state may induce an overestimated Q -value that we needto smooth.

Original: In contrast, if  ( s, ˆ s, a ) < 0 , the perturbed Q -function is underestimated, which doesnot cause a serious problem in ofﬂine RL.
Revised: In contrast, if  ( s, ˆ s, a ) < 0 , the perturbed Q -function is underestimated, which doesnot cause a serious problem in ofﬂine RL.

Original: As a result, we use different weights for  ( s, ˆ s, a ) + and ( s, ˆ s, a )  .
Revised: As a result, we use different weights for  ( s, ˆ s, a ) + and ( s, ˆ s, a )  .

Original: The deﬁnition of L ( · , · ) is give as follows,
Revised: The deﬁnition of L ( · , · ) is give as follows,

Original: [Equation]
Revised: [Equation]

Original: where we choose ⌧  0 .
Revised: where we choose ⌧  0 .

Original: 5 , x + = max( x, 0) and x  = min( x, 0) .
Revised: 5 , x + = max( x, 0) and x  = min( x, 0) .

Original: In this term, we does not introduce
Revised: In this term, we does not introduce

Original: OOD action ˆ a for smoothing since the actions are desired to be close to the behavior actions in thedataset for areas near the ofﬂine data.
Revised: OOD action ˆ a for smoothing since the actions are desired to be close to the behavior actions in thedataset for areas near the ofﬂine data.

Original: Finally, to prevent overestimation of OOD states and actions, we use bootstrapped uncertainty u
Revised: Finally, to prevent overestimation of OOD states and actions, we use bootstrapped uncertainty u

Original: (ˆ s, ˆ a )as a penalty for Q (ˆ s, ˆ a ) , where ˆ a ⇠ ⇡ (ˆ s ) is an OOD action sampled from the current policy ⇡ .
Revised: (ˆ s, ˆ a )as a penalty for Q (ˆ s, ˆ a ) , where ˆ a ⇠ ⇡ (ˆ s ) is an OOD action sampled from the current policy ⇡ .

Original: We remark that a similar OOD sampling is also used in PBRL [4].
Revised: We remark that a similar OOD sampling is also used in PBRL [3].

Original: The difference is that PBRLonly penalizes the OOD actions, while RORL penalizes both the OOD states and actions to provideconservatism for unfamiliar areas.
Revised: The difference is that PBRLonly penalizes the OOD actions, while RORL penalizes both the OOD states and actions to provideconservatism for unfamiliar areas.

Original: The ensemble technique in RL comes from Bootstrapped DQN[35].
Revised: The ensemble technique in RL comes from Bootstrapped DQN[33].

Original: The ensemble forms an estimation of the Q -posterior, which yields diverse predictions on areaswith scarce data.
Revised: The ensemble forms an estimation of the Q -posterior, which yields diverse predictions on areaswith scarce data.

Original: u (ˆ s, ˆ a ) quantiﬁes the deviation of a datapoint from the ofﬂine dataset [2, 4].
Revised: u (ˆ s, ˆ a ) quantiﬁes the deviation of a datapoint from the ofﬂine dataset [2, 3].

Original: Wefollow PBRL and use a loss function as:
Revised: Wefollow PBRL and use a loss function as:

Original: [Equation]
Revised: [Equation]

Original: where the pseudo-target for the OOD datapoints is computed as:
Revised: where the pseudo-target for the OOD datapoints is computed as:

Original: [Equation]
Revised: [Equation]

Original: u (ˆ s, ˆ a ) , where u (ˆ s, ˆ a ) :=
Revised: u (ˆ s, ˆ a ) , where u (ˆ s, ˆ a ) :=

Revised: [Equation]

Revised:

Original: [Equation]
Revised: [Equation]

Original: , The bootstrapped uncertaintyu (ˆ s, ˆ a ) is deﬁned as the standard deviation among the Q -ensemble.
Revised: , The bootstrapped uncertaintyu (ˆ s, ˆ a ) is deﬁned as the standard deviation among the Q -ensemble.

Original: Combining the loss functions above, RORL has the following loss function for each Q  i :
Revised: Combining the loss functions above, RORL has the following loss function for each Q  i :

Original: [Equation]
Revised: [Equation]

Original: Robust Policy We learn a robust policy by using a smooth constraint to make the policy changeless with perturbations.
Revised: Robust Policy We learn a robust policy by using a smooth constraint to make the policy changeless with perturbations.

Original: Similarly, we choose an adversarial state ˆ s 2 B d ( s, ✏ ) that maximizes
Revised: Similarly, we choose an adversarial state ˆ s 2 B d ( s, ✏ ) that maximizes

Original: D J  ⇡ ✓ ( ·| s ) k ⇡ ✓ ( ·| ˆ s )  , and then minimize the policy difference between ⇡ ✓ ( ·| s ) and ⇡ ✓ ( ·| ˆ s ) .
Revised: D J  ⇡ ✓ ( ·| s ) k ⇡ ✓ ( ·| ˆ s )  , and then minimize the policy difference between ⇡ ✓ ( ·| s ) and ⇡ ✓ ( ·| ˆ s ) .

Original: To 163 conclude, we minimize the following loss function for the policy:
Revised: To 159 conclude, we minimize the following loss function for the policy,

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst term aims to maximize the minimum of the ensemble Q -functions to obtain aconservative policy, and the third term is regularization of entropy.
Revised: where the ﬁrst term aims to maximize the minimum of the ensemble Q -functions to obtain aconservative policy, and the third term is regularization of entropy.

Original:  Theoretical Analysis
Revised:  Theoretical Analysis

Original: We analyze the simpliﬁed learning objective of RORL in linear MDPs [21, 22], where the featuremap of the state-action pair takes the form of  : S ⇥ A !
Revised: We analyze the simpliﬁed learning objective of RORL in linear MDPs [20, 21], where the featuremap of the state-action pair takes the form of  : S ⇥ A !

Original: R d , and the transition kernel and thereward function are assumed to be linear in  .
Revised: R d , and the transition kernel and thereward function are assumed to be linear in  .

Original: The parameter e w t of RORL can be solved in closedform following the least squares value iteration algorithm (LSVI), which minimizes the followingloss function.
Revised: The parameter e w t of RORL can be solved in closedform following the least squares value iteration algorithm (LSVI), which minimizes the followingloss function.

Original: [Equation]
Revised: [Equation]

Original: where we have Q w ( s it , a it ))
Revised: where we have Q w ( s it , a it ))

Original: =
Revised: =

Original:  ( s it , a it ) > w since the Q -function is also linear in  .
Revised:  ( s it , a it ) > w since the Q -function is also linear in  .

Original: The ﬁrst termin Eq.
Revised: The ﬁrst termin Eq.

Original: (6) is the ordinary TD-error, where we consider the setting of  = 1 and the Q -targetis y it =
Revised: (7) is the ordinary TD-error, where we consider the setting of  = 1 and the Q -targetis y it =

Original: r ( s it , a it )
Revised: r ( s it , a it )

Original: + V t +

Revised: + V t +

Original: ( s it +1 ) .
Revised: ( s it +1 ) .

Original: The second term is the proposed conservative smoothing loss.
Revised: The second term is the proposed conservative smoothing loss.

Original: Speciﬁcally, ˆ s it ⇠ D ood
Revised: Speciﬁcally, ˆ s it ⇠ D ood

Original: ( s it ) are sampled from a l 1 ball of center s it and norm ✏ > 0 , which can beformulated as ˆ s ⇠ B d ( s it , ✏ ) .
Revised: ( s it ) are sampled from a l 1 ball of center s it and norm ✏ > 0 , which can beformulated as ˆ s ⇠ B d ( s it , ✏ ) .

Original: The third term is the additional OOD-sampling loss [4], which enforcesconservatism for OOD actions.
Revised: The third term is the additional OOD-sampling loss [3], which enforcesconservatism for OOD actions.

Original: Differently, we use perturbed states sampled from D ood = m
Revised: Differently, we use perturbed states sampled from D ood = m

Original: S i =
Revised: S i =

Original: 1 D ood ( s it )rather than states from dataset in PBRL.
Revised: 1 D ood ( s it )rather than states from dataset in PBRL.

Original: The OOD action ˆ a is sampled from policy ⇡ .
Revised: The OOD action ˆ a is sampled from policy ⇡ .

Original: The explicitsolution of Eq.
Revised: The explicitsolution of Eq.

Original: (6) takes the following form by following LSVI:
Revised: (7) takes the following form by following LSVI:

Original: [Equation]
Revised: [Equation]

Original: where the covariance matrix e ⇤ t is deﬁned as
Revised: where the covariance matrix e ⇤ t is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: We denote the ﬁrst term and the second term as e ⇤ in and e ⇤ ood t , which represent the covariance matricesinduced by the ofﬂine samples and OOD samples, respectively.
Revised: We denote the ﬁrst term and the second term as e [Equation] e ⇤ ood t , which represent the covariance matricesinduced by the ofﬂine samples and OOD samples, respectively.

Original: Nevertheless, in linear MDPs, it isdifﬁcult to ensure the covariance
Revised: Nevertheless, in linear MDPs, it isdifﬁcult to ensure the covariance

Original: [Equation]
Revised: [Equation]

Original: ⌫  I , since it requires that the embeddings of the samplesare isotropic to make the eigenvalues of the corresponding covariance matrix lower bounded.
Revised: ⌫  I , since it requires that the embeddings of the samplesare isotropic to make the eigenvalues of the corresponding covariance matrix lower bounded.

Original: Thiscondition holds if we can sample embeddings uniformly from the whole embedding space.
Revised: Thiscondition holds if we can sample embeddings uniformly from the whole embedding space.

Original: However,since the ofﬂine dataset has limited coverage in the state-action space and the OOD samples comefrom a limited l 1 -ball around the ofﬂine data,
Revised: However,since the ofﬂine dataset has limited coverage in the state-action space and the OOD samples comefrom a limited l 1 -ball around the ofﬂine data,

Original: [Equation]
Revised: [Equation]

Original: cannot be guaranteed to be positive deﬁnite.
Revised: cannot be guaranteed to be positive deﬁnite.

Original: PBRL [4] uses the assumption of e ⇤ ood
Revised: PBRL [3] uses the assumption of e ⇤ ood

Revised: [Equation]

Original: t ⌫  I , while it is unachievable empirically.
Revised: ⌫  I , while it is unachievable empirically.

Original: In RORL, we solvethis problem by introducing an additional conservative smoothing loss, which induces a covariancematrix as
Revised: In RORL, we solvethis problem by introducing an additional conservative smoothing loss, which induces a covariancematrix as

Original: [Equation]
Revised: [Equation]

Original: a it )] [  (ˆ s it , a it )   ( s it , a it )] > (i.e.,the third term in Eq.
Revised: ][  (ˆ s it , a it )   ( s it , a it )] > (i.e.,the third term in Eq.

Original: The following theorem gives the guarantees of e ⇤ ood_diff t ⌫  I .
Revised: The following theorem gives the guarantees of e ⇤ ood_diff t ⌫  I .

Original: Theorem 1. Assume [-
-] i 2 [1 , m ] the vector group of all ˆ s it
Revised: Theorem 1 Assume [-
-]9 i 2 [1 , m ] the vector group of all ˆ s it

Original: ⇠ D ood
Revised: ⇠ D ood

Original: ( s it ) : {  (ˆ s it , a it )   ( s it , a it ) } befull rank, then the covariance matrix e ⇤ ood _ di ↵ t is positive-deﬁnite: e ⇤ ood _ di ↵ t ⌫  · I where  > 0 .
Revised: ( s it ) : {  (ˆ s it , a it )   ( s it , a it ) } befull rank, then the covariance matrix e ⇤ ood _ di ↵ t is positive-deﬁnite: e ⇤ ood _ di ↵ t ⌫  · I where  > 0 .

Original: Recall the covariance matrix of PBRL is e ⇤ PBRL t
Revised: Recall the covariance matrix of PBRL is e ⇤ PBRL t

Original: =
Revised: =

Original: e ⇤ in t + e ⇤ ood
Revised: e ⇤ in t + e ⇤ ood

Original: t , and RORL has a covariance matrix ase ⇤ t = e ⇤ PBRL t
Revised: t , and RORL has a covariance matrix ase ⇤ t = e ⇤ PBRL t

Original: + e ⇤ ood_diff t , we have the following corollary based on Theorem 1.
Revised: + e ⇤ ood_diff t , we have the following corollary based on Theorem 1.

Original: Corollary 1. Under the linear MDP assumptions, we have e ⇤ t ⌫ e ⇤ PBRL t .
Revised: Corollary 1 Under the linear MDP assumptions, we have e ⇤ t ⌫ e ⇤ PBRL t .

Original: Further, the covariancematrix e ⇤ t of RORL is positive-deﬁnite: e ⇤ t ⌫  · I , where  > 0 .
Revised: Further, the covariancematrix e ⇤ t of RORL is positive-deﬁnite: e ⇤ t ⌫  · I , where  > 0 .

Original: Recent theoretical analysis shows that an appropriate uncertainty quantiﬁcation is essential to provableefﬁciency in ofﬂine RL [22, 58, 4].
Revised: Recent theoretical analysis shows that an appropriate uncertainty quantiﬁcation is essential to provableefﬁciency in ofﬂine RL [21, 56, 3].

Original: Pessimistic Value Iteration [22] deﬁnes a general ⇠ -uncertaintyquantiﬁer as the penalty and achieves provable efﬁcient pessimism in ofﬂine RL.
Revised: Pessimistic Value Iteration [21] deﬁnes a general ⇠ -uncertaintyquantiﬁer as the penalty and achieves provable efﬁcient pessimism in ofﬂine RL.

Original: In linear MDPs,
Revised: In linear MDPs,

Original: Lower Conﬁdence Bound (LCB)-penalty [1, 21] is known to be a ⇠ -uncertainty quantiﬁer forappropriately selected  t as  lcb ( s t , a t ) =
Revised: Lower Conﬁdence Bound (LCB)-penalty [1, 20] is known to be a ⇠ -uncertainty quantiﬁer forappropriately selected  t as  lcb ( s t , a t ) =

Original:  t · ⇥  ( s t , a t ) >
Revised:  t · ⇥  ( s t , a t ) >

Original: ⇤  1 t  ( s t , a t ) ⇤ 1 / 2 .
Revised: ⇤  1 t  ( s t , a t ) ⇤ 1 / 2 .

Original: Following the analysisof PBRL [4], since the bootstrapped uncertainty is an estimation to the LCB-penalty and the OODsampling provides a covariance matrix of e ⇤ PBRL t , the proposed RORL also form a valid ⇠ -uncertaintyquantiﬁer with the covariance matrix e ⇤ t ⌫  · I given in Corollary 1.
Revised: Following the analysisof PBRL [3], since the bootstrapped uncertainty is an estimation to the LCB-penalty and the OODsampling provides a covariance matrix of e ⇤ PBRL t , the proposed RORL also form a valid ⇠ -uncertaintyquantiﬁer with the covariance matrix e ⇤ t ⌫  · I given in Corollary 1.

Original: This allows us to furthercharacterize the optimality gap based on the pessimistic value iteration [22, 4].
Revised: This allows us to furthercharacterize the optimality gap based on the pessimistic value iteration [21, 3].

Original: We have the followingsuboptimality gap under linear MDP assumptions.
Revised: We have the followingsuboptimality gap under linear MDP assumptions.

Original: Corollary 2. SubOpt( ⇡ ⇤ , ˆ ⇡ ) 
Revised: Corollary 2 SubOpt

Original: [Equation]
Revised: [Equation]

Original: We refer to Appendix A for detailed proof.
Revised: We refer to Appendix A for detailed proof.

Original: The second inequality is directly induced by  lcb t ( s t , a t ) <
Revised: The second inequality is directly induced by  lcb t ( s t , a t ) <

Original:  lcb _ PBRL t ( s t , a t ) .
Revised:  lcb _ PBRL t ( s t , a t ) .

Original: Therefore, RORL enjoys a tighter suboptimality bound than PBRL [4].
Revised: Therefore, RORL enjoys a tighter suboptimality bound than PBRL [3].

Original:  Experiments
Revised:  Experiments

Original: We evaluate our method on D4RL benchmark [11] with various continuous-control tasks and datasets.
Revised: We evaluate our method on D4RL benchmark [10] with various continuous-control tasks and datasets.

Original: We compare RORL with several ofﬂine RL algorithms, including (i) BC that performs behaviorcloning, (ii) CQL [27] that learns conservative value function for OOD actions, (iii) EDAC [2] thatlearns a diversiﬁed Q -ensemble to enforce conservatism, and (iv) PBRL [4] that performs uncertaintypenalization and OOD sampling.
Revised: We compare RORL with several ofﬂine RL algorithms, including (i) BC that performs behaviorcloning, (ii) CQL [25] that learns conservative value function for OOD actions, (iii) EDAC [2] thatlearns a diversiﬁed Q -ensemble to enforce conservatism, and (iv) PBRL [3] that performs uncertaintypenalization and OOD sampling.

Original: Among these methods, EDAC [2] and PBRL [4] are related to
Revised: Among these methods, EDAC [2] and PBRL [3] are related to

Original: RORL since all these methods apply Q -ensemble for conservatism.
Revised: RORL since all these methods apply Q -ensemble for conservatism.

Original: EDAC needs much more Qnetworks (i.e., 10 ⇠ 50) for hopper tasks than PBRL and RORL that use 10 Q -networks.
Revised: EDAC needs much more Qnetworks (i.e., 10 ⇠ 50) for hopper tasks than PBRL and RORL that use 10 Q -networks.

Original: (v)
Revised: (v)

Original: Wealso include a basic SAC-10 algorithm as a baseline [2], which is an extension of SAC with 

Revised: Wealso include a basic SAC-10 algorithm as a baseline [2], which is an extension of SAC with

Revised: Q -functions.

Original: [Table]
Revised: [Table]

Original: 6.1 Benchmark Results
Revised: 6.1 Benchmark Results

Original: We evaluate each method on Gym domain that includes three environments (Half Cheetah, Hopper,and Walker2d) with four non-expert datasets (random, medium, medium-replay, and medium-expert).
Revised: We evaluate each method on Gym domain that includes three environments (Half Cheetah, Hopper,and Walker2d) with four non-expert datasets (random, medium, medium-replay, and medium-expert).

Original: The medium-replay dataset contains experiences collected in training a medium-level policy.
Revised: The medium-replay dataset contains experiences collected in training a medium-level policy.

Original: Therandom or medium dataset is generated by a single random or medium policy.
Revised: Therandom or medium dataset is generated by a single random or medium policy.

Original: The mediumexpert dataset is a mixture of medium and expert datasets.
Revised: The mediumexpert dataset is a mixture of medium and expert datasets.

Original: We set small perturbation scale ✏ to0.001/0.005/0.01 for halfcheetah/hopper/walker2d tasks respectively when training RORL and do notinclude observation perturbation in the testing time.
Revised: We set small perturbation scale ✏ to0.001/0.005/0.01 for halfcheetah/hopper/walker2d tasks respectively when training RORL and do notinclude observation perturbation in the testing time.

Original: Table 1 reports the performance of the average normalized score with standard deviation.
Revised: Table 1 reports the performance of the average normalized score with standard deviation.

Original: (i) SAC-10is unstable in several Walker2d and Hopper tasks since the ensemble number is relatively smallto provide reliable uncertainties.
Revised: (i) SAC-10is unstable in several Walker2d and Hopper tasks since the ensemble number is relatively smallto provide reliable uncertainties.

Original: (ii) EDAC solves this problem by using diversity constraintswhile still requiring 10 ⇠ 50 Q -networks to obtain reasonable performance.
Revised: (ii) EDAC solves this problem by using diversity constraintswhile still requiring 10 ⇠ 50 Q -networks to obtain reasonable performance.

Original: In contrast, RORL onlyuses 10 ensemble Q -networks to outperform EDAC with 10 ⇠ 50 networks.
Revised: In contrast, RORL onlyuses 10 ensemble Q -networks to outperform EDAC with 10 ⇠ 50 networks.

Original: (iii) PBRL chooses analternative OOD-sampling technique to reduce the ensemble numbers.
Revised: (iii) PBRL chooses analternative OOD-sampling technique to reduce the ensemble numbers.

Original: According to the result, RORLsigniﬁcantly outperforms PBRL with the same ensemble number.
Revised: According to the result, RORLsigniﬁcantly outperforms PBRL with the same ensemble number.

Original: The reason is RORL additionallyuses conservative smoothing loss for both OOD states and actions, which improves the generalizationability of the policy and Q -functions.
Revised: The reason is RORL additionallyuses conservative smoothing loss for both OOD states and actions, which improves the generalizationability of the policy and Q -functions.

Original: We remark that RORL signiﬁcantly improves over the current
Revised: We remark that RORL signiﬁcantly improves over the current

Original: SOTA results on walker2d and hopper tasks, which might be because the two tasks require a moreprecise balancing of conservatism and generalization for better performance.
Revised: SOTA results on walker2d and hopper tasks, which might be because the two tasks require a moreprecise balancing of conservatism and generalization for better performance.

Original: 6.2 Adversarial Attack
Revised: 6.2 Adversarial Attack

Original: We adopt three attack methods, namely random , action diff , and min Q following prior works [64, 39].
Revised: We adopt three attack methods, namely random , action diff , and min Q following prior works [62, 37].

Original: Given perturbation scale ✏ , the later two methods perform adversarial perturbation on observationsand are given access to the agent’s policy and value functions.
Revised: Given perturbation scale ✏ , the later two methods perform adversarial perturbation on observationsand are given access to the agent’s policy and value functions.

Original: • random uniformly samples perturbed states in an l 1 ball of norm ✏ .
Revised: • random uniformly samples perturbed states in an l 1 ball of norm ✏ .

Original: • action diff is an effective attack based on the agent’s policy and is proved to maximize an upper bound on the performance difference between perturbed and unperturbed environments [64].
Revised: • action diff is an effective attack based on the agent’s policy and is proved to max- imize an upper bound on the performance difference between perturbed and unper- turbed environments [62].

Original: It directly ﬁnds states in an l 1 ball of norm ✏ to minimize:min ˆ s 2 B d ( s, ✏ )  D J  ⇡ ✓ ( ·| s ) k ⇡ ✓ ( ·| ˆ s )  .•
Revised: It directly ﬁnds states in an l 1 ball of norm ✏ to minimize: min ˆ s 2 B d ( s, ✏ )  D J  ⇡ ✓ ( ·| s ) k ⇡ ✓ ( ·| ˆ s )  .

Original:  min Q requires both the agent’s policy and value function to perform a relatively strongerattack.
Revised: • min Q requires both the agent’s policy and value function to perform a relatively stronger attack.

Original: The attacker ﬁnds a perturbed state to minimize the expected return of taking anaction from that state: min ˆ s 2 B d ( s, ✏ ) Q ( s, ⇡ ✓ (ˆ s )) .
Revised: The attacker ﬁnds a perturbed state to minimize the expected return of taking an action from that state: min ˆ s 2 B d ( s, ✏ ) Q ( s, ⇡ ✓ (ˆ s )) .

Original: For ensemble-based algorithms, Q is setas the mean of ensemble Q functions.
Revised: For ensemble-based algorithms, Q is set as the mean of ensemble Q functions.

Original: In our experiments, the two objectives of action diff and min Q are optimized via two ways.
Revised: In our experiments, the two objectives of action diff and min Q are optimized via two ways.

Original: Speciﬁcally, we optimize the objectives through:
Revised: Speciﬁcally, we optimize the objectives through:

Original: [Figure]
Revised: [Figure]

Original: (1) selecting the best state from uniformly sampled 50 states, which has the advantage ofsimplicity and little computation cost.
Revised: (1) selecting the best state from uniformly sampled 50 states, which has the advantage ofsimplicity and little computation cost.

Original: (2) uniformly sampling 20 initial states and performing gradient decent for 10 steps with a stepsize of 110 ✏ from each initial state to ﬁnd the best perturbed state.
Revised: (2) uniformly sampling 20 initial states and performing gradient decent for 10 steps with a stepsize of 110 ✏ from each initial state to ﬁnd the best perturbed state.

Original: Note that we need to clipthe perturbed states within the l 1 ball at the end of each optimization step.
Revised: Note that we need to clipthe perturbed states within the l 1 ball at the end of each optimization step.

Original: We name thisoptimization method “mixed order” optimization.
Revised: We name this optimization method ‘mixed order’ optimization.

Original: We compare RORL with ensemble-based baselines EDAC and SAC-10 on halfcheetah-medium-v2and walker2d-medium-v2 datasets.
Revised: We compare RORL with ensemble-based baselines EDAC and SAC-10 on halfcheetah-medium-v2and walker2d-medium-v2 datasets.

Original: To handle large adversarial noise, we set the perturbation scale ofthe policy and Q functions as [0.1, 0.05] for halfcheetah-medium-v2 and [0.1, 0.03] for walker2dmedium-v2 in RORL’s training phase.
Revised: To handle large adversarial noise, we set the perturbation scale ofthe policy and Q functions as [0.1, 0.05] for halfcheetah-medium-v2 and [0.1, 0.03] for walker2dmedium-v2 in RORL’s training phase.

Original: More detailed description can be found in Appendix B. Theresults are shown in Figure 4.
Revised: More detailed description can be found in Appendix B. Theresults are shown in Figure 4.

Original: In the results, RORL exhibits improved robustness than other baselinesunder adversarial attacks and decreases the slowest as ✏ increases.
Revised: In the results, RORL exhibits improved robustness than other baselinesunder adversarial attacks and decreases the slowest as ✏ increases.

Original: On the other hand, we ﬁnd thatrandom attack is not effective for ensemble-based ofﬂine RL algorithms, and the “mixed order” attackbrings more signiﬁcant performance drop than vanilla zero-order optimization.
Revised: On the other hand, we ﬁnd thatrandom attack is not effective for ensemble-based ofﬂine RL algorithms, and the ‘mixed order’ attackbrings more signiﬁcant performance drop than vanilla zero-order optimization.

Original: 6.3 Ablations
Revised: 6.3 Ablations

Original: We conducted ablation studies on walker2d-medium-v2 dataset to evaluate the importance of thepolicy smoothing term, the Q function smoothing term and the penalty for OOD states in Figure5.
Revised: We conducted ablation studies on walker2d-medium-v2 dataset to evaluate the importance of thepolicy smoothing term, the Q function smoothing term and the penalty for OOD states in Figure5.

Original: From the results, we can conclude that each loss contributes to the performance of RORL underadversarial observation attacks.
Revised: From the results, we can conclude that each loss contributes to the performance of RORL underadversarial observation attacks.

Original: The policy smoothing loss is essential when the perturbation scale islarge, while the performance of “no P smooth” is higher than RORL when the scale is smaller than 0.1.
Revised: The policy smoothing loss is essential when the perturbation scale islarge, while the performance of ’no P smooth’ is higher than RORL when the scale is smaller than 0.1.

Original: Besides, learning without OOD loss performs worse than RORL at almost all the perturbation scales.
Revised: Besides, learning without OOD loss performs worse than RORL at almost all the perturbation scales.

Original: In addition, Q smooth loss has the minimal impact, which is reasonable since the basic algorithm
Revised: In addition, Q smooth loss has the minimal impact, which is reasonable since the basic algorithm

Original: SAC-10 is based on 10 ensemble Q networks.
Revised: SAC-10 is based on 10 ensemble Q networks.

Original: Our smoothing technique may have a more signiﬁcantimpact on baselines with fewer Q networks, such as CQL in Section 3.
Revised: Our smoothing technique may have a more signiﬁcantimpact on baselines with fewer Q networks, such as CQL in Section 3.

Original:  Related Works
Revised:  Related Works

Original: Ofﬂine RL Research related to ofﬂine RL has experienced explosive growth in recent years.
Revised: Ofﬂine RL Research related to ofﬂine RL has experienced explosive growth in recent years.

Original: Inmodel-free domain, ofﬂine RL methods focus on correcting the extrapolation error [13] in the offpolicy algorithms.
Revised: Inmodel-free domain, ofﬂine RL methods focus on correcting the extrapolation error [12] in the offpolicy algorithms.

Original: The natural idea is to regularize the learned policy near the dataset distribution [52,56, 33, 54, 60, 12].
Revised: The natural idea is to regularize the learned policy near the dataset distribution [51,54, 31, 52, 58, 11].

Original: For example, MARVIL reweights the policy with exponential advantage, whichimplicitly guarantees the policy within the KL-divergence neighborhood of the behavior policy.
Revised: For example, MARVIL reweights the policy with exponential advantage, whichimplicitly guarantees the policy within the KL-divergence neighborhood of the behavior policy.

Original: Another stream of model-free methods prevents the selection of OOD actions by penalizing their
Revised: Another stream of model-free methods prevents the selection of OOD actions by penalizing their

Original: Q-value [26, 27, 2, 8].
Revised: Q-value [24, 25, 2, 7].

Original: With the ensemble Q networks and the additional loss term to diversify theirgradients, EDAC [2] achieves SOTA performance in the D4RL benchmark.
Revised: With the ensemble Q networks and the additional loss term to diversify theirgradients, EDAC [2] achieves SOTA performance in the D4RL benchmark.

Original: Instead of diversifyinggradients, PBRL [4] proposes explicit training of the OOD actions, which achieves comparableperformance with fewer ensemble networks.
Revised: Instead of diversifyinggradients, PBRL [3] proposes explicit training of the OOD actions, which achieves comparableperformance with fewer ensemble networks.

Original: Inspired by EDAC and PBRL, we build our work uponensemble networks, focusing more on smoothness over the state space.
Revised: Inspired by EDAC and PBRL, we build our work uponensemble networks, focusing more on smoothness over the state space.

Original: Besides the surprising empirical results, theoretical analysis of ofﬂine reinforcement learning algorithms is of increasing interest [7, 22, 41, 58, 61].
Revised: Besides the surprising empirical results, theoretical analysis of ofﬂine reinforcement learning algorithms is of increasing interest [6, 21, 39, 56, 59].

Original: Though the assumptions for the dataset vary in thedifferent papers, they all suggest that pessimism and conservatism are necessary for ofﬂine RL.
Revised: Though the assumptions for the dataset vary in thedifferent papers, they all suggest that pessimism and conservatism are necessary for ofﬂine RL.

Original: Ourpaper can be viewed as a robust extension to the previous work [22] (see Section 5 for details).
Revised: Ourpaper can be viewed as a robust extension to the previous work [21] (see Section 5 for details).

Original: Robust RL The research line of robust RL can be traced back to H 1 -control theory [57, 5],where policies are optimized to be well-performed in the worst possible deterministic environment.
Revised: Robust RL The research line of robust RL can be traced back to H 1 -control theory [55, 4],where policies are optimized to be well-performed in the worst possible deterministic environment.

Original: Depending on the deﬁnition, there are different streams of research on robust RL.
Revised: Depending on the deﬁnition, there are different streams of research on robust RL.

Original: As the extensionof robust control to MDPs, Robust MDPs (RMDPs) [34, 19, 42, 17] are proposed to formulate theperturbation of transition probabilities for MDPs.
Revised: As the extensionof robust control to MDPs, Robust MDPs (RMDPs) [32, 18, 41, 16] are proposed to formulate theperturbation of transition probabilities for MDPs.

Original: Though some recent analyses with theoreticalguarantees come out under speciﬁc assumptions for RMDP [66, 59, 29], there is currently no practicalalgorithm to solve RMDP in a large-scale problem, expect some linear approximation attempt [47].
Revised: Though some recent analyses with theoreticalguarantees come out under speciﬁc assumptions for RMDP [64, 57, 27], there is currently no practicalalgorithm to solve RMDP in a large-scale problem, expect some linear approximation attempt [46].

Original: In online RL, domain randomization [49, 31] assumes the model uncertainty can be predeﬁned in datacollection by changing the setup of a simulator.
Revised: In online RL, domain randomization [48, 29] assumes the model uncertainty can be predeﬁned in datacollection by changing the setup of a simulator.

Original: However, it is not practical for ofﬂine RL.
Revised: However, it is not practical for ofﬂine RL.

Original: Robust
Revised: Robust

Original: Adversarial Reinforcement Learning (RARL) [39] and Noisy Robust Markov Decision Process(NR-MDP) [23] study the robust RL with the perturbed actions, showing that the policy robustness toadversarial or noisy actions can also induce robustness for model parameter changes.
Revised: Adversarial Reinforcement Learning (RARL) [37] and Noisy Robust Markov Decision Process(NR-MDP) [22] study the robust RL with the perturbed actions, showing that the policy robustness toadversarial or noisy actions can also induce robustness for model parameter changes.

Original: The most related work to ours is SR 2 L [44], which shows policy smoothing can lead to signiﬁcantperformance improvements in the online setting.
Revised: The most related work to ours is SR 2 L [43], which shows policy smoothing can lead to signiﬁcantperformance improvements in the online setting.

Original: In contrast, we focus on the ofﬂine setting andtackle the potential overestimation of perturbed states.
Revised: In contrast, we focus on the ofﬂine setting andtackle the potential overestimation of perturbed states.

Original: Another related work is S4RL [46], where theauthors study different data augmentation methods to smooth observations in ofﬂine RL.
Revised: Another related work is S4RL [45], where theauthors study different data augmentation methods to smooth observations in ofﬂine RL.

Original: Their resultsupports the necessity of state smoothing.
Revised: Their resultsupports the necessity of state smoothing.

Original: More related works are given in Appendix D.
Revised: More related works are given in Appendix D.

Original:  Conclusion
Revised:  Conclusion

Original: We propose Robust Ofﬂine Reinforcement Learning (RORL) to trade-off conservatism and robustnessfor ofﬂine RL.
Revised: We propose Robust Ofﬂine Reinforcement Learning (RORL) to trade-off conservatism and robustnessfor ofﬂine RL.

Original: To achieve that, we introduce the conservative smoothing for the perturbed stateswhile actively underestimating their values based on pessimistic bootstrapping to keep conservative.
Revised: To achieve that, we introduce the conservative smoothing for the perturbed stateswhile actively underestimating their values based on pessimistic bootstrapping to keep conservative.

Original: We show that RORL can achieve comparable performance with fewer ensemble Q networks than theprevious methods in the ofﬂine RL benchmark.
Revised: We show that RORL can achieve comparable performance with fewer ensemble Q networks than theprevious methods in the ofﬂine RL benchmark.

Original: The result supports that our proposed conservativesmoothing techniques beneﬁt general ofﬂine RL by improving the generalization ability.
Revised: The result supports that our proposed conservativesmoothing techniques beneﬁt general ofﬂine RL by improving the generalization ability.

Original: In addition,we demonstrate that RORL is also robust to adversarial perturbations across the different types ofattacks.
Revised: In addition,we demonstrate that RORL is also robust to adversarial perturbations across the different types ofattacks.

Original: We hope our work can promote the application of ofﬂine RL under real-world engineeringconditions.
Revised: We hope our work can promote the application of ofﬂine RL under real-world engineeringconditions.

Original: The main limitation of our method is that the adversarial state sampling slows down the computingprocess, which may be improved in future work.
Revised: The main limitation of our method is that the adversarial state sampling slows down the computingprocess, which may be improved in future work.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.
Revised: Please read the checklist guidelines carefully for information onhow to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .

Original: You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] See Section ?? .
Revised: [Yes] See Section ?? .•

Original: • Did you include the license to the code and datasets?
Revised: Did you include the license to the code and datasets?

Original: [No] The code and the data areproprietary.•
Revised: [No] The code and the data areproprietary.•

Original: Did you include the license to the code and datasets?
Revised: Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?

Original: [Yes](b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes](c)
Revised: [Yes](c)

Original: Did you discuss any potential negative societal impacts of your work?
Revised: Did you discuss any potential negative societal impacts of your work?

Original: [N/A](d)
Revised: [N/A](d)

Original: Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]2.
Revised: [Yes]2.

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes](b) Did you include complete proofs of all theoretical results?
Revised: [Yes](b) Did you include complete proofs of all theoretical results?

Original: [Yes] See Appendix A.3.
Revised: [Yes] See Appendix A.3.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] See Sec 1 andAppendix B.(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: [Yes] See Sec 1 andAppendix B.(b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes] See Appendix B.
Revised: [Yes] See Appendix B.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] See Sec 6.
Revised: [Yes] See Sec 6.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Appendix B.
Revised: [Yes] See Appendix B.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] We cited D4RL [11] and EDAC[2] for their datasets and code.
Revised: [Yes] We cited D4RL [10]and EDAC[2] for their datasets and code.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Original: We included our code in the anonymized link.
Revised: We included our code in the anonymized link.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [Yes] Opensource code and dataset.
Revised: [Yes] Opensource code and dataset.

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]