Original: A Closer Look at Ofﬂine RL Agents
Revised: A Closer Look at Ofﬂine RL Agents

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Despite recent advances in the ﬁeld of Ofﬂine Reinforcement Learning (RL),less attention has been paid to understanding the behaviors of learned RL agents.
Revised: Despite recent advances in the ﬁeld of Ofﬂine Reinforcement Learning (RL),less attention has been paid to understanding the behaviors of learned RL agents.

Original: As a result, there remain some gaps in our understandings, i.e. , why is one ofﬂine RL agent more performant than another?
Revised: As a result, there remain some gaps in our understandings, i.e. , why is one ofﬂine RL agent more performant than another?

Original: In this work, we ﬁrst introducea set of experiments to evaluate ofﬂine RL agents, focusing on three fundamental aspects: representations, value functions and policies.
Revised: In this work, we ﬁrst introducea set of experiments to evaluate ofﬂine RL agents, focusing on three fundamental aspects: representations, value functions and policies.

Original: Counterintuitively, weshow that a more performant ofﬂine RL agent can learn relatively low-qualityrepresentations and inaccurate value functions.
Revised: Counterintuitively, weshow that a more performant ofﬂine RL agent can learn relatively low-qualityrepresentations and inaccurate value functions.

Original: Furthermore, we demonstratethat the proposed experiment setups can be effectively used to diagnose the bottleneck of ofﬂine RL agents.
Revised: Furthermore, we demonstratethat the proposed experiment setups can be effectively used to diagnose the bottleneck of ofﬂine RL agents.

Original: Inspired by the evaluation results, a novel ofﬂine
Revised: Inspired by the evaluation results, a novel ofﬂine

Original: RL algorithm is proposed by a simple modiﬁcation of IQL and achieves SOTAperformance.
Revised: RL algorithm is proposed by a simple modiﬁcation of IQL and achieves SOTAperformance.

Original: Finally, we investigate when a learned dynamics model is helpfulto model-free ofﬂine RL agents, and introduce an uncertainty-based sample selection method to mitigate the problem of model noises.
Revised: Finally, we investigate when a learned dynamics model is helpfulto model-free ofﬂine RL agents, and introduce an uncertainty-based sample selection method to mitigate the problem of model noises.

Original: Code is available at:https://anonymous.4open.science/r/RIQL-BE73/ .
Revised: Code is available at:https://anonymous.4open.science/r/RIQL-BE73/ .

Original:  Introduction
Revised:  Introduction

Original: Ofﬂine Reinforcement Learning (RL), also known as batch RL, refers to the problem of learningeffective control policies from a ﬁxed ofﬂine dataset [39].
Revised: Ofﬂine Reinforcement Learning (RL), also known as batch RL, refers to the problem of learningeffective control policies from a ﬁxed ofﬂine dataset [42].

Original: Due to the wide availability of loggeddata and the increasing computing power, ofﬂine RL holds the promise for successful real-worldapplications [40].
Revised: Due to the wide availability of loggeddata and the increasing computing power, ofﬂine RL holds the promise for successful real-worldapplications [43].

Original: For example, ofﬂine RL suits well for scenarios where collecting online data istime-consuming, dangerous or unethical, i e. , robotics, self-driving cars and medical treatments [21].
Revised: For example, ofﬂine RL suits well for scenarios where collecting online data istime-consuming, dangerous or unethical, i e. , robotics, self-driving cars and medical treatments [22].

Original: While most off-policy RL algorithms are applicable in the ofﬂine setting, they usually suffer fromthe extrapolation error [18, 35, 11] due to out-of-distribution (OOD) samples.
Revised: While most off-policy RL algorithms are applicable in the ofﬂine setting, they usually suffer fromthe extrapolation error [18, 37, 11] due to out-of-distribution (OOD) samples.

Original: Different solutionshave been proposed to mitigate this problem, i.e. , adding constraints [18, 50], behavior cloning (BC)[7, 57], learning a dynamics model [55, 28, 3], incorporating uncertainties [51], using ensembles [1],or learning pessimistic value functions [36, 5, 26].
Revised: Different solutionshave been proposed to mitigate this problem, i.e. , adding constraints [18, 54], behavior cloning BC) [7, 62], learning a dynamics model [60, 29, 3], incorporating uncertainties [56], using ensembles [1],or learning pessimistic value functions [38, 5, 27].

Original: Recently, ofﬂine RL algorithms have shown to be effective to solve various challenging tasks [46, 42].
Revised: Recently, ofﬂine RL algorithms have shown to be effective to solve various challenging tasks [50, 46].

Original: However, most of these works mainly focus on designing new algorithms and less attention hasbeen paid to understanding the behaviors of the learned ofﬂine RL agents.
Revised: However, most of these works mainly focus on designing new algorithms and less attention has been paid to understanding the behaviors of the learned ofﬂine RL agents.

Original: As a result, some basicquestions are still poorly understood.
Revised: As a result, some basicquestions are still poorly understood.

Original: For example:
Revised: For example:

Original: • Why does one ofﬂine RL agent perform better than other baseline agents in a benchmark task,
Revised: • Why does one ofﬂine RL agent perform better than other baseline agents in a benchmark task,

Original: as illustrated in Fig 1?
Revised: as illustrated in Fig 1?

Original: • Does the more performant agent learn better representations or more accurate value functions?
Revised: • Does the more performant agent learn better representations or more accurate value functions?

Original: • Which of the existing ofﬂine policy evaluation/improvement methods is more effective?
Revised: • Which of the existing ofﬂine policy evaluation/improvement methods is more effective?

Original: • When is a learned dynamics model  helpful to a model-free ofﬂine RL agent?
Revised: • When does a learned dynamics model is helpful to a model-free ofﬂine RL agent?

Original: • Given a learned dynamics model, how can we use the generated samples more efﬁciently?Baseline
Revised: • Given a learned dynamics model, how can we use the generated samples more efﬁciently?

Original: Paradigm Constraint Type Speed Disadvantage
Revised: [Table]

Original: TD3+BC [16]
Revised: TD3BC [16]

Original: Policy-regularized
Revised: Policy-regularized

Original: Actor
Revised: Actor

Original: Fail in difﬁcult tasks
Revised: Fail in difﬁcult tasks

Original: CQL [36]
Revised: CQL [38]

Original: Pessimism
Revised: Pessimism

Original: Critic
Revised: Critic

Original: Low computation efﬁciency
Revised: Low computation efﬁciency

Original: COMBO [54]
Revised: COMBO [59]

Original: Model-based
Revised: Model-based

Original: Critic
Revised: Critic

Original: Sensitive to model noises
Revised: Sensitive to model noises

Original: IQL [30]
Revised: IQL [31]

Original: Generalized BC
Revised: Generalized BC

Original: Both
Revised: Both

Original: Need to tune parameter ⌧
Revised: Need to tune parameter ⌧

Original: In spite of several potential intuitions, there is a lackof clear explanations for these questions.
Revised: In spite of several potential intuitions, there is a lackof clear explanations for these questions.

Original: Such incomplete understanding impedes progress, as researchersfail to identify the full impact of algorithmic changeswithout extensive tuning [12].
Revised: Such incomplete understanding impedes progress, as researchersfail to identify the full impact of algorithmic changeswithout extensive tuning [12].

Original: Given the increasingnumber of newly proposed ofﬂine RL algorithms,we argue that it is time to benchmark current SOTAofﬂine RL algorithms [15] and shed some lights onthese fundamental questions.
Revised: Given the increasingnumber of newly proposed ofﬂine RL algorithms,we argue that it is time to benchmark current SOTAofﬂine RL algorithms [15] and shed some lights onthese fundamental questions.

Original: To this end, we empirically compare four SOTA ofﬂine RL baseline algorithms, as shown in Table 1, from different paradigmson the standard D4RL dataset [14].
Revised: To this end, we empirically compare four SOTA ofﬂine RL baseline algorithms, as shown in Table 1, from different paradigmson the standard D4RL dataset [14].

Original: In this work, we design a comprehensive set of experiments to compare the representations, valuefunctions and policies of ofﬂine RL agents in order to gain a deeper understanding of the behaviors of
Revised: In this work, we design a comprehensive set of experiments to compare the representations, valuefunctions and policies of ofﬂine RL agents in order to gain a deeper understanding of the behaviors of

Original: SOTA ofﬂine RL agents.
Revised: SOTA ofﬂine RL agents.

Original: We identify a surprising discovery that a more performant agent sometimeshas worse representations and inaccurate value functions.
Revised: We identify a surprising discovery that a more performant agent sometimeshas worse representations and inaccurate value functions.

Original: This motivates us to take a closer lookat the learned policies.
Revised: This motivates us to take a closer lookat the learned policies.

Original: Our empirical results show that a performant ofﬂine RL agent is usuallyable to select better sub-optimal actions while avoiding bad ones.
Revised: Our empirical results show that a performant ofﬂine RL agent is usuallyable to select better sub-optimal actions while avoiding bad ones.

Original: Furthermore, we demonstratethat the proposed experiment setups can be used to evaluate the effectiveness of existing policyevaluation/improvement methods.
Revised: Furthermore, we demonstratethat the proposed experiment setups can be used to evaluate the effectiveness of existing policyevaluation/improvement methods.

Original: As a case study, we introduce a variant of IQL [30] by relaxing thein-sample constraint for the policy improvement step, which achieves better performance.
Revised: As a case study, we introduce a variant of IQL [31] by relaxing thein-sample constraint for the policy improvement step, which achieves better performance.

Original: Moreover,we investigate when a learned dynamics model can help a model-free ofﬂine RL agent, and wepropose an uncertainty-based sample selection method to mitigate the problem of model noises.
Revised: Moreover,we investigate when a learned dynamics model can help a model-free ofﬂine RL agent, and wepropose an uncertainty-based sample selection method to mitigate the problem of model noises.

Original: Ourcontributions are as follows:
Revised: Ourcontributions are as follows:

Original: • We conduct extensive experiments, focusing on representations, value functions and policies,to compare different SOTA ofﬂine RL agents and explain some fundamental questions.•
Revised: • We conduct extensive experiments, focusing on representations, value functions and policies,to compare different SOTA ofﬂine RL agents and explain some fundamental questions.•

Original: We show the effectiveness of the proposed experiment setups in diagnosing the bottleneck ofofﬂine RL agents with a new variant of the IQL algorithm that achieves SOTA performance.•
Revised: We show the effectiveness of the proposed experiment setups in diagnosing the bottleneck ofofﬂine RL agents with a new variant of the IQL algorithm that achieves SOTA performance.•

Original: We investigate when a learned dynamics model helps model-free ofﬂine RL agents, and weintroduce an uncertainty-based sample selection method that is more robust to model noises.
Revised: We investigate when a learned dynamics model helps model-free ofﬂine RL agents, and weintroduce an uncertainty-based sample selection method that is more robust to model noises.

Original:  Background
Revised:  Background

Original: A Markov Decision Process (MDP) M = h S , A , R , P ,  i [44] is speciﬁed by a state space S , an 68 action space
Revised: A Markov Decision Process (MDP) M = h S , A , R , P ,  i [48] is speciﬁed by a state space S , an 68 action space

Original: A , a transition kernel P : S ⇥ A !
Revised: A , a transition kernel P : S ⇥ A !

Original:  ( S ) , a reward function R : S ⇥ A !
Revised:  ( S ) , a reward function R : S ⇥ A !

Original: R , 69 and a discount factor  2 [0 , 1) .
Revised: R , 69 and a discount factor  2 [0 , 1) .

Original: The goal is to ﬁnd a policy ⇡ ( a | s ) : S !
Revised: The goal is to ﬁnd a policy ⇡ ( a | s ) : S !

Original:  ( A ) , which maps 70 from state to distribution over actions, that maximizes the expected cumulative discounted reward
Revised:  ( A ) , which maps 70 from state to distribution over actions, that maximizes the expected cumulative discounted reward

Original: J ( ⇡ ) := E ⇡ [
Revised: J ( ⇡ ) := E ⇡ [

Original: P 1 t =

Revised: P 1 t =

Original:  t r t ] .
Revised:  t r t ] .

Original: The performance of the policy can be deﬁned by the value functions
Revised: The performance of the policy can be deﬁned by the value functions

Original: Q ⇡ ( s, a ) :=
Revised: Q ⇡ ( s, a ) :=

Original: E ⇡ [
Revised: E ⇡ [

Original: P 1 t =

Revised: P 1 t =

Original:  t r t | s 0 =
Revised:  t r t | s 0 =

Original: s, a 0 = a ] and V ⇡ ( s ) :
Revised: s, a 0 = a ] and V ⇡ ( s ) :

Original: =
Revised: =

Original: E ⇡ [
Revised: E ⇡ [

Original: P 1 t =

Revised: P 1 t =

Original:  t r t | s 0 =
Revised:  t r t | s 0 =

Original: s ] , where E ⇡ [ · ] is 73 the expected result when following the policy ⇡ .
Revised: s ] , where E ⇡ [ · ] is 73 the expected result when following the policy ⇡ .

Original: In deep RL, Q -function is parameterized with a neural network Q ⇡ ✓ ( s, a ) .
Revised: In deep RL, Q -function is parameterized with a neural network Q ⇡ ✓ ( s, a ) .

Original: Following prior works[20, 34, 32, 38], we denote the penultimate layer of the neural network as the learned representation , ad -dimensional mapping  ( s, a ) : S ⇥ A !
Revised: Following prior works[21, 35, 33, 41], we denote the penultimate layer of the neural network as the learned representation , ad -dimensional mapping  ( s, a ) : S ⇥ A !

Original: R d .
Revised: R d .

Original: Such that the Q -function is linear in the representation
Revised: Such that the Q -function is linear in the representation

Original: Q ⇡✓ ( s, a ) =
Revised: Q ⇡✓ ( s, a ) =

Original: ✓ >  ( s, a ) , where ✓ 2 R d is a vector of weights.
Revised: ✓ >  ( s, a ) , where ✓ 2 R d is a vector of weights.

Original: If the state space S and action space Aare ﬁnite the representation corresponds to a feature matrix  2 R |S|·|A| ⇥ d  whose rows are thevector  ( s i , a i ) 2 R d for the i -th state-action pair ( s i  a i ) 
Revised: If the state space S and action space Aare ﬁnite, then the representation corresponds to a feature matrix  2 R |S|·|A| ⇥ d , whose rows are thevector  ( s i , a i ) 2 R d for the i -th state-action pair ( s i , a i ) .

Original: s a  ( s , a )  c  c
Revised: s a  ( s , a )  c

Original:   ( s , a )  p
Revised:   c  ( s , a )  p

Original: Q  ( s , a )  y ( s , a )   p  ( s , a ) probing target value function critic network
Revised: Q  ( s , a )  y ( s , a )   p  ( s , a ) probing target value function critic network

Original: [Figure]
Revised: [Figure]

Original: Target
Revised: Target

Original: Probing Function
Revised: Probing Function

Original: [Table]
Revised: [Table]

Original: reward r r = f p (  ( s, a ))
Revised: reward r r = f p (  ( s, a ))

Original: next state ss 0 = f p (  ( s, a )) inverse action a a = f p ( ( s ) , ( s 0 ))
Revised: next state ss 0 = f p (  ( s, a )) inverse action a a = f p ( ( s ) , ( s 0 ))

Original: Optimal Policy optimal action a ⇤ a ⇤ =
Revised: Optimal Policy optimal action a ⇤ a ⇤ =

Original: f p ( ( s ))
Revised: f p ( ( s ))

Original: optimal Q ⇤ ( s, a )
Revised: optimal Q ⇤ ( s, a )

Original: Q ⇤ ( s, a ) =
Revised: Q ⇤ ( s, a ) =

Original: f p (  ( s, a )) optimal V ⇤ ( s )
Revised: f p (  ( s, a )) optimal V ⇤ ( s )

Original: V ⇤ ( s ) =
Revised: V ⇤ ( s ) =

Original: f p ( ( s ))
Revised: f p ( ( s ))

Original: In this work, we study the ofﬂine RL setting [40], where we aim to learn a policy ⇡ ( a | s ) purelyfrom a ﬁxed ofﬂine dataset D = { ( s i , a i , s 

Revised: In this work, we study the ofﬂine RL setting [44], where we aim to learn a policy ⇡ ( a | s ) purelyfrom a ﬁxed ofﬂine dataset D = { ( s i , a i , s

Original: i , r i ) } , generated from a behavior policy ⇡  ( a | s ) .
Revised: i , r i ) } , generated from a behavior policy ⇡  ( a | s ) .

Original: Most ofrecent ofﬂine RL algorithms build on the Approximate Dynamics Programming (ADP) method [4]that learn the Q ⇡✓ ( s, a ) by minimizing temporal difference error:
Revised: Most ofrecent ofﬂine RL algorithms build on the Approximate Dynamics Programming (ADP) method [4]that learn the Q ⇡✓ ( s, a ) by minimizing temporal difference error:

Original: [Equation]
Revised: [Equation]

Original: where Q ⇡ ˆ ✓ ( s, a ) is the target network.
Revised: where Q ⇡ ˆ ✓ ( s, a ) is the target network.

Original: A major challenge in ofﬂine RL is the issue of distributionalshift between the learned policy ⇡ ( a | s ) and the behavior policy ⇡  ( a | s ) .
Revised: A major challenge in ofﬂine RL is the issue of distributionalshift between the learned policy ⇡ ( a | s ) and the behavior policy ⇡  ( a | s ) .

Original: Speciﬁcally, the OODactions a 0 can produce erroneously over-estimated target values for Q ⇡
Revised: Speciﬁcally, the OODactions a 0 can produce erroneously over-estimated target values for Q ⇡

Original: ˆ ✓ ( s 0 , a 0 ) in Eq 1.
Revised: ˆ ✓ ( s 0 , a 0 ) in Eq 1.

Original: Therefore,many existing ofﬂine RL algorithms are motivated to constrain the learned policy to stay close tothe behavior policy [50, 16], or penalize large over-estimated Q -values [36, 11].
Revised: Therefore,many existing ofﬂine RL algorithms are motivated to constrain the learned policy to stay close tothe behavior policy [54, 16], or penalize large over-estimated Q -values [38, 11].

Original: We provide moreextensive backgrounds in Appendix A.
Revised: We provide moreextensive backgrounds in Appendix A.

Original:  Representation evaluation experiments
Revised:  Representation evaluation experiments

Original: Inspired by the huge success achieved by representation learning in (un)supervised learning [6, 25, 45],it is natural to ask – does a more performant ofﬂine RL agent learn better representations?
Revised: Inspired by the huge success achieved by representation learning in (un)supervised learning [6, 26, 49],it is natural to ask – does a more performant ofﬂine RL agent learn better representations?

Original: To answerthis question, we ﬁrst leverage the representation probing technique [2, 23, 37] to evaluate the learnedrepresentations of each baseline agent.
Revised: To answerthis question, we ﬁrst leverage the representation probing technique [2, 24, 40] to evaluate the learnedrepresentations of each baseline agent.

Original: Then we use some latest introduced representation metrics[34, 33, 38, 41] as proxies to evaluate the quality of the learned representations.
Revised: Then we use some latest introduced representation metrics[35, 34, 41, 45] as proxies to evaluate the quality of the learned representations.

Original: 3.1 Representation probing experiment
Revised: 3.1 Representation probing experiment

Original: For a state-action pair ( s, a ) , we use the output of the penultimate layer of the critic network as thecritic representation  ( s, a ) 2 R d , as shown in Fig 2.
Revised: For a state-action pair ( s, a ) , we use the output of the penultimate layer of the critic network as thecritic representation  ( s, a ) 2 R d , as shown in Fig 2.

Original: We later train another linear model f p ( · ) topredict a probing target, such as reward r , by linear regression ˆ y ( s, a ) =
Revised: We later train another linear model f p ( · ) topredict a probing target, such as reward r , by linear regression ˆ y ( s, a ) =

Original: f p (  ( s, a ))
Revised: f p (  ( s, a ))

Original: =
Revised: =

Original: ✓ > p  ( s, a ) .
Revised: ✓ > p  ( s, a ) .

Original: We evaluate the actor embedding ( s ) 2 R d in a similar way.
Revised: We evaluate the actor embedding ( s ) 2 R d in a similar way.

Original: To validate whether the learnedrepresentations contain meaningful inductive biases [48, 53], we select two categories of probingtargets (Table 2).
Revised: To validate whether the learnedrepresentations contain meaningful inductive biases [52, 58], we select two categories of probingtargets (Table 2).

Original: We ﬁrst use the next state, reward, and action to detect if the representations learnedany semantics about the transition dynamics .
Revised: We ﬁrst use the next state, reward, and action to detect if the representations learnedany semantics about the transition dynamics .

Original: We then directly use the optimal action and optimalvalue functions to check if the representations learned any information about the optimal policy .
Revised: We then directly use the optimal action and optimalvalue functions to check if the representations learned any information about the optimal policy .

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: In the experiment, an online TD3 agent [17] is used to approximate the optimal policy ⇡ ⇤ .
Revised: In the experiment, an online TD3 agent [17] is used to approximate the optimal policy ⇡ ⇤ .

Original: We useﬁve checkpoints of the online agent (with different levels of performance) to collect 100K transitionsas the probing dataset D probe .
Revised: We useﬁve checkpoints of the online agent (with different levels of performance) to collect 100K transitionsas the probing dataset D probe .

Original: For each probing target, we use a 5-fold cross-validation on D probeto train a linear regression model with Mean Squared Error (MSE) loss.
Revised: For each probing target, we use a 5-fold cross-validation on D probeto train a linear regression model with Mean Squared Error (MSE) loss.

Original: The result of the probingexperiment on halfcheetah-medium-v2 environment is shown in Fig 3.
Revised: The result of the probingexperiment on halfcheetah-medium-v2 environment is shown in Fig 3.

Original: Results on other environmentsare summarized in Appendix C.1.
Revised: Results on other environmentsare summarized in Appendix C.1.

Original: We also list the number of environments where the most performantofﬂine RL agent has the least probing loss in Table 3.
Revised: We also list the number of environments where the most performantofﬂine RL agent has the least probing loss in Table 3.

Original: We can observe that the most performant agent usually has worse probing results except for theoptimal action experiment.
Revised: We can observe that the most performant agent usually has worse probing results except for theoptimal action experiment.

Original: These results indicate that the transition dynamics-based information isnot that important for an ofﬂine RL agent to perform well in the selected benchmark tasks.
Revised: These results indicate that the transition dynamics-based information isnot that important for an ofﬂine RL agent to perform well in the selected benchmark tasks.

Original: Further,many baseline agents suffer from large MSE losses on the optimal value functions Q ⇤ ( s, a ) and
Revised: Further,many baseline agents suffer from large MSE losses on the optimal value functions Q ⇤ ( s, a ) and

Original: V ⇤ ( s ) experiments, which highlights the difﬁculty to learn accurate value functions in such ofﬂinesetting due to the limited data coverage and additional policy/value constraints.
Revised: V ⇤ ( s ) experiments, which highlights the difﬁculty to learn accurate value functions in such ofﬂinesetting due to the limited data coverage and additional policy/value constraints.

Original: In addition, as long asthe actor representation ( s ) preserves the ability to learn good actions (low optimal action probingloss), then the ofﬂine RL agent holds the potential to achieve good performance.
Revised: In addition, as long asthe actor representation ( s ) preserves the ability to learn good actions (low optimal action probingloss), then the ofﬂine RL agent holds the potential to achieve good performance.

Original: 3.2 Representation metric experiment
Revised: 3.2 Representation metric experiment

Original: We further utilize some recently proposed metrics [34, 33] as proxies to evaluate the quality of thelearned representations.
Revised: We further utilize some recently proposed metrics [35, 34] as proxies to evaluate the quality of thelearned representations.

Original: We start with the following deﬁnitions and more details are in Appendix A.2.
Revised: We start with the following deﬁnitions and more details are in Appendix A.2.

Original: Deﬁnition 1 (Feature dot-product) .
Revised: Deﬁnition 1 (Feature dot-product) .

Original: Feature dot-product  ( s, a ) >  ( s 0 , a 0 ) twocritic representations[34], s 0{+
+} ⇠ P ( ·| s, a ) and a 0 ⇠ ⇡ ( ·| s 0 )
Revised: Feature dot-product  ( s, a ) >  ( s 0 , a 0 ) two representations s {+
+} ⇠ P ( ·| s, a ) and a 0 ⇠ ⇡ ( ·| s 0 )

Original: Deﬁnition 2 (Effective rank) .
Revised: Deﬁnition 2 (Effective rank) .

Original: Effective rank srank  (  ) =
Revised: Effective rank srank  (  ) =

Original: min  k : P ki =1  i (  )
Revised: min  k : P ki =1  i (  )

Original: P di =1  i (  )  1   of afeature matrix  2 R |S|·|A| ⇥
Revised: P di =1  i (  )  1   of afeature matrix  2 R |S|·|A| ⇥

Original: d approximates the rank of  [33], where {  i (  ) } are the singular 127 values of  in the decreasing order (  · · ·   d  0 ) and  is a threshold parameter, i.e., 0 . 01 .
Revised: d approximates the rank of  [34], where {  i (  ) } are the singular 127 values of  in the decreasing order (  · · ·   d  0 ) and  is a threshold parameter, i.e., 0 . 01 .

Original: Deﬁnition 3 (Effective dimension) .
Revised: Deﬁnition 3 (Effective dimension) .

Original: Effective dimension d e ↵ (  ) =
Revised: Effective dimension d e ↵ (  ) =

Original: N max i =1 , ··· ,N k P
Revised: N max i =1 , ··· ,N k P

Original:  e i
Revised:  e i

Original: k 22 ofa feature matrix  2 R |S|·|A| ⇥
Revised: k 22 ofa feature matrix  2 R |S|·|A| ⇥

Original: d measures the sparsity of the column space of  [38], where
Revised: d measures the sparsity of the column space of  [41], where

Original: N = |S| · |A| and P  is the orthogonal projector onto the column space of  .
Revised: N = |S| · |A| and P  is the orthogonal projector onto the column space of  .

Original: For the feature dot-product metric, we also compute the cosine similarity  ( s,a ) >  ( s 0 ,a 0 ) k
Revised: For the feature dot-product metric, we also compute the cosine similarity  ( s,a ) >  ( s 0 ,a 0 ) k

Original:  ( s,a ) kk  ( s 0 ,a 0 ) k to decouple 132 the effect the representation norm.
Revised:  ( s,a ) kk  ( s 0 ,a 0 ) k to decouple 132 the effect the representation norm.

Original: To compute the effective rank , we ﬁrst compute the critic representation z i =
Revised: To compute the effective rank , we ﬁrst compute the critic representation z i =

Original:  ( s i , a i ) for each sample in the probing dataset D probe .
Revised:  ( s i , a i ) for each sample in the probing dataset D probe .

Original: Then we approximatethe covariance matrix of  by C (  ) =
Revised: Then we approximatethe covariance matrix of  by C (  ) =

Original: |D probe |
Revised: |D probe |

Original: P i ( z i  ¯ z )( z i  ¯ z ) , and use SVD on C (  ) to computethe singular values {  i (  ) } [27].
Revised: P i ( z i  ¯ z )( z i  ¯ z ) , and use SVD on C (  ) to computethe singular values {  i (  ) } [28].

Original: Unlike the original implementation of the effective dimension whichused a ﬁxed threshold to approximate the rank of  , we instead use the effective rank srank  (  ) .
Revised: Unlike the original implementation of the effective dimension whichused a ﬁxed threshold to approximate the rank of  , we instead use the effective rank srank  (  ) .

Original: More details are discussed in Appendix C.2.
Revised: More details are discussed in Appendix C.

Original: Fig 4 shows the experiment results on the halfcheetah-medium-v2 environment.
Revised: Fig 4 shows the experiment results on the halfcheetah-medium-v2 environment.

Original: Results on otherenvironments are summarized in Appendix C.2.
Revised: Results on otherenvironments are summarized in Appendix C.2.

Original: We also add the result for the online TD3 agentfor reference.
Revised: We also add the result for the online TD3 agentfor reference.

Original: We can observe that the most performant COMBO agent has most severe featureco-adaptation [34] (large feature dot-product) and representation collapse [33] (small effective rank)problems, which indicate it learned pretty “low quality” critic representations.
Revised: We can observe that the most performant COMBO agent has most severe featureco-adaptation [35] (large feature dot-product) and representation collapse [34] (small effective rank)problems, which indicate it learned pretty “low quality” critic representations.

Original: In terms of thesemetrics alone, some baselines agents such as TD3+BC and IQL seem to learn similar or better criticrepresentations than the near-optimal online agent.
Revised: In terms of thesemetrics alone, some baselines agents such as TD3BC and IQL seem to learn similar or better criticrepresentations than the near-optimal online agent.

Original: These results are consistent with our ﬁndings inthe previous probing experiments, which indicate that a performant ofﬂine RL agent sometimes doesnot necessarily need high quality critic representations.
Revised: These results are consistent with our ﬁndings inthe previous probing experiments, which indicate that a performant ofﬂine RL agent sometimes doesnot necessarily need high quality critic representations.

Original: Observation 1.
Revised: Observation 1.

Original: A performant ofﬂine RL agent sometimes learn low-quality representations.
Revised: A performant ofﬂine RL agent sometimes learn low-quality representations.

Original:  Value ranking experiments
Revised:  Value ranking experiments

Original: As described in the last section, a performant ofﬂine RL agent can learn low-quality representations.
Revised: As described in the last section, a performant ofﬂine RL agent can learn low-quality representations.

Original: We therefore turn our focus to the value functions to investigate that – does a performant ofﬂine RLagent learn more accurate Q -functions?
Revised: We therefore turn our focus to the value functions to investigate that – does a performant ofﬂine RLagent learn more accurate Q -functions?

Original: Since the pessimism-based ofﬂine RL algorithms adopt anadditional penalty to learn lower-bounded Q -functions [36, 54], it could be inappropriate to measurethe accuracy of Q -functions using metrics like MSE.
Revised: Since the pessimism-based ofﬂine RL algorithms adopt anadditional penalty to learn lower-bounded Q -functions [38, 59], it could be inappropriate to measurethe accuracy of Q -functions using metrics like MSE.

Original: Therefore, we design the following valueranking experiments that focus on the ability of Q -functions to rank actions.
Revised: Therefore, we design the following valueranking experiments that focus on the ability of Q -functions to rank actions.

Original: The motivation is simple– a “good” Q -function in ofﬂine RL can have large MSE loss, but it should be good at ranking actions,such that Q ⇡ ( s, a i ) >
Revised: The motivation is simple– a “good” Q -function in ofﬂine RL can have large MSE loss, but it should be good at ranking actions,such that Q ⇡ ( s, a i ) >

Original: Q ⇡ ( s, a j ) if Q ⇤ ( s, a i ) >
Revised: Q ⇡ ( s, a j ) if Q ⇤ ( s, a i ) >

Original: Q ⇤ ( s, a j ) where Q ⇤ is the optimal Q -function.
Revised: Q ⇤ ( s, a j ) where Q ⇤ is the optimal Q -function.

Original: In the experiment, we ﬁrst use different behavior policies (four baseline agents and a near-optimalonline agent) to interact with the environment to collect a validation set D value =
Revised: In the experiment, we ﬁrst use different behavior policies (four baseline agents and a near-optimalonline agent) to interact with the environment to collect a validation set D value =

Original: { s 1 , · · · , s N } .
Revised: { s 1 , · · · , s N } .

Original: At each state s i , we use each baseline agent to sample m different actions, i.e. , ⇡ ( ·| s i ) + ✏ where✏ is a Gaussian noise.
Revised: At each state s i , we use each baseline agent to sample m different actions, i.e. , ⇡ ( ·| s i ) + ✏ where✏ is a Gaussian noise.

Original: Therefore, we have M = 4 m different actions
Revised: Therefore, we have M = 4 m different actions

Original: A i = { a i 1 , · · · , a iM } 161 for each state.
Revised: A i = { a i 1 , · · · , a iM } 161 for each state.

Original: Then we use baseline agents to rank the M actions at state s i , i.e. , R ⇡ j ( s i ) =
Revised: Then we use baseline agents to rank the M action at state s i , i.e. , R ⇡ j ( s i ) =

Original: sort ( Q ⇡ j ( s i , a i 1 ) , · · · , Q ⇡ j ( s i , a iM )) where Q ⇡ j is the learned Q -function of the j -th baseline agent.
Revised: sort ( Q ⇡ j ( s i , a i 1 ) , · · · , Q ⇡ j ( s i , a iM )) where Q ⇡ j is the learned Q -function of the j -th baseline agent.

Original: We use two metrics to evaluate the accuracy of the learned Q -functions: (1) Spearman’s rank correlation coefﬁcient (Rank IC) and (2) TopN accuracy.
Revised: We use two metrics to evaluate the accuracy of the learned Q -functions: (1) Spearman’s rank correlation coefﬁcient (Rank IC) and (2) TopN accuracy.

Original: We approximate the optimal rank informationusing the online agent.
Revised: We approximate the optimal rank informationusing the online agent.

Original: Here, the larger rank IC/topN accuracy is, the more accurate is the Q ⇡ ( s, a )at ranking actions.
Revised: Here, the larger rank IC/topN accuracy is, the more accurate is the Q ⇡ ( s, a )at ranking actions.

Original: More details are described in Appendix D.
Revised: More details are described in Appendix D.

Original: Experiment results are shown in Table 4 and Table 5.
Revised: Experiment results are shown in Table 4 and Table 5.

Original: The most performant baseline agents in eachenvironment are colored brown.
Revised: The most performant baseline agents in eachenvironment are colored brown.

Original: We can observe that many best-performing agents have near-zerorank IC and lower TopN accuracy results.
Revised: We can observe that many best-performing agents have near-zerorank IC and lower TopN accuracy results.

Original: This indicates that the learned Q -functions are less capableof distinguishing good actions from the bad ones.
Revised: This indicates that the learned Q -functions are less capableof distinguishing good actions from the bad ones.

Original: Interestingly, such inaccurate Q -functions donot prevent the most performant agent to select sub-optimal actions to achieve good performances.
Revised: Interestingly, such inaccurate Q -functions donot prevent the most performant agent to select sub-optimal actions to achieve good performances.

Original: Further, we can observe that the TD3+BC agent and IQL agent usually learn more accurate value
Revised: Further, we can observe that the TD3BC agent and IQL agent usually learn more accurate value

Original: [Table] [Table]
Revised: [Table] [Table]

Original: functions but achieves worse performance.
Revised: functions but achieves worse performance.

Original: This suggests that the policy evaluation method in TD3+BC and IQL might be more effective but the policy improvement method is limited.
Revised: This suggests that the policy evaluation method in TD3BCand IQL might be more effective but the policy improvement method is limited.

Original: We providemore discussions in section 6.
Revised: We provide morediscussions in section 6.

Original: Observation 2.
Revised: Observation 2.

Original: A more performant ofﬂine RL sometimes learn less accurate Q -functions.
Revised: A more performant ofﬂine RL sometimes learn less accurate Q -functions.

Original:  What does a performant policy look like?
Revised:  What does a performant policy look like?

Original: In previous sections, we show that a performant ofﬂine RL agent sometimes learn relative low-qualityrepresentations and inaccurate value functions.
Revised: In previous sections, we show that a performant ofﬂine RL agent sometimes learn relative low-qualityrepresentations and inaccurate value functions.

Original: In this section, we try to directly compare the learnedpolicies in order to get a glimpse of “what does a performant ofﬂine RL policy look like”.
Revised: In this section, we try to directly compare the learnedpolicies in order to get a glimpse of “what does a performant ofﬂine RL policy look like”.

Original: To thisend, we ﬁrst design a policy ranking experiment to measure how well does a performant ofﬂine RLpolicy work in practice.
Revised: To thisend, we ﬁrst design a policy ranking experiment to measure how well dose a performant ofﬂine RLpolicy work in practice.

Original: Then we check how often do the SOTA ofﬂine RL agents take OOD actions.
Revised: Then we check how often do the SOTA ofﬂine RL agents take OOD actions.

Original: 5.1 Policy ranking experiment
Revised: 5.1 Policy ranking experiment

Original: In this experiment, we ﬁrst use a behavior policy to collect 30K transitions to create a test dataset
Revised: In this experiment, we ﬁrst use a behavior policy to collect 30K transitions to create a test dataset

Original: D policy .
Revised: D policy .

Original: At each state s i , we use four baseline agents and a behavior cloning agent to select anaction, respectively.
Revised: At each state s i , we use four baseline agents and a behavior cloning agent to select anaction, respectively.

Original: Again, we use the online agent to approximate Q ⇤ ( s, a ) to rank the selectedfour actions.
Revised: Again, we use the online agent to approximate Q ⇤ ( s, a ) to rank the selectedfour actions.

Original: We use following two metrics to evaluate the policy ⇡ j of the j -th baseline agent: (1)
Revised: We use following two metrics to evaluate the policy ⇡ j of the j -th baseline agent: (1)

Original: Average percentage of policy ⇡ j that ranks the ﬁrst/last (with largest/smallest Q ⇤ value) across states.(2) Average MSE of the selected action w r. t
Revised: Average percentage of policy ⇡ j that ranks the ﬁrst/last (with largest/smallest Q ⇤ value) across states.(2) Average MSE of the selected action w.r.t.

Original: the optimal action
Revised: the optimal action.

Original: More details are in Appendix E.

Revised: More details are in Appendix E.

Original: Experiment results are shown in Table 6 and Table 7.
Revised: Experiment results are shown in Table 6 and Table 7.

Original: We can observe that the learned policy of
Revised: We can observe that the learned policy of

Original: COMBO [54] agent is quite extreme , which both selects the most optimal and worst actions at thesame time.
Revised: COMBO [59] agent is quite extreme , which both selects the most optimal and worst actions at thesame time.

Original: In simple environments such as halfcheetah and hopper , such behavior helps to attainhigher performance.
Revised: In simple environments such as halfcheetah and hopper , such behavior helps to attainhigher performance.

Original: However, in the more complex walker environment, the higher percentage ofbad actions would lead to early termination and thus achieving lower scores.
Revised: However, in the more complex walker environment, the higher percentage ofbad actions would lead to early termination and thus achieving lower scores.

Original: Moreover, as we canobserve in Table 7 that the most performant ofﬂine RL agent usually have similar or larger MSE lossw.r.t.
Revised: Moreover, as we canobserve in Table 7 that the most performant ofﬂine RL agent usually have similar or larger MSE lossw.r.t.

Original: the optimal action a ⇤ .
Revised: the optimal action a ⇤ .

Original: This indicates that a performant policy is good at selecting better actionsin different states, even though these sampled actions are still sub-optimal (with high MSE).[Table] action ( P w ). Results are averaged over 5 seeds
Revised: This indicates that a performant policy is good at selecting better actionsin different states, even though these sampled actions are still sub-optimal (with high MSE).

Original: [Table]

Original: Observation 3.
Revised: Observation 3.

Original: A performant ofﬂine RL policy needs to strike a balance at selecting good actions while avoiding bad ones.
Revised: A performant ofﬂine RL policy needs to strike a balance at selecting good actions while avoiding bad ones.

Original: Even though the selected actions are usually still sub-optimal.
Revised: Even though the selected actions are usually still sub-optimal.

Revised: [Table]

Revised: ( P w ).

Revised: [Table]

Original: 5.2 OOD action experiment
Revised: 5.2 OOD action experiment

Original: Since existing ofﬂine RL algorithms usually attempt to minimize the negative effects caused by OODsamples.
Revised: Since existing ofﬂine RL algorithms usually attempt to minimize the negative effects caused by OODsamples.

Original: Therefore, we try to measure how often would the baseline agents take OOD actions.
Revised: Therefore, we try to measure how often would the baseline agents take OOD actions.

Original: Inthe experiment, we utilize a learned dynamics model to predict whether a state-action pair ( s, a ) is
Revised: Inthe experiment, we utilize a learned dynamics model to predict whether a state-action pair ( s, a ) is

Original: OOD or not.
Revised: OOD or not.

Original: The learned dynamics models is a probabilistic ensemble [8] as in COMBO [54], whereeach model T i ( s 0 | s, a ) =
Revised: The learned dynamics models is a probabilistic ensemble [8] as in COMBO [59], whereeach model T i ( s 0 | s, a ) =

Original: N ( µ ✓ i ( s, a ) , ⌃ ✓ i ( s, a )) outputs a Gaussian distribution with diagonalcovariance parameterized by ✓ i .
Revised: N ( µ ✓ i ( s, a ) , ⌃ ✓ i ( s, a )) outputs a Gaussian distribution with diagonalcovariance parameterized by ✓ i .

Original: We train the probabilistic ensemble using the D4RL dataset.
Revised: We train the probabilistic ensemble using the D4RL dataset.

Original: We usethe following metric to estimate how likely a state-action pair ( s, a ) is OOD: the uncertainty [55]estimated by the probabilistic ensemble  ( s, a ) = max i =1 , ··· ,N k µ i ✓ ( s, a )  1 N
Revised: We usethe following metric to estimate how likely a state-action pair ( s, a ) is OOD: the uncertainty [60]estimated by the probabilistic ensemble  ( s, a ) = max i =1 , ··· ,N k µ i ✓ ( s, a )  1 N

Original: P
Revised: P

Revised: [Equation]

Revised: =

Revised: [Equation]

Revised: µ

Original: Nj =1 µ j ✓ ( s, a ) k 2 .
Revised: [Equation] ✓ [Equation] s, a ) k 2 .

Original: In particular, we ﬁrst compute  ( s, a ) on the original D4RL ofﬂine dataset, and then compute the ( s, a ) on a dataset collected by the baseline agent.
Revised: In particular, we ﬁrst compute  ( s, a ) on the original D4RL ofﬂine dataset, and then compute the ( s, a ) on a dataset collected by the baseline agent.

Original: More details are described in Appendix E.2.
Revised: More details are described in Appendix E.2.

Original: We report the median of the estimated uncertainty in Table 8.
Revised: We report the median of the estimated uncertainty in Table 8.

Original: As we can observe that the threemodel-free ofﬂine RL baselines usually learn conservative policies which mostly take high conﬁdenceactions with low  ( s, a ) .
Revised: As we can observe that the threemodel-free ofﬂine RL baselines usually learn conservative policies which mostly take high conﬁdenceactions with low  ( s, a ) .

Original: On the other hand, the model-based baseline COMBO learns policies thatsometimes take more risky actions with higher  ( s, a ) .
Revised: On the other hand, the model-based baseline COMBO learns policies thatsometimes take more risky actions with higher  ( s, a ) .

Original: This conﬁrms that taking risky actions inofﬂine RL is a double-edged sword, which sometimes helps to learn from the OOD samples to avoidbeing too conservative and sometimes incurs the extrapolation error.
Revised: This conﬁrms that taking risky actions inofﬂine RL is a double-edged sword, which sometimes helps to learn from the OOD samples to avoidbeing too conservative and sometimes incurs the extrapolation error.

Original: Observation 4.
Revised: Observation 4.

Original: Taking OOD actions in ofﬂine RL could be a double-edged sword that sometimes helps to avoid being over-conservative but sometimes incurs the extrapolation error.
Revised: Taking OOD actions in ofﬂine RL could be a double-edged sword that sometimes helps to avoid being over-conservative but sometimes incurs the extrapolation error

Original: In this section, we conduct a case study to show that the proposed experiments can be used tocompare the effectiveness of existing policy evaluation/improvement methods. We ﬁrst brieﬂy recapthe four baseline algorithms as summarized in Table 9. Here we consider a parameterized Q -function
Revised: [Table]

Original: Q ⇡✓ ( s, a ) , policy ⇡  ( a | s ) and loss weight parameters ↵ ,  that we abuse in different baselines.

Original: [Table]
Revised: [Table]

Revised: Baseline

Original: [Table]
Revised: [Table]

Revised: function

Revised: Actor loss functions

Original:  L TD ( D , ✓ )  L TD 3 ( D ,  ) + C BC ( D ,  ) CQL [36]
Revised: TD3BC [16] L TD ( D , ✓ )  L TD 3 ( D ,  ) + C BC ( D ,  ) CQL [38]

Original: L TD ( D , ✓ ) + ↵ C CQL ( D , ✓ )
Revised: L TD ( D , ✓ ) + ↵ C CQL ( D , ✓ )

Original: L SAC ( D , ✓ ,  ) COMBO [54]
Revised: L SAC ( D , ✓ ,  ) COMBO [59]

Original: L TD ( D c M , ✓ ) + ↵ C CQL ( D c M , ✓ )
Revised: L TD ( D c M , ✓ ) + ↵ C CQL ( D c M , ✓ )

Original: L SAC ( D c M ,  ) IQL [30]
Revised: L SAC ( D c M ,  ) IQL [31]

Original: L ER ( D , ✓ )
Revised: L ER ( D , ✓ )

Original: L AWR ( D ,  )
Revised: L AWR ( D ,  )

Revised:  Case study: relaxed in-sample Q-learning (RIQL)

Revised: In this section, we conduct a case study to show that the proposed experiments can be used tocompare the effectiveness of existing policy evaluation/improvement methods.

Revised: We ﬁrst brieﬂy recapthe four baseline algorithms as summarized in Table 9.

Revised: Here we consider a parameterized Q -function

Revised: Q ⇡✓ ( s, a ) , policy ⇡  ( a | s ) and loss weight parameters ↵ ,  that we abuse in different baselines.

Original: Policy evaluation .
Revised: Policy evaluation .

Original: TD3+BC [16] uses the default ﬁtted Q -evaluation as in Eq 1.
Revised: TD3BC [16] uses the default ﬁtted Q -evaluation as in Eq 1.

Original: The only differenceis that the max operator over next action a 0 is replaced by the policy ⇡  ( s 0 ) :
Revised: The only differenceis that the max operator over next action a 0 is replaced by the policy ⇡  ( s 0 ) :

Original: [Equation]
Revised: [Equation]

Original: CQL [36] further adds a conservative penalty to push down large Q -values for OOD actions:
Revised: CQL [38] further adds a conservative penalty to push down large Q -values for OOD actions:

Original: [Equation]
Revised: [Equation]

Original: COMBO [54] extends CQL by using an augmented dataset D c
Revised: COMBO [59] extends CQL by using an augmented dataset D c

Original: M [Equation]
Revised: M = D

Original: D [ D model for policy evaluation, 226 where
Revised: [ D model for policy evaluation, 226 where

Original: D model is generated using the learned dynamics model.
Revised: D model is generated using the learned dynamics model.

Original: Unlike other three baselines, IQL [30]uses expectile regression (ER)  for policy evaluation purely from in-sample data:
Revised: Unlike other three baselines, IQL [31]uses expectile regression (ER) to for policy evaluation purely from in-sample data:

Original: [Equation]
Revised: [Equation]

Original: where L ⌧ 2 ( u ) = | ⌧  ( u < 0)
Revised: where L ⌧ 2 ( u ) = | ⌧  ( u < 0)

Original: | u 2 and ⌧ 2 (0 , 1) is a hyperparameter.
Revised: | u 2 and ⌧ 2 (0 , 1) is a hyperparameter.

Original: Policy improvement .
Revised: Policy improvement .

Original: TD3+BC adds a simple behavior cloning loss C BC (  ) =
Revised: TD3BC adds a simple behavior cloning loss C BC (  ) =

Original: E [( ⇡ ( s )  a ) 2 ] tothe original TD3 [17] actor loss L TD 3 (  ) =
Revised: E [( ⇡ ( s )  a ) 2 ] tothe original TD3 [17] actor loss L TD 3 (  ) =

Original:  E s ⇠ D [ Q ⇡✓ ( s, ⇡  ( s ))] .
Revised:  E s ⇠ D [ Q ⇡✓ ( s, ⇡  ( s ))] .

Original: CQL and COMBO both usethe default SAC [22] actor loss function L SAC (  ) =
Revised: CQL and COMBO both usethe default SAC [23] actor loss function L SAC (  ) =

Original: E ( s,a ) ⇠ D [  log( ⇡  ( a | s ))
Revised: E ( s,a ) ⇠ D [  log( ⇡  ( a | s ))

Original:  Q ⇡✓ ( s, ⇡  ( a | s ))] .
Revised:  Q ⇡✓ ( s, ⇡  ( a | s ))] .

Original: On the other hand, IQL uses the advantage-weighted regression (AWR)[43] to imitate high-qualityin-sample actions L AWR (  ) = E [Equation] s,
Revised: On the other hand, IQL uses the advantage-weighted regression (AWR)

Original: [Equation]

Original: D [Equation] [Equation] ) log ⇡  
Revised: [47] to imitate high-qualityin-sample actions L AWR (  ) =

Original: [Equation]
Revised: [Equation]

Original: a [Equation] s ) ⇤ , where V ( s ) isa value function that only depends on state s .
Revised: ⇤ , where V ( s ) isa value function that only depends on state s .

Original: Given these different policy evaluation/improvement methods, there is a lack of clear comparison toshow which one is more effective in practice.
Revised: Given these different policy evaluation/improvement methods, there is a lack of clear comparison toshow which one is more effective in practice.

Original: Here, we try to show that the previously introducedexperiment setups can be helpful tools to answer such questions.
Revised: Here, we try to show that the previously introducedexperiment setups can be helpful tools to answer such questions.

Original: Recall the results in the Table 4 and
Revised: Recall the results in the Table 4 and

Original: Table 5, we can observe that IQL [30] usually achieves higher rank IC and larger topN accuracy, whichsuggests that the expectile regression based policy evaluation method in IQL is more capable to learnaccurate value functions.
Revised: Table 5, we can observe that IQL [31] usually achieves higher rank IC and larger topN accuracy, whichsuggests that the expectile regression based policy evaluation method in IQL is more capable to learnaccurate value functions.

Original: However, IQL only achieves the best performance in one benchmark task.
Revised: However, IQL only achieves the best performance in one benchmark task.

Original: In addition, Table 8 indicates that IQL learns conservative policy that usually takes high conﬁdencein-sample data.
Revised: In addition, Table 8 indicates that IQL learns conservative policy that usually takes high conﬁdencein-sample data.

Original: These results suggest that the AWR-based policy-improvement method is effective toavoid taking OOD actions but sometimes it is over-conservative which limits the performance.
Revised: These results suggest that the AWR-based policy-improvement method is effective toavoid taking OOD actions but sometimes it is over-conservative which limits the performance.

Original: Tovalidate this assumption, we introduce a simple variant of IQL, called Relaxed In-sample Q-Learning(RIQL), that replaces the original AWR-based actor loss with:[Equation] a ) ⇠ D [log ⇡  ( a
Revised: Tovalidate this assumption, we introduce a simple variant of IQL, called Relaxed In-sample Q-Learning(RIQL), that replaces the original AWR-based actor loss with:

Original: [Equation]
Revised: [Equation]

Original: In short, we add a KL-divergence constraint to the original SAC actor loss, and we drop the
Revised: In short, we add a KL-divergence constraint to the original SAC actor loss, and we drop the

Original: E ( s,a ) ⇠ D [log ⇡  ( a | s )] term which is independent of actor parameter  .
Revised: E ( s,a ) ⇠ D [log ⇡  ( a | s )] term which is independent of actor parameter  .

Original: The motivation is two- 248 fold: (1) Unlike CQL and COMBO, the learned Q -functions in IQL are less discriminating w.r.t.[Table]
Revised: The motivation is two- 248 fold: (1) Unlike CQL and COMBO, the learned Q -functions in IQL are less discriminating w.r.t.

Original: OOD samples.
Revised: OOD samples.

Original: Thus, we need extra policy constraints to avoid taking too many OOD actions.
Revised: Thus, we need extra policy constraints to avoid taking too many OOD actions.

Original: Wewant to use a less conservative actor loss to enable the policy learning from OOD actions.
Revised: Wewant to use a less conservative actor loss to enable the policy learning from OOD actions.

Original: In addition,
Revised: In addition,

Original: Eq 5 is similar to the actor loss in [11].
Revised: Eq 5 is similar to the actor loss in [11].

Original: The difference is that we keep the entropy term which helpsto prevent the learned policy from collapsing to a single point.
Revised: The difference is that we keep the entropy term which helpsto prevent the learned policy from collapsing to a single point.

Original: Experiment results of RIQL is shownin Table 10 and more details are described in Appendix
Revised: Experiment results of RIQL is shownin Table 10 and more details are described in Appendix

Original: F.
Revised: F.

Original: We can observe that RIQL outperforms
Revised: We can observe that RIQL outperforms

Original: IQL in all benchmark tasks, and it achieves higher total evaluation scores than other baseline agents.
Revised: IQL in all benchmark tasks, and it achieves higher total evaluation scores than other baseline agents.

Original: This result also validates our previous assumption that AWR-based policy improvement method isthe bottleneck for IQL in some tasks which makes it over-conservative. 
Revised: This result also validates our previous assumption that AWR-based policy improvement method isthe bottleneck for IQL in some tasks which makes it over-conservative. [Table]

Original:  Uncertainty-based sample selection for model-based ofﬂine RL
Revised:  Uncertainty-based sample selection for model-based ofﬂine RL

Original: In this section, we try to investigate when does a learned dynamics model is helpful to a model-freeofﬂine RL agent?
Revised: In this section, we try to investigate when does a learned dynamics model is helpful to a model-freeofﬂine RL agent?

Original: In particular, we reuse the learned dynamics model in Section 5.2 to generate fakesamples to train a model-free agent.
Revised: In particular, we reuse the learned dynamics model in Section 5.2 to generate fakesamples to train a model-free agent.

Original: For example, we introduce an MBRL variant of IQL/TD3+BC,named MIQL/MTD3+BC, which learns from the augmented dataset D c
Revised: For example, we introduce an MBRL variant of IQL/TD3BC,named MIQL/MTD3BC, which learns from the augmented dataset D c

Original: M = D
Revised: M = D

Original: [ D model as in  262 MOPO [55].
Revised: [ D model as in COMBO 262 [59].

Original: However, we ﬁnd that such a naive MBRL agent(a simple combination of MOPOand model-free ofﬂine RL agents) usually performs worse than its model-free counterpart (Table11) even though they are the same algorithm except for the training data.
Revised: However, we ﬁnd that such a naive MBRL agent usually performs worse than its model-freecounterpart (Table 11) even though they are the same algorithm except for the training data.

Original: This result indicates thatit is nontrivial to combine a learned dynamics model with model-free ofﬂine RL agents due tothe model noises.
Revised: Thisresult indicates that it is nontrivial to combine a learned dynamics model with model-free ofﬂine RL agents due to the model noises.

Original: COMBO [54] addresses this problem by adding a conservative penalty on themodel-generated samples.
Revised: COMBO [59] addresses this problem by adding a conservativepenalty on the model-generated samples.

Original: Here, we introduce an Uncertainty-based Sample Selection (USS) methodto ﬁlter fake samples with large model noises.
Revised: Here, we introduce an Uncertainty-based Sample Selection(USS) method to ﬁlter fake samples with large model noises.

Original: Since it is hard to measure model noises accurately, we adopt the model uncertainty  ( s, a ) deﬁnedin Section 5.2 as a proxy to approximate model noise.
Revised: Since it is hard to measure model noises accurately, we adopt the model uncertainty  ( s, a ) deﬁnedin Section 5.2 as a proxy to approximate model noise.

Original: In addition, we maintain a dynamic uncertaintythreshold   as follows   ⌘ qbatch ( s, a ) + (1  ⌘ )   , where  qbatch ( s, a ) is the q -th quantile ofthe uncertainties in the sampled fake trajectories.
Revised: In addition, we maintain a dynamic uncertaintythreshold   as follows   ⌘ qbatch ( s, a ) + (1  ⌘ )   , where  qbatch ( s, a ) is the q -th quantile ofthe uncertainties in the sampled fake trajectories.

Original: We only add model-generated samples with anuncertainty lower than   to the model buffer which is later used to train the agent.
Revised: We only add model-generated samples with anuncertainty lower than   to the model buffer which is later used to train the agent.

Original: More details aredescribed in Appendix G. From Table 11, we can observe that the proposed USS trick usually helps toimprove the performance of the MIQL/MTD3+BC agent.
Revised: More details aredescribed in Appendix G. From Table 11, we can observe that the proposed USS trick usually helpsto improve the performance of the MIQL/MTD3BC agent.

Original: Moreover, USS is inferior to IQL/TD3+BCin
Revised: Moreover, USS is inferior to IQL/

Original:  some tasks which shows that it is still challenging to completely solve the model noise problem. 
Revised: TD3BCin some tasks which shows that it is still challenging to completely solve the model noise problem. IQL

Original: [Table]
Revised: [Table]

Original:  Conclusion
Revised:  Conclusion

Original: Besides the rapid development of novel ofﬂine RL algorithms, the behaviors of these ofﬂine RL agentshave not been well studied.
Revised: Besides the rapid development of novel ofﬂine RL algorithms, the behaviors of these ofﬂine RL agentshave not been well studied.

Original: In this work, we take a closer look at SOTA ofﬂine RL agents.
Revised: In this work, we take a closer look at SOTA ofﬂine RL agents.

Original: Speciﬁcally,we introduce a series of experiments to compare the learned representations, value functions andpolicies of four baseline agents.
Revised: Speciﬁcally,we introduce a series of experiments to compare the learned representations, value functions andpolicies of four baseline agents.

Original: Surprisingly, we ﬁnd that a performant agent sometimes has relativelylow quality representations and inaccurate value functions.
Revised: Surprisingly, we ﬁnd that a performant agent sometimes has relativelylow quality representations and inaccurate value functions.

Original: We also show that the proposed experimentsetups can be used to compare the effectiveness of different policy evaluation/improvement methodsby introducing a relaxed version of IQL that achieves SOTA performance.
Revised: We also show that the proposed experimentsetups can be used to compare the effectiveness of different policy evaluation/improvement methodsby introducing a relaxed version of IQL that achieves SOTA performance.

Original: Lastly, we investigate whena learned dynamics model can help model-free ofﬂine RL agents, and we introduce an uncertaintybased sample selection method to mitigate the problem of model noises.
Revised: Lastly, we investigate whena learned dynamics model can help model-free ofﬂine RL agents, and we introduce an uncertaintybased sample selection method to mitigate the problem of model noises.