Original: A Closer Look at Offline RL Agents
Revised: A Closer Look at Ofﬂine RL Agents

Original: Abstract
Revised: Abstract

Original: Despite recent advances in the field of Offline Reinforcement Learning (RL), less attention has been paid to understanding the behaviors of learned RL agents.
Revised: Despite recent advances in the ﬁeld of Ofﬂine Reinforcement Learning (RL), less attention has been paid to understanding the behaviors of learned RL agents.

Original: As a result, there remain some gaps in our understandings, i.e. , why is one offline
Revised: As a result, there remain some gaps in our understandings, i.e. , why is one ofﬂine

Original: RL agent more performant than another?
Revised: RL agent more performant than another?

Original: In this work, we first introduce a set of experiments to evaluate offline RL agents, focusing on three fundamental aspects: representations, value functions and policies.
Revised: In this work, we ﬁrst introduce a set of experiments to evaluate ofﬂine RL agents, focusing on three fundamental aspects: representations, value functions and policies.

Original: Counterintuitively, we show that a more performant offline RL agent can learn relatively low-quality representations and inaccurate value functions.
Revised: Counterintuitively, we show that a more performant ofﬂine RL agent can learn relatively low-quality representations and inaccurate value functions.

Original: Furthermore, we demonstrate that the proposed experiment setups can be effectively used to diagnose the bottleneck of offline RL agents.
Revised: Furthermore, we demonstrate that the proposed experiment setups can be effectively used to diagnose the bottleneck of ofﬂine RL agents.

Original: Inspired by the evaluation results, a novel offline RL algorithm is proposed by a simple modification of IQL and achieves SOTA performance.
Revised: Inspired by the evaluation results, a novel ofﬂine RL algorithm is proposed by a simple modiﬁcation of IQL and achieves SOTA performance.

Original: Finally, we investigate when a learned dynamics model is helpful to model-free offline RL agents, and introduce an uncertainty-based sample selection method to mitigate the problem of model noises.
Revised: Finally, we investigate when a learned dynamics model is helpful to model-free ofﬂine RL agents, and introduce an uncertainty-based sample selection method to mitigate the problem of model noises.

Original: Code is available at: https://github.com/fuyw/RIQL .
Revised: Code is available at: https://github.com/fuyw/RIQL .

Original:  Introduction
Revised:  Introduction

Original: Offline Reinforcement Learning (RL), also known as batch RL, refers to the problem of learning effective control policies from a fixed offline dataset [39].
Revised: Ofﬂine Reinforcement Learning (RL), also known as batch RL, refers to the problem of learning effective control policies from a ﬁxed ofﬂine dataset [39].

Original: Due to the wide availability of loggeddata and the increasing computing power, offline RL holds the promise for successful real-world applications [40].
Revised: Due to the wide availability of loggeddata and the increasing computing power, ofﬂine RL holds the promise for successful real-world applications [40].

Original: For example, offline RL suits well for scenarios where collecting online data is time-consuming, dangerous or unethical, i e. , robotics, self-driving cars and medical treatments [21].
Revised: For example, ofﬂine RL suits well for scenarios where collecting online data is time-consuming, dangerous or unethical, i e. , robotics, self-driving cars and medical treatments [21].

Original: While most off-policy RL algorithms are applicable in the offline setting, they usually suffer from the extrapolation error [18, 35, 11] due to out-of-distribution (OOD) samples.
Revised: While most off-policy RL algorithms are applicable in the ofﬂine setting they usually suffer from the extrapolation error [18, 35, 11] due to out-of-distribution (OOD) samples.

Original: Different solutions have been proposed to mitigate this problem, i e. , adding constraints [18, 50], behavior cloning (BC) [7, 57], learning a dynamics model [55, 28, 3], incorporating uncertainties [51], using ensembles [1], or learning pessimistic value functions [36, 5, 26].
Revised: Different solutions have been proposed to mitigate this problem, i.e. , adding constraints [18, 50], behavior cloning (BC) [7, 57], learning a dynamics model [55, 28, 3], incorporating uncertainties [51], using ensembles [1], or learning pessimistic value functions [36, 5, 26].

Original: Recently, offline RL algorithms have shown to be effective to solve various challenging tasks [46, 42].
Revised: Recently, ofﬂine RL algorithms have shown to be effective to solve various challenging tasks [46, 42].

Original: However, most of these works mainly focus on designing new algorithms and less attention has been paid to understanding the behaviors of the learned offline RL agents.
Revised: However, most of these works mainly focus on designing new algorithms and less attention has been paid to understanding the behaviors of the learned ofﬂine RL agents.

Original: As a result, some basic questions are still poorly understood.
Revised: As a result, some basic questions are still poorly understood.

Original: For example:
Revised: For example:

Original: • Why does one offline RL agent perform better than other baseline agents in a benchmark task, as illustrated in Fig 1?
Revised: • Why does one ofﬂine RL agent perform better than other baseline agents in a benchmark task, as illustrated in Fig 1?

Original: • Does the more performant agent learn better representations or more accurate value functions?
Revised: • Does the more performant agent learn better representations or more accurate value functions?

Original: • Which of the existing offline policy evaluation/improvement methods is more effective?
Revised: • Which of the existing ofﬂine policy evaluation/improvement methods is more effective?

Original: • When is a learned dynamics model helpful to a model-free offline RL agent?
Revised: • When is a learned dynamics model helpful to a model-free ofﬂine RL agent?

Original: • Given a learned dynamics model, how can we use the generated samples more efficiently? Baseline Paradigm
Revised: • Given a learned dynamics model, how can we use the generated samples more efﬁciently?

Original: Constraint Type
Revised: 36th Conference on Neural Information Processing Systems (NeurIPS 2022).

Original: Speed
Revised: [Table]

Original: Disadvantage TD3+BC [16]
Revised: TD3+BC [16]

Original: Policy-regularized
Revised: Policy-regularized

Original: Actor✓
Revised: Actor

Original: Fail in difficult tasks
Revised: Fail in difﬁcult tasks

Original: CQL [36]
Revised: CQL [36]

Original: Pessimism
Revised: Pessimism

Original: Critic✗
Revised: Critic

Original: Low computation efficiency
Revised: Low computation efﬁciency

Original: COMBO [54]
Revised: COMBO [54]

Original: Model-based
Revised: Model-based

Original: Critic✗
Revised: Critic

Original: Sensitive to model noises
Revised: Sensitive to model noises

Original: IQL [30]
Revised: IQL [30]

Original: Generalized BC
Revised: Generalized BC

Original: Both✓
Revised: Both

Original: Need to tune parameterτ
Revised: Need to tune parameter

Revised: [Table]

Original: In spite of several potential intuitions, there is a lack of clear explanations for these questions.
Revised: In spite of several potential intuitions, there is a lack of clear explanations for these questions.

Original: Such incomplete understanding impedes progress, as researchers fail to identify the full impact of algorithmic changes without extensive tuning [12].
Revised: Such incomplete understanding impedes progress, as researchers fail to identify the full impact of algorithmic changes without extensive tuning [12].

Original: Given the increasing number of newly proposed offline RL algorithms, we argue that it is time to benchmark current SOTA offline RL algorithms [15] and shed some lights on these fundamental questions.
Revised: Given the increasing number of newly proposed ofﬂine RL algorithms, we argue that it is time to benchmark current SOTA ofﬂine RL algorithms [15] and shed some lights on these fundamental questions.

Original: To this end, we empirically compare four SOTA offline RL baseline algorithms, as shown in Table 1, from different paradigms on the standard D4RL dataset [14].
Revised: To this end, we empirically compare four SOTA ofﬂine RL baseline algorithms, as shown in Table 1, from different paradigms on the standard D4RL dataset [14].

Original: In this work, we design a comprehensive set of experiments to compare the representations, value functions and policies of offline RL agents in order to gain a deeper understanding of the behaviors of
Revised: In this work, we design a comprehensive set of experiments to compare the representations, value functions and policies of ofﬂine RL agents in order to gain a deeper understanding of the behaviors of

Original: SOTA offline RL agents.
Revised: SOTA ofﬂine RL agents.

Original: We identify a surprising discovery that a more performant agent sometimes has worse representations and inaccurate value functions.
Revised: We identify a surprising discovery that a more performant agent sometimes has worse representations and inaccurate value functions.

Original: This motivates us to take a closer look at the learned policies.
Revised: This motivates us to take a closer look at the learned policies.

Original: Our empirical results show that a performant offline RL agent is usually able to select better sub-optimal actions while avoiding bad ones.
Revised: Our empirical results show that a performant ofﬂine RL agent is usually able to select better sub-optimal actions while avoiding bad ones.

Original: Furthermore, we demonstrate that the proposed experiment setups can be used to evaluate the effectiveness of existing policy evaluation/improvement methods.
Revised: Furthermore, we demonstrate that the proposed experiment setups can be used to evaluate the effectiveness of existing policy evaluation/improvement methods.

Original: As a case study, we introduce a variant of IQL [30] by relaxing the in-sample constraint for the policy improvement step, which achieves better performance.
Revised: As a case study, we introduce a variant of IQL [30] by relaxing the in-sample constraint for the policy improvement step, which achieves better performance.

Original: Moreover, we investigate when a learned dynamics model can help a model-free offline RL agent, and we propose an uncertainty-based sample selection method to mitigate the problem of model noises.
Revised: Moreover, we investigate when a learned dynamics model can help a model-free ofﬂine RL agent, and we propose an uncertainty-based sample selection method to mitigate the problem of model noises.

Original: Our contributions are as follows:
Revised: Our contributions are as follows:

Original: • We conduct extensive experiments, focusing on representations, value functions and policies, to compare different SOTA offline RL agents and explain some fundamental questions.
Revised: • We conduct extensive experiments, focusing on representations, value functions and policies, to compare different SOTA ofﬂine RL agents and explain some fundamental questions.

Original: • We show the effectiveness of the proposed experiment setups in diagnosing the bottleneck of offline RL agents with a new variant of the IQL algorithm that achieves SOTA performance.
Revised: • We show the effectiveness of the proposed experiment setups in diagnosing the bottleneck of ofﬂine RL agents with a new variant of the IQL algorithm that achieves SOTA performance.

Original: • We investigate when a learned dynamics model helps model-free offline RL agents, and we introduce an uncertainty-based sample selection method that is more robust to model noises.
Revised: • We investigate when a learned dynamics model helps model-free ofﬂine RL agents, and we introduce an uncertainty-based sample selection method that is more robust to model noises.

Original:  Background
Revised:  Background

Original: A Markov Decision Process (MDP) M = ⟨S , A , R , P , γ ⟩ [44] is specified by a state space S , an action spaceA , a transition kernel P : S ×
Revised: A Markov Decision Process (MDP) M = h S , A , R , P ,  i [44] is speciﬁed by a state space S , an action space

Revised: A , a transition kernel P : S ⇥ A !

Original: A → ∆( S ) , a reward function R : S ×
Revised:  ( S ) , a reward function R : S ⇥ A !

Original: A → R , and a discount factor γ ∈ [0 , 1) .
Revised: R , and a discount factor  2 [0 , 1) .

Original: The goal is to find a policy π ( a | s ) : S → ∆( A ) , which maps from state to distribution over actions, that maximizes the expected cumulative discounted reward
Revised: The goal is to ﬁnd a policy ⇡ ( a | s ) : S !  ( A ) , which maps from state to distribution over actions, that maximizes the expected cumulative discounted reward

Original: J ( π ) : = E π [ (cid:80) ∞ t =0 γ{+=
 +} t r t ] .
Revised: J ( ⇡ ) := E ⇡ [ P 1 t {+=
 +} t r t ] .

Original: The performance of the policy can be defined by the value functions
Revised: The performance of the policy can be deﬁned by the value functions

Original: Q π ( s, a ) : =
Revised: Q ⇡ ( s, a ) :=

Original: E π [ (cid:80) ∞ t =

Revised: E ⇡ [ P 1 t =

Original: γ t r t | s [-
-] =
Revised:  t r t | s [-
-]0 =

Original: s, a 0 = a ] and V π ( s ) :
Revised: s, a 0 = a ] and V ⇡ ( s ) :

Original: =
Revised: =

Original: E π [ (cid:80) ∞ t =

Revised: E ⇡ [ P 1 t =

Original: γ t r t | s [-
-] =
Revised:  t r t | s [-
-]0 =

Original: s ] , where E π [ · ] is the expected result when following the policy π .
Revised: s ] , where E ⇡ [ · ] is the expected result when following the policy ⇡ .

Original: In deep RL, Q -function is parameterized with a neural network Q π θ ( s, a ) .
Revised: In deep RL, Q -function is parameterized with a neural network Q ⇡ ✓ ( s, a ) .

Original: Following prior works [20, 34, 32, 38], we denote the penultimate layer of the neural network as the learned representation , a d -dimensional mapping ϕ ( s, a ) : S×A → R d .
Revised: Following prior works [20, 34, 32, 38], we denote the penultimate layer of the neural network as the learned representation , a d -dimensional mapping  ( s, a ) : S ⇥ A ! R d .

Original: Such that the Q -function is linear in the representation
Revised: Such that the Q -function is linear in the representation

Original: Q πθ ( s, a ) =θ ⊤ ϕ ( s, a ) ,
Revised: Q ⇡✓ ( s, a ) =

Original:  where θ ∈ R d is a vector of weights.
Revised: ✓ >  ( s, a ) , where ✓ 2 R d is a vector of weights.

Original: If the state space S and action space A are finite, then the representation corresponds to a feature matrix Φ ∈ R |S|·|A|× d , whose rows are the vector ϕ ( s  s  a ϕ
Revised: If the state space S and action space A are ﬁnite, then the representation corresponds to a feature matrix  2 R | s a  ( s , a )  c   c

Original:  ( s , a ) 
Revised:  ( s , a )  p

Original: θ c θ ⊤ c ϕ ( s , a ) θ p Q π ( s , a ) ̂ y ( s , a ) θ ⊤ p ϕ ( s , a ) probing target value function critic network
Revised: Q  ( s , a )  y ( s , a )   p  ( s , a ) probing target value function critic network

Original: [Figure]
Revised: [Figure]

Revised: Target

Original: Probing Function
Revised: Probing Function

Original: [Table]
Revised: [Table]

Original: reward r r = f p ( ϕ ( s, a ))
Revised: reward r r = f p (  ( s, a ))

Original: next state s ′ s ′ = f p ( ϕ ( s, a )) inverse action a a = f p (ψ ( s ) ,ψ ( s ′ ))
Revised: next state ss 0 = f p (  ( s, a )) inverse action a a = f p ( ( s ) , ( s 0 ))

Original: Optimal Policy optimal action a ∗ a ∗ =
Revised: Optimal Policy optimal action a ⇤ a ⇤ =

Original: f p (ψ ( s ))
Revised: f p ( ( s ))

Original: optimal Q ∗ ( s, a )
Revised: optimal Q ⇤ ( s, a )

Original: Q ∗ ( s, a ) = f p ( ϕ ( s, a )) optimal V ∗ ( s )
Revised: Q ⇤ ( s, a ) = f p (  ( s, a )) optimal V ⇤ ( s )

Original: V ∗ ( s ) =
Revised: V ⇤ ( s ) =

Original: f p (ψ ( s ))
Revised: f p ( ( s ))

Original: In this work, we study the offline RL setting [40], where we aim to learn a policy π ( a | s ) purely from a fixed offline dataset D = { ( s i , a i , s ′{+
+}
Revised: In this work, we study the ofﬂine RL setting [40], where we aim to learn a policy ⇡ ( a | s ) purely from a ﬁxed ofﬂine dataset D = { ( s i , a i , s {+
+}

Original: i , r i ) } , generated from a behavior policy π β ( a | s ) .
Revised: i , r i ) } , generated from a behavior policy ⇡  ( a | s ) .

Original: Most of recent offline RL algorithms build on the Approximate Dynamics Programming (ADP) method [4] that learn the Q πθ ( s, a ) by minimizing the temporal difference error:
Revised: Most of recent ofﬂine RL algorithms build on the Approximate Dynamics Programming (ADP) method [4] that learn the Q ⇡✓ ( s, a ) by minimizing the temporal difference error:

Original: [Equation]
Revised: [Equation]

Original: (1) where Q π ˆ θ ( s, a ) is the target network.
Revised: where Q ⇡ ˆ ✓ ( s, a ) is the target network.

Original: A major challenge in offline RL is the issue of distributional shift between the learned policy π ( a | s ) and the behavior policy π β ( a | s ) .
Revised: A major challenge in ofﬂine RL is the issue of distributional shift between the learned policy ⇡ ( a | s ) and the behavior policy ⇡  ( a | s ) .

Original: Specifically, the OOD actions a ′ can produce erroneously over-estimated target values for Q π ˆ θ ( s ′ , a ′ ) in Eq 1.
Revised: Speciﬁcally, the OOD actions a 0 can produce erroneously over-estimated target values for Q ⇡ ˆ ✓ ( s 0 , a 0 ) in Eq 1.

Original: Therefore, many existing offline RL algorithms are motivated to constrain the learned policy to stay close to the behavior policy [50, 16], or penalize large over-estimated Q -values [36, 11].
Revised: Therefore, many existing ofﬂine RL algorithms are motivated to constrain the learned policy to stay close to the behavior policy [50, 16], or penalize large over-estimated Q -values [36, 11].

Original: We provide more extensive backgrounds in Appendix A.
Revised: We provide more extensive backgrounds in Appendix A.

Original:  Representation evaluation experiments
Revised:  Representation evaluation experiments

Original: Inspired by the huge success achieved by representation learning in (un)supervised learning [6, 25, 45], it is natural to ask – does a more performant offline RL agent learn better representations?
Revised: Inspired by the huge success achieved by representation learning in (un)supervised learning [6, 25, 45], it is natural to ask – does a more performant ofﬂine RL agent learn better representations?

Original: To answer this question, we first leverage the representation probing technique [2, 23, 37] to evaluate the learned representations of each baseline agent.
Revised: To answer this question, we ﬁrst leverage the representation probing technique [2, 23, 37] to evaluate the learned representations of each baseline agent.

Original: Then we use some latest introduced representation metrics [34, 33, 38, 41] as proxies to evaluate the quality of the learned representations.
Revised: Then we use some latest introduced representation metrics [34, 33, 38, 41] as proxies to evaluate the quality of the learned representations.

Original: 3.1 Representation probing experiment
Revised: 3.1 Representation probing experiment

Original: For a state-action pair ( s, a ) , we use the output of the penultimate layer of the critic network as the critic representation ϕ ( s, a ) ∈ R d , as shown in Fig 2.
Revised: For a state-action pair ( s, a ) , we use the output of the penultimate layer of the critic network as the critic representation  ( s, a ) 2 R d , as shown in Fig 2.

Original: We later train another linear model f p ( · ) to predict a probing target, such as reward r , by linear regression ˆ y ( s, a ) =
Revised: We later train another linear model f p ( · ) to predict a probing target, such as reward r , by linear regression ˆ y ( s, a ) =

Original: f p ( ϕ ( s, a ))
Revised: f p (  ( s, a ))

Original: =
Revised: =

Original: θ ⊤ p ϕ ( s, a ) .
Revised: ✓ > p  ( s, a ) .

Original: We evaluate the actor embeddingψ ( s ) ∈ R d in a similar way.
Revised: We evaluate the actor embedding ( s ) 2 R d in a similar way.

Original: To validate whether the learned representations contain meaningful inductive biases [48, 53], we select two categories of probing targets (Table 2).
Revised: To validate whether the learned representations contain meaningful inductive biases [48, 53], we select two categories of probing targets (Table 2).

Original: We first use the next state, reward, and action to detect if the representations learned any semantics about the transition dynamics .
Revised: We ﬁrst use the next state, reward, and action to detect if the representations learned any semantics about the transition dynamics .

Original: We then directly use the optimal action and optimal value functions to check if the representations learned any information about the optimal policy .
Revised: We then directly use the optimal action and optimal value functions to check if the representations learned any information about the optimal policy .

Original: [Table]
Revised: [Table]

Original: In the experiment, an online TD3 agent [17] is used to approximate the optimal policy π ∗ .
Revised: In the experiment, an online TD3 agent [17] is used to approximate the optimal policy ⇡ ⇤ .

Original: We use five checkpoints of the online agent (with different levels of performance) to collect 100K transitions as the probing dataset D probe .
Revised: We use ﬁve checkpoints of the online agent (with different levels of performance) to collect 100K transitions as the probing dataset D probe .

Original: For each probing target, we use a 5-fold cross-validation on D probe to train a linear regression model with Mean Squared Error (MSE) loss.
Revised: For each probing target, we use a 5-fold cross-validation on D probe to train a linear regression model with Mean Squared Error (MSE) loss.

Original: The result of the probing experiment on halfcheetah-medium-v2 environment is shown in Fig 3.
Revised: The result of the probing experiment on halfcheetah-medium-v2 environment is shown in Fig 3.

Original: Results on other environments are summarized in Appendix C.1.
Revised: Results on other environments are summarized in Appendix C.1.

Original: We also list the number of environments where the most performant offline RL agent has the least probing loss in Table 3.
Revised: We also list the number of environments where the most performant ofﬂine RL agent has the least probing loss in Table 3.

Original: We can observe that the most performant agent usually has worse probing results except for the optimal action experiment.
Revised: We can observe that the most performant agent usually has worse probing results except for the optimal action experiment.

Original: These results indicate that the transition dynamics-based information is not that important for an offline RL agent to perform well in the selected benchmark tasks.
Revised: These results indicate that the transition dynamics-based information is not that important for an ofﬂine RL agent to perform well in the selected benchmark tasks.

Original: Further, many baseline agents suffer from large MSE losses on the optimal value functions Q ∗ ( s, a ) and
Revised: Further, many baseline agents suffer from large MSE losses on the optimal value functions Q ⇤ ( s, a ) and

Original: V ∗ ( s ) experiments, which highlights the difficulty to learn accurate value functions in such offline setting due to the limited data coverage and additional policy/value constraints.
Revised: V ⇤ ( s ) experiments, which highlights the difﬁculty to learn accurate value functions in such ofﬂine setting due to the limited data coverage and additional policy/value constraints.

Original: In addition, as long as the actor representationψ ( s ) preserves the ability to learn good actions (low optimal action probing loss), then the offline RL agent holds the potential to achieve good performance.
Revised: In addition, as long as the actor representation ( s ) preserves the ability to learn good actions (low optimal action probing loss), then the ofﬂine RL agent holds the potential to achieve good performance.

Original: 3.2 Representation metric experiment
Revised: 3.2 Representation metric experiment

Original: We further utilize some recently proposed metrics [34, 33] as proxies to evaluate the quality of the learned representations.
Revised: We further utilize some recently proposed metrics [34, 33] as proxies to evaluate the quality of the learned representations.

Original: We start with the following definitions and more details are in Appendix A.2.
Revised: We start with the following deﬁnitions and more details are in Appendix A.2.

Original: Definition 1 (Feature dot-product) .
Revised: Deﬁnition 1 (Feature dot-product) .

Original: Feature dot-product ϕ ( s, a ) ⊤ ϕ ( s ′ , a ′ ) is the dot-product of twocritic representations [34], where s ′ ∼ P ( ·| s, a ) and a ′ ∼ π ( ·| s ′ ) is the next state and next action.
Revised: Feature dot-product  ( s, a ) >  ( s 0 , a 0 ) is the dot-product of two

Original: Definition 2 (Effective rank) .
Revised: Deﬁnition 2 (Effective rank) .

Original: Effective rank srank δ (Φ) =
Revised: Effective rank srank  (  ) =

Original: min (cid:8) k : (cid:80) ki =1 σ i (Φ) (cid:80) di
Revised: min  k : P ki =1  i (  )

Original:  =1 σ i (Φ) ≥ 1 − δ (cid:9) of a feature matrix Φ ∈ R |S|·|A|× d approximates the rank of Φ [33], where { σ i (Φ) } are the singular values of Φ in the decreasing order ( σ 1 ≥ · · · ≥ σ d ≥ 0 ) and δ is a threshold parameter, i.e., 0 . 
Revised: P di =1  i (  )  1   of a feature matrix  2 R |S|·|A| ⇥ d approximates the rank of  [33], where {  i (  ) } are the singular values of  in the decreasing order (  · · ·   d  0 ) and  is a threshold parameter, i.e., 0 . 01 .

Original: Definition 3 (Effective dimension) .
Revised: Deﬁnition 3 (Effective dimension) .

Original: Effective dimension d eff (Φ) =
Revised: Effective dimension d e ↵ (  ) =

Original: N max i =1 , ··· ,N ∥ PΦ e i ∥
Revised: N max i =1 , ··· ,N k P

Original:  22 of a feature matrix Φ ∈ R |S|·|A|× d measures the sparsity of the column space of Φ [38], where
Revised:  e i k 22 of a feature matrix  2 R |S|·|A| ⇥ d measures the sparsity of the column space of  [38], where

Original: N = |S| · |A| and P Φ is the orthogonal projector onto the column space of Φ .
Revised: N = |S| · |A| and P  is the orthogonal projector onto the column space of  .

Original: For the feature dot-product metric, we also compute the cosine similarity ϕ ( s,a ) ⊤ ϕ ( s ′ ,a ′ ) ∥ ϕ
Revised: For the feature dot-product metric, we also compute the cosine similarity  ( s,a ) >  ( s 0 ,a 0 ) k

Original:  ( s,a ) ∥∥ ϕ ( s ′ ,a ′ ) ∥ to decouple the effect the representation norm.
Revised:  ( s,a ) kk  ( s 0 ,a 0 ) k to decouple the effect the representation norm.

Original: To compute the effective rank , we first compute the critic representation z i =ϕ
Revised: To compute the effective rank , we ﬁrst compute the critic representation z i =

Original:  ( s i , a i ) for each sample in the probing dataset D probe .
Revised:  ( s i , a i ) for each sample in the probing dataset D probe .

Original: Then we approximate the covariance matrix of Φ by C (Φ)
Revised: Then we approximate the covariance matrix of  by C (  ) =

Original: = 1 |D probe |(cid:80) i
Revised: |D probe |

Original:  ( z i − ¯ z )( z i − ¯ z ) , and use SVD on C (Φ) to compute the singular values { σ i (Φ) } [27].
Revised: P i ( z i  ¯ z )( z i  ¯ z ) , and use SVD on C (  ) to compute the singular values {  i (  ) } [27].

Original: Unlike the original implementation of the effective dimension which used a fixed threshold to approximate the rank of Φ , we instead use the effective rank srank δ (Φ) .
Revised: Unlike the original implementation of the effective dimension which used a ﬁxed threshold to approximate the rank of  , we instead use the effective rank srank  (  ) .

Original: More details are discussed in Appendix C.2.
Revised: More details are discussed in Appendix C.2.

Original: Fig 4 shows the experiment results on the halfcheetah-medium-v2 environment.
Revised: Fig 4 shows the experiment results on the halfcheetah-medium-v2 environment.

Original: Results on other environments are summarized in Appendix C.2.
Revised: Results on other environments are summarized in Appendix C.2.

Original: We also add the result for the online TD3 agent for reference.
Revised: We also add the result for the online TD3 agent for reference.

Original: We can observe that the most performant COMBO agent has most severe feature co-adaptation [34] (large feature dot-product) and representation collapse [33] (small effective rank) problems, which indicate it learned pretty “low quality” critic representations.
Revised: We can observe that the most performant COMBO agent has most severe feature co-adaptation [34] (large feature dot-product) and representation collapse [33] (small effective rank) problems, which indicate it learned pretty “low quality” critic representations.

Original: In terms of these metrics alone, some baselines agents such as TD3+BC and IQL seem to learn similar or better critic representations than the near-optimal online agent.
Revised: In terms of these metrics alone, some baselines agents such as TD3+BC and IQL seem to learn similar or better critic representations than the near-optimal online agent.

Original: These results are consistent with our findings in the previous probing experiments, which indicate that a performant offline RL agent sometimes does not necessarily need high quality critic representations.
Revised: These results are consistent with our ﬁndings in the previous probing experiments, which indicate that a performant ofﬂine RL agent sometimes does not necessarily need high quality critic representations.

Original: Observation 1.
Revised: Observation 1.

Original: A performant offline RL agent sometimes learn low-quality representations.
Revised: A performant ofﬂine RL agent sometimes learn low-quality representations.

Original:  Value ranking experiments
Revised:  Value ranking experiments

Original: As described in the last section, a performant offline RL agent can learn low-quality representations.
Revised: As described in the last section, a performant ofﬂine RL agent can learn low-quality representations.

Original: We therefore turn our focus to the value functions to investigate that – does a performant offline RL agent learn more accurate Q -functions?
Revised: We therefore turn our focus to the value functions to investigate that – does a performant ofﬂine RL agent learn more accurate Q -functions?

Original: Since the pessimism-based offline RL algorithms adopt an additional penalty to learn lower-bounded Q -functions [36, 54], it could be inappropriate to measure the accuracy of Q -functions using metrics like MSE.
Revised: Since the pessimism-based ofﬂine RL algorithms adopt an additional penalty to learn lower-bounded Q -functions [36, 54], it could be inappropriate to measure the accuracy of Q -functions using metrics like MSE.

Original: Therefore, we design the following value ranking experiments that focus on the ability of Q -functions to rank actions.
Revised: Therefore, we design the following value ranking experiments that focus on the ability of Q -functions to rank actions.

Original: The motivation is simple – a “good” Q -function in offline RL can have large MSE loss, but it should be good at ranking actions, such that Q π ( s, a i ) >
Revised: The motivation is simple – a “good” Q -function in ofﬂine RL can have large MSE loss, but it should be good at ranking actions, such that Q ⇡ ( s, a i ) >

Original: Q π ( s, a j ) if Q ∗ ( s, a i ) > Q ∗ ( s, a j ) where Q ∗ is the optimal Q -function.
Revised: Q ⇡ ( s, a j ) if Q ⇤ ( s, a i ) > Q ⇤ ( s, a j ) where Q ⇤ is the optimal Q -function.

Original: In the experiment, we first use different behavior policies (four baseline agents and a near-optimal online agent) to interact with the environment to collect a validation set D value =
Revised: In the experiment, we ﬁrst use different behavior policies (four baseline agents and a near-optimal online agent) to interact with the environment to collect a validation set D value =

Original: { s 1 , · · · , s N } .
Revised: { s 1 , · · · , s N } .

Original: At each state s i , we use each baseline agent to sample m different actions, i.e. , π ( ·| s i ) + ϵ where ϵ is a Gaussian noise.
Revised: At each state s i , we use each baseline agent to sample m different actions, i.e. , ⇡ ( ·| s i ) + ✏ where ✏ is a Gaussian noise.

Original: Therefore, we have M = 4 m different actions
Revised: Therefore, we have M = 4 m different actions

Original: A i = { a i 1 , · · · , a iM } for each state.
Revised: A i = { a i 1 , · · · , a iM } for each state.

Original: Then we use baseline agents to rank the M actions at state s i , i.e. , R π j ( s i )
Revised: Then we use baseline agents to rank the M actions at state s i , i.e. , R ⇡ j ( s i )

Original: =
Revised: =

Original: sort ( Q π j ( s i , a i 1 ) , · · · , Q π j ( s i , a iM )) where Q π j is the learned Q -function of the j -th baseline agent.
Revised: sort ( Q ⇡ j ( s i , a i 1 ) , · · · , Q ⇡ j ( s i , a iM )) where Q ⇡ j is the learned Q -function of the j -th baseline agent.

Original: We use two metrics to evaluate the accuracy of the learned Q -functions: (1) Spearman’s rank correlation coefficient (Rank IC) and (2) TopN accuracy.
Revised: We use two metrics to evaluate the accuracy of the learned Q -functions: (1) Spearman’s rank correlation coefﬁcient (Rank IC) and (2) TopN accuracy.

Original: We approximate the optimal rank information using the online agent.
Revised: We approximate the optimal rank information using the online agent.

Original: Here, the larger rank IC/topN accuracy is, the more accurate is the Q π ( s, a ) at ranking actions.
Revised: Here, the larger rank IC/topN accuracy is, the more accurate is the Q ⇡ ( s, a ) at ranking actions.

Original: More details are described in Appendix D.
Revised: More details are described in Appendix D.

Original: Experiment results are shown in Table 4 and Table 5, where we report the mean result and standard deviation across 5 random seeds.
Revised: Experiment results are shown in Table 4 and Table 5, where we report the mean result and standard deviation across 5 random seeds.

Original: The most performant baseline agents in each environment are colored brown.
Revised: The most performant baseline agents in each environment are colored brown.

Original: We can observe that many best-performing agents have near-zero rank IC and lower
Revised: We can observe that many best-performing agents have near-zero rank IC and lower

Original: TopN accuracy results.
Revised: TopN accuracy results.

Original: This indicates that the learned Q -functions are less capable of distinguishing good actions from the bad ones.
Revised: This indicates that the learned Q -functions are less capable of distinguishing good actions from the bad ones.

Original: Interestingly, such inaccurate Q -functions do not prevent the most performant agent to select sub-optimal actions to achieve good performances.
Revised: Interestingly, such inaccurate Q -functions do not prevent the most performant agent to select sub-optimal actions to achieve good performances.

Original: Further, we can observe
Revised: Further, we can observe

Original: [Table] [Table]
Revised: [Table] [Table]

Original: that the TD3+BC agent and IQL agent usually learn more accurate value functions but achieves worse performance.
Revised: that the TD3+BC agent and IQL agent usually learn more accurate value functions but achieves worse performance.

Original: This suggests that the policy evaluation method in TD3+BC and IQL might be more effective but the policy improvement method is limited.
Revised: This suggests that the policy evaluation method in TD3+BC and IQL might be more effective but the policy improvement method is limited.

Original: We provide more discussions in section 6.
Revised: We provide more discussions in section 6.

Original: Observation 2.
Revised: Observation 2.

Original: A more performant offline RL sometimes learn less accurate Q -functions.
Revised: A more performant ofﬂine RL sometimes learn less accurate Q -functions.

Original:  What does a performant policy look like?
Revised:  What does a performant policy look like?

Original: In previous sections, we show that a performant offline RL agent sometimes learn relative low-quality representations and inaccurate value functions.
Revised: In previous sections, we show that a performant ofﬂine RL agent sometimes learn relative low-quality representations and inaccurate value functions.

Original: In this section, we try to directly compare the learned policies in order to get a glimpse of “what does a performant offline RL policy look like”.
Revised: In this section, we try to directly compare the learned policies in order to get a glimpse of “what does a performant ofﬂine RL policy look like”.

Original: To this end, we first design a policy ranking experiment to measure how well does a performant offline RL policy work in practice.
Revised: To this end, we ﬁrst design a policy ranking experiment to measure how well does a performant ofﬂine RL policy work in practice.

Original: Then we check how often do the SOTA offline RL agents take OOD actions.
Revised: Then we check how often do the SOTA ofﬂine RL agents take OOD actions.

Original: 5.1 Policy ranking experiment
Revised: 5.1 Policy ranking experiment

Original: In this experiment, we first use a behavior policy to collect 30K transitions to create a test dataset
Revised: In this experiment, we ﬁrst use a behavior policy to collect 30K transitions to create a test dataset

Original: D policy .
Revised: D policy .

Original: At each state s i , we use four baseline agents and a behavior cloning agent to select an action, respectively.
Revised: At each state s i , we use four baseline agents and a behavior cloning agent to select an action, respectively.

Original: Again, we use the online agent to approximate Q ∗ ( s, a ) to rank the selected four actions.
Revised: Again, we use the online agent to approximate Q ⇤ ( s, a ) to rank the selected four actions.

Original: We use following two metrics to evaluate the policy π j of the j -th baseline agent: (1)
Revised: We use following two metrics to evaluate the policy ⇡ j of the j -th baseline agent: (1)

Original: Average percentage of policy π j that ranks the first/last (with largest/smallest Q ∗ value) across states.
Revised: Average percentage of policy ⇡ j that ranks the ﬁrst/last (with largest/smallest Q ⇤ value) across states.

Original: (2) Average MSE of the selected action w.r.t.
Revised: (2) Average MSE of the selected action w. r.t.

Original: the optimal action.
Revised: the optimal action

Original: More details are in Appendix E.
Revised: More details are in Appendix E.

Original: Experiment results are shown in Table 6 and Table 7.
Revised: Experiment results are shown in Table 6 and Table 7.

Original: We can observe that the learned policy of
Revised: We can observe that the learned policy of

Original: COMBO [54] agent is quite extreme , which both selects the most optimal and worst actions at the same time.
Revised: COMBO [54] agent is quite extreme , which both selects the most optimal and worst actions at the same time.

Original: In simple environments such as halfcheetah and hopper , such behavior helps to attain higher performance.
Revised: In simple environments such as halfcheetah and hopper , such behavior helps to attain higher performance.

Original: However, in the more complex walker environment, the higher percentage of bad actions would lead to early termination and thus achieving lower scores.
Revised: However, in the more complex walker environment, the higher percentage of bad actions would lead to early termination and thus achieving lower scores.

Original: Moreover, as we can observe in Table 7 that the most performant offline RL agent usually have similar or larger MSE loss w.r.t.
Revised: Moreover, as we can observe in Table 7 that the most performant ofﬂine RL agent usually have similar or larger MSE loss w.r.t.

Original: the optimal action a ∗ .
Revised: the optimal action a ⇤ .

Original: This indicates that a performant policy is good at selecting better actions in different states, even though these sampled actions are still sub-optimal (with high MSE).
Revised: This indicates that a performant policy is good at selecting better actions in different states, even though these sampled actions are still sub-optimal (with high MSE).

Original: Observation 3.
Revised: Observation 3.

Original: A performant offline RL policy needs to strike a balance at selecting good actions while avoiding bad ones.
Revised: A performant ofﬂine RL policy needs to strike a balance at selecting good actions while avoiding bad ones.

Original: Even though the selected actions are usually still sub-optimal.[Table] P [Table]
Revised: Even though the selected actions are usually still sub-optimal.

Original: P o

Original: [Table]P o P w P o
Revised: [Table]

Original: P w P o [Table] action ( P w ).
Revised: action ( P w ).

Original: [Table]
Revised: [Table]

Original: 5.2 OOD action experiment
Revised: 5.2 OOD action experiment

Original: Since existing offline RL algorithms usually attempt to minimize the negative effects caused by OOD samples.
Revised: Since existing ofﬂine RL algorithms usually attempt to minimize the negative effects caused by OOD samples.

Original: Therefore, we try to measure how often would the baseline agents take OOD actions.
Revised: Therefore, we try to measure how often would the baseline agents take OOD actions.

Original: In the experiment, we utilize a learned dynamics model to predict whether a state-action pair ( s, a ) is
Revised: In the experiment, we utilize a learned dynamics model to predict whether a state-action pair ( s, a ) is

Original: OOD or not.
Revised: OOD or not.

Original: The learned dynamics models is a probabilistic ensemble [8] as in COMBO [54], where each model T i ( s ′ | s, a ) =
Revised: The learned dynamics models is a probabilistic ensemble [8] as in COMBO [54], where each model T i ( s 0 | s, a ) =

Original: N ( µ θ i ( s, a ) , Σ θ i ( s, a )) outputs a Gaussian distribution with diagonal covariance parameterized by θ i .
Revised: N ( µ ✓ i ( s, a ) , ⌃ ✓ i ( s, a )) outputs a Gaussian distribution with diagonal covariance parameterized by ✓ i .

Original: We train the probabilistic ensemble using the D4RL dataset.
Revised: We train the probabilistic ensemble using the D4RL dataset.

Original: We use the following metric to estimate how likely a state-action pair ( s, a ) is OOD: the uncertainty [55] estimated by the probabilistic ensemble σ ( s, a ) = max i =1 , ··· ,N ∥ µ iθ ( s, a ) − 1 N (cid:80) Nj =1 µ jθ ( s, a ) ∥ 2 .
Revised: We use the following metric to estimate how likely a state-action pair ( s, a ) is OOD: the uncertainty [55] estimated by the probabilistic ensemble  ( s, a ) = max i =1 , ··· ,N k µ i ✓ ( s, a )  1 N [Equation]

Original: In particular, we first compute σ ( s, a ) on the original D4RL offline dataset, and then compute the σ ( s, a ) on a dataset collected by the baseline agent.
Revised: In particular, we ﬁrst compute  ( s, a ) on the original D4RL ofﬂine dataset, and then compute the  ( s, a ) on a dataset collected by the baseline agent.

Original: More details are described in Appendix E.2.
Revised: More details are described in Appendix E.2.

Original: We report the median of the estimated uncertainty in Table 8.
Revised: We report the median of the estimated uncertainty in Table 8.

Original: As we can observe that the three model-free offline RL baselines usually learn conservative policies which mostly take high confidence actions with low σ ( s, a ) .
Revised: As we can observe that the three model-free ofﬂine RL baselines usually learn conservative policies which mostly take high conﬁdence actions with low  ( s, a ) .

Original: On the other hand, the model-based baseline COMBO learns policies that sometimes take more risky actions with higher σ ( s, a ) .
Revised: On the other hand, the model-based baseline COMBO learns policies that sometimes take more risky actions with higher  ( s, a ) .

Original: This confirms that taking risky actions in offline RL is a double-edged sword, which sometimes helps to learn from the OOD samples to avoid being too conservative and sometimes incurs the extrapolation error.
Revised: This conﬁrms that taking risky actions in ofﬂine RL is a double-edged sword, which sometimes helps to learn from the OOD samples to avoid being too conservative and sometimes incurs the extrapolation error.

Original: Observation 4.
Revised: Observation 4.

Original: Taking OOD actions in offline RL could be a double-edged sword that sometimes helps to avoid being over-conservative but sometimes incurs the extrapolation error.
Revised: Taking OOD actions in ofﬂine RL could be a double-edged sword that sometimes helps to avoid being over-conservative but sometimes incurs the extrapolation error.

Original: [Table]
Revised: [Table]

Original: Baseline
Revised: Baseline

Revised: [Table]

Original: Critic loss function Actor loss functions
Revised: function Actor loss functions

Original: TD3+BC [16]
Revised: TD3+BC [16]

Original: L TD ( D ,θ ) λL TD 3 ( D ,ϕ ) + C BC ( D ,ϕ ) CQL [36]
Revised: L TD ( D , ✓ )  L TD 3 ( D ,  ) + C BC ( D ,  ) CQL [36]

Original: L TD ( D ,θ ) + α C CQL ( D ,θ )
Revised: L TD ( D , ✓ ) + ↵ C CQL ( D , ✓ )

Original: L SAC ( D ,θ,ϕ ) COMBO [54]
Revised: L SAC ( D , ✓ ,  ) COMBO [54]

Original: L TD ( D (cid:99) M ,θ ) + α C CQL ( D (cid:99) M ,θ )
Revised: L TD ( D c M , ✓ ) + ↵ C CQL ( D c M , ✓ )

Original: L SAC ( D (cid:99) M ,ϕ ) IQL [30]
Revised: L SAC ( D c M ,  ) IQL [30]

Original: L ER ( D ,θ )
Revised: L ER ( D , ✓ )

Original: L AWR ( D ,ϕ )
Revised: L AWR ( D ,  )

Original:  Case study: relaxed in-sample Q-learning (RIQL)
Revised:  Case study: relaxed in-sample Q-learning (RIQL)

Original: In this section, we conduct a case study to show that the proposed experiments can be used to compare the effectiveness of existing policy evaluation/improvement methods.
Revised: In this section, we conduct a case study to show that the proposed experiments can be used to compare the effectiveness of existing policy evaluation/improvement methods.

Original: We first briefly recap the four baseline algorithms as summarized in Table 9.
Revised: We ﬁrst brieﬂy recap the four baseline algorithms as summarized in Table 9.

Original: Here we consider a parameterized Q -function
Revised: Here we consider a parameterized Q -function

Original: Q πθ ( s, a ) , policy π ϕ ( a | s ) and loss weight parameters α, λ that we abuse in different baselines.
Revised: Q ⇡✓ ( s, a ) , policy ⇡  ( a | s ) and loss weight parameters ↵ ,  that we abuse in different baselines.

Original: Policy evaluation .
Revised: Policy evaluation .

Original: TD3+BC [16] uses the default fitted Q -evaluation as in Eq 1.
Revised: TD3+BC [16] uses the default ﬁtted Q -evaluation as in Eq 1.

Original: The only difference is that the max operator over next action a ′ is replaced by the policy π ϕ ( s ′ ) :
Revised: The only difference is that the max operator over next action a 0 is replaced by the policy ⇡  ( s 0 ) :

Original: [Equation]
Revised: [Equation]

Original: CQL [36] further adds a conservative penalty to push down large Q -values for OOD actions:
Revised: CQL [36] further adds a conservative penalty to push down large Q -values for OOD actions:

Original: [Equation]
Revised: [Equation]

Original: COMBO [54] extends CQL by using an augmented dataset D (cid:99) M
Revised: COMBO [54] extends CQL by using an augmented dataset D c

Original:  = D ∪ D model for policy evaluation, where D model is generated using the learned dynamics model.
Revised: M = D [ D model for policy evaluation, where D model is generated using the learned dynamics model.

Original: Unlike other three baselines, IQL [30] uses expectile regression (ER) for policy evaluation purely from in-sample data:
Revised: Unlike other three baselines, IQL [30] uses expectile regression (ER) for policy evaluation purely from in-sample data:

Original: [Equation]
Revised: [Equation]

Original: where L τ 2 ( u ) = 
Revised: where L ⌧ 2 ( u ) = | ⌧  ( u < 0)

Original: |τ − 1 ( u < 0) | u 2 and τ ∈ (0 , 1) is a hyperparameter.
Revised: | u 2 and ⌧ 2 (0 , 1) is a hyperparameter.

Original: Policy improvement .
Revised: Policy improvement .

Original: TD3+BC adds a simple behavior cloning loss C BC ( ϕ ) =
Revised: TD3+BC adds a simple behavior cloning loss C BC (  ) =

Original: E [( π ( s ) − a ) 2 ] to the original TD3 [17] actor loss L TD 3 ( ϕ ) =
Revised: E [( ⇡ ( s )  a ) 2 ] to the original TD3 [17] actor loss L TD 3 (  ) =

Original: − E s ∼D [ Q πθ ( s, π ϕ ( s ))] .
Revised:  E s ⇠ D [ Q ⇡✓ ( s, ⇡  ( s ))] .

Original: CQL and COMBO both use the default SAC [22] actor loss function L SAC ( ϕ ) =
Revised: CQL and COMBO both use the default SAC [22] actor loss function L SAC (  ) =

Original: E ( s,a ) ∼D [ λ log( π ϕ ( a | s ))
Revised: E ( s,a ) ⇠ D [  log( ⇡  ( a | s ))

Original: − Q πθ ( s, π ϕ ( a | s ))] .
Revised:  Q ⇡✓ ( s, ⇡  ( a | s ))] .

Original: On the other hand, IQL uses the advantage-weighted regression (AWR) [43] to imitate high-quality in-sample actions 
Revised: On the other hand, IQL uses the advantage-weighted regression (AWR) [43] to imitate high-quality in-sample actions L AWR (  ) = E ( s,a ) ⇠ D

Original: [Equation]
Revised: [Equation]

Original: ) (cid:3) , where Vψ ( s ) is a value function that only depends on state s .
Revised: ⇤ , where V ( s ) is a value function that only depends on state s .

Original: Given these different policy evaluation/improvement methods, there is a lack of clear comparison to show which one is more effective in practice.
Revised: Given these different policy evaluation/improvement methods, there is a lack of clear comparison to show which one is more effective in practice.

Original: Here, we try to show that the previously introduced experiment setups can be helpful tools to answer such questions.
Revised: Here, we try to show that the previously introduced experiment setups can be helpful tools to answer such questions.

Original: Recall the results in the Table 4 and
Revised: Recall the results in the Table 4 and

Original: Table 5, we can observe that IQL [30] usually achieves higher rank IC and larger topN accuracy, which suggests that the expectile regression based policy evaluation method in IQL is more capable to learn accurate value functions.
Revised: Table 5, we can observe that IQL [30] usually achieves higher rank IC and larger topN accuracy, which suggests that the expectile regression based policy evaluation method in IQL is more capable to learn accurate value functions.

Original: However, IQL only achieves the best performance in one benchmark task.
Revised: However, IQL only achieves the best performance in one benchmark task.

Original: In addition, Table 8 indicates that IQL learns conservative policy that usually takes high confidence in-sample data.
Revised: In addition, Table 8 indicates that IQL learns conservative policy that usually takes high conﬁdence in-sample data.

Original: These results suggest that the AWR-based policy-improvement method is effective to avoid taking OOD actions but sometimes it is over-conservative which limits the performance.
Revised: These results suggest that the AWR-based policy-improvement method is effective to avoid taking OOD actions but sometimes it is over-conservative which limits the performance.

Original: To validate this assumption, we introduce a simple variant of IQL, called Relaxed In-sample Q-Learning (RIQL), that replaces the original AWR-based actor loss with:
Revised: To validate this assumption, we introduce a simple variant of IQL, called Relaxed In-sample Q-Learning (RIQL), that replaces the original AWR-based actor loss with:

Original: [Equation]
Revised: [Equation]

Original: In short, we add a KL-divergence constraint to the original SAC actor loss, and we drop the E ( s,a ) ∼D [log π β ( a | s )] term which is independent of actor parameter ϕ .
Revised: In short, we add a KL-divergence constraint to the original SAC actor loss, and we drop the E ( s,a ) ⇠ D [log ⇡  ( a | s )] term which is independent of actor parameter  .

Original: The motivation is twofold: (1) Unlike CQL and COMBO, the learned Q -functions in IQL are less discriminating w.r.t.
Revised: The motivation is two- fold: (1) Unlike CQL and COMBO, the learned Q -functions in IQL are less discriminating w.r.t.

Original: OOD samples.
Revised: OOD samples.

Original: Thus, we need extra policy constraints to avoid taking too many OOD actions.
Revised: Thus, we need extra policy constraints to avoid taking too many OOD actions.

Original: (2) We want to use a less conservative actor loss to enable the policy learning from OOD actions.
Revised: (2) We want to use a less conservative actor loss to enable the policy learning from OOD actions.

Original: In addition,
Revised: In addition,

Original: Eq 5 is similar to the actor loss in [11].
Revised: Eq 5 is similar to the actor loss in [11].

Original: The difference is that we keep the entropy term which helps to prevent the learned policy from collapsing to a single point.
Revised: The difference is that we keep the entropy term which helps to prevent the learned policy from collapsing to a single point.

Original: Experiment results of RIQL is shown in Table 10 and more details are described in Appendix
Revised: Experiment results of RIQL is shown in Table 10 and more details are described in Appendix

Original: F. We can observe that RIQL outperforms
Revised: F. We can observe that RIQL outperforms

Original: IQL in all benchmark tasks, and it achieves higher total evaluation scores than other baseline agents.
Revised: IQL in all benchmark tasks, and it achieves higher total evaluation scores than other baseline agents.

Original: This result also validates our previous assumption that AWR-based policy improvement method is the bottleneck for IQL in some tasks which makes it over-conservative.
Revised: This result also validates our previous assumption that AWR-based policy improvement method is the bottleneck for IQL in some tasks which makes it over-conservative.

Original: [Table]
Revised: [Table]

Original:  Uncertainty-based sample selection for model-based offline RL
Revised:  Uncertainty-based sample selection for model-based ofﬂine RL

Original: We also try to investigate when does a learned dynamics model is helpful to a model-free offline RL agent?
Revised: We also try to investigate when does a learned dynamics model is helpful to a model-free ofﬂine RL agent?

Original: In particular, we reuse the learned dynamics model in Section 5.2 to generate fake samples to train a model-free agent.
Revised: In particular, we reuse the learned dynamics model in Section 5.2 to generate fake samples to train a model-free agent.

Original: For example, we introduce an MBRL variant of IQL/TD3+BC, named
Revised: For example, we introduce an MBRL variant of IQL/TD3+BC, named

Original: MIQL/MTD3+BC, which learns from the augmented dataset D (cid:99) M = D ∪ D model as in MOPO [55].
Revised: MIQL/MTD3+BC, which learns from the augmented dataset D c M = D [ D model as in MOPO [55].

Original: We find that such a naive MBRL agent (a combination of MOPO and model-free offline RL agents) usually performs worse than its model-free counterpart (Table 11) even though they are the same algorithm except for the training data.
Revised: We ﬁnd that such a naive MBRL agent (a combination of MOPO and model-free ofﬂine RL agents) usually performs worse than its model-free counterpart (Table 11) even though they are the same algorithm except for the training data.

Original: This result indicates that it is nontrivial to combine a learned dynamics model with model-free offline RL agents due to the model noises.
Revised: This result indicates that it is nontrivial to combine a learned dynamics model with model-free ofﬂine RL agents due to the model noises.

Original: COMBO [54] addresses this problem by adding a conservative penalty on the model-generated samples.
Revised: COMBO [54] addresses this problem by adding a conservative penalty on the model-generated samples.

Original: Here, we introduce an Uncertainty-based Sample Selection (USS) method to filter fake samples with large model noises.
Revised: Here, we introduce an Uncertainty-based Sample Selection (USS) method to ﬁlter fake samples with large model noises.

Original: Since it is hard to measure model noises accurately, we adopt the model uncertainty σ ( s, a ) defined in Section 5.2 as a proxy to approximate model noise.
Revised: Since it is hard to measure model noises accurately, we adopt the model uncertainty  ( s, a ) deﬁned in Section 5.2 as a proxy to approximate model noise.

Original: In addition, we maintain a dynamic uncertainty threshold δ σ as follows δ σ ← ησ qbatch ( s, a ) + (1 − η ) δ σ , where σ qbatch ( s, a ) is the q -th quantile of the uncertainties in the sampled fake trajectories.
Revised: In addition, we maintain a dynamic uncertainty threshold   as follows   ⌘ qbatch ( s, a ) + (1  ⌘ )   , where  qbatch ( s, a ) is the q -th quantile of the uncertainties in the sampled fake trajectories.

Original: We only add model-generated samples with an uncertainty lower than δ σ to the model buffer which is later used to train the agent.
Revised: We only add model-generated samples with an uncertainty lower than   to the model buffer which is later used to train the agent.

Original: More details are described in Appendix G. From Table 11, we can observe that the proposed USS trick usually helps to improve the performance of the MIQL/MTD3+BC agent.
Revised: More details are described in Appendix G. From Table 11, we can observe that the proposed USS trick usually helps to improve the performance of the MIQL/MTD3+BC agent.

Original: Moreover, USS is inferior to IQL/TD3+BC in some tasks which shows that it is still challenging to completely solve the model noise problem.
Revised: Moreover, USS is inferior to IQL/TD3+BC in some tasks which shows that it is still challenging to completely solve the model noise problem.

Original: [Table]
Revised: [Table]

Original:  Conclusion
Revised:  Conclusion

Original: Besides the rapid development of novel offline RL algorithms, the behaviors of these offline RL agents have not been well studied.In this work, we take a closer look at SOTA offline RL agents.
Revised: Besides the rapid development of novel ofﬂine RL algorithms, the behaviors of these ofﬂine RL agents have not been well studied.

Original: Specifically, we introduce a series of experiments to compare the learned representations, value functions and policies of four baseline agents.

Original: Surprisingly, we find that a performant agent sometimes has relatively low quality representations and inaccurate value functions.

Original: We also show that the proposed experiment setups can be used to compare the effectiveness of different policy evaluation/improvement methods by introducing a relaxed version of IQL that achieves SOTA performance.

Original: Lastly, we investigate when a learned dynamics model can help model-free offline RL agents, and we introduce an uncertaintybased sample selection method to mitigate the problem of model noises.

Original: Appendix

Original: In the appendix, we first provide more background information and introduce some related works.

Original: Then we report the details of experiment setups and additional experiment results on all test environments.

Original: Lastly, we discuss the limitations and future directions of this work.

Original: A Extended background material

Original: Here, we provide a more extensive background review.

Original: Specifically, we present more discussions on offline reinforcement learning and representation learning for reinforcement learning.

Original: A.1 Offline reinforcement learning

Original: Offline reinforcement learning (RL) aims to learn effective control policies purely from fixed precollected datasets.

Original: Recent years have seen a surge of different offline RL methods.

Original: We crudely classify some latest works in the following categories.

Original: Policy-regularized offline RL.

Original: The majority of the recent proposed offline RL algorithms share a similar idea – constraining the learned policy to stay close to the behavior policy.

Original: For example,

Original: BCQ [18] used a variational auto-encoder (VAE) to generate actions that are similar to samples in the offline dataset.

Original: KL-Control [24] and CDC [11] used KL divergence, BEAR [35] adopted maximum mean discrepancy (MMD) divergence, FBRC selected Fisher-divergence [31] as a regularizer in the loss function.

Original: On the other hand, BRAC[50] and BRAC+ [56] proposed general frameworks for the such policy-regularized approaches.

Original: Pessimism-based offline RL.

Original: Another line of research focuses on attacking the over-estimation problem directly by using conservative penalties.

Original: CQL [36] proposed a penalty regularizer for out-ofdistribution (OOD) samples, such that we can learn conservative Q-functions that lower-bounds its true value.

Original: In addition, Buckman et al.

Original: [5] proposed a general theoretical framework to unify some existing pessimism-based approaches.

Original: Recently, CDC [11] introduced a new regularizer that only penalizes high-valued samples selected by the actor.

Original: Generalized behavior cloning (BC).

Original: Some other works resort to weighted behavior cloning, which reduces the RL problem to a supervised learning problem [10].

Original: For example, AWR [43] and CRR [49] used value-weighted regression to filter high-quality samples.

Original: BAIL [7] proposed upper-envelope to select good actions for later imitation learning.

Original: Recently, IQL [30] performed weighted behavioral cloning for policy extraction.

Original: Model-based offline RL.

Original: One motivation for using Model-based RL (MBRL) in the offline setting is to increase the data coverage, where the batch RL agent can learn policies with OOD states [55, 29].

Original: MOPO [55] and MoREL [29] measured the uncertainty of the model’s prediction to formulate an uncertainty-penalized MDP.

Original: COMBO [54] combined CQL with a learned model by penalizing the samples generated by the learned model.

Original: MuZero Unplugged [47] directly used the learned model for policy and value improvement through planning.

Original: Other types of offline RL.

Original: In addition to using uncertainty estimation under the MBRL framework, some recent model-free batch RL algorithms also adopt the uncertainty to mitigate the over-estimation error.

Original: UWAC [52] used dropout-uncertainty estimation to down-weight OOD samples loss.

Original: MSG [19] used an ensemble of independently-updated Q-functions for uncertainty estimation.

Original: Furthermore,

Original: REM [1] showed that standard off-policy RL methods can achieve good performance when the dataset is large and diverse.

Original: A.2 Representation learning for reinforcement learning

Original: Some recent works introduced different representation metrics as proxies to evaluate the quality of the learned representations of an RL agent [34, 33, 38, 41].

Original: We provide a more detailed introduction to the following three metrics that we use in the experiments.

Original: Definition 1 (Feature dot-product) .

Original: Feature dot-product ϕ ( s, a ) ⊤ ϕ ( s ′ , a ′ ) is the dot-product of two critic representations, where s ′ ∼ P ( ·| s, a ) and a ′ ∼ π ( ·| s ′ ) is the next state and next action.

Original: Definition 2 (Effective rank) .

Original: Effective rank srank δ (Φ) =

Original: min (cid:8) k : (cid:80) ki

Original: =1 σ i (Φ) (cid:80) di

Original: =1 σ i (Φ) ≥ 1 − δ (cid:9) of a feature matrix Φ ∈ R |S|·|A|× d approximates the rank of Φ [33], where { σ i (Φ) } are the singular values of Φ in the decreasing order ( σ 1 ≥ · · · ≥ σ d ≥ 0 ) and δ is a threshold parameter, i.e., 0 .

Original: Definition 3 (Effective dimension) .

Original: Effective dimension d eff (Φ) =

Original: N max i =1 , ··· ,N ∥ P Φ e i ∥

Original: 22 of a feature matrix Φ ∈ R |S|·|A|× d measures the sparsity of the column space of Φ [38], where

Original: N = |S| · |A| and P Φ is the orthogonal projector onto the column space of Φ .

Original: DR3 [34] introduced the feature co-adaptation phenomenon where the feature dot-product keeps increasing during the training of a deep Q -Network.

Original: In particular, DR3 provided a theoretical analysis to show that optimizing TD errors with SGD has an implicit “regularizer” that leads to the feature co-adaptation phenomenon which usually hurts the performance.

Original: Later, some related works [33, 38, 41] introduced the representation collapse phenomenon, where the rank of the feature space shrinks during the training step, which limits the learning capacity of an RL agent.

Original: In particular,

Original: Kumar et al.

Original: [33] pointed out that such representation collapse phenomenon is connected to the bootstrapping based update, and provided some theoretical explanations in the context of kernel regression and deep linear networks.

Original: On the other hand, Lan et al.

Original: [38] provided an informative bound on the generalization error w.r.t the effective dimension of state representations, and showed that a smaller effective dimension helps to improve the performance.

Original: To mitigate the problem of representation collapse, InFeR [41] presented a regularization term that forces the trained networks to regress to a random function of their initial output.

Original: Another closely related work is ACL [53], which empirically investigated a number of representation auxiliary losses for RL agents.

Original: Moreover, Deng et al.

Original: [9] also explored the representation bottlenecks for deep neural networks.

Original: A major difference of this work and these related works is that we studied the detailed behaviors, including representations, value function and policies, of offline RL agents.

Original: B Experiment details

Original: In this section, we present more details about the dataset and the baseline algorithms.

Original: B.1 Experiment setups

Original: In the experiment, we evaluate all baseline agents on the standard D4RL benchmark [14].

Original: In specific, we use the “-v2” dataset, which contains more metadata and fixes bugs in the “-v0” dataset 1 , to train each baseline agent for 1M steps.

Original: In all experiments, we report the average result and standard deviation over 5 random seeds.

Original: We run all experiments on a workstation with GeForce GTX

Original: GPU and an Intel Core i9-12900KF CPU.

Original: B.2 Baseline algorithms

Original: We re-implement each baseline algorithm in JAX [13].

Original: For TD3BC [16], CQL [36] and IQL [30], we use the default parameter settings as in the original implementations.

Original: Moreover, we find the default parameters in COMBO [54], which is originally trained on the “-v0” dataset, sometimes fail in the “-v2” dataset.

Original: Thus, we additionally tune the conservative parameter β for COMBO using the offline cross-validation scheme as described in the original paper [54].

Original: For the software, we use the following versions:

Original: • Python 3.

Original: • Jax 0.3.

Original: • Gym 0.23.

Original: • Mujoco 2.1.

Original: • mujoco-py 2.1.2.

Original: • d4rl 1.

Original: The running time for 1M steps for each baseline algorithm in the JAX version is usually faster than the original implementations [30] as summarized in Table 12:

Original: [Table]

Original: C Representation evaluation experiments

Original: In this section, we provide more results of the representation experiments in different environments.

Original: In the experiment, we first train an online TD3 [17] agent for 2M steps to approximate the optimal policy π ∗ ( a | s ) and optimal value functions Q ∗ ( s, a ) and V ∗ ( s ) .

Original: For each probing target, we train a linear model for 200 epochs using the Adam optimizer with learning rate 3e-4.

Original: We adopt an early stopping strategy with patience of 10 epochs.

Original: We report the average result of a 5-fold cross-validation.

Original: The most performant baseline agents in each environment are brown, and the agent with the best probing experiment result is in bold .

Original: C.1 Representation probing experiment results

Original: In the experiment, the COMBO agent crashed in the walker2d-medium-expert-v2 environment for one random seed (learned explosive Q ( s, a ) ).

Original: In some probing experiments, the crashed COMBO agent diverges.

Original: We report the mean and standard deviation of the MSE loss over 5 random seeds.

Original: From Table 13, Table 14 and Table 15, we can observe that the TD3BC agent usually has good probing experiment results (with the least probing loss), however, the TD3BC agent only achieves the best performance in 2 of the 9 environments.

Original: These results show that the transition dynamics-based side information is not that important for offline RL agents in the selected D4RL tasks.

Original: Intuitively, the most performant agent usually has small optimal a ∗ probing loss (Table 16).

Original: This indicates that as long as the actor representation ψ ( s ) preserves the ability to learn good actions (low optimal action probing loss), then the offline RL agent holds the potential to achieve good performance.

Original: From Table 17 and Table 18, we can find that the probing loss w.r.t.

Original: the optimal value functions are usually very large.

Original: Such large probing loss highlights the difficulty to learn accurate value functions in the offline setting due to the limited data coverage and additional policy/value constraints

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: C.2 Representation metric experiment results

Original: Before reporting the results for the representation metric experiment, we first give a brief discussion of the implementation of the effective dimension .

Original: As shown in the following code snippet (Algo 1), the original implementation of the effective dimension 2 uses a hard threshold (1e-5) to approximate the rank of the feature matrix.

Original: However, such a hard threshold is prone to over-estimate the rank when the norm of the representation is large, as we will see in Table 19 and Table 20.

Original: Therefore, we use a modified implementation which approximate the rank using a relative threshold (effective rank).

Original: [Table]

Original: From Table 19, we can observe that the critic constraint in offline RL will make the feature co-adaption more severe.

Original: TD3BC agent does not use any critic constraint, and the consecutive state-action pairs 2 Original implementation of the effective dimension.

Original: are the least similar.

Original: On the other hand, the feature co-adaptation phenomenon is particularly severe when we adds a conservative penalty.

Original: Interestingly, such a large feature dot-product does not prevent the COMBO agent to achieve good performance in some offline RL tasks.

Original: In addition, combined results in Table 19 and Table 20, we can conclude that the increasing feature dot-product is mainly caused by the representation norm.

Original: These results also provide a hint for how the conservative penalty in CQL or COMBO works.

Original: As we can see that the representation norm | ϕ ( s, a ) | is usually large for a

Original: CQL or COMBO agent, then it learns a lower-bounded Q -values Q

Original: θ ( s, a ) =

Original: θ ⊤ ϕ ( s, a ) by a small | θ

Original: | and (or) dissimilar vector direction of θ and ϕ ( s, a ) .

Original: From Table 21 and Table 22, we can observe that the COMBO agent ususally suffers from the representation collapse problem while its model-free counterpart CQL agent doesn’t.

Original: This result indicates that the usage of a learned dynamics model would sometimes affect the learned critic representations.

Original: We leave a deeper investigation for future work. [Table] [Table] [Table]
Revised: In this work, we take a closer look at SOTA ofﬂine RL agents.

Original: [Table]

Original: Fig 5 illustrates the pipeline of value ranking experiment. Firstly, we use a behavior policy π β to interact with the environment to collect a set of states { s 1 , · · · , s N } as the test dataset. In particular, the behavior policy π β is a mixture of four baseline offline RL agents and a near-optimal online agent.
Revised: Speciﬁcally, we introduce a series of experiments to compare the learned representations, value functions and policies of four baseline agents.

Original: We use each agent to collect 50K transitions.

Original: Secondly, at each state s i , we use each baseline agent to select m = 5 different actions a ′ =

Original: π ( ·| s i )

Original: + ϵ , where ϵ i is a Gaussian noise with µ = 0 and σ = 0 .

Original: Thirdly, we use each baseline agent to evaluate the Q -values of the collected state-action pairs.

Original: Lastly, we use the online agent to approximate the optimal agent to compute the Spearman’s rank correlation coefficient (rank IC) and top-N accuracy.

Original: Since this experiment only utilizes the learned Q -function of each baseline agent, therefore, we use the experiment result as a proxy to evaluate how accurate the learned Q -values are at ranking different actions.

Original: Moreover, the experiment result is also a useful evidence to show the effectiveness of different policy evaluation methods.

Original: E Performant policy experiments E.1 Policy ranking experiment

Original: Fig 6 shows the pipeline of the policy ranking experiment.

Original: Similar to the previous value ranking experiment, we use the online agent to approximate optimal Q ∗ ( s, a ) to rank the actions selected by each baseline agents.

Original: We use the following two metrics to evaluate the goodness of the learned policy: (1) Average percentage of policy π j that rank the first/last (with largest/smallest Q ∗ value) across all states 1 |D policy | (cid:80) i 1 { rank( Q ∗ ( π j ( s i ) , s i ))

Original: = 1 } . (2) Average mean square error (MSE) loss of the selected action w.r.t. the optimal action [Figure]
Revised: Surprisingly, we ﬁnd that a performant agent sometimes has relatively low quality representations and inaccurate value functions.

Original: |D policy | (cid:80) t ∥ a ∗ i − π j ( s i ) ∥

Original: [Figure]

Original: E.2 OOD action experiments

Original: To investigate how often do different offline RL agents take OOD actions, we first use each baseline agent to interact with the environment to collect 50K transitions.

Original: Then we use a probabilistic ensemble-based dynamics model [8] as in COMBO [54] to estimate the uncertainty [55] of each state-action pair as σ ( s, a ) = max i =1 , ··· ,N ∥ µ iθ ( s, a ) − 1 N (cid:80) Nj =1 µ jθ ( s, a ) ∥ 2 .

Original: In the experiment, we set the ensemble model number N = 7 as in COMBO.

Original: We report the average median and standard deviation of the uncertainty in the collection samples in Table 8.

Original: In RIQL, we attempt to make a minimal modification w.r.t. the IQL algorithm [30]. Since the motivation of RIQL is to relax the in-sample constraint for the  policy improvement step in IQL, we therefore replace the Advantage Weighted Regression (AWR) loss function with the following:
Revised: We also show that the proposed experiment setups can be used to compare the effectiveness of different policy evaluation/improvement methods by introducing a relaxed version of IQL that achieves SOTA performance.

Original: [Equation]

Original: In short, we add a KL-divergence constraint to the SAC [22] actor loss.

Original: We determine the parameter β

Original: = { 0 . 1 , 0 . 25 , 0 . 5 , 1 . 0 , 1 . 5 } using the offline cross-validation scheme as described in COMBO [54].

Original: For the policy evaluation step, we follow the original expectile regression-based method to update the two value functions Q θ ( s, a ) and V

Original: [Equation]

Original: where Q ˆ θ ( s, a ) is the target Q network, L τ 2 ( u ) =

Original: | τ − 1 ( u < 0) | u 2 and τ ∈ (0 , 1) is a hyperparameter.

Original: The pseudocode of RIQL is described in Algo 2.

Original: In addition, Fig 7 shows the learning curves of

Original: RIQL where we report the average evaluation return over 5 random seeds and the standard deviation is in shadow.

Original: Input: offline dataset D .

Original: Initialize parameters ψ , θ , ˆ θ , ϕ .

Original: Policy evaluation with expectile regression: for each gradient step do ψ ← ψ − λ V ∇ ψ L V ( ψ ) θ ← θ − λ Q ∇ θ L Q ( θ ) ˆ θ ← (1 − α )

Original: ˆ θ + αθ end for

Original: Policy improvement with KL constraint: for each gradient step do ϕ ← ϕ − λ π ∇ ϕ L π ( ϕ ) end for

Original: G Uncertainty-based Sample Selection (USS)

Original: The proposed Uncertainty-based Sample Selection (USS) is a simple trick for model-based offline

Original: RL.

Original: Here we present a detailed pipeline to showcase how it can be implemented.

Original: • We first learn a probabilistic ensemble-based dynamics model as in the COMBO using the offline dataset.

Original: • For every T = 1000 step, we use the offline RL agent to rollout the model to generate 10K short trajectories (with horizon = 5).

Original: • We then use the generated samples to update the dynamic uncertainty threshold δ σ ← ησ qbatch ( s, a ) + (1 − η ) δ σ , where η = 0 .

Original: 1 and σ qbatch ( s, a ) is the median uncertainty of the sampled transitions.

Original: The δ σ is initialized to be the median uncertainty of the transition samples in the first rollout.

Original: We only add samples with uncertainty lower than δ σ to the extra model buffer D model . • We train the agent using the augmented dataset D ∪ D model where D is the original offline dataset. H Limitations andfuture directions In this work, we introduced a series of experiment setups to analyze the behaviors of offline reinforcement learning agents.
Revised: Lastly, we investigate when a learned dynamics model can help model-free ofﬂine RL agents, and we introduce an uncertaintybased sample selection method to mitigate the problem of model noises.

Original: Specifically, we focus on three fundamental aspects of an RL agent – learned representations, value functions, and policies.

Original: From the empirical experiment results, we discover some bottlenecks of current SOTA offline RL agents.

Original: As a case study, we introduced a variant of IQL agent, called RIQL, which uses a relaxed policy improvement method and achieves better performance.

Original: A major limitation of our work is that we need to use an online agent to approximate the optimal agent, which might be unavailable in many problems.

Original: In this work, we empirically show that some performant offline RL agents learned low-quality representations.

Original: Therefore, an interesting future direction is to investigate the effectiveness of the latest representation learning methods in offline RL.

Original: Another future direction is to extend the proposed experiment setups to the online setting.