Original: When does return-conditioned supervised learning work for offline reinforcement learning?
Revised: When does return-conditioned supervised learning work for offline reinforcement learning?

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Several recent works have proposed a class of algorithms for the offline reinforcement learning (RL) problem that we will refer to as return-conditioned supervisedlearning (RCSL).
Revised: Several recent works have proposed a class of algorithms for the offline reinforcement learning (RL) problem that we will refer to as return-conditioned supervisedlearning (RCSL).

Original: RCSL algorithms learn the distribution of actions conditionedon both the state and the return of the trajectory.
Revised: RCSL algorithms learn the distribution of actions conditionedon both the state and the return of the trajectory.

Original: Then they define a policy byconditioning on achieving high return.
Revised: Then they define a policy byconditioning on achieving high return.

Original: In this paper, we provide a rigorous studyof the capabilities and limitations of RCSL, something which is crucially missing in previous work.
Revised: In this paper, we provide a rigorous studyof the capabilities and limitations of RCSL, something which is crucially missing in previous work.

Original: We find that RCSL returns the optimal policy under a setof assumptions that are stronger than those needed for the more traditional dynamic programming-based algorithms.
Revised: We find that RCSL returns the optimal policy under a setof assumptions that are stronger than those needed for the more traditional dynamic programming-based algorithms.

Original: We provide specific examples of MDPsand datasets that illustrate the necessity of these assumptions and the limits of
Revised: We provide specific examples of MDPsand datasets that illustrate the necessity of these assumptions and the limits of

Original: RCSL.
Revised: RCSL.

Original: Finally, we present empirical evidence that these limitations will alsocause issues in practice by providing illustrative experiments in simple point-massenvironments and on datasets from the D4RL benchmark.
Revised: Finally, we present empirical evidence that these limitations will alsocause issues in practice by providing illustrative experiments in simple point-massenvironments and on datasets from the D4RL benchmark.

Original:  Introduction
Revised:  Introduction

Original: In recent years, deep learning has proven to be an exceptionally powerful generic algorithm forsolving supervised learning (SL) tasks.
Revised: In recent years, deep learning has proven to be an exceptionally powerful generic algorithm forsolving supervised learning (SL) tasks.

Original: These approaches tend to be stable, and scale well withcompute and data [17].
Revised: These approaches tend to be stable, and scale well withcompute and data [17].

Original: In contrast, deep reinforcement learning algorithms seem to lack these niceproperties; results are well known to be sensitive to hyperparameters and difficult to replicate.
Revised: In contrast, deep reinforcement learning algorithms seem to lack these niceproperties; results are well known to be sensitive to hyperparameters and difficult to replicate.

Original: Inspite of this, deep reinforcement learning (RL) has achieved impressive feats, such as defeatinghuman champions at Go [24].
Revised: Inspite of this, deep reinforcement learning (RL) has achieved impressive feats, such as defeatinghuman champions at Go [24].

Original: This juxtaposition of success and instability has inspired researchersto explore alternative approaches to reinforcement learning that more closely resemble supervisedlearning in hopes of making deep RL as well-behaved as deep SL.
Revised: This juxtaposition of success and instability has inspired researchersto explore alternative approaches to reinforcement learning that more closely resemble supervisedlearning in hopes of making deep RL as well-behaved as deep SL.

Original: One family of algorithms that has garnered great interest recently is return-conditioned supervisedlearning (RCSL).
Revised: One family of algorithms that has garnered great interest recently is return-conditioned supervisedlearning (RCSL).

Original: The core idea of RCSL is to learn the return-conditional distribution of actionsin each state, and then define a policy by sampling from the distribution of actions that receivehigh return.
Revised: The core idea of RCSL is to learn the return-conditional distribution of actionsin each state, and then define a policy by sampling from the distribution of actions that receivehigh return.

Original: This was first proposed for the online RL setting by work on Upside Down RL [22,25] and Reward Conditioned Policies [21].
Revised: This was first proposed for the online RL setting by work on Upside Down RL [22,25] and Reward Conditioned Policies [21].

Original: The idea was extended to the offline RL setting usingtransformers that condition on the entire history of states rather than just the current Markovian statein the Decision Transformer (DT) work [8, 12].
Revised: The idea was extended to the offline RL setting usingtransformers that condition on the entire history of states rather than just the current Markovian statein the Decision Transformer (DT) work [8, 12].

Original: Recent work on RL via Supervised Learning (RvS)[9] unifies and simplifies ideas from these prior works with ideas about goal-conditioned policies.
Revised: Recent work on RL via Supervised Learning (RvS)[9] unifies and simplifies ideas from these prior works with ideas about goal-conditioned policies.

Original: Importantly, none of this prior work provides theoretical guarantees or analysis of the failure modesof the return-conditioning approach.
Revised: Importantly, none of this prior work provides theoretical guarantees or analysis of the failure modesof the return-conditioning approach.

Original: In contrast, the more established dynamic programming (DP)algorithms for RL are better understood theoretically.
Revised: In contrast, the more established dynamic programming (DP)algorithms for RL are better understood theoretically.

Original: This paper attempts to address this gap inunderstanding, in order to assess when RCSL is a reliable approach for offline RL.
Revised: This paper attempts to address this gap inunderstanding, in order to assess when RCSL is a reliable approach for offline RL.

Original: Specifically, weanswer the following questions:
Revised: Specifically, weanswer the following questions:

Original: • What optimality guarantees can we make for RCSL?
Revised: • What optimality guarantees can we make for RCSL?

Original: Under what conditions are they necessary and sufficient?•
Revised: Under what conditions are they necessary and sufficient?•

Original: In what situations does RCSL fail in theory and in practice?• How does RCSL relate to other approaches, such as DP and behavior cloning (BC)?
Revised: In what situations does RCSL fail in theory and in practice?• How does RCSL relate to other approaches, such as DP and behavior cloning (BC)?

Original: We find that although RCSL does select a near-optimal policy under certain conditions, the necessaryassumptions are more strict than those for DP.
Revised: We find that although RCSL does select a near-optimal policy under certain conditions, the necessaryassumptions are more strict than those for DP.

Original: In particular, RCSL (but not DP) requires nearlydeterministic dynamics in the MDP, knowledge of the proper value to condition on, and for theconditioning value to be supported by the distribution of returns in the dataset.
Revised: In particular, RCSL (but not DP) requires nearlydeterministic dynamics in the MDP, knowledge of the proper value to condition on, and for theconditioning value to be supported by the distribution of returns in the dataset.

Original: We provide simpletabular examples to demonstrate the necessity of these assumptions.
Revised: We provide simpletabular examples to demonstrate the necessity of these assumptions.

Original: The shortcomings of RCSL thatwe identify in theory are verified empirically with some simple experiments using neural models onad-hoc example problems as well as benchmark datasets.
Revised: The shortcomings of RCSL thatwe identify in theory are verified empirically with some simple experiments using neural models onad-hoc example problems as well as benchmark datasets.

Original: We conclude that RCSL alone is unlikelyto be a general solution for offline RL problems, but does show promise in some specific situationssuch as deterministic MDPs with high-quality behavior data.
Revised: We conclude that RCSL alone is unlikelyto be a general solution for offline RL problems, but does show promise in some specific situationssuch as deterministic MDPs with high-quality behavior data.

Original:  Preliminaries2.1 Setup
Revised:  Preliminaries2.1 Setup

Original: We will consider an offline RL setup where we are given a dataset D of trajectories τ =( o 1 , a 1 , r 1 , · · · , o H , a H , r H ) of observations o t ∈ O , actions a t ∈ A , and rewards r t ∈ [0 , 1]generated by some behavior policy β interacting with a finite horizon MDP with horizon H .
Revised: We will consider an offline RL setup where we are given a dataset D of trajectories τ =( o 1 , a 1 , r 1 , · · · , o H , a H , r H ) of observations o t ∈ O , actions a t ∈ A , and rewards r t ∈ [0 , 1]generated by some behavior policy β interacting with a finite horizon MDP with horizon H .

Original: Letg ( τ ) =
Revised: Letg ( τ ) =

Original: (cid:80)
Revised: (cid:80)

Original: Ht
Revised: Ht

Original: =1 r t denote the cumulative return of the trajectory (we will just use g when the trajectoryis clear from context).
Revised: =1 r t denote the cumulative return of the trajectory (we will just use g when the trajectoryis clear from context).

Original: And let J ( π ) =
Revised: And let J ( π ) =

Original: E τ ∼ π
Revised: E τ ∼ π

Original: [ g ( τ )] be the expected return of a policy π .
Revised: [ g ( τ )] be the expected return of a policy π .

Original: We then letthe state representation s t ∈ S be any function of the history of observations, actions, and rewardsup to step t along with o t .
Revised: We then letthe state representation s t ∈ S be any function of the history of observations, actions, and rewardsup to step t along with o t .

Original: To simplify notation in the finite horizon setting, we will sometimes dropthe timestep from s to refer to generic states and assume that we can access the timestep from thestate representation as t ( s ) .
Revised: To simplify notation in the finite horizon setting, we will sometimes dropthe timestep from s to refer to generic states and assume that we can access the timestep from thestate representation as t ( s ) .

Original: Let P π denote the joint distribution over states, actions, rewards, andreturns induced by any policy π .
Revised: Let P π denote the joint distribution over states, actions, rewards, andreturns induced by any policy π .

Original: In this paper, we focus on the RCSL approach that learns by return-conditioned supervised learning.
Revised: In this paper, we focus on the RCSL approach that learns by return-conditioned supervised learning.

Original: Explicitly, at training time this method minimizes the empirical negative log likelihood loss:
Revised: Explicitly, at training time this method minimizes the empirical negative log likelihood loss:

Original: [Equation]
Revised: [Equation]

Original: Then at test time, an algorithm takes the learned policy π along with a conditioning function f ( s ) todefine the test-time policy π f as:
Revised: Then at test time, an algorithm takes the learned policy π along with a conditioning function f ( s ) todefine the test-time policy π f as:

Original: [Equation]
Revised: [Equation]

Original: Nota bene: the Decision Transformer [8] is captured in this framework by defining the state space sothat the state s t at time t also contains all past o t ′ , a t ′ , and r t ′ for t ′ < t .
Revised: Nota bene: the Decision Transformer [8] is captured in this framework by defining the state space sothat the state s t at time t also contains all past o t ′ , a t ′ , and r t ′ for t ′ < t .

Original: In prior work, f is usuallychosen to be a constant at the initial state and to decrease with observed reward along a trajectory,which is captured by a state representation that includes the history of rewards.
Revised: In prior work, f is usuallychosen to be a constant at the initial state and to decrease with observed reward along a trajectory,which is captured by a state representation that includes the history of rewards.

Original: 2.2 The RCSL policy
Revised: 2.2 The RCSL policy

Original: To better understand the objective, it is useful to first consider its optimum in the case of infinitedata.
Revised: To better understand the objective, it is useful to first consider its optimum in the case of infinitedata.

Original: It is clear that our loss function attempts to learn P β ( a | s, g ) where β is the behavior policythat generated the data (and recall that P β refers to the distribution over states, actions, and returnsinduced by β ).
Revised: It is clear that our loss function attempts to learn P β ( a | s, g ) where β is the behavior policythat generated the data (and recall that P β refers to the distribution over states, actions, and returnsinduced by β ).

Original: Factoring this distribution, we quickly see that the optimal policy π RCSL f for a specificconditioning function f can be written as:
Revised: Factoring this distribution, we quickly see that the optimal policy π RCSL f for a specificconditioning function f can be written as:

Original: [Equation]
Revised: [Equation]

Original: Essentially, the RCSL policy re-weights the behavior based on the distribution of future returns.
Revised: Essentially, the RCSL policy re-weights the behavior based on the distribution of future returns.

Original: Connection to distributional RL.
Revised: Connection to distributional RL.

Original: In distributional RL [4], the distribution of future returns undera policy π from state s and action a is defined as: G π ( s, a )
Revised: In distributional RL [4], the distribution of future returns undera policy π from state s and action a is defined as: G π ( s, a )

Original: ∼ g
Revised: ∼ g

Original: =
Revised: =

Original: (cid:80)
Revised: (cid:80)

Original: Ht
Revised: Ht

Original: = t ( s ) r t | τ ∼ π, s t ( s )
Revised: = t ( s ) r t | τ ∼ π, s t ( s )

Original: =
Revised: =

Original: s, a t ( s )
Revised: s, a t ( s )

Original: =
Revised: =

Original: a .
Revised: a .

Original: The RCSL policy is precisely proportional to the product of the behavior policy andthe density of the distributional Q function of the behavior policy (i.e. P β ( g | s, a ) ).
Revised: The RCSL policy is precisely proportional to the product of the behavior policy andthe density of the distributional Q function of the behavior policy (i.e. P β ( g | s, a ) ).

Original: 2.3 Related work
Revised: 2.3 Related work

Original: As noted in the introduction, our work is in direct response to the recent line of literature on RCSL[22, 25, 21, 8, 12, 9].
Revised: As noted in the introduction, our work is in direct response to the recent line of literature on RCSL[22, 25, 21, 8, 12, 9].

Original: Specifically, we will focus on the DT [8] and RvS [9] formulations in ourexperiments since they also focus on the offline RL setting.
Revised: Specifically, we will focus on the DT [8] and RvS [9] formulations in ourexperiments since they also focus on the offline RL setting.

Original: Note that another recent work introducedthe Trajectory Transformer [15] which does not fall under the RCSL umbrella since it performsplanning in the learned model to define a policy.
Revised: Note that another recent work introducedthe Trajectory Transformer [15] which does not fall under the RCSL umbrella since it performsplanning in the learned model to define a policy.

Original: Another relevant predecessor of RCSL comes from work on goal-based RL [16].
Revised: Another relevant predecessor of RCSL comes from work on goal-based RL [16].

Original: Compared to
Revised: Compared to

Original: RCSL, this line of work replaces the target return g in the empirical loss function by a goal state.
Revised: RCSL, this line of work replaces the target return g in the empirical loss function by a goal state.

Original: One instantiation is hindsight experience replay (HER) where each trajectory in the replay bufferis relabeled as if the observed final state was in fact the goal state [2].
Revised: One instantiation is hindsight experience replay (HER) where each trajectory in the replay bufferis relabeled as if the observed final state was in fact the goal state [2].

Original: Another instance is goalconditioned supervised learning [GCSL, 13], which provides more careful analysis and guarantees,but the guarantees (1) are not transferable to the return-conditioned setting, (2) assume bounds on
Revised: Another instance is goalconditioned supervised learning [GCSL, 13], which provides more careful analysis and guarantees,but the guarantees (1) are not transferable to the return-conditioned setting, (2) assume bounds on

Original: L ∞ errors in TV distance instead of dealing with expected loss functions that can be estimated fromdata, and (3) do not provide analysis of the tightness of the bounds.
Revised: L ∞ errors in TV distance instead of dealing with expected loss functions that can be estimated fromdata, and (3) do not provide analysis of the tightness of the bounds.

Original: Concurrent work [32] presents an example where the online upside down RL [22] algorithm candiverge in stochastic environments, however unlike our work they (1) do not present any positiveresults, (2) do not consider issues of sample complexity, and (3) consider the online rather than theoffline variant of the algorithm.
Revised: Concurrent work [32] presents an example where the online upside down RL [22] algorithm candiverge in stochastic environments, however unlike our work they (1) do not present any positiveresults, (2) do not consider issues of sample complexity, and (3) consider the online rather than theoffline variant of the algorithm.

Original:  When does RCSL find the optimal policy?
Revised:  When does RCSL find the optimal policy?

Original: We begin by exploring how RCSL behaves with infinite data and a fully expressive policy class.
Revised: We begin by exploring how RCSL behaves with infinite data and a fully expressive policy class.

Original: In this setting, classic DP algorithms (e.g. Q-learning) are guaranteed to converge to the optimalpolicy under coverage assumptions [26].
Revised: In this setting, classic DP algorithms (e.g. Q-learning) are guaranteed to converge to the optimalpolicy under coverage assumptions [26].

Original: But we now show that this is not the case for RCSL,which requires additional assumptions for a similar guarantee.
Revised: But we now show that this is not the case for RCSL,which requires additional assumptions for a similar guarantee.

Original: Our approach is to first derive apositive result: under certain assumptions, the policy which optimizes the RCSL objective (Section2.2) is guaranteed to be near-optimal.
Revised: Our approach is to first derive apositive result: under certain assumptions, the policy which optimizes the RCSL objective (Section2.2) is guaranteed to be near-optimal.

Original: We then illustrate the limitations of RCSL by providingsimple examples that are nonetheless challenging for these methods in order to demonstrate whyour assumptions are necessary and that our bound is tight.
Revised: We then illustrate the limitations of RCSL by providingsimple examples that are nonetheless challenging for these methods in order to demonstrate whyour assumptions are necessary and that our bound is tight.

Original: Theorem 1 (Alignment with respect to the conditioning function) .
Revised: Theorem 1 (Alignment with respect to the conditioning function) .

Original: Consider an MDP, behavior βand conditioning function f .
Revised: Consider an MDP, behavior βand conditioning function f .

Original: Assume the following:
Revised: Assume the following:

Original: Return coverage: P β ( g = f
Revised: Return coverage: P β ( g = f

Original: ( s 1 ) | s 1 ) ≥
Revised: ( s 1 ) | s 1 ) ≥

Original: α f for all initial states s 1 .
Revised: α f for all initial states s 1 .

Original: Near determinism: P ( r ̸
Revised: Near determinism: P ( r ̸

Original: = r
Revised: = r

Original: ( s, a ) or s
Revised: ( s, a ) or s

Original: ′ ̸
Revised: ′ ̸

Original: = T ( s, a ) | s, a ) ≤ ϵ at all s, a for some functions
Revised: = T ( s, a ) | s, a ) ≤ ϵ at all s, a for some functions

Original: T and r .
Revised: T and r .

Original: Note that this does not constrain the stochasticity of the initial state.
Revised: Note that this does not constrain the stochasticity of the initial state.

Original: Consistency of f : f ( s ) =
Revised: Consistency of f : f ( s ) =

Original: f ( s ′ )
Revised: f ( s ′ )

Original: + r for all s .
Revised: + r for all s .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Moreover, there exist problems where the bound is tight up to constant factors.
Revised: Moreover, there exist problems where the bound is tight up to constant factors.

Original: The proof is in Appendix B.1.
Revised: The proof is in Appendix B.1.

Original: Note that the quantity E s 1 [ f ( s 1 )] is specific to the structure of RCSLalgorithms and captures the notion that the ideal RCSL policy will be able to reproduce policiesof any value when given different conditioning functions (with appropriate data).
Revised: Note that the quantity E s 1 [ f ( s 1 )] is specific to the structure of RCSLalgorithms and captures the notion that the ideal RCSL policy will be able to reproduce policiesof any value when given different conditioning functions (with appropriate data).

Original: The theoremimmediately yields the following corollaries (with proof in Appendix B.1).
Revised: The theoremimmediately yields the following corollaries (with proof in Appendix B.1).

Original: Corollary 1.
Revised: Corollary 1.

Original: Under the assumptions of Theorem 1, there exists a conditioning function f such that
Revised: Under the assumptions of Theorem 1, there exists a conditioning function f such that

Original: [Equation]
Revised: [Equation]

Original: Corollary 2.
Revised: Corollary 2.

Original: If α f > 0 , ϵ = 0 , and f ( s 1 )
Revised: If α f > 0 , ϵ = 0 , and f ( s 1 )

Original: =
Revised: =

Original: V ∗ ( s 1 ) for all initial states s 1 , then J ( π RCSL f ) =
Revised: V ∗ ( s 1 ) for all initial states s 1 , then J ( π RCSL f ) =

Original: J ( π ∗ ) .
Revised: J ( π ∗ ) .

Original: The corollaries tell us that in near determinisitc environments with the proper conditioning functionsand data coverage, it is possible for RCSL to recover near optimal policies.
Revised: The corollaries tell us that in near determinisitc environments with the proper conditioning functionsand data coverage, it is possible for RCSL to recover near optimal policies.

Original: These assumptions aresomewhat strong compared to those needed for DP-based approaches, so we will now explain whythey are necessary for our analysis.
Revised: These assumptions aresomewhat strong compared to those needed for DP-based approaches, so we will now explain whythey are necessary for our analysis.

Original: Tightness.
Revised: Tightness.

Original: To demonstrate tightness we will consider the simple examples in Figure 1.
Revised: To demonstrate tightness we will consider the simple examples in Figure 1.

Original: These
Revised: These

Original: MDPs and behavior policies demonstrate tightness in ϵ and α f up to constant factors, and provideinsight into how stochastic dynamics lead to suboptimal behavior from RCSL algorithms.
Revised: MDPs and behavior policies demonstrate tightness in ϵ and α f up to constant factors, and provideinsight into how stochastic dynamics lead to suboptimal behavior from RCSL algorithms.

Original: [Figure]
Revised: [Figure]

Original: First, consider the example in Figure 1a with conditioning f ( s 1 ) = 1 .
Revised: First, consider the example in Figure 1a with conditioning f ( s 1 ) = 1 .

Original: There is only one possiblepolicy in this case, and it has J ( π ) =
Revised: There is only one possiblepolicy in this case, and it has J ( π ) =

Original: ϵ so that E [ f ( s 1 )] − J ( π ) = 1 − ϵ .
Revised: ϵ so that E [ f ( s 1 )] − J ( π ) = 1 − ϵ .

Original: Note that α f =
Revised: Note that α f =

Original: ϵ , so wehave that ϵ/α f = 1 .
Revised: ϵ , so wehave that ϵ/α f = 1 .

Original: Thus, the bound is tight in ϵ/α f .
Revised: Thus, the bound is tight in ϵ/α f .

Original: This example shows that the goal of achievinga specific desired return is incompatible with stochastic environments.
Revised: This example shows that the goal of achievinga specific desired return is incompatible with stochastic environments.

Original: This first example is somewhat silly since there is only one action, so the learned policy does notactually suffer any regret.
Revised: This first example is somewhat silly since there is only one action, so the learned policy does notactually suffer any regret.

Original: To show that this issue can in fact lead to regret, consider the example in
Revised: To show that this issue can in fact lead to regret, consider the example in

Original: Figure 1b, again with conditioning f
Revised: Figure 1b, again with conditioning f

Original: ( s 1 )
Revised: ( s 1 )

Original: = 1 .
Revised: = 1 .

Original: Then applying the reasoning from Section 2.2,
Revised: Then applying the reasoning from Section 2.2,

Original: [Equation]
Revised: [Equation]

Original: So we get that E [ f ( s 1 )] − J ( π RCSL f ) = 1 − ϵ , while ϵ/α f = ϵ/ ( ϵ/ 2)
Revised: So we get that E [ f ( s 1 )] − J ( π RCSL f ) = 1 − ϵ , while ϵ/α f = ϵ/ ( ϵ/ 2)

Original: = 2 (which is on the sameorder, up to a constant factor).
Revised: = 2 (which is on the sameorder, up to a constant factor).

Original: However, in this case the learned policy π RCSL f suffers substantialregret since the chosen action a 2 has substantially lower expected value than a 1 by 1 − 2 ϵ .
Revised: However, in this case the learned policy π RCSL f suffers substantialregret since the chosen action a 2 has substantially lower expected value than a 1 by 1 − 2 ϵ .

Original: The issue in the second example could be resolved by changing the conditioning function so thatf ( s 1 ) = 1 − ϵ .
Revised: The issue in the second example could be resolved by changing the conditioning function so thatf ( s 1 ) = 1 − ϵ .

Original: Now we will consider the example in Figure 1c where we will see that there existcases where the bias of RCSL in stochastic environments can remain regardless of the conditioningfunction.
Revised: Now we will consider the example in Figure 1c where we will see that there existcases where the bias of RCSL in stochastic environments can remain regardless of the conditioningfunction.

Original: In this MDP, the only returns that are supported are g = 0 or g = 1 .
Revised: In this MDP, the only returns that are supported are g = 0 or g = 1 .

Original: For f ( s 1 ) = 1 ,plugging in to the formula for π f yields
Revised: For f ( s 1 ) = 1 ,plugging in to the formula for π f yields

Original: [Equation]
Revised: [Equation]

Original: Thus, E [ f ( s 1 )] − J ( π RCSL f ) = 1 / 2 and J ( π ∗ ) − J ( π RCSL f ) = 1 / 2 − ϵ .
Revised: Thus, E [ f ( s 1 )] − J ( π RCSL f ) = 1 / 2 and J ( π ∗ ) − J ( π RCSL f ) = 1 / 2 − ϵ .

Original: This shows that merelychanging the conditioning function is not enough to overcome the bias of the RCSL method instochastic environments.
Revised: This shows that merelychanging the conditioning function is not enough to overcome the bias of the RCSL method instochastic environments.

Original: These examples show that even for MDPs that are ϵ -close to being deterministic, the regret of RCSLcan be large.
Revised: These examples show that even for MDPs that are ϵ -close to being deterministic, the regret of RCSLcan be large.

Original: But, in the special case of deterministic MDPs we find that RCSL can indeed recoverthe optimal policy.
Revised: But, in the special case of deterministic MDPs we find that RCSL can indeed recoverthe optimal policy.

Original: And note that we still allow for stochasticity in the initial states in these deterministic MDPs, which provides a rich setting for problems like robotics that requires generalizationover the state space from finite data.
Revised: And note that we still allow for stochasticity in the initial states in these deterministic MDPs, which provides a rich setting for problems like robotics that requires generalizationover the state space from finite data.

Original: In the next section, we will consider more precisely whathappens to RCSL algorithms with finite data and limited model classes.
Revised: In the next section, we will consider more precisely whathappens to RCSL algorithms with finite data and limited model classes.

Original: Trajectory stitching.
Revised: Trajectory stitching.

Original: Another issue often discussed in the offline RL literature is the idea of trajectory stitching [30, 8].
Revised: Another issue often discussed in the offline RL literature is the idea of trajectory stitching [30, 8].

Original: Ideally, an offline RL agent can take suboptimal trajectories that overlapand stitch them into a better policy.
Revised: Ideally, an offline RL agent can take suboptimal trajectories that overlapand stitch them into a better policy.

Original: Clearly, DP-based algorithms can do this, but it is not so clearthat RCSL algorithms can.
Revised: Clearly, DP-based algorithms can do this, but it is not so clearthat RCSL algorithms can.

Original: In Appendix A we provide theoretical and empirical evidence that in factthey cannot perform stitching in general, even with infinite data.
Revised: In Appendix A we provide theoretical and empirical evidence that in factthey cannot perform stitching in general, even with infinite data.

Original: While this does not directly affectour bounds, the failure to perform stitching is an issue of practical importance for RCSL methods.
Revised: While this does not directly affectour bounds, the failure to perform stitching is an issue of practical importance for RCSL methods.

Original:  Sample complexity of RCSL
Revised:  Sample complexity of RCSL

Original: Now that we have a better sense of what policy RCSL will converge to with infinite data, we canconsider how quickly (and under what conditions) it will converge to the policy π f when given finitedata and a limited policy class, as will occur in practice.
Revised: Now that we have a better sense of what policy RCSL will converge to with infinite data, we canconsider how quickly (and under what conditions) it will converge to the policy π f when given finitedata and a limited policy class, as will occur in practice.

Original: We will do this via a reduction from theregret relative to the infinite data solution π f to the expected loss function L minimized at trainingtime by RCSL, which is encoded in the following theorem.
Revised: We will do this via a reduction from theregret relative to the infinite data solution π f to the expected loss function L minimized at trainingtime by RCSL, which is encoded in the following theorem.

Original: Theorem 2 (Reduction of RCSL to SL) .
Revised: Theorem 2 (Reduction of RCSL to SL) .

Original: Consider any function f : S → R such that the followingtwo assumptions hold:
Revised: Consider any function f : S → R such that the followingtwo assumptions hold:

Original: Bounded occupancy mismatch: P π RCSL f
Revised: Bounded occupancy mismatch: P π RCSL f

Original: ( s ) P [Equation] ( s ) ≤ C f for all s .
Revised: ( s ) P β ( s ) ≤ C f for all s .

Original: Return

Original: P g =

Original: ( s ) s ) α for s .

Original: Define the expected loss as L (ˆ π ) =
Revised: Define the expected loss as L (ˆ π ) =

Original: E s ∼ P
Revised: E s ∼ P

Original: [Equation]
Revised: [Equation]

Original: E g ∼ P [Equation] ( ·| s )
Revised: E g ∼ P β ( ·| s )

Original: [ D KL ( P β ( ·| s, g ) ∥
Revised: [ D KL ( P β ( ·| s, g ) ∥

Original: ˆ π ( ·| s, g ))] .
Revised: ˆ π ( ·| s, g ))] .

Original: Then for anyestimated RCSL policy ˆ π that conditions on f at test time (denoted by ˆ π f ), we have that 
Revised: Then for anyestimated RCSL policy ˆ π that conditions on f at test time (denoted by ˆ π f ), we have that [Equation] RCSL

Original: [Equation]
Revised: [Equation]

Original: The proof can be found in Appendix B.3.
Revised: The proof can be found in Appendix B.3.

Original: Note that we require a similar assumption of returncoverage as before to ensure we have sufficient data to define π f .
Revised: Note that we require a similar assumption of returncoverage as before to ensure we have sufficient data to define π f .

Original: We also require an assumption onthe state occupancy of the idealized policy π f relative to β .
Revised: We also require an assumption onthe state occupancy of the idealized policy π f relative to β .

Original: This assumption is needed since the loss
Revised: This assumption is needed since the loss

Original: L (ˆ π ) is optimized on states sampled from P β , but we care about the expected return of the learnedpolicy relative to that of π f , which can be written as an expectation over states sampled from P π f .
Revised: L (ˆ π ) is optimized on states sampled from P β , but we care about the expected return of the learnedpolicy relative to that of π f , which can be written as an expectation over states sampled from P π f .

Original: This gives us a reduction to supervised learning, but to take this all the way to a sample complexitybound we need to control the loss L (ˆ π ) from finite samples.
Revised: This gives us a reduction to supervised learning, but to take this all the way to a sample complexitybound we need to control the loss L (ˆ π ) from finite samples.

Original: The following corollary achieves thisusing standard uniform convergence results from supervised learning [23].
Revised: Letting N denote the size of the dataset,the following corollary uses standard uniform convergence results from supervised learning [23] toyield finite sample bounds.

Original: Corollary 3 (Sample complexity of RCSL) .
Revised: Corollary 3 (Sample complexity of RCSL) .

Original: To get finite data guarantees, add to the above assumptions the assumptions that (1) the policy class Π is finite, (2) | log π ( a | s, g ) − log π
Revised: To get finite data guarantees, add to the above assumptions the assumptions that (1) the policy class Π is finite, (2) | log π ( a | s, g ) − log π

Original: ( a ′ | s ′ , g ′ )
Revised: ( a ′ | s ′ , g ′ )

Original: | ≤ c forany ( a, s, g, a ′ , s ′ , g ′ ) and all π ∈ Π , and (3) the approximation error of Π is bounded by ϵ approx ,i.e. min π ∈ Π L ( π ) ≤ ϵ approx .
Revised: | ≤ c forany ( a, s, g, a ′ , s ′ , g ′ ) and all π ∈ Π , and (3) the approximation error of Π is bounded by ϵ approx ,i.e. min π ∈ Π L ( π ) ≤ ϵ approx .

Original: Then with probability at least 1 − δ ,
Revised: Then with probability at least 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: The proof is in Appendix B.4.
Revised: The proof is in Appendix B.4.

Original: Analyzing the bound, we can see that the dependence on N is interms of a fourth root rather than the square root, but this comes from the fact that we are optimizinga surrogate loss.
Revised: Analyzing the bound, we can see that the dependence on N is interms of a fourth root rather than the square root, but this comes from the fact that we are optimizinga surrogate loss.

Original: Namely the learner optimizes KL divergence, but we ultimately care about regretwhich we access by using the KL term to bound a TV divergence and thus lose a square root factor.
Revised: Namely the learner optimizes KL divergence, but we ultimately care about regretwhich we access by using the KL term to bound a TV divergence and thus lose a square root factor.

Original: Asimilar rate appears, for example, when bounding 0-1 classification loss of logistic regression [3, 5].
Revised: Asimilar rate appears, for example, when bounding 0-1 classification loss of logistic regression [3, 5].

Original: This corollary also tells us something about how the learner will learn to generalize across differentvalues of the return.
Revised: This corollary also tells us something about how the learner will learn to generalize across differentvalues of the return.

Original: If the policy class is small (for some notion of model complexity) and sufficiently structured, then it can use information from the given data to generalize across values of g ,using low-return trajectories to inform the model on high-return trajectories.
Revised: If the policy class is small (for some notion of model complexity) and sufficiently structured, then it can use information from the given data to generalize across values of g ,using low-return trajectories to inform the model on high-return trajectories.

Original: Note that a full sample complexity bound that competes with the optimal policy can be derived bycombining this result with Corollary 1 as follows:
Revised: Note that a full sample complexity bound that competes with the optimal policy can be derived bycombining this result with Corollary 1 as follows:

Original: Corollary 4 (Sample complexity against the optimal policy) .
Revised: Corollary 4 (Sample complexity against the optimal policy) .

Original: Under all of the assumptions of Corollary 1 and Corollary 3 we get:
Revised: Under all of the assumptions of Corollary 1 and Corollary 3 we get:

Original: [Equation]
Revised: [Equation]

Original: Tightness.
Revised: Tightness.

Original: To better understand why the dependence on 1 /α f is tight and potentially exponentialin the horizon H , even in deterministic environments, we offer the example in Figure 2.
Revised: To better understand why the dependence on 1 /α f is tight and potentially exponentialin the horizon H , even in deterministic environments, we offer the example in Figure 2.

Original: Specifically,we claim that any value of f ( s 1 ) where the policy π RCSL f prefers the good action a 1 from s 1 willrequire on the order of 10 H/ 2 samples in expectation to recover as ˆ π f 2 . will yield a policy substantially worse  behavior.[Equation]
Revised: Specifically,we claim that any value of f ( s 1 ) where the policy π RCSL f prefers the good action a 1 from s 1 willrequire on the order of 10 H/ 2 samples in expectation to recover as ˆ π f 2 .2 Except for f ( s 1 ) = 0 , which will yield a policy substantially worse than the behavior.

Original: To see why this is the case,  consider the MDP illustratedin Figure 2 with horizon H ≫ 4 .
Revised: To see why this is the case, we consider the MDP illustrated in Figure 2 with horizon H ≫ 4 .

Original: The MDP has fourstates each with two actions.
Revised: The MDP hasfour states each with two actions.

Original: All transitions and rewardsare deterministic.
Revised: All transitions and rewards are deterministic.

Original: The only actions with non-zero rewardare r ( s 2 , a 1 ) = 1 and r ( s 3 , a 1 ) = 0 .
Revised: The only actions with non-zeroreward are r ( s 2 , a 1 ) = 1 and r ( s 3 , a 1 ) = 0 .

Original: The interestingdecision is at s 1 where a 1 is better than a 2 .
Revised: The interesting decision is at s 1 where a 1 is better than a 2 .

Original: Note that for any integer 1 ≤ k < H/ 2 , we have that
Revised: Note that for any integer 1 ≤ k < H/ 2 , we have that

Original: P β ( g = k | s 1 , a 2 )
Revised: P β ( g = k | s 1 , a 2 )

Original: = 0 .
Revised: = 0 .

Original: 5 2 k = 0 .
Revised: 5 2 k = 0 .

Original: 5 · (0 . 25) k , while
Revised: 5 · (0 . 25) k , while

Original: P β ( g = k | s 1 , a 1 ) = 0 .
Revised: P β ( g = k | s 1 , a 1 ) = 0 .

Original: 5 · (0 . 1) k .
Revised: 5 · (0 . 1) k .

Original: Conditioning on anysuch k will make us more likely to choose the bad actiona 2 from s 1 .
Revised: Conditioning on anysuch k will make us more likely to choose the bad actiona 2 from s 1 .

Original: The only way to increase the likelihood of thegood action a 1 from s 1 and s 2 is to condition on f ( s 1 ) >
Revised: The only way to increase the likelihood of thegood action a 1 from s 1 and s 2 is to condition on f ( s 1 ) >

Original: H/ 2 .
Revised: H/ 2 .

Original: Unfortunately for RCSL, the probability of observing g > H/ 2 is extremely small, since for any such f wehave P β ( g = f ( s 1 )) ≤ 0 .
Revised: Unfortunately for RCSL, the probability of observing g > H/ 2 is extremely small, since for any such f wehave P β ( g = f ( s 1 )) ≤ 0 .

Original: H/ 2 ≤ 10 − H/ 2 .
Revised: H/ 2 ≤ 10 − H/ 2 .

Original: Thus,both α f and the sample complexity of learning for any f that will yield a policy better than thebehavior is exponential in the horizon H .
Revised: Thus,both α f and the sample complexity of learning for any f that will yield a policy better than thebehavior is exponential in the horizon H .

Original: Fundamentally, the problem here is that RCSL uses trajectory-level information instead of performing dynamic programming on individual transitions.
Revised: Fundamentally, the problem here is that RCSL uses trajectory-level information instead of performing dynamic programming on individual transitions.

Original: But, collecting enough trajectory-levelinformation can take exponentially many samples in the horizon.
Revised: But, collecting enough trajectory-levelinformation can take exponentially many samples in the horizon.

Original: In contrast, DP merely requirescoverage of transitions in the MDP to perform planning and thus avoids this issue of exponentialsample complexity.
Revised: In contrast, DP merely requirescoverage of transitions in the MDP to perform planning and thus avoids this issue of exponentialsample complexity.

Original: In the next section we will delve deeper into this comparison with DP-basedapproaches as well as the simple top-% BC baseline.
Revised: In the next section we will delve deeper into this comparison with DP-basedapproaches as well as the simple top-% BC baseline.

Original:  Comparing RCSL with bounds for alternative methods
Revised:  Comparing RCSL with bounds for alternative methods

Original: Now that we understand the rate at which we expect RCSL to converge, we briefly present the convergence rates of two baseline methods for comparison.
Revised: Now that we understand the rate at which we expect RCSL to converge, we briefly present the convergence rates of two baseline methods for comparison.

Original: In particular, we will leverage an existinganalysis of a DP-based algorithm, and conduct a novel analysis of top-% BC.
Revised: In particular, we will leverage an existinganalysis of a DP-based algorithm, and conduct a novel analysis of top-% BC.

Original: We find that the sample complexity of RCSL has a similar rate to top-% BC, and is worse than DP due to the potentiallyexponential dependence on horizon that stems from return coverage.
Revised: We find that the sample complexity of RCSL has a similar rate to top-% BC, and is worse than DP due to the potentiallyexponential dependence on horizon that stems from return coverage.

Original: 5.1 Comparison to dynamic programming.
Revised: 5.1 Comparison to dynamic programming.

Original: We will compare to the state of the art (to our knowledge) bound for a DP-based offline RL algorithm.
Revised: We will compare to the state of the art (to our knowledge) bound for a DP-based offline RL algorithm.

Original: Namely, we will look at the results of [31] for pessimistic soft policy iteration.
Revised: Namely, we will look at the results of [31] for pessimistic soft policy iteration.

Original: Similarresults exist for slightly different algorithms or assumptions [7, 29], but we choose this one since itis both the tightest and more closely aligns with the practical actor-critic algorithms that we use forour experiments.
Revised: Similarresults exist for slightly different algorithms or assumptions [7, 29], but we choose this one since itis both the tightest and more closely aligns with the practical actor-critic algorithms that we use forour experiments.

Original: Their bound makes the following assumptions about the function class F and thedataset (letting T π represent the Bellman operator for policy π ):
Revised: Their bound makes the following assumptions about the function class F and thedataset (letting T π represent the Bellman operator for policy π ):

Original: Realizability: for any policies π, π ′ there exists f ∈ F with ∥ f − T π f ∥ 22 ,P π ′ ≤ ϵ 1 .2.
Revised: Realizability: for any policies π, π ′ there exists f ∈ F with ∥ f − T π f ∥ 22 ,P π ′ ≤ ϵ 1 .2.

Original: Bellman completeness: for any π and f ∈ F there exists f ′ ∈ F such that ∥ f ′ −T π f ∥ 22 ,P β ≤ ϵ 2 .3.
Revised: Bellman completeness: for any π and f ∈ F there exists f ′ ∈ F such that ∥ f ′ −T π f ∥ 22 ,P β ≤ ϵ 2 .

Original: Coverage:
Revised: Coverage:

Original: P π ∗ ( s,a ) P β ( s,a ) ≤ C for all s, a 3 .
Revised: P π ∗ ( s,a ) P β ( s,a ) ≤ C for all s, a 3 .

Original: With these assumptions in place, the sample complexity bound takes the form 4 :
Revised: With these assumptions in place, the sample complexity bound takes the form 4 :

Original: [Equation]
Revised: [Equation]

Original: Note: this is the result for the “information-theoretic” form of the algorithm that cannot be efficientlyimplemented.
Revised: Note: this is the result for the “information-theoretic” form of the algorithm that cannot be efficientlyimplemented.

Original: The paper also provides a “practical” version of the algorithm for which the bound isthe same except that the the square root in the first term is replaced with a fifth root
Revised: The paper also provides a “practical” version of the algorithm for which the bound isthe same except that the the square root in the first term is replaced with a fifth root.

Original: There are several points of comparison with our analysis (specifically, our Corollary 4).
Revised: There are several points of comparison with our analysis (specifically, our Corollary 4).

Original: The firstthing to note is that for RCSL to compete with the optimal policy, we require nearly deterministicdynamics and a priori knowledge of the optimal conditioning function.
Revised: The firstthing to note is that for RCSL to compete with the optimal policy, we require nearly deterministicdynamics and a priori knowledge of the optimal conditioning function.

Original: These assumptions are notrequired for the DP-based algorithm; this is a critical difference, since it is clear that these conditionsoften do not hold in practice.
Revised: These assumptions are notrequired for the DP-based algorithm; this is a critical difference, since it is clear that these conditionsoften do not hold in practice.

Original: Comparing the coverage assumptions, our C f becomes nearly equivalent to C .
Revised: Comparing the coverage assumptions, our C f becomes nearly equivalent to C .

Original: The major differenceis that our analysis of RCSL also requires dependence on return coverage 1 /α f .
Revised: The major differenceis that our analysis of RCSL also requires dependence on return coverage 1 /α f .

Original: This is problematicsince as seen in Section 4, this return coverage dependence can be exponential in horizon in caseswhere the state coverage does not depend on horizon.
Revised: This is problematicsince as seen in Section 4, this return coverage dependence can be exponential in horizon in caseswhere the state coverage does not depend on horizon.

Original: Comparing the approximation error assumptions, we see that the realizability and completenessassumptions required for DP are substantially less intuitive than the standard supervised learningapproximation error assumption needed for RCSL.
Revised: Comparing the approximation error assumptions, we see that the realizability and completenessassumptions required for DP are substantially less intuitive than the standard supervised learningapproximation error assumption needed for RCSL.

Original: These assumptions are not directly comparable,but intuitively the RCSL approximation error assumption is simpler.
Revised: These assumptions are not directly comparable,but intuitively the RCSL approximation error assumption is simpler.

Original: Finally, dependence on H is the same for both methods and dependence on N depends on whichversion of the DP algorithm we compare to.
Revised: Finally, dependence on H is the same for both methods and dependence on N depends on whichversion of the DP algorithm we compare to.

Original: For the information-theoretic algorithm DP has betterdependence on N , but for the practical algorithm RCSL has better dependence.
Revised: For the information-theoretic algorithm DP has betterdependence on N , but for the practical algorithm RCSL has better dependence.

Original: It is not clearwhether the dependence on N in either the RCSL analysis or in the analysis of the practical algorithmfrom [31] is tight, and it is an interesting direction for future work to resolve this issue.
Revised: It is not clearwhether the dependence on N in either the RCSL analysis or in the analysis of the practical algorithmfrom [31] is tight, and it is an interesting direction for future work to resolve this issue.

Original: 5.2 Comparison to top-% behavior cloning.
Revised: 5.2 Comparison to top-% behavior cloning.

Original: The closest algorithm to RCSL is top-% BC, which was introduced as a baseline for Decision Transformers [8].
Revised: The closest algorithm to RCSL is top-% BC, which was introduced as a baseline for Decision Transformers [8].

Original: This algorithm simply sorts all trajectories in the dataset by return and takes the top ρfraction of trajectories to run behavior cloning (for ρ ∈ [0 , 1] ).
Revised: This algorithm simply sorts all trajectories in the dataset by return and takes the top ρfraction of trajectories to run behavior cloning (for ρ ∈ [0 , 1] ).

Original: The most obvious difference betweenthis algorithm and RCSL is that RCSL allows us to plug in different conditioning functions at testtime to produce different policies, while top-% BC learns only one policy.
Revised: The most obvious difference betweenthis algorithm and RCSL is that RCSL allows us to plug in different conditioning functions at testtime to produce different policies, while top-% BC learns only one policy.

Original: However, if we want toachieve high returns, the two algorithms are quite similar.
Revised: However, if we want toachieve high returns, the two algorithms are quite similar.

Original: The full statements and proofs of our theoretical results for top-% BC are deferred to Appendix B.5.
Revised: The full statements and proofs of our theoretical results for top-% BC are deferred to Appendix B.5.

Original: The results are essentially the same as those for RCSL except for two key modifications:
Revised: The results are essentially the same as those for RCSL except for two key modifications:

Original: Defining coverage.
Revised: Defining coverage.

Original: The first difference in the analysis is the notion of coverage.
Revised: The first difference in the analysis is the notion of coverage.

Original: For RCSL weneeded the return distribution to cover the conditioning function f .
Revised: For RCSL weneeded the return distribution to cover the conditioning function f .

Original: For top-% BC we instead let g ρbe the 1 − ρ quantile of the return distribution over trajectories sampled by the behavior β and thendefine coverage as P β ( g ≥ g ρ | s ) ≥ α ρ for all s .
Revised: For top-% BC we instead let g ρbe the 1 − ρ quantile of the return distribution over trajectories sampled by the behavior β and thendefine coverage as P β ( g ≥ g ρ | s ) ≥ α ρ for all s .

Original: This modification is somewhat minor.
Revised: This modification is somewhat minor.

Original: Sample size and generalization.
Revised: Sample size and generalization.

Original: The main difference between RCSL and top-% BC is that the
Revised: The main difference between RCSL and top-% BC is that the

Original: RCSL algorithm attempts to transfer information gained from low-return trajectories while the top-% BC algorithm simply throws those trajectories away.
Revised: RCSL algorithm attempts to transfer information gained from low-return trajectories while the top-% BC algorithm simply throws those trajectories away.

Original: This shows up in the formal bounds sincefor a dataset of size N the top-% BC algorithm only uses ρ · N samples while RCSL uses all N .
Revised: This shows up in the formal bounds sincefor a dataset of size N the top-% BC algorithm only uses ρ · N samples while RCSL uses all N .

Original: Depending on the data distribution, competing with the optimal policy may require setting ρ veryclose to zero (exponentially small in H ) yielding poor sample complexity.
Revised: Depending on the data distribution, competing with the optimal policy may require setting ρ veryclose to zero (exponentially small in H ) yielding poor sample complexity.

Original: These bounds suggest that RCSL can use generalization across returns to provide improvementsin sample complexity over top-% BC by leveraging all of the data.
Revised: These bounds suggest that RCSL can use generalization across returns to provide improvementsin sample complexity over top-% BC by leveraging all of the data.

Original: However, the RCSL model isattempting to learn a richer class of functions that conditions on reward, which may require a largerpolicy class negating some of this benefit.
Revised: However, the RCSL model isattempting to learn a richer class of functions that conditions on reward, which may require a largerpolicy class negating some of this benefit.

Original: Overall, RCSL should expect to beat top-% BC if thebehavior policy is still providing useful information about how to interact with the environment inlow-return trajectories that top-% BC would throw away.
Revised: Overall, RCSL should expect to beat top-% BC if thebehavior policy is still providing useful information about how to interact with the environment inlow-return trajectories that top-% BC would throw away.

Original:  Experiments
Revised:  Experiments

Original: We have illustrated through theory and some simple examples when we expect RCSL to work, butthe theory does not cover all cases that are relevant for practice.
Revised: We have illustrated through theory and some simple examples when we expect RCSL to work, butthe theory does not cover all cases that are relevant for practice.

Original: In particular, it is not clear howthe neural networks trained in practice can leverage generalization across returns.
Revised: In particular, it is not clear howthe neural networks trained in practice can leverage generalization across returns.

Original: Moreover, oneof the key benefits to RCSL approaches (as compared to DP) is that by avoiding the instabilities ofnon-linear off-policy DP in favor of supervised learning, one might hope that RCSL is more stablein practice.
Revised: Moreover, oneof the key benefits to RCSL approaches (as compared to DP) is that by avoiding the instabilities ofnon-linear off-policy DP in favor of supervised learning, one might hope that RCSL is more stablein practice.

Original: In this section we attempt to test these capabilities first through targeted experiments ina point-mass environment and then by comparisons on standard benchmark data.
Revised: In this section we attempt to test these capabilities first through targeted experiments ina point-mass environment and then by comparisons on standard benchmark data.

Original: Throughout this section we will consider six algorithms, two from each of three categories:
Revised: Throughout this section we will consider six algorithms, two from each of three categories:

Original: Behavior cloning (BC): standard behavior cloning (BC) and percentage behavior cloning(%BC) that runsBC on the trajectories with the highest returns [8].
Revised: Behavior cloning (BC): standard behavior cloning (BC) and percentage behavior cloning (%BC) that runs

Revised: BC on the trajectories with the highest returns [8].

Original: Dynamic programming (DP): TD3+BC [11] a simple DP-based offline RL approach and IQL [20] a more stable DP-based offline RL approach.
Revised: Dynamic programming (DP): TD3+BC [11] a simple DP-based offline RL approach andIQL [20] a more stable DP-based offline RL approach.

Original: Return-conditioned supervised learning (RCSL): RvS [9] an RCSL approach using simpleMLP policies, and DT [8] an RCSL approach using transformer policies.
Revised: Return-conditioned supervised learning (RCSL): RvS [9] an RCSL approach using simpleMLP policies, and DT [8] an RCSL approach using transformer policies.

Original: All algorithms are implemented in JAX [6] using flax [14] and the jaxrl framework [19], except for
Revised: All algorithms are implemented in JAX [6] using flax [14] and the jaxrl framework [19], except for

Original: DT which is taken from the original paper.
Revised: DT which is taken from the original paper.

Original: Full details can be found in Appendix C.
Revised: Full details can be found in Appendix C.

Original: 6.1 Point-mass datasets
Revised: 6.1 Point-mass datasets

Original: First, we use targeted experiments to demonstrate how the tabular failure modes illustrated above canarise even in simple deterministic MDPs that may be encountered in continuous control.
Revised: First, we use targeted experiments to demonstrate how the tabular failure modes illustrated above canarise even in simple deterministic MDPs that may be encountered in continuous control.

Original: Specifically,we will focus on the issue of exponential sample complexity discussed in Section 4.
Revised: Specifically,we will focus on the issue of exponential sample complexity discussed in Section 4.

Original: We build ourdatasets in an environment using the Deepmind control suite [27] and MuJoCo simulator [28].
Revised: We build ourdatasets in an environment using the Deepmind control suite [27] and MuJoCo simulator [28].

Original: Theenvironment consists of a point-mass navigating in a 2-d plane.
Revised: Theenvironment consists of a point-mass navigating in a 2-d plane.

Original: To build an example with exponential sample complexity we construct a navigation task with a goalregion in the center of the environment.
Revised: To build an example with exponential sample complexity we construct a navigation task with a goalregion in the center of the environment.

Original: The dataset is constructed by running a behavior policy thatis a random walk that is biased towards the top right of the environment.
Revised: The dataset is constructed by running a behavior policy thatis a random walk that is biased towards the top right of the environment.

Original: To construct different levelsof reward coverage, we consider the environment and dataset under three different reward functions(ordered by probability of seeing a trajectory with high return, from lowest to highest):
Revised: To construct different levelsof reward coverage, we consider the environment and dataset under three different reward functions(ordered by probability of seeing a trajectory with high return, from lowest to highest):

Original: (a) The “ring of fire” reward.
Revised: (a) The “ring of fire” reward.

Original: This reward is 1 within the goal region, -1 in the ring of fireregion surrounding the goal, and 0 otherwise
Revised: This reward is 1 within the goal region, -1 in the ring of fireregion surrounding the goal, and 0 otherwise

Original: (b) The sparse reward.
Revised: (b) The sparse reward.

Original: This reward is 1 within the goal region and 0 otherwise.
Revised: This reward is 1 within the goal region and 0 otherwise.(c)

Original: (c) The dense reward.
Revised: The dense reward.

Original: This reward function is 1 within the goal region and gradually decayswith the Euclidean distance outside of
Revised: This reward function is 1 within the goal region and gradually decayswith the Euclidean distance outside of

Original: it.
Revised: it.

Original: [Figure]
Revised: [Figure]

Original: Intuitively, the ring of fire reward will cause serious problems for RCSL approaches when combinedwith the random walk behavior policy.
Revised: Intuitively, the ring of fire reward will cause serious problems for RCSL approaches when combinedwith the random walk behavior policy.

Original: The issue is that any random walk which reached the goalregion is highly likely to spend more time in the region of negative rewards than in the actual goalstates, since the ring of fire has larger area than the goal.
Revised: The issue is that any random walk which reached the goalregion is highly likely to spend more time in the region of negative rewards than in the actual goalstates, since the ring of fire has larger area than the goal.

Original: As a result, while they are technicallysupported by the distribution, it is unlikely to find many trajectories (if any at all) with positivereturns in the dataset, let alone near-optimal returns.
Revised: As a result, while they are technicallysupported by the distribution, it is unlikely to find many trajectories (if any at all) with positivereturns in the dataset, let alone near-optimal returns.

Original: As a result, the RCSL-based approaches arenot even able to learn to achieve positive returns, as seen in Figure3.
Revised: As a result, the RCSL-based approaches arenot even able to learn to achieve positive returns, as seen in Figure

Original: The sparse reward is also difficult for the RCSL-based algorithms, for similar reasons; howeverthe problem is less extreme since any trajectory that gets positive reward must go to the goal, so there is signal in the returns indicating where the goal is.
Revised: The sparse reward is also difficult for the RCSL-based algorithms, for similar reasons; howeverthe problem is less extreme since any trajectory that gets positive reward must go to the goal, sothere is signal in the returns indicating where the goal is.

Original: In contrast, the dense reward providesenough signal in the returns that RCSL approaches are able to perform well, although still not aswell as IQL.
Revised: In contrast, the dense reward providesenough signal in the returns that RCSL approaches are able to perform well, although still not aswell as IQL.

Original: It is also worth noting that because the datset still does not have full coverage of thestate-space, simple DP-based algorithms like TD3+BC can struggle with training instability.
Revised: It is also worth noting that because the datset still does not have full coverage of thestate-space, simple DP-based algorithms like TD3+BC can struggle with training instability.

Original: 6.2 Benchmark data
Revised: 6.2 Benchmark data

Original: In addition to our targeted experiments we also ran our candidate algorithms on some datasets fromthe D4RL benchmark [10].
Revised: In addition to our targeted experiments we also ran our candidate algorithms on some datasets fromthe D4RL benchmark [10].

Original: These are meant to provide more realistic and larger-scale data scenarios.
Revised: These are meant to provide more realistic and larger-scale data scenarios.

Original: While this also makes these experiments less targeted, we can still see that the insights that wegained in simpler problems can be useful in these larger settings.
Revised: While this also makes these experiments less targeted, we can still see that the insights that wegained in simpler problems can be useful in these larger settings.

Original: We attempt to choose a subsetof the datasets with very different properties from eachother.
Revised: We attempt to choose a subsetof the datasets with very different properties from eachother.

Original: For example, the play data on theant-maze environment is very diverse and plentiful while the human demonstration data on the penenvironment has poor coverage but high values.
Revised: For example, the play data on theant-maze environment is very diverse and plentiful while the human demonstration data on the penenvironment has poor coverage but high values.

Original: Results are shown in Figure 4.
Revised: Results are shown in Figure 4.

Original: We find that for most of the datasets DP-based algorithms TD3+BC and IQL outperform boththe BC-based algorithms and RCSL-based al- gorithms
Revised: We find that for most of the datasets DP-basedalgorithms TD3+BC and IQL outperform boththe BC-based algorithms and RCSL-based algorithms.

Original: This is especially stark on the
Revised: This is especially stark on the

Original: ANTMAZE datasets where the behavior policy is highly stochastic, requiring the learnerto stitch together trajectories to achieve goodperformance.
Revised: ANTMAZE datasets where the behavior policy is highly stochastic, requiring the learnerto stitch together trajectories to achieve goodperformance.

Original: While none of these tasks hasstochastic dynamics, the issues of return coverage and trajectory stitching persist.
Revised: While none of these tasks hasstochastic dynamics, the issues of return coverage and trajectory stitching persist.

Original: In contrast, RCSL performs well when thebehavior policy is already high quality, butnot optimal (as in the PEN - HUMAN task).
Revised: In contrast, RCSL performs well when thebehavior policy is already high quality, butnot optimal (as in the PEN - HUMAN task).

Original: Since the data is suboptimal and reward isdense, there is opportunity for RCSL to outperform the BC-based methods.
Revised: Since the data is suboptimal and reward isdense, there is opportunity for RCSL to outperform the BC-based methods.

Original: Moreover,since the data has poor coverage, standard DPapproaches like TD3+BC are highly unstable.
Revised: Moreover,since the data has poor coverage, standard DPapproaches like TD3+BC are highly unstable.

Original: IQL is more stable and performs similarly to the RCSL-based algorithms, but is outperformed by
Revised: IQL is more stable and performs similarly to the RCSL-based algorithms, but is outperformed by

Original: DT (perhaps due to the use of history-dependent policies).
Revised: DT (perhaps due to the use of history-dependent policies).

Original:  Discussion
Revised:  Discussion

Original: Looking back at our results, we can better place RCSL in relation to the more classical BC and
Revised: Looking back at our results, we can better place RCSL in relation to the more classical BC and

Original: DP algorithms.
Revised: DP algorithms.

Original: Like BC, RCSL relies on supervised learning and thus inherits its simplicity, elegance, and ease of implementation and debugging.
Revised: Like BC, RCSL relies on supervised learning and thus inherits its simplicity, elegance, and ease of implementation and debugging.

Original: However, it also inherits BC’s dependence on thequality of the behavior policy.
Revised: However, it also inherits BC’s dependence on thequality of the behavior policy.

Original: This dependence can be somewhat reduced in (nearly) deterministicenvironments, where conditioning on high returns can break the bias towards the behavior policy.
Revised: This dependence can be somewhat reduced in (nearly) deterministicenvironments, where conditioning on high returns can break the bias towards the behavior policy.

Original: But, the reliance on trajectory-level information still means that RCSL is fundamentally limited bythe quality of the best trajectories in the dataset, which can require a sample complexity exponentialin horizon in order to compete with the optimal policy, even in deterministic environments.
Revised: But, the reliance on trajectory-level information still means that RCSL is fundamentally limited bythe quality of the best trajectories in the dataset, which can require a sample complexity exponentialin horizon in order to compete with the optimal policy, even in deterministic environments.

Original: In contrast, DP methods are capable of learning good policies even when the dataset does not contain any high-return trajectories and the environment is stochastic.
Revised: In contrast, DP methods are capable of learning good policies even when the dataset does not contain any high-return trajectories and the environment is stochastic.

Original: This represents a fundamentalgap between the two approaches that cannot be bridged within the RCSL paradigm.
Revised: This represents a fundamentalgap between the two approaches that cannot be bridged within the RCSL paradigm.

Original: However, empirically, current deep DP algorithms are not well-behaved.
Revised: However, empirically, current deep DP algorithms are not well-behaved.

Original: These algorithms are often unstable anddifficult to debug, although recent work has started to alleviate these issues somewhat [20].
Revised: These algorithms are often unstable anddifficult to debug, although recent work has started to alleviate these issues somewhat [20].

Original: In sum, for tasks where the requirements for RCSL to perform well are met, it is an excellent practical choice, with great advantages in simplicity over DP.
Revised: In sum, for tasks where the requirements for RCSL to perform well are met, it is an excellent practical choice, with great advantages in simplicity over DP.

Original: Since many real-world tasks of relevancehave these attributes, RCSL techniques could have substantial impact.
Revised: Since many real-world tasks of relevancehave these attributes, RCSL techniques could have substantial impact.

Original: But as a general learningparadigm, RCSL is fundamentally limited in ways that DP is not.
Revised: But as a general learningparadigm, RCSL is fundamentally limited in ways that DP is not.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [Yes] SeeAppendix D

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes] See Appendix C
Revised: [Yes] See Appendix C

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi-
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [Yes]
Revised: [Yes]

Original: (d) Did you include the total amount of compute and the type of resources used (e g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Appendix C
Revised: [Yes] See Appendix C

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] See Appendix C
Revised: [Yes] See Appendix C

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data
Revised: (d) Did you discuss whether and how consent was obtained from people whose data

Original: you’re using/curating?
Revised: you’re using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifi-
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifi-

Original: able information or offensive content?
Revised: able information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]