W HEN S HOULD O FFLINE R EINFORCEMENT L EARNING BE P REFERRED O VER B EHAVIORAL C LONING ?
Anonymous authors
Paper under double-blind review
A BSTRACT
Ofﬂine reinforcement learning (RL) algorithms can acquire effective policies by utilizing only previously collected experience, without any online interaction.
While it is widely understood that ofﬂine RL is able to extract good policies even from highly suboptimal data, in practice ofﬂine RL is often used with data that resembles demonstrations.
In this case, one can also use behavioral cloning (BC) algorithms, which mimic a subset of the dataset via supervised learning.
It seems natural to ask: under what environment and dataset conditions can an ofﬂine
RL method outperform BC with an equal amount of expert data, even when BC is a natural choice?
To answer this question, we characterize the properties of environments that allow ofﬂine RL methods to perform better than BC methods even when only provided with expert data.
Additionally, we show that policies trained on suboptimal data that is sufﬁciently noisy can attain better performance than even BC algorithms with expert data, especially on long-horizon problems.
We validate our theoretical results via extensive experiments on both diagnostic and high-dimensional domains including robot manipulation, maze navigation and
Atari games, when learning from a variety of data sources.
We observe that modern ofﬂine RL methods trained on suboptimal, noisy data in sparse reward domains outperform cloning the expert data in several practical problems.
I NTRODUCTION
Ofﬂine reinforcement learning (RL) algorithms aim to leverage large, existing datasets of previously collected data to produce effective policies that generalize across a wide range of scenarios, without the need for costly active data collection.
Many recent ofﬂine RL algorithms (Fujimoto et al., 2018;
Kumar et al., 2019; Wu et al., 2019; Kumar et al., 2020; Yu et al., 2020; Sinha et al., 2021; Kostrikov et al., 2021) can work well even highly suboptimal data.
With recent advances, the performance of ofﬂine RL algorithms has improved signiﬁcantly, and a number of these approaches have been studied theoretically (Wang et al., 2021; Zanette, 2020; Rashidinejad et al., 2021).
While it is clear that ofﬂine
RL algorithms are a good choice when the available data is either random or highly suboptimal, it is less clear if these methods should even be tried when the datasets consist of demonstration that come from expert or near-expert demonstrations.
In these cases, imitation learning algorithms, such as behavorial cloning (BC), can be used to train policies via supervised learning.
It then seems natural to ask: When should we prefer to use ofﬂine RL over imitation learning?
To our knowledge, there has not been a rigorous characterization of when ofﬂine RL perform better than imitation learning.
Existing empirical studies comparing ofﬂine RL to imitation learning have mixed conclusions.
Some works show that ofﬂine RL methods appear to greatly outperform imitation learning, speciﬁcally in environments that require “stitching” parts of suboptimal trajectories (Fu et al., 2020).
In contrast, a number of recent works have argued that BC performs better than ofﬂine
RL on both expert and suboptimal demonstration data over a variety of tasks (Mandlekar et al., 2021;
Florence et al., 2021; Hahn et al., 2021).
This makes it confusing for practitioners to understand whether to use ofﬂine RL or simply run BC on collected demonstrations.
Thus, in this work we aim to understand if there are conditions on the environment or the dataset under which an ofﬂine RL algorithm might outperform BC for a given task, even when BC is provided with expert data or is allowed to use reward as side information.
Our insights can inform a practitioner with sufﬁcient domain knowledge on whether ofﬂine RL is a good idea, even when expert or near-expert data is available and BC is considered to be a natural choice.
Our contribution in this paper is a theoretical and empirical characterization of certain conditions when ofﬂine RL can outperform BC.
Theoretically, we are the ﬁrst to derive conditions on the environment or ofﬂine dataset where ofﬂine RL achieves better worst-case guarantees than even the best-case lower-bound for BC using expert demonstrations .
These conditions are grounded in practical problems, and provide guidance to the practitioner as to whether they should use RL or
BC.
Concretely, we show that in the case of expert data, the error incurred by ofﬂine RL algorithms can scale signiﬁcantly more favorably when the MDP enjoys some structure including horizonindependent returns ( i.e. , sparse rewards), or a low volume of states where it is “critical” to take the same action as the expert (Section 4.2)
Meanwhile, in the case of sufﬁciently noisy data, we show that ofﬂine RL again enjoys better guarantees on long-horizon tasks (Section 4.3).
Finally, since traditional BC methods ignore rewards, we consider generalized BC methods that use the observed rewards to inform learning, and show that it is still preferable to perform ofﬂine RL (Section 4.4).
Empirically, we validate our theoretical conclusions on diagnostic gridworld domains (Fu et al., 2019) and large-scale benchmark problems in robotic manipulation and navigation and Atari games, using human data (Fu et al., 2020), scripted data (Singh et al., 2020) and data generated from
RL policies (Agarwal et al., 2020b).
We verify that in multiple long-horizon problems where the conditions we propose are likely to be satisﬁed, practical ofﬂine RL methods can outperform BC and generalized BC methods.
Using careful ofﬂine tuning practices, we show that it is possible for ofﬂine
RL to outperform cloning an expert dataset for the same task, given equal amounts of data.
R ELATED W ORK
Ofﬂine RL (Lange et al., 2012; Levine et al., 2020) has shown promise in domains such as robotic manipulation (Kalashnikov et al., 2018b; Mandlekar et al., 2020;
Singh et al., 2020;
Kalashnikov et al., 2021), NLP (Jaques et al., 2020) and healthcare (Shortreed et al., 2011; Wang et al., 2018).
The major challenge in ofﬂine RL is distribution shift (Fujimoto et al., 2018; Kumar et al., 2019), where the learned policy might execute out-of-distribution actions.
Prior ofﬂine RL methods can broadly be characterized into two categories: (1) policy-constraint methods that regularize the learned policy to be “close” to the behavior policy either explicitly (Fujimoto et al., 2018;
Kumar et al., 2019; Liu et al., 2020; Wu et al., 2019; Fujimoto & Gu, 2021) or implicitly (Siegel et al., 2020; Peng et al., 2019; Nair et al., 2020), or via importance sampling (Liu et al., 2019; Swaminathan & Joachims, 2015; Nachum et al., 2019), and (2) conservative methods that learn a lower-bound, or conservative, estimate of return and optimize the policy against it (Kumar et al., 2020;
Kostrikov et al., 2021; Kidambi et al., 2020; Yu et al., 2020; 2021).
Our goal is not to devise a new ofﬂine RL algorithm, but rather to understand when existing ofﬂine RL methods can outperform BC.
When do ofﬂine RL methods outperform BC?
Rashidinejad et al.
(2021) derive a conservative ofﬂine
RL algorithm based on lower-conﬁdence bounds (LCB) that provably outperforms BC in the simpler contextual bandits (CB) setting, but do not extend it to MDPs.
While this CB result signals the possibility that ofﬂine RL can outperform BC in theory, this generalization is not trivial, as RL suffers from compounding errors (Munos, 2003; 2005; Wang et al., 2021).
Laroche et al.
Nadjahi et al.
; Kumar et al.
; Liu et al.
Xie et al.
(2021a) present safe policy improvement bounds expressed as improvements over the behavior policy, which imitation aims to recover, but these bounds do not clearly indicate when ofﬂine RL is better or worse.
Empirically,
Fu et al.
(2020) show that ofﬂine RL considerably outperforms BC for tasks that require “stitching” trajectory segments to devise an optimal policy.
In contrast, Mandlekar et al.
; Brandfonbrener et al.
; Chen et al.
; Hahn et al.
(2021) suggest that BC or ﬁltered BC using the top fraction of the data performs better on other tasks.
While results on Adroit domains in D4RL (Fu et al., 2020) show that ofﬂine RL outperforms BC even on expert data, Florence et al.
(2021) reported superior BC results, making it unclear.
Kurenkov & Kolesnikov (2022) emphasize the importance of the online evaluation budget for ofﬂine RL methods and show that BC is more favorable in a limited budget.
We provide a characterization of certain scenarios where we would expect ofﬂine RL to be better than BC, and empirical results verifying that ofﬂine RL performs well on such problems, spanning robotics, navigation, games (Fu et al., 2020; Singh et al., 2020; Bellemare et al., 2013).
Our theoretical analysis combines tools from a number of prior works.
We analyze the total error incurred by RL via an error propagation analysis (Munos, 2003; 2005; Farahmand et al., 2010; Chen & Jiang, 2019; Xie & Jiang, 2020; Liu et al., 2020), which gives rise to bounds with concentrability coefﬁcients that bound the total distributional shift between the learned policy and the data distribution (Xie & Jiang, 2020; Liu et al., 2020).
We use tools from Ren et al.
(2021), which provide horizon-free bounds for standard (non-conservative) ofﬂine Q-learning but relax their strict coverage assumptions.
While our analysis studies a LCB-style algorithm similar to Rashidinejad et al.
(2021), we modify it to use tighter Bernstein bonuses (Zhang et al., 2021; Agarwal et al., 2020a), which is key to improving its suboptimality guarantee.
Xie et al.
(2021b) consider a similar algorithm with
Bernstein bonuses, but take a different approach to analysis it and use it for policy ﬁnetuning.
P ROBLEM S ETUP AND P RELIMINARIES
The goal in reinforcement learning is to learn a policy π ( ·| s ) that maximizes the expected cumulative discounted reward in a Markov decision process (MDP), which is deﬁned by a tuple ( S , A , P, r, γ ) .
S , A represent state and action spaces, P ( s (cid:48) | s , a ) and r ( s , a ) represent the dynamics and mean reward function, and γ ∈ (0 , 1) represents the discount factor.
The effective horizon of the MDP is given by
H = 1 / (1 − γ ) .
The Q-function, Q π ( s , a ) for a given policy π is equal to the discounted long-term reward attained by executing a at the state s and then following policy π thereafter.
Q π satisﬁes the recursion: ∀ s , a ∈ S×A , Q π ( s , a ) =
r ( s , a )+ γ E s (cid:48) ∼ P ( ·| s , a ) , a (cid:48) ∼ π ( ·| s (cid:48) )
[ Q ( s (cid:48) , a (cid:48) )] .
The value function
V π considers the expectation of the Q-function over the policy V π ( s ) =
E a ∼ π ( ·| s )
[ Q π ( s , a )] .
Meanwhile, the Q-function of the optimal policy, Q ∗ , satisﬁes the recursion: Q ∗ ( s , a ) =
r ( s , a ) + E s (cid:48) ∼ P ( ·| s , a ) [max a (cid:48)
Q ∗ ( s (cid:48) , a (cid:48) )] , and the optimal value function is given by V ∗ ( s )
= max a Q ∗ ( s , a ) .
Finally, the expected cumulative discounted reward is given by J ( π ) =
E s 0 ∼ ρ
[ V π ( s 0 )] .
In ofﬂine RL, we are provided with a dataset D of transitions, D = { ( s i , a i , r i , s (cid:48) i ) }
Ni =1 of size |D| = N .
We assume that the dataset D is generated i.i.d.
from a distribution µ ( s , a ) that speciﬁes the effective behavior policy π β ( a | s ) :
= µ ( s , a ) / (cid:80) a µ ( s , a ) .
Note that holds even if the data itself is generated by running a non-Markovian policy π β (Puterman, 1994).
Let n ( s , a ) be the number of times ( s , a ) appear in D , and (cid:98) P ( ·| s , a ) and (cid:98) r
( s , a ) denote the empirical dynamics and reward distributions in D , which may be different from P and r due to stochasticity.
Following Rashidinejad et al.
(2021), the goal is to minimize the suboptimality of the learned policy (cid:98) π :
[Equation]
Dataset and MDP conditions.
Here we introduce some conditions on the ofﬂine dataset and
MDP structure that we make for our analysis.
The ﬁrst characterizes the distribution shift between the data distribution µ ( s , a ) and the normalized state-action marginal of π ∗ , given by d ∗ ( s , a ) =
(1 − γ ) (cid:80) ∞
t =0 γ t P ( s t = s, a t = a ; π ∗ ) , via a concentrability coefﬁcient C ∗ .
Condition 3.1 (Rashidinejad et al.
(2021), Concentrability of the data distribution) .
Deﬁne C ∗ to be the smallest, ﬁnite constant that satisﬁes: d ∗ ( s , a ) /µ
( s , a ) ≤ C ∗ ∀ s ∈ S , a ∈ A .
Intuitively, the coefﬁcient C ∗ scales with how suboptimal the data µ ( s , a ) is relative to the optimal π ∗ , where C ∗ = 1 corresponds to data from π ∗ .
The next condition we consider is that the discounted return for any trajectory in the MDP is bounded by a constant, which w.l.o.g., we assume to be 1.
Condition 3.2 (Ren et al.
(2021), the value of any trajectory is bounded by 1) .
The inﬁnite-horizon discounted return for any trajectory τ =
( s 0 , a 0 , r 0 , s 1 , · · · ) is bounded as (cid:80) ∞ t =
γ t r t ≤ 1 .
This condition holds in sparse-reward tasks, particularly those where an agent succeeds or fails at its task once per episode.
This is common in domains such as robotics (Singh et al., 2020;
Kalashnikov et al., 2018b) and games (Bellemare et al., 2013), where the agent receives a signal upon succeeding a task or winning.
This condition also appears in prior work deriving suboptimality bounds for RL algorithms (Ren et al., 2021; Zhang et al., 2021).
Notation.
Let n ∧ 1 = max { n, 1 } .
Denote ι = polylog( |S| , H, N ) .
We let ι be a polylogarithmic quantity, changing with context.
For d -dimensional vectors x , y , x ( i ) denotes its i -th entry, and deﬁne V ( x , y )
= (cid:80) i
x
( i ) y
( i ) 2 − ( (cid:80) i
x ( i ) y
( i )) 2 .
T HEORETICAL C OMPARISON OF BC AND O FFLINE RL
In this section, we present performance guarantees for BC and ofﬂine RL, and characterize scenarios where ofﬂine RL algorithms will outperform BC.
We ﬁrst present general upper bounds for both algorithms in Section 4.1, by extending prior work to account for the conditions discussed in Section 3.
Then, we compare the performance of BC and RL when provided with the same data generated by an expert in Section 4.2 and when RL is given noisy, suboptimal data in Section 4.3.
Our goal is to characterize the conditions on the environment and ofﬂine dataset where RL can outperform BC.
Furthermore, we provide intuition for when they are likely to hold in Appendix D.
4. G ENERAL P ERFORMANCE G UARANTEES OF BC AND O FFLINE RL
Our goal is to understand if there exist ofﬂine RL methods can outperform BC for a given task.
As our aim is simply to provide a proof of existence, we analyze representative ofﬂine RL and BC algorithms that achieve optimal suboptimality guarantees.
For brevity, we only consider a conservative ofﬂine
RL algorithm (as deﬁned in Section 2) in the main paper and defer analysis of a policy-constraint method to Appendix C. Both algorithms are detailed in Algorithms 1 and 2.
Guarantees for BC.
For analysis purposes, we consider a BC algorithm that matches the empirical behavior policy on states in the ofﬂine dataset, and takes uniform random actions outside the support of the dataset.
This BC algorithm was also used in prior work (Rajaraman et al., 2020), and is no worse than other schemes for acting at out-of-support states, in general.
Denote the learned BC policy as (cid:98) π β , then ∀ s ∈ D , (cid:98) π β ( a | s ) ← n ( s , a ) /n
( s ) , and ∀ s / ∈ D , (cid:98) π β ( a | s ) ← 1 / |A| .
We adapt the results presented in Rajaraman et al.
(2020) to the setting with Conditions 3.1 and 3.2.
BC can only incur a non-zero asymptotic suboptimality ( i.e. , does not decrease to 0 as N → ∞ ) in scenarios where
C ∗ = 1 , as it aims to match the data distribution µ ( s , a ) , and a non-expert dataset will inhibit the cloned policy from matching the expert π ∗ .
The performance for BC is bounded in Theorem 4.1.
Theorem 4.1 (Performance of BC) .
Under Conditions 3.1 and 3.2, the suboptimality of BC satisﬁes
[Equation]
A proof of Theorem 4.1 is presented in Appendix B.1.
The ﬁrst term is the additional suboptimality incurred due to discrepancy between the behavior and optimal policies.
The second term in this bound is derived by bounding the expected visitation frequency of the learned policy (cid:98)
π β onto states not observed in the dataset.
The analysis is similar to that for existing bounds for imitation learning (Ross & Bagnell, 2010; Rajaraman et al., 2020).
We achieve ˜ O ( H ) suboptimality rather than ˜ O
( H 2 ) due to Condition 3.2, since the worst-case suboptimality of any trajectory is 1 rather than H .
Guarantees for conservative ofﬂine RL.
We consider guarantees for a class of ofﬂine RL algorithms that maintain conservative value estimator such that the estimated value lower-bounds the true one, i.e. , (cid:98) V π ≤ V π for policy π .
Existing ofﬂine RL algorithms achieve this by subtracting a penalty from reward either explicitly (Yu et al., 2020; Kidambi et al., 2020) or implicitly (Kumar et al., 2020).
We only analyze one such algorithm that does the former, but we believe the algorithm can serve as a theoretical model for general conservative ofﬂine RL algorithms, where analyzing similar algorithms can be accomplished using the same outline.
The algorithm we consider is similar in spirit to VI - LCB proposed by Rashidinejad et al.
(2021) that subtracts penalty b ( s , a ) from the reward during value iteration; we consider a different penalty that results in a tighter bound.
The estimated Q-values are obtained by iteratively solving the following Bellman backup:
[Equation]
s
[Equation]
The learned policy is then given by
[Equation]
←
[Equation]
Building on work in online RL (Zhang et al., 2021), our speciﬁc b ( s , a ) is derived using Bernstein’s inequality, and is shown below:
[Equation]
The performance of the learned policy (cid:98)
π ∗ can then be bounded as:
Theorem 4.2 (Performance of conservative ofﬂine RL) .
Under Conditions 3.1 and 3.2, the policy (cid:98) π ∗ found by conservative ofﬂine RL algorithm can satisfy
[Equation]
We defer a proof for Theorem 4.2 to Appendix B.2.
On a high level, we ﬁrst show that our algorithm is always conservative, i.e. , ∀ s , (cid:98) V ( s ) ≤ V (cid:98) π ∗ ( s ) , and then bound the total suboptimality incurred as a result of being conservative.
Novelty :
Our bound in in Theorem 4.2 improves on existing bounds several ways: (1) by considering pessimistic value estimates, we are able to remove the strict coverage assumptions used by Ren et al.
(2021), (2) by using variance recursion techniques on the Bernstein bonuses, with Condition 3.2, we save a O ( H 2 ) factor over VI - LCB in Rashidinejad et al.
(2021), and (3) we shave a |S| factor by introducing s -absorbing MDPs for each state as in Agarwal et al.
(2020a).
4. C OMPARISON U NDER E XPERT D ATA
We ﬁrst compare the performance bounds from Section 4.1 when the ofﬂine dataset is generated from expert demonstrations.
In relation to Condition 3.1, this corresponds to small C ∗ .
Speciﬁcally, we consider C ∗ ∈ [1 , 1+ ˜ O
(1 /N )]
so that the suboptimality of BC in Theorem 4.1 scales as ˜ O ( |S| H/N ) .
In this regime, we perform a nuanced comparison by analyzing speciﬁc scenarios where RL may outperform BC.
We consider the case of C ∗ = 1 and C ∗ = 1 + ˜ O
(1 /N ) separately.
What happens when C ∗ = 1 ?
In this case, we derive a information-theoretic lower-bound of |S| H/N for any ofﬂine algorithm.
Our result in Theorem 4.3 utilizes the analysis of Rajaraman et al.
(2020) but additionally factoring in Condition 3.2.
Theorem 4.3 (Information-theoretic lower-bound for ofﬂine learning with C ∗ = 1 ) .
For any learner (cid:98) π , there exists an MDP M satisfying Assumption 3.2, and a deterministic expert π ∗ , such that the expected suboptimality of the learner is lower-bounded:
[Equation]
The proof of Theorem 4.3 uses the same hard instance from Theorem 6.1 of Rajaraman et al.
(2020), except that one factor of H is dropped due to Condition 3.2.
The other factor of H arises from the performance difference lemma and is retained.
In this case, where BC achieves the lower bound up to logarithmic factors, we argue that we cannot improve over BC.
This is because the suboptimality of BC is entirely due to encountering states that do not appear in the dataset; without additional assumptions on the ability to generalize to unseen states, ofﬂine RL must incur the same suboptimality, as both methods would choose actions uniformly at random.
Practical Insight 4.1.
When no assumptions are made on the environment structure, both ofﬂine RL and BC perform equally poorly with trajectories from an expert demonstrator.
However, we argue that even with expert demonstrations as data, C ∗ = 1 is an unrealistic assumption.
Naively, it seems plausible that the expert who collected the dataset did not perform optimally at every transition; this is often true for humans, or stochastic experts as (cid:15) -greedy of maximumentropy policies.
In addition, a scenario where C ∗ > 1 even though the expert behaves optimally is under distribution shift of the environment.
One practical example of this is when the initial state distribution changes between dataset collection and evaluation ( e.g. , in robotics (Singh et al., 2020), or self-driving (Bojarski et al., 2016)).
Since the normalized state-action marginals d ∗ ( s , a ) , µ ( s , a ) are impacted by ρ ( s ) , this would lead to C ∗ > 1 even when the expert behaves exactly as π ∗ .
What happens when C ∗ = 1 + ˜ O
(1 /N ) ?
Here C ∗ is small enough that BC still achieves the same optimal ˜ O ( |S| H/N ) performance guarantee.
However, there is suboptimality incurred by BC for even states that appear in the dataset due to distribution shift, which allows us to argue about structural properties of MDPs that allow ofﬂine RL to perform better across those states, particularly for problems with large effective horizon H .
We motivate one such structure below.
In several practical problem domains, the return of any trajectory can mostly be explained by the actions taken at a small fraction of states, which we call critical states .
This can occur when there exist a large proportion of states where it is not costly to recover after deviating from the optimal trajectory, or when there exist a large volume of optimal trajectories.
We give two example domains: robotic manipulation and navigation.
In manipulation tasks such as grasping, if the robot is not near the object, it can take many different actions and still pick up the object by the end; this is because unless the object breaks, actions taken by the robot are typically reversible (Kalashnikov et al., 2018a).
It is only at a few “critical states” when the robot grasps the object, where the robot should be careful to not drop and break the object.
In navigation, there may exist multiple paths that end at the same goal, particularly in large, unobstructed areas (Savva et al., 2019).
For example, while navigating through a wide tunnel, the exact direction the agent takes may not matter so much as multiple directions will take the agent through the tunnel, in contrast, there are “critical states” like narrow doorways where taking a speciﬁc action is important.
An illustration is shown in Figure 3.
Domains that do not satisfy this structure include cliffwalk environments, where a incorrect action at any state will cause the agent to fall off the cliff (Schaul et al., 2015).
Formally:
Deﬁnition 4.1 (Critical states) .
A state s is said to be non-critical, i.e., s ∈ S \ C if the advantage of any action a ∈ A is close to 0 under the optimal policy, i.e, | max a (cid:48)
Q ∗ ( s , a (cid:48) )
− Q ∗ ( s , a ) | ≤ ε/H .
Condition 4.1 (Volume of critical states is small.) .
∃ p c ∈ (0 , 1) that satisﬁes: |C| ≤ p c | S | .
We can show that, if the MDP satisﬁes having a small fraction or volume of critical states, then conservative ofﬂine RL enjoys stronger guarantees than BC.
Corollary 4.1 (Performance of conservative ofﬂine RL with critical states) .
Under Conditions 4.1, 3.1 and 3.2, a policy (cid:98) π ∗ found by the conservative ofﬂine RL can satisfy
[Equation]
For ε =
O ( √ H ) , if the environment satisﬁes p c
= O (1 / √ H ) ,
meaning we encounter O ( √ H ) critical states on average in any trajectory, then we achieve better scaling in H with ofﬂine RL than
BC.
Note that BC does not enjoy the beneﬁt of few critical states because it is agnostic to the reward of the environment and therefore limited by the O ( H ) suboptimality of the behavior policy.
Practical Insight 4.2.
Ofﬂine RL is preferred over BC, even with expert or near-expert data, when either the initial state distribution changes during deployment and when the environment has a few “critical” states, i.e., states where precisely taking the expert action is necessary.
4. C OMPARISON U NDER N OISY D ATA
In practice, it is often much more tractable to obtain suboptimal demonstrations rather than expert ones.
From Theorem 4.1, we see that for C ∗ = 1 + Ω(1 / √ N ) , BC will incur suboptimality that is worse asymptotically than ofﬂine RL.
In constrast, from Theorem 4.2, we note that ofﬂine RL does not scale nearly as poorly with increasing C ∗ .
Since ofﬂine RL is not as reliant on the performance of the behavior policy due to explicitly modeling the rewards, we hypothesize that RL can actually beneﬁt from suboptimal data that is well-explored.
In this section, we aim to answer the following question: Can ofﬂine RL with suboptimal data outperform BC with an equal amount of expert data?
We show in Corollary 4.2 that if the suboptimal dataset D satisﬁes an additional coverage condition, then running conservative ofﬂine RL can attain ˜ O ( √ H ) suboptimality in the horizon.
This implies, perhaps surprisingly, that for long horizon tasks, ofﬂine RL using noisy, suboptimal data can outperform even BC on expert data.
Formally, our coverage condition is the following:
Condition 4.2 (Coverage of the optimal policy) .
∃ b ∈ [log H/N, 1) such that µ satisﬁes: ∀ ( s , a ) ∈
S ×
A where d ∗ ( s , a ) ≥ b/H , we have µ ( s , a ) ≥ b .
Intuitively, this means that the data distribution puts sufﬁcient mass on states that have non-negligible density in the optimal policy distribution.
Note that this is a weaker condition than prior works that require (1) full coverage of the state-action space, and (2) enforce a constraint on the empirical state-action visitations (cid:98) µ ( s , a ) instead of µ ( s , a ) (Ren et al., 2021; Zhang et al., 2021).
This condition is reasonable when the dataset is collected by (cid:15) -greedy or a maximum-entropy expert, which is a standard assumption in MaxEnt IRL (Ziebart et al., 2008).
Even if the expert is not noisy, we argue that in several real-world applications, creating noisy-expert data is feasible.
In many robotics applications, it is practical to augment expert demonstrations with counterfactual data by using scripted exploration policies (Kalashnikov et al., 2018a; 2021).
Existing work has also simulated trajectories to increase coverage, as was done in self-driving by perturbing the vehicle location (Bojarski et al., 2016), or in robotics using learned simulators (Rao et al., 2020).
For intuition, we provide an illustrative example of the noisy data that can help ofﬂine RL in Figure 4.
Corollary 4.2 (Performance of conservative ofﬂine RL with noisy data) .
If µ satisﬁes Condition 4.2, and under Conditions 3.1 and 3.2, the policy (cid:98) π ∗ found by conservative ofﬂine RL can satisfy:
[Equation]
For some b , ofﬂine RL can improve upon the lower-bound for BC in Theorem 4.3, e.g, if b =
O ( √ H/N ) , then the bound in Corollary 4.2 has ˜ O ( √ H ) scaling rather than ˜ O
( H ) for BC.
Thus, when the data satisﬁes the practical coverage conditions (more discussion in Appendix D), ofﬂine RL performs better in long-horizon tasks compared to BC with the same amount of expert data.
Practical Insight 4.3.
Ofﬂine RL outperforms BC on expert data on long-horizon tasks, when provided with an equal amount of noisy-expert data.
So, if noisy-expert data is easy to collect, doing so and running ofﬂine RL can be a much better option.
4. C OMPARISON OF G ENERALIZED BC M ETHODS AND O FFLINE RL
So far we have studied scenarios where ofﬂine RL can outperform naive BC.
One might now wonder how ofﬂine RL methods perform relative to generalized BC methods that additionally use reward information to inform learning.
We study two such approaches: (1) ﬁltered BC (Chen et al., 2021), which only ﬁts to the top k -percentage of trajectories in D , measured by the total reward, and (2)
BC with one-step policy improvement (Brandfonbrener et al., 2021), which ﬁts a Q-function for the behavior policy, then uses the values to perform one-step of policy improvement over the behavior policy.
In this section, we aim to answer how these methods perform relative to RL.
Filtered BC.
In expectation, this algorithm uses αN samples of the ofﬂine dataset D for α ∈ [0 , 1] to perform BC on.
This means that the upper bound (Theorem 4.1) will have worse scaling in N .
For C ∗ = 1 , this leads to a strictly worse bound than regular BC.
However, for suboptimal data, the ﬁltering step could decrease C ∗ by ﬁltering out suboptimal trajectories, allowing ﬁltered BC to outperform traditional BC.
Nevertheless, from our analysis in Section 4.3, ofﬂine RL is still preferred to ﬁltered BC because RL can leverage the noisy data and potentially achieve sub-linear suboptimality
O ( √ H ) in the horizon, whereas even ﬁltered BC would always incur O ( H ) suboptimality.
BC with policy improvement.
This algorithm utilizes the entire dataset to estimate the Q-value of the behavior policy, (cid:98) Q (cid:98) π β , and performs one step of policy improvement using the estimated Q-function, typically via an advantage-weighted update: (cid:98) π
[Equation]
=
[Equation]
( s ) .
When would this algorithm perform poorly compared to ofﬂine RL?
Intutively, this would happen when multiple steps of policy improvement are needed to effectively discover high-advantage actions under the behavior policy.
This is the case when the the behavior policy puts low density on high-advantage transitions.
In Theorem 4.4, we show that more than one step of policy improvement can improve the policy under Condition 4.2 for the softmax policy parameterization (Agarwal et al., 2021a).
Theorem 4.4 (One-step is worse than k -step policy improvement) .
Assume that the learned policies are represented via a softmax parameterization (Equation 3, Agarwal et al.
(2021a)).
Let (cid:98) π k denote the policy obtained after k -steps of policy improvement using exponentiated advantage weights.
Then, under Condition 4.2, the performance difference between (cid:98) π k and (cid:98) π 1 is lower-bounded by:
[Equation]
N
[Equation]
A proof of Theorem 4.4 is provided in Appendix B.5.
This result implies that when the average exponentiated empirical advantage 1 /k (cid:80) ki =1 log Z t ( s ) is large enough ( i.e. , ≥ c 0 for some universal constant), which is usually the case when the behavior policy is highly suboptimal, then for k =
O ( H ) , multiple steps of policy improvement will improve performance, i.e. , J ( (cid:98) π k ) − J ( (cid:98) π 1 ) =
˜ O ( H − (cid:112) H/N ) , where the gap increases with a longer horizon.
This is typically the case when the structure of the MDP allow for stitching parts of poor-performing trajectories.
One example is in navigation, where trajectories that fail may still contain segments of a successful trajectory.
Practical Insight
Using multiple policy improvement steps (i.e., full ofﬂine RL) can lead to greatly improved performance on long-horizon tasks, particularly when parts of various trajectories can be concatenated or stitched together to give better performance.
E MPIRICAL E VALUATION OF BC AND O FFLINE RL
Having characterized scenarios where at least some ofﬂine RL algorithms can outperform BC in theory, we now validate our results empirically.
Concretely, we aim to answer the following questions:
(1) Does some ofﬂine RL method trained on expert data outperform BC on expert data in
practice
(2) Can ofﬂine RL trained on noisy data outperform BC on expert data?, and (3)
How
does
full
ofﬂine
RL compare to
the
generalized
BC methods
studied in
Section
We will ﬁrst validate our ﬁndings on a tabular gridworld domain, and then on several high-dimensional ofﬂine RL problems.
Diagnostic experiments in gridworld.
We ﬁrst evaluate tabular versions of the BC and ofﬂine RL methods analyzed in Section 4.1 on sparse-reward 10 × 10 gridworlds environments (Fu et al., 2019).
Complete details about the setup can be found in Appendix E.1.
On a high-level, we consider three different environments, each with varying number of critical states, from “Single Critical” with exactly one, to “Cliffwalk” where every state is critical and veering off yields zero reward.
The methods we consider are: naive BC ( BC ), conservative RL ( RL - C ), policy-constraint RL ( RL - PC ), and generalized BC with one-step and k-step policy improvement ( BC - PI , BC - kPI ).
In the left plot of Figure 1, we show the return (normalized by return of the optimal policy) across all the different environments for optimal data ( C ∗ = 1 ) and data generated from the optimal policy but with a different initial state distribution ( C ∗ > 1 but π β ( ·| s ) =
π ∗ ( ·| s ) ).
As expected from our discussion in Section 4.2, BC performs best under C ∗ = 1 , but RL - C and RL - PC performs much better when C ∗ > 1 ; also BC with one-step policy improvement outperforms naive BC for C ∗ > 1 , but does not beat RL.
In Figure 1 (right), we vary C ∗ by interpolating the dataset with one generated by a random policy, where α is the proportion of random data.
RL performs much better over all
BC methods, when the data supporting our analysis in Section 4.3.
Finally, BC with multiple policy improvement steps performs better than one step when the data is noisy, which validates Theorem 4.4.
[Figure]
Evaluation in high-dimensional tasks.
Next, we turn to deep ofﬂine RL.
We consider a diverse set of domains (shown on the right) and behavior policies that are representative of scenarios where we would decide between ofﬂine RL and BC: multi-stage robotic manipulation tasks from state (Adroit domains from Fu et al.
and image observations (Singh et al., 2020), antmaze navigation (Fu et al., 2020), and 7 Atari games (Agarwal et al., 2020b).
We use the scripted expert provided by Fu et al.
(2020) for antmaze
and the one provided by Singh et al.
(2020) for manipulation, an RL-trained expert for Atari, and human expert for Adroit (Rajeswaran et al., 2018).
In scenarios where suboptimal data is used to train ofﬂine RL, we use failed attempts to solve the task from a noisy expert policy ( i.e. , previous policies in the replay buffer for Atari, and noisy scripted experts for antmaze and manipulation).
All these tasks utilize sparse rewards such that the return of any trajectory is bounded by a constant much smaller than the horizon.
We use CQL (Kumar et al., 2020) as a representative ofﬂine RL method, and utilize Brandfonbrener et al.
(2021) as a representative BC-PI method.
Tuning ofﬂine RL and BC.
Na ¨ ıvely running ofﬂine RL can lead to poor performance, as noted by prior works (Mandlekar et al., 2021; Florence et al., 2021).
This is also true for BC, but, some solutions such as early stopping based on validation losses, can help improve performance.
We claim that a similar tuning strategy is also crucial for ofﬂine RL.
In our experiments we utilize the ofﬂine workﬂow proposed by Kumar et al.
(2021c) to perform policy selection, and address overﬁtting and underﬁtting, purely ofﬂine.
When the Q-values learned by CQL are extremely negative (typically on the Adroit domains), we utilize dropout with probability 0 .
4 on the layers of the Q-function to combat overﬁtting.
On the other hand, when the Q-values exhibit a relatively stable trend ( e.g. , in Antmaze or Atari), we utilize the DR3 regularizer (Kumar et al., 2021a) to increase capacity.
Consistent with prior work, we ﬁnd that na¨ıve ofﬂine RL generally performs worse than BC without ofﬂine tuning, but we ﬁnd that ofﬂine-tuned ofﬂine RL generally outperforms BC.
To make a stronger comparison, we tuned BC using the online rollouts.
We applied regularizers such as dropout on the BC policy to prevent overﬁtting in Adroit, and utilized a larger ResNet (He et al., 2016) architecture for the robotic manipulation tasks and Atari domains.
For BC, we report the performance of the best checkpoint found during training, giving BC an unfair advantage, but we still ﬁnd that ofﬂine-tuned ofﬂine RL performs better.
More details about tuning can be found in Appendix F.
Answers to questions (1) to (3).
For (1) , we run CQL and BC on expert data in each task, and present the comparison in Table 1 and Figure 2.
While na¨ıve CQL performs comparable or worse than BC in this case, after ofﬂine tuning, CQL outperforms BC.
This tuning does not require any
[Table]
tuning.
Also note that ofﬂine RL can improve when provided with diverse initial states in the Antmaze domain.
Additionally, note that ofﬂine-tuned ofﬂine RL outperforms BC signiﬁcantly in the manipulation domains.
[Figure]
additional online rollouts.
Note that while BC performs better or comparable to RL for antmaze (large) with expert data, it performs worse than RL when the data admits a more diverse initial state distribution such that C ∗ (cid:54) = 1 , even though the behavior policy matches the expert. [Figure]
[Table]
For (2) , we compare ofﬂine RL trained on noisyexpert data with BC trained on on an equal amount of expert data, on domains where noisyexpert data is easy to generate: (a) manipulation domains (Table 2) and (b) Atari games (Figure 2).
Observe that CQL outperforms BC and also improves over only using expert data.
The performance gap also increases with H , i.e. , open-grasp ( H = 40 ) vs pick-place-open-grasp ( H = 80 ) vs Atari domains ( H = 27000 ).
This validates that some form of ofﬂine RL with noisy-expert data can outperform BC with expert data, particularly on long-horizon tasks.
Finally, for (3) , we compare CQL to a representative BC-PI method (Brandfonbrener et al., 2021) trained using noisy-expert data on Atari domains, which present multiple stitching opportunities.
The BC-PI method estimates the Q-function of the behavior policy using SARSA and then performs one-step of policy improvement.
The results in Figure 2 support what is predicted by our theoretical results, i.e. , BC-PI still performs signiﬁcantly worse than CQL with noisy-expert data, even though we utilized online rollouts for tuning BC-PI and report the best hyperparameters found.
D ISCUSSION
We sought to understand if ofﬂine RL is at all preferable over running BC, even provided with expert or near-expert data.
While in the worst case, both methods attain similar performance on expert data, additional assumptions on the environment can provide certain ofﬂine RL methods with an advantage.
We also show that running RL on noisy-expert, suboptimal data attains more favorable guarantees compared to running BC on expert data for the same task, using equal amounts of data.
Empirically, we observe that ofﬂine-tuned CQL can outperform BC on various practical problem domains, with different kinds of expert policies.
While our work is an initial step towards understanding when RL presents a favorable approach, there is still plenty of room for further investigation.
Our theoretical analysis can be improved to handle function approximation.
Understanding if ofﬂine RL is preferred over BC for other real-world data distributions is also important.
Finally, our work focuses on analyzing cases where we might expect ofﬂine RL to outperform BC.
An interesting direction is to understand cases where the opposite holds; such analysis would further contribute to this discussion.
R EPRODUCIBILITY S TATEMENT
For theoretical results we provide explanations of all the assumptions and a complete proof of the claims in Appendix B.
We provide complete experimental details regarding the tasks and tuning strategy in the Appendix F. Additionally, we follow the recommendations of Agarwal et al.
(2021b)
for reliable evaluation in deep RL and report the statistical uncertainty in reported results including aggregate performance metrics (Figure 2).
We provide individual scores in Table 3 and we will open-source our code.
A P SEUDOCODE FOR A LGORITHMS
Require: Ofﬂine dataset D , discount factor γ , and conﬁdence level δ 1: Compute n ( s , a ) from D , and estimate (cid:98) r ( s , a ) , (cid:98) P ( s (cid:48) | s , a ) , ∀ ( s , a ) ∈ S ×
A 2: Initialize (cid:98) Q ( s , a ) ← 0 , (cid:98) V ( s ) ← 0 , ∀ ( s , a ) 3: for i = 1 , 2 , . . .
, m do
Calculate b ( s , a ) as: b ( s , a ) ← (cid:115) V ( (cid:98) P ( s , a ) , (cid:98) V ) log( |S||A| m/δ ))
( n ( s , a ) ∧ 1) + (cid:115)(cid:98) r
( s , a ) log( |S||A| m/δ ) ( n ( s , a ) ∧ 1) + log( |S||A| m/δ ) ( n ( s , a ) ∧ 1)
Calculate (cid:98) π ∗ ( s ) as: (cid:98) Q ( s , a ) ← (cid:98) r
( s , a ) − b ( s , a ) + γ (cid:98) P
( s , a )
· (cid:98) V (cid:98) V ( s ) ← max a (cid:98) Q ( s , a ) (cid:98) π ∗ ( s )
← arg max a (cid:98) Q ( s , a ) 4:
Return (cid:98) π ∗
Require: Ofﬂine dataset D , discount factor γ , and threshold b 1: Compute n ( s , a ) from D , and estimate (cid:98) r ( s , a ) , (cid:98) P ( s (cid:48) | s , a ) , (cid:98) µ (
[Table]
B P ROOFS B. P ROOF OF T HEOREM 4.
Let π β be the behavior policy that we ﬁt our learned policy (cid:98) π β to.
Recall that the BC algorithm we analyze ﬁts (cid:98) π β to choose actions according to the empirical dataset distribution for states that appear in dataset D , and uniformly at random otherwise.
We have
[Equation]
The following lemma from Rajaraman et al.
(2020) bounds the suboptimality from performing BC on a (potentially stochastic) expert, which we adapt below factoring in bounded returns of trajectories from Condition 3.2.
Lemma B.1 (Theorem 4.4, Rajaraman et al.
The policy returned by BC on behavior policy π β has expected error bounded as
[Equation]
where π β could be stochastic.
Using Lemma B.1, we have E D [ J ( π β ) − J
( (cid:98) π β )] ≤ SHι/N .
What remains is bounding the suboptimality of the behavior policy, which we can upper-bound as
[Equation]
where we use the deﬁnition of C ∗ in Condition 3.1.
Taking the sum of both terms yields the desired result.
B. P ROOF OF T HEOREM 4.
In this section, we proof the performance guarantee for the conservative ofﬂine RL algorithm detailed in Algorithm 1.
Recall that the algorithm we consider builds upon empirical value iteration but subtracts a penalty during each Q -update.
Speciﬁcally, we initialize Q
( s , a ) = 0 , V 0 ( s ) =
for all ( s , a ) .
Let n ( s , a ) be the number of times ( s , a ) appeared in D , and let (cid:98) r
( s , a ) , (cid:98) P ( s , a ) be the empirical estimates of their reward and transition probabilities.
Then, for each iteration i ∈ [ m ] :
[Equation]
for all s , a ,
[Equation]
for all s ,
In our algorithm we deﬁne the penalty function as
[Equation]
where we let ι to capture all poly-logarithmic terms.
As notation, we drop the subscript i to denote the ﬁnal (cid:98) Q and (cid:98) V at iteration m , where m =
H log N .
Finally, the learned policy (cid:98) π ∗ satisﬁes (cid:98) π ∗ ( s ) ∈ arg max a (cid:98) Q ( s , a ) for all s , if multiple such actions exist, then the policy samples an action uniformly at random.
B.2. T ECHNICAL L EMMAS
Lemma B.2 (Bernstein’s inequality) .
Let X, { X i } ni =1 be i.i.d random variables with values in [0 , 1] , and let δ > 0 .
Then we have
[Equation]
n (cid:33) ≤ δ .
Lemma B.3 (Theorem 4, Maurer & Pontil (2009)) .
Let X, { X i } ni =1 with n ≥ 2 be i.i.d random variables with values in [0 , 1] .
Deﬁne ¯ X = 1 n (cid:80) ni
=1 X i and (cid:100)
Var( X ) = 1 n (cid:80) ni
=
( X i − ¯ X ) 2 .
Let δ > 0 .
Then we have
[Equation]
Lemma B.4 (Lemma 4, Ren et al.
Let λ 1 , λ 2 > 0 be constants.
Let f : Z ≥
→ R be a function such that f ( i ) ≤ H, ∀ i and f ( i ) satisﬁes the recursion
[Equation]
Then, we have that f (0) ≤ 6( λ 1 + λ 2 ) .
B.2. P ESSIMISM G UARANTEE
The ﬁrst thing we want to show is that with high probability, the algorithm provides pessimistic value estimates, namely that (cid:98) V i ( s ) ≤ V ∗ ( s ) for all t ∈
[ T ] and s ∈ S .
To do so, we introduce a notion of a “good” event, which occurs when our empirical estimates of the MDP are not far from the true MDP.
We deﬁne E 1 to be the event where
[Equation]
holds for all i ∈ [ m ] and ( s , a ) ∈ S × A .
We also deﬁne E 2 to be the event where
[Equation]
holds for all ( s , a ) .
We want to show that the good event E
= E 1 ∩ E 2 occurs with high probability.
The proof mostly follows from Bernstein’s inequality in Lemma B.2 .
Note that because (cid:98) P ( s , a ) , (cid:98) V i are not independent, we cannot straightforwardly apply Bernstein’s inequality.
We instead use the approach of Agarwal et al.
(2020a) who, for each state s , partition the range of (cid:98) V i ( s ) within a modiﬁed
s -absorbing MDP
to create independence from (cid:98) P .
The following lemma from Agarwal et al.
(2020a) is a result of such analysis, and is slightly modiﬁed below to account for bounded returns of trajectories
, i.e. , (cid:98) V i ( s ) ≤ 1 :
Lemma B.5 (Lemma 9, Agarwal et al. (2020a)) .
For any iteration t , state-action ( s , a ) ∈ S ×
A such that n ( s , a ) ≥ 1 , and δ > 0 , we have
[Equation]
Using this, we can show that E occurs with high probability:
Lemma B.6.
P ( E ) ≥ 1 − 2 |S||A|
mδ.
Proof.
For each i and ( s , a ) , if n ( s , a ) ≤ 1 , then equation 2 and equation 3 hold trivially.
For n ( s , a ) ≥ 2 , we have from Lemma B.5 that
[Equation]
Similarly, we can use Lemma B.3 to derive
[Equation]
where we use that (cid:100)
Var( (cid:98) r
( s , a )) ≤ (cid:98) r
( s , a ) for [0 , 1] rewards, and with slight abuse of notation, let ι capture all constant factors.
Taking the union bound over all i and ( s , a ) yields the desired result.
Now, we can prove that our value estimates are indeed pessimistic.
Lemma B.7 (Pessimism Guarantee) .
On event E , we have that (cid:98) V i ( s ) ≤ V (cid:98)
π ∗ ( s ) ≤
V ∗ ( s ) for any iteration i ∈ [ m ] and state s ∈ S .
Proof.
We aim to prove the following for any i and s : (cid:98) V i − 1 ( s ) ≤ (cid:98) V i ( s ) ≤ V (cid:98)
π ∗ ( s ) ≤
V ∗ ( s ) .
We prove the claims one by one.
[Equation] [Equation]
s ) : This is directly implied by the monotonic update of our algorithm.
(cid:98) V i ( s ) ≤ V (cid:98)
π ∗ ( s ) :
We will prove this via induction.
We have that this holds for (cid:98) V 0 trivially.
Assume it holds for t − 1 , then we have
[Equation]
where we use that
[Equation]
under event E .
Finally, the claim of V (cid:98)
π ∗ ( s ) ≤
V ∗ ( s ) is trivial, which completes the proof of our pessimism guarantee.
B.2. V ALUE D IFFERENCE L EMMA
Now, we are ready to derive the performance guarantee from Theorem 4.2.
The following lemma is a bound on the estimation error of our pessimistic Q -values.
Lemma B.8.
On event E , the following holds for any i ∈ [ m ] and ( s , a ) ∈ S
× A :
[Equation] [Equation]
where f ( · ; π ) satisﬁes f ( s ; π ) =
(cid:80) a π ( a | s ) f
( s , a ) .
Proof.
We have,
[Equation]
−
[Equation]
≤ γP ( s
[Equation]
≤ γP
[Equation]
The ﬁrst inequality is due by deﬁnition of E and the second is because (cid:98) V t − 1 ≥ max a (cid:98) Q t − 1 ( · , a ) ≥ (cid:98) Q i ( · , π ∗ ) .
By recursively applying Lemma B.8, we can derive the following value difference lemma:
Lemma B.9 (Value Difference Lemma) .
On event E , at any iteration i ∈ [ m ] , we have
[Equation]
where d ∗ t ( s , a ) =
P ( s t = s , a t = a ; π ∗ ) .
Proof.
We have,
[Equation]
where we use Lemma B.7 in the ﬁrst inequality.
As shorthand, let P π ∈ R ( S×A ) × ( S×A ) where
P π ( s , a , s (cid:48) , a (cid:48) )
=
P ( s (cid:48) | s , a ) π ( a (cid:48) | s (cid:48) )
be the transition matrix for policy π .
Now, we can apply
Lemma B.8 recursively to derive
[Equation]
where we use that d ∗ t = ρ π ∗ ( P π ∗ ) t .
This yields the desired result.
Now, we are ready to bound the desired quantity SubOpt ( (cid:98) π ∗ ) =
E D [ J ( π ∗ ) − J ( (cid:98) π ∗ )] .
We have
E D [ J ( π ∗ ) − J (
[Equation]
We bound each term individually.
The ﬁrst is bounded as ∆ 1 ≤ P (cid:0)
¯ E (cid:1) ≤ 2 |S||A| mδ ≤ ιN for choice of δ = 1 2 |S||A| HN .
B.2.
B OUND ON ∆
For the second term, we have
[Equation]
9 N
[Equation]
where we use that ρ ( s ) ≤ Hd ∗ ( s , π ∗ ( s )) , and that max p ∈
[0 , 1] p (1 − p ) N ≤ 4 9 N .
B.2. B OUND ON ∆
What remains is bounding the last term, which we know from Lemma B.9 is bounded by
[Equation]
where we use that γ m ≤ 1 N for m =
H log N .
Recall that b t ( s
[Equation]
a ) is given by
[Equation]
We can bound the summation of each term separately.
For the third term we have,
[Equation]
N .
Here we use Jensen’s inequality and that (1 − γ ) (cid:80) ∞ t =1 γ t d ∗ t ( s , a ) ≤ C ∗ µ ( s , a ) for any ( s , a ) .
For the second term, we similarly have
[Equation]
N , where we use Cauchy-Schwarz
[Equation]
then Condition 3.2 to bound the total estimated reward.
Finally
[Equation]
we consider the ﬁrst term of b t ( s , a )
[Equation]
Similar to what was done in Zhang et al.
Ren et al.
(2021) for ﬁnite-horizon MDPs, we can bound this term using variance recursion for inﬁnite-horizon ones.
Deﬁne
[Equation]
Using Lemma 3 of Ren et al.
(2021) for the inﬁnite-horizon case
[Equation]
we have the following recursion:
[Equation]
where
[Equation]
Using Lemma B
[Equation] [Equation]
we can bound f (0) =
O (cid:16) C ∗ |S| Hι N + Φ
[Equation]
(cid:17) .
Using that for constant c ,
[Equation]
we have that
[Equation]
Substituting this back into the inequality for Φ yields,
[Equation]
Finally, we can bound ∆
[Equation]
Combining the bounds for the three terms yields the desired result.
B. P ROOF OF C OROLLARY 4.
The proof of Corollary 4.1 mostly relies on the existing machinery in Appendix B.2.
In this section, we will only point out the differences and defer details to the comprehensive proof in Appendix B.2.
Recall that C is the set of critical states, and from Deﬁnition 4.1, that all s ∈ S \ C satisfy having negligible advantage, i.e. , Q ∗ ( s , π ∗ ( s )) − Q ∗ ( s , a ) ≤ ε/H for any suboptimal action a .
Recall that for any Q, π , we use the notation Q ( s ; π ) =
E a ∼ π [ Q ( s , a )] .
The main difference between this proof and the one for Theorem 4.2 is that under the critical point structure outlined in Section 4.2, we can handle critical and non-critical states differently to derive a tighter bound.
Namely, we bound the suboptimality using:
[Equation]
where we use the performance difference lemma, then condition on whether a state is critical.
We propose an alternative interpretation for the bound.
The ﬁrst term is the value difference under a modiﬁed MDP where the advantage for all states that are non-critical is 0 .
One way to construct such an MDP is to take each s (cid:54)∈ C and modify the reward and transitions so that r ( s , a ) =
r ( s , π ∗ ( s )) , P ( s , a ) = P ( s , π ∗ ( s )) for all actions.
Let us denote the performance function for this modiﬁed MDP be J C , then we have
[Equation]
The ﬁrst term can be bounded as was done in Appendix B.2.
Namely, we can show the following equivalent of Lemma B.9:
[Equation]
Bounding this for i =
Hι can be done exactly as in was done in Appendix B.2.4 and B.2.5, except with the dependence on all states S replaced by C only the critical ones.
We get the following bound:
[Equation]
Combining the above result with Condition 4.1, which bounds |C| ≤ p c |S| , completes the proof.
B. P ROOF OF C OROLLARY 4.
The proof of Corollary 4.2 is a slight modiﬁcation of the one for Theorem 4.2.
For brevity, we will point out the parts of the proof that change, and simply defer to the proof in Appendix B.2 for parts that are similar.
Recall the decomposition for suboptimality in equation 6, which we restate below:
[Equation] [Equation]
[Table]
is
[Table]
B.4. B OUND ON ∆
The bound for ∆ 2 changes slightly from Appendix B
[Equation]
2.4 due to accounting for the lower-bound on µ ( s , a ) ≥ b ≥ log HN .
We have
[Equation]
N , where we use that ρ ( s ) ≤ Hd ∗ ( s , π ∗ ( s )) , and that
[Equation]
B.4. B OUND ON ∆
Due to the lower bound on µ ( s , a ) ≥ b , we can instead bound,
E D
[Equation]
The analysis for bounding ∆ 3 proceeds exactly as in Appendix B.2.5 but using the new bound.
Namely, we end up with the recursion
[Equation]
where
[Equation]
Using Lemma B.4 and proceeding as in Appendix B.2.5 yields the bound
[Equation]
Combining the new bounds for ∆ 2 , ∆ 3 results in the bound in the Corollary 4.2.
B. P ROOF OF T HEOREM 4.
The proof of Theorem 4.4 builds on analysis by Agarwal et al.
(2021a) that we apply to policies with a softmax parameterization, which we deﬁne below.
Deﬁnition B.1 (Softmax parameterization) .
For a given θ ∈ R |S|×|A| , π θ ( a | s ) =
[Equation]
We consider generalized BC algorithms that perform advantage-weighted policy improvement for k improvement steps.
A BC algorithm with k -step policy improvement is deﬁned as follows:
Deﬁnition B.2 (BC with k -step policy improvement) .
Let (cid:98)
A k ( s , a ) denote the advantage of action a at state s under a given policy (cid:98) π k , where the policy (cid:98) π k ( a | s ) is deﬁned via the recursion:
[Equation]
starting from (cid:98) π 0 ( a | s ) = (cid:98) π β .
Then, BC with k -step policy improvement returns (cid:98) π k .
This advantage weighted update is utilized in practical works such as Brandfonbrener et al.
(2021), which ﬁrst estimates the Q-function of the behavior policy using the ofﬂine dataset, i.e, (cid:98) Q 0 ( s , a ) , and then computes (cid:98) π 1 as the ﬁnal policy returned by the algorithm.
To understand the performance difference between multiple values of k , we ﬁrst utilize essentially Lemma 5 from Agarwal et al.
(2021a), which we present below for completeness:
Lemma B.10 (Lower bound on policy improvement in the empirical MDP, (cid:99) M ) .
The iterates (cid:98) π k generated by k -steps of policy improvement, for any initial state distributions ρ 0 ( s ) satisfy the following lower-bound on improvement:
[Equation]
Proof.
We utilize the performance difference lemma in the empirical MDP to show this:
[Equation]
Finally, note that the ﬁnal term log Z t ( s ) is always positive because of Jensen’s inequality, and the fact that the expected advantage under a given policy is 0 for any MDP.
Utilizing Lemma B.10, we can then lower bound the total improvement of the learned policy in the actual MDP as:
[Equation]
N where the (cid:112) C ∗ Hι/N guarantee for terms (a) and (c) arises under the conditions studied in Section 4.3.
Interpretation of Theorem 4.4.
Theorem 4.4 says that if atleast k many updates can be made to the underlying empirical MDP, (cid:99) M , such that each update is non-trivially lower-bounded, i.e., E s ∼ d (cid:98) πk +1 [log Z k ( s )] ≥ c 0 > 0 , then the performance improvement obtained by k -steps of policy improvement is bounded below by kc 0 /η − O ( (cid:112) H/N ) .
This result indicates that if k = O ( H ) many high advantage policy updates are possible in a given empirical MDP, then the methods with that perform O ( H ) steps of policy improvement will attain higher performance than the counterparts that perform only one update.
This is typically the case in maze navigation-style environments, where O ( H ) many possible highadvantage updates are possible on the empirical MDP, especially by “stitching” parts of suboptimal trajectories to obtain a much better trajectory.
Therefore, we expect that in ofﬂine RL problems where stitching is possible, ofﬂine RL algorithms will attain an improved performance compared to one or a few-steps of policy improvement.
C G UARANTEES FOR P OLICY -C ONSTRAINT O FFLINE RL
In this section, we analyze a policy-constraint ofﬂine algorithm (Levine et al., 2020) that constrains the policy to choose a safe set of actions by explicitly preventing action selection from previously unseen, low-density actions.
The algorithm we consider builds upon the MBS - PI algorithm from
Liu et al.
(2020), which truncates Bellman backups and policy improvement steps from low-density, out-of-support state-action pairs.
The algorithm is described in detail in Algorithm 2, but we provide a summary below.
Let (cid:98) µ
( s , a ) denote the empirical state-action distribution and choose a constant b .
Then, let ζ ( s , a ) = 1 { (cid:98) µ ( s , a ) ≥ b } be the indicator of high-density state-action tuples.
The algorithm we analyze performs the following update until convergence:
[Equation]
In order to derive performance guarantees for this generic policy-constraint algorithm, we deﬁne the notion of a ζ − covered policy following Liu et al.
(2020) in Deﬁnition C.1.
The total occupancy of all out-of-support state-action pairs (i.e., ( s , a ) such that ζ ( s , a ) = 0 ) under a ζ -covered policy is bounded by a small constant U , which depends on the threshold b .
Let π ∗ ζ denote the best performing ζ -covered policy.
Deﬁnition C.1 ( ζ -covered) .
π is called ζ -covered if (cid:80) ( s , a ) (1 − ζ ( s , a ))
d π ( s , a ) ≤ (1 − γ )
U
( b ) .
Equipped with this deﬁnition C.1, Lemma C.1 shows that the total value estimation error of any given ζ − covered policy, π , | J ( π ) − (cid:98) J ζ ( π ) | is upper bounded in expectation over the dataset
Lemma C.1 (Value estimation error of a ζ -covered policy) .
For any given ζ -covered policy π , under
Condition 3.2, the estimation error | J ( π ) −
[Equation]
J ζ ( π ) | is bounded as:
[Equation]
Proof.
To prove this lemma
[Equation]
we consider the following decomposition of the policy performance estimate:
[Equation]
[Table]
[Equation]
[Table]
To bound the inner summation over ( s (cid:48) , a (cid:48) ) in term (a), we can apply Lemma B.5 since (cid:98) P ( s (cid:48) | s , a ) and ζ ( s (cid:48) , a (cid:48) ) are not independent, to obtain a horizon-free bound.
Finally, we use Condition 3.1 to bound the density ratios, in expectation over the randomness in dataset D , identical to the proof for the conservative lower-conﬁdence bound method from before.
Formally, using Lemma B.5, we get, with high probability ≥ 1 − δ :
[Equation]
where we utilized the fact that (cid:98) V πζ ≤ (cid:98) V π ≤ 1 due to Condition 3.2.
For bounding ∆ 2 , we note that this term is bounded by the deﬁnition of ζ -covered policy:
[Equation]
Thus
[Equation]
the overall policy evaluation error is given by:
[Equation]
Equation 12 mimics the Φ term in equation 8 that is bounded in Section B.2.5, with an additional offset U ( b ) .
Hence, we can reuse the same machinery to show the bound in expectation over the randomness in the dataset, which completes the proof.
Using Lemma C.1, we can now that the policy constraint algorithm attains a favorable guarantee when compared to the best policy that is ζ -covered:
Theorem C.
Performance of policy-constraint ofﬂine RL) .
Under Condition 3.2, the policy (cid:98) π ∗ incurs bounded suboptimality against the best ζ -covered policy, with high probability ≥ 1 − δ :
[Equation]
To prove this theorem, we use the result of Lemma C.1 for the ﬁxed policy, that is agnostic of the dataset, and then again use the recursion as before to bound the value of the data-dependent policy.
The latter uses Lemma B.5 and ends up attaining a bound previously found in Appendix B.2.5, which completes the proof of this Theorem.
When the term U ( b ) is small, such that U ( b ) ≤ O ( H 0 .
5 − ε ) for ε > 0 , then we ﬁnd that the guarantee in Theorem C.1 matches that in Theorem 4.2, modulo a term that grows slower in the horizon than the other terms in the bound.
If U ( b ) is indeed small, then all properties that applied to conservative ofﬂine RL shall also follow for policy-constraint algorithms.
Note on the bound.
We conjecture that it is possible to get rid of the U ( b ) term, under certain assumptions on the support indicator ζ ( s , a ) , and by relating the values of ζ ( s , a ) and ζ ( s (cid:48) , a (cid:48) ) , at consecutive state-action tuples.
For example,
if ζ ( s (cid:48) , a (cid:48) ) =
= ⇒ ζ ( s , a ) = 1 , then we can derive a stronger guarantee.
D I NTUITIVE I LLUSTRATIONS OF D ERIVED C ONDITIONS AND P RACTICAL G UIDELINES FOR V ERIFYING T HEM
In this section, we present intuitive illustrations of the various conditions we study and discuss practical guidelines that allow a practitioner to verify whether they are likely to hold for their problem domain.
We focus on Conditions 4.1 and 4.2, as Condition 3.2 is satisﬁed in very common settings such as learning from sparse rewards obtained at the end of an episode, indicating success or failure.
D. C ONDITION 4.
To provide intuition behind when this condition is satisﬁed, we ﬁrst provide an example of a sample navigation domain in Figure 3, to build examples of critical states.
As shown in the ﬁgure, the task is to navigate to the high-reward yellow-colored region.
We wish to understand if states marked as A ,
B and C are critical or not.
The region where the agent can move is very wide in the neighborhood around state A , very narrow around state B and wide again around state C .
In this case, as marked on the ﬁgure if the agent executes actions shown via green arrows at the various states, then it is likely to sill ﬁnish the task, while if it executes the actions shown via red arrows it is likely not on track to reach the high-reward region.
Several actions at states A and C allow the agent to still reach the goal, without crashing into the walls, while only one action at state B allows so, as shown in the ﬁgure.
Thus, state B is a “critical state” as per Deﬁnition 4.1.
But since most of the states in the wide regions of the tunnel are non-critical, this navigation domain satisﬁes Condition 4.1.
D. C ONDITION 4.
Now we discuss the intuition behind why ofﬂine RL run on a form of noisy-expert data can outperform
BC on expert data.
As shown in Figure 4, the task is to navigate from the Start to the Goal .
In this case, when BC is trained using only expert trajectories, and the environment consists of a stochastic dynamics, then running the BC policy during evaluation may diverge to low reward regions as shown in the second row, second column of Figure 4.
On the other hand, if RL is provided with some noisy-expert data that visits states around expert trajectories which might eventually lead to low-rewarding states, then an effective ofﬂine RL method should be able to ﬁgure out how to avoid such states and solve the task successfully.
Condition 4.2 requires that state-action tuples with d ∗ ( s , a ) ≥ b/H under the expert policy have high-enough density µ ( s , a ) ≥ b under the data distribution.
There are two ways to attain this condition: (1) the expert policy already sufﬁciently explores the state ( d ∗ ( s , a ) ≥ b ), for example, when the environment is mostly deterministic or when trajectories are cyclic (e.g., in locomotion tasks discussed below), or, (2) the ofﬂine dataset is noisy such that states that are less-frequently visited by the expert are explored more in the data.
The latter can be satisﬁed when the agent is provided with “negative data”, i.e., failed trajectories, starting from states in an optimal trajectory as shown in Figure 4.
Such counterfactual trajectories allow the agent to observe more outcomes starting from a state in the optimal trajectory.
And running ofﬂine RL on them will enable the agent to learn what not to do, as illustrated in Figure 4.
On the other hand, BC cannot use this negative data.
This condition is satisﬁed in the following practical problems:
• Robotics :
In robotics noisy-expert data may not be directly available via demonstrations but can be obtained by a practitioner by running some limited autonomous data collection using noisy scripted policies (e.g., Kalashnikov et al.
(2018a)
and Kalashnikov et al.
(2021) run partly trained RL policies for noisy collection for running ofﬂine RL).
• Autonomous driving and robotics:
In some domains such as autonomous driving and robotics, rolling out short counterfactual trajectories from states visited in the ofﬂine data, and labeling it with failures is a common strategy (Bojarski et al., 2016;
Rao et al., 2020).
A
practitioner could therefore satisfy this condition by tuning the amount of counterfactual data they augment to training.
• Recommender systems :
In recommender systems, Chen et al.
(2019) found that it was practical to add additional stochasticity while rolling out a policy for data collection which could be controlled, enabling the practitioner to satisfy this condition.
• Trajectories consist of cyclic states (e.g., robot locomotion, character animation in computer graphics):
This condition can be satisﬁed in several applications such as locomotion (e.g., in character animation in computer graphics (Peng et al., 2018), or open-world robot navigation (Shah et al., 2021)) where the states observed by the agent are repeated over the course of the trajectory.
Therefore a state with sufﬁcient density under the optimal policy may appear enough times under µ .
D. P RACTICAL G UIDELINES F OR V ERIFYING T HESE C ONDITIONS
These conditions listed above and discussed in the paper can be difﬁcult to check for in practice; for example, it is non-obvious how to quantitatively compute the volume of critical states, or how wellexplored the dataset is.
However, we believe that a practitioner who has sufﬁcient domain-speciﬁc knowledge has enough intuition to qualitatively reason about whether the conditions hold.
We believe that such practitioners can answer the following questions about their particular problem domain:
• Does there only exist a large fraction of states along a trajectory where either multiple good actions exist, or it is easy to recover from suboptimal actions?
• Is the ofﬂine dataset collected from a noisy-expert policy, and if not, can the dataset be augmented using perturbed or simulated trajectories?
If either of those questions can be answered positively, our theoretical and empirical results show that it is favorable to use ofﬂine RL algorithms, even over collecting expert data and using BC.
At least, ofﬂine RL algorithms should be tried on the problem under consideration.
Hence, the goal of our contributions is not to provide a rigid set of guidelines, but rather provide practical advice to the ML practitioner.
We would like to highlight that such non-rigid guidelines exist in general machine learning, beyond RL.
For example, in supervised learning, tuning the architecture of a deep neural network depends heavily on domain knowledge, and choosing kernel in a kernel machine again depends on the domain.
E E XPERIMENTAL D ETAILS
In this section we provide a detailed description of the various tasks used in this paper, and describe the data collection procedures for various tasks considered.
We discuss the details of our tasks and empirical validation at the following website: https://sites.google.com/view/ shouldirunrlorbc/home .
E. T ABULAR G RIDWORLD D OMAINS
The gridworld domains we consider are described by 10 × 10 grids, with a start and goal state, and walls and lava placed in between.
We consider a sparse reward where the agent earns a reward ofupon reaching the goal state; however, if the agent reaches a lava state, then its reward is 0 for the rest of the trajectory.
The agent is able to move in either of the four direction (or choose to stay still); to introduce stochasticity in the transition dynamics, there is a 10% chance that the agent travels in a different direction than commanded.
The exact three gridworlds we evaluate on vary in the number of critical points encountered per trajectory.
We model critical states as holes in walls through which the agent must pass; if the agent chooses a wrong action at those states, it veers off into a lava state.
The exact three gridworlds we evaluate on are: (a) “Single Critical” with one critical state per trajectory, (b) “Multiple Critical” with three critical states per trajectory, and (c) “Cliffwalk”, where every state is critical (Schaul et al., 2015).
The renderings of each gridworld are in Figure 5.
[Figure]
OBOTIC
[Figure]
ANIPULATION D OMAINS
Overview of domains.
These tasks are taken from Singh et al.
The robotic manipulation simulated domains comprise of a 6-DoF WidowX robot that interacts with objects in the environment.
There are three tasks of interest, all of which involve a drawer and a tray.
The objective of each task is to remove obstructions of the drawer, open the drawer, pick an object and place it in a tray.
The obstructions of the drawer were varied giving rise to three different domains — open-grasp (no obstruction of the drawer), close-open-grasp (an open top drawer obstructs the bottom drawer), pick-place-open-grasp (an object obstructs the bottom drawer).
Reward function.
For all the three tasks considered, a reward of +1 is provided when the robot is successfully able to open the drawer of interest (bottom drawer in close-open-grasp and pick-placeopen-grasp; the only drawer in open-grasp) and is able to grasp the object inside it.
If the robot fails at doing so, it gets no reward.
Dataset composition.
For each task, we collected a dataset comprising of 5000 trajectories.
For our experiments where we utilize expert data, we used the (nearly)-expert scripted policy for collecting trajectories and discarded the ones that failed to succeed.
Thus the expert data attains a 100% success rate on this task.
For our experiments with suboptimal data, which is used to train ofﬂine RL, we ran a noisy version of this near-expert scripted policy and collected 5000 trajectories.
The average success rate in the suboptimal data is around 40-50% in both opening and closing the drawers with, 70% success rate in grasping objects, and a 70% success rate in place those objects at random locations in the workspace.
E. A NT M AZE D OMAINS
Overview of the domain.
This task is based on the antmaze-medium and antmaze-large environments from Fu et al.
The goal in this environment is to train an 8-DoF quadraped ant robot to successfully navigate to a given, pre-specifcied target location in a maze.
We consider two different maze layouts provided by Fu et al.
We believe that this domain is well-suited to test BC and
RL methods in the presence of multiple critical points, and is representative of real-world navigation scenarios.
Scripted policies and datasets.
We utilize the scripted policies provided by Fu et al.
(2020) to generate two kinds of expert datasets: ﬁrst, we generate trajectories that actually traverse the path from a given default start location to the target goal location that we consider for evaluation, and second, we generate trajectories that go from multiple random start positions in the maze to the target goal location in the maze.
The latter has a wider coverage and a different initial state distribution compared to what we will test these algorithms on.
We collected a dataset of 500k transitions, which was used by both BC and ofﬂine RL.
Reward functions.
In this task, we consider a sparse binary reward r ( s , a )
=
+1 , if | s (cid:48) − g | ≤ ε = 0 .
5 and 0 otherwise.
This reward is only provided at the end of a trajectory.
This reward function is identical to the one reported by D4RL (Fu et al., 2020), but the dataset composition in our case comes from an expert policy.
E. A DROIT D OMAINS
Overview of the domain.
The Adroit domains (Rajeswaran et al., 2018; Fu et al., 2020) involve controlling a 24-DoF simulated Shadow Hand robot tasked with hammering a nail (hammer), opening a door (door), twirling a pen (pen) or picking up and moving a ball (relocate).
This domain presents itself with narrow data distributions, and we utilize the demonstrations provided by Rajeswaran et al.
(2018) as our expert dataset for this task.
The environments were instantiated via D4RL, and we utilized the environments marked as: hammer-human- door-human- relocate human evaluation.
Reward functions.
We directly utilize the data from D4RL (Fu et al., 2020) for this task.
However, we modify the reward function to be used for RL.
While the D4RL adroit domains provide a dense reward function, with intermediate bonuses provided for various steps, we train ofﬂine RL using a binary reward function.
To compute this binary reward function, we ﬁrst extract the D4RL dataset for these tasks, and then modify the reward function as follows:
[Equation]
The constant thresholds for various tasks are chosen in a way that only any transition that actually activates the ﬂag goal achieved=
True ﬂag in the D4RL Adroit environments attains a reward +1, while other transitions attain a reward 0.
We also evaluarte the performance of various algorithms on this new sparse reward that we consider for our setting.
E. A TARI D OMAINS
We utilized 7 Atari games which are commonly studied in prior work (Kumar et al., 2020; 2021b):
A STERIX , B REAKOUT , S EAQUEST , P ONG , SpaceInvaders, Q*B ERT , E NDURO for our experiments.
We do not modify the Atari domains, directly utilize the sparse reward for RL training and operate in the stochastic Atari setting with sticky actions for our evaluations.
For our experiments, we extracted datasets of different qualities from the DQN-Replay dataset provided by Agarwal et al.
(2020b).
The
DQN-Replay dataset is stored as 50 buffers consisting of sequentially stored data observed during training of an online DQN agent over the course of training.
Expert data.
To obtain expert data for training BC and RL algorithms, we utilized all the data from buffer with id 49 (i.e., the last buffer stored).
Since each buffer in DQN-Replay consists of 1M transition samples, all algorithms training on expert data learn from 1M samples.
[Table] [Table]
Noisy-expert data.
For obtaining noisy-expert data, analogous to the gridworld domains we study, we mix data from the optimal policy (buffer 49) with an equal amount of random exploration data drawn from the initial replay buffers in DQN replay (buffers 0-5).
i.e. we utilize 0.5M samples form buffer 49 in addition to 0.5M samples sampled uniformly at random from the ﬁrst 5 replay buffers.
F T UNING AND H YPERPARAMETERS
In this section, we discuss our tuning strategy for BC and CQL used in our experiments.
Tuning CQL.
We tuned CQL ofﬂine, using recommendations from prior work (Kumar et al., 2021c).
We used default hyperparameters for the CQL algorithm (Q-function learning rate = 3e-4, policy learning rate = 1e-4), based on prior works that utilize these domains.
Note that prior works do not use the kind of data distributions we use, and our expert datasets can be very different in composition compared to some of the other medium or diverse data used by prior work in these domains.
In particular, with regards to the hyperaprameter α in CQL that trades off conservatism and the TD error objective, we used α = 0 .
1 for all Atari games (following Kumar et al.
(2021b)),
and α = 1 .for
the robotic manipulation domains following (Singh et al., 2020).
For the Antmaze and Adroit domains, we ran CQL training with multiple values of α ∈ { 0 . 01 , 0 .
and then picked the smallest α that did not lead to eventually divergent Q-values (either positively or negatively) with more (1M) gradient steps.
Next, we discuss how we regularized the Q-function training and performed policy selection on the various domains.
• Detecting overﬁtting and underﬁtting :
Following Kumar et al.
(2021c), as a ﬁrst step, we detect whether the run is overﬁtting or underﬁtting, by checking the trend in Q-values.
In our experiments, we found that Q-values learned on Adroit domains exhibited a decreasing trend throughout training, from which we concluded it was overﬁtting.
On the Antmaze and Atari experiments, Q-values continued to increase and eventually stabilized, indicating that the run might be underﬁtting (but not overﬁtting).
• Correcting for overﬁtting and policy selection : As recommended, we applied a capacity decreasing regularizer to correct for overﬁtting, by utilizing dropout on every layer of the Q-function.
We ran with three values of dropout parobability, p ∈ { 0 . 1 , 0 . 2 , 0 . 4 } , and found that 0 .
4 was the most effective in alleviating the monotonically decreasing trend in Q-values, so used that for our results.
Then, we performed policy checkpoint selection by picking the earliest checkpoint that appears after the peak in the Q-values for our evaluation.
• Correcting for underﬁtting:
In the Atari and Antmaze domains, we observed that the Q-values exhibited a stable, convergent trend and did not decrease with more training.
Following Kumar et al.
(2021c)
, we concluded that this resembled underﬁtting and utilized a capacity-increasing regularizer (DR3 regularizer (Kumar et al., 2021a)) for addressing this issue.
We used identical hyperparameter for the multiplier ( β ) on this regularizer term for both Atari and Antmaze, β = 0 .
and did not tune it.
Tuning BC.
In all domains, we tested BC with different network architectures.
On the antmaze domain, we evaluated two feed-forward policy architectures of sizes (256 , 256 , 256) and (256 , 256 , 256 , 256 , 256 , 256) and picked the one that performed best online.
ON Adroit domains, we were not able to get a tanh-Gaussian policy, typically used in continuous control to work well, since it overﬁtted very quickly giving rise to worse-than-random performance and therefore, we switched to utilizing a Gaussian policy network with hidden layer sizes (256 , 256 , 256 , 256) , and a learned, state-dependent standard deviation.
To prevent overﬁtting in BC, we applied a strong dropout regularization of p = 0 .
2 after each layer for Adroit domains.
On Atari and the manipulation domains, we utilized a Resnet architecture borrowed from IMPALA (Espeholt et al., 2018), but
without any layer norm.
More details are at: https://sites.google.com/view/shouldirunrlorbc/home .
Tuning BC-PI.
Our BC-PI method is implemented by training a Q-function via SARSA, i.e.,
Q ( s , a ) ← r ( s , a ) + γQ ( s (cid:48) , a (cid:48) ) , where ( s (cid:48) , a (cid:48) ) is the state-action pair that appears next in the trajectory after ( s , a ) in the dataset using the following Bellman error loss function to train Q θ : [Equation] and then performing advantage-weighted policy extraction: π ( a | s ) ∝ (cid:99) π β ( a | s ) · exp( A ( s , a ) /η ) on the ofﬂine dataset D . The loss function for this policy extraction step, following Peng et al.
(2019) is given by: [Equation] where the value function was given by V ( s ) = (cid:80)
a
[Equation]
π β ( a
[Equation]
| s ) Q
θ ( s , a
[Equation]
) and π β is a learned model of the behavior policy, as done in the implementation of Brandfonbrener et al.
This model of the behavior policy is trained according to the tuning protocol for BC, and is hence well-tuned. What we tuned: We tuned the temperature hyperparameter η using multiple values spanning various levels of magnitude: { 0 . 005 , 0 . 05 , 0 . 1 , 0 . 5 , 1 . 0 , 3 . 0 } and additionally tried two different clippings of the advantage values A ( s , a ) : = Q ( s , a ) − V ( s ) between [ − 10 , 2] and [ − 10 , 4] .
The Q-function architecture is identical to tuned CQL, and the policy and the model of the behavior policy both utilize the architecture used by our BC baseline.
Our observations: We summarize our observations below: • We found that in all the runs the temporal difference (TD) error for SARSA was in the range of [0 . 001 , 0 . 003] , indicating that the SARSA Q-function is well behaved. • The optimal hyperparameters that lead to the highest average performance across all games is η = 0 . 005 , and the advantage clipping between [ − 10 , 2] .
We ﬁnd a huge variation in the performance of a given η , but we used a single hyperparameter η across all games, in accordance with the Atari evaluation protocols (Mnih et al., 2013), and as we did for all other baselines. For example, while on some games such as Qbert, BC-PI improves quite a lot and attains 118% normalized return, on Enduro it attains only 78% and on SpaceInvaders it attains 31.7%. • These results perhaps indicate the need for per-state tuning of η , i.e., utilizing η ( s ) , however, this is not covered in our deﬁnition of BC-PI from Section 4.4, and so we chose to utilize a single η across all states. Additionally, we are unaware of any work that utilizes per-state η ( s ) values for exponentiated-advantage weighted policy extraction.
Finally, we would note that we tuned this implementation to the best of our abilities, within our compute capabilities.
We used online rollouts for tuning BC-PI, and the results shown in Figue 2 is what we obtain.
G R EGARDING O FFLINE RL L OWER B OUNDS
Here we address the connection between recently published lower-bounds for ofﬂine RL by Zanette (2020) and Wang et al.
(2021) and our work.
Zanette (2020) worst-case analysis does address function approximation even under realizability and closure.
However, this analysis concerns policy evaluation and applies to policies that go out of the support of the ofﬂine data.
This does not necessarily prove that any conservative ofﬂine RL algorithm (i.e., one that prevents the learned policy from going out of the support of the ofﬂine dataset) would suffer from this issue during policy learning, just that there exist policies for which evaluation would be exponentially bad.
Ofﬂine RL can exactly prevent the policy from going out of the support of the dataset in Zanette (2020)’s counterexample since the dynamics and reward functions are deterministic.
This policy can still improve over behavior cloning by stitching overlapping trajectories, similar to the discussion of one-step vs multi-step PI in Theorem 4.4.
Thus, while the lower-bound applies to off-policy evaluation of some target policies, it doesn’t apply to evaluation of every policy in the
MDP, and speciﬁcally does not apply to in-support policies, which modern ofﬂine RL algorithms produce, in theory and practice.
Certainly, their lower bound would apply to an algorithm aiming to estimate Q ∗ from the ofﬂine data, but that’s not the goal of pessimistic algorithms that we study.
Finally, our practical results indicates that ofﬂine RL methods can be made to perform well in practice despite the possibility of any such lower bound.
The lower bound in Wang et al.
(2020) only applies to non-pessimistic algorithms that do not tackle distributional shift.
In fact, Section 5, Theorem 5.1 in Wang et al.
(2021) provides an upper bound for the performance of ofﬂine policy evaluation with low distribution shift.
The algorithms analyzed in our paper are pessimistic and guarantee low distribution shift between the learned policy and the dataset, and so the lower bound does not apply to the algorithms we analyze.