Hybrid RL: Using both offline and online data can make RL efficient
Abstract
We consider a hybrid reinforcement learning setting (Hybrid RL), in which an agent has access to an offline dataset and the ability to collect experience via real-world online interaction.
The framework mitigates the challenges that arise in both pure offline and online RL settings, allowing for the design of simple and highly effective algorithms, in both theory and practice.
We demonstrate these advantages by adapting the classical Q learning/iteration algorithm to the hybrid setting, which we call Hybrid Q-Learning or Hy-Q.
In our theoretical results, we prove that the algorithm is both computationally and statistically efficient whenever the offline dataset supports a high-quality policy and the environment has bounded bilinear rank.
Notably, we require no assumptions on the coverage provided by the initial distribution, in contrast with guarantees for policy gradient/iteration methods.
In our experimental results, we show that Hy-Q with neural network function approximation outperforms state-of-the-art online, offline, and hybrid RL baselines on challenging benchmarks, including Montezuma’s Revenge.
Introduction
Learning by interacting with an environment, in the standard online reinforcement learning (RL) protocol, has led to impressive results across a number of domains.
State-of-the-art RL algorithms are quite general, employing function approximation to scale to complex environments with minimal domain expertise and inductive bias.
However, online RL agents are also notoriously sample inefficient, often requiring billions of environment interactions to achieve suitable performance.
This issue is particularly salient when the environment requires sophisticated exploration and a high quality reset distribution is unavailable to help overcome the exploration challenge. As a consequence, the practical success of online RL and related policy gradient/improvement methods has been largely restricted to settings where a high quality simulator is available. To overcome the issue of sample inefficiency, attention has turned to the offline RL setting (Levine et al., 2020), where, rather than interacting with the environment, the agent trains on a large dataset of experience collected in some other manner (e.g., by a system running in production or an expert). While these methods still require a large dataset, they mitigate the sample complexity concerns of online RL, since the dataset can be collected without compromising system performance.
However, offline RL methods can suffer from distribution shift , where the state distribution induced by the learned policy differs significantly from the offline distribution (Wang et al., 2021).
Existing provable approaches for addressing distribution shift are computationally intractable, while empirical approaches rely on heuristics that can be sensitive to the domain and offline dataset (as we will see).
In this paper, we focus on a hybrid reinforcement learning setting, which we call Hybrid RL, that draws on the favorable properties of both offline and online settings.
In Hybrid RL, the agent has both an offline dataset and the ability to interact with the environment, as in the traditional online RL setting.
The offline dataset helps address the exploration challenge, allowing us to greatly reduce the number of interactions required.
Simultaneously, we can identify and correct distribution shift issues via online interaction.
Variants of the setting have been studied in a number of empirical works (Rajeswaran et al., 2017; Hester et al., 2018; Nair et al., 2018, 2020; Vecerik et al., 2017) which mainly focus on using expert demonstrations as offline data.
Our algorithmic development is closely related to these works, although our focus is on formalizing the hybrid setting and establishing theoretical guarantees against more general offline datasets.
Hybrid RL is closely related to the reset setting , where the agent can interact with the environment starting from a “nice” distribution.
A number of simple and effective algorithms, including CPI (Kakade & Langford, 2002), PSDP (Bagnell et al., 2003), and policy gradient methods (Kakade, 2001;
Agarwal et al., 2020b)—which have further inspired deep RL methods such as TRPO (Schulman et al., 2015) and PPO (Schulman et al., 2017)—are provably efficient in the reset setting. Yet, a nice reset distribution is a strong requirement (often tantamount to having access to a detailed simulation) and unlikely to be available in real world applications. Hybrid RL differs from the reset setting in that (a) we have an offline dataset, but (b) our online interactions start from the initial distribution of the environment, which is not assumed to have any nice properties. Both features (offline data and a nice reset distribution) facilitate algorithm design by de-emphasizing the exploration challenge.
However,
Hybrid RL is much more practical since an offline dataset is much easier to obtain in practice.
We showcase the Hybrid RL setting with a new algorithm, Hybrid Q learning or Hy-Q (pronounced: Haiku).
The algorithm is a simple adaptation of the classical fitted Q-iteration algorithm (FQI) and accommodates value-based function approximation.
For our theoretical results, we prove that Hy-Q is both statistically and computationally efficient assuming that: (1) the offline distribution covers some high quality policy, (2) the MDP has low bilinear rank, (3) the function approximator is Bellman complete, and (4) we have a least squares regression oracle.
The first three assumptions are standard statistical assumptions in the RL literature while the fourth is a widely used computational abstraction for supervised learning.
No computationally efficient algorithms are known under these assumptions in pure offline or pure online settings, which highlights the advantages of the hybrid setting.
We also implement Hy-Q and evaluate it on two challenging RL benchmarks: a rich observation combination lock (Misra et al., 2020) and Montezuma’s Revenge from the Arcade Learning Environment (Bellemare et al., 2013).
Starting with an offline dataset that contains some transitions from a high quality policy, our approach outperforms: an online RL baseline with theoretical guarantees, an online deep RL baseline tuned for Montezuma’s Revenge, pure offline RL baselines, imitation learning baselines, and existing hybrid methods.
Compared to the online methods, Hy-Q requires only a small fraction of the online experience, demonstrating its sample efficiency.
Compared to the offline and hybrid methods, Hy-Q performs most favorably when the offline dataset also contains many interactions from low quality policies, demonstrating its robustness.
These results reveal the significant benefits that can be realized by combining offline and online data.  Related Works
We discuss related works from four categories: pure online RL, online RL with access to a reset distribution, offline RL, and prior work in hybrid settings. We note that pure online RL refers to the setting where one can only reset the system to initial state distribution of the environment, which is not assumed to provide any form of coverage. Pure online RL Beyond tabular settings, many existing statistically efficient RL algorithms are not computationally tractable, due to the difficulty of implementing optimism. This is true in the linear
MDP (Jin et al., 2020) with large action spaces, the linear Bellman complete model (Zanette et al., 2020; Agarwal et al., 2019), and in the general function approximation setting (Jiang et al., 2017; Sun et al., 2019; Du et al., 2021; Jin et al., 2021a). These computational challenges have inspired results on intractability of aspects of online RL (Dann et al., 2018; Kane et al., 2022).
There are several online RL algorithms that aim to tackle the computational issue via stronger structural assumptions and supervised learning-style computational oracles (Misra et al., 2020; Zhang et al., 2022c; Agarwal et al., 2020a; Uehara et al., 2021;
Modi et al., 2021; Zhang et al., 2022a; Qiu et al., 2022).
Compared to these oracle-based methods, our approach operates in the more general “bilinear rank” setting and relies on a standard supervised learning primitive: least squares regression.
Notably, our oracle admits efficient implementation with linear function approximation, so we obtain an end-to-end computational guarantee; this is not true for prior oracle-based methods.
There are many deep RL methods for the online setting (e.g., Schulman et al.
; Lillicrap et al.
(2016); Haarnoja et al.
; Schrittwieser et al.
Apart from a few exceptions (e.g.,
Burda et al.
; Badia et al.
Guo et al.
, most rely on random exploration and are not capable of strategic exploration.
In our experiments, we test our approach on Montezuma’s Revenge, and we pick R ND (Burda et al., 2018) as a deep RL exploration baseline due to its effectiveness.
Online RL with reset distributions When an exploratory reset distribution is available, a number of statistically and computationally efficient algorithms are known.
The classic algorithms are C PI (Kakade & Langford, 2002), P SDP (Bagnell et al., 2003), Natural Policy Gradient (Kakade, 2001;
Agarwal et al., 2020b), and P OLY T EX (Abbasi-Yadkori et al., 2019).
Uchendu et al.
(2022) recently demonstrated that algorithms like P SDP work well when equipped with modern neural network function approximators.
However, these algorithms (and their analyses) heavily rely on the reset distribution to mitigate the exploration challenge, but such a reset distribution is typically unavailable in practice, unless one also has a simulator.
In contrast, we assume the offline data covers some high quality policy, which helps with exploration, but we do not require an exploratory reset distribution.
This makes the hybrid setting much more practically appealing.
Offline RL Offline RL methods learn policies solely from a given offline dataset, with no interaction whatsoever.
When the dataset has global coverage, algorithms such as FQI (Munos & Szepesv ´ ari, 2008; Chen & Jiang, 2019) or certainty-equivalence model learning (Ross & Bagnell, 2012), can find near-optimal policies in an oracle-efficient manner, via least squares or model-fitting oracles.
However, with only partial coverage, existing methods either (a) are not computationally efficient due to the difficulty of implementing pessimism both in linear settings with large action spaces (Jin et al., 2021b;
Zhang et al., 2022b; Chang et al., 2021) and general function approximation settings (Uehara & Sun, 2021;
Xie et al., 2021a; Jiang & Huang, 2020; Chen & Jiang, 2022; Zhan et al., 2022), or (b) require strong representation conditions such as policy-based Bellman completeness (Xie et al., 2021a; Zanette et al., 2021).
In contrast, in the hybrid setting, we obtain an efficient algorithm under the more natural condition of completeness w.r.t.
, the Bellman optimality operator only.
Among the many empirical offline RL methods (e.g., Kumar et al.
Yu et al.
; Kostrikov et al.
(2021); Fujimoto & Gu (2021))
, we use C QL (Kumar et al., 2020) as a baseline in our experiments, since it has been shown to work in image-based control settings such as Atari games.
Online RL with offline datasets Ross & Bagnell (2012) developed a model-based algorithm for a similar hybrid setting.
In comparison, our approach is model-free and consequently more suitable for high-dimensional state spaces (e.g., raw-pixel images).
Xie et al.
(2021b) studied hybrid RL and show that offline data does not yield statistical improvements in tabular MDPs.
Our work instead focuses on the function approximation setting and demonstrates computational benefits of hybrid RL.
On the empirical side, several works consider combining offline expert demonstrations with online interaction (Rajeswaran et al., 2017; Hester et al., 2018; Nair et al., 2018, 2020; Vecerik et al., 2017).
A common challenge in offline RL is the robustness against low-quality offline dataset.
Previous works mostly focus on expert demonstrations and have no rigorous guarantees for such robustness.
In fact, Nair et al.
(2020) showed that such degradation in performance indeed happens in practice with low-quality offline data.
In our experiments, we observe that DQfD (Hester et al., 2018) also has a similar degradation.
On the other hand, our algorithm is robust to the quality of the offline data.
Note that the core idea of our algorithm is similar to that of Vecerik et al. (2017), who adapt DDPG to the setting of combining RL with expert demonstrations for continuous control. Although
Vecerik et al.
(2017) does not provide any theoretical results, it may be possible to combine our theoretical insights with existing analyses for policy gradient methods to establish some guarantees of the algorithm from Vecerik et al.
(2017) for the hybrid RL setting. We also include a detailed comparison with previous empirical work in Appendix D.  Preliminaries We consider finite horizon Markov Decision Process M ( S , A , H, R, P, d 0 ) , where S is the state space, A is the action space, H denotes the horizon, stochastic rewards R ( s, a ) ∈ ∆([0 , 1]) and
Require:
Value class: F , #iterations: T , offline dataset D νh of size m off = HT/d for h ∈ [ H − 1] . 1: Initialize f 1 h ( s, a ) = 0 .
2: for t = 1 , . . .
, T do 3:
Let π t be the greedy policy w.r.t.
f t i.e., π th ( s ) = argmax a f th ( s, a ) .
For each h , collect m on = H 2 online tuples
D th ∼ d π t h . //
Online collection //
FQI using both online and offline data 5:
Set f t +1 H ( s, a ) = 0 .
6: for h
= H − 1 , . .
. , 0 do 7:
Estimate f t +1 h using least squares regression on the aggregated data: f t +1 h ← argmin f ∈F h (cid:40) (cid:98) E D νh ( f ( s, a ) − r − max a ′ f t +1 h
+1 ( s ′ , a ′ ))
2 + t (cid:88)
τ
=1 (cid:98)
E D τh ( f ( s, a ) − r − max a ′ f t +1 h +1 ( s ′ , a ′ )) (cid:41) (1) 8: end for 9: end for P ( s, a ) ∈ ∆( S ) are the reward and transition distributions at ( s, a ) , and d 0 ∈ ∆( S ) is the initial distribution.
We assume the agent can only reset from d 0 (at the beginning of each episode).
Since the optimal policy is non-stationary in this setting, we define a policy π :=
{ π 0 , . .
, π H − 1 } where π h : S (cid:55)→ ∆( A ) .
Given π , d πh ∈ ∆( S ×A ) denotes the state-action occupancy induced by π at step h .
Given π , we define the state and state-action value functions in the usual manner: V πh ( s ) =
E [ (cid:80) H − 1 τ = h r τ | π, s h
= s ] and Q πh ( s, a ) =
E [ (cid:80) H − 1 τ = h r τ | π, s h
= s, a h
= a ] .
Q ⋆ and V ⋆ denote the optimal value functions.
We define the Bellman operator T such that for any f
: S × A (cid:55)→ R ,
[Equation]
We assume that for each h we have an offline dataset of m off samples ( s, a, r, s ′ ) drawn iid via ( s, a ) ∼ ν h , r ∈ R ( s, a ) , s ′ ∼ P ( s, a ) .
Here ν
= { ν 0 , . .
, ν H − 1 } denote the corresponding offline data distributions.
For a dataset D , we use ˆ E D [ · ] to denote a sample average over this dataset.
For our theoretical results, we will assume that ν covers some high-quality policy.
We consider the value-based function approximation setting, where we are given a function class
F = F 0 ×· · ·×F H − 1 with F h ⊂ S ×A (cid:55)→ [0 , V max ] that we use to approximate the value functions for the underlying MDP.
For ease of notation, we define f = { f 0 , . . .
, f H − 1 } and define π f to be the greedy policy w.r.t.
, f , which chooses actions as π fh ( s ) = argmax a f h ( s, a ) .
Hybrid Q-Learning
In this section, we present our algorithm Hybrid Q Learning – Hy-Q in Algorithm 1.
Hy-Q takes an offline dataset D ν that contains ( s, a, r, s ′ ) tuples and a Q function class F ⊂ S × A (cid:55)→ [0 , H ] as inputs, and outputs a policy that optimizes the given reward function.
The algorithm is conceptually simple: it iteratively executes the Fitted Q Iteration procedure (line 6) using the offline dataset and on-policy samples generated by the learned policies.
Specifically, at iteration t , we have an estimate f t of the Q ⋆ function and we set π t to be the greedy policy for f t .
We execute π t to collect a dataset D th of online samples in line 4.
Then we run FQI, a dynamic programming style algorithm on both the offline dataset D ν and all previously collected online samples {D τh } tτ =1 .
The FQI update works backward from time step H to 0 and computes f t +1 h via least squares regression with input ( s, a ) and regression target r + max a ′ f t +1 h
+
( s ′ , a ′ ) .
Let us make several remarks.
Intuitively, the FQI updates in Hy-Q try to ensure that the estimate f t has small Bellman error under both the offline distribution ν and the online distributions d π t .
The standard offline version of FQI ensures the former, but this alone is insufficient when the offline dataset has poor coverage.
Indeed FQI may have poor performance in such cases (see examples in
Zhan et al., 2022; Chen & Jiang, 2022).
The key insight in Hy-Q is to use online interaction to ensure that we also have small Bellman error on d π t .
As we will see, the moment we find an f t that has small Bellman error on the offline distribution ν and its own greedy policy’s distribution d π t , FQI guarantees that π t will be at least as good as any policy covered by ν .
This observation results in an explore-or-terminate phenomenon: either f t has small Bellman error on its distribution and we are done, or d π t must be significantly different from distributions we have seen previously and we make progress.
Crucially, no explicit exploration is required for this argument, which is precisely how we avoid the computational difficulties with implementing optimism.
Another important point pertains to catastrophic forgetting .
We will see that the size of the offline dataset m off should be comparable to the total amount of online data {D τh } Tτ =1 , so that the two terms in Eq.
1 have similar weight and we ensure low Bellman error on ν throughout the learning process.
In practice, we implement this by having all model updates use a fixed proportion of offline samples even as we collect more online data, so that we do not “forget” the distribution ν .
This is quite different from warm-starting with D ν
and then switching to online RL, which may result in catastrophic forgetting due to a vanishing proportion of offline samples being used for model training as we collect more online samples.
We note that this balancing scheme is analogous to and inspired by the one used by Ross & Bagnell (2012) in the context of model-based RL with a reset distribution.
As in their work, a key practical insight from our analysis is that the offline data should be used throughout training to avoid catastrophic forgetting.
Theoretical Analysis: Low Bilinear Rank Models
In this section we present the main theoretical guarantees for Hy-Q.
We start by stating the main assumptions and definitions for the function approximator, the offline data distribution, and the MDP.
We state the key definitions and then provide some discussion.
Assumption 1 (Realizability and Bellman completeness) .
For any h , we have Q ⋆h ∈ F h .
Additionally, for any f h +1 ∈ F h +1 , we have T f h +1 ∈ F h .
Definition 1 (Bellman error transfer coefficient) .
[-T f h +
-]
For any policy π , define the transfer coefficient as
[Equation]
Definition 2 (Bilinear model (Du et al., 2021)) .
We say that the MDP together with the function class F is a bilinear model of rank d if for any h ∈ [ H − 1] , there exist two (unknown) mappings X h , W h : F (cid:55)→ R d with max f ∥ X h ( f ) ∥ 2 ≤ B X and max f ∥ W h ( f ) ∥ 2 ≤ B W such that:
[Equation]
All concepts defined above are frequently used in the statistical analysis of RL methods with function approximation.
Realizability is the most basic function approximation assumption, but is known to be insufficient for offline RL (Foster et al., 2021) unless other strong assumptions hold (Xie & Jiang, 2021; Zhan et al., 2022; Chen & Jiang, 2022).
Completeness is the most standard strengthening of realizability that is used routinely in both online (Jin et al., 2021a) and offline RL (Munos & Szepesv ´ ari, 2008; Chen & Jiang, 2019) and is known to hold in several settings including the linear
MDP and the linear quadratic regulator.
These assumptions ensure that the dynamic programming updates of FQI are stable in the presence of function approximation.
The transfer coefficient definition above is somewhat non-standard, but is actually weaker than related notions used in prior offline RL results.
First, the average Bellman error appearing in the numerator is weaker than the squared Bellman error notion of (Xie et al., 2021a); a simple calculation shows that C 2 π is upper bounded by their coefficient.
Second, by using Bellman errors, both of these are bounded by notions involving density ratios (Kakade & Langford, 2002; Munos & Szepesv´ari, 2008;
Chen & Jiang, 2019).
Finally, many works, particularly those that do not employ pessimism (Munos & Szepesv´ari, 2008; Chen & Jiang, 2019), require “all-policy” analogs, which places a much stronger requirement on the offline data distribution ν .
In contrast, we will only ask that C π is small for some high-quality policy that we hope to compete with (see Appendix A.5 for more details).
Lastly, the bilinear model was developed in a series of works (Jiang et al., 2017; Jin et al., 2021a;
Du et al., 2021) on sample efficient online RL.[-
-]
The setting is known to capture a wide class of models including linear MDPs, linear Bellman complete models, low-rank MDPs, reactive POMDPs, and more.
As a technical note, the main paper focuses on the “Q-type” version of the bilinear model, but the algorithm and proofs easily extend to the “V-type” version.
See Appendix B for details.
Theorem 1 (Cumulative subpotimality) .
Fix δ ∈ (0 , 1) , number of total offline samples m off =
HT/d and number of online samples at each iteration m on = H 2 , and suppose that the underlying MDP admits Bilinear rank d , and the function class F satisfies Assumption 1.
Then with probability at least 1 − δ , Algorithm 1 obtains the following bound on cumulative subpotimality w.r.t.
any comparator policy π e ,
[Equation]
where π t = π f t is the greedy policy w.r.t.
f t at round t .
A standard online-to-batch conversion (Shalev-Shwartz & Ben-David, 2014) immediately gives the following sample complexity guarantee for Algorithm 1 for finding an ϵ -suboptimal policy w.r.t.
the optimal policy π ∗ for the underlying MDP.
Corollary 1 (Sample complexity) .
Under the assumptions of Theorem 1 if C π ∗ < ∞ then Algorithmcan find an ϵ -suboptimal policy (cid:98) π for which V π ∗ − V (cid:98) π ≤ ϵ with total sample complexity:
[Equation]
The results formalize the statistical properties of Hy-Q.
In terms of sample complexity, a somewhat unique feature of the hybrid setting is that both transfer coefficient and bilinear rank parameters are relevant, whereas these (or related) parameters typically appear in isolation in offline and online RL respectively.
In terms of coverage, Theorem 1 highlights an “oracle property” of Hy-Q: it competes with any policy that is sufficiently covered by the offline dataset.
We also highlight the computational efficiency of Hy-Q: it only requires solving least squares problems over the function class F .
To our knowledge, no purely online or purely offline methods are known to be efficient in this sense, except under much stronger “uniform” coverage conditions.
5.1 The Linear Bellman Completeness Model
We next showcase one example of low bilinear rank models: the popular linear Bellman complete model which captures the linear MDP model (Yang & Wang, 2019; Jin et al., 2020), and instantiate the sample complexity bound in Corollary 1.
Definition 3.
Given a feature function ϕ : S × A (cid:55)→ B d (1) , a model admits linear Bellman completeness if for any w ∈ B d ( B W ) , there exists a w ′ ∈ B d ( B W ) such that [Equation]
Note that the above condition implies that Q ⋆ h ( s, a ) = ⟨ w ⋆ h , ϕ
( s, a ) ⟩ with ∥ w ⋆ h ∥ 2 ≤ B W .
Thus, we can define a function class F h = {⟨ w h , ϕ
( s, a ) ⟩ : w h ∈ R d , ∥ w h ∥ 2 ≤ B W } which by inspection satisfies Assumption 1.
Additionally, this model is also known to have bilinear rank at most d (Du et al., 2021).
Thus, using Corollary 1 we immediately get the following guarantee:
Lemma 1.
Let δ ∈ (0 , 1) , suppose the MDP is linear Bellman complete, C π ∗ < ∞ , and consider
F h defined above.
Then, with probability 1 − δ , Algorithm 1 finds an ϵ -suboptimal policy with total sample complexity:
[Equation]
On the computational side, with F as in Lemma 1, the regression problem in Algorithm 1 reduces to a least squares linear regression with a norm constraint on the weight vector.
This can be solved by convex programming with complexity scaling polynomially in the parameters (Bubeck et al., 2015).
Remark 1 (Efficient implementation) . Algorithm 1 is both statistically and computationally efficient for linear Bellman complete models. This shows a computational benefit of hybrid RL over both pure online and pure offline RL. To the best of our knowledge, no prior methods are known to be computationally efficient in this setting.
Remark 2.
(Linear MDPs)
Since linear Bellman complete models generalize linear MDPs (Yang & Wang, 2019; Jin et al., 2020), Algorithm 1 is computationally efficient for linear MDPs as well even when the action space is large or continuous.
In contrast, prior algorithms rely on optimism or pessimism, which requires solving NP-hard optimization problems (particularly with |A| is large).
Remark 3.
(Relative condition number)
A common coverage metric in these linear MDP models is the relative condition number.
In Appendix A.5, we show that our coefficient C π is upper bounded by the relative condition number of π with respect to ν : E d π ∥ ϕ ∥ Σ − 1 ν , where Σ ν = E s,a ∼ ν ϕ ( s, a ) ϕ ⊤ ( s, a ) .
Concretely, we have C π ≤ (cid:113) max
h E d πh ∥ ϕ ∥ 2Σ − 1 νh . 5.2 Why don’t offline RL methods work? One may wonder why do pure offline RL methods fail to learn when the transfer coefficient is bounded, and why does online access help?
We illustrate with the MDP construction developed by Zhan et al.
; Chen & Jiang (2022), visualized in Figure 1.
Consider two MDPs { M 1 , M 2 } with H = 2 , three states { A, B, C } , two actions { L, R } and the fixed start state A .
The two MDPs have the same dynamics but different rewards.
In both, actions from state B yield reward 1 .
In M 1 , ( C, R ) yields reward 1 while ( C, L ) yields reward 1 in M 2 .
All other rewards are 0 .
In both M 1 and M 2 , an optimal policy is π ∗ ( A ) =
L and π ∗ ( B ) =
π ∗ ( C ) =
Uniform( { L, R } ) .
With F = { Q ⋆ 1 , Q ⋆ 2 } where Q ⋆j is the optimal Q function for M j , then one can easily verify that F satisfies Bellman completeness, for both MDPs.
Finally with offline distribution ν supported on states A and B only (with no coverage on state C ), we have sufficient coverage over d π ⋆ .
However, samples from ν are unable to distinguish between fand f 2 or ( M 1 and M 2 ), since state C is not supported by ν .
Unfortunately, adversarial tie-breaking may result the greedy policies of f 1 and f 2 visiting state C , where we have no information about the correct action.
This issue has been documented before, and in order to address it with pure offline RL, existing approaches require additional structural assumptions.
For instance, Chen &
Jiang (2022) assume that Q ⋆ has a gap, which usually does not hold when action space is large or continuous.
Xie et al. (2021a) assumes policy-dependent Bellman completeness for every possible policy π ∈ Π (which is much stronger than our assumption), and Zhan et al. (2022) assumes a somewhat non-interpretable realizability assumption on some “value” function that does not obey the standard Bellman equation.
In contrast, by combining offline data and online data, our approach focuses on functions that have small Bellman residual under both the offline distribution and the on-policy distributions, which together with the offline data coverage assumption, ensures near optimality.
It is easy to see that the hybrid approach will succeed Figure 1.
Experiments
In this section we discuss empirical results comparing Hy-Q to several representative RL methods on two challenging benchmarks.
Our experiments focus on answering the following questions:
Can Hy-Q efficiently solve problems that SOTA offline RL methods simply cannot?
Can Hy-Q, via the use of offline data, significantly improve the sample efficiency of online RL? 3.
Does Hy-Q scale to challenging deep-RL benchmarks?
Our empirical results provide positive answers to all of these questions.
To study the first two, we consider the diabolical combination lock environment (Misra et al., 2020; Zhang et al., 2022c), a synthetic environment designed to be particularly challenging for online exploration.
The synthetic nature allows us to carefully control the offline data distribution to modulate the difficulty of the setup and also to compare with a provably efficient baseline (Zhang et al., 2022c).
To study the third question, we consider the Montezuma’s Revenge benchmark from the Arcade Learning environment, which is one of the most challenging empirical benchmarks with high-dimensional image inputs, largely due to the difficulties of exploration.
Additional details are deferred to Appendix E.
Hy-Q implementation.
We largely follow Algorithm 1 in our implementation for the combination lock experiment. Particularly, we use a similar function approximation to Zhang et al. (2022c), and a minibatch Adam update on Eq. (1) with the same sampling proportions as in the pseudocode. For
[Figure] Montezuma’s Revenge, in addition to minibatch optimization, since the horizon of the environment is not fixed, we deploy a discounted version of Hy-Q. Concretely, the target value in the Bellman error is calculated from the output of a target network, which is periodically updated, times a discount factor. We refer the readers to Appendix E for more details. Baselines.
We include representative algorithms from four categories: (1) for imitation learning we use Behavior Cloning (B C ) (Bain & Sammut, 1995), (2) for offline RL we use Conservative Q-Learning (C QL ) (Kumar et al., 2020), (3) for online RL we use B RIEE (Zhang et al., 2022c) for combination lock 5 and Random Network Distillation (R ND ) (Burda et al., 2018) for Montezuma’s Revenge, and (4) as a Hybrid-RL baseline we use Deep Q-learning from Demonstrations (D QFD )
(Hester et al., 2018).
We note that D QFD and prior hybrid RL methods combine expert demonstrations with online interactions, but are not necessarily designed to work with general offline datasets.
Results summary.
Overall, we find that Hy-Q performs favorably against all of these baselines.
Compared with offline RL, imitation learning, and prior hybrid methods, Hy-Q is significantly more robust in the presence of a low quality offline data distribution.
Compared with online methods, Hy-Q offers order-of-magnitude savings in the total experience.
6.1 Combination Lock
The combination lock benchmark is depicted in Figure 3 and consists of horizon H = 100 , three latent states for each time step and 10 actions in each state. Each state has a single “good” action that advances down a chain of favorable states from which optimal reward can be obtained. A single incorrect action transitions to an absorbing chain with suboptimal value. The agent operates on high dimensional observations and must use function approximation to succeed.
This is an extremely challenging problem for which many Deep RL methods are known to fail (Misra et al., 2020), in part because (uniform) random exploration only has 10 − H probability of obtaining the optimal reward.
[Figure]
On the other hand, the model has low bilinear rank, so we do have online RL algorithms that are provably sample-efficient: B RIEE currently obtains state of the art sample complexity. However, its sample complexity is still quite large, and we hope that Hybrid RL can address this shortcoming. We are not aware of any experiments with offline RL methods on this benchmark.
We construct two offline datasets for the experiments, both of which are derived from the optimal policy π ⋆ .
In the optimal trajectory dataset we collect full trajectories by following π ⋆ with ϵ -greedy exploration with ϵ = 1 /H . In the optimal occupancy dataset we collect transition tuples from the state-occupancy measure of π ⋆ with random actions. 6 Both datasets have bounded concentrability coefficients (and hence transfer coefficients) with respect to π ⋆ , but the second dataset is much more challenging since the actions do not directly provide information about π ⋆ , as they do in the former. The results are presented in Figure 2.
First, we observe that Hy-Q can reliably solve the task under both offline distributions with relatively low sample complexity (500k offline samples and ≤ 25m
[Figure]
online samples).
In comparison, B C fails completely since both datasets contain random actions.
C QL can solve the task using the trajectory-based dataset with a sample complexity that is comparable to the combined sample size of Hy-Q.
However, C QL fails on the occupancy-based dataset since the actions themselves are not informative.
Indeed the pessimism-inducing regularizer of C QL is constant on this dataset and so the algorithm reduces to F QI .
Finally, Hy-Q can solve the task with a factor of 5-10 reduction in (online and offline) samples when compared with B RIEE .
This demonstrates the robustness and sample efficiency provided by hybrid RL.
6.2 Montezuma’s Revenge
To answer the third question, we turn to Montezuma’s Revenge, an extremely challenging imagebased benchmark environment with sparse rewards.
We follow the setup from Burda et al.
and introduce stochasticity to the original dynamics: with probability 0.25 the environment executes the previous action instead of the current one.
For offline datasets, we first train an “expert policy” π e via R ND to achieve V π e ≈ 6400 .
We create three datasets by mixing samples from π e with those from a random policy: the easy dataset contains only samples from π e , the medium dataset mixes in a 80/20 proportion (80 from π e ), and the hard dataset mixes in a 50/50 proportion.
Here we record full trajectories from both policies in the offline dataset, but measure the proportion using the number of transition tuples instead of trajectories.
We provide 0.1 million offline samples for the hybrid methods, and 1 million samples for the offline and IL methods.
[Figure] Results are displayed in Figure 4.
C QL fails completely on all datasets.
D QFD performs well on the easy dataset due to the large margin loss (Piot et al., 2014) that imitates the policies in the offline dataset.
However, D QFD ’s performance drops as the quality of the offline dataset degrades (medium), and fails when the offline dataset is low quality (hard).
We also observe that B C is a competitive baseline in the first two settings, and thus we view these problems as relatively easy to solve.
Hy-Q is the only method that performs well on the hard dataset.
Note that here, BC’s performance is quite poor.
We also include the comparison with R ND in Figure 5: with only 100k offline samples from any of the three datasets, Hy-Q is over 10x more efficient in terms of online sample complexity.
Conclusion
We demonstrate the potential of hybrid RL with Hy-Q, a simple, theoretically principled, and empirically effective algorithm.
Our theoretical results showcase how Hy-Q circumvents the computational issues of pure offline or online RL, while our empirical results highlight its robustness and sample efficiency.
Yet, Hy-Q is perhaps the most natural hybrid algorithm, and we are optimistic that there is much more potential to unlock from the hybrid setting.
We look forward to studying this in the future.
Reproducibility Statement.
For our theory results, we provide detailed proof in the Appendices.
For experiments, we submit anonymous code in the supplemental materials.
Our (offline) dataset can be reproduced with the attached instructions, and our results could be reproduced with the given random seeds.
For more details, we include implementation, environment and computation hardware details in the Appendices, along with hyperparameters for both our method and the baselines.
A Proofs for Section
Throughout the appendix, we give the proofs under generalizations of the assumptions made in the main body.
Firstly, we assume that Realizability and Bellman completeness only hold approximately:
Assumption 2 (Approximate Realizability and Bellman completeness) .
For any h , there exists a (cid:101) Q ⋆h ∈ F h such that ∥ (cid:101) Q ⋆h − Q ⋆h ∥ ∞ ≤ γ .
Additionally, for any f h +1 ∈ F h +1 , there exists a (cid:98) f h ∈ F h such that ∥ (cid:98) f h − T f h +1 ∥ ∞ ≤ γ .
Additional notation.
Throughout the appendix, we define the feature covariance matrix Σ t ; h as
[Equation]
Furthermore, given a distribution β ∈ ∆( S × A ) and a function f : S × A (cid:55)→ R , we denote its weighted ℓ 2 norm as ∥ f ∥ 2 β :=
(cid:112) E s,a ∼ β f 2 ( s, a ) .
A.1 Supporting technical results
We first develop some useful technical results for our main proofs.
The following form of Freedman’s inequality is a modification of a similar inequality in (Beygelzimer et al., 2011).
Lemma 2 (Freedman’s Inequality) .
Let { X 1 , . . .
, X T } be a sequence of non-negative random variables adapted to the increasing filtration {G 1 , . . .
, G T } of the σ -algebra G .
Further, suppose that | X t | ≤ R almost surely for all t ≤ T .
Then, for any δ > 0 and λ ∈ [0 , 1 / 2 R ] , with probability at least 1 − δ , (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) T (cid:88) t = X
[Equation]
] (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≤ λ
T (cid:88)
[Equation] where E t = E [ · | G t ] denotes the conditional expectation w.r.t.
the filtration G t .
Proof.
Define the random variable Z t =
X t − E t − 1 [ X t ] .
Clearly, { Z t } Tt =1 is a martingale difference sequence adapted to the filtration {F t } Tt =1 .
Furthermore, we have that for any t , | Z t | ≤ 2 R and that
[Equation]
where the last inequality holds because | X t | ≤ R .
Using the form of Freedman’s inequality in Beygelzimer et al.
(2011, Lemma 9), we get that for any λ ∈ [0 , 1 / 2 R ] ,
[Equation]
Plugging in the form of Z t and using (4), we get the desired statement.
The following gives a generalization bound for least squares regression when the samples are adapted to an increasing filtration (and are not necessarily i.i.d.).
The proof follows similarly to Agarwal et al.
(2019, Lemma A.11).
Lemma 3.
(Least squares generalization bound)
Let R > 0 , δ ∈ (0 , 1) and let H : X (cid:55)→ [ − R, R ] be a class of real valued functions.
Let D =
( x 1 , y 1 ) , . . .
, ( x T , y T ) } be a dataset of n points where ( x T , . . .
, x T ) is adapted to the increasing filtration {G 1 , . .
, G T } of the σ -algebra G , and y t is sampled via
[Equation]
where the function h ∗ satisfies approximate realizability i.e.
[Equation]
and { ϵ i } ni =1 are independent random variables adapted to the filtration G t .
Additionally, suppose that max t | y t | ≤ R and max x | h ∗ ( x ) | ≤ R .
Then the least square solution (cid:98) h ← argmin h ∈H [Equation]
Tt = [Equation] h ( x t ) − y t ) 2 satisfies with probability at least 1 − δ ,
[Equation]
Proof.
Consider any fixed function h ∈ H and define the random variable
[Equation]
and note that
[Equation]
where the last line holds because E [ y t | x t ]
= h ∗ ( x t ) and the notation E t − 1 [ · ] :=
E [ · | F t ] .
Furthermore, we also have that
[Equation]
[Equation] Note that the sequence of random variables (cid:8) Z h 1 , . . .
, Z hT (cid:9) are adapted to the filtration {G 1 , . . .
, G T } and that (cid:12)(cid:12)
Z ht (cid:12)(cid:12) ≤ 4 R 2 .
Thus, using Lemma 2 we get that for any λ ∈ [0 , 1 / 8 R 2 ] and δ > 0 , with probability at least 1 − δ ,
[Equation]
where the last inequality uses (5) and (6).
Setting λ
= 1 / 64 R 2 in the above, and taking a union bound over h , we get that for any h ∈ H and δ > 0 , with probability at least 1 − δ ,
[Equation]
Rearranging the terms and using (5) in the above implies that,
[Equation]
and
[Equation]
For the rest of the proof, we condition on the event that (7) holds for all h ∈ H .
Define the function (cid:101) h :=
argmin h ∈H (cid:80)
Tt =1 E
[Equation]
( h ( x t ) − h ∗ ( x t )) 2 (cid:3) .
Using (7), we get that
T (cid:88) t
[Equation]
where the last inequality follows from the approximate realizability assumption.
Let (cid:98) h denote the least squares solution on dataset { ( x t , y t ) } t ≤ T .
By definition, we have that
[Equation]
Combining the above two relations, we get that
[Equation]
Finally, using (7) for the function (cid:98) h , we get that
[Equation]
where the last inequality uses the relation (8).
Lemma 4 (Performance difference lemma) .
For any function f = ( f 0 , . . .
, f H − 1 ) where f h : S × A (cid:55)→ R and h ∈
[ H − 1] , we have
[Equation]
where we define f H ( s, a ) = 0 for all s, a .
Proof.
We will prove using induction that for all h ∈
[ H ] ,
[Equation]
where we additionally define V π f H ( s )
=
for any s ∈ S .
For the base case ( h = H ) [Equation] we note that f H ( s, a ) = [-
-]0 and V π f H = 0 .
Thus, E s ∼ d πfH [max a f H ( s, a ) − V π f H ( s )]
= 0 and (9) holds.
For the induction step, suppose (9) holds for step h
+ 1 .
We note that
[Equation]
where the above equalities follows from the fact that policy π f is defined such that at step h ,
π f ( s ) = argmax a f h ( s, a ) for any s ∈ S .
We further note that
[Equation]
where the first line simply follows by expanding T f h
+1 ( s, a ) , and the second equality follows by expanding V π f h ( s ) and noting that a = π f ( s ) = argmax a f h ( s, a ) .
Plugging in (11) in (10), we get
[Equation]
where the last line holds due to induction hypothesis for step h
+ 1 .
Thus, (9) holds for all h ∈
[ H ] , and in particular for h = 0 .
An application of Triangle inequality gives the desired result.
Lemma 5 (Pseudo-optimism) .
Let π e
=
( π e 0 , . . .
, π eH − 1 ) be a comparator policy, and consider any value function f = ( f 0 , . . .
, f H − 1 ) where f h : S × A (cid:55)→ R .
Then,
[Equation]
where we defined f H ( s, a )
=
for all s, a .
Proof.
The proof is similar to the proof of Lemma 4, and we will prove via induction that for all h ∈ [ H ] ,
[Equation]
The base case for h
= H follows by definition as V π e H ( s ) =
and f H ( s, a ) =
for all s, a .
We next show the induction step.
Suppose (12) holds for time step h + 1 ≤ H .
In the following, we will show that (12) also holds for time step h .
Note that
[Equation]
where the inequality in the first line above follows by replacing max a by a ∼ π eh ( s ) .
For the first term in the above, note that
[Equation]
where the first equality follows by expanding V π e h
[Equation] s ) and T f h[-+
-]
+1 ( s, a ) .
The second equality is due to linearity of expectations.
Using (14) in (13), we get
[Equation]
≤
H − 1 (cid:88)
i
[Equation]
h [Equation] [Equation] where the last line holds due to induction hypothesis for time step h
+1 .
This completes the induction step showing the (12) holds for all h ∈
[ H ] .
The claimed bound follows by invoking (12) with h = 0 and noting that d π e 0 = d 0 .
The following result directly follows from the elliptical potential lemma (Lattimore & Szepesv´ari, 2020, Lemma 19.4).
Lemma 6.
Let X h ( f 1 ) , . . .
, X h ( f T ) ∈
R d be a sequence of vectors with ∥ X h ( f t ) ∥
≤ B X < ∞ for all t ≤ T .
Then,
[Equation]
where the matrix Σ t ; h := (cid:80) tτ
=1 X h ( f τ ) X h
( f τ ) ⊤
+ λ I for t ∈
[ T ] and λ ≥ B 2 X .
Proof.
Since λ ≥ B 2 X , we have that
[Equation]
Thus, using elliptical potential lemma (Lattimore & Szepesv ´ ari, 2020, Lemma 19.4), we get that T (cid:88)
[Equation]
The desired bound follows from Jensen’s inequality which implies that
T
[Equation]
A.2 Proof of Theorem
Before delving into the proof,
we first state
that following generalization bound for FQI.
Lemma 7 (Bellman error bound for FQI) .
Let δ ∈ (0 , 1) and let for h ∈ [ H − 1] and t ∈ [ T ] , f t +1 h be the estimated value function for time step h computed via least square regression using samples in the dataset (cid:0) D ν h , D 1 h , . .
, D t h (cid:1) in (1) in the iteration t of Algorithm 1.
Then, with probability at least 1 − δ , for any h ∈ [ H − 1] and t ∈ [ T ] ,
[Equation]
and t (cid:88)
[Equation]
where ν h denotes the offline data distribution at time h , and the distribution µ τh ∈ ∆( s, a ) is defined such that s ∼ d π τ h and a ∼ π est ( s ; f t ) .
Proof.
Fix t ∈
[ T ] , h ∈
[ H − 1] and f t +1 h +1 ∈ F h
+
and consider the regression problem f t +1 h ← argmin f ∈F h (cid:40) [Equation] E D νh
[Equation]
f [Equation] The above can be thought of as regression problem on dataset D consisting of n = m off + t · m on samples { ( x i , y i ) } i ≤ n where
[Equation]
In particular, we define D such that the first m off samples { ( x i , y i ) } i ≤ m off =
D νh , the next m on samples { ( x i , y i ) } m off + m on i = m off +
= D 1 h , and so on where the samples { ( x i , y i ) } m off
+ τm on i = m off +( τ − 1) m on +
= D τh .
Note that: (a) for any sample ( x = ( s h , a h ) , y =
( r + max a f t +1 h
+
( s h +1 , a )))
in D , we have that
[Equation]
where the last line holds since the approximate bellman completeness assumption implies existence of such a function g , (b) for any sample, | y | ≤ B and f ( s, a ) ≤ B for all s, a , (c) our construction of D implies that the samples { x t , y t } t ≤ T are adapted to an increasing filtration G t (given by the natural filtration induced by the previous random variables), finally (d) the samples in D νh are drawn from the offline distribution ν h , and the samples in D τh are drawn such that s h ∼ d π t h and a h ∼ π est
( s h ; f t ) .
Thus, using Lemma 3, we get that the least square regression solution f t +1 h satisfies
[Equation]
Using the property-(d) in the above, we get that
[Equation]
where the distribution µ τh ∈ ∆( s, a ) is defined by sampling
s ∼ d π τ h and a ∼ π est ( s ; f t ) .
Taking a union bound over h ∈ [ H − 1] and t ∈
[ T ] , and bounding each term separately, gives the desired statement.
We next note a change in distribution lemma which allows us to bound expected bellman error under the ( s, a ) distribution generated by f t in terms of the expected square bellman error w.r.t.
the previous policies data distribution, which is further controlled using regression.
Lemma 8.
For any t ≥ 0 and h ∈
[ H − 1] , we have
[Equation]
where Σ − 1 t − 1 is defined in (3) .
Proof.
Using Cauchy-Schwarz inequality, we get that
[Equation]
f
[Equation] [Equation] where the inequality in the second last line holds by plugging in the bound on ∥ W h ( f t ) ∥ , and the last line holds by using Definition 2 which implies that
[Equation]
where the last inequality is due to Jensen’s inequality.
We now have all the tools to prove Theorem 1.
We first restate the bound with the exact dependence on problem parameters.
Theorem (Theorem 1 restated) .
Let m off =
HT/d and m on = 2 H 2 .
Then, with probability at least 1 − δ , the total complexity of Algorithm 1 is bounded as T
[Equation]
Proof of Theorem 1.
Let π e be any comparator policy with bounded transfer coefficient i.e.
[Equation]
We start by noting that
T
[Equation]
For the first term in the right hand side of (16), note that using Lemma 5 for each f t for 1 ≤ t ≤ T , we get
T (cid:88)
[Equation]
where the second inequality follows from plugging in the definition of C π e in (15).
The last line follows from Lemma 7.
For the second term in (16), using Lemma 4 for each f t for 1 ≤ t ≤ T , we get T [Equation]
[Equation]
where the second line follows from Definition 2, the third line follows from Lemma 8 and by plugging in the bound in Lemma 7.
Using the bound in Lemma 6 in the above, we get that
[Equation]
where the second line follows by plugging in λ
= B 2 X .
Combining the bound (17) and (19), we get that
[Equation]
Plugging in the values of ∆ on and ∆ off in the above, we get that
[Equation]
Setting γ =
(under exact Bellman completeness), we get that
T (cid:88)
[Equation]
Setting m off =
HT/d and m on = 2 H 2 in the above gives the total complexity bound
[Equation]
Proof of Corollary 1.
We next convert the above total complexity bound into sample complexity via a standard online-to-batch conversion.
Setting π e =
π ∗ in (20) and defining the policy (cid:98) π = Uniform (cid:0)(cid:8) π 1 , . .
, π T (cid:9)(cid:1) , we get that
[Equation]
δ (cid:19)(cid:33) .
Thus, we get that for T ≥ (cid:101)
O
[Equation]
, we get that
[Equation]
In these T iterations, the total number of offline samples used is
[Equation]
Furthermore, the number of offline samples collected is
[Equation]
A.3 V-type vs Q-type Bilinear Rank
Our previous result focus on the Q-type bilinear model.
Here we provide the V-type Bilinear rank definition.
This V-type Bilinear rank definition is basically the same as the low Bellman rank model proposed by Jiang et al.
Definition 4 (V-type Bilinear model) .
Consider any pair of functions ( f, g ) with f, g ∈ F .
Denote the greedy policy of f as π f = { π fh := argmax a f h ( s, a ) , ∀ h } .
We say that the MDP together with the function F admits a bilinear structure of rank d if for any h ∈ [ H − 1] , there exist two (unknown) mappings
X h : F (cid:55)→ R d and W h : F (cid:55)→ R d with max f ∥ X h ( f ) ∥ 2 ≤ B X and max f ∥ W h ( f ) ∥ 2 ≤ B W , such that:
[Equation]
Note that different from the Q-type definition, here the action a is taken from the greedy policy with respect to g .
This way max a g ( s, a ) can serve as an approximation of V ⋆ – thus the name of V -type.
To make Hy-Q work for the V-type Bilinear model, we only need to make slight change on the data collection process, i.e., when we collect online batch D h , we sample s ∼ d π t h , a ∼ Uniform ( A ) , s ′ ∼ P ( ·| s, a ) .
Namely the action is taken uniformly randomly here.
We skip the detailed discussion of the V-type model and its sample complexity here.
We refer readers to Du et al. ; Jin et al. (2021a) for more detailed discussion.
A.4 Low-rank MDP
In this section, we briefly introduce the low-rank MDP model which is captured by the V-type bilinear model.
(Du et al., 2021).
Unlike the linear MDP model discussed in Section 5.1, low-rank MDP does not assume the feature ϕ is known a priori.
Definition 5 (Low-rank MDP) .
A MDP is called low-rank MDP if there exists µ ⋆ : S (cid:55)→ R d , ϕ
⋆ : S × A (cid:55)→ R d , such that the transition dynamics P ( s ′ | s, a )
= µ ⋆ ( s ′ )
⊤ ϕ ⋆ ( s, a ) for all s, a, s ′ .
We additionally assume that we are given a realizable representation class Φ such that ϕ ⋆ ∈ Φ , and
that sup s,a ∥ ϕ ⋆ ( s, a ) ∥ 2 ≤ 1 , and ∥ f ⊤ µ ⋆ ∥ 2 ≤ √ d for any f : S (cid:55)→ [ − 1 , 1] .
Consider the function class F h = { w ⊤ ϕ ( s, a ) :
ϕ ∈ Φ , w ∈ B d ( B W ) } , and through the bilinear decomposition we have that B W ≤ 2 √ d .
By inspection, we know that this function class satisfies Assumption 1.
Furthermore, it is well known that the low rank MDP model has V -type bilinear rank of at most d (Du et al., 2021).
For low-rank MDP, the transfer coefficient C π is upper bounded by a relative condition number style quantity defined using the unknown ground truth feature ϕ
⋆ (see Lemma 11).
On the computational side, Algorithm 1 (with the modification of a ∼ Uniform ( A ) in the online data collection step) requires to solve a least squares regression problem at every round.
The objective of this regression problem is a convex functional of the hypothesis f over the constraint set F .
While this is not fully efficiently implementable due to the non-convex constraint set F , our regression problem is still much simpler than the oracle models considered in the prior works for this model (Agarwal et al., 2020a; Uehara et al., 2021;
Modi et al., 2021).
A.5 Bounds on transfer coefficient
Note that C π takes both the distribution shift and the function class into consideration, and is smaller than the existing density ratio based concentrability coefficient (Kakade & Langford, 2002; Munos & Szepesv´ari, 2008; Chen & Jiang, 2019) and also existing Bellman error based concentrability coefficient Xie et al. (2021a).
We formalize this in the following lemma.
Lemma 9.
For any π and offline distribution ν ,
[Equation]
Proof.
Using Jensen’s inequality, we get that
[Equation]
where the second line follows from the Mediant inequality and the last line holds
whenever sup h,s,a d πh ( s,a ) ν h
( s,a ) ≥ 1 .
Next we show that in the linear Bellman complete setting, C π is bounded by the relative condition number.
Lemma 10.
Consider the linear Bellman complete setting (Definition 3) with known feature ϕ .
Suppose that the feature covariance matrix induced by offline distribution ν : Σ
ν h := E
[Equation]
[Equation] ] is invertible.
Then for any policy π , we have
[Equation]
Proof.
Repeat the argument in Lemma 9, we have
[Equation]
Recall that in linear Bellman complete setting, we can write f as w ⊤ ϕ , and for any w that defines f , there exists w ′ such that T f = w ′⊤ ϕ .
Now we proceed to low-rank MDPs.
We show that for low-rank MDPs, C π is bounded by the partial feature coverage.
Lemma 11.
Consider the low-rank MDP setting (Definition 5) where the transition dynamics P is given by P ( s ′ | s, a ) = ⟨ µ ⋆ ( s ′ ) ,
ϕ ⋆ ( s, a ) ⟩ for some µ ⋆ , ϕ ⋆ ∈ R d .
Suppose that the offline distribution ν
=
( ν 0 , . .
, ν H − 1 ) is such that max h max s,a ν h ( s )
ν h ( s,a ) ≤ α for any s, a .
Furthermore, suppose that ν is induced via trajectories i.e. ν 0 ( s ) =
d 0 ( s ) and ν h ( s ) =
E ¯ s, ¯ a ∼ ν [Equation] − 1 P ( s | ¯ s, ¯ a ) for any h ≥ 1 , and that the feature covariance matrix Σ [Equation]
[Equation]
s [Equation] s [Equation]
a [Equation] ⊤ ] is invertible.
Then for any policy π , we have
[Equation]
Proof.
We first upper bound the numerator separately.
First note that for h = 0 ,
[Equation]
where the last inequality follows from our assumption since max s,a d π
[Equation]
s,
[Equation]
[Equation]
≤ max s
[Equation]
[Equation]
≤ α .
Next
[Equation]
for any h ≥
[Equation]
we note that backing up one step and looking at the pair ¯ s, ¯ a that lead to the state s , we get that
[Equation]
[Equation]
where the last line follows from an application of Cauchy-Schwarz inequality.
For the term inside the expectation in the right hand side above, we note that for any s, a ,
[Equation]
[Equation]
23) where ( i ) follows by expanding the norm , ( ii ) follows an application of Jensen’s inequality, ( iii ) is due to our assumption that the offline dataset is generated using trajectories such that ν h ( s )
= E ¯ s, ¯ s ∼ ν h − 1 [ P ( s | ¯ s, ¯ a )] .
Finally, ( iv ) follows from the definition of α .
Plugging (23) in (22), we get that for h ≥ 1 ,
[Equation]
We are now ready to bound the transfer coefficient.
First note that using (21), for any f , E s
[Equation]
Furthermore, for any f , using (24), we get that
[Equation]
where the last line holds for an appropriate choice of λ (e
[Equation]
g. λ = 0 ).
Combining the above two bounds in the definition of C π we get that
[Equation]
B Low Bellman Eluder Dimension problems
To capture the complexity of the MDP and the function class we use the notion of Bellman Eluder dimension introduced by Jin et al.
(2021a).
This complexity measure is a distributional version of the Eluder dimension applied to the class of Bellman residuals w.r.t.
F .
We first introduce the key definitions:
Definition 6 ( ε -independence between distributions (Jin et al., 2021a)) .
Let G be a class of functions defined on a space X , and ν, µ 1 , . . .
, µ n be probability measures over X .
We say ν is ε -independent of { µ 1 , µ 2 , . . . , µ n } with respect to G if there exists g ∈ G such that (cid:112)(cid:80) n i {+=
+}
( E µ i [ g ]) 2 ≤ ε , but | E ν [ g ] | > ε .
Definition 7 (Distributional Eluder (DE) dimension) .
Let G be a function class defined on X , and P be a family of probability measures over X .
The distributional Eluder dimension dim DE ( F , P , ε ) is the length of the longest sequence {
ρ 1 , . . . , ρ n } ⊂ Π such that there exists ε ′ ≥ ε where ρ i is ε ′ -independent of { ρ 1 , . .
, ρ i − 1 } for all i ∈ [ n ] .
Definition 8 (Bellman Eluder (BE) dimension (Jin et al., 2021a)) .
Given a value function class F , let G h :=
( f h − T f h +1 | f ∈ F h , f h +1 ∈ F h +1 ) be the set of Bellman residuals induced by F at step h , and P =
{P h } Hh =1 be a collection of H probability measure families over X × A .
The ϵ -Bellman Eluder dimension of F with respect to P is defined as
[Equation]
We also note the following lemma that controls the rate at which Bellman error accummulates.
Lemma 12 (Lemma 41, (Jin et al., 2021a)) .
Given a function class G defined on a space X with sup g ∈G ,x ∈X | g ( x ) | ≤ C , and a set of probability measures P over X .
Suppose that the sequence { g k } Kk =1 ⊂ G and { µ k } Kk =1 ⊂ P satisfy that (cid:80)
{+k − 1 t =
+}
( E µ t [ g k ]) 2 ≤ β for all k ∈ [ K ] .
Then, for all k ∈ [ K ] and γ > 0 ,
[Equation]
We next state our main theorem whose proof is similar to that of Theorem 1.
Theorem 2 (Cumulative subpotimality) .
Fix δ ∈ (0 , 1) , m off =
HT/d and m on = H 2 , and suppose that the underlying MDP admits Bellman eluder dimention d , and the function class F satisfies Assumption 1.
Then with probability at least 1 − δ , Algorithm 1 obtains the following bound on cumulative subpotimality w.r.t.
any comparator policy π e ,
T (cid:88)
[Equation]
where π t = π f t is the greedy policy w.r.t.
f t at round t and d = dim BE ( F , P F , 1 / √ T ) .
Here P F is the class of occupancy measures that can be be induced by greedy policies w.r.t.
value functions in F .
Proof.
Repeating the analysis till (18) in the proof of Theorem 1, we get that T (cid:88)
[Equation]
Using the bound in Lemma 7 and Lemma 12 in the above, we get that
T
[Equation]
where G h :
=
( f h − T f h +1 | f ∈ F h , f h +1 ∈ F h +1 ) denotes the set of Bellman residuals induced by F at step h , and P =
{P F ; h } Hh =1 is the collection of occupancy measures at step h induced by greedy policies w.r.t.
value functions in F .
We set γ = 1 / √ T and define d = dim BE ( F , P , γ ) =
max h dim DE ( G h , P F ; h , γ ) .
Ignoring the lower order terms, we get that
T
[Equation]
where ≲ hides lower order terms, multiplying constants and log factors.
Setting m off =
HT/d and m on =
H 2 , we get that
[Equation]
[Figure] [Figure]
C Additional Ablation Experiments C.1 Ablation on offline buffer ratio
In this section, we analyze how D QFD performs when the offline buffer sampling ratio schedule is replaced by Hy-Q’s offline buffer sampling ratio schedule. The experiment is conducted on the hard dataset and all setups remain fixed as in the main experiment section. We report the training curves in Figure 6.
We observe that switching the offline buffer ratio schedule does not improve the performance of D QFD on the hard dataset, and we believe this is because of the usage of the imitation learning loss in their offline data loss. C.2 Ablation on pretraining In this section, we analyze the performance difference once we allow Hy-Q to perform pretraining on the offline data before the online stage. For the pretraining, we use the same FQI/DQN style update as Hy-Q trains the offline data during the online stage.
We use the same number of pretrain steps as
D QFD .
We present the result in Figure 7.
We observe that there is no major performance difference with or without the pretraining stage for Hy-Q.
We believe this is because FQI algorithm provably fails when the offline data does not have sufficient global coverage (in fact the hard dataset indeed does not contain global coverage).
D Comparison with previous works
As mentioned in the main text, many previous empirical works consider combining offline expert demonstrations with online interaction (Rajeswaran et al., 2017;
Hester et al., 2018; Nair et al., 2018, 2020; Vecerik et al., 2017; Lee et al., 2022; Jia et al., 2022;
Niu et al., 2022).
Thus the idea of performing RL algorithm on both offline data (expert demonstrations) and online data is also explored in some of the previous works, for example, Vecerik et al.
(2017) runs DDPG on both the online and expert data, and Hester et al.
(2018) uses DQN on both data but with an additional supervised loss.
Since we already compared with Hester et al.
(2018) in the experiment, here we focus on our discussion with Vecerik et al.
We first emphasize that Vecerik et al.
(2017) only focuses on expert demonstrations and their experiments entirely rely on using expert demonstrations, while we focus on more general offline
[Figure]
[Figure]
Note that focus dataset that is not necessarily coming from experts.
Said though, the DDPG-based algorithm from
Vecerik et al.
(2017) potentially can be used when offline data is not from experts.
Although the algorithm from Vecerik et al.
(2017) and Hy-Q share the same high-level intuition that one should perform RL on both the datasets, there are still a few differences : (1) Hy-Q uses Q-learning instead of deterministic policy gradients; note that deterministic policy gradient methods cannot be directly applied to discrete action setting; (2) Hy-Q does not require n-step TD style update, since in off-policy case, without proper importance weighting, n-step TD could incur strong bias.
While proper tuning on n could balance bias and variance, one does not need to tune such n-step at all in Hy-Q; (3) The idea of keeping a non-zero ratio to sample offline dataset is also proposed in Vecerik et al.
Our buffer ratio is derived from our theory analysis but meanwhile proves the advantage of the similar heuristic applied in Vecerik et al.
In their experiment, Vecerik et al.
(2017) only considers expert demonstrations.
In our experiment, we considered offline datasets with different amounts of transitions from very low-quality policies and showed Hy-Q is robust to low-quality transitions in offline data.
Note that some of the differences may seem minor on the implementation level, but they may be important to the theory.
Regarding the experiments, our experimental evaluation adds the following insights over those in Vecerik et al. (2017): (i) hybrid methods can succeed without expert data, (ii) hybrid methods can succeed in hard exploration discrete-action tasks, (iii) the core algorithm ( Q -learning vs DDPG) is not essential although some details may matter. Due to the similarity between the two methods, we believe some of these insights may also translate to Vecerik et al.
and we expect that the choice between Hy-Q and Hy-DDPG will be environment specific, as it is with the purely online versions of these methods. In some situations, Q -learning works does not immediately imply Deterministic policy gradient methods work, nor vice versa.
Nevertheless, it is beyond the scope of this paper to rigorously verify this claim and we deem the study of Actor-critic algorithms in Hybrid RL setting an interesting future direction.
In this section we provide a detailed description of combination lock experiment.
The combination lock environment has a horizon H and 10 actions at each state.
There are three latent states z i,h , i ∈ { 0 , 1 , 2 } for each timestep h , where z i,h , i ∈ { 0 , 1 } are good states and z 2 ,h is the bad state. For each good state, we randomly pick a good action a
i,h , such that in latent state z
i,h , i ∈ { 0 , 1 } ,
Algorithm
Require:
Value function class: F , #iterations: T , Offline dataset D νh of size m off for h ∈
[ H − 1] .
1: Initialize f 1 h ( s, a ) = 0 .
2: for t = 1 , . . .
, T do 3:
Let π t be the greedy policy w.r.t.
f t i.e., π th ( s ) = argmax a f th ( s, a ) .
Online collection 4:
For each h , collect m on online tuples D th ∼ d π t h ◦ Uniform ( A ) .
// FQI using both online and offline data 5:
Set f t +1 H ( s, a ) = 0 .
6: for h
= H − 1 , . .
. , 0 do 7:
Estimate f t +1 h using least squares regression on the aggregated data: f t +1 h ← argmin f ∈F h (cid:40) (cid:98) E D νh ( f ( s, a ) − r − max a ′ f t +1 h
+1 ( s ′ , a ′ ))
2 + t (cid:88)
τ
=1 (cid:98)
E D τh ( f ( s, a ) − r − max a ′ f t +1 h
+1 ( s ′ , a ′ ))
(cid:41) 8: end for 9: end for taking the good action a i,h will result in 0.5 probability of transiting to z 0 ,h +1 and 0.5 probability of transiting to z 1 ,h +1 while taking all other actions will result in a 1 probability of transiting to z 2 ,
h +1 . At z 2 ,h , all actions will result in a deterministic transition to z 2 , h +1 . For the reward, we give an optimal reward of 1 for landing z i,H , i ∈ { 0 , 1 } .
We also give an anti-shaped reward of 0.
for all transitions from a good state to a bad state. All other transitions have a reward of 0. The initial distribution is a uniform distribution over z 0 , 0 and z 1 , 0 .
The observation space has dimension 2 ⌈ log( H +1)
⌉ , created by concatenating a one-hot representation of the latent state and a one-hot representation of the horizon (appending 0 if necessary). Random noise from N (0 , 0 . 1) is added to each dimension, and finally the observation is multiplied by a Hadamard matrix.
Note that in this environment, the agent needs to perform optimally for all H timesteps to hit the final good state for an optimal reward of 1. Once the agent chooses a bad action, it will stay in the bad state until the end with at most 0.1 possible reward for the trajectory received while transitting from a good state to a bad state.
E.2 Implementation Details of Combination Lock experiment
We train H separate Q-functions for all H timesteps.
Our function class consists of an encoder and a decoder.
For the encoder, we feed the observation into one linear layer with 3 outputs, followed by a softmax layer to get a state-representation.
This design of encoder is intended to learn a one-hot representation of the latent state.
We take a Kronecker Product of the state-representation and the action, and feed the result to a linear layer with only one output, which will be our Q value.
In order to stabilize the training, we warm-start the Q-function of timestep h − 1 with the encoder from h Q-function of the current iteration and the decoder from the h − 1 Q-function of the last iteration, for each iteration of training.
One remark is that since combination lock belongs to Block MDPs, we require a V-type algorithm instead of the Q-type algorithm as shown in the main text.
The only difference lies in the online sampling process: instead of sampling from d π t h , for each h , we sample from d π t h ◦ Uniform ( A ) , i.e., we first rollin with respect to π t to timestep h − 1 , then take a random action, observe the transition and collect that tuple.
We provide Algorithm 2 for completeness.
Note that the only difference is in line 4.
For CQL, we implemented the variant of CQL-DQN and picked the peak in the learning curve to report in the main paper (so it should represent an upper bound of the performance of CQL).
E.3 Implementation Details of Montezuma’s Revenge experiment
In this section we provide the detailed algorithm for the discounted setting.
The overall algorithm is described in Algorithm 3.
For the function approximation, we use a class of convolutional neural networks (parameterized by class Θ ) as promoted by the original D QN paper.
We include several
Require:
Value function class: F (induced by Θ ), #iterations: T , Offline dataset D ν of size m off , discounted factor γ , target update frequency n target , learning rate α , offline sample ratio β , exploration rate ϵ , action space A .
4: Sample initial state s d5: for t = 1 , . .
, T do 6:
Let π be the ϵ -greedy policy w.r.t.
f θ i.e., π ( s ) = argmax a f θ ( s, a ) with probability 1 − ϵ and π ( s ) =
U ( A ) with probability ϵ .
Online collection 7:
Interact with the environment for one step: [Equation] = [Equation]
( s )
[Equation]
P ( s, a ) , [Equation] R ( s, a ) . Update online buffer: D = D ∪ { s, a, r, s ′ } .
Discounted minibatch FQI using 9: if t mod n value = 0 then 10:
With probability 1 − β : Sample a minibatch D with size n minibatch from online buffer D .
Otherwise: Sample a minibatch D with size n minibatch from offline buffer D ν .
Perform one-step gradient descent on D :
[Equation]
12: end if // 13: if t mod n target = [-
-]0 then 14:
Set target function to the current value function: ˜ f
= f θ .
15: end if 16:
Update s ← s ′ .
17: end for standard empirical design choices that have been practically proven to stabilize the training: we use Prioritize Experience Replay (Schaul et al., 2015) for our buffer.
We also add Double D QN (Van Hasselt et al., 2016) and Dueling D QN (Wang et al., 2016) during our Q-update.
We also observe that a decaying schedule on the offline sample ratio β and the exploration rate ϵ also helps provide better performance.
Note that an annealing β does not contradict to our comment in Section 4 on catastrophic forgetting because we set β to small after our online trajectory distribution covers d π e .
In addition, we also perform per step update instead of per episode update since this has been the popular design choice and leads to better efficiency in practice.
E.4 Baseline implementation E.4.1 Combination Lock
We use the open-sourced implementation https://github.com/BY571/CQL/tree/main/ CQL-DQN for C QL .
For B RIEE , we use the official code released by the authors: https:// github.com/yudasong/briee , where we rely on the code there for the combination lock environment.
E.4.2 Montezuma’s Revenge
We use the open-sourced implementation https://github.com/jcwleo/ random-network-distillation-pytorch for R ND .
For C QL , we use https: //github.com/takuseno/d3rlpy for their implementation of C QL for atari.
We use https://github.com/felix-kerkhoff/DQfD for D QFD .
For all baselines, we keep the hyperparameters used in these public repositories.
For C QL and D QFD , we provide the offline datasets as described in the main text instead of using the offline dataset provided in the public repositories.
7 All baselines are tested in the same stochastic environment setup as in Burda et al.
E.5 Hardware Infrastructure
We run our experiments on a cluster of computes with Nvidia RTX 3090 GPUs and various CPUs which do not incur any randomness to the results.
E.6 Hyperparameters E.6.1 Combination Lock
We provide the hyperparameters of Hy-Q in Table.
In addition, we provide the hyperparameters we tried for C QL baseline in Table.
[Table]
[Table]
E.6.2 Montezuma’s Revenge
We provide the hyperparameter of Hy-Q in Table.
We reuse many hyperparameter choices from D QFD .
Note that [ a, b ] denotes a decreasing/increasing schedule from a to b .
[Table]